librelist archives

« back to archive

Problem with Numpy ctypeslib?

Problem with Numpy ctypeslib?

From:
Jon Riehl
Date:
2012-06-07 @ 21:18
Hi all,

I'm seeing garbage inputs coming from ctypes wrapped Numpy code.  I
originally thought the problem was in the LLVM code I was generating.
After careful review, I seem to have isolated a problem without doing
any dynamic code generation whatsoever.  For example, given a simple
function such as the following (this function is in a C module I'm
building into a .so using gcc; there is no LLVM interaction
whatsoever):

int _getndim(PyArrayObject * arr)
{
  if (!PyArray_Check(arr)) return -1;
  return arr->nd;
}

I'm seeing the following in gdb:

(numba)jriehl@dingo:~/git/sandbox/llvm-py$ gdb python
...
(gdb) break _getndim
Function "_getndim" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (_getndim) pending.
(gdb) run
...
>>> import ctypes
>>> import numpy as np, numpy.ctypeslib as ctl
>>> numpyext = ctypes.CDLL('./numpyext.so')
>>> numpyext._getndim.restype = ctypes.c_int
>>> numpyext._getndim.argtypes = [ctl.ndpointer(flags = 'C_CONTIGUOUS')]
>>> numpyext._getndim(np.array([1,2,3]))

Breakpoint 1, _getndim (arr=0xe5f190) at numpyext.c:40
40	  if (!PyArray_Check(arr)) return -1;
(gdb) print *arr
$1 = {ob_refcnt = 1, ob_type = 0x2, data = 0x3 <Address 0x3 out of
bounds>, nd = 289, dimensions = 0x7ffff6c7a888,
  strides = 0x7ffff6c7a888, base = 0x0, descr = 0x0, flags = 0,
weakreflist = 0x0}
(gdb) cont
Continuing.

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff3ea5ec1 in _getndim (arr=0xe5f190) at numpyext.c:40
40	  if (!PyArray_Check(arr)) return -1;
(gdb)

Note all the invalid fields in the array data structure (nd = 289?!).
The documentation for numpy.ctypeslib is pretty sparse, so is there
something I'm missing here?

For the extremely curious and/or helpful, all the code I'm using here
is available at: https://github.com/jriehl/sandbox/ (see
.../llvm-py/numpyext.c first)

Thanks!
-Jon

Re: [numba] Problem with Numpy ctypeslib?

From:
Travis Oliphant
Date:
2012-06-08 @ 00:26
A few things: 

1) Make sure to use import_array before calling any C-API calls:  
importing numpy first should work
2)  numpy.ctypeslib has some utility routines to call *other* C-modules 
using NumPy arrays as arguments and for loading shared libraries across 
multiple platforms.   It doesn't really do much that you will need for 
this purpose.
3) You should not use ndpointer as the argtypes (that is for an argument 
that is expecting the void * data-buffer *not* the array object):    
numpy.ndarray should do just fine

Keep us posted. 

-Travis


On Jun 7, 2012, at 4:18 PM, Jon Riehl wrote:

> Hi all,
> 
> I'm seeing garbage inputs coming from ctypes wrapped Numpy code.  I
> originally thought the problem was in the LLVM code I was generating.
> After careful review, I seem to have isolated a problem without doing
> any dynamic code generation whatsoever.  For example, given a simple
> function such as the following (this function is in a C module I'm
> building into a .so using gcc; there is no LLVM interaction
> whatsoever):
> 
> int _getndim(PyArrayObject * arr)
> {
>  if (!PyArray_Check(arr)) return -1;
>  return arr->nd;
> }
> 
> I'm seeing the following in gdb:
> 
> (numba)jriehl@dingo:~/git/sandbox/llvm-py$ gdb python
> ...
> (gdb) break _getndim
> Function "_getndim" not defined.
> Make breakpoint pending on future shared library load? (y or [n]) y
> Breakpoint 1 (_getndim) pending.
> (gdb) run
> ...
>>>> import ctypes
>>>> import numpy as np, numpy.ctypeslib as ctl
>>>> numpyext = ctypes.CDLL('./numpyext.so')
>>>> numpyext._getndim.restype = ctypes.c_int
>>>> numpyext._getndim.argtypes = [ctl.ndpointer(flags = 'C_CONTIGUOUS')]
>>>> numpyext._getndim(np.array([1,2,3]))
> 
> Breakpoint 1, _getndim (arr=0xe5f190) at numpyext.c:40
> 40	  if (!PyArray_Check(arr)) return -1;
> (gdb) print *arr
> $1 = {ob_refcnt = 1, ob_type = 0x2, data = 0x3 <Address 0x3 out of
> bounds>, nd = 289, dimensions = 0x7ffff6c7a888,
>  strides = 0x7ffff6c7a888, base = 0x0, descr = 0x0, flags = 0,
> weakreflist = 0x0}
> (gdb) cont
> Continuing.
> 
> Program received signal SIGSEGV, Segmentation fault.
> 0x00007ffff3ea5ec1 in _getndim (arr=0xe5f190) at numpyext.c:40
> 40	  if (!PyArray_Check(arr)) return -1;
> (gdb)
> 
> Note all the invalid fields in the array data structure (nd = 289?!).
> The documentation for numpy.ctypeslib is pretty sparse, so is there
> something I'm missing here?
> 
> For the extremely curious and/or helpful, all the code I'm using here
> is available at: https://github.com/jriehl/sandbox/ (see
> .../llvm-py/numpyext.c first)
> 
> Thanks!
> -Jon

Re: [numba] Problem with Numpy ctypeslib?

From:
Jon Riehl
Date:
2012-06-08 @ 18:46
Hi Travis,

On Thu, Jun 7, 2012 at 7:26 PM, Travis Oliphant <travis@continuum.io> wrote:
> 3) You should not use ndpointer as the argtypes (that is for an argument
that is expecting the void * data-buffer *not* the array object):   
 numpy.ndarray should do just fine

I was using the website documentation which doesn't give an example
showing what was being called into.  The doc string for the module
makes this point clear.  So, this misunderstanding was clearly where I
went wrong.

Unless there is something better that I'm not aware of, I've got a
proof of concept working using ctypes.py_object for the argument type.
 At some point, I'd like to do some wrapper level type checks, but now
is not that time.  Per my erroneous code, we'd at least want to ensure
that the argument is actually a PyArrayObject, and its data is "C
contiguous" since the code generator will assume this for now.

Thanks!
-Jon

Re: [numba] Problem with Numpy ctypeslib?

From:
Travis Oliphant
Date:
2012-06-08 @ 19:25
On Jun 8, 2012, at 1:46 PM, Jon Riehl wrote:

> Hi Travis,
> 
> On Thu, Jun 7, 2012 at 7:26 PM, Travis Oliphant <travis@continuum.io> wrote:
>> 3) You should not use ndpointer as the argtypes (that is for an 
argument that is expecting the void * data-buffer *not* the array object):
numpy.ndarray should do just fine
> 
> I was using the website documentation which doesn't give an example
> showing what was being called into.  The doc string for the module
> makes this point clear.  So, this misunderstanding was clearly where I
> went wrong.
> 
> Unless there is something better that I'm not aware of, I've got a
> proof of concept working using ctypes.py_object for the argument type.
> At some point, I'd like to do some wrapper level type checks, but now
> is not that time.  Per my erroneous code, we'd at least want to ensure
> that the argument is actually a PyArrayObject, and its data is "C
> contiguous" since the code generator will assume this for now.

That should be a fine place to start, but really discontiguous arrays are 
not that difficult to handle as long as you use the strides array of the 
ArrayObject:

The byte-offset of the element at [i,j,k] is  i*strides[0] + j*strides[1] 
+ k*strides[2]

And to increment one element in along axis 'n'  you add strides[n] to the 
pointer. 

You can and should, though assume aligned and correctly ordered arrays.  
Look at np.require for a function to do that...

-Travis