On 04/11/2012 11:00 PM, Travis Oliphant wrote:
>
>> On 04/10/2012 02:11 AM, Travis Oliphant wrote:
>>> Hi all,
>>>
>>> Some of you are aware of Numba. Numba allows you to create the equivalent
>>> of C-function's dynamically from Python. One purpose of this system is to
>>> allow NumPy to take these functions and use them in operations like ufuncs,
>>> generalized ufuncs, file-reading, fancy-indexing, and so forth. There are
>>> actually many use-cases that one can imagine for such things.
>>>
>>> One question is how do you pass this function pointer to the C-side. On
>>> the Python side, Numba allows you to get the raw integer address of the
>>> equivalent C-function pointer that it just created out of the Python code.
>>> One can think of this as a 32- or 64-bit integer that you can cast to a
>>> C-function pointer.
>>>
>>> Now, how should this C-function pointer be passed from Python to NumPy?
>>> One approach is just to pass it as an integer --- in other words have an
>>> API in C that accepts an integer as the first argument that the internal
>>> function interprets as a C-function pointer.
>>>
>>> This is essentially what ctypes does when creating a ctypes function
>>> pointer out of:
>>>
>>> func = ctypes.CFUNCTYPE(restype, *argtypes)(integer)
>>>
>>> Of course the problem with this is that you can easily hand it integers
>>> which don't make sense and which will cause a segfault when control is
>>> passed to this "function"
>>>
>>> We could also piggy-back on-top of Ctypes and assume that a ctypes
>>> function-pointer object is passed in. This allows some error-checking at
>>> least and also has the benefit that one could use ctypes to access a
>>> c-function library where these functions were defined. I'm leaning towards
>>> this approach.
>>>
>>> Now, the issue is how to get the C-function pointer (that npy_intp integer)
>>> back and hand it off internally. Unfortunately, ctypes does not make it
>>> very easy to get this address (that I can see). There is no ctypes
>>> C-API, for example. There are two potential options:
>>>
>>> 1) Create an API for such Ctypes function pointers in NumPy and use the
>>> ctypes object structure. If ctypes were to ever change it's object
>>> structure we would have to adapt this API.
>>>
>>> Something like this is what is envisioned here:
>>>
>>> typedef struct {
>>> PyObject_HEAD
>>> char *b_ptr;
>>> } _cfuncptr_object;
>>>
>>> then the function pointer is:
>>>
>>> (*((void **)(((_sp_cfuncptr_object *)(obj))->b_ptr)))
>>>
>>> which could be wrapped-up into a nice little NumPy C-API call like
>>>
>>> void * Npy_ctypes_funcptr(obj)
>>>
>>>
>>> 2) Use the Python API of ctypes to do the same thing. This has the
>>> advantage of not needing to mirror the simple _cfuncptr_object structure in
>>> NumPy but it is *much* slower to get the address. It basically does the
>>> equivalent of
>>>
>>> ctypes.cast(obj, ctypes.c_void_p).value
>>>
>>>
>>> There is working code for this in the ctypes_callback branch of my
>>> scipy fork on github.
>>>
>>>
>>> I would like to propose two things:
>>>
>>> * creating a Npy_ctypes_funcptr(obj) function in the C-API of NumPy and
>>> * implement it with the simple pointer dereference above (option #1)
>>>
>>>
>>> Thoughts?
>>
>> I really hope we can find some project-neutral common ground, so that lots
>> of tools (Cython, f2py, numba, C extensions in NumPy and SciPy) can agree on
>> how to "unbox callables".
>>
>> A new extension type in NumPy would not fit this bill I feel. I've created a
>> specification for this; if a number of projects (the ones mentioned above)
>> agree on this or something similar and implement support, we could propose a
>> PEP and do it properly once it has proven itself.
>>
>> http://wiki.cython.org/enhancements/cep1000
>>
>> In Cython, this may take the form
>>
>> def call_callback(object func):
>> cdef double (*typed_func)(int)
>> typed_func = func
>> return typed_func(4)
>>
>> ...it would be awesome if passing a Numba-compiled function just worked in
>> this example.
>
> Yes, I think we should go the Python PEP route. However, it will take some
> time to see that to completion (especially with ctypes already in existence).
> Dag, this would be a very good thing for you to champion however ;-)
I was NOT proposing a PEP.
The spec is created so that it can be implemented *now*, by the tools
"we" control (and still be very efficient). A "sci-PEP", if you will; a
mutual understanding between Cython, NumPy, numba (and ideally f2py,
which already has something similar, if anyone bothers).
When this is implemented in all the tools we care about, we can propose
something even nicer as a PEP, but that's far down the road; it'll be
another couple of years before I'm on Python 3.
>
> In the mean-time, I think we could do as Robert essentially suggested and
> just use Capsule Objects around an agreed-upon simple C-structure:
>
> int id /* Some number that can be used as a "type-check" */
> void *func;
> char *string;
>
> We can then just create some nice functions to go to and from this form in
> NumPy ctypeslib and then use this while the Python PEP gets written and
> adopted.
What is not clear to me is how one get from the Python callable to the
capsule.
Or do you simply intend to pass a non-callable capsule as an argument in
place of the callback?
Dag
_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
