Re: [Numpy-discussion] Using np.frombuffer and cffi.buffer on array of C structs (problem with struct member padding)

2018-01-31 Thread Allan Haldane 

On 01/31/2018 02:06 AM, Joe wrote:

Does someone know of a function or a convenient way to automatically
derive a dtype object from a C typedef struct string or a cffi.typeof()?


I remember when I wrote those docs over a year ago, I searched for an 
established way to do this but didn't find one. If you find/come up with 
one, let us know and I might add a pointer or description of it in the docs.


Searching just now, I came across this recent gist:

https://gist.github.com/inactivist/4ef7058c2132fa16759d

So it looks like we can do something like:

# typemap from C type to numpy type may need some work...
typemap = {'char': 'byte', 'short': 'short', 'int': 'intc',
   'long long': 'longlong', 'size_t': 'intp',
   'float': 'float', 'double': 'double'}

def extract_field_info(tp):
fields = []
for field, fieldtype in tp.fields:
if fieldtype.type.kind == 'primitive':
format = typemap[fieldtype.type.cname]
else:
format = extract_field_info(fieldtype.type)
fields.append((field, format, fieldtype.offset))
return fields

 from cffi import FFI
 ffi = FFI()
 ffi.cdef("struct foo { int a; char b; double d;};")
 tp = ffi.typeof('struct foo')

 names, formats, offsets = zip(*extract_field_info(tp))
 np.dtype({'names': names, 'formats': formats, 'offsets': offsets})
 # compare to: np.dtype('i4,i1,f8', align=True)

I just made that up now and it may be buggy and have missing corner 
cases. But if you agree it works, maybe we can mention it in the 
structured array docs instead of the vague sentence there now, that 
"some work may be needed to obtain correspondence with the C struct".


Allan


Am 27.01.2018 10:30 schrieb Joe:

Thanks for your help on this! This solved my issue.


Am 25.01.2018 um 19:01 schrieb Allan Haldane:

There is a new section discussing alignment in the numpy 1.14 structured
array docs, which has some hints about interfacing with C structs.

These new 1.14 docs are not online yet on scipy.org, but in the meantime
  you can view them here:
https://ahaldane.github.io/user/basics.rec.html#automatic-byte-offsets-and-alignment 



(That links specifically to the discussion of alignments and padding).

Allan

On 01/25/2018 11:33 AM, Chris Barker - NOAA Federal wrote:




The numpy dtype constructor takes an “align” keyword that will pad it
for you.


https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.dtype.html 



-CHB



___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion



___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion


___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion


___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Using np.frombuffer and cffi.buffer on array of C structs (problem with struct member padding)

2018-01-30 Thread Joe 

Does someone know of a function or a convenient way to automatically
derive a dtype object from a C typedef struct string or a cffi.typeof()?

Am 27.01.2018 10:30 schrieb Joe:

Thanks for your help on this! This solved my issue.


Am 25.01.2018 um 19:01 schrieb Allan Haldane:
There is a new section discussing alignment in the numpy 1.14 
structured

array docs, which has some hints about interfacing with C structs.

These new 1.14 docs are not online yet on scipy.org, but in the 
meantime

  you can view them here:
https://ahaldane.github.io/user/basics.rec.html#automatic-byte-offsets-and-alignment

(That links specifically to the discussion of alignments and padding).

Allan

On 01/25/2018 11:33 AM, Chris Barker - NOAA Federal wrote:




The numpy dtype constructor takes an “align” keyword that will pad 
it

for you.


https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.dtype.html

-CHB



___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion



___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion


___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Using np.frombuffer and cffi.buffer on array of C structs (problem with struct member padding)

2018-01-27 Thread Joe 

Thanks for your help on this! This solved my issue.


Am 25.01.2018 um 19:01 schrieb Allan Haldane:

There is a new section discussing alignment in the numpy 1.14 structured
array docs, which has some hints about interfacing with C structs.

These new 1.14 docs are not online yet on scipy.org, but in the meantime
  you can view them here:
https://ahaldane.github.io/user/basics.rec.html#automatic-byte-offsets-and-alignment

(That links specifically to the discussion of alignments and padding).

Allan

On 01/25/2018 11:33 AM, Chris Barker - NOAA Federal wrote:




The numpy dtype constructor takes an “align” keyword that will pad it
for you.


https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.dtype.html

-CHB



___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion



___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion


___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Using np.frombuffer and cffi.buffer on array of C structs (problem with struct member padding)

2018-01-25 Thread Allan Haldane 
There is a new section discussing alignment in the numpy 1.14 structured
array docs, which has some hints about interfacing with C structs.

These new 1.14 docs are not online yet on scipy.org, but in the meantime
 you can view them here:
https://ahaldane.github.io/user/basics.rec.html#automatic-byte-offsets-and-alignment

(That links specifically to the discussion of alignments and padding).

Allan

On 01/25/2018 11:33 AM, Chris Barker - NOAA Federal wrote:
> 
>>
>> The numpy dtype constructor takes an “align” keyword that will pad it
>> for you.
> 
> https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.dtype.html
> 
> -CHB
> 
> 
> 
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
> 

___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Using np.frombuffer and cffi.buffer on array of C structs (problem with struct member padding)

2018-01-25 Thread Chris Barker - NOAA Federal 
The numpy dtype constructor takes an “align” keyword that will pad it for
you.


https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.dtype.html

-CHB
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Using np.frombuffer and cffi.buffer on array of C structs (problem with struct member padding)

2018-01-25 Thread Chris Barker - NOAA Federal 
Sent from my iPhone

> On Jan 25, 2018, at 6:51 AM, Joe  wrote:
>
> Hello,
>
> how I could dynamically handle the dtype of a structured array when reading
> an array of C structs with np.frombuffer (regarding the member padding in the 
> struct).
>
> So far I manually adjusted the dtype of the structured array and added a 
> field for the padding,
> this works on a small scale.
> The structs are not defined within my code, but within a third party and
> basically I am looking for no-worry hassle free way to handle this, because 
> there are a lot of structs
>
> Is there some smart way to do this in Numpy?

The numpy dtype constructor takes an “align” keyword that will pad it for you.

However, if these strict are coming from a lib compiled by a third
party, I’m not sure you can count on the alignment rules being the
same.

So maybe you will need to use the cffi functions :-(

-CHB


>
> So far the best approach seems to parse the struct with the cffi functions
> ffi.sizeof(), ffi.offsetof() and maybe ffi.alignof() to find out where the 
> padding
> happens and add it dynamically to the dtype. But maybe someone has a
> smarter idea how to solve this.
>
> You can find a more detailed description and a working example here:
>
> https://stackoverflow.com/questions/48423725/how-to-handle-member-padding-in-struct-when-reading-cffi-buffer-with-numpy-fromb
>
> Kind regards,
> Joe
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion