Hi Dewey, Thank you for taking the time. My goal is to convert from a variety of big C data structures like this to equivalent Arrow spec/schema. Then, I would like to store them (RecordBatches) to parquet or any other relevant type. The CSV or JSON output from the example C program (smf84fmt) doesn't matter; just wanted to point to the sample data format as in the header file.
I had tried bindgen to create Rust definitions from the header files, but it gets complicated real fast... more than I can comprehend at least. The types get crazier too, with singly linked lists (not there in the linked example, but in other types), etc. Would *really* like to solve this in a systemtic way, without needing to hand code the Arrow schema... Because the C header files are maintained (by a provider), it would work out best if it's possible to create a conversion script, and then use the Arrow schema in Python/Rust/etc. -KB On Wednesday, March 6th, 2024 at 07:59, Dewey Dunnington via user <user@arrow.apache.org> wrote: > Hi KB, > > There might be some other approaches I'm not aware of; however, I had > some fun with Python's cffi package to generate some (untested) > nanoarrow code based on the struct definitions [1]. If all you need > are the types in Python or some other higher-level language (e.g., to > read one of the CSV or JSON files generated by the tool you linked), > you could generate Python code instead. > > I hope that's helpful! > > -dewey > > [1] https://gist.github.com/paleolimbot/e1667a57f837e4db7e973b9677e33ddb > > On Sun, Mar 3, 2024 at 10:08 PM kekronbekron > kekronbek...@protonmail.com wrote: > > > Hello, > > > > Say I have a whole bunch of fully typed (with unions and all) data > > structures like the one here - > > https://github.com/IBM/IBM-Z-zOS/blob/main/SMF-Tools/SMF84Formatter/smf84fmt.h. > > Say I'm parsing bytes with such a header...is it possible to then use > > Arrow's C data interface (or maybe nanoarrow) to painlessly convert such a > > struct to Arrow type(s)? > > > > - KB
