Re: Help understanding IPC Message/Buffer structure

2018-07-12 Thread Randy Zwitch
Thanks Paul, all your detail is super-helpful. I became paralyzed by looking at too many codebases, but now I think I get it. Wes, you’re absolutely right that this is complicated by the lack of support. I’m working this example because it’s a good use of work time, and then hopefully success with

Re: Help understanding IPC Message/Buffer structure

2018-07-12 Thread Wes McKinney
hi Randy, In Julia I think this is complicated by the lack of a Flatbuffers compiler for the language. In the case of Feather files, in Feather.jl they have implemented the Flatbuffers schema in Julia code: https://github.com/JuliaData/Feather.jl/blob/master/src/metadata.jl#L3 So you need to do

Re: Help understanding IPC Message/Buffer structure

2018-07-12 Thread Paul Taylor
Hi Randy, The first four bytes are the int32 length of the flatbuffers Message metadata plus 4 bytes of padding between the length and the Message metadata itself. The Message metadata start

Help understanding IPC Message/Buffer structure

2018-07-12 Thread Randy Zwitch
I’m trying to understand how to parse a Buffer into a Schema, but using using pdb with Python and reading the TS/Python/C++ Arrow source hasn’t really cleared much up for me. Nor has studying https://arrow.apache.org/docs/ipc.html Here’s are the steps of what I’ve tried (the code is Julia, but on