Re: [Python/C-Glib] writing IPC file format column-by-column

2020-09-27 Thread Ishan Anand
::RecordBatch without creating intermediate combined chunks. Do C++ developers have any opinion on this? Thanks, -- kou In "[Python/C-Glib] writing IPC file format column-by-column " on Wed, 9 Sep 2020 10:11:54 +, Ishan Anand wrote: > Hi > > I'm looking at usin

Re: [Python/C-Glib] writing IPC file format column-by-column

2020-09-10 Thread Sutou Kouhei
creating intermediate combined chunks. Do C++ developers have any opinion on this? Thanks, -- kou In "[Python/C-Glib] writing IPC file format column-by-column " on Wed, 9 Sep 2020 10:11:54 +, Ishan Anand wrote: > Hi > > I'm looking at using Arrow primarily o

[Python/C-Glib] writing IPC file format column-by-column

2020-09-09 Thread Ishan Anand
Hi I'm looking at using Arrow primarily on low-resource instances with out of memory datasets. This is the workflow I'm trying to implement. * Write record batches in IPC streaming format to a file from a C runtime. * Consume it one row at a time from python/C by loading the file in chu