Hi Weston, I’m willing but hardly capable. I’m just a copy cat with C/C++. I think that the main thing is that is It possible use RecordBatchReader in Javascript without IPC stream. I haven’t find a way but that doesn’t tell much. Bun is capable to read directly from buffer if the buffer is null terminated or the size is know. I tried to use TotalBufferSize with RecordBatch from ImportRecordBatch but It didn’t work.
K. -- Kimmo Linna Nihtisalontie 3 as 1 02630 ESPOO [email protected] +358 40 590 1074 > On 2. Mar 2023, at 21.05, Weston Pace <[email protected]> wrote: > > I believe you would need a Javascript version of the C data interface. This > should be doable with bun:ffi but I'm not aware of anyone that has done this > before. I also wonder if there is a way to create a C data interface based > on TypedArray that would be usable in both bun and node. I'm also not really > up to speed on what arrow-js has in terms of capabilities so it is possible > it exists and I just didn't know. Is it something you are interested in > contributing? > > On Wed, Mar 1, 2023 at 10:41 PM Kimmo Linna <[email protected] > <mailto:[email protected]>> wrote: >> Hi, >> >> I will get ArrowSchema and ArrowArray directly from DuckDB. I want to >> transfer the RecordBatch to Bun with bun::ffi. At the moment my procedure is >> the following: >> auto schema = arrow::ImportSchema(arrow_schema); >> auto batch = arrow::ImportRecordBatch(arrow_array, *schema); >> auto output_stream = arrow::io::BufferOutputStream::Create(); >> auto batch_writer = arrow::ipc::MakeStreamWriter(*output_stream, *schema); >> auto status = (*batch_writer)->WriteRecordBatch(**batch); >> auto buffer = (*output_stream)->Finish(); >> (*out).address = (void *)(*buffer)->address(); >> (*out).size = (*buffer)->size(); >> >> And then I will read the buffer in Bun with toArrayBuffer and >> RecordBatchReader like this: >> return RecordBatchReader.from( >> toArrayBuffer( >> dab.dab_ipc_address(ipc), 0, Number(dab.dab_ipc_size(ipc)) >> )).readAll()[0]; >> >> I just wonder Is there a way to read RecordBatch directly from RecordBatch >> which is done by ImportRecordBatch or can I do this without OutputStream at >> all? >> >> Best regards, >> >> Kimmo >> >> -- >> Kimmo Linna >> Nihtisalontie 3 as 1 >> 02630 ESPOO >> [email protected] <mailto:[email protected]> >> +358 40 590 1074 >> >> >>
