Hi, I'm running into an error while attempting to load an arrow file. `ls`
lists the size of the file as 6347031002 bytes. Most examples I see online
use `fs.readFileSync` and pass the result to `Table.from`. Since you can't
`readFileSync` (or `readFile` for that matter) a file larger than 2GB, I'm
doing something like this instead:
const stream = fs.createReadStream('data.arrow')
const table = Table.from(stream)
That seems to work for smaller files, but with the 6GB file, I hit the
following error:
(node:17465) UnhandledPromiseRejectionWarning: RangeError: Invalid typed
array length: 6347031002
at new Uint8Array (<anonymous>)
at joinUint8Arrays
(..../node_modules/@apache-arrow/es5-esm/util/buffer.js:79:36)
at byteRange
(..../node_modules/@apache-arrow/es5-esm/io/adapters.js:437:85)
at ..../node_modules/@apache-arrow/es5-esm/io/adapters.js:537:73
at step (..../node_modules/tslib/tslib.js:141:27)
at Object.next (..../node_modules/tslib/tslib.js:122:57)
at resume (..../node_modules/tslib/tslib.js:208:48)
at fulfill (..../node_modules/tslib/tslib.js:210:35)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (internal/process/task_queues.js:93:5)
Does that imply that the largest arrow file I can load is limited by the
largest Uint8Array that can be made (2^32, per
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Errors/Invalid_array_length)?
Or is there a better way that I should be loading this file?
Thanks,
Patrick
