Re: [Python][C++] Chunked Storage of N-dim arrays

2024-04-23 Thread Weston Pace
I've worked quite a bit with tensor data recently and `arrow.Tensor` (or just the underlying FixedSizeList) has worked well for me for an in-memory representation. > If you compress it, you have no means to decompress individual chunks, from what I can tell from prototyping within Python.

[Python][C++] Chunked Storage of N-dim arrays

2024-04-23 Thread Robert McLeod
Hi everyone, For a project I'm working on I've picked Arrow as the library and either Feather or Parquet as our storage format for our tabular data. However, I also have some hyperspectral data to serialize and I'd prefer not to add another big dependency if I can avoid it so I've been trying to