alamb commented on issue #6946: URL: https://github.com/apache/arrow-rs/issues/6946#issuecomment-5063359901
> I started looking into a design that has a stream per column. This results in more object store GETs but allows each column to stream at whatever pace it needs to to feed decoding and minimizes how much data has to be buffered in memory to keep the CPU fully fed. I initially got mixed results, but I'm posting here to bump the issue and share the pain / solution ideas. FWIW I think turning on predicate pushdown also (effectively) increases the GET requests but reduces the buffering (depending on the selectivyt of the filters) and that has seen to be a very mixed bag. > Because our columns are very wide / have large values (e.g. 1MB LLM prompts) buffering at the row group level causes high memory usage. Even if we could set some row limit that's still not optimal since depending on what columns the query reads, the shape of the data, etc. the optimal number varies widely. @adriangb I wonder if you could be slightly more clever and compute the amount of data that would be buffered for some slices of rows, using the OffsetIndex -- you might be able to use that to determine how many rows and bytes there are for each page https://github.com/apache/parquet-format/blob/21ecc2b0f5786070561ae0d34152230853c83e2b/src/main/thrift/parquet.thrift#L1239-L1251 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
