Abdallah-Afifi commented on issue #10644: URL: https://github.com/apache/arrow-rs/issues/10644#issuecomment-5556130122
Opened #11004 for this. While writing the regression test it turned out the empty first page isn't only untested, it's actively harmful. `flush_buffer` caches the first page's preset for the whole column chunk, so the exponent 0 / factor 0 fallback makes every fractional value in every later page an exception. Encoding 3000 values of `i * 0.01`: 3,808 bytes when they are the first page, 31,258 bytes when preceded by an empty one, against 24,000 unencoded. It's also reachable from the normal write path — the column writer counts levels rather than values, so an all-null first page flushes an empty buffer. So the PR carries the fix @alamb suggested as a follow-up along with the test. Happy to split them if you'd rather review separately. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
