JonSnow1807 opened a new issue, #12194: URL: https://github.com/apache/arrow-rs/issues/12194
The Parquet writer encodes definition and repetition levels one value at a time. `RleEncoder` already has a group-of-8 path (`put_batch`) that dictionary indices use. Routing levels through it keeps the encoded bytes identical (88/88 written files hash the same) and, on the `arrow_writer` benches (i7-14700, one pinned core, 5 interleaved rounds, about ±1% noise), takes less time for: - nested columns: `list_struct_with_list` -29.6%, `list_nested` -27.4%, `list_primitive` -23.7%, `map` -15.1% - flat nullable columns: `primitive` -25.6%, `bool` -38% - non-null and all-null columns: -2.6% to +1.1% This is most of apache/datafusion-comet#6918. A few dictionary benches moved 3-11% in the default build, which looks like code placement: the same or fewer instructions, and within ±2% with one codegen unit. I’d split it into a PR adding level-encoding and nested-writer benches, then the change. @HippoBaro does this fit with #11273? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
