Abdallah-Afifi commented on issue #10645: URL: https://github.com/apache/arrow-rs/issues/10645#issuecomment-6066040118
Measured it. Setup: `encode/next_page` only, `main` baseline and variant benched back to back in one session, plus a `main` vs `main` control to establish the drift floor (+0.9% to +1.5% with no code change at all). Runtime chunking vs `main`, drift not subtracted: | | change | | --- | --- | | f32 decimal_with_exceptions | +8.83% | | f32 high_precision | +6.40% | | f64 decimal_with_exceptions | +0.74% | | f64 high_precision | −6.63% | So roughly 5–7% on f32 streaming encode once drift is accounted for; f64 flat or better. I also tried keeping a const-generic `as_chunks::<1024>()` fast path for the default size: +9.24% and +7.49% on those same two f32 benches, i.e. no better than plain `chunks_exact`. I dropped it rather than carry the extra branch for nothing. That also means the cost isn't the chunking loop's const-ness, and I don't have a confirmed explanation for it yet — same for the f64 high_precision improvement, which reproduced across four runs but which I can't account for either. Caveats: M5 laptop rather than CI, single run per variant, on top of that ~1.5% drift floor. Is ~5–7% on f32 streaming acceptable here, or would you rather the vector size be a const generic threaded through the encoder? Happy either way — I just don't want to build out the `WriterProperties` plumbing on the wrong foundation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
