kszucs opened a new pull request, #51692: URL: https://github.com/apache/arrow/pull/51692
### Rationale for this change #51685 keeps one content-defined chunker per column for the whole file, so the file writer now creates the chunkers together. The chunker's hash loop stores its rolling hash state to memory for every hashed byte, because the values it reads through byte pointers may alias that state. That slows down the chunking, and with the chunkers next to each other, the threads writing neighbouring columns also contend for their cache lines. This PR is stacked on #51685, only the last commit is new. ### What changes are included in this PR? - The rolling hash and its chunking state move into a `GearHash` value. `GetChunks()` rolls a local copy of it and stores it back once, so the compiler keeps the state in registers. - `BM_WriteContentDefinedChunking` benchmarks writing 16 int32 columns of 1M rows with content-defined chunking, with and without threads. ### Are these changes tested? The existing CDC tests cover the chunking, the chunk boundaries don't change. Medians of `BM_WriteContentDefinedChunking` on an Apple M4 Max: | | main | #51685 | this PR | |---|---|---|---| | single-threaded | 49.6 ms | 49.6 ms | 36.9 ms | | multi-threaded | 10.5 ms | 15.4 ms | 9.1 ms | ### Are there any user-facing changes? No, writing with content-defined chunking gets faster. ### Was AI used for this PR? In accordance to the [AI generation guidelines](https://arrow.apache.org/docs/dev/developers/overview.html#ai-generated-code), please disclose below whether and how AI was used in this PR. Claude Code wrote the code, tests and description under human direction. **PR code and description written by:** - [ ] Human - [x] AI **Reviewed before submission by:** - [x] Human - [x] AI - [ ] Not reviewed 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
