This is an automated email from the ASF dual-hosted git repository.
pitrou pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow.git
The following commit(s) were added to refs/heads/main by this push:
new 331b0f57748 GH-51471: [C++] Disable simdjson threading in chunker
(#51473)
331b0f57748 is described below
commit 331b0f57748ed2d6864e355a4e1cb7460d04da9a
Author: Alexander Taepper <[email protected]>
AuthorDate: Tue Sep 29 10:14:36 2026 +0200
GH-51471: [C++] Disable simdjson threading in chunker (#51473)
### Rationale for this change
Resolves #51471:
When invoking `simdjson`'s `parse_many`, it uses threading to perform its
stage 1 parsing on the next batch when while parsing the current batch:
https://github.com/simdjson/simdjson/blob/master/doc/parse_many.md?plain=1#L96-L104
In most cases, we do not need to parse any batches after the first one to
find delimiters (in fact, we currently would even error, when the first batch
does not contain the whole first document). By disabling the parser's threading
we can expect performance improvements
### What changes are included in this PR?
This disables threading in the parser by setting `parser_.threaded = false;`
### Are these changes tested?
Yes
before (`origin/main` `bb83012743`):
```
ChunkJSONPrettyPrintedMultipleBlocks 187462 ns 158760 ns
4415 block_size=27.344k
bytes_per_second=1.28328Gi/s json_size=218.757k
```
after (`taepper:GH-51471` `fb56ccb2a1`):
```
ChunkJSONPrettyPrintedMultipleBlocks 98726 ns 98721 ns
7144 block_size=27.344k
bytes_per_second=2.06374Gi/s json_size=218.757k
```
### Are there any user-facing changes?
No
### Was AI used for this PR?
**PR code and description written by:**
- [x] Human
- [ ] AI
**Reviewed before submission by:**
- [x] Human
- [ ] AI
- [ ] Not reviewed
* GitHub Issue: #51471
Authored-by: Alexander Taepper <[email protected]>
Signed-off-by: Antoine Pitrou <[email protected]>
---
cpp/src/arrow/json/chunker.cc | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/cpp/src/arrow/json/chunker.cc b/cpp/src/arrow/json/chunker.cc
index 4bff06e2ca5..c51c9213f93 100644
--- a/cpp/src/arrow/json/chunker.cc
+++ b/cpp/src/arrow/json/chunker.cc
@@ -46,7 +46,12 @@ int64_t ConsumeWhitespace(std::string_view view) {
// and uses the structural indexes computed by simdjson to delimit them.
class ParsingBoundaryFinder : public BoundaryFinder {
public:
- explicit ParsingBoundaryFinder(MemoryPool* pool) : pool_(pool) {}
+ explicit ParsingBoundaryFinder(MemoryPool* pool) : pool_(pool) {
+ // A simdjson document stream may start a thread to index the next batch
in the
+ // background. We do not want to do this eagerly as we might not need to
parse
+ // the next block
+ parser_.threaded = false;
+ }
Status FindFirst(std::string_view partial, std::string_view block,
int64_t* out_pos) override {
