pitrou commented on code in PR #50945:
URL: https://github.com/apache/arrow/pull/50945#discussion_r3980984499
##########
cpp/src/arrow/json/chunker.cc:
##########
@@ -165,6 +91,78 @@ class ParsingBoundaryFinder : public BoundaryFinder {
int64_t* out_pos, int64_t* num_found) override {
return Status::NotImplemented("ParsingBoundaryFinder::FindNth");
}
+
+ private:
+ simdjson::ondemand::parser parser_;
+ // A persistent buffer to keep padded contents for simdjson.
+ // This should be more efficient than allocating a new padded_string
everytime.
+ std::string buffer_;
+
+ simdjson::padded_string_view GetPaddedStringView(std::string_view partial,
+ std::string_view block =
{}) {
+ // Adjust buffer size without copying old contents.
+ buffer_.clear();
+ buffer_.reserve(partial.size() + block.size() +
simdjson::SIMDJSON_PADDING);
+ buffer_.append(partial);
+ buffer_.append(block);
Review Comment:
`partial` being empty doesn't change the fact that we need to copy `block`
to ensure there's enough padding.
We can rework this later by having the `BoundaryFinder` API take `Buffer`
arguments instead of `std::string_view`, because that would let us inspect
their capacity and avoid copying if the `Buffer` has enough padding already.
That said, chunking is already faster with this PR than it used to be with
RapidJSON:
* before:
```
ChunkJSONPrettyPrinted 755476 ns 755351 ns 932
bytes_per_second=276.193Mi/s json_size=218.757k
ChunkJSONLineDelimited 94.6 ns 94.6 ns 7391992
bytes_per_second=0/s json_size=193.757k
```
* after:
```
ChunkJSONPrettyPrinted 634472 ns 634465 ns 1110
bytes_per_second=388.942Mi/s json_size=258.757k
ChunkJSONLineDelimited 94.5 ns 94.5 ns 7418649
bytes_per_second=0/s json_size=193.757k
```
(on Ubuntu 24.04 with a AMD Zen 2 CPU supporting AVX2)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
