Reranko05 commented on PR #51038: URL: https://github.com/apache/arrow/pull/51038#issuecomment-5867437215
@pitrou I investigated this further. When the `ConsumeJsonWhitespace()` guard is removed, `BlockParserWithSchema.Empty` passes when run alone but fails in the full suite. In both cases, `iterate_many()` returns `SUCCESS`. This means SIMDJSON was able to create the document stream successfully and did not report an error while creating it. The stream then contains 0 documents because the input is empty. The difference comes afterward: `truncated_bytes()` returns `0` when the test runs alone, but `4294967289` in the full suite, causing it to report empty input as truncated. I also tried a fresh `simdjson::ondemand::parser` and disabled SIMDJSON threading, but the same discrepancy remained. The `ConsumeJsonWhitespace()` guard avoids sending empty input to `iterate_many()`, and the full suite passes. What do you think is it fine to keep the check? Also regarding the `ConsumeJsonWhitespace()` comment, I think this would be more accurate now: ```cpp // XXX We could try to SIMD-accelerate this routine, but it is called only // a few times per input and is expected to examine a small amount of data. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
