tdcmeehan opened a new pull request, #58704: URL: https://github.com/apache/spark/pull/58704
### What changes were proposed in this pull request? This PR changes multiline JSON reading so that a top-level array of structs is consumed lazily from a single Jackson parser instead of materializing the entire array before returning rows. It also extends `FailureSafeParser` to handle failures raised while advancing a lazy parser iterator, preserves rows emitted before later structural corruption, and adds benchmark cases for both many small elements and fewer large elements. ### Why are the changes needed? The existing parser materializes every element of a multiline top-level JSON array in memory. Large arrays can therefore require memory proportional to the complete document even though Spark consumes the result as an iterator. Streaming elements bounds parser-side memory while retaining the existing valid-row and parse-mode behavior. ### Does this PR introduce _any_ user-facing change? Yes. Reading a multiline top-level JSON array no longer materializes the complete array before producing rows. If malformed structure is encountered after valid rows have already been emitted, those rows remain available and the malformed document is then handled according to the configured parse mode. ### How was this patch tested? Added regression coverage for lazy top-level array parsing and malformed input after partial output, and extended the existing SPARK-3308 top-level-array coverage. Ran: - `catalyst/compile` - `sql/Test/compile` - `sql/testOnly org.apache.spark.sql.execution.datasources.json.JsonV1Suite` (160 tests) - `sql/testOnly org.apache.spark.sql.execution.datasources.json.JsonV2Suite` (161 tests) - `catalyst/scalastyle` - `sql/scalastyle` - `sql/Test/scalastyle` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
