tdcmeehan opened a new pull request, #59138:
URL: https://github.com/apache/spark/pull/59138
### What changes were proposed in this pull request?
This is a follow-up to #58704. With
`spark.sql.json.enableStreamingTopLevelArray` or the
`enableStreamingTopLevelArray` option enabled, `JacksonParser.parseIterator`
resumes after a malformed array element only where the parser cannot have moved
past it. For an object element it took a `PartialResultException` as proof,
since `convertObject` normally consumes through the element's `END_OBJECT`
before raising one.
A syntax error inside a nested object breaks that premise. The nested
`convertObject` fails while reading its next token, the element's
`convertObject` catches the error, and its loop then stops at the nested
object's `END_OBJECT`, still inside the element. The iterator resumed there,
read one of the element's fields as the next element, and returned that field's
object value as a row.
`resumableAfter` now also requires the parser to be back in the top-level
array. Any other position ends the document, which is what the configuration
doc already says for an element whose failure leaves the parser at an unknown
position.
### Why are the changes needed?
Reading `[{"a":1},{"s":{"x":1 2},"t":{"a":9}},{"a":2}]` (a missing comma
inside `s`) with schema `a int, s struct<x: int>, _corrupt_record string` and
streaming enabled returned rows that are not in the data. Writing `D` for the
whole document:
| mode | before | after |
| --- | --- | --- |
| PERMISSIVE | `(1, null, null)`, `(null, null, D)`, `(9, null, null)`,
`(null, null, D)`, `(2, null, null)` | `(1, null, null)`, `(null, null, D)` |
| DROPMALFORMED | `(1, null, null)`, `(9, null, null)`, `(2, null, null)` |
`(1, null, null)` |
`(9, null, null)` is `t`'s value rather than an array element, and it
carries no corrupt record. The failure needs
`spark.sql.json.enablePartialResults`, which defaults to true. With it
disabled, the element's error propagates and already ended the document.
### Does this PR introduce _any_ user-facing change?
No, compared to released versions: SPARK-59417 is only in master. Within
master, and only with streaming enabled (it defaults to false), a malformed
element that leaves the parser inside itself now ends the document instead of
producing rows from its fields.
### How was this patch tested?
Added `multiline top level JSON array ends the document at a syntax error in
a nested object` over PERMISSIVE and DROPMALFORMED. It fails without the fix.
Ran:
- `catalyst/scalastyle`
- `sql/Test/scalastyle`
- `catalyst/testOnly *JacksonParserSuite`
- `sql/testOnly *JsonV1Suite *JsonV2Suite *JSONTarArchiveReadSuite
*JSONZipArchiveReadSuite *JSONSevenZArchiveReadSuite` (582 tests)
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Isaac (Claude Opus 5.5)
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
