HyukjinKwon commented on PR #58607:
URL: https://github.com/apache/spark/pull/58607#issuecomment-5609777045
Closing this.
I looked into whether the depth limit is actually reachable for
engine-produced variants, and it isn't:
- Every JSON -> variant path (`parse_json`, `from_json`, ...) goes through
`VariantBuilder`, which uses `VariantUtil.JSON_FACTORY = new JsonFactory()`.
Jackson's default `StreamReadConstraints` caps nesting at 1000, so parsing
rejects anything deeper at parse time:
```
depth 1000: OK
depth 1001: StreamConstraintsException: Document nesting depth (1001)
exceeds
the maximum allowed (1000, from
StreamReadConstraints.getMaxNestingDepth())
```
- `VariantBuilder.buildJson` is itself recursive (per
`START_OBJECT`/`START_ARRAY`), so even without the Jackson cap it would
overflow at build time -- you can't materialize a variant deep enough to later
overflow `toJson`.
- `toJson`'s recursion only overflows in the low-thousands+, i.e. above the
1000 producer cap. So a variant the engine builds can't reach the `toJson`
overflow.
The only way to get a variant deeper than that into `toJson` is a raw
variant binary read from storage that never went through the builder (a corrupt
or hand-crafted blob). That's a narrow malformed-input hardening case, and
`toJson` isn't the only recursive consumer of such bytes (cast, `variant_get`,
shredding recurse too), so guarding `toJson` alone wouldn't be a complete
answer there anyway.
Given that the guard is unreachable on the normal engine paths, the added
config + `toJson` overload + `VariantCastArgs` plumbing isn't justified. If we
later want read-side defense-in-depth against untrusted stored binaries, a
small fixed internal cap mirroring the parser's 1000 would be the way, without
any new API. Closing for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
