viirya opened a new pull request, #59847:
URL: https://github.com/apache/spark/pull/59847
### What changes were proposed in this pull request?
Backport of ef6f78bc8930157890a0238ac462ad214ead57d3 ("[SQL][MINOR] Bound
nesting depth when converting protobuf Any to JSON") to branch-4.0. The commit
is a clean cherry-pick.
When `convert.any.fields.to.json` is enabled, `ProtobufDeserializer` now
checks the message under each `google.protobuf.Any` field before calling
`JsonFormat.print`. The check walks the nested messages iteratively under a
single depth budget of 100. A record that goes over the budget fails with
`InvalidProtocolBufferException`, which the existing parse-mode handling treats
as a malformed record.
This change is already in master, branch-4.3, branch-4.2 and branch-4.1, but
not in branch-4.0.
### Why are the changes needed?
`JsonFormat.print` re-parses each nested `Any` with a fresh protobuf
recursion limit while it recurses on the JVM stack. A deeply nested Any-in-Any
record can therefore overflow the executor stack. `StackOverflowError` is
fatal, so it escapes the `NonFatal` handling behind PERMISSIVE/FAILFAST and
fails the task instead of producing a malformed record.
### Does this PR introduce _any_ user-facing change?
Yes. With `convert.any.fields.to.json=true`, a record whose message nesting
under an `Any` field is deeper than 100 is now treated as malformed: null in
PERMISSIVE mode, an error in FAILFAST mode. Before, it could crash the task
with `StackOverflowError`. This is the same behavior that master, 4.3, 4.2 and
4.1 already have.
### How was this patch tested?
Includes the original test from ProtobufFunctionsSuite, "deeply nested
Any-in-Any is rejected as malformed instead of overflowing the stack".
`build/sbt 'protobuf/testOnly *Protobuf*Suite'` on JDK 17: 58/58 passed (3
suites).
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor (original commit). The backport was prepared with
Claude Code.
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
