Yicong-Huang opened a new pull request, #58962:
URL: https://github.com/apache/spark/pull/58962
### What changes were proposed in this pull request?
This PR makes Spark Connect Scala UDF payload deserialization tolerant of
`serialVersionUID` drift for `org.apache.spark.sql.types` classes.
- Adds `UdfSerialization`, an `ObjectInputStream`-based reader that, for
classes under `org.apache.spark.sql.types`, rebinds a stream class descriptor
to the local class when their `serialVersionUID`s differ but their **serialized
field layouts are identical** (the complete persistent field set, including
Scala lazy-val `bitmap$*` slots). Any field-layout difference is left untouched
so the standard `serialVersionUID` check still fails fast. Classes outside
`org.apache.spark.sql.types` keep the standard check unchanged.
- Routes `UdfPacket.apply(InputStream)` through it.
### Why are the changes needed?
`org.apache.spark.sql.types` classes (which make up a UDF's input/output
schema) carry no explicit `@SerialVersionUID`, so the JVM auto-computes it from
the whole class shape. That computation folds in compiler-synthesized members
that are irrelevant to serialization -- most notably the `$anonfun$` public
static methods Scala emits for lambdas. A source change that only reshapes a
lambda (for example, rewriting a helper to use `existsRecursively { ... }`)
changes the auto-computed `serialVersionUID` without changing any serialized
field. A plain `ObjectInputStream` then rejects a UDF payload produced by a
different Spark version with `java.io.InvalidClassException: ... local class
incompatible`, even though the payload is field-compatible. This breaks the
ability to run a Scala UDF serialized by one version against another.
### Does this PR introduce _any_ user-facing change?
Yes. A Scala UDF payload whose `org.apache.spark.sql.types` classes differ
from the executing side only in `serialVersionUID` (identical serialized
fields) now deserializes successfully instead of failing with
`InvalidClassException`. Behavior for a genuinely incompatible serialized field
layout is unchanged: it still fails fast.
### How was this patch tested?
New `UdfSerializationSuite`. Using same-length sibling fixture classes plus
an in-place class-name patch of the serialized stream (a deterministic stand-in
for a cross-version producer/consumer pair), it covers:
- SUID-only drift of a `sql.types` class deserializes and recovers the
value, through both the `(bytes, loader)` and `(InputStream)` overloads (and a
plain `ObjectInputStream` is asserted to reject the same stream).
- a genuine serialized field-layout change fails fast with
`InvalidClassException`.
- SUID drift outside `org.apache.spark.sql.types` is not tolerated (fails
fast).
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Isaac (Claude Code)
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
