[
https://issues.apache.org/jira/browse/SPARK-60846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60846:
-----------------------------------
Labels: pull-request-available (was: )
> Accept Decimal from Spark 4.3.0 clients in Scala UDF payloads
> -------------------------------------------------------------
>
> Key: SPARK-60846
> URL: https://issues.apache.org/jira/browse/SPARK-60846
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 5.0.0
> Reporter: Dustin Smith
> Priority: Major
> Labels: pull-request-available
>
> {{UdfSerialization.auditedTransitions}} lets a Spark Connect server read
> Scala UDF payloads whose {{org.apache.spark.sql.types}} classes have a
> different computed {{serialVersionUID}} than the server's build, for audited
> transitions only. The table was audited against the published
> {{spark-sql-api_2.13}} jars of 4.0.0 to 4.0.4, 4.1.0 to 4.1.3 and 4.2.0.
> Spark 4.3.0 shipped {{org.apache.spark.sql.types.Decimal}} with the computed
> {{serialVersionUID}} {{4401554505066564464}}, which matches neither the
> earlier releases ({{1715621871942419369}}) nor master. Master has no
> transition from it, so a Scala UDF payload that carries a {{Decimal}} (for
> example a UDF closure that captures a {{Decimal}}) from a 4.3.0 client is
> rejected with {{InvalidClassException}}.
> Proposed change: audit the classes in the table against the published 4.0.5,
> 4.1.4, 4.2.1 and 4.3.0 jars and add the missing transitions, backed by a
> recorded 4.3.0 class descriptor in the {{UdfSerializationSuite}} test
> resources. Only {{Decimal}} in 4.3.0 needs a new entry: its fields and
> serialization hooks are identical to the already-audited releases, and every
> other audited class in these releases has a {{serialVersionUID}} that is
> already covered.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
