[
https://issues.apache.org/jira/browse/SPARK-48073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17843583#comment-17843583
]
L. C. Hsieh commented on SPARK-48073:
-------------------------------------
The breaking change was introduced by https://github.com/apache/spark/pull/39615
> StateStore schema incompatibility between 3.2 and 3.4
> -----------------------------------------------------
>
> Key: SPARK-48073
> URL: https://issues.apache.org/jira/browse/SPARK-48073
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 3.2.4
> Reporter: L. C. Hsieh
> Priority: Major
>
> One our customer encountered some schema incompatibility problems when
> upgrading from Spark 3.2 to 3.4 with structured streaming application.
> It seems in 3.4 `Encoders.bean()` includes properties with only getter with
> or without setter, whereas in 3.2, only properties with both getter and
> setter are included.
> For example, here are schemas for an AtomicLong property/field generated by
> each version:
> 3.2:
> StructType(StructField(opaque,LongType,true),StructField(plain,LongType,true))
> 3.4:
> StructType(StructField(acquire,LongType,false),StructField(andDecrement,LongType,false),StructField(andIncrement,LongType,false),StructField(opaque,LongType,false),StructField(plain,LongType,false))
> Note that the null ability flag also changes.
> Primitive long schema has nullable=true in 3.2, but false in 3.4.
> I am not sure if the issue is aware by the community before, and if there is
> workaround for that?
> Thanks.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
