[
https://issues.apache.org/jira/browse/SPARK-38412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated SPARK-38412:
---------------------------------
Description:
During the work in SPARK-38204, I figured out the parameter is swapped which
led to test failure on new test (I disabled the schema check for now in the PR
of SPARK-38204).
That allows nullable column to be stored into non-nullable column, which should
be prohibited. This is less likely making runtime problem since state schema is
conceptual one and row can be stored even not respecting the state schema.
The worse problem is happening in opposite way, that disallows non-nullable
column to be stored into nullable column, which should be allowed. Spark fails
the query for this case.
We should fix this to allow the case properly.
was:
During the work in SPARK-38024, I figured out the parameter is swapped which
led to test failure on new test (I disabled the schema check for now in the PR
of SPARK-38024).
That allows nullable column to be stored into non-nullable column, which should
be prohibited. This is less likely making runtime problem since state schema is
conceptual one and row can be stored even not respecting the state schema.
The worse problem is happening in opposite way, that disallows non-nullable
column to be stored into nullable column, which should be allowed. Spark fails
the query for this case.
We should fix this to allow the case properly.
> `from` and `to` is swapped in the StateSchemaCompatibilityChecker
> -----------------------------------------------------------------
>
> Key: SPARK-38412
> URL: https://issues.apache.org/jira/browse/SPARK-38412
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 3.1.2, 3.2.1, 3.3.0
> Reporter: Jungtaek Lim
> Priority: Major
>
> During the work in SPARK-38204, I figured out the parameter is swapped which
> led to test failure on new test (I disabled the schema check for now in the
> PR of SPARK-38204).
> That allows nullable column to be stored into non-nullable column, which
> should be prohibited. This is less likely making runtime problem since state
> schema is conceptual one and row can be stored even not respecting the state
> schema.
> The worse problem is happening in opposite way, that disallows non-nullable
> column to be stored into nullable column, which should be allowed. Spark
> fails the query for this case.
> We should fix this to allow the case properly.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
