HeartSaVioR commented on code in PR #56061:
URL: https://github.com/apache/spark/pull/56061#discussion_r3300563998
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -3403,6 +3403,22 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val STATEFUL_OPERATOR_ALWAYS_NULLABLE_OUTPUT =
+
buildConf("spark.sql.streaming.statefulOperator.alwaysNullableOutput.enabled")
+ .internal()
+ .doc("When true, every streaming stateful operator reports its output
schema with " +
+ "nullable=true on all columns (including nested struct fields, array
elements, and " +
+ "map values), the state schema is widened at every construction site,
and the state " +
+ "schema compatibility checker ignores nullability for stateful
operator schemas. " +
Review Comment:
Makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
