HeartSaVioR commented on code in PR #42940:
URL: https://github.com/apache/spark/pull/42940#discussion_r1329416576
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -2180,6 +2180,17 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val STREAMING_TRIGGER_AVAILABLE_NOW_WRAPPER_ENABLED =
+ buildConf("spark.sql.streaming.triggerAvailableNowWrapper.enabled")
+ .internal()
+ .doc("Whether to use the wrapper implementation of Trigger.AvailableNow
if the source " +
+ "does not support Trigger.AvailableNow. Enabling this allows the
benefits of " +
+ "Trigger.AvailableNow with sources which don't support it, but some
sources " +
+ "may show unexpected behavior including duplication, data loss, etc.
So use with " +
Review Comment:
This new config is marked as "internal", which is not going to be a part of
public documentation. We expect very advanced users and/or operators only know
about the existence of the config and use it as their own risk. We don't even
expose this config in the warning message - that's the way I avoid users from
shooting themselves on the foot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
