peter-toth commented on code in PR #57357:
URL: https://github.com/apache/spark/pull/57357#discussion_r3633021513
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -7864,6 +7864,20 @@ object SQLConf {
.createWithDefault(false)
}
+ val LEGACY_ALLOW_NON_DETERMINISTIC_V2_FILTER_PUSHDOWN =
Review Comment:
It's never the *connector* that depends on non-det pushdown. A connector
just implements `SupportsPushDownV2Filters` and takes whatever Spark hands it.
Getting `rand() > 0.5` pushed in gains it nothing and isn't sound anyway, so no
connector author would set a `pushDownNondeterministicPredicates()` capability
to true on purpose — it'd be dead API. The thing that can accidentally rely on
the old behavior is a user's *workflow* that happened to get the results it
wanted out of Spark's weird old behavior, and the person who knows about that
workflow isn't the connector author.
So the dependency lives at the query/session level, which is exactly where a
`spark.sql.legacy.*` conf belongs: flip the default to the correct behavior
now, give the few who built on the old one a temporary, deprecatable switch.
And flipping the default can't break a connector — keeping non-det filters
post-scan only ever makes results more correct, never crashes a source, so it's
purely opt-back-in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
