longvu-db opened a new pull request, #59236: URL: https://github.com/apache/spark/pull/59236
### What changes were proposed in this pull request? Preserve scalar boolean literals in `V2ExpressionBuilder` behind the internal session configuration `spark.sql.v2ExpressionBuilder.preserveBooleanLiterals.enabled`. Pass `isPredicate` into `translateLiteral`. When the flag is enabled and the literal is a scalar operand, return `LiteralValue`. Retain the existing `AlwaysTrue`, `AlwaysFalse`, and other literal branches for the disabled path and predicate contexts. The flag defaults to false in production and true in tests, using Spark's existing `Utils.isTesting` convention. Add `DataSourceV2BooleanReplaceWhereSuite` with seven predicates, three write paths, and both flag values, for 42 cases. A test-only in-memory V1 adapter supports SQL `INSERT ... REPLACE WHERE`, DataFrameWriter's raw `replaceWhere` option, and DataFrameWriterV2 overwrite. The raw writer-option route uses the existing in-memory provider's `save()` entry point. ### Why are the changes needed? Scalar boolean operands currently become always-true/false predicates rather than literals. Converting the resulting V2 predicates to V1 filters either rejects valid overwrites or silently drops a branch of an `OR` condition. With an initial table containing `(1, true), (2, false), (3, NULL)`: - `b <=> true`, `true <=> b`, `b <=> false`, `NOT (b <=> true)`, and `b IN (true, false)` fail through the V2-to-V1 overwrite path when the flag is disabled. - `b <=> true OR id = 4` drops the boolean branch when disabled, incorrectly retaining `(1, true)`. - Enabling the flag preserves literal operands and overwrites the intended rows. - Constant `true` remains a predicate, and the raw writer-option route is unchanged by the flag. ### Does this PR introduce _any_ user-facing change? No change with the production default. Enabling the internal flag fixes boolean-literal conversion for data sources that consume V2 scalar expressions, including overwrite predicates converted to V1 filters. ### How was this patch tested? - Compiled the changed `SQLConf`, `V2ExpressionBuilder`, and regression suite with the cached Spark dependency classpath. - Verified production default false and test default true in separate JVMs using the changed configuration source. - All 42 regression cases passed in an isolated validation run against the existing local Spark build. Its cached SQL implementation expects an older configuration API, so the validation run used that build's configuration sources with the identical new configuration entry, alongside the actual changed `V2ExpressionBuilder` and test suite. - Verified all seven regression bodies and assertions, including unchanged tables after errors, retained rows on the disabled `OR` path, successful replacements, and null handling. - `git diff --check` passed. - Scalastyle passed on all three changed files with no errors or warnings. - The standard `build/sbt 'sql/testOnly org.apache.spark.sql.connector.DataSourceV2BooleanReplaceWhereSuite'` command could not start because Maven host resolution prevented downloading the required sbt launcher. A full clean-master run remains to be done. This PR is a draft pending an Apache Spark JIRA issue and standard-build validation. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
