sunchao opened a new issue, #6006: URL: https://github.com/apache/datafusion-comet/issues/6006
## Problem and scope Track Spark-compatible evaluation of expressions that can raise a data-dependent error when an enclosing operator or expression skips some input rows. The expression can return the correct result or error when called in isolation and still change a query's outcome if a batch evaluator calls it on rows Spark never evaluates. [#5532](https://github.com/apache/datafusion-comet/issues/5532) demonstrates this for `unbase64` under limits and semi/anti joins. [#5533](https://github.com/apache/datafusion-comet/pull/5533) adds an explicit serde policy, `RequiresSparkEvaluationMask`, for selected operator-level protections. Only `UnBase64` opts in. The policy is deliberately not a complete classification of throwing expressions, and its presence does not prove every enclosing evaluation context is protected. The [review discussion](https://github.com/apache/datafusion-comet/pull/5533#discussion_r3884708892) identified the wider family: ANSI arithmetic overflow, invalid ANSI casts, decimal divide-by-zero, and `element_at`, plus per-row `AND` and `CASE` evaluation. This issue tracks validating and resolving that family; it does not claim that every expression/context combination below is currently broken. In particular, existing native conditional evaluation and targeted serde guards may already preserve a given mask. ## Work to track - Establish which expression instances can raise, accounting for eval mode, input types, nullable children, and Spark version. Start with ANSI arithmetic, ANSI casts, decimal division, array `element_at`, and `unbase64`. Determinism alone does not establish safety. Do not classify all casts or all `element_at` forms identically. - Audit operator-level early termination: limits, already-ordered top-K/window limits, semi/anti join residuals, and empty-input join behavior. Check the actual consumption of each physical strategy. An intervening blocking sort, exchange, or aggregate can change which rows Spark evaluates. - Audit per-row masks independently of operator masks. Representative shapes include `WHERE k = 1 AND unbase64(bad) = X'616263'` and `CASE WHEN k = 1 THEN unbase64(bad) END`, with malformed input only in excluded rows. Verify native execution, predicate placement/reordering, conditional branch selection, nullable operands, and whole-tree JVM dispatch before classifying a case as a mismatch. - Audit aggregate `FILTER` separately. For example, determine whether `max(unbase64(bad)) FILTER (WHERE k = 1)` evaluates the argument only for qualifying rows on each execution path. Aggregate input evaluation is distinct from final-result evaluation beneath an outer limit; #5533 does not fix the aggregate `FILTER` case. - Choose the compatibility/performance approach deliberately: conditional evaluation in native kernels, scoped fallback, or an explicitly documented divergence where a fix is deferred. Expanding a fallback policy to common ANSI arithmetic/casts can affect many valid-data queries, so measure the fallback footprint and end-to-end cost before broader enrollment. ## Validation and completion criteria - For each confirmed mismatch, record Spark/Comet versions and commits, configuration, executed physical plans, and expected versus observed rows/errors. Native execution and JVM dispatch should be tested separately; switching to batch JVM dispatch is not by itself proof of Spark's row-by-row consumption. - Pair a skipped-invalid-input case with valid-input native controls and a case that deliberately consumes the invalid value and must raise. Keep any required physical input order explicit; SQL without `ORDER BY` does not guarantee a row order. - Cover ANSI/legacy/try distinctions where relevant, AQE and reused stages, and supported Spark versions. Include already-correct conditional and `element_at` guards as regression controls. - Add focused regressions for confirmed gaps, document remaining limitations in the compatibility guide, and keep `RequiresSparkEvaluationMask` enrollment and its scope explicit. Do not expand the policy simply because an expression is potentially throwing. ## Related work - [#5532](https://github.com/apache/datafusion-comet/issues/5532) / [#5533](https://github.com/apache/datafusion-comet/pull/5533): reproduced `unbase64` operator-consumption mismatch and its scoped fix. - [#5078](https://github.com/apache/datafusion-comet/issues/5078): per-expression ANSI behavior/error-fidelity audit. This tracking issue adds enclosing evaluation contexts; it does not replace the individual error-class issues. - [#5766](https://github.com/apache/datafusion-comet/pull/5766): preserves `element_at` NULL short-circuiting for deterministic operands and declines nullable nondeterministic operands under ANSI. [#5781](https://github.com/apache/datafusion-comet/issues/5781) separately tracks repeated evaluation of stateful operands in other NULL guards. - [#5218](https://github.com/apache/datafusion-comet/issues/5218) and [#5608](https://github.com/apache/datafusion-comet/issues/5608): fixed dispatcher NULL short-circuit cases that suppressed errors Spark would raise, the opposite outcome direction from eager evaluation of skipped rows. - [#5328](https://github.com/apache/datafusion-comet/issues/5328): fallback-invariance testing provides a useful additional oracle, with execution-path assertions to avoid vacuous comparisons. - [#5962](https://github.com/apache/datafusion-comet/issues/5962): moving reusable expressions before broadcast materialization also requires proving that error evaluation is preserved. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
