rich7420 opened a new pull request, #6180: URL: https://github.com/apache/datafusion-comet/pull/6180
## Which issue does this PR close? Closes #6093. ## Rationale for this change `AtLeastNNonNulls`, used by `DataFrame.na.drop`, currently falls back to Spark. This adds native support without codegen dispatch. ## What changes are included in this PR? - Add a native expression that counts non-null, non-NaN children and preserves Spark's per-row short circuit and ANSI errors. - Embed empty projections into filter output so `count(*)` avoids materializing unused arrays, while preserving row counts and Spark metrics. - Add regression tests, documentation and benchmarks against Spark and Comet fallback. ## How are these changes tested? [Fork CI](https://github.com/rich7420/datafusion-comet/actions/runs/35977330665) passed Rust tests, Spark 4.1 Comet suites, TPC-H/TPC-DS checks and lint. Local Spark 3.5/4.1 expression and projection tests also passed. Disabling the projection optimization makes its regression fail. Local release benchmarks on M3 Pro, Spark 4.1.3/JDK 21, `local[1]`, 1,048,576 rows and 32 columns, `na.drop(16)`; best elapsed milliseconds: | Query | Native | Comet fallback | | --- | ---: | ---: | | String, 50% NULL, count only | 398 | 423 | | String, 50% NULL, aggregate all columns | 629 | 901 | | Double, 0% NULL, aggregate all columns | 426 | 364 | Five benchmark cases passed result and execution-path checks. Some double aggregations remain 14–19% slower than fallback; the original workload is still unverified. Full Spark SQL and other Spark-profile CI results are pending. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
