LinSimon-901101 opened a new pull request, #7155: URL: https://github.com/apache/datafusion-comet/pull/7155
## Which issue does this PR close? Closes #7049. ## Rationale for this change On Spark 3.5+, writing nullable input to required DataSource V2 fields inserts `AssertNotNull`. Without native support, this projection can force otherwise eligible native Iceberg writes to fall back, including unpartitioned tables and writes with `write.distribution-mode=none`. ## What changes are included in this PR? - Add native `AssertNotNull` serialization, planning and evaluation. Preserve the child value and validate logical nulls, including dictionary arrays and outer struct validity. - Preserve Spark's version-specific exception type, message and walked type path. - Support the structural and conditional expressions used by V2 writes while preserving existing JVM codegen dispatch. - Preserve short-circuit and partial-consumption semantics through conservative fallback, including assertions below `LIMIT` and in partially consumed TopK or aggregate result projections. Reuse the existing partial-reader tracking. - Document the supported contexts and fallback restrictions. Native Iceberg writes remain opt-in. ## How are these changes tested? - Native unit tests cover values, slices, scalar and dictionary nulls, empty batches, struct validity and error paths. - Expression regressions cover Spark error parity, conditional and short-circuit evaluation, codegen dispatch, and partial consumption with AQE enabled and disabled. - Iceberg regressions cover SQL inserts and DataFrame appends into required top-level and nested fields, nullable parent structs, limited input, and rejection of null input without committing a snapshot. - Local targeted validation passed 115 JVM test executions across Spark 3.4–4.2 and four native tests. Writer regressions ran on Spark 3.5, 4.0 and 4.1; local Spark 3.4 and 4.2 coverage is expression-only. - The unchanged upstream [Comet CI workflow](https://github.com/LinSimon-901101/datafusion-comet/actions/runs/39644062061) passed on commit `9737fc3c8f37a5fa332fc18ccb640f154559a5da`: 137 successful jobs, 13 expected conditional skips, and no failed or canceled jobs. The complete configured matrices and critical test steps were verified. Validation establishes native execution and semantic parity; throughput has not been benchmarked. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
