peterxcli opened a new pull request, #5631: URL: https://github.com/apache/datafusion-comet/pull/5631
## Which issue does this PR close? Closes #5548. This is the second atomic change in #5546 and builds on the Field-aware Arrow C Data export merged in #5552. It intentionally does not enable native Variant scans. ## Rationale for this change [Spark 4 maps `VariantType` to non-null `value` and `metadata` binary fields](https://github.com/apache/spark/blob/v4.0.4/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaConverter.scala#L790-L794), while [arrow-rs identifies that physical Struct as the `arrow.parquet.variant` extension type](https://github.com/apache/arrow-rs/blob/58.4.0/parquet-variant-compute/src/variant_array.rs#L77-L104). Comet needs both representations: the physical Struct for Arrow transport and the Field-level extension marker to retain its logical Variant identity. Without an explicit protobuf type and Field marker, Comet cannot carry Spark `VariantType` through schema serialization or recover it when importing an Arrow Field on the JVM. This change establishes that type contract while keeping native scan, expression, and operator admission closed. A later scan PR can therefore transport Variant values without treating an ordinary two-binary-field Struct as Variant. ## What changes are included in this PR? - Append `VARIANT = 21` to protobuf `DataTypeId` without renumbering existing values. - Serialize Spark 4 `VariantType` through version-specific shims while keeping Spark 3.x inert. - Map Variant natively to `Struct<value: Binary, metadata: Binary>` and attach the canonical Arrow Variant extension marker to its Field. - Preserve generated extension metadata alongside Spark and Parquet field metadata. - Recover `VariantType` in `Utils.fromArrowField` only for the exact `arrow.parquet.variant` marker; identical unmarked or differently marked Structs remain `StructType`. - Keep direct and nested Variant scans, expressions, and operators on explicit Spark fallback despite making schema serialization available. - Add focused native, JVM, and SQL regression coverage for identity preservation and fallback boundaries. This PR does not add Parquet scan admission, Variant unshredding or normalization, existence defaults, Variant functions or casts, native C2R, shuffle/spill, Python transport, writes, Iceberg support, or nested Variant execution. Those remain in later #5546 subtasks. ## How are these changes tested? ```shell cd native DYLD_LIBRARY_PATH="${JAVA_HOME}/lib/server" cargo test -p datafusion-comet variant cargo fmt --all -- --check cd .. make core mvn -o -ntp -Pspark-4.0 -Dtest=none \ '-Dsuites=org.apache.comet.CometVariantTypeSuite' test mvn -o -ntp -Pspark-4.0 -Dtest=none \ '-Dsuites=org.apache.comet.CometSqlFileTestSuite variant' test mvn -o -ntp -Pspark-3.5 -Dtest=none \ '-Dsuites=org.apache.comet.CometVariantTypeSuite' test git diff --check ``` The focused native tests passed (2 passed, 0 failed), `make core` succeeded, both focused JVM suites passed, the Spark 4 Variant SQL suite passed, Spark 3.5 compiled and passed without exposing Spark 4 Variant classes, and formatting/style checks passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
