raghuvanshraj opened a new issue, #11917: URL: https://github.com/apache/arrow-rs/issues/11917
### Describe the bug When writing an Arrow struct field tagged `ARROW:extension:name = arrow.parquet.variant`, the Parquet writer annotates the group with `VARIANT` but leaves `specification_version` unset: https://github.com/apache/arrow-rs/blob/5b02c08e6ae76bceba53e41e7f66e0876741da4d/parquet/src/arrow/schema/extension.rs#L127-L134 ```rust pub(crate) fn logical_type_for_struct(field: &Field) -> Option<LogicalType> { use parquet_variant_compute::VariantType; if field.has_valid_extension_type::<VariantType>() { Some(LogicalType::variant(None)) ``` This is the only place the writer produces the annotation ([`arrow_to_parquet_type`](https://github.com/apache/arrow-rs/blob/5b02c08e6ae76bceba53e41e7f66e0876741da4d/parquet/src/arrow/schema/mod.rs#L939)). The thrift field is `optional i8 specification_version` ([`parquet.thrift` L416-L420](https://github.com/apache/parquet-format/blob/e365c6f7c05eea69c5aa11f1ead6c3db261ca9f9/src/main/thrift/parquet.thrift#L416-L420)), so the file is well-formed thrift. But the Parquet spec requires the version in the declaration: "The Variant group must be annotated with the `VARIANT` logical type, with the version number included in the declaration" ([`LogicalTypes.md` L602-L603](https://github.com/apache/parquet-format/blob/e365c6f7c05eea69c5aa11f1ead6c3db261ca9f9/LogicalTypes.md#L602-L603)), and every example in `LogicalTypes.md` and `VariantShredding.md` is `VARIANT(1)`. Spark 4.1 enforces this. It reads the annotation through parquet-java, whose thrift getter returns `0` when the field is unset ([`ParquetMetadataConverter` L1430-L1432](https://github.com/apache/parquet-java/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/format/converter/ParquetMetadataConverter.java#L1430-L1432)), and Spark only accepts version 1: https://github.com/apache/spark/blob/branch-4.1/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaConverter.scala#L395 ```scala case v: VariantLogicalTypeAnnotation if v.getSpecVersion == 1 => ``` Any other value falls through to [`unrecognizedParquetTypeError`](https://github.com/apache/spark/blob/branch-4.1/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaConverter.scala#L491) and the read fails with `[PARQUET_TYPE_NOT_RECOGNIZED] Unrecognized Parquet type: ...`. Spark's own writer always sets `1` ([L860](https://github.com/apache/spark/blob/branch-4.1/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaConverter.scala#L860)). The arrow-rs reader is lenient ([`try_add_extension_type` L50-L55](https://github.com/apache/arrow-rs/blob/5b02c08e6ae76bceba53e41e7f66e0876741da4d/parquet/src/arrow/schema/extension.rs#L50-L55) matches `LogicalType::Variant(_)`), so arrow-rs round-trips its own files and the gap only shows when another engine reads them. ### To Reproduce Schema conversion alone shows it (feature `variant_experimental`): ```rust use arrow_schema::{DataType, Field, Schema}; use parquet::arrow::ArrowSchemaConverter; use parquet::basic::LogicalType; use parquet_variant_compute::VariantType; let fields = vec![ Field::new("metadata", DataType::BinaryView, false), Field::new("value", DataType::BinaryView, true), ]; let v = Field::new("v", DataType::Struct(fields.into()), true) .with_extension_type(VariantType); let schema = ArrowSchemaConverter::new() .convert(&Schema::new(vec![v])) .unwrap(); let root = schema.root_schema(); assert_eq!( root.get_fields()[0].get_basic_info().logical_type_ref(), Some(&LogicalType::variant(Some(1))), ); // left: Some(Variant(VariantType { specification_version: None })) ``` End to end: 1. Write a file with one `arrow.parquet.variant` column using `ArrowWriter`. 2. `parquet-schema <file>` prints the group as `VARIANT(None)` ([`printer.rs` L322-L324](https://github.com/apache/arrow-rs/blob/5b02c08e6ae76bceba53e41e7f66e0876741da4d/parquet/src/schema/printer.rs#L322-L324)); parquet-java tools print `VARIANT(0)`. 3. `spark.read.parquet(path)` on Spark 4.1 fails with `PARQUET_TYPE_NOT_RECOGNIZED`. Reproduced 2026-09-25 on Spark 4.1 (AWS Glue) with arrow-rs `<VERSION>`. Code unchanged on `main` at `5b02c08` (60.1.0 dev, 2026-10-26). ### Expected behavior The writer sets `specification_version = 1`, the only version the spec defines, so the file matches `VARIANT(1)` in `LogicalTypes.md` and reads in Spark. ### Possible fix `Some(LogicalType::variant(Some(1)))` in `logical_type_for_struct`, plus a round-trip test asserting the version on the written schema. Happy to open the PR. ### Additional context Found while working on DataFusion Variant support (apache/datafusion#26517). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
