dwsmith1983 commented on code in PR #5654:
URL: https://github.com/apache/datafusion-comet/pull/5654#discussion_r4100259539
##########
native/core/src/parquet/schema_adapter.rs:
##########
@@ -3518,11 +3856,187 @@ mod test {
let physical = Arc::new(Schema::new(vec![
Field::new("a", DataType::Int32, true).with_metadata(id_meta("9"))
]));
- let (remapped, _name_map) =
+ let (remapped, _name_map, _) =
super::remap_physical_schema(&logical, &physical, true, true,
false).unwrap();
assert_eq!(remapped.field(0).name(), "a");
}
+ /// Build a nullable Int64 field carrying a Parquet field ID.
+ fn field_with_id(name: &str, id: i32) -> Field {
+ Field::new(name, DataType::Int64,
true).with_metadata(id_meta(&id.to_string()))
+ }
+
+ /// Write a Parquet file from `file_schema`/`columns`, then scan it with
+ /// `required_schema` through the Spark expression adapter and return the
first batch.
+ async fn scan_with_adapter(
+ file_schema: SchemaRef,
+ columns: Vec<Arc<dyn arrow::array::Array>>,
+ required_schema: SchemaRef,
+ spark_parquet_options: SparkParquetOptions,
+ ) -> Result<RecordBatch, DataFusionError> {
+ scan_with_defaults(
+ file_schema,
+ columns,
+ required_schema,
+ spark_parquet_options,
+ None,
+ )
+ .await
+ }
+
+ /// `scan_with_adapter` with column defaults for fields missing from the
file.
+ async fn scan_with_defaults(
Review Comment:
`scan_with_adapter` and `scan_with_defaults` are gone. The kept test calls
`scan_parquet` as it is. The key-value metadata was there so that a file whose
schema equals the requested one still reaches the adapter, and in this test the
schemas differ by the id on `y`, so the adapter runs without it.
`field_with_id` and `int_field_with_id` are gone too. The tests in all three
files use `id_meta` and `struct_type_with_field_id`, with
`struct_type_with_field_id` now `pub(crate)` in the `schema_adapter` test
module.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
