dwsmith1983 commented on code in PR #6116:
URL: https://github.com/apache/datafusion-comet/pull/6116#discussion_r4100373808
##########
native/core/src/parquet/eager_page_index_reader_factory.rs:
##########
@@ -191,13 +204,40 @@ impl EagerPageIndexReaderFactory {
metadata_cache,
scan_io_metrics,
spark_variant_schema: false,
+ requested_schema_has_field_ids: false,
+ ignore_missing_field_id: false,
}
}
pub fn with_spark_variant_schema(mut self, enabled: bool) -> Self {
self.spark_variant_schema = enabled;
self
}
+
+ /// Arm Spark's missing field id check. A file whose Parquet schema
carries no field id
+ /// is refused on open when `requested_schema_has_field_ids` is set and
+ /// `ignore_missing_field_id` is not. Both default to off, so a factory
that never calls
+ /// this reads every file.
+ pub fn with_missing_field_id_check(
+ mut self,
+ requested_schema_has_field_ids: bool,
+ ignore_missing_field_id: bool,
+ ) -> Self {
+ self.requested_schema_has_field_ids = requested_schema_has_field_ids;
+ self.ignore_missing_field_id = ignore_missing_field_id;
+ self
+ }
+}
+
+/// True when `node` or any node under it carries a field id, the way Spark's
+/// `containsFieldIds` answers it over the raw Parquet schema, message root
included.
+fn parquet_schema_has_field_ids(node: &ParquetType) -> bool {
Review Comment:
Renamed to `contains_field_ids` and `any_root_field_has_id`. The Arrow walk
is gone with the proto change, so those are the only two left.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
