yjshen commented on a change in pull request #811:
URL: https://github.com/apache/arrow-datafusion/pull/811#discussion_r686711509
##########
File path: datafusion/src/datasource/parquet.rs
##########
@@ -120,14 +132,303 @@ impl TableProvider for ParquetTable {
}
fn statistics(&self) -> Statistics {
- self.statistics.clone()
+ self.desc.statistics()
}
fn has_exact_statistics(&self) -> bool {
true
}
}
+#[derive(Debug)]
+/// Descriptor for a parquet root path
+pub struct ParquetRootDesc {
+ /// object store for reading files inside the root path
+ pub object_store: Arc<dyn ObjectStore>,
+ /// metadata for files inside the root path
+ pub descriptor: SourceRootDescriptor,
+}
+
+impl ParquetRootDesc {
+ /// Construct a new parquet descriptor for a root path
+ pub fn new(root_path: &str, context: ExecutionContext) -> Result<Self> {
+ let object_store = context
+ .state
+ .lock()
+ .unwrap()
+ .object_store_registry
+ .store_for_path(root_path);
+ let root_desc = Self::get_source_desc(root_path, object_store.clone(),
"parquet");
Review comment:
Agree, it was restricted to parquet suffix in the original
implementation, so I moved it here. Probably we could make it as an argument
and ask from the user?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
