Github user mallman commented on the issue: https://github.com/apache/spark/pull/16578 > * There is a minor change needed though, "parquetFormat: ParquetFileFormat" should be replaced by "fileFormat: FileFormat" as there is no dependency on the actual ParquetFileFormat class defined in parquet package https://github.com/apache/spark/pull/16578/files?diff=unified#diff-3bad814b3336a83f360d7395bd740759R38 This is true. I hesitate to weaken the match to all instances of `FileFormat` because the only format I have extensive experience and knowledge of is Parquet. There are other formats that could expand upon this work, such as ORC and ROOT, but I have no practical experience working with those. I'd prefer someone who does have such experience build on this PR to make it work with those file formats. Incidentally, I added a `ColumnarFileFormat` trait in this PR. You might consider it a marker for columnar file formats, but all it really does right now is compute the number of physical columns read for a given catalyst schema. This value is used in the description of a physical plan to help a dev/user ensure that their expected column pruning is in fact occurring. > * And may be renaming this ParquetSchemaPruning and taking it outside of the parquet package as it is quite more general than just for parquet, otherwise I have to add a special Rule here, https://github.com/apache/spark/pull/16578/files?diff=unified#diff-2370d8ed85930c93ef8e5ce67abca53fR35 ??? Moving and renaming `ParquetSchemaPruning` makes sense if it's generalized to other file formats.
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org