Github user liancheng commented on the issue:
https://github.com/apache/spark/pull/13371
@yhuai We used to support row group level filter push-down before
refactoring `HadoopFsRelation` into `FileFormat`, but lost it (by accident I
guess) after the refactoring. So now we only have row group level filtering
when the vectorized reader is not used, [see here][1].
And yes, both `ParquetInputFormat` and `ParquetRecordReader` do row group
level filtering.
This LGTM. Thanks for fixing it! Merging to master and 2.0.
[1]:
https://github.com/apache/spark/blob/54f758b5fc60ecb0da6b191939a72ef5829be38c/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala#L371-L378
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
