Github user yhuai commented on the issue:
https://github.com/apache/spark/pull/13371
@viirya I took a look at parquet's code. Seems parquet only evaluate row
group level filters when generating splits
(https://github.com/apache/parquet-mr/blob/apache-parquet-1.7.0/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetInputFormat.java#L673).
With FileSourceStrategy in Spark, I am not sure we will actually evaluate
filter unneeded row groups as expected. Can you take a look? Also, it will be
great if you can have a test to make sure that we actually can skip unneeded
row groups. This test can be created as follows.
1. We first write a parquet file containing multiple row groups. Also,
let's that there is a column `c` and those row groups have disjoint ranges of
`c`'s values.
2. We write a query having a filter on `c` and we make sure that this query
only need a subset of row groups.
3. We verify that we only create splits for the needed row groups.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
