Github user mallman commented on the issue:
https://github.com/apache/spark/pull/22357
> FYI, per further checking code and discussion with @dbtsai regarding with
predicate pushdown, we know that predicate push down only works for primitive
types on Parquet datasource. So both `IsNotNull(employer)` and
`IsNotNull(employer.id)` are not actually pushed down to work at Parquet reader
I would expect `IsNotNull(employer.id)` to be pushed down. In any case, I
misunderstood what that `PushedFilters` metadata item means in the `FileScan`
part of the physical plan. I thought that was a Parquet filter, but sometimes
it is not. In any case, I'm not concerned about supporting filter push down at
this point. My concern is around its side effects, but that has been allayed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
