Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/21143
If we don't care about whole stage codegen, I think it's possible to
dynamically change the filter condition at executors and codegen it.
Now the problem becomes, how would the data source API look like? I'd image
some systems have to do filter pushdown at the driver side, so that they can
connect to the metadata service only once. One possible way is to allow users
to mixin `SupportsPushDownFilter` to both `DataSourceReader` and
`DataReaderFactory`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
