Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/20387
This PR does 3 things:
1. make `DataSourceV2Relation` immutable. This extends the constructor of
`DataSourceV2Relation` to include pushed filters and pruned columns.
2. carry some standard information(table, path, etc.) into
`DataSourceV2Relation`. This extends the constructor of `DataSourceV2Relation`
to include table identifier, path string etc.
3. replace the new operator pushdown rule with `PhysicalOperation`.
It will be great if we only focus on 1, but I'm also OK if we do 1 and 3
together. I don't think we should include 2 here, as it's unclear what's the
benefit. These stardard information are only used to create `DataSourceOptions`
inside `DataSourceV2Relation`, which can also be done in `DataFrameReader`. I
suggest we don't change this part and just keep the `DataSourceOption` in the
constructor of `DataSourceV2Relation`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
