GitHub user maryannxue opened a pull request: https://github.com/apache/spark/pull/21360
[SPARK-24288] Enable preventing predicate pushdown ## What changes were proposed in this pull request? 1. Add DataSet interface "withOptimizerBarrier()" 2. Modify AnalysisBarrier to accommodate two scenarios: 1) analysis-only barriers; and 2) optimizer barriers. 3. Add handling of Barrier in Optimizer (logical plan optimization). 4. Add handling of Barrier in SparkStrategies (logical-to-physical plan translation). ## How was this patch tested? 1. Add DataFrameOptimizerBarrierSuite to ensure: a) Barriers isolate optimization rule applications. b) Plans with optimization Barriers get resolved correctly, same as analysis-only Barriers. c) Barriers preserves ordering. d) Barriers preserves constraints. 2. Add one test in JDBCSuite to verify scenario raised by the user (avoid filter push-down to data source). You can merge this pull request into a Git repository by running: $ git pull https://github.com/maryannxue/spark spark-24288 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/21360.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #21360 ---- commit d57d89ad98d554c020a59c5c44a5f6725d6e916e Author: maryannxue <maryann.xue@...> Date: 2018-05-18T06:34:38Z [SPARK-24288] Enable preventing predicate pushdown ---- --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org