Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21764#discussion_r202521451
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -46,7 +47,23 @@ abstract class Optimizer(sessionCatalog: SessionCatalog)
protected def fixedPoint = FixedPoint(SQLConf.get.optimizerMaxIterations)
- def batches: Seq[Batch] = {
+ protected def postAnalysisBatches: Seq[Batch] = {
+ Batch("Eliminate Distinct", Once, EliminateDistinct) ::
+ // Technically some of the rules in Finish Analysis are not optimizer
rules and belong more
+ // in the analyzer, because they are needed for correctness (e.g.
ComputeCurrentTime).
+ // However, because we also use the analyzer to canonicalized queries
(for view definition),
+ // we do not eliminate subqueries or compute current time in the
analyzer.
+ Batch("Finish Analysis", Once,
+ EliminateSubqueryAliases,
+ EliminateView,
+ ReplaceExpressions,
+ ComputeCurrentTime,
+ GetCurrentDatabase(sessionCatalog),
+ RewriteDistinctAggregates,
+ ReplaceDeduplicateWithAggregate) :: Nil
+ }
+
+ protected def optimizationBatches: Seq[Batch] = {
--- End diff --
In `optimizationBatches`, some rules can't be excluded. Without them, the
affected queries can't be executed. For example,
```Scala
Batch("Replace Operators", fixedPoint,
ReplaceIntersectWithSemiJoin,
ReplaceExceptWithFilter,
ReplaceExceptWithAntiJoin,
ReplaceDistinctWithAggregate)
```
Can we just introduce a black list?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
