[4/4] spark git commit: [SPARK-9830][SQL] Remove AggregateExpression1 and Aggregate Operator used to evaluate AggregateExpression1s
[SPARK-9830][SQL] Remove AggregateExpression1 and Aggregate Operator used to evaluate AggregateExpression1s https://issues.apache.org/jira/browse/SPARK-9830 This PR contains the following main changes. * Removing `AggregateExpression1`. * Removing `Aggregate` operator, which is used to evaluate `AggregateExpression1`. * Removing planner rule used to plan `Aggregate`. * Linking `MultipleDistinctRewriter` to analyzer. * Renaming `AggregateExpression2` to `AggregateExpression` and `AggregateFunction2` to `AggregateFunction`. * Updating places where we create aggregate expression. The way to create aggregate expressions is `AggregateExpression(aggregateFunction, mode, isDistinct)`. * Changing `val`s in `DeclarativeAggregate`s that touch children of this function to `lazy val`s (when we create aggregate expression in DataFrame API, children of an aggregate function can be unresolved). Author: Yin Huai Closes #9556 from yhuai/removeAgg1. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/e0701c75 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/e0701c75 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/e0701c75 Branch: refs/heads/master Commit: e0701c75601c43f69ed27fc7c252321703db51f2 Parents: 6e5fc37 Author: Yin Huai Authored: Tue Nov 10 11:06:29 2015 -0800 Committer: Michael Armbrust Committed: Tue Nov 10 11:06:29 2015 -0800 -- R/pkg/R/functions.R |2 +- python/pyspark/sql/dataframe.py |2 +- python/pyspark/sql/functions.py |2 +- python/pyspark/sql/tests.py |2 +- .../spark/sql/catalyst/CatalystConf.scala | 10 +- .../apache/spark/sql/catalyst/SqlParser.scala | 14 +- .../spark/sql/catalyst/analysis/Analyzer.scala | 26 +- .../sql/catalyst/analysis/CheckAnalysis.scala | 46 +- .../analysis/DistinctAggregationRewriter.scala | 278 + .../catalyst/analysis/FunctionRegistry.scala|2 + .../catalyst/analysis/HiveTypeCoercion.scala| 20 +- .../sql/catalyst/analysis/unresolved.scala |4 + .../apache/spark/sql/catalyst/dsl/package.scala | 22 +- .../expressions/aggregate/Average.scala | 31 +- .../aggregate/CentralMomentAgg.scala| 13 +- .../catalyst/expressions/aggregate/Corr.scala | 15 + .../catalyst/expressions/aggregate/Count.scala | 28 +- .../catalyst/expressions/aggregate/First.scala | 14 +- .../aggregate/HyperLogLogPlusPlus.scala | 17 + .../expressions/aggregate/Kurtosis.scala|2 + .../catalyst/expressions/aggregate/Last.scala | 12 +- .../catalyst/expressions/aggregate/Max.scala| 17 +- .../catalyst/expressions/aggregate/Min.scala| 17 +- .../expressions/aggregate/Skewness.scala|2 + .../catalyst/expressions/aggregate/Stddev.scala | 31 +- .../catalyst/expressions/aggregate/Sum.scala| 29 +- .../catalyst/expressions/aggregate/Utils.scala | 467 .../expressions/aggregate/Variance.scala|7 +- .../expressions/aggregate/interfaces.scala | 57 +- .../sql/catalyst/expressions/aggregates.scala | 1073 -- .../sql/catalyst/optimizer/Optimizer.scala | 23 +- .../spark/sql/catalyst/planning/patterns.scala | 74 -- .../spark/sql/catalyst/plans/QueryPlan.scala| 12 +- .../catalyst/plans/logical/basicOperators.scala |4 +- .../catalyst/analysis/AnalysisErrorSuite.scala | 23 +- .../sql/catalyst/analysis/AnalysisSuite.scala |2 +- .../analysis/DecimalPrecisionSuite.scala|1 + .../analysis/ExpressionTypeCheckingSuite.scala |6 +- .../optimizer/ConstantFoldingSuite.scala|4 +- .../optimizer/FilterPushdownSuite.scala | 14 +- .../scala/org/apache/spark/sql/DataFrame.scala | 13 +- .../org/apache/spark/sql/GroupedData.scala | 45 +- .../scala/org/apache/spark/sql/SQLConf.scala| 20 +- .../apache/spark/sql/execution/Aggregate.scala | 205 .../org/apache/spark/sql/execution/Expand.scala |3 + .../spark/sql/execution/SparkPlanner.scala |1 - .../spark/sql/execution/SparkStrategies.scala | 238 ++-- .../aggregate/AggregationIterator.scala | 28 +- .../aggregate/SortBasedAggregate.scala |4 +- .../SortBasedAggregationIterator.scala |8 +- .../execution/aggregate/TungstenAggregate.scala |6 +- .../aggregate/TungstenAggregationIterator.scala | 36 +- .../spark/sql/execution/aggregate/udaf.scala|2 +- .../spark/sql/execution/aggregate/utils.scala | 20 +- .../spark/sql/expressions/Aggregator.scala |5 +- .../spark/sql/expressions/WindowSpec.scala | 82 +- .../org/apache/spark/sql/expressions/udaf.scala |6 +- .../scala/org/apache/spark/sql/functions.scala | 53 +- .../org/apache/spark/sql/SQLQuerySuite.scal
[4/4] spark git commit: [SPARK-9830][SQL] Remove AggregateExpression1 and Aggregate Operator used to evaluate AggregateExpression1s
[SPARK-9830][SQL] Remove AggregateExpression1 and Aggregate Operator used to evaluate AggregateExpression1s https://issues.apache.org/jira/browse/SPARK-9830 This PR contains the following main changes. * Removing `AggregateExpression1`. * Removing `Aggregate` operator, which is used to evaluate `AggregateExpression1`. * Removing planner rule used to plan `Aggregate`. * Linking `MultipleDistinctRewriter` to analyzer. * Renaming `AggregateExpression2` to `AggregateExpression` and `AggregateFunction2` to `AggregateFunction`. * Updating places where we create aggregate expression. The way to create aggregate expressions is `AggregateExpression(aggregateFunction, mode, isDistinct)`. * Changing `val`s in `DeclarativeAggregate`s that touch children of this function to `lazy val`s (when we create aggregate expression in DataFrame API, children of an aggregate function can be unresolved). Author: Yin Huai Closes #9556 from yhuai/removeAgg1. (cherry picked from commit e0701c75601c43f69ed27fc7c252321703db51f2) Signed-off-by: Michael Armbrust Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/7c4ade0d Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/7c4ade0d Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/7c4ade0d Branch: refs/heads/branch-1.6 Commit: 7c4ade0d7665e0f473d00f4a812fa69a0e0d14b5 Parents: 825e971 Author: Yin Huai Authored: Tue Nov 10 11:06:29 2015 -0800 Committer: Michael Armbrust Committed: Tue Nov 10 11:06:48 2015 -0800 -- R/pkg/R/functions.R |2 +- python/pyspark/sql/dataframe.py |2 +- python/pyspark/sql/functions.py |2 +- python/pyspark/sql/tests.py |2 +- .../spark/sql/catalyst/CatalystConf.scala | 10 +- .../apache/spark/sql/catalyst/SqlParser.scala | 14 +- .../spark/sql/catalyst/analysis/Analyzer.scala | 26 +- .../sql/catalyst/analysis/CheckAnalysis.scala | 46 +- .../analysis/DistinctAggregationRewriter.scala | 278 + .../catalyst/analysis/FunctionRegistry.scala|2 + .../catalyst/analysis/HiveTypeCoercion.scala| 20 +- .../sql/catalyst/analysis/unresolved.scala |4 + .../apache/spark/sql/catalyst/dsl/package.scala | 22 +- .../expressions/aggregate/Average.scala | 31 +- .../aggregate/CentralMomentAgg.scala| 13 +- .../catalyst/expressions/aggregate/Corr.scala | 15 + .../catalyst/expressions/aggregate/Count.scala | 28 +- .../catalyst/expressions/aggregate/First.scala | 14 +- .../aggregate/HyperLogLogPlusPlus.scala | 17 + .../expressions/aggregate/Kurtosis.scala|2 + .../catalyst/expressions/aggregate/Last.scala | 12 +- .../catalyst/expressions/aggregate/Max.scala| 17 +- .../catalyst/expressions/aggregate/Min.scala| 17 +- .../expressions/aggregate/Skewness.scala|2 + .../catalyst/expressions/aggregate/Stddev.scala | 31 +- .../catalyst/expressions/aggregate/Sum.scala| 29 +- .../catalyst/expressions/aggregate/Utils.scala | 467 .../expressions/aggregate/Variance.scala|7 +- .../expressions/aggregate/interfaces.scala | 57 +- .../sql/catalyst/expressions/aggregates.scala | 1073 -- .../sql/catalyst/optimizer/Optimizer.scala | 23 +- .../spark/sql/catalyst/planning/patterns.scala | 74 -- .../spark/sql/catalyst/plans/QueryPlan.scala| 12 +- .../catalyst/plans/logical/basicOperators.scala |4 +- .../catalyst/analysis/AnalysisErrorSuite.scala | 23 +- .../sql/catalyst/analysis/AnalysisSuite.scala |2 +- .../analysis/DecimalPrecisionSuite.scala|1 + .../analysis/ExpressionTypeCheckingSuite.scala |6 +- .../optimizer/ConstantFoldingSuite.scala|4 +- .../optimizer/FilterPushdownSuite.scala | 14 +- .../scala/org/apache/spark/sql/DataFrame.scala | 13 +- .../org/apache/spark/sql/GroupedData.scala | 45 +- .../scala/org/apache/spark/sql/SQLConf.scala| 20 +- .../apache/spark/sql/execution/Aggregate.scala | 205 .../org/apache/spark/sql/execution/Expand.scala |3 + .../spark/sql/execution/SparkPlanner.scala |1 - .../spark/sql/execution/SparkStrategies.scala | 238 ++-- .../aggregate/AggregationIterator.scala | 28 +- .../aggregate/SortBasedAggregate.scala |4 +- .../SortBasedAggregationIterator.scala |8 +- .../execution/aggregate/TungstenAggregate.scala |6 +- .../aggregate/TungstenAggregationIterator.scala | 36 +- .../spark/sql/execution/aggregate/udaf.scala|2 +- .../spark/sql/execution/aggregate/utils.scala | 20 +- .../spark/sql/expressions/Aggregator.scala |5 +- .../spark/sql/expressions/WindowSpec.scala | 82 +- .../org/apache/spark/sql/expressions/udaf.scala |6
