spark git commit: [SPARK-22675][SQL] Refactoring PropagateTypes in TypeCoercion

2017-12-05 Thread wenchen
Repository: spark Updated Branches: refs/heads/master a8af4da12 -> 53e5251bb [SPARK-22675][SQL] Refactoring PropagateTypes in TypeCoercion ## What changes were proposed in this pull request? PropagateTypes are called twice in TypeCoercion. We do not need to call it twice. Instead, we should c

spark git commit: [SPARK-20728][SQL] Make OrcFileFormat configurable between sql/hive and sql/core

2017-12-05 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 53e5251bb -> 326f1d672 [SPARK-20728][SQL] Make OrcFileFormat configurable between sql/hive and sql/core ## What changes were proposed in this pull request? This PR aims to provide a configuration to choose the default `OrcFileFormat` from

spark git commit: [SPARK-22681] Accumulator should only be updated once for each task in result stage

2017-12-05 Thread vanzin
Repository: spark Updated Branches: refs/heads/master 326f1d672 -> 03fdc92e4 [SPARK-22681] Accumulator should only be updated once for each task in result stage ## What changes were proposed in this pull request? As the doc says "For accumulator updates performed inside actions only, Spark g

spark git commit: [SPARK-22162][BRANCH-2.2] Executors and the driver should use consistent JobIDs in the RDD commit protocol

2017-12-05 Thread vanzin
Repository: spark Updated Branches: refs/heads/branch-2.2 f3f8c8767 -> 5b637 [SPARK-22162][BRANCH-2.2] Executors and the driver should use consistent JobIDs in the RDD commit protocol I have modified SparkHadoopMapReduceWriter so that executors and the driver always use consistent JobIds

spark git commit: [SPARK-22701][SQL] add ctx.splitExpressionsWithCurrentInputs

2017-12-05 Thread lixiao
Repository: spark Updated Branches: refs/heads/master 03fdc92e4 -> ced6ccf0d [SPARK-22701][SQL] add ctx.splitExpressionsWithCurrentInputs ## What changes were proposed in this pull request? This pattern appears many times in the codebase: ``` if (ctx.INPUT_ROW == null || ctx.currentVars != nu

spark git commit: [SPARK-22500][SQL][FOLLOWUP] cast for struct can split code even with whole stage codegen

2017-12-05 Thread lixiao
Repository: spark Updated Branches: refs/heads/master ced6ccf0d -> 132a3f470 [SPARK-22500][SQL][FOLLOWUP] cast for struct can split code even with whole stage codegen ## What changes were proposed in this pull request? A followup of https://github.com/apache/spark/pull/19730, we can split th

spark git commit: [SPARK-22662][SQL] Failed to prune columns after rewriting predicate subquery

2017-12-05 Thread lixiao
Repository: spark Updated Branches: refs/heads/master 132a3f470 -> 1e17ab83d [SPARK-22662][SQL] Failed to prune columns after rewriting predicate subquery ## What changes were proposed in this pull request? As a simple example: ``` spark-sql> create table base (a int, b int) using parquet; Ti

spark git commit: [SPARK-20706][SPARK-SHELL] Spark-shell not overriding method/variable definition

2017-12-05 Thread srowen
Repository: spark Updated Branches: refs/heads/master 1e17ab83d -> 59aa3d56a [SPARK-20706][SPARK-SHELL] Spark-shell not overriding method/variable definition ## What changes were proposed in this pull request? [SPARK-20706](https://issues.apache.org/jira/browse/SPARK-20706): Spark-shell not

spark git commit: [SPARK-22686][SQL] DROP TABLE IF EXISTS should not show AnalysisException

2017-12-05 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 59aa3d56a -> 82183f7b5 [SPARK-22686][SQL] DROP TABLE IF EXISTS should not show AnalysisException ## What changes were proposed in this pull request? During [SPARK-22488](https://github.com/apache/spark/pull/19713) to fix view resolution i

spark git commit: [SPARK-22686][SQL] DROP TABLE IF EXISTS should not show AnalysisException

2017-12-05 Thread wenchen
Repository: spark Updated Branches: refs/heads/branch-2.2 5b637 -> 7fd6d535d [SPARK-22686][SQL] DROP TABLE IF EXISTS should not show AnalysisException ## What changes were proposed in this pull request? During [SPARK-22488](https://github.com/apache/spark/pull/19713) to fix view resoluti

spark git commit: [SPARK-20392][SQL] Set barrier to prevent re-entering a tree

2017-12-05 Thread lixiao
Repository: spark Updated Branches: refs/heads/master 82183f7b5 -> 00d176d2f [SPARK-20392][SQL] Set barrier to prevent re-entering a tree ## What changes were proposed in this pull request? The SQL `Analyzer` goes through a whole query plan even most part of it is analyzed. This increases th