spark git commit: [SPARK-19537] Move pendingPartitions to ShuffleMapStage.

2017-02-10 Thread kayousterhout
Repository: spark Updated Branches: refs/heads/master 226d38840 -> 0fbecc736 [SPARK-19537] Move pendingPartitions to ShuffleMapStage. The pendingPartitions instance variable should be moved to ShuffleMapStage, because it is only used by ShuffleMapStages. This change is purely refactoring and

spark git commit: [SPARK-18717][SQL] Make code generation for Scala Map work with immutable.Map also

2017-02-10 Thread lian
Repository: spark Updated Branches: refs/heads/branch-2.1 7b5ea000e -> e580bb035 [SPARK-18717][SQL] Make code generation for Scala Map work with immutable.Map also ## What changes were proposed in this pull request? Fixes compile errors in generated code when user has case class with a

spark git commit: [SPARK-19548][SQL] Support Hive UDFs which return typed Lists/Maps

2017-02-10 Thread wenchen
Repository: spark Updated Branches: refs/heads/master d785217b7 -> 226d38840 [SPARK-19548][SQL] Support Hive UDFs which return typed Lists/Maps ## What changes were proposed in this pull request? This PR adds support for Hive UDFs that return fully typed java Lists or Maps, for example

spark git commit: [SPARK-19549] Allow providing reason for stage/job cancelling

2017-02-10 Thread rxin
Repository: spark Updated Branches: refs/heads/master 3a43ae7c0 -> d785217b7 [SPARK-19549] Allow providing reason for stage/job cancelling ## What changes were proposed in this pull request? This change add an optional argument to `SparkContext.cancelStage()` and `SparkContext.cancelJob()`

spark git commit: [SPARK-18613][ML] make spark.mllib LDA dependencies in spark.ml LDA private

2017-02-10 Thread jkbradley
Repository: spark Updated Branches: refs/heads/master de8a03e68 -> 3a43ae7c0 [SPARK-18613][ML] make spark.mllib LDA dependencies in spark.ml LDA private ## What changes were proposed in this pull request? spark.ml.*LDAModel classes were exposing spark.mllib LDA models via protected methods.

spark git commit: [SPARK-19459][SQL] Add Hive datatype (char/varchar) to StructField metadata

2017-02-10 Thread wenchen
Repository: spark Updated Branches: refs/heads/master dadff5f07 -> de8a03e68 [SPARK-19459][SQL] Add Hive datatype (char/varchar) to StructField metadata ## What changes were proposed in this pull request? Reading from an existing ORC table which contains `char` or `varchar` columns can fail

spark git commit: Encryption of shuffle files

2017-02-10 Thread rxin
Repository: spark Updated Branches: refs/heads/master 8640dc082 -> c5a66356d Encryption of shuffle files Hello According to my understanding of commits 4b4e329e49f8af28fa6301bd06c48d7097eaf9e6 & 8b325b17ecdf013b7a6edcb7ee3773546bd914df, one may now encrypt shuffle files regardless of the

spark git commit: [SPARK-10748][MESOS] Log error instead of crashing Spark Mesos dispatcher when a job is misconfigured

2017-02-10 Thread srowen
Repository: spark Updated Branches: refs/heads/master 8e8afb3a3 -> 8640dc082 [SPARK-10748][MESOS] Log error instead of crashing Spark Mesos dispatcher when a job is misconfigured ## What changes were proposed in this pull request? Now handling the spark exception which gets thrown for

spark git commit: [SPARK-19545][YARN] Fix compile issue for Spark on Yarn when building against Hadoop 2.6.0~2.6.3

2017-02-10 Thread srowen
Repository: spark Updated Branches: refs/heads/master d5593f7f5 -> 8e8afb3a3 [SPARK-19545][YARN] Fix compile issue for Spark on Yarn when building against Hadoop 2.6.0~2.6.3 ## What changes were proposed in this pull request? Due to the newly added API in Hadoop 2.6.4+, Spark builds against

spark git commit: [SPARK-19543] from_json fails when the input row is empty

2017-02-10 Thread hvanhovell
Repository: spark Updated Branches: refs/heads/branch-2.1 ff5818b8c -> 7b5ea000e [SPARK-19543] from_json fails when the input row is empty ## What changes were proposed in this pull request? Using from_json on a column with an empty string results in: java.util.NoSuchElementException: head

spark git commit: [SPARK-19543] from_json fails when the input row is empty

2017-02-10 Thread hvanhovell
Repository: spark Updated Branches: refs/heads/master fd6c3a0b1 -> d5593f7f5 [SPARK-19543] from_json fails when the input row is empty ## What changes were proposed in this pull request? Using from_json on a column with an empty string results in: java.util.NoSuchElementException: head of

spark git commit: [SPARK-19512][BACKPORT-2.1][SQL] codegen for compare structs fails #16852

2017-02-10 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.1 a3d5300a0 -> ff5818b8c [SPARK-19512][BACKPORT-2.1][SQL] codegen for compare structs fails #16852 ## What changes were proposed in this pull request? Set currentVars to null in GenerateOrdering.genComparisons before genCode is called.