spark git commit: [SPARK-16331][SQL] Reduce code generation time

Repository: spark Updated Branches: refs/heads/master aa6564f37 -> 14cf61e90 [SPARK-16331][SQL] Reduce code generation time ## What changes were proposed in this pull request? During the code generation, a `LocalRelation` often has a huge `Vector` object as `data`. In the simple example

spark git commit: [SPARK-14608][ML] transformSchema needs better documentation

Repository: spark Updated Branches: refs/heads/branch-2.0 80a7bff89 -> cc3c44b11 [SPARK-14608][ML] transformSchema needs better documentation ## What changes were proposed in this pull request? jira: https://issues.apache.org/jira/browse/SPARK-14608 PipelineStage.transformSchema currently has

spark git commit: [SPARK-14608][ML] transformSchema needs better documentation

Repository: spark Updated Branches: refs/heads/master 38f4d6f44 -> aa6564f37 [SPARK-14608][ML] transformSchema needs better documentation ## What changes were proposed in this pull request? jira: https://issues.apache.org/jira/browse/SPARK-14608 PipelineStage.transformSchema currently has

spark git commit: [SPARK-15820][PYSPARK][SQL] Add Catalog.refreshTable into python API

Repository: spark Updated Branches: refs/heads/branch-2.0 94d61de9c -> 80a7bff89 [SPARK-15820][PYSPARK][SQL] Add Catalog.refreshTable into python API ## What changes were proposed in this pull request? Add Catalog.refreshTable API into python interface for Spark-SQL. ## How was this patch

spark git commit: [SPARK-15954][SQL] Disable loading test tables in Python tests

Repository: spark Updated Branches: refs/heads/master 4a981dc87 -> 38f4d6f44 [SPARK-15954][SQL] Disable loading test tables in Python tests ## What changes were proposed in this pull request? This patch introduces a flag to disable loading test tables in TestHiveSparkSession and disables

spark git commit: [SPARK-15643][DOC][ML] Add breaking changes to ML migration guide

Repository: spark Updated Branches: refs/heads/master dab105161 -> 4a981dc87 [SPARK-15643][DOC][ML] Add breaking changes to ML migration guide This PR adds the breaking changes from [SPARK-14810](https://issues.apache.org/jira/browse/SPARK-14810) to the migration guide. ## How was this

spark git commit: [SPARK-15643][DOC][ML] Add breaking changes to ML migration guide

Repository: spark Updated Branches: refs/heads/branch-2.0 d3027c45f -> 79c96c999 [SPARK-15643][DOC][ML] Add breaking changes to ML migration guide This PR adds the breaking changes from [SPARK-14810](https://issues.apache.org/jira/browse/SPARK-14810) to the migration guide. ## How was this

spark git commit: [SPARK-16328][ML][MLLIB][PYSPARK] Add 'asML' and 'fromML' conversion methods to PySpark linalg

Repository: spark Updated Branches: refs/heads/branch-2.0 17c7522c8 -> d3027c45f [SPARK-16328][ML][MLLIB][PYSPARK] Add 'asML' and 'fromML' conversion methods to PySpark linalg The move to `ml.linalg` created `asML`/`fromML` utility methods in Scala/Java for converting between

spark git commit: [SPARK-16328][ML][MLLIB][PYSPARK] Add 'asML' and 'fromML' conversion methods to PySpark linalg

Repository: spark Updated Branches: refs/heads/master 85f2303ec -> dab105161 [SPARK-16328][ML][MLLIB][PYSPARK] Add 'asML' and 'fromML' conversion methods to PySpark linalg The move to `ml.linalg` created `asML`/`fromML` utility methods in Scala/Java for converting between representations.

spark git commit: [SPARK-16276][SQL] Implement elt SQL function

Repository: spark Updated Branches: refs/heads/master 3d75a5b2a -> 85f2303ec [SPARK-16276][SQL] Implement elt SQL function ## What changes were proposed in this pull request? This patch implements the elt function, as it is implemented in Hive. ## How was this patch tested? Added expression

spark git commit: [SPARK-16313][SQL] Spark should not silently drop exceptions in file listing

Repository: spark Updated Branches: refs/heads/branch-2.0 4dc7d377f -> 17c7522c8 [SPARK-16313][SQL] Spark should not silently drop exceptions in file listing ## What changes were proposed in this pull request? Spark silently drops exceptions during file listing. This is a very bad behavior

spark git commit: [SPARK-16313][SQL] Spark should not silently drop exceptions in file listing

Repository: spark Updated Branches: refs/heads/master fb41670c9 -> 3d75a5b2a [SPARK-16313][SQL] Spark should not silently drop exceptions in file listing ## What changes were proposed in this pull request? Spark silently drops exceptions during file listing. This is a very bad behavior

spark git commit: [SPARK-16336][SQL] Suggest doing table refresh upon FileNotFoundException

Repository: spark Updated Branches: refs/heads/branch-2.0 03008e049 -> 4dc7d377f [SPARK-16336][SQL] Suggest doing table refresh upon FileNotFoundException ## What changes were proposed in this pull request? This patch appends a message to suggest users running refresh table or reloading data

spark git commit: [SPARK-16336][SQL] Suggest doing table refresh upon FileNotFoundException

Repository: spark Updated Branches: refs/heads/master 5d00a7bc1 -> fb41670c9 [SPARK-16336][SQL] Suggest doing table refresh upon FileNotFoundException ## What changes were proposed in this pull request? This patch appends a message to suggest users running refresh table or reloading data

spark git commit: [SPARK-16256][DOCS] Fix window operation diagram

Repository: spark Updated Branches: refs/heads/branch-2.0 f17ffef38 -> 03008e049 [SPARK-16256][DOCS] Fix window operation diagram Author: Tathagata Das Closes #14001 from tdas/SPARK-16256-2. (cherry picked from commit 5d00a7bc19ddeb1b5247733b55095a03ee7b1a30)

spark git commit: [SPARK-16256][DOCS] Fix window operation diagram

Repository: spark Updated Branches: refs/heads/master c62263340 -> 5d00a7bc1 [SPARK-16256][DOCS] Fix window operation diagram Author: Tathagata Das Closes #14001 from tdas/SPARK-16256-2. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit:

spark git commit: [SPARK-16212][STREAMING][KAFKA] code cleanup from review feedback

Repository: spark Updated Branches: refs/heads/master 46395db80 -> c62263340 [SPARK-16212][STREAMING][KAFKA] code cleanup from review feedback ## What changes were proposed in this pull request? code cleanup in kafka-0-8 to match suggested changes for kafka-0-10 branch ## How was this patch

spark git commit: [SPARK-16289][SQL] Implement posexplode table generating function

Repository: spark Updated Branches: refs/heads/master fdf9f94f8 -> 46395db80 [SPARK-16289][SQL] Implement posexplode table generating function ## What changes were proposed in this pull request? This PR implements `posexplode` table generating function. Currently, master branch raises the

spark git commit: [SPARK-15865][CORE] Blacklist should not result in job hanging with less than 4 executors

Repository: spark Updated Branches: refs/heads/master 07f46afc7 -> fdf9f94f8 [SPARK-15865][CORE] Blacklist should not result in job hanging with less than 4 executors ## What changes were proposed in this pull request? Before this change, when you turn on blacklisting with

spark git commit: [SPARK-13850] Force the sorter to Spill when number of elements in th…

Repository: spark Updated Branches: refs/heads/branch-2.0 98056a1f8 -> f17ffef38 [SPARK-13850] Force the sorter to Spill when number of elements in th… Force the sorter to Spill when number of elements in the pointer array reach a certain size. This is to workaround the issue of timSort

spark git commit: [SPARK-13850] Force the sorter to Spill when number of elements in th…

Repository: spark Updated Branches: refs/heads/master 5344bade8 -> 07f46afc7 [SPARK-13850] Force the sorter to Spill when number of elements in th… ## What changes were proposed in this pull request? Force the sorter to Spill when number of elements in the pointer array reach a certain

spark git commit: [SPARK-15820][PYSPARK][SQL] Add Catalog.refreshTable into python API

Repository: spark Updated Branches: refs/heads/master 5320adc86 -> 5344bade8 [SPARK-15820][PYSPARK][SQL] Add Catalog.refreshTable into python API ## What changes were proposed in this pull request? Add Catalog.refreshTable API into python interface for Spark-SQL. ## How was this patch

spark git commit: [BUILD] Fix version in poms related to kafka-0-10

Repository: spark Updated Branches: refs/heads/branch-2.0 56207fc3b -> 98056a1f8 [BUILD] Fix version in poms related to kafka-0-10 self explanatory Author: Tathagata Das Closes #13994 from tdas/SPARK-12177-1. Project:

spark git commit: [SPARK-16071][SQL] Checks size limit when doubling the array size in BufferHolder

Repository: spark Updated Branches: refs/heads/branch-2.0 6a4f4c1d7 -> 56207fc3b [SPARK-16071][SQL] Checks size limit when doubling the array size in BufferHolder ## What changes were proposed in this pull request? This PR Checks the size limit when doubling the array size in BufferHolder

spark git commit: [SPARK-16071][SQL] Checks size limit when doubling the array size in BufferHolder

Repository: spark Updated Branches: refs/heads/master de8ab313e -> 5320adc86 [SPARK-16071][SQL] Checks size limit when doubling the array size in BufferHolder ## What changes were proposed in this pull request? This PR Checks the size limit when doubling the array size in BufferHolder to

spark git commit: [SPARK-12177][TEST] Removed test to avoid compilation issue in scala 2.10

Repository: spark Updated Branches: refs/heads/branch-2.0 1d274455c -> 6a4f4c1d7 [SPARK-12177][TEST] Removed test to avoid compilation issue in scala 2.10 ## What changes were proposed in this pull request? The commented lines failed scala 2.10 build. This is because of change in behavior

spark git commit: [SPARK-12177][TEST] Removed test to avoid compilation issue in scala 2.10

Repository: spark Updated Branches: refs/heads/master b30a2dc7c -> de8ab313e [SPARK-12177][TEST] Removed test to avoid compilation issue in scala 2.10 ## What changes were proposed in this pull request? The commented lines failed scala 2.10 build. This is because of change in behavior of

spark git commit: [SPARK-16241][ML] model loading backward compatibility for ml NaiveBayes

Repository: spark Updated Branches: refs/heads/master 2c3d96134 -> b30a2dc7c [SPARK-16241][ML] model loading backward compatibility for ml NaiveBayes ## What changes were proposed in this pull request? model loading backward compatibility for ml NaiveBayes ## How was this patch tested?

spark git commit: [SPARK-16241][ML] model loading backward compatibility for ml NaiveBayes

Repository: spark Updated Branches: refs/heads/branch-2.0 c8a7c2305 -> 1d274455c [SPARK-16241][ML] model loading backward compatibility for ml NaiveBayes ## What changes were proposed in this pull request? model loading backward compatibility for ml NaiveBayes ## How was this patch tested?

spark git commit: [SPARK-16257][BUILD] Update spark_ec2.py to support Spark 1.6.2 and 1.6.3.

Repository: spark Updated Branches: refs/heads/branch-1.6 1ac830aca -> ccc7fa357 [SPARK-16257][BUILD] Update spark_ec2.py to support Spark 1.6.2 and 1.6.3. ## What changes were proposed in this pull request? - Adds 1.6.2 and 1.6.3 as supported Spark versions within the bundled spark-ec2

spark git commit: [SPARK-16256][DOCS] Minor fixes on the Structured Streaming Programming Guide

Repository: spark Updated Branches: refs/heads/branch-2.0 3134f116a -> c8a7c2305 [SPARK-16256][DOCS] Minor fixes on the Structured Streaming Programming Guide Author: Tathagata Das Closes #13978 from tdas/SPARK-16256-1. (cherry picked from commit

spark git commit: [SPARK-16256][DOCS] Minor fixes on the Structured Streaming Programming Guide

Repository: spark Updated Branches: refs/heads/master dedbceec1 -> 2c3d96134 [SPARK-16256][DOCS] Minor fixes on the Structured Streaming Programming Guide Author: Tathagata Das Closes #13978 from tdas/SPARK-16256-1. Project:

[1/2] spark git commit: [SPARK-12177][STREAMING][KAFKA] Update KafkaDStreams to new Kafka 0.10 Consumer API

Repository: spark Updated Branches: refs/heads/branch-2.0 a54852350 -> 3134f116a http://git-wip-us.apache.org/repos/asf/spark/blob/3134f116/external/kafka-0-10/src/test/java/org/apache/spark/streaming/kafka010/JavaConsumerStrategySuite.java

[2/2] spark git commit: [SPARK-12177][STREAMING][KAFKA] Update KafkaDStreams to new Kafka 0.10 Consumer API

[SPARK-12177][STREAMING][KAFKA] Update KafkaDStreams to new Kafka 0.10 Consumer API ## What changes were proposed in this pull request? New Kafka consumer api for the released 0.10 version of Kafka ## How was this patch tested? Unit tests, manual tests Author: cody koeninger

[1/2] spark git commit: [SPARK-12177][STREAMING][KAFKA] Update KafkaDStreams to new Kafka 0.10 Consumer API

Repository: spark Updated Branches: refs/heads/master bde1d6a61 -> dedbceec1 http://git-wip-us.apache.org/repos/asf/spark/blob/dedbceec/external/kafka-0-10/src/test/java/org/apache/spark/streaming/kafka010/JavaConsumerStrategySuite.java

[2/2] spark git commit: [SPARK-12177][STREAMING][KAFKA] Update KafkaDStreams to new Kafka 0.10 Consumer API

[SPARK-12177][STREAMING][KAFKA] Update KafkaDStreams to new Kafka 0.10 Consumer API ## What changes were proposed in this pull request? New Kafka consumer api for the released 0.10 version of Kafka ## How was this patch tested? Unit tests, manual tests Author: cody koeninger