spark git commit: [SPARK-14893][SQL] Re-enable HiveSparkSubmitSuite SPARK-8489 test after HiveContext is removed

Repository: spark Updated Branches: refs/heads/branch-2.0 80a4bfa4d -> 1064a3303 [SPARK-14893][SQL] Re-enable HiveSparkSubmitSuite SPARK-8489 test after HiveContext is removed ## What changes were proposed in this pull request? Enable the test that was disabled when HiveContext was removed.

spark git commit: [SPARK-14893][SQL] Re-enable HiveSparkSubmitSuite SPARK-8489 test after HiveContext is removed

Repository: spark Updated Branches: refs/heads/master 08db49126 -> 02c07e899 [SPARK-14893][SQL] Re-enable HiveSparkSubmitSuite SPARK-8489 test after HiveContext is removed ## What changes were proposed in this pull request? Enable the test that was disabled when HiveContext was removed. ##

spark git commit: [SPARK-9926] Parallelize partition logic in UnionRDD.

Repository: spark Updated Branches: refs/heads/branch-2.0 19a14e841 -> 80a4bfa4d [SPARK-9926] Parallelize partition logic in UnionRDD. This patch has the new logic from #8512 that uses a parallel collection to compute partitions in UnionRDD. The rest of #8512 added an alternative code path

spark git commit: [SPARK-9926] Parallelize partition logic in UnionRDD.

Repository: spark Updated Branches: refs/heads/master 5c47db065 -> 08db49126 [SPARK-9926] Parallelize partition logic in UnionRDD. This patch has the new logic from #8512 that uses a parallel collection to compute partitions in UnionRDD. The rest of #8512 added an alternative code path for

spark git commit: [SPARK-15158][CORE] downgrade shouldRollover message to debug level

Repository: spark Updated Branches: refs/heads/branch-2.0 8b4ab590c -> 19a14e841 [SPARK-15158][CORE] downgrade shouldRollover message to debug level ## What changes were proposed in this pull request? set log level to debug when check shouldRollover ## How was this patch tested? It's tested

spark git commit: [SPARK-15158][CORE] downgrade shouldRollover message to debug level

Repository: spark Updated Branches: refs/heads/master 2c170dd3d -> 5c47db065 [SPARK-15158][CORE] downgrade shouldRollover message to debug level ## What changes were proposed in this pull request? set log level to debug when check shouldRollover ## How was this patch tested? It's tested

[2/2] spark git commit: [SPARK-15134][EXAMPLE] Indent SparkSession builder patterns and update binary_classification_metrics_example.py

[SPARK-15134][EXAMPLE] Indent SparkSession builder patterns and update binary_classification_metrics_example.py ## What changes were proposed in this pull request? This issue addresses the comments in SPARK-15031 and also fix java-linter errors. - Use multiline format in SparkSession builder

[2/2] spark git commit: [SPARK-15134][EXAMPLE] Indent SparkSession builder patterns and update binary_classification_metrics_example.py

[SPARK-15134][EXAMPLE] Indent SparkSession builder patterns and update binary_classification_metrics_example.py ## What changes were proposed in this pull request? This issue addresses the comments in SPARK-15031 and also fix java-linter errors. - Use multiline format in SparkSession builder

[1/2] spark git commit: [SPARK-15134][EXAMPLE] Indent SparkSession builder patterns and update binary_classification_metrics_example.py

Repository: spark Updated Branches: refs/heads/master bb9991dec -> 2c170dd3d http://git-wip-us.apache.org/repos/asf/spark/blob/2c170dd3/examples/src/main/python/ml/vector_indexer_example.py -- diff --git

[1/2] spark git commit: [SPARK-15134][EXAMPLE] Indent SparkSession builder patterns and update binary_classification_metrics_example.py

Repository: spark Updated Branches: refs/heads/branch-2.0 e78b31b72 -> 8b4ab590c http://git-wip-us.apache.org/repos/asf/spark/blob/8b4ab590/examples/src/main/python/ml/vector_indexer_example.py -- diff --git

spark git commit: [SPARK-15135][SQL] Make sure SparkSession thread safe

Repository: spark Updated Branches: refs/heads/branch-2.0 59fa480b6 -> e78b31b72 [SPARK-15135][SQL] Make sure SparkSession thread safe ## What changes were proposed in this pull request? Went through SparkSession and its members and fixed non-thread-safe classes used by SparkSession ## How

spark git commit: [SPARK-15135][SQL] Make sure SparkSession thread safe

Repository: spark Updated Branches: refs/heads/master ed6f3f8a5 -> bb9991dec [SPARK-15135][SQL] Make sure SparkSession thread safe ## What changes were proposed in this pull request? Went through SparkSession and its members and fixed non-thread-safe classes used by SparkSession ## How was

spark git commit: [SPARK-15072][SQL][REPL][EXAMPLES] Remove SparkSession.withHiveSupport

Repository: spark Updated Branches: refs/heads/branch-2.0 fe268ee1e -> 59fa480b6 [SPARK-15072][SQL][REPL][EXAMPLES] Remove SparkSession.withHiveSupport ## What changes were proposed in this pull request? Removing the `withHiveSupport` method of `SparkSession`, instead use `enableHiveSupport`

spark git commit: [SPARK-15072][SQL][REPL][EXAMPLES] Remove SparkSession.withHiveSupport

Repository: spark Updated Branches: refs/heads/master 8cba57a75 -> ed6f3f8a5 [SPARK-15072][SQL][REPL][EXAMPLES] Remove SparkSession.withHiveSupport ## What changes were proposed in this pull request? Removing the `withHiveSupport` method of `SparkSession`, instead use `enableHiveSupport` ##

spark git commit: [SPARK-14124][SQL][FOLLOWUP] Implement Database-related DDL Commands

Repository: spark Updated Branches: refs/heads/branch-2.0 b063d9b71 -> fe268ee1e [SPARK-14124][SQL][FOLLOWUP] Implement Database-related DDL Commands What changes were proposed in this pull request? First, a few test cases failed in mac OS X because the property value of

spark git commit: [SPARK-14124][SQL][FOLLOWUP] Implement Database-related DDL Commands

Repository: spark Updated Branches: refs/heads/master 63db2bd28 -> 8cba57a75 [SPARK-14124][SQL][FOLLOWUP] Implement Database-related DDL Commands What changes were proposed in this pull request? First, a few test cases failed in mac OS X because the property value of `java.io.tmpdir`

spark git commit: [MINOR][BUILD] Adds spark-warehouse/ to .gitignore

Repository: spark Updated Branches: refs/heads/branch-2.0 c2b100e50 -> b063d9b71 [MINOR][BUILD] Adds spark-warehouse/ to .gitignore ## What changes were proposed in this pull request? Adds spark-warehouse/ to `.gitignore`. ## How was this patch tested? N/A Author: Cheng Lian

spark git commit: [MINOR][BUILD] Adds spark-warehouse/ to .gitignore

Repository: spark Updated Branches: refs/heads/master 6fcc9 -> 63db2bd28 [MINOR][BUILD] Adds spark-warehouse/ to .gitignore ## What changes were proposed in this pull request? Adds spark-warehouse/ to `.gitignore`. ## How was this patch tested? N/A Author: Cheng Lian

spark git commit: [SPARK-14896][SQL] Deprecate HiveContext in python

; Closes #12917 from andrewor14/deprecate-hive-context-python. (cherry picked from commit fa79d346e1a79ceda6ccd20e74eb850e769556ea) Signed-off-by: Andrew Or <and...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/a

spark git commit: [SPARK-14896][SQL] Deprecate HiveContext in python

; Closes #12917 from andrewor14/deprecate-hive-context-python. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/fa79d346 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/fa79d346 Diff: http://git-wip-us.apache.org/repos/

spark git commit: [MINOR][SQL] Fix typo in DataFrameReader csv documentation

Repository: spark Updated Branches: refs/heads/branch-2.0 701c66729 -> aca46ecf8 [MINOR][SQL] Fix typo in DataFrameReader csv documentation ## What changes were proposed in this pull request? Typo fix ## How was this patch tested? No tests My apologies for the tiny PR, but I stumbled across

spark git commit: [MINOR][SQL] Fix typo in DataFrameReader csv documentation

Repository: spark Updated Branches: refs/heads/master a432a2b86 -> b28137764 [MINOR][SQL] Fix typo in DataFrameReader csv documentation ## What changes were proposed in this pull request? Typo fix ## How was this patch tested? No tests My apologies for the tiny PR, but I stumbled across

spark git commit: [SPARK-15116] In REPL we should create SparkSession first and get SparkContext from it

Repository: spark Updated Branches: refs/heads/master eb019af9a -> a432a2b86 [SPARK-15116] In REPL we should create SparkSession first and get SparkContext from it ## What changes were proposed in this pull request? see https://github.com/apache/spark/pull/12873#discussion_r61993910. The

spark git commit: [SPARK-13001][CORE][MESOS] Prevent getting offers when reached max cores

Repository: spark Updated Branches: refs/heads/branch-2.0 23789e358 -> 1e7d9bfb5 [SPARK-13001][CORE][MESOS] Prevent getting offers when reached max cores Similar to https://github.com/apache/spark/pull/8639 This change rejects offers for 120s when reached `spark.cores.max` in coarse-grained

spark git commit: [SPARK-13001][CORE][MESOS] Prevent getting offers when reached max cores

Repository: spark Updated Branches: refs/heads/master cdce4e62a -> eb019af9a [SPARK-13001][CORE][MESOS] Prevent getting offers when reached max cores Similar to https://github.com/apache/spark/pull/8639 This change rejects offers for 120s when reached `spark.cores.max` in coarse-grained

[3/4] spark git commit: [SPARK-15031][EXAMPLE] Use SparkSession in Scala/Python/Java example.

http://git-wip-us.apache.org/repos/asf/spark/blob/23789e35/examples/src/main/java/org/apache/spark/examples/ml/JavaPolynomialExpansionExample.java -- diff --git

[2/4] spark git commit: [SPARK-15031][EXAMPLE] Use SparkSession in Scala/Python/Java example.

http://git-wip-us.apache.org/repos/asf/spark/blob/23789e35/examples/src/main/python/ml/naive_bayes_example.py -- diff --git a/examples/src/main/python/ml/naive_bayes_example.py b/examples/src/main/python/ml/naive_bayes_example.py

[4/4] spark git commit: [SPARK-15031][EXAMPLE] Use SparkSession in Scala/Python/Java example.

[SPARK-15031][EXAMPLE] Use SparkSession in Scala/Python/Java example. ## What changes were proposed in this pull request? This PR aims to update Scala/Python/Java examples by replacing `SQLContext` with newly added `SparkSession`. - Use **SparkSession Builder Pattern** in 154(Scala 55, Java

[1/4] spark git commit: [SPARK-15031][EXAMPLE] Use SparkSession in Scala/Python/Java example.

Repository: spark Updated Branches: refs/heads/branch-2.0 c0715f33b -> 23789e358 http://git-wip-us.apache.org/repos/asf/spark/blob/23789e35/examples/src/main/scala/org/apache/spark/examples/ml/NGramExample.scala -- diff --git

[4/4] spark git commit: [SPARK-15031][EXAMPLE] Use SparkSession in Scala/Python/Java example.

[SPARK-15031][EXAMPLE] Use SparkSession in Scala/Python/Java example. ## What changes were proposed in this pull request? This PR aims to update Scala/Python/Java examples by replacing `SQLContext` with newly added `SparkSession`. - Use **SparkSession Builder Pattern** in 154(Scala 55, Java

[2/4] spark git commit: [SPARK-15031][EXAMPLE] Use SparkSession in Scala/Python/Java example.

http://git-wip-us.apache.org/repos/asf/spark/blob/cdce4e62/examples/src/main/python/ml/naive_bayes_example.py -- diff --git a/examples/src/main/python/ml/naive_bayes_example.py b/examples/src/main/python/ml/naive_bayes_example.py

[3/4] spark git commit: [SPARK-15031][EXAMPLE] Use SparkSession in Scala/Python/Java example.

http://git-wip-us.apache.org/repos/asf/spark/blob/cdce4e62/examples/src/main/java/org/apache/spark/examples/ml/JavaPolynomialExpansionExample.java -- diff --git

[1/4] spark git commit: [SPARK-15031][EXAMPLE] Use SparkSession in Scala/Python/Java example.

Repository: spark Updated Branches: refs/heads/master cf2e9da61 -> cdce4e62a http://git-wip-us.apache.org/repos/asf/spark/blob/cdce4e62/examples/src/main/scala/org/apache/spark/examples/ml/NGramExample.scala -- diff --git

spark git commit: [SPARK-12299][CORE] Remove history serving functionality from Master

Repository: spark Updated Branches: refs/heads/master 0c00391f7 -> cf2e9da61 [SPARK-12299][CORE] Remove history serving functionality from Master Remove history server functionality from standalone Master. Previously, the Master process rebuilt a SparkUI once the application was completed

spark git commit: [SPARK-12299][CORE] Remove history serving functionality from Master

Repository: spark Updated Branches: refs/heads/branch-2.0 eeb18f6d7 -> c0715f33b [SPARK-12299][CORE] Remove history serving functionality from Master Remove history server functionality from standalone Master. Previously, the Master process rebuilt a SparkUI once the application was

spark git commit: [SPARK-15121] Improve logging of external shuffle handler

Repository: spark Updated Branches: refs/heads/branch-2.0 45862f6c9 -> eeb18f6d7 [SPARK-15121] Improve logging of external shuffle handler ## What changes were proposed in this pull request? Add more informative logging in the external shuffle service to aid in debugging who is connecting

spark git commit: [SPARK-15121] Improve logging of external shuffle handler

Repository: spark Updated Branches: refs/heads/master 6ae9fc00e -> 0c00391f7 [SPARK-15121] Improve logging of external shuffle handler ## What changes were proposed in this pull request? Add more informative logging in the external shuffle service to aid in debugging who is connecting to

spark git commit: [SPARK-15126][SQL] RuntimeConfig.set should return Unit

Repository: spark Updated Branches: refs/heads/branch-2.0 e868a15a7 -> 45862f6c9 [SPARK-15126][SQL] RuntimeConfig.set should return Unit ## What changes were proposed in this pull request? Currently we return RuntimeConfig itself to facilitate chaining. However, it makes the output in

spark git commit: [SPARK-15126][SQL] RuntimeConfig.set should return Unit

Repository: spark Updated Branches: refs/heads/master 0fd3a4748 -> 6ae9fc00e [SPARK-15126][SQL] RuntimeConfig.set should return Unit ## What changes were proposed in this pull request? Currently we return RuntimeConfig itself to facilitate chaining. However, it makes the output in

spark git commit: [SPARK-14414][SQL] Make DDL exceptions more consistent

et al. Author: Andrew Or <and...@databricks.com> Closes #12853 from andrewor14/make-exceptions-consistent. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/6ba17cd1 Tree: http://git-wip-us.apache.org/repos/asf/s

spark git commit: [SPARK-14414][SQL] Make DDL exceptions more consistent

ite` et al. Author: Andrew Or <and...@databricks.com> Closes #12853 from andrewor14/make-exceptions-consistent. (cherry picked from commit 6ba17cd147277a20a7fbb244c040e694de486c36) Signed-off-by: Andrew Or <and...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/sp

spark git commit: [SPARK-15097][SQL] make Dataset.sqlContext a stable identifier for imports

Repository: spark Updated Branches: refs/heads/branch-2.0 5e15615d1 -> 95d359abd [SPARK-15097][SQL] make Dataset.sqlContext a stable identifier for imports ## What changes were proposed in this pull request? Make Dataset.sqlContext a lazy val so that its a stable identifier and can be used

spark git commit: [SPARK-15097][SQL] make Dataset.sqlContext a stable identifier for imports

Repository: spark Updated Branches: refs/heads/master 0903a185c -> 9e4928b7e [SPARK-15097][SQL] make Dataset.sqlContext a stable identifier for imports ## What changes were proposed in this pull request? Make Dataset.sqlContext a lazy val so that its a stable identifier and can be used for

spark git commit: [SPARK-15084][PYTHON][SQL] Use builder pattern to create SparkSession in PySpark.

Repository: spark Updated Branches: refs/heads/branch-2.0 4c7f5a74d -> 5e15615d1 [SPARK-15084][PYTHON][SQL] Use builder pattern to create SparkSession in PySpark. ## What changes were proposed in this pull request? This is a python port of corresponding Scala builder pattern code. `sql.py`

spark git commit: [SPARK-15084][PYTHON][SQL] Use builder pattern to create SparkSession in PySpark.

Repository: spark Updated Branches: refs/heads/master c1839c991 -> 0903a185c [SPARK-15084][PYTHON][SQL] Use builder pattern to create SparkSession in PySpark. ## What changes were proposed in this pull request? This is a python port of corresponding Scala builder pattern code. `sql.py` is

spark git commit: [SPARK-14645][MESOS] Fix python running on cluster mode mesos to have non local uris

Repository: spark Updated Branches: refs/heads/master a8d56f538 -> c1839c991 [SPARK-14645][MESOS] Fix python running on cluster mode mesos to have non local uris ## What changes were proposed in this pull request? Fix SparkSubmit to allow non-local python uris ## How was this patch tested?

spark git commit: [SPARK-14645][MESOS] Fix python running on cluster mode mesos to have non local uris

Repository: spark Updated Branches: refs/heads/branch-2.0 5625b037a -> 4c7f5a74d [SPARK-14645][MESOS] Fix python running on cluster mode mesos to have non local uris ## What changes were proposed in this pull request? Fix SparkSubmit to allow non-local python uris ## How was this patch

spark git commit: [SPARK-14422][SQL] Improve handling of optional configs in SQLConf

Repository: spark Updated Branches: refs/heads/branch-2.0 0d16b7f3a -> 5625b037a [SPARK-14422][SQL] Improve handling of optional configs in SQLConf ## What changes were proposed in this pull request? Create a new API for handling Optional Configs in SQLConf. Right now `getConf` for

spark git commit: [MINOR][DOC] Fixed some python snippets in mllib data types documentation.

Repository: spark Updated Branches: refs/heads/branch-2.0 c212307b9 -> 0d16b7f3a [MINOR][DOC] Fixed some python snippets in mllib data types documentation. ## What changes were proposed in this pull request? Some python snippets is using scala imports and comments. ## How was this patch

spark git commit: [MINOR][DOC] Fixed some python snippets in mllib data types documentation.

Repository: spark Updated Branches: refs/heads/master dbacd9998 -> c4e0fde87 [MINOR][DOC] Fixed some python snippets in mllib data types documentation. ## What changes were proposed in this pull request? Some python snippets is using scala imports and comments. ## How was this patch tested?

spark git commit: [SPARK-15019][SQL] Propagate all Spark Confs to HiveConf created in HiveClientImpl

Repository: spark Updated Branches: refs/heads/master dcfaeadea -> b33d6b728 [SPARK-15019][SQL] Propagate all Spark Confs to HiveConf created in HiveClientImpl ## What changes were proposed in this pull request? This PR makes two changes: 1. We will propagate Spark Confs to HiveConf created

spark git commit: [SPARK-14988][PYTHON] SparkSession API follow-ups

rew Or <and...@databricks.com> Closes #12784 from andrewor14/python-followup. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/d33e3d57 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/d33e3d57 Diff: http://git-wip-us.apache.org/repos/

spark git commit: [MINOR] Follow-up to #12625

rew Or <and...@databricks.com> Closes #12686 from andrewor14/visibility. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/2a3d39f4 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/2a3d39f4 Diff: http://git-wip-us.a

spark git commit: [SPARK-14721][SQL] Remove HiveContext (part 2)

ext in the REPL (fixed by #12589) ## How was this patch tested? No change in functionality. Author: Andrew Or <and...@databricks.com> Closes #12585 from andrewor14/delete-hive-context. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/

spark git commit: [MINOR] Comment whitespace changes in #12553

Repository: spark Updated Branches: refs/heads/master a2e8d4fdd -> ef6be7bed [MINOR] Comment whitespace changes in #12553 Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/ef6be7be Tree:

spark git commit: Revert "[SPARK-14647][SQL] Group SQLContext/HiveContext state into SharedState"

Repository: spark Updated Branches: refs/heads/master 699a4dfd8 -> 7de06a646 Revert "[SPARK-14647][SQL] Group SQLContext/HiveContext state into SharedState" This reverts commit 5cefecc95a5b8418713516802c416cfde5a94a2d. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit:

spark git commit: [SPARK-14592][SQL] Native support for CREATE TABLE LIKE DDL command

Repository: spark Updated Branches: refs/heads/master c971aee40 -> 28efdd3fd [SPARK-14592][SQL] Native support for CREATE TABLE LIKE DDL command ## What changes were proposed in this pull request? JIRA: https://issues.apache.org/jira/browse/SPARK-14592 This patch adds native support for DDL

spark git commit: [SPARK-14499][SQL][TEST] Drop Partition Does Not Delete Data of External Tables

ble will not delete data. cc yhuai andrewor14 How was this patch tested? N/A Author: gatorsmile <gatorsm...@gmail.com> This patch had conflicts when merged, resolved by Committer: Andrew Or <and...@databricks.com> Closes #12350 from gatorsmile/testDropPartition. Project: http://git-wi

spark git commit: [SPARK-14558][CORE] In ClosureCleaner, clean the outer pointer if it's a REPL line object

Repository: spark Updated Branches: refs/heads/master a46f98d3f -> 1d04c86fc [SPARK-14558][CORE] In ClosureCleaner, clean the outer pointer if it's a REPL line object ## What changes were proposed in this pull request? When we clean a closure, if its outermost parent is not a closure, we

spark git commit: [SPARK-14617] Remove deprecated APIs in TaskMetrics

Repository: spark Updated Branches: refs/heads/master dac40b68d -> a46f98d3f [SPARK-14617] Remove deprecated APIs in TaskMetrics ## What changes were proposed in this pull request? This patch removes some of the deprecated APIs in TaskMetrics. This is part of my bigger effort to simplify

spark git commit: [SPARK-14619] Track internal accumulators (metrics) by stage attempt

Repository: spark Updated Branches: refs/heads/master 9fa43a33b -> dac40b68d [SPARK-14619] Track internal accumulators (metrics) by stage attempt ## What changes were proposed in this pull request? When there are multiple attempts for a stage, we currently only reset internal accumulator

spark git commit: [SPARK-14357][CORE] Properly handle the root cause being a commit denied exception

Repository: spark Updated Branches: refs/heads/branch-1.5 4f9d1f8c7 -> 1e61ff4ca [SPARK-14357][CORE] Properly handle the root cause being a commit denied exception ## What changes were proposed in this pull request? When deciding whether a CommitDeniedException caused a task to fail,

spark git commit: [SPARK-14455][STREAMING] Fix NPE in allocatedExecutors when calling in receiver-less scenario

Repository: spark Updated Branches: refs/heads/master 3fb09afd5 -> 2c95e4e96 [SPARK-14455][STREAMING] Fix NPE in allocatedExecutors when calling in receiver-less scenario ## What changes were proposed in this pull request? When calling `ReceiverTracker#allocatedExecutors` in receiver-less

spark git commit: [SPARK-14506][SQL] HiveClientImpl's toHiveTable misses a table property for external tables

Repository: spark Updated Branches: refs/heads/master aea30a1a9 -> 3fb09afd5 [SPARK-14506][SQL] HiveClientImpl's toHiveTable misses a table property for external tables ## What changes were proposed in this pull request? For an external table's metadata (in Hive's representation), its table

spark git commit: [SPARK-14468] Always enable OutputCommitCoordinator

ugh `spark.hadoop.outputCommitCoordination.enabled`, but they really shouldn't... ## How was this patch tested? `OutputCommitCoordinator*Suite` Author: Andrew Or <and...@databricks.com> Closes #12244 from andrewor14/always-occ. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Com

spark git commit: [SPARK-14468] Always enable OutputCommitCoordinator

ugh `spark.hadoop.outputCommitCoordination.enabled`, but they really shouldn't... ## How was this patch tested? `OutputCommitCoordinator*Suite` Author: Andrew Or <and...@databricks.com> Closes #12244 from andrewor14/always-occ. (cherry picked from commit 3e29e372ff518827bae9dcd26087946

spark git commit: [SPARK-14468] Always enable OutputCommitCoordinator

ugh `spark.hadoop.outputCommitCoordination.enabled`, but they really shouldn't... ## How was this patch tested? `OutputCommitCoordinator*Suite` Author: Andrew Or <and...@databricks.com> Closes #12244 from andrewor14/always-occ. (cherry picked from commit 3e29e372ff518827bae9dcd26087946

spark git commit: [DOCS][MINOR] Remove sentence about Mesos not supporting cluster mode.

Repository: spark Updated Branches: refs/heads/branch-1.6 dca0d9a48 -> 8a94a59f9 [DOCS][MINOR] Remove sentence about Mesos not supporting cluster mode. Docs change to remove the sentence about Mesos not supporting cluster mode. It was not. Author: Michael Gummelt

spark git commit: [SPARK-13112][CORE] Make sure RegisterExecutorResponse arrive before LaunchTask

Repository: spark Updated Branches: refs/heads/master c4bb02abf -> f1def573f [SPARK-13112][CORE] Make sure RegisterExecutorResponse arrive before LaunchTask ## What changes were proposed in this pull request? Send `RegisterExecutorResponse` using `executorRef` in order to make sure

spark git commit: [SPARK-14444][BUILD] Add a new scalastyle `NoScalaDoc` to prevent ScalaDoc-style multiline comments

Repository: spark Updated Branches: refs/heads/master 457e58bef -> d717ae1fd [SPARK-1][BUILD] Add a new scalastyle `NoScalaDoc` to prevent ScalaDoc-style multiline comments ## What changes were proposed in this pull request? According to the [Spark Code Style

spark git commit: [SPARK-14424][BUILD][DOCS] Update the build docs to switch from assembly to package and add a no…

Repository: spark Updated Branches: refs/heads/master 9af5423ec -> 457e58bef [SPARK-14424][BUILD][DOCS] Update the build docs to switch from assembly to package and add a no… ## What changes were proposed in this pull request? Change our build docs & shell scripts to that developers are

spark git commit: [SPARK-12133][STREAMING] Streaming dynamic allocation

Repository: spark Updated Branches: refs/heads/master de4792605 -> 9af5423ec [SPARK-12133][STREAMING] Streaming dynamic allocation ## What changes were proposed in this pull request? Added a new Executor Allocation Manager for the Streaming scheduler for doing Streaming Dynamic Allocation.

spark git commit: [SPARK-14391][LAUNCHER] Increase test timeouts.

Repository: spark Updated Branches: refs/heads/master 5a4b11a90 -> de4792605 [SPARK-14391][LAUNCHER] Increase test timeouts. Most of the time tests should still pass really quickly; it's just when machines are overloaded that the tests may take a little time, but that's still preferable over

spark git commit: [SPARK-14383][SQL] missing "|" in the g4 file

Repository: spark Updated Branches: refs/heads/master 5abd02c02 -> 3c8d88216 [SPARK-14383][SQL] missing "|" in the g4 file ## What changes were proposed in this pull request? A very trivial one. It missed "|" between DISTRIBUTE and UNSET. ## How was this patch tested? I do not think it is

spark git commit: [SPARK-14429][SQL] Improve LIKE pattern in "SHOW TABLES / FUNCTIONS LIKE " DDL

some changes to the utility method to make sure we will get the same result as Hive does. A new method was created in StringUtil and test cases were added. andrewor14 Author: bomeng <bm...@us.ibm.com> Closes #12206 from bomeng/SPARK-14429. Project: http://git-wip-us.apache.org/repos/asf/spar

spark git commit: [SPARK-14426][SQL] Merge PerserUtils and ParseUtils

Repository: spark Updated Branches: refs/heads/master 90ca18448 -> 10494feae [SPARK-14426][SQL] Merge PerserUtils and ParseUtils ## What changes were proposed in this pull request? We have ParserUtils and ParseUtils which are both utility collections for use during the parsing process.

spark git commit: [SPARK-14252] Executors do not try to download remote cached blocks

Repository: spark Updated Branches: refs/heads/master 68be5b9e8 -> 78c1076d0 [SPARK-14252] Executors do not try to download remote cached blocks ## What changes were proposed in this pull request? As mentioned in the ticket this was because one get path in the refactored `BlockManager` did

spark git commit: [SPARK-14396][SQL] Throw Exceptions for DDLs of Partitioned Views

Repository: spark Updated Branches: refs/heads/master 48467f4eb -> 68be5b9e8 [SPARK-14396][SQL] Throw Exceptions for DDLs of Partitioned Views What changes were proposed in this pull request? Because the concept of partitioning is associated with physical tables, we disable all the

spark git commit: [SPARK-14416][CORE] Add thread-safe comments for CoarseGrainedSchedulerBackend's fields

Repository: spark Updated Branches: refs/heads/master adbfdb878 -> 48467f4eb [SPARK-14416][CORE] Add thread-safe comments for CoarseGrainedSchedulerBackend's fields ## What changes were proposed in this pull request? While I was reviewing #12078, I found most of

spark git commit: [SPARK-14243][CORE][BACKPORT-1.6] update task metrics when removing blocks

Repository: spark Updated Branches: refs/heads/branch-1.6 285cb9c66 -> cfe9f02ca [SPARK-14243][CORE][BACKPORT-1.6] update task metrics when removing blocks ## What changes were proposed in this pull request? This patch try to update the `updatedBlockStatuses ` when removing blocks, making

[1/2] spark git commit: [SPARK-14123][SPARK-14384][SQL] Handle CreateFunction/DropFunction

Repository: spark Updated Branches: refs/heads/master bc36df127 -> 72544d6f2 http://git-wip-us.apache.org/repos/asf/spark/blob/72544d6f/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala -- diff

[2/2] spark git commit: [SPARK-14123][SPARK-14384][SQL] Handle CreateFunction/DropFunction

[SPARK-14123][SPARK-14384][SQL] Handle CreateFunction/DropFunction ## What changes were proposed in this pull request? This PR implements CreateFunction and DropFunction commands. Besides implementing these two commands, we also change how to manage functions. Here are the main changes. *

spark git commit: [SPARK-11327][MESOS] Backport dispatcher does not respect all args f…

Repository: spark Updated Branches: refs/heads/branch-1.6 f12f11e57 -> 91530b09e [SPARK-11327][MESOS] Backport dispatcher does not respect all args f… Backport for https://github.com/apache/spark/pull/10370 andrewor14 Author: Jo Voordeckers <jo.voordeck...@gmail.com> Closes #1

spark git commit: [SPARK-14358] Change SparkListener from a trait to an abstract class

Repository: spark Updated Branches: refs/heads/master 27dad6f65 -> 714390470 [SPARK-14358] Change SparkListener from a trait to an abstract class ## What changes were proposed in this pull request? Scala traits are difficult to maintain binary compatibility on, and as a result we had to

spark git commit: [SPARK-14364][SPARK] HeartbeatReceiver object should be private

Repository: spark Updated Branches: refs/heads/master 5743c6476 -> 27dad6f65 [SPARK-14364][SPARK] HeartbeatReceiver object should be private ## What changes were proposed in this pull request? It's a mistake that HeartbeatReceiver object was made public in Spark 1.x. ## How was this patch

spark git commit: [SPARK-14304][SQL][TESTS] Fix tests that don't create temp files in the `java.io.tmpdir` folder

Repository: spark Updated Branches: refs/heads/master 3cfbeb70b -> e78540282 [SPARK-14304][SQL][TESTS] Fix tests that don't create temp files in the `java.io.tmpdir` folder ## What changes were proposed in this pull request? If I press `CTRL-C` when running these tests, the temp files will

spark git commit: [SPARK-11327][MESOS] Dispatcher does not respect all args from the Submit request

Repository: spark Updated Branches: refs/heads/master 0abee534f -> 10508f36a [SPARK-11327][MESOS] Dispatcher does not respect all args from the Submit request Supersedes https://github.com/apache/spark/pull/9752 Author: Jo Voordeckers Author: Iulian Dragos

spark git commit: [SPARK-14069][SQL] Improve SparkStatusTracker to also track executor information

Repository: spark Updated Branches: refs/heads/master 4d93b653f -> 0abee534f [SPARK-14069][SQL] Improve SparkStatusTracker to also track executor information ## What changes were proposed in this pull request? Track executor information like host and port, cache size, running tasks. TODO:

spark git commit: [Docs] Update monitoring.md to accurately describe the history server

Repository: spark Updated Branches: refs/heads/master 8a333d2da -> 4d93b653f [Docs] Update monitoring.md to accurately describe the history server It looks like the docs were recently updated to reflect the History Server's support for incomplete applications, but they still had wording that

spark git commit: [SPARK-14243][CORE] update task metrics when removing blocks

Repository: spark Updated Branches: refs/heads/master 446c45bd8 -> 8a333d2da [SPARK-14243][CORE] update task metrics when removing blocks ## What changes were proposed in this pull request? This PR try to use `incUpdatedBlockStatuses ` to update the `updatedBlockStatuses ` when removing

spark git commit: [SPARK-14182][SQL] Parse DDL Command: Alter View

Repository: spark Updated Branches: refs/heads/master ac1b8b302 -> 446c45bd8 [SPARK-14182][SQL] Parse DDL Command: Alter View This PR is to provide native parsing support for DDL commands: `Alter View`. Since its AST trees are highly similar to `Alter Table`. Thus, both implementation are

spark git commit: [SPARK-13796] Redirect error message to logWarning

Repository: spark Updated Branches: refs/heads/master 358692932 -> ac1b8b302 [SPARK-13796] Redirect error message to logWarning ## What changes were proposed in this pull request? Redirect error message to logWarning ## How was this patch tested? Unit tests, manual tests JoshRosen

spark git commit: [SPARK-14124][SQL] Implement Database-related DDL Commands

nds. In the Database-related DDL commands, we will issue an error exception for `ALTER (DATABASE|SCHEMA) database_name SET OWNER [USER|ROLE] user_or_role`. cc yhuai andrewor14 rxin Could you review the changes? Is it in the right direction? Thanks! How was this patch tested? Added a few t

spark git commit: [SPARK-10570][CORE] Add version info to json api

Repository: spark Updated Branches: refs/heads/master 15c0b0006 -> d26c42982 [SPARK-10570][CORE] Add version info to json api Add a new api endpoint `/api/v1/version` to retrieve various version info. This PR only adds support for finding the current spark version, however other version

spark git commit: [SPARK-14232][WEBUI] Fix event timeline display issue when an executor is removed with a multiple line reason.

Repository: spark Updated Branches: refs/heads/branch-1.6 c2ce247ea -> 84ad2544f [SPARK-14232][WEBUI] Fix event timeline display issue when an executor is removed with a multiple line reason. ## What changes were proposed in this pull request? The event timeline doesn't show on job page if

spark git commit: [SPARK-14232][WEBUI] Fix event timeline display issue when an executor is removed with a multiple line reason.

Repository: spark Updated Branches: refs/heads/master d2a819a63 -> 15c0b0006 [SPARK-14232][WEBUI] Fix event timeline display issue when an executor is removed with a multiple line reason. ## What changes were proposed in this pull request? The event timeline doesn't show on job page if an

spark git commit: [SPARK-13845][CORE][BACKPORT-1.6] Using onBlockUpdated to replace onTaskEnd avioding driver OOM

Repository: spark Updated Branches: refs/heads/branch-1.6 a7579444d -> c2ce247ea [SPARK-13845][CORE][BACKPORT-1.6] Using onBlockUpdated to replace onTaskEnd avioding driver OOM ## What changes were proposed in this pull request? We have a streaming job using `FlumePollInputStream` always

spark git commit: [SPARK-13447][YARN][CORE] Clean the stale states for AM failure and restart situation

ean-up work only for dynamic allocation enabled scenario. Here we should also clean the states in `CoarseGrainedSchedulerBackend` for dynamic allocation disabled scenario. Please review, CC andrewor14 lianhuiwang , thanks a lot. ## How was this patch tested? Run the unit test locally, a

spark git commit: [SPARK-13845][CORE] Using onBlockUpdated to replace onTaskEnd avioding driver OOM

Repository: spark Updated Branches: refs/heads/master a916d2a45 -> ad9e3d50f [SPARK-13845][CORE] Using onBlockUpdated to replace onTaskEnd avioding driver OOM ## What changes were proposed in this pull request? We have a streaming job using `FlumePollInputStream` always driver OOM after

spark git commit: [SPARK-14119][SPARK-14120][SPARK-14122][SQL] Throw exception on unsupported DDL commands

his patch tested? `HiveQuerySuite` Author: Andrew Or <and...@databricks.com> Closes #11948 from andrewor14/ddl-role-management. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/a916d2a4 Tree: http://git-wip-us.apache.or

<    1   2   3   4   5   6   7   8   9   10   >