spark git commit: [SPARK-13923][SPARK-14014][SQL] Session catalog follow-ups

ull list of changes in this patch, just trace the commits. ## How was this patch tested? `SessionCatalogSuite` and `CatalogTestCases` Author: Andrew Or <and...@databricks.com> Closes #12006 from andrewor14/session-catalog-followup. Project: http://git-wip-us.apache.org/repos/asf/s

spark git commit: [SPARK-14180][CORE] Fix a deadlock in CoarseGrainedExecutorBackend Shutdown

Repository: spark Updated Branches: refs/heads/master 328c71161 -> 34c0638ee [SPARK-14180][CORE] Fix a deadlock in CoarseGrainedExecutorBackend Shutdown ## What changes were proposed in this pull request? Call `executor.stop` in a new thread to eliminate deadlock. ## How was this patch

spark git commit: [SPARK-14086][SQL] Add DDL commands to ANTLR4 parser

ove them in follow-up PRs. How was this patch tested? The existing DDLCommandSuite. cc rxin andrewor14 yhuai Author: Herman van Hovell <hvanhov...@questtec.nl> Closes #12011 from hvanhovell/SPARK-14086. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http:

[1/3] spark git commit: [SPARK-14014][SQL] Integrate session catalog (attempt #2)

Repository: spark Updated Branches: refs/heads/master 1c70b7650 -> 20ddf5fdd http://git-wip-us.apache.org/repos/asf/spark/blob/20ddf5fd/sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/OrcQuerySuite.scala -- diff --git

[3/3] spark git commit: [SPARK-14014][SQL] Integrate session catalog (attempt #2)

` and `HiveContextSuite`. Author: Andrew Or <and...@databricks.com> Closes #11938 from andrewor14/session-catalog-again. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/20ddf5fd Tree: http://git-wip-us.apache.org/repos/asf/spar

[2/3] spark git commit: [SPARK-14014][SQL] Integrate session catalog (attempt #2)

http://git-wip-us.apache.org/repos/asf/spark/blob/20ddf5fd/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala -- diff --git a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala

[2/2] spark git commit: Revert "[SPARK-14014][SQL] Replace existing catalog with SessionCatalog"

Revert "[SPARK-14014][SQL] Replace existing catalog with SessionCatalog" This reverts commit 5dfc01976bb0d72489620b4f32cc12d620bb6260. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/c44d140c Tree:

[1/2] spark git commit: Revert "[SPARK-14014][SQL] Replace existing catalog with SessionCatalog"

Repository: spark Updated Branches: refs/heads/master cf823bead -> c44d140ca http://git-wip-us.apache.org/repos/asf/spark/blob/c44d140c/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala -- diff --git

spark git commit: [MINOR][COMMON] Fix copy-paste oversight in variable naming

Repository: spark Updated Branches: refs/heads/branch-1.5 9ecd33e96 -> 7a24d94a8 [MINOR][COMMON] Fix copy-paste oversight in variable naming ## What changes were proposed in this pull request? JavaUtils.java has methods to convert time and byte strings for internal use, this change renames

spark git commit: [MINOR][COMMON] Fix copy-paste oversight in variable naming

Repository: spark Updated Branches: refs/heads/master 9f13f0fc1 -> e06493cb7 [MINOR][COMMON] Fix copy-paste oversight in variable naming ## What changes were proposed in this pull request? JavaUtils.java has methods to convert time and byte strings for internal use, this change renames a

spark git commit: [MINOR][DOCS] Added Missing back slashes

Repository: spark Updated Branches: refs/heads/branch-1.6 3519ce9f1 -> 13bd12264 [MINOR][DOCS] Added Missing back slashes ## What changes were proposed in this pull request? When studying spark many users just copy examples on the documentation and paste on their terminals and because of

spark git commit: [MINOR][DOCS] Added Missing back slashes

Repository: spark Updated Branches: refs/heads/master 310981d49 -> 9f13f0fc1 [MINOR][DOCS] Added Missing back slashes ## What changes were proposed in this pull request? When studying spark many users just copy examples on the documentation and paste on their terminals and because of that

spark git commit: [SPARK-12583][MESOS] Mesos shuffle service: Don't delete shuffle files before application has stopped

Repository: spark Updated Branches: refs/heads/master 07cb323e7 -> 310981d49 [SPARK-12583][MESOS] Mesos shuffle service: Don't delete shuffle files before application has stopped ## Problem description: Mesos shuffle service is completely unusable since Spark 1.6.0 . The problem seems to

spark git commit: [SPARK-13833] Guard against race condition when re-caching disk blocks in memory

Repository: spark Updated Branches: refs/heads/master 9a1680c2c -> 9a87afd7d [SPARK-13833] Guard against race condition when re-caching disk blocks in memory When reading data from the DiskStore and attempting to cache it back into the memory store, we should guard against race conditions

spark git commit: [SPARK-13328][CORE] Poor read performance for broadcast variables with dynamic resource allocation

Repository: spark Updated Branches: refs/heads/master eb650a81f -> ff776b2fc [SPARK-13328][CORE] Poor read performance for broadcast variables with dynamic resource allocation When dynamic resource allocation is enabled fetching broadcast variables from removed executors were causing job

spark git commit: [SPARK-13604][CORE] Sync worker's state after registering with master

Repository: spark Updated Branches: refs/heads/master 020ff8cd3 -> 27fe6bacc [SPARK-13604][CORE] Sync worker's state after registering with master ## What changes were proposed in this pull request? Here lists all cases that Master cannot talk with Worker for a while and then network is

[2/2] spark git commit: [SPARK-13696] Remove BlockStore class & simplify interfaces of mem. & disk stores

[SPARK-13696] Remove BlockStore class & simplify interfaces of mem. & disk stores Today, both the MemoryStore and DiskStore implement a common `BlockStore` API, but I feel that this API is inappropriate because it abstracts away important distinctions between the behavior of these two stores.

[1/2] spark git commit: [SPARK-13696] Remove BlockStore class & simplify interfaces of mem. & disk stores

Repository: spark Updated Branches: refs/heads/master 3d2b6f56e -> 81d48532d http://git-wip-us.apache.org/repos/asf/spark/blob/81d48532/core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala -- diff --git

spark git commit: [SPARK-13492][MESOS] Configurable Mesos framework webui URL.

Repository: spark Updated Branches: refs/heads/master 5f7dbdba6 -> a4a0addcc [SPARK-13492][MESOS] Configurable Mesos framework webui URL. ## What changes were proposed in this pull request? Previously the Mesos framework webui URL was being derived only from the Spark UI address leaving no

spark git commit: [MINOR] Fix typo in 'hypot' docstring

Repository: spark Updated Branches: refs/heads/master 238447db5 -> 5f7dbdba6 [MINOR] Fix typo in 'hypot' docstring Minor typo: docstring for pyspark.sql.functions: hypot has extra characters N/A Author: Tristan Reid Closes #11616 from tristanreid/master. Project:

spark git commit: [SPARK-13775] History page sorted by completed time desc by default.

Repository: spark Updated Branches: refs/heads/master 40e067675 -> 238447db5 [SPARK-13775] History page sorted by completed time desc by default. ## What changes were proposed in this pull request? Originally the page is sorted by AppID by default. After tests with users' feedback, we think

spark git commit: [SPARK-13778][CORE] Set the executor state for a worker when removing it

Repository: spark Updated Branches: refs/heads/master 37fcda3e6 -> 40e067675 [SPARK-13778][CORE] Set the executor state for a worker when removing it ## What changes were proposed in this pull request? When a worker is lost, the executors on this worker are also lost. But Master's

spark git commit: [SPARK-13659] Refactor BlockStore put*() APIs to remove returnValues

Repository: spark Updated Branches: refs/heads/master 017cdf2be -> e52e597db [SPARK-13659] Refactor BlockStore put*() APIs to remove returnValues In preparation for larger refactoring, this patch removes the confusing `returnValues` option from the BlockStore put() APIs: returning the value

spark git commit: [SPARK-13689][SQL] Move helper things in CatalystQl to new utils object

rya's changes in #11048. It prefaces the bigger fix for SPARK-13139 to make the diff of that patch smaller. ## How was this patch tested? No change in functionality, so just Jenkins. Author: Andrew Or <and...@databricks.com> Closes #11529 from andrewor14/parser-utils. Project: http://gi

[2/3] spark git commit: [SPARK-13633][SQL] Move things into catalyst.parser package

http://git-wip-us.apache.org/repos/asf/spark/blob/b7d41474/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/CatalystQl.scala -- diff --git

[1/3] spark git commit: [SPARK-13633][SQL] Move things into catalyst.parser package

Repository: spark Updated Branches: refs/heads/master 83302c3bf -> b7d414742 http://git-wip-us.apache.org/repos/asf/spark/blob/b7d41474/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DataTypeParserSuite.scala

[2/2] spark git commit: [SPARK-12817] Add BlockManager.getOrElseUpdate and remove CacheManager

andrewor14 and nongli for review. Note that this changes the locking semantics of a couple of internal BlockManager methods (`doPut()` and `lockNewBlockForWriting`), so please pay attention to the Scaladoc changes and new test cases for those methods. Author: Josh Rosen <joshro...@databricks.com>

spark git commit: [SPARK-13522][CORE] Fix the exit log place for heartbeat

Repository: spark Updated Branches: refs/heads/master 17a253cbf -> 644dbb641 [SPARK-13522][CORE] Fix the exit log place for heartbeat ## What changes were proposed in this pull request? Just fixed the log place introduced by #11401 ## How was this patch tested? unit tests. Author:

spark git commit: [SPARK-13522][CORE] Executor should kill itself when it's unable to heartbeat to driver more than N times

Repository: spark Updated Branches: refs/heads/master bc65f60ef -> 17a253cbf [SPARK-13522][CORE] Executor should kill itself when it's unable to heartbeat to driver more than N times ## What changes were proposed in this pull request? Sometimes, network disconnection event won't be

spark git commit: [SPARK-13519][CORE] Driver should tell Executor to stop itself when cleaning executor's state

Repository: spark Updated Branches: refs/heads/master 1e5fcdf96 -> ad615291f [SPARK-13519][CORE] Driver should tell Executor to stop itself when cleaning executor's state ## What changes were proposed in this pull request? When the driver removes an executor's state, the connection between

[2/2] spark git commit: [SPARK-12757] Add block-level read/write locks to BlockManager

[SPARK-12757] Add block-level read/write locks to BlockManager ## Motivation As a pre-requisite to off-heap caching of blocks, we need a mechanism to prevent pages / blocks from being evicted while they are being read. With on-heap objects, evicting a block while it is being read merely leads

[1/2] spark git commit: [SPARK-12757] Add block-level read/write locks to BlockManager

Repository: spark Updated Branches: refs/heads/master 712995757 -> 633d63a48 http://git-wip-us.apache.org/repos/asf/spark/blob/633d63a4/core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala -- diff --git

spark git commit: [SPARK-13387][MESOS] Add support for SPARK_DAEMON_JAVA_OPTS with MesosClusterDispatcher.

Repository: spark Updated Branches: refs/heads/master f2cfafdfe -> 712995757 [SPARK-13387][MESOS] Add support for SPARK_DAEMON_JAVA_OPTS with MesosClusterDispatcher. ## What changes were proposed in this pull request? Add support for SPARK_DAEMON_JAVA_OPTS with MesosClusterDispatcher. ##

spark git commit: [SPARK-13501] Remove use of Guava Stopwatch

Repository: spark Updated Branches: refs/heads/master 7a6ee8a8f -> f2cfafdfe [SPARK-13501] Remove use of Guava Stopwatch Our nightly doc snapshot builds are failing due to some issue involving the Guava Stopwatch constructor: ``` [error]

spark git commit: [SPARK-12009][YARN] Avoid to re-allocating yarn container while driver want to stop all Executors

Repository: spark Updated Branches: refs/heads/master dc6c5ea4c -> 7a6ee8a8f [SPARK-12009][YARN] Avoid to re-allocating yarn container while driver want to stop all Executors Author: hushan Closes #9992 from suyanNone/tricky. Project:

spark git commit: [SPARK-13298][CORE][UI] Escape "label" to avoid DAG being broken by some special character

Repository: spark Updated Branches: refs/heads/master 33ef3aa7e -> a11b39951 [SPARK-13298][CORE][UI] Escape "label" to avoid DAG being broken by some special character ## What changes were proposed in this pull request? When there are some special characters (e.g., `"`, `\`) in `label`, DAG

spark git commit: [SPARK-13298][CORE][UI] Escape "label" to avoid DAG being broken by some special character

Repository: spark Updated Branches: refs/heads/branch-1.6 699644c69 -> 85e6a2205 [SPARK-13298][CORE][UI] Escape "label" to avoid DAG being broken by some special character ## What changes were proposed in this pull request? When there are some special characters (e.g., `"`, `\`) in `label`,

spark git commit: [SPARK-10749][MESOS] Support multiple roles with mesos cluster mode.

Repository: spark Updated Branches: refs/heads/master 40e6d40fe -> 00461bb91 [SPARK-10749][MESOS] Support multiple roles with mesos cluster mode. Currently the Mesos cluster dispatcher is not using offers from multiple roles correctly, as it simply aggregates all the offers resource values

spark git commit: [SPARK-13371][CORE][STRING] TaskSetManager.dequeueSpeculativeTask compares Option and String directly.

Repository: spark Updated Branches: refs/heads/branch-1.6 66106a660 -> 16f35c4c6 [SPARK-13371][CORE][STRING] TaskSetManager.dequeueSpeculativeTask compares Option and String directly. ## What changes were proposed in this pull request? Fix some comparisons between unequal types that cause

spark git commit: [SPARK-13371][CORE][STRING] TaskSetManager.dequeueSpeculativeTask compares Option and String directly.

Repository: spark Updated Branches: refs/heads/master 892b2dd6d -> 78562535f [SPARK-13371][CORE][STRING] TaskSetManager.dequeueSpeculativeTask compares Option and String directly. ## What changes were proposed in this pull request? Fix some comparisons between unequal types that cause IJ

spark git commit: [SPARK-5095] Fix style in mesos coarse grained scheduler code

Repository: spark Updated Branches: refs/heads/master 42d656814 -> 38bc6018e [SPARK-5095] Fix style in mesos coarse grained scheduler code andrewor14 This addressed your style comments from #10993 Author: Michael Gummelt <mgumm...@mesosphere.io> Closes #11187 from mgummelt/fix_me

spark git commit: [SPARK-5095] remove flaky test

Repository: spark Updated Branches: refs/heads/master 38bc6018e -> 62b1c07e7 [SPARK-5095] remove flaky test Overrode the start() method, which was previously starting a thread causing a race condition. I believe this should fix the flaky test. Author: Michael Gummelt

spark git commit: [SPARK-5095][MESOS] Support launching multiple mesos executors in coarse grained mesos mode.

ved with andrewor14 and pwendell to implement the Mesos scheduler's support of `spark.executor.cores` to be consistent with YARN and Standalone. This PR implements that resolution. This PR implements two high-level features. These two features are co-dependent, so they're implemented both h

spark git commit: [SPARK-13002][MESOS] Send initial request of executors for dyn allocation

Repository: spark Updated Branches: refs/heads/master 66e1383de -> 0bb5b7338 [SPARK-13002][MESOS] Send initial request of executors for dyn allocation Fix for [SPARK-13002](https://issues.apache.org/jira/browse/SPARK-13002) about the initial number of executors when running with dynamic

spark git commit: [SPARK-13214][DOCS] update dynamicAllocation documentation

Repository: spark Updated Branches: refs/heads/master 1ed354a53 -> 66e1383de [SPARK-13214][DOCS] update dynamicAllocation documentation Author: Bill Chambers Closes #11094 from anabranch/dynamic-docs. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit:

spark git commit: [SPARK-13214][DOCS] update dynamicAllocation documentation

Repository: spark Updated Branches: refs/heads/branch-1.6 a907c7c64 -> 3ca5dc307 [SPARK-13214][DOCS] update dynamicAllocation documentation Author: Bill Chambers Closes #11094 from anabranch/dynamic-docs. (cherry picked from commit

spark git commit: [SPARK-13052] waitingApps metric doesn't show the number of apps currently in the WAITING state

Repository: spark Updated Branches: refs/heads/master 7a4b37f02 -> 6dbfc4077 [SPARK-13052] waitingApps metric doesn't show the number of apps currently in the WAITING state Author: Raafat Akkad Closes #10959 from RaafatAkkad/master. Project:

spark git commit: MAINTENANCE: Automated closing of pull requests.

ted by yhuai) Closes #9288 (requested by andrewor14) Closes #9321 (requested by andrewor14) Closes #9935 (requested by JoshRosen) Closes #10442 (requested by andrewor14) Closes #10585 (requested by srowen) Closes #10785 (requested by srowen) Closes #10832 (requested by andrewor14) Closes #10941 (reques

spark git commit: [SPARK-12330][MESOS] Fix mesos coarse mode cleanup

Repository: spark Updated Branches: refs/heads/master dee801adb -> 2eaeafe8a [SPARK-12330][MESOS] Fix mesos coarse mode cleanup In the current implementation the mesos coarse scheduler does not wait for the mesos tasks to complete before ending the driver. This causes a race where the task

spark git commit: [SPARK-13164][CORE] Replace deprecated synchronized buffer in core

Repository: spark Updated Branches: refs/heads/master 2eaeafe8a -> 62a7c2838 [SPARK-13164][CORE] Replace deprecated synchronized buffer in core Building with scala 2.11 results in the warning trait SynchronizedBuffer in package mutable is deprecated: Synchronization via traits is deprecated

spark git commit: [SPARK-13162] Standalone mode does not respect initial executors

ied `spark.dynamicAllocation.initialExecutors`, the config would not take effect. This is similar to #11047 but for standalone mode. Author: Andrew Or <and...@databricks.com> Closes #11054 from andrewor14/standalone-da-initial. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/

spark git commit: [SPARK-13053][TEST] Unignore tests in InternalAccumulatorSuite

rew Or <and...@databricks.com> Closes #10969 from andrewor14/unignore-accum-test. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/15205da8 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/15205da8 Diff: http://git-wip-us.apache.org/

spark git commit: [HOTFIX] Fix style violation caused by c756bda

Repository: spark Updated Branches: refs/heads/master 8e2f29630 -> 7a4b37f02 [HOTFIX] Fix style violation caused by c756bda Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/7a4b37f0 Tree:

spark git commit: [SPARK-13152][CORE] Fix task metrics deprecation warning

now incBytesRead and incRecordsRead are marked as deprecated and for internal use only. We should make private[spark] versions which are not deprecated and switch to those internally so as to not clutter up the warning messages when building. cc andrewor14 who did the initial deprecation Author: Hol

spark git commit: [SPARK-13122] Fix race condition in MemoryStore.unrollSafely()

Repository: spark Updated Branches: refs/heads/master 21112e8a1 -> ff71261b6 [SPARK-13122] Fix race condition in MemoryStore.unrollSafely() https://issues.apache.org/jira/browse/SPARK-13122 A race condition can occur in MemoryStore's unrollSafely() method if two threads that return the same

spark git commit: [SPARK-13122] Fix race condition in MemoryStore.unrollSafely()

Repository: spark Updated Branches: refs/heads/branch-1.6 e81333be0 -> 2f8abb4af [SPARK-13122] Fix race condition in MemoryStore.unrollSafely() https://issues.apache.org/jira/browse/SPARK-13122 A race condition can occur in MemoryStore's unrollSafely() method if two threads that return the

spark git commit: [SPARK-12265][MESOS] Spark calls System.exit inside driver instead of throwing exception

Repository: spark Updated Branches: refs/heads/master 51b03b71f -> a41b68b95 [SPARK-12265][MESOS] Spark calls System.exit inside driver instead of throwing exception This takes over #10729 and makes sure that `spark-shell` fails with a proper error message. There is a slight behavioral

spark git commit: [SPARK-6847][CORE][STREAMING] Fix stack overflow issue when updateStateByKey is followed by a checkpointed dstream

Repository: spark Updated Branches: refs/heads/master c1da4d421 -> 6075573a9 [SPARK-6847][CORE][STREAMING] Fix stack overflow issue when updateStateByKey is followed by a checkpointed dstream Add a local property to indicate if checkpointing all RDDs that are marked with the checkpoint

spark git commit: [SPARK-12979][MESOS] Don’t resolve paths on the local file system in Mesos scheduler

Repository: spark Updated Branches: refs/heads/master a41b68b95 -> c9b89a0a0 [SPARK-12979][MESOS] Don’t resolve paths on the local file system in Mesos scheduler The driver filesystem is likely different from where the executors will run, so resolving paths (and symlinks, etc.) will lead

[8/9] spark git commit: [SPARK-12790][CORE] Remove HistoryServer old multiple files format

http://git-wip-us.apache.org/repos/asf/spark/blob/0df3cfb8/core/src/test/resources/spark-events/local-1422981759269 -- diff --git a/core/src/test/resources/spark-events/local-1422981759269

[7/9] spark git commit: [SPARK-12790][CORE] Remove HistoryServer old multiple files format

http://git-wip-us.apache.org/repos/asf/spark/blob/0df3cfb8/core/src/test/resources/spark-events/local-1422981759269/EVENT_LOG_1 -- diff --git a/core/src/test/resources/spark-events/local-1422981759269/EVENT_LOG_1

[4/9] spark git commit: [SPARK-12790][CORE] Remove HistoryServer old multiple files format

http://git-wip-us.apache.org/repos/asf/spark/blob/0df3cfb8/core/src/test/resources/spark-events/local-1425081759269 -- diff --git a/core/src/test/resources/spark-events/local-1425081759269

[6/9] spark git commit: [SPARK-12790][CORE] Remove HistoryServer old multiple files format

http://git-wip-us.apache.org/repos/asf/spark/blob/0df3cfb8/core/src/test/resources/spark-events/local-1422981780767 -- diff --git a/core/src/test/resources/spark-events/local-1422981780767

[5/9] spark git commit: [SPARK-12790][CORE] Remove HistoryServer old multiple files format

http://git-wip-us.apache.org/repos/asf/spark/blob/0df3cfb8/core/src/test/resources/spark-events/local-1422981780767/EVENT_LOG_1 -- diff --git a/core/src/test/resources/spark-events/local-1422981780767/EVENT_LOG_1

[2/9] spark git commit: [SPARK-12790][CORE] Remove HistoryServer old multiple files format

http://git-wip-us.apache.org/repos/asf/spark/blob/0df3cfb8/core/src/test/resources/spark-events/local-1426533911241 -- diff --git a/core/src/test/resources/spark-events/local-1426533911241

[9/9] spark git commit: [SPARK-12790][CORE] Remove HistoryServer old multiple files format

[SPARK-12790][CORE] Remove HistoryServer old multiple files format Removed isLegacyLogDirectory code path and updated tests andrewor14 Author: felixcheung <felixcheun...@hotmail.com> Closes #10860 from felixcheung/historyserverformat. Project: http://git-wip-us.apache.org/repos/asf/spar

spark git commit: [SPARK-12637][CORE] Print stage info of finished stages properly

Repository: spark Updated Branches: refs/heads/master be7a2fc07 -> 715a19d56 [SPARK-12637][CORE] Print stage info of finished stages properly Improve printing of StageInfo in onStageCompleted See also https://github.com/apache/spark/pull/10585 Author: Sean Owen Closes

[3/9] spark git commit: [SPARK-12790][CORE] Remove HistoryServer old multiple files format

http://git-wip-us.apache.org/repos/asf/spark/blob/0df3cfb8/core/src/test/resources/spark-events/local-1425081759269/EVENT_LOG_1 -- diff --git a/core/src/test/resources/spark-events/local-1425081759269/EVENT_LOG_1

[1/9] spark git commit: [SPARK-12790][CORE] Remove HistoryServer old multiple files format

Repository: spark Updated Branches: refs/heads/master 715a19d56 -> 0df3cfb8a http://git-wip-us.apache.org/repos/asf/spark/blob/0df3cfb8/core/src/test/resources/spark-events/local-1426633911242 -- diff --git

spark git commit: [SPARK-12463][SPARK-12464][SPARK-12465][SPARK-10647][MESOS] Fix zookeeper dir with mesos conf and add docs.

Repository: spark Updated Branches: refs/heads/master 711ce048a -> 51b03b71f [SPARK-12463][SPARK-12464][SPARK-12465][SPARK-10647][MESOS] Fix zookeeper dir with mesos conf and add docs. Fix zookeeper dir configuration used in cluster mode, and also add documentation around these settings.

[1/2] spark git commit: [SPARK-13088] Fix DAG viz in latest version of chrome

gre-d3.min.js @@ -1,4 +1,5 @@ -/* This is a custom version of dagre-d3 on top of v0.4.3. The full list of commits can be found at http://github.com/andrewor14/dagre-d3/ */!function(e){if("object"==typeof exports&&"undefined"!=typeof module)module.exports=e();else if(&qu

spark git commit: [SPARK-13071] Coalescing HadoopRDD overwrites existing input metrics

ks.com> Closes #10973 from andrewor14/fix-input-metrics-coalesce. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/12252d1d Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/12252d1d Diff: http://git-wip-us.apache.org/repos/

spark git commit: [SPARK-13096][TEST] Fix flaky verifyPeakExecutionMemorySet

sed, i.e. until the listener queue is empty. https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-sbt-hadoop-2.7/79/testReport/junit/org.apache.spark.util.collection/ExternalAppendOnlyMapSuite/spilling/ Author: Andrew Or <and...@databricks.com> Closes #10990 from andrewor14/accum-s

[1/2] spark git commit: [SPARK-13088] Fix DAG viz in latest version of chrome

gre-d3.min.js @@ -1,4 +1,5 @@ -/* This is a custom version of dagre-d3 on top of v0.4.3. The full list of commits can be found at http://github.com/andrewor14/dagre-d3/ */!function(e){if("object"==typeof exports&&"undefined"!=typeof module)module.exports=e();else if(&qu

[2/2] spark git commit: [SPARK-13088] Fix DAG viz in latest version of chrome

[SPARK-13088] Fix DAG viz in latest version of chrome Apparently chrome removed `SVGElement.prototype.getTransformToElement`, which is used by our JS library dagre-d3 when creating edges. The real diff can be found here: https://github.com/andrewor14/dagre-d3/commit

[1/2] spark git commit: [SPARK-13088] Fix DAG viz in latest version of chrome

-d3.min.js @@ -1,4 +1,5 @@ -/* This is a custom version of dagre-d3 on top of v0.4.3. The full list of commits can be found at http://github.com/andrewor14/dagre-d3/ */!function(e){if("object"==typeof exports&&"undefined"!=typeof module)module.exports=e();else if(&qu

[2/2] spark git commit: [SPARK-13088] Fix DAG viz in latest version of chrome

[SPARK-13088] Fix DAG viz in latest version of chrome Apparently chrome removed `SVGElement.prototype.getTransformToElement`, which is used by our JS library dagre-d3 when creating edges. The real diff can be found here: https://github.com/andrewor14/dagre-d3/commit

[1/2] spark git commit: [SPARK-13088] Fix DAG viz in latest version of chrome

gre-d3.min.js @@ -1,4 +1,5 @@ -/* This is a custom version of dagre-d3 on top of v0.4.3. The full list of commits can be found at http://github.com/andrewor14/dagre-d3/ */!function(e){if("object"==typeof exports&&"undefined"!=typeof module)module.exports=e();else if(&qu

[2/2] spark git commit: [SPARK-13088] Fix DAG viz in latest version of chrome

[SPARK-13088] Fix DAG viz in latest version of chrome Apparently chrome removed `SVGElement.prototype.getTransformToElement`, which is used by our JS library dagre-d3 when creating edges. The real diff can be found here: https://github.com/andrewor14/dagre-d3/commit

[2/2] spark git commit: [SPARK-13088] Fix DAG viz in latest version of chrome

[SPARK-13088] Fix DAG viz in latest version of chrome Apparently chrome removed `SVGElement.prototype.getTransformToElement`, which is used by our JS library dagre-d3 when creating edges. The real diff can be found here: https://github.com/andrewor14/dagre-d3/commit

spark git commit: [SPARK-12887] Do not expose var's in TaskMetrics

nse in the case of TaskMetrics because these are just aggregated metrics that we want to collect throughout the task, so it doesn't matter who's incrementing them. Parent PR: #10717 Author: Andrew Or <and...@databricks.com> Author: Josh Rosen <joshro...@databricks.com> Aut

spark git commit: [SPARK-10985][CORE] Avoid passing evicted blocks throughout BlockManager

Repository: spark Updated Branches: refs/heads/master 302bb569f -> b8cb548a4 [SPARK-10985][CORE] Avoid passing evicted blocks throughout BlockManager This patch refactors portions of the BlockManager and CacheManager in order to avoid having to pass `evictedBlocks` lists throughout the code.

spark git commit: [SPARK-12174] Speed up BlockManagerSuite getRemoteBytes() test

Repository: spark Updated Branches: refs/heads/master bcc7373f6 -> 25782981c [SPARK-12174] Speed up BlockManagerSuite getRemoteBytes() test This patch significantly speeds up the BlockManagerSuite's "SPARK-9591: getRemoteBytes from another location when Exception throw" test, reducing the

spark git commit: [SPARK-12486] Worker should kill the executors more forcefully if possible.

Repository: spark Updated Branches: refs/heads/master 962aac4db -> 8f659393b [SPARK-12486] Worker should kill the executors more forcefully if possible. This patch updates the ExecutorRunner's terminate path to use the new java 8 API to terminate processes more forcefully if possible. If the

spark git commit: [SPARK-12486] Worker should kill the executors more forcefully if possible.

Repository: spark Updated Branches: refs/heads/branch-1.6 f7a322382 -> cd0203819 [SPARK-12486] Worker should kill the executors more forcefully if possible. This patch updates the ExecutorRunner's terminate path to use the new java 8 API to terminate processes more forcefully if possible. If

spark git commit: [SPARK-12411][CORE] Decrease executor heartbeat timeout to match heartbeat interval

Repository: spark Updated Branches: refs/heads/branch-1.6 5987b1658 -> b49856ae5 [SPARK-12411][CORE] Decrease executor heartbeat timeout to match heartbeat interval Previously, the rpc timeout was the default network timeout, which is the same value the driver uses to determine dead

spark git commit: Doc typo: ltrim = trim from left end, not right

Repository: spark Updated Branches: refs/heads/master 1eb90bc9c -> fc6dbcc70 Doc typo: ltrim = trim from left end, not right Author: pshearer Closes #10414 from pshearer/patch-1. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit:

spark git commit: Doc typo: ltrim = trim from left end, not right

Repository: spark Updated Branches: refs/heads/branch-1.6 d6a519ff2 -> c754a0879 Doc typo: ltrim = trim from left end, not right Author: pshearer Closes #10414 from pshearer/patch-1. (cherry picked from commit fc6dbcc7038c2b030ef6a2dc8be5848499ccee1c)

spark git commit: Doc typo: ltrim = trim from left end, not right

Repository: spark Updated Branches: refs/heads/branch-1.5 eb54c914a -> 4d54ba896 Doc typo: ltrim = trim from left end, not right Author: pshearer Closes #10414 from pshearer/patch-1. (cherry picked from commit fc6dbcc7038c2b030ef6a2dc8be5848499ccee1c)

spark git commit: [SPARK-12392][CORE] Optimize a location order of broadcast blocks by considering preferred local hosts

Repository: spark Updated Branches: refs/heads/master 4883a5087 -> 935f46630 [SPARK-12392][CORE] Optimize a location order of broadcast blocks by considering preferred local hosts When multiple workers exist in a host, we can bypass unnecessary remote access for broadcasts; block managers

spark git commit: [SPARK-12339][SPARK-11206][WEBUI] Added a null check that was removed in

Repository: spark Updated Branches: refs/heads/master fc6dbcc70 -> b0849b8ae [SPARK-12339][SPARK-11206][WEBUI] Added a null check that was removed in Updates made in SPARK-11206 missed an edge case which cause's a NullPointerException when a task is killed. In some cases when a task ends in

spark git commit: [SPARK-2331] SparkContext.emptyRDD should return RDD[T] not EmptyRDD[T]

Repository: spark Updated Branches: refs/heads/master b0849b8ae -> a820ca19d [SPARK-2331] SparkContext.emptyRDD should return RDD[T] not EmptyRDD[T] Author: Reynold Xin Closes #10394 from rxin/SPARK-2331. Project: http://git-wip-us.apache.org/repos/asf/spark/repo

spark git commit: [SPARK-12466] Fix harmless NPE in tests

est/consoleFull This was introduced in #10284. It's harmless because the NPE is caused by a race that occurs mainly in `local-cluster` tests (but don't actually fail the tests). Tested locally to verify that the NPE is gone. Author: Andrew Or <and...@databricks.com> Closes #10417 from andre

spark git commit: [SPARK-12466] Fix harmless NPE in tests

est/consoleFull This was introduced in #10284. It's harmless because the NPE is caused by a race that occurs mainly in `local-cluster` tests (but don't actually fail the tests). Tested locally to verify that the NPE is gone. Author: Andrew Or <and...@databricks.com> Closes #10417 from andre

spark git commit: [SPARK-12345][CORE] Do not send SPARK_HOME through Spark submit REST interface

Repository: spark Updated Branches: refs/heads/master 007a32f90 -> ba9332edd [SPARK-12345][CORE] Do not send SPARK_HOME through Spark submit REST interface It is usually an invalid location on the remote machine executing the job. It is picked up by the Mesos support in cluster mode, and most

spark git commit: [SPARK-12411][CORE] Decrease executor heartbeat timeout to match heartbeat interval

Repository: spark Updated Branches: refs/heads/master 60da0e11f -> 0514e8d4b [SPARK-12411][CORE] Decrease executor heartbeat timeout to match heartbeat interval Previously, the rpc timeout was the default network timeout, which is the same value the driver uses to determine dead executors.

[3/3] spark git commit: Revert "[SPARK-12345][MESOS] Filter SPARK_HOME when submitting Spark jobs with Mesos cluster mode."

Revert "[SPARK-12345][MESOS] Filter SPARK_HOME when submitting Spark jobs with Mesos cluster mode." This reverts commit ad8c1f0b840284d05da737fb2cc5ebf8848f4490. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/a78a91f4

[1/3] spark git commit: Revert "[SPARK-12413] Fix Mesos ZK persistence"

Repository: spark Updated Branches: refs/heads/master ba9332edd -> a78a91f4d Revert "[SPARK-12413] Fix Mesos ZK persistence" This reverts commit 2bebaa39d9da33bc93ef682959cd42c1968a6a3e. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit:

[2/3] spark git commit: Revert "[SPARK-12345][MESOS] Properly filter out SPARK_HOME in the Mesos REST server"

Revert "[SPARK-12345][MESOS] Properly filter out SPARK_HOME in the Mesos REST server" This reverts commit 8184568810e8a2e7d5371db2c6a0366ef4841f70. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/8a9417bc Tree:

spark git commit: Revert "[SPARK-12365][CORE] Use ShutdownHookManager where Runtime.getRuntime.addShutdownHook() is called"

Repository: spark Updated Branches: refs/heads/branch-1.6 1dc71ec77 -> 3b903e44b Revert "[SPARK-12365][CORE] Use ShutdownHookManager where Runtime.getRuntime.addShutdownHook() is called" This reverts commit 4af64385b085002d94c54d11bbd144f9f026bbd8. Project:

<    1   2   3   4   5   6   7   8   9   10   >