[spark] branch branch-2.4 updated: [SPARK-32003][CORE][2.4] When external shuffle service is used, unregister outputs for executor on fetch failure after executor is lost

This is an automated email from the ASF dual-hosted git repository. irashid pushed a commit to branch branch-2.4 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-2.4 by this push: new 39d31dc [SPARK-32003][CORE][2.4] When

[spark] branch branch-3.0 updated: [SPARK-32003][CORE][3.0] When external shuffle service is used, unregister outputs for executor on fetch failure after executor is lost

This is an automated email from the ASF dual-hosted git repository. irashid pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.0 by this push: new fd445cb [SPARK-32003][CORE][3.0] When

[spark] branch master updated: [SPARK-32003][CORE] When external shuffle service is used, unregister outputs for executor on fetch failure after executor is lost

This is an automated email from the ASF dual-hosted git repository. irashid pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new e8c06af [SPARK-32003][CORE] When external

[spark] branch master updated: [SPARK-27324][DOC][CORE] Document configurations related to executor metrics and modify a configuration

This is an automated email from the ASF dual-hosted git repository. irashid pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 387ce89 [SPARK-27324][DOC][CORE] Document

[spark] branch master updated: [SPARK-27189][CORE] Add Executor metrics and memory usage instrumentation to the metrics system

This is an automated email from the ASF dual-hosted git repository. irashid pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 729f43f [SPARK-27189][CORE] Add Executor

[spark] branch master updated: [SPARK-29398][CORE] Support dedicated thread pools for RPC endpoints

This is an automated email from the ASF dual-hosted git repository. irashid pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 2f0a38c [SPARK-29398][CORE] Support dedicated

[spark] branch master updated: [SPARK-28211][CORE][SHUFFLE] Propose Shuffle Driver Components API

This is an automated email from the ASF dual-hosted git repository. irashid pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 2e28622 [SPARK-28211][CORE][SHUFFLE] Propose

[spark] branch master updated: [SPARK-27468][CORE] Track correct storage level of RDDs and partitions

This is an automated email from the ASF dual-hosted git repository. irashid pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new d2f21b0 [SPARK-27468][CORE] Track correct

[spark] branch master updated: [SPARK-29189][SQL] Add an option to ignore block locations when listing file

This is an automated email from the ASF dual-hosted git repository. irashid pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 64fe82b [SPARK-29189][SQL] Add an option

[spark] branch master updated: [SPARK-29072][CORE] Put back usage of TimeTrackingOutputStream for UnsafeShuffleWriter and SortShuffleWriter

This is an automated email from the ASF dual-hosted git repository. irashid pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 67751e2 [SPARK-29072][CORE] Put back usage

[spark] branch master updated: [SPARK-28770][CORE][TEST] Fix ReplayListenerSuite tests that sometimes fail

This is an automated email from the ASF dual-hosted git repository. irashid pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 151b954 [SPARK-28770][CORE][TEST] Fix

[spark-website] branch asf-site updated: CVE-2019-10099

This is an automated email from the ASF dual-hosted git repository. irashid pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/spark-website.git The following commit(s) were added to refs/heads/asf-site by this push: new 6e85b26 CVE-2019-10099 6e85b26

[spark] branch master updated: [SPARK-26755][SCHEDULER] : Optimize Spark Scheduler to dequeue speculative tasks…

This is an automated email from the ASF dual-hosted git repository. irashid pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 70910e6 [SPARK-26755][SCHEDULER] : Optimize

[spark] branch master updated: [SPARK-28005][YARN] Remove unnecessary log from SparkRackResolver

This is an automated email from the ASF dual-hosted git repository. irashid pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 8313015 [SPARK-28005][YARN] Remove unnecessary

[spark] branch master updated: [SPARK-27630][CORE] Properly handle task end events from completed stages

This is an automated email from the ASF dual-hosted git repository. irashid pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 38263f6 [SPARK-27630][CORE] Properly handle

[spark] branch master updated: [SPARK-20286][CORE] Improve logic for timing out executors in dynamic allocation.

This is an automated email from the ASF dual-hosted git repository. irashid pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new b312033 [SPARK-20286][CORE] Improve logic

[spark] branch master updated: [SPARK-27590][CORE] do not consider skipped tasks when scheduling speculative tasks

This is an automated email from the ASF dual-hosted git repository. irashid pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new d124ce9 [SPARK-27590][CORE] do not consider

[spark] branch master updated: [SPARK-13704][CORE][YARN] Reduce rack resolution time

This is an automated email from the ASF dual-hosted git repository. irashid pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 52838e7 [SPARK-13704][CORE][YARN] Reduce rack

[spark] branch master updated: [SPARK-27216][CORE] Upgrade RoaringBitmap to 0.7.45 to fix Kryo unsafe ser/dser issue

This is an automated email from the ASF dual-hosted git repository. irashid pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 69dd44a [SPARK-27216][CORE] Upgrade

[spark] branch branch-2.3 updated: [SPARK-27112][CORE] : Create a resource ordering between threads to r…

This is an automated email from the ASF dual-hosted git repository. irashid pushed a commit to branch branch-2.3 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-2.3 by this push: new 7bb2b42 [SPARK-27112][CORE] : Create

[spark] branch branch-2.4 updated: [SPARK-27112][CORE] : Create a resource ordering between threads to r…

This is an automated email from the ASF dual-hosted git repository. irashid pushed a commit to branch branch-2.4 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-2.4 by this push: new 95e73b3 [SPARK-27112][CORE] : Create

[spark] branch master updated: [SPARK-26288][CORE] add initRegisteredExecutorsDB

This is an automated email from the ASF dual-hosted git repository. irashid pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 8b0aa59 [SPARK-26288][CORE] add

[spark] branch master updated: [SPARK-27112][CORE] : Create a resource ordering between threads to resolve the deadlocks encountered …

This is an automated email from the ASF dual-hosted git repository. irashid pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 7043aee [SPARK-27112][CORE] : Create

[spark] branch master updated: [SPARK-26089][CORE] Handle corruption in large shuffle blocks

This is an automated email from the ASF dual-hosted git repository. irashid pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 688b0c0 [SPARK-26089][CORE] Handle corruption

[spark] branch branch-2.3 updated: [SPARK-23433][SPARK-25250][CORE][BRANCH-2.3] Later created TaskSet should learn about the finished partitions

This is an automated email from the ASF dual-hosted git repository. irashid pushed a commit to branch branch-2.3 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-2.3 by this push: new a1ca566 [SPARK-23433][SPARK-25250

[spark] branch branch-2.4 updated: [SPARK-27065][CORE] avoid more than one active task set managers for a stage

This is an automated email from the ASF dual-hosted git repository. irashid pushed a commit to branch branch-2.4 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-2.4 by this push: new 7df5aa6 [SPARK-27065][CORE] avoid more

[spark] branch branch-2.3 updated: [SPARK-27065][CORE] avoid more than one active task set managers for a stage

This is an automated email from the ASF dual-hosted git repository. irashid pushed a commit to branch branch-2.3 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-2.3 by this push: new 877b8db [SPARK-27065][CORE] avoid more

[spark] branch master updated: [SPARK-27065][CORE] avoid more than one active task set managers for a stage

This is an automated email from the ASF dual-hosted git repository. irashid pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new cb20fbc [SPARK-27065][CORE] avoid more than

[spark] branch branch-2.4 updated: [SPARK-23433][SPARK-25250][CORE] Later created TaskSet should learn about the finished partitions

This is an automated email from the ASF dual-hosted git repository. irashid pushed a commit to branch branch-2.4 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-2.4 by this push: new db86ccb [SPARK-23433][SPARK-25250

[spark] branch master updated: [SPARK-23433][SPARK-25250][CORE] Later created TaskSet should learn about the finished partitions

This is an automated email from the ASF dual-hosted git repository. irashid pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new e5c6143 [SPARK-23433][SPARK-25250][CORE] Later

[spark] branch master updated: [SPARK-25865][CORE] Add GC information to ExecutorMetrics

This is an automated email from the ASF dual-hosted git repository. irashid pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new e5c502c [SPARK-25865][CORE] Add GC information

[spark] branch master updated: [SPARK-26688][YARN] Provide configuration of initially blacklisted YARN nodes

This is an automated email from the ASF dual-hosted git repository. irashid pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new caceaec [SPARK-26688][YARN] Provide

[spark] branch master updated: [SPARK-24522][UI] Create filter to apply HTTP security checks consistently.

This is an automated email from the ASF dual-hosted git repository. irashid pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 2783e4c [SPARK-24522][UI] Create filter

[spark] branch master updated: [SPARK-25689][YARN] Make driver, not AM, manage delegation tokens.

This is an automated email from the ASF dual-hosted git repository. irashid pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 669e8a1 [SPARK-25689][YARN] Make driver

[spark] branch master updated: [SPARK-24958][CORE] Add memory from procfs to executor metrics.

This is an automated email from the ASF dual-hosted git repository. irashid pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 90c77ea [SPARK-24958][CORE] Add memory from

spark git commit: [SPARK-25905][CORE] When getting a remote block, avoid forcing a conversion to a ChunkedByteBuffer

Repository: spark Updated Branches: refs/heads/master cb368f2c2 -> 59741887e [SPARK-25905][CORE] When getting a remote block, avoid forcing a conversion to a ChunkedByteBuffer ## What changes were proposed in this pull request? In `BlockManager`, `getRemoteValues` gets a `ChunkedByteBuffer`

spark git commit: [SPARK-26118][WEB UI] Introducing spark.ui.requestHeaderSize for setting HTTP requestHeaderSize

Repository: spark Updated Branches: refs/heads/master c34c42234 -> ab61ddb34 [SPARK-26118][WEB UI] Introducing spark.ui.requestHeaderSize for setting HTTP requestHeaderSize ## What changes were proposed in this pull request? Introducing spark.ui.requestHeaderSize for configuring Jetty's

spark git commit: [SPARK-26118][WEB UI] Introducing spark.ui.requestHeaderSize for setting HTTP requestHeaderSize

Repository: spark Updated Branches: refs/heads/branch-2.4 096e0d8f0 -> c28a27a25 [SPARK-26118][WEB UI] Introducing spark.ui.requestHeaderSize for setting HTTP requestHeaderSize ## What changes were proposed in this pull request? Introducing spark.ui.requestHeaderSize for configuring Jetty's

spark git commit: [SPARK-25904][CORE] Allocate arrays smaller than Int.MaxValue

Repository: spark Updated Branches: refs/heads/branch-2.4 947462f5a -> 47a668c2f [SPARK-25904][CORE] Allocate arrays smaller than Int.MaxValue JVMs can't allocate arrays of length exactly Int.MaxValue, so ensure we never try to allocate an array that big. This commit changes some defaults &

spark git commit: [SPARK-25904][CORE] Allocate arrays smaller than Int.MaxValue

Repository: spark Updated Branches: refs/heads/master 9e9fa2f69 -> 8fbc1830f [SPARK-25904][CORE] Allocate arrays smaller than Int.MaxValue JVMs can't allocate arrays of length exactly Int.MaxValue, so ensure we never try to allocate an array that big. This commit changes some defaults &

spark git commit: [SPARK-23781][CORE] Merge token renewer functionality into HadoopDelegationTokenManager.

Repository: spark Updated Branches: refs/heads/master af3b81607 -> 68dde3481 [SPARK-23781][CORE] Merge token renewer functionality into HadoopDelegationTokenManager. This avoids having two classes to deal with tokens; now the above class is a one-stop shop for dealing with delegation tokens.

spark git commit: [SPARK-23429][CORE][FOLLOWUP] MetricGetter should rename to ExecutorMetricType in comments

Repository: spark Updated Branches: refs/heads/master ce40efa20 -> a129f0795 [SPARK-23429][CORE][FOLLOWUP] MetricGetter should rename to ExecutorMetricType in comments ## What changes were proposed in this pull request? MetricGetter should rename to ExecutorMetricType in comments. ## How

spark git commit: [SPARK-25704][CORE] Allocate a bit less than Int.MaxValue

Repository: spark Updated Branches: refs/heads/branch-2.4 9c0c6d4d5 -> 1001d2314 [SPARK-25704][CORE] Allocate a bit less than Int.MaxValue JVMs don't you allocate arrays of length exactly Int.MaxValue, so leave a little extra room. This is necessary when reading blocks >2GB off the network

spark git commit: [SPARK-25704][CORE] Allocate a bit less than Int.MaxValue

Repository: spark Updated Branches: refs/heads/master 130121711 -> 43717dee5 [SPARK-25704][CORE] Allocate a bit less than Int.MaxValue JVMs don't you allocate arrays of length exactly Int.MaxValue, so leave a little extra room. This is necessary when reading blocks >2GB off the network (for

spark git commit: [SPARK-20327][YARN] Follow up: fix resource request tests on Hadoop 3.

Repository: spark Updated Branches: refs/heads/master 24f5bbd77 -> 7d425b190 [SPARK-20327][YARN] Follow up: fix resource request tests on Hadoop 3. The test fix is to allocate a `Resource` object only after the resource types have been initialized. Otherwise the YARN classes get in a weird

spark git commit: [SPARK-25535][CORE] Work around bad error handling in commons-crypto.

Repository: spark Updated Branches: refs/heads/master deb9588b2 -> 3eee9e024 [SPARK-25535][CORE] Work around bad error handling in commons-crypto. The commons-crypto library does some questionable error handling internally, which can lead to JVM crashes if some call into native code fails and

[3/3] spark git commit: [PYSPARK][SQL] Updates to RowQueue

[PYSPARK][SQL] Updates to RowQueue Tested with updates to RowQueueSuite (cherry picked from commit 6d742d1bd71aa3803dce91a830b37284cb18cf70) Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/4f10aff4 Tree:

[1/3] spark git commit: [SPARK-25253][PYSPARK] Refactor local connection & auth code

Repository: spark Updated Branches: refs/heads/branch-2.2 bd12eb75d -> 4f10aff40 [SPARK-25253][PYSPARK] Refactor local connection & auth code This eliminates some duplication in the code to connect to a server on localhost to talk directly to the jvm. Also it gives consistent ipv6 and error

[2/3] spark git commit: [PYSPARK] Updates to pyspark broadcast

[PYSPARK] Updates to pyspark broadcast (cherry picked from commit 09dd34cb1706f2477a89174d6a1a0f17ed5b0a65) Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/dd0e7cf5 Tree:

spark git commit: [SPARK-25456][SQL][TEST] Fix PythonForeachWriterSuite

Repository: spark Updated Branches: refs/heads/master 123f0041d -> a6f37b074 [SPARK-25456][SQL][TEST] Fix PythonForeachWriterSuite PythonForeachWriterSuite was failing because RowQueue now needs to have a handle on a SparkEnv with a SerializerManager, so added a mock env with a serializer

spark git commit: [SPARK-25456][SQL][TEST] Fix PythonForeachWriterSuite

Repository: spark Updated Branches: refs/heads/branch-2.4 67f2cb6e0 -> 76514a015 [SPARK-25456][SQL][TEST] Fix PythonForeachWriterSuite PythonForeachWriterSuite was failing because RowQueue now needs to have a handle on a SparkEnv with a SerializerManager, so added a mock env with a

[1/3] spark git commit: [PYSPARK] Updates to pyspark broadcast

Repository: spark Updated Branches: refs/heads/branch-2.4 08f7b145c -> 7beb34175 [PYSPARK] Updates to pyspark broadcast (cherry picked from commit 58419b92673c46911c25bc6c6b13397f880c6424) Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit:

[3/3] spark git commit: [CORE] Updates to remote cache reads

[CORE] Updates to remote cache reads Covered by tests in DistributedSuite (cherry picked from commit a97001d21757ae214c86371141bd78a376200f66) Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/7beb3417 Tree:

[2/3] spark git commit: [PYSPARK][SQL] Updates to RowQueue

[PYSPARK][SQL] Updates to RowQueue Tested with updates to RowQueueSuite (cherry picked from commit 8f5a5a9e5b9f273443b2721f80c99dc7397ef4c0) Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/80e317b3 Tree:

[1/3] spark git commit: [CORE] Updates to remote cache reads

Repository: spark Updated Branches: refs/heads/master 553af22f2 -> a97001d21 [CORE] Updates to remote cache reads Covered by tests in DistributedSuite Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/a97001d2 Tree:

[3/3] spark git commit: [PYSPARK] Updates to pyspark broadcast

[PYSPARK] Updates to pyspark broadcast Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/58419b92 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/58419b92 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/58419b92

[2/3] spark git commit: [PYSPARK][SQL] Updates to RowQueue

[PYSPARK][SQL] Updates to RowQueue Tested with updates to RowQueueSuite Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/8f5a5a9e Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/8f5a5a9e Diff:

spark git commit: [HOTFIX] fix lint-java

Repository: spark Updated Branches: refs/heads/branch-2.3 575fea120 -> f3bbb7ceb [HOTFIX] fix lint-java Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/f3bbb7ce Tree:

[1/4] spark git commit: [CORE] Updates to remote cache reads

Repository: spark Updated Branches: refs/heads/branch-2.3 9ac9f36c4 -> 575fea120 [CORE] Updates to remote cache reads Covered by tests in DistributedSuite Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/575fea12 Tree:

[3/4] spark git commit: [PYSPARK][SQL] Updates to RowQueue

[PYSPARK][SQL] Updates to RowQueue Tested with updates to RowQueueSuite Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/6d742d1b Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/6d742d1b Diff:

[2/4] spark git commit: [PYSPARK] Updates to pyspark broadcast

[PYSPARK] Updates to pyspark broadcast Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/09dd34cb Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/09dd34cb Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/09dd34cb

[4/4] spark git commit: [SPARK-25253][PYSPARK] Refactor local connection & auth code

[SPARK-25253][PYSPARK] Refactor local connection & auth code This eliminates some duplication in the code to connect to a server on localhost to talk directly to the jvm. Also it gives consistent ipv6 and error handling. Two other incidental changes, that shouldn't matter: 1) python barrier

spark git commit: [PYSPARK] Updates to Accumulators

Repository: spark Updated Branches: refs/heads/branch-2.1 a3eb07db3 -> b2e0f68f6 [PYSPARK] Updates to Accumulators (cherry picked from commit 15fc2372269159ea2556b028d4eb8860c4108650) Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit:

spark git commit: [PYSPARK] Updates to Accumulators

Repository: spark Updated Branches: refs/heads/branch-2.2 22ce8051f -> a5624c7ae [PYSPARK] Updates to Accumulators (cherry picked from commit 15fc2372269159ea2556b028d4eb8860c4108650) Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit:

spark git commit: [PYSPARK] Updates to Accumulators

Repository: spark Updated Branches: refs/heads/branch-2.3 5b187a85a -> 8080c937d [PYSPARK] Updates to Accumulators (cherry picked from commit 15fc2372269159ea2556b028d4eb8860c4108650) Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit:

spark git commit: [SPARK-24801][CORE] Avoid memory waste by empty byte[] arrays in SaslEncryption$EncryptedMessage

Repository: spark Updated Branches: refs/heads/master fa09d9192 -> 094aa5971 [SPARK-24801][CORE] Avoid memory waste by empty byte[] arrays in SaslEncryption$EncryptedMessage ## What changes were proposed in this pull request? Initialize SaslEncryption$EncryptedMessage.byteChannel lazily, so

spark git commit: [SPARK-16630][YARN] Blacklist a node if executors won't launch on it

Repository: spark Updated Branches: refs/heads/master c0cad596b -> b56e9c613 [SPARK-16630][YARN] Blacklist a node if executors won't launch on it ## What changes were proposed in this pull request? This change extends YARN resource allocation handling with blacklisting functionality. This

spark git commit: [SPARK-24416] Fix configuration specification for killBlacklisted executors

Repository: spark Updated Branches: refs/heads/master 2824f1436 -> 3af1d3e6d [SPARK-24416] Fix configuration specification for killBlacklisted executors ## What changes were proposed in this pull request? spark.blacklist.killBlacklistedExecutors is defined as (Experimental) If set to

spark git commit: [SPARK-23433][CORE] Late zombie task completions update all tasksets

Repository: spark Updated Branches: refs/heads/branch-2.2 4f1ae3af9 -> 154bbc959 [SPARK-23433][CORE] Late zombie task completions update all tasksets Fetch failure lead to multiple tasksets which are active for a given stage. While there is only one "active" version of the taskset, the

spark git commit: [SPARK-23433][CORE] Late zombie task completions update all tasksets

Repository: spark Updated Branches: refs/heads/branch-2.3 61e7bc0c1 -> 8509284e1 [SPARK-23433][CORE] Late zombie task completions update all tasksets Fetch failure lead to multiple tasksets which are active for a given stage. While there is only one "active" version of the taskset, the

spark git commit: [SPARK-23433][CORE] Late zombie task completions update all tasksets

Repository: spark Updated Branches: refs/heads/master 96a50016b -> 94641fe6c [SPARK-23433][CORE] Late zombie task completions update all tasksets Fetch failure lead to multiple tasksets which are active for a given stage. While there is only one "active" version of the taskset, the earlier

spark git commit: [SPARK-23888][CORE] correct the comment of hasAttemptOnHost()

Repository: spark Updated Branches: refs/heads/master e82cb6834 -> c8f3ac69d [SPARK-23888][CORE] correct the comment of hasAttemptOnHost() TaskSetManager.hasAttemptOnHost had a misleading comment. The comment said that it only checked for running tasks, but really it checked for any tasks

spark git commit: [SPARK-24021][CORE] fix bug in BlacklistTracker's updateBlacklistForFetchFailure

Repository: spark Updated Branches: refs/heads/branch-2.3 32bec6ca3 -> 7fb11176f [SPARK-24021][CORE] fix bug in BlacklistTracker's updateBlacklistForFetchFailure ## What changes were proposed in this pull request? There‘s a miswrite in BlacklistTracker's updateBlacklistForFetchFailure: ```

spark git commit: [SPARK-24021][CORE] fix bug in BlacklistTracker's updateBlacklistForFetchFailure

Repository: spark Updated Branches: refs/heads/master d96c3e33c -> 0deaa5251 [SPARK-24021][CORE] fix bug in BlacklistTracker's updateBlacklistForFetchFailure ## What changes were proposed in this pull request? There‘s a miswrite in BlacklistTracker's updateBlacklistForFetchFailure: ``` val

spark git commit: [SPARK-23948] Trigger mapstage's job listener in submitMissingTasks

Repository: spark Updated Branches: refs/heads/branch-2.3 564019b92 -> 6b99d5bc3 [SPARK-23948] Trigger mapstage's job listener in submitMissingTasks ## What changes were proposed in this pull request? SparkContext submitted a map stage from `submitMapStage` to `DAGScheduler`,

spark git commit: [SPARK-23948] Trigger mapstage's job listener in submitMissingTasks

Repository: spark Updated Branches: refs/heads/master ed4101d29 -> 3990daaf3 [SPARK-23948] Trigger mapstage's job listener in submitMissingTasks ## What changes were proposed in this pull request? SparkContext submitted a map stage from `submitMapStage` to `DAGScheduler`,

spark git commit: [SPARK-22941][CORE] Do not exit JVM when submit fails with in-process launcher.

Repository: spark Updated Branches: refs/heads/master 653fe0241 -> 3cb82047f [SPARK-22941][CORE] Do not exit JVM when submit fails with in-process launcher. The current in-process launcher implementation just calls the SparkSubmit object, which, in case of errors, will more often than not

spark git commit: [SPARK-23053][CORE][BRANCH-2.1] taskBinarySerialization and task partitions calculate in DagScheduler.submitMissingTasks should keep the same RDD checkpoint status

Repository: spark Updated Branches: refs/heads/branch-2.1 1df8020e1 -> 24fe6eb0f [SPARK-23053][CORE][BRANCH-2.1] taskBinarySerialization and task partitions calculate in DagScheduler.submitMissingTasks should keep the same RDD checkpoint status ## What changes were proposed in this pull

spark git commit: [SPARK-23413][UI] Fix sorting tasks by Host / Executor ID at the Stag…

Repository: spark Updated Branches: refs/heads/branch-2.3 0bd7765cd -> 75bb19a01 [SPARK-23413][UI] Fix sorting tasks by Host / Executor ID at the Stag… …e page ## What changes were proposed in this pull request? Fixing exception got at sorting tasks by Host / Executor ID: ```

spark git commit: [SPARK-23413][UI] Fix sorting tasks by Host / Executor ID at the Stage page

Repository: spark Updated Branches: refs/heads/master db45daab9 -> 1dc2c1d5e [SPARK-23413][UI] Fix sorting tasks by Host / Executor ID at the Stage page ## What changes were proposed in this pull request? Fixing exception got at sorting tasks by Host / Executor ID: ```

spark git commit: [SPARK-23235][CORE] Add executor Threaddump to api

Repository: spark Updated Branches: refs/heads/master 2ee76c22b -> a5a4b8350 [SPARK-23235][CORE] Add executor Threaddump to api ## What changes were proposed in this pull request? Extending api with the executor thread dump data. For this new REST URL is introduced: - GET

spark git commit: [SPARK-23053][CORE] taskBinarySerialization and task partitions calculate in DagScheduler.submitMissingTasks should keep the same RDD checkpoint status

Repository: spark Updated Branches: refs/heads/branch-2.2 14b5dbfa9 -> 73263b215 [SPARK-23053][CORE] taskBinarySerialization and task partitions calculate in DagScheduler.submitMissingTasks should keep the same RDD checkpoint status ## What changes were proposed in this pull request? When

spark git commit: [SPARK-23053][CORE] taskBinarySerialization and task partitions calculate in DagScheduler.submitMissingTasks should keep the same RDD checkpoint status

Repository: spark Updated Branches: refs/heads/branch-2.3 1c81c0c62 -> dbb1b399b [SPARK-23053][CORE] taskBinarySerialization and task partitions calculate in DagScheduler.submitMissingTasks should keep the same RDD checkpoint status ## What changes were proposed in this pull request? When

spark git commit: [SPARK-23053][CORE] taskBinarySerialization and task partitions calculate in DagScheduler.submitMissingTasks should keep the same RDD checkpoint status

Repository: spark Updated Branches: refs/heads/master d6e1958a2 -> 091a000d2 [SPARK-23053][CORE] taskBinarySerialization and task partitions calculate in DagScheduler.submitMissingTasks should keep the same RDD checkpoint status ## What changes were proposed in this pull request? When we

spark git commit: [SPARK-23189][CORE][WEB UI] Reflect stage level blacklisting on executor tab

Repository: spark Updated Branches: refs/heads/master 116c581d2 -> d6e1958a2 [SPARK-23189][CORE][WEB UI] Reflect stage level blacklisting on executor tab ## What changes were proposed in this pull request? The purpose of this PR to reflect the stage level blacklisting on the executor tab

spark git commit: [SPARK-23253][CORE][SHUFFLE] Only write shuffle temporary index file when there is not an existing one

Repository: spark Updated Branches: refs/heads/master b9503fcbb -> dd52681bf [SPARK-23253][CORE][SHUFFLE] Only write shuffle temporary index file when there is not an existing one ## What changes were proposed in this pull request? Shuffle Index temporay file is used for atomic creating

spark git commit: [SPARK-23209][core] Allow credential manager to work when Hive not available.

Repository: spark Updated Branches: refs/heads/branch-2.3 438631031 -> 75131ee86 [SPARK-23209][core] Allow credential manager to work when Hive not available. The JVM seems to be doing early binding of classes that the Hive provider depends on, causing an error to be thrown before it was

spark git commit: [SPARK-23209][core] Allow credential manager to work when Hive not available.

Repository: spark Updated Branches: refs/heads/master e30b34f7b -> b834446ec [SPARK-23209][core] Allow credential manager to work when Hive not available. The JVM seems to be doing early binding of classes that the Hive provider depends on, causing an error to be thrown before it was caught

[2/5] spark git commit: [SPARK-22577][CORE] executor page blacklist status should update with TaskSet level blacklisting

http://git-wip-us.apache.org/repos/asf/spark/blob/0ec95bb7/core/src/test/resources/spark-events/application_1516285256255_0012 -- diff --git a/core/src/test/resources/spark-events/application_1516285256255_0012

[4/5] spark git commit: [SPARK-22577][CORE] executor page blacklist status should update with TaskSet level blacklisting

http://git-wip-us.apache.org/repos/asf/spark/blob/0ec95bb7/core/src/test/resources/HistoryServerExpectations/one_stage_attempt_json_expectation.json -- diff --git

[1/5] spark git commit: [SPARK-22577][CORE] executor page blacklist status should update with TaskSet level blacklisting

Repository: spark Updated Branches: refs/heads/master de36f65d3 -> 0ec95bb7d http://git-wip-us.apache.org/repos/asf/spark/blob/0ec95bb7/core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala -- diff --git

[5/5] spark git commit: [SPARK-22577][CORE] executor page blacklist status should update with TaskSet level blacklisting

ndTimeEpoch" : 1515493477606 + } ] +}, { "id" : "app-20161116163331-", "name" : "Spark shell", "attempts" : [ { @@ -9,9 +39,9 @@ "sparkUser" : "jose", "completed" : true, "a

[3/5] spark git commit: [SPARK-22577][CORE] executor page blacklist status should update with TaskSet level blacklisting

http://git-wip-us.apache.org/repos/asf/spark/blob/0ec95bb7/core/src/test/resources/spark-events/app-20180109111548- -- diff --git a/core/src/test/resources/spark-events/app-20180109111548-

spark git commit: [SPARK-23103][CORE] Ensure correct sort order for negative values in LevelDB.

Repository: spark Updated Branches: refs/heads/branch-2.3 4b79514c9 -> d0cb19873 [SPARK-23103][CORE] Ensure correct sort order for negative values in LevelDB. The code was sorting "0" as "less than" negative values, which is a little wrong. Fix is simple, most of the changes are the added

spark git commit: [SPARK-23103][CORE] Ensure correct sort order for negative values in LevelDB.

Repository: spark Updated Branches: refs/heads/master fed2139f0 -> aa3a1276f [SPARK-23103][CORE] Ensure correct sort order for negative values in LevelDB. The code was sorting "0" as "less than" negative values, which is a little wrong. Fix is simple, most of the changes are the added test

spark git commit: [SPARK-20664][CORE] Delete stale application data from SHS.

Repository: spark Updated Branches: refs/heads/branch-2.3 ffe45913d -> 4b79514c9 [SPARK-20664][CORE] Delete stale application data from SHS. Detect the deletion of event log files from storage, and remove data about the related application attempt in the SHS. Also contains code to fix

spark git commit: [SPARK-20664][CORE] Delete stale application data from SHS.

Repository: spark Updated Branches: refs/heads/master 07296a61c -> fed2139f0 [SPARK-20664][CORE] Delete stale application data from SHS. Detect the deletion of event log files from storage, and remove data about the related application attempt in the SHS. Also contains code to fix

spark git commit: [SPARK-22850][CORE] Ensure queued events are delivered to all event queues.

Repository: spark Updated Branches: refs/heads/branch-2.3 cd92913f3 -> bc4bef472 [SPARK-22850][CORE] Ensure queued events are delivered to all event queues. The code in LiveListenerBus was queueing events before start in the queues themselves; so in situations like the following:

spark git commit: [SPARK-22850][CORE] Ensure queued events are delivered to all event queues.

Repository: spark Updated Branches: refs/heads/master 93f92c0ed -> d2cddc88e [SPARK-22850][CORE] Ensure queued events are delivered to all event queues. The code in LiveListenerBus was queueing events before start in the queues themselves; so in situations like the following:

spark git commit: [SPARK-22921][PROJECT-INFRA] Bug fix in jira assigning

Repository: spark Updated Branches: refs/heads/master 66a7d6b30 -> ccda75b0d [SPARK-22921][PROJECT-INFRA] Bug fix in jira assigning Small bug fix from last pr, ran a successful merge with this code. Author: Imran Rashid Closes #20117 from squito/SPARK-22921.

  1   2   3   >