spark git commit: [SPARK-9826] [CORE] Fix cannot use custom classes in log4j.properties

log4j.appender.redis.layout=net.logstash.log4j.JSONEventLayoutV1 Author: michellemay mle...@gmail.com Closes #8109 from michellemay/SPARK-9826. (cherry picked from commit ab7e721cfec63155641e81e72b4ad43cf6a7d4c7) Signed-off-by: Marcelo Vanzin van...@cloudera.com Project: http://git-wip-us.apache.org/repos

spark git commit: [SPARK-9826] [CORE] Fix cannot use custom classes in log4j.properties

/asf/spark/tree/ab7e721c Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/ab7e721c Branch: refs/heads/master Commit: ab7e721cfec63155641e81e72b4ad43cf6a7d4c7 Parents: 738f353 Author: Michel Lemay mle...@gmail.com Authored: Wed Aug 12 16:17:58 2015 -0700 Committer: Marcelo Vanzin van

spark git commit: [SPARK-9826] [CORE] Fix cannot use custom classes in log4j.properties

Aug 12 16:17:58 2015 -0700 Committer: Marcelo Vanzin van...@cloudera.com Committed: Wed Aug 12 16:56:52 2015 -0700 -- .../scala/org/apache/spark/SparkContext.scala | 5 +- .../spark/deploy/history/HistoryServer.scala| 4

spark git commit: [SPARK-9092] Fixed incompatibility when both num-executors and dynamic...

12 16:10:21 2015 -0700 Committer: Marcelo Vanzin van...@cloudera.com Committed: Wed Aug 12 16:10:21 2015 -0700 -- .../main/scala/org/apache/spark/SparkConf.scala | 19 +++ .../scala/org/apache/spark

spark git commit: [SPARK-9092] Fixed incompatibility when both num-executors and dynamic...

, executorAllocationManager in not initialized in the SparkContext. Author: Niranjan Padmanabhan niranjan.padmanab...@cloudera.com Closes #7657 from neurons/SPARK-9092. (cherry picked from commit 738f353988dbf02704bd63f5e35d94402c59ed79) Signed-off-by: Marcelo Vanzin van...@cloudera.com Project

spark git commit: [SPARK-9724] [WEB UI] Avoid unnecessary redirects in the Spark Web UI.

: 0d1d146c220f0d47d0e62b368d5b94d3bd9dd197 Parents: 8ce6096 Author: Rohit Agarwal roh...@qubole.com Authored: Wed Aug 12 17:48:43 2015 -0700 Committer: Marcelo Vanzin van...@cloudera.com Committed: Wed Aug 12 17:48:43 2015 -0700

spark git commit: [SPARK-10059] [YARN] Explicitly add JSP dependencies for tests.

Repository: spark Updated Branches: refs/heads/branch-1.5 9740d43d3 - bfb4c8425 [SPARK-10059] [YARN] Explicitly add JSP dependencies for tests. Author: Marcelo Vanzin van...@cloudera.com Closes #8251 from vanzin/SPARK-10059. (cherry picked from commit

spark git commit: [SPARK-10059] [YARN] Explicitly add JSP dependencies for tests.

Repository: spark Updated Branches: refs/heads/master c90c605dc - ee093c8b9 [SPARK-10059] [YARN] Explicitly add JSP dependencies for tests. Author: Marcelo Vanzin van...@cloudera.com Closes #8251 from vanzin/SPARK-10059. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit

spark git commit: [SPARK-7736] [CORE] [YARN] Make pyspark fail YARN app on failure.

for the application. So, for pyspark, avoid that call and instead throw an exception with the exit code. SparkSubmit handles that exception and exits with the given exit code, while YARN uses the exit code as the failure code for the Spark app. Author: Marcelo Vanzin van...@cloudera.com Closes

spark git commit: [SPARK-9924] [WEB UI] Don't schedule checkForLogs while some of them are already running.

: 76c155d Author: Rohit Agarwal roh...@qubole.com Authored: Mon Aug 17 10:31:57 2015 -0700 Committer: Marcelo Vanzin van...@cloudera.com Committed: Mon Aug 17 10:31:57 2015 -0700 -- .../deploy/history/FsHistoryProvider.scala

spark git commit: [SPARK-7736] [CORE] Fix a race introduced in PythonRunner.

. This change adds code to fix that race. Author: Marcelo Vanzin van...@cloudera.com Closes #8258 from vanzin/SPARK-7736. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/c1840a86 Tree: http://git-wip-us.apache.org/repos/asf

spark git commit: [SPARK-5754] [YARN] Spark/Yarn/Windows driver/executor escaping Fix

better, and wasn't a true solution as I didn't really understand where the problem came from. Albeit being still a bit obscure, I can name the players and have come up with a better explaination of why I am suggesting this fix. I also used vanzin and srowen input to *try* to give a more elegant

spark git commit: [SPARK-9833] [YARN] Add options to disable delegation token retrieval.

being run. Author: Marcelo Vanzin van...@cloudera.com Closes #8134 from vanzin/SPARK-9833. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/5fd53c64 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/5fd53c64 Diff: http

spark git commit: [SPARK-10119] [CORE] Fix isDynamicAllocationEnabled when config is expliticly disabled.

Repository: spark Updated Branches: refs/heads/branch-1.5 1494d58d2 - a59475f5b [SPARK-10119] [CORE] Fix isDynamicAllocationEnabled when config is expliticly disabled. Author: Marcelo Vanzin van...@cloudera.com Closes #8316 from vanzin/SPARK-10119. (cherry picked from commit

spark git commit: [SPARK-10119] [CORE] Fix isDynamicAllocationEnabled when config is expliticly disabled.

Repository: spark Updated Branches: refs/heads/master 373a376c0 - e0dd1309a [SPARK-10119] [CORE] Fix isDynamicAllocationEnabled when config is expliticly disabled. Author: Marcelo Vanzin van...@cloudera.com Closes #8316 from vanzin/SPARK-10119. Project: http://git-wip-us.apache.org/repos

spark git commit: [SPARK-9491] Avoid fetching HBase tokens when not needed.

authentication, then skip getting tokens. Reference: http://hbase.apache.org/book.html#security.prerequisites Author: Marcelo Vanzin van...@cloudera.com Closes #7810 from vanzin/SPARK-9491 and squashes the following commits: a57c776 [Marcelo Vanzin] [SPARK-9491] Avoid fetching HBase tokens when

spark git commit: [SPARK-9491] Avoid fetching HBase tokens when not needed.

authentication, then skip getting tokens. Reference: http://hbase.apache.org/book.html#security.prerequisites Author: Marcelo Vanzin van...@cloudera.com Closes #7810 from vanzin/SPARK-9491 and squashes the following commits: a57c776 [Marcelo Vanzin] [SPARK-9491] Avoid fetching HBase tokens when

spark git commit: [SPARK-9263] Added flags to exclude dependencies when using --packages

file to provide dependencies, which is possible if there is a long list of dependencies or exclusions. cc andrewor14 vanzin pwendell Author: Burak Yavuz brk...@gmail.com Closes #7599 from brkyvz/packages-exclusions and squashes the following commits: 636f410 [Burak Yavuz] addressed nits 6e54ede

spark git commit: [SPARK-9263] Added flags to exclude dependencies when using --packages

to provide dependencies, which is possible if there is a long list of dependencies or exclusions. cc andrewor14 vanzin pwendell Author: Burak Yavuz brk...@gmail.com Closes #7599 from brkyvz/packages-exclusions and squashes the following commits: 636f410 [Burak Yavuz] addressed nits 6e54ede

spark git commit: [SPARK-9327] [DOCS] Fix documentation about classpath config options.

Repository: spark Updated Branches: refs/heads/master 614323406 - 31ec6a871 [SPARK-9327] [DOCS] Fix documentation about classpath config options. Author: Marcelo Vanzin van...@cloudera.com Closes #7651 from vanzin/SPARK-9327 and squashes the following commits: 2923e23 [Marcelo Vanzin

spark git commit: [SPARK-9519] [YARN] Confirm stop sc successfully when application was killed

Vanzin van...@cloudera.com Committed: Wed Aug 5 10:16:12 2015 -0700 -- .../cluster/YarnClientSchedulerBackend.scala| 47 +--- 1 file changed, 32 insertions(+), 15 deletions

spark git commit: [SPARK-9519] [YARN] Confirm stop sc successfully when application was killed

[linweizhong] Add comment explaining the code ad0e23b [linweizhong] Update 243d2c7 [linweizhong] Confirm stop sc successfully when application was killed (cherry picked from commit 7a969a6967c4ecc0f004b73bff27a75257a94e86) Signed-off-by: Marcelo Vanzin van...@cloudera.com Project: http://git-wip

spark git commit: [SPARK-9388] [YARN] Make executor info log messages easier to read.

Repository: spark Updated Branches: refs/heads/master e53534655 - ab78b1d2a [SPARK-9388] [YARN] Make executor info log messages easier to read. Author: Marcelo Vanzin van...@cloudera.com Closes #7706 from vanzin/SPARK-9388 and squashes the following commits: 028b990 [Marcelo Vanzin] Single

spark git commit: [SPARK-8297] [YARN] Scheduler backend is not notified in case node fails in YARN

Author: Marcelo Vanzin van...@cloudera.com Closes #7431 from vanzin/SPARK-8297 and squashes the following commits: 471e4a0 [Marcelo Vanzin] Fix unit test after merge. d4adf4e [Marcelo Vanzin] Merge branch 'master' into SPARK-8297 3b262e8 [Marcelo Vanzin] Merge branch 'master' into SPARK-8297

spark git commit: [SPARK-9645] [YARN] [CORE] Allow shuffle service to read shuffle files.

not cause problems. Author: Marcelo Vanzin van...@cloudera.com Closes #7966 from vanzin/SPARK-9645 and squashes the following commits: 6e07b31 [Marcelo Vanzin] Protect the app dir in standalone mode. 384ba6a [Marcelo Vanzin] [SPARK-9645] [yarn] [core] Allow shuffle service to read shuffle files

spark git commit: [SPARK-9645] [YARN] [CORE] Allow shuffle service to read shuffle files.

not cause problems. Author: Marcelo Vanzin van...@cloudera.com Closes #7966 from vanzin/SPARK-9645 and squashes the following commits: 6e07b31 [Marcelo Vanzin] Protect the app dir in standalone mode. 384ba6a [Marcelo Vanzin] [SPARK-9645] [yarn] [core] Allow shuffle service to read shuffle files

spark git commit: [SPARK-8988] [YARN] Make sure driver log links appear in secure cluste…

: c1be9f309acad4d1b1908fa7800e7ef4f3e872ce Parents: 3ab7525 Author: Hari Shreedharan hshreedha...@apache.org Authored: Mon Jul 27 15:16:46 2015 -0700 Committer: Marcelo Vanzin van...@cloudera.com Committed: Mon Jul 27 15:16:46 2015 -0700

spark git commit: [SPARK-9261] [STREAMING] Avoid calling APIs that expose shaded classes.

methods that do not expose shaded classes. Author: Marcelo Vanzin van...@cloudera.com Closes #7601 from vanzin/SPARK-9261 and squashes the following commits: 4f64a16 [Marcelo Vanzin] [SPARK-9261] [streaming] Avoid calling APIs that expose shaded classes. Project: http://git-wip-us.apache.org/repos

spark git commit: [SPARK-9270] [PYSPARK] allow --name option in pyspark

/heads/master Commit: 9a11396113d4bb0e76e0520df4fc58e7a8ec9f69 Parents: 8399ba1 Author: Cheolsoo Park cheols...@netflix.com Authored: Fri Jul 24 11:56:55 2015 -0700 Committer: Marcelo Vanzin van...@cloudera.com Committed: Fri Jul 24 11:56:55 2015 -0700

spark git commit: [SPARK-9180] fix spark-shell to accept --name option

: 430cd7815dc7875edd126af4b90752ba8a380cf2 Parents: 798dff7 Author: Kenichi Maehashi webmas...@kenichimaehashi.com Authored: Wed Jul 22 16:15:44 2015 -0700 Committer: Marcelo Vanzin van...@cloudera.com Committed: Wed Jul 22 16:15:44 2015 -0700 -- bin

spark git commit: [SPARK-11232][CORE] Use 'offer' instead of 'put' to make sure calling send won't be interrupted

hu Oct 22 11:31:47 2015 -0700 Committer: Marcelo Vanzin <van...@cloudera.com> Committed: Thu Oct 22 11:31:47 2015 -0700 -- .../scala/org/apache/spark/rpc/netty/Dispatcher.scala | 10 +- 1 file changed, 5 in

[2/2] spark git commit: [SPARK-10812] [YARN] Fix shutdown of token renewer.

) at org.apache.spark.scheduler.DAGScheduler.stop(DAGScheduler.scala:1579) at org.apache.spark.SparkContext$$anonfun$stop$7.apply$mcV$sp(SparkContext.scala:1730) at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1185) at org.apache.spark.SparkContext.stop(SparkContext.scala:1729) Author: Marcelo Vanzin

[1/2] spark git commit: [SPARK-10812] [YARN] Spark hadoop util support switching to yarn

pos/asf/spark/diff/c49e0c3f Branch: refs/heads/branch-1.5 Commit: c49e0c3f6d25aa15b7cc25db0e9ae5a869184480 Parents: f9ad0e5 Author: Holden Karau <hol...@pigscanfly.ca> Authored: Mon Sep 28 06:33:45 2015 -0700 Committer: Marcelo Vanzin <van...@cloudera.com> Committed: Thu O

spark git commit: [SPARK-11252][NETWORK] ShuffleClient should release connection after fetching blocks had been completed for external shuffle

ent, connection for NM's shuffleService needs to be closed.andrewor14 rxin vanzin Author: Lianhui Wang <lianhuiwan...@gmail.com> Closes #9227 from lianhuiwang/spark-11252. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit

spark git commit: [SPARK-11655][CORE] Fix deadlock in handling of launcher stop().

work if running Java 7. Author: Marcelo Vanzin <van...@cloudera.com> Closes #9633 from vanzin/SPARK-11655. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/767d288b Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/

spark git commit: [SPARK-11655][CORE] Fix deadlock in handling of launcher stop().

work if running Java 7. Author: Marcelo Vanzin <van...@cloudera.com> Closes #9633 from vanzin/SPARK-11655. (cherry picked from commit 767d288b6b33a79d99324b70c2ac079fcf484a50) Signed-off-by: Marcelo Vanzin <van...@cloudera.com> Project: http://git-wip-us.apache.org/repos/as

spark git commit: [SPARK-11718][YARN][CORE] Fix explicitly killed executor dies silently issue

ete is already processed), here we should store this loss reason for later query. Here this PR chooses solution 2. Please help to review. vanzin I think this part is changed by you previously, would you please help to review? Thanks a lot. Author: jerryshao <ss...@hortonworks.com> Closes #

spark git commit: [SPARK-11718][YARN][CORE] Fix explicitly killed executor dies silently issue

ete is already processed), here we should store this loss reason for later query. Here this PR chooses solution 2. Please help to review. vanzin I think this part is changed by you previously, would you please help to review? Thanks a lot. Author: jerryshao <ss...@hortonworks.com> Clo

spark git commit: [SPARK-11617][NETWORK] Fix leak in TransportFrameDecoder.

ens. Author: Marcelo Vanzin <van...@cloudera.com> Closes #9619 from vanzin/SPARK-11617. (cherry picked from commit 540bf58f18328c68107d6c616ffd70f3a4640054) Signed-off-by: Marcelo Vanzin <van...@cloudera.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit:

spark git commit: [SPARK-11617][NETWORK] Fix leak in TransportFrameDecoder.

ens. Author: Marcelo Vanzin <van...@cloudera.com> Closes #9619 from vanzin/SPARK-11617. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/540bf58f Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/540bf58f Diff: http:

spark git commit: [SPARK-11695][CORE] Set s3a credentials

s3a-creds. (cherry picked from commit cc567b6634c3142125526f4875795c1b1e862838) Signed-off-by: Marcelo Vanzin <van...@cloudera.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/0f39c2d0 Tree: http://git-wip-us.apache.org/

spark git commit: [SPARK-11695][CORE] Set s3a credentials

er Commit: cc567b6634c3142125526f4875795c1b1e862838 Parents: 6fc2740 Author: Chris Bannister <chris.bannis...@swiftkey.com> Authored: Tue Nov 17 10:03:46 2015 -0800 Committer: Marcelo Vanzin <van...@cloudera.com> Committed: Tue Nov 1

spark git commit: [SPARK-11744][LAUNCHER] Fix print version throw exception when using pyspark shell

park/diff/6fc2740e Branch: refs/heads/master Commit: 6fc2740ebb59aca1aa0ee1e93658a7e4e69de33c Parents: 15cc36b Author: jerryshao <ss...@hortonworks.com> Authored: Tue Nov 17 10:01:33 2015 -0800 Committer: Marcelo Vanzin <van...@cloudera.com> Committed: Tue Nov 1

spark git commit: [SPARK-11744][LAUNCHER] Fix print version throw exception when using pyspark shell

<ss...@hortonworks.com> Closes #9721 from jerryshao/SPARK-11744. (cherry picked from commit 6fc2740ebb59aca1aa0ee1e93658a7e4e69de33c) Signed-off-by: Marcelo Vanzin <van...@cloudera.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/

spark git commit: [SPARK-10997][CORE] Add "client mode" to netty rpc env.

connects to the driver in "client mode", and that connection is used for all driver <-> AM communication, and so the AM is properly notified when the connection goes down. Author: Marcelo Vanzin <van...@cloudera.com> Closes #9210 from vanzin/SPARK-10997. Project: http://git-

spark git commit: [SPARK-9817][YARN] Improve the locality calculation of containers by taking pending container requests into consideraion

asf/spark/tree/a930e624 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/a930e624 Branch: refs/heads/master Commit: a930e624eb9feb0f7d37d99dcb8178feb9c0f177 Parents: 74ba952 Author: jerryshao <ss...@hortonworks.com> Authored: Mon Nov 2 10:23:30 2015 -0800 Committer: Marcelo Vanzin <v

spark git commit: [SPARK-11338] [WEBUI] Prepend app links on HistoryPage with uiRoot path

/dc7e399f Branch: refs/heads/master Commit: dc7e399fc01e74f2ba28ebd945785cc0f7759ccd Parents: 643c49c Author: Christian Kadner <ckad...@us.ibm.com> Authored: Sun Nov 1 13:09:42 2015 -0800 Committer: Marcelo Vanzin <van...@cloudera.com> Committed: Sun Nov

spark git commit: [SPARK-11073][CORE][YARN] Remove akka dependency in secret key generation.

fic to that part of the code. Author: Marcelo Vanzin <van...@cloudera.com> Closes #9257 from vanzin/SPARK-11073. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/f8d93ede Tree: http://git-wip-us.apache.org/repos/asf/s

spark git commit: [SPARK-11501][CORE][YARN] Propagate spark.rpc config to executors

park/tree/a4b5cefc Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/a4b5cefc Branch: refs/heads/master Commit: a4b5cefcf1a196e6b257e6127d6b43a7e50200ac Parents: 2e86cf1 Author: Nishkam Ravi <nishkamr...@gmail.com> Authored: Thu Nov 5 09:35:49 2015 -0800 Committer: Marcelo Vanzin <v

spark git commit: [SPARK-11555] spark on yarn spark-class --num-workers doesn't work

8) Signed-off-by: Marcelo Vanzin <van...@cloudera.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/8fb6696c Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/8fb6696c Diff: http://git-wip-us.apache.org/repos

spark git commit: [SPARK-11555] spark on yarn spark-class --num-workers doesn't work

ves <tgra...@staydecay.corp.gq1.yahoo.com> Authored: Fri Nov 6 15:24:33 2015 -0800 Committer: Marcelo Vanzin <van...@cloudera.com> Committed: Fri Nov 6 15:24:33 2015 -0800 -- .../scala/org/apache/spark/deploy/yarn/Clien

spark git commit: [SPARK-11555] spark on yarn spark-class --num-workers doesn't work

8) Signed-off-by: Marcelo Vanzin <van...@cloudera.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/b58f1ce5 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/b58f1ce5 Diff: http://git-wip-us.apache.org/repos

spark git commit: [SPARK-11484][WEBUI] Using proxyBase set by spark AM

ttp://git-wip-us.apache.org/repos/asf/spark/diff/c76865c6 Branch: refs/heads/master Commit: c76865c6220e3e7b2a266bbc4935567ef55303d8 Parents: 9da7cee Author: Srinivasa Reddy Vundela <v...@cloudera.com> Authored: Thu Nov 5 11:30:44 2015 -0800 Committer: Marcelo Vanzin <van...@cloudera.com>

spark git commit: [SPARK-11466][CORE] Avoid mockito in multi-threaded FsHistoryProviderSuite test.

may be the cause of the flakiness, since updating mocks when multiple threads may be using it doesn't work very well. It also allows some other cleanup (= less test code in FsHistoryProvider). Author: Marcelo Vanzin <van...@cloudera.com> Closes #9425 from vanzin/SPARK-11466. Project: http:

spark git commit: [SPARK-11235][NETWORK] Add ability to stream data using network lib.

uld not be reused since their semantics do not allow for the interception approach the new decoder uses. Author: Marcelo Vanzin <van...@cloudera.com> Closes #9206 from vanzin/SPARK-11235. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/

spark git commit: [SPARK-11538][BUILD] Force guava 14 in sbt build.

Repository: spark Updated Branches: refs/heads/master 468ad0ae8 -> 5e31db70b [SPARK-11538][BUILD] Force guava 14 in sbt build. sbt's version resolution code always picks the most recent version, and we don't want that for guava. Author: Marcelo Vanzin <van...@cloudera.com> Closes #

spark git commit: [SPARK-11265][YARN] YarnClient can't get tokens to talk to Hive 1.2.1 in a secure cluster

-0700 Committer: Marcelo Vanzin <van...@cloudera.com> Committed: Sat Oct 31 18:23:15 2015 -0700 -- yarn/pom.xml| 25 +++ .../org/apache/spark/deploy/yarn/Client.scala | 51 +-

spark git commit: [SPARK-10987] [YARN] Workaround for missing netty rpc disconnection event.

e driver when using the netty rpc backend; but that "ack" wasn't being sent because the handler was shutting down the rpc env before returning. So added a change to delay the shutdown a little bit, allowing the ack to be sent back. Author: Marcelo Vanzin <van...@cloudera.com> Closes

spark git commit: [SPARK-11093] [CORE] ChildFirstURLClassLoader#getResources should return all found resources, not just those in the child classloader

ds/master Commit: 0f62c2282bb30cb4fb6eea9d28b198d557a79b22 Parents: 9808052 Author: Adam Lewandowski <alewandow...@ipcoop.com> Authored: Thu Oct 15 09:45:54 2015 -0700 Committer: Marcelo Vanzin <van...@cloudera.com> Committed: Thu Oct 1

spark git commit: [SPARK-11000] [YARN] Load `metadata.Hive` class only when `hive.metastore.uris` was set to avoid bootting the database twice

er Commit: e2dfdbb2c0523517880138f214775f9a896f2271 Parents: 022a8f6 Author: huangzhaowei <carlmartin...@gmail.com> Authored: Sat Oct 17 16:41:49 2015 -0700 Committer: Marcelo Vanzin <van...@cloudera.com> Committed: Sat Oct 1

spark git commit: [SPARK-11040] [NETWORK] Make sure SASL handler delegates all events.

Repository: spark Updated Branches: refs/heads/master 135a2ce5b -> 31f315981 [SPARK-11040] [NETWORK] Make sure SASL handler delegates all events. Author: Marcelo Vanzin <van...@cloudera.com> Closes #9053 from vanzin/SPARK-11040. Project: http://git-wip-us.apache.org/repos/asf/s

spark git commit: [SPARK-11023] [YARN] Avoid creating URIs from local paths directly.

out whether paths are URIs or not, use Utils.resolveURI() which does that for us. Author: Marcelo Vanzin <van...@cloudera.com> Closes #9049 from vanzin/SPARK-11023 and squashes the following commits: 77021f2 [Marcelo Vanzin] [SPARK-11023] [yarn] Avoid creating URIs from local paths

spark git commit: [SPARK-11023] [YARN] Avoid creating URIs from local paths directly.

out whether paths are URIs or not, use Utils.resolveURI() which does that for us. Author: Marcelo Vanzin <van...@cloudera.com> Closes #9049 from vanzin/SPARK-11023 and squashes the following commits: 77021f2 [Marcelo Vanzin] [SPARK-11023] [yarn] Avoid creating URIs from local paths directly.

spark git commit: [SPARK-10812] [YARN] Fix shutdown of token renewer.

185) at org.apache.spark.SparkContext.stop(SparkContext.scala:1729) Author: Marcelo Vanzin <van...@cloudera.com> Closes #8996 from vanzin/SPARK-10812. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/4b747551 Tree: http:

spark git commit: [SPARK-10964] [YARN] Correctly register the AM with the driver.

elf` reference has already been initialized. Author: Marcelo Vanzin <van...@cloudera.com> Closes #9005 from vanzin/SPARK-10964. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/6ca27f85 Tree: http://git-wip-us.apache.org/

spark git commit: [SPARK-10739] [YARN] Add application attempt window for Spark on Yarn

ds/master Commit: f97e9323b526b3d0b0fee0ca03f4276f37bb5750 Parents: 091c2c3 Author: jerryshao <ss...@hortonworks.com> Authored: Mon Oct 12 18:17:28 2015 -0700 Committer: Marcelo Vanzin <van...@cloudera.com> Committed: Mon Oct 1

spark git commit: [SPARK-10858] YARN: archives/jar/files rename with # doesn't work unl

om> Authored: Fri Oct 9 14:06:25 2015 -0700 Committer: Marcelo Vanzin <van...@cloudera.com> Committed: Fri Oct 9 14:06:25 2015 -0700 -- core/src/main/scala/org/apache/spark/util/Utils.scala | 7 +++ core/src/test/s

spark git commit: [SPARK-10858] YARN: archives/jar/files rename with # doesn't work unl

.5 Commit: 156ac279014872de976246bb252dd28df5ab29d6 Parents: f95129c Author: Tom Graves <tgra...@yahoo-inc.com> Authored: Fri Oct 9 14:06:25 2015 -0700 Committer: Marcelo Vanzin <van...@cloudera.com> Committed: Fri Oct 9 14:08:21 2015 -0700 -- core/src/

spark git commit: [SPARK-10745][CORE] Separate configs between shuffle and RPC

er Commit: 7c5b641808740ba5eed05ba8204cdbaf3fc579f5 Parents: a416e41 Author: Shixiong Zhu <shixi...@databricks.com> Authored: Wed Nov 18 12:53:22 2015 -0800 Committer: Marcelo Vanzin <van...@cloudera.com> Committed: Wed Nov 1

spark git commit: [SPARK-10745][CORE] Separate configs between shuffle and RPC

ing/SPARK-10745. (cherry picked from commit 7c5b641808740ba5eed05ba8204cdbaf3fc579f5) Signed-off-by: Marcelo Vanzin <van...@cloudera.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/34a77679 Tree: http://git-wip-us.apach

spark git commit: [SPARK-7336] [HISTORYSERVER] Fix bug that applications status incorrect on JobHistory UI.

def1f63af Parents: 00d9af5 Author: Chuan Shao <shaoch...@huawei.com> Authored: Wed Sep 2 11:02:27 2015 -0700 Committer: Marcelo Vanzin <van...@cloudera.com> Committed: Wed Sep 2 11:02:57 2015 -0700 -- .../deploy/history/Fs

spark git commit: [SPARK-10004] [SHUFFLE] Perform auth checks when clients read shuffle data.

eed to be performed. This change makes sure that only the application that created the shuffle data can read from it. Such checks are only enabled when "spark.authenticate" is enabled, otherwise there's no secure way to make sure that the client is really who it says it is. Author: Marcel

[1/2] spark git commit: [SPARK-7736] [CORE] [YARN] Make pyspark fail YARN app on failure.

nal status for the application. So, for pyspark, avoid that call and instead throw an exception with the exit code. SparkSubmit handles that exception and exits with the given exit code, while YARN uses the exit code as the failure code for the Spark app. Author: Marcelo Vanzin <van...@cloude

[2/2] spark git commit: [SPARK-7736] [CORE] Fix a race introduced in PythonRunner.

[SPARK-7736] [CORE] Fix a race introduced in PythonRunner. The fix for SPARK-7736 introduced a race where a port value of "-1" could be passed down to the pyspark process, causing it to fail to connect back to the JVM. This change adds code to fix that race. Author: Marcelo V

spark git commit: [SPARK-9924] [WEB UI] Don't schedule checkForLogs while some of them …

pache.org/repos/asf/spark/tree/7f10bd62 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/7f10bd62 Branch: refs/heads/branch-1.5 Commit: 7f10bd620857aa558b77efcc4ba58380ea602f9b Parents: 295281f Author: Rohit Agarwal <roh...@qubole.com> Authored: Fri Sep 11 10:03:39 2015 -0700 Committer:

spark git commit: [SPARK-10300] [BUILD] [TESTS] Add support for test tags in run-tests.py.

les. Author: Marcelo Vanzin <van...@cloudera.com> Closes #8437 from vanzin/test-tags. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/8abef21d Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/8abef21d Diff: http:

spark git commit: Revert "[SPARK-10300] [BUILD] [TESTS] Add support for test tags in run-tests.py."

thor: Marcelo Vanzin <van...@cloudera.com> Authored: Tue Sep 15 13:03:38 2015 -0700 Committer: Marcelo Vanzin <van...@cloudera.com> Committed: Tue Sep 15 13:03:38 2015 -0700 -- core/pom.xml

spark git commit: [SPARK-10660] Doc describe error in the "Running Spark on YARN" page

arkOnYarnDocError. (cherry picked from commit c88bb5df94f9696677c3a429472114bc66f32a52) Signed-off-by: Marcelo Vanzin <van...@cloudera.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/eae1566d Tree: http://git-wip-us.

spark git commit: [SPARK-10531] [CORE] AppId is set as AppName in status rest api

: 81b4db3 Author: Jeff Zhang <zjf...@apache.org> Authored: Thu Sep 17 10:25:18 2015 -0700 Committer: Marcelo Vanzin <van...@cloudera.com> Committed: Thu Sep 17 10:25:18 2015 -0700 -- core/src/main/scala/org/apache/spa

spark git commit: [SPARK-10660] Doc describe error in the "Running Spark on YARN" page

arkOnYarnDocError. (cherry picked from commit c88bb5df94f9696677c3a429472114bc66f32a52) Signed-off-by: Marcelo Vanzin <van...@cloudera.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/6ee3cc4f Tree: http://git-wip-us.

spark git commit: [SPARK-10660] Doc describe error in the "Running Spark on YARN" page

bb5df Branch: refs/heads/master Commit: c88bb5df94f9696677c3a429472114bc66f32a52 Parents: 2a508df Author: yangping.wu <wyphao.2...@163.com> Authored: Thu Sep 17 09:52:40 2015 -0700 Committer: Marcelo Vanzin <van...@cloudera.com> Committed: Thu Sep 17 09:52:40 2015 -0700

spark git commit: [SPARK-10172] [CORE] disable sort in HistoryServer webUI

//git-wip-us.apache.org/repos/asf/spark/diff/81b4db37 Branch: refs/heads/master Commit: 81b4db374dd61b6f1c30511c70b6ab2a52c68faa Parents: 136c77d Author: Josiah Samuel <josiah_s...@in.ibm.com> Authored: Thu Sep 17 10:18:21 2015 -0700 Committer: Marcelo Vanzin <van...@cloudera.com> Committed

spark git commit: [SPARK-10172] [CORE] disable sort in HistoryServer webUI

hor: Josiah Samuel <josiah_s...@in.ibm.com> Closes #8506 from josiahsams/SPARK-10172. (cherry picked from commit 81b4db374dd61b6f1c30511c70b6ab2a52c68faa) Signed-off-by: Marcelo Vanzin <van...@cloudera.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wi

spark git commit: [SPARK-10172] [CORE] disable sort in HistoryServer webUI

hor: Josiah Samuel <josiah_s...@in.ibm.com> Closes #8506 from josiahsams/SPARK-10172. (cherry picked from commit 81b4db374dd61b6f1c30511c70b6ab2a52c68faa) Signed-off-by: Marcelo Vanzin <van...@cloudera.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wi

spark git commit: [SPARK-10674] [TESTS] Increase timeouts in SaslIntegrationSuite.

Repository: spark Updated Branches: refs/heads/master 4fbf33286 -> 0f5ef6dfa [SPARK-10674] [TESTS] Increase timeouts in SaslIntegrationSuite. 1s seems to trigger too many times on the jenkins build boxes, so increase the timeout and cross fingers. Author: Marcelo Vanzin <van...@cloude

spark git commit: [SPARK-10058] [CORE] [TESTS] Fix the flaky tests in HeartbeatReceiverSuite

so fixed them. Author: zsxwing <zsxw...@gmail.com> Closes #8946 from zsxwing/SPARK-10058. (cherry picked from commit 9b3e7768a27d51ddd4711c4a68a428a6875bd6d7) Signed-off-by: Marcelo Vanzin <van...@cloudera.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: htt

spark git commit: [SPARK-10058] [CORE] [TESTS] Fix the flaky tests in HeartbeatReceiverSuite

iff: http://git-wip-us.apache.org/repos/asf/spark/diff/9b3e7768 Branch: refs/heads/master Commit: 9b3e7768a27d51ddd4711c4a68a428a6875bd6d7 Parents: f21e2da Author: zsxwing <zsxw...@gmail.com> Authored: Thu Oct 1 07:09:31 2015 -0700 Committer: Marcelo Vanzin <van...@cloudera.com> Committed

spark git commit: [SPARK-10317] [CORE] Compatibility between history server script and functionality

iff: http://git-wip-us.apache.org/repos/asf/spark/diff/f85aa064 Branch: refs/heads/master Commit: f85aa06464a10f5d1563302fd76465dded475a12 Parents: b0baa11 Author: Joshi <rekhajo...@gmail.com> Authored: Fri Oct 2 15:26:11 2015 -0700 Committer: Marcelo Vanzin <van...@cloudera.com> Committ

spark git commit: [SPARK-10812] [YARN] Spark hadoop util support switching to yarn

master Commit: d8d50ed388d2e695b69d2b93a620045ef2f0bc18 Parents: b582499 Author: Holden Karau <hol...@pigscanfly.ca> Authored: Mon Sep 28 06:33:45 2015 -0700 Committer: Marcelo Vanzin <van...@cloudera.com> Committed: Mon Sep 28 06:33:45 2015 -0700 ---

spark git commit: [SPARK-10790] [YARN] Fix initial executor number not set issue and consolidate the codes

park/diff/353c30bd Branch: refs/heads/master Commit: 353c30bd7dfbd3b76fc8bc9a6dfab9321439a34b Parents: d8d50ed Author: jerryshao <ss...@hortonworks.com> Authored: Mon Sep 28 06:38:54 2015 -0700 Committer: Marcelo Vanzin <van...@cloudera.com> Committed: Mon Sep 2

spark git commit: [SPARK-10790] [YARN] Fix initial executor number not set issue and consolidate the codes

<ss...@hortonworks.com> Closes #8910 from jerryshao/SPARK-10790. (cherry picked from commit 353c30bd7dfbd3b76fc8bc9a6dfab9321439a34b) Signed-off-by: Marcelo Vanzin <van...@cloudera.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/a

spark git commit: [SPARK-10481] [YARN] SPARK_PREPEND_CLASSES make spark-yarn related jar could n…

er Commit: c0052d8d09eebadadb5ed35ac512caaf73919551 Parents: 2ddeb63 Author: Jeff Zhang <zjf...@apache.org> Authored: Wed Sep 9 10:26:53 2015 -0700 Committer: Marcelo Vanzin <van...@cloudera.com> Committed: Wed Sep

spark git commit: [SPARK-10901] [YARN] spark.yarn.user.classpath.first doesn't work

/asf/spark/tree/e9783601 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/e9783601 Branch: refs/heads/master Commit: e9783601599758df87418bf61a7b4636f06714fa Parents: 744f03e Author: Thomas Graves <tgra...@staydecay.corp.gq1.yahoo.com> Authored: Tue Oct 6 10:18:50 2015 -0700 Committer

spark git commit: [SPARK-10916] [YARN] Set perm gen size when launching containers on YARN.

ess overridden by the user (or not needed by the vm). Author: Marcelo Vanzin <van...@cloudera.com> Closes #8970 from vanzin/SPARK-10916. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/744f03e7 Tree: http://git-wip-us.apache.

spark git commit: [SPARK-10901] [YARN] spark.yarn.user.classpath.first doesn't work

t;tgra...@staydecay.corp.gq1.yahoo.com> Author: Tom Graves <tgra...@yahoo-inc.com> Closes #8959 from tgravescs/SPARK-10901. (cherry picked from commit e9783601599758df87418bf61a7b4636f06714fa) Signed-off-by: Marcelo Vanzin <van...@cloudera.com> Project: http://git-wip-us.apache.org

spark git commit: [SPARK-10582][YARN][CORE] Fix AM failure situation for dynamic allocation

mic allocation. So when AM is re-registered with driver, state in `ExecutorAllocationManager` and `CoarseGrainedSchedulerBacked` should be reset. This issue is originally addressed in #8737 , here re-opened again. Thanks a lot KaiXinXiaoLei for finding this issue. andrewor14 and vanzin would

spark git commit: [SPARK-3873][BUILD] Add style checker to enforce import ordering.

ead of having a huge change that touches everywhere. Author: Marcelo Vanzin <van...@cloudera.com> Closes #6502 from vanzin/SPARK-3873. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/2ff17bcf Tree: http://git-wip-us.apache.org/

spark git commit: [SPARK-12241][YARN] Improve failure reporting in Yarn client obtainTokenForHBase()

om> Authored: Wed Dec 9 10:25:38 2015 -0800 Committer: Marcelo Vanzin <van...@cloudera.com> Committed: Wed Dec 9 10:25:38 2015 -0800 -- .../org/apache/spark/deploy/yarn/Client.scala | 32 ++--

spark git commit: [SPARK-12084][CORE] Fix codes that uses ByteBuffer.array incorrectly

Shixiong Zhu <shixi...@databricks.com> Authored: Fri Dec 4 17:02:04 2015 -0800 Committer: Marcelo Vanzin <van...@cloudera.com> Committed: Fri Dec 4 17:02:04 2015 -0800 -- .../network/netty/NettyBlockTransfer

spark git commit: [SPARK-12142][CORE]Reply false when container allocator is not ready and reset target

mitter: Marcelo Vanzin <van...@cloudera.com> Committed: Fri Dec 4 16:50:43 2015 -0800 -- .../main/scala/org/apache/spark/ExecutorAllocationManager.scala | 1 + .../scala/org/apache/spark/deploy/yarn/ApplicationMaster.scal

  1   2   3   4   5   6   7   8   9   10   >