spark git commit: [SPARK-2691] [MESOS] Support for Mesos DockerInfo

Repository: spark Updated Branches: refs/heads/master b4b43df8a - 8f50a07d2 [SPARK-2691] [MESOS] Support for Mesos DockerInfo This patch adds partial support for running spark on mesos inside of a docker container. Only fine-grained mode is presently supported, and there is no checking done

[2/2] spark git commit: [SPARK-6229] Add SASL encryption to network library.

[SPARK-6229] Add SASL encryption to network library. There are two main parts of this change: - Extending the bootstrap mechanism in the network library to add a server-side bootstrap (which works a little bit differently than the client-side bootstrap), and to allow the bootstraps to

[1/2] spark git commit: [SPARK-6229] Add SASL encryption to network library.

Repository: spark Updated Branches: refs/heads/master 8f50a07d2 - 38d4e9e44 http://git-wip-us.apache.org/repos/asf/spark/blob/38d4e9e4/network/shuffle/src/test/java/org/apache/spark/network/shuffle/ExternalShuffleSecuritySuite.java

spark git commit: [SPARK-2808][Streaming][Kafka] update kafka to 0.8.2

Repository: spark Updated Branches: refs/heads/master b88c275e6 - 478648407 [SPARK-2808][Streaming][Kafka] update kafka to 0.8.2 i don't think this should be merged until after 1.3.0 is final Author: cody koeninger c...@koeninger.org Author: Helena Edelson helena.edel...@datastax.com Closes

spark git commit: [SPARK-7112][Streaming][WIP] Add a InputInfoTracker to track all the input streams

Repository: spark Updated Branches: refs/heads/master ebc25a4dd - b88c275e6 [SPARK-7112][Streaming][WIP] Add a InputInfoTracker to track all the input streams Author: jerryshao saisai.s...@intel.com Author: Saisai Shao saisai.s...@intel.com Closes #5680 from jerryshao/SPARK-7111 and

spark git commit: [SPARK-7216] [MESOS] Add driver details page to Mesos cluster UI.

Repository: spark Updated Branches: refs/heads/master 099327d53 - 202219341 [SPARK-7216] [MESOS] Add driver details page to Mesos cluster UI. Add a details page that displays Mesos driver in the Mesos cluster UI Author: Timothy Chen tnac...@gmail.com Closes #5763 from

spark git commit: [SPARK-6443] [SPARK SUBMIT] Could not submit app in standalone cluster mode when HA is enabled

Repository: spark Updated Branches: refs/heads/master 202219341 - b4b43df8a [SPARK-6443] [SPARK SUBMIT] Could not submit app in standalone cluster mode when HA is enabled **3/26 update:** * Akka-based: Use an array of `ActorSelection` to represent multiple master. Add an

spark git commit: [SPARK-7309] [CORE] [STREAMING] Shutdown the thread pools in ReceivedBlockHandler and DAGScheduler

Repository: spark Updated Branches: refs/heads/master 98e704580 - ebc25a4dd [SPARK-7309] [CORE] [STREAMING] Shutdown the thread pools in ReceivedBlockHandler and DAGScheduler Shutdown the thread pools in ReceivedBlockHandler and DAGScheduler when stopping them. Author: zsxwing

spark git commit: [SPARK-3444] Provide an easy way to change log level

Repository: spark Updated Branches: refs/heads/master 478648407 - ae98eec73 [SPARK-3444] Provide an easy way to change log level Add support for changing the log level at run time through the SparkContext. Based on an earlier PR, #2433 includes CR feedback from pwendel davies Author:

spark git commit: [SPARK-6999] [SQL] Remove the infinite recursive method (useless)

Repository: spark Updated Branches: refs/heads/master e6fb37712 - 98e704580 [SPARK-6999] [SQL] Remove the infinite recursive method (useless) Remove the method, since it causes infinite recursive calls. And seems it's a dummy method, since we have the API: `def createDataFrame(rowRDD:

spark git commit: [SPARK-7304] [BUILD] Include $@ in call to mvn consistently in make-distribution.sh

Repository: spark Updated Branches: refs/heads/master 41c6a44b1 - e6fb37712 [SPARK-7304] [BUILD] Include $@ in call to mvn consistently in make-distribution.sh Adding the $ allows the caller of this script to supply additional arguments to the mvn command and is consistent with how mvn is

spark git commit: [SPARK-7317] [Shuffle] Expose shuffle handle

Repository: spark Updated Branches: refs/heads/master 38d4e9e44 - b79aeb95b [SPARK-7317] [Shuffle] Expose shuffle handle Details in JIRA, in a nut-shell, all machinary for custom RDD's to leverage spark shuffle directly (without exposing impl details of shuffle) exists - except for this

spark git commit: [SPARK-6954] [YARN] ExecutorAllocationManager can end up requesting a negative n...

Repository: spark Updated Branches: refs/heads/master ae98eec73 - 099327d53 [SPARK-6954] [YARN] ExecutorAllocationManager can end up requesting a negative n... ...umber of executors Author: Sandy Ryza sa...@cloudera.com Closes #5704 from sryza/sandy-spark-6954 and squashes the following

spark git commit: [SPARK-3066] [MLLIB] Support recommendAll in matrix factorization model

Repository: spark Updated Branches: refs/heads/master 3052f4916 - 3b514af8a [SPARK-3066] [MLLIB] Support recommendAll in matrix factorization model This is based on #3098 from debasish83. 1. BLAS' GEMM is used to compute inner products. 2. Reverted changes to MovieLensALS. SPARK-4231 should

spark git commit: [SPARK-5891] [ML] Add Binarizer ML Transformer

Repository: spark Updated Branches: refs/heads/master 3b514af8a - 7630213ca [SPARK-5891] [ML] Add Binarizer ML Transformer JIRA: https://issues.apache.org/jira/browse/SPARK-5891 Author: Liang-Chi Hsieh vii...@gmail.com Closes #5699 from viirya/add_binarizer and squashes the following

spark git commit: [SPARK-4705] Handle multiple app attempts event logs, history server.

Repository: spark Updated Branches: refs/heads/master 7fe0f3f2b - 3052f4916 [SPARK-4705] Handle multiple app attempts event logs, history server. This change modifies the event logging listener to write the logs for different application attempts to different files. The attempt ID is set by

[2/2] spark git commit: [SPARK-4550] In sort-based shuffle, store map outputs in serialized form

[SPARK-4550] In sort-based shuffle, store map outputs in serialized form Refer to the JIRA for the design doc and some perf results. I wanted to call out some of the more possibly controversial changes up front: * Map outputs are only stored in serialized form when Kryo is in use. I'm still

[1/2] spark git commit: [SPARK-4550] In sort-based shuffle, store map outputs in serialized form

Repository: spark Updated Branches: refs/heads/master a9fc50552 - 0a2b15ce4 http://git-wip-us.apache.org/repos/asf/spark/blob/0a2b15ce/core/src/test/scala/org/apache/spark/util/collection/ExternalSorterSuite.scala -- diff

spark git commit: [SPARK-7291] [CORE] Fix a flaky test in AkkaRpcEnvSuite

Repository: spark Updated Branches: refs/heads/master 7cf1eb79b - 14b32886f [SPARK-7291] [CORE] Fix a flaky test in AkkaRpcEnvSuite Read the port from RpcEnv to check the result so that it will success even if port conflicts Author: zsxwing zsxw...@gmail.com Closes #5822 from

spark git commit: [SPARK-7287] enabled fixed test

Repository: spark Updated Branches: refs/heads/master 0a2b15ce4 - 7cf1eb79b [SPARK-7287] enabled fixed test andrewor14 pwendell I reenabled the test. Let's see if it's fixed. I did also notice that `--jars` started to fail after this was ignored though in the JIRA. like

spark git commit: [SPARK-6257] [PYSPARK] [MLLIB] MLlib API missing items in Recommendation

Repository: spark Updated Branches: refs/heads/master 14b32886f - c24aeb6a3 [SPARK-6257] [PYSPARK] [MLLIB] MLlib API missing items in Recommendation Adds rank, recommendUsers and RecommendProducts to MatrixFactorizationModel in PySpark. Author: MechCoder manojkumarsivaraj...@gmail.com

spark git commit: [SPARK-7281] [YARN] Add option to set AM's lib path in client mode.

Repository: spark Updated Branches: refs/heads/master f53a48827 - 7b5dd3e3c [SPARK-7281] [YARN] Add option to set AM's lib path in client mode. Author: Marcelo Vanzin van...@cloudera.com Closes #5813 from vanzin/SPARK-7281 and squashes the following commits: 1cb6f42 [Marcelo Vanzin]

spark git commit: [SPARK-7240][SQL] Single pass covariance calculation for dataframes

Repository: spark Updated Branches: refs/heads/master 7b5dd3e3c - 4dc8d7449 [SPARK-7240][SQL] Single pass covariance calculation for dataframes Added the calculation of covariance between two columns to DataFrames. cc mengxr rxin Author: Burak Yavuz brk...@gmail.com Closes #5825 from

[1/2] spark git commit: Revert [SPARK-7287] enabled fixed test

Repository: spark Updated Branches: refs/heads/master 37537760d - c6d9a4294 Revert [SPARK-7287] enabled fixed test This reverts commit 7cf1eb79b1fa290aa1d867a8a1eaaea86d6b2239. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit:

spark git commit: [SPARK-5342] [YARN] Allow long running Spark apps to run on secure YARN/HDFS

Repository: spark Updated Branches: refs/heads/master 4dc8d7449 - b1f4ca82d [SPARK-5342] [YARN] Allow long running Spark apps to run on secure YARN/HDFS Take 2. Does the same thing as #4688, but fixes Hadoop-1 build. Author: Hari Shreedharan hshreedha...@apache.org Closes #5823 from

spark git commit: [SPARK-7274] [SQL] Create Column expression for array/struct creation.

Repository: spark Updated Branches: refs/heads/master 168603272 - 37537760d [SPARK-7274] [SQL] Create Column expression for array/struct creation. Author: Reynold Xin r...@databricks.com Closes #5802 from rxin/SPARK-7274 and squashes the following commits: 19aecaa [Reynold Xin] Fixed

spark git commit: [SPARK-7213] [YARN] Check for read permissions before copying a Hadoop config file

Repository: spark Updated Branches: refs/heads/master c6d9a4294 - f53a48827 [SPARK-7213] [YARN] Check for read permissions before copying a Hadoop config file Author: Nishkam Ravi nr...@cloudera.com Author: nishkamravi2 nishkamr...@gmail.com Author: nravi nr...@c1704.halxg.cloudera.com

spark git commit: Limit help option regex

Repository: spark Updated Branches: refs/heads/master 7630213ca - c8c481da1 Limit help option regex Added word-boundary delimiters so that embedded text such as -h within command line options and values doesn't trigger the usage script and exit. Author: Chris Biow chris.b...@10gen.com

spark git commit: Limit help option regex

Repository: spark Updated Branches: refs/heads/branch-1.3 f64b99421 - d7269493d Limit help option regex Added word-boundary delimiters so that embedded text such as -h within command line options and values doesn't trigger the usage script and exit. Author: Chris Biow chris.b...@10gen.com

spark git commit: [SPARK-5854] personalized page rank

Repository: spark Updated Branches: refs/heads/master 27de6fef6 - 7d427222d [SPARK-5854] personalized page rank Here's a modification to PageRank which does personalized PageRank. The approach is basically similar to that outlined by Bahmani et al. from 2010

spark git commit: [SPARK-6846] [WEBUI] [HOTFIX] return to GET for kill link in UI since YARN AM won't proxy POST

Repository: spark Updated Branches: refs/heads/master 7d427222d - 1262e310c [SPARK-6846] [WEBUI] [HOTFIX] return to GET for kill link in UI since YARN AM won't proxy POST Partial undoing of SPARK-6846; YARN AM proxy won't forward POSTs, so go back to GET for kill links in Spark UI.

spark git commit: changing persistence engine trait to an abstract class

Repository: spark Updated Branches: refs/heads/master c8c481da1 - 27de6fef6 changing persistence engine trait to an abstract class Author: niranda niranda.per...@gmail.com Closes #5832 from nirandaperera/PersistanceEngine_abstract_class and squashes the following commits: 67b9d5a [niranda]

spark git commit: [SPARK-7183] [NETWORK] Fix memory leak of TransportRequestHandler.streamIds

Repository: spark Updated Branches: refs/heads/master 1262e310c - 168603272 [SPARK-7183] [NETWORK] Fix memory leak of TransportRequestHandler.streamIds JIRA: https://issues.apache.org/jira/browse/SPARK-7183 Author: Liang-Chi Hsieh vii...@gmail.com Closes #5743 from

[3/5] spark git commit: [SPARK-3468] [WEBUI] Timeline-View feature

http://git-wip-us.apache.org/repos/asf/spark/blob/7fe0f3f2/core/src/main/resources/org/apache/spark/ui/static/vis.min.css -- diff --git a/core/src/main/resources/org/apache/spark/ui/static/vis.min.css

[1/5] spark git commit: [SPARK-3468] [WEBUI] Timeline-View feature

Repository: spark Updated Branches: refs/heads/master c24aeb6a3 - 7fe0f3f2b http://git-wip-us.apache.org/repos/asf/spark/blob/7fe0f3f2/core/src/main/scala/org/apache/spark/scheduler/StageInfo.scala -- diff --git

[4/5] spark git commit: [SPARK-3468] [WEBUI] Timeline-View feature

http://git-wip-us.apache.org/repos/asf/spark/blob/7fe0f3f2/core/src/main/resources/org/apache/spark/ui/static/vis.map -- diff --git a/core/src/main/resources/org/apache/spark/ui/static/vis.map

[5/5] spark git commit: [SPARK-3468] [WEBUI] Timeline-View feature

[SPARK-3468] [WEBUI] Timeline-View feature I sometimes trouble-shoot and analyse the cause of long time spending job. At the time, I find the stages which spends long time or fails, then I find the tasks which spends long time or fails, next I analyse the proportion of each phase in a task.

[2/5] spark git commit: [SPARK-3468] [WEBUI] Timeline-View feature

http://git-wip-us.apache.org/repos/asf/spark/blob/7fe0f3f2/core/src/main/resources/org/apache/spark/ui/static/vis.min.js -- diff --git a/core/src/main/resources/org/apache/spark/ui/static/vis.min.js

spark git commit: Ignore flakey test in SparkSubmitUtilsSuite

Repository: spark Updated Branches: refs/heads/master b1f4ca82d - 5c1fabafa Ignore flakey test in SparkSubmitUtilsSuite Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/5c1fabaf Tree: