spark git commit: [CORE] [TEST] Fix SimpleDateParamTest

2015-05-26 Thread irashid
Repository: spark Updated Branches: refs/heads/master 43aa819c0 -> bf49c2213 [CORE] [TEST] Fix SimpleDateParamTest ``` sbt.ForkMain$ForkError: 1424424077190 was not equal to 1424474477190 at org.scalatest.MatchersHelper$.newTestFailedException(MatchersHelper.scala:160) at org

spark git commit: [CORE] [TEST] Fix SimpleDateParamTest

2015-05-26 Thread irashid
Repository: spark Updated Branches: refs/heads/branch-1.4 4b31a07b6 -> 79bb7dcec [CORE] [TEST] Fix SimpleDateParamTest ``` sbt.ForkMain$ForkError: 1424424077190 was not equal to 1424474477190 at org.scalatest.MatchersHelper$.newTestFailedException(MatchersHelper.scala:160) at

spark git commit: [SPARK-7339] [PYSPARK] PySpark shuffle spill memory sometimes are not correct

2015-05-26 Thread davies
Repository: spark Updated Branches: refs/heads/master bf49c2213 -> 8948ad3fb [SPARK-7339] [PYSPARK] PySpark shuffle spill memory sometimes are not correct In PySpark we get memory used before and after spill, then use the difference of these two value as memorySpilled, but if the before value

spark git commit: [SPARK-7339] [PYSPARK] PySpark shuffle spill memory sometimes are not correct

2015-05-26 Thread davies
Repository: spark Updated Branches: refs/heads/branch-1.4 79bb7dcec -> 25b2f95fe [SPARK-7339] [PYSPARK] PySpark shuffle spill memory sometimes are not correct In PySpark we get memory used before and after spill, then use the difference of these two value as memorySpilled, but if the before v

spark git commit: [SPARK-7806][EC2] Fixes that allow the spark_ec2.py tool to run with Python3

2015-05-26 Thread davies
Repository: spark Updated Branches: refs/heads/master 8948ad3fb -> 8dbe0 [SPARK-7806][EC2] Fixes that allow the spark_ec2.py tool to run with Python3 I have used this script to launch, destroy, start, and stop clusters successfully. Author: meawoppl Closes #6336 from meawoppl/py3ec2spa

spark git commit: [SPARK-7806][EC2] Fixes that allow the spark_ec2.py tool to run with Python3

2015-05-26 Thread davies
Repository: spark Updated Branches: refs/heads/branch-1.4 25b2f95fe -> 42070f096 [SPARK-7806][EC2] Fixes that allow the spark_ec2.py tool to run with Python3 I have used this script to launch, destroy, start, and stop clusters successfully. Author: meawoppl Closes #6336 from meawoppl/py3ec

spark git commit: [DOCS] [MLLIB] Fixing misformatted links in v1.4 MLlib Naive Bayes documentation by removing space and newline characters.

2015-05-26 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-1.4 42070f096 -> dfd905df5 [DOCS] [MLLIB] Fixing misformatted links in v1.4 MLlib Naive Bayes documentation by removing space and newline characters. A couple of links in the MLlib Naive Bayes documentation for v1.4 were broken due to the

spark git commit: [DOCS] [MLLIB] Fixing misformatted links in v1.4 MLlib Naive Bayes documentation by removing space and newline characters.

2015-05-26 Thread srowen
Repository: spark Updated Branches: refs/heads/master 8dbe0 -> e5a63a0e3 [DOCS] [MLLIB] Fixing misformatted links in v1.4 MLlib Naive Bayes documentation by removing space and newline characters. A couple of links in the MLlib Naive Bayes documentation for v1.4 were broken due to the add

spark git commit: [SPARK-7854] [TEST] refine Kryo test suite

2015-05-26 Thread srowen
Repository: spark Updated Branches: refs/heads/master e5a63a0e3 -> 63099122d [SPARK-7854] [TEST] refine Kryo test suite this modification is according to JoshRosen 's comments, for details, please refer to [#5934](https://github.com/apache/spark/pull/5934/files#r30949751). Author: Zhang, Liy

spark git commit: Revert "[SPARK-7042] [BUILD] use the standard akka artifacts with hadoop-2.x"

2015-05-26 Thread pwendell
Repository: spark Updated Branches: refs/heads/master 63099122d -> b7d808594 Revert "[SPARK-7042] [BUILD] use the standard akka artifacts with hadoop-2.x" This reverts commit 43aa819c041f6e8301ad1b8f82eb68e14254f636. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://g

spark git commit: [SPARK-7844] [MLLIB] Fix broken tests in KernelDensity

2015-05-26 Thread meng
Repository: spark Updated Branches: refs/heads/branch-1.4 dfd905df5 -> 51d98b0e9 [SPARK-7844] [MLLIB] Fix broken tests in KernelDensity The densities in KernelDensity are scaled down by (number of parallel processes X number of points). It should be just no.of samples. This results in broken

spark git commit: [SPARK-7844] [MLLIB] Fix broken tests in KernelDensity

2015-05-26 Thread meng
Repository: spark Updated Branches: refs/heads/master b7d808594 -> 61664732b [SPARK-7844] [MLLIB] Fix broken tests in KernelDensity The densities in KernelDensity are scaled down by (number of parallel processes X number of points). It should be just no.of samples. This results in broken test

spark git commit: [SPARK-3674] YARN support in Spark EC2

2015-05-26 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 61664732b -> 2e9a5f229 [SPARK-3674] YARN support in Spark EC2 This corresponds to https://github.com/mesos/spark-ec2/pull/116 in the spark-ec2 repo. The only changes required on the spark_ec2.py script is to open the RM port. cc andrewor

spark git commit: [SPARK-3674] YARN support in Spark EC2

2015-05-26 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.4 51d98b0e9 -> d014a447a [SPARK-3674] YARN support in Spark EC2 This corresponds to https://github.com/mesos/spark-ec2/pull/116 in the spark-ec2 repo. The only changes required on the spark_ec2.py script is to open the RM port. cc andr

spark git commit: [SPARK-6602] [CORE] Remove some places in core that calling SparkEnv.actorSystem

2015-05-26 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 2e9a5f229 -> 9f742241c [SPARK-6602] [CORE] Remove some places in core that calling SparkEnv.actorSystem Author: zsxwing Closes #6333 from zsxwing/remove-actor-system-usage and squashes the following commits: f125aa6 [zsxwing] Fix YarnAl

spark git commit: [SPARK-7748] [MLLIB] Graduate spark.ml from alpha

2015-05-26 Thread meng
Repository: spark Updated Branches: refs/heads/master 9f742241c -> 836a75898 [SPARK-7748] [MLLIB] Graduate spark.ml from alpha With descent coverage of feature transformers, algorithms, and model tuning support, it is time to graduate `spark.ml` from alpha. This PR changes all `AlphaComponen

spark git commit: [SPARK-7748] [MLLIB] Graduate spark.ml from alpha

2015-05-26 Thread meng
Repository: spark Updated Branches: refs/heads/branch-1.4 d014a447a -> b5ee7eefd [SPARK-7748] [MLLIB] Graduate spark.ml from alpha With descent coverage of feature transformers, algorithms, and model tuning support, it is time to graduate `spark.ml` from alpha. This PR changes all `AlphaComp

spark git commit: [SPARK-7864] [UI] Do not kill innocent stages from visualization

2015-05-26 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.4 b5ee7eefd -> f9dfa4d0f [SPARK-7864] [UI] Do not kill innocent stages from visualization **Reproduction.** Run a long-running job, go to the job page, expand the DAG visualization, and click into a stage. Your stage is now killed. Why?

spark git commit: [SPARK-7864] [UI] Do not kill innocent stages from visualization

2015-05-26 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 836a75898 -> 8f2082426 [SPARK-7864] [UI] Do not kill innocent stages from visualization **Reproduction.** Run a long-running job, go to the job page, expand the DAG visualization, and click into a stage. Your stage is now killed. Why? This

spark git commit: [SPARK-7883] [DOCS] [MLLIB] Fixing broken trainImplicit Scala example in MLlib Collaborative Filtering documentation.

2015-05-26 Thread meng
Repository: spark Updated Branches: refs/heads/master 8f2082426 -> 0463428b6 [SPARK-7883] [DOCS] [MLLIB] Fixing broken trainImplicit Scala example in MLlib Collaborative Filtering documentation. Fixing broken trainImplicit Scala example in MLlib Collaborative Filtering documentation to match

spark git commit: [SPARK-7883] [DOCS] [MLLIB] Fixing broken trainImplicit Scala example in MLlib Collaborative Filtering documentation.

2015-05-26 Thread meng
Repository: spark Updated Branches: refs/heads/branch-1.4 f9dfa4d0f -> 311fcf67e [SPARK-7883] [DOCS] [MLLIB] Fixing broken trainImplicit Scala example in MLlib Collaborative Filtering documentation. Fixing broken trainImplicit Scala example in MLlib Collaborative Filtering documentation to m

spark git commit: [SPARK-7883] [DOCS] [MLLIB] Fixing broken trainImplicit Scala example in MLlib Collaborative Filtering documentation.

2015-05-26 Thread meng
Repository: spark Updated Branches: refs/heads/branch-1.3 f26e38234 -> 68387e357 [SPARK-7883] [DOCS] [MLLIB] Fixing broken trainImplicit Scala example in MLlib Collaborative Filtering documentation. Fixing broken trainImplicit Scala example in MLlib Collaborative Filtering documentation to m

spark git commit: [SPARK-7883] [DOCS] [MLLIB] Fixing broken trainImplicit Scala example in MLlib Collaborative Filtering documentation.

2015-05-26 Thread meng
Repository: spark Updated Branches: refs/heads/branch-1.2 6c41e1cb9 -> d5763c3b9 [SPARK-7883] [DOCS] [MLLIB] Fixing broken trainImplicit Scala example in MLlib Collaborative Filtering documentation. Fixing broken trainImplicit Scala example in MLlib Collaborative Filtering documentation to m

spark git commit: [SPARK-7883] [DOCS] [MLLIB] Fixing broken trainImplicit Scala example in MLlib Collaborative Filtering documentation.

2015-05-26 Thread meng
Repository: spark Updated Branches: refs/heads/branch-1.0 0afb04250 -> 86ad12d44 [SPARK-7883] [DOCS] [MLLIB] Fixing broken trainImplicit Scala example in MLlib Collaborative Filtering documentation. Fixing broken trainImplicit Scala example in MLlib Collaborative Filtering documentation to m

spark git commit: [SPARK-7883] [DOCS] [MLLIB] Fixing broken trainImplicit Scala example in MLlib Collaborative Filtering documentation.

2015-05-26 Thread meng
Repository: spark Updated Branches: refs/heads/branch-1.1 ee06e9271 -> 672f3228c [SPARK-7883] [DOCS] [MLLIB] Fixing broken trainImplicit Scala example in MLlib Collaborative Filtering documentation. Fixing broken trainImplicit Scala example in MLlib Collaborative Filtering documentation to m

spark git commit: [SPARK-7637] [SQL] O(N) merge implementation for StructType merge

2015-05-26 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master 0463428b6 -> 03668348e [SPARK-7637] [SQL] O(N) merge implementation for StructType merge Contribution is my original work and I license the work to the project under the projects open source license. Author: rowan Closes #6259 from rowa

spark git commit: [SPARK-7858] [SQL] Use output schema, not relation schema, for data source input conversion

2015-05-26 Thread yhuai
Repository: spark Updated Branches: refs/heads/master 03668348e -> 0c33c7b4a [SPARK-7858] [SQL] Use output schema, not relation schema, for data source input conversion In `DataSourceStrategy.createPhysicalRDD`, we use the relation schema as the target schema for converting incoming rows int

spark git commit: [SPARK-7858] [SQL] Use output schema, not relation schema, for data source input conversion

2015-05-26 Thread yhuai
Repository: spark Updated Branches: refs/heads/branch-1.4 311fcf67e -> faadbd4d9 [SPARK-7858] [SQL] Use output schema, not relation schema, for data source input conversion In `DataSourceStrategy.createPhysicalRDD`, we use the relation schema as the target schema for converting incoming rows

spark git commit: [SPARK-7868] [SQL] Ignores _temporary directories in HadoopFsRelation

2015-05-26 Thread yhuai
Repository: spark Updated Branches: refs/heads/master 0c33c7b4a -> b463e6d61 [SPARK-7868] [SQL] Ignores _temporary directories in HadoopFsRelation So that potential partial/corrupted data files left by failed tasks/jobs won't affect normal data scan. Author: Cheng Lian Closes #6411 from li

spark git commit: [SPARK-7868] [SQL] Ignores _temporary directories in HadoopFsRelation

2015-05-26 Thread yhuai
Repository: spark Updated Branches: refs/heads/branch-1.4 faadbd4d9 -> d0bd68ff8 [SPARK-7868] [SQL] Ignores _temporary directories in HadoopFsRelation So that potential partial/corrupted data files left by failed tasks/jobs won't affect normal data scan. Author: Cheng Lian Closes #6411 fro

spark git commit: [SPARK-7535] [.1] [MLLIB] minor changes to the pipeline API

2015-05-26 Thread meng
Repository: spark Updated Branches: refs/heads/branch-1.4 d0bd68ff8 -> 34e233f9c [SPARK-7535] [.1] [MLLIB] minor changes to the pipeline API 1. removed `Params.validateParams(extra)` 2. added `Evaluate.evaluate(dataset, paramPairs*)` 3. updated `RegressionEvaluator` doc jkbradley Author: Xia

spark git commit: [SPARK-7535] [.1] [MLLIB] minor changes to the pipeline API

2015-05-26 Thread meng
Repository: spark Updated Branches: refs/heads/master b463e6d61 -> a9f1c0c57 [SPARK-7535] [.1] [MLLIB] minor changes to the pipeline API 1. removed `Params.validateParams(extra)` 2. added `Evaluate.evaluate(dataset, paramPairs*)` 3. updated `RegressionEvaluator` doc jkbradley Author: Xiangru