spark git commit: [SPARK-4672][Core]Checkpoint() should clear f to shorten the serialization chain

2014-12-03 Thread ankurdave
Repository: spark Updated Branches: refs/heads/master 17c162f66 - 77be8b986 [SPARK-4672][Core]Checkpoint() should clear f to shorten the serialization chain The related JIRA is https://issues.apache.org/jira/browse/SPARK-4672 The f closure of `PartitionsRDD(ZippedPartitionsRDD2)` contains a

spark git commit: [SPARK-4672][Core]Checkpoint() should clear f to shorten the serialization chain

2014-12-03 Thread ankurdave
Repository: spark Updated Branches: refs/heads/branch-1.2 528cce8bc - 667f7ff44 [SPARK-4672][Core]Checkpoint() should clear f to shorten the serialization chain The related JIRA is https://issues.apache.org/jira/browse/SPARK-4672 The f closure of `PartitionsRDD(ZippedPartitionsRDD2)`

spark git commit: [SPARK-4397][Core] Change the 'since' value of '@deprecated' to '1.3.0'

2014-12-03 Thread rxin
Repository: spark Updated Branches: refs/heads/master 77be8b986 - 8af551f71 [SPARK-4397][Core] Change the 'since' value of '@deprecated' to '1.3.0' As #3262 wasn't merged to branch 1.2, the `since` value of `deprecated` should be '1.3.0'. Author: zsxwing zsxw...@gmail.com Closes #3573 from

spark git commit: [SPARK-4710] [mllib] Eliminate MLlib compilation warnings

2014-12-03 Thread meng
Repository: spark Updated Branches: refs/heads/master 8af551f71 - 4ac215115 [SPARK-4710] [mllib] Eliminate MLlib compilation warnings Renamed StreamingKMeans to StreamingKMeansExample to avoid warning about name conflict with StreamingKMeans class. Added import to DecisionTreeRunner to

spark git commit: [SPARK-4710] [mllib] Eliminate MLlib compilation warnings

2014-12-03 Thread meng
Repository: spark Updated Branches: refs/heads/branch-1.2 667f7ff44 - fb14bfdd9 [SPARK-4710] [mllib] Eliminate MLlib compilation warnings Renamed StreamingKMeans to StreamingKMeansExample to avoid warning about name conflict with StreamingKMeans class. Added import to DecisionTreeRunner to

spark git commit: [SPARK-4717][MLlib] Optimize BLAS library to avoid de-reference multiple times in loop

2014-12-03 Thread meng
Repository: spark Updated Branches: refs/heads/branch-1.2 8ff7a286d - b63e94175 [SPARK-4717][MLlib] Optimize BLAS library to avoid de-reference multiple times in loop Have a local reference to `values` and `indices` array in the `Vector` object so JVM can locate the value with one operation

spark git commit: [SPARK-4717][MLlib] Optimize BLAS library to avoid de-reference multiple times in loop

2014-12-03 Thread meng
Repository: spark Updated Branches: refs/heads/master 7fc49ed91 - d00542987 [SPARK-4717][MLlib] Optimize BLAS library to avoid de-reference multiple times in loop Have a local reference to `values` and `indices` array in the `Vector` object so JVM can locate the value with one operation

spark git commit: SPARK-2624 add datanucleus jars to the container in yarn-cluster

2014-12-03 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master d00542987 - a975dc327 SPARK-2624 add datanucleus jars to the container in yarn-cluster If `spark-submit` finds the datanucleus jars, it adds them to the driver's classpath, but does not add it to the container. This patch modifies the

spark git commit: SPARK-2624 add datanucleus jars to the container in yarn-cluster

2014-12-03 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.2 b63e94175 - 163fd785a SPARK-2624 add datanucleus jars to the container in yarn-cluster If `spark-submit` finds the datanucleus jars, it adds them to the driver's classpath, but does not add it to the container. This patch modifies

spark git commit: [SPARK-4701] Typo in sbt/sbt

2014-12-03 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.1 aec20af1f - e484b8af2 [SPARK-4701] Typo in sbt/sbt Modified typo. Author: Masayoshi TSUZUKI tsudu...@oss.nttdata.co.jp Closes #3560 from tsudukim/feature/SPARK-4701 and squashes the following commits: ed2a3f1 [Masayoshi TSUZUKI]

spark git commit: [SPARK-4701] Typo in sbt/sbt

2014-12-03 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master a975dc327 - 96786e3ee [SPARK-4701] Typo in sbt/sbt Modified typo. Author: Masayoshi TSUZUKI tsudu...@oss.nttdata.co.jp Closes #3560 from tsudukim/feature/SPARK-4701 and squashes the following commits: ed2a3f1 [Masayoshi TSUZUKI]

spark git commit: [SPARK-4715][Core] Make sure tryToAcquire won't return a negative value

2014-12-03 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.2 614e68636 - 1ee65b4f9 [SPARK-4715][Core] Make sure tryToAcquire won't return a negative value ShuffleMemoryManager.tryToAcquire may return a negative value. The unit test demonstrates this bug. It will output `0 did not equal -200

spark git commit: [SPARK-4715][Core] Make sure tryToAcquire won't return a negative value

2014-12-03 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.1 e484b8af2 - af7695456 [SPARK-4715][Core] Make sure tryToAcquire won't return a negative value ShuffleMemoryManager.tryToAcquire may return a negative value. The unit test demonstrates this bug. It will output `0 did not equal -200

spark git commit: [SPARK-4642] Add description about spark.yarn.queue to running-on-YARN document.

2014-12-03 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master edd3cd477 - 692f49378 [SPARK-4642] Add description about spark.yarn.queue to running-on-YARN document. Added descriptions about these parameters. - spark.yarn.queue Modified description about the defalut value of this parameter. -

spark git commit: [SPARK-4642] Add description about spark.yarn.queue to running-on-YARN document.

2014-12-03 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.2 1ee65b4f9 - 4a71e0853 [SPARK-4642] Add description about spark.yarn.queue to running-on-YARN document. Added descriptions about these parameters. - spark.yarn.queue Modified description about the defalut value of this parameter. -

spark git commit: [SPARK-4642] Add description about spark.yarn.queue to running-on-YARN document.

2014-12-03 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.1 af7695456 - 3e3cd5ac8 [SPARK-4642] Add description about spark.yarn.queue to running-on-YARN document. Added descriptions about these parameters. - spark.yarn.queue Modified description about the defalut value of this parameter. -

spark git commit: [HOT FIX] [YARN] Check whether `/lib` exists before listing its files

2014-12-03 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.2 4a71e0853 - 38cb2c3a3 [HOT FIX] [YARN] Check whether `/lib` exists before listing its files This is caused by a975dc32799bb8a14f9e1c76defaaa7cfbaf8b53 Author: Andrew Or and...@databricks.com Closes #3589 from andrewor14/yarn-hot-fix

spark git commit: [SPARK-4552][SQL] Avoid exception when reading empty parquet data through Hive

2014-12-03 Thread marmbrus
Repository: spark Updated Branches: refs/heads/branch-1.2 38cb2c3a3 - 47931975e [SPARK-4552][SQL] Avoid exception when reading empty parquet data through Hive This is a very small fix that catches one specific exception and returns an empty table. #3441 will address this in a more

spark git commit: [SPARK-4498][core] Don't transition ExecutorInfo to RUNNING until Driver adds Executor

2014-12-03 Thread joshrosen
Repository: spark Updated Branches: refs/heads/master 513ef82e8 - 96b27855c [SPARK-4498][core] Don't transition ExecutorInfo to RUNNING until Driver adds Executor The ExecutorInfo only reaches the RUNNING state if the Driver is alive to send the ExecutorStateChanged message to master.

spark git commit: [SPARK-4498][core] Don't transition ExecutorInfo to RUNNING until Driver adds Executor

2014-12-03 Thread joshrosen
Repository: spark Updated Branches: refs/heads/branch-1.1 3e3cd5ac8 - 17dfd415f [SPARK-4498][core] Don't transition ExecutorInfo to RUNNING until Driver adds Executor The ExecutorInfo only reaches the RUNNING state if the Driver is alive to send the ExecutorStateChanged message to master.

spark git commit: [SPARK-4085] Propagate FetchFailedException when Spark fails to read local shuffle file.

2014-12-03 Thread pwendell
Repository: spark Updated Branches: refs/heads/master 96b27855c - 1826372d0 [SPARK-4085] Propagate FetchFailedException when Spark fails to read local shuffle file. cc aarondav kayousterhout pwendell This should go into 1.2? Author: Reynold Xin r...@databricks.com Closes #3579 from

spark git commit: [SPARK-4085] Propagate FetchFailedException when Spark fails to read local shuffle file.

2014-12-03 Thread pwendell
Repository: spark Updated Branches: refs/heads/branch-1.2 6b6b7791d - fe28ee2d1 [SPARK-4085] Propagate FetchFailedException when Spark fails to read local shuffle file. cc aarondav kayousterhout pwendell This should go into 1.2? Author: Reynold Xin r...@databricks.com Closes #3579 from

spark git commit: [SPARK-4711] [mllib] [docs] Programming guide advice on choosing optimizer

2014-12-03 Thread meng
Repository: spark Updated Branches: refs/heads/master 1826372d0 - 27ab0b8a0 [SPARK-4711] [mllib] [docs] Programming guide advice on choosing optimizer I have heard requests for the docs to include advice about choosing an optimization method. The programming guide could include a brief

spark git commit: [SPARK-4711] [mllib] [docs] Programming guide advice on choosing optimizer

2014-12-03 Thread meng
Repository: spark Updated Branches: refs/heads/branch-1.2 fe28ee2d1 - 4259ca8dd [SPARK-4711] [mllib] [docs] Programming guide advice on choosing optimizer I have heard requests for the docs to include advice about choosing an optimization method. The programming guide could include a brief

[2/2] spark git commit: [SPARK-4580] [SPARK-4610] [mllib] [docs] Documentation for tree ensembles + DecisionTree API fix

2014-12-03 Thread meng
[SPARK-4580] [SPARK-4610] [mllib] [docs] Documentation for tree ensembles + DecisionTree API fix Major changes: * Added programming guide sections for tree ensembles * Added examples for tree ensembles * Updated DecisionTree programming guide with more info on parameters * **API change**:

spark git commit: [Release] Correctly translate contributors name in release notes

2014-12-03 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 657a88835 - a4dfb4efe [Release] Correctly translate contributors name in release notes This commit involves three main changes: (1) It separates the translation of contributor names from the generation of the contributors list. This is

spark git commit: [Release] Correctly translate contributors name in release notes

2014-12-03 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.2 9880bb481 - f9e1f89b2 [Release] Correctly translate contributors name in release notes This commit involves three main changes: (1) It separates the translation of contributor names from the generation of the contributors list. This

spark git commit: MAINTENANCE: Automated closing of pull requests.

2014-12-03 Thread pwendell
Repository: spark Updated Branches: refs/heads/master a4dfb4efe - 3cdae038f MAINTENANCE: Automated closing of pull requests. This commit exists to close the following pull requests on Github: Closes #1875 (close requested by 'marmbrus') Closes #3566 (close requested by 'andrewor14') Closes