[GitHub] spark pull request: [SPARK-3032][Shuffle] Fix key comparison integ...

2014-09-23 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2514#issuecomment-56629824 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20742/

[GitHub] spark pull request: [SPARK-3032][Shuffle] Fix key comparison integ...

2014-09-23 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2514#issuecomment-56629822 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20742/consoleFull) for PR 2514 at commit [`83acb38`](https://github.com/a

[GitHub] spark pull request: [SPARK-3675][SQL] Allow starting a JDBC server...

2014-09-23 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2515#issuecomment-56629604 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20744/consoleFull) for PR 2515 at commit [`7866fad`](https://github.com/ap

[GitHub] spark pull request: [SPARK-3675][SQL] Allow starting a JDBC server...

2014-09-23 Thread marmbrus
GitHub user marmbrus opened a pull request: https://github.com/apache/spark/pull/2515 [SPARK-3675][SQL] Allow starting a JDBC server on an existing context You can merge this pull request into a Git repository by running: $ git pull https://github.com/marmbrus/spark jdbcExisti

[GitHub] spark pull request: [SPARK-1853] Show Streaming application code c...

2014-09-23 Thread tdas
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/2464#issuecomment-56628837 @mubarak Thank you very much for this fix! Its finally merged! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark pull request: [SPARK-3615][Streaming]Fix Kafka unit test har...

2014-09-23 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/2483#discussion_r17955115 --- Diff: external/kafka/src/test/scala/org/apache/spark/streaming/kafka/KafkaStreamSuite.scala --- @@ -59,16 +58,35 @@ class KafkaStreamSuite extends TestSuite

[GitHub] spark pull request: [SPARK-3615][Streaming]Fix Kafka unit test har...

2014-09-23 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2483#issuecomment-56628168 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20743/consoleFull) for PR 2483 at commit [`863`](https://github.com/ap

[GitHub] spark pull request: [SPARK-3032][Shuffle] Fix key comparison integ...

2014-09-23 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2514#issuecomment-56628035 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20741/consoleFull) for PR 2514 at commit [`fa2a08f`](https://github.com/a

[GitHub] spark pull request: [SPARK-3032][Shuffle] Fix key comparison integ...

2014-09-23 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2514#issuecomment-56628042 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20741/

[GitHub] spark pull request: [SPARK-3615][Streaming]Fix Kafka unit test har...

2014-09-23 Thread tdas
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/2483#issuecomment-56627830 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-3032][Shuffle] Fix key comparison integ...

2014-09-23 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/2514#discussion_r17954749 --- Diff: core/src/test/scala/org/apache/spark/util/collection/ExternalSorterSuite.scala --- @@ -707,4 +707,53 @@ class ExternalSorterSuite extends FunSuit

[GitHub] spark pull request: [SPARK-3032][Shuffle] Fix key comparison integ...

2014-09-23 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/2514#discussion_r17954666 --- Diff: core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala --- @@ -152,7 +152,7 @@ private[spark] class ExternalSorter[K, V, C](

[GitHub] spark pull request: [SPARK-3032][Shuffle] Fix key comparison integ...

2014-09-23 Thread mateiz
Github user mateiz commented on a diff in the pull request: https://github.com/apache/spark/pull/2514#discussion_r17954612 --- Diff: core/src/test/scala/org/apache/spark/util/collection/ExternalSorterSuite.scala --- @@ -707,4 +707,53 @@ class ExternalSorterSuite extends FunSuite w

[GitHub] spark pull request: [SPARK-3032][Shuffle] Fix key comparison integ...

2014-09-23 Thread mateiz
Github user mateiz commented on a diff in the pull request: https://github.com/apache/spark/pull/2514#discussion_r17954596 --- Diff: core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala --- @@ -152,7 +152,7 @@ private[spark] class ExternalSorter[K, V, C](

[GitHub] spark pull request: [SPARK-3032][Shuffle] Fix key comparison integ...

2014-09-23 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2514#issuecomment-56625977 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20742/consoleFull) for PR 2514 at commit [`83acb38`](https://github.com/ap

[GitHub] spark pull request: [SPARK-3032][Shuffle] Fix key comparison integ...

2014-09-23 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/2514#discussion_r17954265 --- Diff: core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala --- @@ -152,7 +152,7 @@ private[spark] class ExternalSorter[K, V, C](

[GitHub] spark pull request: [SPARK-546] Add full outer join to RDD and DSt...

2014-09-23 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/1395#issuecomment-56624824 Go for it @tdas! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this featu

[GitHub] spark pull request: [SPARK-3032][Shuffle] Fix key comparison integ...

2014-09-23 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2514#issuecomment-56624542 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20741/consoleFull) for PR 2514 at commit [`fa2a08f`](https://github.com/ap

[GitHub] spark pull request: [SPARK-3032][Shuffle] Fix key comparison integ...

2014-09-23 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/2514#discussion_r17954025 --- Diff: core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala --- @@ -152,7 +152,7 @@ private[spark] class ExternalSorter[K, V, C](

[GitHub] spark pull request: [SPARK-3032][Shuffle] Fix key comparison integ...

2014-09-23 Thread jerryshao
GitHub user jerryshao opened a pull request: https://github.com/apache/spark/pull/2514 [SPARK-3032][Shuffle] Fix key comparison integer overflow introduced sorting exception Previous key comparison in `ExternalSorter` will get wrong sorting result or exception when key comparison o

[GitHub] spark pull request: stop, start and destroy require the EC2_REGION

2014-09-23 Thread jeffsteinmetz
Github user jeffsteinmetz commented on the pull request: https://github.com/apache/spark/pull/2473#issuecomment-56622767 "For example" fixed in PR --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-3007][SQL]Add Dynamic Partition support...

2014-09-23 Thread baishuo
Github user baishuo commented on the pull request: https://github.com/apache/spark/pull/2226#issuecomment-56621834 I think I should say thank you to @liancheng and @yhuai. During the communication with you, I had learned a lot :) --- If your project is set up for it, you can reply t

[GitHub] spark pull request: [SPARK-3531][SQL]select null from table would ...

2014-09-23 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2396#issuecomment-56621308 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20740/

[GitHub] spark pull request: [SPARK-3531][SQL]select null from table would ...

2014-09-23 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2396#issuecomment-56621305 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20740/consoleFull) for PR 2396 at commit [`2458229`](https://github.com/a

[GitHub] spark pull request: [SPARK-3531][SQL]select null from table would ...

2014-09-23 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2396#issuecomment-56620753 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20739/consoleFull) for PR 2396 at commit [`2458229`](https://github.com/a

[GitHub] spark pull request: [SPARK-3531][SQL]select null from table would ...

2014-09-23 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2396#issuecomment-56620755 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20739/

[GitHub] spark pull request: [SPARK-3407][SQL]Add Date type support

2014-09-23 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2344#issuecomment-56619771 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20737/consoleFull) for PR 2344 at commit [`be6a958`](https://github.com/a

[GitHub] spark pull request: [SPARK-3407][SQL]Add Date type support

2014-09-23 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2344#issuecomment-56619774 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20737/

[GitHub] spark pull request: [SPARK-3389] Add Converter for ease of Parquet...

2014-09-23 Thread mateiz
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/2256#issuecomment-56618758 It looks like we can merge it without a rebase. I'll wait to see whether Nick has any comments because he built this feature. --- If your project is set up for it, you ca

[GitHub] spark pull request: [SPARK-3531][SQL]select null from table would ...

2014-09-23 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2396#issuecomment-56618731 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20740/consoleFull) for PR 2396 at commit [`2458229`](https://github.com/ap

[GitHub] spark pull request: [SPARK-3293] yarn's web show "SUCCEEDED" when ...

2014-09-23 Thread mateiz
Github user mateiz commented on a diff in the pull request: https://github.com/apache/spark/pull/2311#discussion_r17951694 --- Diff: yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala --- @@ -91,7 +94,11 @@ private[spark] class ApplicationMaster(args:

[GitHub] spark pull request: [SPARK-3531][SQL]select null from table would ...

2014-09-23 Thread adrian-wang
Github user adrian-wang commented on the pull request: https://github.com/apache/spark/pull/2396#issuecomment-56618520 retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-3531][SQL]select null from table would ...

2014-09-23 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2396#issuecomment-56618344 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20738/

[GitHub] spark pull request: [SPARK-3531][SQL]select null from table would ...

2014-09-23 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2396#issuecomment-56618196 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20739/consoleFull) for PR 2396 at commit [`2458229`](https://github.com/ap

[GitHub] spark pull request: [SPARK-3293] yarn's web show "SUCCEEDED" when ...

2014-09-23 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/2311#discussion_r17951431 --- Diff: yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala --- @@ -91,7 +94,11 @@ private[spark] class ApplicationMaster(arg

[GitHub] spark pull request: [SPARK-3531][SQL]select null from table would ...

2014-09-23 Thread adrian-wang
Github user adrian-wang commented on a diff in the pull request: https://github.com/apache/spark/pull/2396#discussion_r17951394 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala --- @@ -246,6 +246,7 @@ object HiveMetastoreTypes extends RegexPars

[GitHub] spark pull request: [SPARK-1774] Respect SparkSubmit --jars on YAR...

2014-09-23 Thread uncleGen
Github user uncleGen commented on a diff in the pull request: https://github.com/apache/spark/pull/710#discussion_r17951094 --- Diff: core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala --- @@ -192,15 +236,17 @@ class SparkSubmitSuite extends FunSuite with ShouldMat

[GitHub] spark pull request: [SPARK-3407][SQL]Add Date type support

2014-09-23 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2344#issuecomment-56617051 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20737/consoleFull) for PR 2344 at commit [`be6a958`](https://github.com/ap

[GitHub] spark pull request: [SPARK-3407][SQL]Add Date type support

2014-09-23 Thread adrian-wang
Github user adrian-wang commented on the pull request: https://github.com/apache/spark/pull/2344#issuecomment-56616986 @marmbrus , I have updated on current master and run sbt tests locally. --- If your project is set up for it, you can reply to this email and have your reply appear o

[GitHub] spark pull request: [Build] Diff from branch point

2014-09-23 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2512#issuecomment-56616906 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20736/consoleFull) for PR 2512 at commit [`c065599`](https://github.com/a

[GitHub] spark pull request: [Build] Diff from branch point

2014-09-23 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2512#issuecomment-56616907 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20736/

[GitHub] spark pull request: [SPARK-3615][Streaming]Fix Kafka unit test har...

2014-09-23 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/2483#discussion_r17950810 --- Diff: external/kafka/src/test/scala/org/apache/spark/streaming/kafka/KafkaStreamSuite.scala --- @@ -59,16 +58,35 @@ class KafkaStreamSuite extends Test

[GitHub] spark pull request: [SPARK-3007][SQL]Add Dynamic Partition support...

2014-09-23 Thread baishuo
Github user baishuo commented on the pull request: https://github.com/apache/spark/pull/2226#issuecomment-56616245 had remove "s from title @marmbrus --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does n

[GitHub] spark pull request: [SPARK-3645][SQL] Makes table caching eager by...

2014-09-23 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2513#issuecomment-56615126 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20735/

[GitHub] spark pull request: [SPARK-3645][SQL] Makes table caching eager by...

2014-09-23 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2513#issuecomment-56615122 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20735/consoleFull) for PR 2513 at commit [`b72e24e`](https://github.com/a

[GitHub] spark pull request: SPARK-3642. Document the nuances of shared var...

2014-09-23 Thread CodingCat
Github user CodingCat commented on a diff in the pull request: https://github.com/apache/spark/pull/2490#discussion_r17949828 --- Diff: docs/programming-guide.md --- @@ -1183,6 +1188,10 @@ running on the cluster can then add to it using the `add` method or the `+=` ope However

[GitHub] spark pull request: [Build] Diff from branch point

2014-09-23 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2512#issuecomment-56612753 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20736/consoleFull) for PR 2512 at commit [`c065599`](https://github.com/ap

[GitHub] spark pull request: [Build] Diff from branch point

2014-09-23 Thread nchammas
Github user nchammas commented on a diff in the pull request: https://github.com/apache/spark/pull/2512#discussion_r17949248 --- Diff: dev/run-tests-jenkins --- @@ -92,13 +92,13 @@ function post_message () { merge_note=" * This patch merges cleanly." source_f

[GitHub] spark pull request: [Build] Diff from branch point

2014-09-23 Thread ash211
Github user ash211 commented on a diff in the pull request: https://github.com/apache/spark/pull/2512#discussion_r17949026 --- Diff: dev/run-tests-jenkins --- @@ -92,13 +92,13 @@ function post_message () { merge_note=" * This patch merges cleanly." source_fil

[GitHub] spark pull request: [SPARK-3645][SQL] Makes table caching eager by...

2014-09-23 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2513#issuecomment-56611828 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20735/consoleFull) for PR 2513 at commit [`b72e24e`](https://github.com/ap

[GitHub] spark pull request: [SPARK-3645][SQL] Makes table caching eager by...

2014-09-23 Thread liancheng
GitHub user liancheng opened a pull request: https://github.com/apache/spark/pull/2513 [SPARK-3645][SQL] Makes table caching eager by default and adds syntax for lazy caching Although lazy caching for in-memory table seems consistent with the `RDD.cache()` API, it's relatively conf

[GitHub] spark pull request: [SPARK-2365] Add IndexedRDD, an efficient upda...

2014-09-23 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1297#issuecomment-56610747 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20734/consoleFull) for PR 1297 at commit [`1c864cd`](https://github.com/a

[GitHub] spark pull request: [SPARK-2365] Add IndexedRDD, an efficient upda...

2014-09-23 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1297#issuecomment-56610754 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20734/

[GitHub] spark pull request: [SPARK-3550][MLLIB] Disable automatic rdd cach...

2014-09-23 Thread davies
Github user davies commented on the pull request: https://github.com/apache/spark/pull/2412#issuecomment-56610305 @staple could you rebase this PR? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: add a util method for changing the log level w...

2014-09-23 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2433#issuecomment-56609697 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20732/consoleFull) for PR 2433 at commit [`03ed4f9`](https://github.com/a

[GitHub] spark pull request: [SPARK-2778] [yarn] Add yarn integration tests...

2014-09-23 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2257#issuecomment-56609714 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20733/consoleFull) for PR 2257 at commit [`5c2b56f`](https://github.com/a

[GitHub] spark pull request: add a util method for changing the log level w...

2014-09-23 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2433#issuecomment-56609700 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20732/

[GitHub] spark pull request: [SPARK-2778] [yarn] Add yarn integration tests...

2014-09-23 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2257#issuecomment-56609722 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20733/

[GitHub] spark pull request: [SPARK-3293] yarn's web show "SUCCEEDED" when ...

2014-09-23 Thread mateiz
Github user mateiz commented on a diff in the pull request: https://github.com/apache/spark/pull/2311#discussion_r17948054 --- Diff: yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala --- @@ -91,7 +94,11 @@ private[spark] class ApplicationMaster(args:

[GitHub] spark pull request: [STREAMING] Add redis pub/sub streaming suppor...

2014-09-23 Thread hayesgm
Github user hayesgm commented on the pull request: https://github.com/apache/spark/pull/2348#issuecomment-56608641 @tdas Thanks for looking into this. Let me know how you would like to proceed. --- If your project is set up for it, you can reply to this email and have your reply app

[GitHub] spark pull request: [SPARK-3293] yarn's web show "SUCCEEDED" when ...

2014-09-23 Thread mateiz
Github user mateiz commented on a diff in the pull request: https://github.com/apache/spark/pull/2311#discussion_r17946154 --- Diff: yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala --- @@ -91,7 +94,11 @@ private[spark] class ApplicationMaster(args:

[GitHub] spark pull request: [SPARK-3293] yarn's web show "SUCCEEDED" when ...

2014-09-23 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/2311#discussion_r17946116 --- Diff: yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala --- @@ -91,7 +94,11 @@ private[spark] class ApplicationMaster(args:

[GitHub] spark pull request: [SPARK-3293] yarn's web show "SUCCEEDED" when ...

2014-09-23 Thread mateiz
Github user mateiz commented on a diff in the pull request: https://github.com/apache/spark/pull/2311#discussion_r17946079 --- Diff: yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala --- @@ -91,7 +94,11 @@ private[spark] class ApplicationMaster(args:

[GitHub] spark pull request: [SPARK-3478] [PySpark] Profile the Python task...

2014-09-23 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2351#issuecomment-56605482 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20730/consoleFull) for PR 2351 at commit [`fb9565b`](https://github.com/a

[GitHub] spark pull request: [SPARK-3478] [PySpark] Profile the Python task...

2014-09-23 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2351#issuecomment-56605488 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20730/

[GitHub] spark pull request: [SPARK-2778] [yarn] Add yarn integration tests...

2014-09-23 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2257#issuecomment-56605345 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20733/consoleFull) for PR 2257 at commit [`5c2b56f`](https://github.com/ap

[GitHub] spark pull request: [SPARK-2365] Add IndexedRDD, an efficient upda...

2014-09-23 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1297#issuecomment-56605365 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20734/consoleFull) for PR 1297 at commit [`1c864cd`](https://github.com/ap

[GitHub] spark pull request: add a util method for changing the log level w...

2014-09-23 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2433#issuecomment-56604522 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20732/consoleFull) for PR 2433 at commit [`03ed4f9`](https://github.com/ap

[GitHub] spark pull request: [SPARK-2365] Add IndexedRDD, an efficient upda...

2014-09-23 Thread ankurdave
Github user ankurdave commented on a diff in the pull request: https://github.com/apache/spark/pull/1297#discussion_r17945543 --- Diff: core/src/main/scala/org/apache/spark/rdd/IndexedRDDPartitionLike.scala --- @@ -0,0 +1,426 @@ +/* + * Licensed to the Apache Software Found

[GitHub] spark pull request: [SPARK-2778] [yarn] Add yarn integration tests...

2014-09-23 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/2257#issuecomment-56604168 Having everything in the same file is ok. I think the trick in this case is to convince the child processes launched by Yarn to not use the common log4j configuration, and

[GitHub] spark pull request: [SPARK-2365] Add IndexedRDD, an efficient upda...

2014-09-23 Thread ankurdave
Github user ankurdave commented on a diff in the pull request: https://github.com/apache/spark/pull/1297#discussion_r17945448 --- Diff: core/src/main/scala/org/apache/spark/rdd/IndexedRDDPartitionLike.scala --- @@ -0,0 +1,426 @@ +/* + * Licensed to the Apache Software Found

[GitHub] spark pull request: [SPARK-2778] [yarn] Add yarn integration tests...

2014-09-23 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/2257#issuecomment-56603473 I think you have to SSH into the machines to get them, but even if you have them they're kinda jumbled because we log everything to the same file. Actually here we mig

[GitHub] spark pull request: [SPARK-3614][MLLIB] Add minimumOccurence filte...

2014-09-23 Thread mengxr
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/2494#issuecomment-56602599 @rnowling let's retry :) test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-2778] [yarn] Add yarn integration tests...

2014-09-23 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/2257#issuecomment-56601784 Jenkins, retest this please. Is there any way to access `unit-tests.log` from jenkins? --- If your project is set up for it, you can reply to this email and have

[GitHub] spark pull request: [SPARK-3562]Periodic cleanup event logs

2014-09-23 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/2471#issuecomment-56600443 I think there's a very unlikely race in your code: it's possible, if things are messed up just right, that the reader thread might try to read a log file that is being del

[GitHub] spark pull request: [SPARK-3562]Periodic cleanup event logs

2014-09-23 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/2471#discussion_r17943824 --- Diff: docs/monitoring.md --- @@ -135,6 +135,29 @@ follows: spark.ui.view.acls when the application was run will also have authorization

[GitHub] spark pull request: [SPARK-2778] [yarn] Add yarn integration tests...

2014-09-23 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2257#issuecomment-56600035 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20731/

[GitHub] spark pull request: [SPARK-3562]Periodic cleanup event logs

2014-09-23 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/2471#discussion_r17943751 --- Diff: docs/monitoring.md --- @@ -135,6 +135,29 @@ follows: spark.ui.view.acls when the application was run will also have authorization

[GitHub] spark pull request: [SPARK-3562]Periodic cleanup event logs

2014-09-23 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/2471#discussion_r17943689 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala --- @@ -214,6 +252,32 @@ private[history] class FsHistoryProvider(conf:

[GitHub] spark pull request: [SPARK-3562]Periodic cleanup event logs

2014-09-23 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/2471#discussion_r17943683 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala --- @@ -214,6 +252,32 @@ private[history] class FsHistoryProvider(conf:

[GitHub] spark pull request: [SPARK-3562]Periodic cleanup event logs

2014-09-23 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/2471#discussion_r17943673 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala --- @@ -100,6 +132,12 @@ private[history] class FsHistoryProvider(conf:

[GitHub] spark pull request: [SPARK-3562]Periodic cleanup event logs

2014-09-23 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/2471#discussion_r17943624 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala --- @@ -100,6 +132,12 @@ private[history] class FsHistoryProvider(conf:

[GitHub] spark pull request: [SPARK-3562]Periodic cleanup event logs

2014-09-23 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/2471#discussion_r17943665 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala --- @@ -100,6 +132,12 @@ private[history] class FsHistoryProvider(conf:

[GitHub] spark pull request: [SPARK-3562]Periodic cleanup event logs

2014-09-23 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/2471#discussion_r17943630 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala --- @@ -100,6 +132,12 @@ private[history] class FsHistoryProvider(conf:

[GitHub] spark pull request: [SPARK-2778] [yarn] Add yarn integration tests...

2014-09-23 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2257#issuecomment-56599622 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20729/consoleFull) for PR 2257 at commit [`67f5b02`](https://github.com/a

[GitHub] spark pull request: [SPARK-3562]Periodic cleanup event logs

2014-09-23 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/2471#discussion_r17943561 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala --- @@ -34,10 +34,20 @@ private[history] class FsHistoryProvider(conf: S

[GitHub] spark pull request: [SPARK-2778] [yarn] Add yarn integration tests...

2014-09-23 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2257#issuecomment-56599628 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20729/

[GitHub] spark pull request: [SPARK-3562]Periodic cleanup event logs

2014-09-23 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/2471#discussion_r17943541 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala --- @@ -34,10 +34,20 @@ private[history] class FsHistoryProvider(conf: S

[GitHub] spark pull request: [SPARK-1545] [mllib] Add Random Forests

2014-09-23 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/2435#discussion_r17943480 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/tree/model/RandomForestModel.scala --- @@ -0,0 +1,106 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-1545] [mllib] Add Random Forests

2014-09-23 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/2435#discussion_r17943465 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/tree/impl/DTStatsAggregator.scala --- @@ -189,6 +160,230 @@ private[tree] class DTStatsAggregator(

[GitHub] spark pull request: [SPARK-1545] [mllib] Add Random Forests

2014-09-23 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/2435#discussion_r17943501 --- Diff: mllib/src/test/scala/org/apache/spark/mllib/tree/RandomForestSuite.scala --- @@ -0,0 +1,221 @@ +/* + * Licensed to the Apache Software Founda

[GitHub] spark pull request: [SPARK-1545] [mllib] Add Random Forests

2014-09-23 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/2435#discussion_r17943493 --- Diff: mllib/src/test/scala/org/apache/spark/mllib/tree/RandomForestSuite.scala --- @@ -0,0 +1,221 @@ +/* + * Licensed to the Apache Software Founda

[GitHub] spark pull request: [SPARK-1545] [mllib] Add Random Forests

2014-09-23 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/2435#discussion_r17943476 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/tree/model/RandomForestModel.scala --- @@ -0,0 +1,106 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-1545] [mllib] Add Random Forests

2014-09-23 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/2435#discussion_r17943471 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/tree/impl/DecisionTreeMetadata.scala --- @@ -128,13 +139,34 @@ private[tree] object DecisionTreeMetadata

[GitHub] spark pull request: [SPARK-1545] [mllib] Add Random Forests

2014-09-23 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/2435#discussion_r17943458 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/tree/impl/DTStatsAggregator.scala --- @@ -189,6 +160,230 @@ private[tree] class DTStatsAggregator(

[GitHub] spark pull request: [SPARK-1545] [mllib] Add Random Forests

2014-09-23 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/2435#discussion_r17943451 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/tree/impl/DTStatsAggregator.scala --- @@ -189,6 +160,230 @@ private[tree] class DTStatsAggregator(

[GitHub] spark pull request: [SPARK-1545] [mllib] Add Random Forests

2014-09-23 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/2435#discussion_r17943440 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/tree/impl/BaggedPoint.scala --- @@ -0,0 +1,80 @@ +/* + * Licensed to the Apache Software Foundati

[GitHub] spark pull request: [SPARK-1545] [mllib] Add Random Forests

2014-09-23 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/2435#discussion_r17943453 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/tree/impl/DTStatsAggregator.scala --- @@ -189,6 +160,230 @@ private[tree] class DTStatsAggregator(

[GitHub] spark pull request: [SPARK-1545] [mllib] Add Random Forests

2014-09-23 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/2435#discussion_r17943444 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/tree/impl/BaggedPoint.scala --- @@ -0,0 +1,80 @@ +/* + * Licensed to the Apache Software Foundati

[GitHub] spark pull request: [SPARK-1545] [mllib] Add Random Forests

2014-09-23 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/2435#discussion_r17943438 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/tree/impl/BaggedPoint.scala --- @@ -0,0 +1,80 @@ +/* + * Licensed to the Apache Software Foundati

  1   2   3   4   >