[GitHub] spark pull request: [SPARK-3694] RDD and Task serialization debugg...

2014-12-12 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/3518#discussion_r21786759 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala --- @@ -458,6 +458,17 @@ private[spark] class TaskSetManager( v

[GitHub] spark pull request: [SPARK-3694] RDD and Task serialization debugg...

2014-12-12 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/3518#discussion_r21786754 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -873,6 +943,10 @@ class DAGScheduler( // We've already seriali

[GitHub] spark pull request: [SPARK-3694] RDD and Task serialization debugg...

2014-12-12 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/3518#discussion_r21786752 --- Diff: core/src/main/scala/org/apache/spark/util/SerializationHelper.scala --- @@ -0,0 +1,128 @@ +/* + * Licensed to the Apache Software Foundati

[GitHub] spark pull request: [SPARK-3694] RDD and Task serialization debugg...

2014-12-12 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/3518#discussion_r21786744 --- Diff: docs/configuration.md --- @@ -517,6 +517,14 @@ Apart from these, the following properties are also available, and may be useful

[GitHub] spark pull request: [SPARK-3694] RDD and Task serialization debugg...

2014-12-12 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/3518#issuecomment-66868725 IMO, this patch still needs a lot of work before it will be ready to merge. I'm not convinced that telling me which RDD referenced the unserializable object, by itself

[GitHub] spark pull request: [SPARK-4839] Adding a documentation about dyna...

2014-12-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3689#issuecomment-66868521 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your pro

[GitHub] spark pull request: [SPARK-4839] Adding a documentation about dyna...

2014-12-12 Thread oza
GitHub user oza opened a pull request: https://github.com/apache/spark/pull/3689 [SPARK-4839] Adding a documentation about dynamic resource allocation. Signed-off-by: Tsuyoshi Ozawa You can merge this pull request into a Git repository by running: $ git pull https://github.com

[GitHub] spark pull request: [SPARK-4006] Block Manager - Double Register C...

2014-12-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2854#issuecomment-66868340 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24

[GitHub] spark pull request: [SPARK-4006] Block Manager - Double Register C...

2014-12-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2854#issuecomment-66868337 [Test build #24433 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24433/consoleFull) for PR 2854 at commit [`95ae4db`](https://gith

[GitHub] spark pull request: [SPARK-3967] don’t redundantly overwrite exe...

2014-12-12 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2848#issuecomment-66868021 Hmm, maybe a legitimate test failure? ``` [info] - Distributing files locally security On *** FAILED *** (82 milliseconds) [info] java.io.FileNotFound

[GitHub] spark pull request: [SPARK-4006] Block Manager - Double Register C...

2014-12-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2854#issuecomment-66867662 [Test build #24433 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24433/consoleFull) for PR 2854 at commit [`95ae4db`](https://githu

[GitHub] spark pull request: [SPARK-4006] Block Manager - Double Register C...

2014-12-12 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/2854#issuecomment-66867614 another flaky test. retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-3431] [WIP] Parallelize test execution

2014-12-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3564#issuecomment-66866916 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24

[GitHub] spark pull request: [SPARK-3431] [WIP] Parallelize test execution

2014-12-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3564#issuecomment-66866915 [Test build #24432 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24432/consoleFull) for PR 3564 at commit [`79d3c38`](https://gith

[GitHub] spark pull request: [SPARK-1507][YARN]specify num of cores for AM

2014-12-12 Thread XuTingjun
Github user XuTingjun commented on the pull request: https://github.com/apache/spark/pull/3686#issuecomment-66866834 I have tested, it works in yarn-client and yarn-cluster mode. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark pull request: [SPARK-3431] [WIP] Parallelize test execution

2014-12-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3564#issuecomment-66866635 [Test build #24432 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24432/consoleFull) for PR 3564 at commit [`79d3c38`](https://githu

[GitHub] spark pull request: [SPARK-3431] [WIP] Parallelize test execution

2014-12-12 Thread nchammas
Github user nchammas commented on the pull request: https://github.com/apache/spark/pull/3564#issuecomment-66866615 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not hav

[GitHub] spark pull request: SPARK-4297 [BUILD] Build warning fixes omnibus

2014-12-12 Thread nchammas
Github user nchammas commented on the pull request: https://github.com/apache/spark/pull/3157#issuecomment-66866588 +1 on this kind of cleanup work. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-3967] don’t redundantly overwrite exe...

2014-12-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2848#issuecomment-66866422 [Test build #24431 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24431/consoleFull) for PR 2848 at commit [`bcf2543`](https://gith

[GitHub] spark pull request: [SPARK-3967] don’t redundantly overwrite exe...

2014-12-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2848#issuecomment-66866426 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24

[GitHub] spark pull request: [WIP] Remove many uses of Thread.sleep() from ...

2014-12-12 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/3687#issuecomment-66865631 /cc @tdas. This is a work-in-progress towards removing most of the Thread.sleep() calls. I'm making slow-but-steady progress; would love your feedback + any suggestio

[GitHub] spark pull request: [SPARK-4837] NettyBlockTransferService should ...

2014-12-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3688#issuecomment-66865449 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24

[GitHub] spark pull request: [SPARK-4837] NettyBlockTransferService should ...

2014-12-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3688#issuecomment-66865446 [Test build #24430 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24430/consoleFull) for PR 3688 at commit [`ebd2007`](https://gith

[GitHub] spark pull request: [SPARK-3967] don’t redundantly overwrite exe...

2014-12-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2848#issuecomment-66865189 [Test build #24431 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24431/consoleFull) for PR 2848 at commit [`bcf2543`](https://githu

[GitHub] spark pull request: [SPARK-3431] [WIP] Parallelize test execution

2014-12-12 Thread nchammas
Github user nchammas commented on the pull request: https://github.com/apache/spark/pull/3564#issuecomment-66865164 :+1: Thank you for getting to the bottom of these flaky tests. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark pull request: [SPARK-3967] don’t redundantly overwrite exe...

2014-12-12 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2848#issuecomment-66865120 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not ha

[GitHub] spark pull request: [SPARK-3431] [WIP] Parallelize test execution

2014-12-12 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/3564#issuecomment-66865097 I've opened #3687 to try to systematically address the causes of streaming test flakiness. In the process, I uncovered what I think is a new bug in streaming checkpoin

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-66864596 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-66864594 [Test build #24429 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24429/consoleFull) for PR 3607 at commit [`7fa9e2e`](https://gith

[GitHub] spark pull request: [SPARK-4837] NettyBlockTransferService should ...

2014-12-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3688#issuecomment-66864203 [Test build #24430 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24430/consoleFull) for PR 3688 at commit [`ebd2007`](https://githu

[GitHub] spark pull request: [SPARK-4837] NettyBlockTransferService should ...

2014-12-12 Thread aarondav
GitHub user aarondav opened a pull request: https://github.com/apache/spark/pull/3688 [SPARK-4837] NettyBlockTransferService should use spark.blockManager.port config This is used in NioBlockTransferService here: https://github.com/apache/spark/blob/master/core/src/main/scala/o

[GitHub] spark pull request: [SPARK-4006] Block Manager - Double Register C...

2014-12-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2854#issuecomment-66863703 [Test build #24428 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24428/consoleFull) for PR 2854 at commit [`95ae4db`](https://gith

[GitHub] spark pull request: [SPARK-4006] Block Manager - Double Register C...

2014-12-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2854#issuecomment-66863706 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24

[GitHub] spark pull request: [SPARK-4609] Blacklist hosts rather than execu...

2014-12-12 Thread suyanNone
Github user suyanNone commented on the pull request: https://github.com/apache/spark/pull/3541#issuecomment-66863621 To be frankly, I am fresh in spark. If have local fileSystem accessed errors, should we identify and mark as host-failed? may wrapAs ReadFileException or WriteFileE

[GitHub] spark pull request: [SPARK-4461][YARN] pass extra java options to ...

2014-12-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3409#issuecomment-66863203 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24

[GitHub] spark pull request: [SPARK-4461][YARN] pass extra java options to ...

2014-12-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3409#issuecomment-66863200 [Test build #24427 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24427/consoleFull) for PR 3409 at commit [`08f44a7`](https://gith

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-66863118 [Test build #24429 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24429/consoleFull) for PR 3607 at commit [`7fa9e2e`](https://githu

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-12 Thread WangTaoTheTonic
Github user WangTaoTheTonic commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-66863070 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have t

[GitHub] spark pull request: [SPARK-4006] Block Manager - Double Register C...

2014-12-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2854#issuecomment-66862989 [Test build #24428 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24428/consoleFull) for PR 2854 at commit [`95ae4db`](https://githu

[GitHub] spark pull request: [SPARK-4006] Block Manager - Double Register C...

2014-12-12 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/2854#issuecomment-66862967 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this f

[GitHub] spark pull request: [PySpark] Fix tests with Python 2.6 in 1.0 bra...

2014-12-12 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/3668#issuecomment-66862913 Nice! I'm merging this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have t

[GitHub] spark pull request: [SPARK-4527][SQl]Add BroadcastNestedLoopJoin o...

2014-12-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3395#issuecomment-66862821 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24

[GitHub] spark pull request: [SPARK-4527][SQl]Add BroadcastNestedLoopJoin o...

2014-12-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3395#issuecomment-66862819 [Test build #24425 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24425/consoleFull) for PR 3395 at commit [`ea0e495`](https://gith

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-66862423 [Test build #24426 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24426/consoleFull) for PR 3607 at commit [`7fa9e2e`](https://gith

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-66862427 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24

[GitHub] spark pull request: Remove many uses of Thread.sleep() from stream...

2014-12-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3687#issuecomment-66862227 [Test build #24424 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24424/consoleFull) for PR 3687 at commit [`ad0056b`](https://gith

[GitHub] spark pull request: Remove many uses of Thread.sleep() from stream...

2014-12-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3687#issuecomment-66862230 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24

[GitHub] spark pull request: [WIP][SPARK-4251][SPARK-2352][MLLIB]Add RBM, A...

2014-12-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3222#issuecomment-66861467 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24

[GitHub] spark pull request: [WIP][SPARK-4251][SPARK-2352][MLLIB]Add RBM, A...

2014-12-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3222#issuecomment-66861462 [Test build #24423 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24423/consoleFull) for PR 3222 at commit [`3980c8a`](https://gith

[GitHub] spark pull request: [WIP][SPARK-4251][SPARK-2352][MLLIB]Add RBM, A...

2014-12-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3222#issuecomment-66861201 [Test build #24422 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24422/consoleFull) for PR 3222 at commit [`0e81958`](https://gith

[GitHub] spark pull request: [WIP][SPARK-4251][SPARK-2352][MLLIB]Add RBM, A...

2014-12-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3222#issuecomment-66861205 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24

[GitHub] spark pull request: [SPARK-4461][YARN] pass extra java options to ...

2014-12-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3409#issuecomment-66860916 [Test build #24427 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24427/consoleFull) for PR 3409 at commit [`08f44a7`](https://githu

[GitHub] spark pull request: [SPARK-4461][YARN] pass extra java options to ...

2014-12-12 Thread zhzhan
Github user zhzhan commented on the pull request: https://github.com/apache/spark/pull/3409#issuecomment-66860869 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this featu

[GitHub] spark pull request: Remove many uses of Thread.sleep() from stream...

2014-12-12 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/3687#issuecomment-66860487 I expect this to fail due to https://issues.apache.org/jira/browse/SPARK-4835. --- If your project is set up for it, you can reply to this email and have your reply ap

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-12 Thread WangTaoTheTonic
Github user WangTaoTheTonic commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-66860231 @andrewor14 As `ClientArguments` didn't implement Log class before, I use println to warn. --- If your project is set up for it, you can reply to this email and

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-66860143 [Test build #24426 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24426/consoleFull) for PR 3607 at commit [`7fa9e2e`](https://githu

[GitHub] spark pull request: [SPARK-4527][SQl]Add BroadcastNestedLoopJoin o...

2014-12-12 Thread wangxiaojing
Github user wangxiaojing commented on the pull request: https://github.com/apache/spark/pull/3395#issuecomment-66860102 @liancheng Sorry for the delay in my styling issues, and thanks for your help. --- If your project is set up for it, you can reply to this email and have your rep

[GitHub] spark pull request: [SPARK-4527][SQl]Add BroadcastNestedLoopJoin o...

2014-12-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3395#issuecomment-66859945 [Test build #24425 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24425/consoleFull) for PR 3395 at commit [`ea0e495`](https://githu

[GitHub] spark pull request: Remove many uses of Thread.sleep() from stream...

2014-12-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3687#issuecomment-66859936 [Test build #24424 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24424/consoleFull) for PR 3687 at commit [`ad0056b`](https://githu

[GitHub] spark pull request: Remove many uses of Thread.sleep() from stream...

2014-12-12 Thread JoshRosen
GitHub user JoshRosen opened a pull request: https://github.com/apache/spark/pull/3687 Remove many uses of Thread.sleep() from streaming tests This is a work-in-progress PR towards removing many of the `Thread.sleep()` calls from Spark Streaming's test suite, since I think that thes

[GitHub] spark pull request: [SPARK-4461][YARN] pass extra java options to ...

2014-12-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3409#issuecomment-66859399 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24

[GitHub] spark pull request: [SPARK-4461][YARN] pass extra java options to ...

2014-12-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3409#issuecomment-66859390 [Test build #24421 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24421/consoleFull) for PR 3409 at commit [`08f44a7`](https://gith

[GitHub] spark pull request: [WIP][SPARK-4251][SPARK-2352][MLLIB]Add RBM, A...

2014-12-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3222#issuecomment-66858770 [Test build #24423 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24423/consoleFull) for PR 3222 at commit [`3980c8a`](https://githu

[GitHub] spark pull request: [WIP][SPARK-4251][SPARK-2352][MLLIB]Add RBM, A...

2014-12-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3222#issuecomment-66858513 [Test build #24422 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24422/consoleFull) for PR 3222 at commit [`0e81958`](https://githu

[GitHub] spark pull request: [SPARK-4618][SQL] Make foreign DDL commands op...

2014-12-12 Thread scwf
Github user scwf commented on the pull request: https://github.com/apache/spark/pull/3470#issuecomment-66858359 @marmbrus, updated, is this ok --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-4461][YARN] pass extra java options to ...

2014-12-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3409#issuecomment-66855785 [Test build #24421 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24421/consoleFull) for PR 3409 at commit [`08f44a7`](https://githu

[GitHub] spark pull request: [SPARK-4461][YARN] pass extra java options to ...

2014-12-12 Thread zhzhan
Github user zhzhan commented on the pull request: https://github.com/apache/spark/pull/3409#issuecomment-66855013 Warning is OK to me, but I am against throwing exception. Will update it with logWarning("spark.yarn.am.extraJavaOptions will not take effect in driver mode") --- If you

[GitHub] spark pull request: [SQL] SPARK-4700: Add HTTP protocol spark thri...

2014-12-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3672#issuecomment-66854192 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24

[GitHub] spark pull request: [SQL] SPARK-4700: Add HTTP protocol spark thri...

2014-12-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3672#issuecomment-66854184 [Test build #24420 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24420/consoleFull) for PR 3672 at commit [`31a6520`](https://gith

[GitHub] spark pull request: [SPARK-3431] [WIP] Parallelize test execution

2014-12-12 Thread nchammas
Github user nchammas commented on the pull request: https://github.com/apache/spark/pull/3564#issuecomment-66853045 FYI, [I've posted a question on Stack Overflow](http://stackoverflow.com/questions/27453882/how-can-run-tests-in-parallel-but-get-neatly-ordered-test-output) about the i

[GitHub] spark pull request: [SQL] SPARK-4700: Add HTTP protocol spark thri...

2014-12-12 Thread judynash
Github user judynash commented on the pull request: https://github.com/apache/spark/pull/3672#issuecomment-66843978 Fixed the styling issue and generated SQL Programming Guide doc. See screenshot http://i.imgur.com/l9F4mXq.png --- If your project is set up for it, you can reply to t

[GitHub] spark pull request: [SQL] SPARK-4700: Add HTTP protocol spark thri...

2014-12-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3672#issuecomment-66843487 [Test build #24420 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24420/consoleFull) for PR 3672 at commit [`31a6520`](https://githu

[GitHub] spark pull request: [SPARK-3431] [WIP] Parallelize test execution

2014-12-12 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/3564#issuecomment-66842034 A couple of those are [known flaky tests](https://issues.apache.org/jira/issues/?jql=labels%20%3D%20flaky-test). Specifically, it looks like you've run into [SPARK-48

[GitHub] spark pull request: [SPARK-3431] [WIP] Parallelize test execution

2014-12-12 Thread nchammas
Github user nchammas commented on the pull request: https://github.com/apache/spark/pull/3564#issuecomment-66841320 Hmm, [lots of failures](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24419/testReport/) in this latest test, mostly related to `streaming`, and one

[GitHub] spark pull request: [SPARK-3405] add subnet-id and vpc-id options ...

2014-12-12 Thread jontg
Github user jontg commented on the pull request: https://github.com/apache/spark/pull/2872#issuecomment-66840618 @tylerprete That might occur if your VPC is not set up to auto-assign DNS records. If you can, that is where I would suggest beginning an investigation. --- If your proje

[GitHub] spark pull request: [SPARK-3431] [WIP] Parallelize test execution

2014-12-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3564#issuecomment-66840421 [Test build #24419 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24419/consoleFull) for PR 3564 at commit [`79d3c38`](https://gith

[GitHub] spark pull request: [SPARK-3431] [WIP] Parallelize test execution

2014-12-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3564#issuecomment-66840435 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24

[GitHub] spark pull request: [SPARK-3431] [WIP] Parallelize test execution

2014-12-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3564#issuecomment-66838546 [Test build #24419 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24419/consoleFull) for PR 3564 at commit [`79d3c38`](https://githu

[GitHub] spark pull request: [SPARK-3431] [WIP] Parallelize test execution

2014-12-12 Thread shaneknapp
Github user shaneknapp commented on the pull request: https://github.com/apache/spark/pull/3564#issuecomment-66838179 thanks for the patience @nchammas ! :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-3431] [WIP] Parallelize test execution

2014-12-12 Thread nchammas
Github user nchammas commented on the pull request: https://github.com/apache/spark/pull/3564#issuecomment-66838119 Okie doke @shaneknapp. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-3431] [WIP] Parallelize test execution

2014-12-12 Thread nchammas
Github user nchammas commented on the pull request: https://github.com/apache/spark/pull/3564#issuecomment-66838083 Thanks for sending the update to `SparkSubmitSuite` @JoshRosen. I also made a change in 79d3c38fc7730d4c7f9753e7631d7d100df18c8f to how the forked JVMs buffer th

[GitHub] spark pull request: [SPARK-3431] [WIP] Parallelize test execution

2014-12-12 Thread shaneknapp
Github user shaneknapp commented on the pull request: https://github.com/apache/spark/pull/3564#issuecomment-66838097 jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-3431] [WIP] Parallelize test execution

2014-12-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3564#issuecomment-66838015 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24

[GitHub] spark pull request: [SPARK-3431] [WIP] Parallelize test execution

2014-12-12 Thread shaneknapp
Github user shaneknapp commented on the pull request: https://github.com/apache/spark/pull/3564#issuecomment-66837976 i'm gonna kill the 22418 build and retrigger. need to restart jenkins really quick. --- If your project is set up for it, you can reply to this email and have your r

[GitHub] spark pull request: [SQL] SPARK-4700: Add HTTP protocol spark thri...

2014-12-12 Thread judynash
Github user judynash commented on a diff in the pull request: https://github.com/apache/spark/pull/3672#discussion_r21774497 --- Diff: docs/sql-programming-guide.md --- @@ -938,6 +938,18 @@ Configuration of Hive is done by placing your `hive-site.xml` file in `conf/`. Yo

[GitHub] spark pull request: [SPARK-3431] [WIP] Parallelize test execution

2014-12-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3564#issuecomment-66837328 [Test build #24417 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24417/consoleFull) for PR 3564 at commit [`cc46a61`](https://gith

[GitHub] spark pull request: [SPARK-3431] [WIP] Parallelize test execution

2014-12-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3564#issuecomment-66837335 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24

[GitHub] spark pull request: [SPARK-3431] [WIP] Parallelize test execution

2014-12-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3564#issuecomment-66836113 [Test build #24418 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24418/consoleFull) for PR 3564 at commit [`79d3c38`](https://githu

[GitHub] spark pull request: [SPARK-3431] [WIP] Parallelize test execution

2014-12-12 Thread nchammas
Github user nchammas commented on the pull request: https://github.com/apache/spark/pull/3564#issuecomment-66835945 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not hav

[GitHub] spark pull request: [PySpark] Fix tests with Python 2.6 in 1.0 bra...

2014-12-12 Thread shaneknapp
Github user shaneknapp commented on the pull request: https://github.com/apache/spark/pull/3668#issuecomment-66835922 woot! :) On Fri, Dec 12, 2014 at 12:47 PM, UCB AMPLab wrote: > > Test PASSed. > Refer to this link for build results (access rights to CI serv

[GitHub] spark pull request: [SPARK-3431] [WIP] Parallelize test execution

2014-12-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3564#issuecomment-66835509 [Test build #24417 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24417/consoleFull) for PR 3564 at commit [`cc46a61`](https://githu

[GitHub] spark pull request: [SPARK-2261] Make event logger use a single fi...

2014-12-12 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/1222#issuecomment-66833819 Failures are in streaming tests, which are unrelated to this pr. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark pull request: [SPARK-2261] Make event logger use a single fi...

2014-12-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1222#issuecomment-66833605 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24

[GitHub] spark pull request: [SPARK-2261] Make event logger use a single fi...

2014-12-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1222#issuecomment-66833592 [Test build #24415 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24415/consoleFull) for PR 1222 at commit [`59c561c`](https://gith

[GitHub] spark pull request: [PySpark] Fix tests with Python 2.6 in 1.0 bra...

2014-12-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3668#issuecomment-66832642 [Test build #24414 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24414/consoleFull) for PR 3668 at commit [`b32583d`](https://gith

[GitHub] spark pull request: [PySpark] Fix tests with Python 2.6 in 1.0 bra...

2014-12-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3668#issuecomment-66832650 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24

[GitHub] spark pull request: Test parallelization WIP

2014-12-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3681#issuecomment-66830291 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24

[GitHub] spark pull request: Test parallelization WIP

2014-12-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3681#issuecomment-66830275 [Test build #24416 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24416/consoleFull) for PR 3681 at commit [`3513e84`](https://gith

[GitHub] spark pull request: [SPARK-4461][YARN] pass extra java options to ...

2014-12-12 Thread tgravescs
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/3409#issuecomment-66828213 LGTM. I can go either way on the warning. It might be useful just to print a single line warning stating its not going to be used. 1 log line on executor startup does

[GitHub] spark pull request: [SPARK-3431] [WIP] Parallelize test execution

2014-12-12 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/3564#issuecomment-66827468 The other two test failures _might_ be due to streaming flakiness, but I don't want to rule out the possibility that they could be legitimate failures. We'll know when

  1   2   3   >