[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user squito commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-159682175 (or https://github.com/apache/spark/pull/9941, which is just the bug fixes w/out the other changes ...) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user squito commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-159676268 Hi @JoshRosen , I was just looking into this. With some local testing, I could confirm that this patch did lead to those failures, but that it is fixed by https://github.com/apache/spark/pull/9923 (in particular the handling of an empty stream in `TransportResponseHandler`) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-159077566 Merging to master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user vanzin closed the pull request at: https://github.com/apache/spark/pull/9530 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-159187716 The following two tests have failed for the past three builds in the Hadoop 2.4 Maven Jenkins job: - org.apache.spark.deploy.yarn.YarnClusterSuite.run Spark in yarn-client mode - org.apache.spark.deploy.yarn.YarnClusterSuite.user class path first in client mode According to Jenkins, these failures first started in https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-Maven-with-YARN/HADOOP_PROFILE=hadoop-2.4,label=spark-test/4223/ Given the set of patches in that changeset, my hunch is that this might have somehow been caused by either #9853 or #9530 (this patch). Branch 1.6 only includes #9853 and seems to be passing these tests, so I therefore believe that this patch may be the culprit (since it's only present in master and not 1.6). @vanzin, could you help me look into this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-159025832 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-159027545 **[Test build #46542 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46542/consoleFull)** for PR 9530 at commit [`c5a218d`](https://github.com/apache/spark/commit/c5a218d51c805cb7be8421eb88ed097d1e8dba3b). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-159061302 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46542/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-159061136 **[Test build #46542 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46542/consoleFull)** for PR 9530 at commit [`c5a218d`](https://github.com/apache/spark/commit/c5a218d51c805cb7be8421eb88ed097d1e8dba3b). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-159061299 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-158522687 LGTM. But let's just merge this one to master since it's too risky to add a new feature to 1.6 now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-158179685 **[Test build #46349 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46349/consoleFull)** for PR 9530 at commit [`0e8e4bb`](https://github.com/apache/spark/commit/0e8e4bb10605d218e8c763ac4fd66bf48113b794). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-158216954 **[Test build #46349 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46349/consoleFull)** for PR 9530 at commit [`0e8e4bb`](https://github.com/apache/spark/commit/0e8e4bb10605d218e8c763ac4fd66bf48113b794). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-158217046 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-158217049 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46349/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-158217284 sigh... retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-158220621 **[Test build #46361 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46361/consoleFull)** for PR 9530 at commit [`0e8e4bb`](https://github.com/apache/spark/commit/0e8e4bb10605d218e8c763ac4fd66bf48113b794). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-158281171 **[Test build #46388 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46388/consoleFull)** for PR 9530 at commit [`c5a218d`](https://github.com/apache/spark/commit/c5a218d51c805cb7be8421eb88ed097d1e8dba3b). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-158281223 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46388/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-158262437 **[Test build #46361 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46361/consoleFull)** for PR 9530 at commit [`0e8e4bb`](https://github.com/apache/spark/commit/0e8e4bb10605d218e8c763ac4fd66bf48113b794). * This patch **fails from timeout after a configured wait of \`250m\`**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-158262494 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-158262495 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46361/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-158266769 **[Test build #46388 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46388/consoleFull)** for PR 9530 at commit [`c5a218d`](https://github.com/apache/spark/commit/c5a218d51c805cb7be8421eb88ed097d1e8dba3b). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-158281222 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-157804525 > Maybe you can just submit more prs basing on previous ones I don't know of any clean way of doing that in github; they'd end up including all the code in this PR, which is rather ugly / sub-optimal. I can wait for the config change (I want to push it soon anyway), but I'm not super worried about conflicts. Those are generally not hard to resolve. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-157916377 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-157916378 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46265/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-157934860 **[Test build #46289 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46289/consoleFull)** for PR 9530 at commit [`cfd01bd`](https://github.com/apache/spark/commit/cfd01bddeeedc99b84e756afe97b9731b8e95fc6). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-157916276 **[Test build #46265 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46265/consoleFull)** for PR 9530 at commit [`7cc83e7`](https://github.com/apache/spark/commit/7cc83e73fc0223c716e73b1ec269c484159092f3). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_:\n * `class ALS(override val uid: String) extends Estimator[ALSModel] with ALSParams with Writable `\n --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-157892490 **[Test build #46265 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46265/consoleFull)** for PR 9530 at commit [`7cc83e7`](https://github.com/apache/spark/commit/7cc83e73fc0223c716e73b1ec269c484159092f3). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-157887620 **[Test build #46261 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46261/consoleFull)** for PR 9530 at commit [`d10fa97`](https://github.com/apache/spark/commit/d10fa97c8458b1ca20adfd8ea594c7cc66e99932). * This patch **fails to build**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_:\n * `class ALS(override val uid: String) extends Estimator[ALSModel] with ALSParams with Writable `\n --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-157887628 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-157887632 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46261/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-157952065 **[Test build #46289 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46289/consoleFull)** for PR 9530 at commit [`cfd01bd`](https://github.com/apache/spark/commit/cfd01bddeeedc99b84e756afe97b9731b8e95fc6). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-157952122 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-157952123 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46289/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-157885830 **[Test build #46261 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46261/consoleFull)** for PR 9530 at commit [`d10fa97`](https://github.com/apache/spark/commit/d10fa97c8458b1ca20adfd8ea594c7cc66e99932). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-157462338 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-157462342 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46098/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-157451685 pyspark tests, always flaky... retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-157456927 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46096/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-157456926 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-157457151 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-157457125 Let me take a look at `test_kafka_direct_stream_foreach_get_offsetRanges` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-157463352 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-157467759 **[Test build #46099 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46099/consoleFull)** for PR 9530 at commit [`908a051`](https://github.com/apache/spark/commit/908a05121da0a5957845d4451e3841e6faa71493). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-157536909 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46099/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-157536842 **[Test build #46099 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46099/consoleFull)** for PR 9530 at commit [`908a051`](https://github.com/apache/spark/commit/908a05121da0a5957845d4451e3841e6faa71493). * This patch **fails from timeout after a configured wait of \`250m\`**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-157536908 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-157483860 sasl encryption for file transfers (instead of having to configure ssl for the driver's file server). lower overhead in the driver. but nothing really critical. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-157477553 /cc @rxin is it too risky if adding this one to 1.6? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-157482127 Does it buy us anything to add this to 1.6? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-157587696 Alright I looked a bit closer at this. Since it is not mission critical and it is touching some mission critical part, I think it's best to not to merge this in 1.6. Let's merge this once 1.6 has an rc? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-157595580 I'm mostly thinking it'd make it harder if we found bugs that need to fix. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-157591720 What's the problem with merging to master? Isn't that why we have branch-1.6? I have changes built on top of this that I'm waiting to send for review. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-157598886 Do you mean bugs in this code? Or in 1.6? If in this code, master will eventually go through stabilization also, and at that point if there are bugs, they'll surface. Whether this code goes in now or later doesn't change much in that regard. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-157631673 There are still some network related code that's going into 1.6 (one is the config naming thing). I'm concerned they might conflict. Maybe you can just submit more prs basing on previous ones for reviewing? We should be pretty close to cutting a rc anyway. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-157258738 **[Test build #46041 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46041/consoleFull)** for PR 9530 at commit [`908a051`](https://github.com/apache/spark/commit/908a05121da0a5957845d4451e3841e6faa71493). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-157258806 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-157258810 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46041/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-157231537 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-157234008 **[Test build #46041 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46041/consoleFull)** for PR 9530 at commit [`908a051`](https://github.com/apache/spark/commit/908a05121da0a5957845d4451e3841e6faa71493). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-156583779 **[Test build #45904 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45904/consoleFull)** for PR 9530 at commit [`908a051`](https://github.com/apache/spark/commit/908a05121da0a5957845d4451e3841e6faa71493). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-156601039 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/45904/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-156601001 **[Test build #45904 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45904/consoleFull)** for PR 9530 at commit [`908a051`](https://github.com/apache/spark/commit/908a05121da0a5957845d4451e3841e6faa71493). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_:\n * `case class AppendColumns[T, U](`\n --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-156601037 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-155541099 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-155541164 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-155542223 **[Test build #45540 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45540/consoleFull)** for PR 9530 at commit [`a89e665`](https://github.com/apache/spark/commit/a89e6655e74209d9b89f5dcb2b1b8ed8e3399d42). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-155574831 **[Test build #45540 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45540/consoleFull)** for PR 9530 at commit [`a89e665`](https://github.com/apache/spark/commit/a89e6655e74209d9b89f5dcb2b1b8ed8e3399d42). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-155568326 **[Test build #45547 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45547/consoleFull)** for PR 9530 at commit [`71ac0cf`](https://github.com/apache/spark/commit/71ac0cfdb25d7bb29e180c704a8eac840c7fccd4). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-155574987 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/45540/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-155574985 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-155576989 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-155579680 **[Test build #45552 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45552/consoleFull)** for PR 9530 at commit [`11f61a8`](https://github.com/apache/spark/commit/11f61a88e4a9a1a1a17572c8579143fc856667d9). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/9530#discussion_r44455097
--- Diff: core/src/main/scala/org/apache/spark/rpc/netty/NettyRpcEnv.scala
---
@@ -302,6 +325,138 @@ private[netty] class NettyRpcEnv(
}
}
+ override def fileServer: RpcEnvFileServer = streamManager
+
+ override def openChannel(uri: String): ReadableByteChannel = {
+val parsedUri = new URI(uri)
+require(parsedUri.getHost() != null, "Host name must be defined.")
+require(parsedUri.getPort() > 0, "Port must be defined.")
+require(parsedUri.getPath() != null && parsedUri.getPath().nonEmpty,
"Path must be defined.")
+
+val pipe = Pipe.open()
+val source = new FileDownloadChannel(pipe.source())
+try {
+ val client = fileDownloadClient(parsedUri.getHost(),
parsedUri.getPort())
+ val callback = new FileDownloadCallback(pipe.sink(), source, client)
+ client.stream(parsedUri.getPath(), callback)
+} catch {
+ case e: Exception =>
+pipe.sink().close()
+source.close()
+throw e
+}
+
+source
+ }
+
+ private def fileDownloadClient(host: String, port: Int): TransportClient
= synchronized {
+if (stopped.get()) {
+ throw new IllegalStateException("RpcEnv already stopped.")
+}
+
+val address = RpcAddress(host, port)
+val client =
fileClients.get(address).filter(_.isActive()).getOrElse(newDownloadClient(address))
+
+// Tell the timeout handler this client is in use. This will prevent
the handler from
+// closing the client if the timeout even triggers before data starts
flowing for this
+// download.
+client.synchronized {
+ val timeoutHandler =
client.getChannel().pipeline().get(classOf[TimeoutHandler])
+ timeoutHandler.setInUse(true)
+
+ // After notifying the timeout handler, check that the client is
really active, and if not,
+ // create a new one.
+ if (client.isActive()) client else newDownloadClient(address)
+}
+ }
+
+ /**
+ * Create a new client and install a handler that will respond to
IdleStateEvent. The events
+ * are generated by the IdleStateHandler installed by the
TransportContext when creating
+ * clients, and the timeout value is controlled by the transport
configuration.
+ */
+ private def newDownloadClient(addr: RpcAddress): TransportClient = {
+val c = clientFactory.createUnmanagedClient(addr.host, addr.port)
+c.getChannel().pipeline().addLast("rpcEnvTimeoutHandler", new
TimeoutHandler(c))
+fileClients.put(addr, c)
+c
+ }
+
+ private class TimeoutHandler(client: TransportClient) extends
ChannelInboundHandlerAdapter {
+
+@volatile private var inUse = true
+
+def setInUse(inUse: Boolean): Unit = this.inUse = inUse
+
+override def userEventTriggered(ctx: ChannelHandlerContext, evt:
Object): Unit = {
+ client.synchronized {
+if (!inUse && evt.isInstanceOf[IdleStateEvent] &&
client.isActive()) {
+ logDebug(s"Closing transport client $client after idle timeout.")
+ val socketAddr =
client.getChannel().remoteAddress().asInstanceOf[InetSocketAddress]
+ val address = RpcAddress(socketAddr.getHostName(),
socketAddr.getPort())
+ ctx.close()
+ NettyRpcEnv.this.synchronized {
--- End diff --
The lock order here is in reverse order of `fileDownloadClient`. A
potential dead-lock issue?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/9530#discussion_r44465685
--- Diff: core/src/main/scala/org/apache/spark/rpc/netty/NettyRpcEnv.scala
---
@@ -302,6 +325,138 @@ private[netty] class NettyRpcEnv(
}
}
+ override def fileServer: RpcEnvFileServer = streamManager
+
+ override def openChannel(uri: String): ReadableByteChannel = {
+val parsedUri = new URI(uri)
+require(parsedUri.getHost() != null, "Host name must be defined.")
+require(parsedUri.getPort() > 0, "Port must be defined.")
+require(parsedUri.getPath() != null && parsedUri.getPath().nonEmpty,
"Path must be defined.")
+
+val pipe = Pipe.open()
+val source = new FileDownloadChannel(pipe.source())
+try {
+ val client = fileDownloadClient(parsedUri.getHost(),
parsedUri.getPort())
+ val callback = new FileDownloadCallback(pipe.sink(), source, client)
+ client.stream(parsedUri.getPath(), callback)
+} catch {
+ case e: Exception =>
+pipe.sink().close()
+source.close()
+throw e
+}
+
+source
+ }
+
+ private def fileDownloadClient(host: String, port: Int): TransportClient
= synchronized {
+if (stopped.get()) {
+ throw new IllegalStateException("RpcEnv already stopped.")
+}
+
+val address = RpcAddress(host, port)
+val client =
fileClients.get(address).filter(_.isActive()).getOrElse(newDownloadClient(address))
+
+// Tell the timeout handler this client is in use. This will prevent
the handler from
+// closing the client if the timeout even triggers before data starts
flowing for this
+// download.
+client.synchronized {
+ val timeoutHandler =
client.getChannel().pipeline().get(classOf[TimeoutHandler])
+ timeoutHandler.setInUse(true)
+
+ // After notifying the timeout handler, check that the client is
really active, and if not,
+ // create a new one.
+ if (client.isActive()) client else newDownloadClient(address)
+}
+ }
+
+ /**
+ * Create a new client and install a handler that will respond to
IdleStateEvent. The events
+ * are generated by the IdleStateHandler installed by the
TransportContext when creating
+ * clients, and the timeout value is controlled by the transport
configuration.
+ */
+ private def newDownloadClient(addr: RpcAddress): TransportClient = {
+val c = clientFactory.createUnmanagedClient(addr.host, addr.port)
+c.getChannel().pipeline().addLast("rpcEnvTimeoutHandler", new
TimeoutHandler(c))
+fileClients.put(addr, c)
+c
+ }
+
+ private class TimeoutHandler(client: TransportClient) extends
ChannelInboundHandlerAdapter {
+
+@volatile private var inUse = true
+
+def setInUse(inUse: Boolean): Unit = this.inUse = inUse
+
+override def userEventTriggered(ctx: ChannelHandlerContext, evt:
Object): Unit = {
+ client.synchronized {
+if (!inUse && evt.isInstanceOf[IdleStateEvent] &&
client.isActive()) {
+ logDebug(s"Closing transport client $client after idle timeout.")
+ val socketAddr =
client.getChannel().remoteAddress().asInstanceOf[InetSocketAddress]
+ val address = RpcAddress(socketAddr.getHostName(),
socketAddr.getPort())
+ ctx.close()
+ NettyRpcEnv.this.synchronized {
--- End diff --
Good catch, fixed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/9530#discussion_r44453574
--- Diff: core/src/main/scala/org/apache/spark/rpc/netty/NettyRpcEnv.scala
---
@@ -302,6 +325,138 @@ private[netty] class NettyRpcEnv(
}
}
+ override def fileServer: RpcEnvFileServer = streamManager
+
+ override def openChannel(uri: String): ReadableByteChannel = {
+val parsedUri = new URI(uri)
+require(parsedUri.getHost() != null, "Host name must be defined.")
+require(parsedUri.getPort() > 0, "Port must be defined.")
+require(parsedUri.getPath() != null && parsedUri.getPath().nonEmpty,
"Path must be defined.")
+
+val pipe = Pipe.open()
+val source = new FileDownloadChannel(pipe.source())
+try {
+ val client = fileDownloadClient(parsedUri.getHost(),
parsedUri.getPort())
+ val callback = new FileDownloadCallback(pipe.sink(), source, client)
+ client.stream(parsedUri.getPath(), callback)
+} catch {
+ case e: Exception =>
+pipe.sink().close()
+source.close()
+throw e
+}
+
+source
+ }
+
+ private def fileDownloadClient(host: String, port: Int): TransportClient
= synchronized {
--- End diff --
On second thought I think that's fine since downloading are still in
parallel.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-155567266 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-155567361 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-155576963 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-155610649 **[Test build #45552 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45552/consoleFull)** for PR 9530 at commit [`11f61a8`](https://github.com/apache/spark/commit/11f61a88e4a9a1a1a17572c8579143fc856667d9). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-155610925 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-155610948 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-155610829 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/45552/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-155610828 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-155613207 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-155613209 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/45570/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-155624877 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/45547/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-155624815 **[Test build #45547 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45547/consoleFull)** for PR 9530 at commit [`71ac0cf`](https://github.com/apache/spark/commit/71ac0cfdb25d7bb29e180c704a8eac840c7fccd4). * This patch **fails from timeout after a configured wait of \`250m\`**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-155624875 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-155203661 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/9530#discussion_r44361246
--- Diff: core/src/main/scala/org/apache/spark/rpc/netty/NettyRpcEnv.scala
---
@@ -302,6 +325,138 @@ private[netty] class NettyRpcEnv(
}
}
+ override def fileServer: RpcEnvFileServer = streamManager
+
+ override def openChannel(uri: String): ReadableByteChannel = {
+val parsedUri = new URI(uri)
+require(parsedUri.getHost() != null, "Host name must be defined.")
+require(parsedUri.getPort() > 0, "Port must be defined.")
+require(parsedUri.getPath() != null && parsedUri.getPath().nonEmpty,
"Path must be defined.")
+
+val pipe = Pipe.open()
+val source = new FileDownloadChannel(pipe.source())
+try {
+ val client = fileDownloadClient(parsedUri.getHost(),
parsedUri.getPort())
+ val callback = new FileDownloadCallback(pipe.sink(), source, client)
+ client.stream(parsedUri.getPath(), callback)
+} catch {
+ case e: Exception =>
+pipe.sink().close()
+source.close()
+throw e
+}
+
+source
+ }
+
+ private def fileDownloadClient(host: String, port: Int): TransportClient
= synchronized {
--- End diff --
Downloading from different hosts mean different clients. But different
tasks will still all download from the same host (the driver).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/9530#discussion_r44360229
--- Diff: core/src/main/scala/org/apache/spark/rpc/netty/NettyRpcEnv.scala
---
@@ -302,6 +325,138 @@ private[netty] class NettyRpcEnv(
}
}
+ override def fileServer: RpcEnvFileServer = streamManager
+
+ override def openChannel(uri: String): ReadableByteChannel = {
+val parsedUri = new URI(uri)
+require(parsedUri.getHost() != null, "Host name must be defined.")
+require(parsedUri.getPort() > 0, "Port must be defined.")
+require(parsedUri.getPath() != null && parsedUri.getPath().nonEmpty,
"Path must be defined.")
+
+val pipe = Pipe.open()
+val source = new FileDownloadChannel(pipe.source())
+try {
+ val client = fileDownloadClient(parsedUri.getHost(),
parsedUri.getPort())
+ val callback = new FileDownloadCallback(pipe.sink(), source, client)
+ client.stream(parsedUri.getPath(), callback)
+} catch {
+ case e: Exception =>
+pipe.sink().close()
+source.close()
+throw e
+}
+
+source
+ }
+
+ private def fileDownloadClient(host: String, port: Int): TransportClient
= synchronized {
--- End diff --
So if dowloading from different hosts (different tasks in the same
Executor), they need to connect one by one. Right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/9530#discussion_r44361876
--- Diff: core/src/main/scala/org/apache/spark/rpc/netty/NettyRpcEnv.scala
---
@@ -302,6 +325,138 @@ private[netty] class NettyRpcEnv(
}
}
+ override def fileServer: RpcEnvFileServer = streamManager
+
+ override def openChannel(uri: String): ReadableByteChannel = {
+val parsedUri = new URI(uri)
+require(parsedUri.getHost() != null, "Host name must be defined.")
+require(parsedUri.getPort() > 0, "Port must be defined.")
+require(parsedUri.getPath() != null && parsedUri.getPath().nonEmpty,
"Path must be defined.")
+
+val pipe = Pipe.open()
+val source = new FileDownloadChannel(pipe.source())
+try {
+ val client = fileDownloadClient(parsedUri.getHost(),
parsedUri.getPort())
+ val callback = new FileDownloadCallback(pipe.sink(), source, client)
+ client.stream(parsedUri.getPath(), callback)
+} catch {
+ case e: Exception =>
+pipe.sink().close()
+source.close()
+throw e
+}
+
+source
+ }
+
+ private def fileDownloadClient(host: String, port: Int): TransportClient
= synchronized {
--- End diff --
Just found another case: `fetchFiles` is also used in `DriverRunner` in the
`Worker`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/9530#discussion_r44361532
--- Diff: core/src/main/scala/org/apache/spark/rpc/netty/NettyRpcEnv.scala
---
@@ -302,6 +325,138 @@ private[netty] class NettyRpcEnv(
}
}
+ override def fileServer: RpcEnvFileServer = streamManager
+
+ override def openChannel(uri: String): ReadableByteChannel = {
+val parsedUri = new URI(uri)
+require(parsedUri.getHost() != null, "Host name must be defined.")
+require(parsedUri.getPort() > 0, "Port must be defined.")
+require(parsedUri.getPath() != null && parsedUri.getPath().nonEmpty,
"Path must be defined.")
+
+val pipe = Pipe.open()
+val source = new FileDownloadChannel(pipe.source())
+try {
+ val client = fileDownloadClient(parsedUri.getHost(),
parsedUri.getPort())
+ val callback = new FileDownloadCallback(pipe.sink(), source, client)
+ client.stream(parsedUri.getPath(), callback)
+} catch {
+ case e: Exception =>
+pipe.sink().close()
+source.close()
+throw e
+}
+
+source
+ }
+
+ private def fileDownloadClient(host: String, port: Int): TransportClient
= synchronized {
--- End diff --
Oh, right. All tasks will download from the same driver.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-155133542 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-155134307 **[Test build #45383 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45383/consoleFull)** for PR 9530 at commit [`87b7a91`](https://github.com/apache/spark/commit/87b7a9174be76646ef814b42dc7d0172ccc23552). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-155133513 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11140] [core] Transfer files using netw...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/9530#issuecomment-155166367 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
