[GitHub] spark issue #18211: [WIP][SPARK-20994] Alleviate memory pressure in StreamMa...

2017-06-07 Thread jinxing64
Github user jinxing64 commented on the issue: https://github.com/apache/spark/pull/18211 @vanzin Thanks a lot for comment. I will close this pr and think if there is other solution. --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark issue #18211: [WIP][SPARK-20994] Alleviate memory pressure in StreamMa...

2017-06-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18211 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77790/ Test PASSed. ---

[GitHub] spark issue #18211: [WIP][SPARK-20994] Alleviate memory pressure in StreamMa...

2017-06-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18211 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18211: [WIP][SPARK-20994] Alleviate memory pressure in StreamMa...

2017-06-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18211 **[Test build #77790 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77790/testReport)** for PR 18211 at commit

[GitHub] spark issue #18211: [WIP][SPARK-20994] Alleviate memory pressure in StreamMa...

2017-06-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18211 **[Test build #8 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/8/testReport)** for PR 18211 at commit

[GitHub] spark issue #18211: [WIP][SPARK-20994] Alleviate memory pressure in StreamMa...

2017-06-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18211 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77768/ Test PASSed. ---

[GitHub] spark issue #18211: [WIP][SPARK-20994] Alleviate memory pressure in StreamMa...

2017-06-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18211 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18211: [WIP][SPARK-20994] Alleviate memory pressure in StreamMa...

2017-06-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18211 **[Test build #77768 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77768/testReport)** for PR 18211 at commit

[GitHub] spark issue #18211: [WIP][SPARK-20994] Alleviate memory pressure in StreamMa...

2017-06-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18211 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18211: [WIP][SPARK-20994] Alleviate memory pressure in StreamMa...

2017-06-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18211 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77769/ Test FAILed. ---

[GitHub] spark issue #18211: [WIP][SPARK-20994] Alleviate memory pressure in StreamMa...

2017-06-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18211 **[Test build #77769 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77769/testReport)** for PR 18211 at commit

[GitHub] spark issue #18211: [WIP][SPARK-20994] Alleviate memory pressure in StreamMa...

2017-06-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18211 **[Test build #77769 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77769/testReport)** for PR 18211 at commit

[GitHub] spark issue #18211: [WIP][SPARK-20994] Alleviate memory pressure in StreamMa...

2017-06-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18211 **[Test build #77768 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77768/testReport)** for PR 18211 at commit

[GitHub] spark issue #18211: [WIP][SPARK-20994] Alleviate memory pressure in StreamMa...

2017-06-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18211 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77767/ Test FAILed. ---

[GitHub] spark issue #18211: [WIP][SPARK-20994] Alleviate memory pressure in StreamMa...

2017-06-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18211 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18211: [WIP][SPARK-20994] Alleviate memory pressure in StreamMa...

2017-06-06 Thread jinxing64
Github user jinxing64 commented on the issue: https://github.com/apache/spark/pull/18211 In this pr: 1. Instead of `chunkIndex`, fetch chunk by `String chunkId`. Server doesn't cache the blocks list. 2. In `OpenBlocks`, only metadata(e.g. appId, executorId) of the stream is

[GitHub] spark issue #18211: [WIP][SPARK-20994] Alleviate memory pressure in StreamMa...

2017-06-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18211 **[Test build #77767 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77767/testReport)** for PR 18211 at commit

[GitHub] spark issue #18211: [WIP][SPARK-20994] Alleviate memory pressure in StreamMa...

2017-06-06 Thread jinxing64
Github user jinxing64 commented on the issue: https://github.com/apache/spark/pull/18211 In my cluster, we are suffering from OOM of shuffle-service. We found that a lot of executors are fetching blocks from a single shuffle-service. Analyzing the memory, we found that the