Github user dhruve commented on the issue:
https://github.com/apache/spark/pull/18487
@tgravescs Thanks for merging this. I have created a PR for 2.2
https://github.com/apache/spark/pull/18691
I had to remove a couple of newer config entries which landed while
resolving a
Github user dhruve commented on the issue:
https://github.com/apache/spark/pull/18487
@cloud-fan replied to your comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/18487
@rxin it's kind of a stability fix(make shuffle service more stable), so
I'm ok to backport if the conflict is small.
---
If your project is set up for it, you can reply to this email and have
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/18487
hm is this a bug fix? if not we shouldn't cherry pick it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/18487
cherry pick to 2.2 wasn't clean so can you please put up a separate PR
against branch 2.2
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18487
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79760/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18487
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18487
**[Test build #79760 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79760/testReport)**
for PR 18487 at commit
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/18487
+1, pending jenkins build. if no further comments I'm going to commit this
later today.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18487
**[Test build #79760 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79760/testReport)**
for PR 18487 at commit
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/18487
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/18487
@jinxing64 We have the default set the int max so that by default there is
no performance penalty for users. We have done some testing as Dhruve
mentioned but we don't regularly hit the issue.
Github user dhruve commented on the issue:
https://github.com/apache/spark/pull/18487
@jinxing64 I performed few runs to see if we were observing any performance
issues with the change. I ran a simple word count job over a random set of text
- 3TB. I couldn't get 100's of executors
Github user jinxing64 commented on the issue:
https://github.com/apache/spark/pull/18487
`maxReqsInFlight` and `maxBytesInFlight` is hard to control the # of blocks
in a single request. When # of map is very high, this change can alleviate the
pressure of shuffle server.
@dhruve
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18487
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79614/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18487
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18487
**[Test build #79614 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79614/testReport)**
for PR 18487 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18487
**[Test build #79614 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79614/testReport)**
for PR 18487 at commit
Github user dhruve commented on the issue:
https://github.com/apache/spark/pull/18487
@jiangxb1987 Thanks for the review.
@cloud-fan #18388
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/18487
Will this be convered by https://github.com/apache/spark/pull/18388 ? And
another concern is how shall we expect users to tune this config? Can users
just tune `spark.reducer.maxReqsInFlight`
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/18487
cc @cloud-fan
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so,
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18487
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79477/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18487
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18487
**[Test build #79477 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79477/testReport)**
for PR 18487 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18487
**[Test build #79477 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79477/testReport)**
for PR 18487 at commit
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/18487
Jenkins, test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18487
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18487
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79469/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18487
**[Test build #79469 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79469/testReport)**
for PR 18487 at commit
Github user dhruve commented on the issue:
https://github.com/apache/spark/pull/18487
@jiangxb1987 I have resolved the merge conflicts and reworded the config to
make it more clear.
---
If your project is set up for it, you can reply to this email and have your
reply appear on
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18487
**[Test build #79469 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79469/testReport)**
for PR 18487 at commit
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/18487
This LGTM, @dhruve could you rebase it with the master branch please?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user dhruve commented on the issue:
https://github.com/apache/spark/pull/18487
@jiangxb1987 I have made the changes requested. Can you have a look.
Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18487
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18487
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79292/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18487
**[Test build #79292 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79292/testReport)**
for PR 18487 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18487
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18487
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79290/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18487
**[Test build #79290 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79290/testReport)**
for PR 18487 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18487
**[Test build #79292 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79292/testReport)**
for PR 18487 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18487
**[Test build #79290 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79290/testReport)**
for PR 18487 at commit
Github user dhruve commented on the issue:
https://github.com/apache/spark/pull/18487
@rxin @cloud-fan Can you review this PR?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18487
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18487
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78984/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18487
**[Test build #78984 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78984/testReport)**
for PR 18487 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18487
**[Test build #78984 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78984/testReport)**
for PR 18487 at commit
46 matches
Mail list logo
