Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/16989
how about we make the number of blocks which triggers
`HighlyCompressedMapStatus` configurable? If users wanna more accurate block
size estimation, they should increase that config (it's hard
Github user jinxing64 commented on the issue:
https://github.com/apache/spark/pull/16989
As @mridulm mentioned, in `HighlyCompressedMapStatus` it can be configured
in two respects:
>1. minimum size before we consider something a large block.
>2. The fraction '2' should also
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16989
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16989
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76721/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16989
**[Test build #76721 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76721/testReport)**
for PR 16989 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16989
**[Test build #76721 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76721/testReport)**
for PR 16989 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16989
**[Test build #76720 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76720/testReport)**
for PR 16989 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16989
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76720/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16989
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16989
**[Test build #76720 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76720/testReport)**
for PR 16989 at commit
Github user jinxing64 commented on the issue:
https://github.com/apache/spark/pull/16989
Yes, I will refine :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16989
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76686/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16989
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16989
**[Test build #76686 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76686/testReport)**
for PR 16989 at commit
Github user jinxing64 commented on the issue:
https://github.com/apache/spark/pull/16989
@mridulm
Really thankful for taking time looking into this pr. Really helpful. I
refined according to your comments. Please take another look when you have time
and give more comments : )
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16989
**[Test build #76686 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76686/testReport)**
for PR 16989 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16989
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76627/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16989
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16989
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16989
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76630/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16989
**[Test build #76630 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76630/testReport)**
for PR 16989 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16989
**[Test build #76627 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76627/testReport)**
for PR 16989 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16989
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16989
**[Test build #76622 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76622/testReport)**
for PR 16989 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16989
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76622/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16989
**[Test build #76622 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76622/testReport)**
for PR 16989 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16989
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16989
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76574/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16989
**[Test build #76574 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76574/testReport)**
for PR 16989 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16989
**[Test build #76574 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76574/testReport)**
for PR 16989 at commit
Github user jinxing64 commented on the issue:
https://github.com/apache/spark/pull/16989
Jenkins, test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user jinxing64 commented on the issue:
https://github.com/apache/spark/pull/16989
@cloud-fan
Thank you very much for reviewing this thus far :)
>How about we always fetch to disk if the block size is over
maxBytesInFlight?
I super agree with this. It's to use
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/16989
looks pretty good, one last comment: since the shuffle fetcher doesn't
support spilling to disk, we should be very careful to avoid allocating a lot
of memory. Normally this should be fine as the
Github user jinxing64 commented on the issue:
https://github.com/apache/spark/pull/16989
@cloud-fan
More comments on this ? :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16989
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16989
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76497/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16989
**[Test build #76497 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76497/testReport)**
for PR 16989 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16989
**[Test build #76497 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76497/testReport)**
for PR 16989 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16989
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76490/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16989
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16989
**[Test build #76490 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76490/testReport)**
for PR 16989 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16989
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76489/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16989
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16989
**[Test build #76489 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76489/testReport)**
for PR 16989 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16989
**[Test build #76490 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76490/testReport)**
for PR 16989 at commit
Github user jinxing64 commented on the issue:
https://github.com/apache/spark/pull/16989
@cloud-fan
Really really thankful for reviewing this pr:). I've refined according to
your comments. Please take another look at this when you have time.
---
If your project is set up for
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16989
**[Test build #76489 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76489/testReport)**
for PR 16989 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16989
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16989
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76486/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16989
**[Test build #76486 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76486/testReport)**
for PR 16989 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16989
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76485/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16989
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16989
**[Test build #76485 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76485/testReport)**
for PR 16989 at commit
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/16989
LGTM except some minor comments
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16989
**[Test build #76486 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76486/testReport)**
for PR 16989 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16989
**[Test build #76485 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76485/testReport)**
for PR 16989 at commit
Github user jinxing64 commented on the issue:
https://github.com/apache/spark/pull/16989
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16989
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76483/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16989
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16989
**[Test build #76483 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76483/testReport)**
for PR 16989 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16989
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76415/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16989
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16989
**[Test build #76415 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76415/testReport)**
for PR 16989 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16989
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16989
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76414/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16989
**[Test build #76414 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76414/testReport)**
for PR 16989 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16989
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76412/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16989
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16989
**[Test build #76412 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76412/testReport)**
for PR 16989 at commit
Github user jinxing64 commented on the issue:
https://github.com/apache/spark/pull/16989
@squito
Thanks a lot for your comments : )
Yes, There must be a design doc for discussing. I will prepare and post a
pdf to jira.
---
If your project is set up for it, you can reply to
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16989
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73224/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16989
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16989
**[Test build #73224 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73224/testReport)**
for PR 16989 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16989
**[Test build #73224 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73224/testReport)**
for PR 16989 at commit
Github user squito commented on the issue:
https://github.com/apache/spark/pull/16989
Jenkins, add to whitelist
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user squito commented on the issue:
https://github.com/apache/spark/pull/16989
Hi @jinxing64 I posted a comment on jira about the design -- I think this
is a big enough change that its worth discussing the design first. Its fine to
keep working on the code as a demonstration
Github user jinxing64 commented on the issue:
https://github.com/apache/spark/pull/16989
@vanzin @squito
Would you mind to take a look at this when have time ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16989
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
101 - 178 of 178 matches
Mail list logo
