Github user squito commented on the issue:
https://github.com/apache/spark/pull/21212
@jinxing64
> I guess your concern is ArrayBuffer will do lots of copy as size of
elements grows, and we don't need fast random access in
ShuffleBlockFetcherIterator
my concern wasn't
Github user jinxing64 commented on the issue:
https://github.com/apache/spark/pull/21212
Thanks for merging !
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/21212
thanks, merging to master!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21212
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90283/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21212
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21212
**[Test build #90283 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90283/testReport)**
for PR 21212 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21212
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2968/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21212
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21212
**[Test build #90283 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90283/testReport)**
for PR 21212 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21212
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21212
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2966/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21212
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21212
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90280/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21212
**[Test build #90280 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90280/testReport)**
for PR 21212 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21212
**[Test build #90280 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90280/testReport)**
for PR 21212 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21212
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21212
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90243/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21212
**[Test build #90243 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90243/testReport)**
for PR 21212 at commit
Github user jinxing64 commented on the issue:
https://github.com/apache/spark/pull/21212
@squito
I ananlyzed "YourKit Memory Inspections" to analyze the heap, but didn't
find many duplicate objects in ArrayBuffer. I guess your concern is ArrayBuffer
will do lots of copy as size
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21212
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2944/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21212
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21212
**[Test build #90243 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90243/testReport)**
for PR 21212 at commit
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/21212
> do you mean optimize space usage for MapStatus when there are lots of
consecutive empty-blocks ?
Yea, something like doing an RLE for the size array in
`CompressedMapStatus`. But this
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21212
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90127/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21212
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21212
**[Test build #90127 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90127/testReport)**
for PR 21212 at commit
Github user squito commented on the issue:
https://github.com/apache/spark/pull/21212
can you add a test in MapOutputTrackerSuite and update the pr description
to include all the changes? but overall looks good.
---
Github user squito commented on the issue:
https://github.com/apache/spark/pull/21212
If you have a heap dump, there are tools that can check for wasted space in
ArrayBuffer. Eg. [jxray](http://www.jxray.com/) or [YourKit Memory
Github user jinxing64 commented on the issue:
https://github.com/apache/spark/pull/21212
@squito @cloud-fan @jiangxb1987
Thanks a lot for review.
> shall we also optimize the space usage for MapStatus
@cloud-fan do you mean optimize space usage for MapStatus when
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21212
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2867/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21212
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21212
**[Test build #90127 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90127/testReport)**
for PR 21212 at commit
Github user squito commented on the issue:
https://github.com/apache/spark/pull/21212
This makes sense to me. You should update the comment on
`MapOutputTracker.getMapSizesByExecutorId` to mention that it excludes the
zero-sized blocks, and also remove the filter in
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21212
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90041/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21212
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21212
**[Test build #90041 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90041/testReport)**
for PR 21212 at commit
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/21212
I think it's reasonable to filter out empty shuffle blocks, shall we also
optimize the space usage for `MapStatus`? then we can also reduce network
traffic.
---
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/21212
How much memory did the converted pairs consume? If the empty blocks should
be a issue can we just clean up the empty blocks?
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21212
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21212
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2804/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21212
**[Test build #90041 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90041/testReport)**
for PR 21212 at commit
Github user jinxing64 commented on the issue:
https://github.com/apache/spark/pull/21212
Jenkins, retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands,
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21212
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90032/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21212
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21212
**[Test build #90032 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90032/testReport)**
for PR 21212 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21212
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21212
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2795/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21212
**[Test build #90032 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90032/testReport)**
for PR 21212 at commit
Github user jinxing64 commented on the issue:
https://github.com/apache/spark/pull/21212
@squito @cloud-fan @jiangxb1987
Do you think this make sense?
---
-
To unsubscribe, e-mail:
49 matches
Mail list logo
