Github user advancedxy commented on the issue:
https://github.com/apache/spark/pull/23083
> For the task completion listener, I think it's an overkill to introduce a
new API, do you know where exactly we leak the memory? and can we null it out
when the ShuffleBlockFetcherIterator
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/23083
Looking at the code, we are trying to fix 2 memory leaks: the task
completion listener in `ShuffleBlockFetcherIterator`, and the
`CompletionIterator`. If that's case, can you say that in the PR
Github user szhem commented on the issue:
https://github.com/apache/spark/pull/23083
> So do you mean CoGroupRDDs with multiple input sources will have similar
problems?
Yep, but a little bit different ones
> If so, can you create another Jira?
Will do it
Github user advancedxy commented on the issue:
https://github.com/apache/spark/pull/23083
And another thing:
> P.S. This PR does not cover cases with CoGroupedRDDs which use
ExternalAppendOnlyMap internally, which itself can lead to OutOfMemoryErrors in
many places.
So do you
Github user szhem commented on the issue:
https://github.com/apache/spark/pull/23083
Hi @davies, @advancedxy, @rxin,
You seem to be the last ones who touched the corresponding parts of the
files in this PR.
Could you be so kind to take a look at it?
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23083
**[Test build #4433 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4433/testReport)**
for PR 23083 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23083
**[Test build #4433 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4433/testReport)**
for PR 23083 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23083
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23083
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23083
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
10 matches
Mail list logo
