Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/3656
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enab
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/3656#issuecomment-75132672
Hey @lawlerd this seems to be superseded by #4420. Would you mind closing
this PR? Please follow the latest discussion there.
---
If your project is set up for it, yo
Github user mccheah commented on the pull request:
https://github.com/apache/spark/pull/3656#issuecomment-71526176
Seeing some problems that this PR could address so reviving this thread.
@lawlerd the configurable count would help because if it is known that the
individual obj
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/3656#issuecomment-66676807
@lawlerd things are done this way because estimating the size for every
record would be prohibitively expensive. Also, the trackMemoryThreshold is
required at least until
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3656#issuecomment-66542977
[Test build #24320 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24320/consoleFull)
for PR 3656 at commit
[`2df2e2d`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3656#issuecomment-66542983
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24
Github user lawlerd commented on a diff in the pull request:
https://github.com/apache/spark/pull/3656#discussion_r21644679
--- Diff:
core/src/main/scala/org/apache/spark/util/collection/Spillable.scala ---
@@ -72,8 +69,7 @@ private[spark] trait Spillable[C] extends Logging {
Github user lawlerd commented on the pull request:
https://github.com/apache/spark/pull/3656#issuecomment-66538076
Thanks for the review, @andrewor14!
I figured that to be the case. I weighed this against the hazard of OOMs
and figured it was the lesser evil, but I might have
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3656#issuecomment-66527581
[Test build #24320 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24320/consoleFull)
for PR 3656 at commit
[`2df2e2d`](https://githu
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/3656#issuecomment-66526951
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/3656#issuecomment-66526912
Hey @lawlerd I don't think we want to get rid of this check because we want
the thread to ramp up and not just spill all the time. Would it make sense to
make the spil
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3656#discussion_r21638661
--- Diff:
core/src/main/scala/org/apache/spark/util/collection/Spillable.scala ---
@@ -72,8 +69,7 @@ private[spark] trait Spillable[C] extends Logging {
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3656#issuecomment-66392891
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your pro
GitHub user lawlerd opened a pull request:
https://github.com/apache/spark/pull/3656
[SPARK-4808] Remove Spillable minimum threshold and sampling
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/lawlerd/spark branch-1.2
Alternati
14 matches
Mail list logo
