[
https://issues.apache.org/jira/browse/SPARK-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15278147#comment-15278147
]
Apache Spark commented on SPARK-4452:
-
User 'lianhuiwang' has created a pull request for this issue:
[
https://issues.apache.org/jira/browse/SPARK-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15277583#comment-15277583
]
Apache Spark commented on SPARK-4452:
-
User 'lianhuiwang' has created a pull request for this issue:
[
https://issues.apache.org/jira/browse/SPARK-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15276109#comment-15276109
]
Xin Hao commented on SPARK-4452:
Since this is an old issue which impact Spark since 1.1.0, can the patch
[
https://issues.apache.org/jira/browse/SPARK-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256602#comment-15256602
]
Davies Liu commented on SPARK-4452:
---
We only backport critical bug fix into released branch.
There is
[
https://issues.apache.org/jira/browse/SPARK-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15255658#comment-15255658
]
Romi Kuntsman commented on SPARK-4452:
--
Hi, what's the reason this will only be available in Spark
[
https://issues.apache.org/jira/browse/SPARK-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15030450#comment-15030450
]
Apache Spark commented on SPARK-4452:
-
User 'lianhuiwang' has created a pull request for this issue:
[
https://issues.apache.org/jira/browse/SPARK-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14608618#comment-14608618
]
Apache Spark commented on SPARK-4452:
-
User 'lianhuiwang' has created a pull request
[
https://issues.apache.org/jira/browse/SPARK-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14609144#comment-14609144
]
Josh Rosen commented on SPARK-4452:
---
I've linked this to the Project Tungsten JIRA epic,
[
https://issues.apache.org/jira/browse/SPARK-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14290874#comment-14290874
]
Sean Owen commented on SPARK-4452:
--
Can this JIRA be resolved now that its children are
[
https://issues.apache.org/jira/browse/SPARK-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14290885#comment-14290885
]
Sandy Ryza commented on SPARK-4452:
---
I think there's more to this one, the subtasks
[
https://issues.apache.org/jira/browse/SPARK-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226969#comment-14226969
]
Sandy Ryza commented on SPARK-4452:
---
Thinking about the current change a little more, an
[
https://issues.apache.org/jira/browse/SPARK-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14224996#comment-14224996
]
Tianshuo Deng commented on SPARK-4452:
--
[~sandyr]:
Thanks for the feedback!
For
[
https://issues.apache.org/jira/browse/SPARK-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14222572#comment-14222572
]
Sandy Ryza commented on SPARK-4452:
---
[~tianshuo], I took a look at the patch, and the
[
https://issues.apache.org/jira/browse/SPARK-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14219795#comment-14219795
]
Tianshuo Deng commented on SPARK-4452:
--
Hi, [~matei]:
Hi, Matei
My way of
[
https://issues.apache.org/jira/browse/SPARK-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14220070#comment-14220070
]
Tianshuo Deng commented on SPARK-4452:
--
Here is a link of the diff:
[
https://issues.apache.org/jira/browse/SPARK-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14218289#comment-14218289
]
Tianshuo Deng commented on SPARK-4452:
--
Hi,
While I'm working on this ticket, I have
[
https://issues.apache.org/jira/browse/SPARK-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14218318#comment-14218318
]
Andrew Or commented on SPARK-4452:
--
[~tianshuo] That is a correct assumption for
[
https://issues.apache.org/jira/browse/SPARK-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14216691#comment-14216691
]
Matei Zaharia commented on SPARK-4452:
--
BTW I've thought about this more and here's
[
https://issues.apache.org/jira/browse/SPARK-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14216933#comment-14216933
]
Sandy Ryza commented on SPARK-4452:
---
One issue with a limits-by-object approach is that
[
https://issues.apache.org/jira/browse/SPARK-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14217253#comment-14217253
]
Andrew Or commented on SPARK-4452:
--
I have opened a JIRA that targets on fixing this on a
[
https://issues.apache.org/jira/browse/SPARK-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14217274#comment-14217274
]
Andrew Or commented on SPARK-4452:
--
[~matei] I have implemented your first suggestion
[
https://issues.apache.org/jira/browse/SPARK-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14217331#comment-14217331
]
Matei Zaharia commented on SPARK-4452:
--
Forced spilling is orthogonal to how you set
[
https://issues.apache.org/jira/browse/SPARK-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14217340#comment-14217340
]
Sandy Ryza commented on SPARK-4452:
---
[~matei] my point is not that forced spilling
[
https://issues.apache.org/jira/browse/SPARK-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14215269#comment-14215269
]
Sandy Ryza commented on SPARK-4452:
---
Updated the title to reflect the specific problem.
[
https://issues.apache.org/jira/browse/SPARK-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14215395#comment-14215395
]
Andrew Or commented on SPARK-4452:
--
Hey [~tianshuo] do you see this issue only for
[
https://issues.apache.org/jira/browse/SPARK-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14215411#comment-14215411
]
Tianshuo Deng commented on SPARK-4452:
--
Hi, [~andrewor14]:
Actually hash-based
[
https://issues.apache.org/jira/browse/SPARK-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14215418#comment-14215418
]
Tianshuo Deng commented on SPARK-4452:
--
Hi, [~andrewor14]:
The elementsRead bug that
[
https://issues.apache.org/jira/browse/SPARK-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14215425#comment-14215425
]
Matei Zaharia commented on SPARK-4452:
--
How much of this gets fixed if you fix the
[
https://issues.apache.org/jira/browse/SPARK-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14215427#comment-14215427
]
Andrew Or commented on SPARK-4452:
--
I see, in other words, there are two separate issues
[
https://issues.apache.org/jira/browse/SPARK-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14215434#comment-14215434
]
Tianshuo Deng commented on SPARK-4452:
--
Hi, [~andrewor14],
Yeah exactly. Actually
[
https://issues.apache.org/jira/browse/SPARK-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14215436#comment-14215436
]
Sandy Ryza commented on SPARK-4452:
---
[~andrewor14], IIUC, (2) shouldn't happen in
[
https://issues.apache.org/jira/browse/SPARK-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14215446#comment-14215446
]
Andrew Or commented on SPARK-4452:
--
[~sandyr] hash-based shuffle can still use two
[
https://issues.apache.org/jira/browse/SPARK-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14215448#comment-14215448
]
Sandy Ryza commented on SPARK-4452:
---
Ah, true.
Shuffle data structures can starve
[
https://issues.apache.org/jira/browse/SPARK-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14215449#comment-14215449
]
Tianshuo Deng commented on SPARK-4452:
--
[~matei]:
You are right, it does add more
[
https://issues.apache.org/jira/browse/SPARK-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14215491#comment-14215491
]
Apache Spark commented on SPARK-4452:
-
User 'andrewor14' has created a pull request
[
https://issues.apache.org/jira/browse/SPARK-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14215557#comment-14215557
]
Matei Zaharia commented on SPARK-4452:
--
BTW we may also want to create a separate
[
https://issues.apache.org/jira/browse/SPARK-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14215556#comment-14215556
]
Matei Zaharia commented on SPARK-4452:
--
Got it. It would be fine to do this if you
[
https://issues.apache.org/jira/browse/SPARK-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14215617#comment-14215617
]
Andrew Or commented on SPARK-4452:
--
I have created SPARK-4467 for the `elementsRead` bug
38 matches
Mail list logo