[
https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16080351#comment-16080351
]
Wenchen Fan commented on SPARK-19659:
-
I think spark 1.6 client can talk to spark 2.2 shuffle
[
https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16080340#comment-16080340
]
Thomas Graves commented on SPARK-19659:
---
can you be more specific here, do you mean that if you
[
https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16069396#comment-16069396
]
Apache Spark commented on SPARK-19659:
--
User 'zsxwing' has created a pull request for this issue:
[
https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025900#comment-16025900
]
Apache Spark commented on SPARK-19659:
--
User 'cloud-fan' has created a pull request for this issue:
[
https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15971025#comment-15971025
]
jin xing commented on SPARK-19659:
--
[~cloud_fan]
I refined the the pr. In current change, I'd propose:
[
https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15965452#comment-15965452
]
jin xing commented on SPARK-19659:
--
Yes, I think it's a good idea to leverage memory manager instead of
[
https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15965302#comment-15965302
]
Wenchen Fan commented on SPARK-19659:
-
Seems your model is, all shuffle blocks should be fetched into
[
https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15964596#comment-15964596
]
jin xing commented on SPARK-19659:
--
*bytesShuffleToMemory* is different from *bytesInFlight*.
[
https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15964587#comment-15964587
]
jin xing commented on SPARK-19659:
--
[~cloud_fan]
Thanks a lot for taking look into this and sorry for
[
https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15964548#comment-15964548
]
Wenchen Fan commented on SPARK-19659:
-
[~jinxing6...@126.com] can you say more about fetching shuffle
[
https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15964508#comment-15964508
]
jin xing commented on SPARK-19659:
--
[~irashid]
Tracking memory used by Netty by swapping in our own
[
https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15964476#comment-15964476
]
Imran Rashid commented on SPARK-19659:
--
currently the smallest unit is an entire block. That
[
https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15953968#comment-15953968
]
Wenchen Fan commented on SPARK-19659:
-
What's the smallest unit of fetching remote shuffle blocks?
[
https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952401#comment-15952401
]
Imran Rashid commented on SPARK-19659:
--
[~cloud_fan] The memory is allocated by netty, but its still
[
https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952303#comment-15952303
]
Wenchen Fan commented on SPARK-19659:
-
Is it possible to track shuffle fetch memory? I think the
[
https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900741#comment-15900741
]
jin xing commented on SPARK-19659:
--
[~irashid] [~rxin]
I uploaded SPARK-19659-design-v2.pdf, please take
[
https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15897244#comment-15897244
]
jin xing commented on SPARK-19659:
--
[~irashid]
Thanks a lot for your comments, really helpful. I got
[
https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15895502#comment-15895502
]
Imran Rashid commented on SPARK-19659:
--
I think Reynold has a good point. I really don't like the
[
https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15895411#comment-15895411
]
jin xing commented on SPARK-19659:
--
[~rxin]
Thanks a lot for comment.
Tracking average size and also the
[
https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893900#comment-15893900
]
Reynold Xin commented on SPARK-19659:
-
Rather than this tracking the max for each N/2000 consecutive
[
https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15891999#comment-15891999
]
jin xing commented on SPARK-19659:
--
[~rxin] [~davies] [~andrewor14] [~joshrosen]
I've uploaded a design
[
https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15884789#comment-15884789
]
jin xing commented on SPARK-19659:
--
[~irashid]
I've uploaded a design doc, please take a look and give
[
https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15878389#comment-15878389
]
jin xing commented on SPARK-19659:
--
[~irashid]
Thanks a lot for your comments. I will file a design pdf
[
https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876466#comment-15876466
]
Imran Rashid commented on SPARK-19659:
--
[~jinxing6...@126.com] Thanks for taking this on, I think
[
https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15873616#comment-15873616
]
Apache Spark commented on SPARK-19659:
--
User 'jinxing64' has created a pull request for this issue:
25 matches
Mail list logo