[jira] [Commented] (SPARK-19659) Fetch big blocks to disk when shuffle-read

2017-07-10 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16080351#comment-16080351 ] Wenchen Fan commented on SPARK-19659: - I think spark 1.6 client can talk to spark 2.2 shuffle

[jira] [Commented] (SPARK-19659) Fetch big blocks to disk when shuffle-read

2017-07-10 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16080340#comment-16080340 ] Thomas Graves commented on SPARK-19659: --- can you be more specific here, do you mean that if you

[jira] [Commented] (SPARK-19659) Fetch big blocks to disk when shuffle-read

2017-06-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16069396#comment-16069396 ] Apache Spark commented on SPARK-19659: -- User 'zsxwing' has created a pull request for this issue:

[jira] [Commented] (SPARK-19659) Fetch big blocks to disk when shuffle-read

2017-05-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025900#comment-16025900 ] Apache Spark commented on SPARK-19659: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Commented] (SPARK-19659) Fetch big blocks to disk when shuffle-read

2017-04-17 Thread jin xing (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15971025#comment-15971025 ] jin xing commented on SPARK-19659: -- [~cloud_fan] I refined the the pr. In current change, I'd propose:

[jira] [Commented] (SPARK-19659) Fetch big blocks to disk when shuffle-read

2017-04-12 Thread jin xing (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15965452#comment-15965452 ] jin xing commented on SPARK-19659: -- Yes, I think it's a good idea to leverage memory manager instead of

[jira] [Commented] (SPARK-19659) Fetch big blocks to disk when shuffle-read

2017-04-11 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15965302#comment-15965302 ] Wenchen Fan commented on SPARK-19659: - Seems your model is, all shuffle blocks should be fetched into

[jira] [Commented] (SPARK-19659) Fetch big blocks to disk when shuffle-read

2017-04-11 Thread jin xing (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15964596#comment-15964596 ] jin xing commented on SPARK-19659: -- *bytesShuffleToMemory* is different from *bytesInFlight*.

[jira] [Commented] (SPARK-19659) Fetch big blocks to disk when shuffle-read

2017-04-11 Thread jin xing (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15964587#comment-15964587 ] jin xing commented on SPARK-19659: -- [~cloud_fan] Thanks a lot for taking look into this and sorry for

[jira] [Commented] (SPARK-19659) Fetch big blocks to disk when shuffle-read

2017-04-11 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15964548#comment-15964548 ] Wenchen Fan commented on SPARK-19659: - [~jinxing6...@126.com] can you say more about fetching shuffle

[jira] [Commented] (SPARK-19659) Fetch big blocks to disk when shuffle-read

2017-04-11 Thread jin xing (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15964508#comment-15964508 ] jin xing commented on SPARK-19659: -- [~irashid] Tracking memory used by Netty by swapping in our own

[jira] [Commented] (SPARK-19659) Fetch big blocks to disk when shuffle-read

2017-04-11 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15964476#comment-15964476 ] Imran Rashid commented on SPARK-19659: -- currently the smallest unit is an entire block. That

[jira] [Commented] (SPARK-19659) Fetch big blocks to disk when shuffle-read

2017-04-03 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15953968#comment-15953968 ] Wenchen Fan commented on SPARK-19659: - What's the smallest unit of fetching remote shuffle blocks?

[jira] [Commented] (SPARK-19659) Fetch big blocks to disk when shuffle-read

2017-04-01 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952401#comment-15952401 ] Imran Rashid commented on SPARK-19659: -- [~cloud_fan] The memory is allocated by netty, but its still

[jira] [Commented] (SPARK-19659) Fetch big blocks to disk when shuffle-read

2017-04-01 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952303#comment-15952303 ] Wenchen Fan commented on SPARK-19659: - Is it possible to track shuffle fetch memory? I think the

[jira] [Commented] (SPARK-19659) Fetch big blocks to disk when shuffle-read

2017-03-07 Thread jin xing (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900741#comment-15900741 ] jin xing commented on SPARK-19659: -- [~irashid] [~rxin] I uploaded SPARK-19659-design-v2.pdf, please take

[jira] [Commented] (SPARK-19659) Fetch big blocks to disk when shuffle-read

2017-03-06 Thread jin xing (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15897244#comment-15897244 ] jin xing commented on SPARK-19659: -- [~irashid] Thanks a lot for your comments, really helpful. I got

[jira] [Commented] (SPARK-19659) Fetch big blocks to disk when shuffle-read

2017-03-03 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15895502#comment-15895502 ] Imran Rashid commented on SPARK-19659: -- I think Reynold has a good point. I really don't like the

[jira] [Commented] (SPARK-19659) Fetch big blocks to disk when shuffle-read

2017-03-03 Thread jin xing (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15895411#comment-15895411 ] jin xing commented on SPARK-19659: -- [~rxin] Thanks a lot for comment. Tracking average size and also the

[jira] [Commented] (SPARK-19659) Fetch big blocks to disk when shuffle-read

2017-03-03 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893900#comment-15893900 ] Reynold Xin commented on SPARK-19659: - Rather than this tracking the max for each N/2000 consecutive

[jira] [Commented] (SPARK-19659) Fetch big blocks to disk when shuffle-read

2017-03-02 Thread jin xing (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15891999#comment-15891999 ] jin xing commented on SPARK-19659: -- [~rxin] [~davies] [~andrewor14] [~joshrosen] I've uploaded a design

[jira] [Commented] (SPARK-19659) Fetch big blocks to disk when shuffle-read

2017-02-26 Thread jin xing (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15884789#comment-15884789 ] jin xing commented on SPARK-19659: -- [~irashid] I've uploaded a design doc, please take a look and give

[jira] [Commented] (SPARK-19659) Fetch big blocks to disk when shuffle-read

2017-02-22 Thread jin xing (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15878389#comment-15878389 ] jin xing commented on SPARK-19659: -- [~irashid] Thanks a lot for your comments. I will file a design pdf

[jira] [Commented] (SPARK-19659) Fetch big blocks to disk when shuffle-read

2017-02-21 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876466#comment-15876466 ] Imran Rashid commented on SPARK-19659: -- [~jinxing6...@126.com] Thanks for taking this on, I think

[jira] [Commented] (SPARK-19659) Fetch big blocks to disk when shuffle-read

2017-02-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15873616#comment-15873616 ] Apache Spark commented on SPARK-19659: -- User 'jinxing64' has created a pull request for this issue: