[jira] [Commented] (SPARK-6235) Address various 2G limits

2020-01-16 Thread Samuel Shepard (Jira)
[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017438#comment-17017438 ] Samuel Shepard commented on SPARK-6235: --- [~irashid] I followed your suggestion of looking in the

[jira] [Commented] (SPARK-6235) Address various 2G limits

2019-12-17 Thread Imran Rashid (Jira)
[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16998577#comment-16998577 ] Imran Rashid commented on SPARK-6235: - [~sammysheep] Spark's ML library uses the same jira project,

[jira] [Commented] (SPARK-6235) Address various 2G limits

2019-12-16 Thread Samuel Shepard (Jira)
[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16997798#comment-16997798 ] Samuel Shepard commented on SPARK-6235: --- [~irashid] I meant the former (task result > 2G) as best I

[jira] [Commented] (SPARK-6235) Address various 2G limits

2019-12-16 Thread Imran Rashid (Jira)
[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16997631#comment-16997631 ] Imran Rashid commented on SPARK-6235: - [~sammysheep] are you discussing the use case for task results

[jira] [Commented] (SPARK-6235) Address various 2G limits

2019-12-13 Thread Samuel Shepard (Jira)
[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16995889#comment-16995889 ] Samuel Shepard commented on SPARK-6235: --- One use case could be fetching large results to the driver

[jira] [Commented] (SPARK-6235) Address various 2G limits

2019-04-17 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820309#comment-16820309 ] Imran Rashid commented on SPARK-6235: - [~glal14] actually this was fixed in 2.4. There was one open

[jira] [Commented] (SPARK-6235) Address various 2G limits

2019-04-17 Thread Gowtam Lal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820229#comment-16820229 ] Gowtam Lal commented on SPARK-6235: --- It would be great to see this go out. Any updates? > Address

[jira] [Commented] (SPARK-6235) Address various 2G limits

2018-05-22 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16484069#comment-16484069 ] Imran Rashid commented on SPARK-6235: - Would be nice to find a better home for this, but for now I

[jira] [Commented] (SPARK-6235) Address various 2G limits

2018-05-21 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16482901#comment-16482901 ] Imran Rashid commented on SPARK-6235: - [~tgraves] WAL -- write-ahead-log for receiver-based streaming.

[jira] [Commented] (SPARK-6235) Address various 2G limits

2018-05-21 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16482746#comment-16482746 ] Thomas Graves commented on SPARK-6235: -- >> Still unsupported: * large task results * large blocks

[jira] [Commented] (SPARK-6235) Address various 2G limits

2018-05-21 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16482699#comment-16482699 ] Imran Rashid commented on SPARK-6235: - Linked a [design

[jira] [Commented] (SPARK-6235) Address various 2G limits

2018-05-16 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16477986#comment-16477986 ] Imran Rashid commented on SPARK-6235: - derp, I missed a pretty basic case -- if you cache a large

[jira] [Commented] (SPARK-6235) Address various 2G limits

2018-05-16 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16477827#comment-16477827 ] Imran Rashid commented on SPARK-6235: - I've been testing the current state of the 2GB block limit, and

[jira] [Commented] (SPARK-6235) Address various 2G limits

2018-05-10 Thread Cyanny (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469991#comment-16469991 ] Cyanny commented on SPARK-6235: --- Hi, when will this Jira feature included in a spark releas? [~rxin] >

[jira] [Commented] (SPARK-6235) Address various 2G limits

2017-05-14 Thread Jamie Hutton (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16009808#comment-16009808 ] Jamie Hutton commented on SPARK-6235: - Hi there, Is there any update on when this will be included in

[jira] [Commented] (SPARK-6235) Address various 2G limits

2016-09-22 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15515042#comment-15515042 ] Guoqiang Li commented on SPARK-6235: ping [~rxin] > Address various 2G limits >

[jira] [Commented] (SPARK-6235) Address various 2G limits

2016-09-20 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508670#comment-15508670 ] Guoqiang Li commented on SPARK-6235: [~rxin] Any comments? > Address various 2G limits >

[jira] [Commented] (SPARK-6235) Address various 2G limits

2016-09-07 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15470410#comment-15470410 ] Apache Spark commented on SPARK-6235: - User 'witgo' has created a pull request for this issue:

[jira] [Commented] (SPARK-6235) Address various 2G limits

2016-09-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15467563#comment-15467563 ] Apache Spark commented on SPARK-6235: - User 'witgo' has created a pull request for this issue:

[jira] [Commented] (SPARK-6235) Address various 2G limits

2016-08-15 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15421048#comment-15421048 ] Guoqiang Li commented on SPARK-6235: Yes, it contains a lot of minor changes, eg: Replace ByteBuffer

[jira] [Commented] (SPARK-6235) Address various 2G limits

2016-08-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15421018#comment-15421018 ] Apache Spark commented on SPARK-6235: - User 'witgo' has created a pull request for this issue:

[jira] [Commented] (SPARK-6235) Address various 2G limits

2016-08-15 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15420746#comment-15420746 ] Sean Owen commented on SPARK-6235: -- How does this relate to the existing subtasks and their

[jira] [Commented] (SPARK-6235) Address various 2G limits

2016-08-14 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15420554#comment-15420554 ] Guoqiang Li commented on SPARK-6235: [~hvanhovell] The main changes. 1. Replace DiskStore method

[jira] [Commented] (SPARK-6235) Address various 2G limits

2016-08-12 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15419151#comment-15419151 ] Herman van Hovell commented on SPARK-6235: -- [~gq] it might be a good idea to share some design

[jira] [Commented] (SPARK-6235) Address various 2G limits

2016-08-11 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418225#comment-15418225 ] Guoqiang Li commented on SPARK-6235: I'm doing this work and I'll put the patch in this month. >

[jira] [Commented] (SPARK-6235) Address various 2G limits

2016-08-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15413055#comment-15413055 ] Sean Owen commented on SPARK-6235: -- I think the short answer is, it's very hard. I am not sure it's

[jira] [Commented] (SPARK-6235) Address various 2G limits

2016-08-08 Thread Brian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412727#comment-15412727 ] Brian commented on SPARK-6235: -- How is it possible that Spark 2.0 comes and and this bug isn't solved? A

[jira] [Commented] (SPARK-6235) Address various 2G limits

2015-10-14 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14957546#comment-14957546 ] Reynold Xin commented on SPARK-6235: Is your data skewed? i.e. maybe there is a single key that's

[jira] [Commented] (SPARK-6235) Address various 2G limits

2015-10-14 Thread Glenn Strycker (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14957554#comment-14957554 ] Glenn Strycker commented on SPARK-6235: --- I don't think so, but I can check. My RDD came from an RDD

[jira] [Commented] (SPARK-6235) Address various 2G limits

2015-09-14 Thread Sean McKibben (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14744746#comment-14744746 ] Sean McKibben commented on SPARK-6235: -- When reading from HBase into spark, the regions seem to

[jira] [Commented] (SPARK-6235) Address various 2G limits

2015-09-14 Thread Ram Gande (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14744605#comment-14744605 ] Ram Gande commented on SPARK-6235: -- Any progress on this. We are seeing this issue constantly in our

[jira] [Commented] (SPARK-6235) Address various 2G limits

2015-08-31 Thread Zhen Peng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724609#comment-14724609 ] Zhen Peng commented on SPARK-6235: -- Hi [~rxin], is there any update for this issue? > Address various 2G