[jira] [Commented] (SPARK-4550) In sort-based shuffle, store map outputs in serialized form

2015-05-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14529166#comment-14529166 ] Apache Spark commented on SPARK-4550: - User 'sryza' has created a pull request for

[jira] [Commented] (SPARK-4550) In sort-based shuffle, store map outputs in serialized form

2015-03-26 Thread Sandy Ryza (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14383280#comment-14383280 ] Sandy Ryza commented on SPARK-4550: --- Java serialization appears to write out the full

[jira] [Commented] (SPARK-4550) In sort-based shuffle, store map outputs in serialized form

2015-03-26 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14383260#comment-14383260 ] Reynold Xin commented on SPARK-4550: [~sandyryza] can you investigate what's happening

[jira] [Commented] (SPARK-4550) In sort-based shuffle, store map outputs in serialized form

2015-03-20 Thread Sandy Ryza (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14372232#comment-14372232 ] Sandy Ryza commented on SPARK-4550: --- I spoke briefly with Reynold about this offline,

[jira] [Commented] (SPARK-4550) In sort-based shuffle, store map outputs in serialized form

2015-02-07 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310643#comment-14310643 ] Apache Spark commented on SPARK-4550: - User 'sryza' has created a pull request for

[jira] [Commented] (SPARK-4550) In sort-based shuffle, store map outputs in serialized form

2015-02-06 Thread Sandy Ryza (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310130#comment-14310130 ] Sandy Ryza commented on SPARK-4550: --- I got a working prototype and benchmarked the

[jira] [Commented] (SPARK-4550) In sort-based shuffle, store map outputs in serialized form

2015-02-04 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14304817#comment-14304817 ] Reynold Xin commented on SPARK-4550: Sandy, The proposal seems to assume that objects

[jira] [Commented] (SPARK-4550) In sort-based shuffle, store map outputs in serialized form

2015-02-04 Thread Sandy Ryza (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14304864#comment-14304864 ] Sandy Ryza commented on SPARK-4550: --- I had heard rumors to that effect, so I ran some

[jira] [Commented] (SPARK-4550) In sort-based shuffle, store map outputs in serialized form

2015-02-04 Thread Sandy Ryza (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14304816#comment-14304816 ] Sandy Ryza commented on SPARK-4550: --- WIP branch:

[jira] [Commented] (SPARK-4550) In sort-based shuffle, store map outputs in serialized form

2015-02-04 Thread Sandy Ryza (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14306321#comment-14306321 ] Sandy Ryza commented on SPARK-4550: --- I also just tried this out using an object that's

[jira] [Commented] (SPARK-4550) In sort-based shuffle, store map outputs in serialized form

2015-02-03 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14302999#comment-14302999 ] Patrick Wendell commented on SPARK-4550: The doc alludes to having to (at some

[jira] [Commented] (SPARK-4550) In sort-based shuffle, store map outputs in serialized form

2015-02-02 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14302326#comment-14302326 ] Patrick Wendell commented on SPARK-4550: Yeah, this is a good idea. I don't see

[jira] [Commented] (SPARK-4550) In sort-based shuffle, store map outputs in serialized form

2015-02-02 Thread Sandy Ryza (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14302303#comment-14302303 ] Sandy Ryza commented on SPARK-4550: --- Just posted a design doc. Would love to get

[jira] [Commented] (SPARK-4550) In sort-based shuffle, store map outputs in serialized form

2014-11-21 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14221690#comment-14221690 ] Patrick Wendell commented on SPARK-4550: Not an expert on the internals of this

[jira] [Commented] (SPARK-4550) In sort-based shuffle, store map outputs in serialized form

2014-11-21 Thread Sandy Ryza (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14221710#comment-14221710 ] Sandy Ryza commented on SPARK-4550: --- We don't, though it would allow us to be much more