[jira] [Commented] (SPARK-14654) New accumulator API

2016-04-24 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15255840#comment-15255840 ] holdenk commented on SPARK-14654: - Ah yes, in DAGScheduler right now we cast it to [Any, Any] when we get

[jira] [Commented] (SPARK-14654) New accumulator API

2016-04-24 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15255820#comment-15255820 ] Reynold Xin commented on SPARK-14654: - But Holden the call sites in Spark are already erasing the

[jira] [Commented] (SPARK-14654) New accumulator API

2016-04-24 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15255819#comment-15255819 ] holdenk commented on SPARK-14654: - Even if merge isn't supposed to be called by the end user it would be

[jira] [Commented] (SPARK-14654) New accumulator API

2016-04-23 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15255363#comment-15255363 ] Reynold Xin commented on SPARK-14654: - I see. The merge function isn't supposed to be called by the

[jira] [Commented] (SPARK-14654) New accumulator API

2016-04-23 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15255359#comment-15255359 ] holdenk commented on SPARK-14654: - So ACC isn't the internal buffer type, rather its the type of the

[jira] [Commented] (SPARK-14654) New accumulator API

2016-04-23 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15255357#comment-15255357 ] Reynold Xin commented on SPARK-14654: - No it can't be private spark because it needs to be

[jira] [Commented] (SPARK-14654) New accumulator API

2016-04-23 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15255356#comment-15255356 ] holdenk commented on SPARK-14654: - Since we're talking about average accumulators anyways - what would

[jira] [Commented] (SPARK-14654) New accumulator API

2016-04-23 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15255355#comment-15255355 ] holdenk commented on SPARK-14654: - If were only going to do Long and Double for the easy creation on the

[jira] [Commented] (SPARK-14654) New accumulator API

2016-04-23 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15255312#comment-15255312 ] Reynold Xin commented on SPARK-14654: - I don't get what you are trying to accomplish. It seems like

[jira] [Commented] (SPARK-14654) New accumulator API

2016-04-23 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15255291#comment-15255291 ] holdenk commented on SPARK-14654: - You wouldn't know if you want a counter or overage, but the same

[jira] [Commented] (SPARK-14654) New accumulator API

2016-04-23 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15255169#comment-15255169 ] Reynold Xin commented on SPARK-14654: - How do you know for double whether we'd want a counter or an

[jira] [Commented] (SPARK-14654) New accumulator API

2016-04-23 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15255163#comment-15255163 ] holdenk commented on SPARK-14654: - [~rxin] its true, but compared to the current proposed new API we

[jira] [Commented] (SPARK-14654) New accumulator API

2016-04-22 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15255159#comment-15255159 ] Reynold Xin commented on SPARK-14654: - That only works for the built-in ones and can't work if there

[jira] [Commented] (SPARK-14654) New accumulator API

2016-04-22 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15255123#comment-15255123 ] holdenk commented on SPARK-14654: - Giving it a bit of thought on the flight, is there a reason why we

[jira] [Commented] (SPARK-14654) New accumulator API

2016-04-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15254053#comment-15254053 ] Apache Spark commented on SPARK-14654: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Commented] (SPARK-14654) New accumulator API

2016-04-21 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15253000#comment-15253000 ] holdenk commented on SPARK-14654: - Looks like a nice clean API, out of interest what is `dedup` for? Is

[jira] [Commented] (SPARK-14654) New accumulator API

2016-04-15 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15242540#comment-15242540 ] Reynold Xin commented on SPARK-14654: - cc [~holdenk] and [~imranr] > New accumulator API >

[jira] [Commented] (SPARK-14654) New accumulator API

2016-04-15 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15242536#comment-15242536 ] Reynold Xin commented on SPARK-14654: - Note that one challenge is that we most likely don't want to