[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17534937#comment-17534937 ] Chris Kimmel commented on SPARK-17556: -- [~jlaskowski] I believe the problem is that "Currently in

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17533465#comment-17533465 ] Jacek Laskowski commented on SPARK-17556: - Given: # "I'm running a large query with over

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17533157#comment-17533157 ] Chris Kimmel commented on SPARK-17556: -- Bumping this. I'm running a large query with over 100,000

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198571#comment-17198571 ] L. C. Hsieh commented on SPARK-17556: - We will recently try to pick this up again. > Executor side

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16788206#comment-16788206 ] Eyal Farago commented on SPARK-17556: - why was this abandoned? [~viirya]'s pull request seems

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16770481#comment-16770481 ] t oo commented on SPARK-17556: -- please don't leave us [~scwf] > Executor side broadcast for broadcast

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15937707#comment-15937707 ] Liang-Chi Hsieh commented on SPARK-17556: - We may need to change the Target Version/s for this.

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15537544#comment-15537544 ] Liang-Chi Hsieh commented on SPARK-17556: - Update the design document again to address some

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534774#comment-15534774 ] Liang-Chi Hsieh commented on SPARK-17556: - Update the design document to add more description for

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15529476#comment-15529476 ] Liang-Chi Hsieh commented on SPARK-17556: - For example, assume we have 3 executors and the

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15529439#comment-15529439 ] Liang-Chi Hsieh commented on SPARK-17556: - Actually I am hesitant to add the feature to handle

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15522817#comment-15522817 ] Apache Spark commented on SPARK-17556: -- User 'scwf' has created a pull request for this issue:

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15519497#comment-15519497 ] Yan commented on SPARK-17556: - For 2), I think BitTorrent won't help in the case of all-to-all transfers,

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15518790#comment-15518790 ] Liang-Chi Hsieh commented on SPARK-17556: - For 1). It is true only if your driver is outside of

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15518598#comment-15518598 ] Yan commented on SPARK-17556: - A few comments of mine are as follows: 1) The "one-executor collection"

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516772#comment-15516772 ] Fei Wang commented on SPARK-17556: -- [~viirya] in this case how about notify driver to re-persist the

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516732#comment-15516732 ] Fei Wang commented on SPARK-17556: -- That's a good point! In your solution, the broadcast rdd must

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516668#comment-15516668 ] Liang-Chi Hsieh commented on SPARK-17556: - No. It doesn't. I think the point is not only the

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516639#comment-15516639 ] Fei Wang commented on SPARK-17556: -- Yes, the main different is is does not introduce overhead to driver,

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516577#comment-15516577 ] Liang-Chi Hsieh commented on SPARK-17556: - In other words, from the jira description we say "the

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516568#comment-15516568 ] Liang-Chi Hsieh commented on SPARK-17556: - OK. You create the broadcast object on one executor.

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516545#comment-15516545 ] Fei Wang commented on SPARK-17556: -- Not correct, I just collect the broadcast ref to the driver but not

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516546#comment-15516546 ] Fei Wang commented on SPARK-17556: -- Not correct, I just collect the broadcast ref to the driver but not

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516490#comment-15516490 ] Liang-Chi Hsieh commented on SPARK-17556: - [~Fei Wang] I quickly go through your design doc.

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516478#comment-15516478 ] Liang-Chi Hsieh commented on SPARK-17556: - [~scwf]I already submitted a PR for this. Can you also

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516386#comment-15516386 ] Fei Wang commented on SPARK-17556: -- [~rxin] attached a design doc for the executor based broadcast. Will

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509228#comment-15509228 ] Apache Spark commented on SPARK-17556: -- User 'viirya' has created a pull request for this issue: