[jira] [Commented] (SPARK-12656) Rewrite Intersect phyiscal plan using semi-join
[ https://issues.apache.org/jira/browse/SPARK-12656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085779#comment-15085779 ] Xiao Li commented on SPARK-12656: - Will do it. Thank you! > Rewrite Intersect phyiscal plan using semi-join > --- > > Key: SPARK-12656 > URL: https://issues.apache.org/jira/browse/SPARK-12656 > Project: Spark > Issue Type: Improvement > Components: SQL >Reporter: Reynold Xin > > Our current Intersect physical operator simply delegates to RDD.intersect. We > should remove the Intersect physical operator and simply transform a logical > intersect into a semi-join. This way, we can take advantage of all the > benefits of join implementations (e.g. managed memory, code generation, > broadcast joins). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-12656) Rewrite Intersect phyiscal plan using semi-join
[ https://issues.apache.org/jira/browse/SPARK-12656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15086782#comment-15086782 ] Xiao Li commented on SPARK-12656: - Starting it. Will submit a PR tonight. Thanks! > Rewrite Intersect phyiscal plan using semi-join > --- > > Key: SPARK-12656 > URL: https://issues.apache.org/jira/browse/SPARK-12656 > Project: Spark > Issue Type: Improvement > Components: SQL >Reporter: Reynold Xin > > Our current Intersect physical operator simply delegates to RDD.intersect. We > should remove the Intersect physical operator and simply transform a logical > intersect into a semi-join. This way, we can take advantage of all the > benefits of join implementations (e.g. managed memory, code generation, > broadcast joins). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-12656) Rewrite Intersect phyiscal plan using semi-join
[ https://issues.apache.org/jira/browse/SPARK-12656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15086856#comment-15086856 ] Apache Spark commented on SPARK-12656: -- User 'gatorsmile' has created a pull request for this issue: https://github.com/apache/spark/pull/10630 > Rewrite Intersect phyiscal plan using semi-join > --- > > Key: SPARK-12656 > URL: https://issues.apache.org/jira/browse/SPARK-12656 > Project: Spark > Issue Type: Improvement > Components: SQL >Reporter: Reynold Xin > > Our current Intersect physical operator simply delegates to RDD.intersect. We > should remove the Intersect physical operator and simply transform a logical > intersect into a semi-join. This way, we can take advantage of all the > benefits of join implementations (e.g. managed memory, code generation, > broadcast joins). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-12656) Rewrite Intersect phyiscal plan using semi-join
[ https://issues.apache.org/jira/browse/SPARK-12656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15083858#comment-15083858 ] Reynold Xin commented on SPARK-12656: - cc [~smilegator] one more thing for you to pick up if you are interested. > Rewrite Intersect phyiscal plan using semi-join > --- > > Key: SPARK-12656 > URL: https://issues.apache.org/jira/browse/SPARK-12656 > Project: Spark > Issue Type: Improvement > Components: SQL >Reporter: Reynold Xin > > Our current Intersect physical operator simply delegates to RDD.intersect. We > should remove the Intersect physical operator and simply transform a logical > intersect into a semi-join. This way, we can take advantage of all the > benefits of join implementations (e.g. managed memory, code generation, > broadcast joins). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
