[jira] [Commented] (SPARK-12449) Pushing down arbitrary logical plans to data sources

[ https://issues.apache.org/jira/browse/SPARK-12449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575044#comment-16575044 ] Kyle Prifogle commented on SPARK-12449: --- [~oae]  as far as I can tell the issue lives on here: 

[jira] [Commented] (SPARK-12449) Pushing down arbitrary logical plans to data sources

[ https://issues.apache.org/jira/browse/SPARK-12449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574770#comment-16574770 ] Johannes Zillmann commented on SPARK-12449: --- I'm a bit confused. Reading

[jira] [Commented] (SPARK-12449) Pushing down arbitrary logical plans to data sources

[ https://issues.apache.org/jira/browse/SPARK-12449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548140#comment-16548140 ] Kyle Prifogle commented on SPARK-12449: --- What happened to this initiative?  I came here trying to

[jira] [Commented] (SPARK-12449) Pushing down arbitrary logical plans to data sources

[ https://issues.apache.org/jira/browse/SPARK-12449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156622#comment-16156622 ] Andrew Ash commented on SPARK-12449: [~velvia] I'm not involved with the CatalystSource or SAP

[jira] [Commented] (SPARK-12449) Pushing down arbitrary logical plans to data sources

[ https://issues.apache.org/jira/browse/SPARK-12449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16138856#comment-16138856 ] Evan Chan commented on SPARK-12449: --- Andrew and others: Is there a plan to make this CatalystSource

[jira] [Commented] (SPARK-12449) Pushing down arbitrary logical plans to data sources

[ https://issues.apache.org/jira/browse/SPARK-12449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16138501#comment-16138501 ] Andrew Ash commented on SPARK-12449: Relevant slides:

[jira] [Commented] (SPARK-12449) Pushing down arbitrary logical plans to data sources

[ https://issues.apache.org/jira/browse/SPARK-12449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153931#comment-15153931 ] Stephan Kessler commented on SPARK-12449: - To keep the interfaces stable, the possibility of

[jira] [Commented] (SPARK-12449) Pushing down arbitrary logical plans to data sources

[ https://issues.apache.org/jira/browse/SPARK-12449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153920#comment-15153920 ] Stephan Kessler commented on SPARK-12449: - In our github repo

[jira] [Commented] (SPARK-12449) Pushing down arbitrary logical plans to data sources

[ https://issues.apache.org/jira/browse/SPARK-12449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15151904#comment-15151904 ] Max Seiden commented on SPARK-12449: [~stephank85] Dropping the partitioned bit makes sense for a

[jira] [Commented] (SPARK-12449) Pushing down arbitrary logical plans to data sources

[ https://issues.apache.org/jira/browse/SPARK-12449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15151880#comment-15151880 ] Max Seiden commented on SPARK-12449: Yea, that seems to be the case. There's code in the

[jira] [Commented] (SPARK-12449) Pushing down arbitrary logical plans to data sources

[ https://issues.apache.org/jira/browse/SPARK-12449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15151823#comment-15151823 ] Evan Chan commented on SPARK-12449: --- I think in the case of sources.Expressions, by the time they are

[jira] [Commented] (SPARK-12449) Pushing down arbitrary logical plans to data sources

[ https://issues.apache.org/jira/browse/SPARK-12449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15151368#comment-15151368 ] Max Seiden commented on SPARK-12449: Very interested in checking out that PR! It would be prudent to

[jira] [Commented] (SPARK-12449) Pushing down arbitrary logical plans to data sources

[ https://issues.apache.org/jira/browse/SPARK-12449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15151345#comment-15151345 ] Evan Chan commented on SPARK-12449: --- [~stephank85] would you have any code to share? :D > Pushing

[jira] [Commented] (SPARK-12449) Pushing down arbitrary logical plans to data sources

[ https://issues.apache.org/jira/browse/SPARK-12449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15151339#comment-15151339 ] Stephan Kessler commented on SPARK-12449: - [~maxseiden] good idea! In order to simplify things

[jira] [Commented] (SPARK-12449) Pushing down arbitrary logical plans to data sources

[ https://issues.apache.org/jira/browse/SPARK-12449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15151268#comment-15151268 ] Evan Chan commented on SPARK-12449: --- I agree with [~maxseiden] on a gradual approach to push more down

[jira] [Commented] (SPARK-12449) Pushing down arbitrary logical plans to data sources

[ https://issues.apache.org/jira/browse/SPARK-12449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15151116#comment-15151116 ] Max Seiden commented on SPARK-12449: [~rxin] Given that predicate pushdown via `sources.Filter` is

[jira] [Commented] (SPARK-12449) Pushing down arbitrary logical plans to data sources

[ https://issues.apache.org/jira/browse/SPARK-12449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15150372#comment-15150372 ] Takeshi Yamamuro commented on SPARK-12449: -- I agree though, there are many jira tickets(12506,

[jira] [Commented] (SPARK-12449) Pushing down arbitrary logical plans to data sources

[ https://issues.apache.org/jira/browse/SPARK-12449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15141243#comment-15141243 ] Evan Chan commented on SPARK-12449: --- [~rxin] I agree with [~stephank85] and others that this would be a

[jira] [Commented] (SPARK-12449) Pushing down arbitrary logical plans to data sources

[ https://issues.apache.org/jira/browse/SPARK-12449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093533#comment-15093533 ] Santiago M. Mola commented on SPARK-12449: -- Implementing this interface or an equivalent one

[jira] [Commented] (SPARK-12449) Pushing down arbitrary logical plans to data sources

[ https://issues.apache.org/jira/browse/SPARK-12449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15088334#comment-15088334 ] Yan commented on SPARK-12449: - Stephan, thanks for your explanations and questions. My answers are as

[jira] [Commented] (SPARK-12449) Pushing down arbitrary logical plans to data sources

[ https://issues.apache.org/jira/browse/SPARK-12449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15087983#comment-15087983 ] Yan commented on SPARK-12449: - Stephan, By "partial op" I mean, for instance, partial map-side aggregation.

[jira] [Commented] (SPARK-12449) Pushing down arbitrary logical plans to data sources

[ https://issues.apache.org/jira/browse/SPARK-12449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15080867#comment-15080867 ] Stephan Kessler commented on SPARK-12449: - Hi Yan, this is only true for the

[jira] [Commented] (SPARK-12449) Pushing down arbitrary logical plans to data sources

[ https://issues.apache.org/jira/browse/SPARK-12449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15080858#comment-15080858 ] Stephan Kessler commented on SPARK-12449: - Thank you for your thoughts - unfortunately i did not

[jira] [Commented] (SPARK-12449) Pushing down arbitrary logical plans to data sources

[ https://issues.apache.org/jira/browse/SPARK-12449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15080835#comment-15080835 ] Stephan Kessler commented on SPARK-12449: - Hi Reynold, i fully agree that we need the

[jira] [Commented] (SPARK-12449) Pushing down arbitrary logical plans to data sources

[ https://issues.apache.org/jira/browse/SPARK-12449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070062#comment-15070062 ] Santiago M. Mola commented on SPARK-12449: -- The physical plan would not be consumed by data

[jira] [Commented] (SPARK-12449) Pushing down arbitrary logical plans to data sources

[ https://issues.apache.org/jira/browse/SPARK-12449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070027#comment-15070027 ] Reynold Xin commented on SPARK-12449: - This is great, but our internal logical/physical plan

[jira] [Commented] (SPARK-12449) Pushing down arbitrary logical plans to data sources

[ https://issues.apache.org/jira/browse/SPARK-12449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070088#comment-15070088 ] Yan commented on SPARK-12449: - To push down map-side (partial) ops, Physical Plan would need to be checked I

[jira] [Commented] (SPARK-12449) Pushing down arbitrary logical plans to data sources

[ https://issues.apache.org/jira/browse/SPARK-12449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070135#comment-15070135 ] Yan commented on SPARK-12449: - Conceivably, if only logical plan is used, the Spark SQL execution would

[jira] [Commented] (SPARK-12449) Pushing down arbitrary logical plans to data sources

[ https://issues.apache.org/jira/browse/SPARK-12449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070100#comment-15070100 ] Santiago M. Mola commented on SPARK-12449: -- Well, at least with the implementation presented at

[jira] [Commented] (SPARK-12449) Pushing down arbitrary logical plans to data sources

[ https://issues.apache.org/jira/browse/SPARK-12449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15068447#comment-15068447 ] Yan commented on SPARK-12449: - A few thoughts on the capabilities of this "CatalystSource Interface": 1)

[jira] [Commented] (SPARK-12449) Pushing down arbitrary logical plans to data sources

[ https://issues.apache.org/jira/browse/SPARK-12449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15066407#comment-15066407 ] Stephan Kessler commented on SPARK-12449: - Added the design document. Looking forward for the

[jira] [Commented] (SPARK-12449) Pushing down arbitrary logical plans to data sources

[ https://issues.apache.org/jira/browse/SPARK-12449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15066655#comment-15066655 ] Santiago M. Mola commented on SPARK-12449: -- At Stratio we are interested in this kind of