Re: Possible to push sub-queries down into the DataSource impl?

yes, that's exactly what I was looking for, thanks for the pointer ;-) On Thu, Jul 28, 2016 at 1:07 AM, Takeshi Yamamuro wrote: > Hi, > > Have you seen this ticket? > https://issues.apache.org/jira/browse/SPARK-12449 > > // maropu > > On Thu, Jul 28, 2016 at 2:13 AM,

Re: Possible to push sub-queries down into the DataSource impl?

Hi, Have you seen this ticket? https://issues.apache.org/jira/browse/SPARK-12449 // maropu On Thu, Jul 28, 2016 at 2:13 AM, Timothy Potter wrote: > I'm not looking for a one-off solution for a specific query that can > be solved on the client side as you suggest, but

Re: Possible to push sub-queries down into the DataSource impl?

I'm not looking for a one-off solution for a specific query that can be solved on the client side as you suggest, but rather a generic solution that can be implemented within the DataSource impl itself when it knows a sub-query can be pushed down into the engine. In other words, I'd like to

Re: Possible to push sub-queries down into the DataSource impl?

Why don't you create a dataframe filtered, map it as temporary table and then use it in your query? You can also cache it, of multiple queries on the same inner queries are requested. Il mercoledì 27 luglio 2016, Timothy Potter ha scritto: > Take this simple join: > >

Possible to push sub-queries down into the DataSource impl?

Take this simple join: SELECT m.title as title, solr.aggCount as aggCount FROM movies m INNER JOIN (SELECT movie_id, COUNT(*) as aggCount FROM ratings WHERE rating >= 4 GROUP BY movie_id ORDER BY aggCount desc LIMIT 10) as solr ON solr.movie_id = m.movie_id ORDER BY aggCount DESC I would like