Hi, Josh, Thank you very much for your response. We are trying to push as many ops down to druid as possible. While we are tuning the performance to make it as good as directly call druid api, we still face the join problem though. It’ll be nice if we could make the spark adaptor work. It will make calcite much more powerful than it can be.
I looked up the code base a little bit, the spark adaptor code has not had any serious commits for years. I’m just curious what is the roadmap for the spark adaptor. -JD > On Jun 20, 2017, at 1:29 PM, Josh Elser <[email protected]> wrote: > > Hi JD, > > I don't have a lot of expertise to comment definitively, but let me try to > give you a response with what limited knowledge I do have :) > > I would venture a guess that the calcite-druid adapter could be extended to > push more operations down to Druid in order to do less in the Calcite layer > itself. > > I'm not aware of any sort of "production ready" example of using spark as the > execution engine, either. Maybe there are some examples people have elsewhere > -- asking that question on this mailing list as you have is a good way to get > help. Hopefully someone with more experience than I have can comment! > > - Josh > > On 6/19/17 10:36 PM, JD Zheng wrote: >> Hi, >> We are using calcite druid adaptor to query data stored in druid. But lots >> of operations are not pushed down to druid, the built-in enumerable >> execution engine becomes the bottleneck for some of the queries that’s not >> pushed down. As we also have some use-case to join data in druid from >> outside data source, it seems using spark execution engine is the way to go. >> Has anyone used the spark-adaptor to make spark as the calcite execution >> engine? How mature is the spark-adaptor? Is there any document about how to >> use the spark-adaptor? How does it compare to using hive? >> Thanks, >> -JD
