Hi JD,
I don't have a lot of expertise to comment definitively, but let me try
to give you a response with what limited knowledge I do have :)
I would venture a guess that the calcite-druid adapter could be extended
to push more operations down to Druid in order to do less in the Calcite
layer itself.
I'm not aware of any sort of "production ready" example of using spark
as the execution engine, either. Maybe there are some examples people
have elsewhere -- asking that question on this mailing list as you have
is a good way to get help. Hopefully someone with more experience than I
have can comment!
- Josh
On 6/19/17 10:36 PM, JD Zheng wrote:
Hi,
We are using calcite druid adaptor to query data stored in druid. But lots of
operations are not pushed down to druid, the built-in enumerable execution
engine becomes the bottleneck for some of the queries that’s not pushed down.
As we also have some use-case to join data in druid from outside data source,
it seems using spark execution engine is the way to go.
Has anyone used the spark-adaptor to make spark as the calcite execution
engine? How mature is the spark-adaptor? Is there any document about how to use
the spark-adaptor? How does it compare to using hive?
Thanks,
-JD
