I have been trying to access Druid from Spark via Calcite JDBC, but somehow sql statement generated by Spark causes exceptions. Not sure if it is Calcite JDBC or Druid related. Have you see, or do you think Spark -> Calcite JDBC -> Druid is a right way to connect Spark and Druid?
Thanks Herman. > On Nov 17, 2016, at 19:02, Julian Hyde <jh...@apache.org> wrote: > > Convertlets have a similar effect to planner rules (albeit they act on scalar > expressions, not relational expressions) so people should be able to change > the set of active convertlets. > > Would you like to propose a change that makes the convertlet table pluggable? > Maybe as part of FrameworkConfig? Regardless, please log a JIRA to track this. > > And by the way, RexImpTable, which defines how operators are implemented by > generating java code, should also be pluggable. It’s been on my mind for a > long time to allow the “engine” — related to the data format, and how code is > generated to access fields and evaluate expressions and operators — to be > pluggable. > > Regarding whether the JDBC driver is the right way to embed Calcite. There’s > no easy answer. You might want to embed Calcite as a library in your own > server (as Drill and Hive do). Or you might want to make yourself just an > adapter that runs inside a Calcite JDBC server (as the CSV adapter does). Or > something in the middle, like what Phoenix does: using Calcite for JDBC, SQL, > planning, but with your own metadata and runtime engine. > > As long as you build the valuable stuff into planner rules, new relational > operators (if necessary) and use the schema SPI, you should be able to change > packaging in the future. > > Julian > > > > >> On Nov 17, 2016, at 1:59 PM, Gian Merlino <g...@imply.io> wrote: >> >> Hey Calcites, >> >> I'm working on embedding Calcite into Druid (http://druid.io/, >> https://github.com/druid-io/druid/pull/3682) and am running into a problem >> that is making me wonder if the approach I'm using makes sense. >> >> Consider the expression EXTRACT(YEAR FROM __time). Calcite has a standard >> convertlet rule "convertExtract" that changes this into some arithmetic on >> __time casted to an int type. But Druid has some builtin functions to do >> this, and I'd rather use those than arithmetic (for a bunch of reasons). >> Ideally, in my RelOptRules that convert Calcite rels to Druid queries, I'd >> see the EXTRACT as a normal RexCall with the time flag and an expression to >> apply it to. That's a lot easier to translate than the arithmetic stuff, >> which I'd have to pattern match and undo first before translating. >> >> So the problem I have is that I want to disable convertExtract, but I don't >> see a way to do that or to swap out the convertlet table. >> >> The code I'm using to set up a connection is: >> >> public CalciteConnection createCalciteConnection( >> final DruidSchema druidSchema >> ) throws SQLException >> { >> final Properties props = new Properties(); >> props.setProperty("caseSensitive", "true"); >> props.setProperty("unquotedCasing", "UNCHANGED"); >> final Connection connection = >> DriverManager.getConnection("jdbc:calcite:", props); >> final CalciteConnection calciteConnection = >> connection.unwrap(CalciteConnection.class); >> calciteConnection.getRootSchema().setCacheEnabled(false); >> calciteConnection.getRootSchema().add(DRUID_SCHEMA_NAME, druidSchema); >> return calciteConnection; >> } >> >> This CalciteConnection is then used by the Druid HTTP server to offer a SQL >> API. >> >> Is there some way to swap out the convertlet table that I'm missing? >> >> Also, just in general, am I going about this the right way? Is using the >> JDBC driver the right way to embed Calcite? Or should I be calling into it >> at some lower level? >> >> Thanks! >> >> Gian >
