[ https://issues.apache.org/jira/browse/CALCITE-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364583#comment-16364583 ]
Jesus Camacho Rodriguez commented on CALCITE-2170: -------------------------------------------------- [~julianhyde], will you be taking a look at the PR? I already did one pass, I will be doing another one today and then we would like to merge it. Thanks > Use Druid Expressions capabilities to improve the amount of work that can be > pushed to Druid > -------------------------------------------------------------------------------------------- > > Key: CALCITE-2170 > URL: https://issues.apache.org/jira/browse/CALCITE-2170 > Project: Calcite > Issue Type: New Feature > Components: druid > Reporter: slim bouguerra > Assignee: slim bouguerra > Priority: Major > > Druid 0.11 has newly built in capabilities called Expressions that can be > used to push expression like projects/aggregates/filters. > In order to leverage this new feature, some changes need to be done to the > Druid Calcite adapter. > This is a link to the current supported functions and expressions in Druid > [http://druid.io/docs/latest/misc/math-expr.html] > As you can see from the Docs an expression can be an actual tree of operators, > Expression can be used with Filters, Projects, Aggregates, PostAggregates and > Having filters. For Filters will have new Filter kind called Filter > expression. > FYI, you might ask can we push everything as Expression Filter the short > answer > is no because, other kinds of Druid filters perform better when used, Hence > Expression filter is a plan B sort of thing. In order to push expression as > Projects and Aggregates we will be using Expression based Virtual Columns. > The major change is the merging of the logic of pushdown verification code and > the Translation of RexCall/RexNode to Druid Json, native physical language. > The > main drive behind this redesign is the fact that in order to check if we can > push down a tree of expressions to Druid we have to compute the Druid > Expression > String anyway. Thus instead of having 2 different code paths, one for pushdown > validation and one for Json generation we can have one function that does > both. > For instance instead of having one code path to test and check if a given > filter > can be pushed or not and then having a translation layer code, will have > one function that either returns a valid Druid Filter or null if it is not > possible to pushdown. The same idea will be applied to how we push Projects > and > Aggregates, Post Aggregates and Sort. > Here are the main elements/Classes of the new design. First will be merging > the logic of > Translation of Literals/InputRex/RexCall to a Druid physical representation. > Translate leaf RexNode to Valid pair Druid Column + Extraction functions if > possible > {code:java} > /** > * @param rexNode leaf Input Ref to Druid Column > * @param rowType row type > * @param druidQuery druid query > * > * @return {@link Pair} of Column name and Extraction Function on the top of > the input ref or > * {@link Pair of(null, null)} when can not translate to valid Druid column > */ > protected static Pair<String, ExtractionFunction> toDruidColumn(RexNode > rexNode, > RelDataType rowType, DruidQuery druidQuery > ) > {code} > In the other hand, in order to Convert Literals to Druid Literals will > introduce > {code:java} > /** > * @param rexNode rexNode to translate to Druid literal equivalante > * @param rowType rowType associated to rexNode > * @param druidQuery druid Query > * > * @return non null string or null if it can not translate to valid Druid > equivalent > */ > @Nullable > private static String toDruidLiteral(RexNode rexNode, RelDataType rowType, > DruidQuery druidQuery > ) > {code} > Main new functions used to pushdown nodes and Druid Json generation. > Filter pushdown verification and generates is done via > {code:java} > org.apache.calcite.adapter.druid.DruidJsonFilter#toDruidFilters > {code} > For project pushdown added > {code:java} > org.apache.calcite.adapter.druid.DruidQuery#computeProjectAsScan. > {code} > For Grouping pushdown added > {code:java} > org.apache.calcite.adapter.druid.DruidQuery#computeProjectGroupSet. > {code} > For Aggregation pushdown added > {code:java} > org.apache.calcite.adapter.druid.DruidQuery#computeDruidJsonAgg > {code} > For sort pushdown added > {code:java} > org.apache.calcite.adapter.druid.DruidQuery#computeSort\{code} > Pushing of PostAggregates will be using Expression post Aggregates and use > {code} > org.apache.calcite.adapter.druid.DruidExpressions#toDruidExpression\{code} > to generate expression > For Expression computation most of the work is done here > {code:java} > org.apache.calcite.adapter.druid.DruidExpressions#toDruidExpression\{code} > This static function generates Druid String expression out of a given RexNode > or > returns null if not possible. > {code} > @Nullable > public static String toDruidExpression( > final RexNode rexNode, > final RelDataType inputRowType, > final DruidQuery druidRel > ) > {code:java} > In order to support various kind of expressions added the following interface > {code} > org.apache.calcite.adapter.druid.DruidSqlOperatorConverter\{code} > Thus user can implement custom expression converter based on the SqlOperator > syntax and signature. > {code:java} > public interface DruidSqlOperatorConverter { > /** > * Returns the calcite SQL operator corresponding to Druid operator. > * > * @return operator > */ > SqlOperator calciteOperator(); > /** > * Translate rexNode to valid Druid expression. > * @param rexNode rexNode to translate to Druid expression > * @param rowType row type associated with rexNode > * @param druidQuery druid query used to figure out configs/fields related > like timeZone > * > * @return valid Druid expression or null if it can not convert the rexNode > */ > @Nullable String toDruidExpression(RexNode rexNode, RelDataType rowType, > DruidQuery druidQuery); > } > {code} > The Druid Query Class will provide > org.apache.calcite.adapter.druid.DruidQuery#getOperatorConversionMap which is > a > map of SqlOperator to DruidSqlOperatorConverter. > Any feedback is welcome. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)