Re: Embedding Calcite, adjusting convertlets

[Gian Merlino]

This was totally user error, I was ignoring "fields" and "collation" of the
RelRoot. Passing "collation" to planner.transform stopped it from removing
the sort.

Gian

On Wed, Dec 14, 2016 at 3:21 PM, Gian Merlino  wrote:

> I don't think so, since my test query (SELECT dim1 FROM s.foo GROUP BY
> dim1 ORDER BY dim1 DESC) is sorting on a column that is also projected.
>
> Gian
>
> On Wed, Dec 14, 2016 at 11:58 AM, Julian Hyde  wrote:
>
>> Are you running into some variant of the problems that inspired
>> https://issues.apache.org/jira/browse/CALCITE-819: <
>> https://issues.apache.org/jira/browse/CALCITE-819:> at the root of the
>> tree, columns that are not projected are removed, and if the desired sort
>> order involves non-projected columns, the desired sort order is forgotten).
>>
>> > On Dec 14, 2016, at 11:19 AM, Gian Merlino  wrote:
>> >
>> > Ah, thanks. So if that sort of thing is not a smoking gun, do you have
>> an
>> > idea about where I should look next? If not I'll keep poking around.
>> >
>> > Gian
>> >
>> > On Wed, Dec 14, 2016 at 11:06 AM, Julian Hyde  wrote:
>> >
>> >>> - But its "set" field points to a RelSet with "rels" that _don't_ have
>> >>> _any_ collation traits.
>> >>
>> >> That’s OK. A “subset” (RelSubset) is a collection of RelNodes that are
>> >> logically and physically equivalent (same results, same physical
>> >> properties) whereas a “set” (RelSet) is a collection of RelNodes that
>> are
>> >> logically equivalent.
>> >>
>> >> A set can therefore be considered to be a collection of subsets, each
>> of
>> >> which contains RelNodes. And it used to be implemented that way, but in
>> >> https://issues.apache.org/jira/browse/CALCITE-88 <
>> >> https://issues.apache.org/jira/browse/CALCITE-88> we introduced
>> collation
>> >> as a trait, and that made subsets non-disjoint (a RelNode can be
>> sorted on
>> >> (x, y), and also on (x), and also on (), and also on (z)) so we made
>> >> RelSubset just a view onto a RelSet, filtering the list of RelNodes
>> >> according to the ones that have (“subsume”) the desired traits.
>> >>
>> >> Julian
>> >>
>> >>
>> >>> On Dec 14, 2016, at 10:45 AM, Gian Merlino  wrote:
>> >>>
>> >>> I spent some more time looking into (3) and found that when I had
>> things
>> >>> going through the Planner rather than the JDBC driver, SortRemoveRule
>> was
>> >>> removing sorts when it shouldn't have been. This happens even for
>> simple
>> >>> queries like "SELECT dim1 FROM s.foo GROUP BY dim1 ORDER BY dim1
>> >>> DESC". Removing SortRemoveRule from the planner fixed the broken
>> tests on
>> >>> my end.
>> >>>
>> >>> I dug into that a bit and saw that the call to
>> "convert(sort.getInput(),
>> >>> traits)" in SortRemoveRule was returning a RelSubset that looked a bit
>> >>> funny in the debugger:
>> >>>
>> >>> - The RelSubset's "traitSet" _does_ have the proper collation trait.
>> >>> - But its "set" field points to a RelSet with "rels" that _don't_ have
>> >>> _any_ collation traits.
>> >>>
>> >>> From what I understand that causes Calcite to treat the unsorted and
>> >> sorted
>> >>> rels as equivalent when they in fact aren't. I'm still not sure if
>> this
>> >> is
>> >>> a Calcite bug or user error on my part… I'll keep looking into it
>> unless
>> >>> someone has any bright ideas.
>> >>>
>> >>> fwiw, my Planner construction looks like this:
>> >>>
>> >>>   final FrameworkConfig frameworkConfig = Frameworks
>> >>>   .newConfigBuilder()
>> >>>   .parserConfig(
>> >>>   SqlParser.configBuilder()
>> >>>.setCaseSensitive(true)
>> >>>.setUnquotedCasing(Casing.UNCHANGED)
>> >>>.build()
>> >>>   )
>> >>>   .defaultSchema(rootSchema)
>> >>>   .traitDefs(ConventionTraitDef.INSTANCE,
>> >>> RelCollationTraitDef.INSTANCE)
>> >>>   .programs(Programs.ofRules(myRules))
>> >>>   .executor(new RexExecutorImpl(Schemas.createDataContext(null)))
>> >>>   .context(Contexts.EMPTY_CONTEXT)
>> >>>   .build();
>> >>>
>> >>>   return Frameworks.getPlanner(frameworkConfig);
>> >>>
>> >>> Gian
>> >>>
>> >>> On Sat, Dec 3, 2016 at 5:53 PM, Gian Merlino  wrote:
>> >>>
>>  Sure, I added those first two to the ticket.
>> 
>>  I don't think those are happening with (3) but I'll double check next
>> >> time
>>  I take a look at using the Planner.
>> 
>>  Gian
>> 
>>  On Fri, Dec 2, 2016 at 12:20 PM, Julian Hyde 
>> wrote:
>> 
>> > Can you please add (1) and (2) to https://issues.apache.org/jira
>> > /browse/CALCITE-1525 > >> a/browse/CALCITE-1525>,
>> > which deals with the whole issue of using “Planner” within the JDBC
>> >> driver,
>> > so we can be consistent.
>> >
>> > (3) doesn’t look likely to be related. Do your queries have UNION or
>> > other set-ops? Are you sorting 

Re: Embedding Calcite, adjusting convertlets

[Gian Merlino]

I don't think so, since my test query (SELECT dim1 FROM s.foo GROUP BY dim1
ORDER BY dim1 DESC) is sorting on a column that is also projected.

Gian

On Wed, Dec 14, 2016 at 11:58 AM, Julian Hyde  wrote:

> Are you running into some variant of the problems that inspired
> https://issues.apache.org/jira/browse/CALCITE-819: <
> https://issues.apache.org/jira/browse/CALCITE-819:> at the root of the
> tree, columns that are not projected are removed, and if the desired sort
> order involves non-projected columns, the desired sort order is forgotten).
>
> > On Dec 14, 2016, at 11:19 AM, Gian Merlino  wrote:
> >
> > Ah, thanks. So if that sort of thing is not a smoking gun, do you have an
> > idea about where I should look next? If not I'll keep poking around.
> >
> > Gian
> >
> > On Wed, Dec 14, 2016 at 11:06 AM, Julian Hyde  wrote:
> >
> >>> - But its "set" field points to a RelSet with "rels" that _don't_ have
> >>> _any_ collation traits.
> >>
> >> That’s OK. A “subset” (RelSubset) is a collection of RelNodes that are
> >> logically and physically equivalent (same results, same physical
> >> properties) whereas a “set” (RelSet) is a collection of RelNodes that
> are
> >> logically equivalent.
> >>
> >> A set can therefore be considered to be a collection of subsets, each of
> >> which contains RelNodes. And it used to be implemented that way, but in
> >> https://issues.apache.org/jira/browse/CALCITE-88 <
> >> https://issues.apache.org/jira/browse/CALCITE-88> we introduced
> collation
> >> as a trait, and that made subsets non-disjoint (a RelNode can be sorted
> on
> >> (x, y), and also on (x), and also on (), and also on (z)) so we made
> >> RelSubset just a view onto a RelSet, filtering the list of RelNodes
> >> according to the ones that have (“subsume”) the desired traits.
> >>
> >> Julian
> >>
> >>
> >>> On Dec 14, 2016, at 10:45 AM, Gian Merlino  wrote:
> >>>
> >>> I spent some more time looking into (3) and found that when I had
> things
> >>> going through the Planner rather than the JDBC driver, SortRemoveRule
> was
> >>> removing sorts when it shouldn't have been. This happens even for
> simple
> >>> queries like "SELECT dim1 FROM s.foo GROUP BY dim1 ORDER BY dim1
> >>> DESC". Removing SortRemoveRule from the planner fixed the broken tests
> on
> >>> my end.
> >>>
> >>> I dug into that a bit and saw that the call to
> "convert(sort.getInput(),
> >>> traits)" in SortRemoveRule was returning a RelSubset that looked a bit
> >>> funny in the debugger:
> >>>
> >>> - The RelSubset's "traitSet" _does_ have the proper collation trait.
> >>> - But its "set" field points to a RelSet with "rels" that _don't_ have
> >>> _any_ collation traits.
> >>>
> >>> From what I understand that causes Calcite to treat the unsorted and
> >> sorted
> >>> rels as equivalent when they in fact aren't. I'm still not sure if this
> >> is
> >>> a Calcite bug or user error on my part… I'll keep looking into it
> unless
> >>> someone has any bright ideas.
> >>>
> >>> fwiw, my Planner construction looks like this:
> >>>
> >>>   final FrameworkConfig frameworkConfig = Frameworks
> >>>   .newConfigBuilder()
> >>>   .parserConfig(
> >>>   SqlParser.configBuilder()
> >>>.setCaseSensitive(true)
> >>>.setUnquotedCasing(Casing.UNCHANGED)
> >>>.build()
> >>>   )
> >>>   .defaultSchema(rootSchema)
> >>>   .traitDefs(ConventionTraitDef.INSTANCE,
> >>> RelCollationTraitDef.INSTANCE)
> >>>   .programs(Programs.ofRules(myRules))
> >>>   .executor(new RexExecutorImpl(Schemas.createDataContext(null)))
> >>>   .context(Contexts.EMPTY_CONTEXT)
> >>>   .build();
> >>>
> >>>   return Frameworks.getPlanner(frameworkConfig);
> >>>
> >>> Gian
> >>>
> >>> On Sat, Dec 3, 2016 at 5:53 PM, Gian Merlino  wrote:
> >>>
>  Sure, I added those first two to the ticket.
> 
>  I don't think those are happening with (3) but I'll double check next
> >> time
>  I take a look at using the Planner.
> 
>  Gian
> 
>  On Fri, Dec 2, 2016 at 12:20 PM, Julian Hyde 
> wrote:
> 
> > Can you please add (1) and (2) to https://issues.apache.org/jira
> > /browse/CALCITE-1525  >> a/browse/CALCITE-1525>,
> > which deals with the whole issue of using “Planner” within the JDBC
> >> driver,
> > so we can be consistent.
> >
> > (3) doesn’t look likely to be related. Do your queries have UNION or
> > other set-ops? Are you sorting on columns that do not appear in the
> >> final
> > result?
> >
> > Julian
> >
> >
> >> On Nov 28, 2016, at 10:45 AM, Gian Merlino  wrote:
> >>
> >> I traveled a bit down the Frameworks/Planner road and got most of my
> > tests
> >> passing, but ran into some problems getting them all to work:
> >>
> >> (1) 

Re: Embedding Calcite, adjusting convertlets

[Julian Hyde]

Are you running into some variant of the problems that inspired 
https://issues.apache.org/jira/browse/CALCITE-819: 
 at the root of the tree, 
columns that are not projected are removed, and if the desired sort order 
involves non-projected columns, the desired sort order is forgotten).
 
> On Dec 14, 2016, at 11:19 AM, Gian Merlino  wrote:
> 
> Ah, thanks. So if that sort of thing is not a smoking gun, do you have an
> idea about where I should look next? If not I'll keep poking around.
> 
> Gian
> 
> On Wed, Dec 14, 2016 at 11:06 AM, Julian Hyde  wrote:
> 
>>> - But its "set" field points to a RelSet with "rels" that _don't_ have
>>> _any_ collation traits.
>> 
>> That’s OK. A “subset” (RelSubset) is a collection of RelNodes that are
>> logically and physically equivalent (same results, same physical
>> properties) whereas a “set” (RelSet) is a collection of RelNodes that are
>> logically equivalent.
>> 
>> A set can therefore be considered to be a collection of subsets, each of
>> which contains RelNodes. And it used to be implemented that way, but in
>> https://issues.apache.org/jira/browse/CALCITE-88 <
>> https://issues.apache.org/jira/browse/CALCITE-88> we introduced collation
>> as a trait, and that made subsets non-disjoint (a RelNode can be sorted on
>> (x, y), and also on (x), and also on (), and also on (z)) so we made
>> RelSubset just a view onto a RelSet, filtering the list of RelNodes
>> according to the ones that have (“subsume”) the desired traits.
>> 
>> Julian
>> 
>> 
>>> On Dec 14, 2016, at 10:45 AM, Gian Merlino  wrote:
>>> 
>>> I spent some more time looking into (3) and found that when I had things
>>> going through the Planner rather than the JDBC driver, SortRemoveRule was
>>> removing sorts when it shouldn't have been. This happens even for simple
>>> queries like "SELECT dim1 FROM s.foo GROUP BY dim1 ORDER BY dim1
>>> DESC". Removing SortRemoveRule from the planner fixed the broken tests on
>>> my end.
>>> 
>>> I dug into that a bit and saw that the call to "convert(sort.getInput(),
>>> traits)" in SortRemoveRule was returning a RelSubset that looked a bit
>>> funny in the debugger:
>>> 
>>> - The RelSubset's "traitSet" _does_ have the proper collation trait.
>>> - But its "set" field points to a RelSet with "rels" that _don't_ have
>>> _any_ collation traits.
>>> 
>>> From what I understand that causes Calcite to treat the unsorted and
>> sorted
>>> rels as equivalent when they in fact aren't. I'm still not sure if this
>> is
>>> a Calcite bug or user error on my part… I'll keep looking into it unless
>>> someone has any bright ideas.
>>> 
>>> fwiw, my Planner construction looks like this:
>>> 
>>>   final FrameworkConfig frameworkConfig = Frameworks
>>>   .newConfigBuilder()
>>>   .parserConfig(
>>>   SqlParser.configBuilder()
>>>.setCaseSensitive(true)
>>>.setUnquotedCasing(Casing.UNCHANGED)
>>>.build()
>>>   )
>>>   .defaultSchema(rootSchema)
>>>   .traitDefs(ConventionTraitDef.INSTANCE,
>>> RelCollationTraitDef.INSTANCE)
>>>   .programs(Programs.ofRules(myRules))
>>>   .executor(new RexExecutorImpl(Schemas.createDataContext(null)))
>>>   .context(Contexts.EMPTY_CONTEXT)
>>>   .build();
>>> 
>>>   return Frameworks.getPlanner(frameworkConfig);
>>> 
>>> Gian
>>> 
>>> On Sat, Dec 3, 2016 at 5:53 PM, Gian Merlino  wrote:
>>> 
 Sure, I added those first two to the ticket.
 
 I don't think those are happening with (3) but I'll double check next
>> time
 I take a look at using the Planner.
 
 Gian
 
 On Fri, Dec 2, 2016 at 12:20 PM, Julian Hyde  wrote:
 
> Can you please add (1) and (2) to https://issues.apache.org/jira
> /browse/CALCITE-1525 > a/browse/CALCITE-1525>,
> which deals with the whole issue of using “Planner” within the JDBC
>> driver,
> so we can be consistent.
> 
> (3) doesn’t look likely to be related. Do your queries have UNION or
> other set-ops? Are you sorting on columns that do not appear in the
>> final
> result?
> 
> Julian
> 
> 
>> On Nov 28, 2016, at 10:45 AM, Gian Merlino  wrote:
>> 
>> I traveled a bit down the Frameworks/Planner road and got most of my
> tests
>> passing, but ran into some problems getting them all to work:
>> 
>> (1) "EXPLAIN PLAN FOR" throws NullPointerException during
> Planner.validate.
>> It looks like CalcitePrepareImpl has some special code to handle
> validation
>> of EXPLAIN, but PlannerImpl doesn't. I'm not sure if this is
>> something I
>> should be doing on my end, or if it's a bug in PlannerImpl.
>> (2) I don't see a way to do ?-style prepared statements with bound
>> variables, which _is_ possible with the 

Re: Embedding Calcite, adjusting convertlets

[Gian Merlino]

Ah, thanks. So if that sort of thing is not a smoking gun, do you have an
idea about where I should look next? If not I'll keep poking around.

Gian

On Wed, Dec 14, 2016 at 11:06 AM, Julian Hyde  wrote:

> > - But its "set" field points to a RelSet with "rels" that _don't_ have
> > _any_ collation traits.
>
> That’s OK. A “subset” (RelSubset) is a collection of RelNodes that are
> logically and physically equivalent (same results, same physical
> properties) whereas a “set” (RelSet) is a collection of RelNodes that are
> logically equivalent.
>
> A set can therefore be considered to be a collection of subsets, each of
> which contains RelNodes. And it used to be implemented that way, but in
> https://issues.apache.org/jira/browse/CALCITE-88 <
> https://issues.apache.org/jira/browse/CALCITE-88> we introduced collation
> as a trait, and that made subsets non-disjoint (a RelNode can be sorted on
> (x, y), and also on (x), and also on (), and also on (z)) so we made
> RelSubset just a view onto a RelSet, filtering the list of RelNodes
> according to the ones that have (“subsume”) the desired traits.
>
> Julian
>
>
> > On Dec 14, 2016, at 10:45 AM, Gian Merlino  wrote:
> >
> > I spent some more time looking into (3) and found that when I had things
> > going through the Planner rather than the JDBC driver, SortRemoveRule was
> > removing sorts when it shouldn't have been. This happens even for simple
> > queries like "SELECT dim1 FROM s.foo GROUP BY dim1 ORDER BY dim1
> > DESC". Removing SortRemoveRule from the planner fixed the broken tests on
> > my end.
> >
> > I dug into that a bit and saw that the call to "convert(sort.getInput(),
> > traits)" in SortRemoveRule was returning a RelSubset that looked a bit
> > funny in the debugger:
> >
> > - The RelSubset's "traitSet" _does_ have the proper collation trait.
> > - But its "set" field points to a RelSet with "rels" that _don't_ have
> > _any_ collation traits.
> >
> > From what I understand that causes Calcite to treat the unsorted and
> sorted
> > rels as equivalent when they in fact aren't. I'm still not sure if this
> is
> > a Calcite bug or user error on my part… I'll keep looking into it unless
> > someone has any bright ideas.
> >
> > fwiw, my Planner construction looks like this:
> >
> >final FrameworkConfig frameworkConfig = Frameworks
> >.newConfigBuilder()
> >.parserConfig(
> >SqlParser.configBuilder()
> > .setCaseSensitive(true)
> > .setUnquotedCasing(Casing.UNCHANGED)
> > .build()
> >)
> >.defaultSchema(rootSchema)
> >.traitDefs(ConventionTraitDef.INSTANCE,
> > RelCollationTraitDef.INSTANCE)
> >.programs(Programs.ofRules(myRules))
> >.executor(new RexExecutorImpl(Schemas.createDataContext(null)))
> >.context(Contexts.EMPTY_CONTEXT)
> >.build();
> >
> >return Frameworks.getPlanner(frameworkConfig);
> >
> > Gian
> >
> > On Sat, Dec 3, 2016 at 5:53 PM, Gian Merlino  wrote:
> >
> >> Sure, I added those first two to the ticket.
> >>
> >> I don't think those are happening with (3) but I'll double check next
> time
> >> I take a look at using the Planner.
> >>
> >> Gian
> >>
> >> On Fri, Dec 2, 2016 at 12:20 PM, Julian Hyde  wrote:
> >>
> >>> Can you please add (1) and (2) to https://issues.apache.org/jira
> >>> /browse/CALCITE-1525  a/browse/CALCITE-1525>,
> >>> which deals with the whole issue of using “Planner” within the JDBC
> driver,
> >>> so we can be consistent.
> >>>
> >>> (3) doesn’t look likely to be related. Do your queries have UNION or
> >>> other set-ops? Are you sorting on columns that do not appear in the
> final
> >>> result?
> >>>
> >>> Julian
> >>>
> >>>
>  On Nov 28, 2016, at 10:45 AM, Gian Merlino  wrote:
> 
>  I traveled a bit down the Frameworks/Planner road and got most of my
> >>> tests
>  passing, but ran into some problems getting them all to work:
> 
>  (1) "EXPLAIN PLAN FOR" throws NullPointerException during
> >>> Planner.validate.
>  It looks like CalcitePrepareImpl has some special code to handle
> >>> validation
>  of EXPLAIN, but PlannerImpl doesn't. I'm not sure if this is
> something I
>  should be doing on my end, or if it's a bug in PlannerImpl.
>  (2) I don't see a way to do ?-style prepared statements with bound
>  variables, which _is_ possible with the JDBC driver route.
>  (3) Not sure why this is happening, but for some reason ORDER BY /
> LIMIT
>  clauses are getting ignored sometimes, even when they work with the
> JDBC
>  driver route. This may be something messed up with my rules though and
> >>> may
>  not be Calcite's fault.
> 
>  Julian, do any of these look like bugs that should be raised in jira,
> or
>  are they just stuff I should be dealing with on my side?
> 
> 

Re: Embedding Calcite, adjusting convertlets

[Julian Hyde]

> - But its "set" field points to a RelSet with "rels" that _don't_ have
> _any_ collation traits.

That’s OK. A “subset” (RelSubset) is a collection of RelNodes that are 
logically and physically equivalent (same results, same physical properties) 
whereas a “set” (RelSet) is a collection of RelNodes that are logically 
equivalent.

A set can therefore be considered to be a collection of subsets, each of which 
contains RelNodes. And it used to be implemented that way, but in 
https://issues.apache.org/jira/browse/CALCITE-88 
 we introduced collation as a 
trait, and that made subsets non-disjoint (a RelNode can be sorted on (x, y), 
and also on (x), and also on (), and also on (z)) so we made RelSubset just a 
view onto a RelSet, filtering the list of RelNodes according to the ones that 
have (“subsume”) the desired traits.

Julian


> On Dec 14, 2016, at 10:45 AM, Gian Merlino  wrote:
> 
> I spent some more time looking into (3) and found that when I had things
> going through the Planner rather than the JDBC driver, SortRemoveRule was
> removing sorts when it shouldn't have been. This happens even for simple
> queries like "SELECT dim1 FROM s.foo GROUP BY dim1 ORDER BY dim1
> DESC". Removing SortRemoveRule from the planner fixed the broken tests on
> my end.
> 
> I dug into that a bit and saw that the call to "convert(sort.getInput(),
> traits)" in SortRemoveRule was returning a RelSubset that looked a bit
> funny in the debugger:
> 
> - The RelSubset's "traitSet" _does_ have the proper collation trait.
> - But its "set" field points to a RelSet with "rels" that _don't_ have
> _any_ collation traits.
> 
> From what I understand that causes Calcite to treat the unsorted and sorted
> rels as equivalent when they in fact aren't. I'm still not sure if this is
> a Calcite bug or user error on my part… I'll keep looking into it unless
> someone has any bright ideas.
> 
> fwiw, my Planner construction looks like this:
> 
>final FrameworkConfig frameworkConfig = Frameworks
>.newConfigBuilder()
>.parserConfig(
>SqlParser.configBuilder()
> .setCaseSensitive(true)
> .setUnquotedCasing(Casing.UNCHANGED)
> .build()
>)
>.defaultSchema(rootSchema)
>.traitDefs(ConventionTraitDef.INSTANCE,
> RelCollationTraitDef.INSTANCE)
>.programs(Programs.ofRules(myRules))
>.executor(new RexExecutorImpl(Schemas.createDataContext(null)))
>.context(Contexts.EMPTY_CONTEXT)
>.build();
> 
>return Frameworks.getPlanner(frameworkConfig);
> 
> Gian
> 
> On Sat, Dec 3, 2016 at 5:53 PM, Gian Merlino  wrote:
> 
>> Sure, I added those first two to the ticket.
>> 
>> I don't think those are happening with (3) but I'll double check next time
>> I take a look at using the Planner.
>> 
>> Gian
>> 
>> On Fri, Dec 2, 2016 at 12:20 PM, Julian Hyde  wrote:
>> 
>>> Can you please add (1) and (2) to https://issues.apache.org/jira
>>> /browse/CALCITE-1525 ,
>>> which deals with the whole issue of using “Planner” within the JDBC driver,
>>> so we can be consistent.
>>> 
>>> (3) doesn’t look likely to be related. Do your queries have UNION or
>>> other set-ops? Are you sorting on columns that do not appear in the final
>>> result?
>>> 
>>> Julian
>>> 
>>> 
 On Nov 28, 2016, at 10:45 AM, Gian Merlino  wrote:
 
 I traveled a bit down the Frameworks/Planner road and got most of my
>>> tests
 passing, but ran into some problems getting them all to work:
 
 (1) "EXPLAIN PLAN FOR" throws NullPointerException during
>>> Planner.validate.
 It looks like CalcitePrepareImpl has some special code to handle
>>> validation
 of EXPLAIN, but PlannerImpl doesn't. I'm not sure if this is something I
 should be doing on my end, or if it's a bug in PlannerImpl.
 (2) I don't see a way to do ?-style prepared statements with bound
 variables, which _is_ possible with the JDBC driver route.
 (3) Not sure why this is happening, but for some reason ORDER BY / LIMIT
 clauses are getting ignored sometimes, even when they work with the JDBC
 driver route. This may be something messed up with my rules though and
>>> may
 not be Calcite's fault.
 
 Julian, do any of these look like bugs that should be raised in jira, or
 are they just stuff I should be dealing with on my side?
 
 Btw, I do like that the Frameworks/Planner route gives me back the
>>> RelNode
 itself, since that means I can make the Druid queries directly without
 needing to go through the extra layers of the JDBC driver. That part is
 nice.
 
 Gian
 
 On Wed, Nov 23, 2016 at 10:11 PM, Julian Hyde  wrote:
 
> I don’t know how it’s used outside Calcite. Maybe some others can
>>> 

Re: Embedding Calcite, adjusting convertlets

[Gian Merlino]

I spent some more time looking into (3) and found that when I had things
going through the Planner rather than the JDBC driver, SortRemoveRule was
removing sorts when it shouldn't have been. This happens even for simple
queries like "SELECT dim1 FROM s.foo GROUP BY dim1 ORDER BY dim1
DESC". Removing SortRemoveRule from the planner fixed the broken tests on
my end.

I dug into that a bit and saw that the call to "convert(sort.getInput(),
traits)" in SortRemoveRule was returning a RelSubset that looked a bit
funny in the debugger:

- The RelSubset's "traitSet" _does_ have the proper collation trait.
- But its "set" field points to a RelSet with "rels" that _don't_ have
_any_ collation traits.

>From what I understand that causes Calcite to treat the unsorted and sorted
rels as equivalent when they in fact aren't. I'm still not sure if this is
a Calcite bug or user error on my part… I'll keep looking into it unless
someone has any bright ideas.

fwiw, my Planner construction looks like this:

final FrameworkConfig frameworkConfig = Frameworks
.newConfigBuilder()
.parserConfig(
SqlParser.configBuilder()
 .setCaseSensitive(true)
 .setUnquotedCasing(Casing.UNCHANGED)
 .build()
)
.defaultSchema(rootSchema)
.traitDefs(ConventionTraitDef.INSTANCE,
RelCollationTraitDef.INSTANCE)
.programs(Programs.ofRules(myRules))
.executor(new RexExecutorImpl(Schemas.createDataContext(null)))
.context(Contexts.EMPTY_CONTEXT)
.build();

return Frameworks.getPlanner(frameworkConfig);

Gian

On Sat, Dec 3, 2016 at 5:53 PM, Gian Merlino  wrote:

> Sure, I added those first two to the ticket.
>
> I don't think those are happening with (3) but I'll double check next time
> I take a look at using the Planner.
>
> Gian
>
> On Fri, Dec 2, 2016 at 12:20 PM, Julian Hyde  wrote:
>
>> Can you please add (1) and (2) to https://issues.apache.org/jira
>> /browse/CALCITE-1525 ,
>> which deals with the whole issue of using “Planner” within the JDBC driver,
>> so we can be consistent.
>>
>> (3) doesn’t look likely to be related. Do your queries have UNION or
>> other set-ops? Are you sorting on columns that do not appear in the final
>> result?
>>
>> Julian
>>
>>
>> > On Nov 28, 2016, at 10:45 AM, Gian Merlino  wrote:
>> >
>> > I traveled a bit down the Frameworks/Planner road and got most of my
>> tests
>> > passing, but ran into some problems getting them all to work:
>> >
>> > (1) "EXPLAIN PLAN FOR" throws NullPointerException during
>> Planner.validate.
>> > It looks like CalcitePrepareImpl has some special code to handle
>> validation
>> > of EXPLAIN, but PlannerImpl doesn't. I'm not sure if this is something I
>> > should be doing on my end, or if it's a bug in PlannerImpl.
>> > (2) I don't see a way to do ?-style prepared statements with bound
>> > variables, which _is_ possible with the JDBC driver route.
>> > (3) Not sure why this is happening, but for some reason ORDER BY / LIMIT
>> > clauses are getting ignored sometimes, even when they work with the JDBC
>> > driver route. This may be something messed up with my rules though and
>> may
>> > not be Calcite's fault.
>> >
>> > Julian, do any of these look like bugs that should be raised in jira, or
>> > are they just stuff I should be dealing with on my side?
>> >
>> > Btw, I do like that the Frameworks/Planner route gives me back the
>> RelNode
>> > itself, since that means I can make the Druid queries directly without
>> > needing to go through the extra layers of the JDBC driver. That part is
>> > nice.
>> >
>> > Gian
>> >
>> > On Wed, Nov 23, 2016 at 10:11 PM, Julian Hyde  wrote:
>> >
>> >> I don’t know how it’s used outside Calcite. Maybe some others can
>> chime in.
>> >>
>> >> Thanks for the PR. I logged https://issues.apache.org/jira
>> >> /browse/CALCITE-1509 > a/browse/CALCITE-1509>
>> >> for it, and will commit shortly.
>> >>
>> >> Julian
>> >>
>> >>> On Nov 23, 2016, at 12:32 PM, Gian Merlino  wrote:
>> >>>
>> >>> Do you know examples of projects that use Planner or PlannerImpl
>> >> currently
>> >>> (from "outside")? As far as I can tell, within Calcite itself it's
>> only
>> >>> used in test code. Maybe that'd be a better entry point.
>> >>>
>> >>> In the meantime I raised a PR here for allowing a convertlet table
>> >> override
>> >>> in a CalcitePrepareImpl: https://github.com/apache/calcite/pull/330.
>> >> That
>> >>> was enough to get the JDBC driver on my end to behave how I want it
>> to.
>> >>>
>> >>> Gian
>> >>>
>> >>> On Thu, Nov 17, 2016 at 5:23 PM, Julian Hyde 
>> wrote:
>> >>>
>>  I was wrong earlier… FrameworkConfig already has a getConvertletTable
>>  method. But regarding using FrameworkConfig from within the JDBC
>> driver,

Re: Embedding Calcite, adjusting convertlets

[Gian Merlino]

Sure, I added those first two to the ticket.

I don't think those are happening with (3) but I'll double check next time
I take a look at using the Planner.

Gian

On Fri, Dec 2, 2016 at 12:20 PM, Julian Hyde  wrote:

> Can you please add (1) and (2) to https://issues.apache.org/
> jira/browse/CALCITE-1525  jira/browse/CALCITE-1525>, which deals with the whole issue of using
> “Planner” within the JDBC driver, so we can be consistent.
>
> (3) doesn’t look likely to be related. Do your queries have UNION or other
> set-ops? Are you sorting on columns that do not appear in the final result?
>
> Julian
>
>
> > On Nov 28, 2016, at 10:45 AM, Gian Merlino  wrote:
> >
> > I traveled a bit down the Frameworks/Planner road and got most of my
> tests
> > passing, but ran into some problems getting them all to work:
> >
> > (1) "EXPLAIN PLAN FOR" throws NullPointerException during
> Planner.validate.
> > It looks like CalcitePrepareImpl has some special code to handle
> validation
> > of EXPLAIN, but PlannerImpl doesn't. I'm not sure if this is something I
> > should be doing on my end, or if it's a bug in PlannerImpl.
> > (2) I don't see a way to do ?-style prepared statements with bound
> > variables, which _is_ possible with the JDBC driver route.
> > (3) Not sure why this is happening, but for some reason ORDER BY / LIMIT
> > clauses are getting ignored sometimes, even when they work with the JDBC
> > driver route. This may be something messed up with my rules though and
> may
> > not be Calcite's fault.
> >
> > Julian, do any of these look like bugs that should be raised in jira, or
> > are they just stuff I should be dealing with on my side?
> >
> > Btw, I do like that the Frameworks/Planner route gives me back the
> RelNode
> > itself, since that means I can make the Druid queries directly without
> > needing to go through the extra layers of the JDBC driver. That part is
> > nice.
> >
> > Gian
> >
> > On Wed, Nov 23, 2016 at 10:11 PM, Julian Hyde  wrote:
> >
> >> I don’t know how it’s used outside Calcite. Maybe some others can chime
> in.
> >>
> >> Thanks for the PR. I logged https://issues.apache.org/jira
> >> /browse/CALCITE-1509  jira/browse/CALCITE-1509>
> >> for it, and will commit shortly.
> >>
> >> Julian
> >>
> >>> On Nov 23, 2016, at 12:32 PM, Gian Merlino  wrote:
> >>>
> >>> Do you know examples of projects that use Planner or PlannerImpl
> >> currently
> >>> (from "outside")? As far as I can tell, within Calcite itself it's only
> >>> used in test code. Maybe that'd be a better entry point.
> >>>
> >>> In the meantime I raised a PR here for allowing a convertlet table
> >> override
> >>> in a CalcitePrepareImpl: https://github.com/apache/calcite/pull/330.
> >> That
> >>> was enough to get the JDBC driver on my end to behave how I want it to.
> >>>
> >>> Gian
> >>>
> >>> On Thu, Nov 17, 2016 at 5:23 PM, Julian Hyde  wrote:
> >>>
>  I was wrong earlier… FrameworkConfig already has a getConvertletTable
>  method. But regarding using FrameworkConfig from within the JDBC
> driver,
>  It’s complicated. FrameworkConfig only works if you are “outside”
> >> Calcite,
>  whereas CalcitePrepare is when you are customizing from the inside,
> and
>  sadly CalcitePrepare does not use a FrameworkConfig.
> 
>  Compare and contrast:
>  * CalcitePrepareImpl.getSqlToRelConverter [
> https://github.com/apache/
>  calcite/blob/3f92157d5742dd10f3b828d22d7a753e0a2899cc/core/
> >> src/main/java/
>  org/apache/calcite/prepare/CalcitePrepareImpl.java#L1114 <
>  https://github.com/apache/calcite/blob/3f92157d5742dd10f3b828d22d7a75
>  3e0a2899cc/core/src/main/java/org/apache/calcite/prepare/
>  CalcitePrepareImpl.java#L1114> ]
>  * PlannerImpl.rel [ https://github.com/apache/calcite/blob/
>  105bba1f83cd9631e8e1211d262e4886a4a863b7/core/src/main/java/
>  org/apache/calcite/prepare/PlannerImpl.java#L225 <
>  https://github.com/apache/calcite/blob/105bba1f83cd9631e8e1211d262e48
>  86a4a863b7/core/src/main/java/org/apache/calcite/prepare/
>  PlannerImpl.java#L225> ]
> 
>  The latter uses a convertletTable sourced from a FrameworkConfig.
> 
>  The ideal thing would be to get CalcitePrepareImpl to use a
> PlannerImpl
> >> to
>  do its dirty work. Then “inside” and “outside” would work the same.
> >> Would
>  definitely appreciate that as a patch.
> 
>  If you choose to go the JDBC driver route, you could override
>  Driver.createPrepareFactory to produce a sub-class of CalcitePrepare
> >> that
>  works for your environment, one with an explicit convertletTable
> rather
>  than just using the default.
> 
>  Julian
> 
> 
> > On Nov 17, 2016, at 5:01 PM, Gian Merlino  wrote:
> >
> > Hey Julian,
> >
> > If the convertlets were 

Re: Embedding Calcite, adjusting convertlets

[Julian Hyde]

Can you please add (1) and (2) to 
https://issues.apache.org/jira/browse/CALCITE-1525 
, which deals with the 
whole issue of using “Planner” within the JDBC driver, so we can be consistent.

(3) doesn’t look likely to be related. Do your queries have UNION or other 
set-ops? Are you sorting on columns that do not appear in the final result?

Julian


> On Nov 28, 2016, at 10:45 AM, Gian Merlino  wrote:
> 
> I traveled a bit down the Frameworks/Planner road and got most of my tests
> passing, but ran into some problems getting them all to work:
> 
> (1) "EXPLAIN PLAN FOR" throws NullPointerException during Planner.validate.
> It looks like CalcitePrepareImpl has some special code to handle validation
> of EXPLAIN, but PlannerImpl doesn't. I'm not sure if this is something I
> should be doing on my end, or if it's a bug in PlannerImpl.
> (2) I don't see a way to do ?-style prepared statements with bound
> variables, which _is_ possible with the JDBC driver route.
> (3) Not sure why this is happening, but for some reason ORDER BY / LIMIT
> clauses are getting ignored sometimes, even when they work with the JDBC
> driver route. This may be something messed up with my rules though and may
> not be Calcite's fault.
> 
> Julian, do any of these look like bugs that should be raised in jira, or
> are they just stuff I should be dealing with on my side?
> 
> Btw, I do like that the Frameworks/Planner route gives me back the RelNode
> itself, since that means I can make the Druid queries directly without
> needing to go through the extra layers of the JDBC driver. That part is
> nice.
> 
> Gian
> 
> On Wed, Nov 23, 2016 at 10:11 PM, Julian Hyde  wrote:
> 
>> I don’t know how it’s used outside Calcite. Maybe some others can chime in.
>> 
>> Thanks for the PR. I logged https://issues.apache.org/jira
>> /browse/CALCITE-1509 
>> for it, and will commit shortly.
>> 
>> Julian
>> 
>>> On Nov 23, 2016, at 12:32 PM, Gian Merlino  wrote:
>>> 
>>> Do you know examples of projects that use Planner or PlannerImpl
>> currently
>>> (from "outside")? As far as I can tell, within Calcite itself it's only
>>> used in test code. Maybe that'd be a better entry point.
>>> 
>>> In the meantime I raised a PR here for allowing a convertlet table
>> override
>>> in a CalcitePrepareImpl: https://github.com/apache/calcite/pull/330.
>> That
>>> was enough to get the JDBC driver on my end to behave how I want it to.
>>> 
>>> Gian
>>> 
>>> On Thu, Nov 17, 2016 at 5:23 PM, Julian Hyde  wrote:
>>> 
 I was wrong earlier… FrameworkConfig already has a getConvertletTable
 method. But regarding using FrameworkConfig from within the JDBC driver,
 It’s complicated. FrameworkConfig only works if you are “outside”
>> Calcite,
 whereas CalcitePrepare is when you are customizing from the inside, and
 sadly CalcitePrepare does not use a FrameworkConfig.
 
 Compare and contrast:
 * CalcitePrepareImpl.getSqlToRelConverter [ https://github.com/apache/
 calcite/blob/3f92157d5742dd10f3b828d22d7a753e0a2899cc/core/
>> src/main/java/
 org/apache/calcite/prepare/CalcitePrepareImpl.java#L1114 <
 https://github.com/apache/calcite/blob/3f92157d5742dd10f3b828d22d7a75
 3e0a2899cc/core/src/main/java/org/apache/calcite/prepare/
 CalcitePrepareImpl.java#L1114> ]
 * PlannerImpl.rel [ https://github.com/apache/calcite/blob/
 105bba1f83cd9631e8e1211d262e4886a4a863b7/core/src/main/java/
 org/apache/calcite/prepare/PlannerImpl.java#L225 <
 https://github.com/apache/calcite/blob/105bba1f83cd9631e8e1211d262e48
 86a4a863b7/core/src/main/java/org/apache/calcite/prepare/
 PlannerImpl.java#L225> ]
 
 The latter uses a convertletTable sourced from a FrameworkConfig.
 
 The ideal thing would be to get CalcitePrepareImpl to use a PlannerImpl
>> to
 do its dirty work. Then “inside” and “outside” would work the same.
>> Would
 definitely appreciate that as a patch.
 
 If you choose to go the JDBC driver route, you could override
 Driver.createPrepareFactory to produce a sub-class of CalcitePrepare
>> that
 works for your environment, one with an explicit convertletTable rather
 than just using the default.
 
 Julian
 
 
> On Nov 17, 2016, at 5:01 PM, Gian Merlino  wrote:
> 
> Hey Julian,
> 
> If the convertlets were customizable with a FrameworkConfig, how would
>> I
> use that configure the JDBC driver (given that I'm doing it with the
>> code
> upthread)? Or would that suggest using a different approach to
>> embedding
> Calcite?
> 
> Gian
> 
> On Thu, Nov 17, 2016 at 4:02 PM, Julian Hyde  wrote:
> 
>> Convertlets have a similar effect to planner rules (albeit they act on
>> scalar expressions, 

Re: Embedding Calcite, adjusting convertlets

[Gian Merlino]

I traveled a bit down the Frameworks/Planner road and got most of my tests
passing, but ran into some problems getting them all to work:

(1) "EXPLAIN PLAN FOR" throws NullPointerException during Planner.validate.
It looks like CalcitePrepareImpl has some special code to handle validation
of EXPLAIN, but PlannerImpl doesn't. I'm not sure if this is something I
should be doing on my end, or if it's a bug in PlannerImpl.
(2) I don't see a way to do ?-style prepared statements with bound
variables, which _is_ possible with the JDBC driver route.
(3) Not sure why this is happening, but for some reason ORDER BY / LIMIT
clauses are getting ignored sometimes, even when they work with the JDBC
driver route. This may be something messed up with my rules though and may
not be Calcite's fault.

Julian, do any of these look like bugs that should be raised in jira, or
are they just stuff I should be dealing with on my side?

Btw, I do like that the Frameworks/Planner route gives me back the RelNode
itself, since that means I can make the Druid queries directly without
needing to go through the extra layers of the JDBC driver. That part is
nice.

Gian

On Wed, Nov 23, 2016 at 10:11 PM, Julian Hyde  wrote:

> I don’t know how it’s used outside Calcite. Maybe some others can chime in.
>
> Thanks for the PR. I logged https://issues.apache.org/jira
> /browse/CALCITE-1509 
> for it, and will commit shortly.
>
> Julian
>
> > On Nov 23, 2016, at 12:32 PM, Gian Merlino  wrote:
> >
> > Do you know examples of projects that use Planner or PlannerImpl
> currently
> > (from "outside")? As far as I can tell, within Calcite itself it's only
> > used in test code. Maybe that'd be a better entry point.
> >
> > In the meantime I raised a PR here for allowing a convertlet table
> override
> > in a CalcitePrepareImpl: https://github.com/apache/calcite/pull/330.
> That
> > was enough to get the JDBC driver on my end to behave how I want it to.
> >
> > Gian
> >
> > On Thu, Nov 17, 2016 at 5:23 PM, Julian Hyde  wrote:
> >
> >> I was wrong earlier… FrameworkConfig already has a getConvertletTable
> >> method. But regarding using FrameworkConfig from within the JDBC driver,
> >> It’s complicated. FrameworkConfig only works if you are “outside”
> Calcite,
> >> whereas CalcitePrepare is when you are customizing from the inside, and
> >> sadly CalcitePrepare does not use a FrameworkConfig.
> >>
> >> Compare and contrast:
> >> * CalcitePrepareImpl.getSqlToRelConverter [ https://github.com/apache/
> >> calcite/blob/3f92157d5742dd10f3b828d22d7a753e0a2899cc/core/
> src/main/java/
> >> org/apache/calcite/prepare/CalcitePrepareImpl.java#L1114 <
> >> https://github.com/apache/calcite/blob/3f92157d5742dd10f3b828d22d7a75
> >> 3e0a2899cc/core/src/main/java/org/apache/calcite/prepare/
> >> CalcitePrepareImpl.java#L1114> ]
> >> * PlannerImpl.rel [ https://github.com/apache/calcite/blob/
> >> 105bba1f83cd9631e8e1211d262e4886a4a863b7/core/src/main/java/
> >> org/apache/calcite/prepare/PlannerImpl.java#L225 <
> >> https://github.com/apache/calcite/blob/105bba1f83cd9631e8e1211d262e48
> >> 86a4a863b7/core/src/main/java/org/apache/calcite/prepare/
> >> PlannerImpl.java#L225> ]
> >>
> >> The latter uses a convertletTable sourced from a FrameworkConfig.
> >>
> >> The ideal thing would be to get CalcitePrepareImpl to use a PlannerImpl
> to
> >> do its dirty work. Then “inside” and “outside” would work the same.
> Would
> >> definitely appreciate that as a patch.
> >>
> >> If you choose to go the JDBC driver route, you could override
> >> Driver.createPrepareFactory to produce a sub-class of CalcitePrepare
> that
> >> works for your environment, one with an explicit convertletTable rather
> >> than just using the default.
> >>
> >> Julian
> >>
> >>
> >>> On Nov 17, 2016, at 5:01 PM, Gian Merlino  wrote:
> >>>
> >>> Hey Julian,
> >>>
> >>> If the convertlets were customizable with a FrameworkConfig, how would
> I
> >>> use that configure the JDBC driver (given that I'm doing it with the
> code
> >>> upthread)? Or would that suggest using a different approach to
> embedding
> >>> Calcite?
> >>>
> >>> Gian
> >>>
> >>> On Thu, Nov 17, 2016 at 4:02 PM, Julian Hyde  wrote:
> >>>
>  Convertlets have a similar effect to planner rules (albeit they act on
>  scalar expressions, not relational expressions) so people should be
> >> able to
>  change the set of active convertlets.
> 
>  Would you like to propose a change that makes the convertlet table
>  pluggable? Maybe as part of FrameworkConfig? Regardless, please log a
> >> JIRA
>  to track this.
> 
>  And by the way, RexImpTable, which defines how operators are
> implemented
>  by generating java code, should also be pluggable. It’s been on my
> mind
> >> for
>  a long time to allow the “engine” — related to the data format, and
> how
>  code 

Re: Embedding Calcite, adjusting convertlets

[Julian Hyde]

I don’t know how it’s used outside Calcite. Maybe some others can chime in.

Thanks for the PR. I logged https://issues.apache.org/jira/browse/CALCITE-1509 
 for it, and will commit 
shortly.

Julian

> On Nov 23, 2016, at 12:32 PM, Gian Merlino  wrote:
> 
> Do you know examples of projects that use Planner or PlannerImpl currently
> (from "outside")? As far as I can tell, within Calcite itself it's only
> used in test code. Maybe that'd be a better entry point.
> 
> In the meantime I raised a PR here for allowing a convertlet table override
> in a CalcitePrepareImpl: https://github.com/apache/calcite/pull/330. That
> was enough to get the JDBC driver on my end to behave how I want it to.
> 
> Gian
> 
> On Thu, Nov 17, 2016 at 5:23 PM, Julian Hyde  wrote:
> 
>> I was wrong earlier… FrameworkConfig already has a getConvertletTable
>> method. But regarding using FrameworkConfig from within the JDBC driver,
>> It’s complicated. FrameworkConfig only works if you are “outside” Calcite,
>> whereas CalcitePrepare is when you are customizing from the inside, and
>> sadly CalcitePrepare does not use a FrameworkConfig.
>> 
>> Compare and contrast:
>> * CalcitePrepareImpl.getSqlToRelConverter [ https://github.com/apache/
>> calcite/blob/3f92157d5742dd10f3b828d22d7a753e0a2899cc/core/src/main/java/
>> org/apache/calcite/prepare/CalcitePrepareImpl.java#L1114 <
>> https://github.com/apache/calcite/blob/3f92157d5742dd10f3b828d22d7a75
>> 3e0a2899cc/core/src/main/java/org/apache/calcite/prepare/
>> CalcitePrepareImpl.java#L1114> ]
>> * PlannerImpl.rel [ https://github.com/apache/calcite/blob/
>> 105bba1f83cd9631e8e1211d262e4886a4a863b7/core/src/main/java/
>> org/apache/calcite/prepare/PlannerImpl.java#L225 <
>> https://github.com/apache/calcite/blob/105bba1f83cd9631e8e1211d262e48
>> 86a4a863b7/core/src/main/java/org/apache/calcite/prepare/
>> PlannerImpl.java#L225> ]
>> 
>> The latter uses a convertletTable sourced from a FrameworkConfig.
>> 
>> The ideal thing would be to get CalcitePrepareImpl to use a PlannerImpl to
>> do its dirty work. Then “inside” and “outside” would work the same. Would
>> definitely appreciate that as a patch.
>> 
>> If you choose to go the JDBC driver route, you could override
>> Driver.createPrepareFactory to produce a sub-class of CalcitePrepare that
>> works for your environment, one with an explicit convertletTable rather
>> than just using the default.
>> 
>> Julian
>> 
>> 
>>> On Nov 17, 2016, at 5:01 PM, Gian Merlino  wrote:
>>> 
>>> Hey Julian,
>>> 
>>> If the convertlets were customizable with a FrameworkConfig, how would I
>>> use that configure the JDBC driver (given that I'm doing it with the code
>>> upthread)? Or would that suggest using a different approach to embedding
>>> Calcite?
>>> 
>>> Gian
>>> 
>>> On Thu, Nov 17, 2016 at 4:02 PM, Julian Hyde  wrote:
>>> 
 Convertlets have a similar effect to planner rules (albeit they act on
 scalar expressions, not relational expressions) so people should be
>> able to
 change the set of active convertlets.
 
 Would you like to propose a change that makes the convertlet table
 pluggable? Maybe as part of FrameworkConfig? Regardless, please log a
>> JIRA
 to track this.
 
 And by the way, RexImpTable, which defines how operators are implemented
 by generating java code, should also be pluggable. It’s been on my mind
>> for
 a long time to allow the “engine” — related to the data format, and how
 code is generated to access fields and evaluate expressions and
>> operators —
 to be pluggable.
 
 Regarding whether the JDBC driver is the right way to embed Calcite.
 There’s no easy answer. You might want to embed Calcite as a library in
 your own server (as Drill and Hive do). Or you might want to make
>> yourself
 just an adapter that runs inside a Calcite JDBC server (as the CSV
>> adapter
 does). Or something in the middle, like what Phoenix does: using Calcite
 for JDBC, SQL, planning, but with your own metadata and runtime engine.
 
 As long as you build the valuable stuff into planner rules, new
>> relational
 operators (if necessary) and use the schema SPI, you should be able to
 change packaging in the future.
 
 Julian
 
 
 
 
> On Nov 17, 2016, at 1:59 PM, Gian Merlino  wrote:
> 
> Hey Calcites,
> 
> I'm working on embedding Calcite into Druid (http://druid.io/,
> https://github.com/druid-io/druid/pull/3682) and am running into a
 problem
> that is making me wonder if the approach I'm using makes sense.
> 
> Consider the expression EXTRACT(YEAR FROM __time). Calcite has a
>> standard
> convertlet rule "convertExtract" that changes this into some arithmetic
 on
> __time casted to an int type. But Druid has some builtin functions to
>> 

Re: Embedding Calcite, adjusting convertlets

[Gian Merlino]

Do you know examples of projects that use Planner or PlannerImpl currently
(from "outside")? As far as I can tell, within Calcite itself it's only
used in test code. Maybe that'd be a better entry point.

In the meantime I raised a PR here for allowing a convertlet table override
in a CalcitePrepareImpl: https://github.com/apache/calcite/pull/330. That
was enough to get the JDBC driver on my end to behave how I want it to.

Gian

On Thu, Nov 17, 2016 at 5:23 PM, Julian Hyde  wrote:

> I was wrong earlier… FrameworkConfig already has a getConvertletTable
> method. But regarding using FrameworkConfig from within the JDBC driver,
> It’s complicated. FrameworkConfig only works if you are “outside” Calcite,
> whereas CalcitePrepare is when you are customizing from the inside, and
> sadly CalcitePrepare does not use a FrameworkConfig.
>
> Compare and contrast:
>  * CalcitePrepareImpl.getSqlToRelConverter [ https://github.com/apache/
> calcite/blob/3f92157d5742dd10f3b828d22d7a753e0a2899cc/core/src/main/java/
> org/apache/calcite/prepare/CalcitePrepareImpl.java#L1114 <
> https://github.com/apache/calcite/blob/3f92157d5742dd10f3b828d22d7a75
> 3e0a2899cc/core/src/main/java/org/apache/calcite/prepare/
> CalcitePrepareImpl.java#L1114> ]
>  * PlannerImpl.rel [ https://github.com/apache/calcite/blob/
> 105bba1f83cd9631e8e1211d262e4886a4a863b7/core/src/main/java/
> org/apache/calcite/prepare/PlannerImpl.java#L225 <
> https://github.com/apache/calcite/blob/105bba1f83cd9631e8e1211d262e48
> 86a4a863b7/core/src/main/java/org/apache/calcite/prepare/
> PlannerImpl.java#L225> ]
>
> The latter uses a convertletTable sourced from a FrameworkConfig.
>
> The ideal thing would be to get CalcitePrepareImpl to use a PlannerImpl to
> do its dirty work. Then “inside” and “outside” would work the same. Would
> definitely appreciate that as a patch.
>
> If you choose to go the JDBC driver route, you could override
> Driver.createPrepareFactory to produce a sub-class of CalcitePrepare that
> works for your environment, one with an explicit convertletTable rather
> than just using the default.
>
> Julian
>
>
> > On Nov 17, 2016, at 5:01 PM, Gian Merlino  wrote:
> >
> > Hey Julian,
> >
> > If the convertlets were customizable with a FrameworkConfig, how would I
> > use that configure the JDBC driver (given that I'm doing it with the code
> > upthread)? Or would that suggest using a different approach to embedding
> > Calcite?
> >
> > Gian
> >
> > On Thu, Nov 17, 2016 at 4:02 PM, Julian Hyde  wrote:
> >
> >> Convertlets have a similar effect to planner rules (albeit they act on
> >> scalar expressions, not relational expressions) so people should be
> able to
> >> change the set of active convertlets.
> >>
> >> Would you like to propose a change that makes the convertlet table
> >> pluggable? Maybe as part of FrameworkConfig? Regardless, please log a
> JIRA
> >> to track this.
> >>
> >> And by the way, RexImpTable, which defines how operators are implemented
> >> by generating java code, should also be pluggable. It’s been on my mind
> for
> >> a long time to allow the “engine” — related to the data format, and how
> >> code is generated to access fields and evaluate expressions and
> operators —
> >> to be pluggable.
> >>
> >> Regarding whether the JDBC driver is the right way to embed Calcite.
> >> There’s no easy answer. You might want to embed Calcite as a library in
> >> your own server (as Drill and Hive do). Or you might want to make
> yourself
> >> just an adapter that runs inside a Calcite JDBC server (as the CSV
> adapter
> >> does). Or something in the middle, like what Phoenix does: using Calcite
> >> for JDBC, SQL, planning, but with your own metadata and runtime engine.
> >>
> >> As long as you build the valuable stuff into planner rules, new
> relational
> >> operators (if necessary) and use the schema SPI, you should be able to
> >> change packaging in the future.
> >>
> >> Julian
> >>
> >>
> >>
> >>
> >>> On Nov 17, 2016, at 1:59 PM, Gian Merlino  wrote:
> >>>
> >>> Hey Calcites,
> >>>
> >>> I'm working on embedding Calcite into Druid (http://druid.io/,
> >>> https://github.com/druid-io/druid/pull/3682) and am running into a
> >> problem
> >>> that is making me wonder if the approach I'm using makes sense.
> >>>
> >>> Consider the expression EXTRACT(YEAR FROM __time). Calcite has a
> standard
> >>> convertlet rule "convertExtract" that changes this into some arithmetic
> >> on
> >>> __time casted to an int type. But Druid has some builtin functions to
> do
> >>> this, and I'd rather use those than arithmetic (for a bunch of
> reasons).
> >>> Ideally, in my RelOptRules that convert Calcite rels to Druid queries,
> >> I'd
> >>> see the EXTRACT as a normal RexCall with the time flag and an
> expression
> >> to
> >>> apply it to. That's a lot easier to translate than the arithmetic
> stuff,
> >>> which I'd have to pattern match and undo first before translating.
> 

Re: Embedding Calcite, adjusting convertlets

[Julian Hyde]

I was wrong earlier… FrameworkConfig already has a getConvertletTable method. 
But regarding using FrameworkConfig from within the JDBC driver, It’s 
complicated. FrameworkConfig only works if you are “outside” Calcite, whereas 
CalcitePrepare is when you are customizing from the inside, and sadly 
CalcitePrepare does not use a FrameworkConfig.

Compare and contrast:
 * CalcitePrepareImpl.getSqlToRelConverter [ 
https://github.com/apache/calcite/blob/3f92157d5742dd10f3b828d22d7a753e0a2899cc/core/src/main/java/org/apache/calcite/prepare/CalcitePrepareImpl.java#L1114
 

 ]
 * PlannerImpl.rel [ 
https://github.com/apache/calcite/blob/105bba1f83cd9631e8e1211d262e4886a4a863b7/core/src/main/java/org/apache/calcite/prepare/PlannerImpl.java#L225
 

 ]

The latter uses a convertletTable sourced from a FrameworkConfig. 

The ideal thing would be to get CalcitePrepareImpl to use a PlannerImpl to do 
its dirty work. Then “inside” and “outside” would work the same. Would 
definitely appreciate that as a patch.

If you choose to go the JDBC driver route, you could override 
Driver.createPrepareFactory to produce a sub-class of CalcitePrepare that works 
for your environment, one with an explicit convertletTable rather than just 
using the default.

Julian


> On Nov 17, 2016, at 5:01 PM, Gian Merlino  wrote:
> 
> Hey Julian,
> 
> If the convertlets were customizable with a FrameworkConfig, how would I
> use that configure the JDBC driver (given that I'm doing it with the code
> upthread)? Or would that suggest using a different approach to embedding
> Calcite?
> 
> Gian
> 
> On Thu, Nov 17, 2016 at 4:02 PM, Julian Hyde  wrote:
> 
>> Convertlets have a similar effect to planner rules (albeit they act on
>> scalar expressions, not relational expressions) so people should be able to
>> change the set of active convertlets.
>> 
>> Would you like to propose a change that makes the convertlet table
>> pluggable? Maybe as part of FrameworkConfig? Regardless, please log a JIRA
>> to track this.
>> 
>> And by the way, RexImpTable, which defines how operators are implemented
>> by generating java code, should also be pluggable. It’s been on my mind for
>> a long time to allow the “engine” — related to the data format, and how
>> code is generated to access fields and evaluate expressions and operators —
>> to be pluggable.
>> 
>> Regarding whether the JDBC driver is the right way to embed Calcite.
>> There’s no easy answer. You might want to embed Calcite as a library in
>> your own server (as Drill and Hive do). Or you might want to make yourself
>> just an adapter that runs inside a Calcite JDBC server (as the CSV adapter
>> does). Or something in the middle, like what Phoenix does: using Calcite
>> for JDBC, SQL, planning, but with your own metadata and runtime engine.
>> 
>> As long as you build the valuable stuff into planner rules, new relational
>> operators (if necessary) and use the schema SPI, you should be able to
>> change packaging in the future.
>> 
>> Julian
>> 
>> 
>> 
>> 
>>> On Nov 17, 2016, at 1:59 PM, Gian Merlino  wrote:
>>> 
>>> Hey Calcites,
>>> 
>>> I'm working on embedding Calcite into Druid (http://druid.io/,
>>> https://github.com/druid-io/druid/pull/3682) and am running into a
>> problem
>>> that is making me wonder if the approach I'm using makes sense.
>>> 
>>> Consider the expression EXTRACT(YEAR FROM __time). Calcite has a standard
>>> convertlet rule "convertExtract" that changes this into some arithmetic
>> on
>>> __time casted to an int type. But Druid has some builtin functions to do
>>> this, and I'd rather use those than arithmetic (for a bunch of reasons).
>>> Ideally, in my RelOptRules that convert Calcite rels to Druid queries,
>> I'd
>>> see the EXTRACT as a normal RexCall with the time flag and an expression
>> to
>>> apply it to. That's a lot easier to translate than the arithmetic stuff,
>>> which I'd have to pattern match and undo first before translating.
>>> 
>>> So the problem I have is that I want to disable convertExtract, but I
>> don't
>>> see a way to do that or to swap out the convertlet table.
>>> 
>>> The code I'm using to set up a connection is:
>>> 
>>> public CalciteConnection createCalciteConnection(
>>> final DruidSchema druidSchema
>>> ) throws SQLException
>>> {
>>>   final Properties props = new Properties();
>>>   props.setProperty("caseSensitive", "true");
>>>   props.setProperty("unquotedCasing", "UNCHANGED");
>>>   final Connection connection =
>>> DriverManager.getConnection("jdbc:calcite:", props);
>>>   final CalciteConnection calciteConnection =
>>> connection.unwrap(CalciteConnection.class);
>>>   

Re: Embedding Calcite, adjusting convertlets

[Gian Merlino]

Hey Julian,

If the convertlets were customizable with a FrameworkConfig, how would I
use that configure the JDBC driver (given that I'm doing it with the code
upthread)? Or would that suggest using a different approach to embedding
Calcite?

Gian

On Thu, Nov 17, 2016 at 4:02 PM, Julian Hyde  wrote:

> Convertlets have a similar effect to planner rules (albeit they act on
> scalar expressions, not relational expressions) so people should be able to
> change the set of active convertlets.
>
> Would you like to propose a change that makes the convertlet table
> pluggable? Maybe as part of FrameworkConfig? Regardless, please log a JIRA
> to track this.
>
> And by the way, RexImpTable, which defines how operators are implemented
> by generating java code, should also be pluggable. It’s been on my mind for
> a long time to allow the “engine” — related to the data format, and how
> code is generated to access fields and evaluate expressions and operators —
> to be pluggable.
>
> Regarding whether the JDBC driver is the right way to embed Calcite.
> There’s no easy answer. You might want to embed Calcite as a library in
> your own server (as Drill and Hive do). Or you might want to make yourself
> just an adapter that runs inside a Calcite JDBC server (as the CSV adapter
> does). Or something in the middle, like what Phoenix does: using Calcite
> for JDBC, SQL, planning, but with your own metadata and runtime engine.
>
> As long as you build the valuable stuff into planner rules, new relational
> operators (if necessary) and use the schema SPI, you should be able to
> change packaging in the future.
>
> Julian
>
>
>
>
> > On Nov 17, 2016, at 1:59 PM, Gian Merlino  wrote:
> >
> > Hey Calcites,
> >
> > I'm working on embedding Calcite into Druid (http://druid.io/,
> > https://github.com/druid-io/druid/pull/3682) and am running into a
> problem
> > that is making me wonder if the approach I'm using makes sense.
> >
> > Consider the expression EXTRACT(YEAR FROM __time). Calcite has a standard
> > convertlet rule "convertExtract" that changes this into some arithmetic
> on
> > __time casted to an int type. But Druid has some builtin functions to do
> > this, and I'd rather use those than arithmetic (for a bunch of reasons).
> > Ideally, in my RelOptRules that convert Calcite rels to Druid queries,
> I'd
> > see the EXTRACT as a normal RexCall with the time flag and an expression
> to
> > apply it to. That's a lot easier to translate than the arithmetic stuff,
> > which I'd have to pattern match and undo first before translating.
> >
> > So the problem I have is that I want to disable convertExtract, but I
> don't
> > see a way to do that or to swap out the convertlet table.
> >
> > The code I'm using to set up a connection is:
> >
> >  public CalciteConnection createCalciteConnection(
> >  final DruidSchema druidSchema
> >  ) throws SQLException
> >  {
> >final Properties props = new Properties();
> >props.setProperty("caseSensitive", "true");
> >props.setProperty("unquotedCasing", "UNCHANGED");
> >final Connection connection =
> > DriverManager.getConnection("jdbc:calcite:", props);
> >final CalciteConnection calciteConnection =
> > connection.unwrap(CalciteConnection.class);
> >calciteConnection.getRootSchema().setCacheEnabled(false);
> >calciteConnection.getRootSchema().add(DRUID_SCHEMA_NAME,
> druidSchema);
> >return calciteConnection;
> >  }
> >
> > This CalciteConnection is then used by the Druid HTTP server to offer a
> SQL
> > API.
> >
> > Is there some way to swap out the convertlet table that I'm missing?
> >
> > Also, just in general, am I going about this the right way? Is using the
> > JDBC driver the right way to embed Calcite? Or should I be calling into
> it
> > at some lower level?
> >
> > Thanks!
> >
> > Gian
>
>

Re: Embedding Calcite, adjusting convertlets

[herman...@teeupdata.com]

I have been trying to access Druid from Spark via Calcite JDBC, but somehow sql 
statement generated by Spark causes exceptions. Not sure if it is Calcite JDBC 
or Druid related. Have you see, or do you think Spark -> Calcite JDBC -> Druid 
is a right way to connect Spark and Druid?

Thanks
Herman.


> On Nov 17, 2016, at 19:02, Julian Hyde  wrote:
> 
> Convertlets have a similar effect to planner rules (albeit they act on scalar 
> expressions, not relational expressions) so people should be able to change 
> the set of active convertlets.
> 
> Would you like to propose a change that makes the convertlet table pluggable? 
> Maybe as part of FrameworkConfig? Regardless, please log a JIRA to track this.
> 
> And by the way, RexImpTable, which defines how operators are implemented by 
> generating java code, should also be pluggable. It’s been on my mind for a 
> long time to allow the “engine” — related to the data format, and how code is 
> generated to access fields and evaluate expressions and operators — to be 
> pluggable.
> 
> Regarding whether the JDBC driver is the right way to embed Calcite. There’s 
> no easy answer. You might want to embed Calcite as a library in your own 
> server (as Drill and Hive do). Or you might want to make yourself just an 
> adapter that runs inside a Calcite JDBC server (as the CSV adapter does). Or 
> something in the middle, like what Phoenix does: using Calcite for JDBC, SQL, 
> planning, but with your own metadata and runtime engine.
> 
> As long as you build the valuable stuff into planner rules, new relational 
> operators (if necessary) and use the schema SPI, you should be able to change 
> packaging in the future. 
> 
> Julian
> 
> 
> 
> 
>> On Nov 17, 2016, at 1:59 PM, Gian Merlino  wrote:
>> 
>> Hey Calcites,
>> 
>> I'm working on embedding Calcite into Druid (http://druid.io/,
>> https://github.com/druid-io/druid/pull/3682) and am running into a problem
>> that is making me wonder if the approach I'm using makes sense.
>> 
>> Consider the expression EXTRACT(YEAR FROM __time). Calcite has a standard
>> convertlet rule "convertExtract" that changes this into some arithmetic on
>> __time casted to an int type. But Druid has some builtin functions to do
>> this, and I'd rather use those than arithmetic (for a bunch of reasons).
>> Ideally, in my RelOptRules that convert Calcite rels to Druid queries, I'd
>> see the EXTRACT as a normal RexCall with the time flag and an expression to
>> apply it to. That's a lot easier to translate than the arithmetic stuff,
>> which I'd have to pattern match and undo first before translating.
>> 
>> So the problem I have is that I want to disable convertExtract, but I don't
>> see a way to do that or to swap out the convertlet table.
>> 
>> The code I'm using to set up a connection is:
>> 
>> public CalciteConnection createCalciteConnection(
>> final DruidSchema druidSchema
>> ) throws SQLException
>> {
>>   final Properties props = new Properties();
>>   props.setProperty("caseSensitive", "true");
>>   props.setProperty("unquotedCasing", "UNCHANGED");
>>   final Connection connection =
>> DriverManager.getConnection("jdbc:calcite:", props);
>>   final CalciteConnection calciteConnection =
>> connection.unwrap(CalciteConnection.class);
>>   calciteConnection.getRootSchema().setCacheEnabled(false);
>>   calciteConnection.getRootSchema().add(DRUID_SCHEMA_NAME, druidSchema);
>>   return calciteConnection;
>> }
>> 
>> This CalciteConnection is then used by the Druid HTTP server to offer a SQL
>> API.
>> 
>> Is there some way to swap out the convertlet table that I'm missing?
>> 
>> Also, just in general, am I going about this the right way? Is using the
>> JDBC driver the right way to embed Calcite? Or should I be calling into it
>> at some lower level?
>> 
>> Thanks!
>> 
>> Gian
>