Re: A storage plugin for a custom datasource

[Muhammad Gelbana]

Thanks a lot Aman, my mistake was that there was a logical node between
physical ones, which couldn't be implemented of course.



- Gelbana

On Tue, Jul 11, 2017 at 5:51 PM, Aman Sinha <amansi...@apache.org> wrote:

> For such 'could not be implemented' issues, it is useful to look at Calcite
> trace logs to determine where an appropriate implementation of the logical
> plan node to physical could not be found.   You would need 2 things:  a
> logging.properties file and adding couple of -D flags to your JVM
> properties.
>
> In drill-env.sh:
>
> export DRILL_JAVA_OPTS="$DRILL_JAVA_OPTS *-Dcalcite.debug=true
> -Djava.util.logging.config.file=* "
>
> // Create logging.properties file  (note the output of this will go to your
> $HOME/log directory, so you need to create that).
>
> $ cat logging.properties
>
> handlers=java.util.logging.FileHandler
>
> .level=ALL
>
> #.level= INFO
>
> org.eigenbase.relopt.RelOptPlanner.level=FINER
>
> java.util.logging.FileHandler.pattern=%h/log/java%u.log
>
> On Mon, Jul 10, 2017 at 9:20 AM, Muhammad Gelbana <m.gelb...@gmail.com>
> wrote:
>
> > Forgive me for accidentally sending the previous email before preparing
> it
> > well. Ignore the plans mentioned earlier for now, to continue...
> >
> > Getting back to the rules, I edited the old JDBC physical converter rule
> to
> > be
> >
> > private static class GelbanaPrule extends ConverterRule {
> > > private GelbanaPrule(IncortaLayoutConvention
> > incortaLayoutConvention)
> > > {
> > > super(GelbanaRel.class, incortaLayoutConvention,
> > > Prel.DRILL_PHYSICAL, "PREL_Converter");
> > > }
> > > @Override
> > > public boolean matches(RelOptRuleCall call) {
> > > return super.matches(call);
> > > }
> > > @Override
> > > public RelNode convert(RelNode in) {
> > > RelTraitSet physicalTraits =
> > > in.getTraitSet().replace(getOutTrait());
> > > RelTraitSet noneTraits = in.getTraitSet().replace(
> > Convention.NONE);
> > > return new GelbanaIntermediatePrel(in.getCluster(),
> > > physicalTraits, convert(in, noneTraits));
> > > }
> > > }
> >
> >
> > What happens is that the physical rule is executed successfully but then
> an
> > error (*Node [rel#50:Subset#3.LOGICAL.ANY([]).[]] could not be
> > implemented;
> > planner state*) is thrown from this method:
> > *org.apache.calcite.plan.volcano.RelSubset.CheapestPlanReplacer.visit(
> > RelNode,
> > int, RelNode) *
> >
> > That's because a *RelSubset* is visited but it doesn't have a best
> > performing node and it's cost is infinite.
> >
> > Getting back to the plan included in the previous email, I encapsulated
> the
> > *LogicalAggregate* as an *IncortaRel*, so that I can physically implement
> > the aggregation. I succeeded in the encapsulation but I can't figure out
> > how to fix the "*could not be implemented*" error so far. Would someone
> > please give a hint about how I can approach this error ?
> >
> >
> >
> > - Gelbana
> >
> > On Mon, Jul 10, 2017 at 6:09 PM, Muhammad Gelbana <m.gelb...@gmail.com>
> > wrote:
> >
> > > I'm planning to create a storage plugin for a custom datasource that
> > > accepts queries in the form of XML messages, but it's metadata can be
> > > discovered using the JDBC metadata API. I can work on discovering the
> > > metadata differently but that's not my priority for now.
> > >
> > > So I copied the JDBC storage plugin, Ignored all JDBC rules, and edited
> > > wrote the following JDBC storage rules:
> > >
> > > I renamed *JdbcDrelConverterRule* to *GelbanaRelConverterRule* and
> edited
> > > it's *constructor* and *convert* methods to be:
> > >
> > > public GelbanaRelConverterRule(IncortaLayoutConvention out) {
> > >> super(Aggregate.class, Convention.NONE, out,
> > >> "Incorta_Rel_Converter");
> > >> }
> > >> @Override
> > >> public RelNode convert(RelNode rel) {
> > >> RelTraitSet newTraits = rel.getTraitSet().replace(getOutTrait());
> > >> return new GelbanaRel(rel.getCluster(), newTraits, convert(rel,
> > >> newTraits));
> > >> }
> > >
> > >
> > > 17:57:19.931 [269c5c27-2f94-14ff-1f3e-0035b17b5965:foreman] DEBUG
> > > o.a.d.e.p.s.h.DefaultSqlHandler - HEP:Window Function rewrites
&g

Re: A storage plugin for a custom datasource

[Aman Sinha]

For such 'could not be implemented' issues, it is useful to look at Calcite
trace logs to determine where an appropriate implementation of the logical
plan node to physical could not be found.   You would need 2 things:  a
logging.properties file and adding couple of -D flags to your JVM
properties.

In drill-env.sh:

export DRILL_JAVA_OPTS="$DRILL_JAVA_OPTS *-Dcalcite.debug=true
-Djava.util.logging.config.file=* "

// Create logging.properties file  (note the output of this will go to your
$HOME/log directory, so you need to create that).

$ cat logging.properties

handlers=java.util.logging.FileHandler

.level=ALL

#.level= INFO

org.eigenbase.relopt.RelOptPlanner.level=FINER

java.util.logging.FileHandler.pattern=%h/log/java%u.log

On Mon, Jul 10, 2017 at 9:20 AM, Muhammad Gelbana <m.gelb...@gmail.com>
wrote:

> Forgive me for accidentally sending the previous email before preparing it
> well. Ignore the plans mentioned earlier for now, to continue...
>
> Getting back to the rules, I edited the old JDBC physical converter rule to
> be
>
> private static class GelbanaPrule extends ConverterRule {
> > private GelbanaPrule(IncortaLayoutConvention
> incortaLayoutConvention)
> > {
> > super(GelbanaRel.class, incortaLayoutConvention,
> > Prel.DRILL_PHYSICAL, "PREL_Converter");
> > }
> > @Override
> > public boolean matches(RelOptRuleCall call) {
> > return super.matches(call);
> > }
> > @Override
> > public RelNode convert(RelNode in) {
> > RelTraitSet physicalTraits =
> > in.getTraitSet().replace(getOutTrait());
> > RelTraitSet noneTraits = in.getTraitSet().replace(
> Convention.NONE);
> > return new GelbanaIntermediatePrel(in.getCluster(),
> > physicalTraits, convert(in, noneTraits));
> > }
> > }
>
>
> What happens is that the physical rule is executed successfully but then an
> error (*Node [rel#50:Subset#3.LOGICAL.ANY([]).[]] could not be
> implemented;
> planner state*) is thrown from this method:
> *org.apache.calcite.plan.volcano.RelSubset.CheapestPlanReplacer.visit(
> RelNode,
> int, RelNode) *
>
> That's because a *RelSubset* is visited but it doesn't have a best
> performing node and it's cost is infinite.
>
> Getting back to the plan included in the previous email, I encapsulated the
> *LogicalAggregate* as an *IncortaRel*, so that I can physically implement
> the aggregation. I succeeded in the encapsulation but I can't figure out
> how to fix the "*could not be implemented*" error so far. Would someone
> please give a hint about how I can approach this error ?
>
>
>
> - Gelbana
>
> On Mon, Jul 10, 2017 at 6:09 PM, Muhammad Gelbana <m.gelb...@gmail.com>
> wrote:
>
> > I'm planning to create a storage plugin for a custom datasource that
> > accepts queries in the form of XML messages, but it's metadata can be
> > discovered using the JDBC metadata API. I can work on discovering the
> > metadata differently but that's not my priority for now.
> >
> > So I copied the JDBC storage plugin, Ignored all JDBC rules, and edited
> > wrote the following JDBC storage rules:
> >
> > I renamed *JdbcDrelConverterRule* to *GelbanaRelConverterRule* and edited
> > it's *constructor* and *convert* methods to be:
> >
> > public GelbanaRelConverterRule(IncortaLayoutConvention out) {
> >> super(Aggregate.class, Convention.NONE, out,
> >> "Incorta_Rel_Converter");
> >> }
> >> @Override
> >> public RelNode convert(RelNode rel) {
> >> RelTraitSet newTraits = rel.getTraitSet().replace(getOutTrait());
> >> return new GelbanaRel(rel.getCluster(), newTraits, convert(rel,
> >> newTraits));
> >> }
> >
> >
> > 17:57:19.931 [269c5c27-2f94-14ff-1f3e-0035b17b5965:foreman] DEBUG
> > o.a.d.e.p.s.h.DefaultSqlHandler - HEP:Window Function rewrites (152ms):
> > LogicalProject(EXPR$0=[$1], PROD_CATEGORY=[$0]): rowcount = 150.0,
> > cumulative cost = {3518.75 rows, 3502.0 cpu, 0.0 io, 0.0 network, 0.0
> > memory}, id = 21
> >   LogicalAggregate(group=[{0}], EXPR$0=[COUNT($1)]): rowcount = 150.0,
> > cumulative cost = {3368.75 rows, 3202.0 cpu, 0.0 io, 0.0 network, 0.0
> > memory}, id = 19
> > LogicalProject(PROD_CATEGORY=[$13], Revenue=[$3]): rowcount =
> 1500.0,
> > cumulative cost = {3200.0 rows, 3202.0 cpu, 0.0 io, 0.0 network, 0.0
> > memory}, id = 17
> >   LogicalJoin(condition=[=($0, $13)], joinType=[inner]): rowcount =
> > 1500.0, cumulative cost = {1700.0 rows, 202.0 cpu, 0.0 io, 0.0 network,
> 0.0
> > memo

Re: A storage plugin for a custom datasource

[Muhammad Gelbana]

Forgive me for accidentally sending the previous email before preparing it
well. Ignore the plans mentioned earlier for now, to continue...

Getting back to the rules, I edited the old JDBC physical converter rule to
be

private static class GelbanaPrule extends ConverterRule {
> private GelbanaPrule(IncortaLayoutConvention incortaLayoutConvention)
> {
> super(GelbanaRel.class, incortaLayoutConvention,
> Prel.DRILL_PHYSICAL, "PREL_Converter");
> }
> @Override
> public boolean matches(RelOptRuleCall call) {
> return super.matches(call);
> }
> @Override
> public RelNode convert(RelNode in) {
> RelTraitSet physicalTraits =
> in.getTraitSet().replace(getOutTrait());
> RelTraitSet noneTraits = in.getTraitSet().replace(Convention.NONE);
> return new GelbanaIntermediatePrel(in.getCluster(),
> physicalTraits, convert(in, noneTraits));
> }
> }


What happens is that the physical rule is executed successfully but then an
error (*Node [rel#50:Subset#3.LOGICAL.ANY([]).[]] could not be implemented;
planner state*) is thrown from this method:
*org.apache.calcite.plan.volcano.RelSubset.CheapestPlanReplacer.visit(RelNode,
int, RelNode) *

That's because a *RelSubset* is visited but it doesn't have a best
performing node and it's cost is infinite.

Getting back to the plan included in the previous email, I encapsulated the
*LogicalAggregate* as an *IncortaRel*, so that I can physically implement
the aggregation. I succeeded in the encapsulation but I can't figure out
how to fix the "*could not be implemented*" error so far. Would someone
please give a hint about how I can approach this error ?



- Gelbana

On Mon, Jul 10, 2017 at 6:09 PM, Muhammad Gelbana <m.gelb...@gmail.com>
wrote:

> I'm planning to create a storage plugin for a custom datasource that
> accepts queries in the form of XML messages, but it's metadata can be
> discovered using the JDBC metadata API. I can work on discovering the
> metadata differently but that's not my priority for now.
>
> So I copied the JDBC storage plugin, Ignored all JDBC rules, and edited
> wrote the following JDBC storage rules:
>
> I renamed *JdbcDrelConverterRule* to *GelbanaRelConverterRule* and edited
> it's *constructor* and *convert* methods to be:
>
> public GelbanaRelConverterRule(IncortaLayoutConvention out) {
>> super(Aggregate.class, Convention.NONE, out,
>> "Incorta_Rel_Converter");
>> }
>> @Override
>> public RelNode convert(RelNode rel) {
>> RelTraitSet newTraits = rel.getTraitSet().replace(getOutTrait());
>> return new GelbanaRel(rel.getCluster(), newTraits, convert(rel,
>> newTraits));
>> }
>
>
> 17:57:19.931 [269c5c27-2f94-14ff-1f3e-0035b17b5965:foreman] DEBUG
> o.a.d.e.p.s.h.DefaultSqlHandler - HEP:Window Function rewrites (152ms):
> LogicalProject(EXPR$0=[$1], PROD_CATEGORY=[$0]): rowcount = 150.0,
> cumulative cost = {3518.75 rows, 3502.0 cpu, 0.0 io, 0.0 network, 0.0
> memory}, id = 21
>   LogicalAggregate(group=[{0}], EXPR$0=[COUNT($1)]): rowcount = 150.0,
> cumulative cost = {3368.75 rows, 3202.0 cpu, 0.0 io, 0.0 network, 0.0
> memory}, id = 19
> LogicalProject(PROD_CATEGORY=[$13], Revenue=[$3]): rowcount = 1500.0,
> cumulative cost = {3200.0 rows, 3202.0 cpu, 0.0 io, 0.0 network, 0.0
> memory}, id = 17
>   LogicalJoin(condition=[=($0, $13)], joinType=[inner]): rowcount =
> 1500.0, cumulative cost = {1700.0 rows, 202.0 cpu, 0.0 io, 0.0 network, 0.0
> memory}, id = 15
> JdbcTableScan(table=[[incorta, SALES, Target]]): rowcount = 100.0,
> cumulative cost = {100.0 rows, 101.0 cpu, 0.0 io, 0.0 network, 0.0 memory},
> id = 7
> JdbcTableScan(table=[[incorta, SALES, PRODUCTS]]): rowcount =
> 100.0, cumulative cost = {100.0 rows, 101.0 cpu, 0.0 io, 0.0 network, 0.0
> memory}, id = 8
>
> 17:57:20.094 [269c5c27-2f94-14ff-1f3e-0035b17b5965:foreman] DEBUG
> o.a.d.e.p.s.h.DefaultSqlHandler - HEP_BOTTOM_UP:Directory Prune Planning
> (150ms):
> LogicalProject(EXPR$0=[$1], PROD_CATEGORY=[$0]): rowcount = 150.0,
> cumulative cost = {150.0 rows, 300.0 cpu, 0.0 io, 0.0 network, 0.0 memory},
> id = 37
>   IncortaRel: rowcount = 150.0, cumulative cost = {0.0 rows, 0.0 cpu, 0.0
> io, 0.0 network, 0.0 memory}, id = 45
>
>
> - Gelbana
>

A storage plugin for a custom datasource

[Muhammad Gelbana]

I'm planning to create a storage plugin for a custom datasource that
accepts queries in the form of XML messages, but it's metadata can be
discovered using the JDBC metadata API. I can work on discovering the
metadata differently but that's not my priority for now.

So I copied the JDBC storage plugin, Ignored all JDBC rules, and edited
wrote the following JDBC storage rules:

I renamed *JdbcDrelConverterRule* to *GelbanaRelConverterRule* and edited
it's *constructor* and *convert* methods to be:

public GelbanaRelConverterRule(IncortaLayoutConvention out) {
> super(Aggregate.class, Convention.NONE, out,
> "Incorta_Rel_Converter");
> }
> @Override
> public RelNode convert(RelNode rel) {
> RelTraitSet newTraits = rel.getTraitSet().replace(getOutTrait());
> return new GelbanaRel(rel.getCluster(), newTraits, convert(rel,
> newTraits));
> }


17:57:19.931 [269c5c27-2f94-14ff-1f3e-0035b17b5965:foreman] DEBUG
o.a.d.e.p.s.h.DefaultSqlHandler - HEP:Window Function rewrites (152ms):
LogicalProject(EXPR$0=[$1], PROD_CATEGORY=[$0]): rowcount = 150.0,
cumulative cost = {3518.75 rows, 3502.0 cpu, 0.0 io, 0.0 network, 0.0
memory}, id = 21
  LogicalAggregate(group=[{0}], EXPR$0=[COUNT($1)]): rowcount = 150.0,
cumulative cost = {3368.75 rows, 3202.0 cpu, 0.0 io, 0.0 network, 0.0
memory}, id = 19
LogicalProject(PROD_CATEGORY=[$13], Revenue=[$3]): rowcount = 1500.0,
cumulative cost = {3200.0 rows, 3202.0 cpu, 0.0 io, 0.0 network, 0.0
memory}, id = 17
  LogicalJoin(condition=[=($0, $13)], joinType=[inner]): rowcount =
1500.0, cumulative cost = {1700.0 rows, 202.0 cpu, 0.0 io, 0.0 network, 0.0
memory}, id = 15
JdbcTableScan(table=[[incorta, SALES, Target]]): rowcount = 100.0,
cumulative cost = {100.0 rows, 101.0 cpu, 0.0 io, 0.0 network, 0.0 memory},
id = 7
JdbcTableScan(table=[[incorta, SALES, PRODUCTS]]): rowcount =
100.0, cumulative cost = {100.0 rows, 101.0 cpu, 0.0 io, 0.0 network, 0.0
memory}, id = 8

17:57:20.094 [269c5c27-2f94-14ff-1f3e-0035b17b5965:foreman] DEBUG
o.a.d.e.p.s.h.DefaultSqlHandler - HEP_BOTTOM_UP:Directory Prune Planning
(150ms):
LogicalProject(EXPR$0=[$1], PROD_CATEGORY=[$0]): rowcount = 150.0,
cumulative cost = {150.0 rows, 300.0 cpu, 0.0 io, 0.0 network, 0.0 memory},
id = 37
  IncortaRel: rowcount = 150.0, cumulative cost = {0.0 rows, 0.0 cpu, 0.0
io, 0.0 network, 0.0 memory}, id = 45


- Gelbana