Re: Unable to implement EnumerableAggregate(group=[{}], EXPR$0=[COUNT($0)]) with ReflectiveSchema
We come to the conclusion that fileds for tables and columns of the target
class must be public, and those not for tables and columns must not be public,
are we right?
-- Original --
From: "xiaobo ";;
Send time: Saturday, Feb 12, 2022 9:57 AM
To: "dev";
Subject: Unable to implement EnumerableAggregate(group=[{}],
EXPR$0=[COUNT($0)]) with ReflectiveSchema
Simple count sql against ReflectiveSchema failed with :
Error while executing SQL "select count(a.PA01AI01) from p.PRH_PA01 as a
": Unable to implement EnumerableAggregate(group=[{}], EXPR$0=[COUNT($0)]):
rowcount = 10.0, cumulative cost = {111.25 rows, 101.0 cpu, 0.0 io}, id = 37
EnumerableTableScan(table=[[p, PRH_PA01]]): rowcount = 100.0, cumulative cost
= {100.0 rows, 101.0 cpu, 0.0 io}, id = 26
at
com.xsmartware.javatest.pbc2p.CalciteBaseVarGenerator.generateVariable(CalciteBaseVarGenerator.java:61)
at com.xsmartware.javatest.pbc2p.PBC2PTestRunner.test1(PBC2PTestRunner.java:68)
at com.xsmartware.javatest.pbc2p.PBC2PTestRunner.run(PBC2PTestRunner.java:54)
at
org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:758)
at
org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:748)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:309)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1301)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1290)
at com.xsmartware.javatest.JavaTestApplication.main(JavaTestApplication.java:9)
Caused by: java.sql.SQLException: Error while executing SQL "select
count(a.PA01AI01) from p.PRH_PA01 as a
": Unable to implement EnumerableAggregate(group=[{}], EXPR$0=[COUNT($0)]):
rowcount = 10.0, cumulative cost = {111.25 rows, 101.0 cpu, 0.0 io}, id = 37
EnumerableTableScan(table=[[p, PRH_PA01]]): rowcount = 100.0, cumulative cost
= {100.0 rows, 101.0 cpu, 0.0 io}, id = 26
at org.apache.calcite.avatica.Helper.createException(Helper.java:56)
at org.apache.calcite.avatica.Helper.createException(Helper.java:41)
at
org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:163)
at
org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:227)
at com.xsmartware.common.util.SQLUtil.exeGetLong(SQLUtil.java:42)
at
com.xsmartware.javatest.pbc2p.CalciteBaseVarGenerator.generateVariable(CalciteBaseVarGenerator.java:39)
... 8 more
Caused by: java.lang.IllegalStateException: Unable to implement
EnumerableAggregate(group=[{}], EXPR$0=[COUNT($0)]): rowcount = 10.0,
cumulative cost = {111.25 rows, 101.0 cpu, 0.0 io}, id = 37
EnumerableTableScan(table=[[p, PRH_PA01]]): rowcount = 100.0, cumulative cost
= {100.0 rows, 101.0 cpu, 0.0 io}, id = 26
at
org.apache.calcite.adapter.enumerable.EnumerableRelImplementor.implementRoot(EnumerableRelImplementor.java:114)
at
org.apache.calcite.adapter.enumerable.EnumerableInterpretable.toBindable(EnumerableInterpretable.java:114)
at
org.apache.calcite.prepare.CalcitePrepareImpl$CalcitePreparingStmt.implement(CalcitePrepareImpl.java:1130)
at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:324)
at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:220)
at
org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:647)
at
org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:513)
at
org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:483)
at
org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:249)
at
org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:623)
at
org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:674)
at
org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156)
... 11 more
Suppressed: java.lang.NullPointerException: relFieldType.getComponentType()
for JavaType(interface java.util.List)
at java.base/java.util.Objects.requireNonNull(Objects.java:347)
at
org.apache.calcite.adapter.enumerable.EnumerableTableScan.fieldExpression(EnumerableTableScan.java:269)
at
org.apache.calcite.adapter.enumerable.EnumerableTableScan.toRows(EnumerableTableScan.java:252)
at
org.apache.calcite.adapter.enumerable.EnumerableTableScan.getExpression(EnumerableTableScan.java:209)
at
org.apache.calcite.adapter.enumerable.EnumerableTableScan.implement(EnumerableTableScan.java:340)
at
org.apache.calcite.adapter.enumerable.EnumerableRelImplementor.visitChild(EnumerableRelImplementor.java:104)
at
org.apache.calcite.adapter.enumerable.EnumerableAggregate.implement(EnumerableAggregate.java:105)
at
org.apache.calcite.adapter.enumerable.EnumerableRelImplementor.implementRoot(EnumerableRelImplementor.java:111)
... 22 more
Re: calcite multi-threading problem
In our use case, the Java class which opens calcite connection is contained in
a external Java jar file, which is loaded dynamicly through classloaders other
than the default systemloader like this:
try {
ClassLoader loader = URLClassLoader.newInstance(
new URL[] { new URL("jar:file:" + path + "!/") },
getClass().getClassLoader());
Class clazz = Class.forName(classPath, true, loader);
we mention this because the error message shows a classloader parameter.
-- Original --
From: "xiaobo ";;
Send time: Saturday, Feb 12, 2022 11:56 AM
To: "dev";
Subject: Re: calcite multi-threading problem
And if we remove calcite from our project, janino is not in "resolved
depencies" library list of pom.xml,
So we think it not the janino conflic problem.
-- Original --
From: "xiaobo ";;
Send time: Saturday, Feb 12, 2022 9:49 AM
To: "dev";
Subject: Re: calcite multi-threading problem
can you share the solution for the problem, we are using the following
libraries, and it seems janino 3.1.6 is only dependent by calcite-core.
UTF-8
UTF-8
9
9
9
2.17.1
7.64.0.Final
org.apache.calcite
calcite-core
1.29.0
org.apache.calcite.avatica
avatica-core
1.20.0
org.drools
drools-core
${drools.version}
org.drools
drools-compiler
${drools.version}
org.drools
drools-decisiontables
${drools.version}
-- Original --
From: "xiong duan";;
Send time: Saturday, Feb 12, 2022 8:43 AM
To: "dev";
Subject: Re: calcite multi-threading problem
Yes, It is a dependency issue. I have met the same problem. +1 for Dmitry.
Dmitry Sysolyatin 于2022年2月11日周五 21:18写道：
> Actually, I had the same problem with spark. Spark 3.2.1 uses the old
> version of janino, calcite 1.29 uses the new version of janino and they are
> not compatible.
> If I downgrade janino version to version which spark uses I got an error
> not method found exception
> If I upgrade the janino version to the version which calcite uses I get the
> same error but with another method.
>
> As a result, I built a separate calcite jar package where I shaded janino
> dependency
>
>
>
> On Fri, Feb 11, 2022 at 2:39 PM stanilovsky evgeny <
> estanilovs...@gridgain.com> wrote:
>
> > for example on a current calcite branch:
> > grep janino gradle.properties
> > janino.version=3.1.6
> >
> >
> > > yes, we use drools in the same project, and drools uses janino
> > > too, is there a version list of janino which calcite supports?
> > >
> > >
> > >
> > >
> > > ---Original---
> > > From: "Dmitry Sysolyatin" > > Date: Fri, Feb 11, 2022 18:30 PM
> > > To: "dev" > > Subject: Re: calcite multi-threading problem
> > >
> > >
> > > Hi!
> > >
> >
> It looks like a dependency issue, not like a multi-threading issue. Most
> > >
> >
> likely you have some dependency that use another version of janino but
> > > calcite requires another one
> > >
> > >
> >
> On Fri, Feb 11, 2022 at 12:18 PM xiaobo > > wrote:
> > >
> > > > Hi,
> > > >
> > > >
> > >
> >
> > we open a calcite connection with ReflectiveSchema in one thread, and
> > >
> >
> > execute sqls in other threads against the connection, but failed with
> > > > this error
> > > >
> > > >
> > > >
> > > >
> > > > com.google.common.util.concurrent.ExecutionError:
> > > > java.lang.NoSuchMethodError:
> > > >
> > >
> >
> > org.codehaus.commons.compiler.CompilerFactoryFactory.getDefaultCompilerFactory(Ljava/lang/ClassLoader;)
> > > >
> > > > Lorg/codehaus/commons/compiler/ICompilerFactory;
> > > >
> > > >
> > >
> >
> > do we miss anything regarding to multi-threading with calcite?
> >
>
Re: calcite multi-threading problem
And if we remove calcite from our project, janino is not in "resolved
depencies" library list of pom.xml,
So we think it not the janino conflic problem.
-- Original --
From: "xiaobo ";;
Send time: Saturday, Feb 12, 2022 9:49 AM
To: "dev";
Subject: Re: calcite multi-threading problem
can you share the solution for the problem, we are using the following
libraries, and it seems janino 3.1.6 is only dependent by calcite-core.
UTF-8
UTF-8
9
9
9
2.17.1
7.64.0.Final
org.apache.calcite
calcite-core
1.29.0
org.apache.calcite.avatica
avatica-core
1.20.0
org.drools
drools-core
${drools.version}
org.drools
drools-compiler
${drools.version}
org.drools
drools-decisiontables
${drools.version}
-- Original --
From: "xiong duan";;
Send time: Saturday, Feb 12, 2022 8:43 AM
To: "dev";
Subject: Re: calcite multi-threading problem
Yes, It is a dependency issue. I have met the same problem. +1 for Dmitry.
Dmitry Sysolyatin 于2022年2月11日周五 21:18写道：
> Actually, I had the same problem with spark. Spark 3.2.1 uses the old
> version of janino, calcite 1.29 uses the new version of janino and they are
> not compatible.
> If I downgrade janino version to version which spark uses I got an error
> not method found exception
> If I upgrade the janino version to the version which calcite uses I get the
> same error but with another method.
>
> As a result, I built a separate calcite jar package where I shaded janino
> dependency
>
>
>
> On Fri, Feb 11, 2022 at 2:39 PM stanilovsky evgeny <
> estanilovs...@gridgain.com> wrote:
>
> > for example on a current calcite branch:
> > grep janino gradle.properties
> > janino.version=3.1.6
> >
> >
> > > yes, we use drools in the same project, and drools uses janino
> > > too, is there a version list of janino which calcite supports?
> > >
> > >
> > >
> > >
> > > ---Original---
> > > From: "Dmitry Sysolyatin" > > Date: Fri, Feb 11, 2022 18:30 PM
> > > To: "dev" > > Subject: Re: calcite multi-threading problem
> > >
> > >
> > > Hi!
> > >
> >
> It looks like a dependency issue, not like a multi-threading issue. Most
> > >
> >
> likely you have some dependency that use another version of janino but
> > > calcite requires another one
> > >
> > >
> >
> On Fri, Feb 11, 2022 at 12:18 PM xiaobo > > wrote:
> > >
> > > > Hi,
> > > >
> > > >
> > >
> >
> > we open a calcite connection with ReflectiveSchema in one thread, and
> > >
> >
> > execute sqls in other threads against the connection, but failed with
> > > > this error
> > > >
> > > >
> > > >
> > > >
> > > > com.google.common.util.concurrent.ExecutionError:
> > > > java.lang.NoSuchMethodError:
> > > >
> > >
> >
> > org.codehaus.commons.compiler.CompilerFactoryFactory.getDefaultCompilerFactory(Ljava/lang/ClassLoader;)
> > > >
> > > > Lorg/codehaus/commons/compiler/ICompilerFactory;
> > > >
> > > >
> > >
> >
> > do we miss anything regarding to multi-threading with calcite?
> >
>
Re: can we set a default schema for calcite connection to avoid writing schema names in sql
sorry for the html escape characters,
we tried the following and it does not work
Class.forName("org.apache.calcite.jdbc.Driver");
Properties info = new Properties();
info.setProperty("lex", "JAVA");
info.setProperty(InternalProperty.CASE_SENSITIVE.name(),
"false");
info.setProperty("defaultSchema", "hr");
Connection connection =
DriverManager.getConnection("jdbc:calcite:", info);
CalciteConnection conn =
connection.unwrap(CalciteConnection.class);
SchemaPlus rootSchema = conn.getRootSchema();
Schema schema = new ReflectiveSchema(target);
rootSchema.add(schemaName, schema);
return conn;
-- Original --
From: "xiaobo ";;
Send time: Friday, Feb 11, 2022 11:20 PM
To: "dev";
Subject: can we set a default schema for calcite connection to avoid writing
schema names in sql
we have tried the following and it does not work
Class.forName("org.apache.calcite.jdbc.Driver");
Properties info = new
Properties();
info.setProperty("lex", "JAVA");
info.setProperty(CalciteConnectionProperty.CASE_SENSITIVE.camelName(), "false");
info.setProperty("defaultSchema", "hr");
try {
Connection connection =
DriverManager.getConnection("jdbc:calcite:", info);
CalciteConnection conn =
connection.unwrap(CalciteConnection.class);
SchemaPlus rootSchema = conn.getRootSchema();
Unable to implement EnumerableAggregate(group=[{}], EXPR$0=[COUNT($0)]) with ReflectiveSchema
Simple count sql against ReflectiveSchema failed with :
Error while executing SQL "select count(a.PA01AI01) from p.PRH_PA01 as a
": Unable to implement EnumerableAggregate(group=[{}], EXPR$0=[COUNT($0)]):
rowcount = 10.0, cumulative cost = {111.25 rows, 101.0 cpu, 0.0 io}, id = 37
EnumerableTableScan(table=[[p, PRH_PA01]]): rowcount = 100.0, cumulative cost
= {100.0 rows, 101.0 cpu, 0.0 io}, id = 26
at
com.xsmartware.javatest.pbc2p.CalciteBaseVarGenerator.generateVariable(CalciteBaseVarGenerator.java:61)
at
com.xsmartware.javatest.pbc2p.PBC2PTestRunner.test1(PBC2PTestRunner.java:68)
at
com.xsmartware.javatest.pbc2p.PBC2PTestRunner.run(PBC2PTestRunner.java:54)
at
org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:758)
at
org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:748)
at
org.springframework.boot.SpringApplication.run(SpringApplication.java:309)
at
org.springframework.boot.SpringApplication.run(SpringApplication.java:1301)
at
org.springframework.boot.SpringApplication.run(SpringApplication.java:1290)
at
com.xsmartware.javatest.JavaTestApplication.main(JavaTestApplication.java:9)
Caused by: java.sql.SQLException: Error while executing SQL "select
count(a.PA01AI01) from p.PRH_PA01 as a
": Unable to implement EnumerableAggregate(group=[{}], EXPR$0=[COUNT($0)]):
rowcount = 10.0, cumulative cost = {111.25 rows, 101.0 cpu, 0.0 io}, id = 37
EnumerableTableScan(table=[[p, PRH_PA01]]): rowcount = 100.0, cumulative cost
= {100.0 rows, 101.0 cpu, 0.0 io}, id = 26
at org.apache.calcite.avatica.Helper.createException(Helper.java:56)
at org.apache.calcite.avatica.Helper.createException(Helper.java:41)
at
org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:163)
at
org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:227)
at com.xsmartware.common.util.SQLUtil.exeGetLong(SQLUtil.java:42)
at
com.xsmartware.javatest.pbc2p.CalciteBaseVarGenerator.generateVariable(CalciteBaseVarGenerator.java:39)
... 8 more
Caused by: java.lang.IllegalStateException: Unable to implement
EnumerableAggregate(group=[{}], EXPR$0=[COUNT($0)]): rowcount = 10.0,
cumulative cost = {111.25 rows, 101.0 cpu, 0.0 io}, id = 37
EnumerableTableScan(table=[[p, PRH_PA01]]): rowcount = 100.0, cumulative cost
= {100.0 rows, 101.0 cpu, 0.0 io}, id = 26
at
org.apache.calcite.adapter.enumerable.EnumerableRelImplementor.implementRoot(EnumerableRelImplementor.java:114)
at
org.apache.calcite.adapter.enumerable.EnumerableInterpretable.toBindable(EnumerableInterpretable.java:114)
at
org.apache.calcite.prepare.CalcitePrepareImpl$CalcitePreparingStmt.implement(CalcitePrepareImpl.java:1130)
at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:324)
at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:220)
at
org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:647)
at
org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:513)
at
org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:483)
at
org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:249)
at
org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:623)
at
org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:674)
at
org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156)
... 11 more
Suppressed: java.lang.NullPointerException: relFieldType.getComponentType()
for JavaType(interface java.util.List)
at java.base/java.util.Objects.requireNonNull(Objects.java:347)
at
org.apache.calcite.adapter.enumerable.EnumerableTableScan.fieldExpression(EnumerableTableScan.java:269)
at
org.apache.calcite.adapter.enumerable.EnumerableTableScan.toRows(EnumerableTableScan.java:252)
at
org.apache.calcite.adapter.enumerable.EnumerableTableScan.getExpression(EnumerableTableScan.java:209)
at
org.apache.calcite.adapter.enumerable.EnumerableTableScan.implement(EnumerableTableScan.java:340)
at
org.apache.calcite.adapter.enumerable.EnumerableRelImplementor.visitChild(EnumerableRelImplementor.java:104)
at
org.apache.calcite.adapter.enumerable.EnumerableAggregate.implement(EnumerableAggregate.java:105)
at
org.apache.calcite.adapter.enumerable.EnumerableRelImplementor.implementRoot(EnumerableRelImplementor.java:111)
... 22 more
Re: 回复：cannot be cast to class org.apache.calcite.runtime.FlatLists$ComparableList error
Sorry for the HTML escape characters, I rewrite the problem as following: SQL select count(a.PA01AI01) from p.PRH_PA01 as a failed with : Caused by: java.sql.SQLException: Error while executing SQL "select count(a.PA01AI01) from p.PRH_PA01 as a ": From line 1, column 16 to line 1, column 23: Column 'PA01AI01' not found in table 'a' to resolve the problem we must make the table fields of the target Java class as public, is this a rule for ReflectiveSchema?
Re: calcite multi-threading problem
can you share the solution for the problem, we are using the following
libraries, and it seems janino 3.1.6 is only dependent by calcite-core.
UTF-8
UTF-8
9
9
9
2.17.1
7.64.0.Final
org.apache.calcite
calcite-core
1.29.0
org.apache.calcite.avatica
avatica-core
1.20.0
org.drools
drools-core
${drools.version}
org.drools
drools-compiler
${drools.version}
org.drools
drools-decisiontables
${drools.version}
-- Original --
From: "xiong duan";;
Send time: Saturday, Feb 12, 2022 8:43 AM
To: "dev";
Subject: Re: calcite multi-threading problem
Yes, It is a dependency issue. I have met the same problem. +1 for Dmitry.
Dmitry Sysolyatin 于2022年2月11日周五 21:18写道：
> Actually, I had the same problem with spark. Spark 3.2.1 uses the old
> version of janino, calcite 1.29 uses the new version of janino and they are
> not compatible.
> If I downgrade janino version to version which spark uses I got an error
> not method found exception
> If I upgrade the janino version to the version which calcite uses I get the
> same error but with another method.
>
> As a result, I built a separate calcite jar package where I shaded janino
> dependency
>
>
>
> On Fri, Feb 11, 2022 at 2:39 PM stanilovsky evgeny <
> estanilovs...@gridgain.com> wrote:
>
> > for example on a current calcite branch:
> > grep janino gradle.properties
> > janino.version=3.1.6
> >
> >
> > > yes, we use drools in the same project, and drools uses janino
> > > too, is there a version list of janino which calcite supports?
> > >
> > >
> > >
> > >
> > > ---Original---
> > > From: "Dmitry Sysolyatin" > > Date: Fri, Feb 11, 2022 18:30 PM
> > > To: "dev" > > Subject: Re: calcite multi-threading problem
> > >
> > >
> > > Hi!
> > >
> >
> It looks like a dependency issue, not like a multi-threading issue. Most
> > >
> >
> likely you have some dependency that use another version of janino but
> > > calcite requires another one
> > >
> > >
> >
> On Fri, Feb 11, 2022 at 12:18 PM xiaobo > > wrote:
> > >
> > > > Hi,
> > > >
> > > >
> > >
> >
> > we open a calcite connection with ReflectiveSchema in one thread, and
> > >
> >
> > execute sqls in other threads against the connection, but failed with
> > > > this error
> > > >
> > > >
> > > >
> > > >
> > > > com.google.common.util.concurrent.ExecutionError:
> > > > java.lang.NoSuchMethodError:
> > > >
> > >
> >
> > org.codehaus.commons.compiler.CompilerFactoryFactory.getDefaultCompilerFactory(Ljava/lang/ClassLoader;)
> > > >
> > > > Lorg/codehaus/commons/compiler/ICompilerFactory;
> > > >
> > > >
> > >
> >
> > do we miss anything regarding to multi-threading with calcite?
> >
>
Re: calcite multi-threading problem
sorry about that, I'll correct it. -- Original -- From: "Julian Hyde";; Send time: Saturday, Feb 12, 2022 9:41 AM To: "dev"; Subject: Re: calcite multi-threading problem xiaobo, Your emails are coming through with a lot of HTML escape characters [1]. To make them more readable for everyone else, could you perhaps try sending in “Plain text” mode? Julian [1] https://lists.apache.org/thread/ttrlw4c441br27lppxl77osbgxrv58wn > On Feb 11, 2022, at 5:36 PM, xiaobo wrote: > > can you share the solution for the problem, we are using the following > libraries, and it seems janino 3.1.6 is only dependent by calcite-core. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- Original -- > From: "xiong duan"; Send time: Saturday, Feb 12, 2022 8:43 AM > To: "dev" > Subject: Re: calcite multi-threading problem > > > > Yes, It is a dependency issue. I have met the same problem. +1 for Dmitry. > > Dmitry Sysolyatin > > Actually, I had the same problem with spark. Spark 3.2.1 uses the old > > version of janino, calcite 1.29 uses the new version of janino and they > are > > not compatible. > > If I downgrade janino version to version which spark uses I got an error > > not method found exception > > If I upgrade the janino version to the version which calcite uses I get > the > > same error but with another method. > > > > As a result, I built a separate calcite jar package where I shaded janino > > dependency > > > > > > > > On Fri, Feb 11, 2022 at 2:39 PM stanilovsky evgeny < > > estanilovs...@gridgain.com> wrote: > > > > > for example on a current calcite branch: > > > grep janino gradle.properties > > > janino.version=3.1.6 > > > > > > > > > > yes, we use drools in the same project, and drools > uses janino > > > > too, is there a version list of janino which calcite supports? > > > > > > > > > > > > > > > > > > > > ---Original--- > > > > From: "Dmitry Sysolyatin" > > > Date: Fri, Feb 11, 2022 18:30 PM > > > > To: "dev" > > > Subject: Re: calcite multi-threading problem > > > > > > > > > > > > Hi! > > > > > > > > > > It looks like a dependency issue, not like a multi-threading issue. Most > > > > > > > > > > likely you have some dependency that use another version of janino but > > > > calcite requires another one > > > > > > > > > > > > > > On Fri, Feb 11, 2022 at 12:18 PM xiaobo > > > wrote: > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > we open a calcite connection with ReflectiveSchema in one thread, and > > > > > > > > > > > execute sqls in other threads against the connection, but failed with > > > > > this error > > > > > > > > > > > > > > > > > > > > > > > > > > com.google.common.util.concurrent.ExecutionError: > > > > > java.lang.NoSuchMethodError: > > > > > > > > > > > > > > > > org.codehaus.commons.compiler.CompilerFactoryFactory.getDefaultCompilerFactory(Ljava/lang/ClassLoader;) > > > > > > > > > > > Lorg/codehaus/commons/compiler/ICompilerFactory; > > > > > > > > > > > > > > > > > > > > > do we miss anything regarding to multi-threading with calcite? > > > > >
Re: calcite multi-threading problem
xiaobo, Your emails are coming through with a lot of HTML escape characters [1]. To make them more readable for everyone else, could you perhaps try sending in “Plain text” mode? Julian [1] https://lists.apache.org/thread/ttrlw4c441br27lppxl77osbgxrv58wn > On Feb 11, 2022, at 5:36 PM, xiaobo wrote: > > can you share the solution for the problem, we are using the following > libraries, and it seems janino 3.1.6 is only dependent by calcite-core. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- Original -- > From: "xiong duan"; Send time: Saturday, Feb 12, 2022 8:43 AM > To: "dev" > Subject: Re: calcite multi-threading problem > > > > Yes, It is a dependency issue. I have met the same problem. +1 for Dmitry. > > Dmitry Sysolyatin > > Actually, I had the same problem with spark. Spark 3.2.1 uses the old > > version of janino, calcite 1.29 uses the new version of janino and they > are > > not compatible. > > If I downgrade janino version to version which spark uses I got an error > > not method found exception > > If I upgrade the janino version to the version which calcite uses I get > the > > same error but with another method. > > > > As a result, I built a separate calcite jar package where I shaded janino > > dependency > > > > > > > > On Fri, Feb 11, 2022 at 2:39 PM stanilovsky evgeny < > > estanilovs...@gridgain.com> wrote: > > > > > for example on a current calcite branch: > > > grep janino gradle.properties > > > janino.version=3.1.6 > > > > > > > > > > yes, we use drools in the same project, and drools > uses janino > > > > too, is there a version list of janino which calcite supports? > > > > > > > > > > > > > > > > > > > > ---Original--- > > > > From: "Dmitry Sysolyatin" > > > Date: Fri, Feb 11, 2022 18:30 PM > > > > To: "dev" > > > Subject: Re: calcite multi-threading problem > > > > > > > > > > > > Hi! > > > > > > > > > > It looks like a dependency issue, not like a multi-threading issue. Most > > > > > > > > > > likely you have some dependency that use another version of janino but > > > > calcite requires another one > > > > > > > > > > > > > > On Fri, Feb 11, 2022 at 12:18 PM xiaobo > > > wrote: > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > we open a calcite connection with ReflectiveSchema in one thread, and > > > > > > > > > > > execute sqls in other threads against the connection, but failed with > > > > > this error > > > > > > > > > > > > > > > > > > > > > > > > > > com.google.common.util.concurrent.ExecutionError: > > > > > java.lang.NoSuchMethodError: > > > > > > > > > > > > > > > > org.codehaus.commons.compiler.CompilerFactoryFactory.getDefaultCompilerFactory(Ljava/lang/ClassLoader;) > > > > > > > > > > > Lorg/codehaus/commons/compiler/ICompilerFactory; > > > > > > > > > > > > > > > > > > > > > do we miss anything regarding to multi-threading with calcite? > > > > >
Re: calcite multi-threading problem
can you share the solution for the problem, we are using the following libraries, and it seems janino 3.1.6 is only dependent by calcite-core.
Re: calcite multi-threading problem
Yes, It is a dependency issue. I have met the same problem. +1 for Dmitry. Dmitry Sysolyatin 于2022年2月11日周五 21:18写道： > Actually, I had the same problem with spark. Spark 3.2.1 uses the old > version of janino, calcite 1.29 uses the new version of janino and they are > not compatible. > If I downgrade janino version to version which spark uses I got an error > not method found exception > If I upgrade the janino version to the version which calcite uses I get the > same error but with another method. > > As a result, I built a separate calcite jar package where I shaded janino > dependency > > > > On Fri, Feb 11, 2022 at 2:39 PM stanilovsky evgeny < > estanilovs...@gridgain.com> wrote: > > > for example on a current calcite branch: > > grep janino gradle.properties > > janino.version=3.1.6 > > > > > > > yes, we use drools in the same project, and drools uses janino > > > too, is there a version list of janino which calcite supports? > > > > > > > > > > > > > > > ---Original--- > > > From: "Dmitry Sysolyatin" > > Date: Fri, Feb 11, 2022 18:30 PM > > > To: "dev" > > Subject: Re: calcite multi-threading problem > > > > > > > > > Hi! > > > > > > It looks like a dependency issue, not like a multi-threading issue. Most > > > > > > likely you have some dependency that use another version of janino but > > > calcite requires another one > > > > > > > > > On Fri, Feb 11, 2022 at 12:18 PM xiaobo > > wrote: > > > > > > > Hi, > > > > > > > > > > > > > > > we open a calcite connection with ReflectiveSchema in one thread, and > > > > > > > execute sqls in other threads against the connection, but failed with > > > > this error > > > > > > > > > > > > > > > > > > > > com.google.common.util.concurrent.ExecutionError: > > > > java.lang.NoSuchMethodError: > > > > > > > > > > > org.codehaus.commons.compiler.CompilerFactoryFactory.getDefaultCompilerFactory(Ljava/lang/ClassLoader;) > > > > > > > > Lorg/codehaus/commons/compiler/ICompilerFactory; > > > > > > > > > > > > > > > do we miss anything regarding to multi-threading with calcite? > > >
Re: Contributor role request
Hey Thomas, I've added you as a contributor in jira. Francis On 12/02/2022 9:00 am, Thomas D'Silva wrote: Hi. I would like to contribute to the calcite project, could you please grant me the contributor role? My Apache JIRA login is tdsilva. Thank you, Thomas D'Silva
Contributor role request
Hi. I would like to contribute to the calcite project, could you please grant me the contributor role? My Apache JIRA login is tdsilva. Thank you, Thomas D'Silva
[jira] [Created] (CALCITE-5010) Modify RexBuilder to use convertlets to transform the RexCall when possible
Thomas D'Silva created CALCITE-5010:
---
Summary: Modify RexBuilder to use convertlets to transform the
RexCall when possible
Key: CALCITE-5010
URL: https://issues.apache.org/jira/browse/CALCITE-5010
Project: Calcite
Issue Type: Improvement
Components: core
Reporter: Thomas D'Silva
SqlRexConvertlets are used to transform a SqlCall into a RexCall while parsing
a sql string. A RelNode that is created using RelBuilder does not use the same
transformations. This causes the following error while trying to use RelBuilder
to create a relnode that use {{{}TIMESTAMPDIFF(){}}}.
{code:java}
Suppressed: java.lang.RuntimeException: cannot translate call
TIMESTAMPDIFF($t8, $t4, $t9)
at
org.apache.calcite.adapter.enumerable.RexToLixTranslator.visitCall(RexToLixTranslator.java:1157)
at
org.apache.calcite.adapter.enumerable.RexToLixTranslator.visitCall(RexToLixTranslator.java:98)
{code}
Modify RexBuilder to check if there is a convertlet that exists for the
operator being constructed and transform the RexCall using the convertlet.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
回复：cannot be cast to class org.apache.calcite.runtime.FlatLists$ComparableList error
the first filed's java class is as following, and SQL
select count(a.PA01AI01) from p.PRH_PA01 as a
failed with :
Caused by: java.sql.SQLException: Error while executing SQL "select
count(a.PA01AI01) from p.PRH_PA01 as a
": From line 1, column 16 to line 1, column 23: Column 'PA01AI01' not found in
table 'a'
public class PRH_PA01 {
String PA01AI01;
Date PA01AR01;
String PA01BQ01;
String PA01BD01;
String PA01BI01;
String PA01BI02;
String PA01BD02;
Long PA01CS01 ;
String PA01DQ01 ;
String PA01DQ02 ;
Date PA01DR01;
Date PA01DR02 ;
Long PA01ES01;
List
can we set a default schema for calcite connection to avoid writing schema names in sql
we have tried the following and it does not work
Class.forName("org.apache.calcite.jdbc.Driver");
Properties info = new
Properties();
info.setProperty("lex", "JAVA");
info.setProperty(CalciteConnectionProperty.CASE_SENSITIVE.camelName(), "false");
info.setProperty("defaultSchema", "hr");
try {
Connection connection =
DriverManager.getConnection("jdbc:calcite:", info);
CalciteConnection conn =
connection.unwrap(CalciteConnection.class);
SchemaPlus rootSchema = conn.getRootSchema();
回复：reply： calcite multi-threading problem
I mean it seems not a janino related problem, if we put the connection opening and sql executing operationes in the same thread the same error does not appear, but with a sql related problem with I sent in another mail. -- 原始邮件 -- 发件人: "xiaobo "
reply： calcite multi-threading problem
we use maven and the effective pom contains only the 3.1.0 version of janino as following:
Re: calcite multi-threading problem
Actually, I had the same problem with spark. Spark 3.2.1 uses the old version of janino, calcite 1.29 uses the new version of janino and they are not compatible. If I downgrade janino version to version which spark uses I got an error not method found exception If I upgrade the janino version to the version which calcite uses I get the same error but with another method. As a result, I built a separate calcite jar package where I shaded janino dependency On Fri, Feb 11, 2022 at 2:39 PM stanilovsky evgeny < estanilovs...@gridgain.com> wrote: > for example on a current calcite branch: > grep janino gradle.properties > janino.version=3.1.6 > > > > yes, we use drools in the same project, and drools uses janino > > too, is there a version list of janino which calcite supports? > > > > > > > > > > ---Original--- > > From: "Dmitry Sysolyatin" > Date: Fri, Feb 11, 2022 18:30 PM > > To: "dev" > Subject: Re: calcite multi-threading problem > > > > > > Hi! > > > It looks like a dependency issue, not like a multi-threading issue. Most > > > likely you have some dependency that use another version of janino but > > calcite requires another one > > > > > On Fri, Feb 11, 2022 at 12:18 PM xiaobo > wrote: > > > > > Hi, > > > > > > > > > > we open a calcite connection with ReflectiveSchema in one thread, and > > > > execute sqls in other threads against the connection, but failed with > > > this error > > > > > > > > > > > > > > > com.google.common.util.concurrent.ExecutionError: > > > java.lang.NoSuchMethodError: > > > > > > > org.codehaus.commons.compiler.CompilerFactoryFactory.getDefaultCompilerFactory(Ljava/lang/ClassLoader;) > > > > > > Lorg/codehaus/commons/compiler/ICompilerFactory; > > > > > > > > > > do we miss anything regarding to multi-threading with calcite? >
Re: calcite multi-threading problem
for example on a current calcite branch: grep janino gradle.properties janino.version=3.1.6 yes, we use drools in the same project, and drools uses janino too, is there a version list of janino which calcite supports? ---Original--- From: "Dmitry Sysolyatin"
Re: calcite multi-threading problem
yes, we use drools in the same project, and drools uses janino too, is there a version list of janino which calcite supports? ---Original--- From: "Dmitry Sysolyatin"
Re: calcite multi-threading problem
yes, we use drools in the same project, and drools uses janino too, is there a version list of janino which janino supports? ---Original--- From: "Dmitry Sysolyatin"
Request for review
Hi everyone! I published a PR https://github.com/apache/calcite/pull/2718 concerning the following Jira issue: https://issues.apache.org/jira/browse/CALCITE-5008 Could please someone review the PR? The CI seems to be ok with it. If I'm not doing the correct thing requesting a review here, please let me know how this should be done. Roman Puchkovskiy
Re: calcite multi-threading problem
Hi! It looks like a dependency issue, not like a multi-threading issue. Most likely you have some dependency that use another version of janino but calcite requires another one On Fri, Feb 11, 2022 at 12:18 PM xiaobo wrote: > Hi, > > > we open a calcite connection with ReflectiveSchema in one thread, and > execute sqls in other threads against the connection, but failed with > this error > > > > > com.google.common.util.concurrent.ExecutionError: > java.lang.NoSuchMethodError: > > org.codehaus.commons.compiler.CompilerFactoryFactory.getDefaultCompilerFactory(Ljava/lang/ClassLoader;) > > Lorg/codehaus/commons/compiler/ICompilerFactory; > > > do we miss anything regarding to multi-threading with calcite?
calcite multi-threading problem
Hi, we open a calcite connection with ReflectiveSchema in one thread, and execute sqls in other threads against the connection, but failed with this error com.google.common.util.concurrent.ExecutionError: java.lang.NoSuchMethodError: org.codehaus.commons.compiler.CompilerFactoryFactory.getDefaultCompilerFactory(Ljava/lang/ClassLoader;) Lorg/codehaus/commons/compiler/ICompilerFactory; do we miss anything regarding to multi-threading with calcite?
Re: [DISCUSS] Assigning reviewers
Hey, I think the main problem is still the lack of active committers/reviewers and not so much the assignment. Totally agree - I think this is the core of the problem; the assign-by-files could somewhat help pulling in the right people - but will not make the job done. We could opt for assigning PRs automatically using filename patterns. This can be done rather easily and it's already done in various projects e.g., Hive [1]. I pioneered to make that happen - but I think it didn't really lifted off; for something like this to work; people should sign on volunteeringly (in which I think Calcite is stronger). I would definetly sign on to review some parts - but now I know: that even with these requests there is no guarantee that I'll take a look... One aspect of implementing this was to provide a way to keep an eye on key points of the project (in case of Hive: thrift api; metastore schema) to avoid possible issues arising from conceptional problems. Julian> Suppose that six committers volunteer to review and merge up to six PRs per quarter I think this is an interesting idea but given the broad range of things we have in Calcite; a PR could range from linq4j to say the Geode adapter... If you open https://github.com/apache/calcite/pulls how many you feel confident enough to give a "merge" (optionally after chanages) or "close PR" decision for it? I think what makes this project great is that we have a lot of people here with strong views and high standards - this is good in a sense that the best solution is choosen in most cases. The presence of the above also puts more weight on committers to accept only changes which are up to these high standards. I'm not sure how many of you have merged in a PR from a contributor which contained serious issues - which have surfaced later and caused issues. Did you feeled ashamed that you let that patch in - and was not doing your part correctly. Because I did... I know its community effort and everything but afterall it was me who pushed the merge button... I had the opportunity to experience a support process built on a similar principle "you just have to carry it" and I have to say that a process like this to work I see the following options: * we should have people to ask help from - who actually do respond ** note that in this case the review and the decision to accept the change would *still* stay with the original owner/etc - so it will not mean much improvement beyond having 1 more people in the chain who is shouting out for others... * accept that sometimes non-optimal solution will be choosen (which are in most cases still valuable) I think in case of a release its more clear what people are signing up for; its documented/etc. For reviews its a bit different; with the proposed system: * what if you don't find 6 PRs you are confident to be able to review in a quarter? * there will be 6 people on this mission at the same time - which could further increase the chance that people would not take risks cheers, Zoltan On 2/7/22 10:56 PM, Stamatis Zampetakis wrote: Hello, I think the main problem is still the lack of active committers/reviewers and not so much the assignment. Nevertheless, things would be a bit better if we had people assigned to PRs. This could certainly help at least offload some work from Julian and possibly sensibilize more people towards the reviewing process. We could opt for assigning PRs automatically using filename patterns. This can be done rather easily and it's already done in various projects e.g., Hive [1]. What do you think of putting automatic assignment in place for Calcite? Who is willing to put their name in the list? At this point I am not expecting that people in the list will review everything they are assigned on but maybe they can help out by pointing to other people or simply setting up the expectations about the PR getting reviewed. Best, Stamatis [1] https://github.com/apache/hive/commit/2bd6a9d63c28e5cb9bcc44115262d565cdc2bb90 On Sat, Feb 5, 2022 at 5:52 PM Jing Zhang wrote: Hi, Julian There is no doubt that you are the most prolific contributors on the project. People often hope to hear professional advice from you because you are the Calcite project authority. However people may didn't realize that you are suffering from more and more requests. I want to sincerely apologize because I often disturb you too. I'm really glad to hear your feelings. At the same time, I want to thank you because I got many professional advices from you. The suggestions are very helpful. Back to this topic, having an efficient mechanism to merge contributors' PR is very important to the long-term development of open source projects. I would like to share my thoughts, hope it helps. 1. It might helpful to know which members are proficient in which modules. For example, introduce each member's familiar module on the website[1]. There may be many requests sent to other members. 2. For some module
cannot be cast to class org.apache.calcite.runtime.FlatLists$ComparableList error
Hi,
Our testing code failed with the following error, can you help with it:
Error while executing SQL "select count(1)
from p.PRH_PA01
": class com.xsmartware.pbcm.sdk.pbc2p.PRH_PA01 cannot be cast to class
org.apache.calcite.runtime.FlatLists$ComparableList
(com.xsmartware.pbcm.sdk.pbc2p.PRH_PA01 and
org.apache.calcite.runtime.FlatLists$ComparableList are in unnamed module of
loader 'app')
at
com.xsmartware.javatest.pbc2p.CalciteBaseVarGenerator.generateVariable(CalciteBaseVarGenerator.java:61)
at
com.xsmartware.javatest.pbc2p.PBC2PTestRunner.test1(PBC2PTestRunner.java:68)
at
com.xsmartware.javatest.pbc2p.PBC2PTestRunner.run(PBC2PTestRunner.java:54)
at
org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:758)
at
org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:748)
at
org.springframework.boot.SpringApplication.run(SpringApplication.java:309)
at
org.springframework.boot.SpringApplication.run(SpringApplication.java:1301)
at
org.springframework.boot.SpringApplication.run(SpringApplication.java:1290)
at
com.xsmartware.javatest.JavaTestApplication.main(JavaTestApplication.java:9)
Caused by: java.sql.SQLException: Error while executing SQL "select count(1)
from p.PRH_PA01
": class com.xsmartware.pbcm.sdk.pbc2p.PRH_PA01 cannot be cast to class
org.apache.calcite.runtime.FlatLists$ComparableList
(com.xsmartware.pbcm.sdk.pbc2p.PRH_PA01 and
org.apache.calcite.runtime.FlatLists$ComparableList are in unnamed module of
loader 'app')
at
org.apache.calcite.avatica.Helper.createException(Helper.java:56)
at
org.apache.calcite.avatica.Helper.createException(Helper.java:41)
at
org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:163)
at
org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:227)
at
com.xsmartware.common.util.SQLUtil.exeGetLong(SQLUtil.java:42)
at
com.xsmartware.javatest.pbc2p.CalciteBaseVarGenerator.generateVariable(CalciteBaseVarGenerator.java:39)
... 8 more
Caused by: java.lang.ClassCastException: class
com.xsmartware.pbcm.sdk.pbc2p.PRH_PA01 cannot be cast to class
org.apache.calcite.runtime.FlatLists$ComparableList
(com.xsmartware.pbcm.sdk.pbc2p.PRH_PA01 and
org.apache.calcite.runtime.FlatLists$ComparableList are in unnamed module of
loader 'app')
at Baz$1.apply(Unknown Source)
at Baz$1.apply(Unknown Source)
at
org.apache.calcite.adapter.enumerable.BasicAggregateLambdaFactory$AccumulatorAdderSeq.apply(BasicAggregateLambdaFactory.java:81)
at
org.apache.calcite.linq4j.EnumerableDefaults.aggregate(EnumerableDefaults.java:133)
at
org.apache.calcite.linq4j.DefaultEnumerable.aggregate(DefaultEnumerable.java:107)
at Baz.bind(Unknown Source)
at
org.apache.calcite.jdbc.CalcitePrepare$CalciteSignature.enumerable(CalcitePrepare.java:363)
at
org.apache.calcite.jdbc.CalciteConnectionImpl.enumerable(CalciteConnectionImpl.java:338)
at
org.apache.calcite.jdbc.CalciteMetaImpl._createIterable(CalciteMetaImpl.java:578)
at
org.apache.calcite.jdbc.CalciteMetaImpl.createIterable(CalciteMetaImpl.java:569)
at
org.apache.calcite.avatica.AvaticaResultSet.execute(AvaticaResultSet.java:184)
at
org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:64)
at
org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:43)
at
org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:666)
at
org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:638)
at
org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:674)
at
org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156)
... 11 more
we use ReflectiveSchema with the following target class:
public class PBC2PCalCiteSchema {
public PRH_PA01[] PRH_PA01 = {};
public PRH_PA01CH[] PRH_PA01CH = {};
public PIM_PB01A[] PIM_PB01A= {};
public PIM_PB01B[] PIM_PB01B= {};
public PMM_PB02[] PMM_PB02= {};
public PRM_PB03[] PRM_PB03= {};
public POM_PB04[] POM_PB04= {};
public PSM[]
PSM= {};
public PSM_FACTORS[] PSM_FACTORS= {};
public PCO_PC02[] PCO_PC02= {};
public PCO_PC02AH[] PCO_PC02AH= {};
public PCO_PC02BH[] PCO_PC02BH= {};
public PCO_PC02DH[] PCO_PC02DH= {};
public PCO_PC02KH[] PCO_PC02KH= {};
public PNO_PC03[] PNO_PC03= {};
public PNO_PC030H[] PNO_PC030H= {};
public PPO_PC04[] PPO_PC04= {};
public PPO_PC040H[] PPO_PC040H= {};
public PQO_PC05[] PQO_PC05= {};
public PDA_PD01[] PDA_PD01= {};
public PDA_PD01DH[] PDA_PD01DH= {};
public PDA_PD01EH[] PDA_PD01EH= {};
public PDA_PD01FH[] PDA_PD01FH= {};
public PDA_PD01GH[] PDA_PD01GH= {};
public PDA_PD01HH[] PDA_PD01HH= {};
public PDA_PD01ZH[] PDA_PD01ZH= {};
public P
Re: Allow Cascades driver invoking "derive" on the nodes produced by "passThrough"
Hello everyone,
@Vladimir, +1 on the change introducing "enforceDerive()".
@Roman, could you walk us through the limitations you found that forced you
to copy-paste the whole class?
Maybe there is some middle ground for your problem(s) too, similar in
spirit to what Vladimir proposed for the other limitation.
I am not against making the class more public if necessary, but it would be
nice to have a discussion here before going down that path.
If the discussion leads to a better design of the original class, all
projects would benefit from that.
Best regards,
Alessandro
On Fri, 11 Feb 2022 at 04:14, Roman Kondakov
wrote:
> Hi Vladimir,
>
> +1 for making the rule driver more public. We've faced similar problems
> in the downstream project. The solution was to copy and paste the
> TopDownRuleDrive code with small fixes since it was not possible to
> override the default behavior.
>
> --
> Roman Kondakov
>
>
> On 11.02.2022 02:50, Vladimir Ozerov wrote:
> > Hi,
> >
> > In the Cascades driver, it is possible to propagate the requests top-down
> > using the "passThrough", method and then notify parents bottom-up about
> the
> > concrete physical implementations of inputs using the "derive" method.
> >
> > In some optimizers, the valid parent node cannot be created before the
> > trait sets of inputs are known. An example is a custom distribution trait
> > that includes the number of shards in the system. The parent operator
> alone
> > may guess the distribution keys, but cannot know the number of input
> > shards. To mitigate this, you may create a "template" node with an
> infinite
> > cost from within the optimization rule that will propagate the
> > passThrough/drive calls but would never participate in the final plan.
> >
> > Currency, the top-down driver designed in a way that the nodes created
> from
> > the "passThrough" method are not notified on the "derive" stage. This
> leads
> > to the incomplete exploration of the search space. For example, the rule
> > may produce the node "A1.template" that will be converted into a normal
> > "A1" node in the derive phase. However, if the parent operator produced
> > "A2.template" from "A1.template" using pass-through mechanics, the
> > "A2.template" will never be notified about the concrete input traits,
> > possibly losing the optimal plan. This is especially painful in
> distributed
> > engines, where the number of shards is important for the placement of
> > Shuffle operators.
> >
> > It seems that the problem could be solved with relatively low effort. The
> > "derive" is not invoked on the nodes created from the "passThrough"
> method,
> > because such nodes are placed in the "passThroughCache" collection.
> Instead
> > of doing this unconditionally, we may introduce an additional predicate
> > that would selectively enforce "derive" on such nodes. For example, this
> > could be a default method in the PhysicalNode interface, like:
> >
> > interface PhysicalNode {
> >default boolean enforceDerive() { return false; }
> > }
> >
> > If there are no objections, I'll proceed with this change.
> >
> > Alternatively, we may make the TopDownRuleDriver more "public", so that
> the
> > user can extend it and decide within the driver whether to cache a
> > particular node or not.
> >
> > I would appreciate your feedback on the matter.
> >
> > Regards,
> > Vladimir.
> >
>
