Top down planner
Hello friends, What is the best resource for learning how to use the top down planner flag in calcite? Are there any open source projects that use that capability that I can study to learn more? I would like to see examples of projects that use this capability to "distribute" a plan over multiple nodes using traits like distribution / collation etc. Regards!
Re: mapPartitions
Well I think the expected approach is to translate other operations to relational operators by yourself ;-) I think Calcite won't be the place to have extensions for such translation. The main concern is that those non relational operations are "non-standard". -Rui On Wed, Mar 3, 2021 at 10:12 PM Debajyoti Roy wrote: > Hi All, > > For operators like filter, join, union, aggregate, window the > logical RelNode choices are obvious within org.apache.calcite.rel.logical > package. > > However, for more complex applications that require operations like > mapPartitions, flatMapGroupsWithState, etc. what would be some choices for > logical rel node and relational expression classes in Apache Calcite > (independent of engine)? > > What is a good way to model operators that are not traditionally relational > and hence do not exist in Calcite (looking for hooks / extension points / > roadmaps)? > > Thanks in advance for any pointers, (disclaimer: I am new to Calcite) > Debajyoti Roy >
[jira] [Created] (CALCITE-4521) Support User Defined Table-valued Function
herodu created CALCITE-4521:
---
Summary: Support User Defined Table-valued Function
Key: CALCITE-4521
URL: https://issues.apache.org/jira/browse/CALCITE-4521
Project: Calcite
Issue Type: Improvement
Components: core
Reporter: herodu
It already support TUMBLE/HOP/SESSION windowing table-valued function in
current version. But in fact, the table-valued function is not just the current
features, so we want to support user defined table-valued function, just like
the following query:
{code:java}
// code placeholder
select * from table(UDTVF(table Shipments))
{code}
or
{code:java}
// code placeholder
select * from table(UDTVF(select * from Shipments)){code}
The input parameters of 'UDTVF' is the output rowtype of the inner subquery,
and the output of 'UDTVF' is defined by user, like:
{code:java}
// code placeholder
final TableValuedFunction tableValuedFunction =
(RelDataTypeFactory typeFactory) -> typeFactory.builder()
.add("in1", SqlTypeName.INTEGER)
.add("in2", SqlTypeName.VARCHAR)
.build();
{code}
And the expected output logic plan like:
{code:java}
// code placeholder
LogicalProject(in1=[$0], in2=[$1])
LogicalTableFunctionScan(invocation=[UDTVF($0, $1)],
rowType=[RecordType(INTEGER in1,VARCHAR in2)])
LogicalProject(ORDERID=[$0], ROWTIME=[$1])
LogicalTableScan(table=[[CATALOG, SALES, SHIPMENTS]])
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
mapPartitions
Hi All, For operators like filter, join, union, aggregate, window the logical RelNode choices are obvious within org.apache.calcite.rel.logical package. However, for more complex applications that require operations like mapPartitions, flatMapGroupsWithState, etc. what would be some choices for logical rel node and relational expression classes in Apache Calcite (independent of engine)? What is a good way to model operators that are not traditionally relational and hence do not exist in Calcite (looking for hooks / extension points / roadmaps)? Thanks in advance for any pointers, (disclaimer: I am new to Calcite) Debajyoti Roy
[jira] [Created] (CALCITE-4520) In ImmutableBeans, nullable properties become not-nullable in project that does not use checkerframework
Julian Hyde created CALCITE-4520:
Summary: In ImmutableBeans, nullable properties become
not-nullable in project that does not use checkerframework
Key: CALCITE-4520
URL: https://issues.apache.org/jira/browse/CALCITE-4520
Project: Calcite
Issue Type: Bug
Reporter: Julian Hyde
In ImmutableBeans, nullable properties become not-nullable in project that does
not use checkerframework. In my Morel project, I get the following error:
{noformat}
java.lang.ExceptionInInitializerError
at
org.apache.calcite.prepare.CalcitePrepareImpl.(CalcitePrepareImpl.java:154)
at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:182)
at org.apache.calcite.tools.RelBuilder.create(RelBuilder.java:225)
at net.hydromatic.morel.foreign.Calcite.(Calcite.java:42)
at
net.hydromatic.morel.foreign.Calcite$CalciteMap.(Calcite.java:64)
at net.hydromatic.morel.foreign.Calcite.withDataSets(Calcite.java:56)
at net.hydromatic.morel.Ml.assertEvalSame(Ml.java:319)
at net.hydromatic.morel.AlgebraTest.checkEqual(AlgebraTest.java:234)
at net.hydromatic.morel.AlgebraTest.testNative(AlgebraTest.java:230)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:567)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at
com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
at
com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:230)
at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:58)
Caused by: java.lang.IllegalArgumentException: property
'org.apache.calcite.adapter.enumerable.EnumerableLimitSortRule$Config#Description'
is required and has no default value
at
org.apache.calcite.util.ImmutableBeans.lambda$makeDef$0(ImmutableBeans.java:168)
at
org.apache.calcite.util.ImmutableBeans$BeanImpl.invoke(ImmutableBeans.java:480)
at com.sun.proxy.$Proxy15.description(Unknown Source)
at org.apache.calcite.plan.RelRule.(RelRule.java:115)
at
org.apache.calcite.adapter.enumerable.EnumerableLimitSortRule.(EnumerableLimitSortRule.java:35)
at
org.apache.calcite.adapter.enumerable.EnumerableLimitSortRule$Config.toRule(EnumerableLimitSortRule.java:57)
at
java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:719)
at
org.apache.calcite.util.ImmutableBeans.lambda$makeDef$3(ImmutableBeans.java:299)
at
org.apache.calcite.util.ImmutableBeans$BeanImpl.invoke(ImmutableBeans.java:480)
at com.sun.proxy.$Proxy15.toRule(Unknown Source)
at
org.apache.calcite.adapter.enumerable.EnumerableRules.(EnumerableRules.java:90)
... 31 more
{noformat}
The property {{EnumerableLimitSortRule.Config.Description}} is declared as
nullable (because it has the annotation
{{org.checkerframework.checker.nullness.qual.Nullable}}) but that annotation
does not come through when the class is used in the Morel project, and
therefore the property becomes non-nullable.
This bug was introduced by the [~vlsi]'s change CALCITE-4284.
We should not require client projects to use checkerframework, but if they
don't, they will get this error when they reference nullable properties.
I think that nullable properties should be indicated by an annotation that we
control - go back to
Re: Re-write right outer join as left outer join
Do you mean the join reordering ? You can take a look at LoptOptimizeJoinRule and JoinCommuteRule. Abhishek Agarwal 于2021年3月3日周三 下午5:12写道： > Hello everyone, > > is there an existing rule that can re-write a right outer join as a left > outer join by swapping tables? >
Re-write right outer join as left outer join
Hello everyone, is there an existing rule that can re-write a right outer join as a left outer join by swapping tables?
