Re: Top-down pass of an optimized plan

[Stamatis Zampetakis]

Hi Makis,

+1 to Haisheng said!

Apart from HepPlanner, another class that is used often for pre/post
processing a plan is RelShuttle [1].

Best,
Stamatis

[1]
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/rel/RelShuttle.java

On Tue, Nov 12, 2019 at 11:22 PM Serafeim (Makis) Papadias <
makisnt...@gmail.com> wrote:

> Thanks a lot Haisheng.
>
> I will try it and come back to you if I have some further questions. :)
>
> Best,
> Makis
>
> > On 12 Nov 2019, at 11:04 PM, Haisheng Yuan 
> wrote:
> >
> > Hi Makis,
> >
> > You can use HepPlanner and your customized transoformation rule to
> rewrite the optimized plan. Top-down or Bottom-up can also be specified.
> > Many systems have this kind of post processing phase.
> >
> > - Haisheng
> >
> > --
> > 发件人：Serafeim (Makis) Papadias
> > 日 期：2019年11月13日 05:37:37
> > 收件人：
> > 主 题：Top-down pass of an optimized plan
> >
> > Dear all,
> >
> > I am performing a Volcano optimisation pass first and I receive the root
> of the optimised plan from the standard Program in [1]. Afterwards, I want
> to perform a top-down pass to the optimised plan and change it a bit by
> placing certain operators at specific points.
> >
> > Do you have any ideas on how to do this properly in Calcite?
> >
> > Thanks in advance.
> >
> > Best,
> > Makis
> >
> >
> > [1] —
> https://calcite.apache.org/apidocs/org/apache/calcite/tools/Programs.html
>  >
>
>

[jira] [Created] (CALCITE-3497) Calcite TimestampString should provide getNanosInSecond interface

[Zhenghua Gao (Jira)]

Zhenghua Gao created CALCITE-3497:
-

 Summary: Calcite TimestampString should provide getNanosInSecond 
interface
 Key: CALCITE-3497
 URL: https://issues.apache.org/jira/browse/CALCITE-3497
 Project: Calcite
  Issue Type: Bug
  Components: core
Reporter: Zhenghua Gao






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Re: Re: Optimizer: All the inputs have relevant nodes, however the cost is still infinite.

[Rui Wang]

I found EnumerableTableFunctionScanRule and EnumerableTableFunctionScan are
not tested in unit tests in core (checked by code search in core/test). The
right approach seems to me now is add more tests and then gradually see if
I can narrow down the problem.

Are there recommended places that I should add more tests?


-Rui

On Fri, Nov 8, 2019 at 8:38 PM Rui Wang  wrote:

> Xiening - You can reproduce it by running CoreQuidemTest in [1].
>
>
> Haisheng - those enumerable rules all exist. My change in [1] shouldn't
> require implementing new enumerable rules.
>
>
>
> [1]: https://github.com/amaliujia/calcite/tree/rw-add_TVF_TUMBLE
>
> On Fri, Nov 8, 2019 at 12:04 PM Haisheng Yuan 
> wrote:
>
>> Yes, looks like EnumerableTableFunctionScan doesn't override
>> computeSelfCost.
>>
>> - Haisheng
>>
>> --
>> 发件人：Haisheng Yuan
>> 日 期：2019年11月09日 04:01:19
>> 收件人：Apache Calcite dev list
>> 主 题：Re: Re: Optimizer: All the inputs have relevant nodes, however the
>> cost is still infinite.
>>
>> It is not surprising to get an infinitive cost, since the operators in
>> the plan are logical operators, which need to be converted to physical
>> operators to be costed. Did you try to add some implementation rules to the
>> rule set, e.g. EnumerableProjectRule, EnumerableTableFunctionScanRule, etc..
>>
>> - Haisheng
>>
>> --
>> 发件人：Xiening Dai
>> 日 期：2019年11月09日 03:35:03
>> 收件人：
>> 主 题：Re: Optimizer: All the inputs have relevant nodes, however the cost
>> is still infinite.
>>
>> Are you able to provide a test case to repro on current code base?
>>
>> From the error message, the given subset’s best RelNode is not available,
>> aka cost is infinite. There could be multiple cases leading to it.
>>
>>
>> > On Nov 8, 2019, at 11:28 AM, Rui Wang  wrote:
>> >
>> > My current guess is it could because of `computeSelfCost` of
>> > `EnumerableTableFunctionScan`.
>> >
>> >
>> > -Rui
>> >
>> > On Fri, Nov 8, 2019 at 10:04 AM Rui Wang  wrote:
>> >
>> >> Hello community:
>> >>
>> >> I am stuck an optimizer error while working on Calcite-3272. I am less
>> >> familiar with Calcite's optimizer and struggled to debug the following
>> >> error message. Is there some hints on what direction I should go? I can
>> >> also provide more information if that is helpful.
>> >>
>> >>
>> >>
>> >>> java.sql.SQLException: Error while executing SQL "select * FROM
>> >> TABLE(TUMBLE(TABLE ORDERS, 'ROWTIME', INTERVAL '1' MINUTE))": There
>> are not
>> >> enough rules to produce a node with desired properties:
>> >> convention=ENUMERABLE, sort=[]. All the inputs have relevant nodes,
>> however
>> >> the cost is still infinite.
>> >>> Root: rel#26:Subset#2.ENUMERABLE.[]
>> >>> Original rel:
>> >>> LogicalProject(ROWTIME=[$0], ID=[$1], PRODUCT=[$2], UNITS=[$3],
>> >> wstart=[$4], wend=[$5]): rowcount = 100.0, cumulative cost =
>> >> {1.7976931348623157E308 rows, 1.7976931348623157E308 cpu,
>> >> 1.7976931348623157E308 io}, id = 12
>> >>>  LogicalTableFunctionScan(invocation=[TUMBLE($3, 'ROWTIME',
>> >> 6:INTERVAL MINUTE)], rowType=[RecordType(TIMESTAMP(0) ROWTIME,
>> INTEGER
>> >> ID, VARCHAR(10) PRODUCT, INTEGER UNITS, TIMESTAMP(0) wstart,
>> TIMESTAMP(0)
>> >> wend)]): rowcount = 100.0, cumulative cost = {1.7976931348623157E308
>> rows,
>> >> 1.7976931348623157E308 cpu, 1.7976931348623157E308 io}, id = 10
>> >>>LogicalProject(ROWTIME=[$0], ID=[$1], PRODUCT=[$2], UNITS=[$3]):
>> >> rowcount = 100.0, cumulative cost = {200.0 rows, 501.0 cpu, 0.0 io},
>> id = 8
>> >>>  LogicalTableScan(table=[[ORINOCO, ORDERS]]): rowcount = 100.0,
>> >> cumulative cost = {100.0 rows, 101.0 cpu, 0.0 io}, id = 2
>> >>
>> >>
>> >>
>> >> -Rui
>> >>
>>
>>

Re: Top-down pass of an optimized plan

[Serafeim (Makis) Papadias]

Thanks a lot Haisheng.

I will try it and come back to you if I have some further questions. :)

Best,
Makis

> On 12 Nov 2019, at 11:04 PM, Haisheng Yuan  wrote:
> 
> Hi Makis,
> 
> You can use HepPlanner and your customized transoformation rule to rewrite 
> the optimized plan. Top-down or Bottom-up can also be specified.
> Many systems have this kind of post processing phase. 
> 
> - Haisheng
> 
> --
> 发件人：Serafeim (Makis) Papadias
> 日　期：2019年11月13日 05:37:37
> 收件人：
> 主　题：Top-down pass of an optimized plan
> 
> Dear all,
> 
> I am performing a Volcano optimisation pass first and I receive the root of 
> the optimised plan from the standard Program in [1]. Afterwards, I want to 
> perform a top-down pass to the optimised plan and change it a bit by placing 
> certain operators at specific points.
> 
> Do you have any ideas on how to do this properly in Calcite?
> 
> Thanks in advance.
> 
> Best,
> Makis
> 
> 
> [1] — 
> https://calcite.apache.org/apidocs/org/apache/calcite/tools/Programs.html 
> 

Re: Top-down pass of an optimized plan

[Haisheng Yuan]

Hi Makis,

You can use HepPlanner and your customized transoformation rule to rewrite the 
optimized plan. Top-down or Bottom-up can also be specified.
Many systems have this kind of post processing phase. 

- Haisheng

--
发件人：Serafeim (Makis) Papadias
日　期：2019年11月13日 05:37:37
收件人：
主　题：Top-down pass of an optimized plan

Dear all,

I am performing a Volcano optimisation pass first and I receive the root of the 
optimised plan from the standard Program in [1]. Afterwards, I want to perform 
a top-down pass to the optimised plan and change it a bit by placing certain 
operators at specific points.

Do you have any ideas on how to do this properly in Calcite?

Thanks in advance.

Best,
Makis


[1] — https://calcite.apache.org/apidocs/org/apache/calcite/tools/Programs.html 

Top-down pass of an optimized plan

[Serafeim (Makis) Papadias]

Dear all,

I am performing a Volcano optimisation pass first and I receive the root of the 
optimised plan from the standard Program in [1]. Afterwards, I want to perform 
a top-down pass to the optimised plan and change it a bit by placing certain 
operators at specific points.

Do you have any ideas on how to do this properly in Calcite?

Thanks in advance.

Best,
Makis


[1] — https://calcite.apache.org/apidocs/org/apache/calcite/tools/Programs.html 

Re: Contribution permission request

[Francis Chuang]

Whoops, it looks like I clicked "reply" instead of "reply list" when I 
replied to the original thread.


On 13/11/2019 3:49 am, Josh Elser wrote:

Looks like someone has already added you. Happy contributing.

On 11/12/19 3:52 AM, Amir Gajst wrote:

Hi Calcite community,

My name is Amir Gajst and I'm a developer at Sisense, a BI company.

I’d like to contribute to the community, could you please give me the
necessary permission?
My JIRA username is agajst.


Best，
Amir

Re: Contribution permission request

[Amir Gajst]

Thanks

On Tue, Nov 12, 2019 at 6:49 PM Josh Elser  wrote:

> Looks like someone has already added you. Happy contributing.
>
> On 11/12/19 3:52 AM, Amir Gajst wrote:
> > Hi Calcite community,
> >
> > My name is Amir Gajst and I'm a developer at Sisense, a BI company.
> >
> > I’d like to contribute to the community, could you please give me the
> > necessary permission?
> > My JIRA username is agajst.
> >
> >
> > Best，
> > Amir
> >
> >
>


-- 



[image: Sisense]

*Amir Gajst*
Software Engineer
+972-54-4766239
amir.gajst
[image: Instagram]    [image: LinkedIn]
   [image: Facebook]
   [image: Twitter]

Re: Avatica + GitHub Actions + Windows = AvaticaSpnegoTest - Principal: HTTP/stratum.antpool....@example.com is not known

[Josh Elser]

It's strange that you see this on Windows. Everything should be using 
localhost in the tests. Maybe that means it's something specific to 
Windows? I don't know enough to say if that's a reasonable guess or not.


The principal looked up stems from the hostname you issue a request to.

e.g. if you try to SPNEGO with http://foobar.google.com, the client 
would look up HTTP/foobar.google.com in the KDC.


On 11/11/19 3:54 PM, Vladimir Sitnikov wrote:

localhost but the principal being looked up is FQDN.


Where that FQDN is taken from?

Vladimir

Re: Contribution permission request

[Josh Elser]

Looks like someone has already added you. Happy contributing.

On 11/12/19 3:52 AM, Amir Gajst wrote:

Hi Calcite community,

My name is Amir Gajst and I'm a developer at Sisense, a BI company.

I’d like to contribute to the community, could you please give me the
necessary permission?
My JIRA username is agajst.


Best，
Amir

[jira] [Created] (CALCITE-3496) Hive dialect support with cube and with rollup

[xzh_dz (Jira)]

xzh_dz created CALCITE-3496:
---

 Summary: Hive dialect support with cube and  with rollup
 Key: CALCITE-3496
 URL: https://issues.apache.org/jira/browse/CALCITE-3496
 Project: Calcite
  Issue Type: Wish
Reporter: xzh_dz


Hive dialect support with cube and with rollup



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (CALCITE-3495) RelDecorrelator generate plan with different semantics when handle Aggregate

[Jin Xing (Jira)]

Jin Xing created CALCITE-3495:
-

 Summary: RelDecorrelator generate plan with different semantics 
when handle Aggregate
 Key: CALCITE-3495
 URL: https://issues.apache.org/jira/browse/CALCITE-3495
 Project: Calcite
  Issue Type: Bug
Reporter: Jin Xing






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Contribution permission request

[Amir Gajst]

Hi Calcite community,

My name is Amir Gajst and I'm a developer at Sisense, a BI company.

I’d like to contribute to the community, could you please give me the
necessary permission?
My JIRA username is agajst.


Best，
Amir


-- 



[image: Sisense]

*Amir Gajst*
Software Engineer
+972-54-4766239
amir.gajst
[image: Instagram]    [image: LinkedIn]
   [image: Facebook]
   [image: Twitter]

[jira] [Created] (CALCITE-3494) Support decimal type aggregate in Interpreter

[Wang Yanlin (Jira)]

Wang Yanlin created CALCITE-3494:


 Summary: Support decimal type aggregate in Interpreter
 Key: CALCITE-3494
 URL: https://issues.apache.org/jira/browse/CALCITE-3494
 Project: Calcite
  Issue Type: Improvement
Reporter: Wang Yanlin


Using decimal type aggregate in Interpreter cause exception.

Add this case in *InterpreterTest* to reproduce.
{code:java}
@Test public void testDecimalType() throws Exception {
final String sql = "select x, min(y), max(y), sum(y)\n"
+ "from (values ('a', 1.2), ('a', 2.3), ('a', 15)) as t(x, y)\n"
+ "group by x";
sql(sql).returnsRows("[a, 1.2, 15.0, 18.5]");
  }
{code}

The stack trace is like this

{noformat}
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException

at 
org.apache.calcite.interpreter.AggregateNode$UdaAccumulator.send(AggregateNode.java:662)
at 
org.apache.calcite.interpreter.AggregateNode$AccumulatorList.send(AggregateNode.java:398)
at 
org.apache.calcite.interpreter.AggregateNode$Grouping.send(AggregateNode.java:366)
at 
org.apache.calcite.interpreter.AggregateNode.run(AggregateNode.java:96)
at 
org.apache.calcite.interpreter.Interpreter.start(Interpreter.java:130)
at 
org.apache.calcite.interpreter.Interpreter.enumerator(Interpreter.java:107)
at 
org.apache.calcite.linq4j.AbstractEnumerable.iterator(AbstractEnumerable.java:33)
at 
org.apache.calcite.test.InterpreterTest.assertRows(InterpreterTest.java:173)
at 
org.apache.calcite.test.InterpreterTest.access$100(InterpreterTest.java:51)
at 
org.apache.calcite.test.InterpreterTest$Sql.returnsRows(InterpreterTest.java:122)
at 
org.apache.calcite.test.InterpreterTest$Sql.returnsRows(InterpreterTest.java:105)
at 
org.apache.calcite.test.InterpreterTest.testDecimalType(InterpreterTest.java:441)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at 
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at 
com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
at 
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
at 
com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.calcite.interpreter.AggregateNode$UdaAccumulator.send(AggregateNode.java:660)
... 35 more
Caused by: java.lang.ClassCastException: java.math.BigDecimal cannot be cast to 
java.lang.Long
at 
org.apache.calcite.interpreter.AggregateNode$NumericComparison.add(AggregateNode.java:494)
... 40 more

{noformat}




--
This message was sent by Atlassian Jira
(v8.3.4#803005)