Re: TUMBLE/HOP/SESSION_START/END do not resolve time field correctly

2017-04-25 Thread Timo Walther 
I created CALCITE-1761 for this issue. If you wanna take a look at the 
exception, this is my branch:


https://github.com/twalthr/flink/tree/FLINK-5884
The following test fails right now: 
org.apache.flink.table.api.scala.stream.sql.WindowAggregateTest#testTumbleFunction


Once we have a fix for that we can copy the class to Flink until the 
next Calcite release. We did that with other issues in the past, too.



Am 25/04/17 um 20:12 schrieb Timo Walther:
Thanks for your quick response. Flink does not use the monotonicity 
property yet and we are are also not using the STREAM keyword. Could 
this be a problem?



Am 25/04/17 um 19:39 schrieb Julian Hyde:

I've added a test case (and the test missed in CALCITE-1615) in
https://github.com/julianhyde/calcite/tree/-hop. I cannot
reproduce your problem. Please still log a jira case.

On Tue, Apr 25, 2017 at 10:13 AM, Julian Hyde  wrote:

I just noticed that in
https://issues.apache.org/jira/browse/CALCITE-1615 tests were added to
SqlToRelConverterTest.xml but not to SqlToRelConverterTest.java. We've
been running without tests.

On Tue, Apr 25, 2017 at 10:01 AM, Julian Hyde  wrote:

Can you log a bug please? I will help out if I can.

When this is fixed, I presume you will need a Calcite release at the
appropriate time so that you can release Flink. Can you start a
separate email thread when you know that timing?

On Tue, Apr 25, 2017 at 7:13 AM, Timo Walther  
wrote:

Hi all,


I'm working on integrating START and END for TUMBLE/HOP/SESSION in 
Flink SQL
with logical time indicator columns (e.g. rowtime, proctime). It 
seems there
is a bug in the resolution logic of SqlToRelConverter. Since our 
feature
freeze is next week and this feature should be part of Flink 1.3, 
it would

be great if you can help me with at least a hint for a quick fix.


The problem is as follows:


Input: MyTable(INTEGER a, VARCHAR b, BIGINT c, TIMESTAMP proctime, 
TIMESTAMP

rowtime)

SQL: SELECT COUNT(*), TUMBLE_START(rowtime, INTERVAL '15' MINUTE),
TUMBLE_END(rowtime, INTERVAL '15' MINUTE) FROM MyTable GROUP BY
TUMBLE(rowtime, INTERVAL '15' MINUTE)

Exception:

java.lang.RuntimeException: while converting
TUMBLE_START(`MyTable`.`rowtime`, INTERVAL '15' MINUTE)

 at
org.apache.calcite.sql2rel.ReflectiveConvertletTable$2.convertCall(ReflectiveConvertletTable.java:134) 


 at
org.apache.calcite.sql2rel.SqlNodeToRexConverterImpl.convertCall(SqlNodeToRexConverterImpl.java:61) 


 at
org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:4415) 


 at
org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:3783) 


 at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:137)
 at
org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.convertExpression(SqlToRelConverter.java:4317) 


 at
org.apache.calcite.sql2rel.SqlToRelConverter.createAggImpl(SqlToRelConverter.java:2723) 


 at
org.apache.calcite.sql2rel.SqlToRelConverter.convertAgg(SqlToRelConverter.java:2541) 


 at
org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectImpl(SqlToRelConverter.java:654) 


 at
org.apache.calcite.sql2rel.SqlToRelConverter.convertSelect(SqlToRelConverter.java:616) 


 at
org.apache.calcite.sql2rel.SqlToRelConverter.convertQueryRecursive(SqlToRelConverter.java:2951) 


 at
org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:552) 


 .
Caused by: java.lang.reflect.InvocationTargetException
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 


 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 


 at java.lang.reflect.Method.invoke(Method.java:498)
 at
org.apache.calcite.sql2rel.ReflectiveConvertletTable$2.convertCall(ReflectiveConvertletTable.java:131) 


 ... 42 more
Caused by: java.lang.AssertionError
 at
org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.getRootField(SqlToRelConverter.java:4132) 


 at
org.apache.calcite.sql2rel.SqlToRelConverter.adjustInputRef(SqlToRelConverter.java:3446) 


 at
org.apache.calcite.sql2rel.SqlToRelConverter.convertIdentifier(SqlToRelConverter.java:3421) 


 at
org.apache.calcite.sql2rel.SqlToRelConverter.access$1800(SqlToRelConverter.java:207) 


 at
org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:4424) 


 at
org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java 




The tests in Calcite only cover the case where rowtime is at the 
beginning
of a row. Once rowtime is somewhere else, the indices are messed 
up. I tried

to debug it, but I'm stuck at SqlToRelConverter#convertIdentifier().


Any help is very welcome.


Regards,

Timo

Re: TUMBLE/HOP/SESSION_START/END do not resolve time field correctly

2017-04-25 Thread Timo Walther 
Thanks for your quick response. Flink does not use the monotonicity 
property yet and we are are also not using the STREAM keyword. Could 
this be a problem?



Am 25/04/17 um 19:39 schrieb Julian Hyde:

I've added a test case (and the test missed in CALCITE-1615) in
https://github.com/julianhyde/calcite/tree/-hop. I cannot
reproduce your problem. Please still log a jira case.

On Tue, Apr 25, 2017 at 10:13 AM, Julian Hyde  wrote:

I just noticed that in
https://issues.apache.org/jira/browse/CALCITE-1615 tests were added to
SqlToRelConverterTest.xml but not to SqlToRelConverterTest.java. We've
been running without tests.

On Tue, Apr 25, 2017 at 10:01 AM, Julian Hyde  wrote:

Can you log a bug please? I will help out if I can.

When this is fixed, I presume you will need a Calcite release at the
appropriate time so that you can release Flink. Can you start a
separate email thread when you know that timing?

On Tue, Apr 25, 2017 at 7:13 AM, Timo Walther  wrote:

Hi all,


I'm working on integrating START and END for TUMBLE/HOP/SESSION in Flink SQL
with logical time indicator columns (e.g. rowtime, proctime). It seems there
is a bug in the resolution logic of SqlToRelConverter. Since our feature
freeze is next week and this feature should be part of Flink 1.3, it would
be great if you can help me with at least a hint for a quick fix.


The problem is as follows:


Input: MyTable(INTEGER a, VARCHAR b, BIGINT c, TIMESTAMP proctime, TIMESTAMP
rowtime)

SQL: SELECT COUNT(*), TUMBLE_START(rowtime, INTERVAL '15' MINUTE),
TUMBLE_END(rowtime, INTERVAL '15' MINUTE) FROM MyTable GROUP BY
TUMBLE(rowtime, INTERVAL '15' MINUTE)

Exception:

java.lang.RuntimeException: while converting
TUMBLE_START(`MyTable`.`rowtime`, INTERVAL '15' MINUTE)

 at
org.apache.calcite.sql2rel.ReflectiveConvertletTable$2.convertCall(ReflectiveConvertletTable.java:134)
 at
org.apache.calcite.sql2rel.SqlNodeToRexConverterImpl.convertCall(SqlNodeToRexConverterImpl.java:61)
 at
org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:4415)
 at
org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:3783)
 at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:137)
 at
org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.convertExpression(SqlToRelConverter.java:4317)
 at
org.apache.calcite.sql2rel.SqlToRelConverter.createAggImpl(SqlToRelConverter.java:2723)
 at
org.apache.calcite.sql2rel.SqlToRelConverter.convertAgg(SqlToRelConverter.java:2541)
 at
org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectImpl(SqlToRelConverter.java:654)
 at
org.apache.calcite.sql2rel.SqlToRelConverter.convertSelect(SqlToRelConverter.java:616)
 at
org.apache.calcite.sql2rel.SqlToRelConverter.convertQueryRecursive(SqlToRelConverter.java:2951)
 at
org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:552)
 .
Caused by: java.lang.reflect.InvocationTargetException
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at
org.apache.calcite.sql2rel.ReflectiveConvertletTable$2.convertCall(ReflectiveConvertletTable.java:131)
 ... 42 more
Caused by: java.lang.AssertionError
 at
org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.getRootField(SqlToRelConverter.java:4132)
 at
org.apache.calcite.sql2rel.SqlToRelConverter.adjustInputRef(SqlToRelConverter.java:3446)
 at
org.apache.calcite.sql2rel.SqlToRelConverter.convertIdentifier(SqlToRelConverter.java:3421)
 at
org.apache.calcite.sql2rel.SqlToRelConverter.access$1800(SqlToRelConverter.java:207)
 at
org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:4424)
 at
org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java


The tests in Calcite only cover the case where rowtime is at the beginning
of a row. Once rowtime is somewhere else, the indices are messed up. I tried
to debug it, but I'm stuck at SqlToRelConverter#convertIdentifier().


Any help is very welcome.


Regards,

Timo

Re: TUMBLE/HOP/SESSION_START/END do not resolve time field correctly

2017-04-25 Thread Julian Hyde 
Can you log a bug please? I will help out if I can.

When this is fixed, I presume you will need a Calcite release at the
appropriate time so that you can release Flink. Can you start a
separate email thread when you know that timing?

On Tue, Apr 25, 2017 at 7:13 AM, Timo Walther  wrote:
> Hi all,
>
>
> I'm working on integrating START and END for TUMBLE/HOP/SESSION in Flink SQL
> with logical time indicator columns (e.g. rowtime, proctime). It seems there
> is a bug in the resolution logic of SqlToRelConverter. Since our feature
> freeze is next week and this feature should be part of Flink 1.3, it would
> be great if you can help me with at least a hint for a quick fix.
>
>
> The problem is as follows:
>
>
> Input: MyTable(INTEGER a, VARCHAR b, BIGINT c, TIMESTAMP proctime, TIMESTAMP
> rowtime)
>
> SQL: SELECT COUNT(*), TUMBLE_START(rowtime, INTERVAL '15' MINUTE),
> TUMBLE_END(rowtime, INTERVAL '15' MINUTE) FROM MyTable GROUP BY
> TUMBLE(rowtime, INTERVAL '15' MINUTE)
>
> Exception:
>
> java.lang.RuntimeException: while converting
> TUMBLE_START(`MyTable`.`rowtime`, INTERVAL '15' MINUTE)
>
> at
> org.apache.calcite.sql2rel.ReflectiveConvertletTable$2.convertCall(ReflectiveConvertletTable.java:134)
> at
> org.apache.calcite.sql2rel.SqlNodeToRexConverterImpl.convertCall(SqlNodeToRexConverterImpl.java:61)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:4415)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:3783)
> at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:137)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.convertExpression(SqlToRelConverter.java:4317)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.createAggImpl(SqlToRelConverter.java:2723)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertAgg(SqlToRelConverter.java:2541)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectImpl(SqlToRelConverter.java:654)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertSelect(SqlToRelConverter.java:616)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertQueryRecursive(SqlToRelConverter.java:2951)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:552)
> .
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.calcite.sql2rel.ReflectiveConvertletTable$2.convertCall(ReflectiveConvertletTable.java:131)
> ... 42 more
> Caused by: java.lang.AssertionError
> at
> org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.getRootField(SqlToRelConverter.java:4132)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.adjustInputRef(SqlToRelConverter.java:3446)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertIdentifier(SqlToRelConverter.java:3421)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.access$1800(SqlToRelConverter.java:207)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:4424)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java
>
>
> The tests in Calcite only cover the case where rowtime is at the beginning
> of a row. Once rowtime is somewhere else, the indices are messed up. I tried
> to debug it, but I'm stuck at SqlToRelConverter#convertIdentifier().
>
>
> Any help is very welcome.
>
>
> Regards,
>
> Timo
>