[jira] [Created] (CALCITE-2756) ForEachStatement generates invalid Java Code
Julian Feinauer created CALCITE-2756: Summary: ForEachStatement generates invalid Java Code Key: CALCITE-2756 URL: https://issues.apache.org/jira/browse/CALCITE-2756 Project: Calcite Issue Type: Bug Reporter: Julian Feinauer Assignee: Julian Feinauer Code generated by the ForEachStatement in org.apache.calcite.linq4j.tree looks like: {code:java} for (i : list) { ... } {code} I.e. the Paramter Type for the looping parameter is missing. Thus, this cannot be used for codegen because Janino fails then. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (CALCITE-2756) ForEachStatement generates invalid Java Code
[ https://issues.apache.org/jira/browse/CALCITE-2756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Feinauer closed CALCITE-2756. Resolution: Fixed I just observed that this Code does not (yet) exist in master, thus I fix it in the local branch I'm working on. > ForEachStatement generates invalid Java Code > > > Key: CALCITE-2756 > URL: https://issues.apache.org/jira/browse/CALCITE-2756 > Project: Calcite > Issue Type: Bug >Reporter: Julian Feinauer >Assignee: Julian Feinauer >Priority: Major > > Code generated by the ForEachStatement in org.apache.calcite.linq4j.tree > looks like: > {code:java} > for (i : list) { > ... > } > {code} > I.e. the Paramter Type for the looping parameter is missing. > Thus, this cannot be used for codegen because Janino fails then. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CALCITE-2966) Problem with Code Generation
[ https://issues.apache.org/jira/browse/CALCITE-2966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16807566#comment-16807566 ] Julian Feinauer commented on CALCITE-2966: -- First, thanks [~danny0405]... are we sure that this is a bug or could it be due to me doing something wrong with the API? Can I supply you with more informations about my code? > Problem with Code Generation > > > Key: CALCITE-2966 > URL: https://issues.apache.org/jira/browse/CALCITE-2966 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.20.0 >Reporter: Danny Chan >Assignee: Danny Chan >Priority: Major > Fix For: 1.20.0 > > > From the mailing list: > Hi all, > I have some problems with the code generation from Linq4j which I'm unable to > resolve myself. > Basically, I want to translate a condition from Rex to a Linq4j expression to > use it in generated code. > In my example the Condition is from Match Recognize and in SQL is: > `up."commission" > prev(up."commission")`. > > ``` > RexBuilder rexBuilder = new RexBuilder(implementor.getTypeFactory()); > RexProgramBuilder rexProgramBuilder = new > RexProgramBuilder(physType.getRowType(), rexBuilder); > > rexProgramBuilder.addCondition(entry.getValue()); > > final Expression condition = > RexToLixTranslator.translateCondition(rexProgramBuilder.getProgram(), > (JavaTypeFactory) getCluster().getTypeFactory(), > builder2, > inputGetter1, > implementor.allCorrelateVariables, > implementor.getConformance()); > > builder2.add(Expressions.return_(null, condition)); > ``` > > Here, the condition seems okay, it is: ">(PREV(UP.$4, 0), PREV(UP.$4, 1))", > so it should be a comparison of two variables (I rewrite the PREV with a > custom Input Getter". > But, the generated code (for Janino) is: > > ``` > Object p1 = row_.get($L4J$C$0_1); > org.apache.calcite.test.JdbcTest.Employee p0 = > (org.apache.calcite.test.JdbcTest.Employee) p1; > Object p3 = row_.get($L4J$C$1_1); > org.apache.calcite.test.JdbcTest.Employee p2 = > (org.apache.calcite.test.JdbcTest.Employee) p3; > Object p5 = row_.get($L4J$C$0_1); > org.apache.calcite.test.JdbcTest.Employee p4 = > (org.apache.calcite.test.JdbcTest.Employee) p5; > Object p7 = row_.get($L4J$C$1_1); > org.apache.calcite.test.JdbcTest.Employee p6 = > (org.apache.calcite.test.JdbcTest.Employee) p7; > return p0.commission && p2.commission && p4.commission > p6.commission; > ``` > > This confuses me a lot as I do not know where the check for p0.commission and > p2.commission comes from. > It seems that Linq4j adds them as it expects these variables to be nullable, > but I have no idea on how to avoid this. > These fields are Numeric so I always get a compilation exception. > > Can someone help me with this issue? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CALCITE-2966) Problem with Code Generation
[ https://issues.apache.org/jira/browse/CALCITE-2966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16807828#comment-16807828 ] Julian Feinauer commented on CALCITE-2966: -- Hey [~danny0405]... the full code is in my fork for this commit: https://github.com/JulianFeinauer/calcite/tree/ae8704ab44632ae4bbe51cafef2251920955b183 The bug appears when the "testMatch" Method in the JdbcTest class is run. This is the fastest and easiest way to reproduce it, is this okay for you? > Problem with Code Generation > > > Key: CALCITE-2966 > URL: https://issues.apache.org/jira/browse/CALCITE-2966 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.20.0 >Reporter: Danny Chan >Assignee: Danny Chan >Priority: Major > Fix For: 1.20.0 > > > From the mailing list: > Hi all, > I have some problems with the code generation from Linq4j which I'm unable to > resolve myself. > Basically, I want to translate a condition from Rex to a Linq4j expression to > use it in generated code. > In my example the Condition is from Match Recognize and in SQL is: > `up."commission" > prev(up."commission")`. > > ``` > RexBuilder rexBuilder = new RexBuilder(implementor.getTypeFactory()); > RexProgramBuilder rexProgramBuilder = new > RexProgramBuilder(physType.getRowType(), rexBuilder); > > rexProgramBuilder.addCondition(entry.getValue()); > > final Expression condition = > RexToLixTranslator.translateCondition(rexProgramBuilder.getProgram(), > (JavaTypeFactory) getCluster().getTypeFactory(), > builder2, > inputGetter1, > implementor.allCorrelateVariables, > implementor.getConformance()); > > builder2.add(Expressions.return_(null, condition)); > ``` > > Here, the condition seems okay, it is: ">(PREV(UP.$4, 0), PREV(UP.$4, 1))", > so it should be a comparison of two variables (I rewrite the PREV with a > custom Input Getter". > But, the generated code (for Janino) is: > > ``` > Object p1 = row_.get($L4J$C$0_1); > org.apache.calcite.test.JdbcTest.Employee p0 = > (org.apache.calcite.test.JdbcTest.Employee) p1; > Object p3 = row_.get($L4J$C$1_1); > org.apache.calcite.test.JdbcTest.Employee p2 = > (org.apache.calcite.test.JdbcTest.Employee) p3; > Object p5 = row_.get($L4J$C$0_1); > org.apache.calcite.test.JdbcTest.Employee p4 = > (org.apache.calcite.test.JdbcTest.Employee) p5; > Object p7 = row_.get($L4J$C$1_1); > org.apache.calcite.test.JdbcTest.Employee p6 = > (org.apache.calcite.test.JdbcTest.Employee) p7; > return p0.commission && p2.commission && p4.commission > p6.commission; > ``` > > This confuses me a lot as I do not know where the check for p0.commission and > p2.commission comes from. > It seems that Linq4j adds them as it expects these variables to be nullable, > but I have no idea on how to avoid this. > These fields are Numeric so I always get a compilation exception. > > Can someone help me with this issue? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CALCITE-2966) Problem with Code Generation
[ https://issues.apache.org/jira/browse/CALCITE-2966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16808411#comment-16808411 ] Julian Feinauer commented on CALCITE-2966: -- Hi [~danny0405]... this is the last commit for the branch where multiple people have been working on for a long time now (over 1 year). So its not the commit itself, that is interesting but the state after the commit with all the ~200 commits in the branch. If it would help you, I can do a several branch where I squash all of them together, but in this branch I want to keep the commits as the work was done by several people and I think it would not be right to "mask" that by a single squash commit from myself. So either you checkout the commit in this branch or I prepare another squashed branch for you, what you like better? > Problem with Code Generation > > > Key: CALCITE-2966 > URL: https://issues.apache.org/jira/browse/CALCITE-2966 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.20.0 >Reporter: Danny Chan >Assignee: Danny Chan >Priority: Major > Fix For: 1.20.0 > > > From the mailing list: > Hi all, > I have some problems with the code generation from Linq4j which I'm unable to > resolve myself. > Basically, I want to translate a condition from Rex to a Linq4j expression to > use it in generated code. > In my example the Condition is from Match Recognize and in SQL is: > `up."commission" > prev(up."commission")`. > > ``` > RexBuilder rexBuilder = new RexBuilder(implementor.getTypeFactory()); > RexProgramBuilder rexProgramBuilder = new > RexProgramBuilder(physType.getRowType(), rexBuilder); > > rexProgramBuilder.addCondition(entry.getValue()); > > final Expression condition = > RexToLixTranslator.translateCondition(rexProgramBuilder.getProgram(), > (JavaTypeFactory) getCluster().getTypeFactory(), > builder2, > inputGetter1, > implementor.allCorrelateVariables, > implementor.getConformance()); > > builder2.add(Expressions.return_(null, condition)); > ``` > > Here, the condition seems okay, it is: ">(PREV(UP.$4, 0), PREV(UP.$4, 1))", > so it should be a comparison of two variables (I rewrite the PREV with a > custom Input Getter". > But, the generated code (for Janino) is: > > ``` > Object p1 = row_.get($L4J$C$0_1); > org.apache.calcite.test.JdbcTest.Employee p0 = > (org.apache.calcite.test.JdbcTest.Employee) p1; > Object p3 = row_.get($L4J$C$1_1); > org.apache.calcite.test.JdbcTest.Employee p2 = > (org.apache.calcite.test.JdbcTest.Employee) p3; > Object p5 = row_.get($L4J$C$0_1); > org.apache.calcite.test.JdbcTest.Employee p4 = > (org.apache.calcite.test.JdbcTest.Employee) p5; > Object p7 = row_.get($L4J$C$1_1); > org.apache.calcite.test.JdbcTest.Employee p6 = > (org.apache.calcite.test.JdbcTest.Employee) p7; > return p0.commission && p2.commission && p4.commission > p6.commission; > ``` > > This confuses me a lot as I do not know where the check for p0.commission and > p2.commission comes from. > It seems that Linq4j adds them as it expects these variables to be nullable, > but I have no idea on how to avoid this. > These fields are Numeric so I always get a compilation exception. > > Can someone help me with this issue? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CALCITE-2966) Problem with Code Generation
[ https://issues.apache.org/jira/browse/CALCITE-2966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16814400#comment-16814400 ] Julian Feinauer commented on CALCITE-2966: -- Hey [~danny0405] it seems that there was indeed a Bug in Calcite in the RexImpTable.implementNullSemantics, there was Lines 1029ff {code:java} case FALSE: // v0 != null && v1 != null && f(v0, v1) for (Ord operand : Ord.zip(call.getOperands())) { if (translator.isNullable(operand.e)) { list.add( translator.translate( operand.e, NullAs.IS_NOT_NULL)); translator = translator.setNullable(operand.e, false); } } list.add(implementCall(translator, call, implementor, nullAs)); return Expressions.foldAnd(list); {code} which should be {code:java} case FALSE: // v0 != null && v1 != null && f(v0, v1) for (Ord operand : Ord.zip(call.getOperands())) { if (translator.isNullable(operand.e)) { list.add(Expressions.notEqual(translator.translate( operand.e, NullAs.IS_NOT_NULL), Expressions.constant(null))); translator = translator.setNullable(operand.e, false); } } list.add(implementCall(translator, call, implementor, nullAs)); return Expressions.foldAnd(list); {code} I fixed this in my branch and would submit this together with the PR I'm preparing for the match recognize. But it would be cool if you could also check if this is the correct solution. > Problem with Code Generation > > > Key: CALCITE-2966 > URL: https://issues.apache.org/jira/browse/CALCITE-2966 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.20.0 >Reporter: Danny Chan >Assignee: Danny Chan >Priority: Major > Fix For: 1.20.0 > > > From the mailing list: > Hi all, > I have some problems with the code generation from Linq4j which I'm unable to > resolve myself. > Basically, I want to translate a condition from Rex to a Linq4j expression to > use it in generated code. > In my example the Condition is from Match Recognize and in SQL is: > `up."commission" > prev(up."commission")`. > > ``` > RexBuilder rexBuilder = new RexBuilder(implementor.getTypeFactory()); > RexProgramBuilder rexProgramBuilder = new > RexProgramBuilder(physType.getRowType(), rexBuilder); > > rexProgramBuilder.addCondition(entry.getValue()); > > final Expression condition = > RexToLixTranslator.translateCondition(rexProgramBuilder.getProgram(), > (JavaTypeFactory) getCluster().getTypeFactory(), > builder2, > inputGetter1, > implementor.allCorrelateVariables, > implementor.getConformance()); > > builder2.add(Expressions.return_(null, condition)); > ``` > > Here, the condition seems okay, it is: ">(PREV(UP.$4, 0), PREV(UP.$4, 1))", > so it should be a comparison of two variables (I rewrite the PREV with a > custom Input Getter". > But, the generated code (for Janino) is: > > ``` > Object p1 = row_.get($L4J$C$0_1); > org.apache.calcite.test.JdbcTest.Employee p0 = > (org.apache.calcite.test.JdbcTest.Employee) p1; > Object p3 = row_.get($L4J$C$1_1); > org.apache.calcite.test.JdbcTest.Employee p2 = > (org.apache.calcite.test.JdbcTest.Employee) p3; > Object p5 = row_.get($L4J$C$0_1); > org.apache.calcite.test.JdbcTest.Employee p4 = > (org.apache.calcite.test.JdbcTest.Employee) p5; > Object p7 = row_.get($L4J$C$1_1); > org.apache.calcite.test.JdbcTest.Employee p6 = > (org.apache.calcite.test.JdbcTest.Employee) p7; > return p0.commission && p2.commission && p4.commission > p6.commission; > ``` > > This confuses me a lot as I do not know where the check for p0.commission and > p2.commission comes from. > It seems that Linq4j adds them as it expects these variables to be nullable, > but I have no idea on how to avoid this. > These fields are Numeric so I always get a compilation exception. > > Can someone help me with this issue? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (CALCITE-2966) Problem with Code Generation
[ https://issues.apache.org/jira/browse/CALCITE-2966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Feinauer reassigned CALCITE-2966: Assignee: Julian Feinauer (was: Danny Chan) > Problem with Code Generation > > > Key: CALCITE-2966 > URL: https://issues.apache.org/jira/browse/CALCITE-2966 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.20.0 >Reporter: Danny Chan >Assignee: Julian Feinauer >Priority: Major > Fix For: 1.20.0 > > > From the mailing list: > Hi all, > I have some problems with the code generation from Linq4j which I'm unable to > resolve myself. > Basically, I want to translate a condition from Rex to a Linq4j expression to > use it in generated code. > In my example the Condition is from Match Recognize and in SQL is: > `up."commission" > prev(up."commission")`. > > ``` > RexBuilder rexBuilder = new RexBuilder(implementor.getTypeFactory()); > RexProgramBuilder rexProgramBuilder = new > RexProgramBuilder(physType.getRowType(), rexBuilder); > > rexProgramBuilder.addCondition(entry.getValue()); > > final Expression condition = > RexToLixTranslator.translateCondition(rexProgramBuilder.getProgram(), > (JavaTypeFactory) getCluster().getTypeFactory(), > builder2, > inputGetter1, > implementor.allCorrelateVariables, > implementor.getConformance()); > > builder2.add(Expressions.return_(null, condition)); > ``` > > Here, the condition seems okay, it is: ">(PREV(UP.$4, 0), PREV(UP.$4, 1))", > so it should be a comparison of two variables (I rewrite the PREV with a > custom Input Getter". > But, the generated code (for Janino) is: > > ``` > Object p1 = row_.get($L4J$C$0_1); > org.apache.calcite.test.JdbcTest.Employee p0 = > (org.apache.calcite.test.JdbcTest.Employee) p1; > Object p3 = row_.get($L4J$C$1_1); > org.apache.calcite.test.JdbcTest.Employee p2 = > (org.apache.calcite.test.JdbcTest.Employee) p3; > Object p5 = row_.get($L4J$C$0_1); > org.apache.calcite.test.JdbcTest.Employee p4 = > (org.apache.calcite.test.JdbcTest.Employee) p5; > Object p7 = row_.get($L4J$C$1_1); > org.apache.calcite.test.JdbcTest.Employee p6 = > (org.apache.calcite.test.JdbcTest.Employee) p7; > return p0.commission && p2.commission && p4.commission > p6.commission; > ``` > > This confuses me a lot as I do not know where the check for p0.commission and > p2.commission comes from. > It seems that Linq4j adds them as it expects these variables to be nullable, > but I have no idea on how to avoid this. > These fields are Numeric so I always get a compilation exception. > > Can someone help me with this issue? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CALCITE-1935) Reference implementation for MATCH_RECOGNIZE
[ https://issues.apache.org/jira/browse/CALCITE-1935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16815490#comment-16815490 ] Julian Feinauer commented on CALCITE-1935: -- Hi all, I have a MWE now and am currentlry trying to prepare a PR with merging back master, checkstyle, ... . > Reference implementation for MATCH_RECOGNIZE > > > Key: CALCITE-1935 > URL: https://issues.apache.org/jira/browse/CALCITE-1935 > Project: Calcite > Issue Type: Bug >Reporter: Julian Hyde >Priority: Major > Labels: match > > We now have comprehensive support for parsing and validating MATCH_RECOGNIZE > queries (see CALCITE-1570 and sub-tasks) but we cannot execute them. I know > the purpose of this work is to do CEP within Flink, but a reference > implementation that works on non-streaming data would be valuable. > I propose that we add a class EnumerableMatch that can generate Java code to > evaluate MATCH_RECOGNIZE queries on Enumerable data. It does not need to be > efficient. I don't mind if it (say) buffers all the data in memory and makes > O(n ^ 3) passes over it. People can make it more efficient over time. > When we have a reference implementation, people can start playing with this > feature. And we can start building a corpus of data sets, queries, and their > expected result. The Flink implementation will be able to test against those > same queries, and should give the same results, even though Flink will be > reading streaming data. > Let's create {{match.iq}} with the following query based on > https://oracle-base.com/articles/12c/pattern-matching-in-oracle-database-12cr1: > {code} > !set outputformat mysql > !use match > SELECT * > FROM sales_history MATCH_RECOGNIZE ( > PARTITION BY product > ORDER BY tstamp > MEASURES STRT.tstamp AS start_tstamp, >LAST(UP.tstamp) AS peak_tstamp, >LAST(DOWN.tstamp) AS end_tstamp, >MATCH_NUMBER() AS mno > ONE ROW PER MATCH > AFTER MATCH SKIP TO LAST DOWN > PATTERN (STRT UP+ FLAT* DOWN+) > DEFINE >UP AS UP.units_sold > PREV(UP.units_sold), >FLAT AS FLAT.units_sold = PREV(FLAT.units_sold), >DOWN AS DOWN.units_sold < PREV(DOWN.units_sold) >) MR > ORDER BY MR.product, MR.start_tstamp; > PRODUCTSTART_TSTAM PEAK_TSTAMP END_TSTAMP MNO > -- --- --- --- -- > TWINKIES 01-OCT-2014 03-OCT-2014 06-OCT-2014 1 > TWINKIES 06-OCT-2014 08-OCT-2014 09-OCT-2014 2 > TWINKIES 09-OCT-2014 13-OCT-2014 16-OCT-2014 3 > TWINKIES 16-OCT-2014 18-OCT-2014 20-OCT-2014 4 > 4 rows selected. > !ok > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CALCITE-2966) Problem with Code Generation
[ https://issues.apache.org/jira/browse/CALCITE-2966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16816305#comment-16816305 ] Julian Feinauer commented on CALCITE-2966: -- Hi [~danny0405] all good! I tell you as soon as the PR is in good shape (have some issues with merge back from master) then you can check there. But I guess the snippets I showed above completly remove this issue. > Problem with Code Generation > > > Key: CALCITE-2966 > URL: https://issues.apache.org/jira/browse/CALCITE-2966 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.20.0 >Reporter: Danny Chan >Assignee: Julian Feinauer >Priority: Major > Fix For: 1.20.0 > > > From the mailing list: > Hi all, > I have some problems with the code generation from Linq4j which I'm unable to > resolve myself. > Basically, I want to translate a condition from Rex to a Linq4j expression to > use it in generated code. > In my example the Condition is from Match Recognize and in SQL is: > `up."commission" > prev(up."commission")`. > > ``` > RexBuilder rexBuilder = new RexBuilder(implementor.getTypeFactory()); > RexProgramBuilder rexProgramBuilder = new > RexProgramBuilder(physType.getRowType(), rexBuilder); > > rexProgramBuilder.addCondition(entry.getValue()); > > final Expression condition = > RexToLixTranslator.translateCondition(rexProgramBuilder.getProgram(), > (JavaTypeFactory) getCluster().getTypeFactory(), > builder2, > inputGetter1, > implementor.allCorrelateVariables, > implementor.getConformance()); > > builder2.add(Expressions.return_(null, condition)); > ``` > > Here, the condition seems okay, it is: ">(PREV(UP.$4, 0), PREV(UP.$4, 1))", > so it should be a comparison of two variables (I rewrite the PREV with a > custom Input Getter". > But, the generated code (for Janino) is: > > ``` > Object p1 = row_.get($L4J$C$0_1); > org.apache.calcite.test.JdbcTest.Employee p0 = > (org.apache.calcite.test.JdbcTest.Employee) p1; > Object p3 = row_.get($L4J$C$1_1); > org.apache.calcite.test.JdbcTest.Employee p2 = > (org.apache.calcite.test.JdbcTest.Employee) p3; > Object p5 = row_.get($L4J$C$0_1); > org.apache.calcite.test.JdbcTest.Employee p4 = > (org.apache.calcite.test.JdbcTest.Employee) p5; > Object p7 = row_.get($L4J$C$1_1); > org.apache.calcite.test.JdbcTest.Employee p6 = > (org.apache.calcite.test.JdbcTest.Employee) p7; > return p0.commission && p2.commission && p4.commission > p6.commission; > ``` > > This confuses me a lot as I do not know where the check for p0.commission and > p2.commission comes from. > It seems that Linq4j adds them as it expects these variables to be nullable, > but I have no idea on how to avoid this. > These fields are Numeric so I always get a compilation exception. > > Can someone help me with this issue? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CALCITE-2966) Problem with Code Generation
[ https://issues.apache.org/jira/browse/CALCITE-2966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16816643#comment-16816643 ] Julian Feinauer commented on CALCITE-2966: -- Hey [~danny0405] a short update on this... I am nearly finished with the preparation of the PR and I see some strange bahvior with the snippet I showed above. If I do the "fix" I intended to do above it makes the code generation work for the situation I have with MR but a lot of other tests fail, probably because they rely on a different behavior. So for me its pretty unclear how this should exactly work and I'm bit confused. > Problem with Code Generation > > > Key: CALCITE-2966 > URL: https://issues.apache.org/jira/browse/CALCITE-2966 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.20.0 >Reporter: Danny Chan >Assignee: Julian Feinauer >Priority: Major > Fix For: 1.20.0 > > > From the mailing list: > Hi all, > I have some problems with the code generation from Linq4j which I'm unable to > resolve myself. > Basically, I want to translate a condition from Rex to a Linq4j expression to > use it in generated code. > In my example the Condition is from Match Recognize and in SQL is: > `up."commission" > prev(up."commission")`. > > ``` > RexBuilder rexBuilder = new RexBuilder(implementor.getTypeFactory()); > RexProgramBuilder rexProgramBuilder = new > RexProgramBuilder(physType.getRowType(), rexBuilder); > > rexProgramBuilder.addCondition(entry.getValue()); > > final Expression condition = > RexToLixTranslator.translateCondition(rexProgramBuilder.getProgram(), > (JavaTypeFactory) getCluster().getTypeFactory(), > builder2, > inputGetter1, > implementor.allCorrelateVariables, > implementor.getConformance()); > > builder2.add(Expressions.return_(null, condition)); > ``` > > Here, the condition seems okay, it is: ">(PREV(UP.$4, 0), PREV(UP.$4, 1))", > so it should be a comparison of two variables (I rewrite the PREV with a > custom Input Getter". > But, the generated code (for Janino) is: > > ``` > Object p1 = row_.get($L4J$C$0_1); > org.apache.calcite.test.JdbcTest.Employee p0 = > (org.apache.calcite.test.JdbcTest.Employee) p1; > Object p3 = row_.get($L4J$C$1_1); > org.apache.calcite.test.JdbcTest.Employee p2 = > (org.apache.calcite.test.JdbcTest.Employee) p3; > Object p5 = row_.get($L4J$C$0_1); > org.apache.calcite.test.JdbcTest.Employee p4 = > (org.apache.calcite.test.JdbcTest.Employee) p5; > Object p7 = row_.get($L4J$C$1_1); > org.apache.calcite.test.JdbcTest.Employee p6 = > (org.apache.calcite.test.JdbcTest.Employee) p7; > return p0.commission && p2.commission && p4.commission > p6.commission; > ``` > > This confuses me a lot as I do not know where the check for p0.commission and > p2.commission comes from. > It seems that Linq4j adds them as it expects these variables to be nullable, > but I have no idea on how to avoid this. > These fields are Numeric so I always get a compilation exception. > > Can someone help me with this issue? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CALCITE-1935) Reference implementation for MATCH_RECOGNIZE
[ https://issues.apache.org/jira/browse/CALCITE-1935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16816658#comment-16816658 ] Julian Feinauer commented on CALCITE-1935: -- Hi [~julianhyde] thanks for your offer. And I think I have to come back to it. I'm trying to prepare a PR for master but the branch is pretty huge as it builds on a lot of preivous work from you and others. So, for example there are some tests failing where I do not know how to fix them properly. Overall I would be grateful to have you looking into the code and help me to bring the efforts to a PR-eable state. Some issues I currently have are: - 8 or 9 MR related tests in SqlToRelConverterTest fail (which were implemented by [~ransom]) but only due to another representation in the string, so this should be easy to fix if one knows how this Test Setup works. - The largest Issue I currently have is related to CALCITE-2966. From the Codegeneration I do it seems like there is a Bug in RexImplTable. But when I "fix" it so that the JdbcTest.testMatch test becomes green a whole lot of other tests fail. I try to investigate this behavior since two or three days but I do not understand completly why this is the case. I think this could be some kind of edge case (where the others rely on Boolean fields or primitives or something) and I do not, or do something different. - It would be good to check if there are "leftovers" of non MR related code in these branches which was introduced by me or previous authors while "playing around" The branch I'm currently working on is https://github.com/JulianFeinauer/calcite/tree/1935-mr-prepare-pr. I think if we manage to get everything "green" the next step is to go to the PR phase and check all my solutions and if they are fine or things should be solved differently. But first things first, I guess. > Reference implementation for MATCH_RECOGNIZE > > > Key: CALCITE-1935 > URL: https://issues.apache.org/jira/browse/CALCITE-1935 > Project: Calcite > Issue Type: Bug >Reporter: Julian Hyde >Priority: Major > Labels: match > > We now have comprehensive support for parsing and validating MATCH_RECOGNIZE > queries (see CALCITE-1570 and sub-tasks) but we cannot execute them. I know > the purpose of this work is to do CEP within Flink, but a reference > implementation that works on non-streaming data would be valuable. > I propose that we add a class EnumerableMatch that can generate Java code to > evaluate MATCH_RECOGNIZE queries on Enumerable data. It does not need to be > efficient. I don't mind if it (say) buffers all the data in memory and makes > O(n ^ 3) passes over it. People can make it more efficient over time. > When we have a reference implementation, people can start playing with this > feature. And we can start building a corpus of data sets, queries, and their > expected result. The Flink implementation will be able to test against those > same queries, and should give the same results, even though Flink will be > reading streaming data. > Let's create {{match.iq}} with the following query based on > https://oracle-base.com/articles/12c/pattern-matching-in-oracle-database-12cr1: > {code} > !set outputformat mysql > !use match > SELECT * > FROM sales_history MATCH_RECOGNIZE ( > PARTITION BY product > ORDER BY tstamp > MEASURES STRT.tstamp AS start_tstamp, >LAST(UP.tstamp) AS peak_tstamp, >LAST(DOWN.tstamp) AS end_tstamp, >MATCH_NUMBER() AS mno > ONE ROW PER MATCH > AFTER MATCH SKIP TO LAST DOWN > PATTERN (STRT UP+ FLAT* DOWN+) > DEFINE >UP AS UP.units_sold > PREV(UP.units_sold), >FLAT AS FLAT.units_sold = PREV(FLAT.units_sold), >DOWN AS DOWN.units_sold < PREV(DOWN.units_sold) >) MR > ORDER BY MR.product, MR.start_tstamp; > PRODUCTSTART_TSTAM PEAK_TSTAMP END_TSTAMP MNO > -- --- --- --- -- > TWINKIES 01-OCT-2014 03-OCT-2014 06-OCT-2014 1 > TWINKIES 06-OCT-2014 08-OCT-2014 09-OCT-2014 2 > TWINKIES 09-OCT-2014 13-OCT-2014 16-OCT-2014 3 > TWINKIES 16-OCT-2014 18-OCT-2014 20-OCT-2014 4 > 4 rows selected. > !ok > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CALCITE-2966) Problem with Code Generation
[ https://issues.apache.org/jira/browse/CALCITE-2966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16816857#comment-16816857 ] Julian Feinauer commented on CALCITE-2966: -- Hi [~danny0405], thanks for your comment and the explanation. But if I run the Test JdbcTest.testMatch the original code generates invalid java (and Janino fails then). The test uses the Employee Table and it generates the Java: {code:java} return p0.commission && p2.commission && p4.commission > p6.commission {code} which should be in this case {code:java} p0.commission != null && p2.commission != null && p4.commission > p6.commission {code} Can you see why this does not happen? > Problem with Code Generation > > > Key: CALCITE-2966 > URL: https://issues.apache.org/jira/browse/CALCITE-2966 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.20.0 >Reporter: Danny Chan >Assignee: Julian Feinauer >Priority: Major > Fix For: 1.20.0 > > > From the mailing list: > Hi all, > I have some problems with the code generation from Linq4j which I'm unable to > resolve myself. > Basically, I want to translate a condition from Rex to a Linq4j expression to > use it in generated code. > In my example the Condition is from Match Recognize and in SQL is: > `up."commission" > prev(up."commission")`. > > ``` > RexBuilder rexBuilder = new RexBuilder(implementor.getTypeFactory()); > RexProgramBuilder rexProgramBuilder = new > RexProgramBuilder(physType.getRowType(), rexBuilder); > > rexProgramBuilder.addCondition(entry.getValue()); > > final Expression condition = > RexToLixTranslator.translateCondition(rexProgramBuilder.getProgram(), > (JavaTypeFactory) getCluster().getTypeFactory(), > builder2, > inputGetter1, > implementor.allCorrelateVariables, > implementor.getConformance()); > > builder2.add(Expressions.return_(null, condition)); > ``` > > Here, the condition seems okay, it is: ">(PREV(UP.$4, 0), PREV(UP.$4, 1))", > so it should be a comparison of two variables (I rewrite the PREV with a > custom Input Getter". > But, the generated code (for Janino) is: > > ``` > Object p1 = row_.get($L4J$C$0_1); > org.apache.calcite.test.JdbcTest.Employee p0 = > (org.apache.calcite.test.JdbcTest.Employee) p1; > Object p3 = row_.get($L4J$C$1_1); > org.apache.calcite.test.JdbcTest.Employee p2 = > (org.apache.calcite.test.JdbcTest.Employee) p3; > Object p5 = row_.get($L4J$C$0_1); > org.apache.calcite.test.JdbcTest.Employee p4 = > (org.apache.calcite.test.JdbcTest.Employee) p5; > Object p7 = row_.get($L4J$C$1_1); > org.apache.calcite.test.JdbcTest.Employee p6 = > (org.apache.calcite.test.JdbcTest.Employee) p7; > return p0.commission && p2.commission && p4.commission > p6.commission; > ``` > > This confuses me a lot as I do not know where the check for p0.commission and > p2.commission comes from. > It seems that Linq4j adds them as it expects these variables to be nullable, > but I have no idea on how to avoid this. > These fields are Numeric so I always get a compilation exception. > > Can someone help me with this issue? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (CALCITE-2966) Problem with Code Generation
[ https://issues.apache.org/jira/browse/CALCITE-2966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16816857#comment-16816857 ] Julian Feinauer edited comment on CALCITE-2966 at 4/13/19 7:24 AM: --- Hi [~danny0405], thanks for your comment and the explanation. But if I run the Test JdbcTest.testMatch the original code generates invalid java (and Janino fails then). The test uses the Employee Table and it generates the Java: {code:java} return p0.commission && p2.commission && p4.commission > p6.commission {code} which should be in this case (as comission is a Long Field) {code:java} return p0.commission != null && p2.commission != null && p4.commission > p6.commission {code} Can you see why this does not happen? was (Author: julian.feinauer): Hi [~danny0405], thanks for your comment and the explanation. But if I run the Test JdbcTest.testMatch the original code generates invalid java (and Janino fails then). The test uses the Employee Table and it generates the Java: {code:java} return p0.commission && p2.commission && p4.commission > p6.commission {code} which should be in this case {code:java} p0.commission != null && p2.commission != null && p4.commission > p6.commission {code} Can you see why this does not happen? > Problem with Code Generation > > > Key: CALCITE-2966 > URL: https://issues.apache.org/jira/browse/CALCITE-2966 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.20.0 >Reporter: Danny Chan >Assignee: Julian Feinauer >Priority: Major > Fix For: 1.20.0 > > > From the mailing list: > Hi all, > I have some problems with the code generation from Linq4j which I'm unable to > resolve myself. > Basically, I want to translate a condition from Rex to a Linq4j expression to > use it in generated code. > In my example the Condition is from Match Recognize and in SQL is: > `up."commission" > prev(up."commission")`. > > ``` > RexBuilder rexBuilder = new RexBuilder(implementor.getTypeFactory()); > RexProgramBuilder rexProgramBuilder = new > RexProgramBuilder(physType.getRowType(), rexBuilder); > > rexProgramBuilder.addCondition(entry.getValue()); > > final Expression condition = > RexToLixTranslator.translateCondition(rexProgramBuilder.getProgram(), > (JavaTypeFactory) getCluster().getTypeFactory(), > builder2, > inputGetter1, > implementor.allCorrelateVariables, > implementor.getConformance()); > > builder2.add(Expressions.return_(null, condition)); > ``` > > Here, the condition seems okay, it is: ">(PREV(UP.$4, 0), PREV(UP.$4, 1))", > so it should be a comparison of two variables (I rewrite the PREV with a > custom Input Getter". > But, the generated code (for Janino) is: > > ``` > Object p1 = row_.get($L4J$C$0_1); > org.apache.calcite.test.JdbcTest.Employee p0 = > (org.apache.calcite.test.JdbcTest.Employee) p1; > Object p3 = row_.get($L4J$C$1_1); > org.apache.calcite.test.JdbcTest.Employee p2 = > (org.apache.calcite.test.JdbcTest.Employee) p3; > Object p5 = row_.get($L4J$C$0_1); > org.apache.calcite.test.JdbcTest.Employee p4 = > (org.apache.calcite.test.JdbcTest.Employee) p5; > Object p7 = row_.get($L4J$C$1_1); > org.apache.calcite.test.JdbcTest.Employee p6 = > (org.apache.calcite.test.JdbcTest.Employee) p7; > return p0.commission && p2.commission && p4.commission > p6.commission; > ``` > > This confuses me a lot as I do not know where the check for p0.commission and > p2.commission comes from. > It seems that Linq4j adds them as it expects these variables to be nullable, > but I have no idea on how to avoid this. > These fields are Numeric so I always get a compilation exception. > > Can someone help me with this issue? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CALCITE-2966) Problem with Code Generation
[ https://issues.apache.org/jira/browse/CALCITE-2966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16818341#comment-16818341 ] Julian Feinauer commented on CALCITE-2966: -- Hi [~danny0405], thank you SOO much for your detailed analysis of the problem. Indeed I never worked with the NullAs before, so please excuse my mistake. So how do I propagate the given "nullAs"? If I understand you correctly, the issue is that the IS_NOT_NULL which is the input gets "lost" in the process and so the translator.translate(node) does the wrong thing? So how should I forward the nullAs to the translator or the translation? I would be very grateful for a hint... as I'm still a bit confused... > Problem with Code Generation > > > Key: CALCITE-2966 > URL: https://issues.apache.org/jira/browse/CALCITE-2966 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.20.0 >Reporter: Danny Chan >Assignee: Julian Feinauer >Priority: Major > Fix For: 1.20.0 > > > From the mailing list: > Hi all, > I have some problems with the code generation from Linq4j which I'm unable to > resolve myself. > Basically, I want to translate a condition from Rex to a Linq4j expression to > use it in generated code. > In my example the Condition is from Match Recognize and in SQL is: > `up."commission" > prev(up."commission")`. > > ``` > RexBuilder rexBuilder = new RexBuilder(implementor.getTypeFactory()); > RexProgramBuilder rexProgramBuilder = new > RexProgramBuilder(physType.getRowType(), rexBuilder); > > rexProgramBuilder.addCondition(entry.getValue()); > > final Expression condition = > RexToLixTranslator.translateCondition(rexProgramBuilder.getProgram(), > (JavaTypeFactory) getCluster().getTypeFactory(), > builder2, > inputGetter1, > implementor.allCorrelateVariables, > implementor.getConformance()); > > builder2.add(Expressions.return_(null, condition)); > ``` > > Here, the condition seems okay, it is: ">(PREV(UP.$4, 0), PREV(UP.$4, 1))", > so it should be a comparison of two variables (I rewrite the PREV with a > custom Input Getter". > But, the generated code (for Janino) is: > > ``` > Object p1 = row_.get($L4J$C$0_1); > org.apache.calcite.test.JdbcTest.Employee p0 = > (org.apache.calcite.test.JdbcTest.Employee) p1; > Object p3 = row_.get($L4J$C$1_1); > org.apache.calcite.test.JdbcTest.Employee p2 = > (org.apache.calcite.test.JdbcTest.Employee) p3; > Object p5 = row_.get($L4J$C$0_1); > org.apache.calcite.test.JdbcTest.Employee p4 = > (org.apache.calcite.test.JdbcTest.Employee) p5; > Object p7 = row_.get($L4J$C$1_1); > org.apache.calcite.test.JdbcTest.Employee p6 = > (org.apache.calcite.test.JdbcTest.Employee) p7; > return p0.commission && p2.commission && p4.commission > p6.commission; > ``` > > This confuses me a lot as I do not know where the check for p0.commission and > p2.commission comes from. > It seems that Linq4j adds them as it expects these variables to be nullable, > but I have no idea on how to avoid this. > These fields are Numeric so I always get a compilation exception. > > Can someone help me with this issue? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CALCITE-2966) Problem with Code Generation
[ https://issues.apache.org/jira/browse/CALCITE-2966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16818676#comment-16818676 ] Julian Feinauer commented on CALCITE-2966: -- [~danny0405] please excuse this statement.. but I could kiss you : ) I just tested your proposal and it works flawlessly. I'm really sorry to made you that much of work for such a minor change.. but I think my understanding of the Linq internals is still very limitted, so I couldn't see it. Thank you so much, I will close this issue now, ok? > Problem with Code Generation > > > Key: CALCITE-2966 > URL: https://issues.apache.org/jira/browse/CALCITE-2966 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.20.0 >Reporter: Danny Chan >Assignee: Julian Feinauer >Priority: Major > Fix For: 1.20.0 > > > From the mailing list: > Hi all, > I have some problems with the code generation from Linq4j which I'm unable to > resolve myself. > Basically, I want to translate a condition from Rex to a Linq4j expression to > use it in generated code. > In my example the Condition is from Match Recognize and in SQL is: > `up."commission" > prev(up."commission")`. > > ``` > RexBuilder rexBuilder = new RexBuilder(implementor.getTypeFactory()); > RexProgramBuilder rexProgramBuilder = new > RexProgramBuilder(physType.getRowType(), rexBuilder); > > rexProgramBuilder.addCondition(entry.getValue()); > > final Expression condition = > RexToLixTranslator.translateCondition(rexProgramBuilder.getProgram(), > (JavaTypeFactory) getCluster().getTypeFactory(), > builder2, > inputGetter1, > implementor.allCorrelateVariables, > implementor.getConformance()); > > builder2.add(Expressions.return_(null, condition)); > ``` > > Here, the condition seems okay, it is: ">(PREV(UP.$4, 0), PREV(UP.$4, 1))", > so it should be a comparison of two variables (I rewrite the PREV with a > custom Input Getter". > But, the generated code (for Janino) is: > > ``` > Object p1 = row_.get($L4J$C$0_1); > org.apache.calcite.test.JdbcTest.Employee p0 = > (org.apache.calcite.test.JdbcTest.Employee) p1; > Object p3 = row_.get($L4J$C$1_1); > org.apache.calcite.test.JdbcTest.Employee p2 = > (org.apache.calcite.test.JdbcTest.Employee) p3; > Object p5 = row_.get($L4J$C$0_1); > org.apache.calcite.test.JdbcTest.Employee p4 = > (org.apache.calcite.test.JdbcTest.Employee) p5; > Object p7 = row_.get($L4J$C$1_1); > org.apache.calcite.test.JdbcTest.Employee p6 = > (org.apache.calcite.test.JdbcTest.Employee) p7; > return p0.commission && p2.commission && p4.commission > p6.commission; > ``` > > This confuses me a lot as I do not know where the check for p0.commission and > p2.commission comes from. > It seems that Linq4j adds them as it expects these variables to be nullable, > but I have no idea on how to avoid this. > These fields are Numeric so I always get a compilation exception. > > Can someone help me with this issue? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (CALCITE-2966) Problem with Code Generation
[ https://issues.apache.org/jira/browse/CALCITE-2966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Feinauer resolved CALCITE-2966. -- Resolution: Not A Problem In fact, there was no Bug in Calcite but I simply "used" it wrong. Many thanks to [~danny0405] for the excellent help! > Problem with Code Generation > > > Key: CALCITE-2966 > URL: https://issues.apache.org/jira/browse/CALCITE-2966 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.20.0 >Reporter: Danny Chan >Assignee: Julian Feinauer >Priority: Major > Fix For: 1.20.0 > > > From the mailing list: > Hi all, > I have some problems with the code generation from Linq4j which I'm unable to > resolve myself. > Basically, I want to translate a condition from Rex to a Linq4j expression to > use it in generated code. > In my example the Condition is from Match Recognize and in SQL is: > `up."commission" > prev(up."commission")`. > > ``` > RexBuilder rexBuilder = new RexBuilder(implementor.getTypeFactory()); > RexProgramBuilder rexProgramBuilder = new > RexProgramBuilder(physType.getRowType(), rexBuilder); > > rexProgramBuilder.addCondition(entry.getValue()); > > final Expression condition = > RexToLixTranslator.translateCondition(rexProgramBuilder.getProgram(), > (JavaTypeFactory) getCluster().getTypeFactory(), > builder2, > inputGetter1, > implementor.allCorrelateVariables, > implementor.getConformance()); > > builder2.add(Expressions.return_(null, condition)); > ``` > > Here, the condition seems okay, it is: ">(PREV(UP.$4, 0), PREV(UP.$4, 1))", > so it should be a comparison of two variables (I rewrite the PREV with a > custom Input Getter". > But, the generated code (for Janino) is: > > ``` > Object p1 = row_.get($L4J$C$0_1); > org.apache.calcite.test.JdbcTest.Employee p0 = > (org.apache.calcite.test.JdbcTest.Employee) p1; > Object p3 = row_.get($L4J$C$1_1); > org.apache.calcite.test.JdbcTest.Employee p2 = > (org.apache.calcite.test.JdbcTest.Employee) p3; > Object p5 = row_.get($L4J$C$0_1); > org.apache.calcite.test.JdbcTest.Employee p4 = > (org.apache.calcite.test.JdbcTest.Employee) p5; > Object p7 = row_.get($L4J$C$1_1); > org.apache.calcite.test.JdbcTest.Employee p6 = > (org.apache.calcite.test.JdbcTest.Employee) p7; > return p0.commission && p2.commission && p4.commission > p6.commission; > ``` > > This confuses me a lot as I do not know where the check for p0.commission and > p2.commission comes from. > It seems that Linq4j adds them as it expects these variables to be nullable, > but I have no idea on how to avoid this. > These fields are Numeric so I always get a compilation exception. > > Can someone help me with this issue? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (CALCITE-1935) Reference implementation for MATCH_RECOGNIZE
[ https://issues.apache.org/jira/browse/CALCITE-1935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16816658#comment-16816658 ] Julian Feinauer edited comment on CALCITE-1935 at 4/16/19 11:20 AM: Hi [~julianhyde] thanks for your offer. And I think I have to come back to it. I'm trying to prepare a PR for master but the branch is pretty huge as it builds on a lot of preivous work from you and others. So, for example there are some tests failing where I do not know how to fix them properly. Overall I would be grateful to have you looking into the code and help me to bring the efforts to a PR-eable state. Some issues I currently have are: - 8 or 9 MR related tests in SqlToRelConverterTest fail (which were implemented by [~ransom]) but only due to another representation in the string, so this should be easy to fix if one knows how this Test Setup works. - It would be good to check if there are "leftovers" of non MR related code in these branches which was introduced by me or previous authors while "playing around" The branch I'm currently working on is https://github.com/JulianFeinauer/calcite/tree/1935-mr-prepare-pr. I think if we manage to get everything "green" the next step is to go to the PR phase and check all my solutions and if they are fine or things should be solved differently. But first things first, I guess. was (Author: julian.feinauer): Hi [~julianhyde] thanks for your offer. And I think I have to come back to it. I'm trying to prepare a PR for master but the branch is pretty huge as it builds on a lot of preivous work from you and others. So, for example there are some tests failing where I do not know how to fix them properly. Overall I would be grateful to have you looking into the code and help me to bring the efforts to a PR-eable state. Some issues I currently have are: - 8 or 9 MR related tests in SqlToRelConverterTest fail (which were implemented by [~ransom]) but only due to another representation in the string, so this should be easy to fix if one knows how this Test Setup works. - The largest Issue I currently have is related to CALCITE-2966. From the Codegeneration I do it seems like there is a Bug in RexImplTable. But when I "fix" it so that the JdbcTest.testMatch test becomes green a whole lot of other tests fail. I try to investigate this behavior since two or three days but I do not understand completly why this is the case. I think this could be some kind of edge case (where the others rely on Boolean fields or primitives or something) and I do not, or do something different. - It would be good to check if there are "leftovers" of non MR related code in these branches which was introduced by me or previous authors while "playing around" The branch I'm currently working on is https://github.com/JulianFeinauer/calcite/tree/1935-mr-prepare-pr. I think if we manage to get everything "green" the next step is to go to the PR phase and check all my solutions and if they are fine or things should be solved differently. But first things first, I guess. > Reference implementation for MATCH_RECOGNIZE > > > Key: CALCITE-1935 > URL: https://issues.apache.org/jira/browse/CALCITE-1935 > Project: Calcite > Issue Type: Bug >Reporter: Julian Hyde >Priority: Major > Labels: match > > We now have comprehensive support for parsing and validating MATCH_RECOGNIZE > queries (see CALCITE-1570 and sub-tasks) but we cannot execute them. I know > the purpose of this work is to do CEP within Flink, but a reference > implementation that works on non-streaming data would be valuable. > I propose that we add a class EnumerableMatch that can generate Java code to > evaluate MATCH_RECOGNIZE queries on Enumerable data. It does not need to be > efficient. I don't mind if it (say) buffers all the data in memory and makes > O(n ^ 3) passes over it. People can make it more efficient over time. > When we have a reference implementation, people can start playing with this > feature. And we can start building a corpus of data sets, queries, and their > expected result. The Flink implementation will be able to test against those > same queries, and should give the same results, even though Flink will be > reading streaming data. > Let's create {{match.iq}} with the following query based on > https://oracle-base.com/articles/12c/pattern-matching-in-oracle-database-12cr1: > {code} > !set outputformat mysql > !use match > SELECT * > FROM sales_history MATCH_RECOGNIZE ( > PARTITION BY product > ORDER BY tstamp > MEASURES STRT.tstamp AS start_tstamp, >LAST(UP.tstamp) AS peak_tstamp, >LAST(DOWN.tstamp) AS end_tstamp, >MATCH_NUMBER() AS mno >
[jira] [Assigned] (CALCITE-1935) Reference implementation for MATCH_RECOGNIZE
[ https://issues.apache.org/jira/browse/CALCITE-1935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Feinauer reassigned CALCITE-1935: Assignee: Julian Hyde (was: Julian Feinauer) > Reference implementation for MATCH_RECOGNIZE > > > Key: CALCITE-1935 > URL: https://issues.apache.org/jira/browse/CALCITE-1935 > Project: Calcite > Issue Type: Bug >Reporter: Julian Hyde >Assignee: Julian Hyde >Priority: Major > Labels: match > > We now have comprehensive support for parsing and validating MATCH_RECOGNIZE > queries (see CALCITE-1570 and sub-tasks) but we cannot execute them. I know > the purpose of this work is to do CEP within Flink, but a reference > implementation that works on non-streaming data would be valuable. > I propose that we add a class EnumerableMatch that can generate Java code to > evaluate MATCH_RECOGNIZE queries on Enumerable data. It does not need to be > efficient. I don't mind if it (say) buffers all the data in memory and makes > O(n ^ 3) passes over it. People can make it more efficient over time. > When we have a reference implementation, people can start playing with this > feature. And we can start building a corpus of data sets, queries, and their > expected result. The Flink implementation will be able to test against those > same queries, and should give the same results, even though Flink will be > reading streaming data. > Let's create {{match.iq}} with the following query based on > https://oracle-base.com/articles/12c/pattern-matching-in-oracle-database-12cr1: > {code} > !set outputformat mysql > !use match > SELECT * > FROM sales_history MATCH_RECOGNIZE ( > PARTITION BY product > ORDER BY tstamp > MEASURES STRT.tstamp AS start_tstamp, >LAST(UP.tstamp) AS peak_tstamp, >LAST(DOWN.tstamp) AS end_tstamp, >MATCH_NUMBER() AS mno > ONE ROW PER MATCH > AFTER MATCH SKIP TO LAST DOWN > PATTERN (STRT UP+ FLAT* DOWN+) > DEFINE >UP AS UP.units_sold > PREV(UP.units_sold), >FLAT AS FLAT.units_sold = PREV(FLAT.units_sold), >DOWN AS DOWN.units_sold < PREV(DOWN.units_sold) >) MR > ORDER BY MR.product, MR.start_tstamp; > PRODUCTSTART_TSTAM PEAK_TSTAMP END_TSTAMP MNO > -- --- --- --- -- > TWINKIES 01-OCT-2014 03-OCT-2014 06-OCT-2014 1 > TWINKIES 06-OCT-2014 08-OCT-2014 09-OCT-2014 2 > TWINKIES 09-OCT-2014 13-OCT-2014 16-OCT-2014 3 > TWINKIES 16-OCT-2014 18-OCT-2014 20-OCT-2014 4 > 4 rows selected. > !ok > {code} -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (CALCITE-1935) Reference implementation for MATCH_RECOGNIZE
[ https://issues.apache.org/jira/browse/CALCITE-1935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16896198#comment-16896198 ] Julian Feinauer commented on CALCITE-1935: -- Hey [~julianhyde], I submitted a PR which needs a bit of polishing but I think I need some help for the final steps. See my mail on the list (https://lists.apache.org/thread.html/475eec9bc95fc05ddabe3721aa0fffd545f77bf4fc5f7a9ea79c690e@%3Cdev.calcite.apache.org%3E). The PR can be found here https://github.com/apache/calcite/pull/1343. I hope that I can manage to get these first steps done with your (or others) help Thanks already! > Reference implementation for MATCH_RECOGNIZE > > > Key: CALCITE-1935 > URL: https://issues.apache.org/jira/browse/CALCITE-1935 > Project: Calcite > Issue Type: Bug >Reporter: Julian Hyde >Assignee: Julian Hyde >Priority: Major > Labels: match > > We now have comprehensive support for parsing and validating MATCH_RECOGNIZE > queries (see CALCITE-1570 and sub-tasks) but we cannot execute them. I know > the purpose of this work is to do CEP within Flink, but a reference > implementation that works on non-streaming data would be valuable. > I propose that we add a class EnumerableMatch that can generate Java code to > evaluate MATCH_RECOGNIZE queries on Enumerable data. It does not need to be > efficient. I don't mind if it (say) buffers all the data in memory and makes > O(n ^ 3) passes over it. People can make it more efficient over time. > When we have a reference implementation, people can start playing with this > feature. And we can start building a corpus of data sets, queries, and their > expected result. The Flink implementation will be able to test against those > same queries, and should give the same results, even though Flink will be > reading streaming data. > Let's create {{match.iq}} with the following query based on > https://oracle-base.com/articles/12c/pattern-matching-in-oracle-database-12cr1: > {code} > !set outputformat mysql > !use match > SELECT * > FROM sales_history MATCH_RECOGNIZE ( > PARTITION BY product > ORDER BY tstamp > MEASURES STRT.tstamp AS start_tstamp, >LAST(UP.tstamp) AS peak_tstamp, >LAST(DOWN.tstamp) AS end_tstamp, >MATCH_NUMBER() AS mno > ONE ROW PER MATCH > AFTER MATCH SKIP TO LAST DOWN > PATTERN (STRT UP+ FLAT* DOWN+) > DEFINE >UP AS UP.units_sold > PREV(UP.units_sold), >FLAT AS FLAT.units_sold = PREV(FLAT.units_sold), >DOWN AS DOWN.units_sold < PREV(DOWN.units_sold) >) MR > ORDER BY MR.product, MR.start_tstamp; > PRODUCTSTART_TSTAM PEAK_TSTAMP END_TSTAMP MNO > -- --- --- --- -- > TWINKIES 01-OCT-2014 03-OCT-2014 06-OCT-2014 1 > TWINKIES 06-OCT-2014 08-OCT-2014 09-OCT-2014 2 > TWINKIES 09-OCT-2014 13-OCT-2014 16-OCT-2014 3 > TWINKIES 16-OCT-2014 18-OCT-2014 20-OCT-2014 4 > 4 rows selected. > !ok > {code} -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (CALCITE-1935) Reference implementation for MATCH_RECOGNIZE
[ https://issues.apache.org/jira/browse/CALCITE-1935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16897031#comment-16897031 ] Julian Feinauer commented on CALCITE-1935: -- [~julianhyde] I will do and thank you for your help. I think it would be ideal if you give me some feedback on the questions in my mail then I can continue. Or if you have the time, also do some changes on your own :) > Reference implementation for MATCH_RECOGNIZE > > > Key: CALCITE-1935 > URL: https://issues.apache.org/jira/browse/CALCITE-1935 > Project: Calcite > Issue Type: Bug >Reporter: Julian Hyde >Assignee: Julian Hyde >Priority: Major > Labels: match > > We now have comprehensive support for parsing and validating MATCH_RECOGNIZE > queries (see CALCITE-1570 and sub-tasks) but we cannot execute them. I know > the purpose of this work is to do CEP within Flink, but a reference > implementation that works on non-streaming data would be valuable. > I propose that we add a class EnumerableMatch that can generate Java code to > evaluate MATCH_RECOGNIZE queries on Enumerable data. It does not need to be > efficient. I don't mind if it (say) buffers all the data in memory and makes > O(n ^ 3) passes over it. People can make it more efficient over time. > When we have a reference implementation, people can start playing with this > feature. And we can start building a corpus of data sets, queries, and their > expected result. The Flink implementation will be able to test against those > same queries, and should give the same results, even though Flink will be > reading streaming data. > Let's create {{match.iq}} with the following query based on > https://oracle-base.com/articles/12c/pattern-matching-in-oracle-database-12cr1: > {code} > !set outputformat mysql > !use match > SELECT * > FROM sales_history MATCH_RECOGNIZE ( > PARTITION BY product > ORDER BY tstamp > MEASURES STRT.tstamp AS start_tstamp, >LAST(UP.tstamp) AS peak_tstamp, >LAST(DOWN.tstamp) AS end_tstamp, >MATCH_NUMBER() AS mno > ONE ROW PER MATCH > AFTER MATCH SKIP TO LAST DOWN > PATTERN (STRT UP+ FLAT* DOWN+) > DEFINE >UP AS UP.units_sold > PREV(UP.units_sold), >FLAT AS FLAT.units_sold = PREV(FLAT.units_sold), >DOWN AS DOWN.units_sold < PREV(DOWN.units_sold) >) MR > ORDER BY MR.product, MR.start_tstamp; > PRODUCTSTART_TSTAM PEAK_TSTAMP END_TSTAMP MNO > -- --- --- --- -- > TWINKIES 01-OCT-2014 03-OCT-2014 06-OCT-2014 1 > TWINKIES 06-OCT-2014 08-OCT-2014 09-OCT-2014 2 > TWINKIES 09-OCT-2014 13-OCT-2014 16-OCT-2014 3 > TWINKIES 16-OCT-2014 18-OCT-2014 20-OCT-2014 4 > 4 rows selected. > !ok > {code} -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (CALCITE-1935) Reference implementation for MATCH_RECOGNIZE
[ https://issues.apache.org/jira/browse/CALCITE-1935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16897535#comment-16897535 ] Julian Feinauer commented on CALCITE-1935: -- Hey, I think this is a good way and definitely agree with that as I'm eager to see how you would choose to solve specific problems and how it's done in a compliant way with the project. I hope that the code is not too hard to follow and please come back to me whenever there are problems or something is unclear! > Reference implementation for MATCH_RECOGNIZE > > > Key: CALCITE-1935 > URL: https://issues.apache.org/jira/browse/CALCITE-1935 > Project: Calcite > Issue Type: Bug >Reporter: Julian Hyde >Assignee: Julian Hyde >Priority: Major > Labels: match > > We now have comprehensive support for parsing and validating MATCH_RECOGNIZE > queries (see CALCITE-1570 and sub-tasks) but we cannot execute them. I know > the purpose of this work is to do CEP within Flink, but a reference > implementation that works on non-streaming data would be valuable. > I propose that we add a class EnumerableMatch that can generate Java code to > evaluate MATCH_RECOGNIZE queries on Enumerable data. It does not need to be > efficient. I don't mind if it (say) buffers all the data in memory and makes > O(n ^ 3) passes over it. People can make it more efficient over time. > When we have a reference implementation, people can start playing with this > feature. And we can start building a corpus of data sets, queries, and their > expected result. The Flink implementation will be able to test against those > same queries, and should give the same results, even though Flink will be > reading streaming data. > Let's create {{match.iq}} with the following query based on > https://oracle-base.com/articles/12c/pattern-matching-in-oracle-database-12cr1: > {code} > !set outputformat mysql > !use match > SELECT * > FROM sales_history MATCH_RECOGNIZE ( > PARTITION BY product > ORDER BY tstamp > MEASURES STRT.tstamp AS start_tstamp, >LAST(UP.tstamp) AS peak_tstamp, >LAST(DOWN.tstamp) AS end_tstamp, >MATCH_NUMBER() AS mno > ONE ROW PER MATCH > AFTER MATCH SKIP TO LAST DOWN > PATTERN (STRT UP+ FLAT* DOWN+) > DEFINE >UP AS UP.units_sold > PREV(UP.units_sold), >FLAT AS FLAT.units_sold = PREV(FLAT.units_sold), >DOWN AS DOWN.units_sold < PREV(DOWN.units_sold) >) MR > ORDER BY MR.product, MR.start_tstamp; > PRODUCTSTART_TSTAM PEAK_TSTAMP END_TSTAMP MNO > -- --- --- --- -- > TWINKIES 01-OCT-2014 03-OCT-2014 06-OCT-2014 1 > TWINKIES 06-OCT-2014 08-OCT-2014 09-OCT-2014 2 > TWINKIES 09-OCT-2014 13-OCT-2014 16-OCT-2014 3 > TWINKIES 16-OCT-2014 18-OCT-2014 20-OCT-2014 4 > 4 rows selected. > !ok > {code} -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (CALCITE-1935) Reference implementation for MATCH_RECOGNIZE
[ https://issues.apache.org/jira/browse/CALCITE-1935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16902397#comment-16902397 ] Julian Feinauer commented on CALCITE-1935: -- Hey [~julianhyde], you assigned the issue back to me although I assigned it to you as its up to you to do the code review (which is what has to be done now). But please ping me, whenever I can assist you or answer questions. > Reference implementation for MATCH_RECOGNIZE > > > Key: CALCITE-1935 > URL: https://issues.apache.org/jira/browse/CALCITE-1935 > Project: Calcite > Issue Type: Bug >Reporter: Julian Hyde >Assignee: Julian Feinauer >Priority: Major > Labels: match > > We now have comprehensive support for parsing and validating MATCH_RECOGNIZE > queries (see CALCITE-1570 and sub-tasks) but we cannot execute them. I know > the purpose of this work is to do CEP within Flink, but a reference > implementation that works on non-streaming data would be valuable. > I propose that we add a class EnumerableMatch that can generate Java code to > evaluate MATCH_RECOGNIZE queries on Enumerable data. It does not need to be > efficient. I don't mind if it (say) buffers all the data in memory and makes > O(n ^ 3) passes over it. People can make it more efficient over time. > When we have a reference implementation, people can start playing with this > feature. And we can start building a corpus of data sets, queries, and their > expected result. The Flink implementation will be able to test against those > same queries, and should give the same results, even though Flink will be > reading streaming data. > Let's create {{match.iq}} with the following query based on > https://oracle-base.com/articles/12c/pattern-matching-in-oracle-database-12cr1: > {code} > !set outputformat mysql > !use match > SELECT * > FROM sales_history MATCH_RECOGNIZE ( > PARTITION BY product > ORDER BY tstamp > MEASURES STRT.tstamp AS start_tstamp, >LAST(UP.tstamp) AS peak_tstamp, >LAST(DOWN.tstamp) AS end_tstamp, >MATCH_NUMBER() AS mno > ONE ROW PER MATCH > AFTER MATCH SKIP TO LAST DOWN > PATTERN (STRT UP+ FLAT* DOWN+) > DEFINE >UP AS UP.units_sold > PREV(UP.units_sold), >FLAT AS FLAT.units_sold = PREV(FLAT.units_sold), >DOWN AS DOWN.units_sold < PREV(DOWN.units_sold) >) MR > ORDER BY MR.product, MR.start_tstamp; > PRODUCTSTART_TSTAM PEAK_TSTAMP END_TSTAMP MNO > -- --- --- --- -- > TWINKIES 01-OCT-2014 03-OCT-2014 06-OCT-2014 1 > TWINKIES 06-OCT-2014 08-OCT-2014 09-OCT-2014 2 > TWINKIES 09-OCT-2014 13-OCT-2014 16-OCT-2014 3 > TWINKIES 16-OCT-2014 18-OCT-2014 20-OCT-2014 4 > 4 rows selected. > !ok > {code} -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (CALCITE-1935) Reference implementation for MATCH_RECOGNIZE
[ https://issues.apache.org/jira/browse/CALCITE-1935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16902823#comment-16902823 ] Julian Feinauer commented on CALCITE-1935: -- Hey [~julianhyde], I just checked your branch and agree with all of your changes I've seen so far. I also agree with your proposal to squash, I just did not do it for the exact same reason... that there where multiple authos involved and I wanted to keep the history for the final polishing. And I agree that most of the work was done by either you or Zianghe as it was in excellent shape already when I took over, so lets do it like you suggest. PS.: Did you also check the points from my email, where an explicit look is needed? > Reference implementation for MATCH_RECOGNIZE > > > Key: CALCITE-1935 > URL: https://issues.apache.org/jira/browse/CALCITE-1935 > Project: Calcite > Issue Type: Bug >Reporter: Julian Hyde >Assignee: Julian Feinauer >Priority: Major > Labels: match > > We now have comprehensive support for parsing and validating MATCH_RECOGNIZE > queries (see CALCITE-1570 and sub-tasks) but we cannot execute them. I know > the purpose of this work is to do CEP within Flink, but a reference > implementation that works on non-streaming data would be valuable. > I propose that we add a class EnumerableMatch that can generate Java code to > evaluate MATCH_RECOGNIZE queries on Enumerable data. It does not need to be > efficient. I don't mind if it (say) buffers all the data in memory and makes > O(n ^ 3) passes over it. People can make it more efficient over time. > When we have a reference implementation, people can start playing with this > feature. And we can start building a corpus of data sets, queries, and their > expected result. The Flink implementation will be able to test against those > same queries, and should give the same results, even though Flink will be > reading streaming data. > Let's create {{match.iq}} with the following query based on > https://oracle-base.com/articles/12c/pattern-matching-in-oracle-database-12cr1: > {code} > !set outputformat mysql > !use match > SELECT * > FROM sales_history MATCH_RECOGNIZE ( > PARTITION BY product > ORDER BY tstamp > MEASURES STRT.tstamp AS start_tstamp, >LAST(UP.tstamp) AS peak_tstamp, >LAST(DOWN.tstamp) AS end_tstamp, >MATCH_NUMBER() AS mno > ONE ROW PER MATCH > AFTER MATCH SKIP TO LAST DOWN > PATTERN (STRT UP+ FLAT* DOWN+) > DEFINE >UP AS UP.units_sold > PREV(UP.units_sold), >FLAT AS FLAT.units_sold = PREV(FLAT.units_sold), >DOWN AS DOWN.units_sold < PREV(DOWN.units_sold) >) MR > ORDER BY MR.product, MR.start_tstamp; > PRODUCTSTART_TSTAM PEAK_TSTAMP END_TSTAMP MNO > -- --- --- --- -- > TWINKIES 01-OCT-2014 03-OCT-2014 06-OCT-2014 1 > TWINKIES 06-OCT-2014 08-OCT-2014 09-OCT-2014 2 > TWINKIES 09-OCT-2014 13-OCT-2014 16-OCT-2014 3 > TWINKIES 16-OCT-2014 18-OCT-2014 20-OCT-2014 4 > 4 rows selected. > !ok > {code} -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (CALCITE-1935) Reference implementation for MATCH_RECOGNIZE
[ https://issues.apache.org/jira/browse/CALCITE-1935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16903794#comment-16903794 ] Julian Feinauer commented on CALCITE-1935: -- Hey [~julianhyde] thank for your detailed response, just wanted to ensure that everything is tackled before we consider merging to master. I will look into your Quidem Test and try to figure it out and make it work! I'll do this based off on your branch so that we can easily merge it back to your work. > Reference implementation for MATCH_RECOGNIZE > > > Key: CALCITE-1935 > URL: https://issues.apache.org/jira/browse/CALCITE-1935 > Project: Calcite > Issue Type: Bug >Reporter: Julian Hyde >Assignee: Julian Feinauer >Priority: Major > Labels: match > > We now have comprehensive support for parsing and validating MATCH_RECOGNIZE > queries (see CALCITE-1570 and sub-tasks) but we cannot execute them. I know > the purpose of this work is to do CEP within Flink, but a reference > implementation that works on non-streaming data would be valuable. > I propose that we add a class EnumerableMatch that can generate Java code to > evaluate MATCH_RECOGNIZE queries on Enumerable data. It does not need to be > efficient. I don't mind if it (say) buffers all the data in memory and makes > O(n ^ 3) passes over it. People can make it more efficient over time. > When we have a reference implementation, people can start playing with this > feature. And we can start building a corpus of data sets, queries, and their > expected result. The Flink implementation will be able to test against those > same queries, and should give the same results, even though Flink will be > reading streaming data. > Let's create {{match.iq}} with the following query based on > https://oracle-base.com/articles/12c/pattern-matching-in-oracle-database-12cr1: > {code} > !set outputformat mysql > !use match > SELECT * > FROM sales_history MATCH_RECOGNIZE ( > PARTITION BY product > ORDER BY tstamp > MEASURES STRT.tstamp AS start_tstamp, >LAST(UP.tstamp) AS peak_tstamp, >LAST(DOWN.tstamp) AS end_tstamp, >MATCH_NUMBER() AS mno > ONE ROW PER MATCH > AFTER MATCH SKIP TO LAST DOWN > PATTERN (STRT UP+ FLAT* DOWN+) > DEFINE >UP AS UP.units_sold > PREV(UP.units_sold), >FLAT AS FLAT.units_sold = PREV(FLAT.units_sold), >DOWN AS DOWN.units_sold < PREV(DOWN.units_sold) >) MR > ORDER BY MR.product, MR.start_tstamp; > PRODUCTSTART_TSTAM PEAK_TSTAMP END_TSTAMP MNO > -- --- --- --- -- > TWINKIES 01-OCT-2014 03-OCT-2014 06-OCT-2014 1 > TWINKIES 06-OCT-2014 08-OCT-2014 09-OCT-2014 2 > TWINKIES 09-OCT-2014 13-OCT-2014 16-OCT-2014 3 > TWINKIES 16-OCT-2014 18-OCT-2014 20-OCT-2014 4 > 4 rows selected. > !ok > {code} -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (CALCITE-1935) Reference implementation for MATCH_RECOGNIZE
[ https://issues.apache.org/jira/browse/CALCITE-1935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16903819#comment-16903819 ] Julian Feinauer commented on CALCITE-1935: -- I just created a PR where I added a Test to match.iq that works (and is structurally similar to yours). Perhaps this helps to get your working? I cannot reproduce the compilation error : / > Reference implementation for MATCH_RECOGNIZE > > > Key: CALCITE-1935 > URL: https://issues.apache.org/jira/browse/CALCITE-1935 > Project: Calcite > Issue Type: Bug >Reporter: Julian Hyde >Assignee: Julian Feinauer >Priority: Major > Labels: match > > We now have comprehensive support for parsing and validating MATCH_RECOGNIZE > queries (see CALCITE-1570 and sub-tasks) but we cannot execute them. I know > the purpose of this work is to do CEP within Flink, but a reference > implementation that works on non-streaming data would be valuable. > I propose that we add a class EnumerableMatch that can generate Java code to > evaluate MATCH_RECOGNIZE queries on Enumerable data. It does not need to be > efficient. I don't mind if it (say) buffers all the data in memory and makes > O(n ^ 3) passes over it. People can make it more efficient over time. > When we have a reference implementation, people can start playing with this > feature. And we can start building a corpus of data sets, queries, and their > expected result. The Flink implementation will be able to test against those > same queries, and should give the same results, even though Flink will be > reading streaming data. > Let's create {{match.iq}} with the following query based on > https://oracle-base.com/articles/12c/pattern-matching-in-oracle-database-12cr1: > {code} > !set outputformat mysql > !use match > SELECT * > FROM sales_history MATCH_RECOGNIZE ( > PARTITION BY product > ORDER BY tstamp > MEASURES STRT.tstamp AS start_tstamp, >LAST(UP.tstamp) AS peak_tstamp, >LAST(DOWN.tstamp) AS end_tstamp, >MATCH_NUMBER() AS mno > ONE ROW PER MATCH > AFTER MATCH SKIP TO LAST DOWN > PATTERN (STRT UP+ FLAT* DOWN+) > DEFINE >UP AS UP.units_sold > PREV(UP.units_sold), >FLAT AS FLAT.units_sold = PREV(FLAT.units_sold), >DOWN AS DOWN.units_sold < PREV(DOWN.units_sold) >) MR > ORDER BY MR.product, MR.start_tstamp; > PRODUCTSTART_TSTAM PEAK_TSTAMP END_TSTAMP MNO > -- --- --- --- -- > TWINKIES 01-OCT-2014 03-OCT-2014 06-OCT-2014 1 > TWINKIES 06-OCT-2014 08-OCT-2014 09-OCT-2014 2 > TWINKIES 09-OCT-2014 13-OCT-2014 16-OCT-2014 3 > TWINKIES 16-OCT-2014 18-OCT-2014 20-OCT-2014 4 > 4 rows selected. > !ok > {code} -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (CALCITE-1935) Reference implementation for MATCH_RECOGNIZE
[ https://issues.apache.org/jira/browse/CALCITE-1935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16904354#comment-16904354 ] Julian Feinauer commented on CALCITE-1935: -- The commit is https://github.com/JulianFeinauer/calcite/commit/7c1fc2b4e6b01a625919d7ab9519e1218533010a Regarding your comment with CircularArrayList vs my memory class I do not object. Better solutions should always be used. My work is not the end of MR I only tried to bring all loose ends together and bring it one step further for others to jump in and improve. > Reference implementation for MATCH_RECOGNIZE > > > Key: CALCITE-1935 > URL: https://issues.apache.org/jira/browse/CALCITE-1935 > Project: Calcite > Issue Type: Bug >Reporter: Julian Hyde >Assignee: Julian Feinauer >Priority: Major > Labels: match > > We now have comprehensive support for parsing and validating MATCH_RECOGNIZE > queries (see CALCITE-1570 and sub-tasks) but we cannot execute them. I know > the purpose of this work is to do CEP within Flink, but a reference > implementation that works on non-streaming data would be valuable. > I propose that we add a class EnumerableMatch that can generate Java code to > evaluate MATCH_RECOGNIZE queries on Enumerable data. It does not need to be > efficient. I don't mind if it (say) buffers all the data in memory and makes > O(n ^ 3) passes over it. People can make it more efficient over time. > When we have a reference implementation, people can start playing with this > feature. And we can start building a corpus of data sets, queries, and their > expected result. The Flink implementation will be able to test against those > same queries, and should give the same results, even though Flink will be > reading streaming data. > Let's create {{match.iq}} with the following query based on > https://oracle-base.com/articles/12c/pattern-matching-in-oracle-database-12cr1: > {code} > !set outputformat mysql > !use match > SELECT * > FROM sales_history MATCH_RECOGNIZE ( > PARTITION BY product > ORDER BY tstamp > MEASURES STRT.tstamp AS start_tstamp, >LAST(UP.tstamp) AS peak_tstamp, >LAST(DOWN.tstamp) AS end_tstamp, >MATCH_NUMBER() AS mno > ONE ROW PER MATCH > AFTER MATCH SKIP TO LAST DOWN > PATTERN (STRT UP+ FLAT* DOWN+) > DEFINE >UP AS UP.units_sold > PREV(UP.units_sold), >FLAT AS FLAT.units_sold = PREV(FLAT.units_sold), >DOWN AS DOWN.units_sold < PREV(DOWN.units_sold) >) MR > ORDER BY MR.product, MR.start_tstamp; > PRODUCTSTART_TSTAM PEAK_TSTAMP END_TSTAMP MNO > -- --- --- --- -- > TWINKIES 01-OCT-2014 03-OCT-2014 06-OCT-2014 1 > TWINKIES 06-OCT-2014 08-OCT-2014 09-OCT-2014 2 > TWINKIES 09-OCT-2014 13-OCT-2014 16-OCT-2014 3 > TWINKIES 16-OCT-2014 18-OCT-2014 20-OCT-2014 4 > 4 rows selected. > !ok > {code} -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (CALCITE-1935) Reference implementation for MATCH_RECOGNIZE
[ https://issues.apache.org/jira/browse/CALCITE-1935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16905538#comment-16905538 ] Julian Feinauer commented on CALCITE-1935: -- Thank you so much [~julianhyde] for your support and the final polishing. I am very happy that I was finally able to contribute something 'major' to calcite! I hope that I'm able to improve MATCH_RECOGNIZE in the future. > Reference implementation for MATCH_RECOGNIZE > > > Key: CALCITE-1935 > URL: https://issues.apache.org/jira/browse/CALCITE-1935 > Project: Calcite > Issue Type: Bug >Reporter: Julian Hyde >Assignee: Julian Feinauer >Priority: Major > Labels: match > Fix For: 1.21.0 > > > We now have comprehensive support for parsing and validating MATCH_RECOGNIZE > queries (see CALCITE-1570 and sub-tasks) but we cannot execute them. I know > the purpose of this work is to do CEP within Flink, but a reference > implementation that works on non-streaming data would be valuable. > I propose that we add a class EnumerableMatch that can generate Java code to > evaluate MATCH_RECOGNIZE queries on Enumerable data. It does not need to be > efficient. I don't mind if it (say) buffers all the data in memory and makes > O(n ^ 3) passes over it. People can make it more efficient over time. > When we have a reference implementation, people can start playing with this > feature. And we can start building a corpus of data sets, queries, and their > expected result. The Flink implementation will be able to test against those > same queries, and should give the same results, even though Flink will be > reading streaming data. > Let's create {{match.iq}} with the following query based on > https://oracle-base.com/articles/12c/pattern-matching-in-oracle-database-12cr1: > {code} > !set outputformat mysql > !use match > SELECT * > FROM sales_history MATCH_RECOGNIZE ( > PARTITION BY product > ORDER BY tstamp > MEASURES STRT.tstamp AS start_tstamp, >LAST(UP.tstamp) AS peak_tstamp, >LAST(DOWN.tstamp) AS end_tstamp, >MATCH_NUMBER() AS mno > ONE ROW PER MATCH > AFTER MATCH SKIP TO LAST DOWN > PATTERN (STRT UP+ FLAT* DOWN+) > DEFINE >UP AS UP.units_sold > PREV(UP.units_sold), >FLAT AS FLAT.units_sold = PREV(FLAT.units_sold), >DOWN AS DOWN.units_sold < PREV(DOWN.units_sold) >) MR > ORDER BY MR.product, MR.start_tstamp; > PRODUCTSTART_TSTAM PEAK_TSTAMP END_TSTAMP MNO > -- --- --- --- -- > TWINKIES 01-OCT-2014 03-OCT-2014 06-OCT-2014 1 > TWINKIES 06-OCT-2014 08-OCT-2014 09-OCT-2014 2 > TWINKIES 09-OCT-2014 13-OCT-2014 16-OCT-2014 3 > TWINKIES 16-OCT-2014 18-OCT-2014 20-OCT-2014 4 > 4 rows selected. > !ok > {code} -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (CALCITE-1935) Reference implementation for MATCH_RECOGNIZE
[ https://issues.apache.org/jira/browse/CALCITE-1935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16905847#comment-16905847 ] Julian Feinauer commented on CALCITE-1935: -- Thank you very much [~danny0405]! > Reference implementation for MATCH_RECOGNIZE > > > Key: CALCITE-1935 > URL: https://issues.apache.org/jira/browse/CALCITE-1935 > Project: Calcite > Issue Type: Bug >Reporter: Julian Hyde >Assignee: Julian Feinauer >Priority: Major > Labels: match > Fix For: 1.21.0 > > > We now have comprehensive support for parsing and validating MATCH_RECOGNIZE > queries (see CALCITE-1570 and sub-tasks) but we cannot execute them. I know > the purpose of this work is to do CEP within Flink, but a reference > implementation that works on non-streaming data would be valuable. > I propose that we add a class EnumerableMatch that can generate Java code to > evaluate MATCH_RECOGNIZE queries on Enumerable data. It does not need to be > efficient. I don't mind if it (say) buffers all the data in memory and makes > O(n ^ 3) passes over it. People can make it more efficient over time. > When we have a reference implementation, people can start playing with this > feature. And we can start building a corpus of data sets, queries, and their > expected result. The Flink implementation will be able to test against those > same queries, and should give the same results, even though Flink will be > reading streaming data. > Let's create {{match.iq}} with the following query based on > https://oracle-base.com/articles/12c/pattern-matching-in-oracle-database-12cr1: > {code} > !set outputformat mysql > !use match > SELECT * > FROM sales_history MATCH_RECOGNIZE ( > PARTITION BY product > ORDER BY tstamp > MEASURES STRT.tstamp AS start_tstamp, >LAST(UP.tstamp) AS peak_tstamp, >LAST(DOWN.tstamp) AS end_tstamp, >MATCH_NUMBER() AS mno > ONE ROW PER MATCH > AFTER MATCH SKIP TO LAST DOWN > PATTERN (STRT UP+ FLAT* DOWN+) > DEFINE >UP AS UP.units_sold > PREV(UP.units_sold), >FLAT AS FLAT.units_sold = PREV(FLAT.units_sold), >DOWN AS DOWN.units_sold < PREV(DOWN.units_sold) >) MR > ORDER BY MR.product, MR.start_tstamp; > PRODUCTSTART_TSTAM PEAK_TSTAMP END_TSTAMP MNO > -- --- --- --- -- > TWINKIES 01-OCT-2014 03-OCT-2014 06-OCT-2014 1 > TWINKIES 06-OCT-2014 08-OCT-2014 09-OCT-2014 2 > TWINKIES 09-OCT-2014 13-OCT-2014 16-OCT-2014 3 > TWINKIES 16-OCT-2014 18-OCT-2014 20-OCT-2014 4 > 4 rows selected. > !ok > {code} -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (CALCITE-3294) Implement FINAL Clause for MATCH_RECOGNIZE
Julian Feinauer created CALCITE-3294: Summary: Implement FINAL Clause for MATCH_RECOGNIZE Key: CALCITE-3294 URL: https://issues.apache.org/jira/browse/CALCITE-3294 Project: Calcite Issue Type: Improvement Reporter: Julian Feinauer With CALCITE-1935 the initial support for the MATCH_RECOGNIZE clause was introduced. But it is still lacking several features. One of them is the FINAL Clause which forces the `MEASURE` to act global on all Tuples which were matched by the pattern. See https://oracle-base.com/articles/12c/pattern-matching-in-oracle-database-12cr1 An example query would be: {code} SELECT * FROM sales_history MATCH_RECOGNIZE ( ORDER BY tstamp MEASURES LAST(A.tstamp) AS ts_prev, FINAL LAST(A.tstamp) AS ts_last ALL ROWS PER MATCH PATTERN (A+) DEFINE A AS A.units_sold > 10 ) MR ORDER BY MR.product, MR.start_tstamp; {code} Here, the query matches for each sequence of rows which all have `units_sold > 10`. For the column `ts_prev` it always shows the timestamp of the timestamp of the previous match (the row before). But for `ts_last` it shows the SAME value for each column as the `FINAL` modifier changes teh behavior to apply the `LAST` operator to the record set (similar to a window aggregation on the machted subset of rows). -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (CALCITE-3294) Implement FINAL Clause for MATCH_RECOGNIZE
[ https://issues.apache.org/jira/browse/CALCITE-3294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16915742#comment-16915742 ] Julian Feinauer commented on CALCITE-3294: -- [~julianhyde] I could imagine working on that (although I have not that much time atm, just as a warning). But I would like to have your insights or ideas on how to implement that. My first idea is to treat it somewhat similar to how the Window Aggreation works as it should be really similar under the covers. Any ideas? > Implement FINAL Clause for MATCH_RECOGNIZE > -- > > Key: CALCITE-3294 > URL: https://issues.apache.org/jira/browse/CALCITE-3294 > Project: Calcite > Issue Type: Improvement >Reporter: Julian Feinauer >Priority: Major > > With CALCITE-1935 the initial support for the MATCH_RECOGNIZE clause was > introduced. But it is still lacking several features. > One of them is the FINAL Clause which forces the `MEASURE` to act global on > all Tuples which were matched by the pattern. > See > https://oracle-base.com/articles/12c/pattern-matching-in-oracle-database-12cr1 > An example query would be: > {code} > SELECT * > FROM sales_history MATCH_RECOGNIZE ( > ORDER BY tstamp > MEASURES >LAST(A.tstamp) AS ts_prev, >FINAL LAST(A.tstamp) AS ts_last > ALL ROWS PER MATCH > PATTERN (A+) > DEFINE >A AS A.units_sold > 10 >) MR > ORDER BY MR.product, MR.start_tstamp; > {code} > Here, the query matches for each sequence of rows which all have `units_sold > > 10`. > For the column `ts_prev` it always shows the timestamp of the timestamp of > the previous match (the row before). > But for `ts_last` it shows the SAME value for each column as the `FINAL` > modifier changes teh behavior to apply the `LAST` operator to the record set > (similar to a window aggregation on the machted subset of rows). -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (CALCITE-3302) Add support for CLASSIFIER() command in MATCH_RECOGNIZE
Julian Feinauer created CALCITE-3302: Summary: Add support for CLASSIFIER() command in MATCH_RECOGNIZE Key: CALCITE-3302 URL: https://issues.apache.org/jira/browse/CALCITE-3302 Project: Calcite Issue Type: Improvement Reporter: Julian Feinauer Assignee: Julian Feinauer The CLASSIFIER() command simply returns the defined pattern classifier that was matched for the respective column. A very simple test case which could be added to match.iq is {code:java} select * from "hr"."emps" match_recognize ( order by "empid" desc measures "commission" as c, "empid" as empid, CLASSIFIER() as cl pattern (s up) define up as up."commission" < prev(up."commission")); CEMPID CL - - 1000 100 S 500 200 UP !ok {code} -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (CALCITE-3302) Add support for CLASSIFIER() command in MATCH_RECOGNIZE
[ https://issues.apache.org/jira/browse/CALCITE-3302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Feinauer updated CALCITE-3302: - Description: The CLASSIFIER() command simply returns the defined pattern classifier that was matched for the respective column. A very simple test case which could be added to match.iq is {code:java} select * from "hr"."emps" match_recognize ( order by "empid" desc measures "commission" as c, "empid" as empid, CLASSIFIER() as cl pattern (s up) define up as up."commission" < prev(up."commission")); CEMPID CL - -- 1000 100 S 500 200 UP !ok {code} was: The CLASSIFIER() command simply returns the defined pattern classifier that was matched for the respective column. A very simple test case which could be added to match.iq is {code:java} select * from "hr"."emps" match_recognize ( order by "empid" desc measures "commission" as c, "empid" as empid, CLASSIFIER() as cl pattern (s up) define up as up."commission" < prev(up."commission")); CEMPID CL - - 1000 100 S 500 200 UP !ok {code} > Add support for CLASSIFIER() command in MATCH_RECOGNIZE > --- > > Key: CALCITE-3302 > URL: https://issues.apache.org/jira/browse/CALCITE-3302 > Project: Calcite > Issue Type: Improvement >Reporter: Julian Feinauer >Assignee: Julian Feinauer >Priority: Major > > The CLASSIFIER() command simply returns the defined pattern classifier that > was matched for the respective column. > A very simple test case which could be added to match.iq is > {code:java} > select * > from "hr"."emps" match_recognize ( > order by "empid" desc > measures "commission" as c, > "empid" as empid, > CLASSIFIER() as cl > pattern (s up) > define up as up."commission" < prev(up."commission")); > CEMPID CL > - -- > 1000 100 S > 500 200 UP > !ok > {code} -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (CALCITE-3294) Implement FINAL Clause for MATCH_RECOGNIZE
[ https://issues.apache.org/jira/browse/CALCITE-3294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16918044#comment-16918044 ] Julian Feinauer commented on CALCITE-3294: -- Found this pretty nice expansion https://docs.oracle.com/database/121/DWHSG/pattern.htm#GUID-F7ECC486-0BDA-47AD-90B9-C356F5E18C5B > Implement FINAL Clause for MATCH_RECOGNIZE > -- > > Key: CALCITE-3294 > URL: https://issues.apache.org/jira/browse/CALCITE-3294 > Project: Calcite > Issue Type: Improvement >Reporter: Julian Feinauer >Priority: Major > > With CALCITE-1935 the initial support for the MATCH_RECOGNIZE clause was > introduced. But it is still lacking several features. > One of them is the FINAL Clause which forces the `MEASURE` to act global on > all Tuples which were matched by the pattern. > See > https://oracle-base.com/articles/12c/pattern-matching-in-oracle-database-12cr1 > An example query would be: > {code} > SELECT * > FROM sales_history MATCH_RECOGNIZE ( > ORDER BY tstamp > MEASURES >LAST(A.tstamp) AS ts_prev, >FINAL LAST(A.tstamp) AS ts_last > ALL ROWS PER MATCH > PATTERN (A+) > DEFINE >A AS A.units_sold > 10 >) MR > ORDER BY MR.product, MR.start_tstamp; > {code} > Here, the query matches for each sequence of rows which all have `units_sold > > 10`. > For the column `ts_prev` it always shows the timestamp of the timestamp of > the previous match (the row before). > But for `ts_last` it shows the SAME value for each column as the `FINAL` > modifier changes teh behavior to apply the `LAST` operator to the record set > (similar to a window aggregation on the machted subset of rows). -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (CALCITE-3328) Immutable beans, powered by reflection
[ https://issues.apache.org/jira/browse/CALCITE-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16924744#comment-16924744 ] Julian Feinauer commented on CALCITE-3328: -- I agree that this is a very good idea but also wonder why there is no default for something like that. Wouldnt this be something for apache commons as also other projects may need that? And second question: Do we use custom annotations or do we use generic ones (JEE, ...) for that? > Immutable beans, powered by reflection > -- > > Key: CALCITE-3328 > URL: https://issues.apache.org/jira/browse/CALCITE-3328 > Project: Calcite > Issue Type: Bug >Reporter: Julian Hyde >Priority: Major > > In quite a few places we have immutable config classes, and because they are > immutable we have builders. It all gets quite verbose. > I propose to add a utility class, {{ImmutableBeans}}, that is able to > generate an implementation of a given interface. For example, > {code:java} > interface MyBean { > @Property int getAge(); > MyBean withAge(int age); > @Property String getName(); > MyBean withName(String name); > } > MyBean b = ImmutableBeans.create(MyBean.class); > assertThat(b.withAge(37).withName("Fred").getAge(), is(37)); > {code} > We could replace {{SqlParser.ConfigBuilder}}, {{Frameworks.ConfigBuilder}}, > {{RelBuilder.ConfigBuilder}}, quite a few of the associated {{Config}} > classes, and perhaps quite a few fluent classes that we use in tests. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (CALCITE-3328) Immutable beans, powered by reflection
[ https://issues.apache.org/jira/browse/CALCITE-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16924748#comment-16924748 ] Julian Feinauer commented on CALCITE-3328: -- I see. Its probably debatable which approach fits better. But generally I like to think about donating such things to commons (or else) as one may want to use it in other projects and its cleaner to include commons. > Immutable beans, powered by reflection > -- > > Key: CALCITE-3328 > URL: https://issues.apache.org/jira/browse/CALCITE-3328 > Project: Calcite > Issue Type: Bug >Reporter: Julian Hyde >Assignee: Julian Hyde >Priority: Major > > In quite a few places we have immutable config classes, and because they are > immutable we have builders. It all gets quite verbose. > I propose to add a utility class, {{ImmutableBeans}}, that is able to > generate an implementation of a given interface. For example, > {code:java} > interface MyBean { > @Property int getAge(); > MyBean withAge(int age); > @Property String getName(); > MyBean withName(String name); > } > MyBean b = ImmutableBeans.create(MyBean.class); > assertThat(b.withAge(37).withName("Fred").getAge(), is(37)); > {code} > We could replace {{SqlParser.ConfigBuilder}}, {{Frameworks.ConfigBuilder}}, > {{RelBuilder.ConfigBuilder}}, quite a few of the associated {{Config}} > classes, and perhaps quite a few fluent classes that we use in tests. > Work in progress: > https://github.com/julianhyde/calcite/tree/3328-immutable-beans -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (CALCITE-3336) Intermittent failure of PigRelBuilderStyleTest.testImplWithGroupByCountDistinct
[ https://issues.apache.org/jira/browse/CALCITE-3336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16926140#comment-16926140 ] Julian Feinauer commented on CALCITE-3336: -- Thank you [~zabetak] for creating the issue, I added my configuration. > Intermittent failure of > PigRelBuilderStyleTest.testImplWithGroupByCountDistinct > --- > > Key: CALCITE-3336 > URL: https://issues.apache.org/jira/browse/CALCITE-3336 > Project: Calcite > Issue Type: Bug > Components: pig-adapter >Affects Versions: 1.21.0 > Environment: +Jenkins environment+ > jdk-11-ea+28 > apache-maven-3.5.2 > Ubuntu 16.04 LTS > +Local Environment (jfeinauer)+ > java version "1.8.0_181" > Java(TM) SE Runtime Environment (build 1.8.0_181-b13) > Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode) > Apache Maven 3.6.1 (d66c9c0b3152b2e69ee9bac180bb8fcc8e6af555; > 2019-04-04T12:00:29-07:00) > macOS Mojave 10.14.4 (18E226) >Reporter: Stamatis Zampetakis >Priority: Major > > The following tests fails intermittently with the following stack trace: > {noformat} > [ERROR] Tests run: 8, Failures: 0, Errors: 1, Skipped: 4, Time elapsed: 5.518 > s <<< FAILURE! - in org.apache.calcite.test.PigRelBuilderStyleTest > [ERROR] > testImplWithGroupByCountDistinct(org.apache.calcite.test.PigRelBuilderStyleTest) > Time elapsed: 3.82 s <<< ERROR! > org.apache.pig.impl.logicalLayer.FrontendException: Unable to open iterator > for alias t > at > org.apache.calcite.test.PigRelBuilderStyleTest.assertScriptAndResults(PigRelBuilderStyleTest.java:270) > at > org.apache.calcite.test.PigRelBuilderStyleTest.testImplWithGroupByCountDistinct(PigRelBuilderStyleTest.java:167) > Caused by: org.apache.pig.PigException: Unable to store alias t > at > org.apache.calcite.test.PigRelBuilderStyleTest.assertScriptAndResults(PigRelBuilderStyleTest.java:270) > at > org.apache.calcite.test.PigRelBuilderStyleTest.testImplWithGroupByCountDistinct(PigRelBuilderStyleTest.java:167) > Caused by: org.apache.pig.impl.logicalLayer.FrontendException: Error > processing rule LoadTypeCastInserter > at > org.apache.calcite.test.PigRelBuilderStyleTest.assertScriptAndResults(PigRelBuilderStyleTest.java:270) > at > org.apache.calcite.test.PigRelBuilderStyleTest.testImplWithGroupByCountDistinct(PigRelBuilderStyleTest.java:167) > Caused by: java.lang.NullPointerException > at > org.apache.calcite.test.PigRelBuilderStyleTest.assertScriptAndResults(PigRelBuilderStyleTest.java:270) > at > org.apache.calcite.test.PigRelBuilderStyleTest.testImplWithGroupByCountDistinct(PigRelBuilderStyleTest.java:167) > {noformat} > The problem appeared a few times in Jenkins (e.g., in build > [1330|https://builds.apache.org/job/Calcite-Master/1330/jdk=JDK%2011%20(latest),label_exp=ubuntu&&!cloud-slave&&!H27&&!ubuntu-4/]) > and was observed also by [~julian.feinauer]. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (CALCITE-3336) Intermittent failure of PigRelBuilderStyleTest.testImplWithGroupByCountDistinct
[ https://issues.apache.org/jira/browse/CALCITE-3336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Feinauer updated CALCITE-3336: - Environment: +Jenkins environment+ jdk-11-ea+28 apache-maven-3.5.2 Ubuntu 16.04 LTS +Local Environment (jfeinauer)+ java version "1.8.0_181" Java(TM) SE Runtime Environment (build 1.8.0_181-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode) Apache Maven 3.6.1 (d66c9c0b3152b2e69ee9bac180bb8fcc8e6af555; 2019-04-04T12:00:29-07:00) macOS Mojave 10.14.4 (18E226) was: +Jenkins environment+ jdk-11-ea+28 apache-maven-3.5.2 Ubuntu 16.04 LTS > Intermittent failure of > PigRelBuilderStyleTest.testImplWithGroupByCountDistinct > --- > > Key: CALCITE-3336 > URL: https://issues.apache.org/jira/browse/CALCITE-3336 > Project: Calcite > Issue Type: Bug > Components: pig-adapter >Affects Versions: 1.21.0 > Environment: +Jenkins environment+ > jdk-11-ea+28 > apache-maven-3.5.2 > Ubuntu 16.04 LTS > +Local Environment (jfeinauer)+ > java version "1.8.0_181" > Java(TM) SE Runtime Environment (build 1.8.0_181-b13) > Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode) > Apache Maven 3.6.1 (d66c9c0b3152b2e69ee9bac180bb8fcc8e6af555; > 2019-04-04T12:00:29-07:00) > macOS Mojave 10.14.4 (18E226) >Reporter: Stamatis Zampetakis >Priority: Major > > The following tests fails intermittently with the following stack trace: > {noformat} > [ERROR] Tests run: 8, Failures: 0, Errors: 1, Skipped: 4, Time elapsed: 5.518 > s <<< FAILURE! - in org.apache.calcite.test.PigRelBuilderStyleTest > [ERROR] > testImplWithGroupByCountDistinct(org.apache.calcite.test.PigRelBuilderStyleTest) > Time elapsed: 3.82 s <<< ERROR! > org.apache.pig.impl.logicalLayer.FrontendException: Unable to open iterator > for alias t > at > org.apache.calcite.test.PigRelBuilderStyleTest.assertScriptAndResults(PigRelBuilderStyleTest.java:270) > at > org.apache.calcite.test.PigRelBuilderStyleTest.testImplWithGroupByCountDistinct(PigRelBuilderStyleTest.java:167) > Caused by: org.apache.pig.PigException: Unable to store alias t > at > org.apache.calcite.test.PigRelBuilderStyleTest.assertScriptAndResults(PigRelBuilderStyleTest.java:270) > at > org.apache.calcite.test.PigRelBuilderStyleTest.testImplWithGroupByCountDistinct(PigRelBuilderStyleTest.java:167) > Caused by: org.apache.pig.impl.logicalLayer.FrontendException: Error > processing rule LoadTypeCastInserter > at > org.apache.calcite.test.PigRelBuilderStyleTest.assertScriptAndResults(PigRelBuilderStyleTest.java:270) > at > org.apache.calcite.test.PigRelBuilderStyleTest.testImplWithGroupByCountDistinct(PigRelBuilderStyleTest.java:167) > Caused by: java.lang.NullPointerException > at > org.apache.calcite.test.PigRelBuilderStyleTest.assertScriptAndResults(PigRelBuilderStyleTest.java:270) > at > org.apache.calcite.test.PigRelBuilderStyleTest.testImplWithGroupByCountDistinct(PigRelBuilderStyleTest.java:167) > {noformat} > The problem appeared a few times in Jenkins (e.g., in build > [1330|https://builds.apache.org/job/Calcite-Master/1330/jdk=JDK%2011%20(latest),label_exp=ubuntu&&!cloud-slave&&!H27&&!ubuntu-4/]) > and was observed also by [~julian.feinauer]. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (CALCITE-3341) Implement FINAL modifier functionality
Julian Feinauer created CALCITE-3341: Summary: Implement FINAL modifier functionality Key: CALCITE-3341 URL: https://issues.apache.org/jira/browse/CALCITE-3341 Project: Calcite Issue Type: Bug Reporter: Julian Feinauer And set default modifier to RUNNING that its compliant with Oracle implementation, see CALCITE-3302. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (CALCITE-3345) Implement time_bucket function
Julian Feinauer created CALCITE-3345: Summary: Implement time_bucket function Key: CALCITE-3345 URL: https://issues.apache.org/jira/browse/CALCITE-3345 Project: Calcite Issue Type: New Feature Reporter: Julian Feinauer See here for information on the `time_bucket` function: https://docs.timescale.com/latest/api#time_bucket This is a more powerful version of the standard PostgreSQL date_trunc function. It allows for arbitrary time intervals instead of the second, minute, hour, etc. provided by date_trunc. The return value is the bucket's start time. This would especially help with time averaging but keeps everything SQL compliant. E.g. queries like Example query from (https://www.timescale.com/): {code:sql} SELECT time_bucket('10 seconds', time) AS ten_second, machine_id, avg(temperature) AS "avgT", min(temperature) AS "minT", max(temperature) AS "maxT", last(temperature, time) AS "lastT" FROM measurements WHERE machine_id = 'C931baF7' AND time > now() - interval '150s' GROUP BY ten_second ORDER BY ten_second DESC; {code} -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (CALCITE-3345) Implement time_bucket function
[ https://issues.apache.org/jira/browse/CALCITE-3345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16929866#comment-16929866 ] Julian Feinauer commented on CALCITE-3345: -- Hi, didnt know that, thanks for the link, this looks good. But I would also like to implement this just for conformance with timescaleDB? > Implement time_bucket function > -- > > Key: CALCITE-3345 > URL: https://issues.apache.org/jira/browse/CALCITE-3345 > Project: Calcite > Issue Type: New Feature >Reporter: Julian Feinauer >Priority: Major > > See here for information on the `time_bucket` function: > https://docs.timescale.com/latest/api#time_bucket > This is a more powerful version of the standard PostgreSQL date_trunc > function. It allows for arbitrary time intervals instead of the second, > minute, hour, etc. provided by date_trunc. The return value is the bucket's > start time. > This would especially help with time averaging but keeps everything SQL > compliant. E.g. queries like > Example query from (https://www.timescale.com/): > {code:sql} > SELECT time_bucket('10 seconds', time) AS ten_second, > machine_id, avg(temperature) AS "avgT", > min(temperature) AS "minT", max(temperature) AS "maxT", > last(temperature, time) AS "lastT" > FROM measurements > WHERE machine_id = 'C931baF7' > AND time > now() - interval '150s' > GROUP BY ten_second > ORDER BY ten_second DESC; > {code} -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (CALCITE-2640) Support for SQL on Timeseries / Traces
Julian Feinauer created CALCITE-2640: Summary: Support for SQL on Timeseries / Traces Key: CALCITE-2640 URL: https://issues.apache.org/jira/browse/CALCITE-2640 Project: Calcite Issue Type: New Feature Reporter: Julian Feinauer Assignee: Julian Hyde When working with time series data, e.g., from IoT devices, traces or other sources there is often the need for analysis on "transients". This means that it is important to compare values with their neighbors (i.e., prev / next) to detect jumps or changes of bits. This is possible with the MATCH_RECOGNIZE functionality from SQL 2016 but perhaps there is also the possibility for a "sensible" extension to SQL (like Streaming SQL is) to support these use cases with a "less technical" syntax. Others also use a pseudo-SQL syntax to do "more time series focussed" things (like use GROUP BY to do time averaging in IinfluxDB). See [1] for InfluxDBs SQL-like QL InfluxQL. And it would be better to have a "more standardized" way of the interaction between SQL and timeseries than several SQL-like DSLs. There was also a discussion on the list, see [2]. [1] https://docs.influxdata.com/influxdb/v1.6/query_language/ [2] https://lists.apache.org/thread.html/250575a56165851ab55351b90a26eaa30e84d5bbe2b31203daaaefb9@%3Cdev.calcite.apache.org%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CALCITE-1414) Add a RANDOM function
[ https://issues.apache.org/jira/browse/CALCITE-1414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15751039#comment-15751039 ] Julian Feinauer commented on CALCITE-1414: -- I added a pull request: https://github.com/apache/calcite/pull/338 with my suggestion how to implement this function. > Add a RANDOM function > - > > Key: CALCITE-1414 > URL: https://issues.apache.org/jira/browse/CALCITE-1414 > Project: Calcite > Issue Type: Bug >Reporter: Julian Hyde >Assignee: Julian Hyde > Labels: newbie > > Add a RANDOM function. > It is not standard SQL, but most DBMSs have one, and it be useful. > It will also allow us to better test non-deterministic expressions. > I propose that: > * {{RANDOM(n)}} returns an integer between 0 and n - 1 inclusive. > * There is no facility to specify a seed at this time. > * The random number generator is randomly seeded at query start time based on > (say) the method used by Java's {{new Random()}}; if you run the query twice, > you will (probably) not get the same results. > * If you call the function twice in the same row, it will (probably) return > different results. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CALCITE-1414) Add a RANDOM function
[ https://issues.apache.org/jira/browse/CALCITE-1414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15751039#comment-15751039 ] Julian Feinauer edited comment on CALCITE-1414 at 12/15/16 11:03 AM: - I added a pull request: https://github.com/apache/calcite/pull/339 with my suggestion how to implement this function. I had to resubmit the PR https://github.com/apache/calcite/pull/338 due to a problem with the CI / Validator. Sorry for that. was (Author: julian.feinauer): I added a pull request: https://github.com/apache/calcite/pull/338 with my suggestion how to implement this function. > Add a RANDOM function > - > > Key: CALCITE-1414 > URL: https://issues.apache.org/jira/browse/CALCITE-1414 > Project: Calcite > Issue Type: Bug >Reporter: Julian Hyde >Assignee: Julian Hyde > Labels: newbie > > Add a RANDOM function. > It is not standard SQL, but most DBMSs have one, and it be useful. > It will also allow us to better test non-deterministic expressions. > I propose that: > * {{RANDOM(n)}} returns an integer between 0 and n - 1 inclusive. > * There is no facility to specify a seed at this time. > * The random number generator is randomly seeded at query start time based on > (say) the method used by Java's {{new Random()}}; if you run the query twice, > you will (probably) not get the same results. > * If you call the function twice in the same row, it will (probably) return > different results. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CALCITE-1414) Add a RANDOM function
[ https://issues.apache.org/jira/browse/CALCITE-1414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15760892#comment-15760892 ] Julian Feinauer commented on CALCITE-1414: -- [~julianhyde], I renamed everything to RAND_INTEGER and added a Comment in the Readme.md. Please feel free to give me any more hints if I should change something. > Add a RANDOM function > - > > Key: CALCITE-1414 > URL: https://issues.apache.org/jira/browse/CALCITE-1414 > Project: Calcite > Issue Type: Bug >Reporter: Julian Hyde >Assignee: Julian Feinauer > Labels: newbie > Fix For: 1.11.0 > > > Add a RANDOM function. > It is not standard SQL, but most DBMSs have one, and it be useful. > It will also allow us to better test non-deterministic expressions. > I propose that: > * {{RANDOM\(n)}} returns an integer between 0 and n - 1 inclusive. > * There is no facility to specify a seed at this time. > * The random number generator is randomly seeded at query start time based on > (say) the method used by Java's {{new Random()}}; if you run the query twice, > you will (probably) not get the same results. > * If you call the function twice in the same row, it will (probably) return > different results. -- This message was sent by Atlassian JIRA (v6.3.4#6332)