Re: Review requested [CALCITE-2914] Improve how LatticeSuggester deduces foreign keys

[Yuzhao Chen]

I will do it.

Best,
Danny Chan
在 2019年4月1日 +0800 AM6:39，Julian Hyde ，写道：
> Can someone please review https://github.com/apache/calcite/pull/1141 
>  (for 
> https://issues.apache.org/jira/browse/CALCITE-2914 
> ).
>
> Julian
>

Re: Problem with Code Generation

[Yuzhao Chen]

Julian Feinauer, i have fire a JIRA and prepare to fix it.

[1] https://issues.apache.org/jira/browse/CALCITE-2966

Best,
Danny Chan
在 2019年4月1日 +0800 AM12:45，dev@calcite.apache.org，写道：
>
> Hi all,
>
> I have some problems with the code generation from Linq4j which I'm unable to 
> resolve myself.
> Basically, I want to translate a condition from Rex to a Linq4j expression to 
> use it in generated code.
> In my example the Condition is from Match Recognize and in SQL is: 
> `up."commission" > prev(up."commission")`.
>
> ```
> RexBuilder rexBuilder = new RexBuilder(implementor.getTypeFactory());
> RexProgramBuilder rexProgramBuilder = new 
> RexProgramBuilder(physType.getRowType(), rexBuilder);
>
> rexProgramBuilder.addCondition(entry.getValue());
>
> final Expression condition = 
> RexToLixTranslator.translateCondition(rexProgramBuilder.getProgram(),
> (JavaTypeFactory) getCluster().getTypeFactory(),
> builder2,
> inputGetter1,
> implementor.allCorrelateVariables,
> implementor.getConformance());
>
> builder2.add(Expressions.return_(null, condition));
> ```
>
> Here, the condition seems okay, it is: ">(PREV(UP.$4, 0), PREV(UP.$4, 1))", 
> so it should be a comparison of two variables (I rewrite the PREV with a 
> custom Input Getter".
> But, the generated code (for Janino) is:
>
> ```
> Object p1 = row_.get($L4J$C$0_1);
> org.apache.calcite.test.JdbcTest.Employee p0 = 
> (org.apache.calcite.test.JdbcTest.Employee) p1;
> Object p3 = row_.get($L4J$C$1_1);
> org.apache.calcite.test.JdbcTest.Employee p2 = 
> (org.apache.calcite.test.JdbcTest.Employee) p3;
> Object p5 = row_.get($L4J$C$0_1);
> org.apache.calcite.test.JdbcTest.Employee p4 = 
> (org.apache.calcite.test.JdbcTest.Employee) p5;
> Object p7 = row_.get($L4J$C$1_1);
> org.apache.calcite.test.JdbcTest.Employee p6 = 
> (org.apache.calcite.test.JdbcTest.Employee) p7;
> return p0.commission && p2.commission && p4.commission > p6.commission;
> ```
>
> This confuses me a lot as I do not know where the check for p0.commission and 
> p2.commission comes from.
> It seems that Linq4j adds them as it expects these variables to be nullable, 
> but I have no idea on how to avoid this.
> These fields are Numeric so I always get a compilation exception.
>
> Can someone help me with this issue?

Re: Join, SemiJoin, Correlate

[Yuzhao Chen]

I will take it, hope to make some help.

Best,
Danny Chan
在 2019年4月1日 +0800 PM7:20，Stamatis Zampetakis ，写道：
> It seems that the discusion has somehow converged (at least to the major
> points). I created CALCITE-2969, for whoever decides to tackle this issue.
>
> [1] https://issues.apache.org/jira/browse/CALCITE-2969
>
> Στις Δευ, 25 Μαρ 2019 στις 8:57 μ.μ., ο/η Julian Hyde 
> έγραψε:
>
> > Generally +1 on what Haisheng says. Specifically:
> >
> > I like the idea of renaming EnumerableCorrelate, and making it not extend
> > Correlate. I would choose EnumerableNestedLoopJoin rather than
> > EnumerableNestLoopJoin.
> >
> > Shifting from LogicalCorrelate to LogicalApply is worth considering.
> > LogicalApply is similar to the “map” operator in functional programming, or
> > “selectMany” in LINQ, so is very well-behaved and powerful - a good
> > abstraction.
> >
> > Regarding SemiJoin and EquiJoin. Maybe we could deprecate them, or maybe
> > we could convert them to interfaces. I’ll leave that decision to whoever
> > actually writes the code. If we moved a few things to interfaces (including
> > JoinLike I mentioned earlier) maybe we’d get out of the gridlock caused by
> > the type hierarchy.
> >
> > Regarding when to decorrelate. Decorrelation during sql-to-rel is legacy.
> > We now prefer to decorrelate using rules, in RelNode-land. There may be
> > bugs in the legacy decorrelation and we do not aggressively fix them. We
> > can even start to remove functionality if it helps us make
> > SqlToRelConverter simpler.
> >
> > Julian
> >
> >
> >
> >
> >
> >
> >
> >
> > > On Mar 25, 2019, at 12:23 PM, Haisheng Yuan 
> > wrote:
> > >
> > > I agree with Stamatis that JoinRelType should have values:
> > > Inner, Left (Outer), Full (Outer), Semi, Anti.
> > > The option of right outer join is not necessary, because we can flip the
> > inner/outer to left outer join.
> > >
> > > SemiJoin and EquiJoin can be deprecated.
> > >
> > > EnumerableCorrelate is confusing, correlate is a logical concept, better
> > to rename it to EnumerableNestLoopJoin. SemiJoin can be implemented by
> > nestloop, hashjoin or mergejoin.
> > > I don’t see the necessity of having a separate physical operator
> > EnumerableSemiJoin.
> > > But these are minor naming issue.
> > >
> > > Regarding the LogicalCorrelate, I view it as a kind of operator similar
> > to LogicalApply [1], which is
> > > the logical operator in Microsoft SQL Server and Greenplum Orca
> > optimizer. Both uses LogicalApply
> > > operator to represent the correlated join that inner has reference to
> > the outer variable. The apply may
> > > have different type: cross apply (or inner apply), outer apply, semi
> > apply, anti-semi apply. They are
> > > just subset of join types, maybe it is why it is acciociated with
> > JoinRelType, or reuse. The main
> > > difference between Correlate (or Apply) and Join is (logically
> > speaking): In Correlate, inner has
> > > reference to outer. In Join, inner doesn’t reference outer. NestLoopJoin
> > can implement both.
> > >
> > > With optimizer transformation rules, Correlate (or Apply) can be
> > transformed into a Join, or a Join
> > > is transfomed into a Correlate (Or Apply), in case there is an index can
> > be used in inner relation.
> > >
> > > What I am not comfortable with is:
> > > In SQL Server and GPDB Orca optimzier, Sql is translated into logical
> > relation as it should be (
> > > keep subquery as it is), then use all kinds of apply rules to unnest
> > subqueries based on cost model,
> > > which seems reasonable to me.
> > > But in Calcite, we can not only decorrelate in SqlToRel stage, but also
> > can do it in SubqueryRemoveRule.
> > > Should we unify them all in the rules and keep SqlToRelConverter simple?
> > >
> > > Thanks ~
> > > Haisheng Yuan
> > > --
> > > 发件人：Stamatis Zampetakis
> > > 日 期：2019年03月23日 07:31:35
> > > 收件人：
> > > 主 题：Re: Join, SemiJoin, Correlate
> > >
> > > Since we are discussing this topic I thought it would be could to bring
> > > back
> > > to the surface a similar discussion [1] that has been done a few years
> > ago
> > > in this list.
> > >
> > > I am leaning towards option 3 where JoinRelType has all necessary values:
> > > Inner, Left, Semi, Anti, and Full.
> > > With these changes it seems we could remove (deprecate) also SemiJoin,
> > and
> > > EquiJoin.
> > >
> > > On the physical level we could have:
> > > 1. EnumerableCorrelate or EnumerableNestedLoopJoin;
> > > 2. EnumerableMergeJoin;
> > > 3. EnumerableHashJoin (currently EnumerableJoin)
> > >
> > > and for the above we could pass the JoinRelType throwing an exception
> > when
> > > the specific algorithm cannot be used to implement a specific type of
> > join.
> > >
> > > EnumerableSemiJoin and EnumerableThetaJoin could also be removed and
> > > covered from the above I think.
> > >
> > > Regarding Correlate and LogicalCorrelate, I am not sure what should we
> > do.
> > 

Re: About aggregation problem

[Yuzhao Chen]

Select count(distinct f0) from t
Is supported now, can you please past your error stack trace, is it the parse 
error or plan promotion error ?

Best,
Danny Chan
在 2019年4月2日 +0800 AM11:31，kakasi2...@163.com ，写道：
> hello，
>
> The problem is about use ［count（distinct（ID））］，distinct aggregation 
> notsupported. Does the community have a plan to solve the problem，or have a 
> temporary solution ?
>
>
> thank you

Re: elasticsearch-adapter test failed with "Object 'test'" not found

[Yuzhao Chen]

For Object 'test' not found it means Calcite does not load your son model 
correctly, so it can not find then validate the table there. Maybe you model 
file is not load correctly.

I saw a demo in Calcite code in [1], the path sep is  while your sep is 
<\\>, can you have a try ?

[1] 
https://github.com/apache/calcite/blob/a75a689eff2f1333adc8fb800bdfa077e94da562/core/src/test/java/org/apache/calcite/test/LatticeTest.java#L914

Best,
Danny Chan
在 2019年4月2日 +0800 PM8:33，Maria ，写道：
> Hi, calcite developers. I met a question about es-adapter and still cannot 
> resove it: .
>
>
> 1) calcite version :1.18.0
> 2) ES version : 5.4.1（I know that in calcite-1.18.0 is 6.4.2）
> 3) test sql : select count(*) from test
> 4) elasticsearch-model.json as follow:
> {
> "version": "1.0","defaultSchema": "elasticsearch", "schemas": [ { "type": 
> "custom", "name": "test", "factory": 
> "org.apache.calcite.adapter.elasticsearch.ElasticsearchSchemaFactory", 
> "operand": {
> "coordinates": "{'xx.xx.xx.xx': 9205}",
> "userConfig": "{'bulk.flush.max.actions': 10, 'bulk.flush.max.size.mb': 1}",
> "index": "test" } } ] }
> 5) test-code:
> try {
> Class.forName("org.apache.calcite.jdbc.Driver");
> } catch (ClassNotFoundException e) {
> Throw new RuntimeException(e);
> }
> Properties pp = new Properties();
> pp.put("lex", "JAVA");
> Connection conn = null;
> try {
> conn = 
> DriverManager.getConnection("jdbc:calcite:model=D:\\calcite\\pre-git\\calcite-test\\conf\\elasticsearch-model.json",
>  pp);
> Statement ss = conn.createStatement();
> String sql = "select count(*) from test";
> ResultSet result = ss.escuteQuery(sql);
> output(resultSet, System.out);
> } catch() {
> Throw new RuntimeException("==get data failed" , e);
> }
>
>
> 6) excaptions :
> org.apache.calcite.sql.validate.SqlValidatorException:Object 'test' not found
> Exception in thread "main" java.lang.RuntimeException: ==get data failed
> .
> caused by:org.apache.calcite.runtime.CalciteContestException: From line 1, 
> column 22 to line 1, column 25: Object 'test' not found<4 internal calls>
> at org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:463)
> at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:787)
> at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:772)
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.newValidationError(SqlValidatorImpl.java:4788)
> at 
> org.apache.calcite.sql.validate.IdentifierNamespace.resolveImpl(IdentifierNamespace.java:172)
> .
> 7) other confusions:
> [1] I know that in elasticsearch apdator, test case is done by 
> EmbeddedElasticsearch, why not add one standard JDBC test case?
> [2] model.json in http://calcite.apache.org/docs/elasticsearch_adapter.html 
> need to be update,should add "userConfig", or else will be failed.
>
>
>
>
> any reply will be appreciated
>
>
> Maria.
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>

Re: Unable to use default as database name

[YuZhao Chen]

Calcite's parser use JavaCC and the built in keywords are configured
hard-code, so there is no way to change this. An quote character may be a
better choice.

Jeff Zhang 于2019年4月2日 周二下午9:15写道：

> Thanks Amit. I'd like to keep my system compatible with hive sql, but
> unfortunately hive use default as its default database name. Is there any
> way to customize calcite's parser to allow user to use default ?
>
> Amit Weitzner  于2019年4月2日周二 下午6:36写道：
>
> > Hi Jeff,
> >
> > "default" is a reserved keyword in calcite parser.
> >
> > If you'll add quoting to default it will work:
> > select * from "default".table_1
> > select * from `default`.table_1
> >
> > The quotation marks depend on your configuration in the calcite parser
> (the
> > default is ").
> >
> > Hope this helps,
> > Amit
> >
> >
> > On Tue, Apr 2, 2019 at 12:08 PM Jeff Zhang  wrote:
> >
> > > Hi Folks,
> > >
> > > I try to integrate calcite into our product, and found that I can not
> use
> > > default as the database name. (select * from default.table_1)
> > >
> > > I will hit the following error. Is there any way that I can use default
> > as
> > > database name ? Thanks
> > >
> > > org.apache.calcite.sql.parser.SqlParseException: Encountered "from
> > default"
> > > at line 1, column 10.
> > > Was expecting one of:
> > > 
> > > "ORDER" ...
> > > "LIMIT" ...
> > > "OFFSET" ...
> > > "FETCH" ...
> > > "FROM"  ...
> > > "FROM"  ...
> > > "FROM"  ...
> > > "FROM"  ...
> > > "FROM"  ...
> > > "FROM" "LATERAL" ...
> > > "FROM" "(" ...
> > > "FROM" "UNNEST" ...
> > > "FROM" "TABLE" ...
> > > "," ...
> > > "AS" ...
> > >  ...
> > >  ...
> > >  ...
> > >  ...
> > >  ...
> > > "UNION" ...
> > > "INTERSECT" ...
> > > "EXCEPT" ...
> > > "MINUS" ...
> > >
> > >
> > > at
> > >
> > >
> >
> org.apache.calcite.sql.parser.impl.SqlParserImpl.convertException(SqlParserImpl.java:355)
> > > at
> > >
> > >
> >
> org.apache.calcite.sql.parser.impl.SqlParserImpl.normalizeException(SqlParserImpl.java:143)
> > > at
> org.apache.calcite.sql.parser.SqlParser.parseQuery(SqlParser.java:156)
> > >
> > > --
> > > Best Regards
> > >
> > > Jeff Zhang
> > >
> >
>
>
> --
> Best Regards
>
> Jeff Zhang
>

Re: expose existing materialized views from postgres to calcite

[Yuzhao Chen]

Or you can config your framework config with a schema implemented by your self 
[1], the getTable should be implemented correctly [2]. Then config the 
CalciteCatalogReader with the schema and schemaPath [3].
The validator used your CalciteCatalogReader will recognize your tables in 
Postgres.

[1] 
https://github.com/apache/calcite/blob/a8e71f9f295e36e472d9f650d1c76d2501e1fdbf/core/src/main/java/org/apache/calcite/tools/FrameworkConfig.java#L55
[2] 
https://github.com/apache/calcite/blob/a8e71f9f295e36e472d9f650d1c76d2501e1fdbf/core/src/main/java/org/apache/calcite/schema/Schema.java#L63
[3] 
https://github.com/apache/calcite/blob/a8e71f9f295e36e472d9f650d1c76d2501e1fdbf/core/src/main/java/org/apache/calcite/prepare/CalciteCatalogReader.java#L100

Best,
Danny Chan
在 2019年4月5日 +0800 PM10:00，Mark Pasterkamp ，写道：
> Dear all,
>
> I have connected my postgresql database as a datasource to calcite. I am
> however not able to find out how I can expose the existing materialized
> views in postgres to calcite. I am unable to find much help on the internet
> so if anyone is knowledgeable about this I would really appreciate a small
> code example as to how this is supposed to work in Java.
>
>
> Mark

Re: Welcome to join China User WeChat Group of Calcite

[Yuzhao Chen]

That’s awesome, good news for Chinese  Calcite users, thx for your work, Lai 
Zhou.

Best,
Danny Chan
在 2019年4月9日 +0800 AM11:53，Lai Zhou ，写道：
> hi, guys:
>  I create a WeChat group  for  China Users of Apache Calcite, to make it easy 
> to communicate about Calcite. Now we have 2  committers in the group ,   if 
> you have problems about calcite , join us, let's communicate and discuss here.
>
> In Chinese :
>  
> 为了更方便Calcite中文用户讨论，我建了一个Calcite微信讨论群，现在群里已经有2位活跃的国内committer了，如果大家有问题需要交流讨论，可以加入我们的群组。
> <>
>

Re: Join, SemiJoin, Correlate

[Yuzhao Chen]

Hi, @Haisheng Yuan, @Julian Hyde, @Stamatis Zampetakis, @Walaa Eldin Moustafa

I have did the work for this discussion, and look forward to your suggestions.


### Diff
- Deprecate SemiJoin, EquiJoin, EnumerableSemiJoin, SemiJoinType, 
EnumerableSemiJoinRule, EnumerableThetaJoin
- Make EnumerableMergeJoin extends Join instead of EquiJoin
- Add SEMI and ANTI join type to JoinRelType, add method 
returnsJustFirstInput() to decide if the join only outputs left side
- Correlate use JoinRelType instead of SemiJoinType
- Rename EnumerableCorrelate to EnumerableNestedLoopJoin and make it exptends 
Join instead of Correlate
- Rename EnumerableJoin to EnumerableHashJoin
- EnumerableJoinRule will convert semi-join to EnumerableNestedLoopJoin 
(EnumerableSemiJoin's function is merged into this rule)
- Add method isNonCorrelateSemiJoin() in Join.java to make sure if this join is 
a semi-join (Comes from SemiJoinRule) or comes from 
decorrelation(SubqueryRemoveRule or RelDecorrelator), the returns value true 
means the join is a semi-join equivalent to SemiJoin before this patch.
- Cache the JoinInfo in Join and use it to get leftKeys and rightKeys, merge 
the SemiJoin#computeSelfCost to Join#computeSelfCost
- RelBuilder removes SemiJoinFactory, method #semiJoin now return a LogicalJoin 
with JoinRelType#SEMI

### Rules tweak
- JoinAddRedundantSemiJoinRule now create LogicalJoin with JoinRelType#SEMI 
instead of SemiJoin
- JoinToCorrelateRule remove SEMI instance and change the matchs condition to 
!join.getJoinType().generatesNullsOnLeft() which also allowed ANTI compared 
before this patch.
- SemiJoinRule match SEMI join specificlly

### Metadata tweak
- RelMdAllPredicates, RelMdExpressionLineage: Add full rowType to 
getAllPredicates(Join) cause semi-join only outputs one side
- RelMdColumnUniqueness, RelMdSelectivity, RelMdDistinctRowCount, RelMdSize, 
RelMdUniqueKeys: merge semi-join logic to join


### Test cases change
- MaterializationTest#testJoinMaterialization11 now can materialize 
successfully, cause i allow logical SemiJoin node to match, the original matchs 
SemiJoin as SemiJoin.class.isAssignableFrom(), which i think is wrong cause 
this will only matches subClasses of SemiJoin which is only EnumerableSemiJoin 
before this patch.
- SortRemoveRuleTest#removeSortOverEnumerableCorrelate, because CALCITE-2018, 
the final EnumerableSort's cost was cache by the previous EnumerableSort with 
logical childs, so i remove the EnumerableSortRule and the best plan is correct
- sub-query.iq has better plan for null correlate



Best,
Danny Chan
在 2019年3月21日 +0800 AM3:07，Julian Hyde ，写道：
> I just discovered that Correlate, which is neither a Join nor a SemiJoin, 
> uses SemiJoinType, but SemiJoin does not use SemiJoinType.
>
> Yuck. The Join/SemiJoin/Correlate type hierarchy needs some thought.
>
> Julian
>
>

Re: [DISCUSS] Towards Avatica 1.14.0

[Yuzhao Chen]

I will help to review these codes.

Best,
Danny Chan
在 2019年4月12日 +0800 AM6:48，dev@calcite.apache.org，写道：
>
> CALCITE-2983

Re: [DISCUSS] Towards Avatica 1.14.0

[Yuzhao Chen]

Francis Chuang, i have reviewed these 3 PRs and add some comments, hope can 
make some help for you.

Best,
Danny Chan
在 2019年4月12日 +0800 AM6:48，Francis Chuang ，写道：
> Just a quick update on the release process. I am planning to make rc0
> available for voting next Friday (19/4/2019) if possible.
>
> I know Kevin would like to get CALCITE-2983 in, so the timeline isn't
> fixed and we can push rc0 out a little bit to get it in.
>
> In the meantime, can someone please review (and merge, if they are
> ready) the following PRs?
> - https://github.com/apache/calcite-avatica/pull/94
> - https://github.com/apache/calcite-avatica/pull/86
> - https://github.com/apache/calcite-avatica/pull/85
>
> Francis
>
> On 9/04/2019 8:43 am, Francis Chuang wrote:
> > Many thanks to reviewers for merging PRs and Kevin for upgrading Jetty.
> > There's still CALCITE-2983 [1], but I think I can make rc0 available for
> > voting soon.
> >
> > In the meantime, I'd like to get CALCITE-2882 in if I can hear back from
> > the author: https://github.com/apache/calcite-avatica/pull/86
> >
> > Vladimir, can you please take a look at Stamatis' review:
> > https://github.com/apache/calcite-avatica/pull/85
> >
> > Francis
> >
> > [1] https://issues.apache.org/jira/browse/CALCITE-2983
> >
> > On 29/03/2019 8:41 am, Francis Chuang wrote:
> > > Would anyone be interested in finishing CALCITE-2322
> > > (https://github.com/apache/calcite-avatica/pull/49)?
> > >
> > > There were some minor changes that were requested, but the original
> > > author has disappeared. There does not appear to be any tests for the
> > > change, but I am not sure how much work is required to finish the PR.
> > >
> > > A few comments about the other PRs:
> > > - https://github.com/apache/calcite-avatica/pull/89 has been updated
> > > and is ready for review
> > > - https://github.com/apache/calcite-avatica/pull/86 is ready for
> > > review, but might require more investigation
> > > - https://github.com/apache/calcite-avatica/pull/85 no tests, but
> > > maybe the change is not easy to test?
> > >
> > > Kevin: Do you think these can make it into the release?
> > > - https://github.com/apache/calcite-avatica/pull/28
> > > - https://github.com/apache/calcite-avatica/pull/36
> > >
> > > Francis
> > >
> > > On 27/03/2019 1:03 pm, Yuzhao Chen wrote:
> > > > Thx for your work Francis, the PRs are not too many, I think the
> > > > timeline is ok.
> > > >
> > > > Best,
> > > > Danny Chan
> > > > 在 2019年3月27日 +0800 AM5:19，Francis Chuang
> > > > ，写道：
> > > > > I am currently planning to make rc0 available for voting next Friday 
> > > > > (5
> > > > > April 2019). How does this sound? Does this leave enough time for
> > > > > PRs to
> > > > > get reviewed and other changes to be committed?
> > > > >
> > > > > If the proposed timeline is too tight, please let me know.
> > > > >
> > > > > Francis
> > > > >
> > > > > On 26/03/2019 8:54 am, Julian Hyde wrote:
> > > > > > Thank you for stepping up! The release process - especially for
> > > > > > Calcite, but also for Avatica and Avatica-Go - seems to be getting
> > > > > > heavier these days, and it seems to be good to keep up momentum.
> > > > > >
> > > > > > I’ll review two or three of those PRs. I hope some other committers
> > > > > > will step forward and take a few.
> > > > > >
> > > > > > Julian
> > > > > >
> > > > > >
> > > > > > > On Mar 25, 2019, at 2:43 PM, Francis Chuang
> > > > > > >  wrote:
> > > > > > >
> > > > > > > Hey all,
> > > > > > >
> > > > > > > Now that Kevin has finalized Calcite 1.19.0, I want to talk about
> > > > > > > Avatica.The last release of Avatica was around 4 months ago and I
> > > > > > > am happy to be release manager for 1.14.0.
> > > > > > >
> > > > > > > There are around 10 unreleased commits and 6 pretty fresh PRs that
> > > > > > > I want to get into this release:
> > > > > > > - https://github.com/apache/calcite-avatica/pull/82
> > > > > > > - https://github.com/apache/calcite-avatica/pull/85
> > > > > > > - https://github.com/apache/calcite-avatica/pull/86
> > > > > > > - https://github.com/apache/calcite-avatica/pull/88
> > > > > > > - https://github.com/apache/calcite-avatica/pull/89
> > > > > > > - https://github.com/apache/calcite-avatica/pull/90
> > > > > > >
> > > > > > > If anyone has spare cycles to review these PRs, please review
> > > > > > > them, so we can get them in.
> > > > > > >
> > > > > > > I would like to get 1.14.0 out before the next Calcite release, so
> > > > > > > that Calcite can use 1.14.0. Are there any others issues you guys
> > > > > > > would like to be fixed?
> > > > > > >
> > > > > > > Once Avatica 1.14.0 is released, I will follow it with a release
> > > > > > > of Avatica-Go.
> > > > > > >
> > > > > > > Francis
> > > > > >
> > > >

Re: Join, SemiJoin, Correlate

[YuZhao Chen]

Stamatis and Haisheng Yuan


I did rename EnumerableCorrelate and EnumerableJoin, your suggestions
are reasonable, I would add them back and mark as deprecated.


For Correlate and LogicalCorrelate i already kept the old constructor and
method #create().


I will recheck if there are some APIs that are missing.


Really thx.


Best,

Danny Chan

Re: Join, SemiJoin, Correlate

[YuZhao Chen]

 Thx, Stamatis and Haisheng Yuan

I did rename EnumerableCorrelate and EnumerableJoin, your suggestions
are reasonable, I
would add them back and mark as deprecated.

For Correlate and LogicalCorrelate i already keep the old constructor and
method #create().

I will recheck if there are some APIs that are missing.

Really thx.

Best,
Danny Chan
在 2019年4月14日 +0800 AM5:37，dev@calcite.apache.org，写道：


Stamatis

Re: Join, SemiJoin, Correlate

[Yuzhao Chen]

I also have this question when i was doing this patch, cause Enumerable nodes 
are physical operators(implementation) and should be specific to Calcite. So in 
the beginning i didn’t classify them as public APIs and only keep the 
constructors and methods for logical nodes.

Well this searching is somewhat convincing, there are many interfaces/methods 
are marked as deprecated and to be removed before 2.0, it would be better if 
the principles for backward compatibility are more clear.

Best,
Danny Chan
在 2019年4月14日 +0800 PM7:41，Hongze Zhang ，写道：
> I didn't take look on the PR in detail so far, but it seems that a topic 
> about backward compatibility would be worth to discuss anyway.
>
> Regarding the Enumerable's API, I don't think there are many use cases of 
> them from Calcite users. Although users may create instances in some custom 
> rules, or extend the Enumerable rels' classes to implement some specific 
> behaviors, I am still not sure if such cases are that usual.
>
> For example, I've run a Google search for term "EnumerableJoin.create" on 
> github.com, only 2 results returned[1], and both are from apache/calcite 
> project. Similar result on "EnumerableCorrelate.create". I am pretty sure 
> that Google could not give a precise result about code usage (I don't find a 
> way to search these terms using GitHub code search), but at least it shows 
> some sort of trend. As a comparison there are 33 results[2] for 
> "LogicalJoin.create", some are from external projects.
>
> So my question would be: how much backward compatibility should we respect 
> when we make API changes to Calcite? To me it is not much clear. I know 
> compatibility is very, very important for an Apache project (see "The Apache 
> Project Maturity Model/QU40"[3]), but I am not sure if we should add 
> "@Depracated" to any changed public staffs, the code will be messy and hard 
> to understand.
>
> Anyway my example about EnumerableJoin/Correlate just shows my confusion on a 
> broader topic. So I will be +1 to the consensus that already be achieved so 
> far. But I'll be happy to hear more principles on how to manage the backward 
> compatibility for Calcite, such as: what's the definition about Calcite's 
> public API, or what changes would be considered backward-incompatible, etc. I 
> think that will also benefit our developers a lot.
>
>
> Best,
> Hongze
>
>
> [1] 
> https://www.google.com/search?q=%22EnumerableJoin.create%22+site%3A%3Ahttps%3A%2F%2Fgithub.com
> [2] 
> https://www.google.com/search?q=%22LogicalJoin.create%22+site%3A%3Ahttps%3A%2F%2Fgithub.com
> [3] 
> https://community.apache.org/apache-way/apache-project-maturity-model.html#quality
>
> > On Apr 14, 2019, at 14:53, Walaa Eldin Moustafa  
> > wrote:
> >
> > Agreed, but not sure what would the best way to do it be without
> > making the code very confusing.
> >
> > On Sat, Apr 13, 2019 at 2:46 PM Haisheng Yuan  
> > wrote:
> > >
> > > I share the same concern with you.
> > >
> > >
> > >
> > >
> > >
> > > Thanks~
> > > Haisheng 
> > > Yuan--
> > > 发件人：Stamatis Zampetakis
> > > 日　期：2019年04月14日 05:37:29
> > > 收件人：
> > > 主　题：Re: Join, SemiJoin, Correlate
> > >
> > > Hi Danny,
> > >
> > > Thanks a lot for taking this on, it is a great start!
> > >
> > > I didn't look thoroughly through the PR but I noticed that there are many
> > > renaming/refactoring of public APIs. I am not sure if we should introduce
> > > so many breaking changes without prior notice. A most conservative 
> > > approach
> > > would be to keep existing classes/methods, mark them as deprecated, and
> > > then remove them in one of the coming releases. I am not sure if that is
> > > the right way to go so let's see what the others have to say.
> > >
> > > Best,
> > > Stamatis
> > >
> > > On Fri, Apr 12, 2019 at 9:18 AM Yuzhao Chen  wrote:
> > >
> > > > Hi, @Haisheng Yuan, @Julian Hyde, @Stamatis Zampetakis,
> > > > @Walaa Eldin Moustafa
> > > >
> > > > I have did the work for this discussion, and look forward to your
> > > > suggestions.
> > > >
> > > >
> > > > ### Diff
> > > > - Deprecate SemiJoin, EquiJoin, EnumerableSemiJoin, SemiJoinType,
> > > > EnumerableSemiJoinRule, EnumerableThetaJoin
> > > > - Make EnumerableMergeJoin ex

What is the exactly definition as an equi join ?

[Yuzhao Chen]

If we checkout the java doc for Calcite EuqiJoin, there is definition for it:
> for any join whose condition is based on column equality

But what about if there are function calls in the equi condition operands ? For 
example:
Should we consider

Select A.a, B.b from A join B on cast(A.a as int) = B.b

as an equi join ?

Now Calcite think it is not, which I think will lost some possibilities for sql 
plan promotion, e.g. join condition push down.

Best,
Danny Chan

Re: What is the exactly definition as an equi join ?

[Yuzhao Chen]

For the example I gave, the join condition is push down with 
RelOptUtil.pushDownJoinConditions(), but when we new a join info [1], we only 
decide if the join is EquiJoin if all the operands of operator <=> is inputRef, 
which is the question i ask here.

[1] 
https://github.com/apache/calcite/blob/9538374e8fae5cec7d6f7b270850f5dfb4c1fc06/core/src/main/java/org/apache/calcite/plan/RelOptUtil.java#L1435

Best,
Danny Chan
在 2019年4月15日 +0800 PM7:25，Xiening Dai ，写道：
> I think Calcite always pushes down equal join conditions. In 
> SqlToRelConverter.createJoin(), before ruction returns, it calls 
> RelOptUtil.pushDownJoinConditions(). So in your example, the cast expression 
> will be pushed down and it will still be an equal join.
>
> > On Apr 15, 2019, at 5:40 PM, Yuzhao Chen  wrote:
> >
> > If we checkout the java doc for Calcite EuqiJoin, there is definition for 
> > it:
> > > for any join whose condition is based on column equality
> >
> > But what about if there are function calls in the equi condition operands ? 
> > For example:
> > Should we consider
> >
> > Select A.a, B.b from A join B on cast(A.a as int) = B.b
> >
> > as an equi join ?
> >
> > Now Calcite think it is not, which I think will lost some possibilities for 
> > sql plan promotion, e.g. join condition push down.
> >
> > Best,
> > Danny Chan
>

Re: What is the exactly definition as an equi join ?

[Yuzhao Chen]

Thx Ruben, the issue really answer my questions, I encounter this when dong 
CALCITE-2969, when I refactor SemiJoinRule, I think not only RexFieldAccess, 
any RexCall should fit into this case, only if the RexCall function is 
deterministic, what do you think ?

Best,
Danny Chan
在 2019年4月15日 +0800 PM7:48，Ruben Q L ，写道：
> Danny,
> In the context of https://issues.apache.org/jira/browse/CALCITE-2898, a
> discussion about this topic was started. In that ticket I pointed out that
> Calcite does not recognize "RexFieldAccess = RexInputRef" as an EquiJoin
> condition (even though the RexFieldAccess itself is referencing a
> RexInputRef); which is somewhat similar to the situation that you propose
> "RexCall = RexInputRef". According to Julian Hyde's comment on that
> ticket: *'For
> our purposes, an equi-join is "field = field", not "expression = field".
> Even if that expression is a reference to sub-field'. *However, I agree
> with you and maybe this definition should be reviewed (I believe your
> example and my example should be valid cases of EquiJoin), but possibly
> this will break some pieces of the current code, so the modification might
> not be straightforward.
>
> Best,
> Ruben
>
>
> Le lun. 15 avr. 2019 à 13:25, Xiening Dai  a écrit :
>
> > I think Calcite always pushes down equal join conditions. In
> > SqlToRelConverter.createJoin(), before ruction returns, it calls
> > RelOptUtil.pushDownJoinConditions(). So in your example, the cast
> > expression will be pushed down and it will still be an equal join.
> >
> > > On Apr 15, 2019, at 5:40 PM, Yuzhao Chen  wrote:
> > >
> > > If we checkout the java doc for Calcite EuqiJoin, there is definition
> > for it:
> > > > for any join whose condition is based on column equality
> > >
> > > But what about if there are function calls in the equi condition
> > operands ? For example:
> > > Should we consider
> > >
> > > Select A.a, B.b from A join B on cast(A.a as int) = B.b
> > >
> > > as an equi join ?
> > >
> > > Now Calcite think it is not, which I think will lost some possibilities
> > for sql plan promotion, e.g. join condition push down.
> > >
> > > Best,
> > > Danny Chan
> >
> >

Re: What is the exactly definition as an equi join ?

[Yuzhao Chen]

Now many compute engine do not use Calcite EnumerableXXXs and only use the 
logical node for planning, after all, the Enumerables are implementations are 
only specific to Calcite, I still think Calcite need to give more accurate 
definitions for what equi join is.

Best,
Danny Chan
在 2019年4月16日 +0800 AM12:19，Ruben Q L ，写道：
> Danny,
> I have seen the full picture and I have actually changed mind:
>
> If I am not mistaken, currently the way to make your example (and mine) to
> work as an EquiJoin is using intermediate projections (so that RexCall
> / RexFieldAccess "becomes" RexInputRef):
>
> Select A.a, B.b from A join B on cast(A.a as int) = B.b
>
> => option 1 (analyzed as equijoin)
> Project($0, $2)
> Join(condition: $1 = $2) -- i.e. cast(A.a as int) = B.b
> Project($0=a; $1=cast($0 as int))
> Scan(A)
> Scan(B)
>
> => option 2 (analyzed as non-equijoin)
> Project($0, $1)
> Join(condition: cast($0 as int) = $1) -- i.e. cast(A.a as int) = B.b
> Scan(A)
> Scan(B)
>
> It might seem "wrong", but the thing is, the Enumerable implementations
> that extend EquiJoin (i.e. EnumerableJoin, EnumerableMergeJoin,
> EnumerableSemiJoin) are based on the EquiJoin fields:
> public final ImmutableIntList leftKeys;
> public final ImmutableIntList rightKeys;
>
> And rely on the the fact that they are representing an equality on leftKeys
> and rightKeys field indices, and that we can directly generate accessors
> for these fields without any extra computation (i.e. without any extra
> call). That's the reason why EquiJoin cannot support RexCall
> / RexFieldAccess, because they cannot be translatable to a key (i.e. to a
> field index).
>
> With this situation, we could improve this logic to support more complex
> equijoin conditions; but I think this will not be worth it, because the
> alternative is quite simple: add a projection for the RexCall
> / RexFieldAccess and keep the existing (simple) logic.
>
> For this reason, I think we should stick to the current logic *an equi-join
> is "field = field", not "expression = field" *and I should abandon and
> close https://issues.apache.org/jira/browse/CALCITE-2898
>
> Best,
> Ruben
>
>
> Le lun. 15 avr. 2019 à 14:13, Yuzhao Chen  a écrit :
>
> > Thx Ruben, the issue really answer my questions, I encounter this when
> > dong CALCITE-2969, when I refactor SemiJoinRule, I think not only
> > RexFieldAccess, any RexCall should fit into this case, only if the RexCall
> > function is deterministic, what do you think ?
> >
> > Best,
> > Danny Chan
> > 在 2019年4月15日 +0800 PM7:48，Ruben Q L ，写道：
> > > Danny,
> > > In the context of https://issues.apache.org/jira/browse/CALCITE-2898, a
> > > discussion about this topic was started. In that ticket I pointed out
> > that
> > > Calcite does not recognize "RexFieldAccess = RexInputRef" as an EquiJoin
> > > condition (even though the RexFieldAccess itself is referencing a
> > > RexInputRef); which is somewhat similar to the situation that you propose
> > > "RexCall = RexInputRef". According to Julian Hyde's comment on that
> > > ticket: *'For
> > > our purposes, an equi-join is "field = field", not "expression = field".
> > > Even if that expression is a reference to sub-field'. *However, I agree
> > > with you and maybe this definition should be reviewed (I believe your
> > > example and my example should be valid cases of EquiJoin), but possibly
> > > this will break some pieces of the current code, so the modification
> > might
> > > not be straightforward.
> > >
> > > Best,
> > > Ruben
> > >
> > >
> > > Le lun. 15 avr. 2019 à 13:25, Xiening Dai  a écrit :
> > >
> > > > I think Calcite always pushes down equal join conditions. In
> > > > SqlToRelConverter.createJoin(), before ruction returns, it calls
> > > > RelOptUtil.pushDownJoinConditions(). So in your example, the cast
> > > > expression will be pushed down and it will still be an equal join.
> > > >
> > > > > On Apr 15, 2019, at 5:40 PM, Yuzhao Chen 
> > wrote:
> > > > >
> > > > > If we checkout the java doc for Calcite EuqiJoin, there is definition
> > > > for it:
> > > > > > for any join whose condition is based on column equality
> > > > >
> > > > > But what about if there are function calls in the equi condition
> > > > operands ? For example:
> > > > > Should we consider
> > > > >
> > > > > Select A.a, B.b from A join B on cast(A.a as int) = B.b
> > > > >
> > > > > as an equi join ?
> > > > >
> > > > > Now Calcite think it is not, which I think will lost some
> > possibilities
> > > > for sql plan promotion, e.g. join condition push down.
> > > > >
> > > > > Best,
> > > > > Danny Chan
> > > >
> > > >
> >

Re: Join, SemiJoin, Correlate

[Yuzhao Chen]

Thx Ruben, very appreciate for your suggestions, I will make modifications soon.

The main diff it to move semiJoin physical implementations to 
EnumerableHashJoin, i also see that now we use method `contains` to probe the 
right side(inner loop), but there also 2 implementations for `contains`
One is Hash probe and one is loop look up, I’m not very sure it is indeed a 
hash-semi or nestedLoop-semi.

[1] 
https://github.com/apache/calcite/blob/b03cdc486cf5c7232bbc6fa9b5f02f564e9601c3/linq4j/src/main/java/org/apache/calcite/linq4j/EnumerableDefaults.java#L1312

Best,
Danny Chan
在 2019年4月18日 +0800 AM1:01，Ruben Q L ，写道：
> Hi Danny,
> thanks for kicking off this big task, I have started to add some comments
> in the PR
>
> Best regards,
> Ruben
>
>
> Le lun. 15 avr. 2019 à 20:49, Julian Hyde  a écrit :
>
> > I haven’t reviewed the change, but let’s strike a balance between
> > backwards compatibility and progress. We should allow breaking changes if
> > they make a significant improvement. Of course we should mark APIs
> > deprecated for one or two releases, if possible.
> >
> > > On Apr 15, 2019, at 2:01 AM, Yuzhao Chen  wrote:
> > >
> > > I also have this question when i was doing this patch, cause Enumerable
> > nodes are physical operators(implementation) and should be specific to
> > Calcite. So in the beginning i didn’t classify them as public APIs and only
> > keep the constructors and methods for logical nodes.
> > >
> > > Well this searching is somewhat convincing, there are many
> > interfaces/methods are marked as deprecated and to be removed before 2.0,
> > it would be better if the principles for backward compatibility are more
> > clear.
> > >
> > > Best,
> > > Danny Chan
> > > 在 2019年4月14日 +0800 PM7:41，Hongze Zhang ，写道：
> > > > I didn't take look on the PR in detail so far, but it seems that a
> > topic about backward compatibility would be worth to discuss anyway.
> > > >
> > > > Regarding the Enumerable's API, I don't think there are many use cases
> > of them from Calcite users. Although users may create instances in some
> > custom rules, or extend the Enumerable rels' classes to implement some
> > specific behaviors, I am still not sure if such cases are that usual.
> > > >
> > > > For example, I've run a Google search for term "EnumerableJoin.create"
> > on github.com, only 2 results returned[1], and both are from
> > apache/calcite project. Similar result on "EnumerableCorrelate.create". I
> > am pretty sure that Google could not give a precise result about code usage
> > (I don't find a way to search these terms using GitHub code search), but at
> > least it shows some sort of trend. As a comparison there are 33 results[2]
> > for "LogicalJoin.create", some are from external projects.
> > > >
> > > > So my question would be: how much backward compatibility should we
> > respect when we make API changes to Calcite? To me it is not much clear. I
> > know compatibility is very, very important for an Apache project (see "The
> > Apache Project Maturity Model/QU40"[3]), but I am not sure if we should add
> > "@Depracated" to any changed public staffs, the code will be messy and hard
> > to understand.
> > > >
> > > > Anyway my example about EnumerableJoin/Correlate just shows my
> > confusion on a broader topic. So I will be +1 to the consensus that already
> > be achieved so far. But I'll be happy to hear more principles on how to
> > manage the backward compatibility for Calcite, such as: what's the
> > definition about Calcite's public API, or what changes would be considered
> > backward-incompatible, etc. I think that will also benefit our developers a
> > lot.
> > > >
> > > >
> > > > Best,
> > > > Hongze
> > > >
> > > >
> > > > [1]
> > https://www.google.com/search?q=%22EnumerableJoin.create%22+site%3A%3Ahttps%3A%2F%2Fgithub.com
> > > > [2]
> > https://www.google.com/search?q=%22LogicalJoin.create%22+site%3A%3Ahttps%3A%2F%2Fgithub.com
> > > > [3]
> > https://community.apache.org/apache-way/apache-project-maturity-model.html#quality
> > > >
> > > > > On Apr 14, 2019, at 14:53, Walaa Eldin Moustafa 
> > > > > 
> > wrote:
> > > > >
> > > > > Agreed, but not sure what would the best way to do it be without
> > > > > making the code very confusing.
> > > > >
&g

Re: Using Calcite for JDBC SQL optimizer without execution

[Yuzhao Chen]

Andrew, I’m not very sure if I got your idea right.

Do you want a promoted `Sql` compared  to the original ? Then you should 
transform Calcite’s RelNodes tree to sql, but Calcite do not support this yet, 
The SqlNode unparse to sql is actually supported.

It is not that equivalent for RelNodes tree and sql text, e.g. How to describe 
a SemiJoin in sql ? Another question is the pure text to data source will 
actually have another planning phrase, we can not make sure this plan is the 
best and efficient.

Best,
Danny Chan
在 2019年4月18日 +0800 AM5:22，Andrew O ，写道：
> I'm interested in using Calcite in a "no execute" mode that would
> effectively return the optimized SQL but not actually execute it. The
> intention would be to leverage a) the query planning / optimization b) the
> support for outputting different SQL dialects.
>
> I can see some Tests (e.g. JdbcTest.java) use CalciteAssert and that it has
> a check "planHasSql" function. This seems to use a Hook to capture the sql
> as part of the execution flow, so is not quite what I'm trying to do.
>
> I appreciate Calcite attempts to provide abstractions for different
> execution layers to allow cross adaptor function, so in theory a single SQL
> statement may not always be available. However in my context I know it
> will be executing only against a single JDBC source and not other types.
>
> Is there any suggested / recommended approaches to this, or pointers to
> bits of code to look at?
>
> Thanks in advance
>
> Andrew

How to traverse RelNode’s parent conviniently?

[Yuzhao Chen]

Now for RelNode, we have method getInput()[1]  to fetch the input RelNodes, but 
how we fetch the parent ?

For example, we have plan:

      join-rel
    /             \
scan1     scan2


We can get scan1 and scan2 in join-rel directly with  method getInput, but how 
can we get the join rel in scan1 and scan 2 ?

I know that there is a RelShuttle that can visit every RelNode and if I make a 
cache for the inputs mapping, finally I can get the ‘parents’ from the cache, 
but this is boring code and not that intuitive.

Do you guys have any good ideas ?

[1] 
https://github.com/apache/calcite/blob/ee83efd360793ef4201f4cdfc2af8d837b76ca69/core/src/main/java/org/apache/calcite/rel/RelNode.java#L132


Best,
Danny Chan

Support non-equi join condition for EnumerableJoin, EnumerableMergeJoin and EnumerableSemiJoin ? Just a discussing..

[Yuzhao Chen]

EnumerableJoin EnumerableMergeJoin EnumerableSemiJoin all only support 
equi-join condition[1], and there are some
rules that match EquiJoin specificly, like FilterJoinRule will not push 
non-equi join predicates into the join condition[2], so what is the original 
intention that these join enumetables only have equi-join implementation ?

Do you think there is necessity to support non-join conditions for these joins 
? I just found that there is a PR to support a hash-theta-join for theta join 
with equi-join condition[3].


[1] 
https://github.com/apache/calcite/blob/ee83efd360793ef4201f4cdfc2af8d837b76ca69/linq4j/src/main/java/org/apache/calcite/linq4j/EnumerableDefaults.java#L1125
[2] 
https://github.com/apache/calcite/blob/ee83efd360793ef4201f4cdfc2af8d837b76ca69/core/src/main/java/org/apache/calcite/rel/rules/FilterJoinRule.java#L165
[3] https://github.com/apache/calcite/pull/1156

Best,
Danny Chan

Re: How to traverse RelNode’s parent conviniently?

[Yuzhao Chen]

Thx, Stamatis, that somehow make sense, if i pass around the parent node every 
time I visit a RelNode and keep the parents in the cache, but it is still not 
that intuitive. Actually I what a to add a new RelTrait which bind to a 
specific scope, for example:

         join-rel(trait1)
       /           \
    join2     join3

Join-rel has a trait trait1, and I want all the children of join-rel can see 
this trait, with Calcite’s default metadata handler, I can only see the trait 
from children nodes(traits propagate from the inputs), and I have no idea how 
to propagate a trait reversely?


Best,
Danny Chan
在 2019年4月22日 +0800 PM8:44，Stamatis Zampetakis ，写道：
> Hi Danny,
>
> Apart from RelShuttle there is also RelVisitor which has a visit method
> that provides the parent [1]. Not sure, if it suits your needs.
>
> Best,
> Stamatis
>
> [1]
> https://github.com/apache/calcite/blob/ee83efd360793ef4201f4cdfc2af8d837b76ca69/core/src/main/java/org/apache/calcite/rel/RelVisitor.java#L43
>
>
> On Mon, Apr 22, 2019 at 2:14 PM Yuzhao Chen  wrote:
>
> > Now for RelNode, we have method getInput()[1] to fetch the input
> > RelNodes, but how we fetch the parent ?
> >
> > For example, we have plan:
> >
> > join-rel
> > / \
> > scan1 scan2
> >
> >
> > We can get scan1 and scan2 in join-rel directly with method getInput, but
> > how can we get the join rel in scan1 and scan 2 ?
> >
> > I know that there is a RelShuttle that can visit every RelNode and if I
> > make a cache for the inputs mapping, finally I can get the ‘parents’ from
> > the cache, but this is boring code and not that intuitive.
> >
> > Do you guys have any good ideas ?
> >
> > [1]
> > https://github.com/apache/calcite/blob/ee83efd360793ef4201f4cdfc2af8d837b76ca69/core/src/main/java/org/apache/calcite/rel/RelNode.java#L132
> >
> >
> > Best,
> > Danny Chan
> >

Re: [DISCUSS] Towards Avatica 1.14.0

[Yuzhao Chen]

Review and accepted.

Best,
Danny Chan
在 2019年4月23日 +0800 PM12:14，Francis Chuang ，写道：
> I am planning to make rc0 available for voting at the end of this week.
>
> If possible, can someone please look at CALCITE-2939? PR available here:
> https://github.com/apache/calcite-avatica/pull/94
>
> Francis
>
> On 12/04/2019 10:01 pm, Francis Chuang wrote:
> > Many thanks for jumping on that, Danny!
> >
> > On 12/04/2019 9:55 pm, Yuzhao Chen wrote:
> > > Francis Chuang, i have reviewed these 3 PRs and add some comments,
> > > hope can make some help for you.
> > >
> > > Best,
> > > Danny Chan
> > > 在 2019年4月12日 +0800 AM6:48，Francis Chuang
> > > ，写道：
> > > > Just a quick update on the release process. I am planning to make rc0
> > > > available for voting next Friday (19/4/2019) if possible.
> > > >
> > > > I know Kevin would like to get CALCITE-2983 in, so the timeline isn't
> > > > fixed and we can push rc0 out a little bit to get it in.
> > > >
> > > > In the meantime, can someone please review (and merge, if they are
> > > > ready) the following PRs?
> > > > - https://github.com/apache/calcite-avatica/pull/94
> > > > - https://github.com/apache/calcite-avatica/pull/86
> > > > - https://github.com/apache/calcite-avatica/pull/85
> > > >
> > > > Francis
> > > >
> > > > On 9/04/2019 8:43 am, Francis Chuang wrote:
> > > > > Many thanks to reviewers for merging PRs and Kevin for upgrading 
> > > > > Jetty.
> > > > > There's still CALCITE-2983 [1], but I think I can make rc0 available
> > > > > for
> > > > > voting soon.
> > > > >
> > > > > In the meantime, I'd like to get CALCITE-2882 in if I can hear back
> > > > > from
> > > > > the author: https://github.com/apache/calcite-avatica/pull/86
> > > > >
> > > > > Vladimir, can you please take a look at Stamatis' review:
> > > > > https://github.com/apache/calcite-avatica/pull/85
> > > > >
> > > > > Francis
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/CALCITE-2983
> > > > >
> > > > > On 29/03/2019 8:41 am, Francis Chuang wrote:
> > > > > > Would anyone be interested in finishing CALCITE-2322
> > > > > > (https://github.com/apache/calcite-avatica/pull/49)?
> > > > > >
> > > > > > There were some minor changes that were requested, but the original
> > > > > > author has disappeared. There does not appear to be any tests for 
> > > > > > the
> > > > > > change, but I am not sure how much work is required to finish the 
> > > > > > PR.
> > > > > >
> > > > > > A few comments about the other PRs:
> > > > > > - https://github.com/apache/calcite-avatica/pull/89 has been updated
> > > > > > and is ready for review
> > > > > > - https://github.com/apache/calcite-avatica/pull/86 is ready for
> > > > > > review, but might require more investigation
> > > > > > - https://github.com/apache/calcite-avatica/pull/85 no tests, but
> > > > > > maybe the change is not easy to test?
> > > > > >
> > > > > > Kevin: Do you think these can make it into the release?
> > > > > > - https://github.com/apache/calcite-avatica/pull/28
> > > > > > - https://github.com/apache/calcite-avatica/pull/36
> > > > > >
> > > > > > Francis
> > > > > >
> > > > > > On 27/03/2019 1:03 pm, Yuzhao Chen wrote:
> > > > > > > Thx for your work Francis, the PRs are not too many, I think the
> > > > > > > timeline is ok.
> > > > > > >
> > > > > > > Best,
> > > > > > > Danny Chan
> > > > > > > 在 2019年3月27日 +0800 AM5:19，Francis Chuang
> > > > > > > ，写道：
> > > > > > > > I am currently planning to make rc0 available for voting next
> > > > > > > > Friday (5
> > > > > > > > April 2019). How does this sound? Does this leave enough time 
> > > > > > > > for
> > > > > > > > PRs to
> > > > > > > > get reviewed and other changes to be c

Re: How to traverse RelNode’s parent conviniently?

[Yuzhao Chen]

Julian,

I want to add hint support for Calcite, the initial idea was to tag a 
RelNode(transformed from a SqlNode with hint) with a hit attribute(or trait), 
then I hope that the children (inputs) of it can see this hint, so to make some 
decisions if it should consume or propagate the hint.

The problem I got here is the trait propagate from inputs from, which is the 
opposite as what I need, can you give some suggestions ? If I use 
MetadataHandler to cache and propagate the hints, how to propagate from parents 
to children ?

Best,
Danny Chan
在 2019年4月23日 +0800 AM3:14，Julian Hyde ，写道：
> TL;DR: RelNodes don’t really have parents. Be careful if you are relying on 
> the parent concept too much. Rely on rules instead.
>
> In the Volcano model, a RelNode doesn’t really have a parent. It might be 
> used in several places. (RelSet has a field ‘List parents’ that is 
> kept up to date as planing progresses. But it’s really for Volcano’s internal 
> use.)
>
> Even if you are not using Volcano, there are reasons to want the RelNode 
> graph to be a dag, so again, a RelNode doesn’t have a unique parent.
>
> RelShuttleImpl has a stack. You can use that to find the parent. But the 
> “parent” is just “where we came from as we traversed the RelNode graph”. 
> There may be other “parents” that you do not know about.
>
> If you have a Project and want to find all parents that are Filters, don’t 
> even think about “iterating over the parents” of the Project. Just write a 
> rule that matches a Filter on a Project, and trust Volcano to do its job.
>
> Julian
>
>
>
>
> > On Apr 22, 2019, at 6:15 AM, Yuzhao Chen  wrote:
> >
> > Thx, Stamatis, that somehow make sense, if i pass around the parent node 
> > every time I visit a RelNode and keep the parents in the cache, but it is 
> > still not that intuitive. Actually I what a to add a new RelTrait which 
> > bind to a specific scope, for example:
> >
> > join-rel(trait1)
> > / \
> > join2 join3
> >
> > Join-rel has a trait trait1, and I want all the children of join-rel can 
> > see this trait, with Calcite’s default metadata handler, I can only see the 
> > trait from children nodes(traits propagate from the inputs), and I have no 
> > idea how to propagate a trait reversely?
> >
> >
> > Best,
> > Danny Chan
> > 在 2019年4月22日 +0800 PM8:44，Stamatis Zampetakis ，写道：
> > > Hi Danny,
> > >
> > > Apart from RelShuttle there is also RelVisitor which has a visit method
> > > that provides the parent [1]. Not sure, if it suits your needs.
> > >
> > > Best,
> > > Stamatis
> > >
> > > [1]
> > > https://github.com/apache/calcite/blob/ee83efd360793ef4201f4cdfc2af8d837b76ca69/core/src/main/java/org/apache/calcite/rel/RelVisitor.java#L43
> > >
> > >
> > > On Mon, Apr 22, 2019 at 2:14 PM Yuzhao Chen  wrote:
> > >
> > > > Now for RelNode, we have method getInput()[1] to fetch the input
> > > > RelNodes, but how we fetch the parent ?
> > > >
> > > > For example, we have plan:
> > > >
> > > > join-rel
> > > > / \
> > > > scan1 scan2
> > > >
> > > >
> > > > We can get scan1 and scan2 in join-rel directly with method getInput, 
> > > > but
> > > > how can we get the join rel in scan1 and scan 2 ?
> > > >
> > > > I know that there is a RelShuttle that can visit every RelNode and if I
> > > > make a cache for the inputs mapping, finally I can get the ‘parents’ 
> > > > from
> > > > the cache, but this is boring code and not that intuitive.
> > > >
> > > > Do you guys have any good ideas ?
> > > >
> > > > [1]
> > > > https://github.com/apache/calcite/blob/ee83efd360793ef4201f4cdfc2af8d837b76ca69/core/src/main/java/org/apache/calcite/rel/RelNode.java#L132
> > > >
> > > >
> > > > Best,
> > > > Danny Chan
> > > >
>

Re: Support non-equi join condition for EnumerableJoin, EnumerableMergeJoin and EnumerableSemiJoin ? Just a discussing..

[Yuzhao Chen]

Thx, Julian

Why not just support non-equi join condition for every physical algorithm, it 
does not make much sense if we have both HashJoin and a HashTheraJoin, cause a 
HashThataJoin with empty non-equi join condition is same as  a HashJoin.

And we can remove the limitations in the rule like FilterJoinRule.

Best,

Danny Chan
在 2019年4月23日 +0800 AM3:21，dev@calcite.apache.org，写道：
>
> If there are limitations, over time we would like to remove those 
> limitations, but we will probably do it by adding new algorithms, and 
> therefore new EnumerableXxx classes.

Re: How to traverse RelNode’s parent conviniently?

[Yuzhao Chen]

Thx, Andrew

I don’t want to have a custom RelNode class, I hope all the work about hints 
would be contributed to the community. I want to find an acceptable way to keep 
and propagate the hints if we use the MetadataHandler to cache and query the 
hints.

I don’t think the hints should be mixed into the cost model, that would make 
the cost computation very complex and hard to maintain, we only need the hints 
in our planning phrase to give suggestions, hints is more like another 
guideline for me and transparent to the planner.

Best,
Danny Chan
在 2019年4月23日 +0800 PM2:24，Андрей Цвелодуб ，写道：
> Hi Danny,
>
> I would also agree with Julian on his position. I've tried to get around
> this limitation in several different ways, but none of it ended well :)
>
> For your idea with hints, if you have custom RelNode classes, you can add
> hint as an additional field of the class and you can write a simple rule
> that propagates the hint downwards, step by step. And also include the hint
> in your cost estimation, so that nodes with hints would be more attractive
> to the planner. I'm not sure this would be the most correct way to use the
> cost mechanism, but at least it is straightforward and it works.
>
> Best Regards,
> Andrew Tsvelodub
>
> On Tue, 23 Apr 2019 at 08:44, Yuzhao Chen  wrote:
>
> > Julian,
> >
> > I want to add hint support for Calcite, the initial idea was to tag a
> > RelNode(transformed from a SqlNode with hint) with a hit attribute(or
> > trait), then I hope that the children (inputs) of it can see this hint, so
> > to make some decisions if it should consume or propagate the hint.
> >
> > The problem I got here is the trait propagate from inputs from, which is
> > the opposite as what I need, can you give some suggestions ? If I use
> > MetadataHandler to cache and propagate the hints, how to propagate from
> > parents to children ?
> >
> > Best,
> > Danny Chan
> > 在 2019年4月23日 +0800 AM3:14，Julian Hyde ，写道：
> > > TL;DR: RelNodes don’t really have parents. Be careful if you are relying
> > on the parent concept too much. Rely on rules instead.
> > >
> > > In the Volcano model, a RelNode doesn’t really have a parent. It might
> > be used in several places. (RelSet has a field ‘List parents’ that
> > is kept up to date as planing progresses. But it’s really for Volcano’s
> > internal use.)
> > >
> > > Even if you are not using Volcano, there are reasons to want the RelNode
> > graph to be a dag, so again, a RelNode doesn’t have a unique parent.
> > >
> > > RelShuttleImpl has a stack. You can use that to find the parent. But the
> > “parent” is just “where we came from as we traversed the RelNode graph”.
> > There may be other “parents” that you do not know about.
> > >
> > > If you have a Project and want to find all parents that are Filters,
> > don’t even think about “iterating over the parents” of the Project. Just
> > write a rule that matches a Filter on a Project, and trust Volcano to do
> > its job.
> > >
> > > Julian
> > >
> > >
> > >
> > >
> > > > On Apr 22, 2019, at 6:15 AM, Yuzhao Chen  wrote:
> > > >
> > > > Thx, Stamatis, that somehow make sense, if i pass around the parent
> > node every time I visit a RelNode and keep the parents in the cache, but it
> > is still not that intuitive. Actually I what a to add a new RelTrait which
> > bind to a specific scope, for example:
> > > >
> > > > join-rel(trait1)
> > > > / \
> > > > join2 join3
> > > >
> > > > Join-rel has a trait trait1, and I want all the children of join-rel
> > can see this trait, with Calcite’s default metadata handler, I can only see
> > the trait from children nodes(traits propagate from the inputs), and I have
> > no idea how to propagate a trait reversely?
> > > >
> > > >
> > > > Best,
> > > > Danny Chan
> > > > 在 2019年4月22日 +0800 PM8:44，Stamatis Zampetakis ，写道：
> > > > > Hi Danny,
> > > > >
> > > > > Apart from RelShuttle there is also RelVisitor which has a visit
> > method
> > > > > that provides the parent [1]. Not sure, if it suits your needs.
> > > > >
> > > > > Best,
> > > > > Stamatis
> > > > >
> > > > > [1]
> > > > >
> > https://github.com/apache/calcite/blob/ee83efd360793ef4201f4cdfc2af8d837b76ca69/core/src/main/java/org/apache/calcite/rel/RelVisitor.java#L43
> > > > >
> > > >

Re: How to traverse RelNode’s parent conviniently?

[Yuzhao Chen]

Thx, Julian

I think the hint path is a good way for searching RelNode’s parents, broadly, 
there may be these modules/things need to be modified:

1. Supports hints grammar for parser.jj
2. Cache the hints in the RelNode instance, and add method like 
RelNode#getHints() to fetch all the hints inherited for this node.
3. Modify #copy method for every kind of RelNode so that the hints can be 
copied when creating new equivalent nodes.
4. Add a visitor in after sql-to-rel phrase, to set up full hints list for 
every children RelNode if there exists any.
5. Add hints metadata handler and handles the hints fetching and overriding for 
specific kind of RelNode

The 2 and 3 are the modifications that i really want to confirm, that is, shall 
we store the hints in the RelNode instance ?

These are initial thoughts and if we make agreement, I would output a detail 
design doc which contains:

1. The hints grammar supported for the major sql engines
2. The hints grammar supported for Apache Calcite
3. The interface and design ideas of the proposed modifications


Best,
Danny Chan
在 2019年4月24日 +0800 AM3:04，Julian Hyde ，写道：
> I see that if you have a hint on, say, the root node then it would be nice 
> for its child or grand-child to be able to see that hint.
>
> How about giving each hint an inherit path? Thus given
>
> Filter Hint1
> +- Join
> +- Scan
> +- Project Hint2
> +- Scan
>
>
> Filter would have hints {Hint1[]}
> Join would have hints {Hint1[0]}
> Scan would have hints {Hint1[0, 0]}
> Project would have hints {Hint1[0,1], Hint2}
> Scan2 would have hints {[Hint1[0, 0, 1, 0], Hint2[0]}
>
> You could populate the hints and inherit paths with a single visitor pass 
> after sql-to-rel conversion.
>
> By the way, I still like the idea of having kinds as a kind of RelMetadata, 
> but I realize that a given RelNode might have more than one hint. So I think 
> that the getHints(RelNode) method would return a List, with Hint as 
> follows:
>
> class Hint {
> public final List inheritPath; // immutable, not null
> public final String type; // not null
> public final Object operand; // immutable, may be null, must be JSON data
> }
>
> operand must be JSON-style data (null, boolean, number, String, immutable 
> List of JSON data, or immutable order-preserving Map from String to JSON 
> data).
>
> > On Apr 23, 2019, at 1:25 AM, Yuzhao Chen  wrote:
> >
> > Thx, Andrew
> >
> > I don’t want to have a custom RelNode class, I hope all the work about 
> > hints would be contributed to the community. I want to find an acceptable 
> > way to keep and propagate the hints if we use the MetadataHandler to cache 
> > and query the hints.
> >
> > I don’t think the hints should be mixed into the cost model, that would 
> > make the cost computation very complex and hard to maintain, we only need 
> > the hints in our planning phrase to give suggestions, hints is more like 
> > another guideline for me and transparent to the planner.
> >
> > Best,
> > Danny Chan
> > 在 2019年4月23日 +0800 PM2:24，Андрей Цвелодуб ，写道：
> > > Hi Danny,
> > >
> > > I would also agree with Julian on his position. I've tried to get around
> > > this limitation in several different ways, but none of it ended well :)
> > >
> > > For your idea with hints, if you have custom RelNode classes, you can add
> > > hint as an additional field of the class and you can write a simple rule
> > > that propagates the hint downwards, step by step. And also include the 
> > > hint
> > > in your cost estimation, so that nodes with hints would be more attractive
> > > to the planner. I'm not sure this would be the most correct way to use the
> > > cost mechanism, but at least it is straightforward and it works.
> > >
> > > Best Regards,
> > > Andrew Tsvelodub
> > >
> > > On Tue, 23 Apr 2019 at 08:44, Yuzhao Chen  wrote:
> > >
> > > > Julian,
> > > >
> > > > I want to add hint support for Calcite, the initial idea was to tag a
> > > > RelNode(transformed from a SqlNode with hint) with a hit attribute(or
> > > > trait), then I hope that the children (inputs) of it can see this hint, 
> > > > so
> > > > to make some decisions if it should consume or propagate the hint.
> > > >
> > > > The problem I got here is the trait propagate from inputs from, which is
> > > > the opposite as what I need, can you give some suggestions ? If I use
> > > > MetadataHandler to cache and propagate the hints, how to propagate from
> > > > parents to children ?
> > > >
> 

Re: Support non-equi join condition for EnumerableJoin, EnumerableMergeJoin and EnumerableSemiJoin ? Just a discussing..

[Yuzhao Chen]

Julian,

I also agree with you that we should distinguish between EnumerableHashJoin and 
EnumerableMergeJoin, what i want to address about is if we should extend the 
EnumerableHashJoin and EnumerableMergeJoin to let them support Thera join 
conditions except the EquiJoin conditions.

It seems that Stamatis also agree that we can extend the existing 
EnumerableXXXs, correct me if i have wrong understanding.


Best,
Danny Chan
在 2019年4月24日 +0800 AM2:18，Julian Hyde ，写道：
> I agree with Stamatis.
>
> Let’s suppose we have a merge join and hash join in enumerable convention. 
> The merge join requires its input to be sorted, the hash join does not. They 
> can both perform equi-join. The merge join can also perform joins like
>
> FROM orders
> JOIN shipments
> ON shipments.shipDate BETWEEN orders.orderDate
> AND orders.orderDate + INTERVAL ‘7’ DAY
>
> Should we use the same sub-class of RelNode, EnumerableJoin, for both, or 
> should we create sub-classes EnumerableHashJoin and EnumerableMergeJoin?
>
> I think we should do the latter:
> * They are clearly different algorithms, and we generate different code for 
> them
> * They have different cost models
> * They have different input and output traits (the merge join requires sorted 
> inputs and produces a sorted output)
> * If we have an equi-join, we might want to generate both options and see how 
> they compare (the merge join might win if one or both of the inputs are 
> sorted), and both algorithms used the EnumerableJoin class we would not be 
> able to distinguish the RelNode instances.
>
> Julian
>
>
> > On Apr 23, 2019, at 11:06 AM, Stamatis Zampetakis  wrote:
> >
> > In the literature, there are many algorithms to perform a join and the vast
> > majority of them cannot handle theta conditions (either for performance
> > reasons, or because the algorithm was simply not meant for it).
> >
> > For the existing EnumerableXXXJoin variants, it may be possible to extend
> > the algorithm to handle theta joins without structural changes to the
> > algorithm (as I think it is the case in [3]).
> > For these cases, I agree with you it does not make sense to have many
> > variants (e.g., EnumerableThetaHashJoin seems redundant).
> >
> > In the future we may decide to use other hash-based implementations with
> > certain applicability restrictions (such as apply only on equijoins) but
> > let's not worry too much about that now.
> >
> > Regarding the rules, I think it is useful to have different variants.
> > Consider for example a system that does not have join processing algorithms
> > that can treat theta-joins; pushing non-equi conditions in a join seems
> > undesirable in this case.
> >
> > For more information regarding join processing, there are a few interesting
> > (and old) surveys [4, 5] which outline some of the most typical algorithms
> > that are used in relational databases.
> >
> > Best,
> > Stamatis
> >
> > [4] Query evaluation techniques for Large databases (
> > https://web.stanford.edu/class/cs346/2014/graefe.pdf)
> > [5] Join processing in relational databases (
> > https://www.csd.uoc.gr/~hy460/pdf/p63-mishra.pdf)
> >
> >
> > On Tue, Apr 23, 2019 at 8:05 AM Yuzhao Chen  wrote:
> >
> > > Thx, Julian
> > >
> > > Why not just support non-equi join condition for every physical algorithm,
> > > it does not make much sense if we have both HashJoin and a HashTheraJoin,
> > > cause a HashThataJoin with empty non-equi join condition is same as a
> > > HashJoin.
> > >
> > > And we can remove the limitations in the rule like FilterJoinRule.
> > >
> > > Best,
> > >
> > > Danny Chan
> > > 在 2019年4月23日 +0800 AM3:21，dev@calcite.apache.org，写道：
> > > >
> > > > If there are limitations, over time we would like to remove those
> > > limitations, but we will probably do it by adding new algorithms, and
> > > therefore new EnumerableXxx classes.
> > >
>

Re: JIRA usage reminders

[YuZhao Chen]

+1 for adding the notes to website

Vladimir Sitnikov 于2019年4月25日 周四下午9:52写道：

> Stamatis> If you see things that are
> Stamatis> incorrect or that need to be done differently feel free to
> reply to this
> Stamatis> email.
>
> LGTM.
>
> Stamatis, thanks for the writeup, however I'm inclined to suppose it
> makes sense to put that on the website.
>
> Such mails will be hard to find, especially for the ones who don't
> know such a mail exists at all.
> On the other hand, "/develop/" page is trivial to navigate even by
> just browsing the website.
>
> Vladimir
>

Re: calcite-test-dataset VM has issues with Druid

[Yuzhao Chen]

I also ran this issue, and make sure stop/up will definitely make Druid not 
work any more.

Best,
Danny Chan
在 2019年4月26日 +0800 AM7:10，dev@calcite.apache.org，写道：
>
> Stop/up definitely doesn't
> work.

Re: Support non-equi join condition for EnumerableJoin, EnumerableMergeJoin and EnumerableSemiJoin ? Just a discussing..

[Yuzhao Chen]

Thx, Stamatis

Finally I think that [3] will provide better compatibility, just like you said 
that some engines may not support non-equi HashJoin, so pushing into non-equi 
join conditions seems not that necessary.

The better way is to give more join algorithm :), I will try my best to 
implement more algorithms if I have time.

Best,
Danny Chan
在 2019年4月26日 +0800 PM9:29，Stamatis Zampetakis ，写道：
> Hi Danny,
>
> Yes, I agree that we don't need two variations of hash joins
> (EnumerableHashJoin and EnumerableThetaHashJoin) as it is the code right
> now.
>
> Having that said, I am not sure if we can really handle the general case of
> theta joins with hash-based algorithms.
>
> The approach in [3] does not really handle general theta joins but allows
> to split a theta condition into equi and non-equi parts where the non-equi
> part is executed as a post-join filter.
> Since the post-join filter is part of the join operator we can say that we
> have a theta hash join operator but this is partially true.
>
> For instance, the following query:
>
> SELECT e1.name
> FROM emp e1
> INNER JOIN emp e2
> ON e1.salary < e2.salary
>
> cannot use at all hash join algorithm [3] because there is no equality
> condition.
>
> I wouldn't mind keeping EnumerableHashJoin only for equijoins and execute
> the post-join filter as such but this would make outer joins
> difficult/impossible to handle so I believe [3] is a good step forward.
>
> Best,
> Stamatis
>
>
>
>
>
> On Wed, Apr 24, 2019 at 6:28 AM Yuzhao Chen  wrote:
>
> > Julian,
> >
> > I also agree with you that we should distinguish between
> > EnumerableHashJoin and EnumerableMergeJoin, what i want to address about is
> > if we should extend the EnumerableHashJoin and EnumerableMergeJoin to let
> > them support Thera join conditions except the EquiJoin conditions.
> >
> > It seems that Stamatis also agree that we can extend the existing
> > EnumerableXXXs, correct me if i have wrong understanding.
> >
> >
> > Best,
> > Danny Chan
> > 在 2019年4月24日 +0800 AM2:18，Julian Hyde ，写道：
> > > I agree with Stamatis.
> > >
> > > Let’s suppose we have a merge join and hash join in enumerable
> > convention. The merge join requires its input to be sorted, the hash join
> > does not. They can both perform equi-join. The merge join can also perform
> > joins like
> > >
> > > FROM orders
> > > JOIN shipments
> > > ON shipments.shipDate BETWEEN orders.orderDate
> > > AND orders.orderDate + INTERVAL ‘7’ DAY
> > >
> > > Should we use the same sub-class of RelNode, EnumerableJoin, for both,
> > or should we create sub-classes EnumerableHashJoin and EnumerableMergeJoin?
> > >
> > > I think we should do the latter:
> > > * They are clearly different algorithms, and we generate different code
> > for them
> > > * They have different cost models
> > > * They have different input and output traits (the merge join requires
> > sorted inputs and produces a sorted output)
> > > * If we have an equi-join, we might want to generate both options and
> > see how they compare (the merge join might win if one or both of the inputs
> > are sorted), and both algorithms used the EnumerableJoin class we would not
> > be able to distinguish the RelNode instances.
> > >
> > > Julian
> > >
> > >
> > > > On Apr 23, 2019, at 11:06 AM, Stamatis Zampetakis 
> > wrote:
> > > >
> > > > In the literature, there are many algorithms to perform a join and the
> > vast
> > > > majority of them cannot handle theta conditions (either for performance
> > > > reasons, or because the algorithm was simply not meant for it).
> > > >
> > > > For the existing EnumerableXXXJoin variants, it may be possible to
> > extend
> > > > the algorithm to handle theta joins without structural changes to the
> > > > algorithm (as I think it is the case in [3]).
> > > > For these cases, I agree with you it does not make sense to have many
> > > > variants (e.g., EnumerableThetaHashJoin seems redundant).
> > > >
> > > > In the future we may decide to use other hash-based implementations
> > with
> > > > certain applicability restrictions (such as apply only on equijoins)
> > but
> > > > let's not worry too much about that now.
> > > >
> > > > Regarding the rules, I think it is useful to have different variants.
> > > > Consider for example a system that does n

Re: How to traverse RelNode’s parent conviniently?

[Yuzhao Chen]

Thx, Julian

Let me repeat my thoughts about the details again, in order to implement the 
hints, maybe these things are needed:

The main diff is that we will maintain a global hints cache
1. Supports hints grammar for parser.jj
2. During/after sql-to-rel, we may pass a hints cache to the SqlToRelConverter, 
there is a visitor to setup/init the RelNodes’hints to the cache once at a 
time, this cache scope is global and would be active the whole query planning 
time. The cache only keep hints for few top nodes that really needs
3. In the Planner, add set/get hints cache method, so that in the planning 
rules, we can see the hints cache,
And we can also ban some rule matching in the planner
4. Hook the RelOptCall#transformTo method to handle logic of hints 
propagating(invoke the hints logic again same as sql-to-rel phrase), this will 
also update the global hints cache
It seems that given the global hints cache, we do not need the MetaDataHandler 
any more, this is the thing I most want to make sure.
Hope for your suggestions.

Best,
Danny Chan
在 2019年4月25日 +0800 AM3:07，Julian Hyde ，写道：
> I think it’s OK to attach hints to the (few) RelNodes that come out of the 
> SqlToRelConverter.
>
> But it would be a mistake to try to propagate those hints to all of the 
> RelNodes that are created during query planning. Even if we changed all of 
> the copy methods (a huge task) there are many other ways that RelNodes get 
> created. We would end up with a RelNode graph with lots of hints, and most of 
> those hints would be inaccurate or not applicable.
>
> For a particular hint, say "/*+ nohashjoin */“, some piece of code would need 
> to look at the initial RelNode tree and take its own action: say, build a 
> data structure to be used by planner rules, or enable or disable planner 
> rules.
>
>
> > On Apr 23, 2019, at 9:31 PM, Chunwei Lei  wrote:
> >
> > Thanks Danny.
> >
> > Those are good points. I think it depends on what we consider hint as.
> > IMHO, if we consider hint as a kind of metadata,
> > it is not a good idea to store the hints in the RelNode instance.
> >
> >
> >
> > Best,
> > Chunwei
> >
> > On Wed, Apr 24, 2019 at 11:09 AM Yuzhao Chen  wrote:
> > >
> > > Thx, Julian
> > >
> > > I think the hint path is a good way for searching RelNode’s parents, 
> > > broadly, there may be these modules/things need to be modified:
> > >
> > > 1. Supports hints grammar for parser.jj
> > > 2. Cache the hints in the RelNode instance, and add method like 
> > > RelNode#getHints() to fetch all the hints inherited for this node.
> > > 3. Modify #copy method for every kind of RelNode so that the hints can be 
> > > copied when creating new equivalent nodes.
> > > 4. Add a visitor in after sql-to-rel phrase, to set up full hints list 
> > > for every children RelNode if there exists any.
> > > 5. Add hints metadata handler and handles the hints fetching and 
> > > overriding for specific kind of RelNode
> > >
> > > The 2 and 3 are the modifications that i really want to confirm, that is, 
> > > shall we store the hints in the RelNode instance ?
> > >
> > > These are initial thoughts and if we make agreement, I would output a 
> > > detail design doc which contains:
> > >
> > > 1. The hints grammar supported for the major sql engines
> > > 2. The hints grammar supported for Apache Calcite
> > > 3. The interface and design ideas of the proposed modifications
> > >
> > >
> > > Best,
> > > Danny Chan
> > > 在 2019年4月24日 +0800 AM3:04，Julian Hyde ，写道：
> > > > I see that if you have a hint on, say, the root node then it would be 
> > > > nice for its child or grand-child to be able to see that hint.
> > > >
> > > > How about giving each hint an inherit path? Thus given
> > > >
> > > > Filter Hint1
> > > > +- Join
> > > > +- Scan
> > > > +- Project Hint2
> > > > +- Scan
> > > >
> > > >
> > > > Filter would have hints {Hint1[]}
> > > > Join would have hints {Hint1[0]}
> > > > Scan would have hints {Hint1[0, 0]}
> > > > Project would have hints {Hint1[0,1], Hint2}
> > > > Scan2 would have hints {[Hint1[0, 0, 1, 0], Hint2[0]}
> > > >
> > > > You could populate the hints and inherit paths with a single visitor 
> > > > pass after sql-to-rel conversion.
> > > >
> > > > By the way, I still like the idea of having kinds as a kind of 
> > > > RelMetadata, but I re

Re: Function sets (aka flavor and dialect)

[Yuzhao Chen]

Thx for the discussing, Julian

I’m also confused about the difference between SqlConformance and SqlDialect, 
and now the Flavor, why not just use one word SqlDialect uniformly ? My 
interpretation about sql dialect includes the functions, cause they are part of 
the sql syntax as operators.

Best,
Danny Chan
在 2019年4月27日 +0800 AM2:47，Michael Mior ，写道：
> I think "function set" sounds like a reasonable name. My current
> interpretation of dialect is that it's more related to the SQL syntax
> accepted by each system. I'm not really sure what the intended
> difference is between dialect and conformance is, but it seems like
> perhaps these two concepts could be merged.
> --
> Michael Mior
> mm...@apache.org
>
> Le ven. 26 avr. 2019 à 14:14, Julian Hyde  a écrit :
> >
> > There’s a discussion in https://issues.apache.org/jira/browse/CALCITE-2846 
> >  about reorganizing the 
> > Sql operator table. The idea is for people to be able to start a connection 
> > with, say, the standard set of SQL functions, plus functions to emulate 
> > MySQL, plus spatial functions. And for us to reorganize the code so that if 
> > a function is in both MySQL and Oracle but not in the SQL Standard we only 
> > define that function in one place.
> >
> > We need a word for a "set of functions". (Standard, MySQL, Oracle and 
> > Spatial are examples of sets of functions in the above paragraph.) It is 
> > tempting to call this a dialect, but that word has an existing meaning that 
> > we do not want to change. “Conformance” is another existing concept that we 
> > need to work with. I suggested “flavor” in the JIRA case, but now I’m 
> > thinking it is an arbitrary word that gives very little clue as to its 
> > purpose. The concept is already exposed via the connect-string parameter 
> > “fun" (e.g. “jdbc:calcite:fun=spatial,oracle”).
> >
> > Any ideas for a better word, or a better way of organizing the dialect / 
> > conformance / function set concepts.
> >
> > Julian
> >

Re: How to traverse RelNode’s parent conviniently?

[Yuzhao Chen]

Thx Julian

Mostly got your idea, but one thing needs to confirm:

Now the MetadataHandler is kind of query lazy the cache is code-gen ed in the 
handler class, the metadata also propagate from inputs, when I got an RelNode’s 
hint, how can I cache it in the metadata handler for querying ?

Best,
Danny Chan
在 2019年4月27日 +0800 AM6:57，Julian Hyde ，写道：
> The RelMetadata system is designed for these kinds of annotations - if there 
> is a “global hints cache” there’s no benefit to doing it outside the 
> RelMetadata system.
>
> That said, I don’t know (and I don’t think anyone knows) how we want hints to 
> be propagated as we generate RelNodes from RelNodes. I think we should focus 
> on really simple cases first (e.g. hints about the whole query, or about 
> particular table scans), and not try to automatically propagate them.
>
> We can make the hints propagation mechanism more sophisticated when we have 
> an actual use case to drive us.
>
> Julian
>
>
> > On Apr 26, 2019, at 3:41 PM, Yuzhao Chen  wrote:
> >
> > Thx, Julian
> >
> > Let me repeat my thoughts about the details again, in order to implement 
> > the hints, maybe these things are needed:
> >
> > The main diff is that we will maintain a global hints cache
> > 1. Supports hints grammar for parser.jj
> > 2. During/after sql-to-rel, we may pass a hints cache to the 
> > SqlToRelConverter, there is a visitor to setup/init the RelNodes’hints to 
> > the cache once at a time, this cache scope is global and would be active 
> > the whole query planning time. The cache only keep hints for few top nodes 
> > that really needs
> > 3. In the Planner, add set/get hints cache method, so that in the planning 
> > rules, we can see the hints cache,
> > And we can also ban some rule matching in the planner
> > 4. Hook the RelOptCall#transformTo method to handle logic of hints 
> > propagating(invoke the hints logic again same as sql-to-rel phrase), this 
> > will also update the global hints cache
> > It seems that given the global hints cache, we do not need the 
> > MetaDataHandler any more, this is the thing I most want to make sure.
> > Hope for your suggestions.
> >
> > Best,
> > Danny Chan
> > 在 2019年4月25日 +0800 AM3:07，Julian Hyde ，写道：
> > > I think it’s OK to attach hints to the (few) RelNodes that come out of 
> > > the SqlToRelConverter.
> > >
> > > But it would be a mistake to try to propagate those hints to all of the 
> > > RelNodes that are created during query planning. Even if we changed all 
> > > of the copy methods (a huge task) there are many other ways that RelNodes 
> > > get created. We would end up with a RelNode graph with lots of hints, and 
> > > most of those hints would be inaccurate or not applicable.
> > >
> > > For a particular hint, say "/*+ nohashjoin */“, some piece of code would 
> > > need to look at the initial RelNode tree and take its own action: say, 
> > > build a data structure to be used by planner rules, or enable or disable 
> > > planner rules.
> > >
> > >
> > > > On Apr 23, 2019, at 9:31 PM, Chunwei Lei  wrote:
> > > >
> > > > Thanks Danny.
> > > >
> > > > Those are good points. I think it depends on what we consider hint as.
> > > > IMHO, if we consider hint as a kind of metadata,
> > > > it is not a good idea to store the hints in the RelNode instance.
> > > >
> > > >
> > > >
> > > > Best,
> > > > Chunwei
> > > >
> > > > On Wed, Apr 24, 2019 at 11:09 AM Yuzhao Chen  
> > > > wrote:
> > > > >
> > > > > Thx, Julian
> > > > >
> > > > > I think the hint path is a good way for searching RelNode’s parents, 
> > > > > broadly, there may be these modules/things need to be modified:
> > > > >
> > > > > 1. Supports hints grammar for parser.jj
> > > > > 2. Cache the hints in the RelNode instance, and add method like 
> > > > > RelNode#getHints() to fetch all the hints inherited for this node.
> > > > > 3. Modify #copy method for every kind of RelNode so that the hints 
> > > > > can be copied when creating new equivalent nodes.
> > > > > 4. Add a visitor in after sql-to-rel phrase, to set up full hints 
> > > > > list for every children RelNode if there exists any.
> > > > > 5. Add hints metadata handler and handles the hints fetching and 
> > > > > overriding for specific kind of Re

Re: [ANNOUNCE] New committers: Chunwei Lei

[Yuzhao Chen]

Congratulations, Chunwei!

Best,
Danny Chan
在 2019年4月27日 +0800 AM11:04，Haisheng Yuan ，写道：
> Congratulations, Chunwei!
>
> Thanks ~
> Haisheng Yuan
> --
> 发件人：Francis Chuang
> 日　期：2019年04月27日 10:34:43
> 收件人：
> 主　题：[ANNOUNCE] New committers: Chunwei Lei
>
> Apache Calcite's Project Management Committee (PMC) has invited Chunwei
> Lei to become a committer, and we are pleased to announce that he has
> accepted.
>
> Over the past few months, Chunwei has proposed numerous pull requests,
> reviewed PRs and opened JIRA issues for the project.
>
> Chunwei, welcome, thank you for your contributions, and we look forward
> your further interactions with the community! If you wish, please feel
> free to tell us more about yourself and what you are working on.
>
> Francis (on behalf of the Apache Calcite PMC)

Re: [ANNOUNCE] New committers: Ruben Quesada Lopez

[Yuzhao Chen]

Congratulations, Ruben, and thx for your work!

Best,
Danny Chan
在 2019年4月27日 +0800 AM10:57，Haisheng Yuan ，写道：
> Congratulations, Ruben! I am impressed by your contributions.
>
> Thanks ~
> Haisheng Yuan
> --
> 发件人：Francis Chuang
> 日　期：2019年04月27日 10:39:18
> 收件人：
> 主　题：[ANNOUNCE] New committers: Ruben Quesada Lopez
>
>
> Apache Calcite's Project Management Committee (PMC) has invited Ruben
> Quesada Lopez to become a committer, and we are pleased to announce that
> he has accepted.
>
> In just a few months, Ruben has contributed more than 15 pull requests
> to the project, fixing bugs and implementing new features.
>
> Ruben, welcome, thank you for your contributions, and we look forward
> your further interactions with the community! If you wish, please feel
> free to tell us more about yourself and what you are working on.
>
> Francis (on behalf of the Apache Calcite PMC)

Re: [ANNOUNCE] Stamatis Zampetakis joins Calcite PMC

[Yuzhao Chen]

Congrats, Stamatis. And thx for your help !

Best,
Danny Chan
在 2019年4月27日 +0800 AM10:44，dev@calcite.apache.org，写道：
>
> Congrats, Stamatis. Thanks for your good work.

Re: [ANNOUNCE] New committers: Zhiwei Peng

[Yuzhao Chen]

Congratulations, Zhiwei!

Best,
Danny Chan
在 2019年4月27日 +0800 AM10:37，Francis Chuang ，写道：
> Apache Calcite's Project Management Committee (PMC) has invited Zhiwei
> Peng to become a committer, and we are pleased to announce that he has
> accepted.
>
> Zhiwei has been contributing to Calcite for a while, racking up an
> impressive 20 pull requests, in particular, doing a lot of work to
> improve RexSimplify.
>
> Zhiwei, welcome, thank you for your contributions, and we look forward
> your further interactions with the community! If you wish, please feel
> free to tell us more about yourself and what you are working on.
>
> Francis (on behalf of the Apache Calcite PMC)

Re: [RESULT] [VOTE] Release apache-calcite-avatica-1.14.0 (release candidate 0)

[Yuzhao Chen]

Thx for your work, Francis

Best,
Danny Chan
在 2019年4月29日 +0800 PM1:56，Francis Chuang ，写道：
> Thanks to everyone who has tested the release candidate and given
> their comments and votes.
>
> The tally is as follows.
>
> 3 binding +1s:
> Francis Chuang
> Michael Mior
> Stamatis Zampetakis
>
> 1 non-binding +1s:
> Haisheng Yuan
>
> No 0s or -1s.
>
> Therefore I am delighted to announce that the proposal to release
> Apache Calcite Avatica 1.14.0 has passed.
>
> Thanks everyone. We’ll now roll the release out to the mirrors.
>
> Francis

Re: [DISCUSS] Towards Calcite 1.20.0

[Yuzhao Chen]

Thx Michael for listing these changes.

[CALCITE-3026] has been canceled, see the discussion in [1].
[CALCITE-3037]’s renaming work has already been done in CALCITE-2969, I’m not 
that sure if we should merge the EnumerableCorrelate and 
EnumerableNestedLoopJoin yet, but I think it should not be a release blocker.

[1] https://issues.apache.org/jira/browse/CALCITE-2969

Best,
Danny Chan
在 2019年4月30日 +0800 PM11:15，Michael Mior ，写道：
> Calcite 1.19.0 was released approximately one month ago. This was
> later than we originally planned (although I think with good reason
> and I'm happy with what made it into this release). I don't think
> there's an imminent need for a new release, but I wanted to start the
> discussion now that Avatica has had it's latest release.
>
> There's a few big things in progress below that I thought I'd see if
> we want to try to include in the next release. Apologies if I missed
> any.
>
> CALCITE-1581 UDTF like in hive
> CALCITE-2952 Certify Calcite on JDK 12
> CALCITE-3036 Remove correlate variables from Join
> CALCITE-3037 Rename EnumerableThetaJoin to EnumerableNestedLoopJoin
>
> --
> Michael Mior
> mm...@apache.org

Re: ClassCastException RelOptCostImpl VolcanoCost while using HepPlanner

[Yuzhao Chen]

Why you care about cost when use HepPlanner ? The HepPlanner is aimed for some 
deterministic planning rules, we usually do not need cost in Hep. Some 
exceptions like Join reorder may need a cost.

What kind of planning promotion you did ? I'm kind of curious about it.

Best,
Danny Chan
在 2019年5月1日 +0800 PM9:27，Mark Pasterkamp ，写道：
> Dear all,
>
> While playing around with the HepPlanner I ran into an issue where the
> planner wants to rewrite a query with a union rewrite. When the
> RelMetaDataQuery computes the cost, the cost instance is a VolcanoCost.
> Then when it tries to calculate the cost of one of the union's operands it
> is a RelCostImpl which results in the ClassCastException.
>
> How would I go about solving this issue? As far as my knowledge goes, I am
> not able to change the costhandler of the RelMetaDataQuery. Another
> approach I could see is removing the cast in the VolcanoCost class, but I
> would hope I do not have to do that.
>
>
> With kind regards,
>
> Mark

Re: Virtual key signing party

[Yuzhao Chen]

Hi, Stamatis

I’d also like to attend the party.

===
pub rsa2048 2019-05-02 [SC] [expires: 2021-05-01]
EC891B124DEF50B5800B1DD6F659E64809F2C0F2
uid Yuzhao Chen 
sub rsa2048 2019-05-02 [E] [expires: 2021-05-01]
===

Best,
Danny Chan
在 2019年4月28日 +0800 PM9:48，Stamatis Zampetakis ，写道：
> Hello,
>
> The last time [1] that we did a virtual key signing [2] party was about a
> year ago. Since then the community has grown and I think now is good time
> to expand our web of trust.
>
> We could try to do it via Skype or Google Hangouts on 2 May, 2019 at 18:30
> UTC+3 (which should be convient for a couple of states in US and most
> countries in EU). My user id is zabetak.
>
> Anybody in the list can join the call, and it is strongly suggested for
> future release managers without a trusted key (like my self).
>
> If you would like to attend, please reply to this thread with your public
> keys' fingerprint (gpg --fingerprint) before the meeting.
>
> pub rsa4096 2019-04-25 [SC] [expires: 2022-04-25]
> 76B9 AAE3 E86C 65D6 7A24 33D3 41CB 331D 85CE B407
> uid Stamatis Zampetakis 
> sub rsa4096 2019-04-25 [E] [expires: 2030-04-25]
>
> To verify your identity, please bring with you a government issued ID
> (preferably passport).
>
> If you can't make it, we can try to organize another one soon.
>
> Best,
> Stamatis
>
> [1]
> https://lists.apache.org/thread.html/37cef1fd59169b187e74225fd5b06a4940e3065abad20d161ff05c80@%3Cdev.calcite.apache.org%3E
> [2]
> http://www.cryptnet.net/fdp/crypto/keysigning_party/en/keysigning_party.html

Re: Virtual key signing party

[Yuzhao Chen]

I have the same issue with Haisheng Yuan, what is the correct key server should 
we use ?

Best,
Danny Chan
在 2019年5月3日 +0800 PM10:20，Stamatis Zampetakis ，写道：
> Apologies for the confusion.
>
> If there is still interest, I can participate again next week on
> http://tiny.cc/mku45y.
> We can also change the time to +/- 3 hours if that is more convenient for
> other people.
>
> @Haisheng: I have an issue with pgp.mit.edu keyserver (it seems to be down)
> so I cannot pull your key to my keyring
>
> On Thu, May 2, 2019 at 8:01 PM Julian Hyde  wrote:
>
> > For future meetings I recommend sending out a link like this:
> > https://timeanddate.com/s/3uqu <https://timeanddate.com/s/3uqu>. (The
> > Apache board uses this service to schedule its meetings, since people are
> > dialing in from all over the world.)
> >
> > Sorry I didn’t make it to the party this time. I’ll try to attend the next
> > one.
> >
> > > On May 2, 2019, at 10:38 AM, Vineet Garg  wrote:
> > >
> > > I also converted to wrong time and ended up missing the call. I'll try to
> > > join the next one.
> > >
> > > On Thu, May 2, 2019 at 10:29 AM Haisheng Yuan 
> > > wrote:
> > >
> > > > Sorry, I missed the call. Converted to wrong time. Will join next party.
> > > >
> > > > Thanks ~
> > > > Haisheng Yuan
> > > > --
> > > > 发件人：Haisheng Yuan
> > > > 日 期：2019年05月03日 01:02:14
> > > > 收件人：Stamatis Zampetakis; Apache Calcite dev list<
> > > > dev@calcite.apache.org>
> > > > 主 题：Re: Re: Virtual key signing party
> > > >
> > > > Hi Stamatis,
> > > >
> > > > I use pgp.mit.edu. Can you try again?
> > > >
> > > > Thanks ~
> > > > Haisheng Yuan
> > > > --
> > > > 发件人：Stamatis Zampetakis
> > > > 日 期：2019年05月02日 15:27:29
> > > > 收件人：
> > > > 主 题：Re: Virtual key signing party
> > > >
> > > > Hi again,
> > > >
> > > > @Francis: Thanks for sharing the instructions, they are very useful.
> > > > @Andrei: No worries we can try earlier or later today. Ping me on
> > > > skype/hangouts when you are available and if I am there we can do it
> > then.
> > > > @Haisheng, Danny: I tried to pull your keys to my keyring but with no
> > > > success; which keyserver(s) are you using?
> > > >
> > > > On Thu, May 2, 2019 at 4:08 AM Yuzhao Chen 
> > wrote:
> > > >
> > > > > Hi, Stamatis
> > > > >
> > > > > I’d also like to attend the party.
> > > > >
> > > > > ===
> > > > > pub rsa2048 2019-05-02 [SC] [expires: 2021-05-01]
> > > > > EC891B124DEF50B5800B1DD6F659E64809F2C0F2
> > > > > uid Yuzhao Chen 
> > > > > sub rsa2048 2019-05-02 [E] [expires: 2021-05-01]
> > > > > ===
> > > > >
> > > > > Best,
> > > > > Danny Chan
> > > > > 在 2019年4月28日 +0800 PM9:48，Stamatis Zampetakis ，写道：
> > > > > > Hello,
> > > > > >
> > > > > > The last time [1] that we did a virtual key signing [2] party was
> > > > about a
> > > > > > year ago. Since then the community has grown and I think now is good
> > > > time
> > > > > > to expand our web of trust.
> > > > > >
> > > > > > We could try to do it via Skype or Google Hangouts on 2 May, 2019 at
> > > > > 18:30
> > > > > > UTC+3 (which should be convient for a couple of states in US and 
> > > > > > most
> > > > > > countries in EU). My user id is zabetak.
> > > > > >
> > > > > > Anybody in the list can join the call, and it is strongly suggested
> > for
> > > > > > future release managers without a trusted key (like my self).
> > > > > >
> > > > > > If you would like to attend, please reply to this thread with your
> > > > public
> > > > > > keys' fingerprint (gpg --fingerprint) before the meeting.
> > > > > >
> > > > > > pub rsa4096 2019-04-25 [SC] [expires: 2022-04-25]
> > > > > > 76B9 AAE3 E86C 65D6 7A24 33D3 41CB 331D 85CE B407
> > > > > > uid Stamatis Zampetakis 
> > > > > > sub rsa4096 2019-04-25 [E] [expires: 2030-04-25]
> > > > > >
> > > > > > To verify your identity, please bring with you a government issued 
> > > > > > ID
> > > > > > (preferably passport).
> > > > > >
> > > > > > If you can't make it, we can try to organize another one soon.
> > > > > >
> > > > > > Best,
> > > > > > Stamatis
> > > > > >
> > > > > > [1]
> > > > > >
> > > > >
> > > >
> > https://lists.apache.org/thread.html/37cef1fd59169b187e74225fd5b06a4940e3065abad20d161ff05c80@%3Cdev.calcite.apache.org%3E
> > > > > > [2]
> > > > > >
> > > > >
> > > >
> > http://www.cryptnet.net/fdp/crypto/keysigning_party/en/keysigning_party.html
> > > > >
> > > >
> > > >
> > > >
> >
> >

Re: Difficulties with implementing a table for a custom convention

[Yuzhao Chen]

You can set up the target traits to include the convention you want when invoke 
the Program#run method [1], with the converters as rules of the program.

[1] 
https://github.com/apache/calcite/blob/9ece70f5dcdb00dbc6712496c51f52c05178d4aa/core/src/main/java/org/apache/calcite/tools/Program.java#L38

Best,
Danny Chan
在 2019年5月4日 +0800 PM6:48，Muhammad Gelbana ，写道：
> I implemented a new convention but I'm facing difficulties with
> implementing tables for that convention.
>
> Since I need to apply my convention rules and converters, I assume I must
> implement TranslatableTable so I can override the RelNode produced by the
> toRel method and provide the rules I need through the
> RelNode.register(RelOptPlanner planner) call back (Is there another way ?)
>
> At the same time, I need for my table's RelNode to implement my
> convention's interface.
>
> After implementing TranslatableTable and my convention's interface, I
> discovered that my table's RelNode have to either implement an interpreter
> node or the InterpretableRel interface. But this led to what seems to me as
> that my table scan is executed twice.
>
> I tried many things, such as having my table extend TableScan,
> AbstractEnumerable2 QueryableTable and other interfaces/class that either
> didn't give me what I want or failed with strange exceptions.
>
> What I need is to implement my table scan following the same convention I'm
> writing. So what is the correct way to do this ?
>
> The MongoDB adapter way is to implement the TranslatableTable and the
> QueryableTable interfaces, but that's confusing me. I mean, it's already
> implementing its own new convention interface, why would it need to
> implement another one to implement its table scans ? I thought while
> executing the final plan, the first calcite identified node (enumerable or
> bindable, enumerable as in MongoDB adapter's example), the interpreter
> would execute that node, which will internally execute all its input nodes
> including the table scan, correct ?
>
> Thanks,
> Gelbana

Re: Rewriting queries with Calcite

[Yuzhao Chen]

You can supply a RuleSet when use Program to promote your logical/physical 
plan, the RuleSet is totally can be custom.

[1] 
https://github.com/apache/calcite/blob/9ece70f5dcdb00dbc6712496c51f52c05178d4aa/core/src/main/java/org/apache/calcite/tools/Program.java#L38

Best,
Danny Chan
在 2019年5月6日 +0800 PM11:55，dev@calcite.apache.org，写道：
>
> TranslatableTable

Reorginize the SqlDialects ? Maybe just a discussion

[Yuzhao Chen]

Hi guys, recently there is a issue that want to integrate SqlDialect and 
SqlParser.Config
The background is that there are cases that we both use the SqlDialect and 
SqlParser.config, but these config items are really a mess and come confused me 
a lot, so i list them there and to see if we can make some progress.

### SqlDialect config items
- nullCollation: null values collation
- dataTypeSystem: type system to implement specific data types
- identifierQuoteString: starting quoting charactor
- identifierEndQuoteString: end quoting charactor
- DatabaseProduct: The name really confuse me ? Insn't it just a SqlDialect ?
- identifierEscapedQuote: escape quote for identifierQuoteString

### SqlParser.Config config items
- identifierMaxLength: max length of an identifier
- quotedCasing: casing of quoted identifier
- unquotedCasing: casing of unquoted identifier
- quoting: quoting charactor
- caseSensitive: if to parse the sql case sensitively
- parserFactory: a parser factory to profuce the parser
- SqlConformance: The name really confuse me ? Insn't it just a SqlDialect ?

### SqlConformance config items
- isLiberal: Whether this dialect supports features from a wide variety of 
dialects
- isGroupByAlias: group by column alias
- isGroupByOrdinal: group by column ordinal
- isHavingAlias: is having by column alias
- isSortByOrdinal: sort by column ordinal
- isSortByAlias: sort by column alias
- isSortByAliasObscures: Whether "empno" is invalid in "select empno as x from 
emp order by empno"
- isFromRequired: Whether FROM clause is required in a SELECT statement
- isBangEqualAllowed: Whether the bang-equal token != is allowed as an 
alternative to <> in the parser
- isPercentRemainderAllowed: Whether the "%" operator is allowed by the parser 
as an alternative to the mod function
- isMinusAllowed: Whether MINUS is allowed as an alternative to EXCEPT in the 
parser
- isApplyAllowed()
- ...omitted

We can almost see the the SqlDialect mainly controls how we access other 
datasources(the runtime config), the SqlParser.Config mainly controls the 
behavior of sql parsing(so we can mostly see the keywords xxxAllowed).

Altough the 2 config class aims to different use cases, the both belong to the 
concept of "SqlDialect", so i want to see if we can unify them, a SqlDialect 
config controls everything. In CALCITE-3016 [2], we are planning to introduce 
SqlFlavor which still represent a "SqlDialect", now we have 
SqlDialect/SqlConformance/DatabaseProduct/SqlFlavor, these classes really 
confused me a lot.

So let's just have a discussion about how to re-orginize them or how we can do 
them berrer :)

[1] https://issues.apache.org/jira/browse/CALCITE-3050
[2] https://issues.apache.org/jira/browse/CALCITE-3016


Best,
Danny Chan

Re: [DISCUSS] Towards Avatica 1.15.0

[Yuzhao Chen]

Francis, I will help to take this issue.

Best,
Danny Chan
在 2019年5月9日 +0800 AM6:33，Francis Chuang ，写道：
> According to the issue, it blocks CALCITE-2817 [1], however CALCITE-2817
> was in Calcite 1.19.0, which was released before Avatica 1.14.0.
>
> I am happy to revert CALCITE-2845 as I am pretty keen to get the ball
> rolling for Avatica 1.15.0, but can someone please confirm that it's
> safe to revert CALCITE-2845? We don't want to end up in a situation
> where Avatica 1.15.0 is broken and ends up blocking Avatica-Go and
> Calcite again.
>
> [1] https://issues.apache.org/jira/browse/CALCITE-2817
>
> On 8/05/2019 11:56 pm, Stamatis Zampetakis wrote:
> > Since nobody is actively working on CALCITE-3040 and we would like to
> > release 1.15.0 relatively soon, the only viable solution is to revert
> > CALCITE-2845, hoping that people who downloaded 1.14.0 are not making
> > changes to their code based on that.
> >
> > On Wed, May 8, 2019 at 12:29 AM Francis Chuang 
> > wrote:
> >
> > > Could someone please have a look at CALCITE-3040 [1]?
> > >
> > > CALCITE-3040 is a blocker for 1.15.0 and I can make rc0 available for
> > > voting once it's fixed.
> > >
> > > Francis
> > >
> > > [1] https://issues.apache.org/jira/browse/CALCITE-3040
> > >
> > > On 1/05/2019 10:47 am, Francis Chuang wrote:
> > > > After Avatica 1.14.0 was released, it was reported that the fix for
> > > > CALCITE-2845 [1] broke some existing clients. This is currently being
> > > > tracked in CALCITE-3040 [2].
> > > >
> > > > We need to release Avatica 1.15.0 to fix this issue, so that Calcite can
> > > > upgrade its version of Avatica.
> > > >
> > > > While this is a tiny release, it's also possible to include other fixes,
> > > > if required. Other than CALCITE-3040, are there any other issues you
> > > > guys would like to see fixed?
> > > >
> > > > Francis
> > > >
> > > > [1] https://issues.apache.org/jira/browse/CALCITE-2845
> > > > [2] https://issues.apache.org/jira/browse/CALCITE-3040
> > >
> >

Re: [DISCUSS] Towards Avatica 1.15.0

[Yuzhao Chen]

Broadly I think redundant error msg is much better than nothing, so revert 
CALCITE-2845 is more accepttable.

It seems the most breaking changes are from [1], but there are no more tests so 
I just revert CALCITE-2845, and do these check:

1. Revert CALCITE-2845
1. Test Calcite-Avatica master branch locally and passed
2. Change Calcite master's avatica dependency version to 1.15.0-SNAPSHOT and 
test locally, also passed

Are there any other tests that I should do ? I’m wandering if I missed 
something.

[1] 
https://github.com/apache/calcite-avatica/blob/b7c6de180aa2f9cfa840504012d42c625595161d/core/src/main/java/org/apache/calcite/avatica/AvaticaConnection.java#L559

Best,
Danny Chan
在 2019年5月9日 +0800 AM9:34，Francis Chuang ，写道：
> Many thanks, Danny!
>
> On 9/05/2019 11:33 am, Yuzhao Chen wrote:
> > Francis, I will help to take this issue.
> >
> > Best,
> > Danny Chan
> > 在 2019年5月9日 +0800 AM6:33，Francis Chuang ，写道：
> > > According to the issue, it blocks CALCITE-2817 [1], however CALCITE-2817
> > > was in Calcite 1.19.0, which was released before Avatica 1.14.0.
> > >
> > > I am happy to revert CALCITE-2845 as I am pretty keen to get the ball
> > > rolling for Avatica 1.15.0, but can someone please confirm that it's
> > > safe to revert CALCITE-2845? We don't want to end up in a situation
> > > where Avatica 1.15.0 is broken and ends up blocking Avatica-Go and
> > > Calcite again.
> > >
> > > [1] https://issues.apache.org/jira/browse/CALCITE-2817
> > >
> > > On 8/05/2019 11:56 pm, Stamatis Zampetakis wrote:
> > > > Since nobody is actively working on CALCITE-3040 and we would like to
> > > > release 1.15.0 relatively soon, the only viable solution is to revert
> > > > CALCITE-2845, hoping that people who downloaded 1.14.0 are not making
> > > > changes to their code based on that.
> > > >
> > > > On Wed, May 8, 2019 at 12:29 AM Francis Chuang 
> > > > 
> > > > wrote:
> > > >
> > > > > Could someone please have a look at CALCITE-3040 [1]?
> > > > >
> > > > > CALCITE-3040 is a blocker for 1.15.0 and I can make rc0 available for
> > > > > voting once it's fixed.
> > > > >
> > > > > Francis
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/CALCITE-3040
> > > > >
> > > > > On 1/05/2019 10:47 am, Francis Chuang wrote:
> > > > > > After Avatica 1.14.0 was released, it was reported that the fix for
> > > > > > CALCITE-2845 [1] broke some existing clients. This is currently 
> > > > > > being
> > > > > > tracked in CALCITE-3040 [2].
> > > > > >
> > > > > > We need to release Avatica 1.15.0 to fix this issue, so that 
> > > > > > Calcite can
> > > > > > upgrade its version of Avatica.
> > > > > >
> > > > > > While this is a tiny release, it's also possible to include other 
> > > > > > fixes,
> > > > > > if required. Other than CALCITE-3040, are there any other issues you
> > > > > > guys would like to see fixed?
> > > > > >
> > > > > > Francis
> > > > > >
> > > > > > [1] https://issues.apache.org/jira/browse/CALCITE-2845
> > > > > > [2] https://issues.apache.org/jira/browse/CALCITE-3040
> > > > >
> > > >
> >

Re: [DISCUSS] Towards Avatica 1.15.0

[Yuzhao Chen]

I test Drill 1.16.0 + Squirrel SQL 3.9 + avatica 1.15.0-SNAPSHOT and the error 
msg comes back.

Best,
Danny Chan
在 2019年5月9日 +0800 PM12:12，Francis Chuang ，写道：
> Thanks for looking into this, Danny!
>
> I think that covers all bases. My only suggestion would be to see if you
> can use Squirrel SQL as the client and see if the error messages come
> back:
> https://issues.apache.org/jira/browse/CALCITE-3040?focusedCommentId=16830674&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16830674
>
> On 9/05/2019 1:21 pm, Yuzhao Chen wrote:
> > Broadly I think redundant error msg is much better than nothing, so revert 
> > CALCITE-2845 is more accepttable.
> >
> > It seems the most breaking changes are from [1], but there are no more 
> > tests so I just revert CALCITE-2845, and do these check:
> >
> > 1. Revert CALCITE-2845
> > 1. Test Calcite-Avatica master branch locally and passed
> > 2. Change Calcite master's avatica dependency version to 1.15.0-SNAPSHOT 
> > and test locally, also passed
> >
> > Are there any other tests that I should do ? I’m wandering if I missed 
> > something.
> >
> > [1] 
> > https://github.com/apache/calcite-avatica/blob/b7c6de180aa2f9cfa840504012d42c625595161d/core/src/main/java/org/apache/calcite/avatica/AvaticaConnection.java#L559
> >
> > Best,
> > Danny Chan
> > 在 2019年5月9日 +0800 AM9:34，Francis Chuang ，写道：
> > > Many thanks, Danny!
> > >
> > > On 9/05/2019 11:33 am, Yuzhao Chen wrote:
> > > > Francis, I will help to take this issue.
> > > >
> > > > Best,
> > > > Danny Chan
> > > > 在 2019年5月9日 +0800 AM6:33，Francis Chuang ，写道：
> > > > > According to the issue, it blocks CALCITE-2817 [1], however 
> > > > > CALCITE-2817
> > > > > was in Calcite 1.19.0, which was released before Avatica 1.14.0.
> > > > >
> > > > > I am happy to revert CALCITE-2845 as I am pretty keen to get the ball
> > > > > rolling for Avatica 1.15.0, but can someone please confirm that it's
> > > > > safe to revert CALCITE-2845? We don't want to end up in a situation
> > > > > where Avatica 1.15.0 is broken and ends up blocking Avatica-Go and
> > > > > Calcite again.
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/CALCITE-2817
> > > > >
> > > > > On 8/05/2019 11:56 pm, Stamatis Zampetakis wrote:
> > > > > > Since nobody is actively working on CALCITE-3040 and we would like 
> > > > > > to
> > > > > > release 1.15.0 relatively soon, the only viable solution is to 
> > > > > > revert
> > > > > > CALCITE-2845, hoping that people who downloaded 1.14.0 are not 
> > > > > > making
> > > > > > changes to their code based on that.
> > > > > >
> > > > > > On Wed, May 8, 2019 at 12:29 AM Francis Chuang 
> > > > > > 
> > > > > > wrote:
> > > > > >
> > > > > > > Could someone please have a look at CALCITE-3040 [1]?
> > > > > > >
> > > > > > > CALCITE-3040 is a blocker for 1.15.0 and I can make rc0 available 
> > > > > > > for
> > > > > > > voting once it's fixed.
> > > > > > >
> > > > > > > Francis
> > > > > > >
> > > > > > > [1] https://issues.apache.org/jira/browse/CALCITE-3040
> > > > > > >
> > > > > > > On 1/05/2019 10:47 am, Francis Chuang wrote:
> > > > > > > > After Avatica 1.14.0 was released, it was reported that the fix 
> > > > > > > > for
> > > > > > > > CALCITE-2845 [1] broke some existing clients. This is currently 
> > > > > > > > being
> > > > > > > > tracked in CALCITE-3040 [2].
> > > > > > > >
> > > > > > > > We need to release Avatica 1.15.0 to fix this issue, so that 
> > > > > > > > Calcite can
> > > > > > > > upgrade its version of Avatica.
> > > > > > > >
> > > > > > > > While this is a tiny release, it's also possible to include 
> > > > > > > > other fixes,
> > > > > > > > if required. Other than CALCITE-3040, are there any other 
> > > > > > > > issues you
> > > > > > > > guys would like to see fixed?
> > > > > > > >
> > > > > > > > Francis
> > > > > > > >
> > > > > > > > [1] https://issues.apache.org/jira/browse/CALCITE-2845
> > > > > > > > [2] https://issues.apache.org/jira/browse/CALCITE-3040
> > > > > > >
> > > > > >
> > > >
> >

Re: [DISCUSS] Towards Avatica 1.15.0

[Yuzhao Chen]

Fired a PR in https://github.com/apache/calcite-avatica/pull/97

Best,
Danny Chan
在 2019年5月9日 +0800 PM1:52，Francis Chuang ，写道：
> Thanks, Danny! If you can open a PR for it, I'll try to merge it in
> today and make 1.15.0-rc0 available for voting.
>
> Francis
>
> On 9/05/2019 3:49 pm, Yuzhao Chen wrote:
> > I test Drill 1.16.0 + Squirrel SQL 3.9 + avatica 1.15.0-SNAPSHOT and the 
> > error msg comes back.
> >
> > Best,
> > Danny Chan
> > 在 2019年5月9日 +0800 PM12:12，Francis Chuang ，写道：
> > > Thanks for looking into this, Danny!
> > >
> > > I think that covers all bases. My only suggestion would be to see if you
> > > can use Squirrel SQL as the client and see if the error messages come
> > > back:
> > > https://issues.apache.org/jira/browse/CALCITE-3040?focusedCommentId=16830674&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16830674
> > >
> > > On 9/05/2019 1:21 pm, Yuzhao Chen wrote:
> > > > Broadly I think redundant error msg is much better than nothing, so 
> > > > revert CALCITE-2845 is more accepttable.
> > > >
> > > > It seems the most breaking changes are from [1], but there are no more 
> > > > tests so I just revert CALCITE-2845, and do these check:
> > > >
> > > > 1. Revert CALCITE-2845
> > > > 1. Test Calcite-Avatica master branch locally and passed
> > > > 2. Change Calcite master's avatica dependency version to 
> > > > 1.15.0-SNAPSHOT and test locally, also passed
> > > >
> > > > Are there any other tests that I should do ? I’m wandering if I missed 
> > > > something.
> > > >
> > > > [1] 
> > > > https://github.com/apache/calcite-avatica/blob/b7c6de180aa2f9cfa840504012d42c625595161d/core/src/main/java/org/apache/calcite/avatica/AvaticaConnection.java#L559
> > > >
> > > > Best,
> > > > Danny Chan
> > > > 在 2019年5月9日 +0800 AM9:34，Francis Chuang ，写道：
> > > > > Many thanks, Danny!
> > > > >
> > > > > On 9/05/2019 11:33 am, Yuzhao Chen wrote:
> > > > > > Francis, I will help to take this issue.
> > > > > >
> > > > > > Best,
> > > > > > Danny Chan
> > > > > > 在 2019年5月9日 +0800 AM6:33，Francis Chuang 
> > > > > > ，写道：
> > > > > > > According to the issue, it blocks CALCITE-2817 [1], however 
> > > > > > > CALCITE-2817
> > > > > > > was in Calcite 1.19.0, which was released before Avatica 1.14.0.
> > > > > > >
> > > > > > > I am happy to revert CALCITE-2845 as I am pretty keen to get the 
> > > > > > > ball
> > > > > > > rolling for Avatica 1.15.0, but can someone please confirm that 
> > > > > > > it's
> > > > > > > safe to revert CALCITE-2845? We don't want to end up in a 
> > > > > > > situation
> > > > > > > where Avatica 1.15.0 is broken and ends up blocking Avatica-Go and
> > > > > > > Calcite again.
> > > > > > >
> > > > > > > [1] https://issues.apache.org/jira/browse/CALCITE-2817
> > > > > > >
> > > > > > > On 8/05/2019 11:56 pm, Stamatis Zampetakis wrote:
> > > > > > > > Since nobody is actively working on CALCITE-3040 and we would 
> > > > > > > > like to
> > > > > > > > release 1.15.0 relatively soon, the only viable solution is to 
> > > > > > > > revert
> > > > > > > > CALCITE-2845, hoping that people who downloaded 1.14.0 are not 
> > > > > > > > making
> > > > > > > > changes to their code based on that.
> > > > > > > >
> > > > > > > > On Wed, May 8, 2019 at 12:29 AM Francis Chuang 
> > > > > > > > 
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Could someone please have a look at CALCITE-3040 [1]?
> > > > > > > > >
> > > > > > > > > CALCITE-3040 is a blocker for 1.15.0 and I can make rc0 
> > > > > > > > > available for
> > > > > > > > > voting once it's fixed.
> > > > > > > > >
> > > > > > > > > Francis
> > > > > > > > >
> > > > > > > > > [1] https://issues.apache.org/jira/browse/CALCITE-3040
> > > > > > > > >
> > > > > > > > > On 1/05/2019 10:47 am, Francis Chuang wrote:
> > > > > > > > > > After Avatica 1.14.0 was released, it was reported that the 
> > > > > > > > > > fix for
> > > > > > > > > > CALCITE-2845 [1] broke some existing clients. This is 
> > > > > > > > > > currently being
> > > > > > > > > > tracked in CALCITE-3040 [2].
> > > > > > > > > >
> > > > > > > > > > We need to release Avatica 1.15.0 to fix this issue, so 
> > > > > > > > > > that Calcite can
> > > > > > > > > > upgrade its version of Avatica.
> > > > > > > > > >
> > > > > > > > > > While this is a tiny release, it's also possible to include 
> > > > > > > > > > other fixes,
> > > > > > > > > > if required. Other than CALCITE-3040, are there any other 
> > > > > > > > > > issues you
> > > > > > > > > > guys would like to see fixed?
> > > > > > > > > >
> > > > > > > > > > Francis
> > > > > > > > > >
> > > > > > > > > > [1] https://issues.apache.org/jira/browse/CALCITE-2845
> > > > > > > > > > [2] https://issues.apache.org/jira/browse/CALCITE-3040
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> >

Re: Fixing parenthesized joins

[Yuzhao Chen]

Set it, good luck !

Best,
Danny Chan
在 2019年5月10日 +0800 AM5:21，Muhammad Gelbana ，写道：
> I opened a PR to fix CALCITE-35 [1] but I can't set the affected version
> (1.19) and component (Babel parser).
>
> Would someone please set those ?
> Or may be grant me the necessary privileges to do so on my own ?
>
> [1] https://issues.apache.org/jira/browse/CALCITE-35
>
> Thanks,
> Gelbana

Re: inter-database queries

[Yuzhao Chen]

That's great.

Re: [ANNOUNCE] Apache Calcite Avatica 1.15.0 released

[Yuzhao Chen]

Nice job ! Francis, thx for your work.

Best,
Danny Chan
在 2019年5月13日 +0800 AM9:30，Francis Chuang ，写道：
> The Apache Calcite team is pleased to announce the release of Apache
> Calcite Avatica 1.15.0.
>
> Avatica is a framework for building database drivers. Avatica defines a
> wire API and serialization mechanism for clients to communicate with a
> server as a proxy to a database. The reference Avatica client and server
> are implemented in Java and communicate over HTTP. Avatica is a
> sub-project of Apache Calcite
>
> Apache Calcite Avatica 1.15.0 is a small release that fixes a regression
> causing error messages to not be propagated to clients. There is also a
> minor update to the docker release script to automate the upload of
> release artifacts and build promotion. Consumers of Avatica are
> encouraged to skip the 1.14.0 release and upgrade directly to 1.15.0 due
> to the aforementioned regression. For a full list of changes, please see
> the release notes:
>
> https://calcite.apache.org/avatica/docs/history.html#v1-15-0
>
> The release is available here:
>
> https://calcite.apache.org/avatica/downloads/avatica.html
>
> We welcome your help and feedback. For more information on how to report
> problems and get involved, visit the project website at:
>
> https://calcite.apache.org/avatica/
>
> or the Apache Calcite project website:
>
> https://calcite.apache.org/
>
> Thanks to everyone involved!
>
> Francis Chuang, on behalf of the Apache Calcite team.

Re: Using secondary indexes for accessing tables

[Yuzhao Chen]

Roman Kondakov, I didn’t look into your use case details, but the secondary 
indexes may be used mainly for 2 scenarios:

1. Table index scan
2. The dim table join (key look up)

Calcite does not support secondary indexes metadata yet, or even (indexes). But 
I think you can extend it with the MetadataHandler API [1]. Then you can query 
the indexing info in your planner rules for some special promotion[2], or use 
the Programs [3] directly.

[1] 
https://github.com/apache/calcite/blob/6afa38bae794462e6e250237a1b60cc4220b2885/core/src/main/java/org/apache/calcite/rel/metadata/BuiltInMetadata.java#L40
[2] 
https://github.com/apache/calcite/blob/6afa38bae794462e6e250237a1b60cc4220b2885/core/src/main/java/org/apache/calcite/plan/RelOptRule.java#L41
[3] 
https://github.com/apache/calcite/blob/6afa38bae794462e6e250237a1b60cc4220b2885/core/src/main/java/org/apache/calcite/tools/Program.java#L36

Best,
Danny Chan
在 2019年5月12日 +0800 AM5:16，Roman Kondakov ，写道：
> Hi,
>
> I am investigating the possibility of using Apache Calcite as a
> cost-based optimizer for Apache Ignite project [1]. Apache Calcite looks
> a very powerful and versatile tool for the such kind of tasks. I've
> started this investigation a couple days ago - read docs, played with
> examples, explored some existing integrations (i.e. hive, drill,
> phoenix, etc.)
>
> As I understand [2] Calcite doesn't support secondary indexes, isn't it?
> If so, is there any workarounds to handle it or integrations where this
> feature is implemented? Apache Ignite allows users to create B+tree
> based indexes over the tables but lacks of a good query optimizer. It
> would be great if the execution plan generated by Calcite contains not
> only full table scans but also index scans when appropriate.
>
> Thank you in advance!
>
> [1] https://ignite.apache.org/
> [2]http://mail-archives.apache.org/mod_mbox/calcite-dev/201505.mbox/%3CCAAqfHNoQnr%2BeUVoH9nFxhB7FEDwdYAjV_Smq%3DY75fxsqGs66Yw%40mail.gmail.com%3E
> --
> Kind Regards
> Roman Kondakov
>

Re: [ANNOUNCE] New committer: Danny Chan

[Yuzhao Chen]

Thank you everyone for your kind messages.

Currently I am working in Alibaba Blink SQL Engine team in Hangzhou, Zhejiang, 
China. We are developing a production version of Apache Flink. Our team has 
done many promotions for flink-table module and recently we are merging and 
contributing our codebase to the Apache Flink community.

It is my honor to be part of Apache Calcite community, I will contribute 
continuously to this great project.

Best,
Danny Chan
在 2019年5月14日 +0800 AM6:40，Francis Chuang ，写道：
> Apache Calcite's Project Management Committee (PMC) has invited Danny
> Chan to become a committer, and we are pleased to announce that he has
> accepted.
>
> Danny has been a prolific contributor to Calcite, with CALCITE-2969
> being one of his more complex contributions to date. He has also been
> extremely active on our mailing lists, contributing to many design
> discussions.
>
> Danny, welcome, thank you for your contributions, and we look forward
> your further interactions with the community! If you wish, please feel
> free to tell us more about yourself and what you are working on.
>
> Francis (on behalf of the Apache Calcite PMC)

Re: Customizing Join Algorithm

[Yuzhao Chen]

It seems that you just need a new convention which integrated with you custom 
algorithm, you can reference EnumerableJoin [1] for some inspirations.

[1] 
https://github.com/apache/calcite/blob/c6b6800c220e513f1d9a2b167b2f14cb689c0b06/core/src/main/java/org/apache/calcite/adapter/enumerable/EnumerableJoin.java#L47

Best,
Danny Chan
在 2019年5月17日 +0800 AM8:38，Valeriy Trofimov ，写道：
> Thank you for the replies. I though that rules are used during query
> planning/optimization stage, not to manipulate real data returned after
> executing the query. What I want to do is to use a custom algorithm to join
> data returned from the left and right side of the query after the query is
> executed. So, in my case it would probably be writing a rule that says "if
> you see keyword JOIN, then change the query to not do the join, but to get
> me data from left and right sides and then do the join using my own
> algorithm". Let me know if it's possible.
>
> Thanks,
> Val
>
> On Thu, May 16, 2019 at 3:24 PM Parimarjan Negi 
> wrote:
>
> > Hi,
> >
> > I have been working on a research project that tries out various strategies
> > for join algorithms. Here are some implementations of various baselines
> > like exhaustive search, left deep search etc. which may be helpful for you.
> > Here is the implementation for the exhaustive search rule which uses
> > Calcite' implementations (like BushyJoinOrderRule) as a base:
> >
> >
> > https://github.com/parimarjan/query-optimizer/blob/master/src/main/java/ExhaustiveDPJoinOrderRule.java#L40
> >
> > Best,
> > Pari
> >
> > On Thu, May 16, 2019 at 4:37 PM Ivan Grgurina 
> > wrote:
> >
> > > What do you mean by "custom join algorithm"? You seem to be mixing "SQL
> > > join queries" with "Avatica". Calcite uses planner rules on relational
> > > algebra level, you can write a rule that transforms Join RelNode with
> > your
> > > algorithm. Is that what you want to do?
> > >
> > > ig
> > >
> > >
> > >
> > >
> > > 
> > > From: Valeriy Trofimov 
> > > Sent: Thursday, May 16, 2019 9:47 PM
> > > To: dev@calcite.apache.org
> > > Subject: Customizing Join Algorithm
> > >
> > > Hi All,
> > >
> > > I want to use a custom join algorithm in Calcite SQL join queries. My
> > > questions are:
> > >
> > > 1. Is it possible to do?
> > > 2. If yes, what part of Calcite should I update?
> > >
> > > I've tried to figure it out myself by debugging Calcite join unit tests,
> > > but it looks like Calcite uses Avatica for this.
> > >
> > > Thanks,
> > > Val
> > >
> >

Re: Giving the Calcite logo some love

[Yuzhao Chen]

Sounds great ! Look forward to the new logo !

Best,
Danny Chan
在 2019年5月19日 +0800 PM11:25，Daniel Gruno ，写道：
> I have them at home (traveling atm), can get to them around June 6th if
> need be.
>
> On 19/05/2019 11.01, Stamatis Zampetakis wrote:
> > Hi all,
> >
> > We started this discussion about a year ago and many people were
> > positive with the idea of having a new logo for Calcite.
> > We had some nice proposals at the time and maybe now somebody else has
> > also new ideas/designs to contribute.
> >
> > @Daniel: Is there any chance that you still have the logos you proposed
> > somewhere available? The old link [1] does not work anymore.
> >
> > Best,
> > Stamatis
> >
> > [1] http://www.apache.org/logos/comdev-test/res/calcite/calcite-proposed.svg
> >
> >
> > On Thu, Aug 30, 2018 at 12:02 AM Stamatis Zampetakis  > > wrote:
> >
> > Vladimir>Could you flip rhombus so it goes right-up?
> >
> > https://svgshare.com/s/86r
> >
> > I hope this is what you meant.
> >
> > Best,
> > Stamatis
> >
> > Στις Τετ, 29 Αυγ 2018 στις 6:30 μ.μ., ο/η Michael Mior
> > mailto:mm...@apache.org>> έγραψε:
> >
> > Just a note since we're on the topic that whatever logos we come
> > up with
> > should be sure to have TM clearly indicated.
> >
> > --
> > Michael Mior
> > mm...@apache.org 
> >
> >
> >
> > Le mer. 29 août 2018 à 03:13, Julian Hyde
> > mailto:jhyde.apa...@gmail.com>> a écrit :
> >
> > > Yes indeed!
> > >
> > > If someone feels inspired to produce a logo, here’s my
> > suggestion of a
> > > theme/image: a spider, specifically a Barn Spider (Araneus
> > Cavaticus)[1].
> > > It was the origin of the name “avatica”, connects and spins
> > webs, and the
> > > eponymous individual in Charlotte’s Web had rather exceptional
> > > communication skills.
> > >
> > > Julian
> > >
> > > [1] https://en.m.wikipedia.org/wiki/Barn_spider
> > >
> > > > On Aug 28, 2018, at 9:49 PM, Francis Chuang
> > mailto:francischu...@apache.org>>
> > > wrote:
> > > >
> > > > The designs I have seen so far look really good! Would it
> > also make
> > > sense to design a variant for Avatica as well? This is what
> > the current
> > > Avatica logo looks like:
> > https://calcite.apache.org/avatica/img/logo.png
> > > >
> > > > Francis
> > > >
> > > > > On 29/08/2018 7:08 AM, Vladimir Sitnikov wrote:
> > > > > Stamatis>How about something like the following:
> > > > >
> > > > > There's left-to-right vs right-to-left issue, however I
> > would claim that
> > > > > the direction of improvement is right+up.
> > > > > For instance: BTC price is good when plots go to the right
> > and go
> > > upward.
> > > > >
> > > > > https://svgur.com/s/83y is slanted backward.
> > > > > That creates perception of "Calcite holding back the
> > progress" or
> > > "Apache
> > > > > pushing C away" or something like that.
> > > > > Could you flip rhombus so it goes right-up?
> > > > >
> > > > >
> > > > > Vladimir
> > > > >
> > > >
> > >
> >
>

Re: Has issue when parse DDL file using SqlDdlParserImpl

[Yuzhao Chen]

Sorry I only found the grammar template in our site [1] and some simple test 
demos in ServerParserTest [2].

I think we should support detail doc with demos of all kinds of supported DDLs 
and DMLs, will fire a JIRA about this.

[1] http://calcite.apache.org/docs/reference.html#ddl-extensions
[2] 
https://github.com/apache/calcite/blob/61b858db13c2e1997e92172a09f38c639ffdceee/server/src/test/java/org/apache/calcite/test/ServerParserTest.java#L136

Best,
Danny Chan
在 2019年5月22日 +0800 AM7:19，Phạm Minh Đức ，写道：
> Dear Dev Team's Apache Calcite,
> My name is Duc. I'm working at Toshiba Group in Vietnam.
> I'm implementing how to parser DDL file using library Apache Calcite.
> I'm trying with below solution, but has exception is thrown and i don't know 
> that why.
> InputStream ddlFile = new FileInputStream("ddlTest.sql");
> SqlDdlParserImpl ddlParser = new SqlDdlParserImpl(ddlFile,"UTF8");
> SqlNode node = ddlParser.SqlStmt();
> Could you please send me a example of DDL file or give me a suggestion about 
> this proplem.
> Thank you very much for your hear m !
> Best regards,
> Pham Minh Duc.
> --
> This mail was scanned by BitDefender
> For more information please visit http://www.bitdefender.com

Re: Elasticsearch Adapter. Dropping support for 2.x and 5.x versions.

[Yuzhao Chen]

+1 for only support one valid version.

Best,
Danny Chan
在 2019年5月22日 +0800 AM6:48，Julian Hyde ，写道：
> +1
>
> Better to stay current. And it’s inefficient for us to support multiple 
> versions.
>
> If people need 2.x or 5.x they can use an older version of Calcite (or roll 
> their own version of the adapter).
>
> > On May 21, 2019, at 3:46 PM, Stamatis Zampetakis  wrote:
> >
> > +1
> >
> > On Wed, May 22, 2019 at 12:44 AM Francis Chuang 
> > wrote:
> >
> > > +1 As those versions are already EoLed and are no longer supported and
> > > will not receive any patches, it makes sense for us to stop supporting
> > > them as well.
> > >
> > > On 22/05/2019 8:36 am, Andrei Sereda wrote:
> > > > Hello,
> > > >
> > > > Following CALCITE-3023 Upgrade to Elasticsearch 7.0.0 [1] we'll be
> > > dropping
> > > > support for ES types (not to be confused with calcite types) in elastic
> > > > search [2].
> > > >
> > > > Are people OK with requiring at least 6.x for elastic search adapter ?
> > > >
> > > > According to official support schedule [3] 2.x, 5.x and 6.0.x are 
> > > > already
> > > > EoL.
> > > >
> > > > Regards,
> > > > Andrei.
> > > >
> > > > [1] https://issues.apache.org/jira/browse/CALCITE-3023
> > > > [2]
> > > >
> > > https://www.elastic.co/guide/en/elasticsearch/reference/7.0/removal-of-types.html
> > > > [3] https://www.elastic.co/support/eol
> > > >
> > >
>

Support complete implicit type coercion (DISCUSSION)

[Yuzhao Chen]

Hi, guys.

The implicit type coercion is almost supported by every production 
RDBMS(MYSQL[1], ORACLE[2], SQLSERVER[3]), also some Hadoop data warehouse 
facilitates like HIVE.

As a query optimization engine of many comutation engines(like Apache Flink) 
and some OLAP engines(like Apache Drill), Calcite would supply better 
compatibility for sql query for the underlying engines it adapter with with 
implicit type coercion. There are already some jira issues that are relative 
with this topic more or less:

1. CALCITE-2992: Enhance implicit conversions when generating hash join keys 
for an equiCondition
2. CALCITE-3002: Case statement fails with: SqlValidatorException: Cannot apply 
'=' to arguments of type ' = '
3. CALCITE-1531: SqlValidatorException when boolean operators are used with NULL
4. CALCITE-3081: https://issues.apache.org/jira/browse/CALCITE-3081
5. CALCITE-2829: Use consistent types when processing ranges

I have fired a issue CALCITE-2302 [4] about 1 year ago, with a design 
doc(sowehow rough).

Maybe we should fire a new discussion here, and hope for your suggesions :)

[1] https://dev.mysql.com/doc/refman/5.5/en/type-conversion.html
[2] https://docs.oracle.com/cd/B12037_01/server.101/b10759/sql_elements002.htm
[3] 
https://docs.microsoft.com/en-us/sql/t-sql/data-types/data-type-conversion-database-engine?view=sql-server-2017
[4] https://issues.apache.org/jira/browse/CALCITE-2302
[5] 
https://docs.google.com/document/d/1g2RUnLXyp_LjUlO-wbblKuP5hqEu3a_2Mt2k4dh6RwU/edit#heading=h.77f83nidn37j

Best,
Danny Chan

Re: Support complete implicit type coercion (DISCUSSION)

[Yuzhao Chen]

Thanks for your response, Zhu Feng, I totally agree with you.

The first thing we should consider with implicit type coercion is to make it 
pluggable. In the original PR[1],
I make implementation of different SqlNodes into separate methods, and we can 
inherent the TypeCoercion  interface to make some extension for different Sql 
Dialect.

But I agree with you, we should make the Sql Dialect somehow bindable with 
different TypeCoercion implementations. So user can customize their 
transformation behaviors based on the Sql Dialect they use.

[1] https://github.com/apache/calcite/pull/706

Best,
Danny Chan
在 2019年5月27日 +0800 PM12:51，Zhu Feng ，写道：
> Hi Yuzhao:
> Thanks for raising this discussion. I think this feature is significant to
> Calcite.
> AFAIK, there is no standard on implicit type coercion. Even for those
> widely-adopted RDBMSs (ORACLE, SqlSever, and etc.), we can find some
> "unreasonable" corner cases that are not as user expected.
>
> There are too many factors. Take (1 > '1') as an example, is "casting one
> type to another type directly" (1 > cast('1' as int)) or "casting to common
> types" (cast('1' as double) > cast('1' as double)) more suitable?
> How about (1>'1')?
>
> From my point of view, we can make implicit type coercion as dialect
> interfaces, and provide SqlDialect-specific implementations. In recent
> years, our in-house platform has evolved many times from Oracle to Hive and
> Spark SQL.
> When migrating from one system to another, problems caused by implicit type
> coercion brought us much pain. Different runtime conversion beheviors lead
> to different results even for the same query.
>
> I think a pluggable or dialect-configurable design benefits not only
> Calcite itself but also the engines (Flink) that use Calcite.
>
> Best,
> DonnyZone
>
> Kurt Young  于2019年5月27日周一 上午11:34写道：
>
> > Thanks Danny for pushing this.
> >
> > Just like you said, different engines may use different strategies for
> > implicit type cast, so i
> > think making the whole mechanism pluggable would be a good idea.
> >
> > Best,
> > Kurt
> >
> >
> > On Mon, May 27, 2019 at 11:08 AM Haisheng Yuan 
> > wrote:
> >
> > > Thanks Danny for bringing it up.
> > > This is a useful feature, we should push it forward.
> > >
> > > I went through the design doc, looks good in general.
> > > I will also spend some time on the pull request 706.
> > >
> > > Thanks ~
> > > Haisheng Yuan
> > > --
> > > 发件人：Yuzhao Chen
> > > 日 期：2019年05月27日 10:20:47
> > > 收件人：
> > > 主 题：Support complete implicit type coercion (DISCUSSION)
> > >
> > > Hi, guys.
> > >
> > > The implicit type coercion is almost supported by every production
> > > RDBMS(MYSQL[1], ORACLE[2], SQLSERVER[3]), also some Hadoop data warehouse
> > > facilitates like HIVE.
> > >
> > > As a query optimization engine of many comutation engines(like Apache
> > > Flink) and some OLAP engines(like Apache Drill), Calcite would supply
> > > better compatibility for sql query for the underlying engines it adapter
> > > with with implicit type coercion. There are already some jira issues that
> > > are relative with this topic more or less:
> > >
> > > 1. CALCITE-2992: Enhance implicit conversions when generating hash join
> > > keys for an equiCondition
> > > 2. CALCITE-3002: Case statement fails with: SqlValidatorException: Cannot
> > > apply '=' to arguments of type ' = '
> > > 3. CALCITE-1531: SqlValidatorException when boolean operators are used
> > > with NULL
> > > 4. CALCITE-3081: https://issues.apache.org/jira/browse/CALCITE-3081
> > > 5. CALCITE-2829: Use consistent types when processing ranges
> > >
> > > I have fired a issue CALCITE-2302 [4] about 1 year ago, with a design
> > > doc(sowehow rough).
> > >
> > > Maybe we should fire a new discussion here, and hope for your suggesions
> > :)
> > >
> > > [1] https://dev.mysql.com/doc/refman/5.5/en/type-conversion.html
> > > [2]
> > >
> > https://docs.oracle.com/cd/B12037_01/server.101/b10759/sql_elements002.htm
> > > [3]
> > >
> > https://docs.microsoft.com/en-us/sql/t-sql/data-types/data-type-conversion-database-engine?view=sql-server-2017
> > > [4] https://issues.apache.org/jira/browse/CALCITE-2302
> > > [5]
> > >
> > https://docs.google.com/document/d/1g2RUnLXyp_LjUlO-wbblKuP5hqEu3a_2Mt2k4dh6RwU/edit#heading=h.77f83nidn37j
> > >
> > > Best,
> > > Danny Chan
> > >
> > >
> >

Re: [DISCUSS] Towards Calcite 1.20.0

[Yuzhao Chen]

Thanks so much for your work, Michael,

Let's get CALCITE-3055 into 1.20 version, because  it fix an important function 
regression. I will merge it in if finishes the review.

[1] https://github.com/apache/calcite/pull/1230/files

Best,
Danny Chan
在 2019年5月28日 +0800 AM1:41，Michael Mior ，写道：
> Thanks Julian! I'm hoping we can get 1.20.0 out this week. I did some
> cleanup on JIRA and pinged a few for some status updates and I think
> we're in reasonably good shape.
>
> --
> Michael Mior
> mm...@apache.org
>
> Le mar. 21 mai 2019 à 19:13, Julian Hyde  a écrit :
> >
> > I agree with Michael’s timeline “a week or two”. How about code freeze on 
> > Friday 31st May, 10 days from now, and RC0 on Mon 3rd June?
> >
> > I have fixes for the following:
> >
> > * [CALCITE-3050] Integrate SqlDialect and SqlParser.Config
> > * [CALCITE-3022] Babel: Various SQL parsing issues
> > * [CALCITE-3047] In JDBC adapter, expose multiple schemas of the back-end 
> > database
> > * [CALCITE-3048] Improve how JDBC adapter deduces current schema on Redshift
> >
> > and I will commit them before the release. I will also help
> >
> > * [CALCITE-2969] Improve design of join-like relational expressions queries
> >
> > over the finishing line.
> >
> > Julian
> >
> >
> > > On May 20, 2019, at 6:06 AM, Michael Mior  wrote:
> > >
> > > Just revisiting this now that the Avatica has been fixed. I have the
> > > following list of pending JIRAs:
> > >
> > > [CALCITE-1581] UDTF like in hive
> > > [CALCITE-2812] Add algebraic operators to allow expressing recursive
> > > [CALCITE-2952] Certify Calcite on JDK 12
> > > [CALCITE-2969] Improve design of join-like relational expressions queries
> > > [CALCITE-2973] Allow theta joins that have equi conditions to be
> > > executed using a hash join algorithm
> > > [CALCITE-2992] Enhance implicit conversions when generating hash join
> > > keys for an equi condition
> > > [CALCITE-3036] Remove correlate variables from Join
> > > [CALCITE-3037] Rename EnumerableThetaJoin to EnumerableNestedLoopJoin
> > >
> > > I think we should cut a release soon to keep momentum going. I'd like
> > > to hear from those working on any of the JIRAs to see which ones could
> > > be wrapped up in the next week or two.
> > > --
> > > Michael Mior
> > > mm...@apache.org
> > >
> > > Le mar. 30 avr. 2019 à 11:15, Michael Mior  a écrit :
> > > >
> > > > Calcite 1.19.0 was released approximately one month ago. This was
> > > > later than we originally planned (although I think with good reason
> > > > and I'm happy with what made it into this release). I don't think
> > > > there's an imminent need for a new release, but I wanted to start the
> > > > discussion now that Avatica has had it's latest release.
> > > >
> > > > There's a few big things in progress below that I thought I'd see if
> > > > we want to try to include in the next release. Apologies if I missed
> > > > any.
> > > >
> > > > CALCITE-1581 UDTF like in hive
> > > > CALCITE-2952 Certify Calcite on JDK 12
> > > > CALCITE-3036 Remove correlate variables from Join
> > > > CALCITE-3037 Rename EnumerableThetaJoin to EnumerableNestedLoopJoin
> > > >
> > > > --
> > > > Michael Mior
> > > > mm...@apache.org
> >

Re: calcite close connection

[Yuzhao Chen]

What do you mean by close the Schema ?

Best,
Danny Chan
在 2019年5月28日 +0800 AM11:41，dev@calcite.apache.org，写道：
>
> Schema

Re: Support complete implicit type coercion (DISCUSSION)

[Yuzhao Chen]

Thanks Julian,

I’m glad that we can push this forward.

Some thoughts about making this feature pluggable:


• For function disable: For the current design draft, I have added a flag in 
SqlValidator so use can enable/disable all the implicit coercions.
• For adopting to different sql dialects: It should have different semantics 
for user to adopt to their semantics they want, for the first PR, i only make a 
default one, but I believe we can add more for major SqlConformance
• For making the behavior modular: For current design I split different 
coercion rules into kinds of method  implementations of interface TypeCoercion, 
for following the pattern of (SqlReturnTypeInference, SqlOperandTypeInference, 
SqlOperandTypeChecker), do you mean I need to package these rules into 
different components ? (So we can move them around for reusing)

I firstly designed the TypeCoercion component like pattern RelDataTypeFactory, 
so user can extend and tweak the logic though inheritance or extending the 
default impl( say different SQL dialect may have different TypeCoercion 
instances).

Best,
Danny Chan
在 2019年5月27日 +0800 PM3:24，Julian Hyde ，写道：
> Thanks for starting this discussion.
>
> I agree that implicit type coercion will make Calcite (and systems derived 
> from it) easier to use, and we should do it.
>
> I also agree that it should be “plugggable”, in several senses:
> * First, it should be possible to disable all implicit coercions, and adopt 
> the current behavior based on the SQL standard.
> * Second, where there are differences in semantics in major existing systems, 
> users should be able to choose which semantics they want.
> * Third, we should make the behavior modular. The existing interfaces 
> (SqlReturnTypeInference, SqlOperandTypeInference, SqlOperandTypeChecker) have 
> been very successful, and we should follow that general pattern as we 
> implement implicit type coercion.
>
> Julian
>
>
> > On May 26, 2019, at 11:39 PM, Yuzhao Chen  wrote:
> >
> > Thanks for your response, Zhu Feng, I totally agree with you.
> >
> > The first thing we should consider with implicit type coercion is to make 
> > it pluggable. In the original PR[1],
> > I make implementation of different SqlNodes into separate methods, and we 
> > can inherent the TypeCoercion interface to make some extension for 
> > different Sql Dialect.
> >
> > But I agree with you, we should make the Sql Dialect somehow bindable with 
> > different TypeCoercion implementations. So user can customize their 
> > transformation behaviors based on the Sql Dialect they use.
> >
> > [1] https://github.com/apache/calcite/pull/706
> >
> > Best,
> > Danny Chan
> > 在 2019年5月27日 +0800 PM12:51，Zhu Feng ，写道：
> > > Hi Yuzhao:
> > > Thanks for raising this discussion. I think this feature is significant to
> > > Calcite.
> > > AFAIK, there is no standard on implicit type coercion. Even for those
> > > widely-adopted RDBMSs (ORACLE, SqlSever, and etc.), we can find some
> > > "unreasonable" corner cases that are not as user expected.
> > >
> > > There are too many factors. Take (1 > '1') as an example, is "casting one
> > > type to another type directly" (1 > cast('1' as int)) or "casting to 
> > > common
> > > types" (cast('1' as double) > cast('1' as double)) more suitable?
> > > How about (1>'1')?
> > >
> > > From my point of view, we can make implicit type coercion as dialect
> > > interfaces, and provide SqlDialect-specific implementations. In recent
> > > years, our in-house platform has evolved many times from Oracle to Hive 
> > > and
> > > Spark SQL.
> > > When migrating from one system to another, problems caused by implicit 
> > > type
> > > coercion brought us much pain. Different runtime conversion beheviors lead
> > > to different results even for the same query.
> > >
> > > I think a pluggable or dialect-configurable design benefits not only
> > > Calcite itself but also the engines (Flink) that use Calcite.
> > >
> > > Best,
> > > DonnyZone
> > >
> > > Kurt Young  于2019年5月27日周一 上午11:34写道：
> > >
> > > > Thanks Danny for pushing this.
> > > >
> > > > Just like you said, different engines may use different strategies for
> > > > implicit type cast, so i
> > > > think making the whole mechanism pluggable would be a good idea.
> > > >
> > > > Best,
> > > > Kurt
>

Re: CyclicMetadataException in testPushDownJoinConditionsWithExpandedIsNotDistinctUsingCase

[Yuzhao Chen]

Thanks Ruben for your good analysis.

What I’m confused is that isn’t the static REL_BUILDER more prone to have 
concurrency problems ? And the pushed scans(EMP_SCAN and DEPT_SCAN) are all 
nodes(immutable), how could this be a problem ?

Best,
Danny Chan
在 2019年5月29日 +0800 PM5:37，Ruben Q L ，写道：
> I'm checking the commit [1] and I see something strange in RelOptUtilTest.
> Maybe I'm wrong and it is nothing, but just in case it may help:
>
> With the latest modification, it seems that we have two RelBuilder(s) in
> place:
> - A static one that is created ad-hoc on a static block to generate the
> EMP_SCAN and DEPT_SCAN RelNodes [2]
> - An instance one to be used in the tests, that is initialized on
> the @Before public void setUp() method [3]
>
> Before this commit, the EMP_SCAN / DEPT_SCAN were only used to read their
> rowTypes to test some join auxiliary methods. But the new
> tests testPushDownJoinConditions* actually build a plan and push these
> scans into the RelBuilder to be tested [4] (which is a different one than
> the static RelBuider that created the scans).
> Maybe this is no problem generally, but it can potentially be under certain
> circumstances?, which would explain the randomness of the issue.
> Could this explain the exception?
>
> [1]
> https://github.com/apache/calcite/commit/82e7d4e760cb203d31956c55e38e0fdd56119d58
>
> [2]
> https://github.com/apache/calcite/blob/ac40d6951bc8c475ca6804be6d878107cc2ebb13/core/src/test/java/org/apache/calcite/plan/RelOptUtilTest.java#L71
> [3]
> https://github.com/apache/calcite/blob/ac40d6951bc8c475ca6804be6d878107cc2ebb13/core/src/test/java/org/apache/calcite/plan/RelOptUtilTest.java#L92
> [4]
> https://github.com/apache/calcite/blob/ac40d6951bc8c475ca6804be6d878107cc2ebb13/core/src/test/java/org/apache/calcite/plan/RelOptUtilTest.java#L292
>
>
>
> Le mer. 29 mai 2019 à 02:20, Julian Hyde  a écrit :
>
> > It’s a tough call. It is probable that the problem existed already and the
> > change merely surfaced it.
> >
> > > On May 28, 2019, at 5:17 PM, Stamatis Zampetakis 
> > wrote:
> > >
> > > It is not the only test that is failing after commit [1] but all the new
> > > tests that were added.
> > >
> > > I've seen the problem on Jenkins on all JDKS but I cannot reproduce it
> > > locally.
> > > I guess we have to do with a race condition most likely due to the
> > > concurrent execution of tests with surefire.
> > >
> > > Should we revert the commit till we find a solution?
> > >
> > > [1]
> > >
> > https://github.com/apache/calcite/commit/82e7d4e760cb203d31956c55e38e0fdd56119d58
> > >
> > > On Tue, May 28, 2019 at 7:57 PM Julian Hyde  wrote:
> > >
> > > > I have seen this intermittent failure 3 times in the last week:
> > > >
> > > > [INFO] Running org.apache.calcite.plan.RelOptUtilTest
> > > > [ERROR] Tests run: 11, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> > > > 0.411 s <<< FAILURE! - in org.apache.calcite.plan.RelOptUtilTest
> > > > [ERROR]
> > > >
> > testPushDownJoinConditionsWithExpandedIsNotDistinctUsingCase(org.apache.calcite.plan.RelOptUtilTest)
> > > > Time elapsed: 0.349 s <<< ERROR!
> > > > org.apache.calcite.rel.metadata.CyclicMetadataException
> > > > at
> > > >
> > org.apache.calcite.plan.RelOptUtilTest.testPushDownJoinConditionsWithExpandedIsNotDistinctUsingCase(RelOptUtilTest.java:445)
> > > >
> > > > I have seen it on Oracle JDK 12 and OpenJDK 10. The test was only added
> > on
> > > > May 22 so I assume that it will continue to fail intermittently until
> > we do
> > > > something.
> > > >
> > > > Anyone have any ideas?
> > > >
> > > > Laurent, As you added the test can you please look into it?
> > > >
> > > > Julian
> > > >
> > > >
> >
> >

Re: [DISCUSS] Towards Calcite 1.20.0

[Yuzhao Chen]

Thanks so much for your work, Julian. And sorry for my mistake.

Best,
Danny Chan
在 2019年6月1日 +0800 AM2:28，Julian Hyde ，写道：
> How are we doing? What must-fix bugs remain?
>
> I asked Danny to fix some deprecation warnings, which he duly did[1], but now 
> I think I was mistaken, because he did so by removing a bunch of methods 
> whose arguments were the now-deprecated class SemiJoin. This has become a 
> breaking change with not even a minor release notice, and I think we should 
> back it out before 1.20. I’m going to re-open 3102 and declare it a blocker 
> for 1.20. Sorry I screwed up, Danny! Let’s discuss in the JIRA case.
>
> Julian
>
> [1] https://issues.apache.org/jira/browse/CALCITE-3102 
> <https://issues.apache.org/jira/browse/CALCITE-3102>
>
> > On May 28, 2019, at 5:18 AM, Yuzhao Chen  wrote:
> >
> > Thanks so much for your work, Michael,
> >
> > Let's get CALCITE-3055 into 1.20 version, because it fix an important 
> > function regression. I will merge it in if finishes the review.
> >
> > [1] https://github.com/apache/calcite/pull/1230/files
> >
> > Best,
> > Danny Chan
> > 在 2019年5月28日 +0800 AM1:41，Michael Mior ，写道：
> > > Thanks Julian! I'm hoping we can get 1.20.0 out this week. I did some
> > > cleanup on JIRA and pinged a few for some status updates and I think
> > > we're in reasonably good shape.
> > >
> > > --
> > > Michael Mior
> > > mm...@apache.org
> > >
> > > Le mar. 21 mai 2019 à 19:13, Julian Hyde  a écrit :
> > > >
> > > > I agree with Michael’s timeline “a week or two”. How about code freeze 
> > > > on Friday 31st May, 10 days from now, and RC0 on Mon 3rd June?
> > > >
> > > > I have fixes for the following:
> > > >
> > > > * [CALCITE-3050] Integrate SqlDialect and SqlParser.Config
> > > > * [CALCITE-3022] Babel: Various SQL parsing issues
> > > > * [CALCITE-3047] In JDBC adapter, expose multiple schemas of the 
> > > > back-end database
> > > > * [CALCITE-3048] Improve how JDBC adapter deduces current schema on 
> > > > Redshift
> > > >
> > > > and I will commit them before the release. I will also help
> > > >
> > > > * [CALCITE-2969] Improve design of join-like relational expressions 
> > > > queries
> > > >
> > > > over the finishing line.
> > > >
> > > > Julian
> > > >
> > > >
> > > > > On May 20, 2019, at 6:06 AM, Michael Mior  wrote:
> > > > >
> > > > > Just revisiting this now that the Avatica has been fixed. I have the
> > > > > following list of pending JIRAs:
> > > > >
> > > > > [CALCITE-1581] UDTF like in hive
> > > > > [CALCITE-2812] Add algebraic operators to allow expressing recursive
> > > > > [CALCITE-2952] Certify Calcite on JDK 12
> > > > > [CALCITE-2969] Improve design of join-like relational expressions 
> > > > > queries
> > > > > [CALCITE-2973] Allow theta joins that have equi conditions to be
> > > > > executed using a hash join algorithm
> > > > > [CALCITE-2992] Enhance implicit conversions when generating hash join
> > > > > keys for an equi condition
> > > > > [CALCITE-3036] Remove correlate variables from Join
> > > > > [CALCITE-3037] Rename EnumerableThetaJoin to EnumerableNestedLoopJoin
> > > > >
> > > > > I think we should cut a release soon to keep momentum going. I'd like
> > > > > to hear from those working on any of the JIRAs to see which ones could
> > > > > be wrapped up in the next week or two.
> > > > > --
> > > > > Michael Mior
> > > > > mm...@apache.org
> > > > >
> > > > > Le mar. 30 avr. 2019 à 11:15, Michael Mior  a écrit 
> > > > > :
> > > > > >
> > > > > > Calcite 1.19.0 was released approximately one month ago. This was
> > > > > > later than we originally planned (although I think with good reason
> > > > > > and I'm happy with what made it into this release). I don't think
> > > > > > there's an imminent need for a new release, but I wanted to start 
> > > > > > the
> > > > > > discussion now that Avatica has had it's latest release.
> > > > > >
> > > > > > There's a few big things in progress below that I thought I'd see if
> > > > > > we want to try to include in the next release. Apologies if I missed
> > > > > > any.
> > > > > >
> > > > > > CALCITE-1581 UDTF like in hive
> > > > > > CALCITE-2952 Certify Calcite on JDK 12
> > > > > > CALCITE-3036 Remove correlate variables from Join
> > > > > > CALCITE-3037 Rename EnumerableThetaJoin to EnumerableNestedLoopJoin
> > > > > >
> > > > > > --
> > > > > > Michael Mior
> > > > > > mm...@apache.org
> > > >
>

Re: Pluggable JDBC types

[Yuzhao Chen]

You don’t need to, just define a new type name in parser[1] and translate it to 
VARCHAR is okey.

[1] 
https://github.com/apache/calcite/blob/b0e83c469ff57257c1ea621ff943ca76f626a9b7/server/src/main/codegen/config.fmpp#L375

Best,
Danny Chan
在 2019年6月3日 +0800 AM6:09，Muhammad Gelbana ，写道：
> That I understand now. But how can I support casting to TEXT and having the
> returned column type name as TEXT (ie. Not VARCHAR) ?
>
> Thanks,
> Gelbana
>
>
> On Sun, Jun 2, 2019 at 7:41 PM Julian Hyde  wrote:
>
> > The parser should only parse, not validate. This is a very important
> > organizing principle for the parser.
> >
> > If I write “x :: text” or “x :: foo” it is up to the type system
> > (implemented in the validator and elsewhere) to figure out whether “text”
> > or “foo” are valid types.
> >
> > Logically, “x :: foo” is the same as “CAST(x AS foo)”. The parser should
> > produce the same SqlCall in both cases. Then the parser’s job is done.
> >
> > Julian
> >
> >
> > > On Jun 2, 2019, at 6:42 AM, Muhammad Gelbana 
> > wrote:
> > >
> > > I'm trying to support the PostgreSQL TEXT type[1]. It's basically a
> > VARCHAR.
> > >
> > > As Julian mentioned in his comment on Jira, I don't need to define a
> > > keyword to achieve what I need so I tried exploring that and here is
> > what I
> > > observed so far:
> > >
> > > 1. If I define a new keyword in the parser, I face no trouble whatsoever
> > > except for the numerous wiring I need to do for RexToLixTranslator,
> > > JavaTypeFactoryImpl, SqlTypeAssignmentRules and SqlTypeName. I won't be
> > > suprised if I'm missing anything but doing what I did at first managed to
> > > get my queries through.
> > >
> > > 2. If I define the type by plugging it in through the root schema, I face
> > > two problems: a) The field cannot be declared as nullable because the
> > query
> > > I'm using for testing gets data from (VALUES()) which doesn't produce
> > null
> > > values, so an exception is thrown. b) The returned column type name is
> > > VARCHAR (although I delcared the new plugged type name to be TEXT), the
> > > returned type number is valid though (Types.VARCHAR = 12)
> > >
> > > I think I'm doing something wrong that causes (2.a) but (2.b) seems a
> > like
> > > a bug to me. What do you think ?
> > >
> > > [1] https://issues.apache.org/jira/browse/CALCITE-3108
> > >
> > > Thanks,
> > > Gelbana
> >
> >

Re: Giving the Calcite logo some love

[Yuzhao Chen]

Oh, I see a big hammer, thanks Daniel !

Best,
Danny Chan
在 2019年6月3日 +0800 AM6:21，Daniel Gruno ，写道：
> Found it!
>
> http://humbedooh.com/calcite-proposed.svg
>
> Thanks, Wayback Machine!
>
> On 19/05/2019 10.01, Stamatis Zampetakis wrote:
> > Hi all,
> >
> > We started this discussion about a year ago and many people were
> > positive with the idea of having a new logo for Calcite.
> > We had some nice proposals at the time and maybe now somebody else has
> > also new ideas/designs to contribute.
> >
> > @Daniel: Is there any chance that you still have the logos you proposed
> > somewhere available? The old link [1] does not work anymore.
> >
> > Best,
> > Stamatis
> >
> > [1] http://www.apache.org/logos/comdev-test/res/calcite/calcite-proposed.svg
> >
> >
> > On Thu, Aug 30, 2018 at 12:02 AM Stamatis Zampetakis  > > wrote:
> >
> > Vladimir>Could you flip rhombus so it goes right-up?
> >
> > https://svgshare.com/s/86r
> >
> > I hope this is what you meant.
> >
> > Best,
> > Stamatis
> >
> > Στις Τετ, 29 Αυγ 2018 στις 6:30 μ.μ., ο/η Michael Mior
> > mailto:mm...@apache.org>> έγραψε:
> >
> > Just a note since we're on the topic that whatever logos we come
> > up with
> > should be sure to have TM clearly indicated.
> >
> > --
> > Michael Mior
> > mm...@apache.org 
> >
> >
> >
> > Le mer. 29 août 2018 à 03:13, Julian Hyde
> > mailto:jhyde.apa...@gmail.com>> a écrit :
> >
> > > Yes indeed!
> > >
> > > If someone feels inspired to produce a logo, here’s my
> > suggestion of a
> > > theme/image: a spider, specifically a Barn Spider (Araneus
> > Cavaticus)[1].
> > > It was the origin of the name “avatica”, connects and spins
> > webs, and the
> > > eponymous individual in Charlotte’s Web had rather exceptional
> > > communication skills.
> > >
> > > Julian
> > >
> > > [1] https://en.m.wikipedia.org/wiki/Barn_spider
> > >
> > > > On Aug 28, 2018, at 9:49 PM, Francis Chuang
> > mailto:francischu...@apache.org>>
> > > wrote:
> > > >
> > > > The designs I have seen so far look really good! Would it
> > also make
> > > sense to design a variant for Avatica as well? This is what
> > the current
> > > Avatica logo looks like:
> > https://calcite.apache.org/avatica/img/logo.png
> > > >
> > > > Francis
> > > >
> > > > > On 29/08/2018 7:08 AM, Vladimir Sitnikov wrote:
> > > > > Stamatis>How about something like the following:
> > > > >
> > > > > There's left-to-right vs right-to-left issue, however I
> > would claim that
> > > > > the direction of improvement is right+up.
> > > > > For instance: BTC price is good when plots go to the right
> > and go
> > > upward.
> > > > >
> > > > > https://svgur.com/s/83y is slanted backward.
> > > > > That creates perception of "Calcite holding back the
> > progress" or
> > > "Apache
> > > > > pushing C away" or something like that.
> > > > > Could you flip rhombus so it goes right-up?
> > > > >
> > > > >
> > > > > Vladimir
> > > > >
> > > >
> > >
> >
>

[DISCUSSION] Extension of Metadata Query

[Yuzhao Chen]

Currently we provide answer to metadata query through RelMetadataProvider [1], 
there are some sub-classes of it:

RelMetadataProvider
|
|- VolcanoRelMetadataProvider
|- ChainedRelMetadataProvider/DefaultRelMetadataProvider
|- HepRelMetadataProvider
|- CachingRelMetadataProvider
|- ReflectiveRelMetadataProvider
|- JaninoRelMetadataProvider

The RelMetadataProvider has two methods: #apply and #handlers, the #apply 
method seems a programming interface and there is a demo code how we can use it:

RelMetadataProvider provider;
LogicalFilter filter;
RexNode predicate;
Function function =
provider.apply(LogicalFilter.class, Selectivity.class};
Selectivity selectivity = function.apply(filter);
Double d = selectivity.selectivity(predicate);

But let's see our RelOptCluster's member variables[2], there are 
MetadataFactory and RelMetadataQuery which all can be used to query the 
metadata, for MetadataFactory, there is a default impl named 
MetadataFactoryImpl which will invoke RelMetadataProvider#apply internally, for 
RelMetadataQuery, it will invoke RelMetadataProvider#handlers (finally composed 
and codeden by JaninoRelMetadataProvider).

In our planning phrase, we can invoke RelOptRuleCall#getMetadataQuery to get 
the MQ and query the metadata.

For extension of metadata handlers, we can set our customized 
RelMetadataProvider in RelOptCluster[3]. But for RelMetadataQuery, we have no 
way to extend it now, because the RelOptCluster always has a singleton instance 
[4] which is only the default implementation.


My question is as follows:

1. Why we have 2 methods in RelMetadataProvider, and why we need the 
MetadataFactory and RelMetadataProvider#apply ? It seems that it's function is 
already been overriden by RelMetadataQuery(The difference is that 
MetadataFactory use Reflection and RelMetadataQuery use gened bytes code).
2. We should make the RelMetadataQuery in RelOptCluster pluggable.


[1] 
https://github.com/apache/calcite/blob/b0e83c469ff57257c1ea621ff943ca76f626a9b7/core/src/main/java/org/apache/calcite/rel/metadata/RelMetadataProvider.java#L38
[2] 
https://github.com/apache/calcite/blob/b0e83c469ff57257c1ea621ff943ca76f626a9b7/core/src/main/java/org/apache/calcite/plan/RelOptCluster.java#L49
[3] 
https://github.com/apache/calcite/blob/b0e83c469ff57257c1ea621ff943ca76f626a9b7/core/src/main/java/org/apache/calcite/plan/RelOptCluster.java#L135
[4] 
https://github.com/apache/calcite/blob/b0e83c469ff57257c1ea621ff943ca76f626a9b7/core/src/main/java/org/apache/calcite/plan/RelOptCluster.java#L151



Best,
Danny Chan

Re: sql to rule machine learning

[Yuzhao Chen]

I have also noticed this project, this grammar is too faraway from SQL 
standard. But the idea is good, like the CEP grammar.

Best,
Danny Chan
在 2019年6月3日 +0800 AM10:52，Albert ，写道：
> found some efforts in bringing sql to machine learning world.
> https://github.com/sql-machine-learning/sqlflow/blob/develop/doc/syntax.md
>
> any comments ?
>
> --
> ~~~
> no mistakes
> ~~

Re: Giving the Calcite logo some love

[Yuzhao Chen]

I would prefer 
https://github.com/zabetak/calcite/blob/calcite-logo/site/img/logo-alt2-v1.svg
if the hammer can be smaller :)

Best,
Danny Chan
在 2019年6月5日 +0800 AM8:32，Albert ，写道：
> I will vote "logo-alt1-v5.svg
> <https://github.com/zabetak/calcite/blob/calcite-logo/site/img/logo-alt1-v5.svg>"
> , looks nice.
>
> On Wed, Jun 5, 2019 at 6:11 AM Stamatis Zampetakis 
> wrote:
>
> > I created a branch to gather all alternative logos [1].
> >
> > Among the two aforementioned proposals, I added a few more variants (check
> > logo-alt* under the img directory).
> > Have a look and let me know what you think. I'm open to any ideas and
> > suggestions.
> >
> > [1] https://github.com/zabetak/calcite/tree/calcite-logo/site/img
> >
> > On Tue, Jun 4, 2019 at 7:25 PM Julian Hyde  wrote:
> >
> > > I prefer both over the current logo. (And I made the current logo.)
> > >
> > > Let's keep the discussion going, and get to a new logo.
> > >
> > > On Tue, Jun 4, 2019 at 9:42 AM Ivan Grgurina 
> > wrote:
> > > >
> > > > I prefer the one Daniel sent. It looks cleaner, but maybe the "periodic
> > > table" logo can be made better by simplifying it, the shadow on the
> > letter
> > > C is... unusual.
> > > >
> > > > 
> > > > From: Stamatis Zampetakis 
> > > > Sent: Tuesday, June 4, 2019 6:32 PM
> > > > To: dev@calcite.apache.org; humbed...@apache.org
> > > > Subject: Re: Giving the Calcite logo some love
> > > >
> > > > Thanks for digging this out Daniel!
> > > >
> > > > At this point we have two candidates:
> > > > http://humbedooh.com/calcite-proposed.svg
> > > > https://svgshare.com/s/86r
> > > >
> > > > Do we like any of above more than our current logo (the way they are or
> > > > with slight modifications) ?
> > > >
> > > >
> > > >
> > > >
> > > > On Mon, Jun 3, 2019 at 3:15 AM Yuzhao Chen 
> > wrote:
> > > >
> > > > > Oh, I see a big hammer, thanks Daniel !
> > > > >
> > > > > Best,
> > > > > Danny Chan
> > > > > 在 2019年6月3日 +0800 AM6:21，Daniel Gruno ，写道：
> > > > > > Found it!
> > > > > >
> > > > > > http://humbedooh.com/calcite-proposed.svg
> > > > > >
> > > > > > Thanks, Wayback Machine!
> > > > > >
> > > > > > On 19/05/2019 10.01, Stamatis Zampetakis wrote:
> > > > > > > Hi all,
> > > > > > >
> > > > > > > We started this discussion about a year ago and many people were
> > > > > > > positive with the idea of having a new logo for Calcite.
> > > > > > > We had some nice proposals at the time and maybe now somebody
> > else
> > > has
> > > > > > > also new ideas/designs to contribute.
> > > > > > >
> > > > > > > @Daniel: Is there any chance that you still have the logos you
> > > proposed
> > > > > > > somewhere available? The old link [1] does not work anymore.
> > > > > > >
> > > > > > > Best,
> > > > > > > Stamatis
> > > > > > >
> > > > > > > [1]
> > > > >
> > > http://www.apache.org/logos/comdev-test/res/calcite/calcite-proposed.svg
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Aug 30, 2018 at 12:02 AM Stamatis Zampetakis <
> > > > > zabe...@gmail.com
> > > > > > > <mailto:zabe...@gmail.com>> wrote:
> > > > > > >
> > > > > > > Vladimir>Could you flip rhombus so it goes right-up?
> > > > > > >
> > > > > > > https://svgshare.com/s/86r
> > > > > > >
> > > > > > > I hope this is what you meant.
> > > > > > >
> > > > > > > Best,
> > > > > > > Stamatis
> > > > > > >
> > > > > > > Στις Τετ, 29 Αυγ 2018 στις 6:30 μ.μ., ο/η Michael Mior
> > > > > > > mailto:mm...@apache.org>> έγραψε:
> > > > > > >
> > > > > > > Just a note since we're on the topic that whatever 

Re: Correlate Join SemiJoin transformation

[Yuzhao Chen]

I think the complexity mostly comes from the value generator, which is the 
engineering foundation of current decorrelation.

Best,
Danny Chan
在 2019年6月5日 +0800 AM10:45，Chunwei Lei ，写道：
> Thanks for raising this, Haisheng.
>
> Do you mean that we should have better subquery unnesting?
>
>
>
>
> Best,
> Chunwei
>
>
> On Tue, Jun 4, 2019 at 8:27 AM Haisheng Yuan  wrote:
>
> > Hi,
> >
> > Since we have commited CALCITE-2969, and in the next release of 1.21, we
> > will deprecate EquiJoin and make enumerable hash join support non-equi join
> > condition, the optimization door is open to us.
> >
> > Currently Calcite often generates complicated and inefficient plan for
> > boolean context correlated subquery (putting value context subquery aside,
> > which is more complex during unnesting). e.g.
> > https://issues.apache.org/jira/browse/CALCITE-2857
> > https://issues.apache.org/jira/browse/CALCITE-2874
> >
> > There are many more cases if you want to try and find out. Some underlying
> > reasons are:
> > 1. SemJoin doesn't support non-equi condition
> > 2. Correlate comes with Aggregate in many cases
> >
> > Sometimes, SubqueryRemoveRule generates Correlate with Aggregate,
> > sometimes generates Join with Aggregate. Then we use SemiJoinRule to match
> > Join(Rel, Agg) pattern to see whether we can transform it to semijoin or
> > not. This is counter intuitive, because we already know it will be a
> > SemiJoin or AntiSemi when we generate Correlate for the subquery. In
> > current way, we may miss the chance to get back SemiJoin for it, and we
> > almost lose the opportunity to do IndexScan if there is index available.
> >
> > In addition, I see 2 rules JoinAddRedundantSemiJoinRule and
> > SemiJoinRemoveRule seem to provide the chance to do index scan. But through
> > the test cases related with the 2 rules, I don't see any chance for a
> > better plan with them and they don't seem to be able to generate plan with
> > index scan. I doubt whether they are used at all. Can someone shed some
> > light on this and give us some examples, if I get it wrong?
> >
> > The transformation flow for boolean context sub-query in my head is:
> > sub-query -> Correate(Semi/AntiSemi) without Aggregate (SubqueryRemoveRule)
> > Correlate(Semi/AntiSemi) -> NestedLoopJoin (correlate implementation rule)
> > Correlate(Semi/AntiSemi) -> IndexedNestedLoopJoin (index apply rule)
> > Correlate(Semi/AntiSemi) -> SemiJoin/AntiSemiJoin (Semi Correlate to Semi
> > Join rule)
> > SemiJoin -> InnerJoin(rel, Agg) (SemiJoin to Join rule)
> > InnerJoin(rel, Agg) -> InnerJoin(Agg, rel) (Join reorder rule)
> > SemiJoin/InnerJoin -> HashJoin/MergeJoin/NLJ (Join implementation rule)
> >
> > This is a long shot, and will involve tons of changes.
> > Any thoughts?
> >
> > - Haisheng Yuan

Re: [DISCUSSION] Extension of Metadata Query

[Yuzhao Chen]

Thanks Julian for your detail reply,


1. I checked the Janino gened-code and the RelMetadataQuery/RelMetadataProvidor 
and almost can make sure MetadataQuery only use the 
RelMetadataProvidor#handlers method for multiple dispatch, the 
RelMetadataProvidor#apply is only used for MetadataFactory.
2. I agree that we should provide new RelMetadataProvider for extension, but 
the RelMetadataQuery has top level interfaces for metadata query, these top 
level interfaces should be extendable and sync with our metadata handlers.


Best,
Danny Chan
在 2019年6月5日 +0800 AM1:48，Julian Hyde ，写道：
> > 1. Why we have 2 methods in RelMetadataProvider?
>
> The metadata system is complicated. We need to allow multiple handlers
> for any given call. So, making a metadata call involves multiple
> dispatch [1] based on the metadata method being called, the type of
> RelNode, and the handlers that are registered. Also it needs to cache
> results, and detect cycles. And the dispatch needs to be efficient, so
> we generate janino code to do the dispatch, and re-generate when new
> handlers or sub-classes of RelNode are added.
>
> I forget details, the two methods are basically required to allow us
> to generate code to do the dispatch.
>
> > 2. We should make the RelMetadataQuery in RelOptCluster pluggable.
>
> I disagree. RelMetadataQuery is not an extension point. Its sole
> purpose is to to keep state (the cache and cycle-checking).
> RelMetadataProvider is the extension point. (By analogy, if you are
> un-parsing an AST, you let each AST sub-class handle unparsing itself,
> but the unparsed text goes to a simple StringBuilder. RelMetadataQuery
> is in the role of the StringBuilder. In a complex system, it is nice
> to keep some of the components simple, or at least keep them to
> prescribed roles.)
>
> Julian
>
> [1] https://en.wikipedia.org/wiki/Multiple_dispatch
>
> On Sun, Jun 2, 2019 at 11:19 PM Yuzhao Chen  wrote:
> >
> > Currently we provide answer to metadata query through RelMetadataProvider 
> > [1], there are some sub-classes of it:
> >
> > RelMetadataProvider
> > |
> > |- VolcanoRelMetadataProvider
> > |- ChainedRelMetadataProvider/DefaultRelMetadataProvider
> > |- HepRelMetadataProvider
> > |- CachingRelMetadataProvider
> > |- ReflectiveRelMetadataProvider
> > |- JaninoRelMetadataProvider
> >
> > The RelMetadataProvider has two methods: #apply and #handlers, the #apply 
> > method seems a programming interface and there is a demo code how we can 
> > use it:
> >
> > RelMetadataProvider provider;
> > LogicalFilter filter;
> > RexNode predicate;
> > Function function =
> > provider.apply(LogicalFilter.class, Selectivity.class};
> > Selectivity selectivity = function.apply(filter);
> > Double d = selectivity.selectivity(predicate);
> >
> > But let's see our RelOptCluster's member variables[2], there are 
> > MetadataFactory and RelMetadataQuery which all can be used to query the 
> > metadata, for MetadataFactory, there is a default impl named 
> > MetadataFactoryImpl which will invoke RelMetadataProvider#apply internally, 
> > for RelMetadataQuery, it will invoke RelMetadataProvider#handlers (finally 
> > composed and codeden by JaninoRelMetadataProvider).
> >
> > In our planning phrase, we can invoke RelOptRuleCall#getMetadataQuery to 
> > get the MQ and query the metadata.
> >
> > For extension of metadata handlers, we can set our customized 
> > RelMetadataProvider in RelOptCluster[3]. But for RelMetadataQuery, we have 
> > no way to extend it now, because the RelOptCluster always has a singleton 
> > instance [4] which is only the default implementation.
> >
> >
> > My question is as follows:
> >
> > 1. Why we have 2 methods in RelMetadataProvider, and why we need the 
> > MetadataFactory and RelMetadataProvider#apply ? It seems that it's function 
> > is already been overriden by RelMetadataQuery(The difference is that 
> > MetadataFactory use Reflection and RelMetadataQuery use gened bytes code).
> > 2. We should make the RelMetadataQuery in RelOptCluster pluggable.
> >
> >
> > [1] 
> > https://github.com/apache/calcite/blob/b0e83c469ff57257c1ea621ff943ca76f626a9b7/core/src/main/java/org/apache/calcite/rel/metadata/RelMetadataProvider.java#L38
> > [2] 
> > https://github.com/apache/calcite/blob/b0e83c469ff57257c1ea621ff943ca76f626a9b7/core/src/main/java/org/apache/calcite/plan/RelOptCluster.java#L49
> > [3] 
> > https://github.com/apache/calcite/blob/b0e83c469ff57257c1ea621ff943ca76f626a9b7/core/src/main/java/org/apache/calcite/plan/RelOptCluster.java#L135
> > [4] 
> > https://github.com/apache/calcite/blob/b0e83c469ff57257c1ea621ff943ca76f626a9b7/core/src/main/java/org/apache/calcite/plan/RelOptCluster.java#L151
> >
> >
> >
> > Best,
> > Danny Chan

Re: Execute multiple RelNodes from single RelNode

[Yuzhao Chen]

This seems a requests for multi-sink insert.

> 3) Calcite transforms it into multiple TableModifies

Instead of let Calcite to transform multiple TableModifies, I think you should 
do it by your self, the send each TableModify to Calcite sqlToRel converter.

If you want to insert into multiple sink task to be run in the same plan, this 
is another topic, we may promote one sink node tree a time and finally merge 
all the trees.

Best,
Danny Chan
在 2019年6月5日 +0800 PM7:58，dev@calcite.apache.org，写道：
>
> 3) Calcite transforms it into multiple TableModifies

Re: [DISCUSSION] Extension of Metadata Query

[Yuzhao Chen]

In order to add some top-level query interfaces like columnInterval to topK, 
one way to extend the RelMetadataQuery is “sub-class” it,  but a 
subclass-instance of RelMetadataQuery can not be set into the 
RelOptCluster(with a always default singleton RMQ instance), which is a key 
factor to limit the extension of RelMetadataQuery.

Best,
Danny Chan
在 2019年6月5日 +0800 PM11:49，Julian Hyde ，写道：
> By “extend” do you mean “sub-class”?
>
> > On Jun 4, 2019, at 10:19 PM, Yuzhao Chen  wrote:
> >
> > Thanks Julian for your detail reply,
> >
> >
> > 1. I checked the Janino gened-code and the 
> > RelMetadataQuery/RelMetadataProvidor and almost can make sure MetadataQuery 
> > only use the RelMetadataProvidor#handlers method for multiple dispatch, the 
> > RelMetadataProvidor#apply is only used for MetadataFactory.
> > 2. I agree that we should provide new RelMetadataProvider for extension, 
> > but the RelMetadataQuery has top level interfaces for metadata query, these 
> > top level interfaces should be extendable and sync with our metadata 
> > handlers.
> >
> >
> > Best,
> > Danny Chan
> > 在 2019年6月5日 +0800 AM1:48，Julian Hyde ，写道：
> > > > 1. Why we have 2 methods in RelMetadataProvider?
> > >
> > > The metadata system is complicated. We need to allow multiple handlers
> > > for any given call. So, making a metadata call involves multiple
> > > dispatch [1] based on the metadata method being called, the type of
> > > RelNode, and the handlers that are registered. Also it needs to cache
> > > results, and detect cycles. And the dispatch needs to be efficient, so
> > > we generate janino code to do the dispatch, and re-generate when new
> > > handlers or sub-classes of RelNode are added.
> > >
> > > I forget details, the two methods are basically required to allow us
> > > to generate code to do the dispatch.
> > >
> > > > 2. We should make the RelMetadataQuery in RelOptCluster pluggable.
> > >
> > > I disagree. RelMetadataQuery is not an extension point. Its sole
> > > purpose is to to keep state (the cache and cycle-checking).
> > > RelMetadataProvider is the extension point. (By analogy, if you are
> > > un-parsing an AST, you let each AST sub-class handle unparsing itself,
> > > but the unparsed text goes to a simple StringBuilder. RelMetadataQuery
> > > is in the role of the StringBuilder. In a complex system, it is nice
> > > to keep some of the components simple, or at least keep them to
> > > prescribed roles.)
> > >
> > > Julian
> > >
> > > [1] https://en.wikipedia.org/wiki/Multiple_dispatch
> > >
> > > > On Sun, Jun 2, 2019 at 11:19 PM Yuzhao Chen  
> > > > wrote:
> > > >
> > > > Currently we provide answer to metadata query through 
> > > > RelMetadataProvider [1], there are some sub-classes of it:
> > > >
> > > > RelMetadataProvider
> > > > |
> > > > |- VolcanoRelMetadataProvider
> > > > |- ChainedRelMetadataProvider/DefaultRelMetadataProvider
> > > > |- HepRelMetadataProvider
> > > > |- CachingRelMetadataProvider
> > > > |- ReflectiveRelMetadataProvider
> > > > |- JaninoRelMetadataProvider
> > > >
> > > > The RelMetadataProvider has two methods: #apply and #handlers, the 
> > > > #apply method seems a programming interface and there is a demo code 
> > > > how we can use it:
> > > >
> > > > RelMetadataProvider provider;
> > > > LogicalFilter filter;
> > > > RexNode predicate;
> > > > Function function =
> > > > provider.apply(LogicalFilter.class, Selectivity.class};
> > > > Selectivity selectivity = function.apply(filter);
> > > > Double d = selectivity.selectivity(predicate);
> > > >
> > > > But let's see our RelOptCluster's member variables[2], there are 
> > > > MetadataFactory and RelMetadataQuery which all can be used to query the 
> > > > metadata, for MetadataFactory, there is a default impl named 
> > > > MetadataFactoryImpl which will invoke RelMetadataProvider#apply 
> > > > internally, for RelMetadataQuery, it will invoke 
> > > > RelMetadataProvider#handlers (finally composed and codeden by 
> > > > JaninoRelMetadataProvider).
> > > >
> > > > In our planning phrase, we can invoke RelOptRuleCall#getMetadataQuery 
> > > > to get the MQ and query the metadata.
> > > >

Re: Question about spark adaptor

[Yuzhao Chen]

Luso Clemens,

kindly reminder, you better not to tell your internal team work for your 
companies in the dev mailing list, because it would be recorded permanently.

Just put out your points or questions is okey and we would love to help you.

Best,
Danny Chan
在 2019年6月5日 +0800 PM11:52，Luso Clemens ，写道：
> Hi，
> I'm a system architect work for china merchant bank, I want to use
> calcite to do some federated queries across different databases and push
> calculations to spark, so I enable the spark option but get an no pointer
> exception.
> I think the problem is I didn't configure the spark adaptor, but
> couldn't find any introduction or documents talking about it, except some
> test specs.
> So my question is, is there any way to push calculations into spark,
> or it just unavailable in current version.
> Forgive my bad English.

Re: Parsing DB2 sql statements

[Yuzhao Chen]

Another choice is just like you said, you can add Date as UDF but because
Date is a reserved word, you may need to quote the function name like
`Date`(col1) based on what quoting character you use.

Andrew O  于2019年6月6日周四 下午7:07写道：

> I'm doing a project trying to parse some IBM DB2 sql expressions (to
> analyze table / column usages).
>
> To note,
> 1) these are existing ad-hoc user queries so I can't change their syntax
> 2)  I don't have a connection / schema for the database,  but my
> understanding is that this shouldn't be required by Calcite for this
> kind of work.
>
> Currently I'm hitting some parsing issues with queries like
>
> select Date(x.col1), x.col2 from myTable x
>
> It fails parse at the point of the Date function call. This is trying to
> use the function to convert the value of col1 to a date (almost similar to
> a cast/convert).
>
>
> https://www.ibm.com/support/knowledgecenter/en/SSEPEK_11.0.0/sqlref/src/tpc/db2z_bif_date.html
>
>
> My current thinking would be that this  may need changes to the parser
> grammar (through an extension?). Or perhaps this could be registered as a
> custom User Defined Function of some kind (although Date seems like a
> reserved word)?
>
> Is this the right thinking,  or is there another approach I could look at?
>
> Thanks in advance
>
> Andrew
>

Re: is there CHARINDEX function?

[Yuzhao Chen]

The SUBSTRING function in SqlStdOperatorTable may be what you needs.

[1] 
https://github.com/apache/calcite/blob/614b4350f558a4fabaafc06fef9689ec4c267994/core/src/main/java/org/apache/calcite/sql/fun/SqlStdOperatorTable.java#L1428

Best,
Danny Chan
在 2019年6月7日 +0800 AM1:01，Jess Balint ，写道：
> Is there a function which searches for a substring? Thanks.
>
> Jess

Re: [DISCUSSION] Extension of Metadata Query

[Yuzhao Chen]

Thanks, Stamatis

You are right, this discussion came up due to CALCITE-2885, it is not about the 
performance problem, it is the extension of RelMetadataQuery, because we add 
all kinds of top interfaces in RelMetadataQuery, e.g. the row count query [1].

When we add a new RelMetadataProvider, a corresponding interface/method should 
be added into RelMetadataQuery, but for current RelOpeCluster impl, we could 
not do that(with a always default instance)

As for the  methods in RelMetadataProvider, i only saw the usage of #handlers 
in RelMetadataQuery, and saw the reference you pase, it seems not that relevant 
with this topic. What do you think ?

[1] 
https://github.com/apache/calcite/blob/941cd4e9540e3ef9b7c15daee42831a0c63da8b9/core/src/main/java/org/apache/calcite/rel/metadata/RelMetadataQuery.java#L222

Best,
Danny Chan
在 2019年6月5日 +0800 AM6:44，Stamatis Zampetakis ，写道：
> Thanks for bringing this up Danny.
>
> I guess the discussion came up due to CALCITE-2885 [1]. Looking back into
> it, I am not sure that the intention there is to make the RelMetadataQuery
> pluggable. We could possibly solve the performance problem without
> extending the RelMetadataQuery. We have to look again into this case.
>
> For more details regarding the existence of the two methods in
> RelMetadataProvider have a look in CALCITE-604 [2]. More general for the
> design of RelMetadataProvider you may find useful the description in [3].
>
> Best,
> Stamatis
>
> [1] https://issues.apache.org/jira/browse/CALCITE-2885
> [2] https://issues.apache.org/jira/browse/CALCITE-604
> [3]
> https://web.archive.org/web/20140624040836/www.hydromatic.net/wiki/RelationalExpressionMetadata/
>
> On Tue, Jun 4, 2019 at 7:48 PM Julian Hyde  wrote:
>
> > > 1. Why we have 2 methods in RelMetadataProvider?
> >
> > The metadata system is complicated. We need to allow multiple handlers
> > for any given call. So, making a metadata call involves multiple
> > dispatch [1] based on the metadata method being called, the type of
> > RelNode, and the handlers that are registered. Also it needs to cache
> > results, and detect cycles. And the dispatch needs to be efficient, so
> > we generate janino code to do the dispatch, and re-generate when new
> > handlers or sub-classes of RelNode are added.
> >
> > I forget details, the two methods are basically required to allow us
> > to generate code to do the dispatch.
> >
> > > 2. We should make the RelMetadataQuery in RelOptCluster pluggable.
> >
> > I disagree. RelMetadataQuery is not an extension point. Its sole
> > purpose is to to keep state (the cache and cycle-checking).
> > RelMetadataProvider is the extension point. (By analogy, if you are
> > un-parsing an AST, you let each AST sub-class handle unparsing itself,
> > but the unparsed text goes to a simple StringBuilder. RelMetadataQuery
> > is in the role of the StringBuilder. In a complex system, it is nice
> > to keep some of the components simple, or at least keep them to
> > prescribed roles.)
> >
> > Julian
> >
> > [1] https://en.wikipedia.org/wiki/Multiple_dispatch
> >
> > On Sun, Jun 2, 2019 at 11:19 PM Yuzhao Chen  wrote:
> > >
> > > Currently we provide answer to metadata query through
> > RelMetadataProvider [1], there are some sub-classes of it:
> > >
> > > RelMetadataProvider
> > > |
> > > |- VolcanoRelMetadataProvider
> > > |- ChainedRelMetadataProvider/DefaultRelMetadataProvider
> > > |- HepRelMetadataProvider
> > > |- CachingRelMetadataProvider
> > > |- ReflectiveRelMetadataProvider
> > > |- JaninoRelMetadataProvider
> > >
> > > The RelMetadataProvider has two methods: #apply and #handlers, the
> > #apply method seems a programming interface and there is a demo code how we
> > can use it:
> > >
> > > RelMetadataProvider provider;
> > > LogicalFilter filter;
> > > RexNode predicate;
> > > Function function =
> > > provider.apply(LogicalFilter.class, Selectivity.class};
> > > Selectivity selectivity = function.apply(filter);
> > > Double d = selectivity.selectivity(predicate);
> > >
> > > But let's see our RelOptCluster's member variables[2], there are
> > MetadataFactory and RelMetadataQuery which all can be used to query the
> > metadata, for MetadataFactory, there is a default impl named
> > MetadataFactoryImpl which will invoke RelMetadataProvider#apply internally,
> > for RelMetadataQuery, it will invoke RelMetadataProvider#handlers (finally
> > composed and codeden by JaninoRelMetadataProvider).
> > >
> > > I

Master build is failing

[Yuzhao Chen]

The master branch is failing continuously, since from CALCITE-2822[1],  but I’m 
not sure if this commits has relationship with the failure.

Siddharth and Stamatis Zampetakis, do you have any ideas ?

[1] https://travis-ci.org/apache/calcite/builds/542987336

Best,
Danny Chan

Re: [DISCUSSION] Extension of Metadata Query

[Yuzhao Chen]

We have an interface to fetch the RelMetadataQuery in RelOptRuleCall [1], which 
I believe is the most common routine to query metadata during planning, I’m a 
little confused for your saying

>The methods in RelMetadataQuery are for convenience only.

If these methods in RelMetadataQuery are only for convenience, what is the 
other way I can query a metadata ? The MedataFactory from the RelOptCluster ? 
(Users would never expect to use this because it’s based on Java Reflection and 
has performance problem)

[1] 
https://github.com/apache/calcite/blob/a3c56be7bccc58859524ba39e5b30b7078f97d00/core/src/main/java/org/apache/calcite/plan/RelOptRuleCall.java#L200

Best,
Danny Chan
在 2019年6月12日 +0800 AM6:27，Julian Hyde ，写道：
> The goal is that it should be possible to add a new kind of metadata. The 
> methods in RelMetadataQuery are for convenience only. So you should be able 
> to use your new kind of metadata, and existing ones, without modifying 
> RelMetadataQuery.
>
> If that is not possible, it’s a bug, and you should log a JIRA case.
>
> Given how the methods re-generate handlers (by catching exceptions and 
> modifying mutable private members) I can see that new kinds of metadata might 
> be difficult to add.
>
> Julian
>
>
> > On Jun 9, 2019, at 7:54 PM, Yuzhao Chen  wrote:
> >
> > Thanks, Stamatis
> >
> > You are right, this discussion came up due to CALCITE-2885, it is not about 
> > the performance problem, it is the extension of RelMetadataQuery, because 
> > we add all kinds of top interfaces in RelMetadataQuery, e.g. the row count 
> > query [1].
> >
> > When we add a new RelMetadataProvider, a corresponding interface/method 
> > should be added into RelMetadataQuery, but for current RelOpeCluster impl, 
> > we could not do that(with a always default instance)
> >
> > As for the methods in RelMetadataProvider, i only saw the usage of 
> > #handlers in RelMetadataQuery, and saw the reference you pase, it seems not 
> > that relevant with this topic. What do you think ?
> >
> > [1] 
> > https://github.com/apache/calcite/blob/941cd4e9540e3ef9b7c15daee42831a0c63da8b9/core/src/main/java/org/apache/calcite/rel/metadata/RelMetadataQuery.java#L222
> >
> > Best,
> > Danny Chan
> > 在 2019年6月5日 +0800 AM6:44，Stamatis Zampetakis ，写道：
> > > Thanks for bringing this up Danny.
> > >
> > > I guess the discussion came up due to CALCITE-2885 [1]. Looking back into
> > > it, I am not sure that the intention there is to make the RelMetadataQuery
> > > pluggable. We could possibly solve the performance problem without
> > > extending the RelMetadataQuery. We have to look again into this case.
> > >
> > > For more details regarding the existence of the two methods in
> > > RelMetadataProvider have a look in CALCITE-604 [2]. More general for the
> > > design of RelMetadataProvider you may find useful the description in [3].
> > >
> > > Best,
> > > Stamatis
> > >
> > > [1] https://issues.apache.org/jira/browse/CALCITE-2885
> > > [2] https://issues.apache.org/jira/browse/CALCITE-604
> > > [3]
> > > https://web.archive.org/web/20140624040836/www.hydromatic.net/wiki/RelationalExpressionMetadata/
> > >
> > > On Tue, Jun 4, 2019 at 7:48 PM Julian Hyde  wrote:
> > >
> > > > > 1. Why we have 2 methods in RelMetadataProvider?
> > > >
> > > > The metadata system is complicated. We need to allow multiple handlers
> > > > for any given call. So, making a metadata call involves multiple
> > > > dispatch [1] based on the metadata method being called, the type of
> > > > RelNode, and the handlers that are registered. Also it needs to cache
> > > > results, and detect cycles. And the dispatch needs to be efficient, so
> > > > we generate janino code to do the dispatch, and re-generate when new
> > > > handlers or sub-classes of RelNode are added.
> > > >
> > > > I forget details, the two methods are basically required to allow us
> > > > to generate code to do the dispatch.
> > > >
> > > > > 2. We should make the RelMetadataQuery in RelOptCluster pluggable.
> > > >
> > > > I disagree. RelMetadataQuery is not an extension point. Its sole
> > > > purpose is to to keep state (the cache and cycle-checking).
> > > > RelMetadataProvider is the extension point. (By analogy, if you are
> > > > un-parsing an AST, you let each AST sub-class handle unparsing itself,
> > > > but the unparsed text goes to

Apply for JIRA contributors permission

[Yuzhao Chen]

Hi Guys,

I want to contribute to Apache Calcite.
Would you please give me the permission as a contributor?
My JIRA ID is danny0405.

Best,
Danny Chan

Re: About ElasticsearchToEnumerableConverterRule

[Yuzhao Chen]

Every ConverterRule will match kind of RelNode with special calling convention, 
for ElasticsearchToEnumerableConverterRule, it is ElasticsearchRel.convention, 
you can see it from the constructor:
private ElasticsearchToEnumerableConverterRule(
   RelBuilderFactory relBuilderFactory) {
super(RelNode.class, (Predicate) r -> true,
 ElasticsearchRel.CONVENTION, EnumerableConvention.INSTANCE,
 relBuilderFactory, "ElasticsearchToEnumerableConverterRule");
}

Best,
Yuzhao Chen
在 2019年3月11日 +0800 AM11:04，Jocean shi ，写道：
> Hi All:
> I have a question.
> The ElasticsearchToEnumerableConverterRule is used to match
> RelNode.class. I think this rule can convert all RelNode. but only convert
> the root RelNode in fact. why?
>
> Jocean.shi
> Best wishes

Re: [DISCUSS] Towards Calcite 1.19.0

[Yuzhao Chen]

Nice job, Kevin

Best,
Yuzhao Chen
在 2019年3月11日 +0800 PM9:05，Kevin Risden ，写道：
> As of this morning, all JIRAs tagged for 1.19.0 have been resolved or moved
> out to 1.20.0.
>
> I will be starting the release steps this morning. Please hold off on
> commits to master while the 1.19.0 release is happening.
>
> Kevin Risden
>
>
> On Wed, Mar 6, 2019 at 9:02 AM Kevin Risden  wrote:
>
> > It looks like we haven't made any progress (JIRAs have been opened/closed)
> > towards closing down JIRAs tagged for 1.19.0. There are still 14 (14 on
> > 2/27) open JIRAs tagged for 1.19.0.
> >
> > I will start moving those JIRAs out today to 1.20.0 so I can start to
> > close those this release. We are getting to mid March at this point since
> > we keep postponing this release.
> >
> > Kevin Risden
> >
> >
> > On Thu, Feb 28, 2019 at 5:55 AM YuZhao Chan  wrote:
> >
> > > I would help to review some PRs this weekend，especially [1]. Hope to help
> > > and release some of the burden.
> > > [1]
> > > https://issues.apache.org/jira/browse/CALCITE-2018?jql=project%20%3D%20CALCITE%20AND%20resolution%20%3D%20Unresolved%20AND%20fixVersion%20in%20(EMPTY%2C%20%22next%22)%20AND%20labels%20%3D%20pull-request-available%20ORDER%20BY%20priority%20DESC
> > >
> > > Best,
> > > YuZhao Chan
> > > 在 2019年2月26日 +0800 AM8:14，Julian Hyde ，写道：
> > > > Hey everyone.
> > > >
> > > > There are 108 open pull requests. What are we going to do about it?
> > > >
> > > > Last release I reviewed and committed dozens of pull requests, and I
> > > burned out.
> > > >
> > > > Julian
> > > >
> > > >
> > > > > On Feb 22, 2019, at 1:20 PM,
> > > Kevin Risden
> > >  wrote:
> > > > >
> > > > > Yea I don't mind pushing out the RC towards the end of next week
> > > (beginning
> > > > > of March). I'm not in a rush to build the RC. I just picked Monday to
> > > try
> > > > > to hit the target of February that was arbitrarily picked in the
> > > sand. At
> > > > > this point, the release will most likely happen in early March.
> > > > >
> > > > > Kevin Risden
> > > > >
> > > > >
> > > > > On Fri, Feb 22, 2019 at 4:15 PM Julian Hyde  wrote:
> > > > >
> > > > > > Can you do the RC towards the end of next week?
> > > > > >
> > > > > > I only just saw this message - 4 days after you sent it - because
> > > I’ve
> > > > > > been fighting down a backlog of 500 Apache messages. There are some
> > > changes
> > > > > > I would like to get into the release but I will need a few days.
> > > > > >
> > > > > > Julian
> > > > > >
> > > > > >
> > > > > > > On Feb 18, 2019, at 9:56 AM,
> > > > > > Kevin Risden
> > > > > >  wrote:
> > > > > > >
> > > > > > > It looks like there are some PRs to be reviewed and some changes
> > > to get
> > > > > > > merged in this week.
> > > > > > >
> > > > > > > How does creating the first RC on Monday Feb 25th sound? Does
> > > that give
> > > > > > > enough time this week to get changes into Calcite 1.19.0?
> > > > > > >
> > > > > > > Kevin Risden
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Feb 13, 2019 at 11:08 AM Zoltan Haindrich 
> > > wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > On 2/13/19 4:24 PM, Julian Hyde wrote:
> > > > > > > > > Sorry, there’s been a misunderstanding. Let me clarify. I
> > > didn’t say
> > > > > > > > that your patches were too small. Or intend to imply it.
> > > > > > > > >
> > > > > > > > > When I said “widespread changes for no good reason” - or
> > > something like
> > > > > > > > that - I meant changes to the RexNode format due to removing IS
> > > TRUE
> > > > > > nodes.
> > > > > > > > >
> > > > > > > > > I like small patches, provided each one fixes a well defined
> > > issue and
>

Re: About ElasticsearchToEnumerableConverterRule

[Yuzhao Chen]

Then what kind of Planner you use, if it is Volcano it is not expected to 
happen.

Best,
Yuzhao Chen
在 2019年3月11日 +0800 PM11:08，Jocean shi ，写道：
> Thanks for your help.
> I get your point. But there are many other rules such as :
> private ElasticsearchFilterRule() {
> super(LogicalFilter.class, Convention.NONE, ElasticsearchRel.CONVENTION,
> "ElasticsearchFilterRule");
> }
>
> will convert the Convention.NONE to ElasticsearchRel.CONVENTION.
> The ElasticsearchToEnumerableConverterRule should take effect after that,
> but not
>
>
> Best
> Jocean.shi
>
>
> Yuzhao Chen  于2019年3月11日周一 下午5:34写道：
>
> > Every ConverterRule will match kind of RelNode with special calling
> > convention, for ElasticsearchToEnumerableConverterRule, it is
> > ElasticsearchRel.convention, you can see it from the constructor:
> > private ElasticsearchToEnumerableConverterRule(
> > RelBuilderFactory relBuilderFactory) {
> > super(RelNode.class, (Predicate) r -> true,
> > ElasticsearchRel.CONVENTION, EnumerableConvention.INSTANCE,
> > relBuilderFactory, "ElasticsearchToEnumerableConverterRule");
> > }
> >
> > Best,
> > Yuzhao Chen
> > 在 2019年3月11日 +0800 AM11:04，Jocean shi ，写道：
> > > Hi All:
> > > I have a question.
> > > The ElasticsearchToEnumerableConverterRule is used to match
> > > RelNode.class. I think this rule can convert all RelNode. but only
> > convert
> > > the root RelNode in fact. why?
> > >
> > > Jocean.shi
> > > Best wishes
> >

Re: Apply for JIRA contributors permission

[Yuzhao Chen]

Thx, Julian, I will contribute to Calcite continuously.

Best,
Yuzhao Chen
在 2019年3月12日 +0800 AM5:52，Julian Hyde ，写道：
> I’ve added you as a JIRA contributor.
>
> Note “Contributor” is a JIRA term, not an Apache term. It gives you the right 
> to assign a JIRA case to yourself.
>
> Julian
>
>
> > On Mar 9, 2019, at 5:59 PM, Yuzhao Chen  wrote:
> >
> > Hi Guys,
> >
> > I want to contribute to Apache Calcite.
> > Would you please give me the permission as a contributor?
> > My JIRA ID is danny0405.
> >
> > Best,
> > Danny Chan
>

Re: About ElasticsearchToEnumerableConverterRule

[Yuzhao Chen]

The ConverterRule is a RelOptRule but with default implementation of onMatch:
public void onMatch(RelOptRuleCall call) {
 RelNode rel = call.rel(0);
 if (rel.getTraitSet().contains(inTrait)) {
   final RelNode converted = convert(rel);
   if (converted != null) {
 call.transformTo(converted);
   }
 }
}
So, the first thing is does your ConverterRule matches ?


Best,
Danny Chan
在 2019年3月12日 +0800 PM4:01，dev@calcite.apache.org，写道：
>
> ElasticsearchRel.CONVENTION

Re: About ElasticsearchToEnumerableConverterRule

[Yuzhao Chen]

Yeah, did you replace the Convention trait of the matched node when matches ?

Best,
Danny Chan
在 2019年3月12日 +0800 PM8:59，Jocean shi ，写道：
> I sure this ConverterRule matches. In other world, this method has excuted.
> but the final result is only root RelNode has been replaced.
>
>
> Best
> Jocean.shi

Re: [CALCITE-2843] PR review request

[Yuzhao Chen]

I would take it, hole to help you.

Best,
Danny Chan
在 2019年3月13日 +0800 PM10:52，Muhammad Gelbana ，写道：
> Could someone kindly review this PR please ?
> https://github.com/apache/calcite/pull/1066
>
> Thanks,
> Gelbana

Re: Parantheses in RelBuilder?

[Yuzhao Chen]

Hi, Rakesh Nair.
RelBuilder/RexBuilder are used to construct rel/rex nodes, but parenthesis are 
not, parenthesis only defines the scope and precedence of the relational 
algebra. You only need to construct the rel/rex nodes directly with the defined 
precedence by parenthesis.

Best,
Danny Chan
在 2019年3月14日 +0800 PM7:56，Rakesh Nair ，写道：
> Hello,
> Would you mind giving me some pointers on *creating parenthesis while
> building a RelNode*.
> For example, how to build RelNode for the sample query:
> SELECT * FROM `persons` WHERE *(*`SALARY` >= 10 AND `SALARY` < 20*) *OR
> *(*`SALARY` >= 30 AND `SALARY` < 40*)*
>
> P.S. Need to know how to build the parenthesis only, everything else is
> fine..
> --
> Thanks & Regards,
> Rakesh

Re: About ElasticsearchToEnumerableConverterRule

[Yuzhao Chen]

Hi, Jocean shi

The convention is a kind of trait, and not decided by cost model and meratada. 
If you wanna output relNode  with specified trait def, you should passed the 
trait defs to the program. See [1].

[1] 
https://github.com/apache/calcite/blob/ffca956be03a99cd11e440d652b09674aaa727e6/core/src/main/java/org/apache/calcite/tools/Program.java#L38

public interface Program {
 RelNode run(RelOptPlanner planner, RelNode rel,
 RelTraitSet requiredOutputTraits,
 List materializations,
 List lattices);
}

Best,
Danny Chan
在 2019年3月14日 +0800 AM10:43，Jocean shi ，写道：
> Hi,
>
> In other world. That question can convert to that if a RelNode has a
> RelSet and the RelSet have two RelSubset then how to chose RelSubset
>
> Best
> Jocean.shi
>
>
>
>
> Stamatis Zampetakis  于2019年3月13日周三 下午4:09写道：
>
> > Hi Jocean,
> >
> > It is not very clear what is the problem you are trying to solve.
> > Maybe it could help if you could share the input plan, planner rules, and
> > the expected output.
> >
> >
> > Best,
> > Stamatis
> >
> > Στις Τετ, 13 Μαρ 2019 στις 2:51 π.μ., ο/η Yuzhao Chen <
> > yuzhao@gmail.com>
> > έγραψε:
> >
> > > Yeah, did you replace the Convention trait of the matched node when
> > > matches ?
> > >
> > > Best,
> > > Danny Chan
> > > 在 2019年3月12日 +0800 PM8:59，Jocean shi ，写道：
> > > > I sure this ConverterRule matches. In other world, this method has
> > > excuted.
> > > > but the final result is only root RelNode has been replaced.
> > > >
> > > >
> > > > Best
> > > > Jocean.shi
> > >
> >

Re: [ANNOUNCE] New committer: Stamatis Zampetakis

[Yuzhao Chen]

Congratulations, Stamatis Zampetakis !
Thank you for your work of Apache Calcite.

Best,
Danny Chan
在 2019年3月15日 +0800 PM12:09，Muhammad Gelbana ，写道：
> Congratulations Stamatis :)
>
> Thanks for frequently answering my questions and discussing my raised
> topics.
>
> On Fri, Mar 15, 2019, 2:45 AM Michael Mior  wrote:
>
> > No problem. I'll leave that to you then once the release is done :)
> > Thanks!
> > --
> > Michael Mior
> > mm...@apache.org
> >
> > Le jeu. 14 mars 2019 à 19:03, Stamatis Zampetakis  a
> > écrit :
> > >
> > > Thanks for noticing Michael.
> > >
> > > Actually, I started doing it at some point but then there were
> > > inconsistencies between master, site, and svn, so I decided to do it
> > after
> > > the release where everything is in line.
> > >
> > > On Thu, Mar 14, 2019, 10:15 PM Francis Chuang 
> > > wrote:
> > >
> > > > I also noticed Hongze was not added to the community page as well. Once
> > > > Kevin releases 1.19.0, we should add both of them to the page.
> > > >
> > > > On 15/03/2019 2:11 am, Michael Mior wrote:
> > > > > I just noticed that Stamatis was never added to the community page
> > > > > site. Stamatis, feel free to add yourself once the freeze for the
> > > > > current release is over. Otherwise, I'm happy to do so.
> > > > > --
> > > > > Michael Mior
> > > > > mm...@apache.org
> > > > >
> > > > > Le mer. 30 janv. 2019 à 13:01, Jesus Camacho Rodriguez
> > > > >  a écrit :
> > > > > >
> > > > > > Apache Calcite's Project Management Committee (PMC) has invited
> > > > > > Stamatis Zampetakis to become a committer, and we are pleased to
> > > > > > announce that he has accepted.
> > > > > >
> > > > > > Over the past few months, Stamatis has made several contributions to
> > > > > > Calcite and he is a very active participant in discussions in issues
> > > > > > and mailing lists.
> > > > > >
> > > > > > Stamatis, welcome, thank you for your contributions, and we look
> > > > > > forward your further interactions with the community! If you wish,
> > > > > > please feel free to tell us more about yourself and what you are
> > > > > > working on.
> > > > > >
> > > > > > Jesús (on behalf of the Apache Calcite PMC)
> > > >
> > > >
> >

How to start up the vm environment after i do Vagrant halt ?

[Yuzhao Chen]

Hi, guys.

After I did the cmd: vagrant halt and vagrant up, i can not connect to the 
druid broker again, so what it the right way to halt and start up ?


Best,
Danny Chan

Re: Conditionally Adding Item to a Map

[YuZhao Chen]

 Or you can right a UDF by yourself, if you encouter

Best,
Danny Chan
在 2019年3月21日 +0800 AM2:45，Shahar Cizer Kobrinsky ，写道：

Hey All,

New to Calcite, trying to express a select statement in SQL where i build a
map.
The thing is i want to conditionally add items to it, so for example have
key 'c' with value from column 'c' only if the value is not null

SELECT a,b, map['c',c, 'd', d] as my_map FROM...

Is there a way to conditionally have key/values in the map?

Thanks!
Shahar

Re: Conditionally Adding Item to a Map

[Yuzhao Chen]

Or you can right a UDF by yourself, if you encouter an null value you can just 
return null and in outer you can filter out the null values.

Best,
Danny Chan
在 2019年3月21日 +0800 AM2:45，Shahar Cizer Kobrinsky 
，写道：
> Hey All,
>
> New to Calcite, trying to express a select statement in SQL where i build a
> map.
> The thing is i want to conditionally add items to it, so for example have
> key 'c' with value from column 'c' only if the value is not null
>
> SELECT a,b, map['c',c, 'd', d] as my_map FROM...
>
> Is there a way to conditionally have key/values in the map?
>
> Thanks!
> Shahar

Re: Isssue with th ekeyword Timestamp in sql query validation.

[YuZhao Chen]

timestamp is neither a keyword nor a builtin function in Calcite now, so
what is the stackTrace your query throw ? Can you show us more details ?

Best,
Danny Chan

Suhail Aliyar  于2019年3月22日周五 上午2:26写道：

> Hello,
>
> I am Suhail Aliyar, Backend developer at dexlock technologies, Kochi,
> India.
>
> I am using calcite server for validating SQL queries in my MSSQL server
> using java in my application. While using your latest version(1.18.0) from
> maven, The validator throws exceptions. The queries are given below. I have
> tried the same queries from DBeaver and I am able to fetch results.
>
> Sample Queries:
> 1. select timeStamp from ;
> 2. SELECT   *, FLOOR(timeStamp / (1000)) % 60 AS seconds FROM
> dbo.enriched_video
> 3. SELECT *, CONVERT(VARCHAR,DATEADD(ms,timestamp,0),108) AS time FROM
> dbo.enriched_video
>
> In my table I have a attribute names "timeStamp".
>
> Can anyone please provide me some quick help to make support to these
> keywords. Thanks for helping.
>
> *Thanks & Regards*
> *Suhail Aliyar*
> *Software Engineer*
> *Mob:+91 9846449816*
> *Website: www.dexlock.com *
>

Re: Join, SemiJoin, Correlate

[YuZhao Chen]

I kind of prefer 3, for the reasons below:

1. It makes the concept unified and clean.
2. We somehow can not remove the Correlate, cause it's a good logical
representation for correlated query, it can be de-correlated to a HashJoin
or keep as a NestedLoopJoin, for functionality, it seens like Hive
QBSubQuery[1], and Hive uses JoinType[2] to enumerate different joins. An
enumeration have much cleaner concept than just functional methods.

[1]
https://github.com/apache/hive/blob/2fa22bf360898dc8fd1408bfcc96e1c6aeaf9a53/ql/src/java/org/apache/hadoop/hive/ql/parse/QBSubQuery.java#L41
[2]
https://github.com/apache/hive/blob/2fa22bf360898dc8fd1408bfcc96e1c6aeaf9a53/ql/src/java/org/apache/hadoop/hive/ql/parse/JoinType.java#L26

Best,
Danny Chan

Julian Hyde  于2019年3月22日周五 上午2:55写道：

> I have a few ideas for refactorings. (I’m not convinced by any of them,
> but let me know which you like.)
>
> 1. Get rid of SemiJoinType. It is mis-named (it is not used by SemiJoin,
> it is used by Correlate, but in a field called joinType).
>
> 2. In Correlate, use org.apache.calcite.linq4j.CorrelateJoinType. It has
> the same set of values as SemiJoinType, but it has a better name.
>
> 3. Get rid of both SemiJoinType and CorrelateJoinType, and use JoinRelType
> for everything. We would have to add SEMI and ANTI values. Also some
> methods to find out whether the resulting row type contains fields from the
> left and right inputs or just the left input.
>
> 4. Add “interface JoinLike extends BiRel” and make Join, SemiJoin and
> Correlate implement it. It would have a methods that say whether the LHS
> and RHS generate nulls, and whether the output row type contains columns
> from the right input. This seems attractive because it lets Join, SemiJoin
> and Correlate continue to be structurally different.
>
> Julian
>
>
>
>
> > On Mar 20, 2019, at 6:55 PM, Haisheng Yuan 
> wrote:
> >
> > SubPlan (in Postgres’ term) is a Postgres physical relational node to
> evaluate correlated subquery. What I mean is correlated subquery that can’t
> be decorrelated can’t be implemented by hashjoin or mergejoin. But it is
> off topic.
> >
> > Thanks ~
> > Haisheng Yuan
> > --
> > 发件人：Walaa Eldin Moustafa
> > 日 期：2019年03月21日 09:31:41
> > 收件人：
> > 抄 送：Stamatis Zampetakis
> > 主 题：Re: Re: Join, SemiJoin, Correlate
> >
> > Agreed with Stamatis. Currently: 1) Correlate is tied to IN, EXISTS,
> > NOT IN, NOT EXISTS, and 2) is used as an equivalent to nested loops
> > join. The issues here are: 1) IN, EXISTS, NOT IN, NOT EXISTS can be
> > rewritten as semi/anti joins, and 2) nested loops join is more of a
> > physical operator.
> >
> > It seems that the minimal set of logical join types are INNER, LEFT,
> > RIGHT, OUTER, SEMI, ANTI.
> >
> > So I think Calciate could have one LogicalJoin operator with an
> > attribute to specify the join type (from the above), and a number of
> > physical join operators (hash, merge, nested loops) whose
> > implementation details depend on the the join type.
> >
> > What we lose by this model is the structure of the query (whether
> > there was a sub-plan or not), but I would say that this is actually
> > what is desired from a logical representation -- to abstract away from
> > how the query is written, and how it is structured, as long as there
> > is a canonical representation. There could also be a world where both
> > models coexist (Correlate first then Decorrelate but in the light of a
> > single logical join operator?).
> >
> > @Haisheng, generally, a sub-plan can also be implemented using a
> > variant of hash or merge joins as long as we evaluate the sub-plan
> > independently (without the join predicate), but that is up to the
> > optimizer.
> >
> > Thanks,
> > Walaa.
> >
> > On Wed, Mar 20, 2019 at 5:23 PM Haisheng Yuan 
> wrote:
> >>
> >> SemiJoinType and its relationship with JoinRelType do confuse me a
> little bit.
> >>
> >> But I don’t think we should not have LogicalCorrelate. It is useful to
> represent the lateral or correlated subquery (aka SubPlan in Postgres
> jargon). The LogicalCorrelate can be implemented as NestLoopJoin in
> Calcite, or SubPlan in Postgres’s terminology, but it can’t be implemented
> as HashJoin or MergeJoin.
> >>
> >> Thanks ~
> >> Haisheng Yuan
> >> --
> >> 发件人：Stamatis Zampetakis
> >> 日 期：2019年03月21日 07:13:15
> >> 收件人：
> >> 主 题：Re: Join, SemiJoin, Correlate
> >>
> >> I have bumped into this quite a few times and I think we should really
> try
> >> to improve the design of the join hierarchy.
> >>
> >> From a logical point of view I think it makes sense to have the
> following
> >> operators:
> >> InnerJoin, LeftOuterJoin, FullOuterJoin, SemiJoin, AntiJoin, (GroupJoin)
> >>
> >> Yet I have not thought thoroughly what should become a class, and what a
> >> property of the class (e.g., JoinRelType, SemiJoinType).
> >>
> >> Moreover, Correlate as it is rig

Re: missing Schema.contentsHaveChangedSince in v1.18

[Yuzhao Chen]

I only found a method about cacheing in SchemaPlus [1], but the 
Schema.contentsHaveChangedSince only appears in the doc, i look up the commits 
and it seems that this method is gone from very old version [2], maybe Julian 
can answer your question.

[1] 
https://github.com/apache/calcite/blob/a648f9c12309cc253628930b0cab98591caa66ab/core/src/main/java/org/apache/calcite/schema/SchemaPlus.java#L85
[2] https://github.com/julianhyde/optiq/issues/175

Best,
Danny Chan
在 2019年3月25日 +0800 PM3:11，dev@calcite.apache.org，写道：
>
> Schema

Re: Calcite doesn't work with LOOKAHEAD(3)

[Yuzhao Chen]

Maybe we should fire a jira if it is a bug.

Best,
Danny Chan
在 2019年3月26日 +0800 PM8:33，Hongze Zhang ，写道：
> Ops, correct a typo:
>
> "... after uncommenting a line ..." -> "... after commenting a line
> ...".
>
> Best,
> Hongze
>
> -- Original Message --
> From: "Hongze Zhang" 
> To: dev@calcite.apache.org
> Sent: 2019/3/26 19:28:08
> Subject: Re: Calcite doesn't work with LOOKAHEAD(3)
>
> > Firstly, thank you very much for sharing the case, Rui!
> >
> > I have run a test with the SQL you provided and also run into the same 
> > exception (under a global LOOKAHEAD 3). After debugging the generated 
> > parser code, I think the problem is probably in the generated LOOKAHEAD 
> > method SqlParserImpl#jj_3R_42():
> >
> >
> > > final private boolean jj_3R_42() {
> > > if (!jj_rescan) trace_call("SqlSelect(LOOKING AHEAD...)");
> > > if (jj_scan_token(SELECT)) { if (!jj_rescan) 
> > > trace_return("SqlSelect(LOOKAHEAD FAILED)"); return true; }
> > > if (jj_3R_190()) { if (!jj_rescan) trace_return("SqlSelect(LOOKAHEAD 
> > > FAILED)"); return true; }
> > > { if (!jj_rescan) trace_return("SqlSelect(LOOKAHEAD SUCCEEDED)"); return 
> > > false; }
> > > }
> >
> > The LOOKAHEAD method checks only a single token . This is 
> > definitely not enough since we have already set the number to 3.
> >
> > Unfortunately I didn't find a root cause so far, but after uncommenting a 
> > line[1] in production "SqlSelect()" then everything goes back to normal. 
> > I'm inclined to believe JavaCC has some unexpected behavior when dealing 
> > with LOOKAHEAD on a production with the shape like "SqlSelectKeywords()"[2].
> >
> > Please feel free to log a JIRA ticket with which we can track further 
> > information of the issue.
> >
> > Best,
> > Hongze
> >
> >
> > [1] 
> > https://github.com/apache/calcite/blob/1b430721c0d9e22b2252ffcd893b42959cb7966c/core/src/main/codegen/templates/Parser.jj#L1030
> > [2] 
> > https://github.com/apache/calcite/blob/1b430721c0d9e22b2252ffcd893b42959cb7966c/core/src/main/codegen/templates/Parser.jj#L288
> >
> > -- Original Message --
> > From: "Rui Li" 
> > To: dev@calcite.apache.org
> > Sent: 2019/3/26 16:53:44
> > Subject: Calcite doesn't work with LOOKAHEAD(3)
> >
> > > Hi,
> > >
> > > I'm trying to extend Calcite grammar to support some custom statements. 
> > > And
> > > I need to increase LOOKAHEAD to 3 to resolve some ambiguity. But when I 
> > > did
> > > that, the parser fails to parse queries like:
> > > * select t.key from (select key from src) t*
> > >
> > > With exception:
> > > *Caused by: org.apache.calcite.sql.parser.impl.ParseException:*
> > > *Encountered "( select key" at line 1, column 19.*
> > > *Was expecting one of:*
> > > *  ...*
> > > *  ...*
> > > *  ...*
> > > *  ...*
> > > *  ...*
> > > * "LATERAL" ...*
> > > * "(" "WITH" ...*
> > > *...*
> > >
> > > So I'm wondering whether there's some limitation on the LOOKAHEAD we can
> > > use?
> > >
> > > --
> > > Best regards!
> > > Rui Li