Re: Re: [DISCUSS] On-demand traitset request
Jinfeng Ni's proposal +1, enhance method satisfies maybe more reasonable. Danny Chan's thoughts +1 Regards! Aron Tao XING JIN 于2019年11月8日周五 下午8:09写道： > Hi Haisheng, > > Thanks a lot for sharing this great proposal ~ > For short I understand your idea as below: > 1. Derive the distributions/collations that children COULD/MIGHT offer > 2. Decide the best distributions/collations by first point and computing > logic of operator, say gropuings in Aggregate; > > It comes to me that another important part is that children operator should > also provide the corresponding COST for the POSSIBLE > distribution/collation. > The COST is not for the final plan, but a hypothesis. > > Take below example > SELECT DISTINCT c, b FROM > ( SELECT R.c c, S.b b FROM R, S > WHERE R.a=S.a and R.b=S.b and R.c=S.c) t; > Suppose R is ordered by (c, b, a), and S is ordered by (b, c, a). > Aggregate > +--- InnerJoin > |--- TableScan on R > +--- TableScan on S > > InnerJoin should deliver that its possible collations and corresponding > costs at the same time. > - If ordered by (c, b, a) my cost is ... > - If ordered by (b, c, a) my cost is ... > - If ordered by (a, b, c) my cost is ... > By which Aggregate decide the 'best' required collation. > By this way we can better limit the searching space and also target the > relatively optimized (if not best) plan. > > Also when you say "I didn't say adding to RelNode, but a new API/interface > for physical operator only.", I'm not so clear; > Currently the physical operators in Calcite like EnumerableHashJoin, > EnumerableMergeJoin, when created, their physical behavior(like real > collations) are determined. > So I belive you intend to add new API at upper layer, but there's no > physical optimizing phase in Calcite at this moment. Where do you want to > add the new API, can you specify ? > > Thanks, > Jin > > Jinfeng Ni 于2019年11月6日周三 上午1:56写道： > > > @Haisheng, @Xiening, > > > > Thanks for pointing that previous email out. Overall, I agree that > > the physical trait enforcement should be done in the engine, not in > > the rule. For the rule, it should only specify the request, and the > > corresponding transformation, and let the engine to explore the search > > space. It will be great if we can revamp the Volcano optimizer > > framework, to do that way. > > > > In terms of search space, it's always a tradeoff between the space > > searched and the optimality of the plan found. I think it's fine for > > the engine to explore a potential big search space, as long as it has > > effective "bound-and-prune" strategy. In the original Volcano paper, > > there is a way to prune the search space based on the best plan found > > so far, using the parameter "limit". When an implementable plan is > > found, a "real" cost is obtained, which could be used to prune > > un-necessary search space. That's actually the advantage of Volcano's > > "top-down" approach. However, seems to me that Calcite's Volcano did > > not apply that approach effectively, because of the existence of > > AbstractConverter. > > > > > > On Sun, Nov 3, 2019 at 10:12 PM Haisheng Yuan > > wrote: > > > > > > Hi Jinfeng, > > > > > > I think you might have missed the email about proposed API for physical > > operators I sent out previously in [1]. > > > > > > We don't need request all the permutation, which is also impossible in > > practice, the search space is going to explode. > > > > > > In the example in email [1], I already talked about your concen on > > passing down parent request into children may lead to less optimal plan. > > Besically join operator can send 2 collation optimization requests, one > is > > to pass request down, the other one is ignore the parent's request. > > > > > > Using AbstractConverter to enforce properties is inapporpriate, which > > handles all the optimization work to physical operator providers, meaning > > there is almost no physical level optimization mechanism in Calcite. SQL > > Server and Greenplum's optimizer, which are Cascades framework based, > > implemented the property enforcement in the core optimizer engine, not > > through AbstractConverter and rules, physical operators just need to > > implement those methods (or similar) I mentioned in email [1]. My goal is > > completely abolishing AbstractConverter. > > > > > > [1] > > > http://mail-archives.apache.org/mod_mbox/calcite-dev/201910.mbox/%3cd75b20f4-542a-4a73-897e-66ab426494c1.h.y...@alibaba-inc.com%3e > > > > > > - Haisheng > > > > > > -- > > > 发件人：Jinfeng Ni > > > 日 期：2019年11月01日 14:10:30 > > > 收件人： > > > 主 题：Re: [DISCUSS] On-demand traitset request > > > > > > Hi Xiening, > > > > > > "Let say if R and S doesn’t have sorting properties at all. In your > > > case, we would end up adding enforcers for LHS and RHS to get > > > collation (a, b, c). Then we would need another enforcer to get > > > collation (b, c). This is a sub optimal
When will the exchange node(Distribution) be added to the execution plan
Hi
I wonder when will the exchange node be added to the execution plan. For
example, In Spark, if a join is SMJ(SortMergeJoin), it will add an exchange
and a sort node to the execution plan:
[image: 3631580619602_.pic.jpg]
In Calcite, Let me use CsvTest#testReadme for example and I can find a
sorting trait if the join is SMJ, but I can not find an exchange.
The SQL:
SELECT d.name, COUNT(*) cnt
FROM emps AS e
JOIN depts AS d ON e.deptno = d.deptno
GROUP BY d.name;
The plan in volcano planner, see
`rel#76:EnumerableMergeJoin.ENUMERABLE.[[0], [2]]`, we can see the
conversion and the Collation, but no distribution.
appendix
Set#6, type: RecordType(INTEGER DEPTNO, VARCHAR NAME, INTEGER DEPTNO0)
rel#51:Subset#6.NONE.[], best=null, importance=0.6561
rel#49:LogicalJoin.NONE.[](left=RelSubset#30,right=RelSubset#29,condition==($2,
$0),joinType=inner), rowcount=1500.0, cumulative cost={inf}
rel#60:LogicalProject.NONE.[](input=RelSubset#32,DEPTNO=$1,NAME=$2,DEPTNO0=$0),
rowcount=1500.0, cumulative cost={inf}
rel#55:Subset#6.ENUMERABLE.[], best=rel#78,
importance=0.7291
rel#70:EnumerableProject.ENUMERABLE.[](input=RelSubset#46,DEPTNO=$1,NAME=$2,DEPTNO0=$0),
rowcount=1500.0, cumulative cost={3686.517018598809 rows, 4626.25 cpu, 0.0
io}
rel#76:EnumerableMergeJoin.ENUMERABLE.[[0],
[2]](left=RelSubset#74,right=RelSubset#75,condition==($2,
$0),joinType=inner), rowcount=1500.0, cumulative cost={inf}
rel#78:EnumerableHashJoin.ENUMERABLE.[](left=RelSubset#30,right=RelSubset#69,condition==($0,
$2),joinType=inner), rowcount=1500.0, cumulative cost={2185.517018598809
rows, 126.25 cpu, 0.0 io}
--
Regards!
Aron Tao
--
Regards!
Aron Tao
Re: Calcite-Master - Build # 1588 - Failure
Alessandro> What do you usually do in such cases (transient failures)? There are multiple ways (it no order): A) Ignore the failure (as it is not related to your changes) B) Ignore the failure and add a comment on the PR/JIRA like "CI fails with timeout in org.apache.calcite.adapter.elasticsearch.Projection2Test > initializationError which seems to be unrelated " C) Analyze the root cause and fix it (e.g. provide a separate PR or just add a commit to existing PR) D) Log a JIRA case. JIRA should have a copy of the stacktrace (CI logs might fade over time). See samples at https://issues.apache.org/jira/browse/CALCITE-3750?filter=-2&jql=project%3DCALCITE%20AND%20text%20~%20Intermittent%20AND%20resolution%20%3D%20Unresolved%20%20ORDER%20BY%20created%20DESC E) Send an email to dev@calcite.a.o as you did When I review PRs I can tolerate CI failures (== I can review even if there are failures). However, it still takes time to open the relevant log and verify the reason for the failure, so it really helps when PR author summarizes CI status in case of failure. Vladimir
Re: Calcite-Master - Build # 1588 - Failure
Thanks a lot Vladimir, your fix on master indeed solved my issue. Unfortunately this time AppVeyor failed on ElasticSearchAdapterTest due to a "java.net.SocketTimeoutException" (link to the build https://ci.appveyor.com/project/ApacheSoftwareFoundation/calcite/builds/30510892/job/0o5meoj2omuxadi8 ). What do you usually do in such cases (transient failures)? Shall I wait or try to re-trigger Travis again? Best regards, Alessandro On Sat, 1 Feb 2020 at 22:50, Vladimir Sitnikov wrote: > >My PR is this one: https://github.com/apache/calcite/pull/1774 > > Ok. I see. The error in your PR is exactly the same. > I guess it was caused by > > https://github.com/apache/calcite/commit/bcac62e3dad6137511d4451135daa2d1762ec6ad#diff-c25007f5834d66d28e0b6644edf21325L55-R69 > > In other words, `master` branch build was broken, and PR is validated by > merging with master branch. > I've fixed master branch, so CI in your PR would likely work. > > Vladimir >
Re: Calcite-Master - Build # 1588 - Failure
>My PR is this one: https://github.com/apache/calcite/pull/1774 Ok. I see. The error in your PR is exactly the same. I guess it was caused by https://github.com/apache/calcite/commit/bcac62e3dad6137511d4451135daa2d1762ec6ad#diff-c25007f5834d66d28e0b6644edf21325L55-R69 In other words, `master` branch build was broken, and PR is validated by merging with master branch. I've fixed master branch, so CI in your PR would likely work. Vladimir
Re: Calcite-Master - Build # 1588 - Failure
Hi Vladimir, I see, so most probably not related to the problem I am having. What do you mean by "Calcite never uses Jenkins for PR validation."? My PR is this one: https://github.com/apache/calcite/pull/1774 Best regards, Alessandro On Sat, 1 Feb 2020 at 22:39, Vladimir Sitnikov wrote: > Hi, > > Calcite never uses Jenkins for PR validation. > > >I had a CI failure on my PR (on code unrelated to my PR which concerns > only > CassandraAdapter), > > Can you provide the link to your PR? > > > >Status: Failure > >Check console output at > https://builds.apache.org/job/Calcite-Master/1588/ to > view the results. > > This failure is a true failure. > > The error is below: > > CoreQuidemTest > [13] sql/functions.iq FAILED > org.opentest4j.AssertionFailedError: Files differ: > /home/jenkins/jenkins-slave/workspace/Calcite-Master/jdk/JDK 1.8 > > (latest)/label_exp/ubuntu&&!cloud-slave&&!H27&&!ubuntu-4&&!test-nodes/ws/core/build/resources/test/sql/surefire/sql/ > functions.iq > /home/jenkins/jenkins-slave/workspace/Calcite-Master/jdk/JDK 1.8 > > (latest)/label_exp/ubuntu&&!cloud-slave&&!H27&&!ubuntu-4&&!test-nodes/ws/core/build/resources/test/sql/ > functions.iq > 65a66 > > (1 row) > at > org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:39) > at org.junit.jupiter.api.Assertions.fail(Assertions.java:109) > at org.apache.calcite.test.QuidemTest.checkRun(QuidemTest.java:155) > at org.apache.calcite.test.QuidemTest.test(QuidemTest.java:202) > > Vladimir >
Re: Calcite-Master - Build # 1588 - Failure
Hi, Calcite never uses Jenkins for PR validation. >I had a CI failure on my PR (on code unrelated to my PR which concerns only CassandraAdapter), Can you provide the link to your PR? >Status: Failure >Check console output at https://builds.apache.org/job/Calcite-Master/1588/ to view the results. This failure is a true failure. The error is below: CoreQuidemTest > [13] sql/functions.iq FAILED org.opentest4j.AssertionFailedError: Files differ: /home/jenkins/jenkins-slave/workspace/Calcite-Master/jdk/JDK 1.8 (latest)/label_exp/ubuntu&&!cloud-slave&&!H27&&!ubuntu-4&&!test-nodes/ws/core/build/resources/test/sql/surefire/sql/functions.iq /home/jenkins/jenkins-slave/workspace/Calcite-Master/jdk/JDK 1.8 (latest)/label_exp/ubuntu&&!cloud-slave&&!H27&&!ubuntu-4&&!test-nodes/ws/core/build/resources/test/sql/functions.iq 65a66 > (1 row) at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:39) at org.junit.jupiter.api.Assertions.fail(Assertions.java:109) at org.apache.calcite.test.QuidemTest.checkRun(QuidemTest.java:155) at org.apache.calcite.test.QuidemTest.test(QuidemTest.java:202) Vladimir
Re: Calcite-Master - Build # 1588 - Failure
Hello, I had a CI failure on my PR (on code unrelated to my PR which concerns only CassandraAdapter), as per documentation I have re-triggered with an empty commit and I get errors again. At first I thought about a difference between my local (Linux) setup and the Linux job in Travis, but the second run saw also Windows and MacOSX failure, which passed before the empty commit. Am I the only one experiencing this? Best regards. Alessandro On Sat, 1 Feb 2020 at 22:26, Apache Jenkins Server < jenk...@builds.apache.org> wrote: > The Apache Jenkins build system has built Calcite-Master (build #1588) > > Status: Failure > > Check console output at https://builds.apache.org/job/Calcite-Master/1588/ > to view the results.
Calcite-Master - Build # 1588 - Failure
The Apache Jenkins build system has built Calcite-Master (build #1588) Status: Failure Check console output at https://builds.apache.org/job/Calcite-Master/1588/ to view the results.
