Re: Summary of experience using Calcite

2022-08-04 Thread Vladimir Ozerov
Hi Sandeep, Thank you for sharing your experience. Your feedback matches with what I usually see in practice. IMO there are two main problems with Apache Calcite: 1. Documentation that focuses on a single edge case - the end-to-end execution of federated queries with Enumerable. In

Re: Default operator override in Apache Calcite

2022-07-14 Thread Vladimir Ozerov
d be inferred and which operand combinations are valid or not. This is the key problem: MySQL and Postgres both have PLUS operator, but their behavior is pretty different. чт, 14 июл. 2022 г. в 10:14, Vladimir Ozerov : > Hi Julian, > > The motivation for the proposal is a requirement to cha

Re: Default operator override in Apache Calcite

2022-07-14 Thread Vladimir Ozerov
t; On Jul 10, 2022, at 9:41 PM, Yanjing Wang > wrote: > > > > +1, Thanks Vladimir for pointing the pains out and the solutions looks > more > > clean and extensible. > > > > Vladimir Ozerov 于2022年7月9日周六 15:01写道: > > > >> Hi, > >> > >> Apache Ca

Default operator override in Apache Calcite

2022-07-09 Thread Vladimir Ozerov
Hi, Apache Calcite has a powerful but complicated and non-documented function library. Some projects may require overriding some of the existing operators to introduce custom type deduction, custom validation, etc. This includes the base arithmetic functions (e.g, disallow INT + VARCHAR),

Re: [ANNOUNCE] Vladimir Ozerov joins Calcite PMC

2022-06-06 Thread Vladimir Ozerov
Hi everybody, Thank you very much! пт, 27 мая 2022 г. в 05:27, Yanjing Wang : > Congrats Vladimir! > > Forward Xu 于2022年5月25日周三 15:18写道: > > > Congratulations Vladimir! > > > > > > Best, > > > > ForwardXu > > > > Stamatis Zampetakis 于2022年5月25日周三 15:05写道: > > > > > Congratulations Vladimir.

[jira] [Created] (CALCITE-5175) IndexOutOfBoundsException when query has IN clause on top of the view column

2022-06-01 Thread Vladimir Ozerov (Jira)
Vladimir Ozerov created CALCITE-5175: Summary: IndexOutOfBoundsException when query has IN clause on top of the view column Key: CALCITE-5175 URL: https://issues.apache.org/jira/browse/CALCITE-5175

[jira] [Created] (CALCITE-5174) EnumerableHashJoin filter out NULL values for "IS NOT DISTINCT FROM"

2022-06-01 Thread Vladimir Ozerov (Jira)
Vladimir Ozerov created CALCITE-5174: Summary: EnumerableHashJoin filter out NULL values for "IS NOT DISTINCT FROM" Key: CALCITE-5174 URL: https://issues.apache.org/jira/browse/CA

Re: Changes to the rule pattern interface

2022-04-15 Thread Vladimir Ozerov
alcite config, these two concepts are merged. In some ways it > > makes > > > > things > > > > > simpler for trivial cases. However, it is less formal and causes > > > pain > > > > when > > > > > you have required properties that have

Changes to the rule pattern interface

2022-04-12 Thread Vladimir Ozerov
Hi folks, Rules are an essential part of the Calcite-based query optimizers. A typical optimizer may require dozens of custom rules that are created by extending some Apache Calcite interfaces. During the last two years, there were two major revisions of how rules are created: 1. In early

Re: Why RelBuilder.project unwraps SARGs?

2022-04-04 Thread Vladimir Ozerov
y(). We don’t promise > not to improve our rewrites from one release to the next. > > I think you should fix it. > > Julian > > > On Apr 2, 2022, at 8:30 AM, Vladimir Ozerov wrote: > > > > Hi, > > > > When simplification is enabled in the RelBuilde

[jira] [Created] (CALCITE-5083) In RelBuilder.project_, do not unwrap SARGs

2022-04-04 Thread Vladimir Ozerov (Jira)
Vladimir Ozerov created CALCITE-5083: Summary: In RelBuilder.project_, do not unwrap SARGs Key: CALCITE-5083 URL: https://issues.apache.org/jira/browse/CALCITE-5083 Project: Calcite

Why RelBuilder.project unwraps SARGs?

2022-04-02 Thread Vladimir Ozerov
Hi, When simplification is enabled in the RelBuilder, a call to the RelBuilder.project unwraps SARGs into their flat counterparts [1]. This is not the case for other nodes, like Filter or Join. This behavior may lead to an infinite loop when the ProjectReduceExpressionsRule is used with the

Allow Cascades driver invoking "derive" on the nodes produced by "passThrough"

2022-02-10 Thread Vladimir Ozerov
Hi, In the Cascades driver, it is possible to propagate the requests top-down using the "passThrough", method and then notify parents bottom-up about the concrete physical implementations of inputs using the "derive" method. In some optimizers, the valid parent node cannot be created before the

Re: Sort getting removed during optimization

2022-01-13 Thread Vladimir Ozerov
In my case this was a LogicalProject. > I have no idea how the Project itself then assures that the CollationTrait > is fulfilled. > > By the way, is there a way to shot the Traits from a RelNode Tree? This > could help to analyze this kind of situations? > > Th

Re: [DISCUSS] Writing good summaries for Jira cases

2022-01-12 Thread Vladimir Ozerov
Hi Julian, In my opinion, both ways work well. People tend to think differently. Some prefer symptoms, others - the root cause. I personally prefer the latter for the following reason. If I face a problem, I first try to debug it on my own. The result of the analysis is usually some questionable

Re: Different behavior bewteen '>' and '='

2022-01-11 Thread Vladimir Ozerov
Hi, If I recall correctly, the SQL standard is mostly silent on how one should coerce operands. Therefore different systems implement the coercion at their discretion. Moreover, the type inference might be influenced not only by operands types but by their nature as well. For example, a target

Re: Sort getting removed during optimization

2022-01-11 Thread Vladimir Ozerov
Hi Julian, When invoking the optimizer, you may provide the desired trait set of the top-level node. It might happen, that the specific collation is not requested from the optimizer, and hence the plan with a top-level Sort operator is not chosen. Could you please show how you invoke the planner?

[jira] [Created] (CALCITE-4978) Allow custom constant map in RelOptPredicateList

2022-01-09 Thread Vladimir Ozerov (Jira)
Vladimir Ozerov created CALCITE-4978: Summary: Allow custom constant map in RelOptPredicateList Key: CALCITE-4978 URL: https://issues.apache.org/jira/browse/CALCITE-4978 Project: Calcite

Re: Acquiring original SQL identifier from RexInputRef

2021-12-21 Thread Vladimir Ozerov
ot be > modified. > > Does that make sense? If there is a more intelligent way to conditional > predicates from the SQL query, even if it isn't at the Rex level I would > love to hear suggestions > > [1] - https://github.com/dask-contrib/dask-sql > > On Tue, Dec 21, 2021 at 1:05 PM

Re: Acquiring original SQL identifier from RexInputRef

2021-12-21 Thread Vladimir Ozerov
Hi Jeremy, Could you please share the use case behind this requirement? In the general case, it is not possible to link RelNode's attributes to specific identifiers. For this reason, an attempt to extract such identifier from any "rel" except for the RelRoot might indicate a design issue.

Re: Committer permissions

2021-12-21 Thread Vladimir Ozerov
https://gitbox.apache.org/setup/ to check your count. It includes three > > procedures: > > > > 1) authority for Apache Count > > 2) authority for Github Count > > 3) authority for Apache Organization > > > > > > Vladimir Ozerov 于2021年12月18日周六

Re: RelNode semantic evolution vs adapter implementations

2021-12-21 Thread Vladimir Ozerov
The answer depends on the definition of "wrong". Rules may produce non-equivalent operators, the validator may accidentally change how type is deduced for a specific expression, metadata may cause incorrect cost calculation and pruning of promising plans, leading to a hang in a downstream

Re: Behavior of SINGLE_VALUE aggregate

2021-12-21 Thread Vladimir Ozerov
Hi Chathura, While the SINGLE_VALUE function could be used explicitly, usually this is a result of a scalar subquery unnesting, and the goal of the function is to ensure that the underlying operator returns no more than one row (otherwise, this is not a scalar). Regards, Vladimir. вт, 21 дек.

Re: [Calcite Release] Preparing for 1.29.0 Release

2021-12-18 Thread Vladimir Ozerov
Hi Xiong, Thank you very much. Vladimir. Сб, 18 дек. 2021 г. в 14:32, xiong duan : > Hi Vladimir, > > I have merged. > > Vladimir Ozerov 于2021年12月18日周六 16:58写道: > > > Hi Xiong, > > > > I've just resolved the conflicts. > > > > Regards, > >

Re: [Calcite Release] Preparing for 1.29.0 Release

2021-12-18 Thread Vladimir Ozerov
Hi Xiong, I've just resolved the conflicts. Regards, Vladimir. сб, 18 дек. 2021 г. в 10:48, xiong duan : > Hi, Vladimir. > > The PR[https://github.com/apache/calcite/pull/2464] looks good. If you > have > free time Please resolve the conflicts. Then we can merge it. > > Vl

Committer permissions

2021-12-18 Thread Vladimir Ozerov
Hi, It seems that I do not have the committer permissions to the repo, as I can merge PR neither from GitHub, nor manually. Could you please grant me the required access rights? Please let me know if any additional information is required. Regards, Vladimir.

Re: [Calcite Release] Preparing for 1.29.0 Release

2021-12-17 Thread Vladimir Ozerov
Hi, Could you please also merge https://github.com/apache/calcite/pull/2464 ? I guess it got stuck somehow, as all the comments are addressed. This issue forced us to create a copy of the AggregateUnionTransposeRule in our project. Regards, Vladimir Сб, 18 дек. 2021 г. в 09:51, Rui Wang : >

Re: RelNode semantic evolution vs adapter implementations

2021-12-13 Thread Vladimir Ozerov
Practically, Apache Calcite doesn’t have public API. Real projects usually override not only public extension points, but also internals, such as SqlValidatorImpl or VolcanoPlanner. Given the flexibility that Apache Calcite provides, this lack of public API surface is not necessarily a bad thing,

Re: [DISCUSS] Apache Calcite Online Meetup January 2022

2021-11-18 Thread Vladimir Ozerov
+ 1 >From our side (Querify Labs), we may present one of the difficult use cases we faced this year, such as large-scale join order planning, experience with Cascades integration, planning for heterogeneous systems, advanced statistics management, etc. We may need some time to figure out what can

Re: DISCUSS: merge calcite-avatica and calcite repositories

2021-11-09 Thread Vladimir Ozerov
+1 for a single repo. Вт, 9 нояб. 2021 г. в 19:22, Vladimir Sitnikov : > Michael> is not proposing to change the > Michael>structure of modules within both projects, merely to have the code > for > Michael>both in a single repository. > > I propose to integrate them into a single build, and keep

[jira] [Created] (CALCITE-4700) AggregateUnionTransposeRule produces wrong group sets for the top Aggregate

2021-07-21 Thread Vladimir Ozerov (Jira)
Vladimir Ozerov created CALCITE-4700: Summary: AggregateUnionTransposeRule produces wrong group sets for the top Aggregate Key: CALCITE-4700 URL: https://issues.apache.org/jira/browse/CALCITE-4700

Re: [ANNOUNCE] New committer: Vladimir Ozerov

2021-06-25 Thread Vladimir Ozerov
> Liya Fan > > On Fri, Jun 25, 2021 at 9:07 AM Julian Hyde > wrote: > > > Welcome, Vladimir! > > > > > On Jun 24, 2021, at 6:00 PM, Albert wrote: > > > > > > Congrats. > > > just found the blog: > https://www.querifylabs.com

Re: Trait propagation guidelines

2021-06-25 Thread Vladimir Ozerov
erator "passthrough" and "derive" strategy correctly. > > [1] > https://lists.apache.org/thread.html/r36b25cbe4ca05fb1262c432ad9103f4126b654698481fca0d2a01fe7%40%3Cdev.calcite.apache.org%3E > > Thanks, > Haisheng Yuan > > On 2021/06/14 08:26:31, Vladi

Re: Trait propagation guidelines

2021-06-14 Thread Vladimir Ozerov
don't need the involvement of "derive". > > Haisheng Yuan > > On 2021/06/13 16:58:53, Vladimir Ozerov wrote: > > Hi, > > > > I tried to apply different approaches, but eventually, I failed to > achieve > > my goals. It seems that the current impl

Re: Possibly incorrect assertion in the TopDownRuleDriver.DeriveTrait.derive

2021-06-13 Thread Vladimir Ozerov
; Regards, > Haisheng Yuan > > On 2021/06/06 13:09:16, Vladimir Ozerov wrote: > > Hi, > > > > When doing a trait derivation in the non-OMAKASE mode, the following > lines > > of code are invoked: > > 1: RelSubset relSubset = planner.register(newRel,

[jira] [Created] (CALCITE-4650) TopDownRuleDriver.DeriveTrait.derive produces wrong AssertionError when registering new nodes

2021-06-13 Thread Vladimir Ozerov (Jira)
Vladimir Ozerov created CALCITE-4650: Summary: TopDownRuleDriver.DeriveTrait.derive produces wrong AssertionError when registering new nodes Key: CALCITE-4650 URL: https://issues.apache.org/jira/browse

Re: Trait propagation guidelines

2021-06-13 Thread Vladimir Ozerov
o do this by disabling the pass-through cache completely and confirmed that the required plan is found. Do you have any objections to such a change? Regards, Vladimir. сб, 29 мая 2021 г. в 11:59, Vladimir Ozerov : > Hi Haisheng, Jinpeng > > I think we are more or less on the same pag

Re: Top-down optimizer cannot explore the search space because physical rule is treated as transformation rule

2021-06-13 Thread Vladimir Ozerov
Please disregard this email. I sent it to the wrong thread, sorry. вс, 13 июн. 2021 г. в 19:56, Vladimir Ozerov : > Hi, > > I tried to apply different approaches, but eventually, I failed to achieve > my goals. It seems that the current implementation cannot handle the > re

Re: Top-down optimizer cannot explore the search space because physical rule is treated as transformation rule

2021-06-13 Thread Vladimir Ozerov
logic to avoid risks can > overwrite the method by simply copying several lines of code. That's > totally acceptable. But if errors are issued, that's no longer a choice. > > In any case, if errors should be reported, we should provide an option to > suppress the errors. > >

Possibly incorrect assertion in the TopDownRuleDriver.DeriveTrait.derive

2021-06-06 Thread Vladimir Ozerov
Hi, When doing a trait derivation in the non-OMAKASE mode, the following lines of code are invoked: 1: RelSubset relSubset = planner.register(newRel, rel); 2: assert relSubset.set == planner.getSubset(rel).set; The assertion on the second line may fail because the "newRel" is assigned not the

Re: Top-down optimizer cannot explore the search space because physical rule is treated as transformation rule

2021-05-30 Thread Vladimir Ozerov
but to keep using old versions > of calcite. However, we usually can still get a good plan even without > correct rule types. I don't think it worthwhile to introduce that change. > > Thanks > Jinpeng Wu > > On Sun, May 30, 2021 at 5:19 AM Vladimir Ozerov > wro

Re: Top-down optimizer cannot explore the search space because physical rule is treated as transformation rule

2021-05-29 Thread Vladimir Ozerov
Yuan : > Great, that is the correct way to change it and that should be the default > implementation. > > On 2021/05/28 17:41:15, Vladimir Ozerov wrote: > > BTW, I tried to change the implementation to: > > > > 1: protected boolean isTransformationRule(VolcanoRuleC

Re: Trait propagation guidelines

2021-05-29 Thread Vladimir Ozerov
ve and may decide to abort this transformation > > early. But I would rather let the cost model decide. > > > > > > Admittedly, the current top-down optimization is not pure on-demand > > request oriented, because it will always generate a physical request > > rega

Re: Top-down optimizer cannot explore the search space because physical rule is treated as transformation rule

2021-05-28 Thread Vladimir Ozerov
class failed due to a minor operator reordering. пт, 28 мая 2021 г. в 20:37, Vladimir Ozerov : > Hi Jinpeng, > > Thank you for the clarification. When I saw the code in question for the > first time, my first thought was that it was perhaps designed for gradual > migration. T

Re: Top-down optimizer cannot explore the search space because physical rule is treated as transformation rule

2021-05-28 Thread Vladimir Ozerov
ve multiple parents and current framework cannot > handle such a situation) while it can "derive" input traits. > > Of course, this conclusion may not apply to your product as we could have > quite different rule sets. I am just sharing some of my experiences. Maybe >

Top-down optimizer cannot explore the search space because physical rule is treated as transformation rule

2021-05-28 Thread Vladimir Ozerov
Hi, I have an optimizer that uses top-down VolcanoPlanner and has a ConverterRule for every LogicalNode. I have a new requirement when one of the physical rules must emit several physical nodes instead of one. I tried to convert a ConverterRule to a normal rule that emits physical nodes. Even

Re: Trait propagation guidelines

2021-05-28 Thread Vladimir Ozerov
are basically > "templates" > > > that must be replaced with normal nodes. > > > > There is no difference between distributed and non-distributed engines > when dealing with this. In Orca and CockroachDB optimizer, the nodes > emitted from rules are opera

Re: Trait propagation guidelines

2021-05-27 Thread Vladimir Ozerov
y > a, > > or b, the derive() method will tell you there is an opportunity. This is > > the feature that Calcite's top-down optimizer excels over other > optimizers, > > because this can dramatically reduce the search space while keeping the > > optimal

Re: Trait propagation guidelines

2021-05-27 Thread Vladimir Ozerov
t; 3. This is by design. Nodes produced from "passThrough" and "derive" and > just sibling physical node with different traitset, we only need the > initial physical nodes after implementation to avoid unnecessary > operations. The fundamental reason is, unlike Orca optimizer w

Trait propagation guidelines

2021-05-26 Thread Vladimir Ozerov
Hi, I tried to optimize a certain combination of operators for the distributed engine and got stuck with the trait propagation in the top-down engine. I want to ask the community for advice on whether the problem is solvable with the current Apache Calcite implementation or not. Consider the

Re: AggregateUnionTransposeRule fails when some inputs have unique grouping key

2021-05-25 Thread Vladimir Ozerov
t; On May 21, 2021, at 2:33 AM, Vladimir Ozerov wrote: > > > > Hi, > > > > I created two issues for two distinct bugs in > AggregateUnionTransposeRule: > > > > 1. https://issues.apache.org/jira/browse/CALCITE-4616 - the problem > with > > a partial

Re: Exposing multiple values of a trait from the operator

2021-05-25 Thread Vladimir Ozerov
reenplum-db/gporca/blob/master/libgpopt/include/gpopt/base/CDistributionSpecHashed.h#L44 > > Regards, > Haisheng Yuan > > On 2021/05/25 15:37:32, Vladimir Ozerov wrote: > > Hi, > > > > Consider the distributed SQL engine that uses a distribution property to

Re: Exposing multiple values of a trait from the operator

2021-05-25 Thread Vladimir Ozerov
Hi Vladimir, Thank you for the link. It is very relevant to my problem. I see that in these discussions, there were several ideas and claims, such as that (1) we can get rid of "simplify" altogether, (2) composite traits are rare in practice, (3) composite traits are not designed well in the

Exposing multiple values of a trait from the operator

2021-05-25 Thread Vladimir Ozerov
Hi, Consider the distributed SQL engine that uses a distribution property to model exchanges. Consider the following physical tree. To do the distributed join, we co-locate tuples using the equijoin key. Now the Join operator has two equivalent distributions - [a1] and [b1]. It is critical to

Re: AggregateUnionTransposeRule fails when some inputs have unique grouping key

2021-05-21 Thread Vladimir Ozerov
. в 09:43, JiaTao Tao : > Sounds make sense > > Regards! > > Aron Tao > > > Vladimir Ozerov 于2021年5月19日周三 下午8:57写道: > > > Hi, > > > > The AggregateUnionTransposeRule attempts to push the Aggregate below the > > Union. > > > &

[jira] [Created] (CALCITE-4616) AggregateUnionTransposeRule causes row type mismatch when some inputs have unique grouping key

2021-05-21 Thread Vladimir Ozerov (Jira)
Vladimir Ozerov created CALCITE-4616: Summary: AggregateUnionTransposeRule causes row type mismatch when some inputs have unique grouping key Key: CALCITE-4616 URL: https://issues.apache.org/jira/browse

[jira] [Created] (CALCITE-4615) AggregateUnionTransposeRule may assign wrong aggregate function to input

2021-05-21 Thread Vladimir Ozerov (Jira)
Vladimir Ozerov created CALCITE-4615: Summary: AggregateUnionTransposeRule may assign wrong aggregate function to input Key: CALCITE-4615 URL: https://issues.apache.org/jira/browse/CALCITE-4615

AggregateUnionTransposeRule fails when some inputs have unique grouping key

2021-05-19 Thread Vladimir Ozerov
Hi, The AggregateUnionTransposeRule attempts to push the Aggregate below the Union. Before: Aggregate[group=$0, agg=SUM($1] Union[all] Input1 Input2 After: Aggregate[group=$0, agg=SUM($1] Union[all] Aggregate[group=$0, agg=SUM($1] Input1 Aggregate[group=$0, agg=SUM($1]

Re: Trait propagation in heterogeneous plans

2021-05-06 Thread Vladimir Ozerov
It may propagate the in-core distribution in theory, if the relevant code exists. Practically, there is no such code. For example, consider EnumerableProject: 1. EnumerableProjectRule.convert doesn't propagate input's distribution, thanks to EnumerableProject.create that uses

Re: Trait propagation in heterogeneous plans

2021-05-06 Thread Vladimir Ozerov
Hi, I'd like to stress out that I am not trying to argue about subjective concepts at all. Quite the opposite - I would like to agree or disagree on a set of objective facts and find the solution. Specifically, from what I saw in Calcite's codebase and real projects, I assert the following:

Re: Trait propagation in heterogeneous plans

2021-05-05 Thread Vladimir Ozerov
Hi Vladimir, Julian, I want to distinguish between two cases. Some projects may decide to use Calcite's distribution trait. To my knowledge, this is not a common pattern because it is not really integrated into Calcite. It is not destroyed/adjusted in rules and operators as needed, not

Re: Trait propagation in heterogeneous plans

2021-05-04 Thread Vladimir Ozerov
Hi Vladimir, I couldn't share the reproducer, as it is behind the NDA. But the problem is evident from the code. There are two distinct issues actually: 1. Propagation of unsupported traits in operators. EnumerableProject is not affected. Examples of the problem: EnumerableWindowRule,

Re: Trait propagation in heterogeneous plans

2021-05-04 Thread Vladimir Ozerov
ilarly > CustomToEnumerable only change convention, but retains collation and > distribution, I assume. > But in practice, it should be decided by the operator inventor and the > underlying physical > implementation. > > Hope that answers your question. Feel free to ask if you have more > ques

Re: Time zone management and DST in Avatica

2021-03-31 Thread Vladimir Ozerov
Hi Julian, Vladimir, Israel time zone It is not fixed. Formally, you may find different abbreviations, like IST and IDT. But in real systems, like Postgres, MS SQL, or Java, this is a single time zone with DST changes. The following query in Postgres will return 21:00, 21:00, 22:00 for 24, 25,

Re: Time zone management and DST in Avatica

2021-03-31 Thread Vladimir Ozerov
Hi, Thank you for your feedback. Let me provide the exact example of when the problem happened. Consider the date 2020 Oct 25 in Israel Standard Time [1]. There was DST end at this date. This date represents exactly 18560 days since 01 Jan 1970. When there was 2020 Oct 25 00:00 IST, it was 2020

Time zone management and DST in Avatica

2021-03-28 Thread Vladimir Ozerov
Hi, Avatica assumes that the underlying engine operates on relative time without time zone as opposed to global time (UTC). When you set a temporal value (e.g., a prepared statement parameter), Avatica adds the current offset to the passed time. When you read a temporal value, Avatica subtracts

Trait propagation in heterogeneous plans

2021-03-27 Thread Vladimir Ozerov
Hi, Apache Calcite supports heterogeneous optimization when nodes may have different conventions. The Enumerable rules propagate all traits from inputs. We have doubts whether this is correct or not. Consider the following initial plan, which was created by Apache Calcite after sql-to-rel

[jira] [Created] (CALCITE-4543) Interval literal looses a fractional second when it has scale greater than 3

2021-03-19 Thread Vladimir Ozerov (Jira)
Vladimir Ozerov created CALCITE-4543: Summary: Interval literal looses a fractional second when it has scale greater than 3 Key: CALCITE-4543 URL: https://issues.apache.org/jira/browse/CALCITE-4543

Re: [DISCUSS] Default disable the RexNode normalization(or operands reorder)

2021-03-13 Thread Vladimir Ozerov
or example, pulling up all filters before > attempting to match materialized views. > > Julian > > > On Mar 11, 2021, at 10:37 AM, Vladimir Ozerov > wrote: > > > > in our practice, we also had some problems with normalization. First, we > > observed problems with

Re: Make RelNode attribute order part of MEMO

2021-03-13 Thread Vladimir Ozerov
lenges. > > Julian > > On Sun, Mar 7, 2021 at 11:25 PM Vladimir Ozerov > wrote: > > > > Hi, > > > > Currently, the order of attributes is used to define operators > equivalence. > > This leads to several interesting problems, such as possible duplicat

[jira] [Created] (CALCITE-4535) ServerDdlExecutor cannot execute DROP commands with qualified object names

2021-03-13 Thread Vladimir Ozerov (Jira)
Vladimir Ozerov created CALCITE-4535: Summary: ServerDdlExecutor cannot execute DROP commands with qualified object names Key: CALCITE-4535 URL: https://issues.apache.org/jira/browse/CALCITE-4535

Re: [DISCUSS] Default disable the RexNode normalization(or operands reorder)

2021-03-11 Thread Vladimir Ozerov
in our practice, we also had some problems with normalization. First, we observed problems with the unwanted (and sometimes incorrect) simplification of expressions with CASTs and literals which came from RexSimplify. I couldn't find an easy way to disable that behavior. Note, that RexSimplify may

[jira] [Created] (CALCITE-4533) Fix handling of REPLACE and IF NOT EXISTS keywords for CREATE TABLE/SCHEMA commands

2021-03-11 Thread Vladimir Ozerov (Jira)
Vladimir Ozerov created CALCITE-4533: Summary: Fix handling of REPLACE and IF NOT EXISTS keywords for CREATE TABLE/SCHEMA commands Key: CALCITE-4533 URL: https://issues.apache.org/jira/browse/CALCITE-4533

Re: Question about parallel query planning

2021-03-09 Thread Vladimir Ozerov
*at such = at such scale Вт, 9 марта 2021 г. в 11:01, Vladimir Ozerov : > Hi Jihoon, > > I would say that 5 sec could be actually a pretty good result at such. Did > you have a chance to check which exact rules contributed to the planning > time? You may inject a listener to

Re: Question about parallel query planning

2021-03-09 Thread Vladimir Ozerov
Hi Jihoon, I would say that 5 sec could be actually a pretty good result at such. Did you have a chance to check which exact rules contributed to the planning time? You may inject a listener to VolcanoPlanner to check that. Regards, Vladimir Вт, 9 марта 2021 г. в 05:37, Jihoon Son : > Hi all,

Make RelNode attribute order part of MEMO

2021-03-07 Thread Vladimir Ozerov
Hi, Currently, the order of attributes is used to define operators equivalence. This leads to several interesting problems, such as possible duplication of expressions (e.g., "a,a,b") or additional work in rules to detect trivial projects and/or input permutations (e.g. "a,b -> b,a"). But the

Non-additive costs in heterogeneous engines

2021-02-26 Thread Vladimir Ozerov
Hi, Several products that utilize Apache Calcite for query optimization might use multiple execution units to execute physical operators concurrently. Think of a heterogeneous engine that might split execution between a CPU and a co-processor (GPU, FGPA, etc), or just a multi-core machine in the

Re: Add cross-join suppression to JoinAssociateRule

2021-02-26 Thread Vladimir Ozerov
is applied, and joins contain only those conditions, that couldn't be pushed down. [1] https://github.com/apache/calcite/pull/2359 сб, 20 февр. 2021 г. в 20:34, Vladimir Ozerov : > Thank you for the feedback. I'll try to prototype it for all the affected > rules. > > Regards, > Vladimi

[jira] [Created] (CALCITE-4515) Do not generate the new join tree from commute/associate rules if it contains "always true" condition

2021-02-26 Thread Vladimir Ozerov (Jira)
Vladimir Ozerov created CALCITE-4515: Summary: Do not generate the new join tree from commute/associate rules if it contains "always true" condition Key: CALCITE-4515 URL: https://issues.apach

Re: Add cross-join suppression to JoinAssociateRule

2021-02-20 Thread Vladimir Ozerov
ive to the users the option to use it or not. > > Best, > Stamatis > > On Sun, Feb 14, 2021 at 9:58 AM Vladimir Ozerov > wrote: > > > Hi Julian, > > > > First of all, I want to clarify that if we decide to do the cross-join > > suppression, we would have to do that for

[jira] [Created] (CALCITE-4501) Clear costs in VolcanoPlanner when setting the new root

2021-02-16 Thread Vladimir Ozerov (Jira)
Vladimir Ozerov created CALCITE-4501: Summary: Clear costs in VolcanoPlanner when setting the new root Key: CALCITE-4501 URL: https://issues.apache.org/jira/browse/CALCITE-4501 Project: Calcite

Re: Add cross-join suppression to JoinAssociateRule

2021-02-14 Thread Vladimir Ozerov
> > Julian > > > On Feb 12, 2021, at 23:04, Vladimir Ozerov wrote: > > > > Hi, > > > > Join order planning is an important optimization problem. The widely-used > > heuristic is to consider all bushy trees without cross-joins. There is > > proof [1] t

Add cross-join suppression to JoinAssociateRule

2021-02-12 Thread Vladimir Ozerov
Hi, Join order planning is an important optimization problem. The widely-used heuristic is to consider all bushy trees without cross-joins. There is proof [1] that a pair of commute and associate rules is a complete ruleset to explore all bushy trees without the cross-joins if the initial join

[jira] [Created] (CALCITE-4476) DateTimeUtils.timeStringToUnixDate may produce wrong time

2021-01-23 Thread Vladimir Ozerov (Jira)
Vladimir Ozerov created CALCITE-4476: Summary: DateTimeUtils.timeStringToUnixDate may produce wrong time Key: CALCITE-4476 URL: https://issues.apache.org/jira/browse/CALCITE-4476 Project: Calcite

[jira] [Created] (CALCITE-4466) Do not RelTraitDef.convert if the "from" trait already satisfies the "to" trait.

2021-01-13 Thread Vladimir Ozerov (Jira)
Vladimir Ozerov created CALCITE-4466: Summary: Do not RelTraitDef.convert if the "from" trait already satisfies the "to" trait. Key: CALCITE-4466 URL: https://issues.apache.org/jira

Re: [DISCUSS] Apache Calcite Online Meetup January 2021

2021-01-12 Thread Vladimir Ozerov
*Talk duration*: 30 min. ср, 13 янв. 2021 г. в 10:27, Vladimir Ozerov : > Hi Stamatis, Julian, > > Thank you for releasing the slot. Just to be crystal clear - I was very > late with my proposal, and have no urge to present in January. Both January > and April are perfectly f

Re: [DISCUSS] Apache Calcite Online Meetup January 2021

2021-01-12 Thread Vladimir Ozerov
. In any case, I confirm that I can present in January. Please find the talk details below. *Speaker:* Vladimir Ozerov *Title:* Apache Calcite integration in Hazelcast IMDG *Abstract: * Hazelcast IMDG is a distributed in-memory key-value store. In this talk, I will present how we used Apache Calcite

Re: [DISCUSS] Apache Calcite Online Meetup January 2021

2021-01-12 Thread Vladimir Ozerov
> > Best, > Stamatis > > On Mon, Jan 11, 2021 at 8:35 PM Vladimir Ozerov > wrote: > > > Hi, > > > > I can share our experience with Apache Calcite integration into > Hazelcast > > distributed SQL engine. > > > > Regards, > > Vl

Re: [DISCUSS] Apache Calcite Online Meetup January 2021

2021-01-11 Thread Vladimir Ozerov
Hi, I can share our experience with Apache Calcite integration into Hazelcast distributed SQL engine. Regards, Vladimir Вт, 5 янв. 2021 г. в 00:48, Vineet G : > Hi Stamatis, > > Something has come up and unfortunately I will not be able to present the > talk. > > Vineet > > > On Jan 3, 2021,

[jira] [Created] (CALCITE-4461) Do not cast to logical node inside Enumerable rules

2021-01-11 Thread Vladimir Ozerov (Jira)
Vladimir Ozerov created CALCITE-4461: Summary: Do not cast to logical node inside Enumerable rules Key: CALCITE-4461 URL: https://issues.apache.org/jira/browse/CALCITE-4461 Project: Calcite

Re: Contributor rights

2021-01-11 Thread Vladimir Ozerov
Thank you very much, Francis! вс, 10 янв. 2021 г. в 00:45, Francis Chuang : > Hey Vladimir, > > I've added you to the contributor role in jira. > > Francis > > On 9/01/2021 8:47 pm, Vladimir Ozerov wrote: > > Hi, > > > > Could you please grant me contribu

Contributor rights

2021-01-09 Thread Vladimir Ozerov
Hi, Could you please grant me contributor rights in Calcite JIRA? My username is "vozerov". Thank you. Vladimir.

Re: Need a Jason output if physical tree

2020-12-21 Thread Vladimir Ozerov
Hi Liya, This will not work AFAIK. Consider the query "SELECT a FROM t WHERE b>1". The top-level operator has only one column: Root[$0] Project[$0] Filter[$1>1] Scan[$0=a, $1=b] If you invoke RelMdColumnOrigins on Root[$0], you will get [t.a], but miss [t.b]. To my knowledge, rules

Re: Need a Jason output if physical tree

2020-12-18 Thread Vladimir Ozerov
Hi Liya, I do not think, this is a global decision. If you go bottom-up in the plan, and find the Project operator, than none parent operators will use any columns from the lower scans, other than those present in the Project. It is a common practice for Calcite integrations to restrict the

Re: Decouple core from linq4j and Avatica

2020-11-27 Thread Vladimir Ozerov
; Moving Enumerable out of core may be time consuming and disruptive, > > > > because many core tests are using Enumerable to verify plan quality > and > > > > correctness. > > > > > > > > Best, > > > > Haisheng > > > > > >

Decouple core from linq4j and Avatica

2020-11-24 Thread Vladimir Ozerov
Hi colleagues, Many Calcite integrations use only part of the framework. Specifically, it is common to use only the parser/optimizer part. JDBC and runtime are used less frequently because they are not very well suited for mature processing engines (e.g. Enumerable runs out of memory easily).

RelFieldTrimmer misses optimization opportunity for always false condition

2020-08-11 Thread Vladimir Ozerov
Hi colleagues, Consider the following query. If the RelFieldTrimmer is applied to this, then the expression is reduced to an empty LogicalValues: SELECT field FROM table WHERE TRUE IS FALSE However, the following query will not be simplified, leaving a table scan with "always false" filter:

How to support case-insensitive identifier resolution?

2020-06-16 Thread Vladimir Ozerov
Hi colleagues, I am trying to implement case insensitive resolution of column/table/schema names for our Apache Calcite integration in Hazelcast and got stuck. I hope that the community might help me. Consider that we have a table source that contains the following Java class: class Employee {

Re: Using indexes rather than table scans with Calcite

2020-06-02 Thread Vladimir Ozerov
gt; > >> Let's consider an example > > > >> > > > >> SELECT * FROM A JOIN B ON A.a=B.b WHERE A.c > 100 > > > >> > > > >> where A.a, A.c and B.b are indexed fields. The logical plan for this > > > >> query migh

Re: Using indexes rather than table scans with Calcite

2020-05-31 Thread Vladimir Ozerov
be more efficient > plan in terms of the cost. > > My question is: how rule-based approach handle this scenario? Will it > re-apply IndexScanRule once again to produce PhysicalIndexScan(A.a) and > PhysicalIndexScan(B.b)? Or am I missing the crucial point of a rule-base > appro

  1   2   >