Equivalence of two RexNode in SubstitutionVisitor::splitFilter

[Vineet G]

Hi,

I am running into an issue where a materialize view rewriting is not occurring 
and the issue is narrowed down to usage of splitFilter.

The problem is in figuring out if two RexNode are equivalent or not. For simple 
expressions like (a > 1 AND b < 3) & (b < 3 AND a > 1) it works but for 
expressions like (a +b > 1) & (b + a > 1) splitFilter is unable to figure out 
that these two expressions are equivalent and as a result it ends up returning 
null. 
The reason is that equivalence is determined based on the string representation 
of the expressions, since the later two expressions have different string 
representation they are deemed inequivalent although they are semantically 
equivalent. 

Is there better way to figure out semantic equivalence of two rex node in 
calcite which may be take commutativity and associativity of operands into 
account? 

Thanks,
Vineet Garg

Re: Equivalence of two RexNode in SubstitutionVisitor::splitFilter

[Vineet G]

I believe I found the answer to my question. SubstitutionVisitor already 
provides a way to achieve this using canonicalization. It needs to be improved 
to consider RexNode of type PLUS.
I have created CALCITE-3194 to add this improvement.

Thanks,
Vineet

> On Apr 10, 2020, at 12:00 PM, Vineet G  wrote:
> 
> Hi,
> 
> I am running into an issue where a materialize view rewriting is not 
> occurring and the issue is narrowed down to usage of splitFilter.
> 
> The problem is in figuring out if two RexNode are equivalent or not. For 
> simple expressions like (a > 1 AND b < 3) & (b < 3 AND a > 1) it works but 
> for expressions like (a +b > 1) & (b + a > 1) splitFilter is unable to figure 
> out that these two expressions are equivalent and as a result it ends up 
> returning null. 
> The reason is that equivalence is determined based on the string 
> representation of the expressions, since the later two expressions have 
> different string representation they are deemed inequivalent although they 
> are semantically equivalent. 
> 
> Is there better way to figure out semantic equivalence of two rex node in 
> calcite which may be take commutativity and associativity of operands into 
> account? 
> 
> Thanks,
> Vineet Garg

Re: Understanding annotations of SqlGroupingFunction

[Vineet G]

I expect that the user behavior for the GROUPING in both hive and calcite is 
same. It’s just the documentation which is a bit confusing.
e.g.  comment line on grouping : if both deptno and gender are being grouped

should really mean that the row which represents the grand total i.e  without 
group by expression. will return 3.

FYI Hive had its behavior fixed with 
https://issues.apache.org/jira/browse/HIVE-16102 
 and currently uses GROUPING 
ID function.

Vineet Garg


> On Apr 21, 2020, at 10:42 AM, Julian Hyde  wrote:
> 
> Suppose we have one row that represents the total for department 10, and 
> another that represents the grand total of all departments. Which row would 
> we say that department is “grouped” (in Calcite’s parlance) or “aggregated” 
> in (Hive’s parlance)?
> 
> I find the terms confusing. It’s possible that Calcite has them “wrong”. 
> 
> I would simply run a query like
> 
>  SELECT deptno, job, GROUPING(deptno, job), COUNT(*)
>  FROM emp
>  GROUP BY CUBE (deptno, job)
> 
> and see whether Hive and Calcite return the same result. 
> 
>> On Apr 20, 2020, at 6:58 PM, ZZY  wrote:
>> 
>> Hi, Hyde:
>> It's confused me that some annotations in
>> Calcite(org.apache.calcite.sql.fun.SqlGroupingFunction.java) :
>> /**
>> * The {@code GROUPING} function.
>> *
>> * Accepts 1 or more arguments.
>> * Example: {@code GROUPING(deptno, gender)} returns
>> * 3 if both deptno and gender are being grouped,
>> * 2 if only deptno is being grouped,
>> * 1 if only gender is being groped,
>> * 0 if neither deptno nor gender are being grouped.
>> *
>> * This function is defined in the SQL standard.
>> * {@code GROUPING_ID} is a non-standard synonym.
>> *
>> * Some examples are in {@code agg.iq}.
>> */
>> 
>> The annotations above seems conflicts with other implementations like Hive(
>> https://cwiki.apache.org/confluence/display/Hive/Enhanced+Aggregation%2C+Cube%2C+Grouping+and+Rollup?spm=ata.13261165.0.0.528c6dfcXalQFy#EnhancedAggregation,Cube,GroupingandRollup-Groupingfunction
>> )
>> 
>> Notice that: "The grouping function indicates whether an expression in a
>> GROUP BY clause is aggregated or not for a given row. The value 0
>> represents a column that is part of the grouping set, while the value 1
>> represents a column that is not part of the grouping set. "
>> 
>> 
>> It is clearly that 0 and 1 bit have different interpretation  between
>> annotations in Calcite and in Hive. And I did not figure out why...
>> 
>> Any feedback can give me on this would be highly appreciated.
>> 
>> Best regards!

Re: [ANNOUNCE] New committer: Vineet Garg

[Vineet G]

Thanks a lot guys!

Just to briefly introduce myself - I work with Cloudera (Hortonworks before) on 
Hive and I am a Hive PMC member. As Stamatis noted I have been involved in 
calcite since 2017. It is great honor to be part of this community. I am very 
excited to become committer and I look forward to contributing more.

Regards,
Vineet Garg

> On Apr 26, 2020, at 2:26 PM, Jesus Camacho Rodriguez  
> wrote:
> 
> Congrats Vineet, well deserved!
> 
> -Jesús
> 
> On Sun, Apr 26, 2020 at 3:09 AM Leonard Xu  wrote:
> 
>> Congratulations, Vineet!
>> 
>> Best,
>> Leonard Xu
>>> 在 2020年4月26日，18:07，xu  写道：
>>> 
>>> Congrats, Vineet!
>>> 
>>> Danny Chan  于2020年4月26日周日 下午4:52写道：
>>> 
 Congrats, Vineet!
 
 Best,
 Danny Chan
 在 2020年4月26日 +0800 PM1:55，dev@calcite.apache.org，写道：
> 
> Congrats, Vineet!
 
>>> 
>>> 
>>> --
>>> 
>>> Best regards,
>>> 
>>> Xu
>> 
>> 

Re: [ANNOUNCE] New committer: Wang Yanlin

[Vineet G]

Congratulations!

> On May 1, 2020, at 12:11 PM, Julian Hyde  wrote:
> 
> Yanlin,
> 
> Please create a PR adding an Ant Financial paragraph to that page. in
> the PR, include the URL of the image that you would like me to add to
> the logo diagram.
> 
> Julian
> 
> On Fri, May 1, 2020 at 8:02 AM Wang Yanlin <1989yanlinw...@163.com> wrote:
>> 
>> 
>> 
>> 
>> Thanks Julian, having an entry on the powered by page for Ant Financial 
>> would be nice.
>> 
>> 
>> 
>> [1] is the official homepage of the company I'm serving.
>> 
>> 
>> Yanlin
>> 
>> 
>> 
>> 
>> [1]  https://www.antfin.com/  
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> --
>> 
>> Best,
>> Wang Yanlin
>> 
>> 
>> 
>> 
>> 
>> At 2020-05-01 01:54:38, "Julian Hyde"  wrote:
>>> Welcome, Yanlin!
>>> 
>>> Thanks for your contributions so far, and thanks for introducing yourself. 
>>> I often learn so much from committers’ self-introductions about how Calcite 
>>> is being used.
>>> 
>>> I know we have other Alibaba-related projects on the “powered by” page [1] 
>>> (Flink/Ververica, MaxCompute) but it seems that Ant Financial is a distinct 
>>> business, so deserves its own entry on the page, and logo. What do you 
>>> think?
>>> 
>>> Julian
>>> 
>>> [1] https://calcite.apache.org/docs/powered_by.html#alibaba-maxcompute 
>>> 
>>> 
 On Apr 29, 2020, at 5:50 AM, Wang Yanlin <1989yanlinw...@163.com> wrote:
 
 Hi, guys, thanks for your warm welcome.
 
 
 
 I'm working in Ant Finical, Alibaba  Group. Currently my team is working 
 on building a system to process big data in form of sql.
 We use calcite to parse sql, optimize Relnode and rewrite SqlNode to 
 execute on different engines, like Spark，MaxCompute, HBase and so on.
 Calcite is really a great community, and it's really an honor for me to 
 become calcite committer, hops to make more contribution to calcite.
 
 
 Thanks again.
 
 --
 
 Best,
 Wang Yanlin
 
 
 
 
 
 在 2020-04-29 13:58:35，"Zoltan Haindrich"  写道：
> Congratulations!
> 
> On 4/29/20 7:32 AM, Enrico Olivelli wrote:
>> Congrats!
>> 
>> Enrico
>> 
>> Il Mer 29 Apr 2020, 04:51 Feng Zhu  ha scritto:
>> 
>>> Congrations! Yanlin!
>>> 
>>> best,
>>> Feng
>>> 
>>> Chunwei Lei  于2020年4月29日周三 上午10:16写道：
>>> 
 Congrats, Yanlin!
 
 
 Best,
 Chunwei
 
 
 On Wed, Apr 29, 2020 at 10:07 AM Forward Xu 
 wrote:
 
> Congrats
> 
> 
> Best,
> 
> Forward
> 
> 953396112 <953396...@qq.com> 于2020年4月29日周三 上午8:26写道：
> 
>> Congrats, Wang Yanlin!
>> 
>> 
>> 
>> 
>> ---Original---
>> From: "Stamatis Zampetakis"> Date: Wed, Apr 29, 2020 05:51 AM
>> To: "dev"> Subject: [ANNOUNCE] New committer: Wang Yanlin
>> 
>> 
>> Apache Calcite's Project Management Committee (PMC) has invited Wang
> Yanlin
>> to
>> become a committer, and we are pleased to announce that he has
 accepted.
>> 
>> Wang has pushed numerous fixes and improvements to the project,
>>> landing
> in
>> total
>> the impressive number of 30 commits to the master. Among other
>>> things,
 he
>> contributed some important features in the Interpreter.
>> 
>> Wang, welcome, thank you for your contributions, and we look forward
 your
>> further interactions with the community! If you wish, please feel
>>> free
 to
>> tell
>> us more about yourself and what you are working on.
>> 
>> Stamatis (on behalf of the Apache Calcite PMC)
> 
 
>>> 
>> 
>>> 

Re: [ANNOUNCE] New committer: Jin Xing

[Vineet G]

Congratulations!

> On Apr 30, 2020, at 10:48 AM, Julian Hyde  wrote:
> 
> Welcome Jin! Thanks for your contributions so far, looking forward to more!
> 
> Are you on the MaxCompute project? It’s already on our “powered by” page[1], 
> so I think people are familiar with it.
> 
> Julian
> 
> [1] https://calcite.apache.org/docs/powered_by.html#alibaba-maxcompute 
> 
> 
> 
>> On Apr 29, 2020, at 5:06 AM, XING JIN  wrote:
>> 
>> Thanks a lot ~
>> Calcite is a great project and it's great honor for me to work with you
>> guys. I really appreciate the help from community.
>> I'm working in Alibaba. My team builds big data system to optimize batch
>> and streaming jobs. We use Calcite to process Sql queries and accommodate
>> to different physical engines.
>> I'm very excited to become Calcite committer and looking forward to make
>> more contributions.
>> 
>> Best regards,
>> Jin
>> 
>> 
>> Zoltan Haindrich  于2020年4月29日周三 下午1:58写道：
>> 
>>> Congrats!
>>> 
>>> On 4/29/20 7:32 AM, Enrico Olivelli wrote:
 Congratulations!
 
 Enrico
 
 Il Mer 29 Apr 2020, 04:51 Feng Zhu  ha scritto:
 
> Congrations!
> 
> best,
> Feng
> 
> Chunwei Lei  于2020年4月29日周三 上午10:16写道：
> 
>> Congrats, Jin!
>> 
>> 
>> Best,
>> Chunwei
>> 
>> 
>> On Wed, Apr 29, 2020 at 10:07 AM Forward Xu 
>> wrote:
>> 
>>> Congrats
>>> 
>>> 
>>> best,
>>> 
>>> Forward
>>> 
>>> 953396112 <953396...@qq.com> 于2020年4月29日周三 上午8:21写道：
>>> 
 Congrats, Jin Xing!
 
 
 ---Original---
 From: "Stamatis Zampetakis">>> Date: Wed, Apr 29, 2020 05:47 AM
 To: "dev">>> Subject: [ANNOUNCE] New committer: Jin Xing
 
 
 Apache Calcite's Project Management Committee (PMC) has invited Jin
>> Xing
>>> to
 become a committer, and we are pleased to announce that he has
>> accepted.
 
 Jin has contributed a lot of code in the project and many
 recent improvements in
 materialized view matching have his signature on them. Apart from
> code
 contributions, Jin provides valuable help to the community by doing
>>> reviews
 and
 answering questions in the devlist.
 
 Jin, welcome, thank you for your contributions, and we look forward
> to
>>> your
 further interactions with the community! If you wish, please feel
> free
>> to
 tell
 us more about yourself and what you are working on.
 
 Stamatis (on behalf of the Apache Calcite PMC)
>>> 
>> 
> 
 
>>> 
> 

Re: [ANNOUNCE] New committer: Forward Xu

[Vineet G]

Congratulations!

> On Apr 30, 2020, at 10:57 AM, Julian Hyde  wrote:
> 
> Congratulations and welcome, Forward! Thank you for your contributions.
> 
> It would be great to add TBDS (and its logo) to the “powered by” page[1]. 
> What do you think?
> 
> Julian
> 
> [1] https://calcite.apache.org/docs/powered_by.html 
> 
> 
>> On Apr 29, 2020, at 5:34 AM, Forward Xu  wrote:
>> 
>> Thank you everyone for your warm welcome!
>> I'm working in the TBDS team of Tencent in Shenzhen. TBDS (Tencent Big Data
>> Suite) is similar to Alibaba's EMR, TBDS is a big data ecosystem. I am
>> responsible for Oceanus(flink streaming jobs) and Tdbank (Tencent real-time
>> data collection system). I‘m very happy to become calcite committer and
>> looking forward to make more contributions.
>> 
>> Best,
>> Forward
>> 
>> Zoltan Haindrich  于2020年4月29日周三 下午1:58写道：
>> 
>>> Congratulations!
>>> 
>>> On 4/29/20 7:31 AM, Enrico Olivelli wrote:
 Congrats!
 
 Enrico
 
 Il Mer 29 Apr 2020, 04:52 Feng Zhu  ha scritto:
 
> Congrations! Forward!
> 
> best,
> Feng
> 
> Chunwei Lei  于2020年4月29日周三 上午10:17写道：
> 
>> Congrats, Forward!
>> 
>> 
>> 
>> Best,
>> Chunwei
>> 
>> 
>> On Wed, Apr 29, 2020 at 6:46 AM Rui Wang  wrote:
>> 
>>> Congrats!
>>> 
>>> 
>>> -Rui
>>> 
>>> On Tue, Apr 28, 2020 at 3:04 PM Francis Chuang <
> francischu...@apache.org
>>> 
>>> wrote:
>>> 
 Congrats, Forward!
 
 Francis
 
 On 29/04/2020 7:53 am, Stamatis Zampetakis wrote:
> Apache Calcite's Project Management Committee (PMC) has invited
>> Forward
 Xu
> to
> become a committer, and we are pleased to announce that he has
>>> accepted.
> 
> Forward has been helping the project for some time now. He added
> many
>>> new
> SQL
> functions to the project and is one of our JSON experts. On top of
>>> that,
 and
> other fixes, he is the one who added the Redis adapter to the
>> project.
> 
> Forward, welcome, thank you for your contributions, and we look
>> forward
 to
> your
> further interactions with the community! If you wish, please feel
>> free
>>> to
> tell
> us more about yourself and what you are working on.
> 
> Stamatis (on behalf of the Apache Calcite PMC)
> 
 
>>> 
>> 
> 
 
>>> 
> 

Re: [DISCUSS] Apache Calcite Online Meetup January 2021

[Vineet G]

Hi Stamatis,

Something has come up and unfortunately I will not be able to present the talk.

Vineet

> On Jan 3, 2021, at 1:37 PM, Stamatis Zampetakis  wrote:
> 
> I updated the agenda on meetup to include Julian's talk around spatial
> queries.
> 
> So far we have four presentations lasting approximately 1h45 so I still
> find the duration reasonable.
> 
> Of course if there are more people interested to present something we can
> schedule another meetup in April as Julian suggested.
> I am always happy to see what other people are working on and more Calcite
> use-cases.
> 
> Best,
> Stamatis
> 
> On Sun, Jan 3, 2021 at 2:09 AM Julian Hyde  wrote:
> 
>> In other news I’ll be co-presenting (with Mosha Pasumansky) a talk
>> “Open source SQL - beyond parsers: ZetaSQL and Apache Calcite” at the
>> Northwest Database Society Annual Meeting on January 29th. It’s virtual and
>> free, but you must sign up to attend.
>> 
>> Julian
>> 
>> [1] https://sites.google.com/view/nwds2021
>> 
>>> On Jan 2, 2021, at 12:47 PM, Julian Hyde  wrote:
>>> 
>>> I can give a talk “Implementing spatial queries using algebra
>> rewrites”, 20 minutes.
>>> 
>>> But if that makes the meetup too long, I am equally happy to postpone
>> the talk. How about scheduling another meetup  in say April?
>>> 
>>> Julian
>>> 
 On Dec 31, 2020, at 3:10 AM, Stamatis Zampetakis 
>> wrote:
 
 I just published the event on Meetup [1].
 
 The agenda is not yet finalized so if there are people who would like to
 give a talk or add/remove things from the agenda please reply to this
 thread.
 
 Best,
 Stamatis
 
 [1] https://www.meetup.com/Apache-Calcite/events/275461117/
 
>> On Mon, Nov 30, 2020 at 12:37 AM Rui Wang 
>> wrote:
> 
> Title: event timestamp semantic based streaming SQL
> Abstract: this talk will cover in Calcite Streaming SQL case, how to
>> reason
> data completeness in terms of event timestamp semantic and how to
>> control
> materialization latency given unbounded input data (in Calcite roadmap
>> but
> not implemented yet).
> 
> Duration: 20~30 mins
> 
>> On Tue, Nov 24, 2020 at 8:56 AM Slim Bouguerra 
>> wrote:
>> 
>> this is a great idea thanks @Statmatis looking forward to learning
>> more
>> about Calcite especially the Streaming work.
>> 
>>> On Mon, Nov 23, 2020 at 2:19 PM Rui Wang 
>> wrote:
>> 
>>> Sorry for the late reply Statmatis. I have recently been pretty busy
>> on
>>> work as it is approaching the end of the year.
>>> 
>>> The time in [1] works perfectly for me. I will share the abstract and
>>> expected duration soon (should within this week).
>>> 
>>> 
>>> -Rui
>>> 
>>> On Fri, Nov 20, 2020 at 2:11 AM Stamatis Zampetakis <
>> zabe...@gmail.com
>> 
>>> wrote:
>>> 
 That would be great Vineet!
 
 @Julian, @Rui, @Vineet:
 Can you share a small abstract (2-3 sentences) and expected
>> duration?
 Can you check if the date/times proposed previously [1] work for
>> you.
>> If
 not feel free to propose another slot.
 
 Best,
 Stamatis
 
 [1] https://s.apache.org/uhrzo
 
 On Thu, Nov 19, 2020 at 6:18 PM Vineet Garg 
> wrote:
 
> I think this is a great idea. +1 for the online meetup.
> 
> If there are slots left I can also talk about how Hive leverages
>>> Calcite
 to
> do query optimization.
> 
> -Vineet
> 
> On Fri, Nov 6, 2020 at 7:21 AM Stamatis Zampetakis <
>> zabe...@gmail.com>
> wrote:
> 
>> Let's try to fix the date/time and tentative agenda so that we
> can
>>> add
> some
>> information on meetup [1].
>> 
>> So far we have three presenters, Julian, Rui, and myself. We can
>>> start
> like
>> that and if in the process there are more people interested to
>> give a
> small
>> talk we can update the program.
>> 
>> Let's try to get a date in the last two weeks of January to give
>> us a
 bit
>> more time to prepare. Personally, I don't have a preference for
>> that
> being
>> a business day or not and I am in UTC+1.
>> For instance, how do you feel about Wednesday, 20 January 2021,
 18:00:00
> to
>> 21:00 UTC+1 [2] ?
>> 
>> As soon as we have a rough estimate on the number of attendees we
>> can
> also
>> decide which platform to use for the event.
>> 
>> Even if the number of participants ends up being low, having new
>> material for our website is still a plus.
>> 
>> Best,
>> Stamatis
>> 
>> [1] https://www.meetup.com/Apache-Calcite/
>> [2] https://s.a

Re: [ANNOUNCE] New committers: Zhiwei Peng

[Vineet G]

Congratulations Zhiwei!

On Sat, Apr 27, 2019 at 11:19 AM Kevin Risden  wrote:

> Congrats and welcome!
>
> Kevin Risden
>
>
> On Sat, Apr 27, 2019 at 11:44 AM Stamatis Zampetakis 
> wrote:
>
> > Congratulations Zhiwei!
> >
> > With so many high quality contributions there is nothing left to say.
> Well
> > deserved!
> >
> > On Sat, Apr 27, 2019 at 6:14 AM Chunwei Lei 
> > wrote:
> >
> > > Congratulations, Zhiwei!
> > >
> > >
> > >
> > > Best,
> > > Chunwei
> > >
> > > On Sat, Apr 27, 2019 at 12:13 PM Yuzhao Chen 
> > wrote:
> > > >
> > > > Congratulations, Zhiwei!
> > > >
> > > > Best,
> > > > Danny Chan
> > > > 在 2019年4月27日 +0800 AM10:37，Francis Chuang  > >，写道：
> > > > > Apache Calcite's Project Management Committee (PMC) has invited
> > Zhiwei
> > > > > Peng to become a committer, and we are pleased to announce that he
> > has
> > > > > accepted.
> > > > >
> > > > > Zhiwei has been contributing to Calcite for a while, racking up an
> > > > > impressive 20 pull requests, in particular, doing a lot of work to
> > > > > improve RexSimplify.
> > > > >
> > > > > Zhiwei, welcome, thank you for your contributions, and we look
> > forward
> > > > > your further interactions with the community! If you wish, please
> > feel
> > > > > free to tell us more about yourself and what you are working on.
> > > > >
> > > > > Francis (on behalf of the Apache Calcite PMC)
> > >
> >
>