RelDecorrelator.java - Values Handler

[Mahendrakar Srinivasarao]

Hi Community,

Hope you are doing well.
I have a question about the handling of Values Rel.

File: RelDecorrelator.java

public Frame decorrelateRel(Values rel, boolean isCorVarDefined) {
   return null;
}

Why do we return null for Values Rel. Can you please let me know.


While looking at the join/union handlers (in the same file), if one of
the input frame is null, handler returns null.

JOIN

Input1 Input2


In the above scenario, let's say input1 is decorrelated successfully
but Input2 has Values Rel, in which case we return null as per the
logic in Join (below).

public Frame decorrelateRel(Join rel, boolean isCorVarDefined) {

.

.

.

final RelNode oldLeft = rel.getInput(0);
final RelNode oldRight = rel.getInput(1);

final Frame leftFrame = getInvoke(oldLeft, isCorVarDefined, rel);
final Frame rightFrame = getInvoke(oldRight, isCorVarDefined, rel);

if (leftFrame == null || rightFrame == null) {
  // If any input has not been rewritten, do not rewrite this rel.
  return null;

.

.


.
}




Thanks,
Mahendrakar.

Re: DISCUSS: merge calcite-avatica and calcite repositories

[Jacques Nadeau]

I'm finding it difficult to make sense of all the responses on this thread.
I put up a quick straw poll that will hopefully try to give us a better
sense of where people's heads are at. I think the thread outlines a number
of concerns for both outcomes. I'd ask people to read the thread and then
share their feedback on the survey.

Survey: https://forms.gle/hUZaEP3aqcbe31HC9
Results:
https://docs.google.com/spreadsheets/d/1EXhrRkYp2owGAK2e3bBw1bEvNj5O3V22iTqndBAuyG8/edit?usp=sharing

Hopefully this will cut through some of the noise.



On Wed, Nov 10, 2021 at 1:35 PM Vladimir Sitnikov <
sitnikov.vladi...@gmail.com> wrote:

> s/John/Josh/
>
> sorry for the typo.
>
> Vladimir
>

[jira] [Created] (CALCITE-4884) Provide a new constructor for RelJsonWriter to allow customized JsonBuilder

[Chunwei Lei (Jira)]

Chunwei Lei created CALCITE-4884:


 Summary: Provide a new constructor for RelJsonWriter to allow 
customized JsonBuilder
 Key: CALCITE-4884
 URL: https://issues.apache.org/jira/browse/CALCITE-4884
 Project: Calcite
  Issue Type: Improvement
Reporter: Chunwei Lei


Currently JsonBuilder in RelJsonWriter cannot be customized due to it being 
created in the constructor.  We can provide a new constructor for RelJsonWriter 
to allow customized JsonBuilder.
{code:java}
// code placeholder
public RelJsonWriter() {
  jsonBuilder = new JsonBuilder();
  relList = jsonBuilder.list();
  relJson = new RelJson(jsonBuilder);
}

// the new constrctor suggested.
public RelJsonWriter(JsonBuilder jsonBuilder) {
  this.jsonBuilder = jsonBuilder;
  relList = jsonBuilder.list();
  relJson = new RelJson(jsonBuilder);
}
{code}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: Calcite for macro parsing in SQL ?

[Julian Hyde]

No worries.

Thanks for following up and sharing what worked for you. It will help other 
people who are facing similar problems in future. I’m pleased that you found an 
example that you could copy!

Julian


> On Nov 10, 2021, at 7:03 AM, Cyril Catheu  wrote:
> 
> Also sorry Julian Hyde I forgot to subscribe to the mailing list when I
> sent my question so I did not see your answer 🤦‍♂️.
> I'll have a look at your suggestions.
> 
> On Wed, Nov 10, 2021 at 3:45 PM Cyril Catheu  wrote:
> 
>> Had a deeper look, I can answer my own questions now.
>> 
>> 1. Adding a list of SqlFunction to a parser?
>> No need. The SqlParser will parse unknown symbols that look like function
>> calls as SqlCall nodes with SqlKind "OTHER_FUNCTION"  and
>> SqlFunctionCategory "USER_DEFINED_FUNCTION".
>> 
>> Note: I was actually getting confused by how functions identifiers are
>> escaped in the toString method.
>> For instance: select EXP(col), myMacro(col) from ... will be parsed then
>> returned as String as EXP(`col`), `myMacro`(`col`) from ...
>> Not sure to understand why the EXP is not escaped, but I was able to play
>> with the toSqlString method to get what I want.
>> 
>> 2. Parse a SQL query, traverse the tree recursively, replace some nodes,
>> then get back a Sql string ?
>> Yes, it's a good use case for Calcite. A good inspiration was the Hoist
>> 
>> class. It replaces constants in q SQL string.
>> I implemented something similar that replaces my "USER_DEFINED_FUNCTION"
>> macros.
>> 
>> It works like a charm.
>> Have a nice day.
>> 
>> On Wed, Nov 3, 2021 at 9:53 PM Cyril Catheu  wrote:
>> 
>>> Hey,
>>> 
>>> I'm working on a product that executes arbitrary SQL on a "data source".
>>> A data source implements a SQL language, for instance, it can be MySQL,
>>> Pinot, BigQuery, etc...
>>> I'd like to introduce a macro language on top of the SQL.
>>> Very similar to what is done in grafana:
>>> https://grafana.com/docs/grafana/latest/datasources/mysql/#macros
>>> These macros would be resolved before the SQL is executed on the data
>>> source.
>>> 
>>> A macro looks like a function but is a string replacement, for instance:
>>> __timeFilter(timeColumn, start, end) -->  timeColumn>=start and
>>> timeColumn>> 
>>> Grafana uses regex replacements, but I'm considering parsing the SQL with
>>> Calcite to apply the macros safely and manage nested macros/functions.
>>> 
>>> --> Is this a good use case for Calcite? What seemed interesting to me is
>>> that Calcite has all those SqlDialect implementations.
>>> 
>>> I had this flow in mind:
>>> 1. get the SqlDialect of my datasource
>>> *2.* add to this dialect my custom macros (add to an SqlFunction list? )
>>> 3. parse
>>> 4. replace
>>> 5. rebuild
>>> 
>>> I'm wondering if 2. is possible? I'm not sure to understand how I could
>>> add a list of SqlFunction to a dialect, or build a new dialect from an
>>> existing one dynamically.
>>> 
>>> If you've taken the time to read this message, thanks a lot!
>>> Have a nice day.
>>> 
>>> --
>>> [image: Startree] 
>>> 
>>> Cyril de Catheu
>>> 
>>> Software Engineer, StarTree
>>> 
>>> cy...@startree.ai | +33 684-829-908
>>> 
>>> [image: Linkedin] [image:
>>> Twitter] 
>>> 
>> 

Re: DISCUSS: merge calcite-avatica and calcite repositories

[Vladimir Sitnikov]

s/John/Josh/

sorry for the typo.

Vladimir

Re: DISCUSS: merge calcite-avatica and calcite repositories

[Vladimir Sitnikov]

Josh>because I genuinely do not see how this approach
Josh>creates more code that needs to be written than if the projects were in
Josh>the same repository.

Compare
https://github.com/apache/calcite-avatica/blob/4cf769f8ee32bb3520604e4f3284e6f233f90276/build.gradle.kts
and
https://github.com/apache/calcite/blob/6d51d277b158ff7073f29ada4a96a7a74c0b46fc/build.gradle.kts

Those are duplicate files, and I had to create them twice.
If the code was co-located in a single repository, then a single copy of
the file would be enough.

John>(I am to assume) is a personal motivation for yourself.

Everybody who contributes to the build scripts, dependencies, etc, would
want to avoid duplication of their work.

John>Avatica 1.17.0 was released Jun 2020, 1.18.0 in May 2021. Calcite had 3
John>releases of its own between these.

In other words, Avatica could have three releases **for free**
Of course, you might argue there's "non-trivial procedure for releasing
Calcite",
however, I would claim it does not have to be that way.

"lots of manual steps for the release" add zero to the quality of the
release.
Of course, we might disagree here as well, however, if releasing Calcite is
painful,
then "forking Avatica into a different repository" is a wrong solution.

John>I can still build Avatica with Gradle 6.8.1.

Go ahead and try building Avatica with Java 17.
You will fail quite fast. Have a fun day.

Vladimir

Re: DISCUSS: merge calcite-avatica and calcite repositories

[Josh Elser]

On 11/10/21 3:48 PM, Vladimir Sitnikov wrote:

Josh>If we identify these problems, we can think
Josh>through (from the beginning) if combining repositories is the only
Josh>solution or just one possible solution (and weigh the pros and cons for
Josh>each).

I am sure I did explain the pain points. See points 1..5 (the very
beginning of the mail) in the very first message in this thread.
If that is not enough, I can't really explain better.

CI configuration is duplicated across repositories.
JUnit versions maintenance overheads are duplicated.
Build configuration is duplicated (e.g. code style rules, boilerplate build
scripts, etc)
Cross-repository PRs are non-trivial.
And so on.


These are not things that are broken. CI configuration, JUnit versions, 
and build configuration are already defined for each project. If you 
want to change them, you can submit a change and test them. It sounds 
like you want to be able to change them in exactly one place which (I am 
to assume) is a personal motivation for yourself.


Cross-repository PRs, I've already addressed that this is a solved 
problem. Please illustrate an example how this creates more code that 
you have to write, because I genuinely do not see how this approach 
creates more code that needs to be written than if the projects were in 
the same repository.


1. Update avatica
2. Release avatica
3. Upgrade avatica dependency in calcite with corresponding changes due 
to avatica upgrade.


In your example where avatica is in the code base, it discourages API 
semantics to *not* be compatible by allowing you to quickly change the 
Avatica API and immediately reflect those changes in Calcite. If folks 
start doing this, then you disrupt the entire Calcite developer base who 
has to realize that the API they wanted to use is no longer the correct API.


> If you have suggestions on how to bump JUnit and CI configuration in 
both repositories at once


You are correct in that I do not believe this to be a "problem".

> It is clear that having the repositories separate has no real benefits.

This is not a fact, but this is your opinion.

> committers are the same

The PMC is the same and those authorized to make commits to each are the 
same group, but the individuals are *not*. For example, I know damn well 
that I should not go merging changes that I make into Calcite because I 
do not regularly do this. In the same way, I'd expect 90%+ of the 
Calcite committers have never contributed to Avatica.


If your opinion is some philosophical issue with the PMC governance 
being shared between Avatica and Calcite, we can have that discussion 
instead.


> I am sure those "reasons" have no weight

Again, this is your opinion. The fact is that I have made Avatica 
releases without the burden of making a Calcite release, and I know that 
multiple Calcite releases have been made without the burden of making a 
new Avatica release. This is not "theoretical".


Avatica 1.17.0 was released Jun 2020, 1.18.0 in May 2021. Calcite had 3 
releases of its own between these.


> If so, I just let them do the maintenance.

That is what I would prefer you to do.

> Avatica is stuck with Gradle 6.8.1

I can still build Avatica with Gradle 6.8.1.

Let me be clear to remove all doubt at this point in time: I am -1 on 
combining these repositories based on the current discussion.

Re: DISCUSS: merge calcite-avatica and calcite repositories

[Julian Hyde]

My mistake. I originally wrote “If we want to keep our linear commit history 
(i.e. avoid a huge merge commit), all of the commit hashes will have to 
change”. I believe that is a true statement. I shouldn’t have tried to simplify 
it.

> On Nov 10, 2021, at 3:46 AM, Dmitry Sysolyatin  
> wrote:
> 
> Actually, it is possible to preserve the history of commits when you merge
> two repos -
> https://stackoverflow.com/questions/17371150/moving-git-repository-content-to-another-repository-preserving-history
> 
> On Tue, Nov 9, 2021 at 11:24 PM Julian Hyde  wrote:
> 
>> Merging the repositories is unnecessary, and a massive waste of time.
>> 
>> Any supposed “time savings” due to the the merged repositories will be
>> eaten up by dozens of small issues that will crop up long after the merge.
>> 
>> I was release manager for Calcite and Avatica this time around and it took
>> me about a week, because the process had changed since last time I was
>> release manager. And now we’re going to change the process again?
>> Absolutely fucking insane.
>> 
>> Also, we will lose the history of the Avatica source code. All of the
>> commit hashes for bugs fixed long ago will change.
>> 
>> At the time that we split the repositories, I was probably against the
>> idea, because it was work. Now that we’ve split, I am strongly against
>> recombining the repositories.
>> 
>> Honestly, people, we have better things to do.
>> 
>> Julian
>> 
>> 
>>> On Nov 9, 2021, at 1:03 PM, Stamatis Zampetakis 
>> wrote:
>>> 
>>> The two projects have more or less the same release cadence, they're
>>> maintained by the same community, they're written in the same language,
>> and
>>> they use the same build system.
>>> 
>>> Although I understand the reasons the project was split in the first
>> place,
>>> at the moment I tend to agree that having them in the same repo would be
>>> more practical.
>>> 
>>> Truth is the merge may require quite a bit of work so we should take this
>>> into consideration.
>>> 
>>> If people prefer keeping the projects as they are I am perfectly fine
>> with
>>> that as well. I am leaving the decision to those who feel strongly about
>>> this change.
>>> 
>>> Best,
>>> Stamatis
>>> 
>>> 
>>> 
>>> On Tue, Nov 9, 2021, 7:46 PM Vladimir Ozerov  wrote:
>>> 
 +1 for a single repo.
 
 Вт, 9 нояб. 2021 г. в 19:22, Vladimir Sitnikov <
 sitnikov.vladi...@gmail.com
> :
 
> Michael> is not proposing to change the
> Michael>structure of modules within both projects, merely to have the
 code
> for
> Michael>both in a single repository.
> 
> I propose to integrate them into a single build, and keep the set of
>> the
> published jars.
> However, the modules and dependency structure could be kept as is.
> 
> We might want to rename folders like
> calcite-core
> calcite-linq4j
> ..
> avatica-core
> avatica-server
> 
> However, I am not sure it is that important to discuss folder names
>> now.
> The idea is that as you "open Calcite in IDE, you see both Avatica and
> Calcite modules"
> 
> Michael>Is there any reason we couldn't have a separate release
>> schedule
 if
> Michael>both projects are in the same repository?
> 
> A different schedule means Calcite must support at least two different
> Avatica versions.
> In other words, if we allow clients to update Avatica at their will,
>> then
> we should allow them building Calcite with different Avatica versions,
> which implies Calcite test code should succeed for multiple different
> Avatica vesions.
> 
> That makes it harder to write tests: we have to execute tests with two
> different Avatica releases (or even more than two).
> 
> There are at least two sources for complexity:
> 
> a) We have to write tests that tolerate multiple versions. For
>> instance,
> "if (avatica.18+) {...}" and so on.
> That is not really trivial, especially taking into account some of the
> tests are created with non-yet-popular
> technologies like Quidem where you can't really google solutions. So
>> the
> "trivial" task of "making a test to expect two possible outcomes"
> becomes less trivial as you try to pass the version from GitHub Action
>> to
> Gradle to JUnit to Quidem to no-one-knows-which class.
> If we support one Avatica version only, that is not needed. We just
>> patch
> the test in Avatica and Calcite and that is it.
> Single repo avoids "Gradle vs Quidem" dance.
> 
> b) If we claim that we support 5 different Guava versions, 3 different
 JDK
> versions, 2 different Avatica versions,
> then we have to execute 5*3*2 = 30 combinations of the tests.
> That is not really a full matrix, however, things get way easier if we
> support one Avatica version only.
> The amount of tests we need to do during a proper release is much less,
 and
> it is e

Re: DISCUSS: merge calcite-avatica and calcite repositories

[Josh Elser]

On 11/10/21 3:48 PM, Vladimir Sitnikov wrote:

Josh>I have no objections to combining these two repositories together.

Why don't we just merge the repositories and move on then?
This was in reference to merging calcite-avatica and calcite-avatica-go 
together. I thought this would be apparent given the rest of my 
arguments *against* merging them. Sorry I was not in more clear 
disagreement.

Re: DISCUSS: merge calcite-avatica and calcite repositories

[Vladimir Sitnikov]

Josh>I have no objections to combining these two repositories together.

Why don't we just merge the repositories and move on then?

Josh>If we identify these problems, we can think
Josh>through (from the beginning) if combining repositories is the only
Josh>solution or just one possible solution (and weigh the pros and cons for
Josh>each).

I am sure I did explain the pain points. See points 1..5 (the very
beginning of the mail) in the very first message in this thread.
If that is not enough, I can't really explain better.

CI configuration is duplicated across repositories.
JUnit versions maintenance overheads are duplicated.
Build configuration is duplicated (e.g. code style rules, boilerplate build
scripts, etc)
Cross-repository PRs are non-trivial.
And so on.

If you have suggestions on how to bump JUnit and CI configuration in both
repositories at once,
you are welcome, however, the only alternative "solution" I expect is to
ignore the problem
and treat it as "not a problem".

I acknowledge merging repositories will make some use cases harder.
However, I am sure points 1..5 (see the very first mail) have no solution
besides repository merge.

It is clear that having the repositories separate has no real benefits.
Community is the same, committers are the same, language is the same,
releases are co-located.

There are theoretical reasons like "different repositories allow different
committers in theory" or
"different repositories allow different release schedules" or "because
people perceive that they can use component A without using component B"
However, I am sure those "reasons" have no weight. They are theoretical,
and some of them turned out to be false.
For instance, the release schedule is co-located across several years.
Even at the time of the Avatica split, somebody said that Solr community
did **merge** repositories after a while
exactly because the committers were effectively the same.

Of course, somebody might **believe** that "because people perceive that
they can use component A without using component B"
is way more important than a non-important duplication of maintenance of
CI, dependencies, build scripts across both repositories.

You know what? I am fine to accept people might have those beliefs.
If so, I just let them do the maintenance.

Of course, you might claim that your ability to watch "avatica PRs only" is
far more important.
Is that really the key point of having separate repositories? Really?
GitHub has "code owners" and "labeling" features, so you can watch the
relevant parts of the repository.

Josh>These repositories are separate because they have separate release
Josh>schedules

The release dates for Calcite and Avatica are co-located (see my mail
above).
The feature of "separate release schedules" is not really used in practice.

Having the same version number for Calcite and Avatica would event make it
easier for clients to pick the compatible versions.

Josh> The net amount of code you have to write doesn't
Josh> change with how it works now compared to how you are suggesting it
Josh> should work.

This is false. For instance, "CI configuration, JUnit versions, build
scripts" are **duplicated** across repositories,
so if the repositories are different, then the amount of work is increased.

Here's one more data point: Calcite is built with Gradle 7.3, and Avatica
is stuck with Gradle 6.8.1
Of course, the duplication of work does not exist.

You say "the net amount of code I have to write doesn't change".
You are right. I won't contribute to calcite-avatica repository anymore, so
there's no duplication of work for me.

Josh>If there are breaking changes in Avatica, they would need to be
Josh>accounted for when Calcite is bumped to the next version of Avatica.

calcite-avatica/pull/161 would be way easier if both avatica and calcite
were in a single repository.
There would be no need to think of version compatibility, etc.

Josh> Having a separate repository makes it
Josh>extremely easy for me to watch Avatica commits/pull-requests which I am
Josh>capable of reviewing vs. Calcite pull-requests which I am not
Josh>comfortable reviewing.

Have you considered mail filters and/or code owners
https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners
?
There's an option to automatically label PRs according to the modified
files via GitHub Action.

Josh>rather than starting from the root problem "How can we
Josh>make co-dependent changes easier?"

"co-dependent" changes are not the only problem.
It was just a trigger.

Vladimir

Re: DISCUSS: merge calcite-avatica and calcite repositories

[Josh Elser]

Are you still advocating this change, Vladimir?

If this is something you want to continue discussing (I'm not sure if 
your comment about "[I won't contribute to Avatica]" is saying that you 
don't want to discuss this anymore), could we try to refocus this on 
some specific technical problems that come up due to Avatica and Calcite 
being separate repositories. If we identify these problems, we can think 
through (from the beginning) if combining repositories is the only 
solution or just one possible solution (and weigh the pros and cons for 
each).


It seems like you also have some pain points with Calcite that you want 
to improve, but these are separate from the discussion at hand. Let's 
try to keep those out of a discussion about combining these two 
repositories, please.


On 11/10/21 6:59 AM, Vladimir Sitnikov wrote:

Ok, I did not care when Avatica was split into its own repository, and I
assumed Avatica team would maintain the code.
That turned out to be not quite true, and it turns out there are
non-trivial overheads coming from the split.

Of course, everybody can disagree, so if you disagree, you don't need to
explain that. We just disagree and that is it.

Now I do not care how Avatica would evolve further.
It might be even good because Vladimir S won't bother Avatica with mails,
PRs, and other random items.


What I care much more about is that Calcite tests are broken, the CI
reports failure for every commit, and that is really insane.
Calcite CI must not fail on every build.

I am going to delete/ignore the offending tests in Calcite in 24 hours, and
I let the others deal with the case (fix tests, fix Quidem, merge Avatica,
or whatever).

I would no longer review calcite-avatica PRs.



Julian>I was release manager for Calcite and Avatica this time around and
it took me about a week

Julian, as the repositories are merged, there will be a **single** release
process that releases both Calcite and Avatica in a single go.
It would reduce the friction, and it would reduce cases where the release
procedures and steps are "slightly different".

It looks like you assume the release steps would never change, however, in
practice, the drift of Java, Docker, OS, and so on versions would
affect the release steps anyway.

That is why having Avatica and Calcite in a single repository would
simplify the release procedure.

Julian> And now we’re going to change the process again?

I propose to *drop* the release process for "calcite-avatica-java". The
release of "calcite" would release both things (in a single
calcite-x.y.z.tar.gz file),
so the release steps for Calcite would be intact.

Technically, that changes the process.
In practice, I find that you overreact. The merge would make the release
easier.

On top of that, it was you who voted for splitting Calcite and Avatica, so
it was your conscious decision to accept that "release management would get
harder after the split"

Julian>Any supposed “time savings” due to the merged repositories will be
eaten up by dozens of small issues that will crop up long after the merge

The words are not justified. There will always be some random small issues,
even in the case we keep repositories separate.
On the other hand, there are true issues when repositories are different:
* We have two different release processes, so RM have to know the
differences
* we have slightly different dependency versions, so things like "bump
junit" have to be duplicated
* CI configuration is duplicated across the repositories, and it does have
non-zero maintenance overhead
* cross-repository refactorings are harder to do (like the fix for the
exception message)

Julian, I did put four issues that are real, and they induce noticeable
maintenance churn. Java and JUnit will keep updating and you can't prevent
that.
Please highlight the problems exactly and please stop saying "time savings
will be eaten by dozens".
That is not a healthy discussion.


Also, we will lose the history of the Avatica source code. All of the

commit hashes for bugs fixed long ago will change

We can make calcite-avatica repository read-only, so all commits will be
there.

Vladimir

Re: Calcite for macro parsing in SQL ?

[Cyril Catheu]

Also sorry Julian Hyde I forgot to subscribe to the mailing list when I
sent my question so I did not see your answer 🤦‍♂️.
I'll have a look at your suggestions.

On Wed, Nov 10, 2021 at 3:45 PM Cyril Catheu  wrote:

> Had a deeper look, I can answer my own questions now.
>
> 1. Adding a list of SqlFunction to a parser?
> No need. The SqlParser will parse unknown symbols that look like function
> calls as SqlCall nodes with SqlKind "OTHER_FUNCTION"  and
> SqlFunctionCategory "USER_DEFINED_FUNCTION".
>
> Note: I was actually getting confused by how functions identifiers are
> escaped in the toString method.
> For instance: select EXP(col), myMacro(col) from ... will be parsed then
> returned as String as EXP(`col`), `myMacro`(`col`) from ...
> Not sure to understand why the EXP is not escaped, but I was able to play
> with the toSqlString method to get what I want.
>
> 2. Parse a SQL query, traverse the tree recursively, replace some nodes,
> then get back a Sql string ?
> Yes, it's a good use case for Calcite. A good inspiration was the Hoist
> 
> class. It replaces constants in q SQL string.
> I implemented something similar that replaces my "USER_DEFINED_FUNCTION"
> macros.
>
> It works like a charm.
> Have a nice day.
>
> On Wed, Nov 3, 2021 at 9:53 PM Cyril Catheu  wrote:
>
>> Hey,
>>
>> I'm working on a product that executes arbitrary SQL on a "data source".
>> A data source implements a SQL language, for instance, it can be MySQL,
>> Pinot, BigQuery, etc...
>> I'd like to introduce a macro language on top of the SQL.
>> Very similar to what is done in grafana:
>> https://grafana.com/docs/grafana/latest/datasources/mysql/#macros
>> These macros would be resolved before the SQL is executed on the data
>> source.
>>
>> A macro looks like a function but is a string replacement, for instance:
>>  __timeFilter(timeColumn, start, end) -->  timeColumn>=start and
>> timeColumn>
>> Grafana uses regex replacements, but I'm considering parsing the SQL with
>> Calcite to apply the macros safely and manage nested macros/functions.
>>
>> --> Is this a good use case for Calcite? What seemed interesting to me is
>> that Calcite has all those SqlDialect implementations.
>>
>> I had this flow in mind:
>> 1. get the SqlDialect of my datasource
>> *2.* add to this dialect my custom macros (add to an SqlFunction list? )
>> 3. parse
>> 4. replace
>> 5. rebuild
>>
>> I'm wondering if 2. is possible? I'm not sure to understand how I could
>> add a list of SqlFunction to a dialect, or build a new dialect from an
>> existing one dynamically.
>>
>> If you've taken the time to read this message, thanks a lot!
>> Have a nice day.
>>
>> --
>> [image: Startree] 
>>
>> Cyril de Catheu
>>
>> Software Engineer, StarTree
>>
>> cy...@startree.ai | +33 684-829-908
>>
>> [image: Linkedin] [image:
>> Twitter] 
>>
>

Re: Calcite for macro parsing in SQL ?

[Cyril Catheu]

Had a deeper look, I can answer my own questions now.

1. Adding a list of SqlFunction to a parser?
No need. The SqlParser will parse unknown symbols that look like function
calls as SqlCall nodes with SqlKind "OTHER_FUNCTION"  and
SqlFunctionCategory "USER_DEFINED_FUNCTION".

Note: I was actually getting confused by how functions identifiers are
escaped in the toString method.
For instance: select EXP(col), myMacro(col) from ... will be parsed then
returned as String as EXP(`col`), `myMacro`(`col`) from ...
Not sure to understand why the EXP is not escaped, but I was able to play
with the toSqlString method to get what I want.

2. Parse a SQL query, traverse the tree recursively, replace some nodes,
then get back a Sql string ?
Yes, it's a good use case for Calcite. A good inspiration was the Hoist

class. It replaces constants in q SQL string.
I implemented something similar that replaces my "USER_DEFINED_FUNCTION"
macros.

It works like a charm.
Have a nice day.

On Wed, Nov 3, 2021 at 9:53 PM Cyril Catheu  wrote:

> Hey,
>
> I'm working on a product that executes arbitrary SQL on a "data source".
> A data source implements a SQL language, for instance, it can be MySQL,
> Pinot, BigQuery, etc...
> I'd like to introduce a macro language on top of the SQL.
> Very similar to what is done in grafana:
> https://grafana.com/docs/grafana/latest/datasources/mysql/#macros
> These macros would be resolved before the SQL is executed on the data
> source.
>
> A macro looks like a function but is a string replacement, for instance:
>  __timeFilter(timeColumn, start, end) -->  timeColumn>=start and
> timeColumn
> Grafana uses regex replacements, but I'm considering parsing the SQL with
> Calcite to apply the macros safely and manage nested macros/functions.
>
> --> Is this a good use case for Calcite? What seemed interesting to me is
> that Calcite has all those SqlDialect implementations.
>
> I had this flow in mind:
> 1. get the SqlDialect of my datasource
> *2.* add to this dialect my custom macros (add to an SqlFunction list? )
> 3. parse
> 4. replace
> 5. rebuild
>
> I'm wondering if 2. is possible? I'm not sure to understand how I could
> add a list of SqlFunction to a dialect, or build a new dialect from an
> existing one dynamically.
>
> If you've taken the time to read this message, thanks a lot!
> Have a nice day.
>
> --
> [image: Startree] 
>
> Cyril de Catheu
>
> Software Engineer, StarTree
>
> cy...@startree.ai | +33 684-829-908
>
> [image: Linkedin] [image:
> Twitter] 
>

Re: DISCUSS: merge calcite-avatica and calcite repositories

[Michael Mior]

My mistake. I was unaware of the --allow-unrelated-histories flag :)
--
Michael Mior
mm...@apache.org

Le mer. 10 nov. 2021 à 07:34, Dmitry Sysolyatin
 a écrit :
>
> No. hashes shouldn't change.
> You can try the following steps and check:
>
> git pull https://github.com/apache/calcite.git
> cd calcite
> git remote add avatica https://github.com/apache/calcite-avatica
> git fetch avatica
> git merge avatica/master --allow-unrelated-histories
> git add . && git commit -m "test"
>
> If you check the history log you can see that hash for the commit
> https://github.com/apache/calcite-avatica/commit/4cf769f8ee32bb3520604e4f3284e6f233f90276
> remains the same
>
> On Wed, Nov 10, 2021 at 2:20 PM Michael Mior  wrote:
>
> > > Actually, it is possible to preserve the history of commits when you
> > merge
> > > two repos -
> >
> > You can keep the history, but the hashes will necessarily change
> > because they are based on a different parent. So it will be possible
> > to see the changes, but the hashes would reference the old Avatica
> > repository which would no longer be active.
> >
> > --
> > Michael Mior
> > mm...@apache.org
> >
> > Le mer. 10 nov. 2021 à 06:47, Dmitry Sysolyatin
> >  a écrit :
> > >
> > > Actually, it is possible to preserve the history of commits when you
> > merge
> > > two repos -
> > >
> > https://stackoverflow.com/questions/17371150/moving-git-repository-content-to-another-repository-preserving-history
> > >
> > > On Tue, Nov 9, 2021 at 11:24 PM Julian Hyde 
> > wrote:
> > >
> > > > Merging the repositories is unnecessary, and a massive waste of time.
> > > >
> > > > Any supposed “time savings” due to the the merged repositories will be
> > > > eaten up by dozens of small issues that will crop up long after the
> > merge.
> > > >
> > > > I was release manager for Calcite and Avatica this time around and it
> > took
> > > > me about a week, because the process had changed since last time I was
> > > > release manager. And now we’re going to change the process again?
> > > > Absolutely fucking insane.
> > > >
> > > > Also, we will lose the history of the Avatica source code. All of the
> > > > commit hashes for bugs fixed long ago will change.
> > > >
> > > > At the time that we split the repositories, I was probably against the
> > > > idea, because it was work. Now that we’ve split, I am strongly against
> > > > recombining the repositories.
> > > >
> > > > Honestly, people, we have better things to do.
> > > >
> > > > Julian
> > > >
> > > >
> > > > > On Nov 9, 2021, at 1:03 PM, Stamatis Zampetakis 
> > > > wrote:
> > > > >
> > > > > The two projects have more or less the same release cadence, they're
> > > > > maintained by the same community, they're written in the same
> > language,
> > > > and
> > > > > they use the same build system.
> > > > >
> > > > > Although I understand the reasons the project was split in the first
> > > > place,
> > > > > at the moment I tend to agree that having them in the same repo
> > would be
> > > > > more practical.
> > > > >
> > > > > Truth is the merge may require quite a bit of work so we should take
> > this
> > > > > into consideration.
> > > > >
> > > > > If people prefer keeping the projects as they are I am perfectly fine
> > > > with
> > > > > that as well. I am leaving the decision to those who feel strongly
> > about
> > > > > this change.
> > > > >
> > > > > Best,
> > > > > Stamatis
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Nov 9, 2021, 7:46 PM Vladimir Ozerov 
> > wrote:
> > > > >
> > > > >> +1 for a single repo.
> > > > >>
> > > > >> Вт, 9 нояб. 2021 г. в 19:22, Vladimir Sitnikov <
> > > > >> sitnikov.vladi...@gmail.com
> > > > >>> :
> > > > >>
> > > > >>> Michael> is not proposing to change the
> > > > >>> Michael>structure of modules within both projects, merely to have
> > the
> > > > >> code
> > > > >>> for
> > > > >>> Michael>both in a single repository.
> > > > >>>
> > > > >>> I propose to integrate them into a single build, and keep the set
> > of
> > > > the
> > > > >>> published jars.
> > > > >>> However, the modules and dependency structure could be kept as is.
> > > > >>>
> > > > >>> We might want to rename folders like
> > > > >>> calcite-core
> > > > >>> calcite-linq4j
> > > > >>> ..
> > > > >>> avatica-core
> > > > >>> avatica-server
> > > > >>>
> > > > >>> However, I am not sure it is that important to discuss folder names
> > > > now.
> > > > >>> The idea is that as you "open Calcite in IDE, you see both Avatica
> > and
> > > > >>> Calcite modules"
> > > > >>>
> > > > >>> Michael>Is there any reason we couldn't have a separate release
> > > > schedule
> > > > >> if
> > > > >>> Michael>both projects are in the same repository?
> > > > >>>
> > > > >>> A different schedule means Calcite must support at least two
> > different
> > > > >>> Avatica versions.
> > > > >>> In other words, if we allow clients to update Avatica at their
> > will,
> > > > then
> > > > >>> we should allow them building Calcite with different Avat

Re: DISCUSS: merge calcite-avatica and calcite repositories

[Dmitry Sysolyatin]

No. hashes shouldn't change.
You can try the following steps and check:

git pull https://github.com/apache/calcite.git
cd calcite
git remote add avatica https://github.com/apache/calcite-avatica
git fetch avatica
git merge avatica/master --allow-unrelated-histories
git add . && git commit -m "test"

If you check the history log you can see that hash for the commit
https://github.com/apache/calcite-avatica/commit/4cf769f8ee32bb3520604e4f3284e6f233f90276
remains the same

On Wed, Nov 10, 2021 at 2:20 PM Michael Mior  wrote:

> > Actually, it is possible to preserve the history of commits when you
> merge
> > two repos -
>
> You can keep the history, but the hashes will necessarily change
> because they are based on a different parent. So it will be possible
> to see the changes, but the hashes would reference the old Avatica
> repository which would no longer be active.
>
> --
> Michael Mior
> mm...@apache.org
>
> Le mer. 10 nov. 2021 à 06:47, Dmitry Sysolyatin
>  a écrit :
> >
> > Actually, it is possible to preserve the history of commits when you
> merge
> > two repos -
> >
> https://stackoverflow.com/questions/17371150/moving-git-repository-content-to-another-repository-preserving-history
> >
> > On Tue, Nov 9, 2021 at 11:24 PM Julian Hyde 
> wrote:
> >
> > > Merging the repositories is unnecessary, and a massive waste of time.
> > >
> > > Any supposed “time savings” due to the the merged repositories will be
> > > eaten up by dozens of small issues that will crop up long after the
> merge.
> > >
> > > I was release manager for Calcite and Avatica this time around and it
> took
> > > me about a week, because the process had changed since last time I was
> > > release manager. And now we’re going to change the process again?
> > > Absolutely fucking insane.
> > >
> > > Also, we will lose the history of the Avatica source code. All of the
> > > commit hashes for bugs fixed long ago will change.
> > >
> > > At the time that we split the repositories, I was probably against the
> > > idea, because it was work. Now that we’ve split, I am strongly against
> > > recombining the repositories.
> > >
> > > Honestly, people, we have better things to do.
> > >
> > > Julian
> > >
> > >
> > > > On Nov 9, 2021, at 1:03 PM, Stamatis Zampetakis 
> > > wrote:
> > > >
> > > > The two projects have more or less the same release cadence, they're
> > > > maintained by the same community, they're written in the same
> language,
> > > and
> > > > they use the same build system.
> > > >
> > > > Although I understand the reasons the project was split in the first
> > > place,
> > > > at the moment I tend to agree that having them in the same repo
> would be
> > > > more practical.
> > > >
> > > > Truth is the merge may require quite a bit of work so we should take
> this
> > > > into consideration.
> > > >
> > > > If people prefer keeping the projects as they are I am perfectly fine
> > > with
> > > > that as well. I am leaving the decision to those who feel strongly
> about
> > > > this change.
> > > >
> > > > Best,
> > > > Stamatis
> > > >
> > > >
> > > >
> > > > On Tue, Nov 9, 2021, 7:46 PM Vladimir Ozerov 
> wrote:
> > > >
> > > >> +1 for a single repo.
> > > >>
> > > >> Вт, 9 нояб. 2021 г. в 19:22, Vladimir Sitnikov <
> > > >> sitnikov.vladi...@gmail.com
> > > >>> :
> > > >>
> > > >>> Michael> is not proposing to change the
> > > >>> Michael>structure of modules within both projects, merely to have
> the
> > > >> code
> > > >>> for
> > > >>> Michael>both in a single repository.
> > > >>>
> > > >>> I propose to integrate them into a single build, and keep the set
> of
> > > the
> > > >>> published jars.
> > > >>> However, the modules and dependency structure could be kept as is.
> > > >>>
> > > >>> We might want to rename folders like
> > > >>> calcite-core
> > > >>> calcite-linq4j
> > > >>> ..
> > > >>> avatica-core
> > > >>> avatica-server
> > > >>>
> > > >>> However, I am not sure it is that important to discuss folder names
> > > now.
> > > >>> The idea is that as you "open Calcite in IDE, you see both Avatica
> and
> > > >>> Calcite modules"
> > > >>>
> > > >>> Michael>Is there any reason we couldn't have a separate release
> > > schedule
> > > >> if
> > > >>> Michael>both projects are in the same repository?
> > > >>>
> > > >>> A different schedule means Calcite must support at least two
> different
> > > >>> Avatica versions.
> > > >>> In other words, if we allow clients to update Avatica at their
> will,
> > > then
> > > >>> we should allow them building Calcite with different Avatica
> versions,
> > > >>> which implies Calcite test code should succeed for multiple
> different
> > > >>> Avatica vesions.
> > > >>>
> > > >>> That makes it harder to write tests: we have to execute tests with
> two
> > > >>> different Avatica releases (or even more than two).
> > > >>>
> > > >>> There are at least two sources for complexity:
> > > >>>
> > > >>> a) We have to write tests that tolerate multiple versions. For
> > > instance

Re: DISCUSS: merge calcite-avatica and calcite repositories

[Michael Mior]

> Actually, it is possible to preserve the history of commits when you merge
> two repos -

You can keep the history, but the hashes will necessarily change
because they are based on a different parent. So it will be possible
to see the changes, but the hashes would reference the old Avatica
repository which would no longer be active.

--
Michael Mior
mm...@apache.org

Le mer. 10 nov. 2021 à 06:47, Dmitry Sysolyatin
 a écrit :
>
> Actually, it is possible to preserve the history of commits when you merge
> two repos -
> https://stackoverflow.com/questions/17371150/moving-git-repository-content-to-another-repository-preserving-history
>
> On Tue, Nov 9, 2021 at 11:24 PM Julian Hyde  wrote:
>
> > Merging the repositories is unnecessary, and a massive waste of time.
> >
> > Any supposed “time savings” due to the the merged repositories will be
> > eaten up by dozens of small issues that will crop up long after the merge.
> >
> > I was release manager for Calcite and Avatica this time around and it took
> > me about a week, because the process had changed since last time I was
> > release manager. And now we’re going to change the process again?
> > Absolutely fucking insane.
> >
> > Also, we will lose the history of the Avatica source code. All of the
> > commit hashes for bugs fixed long ago will change.
> >
> > At the time that we split the repositories, I was probably against the
> > idea, because it was work. Now that we’ve split, I am strongly against
> > recombining the repositories.
> >
> > Honestly, people, we have better things to do.
> >
> > Julian
> >
> >
> > > On Nov 9, 2021, at 1:03 PM, Stamatis Zampetakis 
> > wrote:
> > >
> > > The two projects have more or less the same release cadence, they're
> > > maintained by the same community, they're written in the same language,
> > and
> > > they use the same build system.
> > >
> > > Although I understand the reasons the project was split in the first
> > place,
> > > at the moment I tend to agree that having them in the same repo would be
> > > more practical.
> > >
> > > Truth is the merge may require quite a bit of work so we should take this
> > > into consideration.
> > >
> > > If people prefer keeping the projects as they are I am perfectly fine
> > with
> > > that as well. I am leaving the decision to those who feel strongly about
> > > this change.
> > >
> > > Best,
> > > Stamatis
> > >
> > >
> > >
> > > On Tue, Nov 9, 2021, 7:46 PM Vladimir Ozerov  wrote:
> > >
> > >> +1 for a single repo.
> > >>
> > >> Вт, 9 нояб. 2021 г. в 19:22, Vladimir Sitnikov <
> > >> sitnikov.vladi...@gmail.com
> > >>> :
> > >>
> > >>> Michael> is not proposing to change the
> > >>> Michael>structure of modules within both projects, merely to have the
> > >> code
> > >>> for
> > >>> Michael>both in a single repository.
> > >>>
> > >>> I propose to integrate them into a single build, and keep the set of
> > the
> > >>> published jars.
> > >>> However, the modules and dependency structure could be kept as is.
> > >>>
> > >>> We might want to rename folders like
> > >>> calcite-core
> > >>> calcite-linq4j
> > >>> ..
> > >>> avatica-core
> > >>> avatica-server
> > >>>
> > >>> However, I am not sure it is that important to discuss folder names
> > now.
> > >>> The idea is that as you "open Calcite in IDE, you see both Avatica and
> > >>> Calcite modules"
> > >>>
> > >>> Michael>Is there any reason we couldn't have a separate release
> > schedule
> > >> if
> > >>> Michael>both projects are in the same repository?
> > >>>
> > >>> A different schedule means Calcite must support at least two different
> > >>> Avatica versions.
> > >>> In other words, if we allow clients to update Avatica at their will,
> > then
> > >>> we should allow them building Calcite with different Avatica versions,
> > >>> which implies Calcite test code should succeed for multiple different
> > >>> Avatica vesions.
> > >>>
> > >>> That makes it harder to write tests: we have to execute tests with two
> > >>> different Avatica releases (or even more than two).
> > >>>
> > >>> There are at least two sources for complexity:
> > >>>
> > >>> a) We have to write tests that tolerate multiple versions. For
> > instance,
> > >>> "if (avatica.18+) {...}" and so on.
> > >>> That is not really trivial, especially taking into account some of the
> > >>> tests are created with non-yet-popular
> > >>> technologies like Quidem where you can't really google solutions. So
> > the
> > >>> "trivial" task of "making a test to expect two possible outcomes"
> > >>> becomes less trivial as you try to pass the version from GitHub Action
> > to
> > >>> Gradle to JUnit to Quidem to no-one-knows-which class.
> > >>> If we support one Avatica version only, that is not needed. We just
> > patch
> > >>> the test in Avatica and Calcite and that is it.
> > >>> Single repo avoids "Gradle vs Quidem" dance.
> > >>>
> > >>> b) If we claim that we support 5 different Guava versions, 3 different
> > >> JDK
> > >>> versions, 2 different Av

Re: DISCUSS: merge calcite-avatica and calcite repositories

[Vladimir Sitnikov]

Ok, I did not care when Avatica was split into its own repository, and I
assumed Avatica team would maintain the code.
That turned out to be not quite true, and it turns out there are
non-trivial overheads coming from the split.

Of course, everybody can disagree, so if you disagree, you don't need to
explain that. We just disagree and that is it.

Now I do not care how Avatica would evolve further.
It might be even good because Vladimir S won't bother Avatica with mails,
PRs, and other random items.


What I care much more about is that Calcite tests are broken, the CI
reports failure for every commit, and that is really insane.
Calcite CI must not fail on every build.

I am going to delete/ignore the offending tests in Calcite in 24 hours, and
I let the others deal with the case (fix tests, fix Quidem, merge Avatica,
or whatever).

I would no longer review calcite-avatica PRs.



Julian>I was release manager for Calcite and Avatica this time around and
it took me about a week

Julian, as the repositories are merged, there will be a **single** release
process that releases both Calcite and Avatica in a single go.
It would reduce the friction, and it would reduce cases where the release
procedures and steps are "slightly different".

It looks like you assume the release steps would never change, however, in
practice, the drift of Java, Docker, OS, and so on versions would
affect the release steps anyway.

That is why having Avatica and Calcite in a single repository would
simplify the release procedure.

Julian> And now we’re going to change the process again?

I propose to *drop* the release process for "calcite-avatica-java". The
release of "calcite" would release both things (in a single
calcite-x.y.z.tar.gz file),
so the release steps for Calcite would be intact.

Technically, that changes the process.
In practice, I find that you overreact. The merge would make the release
easier.

On top of that, it was you who voted for splitting Calcite and Avatica, so
it was your conscious decision to accept that "release management would get
harder after the split"

Julian>Any supposed “time savings” due to the merged repositories will be
eaten up by dozens of small issues that will crop up long after the merge

The words are not justified. There will always be some random small issues,
even in the case we keep repositories separate.
On the other hand, there are true issues when repositories are different:
* We have two different release processes, so RM have to know the
differences
* we have slightly different dependency versions, so things like "bump
junit" have to be duplicated
* CI configuration is duplicated across the repositories, and it does have
non-zero maintenance overhead
* cross-repository refactorings are harder to do (like the fix for the
exception message)

Julian, I did put four issues that are real, and they induce noticeable
maintenance churn. Java and JUnit will keep updating and you can't prevent
that.
Please highlight the problems exactly and please stop saying "time savings
will be eaten by dozens".
That is not a healthy discussion.

>Also, we will lose the history of the Avatica source code. All of the
commit hashes for bugs fixed long ago will change

We can make calcite-avatica repository read-only, so all commits will be
there.

Vladimir

Re: DISCUSS: merge calcite-avatica and calcite repositories

[Dmitry Sysolyatin]

Actually, it is possible to preserve the history of commits when you merge
two repos -
https://stackoverflow.com/questions/17371150/moving-git-repository-content-to-another-repository-preserving-history

On Tue, Nov 9, 2021 at 11:24 PM Julian Hyde  wrote:

> Merging the repositories is unnecessary, and a massive waste of time.
>
> Any supposed “time savings” due to the the merged repositories will be
> eaten up by dozens of small issues that will crop up long after the merge.
>
> I was release manager for Calcite and Avatica this time around and it took
> me about a week, because the process had changed since last time I was
> release manager. And now we’re going to change the process again?
> Absolutely fucking insane.
>
> Also, we will lose the history of the Avatica source code. All of the
> commit hashes for bugs fixed long ago will change.
>
> At the time that we split the repositories, I was probably against the
> idea, because it was work. Now that we’ve split, I am strongly against
> recombining the repositories.
>
> Honestly, people, we have better things to do.
>
> Julian
>
>
> > On Nov 9, 2021, at 1:03 PM, Stamatis Zampetakis 
> wrote:
> >
> > The two projects have more or less the same release cadence, they're
> > maintained by the same community, they're written in the same language,
> and
> > they use the same build system.
> >
> > Although I understand the reasons the project was split in the first
> place,
> > at the moment I tend to agree that having them in the same repo would be
> > more practical.
> >
> > Truth is the merge may require quite a bit of work so we should take this
> > into consideration.
> >
> > If people prefer keeping the projects as they are I am perfectly fine
> with
> > that as well. I am leaving the decision to those who feel strongly about
> > this change.
> >
> > Best,
> > Stamatis
> >
> >
> >
> > On Tue, Nov 9, 2021, 7:46 PM Vladimir Ozerov  wrote:
> >
> >> +1 for a single repo.
> >>
> >> Вт, 9 нояб. 2021 г. в 19:22, Vladimir Sitnikov <
> >> sitnikov.vladi...@gmail.com
> >>> :
> >>
> >>> Michael> is not proposing to change the
> >>> Michael>structure of modules within both projects, merely to have the
> >> code
> >>> for
> >>> Michael>both in a single repository.
> >>>
> >>> I propose to integrate them into a single build, and keep the set of
> the
> >>> published jars.
> >>> However, the modules and dependency structure could be kept as is.
> >>>
> >>> We might want to rename folders like
> >>> calcite-core
> >>> calcite-linq4j
> >>> ..
> >>> avatica-core
> >>> avatica-server
> >>>
> >>> However, I am not sure it is that important to discuss folder names
> now.
> >>> The idea is that as you "open Calcite in IDE, you see both Avatica and
> >>> Calcite modules"
> >>>
> >>> Michael>Is there any reason we couldn't have a separate release
> schedule
> >> if
> >>> Michael>both projects are in the same repository?
> >>>
> >>> A different schedule means Calcite must support at least two different
> >>> Avatica versions.
> >>> In other words, if we allow clients to update Avatica at their will,
> then
> >>> we should allow them building Calcite with different Avatica versions,
> >>> which implies Calcite test code should succeed for multiple different
> >>> Avatica vesions.
> >>>
> >>> That makes it harder to write tests: we have to execute tests with two
> >>> different Avatica releases (or even more than two).
> >>>
> >>> There are at least two sources for complexity:
> >>>
> >>> a) We have to write tests that tolerate multiple versions. For
> instance,
> >>> "if (avatica.18+) {...}" and so on.
> >>> That is not really trivial, especially taking into account some of the
> >>> tests are created with non-yet-popular
> >>> technologies like Quidem where you can't really google solutions. So
> the
> >>> "trivial" task of "making a test to expect two possible outcomes"
> >>> becomes less trivial as you try to pass the version from GitHub Action
> to
> >>> Gradle to JUnit to Quidem to no-one-knows-which class.
> >>> If we support one Avatica version only, that is not needed. We just
> patch
> >>> the test in Avatica and Calcite and that is it.
> >>> Single repo avoids "Gradle vs Quidem" dance.
> >>>
> >>> b) If we claim that we support 5 different Guava versions, 3 different
> >> JDK
> >>> versions, 2 different Avatica versions,
> >>> then we have to execute 5*3*2 = 30 combinations of the tests.
> >>> That is not really a full matrix, however, things get way easier if we
> >>> support one Avatica version only.
> >>> The amount of tests we need to do during a proper release is much less,
> >> and
> >>> it is easier to commit
> >>> changes that touch Avatica and Calcite at the same time.
> >>>
> >>>
> >>> Vladimir
> >>>
> >>
>
>

Re: CALCITE-3147 | Not Supported

[Stamatis Zampetakis]

Hi Ravi,

Quite an interesting literal :) Indeed It doesn't look valid.

Probably you are hitting a bug, please create a JIRA and include also the
original query that you are trying to parse.

Best,
Stamatis

On Wed, Nov 10, 2021 at 7:32 AM Ravi Kapoor  wrote:

> Hi Team,
> As per https://issues.apache.org/jira/browse/CALCITE-3147
> the triple quoted string for BigQuerySqlDialect should also be
> supported in case of query falls in multiple lines.
> However, I cannot see it working. Neither do we have a test case
> supporting the use case.
> Can you please provide some insights on that?
>
> I have a use case where RexLiteral value falls under multiple new
> lines in BQ dialect and the
> unparsed string generated as shown:
>
> 'SELECT wuaht_.report_access_brch_cd as branch_cd,
> wuaht_.report_access_brch_nm AS branch_name
>  FROM   BRCH_INCTV_PROD_DATA.wire_user_access_hier_1 as wuaht_
>  WHERE  wuaht_.system_access_swb_prsn_id = ?
>  GROUP BY 1,2;';  -> RexLiteral
> which fails in validation.
>
> I can set it as .withLiteralQuoteString(" ''' ") to the context
> But that would make all literals wrapped in triple quotes.
> Can we wrap it in triple quotes only when drills down to multiple lines?
>
> Thanks,
> Ravi
>

[jira] [Created] (CALCITE-4883) The traitset of Exchange operator losses the distribution when externalizing Exchange operator from JSON

[Chunwei Lei (Jira)]

Chunwei Lei created CALCITE-4883:


 Summary: The traitset of Exchange operator losses the distribution 
when externalizing Exchange operator from JSON
 Key: CALCITE-4883
 URL: https://issues.apache.org/jira/browse/CALCITE-4883
 Project: Calcite
  Issue Type: Bug
Reporter: Chunwei Lei
Assignee: Chunwei Lei


{code:java}
// code placeholder
// input.getTraitSet() might have no same distribution
protected Exchange(RelInput input) {
  this(input.getCluster(), input.getTraitSet(), input.getInput(),
  RelDistributionTraitDef.INSTANCE.canonize(input.getDistribution()));
} 

{code}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)