Applications for Travel Assistance to ApacheCon NA 2022 now open

[Ruben Q L]

The ASF Travel Assistance Committee (TAC) is pleased to announce that travel
assistance applications for ApacheCon NA 2022 are now open!

We will be supporting ApacheCon North America in New Orleans, Louisiana,
on October 3rd through 6th, 2022.

TAC exists to help those that would like to attend ApacheCon events, but
are unable to do so for financial reasons. This year, We are supporting
both committers and non-committers involved with projects at the
Apache Software Foundation, or open source projects in general.

For more info on this year's applications and qualifying criteria, please
visit the TAC website at http://www.apache.org/travel/
Applications opened today and will close on the 1st of July 2022.

Important: Applicants have until the closing date above to submit their
applications (which should contain as much supporting material as required
to efficiently and accurately process their request), this will enable TAC
to announce successful awards shortly afterwards.

As usual, TAC expects to deal with a range of applications from a diverse
range of backgrounds. We therefore encourage (as always) anyone thinking
about sending in an application to do so ASAP.

Re: Contributing a custom SQL dialect of Firebolt to Calcite

[Raghav Sharma]

Thankyou for the update.

We will file a JIRA case with the link of the PR raised for the Calcite
team to look into the code of the 'FireboltSqlDialect.java' file.

Also, like it was mentioned before we did add a value 'FIREBOLT' to the
list of allowable values for the 'TEST_DB' inside
'CalciteSystemProperty.java' along with an enum to 'DatabaseInstance' of
the same in 'CalciteAssert.java'. When we tried to run the command: ‘./gradlew
-Dcalcite.test.db=FIREBOLT test’, it couldn't locate the driver. We also
checked with an existing enum i.e 'POSTGRESQL', the same issue persists. It
could not locate the driver for postgresql too.

PFB the test report for FIREBOLT:
7023 tests completed, 427 failed, 96 skipped
Most of these failed due to the driver issue. If this is broken, what
solution do you have?


Best Regards,
Raghav Sharma | SDE 1 | Data Engineering
M: +91 7087637086 I Mail: ragh...@sigmoidanalytics.com



On Sat, Apr 2, 2022 at 1:58 AM Julian Hyde  wrote:

> Sorry I was slow to respond to your earlier message.
>
> (No need to add me to the thread - I am on the list, just backlogged. By
> the way, I am on vacation for the next ten days, so don’t expect responses
> from me. Other project members may or may not be able to help.)
>
> Please log a JIRA case, as I asked. Then we can have discussions in that
> case.
>
> Re 1. I’m surprised that double-quote doesn’t work in
> RelToSqlConverterTest. Other dialects, such as Postgres, also use
> double-quote to quote identifiers. If you post a PR we could perhaps look
> at your code.
>
> Re 2. I think you will need to add “FIREBOLT” to the list of allowable
> values for the TEST_DB parameter [1] and also add a value to enum
> DatabaseInstance [2].
>
> Julian
>
> [1]
> https://github.com/apache/calcite/blob/a8a6569e6ba75efe9d5725c49338a7f181d3ab5c/core/src/main/java/org/apache/calcite/config/CalciteSystemProperty.java#L166
> <
> https://github.com/apache/calcite/blob/a8a6569e6ba75efe9d5725c49338a7f181d3ab5c/core/src/main/java/org/apache/calcite/config/CalciteSystemProperty.java#L166>
>
>
> [2]
> https://github.com/apache/calcite/blob/bf56743554ea27d250d41db2eb83806f9e626b55/testkit/src/main/java/org/apache/calcite/test/CalciteAssert.java#L1904
> <
> https://github.com/apache/calcite/blob/bf56743554ea27d250d41db2eb83806f9e626b55/testkit/src/main/java/org/apache/calcite/test/CalciteAssert.java#L1904>
>
>
> > On Apr 1, 2022, at 5:07 AM, Raghav Sharma 
> wrote:
> >
> > Adding Julian to the thread.
> >
> >
> > Thanks & Regards,
> > Raghav Sharma | SDE 1 | Data Engineering
> > M: +91 7087637086 I Mail: ragh...@sigmoidanalytics.com
> > 
> >
> >
> > On Thu, Mar 31, 2022 at 2:39 PM Raghav Sharma <
> ragh...@sigmoidanalytics.com>
> > wrote:
> >
> >> Hey Julian!
> >> I followed your suggestions related to the dialect/plugin we want to
> >> contribute to Calcite for Firebolt. Some of the issues that we faced so
> far:
> >>
> >> 1. Tests are failing in the RelToSqlConverterTest.java file for Firebolt
> >> reason being the fact the actual parsed query does not contain the
> >> 'Identifier Quote String'. For Firebolt, we've chosen '\"' as the
> >> 'Identifier Quote String'. Have followed the approach that other
> dialects
> >> have used and therefore have added support for this in the dialect file
> as
> >> well as in SqlDialect.java and SqlDialectFactoryImpl.java. Have even
> >> implemented the method 'withFirebolt()' in the test file. Please guide
> what
> >> else needs to be done for this to be fixed and our test cases to be
> passed.
> >>
> >> 2. After adding 'FIREBOLT' to "TEST_DB' in CalciteSystemProperty.java
> and
> >> running the command: ‘./gradlew -Dcalcite.test.db=FIREBOLT test’,
> >> Calcite throws an error that it could not find the enum for 'FIREBOLT'
> in
> >> CalciteAssert.java file. Even after adding the enum either with
> Firebolt's
> >> connection string or using one of the given ones, say POSTGRESQL, it
> >> couldn't locate the driver in both cases. Need your suggestion to tackle
> >> this one.
> >>
> >>
> >> Thanks & Regards,
> >> Raghav Sharma | SDE 1 | Data Engineering
> >> M: +91 7087637086 I Mail: ragh...@sigmoidanalytics.com
> >> 
> >>
> >>
> >> On Tue, Mar 29, 2022 at 1:50 AM Raghav Sharma <
> >> ragh...@sigmoidanalytics.com> wrote:
> >>
> >>> Thank you for the clarification Julian. I will send a mail to
> >>> dev-subscr...@calcite.apache.org for subscribing.
> >>>
> >>>
> >>>
> >>> On Tue, 29 Mar 2022 at 1:45 AM Julian Hyde  wrote:
> >>>
>  You did receive a response[1]. But you didn't receive it because you
>  are not subscribed to this list. Please subscribe[2].
> 
>  Julian
> 
>  [1] https://lists.apache.org/thread/39obrhwpd95bvhc9sb2n4z6zrrwbc21x
> 
>  [2] https://calcite.apache.org/community/#mailing-lists
> 
>  On Mon, Mar 28, 2022 at 1:11 PM Raghav Sharma
>   wrote:
> >
> > Gentle reminder!
> > We are expecting 

Re: Contributing a custom SQL dialect of Firebolt to Calcite

[Stamatis Zampetakis]

Hi Raghav,

Some tests which make use of the TEST_DB property assume that the database
has the foodmart dataset (possibly others as well) loaded [1].
If you need to run tests with Firebolt, Postgres, etc., you have to
manually load the dataset there. To do that you can take some inspiration
from calcite-test-dataset [2] because the latter does not fully work or
maintained at the moment.

Regarding the missing drivers note that the respective dependencies are
selectively fetched by gradle when you run the inteTest* tasks, e.g.,:
./gradlew integTestPostgresql

In order to run tests against Firebolt you will also have to touch a few
places in the build script [3, 4, 5].

Best,
Stamatis

[1]
https://github.com/apache/calcite/blob/a81cfb2ad001589929e190939cf4db928ebac386/core/src/main/java/org/apache/calcite/config/CalciteSystemProperty.java#L163
[2]
https://github.com/vlsi/calcite-test-dataset/blob/829ea54433dd9a4cf4d4a37da13c2dea66a24524/postgresql/pom.xml
[3]
https://github.com/apache/calcite/blob/a81cfb2ad001589929e190939cf4db928ebac386/core/build.gradle.kts#L43
[4]
https://github.com/apache/calcite/blob/a81cfb2ad001589929e190939cf4db928ebac386/core/build.gradle.kts#L84
[5]
https://github.com/apache/calcite/blob/a81cfb2ad001589929e190939cf4db928ebac386/core/build.gradle.kts#L266

On Mon, Apr 4, 2022 at 11:47 AM Raghav Sharma 
wrote:

> Thankyou for the update.
>
> We will file a JIRA case with the link of the PR raised for the Calcite
> team to look into the code of the 'FireboltSqlDialect.java' file.
>
> Also, like it was mentioned before we did add a value 'FIREBOLT' to the
> list of allowable values for the 'TEST_DB' inside
> 'CalciteSystemProperty.java' along with an enum to 'DatabaseInstance' of
> the same in 'CalciteAssert.java'. When we tried to run the command:
> ‘./gradlew
> -Dcalcite.test.db=FIREBOLT test’, it couldn't locate the driver. We also
> checked with an existing enum i.e 'POSTGRESQL', the same issue persists. It
> could not locate the driver for postgresql too.
>
> PFB the test report for FIREBOLT:
> 7023 tests completed, 427 failed, 96 skipped
> Most of these failed due to the driver issue. If this is broken, what
> solution do you have?
>
>
> Best Regards,
> Raghav Sharma | SDE 1 | Data Engineering
> M: +91 7087637086 I Mail: ragh...@sigmoidanalytics.com
> 
>
>
> On Sat, Apr 2, 2022 at 1:58 AM Julian Hyde  wrote:
>
> > Sorry I was slow to respond to your earlier message.
> >
> > (No need to add me to the thread - I am on the list, just backlogged. By
> > the way, I am on vacation for the next ten days, so don’t expect
> responses
> > from me. Other project members may or may not be able to help.)
> >
> > Please log a JIRA case, as I asked. Then we can have discussions in that
> > case.
> >
> > Re 1. I’m surprised that double-quote doesn’t work in
> > RelToSqlConverterTest. Other dialects, such as Postgres, also use
> > double-quote to quote identifiers. If you post a PR we could perhaps look
> > at your code.
> >
> > Re 2. I think you will need to add “FIREBOLT” to the list of allowable
> > values for the TEST_DB parameter [1] and also add a value to enum
> > DatabaseInstance [2].
> >
> > Julian
> >
> > [1]
> >
> https://github.com/apache/calcite/blob/a8a6569e6ba75efe9d5725c49338a7f181d3ab5c/core/src/main/java/org/apache/calcite/config/CalciteSystemProperty.java#L166
> > <
> >
> https://github.com/apache/calcite/blob/a8a6569e6ba75efe9d5725c49338a7f181d3ab5c/core/src/main/java/org/apache/calcite/config/CalciteSystemProperty.java#L166
> >
> >
> >
> > [2]
> >
> https://github.com/apache/calcite/blob/bf56743554ea27d250d41db2eb83806f9e626b55/testkit/src/main/java/org/apache/calcite/test/CalciteAssert.java#L1904
> > <
> >
> https://github.com/apache/calcite/blob/bf56743554ea27d250d41db2eb83806f9e626b55/testkit/src/main/java/org/apache/calcite/test/CalciteAssert.java#L1904
> >
> >
> >
> > > On Apr 1, 2022, at 5:07 AM, Raghav Sharma <
> ragh...@sigmoidanalytics.com>
> > wrote:
> > >
> > > Adding Julian to the thread.
> > >
> > >
> > > Thanks & Regards,
> > > Raghav Sharma | SDE 1 | Data Engineering
> > > M: +91 7087637086 I Mail: ragh...@sigmoidanalytics.com
> > > 
> > >
> > >
> > > On Thu, Mar 31, 2022 at 2:39 PM Raghav Sharma <
> > ragh...@sigmoidanalytics.com>
> > > wrote:
> > >
> > >> Hey Julian!
> > >> I followed your suggestions related to the dialect/plugin we want to
> > >> contribute to Calcite for Firebolt. Some of the issues that we faced
> so
> > far:
> > >>
> > >> 1. Tests are failing in the RelToSqlConverterTest.java file for
> Firebolt
> > >> reason being the fact the actual parsed query does not contain the
> > >> 'Identifier Quote String'. For Firebolt, we've chosen '\"' as the
> > >> 'Identifier Quote String'. Have followed the approach that other
> > dialects
> > >> have used and therefore have added support for this in the dialect
> file
> > as
> > >> well as in SqlDialect.java and SqlDialectFactoryImpl.java. Have even
> >

Re: [DISCUSS] Github PR link to JIRA issue

[Michael Mior]

Autolinks are listed as an "upcoming feature." I don't believe they can be
currently controlled via .asf.yaml.

--
Michael Mior
mm...@apache.org


Le sam. 2 avr. 2022 à 01:19, Julian Hyde  a écrit :

> The present hyperlinking was enabled by Stamatis:
> https://issues.apache.org/jira/browse/CALCITE-4104 <
> https://issues.apache.org/jira/browse/CALCITE-4104>
>
> I’m not exactly sure how the .asf.yaml file accomplishes this. Maybe it
> instructs some ASF robot to modify the GitHub project settings on our
> behalf.
>
> More info on the .asf.yaml file:
> https://cwiki.apache.org/confluence/display/INFRA/Git+-+.asf.yaml+features
> <
> https://cwiki.apache.org/confluence/display/INFRA/Git+-+.asf.yaml+features>
>
>
> Julian
>
>
> > On Apr 1, 2022, at 9:18 PM, Haisheng Yuan  wrote:
> >
> > Yes, that is what I meant.
> >
> > I was expecting to see the link to JIRA in PR's commit page:
> > https://github.com/apache/calcite/pull/2752/commits
> >
> > I am lazy to navigate to JIRA manually. :)
> >
> > Haisheng
> >
> > On 2022/04/01 01:04:19 Julian Hyde wrote:
> >> Ah, I see. In the page for a commit (e.g. [1]) the text
> “[CALCITE-5064]” appears as a blue hyperlink whereas the page for the
> corresponding pull request (e.g. [2]) the “[CALCITE-5064]” text is not a
> hyperlink.
> >>
> >> Is that what you meant, Haisheng?
> >>
> >> Julian
> >>
> >>
> >> [1]
> https://github.com/apache/calcite/commit/d85b2a602a547290bd5be0bba68092b702400731
> <
> https://github.com/apache/calcite/commit/d85b2a602a547290bd5be0bba68092b702400731
> >
> >>
> >> [2] https://github.com/apache/calcite/pull/2752 <
> https://github.com/apache/calcite/pull/2752>
> >>
> >>> On Mar 31, 2022, at 11:05 AM, Michael Mior  wrote:
> >>>
> >>> Stamatis,
> >>>
> >>> Unless I'm misunderstanding, Haisheng was referring to links in the
> >>> opposite direction. That is, linking to JIRA issues from GitHub. The
> >>> setting you reference is for creating a link from JIRA to GitHub.
> >>>
> >>> --
> >>> Michael Mior
> >>> mm...@apache.org
> >>>
> >>>
> >>> Le jeu. 31 mars 2022 à 13:53, Stamatis Zampetakis 
> a
> >>> écrit :
> >>>
>  It's already done via .asf.yaml [1].
> 
>  Best,
>  Stamatis
> 
>  [1]
> 
> 
> https://github.com/apache/calcite/blob/88cc385f98c551c1aca7ffab101934f1c34fdffd/.asf.yaml#L35
> 
>  On Thu, Mar 31, 2022 at 7:19 PM Michael Mior 
> wrote:
> 
> > Not sure what might have changed, but here's the GitHub
> documentation on
> > the feature. If this isn't working as expected, I would contact
> INFRA to
> > make sure things are correctly configured. (Apparently in the future,
>  this
> > may be done via .asf.yaml)
> >
> >
> >
> 
> https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/managing-repository-settings/configuring-autolinks-to-reference-external-resources
> >
> >
> > --
> > Michael Mior
> > mm...@apache.org
> >
> >
> > Le mer. 30 mars 2022 à 13:46, Haisheng Yuan  a
> écrit :
> >
> >> Hi all,
> >>
> >> Previously, the JIRA issue e.g. [CALCITE-6789] in Calcite github PR
> and
> >> commit message will be automatically linked to the JIRA site. Now
> there
> > is
> >> no link anymore.
> >>
> >> Does anyone know what happened? What can we do to add the link back?
> >>
> >> Thanks,
> >> Haisheng Yuan
> >>
> >
> 
> >>
> >>
>
>

Request to be a contributor in jira

[??????]

Hello everybody:
  My name is Jiajun Xie, I have completed some tickets in JIRA. For 
example, https://issues.apache.org/jira/browse/CALCITE-4323 and 
https://issues.apache.org/jira/browse/CALCITE-4997.
  I'm ready to 
solve https://issues.apache.org/jira/browse/CALCITE-5048. Please assign 
the ticket to me. 
Thanks~

[jira] [Created] (CALCITE-5083) In RelBuilder.project_, do not unwrap SARGs

[Vladimir Ozerov (Jira)]

Vladimir Ozerov created CALCITE-5083:


 Summary: In RelBuilder.project_, do not unwrap SARGs
 Key: CALCITE-5083
 URL: https://issues.apache.org/jira/browse/CALCITE-5083
 Project: Calcite
  Issue Type: Improvement
  Components: core
Affects Versions: 1.30.0
Reporter: Vladimir Ozerov
Assignee: Vladimir Ozerov


The {{RelBuilder.project_}} method forcefully unwraps SARG expressions. This 
way, we may lose SARG-related optimization opportunities and even get an 
infinite loop when some rules are applied heuristically. For example, the 
{{ProjectReduceExpressionsRule}} may simplify an expression to a SARG which 
would be unwrapped back to the original form in the RelBuilder, leading to an 
infinite execution of the rule.

The aim of this ticket is to remove the unconditional SARG unwrap from the 
{{RelBuilder}}.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: Why RelBuilder.project unwraps SARGs?

[Vladimir Ozerov]

Hi Julian,

Thank you for the comments. I've created the PR:
https://github.com/apache/calcite/pull/2762

Regards,
Vladimir.

сб, 2 апр. 2022 г. в 19:27, Julian Hyde :

> Good catch. The unwrapping was not intentional on my part. Removing the
> unwrapping seems like a benefit (less work for RelBuilder, therefore better
> planning performance; SARGs in more places, therefore better optimization;
> also the avoidance of cycles as you mention).
>
> There’s a small risk that downstream projects are relying on this
> unwrapping but they can control using config.simplify(). We don’t promise
> not to improve our rewrites from one release to the next.
>
> I think you should fix it.
>
> Julian
>
> > On Apr 2, 2022, at 8:30 AM, Vladimir Ozerov  wrote:
> >
> > Hi,
> >
> > When simplification is enabled in the RelBuilder, a call to the
> > RelBuilder.project unwraps SARGs into their flat counterparts [1]. This
> is
> > not the case for other nodes, like Filter or Join. This behavior may lead
> > to an infinite loop when the ProjectReduceExpressionsRule is used with
> the
> > HepPlanner (which is a common pattern);
> >
> >   1. The original Project(e) has the expression e.
> >   2. ProjectReduceExpressionsRule simplifies it to SARG(e).
> >   3. ProjectReduceExpressionsRule calls RelBuilder.project which returns
> >   back Project(e) instead of Project(SARG(e)).
> >   4. ProjectReduceExpressionsRule calls
> Project(e).transformTo(Project(e))
> >   which schedules invocation of the rule again, leading to a hang.
> >
> > Shall we remove this unwrapping? No tests are affected except for two
> > trivial failures in MaterializedViewRelOptRulesTest.
> >
> > Regards,
> > Vladimir.
> >
> > [1]
> >
> https://github.com/apache/calcite/blob/a81cfb2ad001589929e190939cf4db928ebac386/core/src/main/java/org/apache/calcite/tools/RelBuilder.java#L1948
>

[jira] [Created] (CALCITE-5084) Support ROWS syntax with TABLESAMPLE

[Rohan Garg (Jira)]

Rohan Garg created CALCITE-5084:
---

 Summary: Support ROWS syntax with TABLESAMPLE
 Key: CALCITE-5084
 URL: https://issues.apache.org/jira/browse/CALCITE-5084
 Project: Calcite
  Issue Type: Task
Reporter: Rohan Garg


Currently, Calcite provides a useful syntax for TABLESAMPLE which allows users 
to sample the data being processed. It has two main parameters : 
1. sampling algorithm (BERNOULLI or SYSTEM)
2. sampling percentage (a value between 0 and 100 to indicate rate of sampling)

While percentage is generally good, it is not always possible to provide a 
decent value for it if the user is unaware of the row counts. Further incase of 
subqueries (assuming that the underlying system handles tablesample with 
subqueries), it becomes even more difficult to estimate the correct percentage 
value.

Most likely the 'n ROWS' syntax is not a part of the SQL standard and hence 
wasn't included in the default calcite grammar. But, a few systems have 
implemented it in their dialects : 
1. MS SQL Server : 
[https://docs.microsoft.com/en-us/sql/t-sql/queries/from-transact-sql?view=sql-server-ver15#tablesample-clause]
2. Snowflake : 
[https://docs.snowflake.com/en/sql-reference/constructs/sample.html]
3. Google Spanner : 
[https://cloud.google.com/spanner/docs/reference/standard-sql/query-syntax#tablesample_operator]
4. Apache Spark : 
[https://spark.apache.org/docs/latest/sql-ref-syntax-qry-select-sampling.html]
So, it would be a useful addition to Calcite.

Derived from https://issues.apache.org/jira/browse/CALCITE-5074



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: Request to be a contributor in jira

[Francis Chuang]

I've added you as a contributor in jira and I also assigned you to 
CALCITE-5048.


Francis

On 4/04/2022 11:20 pm, 谢佳君 wrote:

Hello everybody:
  My name is Jiajun Xie, I have completed some tickets in JIRA. For example, 
https://issues.apache.org/jira/browse/CALCITE-4323 and 
https://issues.apache.org/jira/browse/CALCITE-4997.
  I'm ready to 
solve https://issues.apache.org/jira/browse/CALCITE-5048. Please assign the ticket to 
me. 
Thanks~

[jira] [Created] (CALCITE-5085) New Firebolt Dialect

[Kevin Marr (Jira)]

Kevin Marr created CALCITE-5085:
---

 Summary: New Firebolt Dialect
 Key: CALCITE-5085
 URL: https://issues.apache.org/jira/browse/CALCITE-5085
 Project: Calcite
  Issue Type: New Feature
  Components: jdbc-adapter
Reporter: Kevin Marr


Firebolt is a columnar MPP cloud data warehouse with decoupled storage and 
compute. It closely follows PostgreSQL but has enough differences to warrant a 
separate dialect in Calcite. We would like to add this dialect to make it 
easier for any BI/modeling vendors using Calcite to implement Firebolt 
connectors.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Vararg/Variadic UDFs or workarounds?

[Ian Bertolacci]

Howdy!
We’re trying to add a vararg/variadic UDF, but cannot seem to make it work out.

In our system, we define our UDFs to the Schema’s function multimap, and so 
have classes with methods which are provided to 
ScalarFunctionImpl.create(theClass, “methodName” ) to create the Function 
object, and from which the parser/validator draws the SQL function signature.
However, there doesn’t seem to be a way do define such a method where 
ScalarFunctionImpl infers the variadic function signature, but rather as 
accepting a single List parameter.

I see that CALCITE-2772 was raised to solve this but the work seems to have 
stalled in review.

Is there any plan to restart this work?
And in the mean time, is there a good way of defining variadic UDFs?

One solution we’ve come up with is to define a function with all (except the 
first) optional parameters so that it *looks* like a variadic function, but 
this is limited to 254 parameters by the JVM.
While 254 parameters is quite a lot, we predict that our users may exceed this 
limit.
Thanks!
-Ian J. Bertolacci