[jira] [Created] (CALCITE-3608) Promote RelOptUtil.createCastRel to not create new projection if the input rel is already a project
Danny Chen created CALCITE-3608: --- Summary: Promote RelOptUtil.createCastRel to not create new projection if the input rel is already a project Key: CALCITE-3608 URL: https://issues.apache.org/jira/browse/CALCITE-3608 Project: Calcite Issue Type: Bug Components: core Affects Versions: 1.21.0 Reporter: Danny Chen Assignee: Danny Chen Fix For: 1.22.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-3607) Support LogicalTableModify in RelShuttle
xzh_dz created CALCITE-3607: --- Summary: Support LogicalTableModify in RelShuttle Key: CALCITE-3607 URL: https://issues.apache.org/jira/browse/CALCITE-3607 Project: Calcite Issue Type: Wish Components: core Reporter: xzh_dz RelShuttle don't support LogicalTableModify,this pr try to support LogicalTableModify in RelShuttle。 -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [DISCUSS] Move CalciteAssert and similar "test framework" classes to its own testlib module
Hi, Vladimir
> I suggest we do not publish testlib artifacts to Nexus (who needs
> Calcite-related assertions and things like that?)
+1 to have the module be published.
Some of my users write tests based on functionalities provided from Calcite
test framework.
Best,
Jin
Michael Mior 于2019年12月18日周三 上午7:47写道：
> Sounds good to me. Although I'd still like to see the module be
> published. I'm currently using it my notebooks project
> (https://github.com/michaelmior/calcite-notebooks) because some of the
> test code removes boilerplate when writing examples.
> --
> Michael Mior
> mm...@apache.org
>
> Le mar. 17 déc. 2019 à 15:49, Vladimir Sitnikov
> a écrit :
> >
> > Hi,
> >
> > Currently
> > org.apache.calcite.test.CalciteAssert, org.apache.calcite.util.TestUtil,
> > and probably other classes are hard to find.
> > I suggest we create a testlib module (or something like that), that would
> > contain "test framework" code.
> >
> > That would make test framework easier to discover (one could open the
> full
> > set of packages in the testlib),
> > and it would make "autocomplete" better (especially for adapters).
> >
> > I suggest we do not publish testlib artifacts to Nexus (who needs
> > Calcite-related assertions and things like that?)
> >
> > Currently testImplementation(project(":core", "testClasses")) is used in
> > lots of places, and it causes "core"
> > test clases appear in autocomplete menu for adapters.
> > For example, typing "new Sql..." in GeodeAssertions suggests
> > SqlAdvisorJdbcTest which is valid, yet it is weird.
> >
> > What do you think?
> >
> > Just in case you did not know: file movement in Git keeps history (even
> in
> > case there are slight modifications).
> > However, the key idea here is to keep files in their current packages,
> but
> > move them into teslib module.
> >
> > Vladimir
>
[jira] [Created] (CALCITE-3606) batch insert failed
Ran Cao created CALCITE-3606:
Summary: batch insert failed
Key: CALCITE-3606
URL: https://issues.apache.org/jira/browse/CALCITE-3606
Project: Calcite
Issue Type: Wish
Components: core
Affects Versions: 1.21.0
Reporter: Ran Cao
when I try to execute sql like (insert into example_table (column1,column2)
values (value1,value2),(value1,value2)), it will failed with error message like
this: column "EXPR$0" of relation "example_table" does not exist. I find the
reason is that when converting SqlNode(insert sql) to RelNode(TableModify), one
of the steps is to change the column that stored in RelDataType from the fake
column name (like "EXPR$0") to the real column name (like "id"). But when the
values part in sql is more than one , the step above-mentioned will skip
because the RelNode is instance of LogicalUnion instead of Project, the code
refered to org.apache.calcite.tools.RelBuilder line 1461:
if (input instanceof Project && fieldNames != null) {
// change the column name
}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: Quicksql
Some updates. Recently i took a look at their doc and source code, and found this project uses SQL parsing and Relational algebra of Calcite to get query plan, and also translates to spark SQL for joining different datasources, or corresponding query for single datasource. Although it copies many classes from Calcite, the idea of QuickSQL seems some of interests, and code is succinct. Best, Trista Juan Pan (Trista) Senior DBA & PPMC of Apache ShardingSphere(Incubating) E-mail: panj...@apache.org On 12/13/2019 17:16，Juan Pan wrote： Yes, indeed. Juan Pan (Trista) Senior DBA & PPMC of Apache ShardingSphere(Incubating) E-mail: panj...@apache.org On 12/12/2019 18:00，Alessandro Solimando wrote： Adapters must be needed by data sources not supporting SQL, I think this is what Juan Pan was asking for. On Thu, 12 Dec 2019 at 04:05, Haisheng Yuan wrote: Nope, it doesn't use any adapters. It just submits partial SQL query to different engines. If query contains table from single source, e.g. select count(*) from hive_table1, hive_table2 where a=b; then the whole query will be submitted to hive. Otherwise, e.g. select distinct a,b from hive_table union select distinct a,b from mysql_table; The following query will be submitted to Spark and executed by Spark: select a,b from spark_tmp_table1 union select a,b from spark_tmp_table2; spark_tmp_table1: select distinct a,b from hive_table spark_tmp_table2: select distinct a,b from mysql_table On 2019/12/11 04:27:07, "Juan Pan" wrote: Hi Haisheng, The query on different data source will then be registered as temp spark tables (with filter or join pushed in), the whole query is rewritten as SQL text over these temp tables and submitted to Spark. Does it mean QuickSQL also need adaptors to make query executed on different data source? Yes, virtualization is one of Calcite’s goals. In fact, when I created Calcite I was thinking about virtualization + in-memory materialized views. Not only the Spark convention but any of the “engine” conventions (Drill, Flink, Beam, Enumerable) could be used to create a virtual query engine. Basically, i like and agree with Julian’s statement. It is a great idea which personally hope Calcite move towards. Give my best wishes to Calcite community. Thanks, Trista Juan Pan panj...@apache.org Juan Pan(Trista), Apache ShardingSphere On 12/11/2019 10:53，Haisheng Yuan wrote： As far as I know, users still need to register tables from other data sources before querying it. QuickSQL uses Calcite for parsing queries and optimizing logical expressions with several transformation rules. The query on different data source will then be registered as temp spark tables (with filter or join pushed in), the whole query is rewritten as SQL text over these temp tables and submitted to Spark. - Haisheng -- 发件人：Rui Wang 日 期：2019年12月11日 06:24:45 收件人： 主 题：Re: Quicksql The co-routine model sounds fitting into Streaming cases well. I was thinking how should Enumerable interface work with streaming cases but now I should also check Interpreter. -Rui On Tue, Dec 10, 2019 at 1:33 PM Julian Hyde wrote: The goal (or rather my goal) for the interpreter is to replace Enumerable as the quick, easy default convention. Enumerable is efficient but not that efficient (compared to engines that work on off-heap data representing batches of records). And because it generates java byte code there is a certain latency to getting a query prepared and ready to run. It basically implements the old Volcano query evaluation model. It is single-threaded (because all work happens as a result of a call to 'next()' on the root node) and cannot handle branching data-flow graphs (DAGs). The Interpreter operates uses a co-routine model (reading from queues, writing to queues, and yielding when there is no work to be done) and therefore could be more efficient than enumerable in a single-node multi-core system. Also, there is little start-up time, which is important for small queries. I would love to add another built-in convention that uses Arrow as data format and generates co-routines for each operator. Those co-routines could be deployed in a parallel and/or distributed data engine. Julian On Tue, Dec 10, 2019 at 3:47 AM Zoltan Farkas wrote: What is the ultimate goal of the Calcite Interpreter? To provide some context, I have been playing around with calcite + REST (see https://github.com/zolyfarkas/jaxrs-spf4j-demo/wiki/AvroCalciteRest < https://github.com/zolyfarkas/jaxrs-spf4j-demo/wiki/AvroCalciteRest> for detail of my experiments) —Z On Dec 9, 2019, at 9:05 PM, Julian Hyde wrote: Yes, virtualization is one of Calcite’s goals. In fact, when I created Calcite I was thinking about virtualization + in-memory materialized views. Not only the Spark convention but any of the “engine” conventions (Drill, Flink, Beam, Enumerable) could be
Re: [DISCUSS] Remove Kotlin
I've used the new version calcite with new version of IntelliJ, everything works. I like that. I can see valadmir put some efforts in this, I respect that. and all effort put in to the codebase should be respected. from my side, I don't contribute as much now, but occasionally I would look at the new stuff added so as long I can REPL the code I am okay with it. as for 'kotlin', like when it was first brought up in the calcite mail thread, I am curious about that and would be willing to learn more. On Wed, Dec 18, 2019 at 7:45 AM Michael Mior wrote: > Le mar. 17 déc. 2019 à 15:26, Vladimir Sitnikov > a écrit : > > > > Vladimir>Quidem, CalciteAssert > > Michael>If you want to propose removing either of these, we could have a > > Michael>discussion about it, but you're talking about code which is > already > > Michael>heavily used throughout Calcite. > > > > The point of "we assume contributors are good at Java, thus we must keep > > the code to be Java-only" is weak. > > New contributors will likely see Quidem and CalciteAssert for the first > > time, and Java knowledge does not help there. > > > > I didn't make that point. Those are you words. > > > It does not imply that languages like Quidem and/or CalciteAssert are a > bad > > fit for their job, but it is wrong to judge > > based solely on "it is not Java". > > > > Michael>The consensus from the discussion you started seems to be that > > Michael>Kotlin should not be added to the tests > > > > It is not like that. > > I counted at least 5 different contributors stating they did not think > Kotlin should be introduced into test code. You seemed to be the only > one in the discussion strongly promoting it. If that's not consensus, > I must have misinterpreted the discussion. > > > > > Michael>I agree that for these specific tests, readability is improved > > > > That is exactly my point. There's an improvement, the downsides are > small, > > so I just committed it. > > > > Michael>But many tests require more than this > > > > That is to be discussed on a test by test basis (or use-case by > use-case). > > For instance, strings (especially, multi-line ones) with $ is an issue > for > > Kotlin for now. > > > > Vladimir > -- ~~~ no mistakes ~~
Re: [DISCUSS] Move CalciteAssert and similar "test framework" classes to its own testlib module
Sounds good to me. Although I'd still like to see the module be
published. I'm currently using it my notebooks project
(https://github.com/michaelmior/calcite-notebooks) because some of the
test code removes boilerplate when writing examples.
--
Michael Mior
mm...@apache.org
Le mar. 17 déc. 2019 à 15:49, Vladimir Sitnikov
a écrit :
>
> Hi,
>
> Currently
> org.apache.calcite.test.CalciteAssert, org.apache.calcite.util.TestUtil,
> and probably other classes are hard to find.
> I suggest we create a testlib module (or something like that), that would
> contain "test framework" code.
>
> That would make test framework easier to discover (one could open the full
> set of packages in the testlib),
> and it would make "autocomplete" better (especially for adapters).
>
> I suggest we do not publish testlib artifacts to Nexus (who needs
> Calcite-related assertions and things like that?)
>
> Currently testImplementation(project(":core", "testClasses")) is used in
> lots of places, and it causes "core"
> test clases appear in autocomplete menu for adapters.
> For example, typing "new Sql..." in GeodeAssertions suggests
> SqlAdvisorJdbcTest which is valid, yet it is weird.
>
> What do you think?
>
> Just in case you did not know: file movement in Git keeps history (even in
> case there are slight modifications).
> However, the key idea here is to keep files in their current packages, but
> move them into teslib module.
>
> Vladimir
Re: [DISCUSS] Remove Kotlin
Le mar. 17 déc. 2019 à 15:26, Vladimir Sitnikov a écrit : > > Vladimir>Quidem, CalciteAssert > Michael>If you want to propose removing either of these, we could have a > Michael>discussion about it, but you're talking about code which is already > Michael>heavily used throughout Calcite. > > The point of "we assume contributors are good at Java, thus we must keep > the code to be Java-only" is weak. > New contributors will likely see Quidem and CalciteAssert for the first > time, and Java knowledge does not help there. > I didn't make that point. Those are you words. > It does not imply that languages like Quidem and/or CalciteAssert are a bad > fit for their job, but it is wrong to judge > based solely on "it is not Java". > > Michael>The consensus from the discussion you started seems to be that > Michael>Kotlin should not be added to the tests > > It is not like that. I counted at least 5 different contributors stating they did not think Kotlin should be introduced into test code. You seemed to be the only one in the discussion strongly promoting it. If that's not consensus, I must have misinterpreted the discussion. > > Michael>I agree that for these specific tests, readability is improved > > That is exactly my point. There's an improvement, the downsides are small, > so I just committed it. > > Michael>But many tests require more than this > > That is to be discussed on a test by test basis (or use-case by use-case). > For instance, strings (especially, multi-line ones) with $ is an issue for > Kotlin for now. > > Vladimir
Re: [DISCUSS] Tests vs multiline strings
Undeniably multi-line strings can improve the readability of the code and facilitate reviews but I would prefer if we didn't refactor existing tests from Java to Kotlin mainly for two reasons: 1. Losing the history is an important disadvantage for me. Quite often, I find my self looking in the history of the tests to find the parts of the core that were changed. Or the opposite, looking for instance how an API works by checking the tests that were added in a particular commit. 2. The feature is already present as experimental in Java 13 [1] so we could start using it in the near future (eventually we will start using a newer jdk) so from my point of view there is no big rush to use Kotlin for this. If I had to choose between the history and multi-line strings I would opt for the history. I lived with normal string concatenation for a while so I am accustomed to reading the code the way it is without too much struggle. No objections for using Kotlin for new tests. Best, Stamatis [1] https://openjdk.java.net/jeps/355 On Tue, Dec 17, 2019 at 1:27 AM Kevin Risden wrote: > I don't see a major benefit to switching to an entirely new language to get > multiline strings. I agree that sticking to Java makes sense. > > Kevin Risden > > > On Mon, Dec 16, 2019 at 12:50 AM Vladimir Sitnikov < > sitnikov.vladi...@gmail.com> wrote: > > > Feng>Introducing another language > > > > It is the same language that is used for the build scripts, so it is not > a > > new language. > > > > Danny> in the code evolving and the Scala has many > > Danny> tricky problems especially the version compatibility > > > > Kotlin has strong backward compatibility. > > > > Julian>Transitioning our tests to a different language (Kotlin) is a > > Julian> drastic solution. It requires developers to understand a new > > language, > > > > Note: most of the time, test code is just a glue code between input data > > and asserts. The complicated logic is not there (which is good by the > way). > > > > At the same time, it means it will be very little new to understand. > > > > Vladimir > > >
Re: [DISCUSS] Move CalciteAssert and similar "test framework" classes to its own testlib module
It probably makes sense.
But I am exhausted, utterly exhausted, with the "moving stuff around"
that Vladimir has done over the last few months.
For example. I ran into a problem running SqlPrettyWriterTest a few
days ago[1] that did not exist a few months ago (before the
gradle/junit upgrade). After I solved that problem, I can into
another, and another. After 3 or 4 days, I am still unable to run the
test and have Intellij show the difference in a pop-up window.
What caused these problems? Maybe the transition from maven to gradle.
Maybe I didn't adequately clean out generated files in my sandox when
I transitioned my Intellij project to gradle. Maybe I should be using
gradle-based tests in Intellij rather than junit-based tests. Maybe
the transition from junit4 to junit5. Who knows?
All of those changes were done by Vladimir. Not one of these changes
adds a single feature, or fixes a single bug, in "Calcite the
product", the library we deliver to our end-users. All of them are
supposedly to make developers' lives easier, but I have lost a week.
Julian
[1] https://issues.apache.org/jira/browse/CALCITE-3595
On Tue, Dec 17, 2019 at 12:49 PM Vladimir Sitnikov
wrote:
>
> Hi,
>
> Currently
> org.apache.calcite.test.CalciteAssert, org.apache.calcite.util.TestUtil,
> and probably other classes are hard to find.
> I suggest we create a testlib module (or something like that), that would
> contain "test framework" code.
>
> That would make test framework easier to discover (one could open the full
> set of packages in the testlib),
> and it would make "autocomplete" better (especially for adapters).
>
> I suggest we do not publish testlib artifacts to Nexus (who needs
> Calcite-related assertions and things like that?)
>
> Currently testImplementation(project(":core", "testClasses")) is used in
> lots of places, and it causes "core"
> test clases appear in autocomplete menu for adapters.
> For example, typing "new Sql..." in GeodeAssertions suggests
> SqlAdvisorJdbcTest which is valid, yet it is weird.
>
> What do you think?
>
> Just in case you did not know: file movement in Git keeps history (even in
> case there are slight modifications).
> However, the key idea here is to keep files in their current packages, but
> move them into teslib module.
>
> Vladimir
[DISCUSS] Move CalciteAssert and similar "test framework" classes to its own testlib module
Hi,
Currently
org.apache.calcite.test.CalciteAssert, org.apache.calcite.util.TestUtil,
and probably other classes are hard to find.
I suggest we create a testlib module (or something like that), that would
contain "test framework" code.
That would make test framework easier to discover (one could open the full
set of packages in the testlib),
and it would make "autocomplete" better (especially for adapters).
I suggest we do not publish testlib artifacts to Nexus (who needs
Calcite-related assertions and things like that?)
Currently testImplementation(project(":core", "testClasses")) is used in
lots of places, and it causes "core"
test clases appear in autocomplete menu for adapters.
For example, typing "new Sql..." in GeodeAssertions suggests
SqlAdvisorJdbcTest which is valid, yet it is weird.
What do you think?
Just in case you did not know: file movement in Git keeps history (even in
case there are slight modifications).
However, the key idea here is to keep files in their current packages, but
move them into teslib module.
Vladimir
Re: [DISCUSS] Remove Kotlin
Vladimir>Quidem, CalciteAssert Michael>If you want to propose removing either of these, we could have a Michael>discussion about it, but you're talking about code which is already Michael>heavily used throughout Calcite. The point of "we assume contributors are good at Java, thus we must keep the code to be Java-only" is weak. New contributors will likely see Quidem and CalciteAssert for the first time, and Java knowledge does not help there. It does not imply that languages like Quidem and/or CalciteAssert are a bad fit for their job, but it is wrong to judge based solely on "it is not Java". Michael>The consensus from the discussion you started seems to be that Michael>Kotlin should not be added to the tests It is not like that. Michael>I agree that for these specific tests, readability is improved That is exactly my point. There's an improvement, the downsides are small, so I just committed it. Michael>But many tests require more than this That is to be discussed on a test by test basis (or use-case by use-case). For instance, strings (especially, multi-line ones) with $ is an issue for Kotlin for now. Vladimir
Re: [DISCUSS] Remove Kotlin
Le mar. 17 déc. 2019 à 10:59, Vladimir Sitnikov
a écrit :
>
> Michael>However, we know our
> Michael>current contributors are reasonably fluent in Java. I'm not sure
> about
> Michael>Kotlin.
>
> 1) New contributions are not fluent in Quidem at all
> 2) New contributions are not fluent in CalciteAssert at all
If you want to propose removing either of these, we could have a
discussion about it, but you're talking about code which is already
heavily used throughout Calcite.
> 3) As I said, if someone does not want to get familiar with Kotlin, they
> can write tests in Java (or XML or whatever we have)
> 4) I'm sure even those who have never used Kotlin before could add tests to
> by using battle-tested ctrl+c/v combinations:
> https://github.com/vlsi/calcite/blob/0d6bec4140da46af07d58cf07a5e682d61529603/core/src/test/java/org/apache/calcite/test/SqlValidatorDynamicTest.kt#L55
> 5) If logic is required, then "if", "for", "try", "method calls" work very
> similar to Java, and documentation is there on StackOverflow, Google, and
> so on.
I would agree with those points.
>
> Michael>I assume that similar concerns would have been raised
>
> Frankly speaking, I find those concerns too moot to ever rise.
> Of course, it is valid that "adding a language" is a significant change,
> however, I am not among those who would veto such changes based on very
> generic statements like "ok, the new language would be hard for new
> contributors" or "ok, the new language would complicate maintenance".
> I assume everybody here knows "adding new languages" requires good
> justification.
>
> I find it truly nonappealing to receive opinions like "it would complicate
> maintenance" when not a single clue is given on how maintenance could screw
> up.
> If someone proposes COBOL, then I would definitely question that proposal :)
> If someone proposes OCaml I would get curious how that would play rather
> than declare "it is not proven yet to be the best option".
>
> At the same time, if someone is going to play "oh, X is a new language"
> card, then I would highlight that Pig, Redis, Mongo, and Quidem are all
> **languages** right inside Calcite's codebase.
> It is not something theoretical, but I personally did run into exceptions
> from Pig tests similar to https://issues.apache.org/jira/browse/CALCITE-3336
> Adding Kotlin to the mix of Pig, Piglet, and Mongo is innocent and it is a
> small change :)
>
I didn't suggest I intend to veto any changes. A discussion was
started, and I'm sharing my opinion.
>
> Recently the build system was switched from Maven to Gradle.
> Before the migration there were lots of emails coming back and forth like
> "oh, the release process is too complicated, it would take ages to fully
> migrate to another build system" and so on.
>
> Have you seen the vote regarding Avatica release? It goes smooth.
> Has Gradle resolved Maven's pain points I declared? It has.
> Have you heard feedback on that change? The feedback seems to be positive
> so far.
Yes, and although I haven't had touch thee new build system much
myself, it seems to have worked out very wlel.
>
> I assume we follow "commit then review" policy when committers commit
> whatever they think is good, and in the rare cases when they are not sure
> they discuss in JIRA, mailing list, and so on.
Agreed. The consensus from the discussion you started seems to be that
Kotlin should not be added to the tests.
>
> Michael>But I think that's a straw man. Rewriting some things in Kotlin
> still
> Michael>has some of the same disadvantages
>
> I'm sure we can't discuss "Rewriting some things" without mentioning the
> "things" explicitly.
> In my view, rewriting SqlValidatorMatchTest in Kotlin had no significant
> enough disadvantages.
>
I agree, but you seem to be inferring that others think the "things"
is "everything" when no one has said this. (It's certainly not the
argument I'm making.)
> Are there tests that degrade when rewritten? Of course, there are.
> Should we discuss it like "has some of the same disadvantages"? I don't
> think so.
>
>
> Vladimir>
> https://github.com/apache/calcite/pull/1657/commits/0d6bec4140da46af07d58cf07a5e682d61529603#diff-7a7027c499b6e4f5e7448b7b971052f1R85-R94
> Vladimir> much easier to read and update than similar tests in Java.
> Michael>Assuming you understand, Kotlin, yes.
>
> Really?
> Did you just say it requires one to understand Kotlin to successfully
> copy-paste the following test?
>
> @Test
> fun `ambiguous dynamic star4`() {
> sql(
> """
> select n.n_nation
> from (select * from "SALES".NATION) as n,
>(select * from "SALES".CUSTOMER)
> """.trimIndent()
> ).type("RecordType(ANY N_NATION) NOT NULL")
> }
No, I didn't say anything about copy and pasting. If all that is
required for a test is copy and pasting, then sure this should be
doable without knowing Kotlin. But many tests require more than
Re: [DISCUSS] Remove Kotlin
Michael>However, we know our
Michael>current contributors are reasonably fluent in Java. I'm not sure
about
Michael>Kotlin.
1) New contributions are not fluent in Quidem at all
2) New contributions are not fluent in CalciteAssert at all
3) As I said, if someone does not want to get familiar with Kotlin, they
can write tests in Java (or XML or whatever we have)
4) I'm sure even those who have never used Kotlin before could add tests to
by using battle-tested ctrl+c/v combinations:
https://github.com/vlsi/calcite/blob/0d6bec4140da46af07d58cf07a5e682d61529603/core/src/test/java/org/apache/calcite/test/SqlValidatorDynamicTest.kt#L55
5) If logic is required, then "if", "for", "try", "method calls" work very
similar to Java, and documentation is there on StackOverflow, Google, and
so on.
Michael>I assume that similar concerns would have been raised
Frankly speaking, I find those concerns too moot to ever rise.
Of course, it is valid that "adding a language" is a significant change,
however, I am not among those who would veto such changes based on very
generic statements like "ok, the new language would be hard for new
contributors" or "ok, the new language would complicate maintenance".
I assume everybody here knows "adding new languages" requires good
justification.
I find it truly nonappealing to receive opinions like "it would complicate
maintenance" when not a single clue is given on how maintenance could screw
up.
If someone proposes COBOL, then I would definitely question that proposal :)
If someone proposes OCaml I would get curious how that would play rather
than declare "it is not proven yet to be the best option".
At the same time, if someone is going to play "oh, X is a new language"
card, then I would highlight that Pig, Redis, Mongo, and Quidem are all
**languages** right inside Calcite's codebase.
It is not something theoretical, but I personally did run into exceptions
from Pig tests similar to https://issues.apache.org/jira/browse/CALCITE-3336
Adding Kotlin to the mix of Pig, Piglet, and Mongo is innocent and it is a
small change :)
Recently the build system was switched from Maven to Gradle.
Before the migration there were lots of emails coming back and forth like
"oh, the release process is too complicated, it would take ages to fully
migrate to another build system" and so on.
Have you seen the vote regarding Avatica release? It goes smooth.
Has Gradle resolved Maven's pain points I declared? It has.
Have you heard feedback on that change? The feedback seems to be positive
so far.
I assume we follow "commit then review" policy when committers commit
whatever they think is good, and in the rare cases when they are not sure
they discuss in JIRA, mailing list, and so on.
Michael>But I think that's a straw man. Rewriting some things in Kotlin
still
Michael>has some of the same disadvantages
I'm sure we can't discuss "Rewriting some things" without mentioning the
"things" explicitly.
In my view, rewriting SqlValidatorMatchTest in Kotlin had no significant
enough disadvantages.
Are there tests that degrade when rewritten? Of course, there are.
Should we discuss it like "has some of the same disadvantages"? I don't
think so.
Vladimir>
https://github.com/apache/calcite/pull/1657/commits/0d6bec4140da46af07d58cf07a5e682d61529603#diff-7a7027c499b6e4f5e7448b7b971052f1R85-R94
Vladimir> much easier to read and update than similar tests in Java.
Michael>Assuming you understand, Kotlin, yes.
Really?
Did you just say it requires one to understand Kotlin to successfully
copy-paste the following test?
@Test
fun `ambiguous dynamic star4`() {
sql(
"""
select n.n_nation
from (select * from "SALES".NATION) as n,
(select * from "SALES".CUSTOMER)
""".trimIndent()
).type("RecordType(ANY N_NATION) NOT NULL")
}
I'm sure it does not require Kotlin understanding.
It is clear the test is doing something with SQL, and it has something to
do with the type of a record.
Would Java make this test much more accessible? Of course, it would not.
Java does not help to understand what the test is doing here. Java does not
help to understand what is really important in the test.
Java does not help to undestand which of the two stars is dynamic here, and
nothing explains why the test is "ambiguous".
On the other hand, Kotlin improves test readability much, as the SQL is now
easy to get.
Vladimir
Re: [DISCUSS] Remove Kotlin
Hello Calcite Team, I believe products are advancing through constant improvements rather than freezing codebase. Why does nobody consider Kotlin as an opportunity to improve Calcite? Indeed, most of the replies sound like: "I don't want to learn Kotlin because it requires extra efforts, etc, etc." Sincerely, Igor On Tue, Dec 17, 2019 at 4:20 PM Michael Mior wrote: > Le mar. 17 déc. 2019 à 03:30, Vladimir Sitnikov > a écrit : > > > > Kevin>Focusing on the technical side of things, I agree that introducing > a > > new > > Kevin>language is of little benefit currently > > > > Kevin, what is your opinion on removing Quidem language? > > Focusing on the technical side, it is a standalone language. > > The language is not Java, it has limited tooling, it has a limited set of > > users, it has 0 or so questions on StackOverflow and so on. > > I'm not Kevin, but removing a language is quite different from > introducing a new language. I wasn't around when Quidem tests were > introduced, but I assume that similar concerns would have been raised. > > > > > Kevin>I agree that introducing a new > > Kevin>language is of little benefit currently > > > > Keeping tests easy to read, easy to edit, easy to create, easy to debug, > > and easy to maintain is hard. > > Making tests easy to read simplifies code reviews that pay off in the > long > > run. > > Agreed on those points. I'm not convinced though that having another > language tests are written in will simplify code review. It may be > true for someone equally literate in Java and Kotlin that in some > instances, Kotlin tests will be easier to read. However, we know our > current contributors are reasonably fluent in Java. I'm not sure about > Kotlin. > > > It is sad you mention "code readability" as "little benefit currently" > item. > > > > Kevin>surprised that a change went in to > > Kevin>switch to Kotlin especially after the discussion that is happening > on > > the > > Kevin>mailing list. > > > > Those are two different items. The discussion re $ is still open, and > > there's no clear answer yet. > > At the same time, in the very same thread, there were explicit opinions > > that "tests that do not use $ might still benefit from improved > > readability". > > > > The change in [2] is not related to the $-discussion, so I don't see why > > people relate them. > > > > Kevin>I agree that introducing a new > > Kevin>language is of little benefit currently > > > > It is not introducing a brand new language. The very same language is > used > > in the build scripts. > > The language was designed for interoperability with Java, and it does > > improve things if used appropriately. > > > > For instance, tests like > > > https://github.com/apache/calcite/pull/1657/commits/0d6bec4140da46af07d58cf07a5e682d61529603#diff-7a7027c499b6e4f5e7448b7b971052f1R85-R94 > > are > > much easier to read and update than similar tests in Java. > > Assuming you understand, Kotlin, yes. I agree it's not exactly > introducing a new language, but I think using Kotlin in tests is quite > different from using it in build scripts. We expect most contributors > to write tests. If build script editing is a bit less accessible, I > think that's less of a concern. > > > > > Note: Kotlin tests are still regular JUnit5 tests, so they get proper > > statistics in the CI outputs like test count, skipped test count, failure > > count, test duration. > > > > Kevin>In my opinion there are more negatives than positives > > Kevin>currently. > > > > What your opinion re Kotlin is based on besides angst and "introducing a > > new language"? > > I can easily relate how the items play for writing tests COBOL, but, > well, > > I hope we don't consider that :) > > > > Kotlin is a modern language with good tooling. > > It was designed for smooth Java interop which plays well for maintenance. > > The code is easy to read, and it is copy-paste friendly in the same way > as > > almost any other language we currently have in the source repository. > > So newbies could copy-paste Kotlin code or Java code or Quidem code. > > > > For instance, people might still create tests in Java and call methods > > written in Kotlin. They might create "pure Java" tests if they like. > > They might even create Quidem tests, but they won't be able to call > > Java/Kotlin methods (and/or extend classes) then. > > > > It looks like you express an opinion on "let's rewrite everything in > > Kotlin" rather than "let's pick the proper tools on a case by case > basis". > > > > Currently, there are valid reasons against migrating all the tests to > > Kotlin, and I guess we have never discussed that. > > At the same time, we are not discussing "rewrite everything in Kotlin" > here. > > I don't think anyone was discussing rewriting everything in Kotlin. > But I think that's a straw man. Rewriting some things in Kotlin still > has some of the same disadvantages. > > > > > Vladimir >
Re: [DISCUSS] Remove Kotlin
Le mar. 17 déc. 2019 à 03:30, Vladimir Sitnikov a écrit : > > Kevin>Focusing on the technical side of things, I agree that introducing a > new > Kevin>language is of little benefit currently > > Kevin, what is your opinion on removing Quidem language? > Focusing on the technical side, it is a standalone language. > The language is not Java, it has limited tooling, it has a limited set of > users, it has 0 or so questions on StackOverflow and so on. I'm not Kevin, but removing a language is quite different from introducing a new language. I wasn't around when Quidem tests were introduced, but I assume that similar concerns would have been raised. > > Kevin>I agree that introducing a new > Kevin>language is of little benefit currently > > Keeping tests easy to read, easy to edit, easy to create, easy to debug, > and easy to maintain is hard. > Making tests easy to read simplifies code reviews that pay off in the long > run. Agreed on those points. I'm not convinced though that having another language tests are written in will simplify code review. It may be true for someone equally literate in Java and Kotlin that in some instances, Kotlin tests will be easier to read. However, we know our current contributors are reasonably fluent in Java. I'm not sure about Kotlin. > It is sad you mention "code readability" as "little benefit currently" item. > > Kevin>surprised that a change went in to > Kevin>switch to Kotlin especially after the discussion that is happening on > the > Kevin>mailing list. > > Those are two different items. The discussion re $ is still open, and > there's no clear answer yet. > At the same time, in the very same thread, there were explicit opinions > that "tests that do not use $ might still benefit from improved > readability". > > The change in [2] is not related to the $-discussion, so I don't see why > people relate them. > > Kevin>I agree that introducing a new > Kevin>language is of little benefit currently > > It is not introducing a brand new language. The very same language is used > in the build scripts. > The language was designed for interoperability with Java, and it does > improve things if used appropriately. > > For instance, tests like > https://github.com/apache/calcite/pull/1657/commits/0d6bec4140da46af07d58cf07a5e682d61529603#diff-7a7027c499b6e4f5e7448b7b971052f1R85-R94 > are > much easier to read and update than similar tests in Java. Assuming you understand, Kotlin, yes. I agree it's not exactly introducing a new language, but I think using Kotlin in tests is quite different from using it in build scripts. We expect most contributors to write tests. If build script editing is a bit less accessible, I think that's less of a concern. > > Note: Kotlin tests are still regular JUnit5 tests, so they get proper > statistics in the CI outputs like test count, skipped test count, failure > count, test duration. > > Kevin>In my opinion there are more negatives than positives > Kevin>currently. > > What your opinion re Kotlin is based on besides angst and "introducing a > new language"? > I can easily relate how the items play for writing tests COBOL, but, well, > I hope we don't consider that :) > > Kotlin is a modern language with good tooling. > It was designed for smooth Java interop which plays well for maintenance. > The code is easy to read, and it is copy-paste friendly in the same way as > almost any other language we currently have in the source repository. > So newbies could copy-paste Kotlin code or Java code or Quidem code. > > For instance, people might still create tests in Java and call methods > written in Kotlin. They might create "pure Java" tests if they like. > They might even create Quidem tests, but they won't be able to call > Java/Kotlin methods (and/or extend classes) then. > > It looks like you express an opinion on "let's rewrite everything in > Kotlin" rather than "let's pick the proper tools on a case by case basis". > > Currently, there are valid reasons against migrating all the tests to > Kotlin, and I guess we have never discussed that. > At the same time, we are not discussing "rewrite everything in Kotlin" here. I don't think anyone was discussing rewriting everything in Kotlin. But I think that's a straw man. Rewriting some things in Kotlin still has some of the same disadvantages. > > Vladimir
Re: [VOTE] Release apache-calcite-avatica-1.16.0 (release candidate 1)
Given the discussion on [1] the problem of building .tar.gz on Windows was anticipated so I am not changing my vote. Nevertheless it might be a bit annoying for Windows users so we have to see how we can improve on this aspect. [1] https://lists.apache.org/thread.html/280c02113c31e7ae2186ee42c1ac1e1010c7390863e791d5778f1578%40%3Cdev.calcite.apache.org%3E On Tue, Dec 17, 2019 at 10:58 AM Danny Chan wrote: > MacOS 10.14, JDK 1.8.0_161, Gradle 6.0.1 > * Checked signatures and checksums OK > * Went over release note OK > * Run build and tests (./gradlew clean build) on git repo OK > * Run build and Calcite tests on Calcite current master (./gradlew clean > build) with Avatica 1.16.0 OK > * Run diff between staged sources and git commit (diff -qr > ~/workspace/avatica-dev ~/Desktop/apache-calcite-avatica-1.16.0-src) ? > > Files /Users/chenyuzhao/workspace/avatica-dev/LICENSE and > /Users/chenyuzhao/Desktop/apache-calcite-avatica-1.16.0-src/LICENSE differ > Only in /Users/chenyuzhao/Desktop/apache-calcite-avatica-1.16.0-src: > licenses > > There are quite a few commits that does not have a JIRA issue to trace, > which is hard to understand, > instead of list the commit message, maybe we should give a broad summary > of them. > > +1 (binding) > > Best, > Danny Chan > 在 2019年12月12日 +0800 AM6:20，Francis Chuang ，写道： > > Hi all, > > > > I have created a build for Apache Calcite Avatica 1.16.0, release > > candidate 1. > > > > Thanks to everyone who has contributed to this release. > > > > You can read the release notes here: > > > https://github.com/apache/calcite-avatica/blob/512bbee4aa24ef9fb8106d0286d1243679dce2d0/site/_docs/history.md > > > > The commit to be voted upon: > > > https://gitbox.apache.org/repos/asf?p=calcite-avatica.git;a=commit;h=512bbee4aa24ef9fb8106d0286d1243679dce2d0 > > > > Its hash is 512bbee4aa24ef9fb8106d0286d1243679dce2d0 > > > > Tag: > > > https://gitbox.apache.org/repos/asf?p=calcite-avatica.git;a=tag;h=refs/tags/avatica-1.16.0-rc1 > > > > The artifacts to be voted on are located here: > > > https://dist.apache.org/repos/dist/dev/calcite/apache-calcite-avatica-1.16.0-rc1 > > (revision 37181) > > > > The hashes of the artifacts are as follows: > > > 102d3ab0e90dd1db5e012a966d265bdfa8a0f24f9016a4187a6e5f0135a14770da124493dd2c7a18c9d8d8b9af5ecf4f5aceb90d48421251f38bc6ce6f5be697 > > *apache-calcite-avatica-1.16.0-src.tar.gz > > > > A staged Maven repository is available for review at: > > > https://repository.apache.org/content/repositories/orgapachecalcite-1071/org/apache/calcite/ > > > > Release artifacts are signed with the following key: > > https://people.apache.org/keys/committer/francischuang.asc > > https://www.apache.org/dist/calcite/KEYS > > > > N.B. > > To create the jars and test Apache Calcite Avatica: "./gradlew build > > -PskipSigning". > > > > If you do not have a Java environment available, you can run the tests > > using docker. To do so, install docker and docker-compose, then run > > "docker-compose run test" from the root of the directory. > > > > Please vote on releasing this package as Apache Calcite Avatica 1.16.0. > > > > The vote is open for the next 72 hours and passes if a majority of at > > least three +1 PMC votes are cast. > > > > [ ] +1 Release this package as Apache Calcite 1.16.0 > > [ ] 0 I don't feel strongly about it, but I'm okay with the release > > [ ] -1 Do not release this package because... > > > > > > Here is my vote: > > > > +1 (binding) > > > > Francis >
Re: [VOTE] Release apache-calcite-avatica-1.16.0 (release candidate 1)
MacOS 10.14, JDK 1.8.0_161, Gradle 6.0.1 * Checked signatures and checksums OK * Went over release note OK * Run build and tests (./gradlew clean build) on git repo OK * Run build and Calcite tests on Calcite current master (./gradlew clean build) with Avatica 1.16.0 OK * Run diff between staged sources and git commit (diff -qr ~/workspace/avatica-dev ~/Desktop/apache-calcite-avatica-1.16.0-src) ? Files /Users/chenyuzhao/workspace/avatica-dev/LICENSE and /Users/chenyuzhao/Desktop/apache-calcite-avatica-1.16.0-src/LICENSE differ Only in /Users/chenyuzhao/Desktop/apache-calcite-avatica-1.16.0-src: licenses There are quite a few commits that does not have a JIRA issue to trace, which is hard to understand, instead of list the commit message, maybe we should give a broad summary of them. +1 (binding) Best, Danny Chan 在 2019年12月12日 +0800 AM6:20，Francis Chuang ，写道： > Hi all, > > I have created a build for Apache Calcite Avatica 1.16.0, release > candidate 1. > > Thanks to everyone who has contributed to this release. > > You can read the release notes here: > https://github.com/apache/calcite-avatica/blob/512bbee4aa24ef9fb8106d0286d1243679dce2d0/site/_docs/history.md > > The commit to be voted upon: > https://gitbox.apache.org/repos/asf?p=calcite-avatica.git;a=commit;h=512bbee4aa24ef9fb8106d0286d1243679dce2d0 > > Its hash is 512bbee4aa24ef9fb8106d0286d1243679dce2d0 > > Tag: > https://gitbox.apache.org/repos/asf?p=calcite-avatica.git;a=tag;h=refs/tags/avatica-1.16.0-rc1 > > The artifacts to be voted on are located here: > https://dist.apache.org/repos/dist/dev/calcite/apache-calcite-avatica-1.16.0-rc1 > (revision 37181) > > The hashes of the artifacts are as follows: > 102d3ab0e90dd1db5e012a966d265bdfa8a0f24f9016a4187a6e5f0135a14770da124493dd2c7a18c9d8d8b9af5ecf4f5aceb90d48421251f38bc6ce6f5be697 > *apache-calcite-avatica-1.16.0-src.tar.gz > > A staged Maven repository is available for review at: > https://repository.apache.org/content/repositories/orgapachecalcite-1071/org/apache/calcite/ > > Release artifacts are signed with the following key: > https://people.apache.org/keys/committer/francischuang.asc > https://www.apache.org/dist/calcite/KEYS > > N.B. > To create the jars and test Apache Calcite Avatica: "./gradlew build > -PskipSigning". > > If you do not have a Java environment available, you can run the tests > using docker. To do so, install docker and docker-compose, then run > "docker-compose run test" from the root of the directory. > > Please vote on releasing this package as Apache Calcite Avatica 1.16.0. > > The vote is open for the next 72 hours and passes if a majority of at > least three +1 PMC votes are cast. > > [ ] +1 Release this package as Apache Calcite 1.16.0 > [ ] 0 I don't feel strongly about it, but I'm okay with the release > [ ] -1 Do not release this package because... > > > Here is my vote: > > +1 (binding) > > Francis
Re: [VOTE] Release apache-calcite-avatica-1.16.0 (release candidate 1)
Contributors and Committers who have voted, please review this issue and
let me know if this changes your vote.
On 17/12/2019 8:07 pm, Vladimir Sitnikov wrote:
https://dist.apache.org/repos/dist/dev/calcite/apache-calcite-avatica-1.16.0-rc1/apache-calcite-avatica-1.16.0-src.tar.gz
does not build if Windows with default settings is used.
The error messages for "./gradlew build" are as follows:
-// dependency on it during compilation\n
-apiv("com.beust:jcommander")\n
... (116 more lines that didn't fit)
Run 'gradlew spotlessApply' to fix these violations.
It means \n files are not expected for Windows.
I'll leave the decision on the severity of that to the RM.
Vladimir
Re: [VOTE] Release apache-calcite-avatica-1.16.0 (release candidate 1)
https://dist.apache.org/repos/dist/dev/calcite/apache-calcite-avatica-1.16.0-rc1/apache-calcite-avatica-1.16.0-src.tar.gz
does not build if Windows with default settings is used.
The error messages for "./gradlew build" are as follows:
-// dependency on it during compilation\n
-apiv("com.beust:jcommander")\n
... (116 more lines that didn't fit)
Run 'gradlew spotlessApply' to fix these violations.
It means \n files are not expected for Windows.
I'll leave the decision on the severity of that to the RM.
Vladimir
Re: [DISCUSS] Remove Kotlin
Kevin>Focusing on the technical side of things, I agree that introducing a new Kevin>language is of little benefit currently Kevin, what is your opinion on removing Quidem language? Focusing on the technical side, it is a standalone language. The language is not Java, it has limited tooling, it has a limited set of users, it has 0 or so questions on StackOverflow and so on. Kevin>I agree that introducing a new Kevin>language is of little benefit currently Keeping tests easy to read, easy to edit, easy to create, easy to debug, and easy to maintain is hard. Making tests easy to read simplifies code reviews that pay off in the long run. It is sad you mention "code readability" as "little benefit currently" item. Kevin>surprised that a change went in to Kevin>switch to Kotlin especially after the discussion that is happening on the Kevin>mailing list. Those are two different items. The discussion re $ is still open, and there's no clear answer yet. At the same time, in the very same thread, there were explicit opinions that "tests that do not use $ might still benefit from improved readability". The change in [2] is not related to the $-discussion, so I don't see why people relate them. Kevin>I agree that introducing a new Kevin>language is of little benefit currently It is not introducing a brand new language. The very same language is used in the build scripts. The language was designed for interoperability with Java, and it does improve things if used appropriately. For instance, tests like https://github.com/apache/calcite/pull/1657/commits/0d6bec4140da46af07d58cf07a5e682d61529603#diff-7a7027c499b6e4f5e7448b7b971052f1R85-R94 are much easier to read and update than similar tests in Java. Note: Kotlin tests are still regular JUnit5 tests, so they get proper statistics in the CI outputs like test count, skipped test count, failure count, test duration. Kevin>In my opinion there are more negatives than positives Kevin>currently. What your opinion re Kotlin is based on besides angst and "introducing a new language"? I can easily relate how the items play for writing tests COBOL, but, well, I hope we don't consider that :) Kotlin is a modern language with good tooling. It was designed for smooth Java interop which plays well for maintenance. The code is easy to read, and it is copy-paste friendly in the same way as almost any other language we currently have in the source repository. So newbies could copy-paste Kotlin code or Java code or Quidem code. For instance, people might still create tests in Java and call methods written in Kotlin. They might create "pure Java" tests if they like. They might even create Quidem tests, but they won't be able to call Java/Kotlin methods (and/or extend classes) then. It looks like you express an opinion on "let's rewrite everything in Kotlin" rather than "let's pick the proper tools on a case by case basis". Currently, there are valid reasons against migrating all the tests to Kotlin, and I guess we have never discussed that. At the same time, we are not discussing "rewrite everything in Kotlin" here. Vladimir
