Re: RECURSIVE keyword not supported?
Hey Ruben, Stamatis, Thanks a lot, please find a PR for that doc change here: https://github.com/apache/calcite/pull/3081 Happy to make any further changes as you see fit. Best, --Gunnar Am Mi., 22. Feb. 2023 um 10:58 Uhr schrieb Stamatis Zampetakis < zabe...@gmail.com>: > Echoing Ruben; all contributions are welcome and much appreciated > especially when it comes to documentation that can be easily left behind! > > Best, > Stamatis > > On Wed, Feb 22, 2023 at 9:53 AM Ruben Q L wrote: > > > Hello Gunnar, > > > > IIRC the idea when that doc [1] was written was providing an alternative > > syntax that people would understand (since probably the RelBuilder > example > > would be too abstract or complex for most users). > > Perhaps as it is written right now it can be misleading and we should > > clarify that at the moment SQL is not supported for recursive queries, > and > > it is only shown for illustrative purposes. > > Contributions are always welcome, including the documentation! Feel free > to > > open a PR to improve this part, I'll happily take a look at it. > > > > Best, > > Ruben > > > > [1] https://github.com/apache/calcite/blob/main/site/_docs/algebra.md > > > > > > On Wed, Feb 22, 2023 at 7:32 AM Gunnar Morling > > wrote: > > > > > Hey Julien, > > > > > > Am Di., 21. Feb. 2023 um 23:01 Uhr schrieb Julian Hyde < > jh...@apache.org > > >: > > > > > > > A classic problem of open source development. Should we accept a > > > > contribution for recursive queries even though it doesn't have SQL > > > > support? And if we choose to accept it without SQL support, do we > show > > > > the equivalent SQL in our documentation? > > > > > > > > > > Yes, the struggle is real, and there's no simple answer. Being a > > maintainer > > > of multiple open-source projects myself, I very much can relate to > that. > > > FWIW, I think you made the right call by accepting the feature in its > > > current state. It then comes down to expectation management, i.e. being > > > very transparent about what is supported and what isn't. > > > > > > > > > > > I agree that the documentation is a bit misleading. The caveats about > > > > "experimental APIs" mostly cover it, but it should say that "WITH > > > > RECURSIVE" SQL is not currently supported. But you are also > > > > complaining that the free lunch has french fries and baked potatoes > > > > but no mashed potatoes. > > > > > > > > > > Ugh, now that's a bit harsh, don't you think? I've asked for > > clarification > > > about how to use this feature, and when learning that right now it > isn't, > > > I've offered to help clarify that fact in the doc (as much as I'd love > > help > > > building the actual feature, it's just beyond the scope of what my time > > > currently allows for, unfortunately). I think that hardly qualifies as > > > "complaining". > > > > > > Best, > > > > > > --Gunnar > > > > > > Julian > > > > > > > > On Tue, Feb 21, 2023 at 1:06 PM Gunnar Morling > > > > wrote: > > > > > > > > > > Hey Ruben, > > > > > > > > > > Thanks a lot for the quick response! In that light, isn't having > that > > > SQL > > > > > example in the docs a bit confusing then? Might be worth clarifying > > for > > > > the > > > > > time being that this is a hypothetical syntax and recursive queries > > are > > > > > only supported in the RelBuilder API? Happy to send a PR for doing > > that > > > > doc > > > > > update, if you think it makes sense and can point me to the source > of > > > > that > > > > > page. > > > > > > > > > > Thanks again, > > > > > > > > > > --Gunnar > > > > > > > > > > > > > > > Am Di., 21. Feb. 2023 um 20:56 Uhr schrieb Ruben Q L < > > > rube...@gmail.com > > > > >: > > > > > > > > > > > Hello Gunnar, > > > > > > > > > > > > At the moment, recursive queries are only supported via > RelBuilder > > > > API, not > > > > > > via SQL (the documen
Re: RECURSIVE keyword not supported?
Hey Julien, Am Di., 21. Feb. 2023 um 23:01 Uhr schrieb Julian Hyde : > A classic problem of open source development. Should we accept a > contribution for recursive queries even though it doesn't have SQL > support? And if we choose to accept it without SQL support, do we show > the equivalent SQL in our documentation? > Yes, the struggle is real, and there's no simple answer. Being a maintainer of multiple open-source projects myself, I very much can relate to that. FWIW, I think you made the right call by accepting the feature in its current state. It then comes down to expectation management, i.e. being very transparent about what is supported and what isn't. > > I agree that the documentation is a bit misleading. The caveats about > "experimental APIs" mostly cover it, but it should say that "WITH > RECURSIVE" SQL is not currently supported. But you are also > complaining that the free lunch has french fries and baked potatoes > but no mashed potatoes. > Ugh, now that's a bit harsh, don't you think? I've asked for clarification about how to use this feature, and when learning that right now it isn't, I've offered to help clarify that fact in the doc (as much as I'd love help building the actual feature, it's just beyond the scope of what my time currently allows for, unfortunately). I think that hardly qualifies as "complaining". Best, --Gunnar Julian > > On Tue, Feb 21, 2023 at 1:06 PM Gunnar Morling > wrote: > > > > Hey Ruben, > > > > Thanks a lot for the quick response! In that light, isn't having that SQL > > example in the docs a bit confusing then? Might be worth clarifying for > the > > time being that this is a hypothetical syntax and recursive queries are > > only supported in the RelBuilder API? Happy to send a PR for doing that > doc > > update, if you think it makes sense and can point me to the source of > that > > page. > > > > Thanks again, > > > > --Gunnar > > > > > > Am Di., 21. Feb. 2023 um 20:56 Uhr schrieb Ruben Q L >: > > > > > Hello Gunnar, > > > > > > At the moment, recursive queries are only supported via RelBuilder > API, not > > > via SQL (the documentation shows the SQL equivalent to the RelBuider > code > > > just below, but Calcite currently only supports the latter, not the > > > former). > > > > > > There is an old ticket about supporting this feature in SQL [1] , but > so > > > far no work has been done in this regard. > > > > > > Best regards, > > > Ruben > > > > > > [1] https://issues.apache.org/jira/browse/CALCITE-129 > > > > > > > > > On Tue, Feb 21, 2023 at 7:23 PM Gunnar Morling > > > wrote: > > > > > > > Hi, > > > > > > > > I'm trying to run the recursive example from the Calcite algebra docs > > > [1], > > > > but I'm always getting an error at the RECURSIVE keyword: > > > > > > > > WITH RECURSIVE aux(i) AS ( > > > > VALUES (1) > > > > UNION ALL > > > > SELECT i+1 FROM aux WHERE i < 10 > > > > ) > > > > SELECT * FROM aux > > > > > > > > "Incorrect syntax near the keyword 'RECURSIVE' at line 1, column > 6" > > > > > > > > I saw this is considered an experimental feature for the time being, > is > > > > there some flag/option I need to specify when creating my connection > to > > > > enable support for recursive queries? > > > > > > > > Thanks a lot for any help, > > > > > > > > --Gunnar > > > > > > > > [1] https://calcite.apache.org/docs/algebra.html#recursive-queries > > > > > > > >
Re: RECURSIVE keyword not supported?
Hey Ruben, Thanks a lot for the quick response! In that light, isn't having that SQL example in the docs a bit confusing then? Might be worth clarifying for the time being that this is a hypothetical syntax and recursive queries are only supported in the RelBuilder API? Happy to send a PR for doing that doc update, if you think it makes sense and can point me to the source of that page. Thanks again, --Gunnar Am Di., 21. Feb. 2023 um 20:56 Uhr schrieb Ruben Q L : > Hello Gunnar, > > At the moment, recursive queries are only supported via RelBuilder API, not > via SQL (the documentation shows the SQL equivalent to the RelBuider code > just below, but Calcite currently only supports the latter, not the > former). > > There is an old ticket about supporting this feature in SQL [1] , but so > far no work has been done in this regard. > > Best regards, > Ruben > > [1] https://issues.apache.org/jira/browse/CALCITE-129 > > > On Tue, Feb 21, 2023 at 7:23 PM Gunnar Morling > wrote: > > > Hi, > > > > I'm trying to run the recursive example from the Calcite algebra docs > [1], > > but I'm always getting an error at the RECURSIVE keyword: > > > > WITH RECURSIVE aux(i) AS ( > > VALUES (1) > > UNION ALL > > SELECT i+1 FROM aux WHERE i < 10 > > ) > > SELECT * FROM aux > > > > "Incorrect syntax near the keyword 'RECURSIVE' at line 1, column 6" > > > > I saw this is considered an experimental feature for the time being, is > > there some flag/option I need to specify when creating my connection to > > enable support for recursive queries? > > > > Thanks a lot for any help, > > > > --Gunnar > > > > [1] https://calcite.apache.org/docs/algebra.html#recursive-queries > > >
RECURSIVE keyword not supported?
Hi, I'm trying to run the recursive example from the Calcite algebra docs [1], but I'm always getting an error at the RECURSIVE keyword: WITH RECURSIVE aux(i) AS ( VALUES (1) UNION ALL SELECT i+1 FROM aux WHERE i < 10 ) SELECT * FROM aux "Incorrect syntax near the keyword 'RECURSIVE' at line 1, column 6" I saw this is considered an experimental feature for the time being, is there some flag/option I need to specify when creating my connection to enable support for recursive queries? Thanks a lot for any help, --Gunnar [1] https://calcite.apache.org/docs/algebra.html#recursive-queries
Visualizing Calcite data in Apache Superset
Hey all, I'm looking into ways of visualizing data provided via Calcite using Apache Superset. For this, I think I'd have to expose some sort of endpoint which Superset can connect to, potentially implementing an existing wire protocol, such as the one of Postgres. That way, Superset could connect to my Calcite-based application just as to any other Postgres application. Has anyone given something like this a try? Thanks for any hints, --Gunnar
Re: Extend parser.jj in calcite, to support the operator '<=>'
> We’d want to support it in Babel but not the core parser. I've come across that differentiation a few times when reading earlier e-mails. Is there any resource explaining the difference between these two parsers, what they are used for, and which sort of change should be done in which? Thanks a lot, --Gunnar Am Fr., 7. Jan. 2022 um 20:58 Uhr schrieb Julian Hyde < jhyde.apa...@gmail.com>: > It seems analogous to the Postgres-style ‘::’ cast operator. We’d want to > support it in Babel but not the core parser. Can you log a JIRA case > analogous to https://issues.apache.org/jira/browse/CALCITE-2843 < > https://issues.apache.org/jira/browse/CALCITE-2843>. > > > On Jan 7, 2022, at 3:06 AM, 徐仁和 wrote: > > > > Hi calcite community. > > > > In my case, we need calcite's parser support operator '<=>' of mysql. > > > > Link: > > > https://dev.mysql.com/doc/refman/8.0/en/comparison-operators.html#operator_equal-to > > > > I debug it, I find I need to modify the Parser.jj in calcite, to define a > > new token for '<=>'. > > > > So, could I expand the Parser.jj of calcite, and support this new > operator. > > > > Best, > > XuRenhe > >
Re: [DISCUSS] SBOM (Software Bill of Materials)
Am Di., 4. Jan. 2022 um 21:41 Uhr schrieb Julian Hyde < jhyde.apa...@gmail.com>: > No, I don’t think it matters in this case. But consistent use of ROOT is > useful, because someone in future might be tracking down a bug, and if they > see ENGLISH it’s one more hypothesis they’d have to discount. > That makes sense. Just let me know if you see the need for other changes to that PR. I may look into some of the other dependencies you mentioned as being rarely used, as I find the time. Any thoughts on this one: > Re Janino, is there any reason for not using the compiler implementation coming with the JDK ? Thanks, --Gunnar > > On Jan 4, 2022, at 12:31 PM, Gunnar Morling > wrote: > > > > Am Di., 4. Jan. 2022 um 20:51 Uhr schrieb Julian Hyde < > > jhyde.apa...@gmail.com>: > > > >> If a method needs a locale, always pass Locale.ROOT. > >> > > > > Ok, I've changed it accordingly. Do you think it actually matters for the > > case at hand? > > > >> On Jan 4, 2022, at 9:13 AM, Gunnar Morling > >> wrote: > >>> > >>> Am Di., 4. Jan. 2022 um 09:39 Uhr schrieb Julian Hyde < > >>> jhyde.apa...@gmail.com>: > >>> > >>>> Please log a jira case for the commons-lang3 change. > >>> > >>> > >>> Logged https://issues.apache.org/jira/browse/CALCITE-4975. > >>> > >>> > >>>> It looks good. One or two places I’d create a function rather than > >> having > >>>> a blob of code inline. > >>>> > >>> > >>> Sure; just let me know where exactly. > >>> > >>> Your use of default locale in the CSV adapter looks wrong. Calcite is a > >>>> server, so never uses default locale or time zone. In fact we use > >>>> forbiddenApis to check, so we should add a few methods to its > >> configuration. > >>> > >>> > >>> Yeah, I had been pondering about this; I don't think it matters, the > >> locale > >>> should not make any difference for these specific formats, as they > don't > >>> contain any locale-specific patterns (unlike, say, "MMM"). I've changed > >> it > >>> to Locale.ENGLISH now, just in case. In fact, I wanted to use the > >>> ofPattern() method without the Locale parameter, but this failed the > >>> forbiddenApis check as well :) > >>> > >>> Julian > >>> > >>> > >>> Best, > >>> > >>> --Gunnar > >>> > >>> > >>>> > >>>> > >>>>> On Jan 3, 2022, at 12:30 PM, Gunnar Morling > >>>> wrote: > >>>>> > >>>>> Hi, > >>>>> > >>>>> Thanks a lot for this, I think trimming down the dependencies of > >> Calcite > >>>>> will be of great help for its adoption. > >>>>> > >>>>>> So, the easiest way to reduce dependencies would be to make certain > >>>>> classes of SQL functions optional (i.e. move them out of core). > >>>>> > >>>>> That sounds like a good idea. > >>>>> > >>>>>> commons-lang3, commons-codec, commons-io are probably only used in > one > >>>> or > >>>>> two places each; > >>>>> > >>>>> To make some progress there, I've created PR > >>>>> https://github.com/apache/calcite/pull/2672 which removes the > >>>> dependency to > >>>>> commons-lang3 from the entire code base. Any feedback on that PR > would > >>>>> be appreciated (I still need to log an issue, but wanted to share > >> quickly > >>>>> what I had). I can try and take a look at the other ones, if there's > >>>>> interest in this. > >>>>> > >>>>> Re Janino, is there any reason for not using the compiler > >> implementation > >>>>> coming with the JDK? Alternatively, one could also consider to > generate > >>>>> byte code directly using ASM, which wouldn't be beneficial > >>>> dependency-wise, > >>>>> but it may improve the performance of this generation step (I still > >> lack > >>>>> insight why this is done in the first place). > >>>>> > >>>>> Thanks, > >>>>&
Re: [DISCUSS] SBOM (Software Bill of Materials)
Am Di., 4. Jan. 2022 um 20:51 Uhr schrieb Julian Hyde <
jhyde.apa...@gmail.com>:
> If a method needs a locale, always pass Locale.ROOT.
>
Ok, I've changed it accordingly. Do you think it actually matters for the
case at hand?
> On Jan 4, 2022, at 9:13 AM, Gunnar Morling
> wrote:
> >
> > Am Di., 4. Jan. 2022 um 09:39 Uhr schrieb Julian Hyde <
> > jhyde.apa...@gmail.com>:
> >
> >> Please log a jira case for the commons-lang3 change.
> >
> >
> > Logged https://issues.apache.org/jira/browse/CALCITE-4975.
> >
> >
> >> It looks good. One or two places I’d create a function rather than
> having
> >> a blob of code inline.
> >>
> >
> > Sure; just let me know where exactly.
> >
> > Your use of default locale in the CSV adapter looks wrong. Calcite is a
> >> server, so never uses default locale or time zone. In fact we use
> >> forbiddenApis to check, so we should add a few methods to its
> configuration.
> >
> >
> > Yeah, I had been pondering about this; I don't think it matters, the
> locale
> > should not make any difference for these specific formats, as they don't
> > contain any locale-specific patterns (unlike, say, "MMM"). I've changed
> it
> > to Locale.ENGLISH now, just in case. In fact, I wanted to use the
> > ofPattern() method without the Locale parameter, but this failed the
> > forbiddenApis check as well :)
> >
> > Julian
> >
> >
> > Best,
> >
> > --Gunnar
> >
> >
> >>
> >>
> >>> On Jan 3, 2022, at 12:30 PM, Gunnar Morling
> >> wrote:
> >>>
> >>> Hi,
> >>>
> >>> Thanks a lot for this, I think trimming down the dependencies of
> Calcite
> >>> will be of great help for its adoption.
> >>>
> >>>> So, the easiest way to reduce dependencies would be to make certain
> >>> classes of SQL functions optional (i.e. move them out of core).
> >>>
> >>> That sounds like a good idea.
> >>>
> >>>> commons-lang3, commons-codec, commons-io are probably only used in one
> >> or
> >>> two places each;
> >>>
> >>> To make some progress there, I've created PR
> >>> https://github.com/apache/calcite/pull/2672 which removes the
> >> dependency to
> >>> commons-lang3 from the entire code base. Any feedback on that PR would
> >>> be appreciated (I still need to log an issue, but wanted to share
> quickly
> >>> what I had). I can try and take a look at the other ones, if there's
> >>> interest in this.
> >>>
> >>> Re Janino, is there any reason for not using the compiler
> implementation
> >>> coming with the JDK? Alternatively, one could also consider to generate
> >>> byte code directly using ASM, which wouldn't be beneficial
> >> dependency-wise,
> >>> but it may improve the performance of this generation step (I still
> lack
> >>> insight why this is done in the first place).
> >>>
> >>> Thanks,
> >>>
> >>> --Gunnar
> >>>
> >>>> Am Fr., 31. Dez. 2021 um 00:56 Uhr schrieb Julian Hyde <
> >>>> jhyde.apa...@gmail.com>:
> >>>>
> >>>> Regarding dependencies. Here are the runtime dependencies from
> >>>> core/build.gradle.kts (ignoring test and annotation libraries):
> >>>>
> >>>> * api("com.esri.geometry:esri-geometry-api")
> >>>> * api("com.fasterxml.jackson.core:jackson-annotations")
> >>>> * api("com.google.guava:guava")
> >>>> * api("org.apache.calcite.avatica:avatica-core")
> >>>> * api("org.slf4j:slf4j-api")
> >>>> * implementation("com.fasterxml.jackson.core:jackson-core")
> >>>> * implementation("com.fasterxml.jackson.core:jackson-databind")
> >>>> *
> >>>>
> >>
> implementation("com.fasterxml.jackson.dataformat:jackson-dataformat-yaml")
> >>>> * implementation("com.google.uzaygezen:uzaygezen-core")
> >>>> * implementation("com.jayway.jsonpath:json-path")
> >>>> * implementation("com.yahoo.datasketches:sketches-core")
> >>>> * implementation("commons-codec:commons-codec")
> >>>> * implementation("net.hydromatic:aggdesigne
Re: [DISCUSS] SBOM (Software Bill of Materials)
Am Di., 4. Jan. 2022 um 09:39 Uhr schrieb Julian Hyde <
jhyde.apa...@gmail.com>:
> Please log a jira case for the commons-lang3 change.
Logged https://issues.apache.org/jira/browse/CALCITE-4975.
> It looks good. One or two places I’d create a function rather than having
> a blob of code inline.
>
Sure; just let me know where exactly.
Your use of default locale in the CSV adapter looks wrong. Calcite is a
> server, so never uses default locale or time zone. In fact we use
> forbiddenApis to check, so we should add a few methods to its configuration.
Yeah, I had been pondering about this; I don't think it matters, the locale
should not make any difference for these specific formats, as they don't
contain any locale-specific patterns (unlike, say, "MMM"). I've changed it
to Locale.ENGLISH now, just in case. In fact, I wanted to use the
ofPattern() method without the Locale parameter, but this failed the
forbiddenApis check as well :)
Julian
Best,
--Gunnar
>
>
> > On Jan 3, 2022, at 12:30 PM, Gunnar Morling
> wrote:
> >
> > Hi,
> >
> > Thanks a lot for this, I think trimming down the dependencies of Calcite
> > will be of great help for its adoption.
> >
> >> So, the easiest way to reduce dependencies would be to make certain
> > classes of SQL functions optional (i.e. move them out of core).
> >
> > That sounds like a good idea.
> >
> >> commons-lang3, commons-codec, commons-io are probably only used in one
> or
> > two places each;
> >
> > To make some progress there, I've created PR
> > https://github.com/apache/calcite/pull/2672 which removes the
> dependency to
> > commons-lang3 from the entire code base. Any feedback on that PR would
> > be appreciated (I still need to log an issue, but wanted to share quickly
> > what I had). I can try and take a look at the other ones, if there's
> > interest in this.
> >
> > Re Janino, is there any reason for not using the compiler implementation
> > coming with the JDK? Alternatively, one could also consider to generate
> > byte code directly using ASM, which wouldn't be beneficial
> dependency-wise,
> > but it may improve the performance of this generation step (I still lack
> > insight why this is done in the first place).
> >
> > Thanks,
> >
> > --Gunnar
> >
> >> Am Fr., 31. Dez. 2021 um 00:56 Uhr schrieb Julian Hyde <
> >> jhyde.apa...@gmail.com>:
> >>
> >> Regarding dependencies. Here are the runtime dependencies from
> >> core/build.gradle.kts (ignoring test and annotation libraries):
> >>
> >> * api("com.esri.geometry:esri-geometry-api")
> >> * api("com.fasterxml.jackson.core:jackson-annotations")
> >> * api("com.google.guava:guava")
> >> * api("org.apache.calcite.avatica:avatica-core")
> >> * api("org.slf4j:slf4j-api")
> >> * implementation("com.fasterxml.jackson.core:jackson-core")
> >> * implementation("com.fasterxml.jackson.core:jackson-databind")
> >> *
> >>
> implementation("com.fasterxml.jackson.dataformat:jackson-dataformat-yaml")
> >> * implementation("com.google.uzaygezen:uzaygezen-core")
> >> * implementation("com.jayway.jsonpath:json-path")
> >> * implementation("com.yahoo.datasketches:sketches-core")
> >> * implementation("commons-codec:commons-codec")
> >> * implementation("net.hydromatic:aggdesigner-algorithm")
> >> * implementation("org.apache.commons:commons-dbcp2")
> >> * implementation("org.apache.commons:commons-lang3")
> >> * implementation("commons-io:commons-io")
> >> * implementation("org.codehaus.janino:commons-compiler")
> >> * implementation("org.codehaus.janino:janino")
> >>
> >> A few libraries are used only for a narrow range of functionality:
> >> * esri-geometry and uzaygezen-core are used by geospatial functions;
> >> * sketches-core is used by the HLL aggregate functions;
> >> * json-path is used by some JSON functions;
> >> * jackson-core, jackson-databind, jackson-dataformat-yaml are used to
> >> load models, and to serialize RelNodes to and from JSON;
> >> * commons-lang3, commons-codec, commons-io are probably only used in one
> >> or two places each;
> >> * aggdesigner-algotihm is used for recommending materialized views.
> >>
> >> So, the easiest way to reduce dependencies would be to make certain
> >> cla
[jira] [Created] (CALCITE-4975) Remove dependency to commons-lang3
Gunnar Morling created CALCITE-4975: --- Summary: Remove dependency to commons-lang3 Key: CALCITE-4975 URL: https://issues.apache.org/jira/browse/CALCITE-4975 Project: Calcite Issue Type: Improvement Reporter: Gunnar Morling It's used only in a few places, so the dependency can easily be removed, reducing the number of transitive dependencies introduced to an application via Calcite. -- This message was sent by Atlassian Jira (v8.20.1#820001)
Re: [DISCUSS] SBOM (Software Bill of Materials)
Hi,
Thanks a lot for this, I think trimming down the dependencies of Calcite
will be of great help for its adoption.
> So, the easiest way to reduce dependencies would be to make certain
classes of SQL functions optional (i.e. move them out of core).
That sounds like a good idea.
> commons-lang3, commons-codec, commons-io are probably only used in one or
two places each;
To make some progress there, I've created PR
https://github.com/apache/calcite/pull/2672 which removes the dependency to
commons-lang3 from the entire code base. Any feedback on that PR would
be appreciated (I still need to log an issue, but wanted to share quickly
what I had). I can try and take a look at the other ones, if there's
interest in this.
Re Janino, is there any reason for not using the compiler implementation
coming with the JDK? Alternatively, one could also consider to generate
byte code directly using ASM, which wouldn't be beneficial dependency-wise,
but it may improve the performance of this generation step (I still lack
insight why this is done in the first place).
Thanks,
--Gunnar
Am Fr., 31. Dez. 2021 um 00:56 Uhr schrieb Julian Hyde <
jhyde.apa...@gmail.com>:
> Regarding dependencies. Here are the runtime dependencies from
> core/build.gradle.kts (ignoring test and annotation libraries):
>
> * api("com.esri.geometry:esri-geometry-api")
> * api("com.fasterxml.jackson.core:jackson-annotations")
> * api("com.google.guava:guava")
> * api("org.apache.calcite.avatica:avatica-core")
> * api("org.slf4j:slf4j-api")
> * implementation("com.fasterxml.jackson.core:jackson-core")
> * implementation("com.fasterxml.jackson.core:jackson-databind")
> *
> implementation("com.fasterxml.jackson.dataformat:jackson-dataformat-yaml")
> * implementation("com.google.uzaygezen:uzaygezen-core")
> * implementation("com.jayway.jsonpath:json-path")
> * implementation("com.yahoo.datasketches:sketches-core")
> * implementation("commons-codec:commons-codec")
> * implementation("net.hydromatic:aggdesigner-algorithm")
> * implementation("org.apache.commons:commons-dbcp2")
> * implementation("org.apache.commons:commons-lang3")
> * implementation("commons-io:commons-io")
> * implementation("org.codehaus.janino:commons-compiler")
> * implementation("org.codehaus.janino:janino")
>
> A few libraries are used only for a narrow range of functionality:
> * esri-geometry and uzaygezen-core are used by geospatial functions;
> * sketches-core is used by the HLL aggregate functions;
> * json-path is used by some JSON functions;
> * jackson-core, jackson-databind, jackson-dataformat-yaml are used to
> load models, and to serialize RelNodes to and from JSON;
> * commons-lang3, commons-codec, commons-io are probably only used in one
> or two places each;
> * aggdesigner-algotihm is used for recommending materialized views.
>
> So, the easiest way to reduce dependencies would be to make certain
> classes of SQL functions optional (i.e. move them out of core).
>
> Julian
>
>
>
> > On Dec 29, 2021, at 1:30 PM, Jacques Nadeau wrote:
> >
> > WRT SBOM (Julian): My general experience is that most large orgs use
> > scanners now (either open or closed) and they will scan whether you have
> a
> > bill of materials or not. I wouldn't worry about adding something
> > additional.
> >
> > WRT too many dependencies (Gunnar): I completely agree with the general
> > feeling of too many (and with Guava, jackson less so). I think the core
> > challenge (no pun intended) is that calcite-core is really a lot of
> > different components. For example, I have frequently wished that parser,
> > planner and enumerable were separate modules. And if they were, I'd guess
> > that each would have a narrower dependency range. I've also wished many
> > times that runtime compilation was an optional addon as opposed to
> > required/coupled in the core...
> >
> > When I've thought about how to dissect in the past, I think the big
> > challenge would be tests, where things are sometimes mixed together.
> > Breaking change possibilities could be at least somewhat mitigated by
> > moving classes but not packages.
> >
> > On Wed, Dec 29, 2021 at 1:51 AM Gunnar Morling
> > wrote:
> >
> >> Hi,
> >>
> >> In a way, Calcite's build configuration as well as the published POM
> could
> >> be considered as such an SBOM? In particular when looking at the latter
> >> through services like mvnrepository [1], you get quite a good view on
&
Re: Trying to call "PreparedStatement.run()" throws "Cannot invoke "calcite.schema.Schema.getTable(String)" because the return value of "calcite.schema.SchemaPlus.getSubSchema(String)" is null"
Am Mo., 3. Jan. 2022 um 20:44 Uhr schrieb Gavin Ray : > @Stamatis > > Thank you, I actually did not know that! That's useful information to > have =) > > @Julian > > Is it possible to programmatically generate the JSON model? Why I ask is > because part of the functionality I'm trying to mirror is the ability to > add > data sources dynamically while the app server is running > > So you might send a POST /metadata/datasource saying "Here is the info of a > new > data source I would like you to connect to" And then I need to figure out > how to > handle creating a Calcite source from it and adding the new datasource + > schemas > to the global pool of registered ones, if that makes sense > > Worst case I could write the JSON file to disk and load from the tempfile I > suppose, right? > No need for writing to disk, you just can pass in a String representing the JSON model. Here's an example: https://github.com/moditect/jfr-analytics/blob/main/src/test/java/org/moditect/jfranalytics/JfrSchemaFactoryTest.java#L381-L388 Hth, --Gunnar > > > > On Mon, Jan 3, 2022 at 1:30 PM Julian Hyde wrote: > > > > Will try to figure out how to manually perform the process of creating > a > > > proper "Connection" object and adding the schemas to it. > > > > > > Generally the best way is to write a JSON model, and then create a > > connection using ‘jdbc:calcite:…model=...'. For that to work, your > > component needs to implement ‘interface SchemaFactory’, i.e. be an > adapter. > > > > Julian > > > > >
Re: [DISCUSS] SBOM (Software Bill of Materials)
Hi, In a way, Calcite's build configuration as well as the published POM could be considered as such an SBOM? In particular when looking at the latter through services like mvnrepository [1], you get quite a good view on the dependency versions, licenses, any potential CVEs, etc. I think this should satisfy most user needs around this? Or are you referring to the notion of Maven BOM POMs specifically [2], i.e. the notion of publishing a POM with all the Calcite component versions which people can then use with Maven's import scope (there should be something comparable for Gradle)? If so, that could be useful for users working with multiple Calcite components, though I think the usability improvement provided by such BOM POM wouldn't be huge. I wanted to bring up a related matter though. Coming to Calcite as a user just recently (loving the possibilities it provides!), I was surprised by the large number of dependencies of the project. It looks like 1.29 improves that a little bit (no more kotlin-stdlib, no more transitive dependency to log4j 1.x), but the transitive hull of all dependencies of calcite-core still is quite big. I lack insight about what the different dependencies are used for; but as an application developer, Guava for instance is a dependency which I'd prefer to not get pushed onto the classpath transitively. Jackson is another heavy one; depending on how it's used, perhaps this could be pushed into some separate module which users could optionally pull in? That'd help to avoid having it around when users work with other JSON libs themselves and don't require JSON support in Calcite. >From a supply chain perspective, the less transitive dependencies a library like Calcite introduces to my project, the better IMHO. Less potential for version conflicts with my own (or other transitive) dependencies, and also less potential for introducing CVEs to the dependency graph, as e.g. in the case of the Guava version currently used by Calcite; I suppose it does not impact the usage in Calcite, but these things tend to be tricky to reason about, and typical CVE reporting tooling will now create a warning for a project using Calcite, no matter whether that specific issue actually is a problem or not. Best, --Gunnar [1] https://mvnrepository.com/artifact/org.apache.calcite/calcite-core/1.29.0 [2] https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html#bill-of-materials-bom-poms Am Mi., 29. Dez. 2021 um 02:27 Uhr schrieb Julian Hyde < jhyde.apa...@gmail.com>: > In the wake of the log4j CVEs [1], people are asking how to improve the > security of open source projects, and one idea is to provide a SBOM > (Software Bill of Materials) [2] along with each release. > > I had not heard of SBOM until a couple of days ago. Is anyone on this list > familiar with SBOMs and their use? Should Calcite be providing an SBOM? Are > people aware of SBOM initiatives in other projects? What, in your opinion, > is the priority of this issue? > > Julian > > [1] > https://thehackernews.com/2021/12/second-log4j-vulnerability-cve-2021.html > > [2] https://en.wikipedia.org/wiki/Software_bill_of_materials >
Re: How to obtain metadata for tables in sub-schemas?
Am Di., 28. Dez. 2021 um 23:13 Uhr schrieb Julian Hyde <
jhyde.apa...@gmail.com>:
> That looks right. So we may have a bug. Also try
>
> md.getTables(null, null, "%", new String[] {“TABLE”}));
>
> and
>
> md.getTables(“", null, "%", new String[] {“TABLE”}));
>
Same result, no sub-schema tables are returned.
I've logged https://issues.apache.org/jira/browse/CALCITE-4969 for this.
Julian
>
Thanks,
--Gunnar
>
>
> > On Dec 28, 2021, at 1:45 PM, Gunnar Morling
> wrote:
> >
> > Hi,
> >
> > I have created a custom Calcite schema factory which produces a parent
> > schema which in turn contains multiple sub-schemas. I can query those
> > tables as intended, e.g. SELECT * FROM schemaname.tablename.
> >
> > I am struggling though to obtain all the tables via JDBC metadata, the
> > following doesn't return any subschema tables (note how the schema filter
> > is null):
> >
> > DatabaseMetaData md = connection.getMetaData();
> > ResultSet rs = md.getTables(null, null, "%", null));
> >
> > Is this an expected behavior? If so, what would I have to do in order to
> > obtain the metadata for tables in any sub-schema? I don't think I should
> > return any sub-schema tables from getTable() in the parent schema, or
> > should I?
> >
> > Thanks a lot for any advice,
> >
> > --Gunnar
>
>
[jira] [Created] (CALCITE-4969) Can't retrieve metadata for tables in sub-schemas
Gunnar Morling created CALCITE-4969:
---
Summary: Can't retrieve metadata for tables in sub-schemas
Key: CALCITE-4969
URL: https://issues.apache.org/jira/browse/CALCITE-4969
Project: Calcite
Issue Type: Bug
Affects Versions: 1.28.0
Reporter: Gunnar Morling
I have created a custom Calcite schema factory which produces a parent schema,
which in turn contains multiple sub-schemas. I can query those tables as
intended, e.g. SELECT * FROM schemaname.tablename.
Metadata retrieval for these tables via JDBC fails though; the following
doesn't return any subschema tables (note how the schema filter is null):
{code}
DatabaseMetaData md = connection.getMetaData();
ResultSet rs = md.getTables(null, null, "%", null));
{code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
