This reminds me that there’s an opportunity to make it easier to write adapters. There is a continuum of backends for adapters, from simple (such as CSV, which supports project and maybe filter) to powerful (such as Elasticsearch, which supports full relational algebra and then some). I am thinking about the latter case.
Let’s suppose we are writing an adapter for a database Foo which is approximately as powerful as SQL. Typically we write such adapters by creating a Foo convention, then a FooToEnumerableConverter, and then write rules to push each relational operator down through the membrane. (Elasticsearch, Cassandra and Druid are all in this category, and all use the basic ‘membrane’ principle, but it is worrying that their adapters are so dissimilar.) If Foo has ten relational operators, you need to write ten RelNode classes and ten rules. That is a considerable amount of code, which ends up looking like boilerplate. I wonder whether we can remove the need to create those Foo-specific classes and rules. Could people just write a FooQuery relational expression that had a sequence of lightweight relational operations inside it? Note that in MutableRel we have lightweight equivalents to RelNodes, for example MutableProject [1], and they could be repurposed. The adapter would still have some hard work to do: translating RexNode to their expression language (splitting or rejecting expressions that they can’t handle), figuring out which sequences of relational operations are valid, and translating the FooQuery. But if we reduce the boilerplate, the adapter author can focus on the hard parts. Julian [1] https://github.com/apache/calcite/blob/4bc916619fd286b2c0cc4d5c653c96a68801d74e/core/src/main/java/org/apache/calcite/rel/mutable/MutableProject.java#L34 > On Jan 23, 2022, at 12:28 PM, Stamatis Zampetakis <zabe...@gmail.com> wrote: > > To the best of my knowledge there is no document getting into details about > the Queryable interface etc. The best way to approach this as Gavin > mentioned is to see how it is used in the project and outside of it. > > One thing that I wanted to mention is that you can build a system using > Calcite without using these interfaces at all. If you consider for instance > the druid module in Calcite I think it doesn't rely on Queryable at all. > > Best, > Stamatis > > > On Wed, Jan 19, 2022 at 4:00 PM Gavin Ray <ray.gavi...@gmail.com> wrote: > >> My apologies, I didn't realize that the updated search wasn't public now >> (it opened for me a few days ago) >> >> Regular GH links: >> https://github.com/search?q=rawqueryable+language%3AJava&type=code >> >> Searching "CalciteConnection" across all repos, but removing >> "apache/calcite" from the results: >> >> https://github.com/search?q=CalciteConnection+-repo%3Aapache%2Fcalcite&type=code >> >> >> >> On Tue, Jan 18, 2022 at 12:17 PM M Singh <mans2si...@yahoo.com.invalid> >> wrote: >> >>> Thanks Gavin for your pointers. I am in the waiting list for the search >>> tool you mentioned. I will continue to search on google in the >> meanwhile. >>> Mans >>> >>> >>> On Tuesday, January 18, 2022, 08:44:31 AM EST, Gavin Ray < >>> ray.gavi...@gmail.com> wrote: >>> >>> Not to be dismissive, but one thing I've done that has been helpful for >>> me, >>> is to search all public repos for usage/examples of the thing I am trying >>> to understand. >>> For example, this "RawQueryable" does not appear to be utilized in any >>> user-facing code on Github, so it may not be useful for you: >>> >>> >> https://cs.github.com/?q=rawqueryable%20language%3AJava&scopeName=All%20repos&scope= >>> >>> However, take something like "CalciteConnection" or "RelNode", try to >>> remove the Calcite repo itself, and you'll get a lot of hits and usage >>> examples: >>> >>> >> https://cs.github.com/?scopeName=All+repos&scope=&q=CalciteConnection+%28NOT+path%3A%22calcite%2F%22%29 >>> >>> (I find that searching Scala repos usually gives more relevant results, >> due >>> to lots of Java results being duplicates) >>> >>> This isn't nearly as good as getting a direct answer, but sometimes you >>> might be able to find what you're looking for >>> >>> >>> >>> >>> On Tue, Jan 18, 2022 at 5:39 AM M Singh <mans2si...@yahoo.com.invalid> >>> wrote: >>> >>>> Hey Folks: >>>> Just wanted to see if you have any pointers to help me understand the >>>> queryable concepts. >>>> Thanks >>>> On Friday, January 14, 2022, 09:09:06 AM EST, M Singh >>>> <mans2si...@yahoo.com.invalid> wrote: >>>> >>>> Hi Folks: >>>> A few newbie questions - >>>> 1. I wanted to find out what are the benefits of a queryable table ( >>>> >>> >> https://calcite.apache.org/javadocAggregate/org/apache/calcite/schema/QueryableTable.html >> ) >>> and >>>> what are the pros and cons of using it. >>>> 2. What is the role of Querable interface ( >>>> >>> >> https://calcite.apache.org/javadocAggregate/org/apache/calcite/linq4j/Queryable.html >>> ) >>>> ?3. There is a getExpression methods in queryabletable ( >>>> >>> >> https://calcite.apache.org/javadocAggregate/org/apache/calcite/linq4j/tree/Expression.html >>> ), >>>> and I wanted to find out if it is required and what are the pros/cons >> of >>>> implementing it. >>>> 4. What role does RawQueralbe play ? ( >>>> >>> >> https://calcite.apache.org/javadocAggregate/org/apache/calcite/linq4j/RawQueryable.html >>> ). >>>> It has a method getProvider method that returns QueryProvider. What >>> roles >>>> does query provider play ?5. What role does ExtendedQuerable play ? >>>> Please let me know if there are any basic >>>> examples/blogs/documentation/etc. >>>> Thanks for your help. >>>> Mans >>> >>
