The guys at Contiamo [1] are doing similar stuff. I added the nice talk [2] by Chris in ApacheCon to our website.
[1] https://www.contiamo.com/ [2] https://youtu.be/4JAOkLKrcYE On Tue, Dec 10, 2019 at 3:05 AM Julian Hyde <jh...@apache.org> wrote: > Yes, virtualization is one of Calcite’s goals. In fact, when I created > Calcite I was thinking about virtualization + in-memory materialized views. > Not only the Spark convention but any of the “engine” conventions (Drill, > Flink, Beam, Enumerable) could be used to create a virtual query engine. > > See e.g. a talk I gave in 2013 about Optiq (precursor to Calcite) > https://www.slideshare.net/julianhyde/optiq-a-dynamic-data-management-framework > < > https://www.slideshare.net/julianhyde/optiq-a-dynamic-data-management-framework > >. > > Julian > > > > > On Dec 9, 2019, at 2:29 PM, Muhammad Gelbana <mgelb...@apache.org> > wrote: > > > > I recently contacted one of the active contributors asking about the > > purpose of the project and here's his reply: > > > > From my understanding, Quicksql is a data virtualization platform. It can > >> query multiple data sources altogether and in a distributed way; Say, > you > >> can write a SQL with a MySql table join with an Elasticsearch table. > >> Quicksql can recognize that, and then generate Spark code, in which it > will > >> fetch the MySQL/ES data as a temporary table separately, and then join > them > >> in Spark. The execution is in Spark so it is totally distributed. The > user > >> doesn't need to aware of where the table is from. > >> > > > > I understand that the Spark convention Calcite has attempts to achieve > the > > same goal, but it isn't fully implemented yet. > > > > > > On Tue, Oct 29, 2019 at 9:43 PM Julian Hyde <jh...@apache.org> wrote: > > > >> Anyone know anything about Quicksql? It seems to be quite a popular > >> project, and they have an internal fork of Calcite. > >> > >> https://github.com/Qihoo360/ <https://github.com/Qihoo360/> > >> > >> > >> > https://github.com/Qihoo360/Quicksql/tree/master/analysis/src/main/java/org/apache/calcite > >> < > >> > https://github.com/Qihoo360/Quicksql/tree/master/analysis/src/main/java/org/apache/calcite > >>> > >> > >> Julian > >> > >> > >
