Hi, everyone:
Glad to see a lot of old friends here. Quicksql is a project born in early
2019. It was designed to solve the problem of long and complex work flow in
the big data field with many data sources, many compute engines, and many
types of syntax. The core idea is `Connect All Data Sources with One Extra
Parsing Cost`.
Because it involves standard SQL parsing, we finally chose Calcite as the
parsing engine that has the best SQL compatibility. Thanks to the excellent
architecture and toolkits provided by Calcite, Quicksql has made some
extensions on this basis and made more logical plans Rich definitions
enable single data source and multi-source queries to be described. For
single data sources, an end-to-end connection query is directly
established, and for multiple data sources, logical plans are divided and
pushed down, final interpreted as the code of the compute engine (such as
Spark, Flink) with distributed computing capabilities for data merge.
Based on this design, Quicksql makes extensive use of the ability of
Calcite Adapter \ Dialect \ UDF to provide syntax adaptation compatibility
for various data sources and compute engines, and also uses Avatica as a
JDBC protocol. We are very grateful for the excellent artwork provided by
the Calcite community.
At the beginning of the project, Quicksql was confused about the
application areas. After one year of polishing, Quicksql has successfully
applied two areas:
1. Interactive Query Engine: Provides big data interactive query and BI
analysis with standard SQL syntax, and response time is in seconds to
minutes.
2. ETL Compute Engine: SQL-based ETL for multi-data source, which can use
optimization capabilities of SQL for data cleaning \ transformation \ join,
etc.
In the future, we will also focus on dynamic engine selection, so that
engines such as Hive, Spark, and Presto can run more suitable SQL.
Looking forward to working with the Calcite community to do some
interesting things and explore the unlimited possibilities of SQL
Siyuan Liu
On Mon, Mar 2, 2020 at 3:45 PM Francis Du wrote:
> Hi everyone:
>
> Allow me to introduce my good friend Siyuan Liu, who is the leader of
> Quicksql project.
>
> I CC to him and ask him to introduce the project to us.Here is the
> documentation link for
>
> Quicksql [1].
>
> [1]. https://quicksql.readthedocs.io/en/latest/
>
> Regards,
> Francis
>
> Juan Pan 于2019年12月23日周一 上午11:44写道：
>
>> Thanks Gelbana,
>>
>>
>> Very appreciated your explanation, which sheds me some light on exploring
>> Calcite. :)
>>
>>
>> Best wishes,
>> Trista
>>
>>
>> Juan Pan (Trista)
>>
>> Senior DBA & PPMC of Apache ShardingSphere(Incubating)
>> E-mail: panj...@apache.org
>>
>>
>>
>>
>> On 12/22/2019 05:58，Muhammad Gelbana wrote：
>> I am curious how to join the tables from different datasources.
>> Based on Calcite's conventions concept, the Join operator and its input
>> operators should all have the same convention. If they don't, the
>> convention different from the Join operator's convention will have to
>> register a converter rule. This rule should produce an operator that only
>> converts from that convention to the Join operator's convention.
>>
>> This way the Join operator will be able to handle the data obtained from
>> its input operators because it understands the data structure.
>>
>> Thanks,
>> Gelbana
>>
>>
>> On Wed, Dec 18, 2019 at 5:08 AM Juan Pan wrote:
>>
>> Some updates.
>>
>>
>> Recently i took a look at their doc and source code, and found this
>> project uses SQL parsing and Relational algebra of Calcite to get query
>> plan, and also translates to spark SQL for joining different datasources,
>> or corresponding query for single datasource.
>>
>>
>> Although it copies many classes from Calcite, the idea of QuickSQL seems
>> some of interests, and code is succinct.
>>
>>
>> Best,
>> Trista
>>
>>
>> Juan Pan (Trista)
>>
>> Senior DBA & PPMC of Apache ShardingSphere(Incubating)
>> E-mail: panj...@apache.org
>>
>>
>>
>>
>> On 12/13/2019 17:16，Juan Pan wrote：
>> Yes, indeed.
>>
>>
>> Juan Pan (Trista)
>>
>> Senior DBA & PPMC of Apache ShardingSphere(Incubating)
>> E-mail: panj...@apache.org
>>
>>
>>
>>
>> On 12/12/2019 18:00，Alessandro Solimando
>> wrote：
>> Adapters must be needed by data sources not supporting SQL, I think this
>> is
>> what Juan Pan was asking for.
>>
>> On Thu, 12 Dec 2019 at 04:05, Haisheng Yuan wrote:
>>
>> Nope
