Message-
From: Roman Kondakov
Sent: Monday, November 18, 2019 11:04 AM
To: dev@ignite.apache.org
Subject: Re: New SQL execution engine
Hi, Steve
This behavior is actually not a bug, but this is not obvious. I'll try to
explain.
When query parallelism = N is turned on, it means that each
Hi, Steve
This behavior is actually not a bug, but this is not obvious. I'll try
to explain.
When query parallelism = N is turned on, it means that each cache is
divided into N parts from the SQL point of view. Every SQL query is
executed independently over each particular part, and then
:13 AM
To: dev
Subject: Re: New SQL execution engine
Steve,
Yep, unfortunately query parallelism in current flavor is counter-intuitive.
But it was designed so =( As Roman wrote
> And of course this feature should also be available in the new engine, though
> it's architecture may be c
Steve,
Yep, unfortunately query parallelism in current flavor is
counter-intuitive. But it was designed so =( As Roman wrote
> And of course this feature should also be available in the new engine, though
> it's architecture may be changed.
The architecture of parallel execution will be
Actually I am now wondering whether this is not just a bug and that I should
record it as such. As the behavior is different with and without the
parallelism and there is no warning during execution or in the api.
Any thought?
--
Sent from:
not return the
same result. I understand that it probably because I did not set an affinity
but it is very counter-intuitive.
Am I missing something?
-Original Message-
From: Roman Kondakov
Sent: Friday, November 15, 2019 11:46 AM
To: dev@ignite.apache.org
Subject: Re: New SQL execution
Hi Steve,
it is possible to execute queries in parallel even in the current
engine, see docs here [1]. And of course this feature should also be
available in the new engine, though it's architecture may be changed.
[1]
Dear all,
would it be possible to also have then // execution of sql queries on single
node with that approach?
My understanding is that, for the moment, the SQL queries a re
single-threaded for a given node if there is no affinity.
Best Regards
--
Sent from:
Nikolay,
Guys updated IEP [1]. Could you please check it? Are there any missing
parts needed at that stage?
[1]
https://cwiki.apache.org/confluence/display/IGNITE/IEP-37%3A+New+query+execution+engine
вт, 1 окт. 2019 г. в 12:19, Ivan Pavlukhin :
>
> Folks,
>
> I marked IEP-33 as obsolete. Also
Nikolay,
The document you edited is wrong (and outdated).
Since the author meant another idea, I decided not to change IEP-35 and
create a new one - IEP-37 (https://cwiki.apache.org/confluence/x/NBLABw).
It's already have a number of key requirements.
Regards,
Igor
вт, 1 окт. 2019 г., 6:14
Hello, Igniters.
I extends IEP [1] with the tickets caused by H2 limitations.
Please, let's write down requirements for engine in the IEP.
https://cwiki.apache.org/confluence/display/IGNITE/IEP-33%3A+New+SQL+executor+engine+infrastructure
В Пн, 30/09/2019 в 17:20 -0700, Denis Magda пишет:
>
Ivan, we need more of these discussions, totally agree with you ;)
I've updated the Motivation paragraph outlining some high-level users we
see by working with our users. Hope it helps. Let's carry on and let me
send a note to Apache Calcite community.
-
Denis
On Mon, Sep 30, 2019 at 1:56 AM
Folks,
Thanks everyone for a hot discussion! Not every open source community
has such open and boiling discussions. It means that people here
really do care. And I am proud of it!
As I understood, nobody is strictly against the proposed initiative.
And I am glad that we can move forward (with
Hello, Denis.
Thanks for the clarifications.
Sounds good for me.
All I try to say in this thread:
Guys, please, let's take a step back and write down requirements(what we want
to get with SQL engine).
Which features and use-cases are primary for us.
I'm sure you have done it, already during
> I think, we should discuss the idea in general.
Everybody likes the idea so far :)
The issues in details, as usual.
В Пт, 27/09/2019 в 19:03 +0300, Seliverstov Igor пишет:
> Nikolay,
>
> > What project hosted Calcite based engine?
>
>
> Currently the prototype is placed in my personal
Ignite mates, let me try to move the discussion in a constructive way. It
looks like we set a wrong context from the very beginning.
Before proposing this idea to the community, some of us were
discussing/researching the topic in different groups (the one need to think
it through first before
Nikolay,
> What project hosted Calcite based engine?
Currently the prototype is placed in my personal Ignite fork. I need an
appropriate ticket before pushing it to ASF git repository.
At first, I think, we should discuss the idea in general.
> Personally, I'm against the support of two
Igor.
> There is no decision, here we should decide.
Great.
> At now Calcite based engine is placed in different module
What project hosted Calcite based engine?
> It’s possible to develop it as an experimental extension at first (not a
> replacement)
For me, Ignite 3 are the place where
Hello, Andrey.
Thanks, it's more clear now.
> I agree, we should make IEP clear to everyone in community who want to be
> involved in IEP implementation at first.
Great!
Looking forward for IEP clarification.
В Пт, 27/09/2019 в 18:07 +0300, Andrey Mashenkov пишет:
> Nikolay, Igor.
>
>
Folks,
especially Ignite PMCs,
Are there any plans about how Ignite SQL will be evolved? It is a very
interesting thread on how Ignite SQL as a product will be developed
for the near future e.g. supporting new standards etc.
According to documentation Ignite complies with SQL ANSI-99 [2] but in
Nikolay,
At last we have better questions.
There is no decision, here we should decide.
Doing nothing isn’t a decision, it’s just doing nothing
Spark Catalyst is a good example, but under the hood it has absolutely the same
idea, but adopted to Spark. Calcite is the same, but general. That’s
Nikolay, Igor.
Implementing from scratch is an option, of course.
If we decide to go this way then we definitely won't to spend long nights
to invent "yet another SQL parser" with all the stuff related to query
rewrite rules (e.g. IN -> JOIN) or type casting \ validation \ conversion.
We thought
Igor.
> The main issue - there is no *selection*.
1. I don't remember community decision about this.
2. We should avoid to make such long-term decision so quickly.
We done this kind of decision with H2 and come to the point when we should
review it.
> 1) Implementing white papers from scratch
Thanks, Andrey!
Will take a loo, shortly.
В Пт, 27/09/2019 в 17:19 +0300, Andrey Mashenkov пишет:
> Issues can't be resolved without changes in H2.
> Hope, this will be enough.
>
> https://issues.apache.org/jira/browse/IGNITE-10598
> https://issues.apache.org/jira/browse/IGNITE-11473
>
Issues can't be resolved without changes in H2.
Hope, this will be enough.
https://issues.apache.org/jira/browse/IGNITE-10598
https://issues.apache.org/jira/browse/IGNITE-11473
https://issues.apache.org/jira/browse/IGNITE-11444
https://issues.apache.org/jira/browse/IGNITE-5289
Nikolay,
The main issue - there is no *selection*.
There is a field of knowledge - relational algebra, which describes how to
transform relational expressions saving their semantics, and a couple of
implementations (Calcite is only one written in Java).
There are only two alternatives:
1)
Roman.
> Nikolay, Maxim, I understand that our arguments may not be as obvious
> for you as it obvious for SQL team. So, please arrange your questions in
> a more constructive way.
What is SQL team?
I only know Ignite community :)
Please, share you knowledge in IEP.
I want to join to the
Maxim, Nikolay,
I've listed two issues which show the ideological flaws of the current
engine.
1. IGNITE-11448 - Open. This ticket describes the impossibility of
executing queries which can not be fit in the hardcoded one pass
map-reduce paradigm.
2. IGNITE-6085 - Closed (won't fix) -
Hello, Alexey.
Thanks for the details.
> Now, as for alternatives for Apache Calcite
I want to discuss our *requirements* for the new engine first.
Can we do it?
The main reason to do it - We should avoid wrong technical decision.
We made one with H2 and we shouldn't do it again.
> As for the
Nikolay, Maxim,
Asking to provide a list of issues with the current H2 is pointless because
it has a fundamental architectural flow, not just a bunch of bugs:
Currently, the query execution is limited to a two-phase map-reduce task
(with an optional remote cursor when 'distributed joins' flag is
Folks,
I agree with Nikolay, the idea of replacing the H2 engine with the
most suitable one is reasonable. But since such change is major we
should have a strong argumentation on it even for members with are
working outside the SQL-team.
I think it is really necessary to have:
1. The list of
Hello, Roman.
All I see is links to two tickets:
IGNITE-11448 - Open
IGNITE-6085 - Closed
Other issues described poorly and have not ticket links.
We can't discuss such a huge change as an execution engine replacement with
descrition like:
"No data co-location control, i.e. arbitrary data can
Hello Nikolay,
please see IEP--37 [1]. Issues are there.
[1]
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=130028084
--
Kind Regards
Roman Kondakov
On 27.09.2019 14:20, Nikolay Izhikov wrote:
Hello, Roman.
Also Apache Calcite is commonly used in popular Apache
Hello, Roman.
> Also Apache Calcite is commonly used in popular Apache projects
I don't think it's a good point.
H2 is also commonly used.
But, it doesn't conform to Ignite requirements.
Can you, please, write down issues and engine requirements to the IEP?
So we can discuss each point
Hello, Andrey.
> Ignite SQL layer has some issues that can't be fix with changes in Ignite
> only, and we are blocked with H2.
What are these issues?
Can you make it specific and send a tickets for this issues?
> 3. Replace H2 with smth else.
Actually, I support this decision in general.
But,
Hello Nikolay.
You've asked very good questions. I'll try to answer.
1. What the exact issues with the H2 integration?
Can you send a tickets links?
Can we label all H2 integration issues in JIRA? I propose to use "h2" label.
Current SQL engine is confined in the single-pass map-reduce
Hi Nikolay,
Let me add my 5- cent here.
Ignite SQL layer has some issues that can't be fix with changes in Ignite
only, and we are blocked with H2.
To resolve these issues we can:
1. Donate some changes to H2 and wait for it's next release. But there are
more cons than pros and I think we can't
Hi Igor!
In my opinion using Apache Calcite for distributed SQL query
optimization and planning is much more promising approach than using H2.
H2 is not suitable for distributed query execution and also it has very
limited abilities for query optimization. While Apache Calcite is the
open
Hello, Igor.
Thanks for starting this discussion.
I think we should take a step back in it and answer the following questions:
1. What the exact issues with the H2 integration?
Can you send a tickets links?
Can we label all H2 integration issues in JIRA? I propose to use "h2" label.
2. What
Hi Igniters!
As you might know currently we have many open issues relating to current H2
based engine and its execution flow.
Some of them are critical (like impossibility to execute particular queries),
some of them are majors (like impossibility to execute particular queries
without
40 matches
Mail list logo
