Denis, Would be nice to see real use-cases of affinity call + local SQL combination. Generally, new engine will be able to infer collocation resulting in the same collocated execution automatically.
пт, 1 нояб. 2019 г. в 19:11, Denis Magda <dma...@apache.org>: > > Hi Igor, > > Local queries feature is broadly used together with affinity-based compute > tasks: > https://apacheignite.readme.io/docs/collocate-compute-and-data#section-affinity-call-and-run-methods > > The use case is as follows. The user knows that all required data needed > for computation is collocated, and SQL is used as an advanced API for data > retrieval from the computation code. The affinity task ensures that > partitions won't be discarded from the node(s) if the topology changes > during the task execution and, thus, it's safe to run SQL locally skipping > distributed phases. > > The combination of affinity compute tasks with local SQL is a real and > valuable use case, and this is what we need to support with Calcite. Do you > see any challenges? > > - > Denis > > > On Fri, Nov 1, 2019 at 8:46 AM Roman Kondakov <kondako...@mail.ru.invalid> > wrote: > > > Hi Igor! > > > > IMO we need to maintain the backward compatibility between old and new > > query engines as much as possible. And therefore we shouldn't change the > > behavior of local queries. > > > > So, for local queries Calcite's planner shouldn't consider the > > distribution trait at all. > > > > > > -- > > Kind Regards > > Roman Kondakov > > > > On 01.11.2019 17:07, Seliverstov Igor wrote: > > > Hi Igniters, > > > > > > Working on new generation of Ignite SQL I faced a question: «Do we need > > local queries at all and, if so, what semantic they should have?». > > > > > > Current planing flow consists of next steps: > > > > > > 1) Parsing SQL to AST > > > 2) Validating AST (against Schema) > > > 3) Optimizing (Building execution graph) > > > 4) Splitting (into query fragments which executes on target nodes) > > > 5) Mapping (query fragments to nodes/partitions) > > > > > > At last step we check that all Fragment sources (a table or result) have > > the same distribution (in other words all sources have to be co-located) > > > > > > Planner and Splitter guarantee that all caches in a Fragment are > > co-located, an Exchange is produced otherwise. But if we force local > > execution we cannot produce Exchanges, that means we may face two > > non-co-located caches inside a single query fragment (result of local query > > planning is a single query fragment). So, we cannot pass the check. > > > > > > Should we throw an exception or omit the check for local query planning > > or prohibit local queries at all? > > > > > > Your thoughts? > > > > > > Regards, > > > Igor > > -- Best regards, Ivan Pavlukhin
