Alexey, Nikolay, Thank you for sharing details!
вт, 1 окт. 2019 г. в 07:42, Alexey Zinoviev <zaleslaw....@gmail.com>: > > Great talk and paper, I've learnt it last year > > пн, 30 сент. 2019 г., 21:42 Nikolay Izhikov <nizhi...@apache.org>: > > > Yes, I can :) > > > > В Пн, 30/09/2019 в 11:40 -0700, Denis Magda пишет: > > > Nikolay, > > > > > > Would you be able to review the changes? I'm not sure there is a better > > candidate for now. > > > > > > - > > > Denis > > > > > > > > > On Mon, Sep 30, 2019 at 11:01 AM Nikolay Izhikov <nizhi...@apache.org> > > wrote: > > > > Hello, Ivan. > > > > > > > > I had a talk about internals of Spark integration in Ignite. > > > > It answers on question why we should use Spark internals. > > > > > > > > You can take a look at my meetup talk(in Russian) [1] or read an > > article if you prefer text [2]. > > > > > > > > [1] https://www.youtube.com/watch?v=CzbAweNKEVY > > > > [2] https://habr.com/ru/company/sberbank/blog/427297/ > > > > > > > > В Пн, 30/09/2019 в 20:29 +0300, Alexey Zinoviev пишет: > > > > > Yes, as I understand it uses Spark internals from the first commit))) > > > > > The reason - we take Spark SQL query execution plan and try to > > execute it > > > > > on Ignite cluster > > > > > Also we inherit a lot of Developer API related classes that could be > > > > > unstable. Spark has no good point for extension and this is a reason > > why we > > > > > should go deeper > > > > > > > > > > пн, 30 сент. 2019 г. в 20:17, Ivan Pavlukhin <vololo...@gmail.com>: > > > > > > > > > > > Hi Alexey, > > > > > > > > > > > > As an external watcher very far from Ignite Spark integration I > > would > > > > > > like to ask a humble question for my understanding. Why this > > > > > > integration uses Spark internals? Is it a common approach for > > > > > > integrating with Spark? > > > > > > > > > > > > пн, 30 сент. 2019 г. в 16:17, Alexey Zinoviev < > > zaleslaw....@gmail.com>: > > > > > > > > > > > > > > Hi, Igniters > > > > > > > I've started the work on the Spark 2.4 support > > > > > > > > > > > > > > We started the discussion here, in > > > > > > > https://issues.apache.org/jira/browse/IGNITE-12054 > > > > > > > > > > > > > > The Spark internals were totally refactored between 2.3 and 2.4 > > versions, > > > > > > > main changes touches > > > > > > > > > > > > > > - External catalog and listeners refactoring > > > > > > > - Changes of HAVING operator semantic support > > > > > > > - Push-down NULL filters generation in JOIN plans > > > > > > > - minor changes in Plan Generation that should be adopted in > > our > > > > > > > integration module > > > > > > > > > > > > > > I propose the initial solution here via creation of new module > > spark-2.4 > > > > > > > here https://issues.apache.org/jira/browse/IGNITE-12247 and > > addition of > > > > > > > > > > > > new > > > > > > > profile spark-2.4 (to avoid possible clashes with another spark > > versions) > > > > > > > > > > > > > > Also I've transformed ticket to an Umbrella ticket and created a > > few > > > > > > > tickets for muted tests (around 7 from 211 tests are muted now) > > > > > > > > > > > > > > Please, if somebody interested in it, make an initial review of > > modular > > > > > > > ignite structure and changes (without deep diving into Spark > > code). > > > > > > > > > > > > > > And yes, the proposed code is a copy-paste of spark-ignite > > module with a > > > > > > > few fixes > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Best regards, > > > > > > Ivan Pavlukhin > > > > > > > > -- Best regards, Ivan Pavlukhin
