Hello, Alexey. Can we somehow highlight changes in Spark-2.4 module comparing to 2.3 one? For now the changes look too huge for me (+11,681 −1).
Are we sure we want to add those huge piece of code to support two versions? Can we extract unchanged parts(based on spark public API) and keep them in one copy? > 18 нояб. 2019 г., в 23:47, Denis Magda <dma...@apache.org> написал(а): > > Alexey, thanks for the details and for reaching out this milestone with the > 2.4 support. > > Generally, I would advise us to merge the changes to the master only after > we confirm the failing tests are not regressions. We should either remove > them or replace them with some others or just fix. > > - > Denis > > > On Mon, Nov 18, 2019 at 10:06 AM Alexey Zinoviev <zaleslaw....@gmail.com> > wrote: > >> Right, a few tests from 200 are failed due to known issue and couldnt be >> fixed immediately, related to rare cases. These tests are copies of 2.3 >> tests and part of them could have no meaning for 2.4 due to Spark changed >> behaviour. >> >> пн, 18 нояб. 2019 г., 19:42 Denis Magda <dma...@apache.org>: >> >>> Alexey, >>> >>> Please help to understand what it means that 2.4 integration supports >> "95% >>> of tests of 2.3". Does it mean that 5% of existing tests are failing and, >>> basically, need to be fixed? >>> >>> - >>> Denis >>> >>> >>> On Mon, Nov 18, 2019 at 6:52 AM Alexey Zinoviev <zaleslaw....@gmail.com> >>> wrote: >>> >>>> Dear Nikolay Izhikov, I've recreated the PR for 2.4 initial support >>>> >>>> The last commit >>>> >>>> >>> >> https://github.com/apache/ignite/pull/7058/commits/60386802299deedc6ed60bf4736e922201a67fb8 >>>> contains >>>> real changes from Spark 2.3 >>>> >>>> I suggest to merge to master this initial solution with 95% support of >>>> Spark 2.4 and continue work on known issues listed in JIRA >>>> >>>> This solution supports the new Spark version for all examples and 95% >> of >>>> tests of 2.3. >>>> >>>> вт, 1 окт. 2019 г. в 08:48, Ivan Pavlukhin <vololo...@gmail.com>: >>>> >>>>> Alexey, Nikolay, >>>>> >>>>> Thank you for sharing details! >>>>> >>>>> вт, 1 окт. 2019 г. в 07:42, Alexey Zinoviev <zaleslaw....@gmail.com >>> : >>>>>> >>>>>> Great talk and paper, I've learnt it last year >>>>>> >>>>>> пн, 30 сент. 2019 г., 21:42 Nikolay Izhikov <nizhi...@apache.org>: >>>>>> >>>>>>> Yes, I can :) >>>>>>> >>>>>>> В Пн, 30/09/2019 в 11:40 -0700, Denis Magda пишет: >>>>>>>> Nikolay, >>>>>>>> >>>>>>>> Would you be able to review the changes? I'm not sure there is >> a >>>>> better >>>>>>> candidate for now. >>>>>>>> >>>>>>>> - >>>>>>>> Denis >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Sep 30, 2019 at 11:01 AM Nikolay Izhikov < >>>>> nizhi...@apache.org> >>>>>>> wrote: >>>>>>>>> Hello, Ivan. >>>>>>>>> >>>>>>>>> I had a talk about internals of Spark integration in Ignite. >>>>>>>>> It answers on question why we should use Spark internals. >>>>>>>>> >>>>>>>>> You can take a look at my meetup talk(in Russian) [1] or read >>> an >>>>>>> article if you prefer text [2]. >>>>>>>>> >>>>>>>>> [1] https://www.youtube.com/watch?v=CzbAweNKEVY >>>>>>>>> [2] https://habr.com/ru/company/sberbank/blog/427297/ >>>>>>>>> >>>>>>>>> В Пн, 30/09/2019 в 20:29 +0300, Alexey Zinoviev пишет: >>>>>>>>>> Yes, as I understand it uses Spark internals from the first >>>>> commit))) >>>>>>>>>> The reason - we take Spark SQL query execution plan and try >>> to >>>>>>> execute it >>>>>>>>>> on Ignite cluster >>>>>>>>>> Also we inherit a lot of Developer API related classes that >>>>> could be >>>>>>>>>> unstable. Spark has no good point for extension and this >> is a >>>>> reason >>>>>>> why we >>>>>>>>>> should go deeper >>>>>>>>>> >>>>>>>>>> пн, 30 сент. 2019 г. в 20:17, Ivan Pavlukhin < >>>>> vololo...@gmail.com>: >>>>>>>>>> >>>>>>>>>>> Hi Alexey, >>>>>>>>>>> >>>>>>>>>>> As an external watcher very far from Ignite Spark >>>> integration I >>>>>>> would >>>>>>>>>>> like to ask a humble question for my understanding. Why >>> this >>>>>>>>>>> integration uses Spark internals? Is it a common approach >>> for >>>>>>>>>>> integrating with Spark? >>>>>>>>>>> >>>>>>>>>>> пн, 30 сент. 2019 г. в 16:17, Alexey Zinoviev < >>>>>>> zaleslaw....@gmail.com>: >>>>>>>>>>>> >>>>>>>>>>>> Hi, Igniters >>>>>>>>>>>> I've started the work on the Spark 2.4 support >>>>>>>>>>>> >>>>>>>>>>>> We started the discussion here, in >>>>>>>>>>>> https://issues.apache.org/jira/browse/IGNITE-12054 >>>>>>>>>>>> >>>>>>>>>>>> The Spark internals were totally refactored between 2.3 >>> and >>>>> 2.4 >>>>>>> versions, >>>>>>>>>>>> main changes touches >>>>>>>>>>>> >>>>>>>>>>>> - External catalog and listeners refactoring >>>>>>>>>>>> - Changes of HAVING operator semantic support >>>>>>>>>>>> - Push-down NULL filters generation in JOIN plans >>>>>>>>>>>> - minor changes in Plan Generation that should be >>>> adopted >>>>> in >>>>>>> our >>>>>>>>>>>> integration module >>>>>>>>>>>> >>>>>>>>>>>> I propose the initial solution here via creation of new >>>>> module >>>>>>> spark-2.4 >>>>>>>>>>>> here >> https://issues.apache.org/jira/browse/IGNITE-12247 >>>> and >>>>>>> addition of >>>>>>>>>>> >>>>>>>>>>> new >>>>>>>>>>>> profile spark-2.4 (to avoid possible clashes with >> another >>>>> spark >>>>>>> versions) >>>>>>>>>>>> >>>>>>>>>>>> Also I've transformed ticket to an Umbrella ticket and >>>>> created a >>>>>>> few >>>>>>>>>>>> tickets for muted tests (around 7 from 211 tests are >>> muted >>>>> now) >>>>>>>>>>>> >>>>>>>>>>>> Please, if somebody interested in it, make an initial >>>> review >>>>> of >>>>>>> modular >>>>>>>>>>>> ignite structure and changes (without deep diving into >>>> Spark >>>>>>> code). >>>>>>>>>>>> >>>>>>>>>>>> And yes, the proposed code is a copy-paste of >>> spark-ignite >>>>>>> module with a >>>>>>>>>>>> few fixes >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Best regards, >>>>>>>>>>> Ivan Pavlukhin >>>>>>>>>>> >>>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Best regards, >>>>> Ivan Pavlukhin >>>>> >>>> >>> >>
