> P.S.: I'd like to take your feedback serious, the AIP process description in Confluence just tells: "Once you or someone else feels like there’s a rough consensus on the idea and there’s no strong opposition, you can move your proposal to the Vote phase." - neither this nor the structure template mention that a technical spec or PR must be provided prior vote. If you feel that an AIP should include this then I assume the contribution docs need to be adjusted.
Just one - comment here (I am at Pycon US now). I think it's hard to describe in detail the requirements because it will be more of a case-by-case, but I think you should simply compare this with other AIPs where a lot of details have been worked out before voting - often with code snippets, or detailed description of the APIs or communication involved. It's just difficult to understand what exactly changes it will bring - will it use AIP-44 or not? - not clear, will it change how the heartbeat will work ? How are we going to handle retries, what the API proposal will be in general (snippets?). Even if it is not a final version, but first "pass" of how roughly all those things will look like - even if not working, what changes it will impose on each component is usually what you will find in recently created AIPs. I think a LOT depends on the implementation choices here - including whether we **really** want to do it in 2.* or maybe that is simply something that should be Airflow 3 - because we will find out that what is needed to implement is basically already available in what we will have in Airflow 3 or because we will be able to implement is in a much simpler way in Airflow 3. Or maybe Airflow 3 will implement all the needs there simply "via" task isolation. And I know it means "delay" - (and for example waiting for the detailed Task Isolation proposal that Ash works on and that will undoubtedly contain a lot of details) - but this is also exactly the reason why i put on hold AIP-67 "Multi-team" - even if I had it much more detailed and knew what it means for most of the internals, because I think we should put it in perspective of Airflow 3 vision and decide: do we want it in Airflow 2 ? Maybe differently in Airflow 2 and Airflow 3 or Airflow 2 only? I do not know. In the case of this AIP, the thing is that it is too high level to understand how it maps into changes to the components. How things will change and which components are impacted here ? I have no idea. will we change "local task" execution and heartbeat and status updates / retry/ execution semantics that are currently all run (and very tightly coupled with how "local task" implementation works"). Actually one reason why Airflow 3 was a "thing" was that we need to remove a lot of coupling from "local task" and "Airflow DB" to make it easier and less of a maintenance burden (and breaking compatibility of how tasks work). And this is why AIP-44 went in the other direction - don't change anything, just wrap DB calls in gRPc (but keep compatibility on how tasks work and interact with Airflow). And to be honest - from the current AIP it's entirely unclear whether it is closer to AIP-44 or future isolation where the API will be completely new. So I simply do not know what I am voting on. Maybe to put it in short terms - I see AIP voting not voting on "Idea" but voting on "Implementation proposal" - and for me, the current status is that it is an "Idea" but a lot of Implementation proposal needs to be hashed out. J. On Sat, May 18, 2024 at 2:45 PM Scheffler Jens (XC-AS/EAE-ADA-T) <jens.scheff...@de.bosch.com.invalid> wrote: > Hi Vikram, Jarek, Ash, > > Thanks for your fast feedback. So I understand you ask for more details. > Then I will cancel the vote if you think the AIP is not ready. > > I would have expected with my earlier call for discussion to receive more > feedback on what you expect missing. Please add comments to confluence or > post a message to me. That would help making the AIP to the level of detail > you expect. > A PR is not ready helping to discover, it is rather a skeleton but a lot > of pieces are not decided. I would have planned to tackle this during > implementation. And then - once code is mature - there are PRs planned of > course where also serious code feedback will be needed. > But my intent was not to spent 100's of hours in a detailed concept then > to get a push-back later. That is why I called for vote. If you can also > please express the "do this" or "don't this" in the AIP feedback that will > help that I am not wasting my time hitting the wall later. Also I was > seeking for contribution, so if you have ideas or recommendation, please > keep them coming. > > Immediate feedback to some of your questions: > - @ash: What do you expect in a level of detail in a PR to review before > accepting the AIP? Are you looking for certain patterns, modules which are > used? > - @jarek: PoC can be made but I fear that is already 50% of effort to a > MVP. That is why I was seeking for feedback if this would be the right way > before spending efforts. > - @jarek: AIP-44 would certainly be leveraged. I have no plan to replicate > existing API and I know that it will be a challenge to de-couple the task > execution. Do you expect me to evaluate all details before a vote? I would > have planned a pragmatic approach, once some dependency missing I would > maybe have contributed to AIP-44 efforts to close gaps. But AIP-44 does not > make a remote scheduling/execution possible as it does not include the > remote worker and executor component. That is to be added in AIP-69. > - @jarek/@ash: You briefly talked about the "task isolation" but are there > any concepts you can share which will help? I understood you are _thinking_ > about it and there will be papers _soon_? If there is anything that > contributes, please share. I have no glassball. > @jarek: Lifecyle of task and adoption etc.: Yes this is something to be > addressed and is a must to be included. Mainly it would be around a > heartbeat logic. I am sure there will be something to be done and > resiliency will be covered. If you have requirements or ideas please > contribute. Do you see this as a must have in the AIP before going to vote? > Or is it just important it is covered? > @jarek: API promises on exactly once: My plan is to use and rely on DB > locks and transactions from the API. Assuming that something can go wrong > also between API and remote side I would add an additional confirmation in > the heartbeat when the task was accepted and is starting on remote (to > cover the use case that a task is assigned and DB transaction committed and > response is not reaching the remote because of... line breaks down). Als > certainly if a remote worker is "lost" and does not heartbeat for a (to be > configured) timeout then a task must be re-assigned or assumed to be failed. > > P.S.: I'd like to take your feedback serious, the AIP process description > in Confluence just tells: "Once you or someone else feels like there’s a > rough consensus on the idea and there’s no strong opposition, you can move > your proposal to the Vote phase." - neither this nor the structure template > mention that a technical spec or PR must be provided prior vote. If you > feel that an AIP should include this then I assume the contribution docs > need to be adjusted. > > Mit freundlichen Grüßen / Best regards > > Jens Scheffler > > Alliance: Enabler - Tech Lead (XC-AS/EAE-ADA-T) > Robert Bosch GmbH | Hessbruehlstraße 21 | 70565 Stuttgart-Vaihingen | > GERMANY | www.bosch.com > Tel. +49 711 811-91508 | Mobil +49 160 90417410 | > jens.scheff...@de.bosch.com > > Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart, HRB 14000; > Aufsichtsratsvorsitzender: Prof. Dr. Stefan Asenkerschbaumer; > Geschäftsführung: Dr. Stefan Hartung, Dr. Christian Fischer, Dr. Markus > Forschner, > Stefan Grosch, Dr. Markus Heyn, Dr. Frank Meyer, Dr. Tanja Rückert > > -----Original Message----- > From: Vikram Koka <vik...@astronomer.io.INVALID> > Sent: Saturday, May 18, 2024 6:57 PM > To: dev@airflow.apache.org > Subject: Re: [VOTE] AIP-69 Remote Executor > > I agree with Jarek and Ash on this. > > I believe that the AIP as written documents the “what” and the “why” very > well, but is too light on the “how”. > > I very much would like to see this AIP become reality as well, but I > believe that we need some foundational elements such as API-44 and the > “task context” concept to take this AIP to fruition with enough > functionality to be meaningful. > > It’s entire possible that you are proposing something in between that’s > feasible in the short-term, but that’s not clear to me yet. > > Vikram > > > On Sat, May 18, 2024 at 8:46 AM Jarek Potiuk <ja...@potiuk.com> wrote: > > > Agree with Ash. I also have a feeling that this is a very generic > > description and some POC describing the approach we would like to take > > here is needed to be able to vote on it. Feels a bit too early to vote > on it. > > > > A lot of internals of the current (Airflow 2) way Airflow handling of > > tasks running in executor are about database communication and if you > > look closely, decoupling those internals to make it works with the > > current executor API might be either very difficult or complex if we > > stick to the current task <> airflow communication patterns. In some > > ways, you already get "Remote Executor" by simply completing AIP-44 > > (which precisely provides an HTTP-based. interface between tasks and > > scheduler in a very similar fashion as AIP-69 proposal, but without > > changing the communication patterns. > > > > As I see it it generally looks like AIP-69 is either the same as > > AIP-44 or the same as the future "Airflow 3" task isolation we've been > > discussing about and Ash is working on. Both AIP-44 and the future > > Task Isolation aim to solve pretty much the same problem but AIP-44 in > > a very "backwards compatible" way and limited change on how to achieve > "remote execution" > > without actually making a lot of ripple effect on all other components. > > > > So I would really love to understand if AIP-69 is really something > > in-between and how - but the "lightweitness" of description makes it > > difficult to understand how different AIP-69 is from those two (of > > course we should see the future task isolation AIP as well to > > understand it better and know what kind of back-compatibilities it will > involve in Airflow 3). > > > > I think at the very minimum we should see the proposal of how the API > > will look like between the task and executor (including the whole > > life-cycle of tasks), the way how we are going to implement all the > > complexity involved with task adoption, edge cases of scheduling, > > execution semantic promises the API will hold (exactly-once, > > at-most-once, at least once) - something that comes as given for > > celery, the queuing mechanism and technologies used (how do we handle > > distributed case where you have to manage multiple workers running and > how the tasks will be distributed etc. etc. > > > > For me the AIP currently is mostly documenting a "wishlist" but the > > implementation details on which of those wishes we implement, and > > which not, and how is very much absent. > > > > J. > > > > On Sat, May 18, 2024 at 10:41 AM Ash Berlin-Taylor <a...@apache.org> > wrote: > > > > > Can we have a link to the pr please? The AIP doc itself is still > > > light on what changes are actually needed > > > > > > On 18 May 2024 14:56:57 BST, Aritra Basu <aritrabasu1...@gmail.com> > > wrote: > > > >+1 (non-binding) > > > >The proposal was a good read, would love to see it come up and > > > >would > > love > > > >to help out if you need a helping hand. > > > > > > > >-- > > > >Regards, > > > >Aritra Basu > > > > > > > >On Sat, May 18, 2024, 7:15 PM Christian Schilling > > > ><christian.lellm...@googlemail.com.invalid> wrote: > > > > > > > >> Hi Jens, > > > >> > > > >> Thank you very much for the proposal! > > > >> This would be cool to have such a feature in Airflow. > > > >> > > > >> +1 non-binding > > > >> > > > >> Best, > > > >> > > > >> Chris > > > >> > > > >> Scheffler Jens (XC-AS/EAE-ADA-T) <jens.scheff...@de.bosch.com > > .invalid> > > > >> schrieb am Sa., 18. Mai 2024, 15:40: > > > >> > > > >> > Hi all, > > > >> > > > > >> > > > > >> > > > > >> > Discussion thread: > > > >> > > > > >> > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F > > > >> > %2Flists.apache.org%2Fthread%2F8hlltm9brdxqyf8jyw1syrfb11hl52k5 > > > >> > &data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C4300b2481a734dd > > > >> > aadac08dc775b921e%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C > > > >> > 638516482373030237%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAi > > > >> > LCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sda > > > >> > ta=E4blX6HbfvW73CkOWMxQoc5XSYDiIToAcLqbh9zVFE8%3D&reserved=0 > > > >> > > > > >> > > > > >> > > > > >> > I would like to officially call for a vote to add a Remote > > > >> > Executor feature to Airflow – via a provider package. All > > > >> > details are > > > documented > > > >> in: > > > >> > > > > >> > > > > > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwik > > i.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-69%2BRemote%2BExec > > utor&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C4300b2481a734ddaad > > ac08dc775b921e%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C6385164823 > > 73039443%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiL > > CJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=Hbvz5Ct1PE52pUDFDAeHO > > SYncOZ%2BOFi9eZQilaxeu9E%3D&reserved=0 > > > >> > > > > >> > > > > >> > > > > >> > Since the first draft was raised and the discussion thread went > > > along, I > > > >> > integrated a bit of feedback and now also added some technical > > > details as > > > >> > proposed development. After successful vote I’d raise a draft > > > >> > PR as > > > PoC. > > > >> > As a big wave of emails was posted by Jarek after I dropped > > > >> > this AIP > > > I’d > > > >> > like to highlight that I propose to make this a tactical > > > implementation > > > >> > which might be a base for some discussion how to distribute > > > >> > work in > > a > > > >> > future Airflow 3.0. I would assume if interfaces and structures > > > change, > > > >> > rework will be needed and it is fully accepted that breaking > > > >> > changes > > > need > > > >> > to be applied if moving to Airflow 3. > > > >> > > > > >> > > > > >> > > > > >> > Looking forward to releasing this for Airflow 2.10 (but > > > >> > depending > > how > > > >> fast > > > >> > I can make it and also depending on if somebody wants to join > > forces). > > > >> > > > > >> > > > > >> > > > > >> > Consider this my +1 binding vote. > > > >> > > > > >> > > > > >> > > > > >> > The vote will last until 6:00 PM GMT/UTC on May 23, 2024, and > > > >> > until > > at > > > >> > least 3 binding votes have been cast. I have it a bit longer as > > usual > > > >> > because of a public holiday in some countries. > > > >> > > > > >> > > > > >> > > > > >> > Please vote accordingly: > > > >> > > > > >> > > > > >> > > > > >> > [ ] + 1 approve > > > >> > > > > >> > [ ] + 0 no opinion > > > >> > > > > >> > [ ] - 1 disapprove with the reason > > > >> > > > > >> > > > > >> > > > > >> > Only votes from PMC members and committers are binding, but > > > >> > other > > > members > > > >> > of the community are encouraged to check the AIP and vote with > > > >> > "(non-binding)". > > > >> > > > > >> > Mit freundlichen Grüßen / Best regards > > > >> > > > > >> > Jens Scheffler > > > >> > > > > >> > Alliance: Enabler - Tech Lead (XC-AS/EAE-ADA-T) Robert Bosch > > > >> > GmbH | Hessbruehlstraße 21 | 70565 Stuttgart-Vaihingen > > | > > > >> > GERMANY | www.bosch.com > > > >> > Tel. +49 711 811-91508 | Mobil +49 160 90417410 | > > > >> > jens.scheff...@de.bosch.com<mailto:jens.scheff...@de.bosch.com> > > > >> > > > > >> > Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart, HRB > > > >> > 14000; > > > >> > Aufsichtsratsvorsitzender: Prof. Dr. Stefan Asenkerschbaumer; > > > >> > Geschäftsführung: Dr. Stefan Hartung, Dr. Christian Fischer, Dr. > > > Markus > > > >> > Forschner, > > > >> > Stefan Grosch, Dr. Markus Heyn, Dr. Frank Meyer, Dr. Tanja > > > >> > Rückert > > > >> > > > > >> > > > > > >
