Hey Jens, I looked at the AIPs when you created them and I very much like the directions put there - but it also got me into a lot of thinking on the future of Airflow and AIPs. See the thread I started https://lists.apache.org/thread/3chvg9964zvh15mtrbl073f4oj3nlzp2 - about Airflow 3.
I think in both cases (especially about AIP-689 but also AIP-69) - it would make a wealth of difference if we treat them in the context of Airflow 2, or (maybe) in the context of Airflow 3 which might start from taking the best of Airflow 2 and get rid of all the unnecessary baggage it has. In the past many similar efforts like AIP-69 had stalled in general because they were far too complex to implement taking into account backwards compatibility expectations of Airflow 2. And I think it's the right time we should get to terms with the future of Airflow - whether/to what extent we want Airflow 3 to come, what level of compatibility it should have, which assumptions should be dropped. I personally have a feeling that AIP-69 would have been way easier to bring as one of Airflow 3 "foundational" AIP that could define the "new" remote architecture of Airflow rather than "plugin" to existing one. Dropping Celery & K8s Options, leaving the Remote + Local variant of it as the only ones, without the direct DB communication channel we have now and replacing it with smth else. That would be my first comment on it and a question - should we get a bit more clarity on what "future" of Airflow is before we discuss details and approach there. J. On Fri, Apr 19, 2024 at 10:06 PM Scheffler Jens (XC-AS/EAE-ADA-T) <[email protected]> wrote: > Dear Dev-Community, > > mainly triggered by the deadline for CFP for Summit I dropped two “brand > new” AIP’s as ideas that are running in my head for a longer time. Note > these are DRAFT versions as a first write-up as solution concept and are > lagging technical design and implementation yet. > > I’d kindly ask for feedback and review in Confluence cWiki (via Comments) > – and also am seeking for people who like to join forces. > > AIP-68: Extended Plugin Interface for Custom Grid View Panels > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-68+Extended+Plugin+Interface+for+Custom+Grid+View+Panels > > Main motivation is that the UI has developed a lot in the recent time and > AIP-38 is near completion. But it is focusing on technical details and logs > – and for most business users it is hard to read, missing business > perspective. I propose to extend the Plugin interface allowing > customizations on various new levels such that customer specific business > information can be embedded, > > AIP-69: Remote Executor > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-69+Remote+Executor > > Airflow can be deployed in cloud or on-prem and various options of > deployment are possible. But it lags the option to easily form a secure and > lean distributed setup for cases where individual nodes are far far away > from the core of deployment. This imposes problems in opening firewalls and > might raise risk of security. Therefore I propose to add a “Remote > Executor” such that workload can easily distributed to remote locations – > also with a chance that it is easier for cases where people want to > distribute workload to Windows (yeah there are really people around who > still have this). > > Looking forward for (constructive) feedback, discussion and opinions. > > Again, note: DRAFT means open to discussion, nothing fixed, nothing coded > yet. Many implementation options possible – and in case of interest please > join forces with me 😃 > Once the discussion is calming I’d call for a vote separately like usually. > > Mit freundlichen Grüßen / Best regards > > Jens Scheffler > > Alliance: Enabler - Tech Lead (XC-AS/EAE-ADA-T) > Robert Bosch GmbH | Hessbruehlstraße 21 | 70565 Stuttgart-Vaihingen | > GERMANY | www.bosch.com > Tel. +49 711 811-91508 | Mobil +49 160 90417410 | > [email protected]<mailto:[email protected]> > > Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart, HRB 14000; > Aufsichtsratsvorsitzender: Prof. Dr. Stefan Asenkerschbaumer; > Geschäftsführung: Dr. Stefan Hartung, Dr. Christian Fischer, Dr. Markus > Forschner, > Stefan Grosch, Dr. Markus Heyn, Dr. Frank Meyer, Dr. Tanja Rückert > >
