Good work on the AIPs @Scheffler Jens (XC-DX/ETV5) <[email protected]>!
I spent some time understanding your thoughts and I like the ideas and the direction you are heading too. I also agree with Jarek, that we might need to start thinking in terms of Airflow 3, especially now that you brought up AIP 68. Thanks & Regards, Amogh Desai On Sat, Apr 20, 2024 at 3:55 PM Jarek Potiuk <[email protected]> wrote: > Hey Jens, > > I looked at the AIPs when you created them and I very much like the > directions put there - but it also got me into a lot of thinking on the > future of Airflow and AIPs. See the thread I started > https://lists.apache.org/thread/3chvg9964zvh15mtrbl073f4oj3nlzp2 - about > Airflow 3. > > I think in both cases (especially about AIP-689 but also AIP-69) - it would > make a wealth of difference if we treat them in the context of Airflow 2, > or (maybe) in the context of Airflow 3 which might start from taking the > best of Airflow 2 and get rid of all the unnecessary baggage it has. In the > past many similar efforts like AIP-69 had stalled in general because they > were far too complex to implement taking into account backwards > compatibility expectations of Airflow 2. > > And I think it's the right time we should get to terms with the future of > Airflow - whether/to what extent we want Airflow 3 to come, what level of > compatibility it should have, which assumptions should be dropped. I > personally have a feeling that AIP-69 would have been way easier to bring > as one of Airflow 3 "foundational" AIP that could define the "new" remote > architecture of Airflow rather than "plugin" to existing one. Dropping > Celery & K8s Options, leaving the Remote + Local variant of it as the only > ones, without the direct DB communication channel we have now and replacing > it with smth else. > > That would be my first comment on it and a question - should we get a bit > more clarity on what "future" of Airflow is before we discuss details and > approach there. > > J. > > On Fri, Apr 19, 2024 at 10:06 PM Scheffler Jens (XC-AS/EAE-ADA-T) > <[email protected]> wrote: > > > Dear Dev-Community, > > > > mainly triggered by the deadline for CFP for Summit I dropped two “brand > > new” AIP’s as ideas that are running in my head for a longer time. Note > > these are DRAFT versions as a first write-up as solution concept and are > > lagging technical design and implementation yet. > > > > I’d kindly ask for feedback and review in Confluence cWiki (via Comments) > > – and also am seeking for people who like to join forces. > > > > AIP-68: Extended Plugin Interface for Custom Grid View Panels > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-68+Extended+Plugin+Interface+for+Custom+Grid+View+Panels > > > > Main motivation is that the UI has developed a lot in the recent time and > > AIP-38 is near completion. But it is focusing on technical details and > logs > > – and for most business users it is hard to read, missing business > > perspective. I propose to extend the Plugin interface allowing > > customizations on various new levels such that customer specific business > > information can be embedded, > > > > AIP-69: Remote Executor > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-69+Remote+Executor > > > > Airflow can be deployed in cloud or on-prem and various options of > > deployment are possible. But it lags the option to easily form a secure > and > > lean distributed setup for cases where individual nodes are far far away > > from the core of deployment. This imposes problems in opening firewalls > and > > might raise risk of security. Therefore I propose to add a “Remote > > Executor” such that workload can easily distributed to remote locations – > > also with a chance that it is easier for cases where people want to > > distribute workload to Windows (yeah there are really people around who > > still have this). > > > > Looking forward for (constructive) feedback, discussion and opinions. > > > > Again, note: DRAFT means open to discussion, nothing fixed, nothing coded > > yet. Many implementation options possible – and in case of interest > please > > join forces with me 😃 > > Once the discussion is calming I’d call for a vote separately like > usually. > > > > Mit freundlichen Grüßen / Best regards > > > > Jens Scheffler > > > > Alliance: Enabler - Tech Lead (XC-AS/EAE-ADA-T) > > Robert Bosch GmbH | Hessbruehlstraße 21 | 70565 Stuttgart-Vaihingen | > > GERMANY | www.bosch.com > > Tel. +49 711 811-91508 | Mobil +49 160 90417410 | > > [email protected]<mailto:[email protected]> > > > > Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart, HRB 14000; > > Aufsichtsratsvorsitzender: Prof. Dr. Stefan Asenkerschbaumer; > > Geschäftsführung: Dr. Stefan Hartung, Dr. Christian Fischer, Dr. Markus > > Forschner, > > Stefan Grosch, Dr. Markus Heyn, Dr. Frank Meyer, Dr. Tanja Rückert > > > > >
