BTW, I am planning to turn the doc into a blog post on Airflow Medium (I think it provides a nice description on the extra usage for Airflow) - so I want to make a blog about "Airflow 2 how to use extras" - following with "Airflow 3 how to use extras" once we come to conclusions re: packaging and extras for them - so I'd appreciate any comments in general :D
On Fri, Oct 18, 2024 at 8:17 AM Jarek Potiuk <ja...@potiuk.com> wrote: > Also one more thing added - I realized I have forgotten one more thing - > pre-installed providers. Those are automatically added to 'requirements" in > "standard" installation, and in "editable" mode - their dependencies are > added instead (not the providers themselves). This is an important case for > both editable and standard mode. > > The doc is updated accordingly. > > J. > > > On Thu, Oct 17, 2024 at 2:56 PM Jarek Potiuk <ja...@potiuk.com> wrote: > >> Also one more comment. I added this paragraph (and highlighted it): >> >> > The approach to extras depends very much on the packaging decisions we >> will make before releasing Airflow 3. Currently we only have one >> “apache-airflow” package and all extras are defined there. In Airflow 3 we >> will have more packages (at least 2) and some extras might or might not be >> available in some of those packages (and different packages might have >> different extras. >> >> We are (currently offline to present it soon for wider discussion) >> discussing the packaging approach - there is a draft document from Vikram >> few of us are commenting on and it is parallel discussion that will have >> impact on the "extras" decisions. >> >> J. >> >> >> >> On Thu, Oct 17, 2024 at 12:25 PM Jarek Potiuk <ja...@potiuk.com> wrote: >> >>> "separate POM" -> Separate pyproject.toml :D,, I've been in too many >>> discussions in Apache Software Foundation about java dependencies and POM >>> files and Maven (ough!) it seems :D >>> >>> On Thu, Oct 17, 2024 at 12:22 PM Jarek Potiuk <ja...@potiuk.com> wrote: >>> >>>> Hello here, >>>> >>>> As discussed last week on slack, I prepared a comprehensive document >>>> describing the current state of extras in Airflow (or at least my view on >>>> it :) ). >>>> >>>> >>>> https://docs.google.com/document/d/1e1T3u34bqo7yYSJVvq1NlEJJhA5_hUNXOXo7J6KFg_0/edit?usp=sharing >>>> >>>> The document describes the current state of extras as we use in Airflow >>>> 2 (and same usage so far in Airflow 3) - split into development and user >>>> use cases (which roughly splits to "standard" and "editable" installation. >>>> It describes the different behaviour of extras in "standard" and "editable" >>>> environments (via hatch_build.py hooks) as well as potential future and >>>> very recent developments that might impact the way we will think about >>>> extras for Airflow 3. >>>> >>>> For example the recent development about - accepted last week PEP 735 >>>> that might change the way we will define development groups. But tools need >>>> to catch up. >>>> >>>> Also the move to "uv workspaces" might (and will) change some of the >>>> ways we interact with Airflow - especially for development, which might >>>> render some of the extra use cases obsolete, but IMHO we need to do some >>>> preparatory work (and further splitting providers to per-provider >>>> sub-directories and each provider having a separate POM is a prerequisite >>>> for that). >>>> >>>> There are also some other potential changes and future work we can do >>>> (all marked in yellow) that might impact the way we think about and define >>>> our extras. >>>> >>>> Finally - at the end - it contains a summary of all the use-cases (I >>>> found 6 in total) how extras are currently used - user and development use >>>> cases. >>>> >>>> I know that there are voices that maybe we should get rid of some of >>>> the extras - so having summary of the use cases might be helpful to discuss >>>> it: >>>> >>>> * whether the use case is actually "useful" >>>> * whether we have some way to replace the use of extras to achieve >>>> similar behaviour >>>> >>>> I propose - if you have detailed comments to the document - let's >>>> discuss it there (everyone has commenter access by following the link). If >>>> you have some proposals on what to do after reading and discussing details >>>> - we should discuss any proposals in this thread >>>> >>>> Have a nice read :) >>>> >>>> J. >>>> >>>> >>>> >>>
