Also one more comment. I added this paragraph (and highlighted it): > The approach to extras depends very much on the packaging decisions we will make before releasing Airflow 3. Currently we only have one “apache-airflow” package and all extras are defined there. In Airflow 3 we will have more packages (at least 2) and some extras might or might not be available in some of those packages (and different packages might have different extras.
We are (currently offline to present it soon for wider discussion) discussing the packaging approach - there is a draft document from Vikram few of us are commenting on and it is parallel discussion that will have impact on the "extras" decisions. J. On Thu, Oct 17, 2024 at 12:25 PM Jarek Potiuk <ja...@potiuk.com> wrote: > "separate POM" -> Separate pyproject.toml :D,, I've been in too many > discussions in Apache Software Foundation about java dependencies and POM > files and Maven (ough!) it seems :D > > On Thu, Oct 17, 2024 at 12:22 PM Jarek Potiuk <ja...@potiuk.com> wrote: > >> Hello here, >> >> As discussed last week on slack, I prepared a comprehensive document >> describing the current state of extras in Airflow (or at least my view on >> it :) ). >> >> >> https://docs.google.com/document/d/1e1T3u34bqo7yYSJVvq1NlEJJhA5_hUNXOXo7J6KFg_0/edit?usp=sharing >> >> The document describes the current state of extras as we use in Airflow 2 >> (and same usage so far in Airflow 3) - split into development and user use >> cases (which roughly splits to "standard" and "editable" installation. It >> describes the different behaviour of extras in "standard" and "editable" >> environments (via hatch_build.py hooks) as well as potential future and >> very recent developments that might impact the way we will think about >> extras for Airflow 3. >> >> For example the recent development about - accepted last week PEP 735 >> that might change the way we will define development groups. But tools need >> to catch up. >> >> Also the move to "uv workspaces" might (and will) change some of the ways >> we interact with Airflow - especially for development, which might render >> some of the extra use cases obsolete, but IMHO we need to do some >> preparatory work (and further splitting providers to per-provider >> sub-directories and each provider having a separate POM is a prerequisite >> for that). >> >> There are also some other potential changes and future work we can do >> (all marked in yellow) that might impact the way we think about and define >> our extras. >> >> Finally - at the end - it contains a summary of all the use-cases (I >> found 6 in total) how extras are currently used - user and development use >> cases. >> >> I know that there are voices that maybe we should get rid of some of the >> extras - so having summary of the use cases might be helpful to discuss it: >> >> * whether the use case is actually "useful" >> * whether we have some way to replace the use of extras to achieve >> similar behaviour >> >> I propose - if you have detailed comments to the document - let's discuss >> it there (everyone has commenter access by following the link). If you have >> some proposals on what to do after reading and discussing details - we >> should discuss any proposals in this thread >> >> Have a nice read :) >> >> J. >> >> >> >
