+1 -- also been on my list of things I've thought about doing. (Get out of my head!)
On Dec 20 2022, at 3:47 pm, Jarek Potiuk <[email protected]> wrote: > Hello everyone, > > I have a proposal (Draft PR here: > https://github.com/apache/airflow/pull/28495) how we can change the approach > to default values for Airflow configuration - namely to keep "defaults" > internally in airflow application rather than with generated "airflow.cfg" > file. > > This has been popping up in casual conversations we were having at various > occasions, and I wanted to implement another fix (with different default on > MacOS) and decided to give it a try today - and I believe (with some more > testing and likely some tweaks) - I pretty much achieved what I think makes > sense (still a few small things around dumping values for diagnostics need to > be updated) > > I believe what I propose is fully backwards compatible, but it has some nice > future-looking properties. > > The gist of the proposal: > > * defaults for config values are kept internally as a generated Python > dictionary rather than a generated .cfg file (generated from the same single > source of truth - .yaml config file). > > * the "airflow.cfg" file is still generated when it is missing - but with > commented out values. This way first-time users can still see available > sections and options and can easily change them in the config file > > * if you have a configuration file already generated - nothing changes - the > config values will still be used as they used to be and the internal defaults > are not used. > > Doing it this way has some nice properties: > > * we can easily change defaults in the future and rename properties > * we can easily know if the value comes from internal defaults or from a file > (even if it has the same value as default) > * we can have dynamic defaults depending on different factors (for example > disable forking on MacOS dynamically) > > Let me know what you think. > > J.
