GitHub user potiuk edited a comment on the discussion: Versioning Problem of (likely badly written) Dynamically Created DAGS
Thanks for testing it! > * if you are reasoning like the "git command" this is an Airflow's "subtle" > bug. > * If you are reasoning looking at the final results of the "compiled dynamic > code" this is the perfectly expected behaviour. The answer is easy. Option 2. It's always been the goal of DAG versioning - and you can read more about it in Airflow Improvement proposals (3 of them) describing the idea, implementaiton, goals, and future of Dag versioning - some of the things there are not yet implemented 100% https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-63%3A+DAG+Versioning > Actually if you talk to a developer about "code's versioning" I think that > the common mindset is totally "git influenced". So if we align Airflow to this mindset it is much simpler to explain and document the mechanism... but probably it is not so simple to implement it in the "dag-processor". No. It's not a question of simpllicity. It's a question of deliberate choice and probably several tens of discussion points and countless hours of talking and agreeing on the direction by those who proposed, discussed, got approved and finally implemented it. This reflects the very "Fabric" of Airflow - where from the very beginning, dynamig Dag generation actually allowed to effectively have different versions of dag (depending on the input parameters). But before Dag versioning it has never been captured and caused all kind of - mostly - ui and rerun / backfill problems when Dynamically generated dags changed. So - from the very beginnig of the Dag Versioning discussions (and if you look at the AIPs they have linked in discussions at devlist and those discussions refer not only to other discussions but also to previous version of Dag Versioning proposal - rejected and abandoned https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-36+DAG+Versioning - this was very well discussed and agreed to. Now - It seems you indeed pointed to a problem - that people might understand it differently. There is no reason or possibility to change it to the "git" approach, but since you now understand and got explained how it works we would be very happy if you could help to make it clearer: 1) Updating our documentation to make it more explicit and clearer. People like you, who got burned and understood things differently (but are smart and understand the reasoning now) are the best to put it in the words and describing things in the way that other people like them are not making the same (wrong) assumptions you did. Simply - after all the years of discussing it, things are "obvious" for maintainers and creators - but they are "non obvious" for newcomers, so adding a PR where you could describe is better - is golden (and fantastic opportunity to contribute back for the free software you get to use). 2) Describing your and similar use-cases to make it easier for people to see and investigate their issues when they hit similar things - even if it is starting "FAQ" or "common mistakes" or smth like that - coming from someone who had, investigated, and found the issues after having potential reasons explained by maintainer - is again golden and fantastic opportunity to contribute a PR 3) More complex and sophisticated - figuring out some ways of automated detection and flagging similar problems in Dags written by people would be a fantastic opportunity to contribute back "code" contribution. There are many ways and many "bad practices" that could be flagged there - it could be based on static code analysis, or detecting situations where indeed every time you parse you have different versions and it keeps on changing continuously .. World is your oyster. Another great opportunity to contribute. Adding doc PR is super easy "suggest a change on this page" button (bottom-right) opens a PR where you can add documentation - that could be first step, later leading to a code contribution.. That would be fantastic output of that discussion. GitHub link: https://github.com/apache/airflow/discussions/54348#discussioncomment-14093597 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
