GitHub user potiuk edited a comment on the discussion: Versioning Problem of (likely badly written) Dynamically Created DAGS
> The reality is that is dynamically generated and so it is obvious that > changes dynamically. That's not true in general. If your intention to generate a different dag every time it is parsed, then yes it will be dynamically changing and. as far as I know - you can disable versioning https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#disable-bundle-versioning for that bundle if your bundle contains DAG that change frequently and it is expected (but this is generally a bad practice) Dynamically generated dags are NOT supposed to dynamically change every time they are parsed. They MAY change slower or faster - depending on how often the "input" to those dynamic dags change. For example if you have list of countries, and you generate dynamic dags based on that - your parsed dag should change only when you add a new country - every time (providing that your code to generate dynamic dag is well written) , parsing the same Dag **should** produce the same - dynamically generated - parsed representation. So in your case you likely: a) your intention is to change your dag every time it is parssed (which is quite strange, but then you should disable versioning for that bundle) b) there is a problem with dynamic dag generation code - it **should** produce the same parsed representation, every time Dag is parsed. There might be two resons for b) 1) your dynamic dag generation has problems and does not produce stable dag object structure or stable parameters. This might change for example when you do not use 'fixed` start date - but use `days_ago` or similar - which is a bad practice, because days_ago() will change every time your dag is parsed. Or you can pass time() or other dynamically generated (and ever changing) content as part of your dag definition. Or you create objects in a random sequence every time you build your dynamic dag. This might happen for example when you run "SELECT * FROM" and do not ORDER the select - because the databases return data in such select in random order in general, and the order might change any time. 2) there is a bug in our Dag serialization code Most likely it's 1) and you have to fix your dynamic dag generation to be "stable" if you want to version dag. Which means that serialized version of your Dag should be THE SAME if Dag parsing on your Dag is executed - if none of the inputs changed - and they should remain stable over time, so they should be independent on the time when they are parsed. It could also be 2) - but in order to know whether it's 1) or 2) you have to look at the serialized dag table (see https://airflow.apache.org/docs/apache-airflow/stable/database-erd-ref.html ) find two versions of your Dags and find out what's the difference betwen the dags and what kind of "stability" issue in dynamic dag generation causes it. The "compressed" representation in this table is compressed with `.gzip` so you need to retrieve the blobs and decompress them -> you will get .json representation of the dags that you should be able to compare and see where the problem is coming from. Also it might be easier for you if you review your dynamic dag generation looking at potential stability issues from those described above. Once you find the reasons - please let us know - it would be interesting to find out what stability issues people like you might have - or maybe you will find some of the problems coming from our serializaiton. So we are looking forward for your investigation. Converting it into a discussion until you investigate it and provide more information. GitHub link: https://github.com/apache/airflow/discussions/54348#discussioncomment-14069763 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
