Hi, By slowly, you mean that it can change between dag runs (as it is actually currently)? To be honest current semantic is enough for me. The last thing I would expect from orchestrator is to have mutating "dag run" or even "dag" definition (as it would be against lineage). However for dag it is still acceptable to have "append only" for new nodes and edges, where old and not used anymore paths you can just skip in future dag runs. I will trace the page you provided and try to contribute somehow.
Cheers, JB śr., 16 wrz 2020, 14:53 użytkownik Jarek Potiuk <[email protected]> napisał: > Just a few comments: > - the structure can change dynamically. Just "slowly" (I know it's not > very precise). They will work fine when the structure change "slowly" and > the UI will also not be fully reflecting the reality while transition is > happening but it will work. > - we are working on becoming "true" to the dynamic promise - we have a > work in progress about DAG versioning > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-36+DAG+Versioning > and once implemented, the dynamic nature that was promised but not yet > delivered. I'd really love if you join the discussions there and maybe even > contribute to making it happen :) > > J. > > On Wed, Sep 16, 2020 at 9:30 AM Johnny Bravo <[email protected]> > wrote: > >> Hi! >> >> I'm a user of your great tool. I have already created few DAGs and >> some of them are even quite complex (in terms of carefully crafted >> custom operators as well as in terms of size and complexity of >> dependencies between tasks). Before getting in that place I've gone a >> long way for understanding the semantics used by Airflow. >> >> Recently I have entered again to the Airflow Apache page and to my >> surprise found that actually the part on which I spent most of time >> during solution development is somehow wrongly described on the >> Airflow page. It is all about the word "dynamic" on your principles >> list. >> >> Airflow DAGs are not dynamic. To shorten discussion, it should be >> called "Programmable", which means that we can craft DAG with the use >> of algorithms or external knowledge, but final shape cannot change >> during execution as the DAG must be fully known before running so >> actually these are static. >> >> Lets now find out meaning of two words: >> >> dynamic - "(of a process or system) characterized by constant change, >> activity, or progress." (afer google translator) and "In general, >> dynamic means energetic, capable of action and/or change, or forceful, >> while static means stationary or fixed. In computer terminology, >> dynamic usually means capable of action and/or change, while static >> means fixed." (after techtarget.com). >> >> >> >> programmable - (of a computer or other machine) able to be provided >> with coded instructions for the automatic performance of a task (after >> google translator) and here Computer_programming @ wikipedia.org >> >> So for the sake of stopping giving newbies' heads headaches, stop >> using the word "dynamic" in the current context. >> >> Change my mind ;-) >> >> Cheers, >> JB. >> > > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> > >
