Raul824 commented on issue #55489: URL: https://github.com/apache/airflow/issues/55489#issuecomment-3283404264
Well my main concern is external dag dependency logic creation across different strategies. as @Lee-W @jroachgolf84 you guys suggest getting source run id and yield metadata is possible from Asset Triggered dag. But my main point is A consistent date/timestamp which is common across the different triggering strategies let me try to elaborate. Let's take three daily scheduled dags DAG_SCHED_A, DAG_SCHED_B and DAG_SCHED_X. DAG_SCHED_A is triggering a dag let's name it DAG_TRIG_C. DAG_TRIG_C is triggering a dag using asset aware scheduling DAG_ASSET_D. DAG_ASSET_D has to run after DAG_SCHED_X and DAG_SCHED_B has to run after DAG_ASSET_D. Due to different logical date generation it becomes very hard to set these dependencies. DAG_ASSET_D external Dependency on DAG_SCHED_X --> It can be achieved by yield metadata or extracting source run id and checking the dependency based on source run logical date. DAG_SCHED_B external Dependency on DAG_ASSET_D --> This becomes tricky as there is no logical date in DAG_ASSET_D, and getting source run logical date of DAG_ASSET_D from DAG_SCHED_B would be too complex. If a logical date can be passed to Asset Triggered dag that would actually solve the problem of bridging the gap between dependencies. **From your first scheduled dag itself you can pass a logical date to your triggered dag (which is already possible in trigger dag operator) then from your triggered dag if you can pass a logical date to you Asset Triggered dag it would bridge the missing link to align your dependencies based on one logical date of first Upstream Dag.** I know I can re-write most of these to use triggered dag instead of Asset Aware but, **I think logical date as None has opened this possibility to open an option of passing a logical date to Asset Triggered dag, it can be optional but it would help align your dags for better external dag dependencies.** -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
