GitHub user rt849122 added a comment to the discussion: Large DAG with conditional subset execution: Is DAG Versioning + Dynamic DAG Generation the right approach?
Additional context: This is a deep learning use case. Our DAG contains a complete pipeline including data preprocessing, feature engineering, and model training. Each time we trigger a DAG run, we need to selectively run specific stages—for example, sometimes only data cleaning, sometimes only model training with existing cleaned data, or sometimes the full pipeline from scratch." GitHub link: https://github.com/apache/airflow/discussions/64344#discussioncomment-16353044 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
