GitHub user Foxigod created a discussion: Using Params with .override()?
Working with Airflow version 3.1+ (3.1.7 currently). I've come across this task parametrization feature in Airflow to reuse decorated tasks: https://airflow.apache.org/docs/apache-airflow/stable/tutorial/taskflow.html#task-parameterization , e.g. to instantiate tasks multiple times with different task_ids. Even though this looks to be for portability between DAGs, or instantiation of multiple similar tasks within a single DAG, I am interested in the simple ability to override chosen attributes in the `@task` decorator. Is it possible to couple this somehow with Params, the ones provided at trigger-time of a DAG, such that some attributes of the `@task` decorator may be modified depending on what the user inputs at trigger-time? ### Motivation We have a custom Executor which we specify in the `@task` decorator and provide an `executor_config` attribute for configuration. This executor_config, among other things, includes which machine to run on and e.g. how many resources to allocate (particularly how many nodes to use in an HPC setting). The custom Executor then allocates these resources for the job to run. Many workflows are of the nature of being able to run with multiple such different configurations (especially varying the number of nodes), however, we are having trouble understanding if this .override functionality has the ability to work in tandem with the Params specified at trigger-time. I see something similar at: https://github.com/apache/airflow/discussions/46880, where a user wants to launch tasks on a user-specified data center at trigger-time, however their Airflow version is 2.x. Is this situation still the same, i.e. is the `task_instance_mutation_hook` cluster policy the "right" way forward here, or is there a better way to achieve this? Could the solution be to implement custom operators with these specific fields (machine, number of nodes) as templatable, and then possibly create a custom decorator to align with the TaskFlow API syntax (or is that somehow easier said than done)? Here seems to be another discussion where a user is expressing a similar desire to use Params to override some specific attributes of the task: https://github.com/apache/airflow/discussions/21627. In essence I guess my discussion is somewhat reviving those older ones in the context of a more recent Airflow version. GitHub link: https://github.com/apache/airflow/discussions/63191 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
