o-nikolas commented on issue #35490: URL: https://github.com/apache/airflow/issues/35490#issuecomment-1808741730
> Thanks for starting this @o-nikolas! > > It is important to note that you can use `propagateTags=TASK_DEFINITION` as a way of setting default tags to all the tasks that Airflow launches. I haven't implemented tagging for my own environment yet, but that would be my approach to global tagging. That would render settings tags via `run_task_kwargs` obsolete and then tags set with `executor_config` could always be the complete set of tags that are passed to the RunTask API. It's not really so much that tags are the specific case that needs a work around, just the example I chose, there are many others. > I'm also personally not a huge fan of the `RUN_TASK_KWARGS` config option. We already have explicit config options for several attributes (cluster, networkConfiguration, taskDefinition, platformVersion, launchType). I'd rather have more of the same. If we dictate what attributes can be set on a config level then it makes handling the `executor_config` much simpler. For example, I don't see the need of having `overrides` supported as a global config option when all of those properties can be set in the task definition. I'm not in favour of this approach. There are many config options for ECS RunTask, creating a first class Airflow config option for each one is untenable. Especially because that couples us very tightly to the ECS API, every change it makes we must reflect in our config options. The few important configs we have now plus the catch-all of the `run_task_kwargs` gives users full control without overburdening the maintenance and quality of the underlying code. We have the same situation with Airflow operators (which usually wrap an underlying Boto API or two), those operators cannot possibly take every parameter that the underlying Boto API can, so we pull out some of the mandatory and important ones, and then often have a <boto_api>_kwargs arg that allows the user to specify anything else. > > As well as from @mshober who has an implementation of this, so I'd be interested to hear what approach they took and how it's working out for them: > > My implementation is very specific and abstracted to my use case: > > ... > > so my code is likely not too helpful. I'll spend some time thinking about how it can be improved to fit all use cases. Mmm, yeah, that is very specific to your usecase it seems. Thanks for sharing it either way! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
