syedahsn commented on issue #35490:
URL: https://github.com/apache/airflow/issues/35490#issuecomment-1901325653
The difference that I was pointing out between our suggestions was that if
we don't specify a key in `executor_config` which exists in `RUN_TASK_KWARGS`,
that key should not be removed from the final config, no matter what level it
is at.
So in your example, even though `executor_config` did not include a
`startedBy` key, it was still included in the final config. However,
`overrides` was included in both `RUN_TASK_KWARGS` and `executor_config`, but
because it is a dictionary, we look at the individual keys, and only change a
value if the key is being specified in the `executor_config`.
So in the example, there is `cpu` key in the `overrides` dictionary in the
`RUN_TASK_KWARGS` config, but `cpu` is not in the `overrides` dictionary in
`executor_config` therefore we leave it as is. As a result, the final config
has a `cpu` key in the `overrides` dictionary.
The difference between our approach is what happens in `containerOverrides`.
You are suggesting that because we are defining that in the `executor_config`,
it should overwrite the existing value in `RUN_TASK_KWARGS`. My suggestion is
to use the same logic that we are using in the overall config, and evaluate key
by key. We would check to see if the key exists in the dictionary, and update
if it does, and add it if it doesn't.
As for issue of removing a tag (or any key) from a dictionary, maybe we can
use an empty value (i.e. `""`) as a way to remove an existing key from the
config.
```
RUN_TASK_KWARGS = {
"tags": [{"key": "fish", "value": "banana"}, {"key": "peanut", "value":
"sauce"}],
"startedBy": "Niko",
"overrides": {
"containerOverrides": [{"memory": 500, "environment":[{"name":
"FOO", "value": "BAR"}]}],
"cpu": "5",
}
}
executor_config={
"tags": [{"key": "FOO", "value": "BAR"},{"key": "fish", "value": ""}],
"overrides": {
"containerOverrides": [{"memory": 740, "environment": [{"name": "FOO",
"value": ""}, {"name": "fish", "value": "banana"}]}]
},
}
```
The final config will be :
```
Config = {
"tags": [{"key": "peanut", "value": "sauce"}, {"key": "FOO", "value":
"BAR"}],
"startedBy": "Niko",
"overrides": {
"containerOverrides": [{"memory": 740, "environment":[{"name":
"fish", "value": "banana"}]}],
"cpu": "5",
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
