theobjectivedad opened a new issue, #33036:
URL: https://github.com/apache/airflow/issues/33036
### Description
This feature request proposes an update to
``airflow.providers.docker.operators.docker.DockerOperator`` where DAG
parameters can optionally be passed into the container as environment
variables. For example:
```python
task = DockerConfOperator(
task_id="test_task",
params_in_env=True, # Defaults to False for compatibility
...
)
```
When enabled, DAG parameters will be passed into the container's
environment. This would be a fairly simple update to ``execute`` so that it
would pass the DAG context to ``_run_image`` which would pass the context to
``_run_image_with_mounts``. From here, ``context["params"]`` would simply be
added to the container environment:
```python
self.container = self.cli.create_container(
command=self.format_command(self.command),
name=self.container_name,
environment={
**self._private_environment,
**self.environment,
**context["params"]
},
...
)
```
### Use case/motivation
**Scenario:**
DAG1 defines a DockerOperator task and provided a default environment
variables, volume mounts, GPU devices, secrets, and other container-level
settings for it to run in the desired environment. This configuration can be
somewhat sophisticated depending on the environment.
Other DAGS leverage the TriggerDagRunOperator to invoke DAG1 and pass in
``conf`` parameters that will override environment variables in the container
started by DAG1:
```python
my_task = TriggerDagRunOperator(
task_id="inference_task",
trigger_dag_id="DAG1",
conf={
"ENV_VAR1": "VAL1",
"ENV_VAR2": "VAL2",
},
)
```
Currently, it isn't straightforward to pass ``conf`` through dynamically to
the container started by DAG1 since ``conf`` is part of the DAG context. An
obvious solution would be to use templates however this would require that
``DAG1`` map each environment variable individually which could be too
cumbersome for some applications and provide little value in the end (just
consider how many environment-based configurations Airflow itself has).
With this update, ``conf`` would simply be passed into the container's
environment when the feature is enabled and no additional mapping needed.
### Related issues
_No response_
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
