germanium opened a new issue, #34461:
URL: https://github.com/apache/airflow/issues/34461
### Apache Airflow version
2.7.1
### What happened
Passing environment variables dynamically to the
`DatabricksSubmitRunOperator` results in all tasks having the environment
variable from the last task in the list
```python
from airflow.providers.databricks.operators.databricks import
DatabricksSubmitRunOperator
from airflow.models import DAG
from datetime import datetime
DAG_INSTANCE = DAG(
dag_id="my_test_dag",
start_date=datetime(year=2020, month=11, day=23),
)
env_vars = {
"CONST_VAR": "some_var"
}
for my_var in ["var_a", "var_b"]:
env_vars["MY_VAR"] = my_var
DatabricksSubmitRunOperator(
task_id=f"processing_{my_var}",
dag=DAG_INSTANCE,
retries=1,
json={
"run_name": f"processing_{my_var}",
"new_cluster": {
"spark_version": "13.1.x-scala2.12"
"spark_env_vars": env_vars,
}
}
)
```
For the two task the UI shows that `MY_VAR` was assigned `var_b`, the last
variable in the list.
Task Instance details for processing_var_a
```
{
"new_cluster": {
"spark_env_vars": {
"MY_VAR": "var_b",
"CONST_VAR": "some_var"
},
"spark_version": "13.1.x-scala2.12"
},
"run_name": "processing_var_a"
}
```
Task Instance details for processing_var_b
```
{
"new_cluster": {
"spark_env_vars": {
"MY_VAR": "var_b",
"CONST_VAR": "some_var"
},
"spark_version": "13.1.x-scala2.12"
},
"run_name": "processing_var_b"
}
```****
### What you think should happen instead
Each task should have its dynamically assigned `spark_env_vars`:
`processing_var_a` should have `MY_VAR=var_a` and `processing_var_b`
`MY_VAR=var_b`
The correct behavior was observed up until
`apache-airflow-providers-databricks = "4.0.0"`
### How to reproduce
Create a DAG similar to the one in the `What happened` section and run it.
### Operating System
debian:11-slim
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon = "3.4.0"
apache-airflow-providers-cncf-kubernetes = "4.0.2"
apache-airflow-providers-databricks = "4.5.0"
apache-airflow-providers-datadog = "2.0.4"
apache-airflow-providers-mongo = "2.3.3"
apache-airflow-providers-pagerduty = "2.1.3"
apache-airflow-providers-postgres = "4.1.0"
apache-airflow-providers-slack = "4.2.3"
apache-airflow-providers-snowflake = "2.7.0"
### Deployment
Docker-Compose
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
