diederikperdok opened a new issue, #74095: URL: https://github.com/apache/airflow/issues/74095
### Under which category would you file this issue? Providers ### Apache Airflow version 3.3.2 ### What happened and how to reproduce it? `DatabricksSubmitRunOperator` and `DatabricksRunNowOperator` automatically inject DAG parameters into the named CLI parameters of Python wheel tasks since `apache-airflow-providers-databricks` version 7.16. If `named_parameters` is already set then no injection happens, but if `parameters` are already set then injection does still happen. Because setting both `named_parameters` and `parameters` is invalid, the task fails. Note: An opt-out is available for `DatabricksRunNowOperator` (by passing `forward_dag_params=False`) but not for `DatabricksSubmitRunOperator`. ### What you think should happen instead? My preference would be to remove this parameter injection behaviour altogether. After all, [Explicit is better than implicit](https://peps.python.org/pep-0020/) and I suspect it causes more problems than it solves. Furthermore, if the databricks provider follows semver (?) it should not have been released as part of a minor version update because it is very much breaking. Should the maintainers wish to preserve this parameter injection, then it should be modified to only take place when no `named_parameters` **and** no `parameters` are already set. ### Operating System Linux ### Deployment Official Apache Airflow Helm Chart ### Apache Airflow Provider(s) databricks ### Versions of Apache Airflow Providers I'm on `apache-airflow-providers-databricks` version 7.20 but I suspect the issue exists since 7.16. ### Official Helm Chart version Not Applicable ### Kubernetes Version _No response_ ### Helm Chart configuration _No response_ ### Docker Image customizations _No response_ ### Anything else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
