potiuk opened a new issue #22188: URL: https://github.com/apache/airflow/issues/22188
### Description I've seen recently at least two cases that the users wanted to use constructor kwargs values in jinja templates as part of the Jinja `Context`. ### Use case/motivation While this is not a trivial thing (because of potentially recursive references - even going through a chain of dependencies), this seems as an appealing use case. A good example (slightly modified: https://apache-airflow.slack.com/archives/CCR6P6JRL/p1647017726404589?thread_ts=1646967780.298339&cid=CCR6P6JRL) : ``` copy_into_table = SnowflakeOperator( task_id='copy_into_table', snowflake_conn_id='snflk_conn_id', sql='load_abc.sql', warehouse='main_xs_wh', schema='DBT_BASE', sql_params={ 'snflk_schema':'DBT_BASE', 's3_path': '@public.sf_stage/airflow/wsq-extract/{{var.value.snflk_env_var}}/', 'file_format': 'public.skip_header', 'batch_id':batch_id }, ) ``` In the sql file ``` FROM {{kwargs.sql_params.s3_path}}{{ds.replace('-', '/')}}/{{execution_date.int_timestamp}}.csv.gz (file_format=> {{kwargs.sql_params.file_format}}) ``` This allows to easily customize the content of rendered JINJA parameters not only based on state of the dag or ti, but also on the args of the Operator, which allows to reuse the operator more easily with different parameters relying on Airflow's Jinja engine and no-need to modify the code of existing operators, rather than writing extra execute() code to parse and process the parameters. There are a few things that might be discussed here: 1) The {{ kwargs }} should only keep the original "non-templated" copy of the values of the kwargs. I.e. if there are some templated_fields there, the original `{{ jinja }}` value should be returned when kwarg is accessed (otherwise we are vulnerable to recursion or we start to depend on sequence of processing of the kwargs by the templating engine) 2) We should consider to have some "generic" kwarg name for BaseOperator ("extra" ?). Currently when we pass an unknown kwarg, to Base Operator vhen `operators/allow_illegal_arguments` config is set to True (which is default and disabling it wil be dropped in the future). Otherwise it will be difficult to "add" new parameters dynamically - for example above "sql_params" is an illegal parameter in `SnowflakeOperator` - and it would be nice to have a way to pass such "jinj" context values to all operators. We could define such kwarg name and always allow it to be set on any operator. ### Related issues _No response_ ### Are you willing to submit a PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
