Joel Croteau created AIRFLOW-4408: ------------------------------------- Summary: Add template expansion support to properties arguments for DataProc operators Key: AIRFLOW-4408 URL: https://issues.apache.org/jira/browse/AIRFLOW-4408 Project: Apache Airflow Issue Type: Improvement Components: contrib, gcp, operators Affects Versions: 1.10.3 Reporter: Joel Croteau
Most GCP Dataproc operators take a `properties` argument of some sort, in the form of a `dict` of string->string that sets details of how the job is to be run on the cluster and other useful properties. It would be very nice if we could expand templates in the components of this `dict` in the same way we do for other string arguments. In particular, I am enabling log aggregation on a cluster, and would like to specify `yarn:yarn.nodemanager.remote-app-log-dir` to include the cluster name, which itself includes a template for the run date. It seems like it would be easy enough to iterate through the keys and values of the dict and build a new dict with the template-expanded results. I guess one potential problem with this is it might be possible that distinct templated keys might resolve to the same string after expansion, but this seems like a corner case we could deal with by raising an exception, or even just not expand keys and only expand values, as I'm not really sure how useful templated property names would actually be. -- This message was sent by Atlassian JIRA (v7.6.3#76005)