raph-p commented on issue #37217:
URL: https://github.com/apache/airflow/issues/37217#issuecomment-3266776436
In case somebody encountered this issue, a quick & dirty hack is to extend
the `CloudBatchSubmitJobOperator` class by creating a custom class that
inherits from it. It adds the whole `job` field as template, allowing to pass
the command (or env vars, or even more) as jinja template.
We also decided to add the `job_name`as well as the job_name, in case some
would pass a dynamic name (i.e. including the execution date for example), as
the name of a Batch Job must be unique.
I'm not sure I'll have the time to do a fix in the `airflow.provider`
myself, as I've never contributed to an open-source project and I'm not sure
where to start.
```python
class CustomCloudBatchSubmitJobOperator(CloudBatchSubmitJobOperator):
"""
Extends CloudBatchSubmitJobOperator by allowing jinja substitution of the
"job_name" and "job"
"""
template_fields = ("project_id", "region", "gcp_conn_id",
"impersonation_chain", "job_name", "job"). # this is the important line
def __init__(
self,
project_id: str,
region: str,
job_name: str,
job: dict | Job,
polling_period_seconds: float = 10,
timeout_seconds: float | None = None,
gcp_conn_id: str = "google_cloud_default",
impersonation_chain: str | Sequence[str] | None = None,
deferrable: bool = conf.getboolean("operators",
"default_deferrable", fallback=False),
**kwargs,
) -> None:
super().__init__(
project_id=project_id,
region=region,
job_name=job_name,
job=job,
polling_period_seconds=polling_period_seconds,
timeout_seconds=timeout_seconds,
gcp_conn_id=gcp_conn_id,
impersonation_chain=impersonation_chain,
deferrable=deferrable,
**kwargs,
)
```
Regards
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
