atikulmunna opened a new pull request, #70723:
URL: https://github.com/apache/airflow/pull/70723
Moves the `match_glob` support check out of `GCSToS3Operator.__init__` so it
runs after template rendering.
`match_glob` is a template field, so the constructor only ever sees the
un-rendered Jinja expression. The check `if not self.__is_match_glob_supported
and match_glob:` was therefore testing a Jinja string for truthiness, which
meant a templated `match_glob` got rejected at Dag-parse time no matter what it
rendered to, and the failure showed up as a Dag import error rather than
against the task.
This one inspects the value rather than just asking whether the argument was
passed, so going by the guidance in #70296 it moves to `execute()` instead of
being rewritten in place. The check now sits on the `elif` branch of the
existing `__is_match_glob_supported` test, which is the point where the
rendered value would otherwise be handed to `GCSHook.list()`.
Same change as #70574, which did this for the sibling
`GCSToAzureBlobStorageOperator` in
`providers/microsoft/azure/.../transfers/gcs_to_wasb.py`.
This change:
* removes the `match_glob` support check from `__init__`
* raises from `execute()` instead, on the `elif` branch of the existing
`__is_match_glob_supported` test
* adds `test_execute__match_glob_requires_recent_google_provider`, since
nothing covered this error before
* removes `GCSToS3Operator` from
`scripts/ci/prek/validate_operators_init_exemptions.txt`
I left the exception as `AirflowException` rather than narrowing it to
`ValueError` the way #70359 did for `S3DeleteObjectsOperator`, just to keep
this diff to the move itself and not change behaviour for anyone catching
`AirflowException`. Glad to narrow it here too if you'd rather have it in the
same PR.
@shahar1
Ran locally, both clean:
```bash
python scripts/ci/prek/validate_operators_init.py \
providers/amazon/src/airflow/providers/amazon/aws/transfers/gcs_to_s3.py
ruff check && ruff format --check # on the two changed files
```
Being upfront: I couldn't run the unit tests locally. I'm on Windows without
a working Docker daemon, so breeze wasn't available, and Airflow doesn't
support running the suite natively there. Relying on CI for this one:
```bash
breeze run pytest
providers/amazon/tests/unit/amazon/aws/transfers/test_gcs_to_s3.py
```
If CI turns anything up I'll fix it on the branch.
related: #70296
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: Claude Code following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
---
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
