AlejandroMorgante opened a new pull request, #69930:
URL: https://github.com/apache/airflow/pull/69930
Split the AWS Glue crawler lifecycle into explicit create, update, run, and
delete operators while keeping the existing `GlueCrawlerOperator` backward
compatible during its deprecation period.
## Motivation
`GlueCrawlerOperator` currently combines resource provisioning,
configuration changes, and execution. This means a Dag intended only to run an
existing crawler can also create or modify it. Operation-specific operators
make each task's intent and required AWS permissions explicit.
## Operators
- `GlueCrawlerCreateOperator` creates a crawler from its Boto3 configuration.
- `GlueCrawlerUpdateOperator` updates an existing crawler without starting
it.
- `GlueCrawlerRunOperator` starts an existing crawler and supports
synchronous or deferrable completion waits through the existing
`GlueCrawlerCompleteTrigger`.
- `GlueCrawlerDeleteOperator` deletes an existing crawler through the Boto3
client exposed by `GlueCrawlerHook`.
The existing `GlueCrawlerOperator` remains available for compatibility and
emits an `AirflowProviderDeprecationWarning` directing new Dags to the
operation-specific operators.
## System test
The Glue system test now validates the complete lifecycle:
1. Create the crawler.
2. Update its description and verify the change through Boto3.
3. Run it with `deferrable=True`.
4. Run it with `deferrable=False`.
5. Delete it during teardown with the new delete operator.
## Validation
Validated end-to-end against a real AWS environment. The system test
created, updated, ran, and deleted the Glue crawler, then confirmed that the
crawler, Glue database, Glue job, and S3 bucket no longer existed.
The system test can be reproduced from the Airflow checkout with
`AWS_PROFILE`, `AWS_REGION`, and `ROLE_ARN` configured in
`files/airflow-breeze-config/environment_variables.env`:
```bash
SYSTEM_TESTS_ENV_ID=<unique-id> \
breeze testing system-tests \
--forward-credentials \
--test-timeout 240 \
providers/amazon/tests/system/amazon/aws/example_glue.py \
-q
```
Latest result: `1 passed in 191.86s`. The focused Glue crawler operator and
trigger tests also pass: `66 passed`, with 100% coverage of the modified
operator module. Provider mypy, regular and manual prek checks, and the Amazon
provider documentation build also pass.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Codex (GPT-5)
Generated-by: Codex (GPT-5) following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
