seanghaeli opened a new pull request, #69838:
URL: https://github.com/apache/airflow/pull/69838
## Problem
The `example_neptune_analytics` system test fails on the **deferrable** path
while passing on the non-deferrable path. Two pre-existing bugs in the Neptune
Analytics code (both introduced together in #64274, never exercised by CI
because system tests only run in the MWAA pipeline):
1. **Custom waiter never loads.** `NeptuneAnalyticsHook` sets `client_type =
"neptune-graph"`, and `AwsGenericHook.waiter_path` derives the waiter filename
from `client_type` (→ `waiters/neptune-graph.json`). But the file shipped as
`waiters/neptune_analytics.json` (underscore), so `_list_custom_waiters()`
returns `[]` and `get_waiter("import_task_cancelled")` silently falls through
to botocore's **official** `ImportTaskCancelled` waiter. That official waiter
treats any status other than `CANCELLING`/`CANCELLED` as **failure** — so a
small, fast-completing import that reaches `SUCCEEDED` before the cancel poll
raises `NeptuneImportTaskCancellationFailedError`. The team's custom acceptors
(which correctly treat `SUCCEEDED`/`FAILED`/`CANCELLED` as success) were dead
code.
2. **Region/verify/botocore_config not forwarded to triggers.** Every
`self.defer(trigger=Neptune*Trigger(...))` passed only `aws_conn_id` + resource
ids + waiter timing, omitting `region_name`, `verify`, `botocore_config`. The
trigger's `hook()` then rebuilt the async hook with `region_name=None`. The
sync path uses `self.hook` (region resolved from the connection) so it passes;
the triggerer (separate process) can fail with `NoRegionError` / wrong-region
`ResourceNotFound`.
## Fix
1. `git mv waiters/neptune_analytics.json waiters/neptune-graph.json` so the
filename matches `client_type` and the custom waiter loads. No code referenced
the old name (it's derived), so the rename is safe.
2. Forward `region_name=self.region_name`, `verify=self.verify`,
`botocore_config=self.botocore_config` to all **8** `self.defer(...)` sites.
The triggers already accept these via `AwsBaseWaiterTrigger`; the operators
just weren't passing them.
## Tests
- New `waiters/test_neptune_analytics.py`: asserts the custom waiter loads
and behaviorally that `SUCCEEDED`/`FAILED`/`CANCELLED` are success,
`ERROR_ENCOUNTERED` fails.
- New `TestNeptuneDeferForwardsHookParams` (7 tests, one per deferrable
operator): asserts region/verify/config reach the trigger. All failed against
the unfixed operator, pass after the fix.
- Full neptune suite: **90 passed**; entire waiters test dir (rename
regression guard): **148 passed**; ruff + prek clean.
## Verification note
The waiter bug (#1) was reproduced locally with the exact botocore error
string. A verbatim triggerer traceback could not be pulled from the pipeline
CloudWatch groups, so "same failure as pipeline" rests on mechanism match
(medium-high confidence), not a matched log line. Full Breeze system-test
against live AWS not run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
