o-nikolas opened a new pull request, #71745: URL: https://github.com/apache/airflow/pull/71745
The `example_bedrock_retrieve_and_generate` system test intermittently fails in the ingestion job sensor with: ``` Bedrock ingestion job sensor failed. Failure reasons: ["Encountered error: Request failed: [security_exception] authentication/authorization failure. Call to Amazon OpenSearch Serverless Vector Database did not succeed."] ``` (This failure reason became visible thanks to #71124.) OpenSearch Serverless data access policies propagate **per principal**. The existing index readiness probe authenticates as the task's own principal, which proves nothing about the Knowledge Base execution role that actually performs the ingestion. When the collection activates quickly, Bedrock can call AOSS before the access policy has propagated for the Knowledge Base role, and the ingestion job fails. The deferrable path never hits this because its longer runtime provides enough propagation slack. This PR assumes the Knowledge Base role and runs the readiness probe with its credentials, so the probe only passes once the ingesting principal is authorized. No retry changes are needed: `AuthorizationException` and `AuthenticationException` subclass `TransportError`, so the existing tenacity retry (30 attempts x 5s) already absorbs the interim 403s. Note for AWS-side reviewers: this requires the test execution role to be allowed to assume the Knowledge Base role. That trust policy change is deployed on our side (internal ref: CR-297204508 / P482595782). --- ^ Add meaningful description above Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
