vgkowski opened a new pull request, #70768:
URL: https://github.com/apache/airflow/pull/70768
Add `EmrServerlessStopSessionOperator`, which terminates an EMR Serverless
interactive
(Spark Connect) session and optionally waits until it reaches the
`TERMINATED` state.
Why:
- Interactive sessions keep provisioned Spark drivers/executors running
until they are
terminated or hit their idle timeout, so users need an explicit way to
tear a session
down as part of a DAG.
Changes:
- `EmrServerlessHook.terminate_session` calls the `TerminateSession` API.
- `EmrServerlessStopSessionTrigger` and the `serverless_session_terminated`
waiter support
synchronous waiting and deferrable mode (parity with the existing
`EmrServerlessStopApplicationOperator`).
- Docs (howto section), a system-test example step, and unit tests
(including deferrable and
`execute_complete` paths).
The feature is gated on `botocore >= 1.43.0` at runtime (introduced in the
base PR), so the
provider's dependency floor is unchanged.
- related: #69207
> [!NOTE]
> This PR is the top of a stack and depends on:
> - https://github.com/apache/airflow/pull/70763
(`EmrServerlessStartSessionOperator` + shared session plumbing)
> - https://github.com/apache/airflow/pull/70766
(`EmrServerlessGetSessionEndpointOperator`)
>
> Please review/merge those first. This PR's base branch is
> `amazon-emr-serverless-get-session-endpoint` until the earlier PRs land,
then it retargets
> to `main`.
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Kiro (AI coding assistant)
Generated-by: Kiro, following the Gen-AI guidelines in
contributing-docs/05_pull_requests.rst
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
