dongjoon-hyun opened a new pull request, #58361: URL: https://github.com/apache/spark/pull/58361
### What changes were proposed in this pull request? This PR adds `(hold)` and `(resume)` controls to the standalone Master web UI, next to the existing `(kill)` link, building on the hold status reported by SPARK-59055: - https://github.com/apache/spark/pull/58322 The Master does not hold the application itself; it forwards the request to the driver, which owns the hold (`SparkContext.holdExecutors()` / `resumeExecutors()`, SPARK-58828). Holding it on the Master side instead would bypass the driver's registration guard, be undone by the next `requestTotalExecutors`, and disagree with the driver's own `/holdstatus` endpoint. ```mermaid sequenceDiagram participant W as Master Web UI participant M as Master participant C as StandaloneAppClient participant B as StandaloneSchedulerBackend participant S as SparkContext W->>M: POST /app/hold or /app/resume<br/>RequestApplicationHold (send) M->>C: SetApplicationHold (ask, not awaited on the dispatcher) C->>B: listener.holdApplication Note over B,S: dedicated single thread (standalone-hold-resume) B->>S: holdExecutors() / resumeExecutors() S-->>B: reportExecutorHoldStatus(supported, held) B-->>C: reportHoldStatus C-->>M: ApplicationHoldUpdated (SPARK-59055) C-->>M: reply whether acknowledged (logged on the Master) ``` - Two new messages: `RequestApplicationHold` (Master UI to Master) and `SetApplicationHold` (Master to app client, replying whether the request was acknowledged). The resulting state comes back through the existing `ApplicationHoldUpdated` push, unchanged from SPARK-59055. - The controls follow the existing `(kill)` pattern: a `POST`-only form, registered under `spark.ui.holdEnabled` the way the kill handlers are registered under `spark.ui.killEnabled`, and requiring modify permissions. The generic `confirm-link` handler in `webui.js` now submits the form it is wrapped in, so it can guard a `POST` control and not only a navigation. - `spark.ui.holdEnabled` on the application is honored as a static policy: it travels with the registration as a new `ApplicationDescription.holdEnabled` field, following the `initialExecutorLimit` precedent. The Master offers no controls for an application that disabled it and rejects its hold requests without the round trip to the driver, which also rejects them as a second line of defense. The dynamic hold status reported by SPARK-59055 is untouched, so such an application stays fully visible -- state annotation, `held`, and `draining` -- when it is held programmatically; only the controls are withheld. Since the description is persisted for recovery, a failed-over Master knows the policy before the driver re-reports. Nothing blocks on the driver: the Master `send`s rather than asks from the Jetty thread, the Master does not wait on its ask to the driver, and the driver serves the request on a dedicated single-thread pool rather than on an RPC endpoint thread. Like the kill links, the request itself gets no inline feedback: the UI reflects the resulting state, and the outcome of the request is logged on the Master. No new configuration and no new public API. ### Why are the changes needed? SPARK-59055 made the hold status visible on the Master page, but acting on it still requires reaching each driver's own web UI. In a cluster where drivers are not directly reachable, the Master page is the operator's only console, and it is already where applications are killed. Holding is the graceful alternative to killing -- the application gives back its executors, keeps its driver and the shuffle output already written, and can be resumed later -- so it belongs next to `(kill)`. ### Does this PR introduce _any_ user-facing change? Yes, additive. Running applications that report themselves as holdable gain a `(hold)` link, and a `(resume)` link while held, on the Master web UI. No existing endpoint or field changes its meaning. ### How was this patch tested? Pass the CIs with new unit tests: - `MasterWebUISuite`: `/app/hold/` and `/app/resume/` forward the request to the Master; the page offers the hold or the resume control but never both, and offers neither for an application that disabled holding while still annotating its state. - `ReadOnlyMasterWebUISuite`: both endpoints are absent when `spark.ui.holdEnabled` is false on the Master. - `AppClientSuite`: a hold and a resume requested from a real Master reach the driver's listener, and a hold request for an application that disabled holding is rejected by the Master without reaching the driver. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Fable 5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
