hellodml opened a new issue, #18570: URL: https://github.com/apache/dolphinscheduler/issues/18570
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues. ### What happened On a standalone deployment, the master can enter a state where it permanently stops consuming commands and never recovers without a restart. Commands pile up in t_ds_command while the master logs The current server is overload, cannot consumes commands. indefinitely. We traced it to a self-reinforcing loop between server-load protection and slot rebalancing: 1.A scheduling burst pushes the live workflow-instance count above master.server-load-protection.max-concurrent-workflow-instances. MasterServerLoadProtection.isOverload() returns true and the master reports itself as ServerStatus.BUSY in its heartbeat. 2.MasterSlotManager.doReBalance() looks for the current master inside the NORMAL-only server list: `// MasterClusters.getNormalServers() .filter(masterServer -> masterServer.getServerStatus() == ServerStatus.NORMAL)` Because the master just marked itself BUSY, it is no longer in that list, so it cannot find itself: `WARN MasterSlotManager - Do rebalance failed, cannot found the current master: 172.18.0.2:5678` currentSlot is set to -1. 3.IdSlotBasedCommandFetcher.fetchCommands() then short-circuits: `WARN IdSlotBasedCommandFetcher - MasterSlotManager check slot (-1 -> 1) is invalidated.` No commands are fetched at all. 4.Commands accumulate. Any workflow instance already created but stuck (see the secondary issue below) keeps the instance count above the threshold, so the master stays BUSY forever. The condition that stops the master from working is the same condition that prevents it from recovering. **Secondary issue: duplicate command handling leaks tasks** While the slot flaps between valid and invalid, an already-fetched-but-not-yet-deleted command can be fetched a second time. Deduplication relies on the delete succeeding: `// WorkflowExecutionFactory.deleteCommandOrThrow() boolean deleteResult = commandDao.deleteById(command.getId()); if (!deleteResult) { throw new CommandDuplicateHandleException(command); }` The second handler throws, and CommandEngine marks the workflow instance FAILURE and moves the command to t_ds_error_command. But the first handler has already published TaskStartLifecycleEvent and inserted the task into the task-group queue. The result is an inconsistent pair that nothing ever reconciles: - t_ds_workflow_instance.state = 6 (FAILURE) (or stuck at 1 RUNNING) - t_ds_task_instance.state = 0 (SUBMITTED), start_time/host/pid all null, no log file Those workflow instances count toward max-concurrent-workflow-instances forever, which feeds straight back into step 1. ### What you expected to happen 1.A master should not exclude itself from slot rebalancing because of its own load status. Load protection should throttle intake, not make the node invisible to its own slot computation — otherwise overload becomes unrecoverable. 2.When CommandDuplicateHandleException is raised, the tasks already dispatched by the winning handler should be reconciled rather than left in state=0 indefinitely. ### How to reproduce 1.Standalone deployment, single master. 2.Set master.server-load-protection.max-concurrent-workflow-instances below the size of one scheduling burst (we had ~44 workflows firing on the same cron tick with the limit at 20). 3.Let it run. Each burst trips overload; occasionally a duplicate-handled command leaves a workflow instance stuck. Once the number of stuck instances reaches the limit, the master stops consuming commands permanently. ### Anything else <h2 style="box-sizing: border-box; border: 0px solid color(srgb 0.0431373 0.0431373 0.0431373 / 0.1); margin: 0px 0px 16px; padding: 0px; outline-color: rgb(37, 106, 191); scrollbar-width: thin; scrollbar-color: rgba(30, 30, 29, 0.35) rgba(0, 0, 0, 0); font-size: 17px; font-weight: 580; line-height: 22px; letter-spacing: -0.01em; color: rgb(11, 11, 11); margin-block: 24px 8px; padding-block: 0px; font-family: anthropic-sans, system-ui, "Segoe UI", Roboto, Helvetica, Arial, sans-serif; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(252, 252, 251); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">Observed impact</h2><p style="box-sizing: border-box; border: 0px solid color(srgb 0.0431373 0.0431373 0.0431373 / 0.1); ma rgin: 0px 0px 16px; padding: 0px; outline-color: rgb(37, 106, 191); scrollbar-width: thin; scrollbar-color: rgba(30, 30, 29, 0.35) rgba(0, 0, 0, 0); color: rgb(11, 11, 11); font-family: anthropic-sans, system-ui, "Segoe UI", Roboto, Helvetica, Arial, sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(252, 252, 251); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">Production standalone instance, 57 workflows on 10-minute crons:</p><div class="overflow-x-auto" style="box-sizing: border-box; border: 0px solid color(srgb 0.0431373 0.0431373 0.0431373 / 0.1); margin: 0px 0px 16px; padding: 0px; outline-color: rgb(37, 106, 191); scrollbar-width: thin; scrol lbar-color: rgba(30, 30, 29, 0.35) rgba(0, 0, 0, 0); overflow-x: auto; color: rgb(11, 11, 11); font-family: anthropic-sans, system-ui, "Segoe UI", Roboto, Helvetica, Arial, sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(252, 252, 251); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"> max-concurrent-workflow-instances | Do rebalance failed/h | cannot consumes commands/h -- | -- | -- 20 (burst size 44) | 56 – 120 | 1438 – 3597 100 | 13 | 160 </div><p style="box-sizing: border-box; border: 0px solid color(srgb 0.0431373 0.0431373 0.0431373 / 0.1); margin: 0px 0px 16px; padding: 0px; outline-color: rgb(37, 106, 191); scrollbar-width: thin; scrollbar-color: rgba(30, 30, 29, 0.35) rgba(0, 0, 0, 0); color: rgb(11, 11, 11); font-family: anthropic-sans, system-ui, "Segoe UI", Roboto, Helvetica, Arial, sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(252, 252, 251); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">Worst incident:<span> </span><strong style="box-sizing: border-box; border: 0px solid color(srgb 0.0431373 0.0431373 0.0431373 / 0.1); margin: 0px; padding: 0px; outline-color: rgb( 37, 106, 191); scrollbar-width: thin; scrollbar-color: rgba(30, 30, 29, 0.35) rgba(0, 0, 0, 0); font-weight: 500;">100 minutes of total scheduling outage</strong>, 34 consecutive batches / 690 workflow instances never executed, requiring a restart and manual data backfill.</p><p style="box-sizing: border-box; border: 0px solid color(srgb 0.0431373 0.0431373 0.0431373 / 0.1); margin: 0px 0px 16px; padding: 0px; outline-color: rgb(37, 106, 191); scrollbar-width: thin; scrollbar-color: rgba(30, 30, 29, 0.35) rgba(0, 0, 0, 0); color: rgb(11, 11, 11); font-family: anthropic-sans, system-ui, "Segoe UI", Roboto, Helvetica, Arial, sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(252, 252, 251); text-decoratio n-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">Raising the limit well above the burst size is an effective workaround because it keeps the master out of BUSY, but it is a workaround — any condition that makes a master BUSY (cpu/memory/disk thresholds too) will trigger the same deadlock.</p><h2 style="box-sizing: border-box; border: 0px solid color(srgb 0.0431373 0.0431373 0.0431373 / 0.1); margin: 0px 0px 16px; padding: 0px; outline-color: rgb(37, 106, 191); scrollbar-width: thin; scrollbar-color: rgba(30, 30, 29, 0.35) rgba(0, 0, 0, 0); font-size: 17px; font-weight: 580; line-height: 22px; letter-spacing: -0.01em; color: rgb(11, 11, 11); margin-block: 24px 8px; padding-block: 0px; font-family: anthropic-sans, system-ui, "Segoe UI", Roboto, Helvetica, Arial, sans-serif; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; wor d-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(252, 252, 251); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">Suggested fix</h2><p style="box-sizing: border-box; border: 0px solid color(srgb 0.0431373 0.0431373 0.0431373 / 0.1); margin: 0px 0px 16px; padding: 0px; outline-color: rgb(37, 106, 191); scrollbar-width: thin; scrollbar-color: rgba(30, 30, 29, 0.35) rgba(0, 0, 0, 0); color: rgb(11, 11, 11); font-family: anthropic-sans, system-ui, "Segoe UI", Roboto, Helvetica, Arial, sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(252, 252, 251); text-decoration-thickness: initial; text-decoration-style: initia l; text-decoration-color: initial;">Make<span> </span><code node="[object Object]" data-epitaxy-inline-code="" style="box-sizing: border-box; border: 0px solid color(srgb 0.0431373 0.0431373 0.0431373 / 0.1); margin: 0px; padding: 0px; outline-color: rgb(37, 106, 191); scrollbar-width: thin; scrollbar-color: rgba(30, 30, 29, 0.35) rgba(0, 0, 0, 0); font-family: anthropic-mono, ui-monospace, monospace; font-feature-settings: "calt" 0, "liga" 0; font-variation-settings: normal; font-size: 12px; font-variant-ligatures: none; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(11, 11, 11, 0.04); color: rgb(11, 11, 11); border-radius: 4px; padding-block: 1px; padding-inline: 2px; line-height: 1.4;">MasterSlotManager.doReBalance()</code><span> </span>locate the current master regardless of its own<span> </span><code node="[object Object]" data-epitaxy-inline-code="" style="box-sizing: border-box; border: 0px solid color(srgb 0.0431373 0.0431373 0 .0431373 / 0.1); margin: 0px; padding: 0px; outline-color: rgb(37, 106, 191); scrollbar-width: thin; scrollbar-color: rgba(30, 30, 29, 0.35) rgba(0, 0, 0, 0); font-family: anthropic-mono, ui-monospace, monospace; font-feature-settings: "calt" 0, "liga" 0; font-variation-settings: normal; font-size: 12px; font-variant-ligatures: none; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(11, 11, 11, 0.04); color: rgb(11, 11, 11); border-radius: 4px; padding-block: 1px; padding-inline: 2px; line-height: 1.4;">ServerStatus</code>, or keep the last known-good slot when the master is not found in the NORMAL list, instead of setting<span> </span><code node="[object Object]" data-epitaxy-inline-code="" style="box-sizing: border-box; border: 0px solid color(srgb 0.0431373 0.0431373 0.0431373 / 0.1); margin: 0px; padding: 0px; outline-color: rgb(37, 106, 191); scrollbar-width: thin; scrollbar-color: rgba(30, 30, 29, 0.35) rgba(0, 0, 0, 0); font-family: anthropic-mono, ui-monospace, monospace; font-feature-settings: "calt" 0, "liga" 0; font-variation-settings: normal; font-size: 12px; font-variant-ligatures: none; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(11, 11, 11, 0.04); color: rgb(11, 11, 11); border-radius: 4px; padding-block: 1px; padding-inline: 2px; line-height: 1.4;">currentSlot = -1</code>.</p><br class="Apple-interchange-newline"> Environment Deployment: standalone (Docker), single master Registry: JDBC (MySQL/PolarDB) JDK 8 ### Version 3.4.2 ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
