dloszewski-cni opened a new issue, #14214: URL: https://github.com/apache/cloudstack/issues/14214
### problem <p style="box-sizing: border-box; border: 0px solid color(srgb 1 1 1 / 0.1); margin: 0px 0px 16px; padding: 0px; outline-color: rgb(85, 152, 231); scrollbar-width: thin; app-region: initial; text-rendering: optimizelegibility; color: rgb(240, 239, 236); font-family: anthropic-sans, system-ui, "Segoe UI", Roboto, Helvetica, Arial, "PingFang SC", "PingFang TC", "Hiragino Sans", "Apple SD Gothic Neo", "Kohinoor Devanagari", "Kohinoor Bangla", "Kohinoor Telugu", "Tamil Sangam MN", "Kohinoor Gujarati", "Malayalam Sangam MN", "Nirmala UI", "Noto Sans Devanagari UI", "Noto Sans Devanagari", "Noto Sans Bengali UI", "Noto Sans Bengali", "Noto Sans Telugu UI", "Noto Sans Telugu", "Noto Sans Tamil UI", "Noto Sans Tamil", "Noto Sans Gujarati UI", "Noto Sans Gujarati", "No to Sans Kannada UI", "Noto Sans Kannada", "Noto Sans Malayalam UI", "Noto Sans Malayalam", Thonburi, "Leelawadee UI", "Noto Sans Thai UI", "Noto Sans Thai", Kefa, Ebrima, "Noto Sans Ethiopic", "Abyssinica SIL", sans-serif, ui-sans-serif, system-ui, -apple-system, "Segoe UI", Roboto, "PingFang SC", "PingFang TC", "Hiragino Sans", "Apple SD Gothic Neo", sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(26, 26, 25); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">A stale<span> </span><code data-epitaxy-inline-code="" style="box-s izing: border-box; border: 0px solid color(srgb 1 1 1 / 0.1); margin: 0px; padding: 0px; outline-color: rgb(85, 152, 231); scrollbar-width: thin; app-region: initial; font-family: anthropic-mono, ui-monospace, monospace, "SF Mono", ui-monospace, Menlo, Consolas, monospace; font-feature-settings: "calt" 0, "liga" 0; font-variation-settings: normal; font-size: 12px; font-variant-ligatures: none; text-rendering: optimizelegibility; background: none 0% 0% / auto repeat scroll padding-box border-box color(srgb 1 1 1 / 0.05); color: rgb(240, 239, 236); border-radius: 4px; padding-block: 1px; padding-inline: 2px; line-height: 1.4;">DetachVolume</code><span> </span>async job that completes<span> </span><strong style="box-sizing: border-box; border: 0px solid color(srgb 1 1 1 / 0.1); margin: 0px; padding: 0px; outline-color: rgb(85, 152, 231); scrollbar-width: thin; app-region: initial; font-weight: 500; --cds-font-axis-wght: "wght" 560,; font-variat ion-settings: "GRAD" 0, "wght" 560; text-rendering: optimizelegibility;">after</strong><span> </span>the same volume has already been re-attached (to another VM) clears the volume's attachment state in the database unconditionally. The result is a persistent split-brain: the DB shows the volume detached while the hypervisor still has the disk plugged into the original guest. Every subsequent<span> </span><code data-epitaxy-inline-code="" style="box-sizing: border-box; border: 0px solid color(srgb 1 1 1 / 0.1); margin: 0px; padding: 0px; outline-color: rgb(85, 152, 231); scrollbar-width: thin; app-region: initial; font-family: anthropic-mono, ui-monospace, monospace, "SF Mono", ui-monospace, Menlo, Consolas, monospace; font-feature-settings: "calt" 0, "liga" 0; font-variation-settings: normal; font-size: 12px; font-variant-ligatures: none; text-rendering: optimizelegibility; background: none 0% 0% / auto repeat scroll padding-box bo rder-box color(srgb 1 1 1 / 0.05); color: rgb(240, 239, 236); border-radius: 4px; padding-block: 1px; padding-inline: 2px; line-height: 1.4;">AttachVolume</code><span> </span>to that guest is then allocated the "free" device id and fails in libvirt with<span> </span><code data-epitaxy-inline-code="" style="box-sizing: border-box; border: 0px solid color(srgb 1 1 1 / 0.1); margin: 0px; padding: 0px; outline-color: rgb(85, 152, 231); scrollbar-width: thin; app-region: initial; font-family: anthropic-mono, ui-monospace, monospace, "SF Mono", ui-monospace, Menlo, Consolas, monospace; font-feature-settings: "calt" 0, "liga" 0; font-variation-settings: normal; font-size: 12px; font-variant-ligatures: none; text-rendering: optimizelegibility; background: none 0% 0% / auto repeat scroll padding-box border-box color(srgb 1 1 1 / 0.05); color: rgb(240, 239, 236); border-radius: 4px; padding-block: 1px; padding-inline: 2px; line-height: 1.4;">XML error: target ' vdX' duplicated</code>, permanently rejecting all volume attaches to that VM until an operator manually runs<span> </span><code data-epitaxy-inline-code="" style="box-sizing: border-box; border: 0px solid color(srgb 1 1 1 / 0.1); margin: 0px; padding: 0px; outline-color: rgb(85, 152, 231); scrollbar-width: thin; app-region: initial; font-family: anthropic-mono, ui-monospace, monospace, "SF Mono", ui-monospace, Menlo, Consolas, monospace; font-feature-settings: "calt" 0, "liga" 0; font-variation-settings: normal; font-size: 12px; font-variant-ligatures: none; text-rendering: optimizelegibility; background: none 0% 0% / auto repeat scroll padding-box border-box color(srgb 1 1 1 / 0.05); color: rgb(240, 239, 236); border-radius: 4px; padding-block: 1px; padding-inline: 2px; line-height: 1.4;">virsh detach-disk</code>.</p><p style="box-sizing: border-box; border: 0px solid color(srgb 1 1 1 / 0.1); margin: 0px 0px 16px; padding: 0px; outline-color: rgb(85, 152, 231); scrollbar-width: thin; app-region: initial; text-rendering: optimizelegibility; color: rgb(240, 239, 236); font-family: anthropic-sans, system-ui, "Segoe UI", Roboto, Helvetica, Arial, "PingFang SC", "PingFang TC", "Hiragino Sans", "Apple SD Gothic Neo", "Kohinoor Devanagari", "Kohinoor Bangla", "Kohinoor Telugu", "Tamil Sangam MN", "Kohinoor Gujarati", "Malayalam Sangam MN", "Nirmala UI", "Noto Sans Devanagari UI", "Noto Sans Devanagari", "Noto Sans Bengali UI", "Noto Sans Bengali", "Noto Sans Telugu UI", "Noto Sans Telugu", "Noto Sans Tamil UI", "Noto Sans Tamil", "Noto Sans Gujarati UI", "Noto Sans Gujarati", "Noto Sans Kannada UI", "Noto Sans Kannada", "Noto Sans Malayalam UI", "Noto Sans Malayalam", Thonburi, &q uot;Leelawadee UI", "Noto Sans Thai UI", "Noto Sans Thai", Kefa, Ebrima, "Noto Sans Ethiopic", "Abyssinica SIL", sans-serif, ui-sans-serif, system-ui, -apple-system, "Segoe UI", Roboto, "PingFang SC", "PingFang TC", "Hiragino Sans", "Apple SD Gothic Neo", sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(26, 26, 25); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">Root factors, as observed:</p><ol style="box-sizing: border-box; border: 0px solid color(srgb 1 1 1 / 0.1); margin: 0px 0px 16px; padding: 0px 0px 0px 20px; outline-color: rgb(85, 152, 231); scrollbar-width : thin; app-region: initial; list-style: decimal; text-rendering: optimizelegibility; gap: 6px; flex-direction: column; display: flex; color: rgb(240, 239, 236); font-family: anthropic-sans, system-ui, "Segoe UI", Roboto, Helvetica, Arial, "PingFang SC", "PingFang TC", "Hiragino Sans", "Apple SD Gothic Neo", "Kohinoor Devanagari", "Kohinoor Bangla", "Kohinoor Telugu", "Tamil Sangam MN", "Kohinoor Gujarati", "Malayalam Sangam MN", "Nirmala UI", "Noto Sans Devanagari UI", "Noto Sans Devanagari", "Noto Sans Bengali UI", "Noto Sans Bengali", "Noto Sans Telugu UI", "Noto Sans Telugu", "Noto Sans Tamil UI", "Noto Sans Tamil", "Noto Sans Gujarati UI", "Noto Sans Gujarati", "Noto Sans Kannada UI", "Noto Sans Kannada", "Noto Sans Malayalam UI", & quot;Noto Sans Malayalam", Thonburi, "Leelawadee UI", "Noto Sans Thai UI", "Noto Sans Thai", Kefa, Ebrima, "Noto Sans Ethiopic", "Abyssinica SIL", sans-serif, ui-sans-serif, system-ui, -apple-system, "Segoe UI", Roboto, "PingFang SC", "PingFang TC", "Hiragino Sans", "Apple SD Gothic Neo", sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(26, 26, 25); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><li style="box-sizing: border-box; border: 0px solid color(srgb 1 1 1 / 0.1); margin: 0px; padding: 0px; outline-color: rgb(85, 152, 231); scrollbar-width: thin; app-region: initial; text-rendering: optimizelegibility;">VM work jobs are serialized<span> </span><strong style="box-sizing: border-box; border: 0px solid color(srgb 1 1 1 / 0.1); margin: 0px; padding: 0px; outline-color: rgb(85, 152, 231); scrollbar-width: thin; app-region: initial; font-weight: 500; --cds-font-axis-wght: "wght" 560,; font-variation-settings: "GRAD" 0, "wght" 560; text-rendering: optimizelegibility;">per VM, not per volume</strong><span> </span>— so a detach of volume V from VM-A and an attach of volume V to VM-B run on independent queues with no ordering guarantee.</li><li style="box-sizing: border-box; border: 0px solid color(srgb 1 1 1 / 0.1); margin: 0px; padding: 0px; outline-color: rgb(85, 152, 231); scrollbar-width: thin; app-region: initial; text-rendering: optimizelegibility;">Clients that retry (Kubernetes CSI external-attacher, ~60 s retry) stack multiple identical<span> </span><code data-epitaxy-inline-code="" style=" box-sizing: border-box; border: 0px solid color(srgb 1 1 1 / 0.1); margin: 0px; padding: 0px; outline-color: rgb(85, 152, 231); scrollbar-width: thin; app-region: initial; font-family: anthropic-mono, ui-monospace, monospace, "SF Mono", ui-monospace, Menlo, Consolas, monospace; font-feature-settings: "calt" 0, "liga" 0; font-variation-settings: normal; font-size: 12px; font-variant-ligatures: none; text-rendering: optimizelegibility; background: none 0% 0% / auto repeat scroll padding-box border-box color(srgb 1 1 1 / 0.05); color: rgb(240, 239, 236); border-radius: 4px; padding-block: 1px; padding-inline: 2px; line-height: 1.4;">DetachVolume</code><span> </span>jobs on the source VM's queue. The first succeeds; the stragglers also "succeed" later.</li><li style="box-sizing: border-box; border: 0px solid color(srgb 1 1 1 / 0.1); margin: 0px; padding: 0px; outline-color: rgb(85, 152, 231); scrollbar-width: thin; app-region: initial; text-rendering: opti mizelegibility;">The straggler's completion path clears<span> </span><code data-epitaxy-inline-code="" style="box-sizing: border-box; border: 0px solid color(srgb 1 1 1 / 0.1); margin: 0px; padding: 0px; outline-color: rgb(85, 152, 231); scrollbar-width: thin; app-region: initial; font-family: anthropic-mono, ui-monospace, monospace, "SF Mono", ui-monospace, Menlo, Consolas, monospace; font-feature-settings: "calt" 0, "liga" 0; font-variation-settings: normal; font-size: 12px; font-variant-ligatures: none; text-rendering: optimizelegibility; background: none 0% 0% / auto repeat scroll padding-box border-box color(srgb 1 1 1 / 0.05); color: rgb(240, 239, 236); border-radius: 4px; padding-block: 1px; padding-inline: 2px; line-height: 1.4;">volumes.instance_id</code><span> </span>/<span> </span><code data-epitaxy-inline-code="" style="box-sizing: border-box; border: 0px solid color(srgb 1 1 1 / 0.1); margin: 0px; padding: 0px; outline-color: rgb(85, 152 , 231); scrollbar-width: thin; app-region: initial; font-family: anthropic-mono, ui-monospace, monospace, "SF Mono", ui-monospace, Menlo, Consolas, monospace; font-feature-settings: "calt" 0, "liga" 0; font-variation-settings: normal; font-size: 12px; font-variant-ligatures: none; text-rendering: optimizelegibility; background: none 0% 0% / auto repeat scroll padding-box border-box color(srgb 1 1 1 / 0.05); color: rgb(240, 239, 236); border-radius: 4px; padding-block: 1px; padding-inline: 2px; line-height: 1.4;">device_id</code><span> </span><strong style="box-sizing: border-box; border: 0px solid color(srgb 1 1 1 / 0.1); margin: 0px; padding: 0px; outline-color: rgb(85, 152, 231); scrollbar-width: thin; app-region: initial; font-weight: 500; --cds-font-axis-wght: "wght" 560,; font-variation-settings: "GRAD" 0, "wght" 560; text-rendering: optimizelegibility;">without verifying the row still refers to the attachment it deta ched</strong><span> </span>— no compare-and-swap. If the volume was re-attached in the meantime, the<span> </span><em style="box-sizing: border-box; border: 0px solid color(srgb 1 1 1 / 0.1); margin: 0px; padding: 0px; outline-color: rgb(85, 152, 231); scrollbar-width: thin; app-region: initial; text-rendering: optimizelegibility; font-style: italic;">new</em><span> </span>attachment's state is destroyed.</li><li style="box-sizing: border-box; border: 0px solid color(srgb 1 1 1 / 0.1); margin: 0px; padding: 0px; outline-color: rgb(85, 152, 231); scrollbar-width: thin; app-region: initial; text-rendering: optimizelegibility;">The corrupting event logs at INFO level as a successful detach — nothing flags the inconsistency until a later attach collides.</li></ol><h3 style="box-sizing: border-box; border: 0px solid color(srgb 1 1 1 / 0.1); margin: 0px 0px 16px; padding: 0px; outline-color: rgb(85, 152, 231); scrollbar-width: thin; app-region: initial; font-size: 15px; font-weight : 580; text-rendering: optimizelegibility; font-variation-settings: "wght" 540; letter-spacing: -0.01em; margin-block: 20px 6px; padding-block: 0px; color: rgb(240, 239, 236); font-family: anthropic-sans, system-ui, "Segoe UI", Roboto, Helvetica, Arial, "PingFang SC", "PingFang TC", "Hiragino Sans", "Apple SD Gothic Neo", "Kohinoor Devanagari", "Kohinoor Bangla", "Kohinoor Telugu", "Tamil Sangam MN", "Kohinoor Gujarati", "Malayalam Sangam MN", "Nirmala UI", "Noto Sans Devanagari UI", "Noto Sans Devanagari", "Noto Sans Bengali UI", "Noto Sans Bengali", "Noto Sans Telugu UI", "Noto Sans Telugu", "Noto Sans Tamil UI", "Noto Sans Tamil", "Noto Sans Gujarati UI", "Noto Sans Gujarati", "Noto Sans Kannada UI", "Noto Sans Kannada", "Noto Sa ns Malayalam UI", "Noto Sans Malayalam", Thonburi, "Leelawadee UI", "Noto Sans Thai UI", "Noto Sans Thai", Kefa, Ebrima, "Noto Sans Ethiopic", "Abyssinica SIL", sans-serif, ui-sans-serif, system-ui, -apple-system, "Segoe UI", Roboto, "PingFang SC", "PingFang TC", "Hiragino Sans", "Apple SD Gothic Neo", sans-serif; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(26, 26, 25); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">Observed occurrences (four in 20 days, production; all on 4.22.1.0, KVM, RBD primary storage)</h3><p style="box-sizing: border-box; border: 0px solid color(srgb 1 1 1 / 0.1); margin: 0px 0px 16px; padding: 0px; outline-color: rgb(85, 152, 231); scrollbar-width: thin; app-region: initial; text-rendering: optimizelegibility; color: rgb(240, 239, 236); font-family: anthropic-sans, system-ui, "Segoe UI", Roboto, Helvetica, Arial, "PingFang SC", "PingFang TC", "Hiragino Sans", "Apple SD Gothic Neo", "Kohinoor Devanagari", "Kohinoor Bangla", "Kohinoor Telugu", "Tamil Sangam MN", "Kohinoor Gujarati", "Malayalam Sangam MN", "Nirmala UI", "Noto Sans Devanagari UI", "Noto Sans Devanagari", "Noto Sans Bengali UI", "Noto Sans Bengali", "Noto Sans Telugu UI", "Noto Sans Telugu", "Noto Sans Tamil UI", "Noto Sans Tamil", "Noto Sans Gujarati UI", "Noto Sans Gujarati", "Noto Sans Kannada UI", "Noto Sans Kannada", "Noto Sans Malayalam UI", "No to Sans Malayalam", Thonburi, "Leelawadee UI", "Noto Sans Thai UI", "Noto Sans Thai", Kefa, Ebrima, "Noto Sans Ethiopic", "Abyssinica SIL", sans-serif, ui-sans-serif, system-ui, -apple-system, "Segoe UI", Roboto, "PingFang SC", "PingFang TC", "Hiragino Sans", "Apple SD Gothic Neo", sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(26, 26, 25); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><strong style="box-sizing: border-box; border: 0px solid color(srgb 1 1 1 / 0.1); margin: 0px; padding: 0px; outline-color: rgb(85, 152, 231); scrollbar-width: thin; app -region: initial; font-weight: 500; --cds-font-axis-wght: "wght" 560,; font-variation-settings: "GRAD" 0, "wght" 560; text-rendering: optimizelegibility;">Occurrence 1 — 2026-08-27/28, "worker7":</strong><span> </span>straggler detach<span> </span><strong style="box-sizing: border-box; border: 0px solid color(srgb 1 1 1 / 0.1); margin: 0px; padding: 0px; outline-color: rgb(85, 152, 231); scrollbar-width: thin; app-region: initial; font-weight: 500; --cds-font-axis-wght: "wght" 560,; font-variation-settings: "GRAD" 0, "wght" 560; text-rendering: optimizelegibility;">job-148750</strong><span> </span>cleared the state of an attachment established by job-148738; collisions observed via job-148762 et seq.</p><p style="box-sizing: border-box; border: 0px solid color(srgb 1 1 1 / 0.1); margin: 0px 0px 16px; padding: 0px; outline-color: rgb(85, 152, 231); scrollbar-width: thin; app-region: initial; text-rendering: optimizelegi bility; color: rgb(240, 239, 236); font-family: anthropic-sans, system-ui, "Segoe UI", Roboto, Helvetica, Arial, "PingFang SC", "PingFang TC", "Hiragino Sans", "Apple SD Gothic Neo", "Kohinoor Devanagari", "Kohinoor Bangla", "Kohinoor Telugu", "Tamil Sangam MN", "Kohinoor Gujarati", "Malayalam Sangam MN", "Nirmala UI", "Noto Sans Devanagari UI", "Noto Sans Devanagari", "Noto Sans Bengali UI", "Noto Sans Bengali", "Noto Sans Telugu UI", "Noto Sans Telugu", "Noto Sans Tamil UI", "Noto Sans Tamil", "Noto Sans Gujarati UI", "Noto Sans Gujarati", "Noto Sans Kannada UI", "Noto Sans Kannada", "Noto Sans Malayalam UI", "Noto Sans Malayalam", Thonburi, "Leelawadee UI", "Noto Sans Thai UI", "Noto Sans Thai", Kefa, Ebrima, "Noto Sans Ethiopic", "Abyssinica SIL", sans-serif, ui-sans-serif, system-ui, -apple-system, "Segoe UI", Roboto, "PingFang SC", "PingFang TC", "Hiragino Sans", "Apple SD Gothic Neo", sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(26, 26, 25); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><strong style="box-sizing: border-box; border: 0px solid color(srgb 1 1 1 / 0.1); margin: 0px; padding: 0px; outline-color: rgb(85, 152, 231); scrollbar-width: thin; app-region: initial; font-weight: 500; --cds-font-axis-wght: "wght" 560,; font-variation-settings: "GRAD&quo t; 0, "wght" 560; text-rendering: optimizelegibility;">Occurrence 2 — 2026-09-08, "worker7"/"worker8"</strong><span> </span>(management-server.log timeline, UTC):</p><div class="epitaxy-table-scroll" style="box-sizing: border-box; border: 0px solid color(srgb 1 1 1 / 0.1); margin: 0px 0px 16px; padding: 0px; outline-color: rgb(85, 152, 231); scrollbar-width: auto; app-region: initial; text-rendering: optimizelegibility; scrollbar-color: auto; overflow-x: auto; color: rgb(240, 239, 236); font-family: anthropic-sans, system-ui, "Segoe UI", Roboto, Helvetica, Arial, "PingFang SC", "PingFang TC", "Hiragino Sans", "Apple SD Gothic Neo", "Kohinoor Devanagari", "Kohinoor Bangla", "Kohinoor Telugu", "Tamil Sangam MN", "Kohinoor Gujarati", "Malayalam Sangam MN", "Nirmala UI", "Noto Sans Devanagari UI", "Noto Sans Devanagari", "Noto Sans Bengali UI", "Noto Sans Bengali", "Noto Sans Telugu UI", "Noto Sans Telugu", "Noto Sans Tamil UI", "Noto Sans Tamil", "Noto Sans Gujarati UI", "Noto Sans Gujarati", "Noto Sans Kannada UI", "Noto Sans Kannada", "Noto Sans Malayalam UI", "Noto Sans Malayalam", Thonburi, "Leelawadee UI", "Noto Sans Thai UI", "Noto Sans Thai", Kefa, Ebrima, "Noto Sans Ethiopic", "Abyssinica SIL", sans-serif, ui-sans-serif, system-ui, -apple-system, "Segoe UI", Roboto, "PingFang SC", "PingFang TC", "Hiragino Sans", "Apple SD Gothic Neo", sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit- text-stroke-width: 0px; white-space: normal; background-color: rgb(26, 26, 25); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"> Time | Job | Event -- | -- | -- 14:34:56 | job-210187 | DetachVolume #1 of volume e949a8c4 from worker1 (CSI-initiated) 14:35:56–14:37:57 | job-210205 / 210223 / 210235 | CSI ~60 s retries stack detaches #2–4 on worker1's queue 14:38:51 | job-210187 SUCCEEDED | real unplug from worker1 14:40:10→19 | job-210247 SUCCEEDED | volume attached to worker7, deviceid 2 (vdc), DB correct 14:42:40 | job-210235 SUCCEEDED | straggler completes 2m21s after the attach and clears the worker7 attachment row — orphan created; logged as a successful detach at INFO 14:43:04 onward | jobs 210253/210259/210265/210268/210277 FAILED 530 | attach collisions begin </div><p style="box-sizing: border-box; border: 0px solid color(srgb 1 1 1 / 0.1); margin: 0px 0px 16px; padding: 0px; outline-color: rgb(85, 152, 231); scrollbar-width: thin; app-region: initial; text-rendering: optimizelegibility; color: rgb(240, 239, 236); font-family: anthropic-sans, system-ui, "Segoe UI", Roboto, Helvetica, Arial, "PingFang SC", "PingFang TC", "Hiragino Sans", "Apple SD Gothic Neo", "Kohinoor Devanagari", "Kohinoor Bangla", "Kohinoor Telugu", "Tamil Sangam MN", "Kohinoor Gujarati", "Malayalam Sangam MN", "Nirmala UI", "Noto Sans Devanagari UI", "Noto Sans Devanagari", "Noto Sans Bengali UI", "Noto Sans Bengali", "Noto Sans Telugu UI", "Noto Sans Telugu", "Noto Sans Tamil UI", "Noto Sans Tamil", "Noto Sans Gujarati UI", "Noto Sans Gujarati", &q uot;Noto Sans Kannada UI", "Noto Sans Kannada", "Noto Sans Malayalam UI", "Noto Sans Malayalam", Thonburi, "Leelawadee UI", "Noto Sans Thai UI", "Noto Sans Thai", Kefa, Ebrima, "Noto Sans Ethiopic", "Abyssinica SIL", sans-serif, ui-sans-serif, system-ui, -apple-system, "Segoe UI", Roboto, "PingFang SC", "PingFang TC", "Hiragino Sans", "Apple SD Gothic Neo", sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(26, 26, 25); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">Two points specific to this occurrence:</p><ul style="box-sizi ng: border-box; border: 0px solid color(srgb 1 1 1 / 0.1); margin: 0px 0px 16px; padding: 0px 0px 0px 20px; outline-color: rgb(85, 152, 231); scrollbar-width: thin; app-region: initial; list-style: outside; text-rendering: optimizelegibility; gap: 6px; flex-direction: column; display: flex; color: rgb(240, 239, 236); font-family: anthropic-sans, system-ui, "Segoe UI", Roboto, Helvetica, Arial, "PingFang SC", "PingFang TC", "Hiragino Sans", "Apple SD Gothic Neo", "Kohinoor Devanagari", "Kohinoor Bangla", "Kohinoor Telugu", "Tamil Sangam MN", "Kohinoor Gujarati", "Malayalam Sangam MN", "Nirmala UI", "Noto Sans Devanagari UI", "Noto Sans Devanagari", "Noto Sans Bengali UI", "Noto Sans Bengali", "Noto Sans Telugu UI", "Noto Sans Telugu", "Noto Sans Tamil UI", "Noto Sans Tamil", "Noto S ans Gujarati UI", "Noto Sans Gujarati", "Noto Sans Kannada UI", "Noto Sans Kannada", "Noto Sans Malayalam UI", "Noto Sans Malayalam", Thonburi, "Leelawadee UI", "Noto Sans Thai UI", "Noto Sans Thai", Kefa, Ebrima, "Noto Sans Ethiopic", "Abyssinica SIL", sans-serif, ui-sans-serif, system-ui, -apple-system, "Segoe UI", Roboto, "PingFang SC", "PingFang TC", "Hiragino Sans", "Apple SD Gothic Neo", sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(26, 26, 25); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><li s tyle="box-sizing: border-box; border: 0px solid color(srgb 1 1 1 / 0.1); margin: 0px; padding: 0px; outline-color: rgb(85, 152, 231); scrollbar-width: thin; app-region: initial; text-rendering: optimizelegibility;">The stale attachment's consumer<span> </span><strong style="box-sizing: border-box; border: 0px solid color(srgb 1 1 1 / 0.1); margin: 0px; padding: 0px; outline-color: rgb(85, 152, 231); scrollbar-width: thin; app-region: initial; font-weight: 500; --cds-font-axis-wght: "wght" 560,; font-variation-settings: "GRAD" 0, "wght" 560; text-rendering: optimizelegibility;">kept running and doing I/O on worker2 for five days</strong><span> </span>while the DB said the volume was detached — the DB lost track of a live, in-use attachment. That is a data-integrity hazard: the volume could have been offered to a second VM while mounted read-write in the first, and a live detach by an operator trusting the DB would have pulled a disk out from under a running workload.</li><li style="box-sizing: border-box; border: 0px solid color(srgb 1 1 1 / 0.1); margin: 0px; padding: 0px; outline-color: rgb(85, 152, 231); scrollbar-width: thin; app-region: initial; text-rendering: optimizelegibility;">A stale detach that reaches the hypervisor path fails with an NPE in<span> </span><code data-epitaxy-inline-code="" style="box-sizing: border-box; border: 0px solid color(srgb 1 1 1 / 0.1); margin: 0px; padding: 0px; outline-color: rgb(85, 152, 231); scrollbar-width: thin; app-region: initial; font-family: anthropic-mono, ui-monospace, monospace, "SF Mono", ui-monospace, Menlo, Consolas, monospace; font-feature-settings: "calt" 0, "liga" 0; font-variation-settings: normal; font-size: 12px; font-variant-ligatures: none; text-rendering: optimizelegibility; background: none 0% 0% / auto repeat scroll padding-box border-box color(srgb 1 1 1 / 0.05); color: rgb(240, 239, 236); border-radius: 4px; padding-block: 1px; pad ding-inline: 2px; line-height: 1.4;">DiskTO.getDiskSeq()</code><span> </span>rather than a clean "volume is not attached to this VM" rejection; a stale detach that does<span> </span><em style="box-sizing: border-box; border: 0px solid color(srgb 1 1 1 / 0.1); margin: 0px; padding: 0px; outline-color: rgb(85, 152, 231); scrollbar-width: thin; app-region: initial; text-rendering: optimizelegibility; font-style: italic;">not</em><span> </span>reach that path (job-185059) succeeds and clears state it does not own. Which one a straggler hits looks timing-dependent.</li></ul><p style="box-sizing: border-box; border: 0px solid color(srgb 1 1 1 / 0.1); margin: 0px 0px 16px; padding: 0px; outline-color: rgb(85, 152, 231); scrollbar-width: thin; app-region: initial; text-rendering: optimizelegibility; color: rgb(240, 239, 236); font-family: anthropic-sans, system-ui, "Segoe UI", Roboto, Helvetica, Arial, "PingFang SC", "PingFang TC", "Hiragino Sans", "Apple SD Gothic Neo", "Kohinoor Devanagari", "Kohinoor Bangla", "Kohinoor Telugu", "Tamil Sangam MN", "Kohinoor Gujarati", "Malayalam Sangam MN", "Nirmala UI", "Noto Sans Devanagari UI", "Noto Sans Devanagari", "Noto Sans Bengali UI", "Noto Sans Bengali", "Noto Sans Telugu UI", "Noto Sans Telugu", "Noto Sans Tamil UI", "Noto Sans Tamil", "Noto Sans Gujarati UI", "Noto Sans Gujarati", "Noto Sans Kannada UI", "Noto Sans Kannada", "Noto Sans Malayalam UI", "Noto Sans Malayalam", Thonburi, "Leelawadee UI", "Noto Sans Thai UI", "Noto Sans Thai", Kefa, Ebrima, "Noto Sans Ethiopic", "Abyssinica SIL", sans-serif, ui-sans-serif, system-ui, -apple-system, "Segoe UI", Roboto, "PingFang SC", "PingFang T C", "Hiragino Sans", "Apple SD Gothic Neo", sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(26, 26, 25); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><strong style="box-sizing: border-box; border: 0px solid color(srgb 1 1 1 / 0.1); margin: 0px; padding: 0px; outline-color: rgb(85, 152, 231); scrollbar-width: thin; app-region: initial; font-weight: 500; --cds-font-axis-wght: "wght" 560,; font-variation-settings: "GRAD" 0, "wght" 560; text-rendering: optimizelegibility;">Occurrence 4 — attach-side variant, 2026-09-10 → 09-16, "worker4":</strong><span> </span>the mirror image of the straggler-detach cases: here the DB lost a<span> </span><strong style="box-sizing: border-box; border: 0px solid color(srgb 1 1 1 / 0.1); margin: 0px; padding: 0px; outline-color: rgb(85, 152, 231); scrollbar-width: thin; app-region: initial; font-weight: 500; --cds-font-axis-wght: "wght" 560,; font-variation-settings: "GRAD" 0, "wght" 560; text-rendering: optimizelegibility;">successful attach</strong>. Volume 1a6a7be4 (an RWO PVC backing a database StatefulSet member) was live at target vde / deviceid 4 on worker4 — libvirt and the Kubernetes VolumeAttachment both showed attached — while the volumes table showed it detached (state Ready, no instance). The corrupting job aged out of the async_job table before discovery (~6 days in), so the exact write-loss path is unrecovered, but the allocator consequence is identical: deviceid 4 is considered free, every subsequent AttachVolume to worker4 picks it, and fails<span> </span><code data-epitaxy-inline-code="" style ="box-sizing: border-box; border: 0px solid color(srgb 1 1 1 / 0.1); margin: 0px; padding: 0px; outline-color: rgb(85, 152, 231); scrollbar-width: thin; app-region: initial; font-family: anthropic-mono, ui-monospace, monospace, "SF Mono", ui-monospace, Menlo, Consolas, monospace; font-feature-settings: "calt" 0, "liga" 0; font-variation-settings: normal; font-size: 12px; font-variant-ligatures: none; text-rendering: optimizelegibility; background: none 0% 0% / auto repeat scroll padding-box border-box color(srgb 1 1 1 / 0.05); color: rgb(240, 239, 236); border-radius: 4px; padding-block: 1px; padding-inline: 2px; line-height: 1.4;">target 'vde' duplicated</code><span> </span>— measured<span> </span><strong style="box-sizing: border-box; border: 0px solid color(srgb 1 1 1 / 0.1); margin: 0px; padding: 0px; outline-color: rgb(85, 152, 231); scrollbar-width: thin; app-region: initial; font-weight: 500; --cds-font-axis-wght: "wght" 560,; font- variation-settings: "GRAD" 0, "wght" 560; text-rendering: optimizelegibility;">~3,700 failed attach events/day for six days</strong>, with four unrelated PVCs unable to schedule onto the VM.</p><p style="box-sizing: border-box; border: 0px solid color(srgb 1 1 1 / 0.1); margin: 0px 0px 16px; padding: 0px; outline-color: rgb(85, 152, 231); scrollbar-width: thin; app-region: initial; text-rendering: optimizelegibility; color: rgb(240, 239, 236); font-family: anthropic-sans, system-ui, "Segoe UI", Roboto, Helvetica, Arial, "PingFang SC", "PingFang TC", "Hiragino Sans", "Apple SD Gothic Neo", "Kohinoor Devanagari", "Kohinoor Bangla", "Kohinoor Telugu", "Tamil Sangam MN", "Kohinoor Gujarati", "Malayalam Sangam MN", "Nirmala UI", "Noto Sans Devanagari UI", "Noto Sans Devanagari", "Noto Sans Bengali UI", "Noto Sans Be ngali", "Noto Sans Telugu UI", "Noto Sans Telugu", "Noto Sans Tamil UI", "Noto Sans Tamil", "Noto Sans Gujarati UI", "Noto Sans Gujarati", "Noto Sans Kannada UI", "Noto Sans Kannada", "Noto Sans Malayalam UI", "Noto Sans Malayalam", Thonburi, "Leelawadee UI", "Noto Sans Thai UI", "Noto Sans Thai", Kefa, Ebrima, "Noto Sans Ethiopic", "Abyssinica SIL", sans-serif, ui-sans-serif, system-ui, -apple-system, "Segoe UI", Roboto, "PingFang SC", "PingFang TC", "Hiragino Sans", "Apple SD Gothic Neo", sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(26, 26, 25); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">This occurrence also<span> </span><em style="box-sizing: border-box; border: 0px solid color(srgb 1 1 1 / 0.1); margin: 0px; padding: 0px; outline-color: rgb(85, 152, 231); scrollbar-width: thin; app-region: initial; text-rendering: optimizelegibility; font-style: italic;">nearly realized</em><span> </span>the data-integrity hazard flagged in occurrence 3: when the consuming pod was later evicted, the CSI ControllerUnpublish "succeeded" as a no-op (the DB already said detached), and CloudStack then<span> </span><strong style="box-sizing: border-box; border: 0px solid color(srgb 1 1 1 / 0.1); margin: 0px; padding: 0px; outline-color: rgb(85, 152, 231); scrollbar-width: thin; app-region: initial; font-weight: 500; --cds-font-axis-wght: "wght" 560,; font-variation-settings: "GRAD" 0, "wght" 560; text-rendering: optimizelegibility;">attached the volume to a second VM while the original guest's qemu still held the RBD image open read-write</strong><span> </span>— a silent double-open of a database volume across two hypervisors, with no error or warning anywhere in the API path. No corruption resulted only because the first guest had unmounted the filesystem before the re-attach.</p><h3 style="box-sizing: border-box; border: 0px solid color(srgb 1 1 1 / 0.1); margin: 0px 0px 16px; padding: 0px; outline-color: rgb(85, 152, 231); scrollbar-width: thin; app-region: initial; font-size: 15px; font-weight: 580; text-rendering: optimizelegibility; font-variation-settings: "wght" 540; letter-spacing: -0.01em; margin-block: 20px 6px; padding-block: 0px; color: rgb(240, 239, 236); font-family: anthropic-sans, system-ui, "Segoe UI", Roboto, Helvetica, Arial, "PingFang SC", "PingFang TC", "Hiragino Sans", "Apple SD Gothic Neo", "Kohinoor Devanagari", "Kohinoor Bangla", "Kohinoor Telugu", "Tamil Sangam MN", "Kohinoor Gujarati", "Malayalam Sangam MN", "Nirmala UI", "Noto Sans Devanagari UI", "Noto Sans Devanagari", "Noto Sans Bengali UI", "Noto Sans Bengali", "Noto Sans Telugu UI", "Noto Sans Telugu", "Noto Sans Tamil UI", "Noto Sans Tamil", "Noto Sans Gujarati UI", "Noto Sans Gujarati", "Noto Sans Kannada UI", "Noto Sans Kannada", "Noto Sans Malayalam UI", "Noto Sans Malayalam", Thonburi, "Leelawadee UI", "Noto Sans Thai UI", "Noto Sans Thai", Kefa, Ebrima, "Noto Sans Ethiopic", "Abyssinica SIL", sans-serif, ui-sans-serif, system-ui, -apple-system, "Segoe UI", Roboto, "PingFang SC", "PingFang TC", "Hiragino Sans", "Apple SD Gothic Neo", sans-serif; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(26, 26, 25); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">Actual results</h3><p style="box-sizing: border-box; border: 0px solid color(srgb 1 1 1 / 0.1); margin: 0px 0px 16px; padding: 0px; outline-color: rgb(85, 152, 231); scrollbar-width: thin; app-region: initial; text-rendering: optimizelegibility; color: rgb(240, 239, 236); font-family: anthropic-sans, system-ui, "Segoe UI", Roboto, Helvetica, Arial, "PingFang SC", "PingFang TC", "Hiragino Sans", "Apple SD Gothic Neo", "Kohinoor Devanagari", "Kohinoor Bangla", "Kohinoor Telugu", "Tamil Sangam MN", &quo t;Kohinoor Gujarati", "Malayalam Sangam MN", "Nirmala UI", "Noto Sans Devanagari UI", "Noto Sans Devanagari", "Noto Sans Bengali UI", "Noto Sans Bengali", "Noto Sans Telugu UI", "Noto Sans Telugu", "Noto Sans Tamil UI", "Noto Sans Tamil", "Noto Sans Gujarati UI", "Noto Sans Gujarati", "Noto Sans Kannada UI", "Noto Sans Kannada", "Noto Sans Malayalam UI", "Noto Sans Malayalam", Thonburi, "Leelawadee UI", "Noto Sans Thai UI", "Noto Sans Thai", Kefa, Ebrima, "Noto Sans Ethiopic", "Abyssinica SIL", sans-serif, ui-sans-serif, system-ui, -apple-system, "Segoe UI", Roboto, "PingFang SC", "PingFang TC", "Hiragino Sans", "Apple SD Gothic Neo", sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-var iant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(26, 26, 25); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">Unconditional clear of the attachment row; persistent DB↔libvirt divergence; all subsequent attaches to the affected VM fail with libvirt duplicated-target errors (530) until manual operator intervention (<code data-epitaxy-inline-code="" style="box-sizing: border-box; border: 0px solid color(srgb 1 1 1 / 0.1); margin: 0px; padding: 0px; outline-color: rgb(85, 152, 231); scrollbar-width: thin; app-region: initial; font-family: anthropic-mono, ui-monospace, monospace, "SF Mono", ui-monospace, Menlo, Consolas, monospace; font-feature-settings: "calt" 0, "liga" 0; font-variation-settings: normal; font-size: 12 px; font-variant-ligatures: none; text-rendering: optimizelegibility; background: none 0% 0% / auto repeat scroll padding-box border-box color(srgb 1 1 1 / 0.05); color: rgb(240, 239, 236); border-radius: 4px; padding-block: 1px; padding-inline: 2px; line-height: 1.4;">virsh detach-disk <vm> <target> --live</code><span> </span>on the host).</p> ### versions 4.22.1.0 (KVM on Ubuntu, Ceph RBD primary storage, advanced networking) Also reviewed 4.23.0 (2026-09-17): no change to the DetachVolume completion / volume-attachment state path that would address this behavior, and no existing upstream issue or PR describing it that we could find. Component: Volume lifecycle / async job orchestration (DetachVolume completion), KVM Kubernetes CSI driver (csi.cloudstack.apache.org) drives the volume attach/detach traffic. ### The steps to reproduce the bug Environment: CloudStack 4.22.1.0, KVM, Ceph RBD primary storage, a Kubernetes cluster using the CloudStack CSI driver (csi.cloudstack.apache.org), where pod rescheduling moves an RWO PVC between worker VMs. 1. Have a data volume attached to worker VM-A. 2. Trigger rapid relocation of the volume (e.g. reschedule the consuming pod to VM-B). The CSI external-attacher submits DetachVolume; under load its ~60 s retries enqueue several duplicate detach jobs on VM-A's work queue. 3. First detach job completes (real unplug from VM-A); CSI attaches the volume to VM-B; DB is briefly correct (volume → VM-B, deviceid N). 4. A straggler detach job from step 2 completes minutes later and clears the volume's attachment row. 5. DB now: volume "Ready"/detached. Hypervisor: disk still defined in VM-B's domain at target vdN. (In one occurrence the stale disk remained in the previous VM's domain instead — both variants observed.) 6. Any subsequent AttachVolume to the affected VM picks deviceid N and fails: org.libvirt.LibvirtException: XML error: target 'vdN' duplicated for disk sources '<stale-uuid>' and '<new-uuid>' (API errorcode 530). The VM rejects all further volume attaches. ### What to do about it? Expected behavior / suggested fix: DetachVolume completion should only clear the volume's attachment state if it still refers to the attachment the job actually detached — i.e. a conditional update / compare-and-swap on (volume_id, instance_id[, device_id]) against the job's source VM. A stale detach whose volume has since been re-attached elsewhere should complete as a no-op (or fail), leaving the newer attachment state intact. Ideally, duplicate detach jobs for the same volume would also be coalesced, or volume-level serialization applied. Workaround: Manual: compare virsh domblklist against the volumes table per VM; live detach the orphaned target. For the attach-side variant (occurrence 4), a gentler remediation avoids touching a possibly-live disk: drain the guest's workloads, then stop/start the VM through CloudStack — on start the domain XML is rebuilt from the (now-consistent) DB and the stale target disappears. Mitigations deployed on our side: reconciliation alerting between libvirt and the DB, and increasing the CSI external-attacher retry backoff to reduce detach-job stacking -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
