Mikhail-Aoun commented on issue #7653: URL: https://github.com/apache/storm/issues/7653#issuecomment-5615868841
We tested the fix. Most of the time, our topology dependencies are cleaned up correctly, but sometimes they still remain in the blobstore. The following suspicious log pattern can be observed in the Nimbus logs when a dependency is not cleaned up: 1. Nimbus log: `"[INFO] Removing dependency blobs of..."`, which indicates the start of the cleanup implemented by the fix. 2. `StormClusterStateImpl` log: `"[INFO] set-path: /blobstore/..."` for the same dependency that was previously removed. For example, in one of my recent executions: ```text nimbus.log:2026-09-10 05:14:09.061 o.a.s.c.StormClusterStateImpl [INFO] set-path: /blobstore/dep-apeframework-0.0.11710-SNAPSHOT-f2d87a33-a8ad-4a95-a567-71e96a2b8695.jar/spc-scientificprocessingcluster-sni-stormnimbus-1.storm-nimbus.val.svc.cluster.local:6627:0-2 nimbus.log:2026-09-10 05:14:09.625 o.a.s.d.n.Nimbus [INFO] Created download session 40814946-e3bd-4240-b022-1565057c2fb1 for dep-apeframework-0.0.11710-SNAPSHOT-f2d87a33-a8ad-4a95-a567-71e96a2b8695.jar nimbus.log:2026-09-10 05:14:09.655 o.a.s.d.n.Nimbus [INFO] Created download session dd541b11-04e0-4f33-9848-5674820445fb for dep-apeframework-0.0.11710-SNAPSHOT-f2d87a33-a8ad-4a95-a567-71e96a2b8695.jar nimbus.log:2026-09-10 05:14:20.292 o.a.s.d.n.Nimbus [INFO] Created download session 2338a9c0-6fa8-4fca-bb52-60be4ce9e9fb for dep-apeframework-0.0.11710-SNAPSHOT-f2d87a33-a8ad-4a95-a567-71e96a2b8695.jar nimbus.log:2026-09-10 05:19:38.715 o.a.s.d.n.Nimbus [INFO] Removing dependency blobs of val-ipc-fci-nrt_Clmcla-RCFD_2017100_0096-2354-1789017251 - [dep-apeframework-0.0.11710-SNAPSHOT-f2d87a33-a8ad-4a95-a567-71e96a2b8695.jar] nimbus.log:2026-09-10 05:20:10.235 o.a.s.c.StormClusterStateImpl [INFO] set-path: /blobstore/dep-apeframework-0.0.11710-SNAPSHOT-f2d87a33-a8ad-4a95-a567-71e96a2b8695.jar/spc-scientificprocessingcluster-sni-stormnimbus-1.storm-nimbus.val.svc.cluster.local:6627:0-0 nimbus.log:2026-09-10 05:20:11.466 o.a.s.c.StormClusterStateImpl [INFO] set-path: /blobstore/dep-apeframework-0.0.11710-SNAPSHOT-f2d87a33-a8ad-4a95-a567-71e96a2b8695.jar/spc-scientificprocessingcluster-sni-stormnimbus-1.storm-nimbus.val.svc.cluster.local:6627:0-1 ``` When the dependency is correctly removed, we do not observe the last two `StormClusterStateImpl` log entries: ```text nimbus.log:2026-09-10 08:35:04.722 o.a.s.c.StormClusterStateImpl [INFO] set-path: /blobstore/dep-apeframework-0.0.11710-SNAPSHOT-8135d1cd-0fd1-4b67-9350-3909723c4078.jar/spc-scientificprocessingcluster-sni-stormnimbus-1.storm-nimbus.val.svc.cluster.local:6627:0-2 nimbus.log:2026-09-10 08:35:04.974 o.a.s.d.n.Nimbus [INFO] Created download session c5c8c393-f6d0-416b-b92d-1eb5bd7f8f46 for dep-apeframework-0.0.11710-SNAPSHOT-8135d1cd-0fd1-4b67-9350-3909723c4078.jar nimbus.log:2026-09-10 08:35:04.997 o.a.s.d.n.Nimbus [INFO] Created download session 6ec96c26-29fb-446d-acc9-16fd3f6aa63f for dep-apeframework-0.0.11710-SNAPSHOT-8135d1cd-0fd1-4b67-9350-3909723c4078.jar nimbus.log:2026-09-10 08:35:13.038 o.a.s.d.n.Nimbus [INFO] Created download session f5cae8e4-1a1e-4963-954d-d717e1767bf2 for dep-apeframework-0.0.11710-SNAPSHOT-8135d1cd-0fd1-4b67-9350-3909723c4078.jar nimbus.log:2026-09-10 08:40:18.801 o.a.s.d.n.Nimbus [INFO] Removing dependency blobs of val-ipc-fci-nrt_Fir-RCFD_2017100_0116-2553-1789029304 - [dep-apeframework-0.0.11710-SNAPSHOT-8135d1cd-0fd1-4b67-9350-3909723c4078.jar] ``` Please do not hesitate to ask if you need more details or additional logs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
