Hi Stephan,
"4.20.0" should be fine as 4.20.0 and 4.20.1 systemvm templates are both
based on Debian 12.X
The 4.19.x system template is based on Debian 11. live-patch also worked in
my testing few months ago, but I personally do not recommend it ("4.19.x")
Kind regards,
Wei
On Fri, Jun 20, 2025 at 10:00 PM Stephan Bienek <stephan....@bienek.org>
wrote:
> Hi Wei,
>
> an interesting thought - indeed the setting states "4.20.1".
> I can't remember manually changing the value after the update.
>
> Do you you know if changing it to 4.20.0 would be safe way or could be
> causing any potential issues?
>
> Thanks,
> Stephan
>
> > Wei ZHOU <ustcweiz...@gmail.com> hat am 20.06.2025 19:21 CEST
> geschrieben:
> >
> >
> > What's the value of the global setting "minreq.sysvmtemplate.version" ?
> >
> >
> > -Wei
> >
> > On Fri, Jun 20, 2025 at 6:17 PM Stephan Bienek <stephan....@bienek.org>
> > wrote:
> >
> > > Dear Community,
> > >
> > > i am trying to better understand the Virtual Router Live Patching aka
> VR
> > > Update without service interruption.
> > >
> > > We did update a CloudStack 4.20.0 Cluster to 4.20.1
> > > As expected, shutdown VMs connected to unpatched Isolated networks
> can't
> > > start and firewall rules etc can't be changed complaining about a
> required
> > > update of the VR.
> > >
> > > When i execute the live patching of an isolated network or of a VR, i
> > > always receive the following error message "Failed to restart network"
> and
> > > i see the log entry "Unable to send command. Router requires upgrade"
> > >
> > > 2025-06-19 13:03:26,740 WARN [o.a.c.e.o.NetworkOrchestrator]
> > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> > > (logid:c02acb59) Failed to implement network Network {"id": 279,
> "name":
> > > "TESTNETWORK LIVEPATCH", "uuid":
> "8a8d2f9b-98c5-4509-a9a7-8b0a603c55c7",
> > > "networkofferingid": 8} elements and resources as a part of network
> restart
> > > due to com.cloud.exception.ResourceUnavailableException: Resource
> > > [VirtualRouter:922] is unreachable: Unable to send command. Router
> requires
> > > upgrade
> > > 2025-06-19 13:03:26,785 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502]) (logid:c02acb59)
> Complete
> > > async job-23502, jobStatus: FAILED, resultCode: 530, result:
> > >
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed
> > > to restart network"}
> > >
> > > From the logs, shortly before i see a success message "Successfully
> > > patched router VM":
> > >
> > > 2025-06-19 13:03:26,606 INFO [o.a.c.e.o.NetworkOrchestrator]
> > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> > > (logid:c02acb59) Successfully patched router VM instance
> > >
> {"id":922,"instanceName":"r-922-VM","state":"Running","type":"DomainRouter","uuid":"79a93a76-b1c7-4627-ae4c-d9b1f0a6bdfa"}
> > >
> > > Is it because the systemvm template was updated (Debian 11 to Debian
> 12)
> > > and live patching can only be done on the same systemvm template
> version?
> > >
> > > 2025-06-19 13:03:26,738 DEBUG [c.c.n.r.NetworkHelperImpl]
> > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> > > (logid:c02acb59) Router requires upgrade. Unable to send command to
> router:
> > > VM instance
> > >
> {"id":922,"instanceName":"r-922-VM","state":"Running","type":"DomainRouter","uuid":"79a93a76-b1c7-4627-ae4c-d9b1f0a6bdfa"},
> > > router template version: Cloudstack Release 4.20.0 Fri Sep 6 03:45:27
> AM
> > > UTC 2024, minimal required version: 4.20.1
> > >
> > > The only way to get the network/VR fully functional again, allowing
> > > stopped VMs to start and changing firewall rules etc is to execute a
> > > network restart with cleanup or execute "Upgrade router to use newer
> > > template" - which comes with a service impact as expected.
> > >
> > > Full log attached to the end of this mail, in case i am missing
> something
> > > obvious.
> > >
> > > Thanks in advance for helping me to understand the requirements for
> > > non-service impacting live patching.
> > >
> > > Best regards,
> > > Stephan
> > >
> > > ############
> > > ### full log ###
> > > ############
> > >
> > > 2025-06-19 13:02:57,903 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > (qtp1404565079-435:[ctx-ccc0da24, ctx-7ab4ba90]) (logid:64ccbfcd)
> submit
> > > async job-23502, details: AsyncJob
> > >
> {"accountId":4,"cmd":"org.apache.cloudstack.api.command.user.network.RestartNetworkCmd","cmdInfo":"{\"response\":\"json\",\"ctxUserId\":\"9\",\"livepatch\":\"true\",\"sessionkey\":\"1234567890abcdefg\",\"httpmethod\":\"GET\",\"ctxStartEventId\":\"82573\",\"id\":\"8a8d2f9b-98c5-4509-a9a7-8b0a603c55c7\",\"ctxDetails\":\"{\\\"interface
> > >
> com.cloud.network.Network\\\":\\\"8a8d2f9b-98c5-4509-a9a7-8b0a603c55c7\\\"}\",\"ctxAccountId\":\"4\",\"uuid\":\"8a8d2f9b-98c5-4509-a9a7-8b0a603c55c7\",\"cmdEventType\":\"NETWORK.RESTART\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":23502,"initMsid":33830401474176,"instanceId":279,"instanceType":"Network","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":9,"uuid":"c02acb59-37e0-4bba-bd8e-82294c6
> > > 6c505"}
> > > 2025-06-19 13:02:57,908 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5]
> > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502]) (logid:c02acb59)
> Executing
> > > AsyncJob
> > >
> {"accountId":4,"cmd":"org.apache.cloudstack.api.command.user.network.RestartNetworkCmd","cmdInfo":"{\"response\":\"json\",\"ctxUserId\":\"9\",\"livepatch\":\"true\",\"sessionkey\":\"1234567890abcdefg\",\"httpmethod\":\"GET\",\"ctxStartEventId\":\"82573\",\"id\":\"8a8d2f9b-98c5-4509-a9a7-8b0a603c55c7\",\"ctxDetails\":\"{\\\"interface
> > >
> com.cloud.network.Network\\\":\\\"8a8d2f9b-98c5-4509-a9a7-8b0a603c55c7\\\"}\",\"ctxAccountId\":\"4\",\"uuid\":\"8a8d2f9b-98c5-4509-a9a7-8b0a603c55c7\",\"cmdEventType\":\"NETWORK.RESTART\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":23502,"initMsid":33830401474176,"instanceId":279,"instanceType":"Network","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":9,"uuid":"c02acb59-37e0-4bba-bd8e-82294c66c505"}
> > > 2025-06-19 13:02:57,948 DEBUG [c.c.u.AccountManagerImpl]
> > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> > > (logid:c02acb59) Account [Account
> > >
> [{"accountName":"Cloudstack_Admins","id":4,"uuid":"742dba95-8709-4aba-92c8-fb472e118ddb"}]]
> > > has access to resource.
> > > 2025-06-19 13:02:57,961 DEBUG [c.c.a.ApiServlet]
> > > (qtp1404565079-435:[ctx-d1310f58]) (logid:bf1fd0ff) ===START===
> 192.168.1.5
> > > -- GET
> > >
> jobId=c02acb59-37e0-4bba-bd8e-82294c66c505&command=queryAsyncJobResult&response=json&sessionkey=1234567890abcdefg
> > > 2025-06-19 13:02:58,007 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> > > (logid:c02acb59) Restarting network Network {"id": 279, "name":
> > > "TESTNETWORK LIVEPATCH", "uuid":
> "8a8d2f9b-98c5-4509-a9a7-8b0a603c55c7",
> > > "networkofferingid": 8}...
> > > 2025-06-19 13:02:58,039 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> > > (logid:c02acb59) Wait time setting on
> > > com.cloud.agent.api.PatchSystemVmCommand is 600000 seconds
> > > 2025-06-19 13:02:58,046 DEBUG [c.c.a.m.ClusteredAgentAttache]
> > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> > > (logid:c02acb59) Seq 3902-2730025799116652757: Forwarding Seq
> > > 3986-2730025799116652757: { Cmd , MgmtId: 33830401474176, via:
> > > 3986(kvm-001), Ver: v1, Flags: 100011,
> > >
> [{"com.cloud.agent.api.PatchSystemVmCommand":{"forced":"true","accessDetails":{"
> > > router.name
> ":"r-922-VM","router.ip":"169.254.77.6"},"wait":"0","bypassHostMaintenance":"false"}}]
> > > } to 24728157879110
> > > 2025-06-19 13:02:58,050 DEBUG [c.c.a.ApiServlet]
> > > (qtp1404565079-435:[ctx-d1310f58, ctx-531f992f]) (logid:bf1fd0ff)
> ===END===
> > > 192.168.1.5 -- GET
> > >
> jobId=c02acb59-37e0-4bba-bd8e-82294c66c505&command=queryAsyncJobResult&response=json&sessionkey=1234567890abcdefg
> > > 2025-06-19 13:03:26,591 DEBUG [c.c.a.t.Request]
> > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> > > (logid:c02acb59) Seq 3986-2730025799116652757: Received: { Ans: ,
> MgmtId:
> > > 33830401474176, via: 3986(kvm-001), Ver: v1, Flags: 10, {
> > > PatchSystemVmAnswer } }
> > > 2025-06-19 13:03:26,591 INFO [c.c.s.ManagementServerImpl]
> > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> > > (logid:c02acb59) Successfully patched system VM r-922-VM
> > > 2025-06-19 13:03:26,606 INFO [o.a.c.e.o.NetworkOrchestrator]
> > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> > > (logid:c02acb59) Successfully patched router VM instance
> > >
> {"id":922,"instanceName":"r-922-VM","state":"Running","type":"DomainRouter","uuid":"79a93a76-b1c7-4627-ae4c-d9b1f0a6bdfa"}
> > > 2025-06-19 13:03:26,607 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> > > (logid:c02acb59) Implementing the network Network {"id": 279, "name":
> > > "TESTNETWORK LIVEPATCH", "uuid":
> "8a8d2f9b-98c5-4509-a9a7-8b0a603c55c7",
> > > "networkofferingid": 8} elements and resources as a part of network
> restart
> > > without cleanup
> > > 2025-06-19 13:03:26,638 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> > > (logid:c02acb59) Asking VirtualRouter to implement Network {"id": 279,
> > > "name": "TESTNETWORK LIVEPATCH", "uuid":
> > > "8a8d2f9b-98c5-4509-a9a7-8b0a603c55c7", "networkofferingid": 8}
> > > 2025-06-19 13:03:26,668 DEBUG [c.c.a.ApiServlet]
> > > (qtp1404565079-321:[ctx-fcacff74]) (logid:a66d7bee) ===START===
> 192.168.1.5
> > > -- GET
> > >
> jobId=c02acb59-37e0-4bba-bd8e-82294c66c505&command=queryAsyncJobResult&response=json&sessionkey=1234567890abcdefg
> > > 2025-06-19 13:03:26,704 DEBUG [c.c.n.NetworkModelImpl]
> > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> > > (logid:c02acb59) Service SecurityGroup is not supported in the network
> > > Network {"id": 279, "name": "TESTNETWORK LIVEPATCH", "uuid":
> > > "8a8d2f9b-98c5-4509-a9a7-8b0a603c55c7", "networkofferingid": 8}
> > > 2025-06-19 13:03:26,733 DEBUG [c.c.u.s.Script]
> > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> > > (logid:c02acb59) Looking for vms/cloud-scripts.tgz in the classpath
> > > 2025-06-19 13:03:26,733 DEBUG [c.c.u.s.Script]
> > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> > > (logid:c02acb59) System resource:
> > > file:/usr/share/cloudstack-common/vms/cloud-scripts.tgz
> > > 2025-06-19 13:03:26,733 DEBUG [c.c.u.s.Script]
> > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> > > (logid:c02acb59) Absolute path =
> > > /usr/share/cloudstack-common/vms/cloud-scripts.tgz
> > > 2025-06-19 13:03:26,738 DEBUG [c.c.n.r.NetworkHelperImpl]
> > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> > > (logid:c02acb59) Router requires upgrade. Unable to send command to
> router:
> > > VM instance
> > >
> {"id":922,"instanceName":"r-922-VM","state":"Running","type":"DomainRouter","uuid":"79a93a76-b1c7-4627-ae4c-d9b1f0a6bdfa"},
> > > router template version: Cloudstack Release 4.20.0 Fri Sep 6 03:45:27
> AM
> > > UTC 2024, minimal required version: 4.20.1
> > > 2025-06-19 13:03:26,740 WARN [o.a.c.e.o.NetworkOrchestrator]
> > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> > > (logid:c02acb59) Failed to implement network Network {"id": 279,
> "name":
> > > "TESTNETWORK LIVEPATCH", "uuid":
> "8a8d2f9b-98c5-4509-a9a7-8b0a603c55c7",
> > > "networkofferingid": 8} elements and resources as a part of network
> restart
> > > due to com.cloud.exception.ResourceUnavailableException: Resource
> > > [VirtualRouter:922] is unreachable: Unable to send command. Router
> requires
> > > upgrade
> > > 2025-06-19 13:03:26,741 WARN [c.c.n.NetworkServiceImpl]
> > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> > > (logid:c02acb59) Network Network {"id": 279, "name": "TESTNETWORK
> > > LIVEPATCH", "uuid": "8a8d2f9b-98c5-4509-a9a7-8b0a603c55c7",
> > > "networkofferingid": 8} failed to restart.
> > > 2025-06-19 13:03:26,741 WARN [o.a.c.m.w.WebhookServiceImpl]
> > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> > > (logid:c02acb59) Skipping delivering event Event
> > >
> {"description":"{\"event\":\"NETWORK.RESTART\",\"status\":\"Completed\"}","eventId":null,"eventType":"NETWORK.RESTART","eventUuid":null,"resourceType":"Network","resourceUUID":null}
> > > to any webhook as account ID is missing
> > > 2025-06-19 13:03:26,742 WARN [o.a.c.f.e.EventDistributorImpl]
> > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> > > (logid:c02acb59) Failed to publish event [category: ActionEvent, type:
> > > NETWORK.RESTART] on bus webhookEventBus
> > > 2025-06-19 13:03:26,752 DEBUG [c.c.a.ApiServlet]
> > > (qtp1404565079-321:[ctx-fcacff74, ctx-dc88a930]) (logid:a66d7bee)
> ===END===
> > > 192.168.1.5 -- GET
> > >
> jobId=c02acb59-37e0-4bba-bd8e-82294c66c505&command=queryAsyncJobResult&response=json&sessionkey=1234567890abcdefg
> > > 2025-06-19 13:03:26,785 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502]) (logid:c02acb59)
> Complete
> > > async job-23502, jobStatus: FAILED, resultCode: 530, result:
> > >
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed
> > > to restart network"}
> > > 2025-06-19 13:03:26,789 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502]) (logid:c02acb59)
> Publish
> > > async job-23502 complete on message bus
> > > 2025-06-19 13:03:26,789 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502]) (logid:c02acb59) Wake
> up
> > > jobs related to job-23502
> > > 2025-06-19 13:03:26,789 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502]) (logid:c02acb59)
> Update db
> > > status for job-23502
> > > 2025-06-19 13:03:26,793 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502]) (logid:c02acb59) Wake
> up
> > > jobs joined with job-23502 and disjoin all subjobs created from job-
> 23502
> > > 2025-06-19 13:03:26,824 DEBUG [c.c.a.ApiServer]
> > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502]) (logid:c02acb59)
> Retrieved
> > > cmdEventType from job info: NETWORK.RESTART
> > > 2025-06-19 13:03:26,832 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5]
> > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502]) (logid:c02acb59) Done
> > > executing
> org.apache.cloudstack.api.command.user.network.RestartNetworkCmd
> > > for job-23502
> > > 2025-06-19 13:03:26,832 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502]) (logid:c02acb59) Remove
> > > job-23502 from job monitoring
>
