Shivamkashyap08 opened a new issue, #13009:
URL: https://github.com/apache/cloudstack/issues/13009
Hi CloudStack Community,
We recently upgraded our CloudStack environment from version 4.19.2.0 to
4.20.2.0. The upgrade process completed successfully; however, we are
encountering an issue post-upgrade.
Environment Details:
Current Cloudstack Version - 4.19.2.0
Need to Upgrade to CloudStack Version: 4.20.2.0
Hypervisor: Citrix XenServer
Zone Types: EC Zone & VPC Zone(both)
Issue Description:
After the upgrade:
Hosts in the VPC Zone are functioning correctly and appear in the UP state
in the CloudStack UI. However, hosts in the EC Zone have gone into an Alert
state and are not transitioning back to UP.
Observations:
Upon analyzing the logs, we observed failures related to the callHostPlugin.
After logging into the hypervisor, we found that the task host.call_plugin
is stuck in a pending state.
Request:
Could you please help us identify the root cause of this issue and suggest
possible steps to resolve it?
I have attached the relevant error logs and hypervisor task list output for
reference.
2026-03-28 03:21:35,277 INFO [c.c.a.m.ClusteredAgentManagerImpl]
(DirectAgentCronJob-4:[ctx-8fed6763]) (logid:e1cdc087) PingMap for agent: 70098
(null) will not be updated because agent is no longer in the PingMap
2026-03-28 03:21:35,280 DEBUG [c.c.h.x.r.XenServerConnectionPool]
(DirectAgent-46:[ctx-ea6d6da5]) (logid:59e687ae) XmlRpcException for method:
host.call_plugin due to org.apache.xmlrpc.XmlRpcException: Failed to create
input stream: Read timed out
2026-03-28 03:21:35,280 DEBUG [c.c.h.x.r.XenServer650Resource]
(DirectAgent-46:[ctx-ea6d6da5]) (logid:59e687ae) callHostPlugin failed for cmd:
can_bridge_firewall with args instance: VM, host_uuid:
2ec20f60-ea68-4ec8-89bb-3379c14395f5, due to Failed to create input stream:
Read timed out
2026-03-28 03:21:35,280 WARN [c.c.h.x.r.w.x.CitrixSetupCommandWrapper]
(DirectAgent-46:[ctx-ea6d6da5]) (logid:59e687ae) Unable to setup
com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed for cmd:
can_bridge_firewall with args instance: VM, host_uuid:
2ec20f60-ea68-4ec8-89bb-3379c14395f5, due to Failed to create input stream:
Read timed out
at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.callHostPlugin(CitrixResourceBase.java:380)
at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.canBridgeFirewall(CitrixResourceBase.java:497)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixSetupCommandWrapper.execute(CitrixSetupCommandWrapper.java:67)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixSetupCommandWrapper.execute(CitrixSetupCommandWrapper.java:44)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1778)
at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:304)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
2026-03-28 03:21:35,280 DEBUG [c.c.a.m.D.Task]
(DirectAgent-46:[ctx-ea6d6da5]) (logid:59e687ae) Seq 70097-597289900580012033:
Response Received:
2026-03-28 03:21:35,281 DEBUG [c.c.a.t.Request]
(DirectAgent-46:[ctx-ea6d6da5]) (logid:59e687ae) Seq 70097-597289900580012033:
Processing: { Ans: , MgmtId: 165385370412082, via: 70097, Ver: v1, Flags: 110,
[{"com.cloud.agent.api.SetupAnswer":{"_reconnect":"true","result":"false","details":"callHostPlugin
failed for cmd: can_bridge_firewall with args instance: VM, host_uuid:
2ec20f60-ea68-4ec8-89bb-3379c14395f5, due to Failed to create input stream:
Read timed out","wait":"0","bypassHostMaintenance":"false"}}] }
Cmd - xe task-list
uuid ( RO) : 77840f1d-70fb-2a31-c941-ed93c75e68be
name-label ( RO): host.call_plugin
name-description ( RO):
status ( RO): pending
progress ( RO): 0.000
uuid ( RO) : 2ac624c1-9584-d56e-cae7-1988fba8c0b2
name-label ( RO): host.call_plugin
name-description ( RO):
status ( RO): pending
progress ( RO): 0.000
uuid ( RO) : 6a9ce2ab-3353-4bc3-b23c-b17cf4eafb28
name-label ( RO): host.call_plugin
name-description ( RO):
status ( RO): pending
progress ( RO): 0.000
Please note - I have rolled back the upgrade, and it is running on
CloudStack version 4.19.2.0.
Thanks in advance for your support.
Thanks
Shivam Kashyap
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
