Hi Granwille,
Is the global setting 'secstorage.allowed.internal.sites' set to '192.168.50.6'
? If so, I believe it needs to be a CIDR and not an IP address. Could you try
setting it to 192.168.50.6/32 and restart the SSVM?
Regards,
Pearl
________________________________
From: Granwille Strauss
Sent: Wednesday, January 08, 2025 5:26 PM
To: users@cloudstack.apache.org
Subject: Re: SSVM Agent Alert State | 4.20 Upgrade
Additional log entries I am seeing from the management log for more context:
2025-01-09 00:19:31,259 WARN [c.c.a.m.A.HandleAgentConnectTask]
(AgentConnectTaskPool-194:[ctx-60959b5d]) (logid:7e1e2ebe) Unable to create
attache for agent: Seq 0-197: { Cmd , MgmtId: -1, via: 0, Ver: v1, Flags: 101,
[{"com.cloud.agent.api.StartupSecondaryStorageCommand":{"type":"SecondaryStorage","dataCenter":"16","pod":"15","guid":"s-247-VM-PremiumSecondaryStorageResource","name":"s-247-VM","id":"0","version":"4.20.0.0","iqn":"NoIqn","publicIpAddress":"197.189.211.3","publicNetmask":"255.255.255.248","publicMacAddress":"1e:00:07:00:00:1f","privateIpAddress":"192.168.50.57","privateMacAddress":"1e:00:14:00:00:08","privateNetmask":"255.255.255.0","storageIpAddress":"192.168.50.57","storageNetmask":"255.255.255.0","storageMacAddress":"1e:00:14:00:00:08","resourceName":"PremiumSecondaryStorageResource","msHostList":"CMPUBLICIP,192.168.50.2","wait":"0","bypassHostMaintenance":"false"}}]
}
2025-01-09 00:19:10,309 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(AgentManager-Handler-12:[]) (logid:) Ping from 69(s-247-VM)
2025-01-09 00:19:10,309 INFO [c.c.a.m.ClusteredAgentManagerImpl]
(AgentManager-Handler-12:[]) (logid:) PingMap for agent: 69 will not be updated
because agent is no longer in the PingMap
025-01-09 00:19:14,974 DEBUG [o.a.c.h.H.HAManagerBgPollTask]
(BackgroundTaskPollManager-2:[ctx-ee4b6523]) (logid:c4a5a9ac) HA health check
task is running...
2025-01-09 00:19:15,807 DEBUG [c.c.a.t.Request] (AgentManager-Handler-15:[])
(logid:) Seq 69-4465600505514557442: Processing: { Ans: , MgmtId:
66988330791813, via: 69, Ver: v1, Flags: 110,
[{"com.cloud.agent.api.SecStorageSetupAnswer":{"_dir":"b702a7e0-8545-3f2a-a7ba-9f1813f37b71","result":"true","details":"success","wait":"0","bypassHostMaintenance":"false"}}]
}
2025-01-09 00:19:15,807 DEBUG [c.c.a.m.ClusteredAgentAttache]
(AgentManager-Handler-15:[]) (logid:) Seq 69-4465600505514557442: No more
commands found
2025-01-09 00:19:15,807 DEBUG [c.c.a.t.Request]
(AgentConnectTaskPool-193:[ctx-9be0a9ec]) (logid:87642208) Seq
69-4465600505514557442: Received: { Ans: , MgmtId: 66988330791813, via:
69(s-247-VM), Ver: v1, Flags: 110, { SecStorageSetupAnswer } }
2025-01-09 00:19:15,807 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(AgentConnectTaskPool-193:[ctx-9be0a9ec]) (logid:87642208) Details from
executing class com.cloud.agent.api.SecStorageSetupCommand: success
2025-01-09 00:19:15,808 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl]
(AgentConnectTaskPool-193:[ctx-9be0a9ec]) (logid:87642208) Successfully
programmed secondary storage [backup.namhost.com] in secondary storage VM
[s-247-VM].
2025-01-09 00:19:15,809 ERROR [c.c.a.m.ClusteredAgentManagerImpl]
(AgentConnectTaskPool-193:[ctx-9be0a9ec]) (logid:87642208) Monitor
SecondaryStorageListener says there is an error in the connect process for 69
due to Invalid CIDR: 192.168.50.6
com.cloud.utils.exception.CloudRuntimeException: Invalid CIDR: 192.168.50.6
at com.cloud.utils.net.NetUtils.getCleanIp4Cidr(NetUtils.java:634)
at
org.apache.cloudstack.secondarystorage.SecondaryStorageManagerImpl.getAllowedInternalSiteCidrs(SecondaryStorageManagerImpl.java:404)
at
org.apache.cloudstack.secondarystorage.SecondaryStorageManagerImpl.generateFirewallConfiguration(SecondaryStorageManagerImpl.java:437)
at
com.cloud.storage.secondary.SecondaryStorageListener.processConnect(SecondaryStorageListener.java:87)
at
com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:553)
at
com.cloud.agent.manager.AgentManagerImpl.sendReadyAndGetAttache(AgentManagerImpl.java:1116)
at
com.cloud.agent.manager.AgentManagerImpl.handleConnectedAgent(AgentManagerImpl.java:1135)
at
com.cloud.agent.manager.AgentManagerImpl$HandleAgentConnectTask.runInContext(AgentManagerImpl.java:1227)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
2025-01-09 00:19:15,809 WARN [c.c.u.d.GlobalLock]
(AgentConnectTaskPool-193:[ctx-9be0a9ec]) (logid:87642208) Global lock
re-entrance detected
2025-01-09 00:19:15,809 INFO [c.c.a.m.ClusteredAgentManagerImpl]
(AgentConnectTaskPool-193:[ctx-9be0a9ec]) (logid:87642208) Host 69 is
disconnecting with event AgentDisconnected
2025-01-09 00:19:15,810 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(AgentConnectTaskPool-193:[ctx-9be0a9ec]) (logid:87642208) The next status of
agent bdfd8082-ba90-4c44-bfdb-e51ee6243271 is Alert, current status is
Connecting
2025-01-09 00:19:15,814 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(AgentConnectTaskPool-193:[ctx-9be0a9ec]) (logid:87642208) Failed to handle
host connection: com.cloud.utils.exception.CloudRuntimeException: Unable to
connect 69
at
com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:572)
at
com.cloud.agent.manager.AgentManagerImpl.sendReadyAndGetAttache(AgentManagerImpl.java:1116)
at
com.cloud.agent.manager.AgentManagerImpl.handleConnectedAgent(AgentManagerImpl.java:1135)
at
com.cloud.agent.manager.AgentManagerImpl$HandleAgentConnectTask.runInContext(AgentManagerImpl.java:1227)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: com.cloud.utils.exception.CloudRuntimeException: Invalid CIDR:
192.168.50.6
at com.cloud.utils.net.NetUtils.getCleanIp4Cidr(NetUtils.java:634)
at
org.apache.cloudstack.secondarystorage.SecondaryStorageManagerImpl.getAllowedInternalSiteCidrs(SecondaryStorageManagerImpl.java:404)
at
org.apache.cloudstack.secondarystorage.SecondaryStorageManagerImpl.generateFirewallConfiguration(SecondaryStorageManagerImpl.java:437)
at
com.cloud.storage.secondary.SecondaryStorageListener.processConnect(SecondaryStorageListener.java:87)
at
com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:553)
... 11 more
2025-01-09 00:19:15,814 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(AgentConnectTaskPool-193:[ctx-9be0a9ec]) (logid:87642208) Failed to send ready
command:java.nio.channels.ClosedChannelException
I have no idea what this means or where to begin.
On 1/9/25 00:04, Granwille Strauss wrote:
Good Day,
I just completed the upgrade to 4.20 and now my SSVM agent remains in alert
state. I use KVM setup on Almalinux 8 and I have followed the upgrade
documentation from 4.19.x to 4.20 thoroughly as always. I have Java 17 running
and the cloudstack-agent is runinng fine too on the KVM host.
java -version
openjdk version "17.0.13" 2024-10-15 LTS
OpenJDK Runtime Environment (Red_Hat-17.0.13.0.11-1) (build 17.0.13+11-LTS)
OpenJDK 64-Bit Server VM (Red_Hat-17.0.13.0.11-1) (build 17.0.13+11-LTS, mixed
mode, sharing)
Inside the SSVM, the /usr/local/cloud/systemvm/ssvm-check.sh passes too:
root@s-247-VM:~# /usr/local/cloud/systemvm/ssvm-check.sh
================================================
First DNS server is 1.1.1.1
PING 1.1.1.1 (1.1.1.1): 56 data bytes
64 bytes from 1.1.1.1: icmp_seq=0 ttl=58 time=1.634 ms
64 bytes from 1.1.1.1: icmp_seq=1 ttl=58 time=1.541 ms
--- 1.1.1.1 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 1.541/1.587/1.634/0.047 ms
Good: Can ping DNS server
================================================
Good: DNS resolves cloudstack.apache.org
================================================
nfs is currently mounted
Mount point is /mnt/SecStorage/a863afaf-19a6-3827-8c49-0c5e6346b6c8
Good: Can write to mount point
Mount point is /mnt/SecStorage/b702a7e0-8545-3f2a-a7ba-9f1813f37b71
Good: Can write to mount point
================================================
Management server is 192.168.50.2. Checking connectivity.
Good: Can connect to management server 192.168.50.2 port 8250
================================================
Good: Java process is running
================================================
Tests Complete. Look for ERROR or WARNING above.
Tailing the /var/log/cloud/cloud.log log gives me the following constant error:
2025-01-08T21:59:21,219 INFO [utils.nio.NioClient] (Agent-Handler-1:[])
Connected to 156.38.173.122:8250
2025-01-08T21:59:21,296 INFO [cloud.agent.Agent] (Agent-Handler-2:[]) Process
agent startup answer, agent id = 0
2025-01-08T21:59:21,301 INFO [storage.resource.NfsSecondaryStorageResource]
(agentRequest-Handler-2:[]) Determined host 192.168.50.2 corresponds to IP
192.168.50.2
2025-01-08T21:59:21,302 INFO [storage.resource.NfsSecondaryStorageResource]
(agentRequest-Handler-2:[]) Determined host 192.168.50.2 corresponds to IP
192.168.50.2
2025-01-08T21:59:21,298 INFO [cloud.agent.Agent] (Agent-Handler-2:[]) Startup
Response Received: agent id = 0
2025-01-08T21:59:23,822 INFO [storage.resource.NfsSecondaryStorageResource]
(agentRequest-Handler-2:[]) created a thread to start post upload server
2025-01-08T21:59:23,843 INFO [handler.logging.LoggingHandler]
(nioEventLoopGroup-66-1:[]) [id: 0x8346baf0] REGISTERED
2025-01-08T21:59:23,845 INFO [handler.logging.LoggingHandler]
(nioEventLoopGroup-66-1:[]) [id: 0x8346baf0] BIND: 0.0.0.0/0.0.0.0:8210
2025-01-08T21:59:23,846 INFO [handler.logging.LoggingHandler]
(nioEventLoopGroup-66-1:[]) [id: 0x8346baf0] CLOSE
2025-01-08T21:59:23,846 INFO [handler.logging.LoggingHandler]
(nioEventLoopGroup-66-1:[]) [id: 0x8346baf0] UNREGISTERED
2025-01-08T21:59:23,857 INFO [storage.resource.NfsSecondaryStorageResource]
(Thread-34:[]) shutting down post upload server
2025-01-08T21:59:23,880 INFO [storage.resource.NfsSecondaryStorageResource]
(agentRequest-Handler-4:[]) Determined host 192.168.50.6 corresponds to IP
192.168.50.6
2025-01-08T21:59:23,885 INFO [storage.resource.NfsSecondaryStorageResource]
(agentRequest-Handler-4:[]) Determined host 192.168.50.6 corresponds to IP
192.168.50.6
2025-01-08T21:59:26,222 INFO [cloud.agent.Agent] (Agent-Handler-1:[])
Connected to the host: 156.38.173.122
Can anyone please help with this, please? I have destroyed and rebooted the
SSVM multiple times and have it re-created on different KVM hosts, the results
remain the same.
Any assistance will be highly appreciated, thank you.
--
Regards / Groete
[https://outlook.office.com/mail/inbox/id/AAQkAGU4ZGNhYmI2LTJmNTQtNDYyNi04NDcxLThlYTQzZmIyOTc0YQAQAMGHvWrFY0JCplMUnaD6GYo%3D]<https://www.namhost.com>
Granwille Strauss // Senior Systems Admin
e: granwi...@namhost.com<mailto:granwi...@namhost.com>
m: +264 81 323 1260<tel:+264813231260>
w: www.namhost.com<https://www.namhost.com/>
[https://outlook.office.com/mail/inbox/id/AAQkAGU4ZGNhYmI2LTJmNTQtNDYyNi04NDcxLThlYTQzZmIyOTc0YQAQAMGHvWrFY0JCplMUnaD6GYo%3D]
<https://www.facebook.com/namhost>
[https://outlook.office.com/mail/inbox/id/AAQkAGU4ZGNhYmI2LTJmNTQtNDYyNi04NDcxLThlYTQzZmIyOTc0YQAQAMGHvWrFY0JCplMUnaD6GYo%3D]
<https://twitter.com/namhost>
[https://outlook.office.com/mail/inbox/id/AAQkAGU4ZGNhYmI2LTJmNTQtNDYyNi04NDcxLThlYTQzZmIyOTc0YQAQAMGHvWrFY0JCplMUnaD6GYo%3D]
<https://www.instagram.com/namhostinternetservices/>
[https://outlook.office.com/mail/inbox/id/AAQkAGU4ZGNhYmI2LTJmNTQtNDYyNi04NDcxLThlYTQzZmIyOTc0YQAQAMGHvWrFY0JCplMUnaD6GYo%3D]
<https://www.linkedin.com/company/namhost>
[https://outlook.office.com/mail/inbox/id/AAQkAGU4ZGNhYmI2LTJmNTQtNDYyNi04NDcxLThlYTQzZmIyOTc0YQAQAMGHvWrFY0JCplMUnaD6GYo%3D]
<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
[https://outlook.office.com/mail/inbox/id/AAQkAGU4ZGNhYmI2LTJmNTQtNDYyNi04NDcxLThlYTQzZmIyOTc0YQAQAMGHvWrFY0JCplMUnaD6GYo%3D]<https://www.namhost.com/your-move>
Namhost Internet Services (Pty) Ltd ,
24 Black Eagle Rd, Hermanus, 7210, RSA
The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It is
forbidden to copy, forward, or in any way reveal the contents of this message
to anyone without our explicit consent. The integrity and security of this
email cannot be guaranteed over the Internet. Therefore, the sender will not be
held liable for any damage caused by the message. For our full privacy policy
and disclaimers, please go to https://www.namhost.com/privacy-policy
--
Regards / Groete
[https://outlook.office.com/mail/inbox/id/AAQkAGU4ZGNhYmI2LTJmNTQtNDYyNi04NDcxLThlYTQzZmIyOTc0YQAQAMGHvWrFY0JCplMUnaD6GYo%3D]<https://www.namhost.com>
Granwille Strauss // Senior Systems Admin
e: granwi...@namhost.com<mailto:granwi...@namhost.com>
m: +264 81 323 1260<tel:+264813231260>
w: www.namhost.com<https://www.namhost.com/>
[https://outlook.office.com/mail/inbox/id/AAQkAGU4ZGNhYmI2LTJmNTQtNDYyNi04NDcxLThlYTQzZmIyOTc0YQAQAMGHvWrFY0JCplMUnaD6GYo%3D]
<https://www.facebook.com/namhost>
[https://outlook.office.com/mail/inbox/id/AAQkAGU4ZGNhYmI2LTJmNTQtNDYyNi04NDcxLThlYTQzZmIyOTc0YQAQAMGHvWrFY0JCplMUnaD6GYo%3D]
<https://twitter.com/namhost>
[https://outlook.office.com/mail/inbox/id/AAQkAGU4ZGNhYmI2LTJmNTQtNDYyNi04NDcxLThlYTQzZmIyOTc0YQAQAMGHvWrFY0JCplMUnaD6GYo%3D]
<https://www.instagram.com/namhostinternetservices/>
[https://outlook.office.com/mail/inbox/id/AAQkAGU4ZGNhYmI2LTJmNTQtNDYyNi04NDcxLThlYTQzZmIyOTc0YQAQAMGHvWrFY0JCplMUnaD6GYo%3D]
<https://www.linkedin.com/company/namhost>
[https://outlook.office.com/mail/inbox/id/AAQkAGU4ZGNhYmI2LTJmNTQtNDYyNi04NDcxLThlYTQzZmIyOTc0YQAQAMGHvWrFY0JCplMUnaD6GYo%3D]
<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
[https://outlook.office.com/mail/inbox/id/AAQkAGU4ZGNhYmI2LTJmNTQtNDYyNi04NDcxLThlYTQzZmIyOTc0YQAQAMGHvWrFY0JCplMUnaD6GYo%3D]<https://www.namhost.com/your-move>
Namhost Internet Services (Pty) Ltd ,
24 Black Eagle Rd, Hermanus, 7210, RSA
The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It is
forbidden to copy, forward, or in any way reveal the contents of this message
to anyone without our explicit consent. The integrity and security of this
email cannot be guaranteed over the Internet. Therefore, the sender will not be
held liable for any damage caused by the message. For our full privacy policy
and disclaimers, please go to https://www.namhost.com/privacy-policy
