Hello,
All VM domains in KVM suddenly terminated but CloudStack was thinking they
are running as states were not updated in the database. I just forcefully
stopped them and now they cannot start due to insufficient capacity. Below
is the agent log. i tried restarting systemd-machind, libvirtd and agent.
Also checked all ports. Port 8250 is open.
in:null) (logid:) Connecting to 79.143.86.149:8250
2021-09-04 12:16:02,000 INFO [utils.nio.Link] (main:null) (logid:) Conf
file found: /etc/cloudstack/agent/agent.properties
2021-09-04 12:16:02,255 ERROR [utils.nio.Link] (main:null) (logid:) SSL
error caught during unwrap data: Received fatal alert: certificate_unknown,
for local address=/79.143.86.149:53860, remote address=/79.143.86.149:8250.
The client may have invalid ca-certificates.
2021-09-04 12:16:02,255 ERROR [utils.nio.NioClient] (main:null) (logid:)
SSL Handshake failed while connecting to host: 79.143.86.149 port: 8250
2021-09-04 12:16:02,255 ERROR [utils.nio.NioConnection] (main:null)
(logid:) Unable to initialize the threads.
java.io.IOException: SSL Handshake failed while connecting to host:
79.143.86.149 port: 8250
at com.cloud.utils.nio.NioClient.init(NioClient.java:67)
at com.cloud.utils.nio.NioConnection.start(NioConnection.java:95)
at com.cloud.agent.Agent.start(Agent.java:293)
at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:455)
at
com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:422)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:406)
at com.cloud.agent.AgentShell.start(AgentShell.java:512)
at com.cloud.agent.AgentShell.main(AgentShell.java:547)
2021-09-04 12:16:02,255 INFO [utils.nio.NioClient] (main:null) (logid:)
NioClient connection closed
2021-09-04 12:16:02,255 INFO [cloud.agent.Agent] (main:null) (logid:)
Attempted to connect to the server, but received an unexpected exception,
trying again...
com.cloud.utils.exception.NioConnectionException: SSL Handshake failed
while connecting to host: 79.143.86.149 port: 8250
at com.cloud.utils.nio.NioConnection.start(NioConnection.java:101)
at com.cloud.agent.Agent.start(Agent.java:293)
at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:455)
at
com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:422)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:406)
at com.cloud.agent.AgentShell.start(AgentShell.java:512)
at com.cloud.agent.AgentShell.main(AgentShell.java:547)
Caused by: java.io.IOException: SSL Handshake failed while connecting to
host: 79.143.86.149 port: 8250
at com.cloud.utils.nio.NioClient.init(NioClient.java:67)
at com.cloud.utils.nio.NioConnection.start(NioConnection.java:95)
... 6 more
2021-09-04 12:16:07,256 INFO [cloud.agent.Agent] (main:null) (logid:)
Connecting to host:79.143.86.149
2021-09-04 12:16:07,256 INFO [utils.nio.NioClient] (main:null) (logid:)
Connecting to 79.143.86.149:8250
2021-09-04 12:16:07,257 INFO [utils.nio.Link] (main:null) (logid:) Conf
file found: /etc/cloudstack/agent/agent.properties
2021-09-04 12:16:07,521 ERROR [utils.nio.Link] (main:null) (logid:) SSL
error caught during unwrap data: Received fatal alert: certificate_unknown,
for local address=/79.143.86.149:53862, remote address=/79.143.86.149:8250.
The client may have invalid ca-certificates.
2021-09-04 12:16:07,521 ERROR [utils.nio.NioClient] (main:null) (logid:)
SSL Handshake failed while connecting to host: 79.143.86.149 port: 8250
2021-09-04 12:16:07,521 ERROR [utils.nio.NioConnection] (main:null)
(logid:) Unable to initialize the threads.
java.io.IOException: SSL Handshake failed while connecting to host:
79.143.86.149 port: 8250
at com.cloud.utils.nio.NioClient.init(NioClient.java:67)
at com.cloud.utils.nio.NioConnection.start(NioConnection.java:95)
at com.cloud.agent.Agent.start(Agent.java:293)
at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:455)
at
com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:422)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:406)
at com.cloud.agent.AgentShell.start(AgentShell.java:512)
at com.cloud.agent.AgentShell.main(AgentShell.java:547)
2021-09-04 12:16:07,522 INFO [utils.nio.NioClient] (main:null) (logid:)
NioClient connection closed
2021-09-04 12:16:07,522 INFO [cloud.agent.Agent] (main:null) (logid:)
Attempted to connect to the server, but received an unexpected exception,
trying again...
com.cloud.utils.exception.NioConnectionException: SSL Handshake failed
while connecting to host: 79.143.86.149 port: 8250
at com.cloud.utils.nio.NioConnection.start(NioConnection.java:101)
at com.cloud.agent.Agent.start(Agent.java:293)
at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:455)
at
com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:422)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:406)
at com.cloud.agent.AgentShell.start(AgentShell.java:512)
at com.cloud.agent.AgentShell.main(AgentShell.java:547)
Caused by: java.io.IOException: SSL Handshake failed while connecting to
host: 79.143.86.149 port: 8250
at com.cloud.utils.nio.NioClient.init(NioClient.java:67)
at com.cloud.utils.nio.NioConnection.start(NioConnection.java:95)
... 6 more
2021-09-04 12:16:12,522 INFO [cloud.agent.Agent] (main:null) (logid:)
Connecting to host:79.143.86.149
2021-09-04 12:16:12,523 INFO [utils.nio.NioClient] (main:null) (logid:)
Connecting to 79.143.86.149:8250
2021-09-04 12:16:12,523 INFO [utils.nio.Link] (main:null) (logid:) Conf
file found: /etc/cloudstack/agent/agent.properties
2021-09-04 12:16:12,776 ERROR [utils.nio.Link] (main:null) (logid:) SSL
error caught during unwrap data: Received fatal alert: certificate_unknown,
for local address=/79.143.86.149:53864, remote address=/79.143.86.149:8250.
The client may have invalid ca-certificates.
2021-09-04 12:16:12,776 ERROR [utils.nio.NioClient] (main:null) (logid:)
SSL Handshake failed while connecting to host: 79.143.86.149 port: 8250
2021-09-04 12:16:12,776 ERROR [utils.nio.NioConnection] (main:null)
(logid:) Unable to initialize the threads.
java.io.IOException: SSL Handshake failed while connecting to host:
79.143.86.149 port: 8250
at com.cloud.utils.nio.NioClient.init(NioClient.java:67)
at com.cloud.utils.nio.NioConnection.start(NioConnection.java:95)
at com.cloud.agent.Agent.start(Agent.java:293)
at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:455)
at
com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:422)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:406)
at com.cloud.agent.AgentShell.start(AgentShell.java:512)
at com.cloud.agent.AgentShell.main(AgentShell.java:547)
2021-09-04 12:16:12,777 INFO [utils.nio.NioClient] (main:null) (logid:)
NioClient connection closed
2021-09-04 12:16:12,777 INFO [cloud.agent.Agent] (main:null) (logid:)
Attempted to connect to the server, but received an unexpected exception,
trying again...
com.cloud.utils.exception.NioConnectionException: SSL Handshake failed
while connecting to host: 79.143.86.149 port: 8250
at com.cloud.utils.nio.NioConnection.start(NioConnection.java:101)
at com.cloud.agent.Agent.start(Agent.java:293)
at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:455)
at
com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:422)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:406)
at com.cloud.agent.AgentShell.start(AgentShell.java:512)
at com.cloud.agent.AgentShell.main(AgentShell.java:547)
Caused by: java.io.IOException: SSL Handshake failed while connecting to
host: 79.143.86.149 port: 8250
at com.cloud.utils.nio.NioClient.init(NioClient.java:67)
at com.cloud.utils.nio.NioConnection.start(NioConnection.java:95)
... 6 more
