looks like your ssl keys got corrupted, I suggest removing them and re-adding the hosts (as a first try. Hope this is not your production environment.
On Sat, Sep 4, 2021 at 12:20 PM Fariborz Navidan <[email protected]> wrote: > Hello, > > All VM domains in KVM suddenly terminated but CloudStack was thinking they > are running as states were not updated in the database. I just forcefully > stopped them and now they cannot start due to insufficient capacity. Below > is the agent log. i tried restarting systemd-machind, libvirtd and agent. > Also checked all ports. Port 8250 is open. > > in:null) (logid:) Connecting to 79.143.86.149:8250 > 2021-09-04 12:16:02,000 INFO [utils.nio.Link] (main:null) (logid:) Conf > file found: /etc/cloudstack/agent/agent.properties > > > > > > > > 2021-09-04 12:16:02,255 ERROR [utils.nio.Link] (main:null) (logid:) SSL > error caught during unwrap data: Received fatal alert: certificate_unknown, > for local address=/79.143.86.149:53860, remote address=/79.143.86.149:8250 > . > The client may have invalid ca-certificates. > 2021-09-04 12:16:02,255 ERROR [utils.nio.NioClient] (main:null) (logid:) > SSL Handshake failed while connecting to host: 79.143.86.149 port: 8250 > 2021-09-04 12:16:02,255 ERROR [utils.nio.NioConnection] (main:null) > (logid:) Unable to initialize the threads. > java.io.IOException: SSL Handshake failed while connecting to host: > 79.143.86.149 port: 8250 > at com.cloud.utils.nio.NioClient.init(NioClient.java:67) > at com.cloud.utils.nio.NioConnection.start(NioConnection.java:95) > at com.cloud.agent.Agent.start(Agent.java:293) > at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:455) > at > com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:422) > at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:406) > at com.cloud.agent.AgentShell.start(AgentShell.java:512) > at com.cloud.agent.AgentShell.main(AgentShell.java:547) > 2021-09-04 12:16:02,255 INFO [utils.nio.NioClient] (main:null) (logid:) > NioClient connection closed > 2021-09-04 12:16:02,255 INFO [cloud.agent.Agent] (main:null) (logid:) > Attempted to connect to the server, but received an unexpected exception, > trying again... > com.cloud.utils.exception.NioConnectionException: SSL Handshake failed > while connecting to host: 79.143.86.149 port: 8250 > at com.cloud.utils.nio.NioConnection.start(NioConnection.java:101) > at com.cloud.agent.Agent.start(Agent.java:293) > at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:455) > at > com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:422) > at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:406) > at com.cloud.agent.AgentShell.start(AgentShell.java:512) > at com.cloud.agent.AgentShell.main(AgentShell.java:547) > Caused by: java.io.IOException: SSL Handshake failed while connecting to > host: 79.143.86.149 port: 8250 > at com.cloud.utils.nio.NioClient.init(NioClient.java:67) > at com.cloud.utils.nio.NioConnection.start(NioConnection.java:95) > ... 6 more > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2021-09-04 12:16:07,256 INFO [cloud.agent.Agent] (main:null) (logid:) > Connecting to host:79.143.86.149 > 2021-09-04 12:16:07,256 INFO [utils.nio.NioClient] (main:null) (logid:) > Connecting to 79.143.86.149:8250 > 2021-09-04 12:16:07,257 INFO [utils.nio.Link] (main:null) (logid:) Conf > file found: /etc/cloudstack/agent/agent.properties > 2021-09-04 12:16:07,521 ERROR [utils.nio.Link] (main:null) (logid:) SSL > error caught during unwrap data: Received fatal alert: certificate_unknown, > for local address=/79.143.86.149:53862, remote address=/79.143.86.149:8250 > . > The client may have invalid ca-certificates. > 2021-09-04 12:16:07,521 ERROR [utils.nio.NioClient] (main:null) (logid:) > SSL Handshake failed while connecting to host: 79.143.86.149 port: 8250 > 2021-09-04 12:16:07,521 ERROR [utils.nio.NioConnection] (main:null) > (logid:) Unable to initialize the threads. > java.io.IOException: SSL Handshake failed while connecting to host: > 79.143.86.149 port: 8250 > at com.cloud.utils.nio.NioClient.init(NioClient.java:67) > at com.cloud.utils.nio.NioConnection.start(NioConnection.java:95) > at com.cloud.agent.Agent.start(Agent.java:293) > at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:455) > at > com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:422) > at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:406) > at com.cloud.agent.AgentShell.start(AgentShell.java:512) > at com.cloud.agent.AgentShell.main(AgentShell.java:547) > 2021-09-04 12:16:07,522 INFO [utils.nio.NioClient] (main:null) (logid:) > NioClient connection closed > 2021-09-04 12:16:07,522 INFO [cloud.agent.Agent] (main:null) (logid:) > Attempted to connect to the server, but received an unexpected exception, > trying again... > com.cloud.utils.exception.NioConnectionException: SSL Handshake failed > while connecting to host: 79.143.86.149 port: 8250 > at com.cloud.utils.nio.NioConnection.start(NioConnection.java:101) > at com.cloud.agent.Agent.start(Agent.java:293) > at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:455) > at > com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:422) > at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:406) > at com.cloud.agent.AgentShell.start(AgentShell.java:512) > at com.cloud.agent.AgentShell.main(AgentShell.java:547) > Caused by: java.io.IOException: SSL Handshake failed while connecting to > host: 79.143.86.149 port: 8250 > at com.cloud.utils.nio.NioClient.init(NioClient.java:67) > at com.cloud.utils.nio.NioConnection.start(NioConnection.java:95) > ... 6 more > 2021-09-04 12:16:12,522 INFO [cloud.agent.Agent] (main:null) (logid:) > Connecting to host:79.143.86.149 > 2021-09-04 12:16:12,523 INFO [utils.nio.NioClient] (main:null) (logid:) > Connecting to 79.143.86.149:8250 > 2021-09-04 12:16:12,523 INFO [utils.nio.Link] (main:null) (logid:) Conf > file found: /etc/cloudstack/agent/agent.properties > 2021-09-04 12:16:12,776 ERROR [utils.nio.Link] (main:null) (logid:) SSL > error caught during unwrap data: Received fatal alert: certificate_unknown, > for local address=/79.143.86.149:53864, remote address=/79.143.86.149:8250 > . > The client may have invalid ca-certificates. > 2021-09-04 12:16:12,776 ERROR [utils.nio.NioClient] (main:null) (logid:) > SSL Handshake failed while connecting to host: 79.143.86.149 port: 8250 > 2021-09-04 12:16:12,776 ERROR [utils.nio.NioConnection] (main:null) > (logid:) Unable to initialize the threads. > java.io.IOException: SSL Handshake failed while connecting to host: > 79.143.86.149 port: 8250 > at com.cloud.utils.nio.NioClient.init(NioClient.java:67) > at com.cloud.utils.nio.NioConnection.start(NioConnection.java:95) > at com.cloud.agent.Agent.start(Agent.java:293) > at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:455) > at > com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:422) > at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:406) > at com.cloud.agent.AgentShell.start(AgentShell.java:512) > at com.cloud.agent.AgentShell.main(AgentShell.java:547) > 2021-09-04 12:16:12,777 INFO [utils.nio.NioClient] (main:null) (logid:) > NioClient connection closed > 2021-09-04 12:16:12,777 INFO [cloud.agent.Agent] (main:null) (logid:) > Attempted to connect to the server, but received an unexpected exception, > trying again... > com.cloud.utils.exception.NioConnectionException: SSL Handshake failed > while connecting to host: 79.143.86.149 port: 8250 > at com.cloud.utils.nio.NioConnection.start(NioConnection.java:101) > at com.cloud.agent.Agent.start(Agent.java:293) > at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:455) > at > com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:422) > at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:406) > at com.cloud.agent.AgentShell.start(AgentShell.java:512) > at com.cloud.agent.AgentShell.main(AgentShell.java:547) > Caused by: java.io.IOException: SSL Handshake failed while connecting to > host: 79.143.86.149 port: 8250 > at com.cloud.utils.nio.NioClient.init(NioClient.java:67) > at com.cloud.utils.nio.NioConnection.start(NioConnection.java:95) > ... 6 more > -- Daan
