justinestruch commented on issue #11081: URL: https://github.com/apache/cloudstack/issues/11081#issuecomment-3110104679
> ### problem > Currently, the systemvms using the template (https://download.cloudstack.org/systemvm/4.20/systemvmtemplate-4.20.1-x86_64-vmware.ova ) and running on vmware 8u03 and Cloudstack 4.20.1 are not connecting to the management server, the agent state is not up > > ### versions > Cloudstack: 4.20.1 Vmware: 8u03 systemvm template : https://download.cloudstack.org/systemvm/4.20/systemvmtemplate-4.20.1-x86_64-vmware.ova > > ### The steps to reproduce the bug > 1. Create a clousstack (4.20.1) env with vmware 8u03 > 2. Observe the systemvm state , the agent state will not be up > 3. Check the logs of the systemvm > > logs > > ``` > 2025-06-23T06:50:39,496 INFO [utils.nio.NioClient] (Agent-Handler-2:[]) Connected to 10.0.35.27:8250 > 2025-06-23T06:50:39,497 INFO [utils.nio.Link] (Agent-Handler-2:[]) Conf file found: /usr/local/cloud/systemvm/conf/agent.properties > 2025-06-23T06:50:39,754 ERROR [utils.nio.Link] (Agent-Handler-2:[]) SSL error caught during wrap data: No trusted certificate found, for local address=/10.0.43.213:44648, remote address=/10.0.35.27:8250. > 2025-06-23T06:50:39,757 INFO [utils.nio.NioClient] (Agent-Handler-2:[]) SSL: Handshake done > 2025-06-23T06:50:39,805 INFO [cloud.agent.Agent] (Agent-Handler-2:[]) Lost connection to host: 10.0.35.27. Attempting reconnection while we still have 0 commands in progress. > 2025-06-23T06:50:39,810 INFO [utils.nio.NioClient] (Agent-Handler-2:[]) NioClient connection closed > 2025-06-23T06:50:39,813 ERROR [utils.nio.Link] (Agent-Handler-2:[]) SSL error caught during wrap data: No trusted certificate found, for local address=/10.0.43.213:44640, remote address=/10.0.35.27:8250. > 2025-06-23T06:50:39,814 INFO [utils.nio.NioClient] (Agent-Handler-2:[]) SSL: Handshake done > 2025-06-23T06:50:39,822 WARN [cloud.agent.Agent] (Agent-Handler-1:[]) Unable to send request to /10.0.35.27:8250 due to 'null', request: null > 2025-06-23T06:50:39,809 ERROR [utils.nio.NioClient] (Agent-Handler-2:[]) IOException while connecting to 10.0.35.27:8250 java.nio.channels.ClosedChannelException > at java.base/sun.nio.ch.SocketChannelImpl.ensureOpenAndConnected(SocketChannelImpl.java:215) > at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:403) > at com.cloud.utils.nio.Link.doHandshakeUnwrap(Link.java:487) > at com.cloud.utils.nio.Link.doHandshake(Link.java:627) > at com.cloud.utils.nio.NioClient.init(NioClient.java:74) > at com.cloud.utils.nio.NioConnection.start(NioConnection.java:112) > at com.cloud.agent.Agent.reconnect(Agent.java:655) > at com.cloud.agent.Agent$ServerHandler.doTask(Agent.java:1233) > at com.cloud.utils.nio.Task.call(Task.java:83) > at com.cloud.utils.nio.Task.call(Task.java:29) > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) > at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) > at java.base/java.lang.Thread.run(Thread.java:840) > > 2025-06-23T06:50:39,851 ERROR [utils.nio.NioClient] (Agent-Handler-2:[]) Unable to initialize the threads. java.nio.channels.ClosedChannelException > ``` > > ### What to do about it? > There should be no ssl related errors when using the systemvm template > > https://download.cloudstack.org/systemvm/4.20/systemvmtemplate-4.20.1-x86_64-vmware.ova > > As a workaround, execute the following command on the systemvm's to reimport cloud.ca.crt into cloud.jks -trustcacerts is removed from the keytool command, so `cacerts` will not be checked when import the ca cert > > ``` > KS_FILE=/usr/local/cloud/systemvm/conf/cloud.jks KS_PASS=$(grep keystore.passphrase /usr/local/cloud/systemvm/conf/agent.properties |cut -d "=" -f2) keytool -import -noprompt -storepass "$KS_PASS" -alias "cloudca.1" -file "/usr/local/cloud/systemvm/conf/cloud.ca.crt" -keystore "$KS_FILE" > ``` @ > ### problem > Currently, the systemvms using the template (https://download.cloudstack.org/systemvm/4.20/systemvmtemplate-4.20.1-x86_64-vmware.ova ) and running on vmware 8u03 and Cloudstack 4.20.1 are not connecting to the management server, the agent state is not up > > ### versions > Cloudstack: 4.20.1 Vmware: 8u03 systemvm template : https://download.cloudstack.org/systemvm/4.20/systemvmtemplate-4.20.1-x86_64-vmware.ova > > ### The steps to reproduce the bug > 1. Create a clousstack (4.20.1) env with vmware 8u03 > 2. Observe the systemvm state , the agent state will not be up > 3. Check the logs of the systemvm > > logs > > ``` > 2025-06-23T06:50:39,496 INFO [utils.nio.NioClient] (Agent-Handler-2:[]) Connected to 10.0.35.27:8250 > 2025-06-23T06:50:39,497 INFO [utils.nio.Link] (Agent-Handler-2:[]) Conf file found: /usr/local/cloud/systemvm/conf/agent.properties > 2025-06-23T06:50:39,754 ERROR [utils.nio.Link] (Agent-Handler-2:[]) SSL error caught during wrap data: No trusted certificate found, for local address=/10.0.43.213:44648, remote address=/10.0.35.27:8250. > 2025-06-23T06:50:39,757 INFO [utils.nio.NioClient] (Agent-Handler-2:[]) SSL: Handshake done > 2025-06-23T06:50:39,805 INFO [cloud.agent.Agent] (Agent-Handler-2:[]) Lost connection to host: 10.0.35.27. Attempting reconnection while we still have 0 commands in progress. > 2025-06-23T06:50:39,810 INFO [utils.nio.NioClient] (Agent-Handler-2:[]) NioClient connection closed > 2025-06-23T06:50:39,813 ERROR [utils.nio.Link] (Agent-Handler-2:[]) SSL error caught during wrap data: No trusted certificate found, for local address=/10.0.43.213:44640, remote address=/10.0.35.27:8250. > 2025-06-23T06:50:39,814 INFO [utils.nio.NioClient] (Agent-Handler-2:[]) SSL: Handshake done > 2025-06-23T06:50:39,822 WARN [cloud.agent.Agent] (Agent-Handler-1:[]) Unable to send request to /10.0.35.27:8250 due to 'null', request: null > 2025-06-23T06:50:39,809 ERROR [utils.nio.NioClient] (Agent-Handler-2:[]) IOException while connecting to 10.0.35.27:8250 java.nio.channels.ClosedChannelException > at java.base/sun.nio.ch.SocketChannelImpl.ensureOpenAndConnected(SocketChannelImpl.java:215) > at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:403) > at com.cloud.utils.nio.Link.doHandshakeUnwrap(Link.java:487) > at com.cloud.utils.nio.Link.doHandshake(Link.java:627) > at com.cloud.utils.nio.NioClient.init(NioClient.java:74) > at com.cloud.utils.nio.NioConnection.start(NioConnection.java:112) > at com.cloud.agent.Agent.reconnect(Agent.java:655) > at com.cloud.agent.Agent$ServerHandler.doTask(Agent.java:1233) > at com.cloud.utils.nio.Task.call(Task.java:83) > at com.cloud.utils.nio.Task.call(Task.java:29) > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) > at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) > at java.base/java.lang.Thread.run(Thread.java:840) > > 2025-06-23T06:50:39,851 ERROR [utils.nio.NioClient] (Agent-Handler-2:[]) Unable to initialize the threads. java.nio.channels.ClosedChannelException > ``` > > ### What to do about it? > There should be no ssl related errors when using the systemvm template > > https://download.cloudstack.org/systemvm/4.20/systemvmtemplate-4.20.1-x86_64-vmware.ova > > As a workaround, execute the following command on the systemvm's to reimport cloud.ca.crt into cloud.jks -trustcacerts is removed from the keytool command, so `cacerts` will not be checked when import the ca cert > > ``` > KS_FILE=/usr/local/cloud/systemvm/conf/cloud.jks KS_PASS=$(grep keystore.passphrase /usr/local/cloud/systemvm/conf/agent.properties |cut -d "=" -f2) keytool -import -noprompt -storepass "$KS_PASS" -alias "cloudca.1" -file "/usr/local/cloud/systemvm/conf/cloud.ca.crt" -keystore "$KS_FILE" > ``` I confirm I had the same issue, the system VMs would deploy but the agent status would never come up. Your workaround worked for me as well. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@cloudstack.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
