Hi,
We already tested in all the ways is possible, this problem is related to
these specific servers, 100%.
The last test we did, was to add the ESXI server to the zone we already have
and works, we disabled all hosts from that zone and destroyed the SSVM to force
a redeploy on this ESXI, of course, we have ended with the same error.
I want to mention that we have the exact same ESXI version and hardware, for
me looks like there is a BIOS/HW/DISK issue.
FYI:
"I did this in 3 different new Setup (new Management install, NFS, etc), no
matter if I add as first zone or second zone.
" I have tested this with multiple Cloudstack versions, 4.15.0, 4.15.1,
4.15.2 and VMware, 6.5 and with 6.7 different patches and with the last
patches. I'm 100% that is related to these 2 servers, I do not understand what
is wrong with these servers, this is the problem.
We have 2 identical servers' hardware/ESXI, the only difference is the
location, the not working, are in UK, and any other location we have, DE, NYC,
FR, CA, etc. works fine. But we tested with the exact same servers which are
from DE(Germany) works perfect, with any ACS or VMware version (also same CIDR
range/size for public and private), when we test with UK, we get same error no
matter what."
I want to mention that we have these servers (UK, DE, FR, CA ) from OVH in
vRack and the UK, are not working. ( I manage these server for more than 5
years )"
Regards,
Cristian
-----Original Message-----
From: Daan Hoogland <daan.hoogl...@gmail.com>
Sent: Tuesday, September 28, 2021 11:00 AM
To: users <users@cloudstack.apache.org>
Subject: Re: Unable to read/process CSR: Command failed due to Exception:
java.lang.Exception
Christian, did you solve this (i just encountered your mail and see it is yet
unanswered) the problem is with the certificate generation, it seems to try and
generate from an empty string (the csr/certificate signing request) Not being
able to sign in to the SVM is a problem but on vmware you would use a specific
command as described in [1]. Did you try `ssh -i /opt/xensource/bin/id_rsa --p
3922 root@privateIP <root@10.209.161.128> OfTheHost`?
[1]
https://cwiki.apache.org/confluence/display/CLOUDSTACK/SSVM%2C+templates%2C+Secondary+storage+troubleshooting
On Thu, Sep 23, 2021 at 11:31 AM <cristian.c@istream.today> wrote:
> Hello,
>
>
>
> I'm not sure why I get this error (added new vmware zone),
> there are not permission issues for systemvms folder, and I do not see
> any error above this. If I try to login to login into SSVM/Proxy
> console via cloudstack management, I get this access denied.
>
>
>
> Any suggestion?
>
>
>
>
>
>
>
>
>
> 021-09-23 05:22:15,012 ERROR [c.c.u.s.SshHelper]
> (DirectAgent-283:ctx-b1e493bd lnd-uk-001.shape.host,
> job-34950/job-34972,
> cmd: SetupKeyStoreCommand) (logid:f3cd2252) Failed to authentication
> SSH user root on host 10.15.0.160
>
> 2021-09-23 05:22:15,012 INFO [c.c.h.v.u.VmwareHelper]
> (DirectAgent-283:ctx-b1e493bd lnd-uk-001.shape.host,
> job-34950/job-34972,
> cmd: SetupKeyStoreCommand) (logid:f3cd2252) [ignored]failed to get
> message for exception: Failed to authentication SSH user root on host
> 10.15.0.160
>
> 2021-09-23 05:22:15,012 ERROR [c.c.h.v.r.VmwareResource]
> (DirectAgent-283:ctx-b1e493bd lnd-uk-001.shape.host,
> job-34950/job-34972,
> cmd: SetupKeyStoreCommand) (logid:f3cd2252) Command failed due to
> Exception:
> java.lang.Exception
>
> Message: Failed to authentication SSH user root on host 10.15.0.160
>
>
>
> 2021-09-23 05:22:15,012 DEBUG [c.c.h.v.r.VmwareResource]
> (DirectAgent-283:ctx-b1e493bd lnd-uk-001.shape.host,
> job-34950/job-34972,
> cmd: SetupKeyStoreCommand) (logid:f3cd2252) keystore-setup execution
> result:
> false
>
> 2021-09-23 05:22:15,014 DEBUG [c.c.a.m.DirectAgentAttache]
> (DirectAgent-283:ctx-b1e493bd) (logid:f3cd2252) Seq 40-1394708509601300577:
> Response Received:
>
> 2021-09-23 05:22:15,016 DEBUG [c.c.a.t.Request]
> (DirectAgent-283:ctx-b1e493bd) (logid:f3cd2252) Seq 40-1394708509601300577:
> Processing: { Ans: , MgmtId: 345049356158, via:
> 40(lnd-uk-001.shape.host),
> Ver: v1, Flags: 10,
> [{"org.apache.cloudstack.ca
> .SetupKeystoreAnswer":{"result":"true","wait":"0"
> ,"bypassHostMaintenance":"false"}}] }
>
> 2021-09-23 05:22:15,016 DEBUG [c.c.a.t.Request]
> (Work-Job-Executor-14:ctx-10c822ed job-34950/job-34972 ctx-67aa0df0)
> (logid:f3cd2252) Seq 40-1394708509601300577: Received: { Ans: , MgmtId:
> 345049356158, via: 40(lnd-uk-001.test.host), Ver: v1, Flags: 10, {
> SetupKeystoreAnswer } }
>
> 2021-09-23 05:22:15,022 ERROR [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-14:ctx-10c822ed job-34950/job-34972 ctx-67aa0df0)
> (logid:f3cd2252) Retrying after catching exception while trying to
> secure agent for systemvm id=1955
>
> com.cloud.utils.exception.CloudRuntimeException: Unable to
> read/process
> CSR:
> Command failed due to Exception: java.lang.Exception
>
> Message: Failed to authentication SSH user root on host 10.15.0.160
>
>
>
> at
> org.apache.cloudstack.ca
> .provider.RootCAProvider.generateCertificateUsingCsr
> (RootCAProvider.java:170)
>
> at
> org.apache.cloudstack.ca
> .provider.RootCAProvider.issueCertificate(RootCAProv
> ider.java:230)
>
> at
> org.apache.cloudstack.ca
> .CAManagerImpl.issueCertificate(CAManagerImpl.java:1
> 64)
>
> at
> jdk.internal.reflect.GeneratedMethodAccessor318.invoke(Unknown
> Source)
>
> at
>
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Del
> egatin
> gMethodAccessorImpl.java:43)
>
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>
> at
>
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflectio
> n(AopU
> tils.java:344)
>
> at
>
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoi
> npoint
> (ReflectiveMethodInvocation.java:198)
>
> at
>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(R
> eflect
> iveMethodInvocation.java:163)
>
> at
>
> org.apache.cloudstack.network.contrail.management.EventUtils$EventInte
> rcepto
> r.invoke(EventUtils.java:107)
>
> at
>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(R
> eflect
> iveMethodInvocation.java:175)
>
> at
>
> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.j
> ava:51
> )
>
> at
>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(R
> eflect
> iveMethodInvocation.java:175)
>
> at
>
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke
> (Expos
> eInvocationInterceptor.java:97)
>
> at
>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(R
> eflect
> iveMethodInvocation.java:186)
>
> at
>
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamic
> AopPro
> xy.java:215)
>
> at com.sun.proxy.$Proxy292.issueCertificate(Unknown Source)
>
> at
>
> com.cloud.vm.VirtualMachineManagerImpl.setupAgentSecurity(VirtualMachi
> neMana
> gerImpl.java:970)
>
> at
>
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachine
> Manage
> rImpl.java:1241)
>
> at
>
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachine
> Manage
> rImpl.java:5502)
>
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
>
> at
>
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeM
> ethodA
> ccessorImpl.java:62)
>
> at
>
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Del
> egatin
> gMethodAccessorImpl.java:43)
>
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>
> at
>
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerPro
> xy.jav
> a:107)
>
> at
>
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineM
> anager
> Impl.java:5669)
>
> at
> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>
> at
>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInC
> ontext
> (AsyncJobManagerImpl.java:620)
>
> at
>
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(Man
> agedCo
> ntextRunnable.java:48)
>
> at
>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.cal
> l(Defa
> ultManagedContext.java:55)
>
> at
>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callW
> ithCon
> text(DefaultManagedContext.java:102)
>
> at
>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWi
> thCont
> ext(DefaultManagedContext.java:52)
>
> at
>
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Manag
> edCont
> extRunnable.java:45)
>
> at
>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(As
> yncJob
> ManagerImpl.java:568)
>
> at
>
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executor
> s.java
> :515)
>
> at
> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>
> at
>
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
> Execut
> or.java:1128)
>
> at
>
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
> lExecu
> tor.java:628)
>
> at java.base/java.lang.Thread.run(Thread.java:829)
>
>
>
>
>
>
>
> Regards,
>
> Cristian
>
>
--
Daan
