Re: host time out after cloudstack management restore
glad to hear it works for you. the MS should be portable across platforms but some scripts may have been an issue. On Tue, Aug 9, 2022 at 9:08 AM Cristian Ciobanu wrote: > Yes, it was not related to that, I just reinstalled again with Ubuntu and > restored the DB, everything works fine. > > Regards, > Cristian > > On Tue, Aug 9, 2022, 09:47 Daan Hoogland wrote: > > > Christian, can you reached the vcenter as well from the MS? And can the > > vcenter reach the hosts? > > > > On Mon, 8 Aug 2022, 17:56 , wrote: > > > > > Hello, > > > > > > > > > > > > I had a server fail today, because of that I restored the > > Cloudstack > > > management on a fresh install and used the last db backup I had. > > > Everything > > > looks fine in the UI, but not working, not sure why I see time out for > > all > > > hosts, this is not true, I can reach all the hosts via ip or hostname > > from > > > cloudstack management. ( all the hosts are connected in vCenter) > > > > > > > > > > > > I see these events in the logs for all hosts. > > > > > > > > > > > > 2022-08-08 15:53:16,533 WARN [c.c.v.UserVmManagerImpl] > > > (StatsCollector-2:ctx-faa5a90c) (logid:0c0862da) Unable to obtain VM > > > statistics. > > > > > > 2022-08-08 15:53:16,548 DEBUG [c.c.a.m.ClusteredAgentAttache] > > > (StatsCollector-2:ctx-faa5a90c) (logid:0c0862da) Seq > > > 42-8639592935156875294: > > > Forwarding null to 39908814623087 > > > > > > 2022-08-08 15:53:16,548 DEBUG [c.c.a.m.ClusteredAgentAttache] > > > (AgentManager-Handler-8:null) (logid:) Seq 42-8639592935156875294: > > Routing > > > from 2130706433 > > > > > > 2022-08-08 15:53:16,548 DEBUG [c.c.a.m.ClusteredAgentAttache] > > > (AgentManager-Handler-8:null) (logid:) Seq 42-8639592935156875294: Link > > is > > > closed > > > > > > 2022-08-08 15:53:16,548 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > > > (AgentManager-Handler-8:null) (logid:) Seq 42-8639592935156875294: > MgmtId > > > 2130706433: Req: Resource [Host:42] is unreachable: Host 42: Link is > > closed > > > > > > > > > > > > > > > > > > > > > > > > 2022-08-08 15:54:13,213 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > > > (AgentManager-Handler-3:null) (logid:) Seq 37--1: MgmtId 2130706433: > Req: > > > Routing to peer > > > > > > 2022-08-08 15:54:13,213 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > > > (AgentManager-Handler-4:null) (logid:) Seq 37--1: MgmtId 2130706433: > Req: > > > Cancel request received > > > > > > 2022-08-08 15:54:13,213 DEBUG [c.c.a.m.AgentAttache] > > > (AgentManager-Handler-4:null) (logid:) Seq 37-3913346601208250383: > > > Cancelling. > > > > > > 2022-08-08 15:54:13,213 DEBUG [c.c.a.m.AgentAttache] > > > (StatsCollector-6:ctx-353c6d50) (logid:7f9ce0ca) Seq > > > 37-3913346601208250383: > > > Waiting some more time because this is the current command > > > > > > 2022-08-08 15:54:13,213 DEBUG [c.c.a.m.AgentAttache] > > > (StatsCollector-6:ctx-353c6d50) (logid:7f9ce0ca) Seq > > > 37-3913346601208250383: > > > Waiting some more time because this is the current command > > > > > > 2022-08-08 15:54:13,213 INFO [c.c.u.e.CSExceptionErrorCode] > > > (StatsCollector-6:ctx-353c6d50) (logid:7f9ce0ca) Could not find > > exception: > > > com.cloud.exception.OperationTimedoutException in error code list for > > > exceptions > > > > > > 2022-08-08 15:54:13,213 WARN [c.c.a.m.AgentAttache] > > > (StatsCollector-6:ctx-353c6d50) (logid:7f9ce0ca) Seq > > > 37-3913346601208250383: > > > Timed out on null > > > > > > 2022-08-08 15:54:13,213 DEBUG [c.c.a.m.AgentAttache] > > > (StatsCollector-6:ctx-353c6d50) (logid:7f9ce0ca) Seq > > > 37-3913346601208250383: > > > Cancelling. > > > > > > > > > > > > > > > > > > > > > > > >The only difference right now, is the OS, I restored on rocky Linux > 8, > > > previously I had ubuntu 20.04. > > > > > > > > > > > > The cloudstack version is 4.16.1.0 > > > > > > Regards, > > > > > > Cristian > > > > > > > > > -- Daan
Re: host time out after cloudstack management restore
Yes, it was not related to that, I just reinstalled again with Ubuntu and restored the DB, everything works fine. Regards, Cristian On Tue, Aug 9, 2022, 09:47 Daan Hoogland wrote: > Christian, can you reached the vcenter as well from the MS? And can the > vcenter reach the hosts? > > On Mon, 8 Aug 2022, 17:56 , wrote: > > > Hello, > > > > > > > > I had a server fail today, because of that I restored the > Cloudstack > > management on a fresh install and used the last db backup I had. > > Everything > > looks fine in the UI, but not working, not sure why I see time out for > all > > hosts, this is not true, I can reach all the hosts via ip or hostname > from > > cloudstack management. ( all the hosts are connected in vCenter) > > > > > > > > I see these events in the logs for all hosts. > > > > > > > > 2022-08-08 15:53:16,533 WARN [c.c.v.UserVmManagerImpl] > > (StatsCollector-2:ctx-faa5a90c) (logid:0c0862da) Unable to obtain VM > > statistics. > > > > 2022-08-08 15:53:16,548 DEBUG [c.c.a.m.ClusteredAgentAttache] > > (StatsCollector-2:ctx-faa5a90c) (logid:0c0862da) Seq > > 42-8639592935156875294: > > Forwarding null to 39908814623087 > > > > 2022-08-08 15:53:16,548 DEBUG [c.c.a.m.ClusteredAgentAttache] > > (AgentManager-Handler-8:null) (logid:) Seq 42-8639592935156875294: > Routing > > from 2130706433 > > > > 2022-08-08 15:53:16,548 DEBUG [c.c.a.m.ClusteredAgentAttache] > > (AgentManager-Handler-8:null) (logid:) Seq 42-8639592935156875294: Link > is > > closed > > > > 2022-08-08 15:53:16,548 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > > (AgentManager-Handler-8:null) (logid:) Seq 42-8639592935156875294: MgmtId > > 2130706433: Req: Resource [Host:42] is unreachable: Host 42: Link is > closed > > > > > > > > > > > > > > > > 2022-08-08 15:54:13,213 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > > (AgentManager-Handler-3:null) (logid:) Seq 37--1: MgmtId 2130706433: Req: > > Routing to peer > > > > 2022-08-08 15:54:13,213 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > > (AgentManager-Handler-4:null) (logid:) Seq 37--1: MgmtId 2130706433: Req: > > Cancel request received > > > > 2022-08-08 15:54:13,213 DEBUG [c.c.a.m.AgentAttache] > > (AgentManager-Handler-4:null) (logid:) Seq 37-3913346601208250383: > > Cancelling. > > > > 2022-08-08 15:54:13,213 DEBUG [c.c.a.m.AgentAttache] > > (StatsCollector-6:ctx-353c6d50) (logid:7f9ce0ca) Seq > > 37-3913346601208250383: > > Waiting some more time because this is the current command > > > > 2022-08-08 15:54:13,213 DEBUG [c.c.a.m.AgentAttache] > > (StatsCollector-6:ctx-353c6d50) (logid:7f9ce0ca) Seq > > 37-3913346601208250383: > > Waiting some more time because this is the current command > > > > 2022-08-08 15:54:13,213 INFO [c.c.u.e.CSExceptionErrorCode] > > (StatsCollector-6:ctx-353c6d50) (logid:7f9ce0ca) Could not find > exception: > > com.cloud.exception.OperationTimedoutException in error code list for > > exceptions > > > > 2022-08-08 15:54:13,213 WARN [c.c.a.m.AgentAttache] > > (StatsCollector-6:ctx-353c6d50) (logid:7f9ce0ca) Seq > > 37-3913346601208250383: > > Timed out on null > > > > 2022-08-08 15:54:13,213 DEBUG [c.c.a.m.AgentAttache] > > (StatsCollector-6:ctx-353c6d50) (logid:7f9ce0ca) Seq > > 37-3913346601208250383: > > Cancelling. > > > > > > > > > > > > > > > >The only difference right now, is the OS, I restored on rocky Linux 8, > > previously I had ubuntu 20.04. > > > > > > > > The cloudstack version is 4.16.1.0 > > > > Regards, > > > > Cristian > > > > >
Re: host time out after cloudstack management restore
Christian, can you reached the vcenter as well from the MS? And can the vcenter reach the hosts? On Mon, 8 Aug 2022, 17:56 , wrote: > Hello, > > > > I had a server fail today, because of that I restored the Cloudstack > management on a fresh install and used the last db backup I had. > Everything > looks fine in the UI, but not working, not sure why I see time out for all > hosts, this is not true, I can reach all the hosts via ip or hostname from > cloudstack management. ( all the hosts are connected in vCenter) > > > > I see these events in the logs for all hosts. > > > > 2022-08-08 15:53:16,533 WARN [c.c.v.UserVmManagerImpl] > (StatsCollector-2:ctx-faa5a90c) (logid:0c0862da) Unable to obtain VM > statistics. > > 2022-08-08 15:53:16,548 DEBUG [c.c.a.m.ClusteredAgentAttache] > (StatsCollector-2:ctx-faa5a90c) (logid:0c0862da) Seq > 42-8639592935156875294: > Forwarding null to 39908814623087 > > 2022-08-08 15:53:16,548 DEBUG [c.c.a.m.ClusteredAgentAttache] > (AgentManager-Handler-8:null) (logid:) Seq 42-8639592935156875294: Routing > from 2130706433 > > 2022-08-08 15:53:16,548 DEBUG [c.c.a.m.ClusteredAgentAttache] > (AgentManager-Handler-8:null) (logid:) Seq 42-8639592935156875294: Link is > closed > > 2022-08-08 15:53:16,548 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > (AgentManager-Handler-8:null) (logid:) Seq 42-8639592935156875294: MgmtId > 2130706433: Req: Resource [Host:42] is unreachable: Host 42: Link is closed > > > > > > > > 2022-08-08 15:54:13,213 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > (AgentManager-Handler-3:null) (logid:) Seq 37--1: MgmtId 2130706433: Req: > Routing to peer > > 2022-08-08 15:54:13,213 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > (AgentManager-Handler-4:null) (logid:) Seq 37--1: MgmtId 2130706433: Req: > Cancel request received > > 2022-08-08 15:54:13,213 DEBUG [c.c.a.m.AgentAttache] > (AgentManager-Handler-4:null) (logid:) Seq 37-3913346601208250383: > Cancelling. > > 2022-08-08 15:54:13,213 DEBUG [c.c.a.m.AgentAttache] > (StatsCollector-6:ctx-353c6d50) (logid:7f9ce0ca) Seq > 37-3913346601208250383: > Waiting some more time because this is the current command > > 2022-08-08 15:54:13,213 DEBUG [c.c.a.m.AgentAttache] > (StatsCollector-6:ctx-353c6d50) (logid:7f9ce0ca) Seq > 37-3913346601208250383: > Waiting some more time because this is the current command > > 2022-08-08 15:54:13,213 INFO [c.c.u.e.CSExceptionErrorCode] > (StatsCollector-6:ctx-353c6d50) (logid:7f9ce0ca) Could not find exception: > com.cloud.exception.OperationTimedoutException in error code list for > exceptions > > 2022-08-08 15:54:13,213 WARN [c.c.a.m.AgentAttache] > (StatsCollector-6:ctx-353c6d50) (logid:7f9ce0ca) Seq > 37-3913346601208250383: > Timed out on null > > 2022-08-08 15:54:13,213 DEBUG [c.c.a.m.AgentAttache] > (StatsCollector-6:ctx-353c6d50) (logid:7f9ce0ca) Seq > 37-3913346601208250383: > Cancelling. > > > > > > > >The only difference right now, is the OS, I restored on rocky Linux 8, > previously I had ubuntu 20.04. > > > > The cloudstack version is 4.16.1.0 > > Regards, > > Cristian > >
host time out after cloudstack management restore
Hello, I had a server fail today, because of that I restored the Cloudstack management on a fresh install and used the last db backup I had. Everything looks fine in the UI, but not working, not sure why I see time out for all hosts, this is not true, I can reach all the hosts via ip or hostname from cloudstack management. ( all the hosts are connected in vCenter) I see these events in the logs for all hosts. 2022-08-08 15:53:16,533 WARN [c.c.v.UserVmManagerImpl] (StatsCollector-2:ctx-faa5a90c) (logid:0c0862da) Unable to obtain VM statistics. 2022-08-08 15:53:16,548 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:ctx-faa5a90c) (logid:0c0862da) Seq 42-8639592935156875294: Forwarding null to 39908814623087 2022-08-08 15:53:16,548 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-8:null) (logid:) Seq 42-8639592935156875294: Routing from 2130706433 2022-08-08 15:53:16,548 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-8:null) (logid:) Seq 42-8639592935156875294: Link is closed 2022-08-08 15:53:16,548 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:null) (logid:) Seq 42-8639592935156875294: MgmtId 2130706433: Req: Resource [Host:42] is unreachable: Host 42: Link is closed 2022-08-08 15:54:13,213 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:null) (logid:) Seq 37--1: MgmtId 2130706433: Req: Routing to peer 2022-08-08 15:54:13,213 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:null) (logid:) Seq 37--1: MgmtId 2130706433: Req: Cancel request received 2022-08-08 15:54:13,213 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-4:null) (logid:) Seq 37-3913346601208250383: Cancelling. 2022-08-08 15:54:13,213 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-6:ctx-353c6d50) (logid:7f9ce0ca) Seq 37-3913346601208250383: Waiting some more time because this is the current command 2022-08-08 15:54:13,213 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-6:ctx-353c6d50) (logid:7f9ce0ca) Seq 37-3913346601208250383: Waiting some more time because this is the current command 2022-08-08 15:54:13,213 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-6:ctx-353c6d50) (logid:7f9ce0ca) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2022-08-08 15:54:13,213 WARN [c.c.a.m.AgentAttache] (StatsCollector-6:ctx-353c6d50) (logid:7f9ce0ca) Seq 37-3913346601208250383: Timed out on null 2022-08-08 15:54:13,213 DEBUG [c.c.a.m.AgentAttache] (StatsCollector-6:ctx-353c6d50) (logid:7f9ce0ca) Seq 37-3913346601208250383: Cancelling. The only difference right now, is the OS, I restored on rocky Linux 8, previously I had ubuntu 20.04. The cloudstack version is 4.16.1.0 Regards, Cristian
