Hi Andrija, Setting the host to management IP, and not our LB VIP. The agent process seems to stall
2020-03-18 07:44:01,969 DEBUG [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:null) (logid:) cpus=32, speed=3800, ram=269385596928, _dom0MinMem=1073741824, _dom0OvercommitMem=0, cpu sockets=2 2020-03-18 07:44:01,972 DEBUG [cloud.resource.ServerResourceBase] (Agent-Handler-1:null) (logid:) Parameters for private nic: 10.10.240.126 - a0:36:9f:24:32:6a-255.255.255.0 2020-03-18 07:44:01,973 DEBUG [cloud.resource.ServerResourceBase] (Agent-Handler-1:null) (logid:) Parameters for storage nic: 10.10.240.126 - a0:36:9f:24:32:6a-255.255.255.0 2020-03-18 07:44:01,973 DEBUG [cloud.resource.ServerResourceBase] (Agent-Handler-1:null) (logid:) Parameters for public nic: 10.10.240.126 - a0:36:9f:24:32:6a-255.255.255.0 2020-03-18 07:44:01,973 DEBUG [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:null) (logid:) Executing: /usr/share/cloudstack-common/scripts/vm/hypervisor/versions.sh 2020-03-18 07:44:01,984 DEBUG [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:null) (logid:) Execution is successful. 2020-03-18 07:44:01,985 DEBUG [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:null) (logid:) Executing: sudo grep InitiatorName= /etc/iscsi/initiatorname.iscsi 2020-03-18 07:44:01,992 DEBUG [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:null) (logid:) Execution is successful. 2020-03-18 07:44:01,993 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Attempting to create storage pool 257c9140-3863-4be2-85e2-9835650f1b45 (Filesystem) in libvirt 2020-03-18 07:44:01,993 DEBUG [kvm.resource.LibvirtConnection] (Agent-Handler-1:null) (logid:) Looking for libvirtd connection at: qemu:///system HALTS HERE AND DOES NOT CONTINUE I waited 10 mins for a timeout then stopped the agent Thanks Adam -----Original Message----- From: Andrija Panic <andrija.pa...@gmail.com> Sent: 17 March 2020 18:52 To: users <users@cloudstack.apache.org> Subject: Re: Agent staying in alert mode ** This mail originated from OUTSIDE the Oakford corporate network. Treat hyperlinks and attachments in this email with caution. ** Can you change that to the IP of the mgmt server and then restart the agent? it should always point to the IP not hostname (what is that value there, "who put it there" etc - perhaps a valid question?) Best, On Tue, 17 Mar 2020 at 15:38, Adam Witwicki <awitwi...@oakfordis.com> wrote: > Hi Suresh > > /etc/cloudstack/agent/agent.properties > host=cs.cloudstack > > telnet > user@ois-mh-p1-c1-h2-b:~$ telnet cs.cloudstack 8250 Trying > 10.10.240.10... > Connected to cs.cloudstack. > Escape character is '^]'. > > > Same for all my other KVM nodes. 7 of 8 connected > > Thanks > > Adam > > -----Original Message----- > From: Suresh Kumar Anaparti <sureshkumar.anapa...@gmail.com> > Sent: 17 March 2020 13:11 > To: users@cloudstack.apache.org > Subject: Re: Agent staying in alert mode > > ** This mail originated from OUTSIDE the Oakford corporate network. > Treat hyperlinks and attachments in this email with caution. ** > > Hi Adam, > > What is the value of the host parameter in > "/etc/cloudstack/agent/agent.properties" file of the KVM host with > agent connectivity issue. Can you telnet to that management server on > port 8250 from the KVM host. > > Are other KVM agents connected to same management server? > > Thanks, > Suresh > > On Tue, Mar 17, 2020 at 3:05 PM Adam Witwicki > <awitwi...@oakfordis.com> > wrote: > > > Hi Suresh, > > > > Thanks for looking > > > > All KVM hosts can connect to management server - via ping > > > > The mgmt_server_id is Null for the alert host > > > > Host > > > > name;uuid;status;type;private_ip_address;private_netmask;private_mac > > _a > > ddress;storage_ip_address;storage_netmask;storage_mac_address;storag > > e_ > > ip_address_2;storage_mac_address_2;storage_netmask_2;cluster_id;publ > > ic > > _ip_address;public_netmask;public_mac_address;proxy_port;data_center > > _i > > d;pod_id;cpu_sockets;cpus;speed;url;fs_type;hypervisor_type;hypervis > > or > > _version;ram;resource;version;parent;total_size;capabilities;guid;av > > ai > > lable;setup;dom0_memory;last_ping;mgmt_server_id;disconnected;create > > d; > > removed;update_count;resource_state;owner;lastUpdated;engine_state > > ois-mh-p1-c1-h2-b;3d5e4c35-7e10-4560-8a2e-5ebf770e837d;Alert;Routing > > ;1 > > 0.10.240.126;255.255.255.0;a0:36:9f:24:32:6a;10.10.240.126;255.255.2 > > 55 > > .0;a0:36:9f:24:32:6a;\N;\N;\N;5;10.10.240.126;255.255.255.0;a0:36:9f > > :2 > > 4:32:6a;\N;5;5;2;32;3800;iqn.1993-08.org.debian:01:7c8e527ccba4;\N;K > > VM > > ;\N;269385596928;\N;4.11.0.0;\N;\N;hvm,snapshot;44190139-5b1b-3642-b > > 78 > > 6-41ca577081e9-LibvirtComputingResource;1;0;0;1546450279;\N;2019-12- > > 24 > > 09:25:53;2019-06-11 21:59:43;\N;2516;Enabled;\N;\N;Disabled > > > > mshost > > > > msid;runid;name;state;version;service_ip;service_port;last_update;re > > mo > > ved;alert_count > > 91765225987;1584406841937;ois-mh-csmg-01;Up;4.11.0.0;10.10.240.21;90 > > 90 > > ;2020-03-17 > > 09:33:07;\N;0 > > 91765663747;1584406829845;ois-mh-csmg-02;Up;4.11.0.0;10.10.240.22;90 > > 90 > > ;2020-03-17 > > 09:33:09;\N;0 > > 91766390275;1584406828383;ois-mh-csmg-03;Up;4.11.0.0;10.10.240.23;90 > > 90 > > ;2020-03-17 > > 09:33:05;\N;0 > > > > > > Adam > > > > -----Original Message----- > > From: Suresh Kumar Anaparti <sureshkumar.anapa...@gmail.com> > > Sent: 16 March 2020 19:35 > > To: users@cloudstack.apache.org > > Subject: Re: Agent staying in alert mode > > > > ** This mail originated from OUTSIDE the Oakford corporate network. > > Treat hyperlinks and attachments in this email with caution. ** > > > > Hi Adam, > > > > Seems it is clustered environment and other node(s) are being > > disconnected, from the logs. Can you check the management host of > > connected agents in the KVM host using net stat and the host value > > in "/etc/cloudstack/agent/agent.properties". > > > > Also, check the ownership of the host in mgmt_server_id column of > > the host table and validate it against the management server details > > in > mshost table. > > > > Thanks, > > Suresh > > > > On Mon, Mar 16, 2020 at 4:37 PM Adam Witwicki > > <awitwi...@oakfordis.com> > > wrote: > > > > > Hi Guys, > > > > > > I need some help, > > > > > > One of my KVM agents is stuck in alert state - no amount of > > > restarting the agent helps > > > > > > > > > > > > Cloudstack version 4.11.0.0 > > > > > > Logs > > > Management https://pastebin.com/JY2bvqmM Agent > > > https://pastebin.com/G7g8wXNN > > > > > > All the other agents connect fine > > > > > > Thanks > > > > > > Adam > > > > > > > > > > > > Disclaimer Notice: > > > This email has been sent by Oakford Technology Limited, while we > > > have checked this e-mail and any attachments for viruses, we can > > > not guarantee that they are virus-free. You must therefore take > > > full responsibility for virus checking. > > > This message and any attachments are confidential and should only > > > be read by those to whom they are addressed. If you are not the > > > intended recipient, please contact us, delete the message from > > > your computer and destroy any copies. Any distribution or copying > > > without our prior permission is prohibited. > > > Internet communications are not always secure and therefore > > > Oakford Technology Limited does not accept legal responsibility > > > for this > message. > > > The recipient is responsible for verifying its authenticity before > > > acting on the contents. Any views or opinions presented are solely > > > those of the author and do not necessarily represent those of > > > Oakford > > Technology Limited. > > > Registered address: Oakford Technology Limited, The Manor House, > > > Potterne, Wiltshire. SN10 5PN. > > > Registered in England and Wales No. 5971519 > > > > > > > > Disclaimer Notice: > > This email has been sent by Oakford Technology Limited, while we > > have checked this e-mail and any attachments for viruses, we can not > > guarantee that they are virus-free. You must therefore take full > > responsibility for virus checking. > > This message and any attachments are confidential and should only be > > read by those to whom they are addressed. If you are not the > > intended recipient, please contact us, delete the message from your > > computer and destroy any copies. Any distribution or copying without > > our prior permission is prohibited. > > Internet communications are not always secure and therefore Oakford > > Technology Limited does not accept legal responsibility for this message. > > The recipient is responsible for verifying its authenticity before > > acting on the contents. Any views or opinions presented are solely > > those of the author and do not necessarily represent those of > > Oakford > Technology Limited. > > Registered address: Oakford Technology Limited, The Manor House, > > Potterne, Wiltshire. SN10 5PN. > > Registered in England and Wales No. 5971519 > > > > > Disclaimer Notice: > This email has been sent by Oakford Technology Limited, while we have > checked this e-mail and any attachments for viruses, we can not > guarantee that they are virus-free. You must therefore take full > responsibility for virus checking. > This message and any attachments are confidential and should only be > read by those to whom they are addressed. If you are not the intended > recipient, please contact us, delete the message from your computer > and destroy any copies. Any distribution or copying without our prior > permission is prohibited. > Internet communications are not always secure and therefore Oakford > Technology Limited does not accept legal responsibility for this message. > The recipient is responsible for verifying its authenticity before > acting on the contents. Any views or opinions presented are solely > those of the author and do not necessarily represent those of Oakford > Technology Limited. > Registered address: Oakford Technology Limited, The Manor House, > Potterne, Wiltshire. SN10 5PN. > Registered in England and Wales No. 5971519 > > -- Andrija Panić Disclaimer Notice: This email has been sent by Oakford Technology Limited, while we have checked this e-mail and any attachments for viruses, we can not guarantee that they are virus-free. You must therefore take full responsibility for virus checking. This message and any attachments are confidential and should only be read by those to whom they are addressed. If you are not the intended recipient, please contact us, delete the message from your computer and destroy any copies. Any distribution or copying without our prior permission is prohibited. Internet communications are not always secure and therefore Oakford Technology Limited does not accept legal responsibility for this message. The recipient is responsible for verifying its authenticity before acting on the contents. Any views or opinions presented are solely those of the author and do not necessarily represent those of Oakford Technology Limited. Registered address: Oakford Technology Limited, The Manor House, Potterne, Wiltshire. SN10 5PN. Registered in England and Wales No. 5971519
