VM status is still running because we cannot determine the status of VM on host which is in disconnected state. Host is in disconnected state may be due to network glitch but may be VM is still running on it.
Regards, Anshul On 16-Jul-2015, at 6:04 pm, Luciano Castro <luciano.cas...@gmail.com<mailto:luciano.cas...@gmail.com>> wrote: Hi! * Yes, Offer HA. * The status os KVM hosts reported is "Disconnected", and curiously the VM status is still "Running..." I did the same test again: 1) shutdown KVM host 12:13 UTC 2) Logs (management server) [root@1q2 ~]# grep -i -E 'warn|exception|unable|fail|error' /var/log/cloudstack/management/management-server.log|tail -50 2015-07-16 12:13:21,195 WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-14:ctx-b1ba1de6 ctx-5c5114f1) Received unknown parameters for command listNetworkOfferings. Unknown parameters : projectid 2015-07-16 12:13:31,395 WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-4:ctx-78cb8329 ctx-ea77f742) Received unknown parameters for command listNetworkOfferings. Unknown parameters : projectid 2015-07-16 12:13:40,975 WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-19:ctx-6e2ab11e ctx-f0bc3e8d) Received unknown parameters for command expungeVirtualMachine. Unknown parameters : projectid 2015-07-16 12:13:40,998 WARN [c.c.a.d.ParamGenericValidationWorker] (API-Job-Executor-54:ctx-a6765444 job-407 ctx-e7d4ae6e) Received unknown parameters for command expungeVirtualMachine. Unknown parameters : projectid 2015-07-16 12:13:44,162 WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-3:ctx-ca71fefa ctx-3a46d16f) Received unknown parameters for command queryAsyncJobResult. Unknown parameters : projectid 2015-07-16 12:13:47,163 WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-2:ctx-9773fe5c ctx-20f2c53d) Received unknown parameters for command listNetworkOfferings. Unknown parameters : projectid 2015-07-16 12:13:51,478 WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-4:ctx-986e5695 ctx-cfd9dc2a) Received unknown parameters for command listNetworkOfferings. Unknown parameters : projectid 2015-07-16 12:13:57,139 WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-21:ctx-3ed5162d ctx-0b378da3) Received unknown parameters for command listZones. Unknown parameters : projectid 2015-07-16 12:13:58,786 WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-19:ctx-ef9b46ae ctx-1ec7c374) Received unknown parameters for command listZones. Unknown parameters : projectid 2015-07-16 12:13:58,991 WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-3:ctx-c40727ff ctx-129cff5b) Received unknown parameters for command listHosts. Unknown parameters : projectid 2015-07-16 12:14:01,369 WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-9:ctx-d39d30d5 ctx-7d1296f3) Received unknown parameters for command listHosts. Unknown parameters : projectid 2015-07-16 12:14:01,553 WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-6:ctx-f02aae15 ctx-b675547d) Received unknown parameters for command listDedicatedHosts. Unknown parameters : projectid 2015-07-16 12:14:01,725 WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-25:ctx-4de9059a ctx-15ac62c3) Received unknown parameters for command listConfigurations. Unknown parameters : projectid 2015-07-16 12:14:01,908 WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-23:ctx-25fb3589 ctx-d4274e7c) Received unknown parameters for command listOsCategories. Unknown parameters : projectid 2015-07-16 12:14:02,081 WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-2:ctx-a90cf3ca ctx-6e7a01fc) Received unknown parameters for command listHosts. Unknown parameters : projectid 2015-07-16 12:14:02,266 WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-12:ctx-b8f987fa ctx-8a7f2922) Received unknown parameters for command listDedicatedHosts. Unknown parameters : projectid 2015-07-16 12:14:02,442 WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-7:ctx-64be7d4b ctx-36a9864d) Received unknown parameters for command listConfigurations. Unknown parameters : projectid 2015-07-16 12:14:02,650 WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-1:ctx-9b0043f1 ctx-36482b14) Received unknown parameters for command listOsCategories. Unknown parameters : projectid 2015-07-16 12:14:14,981 WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-16:ctx-5d62538b ctx-31cf1646) Received unknown parameters for command listZones. Unknown parameters : projectid 2015-07-16 12:14:17,445 WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-17:ctx-ef67f80f ctx-e953915f) Received unknown parameters for command listZones. Unknown parameters : projectid 2015-07-16 12:14:17,654 WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-11:ctx-7a3e6ce3 ctx-7d06a5a4) Received unknown parameters for command listHosts. Unknown parameters : projectid 2015-07-16 12:14:24,537 WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-4:ctx-596c9e4b ctx-90642717) Received unknown parameters for command findHostsForMigration. Unknown parameters : projectid 2015-07-16 12:14:28,684 WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-20:ctx-2b17bb98 ctx-a489da47) Received unknown parameters for command migrateVirtualMachine. Unknown parameters : projectid 2015-07-16 12:14:28,708 WARN [c.c.a.d.ParamGenericValidationWorker] (API-Job-Executor-55:ctx-0b76e8ae job-409 ctx-d346cca9) Received unknown parameters for command migrateVirtualMachine. Unknown parameters : projectid 2015-07-16 12:14:31,880 WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-10:ctx-b9b78739 ctx-bc42f26e) Received unknown parameters for command queryAsyncJobResult. Unknown parameters : projectid 2015-07-16 12:14:34,873 WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-22:ctx-22941163 ctx-737b322d) Received unknown parameters for command queryAsyncJobResult. Unknown parameters : projectid 2015-07-16 12:15:47,758 WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-25:ctx-bd9f0a98 ctx-97c7b112) Received unknown parameters for command listNetworkOfferings. Unknown parameters : projectid 2015-07-16 12:15:50,466 WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-16:ctx-c10f59b2 ctx-405d4cf3) Received unknown parameters for command listZones. Unknown parameters : projectid 2015-07-16 12:15:54,480 WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-17:ctx-9bb4f185 ctx-9f2b37a9) Received unknown parameters for command listZones. Unknown parameters : projectid 2015-07-16 12:15:54,684 WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-11:ctx-6a51a841 ctx-a0d37d54) Received unknown parameters for command listHosts. Unknown parameters : projectid 2015-07-16 12:15:58,044 WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-22:ctx-656c8c86 ctx-aef2da97) Received unknown parameters for command listNetworkOfferings. Unknown parameters : projectid 2015-07-16 12:16:37,381 DEBUG [c.c.c.CapacityManagerImpl] (API-Job-Executor-57:ctx-e1ed3c62 job-412 ctx-e861a024) STATS: Failed to alloc resource from host: 3 reservedCpu: 0, requested cpu: 4000, reservedMem: 0, requested mem: 2147483648 2015-07-16 12:21:18,239 WARN [c.c.a.m.AgentManagerImpl] (AgentMonitor-1:ctx-9eef326e) Disconnect agent for CPVM/SSVM due to physical connection close. host: 9 2015-07-16 12:21:18,243 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-3:ctx-eafd1ed4) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2015-07-16 12:21:18,243 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-eafd1ed4) Seq 9-8454382400481264659: Timed out on null 2015-07-16 12:21:18,243 DEBUG [o.a.c.s.RemoteHostEndPoint] (StatsCollector-3:ctx-eafd1ed4) Failed to send command, due to Agent:9, com.cloud.exception.OperationTimedoutException: Commands 8454382400481264659 to Host 9 timed out after 3600 2015-07-16 12:21:18,243 ERROR [c.c.s.StatsCollector] (StatsCollector-3:ctx-eafd1ed4) Error trying to retrieve storage stats com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:9, com.cloud.exception.OperationTimedoutException: Commands 8454382400481264659 to Host 9 timed out after 3600 2015-07-16 12:22:09,951 WARN [o.a.c.alerts] (HA-1:ctx-6b60e29b) alertType:: 13 // dataCenterId:: 0 // podId:: 0 // clusterId:: null // message:: No usage server process running [root@1q2 ~]# grep -i -E 'warn|exception|unable|fail|error' /var/log/cloudstack/management/catalina.out|tail -50 WARN [o.a.c.alerts] (HA-1:ctx-830dab68) alertType:: 13 // dataCenterId:: 0 // podId:: 0 // clusterId:: null // message:: No usage server process running WARN [o.a.c.alerts] (HA-1:ctx-e45484a1) alertType:: 13 // dataCenterId:: 0 // podId:: 0 // clusterId:: null // message:: No usage server process running WARN [o.a.c.alerts] (HA-1:ctx-c27a224e) alertType:: 13 // dataCenterId:: 0 // podId:: 0 // clusterId:: null // message:: No usage server process running WARN [o.a.c.alerts] (HA-1:ctx-259fdd28) alertType:: 13 // dataCenterId:: 0 // podId:: 0 // clusterId:: null // message:: No usage server process running WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-16:ctx-40e06aca ctx-d419dc55) Received unknown parameters for command listNetworkOfferings. Unknown parameters : projectid WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-22:ctx-4b3562fd ctx-3341445a) Received unknown parameters for command listZones. Unknown parameters : projectid WARN [o.a.c.alerts] (HA-1:ctx-f34fcc40) alertType:: 13 // dataCenterId:: 0 // podId:: 0 // clusterId:: null // message:: No usage server process running WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-19:ctx-8f8e296e ctx-a6f598c1) Received unknown parameters for command listZones. Unknown parameters : projectid WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-3:ctx-8825ea69 ctx-fb7312c0) Received unknown parameters for command listZones. Unknown parameters : projectid WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-9:ctx-d6a4922e ctx-b3bad36d) Received unknown parameters for command listHosts. Unknown parameters : projectid WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-17:ctx-f7e28168 ctx-e966ba7a) Received unknown parameters for command listServiceOfferings. Unknown parameters : listall WARN [o.a.c.alerts] (HA-1:ctx-42103a07) alertType:: 13 // dataCenterId:: 0 // podId:: 0 // clusterId:: null // message:: No usage server process running WARN [o.a.c.alerts] (HA-1:ctx-5cac1938) alertType:: 13 // dataCenterId:: 0 // podId:: 0 // clusterId:: null // message:: No usage server process running WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-14:ctx-b1ba1de6 ctx-5c5114f1) Received unknown parameters for command listNetworkOfferings. Unknown parameters : projectid WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-4:ctx-78cb8329 ctx-ea77f742) Received unknown parameters for command listNetworkOfferings. Unknown parameters : projectid WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-19:ctx-6e2ab11e ctx-f0bc3e8d) Received unknown parameters for command expungeVirtualMachine. Unknown parameters : projectid WARN [c.c.a.d.ParamGenericValidationWorker] (API-Job-Executor-54:ctx-a6765444 job-407 ctx-e7d4ae6e) Received unknown parameters for command expungeVirtualMachine. Unknown parameters : projectid WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-3:ctx-ca71fefa ctx-3a46d16f) Received unknown parameters for command queryAsyncJobResult. Unknown parameters : projectid WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-2:ctx-9773fe5c ctx-20f2c53d) Received unknown parameters for command listNetworkOfferings. Unknown parameters : projectid WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-4:ctx-986e5695 ctx-cfd9dc2a) Received unknown parameters for command listNetworkOfferings. Unknown parameters : projectid WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-21:ctx-3ed5162d ctx-0b378da3) Received unknown parameters for command listZones. Unknown parameters : projectid WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-19:ctx-ef9b46ae ctx-1ec7c374) Received unknown parameters for command listZones. Unknown parameters : projectid WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-3:ctx-c40727ff ctx-129cff5b) Received unknown parameters for command listHosts. Unknown parameters : projectid WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-9:ctx-d39d30d5 ctx-7d1296f3) Received unknown parameters for command listHosts. Unknown parameters : projectid WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-6:ctx-f02aae15 ctx-b675547d) Received unknown parameters for command listDedicatedHosts. Unknown parameters : projectid WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-25:ctx-4de9059a ctx-15ac62c3) Received unknown parameters for command listConfigurations. Unknown parameters : projectid WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-23:ctx-25fb3589 ctx-d4274e7c) Received unknown parameters for command listOsCategories. Unknown parameters : projectid WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-2:ctx-a90cf3ca ctx-6e7a01fc) Received unknown parameters for command listHosts. Unknown parameters : projectid WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-12:ctx-b8f987fa ctx-8a7f2922) Received unknown parameters for command listDedicatedHosts. Unknown parameters : projectid WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-7:ctx-64be7d4b ctx-36a9864d) Received unknown parameters for command listConfigurations. Unknown parameters : projectid WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-1:ctx-9b0043f1 ctx-36482b14) Received unknown parameters for command listOsCategories. Unknown parameters : projectid WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-16:ctx-5d62538b ctx-31cf1646) Received unknown parameters for command listZones. Unknown parameters : projectid WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-17:ctx-ef67f80f ctx-e953915f) Received unknown parameters for command listZones. Unknown parameters : projectid WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-11:ctx-7a3e6ce3 ctx-7d06a5a4) Received unknown parameters for command listHosts. Unknown parameters : projectid WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-4:ctx-596c9e4b ctx-90642717) Received unknown parameters for command findHostsForMigration. Unknown parameters : projectid WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-20:ctx-2b17bb98 ctx-a489da47) Received unknown parameters for command migrateVirtualMachine. Unknown parameters : projectid WARN [c.c.a.d.ParamGenericValidationWorker] (API-Job-Executor-55:ctx-0b76e8ae job-409 ctx-d346cca9) Received unknown parameters for command migrateVirtualMachine. Unknown parameters : projectid WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-10:ctx-b9b78739 ctx-bc42f26e) Received unknown parameters for command queryAsyncJobResult. Unknown parameters : projectid WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-22:ctx-22941163 ctx-737b322d) Received unknown parameters for command queryAsyncJobResult. Unknown parameters : projectid WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-25:ctx-bd9f0a98 ctx-97c7b112) Received unknown parameters for command listNetworkOfferings. Unknown parameters : projectid WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-16:ctx-c10f59b2 ctx-405d4cf3) Received unknown parameters for command listZones. Unknown parameters : projectid WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-17:ctx-9bb4f185 ctx-9f2b37a9) Received unknown parameters for command listZones. Unknown parameters : projectid WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-11:ctx-6a51a841 ctx-a0d37d54) Received unknown parameters for command listHosts. Unknown parameters : projectid WARN [c.c.a.d.ParamGenericValidationWorker] (catalina-exec-22:ctx-656c8c86 ctx-aef2da97) Received unknown parameters for command listNetworkOfferings. Unknown parameters : projectid WARN [c.c.a.m.AgentManagerImpl] (AgentMonitor-1:ctx-9eef326e) Disconnect agent for CPVM/SSVM due to physical connection close. host: 9 INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-3:ctx-eafd1ed4) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-eafd1ed4) Seq 9-8454382400481264659: Timed out on null ERROR [c.c.s.StatsCollector] (StatsCollector-3:ctx-eafd1ed4) Error trying to retrieve storage stats com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:9, com.cloud.exception.OperationTimedoutException: Commands 8454382400481264659 to Host 9 timed out after 3600 WARN [o.a.c.alerts] (HA-1:ctx-6b60e29b) alertType:: 13 // dataCenterId:: 0 // podId:: 0 // clusterId:: null // message:: No usage server process running PS: I was loggen in the vm with offer HA, and I started a ping. I lost the connection after 30 icmp_seq. Thanks! On Wed, Jul 15, 2015 at 7:11 PM, Somesh Naidu <somesh.na...@citrix.com<mailto:somesh.na...@citrix.com>> wrote: "TAG H.A checked" - did you mean "Offer HA"? Also, after you shutdown host A, what was the status of the host reported in Cloudstack? Also, did you check the management server logs to see what the investigators reported? Regards, Somesh -----Original Message----- From: Luciano Castro [mailto:luciano.cas...@gmail.com] Sent: Wednesday, July 15, 2015 2:46 PM To: users@cloudstack.apache.org<mailto:users@cloudstack.apache.org> Subject: Fwd: HA feature - KVM - CloudStack 4.5.1 Hi there, I´m new in the list, and I´m using 4.5.1 CloudStack version, and I started making some tests with H.A feature for VMs, but I didn´t have success. I don´t know if it´s a bug, or something that I´m doing wrong. Scenario: I have 1 manager server and 2 KVM hosts ('A' and 'B'). 1) I created a "service offering" (computing offering) with the TAG H.A checked. 2) then, I created a VM with this offering, and the VM is in KVM 'A'. 3) I did a shutdown in the KVM host 'A', and I expected that this VM, after some minutes will be available in another KVM host ('B') But it didn´t happened. Any ideias?? thanks! -- Luciano Castro -- Luciano Castro -- Luciano Castro
