RE: Cluster HA - KVM Hosts
Hello, I changed timers for HA settings to minimize the number of attempts to recover the hosts and maximize the SLA for VMs. For now if I run the echo c > /proc/sysrq-trigger the host failed and the HA state show Degraded and after a while changed to recovered state, however the VMs is not accessible. How can I change settings to disable the number of attempts for recover or set it to 1 time attempt for recover and if host not recovered (OOBM ON and server loading failed) trigger HA event and restart VMs on another host (like VMware VCenter perform the HA event) Appreciate your support Regards, Jafar Aghabalayev -Original Message- From: Boris Stoyanov Sent: Saturday, February 11, 2023 2:37 PM To: users@cloudstack.apache.org; Wei ZHOU Subject: Re: Cluster HA - KVM Hosts CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Please report all suspicious emails to s...@pasha-technology.com. How did you power off the host? If you shut it down gracefully HA will not kick in, hence considered an intended action. The goal of this feature is to guard against unexpected events, like for example host has crashed. If you're looking to test it out you can try to provoke a kernel panic in the kvm host, something like: echo c > /proc/sysrq-trigger Then soon HA investigator will determin the host as unhealthy, (pings failing), soon after that there will be an activity checks on the VMs disks running on the host. It will check if there's any changes and disk activity, if that also fails it will determine the host for recovery. It will issue a restart with the IPMI tool, if number of restarts (depending on configuration) also fails it will fence the host and will start the VMs on another host in the same cluster. I hope this is insightful for you. Let me know how it goes at your end. Bobby. From: Jafar Aghabalayev Date: Saturday, 11 February 2023, 11:21 To: Wei ZHOU , users@cloudstack.apache.org Subject: Re: Cluster HA - KVM Hosts Hello, I created new offering, with HA enabled and used it Sent from Outlook for Android<https://aka.ms/AAb9ysg> From: Wei ZHOU Sent: Saturday, February 11, 2023 12:44:45 PM To: users@cloudstack.apache.org Cc: Jafar Aghabalayev Subject: Re: Cluster HA - KVM Hosts CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Please report all suspicious emails to s...@pasha-technology.com. Hi, Is HA enabled on the VM? You can check the vm details. HA is not enabled for built-in service offerings (Small/Medium). -Wei On Saturday, 11 February 2023, pspa...@hotmail.com<mailto:pspa...@hotmail.com> mailto:pspa...@hotmail.com>> wrote: Hi Jafar, FYI. VM HA does not work when server is stopped gracefully * Issue #4211 * apache/cloudstack | | | | || | | | | | VM HA does not work when server is stopped gracefully * Issue #4211 * apache/cloudstack When we stop a server (hypervisor) gracefully, vms on it will NOT be started on other servers even HA is enabled. ISSUE TYPE Bug Report COMPONENT NAME VM HA CLOUDSTACK VERSION 4.14/4.15 CONFIGURATI... | | | | Thanks Pradeep Sent from Yahoo Mail on Android On Sat, 11 Feb 2023 at 12:02 am, Jafar Aghabalayevmailto:jafar.aghabala...@pasha-technology.com>> wrote: Hello Community, I have configured cluster with 2 KVM Hosts. HA enabled for hosts, ha.tag attribute on global configuration is the same as for ha.tag attribute for hosts. VMs are running with HA enabled offering. NFS used as primary storage. I tried to shutdown one of the hosts, however HA not performed. State and power state indicated as down, but resource state show as up and VM located at the failed host show in running state (in real it is unaccessible). I tried to stop libvirtd service and the result same. Can anyone help me with this issue?
RE: Cluster HA - KVM Hosts
Hello, In case if i used local storage for system VMs (using local storage for system vms enabled) and for guest VMs shared NFS storage in use, can it cause problem for guest VMs HA trigger? Regards, Jafar Aghabalayev -Original Message- From: Jafar Aghabalayev Sent: Tuesday, February 21, 2023 4:58 PM To: users@cloudstack.apache.org Cc: Wei ZHOU ; boris.stoya...@shapeblue.com Subject: RE: Cluster HA - KVM Hosts Hello, Thank you for your response. One primary storage in use and it is NFS and VM volumes located on this NFS storage. After unplugged the power cable State show alert and power state unknown. Regards, Jafar Aghabalayev -Original Message- From: Vivek Kumar Sent: Tuesday, February 21, 2023 4:10 PM To: users@cloudstack.apache.org Cc: Wei ZHOU ; boris.stoya...@shapeblue.com Subject: Re: Cluster HA - KVM Hosts CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Please report all suspicious emails to s...@pasha-technology.com. Hello Jafar, This is the way to test the HA. I did same before putting into the production. When you unplugged the cable of physical hosts, what is the current state of host ? Is it showing “Disconnected”, “Alert” or “Down”, When I tested the HA it should have atleast one NFS as primary storage, it worked in my case with NFS as a primary storage or atleast one of the primary storage on NFS . So without any NFS storage, whenever I disconnected the host, it was either showing in “Disconnected” or “alert” and then HA will not work. Vivek Kumar Sr. Manager - Cloud & DevOps TechOps | Indiqus Technologies vivek.ku...@indiqus.com <mailto:vivek.ku...@indiqus.com> www.indiqus.com <https://www.indiqus.com/> > On 21-Feb-2023, at 5:29 PM, Jafar Aghabalayev > wrote: > > Hello Community, > My further steps are below: > > 1. I fixed the error related to Java and HA state display correct > state (Available for both hosts) 2. I shutdown the host by unplugging > power cable 3. The HA state show fencing state 4. No VMs migrated to > the second available host (all VMs used offering with HA enabled) > > Is there are any other options to test HA? > > > > > Regards, > Jafar Aghabalayev > > > -Original Message- > From: Jafar Aghabalayev > Sent: Monday, February 13, 2023 9:13 PM > To: users@cloudstack.apache.org; Wei ZHOU ; > boris.stoya...@shapeblue.com > Subject: RE: Cluster HA - KVM Hosts > > CAUTION: This email originated from outside the organization. Do not click > links or open attachments unless you recognize the sender and know the > content is safe. Please report all suspicious emails to > s...@pasha-technology.com. > > Hello, > I run the echo c > /proc/sysrq-trigger command and host went to > restart (is the HA provider initialize OOBM restart or not ?), however > after host up cloudstack agent not in running state > > Process: 4107 ExecStart=/usr/bin/java $JAVA_OPTS $JAVA_DEBUG -cp > $CLASSPATH $JAVA_CLASS (code=exited, status=0/SUCCESS) > > The HA state stucking in Suspect state for a while and then host went to > shutdown. Cloudstack show as Down state, resource state - Maintance and Power > state Off. However instances not rebooted on healty host within the same > cluster. Is any recommendation about that? > > Thank you for your time and efforts. > > > Regards, > > > > Jafar Aghabalayev | Senior IT Infrastructure Engineer > 157 Azadliq Avenue, AZ1106, Baku, Azerbaijan > mob: +994 55 900 19 34 > email: jafar.aghabala...@pasha-technology.com > > This communication contains information issued by "PASHA Technology" LLC. > This e-mail message and all attachments transmitted with it are intended > solely for the use of the addressee and may contain legally privileged and > confidential information. If the reader of this message is not the intended > recipient, or an employee or agent responsible for delivering this message to > the intended recipient, the reader is hereby notified that any dissemination, > distribution, copying, or other use of this message or its attachments is > strictly prohibited. If you have received this message in error, please > notify the sender immediately by replying to this message and please delete > it from your computer. Within the bounds of law "PASHA Technology" LLC may > monitor electronic transmissions through its internal and external networks > to ensure compliance with internal policies and for legitimate business > purposes. > > -Original Message- > From: Boris Stoyanov > Sent: Saturday, February 11, 2023 2:37 PM > To: users@cloudstack.apache.org; Wei ZHOU > Subject: Re:
RE: Cluster HA - KVM Hosts
Hello, Thank you for your response. One primary storage in use and it is NFS and VM volumes located on this NFS storage. After unplugged the power cable State show alert and power state unknown. Regards, Jafar Aghabalayev -Original Message- From: Vivek Kumar Sent: Tuesday, February 21, 2023 4:10 PM To: users@cloudstack.apache.org Cc: Wei ZHOU ; boris.stoya...@shapeblue.com Subject: Re: Cluster HA - KVM Hosts CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Please report all suspicious emails to s...@pasha-technology.com. Hello Jafar, This is the way to test the HA. I did same before putting into the production. When you unplugged the cable of physical hosts, what is the current state of host ? Is it showing “Disconnected”, “Alert” or “Down”, When I tested the HA it should have atleast one NFS as primary storage, it worked in my case with NFS as a primary storage or atleast one of the primary storage on NFS . So without any NFS storage, whenever I disconnected the host, it was either showing in “Disconnected” or “alert” and then HA will not work. Vivek Kumar Sr. Manager - Cloud & DevOps TechOps | Indiqus Technologies vivek.ku...@indiqus.com <mailto:vivek.ku...@indiqus.com> www.indiqus.com <https://www.indiqus.com/> > On 21-Feb-2023, at 5:29 PM, Jafar Aghabalayev > wrote: > > Hello Community, > My further steps are below: > > 1. I fixed the error related to Java and HA state display correct > state (Available for both hosts) 2. I shutdown the host by unplugging > power cable 3. The HA state show fencing state 4. No VMs migrated to > the second available host (all VMs used offering with HA enabled) > > Is there are any other options to test HA? > > > > > Regards, > Jafar Aghabalayev > > > -Original Message- > From: Jafar Aghabalayev > Sent: Monday, February 13, 2023 9:13 PM > To: users@cloudstack.apache.org; Wei ZHOU ; > boris.stoya...@shapeblue.com > Subject: RE: Cluster HA - KVM Hosts > > CAUTION: This email originated from outside the organization. Do not click > links or open attachments unless you recognize the sender and know the > content is safe. Please report all suspicious emails to > s...@pasha-technology.com. > > Hello, > I run the echo c > /proc/sysrq-trigger command and host went to > restart (is the HA provider initialize OOBM restart or not ?), however > after host up cloudstack agent not in running state > > Process: 4107 ExecStart=/usr/bin/java $JAVA_OPTS $JAVA_DEBUG -cp > $CLASSPATH $JAVA_CLASS (code=exited, status=0/SUCCESS) > > The HA state stucking in Suspect state for a while and then host went to > shutdown. Cloudstack show as Down state, resource state - Maintance and Power > state Off. However instances not rebooted on healty host within the same > cluster. Is any recommendation about that? > > Thank you for your time and efforts. > > > Regards, > > > > Jafar Aghabalayev | Senior IT Infrastructure Engineer > 157 Azadliq Avenue, AZ1106, Baku, Azerbaijan > mob: +994 55 900 19 34 > email: jafar.aghabala...@pasha-technology.com > > This communication contains information issued by "PASHA Technology" LLC. > This e-mail message and all attachments transmitted with it are intended > solely for the use of the addressee and may contain legally privileged and > confidential information. If the reader of this message is not the intended > recipient, or an employee or agent responsible for delivering this message to > the intended recipient, the reader is hereby notified that any dissemination, > distribution, copying, or other use of this message or its attachments is > strictly prohibited. If you have received this message in error, please > notify the sender immediately by replying to this message and please delete > it from your computer. Within the bounds of law "PASHA Technology" LLC may > monitor electronic transmissions through its internal and external networks > to ensure compliance with internal policies and for legitimate business > purposes. > > -Original Message- > From: Boris Stoyanov > Sent: Saturday, February 11, 2023 2:37 PM > To: users@cloudstack.apache.org; Wei ZHOU > Subject: Re: Cluster HA - KVM Hosts > > How did you power off the host? If you shut it down gracefully HA will not > kick in, hence considered an intended action. The goal of this feature is to > guard against unexpected events, like for example host has crashed. > > If you're looking to test it out you can try to provoke a kernel panic > in the kvm host, something like: echo c > /proc/sysrq-trigger > &g
Re: Cluster HA - KVM Hosts
Hello Jafar, This is the way to test the HA. I did same before putting into the production. When you unplugged the cable of physical hosts, what is the current state of host ? Is it showing “Disconnected”, “Alert” or “Down”, When I tested the HA it should have atleast one NFS as primary storage, it worked in my case with NFS as a primary storage or atleast one of the primary storage on NFS . So without any NFS storage, whenever I disconnected the host, it was either showing in “Disconnected” or “alert” and then HA will not work. Vivek Kumar Sr. Manager - Cloud & DevOps TechOps | Indiqus Technologies vivek.ku...@indiqus.com <mailto:vivek.ku...@indiqus.com> www.indiqus.com <https://www.indiqus.com/> > On 21-Feb-2023, at 5:29 PM, Jafar Aghabalayev > wrote: > > Hello Community, > My further steps are below: > > 1. I fixed the error related to Java and HA state display correct state > (Available for both hosts) > 2. I shutdown the host by unplugging power cable > 3. The HA state show fencing state > 4. No VMs migrated to the second available host (all VMs used offering with > HA enabled) > > Is there are any other options to test HA? > > > > > Regards, > Jafar Aghabalayev > > > -Original Message- > From: Jafar Aghabalayev > Sent: Monday, February 13, 2023 9:13 PM > To: users@cloudstack.apache.org; Wei ZHOU ; > boris.stoya...@shapeblue.com > Subject: RE: Cluster HA - KVM Hosts > > CAUTION: This email originated from outside the organization. Do not click > links or open attachments unless you recognize the sender and know the > content is safe. Please report all suspicious emails to > s...@pasha-technology.com. > > Hello, > I run the echo c > /proc/sysrq-trigger command and host went to restart (is > the HA provider initialize OOBM restart or not ?), however after host up > cloudstack agent not in running state > > Process: 4107 ExecStart=/usr/bin/java $JAVA_OPTS $JAVA_DEBUG -cp $CLASSPATH > $JAVA_CLASS (code=exited, status=0/SUCCESS) > > The HA state stucking in Suspect state for a while and then host went to > shutdown. Cloudstack show as Down state, resource state - Maintance and Power > state Off. However instances not rebooted on healty host within the same > cluster. Is any recommendation about that? > > Thank you for your time and efforts. > > > Regards, > > > > Jafar Aghabalayev | Senior IT Infrastructure Engineer > 157 Azadliq Avenue, AZ1106, Baku, Azerbaijan > mob: +994 55 900 19 34 > email: jafar.aghabala...@pasha-technology.com > > This communication contains information issued by "PASHA Technology" LLC. > This e-mail message and all attachments transmitted with it are intended > solely for the use of the addressee and may contain legally privileged and > confidential information. If the reader of this message is not the intended > recipient, or an employee or agent responsible for delivering this message to > the intended recipient, the reader is hereby notified that any dissemination, > distribution, copying, or other use of this message or its attachments is > strictly prohibited. If you have received this message in error, please > notify the sender immediately by replying to this message and please delete > it from your computer. Within the bounds of law "PASHA Technology" LLC may > monitor electronic transmissions through its internal and external networks > to ensure compliance with internal policies and for legitimate business > purposes. > > -Original Message- > From: Boris Stoyanov > Sent: Saturday, February 11, 2023 2:37 PM > To: users@cloudstack.apache.org; Wei ZHOU > Subject: Re: Cluster HA - KVM Hosts > > How did you power off the host? If you shut it down gracefully HA will not > kick in, hence considered an intended action. The goal of this feature is to > guard against unexpected events, like for example host has crashed. > > If you're looking to test it out you can try to provoke a kernel panic in the > kvm host, something like: echo c > /proc/sysrq-trigger > > Then soon HA investigator will determin the host as unhealthy, (pings > failing), soon after that there will be an activity checks on the VMs disks > running on the host. It will check if there's any changes and disk activity, > if that also fails it will determine the host for recovery. It will issue a > restart with the IPMI tool, if number of restarts (depending on > configuration) also fails it will fence the host and will start the VMs on > another host in the same cluster. > > I hope this is insightful for you. Let me know how it goes at your end. > > Bobby. > &g
RE: Cluster HA - KVM Hosts
Hello Community, My further steps are below: 1. I fixed the error related to Java and HA state display correct state (Available for both hosts) 2. I shutdown the host by unplugging power cable 3. The HA state show fencing state 4. No VMs migrated to the second available host (all VMs used offering with HA enabled) Is there are any other options to test HA? Regards, Jafar Aghabalayev -Original Message- From: Jafar Aghabalayev Sent: Monday, February 13, 2023 9:13 PM To: users@cloudstack.apache.org; Wei ZHOU ; boris.stoya...@shapeblue.com Subject: RE: Cluster HA - KVM Hosts CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Please report all suspicious emails to s...@pasha-technology.com. Hello, I run the echo c > /proc/sysrq-trigger command and host went to restart (is the HA provider initialize OOBM restart or not ?), however after host up cloudstack agent not in running state Process: 4107 ExecStart=/usr/bin/java $JAVA_OPTS $JAVA_DEBUG -cp $CLASSPATH $JAVA_CLASS (code=exited, status=0/SUCCESS) The HA state stucking in Suspect state for a while and then host went to shutdown. Cloudstack show as Down state, resource state - Maintance and Power state Off. However instances not rebooted on healty host within the same cluster. Is any recommendation about that? Thank you for your time and efforts. Regards, Jafar Aghabalayev | Senior IT Infrastructure Engineer 157 Azadliq Avenue, AZ1106, Baku, Azerbaijan mob: +994 55 900 19 34 email: jafar.aghabala...@pasha-technology.com This communication contains information issued by "PASHA Technology" LLC. This e-mail message and all attachments transmitted with it are intended solely for the use of the addressee and may contain legally privileged and confidential information. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, the reader is hereby notified that any dissemination, distribution, copying, or other use of this message or its attachments is strictly prohibited. If you have received this message in error, please notify the sender immediately by replying to this message and please delete it from your computer. Within the bounds of law "PASHA Technology" LLC may monitor electronic transmissions through its internal and external networks to ensure compliance with internal policies and for legitimate business purposes. -Original Message- From: Boris Stoyanov Sent: Saturday, February 11, 2023 2:37 PM To: users@cloudstack.apache.org; Wei ZHOU Subject: Re: Cluster HA - KVM Hosts How did you power off the host? If you shut it down gracefully HA will not kick in, hence considered an intended action. The goal of this feature is to guard against unexpected events, like for example host has crashed. If you're looking to test it out you can try to provoke a kernel panic in the kvm host, something like: echo c > /proc/sysrq-trigger Then soon HA investigator will determin the host as unhealthy, (pings failing), soon after that there will be an activity checks on the VMs disks running on the host. It will check if there's any changes and disk activity, if that also fails it will determine the host for recovery. It will issue a restart with the IPMI tool, if number of restarts (depending on configuration) also fails it will fence the host and will start the VMs on another host in the same cluster. I hope this is insightful for you. Let me know how it goes at your end. Bobby. From: Jafar Aghabalayev Date: Saturday, 11 February 2023, 11:21 To: Wei ZHOU , users@cloudstack.apache.org Subject: Re: Cluster HA - KVM Hosts Hello, I created new offering, with HA enabled and used it Sent from Outlook for Android<https://aka.ms/AAb9ysg> From: Wei ZHOU Sent: Saturday, February 11, 2023 12:44:45 PM To: users@cloudstack.apache.org Cc: Jafar Aghabalayev Subject: Re: Cluster HA - KVM Hosts CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Please report all suspicious emails to s...@pasha-technology.com. Hi, Is HA enabled on the VM? You can check the vm details. HA is not enabled for built-in service offerings (Small/Medium). -Wei On Saturday, 11 February 2023, pspa...@hotmail.com<mailto:pspa...@hotmail.com> mailto:pspa...@hotmail.com>> wrote: Hi Jafar, FYI. VM HA does not work when server is stopped gracefully * Issue #4211 * apache/cloudstack | | | | || | | | | | VM HA does not work when server is stopped gracefully * Issue #4211 * apache/cloudstack When we stop a server (hypervisor) gracefully, vms on it will NOT be started on other servers even HA is enabled. ISSUE TY
Re: Cluster HA - KVM Hosts
Hi Jafar, this issue has been fixed since 4.14 as far as I read it, so it should be in 4.17. As for the failures you have with agent not starting you should check the management and agent logs, you can trace what did restarted the KVM host (likely OOBM action) and why is the agent not starting. From: Jafar Aghabalayev Date: Monday, 13 February 2023, 19:14 To: pspa...@hotmail.com , users@cloudstack.apache.org Subject: RE: Cluster HA - KVM Hosts Hello, Do you have any information about fixing this bug on cloudstack ver 4.17 ? Thank you Regards, [cid:image001.png@01D93FF0.1A70EED0] Jafar Aghabalayev | Senior IT Infrastructure Engineer 157 Azadliq Avenue, AZ1106, Baku, Azerbaijan mob: +994 55 900 19 34 email: jafar.aghabala...@pasha-technology.com<mailto:jafar.aghabala...@pasha-technology.com> This communication contains information issued by "PASHA Technology" LLC. This e-mail message and all attachments transmitted with it are intended solely for the use of the addressee and may contain legally privileged and confidential information. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, the reader is hereby notified that any dissemination, distribution, copying, or other use of this message or its attachments is strictly prohibited. If you have received this message in error, please notify the sender immediately by replying to this message and please delete it from your computer. Within the bounds of law "PASHA Technology" LLC may monitor electronic transmissions through its internal and external networks to ensure compliance with internal policies and for legitimate business purposes. From: pspa...@hotmail.com Sent: Saturday, February 11, 2023 9:02 AM To: users@cloudstack.apache.org; Jafar Aghabalayev Subject: Re: Cluster HA - KVM Hosts CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Please report all suspicious emails to s...@pasha-technology.com<mailto:s...@pasha-technology.com>. Hi Jafar, FYI. VM HA does not work when server is stopped gracefully · Issue #4211 · apache/cloudstack<https://github.com/apache/cloudstack/issues/4211> VM HA does not work when server is stopped gracefully · Issue #4211 · apache/cloudstack When we stop a server (hypervisor) gracefully, vms on it will NOT be started on other servers even HA is enabled. ISSUE TYPE Bug Report COMPONENT NAME VM HA CLOUDSTACK VERSION 4.14/4.15 CONFIGURATI... Thanks Pradeep Sent from Yahoo Mail on Android<https://go.onelink.me/107872968?pid=InProduct&c=Global_Internal_YGrowth_AndroidEmailSig__AndroidUsers&af_wl=ym&af_sub1=Internal&af_sub2=Global_YGrowth&af_sub3=EmailSignature> On Sat, 11 Feb 2023 at 12:02 am, Jafar Aghabalayev mailto:jafar.aghabala...@pasha-technology.com>> wrote: Hello Community, I have configured cluster with 2 KVM Hosts. HA enabled for hosts, ha.tag attribute on global configuration is the same as for ha.tag attribute for hosts. VMs are running with HA enabled offering. NFS used as primary storage. I tried to shutdown one of the hosts, however HA not performed. State and power state indicated as down, but resource state show as up and VM located at the failed host show in running state (in real it is unaccessible). I tried to stop libvirtd service and the result same. Can anyone help me with this issue?
RE: Cluster HA - KVM Hosts
Hello, Do you have any information about fixing this bug on cloudstack ver 4.17 ? Thank you Regards, [cid:image001.png@01D93FF0.1A70EED0] Jafar Aghabalayev | Senior IT Infrastructure Engineer 157 Azadliq Avenue, AZ1106, Baku, Azerbaijan mob: +994 55 900 19 34 email: jafar.aghabala...@pasha-technology.com<mailto:jafar.aghabala...@pasha-technology.com> This communication contains information issued by "PASHA Technology" LLC. This e-mail message and all attachments transmitted with it are intended solely for the use of the addressee and may contain legally privileged and confidential information. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, the reader is hereby notified that any dissemination, distribution, copying, or other use of this message or its attachments is strictly prohibited. If you have received this message in error, please notify the sender immediately by replying to this message and please delete it from your computer. Within the bounds of law "PASHA Technology" LLC may monitor electronic transmissions through its internal and external networks to ensure compliance with internal policies and for legitimate business purposes. From: pspa...@hotmail.com Sent: Saturday, February 11, 2023 9:02 AM To: users@cloudstack.apache.org; Jafar Aghabalayev Subject: Re: Cluster HA - KVM Hosts CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Please report all suspicious emails to s...@pasha-technology.com<mailto:s...@pasha-technology.com>. Hi Jafar, FYI. VM HA does not work when server is stopped gracefully · Issue #4211 · apache/cloudstack<https://github.com/apache/cloudstack/issues/4211> VM HA does not work when server is stopped gracefully · Issue #4211 · apache/cloudstack When we stop a server (hypervisor) gracefully, vms on it will NOT be started on other servers even HA is enabled. ISSUE TYPE Bug Report COMPONENT NAME VM HA CLOUDSTACK VERSION 4.14/4.15 CONFIGURATI... Thanks Pradeep Sent from Yahoo Mail on Android<https://go.onelink.me/107872968?pid=InProduct&c=Global_Internal_YGrowth_AndroidEmailSig__AndroidUsers&af_wl=ym&af_sub1=Internal&af_sub2=Global_YGrowth&af_sub3=EmailSignature> On Sat, 11 Feb 2023 at 12:02 am, Jafar Aghabalayev mailto:jafar.aghabala...@pasha-technology.com>> wrote: Hello Community, I have configured cluster with 2 KVM Hosts. HA enabled for hosts, ha.tag attribute on global configuration is the same as for ha.tag attribute for hosts. VMs are running with HA enabled offering. NFS used as primary storage. I tried to shutdown one of the hosts, however HA not performed. State and power state indicated as down, but resource state show as up and VM located at the failed host show in running state (in real it is unaccessible). I tried to stop libvirtd service and the result same. Can anyone help me with this issue?
RE: Cluster HA - KVM Hosts
Hello, I run the echo c > /proc/sysrq-trigger command and host went to restart (is the HA provider initialize OOBM restart or not ?), however after host up cloudstack agent not in running state Process: 4107 ExecStart=/usr/bin/java $JAVA_OPTS $JAVA_DEBUG -cp $CLASSPATH $JAVA_CLASS (code=exited, status=0/SUCCESS) The HA state stucking in Suspect state for a while and then host went to shutdown. Cloudstack show as Down state, resource state - Maintance and Power state Off. However instances not rebooted on healty host within the same cluster. Is any recommendation about that? Thank you for your time and efforts. Regards, Jafar Aghabalayev | Senior IT Infrastructure Engineer 157 Azadliq Avenue, AZ1106, Baku, Azerbaijan mob: +994 55 900 19 34 email: jafar.aghabala...@pasha-technology.com This communication contains information issued by "PASHA Technology" LLC. This e-mail message and all attachments transmitted with it are intended solely for the use of the addressee and may contain legally privileged and confidential information. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, the reader is hereby notified that any dissemination, distribution, copying, or other use of this message or its attachments is strictly prohibited. If you have received this message in error, please notify the sender immediately by replying to this message and please delete it from your computer. Within the bounds of law "PASHA Technology" LLC may monitor electronic transmissions through its internal and external networks to ensure compliance with internal policies and for legitimate business purposes. -Original Message- From: Boris Stoyanov Sent: Saturday, February 11, 2023 2:37 PM To: users@cloudstack.apache.org; Wei ZHOU Subject: Re: Cluster HA - KVM Hosts How did you power off the host? If you shut it down gracefully HA will not kick in, hence considered an intended action. The goal of this feature is to guard against unexpected events, like for example host has crashed. If you're looking to test it out you can try to provoke a kernel panic in the kvm host, something like: echo c > /proc/sysrq-trigger Then soon HA investigator will determin the host as unhealthy, (pings failing), soon after that there will be an activity checks on the VMs disks running on the host. It will check if there's any changes and disk activity, if that also fails it will determine the host for recovery. It will issue a restart with the IPMI tool, if number of restarts (depending on configuration) also fails it will fence the host and will start the VMs on another host in the same cluster. I hope this is insightful for you. Let me know how it goes at your end. Bobby. From: Jafar Aghabalayev Date: Saturday, 11 February 2023, 11:21 To: Wei ZHOU , users@cloudstack.apache.org Subject: Re: Cluster HA - KVM Hosts Hello, I created new offering, with HA enabled and used it Sent from Outlook for Android<https://aka.ms/AAb9ysg> From: Wei ZHOU Sent: Saturday, February 11, 2023 12:44:45 PM To: users@cloudstack.apache.org Cc: Jafar Aghabalayev Subject: Re: Cluster HA - KVM Hosts CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Please report all suspicious emails to s...@pasha-technology.com. Hi, Is HA enabled on the VM? You can check the vm details. HA is not enabled for built-in service offerings (Small/Medium). -Wei On Saturday, 11 February 2023, pspa...@hotmail.com<mailto:pspa...@hotmail.com> mailto:pspa...@hotmail.com>> wrote: Hi Jafar, FYI. VM HA does not work when server is stopped gracefully · Issue #4211 · apache/cloudstack | | | | || | | | | | VM HA does not work when server is stopped gracefully · Issue #4211 · apache/cloudstack When we stop a server (hypervisor) gracefully, vms on it will NOT be started on other servers even HA is enabled. ISSUE TYPE Bug Report COMPONENT NAME VM HA CLOUDSTACK VERSION 4.14/4.15 CONFIGURATI... | | | | Thanks Pradeep Sent from Yahoo Mail on Android On Sat, 11 Feb 2023 at 12:02 am, Jafar Aghabalayevmailto:jafar.aghabala...@pasha-technology.com>> wrote: Hello Community, I have configured cluster with 2 KVM Hosts. HA enabled for hosts, ha.tag attribute on global configuration is the same as for ha.tag attribute for hosts. VMs are running with HA enabled offering. NFS used as primary storage. I tried to shutdown one of the hosts, however HA not performed. State and power state indicated as down, but resource state show as up and VM located at the failed host show in running state (in real it is unaccessible). I tried to stop libvirtd service and the result same. Can anyone help me with this issue?
Re: Cluster HA - KVM Hosts
How did you power off the host? If you shut it down gracefully HA will not kick in, hence considered an intended action. The goal of this feature is to guard against unexpected events, like for example host has crashed. If you’re looking to test it out you can try to provoke a kernel panic in the kvm host, something like: echo c > /proc/sysrq-trigger Then soon HA investigator will determin the host as unhealthy, (pings failing), soon after that there will be an activity checks on the VMs disks running on the host. It will check if there’s any changes and disk activity, if that also fails it will determine the host for recovery. It will issue a restart with the IPMI tool, if number of restarts (depending on configuration) also fails it will fence the host and will start the VMs on another host in the same cluster. I hope this is insightful for you. Let me know how it goes at your end. Bobby. From: Jafar Aghabalayev Date: Saturday, 11 February 2023, 11:21 To: Wei ZHOU , users@cloudstack.apache.org Subject: Re: Cluster HA - KVM Hosts Hello, I created new offering, with HA enabled and used it Sent from Outlook for Android<https://aka.ms/AAb9ysg> From: Wei ZHOU Sent: Saturday, February 11, 2023 12:44:45 PM To: users@cloudstack.apache.org Cc: Jafar Aghabalayev Subject: Re: Cluster HA - KVM Hosts CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Please report all suspicious emails to s...@pasha-technology.com. Hi, Is HA enabled on the VM? You can check the vm details. HA is not enabled for built-in service offerings (Small/Medium). -Wei On Saturday, 11 February 2023, pspa...@hotmail.com<mailto:pspa...@hotmail.com> mailto:pspa...@hotmail.com>> wrote: Hi Jafar, FYI. VM HA does not work when server is stopped gracefully · Issue #4211 · apache/cloudstack | | | | || | | | | | VM HA does not work when server is stopped gracefully · Issue #4211 · apache/cloudstack When we stop a server (hypervisor) gracefully, vms on it will NOT be started on other servers even HA is enabled. ISSUE TYPE Bug Report COMPONENT NAME VM HA CLOUDSTACK VERSION 4.14/4.15 CONFIGURATI... | | | | Thanks Pradeep Sent from Yahoo Mail on Android On Sat, 11 Feb 2023 at 12:02 am, Jafar Aghabalayevmailto:jafar.aghabala...@pasha-technology.com>> wrote: Hello Community, I have configured cluster with 2 KVM Hosts. HA enabled for hosts, ha.tag attribute on global configuration is the same as for ha.tag attribute for hosts. VMs are running with HA enabled offering. NFS used as primary storage. I tried to shutdown one of the hosts, however HA not performed. State and power state indicated as down, but resource state show as up and VM located at the failed host show in running state (in real it is unaccessible). I tried to stop libvirtd service and the result same. Can anyone help me with this issue?
Re: Cluster HA - KVM Hosts
Hello, I created new offering, with HA enabled and used it Sent from Outlook for Android<https://aka.ms/AAb9ysg> From: Wei ZHOU Sent: Saturday, February 11, 2023 12:44:45 PM To: users@cloudstack.apache.org Cc: Jafar Aghabalayev Subject: Re: Cluster HA - KVM Hosts CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Please report all suspicious emails to s...@pasha-technology.com. Hi, Is HA enabled on the VM? You can check the vm details. HA is not enabled for built-in service offerings (Small/Medium). -Wei On Saturday, 11 February 2023, pspa...@hotmail.com<mailto:pspa...@hotmail.com> mailto:pspa...@hotmail.com>> wrote: Hi Jafar, FYI. VM HA does not work when server is stopped gracefully · Issue #4211 · apache/cloudstack | | | | || | | | | | VM HA does not work when server is stopped gracefully · Issue #4211 · apache/cloudstack When we stop a server (hypervisor) gracefully, vms on it will NOT be started on other servers even HA is enabled. ISSUE TYPE Bug Report COMPONENT NAME VM HA CLOUDSTACK VERSION 4.14/4.15 CONFIGURATI... | | | | Thanks Pradeep Sent from Yahoo Mail on Android On Sat, 11 Feb 2023 at 12:02 am, Jafar Aghabalayevmailto:jafar.aghabala...@pasha-technology.com>> wrote: Hello Community, I have configured cluster with 2 KVM Hosts. HA enabled for hosts, ha.tag attribute on global configuration is the same as for ha.tag attribute for hosts. VMs are running with HA enabled offering. NFS used as primary storage. I tried to shutdown one of the hosts, however HA not performed. State and power state indicated as down, but resource state show as up and VM located at the failed host show in running state (in real it is unaccessible). I tried to stop libvirtd service and the result same. Can anyone help me with this issue?
Re: Cluster HA - KVM Hosts
Hi, Is HA enabled on the VM? You can check the vm details. HA is not enabled for built-in service offerings (Small/Medium). -Wei On Saturday, 11 February 2023, pspa...@hotmail.com wrote: > Hi Jafar, > FYI. > VM HA does not work when server is stopped gracefully · Issue #4211 · > apache/cloudstack > | > | > | > | || > >| > > | > | > | | > VM HA does not work when server is stopped gracefully · Issue #4211 · > apache/cloudstack > > When we stop a server (hypervisor) gracefully, vms on it will NOT be > started on other servers even HA is enabled. ISSUE TYPE Bug Report > COMPONENT NAME VM HA CLOUDSTACK VERSION 4.14/4.15 CONFIGURATI... > | | > > | > > | > > > > Thanks Pradeep > > Sent from Yahoo Mail on Android > > On Sat, 11 Feb 2023 at 12:02 am, Jafar Aghabalayev pasha-technology.com> wrote: Hello Community, > > I have configured cluster with 2 KVM Hosts. > HA enabled for hosts, ha.tag attribute on global configuration is the same > as for ha.tag attribute for hosts. VMs are running with HA enabled > offering. NFS used as primary storage. > I tried to shutdown one of the hosts, however HA not performed. State and > power state indicated as down, but resource state show as up and VM located > at the failed host show in running state (in real it is unaccessible). I > tried to stop libvirtd service and the result same. > > Can anyone help me with this issue? > > > >
Re: Cluster HA - KVM Hosts
Hello, OOBM also configured using IPMI and when i shut the host, power state show correct state and im able to control (power off, power on) host using OOBM on cloudstack Sent from Outlook for Android<https://aka.ms/AAb9ysg> From: Simon Weller Sent: Friday, February 10, 2023 11:07:15 PM To: users@cloudstack.apache.org Subject: Re: Cluster HA - KVM Hosts CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Please report all suspicious emails to s...@pasha-technology.com. Hey Jafar, For HA to work on KVM, you need to configure each host with out-of-band management using ipmitool or Redfish. This is so Cloudstack can fence your nodes. This covers the concept - https://docs.cloudstack.apache.org/en/latest/adminguide/reliability.html?highlight=fencing#ha-enabled-hosts Out-of-band documentation is here - https://docs.cloudstack.apache.org/en/latest/adminguide/hosts.html#out-of-band-management For details on how the feature works, check out the original design document - https://cwiki.apache.org/confluence/display/CLOUDSTACK/Host+HA -Si On Fri, Feb 10, 2023 at 12:32 PM Jafar Aghabalayev < jafar.aghabala...@pasha-technology.com> wrote: > Hello Community, > > I have configured cluster with 2 KVM Hosts. > HA enabled for hosts, ha.tag attribute on global configuration is the same > as for ha.tag attribute for hosts. VMs are running with HA enabled > offering. NFS used as primary storage. > I tried to shutdown one of the hosts, however HA not performed. State and > power state indicated as down, but resource state show as up and VM located > at the failed host show in running state (in real it is unaccessible). I > tried to stop libvirtd service and the result same. > > Can anyone help me with this issue? > > >
Re: Cluster HA - KVM Hosts
Hi Jafar, FYI. VM HA does not work when server is stopped gracefully · Issue #4211 · apache/cloudstack | | | | || | | | | | VM HA does not work when server is stopped gracefully · Issue #4211 · apache/cloudstack When we stop a server (hypervisor) gracefully, vms on it will NOT be started on other servers even HA is enabled. ISSUE TYPE Bug Report COMPONENT NAME VM HA CLOUDSTACK VERSION 4.14/4.15 CONFIGURATI... | | | | Thanks Pradeep Sent from Yahoo Mail on Android On Sat, 11 Feb 2023 at 12:02 am, Jafar Aghabalayev wrote: Hello Community, I have configured cluster with 2 KVM Hosts. HA enabled for hosts, ha.tag attribute on global configuration is the same as for ha.tag attribute for hosts. VMs are running with HA enabled offering. NFS used as primary storage. I tried to shutdown one of the hosts, however HA not performed. State and power state indicated as down, but resource state show as up and VM located at the failed host show in running state (in real it is unaccessible). I tried to stop libvirtd service and the result same. Can anyone help me with this issue?
Re: Cluster HA - KVM Hosts
Hey Jafar, For HA to work on KVM, you need to configure each host with out-of-band management using ipmitool or Redfish. This is so Cloudstack can fence your nodes. This covers the concept - https://docs.cloudstack.apache.org/en/latest/adminguide/reliability.html?highlight=fencing#ha-enabled-hosts Out-of-band documentation is here - https://docs.cloudstack.apache.org/en/latest/adminguide/hosts.html#out-of-band-management For details on how the feature works, check out the original design document - https://cwiki.apache.org/confluence/display/CLOUDSTACK/Host+HA -Si On Fri, Feb 10, 2023 at 12:32 PM Jafar Aghabalayev < jafar.aghabala...@pasha-technology.com> wrote: > Hello Community, > > I have configured cluster with 2 KVM Hosts. > HA enabled for hosts, ha.tag attribute on global configuration is the same > as for ha.tag attribute for hosts. VMs are running with HA enabled > offering. NFS used as primary storage. > I tried to shutdown one of the hosts, however HA not performed. State and > power state indicated as down, but resource state show as up and VM located > at the failed host show in running state (in real it is unaccessible). I > tried to stop libvirtd service and the result same. > > Can anyone help me with this issue? > > >
Cluster HA - KVM Hosts
Hello Community, I have configured cluster with 2 KVM Hosts. HA enabled for hosts, ha.tag attribute on global configuration is the same as for ha.tag attribute for hosts. VMs are running with HA enabled offering. NFS used as primary storage. I tried to shutdown one of the hosts, however HA not performed. State and power state indicated as down, but resource state show as up and VM located at the failed host show in running state (in real it is unaccessible). I tried to stop libvirtd service and the result same. Can anyone help me with this issue?
