Re: Failed to setup certificates for system vm error on 2nd Zone after upgrading to 4.17.2

[Antoine Boucher]

Is there any reason why the /usr/local/cloud/systemvm directory would be 
missing from the SSVM one one zone but not on an other?

Is the directory added during the systemvm creation or is it part of the base 
system VM template?

Both /etc/cloudstack-release show: Cloudstack Release 4.17.2 Fri 09 Dec 2022 
12:51:18 PM UTC

Regards,
Antoine



> On Feb 21, 2023, at 3:31 PM, Antoine Boucher  wrote:
> 
> I compared the system template fro KVM of both zone and they seem identical.  
> 
> 
> The /usr/local/cloud/systemvm/ directory exist on SSVM on zone 1 exist but 
> the /usr/local/cloud/ directory of the zone 2 SSVM does not exist?
> 
> 
> 
> Antoine Boucher
> antoi...@haltondc.com
> [o] +1-226-505-9734
> www.haltondc.com
> 
> “Data security made simple”
> 
> 
> 
> 
> 
> Confidentiality Warning: This message and any attachments are intended only 
> for the use of the intended recipient(s), are confidential, and may be 
> privileged. If you are not the intended recipient, you are hereby notified 
> that any review, retransmission, conversion to hard copy, copying, 
> circulation or other use of this message and any attachments is strictly 
> prohibited. If you are not the intended recipient, please notify the sender 
> immediately by return e-mail, and delete this message and any attachments 
> from your system.
> 
> 
>> On Feb 21, 2023, at 1:31 PM, Antoine Boucher  wrote:
>> 
>> 4.17.2
>> 
>> Antoine Boucher
>> antoi...@haltondc.com
>> [o] +1-226-505-9734
>> www.haltondc.com
>> 
>> “Data security made simple and affordable”
>> 
>> 
>> On Feb 21, 2023, at 11:51, Wei ZHOU  wrote:
>> 
>> can you check the /etc/cloudstack-release in ssvm ?
>> 
>> -Wei
>> 
>> On Tuesday, 21 February 2023, Antoine Boucher > > wrote:
>>> Here are the logs from daemon.log of the failing SSVM
>>> 
>>> Feb 21 02:36:16 systemvm systemd[1963]: cloud.service: Failed to locate 
>>> executable /usr/local/cloud/systemvm/_run.sh: No such file or directory
>>> Feb 21 02:36:16 systemvm systemd[1963]: cloud.service: Failed at step EXEC 
>>> spawning /usr/local/cloud/systemvm/_run.sh: No such file or directory
>>> Feb 21 02:36:16 systemvm systemd[1]: cloud.service: Main process exited, 
>>> code=exited, status=203/EXEC
>>> Feb 21 02:36:16 systemvm systemd[1]: cloud.service: Failed with result 
>>> 'exit-code'.
>>> Feb 21 02:36:16 systemvm systemd[1]: systemd-update-utmp-runlevel.service: 
>>> Succeeded.
>>> Feb 21 02:36:16 systemvm systemd[1]: Finished Update UTMP about System 
>>> Runlevel Changes.
>>> Feb 21 02:36:16 systemvm systemd[1]: Startup finished in 2.285s (kernel) + 
>>> 2min 16.468s (userspace) = 2min 18.754s.
>>> Feb 21 02:36:16 systemvm systemd[1]: cloud.service: Scheduled restart job, 
>>> restart counter is at 1.
>>> Feb 21 02:36:16 systemvm systemd[1]: Stopped CloudStack Agent service.
>>> 
>>> This repeats 5 times and fails
>>> 
>>> The /usr/local/cloud/systemvm/ directory does not exist on the SSVM.  
>>> 
>>> 
>>> 
>>> 
>>> Antoine Boucher
>>> antoi...@haltondc.com 
>>> [o] +1-226-505-9734
>>> www.haltondc.com 
>>> 
>>> “Data security made simple”
>>> 
>>> 
>>> 
>>> 
>>> 
>>> Confidentiality Warning: This message and any attachments are intended only 
>>> for the use of the intended recipient(s), are confidential, and may be 
>>> privileged. If you are not the intended recipient, you are hereby notified 
>>> that any review, retransmission, conversion to hard copy, copying, 
>>> circulation or other use of this message and any attachments is strictly 
>>> prohibited. If you are not the intended recipient, please notify the sender 
>>> immediately by return e-mail, and delete this message and any attachments 
>>> from your system.
>>> 
>>> 
 On Feb 21, 2023, at 7:53 AM, Antoine Boucher >>> > wrote:
 
 They are not old they are the new ones.  I have deleted them a number of 
 times after the upgrade but the issue prevails.  
 
 Antoine Boucher
 antoi...@haltondc.com 
 [o] +1-226-505-9734
 www.haltondc.com 
 
 “Data security made simple and affordable”
 
 
 On Feb 21, 2023, at 00:24, Rohit Yadav >>> > wrote:
 
 
 You can destroy these old systemvms or do a stop and start on them. 
 
 Regards.
 
   
 
   
 From: Antoine Boucher >>> >
 Sent: Tuesday, February 21, 2023 10:40:46 AM
 To: users >>> >
 Subject: Failed to setup certificates for system vm error on 2nd Zone 
 after upgrading to 4.17.2
  
 After upgrading my two zone ACS from 4.16.2 to 4.17.2, the system VM (and 
 VRs) of the first zone upgraded without issues but the system VMs 
 (s-318-VM and v-317-VM) of the second zone (Kitchener1) are no longer able 
 to establish conne

Re: Failed to setup certificates for system vm error on 2nd Zone after upgrading to 4.17.2

[Wei ZHOU]

The scripts are transferred from kvm host via scp command. It would be good
to check
- kvm host has cloudstack-agent installed and upgraded
- scp to ssvm/router vms via cloud0 is allowed on kvm host

-Wei



On Wednesday, 22 February 2023, Antoine Boucher 
wrote:

> Is there any reason why the /usr/local/cloud/systemvm directory would be
> missing from the SSVM one one zone but not on an other?
>
> Is the directory added during the systemvm creation or is it part of the
> base system VM template?
>
> Both /etc/cloudstack-release show: Cloudstack Release 4.17.2 Fri 09 Dec
> 2022 12:51:18 PM UTC
>
> Regards,
> Antoine
>
>
>
> > On Feb 21, 2023, at 3:31 PM, Antoine Boucher 
> wrote:
> >
> > I compared the system template fro KVM of both zone and they seem
> identical.
> >
> >
> > The /usr/local/cloud/systemvm/ directory exist on SSVM on zone 1 exist
> but the /usr/local/cloud/ directory of the zone 2 SSVM does not exist?
> >
> >
> >
> > Antoine Boucher
> > antoi...@haltondc.com
> > [o] +1-226-505-9734
> > www.haltondc.com
> >
> > “Data security made simple”
> >
> >
> > 
> >
> >
> > Confidentiality Warning: This message and any attachments are intended
> only for the use of the intended recipient(s), are confidential, and may be
> privileged. If you are not the intended recipient, you are hereby notified
> that any review, retransmission, conversion to hard copy, copying,
> circulation or other use of this message and any attachments is strictly
> prohibited. If you are not the intended recipient, please notify the sender
> immediately by return e-mail, and delete this message and any attachments
> from your system.
> >
> >
> >> On Feb 21, 2023, at 1:31 PM, Antoine Boucher 
> wrote:
> >>
> >> 4.17.2
> >>
> >> Antoine Boucher
> >> antoi...@haltondc.com
> >> [o] +1-226-505-9734
> >> www.haltondc.com
> >>
> >> “Data security made simple and affordable”
> >>
> >>
> >> On Feb 21, 2023, at 11:51, Wei ZHOU  wrote:
> >>
> >> can you check the /etc/cloudstack-release in ssvm ?
> >>
> >> -Wei
> >>
> >> On Tuesday, 21 February 2023, Antoine Boucher  > wrote:
> >>> Here are the logs from daemon.log of the failing SSVM
> >>>
> >>> Feb 21 02:36:16 systemvm systemd[1963]: cloud.service: Failed to
> locate executable /usr/local/cloud/systemvm/_run.sh: No such file or
> directory
> >>> Feb 21 02:36:16 systemvm systemd[1963]: cloud.service: Failed at step
> EXEC spawning /usr/local/cloud/systemvm/_run.sh: No such file or directory
> >>> Feb 21 02:36:16 systemvm systemd[1]: cloud.service: Main process
> exited, code=exited, status=203/EXEC
> >>> Feb 21 02:36:16 systemvm systemd[1]: cloud.service: Failed with result
> 'exit-code'.
> >>> Feb 21 02:36:16 systemvm systemd[1]: systemd-update-utmp-runlevel.service:
> Succeeded.
> >>> Feb 21 02:36:16 systemvm systemd[1]: Finished Update UTMP about System
> Runlevel Changes.
> >>> Feb 21 02:36:16 systemvm systemd[1]: Startup finished in 2.285s
> (kernel) + 2min 16.468s (userspace) = 2min 18.754s.
> >>> Feb 21 02:36:16 systemvm systemd[1]: cloud.service: Scheduled restart
> job, restart counter is at 1.
> >>> Feb 21 02:36:16 systemvm systemd[1]: Stopped CloudStack Agent service.
> >>>
> >>> This repeats 5 times and fails
> >>>
> >>> The /usr/local/cloud/systemvm/ directory does not exist on the SSVM.
> >>>
> >>>
> >>>
> >>>
> >>> Antoine Boucher
> >>> antoi...@haltondc.com 
> >>> [o] +1-226-505-9734
> >>> www.haltondc.com 
> >>>
> >>> “Data security made simple”
> >>>
> >>>
> >>> 
> >>>
> >>>
> >>> Confidentiality Warning: This message and any attachments are intended
> only for the use of the intended recipient(s), are confidential, and may be
> privileged. If you are not the intended recipient, you are hereby notified
> that any review, retransmission, conversion to hard copy, copying,
> circulation or other use of this message and any attachments is strictly
> prohibited. If you are not the intended recipient, please notify the sender
> immediately by return e-mail, and delete this message and any attachments
> from your system.
> >>>
> >>>
>  On Feb 21, 2023, at 7:53 AM, Antoine Boucher  > wrote:
> 
>  They are not old they are the new ones.  I have deleted them a number
> of times after the upgrade but the issue prevails.
> 
>  Antoine Boucher
>  antoi...@haltondc.com 
>  [o] +1-226-505-9734
>  www.haltondc.com 
> 
>  “Data security made simple and affordable”
> 
> 
>  On Feb 21, 2023, at 00:24, Rohit Yadav  > wrote:
> 
>  
>  You can destroy these old systemvms or do a stop and start on them.
> 
>  Regards.
> 
> 
> 
> 
>  From: Antoine Boucher  antoi...@haltondc.com>>
>  Sent: Tuesday, February 21, 2023 10:40:46 AM
>  To: users mailto:users@cloudstack.
> apache.org>>
>  Subje

Re: Stuck in Preparing for maintenance on primary storage

[Simon Weller]

Jeremy,

Any chance you have a write permission problem on your new NFS server?
Those errors indicate an underlying storage issue.

-Si

On Tue, Feb 21, 2023, 11:46 PM Jeremy Hansen 
wrote:

> Oh and the system vm’s continue to stay in Starting state.
>
> -jeremy
>
>
>
> On Tuesday, Feb 21, 2023 at 9:44 PM, Me  wrote:
> The vm’s finally stopped and restarted.  This is what I’m seeing in dmesg
> on the secondary storage vm:
>
> root@s-60-VM:~# dmesg  | grep -i error
> [3.861852] blk_update_request: I/O error, dev vda, sector 6787872 op
> 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
> [3.865833] blk_update_request: I/O error, dev vda, sector 6787872 op
> 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
> [3.869553] systemd[1]: Failed to read configured hostname:
> Input/output error
> [4.560419] EXT4-fs (vda6): re-mounted. Opts: errors=remount-ro
> [4.646460] blk_update_request: I/O error, dev vda, sector 6787160 op
> 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
> [4.650710] blk_update_request: I/O error, dev vda, sector 6787160 op
> 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
> [4.975915] blk_update_request: I/O error, dev vda, sector 6787856 op
> 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
> [4.980318] blk_update_request: I/O error, dev vda, sector 6787856 op
> 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
> [5.018828] blk_update_request: I/O error, dev vda, sector 6787136 op
> 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
> [5.022976] blk_update_request: I/O error, dev vda, sector 6787136 op
> 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
> [5.026750] blk_update_request: I/O error, dev vda, sector 6787136 op
> 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
> [5.460315] blk_update_request: I/O error, dev vda, sector 6787856 op
> 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
> [   10.415215] print_req_error: 16 callbacks suppressed
> [   10.415219] blk_update_request: I/O error, dev vda, sector 6787864 op
> 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
> [   13.362595] blk_update_request: I/O error, dev vda, sector 6787136 op
> 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
> [   13.388990] blk_update_request: I/O error, dev vda, sector 6787136 op
> 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
> [   13.787276] blk_update_request: I/O error, dev vda, sector 6399408 op
> 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
> [   13.791575] blk_update_request: I/O error, dev vda, sector 6399408 op
> 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
> [   14.632299] blk_update_request: I/O error, dev vda, sector 6787136 op
> 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
> [   14.658283] blk_update_request: I/O error, dev vda, sector 6787136 op
> 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
>
> -jeremy
>
>
>
> On Tuesday, Feb 21, 2023 at 8:57 PM, Me  wrote:
> The node cloudstack is claiming the system vm’s is starting on shows no
> signs of any vm’s running.  virsh list is black.
>
> Thanks
> -jeremy
>
>
>
> On Tuesday, Feb 21, 2023 at 8:23 PM, Me  wrote:
> Also, just to note, I’m not sure how much made it in to the logs.  The
> system vm’s are stuck in starting state and trying to kill through the
> interface doesn’t seem to do anything.
>
> -jeremy
>
>
>
>
> On Tuesday, Feb 21, 2023 at 8:20 PM, Me  wrote:
> Is there something else I can use to submit logs?  Too much for pastebin.
>
> Thanks
> -jeremy
>
>
>
> On Tuesday, Feb 21, 2023 at 7:07 PM, Simon Weller 
> wrote:
> Can you pull some management server logs and also put the CloudStack KVM
> agent into debug mode before destroying the ssvm and share the logs?
>
>
> https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=30147350#content/view/30147350
>
> On Tue, Feb 21, 2023, 8:33 PM Jeremy Hansen 
> wrote:
>
> Yes. It’s just a different partition on the same nfs server.
>
>
>
> On Tuesday, Feb 21, 2023 at 6:02 PM, Simon Weller 
> wrote:
> The new and old primary storage is in the same zone, correct?
> Did you also change out the secondary storage?
>
> On Tue, Feb 21, 2023, 7:59 PM Jeremy Hansen 
> wrote:
>
> Yes. On Kvm. I’ve been trying to destroy them from the interface and it
> just keep churning. I did a destroy with virsh, but no status changed in
> the interface. Also, the newly created ones don’t seem to bring up their
> agent and never fully start.
>
> Thanks
>
>
>
> On Tuesday, Feb 21, 2023 at 4:37 PM, Simon Weller 
> wrote:
> Just destroy the old system VMs and they will be recreated on available
> storage.
>
> Are you on KVM?
>
>
>
> On Tue, Feb 21, 2023, 6:14 PM Jeremy Hansen 
> wrote:
>
> How do I completely recreate the system vm?
>
> I was able to get the old storage in to full maintenance and deleted it,
> so maybe the system vm are still using the old storage? Is there a way to
> tell the system vm’s to use the new storage? Db change?
>
> Thanks!
>
>
>
> On Tuesday, Feb 21, 2023 at 1:36 PM, Simon Weller 
> wrote:
> Hey Jeremy,
>
> Is there anything in the management logs that indicate w

Apache CloudStack at CloudFest

[Ivet Petrova]

Hi all,

I want to promote a bit our community participation at CloudFest.
One idea I got is to prepare images like the one below, which our community 
members, which will be at the event can share on social media.

So anybody who wants to participate in the online promotion and share such 
image on social?

[cid:7406BC40-07A2-4713-8CE1-3A20E6767935]

Kind regards,


 

RE: Cluster HA - KVM Hosts

[Jafar Aghabalayev]

Hello,

In case if i used local storage for system VMs (using local storage for system 
vms enabled) and for guest VMs shared NFS storage in use, can it cause problem 
for guest VMs HA trigger?

Regards,



Jafar Aghabalayev 

-Original Message-
From: Jafar Aghabalayev 
Sent: Tuesday, February 21, 2023 4:58 PM
To: users@cloudstack.apache.org
Cc: Wei ZHOU ; boris.stoya...@shapeblue.com
Subject: RE: Cluster HA - KVM Hosts


Hello,

Thank you for your response.

One primary storage in use and it is NFS and VM volumes located on this NFS 
storage.

After unplugged the power cable State show alert and power state unknown.


Regards,
Jafar Aghabalayev 

-Original Message-
From: Vivek Kumar 
Sent: Tuesday, February 21, 2023 4:10 PM
To: users@cloudstack.apache.org
Cc: Wei ZHOU ; boris.stoya...@shapeblue.com
Subject: Re: Cluster HA - KVM Hosts

CAUTION: This email originated from outside the organization. Do not click 
links or open attachments unless you recognize the sender and know the content 
is safe. Please report all suspicious emails to s...@pasha-technology.com.

Hello Jafar,

This is the way to test the HA. I did same before putting into the production.

When you unplugged the cable of physical hosts, what is the current state of 
host ? Is it showing  “Disconnected”, “Alert” or “Down”, When I tested the HA 
it should have atleast  one NFS as primary storage,  it worked in my case with 
NFS as a primary storage or atleast one of the primary storage on NFS .  So 
without any NFS storage, whenever I disconnected the host, it was either 
showing in “Disconnected” or “alert” and then HA will not work.



Vivek Kumar
Sr. Manager - Cloud & DevOps
TechOps | Indiqus Technologies

vivek.ku...@indiqus.com 
www.indiqus.com 




> On 21-Feb-2023, at 5:29 PM, Jafar Aghabalayev 
>  wrote:
>
> Hello Community,
> My further steps are below:
>
> 1. I fixed the error related to Java and HA state display correct 
> state (Available for both hosts) 2. I shutdown the host by unplugging 
> power cable 3. The HA state show fencing state 4. No VMs migrated to 
> the second available host (all VMs used offering with HA enabled)
>
> Is there are any other options to test HA?
>
>
>
>
> Regards,
> Jafar Aghabalayev
>
>
> -Original Message-
> From: Jafar Aghabalayev 
> Sent: Monday, February 13, 2023 9:13 PM
> To: users@cloudstack.apache.org; Wei ZHOU ; 
> boris.stoya...@shapeblue.com
> Subject: RE: Cluster HA - KVM Hosts
>
> CAUTION: This email originated from outside the organization. Do not click 
> links or open attachments unless you recognize the sender and know the 
> content is safe. Please report all suspicious emails to 
> s...@pasha-technology.com.
>
> Hello,
> I run the echo c > /proc/sysrq-trigger command and host went to 
> restart (is the HA provider initialize OOBM restart or not ?), however 
> after host up cloudstack agent not in running state
>
>  Process: 4107 ExecStart=/usr/bin/java $JAVA_OPTS $JAVA_DEBUG -cp 
> $CLASSPATH $JAVA_CLASS (code=exited, status=0/SUCCESS)
>
> The HA state stucking in Suspect state for a while and then host went to 
> shutdown. Cloudstack show as Down state, resource state - Maintance and Power 
> state Off. However instances not rebooted on healty host within the same 
> cluster. Is any recommendation about that?
>
> Thank you for your time and efforts.
>
>
> Regards,
>
>
>
> Jafar Aghabalayev | Senior IT Infrastructure Engineer
> 157 Azadliq Avenue, AZ1106, Baku, Azerbaijan
> mob: +994 55 900 19 34
> email: jafar.aghabala...@pasha-technology.com
>
> This communication contains information issued by "PASHA Technology" LLC. 
> This e-mail message and all attachments transmitted with it are intended 
> solely for the use of the addressee and may contain legally privileged and 
> confidential information. If the reader of this message is not the intended 
> recipient, or an employee or agent responsible for delivering this message to 
> the intended recipient, the reader is hereby notified that any dissemination, 
> distribution, copying, or other use of this message or its attachments is 
> strictly prohibited. If you have received this message in error, please 
> notify the sender immediately by replying to this message and please delete 
> it from your computer. Within the bounds of law "PASHA Technology" LLC may 
> monitor electronic transmissions through its internal and external networks 
> to ensure compliance with internal policies and for legitimate business 
> purposes.
>
> -Original Message-
> From: Boris Stoyanov 
> Sent: Saturday, February 11, 2023 2:37 PM
> To: users@cloudstack.apache.org; Wei ZHOU 
> Subject: Re: Cluster HA - KVM Hosts
>
> How did you power off the host? If you shut it down gracefully HA will not 
> kick in, hence considered an intended action. The goal of this feature is to 
> guard against unexpected events, like for example host has crashed.
>
> If you're lo

Re: Failed to setup certificates for system vm error on 2nd Zone after upgrading to 4.17.2

[Antoine Boucher]

Hi Wei,

Thank you for your information.  It turns out that the host was not properly 
updated to 4.17.2

All work now.

Thanks again!


Regards,
Antoine


> On Feb 22, 2023, at 7:51 AM, Wei ZHOU  wrote:
> 
> The scripts are transferred from kvm host via scp command. It would be good
> to check
> - kvm host has cloudstack-agent installed and upgraded
> - scp to ssvm/router vms via cloud0 is allowed on kvm host
> 
> -Wei
> 
> 
> 
> On Wednesday, 22 February 2023, Antoine Boucher  >
> wrote:
> 
>> Is there any reason why the /usr/local/cloud/systemvm directory would be
>> missing from the SSVM one one zone but not on an other?
>> 
>> Is the directory added during the systemvm creation or is it part of the
>> base system VM template?
>> 
>> Both /etc/cloudstack-release show: Cloudstack Release 4.17.2 Fri 09 Dec
>> 2022 12:51:18 PM UTC
>> 
>> Regards,
>> Antoine
>> 
>> 
>> 
>>> On Feb 21, 2023, at 3:31 PM, Antoine Boucher 
>> wrote:
>>> 
>>> I compared the system template fro KVM of both zone and they seem
>> identical.
>>> 
>>> 
>>> The /usr/local/cloud/systemvm/ directory exist on SSVM on zone 1 exist
>> but the /usr/local/cloud/ directory of the zone 2 SSVM does not exist?
>>> 
>>> 
>>> 
>>> Antoine Boucher
>>> antoi...@haltondc.com
>>> [o] +1-226-505-9734
>>> www.haltondc.com
>>> 
>>> “Data security made simple”
>>> 
>>> 
>>> 
>>> 
>>> 
>>> Confidentiality Warning: This message and any attachments are intended
>> only for the use of the intended recipient(s), are confidential, and may be
>> privileged. If you are not the intended recipient, you are hereby notified
>> that any review, retransmission, conversion to hard copy, copying,
>> circulation or other use of this message and any attachments is strictly
>> prohibited. If you are not the intended recipient, please notify the sender
>> immediately by return e-mail, and delete this message and any attachments
>> from your system.
>>> 
>>> 
 On Feb 21, 2023, at 1:31 PM, Antoine Boucher 
>> wrote:
 
 4.17.2
 
 Antoine Boucher
 antoi...@haltondc.com
 [o] +1-226-505-9734
 www.haltondc.com
 
 “Data security made simple and affordable”
 
 
 On Feb 21, 2023, at 11:51, Wei ZHOU  wrote:
 
 can you check the /etc/cloudstack-release in ssvm ?
 
 -Wei
 
 On Tuesday, 21 February 2023, Antoine Boucher > > wrote:
> Here are the logs from daemon.log of the failing SSVM
> 
> Feb 21 02:36:16 systemvm systemd[1963]: cloud.service: Failed to
>> locate executable /usr/local/cloud/systemvm/_run.sh: No such file or
>> directory
> Feb 21 02:36:16 systemvm systemd[1963]: cloud.service: Failed at step
>> EXEC spawning /usr/local/cloud/systemvm/_run.sh: No such file or directory
> Feb 21 02:36:16 systemvm systemd[1]: cloud.service: Main process
>> exited, code=exited, status=203/EXEC
> Feb 21 02:36:16 systemvm systemd[1]: cloud.service: Failed with result
>> 'exit-code'.
> Feb 21 02:36:16 systemvm systemd[1]: systemd-update-utmp-runlevel.service:
>> Succeeded.
> Feb 21 02:36:16 systemvm systemd[1]: Finished Update UTMP about System
>> Runlevel Changes.
> Feb 21 02:36:16 systemvm systemd[1]: Startup finished in 2.285s
>> (kernel) + 2min 16.468s (userspace) = 2min 18.754s.
> Feb 21 02:36:16 systemvm systemd[1]: cloud.service: Scheduled restart
>> job, restart counter is at 1.
> Feb 21 02:36:16 systemvm systemd[1]: Stopped CloudStack Agent service.
> 
> This repeats 5 times and fails
> 
> The /usr/local/cloud/systemvm/ directory does not exist on the SSVM.
> 
> 
> 
> 
> Antoine Boucher
> antoi...@haltondc.com  
> 
> [o] +1-226-505-9734
> www.haltondc.com  
> 
> “Data security made simple”
> 
> 
> 
> 
> 
> Confidentiality Warning: This message and any attachments are intended
>> only for the use of the intended recipient(s), are confidential, and may be
>> privileged. If you are not the intended recipient, you are hereby notified
>> that any review, retransmission, conversion to hard copy, copying,
>> circulation or other use of this message and any attachments is strictly
>> prohibited. If you are not the intended recipient, please notify the sender
>> immediately by return e-mail, and delete this message and any attachments
>> from your system.
> 
> 
>> On Feb 21, 2023, at 7:53 AM, Antoine Boucher > 
>> > wrote:
>> 
>> They are not old they are the new ones.  I have deleted them a number
>> of times after the upgrade but the issue prevails.
>> 
>> Antoine Boucher
>> antoi...@haltondc.com  
>> 
>> [o] +1-226-505-9734
>> www.haltondc.com 

RE: Cluster HA - KVM Hosts

[Jafar Aghabalayev]

Hello,

I changed timers for HA settings to minimize the number of attempts to recover 
the hosts and maximize the SLA for VMs.

For now if I run the echo c > /proc/sysrq-trigger the host failed and the HA 
state show Degraded and after a while changed to recovered state, however the 
VMs is not accessible.

How can I change settings to disable the number of attempts for recover or set 
it to 1 time attempt for recover and if host not recovered (OOBM ON and server 
loading failed) trigger HA event and restart VMs on another host (like VMware 
VCenter perform the HA event)


Appreciate your support 

Regards,

Jafar Aghabalayev 

-Original Message-
From: Boris Stoyanov  
Sent: Saturday, February 11, 2023 2:37 PM
To: users@cloudstack.apache.org; Wei ZHOU 
Subject: Re: Cluster HA - KVM Hosts

CAUTION: This email originated from outside the organization. Do not click 
links or open attachments unless you recognize the sender and know the content 
is safe. Please report all suspicious emails to s...@pasha-technology.com.

How did you power off the host? If you shut it down gracefully HA will not kick 
in, hence considered an intended action. The goal of this feature is to guard 
against unexpected events, like for example host has crashed.

If you're looking to test it out you can try to provoke a kernel panic in the 
kvm host, something like: echo c > /proc/sysrq-trigger

Then soon HA investigator will determin the host as unhealthy, (pings failing), 
soon after that there will be an activity checks on the VMs disks running on 
the host. It will check if there's any changes and disk activity, if that also 
fails it will determine the host for recovery. It will issue a restart with the 
IPMI tool, if number of restarts (depending on configuration) also fails it 
will fence the host and will start the VMs on another host in the same cluster.

I hope this is insightful for you. Let me know how it goes at your end.

Bobby.

From: Jafar Aghabalayev 
Date: Saturday, 11 February 2023, 11:21
To: Wei ZHOU , users@cloudstack.apache.org 

Subject: Re: Cluster HA - KVM Hosts
Hello,
I created new offering, with HA enabled and used it

Sent from Outlook for Android 

From: Wei ZHOU 
Sent: Saturday, February 11, 2023 12:44:45 PM
To: users@cloudstack.apache.org 
Cc: Jafar Aghabalayev 
Subject: Re: Cluster HA - KVM Hosts


CAUTION: This email originated from outside the organization. Do not click 
links or open attachments unless you recognize the sender and know the content 
is safe. Please report all suspicious emails to s...@pasha-technology.com.

Hi,

Is HA enabled on the VM?

You can check the vm details. HA is not enabled for built-in service offerings 
(Small/Medium).

-Wei





On Saturday, 11 February 2023, pspa...@hotmail.com 
mailto:pspa...@hotmail.com>> wrote:
Hi Jafar,
FYI.
VM HA does not work when server is stopped gracefully * Issue #4211 * 
apache/cloudstack
|
|
|
|   ||

   |

  |
|
|   |
VM HA does not work when server is stopped gracefully * Issue #4211 * 
apache/cloudstack

When we stop a server (hypervisor) gracefully, vms on it will NOT be started on 
other servers even HA is enabled. ISSUE TYPE Bug Report COMPONENT NAME VM HA 
CLOUDSTACK VERSION 4.14/4.15 CONFIGURATI...
  |   |

  |

  |



Thanks Pradeep

Sent from Yahoo Mail on Android

  On Sat, 11 Feb 2023 at 12:02 am, Jafar 
Aghabalayevmailto:jafar.aghabala...@pasha-technology.com>>
 wrote:   Hello Community,

I have configured cluster with 2 KVM Hosts.
HA enabled for hosts, ha.tag attribute on global configuration is the same as 
for ha.tag attribute for hosts. VMs are running with HA enabled offering. NFS 
used as primary storage.
I tried to shutdown one of the hosts, however HA not performed. State and power 
state indicated as down, but resource state show as up and VM located at the 
failed host show in running state (in real it is unaccessible). I tried to stop 
libvirtd service and the result same.

Can anyone help me with this issue?

Re: Stuck in Preparing for maintenance on primary storage

[Jeremy Hansen]

No issue with writes:

192.168.210.23:/exports/cloudstorage/primary 49T 57G 47T 1% 
/mnt/11cd19d0-f207-3d01-880f-8d01d4b15020
tmpfs 6.3G 0 6.3G 0% /run/user/0
192.168.210.23:/exports/cloudstorage/secondary 49T 57G 47T 1% 
/var/cloudstack/mnt/161333239336.2b9f6261

[root@droid 11cd19d0-f207-3d01-880f-8d01d4b15020]# touch 
/var/cloudstack/mnt/161333239336.2b9f6261/file
[root@droid 11cd19d0-f207-3d01-880f-8d01d4b15020]# ls -lad 
/var/cloudstack/mnt/161333239336.2b9f6261/file
-rw-r--r-- 1 root root 0 Feb 22 17:30 
/var/cloudstack/mnt/161333239336.2b9f6261/file

[root@droid ~]# touch /mnt/11cd19d0-f207-3d01-880f-8d01d4b15020/file
[root@droid ~]# ls -ald /mnt/11cd19d0-f207-3d01-880f-8d01d4b15020/file
-rw-r--r-- 1 root root 0 Feb 22 17:31 
/mnt/11cd19d0-f207-3d01-880f-8d01d4b15020/file

-jeremy

> On Wednesday, Feb 22, 2023 at 5:07 AM, Simon Weller  (mailto:siwelle...@gmail.com)> wrote:
> Jeremy,
>
> Any chance you have a write permission problem on your new NFS server?
> Those errors indicate an underlying storage issue.
>
> -Si
>
> On Tue, Feb 21, 2023, 11:46 PM Jeremy Hansen 
> wrote:
>
> > Oh and the system vm’s continue to stay in Starting state.
> >
> > -jeremy
> >
> >
> >
> > On Tuesday, Feb 21, 2023 at 9:44 PM, Me  wrote:
> > The vm’s finally stopped and restarted. This is what I’m seeing in dmesg
> > on the secondary storage vm:
> >
> > root@s-60-VM:~# dmesg | grep -i error
> > [ 3.861852] blk_update_request: I/O error, dev vda, sector 6787872 op
> > 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
> > [ 3.865833] blk_update_request: I/O error, dev vda, sector 6787872 op
> > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
> > [ 3.869553] systemd[1]: Failed to read configured hostname:
> > Input/output error
> > [ 4.560419] EXT4-fs (vda6): re-mounted. Opts: errors=remount-ro
> > [ 4.646460] blk_update_request: I/O error, dev vda, sector 6787160 op
> > 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
> > [ 4.650710] blk_update_request: I/O error, dev vda, sector 6787160 op
> > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
> > [ 4.975915] blk_update_request: I/O error, dev vda, sector 6787856 op
> > 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
> > [ 4.980318] blk_update_request: I/O error, dev vda, sector 6787856 op
> > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
> > [ 5.018828] blk_update_request: I/O error, dev vda, sector 6787136 op
> > 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
> > [ 5.022976] blk_update_request: I/O error, dev vda, sector 6787136 op
> > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
> > [ 5.026750] blk_update_request: I/O error, dev vda, sector 6787136 op
> > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
> > [ 5.460315] blk_update_request: I/O error, dev vda, sector 6787856 op
> > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
> > [ 10.415215] print_req_error: 16 callbacks suppressed
> > [ 10.415219] blk_update_request: I/O error, dev vda, sector 6787864 op
> > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
> > [ 13.362595] blk_update_request: I/O error, dev vda, sector 6787136 op
> > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
> > [ 13.388990] blk_update_request: I/O error, dev vda, sector 6787136 op
> > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
> > [ 13.787276] blk_update_request: I/O error, dev vda, sector 6399408 op
> > 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
> > [ 13.791575] blk_update_request: I/O error, dev vda, sector 6399408 op
> > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
> > [ 14.632299] blk_update_request: I/O error, dev vda, sector 6787136 op
> > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
> > [ 14.658283] blk_update_request: I/O error, dev vda, sector 6787136 op
> > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
> >
> > -jeremy
> >
> >
> >
> > On Tuesday, Feb 21, 2023 at 8:57 PM, Me  wrote:
> > The node cloudstack is claiming the system vm’s is starting on shows no
> > signs of any vm’s running. virsh list is black.
> >
> > Thanks
> > -jeremy
> >
> >
> >
> > On Tuesday, Feb 21, 2023 at 8:23 PM, Me  wrote:
> > Also, just to note, I’m not sure how much made it in to the logs. The
> > system vm’s are stuck in starting state and trying to kill through the
> > interface doesn’t seem to do anything.
> >
> > -jeremy
> >
> >
> >
> >
> > On Tuesday, Feb 21, 2023 at 8:20 PM, Me  wrote:
> > Is there something else I can use to submit logs? Too much for pastebin.
> >
> > Thanks
> > -jeremy
> >
> >
> >
> > On Tuesday, Feb 21, 2023 at 7:07 PM, Simon Weller 
> > wrote:
> > Can you pull some management server logs and also put the CloudStack KVM
> > agent into debug mode before destroying the ssvm and share the logs?
> >
> >
> > https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=30147350#content/view/30147350
> >
> > On Tue, Feb 21, 2023, 8:33 PM Jeremy Hansen 
> > wrote:
> >
> > Yes. It’s just a different partition on the same nfs server.
> >
> >
> >
> > On Tuesday, Feb 21, 2023 at 6:02 PM, Simon Weller 
> > wrote:
> > The new and old primary storage is 

Re: Stuck in Preparing for maintenance on primary storage

[Simon Weller]

The VM filesystem is putting itself in read-only mode, so something odd
appears to be going on.

On Wed, Feb 22, 2023, 7:07 AM Simon Weller  wrote:

> Jeremy,
>
> Any chance you have a write permission problem on your new NFS server?
> Those errors indicate an underlying storage issue.
>
> -Si
>
> On Tue, Feb 21, 2023, 11:46 PM Jeremy Hansen 
> wrote:
>
>> Oh and the system vm’s continue to stay in Starting state.
>>
>> -jeremy
>>
>>
>>
>> On Tuesday, Feb 21, 2023 at 9:44 PM, Me  wrote:
>> The vm’s finally stopped and restarted.  This is what I’m seeing in dmesg
>> on the secondary storage vm:
>>
>> root@s-60-VM:~# dmesg  | grep -i error
>> [3.861852] blk_update_request: I/O error, dev vda, sector 6787872 op
>> 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
>> [3.865833] blk_update_request: I/O error, dev vda, sector 6787872 op
>> 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
>> [3.869553] systemd[1]: Failed to read configured hostname:
>> Input/output error
>> [4.560419] EXT4-fs (vda6): re-mounted. Opts: errors=remount-ro
>> [4.646460] blk_update_request: I/O error, dev vda, sector 6787160 op
>> 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
>> [4.650710] blk_update_request: I/O error, dev vda, sector 6787160 op
>> 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
>> [4.975915] blk_update_request: I/O error, dev vda, sector 6787856 op
>> 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
>> [4.980318] blk_update_request: I/O error, dev vda, sector 6787856 op
>> 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
>> [5.018828] blk_update_request: I/O error, dev vda, sector 6787136 op
>> 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
>> [5.022976] blk_update_request: I/O error, dev vda, sector 6787136 op
>> 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
>> [5.026750] blk_update_request: I/O error, dev vda, sector 6787136 op
>> 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
>> [5.460315] blk_update_request: I/O error, dev vda, sector 6787856 op
>> 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
>> [   10.415215] print_req_error: 16 callbacks suppressed
>> [   10.415219] blk_update_request: I/O error, dev vda, sector 6787864 op
>> 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
>> [   13.362595] blk_update_request: I/O error, dev vda, sector 6787136 op
>> 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
>> [   13.388990] blk_update_request: I/O error, dev vda, sector 6787136 op
>> 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
>> [   13.787276] blk_update_request: I/O error, dev vda, sector 6399408 op
>> 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
>> [   13.791575] blk_update_request: I/O error, dev vda, sector 6399408 op
>> 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
>> [   14.632299] blk_update_request: I/O error, dev vda, sector 6787136 op
>> 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
>> [   14.658283] blk_update_request: I/O error, dev vda, sector 6787136 op
>> 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
>>
>> -jeremy
>>
>>
>>
>> On Tuesday, Feb 21, 2023 at 8:57 PM, Me  wrote:
>> The node cloudstack is claiming the system vm’s is starting on shows no
>> signs of any vm’s running.  virsh list is black.
>>
>> Thanks
>> -jeremy
>>
>>
>>
>> On Tuesday, Feb 21, 2023 at 8:23 PM, Me  wrote:
>> Also, just to note, I’m not sure how much made it in to the logs.  The
>> system vm’s are stuck in starting state and trying to kill through the
>> interface doesn’t seem to do anything.
>>
>> -jeremy
>>
>>
>>
>>
>> On Tuesday, Feb 21, 2023 at 8:20 PM, Me  wrote:
>> Is there something else I can use to submit logs?  Too much for pastebin.
>>
>> Thanks
>> -jeremy
>>
>>
>>
>> On Tuesday, Feb 21, 2023 at 7:07 PM, Simon Weller 
>> wrote:
>> Can you pull some management server logs and also put the CloudStack KVM
>> agent into debug mode before destroying the ssvm and share the logs?
>>
>>
>> https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=30147350#content/view/30147350
>>
>> On Tue, Feb 21, 2023, 8:33 PM Jeremy Hansen 
>> wrote:
>>
>> Yes. It’s just a different partition on the same nfs server.
>>
>>
>>
>> On Tuesday, Feb 21, 2023 at 6:02 PM, Simon Weller 
>> wrote:
>> The new and old primary storage is in the same zone, correct?
>> Did you also change out the secondary storage?
>>
>> On Tue, Feb 21, 2023, 7:59 PM Jeremy Hansen 
>> wrote:
>>
>> Yes. On Kvm. I’ve been trying to destroy them from the interface and it
>> just keep churning. I did a destroy with virsh, but no status changed in
>> the interface. Also, the newly created ones don’t seem to bring up their
>> agent and never fully start.
>>
>> Thanks
>>
>>
>>
>> On Tuesday, Feb 21, 2023 at 4:37 PM, Simon Weller 
>> wrote:
>> Just destroy the old system VMs and they will be recreated on available
>> storage.
>>
>> Are you on KVM?
>>
>>
>>
>> On Tue, Feb 21, 2023, 6:14 PM Jeremy Hansen 
>> wrote:
>>
>> How do I completely recreate the system vm?
>>
>> I was able to get the old storage in to full maintenance and deleted