Well - I found a "workaround", but it really can't be explained. I reverted to a snapshot THREE times and retraced my steps just to validate what I'm seeing.
Short summary - When the zone is enabled and the systemvms are created a folder is created at /var/cloudstack/mnt/VM/<uid folder>. This folder has the nobody:nobody attribute and the error Unable to copy systemvm ISO on secondary storage. Next, I removed the Secondary Storage entry, restarted services, re-added Secondary Storage, restarted services, AND systemvms deploy. Attributes of UID Folder at initial systemvm build: drwxr-xr-x. 3 cloud cloud 34 Jun 18 16:27 . drwxrwx---. 3 root cloud 16 Jun 18 16:27 .. drwxrwxrwt. 6 nobody nobody 4096 Jun 18 15:51 345050034639.51841a7 *** ACS tries to set perms on this folder but fails *** 2021-06-18 16:27:25,862 DEBUG [c.c.h.v.m.VmwareManagerImpl] (DirectAgent-3:ctx-a90e1ecb usphlmvesxt01.phl.global.corp.sap, job-9/job-51, cmd: CopyCommand) (logid:3d148cdf) chmod: changing permissions of ‘/var/cloudstack/mnt/VM/345050034639.51841a7’: Operation not permitted 2021-06-18 16:27:25,865 WARN [c.c.h.v.m.VmwareManagerImpl] (DirectAgent-3:ctx-a90e1ecb usphlmvesxt01.phl.global.corp.sap, job-9/job-51, cmd: CopyCommand) (logid:3d148cdf) Unable to set permissions for /var/cloudstack/mnt/VM/345050034639.51841a7 due to chmod: changing permissions of ‘/var/cloudstack/mnt/VM/345050034639.51841a7’: Operation not permitted ***When you remove the SS in UI and restart the services, ACS will clear this folder*** 2021-06-18 16:31:07,962 INFO [c.c.h.v.m.VmwareManagerImpl] (SpringContextShutdownHook:null) (logid:) shutting down scheduled tasks 2021-06-18 16:31:07,962 INFO [c.c.h.v.m.VmwareManagerImpl] (SpringContextShutdownHook:null) (logid:) Cleanup mounted NFS mount points used in current session 2021-06-18 16:31:07,962 INFO [c.c.h.v.m.VmwareManagerImpl] (SpringContextShutdownHook:null) (logid:) umount NFS mount: /var/cloudstack/mnt/VM/345050034639.51841a7 2021-06-18 16:31:07,962 DEBUG [c.c.h.v.m.VmwareManagerImpl] (SpringContextShutdownHook:null) (logid:) Executing: sudo umount /var/cloudstack/mnt/VM/345050034639.51841a7 ***Re-adding the SS in the UI and restarting services ACS CAN set the perms for the new folder *** 2021-06-18 16:38:56,254 DEBUG [c.c.h.v.m.VmwareManagerImpl] (DirectAgent-15:ctx-f9c800cf usphlmvesxt02.phl.global.corp.sap, job-9/job-58, cmd: StartCommand) (logid:3d148cdf) Executing: sudo chmod 1777 /var/cloudstack/mnt/VM/345050034639.50b47f15 2021-06-18 16:38:56,269 DEBUG [c.c.h.v.m.VmwareManagerImpl] (DirectAgent-15:ctx-f9c800cf usphlmvesxt02.phl.global.corp.sap, job-9/job-58, cmd: StartCommand) (logid:3d148cdf) Executing while with timeout : 1440000 2021-06-18 16:38:56,293 DEBUG [c.c.h.v.m.VmwareManagerImpl] (DirectAgent-15:ctx-f9c800cf usphlmvesxt02.phl.global.corp.sap, job-9/job-58, cmd: StartCommand) (logid:3d148cdf) Execution is successful. Attributes after SS removed, services restarted, SS added, services restarted: drwxr-xr-x. 3 cloud cloud 35 Jun 18 16:38 . drwxrwx---. 3 root cloud 16 Jun 18 16:27 .. drwxrwxrwt. 6 root root 4096 Jun 18 15:51 345050034639.50b47f15 So you can see that root:root is applied AFTER the original SS is removed and readded into the zone. I don't why the original attempt by ACS to build this directory fails. Look forward to your thoughts! Have a great weekend! Mike -----Original Message----- From: Corey, Mike <mike.co...@sap.com.INVALID> Sent: Friday, June 18, 2021 11:47 AM To: us...@cloudstack.apache.org; dev@cloudstack.apache.org Subject: [CAUTION] RE: Issues Found Apache CloudStack 4.15.1.0 (RC2) Honestly, I can live with having to go in after the wizard completes and modify the traffic labels. What I'm hitting regarding permissions on the Secondary Storage is more of a priority. The systemVM builds are just looping because they can't copy/mount the systemvm.iso to the proper directory. What I did in previous builds, both 4.14 and 4.15.0, was to manually create the systemvm folder and set ownership (chown -R) to cloud:cloud. This allowed for the copying of template and systemvm.iso files to work. However, now with 4.15.1, I'm getting permissions error when copying the systemvm.iso to the /var/cloudstack/mnt/VM/<UID</systemvm/.iso. 021-06-18 11:30:32,062 ERROR [c.c.h.v.m.VmwareManagerImpl] (DirectAgent-2:ctx-8e03b79a <managementserver>, job-9/job-47, cmd: StartCommand) (logid:3d148cdf) Unexpected exception java.nio.file.AccessDeniedException: /var/cloudstack/mnt/VM/345050034639.1562462a/systemvm/systemvm-4.15.1.0.iso 2021-06-18 11:30:32,064 ERROR [c.c.h.v.m.VmwareManagerImpl] (DirectAgent-2:ctx-8e03b79a <managementserver>, job-9/job-47, cmd: StartCommand) (logid:3d148cdf) Unable to copy systemvm ISO on secondary storage. src location: /usr/share/cloudstack-common/vms/systemvm.iso, dest location: /var/cloudstack/mnt/VM/345050034639.1562462a/systemvm/systemvm-4.15.1.0.iso Permissions for /usr/share/cloudstack-common/ drwxr-xr-x. 5 root root 43 May 3 10:21 . drwxr-xr-x. 101 root root 4096 May 3 10:21 .. drwxr-xr-x. 2 root root 30 Jun 16 15:44 lib drwxr-xr-x. 7 root root 93 Jun 16 15:44 scripts drwxr-xr-x. 2 root root 75 Jun 16 15:45 vms Permissions for /var/cloudstack/mnt/VM/ drwxr-xr-x. 3 cloud cloud 35 Jun 18 11:30 . drwxrwx---. 3 root cloud 16 Jun 18 11:30 .. drwxrwxrwt. 4 nobody nobody 4096 Jun 18 09:38 345050034639.1562462a Any ideas? Many thanks! Mike -----Original Message----- From: Corey, Mike <mike.co...@sap.com.INVALID> Sent: Thursday, June 17, 2021 12:10 PM To: us...@cloudstack.apache.org; dev@cloudstack.apache.org Subject: [CAUTION] Issues Found Apache CloudStack 4.15.1.0 (RC2) Hi, Thanks for pushing this out. I'm looking forward to trying the template/instance deployment in my VMware PILOT. A couple items I noticed off the "new" build are: 1 - During zone creation with VMware and setting up the physical networks - adding the traffic label to use a VDS does NOT keep/take/apply. Once the zone is created and you go into the physical networks, the VDS traffic label is blank when it should be in this format "vSwtichName,VLAN,typeofswitch". The only physical network traffic label that saved during zone setup wizard was for the Management stack; my storage and guest physical network traffic labels did not save from the wizard. 2 - Initial SystemVM deployment, the secondary storage permission do not allow the copy of the systemvm.iso to the secondary/systemvm/ folder. I had to first create a /mnt/secondary/systemvm/ folder and chmod -R for this copy to function. More to come... Mike -----Original Message----- From: Rohit Yadav <ro...@apache.org> Sent: Wednesday, June 16, 2021 12:28 PM To: dev@cloudstack.apache.org; us...@cloudstack.apache.org Subject: [VOTE] Apache CloudStack 4.15.1.0 (RC2) Hi All, I've created a 4.15.1.0 release, with the following artifacts up for a vote: Git Branch: https://github.com/apache/cloudstack/tree/4.15.1.0-RC20210616T2128 Commit SHA: 3afd37022b9dac52cd146dccada6012e47a80232 Source release (checksums and signatures are available at the same location): https://dist.apache.org/repos/dist/dev/cloudstack/4.15.1.0/ PGP release keys (signed using 5ED1E1122DC5E8A4A45112C2484248210EE3D884): https://dist.apache.org/repos/dist/release/cloudstack/KEYS The vote will be open for the next week until 22 June 2021. For sanity in tallying the vote, can PMC members please be sure to indicate "(binding)" with their vote? [ ] +1 approve [ ] +0 no opinion [ ] -1 disapprove (and reason why) For users convenience, the packages from this release candidate and 4.15.1 systemvmtemplates are available here: https://download.cloudstack.org/testing/4.15.1.0-RC2/ https://download.cloudstack.org/systemvm/4.15/ Documentation is not published yet, but the following may be referenced for upgrade related tests: (there's a new 4.15.1 systemvmtemplate to be registered prior to upgrade) https://github.com/apache/cloudstack-documentation/tree/4.15/source/upgrading/upgrade Regards.
