Hi Jordan, Glad that it helped to find the problem.
To know exactly what is happening when a command runs via vmops, from cloud.log you can see that command is initiated something like "DEBUG [root] #### CLOUD enter delete_secondary_storage_folder ####" and you can look for the command definition in vmops or vmopsSnapshots scripts under /etc/xapi.d/plugins. Regards, Harikrishna ________________________________ From: Yordan Kostov <yord...@nsogroup.com> Sent: Friday, May 7, 2021 7:38 PM To: users@cloudstack.apache.org <users@cloudstack.apache.org> Subject: RE: cannot create template? NFS is now able to mount for template provisioning. Updated the XCPNG hosts and reboot and voila it is working! So little thing to cause so much trouble. Thank you for the tips Harikrishna! By the way how do you find what command Cloudstack is using at such occasions? It is a nice approach for troubleshooting purposes. Regards, Jordan -----Original Message----- From: Yordan Kostov <yord...@nsogroup.com> Sent: Friday, May 7, 2021 11:10 AM To: users@cloudstack.apache.org Subject: RE: cannot create template? [X] This message came from outside your organization Thank you for the tip! This command actually those not work - "mount -o soft,tcp,timeo=133,retrans=1 <remote directory> <local directory>" Verbose output reports: mount.nfs: timeout set for Fri May 7 09:58:35 2021 mount.nfs: trying text-based options 'soft,tcp,timeo=10,retrans=1,vers=4.1,addr=A.A.A.A,clientaddr=B.B.B.B' mount.nfs: mount(2): Input/output error mount.nfs: mount system call failed My previous tests (and cloudstack mounts) that were working were all on NFSv3. This is very weird that certain Cloudstack mounts are made through NFSv3 and others through v4. I do tried to mount nfs v4 from XenCenter and it failed the same way. - there is nothing in the XCPNG logs - SMlog or xensource log - there is no traffic coming to the NFS server nor any traffic is seen on the firewall. I believe there is something wrong with the XCP and Nfs v4 mount request, as soon as execute the command with NFS v3 specified it works "mount -o soft,tcp,timeo=133,retrans=1, vers=3 <remote directory> <local directory>" I will continue to investigate. Regards, Jordan -----Original Message----- From: Harikrishna Patnala <harikrishna.patn...@shapeblue.com> Sent: Thursday, May 6, 2021 5:43 AM To: users@cloudstack.apache.org Subject: Re: cannot create template? [X] This message came from outside your organization Hi Jordon, You can try running the below mount command on XCP host which runs during this operation to see what is the actual mount problem, you can check it in verbose mode too. "mount -o soft,tcp,timeo=133,retrans=1 <remote directory> <local directory>" The other error raises a concern of adding unsupported XCP 8.2 version is added properly in CloudStack. I see your previous email thread about XCP-ng 8.2 support. "Failed to create templateUnsupported command issued: org.apache.cloudstack.storage.command.CopyCommand. Are you sure you got the right type of server?" Can you check around the above exception if you get any clue around why exactly it failed. You can try the same operations with some lower versions of XCP-ng and see if it fails there too. Regards, Harikrishna ________________________________ From: Yordan Kostov <yord...@nsogroup.com> Sent: Wednesday, May 5, 2021 6:17 PM To: users@cloudstack.apache.org <users@cloudstack.apache.org> Subject: RE: cannot create template? It seems the host tries to mount the NFS and move the template there. Unfortunately it cannot happen even though NFS can be mounted manualy or through Cloudstack when instances are deployed. Output from XCPNG log files -> /var/log/cloud/cloud.log the following messages are seen. DEBUG [root] Unexpected error while trying to mount x.x.x.x:/var/secondary to /var/run/cloud_mount/5c3a0c95-719c-4b1b-9d09-77f1dd475003 DEBUG [root] create_secondary_storage_folder failed. DEBUG [root] CommandException raised while trying to umount /var/run/cloud_mount/5c3a0c95-719c-4b1b-9d09-77f1dd475003 DEBUG [root] #### CLOUD enter delete_secondary_storage_folder #### DEBUG [root] delete_secondary_storage_folder, args: {'folder': 'template/tmpl/2/211', 'remoteMountPath': 'x.x.x.x:/var/secondary'} DEBUG [root] Unexpected error while trying to mount x.x.x.x:/var/secondary to /var/run/cloud_mount/244482cf-48e4-4acf-bcd3-6c37bfac5014 DEBUG [root] delete_secondary_storage_folder failed. DEBUG [root] CommandException raised while trying to umount /var/run/cloud_mount/244482cf-48e4-4acf-bcd3-6c37bfac5014 NFS firewall and XCPNG iptables are disabled. Best regards, Jordan From: Yordan Kostov Sent: Wednesday, May 5, 2021 1:15 PM To: 'users@cloudstack.apache.org' <users@cloudstack.apache.org> Subject: cannot create template? Hey everyone, Bumped into the following error today while trying to create a template from ROOT volume. Environment is: CS 4.15 + XCPNG 8.2 + presetup - Fiber storage The process failed with the following error message in the management server logs: callHostPlugin failed for cmd: create_secondary_storage_folder with args remoteMountPath: X.X.X.X:/var/secondary, newFolder: template/tmpl/2/206, due to There was a failure communicating with the plugin. GUI error is - (Centos 7) Failed to create templateUnsupported command issued: org.apache.cloudstack.storage.command.CopyCommand. Are you sure you got the right type of server? I am not really sure where to start here, there are no blockages on the networking side, also the NFS can mount to the hosts successfully and instances can be deployed from ISOs. Best Regards, Jordan <font size="2"><font color="#D8D8D8">11!</font>