Re: Feature Cloudstack 4.15
Hi Benoit, Yes, it's possible to write a backup and recovery plugin. The framework exposes interfaces that your plugin or an external service managed by the plugin can execute/implement. For example, taking backup, restoring backup to primary storage etc. Depending on what/how you're integrating it the B&R framework itself may required to be refactored to accomodate the new plugin. If it helps you can look into the current B&R framework and plugin (see the dummy provider for example) or the original pull request for implementation details - https://github.com/apache/cloudstack/pull/3553 Regards. From: benoit lair Sent: Friday, September 3, 2021 17:38 To: d...@cloudstack.apache.org ; users@cloudstack.apache.org Subject: Feature Cloudstack 4.15 Hi , I am trying to use Backup and Recovery Framework with ACS 4.15.1 I would like to implement it with Xcp-NG servers What i see is that only Veeam with Vmware is ready Would it be possible to have an interface in order to define a custom External Provider (3rd Party Backup Solutions like bacula, amanda or backuppc ) like described here : https://cwiki.apache.org/confluence/display/CLOUDSTACK/Backup+and+Recovery+Framework I was thinking about a form giving the commands to execute for each type of Backup API Call of the framework Thanks for your help and ideas Regards, Benoit
Re: XCP-ng Backup Cloudstack 4.15
+1 what Yordan shared, in addition The B&R framework is extensible with the provider model, that is you can write plugins. The framework assumes that plugin is responsible for actually carrying out the operation, i.e. execution. The plugin however can either choose to do those operations or delegate to a service that performs the operation. There is a dummy provider/plugin which is for testing purposes of the framework. We've only one backup vendor support that is Veeam and the Veeam B&R provider/plugin simply delegates B&R framework operations to the Veeam manager/service. At the time of writing it was only tested with Veeam & VMware, but now Veeam has added support for other hypervisors (I've not checked the list myself so not sure if Xen/Xcp-ng is already on it or not). So, in theory with some changes, the Veeam B&R provider can support any hypervisor that Veeam supports. Regards From: Yordan Kostov Sent: Friday, September 3, 2021 15:00 To: users@cloudstack.apache.org Subject: RE: XCP-ng Backup Cloudstack 4.15 Hey Benoit, I am also interested is such integration to external vendor for XCP hypervisor. Lately my attention is in other matters but I do begun thinking on an external solution for a cold backup. Here is a design concept: Backup: 1. ACS framework does occasional volume backups and keeps them on NFS share (hot backup) 2. Veeam collects the backup folder structure and contents once every X and keeps it on tape (cold backup) In this case ACS framework is what the backup solution for hot backup and Veeam is for cold. Hot restore: 1. Happens through ACS Framework and is done by the end user. Cold restore: 1. admin dumps a cold backup to a NFS 2. in an automated manner single or all volumes are imported into ACS and assigned to the proper account owners of the original VMs. 3. ACS user restore VMs from volumes as Hot restore. From everything above what is not possible is how the restored folder structure and volumes can be automatically imported and assigned to the proper accounts? What I came up it as least effort solution that also is resilient is on the backup NFS server to make a cron script that executes once an hour and scans the /backup partition. ACS NFS backup partition has the following structure //snapshots/account_id/VOLUME_ID /master_and_delta_files In each VOLUME_ID directory a YAML file will be created listing: - owner acc id - origin volume id - origin volume name (taken from ACS DB) - list of mater and delta files Consecutive runs of the script will compare if VOLUME_ID folder contents are different than the YAML and will update it. Usually that will before Veeam backup occurs. Then Veeam backup will occur. There is one caveat here - hot backups can occur anytime depending on user settings while cold backup happens once per time frame. There can be the case where cold backup occurs while hot backup jobs are running. This should be avoided. When restore is required specific volume will be restored from Veeam and with restore shell script it will take the volume YAML conif: - converge deltas - set new volume name as RESTORED_ - import the volume to ACS under the origin account ID Moreover a secondary script can create VMs from volumes for a case of mass cold restore but that will be at later stage. Again that is just a concept and if anyone has an idea on how to improve or simplify that will be great! Best regards, Jordan -Original Message- From: benoit lair Sent: 02 септември 2021 г. 18:36 To: users@cloudstack.apache.org Subject: Re: XCP-ng Backup Cloudstack 4.15 [X] This message came from outside your organization Is there a way to implement ourself a custom external provider in order to backup VMs ? Le jeu. 2 sept. 2021 à 15:52, benoit lair a écrit : > Hello, > > I am interested too in doing Backup VM for Xcp-NG Would you have a > solution for using Veeam like Yordan aims ? > > Le lun. 12 juil. 2021 à 13:30, Abishek Budhathoki > a écrit : > >> Thank You for the response. Really apricated. >> >> On 2021/07/12 09:41:07, Rohit Yadav wrote: >> > Hi Abishek, >> > >> > That's right, the current Backup & Recovery framework only supports >> Veeam provider on VMware. >> > >> > For XenServer/xcpng, we don't have a plugin/provider, however >> > volume >> snapshots can be used to backup snapshots on secondary storage. >> > >> > Regards. >> > >> > Regards, >> > Rohit Yadav >> > >> > >> > From: Abishek Budhathoki >> > Sent: Saturday, July 10, 2021 7:42:12 PM >> > To: users@cloudstack.apache.org >> > Subject: XCP-ng Backup Cloudstack 4.15 >> > >> > Hello EveryOne, >> > >> > I am trying clo
Re: RE: Remove host from MySQL?
Thanks for that reply Jordan, when I did that, I got: mysql> delete from host where id=12; ERROR 1451 (23000): Cannot delete or update a parent row: a foreign key constraint fails (`cloud`.`vm_instance`, CONSTRAINT `fk_vm_instance__last_host_id` FOREIGN KEY (`last_host_id`) REFERENCES `host` (`id`)) mysql> Any idea? Thanks, Jim
Secondary storage missing in one host.
Hell All, I have setup cloudstack 4.15.1 with 2 KVM host machine. Everything is working fine but I was unable to migrate systsem vm's to host 2. All is working fine on host 1. Error is as follows: (s-7-VM) Resource [Host:9] is unreachable: Host 9: Unable to prepare for migration due to com.cloud.utils.exception.CloudRuntimeException: Could not fetch storage pool b3ecc007-5403-30c4-93f6-b6c382d70b5f from libvirt When I looked into the issue the in host 2 the primary nfs storage was mounted but secondary storage was missing. Can anybody guide me to remount the secondary storage in the second host. I will be very grateful for any help. Thank You.
Re: High increase in bandwidth usage
Hi I am facing the same problem. Please suggest any solution group members. Thanks in advance On Tue, 7 Sep, 2021, 11:30 pm R R, wrote: > I installed a cloudstack server on a bare metal server (all in one > installation). The bandwidth usage was normal. After a couple days, the > bandwidth usage was very high, got several emails as well from the DC. I > tried to limit it using wondershaper. Worked for a while, but then I was > locked out of the machine. Couldn't ssh into the machine. Had to format the > machine. > > The same thing happened again. I am able to ssh into the system for now, > bandwidth usage is high, cloudstack server isn't responding. Attaching ss > of cloudstack management server logs. > > Please address me if I am doing something wrong, or the solution to this > problem. >
RE: High increase in bandwidth usage
Hi, That would be bandwidth between which hosts? Also, what exactly would you call normal and excessive bandwidth usage? Regards Alex -Original Message- From: Saurabh Rapatwar Sent: 08 September 2021 16:46 To: users@cloudstack.apache.org Cc: d...@cloudstack.apache.org Subject: Re: High increase in bandwidth usage Hi I am facing the same problem. Please suggest any solution group members. Thanks in advance On Tue, 7 Sep, 2021, 11:30 pm R R, wrote: > I installed a cloudstack server on a bare metal server (all in one > installation). The bandwidth usage was normal. After a couple days, > the bandwidth usage was very high, got several emails as well from the > DC. I tried to limit it using wondershaper. Worked for a while, but > then I was locked out of the machine. Couldn't ssh into the machine. > Had to format the machine. > > The same thing happened again. I am able to ssh into the system for > now, bandwidth usage is high, cloudstack server isn't responding. > Attaching ss of cloudstack management server logs. > > Please address me if I am doing something wrong, or the solution to > this problem. >
Re: Secondary storage missing in one host.
Abishek, can you install a VM directly on host 2? and can you mount the secondary storage by hand? On Wed, Sep 8, 2021 at 4:45 PM Abishek wrote: > Hell All, > > I have setup cloudstack 4.15.1 with 2 KVM host machine. Everything is > working fine but I was unable to migrate systsem vm's to host 2. All is > working fine on host 1. Error is as follows: > (s-7-VM) Resource [Host:9] is unreachable: Host 9: Unable to prepare for > migration due to com.cloud.utils.exception.CloudRuntimeException: Could not > fetch storage pool b3ecc007-5403-30c4-93f6-b6c382d70b5f from libvirt > > When I looked into the issue the in host 2 the primary nfs storage was > mounted but secondary storage was missing. Can anybody guide me to remount > the secondary storage in the second host. > I will be very grateful for any help. > > Thank You. > -- Daan
Re: Secondary storage missing in one host.
Hi Abishek, It is not needed to mount secondary storage on kvm hosts during vm migration. Can you post more lines in agent.log ? -Wei On Wed, 8 Sept 2021 at 16:45, Abishek wrote: > Hell All, > > I have setup cloudstack 4.15.1 with 2 KVM host machine. Everything is > working fine but I was unable to migrate systsem vm's to host 2. All is > working fine on host 1. Error is as follows: > (s-7-VM) Resource [Host:9] is unreachable: Host 9: Unable to prepare for > migration due to com.cloud.utils.exception.CloudRuntimeException: Could not > fetch storage pool b3ecc007-5403-30c4-93f6-b6c382d70b5f from libvirt > > When I looked into the issue the in host 2 the primary nfs storage was > mounted but secondary storage was missing. Can anybody guide me to remount > the secondary storage in the second host. > I will be very grateful for any help. > > Thank You. >
Re: [VOTE] Apache CloudStack 4.15.2.0 (RC1)
signing correct, archive contents sane (no further testing done) +1 (binding) On Tue, Sep 7, 2021 at 2:53 PM Rohit Yadav wrote: > All, > > I've created a 4.15.2.0 release, with the following artifacts up for a > vote: > > Git Branch and Commit SHA: > https://github.com/apache/cloudstack/tree/4.15.2.0-RC20210907T1815 > Commit: 5ba2867598ecf7ce16807e78d5033c342a2a52d7 > > Source release (checksums and signatures are available at the same > location): > https://dist.apache.org/repos/dist/dev/cloudstack/4.15.2.0/ > > PGP release keys (signed using 5ED1E1122DC5E8A4A45112C2484248210EE3D884): > https://dist.apache.org/repos/dist/release/cloudstack/KEYS > > The vote will be open for a week until 13 September 2021. > > For sanity in tallying the vote, can PMC members please be sure to indicate > "(binding)" with their vote? > > [ ] +1 approve > [ ] +0 no opinion > [ ] -1 disapprove (and reason why) > > For users convenience, the packages from this release candidate (RC1) > will be available here shortly: > https://download.cloudstack.org/testing/4.15.2.0-RC1/ > > There is no new systemvmtemplate for 4.15.2, the 4.15.1 > systemvmtemplate can be used from here: > https://download.cloudstack.org/systemvm/4.15/ > > Docs are not published yet but upgrade notes are similar to the one > below without the requirement of registering a new systemvmtemplate: > > https://github.com/apache/cloudstack-documentation/tree/4.15/source/upgrading/upgrade > > Regards. > -- Daan
Re: High increase in bandwidth usage
This should not happen, Cloudstack Is just web application, it does not consume any bandwidth . The one consume bandwidth is the VM inside . On Wed, Sep 8, 2021 at 10:52 PM Alex Mattioli wrote: > Hi, > > That would be bandwidth between which hosts? Also, what exactly would > you call normal and excessive bandwidth usage? > > Regards > Alex > > > > > -Original Message- > From: Saurabh Rapatwar > Sent: 08 September 2021 16:46 > To: users@cloudstack.apache.org > Cc: d...@cloudstack.apache.org > Subject: Re: High increase in bandwidth usage > > Hi > > I am facing the same problem. Please suggest any solution group members. > > Thanks in advance > > On Tue, 7 Sep, 2021, 11:30 pm R R, wrote: > > > I installed a cloudstack server on a bare metal server (all in one > > installation). The bandwidth usage was normal. After a couple days, > > the bandwidth usage was very high, got several emails as well from the > > DC. I tried to limit it using wondershaper. Worked for a while, but > > then I was locked out of the machine. Couldn't ssh into the machine. > > Had to format the machine. > > > > The same thing happened again. I am able to ssh into the system for > > now, bandwidth usage is high, cloudstack server isn't responding. > > Attaching ss of cloudstack management server logs. > > > > Please address me if I am doing something wrong, or the solution to > > this problem. > > > -- Regards, Hean Seng
Re: Remove host from MySQL?
Hi James, You can update db by sql Update vm_instance set last_host_id =NULL where last_host_id=12; -Wei On Tuesday, 7 September 2021, James Steele wrote: > Thanks for that reply Jordan, > > when I did that, I got: > > mysql> delete from host where id=12; > ERROR 1451 (23000): Cannot delete or update a parent row: a foreign key > constraint fails (`cloud`.`vm_instance`, CONSTRAINT > `fk_vm_instance__last_host_id` FOREIGN KEY (`last_host_id`) REFERENCES > `host` (`id`)) > mysql> > > Any idea? Thanks, Jim >
RE: RE: Remove host from MySQL?
This means in table vm_instance there are VM instances that are still noted to reside on that host. From what I have seen that is a common ACS behavior. What you need to do is go to that table, filter by host and make a list of those VMs. Find if they really exist as you don’t want to remove real ones. After that is verified you can delete them one by one based on VM id or all of them at once based on last_host_id. Regards, Jordan -Original Message- From: James Steele Sent: 07 септември 2021 г. 12:00 To: users@cloudstack.apache.org Subject: Re: RE: Remove host from MySQL? [X] This message came from outside your organization Thanks for that reply Jordan, when I did that, I got: mysql> delete from host where id=12; ERROR 1451 (23000): Cannot delete or update a parent row: a foreign key constraint fails (`cloud`.`vm_instance`, CONSTRAINT `fk_vm_instance__last_host_id` FOREIGN KEY (`last_host_id`) REFERENCES `host` (`id`)) mysql> Any idea? Thanks, Jim
RE: RE: Remove host from MySQL?
Just saw Wei response, I suggest you take his approach as it is safer than mine. Best regards, Jordan -Original Message- From: Yordan Kostov Sent: 09 септември 2021 г. 8:41 To: users@cloudstack.apache.org Subject: RE: RE: Remove host from MySQL? [X] This message came from outside your organization This means in table vm_instance there are VM instances that are still noted to reside on that host. From what I have seen that is a common ACS behavior. What you need to do is go to that table, filter by host and make a list of those VMs. Find if they really exist as you don’t want to remove real ones. After that is verified you can delete them one by one based on VM id or all of them at once based on last_host_id. Regards, Jordan -Original Message- From: James Steele Sent: 07 септември 2021 г. 12:00 To: users@cloudstack.apache.org Subject: Re: RE: Remove host from MySQL? [X] This message came from outside your organization Thanks for that reply Jordan, when I did that, I got: mysql> delete from host where id=12; ERROR 1451 (23000): Cannot delete or update a parent row: a foreign key constraint fails (`cloud`.`vm_instance`, CONSTRAINT `fk_vm_instance__last_host_id` FOREIGN KEY (`last_host_id`) REFERENCES `host` (`id`)) mysql> Any idea? Thanks, Jim
Re: Secondary storage missing in one host.
Yes I was able to install the VM in the second host but was not able to migrate the system vm to second host. Also I was able to mount the secondary storage in the 2nd host. I was messing around with adding new secondary storage and migrating the content to new storage. I think the issue was caused by it. On 2021/09/08 15:10:10, Daan Hoogland wrote: > Abishek, > can you install a VM directly on host 2? > and can you mount the secondary storage by hand? > > On Wed, Sep 8, 2021 at 4:45 PM Abishek wrote: > > > Hell All, > > > > I have setup cloudstack 4.15.1 with 2 KVM host machine. Everything is > > working fine but I was unable to migrate systsem vm's to host 2. All is > > working fine on host 1. Error is as follows: > > (s-7-VM) Resource [Host:9] is unreachable: Host 9: Unable to prepare for > > migration due to com.cloud.utils.exception.CloudRuntimeException: Could not > > fetch storage pool b3ecc007-5403-30c4-93f6-b6c382d70b5f from libvirt > > > > When I looked into the issue the in host 2 the primary nfs storage was > > mounted but secondary storage was missing. Can anybody guide me to remount > > the secondary storage in the second host. > > I will be very grateful for any help. > > > > Thank You. > > > > > -- > Daan >
Re: Secondary storage missing in one host.
Hello Wei, I was playing around with adding new secondary storage and migrating the secondary storage to new storage and then the issue arised. But I reverted the process and everything started working fine. I am working on a test environment and I have deleted the host yesterday and I don't have agent logs of that host currently. I will try to recreate the issue again and will revert you back if it arises again. Thank You. On 2021/09/08 15:19:23, Wei ZHOU wrote: > Hi Abishek, > > It is not needed to mount secondary storage on kvm hosts during vm > migration. > Can you post more lines in agent.log ? > > -Wei > > On Wed, 8 Sept 2021 at 16:45, Abishek wrote: > > > Hell All, > > > > I have setup cloudstack 4.15.1 with 2 KVM host machine. Everything is > > working fine but I was unable to migrate systsem vm's to host 2. All is > > working fine on host 1. Error is as follows: > > (s-7-VM) Resource [Host:9] is unreachable: Host 9: Unable to prepare for > > migration due to com.cloud.utils.exception.CloudRuntimeException: Could not > > fetch storage pool b3ecc007-5403-30c4-93f6-b6c382d70b5f from libvirt > > > > When I looked into the issue the in host 2 the primary nfs storage was > > mounted but secondary storage was missing. Can anybody guide me to remount > > the secondary storage in the second host. > > I will be very grateful for any help. > > > > Thank You. > > >
