JoaoJandre opened a new pull request, #12758: URL: https://github.com/apache/cloudstack/pull/12758
### Description This PR adds a new native incremental backup provider for KVM. The design document which goes into details of the implementation can be found on https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406622120. The validation process which is detailed in the design document will be added to this PR soon. The file extraction process will be added in a later PR. This PR adds a few new APIs: - The `createNativeBackupOffering` API has the following parameters: | Parameter | Description | Default Value | Required | | ------------------- | -------------------------------------------------------------------- | ------------- | -------- | | `name` | Specifies the name of the offering | - | Yes | | `compress` | Specifies whether the offering supports backup compression | `false` | No | | `validate` | Specifies whether the offering supports backup validation | `false` | No | | `allowQuickRestore` | Specifies whether the offering supports quick restore | `false` | No | | `allowExtractFile` | Specifies whether the offering supports file extraction from backups | `false` | No | | `backupchainsize` | Backup chain size for backups created with this offering. If this is set, it overrides the backup.chain.size setting | - | No | | `compressionlibrary` | Specifies the compression library for offerings that support compression. Accepted values are zstd and zlib. By default, zstd is used for images that support it. If the image only supports zlib, it will be used regardless of this parameter. | `zstd` | No | - The `deleteNativeBackupOffering` API has the following parameter: | Parameter | Description | Required | | --------- | ----------------------------------------- | -------- | | `id` | Identifier of the native backup offering. | Yes | A native backup offering can only be removed if it is not currently imported. - The `listNativeBackupOfferings` API has the following parameters: | Parameter | Description | Required | | | ------------------- | --------------------------------------------------------------- | -------- | -- | | `id` | Identifier of the offering. | No | | | `compress` | Lists only offerings that support backup compression. | No | | | `validate` | Lists only offerings that support backup validation. | No | | | `allowQuickRestore` | Lists only offerings that support quick restore. | No | | | `allowExtractFile` | Lists only offerings that support file extraction from backups. | No | | | `showRemoved` | Lists also offerings that have already been removed. | `false` | No | - The `listBackupCompressionJobs` has the following parameters | Parameter | Description | | --- | --- | | `id` | List only the job with the specified ID | | `backupid` | List jobs associated with the specified backup | | `hostid` | List jobs associated with the specified host. When this parameter is provided, the `executing` parameter is implicit | | `zoneid` | List jobs associated with the specified zone | | `type` | List jobs of the specified type. Accepts `Starting` or `Finalizing` | | `executing` | List jobs that are currently executing | | `scheduled` | List jobs scheduled to run in the future | By default, lists all offerings that have not been removed. </details> It also adds parameters to the following APIs: - The `isolated` parameter was added to the `createBackup` and `createBackupSchedule` APIs - The `quickRestore` parameter was added to the `restoreBackup`, `restoreVolumeFromBackupAndAttachToVM` and `createVMFromBackup` APIs - The `hostId` parameter was added to the `restoreBackup` and `restoreVolumeFromBackupAndAttachToVM` APIs, which can only be used by root admins and only when quick restore is true. New settings were also added: | Configuration | Description | Default Value | | --- | --- | --- | | `backup.chain.size` | Determines the max size of a backup chain. If cloud admins set it to 1 , all the backups will be full backups. With values lower than 1, the backup chain will be unlimited, unless it is stopped by another process. Please note that unlimited backup chains have a higher chance of getting corrupted, as new backups will be dependent on all of the older ones. | 8 | | `knib.timeout` | Timeout, in seconds, to execute KNIB commands. After the command times out, the Management Server will still wait for another knib.timeout seconds to receive a response from the Agent. | 43200 | | `backup.compression.task.enabled` | Determines whether the task responsible for scheduling compression jobs is active. If not, compression jobs will not run | `true` | | `backup.compression.max.concurrent.compressions.per.host` | Maximum number of concurrent compression jobs. Compression finalization jobs ignore this setting | `5` | | `backup.compression.max.job.retries` | Maximum number of attempts for executing compression jobs | `2` | | `backup.compression.retry.interval` | Interval, in minutes, between attempts to run compression jobs | `60` | | `backup.compression.timeout` | Timeout, in seconds, for running compression jobs | `28800` | | `backup.compression.minimum.free.storage` | Minimum required available storage to start the backup compression process. This setting accepts a real number that is multiplied by the total size of the backup to determine the necessary available space. By default, the storage must have the same amount of available space as the space occupied by the backup. | `1` | | `backup.compression.coroutines` | Number of coroutines used for the compression process, each coroutine has its own thread | `1` | | `backup.compression.rate.limit` | Compression rate limit, in MB/s. Values less than 1 disable the limit | `0` | ### Types of changes - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [X] New feature (non-breaking change which adds functionality) - [ ] Bug fix (non-breaking change which fixes an issue) - [ ] Enhancement (improves an existing feature and functionality) - [ ] Cleanup (Code refactoring and cleanup, that may add test cases) - [ ] Build/CI - [ ] Test (unit or integration test code) ### Feature/Enhancement Scale or Bug Severity #### Feature/Enhancement Scale - [X] Major - [ ] Minor #### Bug Severity - [ ] BLOCKER - [ ] Critical - [ ] Major - [ ] Minor - [ ] Trivial ### Screenshots (if appropriate): ### How Has This Been Tested? #### Tests related to disk-only VM snapshots | N | Test | Result | | --- | --- | --- | | 1 | Take disk-only VM snapshot | ok | | 2 | Take disk-only VM snapshot again | ok | | 3 | Stop VM, revert to snapshot 2, start VM | Correct deltas found in the VM volume chain | | 4 | Stop VM, revert to snapshot 1, start VM | Correct deltas found in the VM volume chain | | 5 | Take disk-only VM snapshot | ok | | 6 | Remove disk-only VM snapshot 1 | Marked as removed, not removed from storage | | 7 | Remove disk-only VM snapshot 3 | Merged with current volume | | 8 | Remove disk-only VM snapshot 2 | Removed, snap 1 merged with the current volume | #### Basic tests with backup Using `backup.chain.size`=3 | N | Test | Result | | --- | --- | --- | | 1 | With the VM stopped, I created a backup (b1) | Full backup created | | 2 | Started VM, wrote data, created a second backup (b2) | Incremental backup created | | 3 | Stopped the VM, went back to backup 1, started | Ok, VM without data | | 4 | Stopped the VM, went back to backup 2, started | Ok, data from the backup created in test 2 present | | 5 | Created 4 backups (b3, b4, b5, b6) | b3 was a full, b4 and b5 incremental, b6 full | | 6 | Removing the last backup (b6) | the delta on the primary was merged with the volume | | 7 | Removed backups b4 and b5 | they were marked as removed, but not deleted from storage | | 8 | Batch removing the remaining backups | Ok, all removed | | 9 | Created a new backup | ok | | 10 | Detached the VM from the offer | Deltas were merged on primary | | 11 | Removing this last backup | ok | #### Interactions with other functionalities I created a new VM with a root disk and a data disk for the tests below. | N | Test | Result | | --- | --- | --- | | 1 | Took a new backup and migrated the VM | ok | | 2 | Migrated the VM + one of the volumes | ok, the migrated volume had no delta on the primary, the other volume still had a delta | | 3 | Took a new backup | For the volume that was not migrated the backup was incremental, for the migrated volume, it was a full backup | | 4 | I took 2 backups | OK, the finished normally | | 5 | Try restoring one of the backups from before the migration | OK | | 6 | Created file 1, created backup b1 | OK | | 7 | Created file 2, created VM snap s1 | OK | | 8 | Created file 3, created VM snap s2 | OK | | 9 | Created file 4, created backup b2 | OK | | 10 | Created file 5, created backup b3 | OK | | 11 | Stopped the VM, restored VM snap s1, started | Files 1 and 2 present | | 12 | Stopped the VM, restored VM snap s2, started | Files 1, 2 and 3 present | | 13 | Removed VM snapshots | ok | | 14 | Restored backup b1, started | file 1 present | | 15 | Restored backup b2, started | files 1, 2, 3 and 4 present | | 16 | Restored backup b3, started | files 1, 2, 3, 4 and 5 present | | 17 | Took a new backup b4 | ok | | 18 | Attached a new volume, wrote data, took a backup b5 | ok | | 19 | Stopped the VM, restored backup b4, started the VM | the new volume was not affected by the restoration | | 20 | Detached the volume, restored backup b5 | a new volume was created and attached to the VM, the files were there | | 21 | Created a backup | ok | | 22 | Created a volume snapshot | OK | | 23 | Revert volume snapshot | I verified that the delta on the primary left by the last backup was removed | #### Configuration Tests - I changed the value of the backup.compression.task.enabled setting and verified that no new jobs were started. I verified that when returned to true, they were executed. - I changed the value of the backup.compression.max.concurrent.compressions.per.host setting and verified that the number of jobs executed simultaneously for each host was relative to the value of the setting. I also verified that the value -1 does not limit the number of jobs executed by the host. - I verified that the number of retries respects the backup.compression.max.job.retries setting. - I verified that the time between retries respects the backup.compression.retry.interval setting. - I changed the value of the backup.compression.minimum.free.storage setting and verified that the job failed if there was not enough free space. - I changed the value of the backup.compression.coroutines setting and verified that the value passed to qemu-img was reflected. - I changed the value of the backup.compression.rate.limit setting and verified that the value was passed to qemu-img. #### Compression Tests Tests performed with an offer that provides compressed backups support | Test | Result | | --- | --- | | Create full backup | Backup created and compressed | | Create incremental backup | Backup created and compressed | | Create 10 backups of the same machine | Backups created sequentially, but compressed in parallel | Tests with `restoreVolumeFromBackupAndAttachToVM` | N | Test | Result | | --- | --- | --- | | 1 | Restore volume A from VM 1 to the same VM while it is stopped | New volume created, restored, and attached | | 2 | Restore volume B from VM 1 to the same VM while it is running | New volume created, restored, and attached | | 3 | Restore volume B from VM 1 to VM 2 while it is running | New volume created, restored, and attached | | 4 | Restore volume A from VM 1 to VM 2 while it is running, even though the VM was deleted | New volume created, restored, and attached | | 5 | Restore a volume to a VM using quickrestore | New volume created, attached, and consolidated with the backup | | 6 | Restore a volume to a stopped VM using quickrestore and specifying the hostId | New volume created, attached, VM started on the specified host, and volume consolidated | #### Tests with `restoreBackup` | N | Test | Result | | --- | --- | --- | | 1 | Restore VM without quickrestore | OK | | 2 | Restore VM with quickrestore | Volumes restored, VM started, and volumes consolidated | | 3 | Restore VM with quickrestore and specifying hostId | Volumes restored, VM started on the specified host, and volumes consolidated | | 4 | Detach a volume from the VM and repeat test 3 | Detached volume duplicated, attached to the VM, restored, and VM started on the host, volumes consolidated | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
