RE: Avoid re-redownloading URLs when migrating secondary storage
Hi everyone, I have seen problems caused by this behavior (always trying to download public templates with URLs, and only migrating private templates and public templates that do not have URLs) in some environments. For instance, companies create temporary URLs to download templates from, and delete them after the template gets downloaded; due to the existing behavior, when a new image store is added and data is migrated to it, ACS does not move the template to the new image store, and does not download the template in the new image store either, forcing manual intervention to move it. Also, suppose you download a template to an image store from a URL. Some time passes, and the template in that URL gets updated. Then, you add a new secondary storage that tries to download the updated template from the same URL. After the download finishes, ACS will compare the downloaded template's checksum with the existing template's checksum, and fail to synchronize the template because they differ. This way, you will also be forced to manually move the template from the old image store to the new one. Because of the aforementioned problems, I have been investigating the development of an extension (such as some new configurations) that would allow operators to migrate public templates with URLs (avoiding the first issue) and to prioritize copying templates from existing image stores instead of downloading them (avoiding both issues). I will keep this thread updated as I make progress. Other opinions on this would also be appreciated. Best regards. On 2024/04/25 08:51:05 Mevludin Blazevic wrote: > Hi all, > > it looks like with every secondary storage migration Cloudstack tries to > re-download all templates with URLs instead of simply grabbing the > already downloaded template from the old image store. If that's true, > how to avoid that behaviour? > > Mevludin > > >
Re: Avoid re-redownloading URLs when migrating secondary storage
Hi Pearl, I think due to download errors from some of the public templates (official linux cloud templates are affected), the migration process stucks is not proceeding. I have deleted these templates and redownloaded them manually. Still, the job is still in status "scheduled". @Jithin: Yes, I am using the Cloudstack feature for migrating the storage. Mevludin Am 26.04.2024 um 07:20 schrieb Jithin Raju: Hi Mevludin, Just curious about how you are migrating the secondary storage. Are you using the CloudStack feature to migrate data across secondary storage? So you added a new secondary storage and cloudstack already started syncing the public templates from URLs? -Jithin From: Pearl d'Silva Date: Friday, 26 April 2024 at 3:11 AM To: users Subject: Re: Avoid re-redownloading URLs when migrating secondary storage Hi Mevludin, When migrating data to another secondary store, it only considers private templates and public templates with no url - those which are created from snapshots or volumes. Public templates with URLs are automatically synced(downloaded) on to the newly added secondary store. So, they aren't ideally getting duplicated on the new secondary store. Regards, Pearl From: Mevludin Blazevic Sent: April 25, 2024 4:51 AM To: users Subject: Avoid re-redownloading URLs when migrating secondary storage Hi all, it looks like with every secondary storage migration Cloudstack tries to re-download all templates with URLs instead of simply grabbing the already downloaded template from the old image store. If that's true, how to avoid that behaviour? Mevludin -- --- Mevludin Blazevic, M.Sc. Research Assistant / Cloud Administrator University of Koblenz Center for Information and Media Technologies (CIMT) Research Group Information Systems and Smart Data Universitätsstr. 1 | 56070 Koblenz | Germany P: +49 261 287 1326 M: mblaze...@uni-koblenz.de <mailto:%7B%7B%20email%20%7D%7D> W: https://uni-ko.de/riehle <https://www.uni-koblenz.de/Service/%7B%7B%20web%20%7D%7D> <https://www.instagram.com/uni_koblenz/> <https://www.facebook.com/universitaet.koblenz> <https://twitter.com/unikoblenzde> <https://www.youtube.com/@universitaetkoblenz> <https://de.linkedin.com/company/universit%C3%A4t-koblenz> <https://www.uni-koblenz.de>
Re: Avoid re-redownloading URLs when migrating secondary storage
Hi Mevludin, Just curious about how you are migrating the secondary storage. Are you using the CloudStack feature to migrate data across secondary storage? So you added a new secondary storage and cloudstack already started syncing the public templates from URLs? -Jithin From: Pearl d'Silva Date: Friday, 26 April 2024 at 3:11 AM To: users Subject: Re: Avoid re-redownloading URLs when migrating secondary storage Hi Mevludin, When migrating data to another secondary store, it only considers private templates and public templates with no url - those which are created from snapshots or volumes. Public templates with URLs are automatically synced(downloaded) on to the newly added secondary store. So, they aren't ideally getting duplicated on the new secondary store. Regards, Pearl From: Mevludin Blazevic Sent: April 25, 2024 4:51 AM To: users Subject: Avoid re-redownloading URLs when migrating secondary storage Hi all, it looks like with every secondary storage migration Cloudstack tries to re-download all templates with URLs instead of simply grabbing the already downloaded template from the old image store. If that's true, how to avoid that behaviour? Mevludin
Re: Avoid re-redownloading URLs when migrating secondary storage
Hi Mevludin, When migrating data to another secondary store, it only considers private templates and public templates with no url - those which are created from snapshots or volumes. Public templates with URLs are automatically synced(downloaded) on to the newly added secondary store. So, they aren't ideally getting duplicated on the new secondary store. Regards, Pearl From: Mevludin Blazevic Sent: April 25, 2024 4:51 AM To: users Subject: Avoid re-redownloading URLs when migrating secondary storage Hi all, it looks like with every secondary storage migration Cloudstack tries to re-download all templates with URLs instead of simply grabbing the already downloaded template from the old image store. If that's true, how to avoid that behaviour? Mevludin
Avoid re-redownloading URLs when migrating secondary storage
Hi all, it looks like with every secondary storage migration Cloudstack tries to re-download all templates with URLs instead of simply grabbing the already downloaded template from the old image store. If that's true, how to avoid that behaviour? Mevludin
Re: Migrating secondary storage
Hi Jeremy,
You can enable trace logs to check the states. You can also query the database
to check which resources are not ready.
Here are some queries you can use:
SELECT uuid, name, store_id, state FROM template_view WHERE state NOT IN
('Ready', 'Allocated', 'Destroying', 'Destroyed', 'Failed') AND removed IS NULL;
SELECT uuid, name, store_id, store_state FROM snapshot_view WHERE store_state
NOT IN ('Ready', 'Allocated', 'Destroying', 'Destroyed', 'Failed') AND removed
IS NULL;
SELECT volumes.uuid, volumes.name, volume_store_ref.store_id,
volume_store_ref.state
FROM volumes
INNER JOIN volume_store_ref ON volumes.id = volume_store_ref.volume_id
WHERE volumes.removed IS NULL AND volume_store_ref.state NOT IN ('Ready',
'Allocated', 'Destroying', 'Destroyed', 'Failed');
From: Jeremy Hansen
Sent: Saturday, January 27, 2024 5:06 AM
To: Vivek Kumar via users
Subject: Migrating secondary storage
I’m trying to migrate to new secondary storage. I’m receiving this error:
Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]:
com.cloud.utils.exception.CloudRuntimeException: Complete migration failed as
there are data objects which are not Ready - i.e, they may be in Migrating,
creating, copying, etc. states
Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at
org.apache.cloudstack.engine.orchestration.DataMigrationUtility.checkIfCompleteMigrationPossible(DataMigrationUtility.java:122)
Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at
org.apache.cloudstack.engine.orchestration.StorageOrchestrator.migrateData(StorageOrchestrator.java:149)
Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at
com.cloud.storage.ImageStoreServiceImpl.migrateData(ImageStoreServiceImpl.java:157)
Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at
java.base/java.lang.reflect.Method.invoke(Method.java:566)
Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at
org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at
com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at
com.sun.proxy.$Proxy386.migrateData(Unknown Source)
Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at
org.apache.cloudstack.api.command.admin.storage.MigrateSecondaryStorageDataCmd.execute(MigrateSecondaryStorageDataCmd.java:100)
Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at
com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at
com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:112)
Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564
Migrating secondary storage
I’m trying to migrate to new secondary storage. I’m receiving this error: Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: com.cloud.utils.exception.CloudRuntimeException: Complete migration failed as there are data objects which are not Ready - i.e, they may be in Migrating, creating, copying, etc. states Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at org.apache.cloudstack.engine.orchestration.DataMigrationUtility.checkIfCompleteMigrationPossible(DataMigrationUtility.java:122) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at org.apache.cloudstack.engine.orchestration.StorageOrchestrator.migrateData(StorageOrchestrator.java:149) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at com.cloud.storage.ImageStoreServiceImpl.migrateData(ImageStoreServiceImpl.java:157) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at java.base/java.lang.reflect.Method.invoke(Method.java:566) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at com.sun.proxy.$Proxy386.migrateData(Unknown Source) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at org.apache.cloudstack.api.command.admin.storage.MigrateSecondaryStorageDataCmd.execute(MigrateSecondaryStorageDataCmd.java:100) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:112) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at
