Hi;
We are using Cloudstack 4.9.3, we hit an error when trying to delete
template from cloudstack ui. Our developer send 7 template creation
requests for same snapshot each waiting 5-10 seconds between each requests.
Now we have 7 templates all in Creating state and when we try to delete
them we get "Failed to update state" error from interface. The detailed
cloudstack management logs are attached.
We also checked mysql db for templates status and all are in Creating
state, we restarted cloudstack management server to see if this template
creating process maybe rolled back but we again could not delete template
and we got same error.
Because these templates remain on cloudstack our user could not take any
more actions because they hit to the resource limitation for user/domain.
Is there a way to manually update Creating state to maybe Ready and try to
delete this templates after that?
We only like to delete this templates in a proper way, does anyone could
give us an advice?
Thanks
Özhan
Mysql Records:
mysql> select * from template_store_ref where state = 'Creating';
+----+----------+-------------+---------------------+--------------+--------+--------------+------+------------+---------------+----------------+-----------+------------+---------------------+------+----------+-----------+---------+--------------+---------+---------------------+----------------------+--------------+
| id | store_id | template_id | created | last_updated | job_id
| download_pct | size | store_role | physical_size | download_state |
error_str | local_path | install_path | url | state | destroyed
| is_copy | update_count | ref_cnt | updated |
download_url_created | download_url |
+----+----------+-------------+---------------------+--------------+--------+--------------+------+------------+---------------+----------------+-----------+------------+---------------------+------+----------+-----------+---------+--------------+---------+---------------------+----------------------+--------------+
| 23 | 1 | 213 | 2018-01-15 08:24:44 | NULL | NULL
| 0 | NULL | Image | 0 | NULL |
NULL | NULL | template/tmpl/4/213 | NULL | Creating | 0
| 0 | 1 | 0 | 2018-01-15 08:24:44 | NULL
| NULL |
| 24 | 1 | 214 | 2018-01-15 08:27:08 | NULL | NULL
| 0 | NULL | Image | 0 | NULL |
NULL | NULL | template/tmpl/4/214 | NULL | Creating | 0
| 0 | 1 | 0 | 2018-01-15 08:27:08 | NULL
| NULL |
| 25 | 1 | 215 | 2018-01-15 08:27:22 | NULL | NULL
| 0 | NULL | Image | 0 | NULL |
NULL | NULL | template/tmpl/4/215 | NULL | Creating | 0
| 0 | 1 | 0 | 2018-01-15 08:27:22 | NULL
| NULL |
| 26 | 1 | 216 | 2018-01-15 08:27:33 | NULL | NULL
| 0 | NULL | Image | 0 | NULL |
NULL | NULL | template/tmpl/4/216 | NULL | Creating | 0
| 0 | 1 | 0 | 2018-01-15 08:27:33 | NULL
| NULL |
| 27 | 1 | 217 | 2018-01-15 08:28:54 | NULL | NULL
| 0 | NULL | Image | 0 | NULL |
NULL | NULL | template/tmpl/4/217 | NULL | Creating | 0
| 0 | 1 | 0 | 2018-01-15 08:28:54 | NULL
| NULL |
| 28 | 1 | 218 | 2018-01-15 08:29:19 | NULL | NULL
| 0 | NULL | Image | 0 | NULL |
NULL | NULL | template/tmpl/4/218 | NULL | Creating | 0
| 0 | 1 | 0 | 2018-01-15 08:29:20 | NULL
| NULL |
| 29 | 1 | 219 | 2018-01-15 08:29:36 | NULL | NULL
| 0 | NULL | Image | 0 | NULL |
NULL | NULL | template/tmpl/4/219 | NULL | Creating | 0
| 0 | 1 | 0 | 2018-01-15 08:29:36 | NULL
| NULL |
+----+----------+-------------+---------------------+--------------+--------+--------------+------+------------+---------------+----------------+-----------+------------+---------------------+------+----------+-----------+---------+--------------+---------+---------------------+----------------------+--------------+
7 rows in set (0.00 sec)
mysql>
mysql> select * from vm_template where display_text like '%api55';
+-----+--------------------------------------------+------------------+--------------------------------------+--------+----------+------+-----+------+------+--------+---------------------+---------+------------+----------+------------------+-----------------+---------------+-------------+----------+-------------+-------------+-------------+-----------------+--------------------+--------------+----------+------+--------+--------------+---------+----------------------+
| id | unique_name | name |
uuid | public | featured | type | hvm |
bits | url | format | created | removed | account_id |
checksum | display_text | enable_password | enable_sshkey | guest_os_id
| bootable | prepopulate | cross_zones | extractable | hypervisor_type |
source_template_id | template_tag | sort_key | size | state | update_count
| updated | dynamically_scalable |
+-----+--------------------------------------------+------------------+--------------------------------------+--------+----------+------+-----+------+------+--------+---------------------+---------+------------+----------+------------------+-----------------+---------------+-------------+----------+-------------+-------------+-------------+-----------------+--------------------+--------------+----------+------+--------+--------------+---------+----------------------+
| 213 | 213-4-6c7694ef-cb29-3c06-878f-301300c71a21 | snaptovm - api55 |
094de09d-0beb-4e4f-8f3c-bab492f47645 | 0 | 0 | USER | 1 |
64 | NULL | RAW | 2018-01-15 08:24:44 | NULL | 4 | NULL
| snaptovm - api55 | 1 | 0 | 254 |
1 | 0 | 0 | 1 | KVM |
201 | NULL | 0 | NULL | Active | 0 | NULL
| 0 |
| 214 | 214-4-c9071093-06cd-3351-a1e6-e5570b16c27a | snaptovm - api55 |
57986622-f5fb-4200-a24a-95651ca086bc | 0 | 0 | USER | 1 |
64 | NULL | RAW | 2018-01-15 08:27:08 | NULL | 4 | NULL
| snaptovm - api55 | 1 | 0 | 254 |
1 | 0 | 0 | 1 | KVM |
201 | NULL | 0 | NULL | Active | 0 | NULL
| 0 |
| 215 | 215-4-ee8ec3d2-8880-35e0-b1c9-0365b9c1839e | snaptovm - api55 |
95cb604b-e27c-4385-b2fc-9123c856549c | 0 | 0 | USER | 1 |
64 | NULL | RAW | 2018-01-15 08:27:22 | NULL | 4 | NULL
| snaptovm - api55 | 1 | 0 | 254 |
1 | 0 | 0 | 1 | KVM |
201 | NULL | 0 | NULL | Active | 0 | NULL
| 0 |
| 216 | 216-4-0ee2f1c7-ff28-3dbc-b58a-cae1e2f71a6b | snaptovm - api55 |
533cea26-243d-43cf-ba2f-53c8daf0dbfe | 0 | 0 | USER | 1 |
64 | NULL | RAW | 2018-01-15 08:27:32 | NULL | 4 | NULL
| snaptovm - api55 | 1 | 0 | 254 |
1 | 0 | 0 | 1 | KVM |
201 | NULL | 0 | NULL | Active | 0 | NULL
| 0 |
| 217 | 217-4-3f5093c7-335c-3fc5-b5ff-7c0481dbf5e4 | snaptovm - api55 |
c3cc1614-0ac8-437d-ba1a-ce8253237c81 | 0 | 0 | USER | 1 |
64 | NULL | RAW | 2018-01-15 08:28:53 | NULL | 4 | NULL
| snaptovm - api55 | 1 | 0 | 254 |
1 | 0 | 0 | 1 | KVM |
201 | NULL | 0 | NULL | Active | 0 | NULL
| 0 |
| 218 | 218-4-d49b0a44-efef-337d-9bdc-5c859f67de32 | snaptovm - api55 |
e8f801e9-4c8a-420e-9df4-a26f98bb6c03 | 0 | 0 | USER | 1 |
64 | NULL | RAW | 2018-01-15 08:29:19 | NULL | 4 | NULL
| snaptovm - api55 | 1 | 0 | 254 |
1 | 0 | 0 | 1 | KVM |
201 | NULL | 0 | NULL | Active | 0 | NULL
| 0 |
| 219 | 219-4-9aece59e-c8d5-3eb8-9035-8db61072d007 | snaptovm - api55 |
c14dcd71-c001-408e-924f-f5cd9fc88d06 | 0 | 0 | USER | 1 |
64 | NULL | RAW | 2018-01-15 08:29:35 | NULL | 4 | NULL
| snaptovm - api55 | 1 | 0 | 254 |
1 | 0 | 0 | 1 | KVM |
201 | NULL | 0 | NULL | Active | 0 | NULL
| 0 |
+-----+--------------------------------------------+------------------+--------------------------------------+--------+----------+------+-----+------+------+--------+---------------------+---------+------------+----------+------------------+-----------------+---------------+-------------+----------+-------------+-------------+-------------+-----------------+--------------------+--------------+----------+------+--------+--------------+---------+----------------------+
7 rows in set (0.01 sec)
mysql>
Management Server Logs:
2018-01-15 12:02:35,069 ERROR [c.c.a.ApiAsyncJobDispatcher]
(API-Job-Executor-5:ctx-9135400d job-1133) (logid:ea158f63) Unexpected
exception while executing org.apache.cloudstack.api.c
ommand.user.template.DeleteTemplateCmd
com.cloud.utils.exception.CloudRuntimeException: Failed to update state
at
org.apache.cloudstack.storage.image.store.TemplateObject.processEvent(TemplateObject.java:174)
at
org.apache.cloudstack.storage.image.TemplateServiceImpl.deleteTemplateAsync(TemplateServiceImpl.java:698)
at
com.cloud.template.HypervisorTemplateAdapter.delete(HypervisorTemplateAdapter.java:401)
at
com.cloud.template.TemplateManagerImpl.deleteTemplate(TemplateManagerImpl.java:1192)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at
com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy183.deleteTemplate(Unknown Source)
at
org.apache.cloudstack.api.command.user.template.DeleteTemplateCmd.execute(DeleteTemplateCmd.java:113)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
at
com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:473)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.cloud.utils.fsm.NoTransitionException: Unable to transition
to a new state from Creating via DestroyRequested
at
com.cloud.utils.fsm.StateMachine2.getTransition(StateMachine2.java:108)
at
com.cloud.utils.fsm.StateMachine2.getNextState(StateMachine2.java:94)
at
com.cloud.utils.fsm.StateMachine2.transitTo(StateMachine2.java:124)
... 35 more
2018-01-15 12:02:35,071 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-5:ctx-9135400d job-1133) (logid:ea158f63) Complete async
job-1133, jobStatus: FAILED, resultCode: 530,
result:
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
to update state"}
