Hi,
Template creation happens on secondary storage via SSVM. Try restarting the
SSVM. That should stop the template creation activity. I am not very sure
here.
Although, I believe there is some time needed even before cloud stack marks
the template in error state so that you can remove it.
OR
There is an option to allow user/domain to have more quota for parameters
like template etc. Change that.
--
Makrand
On Mon, Jan 15, 2018 at 3:01 PM, Özhan Rüzgar Karaman <
oruzgarkara...@gmail.com> wrote:
> We have one more finding, issue occur because we did not wait for template
> creation process to finish, we have checked the logs and we restarted the
> management server even template creation process did not finished yet :) We
> need to find a way to rollback these template creation process like
> manually updating template table...
>
> 2018-01-15 11:41:14,779 WARN [o.a.c.f.j.i.AsyncJobMonitor]
> (Timer-1:ctx-55f871c2) (logid:5a3f1482) Task (job-1112) has been pending
> for 990 seconds
> 2018-01-15 11:42:14,778 WARN [o.a.c.f.j.i.AsyncJobMonitor]
> (Timer-1:ctx-988624e1) (logid:cae141bb) Task (job-1118) has been pending
> for 758 seconds
> 2018-01-15 11:42:14,778 WARN [o.a.c.f.j.i.AsyncJobMonitor]
> (Timer-1:ctx-988624e1) (logid:cae141bb) Task (job-1117) has been pending
> for 774 seconds
> 2018-01-15 11:42:14,778 WARN [o.a.c.f.j.i.AsyncJobMonitor]
> (Timer-1:ctx-988624e1) (logid:cae141bb) Task (job-1116) has been pending
> for 800 seconds
> 2018-01-15 11:42:14,778 WARN [o.a.c.f.j.i.AsyncJobMonitor]
> (Timer-1:ctx-988624e1) (logid:cae141bb) Task (job-1115) has been pending
> for 881 seconds
> 2018-01-15 11:42:14,779 WARN [o.a.c.f.j.i.AsyncJobMonitor]
> (Timer-1:ctx-988624e1) (logid:cae141bb) Task (job-1114) has been pending
> for 892 seconds
> 2018-01-15 11:42:14,779 WARN [o.a.c.f.j.i.AsyncJobMonitor]
> (Timer-1:ctx-988624e1) (logid:cae141bb) Task (job-1113) has been pending
> for 905 seconds
> 2018-01-15 11:42:14,779 WARN [o.a.c.f.j.i.AsyncJobMonitor]
> (Timer-1:ctx-988624e1) (logid:cae141bb) Task (job-1112) has been pending
> for 1050 seconds
>
>
> On Mon, Jan 15, 2018 at 12:21 PM, Özhan Rüzgar Karaman <
> oruzgarkara...@gmail.com> wrote:
>
> > Hi;
> > We are using Cloudstack 4.9.3, we hit an error when trying to delete
> > template from cloudstack ui. Our developer send 7 template creation
> > requests for same snapshot each waiting 5-10 seconds between each
> requests.
> >
> > Now we have 7 templates all in Creating state and when we try to delete
> > them we get "Failed to update state" error from interface. The detailed
> > cloudstack management logs are attached.
> >
> > We also checked mysql db for templates status and all are in Creating
> > state, we restarted cloudstack management server to see if this template
> > creating process maybe rolled back but we again could not delete template
> > and we got same error.
> >
> > Because these templates remain on cloudstack our user could not take any
> > more actions because they hit to the resource limitation for user/domain.
> >
> > Is there a way to manually update Creating state to maybe Ready and try
> to
> > delete this templates after that?
> >
> > We only like to delete this templates in a proper way, does anyone could
> > give us an advice?
> >
> > Thanks
> > Özhan
> >
> > Mysql Records:
> > mysql> select * from template_store_ref where state = 'Creating';
> > +----+----------+-------------+---------------------+-------
> > -------+--------+--------------+------+------------+--------
> > -------+----------------+-----------+------------+----------
> > -----------+------+----------+-----------+---------+--------
> > ------+---------+---------------------+---------------------
> > -+--------------+
> > | id | store_id | template_id | created | last_updated |
> > job_id | download_pct | size | store_role | physical_size |
> download_state
> > | error_str | local_path | install_path | url | state |
> > destroyed | is_copy | update_count | ref_cnt | updated |
> > download_url_created | download_url |
> > +----+----------+-------------+---------------------+-------
> > -------+--------+--------------+------+------------+--------
> > -------+----------------+-----------+------------+----------
> > -----------+------+----------+-----------+---------+--------
> > ------+---------+---------------------+---------------------
> > -+--------------+
> > | 23 | 1 | 213 | 2018-01-15 08:24:44 | NULL | NULL
> > | 0 | NULL | Image | 0 | NULL |
> > NULL | NULL | template/tmpl/4/213 | NULL | Creating |
> 0
> > | 0 | 1 | 0 | 2018-01-15 08:24:44 | NULL
> > | NULL |
> > | 24 | 1 | 214 | 2018-01-15 08:27:08 | NULL | NULL
> > | 0 | NULL | Image | 0 | NULL |
> > NULL | NULL | template/tmpl/4/214 | NULL | Creating |
> 0
> > | 0 | 1 | 0 | 2018-01-15 08:27:08 | NULL
> > | NULL |
> > | 25 | 1 | 215 | 2018-01-15 08:27:22 | NULL | NULL
> > | 0 | NULL | Image | 0 | NULL |
> > NULL | NULL | template/tmpl/4/215 | NULL | Creating |
> 0
> > | 0 | 1 | 0 | 2018-01-15 08:27:22 | NULL
> > | NULL |
> > | 26 | 1 | 216 | 2018-01-15 08:27:33 | NULL | NULL
> > | 0 | NULL | Image | 0 | NULL |
> > NULL | NULL | template/tmpl/4/216 | NULL | Creating |
> 0
> > | 0 | 1 | 0 | 2018-01-15 08:27:33 | NULL
> > | NULL |
> > | 27 | 1 | 217 | 2018-01-15 08:28:54 | NULL | NULL
> > | 0 | NULL | Image | 0 | NULL |
> > NULL | NULL | template/tmpl/4/217 | NULL | Creating |
> 0
> > | 0 | 1 | 0 | 2018-01-15 08:28:54 | NULL
> > | NULL |
> > | 28 | 1 | 218 | 2018-01-15 08:29:19 | NULL | NULL
> > | 0 | NULL | Image | 0 | NULL |
> > NULL | NULL | template/tmpl/4/218 | NULL | Creating |
> 0
> > | 0 | 1 | 0 | 2018-01-15 08:29:20 | NULL
> > | NULL |
> > | 29 | 1 | 219 | 2018-01-15 08:29:36 | NULL | NULL
> > | 0 | NULL | Image | 0 | NULL |
> > NULL | NULL | template/tmpl/4/219 | NULL | Creating |
> 0
> > | 0 | 1 | 0 | 2018-01-15 08:29:36 | NULL
> > | NULL |
> > +----+----------+-------------+---------------------+-------
> > -------+--------+--------------+------+------------+--------
> > -------+----------------+-----------+------------+----------
> > -----------+------+----------+-----------+---------+--------
> > ------+---------+---------------------+---------------------
> > -+--------------+
> > 7 rows in set (0.00 sec)
> >
> > mysql>
> >
> > mysql> select * from vm_template where display_text like '%api55';
> > +-----+--------------------------------------------+--------
> > ----------+--------------------------------------+--------+-
> > ---------+------+-----+------+------+--------+--------------
> > -------+---------+------------+----------+------------------
> > +-----------------+---------------+-------------+----------+
> > -------------+-------------+-------------+-----------------+
> > --------------------+--------------+----------+------+------
> > --+--------------+---------+----------------------+
> > | id | unique_name | name |
> > uuid | public | featured | type | hvm |
> > bits | url | format | created | removed | account_id |
> > checksum | display_text | enable_password | enable_sshkey |
> guest_os_id
> > | bootable | prepopulate | cross_zones | extractable | hypervisor_type |
> > source_template_id | template_tag | sort_key | size | state |
> update_count
> > | updated | dynamically_scalable |
> > +-----+--------------------------------------------+--------
> > ----------+--------------------------------------+--------+-
> > ---------+------+-----+------+------+--------+--------------
> > -------+---------+------------+----------+------------------
> > +-----------------+---------------+-------------+----------+
> > -------------+-------------+-------------+-----------------+
> > --------------------+--------------+----------+------+------
> > --+--------------+---------+----------------------+
> > | 213 | 213-4-6c7694ef-cb29-3c06-878f-301300c71a21 | snaptovm - api55 |
> > 094de09d-0beb-4e4f-8f3c-bab492f47645 | 0 | 0 | USER | 1 |
> > 64 | NULL | RAW | 2018-01-15 08:24:44 | NULL | 4 | NULL
> > | snaptovm - api55 | 1 | 0 | 254 |
> > 1 | 0 | 0 | 1 | KVM |
> > 201 | NULL | 0 | NULL | Active | 0 | NULL
> > | 0 |
> > | 214 | 214-4-c9071093-06cd-3351-a1e6-e5570b16c27a | snaptovm - api55 |
> > 57986622-f5fb-4200-a24a-95651ca086bc | 0 | 0 | USER | 1 |
> > 64 | NULL | RAW | 2018-01-15 08:27:08 | NULL | 4 | NULL
> > | snaptovm - api55 | 1 | 0 | 254 |
> > 1 | 0 | 0 | 1 | KVM |
> > 201 | NULL | 0 | NULL | Active | 0 | NULL
> > | 0 |
> > | 215 | 215-4-ee8ec3d2-8880-35e0-b1c9-0365b9c1839e | snaptovm - api55 |
> > 95cb604b-e27c-4385-b2fc-9123c856549c | 0 | 0 | USER | 1 |
> > 64 | NULL | RAW | 2018-01-15 08:27:22 | NULL | 4 | NULL
> > | snaptovm - api55 | 1 | 0 | 254 |
> > 1 | 0 | 0 | 1 | KVM |
> > 201 | NULL | 0 | NULL | Active | 0 | NULL
> > | 0 |
> > | 216 | 216-4-0ee2f1c7-ff28-3dbc-b58a-cae1e2f71a6b | snaptovm - api55 |
> > 533cea26-243d-43cf-ba2f-53c8daf0dbfe | 0 | 0 | USER | 1 |
> > 64 | NULL | RAW | 2018-01-15 08:27:32 | NULL | 4 | NULL
> > | snaptovm - api55 | 1 | 0 | 254 |
> > 1 | 0 | 0 | 1 | KVM |
> > 201 | NULL | 0 | NULL | Active | 0 | NULL
> > | 0 |
> > | 217 | 217-4-3f5093c7-335c-3fc5-b5ff-7c0481dbf5e4 | snaptovm - api55 |
> > c3cc1614-0ac8-437d-ba1a-ce8253237c81 | 0 | 0 | USER | 1 |
> > 64 | NULL | RAW | 2018-01-15 08:28:53 | NULL | 4 | NULL
> > | snaptovm - api55 | 1 | 0 | 254 |
> > 1 | 0 | 0 | 1 | KVM |
> > 201 | NULL | 0 | NULL | Active | 0 | NULL
> > | 0 |
> > | 218 | 218-4-d49b0a44-efef-337d-9bdc-5c859f67de32 | snaptovm - api55 |
> > e8f801e9-4c8a-420e-9df4-a26f98bb6c03 | 0 | 0 | USER | 1 |
> > 64 | NULL | RAW | 2018-01-15 08:29:19 | NULL | 4 | NULL
> > | snaptovm - api55 | 1 | 0 | 254 |
> > 1 | 0 | 0 | 1 | KVM |
> > 201 | NULL | 0 | NULL | Active | 0 | NULL
> > | 0 |
> > | 219 | 219-4-9aece59e-c8d5-3eb8-9035-8db61072d007 | snaptovm - api55 |
> > c14dcd71-c001-408e-924f-f5cd9fc88d06 | 0 | 0 | USER | 1 |
> > 64 | NULL | RAW | 2018-01-15 08:29:35 | NULL | 4 | NULL
> > | snaptovm - api55 | 1 | 0 | 254 |
> > 1 | 0 | 0 | 1 | KVM |
> > 201 | NULL | 0 | NULL | Active | 0 | NULL
> > | 0 |
> > +-----+--------------------------------------------+--------
> > ----------+--------------------------------------+--------+-
> > ---------+------+-----+------+------+--------+--------------
> > -------+---------+------------+----------+------------------
> > +-----------------+---------------+-------------+----------+
> > -------------+-------------+-------------+-----------------+
> > --------------------+--------------+----------+------+------
> > --+--------------+---------+----------------------+
> > 7 rows in set (0.01 sec)
> >
> > mysql>
> >
> >
> > Management Server Logs:
> >
> > 2018-01-15 12:02:35,069 ERROR [c.c.a.ApiAsyncJobDispatcher]
> > (API-Job-Executor-5:ctx-9135400d job-1133) (logid:ea158f63) Unexpected
> > exception while executing org.apache.cloudstack.api.c
> > ommand.user.template.DeleteTemplateCmd
> > com.cloud.utils.exception.CloudRuntimeException: Failed to update state
> > at org.apache.cloudstack.storage.image.store.TemplateObject.
> > processEvent(TemplateObject.java:174)
> > at org.apache.cloudstack.storage.image.TemplateServiceImpl.
> > deleteTemplateAsync(TemplateServiceImpl.java:698)
> > at com.cloud.template.HypervisorTemplateAdapter.delete(
> > HypervisorTemplateAdapter.java:401)
> > at com.cloud.template.TemplateManagerImpl.deleteTemplate(
> > TemplateManagerImpl.java:1192)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at sun.reflect.NativeMethodAccessorImpl.invoke(
> > NativeMethodAccessorImpl.java:57)
> > at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> > DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:606)
> > at org.springframework.aop.support.AopUtils.
> > invokeJoinpointUsingReflection(AopUtils.java:317)
> > at org.springframework.aop.framework.ReflectiveMethodInvocation.
> > invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> > at org.springframework.aop.framework.ReflectiveMethodInvocation.
> > proceed(ReflectiveMethodInvocation.java:150)
> > at org.apache.cloudstack.network.contrail.management.
> > EventUtils$EventInterceptor.invoke(EventUtils.java:107)
> > at org.springframework.aop.framework.ReflectiveMethodInvocation.
> > proceed(ReflectiveMethodInvocation.java:161)
> > at com.cloud.event.ActionEventInterceptor.invoke(
> > ActionEventInterceptor.java:51)
> > at org.springframework.aop.framework.ReflectiveMethodInvocation.
> > proceed(ReflectiveMethodInvocation.java:161)
> > at org.springframework.aop.interceptor.
> > ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> > at org.springframework.aop.framework.ReflectiveMethodInvocation.
> > proceed(ReflectiveMethodInvocation.java:172)
> > at org.springframework.aop.framework.JdkDynamicAopProxy.
> > invoke(JdkDynamicAopProxy.java:204)
> > at com.sun.proxy.$Proxy183.deleteTemplate(Unknown Source)
> > at org.apache.cloudstack.api.command.user.template.
> > DeleteTemplateCmd.execute(DeleteTemplateCmd.java:113)
> > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
> > at com.cloud.api.ApiAsyncJobDispatcher.runJob(
> > ApiAsyncJobDispatcher.java:108)
> > at org.apache.cloudstack.framework.jobs.impl.
> > AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
> > at org.apache.cloudstack.managed.context.
> > ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> > at org.apache.cloudstack.managed.context.impl.
> > DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> > at org.apache.cloudstack.managed.context.impl.
> > DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> > at org.apache.cloudstack.managed.context.impl.
> > DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> > at org.apache.cloudstack.managed.context.
> > ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> > at org.apache.cloudstack.framework.jobs.impl.
> > AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
> > at java.util.concurrent.Executors$RunnableAdapter.
> > call(Executors.java:473)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(
> > ThreadPoolExecutor.java:1152)
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > ThreadPoolExecutor.java:622)
> > at java.lang.Thread.run(Thread.java:748)
> > Caused by: com.cloud.utils.fsm.NoTransitionException: Unable to
> > transition to a new state from Creating via DestroyRequested
> > at com.cloud.utils.fsm.StateMachine2.getTransition(
> > StateMachine2.java:108)
> > at com.cloud.utils.fsm.StateMachine2.getNextState(
> > StateMachine2.java:94)
> > at com.cloud.utils.fsm.StateMachine2.transitTo(
> > StateMachine2.java:124)
> > ... 35 more
> > 2018-01-15 12:02:35,071 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (API-Job-Executor-5:ctx-9135400d job-1133) (logid:ea158f63) Complete
> > async job-1133, jobStatus: FAILED, resultCode: 530,
> > result: org.apache.cloudstack.api.response.ExceptionResponse/
> > null/{"uuidList":[],"errorcode":530,"errortext":"Failed to update
> state"}
> >
>
