Hi Makrand;
We do not want to update user/domain resources, we like to find a way to
remove templates which are problematic. We also restarted ssvm but it did
not solve the issue, templates are still on Creating state on database.
We could update the templates status field on template_store_ref table from
Creating to maybe Ready and then try to delete template but we do not know
which state is correct for this action?
So if an developer could give us a clue that would be great.
Thanks
Özhan
On Mon, Jan 15, 2018 at 1:34 PM, Makrand <makrandsa...@gmail.com> wrote:
> Hi,
>
> Template creation happens on secondary storage via SSVM. Try restarting the
> SSVM. That should stop the template creation activity. I am not very sure
> here.
>
> Although, I believe there is some time needed even before cloud stack marks
> the template in error state so that you can remove it.
>
> OR
>
> There is an option to allow user/domain to have more quota for parameters
> like template etc. Change that.
>
>
>
>
> --
> Makrand
>
>
> On Mon, Jan 15, 2018 at 3:01 PM, Özhan Rüzgar Karaman <
> oruzgarkara...@gmail.com> wrote:
>
> > We have one more finding, issue occur because we did not wait for
> template
> > creation process to finish, we have checked the logs and we restarted the
> > management server even template creation process did not finished yet :)
> We
> > need to find a way to rollback these template creation process like
> > manually updating template table...
> >
> > 2018-01-15 11:41:14,779 WARN [o.a.c.f.j.i.AsyncJobMonitor]
> > (Timer-1:ctx-55f871c2) (logid:5a3f1482) Task (job-1112) has been pending
> > for 990 seconds
> > 2018-01-15 11:42:14,778 WARN [o.a.c.f.j.i.AsyncJobMonitor]
> > (Timer-1:ctx-988624e1) (logid:cae141bb) Task (job-1118) has been pending
> > for 758 seconds
> > 2018-01-15 11:42:14,778 WARN [o.a.c.f.j.i.AsyncJobMonitor]
> > (Timer-1:ctx-988624e1) (logid:cae141bb) Task (job-1117) has been pending
> > for 774 seconds
> > 2018-01-15 11:42:14,778 WARN [o.a.c.f.j.i.AsyncJobMonitor]
> > (Timer-1:ctx-988624e1) (logid:cae141bb) Task (job-1116) has been pending
> > for 800 seconds
> > 2018-01-15 11:42:14,778 WARN [o.a.c.f.j.i.AsyncJobMonitor]
> > (Timer-1:ctx-988624e1) (logid:cae141bb) Task (job-1115) has been pending
> > for 881 seconds
> > 2018-01-15 11:42:14,779 WARN [o.a.c.f.j.i.AsyncJobMonitor]
> > (Timer-1:ctx-988624e1) (logid:cae141bb) Task (job-1114) has been pending
> > for 892 seconds
> > 2018-01-15 11:42:14,779 WARN [o.a.c.f.j.i.AsyncJobMonitor]
> > (Timer-1:ctx-988624e1) (logid:cae141bb) Task (job-1113) has been pending
> > for 905 seconds
> > 2018-01-15 11:42:14,779 WARN [o.a.c.f.j.i.AsyncJobMonitor]
> > (Timer-1:ctx-988624e1) (logid:cae141bb) Task (job-1112) has been pending
> > for 1050 seconds
> >
> >
> > On Mon, Jan 15, 2018 at 12:21 PM, Özhan Rüzgar Karaman <
> > oruzgarkara...@gmail.com> wrote:
> >
> > > Hi;
> > > We are using Cloudstack 4.9.3, we hit an error when trying to delete
> > > template from cloudstack ui. Our developer send 7 template creation
> > > requests for same snapshot each waiting 5-10 seconds between each
> > requests.
> > >
> > > Now we have 7 templates all in Creating state and when we try to delete
> > > them we get "Failed to update state" error from interface. The detailed
> > > cloudstack management logs are attached.
> > >
> > > We also checked mysql db for templates status and all are in Creating
> > > state, we restarted cloudstack management server to see if this
> template
> > > creating process maybe rolled back but we again could not delete
> template
> > > and we got same error.
> > >
> > > Because these templates remain on cloudstack our user could not take
> any
> > > more actions because they hit to the resource limitation for
> user/domain.
> > >
> > > Is there a way to manually update Creating state to maybe Ready and try
> > to
> > > delete this templates after that?
> > >
> > > We only like to delete this templates in a proper way, does anyone
> could
> > > give us an advice?
> > >
> > > Thanks
> > > Özhan
> > >
> > > Mysql Records:
> > > mysql> select * from template_store_ref where state = 'Creating';
> > > +----+----------+-------------+---------------------+-------
> > > -------+--------+--------------+------+------------+--------
> > > -------+----------------+-----------+------------+----------
> > > -----------+------+----------+-----------+---------+--------
> > > ------+---------+---------------------+---------------------
> > > -+--------------+
> > > | id | store_id | template_id | created | last_updated |
> > > job_id | download_pct | size | store_role | physical_size |
> > download_state
> > > | error_str | local_path | install_path | url | state |
> > > destroyed | is_copy | update_count | ref_cnt | updated |
> > > download_url_created | download_url |
> > > +----+----------+-------------+---------------------+-------
> > > -------+--------+--------------+------+------------+--------
> > > -------+----------------+-----------+------------+----------
> > > -----------+------+----------+-----------+---------+--------
> > > ------+---------+---------------------+---------------------
> > > -+--------------+
> > > | 23 | 1 | 213 | 2018-01-15 08:24:44 | NULL |
> NULL
> > > | 0 | NULL | Image | 0 | NULL |
> > > NULL | NULL | template/tmpl/4/213 | NULL | Creating |
> > 0
> > > | 0 | 1 | 0 | 2018-01-15 08:24:44 | NULL
> > > | NULL |
> > > | 24 | 1 | 214 | 2018-01-15 08:27:08 | NULL |
> NULL
> > > | 0 | NULL | Image | 0 | NULL |
> > > NULL | NULL | template/tmpl/4/214 | NULL | Creating |
> > 0
> > > | 0 | 1 | 0 | 2018-01-15 08:27:08 | NULL
> > > | NULL |
> > > | 25 | 1 | 215 | 2018-01-15 08:27:22 | NULL |
> NULL
> > > | 0 | NULL | Image | 0 | NULL |
> > > NULL | NULL | template/tmpl/4/215 | NULL | Creating |
> > 0
> > > | 0 | 1 | 0 | 2018-01-15 08:27:22 | NULL
> > > | NULL |
> > > | 26 | 1 | 216 | 2018-01-15 08:27:33 | NULL |
> NULL
> > > | 0 | NULL | Image | 0 | NULL |
> > > NULL | NULL | template/tmpl/4/216 | NULL | Creating |
> > 0
> > > | 0 | 1 | 0 | 2018-01-15 08:27:33 | NULL
> > > | NULL |
> > > | 27 | 1 | 217 | 2018-01-15 08:28:54 | NULL |
> NULL
> > > | 0 | NULL | Image | 0 | NULL |
> > > NULL | NULL | template/tmpl/4/217 | NULL | Creating |
> > 0
> > > | 0 | 1 | 0 | 2018-01-15 08:28:54 | NULL
> > > | NULL |
> > > | 28 | 1 | 218 | 2018-01-15 08:29:19 | NULL |
> NULL
> > > | 0 | NULL | Image | 0 | NULL |
> > > NULL | NULL | template/tmpl/4/218 | NULL | Creating |
> > 0
> > > | 0 | 1 | 0 | 2018-01-15 08:29:20 | NULL
> > > | NULL |
> > > | 29 | 1 | 219 | 2018-01-15 08:29:36 | NULL |
> NULL
> > > | 0 | NULL | Image | 0 | NULL |
> > > NULL | NULL | template/tmpl/4/219 | NULL | Creating |
> > 0
> > > | 0 | 1 | 0 | 2018-01-15 08:29:36 | NULL
> > > | NULL |
> > > +----+----------+-------------+---------------------+-------
> > > -------+--------+--------------+------+------------+--------
> > > -------+----------------+-----------+------------+----------
> > > -----------+------+----------+-----------+---------+--------
> > > ------+---------+---------------------+---------------------
> > > -+--------------+
> > > 7 rows in set (0.00 sec)
> > >
> > > mysql>
> > >
> > > mysql> select * from vm_template where display_text like '%api55';
> > > +-----+--------------------------------------------+--------
> > > ----------+--------------------------------------+--------+-
> > > ---------+------+-----+------+------+--------+--------------
> > > -------+---------+------------+----------+------------------
> > > +-----------------+---------------+-------------+----------+
> > > -------------+-------------+-------------+-----------------+
> > > --------------------+--------------+----------+------+------
> > > --+--------------+---------+----------------------+
> > > | id | unique_name | name |
> > > uuid | public | featured | type | hvm |
> > > bits | url | format | created | removed | account_id |
> > > checksum | display_text | enable_password | enable_sshkey |
> > guest_os_id
> > > | bootable | prepopulate | cross_zones | extractable | hypervisor_type
> |
> > > source_template_id | template_tag | sort_key | size | state |
> > update_count
> > > | updated | dynamically_scalable |
> > > +-----+--------------------------------------------+--------
> > > ----------+--------------------------------------+--------+-
> > > ---------+------+-----+------+------+--------+--------------
> > > -------+---------+------------+----------+------------------
> > > +-----------------+---------------+-------------+----------+
> > > -------------+-------------+-------------+-----------------+
> > > --------------------+--------------+----------+------+------
> > > --+--------------+---------+----------------------+
> > > | 213 | 213-4-6c7694ef-cb29-3c06-878f-301300c71a21 | snaptovm - api55
> |
> > > 094de09d-0beb-4e4f-8f3c-bab492f47645 | 0 | 0 | USER | 1
> |
> > > 64 | NULL | RAW | 2018-01-15 08:24:44 | NULL | 4 | NULL
> > > | snaptovm - api55 | 1 | 0 | 254 |
> > > 1 | 0 | 0 | 1 | KVM |
> > > 201 | NULL | 0 | NULL | Active | 0 |
> NULL
> > > | 0 |
> > > | 214 | 214-4-c9071093-06cd-3351-a1e6-e5570b16c27a | snaptovm - api55
> |
> > > 57986622-f5fb-4200-a24a-95651ca086bc | 0 | 0 | USER | 1
> |
> > > 64 | NULL | RAW | 2018-01-15 08:27:08 | NULL | 4 | NULL
> > > | snaptovm - api55 | 1 | 0 | 254 |
> > > 1 | 0 | 0 | 1 | KVM |
> > > 201 | NULL | 0 | NULL | Active | 0 |
> NULL
> > > | 0 |
> > > | 215 | 215-4-ee8ec3d2-8880-35e0-b1c9-0365b9c1839e | snaptovm - api55
> |
> > > 95cb604b-e27c-4385-b2fc-9123c856549c | 0 | 0 | USER | 1
> |
> > > 64 | NULL | RAW | 2018-01-15 08:27:22 | NULL | 4 | NULL
> > > | snaptovm - api55 | 1 | 0 | 254 |
> > > 1 | 0 | 0 | 1 | KVM |
> > > 201 | NULL | 0 | NULL | Active | 0 |
> NULL
> > > | 0 |
> > > | 216 | 216-4-0ee2f1c7-ff28-3dbc-b58a-cae1e2f71a6b | snaptovm - api55
> |
> > > 533cea26-243d-43cf-ba2f-53c8daf0dbfe | 0 | 0 | USER | 1
> |
> > > 64 | NULL | RAW | 2018-01-15 08:27:32 | NULL | 4 | NULL
> > > | snaptovm - api55 | 1 | 0 | 254 |
> > > 1 | 0 | 0 | 1 | KVM |
> > > 201 | NULL | 0 | NULL | Active | 0 |
> NULL
> > > | 0 |
> > > | 217 | 217-4-3f5093c7-335c-3fc5-b5ff-7c0481dbf5e4 | snaptovm - api55
> |
> > > c3cc1614-0ac8-437d-ba1a-ce8253237c81 | 0 | 0 | USER | 1
> |
> > > 64 | NULL | RAW | 2018-01-15 08:28:53 | NULL | 4 | NULL
> > > | snaptovm - api55 | 1 | 0 | 254 |
> > > 1 | 0 | 0 | 1 | KVM |
> > > 201 | NULL | 0 | NULL | Active | 0 |
> NULL
> > > | 0 |
> > > | 218 | 218-4-d49b0a44-efef-337d-9bdc-5c859f67de32 | snaptovm - api55
> |
> > > e8f801e9-4c8a-420e-9df4-a26f98bb6c03 | 0 | 0 | USER | 1
> |
> > > 64 | NULL | RAW | 2018-01-15 08:29:19 | NULL | 4 | NULL
> > > | snaptovm - api55 | 1 | 0 | 254 |
> > > 1 | 0 | 0 | 1 | KVM |
> > > 201 | NULL | 0 | NULL | Active | 0 |
> NULL
> > > | 0 |
> > > | 219 | 219-4-9aece59e-c8d5-3eb8-9035-8db61072d007 | snaptovm - api55
> |
> > > c14dcd71-c001-408e-924f-f5cd9fc88d06 | 0 | 0 | USER | 1
> |
> > > 64 | NULL | RAW | 2018-01-15 08:29:35 | NULL | 4 | NULL
> > > | snaptovm - api55 | 1 | 0 | 254 |
> > > 1 | 0 | 0 | 1 | KVM |
> > > 201 | NULL | 0 | NULL | Active | 0 |
> NULL
> > > | 0 |
> > > +-----+--------------------------------------------+--------
> > > ----------+--------------------------------------+--------+-
> > > ---------+------+-----+------+------+--------+--------------
> > > -------+---------+------------+----------+------------------
> > > +-----------------+---------------+-------------+----------+
> > > -------------+-------------+-------------+-----------------+
> > > --------------------+--------------+----------+------+------
> > > --+--------------+---------+----------------------+
> > > 7 rows in set (0.01 sec)
> > >
> > > mysql>
> > >
> > >
> > > Management Server Logs:
> > >
> > > 2018-01-15 12:02:35,069 ERROR [c.c.a.ApiAsyncJobDispatcher]
> > > (API-Job-Executor-5:ctx-9135400d job-1133) (logid:ea158f63) Unexpected
> > > exception while executing org.apache.cloudstack.api.c
> > > ommand.user.template.DeleteTemplateCmd
> > > com.cloud.utils.exception.CloudRuntimeException: Failed to update
> state
> > > at org.apache.cloudstack.storage.image.store.TemplateObject.
> > > processEvent(TemplateObject.java:174)
> > > at org.apache.cloudstack.storage.image.TemplateServiceImpl.
> > > deleteTemplateAsync(TemplateServiceImpl.java:698)
> > > at com.cloud.template.HypervisorTemplateAdapter.delete(
> > > HypervisorTemplateAdapter.java:401)
> > > at com.cloud.template.TemplateManagerImpl.deleteTemplate(
> > > TemplateManagerImpl.java:1192)
> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > at sun.reflect.NativeMethodAccessorImpl.invoke(
> > > NativeMethodAccessorImpl.java:57)
> > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> > > DelegatingMethodAccessorImpl.java:43)
> > > at java.lang.reflect.Method.invoke(Method.java:606)
> > > at org.springframework.aop.support.AopUtils.
> > > invokeJoinpointUsingReflection(AopUtils.java:317)
> > > at org.springframework.aop.framework.
> ReflectiveMethodInvocation.
> > > invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> > > at org.springframework.aop.framework.
> ReflectiveMethodInvocation.
> > > proceed(ReflectiveMethodInvocation.java:150)
> > > at org.apache.cloudstack.network.contrail.management.
> > > EventUtils$EventInterceptor.invoke(EventUtils.java:107)
> > > at org.springframework.aop.framework.
> ReflectiveMethodInvocation.
> > > proceed(ReflectiveMethodInvocation.java:161)
> > > at com.cloud.event.ActionEventInterceptor.invoke(
> > > ActionEventInterceptor.java:51)
> > > at org.springframework.aop.framework.
> ReflectiveMethodInvocation.
> > > proceed(ReflectiveMethodInvocation.java:161)
> > > at org.springframework.aop.interceptor.
> > > ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.
> java:91)
> > > at org.springframework.aop.framework.
> ReflectiveMethodInvocation.
> > > proceed(ReflectiveMethodInvocation.java:172)
> > > at org.springframework.aop.framework.JdkDynamicAopProxy.
> > > invoke(JdkDynamicAopProxy.java:204)
> > > at com.sun.proxy.$Proxy183.deleteTemplate(Unknown Source)
> > > at org.apache.cloudstack.api.command.user.template.
> > > DeleteTemplateCmd.execute(DeleteTemplateCmd.java:113)
> > > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:
> 150)
> > > at com.cloud.api.ApiAsyncJobDispatcher.runJob(
> > > ApiAsyncJobDispatcher.java:108)
> > > at org.apache.cloudstack.framework.jobs.impl.
> > > AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
> > > at org.apache.cloudstack.managed.context.
> > > ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> > > at org.apache.cloudstack.managed.context.impl.
> > > DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> > > at org.apache.cloudstack.managed.context.impl.
> > > DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> > > at org.apache.cloudstack.managed.context.impl.
> > > DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> > > at org.apache.cloudstack.managed.context.
> > > ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> > > at org.apache.cloudstack.framework.jobs.impl.
> > > AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
> > > at java.util.concurrent.Executors$RunnableAdapter.
> > > call(Executors.java:473)
> > > at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> > > at java.util.concurrent.ThreadPoolExecutor.runWorker(
> > > ThreadPoolExecutor.java:1152)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > > ThreadPoolExecutor.java:622)
> > > at java.lang.Thread.run(Thread.java:748)
> > > Caused by: com.cloud.utils.fsm.NoTransitionException: Unable to
> > > transition to a new state from Creating via DestroyRequested
> > > at com.cloud.utils.fsm.StateMachine2.getTransition(
> > > StateMachine2.java:108)
> > > at com.cloud.utils.fsm.StateMachine2.getNextState(
> > > StateMachine2.java:94)
> > > at com.cloud.utils.fsm.StateMachine2.transitTo(
> > > StateMachine2.java:124)
> > > ... 35 more
> > > 2018-01-15 12:02:35,071 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > (API-Job-Executor-5:ctx-9135400d job-1133) (logid:ea158f63) Complete
> > > async job-1133, jobStatus: FAILED, resultCode: 530,
> > > result: org.apache.cloudstack.api.response.ExceptionResponse/
> > > null/{"uuidList":[],"errorcode":530,"errortext":"Failed to update
> > state"}
> > >
> >
>
