The answer is in my previous email, last line: > You can check if there is something wrong in state.db by passing the > contents of it in an XML validator. There will probably be some broken XML > tags there. >
Last time I faced this problem, I put the host in maintenance, powered off or migrate host's VMs, then I opened the file with an XML validator, found the entries that were corrupted and removed them. Then I replaced the old state.db file with this modified file. Afterward, reboot the server and everything should be ok. No need to tell you to do a backup of things, check hypervisors community…., right? http://discussions.citrix.com/topic/241761-help-power-failed-now-xapi-will-not-start/ <http://www.google.com/url?q=http%3A%2F%2Fdiscussions.citrix.com%2Ftopic%2F241761-help-power-failed-now-xapi-will-not-start%2F&sa=D&sntz=1&usg=AFQjCNEMwpDSMLgdddVdD6iDcZ4--yxtkQ> On Wed, Mar 15, 2017 at 2:05 PM, Fabiano A. Souza <f.so...@fdsouza.com.br> wrote: > Very thanks Rafael. I was already imagining that this would be the > problem. Do you know any way to fix this problem in XenServer? > > By your experience do you believe that recreating the pool in XenServer > will correct this problem? > > Again thanks so much for your help. > > --- > FABIANO A. SOUZA > _Assistência e Consultoria em TI_ > f.so...@fdsouza.com.br > http://www.fdsouza.com.br > > Em 15/03/2017 14:36, Rafael Weingärtner escreveu: > > > Looking at the log from your XenServer, and as I said before the problem > is > > not in ACS. > > > > This kind of log entries: > > > > Db_exn.DBCache_NotFound("missing row", "VBD", > > > >> "OpaqueRef:e1136dfc-cdbc-ba21-3eb8-94838cc75030") > > > > or > > > >> D:424e469aca54 failed with exception Db_exn.DBCache_NotFound("missing > >> row", "task", > > > > indicate a problem in the hypervisor metadata. In your case, XenServer > uses > > a file called state.db. This file contains metadata in XML format. > Probably > > your force reboot corrupted some entries there (I have already had > similar > > problems before). You can check if there is something wrong in state.db > by > > passing the contents of it in a XML validator. There will probably be > some > > broken XML tags there. > > > > On Wed, Mar 15, 2017 at 12:15 PM, Fabiano A. Souza < > f.so...@fdsouza.com.br> > > wrote: > > > >> Yes. I set up the DataCenter with the Advanced Network configuration. > >> > >> I'm attaching XenServer file log detailed sample. > >> > >> Thank you so much for your help. > >> --- > >> > >> *Fabiano A. Souza* > >> *Assistência e Consultoria em TI* > >> f.so...@fdsouza.com.br > >> http://www.fdsouza.com.br > >> > >> Em 15/03/2017 12:52, Rafael Weingärtner escreveu: > >> > >> I am not sure if this is a complete destroy and if ACS re-uses something > >> from the old VMs. > >> > >> From your logs, > >> > >> VIF.unplug > >> R:68394c4e8e7c failed with exception Db_exn.DBCache_NotFound("missing > >> > >> This probably happened because when the VM was being created and the VIF > >> was not created in the hypervisor. There are probably other error log > >> entries before this one. From your other log: > >> > >> errorInfo: [HOST_CANNOT_ATTACH_NETWORK, > >> > >> It is clear that there is something wrong in your XenServer. If you > want us > >> to help you, you need to provide a more log entries (not scattered > bits). > >> Also, in the XenServer logs, there will be plenty of exceptions there > >> regarding this problem. > >> You said that user VMs are working. I am guessing you have an advanced > >> networking set up. Maybe the Guest network was not affected, but the > >> Management or Storage network configuration in the XenServer may be > >> affected by some misconfiguration/corruption that happened during the > >> force-reboot. > >> > >> On Wed, Mar 15, 2017 at 11:42 AM, Fabiano A. Souza > <f.so...@fdsouza.com.brwrote: > >> > >> The ACS is trying to do this itself. > >> > >> ACS create, trying start, after error stop and destroy the systemvms > >> instance. The instance name or ID is being incremented at each attempt > >> to create these instances (s-300-VM, s-301-VM...). > >> > >> --- > >> FABIANO A. SOUZA > >> _Assistência e Consultoria em TI_ > >> f.so...@fdsouza.com.br > >> http://www.fdsouza.com.br > >> > >> Em 15/03/2017 12:27, Rafael Weingärtner escreveu: > >> > >> You said you forced a reboot. It looks like something got corrupted in > >> > >> your > >> > >> host. > >> > >> I would not do some drastic measure like destroying pool. > >> > >> Have you tried to destroy the system VMs and then let ACS re-create > them? > >> > >> On Wed, Mar 15, 2017 at 11:23 AM, Fabiano A. Souza < > >> > >> f.so...@fdsouza.com.br> > >> > >> wrote: > >> > >> Hi Rafael > >> > >> I have some other Guest (Windows/Linux) include VRs that works normally. > >> Only systemvms has problem to deploy. I use one pool with two XenServer > >> 7 host. In xenresource.log I see this error below. > >> > >> Mar 15 08:31:32 fl-mia-compute1 xapi: [error|fl-mia-compute1|605 INET > >> :::80|dispatch:VIF.unplug D:6cce7f031461|backtrace] VIF.unplug > >> R:68394c4e8e7c failed with exception Db_exn.DBCache_NotFound("missing > >> row", "VIF", "OpaqueRef:20e49fbf-5c92-ef21-805b-13b1997e326e") > >> > >> Mar 15 08:31:32 fl-mia-compute1 xapi: [error|fl-mia-compute1|605 INET > >> :::80|dispatch:VIF.unplug D:b37f4fd2de0c|backtrace] VIF.unplug > >> R:fcca316e07d3 failed with exception Db_exn.DBCache_NotFound("missing > >> row", "VIF", "OpaqueRef:4d7c3a36-982a-825e-e71f-8978108ec56b") > >> > >> I'm thinking remove the hosts from CloudStack and destroy pool. Create > >> again the pool and re-attach them again on CloudStack configuration. Do > >> you believe that can I have problem to start the guests instance after > >> that? > >> > >> --- > >> FABIANO A. SOUZA > >> _Assistência e Consultoria em TI_ > >> f.so...@fdsouza.com.br > >> http://www.fdsouza.com.br > >> > >> Em 15/03/2017 11:28, Rafael Weingärtner escreveu: > >> > >> This is not an Apache CloudStack (ACS) problem per se. > >> "HOST_CANNOT_ATTACH_NETWOR" > >> Something is wrong with your hosts. Are all of the networks of yours > >> > >> hosts working? is everything in place as ACS expects? > >> > >> On Wed, Mar 15, 2017 at 10:26 AM, Fabiano A. Souza < > >> > >> f.so...@fdsouza.com.br> wrote: > >> > >> After my host with XenServer 7 forced reboot the systemvms can't start > >> anymore. > >> > >> Can you help me? > >> > >> 2017-03-14 09:42:10,724 WARN [c.c.h.x.r.CitrixResourceBase] > >> (DirectAgent-184:ctx-f12cdd04) (logid:4552cd9d) Task failed! Task > >> record: uuid: 84f3483d-1021-22fb-be39-2eb15366e03e > >> nameLabel: Async.VM.start_on > >> nameDescription: > >> allowedOperations: [] > >> currentOperations: {} > >> created: Tue Mar 14 09:42:46 BRT 2017 > >> finished: Tue Mar 14 09:42:46 BRT 2017 > >> status: failure > >> residentOn: com.xensource.xenapi.Host@4e05ddb9 > >> progress: 1.0 > >> type: <none/> > >> result: > >> errorInfo: [HOST_CANNOT_ATTACH_NETWORK, > >> OpaqueRef:3269022e-9b3d-88ca-8fb1-b15f3948b3a9, > >> OpaqueRef:cd7b3c3e-92fd-18a3-eaa7-099aa066bb46] > >> otherConfig: {} > >> subtaskOf: com.xensource.xenapi.Task@aaf13f6f > >> subtasks: [] > >> > >> 2017-03-14 09:42:10,730 WARN [c.c.h.x.r.CitrixResourceBase] > >> (DirectAgent-184:ctx-f12cdd04) (logid:4552cd9d) Unable to start > >> VM(s-294-VM) on host(b8324267-2cbe-4aa9-9b26-215375cb5553) due to Task > >> failed! Task record: uuid: 84f3483d-1021-22fb-be39-2eb15366e03e > >> nameLabel: Async.VM.start_on > >> nameDescription: > >> allowedOperations: [] > >> currentOperations: {} > >> created: Tue Mar 14 09:42:46 BRT 2017 > >> finished: Tue Mar 14 09:42:46 BRT 2017 > >> status: failure > >> residentOn: com.xensource.xenapi.Host@4e05ddb9 > >> progress: 1.0 > >> type: <none/> > >> result: > >> errorInfo: [HOST_CANNOT_ATTACH_NETWORK, > >> OpaqueRef:3269022e-9b3d-88ca-8fb1-b15f3948b3a9, > >> OpaqueRef:cd7b3c3e-92fd-18a3-eaa7-099aa066bb46] > >> otherConfig: {} > >> subtaskOf: com.xensource.xenapi.Task@aaf13f6f > >> subtasks: [] > >> > >> Task failed! Task record: uuid: 84f3483d-1021-22fb-be39-2eb15366e03e > >> nameLabel: Async.VM.start_on > >> nameDescription: > >> allowedOperations: [] > >> currentOperations: {} > >> created: Tue Mar 14 09:42:46 BRT 2017 > >> finished: Tue Mar 14 09:42:46 BRT 2017 > >> status: failure > >> residentOn: com.xensource.xenapi.Host@4e05ddb9 > >> progress: 1.0 > >> type: <none/> > >> result: > >> errorInfo: [HOST_CANNOT_ATTACH_NETWORK, > >> OpaqueRef:3269022e-9b3d-88ca-8fb1-b15f3948b3a9, > >> OpaqueRef:cd7b3c3e-92fd-18a3-eaa7-099aa066bb46] > >> otherConfig: {} > >> subtaskOf: com.xensource.xenapi.Task@aaf13f6f > >> subtasks: [] > >> > >> at > >> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase. > >> checkForSuccess(CitrixResourceBase.java:471) > >> at > >> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.startVM( > >> CitrixResourceBase.java:4900) > >> at > >> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase. > >> CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:128) > >> at > >> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase. > >> CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:53) > >> at > >> com.cloud.hypervisor.xenserver.resource.wrapper. > >> xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122) > >> at > >> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase. > >> > >> executeRequest( > >> > >> CitrixResourceBase.java:1687) > >> at > >> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext( > >> DirectAgentAttache.java:315) > >> at > >> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run( > >> ManagedContextRunnable.java:49) > >> at > >> org.apache.cloudstack.managed.context.impl. > >> > >> DefaultManagedContext$1.call( > >> > >> DefaultManagedContext.java:56) > >> at > >> org.apache.cloudstack.managed.context.impl.DefaultManagedContext. > >> callWithContext(DefaultManagedContext.java:103) > >> at > >> org.apache.cloudstack.managed.context.impl.DefaultManagedContext. > >> runWithContext(DefaultManagedContext.java:53) > >> at > >> org.apache.cloudstack.managed.context.ManagedContextRunnable.run( > >> ManagedContextRunnable.java:46) > >> at > >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) > >> at > >> java.util.concurrent.ScheduledThreadPoolExecutor$ > >> ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > >> at > >> java.util.concurrent.ScheduledThreadPoolExecutor$ > >> > >> ScheduledFutureTask.run( > >> > >> ScheduledThreadPoolExecutor.java:293) > >> at > >> java.util.concurrent.ThreadPoolExecutor.runWorker( > >> ThreadPoolExecutor.java:1142) > >> at > >> java.util.concurrent.ThreadPoolExecutor$Worker.run( > >> ThreadPoolExecutor.java:617) > >> at java.lang.Thread.run(Thread.java:745) > >> > >> -- > >> FABIANO A. SOUZA > >> _Assistência e Consultoria em TI_ > >> f.so...@fdsouza.com.br > >> http://www.fdsouza.com.br > -- Rafael Weingärtner