Ahoj Jakub, Just one more idea: could you turn statement logging on in postgresql before you try again? It is in /var/lib/pgsql/data/postgresql.conf log_statement = 'all'
----- Original Message ----- > From: "Jakub Bittner" <j.bitt...@nbu.cz> > To: "Itamar Heim" <ih...@redhat.com> > Cc: users@ovirt.org > Sent: Tuesday, August 13, 2013 5:20:29 PM > Subject: Re: [Users] All VMs disappeared > > Dne 13.8.2013 17:02, Itamar Heim napsal(a): > > On 08/13/2013 05:31 PM, Jakub Bittner wrote: > >> Dne 13.8.2013 15:39, Itamar Heim napsal(a): > >>> On 08/13/2013 08:50 AM, Jakub Bittner wrote: > >>>> Dne 12.8.2013 22:44, Itamar Heim napsal(a): > >>>>> On 08/12/2013 06:59 PM, Jakub Bittner wrote: > >>>>>> Dne 12.8.2013 14:54, Jakub Bittner napsal(a): > >>>>>>> Dne 12.8.2013 14:29, Laszlo Hornyak napsal(a): > >>>>>>>> I looked around Noam's patch and that should not cause such > >>>>>>>> behavior. > >>>>>>>> I am wondering how that lost VM's could happen. > >>>>>>>> > >>>>>>>> Jakub, can you give a more detailed description what you were > >>>>>>>> doing > >>>>>>>> with oVirt when this happened? Maybe the bug is still there. > >>>>>>>> > >>>>>>>> Thank you, > >>>>>>>> Laszlo > >>>>>>>> > >>>>>>>> ----- Original Message ----- > >>>>>>>>> From: "Greg Sheremeta" <gsher...@redhat.com> > >>>>>>>>> To: "Laszlo Hornyak" <lhorn...@redhat.com> > >>>>>>>>> Cc: "Jakub Bittner" <j.bitt...@nbu.cz>, "Noam Slomianko" > >>>>>>>>> <nslom...@redhat.com>, users@ovirt.org > >>>>>>>>> Sent: Monday, August 12, 2013 1:23:40 PM > >>>>>>>>> Subject: Re: [Users] All VMs disappeared > >>>>>>>>> > >>>>>>>>> Not the one I fixed, 987907. It was a simple UI > >>>>>>>>> NullPointerException. > >>>>>>>>> > >>>>>>>>> Greg > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> ----- Original Message ----- > >>>>>>>>>> From: "Laszlo Hornyak" <lhorn...@redhat.com> > >>>>>>>>>> To: "Jakub Bittner" <j.bitt...@nbu.cz>, "Noam Slomianko" > >>>>>>>>>> <nslom...@redhat.com>, "Greg Sheremeta" > >>>>>>>>>> <gsher...@redhat.com> > >>>>>>>>>> Cc: users@ovirt.org > >>>>>>>>>> Sent: Monday, August 12, 2013 7:21:26 AM > >>>>>>>>>> Subject: Re: [Users] All VMs disappeared > >>>>>>>>>> > >>>>>>>>>> Well if they are no longer in DB then that explains why the > >>>>>>>>>> exception no > >>>>>>>>>> longer occurs, but at the cost of database corruption. > >>>>>>>>>> Noam and Greg, can these bugs cause data corruption? > >>>>>>>>>> > >>>>>>>>>> Thank you, > >>>>>>>>>> Laszlo > >>>>>>>>>> > >>>>>>>>>> ----- Original Message ----- > >>>>>>>>>>> From: "Jakub Bittner" <j.bitt...@nbu.cz> > >>>>>>>>>>> To: "Greg Sheremeta" <gsher...@redhat.com> > >>>>>>>>>>> Cc: users@ovirt.org, "Laszlo Hornyak" <lhorn...@redhat.com> > >>>>>>>>>>> Sent: Monday, August 12, 2013 9:05:09 AM > >>>>>>>>>>> Subject: Re: [Users] All VMs disappeared > >>>>>>>>>>> > >>>>>>>>>>> Dne 10.8.2013 01:54, Greg Sheremeta napsal(a): > >>>>>>>>>>>> It could also be this bug[1], for which I just submitted a > >>>>>>>>>>>> fix. > >>>>>>>>>>>> > >>>>>>>>>>>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=987907 > >>>>>>>>>>>> > >>>>>>>>>>>> You can work around it by typing just "VMs:" (without the > >>>>>>>>>>>> quotes) in > >>>>>>>>>>>> the > >>>>>>>>>>>> search bar. > >>>>>>>>>>>> > >>>>>>>>>>>> Greg > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> ----- Original Message ----- > >>>>>>>>>>>>> From: "Laszlo Hornyak" <lhorn...@redhat.com> > >>>>>>>>>>>>> To: "Jakub Bittner" <j.bitt...@nbu.cz> > >>>>>>>>>>>>> Cc: users@ovirt.org > >>>>>>>>>>>>> Sent: Friday, August 9, 2013 11:21:26 AM > >>>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared > >>>>>>>>>>>>> > >>>>>>>>>>>>> Hi Jakub, > >>>>>>>>>>>>> > >>>>>>>>>>>>> Could you check through DB or REST-API if the VM's are in > >>>>>>>>>>>>> your > >>>>>>>>>>>>> DB? > >>>>>>>>>>>>> select * from vm_static; > >>>>>>>>>>>>> or > >>>>>>>>>>>>> curl -u admin@internal:blablabla [engine-url]api/vms > >>>>>>>>>>>>> > >>>>>>>>>>>>> It seems Noam fixed this issue already in > >>>>>>>>>>>>> c2295c31fa645e1ba1b94cd557bd1fecb40c8829. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Thank you, > >>>>>>>>>>>>> Laszlo > >>>>>>>>>>>>> > >>>>>>>>>>>>> ----- Original Message ----- > >>>>>>>>>>>>>> From: "Jakub Bittner" <j.bitt...@nbu.cz> > >>>>>>>>>>>>>> To: users@ovirt.org > >>>>>>>>>>>>>> Sent: Friday, August 9, 2013 1:06:53 PM > >>>>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Dne 9.8.2013 09:48, Jakub Bittner napsal(a): > >>>>>>>>>>>>>>> Hello, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Iam running ovirt 3.3.0.beta1 on centos 6,4 and all our VMs > >>>>>>>>>>>>>>> disappeared from VMs tab. Nodes running on centos too. > >>>>>>>>>>>>>>> Repeating > >>>>>>>>>>>>>>> problem in log is: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> 2013-08-09 09:44:24,203 WARN > >>>>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager] > >>>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-45) Failed to refresh VDS , > >>>>>>>>>>>>>>> vds = > >>>>>>>>>>>>>>> 7cb6aedf-47bc-40b0-877f-2a537fca5c64 : node2.x.com, error = > >>>>>>>>>>>>>>> java.lang.NullPointerException, continuing.: > >>>>>>>>>>>>>>> java.lang.NullPointerException > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> [vdsbroker.jar:] > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> [vdsbroker.jar:] > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> [vdsbroker.jar:] > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> [vdsbroker.jar:] > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> [vdsbroker.jar:] > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown > >>>>>>>>>>>>>>> Source) [:1.7.0_25] > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> [rt.jar:1.7.0_25] > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>> java.lang.reflect.Method.invoke(Method.java:606) > >>>>>>>>>>>>>>> [rt.jar:1.7.0_25] > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> [scheduler.jar:] > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213) > >>>>>>>>>>>>>>> [quartz.jar:] > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> [quartz.jar:] > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> 2013-08-09 09:44:27,039 WARN > >>>>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager] > >>>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-47) Failed to refresh VDS , > >>>>>>>>>>>>>>> vds = > >>>>>>>>>>>>>>> 653e34a6-6b7e-4757-9042-dc02e7a0cddf : node3.x.com, error = > >>>>>>>>>>>>>>> java.lang.NullPointerException, continuing.: > >>>>>>>>>>>>>>> java.lang.NullPointerException > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> [vdsbroker.jar:] > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> [vdsbroker.jar:] > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> [vdsbroker.jar:] > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> [vdsbroker.jar:] > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> [vdsbroker.jar:] > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown > >>>>>>>>>>>>>>> Source) [:1.7.0_25] > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> [rt.jar:1.7.0_25] > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>> java.lang.reflect.Method.invoke(Method.java:606) > >>>>>>>>>>>>>>> [rt.jar:1.7.0_25] > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> [scheduler.jar:] > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213) > >>>>>>>>>>>>>>> [quartz.jar:] > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> [quartz.jar:] > >>>>>>>>>>>>>>> _______________________________________________ > >>>>>>>>>>>>>>> Users mailing list > >>>>>>>>>>>>>>> Users@ovirt.org > >>>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users > >>>>>>>>>>>>>> I restarted management node and both nodes and this problem > >>>>>>>>>>>>>> disappeared. > >>>>>>>>>>>>>> But I still can not see any VMs. > >>>>>>>>>>>>>> _______________________________________________ > >>>>>>>>>>>>>> Users mailing list > >>>>>>>>>>>>>> Users@ovirt.org > >>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users > >>>>>>>>>>>>>> > >>>>>>>>>>>>> _______________________________________________ > >>>>>>>>>>>>> Users mailing list > >>>>>>>>>>>>> Users@ovirt.org > >>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users > >>>>>>>>>>>>> > >>>>>>>>>>> Hi, > >>>>>>>>>>> > >>>>>>>>>>> I do not have them in database. Both commands shows nothing > >>>>>>>>>>> > >>>>>>> Hello, > >>>>>>> > >>>>>>> we run ovirt 3.2.1 (or 2). I upgraded to RC1, everything was great > >>>>>>> (except SPICE. VNC and creating new WM due to serialization bug.). > >>>>>>> Because those bugs were irritating, we switched to nightly. Still > >>>>>>> works fine (but bug with VMs is not fixed there), than I installed > >>>>>>> new > >>>>>>> ovirt engine management server in KVM virtualization (using > >>>>>>> libvirtd) > >>>>>>> due to taking snapshots before deploying nightly builds. New > >>>>>>> virtualized engine works great and I delete old one. Next day I > >>>>>>> came > >>>>>>> to work and engine web page was very very slow, every operation > >>>>>>> takes > >>>>>>> ages, that time I found there were no VMs, no HDDs and so. I > >>>>>>> converted > >>>>>>> its qcow2 image to raw, because I think it has slow DB because of > >>>>>>> running i qcow2. Did not helped, than I take a look in nodes > >>>>>>> logs and > >>>>>>> found this problem, which I posted in first email (VMs still > >>>>>>> running, > >>>>>>> but not visible in engine VMs tab, but on host I can see number of > >>>>>>> running VMs). After restart bug disappeared, but VMs too. > >>>>>>> Today I reinstalled engine with oVirt 3.3.0rc1. But all my VMs are > >>>>>>> gone. Even from storage. (but disappearing from storage may be my > >>>>>>> fault during recovery). > >>>>>>> > >>>>>>> I can provide logs from both two nodes and (hopefully) from engine > >>>>>>> too. I can provide you with those logs, but not publicly. Sorry. > >>>>>>> > >>>>>>> Thank you, > >>>>>>> Jakub Bittner, NSA Czech > >>>>>>> _______________________________________________ > >>>>>>> Users mailing list > >>>>>>> Users@ovirt.org > >>>>>>> http://lists.ovirt.org/mailman/listinfo/users > >>>>>> > >>>>>> > >>>>>> Well I reinstalled to 3.3.0 RC1, rebooted server and my VMs are gone > >>>>>> again. > >>>>>> _______________________________________________ > >>>>>> Users mailing list > >>>>>> Users@ovirt.org > >>>>>> http://lists.ovirt.org/mailman/listinfo/users > >>>>> > >>>>> can you verify the db from before the upgrade has the VMs? > >>>> > >>>> I dont know what you mean by verify database. Please, could you > >>>> point me > >>>> how to do it? Thank you > >>>> _______________________________________________ > >>>> Users mailing list > >>>> Users@ovirt.org > >>>> http://lists.ovirt.org/mailman/listinfo/users > >>> > >>> I assume from your expectation to see the VMs from the previous > >>> install that you upgraded your system rather than do a clean install? > >>> in an upgrade, there should be a backup of the db from before the > >>> upgrade. > >>> (in a clean install, engine is not aware of VMs running on hosts from > >>> previous installs - we need to cover "import existing storage domain" > >>> in a future version to close this gap. current alternatives around > >>> this are to convert the storage domain to an export domain (for an nfs > >>> data domain) an re-import them to the system or a few other options. > >>> > >>> how many VMs are there? > >> They(VMs) are deleted. So, my situation is this: > >> > >> I have clean install of ovirt 3.3.0beta with imported VMs from export > >> domain. Everything works except snapshots. But when I restart ovirt mgmt > >> server, VMs will disappear. Right now I am doing hourly backup of DB. > > > > this is very fishy... > > can you arrange for a remote access session for someone to take a look? > > > >> > >> I can not use workaround with converting domain to export domain, > >> because we use iscsi. It would be perfect if I can retake existing > >> domain. Another gap is if I dont detach export domain I can not attach > >> it to new datacenter and I have to manually edit some files. It would be > >> absolutely perfect to do it from ovirt web gui (for example by "force > >> override button") > > > > i thought 3.3 allowed for that, but i could be wrong. > > > Well, I will backup all VMs to export domain, than create snapshot of > ovirt and I will try to restart that server if its going to delete VMs > from DB. Than I will report back. I think, that remote session will not > be possible, but I can be wrong (I hope). > > > _______________________________________________ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users