On Fri, Aug 19, 2016 at 12:07 PM, Carlos Rodrigues <c...@eurotux.com> wrote: > On Fri, 2016-08-19 at 10:47 +0100, Carlos Rodrigues wrote: >> On Fri, 2016-08-19 at 11:36 +0200, Simone Tiraboschi wrote: >> > >> > >> > >> > On Fri, Aug 19, 2016 at 11:29 AM, Carlos Rodrigues <c...@eurotux.co >> > m> >> > wrote: >> > > >> > > After night, the OVF_STORE it was created: >> > > >> > >> > It's quite strange that it got so long but now it looks fine. >> > >> > If the ISO_DOMAIN that I see in your screenshot is served by the >> > engine VM itself, I suggest to remove it and export from an >> > external >> > server. >> > Serving the ISO storage domain from the engine VM itself is not a >> > good idea since when the engine VM is down you can experiment long >> > delays before getting the engine VM restarted due to the >> > unavailable >> > storage domain. >> >> Ok, thank you for advice. >> >> Now, apparently is all ok. I'll do more tests with HA and any issue >> i'll tell you. >> >> Thank you for your support. >> >> Regards, >> Carlos Rodrigues >> > > I shutdown the network of host with engine VM and i expected that other > host fence the host and start engine VM but i don't see any fence > action and the "free" host keep trying to start VM but get and error of > sanlock > > Aug 19 11:03:03 ied-blade11.install.eurotux.local kernel: qemu-kvm: > sending ioctl 5326 to a partition! > Aug 19 11:03:03 ied-blade11.install.eurotux.local kernel: qemu-kvm: > sending ioctl 80200204 to a partition! > Aug 19 11:03:03 ied-blade11.install.eurotux.local kvm[7867]: 1 guest > now active > Aug 19 11:03:03 ied-blade11.install.eurotux.local sanlock[884]: 2016- > 08-19 11:03:03+0100 1023 [903]: r3 paxos_acquire owner 1 delta 1 9 > 245502 alive > Aug 19 11:03:03 ied-blade11.install.eurotux.local sanlock[884]: 2016- > 08-19 11:03:03+0100 1023 [903]: r3 acquire_token held error -243 > Aug 19 11:03:03 ied-blade11.install.eurotux.local sanlock[884]: 2016- > 08-19 11:03:03+0100 1023 [903]: r3 cmd_acquire 2,9,7862 acquire_token > -243 lease owned by other host > Aug 19 11:03:03 ied-blade11.install.eurotux.local libvirtd[1369]: > resource busy: Failed to acquire lock: error -243 > Aug 19 11:03:03 ied-blade11.install.eurotux.local kernel: ovirtmgmt: > port 2(vnet0) entered disabled state > Aug 19 11:03:03 ied-blade11.install.eurotux.local kernel: device vnet0 > left promiscuous mode > Aug 19 11:03:03 ied-blade11.install.eurotux.local kernel: ovirtmgmt: > port 2(vnet0) entered disabled state > Aug 19 11:03:03 ied-blade11.install.eurotux.local kvm[7885]: 0 guests > now active > Aug 19 11:03:03 ied-blade11.install.eurotux.local systemd- > machined[7863]: Machine qemu-4-HostedEngine terminated.
Maybe you hit this one: https://bugzilla.redhat.com/show_bug.cgi?id=1322849 Can you please check it as described in comment 28 and eventually apply the workaround in comment 18? >> > > Regards, >> > > Carlos Rodrigues >> > > >> > > On Fri, 2016-08-19 at 08:29 +0200, Simone Tiraboschi wrote: >> > > > >> > > > >> > > > >> > > > On Thu, Aug 18, 2016 at 6:38 PM, Carlos Rodrigues <cmar@eurotux >> > > > .c >> > > > om> wrote: >> > > > > >> > > > > On Thu, 2016-08-18 at 17:45 +0200, Simone Tiraboschi wrote: >> > > > > > >> > > > > > On Thu, Aug 18, 2016 at 5:43 PM, Carlos Rodrigues <cmar@eur >> > > > > > ot >> > > > > > ux.com> wrote: >> > > > > > > >> > > > > > > >> > > > > > > I increase hosted_engine disk space to 160G. How do i >> > > > > > > force >> > > > > > > to create >> > > > > > > OVF_STORE. >> > > > > > >> > > > > > I think that restarting the engine on the engine VM will >> > > > > > trigger it >> > > > > > although I'm not sure that it was a size issue. >> > > > > > >> > > > > >> > > > > I found to OVF_STORE on another storage domain with "Domain >> > > > > Type" "Data (Master)" >> > > > > >> > > > > >> > > > >> > > > Each storage domain has its own OVF_STORE volumes; you should >> > > > get >> > > > them also on the hosted-engine storage domain. >> > > > Not really sure about how to trigger it again; adding Roy here. >> > > > >> > > > >> > > > >> > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > > >> > > > > > > >> > > > > > > >> > > > > > > Regards, >> > > > > > > Carlos Rodrigues >> > > > > > > >> > > > > > > On Thu, 2016-08-18 at 12:14 +0100, Carlos Rodrigues >> > > > > > > wrote: >> > > > > > > > >> > > > > > > > >> > > > > > > > On Thu, 2016-08-18 at 12:34 +0200, Simone Tiraboschi >> > > > > > > > wrote: >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > On Thu, Aug 18, 2016 at 12:11 PM, Carlos Rodrigues >> > > > > > > > > <cma >> > > > > > > > > r...@eurotux.co >> > > > > > > > > m> >> > > > > > > > > wrote: >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > On Thu, 2016-08-18 at 11:53 +0200, Simone >> > > > > > > > > > Tiraboschi >> > > > > > > > > > wrote: >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > On Thu, Aug 18, 2016 at 11:50 AM, Carlos >> > > > > > > > > > > Rodrigues >> > > > > > > > > > > <cmar@eurotu >> > > > > > > > > > > x. >> > > > > > > > > > > com> >> > > > > > > > > > > wrote: >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > On Thu, 2016-08-18 at 11:42 +0200, Simone >> > > > > > > > > > > > Tiraboschi wrote: >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > On Thu, Aug 18, 2016 at 11:25 AM, Carlos >> > > > > > > > > > > > > Rodrigues <cmar@eu >> > > > > > > > > > > > > ro >> > > > > > > > > > > > > tux. >> > > > > > > > > > > > > com> wrote: >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > On Thu, 2016-08-18 at 11:04 +0200, Simone >> > > > > > > > > > > > > > Tiraboschi >> > > > > > > > > > > > > > wrote: >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > On Thu, Aug 18, 2016 at 10:36 AM, Carlos >> > > > > > > > > > > > > > > Rodrigues >> > > > > > > > > > > > > > > <cmar@ >> > > > > > > > > > > > > > > euro >> > > > > > > > > > > > > > > tux.com> >> > > > > > > > > > > > > > > wrote: >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > On Thu, 2016-08-18 at 10:27 +0200, >> > > > > > > > > > > > > > > > Simone >> > > > > > > > > > > > > > > > Tiraboschi >> > > > > > > > > > > > > > > > wrote: >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > On Thu, Aug 18, 2016 at 10:22 AM, >> > > > > > > > > > > > > > > > > Carlos Rodrigues >> > > > > > > > > > > > > > > > > <cmar@ >> > > > > > > > > > > > > > > > > eurotux. >> > > > > > > > > > > > > > > > > com> >> > > > > > > > > > > > > > > > > wrote: >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > On Thu, 2016-08-18 at 08:54 +0200, >> > > > > > > > > > > > > > > > > > Simone >> > > > > > > > > > > > > > > > > > Tiraboschi >> > > > > > > > > > > > > > > > > > wrote: >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > On Tue, Aug 16, 2016 at 12:53 PM, >> > > > > > > > > > > > > > > > > > > Carlos >> > > > > > > > > > > > > > > > > > > Rodrigues <c >> > > > > > > > > > > > > > > > > > > mar@euro >> > > > > > > > > > > > > > > > > > > tux. >> > > > > > > > > > > > > > > > > > > com> >> > > > > > > > > > > > > > > > > > > wrote: >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > On Sun, 2016-08-14 at 14:22 >> > > > > > > > > > > > > > > > > > > > +0300, Roy Golan >> > > > > > > > > > > > > > > > > > > > wrote: >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > On 12 August 2016 at 20:23, >> > > > > > > > > > > > > > > > > > > > > Carlos >> > > > > > > > > > > > > > > > > > > > > Rodrigues >> > > > > > > > > > > > > > > > > > > > > <cma >> > > > > > > > > > > > > > > > > > > > > r@eurotu >> > > > > > > > > > > > > > > > > > > > > x.co >> > > > > > > > > > > > > > > > > > > > > m> >> > > > > > > > > > > > > > > > > > > > > wrote: >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > Hello, >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > I have one cluster with two >> > > > > > > > > > > > > > > > > > > > > > hosts with >> > > > > > > > > > > > > > > > > > > > > > power >> > > > > > > > > > > > > > > > > > > > > > management >> > > > > > > > > > > > > > > > > > > > > > correctly >> > > > > > > > > > > > > > > > > > > > > > configured and one virtual >> > > > > > > > > > > > > > > > > > > > > > machine with >> > > > > > > > > > > > > > > > > > > > > > HostedEngine >> > > > > > > > > > > > > > > > > > > > > > over >> > > > > > > > > > > > > > > > > > > > > > shared >> > > > > > > > > > > > > > > > > > > > > > storage with FiberChannel. >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > When i shutdown the network >> > > > > > > > > > > > > > > > > > > > > > of host with >> > > > > > > > > > > > > > > > > > > > > > HostedEngine >> > > > > > > > > > > > > > > > > > > > > > VM, it >> > > > > > > > > > > > > > > > > > > > > > should be >> > > > > > > > > > > > > > > > > > > > > > possible the HostedEngine >> > > > > > > > > > > > > > > > > > > > > > VM >> > > > > > > > > > > > > > > > > > > > > > migrate >> > > > > > > > > > > > > > > > > > > > > > automatically to >> > > > > > > > > > > > > > > > > > > > > > another >> > > > > > > > > > > > > > > > > > > > > > host? >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > migrate on which network? >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > What is the expected >> > > > > > > > > > > > > > > > > > > > > > behaviour on this HA >> > > > > > > > > > > > > > > > > > > > > > scenario? >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > After a few minutes your vm >> > > > > > > > > > > > > > > > > > > > > will be >> > > > > > > > > > > > > > > > > > > > > shutdown >> > > > > > > > > > > > > > > > > > > > > by >> > > > > > > > > > > > > > > > > > > > > the High >> > > > > > > > > > > > > > > > > > > > > Availability >> > > > > > > > > > > > > > > > > > > > > agent, as it can't see >> > > > > > > > > > > > > > > > > > > > > network, >> > > > > > > > > > > > > > > > > > > > > and started >> > > > > > > > > > > > > > > > > > > > > on >> > > > > > > > > > > > > > > > > > > > > another >> > > > > > > > > > > > > > > > > > > > > host. >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > I'm testing this scenario and >> > > > > > > > > > > > > > > > > > > > after shutdown >> > > > > > > > > > > > > > > > > > > > network, it >> > > > > > > > > > > > > > > > > > > > should >> > > > > > > > > > > > > > > > > > > > be >> > > > > > > > > > > > > > > > > > > > expected that agent shutdown ha >> > > > > > > > > > > > > > > > > > > > and started >> > > > > > > > > > > > > > > > > > > > on >> > > > > > > > > > > > > > > > > > > > another >> > > > > > > > > > > > > > > > > > > > host, >> > > > > > > > > > > > > > > > > > > > but >> > > > > > > > > > > > > > > > > > > > after >> > > > > > > > > > > > > > > > > > > > couple minutes nothing happens >> > > > > > > > > > > > > > > > > > > > and on host >> > > > > > > > > > > > > > > > > > > > with >> > > > > > > > > > > > > > > > > > > > network we >> > > > > > > > > > > > > > > > > > > > getting >> > > > > > > > > > > > > > > > > > > > the >> > > > > > > > > > > > > > > > > > > > following messages: >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > Aug 16 11:44:08 ied- >> > > > > > > > > > > > > > > > > > > > blade11.install.eurotux.local >> > > > > > > > > > > > > > > > > > > > ovirt-ha- >> > > > > > > > > > > > > > > > > > > > agent[2779]: >> > > > > > > > > > > > > > > > > > > > ovirt-ha-agent >> > > > > > > > > > > > > > > > > > > > ovirt_hosted_engine_ha.agent.ho >> > > > > > > > > > > > > > > > > > > > st >> > > > > > > > > > > > > > > > > > > > ed_engine.Ho >> > > > > > > > > > > > > > > > > > > > st >> > > > > > > > > > > > > > > > > > > > edEn >> > > > > > > > > > > > > > > > > > > > gine.con >> > > > > > > > > > > > > > > > > > > > fig >> > > > > > > > > > > > > > > > > > > > ERROR >> > > > > > > > > > > > > > > > > > > > Unable to get vm.conf from >> > > > > > > > > > > > > > > > > > > > OVF_STORE, falling >> > > > > > > > > > > > > > > > > > > > back >> > > > > > > > > > > > > > > > > > > > to >> > > > > > > > > > > > > > > > > > > > initial >> > > > > > > > > > > > > > > > > > > > vm.conf >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > I think the HA agent its trying >> > > > > > > > > > > > > > > > > > > > to get vm >> > > > > > > > > > > > > > > > > > > > configuration but >> > > > > > > > > > > > > > > > > > > > some >> > > > > > > > > > > > > > > > > > > > how it >> > > > > > > > > > > > > > > > > > > > can't get vm.conf to start VM. >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > No, this is a different issues. >> > > > > > > > > > > > > > > > > > > In 3.6 we added a feature to let >> > > > > > > > > > > > > > > > > > > the engine >> > > > > > > > > > > > > > > > > > > manage >> > > > > > > > > > > > > > > > > > > also the >> > > > > > > > > > > > > > > > > > > engine VM >> > > > > > > > > > > > > > > > > > > itself; ovirt-ha-agent will >> > > > > > > > > > > > > > > > > > > pickup >> > > > > > > > > > > > > > > > > > > the latest >> > > > > > > > > > > > > > > > > > > engine >> > > > > > > > > > > > > > > > > > > VM >> > > > > > > > > > > > > > > > > > > configuration >> > > > > > > > > > > > > > > > > > > from the OVF_STORE which is >> > > > > > > > > > > > > > > > > > > managed >> > > > > > > > > > > > > > > > > > > by the >> > > > > > > > > > > > > > > > > > > engine. >> > > > > > > > > > > > > > > > > > > If something goes wrong, ovirt- >> > > > > > > > > > > > > > > > > > > ha- >> > > > > > > > > > > > > > > > > > > agent could >> > > > > > > > > > > > > > > > > > > fallback to the >> > > > > > > > > > > > > > > > > > > initial >> > > > > > > > > > > > > > > > > > > (bootstrap time) vm.conf. This >> > > > > > > > > > > > > > > > > > > will >> > > > > > > > > > > > > > > > > > > normally >> > > > > > > > > > > > > > > > > > > happen >> > > > > > > > > > > > > > > > > > > till you >> > > > > > > > > > > > > > > > > > > add >> > > > > > > > > > > > > > > > > > > your >> > > > > > > > > > > > > > > > > > > first regular storage domain and >> > > > > > > > > > > > > > > > > > > the engine >> > > > > > > > > > > > > > > > > > > imports >> > > > > > > > > > > > > > > > > > > the >> > > > > > > > > > > > > > > > > > > engine >> > > > > > > > > > > > > > > > > > > VM. >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > But i already have my first storage >> > > > > > > > > > > > > > > > > > domain and >> > > > > > > > > > > > > > > > > > storage >> > > > > > > > > > > > > > > > > > engine >> > > > > > > > > > > > > > > > > > domain >> > > > > > > > > > > > > > > > > > and already imported engine VM. >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > I'm using 4.0 version. >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > This seams an issue, can you please >> > > > > > > > > > > > > > > > > share your >> > > > > > > > > > > > > > > > > /var/log/ovirt-hosted-engine- >> > > > > > > > > > > > > > > > > ha/agent.log ? >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > I sent it in attachment. >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > Nothing strange here; >> > > > > > > > > > > > > > > do you see a couple of disks with alias >> > > > > > > > > > > > > > > OVF_STORE on >> > > > > > > > > > > > > > > the >> > > > > > > > > > > > > > > hosted- >> > > > > > > > > > > > > > > engine >> > > > > > > > > > > > > > > storage domain if you check it from the >> > > > > > > > > > > > > > > engine? >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > Do you mean any disk label? >> > > > > > > > > > > > > > I don't have it anyone: >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > [root@ied-blade11 ~]# ls /dev/disk/by- >> > > > > > > > > > > > > > label/ >> > > > > > > > > > > > > > ls: cannot access /dev/disk/by-label/: No >> > > > > > > > > > > > > > such file or >> > > > > > > > > > > > > > directory >> > > > > > > > > > > > > >> > > > > > > > > > > > > No I mean: go to the engine web-ui, select >> > > > > > > > > > > > > the >> > > > > > > > > > > > > hosted- >> > > > > > > > > > > > > engine >> > > > > > > > > > > > > storage >> > > > > > > > > > > > > domain, check the disks there. >> > > > > > > > > > > > >> > > > > > > > > > > > No, the alias is virtio-disk0. >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > And this is the engine VM disk, so the issue is >> > > > > > > > > > > why >> > > > > > > > > > > the engine >> > > > > > > > > > > has >> > > > > > > > > > > still to create the OVF_STORE. >> > > > > > > > > > > Can you please share your engine.log from the >> > > > > > > > > > > engine VM? >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > Go in attachment. >> > > > > > > > > >> > > > > > > > > The creation of the OVF_STORE disk failed but it's >> > > > > > > > > not >> > > > > > > > > that clear >> > > > > > > > > why: >> > > > > > > > > >> > > > > > > > > 2016-08-17 08:43:33,538 ERROR >> > > > > > > > > [org.ovirt.engine.core.bll.storage.ovfstore.CreateOvf >> > > > > > > > > Vo >> > > > > > > > > lumeForStora >> > > > > > > > > ge >> > > > > > > > > DomainCommand] >> > > > > > > > > (DefaultQuartzScheduler6) [6f1f1fd4] Ending command >> > > > > > > > > 'org.ovirt.engine.core.bll.storage.ovfstore.CreateOvf >> > > > > > > > > Vo >> > > > > > > > > lumeForStora >> > > > > > > > > ge >> > > > > > > > > DomainCommand' >> > > > > > > > > with failure. >> > > > > > > > > 2016-08-17 08:43:33,540 ERROR >> > > > > > > > > [org.ovirt.engine.core.bll.storage.disk.AddDiskComman >> > > > > > > > > d] >> > > > > > > > > (DefaultQuartzScheduler6) [6f1f1fd4] Ending command >> > > > > > > > > 'org.ovirt.engine.core.bll.storage.disk.AddDiskComman >> > > > > > > > > d' >> > > > > > > > > with >> > > > > > > > > failure. >> > > > > > > > > 2016-08-17 08:43:33,541 WARN >> > > > > > > > > [org.ovirt.engine.core.bll.storage.disk.AddDiskComman >> > > > > > > > > d] >> > > > > > > > > (DefaultQuartzScheduler6) [6f1f1fd4] >> > > > > > > > > VmCommand::EndVmCommand: Vm is >> > > > > > > > > null - not performing endAction on Vm >> > > > > > > > > 2016-08-17 08:43:33,553 ERROR >> > > > > > > > > [org.ovirt.engine.core.dal.dbbroker.auditloghandling. >> > > > > > > > > Au >> > > > > > > > > ditLogDirect >> > > > > > > > > or >> > > > > > > > > ] >> > > > > > > > > (DefaultQuartzScheduler6) [6f1f1fd4] Correlation ID: >> > > > > > > > > 6f1f1fd4, Call >> > > > > > > > > Stack: null, Custom Event ID: -1, Message: Add-Disk >> > > > > > > > > operation >> > > > > > > > > failed >> > > > > > > > > to complete. >> > > > > > > > > 2016-08-17 08:43:33,557 WARN >> > > > > > > > > [org.ovirt.engine.core.dal.dbbroker.auditloghandling. >> > > > > > > > > Au >> > > > > > > > > ditLogDirect >> > > > > > > > > or >> > > > > > > > > ] >> > > > > > > > > (DefaultQuartzScheduler6) [] Correlation ID: >> > > > > > > > > 19ac5bda, >> > > > > > > > > Call Stack: >> > > > > > > > > null, Custom Event ID: -1, Message: Failed to create >> > > > > > > > > OVF store disk >> > > > > > > > > for Storage Domain hosted_storage. >> > > > > > > > > OVF data won't be updated meanwhile for that domain. >> > > > > > > > > 2016-08-17 08:43:33,585 INFO >> > > > > > > > > [org.ovirt.engine.core.bll.SerialChildCommandsExecuti >> > > > > > > > > on >> > > > > > > > > Callback] >> > > > > > > > > (DefaultQuartzScheduler6) [5f5a8daf] Command >> > > > > > > > > 'ProcessOvfUpdateForStorageDomain' (id: >> > > > > > > > > '71aaaafe-7b9e-45e8-a40c-6d33bdf646a0') waiting on >> > > > > > > > > child command >> > > > > > > > > id: >> > > > > > > > > 'eb2e6f1a-c756-4ccd-85a1-60d97d6880de' >> > > > > > > > > type:'CreateOvfVolumeForStorageDomain' to complete >> > > > > > > > > 2016-08-17 08:43:33,595 ERROR >> > > > > > > > > [org.ovirt.engine.core.bll.storage.ovfstore.CreateOvf >> > > > > > > > > Vo >> > > > > > > > > lumeForStora >> > > > > > > > > ge >> > > > > > > > > DomainCommand] >> > > > > > > > > (DefaultQuartzScheduler6) [5d314e49] Ending command >> > > > > > > > > 'org.ovirt.engine.core.bll.storage.ovfstore.CreateOvf >> > > > > > > > > Vo >> > > > > > > > > lumeForStora >> > > > > > > > > ge >> > > > > > > > > DomainCommand' >> > > > > > > > > with failure. >> > > > > > > > > 2016-08-17 08:43:33,596 ERROR >> > > > > > > > > [org.ovirt.engine.core.bll.storage.disk.AddDiskComman >> > > > > > > > > d] >> > > > > > > > > (DefaultQuartzScheduler6) [5d314e49] Ending command >> > > > > > > > > 'org.ovirt.engine.core.bll.storage.disk.AddDiskComman >> > > > > > > > > d' >> > > > > > > > > with >> > > > > > > > > failure. >> > > > > > > > > 2016-08-17 08:43:33,596 WARN >> > > > > > > > > [org.ovirt.engine.core.bll.storage.disk.AddDiskComman >> > > > > > > > > d] >> > > > > > > > > (DefaultQuartzScheduler6) [5d314e49] >> > > > > > > > > VmCommand::EndVmCommand: Vm is >> > > > > > > > > null - not performing endAction on Vm >> > > > > > > > > 2016-08-17 08:43:33,602 ERROR >> > > > > > > > > [org.ovirt.engine.core.dal.dbbroker.auditloghandling. >> > > > > > > > > Au >> > > > > > > > > ditLogDirect >> > > > > > > > > or >> > > > > > > > > ] >> > > > > > > > > (DefaultQuartzScheduler6) [5d314e49] Correlation ID: >> > > > > > > > > 5d314e49, Call >> > > > > > > > > Stack: null, Custom Event ID: -1, Message: Add-Disk >> > > > > > > > > operation >> > > > > > > > > failed >> > > > > > > > > to complete. >> > > > > > > > > 2016-08-17 08:43:33,605 WARN >> > > > > > > > > [org.ovirt.engine.core.dal.dbbroker.auditloghandling. >> > > > > > > > > Au >> > > > > > > > > ditLogDirect >> > > > > > > > > or >> > > > > > > > > ] >> > > > > > > > > (DefaultQuartzScheduler6) [] Correlation ID: >> > > > > > > > > 5f5a8daf, >> > > > > > > > > Call Stack: >> > > > > > > > > null, Custom Event ID: -1, Message: Failed to create >> > > > > > > > > OVF store disk >> > > > > > > > > for Storage Domain hosted_storage. >> > > > > > > > > OVF data won't be updated meanwhile for that domain. >> > > > > > > > > 2016-08-17 08:43:36,460 INFO >> > > > > > > > > [org.ovirt.engine.core.bll.scheduling.HaReservationHa >> > > > > > > > > nd >> > > > > > > > > ling] >> > > > > > > > > (DefaultQuartzScheduler7) [5d314e49] HA reservation >> > > > > > > > > status for >> > > > > > > > > cluster >> > > > > > > > > 'Default' is 'OK' >> > > > > > > > > 2016-08-17 08:43:36,662 INFO >> > > > > > > > > [org.ovirt.engine.core.bll.SerialChildCommandsExecuti >> > > > > > > > > on >> > > > > > > > > Callback] >> > > > > > > > > (DefaultQuartzScheduler4) [5f5a8daf] Command >> > > > > > > > > 'ProcessOvfUpdateForStorageDomain' id: >> > > > > > > > > '71aaaafe-7b9e-45e8-a40c-6d33bdf646a0' child commands >> > > > > > > > > '[84959a4b-6a10-4d22-b37e-6c154e17a0da, >> > > > > > > > > eb2e6f1a-c756-4ccd-85a1-60d97d6880de]' executions >> > > > > > > > > were >> > > > > > > > > completed, >> > > > > > > > > status 'FAILED' >> > > > > > > > > 2016-08-17 08:43:37,691 ERROR >> > > > > > > > > [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOv >> > > > > > > > > fU >> > > > > > > > > pdateForStor >> > > > > > > > > ag >> > > > > > > > > eDomainCommand] >> > > > > > > > > (DefaultQuartzScheduler6) [5f5a8daf] Ending command >> > > > > > > > > 'org.ovirt.engine.core.bll.storage.ovfstore.ProcessOv >> > > > > > > > > fU >> > > > > > > > > pdateForStor >> > > > > > > > > ag >> > > > > > > > > eDomainCommand' >> > > > > > > > > with failure. >> > > > > > > > > >> > > > > > > > > Can you please check vdsm logs for that time frame on >> > > > > > > > > the SPM host? >> > > > > > > > > >> > > > > > > > >> > > > > > > > I sent in attachment the vdsm logs from both hosts, but >> > > > > > > > i >> > > > > > > > think the >> > > > > > > > SPM >> > > > > > > > host on this time frame it was ied-blade13 >> > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > It seams that you also have an issue in the SPM >> > > > > > > > > election procedure: >> > > > > > > > > >> > > > > > > > > 2016-08-17 18:04:31,053 ERROR >> > > > > > > > > [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyDa >> > > > > > > > > ta >> > > > > > > > > ] >> > > > > > > > > (DefaultQuartzScheduler1) [] SPM Init: could not find >> > > > > > > > > reported vds >> > > > > > > > > or >> > > > > > > > > not up - pool: 'Default' vds_spm_id: '2' >> > > > > > > > > 2016-08-17 18:04:31,076 INFO >> > > > > > > > > [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyDa >> > > > > > > > > ta >> > > > > > > > > ] >> > > > > > > > > (DefaultQuartzScheduler1) [] SPM selection - vds >> > > > > > > > > seems >> > > > > > > > > as spm >> > > > > > > > > 'hosted_engine_2' >> > > > > > > > > 2016-08-17 18:04:31,076 WARN >> > > > > > > > > [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyDa >> > > > > > > > > ta >> > > > > > > > > ] >> > > > > > > > > (DefaultQuartzScheduler1) [] spm vds is non >> > > > > > > > > responsive, >> > > > > > > > > stopping >> > > > > > > > > spm >> > > > > > > > > selection. >> > > > > > > > > 2016-08-17 18:04:31,539 INFO >> > > > > > > > > [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatis >> > > > > > > > > ti >> > > > > > > > > csFetcher] >> > > > > > > > > (DefaultQuartzScheduler7) [] Fetched 1 VMs from VDS >> > > > > > > > > '06372186-572c-41ad-916f-7cbb0aba5302' >> > > > > > > > > >> > > > > > > > > probably due to: >> > > > > > > > > 2016-08-17 18:02:33,569 ERROR >> > > > > > > > > [org.ovirt.engine.core.vdsbroker.monitoring.HostMonit >> > > > > > > > > or >> > > > > > > > > ing] >> > > > > > > > > (DefaultQuartzScheduler6) [] Failure to refresh Vds >> > > > > > > > > runtime info: >> > > > > > > > > VDSGenericException: VDSNetworkException: Message >> > > > > > > > > timeout which can >> > > > > > > > > be >> > > > > > > > > caused by communication issues >> > > > > > > > > 2016-08-17 18:02:33,569 ERROR >> > > > > > > > > [org.ovirt.engine.core.vdsbroker.monitoring.HostMonit >> > > > > > > > > or >> > > > > > > > > ing] >> > > > > > > > > (DefaultQuartzScheduler6) [] Exception: >> > > > > > > > > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkE >> > > > > > > > > xc >> > > > > > > > > eption: >> > > > > > > > > VDSGenericException: VDSNetworkException: Message >> > > > > > > > > timeout which can >> > > > > > > > > be >> > > > > > > > > caused by communication issues >> > > > > > > > > >> > > > > > > > >> > > > > > > > This messages maybe cause by connection issues from >> > > > > > > > yesterday at 6pm. >> > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > can you please check if the engine VM could correctly >> > > > > > > > > resolve and >> > > > > > > > > reach each host? >> > > > > > > > >> > > > > > > > Now i can read engine VM from both hosts >> > > > > > > > >> > > > > > > > [root@ied-blade11 ~]# ping ied-hosted-engine >> > > > > > > > PING ied-hosted-engine.install.eurotux.local >> > > > > > > > (10.10.4.115) 56(84) >> > > > > > > > bytes >> > > > > > > > of data. >> > > > > > > > 64 bytes from ied-hosted-engine.install.eurotux.local >> > > > > > > > (10.10.4.115): >> > > > > > > > icmp_seq=1 ttl=64 time=0.179 ms >> > > > > > > > 64 bytes from ied-hosted-engine.install.eurotux.local >> > > > > > > > (10.10.4.115): >> > > > > > > > icmp_seq=2 ttl=64 time=0.141 ms >> > > > > > > > ^C >> > > > > > > > --- ied-hosted-engine.install.eurotux.local ping >> > > > > > > > statistics --- >> > > > > > > > 2 packets transmitted, 2 received, 0% packet loss, time >> > > > > > > > 999ms >> > > > > > > > rtt min/avg/max/mdev = 0.141/0.160/0.179/0.019 ms >> > > > > > > > [root@ied-blade13 ~]# ping ied-hosted-engine >> > > > > > > > PING ied-hosted-engine.install.eurotux.local >> > > > > > > > (10.10.4.115) 56(84) >> > > > > > > > bytes >> > > > > > > > of data. >> > > > > > > > 64 bytes from ied-hosted-engine.install.eurotux.local >> > > > > > > > (10.10.4.115): >> > > > > > > > icmp_seq=1 ttl=64 time=0.172 ms >> > > > > > > > 64 bytes from ied-hosted-engine.install.eurotux.local >> > > > > > > > (10.10.4.115): >> > > > > > > > icmp_seq=2 ttl=64 time=0.169 ms >> > > > > > > > ^C >> > > > > > > > --- ied-hosted-engine.install.eurotux.local ping >> > > > > > > > statistics --- >> > > > > > > > 2 packets transmitted, 2 received, 0% packet loss, time >> > > > > > > > 1000ms >> > > > > > > > rtt min/avg/max/mdev = 0.169/0.170/0.172/0.013 ms >> > > > > > > > >> > > > > > > > >> > > > > > > > I have a message critical of low disk space on >> > > > > > > > hosted_storage domain. >> > > > > > > > I have 50G of disk and i created i VM with 40G. Do i >> > > > > > > > need >> > > > > > > > more space >> > > > > > > > of >> > > > > > > > OVF_STORAGE? >> > > > > > > > What is the minimum requirements of disk space for >> > > > > > > > deploy >> > > > > > > > engine VM? >> > > > > > > > >> > > > > > > > Regards, >> > > > > > > > Carlos >> > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > Regards, >> > > > > > > > > > > > > > > > > > > > Carlos Rodrigues >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > Regards, >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > -- >> > > > > > > > > > > > > > > > > > > > > > Carlos Rodrigues >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > Engenheiro de Software >> > > > > > > > > > > > > > > > > > > > > > Sénior >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > Eurotux Informática, S.A. | >> > > > > > > > > > > > > > > > > > > > > > w >> > > > > > > > > > > > > > > > > > > > > > ww.eurotux.c >> > > > > > > > > > > > > > > > > > > > > > om >> > > > > > > > > > > > > > > > > > > > > > (t) +351 253 680 300 (m) >> > > > > > > > > > > > > > > > > > > > > > +351 >> > > > > > > > > > > > > > > > > > > > > > 911 926 110 >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > ___________________________ >> > > > > > > > > > > > > > > > > > > > > > __ >> > > > > > > > > > > > > > > > > > > > > > ____________ >> > > > > > > > > > > > > > > > > > > > > > __ >> > > > > > > > > > > > > > > > > > > > > > ____ >> > > > > > > > > > > > > > > > > > > > > > Users mailing list >> > > > > > > > > > > > > > > > > > > > > > Users@ovirt.org >> > > > > > > > > > > > > > > > > > > > > > http://lists.ovirt.org/mail >> > > > > > > > > > > > > > > > > > > > > > ma >> > > > > > > > > > > > > > > > > > > > > > n/listinfo/u >> > > > > > > > > > > > > > > > > > > > > > se >> > > > > > > > > > > > > > > > > > > > > > rs >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > -- >> > > > > > > > > > > > > > > > > > > > Carlos Rodrigues >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > Engenheiro de Software Sénior >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > Eurotux Informática, S.A. | >> > > > > > > > > > > > > > > > > > > > www.e >> > > > > > > > > > > > > > > > > > > > urotux.com >> > > > > > > > > > > > > > > > > > > > (t) +351 253 680 300 (m) +351 >> > > > > > > > > > > > > > > > > > > > 911 >> > > > > > > > > > > > > > > > > > > > 926 110 >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > _______________________________ >> > > > > > > > > > > > > > > > > > > > __ >> > > > > > > > > > > > > > > > > > > > ____________ >> > > > > > > > > > > > > > > > > > > > __ >> > > > > > > > > > > > > > > > > > > > Users mailing list >> > > > > > > > > > > > > > > > > > > > Users@ovirt.org >> > > > > > > > > > > > > > > > > > > > http://lists.ovirt.org/mailman/ >> > > > > > > > > > > > > > > > > > > > li >> > > > > > > > > > > > > > > > > > > > stinfo/users >> > > > > > > > > > > > > > > > > > -- >> > > > > > > > > > > > > > > > > > Carlos Rodrigues >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > Engenheiro de Software Sénior >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > Eurotux Informática, S.A. | www.eur >> > > > > > > > > > > > > > > > > > ot >> > > > > > > > > > > > > > > > > > ux.com >> > > > > > > > > > > > > > > > > > (t) +351 253 680 300 (m) +351 911 >> > > > > > > > > > > > > > > > > > 926 >> > > > > > > > > > > > > > > > > > 110 >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > -- >> > > > > > > > > > > > > > > > Carlos Rodrigues >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > Engenheiro de Software Sénior >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > Eurotux Informática, S.A. | www.eurotux >> > > > > > > > > > > > > > > > .c >> > > > > > > > > > > > > > > > om >> > > > > > > > > > > > > > > > (t) +351 253 680 300 (m) +351 911 926 >> > > > > > > > > > > > > > > > 110 >> > > > > > > > > > > > > > -- >> > > > > > > > > > > > > > Carlos Rodrigues >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > Engenheiro de Software Sénior >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > Eurotux Informática, S.A. | www.eurotux.com >> > > > > > > > > > > > > > (t) +351 253 680 300 (m) +351 911 926 110 >> > > > > > > > > > > > > > >> > > > > > > > > > > > -- >> > > > > > > > > > > > Carlos Rodrigues >> > > > > > > > > > > > >> > > > > > > > > > > > Engenheiro de Software Sénior >> > > > > > > > > > > > >> > > > > > > > > > > > Eurotux Informática, S.A. | www.eurotux.com >> > > > > > > > > > > > (t) +351 253 680 300 (m) +351 911 926 110 >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > -- >> > > > > > > > > > Carlos Rodrigues >> > > > > > > > > > >> > > > > > > > > > Engenheiro de Software Sénior >> > > > > > > > > > >> > > > > > > > > > Eurotux Informática, S.A. | www.eurotux.com >> > > > > > > > > > (t) +351 253 680 300 (m) +351 911 926 110 >> > > > > > > > _______________________________________________ >> > > > > > > > Users mailing list >> > > > > > > > Users@ovirt.org >> > > > > > > > http://lists.ovirt.org/mailman/listinfo/users >> > > > > > > -- >> > > > > > > Carlos Rodrigues >> > > > > > > >> > > > > > > Engenheiro de Software Sénior >> > > > > > > >> > > > > > > Eurotux Informática, S.A. | www.eurotux.com >> > > > > > > (t) +351 253 680 300 (m) +351 911 926 110 >> > > > > > > >> > > > > -- >> > > > > Carlos Rodrigues >> > > > > >> > > > > Engenheiro de Software Sénior >> > > > > >> > > > > Eurotux Informática, S.A. | www.eurotux.com >> > > > > (t) +351 253 680 300 (m) +351 911 926 110 >> > > > > >> > > > > >> > > > >> > > -- >> > > Carlos Rodrigues >> > > >> > > Engenheiro de Software Sénior >> > > >> > > Eurotux Informática, S.A. | www.eurotux.com >> > > (t) +351 253 680 300 (m) +351 911 926 110 >> > > >> > > >> > > -- > Carlos Rodrigues > > Engenheiro de Software Sénior > > Eurotux Informática, S.A. | www.eurotux.com > (t) +351 253 680 300 (m) +351 911 926 110 > > _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users