[ovirt-users] Recovering from a multi-node failure

[Jim Kusznir]

Hi all:

Today has been rough.  two of my three nodes went down today, and self heal
has not been healing well.  4 hours later, VMs are running.  but the engine
is not happy.  It claims the storage domain is down (even though it is up
on all hosts and VMs are running).  I'm getting a ton of these messages
logging:

VDSM engine3 command HSMGetAllTasksStatusesVDS failed: Not SPM

Aug 4, 2017 7:23:00 PM

VDSM engine3 command SpmStatusVDS failed: Error validating master storage
domain: ('MD read error',)

Aug 4, 2017 7:22:49 PM

VDSM engine3 command ConnectStoragePoolVDS failed: Cannot find master
domain: u'spUUID=5868392a-0148-02cf-014d-0121,
msdUUID=cdaf180c-fde6-4cb3-b6e5-b6bd869c8770'

Aug 4, 2017 7:22:47 PM

VDSM engine1 command ConnectStoragePoolVDS failed: Cannot find master
domain: u'spUUID=5868392a-0148-02cf-014d-0121,
msdUUID=cdaf180c-fde6-4cb3-b6e5-b6bd869c8770'

Aug 4, 2017 7:22:46 PM

VDSM engine2 command SpmStatusVDS failed: Error validating master storage
domain: ('MD read error',)

Aug 4, 2017 7:22:44 PM

VDSM engine2 command ConnectStoragePoolVDS failed: Cannot find master
domain: u'spUUID=5868392a-0148-02cf-014d-0121,
msdUUID=cdaf180c-fde6-4cb3-b6e5-b6bd869c8770'

Aug 4, 2017 7:22:42 PM

VDSM engine1 command HSMGetAllTasksStatusesVDS failed: Not SPM: ()



I cannot set an SPM as it claims the storage domain is down; I cannot set
the storage domain up.

Also in the storage realm, one of my exports shows substantially less data
than is actually there.

Here's what happened, as best as I understood them:
I went to do maintence on ovirt2 (needed to replace a faulty ram stick and
rework the disk).  I put it in maintence mode, then shut it down and did my
work.  In the process, much of the disk contents was lost (all the gluster
data).  I figure, no big deal, the gluster data is redundant on the
network, it will heal when it comes back up.

While I was doing maintence, all but one of the VMs were running on
engine1.  When I turned on engine2, all of the sudden, all vms including
the main engine stop and go non-responsive.  As far as I can tell, this
should not have happened, as I turned ON one host, but none the less, I
waited for recovery to occur (while customers started calling asking why
everything stopped working).  As I waited, I  was checking, and gluster
volume status only showed ovirt1 and ovirt2Apparently gluster had
stopped/failed at some point on ovirt3.  I assume that was the cause of the
outage, still, if everything was working fine with ovirt1 gluster, and
ovirt2 powers on with a very broke gluster (the volume status was showing
NA for the port fileds for the gluster volumes), I would not expect to have
a working gluster go stupid like that.

After starting ovirt3 glusterd and checking the status, all three showed
ovirt1 and ovirt3 as operational, and ovirt2 as NA.  Unfortunately,
recovery was still not happening, so I did some googling and found about
the commands to inquire about the hosted-engine status.  It appeared to be
stuck "paused" and I couldn't find a way to unpause it, so I poweroff'ed
it, then started it manually on engine 1, and the cluster came back up.  It
showed all VMs paused.  I was able to unpause them and they worked again.

So now I began to work the ovirt2 gluster healing problem.  It didn't
appear to be self-healing, but eventually I found this document:
https://support.rackspace.com/how-to/recover-from-a-failed-server-in-a-glusterfs-array/
and from that found the magic xattr commands.  After setting them, gluster
volumes on ovirt2 came online.  I told iso to heal, and it did but only
came up about half as much data as it should have.  I told it heal full,
and it did finish off the remaining data, and came up to full.  I then told
engine to do a full heal (gluster volume heal engine full), and it
transferred its data from the other gluster hosts too.  However, it said it
was done when it hit 9.7GB while there was 15GB on disk!  It is still stuck
that way; ovirt gui and gluster volume heal engine info both show the
volume fully healed, but it is not:
[root@ovirt1 ~]# df -h
Filesystem Size  Used Avail Use% Mounted on
/dev/mapper/centos_ovirt-root   20G  4.2G   16G  21% /
devtmpfs16G 0   16G   0% /dev
tmpfs   16G   16K   16G   1% /dev/shm
tmpfs   16G   26M   16G   1% /run
tmpfs   16G 0   16G   0% /sys/fs/cgroup
/dev/mapper/gluster-engine  25G   12G   14G  47% /gluster/brick1
/dev/sda1  497M  315M  183M  64% /boot
/dev/mapper/gluster-data   136G  124G   13G  92% /gluster/brick2
/dev/mapper/gluster-iso 25G  7.3G   18G  29% /gluster/brick4
tmpfs  3.2G 0  3.2G   0% /run/user/0
192.168.8.11:/engine15G  9.7G  5.4G  65%
/rhev/data-center/mnt/glusterSD/192.168.8.11:_engine
192.168.8.11:/data 136G  124G   13G  92%
/rhev/

Re: [ovirt-users] Software RAID on oVirt Node

[Vinícius Ferrão]

> On 4 Aug 2017, at 16:20, Chris Adams  wrote:
> 
> Once upon a time, Vinícius Ferrão  said:
>> On typical deployment scenarios of oVirt which is the recommended RAID 
>> technologies for oVirt Node installation? Should I use controller based RAID 
>> or mdadm can be used instead? Is this recommended?
>> 
>> I’m asking this because other vendors requires hardware RAID, even those 
>> 100% based on CentOS, like XenServer. There’s not even a way to install it 
>> with mdadm (Software Raid).
> 
> I use Linux software RAID under oVirt just fine.  I'm not using oVirt
> Node though (I just installed CentOS and then installed oVirt).  Note
> that I have an iSCSI SAN for VM storage - things might be different if
> you are planning to use the local disks for VMs (local storage or
> Gluster).

Hello Chris,

On non-node installation I can’t see any problems as you said, but due to the 
appliance nature of oVirt Node I don’t know if this would be a supported 
scenario. Anyway you raised a good point: local storage. I’m not needing this, 
perhaps someone on the list will be using this feature.

V.

> 
> -- 
> Chris Adams 
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users

___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] [ANN] oVirt 4.1.5 Second Release Candidate is now available

[Gianluca Cecchi]

On Fri, Aug 4, 2017 at 5:42 PM, Sandro Bonazzola 
wrote:

> The oVirt Project is pleased to announce the availability of the Second
> Release Candidate of oVirt 4.1.5 for testing, as of August 4th, 2017
>
>
Was the first 4.1.5 RC a secret ;-) or did I miss the announcement?

An happy weekend to all the list
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] Software RAID on oVirt Node

[Chris Adams]

Once upon a time, Vinícius Ferrão  said:
> On typical deployment scenarios of oVirt which is the recommended RAID 
> technologies for oVirt Node installation? Should I use controller based RAID 
> or mdadm can be used instead? Is this recommended?
> 
> I’m asking this because other vendors requires hardware RAID, even those 100% 
> based on CentOS, like XenServer. There’s not even a way to install it with 
> mdadm (Software Raid).

I use Linux software RAID under oVirt just fine.  I'm not using oVirt
Node though (I just installed CentOS and then installed oVirt).  Note
that I have an iSCSI SAN for VM storage - things might be different if
you are planning to use the local disks for VMs (local storage or
Gluster).

-- 
Chris Adams 
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

[ovirt-users] Software RAID on oVirt Node

[Vinícius Ferrão]

Hello,

On typical deployment scenarios of oVirt which is the recommended RAID 
technologies for oVirt Node installation? Should I use controller based RAID or 
mdadm can be used instead? Is this recommended?

I’m asking this because other vendors requires hardware RAID, even those 100% 
based on CentOS, like XenServer. There’s not even a way to install it with 
mdadm (Software Raid).

Thanks,
V.

___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

[ovirt-users] [ANN] oVirt 4.1.5 Second Release Candidate is now available

[Sandro Bonazzola]

The oVirt Project is pleased to announce the availability of the Second
Release Candidate of oVirt 4.1.5 for testing, as of August 4th, 2017

This is pre-release software. Please take a look at our community page[1]
to know how to ask questions and interact with developers and users.
All issues or bugs should be reported via oVirt Bugzilla[2].
This pre-release should not to be used in production.

This update is the second release candidate of the fifth in a series of
stabilization updates to the 4.1 series.
4.1.5 brings more than 20 enhancements and more than 50 bugfixes,
including more than 20 high or urgent
severity fixes, on top of oVirt 4.1 series

This release is available now for:
* Fedora 24 (tech preview)
* Red Hat Enterprise Linux 7.3 or later
* CentOS Linux (or similar) 7.3 or later

This release supports Hypervisor Hosts running:
* Red Hat Enterprise Linux 7.3 or later
* CentOS Linux (or similar) 7.3 or later
* oVirt Node 4.1
* Fedora 24 (tech preview)

See the release notes draft [3] for installation / upgrade instructions and
a list of new features and bugs fixed.

Notes:
- oVirt Appliance is already available
- oVirt Live is already available[4]
- oVirt Node is already available[4]

Additional Resources:
* Read more about the oVirt 4.1.5 release highlights:
http://www.ovirt.org/release/4.1.5/
* Get more oVirt Project updates on Twitter: https://twitter.com/ovirt
* Check out the latest project news on the oVirt blog:
http://www.ovirt.org/blog/

[1] https://www.ovirt.org/community/
[2] https://bugzilla.redhat.com/enter_bug.cgi?classification=oVirt
[3] http://www.ovirt.org/release/4.1.5/
[4] http://resources.ovirt.org/pub/ovirt-4.1-pre/iso/


-- 

SANDRO BONAZZOLA

ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D

Red Hat EMEA 

TRIED. TESTED. TRUSTED. 
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt 4.1 : Can't install guest tools for Zentyal 5.0 (based on ubuntu 16.04 xenial)

[Wesley Stewart]

Just wanted to add my input.  I just recently noticed the same thing.
Luckily i was just testing Zentyal, but when I installed python-apt after
reading the error message, apt seemed to completely break.  I would be
curious on a workaround/fix for this as well.


On Fri, Aug 4, 2017 at 9:28 AM, yayo (j)  wrote:

> Hi all,
>
> I have this problem: I'm tring to install the guest tools following this
> guide: https://www.ovirt.org/documentation/how-to/guest-
> agent/install-the-guest-agent-in-ubuntu/#for-ubuntu-1604
>
> This is the sources:
>
> deb http://it.archive.ubuntu.com/ubuntu/ xenial main restricted
> deb http://it.archive.ubuntu.com/ubuntu/ xenial-updates main restricted
> deb http://it.archive.ubuntu.com/ubuntu/ xenial universe
> deb http://it.archive.ubuntu.com/ubuntu/ xenial-updates universe
> deb http://it.archive.ubuntu.com/ubuntu/ xenial multiverse
> deb http://it.archive.ubuntu.com/ubuntu/ xenial-updates multiverse
> deb http://it.archive.ubuntu.com/ubuntu/ xenial-backports main restricted
> universe multiverse
>
> deb http://archive.zentyal.org/zentyal 5.0 main extra
>
> deb http://security.ubuntu.com/ubuntu xenial-security main restricted
> deb http://security.ubuntu.com/ubuntu xenial-security universe
> deb http://security.ubuntu.com/ubuntu xenial-security multiverse
>
>
> But when I try to install, I have this error:
>
>
> root@vmdczen01:~# apt-get install ovirt-guest-agent
> Reading package lists... Done
> Building dependency tree
> Reading state information... Done
> Some packages could not be installed. This may mean that you have
> requested an impossible situation or if you are using the unstable
> distribution that some required packages have not yet been created
> or been moved out of Incoming.
> The following information may help to resolve the situation:
>
> The following packages have unmet dependencies:
>  ovirt-guest-agent : Depends: python-apt but it is not going to be
> installed
> E: Unable to correct problems, you have held broken packages.
>
>
> I've found that python-apt is in "main" repo:
>
>
> https://packages.ubuntu.com/xenial/amd64/python-apt/download
>
>
> So, what's the problem?
>
>
> Can you help me?
> Thank you
>
>
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

[ovirt-users] ovirt 4.1 : Can't install guest tools for Zentyal 5.0 (based on ubuntu 16.04 xenial)

[yayo (j)]

Hi all,

I have this problem: I'm tring to install the guest tools following this
guide:
https://www.ovirt.org/documentation/how-to/guest-agent/install-the-guest-agent-in-ubuntu/#for-ubuntu-1604

This is the sources:

deb http://it.archive.ubuntu.com/ubuntu/ xenial main restricted
deb http://it.archive.ubuntu.com/ubuntu/ xenial-updates main restricted
deb http://it.archive.ubuntu.com/ubuntu/ xenial universe
deb http://it.archive.ubuntu.com/ubuntu/ xenial-updates universe
deb http://it.archive.ubuntu.com/ubuntu/ xenial multiverse
deb http://it.archive.ubuntu.com/ubuntu/ xenial-updates multiverse
deb http://it.archive.ubuntu.com/ubuntu/ xenial-backports main restricted
universe multiverse

deb http://archive.zentyal.org/zentyal 5.0 main extra

deb http://security.ubuntu.com/ubuntu xenial-security main restricted
deb http://security.ubuntu.com/ubuntu xenial-security universe
deb http://security.ubuntu.com/ubuntu xenial-security multiverse


But when I try to install, I have this error:


root@vmdczen01:~# apt-get install ovirt-guest-agent
Reading package lists... Done
Building dependency tree
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 ovirt-guest-agent : Depends: python-apt but it is not going to be installed
E: Unable to correct problems, you have held broken packages.


I've found that python-apt is in "main" repo:


https://packages.ubuntu.com/xenial/amd64/python-apt/download


So, what's the problem?


Can you help me?
Thank you
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] Cannot set a quota to limit resources for each user

[Staniforth, Paul]

Hi Soumya,
You can copy  quotas, what I can't figure out is how to set 
a consumer from the API.

In table 7.263 Attributes Summary at
https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.1/html/rest_api_guide/types#types-quota

it show a users attribute but I only get


testcli



20
80

20
80


Regards,
   Paul S.


From: users-boun...@ovirt.org  on behalf of Soumya 
Koduri 
Sent: 03 August 2017 15:16
To: Artyom Lukianov
Cc: users
Subject: Re: [ovirt-users] Cannot set a quota to limit resources for each   
user

Hi Artyom,

On 08/03/2017 04:07 PM, Artyom Lukianov wrote:
> Hi Soumya,
>
>  1. Yes, you correct a quota is shared among all quota consumers.
>  2. No, it not possible, the single possibility as you said is to create
> a separate quota for each user. I believe it was designed to make
> quota for group of users at first place, but I think it can be a
> good RFE :)

Thanks for confirming. I shall file a RFE then.

Regards,
Soumya

>
> Best Regards
>
> On Thu, Aug 3, 2017 at 10:54 AM, Soumya Koduri  > wrote:
>
> Hi,
>
> We have a use-case to limit VM resources for each user and were
> following guidelines specified in the admin guide to set quota and
> limit resources for each user [section:  16.8. Using Quota to Limit
> Resources by User]. However looks like that quota is shared by all
> the users added as consumers.
>
> Suppose I have created a quota (say quota1) to limit the storage
> capacity to 100GB for each user. Once I add user1 and user2 as
> consumers to that quota, seems like both users combined are entitled
> to 100GB. Is my understanding correct?
>
> Please let me know if there is any way to configure a single quota
> which could be applied for each user individually (i.e, in the above
> eg., each user should be limited to 100GB storage capacity).
>
> Or is the only way this can be done is by creating separate quota
> for each user [which seems like tedious process and cannot scale]?
>
>
> Thanks,
> Soumya
> ___
> Users mailing list
> Users@ovirt.org 
> http://lists.ovirt.org/mailman/listinfo/users
> 
>
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
To view the terms under which this email is distributed, please go to:-
http://disclaimer.leedsbeckett.ac.uk/disclaimer/disclaimer.html
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] Python errors with ovirt 4.1.4

[Staniforth, Paul]

Hello,

   I have 3 nodes and used the engine to update them to


ovirt-node-ng-4.1.4-0.20170728.0


but the engine still reported a new update which I tried but it failed.


On the nodes yum check-update showed an update for


ovirt-node-ng-nodectl.noarch4.1.4-0.20170728.0.el7


installing this produces the same errors when logging into the node or running 
nodectl motd.

nodectl check and info where fine but the engine produced errors when checking 
for updates.


I used yum history to rollback the ovirt-node-ng-nodectl.noarch.


I now have no errors but strangely the engine reports 2 nodes have updates 
available but not the 3rd which wasn't the one I did a nodectl update on.


Regards,

   Paul S.



From: users-boun...@ovirt.org  on behalf of david 
caughey 
Sent: 02 August 2017 10:48
To: Users@ovirt.org
Subject: [ovirt-users] Python errors with ovirt 4.1.4

Hi Folks,

I'm testing out the new version with the 4.1.4 ovirt iso and am getting errors 
directly after install:

Last login: Wed Aug  2 10:17:56 2017
Traceback (most recent call last):
  File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
  File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
  File "/usr/lib/python2.7/site-packages/nodectl/__main__.py", line 42, in 

CliApplication()
  File "/usr/lib/python2.7/site-packages/nodectl/__init__.py", line 200, in 
CliApplication
return cmdmap.command(args)
  File "/usr/lib/python2.7/site-packages/nodectl/__init__.py", line 118, in 
command
return self.commands[command](**kwargs)
  File "/usr/lib/python2.7/site-packages/nodectl/__init__.py", line 102, in motd
machine_readable=True).output, self.machine).write()
  File "/usr/lib/python2.7/site-packages/nodectl/status.py", line 51, in 
__init__
self._update_info(status)
  File "/usr/lib/python2.7/site-packages/nodectl/status.py", line 78, in 
_update_info
if "ok" not in status.lower():
AttributeError: Status instance has no attribute 'lower'
Admin Console: https://192.168.122.61:9090/

The admin console seems to work fine.

Are these issues serious or can they be ignored.

BR/David
To view the terms under which this email is distributed, please go to:-
http://disclaimer.leedsbeckett.ac.uk/disclaimer/disclaimer.html
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users