Itamar, I am addressing this to you because one of your assignments seems to
be to coordinate other oVirt contributors when dealing with issues that are
raised on the ovirt-users email list.
As you are aware, there is an ongoing split-brain problem with running
sanlock on replicated gluster storage. Personally, I believe that this is
the 5th time that I have been bitten by this sanlock+gluster problem.
I believe that the following are true (if not, my entire request is probably
off base).
* ovirt uses sanlock in such a way that when the sanlock storage is on a
replicated gluster file system, very small storage disruptions can result
in a gluster split-brain on the sanlock space
o gluster is aware of the problem, and is working on a different way of
replicating data, which will reduce these problems.
* most (maybe all) of the sanlock locks have a short duration, measured in
seconds
* there are only a couple of things that a user can safely do from the
command line when a file is in split-brain
o delete the file
o rename (mv) the file
* x
_How did I get into this mess?_
had 3 hosts running ovirt 3.3
each hosted VMs
gluster replica 3 storage
engine was external to cluster
upgraded 3 hosts from ovirt 3.3 to 3.4
hosted-engine deploy
used new gluster volume (accessed via nfs) for storage
storage was accessed using localhost:engVM1 link (localhost was
probably a poor choice)
created new engine on VM (did not transfer any data from old engine)
added 3 hosts to new engine via web-gui
ran above setup for 3 days
shut entire system down before I left on vacation (holiday)
came back from vacation
powered on hosts
found that iptables did not have rules for gluster access
(a continuing problem if host installation is allowed to set up firewall)
added rules for gluster
glusterfs now up and running
added storage manually
tried "hosted-engine --vm-start"
vm did not start
logs show sanlock errors
"gluster volume heal engVM1full:
"gluster volume heal engVM1 info split-brain" showed 6 files in split-brain
all 5 prefixed by /rhev/data-center/mnt/localhost\:_engVM1
UUID/dom_md/ids
UUID/images/UUID/UUID (VM hard disk)
UUID/images/UUID/UUID.lease
UUID/ha_agent/hosted-engine.lockspace
UUID/ha_agent/hosted-engine.metadata
I copied each of the above files off of each of the three bricks to a safe
place (15 files copied)
I renamed the 5 files on /rhev/....
I copied the 5 files from one of the bricks to /rhev/
files can now be read OK (e.g. cat ids)
sanlock.log shows error sets like these:
2014-05-20 03:23:39-0400 36199 [2843]: s3358 lockspace
5ebb3b40-a394-405b-bbac-4c0e21ccd659:1:/rhev/data-center/mnt/localhost:_engVM1/5ebb3b40-a394-405b-bbac-4c0e21ccd659/dom_md/ids:0
2014-05-20 03:23:39-0400 36199 [18873]: open error -5
/rhev/data-center/mnt/localhost:_engVM1/5ebb3b40-a394-405b-bbac-4c0e21ccd659/dom_md/ids
2014-05-20 03:23:39-0400 36199 [18873]: s3358 open_disk
/rhev/data-center/mnt/localhost:_engVM1/5ebb3b40-a394-405b-bbac-4c0e21ccd659/dom_md/ids
error -5
2014-05-20 03:23:40-0400 36200 [2843]: s3358 add_lockspace fail result -19
I am now stuck
What I would like to see in ovirt to help me (and others like me). Alternates
listed in order from most desirable (automatic) to least desirable (set of
commands to type, with lots of variables to figure out).
1. automagic recovery
* When a host is not able to access sanlock, it writes a small "problem"
text file into the shared storage
o the host-ID as part of the name (so only one host ever accesses that
file)
o a status number for the error causing problems
o time stamp
o time stamp when last sanlock lease will expire
o if sanlock is able to access the file, the "problem" file is deleted
* when time passes for its last sanlock lease to be expired, highest number
host does a survey
o did all other hosts create "problem" files?
o do all "problem" files show same (or compatible) error codes related
to file access problems?
o are all hosts communicating by network?
o if yes to all above
* delete all sanlock storage space
* initialize sanlock from scratch
* restart whatever may have given up because of sanlock
* restart VM if necessary
2. recovery subcommand
* add "hosted-engine --lock-initialize" command that would delete sanlock,
start over from scratch
3. script
* publish a script (in ovirt packages or available on web) which, when run,
does all (or most) of the recovery process needed.
4. commands
* publish on the web a "recipe" for dealing with files that commonly go
split-brain
o ids
o *.lease
o *.lockspace
Any chance of any help on any of the above levels?
Ted Miller
Elkhart, IN, USA
_______________________________________________
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
