>>> Dejan Muhamedagic <[email protected]> schrieb am 26.06.2017 um 17:05 in Nachricht <[email protected]>: > Hi, > > On Wed, Jun 21, 2017 at 04:40:47PM +0200, Lars Ellenberg wrote: >> >> Repost to a wider audience, to raise awareness for this. >> ocf_take_lock may or may not be better than nothing. >> >> It at least "annotates" that the auther would like to protect something >> that is considered a "critical region" of the resource agent. >> >> At the same time, it does NOT deliver what the name seems to imply. >> > > Lars, many thanks for the analysis and bringing this up again. > > I'm not going to take on the details below, just to say that > there's now a pull request for the issue: > > https://github.com/ClusterLabs/resource-agents/pull/995 > > In short, it consists of reducing the race window size (by using > mkdir*), double test for stale locks, and improved random number > function. I ran numerous tests with and without stale locks and > it seems to hold quite well. > > The comments there contain a detailed description of the > approach. > > Please review and comment whoever finds time. > > Cheers, > > Dejan > > *) Though the current implementation uses just a file and the > proposed one directories, the locks are short lived and there > shouldn't be problems on upgrades. > >> I think I brought this up a few times over the years, but was not noisy >> enough about it, because it seemed not important enough: no-one was >> actually using this anyways. >> >> But since new usage has been recently added with >> [ClusterLabs/resource-agents] targetcli lockfile (#917) >> here goes: >> >> On Wed, Jun 07, 2017 at 02:49:41PM -0700, Dejan Muhamedagic wrote: >> > On Wed, Jun 07, 2017 at 05:52:33AM -0700, Lars Ellenberg wrote: >> > > Note: ocf_take_lock is NOT actually safe to use. >> > > >> > > As implemented, it uses "echo $pid > lockfile" to create the lockfile, >> > > which means if several such "ocf_take_lock" happen at the same time, >> > > they all "succeed", only the last one will be the "visible" one to >> > > future > waiters. >> > >> > Ugh. >> >> Exactly. >> >> Reproducer: >> ############################################################# >> #!/bin/bash >> export OCF_ROOT=/usr/lib/ocf/ ; >> . /usr/lib/ocf/lib/heartbeat/ocf-shellfuncs ; >> >> x() ( >> ocf_take_lock dummy-lock ; >> ocf_release_lock_on_exit dummy-lock ; >> set -C; >> echo x > protected && sleep 0.15 && rm -f protected || touch BROKEN; >> ); >> >> mkdir -p /run/ocf_take_lock_demo >> cd /run/ocf_take_lock_demo >> rm -f BROKEN; i=0; >> time while ! test -e BROKEN; do >> x & x & >> wait; >> i=$(( i+1 )); >> done ; >> test -e BROKEN && echo "reproduced race in $i iterations" >> ############################################################# >> >> x() above takes, and, because of the () subshell and >> ocf_release_lock_on_exit, releases the "dummy-lock", >> and within the protected region of code, >> creates and removes a file "protected". >> >> If ocf_take_lock was good, there could never be two instances >> inside the lock, so echo x > protected should never fail. >> >> With the current implementation of ocf_take_lock, >> it takes "just a few" iterations here to reproduce the race. >> (usually within a minute). >> >> The races I see in ocf_take_lock: >> "creation race": >> test -e $lock >> # someone else may create it here >> echo $$ > $lock >> # but we override it with ours anyways >> >> "still empty race": >> test -e $lock # maybe it already exists (open O_CREAT|O_TRUNC) >> # but does not yet contain target pid, >> pid=`cat $lock` # this one is empty, >> kill -0 $pid # and this one fails >> and thus a "just being created" one is considered stale >> >> There are other problems around "stale pid file detection", >> but let's not go into that minefield right now. >> >> > > Maybe we should change it to >> > > ``` >> > > while ! ( set -C; echo $pid > lockfile ); do >> > > if test -e lockfile ; then >> > > : error handling for existing lockfile, stale lockfile detection >> > > else >> > > : error handling for not being able to create lockfile >> > > fi >> > > done >> > > : only reached if lockfile was successfully created >> > > ``` >> > > >> > > (or use flock or other tools designed for that purpose) >> > >> > flock would probably be the easiest. mkdir would do too, but for >> > upgrade issues. >> >> and, being part of util-linux, flock should be available "everywhere". >> >> but because writing "wrappers" around flock similar to the intended >> semantics of ocf_take_lock and ocf_release_lock_on_exit is not easy >> either, usually you'd be better of using flock directly in the RA. >> >> so, still trying to do this with shell: >> >> "set -C" (respectively set -o noclober): >> If set, disallow existing regular files to be overwritten >> by redirection of output. >> >> normal '>' means: O_WRONLY|O_CREAT|O_TRUNC, >> set -C '>' means: O_WRONLY|O_CREAT|O_EXCL >> >> using "set -C ; echo $$ > $lock" instead of >> "test -e $lock || echo $$ > $lock" >> gets rid of the "creation race". >> >> To get rid of the "still empty race", >> we'd need to play games with hardlinks: >> >> ( >> set -C >> success=false >> if echo $$ > .$lock ; then >> ln .$lock $lock && success=true >> rm -f .$lock >> fi >> $success >> ) >> >> That should be "good enough", >> and much better than what we have now. >> >> >> >> back to a possible "flock" wrapper, >> maybe something like this: >> >> ocf_synchronize() { >> local lockfile=$1 >> shift >> ( >> flock -x 8 || exit 1 >> ( "$@" ) 8>&- >> ) 8> "$lockfile" >> } >> # and then >> ocf_synchronize my_exclusive_shell_function with some args >> >> As this runs in subshells, >> it would not be able to change any variales visible >> to the rest of the script, which may limit its usefulness. >> >> >> or maybe, have a ocf_rarun_synchronized(), >> that will do (directly from the flock man page): >> [ "${FLOCKER}" != "$0" ] && exec env FLOCKER="$0" flock -e "$0" "$0" "$@" >> || > : >> with nonblocking and timeout variants? >> >> >> I'd very much like some volunteer to step forward >> and implement the actual patch... >> >> >> Cheers, >> >> Lars >> >> >> _______________________________________________ >> Users mailing list: [email protected] >> http://lists.clusterlabs.org/mailman/listinfo/users >> >> Project Home: http://www.clusterlabs.org >> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf >> Bugs: http://bugs.clusterlabs.org > > _______________________________________________ > Users mailing list: [email protected] > http://lists.clusterlabs.org/mailman/listinfo/users > > Project Home: http://www.clusterlabs.org > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > Bugs: http://bugs.clusterlabs.org
_______________________________________________ Users mailing list: [email protected] http://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
