Certainly using Windows' file system lock is appropriate for locking its files. I thought the case we were talking about was just abusing that ability as a generic cross-process lock.
I wasn't aware of how configstore was using fslock. On Tue, Dec 1, 2015 at 11:58 AM roger peppe <roger.pe...@canonical.com> wrote: > On 1 December 2015 at 16:43, Nate Finch <nate.fi...@canonical.com> wrote: > > I think half the problem is that someone named the package fslock and not > > oslock, so we're stuck asking the wrong question. > > > > If the question is "How do I acquire an OS-level lock for a process?" > The > > answer on Windows is "Use a named mutex". > > That's not the question we need to answer here, for the configstore case > anyway. In that case, we really do want to protect access to a shared data > store in the filesystem, so the term "fslock" is appropriate I think. > > > Dave seems to be saying that the > > only problem with unix domain sockets is if you try to think about them > like > > files... which may be a result of us thinking too much about the solution > > and not the problem. (again, I'm not an expert, you guys can duke out > what > > the best solution is) I think using hackery with the filesystem locking > on > > Windows is a mistake. > > Is there something wrong with using the Windows file-exclusive-open > semantics to implement a lock associated with a entity in the file system? > Isn't that what it's for? > > > > On Tue, Dec 1, 2015 at 4:10 AM roger peppe <roger.pe...@canonical.com> > > wrote: > >> > >> So I'm with axw on this one - flock seems like it is a reasonable tool > for > >> the job here. FWIW a Unix domain socket also suffers from the "won't > >> work across NFS" problem. Abstract unix domain sockets also > >> have the problem that they won't work with long file paths (what > >> were they thinking?) > >> > >> We have been using github.com/camlistore/lock and although it's not > >> totally ideal (see https://github.com/camlistore/lock/issues/10 ) > >> it does the job. Note that it's non-blocking, so a blocking > >> layer above it is necessary, for example see the lockFile > >> function in > >> https://github.com/juju/persistent-cookiejar/blob/master/serialize.go > >> > >> The Windows named mutex thing does sound interesting because > >> it's a blocking API, which is actually what we need. On the other > >> hand, under Windows, files opened for writing are locked by default > >> anyway, so it might be easier just to leverage that property. > >> The camlistore lock code could use some improvement for the > >> Windows platform - we could either fork it, or bradfitz would > >> probably be amenable to a PR. > >> > >> cheers, > >> rog. > >> > >> On 1 December 2015 at 04:12, Nate Finch <nate.fi...@canonical.com> > wrote: > >> > I'm not a linux expert, but definitely a named mutex is exactly the > >> > correct > >> > thing to use for Windows. Using something else for this purpose would > >> > be > >> > very surprising to a Windows dev and much more likely to be buggy. I > >> > don't > >> > see any reason we need to use the exact same implementation on all > OSes > >> > if > >> > there is something that does exactly the right thing without any > >> > finagling. > >> > FWIW, mutexes do die with the process: > >> > > >> > > https://msdn.microsoft.com/en-us/library/windows/desktop/ms682411(v=vs.85).aspx > >> > > >> > On Mon, Nov 30, 2015 at 8:29 PM Andrew Wilkins > >> > <andrew.wilk...@canonical.com> wrote: > >> >> > >> >> On Tue, Dec 1, 2015 at 9:07 AM David Cheney > >> >> <david.che...@canonical.com> > >> >> wrote: > >> >>> > >> >>> http://0pointer.de/blog/projects/locking.html > >> >>> > >> >>> In short, opening the same file twice and asserting a lock on it > will > >> >>> succeed. > >> >> > >> >> > >> >> Thanks. The article is a bit exasperated. Yes, there are problems to > be > >> >> aware of, but it doesn't make them unusable in all cases. > >> >> - Juju agents don't get installed onto NFS file systems, so doesn't > >> >> matter for the agents. > >> >> - We're in full control of the files we're locking, we're not > locking > >> >> some file like /etc/passwd where some other random bit of code in the > >> >> process is going to open/close it and release the lock by accident. > >> >> - We don't need byte-range locking. > >> >> > >> >> So only the original uncertainty remains: do we care about clients > >> >> running > >> >> their home directory on an NFS share, where the NFS *server* is too > old > >> >> to > >> >> support flock? > >> >> > >> >> Maybe a named mutex on Windows and a domain socket on *NIX is the way > >> >> to > >> >> go. I'm not dead set on flock; I just want something that is simple, > >> >> robust > >> >> and portable. > >> >> > >> >> Cheers, > >> >> Andrew > >> >> > >> >>> On Tue, Dec 1, 2015 at 12:04 PM, Andrew Wilkins > >> >>> <andrew.wilk...@canonical.com> wrote: > >> >>> > On Tue, Dec 1, 2015 at 8:57 AM David Cheney > >> >>> > <david.che...@canonical.com> > >> >>> > wrote: > >> >>> >> > >> >>> >> fcntl won't work in threaded go applications, it barely works in > >> >>> >> non > >> >>> >> threaded applications. > >> >>> > > >> >>> > > >> >>> > This is news to me. I've used it plenty in the past, in > >> >>> > multi-threaded > >> >>> > programs. Please point me at relevant literature that explains > where > >> >>> > it > >> >>> > doesn't work. > >> >>> > > >> >>> >> > >> >>> >> I'm not interested in "doesn't work on windows" arguments. Yes, > we > >> >>> >> have to support windows, but that doesn't mean we have to be > >> >>> >> dragged > >> >>> >> down to it's lowest common denominator. > >> >>> > > >> >>> > > >> >>> > Agreed, if we're actually crippling anything. > >> >>> > > >> >>> >> > >> >>> >> I think it's fine to develop a lock type that does the best > >> >>> >> available > >> >>> >> for each platform using conditional compilation. > >> >>> > > >> >>> > > >> >>> > Again, agreed, but only if there's something to be gained by doing > >> >>> > this. I'm > >> >>> > still not convinced there is. > >> >>> > > >> >>> >> On Tue, Dec 1, 2015 at 11:47 AM, Andrew Wilkins > >> >>> >> <andrew.wilk...@canonical.com> wrote: > >> >>> >> > On Tue, Dec 1, 2015 at 8:03 AM David Cheney > >> >>> >> > <david.che...@canonical.com> > >> >>> >> > wrote: > >> >>> >> >> > >> >>> >> >> On Tue, Dec 1, 2015 at 11:00 AM, Andrew Wilkins > >> >>> >> >> <andrew.wilk...@canonical.com> wrote: > >> >>> >> >> > On Tue, Dec 1, 2015 at 7:57 AM David Cheney > >> >>> >> >> > <david.che...@canonical.com> > >> >>> >> >> > wrote: > >> >>> >> >> >> > >> >>> >> >> >> Doesn't look like there is windows support, and it uses > fcntl > >> >>> >> >> >> (flock) > >> >>> >> >> >> under the hood, which is what we have now. > >> >>> >> >> > > >> >>> >> >> > > >> >>> >> >> > flock isn't the problematic thing Tim is talking about. > >> >>> >> >> > utils/fslock > >> >>> >> >> > attempts to create a directory in a known location, and if > it > >> >>> >> >> > succeeds, > >> >>> >> >> > it > >> >>> >> >> > "holds the lock". Unlocking means removing the directory. > >> >>> >> >> > >> >>> >> >> The problem is if the process dies/exits/goes mental while > >> >>> >> >> holding > >> >>> >> >> the > >> >>> >> >> lock we get into this existential gridlock where we're not > sure > >> >>> >> >> if > >> >>> >> >> the > >> >>> >> >> process _is_ alive, so we shouldn't remove the lock, or the > >> >>> >> >> process > >> >>> >> >> is > >> >>> >> >> dead, so we should remove the lock. > >> >>> >> >> > >> >>> >> >> abstract unix domain sockets do not have this problem on > >> >>> >> >> windows; > >> >>> >> >> kill > >> >>> >> >> the process, file is removed from the abstract namespace. > >> >>> >> > > >> >>> >> > > >> >>> >> > POSIX advisory file locks (flock) do not have this problem > >> >>> >> > either. > >> >>> >> > See: > >> >>> >> > man(2) fcntl, section "Advisory record locking". When the file > >> >>> >> > descriptor is > >> >>> >> > closed, the lock is released; file descriptors are closed when > >> >>> >> > the > >> >>> >> > process > >> >>> >> > dies. > >> >>> >> > > >> >>> >> > We could build a mutex on top of a unix domain socket, but then > >> >>> >> > we'll > >> >>> >> > have > >> >>> >> > something completely separate for Windows. Shared named mutex? > >> >>> >> > I'm > >> >>> >> > not > >> >>> >> > convinced the overall solution would be any more robust, and > I'm > >> >>> >> > pretty > >> >>> >> > sure > >> >>> >> > it's going to be more complicated. Happy to be proven wrong > >> >>> >> > though. > >> >>> >> > > >> >>> >> >> > > >> >>> >> >> > We would have to contribute Windows support, but it's not > >> >>> >> >> > hard, > >> >>> >> >> > I've > >> >>> >> >> > done it > >> >>> >> >> > before. > >> >>> >> >> > > >> >>> >> >> >> > >> >>> >> >> >> On Tue, Dec 1, 2015 at 10:55 AM, Casey Marshall > >> >>> >> >> >> <casey.marsh...@canonical.com> wrote: > >> >>> >> >> >> > How about github.com/camlistore/lock ? > >> >>> >> >> >> > > >> >>> >> >> >> > On Mon, Nov 30, 2015 at 5:43 PM, Tim Penhey > >> >>> >> >> >> > <tim.pen...@canonical.com> > >> >>> >> >> >> > wrote: > >> >>> >> >> >> >> > >> >>> >> >> >> >> Hi folks, > >> >>> >> >> >> >> > >> >>> >> >> >> >> The fslock was a mistake that I added to the codebase > some > >> >>> >> >> >> >> time > >> >>> >> >> >> >> back. > >> >>> >> >> >> >> It > >> >>> >> >> >> >> provided an overly simplistic solution to a more complex > >> >>> >> >> >> >> problem. > >> >>> >> >> >> >> > >> >>> >> >> >> >> Really the filesystem shouldn't be used as a locking > >> >>> >> >> >> >> mechanism. > >> >>> >> >> >> >> > >> >>> >> >> >> >> Most of the code that exists for the fslock now is > working > >> >>> >> >> >> >> around > >> >>> >> >> >> >> its > >> >>> >> >> >> >> deficiencies. Instead we should be looking for a better > >> >>> >> >> >> >> replacement. > >> >>> >> >> >> >> > >> >>> >> >> >> >> Some "features" that were added to fslock were added to > >> >>> >> >> >> >> work > >> >>> >> >> >> >> around > >> >>> >> >> >> >> the > >> >>> >> >> >> >> issue that the lock did not die with the process that > >> >>> >> >> >> >> created > >> >>> >> >> >> >> it, > >> >>> >> >> >> >> so > >> >>> >> >> >> >> some mechanism was needed to determine whether the lock > >> >>> >> >> >> >> should be > >> >>> >> >> >> >> broken > >> >>> >> >> >> >> or not. > >> >>> >> >> >> >> > >> >>> >> >> >> >> What we really need is a good OS agnostic abstraction > that > >> >>> >> >> >> >> provides > >> >>> >> >> >> >> the > >> >>> >> >> >> >> ability to create a "named" lock, acquire the lock, > >> >>> >> >> >> >> release > >> >>> >> >> >> >> the > >> >>> >> >> >> >> lock, > >> >>> >> >> >> >> and make sure that the lock dies when the process dies, > so > >> >>> >> >> >> >> another > >> >>> >> >> >> >> process that is waiting can acquire the lock. This way > no > >> >>> >> >> >> >> "BreakLock" > >> >>> >> >> >> >> functionality is required, nor do we need to try and do > >> >>> >> >> >> >> think > >> >>> >> >> >> >> like > >> >>> >> >> >> >> remember which process owns the lock. > >> >>> >> >> >> >> > >> >>> >> >> >> >> So... > >> >>> >> >> >> >> > >> >>> >> >> >> >> We have three current operating systems we need to > >> >>> >> >> >> >> support: > >> >>> >> >> >> >> > >> >>> >> >> >> >> Linux - Ubuntu and CentOS > >> >>> >> >> >> >> MacOS - client only - bit the CLI uses a lock for the > >> >>> >> >> >> >> local > >> >>> >> >> >> >> cache > >> >>> >> >> >> >> Windows > >> >>> >> >> >> >> > >> >>> >> >> >> >> For Linux, and possibly MacOS, flock is a possibility, > but > >> >>> >> >> >> >> can we > >> >>> >> >> >> >> do > >> >>> >> >> >> >> better? Is there something that is better suited? > >> >>> >> >> >> >> > >> >>> >> >> >> >> For Windows, while you can create global semaphores or > >> >>> >> >> >> >> mutex > >> >>> >> >> >> >> instances, > >> >>> >> >> >> >> I'm not sure of entities that die with the process. Can > >> >>> >> >> >> >> people > >> >>> >> >> >> >> recommend > >> >>> >> >> >> >> solutions? > >> >>> >> >> >> >> > >> >>> >> >> >> >> Cheers, > >> >>> >> >> >> >> Tim > >> >>> >> >> >> >> > >> >>> >> >> >> >> -- > >> >>> >> >> >> >> Juju-dev mailing list > >> >>> >> >> >> >> Juju-dev@lists.ubuntu.com > >> >>> >> >> >> >> Modify settings or unsubscribe at: > >> >>> >> >> >> >> https://lists.ubuntu.com/mailman/listinfo/juju-dev > >> >>> >> >> >> > > >> >>> >> >> >> > > >> >>> >> >> >> > > >> >>> >> >> >> > -- > >> >>> >> >> >> > Juju-dev mailing list > >> >>> >> >> >> > Juju-dev@lists.ubuntu.com > >> >>> >> >> >> > Modify settings or unsubscribe at: > >> >>> >> >> >> > https://lists.ubuntu.com/mailman/listinfo/juju-dev > >> >>> >> >> >> > > >> >>> >> >> >> > >> >>> >> >> >> -- > >> >>> >> >> >> Juju-dev mailing list > >> >>> >> >> >> Juju-dev@lists.ubuntu.com > >> >>> >> >> >> Modify settings or unsubscribe at: > >> >>> >> >> >> https://lists.ubuntu.com/mailman/listinfo/juju-dev > >> >> > >> >> -- > >> >> Juju-dev mailing list > >> >> Juju-dev@lists.ubuntu.com > >> >> Modify settings or unsubscribe at: > >> >> https://lists.ubuntu.com/mailman/listinfo/juju-dev > >> > > >> > > >> > -- > >> > Juju-dev mailing list > >> > Juju-dev@lists.ubuntu.com > >> > Modify settings or unsubscribe at: > >> > https://lists.ubuntu.com/mailman/listinfo/juju-dev > >> > >
-- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
