I think half the problem is that someone named the package fslock and not oslock, so we're stuck asking the wrong question.
If the question is "How do I acquire an OS-level lock for a process?" The answer on Windows is "Use a named mutex". Dave seems to be saying that the only problem with unix domain sockets is if you try to think about them like files... which may be a result of us thinking too much about the solution and not the problem. (again, I'm not an expert, you guys can duke out what the best solution is) I think using hackery with the filesystem locking on Windows is a mistake. On Tue, Dec 1, 2015 at 4:10 AM roger peppe <roger.pe...@canonical.com> wrote: > So I'm with axw on this one - flock seems like it is a reasonable tool for > the job here. FWIW a Unix domain socket also suffers from the "won't > work across NFS" problem. Abstract unix domain sockets also > have the problem that they won't work with long file paths (what > were they thinking?) > > We have been using github.com/camlistore/lock and although it's not > totally ideal (see https://github.com/camlistore/lock/issues/10 ) > it does the job. Note that it's non-blocking, so a blocking > layer above it is necessary, for example see the lockFile > function in > https://github.com/juju/persistent-cookiejar/blob/master/serialize.go > > The Windows named mutex thing does sound interesting because > it's a blocking API, which is actually what we need. On the other > hand, under Windows, files opened for writing are locked by default > anyway, so it might be easier just to leverage that property. > The camlistore lock code could use some improvement for the > Windows platform - we could either fork it, or bradfitz would > probably be amenable to a PR. > > cheers, > rog. > > On 1 December 2015 at 04:12, Nate Finch <nate.fi...@canonical.com> wrote: > > I'm not a linux expert, but definitely a named mutex is exactly the > correct > > thing to use for Windows. Using something else for this purpose would be > > very surprising to a Windows dev and much more likely to be buggy. I > don't > > see any reason we need to use the exact same implementation on all OSes > if > > there is something that does exactly the right thing without any > finagling. > > FWIW, mutexes do die with the process: > > > https://msdn.microsoft.com/en-us/library/windows/desktop/ms682411(v=vs.85).aspx > > > > On Mon, Nov 30, 2015 at 8:29 PM Andrew Wilkins > > <andrew.wilk...@canonical.com> wrote: > >> > >> On Tue, Dec 1, 2015 at 9:07 AM David Cheney <david.che...@canonical.com > > > >> wrote: > >>> > >>> http://0pointer.de/blog/projects/locking.html > >>> > >>> In short, opening the same file twice and asserting a lock on it will > >>> succeed. > >> > >> > >> Thanks. The article is a bit exasperated. Yes, there are problems to be > >> aware of, but it doesn't make them unusable in all cases. > >> - Juju agents don't get installed onto NFS file systems, so doesn't > >> matter for the agents. > >> - We're in full control of the files we're locking, we're not locking > >> some file like /etc/passwd where some other random bit of code in the > >> process is going to open/close it and release the lock by accident. > >> - We don't need byte-range locking. > >> > >> So only the original uncertainty remains: do we care about clients > running > >> their home directory on an NFS share, where the NFS *server* is too old > to > >> support flock? > >> > >> Maybe a named mutex on Windows and a domain socket on *NIX is the way to > >> go. I'm not dead set on flock; I just want something that is simple, > robust > >> and portable. > >> > >> Cheers, > >> Andrew > >> > >>> On Tue, Dec 1, 2015 at 12:04 PM, Andrew Wilkins > >>> <andrew.wilk...@canonical.com> wrote: > >>> > On Tue, Dec 1, 2015 at 8:57 AM David Cheney > >>> > <david.che...@canonical.com> > >>> > wrote: > >>> >> > >>> >> fcntl won't work in threaded go applications, it barely works in non > >>> >> threaded applications. > >>> > > >>> > > >>> > This is news to me. I've used it plenty in the past, in > multi-threaded > >>> > programs. Please point me at relevant literature that explains where > it > >>> > doesn't work. > >>> > > >>> >> > >>> >> I'm not interested in "doesn't work on windows" arguments. Yes, we > >>> >> have to support windows, but that doesn't mean we have to be dragged > >>> >> down to it's lowest common denominator. > >>> > > >>> > > >>> > Agreed, if we're actually crippling anything. > >>> > > >>> >> > >>> >> I think it's fine to develop a lock type that does the best > available > >>> >> for each platform using conditional compilation. > >>> > > >>> > > >>> > Again, agreed, but only if there's something to be gained by doing > >>> > this. I'm > >>> > still not convinced there is. > >>> > > >>> >> On Tue, Dec 1, 2015 at 11:47 AM, Andrew Wilkins > >>> >> <andrew.wilk...@canonical.com> wrote: > >>> >> > On Tue, Dec 1, 2015 at 8:03 AM David Cheney > >>> >> > <david.che...@canonical.com> > >>> >> > wrote: > >>> >> >> > >>> >> >> On Tue, Dec 1, 2015 at 11:00 AM, Andrew Wilkins > >>> >> >> <andrew.wilk...@canonical.com> wrote: > >>> >> >> > On Tue, Dec 1, 2015 at 7:57 AM David Cheney > >>> >> >> > <david.che...@canonical.com> > >>> >> >> > wrote: > >>> >> >> >> > >>> >> >> >> Doesn't look like there is windows support, and it uses fcntl > >>> >> >> >> (flock) > >>> >> >> >> under the hood, which is what we have now. > >>> >> >> > > >>> >> >> > > >>> >> >> > flock isn't the problematic thing Tim is talking about. > >>> >> >> > utils/fslock > >>> >> >> > attempts to create a directory in a known location, and if it > >>> >> >> > succeeds, > >>> >> >> > it > >>> >> >> > "holds the lock". Unlocking means removing the directory. > >>> >> >> > >>> >> >> The problem is if the process dies/exits/goes mental while > holding > >>> >> >> the > >>> >> >> lock we get into this existential gridlock where we're not sure > if > >>> >> >> the > >>> >> >> process _is_ alive, so we shouldn't remove the lock, or the > process > >>> >> >> is > >>> >> >> dead, so we should remove the lock. > >>> >> >> > >>> >> >> abstract unix domain sockets do not have this problem on windows; > >>> >> >> kill > >>> >> >> the process, file is removed from the abstract namespace. > >>> >> > > >>> >> > > >>> >> > POSIX advisory file locks (flock) do not have this problem either. > >>> >> > See: > >>> >> > man(2) fcntl, section "Advisory record locking". When the file > >>> >> > descriptor is > >>> >> > closed, the lock is released; file descriptors are closed when the > >>> >> > process > >>> >> > dies. > >>> >> > > >>> >> > We could build a mutex on top of a unix domain socket, but then > >>> >> > we'll > >>> >> > have > >>> >> > something completely separate for Windows. Shared named mutex? I'm > >>> >> > not > >>> >> > convinced the overall solution would be any more robust, and I'm > >>> >> > pretty > >>> >> > sure > >>> >> > it's going to be more complicated. Happy to be proven wrong > though. > >>> >> > > >>> >> >> > > >>> >> >> > We would have to contribute Windows support, but it's not hard, > >>> >> >> > I've > >>> >> >> > done it > >>> >> >> > before. > >>> >> >> > > >>> >> >> >> > >>> >> >> >> On Tue, Dec 1, 2015 at 10:55 AM, Casey Marshall > >>> >> >> >> <casey.marsh...@canonical.com> wrote: > >>> >> >> >> > How about github.com/camlistore/lock ? > >>> >> >> >> > > >>> >> >> >> > On Mon, Nov 30, 2015 at 5:43 PM, Tim Penhey > >>> >> >> >> > <tim.pen...@canonical.com> > >>> >> >> >> > wrote: > >>> >> >> >> >> > >>> >> >> >> >> Hi folks, > >>> >> >> >> >> > >>> >> >> >> >> The fslock was a mistake that I added to the codebase some > >>> >> >> >> >> time > >>> >> >> >> >> back. > >>> >> >> >> >> It > >>> >> >> >> >> provided an overly simplistic solution to a more complex > >>> >> >> >> >> problem. > >>> >> >> >> >> > >>> >> >> >> >> Really the filesystem shouldn't be used as a locking > >>> >> >> >> >> mechanism. > >>> >> >> >> >> > >>> >> >> >> >> Most of the code that exists for the fslock now is working > >>> >> >> >> >> around > >>> >> >> >> >> its > >>> >> >> >> >> deficiencies. Instead we should be looking for a better > >>> >> >> >> >> replacement. > >>> >> >> >> >> > >>> >> >> >> >> Some "features" that were added to fslock were added to > work > >>> >> >> >> >> around > >>> >> >> >> >> the > >>> >> >> >> >> issue that the lock did not die with the process that > created > >>> >> >> >> >> it, > >>> >> >> >> >> so > >>> >> >> >> >> some mechanism was needed to determine whether the lock > >>> >> >> >> >> should be > >>> >> >> >> >> broken > >>> >> >> >> >> or not. > >>> >> >> >> >> > >>> >> >> >> >> What we really need is a good OS agnostic abstraction that > >>> >> >> >> >> provides > >>> >> >> >> >> the > >>> >> >> >> >> ability to create a "named" lock, acquire the lock, release > >>> >> >> >> >> the > >>> >> >> >> >> lock, > >>> >> >> >> >> and make sure that the lock dies when the process dies, so > >>> >> >> >> >> another > >>> >> >> >> >> process that is waiting can acquire the lock. This way no > >>> >> >> >> >> "BreakLock" > >>> >> >> >> >> functionality is required, nor do we need to try and do > think > >>> >> >> >> >> like > >>> >> >> >> >> remember which process owns the lock. > >>> >> >> >> >> > >>> >> >> >> >> So... > >>> >> >> >> >> > >>> >> >> >> >> We have three current operating systems we need to support: > >>> >> >> >> >> > >>> >> >> >> >> Linux - Ubuntu and CentOS > >>> >> >> >> >> MacOS - client only - bit the CLI uses a lock for the local > >>> >> >> >> >> cache > >>> >> >> >> >> Windows > >>> >> >> >> >> > >>> >> >> >> >> For Linux, and possibly MacOS, flock is a possibility, but > >>> >> >> >> >> can we > >>> >> >> >> >> do > >>> >> >> >> >> better? Is there something that is better suited? > >>> >> >> >> >> > >>> >> >> >> >> For Windows, while you can create global semaphores or > mutex > >>> >> >> >> >> instances, > >>> >> >> >> >> I'm not sure of entities that die with the process. Can > >>> >> >> >> >> people > >>> >> >> >> >> recommend > >>> >> >> >> >> solutions? > >>> >> >> >> >> > >>> >> >> >> >> Cheers, > >>> >> >> >> >> Tim > >>> >> >> >> >> > >>> >> >> >> >> -- > >>> >> >> >> >> Juju-dev mailing list > >>> >> >> >> >> Juju-dev@lists.ubuntu.com > >>> >> >> >> >> Modify settings or unsubscribe at: > >>> >> >> >> >> https://lists.ubuntu.com/mailman/listinfo/juju-dev > >>> >> >> >> > > >>> >> >> >> > > >>> >> >> >> > > >>> >> >> >> > -- > >>> >> >> >> > Juju-dev mailing list > >>> >> >> >> > Juju-dev@lists.ubuntu.com > >>> >> >> >> > Modify settings or unsubscribe at: > >>> >> >> >> > https://lists.ubuntu.com/mailman/listinfo/juju-dev > >>> >> >> >> > > >>> >> >> >> > >>> >> >> >> -- > >>> >> >> >> Juju-dev mailing list > >>> >> >> >> Juju-dev@lists.ubuntu.com > >>> >> >> >> Modify settings or unsubscribe at: > >>> >> >> >> https://lists.ubuntu.com/mailman/listinfo/juju-dev > >> > >> -- > >> Juju-dev mailing list > >> Juju-dev@lists.ubuntu.com > >> Modify settings or unsubscribe at: > >> https://lists.ubuntu.com/mailman/listinfo/juju-dev > > > > > > -- > > Juju-dev mailing list > > Juju-dev@lists.ubuntu.com > > Modify settings or unsubscribe at: > > https://lists.ubuntu.com/mailman/listinfo/juju-dev > > >
-- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
