Re: Depending on libnotify
On 7/4/14, 7:46 AM, David Rajchenbach-Teller wrote: Well, we basically have the Windows version implemented already. We could possibly use Watchman for other platforms. How do others feel about introducing a dependency towards Watchman? The inotify limitation on watchers is indeed annoying in the long run, but ok for the kind of applications we have in mind, so that's ok for me. If you don't need the robustness of watchman, why add a dependency for it? It sounds like a one-off low-level interface to inotify (which you've already built?) is sufficient. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Depending on libnotify
Philip Chee writes: On 02/07/2014 18:13, David Rajchenbach-Teller wrote: We had libnotify support but this was removed from the tree on the grounds that it didn't suit our needs. I don't think those were the grounds. AIUI it was just that people didn't want to put effort into a lower priority platform, so it was easier to remove existing support. But libnotify provides something quite different from the layer on inotify that David wants here. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Depending on libnotify
On 02/07/2014 18:13, David Rajchenbach-Teller wrote: For its recently landed WebIDE, the devtools team needs a component to watch changes in a directory (they are currently using a somewhat hackish workaround that they want to replace with something better supported). Both Windows and MacOS X provide built-in functions for this purpose but most Unices including Linux do not. Under Linux, I would like to use libnotify. It is not quite clear to me whether this is something I can do at the moment. Apparently, we already have dependencies towards libnotify in our gnome bindings and our gtk WebAppRT. We had libnotify support but this was removed from the tree on the grounds that it didn't suit our needs. Phil -- Philip Chee phi...@aleytys.pc.my, philip.c...@gmail.com http://flashblock.mozdev.org/ http://xsidebar.mozdev.org Guard us from the she-wolf and the wolf, and guard us from the thief, oh Night, and so be good for us to pass. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Depending on libnotify
Well, we basically have the Windows version implemented already. We could possibly use Watchman for other platforms. How do others feel about introducing a dependency towards Watchman? I'm not so happy about the idea of using libuv. We already have many ways of performing I/O and introducing yet another one doesn't sound good, unless we decide to make a coordinated effort to migrate Necko towards libuv. Plus, if I recall correctly, libuv makes no effort to allow access to non-POSIXy stuff, which is bad for us. The inotify limitation on watchers is indeed annoying in the long run, but ok for the kind of applications we have in mind, so that's ok for me. Thanks, David On 02/07/14 18:45, Gregory Szorc wrote: I've written a cross-platform file watching API for Windows/OS X/Linux before. It's no fun. -- David Rajchenbach-Teller, PhD Performance Team, Mozilla signature.asc Description: OpenPGP digital signature ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Depending on libnotify
On Wed, Jul 02, 2014 at 12:13:44PM +0200, David Rajchenbach-Teller wrote: For its recently landed WebIDE, the devtools team needs a component to watch changes in a directory (they are currently using a somewhat hackish workaround that they want to replace with something better supported). Both Windows and MacOS X provide built-in functions for this purpose but most Unices including Linux do not. Under Linux, I would like to use libnotify. It is not quite clear to me whether this is something I can do at the moment. Apparently, we already have dependencies towards libnotify in our gnome bindings and our gtk WebAppRT. Can someone shed some light on the subject? libnotify != libinotify. You're talking about the former, but you're thinking about the latter. Note that inotify is not really well suited for some kinds of directory monitoring (so, depends what you really need), and is phased out in favor of fanotify. Also note that glib, which we *do* depend on has a file monitor API, but afaik it's inotify-based and I don't know how well it overcomes the inotify limitations. Mike ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Depending on libnotify
The main usecase is an editor that needs to 1/ reload resources when they have been modified by an outside application; 2/ see when resources are added or removed. In other words, watching a directory (recursively) for add/remove/modify changes to files. Which library do you suggest? Cheers, David On 02/07/14 13:19, Mike Hommey wrote: libnotify != libinotify. You're talking about the former, but you're thinking about the latter. Note that inotify is not really well suited for some kinds of directory monitoring (so, depends what you really need), and is phased out in favor of fanotify. Also note that glib, which we *do* depend on has a file monitor API, but afaik it's inotify-based and I don't know how well it overcomes the inotify limitations. Mike ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform -- David Rajchenbach-Teller, PhD Performance Team, Mozilla signature.asc Description: OpenPGP digital signature ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Depending on libnotify
On Wed, Jul 2, 2014 at 1:41 PM, David Rajchenbach-Teller dtel...@mozilla.com wrote: Which library do you suggest? Take a look at watchman: https://github.com/facebook/watchman Cheers, Dirkjan ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Depending on libnotify
On Wed, Jul 02, 2014 at 02:23:51PM +0200, Dirkjan Ochtman wrote: On Wed, Jul 2, 2014 at 1:41 PM, David Rajchenbach-Teller dtel...@mozilla.com wrote: Which library do you suggest? Take a look at watchman: https://github.com/facebook/watchman That one uses inotify as well. Might as well use glib. Mike ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Depending on libnotify
Hello :-), On 02/07/2014 13:41, David Rajchenbach-Teller wrote: Which library do you suggest? libuv also allows to watch a FS, whatever the platform: https://github.com/joyent/libuv. Cheers. -- Ivan Enderlin Developer of Hoa http://hoa-project.net/ PhD. student at DISC/Femto-ST (Vesontio) and INRIA (Cassis) http://disc.univ-fcomte.fr/ and http://www.inria.fr/ Member of HTML and WebApps Working Group of W3C http://w3.org/ ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Depending on libnotify
On Wed, Jul 02, 2014 at 01:41:54PM +0200, David Rajchenbach-Teller wrote: The main usecase is an editor that needs to 1/ reload resources when they have been modified by an outside application; 2/ see when resources are added or removed. In other words, watching a directory (recursively) for add/remove/modify changes to files. Which library do you suggest? I don't know what the windows and osx file monitoring APIs offer. If they have the same kind of drawbacks as inotify for recursive directory monitoring, then it's probably fine to go with inotify, in which case glib is the best choice, as we already depend on it. Otherwise, I'd go with a little shim around the fanotify system calls. There is fanotify support since glibc 2.13, but I'm not sure that's old enough for all the systems we support. (our current glibc dependency is on a version = 2.7, fwiw) Mike ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Depending on libnotify
According to the manpage of fanotify, « there is no support for create, delete, and move events, » which would make it useless for our scenario. What kind of drawbacks are you talking about in the case of inotify? The possible race condition that [1] attempts to solve? Are there any build incantations needed to use glib? Thanks, David [1] https://mail.gnome.org/archives/dashboard-hackers/2004-October/msg00022.html On 02/07/14 15:07, Mike Hommey wrote: I don't know what the windows and osx file monitoring APIs offer. If they have the same kind of drawbacks as inotify for recursive directory monitoring, then it's probably fine to go with inotify, in which case glib is the best choice, as we already depend on it. Otherwise, I'd go with a little shim around the fanotify system calls. There is fanotify support since glibc 2.13, but I'm not sure that's old enough for all the systems we support. (our current glibc dependency is on a version = 2.7, fwiw) Mike -- David Rajchenbach-Teller, PhD Performance Team, Mozilla signature.asc Description: OpenPGP digital signature ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Depending on libnotify
I've written a cross-platform file watching API for Windows/OS X/Linux before. It's no fun. I would strongly recommend using an existing higher-level abstraction layer because implementing this ourselves will lead to constant edge case discovery and bug fixing. I'd like to think that any reasonably mature abstraction layer has already solved these problems. Facebook's watchman is an excellent project and tool. However, it doesn't support Windows and supporting Windows is a non-goal for them. Porting watchman to Windows would be a lot of work. Now, I'd love to see this done because I'd love to integrate watchman into the build system so we can do things like kick off builds in the background when a file changes. There is also a hgwatchman Mercurial extension that makes `hg status` and other dirstate-crawling APIs nearly instantaneous. Both are very compelling benefits to developer productivity and would fall out nicely if we adopted watchman in Gecko/Firefox. libuv is a contender. Feels a bit bloaty since we won't get much use out of most of the APIs (or could we - perhaps Necko, others could use some of their APIs?). If we integrate libuv, we could statically link it, not export its symbols, and hope the linker will strip enough unused code to get it down to a respectable size. To more directly answer your question, inotify has several limitations. For example, there is no recursive directory watching API. To do that, you need to register watchers on every directory and then add/remove watchers as necessary when directories are added and removed. Furthermore, the default limit on max inotify watchers is pretty conservative - 1024 or so. Large directory trees (like mozilla-central) almost certainly exhaust this limit and require the user adjust kernel limits. IIRC some platforms have limitations where it is possible to overflow the watching subsystem. If there is a massive filesystem change - say a VCS update - some changes can get dropped. If you are relying on watching for anything more than opportunistic event reaction, your implementation complexity quickly grows to deal with recovering from overflow and/or memory bloat. These advanced cases are where I believe watchman excels. Facebook rolled out watchman side-by-side with filesystem-based `hg status` and compared discrepancies among their developer user base until they were confident that watchman was just as accurate as consulting the filesystem. A basic glance at libuv's source reveals that they don't employ these advanced discrepancy mitigation techniques. But if all you care about is opportunistic reaction to some filesystem change, libuv's relative primitives should be satisfactory. On 7/2/14, 6:29 AM, David Rajchenbach-Teller wrote: According to the manpage of fanotify, « there is no support for create, delete, and move events, » which would make it useless for our scenario. What kind of drawbacks are you talking about in the case of inotify? The possible race condition that [1] attempts to solve? Are there any build incantations needed to use glib? Thanks, David [1] https://mail.gnome.org/archives/dashboard-hackers/2004-October/msg00022.html On 02/07/14 15:07, Mike Hommey wrote: I don't know what the windows and osx file monitoring APIs offer. If they have the same kind of drawbacks as inotify for recursive directory monitoring, then it's probably fine to go with inotify, in which case glib is the best choice, as we already depend on it. Otherwise, I'd go with a little shim around the fanotify system calls. There is fanotify support since glibc 2.13, but I'm not sure that's old enough for all the systems we support. (our current glibc dependency is on a version = 2.7, fwiw) Mike ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
