Re: inotify for BSD?

[William Ahern]

On Wed, May 15, 2013 at 01:52:45PM +0200, Peter J. Philipp wrote:
> On 05/15/13 13:41, Jérémie Courrèges-Anglas wrote:
> >Doesn't kqueue() fit your needs?
> >
> 
> Thank you for your reply,
> 
> I've never used kqueue before, does this only report events on 
> descriptors that have been opened?

Yes, but that's a good thing, IMO.
 
> I'm wondering if an implementation is done to recurseively watch 
> directories in inotify (as written about in the limitations), then it 
> would require a lot less filedescriptors even for kqueue correct?  And 
> thus make monitoring a filesystem's events a lot more efficient?
> 

inotify doesn't report recursively either.

The problem with inotify is handling overflow. The kernel can't be expected
to buffer every single event forever. Eventually it starts to drop events,
so no matter what interface you use, you always need to write code to
manually walk directories and look for additions, deletions, or updates
yourself. So inotify isn't as slick as it appears on its face.

The nice thing about kqueue is that with a reference to the descriptor
events are never lost--they just flip a bit in the descriptor book-keeping.
That has downsides, obviously, but for the most part makes it easier to use.

For my Lua cqueues* project I've written a small notification library in a
single C source file which can use BSD kqueue EVFILT_VNODE, Linux inotify,
and Solaris port_associate PORT_SOURCE_FILE. You can find it in
src/lib/notify.c, and you should be able to use it standalone with no or
minimal tweaking. The C interface is basically the same as the Lua interface
described in the User Guide--you create a notification object bound to a
single directory using notify_opendir(), add files to watch with
notify_add(), process events with notify_step(), retrieve notifications with
notify_get(), and use notify_pollfd() and notify_timeout() for interfacing
with your event loop.

It doesn't handle recursion, and in fact doesn't even handle overflow yet
(it sets a critical flag which I need to write code to process). I also want
to add globbing support so that, e.g., "*" would report on all files in a
directory. It's a work-in-progress, but might help you understand how to use
the different APIs.


* http://25thandclement.com/~william/projects/cqueues.html

Re: inotify for BSD?

[Tomas Bodzar]

On Wed, May 15, 2013 at 1:25 PM, Peter J. Philipp  wrote:

> Hi,
>
> This is a question for devs really.  An inotify for BSD would be useful
> for me.  The URL for inotify explanation is at wikipedia:
> http://en.wikipedia.org/wiki/**Inotify, 
> would you say something like this being added to /sys/kern/vfs_vops.c
> would be the right place for it?  If it's finished would something like
> this be included in OpenBSD, or would one have to maintain external patches
> across releases?
>
> Thanks,
>
> -peter
>
>
Maybe this http://www.tedunangst.com/kqueue.pdf and this
http://people.freebsd.org/~jlemon/papers/kqueue.pdf (Dfly/FBSD related) can
help you somewhat.

Re: inotify for BSD?

[Antoine Jacoutot]

On Wed, May 15, 2013 at 01:25:53PM +0200, Peter J. Philipp wrote:
> Hi,
> 
> This is a question for devs really.  An inotify for BSD would be
> useful for me.  The URL for inotify explanation is at wikipedia:
> http://en.wikipedia.org/wiki/Inotify , would you say something like
> this being added to /sys/kern/vfs_vops.c would be the right place
> for it?  If it's finished would something like this be included in
> OpenBSD, or would one have to maintain external patches across
> releases?

There was an upcoming effort, but it's unfinished:
http://dmitrymatveev.co.uk/gsoc11?tag=libinotify

git clone git://github.com/dmatveev/libinotify-kqueue.git

-- 
Antoine

Re: inotify for BSD?

[Peter J. Philipp]

On 05/15/13 14:20, Jérémie Courrèges-Anglas wrote:

"Peter J. Philipp"  writes:


On 05/15/13 13:41, Jérémie Courrèges-Anglas wrote:

Doesn't kqueue() fit your needs?


Thank you for your reply,

I've never used kqueue before, does this only report events on descriptors
that have been opened?

I think so.


Ok, hmm.  My box has 162,000 directories as found with a find / -type d 
-print | wc -l, I'd like to monitor the entire tree and I don't want to 
open 162,000 descriptors to see if someone opened a file in some remote 
corner of my system.


Do you think kqueue can be reworked to look at entire directory trees?  
I'm almost a believer in what it can do now that you pointed it out to 
me and I read the manpage a little.  I just know too little about it to 
judge whether the code allows modifications to look at entire directory 
trees.



I'm wondering if an implementation is done to recurseively watch directories
in inotify (as written about in the limitations), then it would require a lot
less filedescriptors even for kqueue correct?  And thus make monitoring
a filesystem's events a lot more efficient?

As is, kqueue() won't monitor a directory tree recursively.  But there
are examples of kqueue() use; see for example the sysutils/gamin ports
(also devel/glib2 uses it for GIOs, I think).


Thanks, I'll take a look.

-peter

Re: inotify for BSD?

[Jérémie Courrèges-Anglas]

"Peter J. Philipp"  writes:

> On 05/15/13 13:41, Jérémie Courrèges-Anglas wrote:
>> Doesn't kqueue() fit your needs?
>>
>
> Thank you for your reply,
>
> I've never used kqueue before, does this only report events on descriptors
> that have been opened?

I think so.

> I'm wondering if an implementation is done to recurseively watch directories
> in inotify (as written about in the limitations), then it would require a lot
> less filedescriptors even for kqueue correct?  And thus make monitoring
> a filesystem's events a lot more efficient?

As is, kqueue() won't monitor a directory tree recursively.  But there
are examples of kqueue() use; see for example the sysutils/gamin ports
(also devel/glib2 uses it for GIOs, I think).

> -peter

-- 
Jérémie Courrèges-Anglas
PGP Key fingerprint: 61DB D9A0 00A4 67CF 2A90  8961 6191 8FBF 06A1 1494

Re: inotify for BSD?

[Peter J. Philipp]

On 05/15/13 13:41, Jérémie Courrèges-Anglas wrote:

Doesn't kqueue() fit your needs?



Thank you for your reply,

I've never used kqueue before, does this only report events on 
descriptors that have been opened?


I'm wondering if an implementation is done to recurseively watch 
directories in inotify (as written about in the limitations), then it 
would require a lot less filedescriptors even for kqueue correct?  And 
thus make monitoring a filesystem's events a lot more efficient?


-peter

Re: inotify for BSD?

[Jérémie Courrèges-Anglas]

Doesn't kqueue() fit your needs?

-- 
Jérémie Courrèges-Anglas
PGP Key fingerprint: 61DB D9A0 00A4 67CF 2A90  8961 6191 8FBF 06A1 1494