Re: inotify, new idea?
Am 26.05.2014 01:46, schrieb Marian Marinov: > Have anyone of you looked at this: https://github.com/1and1/linux-filemon/ > > I haven't stress tested it, but in the past I ported it to more recent > kernels: > https://github.com/hackman/filemon-patches > > It is not polished, but it works. > > I'm considering fixing some of the issues at has for use in my home setup. Well, the patch itself has documented its major flaw: + * WARNING! This patch has been designed as EXPERIMENTAL. Its usage + * is DANGEROUS, because some filesystems could get exhausted by the + * masses of ORPHAN INODES! Thanks, //richard signature.asc Description: OpenPGP digital signature
Re: inotify, new idea?
On 05/24/2014 02:34 PM, Richard Weinberger wrote: > Am 24.05.2014 09:52, schrieb Michael Kerrisk (man-pages): >> On 04/21/2014 10:42 AM, Richard Weinberger wrote: >>> Am 21.04.2014 09:24, schrieb Michael Kerrisk: > Does recursive monitoring even work with inotify? > Last time I've tried it did failed as soon I did a mkdir -p a/b/c/d > because > mkdir() raced against the thread which installes the new watches. As I understand it, you have to program to deal with the races (rescan directories after adding watches). I recently did a lot of work updating the inotify(7) man page to discuss all the issues that I know of, and their remedies. If I missed anything, I'd appreciate a note on it, so that it can be added. See http://man7.org/linux/man-pages/man7/inotify.7.html#NOTES >>> >>> I'm aware of the rescan hack, but in my case it does not help >>> because my program must not miss any event. >>> Currently I'm using a fuse overlay filesystem to log everything. >>> Not perfect but works... :-) >> >> Richard, >> >> A late follow up question. How does your application deal with the >> event overflow problem (i.e., when you get a large number of events >> much faster than your application can deal with them? > > The downside of the FUSE approach is that you have to intercept > every filesystem function. > This can be a performance issue. > But due to this design the overflow problem cannot happen as the > FUSE filesystem blocks until the event has been proceed. Ahh -- that clears things up for me. Thanks, Richard. Cheers, Michael -- Michael Kerrisk Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/ Linux/UNIX System Programming Training: http://man7.org/training/ -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: inotify, new idea?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 05/24/2014 03:34 PM, Richard Weinberger wrote: > Am 24.05.2014 09:52, schrieb Michael Kerrisk (man-pages): >> On 04/21/2014 10:42 AM, Richard Weinberger wrote: >>> Am 21.04.2014 09:24, schrieb Michael Kerrisk: > Does recursive monitoring even work with inotify? Last time I've tried it > did failed as soon I did a mkdir > -p a/b/c/d because mkdir() raced against the thread which installes the > new watches. As I understand it, you have to program to deal with the races (rescan directories after adding watches). I recently did a lot of work updating the inotify(7) man page to discuss all the issues that I know of, and their remedies. If I missed anything, I'd appreciate a note on it, so that it can be added. See http://man7.org/linux/man-pages/man7/inotify.7.html#NOTES >>> >>> I'm aware of the rescan hack, but in my case it does not help because my >>> program must not miss any event. >>> Currently I'm using a fuse overlay filesystem to log everything. Not >>> perfect but works... :-) >> >> Richard, >> >> A late follow up question. How does your application deal with the event >> overflow problem (i.e., when you get a >> large number of events much faster than your application can deal with them? > > The downside of the FUSE approach is that you have to intercept every > filesystem function. This can be a > performance issue. But due to this design the overflow problem cannot happen > as the FUSE filesystem blocks until > the event has been proceed. > Have anyone of you looked at this: https://github.com/1and1/linux-filemon/ I haven't stress tested it, but in the past I ported it to more recent kernels: https://github.com/hackman/filemon-patches It is not polished, but it works. I'm considering fixing some of the issues at has for use in my home setup. Marian > Thanks, //richard -- To unsubscribe from this list: send the line > "unsubscribe linux-kernel" in the body of a > message to majord...@vger.kernel.org More majordomo info at > http://vger.kernel.org/majordomo-info.html Please read > the FAQ at http://www.tux.org/lkml/ > - -- Marian Marinov Founder & CEO of 1H Ltd. Jabber/GTalk: hack...@jabber.org ICQ: 7556201 Mobile: +359 886 660 270 -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.22 (GNU/Linux) iEYEARECAAYFAlOCgOwACgkQ4mt9JeIbjJSXrgCdHCFADxvKN3Hxb4qinc/5Fj6V WYMAnArk3Zb8YqTUNoyNCh1zYplUUx9B =y/Sm -END PGP SIGNATURE- -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: inotify, new idea?
Am 24.05.2014 09:52, schrieb Michael Kerrisk (man-pages): > On 04/21/2014 10:42 AM, Richard Weinberger wrote: >> Am 21.04.2014 09:24, schrieb Michael Kerrisk: Does recursive monitoring even work with inotify? Last time I've tried it did failed as soon I did a mkdir -p a/b/c/d because mkdir() raced against the thread which installes the new watches. >>> >>> As I understand it, you have to program to deal with the races (rescan >>> directories after adding watches). I recently did a lot of work >>> updating the inotify(7) man page to discuss all the issues that I know >>> of, and their remedies. If I missed anything, I'd appreciate a note on >>> it, so that it can be added. See >>> http://man7.org/linux/man-pages/man7/inotify.7.html#NOTES >> >> I'm aware of the rescan hack, but in my case it does not help >> because my program must not miss any event. >> Currently I'm using a fuse overlay filesystem to log everything. >> Not perfect but works... :-) > > Richard, > > A late follow up question. How does your application deal with the > event overflow problem (i.e., when you get a large number of events > much faster than your application can deal with them? The downside of the FUSE approach is that you have to intercept every filesystem function. This can be a performance issue. But due to this design the overflow problem cannot happen as the FUSE filesystem blocks until the event has been proceed. Thanks, //richard -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: inotify, new idea?
On 04/21/2014 10:42 AM, Richard Weinberger wrote: > Am 21.04.2014 09:24, schrieb Michael Kerrisk: >>> Does recursive monitoring even work with inotify? >>> Last time I've tried it did failed as soon I did a mkdir -p a/b/c/d because >>> mkdir() raced against the thread which installes the new watches. >> >> As I understand it, you have to program to deal with the races (rescan >> directories after adding watches). I recently did a lot of work >> updating the inotify(7) man page to discuss all the issues that I know >> of, and their remedies. If I missed anything, I'd appreciate a note on >> it, so that it can be added. See >> http://man7.org/linux/man-pages/man7/inotify.7.html#NOTES > > I'm aware of the rescan hack, but in my case it does not help > because my program must not miss any event. > Currently I'm using a fuse overlay filesystem to log everything. > Not perfect but works... :-) Richard, A late follow up question. How does your application deal with the event overflow problem (i.e., when you get a large number of events much faster than your application can deal with them? Cheers, Michael -- Michael Kerrisk Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/ Linux/UNIX System Programming Training: http://man7.org/training/ -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: inotify, new idea?
Seems like also 'mv' events are also not supported by fanotify, like creation/deletion. Therefore not an option (for lsyncd) probably. Maybe loggedfs is (if time permits...). Thanks for the discussion. Jos On Mon, Apr 21, 2014 at 5:01 PM, Richard Weinberger wrote: > > > Am 21.04.2014 15:31, schrieb Michael Kerrisk (man-pages): >> On 04/21/2014 10:42 AM, Richard Weinberger wrote: >>> Am 21.04.2014 09:24, schrieb Michael Kerrisk: > Does recursive monitoring even work with inotify? > Last time I've tried it did failed as soon I did a mkdir -p a/b/c/d > because > mkdir() raced against the thread which installes the new watches. As I understand it, you have to program to deal with the races (rescan directories after adding watches). I recently did a lot of work updating the inotify(7) man page to discuss all the issues that I know of, and their remedies. If I missed anything, I'd appreciate a note on it, so that it can be added. See http://man7.org/linux/man-pages/man7/inotify.7.html#NOTES >>> >>> I'm aware of the rescan hack, but in my case it does not help >>> because my program must not miss any event. >> >> Then, we're understand the same thing: you're out of luck :-}. >> >>> Currently I'm using a fuse overlay filesystem to log everything. >>> Not perfect but works... :-) >> >> Interesting notion. I need to get to grips with FUSE... > > e.g. http://loggedfs.sourceforge.net/ > > Thanks, > //richard -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: inotify, new idea?
Am 21.04.2014 15:31, schrieb Michael Kerrisk (man-pages): > On 04/21/2014 10:42 AM, Richard Weinberger wrote: >> Am 21.04.2014 09:24, schrieb Michael Kerrisk: Does recursive monitoring even work with inotify? Last time I've tried it did failed as soon I did a mkdir -p a/b/c/d because mkdir() raced against the thread which installes the new watches. >>> >>> As I understand it, you have to program to deal with the races (rescan >>> directories after adding watches). I recently did a lot of work >>> updating the inotify(7) man page to discuss all the issues that I know >>> of, and their remedies. If I missed anything, I'd appreciate a note on >>> it, so that it can be added. See >>> http://man7.org/linux/man-pages/man7/inotify.7.html#NOTES >> >> I'm aware of the rescan hack, but in my case it does not help >> because my program must not miss any event. > > Then, we're understand the same thing: you're out of luck :-}. > >> Currently I'm using a fuse overlay filesystem to log everything. >> Not perfect but works... :-) > > Interesting notion. I need to get to grips with FUSE... e.g. http://loggedfs.sourceforge.net/ Thanks, //richard -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: inotify, new idea?
On 04/21/2014 10:42 AM, Richard Weinberger wrote: > Am 21.04.2014 09:24, schrieb Michael Kerrisk: >>> Does recursive monitoring even work with inotify? >>> Last time I've tried it did failed as soon I did a mkdir -p a/b/c/d because >>> mkdir() raced against the thread which installes the new watches. >> >> As I understand it, you have to program to deal with the races (rescan >> directories after adding watches). I recently did a lot of work >> updating the inotify(7) man page to discuss all the issues that I know >> of, and their remedies. If I missed anything, I'd appreciate a note on >> it, so that it can be added. See >> http://man7.org/linux/man-pages/man7/inotify.7.html#NOTES > > I'm aware of the rescan hack, but in my case it does not help > because my program must not miss any event. Then, we're understand the same thing: you're out of luck :-}. > Currently I'm using a fuse overlay filesystem to log everything. > Not perfect but works... :-) Interesting notion. I need to get to grips with FUSE... -- Michael Kerrisk Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/ Linux/UNIX System Programming Training: http://man7.org/training/ -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: inotify, new idea?
Am 21.04.2014 09:24, schrieb Michael Kerrisk: >> Does recursive monitoring even work with inotify? >> Last time I've tried it did failed as soon I did a mkdir -p a/b/c/d because >> mkdir() raced against the thread which installes the new watches. > > As I understand it, you have to program to deal with the races (rescan > directories after adding watches). I recently did a lot of work > updating the inotify(7) man page to discuss all the issues that I know > of, and their remedies. If I missed anything, I'd appreciate a note on > it, so that it can be added. See > http://man7.org/linux/man-pages/man7/inotify.7.html#NOTES I'm aware of the rescan hack, but in my case it does not help because my program must not miss any event. Currently I'm using a fuse overlay filesystem to log everything. Not perfect but works... :-) Thanks, //richard -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: inotify, new idea?
On Mon, Apr 21, 2014 at 12:22 AM, Richard Weinberger wrote: > On Fri, Apr 18, 2014 at 6:15 AM, Michael Kerrisk > wrote: >> On Thu, Apr 17, 2014 at 11:28 PM, Lennart Sorensen >> wrote: >>> On Thu, Apr 17, 2014 at 11:00:37PM +0200, Jos Huisken wrote: I was trying to maintain a local and remote directory in sync with lsync, using inotify. I happen to have >4M files and >400k directories... running over /proc/sys/fs/inotify/max_user_watches >>> >>> Would fanotify perhaps be a better interface to use? >> >> (One of us is misunderstanding fanotify; it might be me.) >> >> Did you look at fanotify closely? I don't think it could be used for >> this task -- does not notify linka dn unlink events, difficult to set >> up recursive monitoring, etc. > > Does recursive monitoring even work with inotify? > Last time I've tried it did failed as soon I did a mkdir -p a/b/c/d because > mkdir() raced against the thread which installes the new watches. As I understand it, you have to program to deal with the races (rescan directories after adding watches). I recently did a lot of work updating the inotify(7) man page to discuss all the issues that I know of, and their remedies. If I missed anything, I'd appreciate a note on it, so that it can be added. See http://man7.org/linux/man-pages/man7/inotify.7.html#NOTES Cheers, Michael -- Michael Kerrisk Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/ Author of "The Linux Programming Interface", http://blog.man7.org/ -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: inotify, new idea?
On Fri, Apr 18, 2014 at 6:15 AM, Michael Kerrisk wrote: > On Thu, Apr 17, 2014 at 11:28 PM, Lennart Sorensen > wrote: >> On Thu, Apr 17, 2014 at 11:00:37PM +0200, Jos Huisken wrote: >>> I was trying to maintain a local and remote directory in sync with >>> lsync, using inotify. >>> I happen to have >4M files and >400k directories... running over >>> /proc/sys/fs/inotify/max_user_watches >> >> Would fanotify perhaps be a better interface to use? > > (One of us is misunderstanding fanotify; it might be me.) > > Did you look at fanotify closely? I don't think it could be used for > this task -- does not notify linka dn unlink events, difficult to set > up recursive monitoring, etc. Does recursive monitoring even work with inotify? Last time I've tried it did failed as soon I did a mkdir -p a/b/c/d because mkdir() raced against the thread which installes the new watches. -- Thanks, //richard -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: inotify, new idea?
On Fri, Apr 18, 2014 at 06:15:40AM +0200, Michael Kerrisk wrote: > (One of us is misunderstanding fanotify; it might be me.) No I think you are right, and that the misunderstanding is on my part. I had not realized that fanotify did not provide create/delete events, only access events. > Did you look at fanotify closely? I don't think it could be used for > this task -- does not notify linka dn unlink events, difficult to set > up recursive monitoring, etc. It would appear you are correct. -- Len Sorensen -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: inotify, new idea?
On Thu, Apr 17, 2014 at 11:28 PM, Lennart Sorensen wrote: > On Thu, Apr 17, 2014 at 11:00:37PM +0200, Jos Huisken wrote: >> I was trying to maintain a local and remote directory in sync with >> lsync, using inotify. >> I happen to have >4M files and >400k directories... running over >> /proc/sys/fs/inotify/max_user_watches > > Would fanotify perhaps be a better interface to use? (One of us is misunderstanding fanotify; it might be me.) Did you look at fanotify closely? I don't think it could be used for this task -- does not notify linka dn unlink events, difficult to set up recursive monitoring, etc. Cheers, Michael -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: inotify, new idea?
On Thu, Apr 17, 2014 at 11:00:37PM +0200, Jos Huisken wrote: > I was trying to maintain a local and remote directory in sync with > lsync, using inotify. > I happen to have >4M files and >400k directories... running over > /proc/sys/fs/inotify/max_user_watches Would fanotify perhaps be a better interface to use? -- Len Sorensen -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
