Re: Continually count the number of open files

[Mateusz Guzik]

On 9/13/23, David Chisnall  wrote:
> On 12 Sep 2023, at 17:19, Bakul Shah  wrote:
>>
>> FreeBSD
>> should add inotify.
>
> inotify is also probably not the right thing.  If someone is interested in
> adding this, Apple’s fsevents API is a better inspiration.  It is carefully
> designed to ensure that the things monitoring for events can’t ever block
> filesystem operations from making progress.

I'm not sure what you mean here specifically and I don't see anything
careful about what they did.

>From userspace POV the API is allowed to drop events, which makes life
easy on this front and is probably the right call.

The implementation is utterly horrid. For one, the non-blocking aspect
starts with the obvious equivalent of uma_zalloc(..., M_NOWAIT) and
bailing if it fails, except if you read past that to actual
registration it can perform an alloc which can block indefinitely
while holding on to some vnodes:
// if we haven't gotten the path yet, get it.
if (pathbuff == NULL) {
pathbuff = get_pathbuff();
pathbuff_len = MAXPATHLEN;

where get_pathbuf is:
return zalloc(ZV_NAMEI);

So the notification routine can block indefinitely in a low-memory
condition. I tried to figure out if this is ever called without other
vnodes write-locked (as opposed to "just" refed), but their code is
such a mess that my level of curiosity was dwarfed by difficulty of
getting a definitive answer.

Other than that it is terribly inefficient and artificially limited to
8 processes which can do anything.

That is to say it is unfit for anything but laptop-scale usage.

Perhaps you meant it does not block if the watchers decide to not
process any events, but that's almost inherently true if one allows
for lossy notifications.

> I think there’s a nice design
> possible with a bloom filter in the kernel of events that ensures that
> monitors may get spurious events but don’t miss out on anything.
>
[snip]
>  I think the right kernel API would walk the directory and add the vnodes to
> a bloom filter and trigger a notification on a match in the filter.  You’d
> then have occasional spurious notifications but you’d have something that
> could be monitored via kqueue and could be made to not block anything else
> in the kernel.
>

I don't see how this can work.

A directory can have more inodes than you can have vnodes at any
point. So if you add vnodes to a list as you go, they may fall off of
so that you can continue adding other entries.

But perhaps you mean you could store the inode number as opposed to
holding to the vnode? Even then, the number of entries to scan to make
it happen is so big that it is going to be impractical on anything but
laptop-scale.

What can be fast is checking if the parent dir wants notifications,
but this ignores changes to hardlinks. Except *currently* the VFS
layer does not reliably track who the parent is (and in fact it can
fail to spot one).

The VFS layer contains a lot of cruft and design decisions which at
least today are questionable at best, but fixable. A big chunk of this
concerns name caching, which currently is entirely optional. Should
someone want to propose an API for file notification changes, they
need to state something which if implemented does not result in
unfixable drag on the layer, even if initial implementation would be
suboptimal. Handling arbitrary hardlinks looks like a drag to me, but
I'm happy to review an implementation which avoids being a problem.

That is to say, a laptop-scale API can probably be implemented as is,
but solution which can provide reliable events (not to be confused
with reliably notifying about all events) would require numerous
changes.

-- 
Mateusz Guzik 

Re: Continually count the number of open files

[Tomoaki AOKI]

On Wed, 13 Sep 2023 11:52:19 -0700
Bakul Shah  wrote:

> On Sep 12, 2023, at 11:59 PM, Graham Perrin  wrote:
> > 
> > (I'm a tcsh user, I can easily 'sh' before running the command.)
> 
> You can switch to zsh. Most of csh/tcsh + sh + many more features.
> 
> > baloo is not used in 273669.
> 
> It certainly feels like an inotify like use or a file-descr leak.
> The bug reporter can try "procstat fd " on running processes
> to see which one has all those open files. Another thing worth
> trying is to run under ktrace -di to see which syscalls were made.

Additional note.

For emergency I heve a line below in ~/.tcshrc.mine and a flag file
~/.Use_zsh.
This way, you can switch back to tcsh by deleting the flag file
whenever you want. No need to update master.passwd entry, as the login
shell itself is still tcsh and zsh is `exec`'ed from tcsh.

if ( -X zsh && -f ~/.Use_zsh ) exec zsh

-- 
Tomoaki AOKI

Re: Continually count the number of open files

[Bakul Shah]

On Sep 12, 2023, at 11:59 PM, Graham Perrin  wrote:
> 
> (I'm a tcsh user, I can easily 'sh' before running the command.)

You can switch to zsh. Most of csh/tcsh + sh + many more features.

> baloo is not used in 273669.

It certainly feels like an inotify like use or a file-descr leak.
The bug reporter can try "procstat fd " on running processes
to see which one has all those open files. Another thing worth
trying is to run under ktrace -di to see which syscalls were made.

Re: Continually count the number of open files

[David Chisnall]

On 12 Sep 2023, at 17:19, Bakul Shah  wrote:
> 
> FreeBSD
> should add inotify.

inotify is also probably not the right thing.  If someone is interested in 
adding this, Apple’s fsevents API is a better inspiration.  It is carefully 
designed to ensure that the things monitoring for events can’t ever block 
filesystem operations from making progress.  I think there’s a nice design 
possible with a bloom filter in the kernel of events that ensures that monitors 
may get spurious events but don’t miss out on anything.  

On macOS because files have a stronger idea of which directory they live in and 
so it’s easier to have an API that notifies for changes to files in a 
directory.  inotify monitors individual files but loses notifications if you 
write through the wrong hard link to a file (hard link foo from a/foo to b/foo, 
use inotify to watch a, write though b/foo, observe no notification).  I think 
the right kernel API would walk the directory and add the vnodes to a bloom 
filter and trigger a notification on a match in the filter.  You’d then have 
occasional spurious notifications but you’d have something that could be 
monitored via kqueue and could be made to not block anything else in the kernel.

If anyone is interested in improving the current kqueue code here: there’s 
currently no mechanism for tracking when the last writable file descriptor for 
a file has been closed, which is useful for consuming files that are dropped 
via sftp or similar.  NOTE_CLOSED_WRITE is hard to use without race conditions 
and tells you only that *a* file descriptor with write permission has closed, 
not that the last one has closed.  I’m currently resorting to a process that 
runs as root that uses libprocstat to walk the entire list of open file 
descriptors and report that they’re closed, which is incredibly suboptimal.

David

Re: Continually count the number of open files

[Graham Perrin]

On 12/09/2023 17:19, Bakul Shah wrote:

On Sep 11, 2023, at 11:38 PM, Graham Perrin  wrote:

Can anything like systat(1) present a count, continually?

How about

while sleep 0.1; do sysctl -n kern.openfiles; done



That's ideal, thanks. I knew about the sysctl, but not how to form a 
command like the one above.


(I'm a tcsh user, I can easily 'sh' before running the command.)



Or you can write a small program using sysctl(3).


I'd like to monitor, after log in to Plasma (X11), in connection with 
.

Not sure checking how many files are open will help you.
Looks like "baloo" is using inotify to watch changes on
every file & directory or something. Simulating inotify
with kqueue under FreeBSD doesn't scale well. FreeBSD
should add inotify.


baloo is not used in 273669.

I'm beginning to investigate something unrelated to KDE that has bugged 
me, on FreeBSD, for a very long time.


Thanks

Re: Continually count the number of open files

[Bakul Shah]

On Sep 11, 2023, at 11:38 PM, Graham Perrin  wrote:
> 
> Can anything like systat(1) present a count, continually?

How about 

while sleep 0.1; do sysctl -n kern.openfiles; done

Or you can write a small program using sysctl(3).

> 
> I'd like to monitor, after log in to Plasma (X11), in connection with 
> .

Not sure checking how many files are open will help you.
Looks like "baloo" is using inotify to watch changes on
every file & directory or something. Simulating inotify
with kqueue under FreeBSD doesn't scale well. FreeBSD
should add inotify.

Re: Continually count the number of open files

[Ronald Klop]

On 9/12/23 08:38, Graham Perrin wrote:

Can anything like systat(1) present a count, continually?

I'd like to monitor, after log in to Plasma (X11), in connection with 
.





I don't think you mean:
# sysctl -d kern.openfiles
kern.openfiles: System-wide number of open files

It is more like a gauge than a counter.

Regards,
Ronald.

Continually count the number of open files

[Graham Perrin]

Can anything like systat(1) present a count, continually?

I'd like to monitor, after log in to Plasma (X11), in connection with 
.