Re: Detect when a file has been recreated using a stat watcher
On Wed, Dec 10, 2008 at 09:12:35PM +0200, Graham Leggett <[EMAIL PROTECTED]> wrote: > Given a stat watcher, how do I detect whether a file has been deleted > and recreated? When a path is not there, st_nlink becomes zero, when it gets recreated nlink goes to one again. > I can detect when a file has been deleted by looking at stat.st_nlink, > but if the file is recreated st_nlink goes back to 1. Yes. > I would have thought that st_ino would change if the file is recreated, > but it doesn't seem to do so. Then your filesystem reuses the same inode number. > Can anyone confirm what the correct way is to handle this? What do you perceive as incorrect? If you are concerned about races, when you are not looking fast enough, then there is no way to avoid those. -- The choice of a Deliantra, the free code+content MORPG -==- _GNU_ http://www.deliantra.net ==-- _ generation ---==---(_)__ __ __ Marc Lehmann --==---/ / _ \/ // /\ \/ / [EMAIL PROTECTED] -=/_/_//_/\_,_/ /_/\_\ ___ libev mailing list libev@lists.schmorp.de http://lists.schmorp.de/cgi-bin/mailman/listinfo/libev
Re: Detect when a file has been recreated using a stat watcher
Marc Lehmann wrote: I would have thought that st_ino would change if the file is recreated, but it doesn't seem to do so. Then your filesystem reuses the same inode number. Can anyone confirm what the correct way is to handle this? What do you perceive as incorrect? If you are concerned about races, when you are not looking fast enough, then there is no way to avoid those. I want to find a way to detect logfile rotation, so the file gets rotated out the way and a new file is created. I have an fd open to the file, which will never increase in size because the file has been deleted, but my app doesn't yet know that. Is there a way to detect this? Regards, Graham -- smime.p7s Description: S/MIME Cryptographic Signature ___ libev mailing list libev@lists.schmorp.de http://lists.schmorp.de/cgi-bin/mailman/listinfo/libev
Re: Detect when a file has been recreated using a stat watcher
On Thu, Dec 11, 2008 at 01:01:00AM +0200, Graham Leggett wrote: > I have an fd open to the file, which will never increase in size because > the file has been deleted, but my app doesn't yet know that. > > Is there a way to detect this? An ev_stat watcher should reliably detect that case, as logfile rotation is not very often done, so the ctime should reliably change. -- The choice of a Deliantra, the free code+content MORPG -==- _GNU_ http://www.deliantra.net ==-- _ generation ---==---(_)__ __ __ Marc Lehmann --==---/ / _ \/ // /\ \/ / p...@goof.com -=/_/_//_/\_,_/ /_/\_\ ___ libev mailing list libev@lists.schmorp.de http://lists.schmorp.de/cgi-bin/mailman/listinfo/libev
Re: Detect when a file has been recreated using a stat watcher
Marc Lehmann wrote: I have an fd open to the file, which will never increase in size because the file has been deleted, but my app doesn't yet know that. Is there a way to detect this? An ev_stat watcher should reliably detect that case, as logfile rotation is not very often done, so the ctime should reliably change. I tried this out on MacOSX 10.5, and found that monitoring ctime for file rotation changes were giving me lots of false positives (the file wasn't rotated at all, but every few seconds I was detecting "changes"). Turns out according to the man pages for the stat structure, ctime is described as follows: struct timespec st_ctimespec; /* time of last file status change */ The clarification of ctime further down is as follows: st_ctime Time when file status was last changed (inode data modi- fication). Changed by the chmod(2), chown(2), link(2), mknod(2), rename(2), unlink(2), utimes(2) and write(2) system calls. The fact that ctime is updated on write() means that in theory, it cannot be used to detect file rotation. There is also a stat64 structure, with higher precision on the file times, as follows: struct timespec st_ctimespec; /* time of last status change */ struct timespec st_birthtimespec; /* time of file creation(birth) */ This second stat64 definition would imply that "last status change" and "file creation" (what I am looking for) are two separate things. My question is, is libev using the stat64 structure by way of comparison? Is there a way for me to detect whether the stat or the stat64 structure is in use at a given time? Is there a cross platform method to determine file rotation, by looking at the attr and prev variables returned by a stat event? Regards, Graham -- smime.p7s Description: S/MIME Cryptographic Signature ___ libev mailing list libev@lists.schmorp.de http://lists.schmorp.de/cgi-bin/mailman/listinfo/libev
Re: Detect when a file has been recreated using a stat watcher
On Mon, Dec 22, 2008 at 08:11:57PM +0200, Graham Leggett wrote: > I tried this out on MacOSX 10.5, and found that monitoring ctime for > file rotation changes were giving me lots of false positives (the file > wasn't rotated at all, but every few seconds I was detecting "changes"). Yes, likely. > st_ctime Time when file status was last changed (inode data > > The fact that ctime is updated on write() means that in theory, it > cannot be used to detect file rotation. Just fyi: write *may* update the ctime, but doesn't do so in every case. It will likely do so in your case, however, as the size changes. I don't understand why it can't be used to detetc logfile rottaion, though? The ctime changes, too, with logfile rotation. > There is also a stat64 structure, with higher precision on the file > times, as follows: Please note that the *structure* has more space for this, but most filesystems do not actually support fractional times. So using stat64 doesn't magically give you higher accuracy. > This second stat64 definition would imply that "last status change" and > "file creation" (what I am looking for) are two separate things. Absolutely. But the ctime likely changes on file creation, or the size, or something else (such as the inode number). This all depend son how your logfiles get rotated - the file might get trucnated, deleted or somethign else might happen. > My question is, is libev using the stat64 structure by way of comparison? No. And it wouldn't help your case unless you require the logifle to be on a specific filesystem supporting higher resolution timestamps. And even then, the higher resolution might sitll just be centiseconds or so. All this doesn't help you. The documentation (that you somehow fail to know, for inexplicable reasons) explains this case in mroe details and offers ways to implement this properly in your app. > Is there a way for me to detect whether the stat or the stat64 structure > is in use at a given time? No, that usually depends on how libev was compiled. Note that stat64 on most unices doesn't have a higher resolution, as stat already provides that, and stat64 is only for LFS. some systems (freebsd for example) sanely default their compilation environment to lfs, others (debian) inexplicably default to 32 bit and try to patch eahc and every app to separately enable lfs, which would presumably give you struct stat64. > Is there a cross platform method to determine file rotation, by looking > at the attr and prev variables returned by a stat event? Yes, but for some reason you claim it wouldn't work. -- The choice of a Deliantra, the free code+content MORPG -==- _GNU_ http://www.deliantra.net ==-- _ generation ---==---(_)__ __ __ Marc Lehmann --==---/ / _ \/ // /\ \/ / p...@goof.com -=/_/_//_/\_,_/ /_/\_\ ___ libev mailing list libev@lists.schmorp.de http://lists.schmorp.de/cgi-bin/mailman/listinfo/libev
Re: Detect when a file has been recreated using a stat watcher
Marc Lehmann wrote: Absolutely. But the ctime likely changes on file creation, or the size, or something else (such as the inode number). This all depend son how your logfiles get rotated - the file might get trucnated, deleted or somethign else might happen. So I should be looking at the size of the file, rather than the ctime of the file? My question is, is libev using the stat64 structure by way of comparison? No. And it wouldn't help your case unless you require the logifle to be on a specific filesystem supporting higher resolution timestamps. And even then, the higher resolution might sitll just be centiseconds or so. All this doesn't help you. The documentation (that you somehow fail to know, for inexplicable reasons) explains this case in mroe details and offers ways to implement this properly in your app. Maybe I am looking at different documentation to you. I am looking at the following URL: http://pod.tst.eu/http://cvs.schmorp.de/libev/ev.pod#code_ev_stat_code_did_the_file_attri The headings I see are "ABI Issues (Largefile Support)", "Inotify and Kqueue", "stat () is a synchronous operation" and "The special problem of stat time resolution". The word "rotate" doesn't appear at all in the docs I am reading. Is there some other docs that I should be reading but am not aware of? Is there a cross platform method to determine file rotation, by looking at the attr and prev variables returned by a stat event? Yes, but for some reason you claim it wouldn't work. So far I was led to believe that ctime can be used to detect file rotation, but the manpage for stat, and the code that I have in testing contradict this. Should I be using the file length instead? Regards, Graham -- smime.p7s Description: S/MIME Cryptographic Signature ___ libev mailing list libev@lists.schmorp.de http://lists.schmorp.de/cgi-bin/mailman/listinfo/libev
Re: Detect when a file has been recreated using a stat watcher
On Tue, Dec 23, 2008 at 08:21:59PM +0200, Graham Leggett wrote: >> This all depend son how your logfiles get rotated - the file might get >> trucnated, deleted or somethign else might happen. > > So I should be looking at the size of the file, rather than the ctime of > the file? I don't know, it depends entirely on how you define "logfile rotation", which probably relies on whatever program does the actual rotation. >> a specific filesystem supporting higher resolution timestamps. And even >> then, the higher resolution might sitll just be centiseconds or so. >> >> All this doesn't help you. The documentation (that you somehow fail to >> know, for inexplicable reasons) explains this case in mroe details and >> offers ways to implement this properly in your app. > > Maybe I am looking at different documentation to you. No, you are just being difficult. We were talking about timestamp resolution, and what to do about it. And this is cetrainly being explained in the docs. > The headings I see are "ABI Issues (Largefile Support)", "Inotify and It might help to read more than just the headings. > "The special problem of stat time resolution". But there it is. > The word "rotate" doesn't appear at all in the > docs I am reading. I was replying to you talking about stat time resolution, and even quoted it, and even *you* quoted it, and then completely ignored it, claiming it has nothing to do with logfile rotation. Think before replying. > Is there some other docs that I should be reading but am not aware of? No, you might be well-advised, however, to actually read the e-mails you read and the replies you get. Much less embarrassing. >>> Is there a cross platform method to determine file rotation, by >>> looking at the attr and prev variables returned by a stat event? >> >> Yes, but for some reason you claim it wouldn't work. > > So far I was led to believe that ctime can be used to detect file > rotation, but the manpage for stat, and the code that I have in testing > contradict this. It can, indeed, as it changes whenever the logfile gets rotated. Other things might change, too, such as the filesize, or the inode number, all of which would be hints that the logfile actually was rotated. If you are looking for a logfile rotation watcher in libev, then sorry, there isn't one, you still have to know what you are doing. > Should I be using the file length instead? I don't think so - but it all depends on how you define it. Note that the only sure way of detecting logfile rotation in the general case is to let the program that rotates the log tell you. Eberything else is just heuristics. Note that, if you get more than a log message per second, then ev_stat likely isn't the most efficient way to detect that. -- The choice of a Deliantra, the free code+content MORPG -==- _GNU_ http://www.deliantra.net ==-- _ generation ---==---(_)__ __ __ Marc Lehmann --==---/ / _ \/ // /\ \/ / p...@goof.com -=/_/_//_/\_,_/ /_/\_\ ___ libev mailing list libev@lists.schmorp.de http://lists.schmorp.de/cgi-bin/mailman/listinfo/libev
Re: Detect when a file has been recreated using a stat watcher
Marc Lehmann wrote: I don't know, it depends entirely on how you define "logfile rotation", which probably relies on whatever program does the actual rotation. As in the file gets deleted and/or truncated, and the event driven application watching the file gets notified by some mechanism that this has happened. We were talking about timestamp resolution, and what to do about it. And this is cetrainly being explained in the docs. Er, no, I think you have misunderstood my question. I am not talking at all about timestamp resolution, a concept which you are 100% correct is adequately explained in the docs. What I am talking about is the detection of when a logfile is moved/deleted/truncated and a new logfile is started from scratch. From my event driven app, I need to detect when this has occurred, and close and reopen the logfile in question. (tail -F, in other words) I don't think so - but it all depends on how you define it. Note that the only sure way of detecting logfile rotation in the general case is to let the program that rotates the log tell you. Eberything else is just heuristics. Hmmm... Both inotify and kqueue can (apparently) detect file deletion/renaming events, although it seems that libev does not (yet) expose these events via any API as yet. Regards, Graham -- smime.p7s Description: S/MIME Cryptographic Signature ___ libev mailing list libev@lists.schmorp.de http://lists.schmorp.de/cgi-bin/mailman/listinfo/libev
Re: Detect when a file has been recreated using a stat watcher
On Wed, Dec 24, 2008 at 12:40:46PM +0200, Graham Leggett wrote: > >I don't know, it depends entirely on how you define "logfile rotation", > >which probably relies on whatever program does the actual rotation. > > As in the file gets deleted and/or truncated, and the event driven > application watching the file gets notified by some mechanism that this > has happened. Well, the ctime would likely change in those cases, as well as (likely) the inode number and (likely) the size, all of which could be detected via a stat watcher. > >We were talking about timestamp resolution, and what to do about > >it. And this is cetrainly being explained in the docs. > > Er, no, I think you have misunderstood my question. No, you completely midrepresent your e-mail, and that's no fun. > I am not talking at all about timestamp resolution, You *were* talking about stat time resolution. > Both inotify and kqueue can (apparently) detect file deletion/renaming > events, Firts of all, no, they can't, at least not with certainty. This detection only works on some filesystems for example. > although it seems that libev does not (yet) expose these events libev *caqnnot* expose these events as such because a) libev tries to be reliable and b) libev is a portable library, and the only thing available portably is stat. libev is *not* a wrapper around inotify, it only *uses* inotify to implement its (portable) stat watchers, and it only uses inotify as a hint mechanism (and in this case it actually does check for file deletions). So if you want to write an OS X only program, then you should be free to use kqueue, similarly for gnu/linux, but it's not acceptable for libev to become, say, a linux-only library. -- The choice of a Deliantra, the free code+content MORPG -==- _GNU_ http://www.deliantra.net ==-- _ generation ---==---(_)__ __ __ Marc Lehmann --==---/ / _ \/ // /\ \/ / p...@goof.com -=/_/_//_/\_,_/ /_/\_\ ___ libev mailing list libev@lists.schmorp.de http://lists.schmorp.de/cgi-bin/mailman/listinfo/libev
