On Jan 25, 2008 12:20 PM, <[EMAIL PROTECTED]> wrote: > "Jay Savage" <[EMAIL PROTECTED]> writes: > > > Finally, Tom's points are important. How do you *know* that the files > > (in this case a single directory) changed *during the sleep*? Do you > > know that the output wasn't buffered? That the system didn't delay the > > writes for some reason? That you were even watching the files you > > thought you were? > > > > Your best bet to make sure the module is working would, instead of > > sleeping between scans and trying to modify the watched files from an > > external process during a narrow window, would probably be to perform > > your initial scan; open one of the watched files for writing; write to > > it; flush the buffer; close it; and then rescan. > > Sounds like a plan for testing but the end result sought in this case > is a monitor that *will* see changes done by external processes in some > reasonable time frame. >
Exactly; it's a plan for testing. As with any test, it's best to vary as few variables as possible in any test. Start with a situation that is completely in your control. Once you can make it work when all the IO is performed in your script, on absolute paths, and that you really understand what File::Monitor is doing, then you can start building the "real world" interface. You'll know at what point in the process the script breaks, and be able to figure out whether you have a problem with buffered IO, or relative pathnames, or whatever. Until you isolate the problem, it's all guesswork. > (Assuming for a moment I do get it figured out with the help in this > thread how to use File::Monitor) > > How long of a sleep would be required to *KNOW* a change has happened > with something external writing to files? Is there no way to guess > that closely... too many variables may come into play? Or could one > create a monitor that will *know* inside of ten seconds that a file > has been written to? > Only you know that, because only you know the behavior of the other process(es) writing to the directories and files you are watching, and the behavior of the OS and filesystem. Does your OS/FS guarantee atomic writes? Is the FS mounted via NFS or some other remote technology? I can think of a number of situations where two processes will get different views of the FS, although not many where they would be out of sync by more than a second. > Would *knowing* about file creation have the same problems? Or is that > something the monitor would *know* for sure quickly? > I don't think you're really asking the question you think you are. Think of IO as a four-step operation. This is an oversimplification, but it will help clarify what File::Monitor can tell you, and when: 1) a process that wants to write, asks the Operating System: "please write this." 2) the Operating System says "ok" 3) the Operating System writes the data to the file system 4) the operating system updates the file's metadata. Step three is actually very complicated in its own right, but we'll ignore that for now. The tricky part for monitoring is that steps 2, 3, and 4 can happen in any order, and with virtually any amount of time between them. Sometime, the Os will tell the writing process that the data has been written when it actually hasn't; that is buffering. In that case, the OS saves up the IO until there is enough to be worth writing. The writing process go about its business--its job is done--but other processes won't see the data yet. Closing the file handle or terminating the writing process will often flush the buffer, but not always; it all depends on the OS. Sometimes, there can also be a lag between writing the data and updating the metadata. And sometimes, the metadata can even be updated first. Most people frown on both these behaviors and most--though not all--modern OS and FS implement what are known as atomic writes, which means that steps 3 and 4 are guaranteed to by synchronized. Where is all this headed? File monitoring relies on Step 4. Step 4 is where the directory pointers and block counts get updated, and the ctime and mtime get rewritten. Once your system performs Step 4, File::Monitor will see the change on its next run. How long that takes will depend on how the writing process is written, and at least partially on what OS its running on. HTH, -- jay -------------------------------------------------- This email and attachment(s): [ ] blogable; [ x ] ask first; [ ] private and confidential daggerquill [at] gmail [dot] com http://www.tuaw.com http://www.downloadsquad.com http://www.engatiki.org values of β will give rise to dom!
