Re: Hmm....is a hot directory possible?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I think this is the closest thing to what I am looking for. I will read up on this and plug it into the project I am working on. Thanks! On Sunday 29 June 2003 11:16, Harry Putnam wrote: > "Wiggins d'Anconia" <[EMAIL PROTECTED]> writes: > > FAM provides something similar and I believe uses hooks in the linux > > kernel on that system and other means on other *nixes. There even > > exists a Perl module to hook into it (SGI::FAM) good luck getting it > > to work though, I have had some problems in the recent past as have > > other posters. > > Following up on this a little, and in case the OP is still looking > for something. I found this application on FAM sites > > fileschanged > > http://fileschanged.sourceforge.net/ > > A command line tool that uses FAM. > >$ fileschanged -r -f /home/myhomedir > >...will monitor all files in your home directory, and all >directories and subdirectories in it. For example if you were to >receive mail, while you were monitoring your home directory you >might see something like: -BEGIN PGP SIGNATURE- Version: GnuPG v1.0.7 (GNU/Linux) iD8DBQE/AESqkXvEbfjFKqoRAtlOAJ9QmsALQ8xUK4URSHKPkGnSrdZyWACfe9cY k//jgK4hSUfOwyEY4ZSS8tw= =V0Bt -END PGP SIGNATURE- -- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Hmm....is a hot directory possible?
"Wiggins d'Anconia" <[EMAIL PROTECTED]> writes: > FAM provides something similar and I believe uses hooks in the linux > kernel on that system and other means on other *nixes. There even > exists a Perl module to hook into it (SGI::FAM) good luck getting it > to work though, I have had some problems in the recent past as have > other posters. Following up on this a little, and in case the OP is still looking for something. I found this application on FAM sites fileschanged http://fileschanged.sourceforge.net/ A command line tool that uses FAM. $ fileschanged -r -f /home/myhomedir ...will monitor all files in your home directory, and all directories and subdirectories in it. For example if you were to receive mail, while you were monitoring your home directory you might see something like: -- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Hmm....is a hot directory possible?
david <[EMAIL PROTECTED]> writes: >> This is not what the OP asked. But I wondered if one can determine >> if a file has been writen to or changed inside a directory by looking >> at a stat on the directory. > > if i am not dreaming, OP asks for whether there is new files adding to the Maybe why I said `This is not what the OP asked...' .. hehe. Thanks for the other tips about the cache stuff and loop speed. -- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Hmm....is a hot directory possible?
Harry Putnam wrote: > david <[EMAIL PROTECTED]> writes: > >> >> you could take a look at the stat function provided by Perl to see if the >> directory's last modified time or inode change time changed: > > This is not what the OP asked. But I wondered if one can determine > if a file has been writen to or changed inside a directory by looking > at a stat on the directory. if i am not dreaming, OP asks for whether there is new files adding to the directory, not that if a file change inside a directory. yes, both mtime and inctime reflect that and stat report that as well if new files are added or delete. no, stat won't report that if a file only changes. that has very little to do with the parent directory. > > File changes don't seem to be reflected in mtime, unless a new file > is added or one taken away. Ditto for atime and ctime. correct. OP doesn't care about file change (again, if i am not dreaming). only additon and deletion from the directory. > > So is stat not able to determine if a file has been written to by > looking at the parent dir? not sure what you mean but: #-- #-- stat report mtime and inctime change in /tmp for the first touch but not #-- the second because it only touches it. #-- rm -f /tmp/file touch /tmp/file touch /tmp/file the various posters' comment on hooking is your best bet if your OS supports it. otherwise a combination of cache/compare/mtime is needed because: touch /tmp/file rm -f /tmp/file touch /tmp/file rm -f /tmp/file fakes the cache/compare method by not seeing the file as they come-and-go. the tricky part is you have to do the checking really fast to make the algr. reliable. otherwise, the race condition thingy is going to get you. david -- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Hmm....is a hot directory possible?
Gary Stainburn wrote: On Thursday 26 Jun 2003 10:38 pm, Jenda Krynicky wrote: From: Chris Zimmerman <[EMAIL PROTECTED]> Is there some way that I can write a bit of code that will watch a directory and as soon as a file is written to that directory, something is run against that file? What would be the best way to turn this into a daemon? Depends on the OS. Under windows it'd be Win32::ChangeNotify and Win32::Daemon(::Simple)? I know a while back (12 months?) a talk was given at our local LUG - http://www.wylug.org.uk - about hooks being placed in the Linux kernel so that user mode processed could register an interest in a file or directory and would receive a SIG if any change occured. FAM provides something similar and I believe uses hooks in the linux kernel on that system and other means on other *nixes. There even exists a Perl module to hook into it (SGI::FAM) good luck getting it to work though, I have had some problems in the recent past as have other posters. If someone is looking for a non-blocking way to detect new files or changes to files, etc. and doesn't mind a somewhat steep learning curve I would suggest POE, and possibly its pre-built PoCo::DirWatch component. The OP should see my posts here for somewhat on the subject: http://nntp.x.perl.org/group/perl.beginners/44639 http://nntp.x.perl.org/group/perl.beginners/44673 I would also suggest using MD5 or SHA1 hashing or something similar to detect true changes to a file, mtime can be faked, and checking a directory mtime on an interval may not lead to any reliable way to determine if the contents of said directory has changed. Combining the hash caching manner suggested previously with using a checksum hash of the files in the directory should give better results in detecting individual file changes. http://danconia.org The idea was so that progs like 'top' could become much more efficient by sleeping instead of polling, but I don't know how far along this is, or whether anyone's written a perl module for it yet. Gary Jenda = [EMAIL PROTECTED] === http://Jenda.Krynicky.cz = When it comes to wine, women and song, wizards are allowed to get drunk and croon as much as they like. -- Terry Pratchett in Sourcery -- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Hmm....is a hot directory possible?
On Thursday 26 Jun 2003 10:38 pm, Jenda Krynicky wrote: > From: Chris Zimmerman <[EMAIL PROTECTED]> > > > Is there some way that I can write a bit of code that will watch a > > directory and as soon as a file is written to that directory, > > something is run against that file? What would be the best way to > > turn this into a daemon? > > Depends on the OS. > Under windows it'd be Win32::ChangeNotify and > Win32::Daemon(::Simple)? I know a while back (12 months?) a talk was given at our local LUG - http://www.wylug.org.uk - about hooks being placed in the Linux kernel so that user mode processed could register an interest in a file or directory and would receive a SIG if any change occured. The idea was so that progs like 'top' could become much more efficient by sleeping instead of polling, but I don't know how far along this is, or whether anyone's written a perl module for it yet. Gary > > Jenda > = [EMAIL PROTECTED] === http://Jenda.Krynicky.cz = > When it comes to wine, women and song, wizards are allowed > to get drunk and croon as much as they like. > -- Terry Pratchett in Sourcery -- Gary Stainburn This email does not contain private or confidential material as it may be snooped on by interested government parties for unknown and undisclosed purposes - Regulation of Investigatory Powers Act, 2000 -- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Hmm....is a hot directory possible?
Sorry, Harry. I emailed to soon. I read the email as I commented to much, but really didn't comment at all. Which is not good! Best Regards, Jaimee -Original Message- From: news [mailto:[EMAIL PROTECTED] Behalf Of Harry Putnam Sent: Thursday, June 26, 2003 9:44 PM To: [EMAIL PROTECTED] Subject: Re: Hmm....is a hot directory possible? "jandrspencer" <[EMAIL PROTECTED]> writes: > Please don't ever tell someone how to code their source. Jeez! Is right. I don't see any smileys here so I guess you were offended. Even though it was clearly said in complete jest. Not sure how to respond. I guess its enough to say no ill intent was involved. -- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Hmm....is a hot directory possible?
"jandrspencer" <[EMAIL PROTECTED]> writes: > Please don't ever tell someone how to code their source. Jeez! Is right. I don't see any smileys here so I guess you were offended. Even though it was clearly said in complete jest. Not sure how to respond. I guess its enough to say no ill intent was involved. -- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Hmm....is a hot directory possible?
Sorry, Chris. Harry, is right I should have explained better with my comments. Regards, Jaimee -- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Hmm....is a hot directory possible?
Harry, Please don't ever tell someone how to code their source. Jeez! Is right. -Original Message- From: Harry Putnam [mailto:[EMAIL PROTECTED] Sent: Thursday, June 26, 2003 6:19 PM To: [EMAIL PROTECTED] Subject: Re: Hmm....is a hot directory possible? <[EMAIL PROTECTED]> writes: > I keep forgetting to post the hold group. Hopes this helps. > > #!/usr/bin/perl -w [...] Jeez.. I wish you wouldn't over comment like that. Makes it too easy to figure out what is going on : ) -- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Hmm....is a hot directory possible?
david <[EMAIL PROTECTED]> writes: > > you could take a look at the stat function provided by Perl to see if the > directory's last modified time or inode change time changed: This is not what the OP asked. But I wondered if one can determine if a file has been writen to or changed inside a directory by looking at a stat on the directory. File changes don't seem to be reflected in mtime, unless a new file is added or one taken away. Ditto for atime and ctime. So is stat not able to determine if a file has been written to by looking at the parent dir? -- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Hmm....is a hot directory possible?
<[EMAIL PROTECTED]> writes: > I keep forgetting to post the hold group. Hopes this helps. > > #!/usr/bin/perl -w [...] Jeez.. I wish you wouldn't over comment like that. Makes it too easy to figure out what is going on : ) -- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Hmm....is a hot directory possible?
From: Chris Zimmerman <[EMAIL PROTECTED]> > Is there some way that I can write a bit of code that will watch a > directory and as soon as a file is written to that directory, > something is run against that file? What would be the best way to > turn this into a daemon? Depends on the OS. Under windows it'd be Win32::ChangeNotify and Win32::Daemon(::Simple)? Jenda = [EMAIL PROTECTED] === http://Jenda.Krynicky.cz = When it comes to wine, women and song, wizards are allowed to get drunk and croon as much as they like. -- Terry Pratchett in Sourcery -- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Re: Hmm....is a hot directory possible?
Hello,
I should add that in the below script MAKEPORT is another script that I have
performing once there is a change in the directory.
>
> From: <[EMAIL PROTECTED]>
> Date: 2003/06/26 Thu PM 05:20:51 EDT
> To: [EMAIL PROTECTED], [EMAIL PROTECTED]
> Subject: Re: Re: Hmm....is a hot directory possible?
>
> I keep forgetting to post the hold group. Hopes this helps.
>
> #!/usr/bin/perl -w
>
> use strict;
> use POSIX qw(setsid);
>
> # set costants
> my$MAKEPORT="/home/jspencer/bin/make-port";
>
>
> # daemonize the program
> &daemonize;
>
> while(1) {
> # set costants
> [EMAIL PROTECTED]("/home/jspencer/acucorp/std/std-unix-misc.tar.gz") ;
> my$FIRSTTIME = "$FILETIME[9]";
> my$SECTIME=scalar time;
>
> sleep(2);
>
> if (($FIRSTTIME + 2) > $SECTIME) {
> system($MAKEPORT);
> }}
>
> sub daemonize {
>
> my $outlog = '/home/jspencer/bin/daemons/logs/hotfolder_out.log';
> my $errorlog = '/home/jspencer/bin/daemons/logs/hotfolder_error.log';
>
>
> chdir '/' or die "Can't chdir to /: $!";
> umask 0;
> open STDIN, '/dev/null' or die "Can't read /dev/null: $!";
> open STDOUT, ">$outlog" or die "Can't write to /dev/null: $!";
> open STDERR, ">$errorlog" or die "Can't write to /dev/null: $!";
> defined(my $pid = fork) or die "Can't fork: $!";
> exit if $pid;
> setsidor die "Can't start a new session: $!";
> }
> >
> > From: david <[EMAIL PROTECTED]>
> > Date: 2003/06/26 Thu PM 04:16:49 EDT
> > To: [EMAIL PROTECTED]
> > Subject: Re: Hmmis a hot directory possible?
> >
> > Chris Zimmerman wrote:
> >
> > > -BEGIN PGP SIGNED MESSAGE-
> > > Hash: SHA1
> > >
> > > Is there some way that I can write a bit of code that will watch a
> > > directory and as soon as a file is written to that directory, something is
> > > run against
> > > that file? What would be the best way to turn this into a daemon?
> > >
> >
> > you could take a look at the stat function provided by Perl to see if the
> > directory's last modified time or inode change time changed:
> >
> > #!/usr/bin/perl -w
> > use strict;
> >
> > my($pmm,$pic);
> >
> > while(1){
> >
> > my($mm,$ic) = (stat('/tmp'))[9,10];
> >
> > if($pmm and $pic and $pmm != $mm || $pic != $ic){
> > print "some change to /tmp\n";
> > }else{
> > print ".\n";
> > }
> >
> > $pmm=$mm;
> > $pic=$ic;
> >
> > sleep(7);
> > }
> >
> > __END__
> >
> > 1. this is not a daemon.
> > 2. this only reports there are some changes (it could be adding a file or
> > deleting a file,etc) in /tmp but you don't know what really happened there.
> > 3. this only reports changes to /tmp not knowing any change below the /tmp
> > level. for example:
> >
> > #--
> > #-- the scrpipt report the following 3 changes to /tmp
> > #--
> > mkdir /tmp/another
> > touch /tmp/hi
> > rm -f /tmp/hi
> >
> > #--
> > #-- but doesn't know the following 3 changes
> > #--
> > mkdir /tmp/another/yet
> > touch /tmp/another/yet/file
> > rm -fr /tmp/another/yet
> >
> > because there is really no change to /tmp, only its child directory.
> >
> > solution to #2 and #3 can do done with a different approch. something like
> > the following might work:
> >
> > 1. recursively cache (in a hash) all sub directories and files under /tmp
> > during start up of your daemon.
> > 2. once a while, do the same resursive scan for the /tmp directory and
> > compare the directory content with the hash you cached a while ago.
> > 3. if there is any differences, you know something has changed and because
> > you have 2 hashs, you can easily find out what really happened. for
> > example, if the first hash has an entry where the second hash doesn't, you
> > know something has been deleted from the directory. Or if the second hash
> > has something that's missing from the first hash, you know there are new
> > files.
> > 4. update the cache to be the most recent scan. repeat step #2.
> >
> > File::Find module can help you do the recursive scan portion fairly easi
Re: Re: Hmm....is a hot directory possible?
I keep forgetting to post the hold group. Hopes this helps.
#!/usr/bin/perl -w
use strict;
use POSIX qw(setsid);
# set costants
my$MAKEPORT="/home/jspencer/bin/make-port";
# daemonize the program
&daemonize;
while(1) {
# set costants
[EMAIL PROTECTED]("/home/jspencer/acucorp/std/std-unix-misc.tar.gz") ;
my$FIRSTTIME = "$FILETIME[9]";
my$SECTIME=scalar time;
sleep(2);
if (($FIRSTTIME + 2) > $SECTIME) {
system($MAKEPORT);
}}
sub daemonize {
my $outlog = '/home/jspencer/bin/daemons/logs/hotfolder_out.log';
my $errorlog = '/home/jspencer/bin/daemons/logs/hotfolder_error.log';
chdir '/' or die "Can't chdir to /: $!";
umask 0;
open STDIN, '/dev/null' or die "Can't read /dev/null: $!";
open STDOUT, ">$outlog" or die "Can't write to /dev/null: $!";
open STDERR, ">$errorlog" or die "Can't write to /dev/null: $!";
defined(my $pid = fork) or die "Can't fork: $!";
exit if $pid;
setsid or die "Can't start a new session: $!";
}
>
> From: david <[EMAIL PROTECTED]>
> Date: 2003/06/26 Thu PM 04:16:49 EDT
> To: [EMAIL PROTECTED]
> Subject: Re: Hmmis a hot directory possible?
>
> Chris Zimmerman wrote:
>
> > -BEGIN PGP SIGNED MESSAGE-
> > Hash: SHA1
> >
> > Is there some way that I can write a bit of code that will watch a
> > directory and as soon as a file is written to that directory, something is
> > run against
> > that file? What would be the best way to turn this into a daemon?
> >
>
> you could take a look at the stat function provided by Perl to see if the
> directory's last modified time or inode change time changed:
>
> #!/usr/bin/perl -w
> use strict;
>
> my($pmm,$pic);
>
> while(1){
>
> my($mm,$ic) = (stat('/tmp'))[9,10];
>
> if($pmm and $pic and $pmm != $mm || $pic != $ic){
> print "some change to /tmp\n";
> }else{
> print ".\n";
> }
>
> $pmm=$mm;
> $pic=$ic;
>
> sleep(7);
> }
>
> __END__
>
> 1. this is not a daemon.
> 2. this only reports there are some changes (it could be adding a file or
> deleting a file,etc) in /tmp but you don't know what really happened there.
> 3. this only reports changes to /tmp not knowing any change below the /tmp
> level. for example:
>
> #--
> #-- the scrpipt report the following 3 changes to /tmp
> #--
> mkdir /tmp/another
> touch /tmp/hi
> rm -f /tmp/hi
>
> #--
> #-- but doesn't know the following 3 changes
> #--
> mkdir /tmp/another/yet
> touch /tmp/another/yet/file
> rm -fr /tmp/another/yet
>
> because there is really no change to /tmp, only its child directory.
>
> solution to #2 and #3 can do done with a different approch. something like
> the following might work:
>
> 1. recursively cache (in a hash) all sub directories and files under /tmp
> during start up of your daemon.
> 2. once a while, do the same resursive scan for the /tmp directory and
> compare the directory content with the hash you cached a while ago.
> 3. if there is any differences, you know something has changed and because
> you have 2 hashs, you can easily find out what really happened. for
> example, if the first hash has an entry where the second hash doesn't, you
> know something has been deleted from the directory. Or if the second hash
> has something that's missing from the first hash, you know there are new
> files.
> 4. update the cache to be the most recent scan. repeat step #2.
>
> File::Find module can help you do the recursive scan portion fairly easily.
> you can take the same approach but instead of caching the directory
> contents, you can cache each sub directory's last modified time instead.
> this will reduce the size of your hash a bit. either way, even this
> approach has many drawback:
>
> 1. if you target directory is huge, the scaning part will take a long time
> which brings us to the 2 drawback.
> 2. race condition. it's totally possible for a file to appear and disappear
> during your directory scan especially if the scan takes a long time or the
> directory is "busy" (means there are tons of activity in the directory so
> files appear and disappear really fast). your scan will miss those.
>
> you might need to apply some kind of locking to the directory during the
> scan. finally, take a look in CPAN to see if something comes up. on top of
> my head, i don't remember any modules that does what you want. good luck.
>
> david
>
> --
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>
>
--
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: Hmm....is a hot directory possible?
Chris Zimmerman wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> Is there some way that I can write a bit of code that will watch a
> directory and as soon as a file is written to that directory, something is
> run against
> that file? What would be the best way to turn this into a daemon?
>
you could take a look at the stat function provided by Perl to see if the
directory's last modified time or inode change time changed:
#!/usr/bin/perl -w
use strict;
my($pmm,$pic);
while(1){
my($mm,$ic) = (stat('/tmp'))[9,10];
if($pmm and $pic and $pmm != $mm || $pic != $ic){
print "some change to /tmp\n";
}else{
print ".\n";
}
$pmm=$mm;
$pic=$ic;
sleep(7);
}
__END__
1. this is not a daemon.
2. this only reports there are some changes (it could be adding a file or
deleting a file,etc) in /tmp but you don't know what really happened there.
3. this only reports changes to /tmp not knowing any change below the /tmp
level. for example:
#--
#-- the scrpipt report the following 3 changes to /tmp
#--
mkdir /tmp/another
touch /tmp/hi
rm -f /tmp/hi
#--
#-- but doesn't know the following 3 changes
#--
mkdir /tmp/another/yet
touch /tmp/another/yet/file
rm -fr /tmp/another/yet
because there is really no change to /tmp, only its child directory.
solution to #2 and #3 can do done with a different approch. something like
the following might work:
1. recursively cache (in a hash) all sub directories and files under /tmp
during start up of your daemon.
2. once a while, do the same resursive scan for the /tmp directory and
compare the directory content with the hash you cached a while ago.
3. if there is any differences, you know something has changed and because
you have 2 hashs, you can easily find out what really happened. for
example, if the first hash has an entry where the second hash doesn't, you
know something has been deleted from the directory. Or if the second hash
has something that's missing from the first hash, you know there are new
files.
4. update the cache to be the most recent scan. repeat step #2.
File::Find module can help you do the recursive scan portion fairly easily.
you can take the same approach but instead of caching the directory
contents, you can cache each sub directory's last modified time instead.
this will reduce the size of your hash a bit. either way, even this
approach has many drawback:
1. if you target directory is huge, the scaning part will take a long time
which brings us to the 2 drawback.
2. race condition. it's totally possible for a file to appear and disappear
during your directory scan especially if the scan takes a long time or the
directory is "busy" (means there are tons of activity in the directory so
files appear and disappear really fast). your scan will miss those.
you might need to apply some kind of locking to the directory during the
scan. finally, take a look in CPAN to see if something comes up. on top of
my head, i don't remember any modules that does what you want. good luck.
david
--
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
