Re: [PHP] How to make sure that the target file to read is not under writing by others?

2009-08-20 Thread Tom Worster
On 8/19/09 9:56 PM, Clancy clanc...@cybec.com.au wrote:

 I gather from this discussion that PHP allows two users to open a file for
 R/W? I had assumed it wouldn't.

i think php does allow this. but i'm not sure all file systems do.



-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php



RE: [PHP] How to make sure that the target file to read is not under writing by others?

2009-08-20 Thread Arno Kuhl
-Original Message-
From: Tom Worster [mailto:f...@thefsb.org] 
Sent: 20 August 2009 01:28 PM
To: Clancy; php-general@lists.php.net
Subject: Re: [PHP] How to make sure that the target file to read is not
under writing by others?

On 8/19/09 9:56 PM, Clancy clanc...@cybec.com.au wrote:

 I gather from this discussion that PHP allows two users to open a file 
 for R/W? I had assumed it wouldn't.

i think php does allow this. but i'm not sure all file systems do.

--

PHP does allow more than one process to open a file for read-write, as does
every other filesystem I know of (PHP doesn't have a filesystem, it
implements a subset of file commands common to all OS's [some differences
between *nix and Windows] that it uses to communicate with the underlying OS
filesystem). If there's a need to ensure that multiple processes don't step
on each others toes the file can be locked. The two most common file
accesses are read and append, where a lock is mostly not required. It's only
in special cases where you want to write or update something and want to be
sure that another process isn't trying to update the same data or trying to
read data that may no longer be consistent (until you finish your update)
that you'd want to lock it.

Cheers
Arno


-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php



[PHP] How to make sure that the target file to read is not under writing by others?

2009-08-19 Thread Dengxule
Hi everyone:

I have a crontab command to execuate my php-script every half an hour.

The mission of the php-script is to open a file(log file), examine it.

The target file(log file) is transported to local every half an hour.

I've no idea how much time it will costs and on the other hand, i want to
make sure the file i'm openning is Completely Written.

Any instruction will be grateful.


PS: I've made a test. My php-script just fopen a local file being
transported in, and the fopen returns no FALSE value but a resource.


Dengxule
2009/08/19


Re: [PHP] How to make sure that the target file to read is not under writing by others?

2009-08-19 Thread Ashley Sheridan
On Wed, 2009-08-19 at 15:55 +0800, Dengxule wrote:
 Hi everyone:
 
 I have a crontab command to execuate my php-script every half an hour.
 
 The mission of the php-script is to open a file(log file), examine it.
 
 The target file(log file) is transported to local every half an hour.
 
 I've no idea how much time it will costs and on the other hand, i want to
 make sure the file i'm openning is Completely Written.
 
 Any instruction will be grateful.
 
 
 PS: I've made a test. My php-script just fopen a local file being
 transported in, and the fopen returns no FALSE value but a resource.
 
 
 Dengxule
 2009/08/19

I had to do something similar in my last job. First thing to do is
figure out how large the log you expect to parse is going to be, and
from there, have a guess at how long it will take to transfer fully from
elsewhere.

I'd make the cron script a bit more frequent, but only ask it to look
for files that have changed between x and y minutes ago, where y is a
minute or two over the transfer duration. Then you can either mark the
file in some way, or move it to another location to indicate it has been
processed.

Thanks,
Ash
http://www.ashleysheridan.co.uk




-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php



RE: [PHP] How to make sure that the target file to read is not under writing by others?

2009-08-19 Thread Arno Kuhl
-Original Message-
From: Dengxule [mailto:dengx...@gmail.com] 
Sent: 19 August 2009 09:56 AM
To: Php Maillist
Subject: [PHP] How to make sure that the target file to read is not under
writing by others?

Hi everyone:

I have a crontab command to execuate my php-script every half an hour.

The mission of the php-script is to open a file(log file), examine it.

The target file(log file) is transported to local every half an hour.

I've no idea how much time it will costs and on the other hand, i want to
make sure the file i'm openning is Completely Written.

Any instruction will be grateful.


PS: I've made a test. My php-script just fopen a local file being
transported in, and the fopen returns no FALSE value but a resource.


Dengxule
2009/08/19


Check the timestamp of the log file. Alternatively you can have a log
control file that is updated by the program transferring the log file only
when the transfer is completed, and work on that.

Cheers
Arno


-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php



RE: [PHP] How to make sure that the target file to read is not under writing by others?

2009-08-19 Thread Ashley Sheridan
On Wed, 2009-08-19 at 11:55 +0200, Arno Kuhl wrote:
 -Original Message-
 From: Dengxule [mailto:dengx...@gmail.com] 
 Sent: 19 August 2009 09:56 AM
 To: Php Maillist
 Subject: [PHP] How to make sure that the target file to read is not under
 writing by others?
 
 Hi everyone:
 
 I have a crontab command to execuate my php-script every half an hour.
 
 The mission of the php-script is to open a file(log file), examine it.
 
 The target file(log file) is transported to local every half an hour.
 
 I've no idea how much time it will costs and on the other hand, i want to
 make sure the file i'm openning is Completely Written.
 
 Any instruction will be grateful.
 
 
 PS: I've made a test. My php-script just fopen a local file being
 transported in, and the fopen returns no FALSE value but a resource.
 
 
 Dengxule
 2009/08/19
 
 
 Check the timestamp of the log file. Alternatively you can have a log
 control file that is updated by the program transferring the log file only
 when the transfer is completed, and work on that.
 
 Cheers
 Arno
 
 
The log control file wouldn't work, as how would you know when to open
the control file?!

Thanks,
Ash
http://www.ashleysheridan.co.uk




-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php



RE: [PHP] How to make sure that the target file to read is not under writing by others?

2009-08-19 Thread Arno Kuhl
-Original Message-
From: Ashley Sheridan [mailto:a...@ashleysheridan.co.uk] 
Sent: 19 August 2009 11:57 AM
To: a...@dotcontent.net
Cc: 'Dengxule'; 'Php Maillist'
Subject: RE: [PHP] How to make sure that the target file to read is not
under writing by others?

On Wed, 2009-08-19 at 11:55 +0200, Arno Kuhl wrote:
 -Original Message-
 From: Dengxule [mailto:dengx...@gmail.com]
 Sent: 19 August 2009 09:56 AM
 To: Php Maillist
 Subject: [PHP] How to make sure that the target file to read is not 
 under writing by others?
 
 Hi everyone:
 
 I have a crontab command to execuate my php-script every half an hour.
 
 The mission of the php-script is to open a file(log file), examine it.
 
 The target file(log file) is transported to local every half an hour.
 
 I've no idea how much time it will costs and on the other hand, i want 
 to make sure the file i'm openning is Completely Written.
 
 Any instruction will be grateful.
 
 
 PS: I've made a test. My php-script just fopen a local file being 
 transported in, and the fopen returns no FALSE value but a resource.
 
 
 Dengxule
 2009/08/19
 
 
 Check the timestamp of the log file. Alternatively you can have a log 
 control file that is updated by the program transferring the log file 
 only when the transfer is completed, and work on that.
 
 Cheers
 Arno
 
 
The log control file wouldn't work, as how would you know when to open the
control file?!

Thanks,
Ash
http://www.ashleysheridan.co.uk

--

You can open the control file any time you want. The info in the control
file tells you whether to proceed. The transfer program adds the entry when
the log has been transferred, the processing script updates the entry to
indicate it's been processed. You can either add a new entry each time
(useful for monitoring to fine-tune the timing, or for keeping a history) or
work on a single entry that's overwritten each time the log is transferred.

Cheers
Arno


-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php



RE: [PHP] How to make sure that the target file to read is not under writing by others?

2009-08-19 Thread Ashley Sheridan
On Wed, 2009-08-19 at 12:56 +0200, Arno Kuhl wrote:
 -Original Message-
 From: Ashley Sheridan [mailto:a...@ashleysheridan.co.uk] 
 Sent: 19 August 2009 11:57 AM
 To: a...@dotcontent.net
 Cc: 'Dengxule'; 'Php Maillist'
 Subject: RE: [PHP] How to make sure that the target file to read is not
 under writing by others?
 
 On Wed, 2009-08-19 at 11:55 +0200, Arno Kuhl wrote:
  -Original Message-
  From: Dengxule [mailto:dengx...@gmail.com]
  Sent: 19 August 2009 09:56 AM
  To: Php Maillist
  Subject: [PHP] How to make sure that the target file to read is not 
  under writing by others?
  
  Hi everyone:
  
  I have a crontab command to execuate my php-script every half an hour.
  
  The mission of the php-script is to open a file(log file), examine it.
  
  The target file(log file) is transported to local every half an hour.
  
  I've no idea how much time it will costs and on the other hand, i want 
  to make sure the file i'm openning is Completely Written.
  
  Any instruction will be grateful.
  
  
  PS: I've made a test. My php-script just fopen a local file being 
  transported in, and the fopen returns no FALSE value but a resource.
  
  
  Dengxule
  2009/08/19
  
  
  Check the timestamp of the log file. Alternatively you can have a log 
  control file that is updated by the program transferring the log file 
  only when the transfer is completed, and work on that.
  
  Cheers
  Arno
  
  
 The log control file wouldn't work, as how would you know when to open the
 control file?!
 
 Thanks,
 Ash
 http://www.ashleysheridan.co.uk
 
 --
 
 You can open the control file any time you want. The info in the control
 file tells you whether to proceed. The transfer program adds the entry when
 the log has been transferred, the processing script updates the entry to
 indicate it's been processed. You can either add a new entry each time
 (useful for monitoring to fine-tune the timing, or for keeping a history) or
 work on a single entry that's overwritten each time the log is transferred.
 
 Cheers
 Arno
 
 
No, what you're saying is 'use a log file in order to know when to look
at another log file'. What would happen if you tried to access the
control log file whilst it was in the process of being written to?
Admittedly, you reduce your chances of failure because there is less
data being written to the control log than the actual log, but the
chance of reading incomplete data is still there!

Thanks,
Ash
http://www.ashleysheridan.co.uk




-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php



Re: [PHP] How to make sure that the target file to read is not under writing by others?

2009-08-19 Thread Nitebirdz
On Wed, Aug 19, 2009 at 11:59:39AM +0100, Ashley Sheridan wrote:
  
 No, what you're saying is 'use a log file in order to know when to look
 at another log file'. What would happen if you tried to access the
 control log file whilst it was in the process of being written to?
 Admittedly, you reduce your chances of failure because there is less
 data being written to the control log than the actual log, but the
 chance of reading incomplete data is still there!
 

WARNING: total newbie here.

If I understood Arno correctly, he was recommending to implement
something like the old /var/run/*pid files in UNIX.  That way, you can
control whether or not the previous run is already done with the file
before you move on.  


-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php



Re: [PHP] How to make sure that the target file to read is not under writing by others?

2009-08-19 Thread Kyle Smith

Nitebirdz wrote:

On Wed, Aug 19, 2009 at 11:59:39AM +0100, Ashley Sheridan wrote:
  
 
No, what you're saying is 'use a log file in order to know when to look

at another log file'. What would happen if you tried to access the
control log file whilst it was in the process of being written to?
Admittedly, you reduce your chances of failure because there is less
data being written to the control log than the actual log, but the
chance of reading incomplete data is still there!




WARNING: total newbie here.

If I understood Arno correctly, he was recommending to implement
something like the old /var/run/*pid files in UNIX.  That way, you can
control whether or not the previous run is already done with the file
before you move on. 
That is definitely the correct approach.  Have the script which copies 
the log file touch a file called 'log_file.write' or some such.  When 
it's done, remove the file.  Your PHP script should exit if that file 
exists.  Of course, given the 30 minute cron cycle, it would then have 
to wait until the next cycle.  Maybe run it more often.


Re: [PHP] How to make sure that the target file to read is not under writing by others?

2009-08-19 Thread Tom Worster
On 8/19/09 3:55 AM, Dengxule dengx...@gmail.com wrote:

 I have a crontab command to execuate my php-script every half an hour.
 
 The mission of the php-script is to open a file(log file), examine it.
 
 The target file(log file) is transported to local every half an hour.
 
 I've no idea how much time it will costs and on the other hand, i want to
 make sure the file i'm openning is Completely Written.
 
 Any instruction will be grateful.

perhaps you could use cronolog (http://cronolog.org/) to rotate the log
files every half hour? that way you could be sure that the file you open to
process/copy is one that cronolog has finished writing to.



-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php



RE: [PHP] How to make sure that the target file to read is not under writing by others?

2009-08-19 Thread Arno Kuhl
-Original Message-
From: Ashley Sheridan [mailto:a...@ashleysheridan.co.uk] 
Sent: 19 August 2009 01:00 PM
To: a...@dotcontent.net
Cc: 'Dengxule'; 'Php Maillist'
Subject: RE: [PHP] How to make sure that the target file to read is not
under writing by others?

On Wed, 2009-08-19 at 12:56 +0200, Arno Kuhl wrote:
 -Original Message-
 From: Ashley Sheridan [mailto:a...@ashleysheridan.co.uk]
 Sent: 19 August 2009 11:57 AM
 To: a...@dotcontent.net
 Cc: 'Dengxule'; 'Php Maillist'
 Subject: RE: [PHP] How to make sure that the target file to read is 
 not under writing by others?
 
 On Wed, 2009-08-19 at 11:55 +0200, Arno Kuhl wrote:
  -Original Message-
  From: Dengxule [mailto:dengx...@gmail.com]
  Sent: 19 August 2009 09:56 AM
  To: Php Maillist
  Subject: [PHP] How to make sure that the target file to read is not 
  under writing by others?
  
  Hi everyone:
  
  I have a crontab command to execuate my php-script every half an hour.
  
  The mission of the php-script is to open a file(log file), examine it.
  
  The target file(log file) is transported to local every half an hour.
  
  I've no idea how much time it will costs and on the other hand, i 
  want to make sure the file i'm openning is Completely Written.
  
  Any instruction will be grateful.
  
  
  PS: I've made a test. My php-script just fopen a local file being 
  transported in, and the fopen returns no FALSE value but a resource.
  
  
  Dengxule
  2009/08/19
  
  
  Check the timestamp of the log file. Alternatively you can have a 
  log control file that is updated by the program transferring the log 
  file only when the transfer is completed, and work on that.
  
  Cheers
  Arno
  
  
 The log control file wouldn't work, as how would you know when to open 
 the control file?!
 
 Thanks,
 Ash
 http://www.ashleysheridan.co.uk
 
 --
 
 You can open the control file any time you want. The info in the 
 control file tells you whether to proceed. The transfer program adds 
 the entry when the log has been transferred, the processing script 
 updates the entry to indicate it's been processed. You can either add 
 a new entry each time (useful for monitoring to fine-tune the timing, 
 or for keeping a history) or work on a single entry that's overwritten
each time the log is transferred.
 
 Cheers
 Arno
 
 
No, what you're saying is 'use a log file in order to know when to look at
another log file'. What would happen if you tried to access the control log
file whilst it was in the process of being written to?
Admittedly, you reduce your chances of failure because there is less data
being written to the control log than the actual log, but the chance of
reading incomplete data is still there!

Thanks,
Ash
http://www.ashleysheridan.co.uk



A control file is used to control processing, a log file is used to log
events. I'm definitely talking about a control file. If the control file
happens to keep a log then it's a bonus, but not its primary purpose.

Access the control file while being written to? I'm talking about a short
single-line entry, and the file is only written to when the write command is
processed. Copying a file is very different to issuing a single write
command. Don't forget that PHP and webservers run on top of operating
systems, and the OS ultimately handles all file activity. The OS ensures
that another process can't read a file that is halfway through a write
command (any computer would instantly crash if that sort of basic
housekeeping wasn't done). 

If you don't trust the OS then file locking can be used (though pointless in
this case). For the ultra-paranoids who don't trust OS's or locks you can
either (a) choose a different career, or (b) add a time-stamp to the
beginning of the entry and check that the entry wasn't written in the last
few nanoseconds (if you get this far I'd recommend option a). You can even
go so far as to check the length of the timestamp string to be sure that you
got all of it, and if it's too short you know you're busy reading a file
that's halfway through a write command, and sleep for a millisecond and then
try again ;)

Cheers
Arno


-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php



RE: [PHP] How to make sure that the target file to read is not under writing by others?

2009-08-19 Thread Ashley Sheridan
On Wed, 2009-08-19 at 15:31 +0200, Arno Kuhl wrote:
 (any computer would instantly crash if that sort of basic
 housekeeping wasn't done)

No, it really wouldn't! If it did, then you'd never have video playing
software out there that supported broken downloads, no preview software
for semi-downloaded items. Have you ever been sent a large jpeg over a
slow connection on a messenger program? You can open the file way before
it's finished downloading and actually see it progressively load in.

Thanks,
Ash
http://www.ashleysheridan.co.uk




-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php



RE: [PHP] How to make sure that the target file to read is not under writing by others?

2009-08-19 Thread Arno Kuhl
-Original Message-
From: Ashley Sheridan [mailto:a...@ashleysheridan.co.uk] 
Sent: 19 August 2009 03:34 PM
To: a...@dotcontent.net
Cc: 'Dengxule'; 'Php Maillist'
Subject: RE: [PHP] How to make sure that the target file to read is not
under writing by others?

On Wed, 2009-08-19 at 15:31 +0200, Arno Kuhl wrote:
 (any computer would instantly crash if that sort of basic housekeeping 
 wasn't done)

No, it really wouldn't! If it did, then you'd never have video playing
software out there that supported broken downloads, no preview software for
semi-downloaded items. Have you ever been sent a large jpeg over a slow
connection on a messenger program? You can open the file way before it's
finished downloading and actually see it progressively load in.

Thanks,
Ash
http://www.ashleysheridan.co.uk

---

If anyone was left a bit unsure about whether or not a write command in PHP
could result in a file on a local filesystem being left in a semi-written
state (i.e. some other script started reading the file while your script had
only written part of its output string) rest assured that fwrite() always
writes *all* the contents to the file without the need of flock(), by the
time any other process (whether PHP or not) reads that file (assuming you're
not writing megabytes of data with a single fwrite()). To quote the PHP
manual:

If handle was fopen()ed in append mode, fwrite()s are atomic (unless the
size of string exceeds the filesystem's block size, on some platforms, and
as long as the file is on a local filesystem). That is, there is no need to
flock() a resource before calling fwrite(); all of the data will be written
without interruption.

Cheers
Arno


-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php



Re: [PHP] How to make sure that the target file to read is not under writing by others?

2009-08-19 Thread Clancy
On Wed, 19 Aug 2009 08:13:56 -0400, kyle.sm...@inforonics.com (Kyle Smith) 
wrote:

Nitebirdz wrote:
 On Wed, Aug 19, 2009 at 11:59:39AM +0100, Ashley Sheridan wrote:
   
  
 No, what you're saying is 'use a log file in order to know when to look
 at another log file'. What would happen if you tried to access the
 control log file whilst it was in the process of being written to?
 Admittedly, you reduce your chances of failure because there is less
 data being written to the control log than the actual log, but the
 chance of reading incomplete data is still there!

 

 WARNING: total newbie here.

 If I understood Arno correctly, he was recommending to implement
 something like the old /var/run/*pid files in UNIX.  That way, you can
 control whether or not the previous run is already done with the file
 before you move on. 
That is definitely the correct approach.  Have the script which copies 
the log file touch a file called 'log_file.write' or some such.  When 
it's done, remove the file.  Your PHP script should exit if that file 
exists.  Of course, given the 30 minute cron cycle, it would then have 
to wait until the next cycle.  Maybe run it more often.

I gather from this discussion that PHP allows two users to open a file for R/W? 
I had
assumed it wouldn't.

Anyway, how about:

$user = 'Fred'; $try_again = true;
If (!file_exists('Busy.txt'))
{
File_put_contents ('Busy.txt', $user);
If ($user == File_get_contents ('Busy.txt')
{
// Do what you have to
Unlink ('Busy.txt');
$try_again = false;
}
}
if ($try_again)
{
// Come back in 5 minutes
}

This has a theoretical weakness, in that Joe could check for Busy.txt in the 
interval
between your checking and writing it, and then write his copy after you had 
read your
copy. A moments delay between writing and re-reading would fix this.


-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php