RE: [PHP] How to make sure that the target file to read is not under writing by others?
-Original Message- From: Tom Worster [mailto:f...@thefsb.org] Sent: 20 August 2009 01:28 PM To: Clancy; php-general@lists.php.net Subject: Re: [PHP] How to make sure that the target file to read is not under writing by others? On 8/19/09 9:56 PM, "Clancy" wrote: > I gather from this discussion that PHP allows two users to open a file > for R/W? I had assumed it wouldn't. i think php does allow this. but i'm not sure all file systems do. -- PHP does allow more than one process to open a file for read-write, as does every other filesystem I know of (PHP doesn't have a filesystem, it implements a subset of file commands common to all OS's [some differences between *nix and Windows] that it uses to communicate with the underlying OS filesystem). If there's a need to ensure that multiple processes don't step on each others toes the file can be locked. The two most common file accesses are read and append, where a lock is mostly not required. It's only in special cases where you want to write or update something and want to be sure that another process isn't trying to update the same data or trying to read data that may no longer be consistent (until you finish your update) that you'd want to lock it. Cheers Arno -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] How to make sure that the target file to read is not under writing by others?
On 8/19/09 9:56 PM, "Clancy" wrote: > I gather from this discussion that PHP allows two users to open a file for > R/W? I had assumed it wouldn't. i think php does allow this. but i'm not sure all file systems do. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] How to make sure that the target file to read is not under writing by others?
On Wed, 19 Aug 2009 08:13:56 -0400, kyle.sm...@inforonics.com (Kyle Smith)
wrote:
>Nitebirdz wrote:
>> On Wed, Aug 19, 2009 at 11:59:39AM +0100, Ashley Sheridan wrote:
>>
>>>
>>> No, what you're saying is 'use a log file in order to know when to look
>>> at another log file'. What would happen if you tried to access the
>>> control log file whilst it was in the process of being written to?
>>> Admittedly, you reduce your chances of failure because there is less
>>> data being written to the control log than the actual log, but the
>>> chance of reading incomplete data is still there!
>>>
>>>
>>
>> WARNING: total newbie here.
>>
>> If I understood Arno correctly, he was recommending to implement
>> something like the old "/var/run/*pid" files in UNIX. That way, you can
>> control whether or not the previous run is already done with the file
>> before you move on.
>That is definitely the correct approach. Have the script which copies
>the log file touch a file called 'log_file.write' or some such. When
>it's done, remove the file. Your PHP script should exit if that file
>exists. Of course, given the 30 minute cron cycle, it would then have
>to wait until the next cycle. Maybe run it more often.
I gather from this discussion that PHP allows two users to open a file for R/W?
I had
assumed it wouldn't.
Anyway, how about:
$user = 'Fred'; $try_again = true;
If (!file_exists('Busy.txt'))
{
File_put_contents ('Busy.txt', $user);
If ($user == File_get_contents ('Busy.txt')
{
// Do what you have to
Unlink ('Busy.txt');
$try_again = false;
}
}
if ($try_again)
{
// Come back in 5 minutes
}
This has a theoretical weakness, in that Joe could check for Busy.txt in the
interval
between your checking and writing it, and then write his copy after you had
read your
copy. A moments delay between writing and re-reading would fix this.
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
RE: [PHP] How to make sure that the target file to read is not under writing by others?
-Original Message- From: Ashley Sheridan [mailto:a...@ashleysheridan.co.uk] Sent: 19 August 2009 03:34 PM To: a...@dotcontent.net Cc: 'Dengxule'; 'Php Maillist' Subject: RE: [PHP] How to make sure that the target file to read is not under writing by others? On Wed, 2009-08-19 at 15:31 +0200, Arno Kuhl wrote: > (any computer would instantly crash if that sort of basic housekeeping > wasn't done) No, it really wouldn't! If it did, then you'd never have video playing software out there that supported broken downloads, no preview software for semi-downloaded items. Have you ever been sent a large jpeg over a slow connection on a messenger program? You can open the file way before it's finished downloading and actually see it progressively load in. Thanks, Ash http://www.ashleysheridan.co.uk --- If anyone was left a bit unsure about whether or not a write command in PHP could result in a file on a local filesystem being left in a semi-written state (i.e. some other script started reading the file while your script had only written part of its output string) rest assured that fwrite() always writes *all* the contents to the file without the need of flock(), by the time any other process (whether PHP or not) reads that file (assuming you're not writing megabytes of data with a single fwrite()). To quote the PHP manual: "If handle was fopen()ed in append mode, fwrite()s are atomic (unless the size of string exceeds the filesystem's block size, on some platforms, and as long as the file is on a local filesystem). That is, there is no need to flock() a resource before calling fwrite(); all of the data will be written without interruption." Cheers Arno -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
RE: [PHP] How to make sure that the target file to read is not under writing by others?
On Wed, 2009-08-19 at 15:31 +0200, Arno Kuhl wrote: > (any computer would instantly crash if that sort of basic > housekeeping wasn't done) No, it really wouldn't! If it did, then you'd never have video playing software out there that supported broken downloads, no preview software for semi-downloaded items. Have you ever been sent a large jpeg over a slow connection on a messenger program? You can open the file way before it's finished downloading and actually see it progressively load in. Thanks, Ash http://www.ashleysheridan.co.uk -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
RE: [PHP] How to make sure that the target file to read is not under writing by others?
-Original Message- From: Ashley Sheridan [mailto:a...@ashleysheridan.co.uk] Sent: 19 August 2009 01:00 PM To: a...@dotcontent.net Cc: 'Dengxule'; 'Php Maillist' Subject: RE: [PHP] How to make sure that the target file to read is not under writing by others? On Wed, 2009-08-19 at 12:56 +0200, Arno Kuhl wrote: > -Original Message- > From: Ashley Sheridan [mailto:a...@ashleysheridan.co.uk] > Sent: 19 August 2009 11:57 AM > To: a...@dotcontent.net > Cc: 'Dengxule'; 'Php Maillist' > Subject: RE: [PHP] How to make sure that the target file to read is > not under writing by others? > > On Wed, 2009-08-19 at 11:55 +0200, Arno Kuhl wrote: > > -Original Message- > > From: Dengxule [mailto:dengx...@gmail.com] > > Sent: 19 August 2009 09:56 AM > > To: Php Maillist > > Subject: [PHP] How to make sure that the target file to read is not > > under writing by others? > > > > Hi everyone: > > > > I have a crontab command to execuate my php-script every half an hour. > > > > The mission of the php-script is to open a file(log file), examine it. > > > > The target file(log file) is transported to local every half an hour. > > > > I've no idea how much time it will costs and on the other hand, i > > want to make sure the file i'm openning is Completely Written. > > > > Any instruction will be grateful. > > > > > > PS: I've made a test. My php-script just "fopen" a local file being > > transported in, and the "fopen" returns no FALSE value but a resource. > > > > > > Dengxule > > 2009/08/19 > > > > > > Check the timestamp of the log file. Alternatively you can have a > > log control file that is updated by the program transferring the log > > file only when the transfer is completed, and work on that. > > > > Cheers > > Arno > > > > > The log control file wouldn't work, as how would you know when to open > the control file?! > > Thanks, > Ash > http://www.ashleysheridan.co.uk > > -- > > You can open the control file any time you want. The info in the > control file tells you whether to proceed. The transfer program adds > the entry when the log has been transferred, the processing script > updates the entry to indicate it's been processed. You can either add > a new entry each time (useful for monitoring to fine-tune the timing, > or for keeping a history) or work on a single entry that's overwritten each time the log is transferred. > > Cheers > Arno > > No, what you're saying is 'use a log file in order to know when to look at another log file'. What would happen if you tried to access the control log file whilst it was in the process of being written to? Admittedly, you reduce your chances of failure because there is less data being written to the control log than the actual log, but the chance of reading incomplete data is still there! Thanks, Ash http://www.ashleysheridan.co.uk A control file is used to control processing, a log file is used to log events. I'm definitely talking about a control file. If the control file happens to keep a log then it's a bonus, but not its primary purpose. Access the control file while being written to? I'm talking about a short single-line entry, and the file is only written to when the write command is processed. Copying a file is very different to issuing a single write command. Don't forget that PHP and webservers run on top of operating systems, and the OS ultimately handles all file activity. The OS ensures that another process can't read a file that is halfway through a write command (any computer would instantly crash if that sort of basic housekeeping wasn't done). If you don't trust the OS then file locking can be used (though pointless in this case). For the ultra-paranoids who don't trust OS's or locks you can either (a) choose a different career, or (b) add a time-stamp to the beginning of the entry and check that the entry wasn't written in the last few nanoseconds (if you get this far I'd recommend option a). You can even go so far as to check the length of the timestamp string to be sure that you got all of it, and if it's too short you know you're busy reading a file that's halfway through a write command, and sleep for a millisecond and then try again ;) Cheers Arno -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] How to make sure that the target file to read is not under writing by others?
On 8/19/09 3:55 AM, "Dengxule" wrote: > I have a crontab command to execuate my php-script every half an hour. > > The mission of the php-script is to open a file(log file), examine it. > > The target file(log file) is transported to local every half an hour. > > I've no idea how much time it will costs and on the other hand, i want to > make sure the file i'm openning is Completely Written. > > Any instruction will be grateful. perhaps you could use cronolog (http://cronolog.org/) to rotate the log files every half hour? that way you could be sure that the file you open to process/copy is one that cronolog has finished writing to. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] How to make sure that the target file to read is not under writing by others?
Nitebirdz wrote: On Wed, Aug 19, 2009 at 11:59:39AM +0100, Ashley Sheridan wrote: No, what you're saying is 'use a log file in order to know when to look at another log file'. What would happen if you tried to access the control log file whilst it was in the process of being written to? Admittedly, you reduce your chances of failure because there is less data being written to the control log than the actual log, but the chance of reading incomplete data is still there! WARNING: total newbie here. If I understood Arno correctly, he was recommending to implement something like the old "/var/run/*pid" files in UNIX. That way, you can control whether or not the previous run is already done with the file before you move on. That is definitely the correct approach. Have the script which copies the log file touch a file called 'log_file.write' or some such. When it's done, remove the file. Your PHP script should exit if that file exists. Of course, given the 30 minute cron cycle, it would then have to wait until the next cycle. Maybe run it more often.
Re: [PHP] How to make sure that the target file to read is not under writing by others?
On Wed, Aug 19, 2009 at 11:59:39AM +0100, Ashley Sheridan wrote: > > No, what you're saying is 'use a log file in order to know when to look > at another log file'. What would happen if you tried to access the > control log file whilst it was in the process of being written to? > Admittedly, you reduce your chances of failure because there is less > data being written to the control log than the actual log, but the > chance of reading incomplete data is still there! > WARNING: total newbie here. If I understood Arno correctly, he was recommending to implement something like the old "/var/run/*pid" files in UNIX. That way, you can control whether or not the previous run is already done with the file before you move on. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
RE: [PHP] How to make sure that the target file to read is not under writing by others?
On Wed, 2009-08-19 at 12:56 +0200, Arno Kuhl wrote: > -Original Message- > From: Ashley Sheridan [mailto:a...@ashleysheridan.co.uk] > Sent: 19 August 2009 11:57 AM > To: a...@dotcontent.net > Cc: 'Dengxule'; 'Php Maillist' > Subject: RE: [PHP] How to make sure that the target file to read is not > under writing by others? > > On Wed, 2009-08-19 at 11:55 +0200, Arno Kuhl wrote: > > -Original Message- > > From: Dengxule [mailto:dengx...@gmail.com] > > Sent: 19 August 2009 09:56 AM > > To: Php Maillist > > Subject: [PHP] How to make sure that the target file to read is not > > under writing by others? > > > > Hi everyone: > > > > I have a crontab command to execuate my php-script every half an hour. > > > > The mission of the php-script is to open a file(log file), examine it. > > > > The target file(log file) is transported to local every half an hour. > > > > I've no idea how much time it will costs and on the other hand, i want > > to make sure the file i'm openning is Completely Written. > > > > Any instruction will be grateful. > > > > > > PS: I've made a test. My php-script just "fopen" a local file being > > transported in, and the "fopen" returns no FALSE value but a resource. > > > > > > Dengxule > > 2009/08/19 > > > > > > Check the timestamp of the log file. Alternatively you can have a log > > control file that is updated by the program transferring the log file > > only when the transfer is completed, and work on that. > > > > Cheers > > Arno > > > > > The log control file wouldn't work, as how would you know when to open the > control file?! > > Thanks, > Ash > http://www.ashleysheridan.co.uk > > -- > > You can open the control file any time you want. The info in the control > file tells you whether to proceed. The transfer program adds the entry when > the log has been transferred, the processing script updates the entry to > indicate it's been processed. You can either add a new entry each time > (useful for monitoring to fine-tune the timing, or for keeping a history) or > work on a single entry that's overwritten each time the log is transferred. > > Cheers > Arno > > No, what you're saying is 'use a log file in order to know when to look at another log file'. What would happen if you tried to access the control log file whilst it was in the process of being written to? Admittedly, you reduce your chances of failure because there is less data being written to the control log than the actual log, but the chance of reading incomplete data is still there! Thanks, Ash http://www.ashleysheridan.co.uk -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
RE: [PHP] How to make sure that the target file to read is not under writing by others?
-Original Message- From: Ashley Sheridan [mailto:a...@ashleysheridan.co.uk] Sent: 19 August 2009 11:57 AM To: a...@dotcontent.net Cc: 'Dengxule'; 'Php Maillist' Subject: RE: [PHP] How to make sure that the target file to read is not under writing by others? On Wed, 2009-08-19 at 11:55 +0200, Arno Kuhl wrote: > -Original Message- > From: Dengxule [mailto:dengx...@gmail.com] > Sent: 19 August 2009 09:56 AM > To: Php Maillist > Subject: [PHP] How to make sure that the target file to read is not > under writing by others? > > Hi everyone: > > I have a crontab command to execuate my php-script every half an hour. > > The mission of the php-script is to open a file(log file), examine it. > > The target file(log file) is transported to local every half an hour. > > I've no idea how much time it will costs and on the other hand, i want > to make sure the file i'm openning is Completely Written. > > Any instruction will be grateful. > > > PS: I've made a test. My php-script just "fopen" a local file being > transported in, and the "fopen" returns no FALSE value but a resource. > > > Dengxule > 2009/08/19 > > > Check the timestamp of the log file. Alternatively you can have a log > control file that is updated by the program transferring the log file > only when the transfer is completed, and work on that. > > Cheers > Arno > > The log control file wouldn't work, as how would you know when to open the control file?! Thanks, Ash http://www.ashleysheridan.co.uk -- You can open the control file any time you want. The info in the control file tells you whether to proceed. The transfer program adds the entry when the log has been transferred, the processing script updates the entry to indicate it's been processed. You can either add a new entry each time (useful for monitoring to fine-tune the timing, or for keeping a history) or work on a single entry that's overwritten each time the log is transferred. Cheers Arno -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
RE: [PHP] How to make sure that the target file to read is not under writing by others?
On Wed, 2009-08-19 at 11:55 +0200, Arno Kuhl wrote: > -Original Message- > From: Dengxule [mailto:dengx...@gmail.com] > Sent: 19 August 2009 09:56 AM > To: Php Maillist > Subject: [PHP] How to make sure that the target file to read is not under > writing by others? > > Hi everyone: > > I have a crontab command to execuate my php-script every half an hour. > > The mission of the php-script is to open a file(log file), examine it. > > The target file(log file) is transported to local every half an hour. > > I've no idea how much time it will costs and on the other hand, i want to > make sure the file i'm openning is Completely Written. > > Any instruction will be grateful. > > > PS: I've made a test. My php-script just "fopen" a local file being > transported in, and the "fopen" returns no FALSE value but a resource. > > > Dengxule > 2009/08/19 > > > Check the timestamp of the log file. Alternatively you can have a log > control file that is updated by the program transferring the log file only > when the transfer is completed, and work on that. > > Cheers > Arno > > The log control file wouldn't work, as how would you know when to open the control file?! Thanks, Ash http://www.ashleysheridan.co.uk -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
RE: [PHP] How to make sure that the target file to read is not under writing by others?
-Original Message- From: Dengxule [mailto:dengx...@gmail.com] Sent: 19 August 2009 09:56 AM To: Php Maillist Subject: [PHP] How to make sure that the target file to read is not under writing by others? Hi everyone: I have a crontab command to execuate my php-script every half an hour. The mission of the php-script is to open a file(log file), examine it. The target file(log file) is transported to local every half an hour. I've no idea how much time it will costs and on the other hand, i want to make sure the file i'm openning is Completely Written. Any instruction will be grateful. PS: I've made a test. My php-script just "fopen" a local file being transported in, and the "fopen" returns no FALSE value but a resource. Dengxule 2009/08/19 Check the timestamp of the log file. Alternatively you can have a log control file that is updated by the program transferring the log file only when the transfer is completed, and work on that. Cheers Arno -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] How to make sure that the target file to read is not under writing by others?
On Wed, 2009-08-19 at 15:55 +0800, Dengxule wrote: > Hi everyone: > > I have a crontab command to execuate my php-script every half an hour. > > The mission of the php-script is to open a file(log file), examine it. > > The target file(log file) is transported to local every half an hour. > > I've no idea how much time it will costs and on the other hand, i want to > make sure the file i'm openning is Completely Written. > > Any instruction will be grateful. > > > PS: I've made a test. My php-script just "fopen" a local file being > transported in, and the "fopen" returns no FALSE value but a resource. > > > Dengxule > 2009/08/19 I had to do something similar in my last job. First thing to do is figure out how large the log you expect to parse is going to be, and from there, have a guess at how long it will take to transfer fully from elsewhere. I'd make the cron script a bit more frequent, but only ask it to look for files that have changed between x and y minutes ago, where y is a minute or two over the transfer duration. Then you can either mark the file in some way, or move it to another location to indicate it has been processed. Thanks, Ash http://www.ashleysheridan.co.uk -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
[PHP] How to make sure that the target file to read is not under writing by others?
Hi everyone: I have a crontab command to execuate my php-script every half an hour. The mission of the php-script is to open a file(log file), examine it. The target file(log file) is transported to local every half an hour. I've no idea how much time it will costs and on the other hand, i want to make sure the file i'm openning is Completely Written. Any instruction will be grateful. PS: I've made a test. My php-script just "fopen" a local file being transported in, and the "fopen" returns no FALSE value but a resource. Dengxule 2009/08/19
