Re: [SLUG] Send EOF to Browser from LAMP stack.
hi, 2010/1/21 Peter Rundle pe...@aerodonetix.com.au: It's alleged Ken Foskey did scribe: You could try closing STDOUT which will tell apache that your script has stopped output. This is interesting idea, I think I will give that a try if I can find out how to get hold of the STDOUT file pointer. In perl I executed a background task with an system( command ); to perform the background tasks. I then emailed a reponse to the client to tell them the job was done. That's the kinda thing I need to do. I was hoping to avoid doing a system command because the action I need to do is easily done right away in the php (database connection is already open with right privileges etc). I just need to let the browser know that there's not gonna be any more output, it's finished go and render the page and be happy. If I call a system command I have to pass all the info I current have in the application open a new connection to the database in the other process etc. Doable but if I can just close the network connection that'd be neater. Cron jobs aren't the go, this is an event driven task that needs to happen when the event occurs, not some minutes/hours later when the cron jobs wakes up at the specified interval. i'm interested in the requirements that led to this problem. to be honest, it sounds a bit fishy from a design point of view. maybe it just has to be that way, but requiring big chunks of computation that have to happen straight away, are triggered by network requests (that don't need to see the results of the processing in real time) is not something i'd allow unless absolutely necessary. at the very least, i'd want the resource that triggers that access controlled. sorry if this is a blind alley, but this is a problem i would be trying *not* to solve if possible. any architecture that requires this will be harder to scale and easier to DOS, which might not bite you straight away, but will probably bite you at some point. so, the client request that triggers the processing doesn't see the results. what is it about the app that requires it to happen straight away? is it a consistency issue - no other client should see the site before the processing is done? would it be enough for other clients to just see the site with all or none of the processing finished? cheers, justin -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Send EOF to Browser from LAMP stack.
I said it might take a few seconds, I didn't say it was computationally heavy. justin randell wrote: hi, 2010/1/21 Peter Rundle pe...@aerodonetix.com.au: It's alleged Ken Foskey did scribe: You could try closing STDOUT which will tell apache that your script has stopped output. This is interesting idea, I think I will give that a try if I can find out how to get hold of the STDOUT file pointer. In perl I executed a background task with an system( command ); to perform the background tasks. I then emailed a reponse to the client to tell them the job was done. That's the kinda thing I need to do. I was hoping to avoid doing a system command because the action I need to do is easily done right away in the php (database connection is already open with right privileges etc). I just need to let the browser know that there's not gonna be any more output, it's finished go and render the page and be happy. If I call a system command I have to pass all the info I current have in the application open a new connection to the database in the other process etc. Doable but if I can just close the network connection that'd be neater. Cron jobs aren't the go, this is an event driven task that needs to happen when the event occurs, not some minutes/hours later when the cron jobs wakes up at the specified interval. i'm interested in the requirements that led to this problem. to be honest, it sounds a bit fishy from a design point of view. maybe it just has to be that way, but requiring big chunks of computation that have to happen straight away, are triggered by network requests (that don't need to see the results of the processing in real time) is not something i'd allow unless absolutely necessary. at the very least, i'd want the resource that triggers that access controlled. sorry if this is a blind alley, but this is a problem i would be trying *not* to solve if possible. any architecture that requires this will be harder to scale and easier to DOS, which might not bite you straight away, but will probably bite you at some point. so, the client request that triggers the processing doesn't see the results. what is it about the app that requires it to happen straight away? is it a consistency issue - no other client should see the site before the processing is done? would it be enough for other clients to just see the site with all or none of the processing finished? cheers, justin -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Send EOF to Browser from LAMP stack.
hi, 2010/1/21 Peter Rundle pe...@aerodonetix.com.au: I said it might take a few seconds, I didn't say it was computationally heavy. fair enough, but still worth questioning, i think. for a typical app, i'd be concerned if a fat apache child process was spending more than a quarter to a third of a second servicing a single request. cheers justin justin randell wrote: hi, 2010/1/21 Peter Rundle pe...@aerodonetix.com.au: It's alleged Ken Foskey did scribe: You could try closing STDOUT which will tell apache that your script has stopped output. This is interesting idea, I think I will give that a try if I can find out how to get hold of the STDOUT file pointer. In perl I executed a background task with an system( command ); to perform the background tasks. I then emailed a reponse to the client to tell them the job was done. That's the kinda thing I need to do. I was hoping to avoid doing a system command because the action I need to do is easily done right away in the php (database connection is already open with right privileges etc). I just need to let the browser know that there's not gonna be any more output, it's finished go and render the page and be happy. If I call a system command I have to pass all the info I current have in the application open a new connection to the database in the other process etc. Doable but if I can just close the network connection that'd be neater. Cron jobs aren't the go, this is an event driven task that needs to happen when the event occurs, not some minutes/hours later when the cron jobs wakes up at the specified interval. i'm interested in the requirements that led to this problem. to be honest, it sounds a bit fishy from a design point of view. maybe it just has to be that way, but requiring big chunks of computation that have to happen straight away, are triggered by network requests (that don't need to see the results of the processing in real time) is not something i'd allow unless absolutely necessary. at the very least, i'd want the resource that triggers that access controlled. sorry if this is a blind alley, but this is a problem i would be trying *not* to solve if possible. any architecture that requires this will be harder to scale and easier to DOS, which might not bite you straight away, but will probably bite you at some point. so, the client request that triggers the processing doesn't see the results. what is it about the app that requires it to happen straight away? is it a consistency issue - no other client should see the site before the processing is done? would it be enough for other clients to just see the site with all or none of the processing finished? cheers, justin -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Send EOF to Browser from LAMP stack.
On Thu, Jan 21, 2010 at 04:28:30PM +0100, justin randell wrote: hi, 2010/1/21 Peter Rundle pe...@aerodonetix.com.au: I said it might take a few seconds, I didn't say it was computationally heavy. fair enough, but still worth questioning, i think. for a typical app, i'd be concerned if a fat apache child process was spending more than a quarter to a third of a second servicing a single request. We have a similar issue here at work, and for us it just creeped up on us. The code was written to deal with uploads and parse them / process them / stick bits into the database. As time goes by the uploads grow from kb to Mb and change from simple format to vast swathes of @#...@^ xml. The same job which used to take less than a few seconds now causes timeouts in apache which we've had to raise and/or the browser. Which we can't do much about. This is all in java so it's not too difficult to make it asynch wrt to the original http request. And leave the use perhaps with an ajaxy update page. I know next to nothing about PHP, but can't you spawn another thread unrelated to a client http request? Then you can finish this page, redirect them to another page which just waits for the result to come back - checking now and then via the db or some file on disk. Other respondents of cron fail (AFAICS) to satisfy the responsiveness that the original poster wants. Matt -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
[SLUG] Send EOF to Browser from LAMP stack.
Hi Sluggers, I hope this question is appropriate for this list. I have a PHP web-site running on Apache and Linux. A PHP routine produces a page that is sent back to the browser, but then it has some house-keeping to do which takes some time, perhaps many seconds but the housekeeping doesn't result in any more output to the browser (any output from that point on goes to a log). What I would like to do is end/close the http request so that the browser gets the HTTP equivelent of an EOF but allow the php script to keep running. Now flush() does send the output to date to the browser but the browsers busy icon keeps running because the http session isn't closed until the php ends. I thought of doing a fork but the PHP docs say that fork doesn't work when php is running under apache. I could write a shell script and invoke that with a system/exec call from php and have the shell run into the background and do the house-keeping thus allowing the php to finsih, but I'm wondering if sluggers know of a better way (tm). Thanks Pete -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Send EOF to Browser from LAMP stack.
quote who=Peter Rundle What I would like to do is end/close the http request so that the browser gets the HTTP equivelent of an EOF but allow the php script to keep running. Now flush() does send the output to date to the browser but the browsers busy icon keeps running because the http session isn't closed until the php ends. You're very likely to find a solution to this in the WordPress code base, particularly related to the WP-Cron code (an implementation of poor man's cron, in order to run scheduled jobs based on client requests rather than exact times as with Really Proper Cron). - Jeff -- linux.conf.au 2010: Wellington, NZhttp://www.lca2010.org.nz/ Try Thunderbird, like Evolution but without all the features. - Pia Waugh -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Send EOF to Browser from LAMP stack.
On Thu, 2010-01-21 at 08:22 +1100, Peter Rundle wrote: Hi Sluggers, I hope this question is appropriate for this list. I have a PHP web-site running on Apache and Linux. A PHP routine produces a page that is sent back to the browser, but then it has some house-keeping to do which takes some time, perhaps many seconds but the housekeeping doesn't result in any more output to the browser (any output from that point on goes to a log). What I would like to do is end/close the http request so that the browser gets the HTTP equivelent of an EOF but allow the php script to keep running. Now flush() does send the output to date to the browser but the browsers busy icon keeps running because the http session isn't closed until the php ends. I thought of doing a fork but the PHP docs say that fork doesn't work when php is running under apache. I could write a shell script and invoke that with a system/exec call from php and have the shell run into the background and do the house-keeping thus allowing the php to finsih, but I'm wondering if sluggers know of a better way (tm). Thanks Pete You could try closing STDOUT which will tell apache that your script has stopped output. In perl I executed a background task with an system( command ); to perform the background tasks. I then emailed a reponse to the client to tell them the job was done. Ta Ken -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Send EOF to Browser from LAMP stack.
hi, 2010/1/20 Peter Rundle pe...@aerodonetix.com.au: Hi Sluggers, I hope this question is appropriate for this list. I have a PHP web-site running on Apache and Linux. A PHP routine produces a page that is sent back to the browser, but then it has some house-keeping to do which takes some time, perhaps many seconds but the housekeeping doesn't result in any more output to the browser (any output from that point on goes to a log). http://php.net/flush and possibly http://php.net/ob_flush are likely the droids you are looking for. cheers justin -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Send EOF to Browser from LAMP stack.
hi, 2010/1/20 Peter Rundle pe...@aerodonetix.com.au: Hi Sluggers, I hope this question is appropriate for this list. I have a PHP web-site running on Apache and Linux. A PHP routine produces a page that is sent back to the browser, but then it has some house-keeping to do which takes some time, perhaps many seconds but the housekeeping doesn't result in any more output to the browser (any output from that point on goes to a log). What I would like to do is end/close the http request so that the browser gets the HTTP equivelent of an EOF but allow the php script to keep running. Now flush() does send the output to date to the browser but the browsers busy icon keeps running because the http session isn't closed until the php ends. woops, ignore my last post about flush(), should have read the whole post. i blame the wine and being on holiday in europe... I thought of doing a fork but the PHP docs say that fork doesn't work when php is running under apache. I could write a shell script and invoke that with a system/exec call from php and have the shell run into the background and do the house-keeping thus allowing the php to finsih, but I'm wondering if sluggers know of a better way (tm). how immediate does this need to be? unless this really needs to run straight away, i'd put the needs background work request in a simple queue and process it via a cron script. IMHO, putting a layer between a web request and any serious out-of-band processing is the best way to handle these cases. cheers justin -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Send EOF to Browser from LAMP stack.
how immediate does this need to be? unless this really needs to run
straight away, i'd put the needs background work request in a simple
queue and process it via a cron script. IMHO, putting a layer between
a web request and any serious out-of-band processing is the best way
to handle these cases.
You could set up a cron job to poll a php script hourly.
php /var/www/html/cron/cron.php
Have that script query a queue:
SELECT script_name, parameters FROM cron_table WHERE done = 'NO';
Iterate through the results and run the script(s):
foreach($results as $row)
{
extract($row);
// assuming the params are in the formate a:1/b:2/c:3
$params = explode('/', $row['parameters']);
// populate the variables that the script is expecting
foreach($params as $p)
{
$z = explode(':', $p);
$$p[0] = $p[1];
}
// include the script that actually does stuff
include '/var/www/html/cron/' . $row['script_name'];
}
Using this way you only have to set up one cron job and you can point it
at any PHP script by just getting the page to insert a row in a database.
Just an idea, code probably is broken somewhere but anyway...
Cheers
P
--
*
www.recruitonline.com.au http://www.recruitonline.com.au
*
* *Recruitment, Advertising, Document Managment, CRM, Online Storage
and Search hosted services.*
--
SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/
Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Send EOF to Browser from LAMP stack.
It's alleged Ken Foskey did scribe: You could try closing STDOUT which will tell apache that your script has stopped output. This is interesting idea, I think I will give that a try if I can find out how to get hold of the STDOUT file pointer. In perl I executed a background task with an system( command ); to perform the background tasks. I then emailed a reponse to the client to tell them the job was done. That's the kinda thing I need to do. I was hoping to avoid doing a system command because the action I need to do is easily done right away in the php (database connection is already open with right privileges etc). I just need to let the browser know that there's not gonna be any more output, it's finished go and render the page and be happy. If I call a system command I have to pass all the info I current have in the application open a new connection to the database in the other process etc. Doable but if I can just close the network connection that'd be neater. Cron jobs aren't the go, this is an event driven task that needs to happen when the event occurs, not some minutes/hours later when the cron jobs wakes up at the specified interval. Thanks Pete P.S if people reply to my email address it will probably block, just reply to slug I'm subscribed -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Send EOF to Browser from LAMP stack.
2010/1/21 Peter Rundle pe...@aerodonetix.com.au: Cron jobs aren't the go, this is an event driven task that needs to happen when the event occurs, not some minutes/hours later when the cron jobs wakes up at the specified interval. Cron is not the only way to process things in the background. For instance - how about the request generates a file in a specified directory, a daemon which keeps watching this directory (stat() on the directory, or inotify) will notice the new file and act based on job data contained in the file. Just make sure to cover corner cases - e.g. * create the file under some temporary name (e.g. .filename) until it contains all the required information then rename it to the name/directory under which the background process looks for it. * Make sure to use a unique file name for each request. * Monitor the background process with monit. * Monitor for accumulation of jobs in the queue Cheers, --Amos -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
