php-general Digest 12 Sep 2011 13:05:28 -0000 Issue 7474
Topics (messages 314772 through 314773):
Re: PHP cron job optimization
314772 by: Stuart Dallas
314773 by: Igor Escobar
Administrivia:
To subscribe to the digest, e-mail:
php-general-digest-subscr...@lists.php.net
To unsubscribe from the digest, e-mail:
php-general-digest-unsubscr...@lists.php.net
To post to the list, e-mail:
php-gene...@lists.php.net
----------------------------------------------------------------------
--- Begin Message ---
On 10 Sep 2011, at 09:35, muad shibani wrote:
> I want to design an application that reads news from RSS sources.
> I have about 1000 RSS feed to collect from.
>
> I also will use Cron jobs every 15 minutes to collect the data.
> the question is: Is there a clever way to collect all those feed items
> without exhausting the server
> any Ideas
I designed a job queuing system a while back when I had a similar problem. You
can read about it here: http://stut.net/2009/05/29/php-job-queue/. Set that
type of system up and add a job for each feed, set to run every 15 minutes. You
can then watch the server and tune the number of concurrent job processors so
you get the optimum balance between load and speed.
-Stuart
--
Stuart Dallas
3ft9 Ltd
http://3ft9.com/
--- End Message ---
--- Begin Message ---
Use PHP threads. Do the job separately.. in parts... in other words... you
can't read all them at once.
You can read a little more about php multithreading here:
http://blog.motane.lu/2009/01/02/multithreading-in-php/
You can use a non-relational database like mongo or couchdb to manage where
you stop and where you have to look back to the RSS feed as well.
[]'s
Regards,
Igor Escobar
*Software Engineer
*
+ http://blog.igorescobar.com
+ http://www.igorescobar.com
+ @igorescobar <http://www.twitter.com/igorescobar>
On Sat, Sep 10, 2011 at 10:37 PM, Stuart Dallas <stu...@3ft9.com> wrote:
> On 10 Sep 2011, at 09:35, muad shibani wrote:
>
> > I want to design an application that reads news from RSS sources.
> > I have about 1000 RSS feed to collect from.
> >
> > I also will use Cron jobs every 15 minutes to collect the data.
> > the question is: Is there a clever way to collect all those feed items
> > without exhausting the server
> > any Ideas
>
> I designed a job queuing system a while back when I had a similar problem.
> You can read about it here: http://stut.net/2009/05/29/php-job-queue/. Set
> that type of system up and add a job for each feed, set to run every 15
> minutes. You can then watch the server and tune the number of concurrent job
> processors so you get the optimum balance between load and speed.
>
> -Stuart
>
> --
> Stuart Dallas
> 3ft9 Ltd
> http://3ft9.com/
> --
> PHP General Mailing List (http://www.php.net/)
> To unsubscribe, visit: http://www.php.net/unsub.php
>
>
--- End Message ---
