php-general Digest 11 Sep 2011 00:16:35 -0000 Issue 7473
Topics (messages 314768 through 314771):
Re: PHP cron job optimization
314768 by: Sean Greenslade
314769 by: Gary Golden
314770 by: MUAD SHIBANI
314771 by: Eric Butera
Administrivia:
To subscribe to the digest, e-mail:
php-general-digest-subscr...@lists.php.net
To unsubscribe from the digest, e-mail:
php-general-digest-unsubscr...@lists.php.net
To post to the list, e-mail:
php-gene...@lists.php.net
----------------------------------------------------------------------
--- Begin Message ---
On Sat, Sep 10, 2011 at 4:35 AM, muad shibani <muad.shib...@gmail.com>wrote:
> I want to design an application that reads news from RSS sources.
> I have about 1000 RSS feed to collect from.
>
> I also will use Cron jobs every 15 minutes to collect the data.
> the question is: Is there a clever way to collect all those feed items
> without exhausting the server
> any Ideas
> Thank you in advance
>
Do them one at a time. Fetching web pages isn't a particularly taxing job
for any decent server.
My advice is just to try it. Even if you solution isn't 'elegant' or
'clever,' if it works and doesn't bog down the server, that's a win in my
book.
--
--Zootboy
Sent from my PC.
--- End Message ---
--- Begin Message ---
> I want to design an application that reads news from RSS sources.
>> I have about 1000 RSS feed to collect from.
>>
>> I also will use Cron jobs every 15 minutes to collect the data.
>> the question is: Is there a clever way to collect all those feed items
>> without exhausting the server
>> any Ideas
>> Thank you in advance
Just watch your memory and you'll be fine.
As was stated fetching an rss feed is fast and cheap, so I would
contrariwise think about paralleling if I would you.
signature.asc
Description: OpenPGP digital signature
--- End Message ---
--- Begin Message ---
thanks a lot after I test it I will share the code .. Thanks again
On Sat, Sep 10, 2011 at 11:20 AM, Gary Golden <m...@garygolden.me> wrote:
> > I want to design an application that reads news from RSS sources.
> >> I have about 1000 RSS feed to collect from.
> >>
> >> I also will use Cron jobs every 15 minutes to collect the data.
> >> the question is: Is there a clever way to collect all those feed items
> >> without exhausting the server
> >> any Ideas
> >> Thank you in advance
> Just watch your memory and you'll be fine.
> As was stated fetching an rss feed is fast and cheap, so I would
> contrariwise think about paralleling if I would you.
>
>
--
*
_______________
Muad Shibani*
*
*
Aden Yemen
Mobile: 00967 733045678
Mobile: 00967 711811232
www.muadshibani.com
--- End Message ---
--- Begin Message ---
On Sat, Sep 10, 2011 at 1:47 PM, Sean Greenslade <zootboys...@gmail.com> wrote:
> On Sat, Sep 10, 2011 at 4:35 AM, muad shibani <muad.shib...@gmail.com>wrote:
>
>> I want to design an application that reads news from RSS sources.
>> I have about 1000 RSS feed to collect from.
>>
>> I also will use Cron jobs every 15 minutes to collect the data.
>> the question is: Is there a clever way to collect all those feed items
>> without exhausting the server
>> any Ideas
>> Thank you in advance
>>
>
> Do them one at a time. Fetching web pages isn't a particularly taxing job
> for any decent server.
>
> My advice is just to try it. Even if you solution isn't 'elegant' or
> 'clever,' if it works and doesn't bog down the server, that's a win in my
> book.
>
> --
> --Zootboy
>
> Sent from my PC.
>
Traversing the internet is one of the most expensive IO operations
that can be performed. The OP should try and split the work into
several processes and utilize async connections via a mechanism like
curl_multi [1]. Doing 1,000 connections one at a time is abysmal if
you're blocked for even 3 seconds per connection.
I'd work by making a two step process that has one cron job for
constantly fetching/updating feeds into a store & then another process
that constantly checks for updates inside aforementioned store.
[1] http://us3.php.net/manual/en/function.curl-multi-add-handle.php
--- End Message ---
