Re: [PHP] PHP cron job optimization
Use PHP threads. Do the job separately.. in parts... in other words... you can't read all them at once. You can read a little more about php multithreading here: http://blog.motane.lu/2009/01/02/multithreading-in-php/ You can use a non-relational database like mongo or couchdb to manage where you stop and where you have to look back to the RSS feed as well. []'s Regards, Igor Escobar *Software Engineer * + http://blog.igorescobar.com + http://www.igorescobar.com + @igorescobar http://www.twitter.com/igorescobar On Sat, Sep 10, 2011 at 10:37 PM, Stuart Dallas stu...@3ft9.com wrote: On 10 Sep 2011, at 09:35, muad shibani wrote: I want to design an application that reads news from RSS sources. I have about 1000 RSS feed to collect from. I also will use Cron jobs every 15 minutes to collect the data. the question is: Is there a clever way to collect all those feed items without exhausting the server any Ideas I designed a job queuing system a while back when I had a similar problem. You can read about it here: http://stut.net/2009/05/29/php-job-queue/. Set that type of system up and add a job for each feed, set to run every 15 minutes. You can then watch the server and tune the number of concurrent job processors so you get the optimum balance between load and speed. -Stuart -- Stuart Dallas 3ft9 Ltd http://3ft9.com/ -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] PHP cron job optimization
Other good point is: always set a timeout connection when you're getting the RSS data to avoid your thread get stuck unnecessary. Use cURL (is much more faster then file_get_contents). Multithreading in PHP with cURL http://devzone.zend.com/article/3341 Regards, Igor Escobar *Software Engineer * + http://blog.igorescobar.com + http://www.igorescobar.com + @igorescobar http://www.twitter.com/igorescobar On Mon, Sep 12, 2011 at 10:05 AM, Igor Escobar titiolin...@gmail.comwrote: Use PHP threads. Do the job separately.. in parts... in other words... you can't read all them at once. You can read a little more about php multithreading here: http://blog.motane.lu/2009/01/02/multithreading-in-php/ You can use a non-relational database like mongo or couchdb to manage where you stop and where you have to look back to the RSS feed as well. []'s Regards, Igor Escobar *Software Engineer * + http://blog.igorescobar.com + http://www.igorescobar.com + @igorescobar http://www.twitter.com/igorescobar On Sat, Sep 10, 2011 at 10:37 PM, Stuart Dallas stu...@3ft9.com wrote: On 10 Sep 2011, at 09:35, muad shibani wrote: I want to design an application that reads news from RSS sources. I have about 1000 RSS feed to collect from. I also will use Cron jobs every 15 minutes to collect the data. the question is: Is there a clever way to collect all those feed items without exhausting the server any Ideas I designed a job queuing system a while back when I had a similar problem. You can read about it here: http://stut.net/2009/05/29/php-job-queue/. Set that type of system up and add a job for each feed, set to run every 15 minutes. You can then watch the server and tune the number of concurrent job processors so you get the optimum balance between load and speed. -Stuart -- Stuart Dallas 3ft9 Ltd http://3ft9.com/ -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] PHP cron job optimization
On Mon, Sep 12, 2011 at 9:37 AM, Igor Escobar titiolin...@gmail.com wrote: Other good point is: always set a timeout connection when you're getting the RSS data to avoid your thread get stuck unnecessary. Use cURL (is much more faster then file_get_contents). Multithreading in PHP with cURL http://devzone.zend.com/article/3341 Regards, Igor Escobar *Software Engineer * + http://blog.igorescobar.com + http://www.igorescobar.com + @igorescobar http://www.twitter.com/igorescobar On Mon, Sep 12, 2011 at 10:05 AM, Igor Escobar titiolin...@gmail.comwrote: Use PHP threads. Do the job separately.. in parts... in other words... you can't read all them at once. You can read a little more about php multithreading here: http://blog.motane.lu/2009/01/02/multithreading-in-php/ You can use a non-relational database like mongo or couchdb to manage where you stop and where you have to look back to the RSS feed as well. []'s Regards, Igor Escobar *Software Engineer * + http://blog.igorescobar.com + http://www.igorescobar.com + @igorescobar http://www.twitter.com/igorescobar On Sat, Sep 10, 2011 at 10:37 PM, Stuart Dallas stu...@3ft9.com wrote: On 10 Sep 2011, at 09:35, muad shibani wrote: I want to design an application that reads news from RSS sources. I have about 1000 RSS feed to collect from. I also will use Cron jobs every 15 minutes to collect the data. the question is: Is there a clever way to collect all those feed items without exhausting the server any Ideas I designed a job queuing system a while back when I had a similar problem. You can read about it here: http://stut.net/2009/05/29/php-job-queue/. Set that type of system up and add a job for each feed, set to run every 15 minutes. You can then watch the server and tune the number of concurrent job processors so you get the optimum balance between load and speed. -Stuart -- Stuart Dallas 3ft9 Ltd http://3ft9.com/ -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php Thread != Multi Process. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] PHP cron job optimization
@Eric ok ;) Regards, Igor Escobar *Software Engineer * + http://blog.igorescobar.com + http://www.igorescobar.com + @igorescobar http://www.twitter.com/igorescobar On Mon, Sep 12, 2011 at 10:52 AM, Eric Butera eric.but...@gmail.com wrote: On Mon, Sep 12, 2011 at 9:37 AM, Igor Escobar titiolin...@gmail.com wrote: Other good point is: always set a timeout connection when you're getting the RSS data to avoid your thread get stuck unnecessary. Use cURL (is much more faster then file_get_contents). Multithreading in PHP with cURL http://devzone.zend.com/article/3341 Regards, Igor Escobar *Software Engineer * + http://blog.igorescobar.com + http://www.igorescobar.com + @igorescobar http://www.twitter.com/igorescobar On Mon, Sep 12, 2011 at 10:05 AM, Igor Escobar titiolin...@gmail.com wrote: Use PHP threads. Do the job separately.. in parts... in other words... you can't read all them at once. You can read a little more about php multithreading here: http://blog.motane.lu/2009/01/02/multithreading-in-php/ You can use a non-relational database like mongo or couchdb to manage where you stop and where you have to look back to the RSS feed as well. []'s Regards, Igor Escobar *Software Engineer * + http://blog.igorescobar.com + http://www.igorescobar.com + @igorescobar http://www.twitter.com/igorescobar On Sat, Sep 10, 2011 at 10:37 PM, Stuart Dallas stu...@3ft9.com wrote: On 10 Sep 2011, at 09:35, muad shibani wrote: I want to design an application that reads news from RSS sources. I have about 1000 RSS feed to collect from. I also will use Cron jobs every 15 minutes to collect the data. the question is: Is there a clever way to collect all those feed items without exhausting the server any Ideas I designed a job queuing system a while back when I had a similar problem. You can read about it here: http://stut.net/2009/05/29/php-job-queue/. Set that type of system up and add a job for each feed, set to run every 15 minutes. You can then watch the server and tune the number of concurrent job processors so you get the optimum balance between load and speed. -Stuart -- Stuart Dallas 3ft9 Ltd http://3ft9.com/ -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php Thread != Multi Process.
[PHP] PHP cron job optimization
I want to design an application that reads news from RSS sources. I have about 1000 RSS feed to collect from. I also will use Cron jobs every 15 minutes to collect the data. the question is: Is there a clever way to collect all those feed items without exhausting the server any Ideas Thank you in advance
Re: [PHP] PHP cron job optimization
On Sat, Sep 10, 2011 at 4:35 AM, muad shibani muad.shib...@gmail.comwrote: I want to design an application that reads news from RSS sources. I have about 1000 RSS feed to collect from. I also will use Cron jobs every 15 minutes to collect the data. the question is: Is there a clever way to collect all those feed items without exhausting the server any Ideas Thank you in advance Do them one at a time. Fetching web pages isn't a particularly taxing job for any decent server. My advice is just to try it. Even if you solution isn't 'elegant' or 'clever,' if it works and doesn't bog down the server, that's a win in my book. -- --Zootboy Sent from my PC.
Re: [PHP] PHP cron job optimization
I want to design an application that reads news from RSS sources. I have about 1000 RSS feed to collect from. I also will use Cron jobs every 15 minutes to collect the data. the question is: Is there a clever way to collect all those feed items without exhausting the server any Ideas Thank you in advance Just watch your memory and you'll be fine. As was stated fetching an rss feed is fast and cheap, so I would contrariwise think about paralleling if I would you. signature.asc Description: OpenPGP digital signature
Re: [PHP] PHP cron job optimization
thanks a lot after I test it I will share the code .. Thanks again On Sat, Sep 10, 2011 at 11:20 AM, Gary Golden m...@garygolden.me wrote: I want to design an application that reads news from RSS sources. I have about 1000 RSS feed to collect from. I also will use Cron jobs every 15 minutes to collect the data. the question is: Is there a clever way to collect all those feed items without exhausting the server any Ideas Thank you in advance Just watch your memory and you'll be fine. As was stated fetching an rss feed is fast and cheap, so I would contrariwise think about paralleling if I would you. -- * ___ Muad Shibani* * * Aden Yemen Mobile: 00967 733045678 Mobile: 00967 711811232 www.muadshibani.com
Re: [PHP] PHP cron job optimization
On Sat, Sep 10, 2011 at 1:47 PM, Sean Greenslade zootboys...@gmail.com wrote: On Sat, Sep 10, 2011 at 4:35 AM, muad shibani muad.shib...@gmail.comwrote: I want to design an application that reads news from RSS sources. I have about 1000 RSS feed to collect from. I also will use Cron jobs every 15 minutes to collect the data. the question is: Is there a clever way to collect all those feed items without exhausting the server any Ideas Thank you in advance Do them one at a time. Fetching web pages isn't a particularly taxing job for any decent server. My advice is just to try it. Even if you solution isn't 'elegant' or 'clever,' if it works and doesn't bog down the server, that's a win in my book. -- --Zootboy Sent from my PC. Traversing the internet is one of the most expensive IO operations that can be performed. The OP should try and split the work into several processes and utilize async connections via a mechanism like curl_multi [1]. Doing 1,000 connections one at a time is abysmal if you're blocked for even 3 seconds per connection. I'd work by making a two step process that has one cron job for constantly fetching/updating feeds into a store then another process that constantly checks for updates inside aforementioned store. [1] http://us3.php.net/manual/en/function.curl-multi-add-handle.php -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] PHP cron job optimization
On 10 Sep 2011, at 09:35, muad shibani wrote: I want to design an application that reads news from RSS sources. I have about 1000 RSS feed to collect from. I also will use Cron jobs every 15 minutes to collect the data. the question is: Is there a clever way to collect all those feed items without exhausting the server any Ideas I designed a job queuing system a while back when I had a similar problem. You can read about it here: http://stut.net/2009/05/29/php-job-queue/. Set that type of system up and add a job for each feed, set to run every 15 minutes. You can then watch the server and tune the number of concurrent job processors so you get the optimum balance between load and speed. -Stuart -- Stuart Dallas 3ft9 Ltd http://3ft9.com/ -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
