Yea Hi Mambe, Thanks for the feedback. I have mentioned the details of my application in the above post. I have tried doing this crawling job using php-multi curl and I am getting results which are good enough but the problem I am facing is that it is taking hell lot of time to get the contents of the urls. I have done this without using any API or conversions.
So, in order to crawl in lesser time limits and also helps me to scale my application, I have chosen Nutch crawler. Thanks and regards,* *Ch. Arjun Kumar Reddy On Wed, Jan 26, 2011 at 9:19 PM, Churchill Nanje Mambe < mambena...@afrovisiongroup.com> wrote: > hello > you have to use the short url APIs and get the long URLs... its abit > complex as you have to determine the url if its short, then determine the > url shortening service used eg: tinyurl.com bit.ly or goo.gl and then you > use their respective api and send in the url and they will return the long > url... I used this before but it was a simple php based aggregator and not > nutch >
