Thanks. I was after Common Crawl. Thanks for the pointer. Ridwan ________________________________ From: Gora Mohanty <[email protected]> Sent: Tuesday, October 29, 2024 6:02:08 AM To: [email protected] <[email protected]> Subject: Re: AWS Service that I can use to crawl the entire web
On Mon, 28 Oct 2024 at 16:06, Ridwan Naibi <[email protected]> wrote: > Hello everyone, > > Can I please know an AWS service that I can use to crawl the entire web? > It was suggested to me as an easy replacement nutch. It seems to be that > answer was given to me but I can't find it in the email thread. Thanks. If you really mean to crawl the entire web, it would make sense to start by looking at https://commoncrawl.org/ Regards, Gora
