On Monday, June 25, 2012 11:35:28 AM UTC+12, David Neilsen wrote: > > Looks pretty interesting. > > Does it just scrape the home page or does it go deeper? > > Also is there any was to see the scraped HTML? > > David Neilsen | 07 834 3366 | PANmedia ® > > > On Sat, Jun 23, 2012 at 12:48 AM, Nick <[email protected]> wrote: > >> Evening (or morning to be precise), >> >> Just wanted to share something that I've spent a little bit of time on, I >> got it up the other week and finally thought I'd let other people see it. >> >> http://www.crawl.co.nz/ >> >> Basically, I'm trying to get all the NZ domains, my first "crawl" i've >> come up with around a third, which isn't to bad. I've put up an index of >> the crawl here, unlike search engines you can search the html here, so you >> can see which sites running jquery, or some other piece of js. Can also do >> ands and ors, search ips and headers. >> >> Have a look :) Working on getting another 20-30% on the next crawl. >> >> Any questions, just ask. >> >> -- >> NZ PHP Users Group: http://groups.google.com/group/nzphpug >> To post, send email to [email protected] >> To unsubscribe, send email to >> [email protected] > > > On Monday, June 25, 2012 11:35:28 AM UTC+12, David Neilsen wrote: > > Looks pretty interesting. > > Does it just scrape the home page or does it go deeper? > > Also is there any was to see the scraped HTML? > > David Neilsen | 07 834 3366 | PANmedia ® > > > On Sat, Jun 23, 2012 at 12:48 AM, Nick <[email protected]> wrote: > >> Evening (or morning to be precise), >> >> Just wanted to share something that I've spent a little bit of time on, I >> got it up the other week and finally thought I'd let other people see it. >> >> http://www.crawl.co.nz/ >> >> Basically, I'm trying to get all the NZ domains, my first "crawl" i've >> come up with around a third, which isn't to bad. I've put up an index of >> the crawl here, unlike search engines you can search the html here, so you >> can see which sites running jquery, or some other piece of js. Can also do >> ands and ors, search ips and headers. >> >> Have a look :) Working on getting another 20-30% on the next crawl. >> >> Any questions, just ask. >> >> -- >> NZ PHP Users Group: http://groups.google.com/group/nzphpug >> To post, send email to [email protected] >> To unsubscribe, send email to >> [email protected] > > >
-- NZ PHP Users Group: http://groups.google.com/group/nzphpug To post, send email to [email protected] To unsubscribe, send email to [email protected]
