Hi, We at the National Library will soon begin our 2022 New Zealand domain crawl. As in previous years, the crawl is being run by Internet Archive, and some of you may soon see the crawlers as IA go through their initial setup with some test crawls.
The user agent string their crawlers will be using is: NLNZ_IAHarvester2022 For more information on our domain crawls please see http://natlib.govt.nz/publishers-and-authors/web-harvesting/domain-harvest While we do our best to crawl respectfully, obey robots.txt, and break out of any crawler traps before grinding too much bandwidth, if you encounter any issues from us then please get in contact. Cheers, Ben O’Brien | Digital Preservation Web Engineer | Preservation, Research and Consultancy National Library of New Zealand | Te Puna Mātauranga o Aotearoa PO Box 1467, Wellington 6140, New Zealand | www.natlib.govt.nz<http://www.natlib.govt.nz/> [Logo-test]
_______________________________________________ NZNOG mailing list -- nznog@list.waikato.ac.nz To unsubscribe send an email to nznog-le...@list.waikato.ac.nz
