What happens to those that ignore robot.txt files? On Thu, Sep 3, 2015 at 3:48 PM Tony Harminc <t...@harminc.net> wrote:
> On 3 September 2015 at 11:18, Andy Higgins <ahigg...@transunion.com> > wrote: > >> I don't remember what was there originally... > > > > The Internet Archive Wayback Machine does: > > > > > http://web.archive.org/web/*/http://www-03.ibm.com/systems/z/os/zos/library/bkserv/index.html > > That's surprising - most IBM pages have a robots.txt that doesn't want > anything looked at/crawled/remembered. > > Tony H. > > ---------------------------------------------------------------------- > For IBM-MAIN subscribe / signoff / archive access instructions, > send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN > ---------------------------------------------------------------------- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
