The IESG has received a request from an individual submitter to consider the following document: - 'Robots Exclusion Protocol' <draft-koster-rep-08.txt> as Proposed Standard
The IESG plans to make a decision in the next few weeks, and solicits final comments on this action. Please send substantive comments to the last-c...@ietf.org mailing lists by 2022-06-02. Exceptionally, comments may be sent to i...@ietf.org instead. In either case, please retain the beginning of the Subject line to allow automated sorting. Abstract This document specifies and extends the "Robots Exclusion Protocol" method originally defined by Martijn Koster in 1996 for service owners to control how content served by their services may be accessed, if at all, by automatic clients known as crawlers. The file can be obtained via https://datatracker.ietf.org/doc/draft-koster-rep/ No IPR declarations have been submitted directly on this I-D. The document contains these normative downward references. See RFC 3967 for additional information: rfc1945: Hypertext Transfer Protocol -- HTTP/1.0 (Informational - Internet Engineering Task Force (IETF)) _______________________________________________ IETF-Announce mailing list IETF-Announce@ietf.org https://www.ietf.org/mailman/listinfo/ietf-announce
