A new Request for Comments is now available in online RFC libraries. RFC 9309
Title: Robots Exclusion Protocol Author: M. Koster, G. Illyes, H. Zeller, L. Sassman Status: Standards Track Stream: IETF Date: September 2022 Mailbox: m.kos...@greenhills.co.uk, garyill...@google.com, hen...@google.com, li...@google.com Pages: 12 Updates/Obsoletes/SeeAlso: None I-D Tag: draft-koster-rep-12.txt URL: https://www.rfc-editor.org/info/rfc9309 DOI: 10.17487/RFC9309 This document specifies and extends the "Robots Exclusion Protocol" method originally defined by Martijn Koster in 1994 for service owners to control how content served by their services may be accessed, if at all, by automatic clients known as crawlers. Specifically, it adds definition language for the protocol, instructions for handling errors, and instructions for caching. This is now a Proposed Standard. STANDARDS TRACK: This document specifies an Internet Standards Track protocol for the Internet community, and requests discussion and suggestions for improvements. Please refer to the current edition of the Official Internet Protocol Standards (https://www.rfc-editor.org/standards) for the standardization state and status of this protocol. Distribution of this memo is unlimited. This announcement is sent to the IETF-Announce and rfc-dist lists. To subscribe or unsubscribe, see https://www.ietf.org/mailman/listinfo/ietf-announce https://mailman.rfc-editor.org/mailman/listinfo/rfc-dist For searching the RFC series, see https://www.rfc-editor.org/search For downloading RFCs, see https://www.rfc-editor.org/retrieve/bulk Requests for special distribution should be addressed to either the author of the RFC in question, or to rfc-edi...@rfc-editor.org. Unless specifically noted otherwise on the RFC itself, all RFCs are for unlimited distribution. The RFC Editor Team Association Management Solutions, LLC _______________________________________________ IETF-Announce mailing list IETF-Announce@ietf.org https://www.ietf.org/mailman/listinfo/ietf-announce