On Wed, Sep 25, 2013 at 7:58 PM, Stephen J. Turnbull <step...@xemacs.org>wrote:
> > It might work that way, but it shouldn't. From the Robot Exclusion > Standard: > > Disallow > > The value of this field specifies a partial URL that is not to be > visited. This can be a full path, or a partial path; any URL that > starts with this value will not be retrieved. For example, Disallow: > /help disallows both /help.html and /help/index.html[.] > > So just > > Disallow: /msg > > is presumably what you want. > Thanks for the hint. It is definitely works as confirmed by webmasters doc from one of the robot owners. https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt -- anatoly t.
_______________________________________________ Tracker-discuss mailing list Tracker-discuss@python.org https://mail.python.org/mailman/listinfo/tracker-discuss
