[issue21475] Support the Sitemap extension in robotparser

[Ned Deily]

Ned Deily  added the comment:

Thanks for the patch, Peter, and thanks for the PR and test, Lady Red!  Merged 
for release in 3.8.0.

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21475] Support the Sitemap extension in robotparser

[Ned Deily]

Ned Deily  added the comment:


New changeset 5db5c0669e624767375593cc1a01f32092c91c58 by Ned Deily 
(Christopher Beacham) in branch 'master':
bpo-21475: Support the Sitemap extension in robotparser (GH-6883)
https://github.com/python/cpython/commit/5db5c0669e624767375593cc1a01f32092c91c58


--
nosy: +ned.deily

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21475] Support the Sitemap extension in robotparser

[Lady Red]

Lady Red  added the comment:

Sorry, wrong PR number. it is 6883, and attached to this ticket

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21475] Support the Sitemap extension in robotparser

[Lady Red]

Lady Red  added the comment:

I wrote a test for this as it seems to have been abandoned, and opened a PR.  
https://github.com/python/cpython/pull/6878

--
nosy: +mcsc...@gmail.com

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21475] Support the Sitemap extension in robotparser

[Roundup Robot]

Change by Roundup Robot :


--
pull_requests: +6556

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21475] Support the Sitemap extension in robotparser

[Steven Steven]

Steven Steven  added the comment:

Kindly add a test for this issue

--
nosy: +stevensalbert

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21475] Support the Sitemap extension in robotparser

[Stéphane Wirtel]

Stéphane Wirtel  added the comment:

Hi @berker and @pwirtz.

could you write a test for this issue?

thanks

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21475] Support the Sitemap extension in robotparser

[Raymond Hettinger]

Change by Raymond Hettinger :


--
versions: +Python 3.8 -Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21475] Support the Sitemap extension in robotparser

[STINNER Victor]

Changes by STINNER Victor :


--
dependencies: +Rewrite test_robotparser

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21475] Support the Sitemap extension in robotparser

[Berker Peksag]

Berker Peksag added the comment:

Peter didn't write a test because issue 25497 (test_robotparser rewrite) needs 
to be committed first. See msg253016 in issue 25400 for more information.

--
stage: needs patch -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21475] Support the Sitemap extension in robotparser

[Stéphane Wirtel]

Stéphane Wirtel added the comment:

Add a test with your patch.

Thank you

--
nosy: +matrixise

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21475] Support the Sitemap extension in robotparser

[Peter Wirtz]

Peter Wirtz added the comment:

Here is a patch that provides support for the Sitemap extension.

--
keywords: +patch
Added file: http://bugs.python.org/file40791/robotparser_site_maps_v1.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21475] Support the Sitemap extension in robotparser

[Berker Peksag]

Berker Peksag added the comment:

issue 25400 is not a blocker of this, so feel free to write a patch.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21475] Support the Sitemap extension in robotparser

[Peter Wirtz]

Peter Wirtz added the comment:

I would like to tackle this issue. Should I wait for issue25400 to be resolved 
first?

--
nosy: +pwirtz

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21475] Support the Sitemap extension in robotparser

[Berker Peksag]

Berker Peksag added the comment:

The Crawl-delay part(issue 16099) is now committed.

--
stage:  -> needs patch
title: Support the Sitemap and Crawl-delay extensions in robotparser -> Support 
the Sitemap extension in robotparser
versions: +Python 3.6 -Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com