[Pywikibot-commits] [Gerrit] Skip tests that use a site which is failing - change (pywikibot/core)
jenkins-bot has submitted this change and it was merged.
Change subject: Skip tests that use a site which is failing
..
Skip tests that use a site which is failing
Add 'hostname' to TestCase.sites dict, and skip class when
failed to receive a useful response for the hosts homepage.
Re-enable weblib tests on travis-ci, instead using the new
automatic skip functionality to avoid errors due to WebCite
serving invalid XML and having an undecodable homepage.
Mention hostnames in test method docstring where the hostname
is not easily guessed.
Bug: T58963
Change-Id: I8fdcdaa0fab3b680d35b81b20a12ff5b786f779d
---
M pywikibot/weblib.py
M tests/__init__.py
M tests/aspects.py
M tests/data_ingestion_tests.py
M tests/http_tests.py
M tests/interwiki_link_tests.py
M tests/weblib_tests.py
M tests/wikidataquery_tests.py
M tests/wikistats_tests.py
9 files changed, 202 insertions(+), 39 deletions(-)
Approvals:
XZise: Looks good to me, approved
jenkins-bot: Verified
diff --git a/pywikibot/weblib.py b/pywikibot/weblib.py
index f682fec..8f0622d 100644
--- a/pywikibot/weblib.py
+++ b/pywikibot/weblib.py
@@ -36,7 +36,7 @@
query['timestamp'] = timestamp
uri = uri + urlencode(query)
-jsontext = http.request(uri=uri, site=None)
+jsontext = http.fetch(uri).content
if closest in jsontext:
data = json.loads(jsontext)
return data['archived_snapshots']['closest']['url']
@@ -65,7 +65,7 @@
query['date'] = timestamp
uri = uri + urlencode(query)
-xmltext = http.request(uri=uri, site=None)
+xmltext = http.fetch(uri).content
if success in xmltext:
data = ET.fromstring(xmltext)
return data.find('.//webcite_url').text
diff --git a/tests/__init__.py b/tests/__init__.py
index 32eaa93..4edf26a 100644
--- a/tests/__init__.py
+++ b/tests/__init__.py
@@ -94,9 +94,6 @@
'ui',
]
-if os.environ.get('TRAVIS', 'false') == 'true':
-disabled_test_modules.append('weblib')
-
disabled_tests = {
'textlib': [
'test_interwiki_format', # example; very slow test
diff --git a/tests/aspects.py b/tests/aspects.py
index 6466018..ab9146c 100644
--- a/tests/aspects.py
+++ b/tests/aspects.py
@@ -37,9 +37,10 @@
import pywikibot
-from pywikibot import config, log, Site
+from pywikibot import config, log, ServerError, Site
from pywikibot.site import BaseSite
from pywikibot.family import WikimediaFamily
+from pywikibot.comms import http
from pywikibot.data.api import Request as _original_Request
import tests
@@ -382,6 +383,70 @@
super(CacheInfoMixin, self).tearDown()
+class CheckHostnameMixin(TestCaseBase):
+
+Check the hostname is online before running tests.
+
+_checked_hostnames = {}
+
+@classmethod
+def setUpClass(cls):
+
+Set up the test class.
+
+Prevent tests running if the host is down.
+
+super(CheckHostnameMixin, cls).setUpClass()
+
+if not hasattr(cls, 'sites'):
+return
+
+for key, data in cls.sites.items():
+if 'hostname' not in data:
+raise Exception('%s: hostname not defined for %s'
+% (cls.__name__, key))
+hostname = data['hostname']
+
+if hostname in cls._checked_hostnames:
+if isinstance(cls._checked_hostnames[hostname], Exception):
+raise unittest.SkipTest(
+'%s: hostname %s failed (cached): %s'
+% (cls.__name__, hostname,
+ cls._checked_hostnames[hostname]))
+elif cls._checked_hostnames[hostname] is False:
+raise unittest.SkipTest('%s: hostname %s failed (cached)'
+% (cls.__name__, hostname))
+else:
+continue
+
+e = None
+try:
+if '://' not in hostname:
+hostname = 'http://' + hostname
+r = http.fetch(uri=hostname,
+ default_error_handling=False)
+if r.exception:
+e = r.exception
+else:
+if r.status not in [200, 301, 302, 303, 307, 308]:
+raise ServerError('HTTP status: %d' % r.status)
+r.content # default decode may raise exception
+except Exception as e2:
+pywikibot.error('%s: accessing %s caused exception:'
+% (cls.__name__, hostname))
+pywikibot.exception(e2, tb=True)
+e = e2
+pass
+
+if e:
+cls._checked_hostnames[hostname] = e
+raise unittest.SkipTest(
+'%s: hostname %s failed: %s'
+% (cls.__name__, hostname, e))
+
+
[Pywikibot-commits] Passed: wikimedia/pywikibot-core#1797 (master - 5cd4dd7)
Build Update for wikimedia/pywikibot-core - Build: #1797 Status: Passed Duration: 43 minutes and 53 seconds Commit: 5cd4dd7 (master) Author: jenkins-bot Message: Merge Skip tests that use a site which is failing View the changeset: https://github.com/wikimedia/pywikibot-core/compare/275feb5d60b6...5cd4dd7755a9 View the full build log and details: https://travis-ci.org/wikimedia/pywikibot-core/builds/43995131 -- You can configure recipients for build notifications in your .travis.yml file. See http://docs.travis-ci.com/user/notifications ___ Pywikibot-commits mailing list Pywikibot-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-commits
[Pywikibot-commits] Passed: wikimedia/pywikibot-core#1798 (master - 73dc493)
Build Update for wikimedia/pywikibot-core - Build: #1798 Status: Passed Duration: 46 minutes and 13 seconds Commit: 73dc493 (master) Author: jenkins-bot Message: Merge import pywikibot.tools.MediaWikiVersion as-is instead of as 'LV' View the changeset: https://github.com/wikimedia/pywikibot-core/compare/5cd4dd7755a9...73dc493a0a4f View the full build log and details: https://travis-ci.org/wikimedia/pywikibot-core/builds/44016205 -- You can configure recipients for build notifications in your .travis.yml file. See http://docs.travis-ci.com/user/notifications ___ Pywikibot-commits mailing list Pywikibot-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-commits
[Pywikibot-commits] [Gerrit] [IMPROV] Pagegens: Use -logevents instead of -*log - change (pywikibot/core)
jenkins-bot has submitted this change and it was merged.
Change subject: [IMPROV] Pagegens: Use -logevents instead of -*log
..
[IMPROV] Pagegens: Use -logevents instead of -*log
With a887aff93c0aa0859a93cac1e57f4738de59787e the LogeventsPageGenerator
was introduced which uses the -*log notation from compat (e.g. -movelog)
but it's not feasable to support this with the upcoming argparse patch,
because it doesn't allow such dynamic arguments.
This is a replacement so that when the argparse patch lands, that the
users hopefully haven't used the old notation. Instead of semicolons
it's using normal commas which are safer to use on the bash (just the
new notation). It also does only allow positive integers as a total
value (both notations).
Change-Id: If6c24333ea892386203dc9a4a289ef7f05eb
---
M pywikibot/pagegenerators.py
1 file changed, 59 insertions(+), 23 deletions(-)
Approvals:
John Vandenberg: Looks good to me, approved
jenkins-bot: Verified
diff --git a/pywikibot/pagegenerators.py b/pywikibot/pagegenerators.py
index e5c0e31..1e30a03 100644
--- a/pywikibot/pagegenerators.py
+++ b/pywikibot/pagegenerators.py
@@ -87,18 +87,22 @@
-search Work on all pages that are found in a MediaWiki search
across all namespaces.
--logeventlogWork on articles that were on a specified special:log.
+-logeventsWork on articles that were on a specified Special:Log.
+ The value may be a comma separated list of three values:
+ logevent,username,total
+ To use the default value, use an empty string.
You have options for every type of logs given by the
- logevent parameter which could be one of the following:
+ log event parameter which could be one of the following:
block, protect, rights, delete, upload, move, import,
patrol, merge, suppress, review, stable, gblblock,
renameuser, globalauth, gblrights, abusefilter, newusers
+ It uses the default number of pages 10.
Examples:
- -movelog gives 500 pages from move log (should be redirects)
- -deletelog:10 gives 10 pages from deletion log
- -protect:Dummy gives 500 pages from protect by user Dummy
- -patrol:Dummy;20 gives 20 pages patroled by user Dummy
- In some cases this must be written as -patrol:Dummy;20
+ -logevents:move gives pages from move log (should be
redirects)
+ -logevents:delete,,20 gives 20 pages from deletion log
+ -logevents:protect,Usr gives pages from protect by user Usr
+ -logevents:patrol,Usr,20 gives 20 patroled pages by user Usr
+ In some cases it must be written as
-logevents:patrol,Usr,20
-namespaces Filter the page generator to only yield pages in the
-namespacespecified namespaces. Separate multiple namespace
@@ -344,6 +348,35 @@
start=startfrom,
recurse=recurse,
content=content)
+
+def _parse_log_events(self, logtype, user=None, total=None):
+
+Parse the -logevent argument information.
+
+@param logtype: A valid logtype
+@type logtype: str
+@param user: A username associated to the log events. Ignored if
+empty string or None.
+@type user: str
+@param total: The total amount of pages returned.
+@type total: str (castable to int) or int (positive)
+@return: The generator or None if invalid 'total' value.
+@rtype: LogeventsPageGenerator
+
+# TODO: Check if logtype is one of the allowed log types
+# TODO: -*log used 500 as default total, also use with -logevents?
+if total is not None:
+try:
+total = int(total)
+if total = 0:
+raise ValueError
+except ValueError:
+pywikibot.error(u'Total number of log ({0}) events must be a '
+'positive int.'.format(total))
+return None
+# 'user or None', because user might be an empty string when
+# 'foo,,bar' was used.
+return LogeventsPageGenerator(logtype, user or None, total=total)
def handleArg(self, arg):
Parse one argument at a time.
@@ -609,29 +642,32 @@
elif arg.startswith('-intersect'):
self.intersect = True
return True
+elif arg.startswith('-logevents:'):
+gen = self._parse_log_events(*arg[len('-logevents:'):].split(','))
elif arg.startswith('-'):
mode, log, user = arg.partition('log')
# exclude -log, -nolog
if log
[Pywikibot-commits] Errored: wikimedia/pywikibot-core#1799 (master - 839910d)
Build Update for wikimedia/pywikibot-core - Build: #1799 Status: Errored Duration: 46 minutes and 4 seconds Commit: 839910d (master) Author: jenkins-bot Message: Merge [IMPROV] Pagegens: Use -logevents instead of -*log View the changeset: https://github.com/wikimedia/pywikibot-core/compare/73dc493a0a4f...839910d4d853 View the full build log and details: https://travis-ci.org/wikimedia/pywikibot-core/builds/44035686 -- You can configure recipients for build notifications in your .travis.yml file. See http://docs.travis-ci.com/user/notifications ___ Pywikibot-commits mailing list Pywikibot-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-commits
[Pywikibot-commits] [Gerrit] Fix KeyError: allpages for interwiki. - change (pywikibot/compat)
jenkins-bot has submitted this change and it was merged. Change subject: Fix KeyError: allpages for interwiki. .. Fix KeyError: allpages for interwiki. Do not raise exception for api warnings. Add rawcontinue to API query. Bug: T74667 Change-Id: I828d412cd50a6d5c15ef45610400c3a6f1c6a384 --- M wikipedia.py 1 file changed, 4 insertions(+), 3 deletions(-) Approvals: John Vandenberg: Looks good to me, approved Malafaya: Looks good to me, but someone else must approve jenkins-bot: Verified diff --git a/wikipedia.py b/wikipedia.py index 868688d..3f3496a 100644 --- a/wikipedia.py +++ b/wikipedia.py @@ -8561,7 +8561,8 @@ 'list': 'allpages', 'aplimit': config.special_page_limit, 'apnamespace': namespace, -'apfrom': start +'apfrom': start, +'rawcontinue': '' } if not includeredirects: @@ -8576,8 +8577,8 @@ if verbose: debug(u'allpages data.keys() %s' % data.keys()) if 'warnings' in data: -warning = data['warnings']['allpages']['*'] -raise RuntimeError(API query warning: %s % warning) +warning = data['warnings']['query']['*'] +pywikibot.warning(API query warning: %s % warning) if 'error' in data: raise RuntimeError(API query error: %s % data) if 'allpages' not in data['query']: -- To view, visit https://gerrit.wikimedia.org/r/176910 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I828d412cd50a6d5c15ef45610400c3a6f1c6a384 Gerrit-PatchSet: 4 Gerrit-Project: pywikibot/compat Gerrit-Branch: master Gerrit-Owner: Malafaya malaf...@clix.pt Gerrit-Reviewer: John Vandenberg jay...@gmail.com Gerrit-Reviewer: Ladsgroup ladsgr...@gmail.com Gerrit-Reviewer: Malafaya malaf...@clix.pt Gerrit-Reviewer: Merlijn van Deen valhall...@arctus.nl Gerrit-Reviewer: Mpaa mpaa.w...@gmail.com Gerrit-Reviewer: Ricordisamoa ricordisa...@openmailbox.org Gerrit-Reviewer: Xqt i...@gno.de Gerrit-Reviewer: jenkins-bot ___ Pywikibot-commits mailing list Pywikibot-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-commits
[Pywikibot-commits] [Gerrit] standardize commonscat.py - change (pywikibot/core)
jenkins-bot has submitted this change and it was merged.
Change subject: standardize commonscat.py
..
standardize commonscat.py
CommonscatBot extends pywikibot.Bot
and uses self.userPut() with built-in 'always'
Change-Id: I8d9019ab154a2bca1cb9fe706e832e6dcfde4120
---
M scripts/commonscat.py
1 file changed, 18 insertions(+), 27 deletions(-)
Approvals:
Xqt: Looks good to me, approved
jenkins-bot: Verified
diff --git a/scripts/commonscat.py b/scripts/commonscat.py
index 1a970e3..33a1e1d 100755
--- a/scripts/commonscat.py
+++ b/scripts/commonscat.py
@@ -243,11 +243,13 @@
Commons categorisation bot.
-def __init__(self, generator, always, summary=None):
+def __init__(self, generator, **kwargs):
Constructor.
-super(CommonscatBot, self).__init__(always=always)
+self.availableOptions.update({
+'summary': None,
+})
+super(CommonscatBot, self).__init__(**kwargs)
self.generator = generator
-self.summary = summary
self.site = pywikibot.Site()
def treat(self, page):
@@ -271,7 +273,7 @@
def getCommonscatTemplate(self, code=None):
Get the template name of a site. Expects the site code.
-Return as tuple containing the primary template and it's alternatives
+Return as tuple containing the primary template and its alternatives.
if code in commonscatTemplates:
@@ -349,8 +351,9 @@
else:
textToAdd = u'{{%s|%s}}' % (primaryCommonscat,
commonscatLink)
-always = self.getOption('always')
-rv = add_text(page, textToAdd, self.summary, always=always)
+rv = add_text(page, textToAdd,
+ self.getOption('summary'),
+ always=self.getOption('always'))
self.options['always'] = rv[2]
return True
return True
@@ -381,27 +384,16 @@
page.get())
else: # nothing left to do
return
-if self.summary:
-comment = self.summary
+if self.getOption('summary'):
+comment = self.getOption('summary')
else:
comment = i18n.translate(page.site,
msg_change,
fallback=True) % {'oldcat': oldcat,
'newcat': newcat}
-try:
-self.userPut(page, page.text, newtext, comment=comment)
-except pywikibot.LockedPage:
-pywikibot.output(uPage %s is locked; skipping.
- % page.title(asLink=True))
-except pywikibot.EditConflict:
-pywikibot.output(
-u'Skipping %s because of edit conflict'
-% (page.title()))
-except pywikibot.SpamfilterError as error:
-pywikibot.output(
-u'Cannot change %s because of spam blacklist entry %s'
-% (page.title(), error.url))
+self.userPut(page, page.text, newtext, comment=comment,
+ ignore_save_related_errors=True)
def findCommonscatLink(self, page=None):
Find CommonsCat template on interwiki pages.
@@ -529,10 +521,9 @@
@param args: command line arguments
@type args: list of unicode
-summary = None
+options = {}
generator = None
checkcurrent = False
-always = False
ns = []
ns.append(14)
@@ -543,13 +534,13 @@
for arg in local_args:
if arg.startswith('-summary'):
if len(arg) == 8:
-summary = pywikibot.input(u'What summary do you want to use?')
+options['summary'] = pywikibot.input(u'What summary do you
want to use?')
else:
-summary = arg[9:]
+options['summary'] = arg[9:]
elif arg.startswith('-checkcurrent'):
checkcurrent = True
elif arg == '-always':
-always = True
+options['always'] = True
else:
genFactory.handleArg(arg)
@@ -568,7 +559,7 @@
if generator:
pregenerator = pagegenerators.PreloadingGenerator(generator)
-bot = CommonscatBot(pregenerator, always, summary)
+bot = CommonscatBot(pregenerator, **options)
bot.run()
else:
pywikibot.showHelp()
--
To view, visit https://gerrit.wikimedia.org/r/154829
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I8d9019ab154a2bca1cb9fe706e832e6dcfde4120
Gerrit-PatchSet: 3
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Owner: Ricordisamoa ricordisa...@openmailbox.org
Gerrit-Reviewer: John Vandenberg jay...@gmail.com
Gerrit-Reviewer: Ladsgroup
[Pywikibot-commits] Failed: wikimedia/pywikibot-core#1801 (master - 3513a59)
Build Update for wikimedia/pywikibot-core - Build: #1801 Status: Failed Duration: 47 minutes and 19 seconds Commit: 3513a59 (master) Author: jenkins-bot Message: Merge standardize commonscat.py View the changeset: https://github.com/wikimedia/pywikibot-core/compare/463adf6b2175...3513a59687bd View the full build log and details: https://travis-ci.org/wikimedia/pywikibot-core/builds/44055431 -- You can configure recipients for build notifications in your .travis.yml file. See http://docs.travis-ci.com/user/notifications ___ Pywikibot-commits mailing list Pywikibot-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-commits
