[Pywikipedia-bugs] [Maniphest] [Created] T144023: Maxlag lag incorrectly causing bot to crash with pywikibot.data.api.APIError
Multichill created this task.Multichill added projects: Pywikibot-core, MediaWiki-API.Herald added subscribers: pywikibot-bugs-list, Aklapper.
TASK DESCRIPTIONPywikibot checks if maxlag is returned ( https://phabricator.wikimedia.org/diffusion/PWBC/browse/master/pywikibot/data/api.py;31c0234c3e0dd5f0740aa7d69d94651fbcb8bea8$2075 )
lagpattern = re.compile(r"Waiting for [\d.]+: (?P\d+) seconds? lagged")
if code == "maxlag":
lag = lagpattern.search(info)
if lag:
pywikibot.log(
u"Pausing due to database lag: " + info)
self.site.throttle.lag(int(lag.group("lag")))
continue
https://phabricator.wikimedia.org/diffusion/MW/browse/master/includes/api/ApiMain.php;cfa04ceb2b8878ab237d8dd2bdbd578972bce80e$1122
Parsing a string in the api output is just horrible, but seems to be the only way to retrieve the maxlag output. The output was an integer in seconds, but now a float so the regex doesn't match anymore. Simple fix is to update the regex to r"Waiting for [\d.]+: (?P\d+(\.\d+)?) seconds? lagged"
Bigger question is if we want to know the maximum lag and if we want to parse strings. I see several options here:
Just patch it up (short term solution, should do this whatever longer term we choose)
Stop using the lag info for backing up and instead just use the standard incremental backing off
Update the mediawiki api to include the lag in the output in a structured format so we don't have to do evil string parsing
TASK DETAILhttps://phabricator.wikimedia.org/T144023EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MultichillCc: Xqt, jayvdb, valhallasw, Aklapper, pywikibot-bugs-list, Multichill, Sethakill, Mdupont, JJMC89, dg711, Anomie, Legoktm___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Commented On] T142155: Bot can't login. keyError in GetCookie
Multichill added a comment. In T142155#2529638, @Tgr wrote: Reverted in 1.28.0-wmf.13 (meaning it will last until Tuesday). Is anyone planning to fix PWB and contact users about updating (or using OAuth)? Thank you Gergő. I can confirm too that login works again. Tuesday is way too early. Can you push it back at least 2 months?TASK DETAILhttps://phabricator.wikimedia.org/T142155EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MultichillCc: zhuyifei1999, Stashbot, gerritbot, DrTrigon, Betacommand, Lokal_Profil, Tgr, Udo_T, Legoktm, JJMC89, Pywikibot-Announce, Yamaha5, jayvdb, Ladsgroup, Xqt, valhallasw, Multichill, Anomie, CodeCat, TerraCodes, Luke081515, He7d3r, Basilicofresco, Jagwar, MarcoAurelio, Dalba, Aklapper, Joutbis, pywikibot-bugs-list, Mdupont___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Commented On] T142155: Bot can't login. keyError in GetCookie
Multichill added a comment. In T142155#2529359, @MarcoAurelio wrote: Not a tech so pardon my ignorance, but I feel it'd be easier to solve this from the pywikibot side rather than on the mediawiki side? In other words, this is not that simple than to submit a revert-patch for @Anomie's change and have it merged on gerrit, it needs to be backported or something, right? Having everyone update every once in a while is not problem, forcing every bot operator out there to update *now* to the the latest version is not cool.TASK DETAILhttps://phabricator.wikimedia.org/T142155EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MultichillCc: Legoktm, JJMC89, Pywikibot-Announce, Yamaha5, jayvdb, Ladsgroup, Xqt, valhallasw, Multichill, Anomie, CodeCat, TerraCodes, Luke081515, He7d3r, Basilicofresco, Jagwar, MarcoAurelio, Dalba, Aklapper, Joutbis, pywikibot-bugs-list, Mdupont___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Commented On] T142155: Bot can't login. keyError in GetCookie
Multichill added a comment. Emailed the announce list, see https://lists.wikimedia.org/pipermail/pywikibot-announce/2016-August/10.htmlTASK DETAILhttps://phabricator.wikimedia.org/T142155EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MultichillCc: Pywikibot-Announce, Yamaha5, jayvdb, Ladsgroup, Xqt, valhallasw, Multichill, Anomie, CodeCat, TerraCodes, Luke081515, He7d3r, Basilicofresco, Jagwar, MarcoAurelio, Dalba, Aklapper, Joutbis, pywikibot-bugs-list, Mdupont___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Changed Subscribers] T142155: Bot can't login. keyError in GetCookie
Multichill added subscribers: Anomie, Multichill.Multichill added a comment. @Anomie could you undo your change? It broke all bots.TASK DETAILhttps://phabricator.wikimedia.org/T142155EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MultichillCc: Multichill, Anomie, CodeCat, TerraCodes, Luke081515, He7d3r, Basilicofresco, Jagwar, MarcoAurelio, Dalba, Aklapper, Joutbis, pywikibot-bugs-list, Mdupont, jayvdb___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Commented On] T112140: Provide a wrapper function in pywikibot around wbparsevalue
Multichill added a comment. Item: *https://www.wikidata.org/w/api.php?action=""> Property: https://www.wikidata.org/w/api.php?action=""> FilePage: *https://www.wikidata.org/w/api.php?action=""> Checks if it exists so https://www.wikidata.org/w/api.php?action=""> gives an error Coordinate: https://www.wikidata.org/w/api.php?action=""> https://phabricator.wikimedia.org/diffusion/PWBC/browse/master/pywikibot/__init__.py;b6e933d4e33e1035457755015b29da74b8c8160a$304 Etc, all the output we get seem and should match the constructors we have for the objects. This only makes sense for data types that are subclasses of WbRepresentation https://phabricator.wikimedia.org/diffusion/PWBC/browse/master/pywikibot/_wbtypes.py%3Bb6e933d4e33e1035457755015b29da74b8c8160a . These classes all have a fromWikibase function. So the logic should probably be: DataSite.parsevalue(datatype, value, options, validate) Check that datatype is the name of a subclass of " WbRepresentation" Throw it to the server If the result is ok, return .fromWikibase() Something like that? This way, if a new datatype gets added, this part of the code doesn't need to change.TASK DETAILhttps://phabricator.wikimedia.org/T112140EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: ArthurPSmith, MultichillCc: Tobias1984, ArthurPSmith, Aklapper, pywikibot-bugs-list, Multichill, Mdupont, D3r1ck01, Izno, Wikidata-bugs, aude, jayvdb, Ricordisamoa, Mbch331___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Commented On] T112140: Provide a wrapper function in pywikibot around wbparsevalue
Multichill added a comment. I don't get it. https://www.wikidata.org/w/api.php?action=""> output seems suitable to build a WbTime object. with https://phabricator.wikimedia.org/diffusion/PWBC/browse/master/pywikibot/__init__.py;b6e933d4e33e1035457755015b29da74b8c8160a$512 . What's wrong with that?TASK DETAILhttps://phabricator.wikimedia.org/T112140EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: ArthurPSmith, MultichillCc: Tobias1984, ArthurPSmith, Aklapper, pywikibot-bugs-list, Multichill, Mdupont, D3r1ck01, Izno, Wikidata-bugs, aude, jayvdb, Ricordisamoa, Mbch331___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Commented On] T112140: Provide a wrapper function in pywikibot around wbparsevalue
Multichill added a comment. In T112140#2429651, @ArthurPSmith wrote: I'm going to have a shot at implementing this - it looks like it will be useful for a number of other open phabricator issues for pywikibot. I was figuring a function that will take all the parameters the API offers (datatype - a string, values - a list of strings, options - a dict, validate - boolean). Any other recommendations? The function should return an object. Possibilities seem to be commonsMedia, globe-coordinate, monolingualtext, quantity, string, time, url, external-id, wikibase-item, wikibase-property, mathTASK DETAILhttps://phabricator.wikimedia.org/T112140EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: ArthurPSmith, MultichillCc: Tobias1984, ArthurPSmith, Aklapper, pywikibot-bugs-list, Multichill, Mdupont, D3r1ck01, Izno, Wikidata-bugs, aude, jayvdb, Ricordisamoa, Mbch331___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Commented On] T133445: All pywikibot logging seems to be dumped in pwb-bot.log instead of different files
Multichill added a comment. In https://phabricator.wikimedia.org/T133445#2232161, @valhallasw wrote: > You need to call pywikibot.handle_args() to set up logging: Looks like a good work around, but not really a fix. pwb.py should just initialize this. TASK DETAIL https://phabricator.wikimedia.org/T133445 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Multichill Cc: valhallasw, Aklapper, pywikibot-bugs-list, Multichill, Mdupont, jayvdb ___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Commented On] T133445: All pywikibot logging seems to be dumped in pwb-bot.log instead of different files
Multichill added a comment. Did some more digging. Might be around a bit longer and I just started using the pwb.py wrapper more often. I do have a login-bot.log from Apr 22 20:13 when I ran pwb.py login.py, The "/usr/bin/python /data/project/multichill/pywikibot/pwb.py /data/project/multichill/code/bot/wikidata/getty_import.py" seem all to end up in the pwb-bot.log so maybe that doesn't get parsed properly? TASK DETAIL https://phabricator.wikimedia.org/T133445 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Multichill Cc: Aklapper, pywikibot-bugs-list, Multichill, Mdupont, jayvdb ___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Edited] T133445: All pywikibot logging seems to be dumped in pwb-bot.log instead of different files
Multichill edited the task description. TASK DETAIL https://phabricator.wikimedia.org/T133445 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Multichill Cc: Aklapper, pywikibot-bugs-list, Multichill, Mdupont, jayvdb ___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Created] T133445: All pywikibot logging seems to be dumped in pwb-bot.log instead of different files
Multichill created this task. Herald added subscribers: pywikibot-bugs-list, Aklapper. TASK DESCRIPTION For some reason all Pywikibot logging ends up in pwb-bot.log now instead of -bot.log. This is very annoying when you run multiple bots and you want to debug one of them. Seems to be introduced quite recently because I still have tons of logs from March and I'm pretty sure I did git pulls several times in March. My logging configuration in user-config.py: log = ['*'] I run Pywikibot: [https] r-pywikibot-core.git (https://phabricator.wikimedia.org/rPWBC75134ba52d9a380af6ffcddd80ef0a07d1ea511a, g7223, 2016/04/21, 22:05:31, ok) TASK DETAIL https://phabricator.wikimedia.org/T133445 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Multichill Cc: Aklapper, pywikibot-bugs-list, Multichill, Mdupont, jayvdb ___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Updated] T133444: Occasional Unexpected offset warnings with one byte off in Pywikibot
Multichill added a blocked task: T61914: Chunked upload (tracking). TASK DETAIL https://phabricator.wikimedia.org/T133444 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Multichill Cc: Aklapper, Steinsplitter, Avicennasis, pywikibot-bugs-list, Multichill, Mdupont, Bawolff, jayvdb, Fabrice_Florin, Tgr ___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Edited] T133443: pywikibot WARNING Offset was not supplied. with chunked upload
Multichill edited the task description. TASK DETAIL https://phabricator.wikimedia.org/T133443 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Multichill Cc: Aklapper, Avicennasis, pywikibot-bugs-list, Multichill, Mdupont, jayvdb ___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Created] T133444: Occasional Unexpected offset warnings with one byte off in Pywikibot
Multichill created this task.
Herald added subscribers: pywikibot-bugs-list, Avicennasis, Steinsplitter,
Aklapper.
Herald added a project: Multimedia.
TASK DESCRIPTION
I seem to get occasional warnings that the offset is wrong with chunked
uploading:
$ less pwb-bot.log | grep offset
2016-04-23 09:15:30site.py, 5896 in upload: VERBOSE
Old offset: 4700; Returned offset: 4799; Chunk size: 100
2016-04-23 09:15:30site.py, 5897 in upload: WARNING
Unexpected offset.
2016-04-23 09:28:52site.py, 5896 in upload: VERBOSE
Old offset: 1800; Returned offset: 1899; Chunk size: 100
2016-04-23 09:28:52site.py, 5897 in upload: WARNING
Unexpected offset.
2016-04-23 10:08:41site.py, 5896 in upload: VERBOSE
Old offset: 500; Returned offset: 599; Chunk size: 100
2016-04-23 10:08:41site.py, 5897 in upload: WARNING
Unexpected offset.
2016-04-23 10:16:18site.py, 5896 in upload: VERBOSE
Old offset: 3000; Returned offset: 3099; Chunk size: 100
2016-04-23 10:16:18site.py, 5897 in upload: WARNING
Unexpected offset.
2016-04-23 10:24:31site.py, 5896 in upload: VERBOSE
Old offset: 3700; Returned offset: 3799; Chunk size: 100
2016-04-23 10:24:31site.py, 5897 in upload: WARNING
Unexpected offset.
2016-04-23 10:45:01site.py, 5896 in upload: VERBOSE
Old offset: 2400; Returned offset: 2499; Chunk size: 100
2016-04-23 10:45:01site.py, 5897 in upload: WARNING
Unexpected offset.
It's always expected-1
Example upload:
2016-04-23 10:15:47getty_import.py, 297 inrun: INFO
File:Jacob van Hulsdonck - Still Life with Lemons, Oranges and a Pomegran
ate - 86.PB.538 - J. Paul Getty Museum.jpg
2016-04-23 10:15:47getty_import.py, 299 inrun: INFO
{{User:Multichill/J. Paul Getty Museum
| creator=Jacob van Hulsdonck
|
| date=about 1620 - 1640
|
| id=86.PB.538
|
| imageurl=http://www.getty.edu/museum/media/images/web/download/00091101.jpg
|
| location=Q29247
|
| medium=Oil on panel
|
| title=Still Life with Lemons, Oranges and a Pomegranate
|
|
url=http://www.getty.edu/art/collection/objects/791/jacob-van-hulsdonck-still-life-with-lemons-oranges-and-a-pomegranate-flemish-about-1620-1640/
|
| wikidata=Q20180102
|
}}
2016-04-23 10:15:48 api.py, 1925 in submit: VERBOSE
Submitting unthrottled action 'upload'.
.
2016-04-23 10:16:15 api.py, 1925 in submit: VERBOSE
Submitting unthrottled action 'upload'.
2016-04-23 10:16:16 api.py, 1925 in submit: VERBOSE
Submitting unthrottled action 'upload'.
2016-04-23 10:16:17 api.py, 1925 in submit: VERBOSE
Submitting unthrottled action 'upload'.
2016-04-23 10:16:18site.py, 5896 in upload: VERBOSE
Old offset: 3000; Returned offset: 3099; Chunk size: 100
2016-04-23 10:16:18site.py, 5897 in upload: WARNING
Unexpected offset.
2016-04-23 10:16:18 api.py, 1925 in submit: VERBOSE
Submitting unthrottled action 'upload'.
2016-04-23 10:16:19 api.py, 1925 in submit: VERBOSE
Submitting unthrottled action 'upload'.
2016-04-23 10:16:20 api.py, 1925 in submit: VERBOSE
Submitting unthrottled action 'upload'.
..
2016-04-23 10:16:34 api.py, 1925 in submit: VERBOSE
Submitting unthrottled action 'upload'.
2016-04-23 10:16:44site.py, 5900 in upload: WARNING
Offset was not supplied. <- this is a different bug, see
https://phabricator.wikimedia.org/T133443
2016-04-23 10:16:44site
[Pywikipedia-bugs] [Maniphest] [Created] T133443: pywikibot WARNING Offset was not supplied. with chunked upload
Multichill created this task. Herald added subscribers: pywikibot-bugs-list, Avicennasis, Aklapper. TASK DESCRIPTION I'm playing around with the new upload implementation in site and chunked uploading. Chunked uploading seems to work ok, but on every upload I get a warning: 2016-04-23 08:53:29 api.py, 1925 in submit: VERBOSE Submitting unthrottled action 'upload'. (repeated for every chunk) 2016-04-23 08:54:31 api.py, 1925 in submit: VERBOSE Submitting unthrottled action 'upload'. 2016-04-23 08:54:40site.py, 5900 in upload: WARNING Offset was not supplied. 2016-04-23 08:54:40site.py, 5903 in upload: VERBOSE Finished uploading last chunk. 2016-04-23 08:54:49site.py, 5978 in upload: INFO Upload successful. My assumption is that the last chunk isn't handled properly triggering this warning. TASK DETAIL https://phabricator.wikimedia.org/T133443 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Multichill Cc: Aklapper, Avicennasis, pywikibot-bugs-list, Multichill, Mdupont, jayvdb ___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Updated] T133443: pywikibot WARNING Offset was not supplied. with chunked upload
Multichill added a blocked task: T61914: Chunked upload (tracking). TASK DETAIL https://phabricator.wikimedia.org/T133443 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Multichill Cc: Aklapper, Avicennasis, pywikibot-bugs-list, Multichill, Mdupont, jayvdb ___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Commented On] T131592: Add retry logic to sparql data layer in Pywikibot
Multichill added a comment. Talked with @Smalyshev about this yesterday. api.php ( https://git.wikimedia.org/blob/pywikibot%2Fcore.git/master/pywikibot%2Fdata%2Fapi.py ) is build in a way that it recovers on quite a few errors. Logic can be copied from there. Most network errors (no internet connection, etc.) can probably be recovered. Application errors are probably a bit harder to decide if it can be recovered or not. The trigger for me filing this task was WDQ constantly failing. Not really worth more time in that, but mysql should probably also recover. TASK DETAIL https://phabricator.wikimedia.org/T131592 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Multichill Cc: jayvdb, Lokal_Profil, Aklapper, Smalyshev, pywikibot-bugs-list, Multichill, Ricordisamoa ___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Closed] T129021: ReferringPageGenerator only returns 500 pages instead of all
Multichill closed this task as "Resolved". TASK DETAIL https://phabricator.wikimedia.org/T129021 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Multichill Cc: Xqt, Aklapper, pywikibot-bugs-list, Multichill ___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Updated] T129021: ReferringPageGenerator only returns 500 pages instead of all
Multichill added a comment. Looks like the local copy got stuck in a really old version: $ git pull Updating https://phabricator.wikimedia.org/rPWBC783b33c61495d6d80e17f841620ac0581d41e567..https://phabricator.wikimedia.org/rPWBCd82b9c3a62cad81fb1e44d1c42ca68ebb3c32dbc fatal: Not a git repository: /data/project/railways/core/.git/modules/externals/httplib2 tools.railways@tools-bastion-05:~/pywikibot-broken$ git log | head commit https://phabricator.wikimedia.org/rPWBC783b33c61495d6d80e17f841620ac0581d41e567 Author: notconfusing <isa...@gmail.com> Date: Thu Oct 24 16:07:23 2013 -0700 Pulled a fresh copy: tools.railways@tools-bastion-05:~/pywikibot$ git log | head commit https://phabricator.wikimedia.org/rPWBCd82b9c3a62cad81fb1e44d1c42ca68ebb3c32dbc Merge: https://phabricator.wikimedia.org/rPWBCe65b6e7210c13a9eb236adf4ebed33647e98cc4c https://phabricator.wikimedia.org/rPWBC45ce85e2593d5e2a5c72af2f3913549e0da437f5 Author: jenkins-bot <jenkins-...@gerrit.wikimedia.org> Date: Sun Mar 6 09:13:40 2016 + Now it works and seems to be much faster too :-) TASK DETAIL https://phabricator.wikimedia.org/T129021 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Multichill Cc: Xqt, Aklapper, pywikibot-bugs-list, Multichill ___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Triaged] T129021: ReferringPageGenerator only returns 500 pages instead of all
Multichill triaged this task as "High" priority. TASK DETAIL https://phabricator.wikimedia.org/T129021 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Multichill Cc: Aklapper, pywikibot-bugs-list, Multichill ___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Created] T129021: ReferringPageGenerator only returns 500 pages instead of all
Multichill created this task.
Herald added subscribers: pywikibot-bugs-list, Aklapper.
TASK DESCRIPTION
I dusted off an old bot that used to work. It uses the following code to get
all items that use a certain property:
repo = pywikibot.Site().data_repository()
ppage = pywikibot.PropertyPage(repo, u'Property:P197')
gen =
pagegenerators.NamespaceFilterPageGenerator(pagegenerators.ReferringPageGenerator(ppage,
withTemplateInclusion=False, onlyTemplateInclusion=False), namespaces=[0])
This used to return all usage, but now seems to get stuck at about 500 items.
The last item I get is Q455649 and that so happens to be the last item on
https://www.wikidata.org/w/index.php?title=Special:WhatLinksHere/Property:P197=500
. So it looks like not all items are retrieved.
pagegenerators.ReferringPageGenerator is supposed to return all usage:
def ReferringPageGenerator(referredPage, followRedirects=False,
withTemplateInclusion=True,
onlyTemplateInclusion=False,
total=None, content=False):
"""Yield all pages referring to a specific page."""
return referredPage.getReferences(
follow_redirects=followRedirects,
withTemplateInclusion=withTemplateInclusion,
onlyTemplateInclusion=onlyTemplateInclusion,
total=total, content=content)
Total is not set so it should pass total=None to Page.getReferences()
def getReferences(self, follow_redirects=True, withTemplateInclusion=True,
onlyTemplateInclusion=False, redirectsOnly=False,
namespaces=None, total=None, content=False):
"""
Return an iterator all pages that refer to or embed the page.
If you need a full list of referring pages, use
C{pages = list(s.getReferences())}
@param follow_redirects: if True, also iterate pages that link to a
redirect pointing to the page.
@param withTemplateInclusion: if True, also iterate pages where self
is used as a template.
@param onlyTemplateInclusion: if True, only iterate pages where self
is used as a template.
@param redirectsOnly: if True, only iterate redirects to self.
@param namespaces: only iterate pages in these namespaces
@param total: iterate no more than this number of pages in total
@param content: if True, retrieve the content of the current version
of each referring page (default False)
"""
# N.B.: this method intentionally overlaps with backlinks() and
# embeddedin(). Depending on the interface, it may be more efficient
# to implement those methods in the site interface and then combine
# the results for this method, or to implement this method and then
# split up the results for the others.
return self.site.pagereferences(
self,
followRedirects=follow_redirects,
filterRedirects=redirectsOnly,
withTemplateInclusion=withTemplateInclusion,
onlyTemplateInclusion=onlyTemplateInclusion,
namespaces=namespaces,
total=total,
content=content
)
This passes on the work to Site.pagereferences() again with total=None and
withTemplateInclusion=False, onlyTemplateInclusion=False
def pagereferences(self, page, followRedirects=False, filterRedirects=None,
withTemplateInclusion=True,
onlyTemplateInclusion=False,
namespaces=None, total=None, content=False):
"""
Convenience method combining pagebacklinks and page_embeddedin.
@param namespaces: If present, only return links from the namespaces
in this list.
@type namespaces: iterable of basestring or Namespace key,
or a single instance of those types. May be a '|' separated
list of namespace identifiers.
@raises KeyError: a namespace identifier was not resolved
@raises TypeError: a namespace identifier has an inappropriate
type such as NoneType or bool
"""
if onlyTemplateInclusion:
return self.page_embeddedin(page, namespaces=namespaces,
filterRedirects=filterRedirects,
total=total, content=content)
if not withTemplateInclusion:
return self.pagebacklinks(page, followRedirects=followRedirects,
filterRedirects=filterRedirects,
namespaces=namespaces,
total=total, content=content)
[Pywikipedia-bugs] [Maniphest] [Commented On] T128946: Move out opencv from pywikibot-compat
Multichill added a comment. Why are we wasting time on compat? TASK DETAIL https://phabricator.wikimedia.org/T128946 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Multichill Cc: Multichill, DaBPunkt, pywikibot-bugs-list, jayvdb, DrTrigon, Aklapper, AbdealiJK, Anshoe ___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Updated] T128202: Implement badges support in pywikibot
Multichill added a blocked task: T42810: Wikibase badges (tracking). TASK DETAIL https://phabricator.wikimedia.org/T128202 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Multichill Cc: Aklapper, StudiesWorld, pywikibot-bugs-list, Multichill, D3r1ck01, Izno, Wikidata-bugs, aude, Ricordisamoa, Mbch331 ___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Updated] T128202: Implement badges support in pywikibot
Multichill added a blocking task: T66457: refactor sitelinks structure to support badges. TASK DETAIL https://phabricator.wikimedia.org/T128202 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Multichill Cc: Aklapper, StudiesWorld, pywikibot-bugs-list, Multichill, D3r1ck01, Izno, Wikidata-bugs, aude, Ricordisamoa, Mbch331 ___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Updated] T66457: refactor sitelinks structure to support badges
Multichill added a blocked task: T128202: Implement badges support in pywikibot. TASK DETAIL https://phabricator.wikimedia.org/T66457 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Multichill Cc: gerritbot, Aklapper, Bene, jayvdb, Ricordisamoa, pywikibot-bugs-list, Liuxinyu970226 ___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Created] T128202: Implement badges support in pywikibot
Multichill created this task. Herald added subscribers: pywikibot-bugs-list, StudiesWorld, Aklapper. TASK DESCRIPTION Wikibase badges have been around for quite some time. Should be implemented in Pywikibot (expand a bit) TASK DETAIL https://phabricator.wikimedia.org/T128202 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Multichill Cc: Aklapper, StudiesWorld, pywikibot-bugs-list, Multichill, D3r1ck01, Izno, Wikidata-bugs, aude, Ricordisamoa, Mbch331 ___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Updated] T66457: refactor sitelinks structure to support badges
Multichill removed a blocking task: T42810: Wikibase badges (tracking). TASK DETAIL https://phabricator.wikimedia.org/T66457 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Multichill Cc: gerritbot, Aklapper, Bene, jayvdb, Ricordisamoa, pywikibot-bugs-list, Liuxinyu970226 ___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Updated] T100123: API code 'readonly' not handled
Multichill added a comment. On Wikidata to database slips into read-only mode several times a day (see https://phabricator.wikimedia.org/T123867) and every time bots crash. Fairly annoying. TASK DETAIL https://phabricator.wikimedia.org/T100123 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Multichill Cc: Legoktm, XZise, Multichill, pywikibot-bugs-list, jayvdb, Aklapper, Izno, Wikidata-bugs, aude, Mbch331 ___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Commented On] T104522: Wikibase API warning of unknown format
Multichill added a comment. This is just annoying for some time, not actually breaking anything so "Unbreak Now!" might be a bit too much. TASK DETAIL https://phabricator.wikimedia.org/T104522 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Multichill Cc: Multichill, Ricordisamoa, Aklapper, jayvdb, pywikibot-bugs-list, Wikidata-bugs, aude, Mbch331 ___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Changed Subscribers] T104522: Wikibase API warning of unknown format
Multichill added a subscriber: Multichill. TASK DETAIL https://phabricator.wikimedia.org/T104522 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Multichill Cc: Multichill, Ricordisamoa, Aklapper, jayvdb, pywikibot-bugs-list, Wikidata-bugs, aude, Mbch331 ___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Commented On] T120327: SUL login fails if pywikibot is the first visit to the wiki
Multichill added a comment. Just ran into this. https://phabricator.wikimedia.org/rPWBC261aa028acb1fda727f7ee3418c5fd8db189bc6a broke it. Work around is to comment out self.check_user_exists() on line 254 TASK DETAIL https://phabricator.wikimedia.org/T120327 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Multichill Cc: Multichill, zhuyifei1999, Aklapper, jayvdb, pywikibot-bugs-list ___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Changed Subscribers] T120327: SUL login fails if pywikibot is the first visit to the wiki
Multichill added a subscriber: Multichill. TASK DETAIL https://phabricator.wikimedia.org/T120327 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Multichill Cc: Multichill, zhuyifei1999, Aklapper, jayvdb, pywikibot-bugs-list ___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Commented On] T103063: Drop py2.6 support
Multichill added a comment. Interesting post @jayvdb . I'll follow up on that next week. TASK DETAIL https://phabricator.wikimedia.org/T103063 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jayvdb, Multichill Cc: XZise, Multichill, jayvdb, Aklapper, pywikibot-bugs-list, Ricordisamoa ___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Changed Subscribers] T115679: Handle property datatype changing gracefully
Multichill added a subscriber: Multichill. TASK DETAIL https://phabricator.wikimedia.org/T115679 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Multichill Cc: Multichill, Legoktm, Aklapper, jayvdb, pywikibot-bugs-list, Ricordisamoa ___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Changed Subscribers] T115679: Handle property datatype changing gracefully
Multichill added subscribers: Lydia_Pintscher, hoo. Multichill added a project: Wikidata. TASK DETAIL https://phabricator.wikimedia.org/T115679 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Multichill Cc: hoo, Lydia_Pintscher, Multichill, Legoktm, Aklapper, jayvdb, pywikibot-bugs-list, Wikidata-bugs, aude, Ricordisamoa ___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Commented On] T56311: Random api errors
Multichill added a comment. In https://phabricator.wikimedia.org/T56311#1713509, @jayvdb wrote: > Has this problem occurred since about April 2015 when T61678: Implement > badtoken detection and recovery <https://phabricator.wikimedia.org/T61678> > was fixed? A lot of things have changed since this bug was filed. Pywikibot is much more stable. Let's close this one as resolved. If we run into new problems we can file a new more specific bug. TASK DETAIL https://phabricator.wikimedia.org/T56311 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Multichill Cc: Aklapper, Nemo_bis, Mpaa, Xqt, jayvdb, pywikibot-bugs-list, Multichill, Ladsgroup, Ricordisamoa ___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Updated] T66503: Add datatype time to harvest_template.py for importing dates
Multichill added a comment. I think that if https://phabricator.wikimedia.org/T112141 gets implemented that this works too without any extra code. TASK DETAIL https://phabricator.wikimedia.org/T66503 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Multichill Cc: Aklapper, Nemo_bis, jayvdb, Ricordisamoa, pywikibot-bugs-list, Multichill, Omegat ___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Updated] T66502: Add more advanced time/date parsing functionality
Multichill added a comment. For the Wikidata part it might be better to not do it ourselves, but outsource it, see https://phabricator.wikimedia.org/T112140 TASK DETAIL https://phabricator.wikimedia.org/T66502 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Multichill Cc: Aklapper, jayvdb, Ricordisamoa, pywikibot-bugs-list, Multichill, Ladsgroup ___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Created] T112130: Implement unit support in pywikibot WbQuanity
Multichill created this task. Multichill added a subscriber: Multichill. Multichill added projects: Pywikibot-Wikidata, pywikibot-core, Wikidata. Herald added subscribers: pywikibot-bugs-list, Aklapper. TASK DESCRIPTION Units just got deployed to Wikidata. Pywikibot should also implement support for units. Looks like WbQuantity is already prepared, but not completed for this: ``` class WbQuantity(object): """A Wikibase quantity representation.""" def __init__(self, amount, unit=None, error=None): u""" Create a new WbQuantity object. @param amount: number representing this quantity @type amount: float @param unit: not used (only unit-less quantities are supported) @param error: the uncertainty of the amount (e.g. ±1) @type error: float, or tuple of two floats, where the first value is the upper error and the second is the lower error value. ``` The unit should be a Wikidata item. More info at: https://www.wikidata.org/wiki/Wikidata:Units & https://www.mediawiki.org/wiki/Wikibase/DataModel#Quantities TASK DETAIL https://phabricator.wikimedia.org/T112130 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Multichill Cc: Aklapper, pywikibot-bugs-list, Multichill, Wikidata-bugs, aude, jayvdb, Ricordisamoa ___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Updated] T112140: Provide a wrapper function in pywikibot around wbparsevalue
Multichill added a blocked task: T112141: Update harvest_template.py to use wbparsevalue and accept arbitrary datatypes. TASK DETAIL https://phabricator.wikimedia.org/T112140 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Multichill Cc: Aklapper, pywikibot-bugs-list, Multichill, Wikidata-bugs, aude, jayvdb, Ricordisamoa ___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Updated] T112141: Update harvest_template.py to use wbparsevalue and accept arbitrary datatypes
Multichill added a blocking task: T112140: Provide a wrapper function in pywikibot around wbparsevalue. TASK DETAIL https://phabricator.wikimedia.org/T112141 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Multichill Cc: Multichill, pywikibot-bugs-list, Aklapper, Wikidata-bugs, aude, jayvdb, Ricordisamoa ___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Created] T112140: Provide a wrapper function in pywikibot around wbparsevalue
Multichill created this task. Multichill added a subscriber: Multichill. Multichill added projects: Pywikibot-Wikidata, pywikibot-core, Wikidata. Herald added subscribers: pywikibot-bugs-list, Aklapper. TASK DESCRIPTION The Wikidata api provides a function to parse a string and to try return some datatype. See https://www.wikidata.org/w/api.php?action=help=1#wbparsevalue . We should probably have a wrapper function for this in the site object so we can outsource some of the string parsing to that instead of re-inventing the wheel ourselves. TASK DETAIL https://phabricator.wikimedia.org/T112140 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Multichill Cc: Aklapper, pywikibot-bugs-list, Multichill, Wikidata-bugs, aude, jayvdb, Ricordisamoa ___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Changed Subscribers] T112156: WbQuantity should require an explicit error margin
Multichill added a subscriber: Multichill. TASK DETAIL https://phabricator.wikimedia.org/T112156 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Multichill Cc: Multichill, Aklapper, pywikibot-bugs-list, daniel, Wikidata-bugs, aude, Ricordisamoa ___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Retitled] T112130: Pywikibot crashes on items with quantities with units. Need to implement unit support in pywikibot WbQuanity
Multichill changed the title from "Implement unit support in pywikibot WbQuanity" to "Pywikibot crashes on items with quantities with units. Need to implement unit support in pywikibot WbQuanity". Multichill edited the task description. Multichill set Security to None. TASK DETAIL https://phabricator.wikimedia.org/T112130 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Multichill Cc: Aklapper, pywikibot-bugs-list, Multichill, Wikidata-bugs, aude, jayvdb, Ricordisamoa ___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Created] T112141: Update harvest_template.py to use wbparsevalue and accept arbitrary datatypes
Multichill created this task. Multichill added subscribers: Aklapper, pywikibot-bugs-list, Multichill. Multichill added projects: Wikidata, pywikibot-core, Pywikibot-Wikidata. TASK DESCRIPTION Currently harvest_template.py supports some hard coded data types (images, strings and items). For other types it should try to have wbparsevalue parse it and use that output to add the claim. An example would be to add birth dates based on fields in an infobox. https://en.wikipedia.org/wiki/Vincent_van_Gogh contains "birth_date= 30 March 1853" and we want to add https://www.wikidata.org/wiki/Property:P569 -> https://www.wikidata.org/w/api.php?action=wbparsevalue=time=30%20March%201853 TASK DETAIL https://phabricator.wikimedia.org/T112141 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Multichill Cc: Multichill, pywikibot-bugs-list, Aklapper, Wikidata-bugs, aude, jayvdb, Ricordisamoa ___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Commented On] T110819: Q404 is alive; test_item_missing failing
Multichill added a subscriber: Multichill. Multichill added a comment. Looks solved to me :-) TASK DETAIL https://phabricator.wikimedia.org/T110819 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Multichill Cc: Multichill, gerritbot, Addshore, Aklapper, jayvdb, pywikibot-bugs-list, Wikidata-bugs, aude, Ricordisamoa, Malyacko ___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Commented On] T110572: Handle search api errors better
Multichill added a comment. Maybe file an upstream bug that mediawiki should implement different types of gsrsearch-errors so that we know what we're hitting? TASK DETAIL https://phabricator.wikimedia.org/T110572 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Multichill Cc: jayvdb, Aklapper, pywikibot-bugs-list, Multichill, Malyacko ___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Commented On] T110388: filter namespace generator should not crash on unknown namespace numbers
Multichill added a comment. Thanks for the comment Jay, that seems a good direction. TASK DETAIL https://phabricator.wikimedia.org/T110388 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jayvdb, Multichill Cc: jayvdb, XZise, Mpaa, Aklapper, Multichill, JeanFred, pywikibot-bugs-list, Malyacko ___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Created] T110559: Make sure that Pywikibot still works with new api data output format
Multichill created this task.
Multichill added a subscriber: Multichill.
Multichill added a project: Pywikibot-Wikidata.
Herald added subscribers: pywikibot-bugs-list, Aklapper.
TASK DESCRIPTION
In T95168 the Wikidata api is slightly changed. In the old format if an item
doesn't have claims, you won't see claim in the api. In the new format will
return 'claims: {}' as part of the output. Same goes for descriptions,
aliases and sitelinks. More info at
https://www.wikidata.org/wiki/User:Addshore/API_Break_September_2015 . We just
need to make sure pywikibot can work with both formats.
TASK DETAIL
https://phabricator.wikimedia.org/T110559
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Multichill
Cc: Aklapper, pywikibot-bugs-list, Multichill, Ricordisamoa, Malyacko
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Updated] T110559: Make sure that Pywikibot still works with new api data output format
Multichill added a blocking task: T95168: [Task] Remove calls to getIsRawMode() in Wikibase, and calls to setRawMode() in tests. TASK DETAIL https://phabricator.wikimedia.org/T110559 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Multichill Cc: Aklapper, pywikibot-bugs-list, Multichill, Ricordisamoa, Malyacko ___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Created] T110572: Handle search api errors better
Multichill created this task. Multichill added a subscriber: Multichill. Multichill added projects: pywikibot-core, Pywikibot-pagegenerators. Herald added subscribers: pywikibot-bugs-list, Aklapper. TASK DESCRIPTION So I was working on matching some painters and I hit a search error ``` Traceback (most recent call last): File C:\pywikibot\coredev\painting_add_creator.py, line 315, in module main() File C:\pywikibot\coredev\painting_add_creator.py, line 310, in main paintingBot.run() File C:\pywikibot\coredev\painting_add_creator.py, line 154, in run creatorItem = self.getCreator(creator) File C:\pywikibot\coredev\painting_add_creator.py, line 196, in getCreator for creatoritem in creategen: File C:\pywikibot\coredev\pywikibot\pagegenerators.py, line 1619, in Preload ingItemGenerator for page in generator: File C:\pywikibot\coredev\pywikibot\pagegenerators.py, line 1674, in Wikibas eItemGenerator for page in gen: File C:\pywikibot\coredev\pywikibot\pagegenerators.py, line 1998, in SearchP ageGenerator namespaces=namespaces): File C:\pywikibot\coredev\pywikibot\data\api.py, line 2265, in __iter__ self.data = self.request.submit() File C:\pywikibot\coredev\pywikibot\data\api.py, line 1734, in submit raise APIError(**result['error']) pywikibot.data.api.APIError: gsrsearch-error: Search is currently too busy. Ple ase try again later. class 'pywikibot.data.api.APIError' CRITICAL: Closing network session. ``` Would be nicer to have error handling for the gsrsearch-error errors. When hit, it should probably wait and try later for maxtries just like other functions in pywikibot. TASK DETAIL https://phabricator.wikimedia.org/T110572 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Multichill Cc: Aklapper, pywikibot-bugs-list, Multichill, jayvdb, Malyacko ___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Reopened] T110388: filter namespace generator should not crash on unknown namespace numbers
Multichill reopened this task as Open. TASK DETAIL https://phabricator.wikimedia.org/T110388 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jayvdb, Multichill Cc: jayvdb, XZise, Mpaa, Aklapper, Multichill, JeanFred, pywikibot-bugs-list, Malyacko ___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Commented On] T110388: filter namespace generator should not crash on unknown namespace numbers
Multichill added a comment. Namespaces get added and deleted all the time. This is a filter, it shouldn't crash when trying to filter out a namespace that isn't being supplied in the first place. Solution would be to not remove the namespaces (ever), but mark them as deprecated and only throw a deprecation warning when these are encountered. TASK DETAIL https://phabricator.wikimedia.org/T110388 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jayvdb, Multichill Cc: jayvdb, XZise, Mpaa, Aklapper, Multichill, JeanFred, pywikibot-bugs-list, Malyacko ___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Updated] T110388: filter namespace generator should not crash on unknown namespace numbers
Multichill added a project: Pywikibot-pagegenerators. Multichill set Security to None. TASK DETAIL https://phabricator.wikimedia.org/T110388 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Multichill Cc: Aklapper, Multichill, JeanFred, pywikibot-bugs-list, jayvdb, Malyacko ___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Commented On] T104787: Rewrite/Update/Validate pagegenerators YearPageGenerator and DayPageGenerator using Wikidata
Multichill added a comment. Also worth discussing, .. what are these two generators and date.py being used for? Or, should we simply drop date.py and these two generators? I think I used it once, by far not enough to keep it around and spend a lot of effort on it. Maybe ask around on the mailing list? Wait a couple of weeks, if no response, add a big deprecation warning pointing to this bug? TASK DETAIL https://phabricator.wikimedia.org/T104787 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Multichill Cc: Ricordisamoa, valhallasw, Xqt, Aklapper, Multichill, jayvdb, pywikibot-bugs-list, Malyacko, P.Copp ___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Commented On] T74101: cmstartsortkey: DEPRECATED! Use starthexsortkey instead
Multichill added a subscriber: Multichill. Multichill added a comment. Bump pagegenerators.CategorizedPageGenerator(workcat, namespaces=6, start=u'Ambrosius') WARNING: API warning (categorymembers): The gcmstartsortkey parameter has been deprecated. TASK DETAIL https://phabricator.wikimedia.org/T74101 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Multichill Cc: Multichill, Ricordisamoa, pywikibot-bugs-list, Mpaa, jayvdb, Malyacko, P.Copp ___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Created] T104780: wikidataquery STRING unicode issues
Multichill created this task. Multichill added a subscriber: Multichill. Multichill added projects: pywikibot-core, Pywikibot-Wikidata. Herald added subscribers: pywikibot-bugs-list, Aklapper. TASK DESCRIPTION Looking in wikidataquery.py everything is str(), not unicode(). No problem for most of it except when querying STRING[some string]. If you do that and the string happens to contain a non-ascii character: ``` query = 'CLAIM[31:3305213] AND CLAIM[195:%s] AND STRING[217:%s]' % (instid, metadata.get(idField),) data = wd_query.query(wd_queryset) File C:\pywikibot\coredev\pywikibot\data\wikidataquery.py, line 591, in quer y fullQueryString = self.getQueryString(q, labels, props) File C:\pywikibot\coredev\pywikibot\data\wikidataquery.py, line 481, in getQ ueryString qStr = q=%s % quote(str(q)) File C:\pywikibot\coredev\pywikibot\data\wikidataquery.py, line 107, in __st r__ s = bracketIfQuerySet(self.qs[0]) File C:\pywikibot\coredev\pywikibot\data\wikidataquery.py, line 105, in brac ketIfQuerySet return str(q) UnicodeEncodeError: 'ascii' codec can't encode character u'\xe3' in position 76: ordinal not in range(128) type 'exceptions.UnicodeEncodeError' CRITICAL: Closing network session. Pywikibot: [ssh] pywikibot-core.git (f0e0d87, g5686, 2015/06/06, 13:11:19, OUTDA TED) Release version: 2.0b3 requests version: 2.4.1 cacerts: C:\Python27\lib\site-packages\requests\cacert.pem certificate test: ok Python: 2.7.2 (default, Jun 12 2011, 15:08:59) [MSC v.1500 32 bit (Intel)] ``` TASK DETAIL https://phabricator.wikimedia.org/T104780 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Multichill Cc: Aklapper, pywikibot-bugs-list, Multichill, jayvdb, Ricordisamoa, Malyacko, P.Copp ___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Commented On] T100123: API code 'readonly' not handled
Multichill added a comment. [21:51:25] multichill In https://phabricator.wikimedia.org/T100123 the last question. You might be able to answer that [21:51:31] anomie looks [21:51:35] multichill How long should we back off when we hit read-only? [21:55:18] anomie multichill: For the particular situation there, The database has been automatically locked while the slave database servers catch up to the master, you're basically waiting for maxlag to go back down. When using the actual maxlag parameter, the recommendation is 5 seconds. [21:56:14] multichill So just the normal 5, 10, 20, 40 etc routine TASK DETAIL https://phabricator.wikimedia.org/T100123 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Multichill Cc: Multichill, pywikibot-bugs-list, jayvdb, Aklapper, Wikidata-bugs, aude, Malyacko, P.Copp ___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Commented On] T100123: API code 'readonly' not handled
Multichill added a subscriber: Multichill.
Multichill added a comment.
I ran into this twice tonight while running on Wikidata:
2015-07-01 22:32:31 api.py, 1649 in submit: VERBOSE
Pausing due to database lag: Waiting for 10.64.16.15: 6 seconds lagged
2015-07-01 22:32:31throttle.py, 249 in wait: INFO
Sleeping for 5.0 seconds, 2015-07-01 22:32:31
2015-07-01 22:32:36throttle.py, 140 in checkMultiplicity: VERBOSE
Found 1 wikidata:wikidata processes running, including this one.
2015-07-01 22:32:36 api.py, 1649 in submit: VERBOSE
Pausing due to database lag: Waiting for 10.64.16.144: 9 seconds lagged
2015-07-01 22:32:36throttle.py, 249 in wait: INFO
Sleeping for 5.0 seconds, 2015-07-01 22:32:36
2015-07-01 22:32:42 api.py, 1730 in submit: VERBOSE
API Error: query=
{'action': [u'wbsetreference'],
u'assert': [u'user'],
u'baserevid': [225657903],
u'bot': [1],
u'format': [u'json'],
u'maxlag': ['5'],
u'snaks': [u'{P813: [{datavalue: {type: time, value: {after: 0,
precision: 11, time: +0002015-07-01T00:00:00Z, timezone: 0,
calendarmodel: http://www.wikidata.org/entity/Q1985727;, before: 0}},
property: P813, snaktype: value}], P854: [{datavalue: {type:
string, value:
http://www.mfa.org/collections/object/reading-on-the-rocks-grand-manan-33709},
property: P854, snaktype: value}]}'],
'statement': [u'Q20635272$AF33EB11-5184-4212-9CE4-88BC1B5823CC'],
u'token': [u'5345asdfasdf44878+\\']}
2015-07-01 22:32:42 api.py, 1732 in submit: VERBOSE
response=
{u'servedby': u'mw1122', u'error': {u'info': u'The wiki is currently in
read-only mode', u'readonlyreason': u'The database has been automatically
locked while the slave database servers catch up to the master', u'code':
u'readonly', u'help': u'See https://www.wikidata.org/w/api.php for API usage'}}
File C:\pywikibot\coredev\pywikibot\data\api.py, line 1734, in submit
raise APIError(**result['error'])
pywikibot.data.api.APIError: readonly: The wiki is currently in read-only mode
class 'pywikibot.data.api.APIError'
CRITICAL: Closing network session.
Would indeed be much nicer if the bot would do the graceful back off and
recover logic.
TASK DETAIL
https://phabricator.wikimedia.org/T100123
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Multichill
Cc: Multichill, pywikibot-bugs-list, jayvdb, Aklapper, Malyacko, P.Copp
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Commented On] T85656: detect and correctly handle edits that trigger abusefilter rules
Multichill added a comment. We could setup some test cases on one of the test wiki, hit it with a bot and see what the api spits out. Come to think of it, these would also be excellent integrated unit tests to setup ;-) TASK DETAIL https://phabricator.wikimedia.org/T85656 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Multichill Cc: XZise, Multichill, NiharikaKohli, Qgil, Xqt, Mpaa, jayvdb, Aklapper, pywikibot-bugs-list, Ryasmeen, Gryllida, Malyacko, P.Copp ___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Commented On] T85656: detect and correctly handle edits that trigger abusefilter rules
Multichill added a subscriber: Multichill. Multichill added a comment. I run into this (misconfigured?) abuse filter every once in a while: File C:\pywikibot\coredev\pywikibot\page.py, line 4194, in addSources data = self.repo.editSource(self, claims, new=True, **kwargs) File C:\pywikibot\coredev\pywikibot\site.py, line 1087, in callee return fn(self, *args, **kwargs) File C:\pywikibot\coredev\pywikibot\site.py, line 5882, in editSource data = req.submit() File C:\pywikibot\coredev\pywikibot\data\api.py, line 1734, in submit raise APIError(**result['error']) pywikibot.data.api.APIError: failed-save: abusefilter-warning-63 class 'pywikibot.data.api.APIError' CRITICAL: Closing network session. The only action for this filter (https://www.wikidata.org/wiki/Special:AbuseFilter/63) is to warn. The bot shouldn't throw an api error, the bot should may show a warning and retry. TASK DETAIL https://phabricator.wikimedia.org/T85656 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Multichill Cc: Multichill, NiharikaKohli, Qgil, Xqt, Mpaa, jayvdb, Aklapper, pywikibot-bugs-list, Ryasmeen, Gryllida, Malyacko, P.Copp ___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Changed Subscribers] T102741: Python library for Wikibase data values
Multichill added a subscriber: Multichill. TASK DETAIL https://phabricator.wikimedia.org/T102741 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Multichill Cc: Multichill, XZise, Ricordisamoa, Yurik, Husky, mkroetzsch, Denny, jcreus, Legoktm, Pyfisch, johl, Lucie, JeroenDeDauw, Aklapper, jayvdb, pywikibot-bugs-list, Wikidata-bugs, aude, Malyacko, P.Copp ___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
[Pywikipedia-bugs] [Maniphest] [Commented On] T101214: Decommission compat
Multichill added a comment. Announcement is included in https://meta.wikimedia.org/wiki/Tech/News/2015/25 . I added a link to https://www.mediawiki.org/wiki/Manual:Pywikibot/Compat_deprecation . We might want to mark that page for translation. TASK DETAIL https://phabricator.wikimedia.org/T101214 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Multichill Cc: Liuxinyu970226, Mpaa, Ricordisamoa, JAnD, XZise, Xqt, Legoktm, valhallasw, Ladsgroup, pywikibot-bugs-list, Multichill, jayvdb, Aklapper, Johan, Anshoe, gpaumier, Jay8g, Krenair, Quiddity, greg ___ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
