[Pywikipedia-bugs] [Maniphest] [Commented On] T238291: noreferences.py error

[Kaleem_Bhatti]

Kaleem_Bhatti added a comment.


  I want edit scripts from "core" directory

TASK DETAIL
  https://phabricator.wikimedia.org/T238291

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Kaleem_Bhatti
Cc: Dvorapa, Xqt, Aklapper, pywikibot-bugs-list, Kaleem_Bhatti, Zkhalido, 
Viztor, DannyS712, Wenyi, Tbscho, MayS, Mdupont, JJMC89, Altostratus, 
Avicennasis, mys_721tx, jayvdb, Dalba, Masti, Alchimista, Rxy
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T238291: noreferences.py error

[Kaleem_Bhatti]

Kaleem_Bhatti added a comment.


  @Dvorapa how I can find this core directore 
  /mnt/nfs/labstore-secondary-tools-project/toolname/pywikibot-core
  I have installet it from git and here
   $ git clone --recursive --branch stable
  https://gerrit.wikimedia.org/r/pywikibot/core.git pywikibot-core
  $ cd pywikibot-core

TASK DETAIL
  https://phabricator.wikimedia.org/T238291

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Kaleem_Bhatti
Cc: Dvorapa, Xqt, Aklapper, pywikibot-bugs-list, Kaleem_Bhatti, Zkhalido, 
Viztor, DannyS712, Wenyi, Tbscho, MayS, Mdupont, JJMC89, Altostratus, 
Avicennasis, mys_721tx, jayvdb, Dalba, Masti, Alchimista, Rxy
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T238291: noreferences.py error

[Dvorapa]

Dvorapa added a comment.


  I'm happy to help, but please ask again, currently I don't understand

TASK DETAIL
  https://phabricator.wikimedia.org/T238291

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Kaleem_Bhatti, Dvorapa
Cc: Dvorapa, Xqt, Aklapper, pywikibot-bugs-list, Kaleem_Bhatti, Zkhalido, 
Viztor, DannyS712, Wenyi, Tbscho, MayS, Mdupont, JJMC89, Altostratus, 
Avicennasis, mys_721tx, jayvdb, Dalba, Masti, Alchimista, Rxy
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T238291: noreferences.py error

[Kaleem_Bhatti]

Kaleem_Bhatti added a comment.


  
  
  In T238291#5668955 , 
@Kaleem_Bhatti wrote:
  
  > In T238291#5668954 , 
@Dvorapa wrote:
  >
  >> Then it is where you installed it
  >
  > from here
  > $ git clone --recursive --branch stable
  >  https://gerrit.wikimedia.org/r/pywikibot/core.git pywikibot-core
  > $ cd pywikibot-core
  
  @Dvorapa  please tell me this

TASK DETAIL
  https://phabricator.wikimedia.org/T238291

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Kaleem_Bhatti
Cc: Dvorapa, Xqt, Aklapper, pywikibot-bugs-list, Kaleem_Bhatti, Zkhalido, 
Viztor, DannyS712, Wenyi, Tbscho, MayS, Mdupont, JJMC89, Altostratus, 
Avicennasis, mys_721tx, jayvdb, Dalba, Masti, Alchimista, Rxy
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T238471: Performance problems with pywikibot's pagegenerator

[Xqt]

Xqt added a comment.


  Decreasing maxlag parameter may lead to decreasing performance because some 
queries are probably waiting for a lower server lag

TASK DETAIL
  https://phabricator.wikimedia.org/T238471

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Xqt
Cc: Xqt, Mpaa, Dvorapa, Aklapper, pywikibot-bugs-list, Tilomi, Zkhalido, 
darthmon_wmde, Viztor, DannyS712, Nandana, Wenyi, Lahi, Gq86, 
GoranSMilovanovic, QZanden, Tbscho, MayS, LawExplorer, Mdupont, JJMC89, 
_jensen, rosalieper, Altostratus, Avicennasis, Scott_WUaS, mys_721tx, 
Wikidata-bugs, aude, jayvdb, Ricordisamoa, Dalba, Masti, Alchimista, Mbch331, 
Rxy
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Updated] T238471: Performance problems with pywikibot's pagegenerator

[Dvorapa]

Dvorapa added a project: Pywikibot-Wikidata.

TASK DETAIL
  https://phabricator.wikimedia.org/T238471

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa
Cc: Mpaa, Dvorapa, Aklapper, pywikibot-bugs-list, Tilomi, Zkhalido, 
darthmon_wmde, Viztor, DannyS712, Nandana, Wenyi, Lahi, Gq86, 
GoranSMilovanovic, QZanden, Tbscho, MayS, LawExplorer, Mdupont, JJMC89, 
_jensen, rosalieper, Altostratus, Avicennasis, Scott_WUaS, mys_721tx, 
Wikidata-bugs, aude, jayvdb, Ricordisamoa, Dalba, Masti, Alchimista, Mbch331, 
Rxy
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Updated] T210322: paraminfo_tests fails several times due to timeout at beta.wmflabs.org

[Dvorapa]

Dvorapa added a project: Pywikibot-Wikidata.

TASK DETAIL
  https://phabricator.wikimedia.org/T210322

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa
Cc: Dvorapa, Krenair, gerritbot, Framawiki, Aklapper, pywikibot-bugs-list, 
Dalba, Xqt, Zkhalido, Hook696, Daryl-TTMG, RomaAmorRoma, 0010318400, 
E.S.A-Sheild, darthmon_wmde, Meekrab2012, joker88john, Viztor, DannyS712, 
CucyNoiD, Nandana, NebulousIris, Wenyi, Gaboe420, Versusxo, Majesticalreaper22, 
Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Af420, Darkminds3113, Bsandipan, 
Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, 
Tbscho, MayS, LawExplorer, WSH1906, Lewizho99, Mdupont, JJMC89, Maathavan, 
_jensen, rosalieper, Altostratus, Avicennasis, Scott_WUaS, mys_721tx, 
Wikidata-bugs, aude, jayvdb, Ricordisamoa, Masti, Alchimista, Mbch331, Rxy
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Updated] T112243: Overwriting a redirect using wbeditentity causes an exception

[Dvorapa]

Dvorapa added a project: Pywikibot-Wikidata.

TASK DETAIL
  https://phabricator.wikimedia.org/T112243

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa
Cc: Lydia_Pintscher, StudiesWorld, hoo, Ricordisamoa, pywikibot-bugs-list, 
jayvdb, Aklapper, Zkhalido, darthmon_wmde, Viztor, DannyS712, Nandana, Wenyi, 
Lahi, Gq86, GoranSMilovanovic, QZanden, Tbscho, MayS, LawExplorer, Mdupont, 
JJMC89, Dvorapa, _jensen, rosalieper, Altostratus, Avicennasis, Scott_WUaS, 
mys_721tx, Wikidata-bugs, aude, Dalba, Masti, Alchimista, Mbch331, Rxy
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Updated] T221556: pywikibot's interwikidata.py won't handle projects where one externally-hosted language doesn't have access to the wikibase repo

[Dvorapa]

Dvorapa added a project: Pywikibot-Wikidata.

TASK DETAIL
  https://phabricator.wikimedia.org/T221556

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa
Cc: Dvorapa, Ladsgroup, Xqt, Aklapper, pywikibot-bugs-list, Carlb, Zkhalido, 
darthmon_wmde, Viztor, DannyS712, Nandana, Wenyi, Lahi, Gq86, 
GoranSMilovanovic, QZanden, Tbscho, MayS, LawExplorer, Mdupont, JJMC89, 
_jensen, rosalieper, Altostratus, Avicennasis, Scott_WUaS, mys_721tx, 
Wikidata-bugs, aude, jayvdb, Ricordisamoa, Dalba, Masti, Alchimista, Mbch331, 
Rxy
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Closed] T238291: noreferences.py error

[Dvorapa]

Dvorapa closed this task as "Resolved".
Dvorapa assigned this task to Kaleem_Bhatti.
Dvorapa added a comment.


  The issue seems to be fixed by 
https://gerrit.wikimedia.org/r/#/c/pywikibot/core/+/546388/ in the current 
master => so it should be fixed in the next stable release

TASK DETAIL
  https://phabricator.wikimedia.org/T238291

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Kaleem_Bhatti, Dvorapa
Cc: Dvorapa, Xqt, Aklapper, pywikibot-bugs-list, Kaleem_Bhatti, Zkhalido, 
Viztor, DannyS712, Wenyi, Tbscho, MayS, Mdupont, JJMC89, Altostratus, 
Avicennasis, mys_721tx, jayvdb, Dalba, Masti, Alchimista, Rxy
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T238471: Performance problems with pywikibot's pagegenerator

[Dvorapa]

Dvorapa added a comment.


  Perhaps try WikidataIntegrator instead of Pywikibot (if it offers the same 
functionality you need). I'm unsure what is its current state of development, 
so make sure you don't throw away your original code. A performance comparison 
of the two libraries from your case would be interesting.

TASK DETAIL
  https://phabricator.wikimedia.org/T238471

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa
Cc: Mpaa, Dvorapa, Aklapper, pywikibot-bugs-list, Tilomi, Zkhalido, 
darthmon_wmde, Viztor, DannyS712, Nandana, Wenyi, Lahi, Gq86, 
GoranSMilovanovic, QZanden, Tbscho, MayS, LawExplorer, Mdupont, JJMC89, 
_jensen, rosalieper, Altostratus, Avicennasis, Scott_WUaS, mys_721tx, 
Wikidata-bugs, aude, jayvdb, Dalba, Masti, Alchimista, Mbch331, Rxy
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T238471: Performance problems with pywikibot's pagegenerator

[Tilomi]

Tilomi added a comment.


  I thought about data dumps aswell. But i think it's a bit overkill to 
download 60 Gb to query about 100 Mb of data so i wanted to check first if 
there are any other possibilities regarding the SPARQL endpoint or pywikibot.

TASK DETAIL
  https://phabricator.wikimedia.org/T238471

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Tilomi
Cc: Mpaa, Dvorapa, Aklapper, pywikibot-bugs-list, Tilomi, Zkhalido, 
darthmon_wmde, Viztor, DannyS712, Nandana, Wenyi, Lahi, Gq86, 
GoranSMilovanovic, QZanden, Tbscho, MayS, LawExplorer, Mdupont, JJMC89, 
_jensen, rosalieper, Altostratus, Avicennasis, Scott_WUaS, mys_721tx, 
Wikidata-bugs, aude, jayvdb, Dalba, Masti, Alchimista, Mbch331, Rxy
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T238471: Performance problems with pywikibot's pagegenerator

[Mpaa]

Mpaa added a comment.


  Have you considered to get data from dumps, if possible?

TASK DETAIL
  https://phabricator.wikimedia.org/T238471

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Mpaa
Cc: Mpaa, Dvorapa, Aklapper, pywikibot-bugs-list, Tilomi, Zkhalido, 
darthmon_wmde, Viztor, DannyS712, Nandana, Wenyi, Lahi, Gq86, 
GoranSMilovanovic, QZanden, Tbscho, MayS, LawExplorer, Mdupont, JJMC89, 
_jensen, rosalieper, Altostratus, Avicennasis, Scott_WUaS, mys_721tx, 
Wikidata-bugs, aude, jayvdb, Dalba, Masti, Alchimista, Mbch331, Rxy
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T238471: Performance problems with pywikibot's pagegenerator

[Tilomi]

Tilomi added a comment.


  I played around with them, but i saw no change in performance. I set 
following options to following values because i thought they would affect the 
performance:
  minthrottle = 0 // default value
  maxthrottle = 1 // default 60
  put_throttle = 0 // 10 (i only read pages so this was probably unnecessary)
  maxlag = 1 // default 5
  step = -1 // default value
  
  For 100 pages it takes about 1 minute and 20 seconds. For 1000 it takes ~12 
minutes. I have the same times when i tested it with the default config.

TASK DETAIL
  https://phabricator.wikimedia.org/T238471

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Tilomi
Cc: Dvorapa, Aklapper, pywikibot-bugs-list, Tilomi, Zkhalido, darthmon_wmde, 
Viztor, DannyS712, Nandana, Wenyi, Lahi, Gq86, GoranSMilovanovic, QZanden, 
Tbscho, MayS, LawExplorer, Mdupont, JJMC89, _jensen, rosalieper, Altostratus, 
Avicennasis, Scott_WUaS, mys_721tx, Wikidata-bugs, aude, jayvdb, Dalba, Masti, 
Alchimista, Mbch331, Rxy
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T238291: noreferences.py error

[Kaleem_Bhatti]

Kaleem_Bhatti added a comment.


  In T238291#5668954 , 
@Dvorapa wrote:
  
  > Then it is where you installed it
  
  from here
  $ git clone --recursive --branch stable
   https://gerrit.wikimedia.org/r/pywikibot/core.git pywikibot-core
  $ cd pywikibot-core

TASK DETAIL
  https://phabricator.wikimedia.org/T238291

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Kaleem_Bhatti
Cc: Dvorapa, Xqt, Aklapper, pywikibot-bugs-list, Kaleem_Bhatti, Zkhalido, 
Viztor, DannyS712, Wenyi, Tbscho, MayS, Mdupont, JJMC89, Altostratus, 
Avicennasis, mys_721tx, jayvdb, Dalba, Masti, Alchimista, Rxy
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T238291: noreferences.py error

[Dvorapa]

Dvorapa added a comment.


  Then it is where you installed it

TASK DETAIL
  https://phabricator.wikimedia.org/T238291

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa
Cc: Dvorapa, Xqt, Aklapper, pywikibot-bugs-list, Kaleem_Bhatti, Zkhalido, 
Viztor, DannyS712, Wenyi, Tbscho, MayS, Mdupont, JJMC89, Altostratus, 
Avicennasis, mys_721tx, jayvdb, Dalba, Masti, Alchimista, Rxy
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T238291: noreferences.py error

[Kaleem_Bhatti]

Kaleem_Bhatti added a comment.


  In T238291#5668894 , 
@Dvorapa wrote:
  
  > It depends how you installed them.
  
  I have intalled from git

TASK DETAIL
  https://phabricator.wikimedia.org/T238291

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Kaleem_Bhatti
Cc: Dvorapa, Xqt, Aklapper, pywikibot-bugs-list, Kaleem_Bhatti, Zkhalido, 
Viztor, DannyS712, Wenyi, Tbscho, MayS, Mdupont, JJMC89, Altostratus, 
Avicennasis, mys_721tx, jayvdb, Dalba, Masti, Alchimista, Rxy
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T238471: Performance problems with pywikibot's pagegenerator

[Dvorapa]

Dvorapa added a comment.


  Have you tried to play around with "SETTINGS TO AVOID SERVER OVERLOAD" 
options in Pywikibot's user-config.py?

TASK DETAIL
  https://phabricator.wikimedia.org/T238471

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa
Cc: Dvorapa, Aklapper, pywikibot-bugs-list, Tilomi, Zkhalido, darthmon_wmde, 
Viztor, DannyS712, Nandana, Wenyi, Lahi, Gq86, GoranSMilovanovic, QZanden, 
Tbscho, MayS, LawExplorer, Mdupont, JJMC89, _jensen, rosalieper, Altostratus, 
Avicennasis, Scott_WUaS, mys_721tx, Wikidata-bugs, aude, jayvdb, Dalba, Masti, 
Alchimista, Mbch331, Rxy
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T238291: noreferences.py error

[Dvorapa]

Dvorapa added a comment.


  It depends how you installed them.

TASK DETAIL
  https://phabricator.wikimedia.org/T238291

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa
Cc: Dvorapa, Xqt, Aklapper, pywikibot-bugs-list, Kaleem_Bhatti, Zkhalido, 
Viztor, DannyS712, Wenyi, Tbscho, MayS, Mdupont, JJMC89, Altostratus, 
Avicennasis, mys_721tx, jayvdb, Dalba, Masti, Alchimista, Rxy
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T238291: noreferences.py error

[Kaleem_Bhatti]

Kaleem_Bhatti added a comment.


  @Dvorapa how I can open or edit core folder and scripts

TASK DETAIL
  https://phabricator.wikimedia.org/T238291

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Kaleem_Bhatti
Cc: Dvorapa, Xqt, Aklapper, pywikibot-bugs-list, Kaleem_Bhatti, Zkhalido, 
Viztor, DannyS712, Wenyi, Tbscho, MayS, Mdupont, JJMC89, Altostratus, 
Avicennasis, mys_721tx, jayvdb, Dalba, Masti, Alchimista, Rxy
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T238291: noreferences.py error

[Kaleem_Bhatti]

Kaleem_Bhatti added a comment.


  In T238291#5668763 , 
@Dvorapa wrote:
  
  > It's a "core" folder in your home directory I believe. The error says your 
Wikipedia (wikiproject) is missing in the script's configuration. We want to 
add it there.
  
  I'm running in sindhi (sd) wikipedia

TASK DETAIL
  https://phabricator.wikimedia.org/T238291

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Kaleem_Bhatti
Cc: Dvorapa, Xqt, Aklapper, pywikibot-bugs-list, Kaleem_Bhatti, Zkhalido, 
Viztor, DannyS712, Wenyi, Tbscho, MayS, Mdupont, JJMC89, Altostratus, 
Avicennasis, mys_721tx, jayvdb, Dalba, Masti, Alchimista, Rxy
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Created] T238471: Performance problems with pywikibot's pagegenerator

[Tilomi]

Tilomi created this task.
Tilomi added projects: Pywikibot, Wikidata.
Restricted Application added subscribers: pywikibot-bugs-list, Aklapper.

TASK DESCRIPTION
  Hi, i'm currently working on an open source university project 
(https://openartbrowser.org/ , 
https://github.com/hochschule-darmstadt/openartbrowser).
  
  We query different datasets from wikidata about art. In our ETL process we 
get the data we want with help of the pywikibot libary. This libary loads the 
wikidata sites by their qId's which we query beforehand with SPARQL 
(pagegenerator.WikidataSPARQLPageGenerator). This whole process of extracting 
around 150.000 entries took us 47 hours last time measured at the ending of 
october.
  
  We want to improve our crawler performance so that we can test new features 
faster.
  
  Our implementation can be viewed here 
https://github.com/hochschule-darmstadt/openartbrowser/blob/staging/scripts/Wikidata%20crawler/ArtOntologyCrawler.py
 in the extract_artworks function. This function first queries all qIds of 
Paintings, Drawings and Sculptures. Next we iterate over the items returned by 
the page generator. When measuring the times i came accross that the item.get() 
takes from 0.5 to 3-4 seconds. I assume that this is a page load for all data 
on the page of an wikidata entity.
  
  The only possibility i see at the moment to improve this data extraction is 
multi-threading because wikidata allows five queries in parallel (which equals 
five page loads). A direct SPARQL queries seems to be not possible because 
requests are very limited (see 
https://www.mediawiki.org/wiki/Wikidata_Query_Service/User_Manual#Query_limits).
  
  Maybe there is another way of solving this performance issue. 
  Best regards.

TASK DETAIL
  https://phabricator.wikimedia.org/T238471

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Tilomi
Cc: Aklapper, pywikibot-bugs-list, Tilomi, Zkhalido, darthmon_wmde, Viztor, 
DannyS712, Nandana, Wenyi, Lahi, Gq86, GoranSMilovanovic, QZanden, Tbscho, 
MayS, LawExplorer, Mdupont, JJMC89, Dvorapa, _jensen, rosalieper, Altostratus, 
Avicennasis, Scott_WUaS, mys_721tx, Wikidata-bugs, aude, jayvdb, Dalba, Masti, 
Alchimista, Mbch331, Rxy
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T238291: noreferences.py error

[Dvorapa]

Dvorapa added a comment.


  It's a "core" folder in your home directory I believe. The error says your 
Wikipedia (wikiproject) is missing in the script's configuration. We want to 
add it there.

TASK DETAIL
  https://phabricator.wikimedia.org/T238291

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa
Cc: Dvorapa, Xqt, Aklapper, pywikibot-bugs-list, Kaleem_Bhatti, Zkhalido, 
Viztor, DannyS712, Wenyi, Tbscho, MayS, Mdupont, JJMC89, Altostratus, 
Avicennasis, mys_721tx, jayvdb, Dalba, Masti, Alchimista, Rxy
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T238448: Get family info from wikidata

[Mpaa]

Mpaa added a comment.


  In T238448#5668653 , 
@Bugreporter wrote:
  
  > not all (special) projects are Wikidata clients.
  
  Then we can do it for those who are and leave the others hard-coded.

TASK DETAIL
  https://phabricator.wikimedia.org/T238448

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Mpaa
Cc: Bugreporter, Aklapper, pywikibot-bugs-list, Mpaa, Zkhalido, Viztor, 
DannyS712, Wenyi, Tbscho, MayS, Mdupont, JJMC89, Dvorapa, Altostratus, 
Avicennasis, mys_721tx, jayvdb, Dalba, Masti, Alchimista, Rxy
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T235500: Pywikibot cannot create new property

[Mpaa]

Mpaa added a comment.


  In T235500#5663181 , 
@DD063520 wrote:
  
  > @Xqt : hi, would you consider again this patch, currently I make this 
modification locally to have some code running
  
  You can submit patches directly:
  https://www.mediawiki.org/wiki/Manual:Pywikibot/Development

TASK DETAIL
  https://phabricator.wikimedia.org/T235500

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Mpaa
Cc: Mpaa, Addshore, Xqt, Aklapper, DD063520, pywikibot-bugs-list, Zkhalido, 
Viztor, DannyS712, Wenyi, Tbscho, MayS, Mdupont, JJMC89, Dvorapa, Altostratus, 
Avicennasis, mys_721tx, jayvdb, Ricordisamoa, Dalba, Masti, Alchimista, Rxy
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Closed] T238404: Pywikibot fails when importing entity Q1410828

[Mpaa]

Mpaa closed this task as "Invalid".

TASK DETAIL
  https://phabricator.wikimedia.org/T238404

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Mpaa
Cc: Dvorapa, Mpaa, Xqt, Aklapper, DD063520, pywikibot-bugs-list, Zkhalido, 
darthmon_wmde, Viztor, DannyS712, Nandana, Wenyi, Lahi, Gq86, 
GoranSMilovanovic, QZanden, Tbscho, MayS, LawExplorer, Mdupont, JJMC89, 
_jensen, rosalieper, Altostratus, Avicennasis, Scott_WUaS, mys_721tx, 
Wikidata-bugs, aude, jayvdb, Dalba, Masti, Alchimista, Mbch331, Rxy
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs