[Pywikipedia-bugs] [Maniphest] [Commented On] T238471: Performance problems with pywikibot's pagegenerator

gerritbot added a comment. Change 660809 **merged** by jenkins-bot: [pywikibot/core@master] [IMPR] Create a SiteLink with __getitem__ method https://gerrit.wikimedia.org/r/660809 TASK DETAIL https://phabricator.wikimedia.org/T238471 EMAIL PREFERENCES

[Pywikipedia-bugs] [Maniphest] [Commented On] T238471: Performance problems with pywikibot's pagegenerator

gerritbot added a comment. Change 660809 had a related patch set uploaded (by Xqt; owner: Xqt): [pywikibot/core@master] [IMPR] Create a SiteLink with __getitem__ method https://gerrit.wikimedia.org/r/660809 TASK DETAIL https://phabricator.wikimedia.org/T238471 EMAIL PREFERENCES

[Pywikipedia-bugs] [Maniphest] [Commented On] T238471: Performance problems with pywikibot's pagegenerator

Xqt added a comment. You may use preload_sites.py maintenance script in meantime until the underlying problem has a better solution. The script is introduced with release 6.0.0, currently master branch. TASK DETAIL https://phabricator.wikimedia.org/T238471 EMAIL PREFERENCES

[Pywikipedia-bugs] [Maniphest] [Commented On] T238471: Performance problems with pywikibot's pagegenerator

Tilomi added a comment. Ok interesting, so i guess i have to wait until this issue is resolved or is there another functionality which i can use that won't load and initialize sitelinks? For a new feature we use the wikipedia sitelink but the performance problem occured already before we

[Pywikipedia-bugs] [Maniphest] [Commented On] T238471: Performance problems with pywikibot's pagegenerator

Dvorapa added a comment. In T238471#5669504 , @matej_suchanek wrote: >> When measuring the times i came accross that the item.get() takes from 0.5 to 3-4 seconds. > > Probably T226157: Data retrieval may be very long and heavy

[Pywikipedia-bugs] [Maniphest] [Commented On] T238471: Performance problems with pywikibot's pagegenerator

Tilomi added a comment. Thanks for the answers. I also tried the default value and higher values like 10, 20 or 60 for maxlag. But there were unfortunately no improvments with maxlag changed. I'll take a look at WikidataIntegrator and compare it with pywikibot. TASK DETAIL

[Pywikipedia-bugs] [Maniphest] [Commented On] T238471: Performance problems with pywikibot's pagegenerator

Xqt added a comment. Decreasing maxlag parameter may lead to decreasing performance because some queries are probably waiting for a lower server lag TASK DETAIL https://phabricator.wikimedia.org/T238471 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/

[Pywikipedia-bugs] [Maniphest] [Commented On] T238471: Performance problems with pywikibot's pagegenerator

Dvorapa added a comment. Perhaps try WikidataIntegrator instead of Pywikibot (if it offers the same functionality you need). I'm unsure what is its current state of development, so make sure you don't throw away your original code. A performance comparison of the two libraries from your

[Pywikipedia-bugs] [Maniphest] [Commented On] T238471: Performance problems with pywikibot's pagegenerator

Tilomi added a comment. I thought about data dumps aswell. But i think it's a bit overkill to download 60 Gb to query about 100 Mb of data so i wanted to check first if there are any other possibilities regarding the SPARQL endpoint or pywikibot. TASK DETAIL

[Pywikipedia-bugs] [Maniphest] [Commented On] T238471: Performance problems with pywikibot's pagegenerator

Mpaa added a comment. Have you considered to get data from dumps, if possible? TASK DETAIL https://phabricator.wikimedia.org/T238471 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mpaa Cc: Mpaa, Dvorapa, Aklapper, pywikibot-bugs-list, Tilomi,

[Pywikipedia-bugs] [Maniphest] [Commented On] T238471: Performance problems with pywikibot's pagegenerator

Tilomi added a comment. I played around with them, but i saw no change in performance. I set following options to following values because i thought they would affect the performance: minthrottle = 0 // default value maxthrottle = 1 // default 60 put_throttle = 0 // 10 (i only read

[Pywikipedia-bugs] [Maniphest] [Commented On] T238471: Performance problems with pywikibot's pagegenerator

Dvorapa added a comment. Have you tried to play around with "SETTINGS TO AVOID SERVER OVERLOAD" options in Pywikibot's user-config.py? TASK DETAIL https://phabricator.wikimedia.org/T238471 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Dvorapa