[Pywikipedia-bugs] [Maniphest] [Commented On] T348925: preload_sites.py script is too slow

2024-09-26 Thread gerritbot 
gerritbot added a comment.


  Change #1074562 **merged** by jenkins-bot:
  
  [pywikibot/core@master] pywikibot.scripts: Remove preload_sites.py
  
  https://gerrit.wikimedia.org/r/1074562

TASK DETAIL
  https://phabricator.wikimedia.org/T348925

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Xqt, gerritbot
Cc: ericpien, Aklapper, pywikibot-bugs-list, Xqt, mevo, Isabelladantes1983, 
Themindcoder, Adamm71, Hellket777, LisafBia6531, Susie413113, 786, Biggs657, 
PotsdamLamb, Jyoo1011, JohnsonLee01, Juan90264, SHEKH, Dijkstra, Alter-paule, 
Beast1978, Un1tY, Khutuck, Zkhalido, Hook696, Kent7301, Aram, Viztor, CucyNoiD, 
Wenyi, Gaboe420, Giuliamocci, Cpaulf30, Af420, Bsandipan, Tbscho, MayS, 
Vali.matei, Lewizho99, Mdupont, JJMC89, Maathavan, Dvorapa, Altostratus, 
Neuronton, Avicennasis, Wong128hk, mys_721tx, Dinoguy1000, jayvdb, Masti, 
Alchimista, Jay8g
___
pywikibot-bugs mailing list -- pywikibot-bugs@lists.wikimedia.org
To unsubscribe send an email to pywikibot-bugs-le...@lists.wikimedia.org

[Pywikipedia-bugs] [Maniphest] [Commented On] T348925: preload_sites.py script is too slow

2024-09-21 Thread gerritbot 
gerritbot added a comment.


  Change #1074562 had a related patch set uploaded (by Xqt; author: Xqt):
  
  [pywikibot/core@master] pywikibot.scripts: Remove preload_sites.py
  
  https://gerrit.wikimedia.org/r/1074562

TASK DETAIL
  https://phabricator.wikimedia.org/T348925

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Xqt, gerritbot
Cc: ericpien, Aklapper, pywikibot-bugs-list, Xqt, mevo, Susie413113, 
PotsdamLamb, Jyoo1011, JohnsonLee01, SHEKH, Dijkstra, Khutuck, Zkhalido, Aram, 
Viztor, Wenyi, Tbscho, MayS, Vali.matei, Mdupont, JJMC89, Dvorapa, Altostratus, 
Avicennasis, Wong128hk, mys_721tx, Dinoguy1000, jayvdb, Masti, Alchimista, Jay8g
___
pywikibot-bugs mailing list -- pywikibot-bugs@lists.wikimedia.org
To unsubscribe send an email to pywikibot-bugs-le...@lists.wikimedia.org

[Pywikipedia-bugs] [Maniphest] [Commented On] T348925: preload_sites.py script is too slow

2024-05-26 Thread gerritbot 
gerritbot added a comment.


  Change #1035864 **merged** by jenkins-bot:
  
  [pywikibot/core@master] Revert "[IMPR] use CachedRequest for userinfo 
requests"
  
  https://gerrit.wikimedia.org/r/1035864

TASK DETAIL
  https://phabricator.wikimedia.org/T348925

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Xqt, gerritbot
Cc: ericpien, Aklapper, pywikibot-bugs-list, Xqt, mevo, Isabelladantes1983, 
Themindcoder, Adamm71, Hellket777, LisafBia6531, Susie413113, 786, Biggs657, 
PotsdamLamb, Jyoo1011, JohnsonLee01, Juan90264, SHEKH, Dijkstra, Alter-paule, 
Beast1978, Un1tY, Khutuck, Zkhalido, Hook696, Kent7301, Aram, Viztor, CucyNoiD, 
Wenyi, Gaboe420, Giuliamocci, Cpaulf30, Af420, Bsandipan, Tbscho, MayS, 
Vali.matei, Lewizho99, Mdupont, JJMC89, Maathavan, Dvorapa, Altostratus, 
Neuronton, Avicennasis, Wong128hk, mys_721tx, Dinoguy1000, jayvdb, Masti, 
Alchimista, Jay8g
___
pywikibot-bugs mailing list -- pywikibot-bugs@lists.wikimedia.org
To unsubscribe send an email to pywikibot-bugs-le...@lists.wikimedia.org

[Pywikipedia-bugs] [Maniphest] [Commented On] T348925: preload_sites.py script is too slow

2024-05-26 Thread gerritbot 
gerritbot added a comment.


  Change #1035864 had a related patch set uploaded (by Xqt; author: Xqt):
  
  [pywikibot/core@master] Revert "[IMPR] use CachedRequest for userinfo 
requests"
  
  https://gerrit.wikimedia.org/r/1035864

TASK DETAIL
  https://phabricator.wikimedia.org/T348925

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Xqt, gerritbot
Cc: ericpien, Aklapper, pywikibot-bugs-list, Xqt, mevo, Susie413113, 
PotsdamLamb, Jyoo1011, JohnsonLee01, SHEKH, Dijkstra, Khutuck, Zkhalido, Aram, 
Viztor, Wenyi, Tbscho, MayS, Vali.matei, Mdupont, JJMC89, Dvorapa, Altostratus, 
Avicennasis, Wong128hk, mys_721tx, Dinoguy1000, jayvdb, Masti, Alchimista, Jay8g
___
pywikibot-bugs mailing list -- pywikibot-bugs@lists.wikimedia.org
To unsubscribe send an email to pywikibot-bugs-le...@lists.wikimedia.org

[Pywikipedia-bugs] [Maniphest] [Commented On] T348925: preload_sites.py script is too slow

2024-05-25 Thread gerritbot 
gerritbot added a comment.


  Change #1034059 **merged** by jenkins-bot:
  
  [pywikibot/core@master] [IMPR] use CachedRequest for userinfo requests
  
  https://gerrit.wikimedia.org/r/1034059

TASK DETAIL
  https://phabricator.wikimedia.org/T348925

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Xqt, gerritbot
Cc: ericpien, Aklapper, pywikibot-bugs-list, Xqt, mevo, Isabelladantes1983, 
Themindcoder, Adamm71, Hellket777, LisafBia6531, Susie413113, 786, Biggs657, 
PotsdamLamb, Jyoo1011, JohnsonLee01, Juan90264, SHEKH, Dijkstra, Alter-paule, 
Beast1978, Un1tY, Khutuck, Zkhalido, Hook696, Kent7301, Viztor, CucyNoiD, 
Wenyi, Gaboe420, Giuliamocci, Cpaulf30, Af420, Bsandipan, Tbscho, MayS, 
Vali.matei, Lewizho99, Mdupont, JJMC89, Maathavan, Dvorapa, Altostratus, 
Neuronton, Avicennasis, Wong128hk, mys_721tx, Dinoguy1000, jayvdb, Masti, 
Alchimista, Jay8g
___
pywikibot-bugs mailing list -- pywikibot-bugs@lists.wikimedia.org
To unsubscribe send an email to pywikibot-bugs-le...@lists.wikimedia.org

[Pywikipedia-bugs] [Maniphest] [Commented On] T348925: preload_sites.py script is too slow

2024-05-20 Thread Xqt 
Xqt added a comment.


  The regression was introduced with Pywikibot 8.1 (see above). Previous runs 
were 30 times faster for the first time and 300 times for the second. The patch 
above fastens up the second call but the api calls seems sequentiell instead of 
simultaneously which would be expected with concurrent.futures.

TASK DETAIL
  https://phabricator.wikimedia.org/T348925

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Xqt
Cc: ericpien, Aklapper, pywikibot-bugs-list, Xqt, mevo, Isabelladantes1983, 
Themindcoder, Adamm71, Jersione, Hellket777, LisafBia6531, Susie413113, 786, 
Biggs657, PotsdamLamb, Jyoo1011, JohnsonLee01, Juan90264, SHEKH, Dijkstra, 
Alter-paule, Beast1978, Un1tY, Khutuck, Zkhalido, Hook696, Kent7301, 
joker88john, Viztor, CucyNoiD, Wenyi, Gaboe420, Giuliamocci, Cpaulf30, Af420, 
Bsandipan, Tbscho, MayS, Vali.matei, Lewizho99, Mdupont, JJMC89, Maathavan, 
Dvorapa, Altostratus, Neuronton, Avicennasis, Wong128hk, mys_721tx, 
Dinoguy1000, jayvdb, Masti, Alchimista, Jay8g
___
pywikibot-bugs mailing list -- pywikibot-bugs@lists.wikimedia.org
To unsubscribe send an email to pywikibot-bugs-le...@lists.wikimedia.org

[Pywikipedia-bugs] [Maniphest] [Commented On] T348925: preload_sites.py script is too slow

2024-05-20 Thread gerritbot 
gerritbot added a comment.


  Change #1034059 had a related patch set uploaded (by Xqt; author: Xqt):
  
  [pywikibot/core@master] [IMPR] use CachedRequest for userinfo requests
  
  https://gerrit.wikimedia.org/r/1034059

TASK DETAIL
  https://phabricator.wikimedia.org/T348925

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: ericpien, Aklapper, pywikibot-bugs-list, Xqt, mevo, Susie413113, 
PotsdamLamb, Jyoo1011, JohnsonLee01, SHEKH, Dijkstra, Khutuck, Zkhalido, 
Viztor, Wenyi, Tbscho, MayS, Vali.matei, Mdupont, JJMC89, Dvorapa, Altostratus, 
Avicennasis, Wong128hk, mys_721tx, Dinoguy1000, jayvdb, Masti, Alchimista, Jay8g
___
pywikibot-bugs mailing list -- pywikibot-bugs@lists.wikimedia.org
To unsubscribe send an email to pywikibot-bugs-le...@lists.wikimedia.org

[Pywikipedia-bugs] [Maniphest] [Commented On] T348925: preload_sites.py script is too slow

2024-05-20 Thread Xqt 
Xqt added a comment.


  Hi @ericpien: I think `preload_sites` script is a good measurement. After 
collecting the `siteinfo` (and after Pywikibot 8.1 `userinfo`) the second call 
must much more faster. Thank you for your patch but I would suggest to use the 
already implemented CachedRequest 

 to cache the userinfo which is already used for `siteinfo`. The advantage is 
that parallel running task will also benefit from it. At userinfo 

 method the cached request can be implemented like
  
if not hasattr(self, '_userinfo'):
uirequest = self._request(
expiry=1,
parameters=dict(
action='query',
meta='userinfo',
uiprop='blockinfo|hasmsg|groups|rights|ratelimits',
formatversion=2,
)
)
uidata = uirequest.submit()
  
  The 2nd run needs only ~30 seconds with this patch compared to 8 minutes 
without it. Maybe we could add some runtime test like this to our test matrix.

TASK DETAIL
  https://phabricator.wikimedia.org/T348925

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Xqt
Cc: ericpien, Aklapper, pywikibot-bugs-list, Xqt, mevo, Susie413113, 
PotsdamLamb, Jyoo1011, JohnsonLee01, SHEKH, Dijkstra, Khutuck, Zkhalido, 
Viztor, Wenyi, Tbscho, MayS, Vali.matei, Mdupont, JJMC89, Dvorapa, Altostratus, 
Avicennasis, Wong128hk, mys_721tx, Dinoguy1000, jayvdb, Masti, Alchimista, Jay8g
___
pywikibot-bugs mailing list -- pywikibot-bugs@lists.wikimedia.org
To unsubscribe send an email to pywikibot-bugs-le...@lists.wikimedia.org

[Pywikipedia-bugs] [Maniphest] [Commented On] T348925: preload_sites.py script is too slow

2024-05-19 Thread ericpien 
ericpien added a comment.


  Hi @Xqt what tests would you recommend to ensure the change is not causing 
regression?
  
  I have the code change: 
https://gerrit.wikimedia.org/r/c/pywikibot/core/+/1033715 where userinfo is 
being cached and read based on family name and username matches.
  
  While testing, I noticed cookies are added to `http.cookie_jar` when the api 
call is actually made to retrieve userinfo but I am unsure if this is the 
cookie that is meant to be loaded in `preload_sites` script.

TASK DETAIL
  https://phabricator.wikimedia.org/T348925

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ericpien
Cc: ericpien, Aklapper, pywikibot-bugs-list, Xqt, mevo, Susie413113, 
PotsdamLamb, Jyoo1011, JohnsonLee01, SHEKH, Dijkstra, Khutuck, Zkhalido, 
Viztor, Wenyi, Tbscho, MayS, Vali.matei, Mdupont, JJMC89, Dvorapa, Altostratus, 
Avicennasis, Wong128hk, mys_721tx, Dinoguy1000, jayvdb, Masti, Alchimista, Jay8g
___
pywikibot-bugs mailing list -- pywikibot-bugs@lists.wikimedia.org
To unsubscribe send an email to pywikibot-bugs-le...@lists.wikimedia.org