[Pywikipedia-bugs] [Maniphest] [Commented On] T364393: pywikibot does not handle properly 502 Server Error reading pages

[gerritbot]

gerritbot added a comment.


  Change #1029487 **merged** by jenkins-bot:
  
  [pywikibot/core@master] [fix] retry api request on ServerError
  
  https://gerrit.wikimedia.org/r/1029487

TASK DETAIL
  https://phabricator.wikimedia.org/T364393

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Xqt, gerritbot
Cc: JJMC89, matej_suchanek, Xqt, Aklapper, Masti, pywikibot-bugs-list, mevo, 
Isabelladantes1983, Themindcoder, Adamm71, Jersione, Hellket777, LisafBia6531, 
786, Biggs657, PotsdamLamb, Jyoo1011, JohnsonLee01, Juan90264, SHEKH, Dijkstra, 
Alter-paule, Beast1978, Un1tY, Khutuck, Zkhalido, Hook696, Kent7301, 
joker88john, Viztor, CucyNoiD, Wenyi, Gaboe420, Giuliamocci, Cpaulf30, Af420, 
Bsandipan, Tbscho, MayS, Lewizho99, Mdupont, Maathavan, Dvorapa, Altostratus, 
Neuronton, Avicennasis, mys_721tx, jayvdb, Alchimista
___
pywikibot-bugs mailing list -- pywikibot-bugs@lists.wikimedia.org
To unsubscribe send an email to pywikibot-bugs-le...@lists.wikimedia.org

[Pywikipedia-bugs] [Maniphest] [Commented On] T364393: pywikibot does not handle properly 502 Server Error reading pages

[Xqt]

Xqt added a comment.


  > I think so. But can we handle the timeouts in pw? As the other erros from 
API are handled with incremental delays and retry.
  
  We can only increase the socket_timeout value for real timeout exceptions. 
[1] Maybe within exception handling callback in http module. [3] For wikidata 
this should be recommended currently (somehow). [2] But this will not help vor 
ServerError exceptions. Maybe we can show the html response in such case (or 
log it) for further debugging also for upstream issues.
  
  [1] 
https://doc.wikimedia.org/pywikibot/master/api_ref/pywikibot.config.html#http-settings
  [2] 
https://doc.wikimedia.org/pywikibot/master/api_ref/pywikibot.site.html#pywikibot.site._generators.GeneratorsMixin.alllinks
  [3] 
https://doc.wikimedia.org/pywikibot/master/api_ref/pywikibot.comms.html#comms.http.error_handling_callback

TASK DETAIL
  https://phabricator.wikimedia.org/T364393

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Xqt
Cc: JJMC89, matej_suchanek, Xqt, Aklapper, Masti, pywikibot-bugs-list, mevo, 
Isabelladantes1983, Themindcoder, Adamm71, Jersione, Hellket777, LisafBia6531, 
786, Biggs657, PotsdamLamb, Jyoo1011, JohnsonLee01, Juan90264, SHEKH, Dijkstra, 
Alter-paule, Beast1978, Un1tY, Khutuck, Zkhalido, Hook696, Kent7301, 
joker88john, Viztor, CucyNoiD, Wenyi, Gaboe420, Giuliamocci, Cpaulf30, Af420, 
Bsandipan, Tbscho, MayS, Lewizho99, Mdupont, Maathavan, Dvorapa, Altostratus, 
Neuronton, Avicennasis, mys_721tx, jayvdb, Alchimista
___
pywikibot-bugs mailing list -- pywikibot-bugs@lists.wikimedia.org
To unsubscribe send an email to pywikibot-bugs-le...@lists.wikimedia.org

[Pywikipedia-bugs] [Maniphest] [Commented On] T364393: pywikibot does not handle properly 502 Server Error reading pages

[Masti]

Masti added a comment.


  In T364393#9783360 , @Xqt 
wrote:
  
  > @Masti: I assumed the -start:! tested it on several wikis but didn’t run 
into this issue. I made a patch and both server errors should lead to retry 
loops.
  >
  > I still think this has something to do with the database conversion of MW 
1.43: some api requests are very lame or fails with timeout or server error.
  
  I think so. But can we handle the timeouts in pw? As the other erros from API 
are handled with incremental delays and retry.

TASK DETAIL
  https://phabricator.wikimedia.org/T364393

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Xqt, Masti
Cc: JJMC89, matej_suchanek, Xqt, Aklapper, Masti, pywikibot-bugs-list, mevo, 
Isabelladantes1983, Themindcoder, Adamm71, Jersione, Hellket777, LisafBia6531, 
786, Biggs657, PotsdamLamb, Jyoo1011, JohnsonLee01, Juan90264, SHEKH, Dijkstra, 
Alter-paule, Beast1978, Un1tY, Khutuck, Zkhalido, Hook696, Kent7301, 
joker88john, Viztor, CucyNoiD, Wenyi, Gaboe420, Giuliamocci, Cpaulf30, Af420, 
Bsandipan, Tbscho, MayS, Lewizho99, Mdupont, Maathavan, Dvorapa, Altostratus, 
Neuronton, Avicennasis, mys_721tx, jayvdb, Alchimista
___
pywikibot-bugs mailing list -- pywikibot-bugs@lists.wikimedia.org
To unsubscribe send an email to pywikibot-bugs-le...@lists.wikimedia.org

[Pywikipedia-bugs] [Maniphest] [Commented On] T364393: pywikibot does not handle properly 502 Server Error reading pages

[Xqt]

Xqt added a comment.


  @Masti: I assumed the -start:! tested it on several wikis but didn’t run into 
this issue. I made a patch and both server errors should lead to retry loops.
  
  I still think this has something to do with the database conversion of MW 
1.43: some api requests are very lame or fails with timeout or server error.

TASK DETAIL
  https://phabricator.wikimedia.org/T364393

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Xqt
Cc: JJMC89, matej_suchanek, Xqt, Aklapper, Masti, pywikibot-bugs-list, mevo, 
Isabelladantes1983, Themindcoder, Adamm71, Jersione, Hellket777, LisafBia6531, 
786, Biggs657, PotsdamLamb, Jyoo1011, JohnsonLee01, Juan90264, SHEKH, Dijkstra, 
Alter-paule, Beast1978, Un1tY, Khutuck, Zkhalido, Hook696, Kent7301, 
joker88john, Viztor, CucyNoiD, Wenyi, Gaboe420, Giuliamocci, Cpaulf30, Af420, 
Bsandipan, Tbscho, MayS, Lewizho99, Mdupont, Maathavan, Dvorapa, Altostratus, 
Neuronton, Avicennasis, mys_721tx, jayvdb, Alchimista
___
pywikibot-bugs mailing list -- pywikibot-bugs@lists.wikimedia.org
To unsubscribe send an email to pywikibot-bugs-le...@lists.wikimedia.org

[Pywikipedia-bugs] [Maniphest] [Commented On] T364393: pywikibot does not handle properly 502 Server Error reading pages

[Masti]

Masti added a comment.


  In T364393#9782642 , @Xqt 
wrote:
  
  > @Masti: What was your pagegenerator option when running your script?
  
  -start:'!'
  
  this and similar scripts sometimes run OK. but crash from time to time.
  
  BTW: sometimes the server error is 503 like:
  ERROR: Traceback (most recent call last):
  
File "/home/masti/pw/core/pywikibot/data/api/_requests.py", line 684, in 
_http_request
  response = http.request(self.site, uri=uri,
File "/home/masti/pw/core/pywikibot/comms/http.py", line 283, in request
  r = fetch(baseuri, headers=headers, **kwargs)
File "/home/masti/pw/core/pywikibot/comms/http.py", line 457, in fetch
  callback(response)
File "/home/masti/pw/core/pywikibot/comms/http.py", line 354, in 
error_handling_callback
  raise ServerError(
  
  pywikibot.exceptions.ServerError: 503 Server Error: Service Unavailable
  
  Traceback (most recent call last):
  
File "/home/masti/pw/core/pwb.py", line 40, in 
  sys.exit(main())
File "/home/masti/pw/core/pwb.py", line 36, in main
  runpy.run_path(str(path), run_name='__main__')
File "/usr/lib/python3.10/runpy.py", line 289, in run_path
  return _run_module_code(code, init_globals, run_name,
File "/usr/lib/python3.10/runpy.py", line 96, in _run_module_code
  _run_code(code, mod_globals, init_globals,
File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
  exec(code, run_globals)
File "/home/masti/pw/core/pywikibot/scripts/wrapper.py", line 521, in 

  main()
File "/home/masti/pw/core/pywikibot/scripts/wrapper.py", line 505, in main
  if not execute():
File "/home/masti/pw/core/pywikibot/scripts/wrapper.py", line 492, in 
execute
  run_python_file(filename, script_args, module)
File "/home/masti/pw/core/pywikibot/scripts/wrapper.py", line 149, in 
run_python_file
  exec(compile(source, filename, 'exec', dont_inherit=True),
File "masti/ms-contains.py", line 433, in 
  main()
File "masti/ms-contains.py", line 427, in main
  bot.run()  # guess what it does
File "masti/ms-contains.py", line 147, in run
  for page in self.generator:
File "/home/masti/pw/core/pywikibot/pagegenerators/__init__.py", line 660, 
in PreloadingGenerator
  yield from site.preloadpages(group, groupsize=groupsize,
File "/home/masti/pw/core/pywikibot/site/_generators.py", line 202, in 
preloadpages
  for pagedata in rvgen:
File "/usr/lib/python3.10/_collections_abc.py", line 330, in __next__
  return self.send(None)
File "/home/masti/pw/core/pywikibot/tools/collections.py", line 279, in send
  return next(self._started_gen)
File "/home/masti/pw/core/pywikibot/data/api/_generators.py", line 781, in 
generator
  yield from super().generator
File "/home/masti/pw/core/pywikibot/data/api/_generators.py", line 607, in 
generator
  self.data = self.request.submit()
File "/home/masti/pw/core/pywikibot/data/api/_requests.py", line 993, in 
submit
  response, use_get = self._http_request(use_get, uri, body, headers,
File "/home/masti/pw/core/pywikibot/data/api/_requests.py", line 684, in 
_http_request
  response = http.request(self.site, uri=uri,
File "/home/masti/pw/core/pywikibot/comms/http.py", line 283, in request
  r = fetch(baseuri, headers=headers, **kwargs)
File "/home/masti/pw/core/pywikibot/comms/http.py", line 457, in fetch
  callback(response)
File "/home/masti/pw/core/pywikibot/comms/http.py", line 354, in 
error_handling_callback
  raise ServerError(
  
  pywikibot.exceptions.ServerError: 503 Server Error: Service Unavailable
  CRITICAL: Exiting due to uncaught exception ServerError: 503 Server Error: 
Service Unavailable

TASK DETAIL
  https://phabricator.wikimedia.org/T364393

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Xqt, Masti
Cc: JJMC89, matej_suchanek, Xqt, Aklapper, Masti, pywikibot-bugs-list, mevo, 
Isabelladantes1983, Themindcoder, Adamm71, Jersione, Hellket777, LisafBia6531, 
786, Biggs657, PotsdamLamb, Jyoo1011, JohnsonLee01, Juan90264, SHEKH, Dijkstra, 
Alter-paule, Beast1978, Un1tY, Khutuck, Zkhalido, Hook696, Kent7301, 
joker88john, Viztor, CucyNoiD, Wenyi, Gaboe420, Giuliamocci, Cpaulf30, Af420, 
Bsandipan, Tbscho, MayS, Lewizho99, Mdupont, Maathavan, Dvorapa, Altostratus, 
Neuronton, Avicennasis, mys_721tx, jayvdb, Alchimista
___
pywikibot-bugs mailing list -- pywikibot-bugs@lists.wikimedia.org
To unsubscribe send an email to pywikibot-bugs-le...@lists.wikimedia.org

[Pywikipedia-bugs] [Maniphest] [Commented On] T364393: pywikibot does not handle properly 502 Server Error reading pages

[gerritbot]

gerritbot added a comment.


  Change #1029487 had a related patch set uploaded (by Xqt; author: Xqt):
  
  [pywikibot/core@master] [fix] retry api request on ServerError
  
  https://gerrit.wikimedia.org/r/1029487

TASK DETAIL
  https://phabricator.wikimedia.org/T364393

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: JJMC89, matej_suchanek, Xqt, Aklapper, Masti, pywikibot-bugs-list, mevo, 
PotsdamLamb, Jyoo1011, JohnsonLee01, SHEKH, Dijkstra, Khutuck, Zkhalido, 
Viztor, Wenyi, Tbscho, MayS, Mdupont, Dvorapa, Altostratus, Avicennasis, 
mys_721tx, jayvdb, Alchimista
___
pywikibot-bugs mailing list -- pywikibot-bugs@lists.wikimedia.org
To unsubscribe send an email to pywikibot-bugs-le...@lists.wikimedia.org

[Pywikipedia-bugs] [Maniphest] [Commented On] T364393: pywikibot does not handle properly 502 Server Error reading pages

[Xqt]

Xqt added a comment.


  @Masti: What was your pagegenerator option when running your script?

TASK DETAIL
  https://phabricator.wikimedia.org/T364393

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Xqt
Cc: JJMC89, matej_suchanek, Xqt, Aklapper, Masti, pywikibot-bugs-list, mevo, 
PotsdamLamb, Jyoo1011, JohnsonLee01, SHEKH, Dijkstra, Khutuck, Zkhalido, 
Viztor, Wenyi, Tbscho, MayS, Mdupont, Dvorapa, Altostratus, Avicennasis, 
mys_721tx, jayvdb, Alchimista
___
pywikibot-bugs mailing list -- pywikibot-bugs@lists.wikimedia.org
To unsubscribe send an email to pywikibot-bugs-le...@lists.wikimedia.org

[Pywikipedia-bugs] [Maniphest] [Commented On] T364393: pywikibot does not handle properly 502 Server Error reading pages

[matej_suchanek]

matej_suchanek added a comment.


  I have been running into these lately, too. Especially when calling 
`DataSite.loadrevisions`.

TASK DETAIL
  https://phabricator.wikimedia.org/T364393

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: matej_suchanek
Cc: matej_suchanek, Xqt, Aklapper, Masti, pywikibot-bugs-list, mevo, 
PotsdamLamb, Jyoo1011, JohnsonLee01, SHEKH, Dijkstra, Khutuck, Zkhalido, 
Viztor, Wenyi, Tbscho, MayS, Mdupont, JJMC89, Dvorapa, Altostratus, 
Avicennasis, mys_721tx, jayvdb, Alchimista
___
pywikibot-bugs mailing list -- pywikibot-bugs@lists.wikimedia.org
To unsubscribe send an email to pywikibot-bugs-le...@lists.wikimedia.org

[Pywikipedia-bugs] [Maniphest] [Commented On] T364393: pywikibot does not handle properly 502 Server Error reading pages

[Xqt]

Xqt added a comment.


  Thank you for this investigation.

TASK DETAIL
  https://phabricator.wikimedia.org/T364393

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Xqt
Cc: Xqt, Aklapper, Masti, pywikibot-bugs-list, mevo, PotsdamLamb, Jyoo1011, 
JohnsonLee01, SHEKH, Dijkstra, Khutuck, Zkhalido, Viztor, Wenyi, Tbscho, MayS, 
Mdupont, JJMC89, Dvorapa, Altostratus, Avicennasis, mys_721tx, jayvdb, 
Alchimista
___
pywikibot-bugs mailing list -- pywikibot-bugs@lists.wikimedia.org
To unsubscribe send an email to pywikibot-bugs-le...@lists.wikimedia.org

[Pywikipedia-bugs] [Maniphest] [Commented On] T364393: pywikibot does not handle properly 502 Server Error reading pages

[Masti]

Masti added a comment.


  I increased the timeout to 120s but it crashed again with same error.

TASK DETAIL
  https://phabricator.wikimedia.org/T364393

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Masti
Cc: Xqt, Aklapper, Masti, pywikibot-bugs-list, mevo, PotsdamLamb, Jyoo1011, 
JohnsonLee01, SHEKH, Dijkstra, Khutuck, Zkhalido, Viztor, Wenyi, Tbscho, MayS, 
Mdupont, JJMC89, Dvorapa, Altostratus, Avicennasis, mys_721tx, jayvdb, 
Alchimista
___
pywikibot-bugs mailing list -- pywikibot-bugs@lists.wikimedia.org
To unsubscribe send an email to pywikibot-bugs-le...@lists.wikimedia.org

[Pywikipedia-bugs] [Maniphest] [Commented On] T364393: pywikibot does not handle properly 502 Server Error reading pages

[Masti]

Masti added a comment.


  I will test and come back with the answer. Unfortunately it is not easy 
replicated so one have to wait till it happens.

TASK DETAIL
  https://phabricator.wikimedia.org/T364393

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Masti
Cc: Xqt, Aklapper, Masti, pywikibot-bugs-list, mevo, PotsdamLamb, Jyoo1011, 
JohnsonLee01, SHEKH, Dijkstra, Khutuck, Zkhalido, Viztor, Wenyi, Tbscho, MayS, 
Mdupont, JJMC89, Dvorapa, Altostratus, Avicennasis, mys_721tx, jayvdb, 
Alchimista
___
pywikibot-bugs mailing list -- pywikibot-bugs@lists.wikimedia.org
To unsubscribe send an email to pywikibot-bugs-le...@lists.wikimedia.org

[Pywikipedia-bugs] [Maniphest] [Commented On] T364393: pywikibot does not handle properly 502 Server Error reading pages

[Xqt]

Xqt added a comment.


  Does it helps to increase the socket timeout (read timeout i.e. the second 
value of that tuple)?
  
https://doc.wikimedia.org/pywikibot/master/api_ref/pywikibot.config.html#http-settings

TASK DETAIL
  https://phabricator.wikimedia.org/T364393

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Xqt
Cc: Xqt, Aklapper, Masti, pywikibot-bugs-list, mevo, PotsdamLamb, Jyoo1011, 
JohnsonLee01, SHEKH, Dijkstra, Khutuck, Zkhalido, Viztor, Wenyi, Tbscho, MayS, 
Mdupont, JJMC89, Dvorapa, Altostratus, Avicennasis, mys_721tx, jayvdb, 
Alchimista
___
pywikibot-bugs mailing list -- pywikibot-bugs@lists.wikimedia.org
To unsubscribe send an email to pywikibot-bugs-le...@lists.wikimedia.org