[Pywikipedia-bugs] [Maniphest] [Commented On] T364393: pywikibot does not handle properly 502 Server Error reading pages

2024-05-07 Thread Xqt 
Xqt added a comment.


  Thank you for this investigation.

TASK DETAIL
  https://phabricator.wikimedia.org/T364393

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Xqt
Cc: Xqt, Aklapper, Masti, pywikibot-bugs-list, mevo, PotsdamLamb, Jyoo1011, 
JohnsonLee01, SHEKH, Dijkstra, Khutuck, Zkhalido, Viztor, Wenyi, Tbscho, MayS, 
Mdupont, JJMC89, Dvorapa, Altostratus, Avicennasis, mys_721tx, jayvdb, 
Alchimista
___
pywikibot-bugs mailing list -- pywikibot-bugs@lists.wikimedia.org
To unsubscribe send an email to pywikibot-bugs-le...@lists.wikimedia.org

[Pywikipedia-bugs] [Maniphest] [Commented On] T364393: pywikibot does not handle properly 502 Server Error reading pages

2024-05-07 Thread Masti 
Masti added a comment.


  I increased the timeout to 120s but it crashed again with same error.

TASK DETAIL
  https://phabricator.wikimedia.org/T364393

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Masti
Cc: Xqt, Aklapper, Masti, pywikibot-bugs-list, mevo, PotsdamLamb, Jyoo1011, 
JohnsonLee01, SHEKH, Dijkstra, Khutuck, Zkhalido, Viztor, Wenyi, Tbscho, MayS, 
Mdupont, JJMC89, Dvorapa, Altostratus, Avicennasis, mys_721tx, jayvdb, 
Alchimista
___
pywikibot-bugs mailing list -- pywikibot-bugs@lists.wikimedia.org
To unsubscribe send an email to pywikibot-bugs-le...@lists.wikimedia.org

[Pywikipedia-bugs] [Maniphest] [Commented On] T364393: pywikibot does not handle properly 502 Server Error reading pages

2024-05-07 Thread Masti 
Masti added a comment.


  I will test and come back with the answer. Unfortunately it is not easy 
replicated so one have to wait till it happens.

TASK DETAIL
  https://phabricator.wikimedia.org/T364393

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Masti
Cc: Xqt, Aklapper, Masti, pywikibot-bugs-list, mevo, PotsdamLamb, Jyoo1011, 
JohnsonLee01, SHEKH, Dijkstra, Khutuck, Zkhalido, Viztor, Wenyi, Tbscho, MayS, 
Mdupont, JJMC89, Dvorapa, Altostratus, Avicennasis, mys_721tx, jayvdb, 
Alchimista
___
pywikibot-bugs mailing list -- pywikibot-bugs@lists.wikimedia.org
To unsubscribe send an email to pywikibot-bugs-le...@lists.wikimedia.org

[Pywikipedia-bugs] [Maniphest] [Commented On] T364393: pywikibot does not handle properly 502 Server Error reading pages

2024-05-07 Thread Xqt 
Xqt added a comment.


  Does it helps to increase the socket timeout (read timeout i.e. the second 
value of that tuple)?
  
https://doc.wikimedia.org/pywikibot/master/api_ref/pywikibot.config.html#http-settings

TASK DETAIL
  https://phabricator.wikimedia.org/T364393

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Xqt
Cc: Xqt, Aklapper, Masti, pywikibot-bugs-list, mevo, PotsdamLamb, Jyoo1011, 
JohnsonLee01, SHEKH, Dijkstra, Khutuck, Zkhalido, Viztor, Wenyi, Tbscho, MayS, 
Mdupont, JJMC89, Dvorapa, Altostratus, Avicennasis, mys_721tx, jayvdb, 
Alchimista
___
pywikibot-bugs mailing list -- pywikibot-bugs@lists.wikimedia.org
To unsubscribe send an email to pywikibot-bugs-le...@lists.wikimedia.org

[Pywikipedia-bugs] [Maniphest] [Created] T364393: pywikibot does not handle properly 502 Server Error reading pages

2024-05-07 Thread Masti 
Masti created this task.
Masti added a project: Pywikibot.
Restricted Application added subscribers: pywikibot-bugs-list, Aklapper.

TASK DESCRIPTION
  while running pywikibot on pl.wikipedia I quite often encounter code crashes 
after page generator encounters 502 Server Error from wikimedia API
  
  **What happens?**:
  script throws exception and crashes
  Retrieving 50 pages from wikipedia:pl.
  ERROR: Traceback (most recent call last):
  
File "/home/masti/pw/core/pywikibot/data/api/_requests.py", line 684, in 
_http_request
  response = http.request(self.site, uri=uri,
File "/home/masti/pw/core/pywikibot/comms/http.py", line 283, in request
  r = fetch(baseuri, headers=headers, **kwargs)
File "/home/masti/pw/core/pywikibot/comms/http.py", line 457, in fetch
  callback(response)
File "/home/masti/pw/core/pywikibot/comms/http.py", line 354, in 
error_handling_callback
  raise ServerError(
  
  pywikibot.exceptions.ServerError: 502 Server Error: Server Hangup
  
  Traceback (most recent call last):
  
File "/home/masti/pw/core/pwb.py", line 40, in 
  sys.exit(main())
File "/home/masti/pw/core/pwb.py", line 36, in main
  runpy.run_path(str(path), run_name='__main__')
File "/usr/lib/python3.10/runpy.py", line 289, in run_path
  return _run_module_code(code, init_globals, run_name,
File "/usr/lib/python3.10/runpy.py", line 96, in _run_module_code
  _run_code(code, mod_globals, init_globals,
File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
  exec(code, run_globals)
File "/home/masti/pw/core/pywikibot/scripts/wrapper.py", line 521, in 

  main()
File "/home/masti/pw/core/pywikibot/scripts/wrapper.py", line 505, in main
  if not execute():
File "/home/masti/pw/core/pywikibot/scripts/wrapper.py", line 492, in 
execute
  run_python_file(filename, script_args, module)
File "/home/masti/pw/core/pywikibot/scripts/wrapper.py", line 149, in 
run_python_file
  exec(compile(source, filename, 'exec', dont_inherit=True),
File "masti/ms-contains.py", line 433, in 
  main()
File "masti/ms-contains.py", line 427, in main
  bot.run()  # guess what it does
File "masti/ms-contains.py", line 147, in run
  for page in self.generator:
File "/home/masti/pw/core/pywikibot/pagegenerators/__init__.py", line 660, 
in PreloadingGenerator
  yield from site.preloadpages(group, groupsize=groupsize,
File "/home/masti/pw/core/pywikibot/site/_generators.py", line 202, in 
preloadpages
  for pagedata in rvgen:
File "/usr/lib/python3.10/_collections_abc.py", line 330, in __next__
  return self.send(None)
File "/home/masti/pw/core/pywikibot/tools/collections.py", line 279, in send
  return next(self._started_gen)
File "/home/masti/pw/core/pywikibot/data/api/_generators.py", line 781, in 
generator
  yield from super().generator
File "/home/masti/pw/core/pywikibot/data/api/_generators.py", line 607, in 
generator
  self.data = self.request.submit()
File "/home/masti/pw/core/pywikibot/data/api/_requests.py", line 993, in 
submit
  response, use_get = self._http_request(use_get, uri, body, headers,
File "/home/masti/pw/core/pywikibot/data/api/_requests.py", line 684, in 
_http_request
  response = http.request(self.site, uri=uri,
File "/home/masti/pw/core/pywikibot/comms/http.py", line 283, in request
  r = fetch(baseuri, headers=headers, **kwargs)
File "/home/masti/pw/core/pywikibot/comms/http.py", line 457, in fetch
  callback(response)
File "/home/masti/pw/core/pywikibot/comms/http.py", line 354, in 
error_handling_callback
  raise ServerError(
  
  pywikibot.exceptions.ServerError: 502 Server Error: Server Hangup
  CRITICAL: Exiting due to uncaught exception ServerError: 502 Server Error: 
Server Hangup
  
  **What should have happened instead?**:
  wait for server to come back as the errors are temporary
  
  **Software version** (on `Special:Version` page; skip for WMF-hosted wikis 
like Wikipedia):
  Pywikibot: [https] masti01-pywikibot.git (04383ba 
,
 g18666, 2024/05/06, 16:04:47, master)
  Release version: 9.2.0.dev2
  packaging version: 24.0
  mwparserfromhell version: 0.6.6
  wikitextparser version: n/a
  requests version: 2.31.0
  
cacerts: 
/home/masti/pw/core/venv/lib/python3.10/site-packages/certifi/cacert.pem
  certificate test: ok
  
  Python: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]
  
  **Other information** (browser name/version, screenshots, etc.):

TASK DETAIL
  https://phabricator.wikimedia.org/T364393

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Masti
Cc: Aklapper, Masti, pywikibot-bugs-list, mevo, PotsdamLamb, Jyoo1011, 
JohnsonLee01, SHEKH, Dijkstra, Khutuck, Zkhalido, Viztor, Wenyi, Tbscho, MayS, 
Mdupont, JJMC89, Dvorapa, Altostratus, Avicennasis,