[Pywikipedia-bugs] [Maniphest] [Commented On] T261050: Frequent "Invalid CSRF token" errors on Wikimedia Commons using Pywikibot since August 2020

[Xqt]

Xqt added a comment.


  @Multichill: do you have any tracebacks for investigation?

TASK DETAIL
  https://phabricator.wikimedia.org/T261050

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Xqt
Cc: matej_suchanek, daniel, eprodromou, Tgr, Dvorapa, Xqt, Aklapper, Reedy, 
pywikibot-bugs-list, Multichill, JohnsonLee01, SHEKH, Naike, Dijkstra, CBogen, 
Khutuck, Zkhalido, WDoranWMF, Viztor, DannyS712, Wenyi, Amorymeltzer, 
Ramsey-WMF, Tbscho, MayS, Sethakill, Mdupont, JJMC89, dg711, Poyekhali, 
Agabi10, Altostratus, Taiwania_Justo, Avicennasis, Pchelolo, Ixocactus, 
Wong128hk, mys_721tx, Bawolff, El_Grafo, Dinoguy1000, jayvdb, Anomie, Masti, 
Alchimista, Steinsplitter, Rxy, Keegan, Legoktm
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T261050: Frequent "Invalid CSRF token" errors on Wikimedia Commons using Pywikibot since August 2020

[Tgr]

Tgr added a comment.


  Tokens never expire, but the session that holds them can. That behavior 
changed recently with the switch to Kask (less recently than August, though): 
before we had theoretically-infinite sessions which could get evicted, now we 
have a set session lifetime (1 day, IIRC?).
  
  The session also gets terminted when authentication fails, so possibly the 
bot actually gets logged out and the CSRF error is an effect of that.

TASK DETAIL
  https://phabricator.wikimedia.org/T261050

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Tgr
Cc: matej_suchanek, daniel, eprodromou, Tgr, Dvorapa, Xqt, Aklapper, Reedy, 
pywikibot-bugs-list, Multichill, JohnsonLee01, SHEKH, Naike, Dijkstra, CBogen, 
Khutuck, Zkhalido, WDoranWMF, Viztor, DannyS712, Wenyi, Amorymeltzer, 
Ramsey-WMF, Tbscho, MayS, Sethakill, Mdupont, JJMC89, dg711, Poyekhali, 
Agabi10, Altostratus, Taiwania_Justo, Avicennasis, Pchelolo, Ixocactus, 
Wong128hk, mys_721tx, Bawolff, El_Grafo, Dinoguy1000, jayvdb, Anomie, Masti, 
Alchimista, Steinsplitter, Rxy, Keegan, Legoktm
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T261050: Frequent "Invalid CSRF token" errors on Wikimedia Commons using Pywikibot since August 2020

[Multichill]

Multichill added a comment.


  I think this happens:
  
  - Bot fires up and gets some tokens
  - Bot does a lot of edits all with the same token
  - For some reason after a long time the token is not valid anymore
  - The bot continues to try to edit with the old invalid token instead of 
getting a new one
  
  How long is a token supposed to be valid? Forever? The Pywikibot code seems 
to assume that it can use the same token for all edits in a one run. One run 
can take days or weeks. As a work around I now force the site to get a new 
token when I run into a problem. That seems to be stable.
  
  So I think Pywikibot needs to be updated in the framework itself that if it 
runs into a token error, the a new token gets retrieved, updated in the token 
cache and the edit should be retried. Agree @Xqt ?
  
  Most relevant code is in 
https://github.com/wikimedia/pywikibot/blob/master/pywikibot/site/__init__.py 
btw.

TASK DETAIL
  https://phabricator.wikimedia.org/T261050

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Multichill
Cc: matej_suchanek, daniel, Krinkle, eprodromou, Tgr, Dvorapa, Xqt, Aklapper, 
Reedy, pywikibot-bugs-list, Multichill, JohnsonLee01, SHEKH, Naike, Dijkstra, 
CBogen, Khutuck, Zkhalido, WDoranWMF, Viztor, DannyS712, Wenyi, Amorymeltzer, 
Ramsey-WMF, Tbscho, MayS, Sethakill, Mdupont, JJMC89, dg711, Poyekhali, 
Agabi10, Altostratus, Taiwania_Justo, Avicennasis, Pchelolo, Ixocactus, 
Wong128hk, mys_721tx, Bawolff, El_Grafo, Dinoguy1000, jayvdb, Anomie, Masti, 
Alchimista, Steinsplitter, Rxy, Keegan, Legoktm
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T261050: Frequent "Invalid CSRF token" errors on Wikimedia Commons using Pywikibot since August 2020

[Xqt]

Xqt added a comment.


  site.tokens property is the container object which holds tokens. It uses 
site.get_tokens tonlosere tokens.

TASK DETAIL
  https://phabricator.wikimedia.org/T261050

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Xqt
Cc: matej_suchanek, daniel, Krinkle, eprodromou, Tgr, Dvorapa, Xqt, Aklapper, 
Reedy, pywikibot-bugs-list, Multichill, JohnsonLee01, SHEKH, Naike, Dijkstra, 
CBogen, Khutuck, Zkhalido, WDoranWMF, Viztor, DannyS712, Wenyi, Amorymeltzer, 
Ramsey-WMF, Tbscho, MayS, Sethakill, Mdupont, JJMC89, dg711, Poyekhali, 
Agabi10, Altostratus, Taiwania_Justo, Avicennasis, Pchelolo, Ixocactus, 
Wong128hk, mys_721tx, Bawolff, El_Grafo, Dinoguy1000, jayvdb, Anomie, Masti, 
Alchimista, Steinsplitter, Rxy, Keegan, Legoktm
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T261050: Frequent "Invalid CSRF token" errors on Wikimedia Commons using Pywikibot since August 2020

[Multichill]

Multichill added a comment.


  I think I found what is going wrong. site.get_tokens(['csrf']) gets you a new 
token, but doesn't update the internal token state:
  
  >>> import pywikibot
  >>> site = pywikibot.Site(u'commons', u'commons')
  >>> print (site.tokens['csrf'])
  
  1ade99bef6b895e188692438b12cbd9e5f4aacac+\
  
  >>> site.get_tokens(['csrf'])
  
  {'csrf': '113400e3b87ae1d1d53ec4ebee9bde105f4aacb8+\\'}
  
  >>> print (site.tokens['csrf'])
  
  1ade99bef6b895e188692438b12cbd9e5f4aacac+\
  
  Looks like a site.tokens.load_tokens(['csrf']) does refresh the token, all 
quite confusing.

TASK DETAIL
  https://phabricator.wikimedia.org/T261050

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Multichill
Cc: matej_suchanek, daniel, Krinkle, eprodromou, Tgr, Dvorapa, Xqt, Aklapper, 
Reedy, pywikibot-bugs-list, Multichill, JohnsonLee01, SHEKH, Naike, Dijkstra, 
CBogen, Khutuck, Zkhalido, WDoranWMF, Viztor, DannyS712, Wenyi, Amorymeltzer, 
Ramsey-WMF, Tbscho, MayS, Sethakill, Mdupont, JJMC89, dg711, Poyekhali, 
Agabi10, Altostratus, Taiwania_Justo, Avicennasis, Pchelolo, Ixocactus, 
Wong128hk, mys_721tx, Bawolff, El_Grafo, Dinoguy1000, jayvdb, Anomie, Masti, 
Alchimista, Steinsplitter, Rxy, Keegan, Legoktm
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T261050: Frequent "Invalid CSRF token" errors on Wikimedia Commons using Pywikibot since August 2020

[Multichill]

Multichill added a comment.


  In T261050#6414369 , @Tgr 
wrote:
  
  > If the error is easy enough to reproduce, you could try setting the 
X-Wikimedia-Debug  header 
to get some pretty detailed logs.
  
  No, it's not, it seems to be intermittent. I added a bit more debugging code 
to my bot. When it hits the exception, it will print the current token, grab a 
new one and print the new one. The new and old tokens are the same so either 
the bot is not actually getting a new token or the api is returning the same 
one. I'll investigate a bit more.

TASK DETAIL
  https://phabricator.wikimedia.org/T261050

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Multichill
Cc: matej_suchanek, daniel, Krinkle, eprodromou, Tgr, Dvorapa, Xqt, Aklapper, 
Reedy, pywikibot-bugs-list, Multichill, JohnsonLee01, SHEKH, Naike, Dijkstra, 
CBogen, Khutuck, Zkhalido, WDoranWMF, Viztor, DannyS712, Wenyi, Amorymeltzer, 
Ramsey-WMF, Tbscho, MayS, Sethakill, Mdupont, JJMC89, dg711, Poyekhali, 
Agabi10, Altostratus, Taiwania_Justo, Avicennasis, Pchelolo, Ixocactus, 
Wong128hk, mys_721tx, Bawolff, El_Grafo, Dinoguy1000, jayvdb, Anomie, Masti, 
Alchimista, Steinsplitter, Rxy, Keegan, Legoktm
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T261050: Frequent "Invalid CSRF token" errors on Wikimedia Commons using Pywikibot since August 2020

[matej_suchanek]

matej_suchanek added a comment.


  https://travis-ci.org/github/wikimedia/pywikibot/jobs/721977220 (large page) 
got this warning once and then on every next attempt:
  
WARNING: API error badtoken: Invalid CSRF token.
ERROR: Received incorrect login token. Forcing re-login.

TASK DETAIL
  https://phabricator.wikimedia.org/T261050

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: matej_suchanek
Cc: matej_suchanek, daniel, Krinkle, eprodromou, Tgr, Dvorapa, Xqt, Aklapper, 
Reedy, pywikibot-bugs-list, Multichill, JohnsonLee01, SHEKH, Naike, Dijkstra, 
CBogen, Khutuck, Zkhalido, WDoranWMF, Viztor, DannyS712, Wenyi, Amorymeltzer, 
Ramsey-WMF, Tbscho, MayS, Sethakill, Mdupont, JJMC89, dg711, Poyekhali, 
Agabi10, Altostratus, Taiwania_Justo, Avicennasis, Pchelolo, Ixocactus, 
Wong128hk, mys_721tx, Bawolff, El_Grafo, Dinoguy1000, jayvdb, Anomie, Masti, 
Alchimista, Steinsplitter, Rxy, Keegan, Legoktm
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T261050: Frequent "Invalid CSRF token" errors on Wikimedia Commons using Pywikibot since August 2020

[Tgr]

Tgr added a comment.


  If the error is easy enough to reproduce, you could try setting the 
X-Wikimedia-Debug  header 
to get some pretty detailed logs.

TASK DETAIL
  https://phabricator.wikimedia.org/T261050

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Tgr
Cc: daniel, Krinkle, eprodromou, Tgr, Dvorapa, Xqt, Aklapper, Reedy, 
pywikibot-bugs-list, Multichill, JohnsonLee01, SHEKH, Naike, Dijkstra, CBogen, 
Khutuck, Zkhalido, WDoranWMF, Viztor, DannyS712, Wenyi, Amorymeltzer, 
Ramsey-WMF, Tbscho, MayS, Sethakill, Mdupont, JJMC89, dg711, Poyekhali, 
Agabi10, Altostratus, Taiwania_Justo, Avicennasis, Pchelolo, Ixocactus, 
Wong128hk, mys_721tx, Bawolff, El_Grafo, Dinoguy1000, jayvdb, Anomie, Masti, 
Alchimista, Steinsplitter, Rxy, Keegan, Legoktm
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T261050: Frequent "Invalid CSRF token" errors on Wikimedia Commons using Pywikibot since August 2020

[Multichill]

Multichill added a comment.


  In T261050#6410732 , @Tgr 
wrote:
  
  > What login method are you using?
  
  Classic login
  
  > In T261050#6410502 , 
@daniel wrote:
  >
  >> Pinging @krinkle in case it's a main stash thing.
  >
  > The main stash is not involved with CSRF tokens, it's either a 
SessionManager thing or a Kask thing (assuming it's on our side).
  
  I was unable to find any relevant logging in logstash.

TASK DETAIL
  https://phabricator.wikimedia.org/T261050

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Multichill
Cc: daniel, Krinkle, eprodromou, Tgr, Dvorapa, Xqt, Aklapper, Reedy, 
pywikibot-bugs-list, Multichill, JohnsonLee01, SHEKH, Naike, Dijkstra, CBogen, 
Khutuck, Zkhalido, WDoranWMF, Viztor, DannyS712, Wenyi, Amorymeltzer, 
Ramsey-WMF, Tbscho, MayS, Sethakill, Mdupont, JJMC89, dg711, Poyekhali, 
Agabi10, Altostratus, Taiwania_Justo, Avicennasis, Pchelolo, Ixocactus, 
Wong128hk, mys_721tx, Bawolff, El_Grafo, Dinoguy1000, jayvdb, Anomie, Masti, 
Alchimista, Steinsplitter, Rxy, Keegan, Legoktm
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T261050: Frequent "Invalid CSRF token" errors on Wikimedia Commons using Pywikibot since August 2020

[Tgr]

Tgr added a comment.


  What login method are you using?
  
  In T261050#6410502 , 
@daniel wrote:
  
  > Pinging @krinkle in case it's a main stash thing.
  
  The main stash is not involved with CSRF tokens, it's either a SessionManager 
thing or a Kask thing (assuming it's on our side).

TASK DETAIL
  https://phabricator.wikimedia.org/T261050

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Tgr
Cc: daniel, Krinkle, eprodromou, Tgr, Dvorapa, Xqt, Aklapper, Reedy, 
pywikibot-bugs-list, Multichill, JohnsonLee01, SHEKH, Naike, Dijkstra, CBogen, 
Khutuck, Zkhalido, WDoranWMF, Viztor, DannyS712, Wenyi, Amorymeltzer, 
Ramsey-WMF, Tbscho, MayS, Sethakill, Mdupont, JJMC89, dg711, Poyekhali, 
Agabi10, Altostratus, Taiwania_Justo, Avicennasis, Pchelolo, Ixocactus, 
Wong128hk, mys_721tx, Bawolff, El_Grafo, Dinoguy1000, jayvdb, Anomie, Masti, 
Alchimista, Steinsplitter, Rxy, Keegan, Legoktm
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T261050: Frequent "Invalid CSRF token" errors on Wikimedia Commons using Pywikibot since August 2020

[Multichill]

Multichill added a comment.


  @Xqt what is the correct way to force the bot to get a new token? Just 
site.get_tokens() or something else? I noticed some of my bots now stuck with 
100s of errors in a row.

TASK DETAIL
  https://phabricator.wikimedia.org/T261050

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Multichill
Cc: Dvorapa, Xqt, Aklapper, Reedy, pywikibot-bugs-list, Multichill, 
JohnsonLee01, SHEKH, Dijkstra, CBogen, Khutuck, Zkhalido, WDoranWMF, 
holger.knust, EvanProdromou, Viztor, DannyS712, Wenyi, Amorymeltzer, 
Ramsey-WMF, Tbscho, MayS, Sethakill, Mdupont, JJMC89, dg711, Poyekhali, 
Agabi10, Altostratus, Taiwania_Justo, Avicennasis, Pchelolo, Ixocactus, 
Wong128hk, mys_721tx, Bawolff, El_Grafo, Dinoguy1000, jayvdb, Anomie, Masti, 
Alchimista, Steinsplitter, Rxy, Keegan, Legoktm
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs