jayvdb added a comment. @Tgr, I am not sure what we can or should do to help, but if you can give us some suggestions we might be able to help.
It is occurring all over the place, at any test, before or after the first login. To give you some idea of frequency, each test job performs about 40 tests, and each build includes two jobs : en.wp.beta and zh.wp.beta. So ~80 test methods are performed, and my guess is about 300 API calls are performed (We could add a API call counter if it helps). And as previously said, almost every test build fails, with one or both of the jobs experiencing this problem. So, my back of envelope guess is ~ 1 in 300 API calls fails. These exceptions are always the result of an API error with `code` = `'mwoauth-invalid-authorization'` and `info` = `The authorization headers in your request are not valid: Nonce already used: dddd`. We could include in the exception the API `servedby` value. But I am guessing that the https://phabricator.wikimedia.org/tag/operations/ team would have spotted the problem if it was a particular host more frequently causing the problem. There is some variation in the oauth logging: Some of the nose logs include requests_oauthlib.oauth1_auth: DEBUG: Signing request <PreparedRequest [GET]> using client <Client nonce=None, signature_method=HMAC-SHA1, realm=None, encoding=utf-8, timestamp=None, resource_owner_secret=****, decoding=utf-8, verifier=None, signature_type=AUTH_HEADER, rsa_key=None, resource_owner_key=28c3e24fb4e98a9ddffdf46b120f381e, client_secret=****, callback_uri=None, client_key=6ad8150a40735dac98d4f07826061cd1> Others are different: requests_oauthlib.oauth1_auth: DEBUG: Signing request <PreparedRequest [POST]> using client <Client resource_owner_secret=****, client_secret=****, realm=None, verifier=None, signature_method=HMAC-SHA1, decoding=utf-8, rsa_key=None, client_key=6ad8150a40735dac98d4f07826061cd1, signature_type=AUTH_HEADER, encoding=utf-8, resource_owner_key=28c3e24fb4e98a9ddffdf46b120f381e, nonce=None, callback_uri=None, timestamp=None> Nonce error before log in: https://travis-ci.org/wikimedia/pywikibot-core/jobs/77787307#L371 - Nonce failed https://travis-ci.org/wikimedia/pywikibot-core/jobs/77787307#L410 - log in But mostly it occurs after the log in (but this isnt unusual, as the login happens early in the test sequence) https://travis-ci.org/wikimedia/pywikibot-core/jobs/79135083#L440 - log in https://travis-ci.org/wikimedia/pywikibot-core/jobs/79135083#L672 - Nonce failed https://travis-ci.org/wikimedia/pywikibot-core/jobs/79135083#L763 - the API params which caused the problem were: b'titles=%E9%A6%96%E9%A1%B5&maxlag=5&action=query&prop=info&meta=userinfo&uiprop=blockinfo%7Chasmsg&inprop=protection&format=json&indexpageids=&continue= https://travis-ci.org/wikimedia/pywikibot-core/jobs/79188747#L440 https://travis-ci.org/wikimedia/pywikibot-core/jobs/79188747#L643 https://travis-ci.org/jayvdb/pywikibot-core/jobs/77788314#L549 https://travis-ci.org/jayvdb/pywikibot-core/jobs/77788314#L727 Another thing we could do is use https://pypi.python.org/pypi/nose-randomly , to re-order the tests randomly, but that doesnt have a high chance of helping flush out the problem, but will make logs less easy to compare between test builds. Along these lines, we could change were the login action occurs, moving it later in the sequence to see if it is related. Also, as this seems to be an intermittent problem, pywikibot could simply retry whenever we encounter `The authorization headers in your request are not valid: Nonce already used: `. If the pywikibot builds stop being red so frequently, we can assume that the problem is recoverable within a small timescale. TASK DETAIL https://phabricator.wikimedia.org/T109173 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jayvdb Cc: Tgr, greg, Aklapper, Luke081515, pywikibot-bugs-list, jayvdb, scfc, hashar _______________________________________________ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
