[Wikidata-bugs] [Maniphest] [Commented On] T178445: flapping monitoring for recommendation_api on scb
Gehel added a comment. I'm not sure why we have a retry in the first place. An exponential back-off would be good, or to honor the "Retry-After" HTTP header sent by WDQS on 429 errors.TASK DETAILhttps://phabricator.wikimedia.org/T178445EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: GehelCc: Joe, Volans, mobrovac, Smalyshev, Gehel, Stashbot, Aklapper, Dzahn, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, Avner, Zppix, debt, Jonas, FloNight, Xmlizer, Eevans, jkroll, Hardikj, Wikidata-bugs, Jdouglas, aude, Tobias1984, Capt_Swing, Manybubbles, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T178445: flapping monitoring for recommendation_api on scb
Gehel added a comment. Looking at logs in logstash, it seems we throttled Recommendation API only when WDQS was overwhelmed by another user. Throttling is done on overall request time and the service was most probably already not responding correctly, or in a reasonable time. My understanding at this point is that the flapping and alerting was reasonable. WDQS was returning 429 (throttling), but without throttling, it would probably have timed out. Forwarding x-client-ip might still make sense. Or sending a user-agent combining the current UA and the end user UA.TASK DETAILhttps://phabricator.wikimedia.org/T178445EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: GehelCc: mobrovac, Smalyshev, Gehel, Stashbot, Aklapper, Dzahn, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, Avner, Zppix, Volans, debt, Jonas, FloNight, Xmlizer, Eevans, jkroll, Hardikj, Wikidata-bugs, Jdouglas, aude, Tobias1984, Capt_Swing, Manybubbles, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T178445: flapping monitoring for recommendation_api on scb
mobrovac added a comment. In T178445#3692895, @Smalyshev wrote: @mobrovac what's the rate the requests are currently sent at? IIRC the limits we have are pretty generous, but depends on the use case of course. This is the part that I don't understand - the rate of the public end point is 0, which means that the Recommendation API service contacts WDQS on;y 4 times every 60 seconds due to the automatic monitoring script sending health-check requests to it, which doesn't seem like something that should trigger rate limiting. In T178445#3692899, @Gehel wrote: We could add an exception for specific user agent. Or have Recommendation API propagate some way to bucket by end user. This needs a little bit of thinking. The latter would be ideal. We could forward the client's IP via the x-client-ip header.TASK DETAILhttps://phabricator.wikimedia.org/T178445EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: mobrovacCc: mobrovac, Smalyshev, Gehel, Stashbot, Aklapper, Dzahn, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, Avner, Zppix, Volans, debt, Jonas, FloNight, Xmlizer, Eevans, jkroll, Hardikj, Wikidata-bugs, Jdouglas, aude, Tobias1984, Capt_Swing, Manybubbles, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T178445: flapping monitoring for recommendation_api on scb
Gehel added a comment. @mobrovac it is possible if we can identify internal traffic. The throttling we apply is bucketed by user agent / IP, so I suspect that all the recommendation API traffic end up in the same bucket. We could add an exception for specific user agent. Or have Recommendation API propagate some way to bucket by end user. This needs a little bit of thinking.TASK DETAILhttps://phabricator.wikimedia.org/T178445EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: GehelCc: mobrovac, Smalyshev, Gehel, Stashbot, Aklapper, Dzahn, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, Avner, Zppix, Volans, debt, Jonas, FloNight, Xmlizer, Eevans, jkroll, Hardikj, Wikidata-bugs, Jdouglas, aude, Tobias1984, Capt_Swing, Manybubbles, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T178445: flapping monitoring for recommendation_api on scb
Smalyshev added a comment. @mborovac what's the rate the requests are currently sent at? IIRC the limits we have are pretty generous, but depends on the use case of course.TASK DETAILhttps://phabricator.wikimedia.org/T178445EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: mobrovac, Smalyshev, Gehel, Stashbot, Aklapper, Dzahn, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, Avner, Zppix, Volans, debt, Jonas, FloNight, Xmlizer, Eevans, jkroll, Hardikj, Wikidata-bugs, Jdouglas, aude, Tobias1984, Capt_Swing, Manybubbles, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
