[Wikidata-bugs] [Maniphest] [Commented On] T240442: Design a continuous throttling policy for Wikidata bots

tfmorris added a comment. In T240442#5851541 , @Addshore wrote: > In T240442#5834866 , @Ladsgroup wrote: > >> Very broad idea, feel free to discard, I think using industry-wide

[Wikidata-bugs] [Maniphest] [Commented On] T240442: Design a continuous throttling policy for Wikidata bots

Addshore added a comment. In T240442#5945682 , @Ladsgroup wrote: > What do you think? Definitely worth considering. Could be worth an RFC to get wider involvement? This is essentially edit rate limiting for an entire site.

[Wikidata-bugs] [Maniphest] [Commented On] T240442: Design a continuous throttling policy for Wikidata bots

Ladsgroup added a comment. I have an idea. I think we should use PoolCounter (which is basically a SaaS, Semaphore as a service) to put a cap on edits happening on wikidata at the same time. This is being used when an article is being reparsed as

[Wikidata-bugs] [Maniphest] [Commented On] T240442: Design a continuous throttling policy for Wikidata bots

Xqt added a comment. In addition: should read access also be throttled? TASK DETAIL https://phabricator.wikimedia.org/T240442 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Xqt Cc: Strainu, Xqt, Dvorapa, Ladsgroup, ArthurPSmith, Addshore,

[Wikidata-bugs] [Maniphest] [Commented On] T240442: Design a continuous throttling policy for Wikidata bots

Addshore added a comment. In T240442#5834866 , @Ladsgroup wrote: > Very broad idea, feel free to discard, I think using industry-wide standards for throttling like `token bucket`, `leaky bucket`, `fixed-window counter` or

[Wikidata-bugs] [Maniphest] [Commented On] T240442: Design a continuous throttling policy for Wikidata bots

Ladsgroup added a comment. Very broad idea, feel free to discard, I think using industry-wide standards for throttling like `token bucket`, `leaky bucket`, `fixed-window counter` or `sliding-window counter` might help here. TASK DETAIL https://phabricator.wikimedia.org/T240442 EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T240442: Design a continuous throttling policy for Wikidata bots

Addshore added a comment. It's possible that we could add some sort of suggested wait between actions to the output of maxlag, if that could make things easier. It would avoid individuals trying to figure out how long to wait.. That's kind of what maxlag is, the time that you should

[Wikidata-bugs] [Maniphest] [Commented On] T240442: Design a continuous throttling policy for Wikidata bots

Pintoch added a comment. It is actually possible to retrieve the current maxlag value from the API without making any edit (see @Addshore's comment above). So, just retrieve the current maxlag value and compute your desired edit rate for this maxlag with the function plotted above. Then

[Wikidata-bugs] [Maniphest] [Commented On] T240442: Design a continuous throttling policy for Wikidata bots

ArthurPSmith added a comment. Just saw this - I'm wondering technically how you would implement it? You could generate a random number between 2.5 and 5, and if maxlag is greater than your random number deny the edit? TASK DETAIL https://phabricator.wikimedia.org/T240442 EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T240442: Design a continuous throttling policy for Wikidata bots

Pintoch added a comment. Thanks! I think dynamically changing the maxlag value is likely to still introduce some thresholds, whereas a continuous slowdown (by retrieving the lag and compute one's edit rate based on it) should in theory reach an equilibrium point. In the meantime,

[Wikidata-bugs] [Maniphest] [Commented On] T240442: Design a continuous throttling policy for Wikidata bots

Addshore added a comment. As reported in IRC, maxlag can be checked with, for example, https://www.wikidata.org/w/api.php?action=query=json=-1 Client could also consider dynamically changing their maxlag value, rather than always having it set to 5. TASK DETAIL

[Wikidata-bugs] [Maniphest] [Commented On] T240442: Design a continuous throttling policy for Wikidata bots

Pintoch added a comment. If clients are able to retrieve the current lag periodically (through some MediaWiki API call? which one?), then this should not require any server-side change. Clients can continue to use `maxlag=5` but to also throttle themselves using the smoothed function