[Wikidata-bugs] [Maniphest] [Commented On] T159618: Job queue rising to nearly 3 million jobs
Betacommand added a comment. In T159618#3080947, @Legoktm wrote: In T159618#3079361, @Esc3300 wrote: I think it might be worth attempting to determine the factors that lead to the rapid raise. The edit rate didn't seem that high and we could easily have several bots that resulted in the same rate. I disagree. According to https://wikipulse.herokuapp.com/, the edit rate on Wikidata right now is anywhere from 150-220 epm. Adding an extra 600 epm on top of that is what caused the problem. The edit rate may have been the issue, but we should still utilize the tools we have (maxlag) to notify bots that the server is under high load. If we throw a check in maxlag value calculation checking for the number of JobQueue entries and then raising the maxlag to indicate it, it would prevent bots from causing this issue again. Regardless of whether its one bot or several causing the spike, the existing maxlag checks could be used to notify all bots to back off.TASK DETAILhttps://phabricator.wikimedia.org/T159618EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: BetacommandCc: Esc3300, Lydia_Pintscher, Emijrp, Betacommand, BethNaught, TerraCodes, Jay8g, Joe, aaron, Legoktm, Aklapper, Th3d3v1ls, Hfbn0, Zppix, D3r1ck01, Izno, Wikidata-bugs, aude, faidon, Mbch331, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T159618: Job queue rising to nearly 3 million jobs
Legoktm added a comment. In T159618#3079361, @Esc3300 wrote: I think it might be worth attempting to determine the factors that lead to the rapid raise. The edit rate didn't seem that high and we could easily have several bots that resulted in the same rate. I disagree. According to https://wikipulse.herokuapp.com/, the edit rate on Wikidata right now is anywhere from 150-220 epm. Adding an extra 600 epm on top of that is what caused the problem.TASK DETAILhttps://phabricator.wikimedia.org/T159618EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: LegoktmCc: Esc3300, Lydia_Pintscher, Emijrp, Betacommand, BethNaught, TerraCodes, Jay8g, Joe, aaron, Legoktm, Aklapper, Th3d3v1ls, Hfbn0, Zppix, D3r1ck01, Izno, Wikidata-bugs, aude, faidon, Mbch331, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T159618: Job queue rising to nearly 3 million jobs
Esc3300 added a comment. Many seem to be descriptions for category items (Something that might not be of much use to Wikipedia). On a few items I check, some languages use only the sitelink (en, fr), others all entity data (sv, es). Maybe this is linked to descriptions from Wikidata being used there.TASK DETAILhttps://phabricator.wikimedia.org/T159618EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Esc3300Cc: Esc3300, Lydia_Pintscher, Emijrp, Betacommand, BethNaught, TerraCodes, Jay8g, Joe, aaron, Legoktm, Aklapper, Th3d3v1ls, Zppix, D3r1ck01, Izno, Wikidata-bugs, aude, faidon, Mbch331, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T159618: Job queue rising to nearly 3 million jobs
Emijrp added a comment. @Esc3300 About 3 days (72 hours), at a roughly edit rate of 600 epm, I did 2.5 million edits, more or less. Just a note, my bot edits add descriptions in dozens of languages.[1] I don't know if that makes jobs difficult. [1] https://www.wikidata.org/w/index.php?title=Special:Contributions/Emijrpbot=prev=20170305004655=EmijrpbotTASK DETAILhttps://phabricator.wikimedia.org/T159618EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: EmijrpCc: Esc3300, Lydia_Pintscher, Emijrp, Betacommand, BethNaught, TerraCodes, Jay8g, Joe, aaron, Legoktm, Aklapper, Th3d3v1ls, Zppix, D3r1ck01, Izno, Wikidata-bugs, aude, faidon, Mbch331, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T159618: Job queue rising to nearly 3 million jobs
Esc3300 added a comment. I think it might be worth attempting to determine the factors that lead to the rapid raise. The edit rate didn't seem that high and we could easily have several bots that resulted in the same rate. The number of sitelinks on these items might have increased the impact on the job queue. Merely creating new items without any sitelinks probably has a lower impact. @Emijrp For how long did it run at that rate?TASK DETAILhttps://phabricator.wikimedia.org/T159618EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Esc3300Cc: Esc3300, Lydia_Pintscher, Emijrp, Betacommand, BethNaught, TerraCodes, Jay8g, Joe, aaron, Legoktm, Aklapper, Th3d3v1ls, Zppix, D3r1ck01, Izno, Wikidata-bugs, aude, faidon, Mbch331, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T159618: Job queue rising to nearly 3 million jobs
Emijrp added a comment. In T159618#3074646, @Lydia_Pintscher wrote: Anything else we need to do here? @Joe @Lydia_Pintscher Is the Betacommand suggestion feasible?TASK DETAILhttps://phabricator.wikimedia.org/T159618EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: EmijrpCc: Lydia_Pintscher, Emijrp, Betacommand, BethNaught, TerraCodes, Jay8g, Joe, aaron, Legoktm, Aklapper, Th3d3v1ls, Zppix, D3r1ck01, Izno, Wikidata-bugs, aude, faidon, Mbch331, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T159618: Job queue rising to nearly 3 million jobs
Lydia_Pintscher added a comment. Cool. Thanks!TASK DETAILhttps://phabricator.wikimedia.org/T159618EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Lydia_PintscherCc: Lydia_Pintscher, Emijrp, Betacommand, BethNaught, TerraCodes, Jay8g, Joe, aaron, Legoktm, Aklapper, Th3d3v1ls, Zppix, D3r1ck01, Izno, Wikidata-bugs, aude, faidon, Mbch331, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T159618: Job queue rising to nearly 3 million jobs
Joe added a comment. @Lydia_Pintscher not really, I'm monitoring the jobqueue and it's constantly decreasing in size. We should be ok.TASK DETAILhttps://phabricator.wikimedia.org/T159618EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: JoeCc: Lydia_Pintscher, Emijrp, Betacommand, BethNaught, TerraCodes, Jay8g, Joe, aaron, Legoktm, Aklapper, Th3d3v1ls, Zppix, D3r1ck01, Izno, Wikidata-bugs, aude, faidon, Mbch331, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T159618: Job queue rising to nearly 3 million jobs
Lydia_Pintscher added a comment. Anything else we need to do here?TASK DETAILhttps://phabricator.wikimedia.org/T159618EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Lydia_PintscherCc: Lydia_Pintscher, Emijrp, Betacommand, BethNaught, TerraCodes, Jay8g, Joe, aaron, Legoktm, Aklapper, Th3d3v1ls, Zppix, D3r1ck01, Izno, Wikidata-bugs, aude, faidon, Mbch331, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T159618: Job queue rising to nearly 3 million jobs
Emijrp added a comment. Done. I added a put_throttle = 3 seconds. Anyway I think I will wait for the job queue to go down.TASK DETAILhttps://phabricator.wikimedia.org/T159618EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: EmijrpCc: Emijrp, Betacommand, BethNaught, TerraCodes, Jay8g, Joe, aaron, Legoktm, Aklapper, Th3d3v1ls, Zppix, D3r1ck01, Izno, Wikidata-bugs, aude, faidon, Mbch331, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T159618: Job queue rising to nearly 3 million jobs
Legoktm added a comment. In T159618#3073691, @Emijrp wrote: All my bots follow the maxlag policy, as defined by default in Pywikibot user-config.py. Can you add a ratelimit to your bot? 20-30 epm would be reasonable for now.TASK DETAILhttps://phabricator.wikimedia.org/T159618EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: LegoktmCc: Emijrp, Betacommand, BethNaught, TerraCodes, Jay8g, Joe, aaron, Legoktm, Aklapper, Th3d3v1ls, Zppix, D3r1ck01, Izno, Wikidata-bugs, aude, faidon, Mbch331, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T159618: Job queue rising to nearly 3 million jobs
Emijrp added a comment. All my bots follow the maxlag policy, as defined by default in Pywikibot user-config.py.TASK DETAILhttps://phabricator.wikimedia.org/T159618EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: EmijrpCc: Emijrp, Betacommand, BethNaught, TerraCodes, Jay8g, Joe, aaron, Legoktm, Aklapper, Th3d3v1ls, Zppix, D3r1ck01, Izno, Wikidata-bugs, aude, faidon, Mbch331, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T159618: Job queue rising to nearly 3 million jobs
Betacommand added a comment. Suggestion, when the job queue gets too high set the maxlag parameter to a higher value, most bots use that as a throttle.TASK DETAILhttps://phabricator.wikimedia.org/T159618EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: BetacommandCc: Betacommand, BethNaught, TerraCodes, Jay8g, Joe, aaron, Legoktm, Aklapper, Th3d3v1ls, Zppix, D3r1ck01, Izno, Wikidata-bugs, aude, faidon, Mbch331, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T159618: Job queue rising to nearly 3 million jobs
Legoktm added a comment. 00:47, 5 March 2017 Sjoerddebruin (talk | contribs) blocked Emijrpbot (talk | contribs) with an expiration time of indefinite (account creation disabled, autoblock disabled) (Please respect the bot policy, too high editing rate) https://www.wikidata.org/wiki/User_talk:Emijrp#Slow_down_your_bot_please It was going 600+ epm. Job queue appears to be dropping now...TASK DETAILhttps://phabricator.wikimedia.org/T159618EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: LegoktmCc: BethNaught, TerraCodes, Jay8g, Joe, aaron, Legoktm, Aklapper, Th3d3v1ls, Zppix, D3r1ck01, Izno, Wikidata-bugs, aude, faidon, Mbch331, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
