| mobrovac added a comment. |
In T175316#3608388, @daniel wrote:@mobrovac how about a very large number of very small jobs? e.g. a million jobs to purge a million pages from cdn?
Note that we introduced batching only a few weeks ago, at the explicit request of the performance folks. We had one job per purge before. It caused problems.
This is where the improvement part of the discussion comes in :) For example, in the concrete case of CDN purges, the EventBus/ChangeProp system supports that out of the box, so instead of having the flow MW -> EB -> CP -> JR -> V we could have MW -> EB -> CP -> V. This is the way we are doing it for async updates, doing around 500-1000 purges per second.
I think we should sit down together (Wikidata/Services) and go over the operational side of WD jobs and see (a) how can the new JQ system best support it; and (b) what changes/improvements can be made on both sides to make the system more performant and robust. (Disclaimer: I am not claiming that there is something wrong, and confess my ignorance when it comes to WD, but as this ticket illustrates, there is definitely room for improvement)
But, this is getting a bit out of scope of this concrete ticket. For now, let's just try to get the size of the jobs below the 4MB mark? :)
Cc: mobrovac, Stashbot, Legoktm, hoo, Addshore, aude, gerritbot, Ladsgroup, daniel, GWicke, Aklapper, Pchelolo, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Lewizho99, Maathavan, Izno, Eevans, JAllemandou, Hardikj, Wikidata-bugs, Mbch331, jeremyb
_______________________________________________ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
