ACN: Thanks! We’ve created a ticket for that one to help collaborate and surface the process here: https://phabricator.wikimedia.org/T267992 <https://phabricator.wikimedia.org/T267992> Anybody working on that, please add info there.
Brooke Storm Staff SRE Wikimedia Cloud Services bst...@wikimedia.org <mailto:bst...@wikimedia.org> IRC: bstorm > On Nov 17, 2020, at 12:01 PM, AntiCompositeNumber > <anticompositenum...@gmail.com> wrote: > > I took a look at converting the query used for GreenC Bot's Job 10, > which tracks enwiki files that "shadow" a different file on Commons. > It is currently run daily, and the query executes in about 60-90 > seconds. I tried three methods to recreate that query without a SQL > cross-database join. The naive method of "just give me all the files" > didn't work because it timed out somewhere. The paginated version of > that query was on track to take over 5 hours to complete. A similar > method that emulates a subquery instead of a join was projected to > take about 6 hours. Both stopped early because I got bored of watching > them and PAWS doesn't work unattended. I also wasn't able to properly > test them because people kept fixing the shadowed files before the > script got to them. The code is at > <https://public.paws.wmcloud.org/User:AntiCompositeBot/ShadowsCommonsQuery.ipynb>. > > ACN > > On Tue, Nov 17, 2020 at 1:02 PM Maarten Dammers <maar...@mdammers.nl> wrote: >> >> Hi Joaquin, >> >> On 16-11-2020 21:42, Joaquin Oltra Hernandez wrote: >> >> Hi Maarten, >> >> I believe this work started many years ago, and it was paused, and recently >> restarted because of the stability and performance problems in the last >> years. >> >> You do realize the current setup was announced as new 3 years ago? See >> https://phabricator.wikimedia.org/phame/post/view/70/new_wiki_replica_servers_ready_for_use/ >> . >> >> I'm sorry about the extra work this will cause, I hope the improved >> stability and performance will make it worth it for you, and that you will >> reconsider and migrate your code to work on the new architecture (or reach >> out for specific help if you need it). >> >> No, saying sorry won't make it right and no, it won't make it worth it for >> me. If I want very stable access to a single wiki, I'll use the API of that >> wiki. >> >> -- >> Joaquin Oltra Hernandez >> Developer Advocate - Wikimedia Foundation >> >> It currently doesn't really feel to me that you're advocating for the >> developers, it feels more like you're the unlucky person having to sell the >> bad WMF management decisions to the angry developers. >> >> Maarten >> >> _______________________________________________ >> Wikimedia Cloud Services mailing list >> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org) >> https://lists.wikimedia.org/mailman/listinfo/cloud > > _______________________________________________ > Wikimedia Cloud Services mailing list > Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org) > https://lists.wikimedia.org/mailman/listinfo/cloud
signature.asc
Description: Message signed with OpenPGP
_______________________________________________ Wikimedia Cloud Services mailing list Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud
