On Thu, Apr 21, 2016 at 1:45 AM, Brion Vibber <bvib...@wikimedia.org> wrote: > Over in TimedMediaHandler extension, we've had a number of cases where old > code did things that were convenient in terms of squishing read-write > operations into data getters, that got removed due to problems with long > running transactions or needing to refactor things to support > future-facing multi-DC work where we want requests to be able to more > reliably distinguish between read-only and read-write. And we sometimes > want to put some of those clever hacks back and add more. ;) > > For instance in https://gerrit.wikimedia.org/r/284368 we'd like to remove > transcode derivative files of types/resolutions that have been disabled > automatically when we come across them. But I'm a bit unsure it's safe to > do so. > > Note that we could fire off a job queue background task to do the actual > removal... But is it also safe to do that on a read-only request? > https://www.mediawiki.org/wiki/Requests_for_comment/Master_%26_slave_datacenter_strategy_for_MediaWiki > seems to indicate job queueing will be safe, but would like to confirm > that. :) > > Similarly in https://gerrit.wikimedia.org/r/#/c/284269/ we may wish to > trigger missing transcodes to run on demand, similarly. The actual re > encoding happens in a background job, but we have to fire it off, and we > have to record that we fired it off so we don't duplicate it... > > (This would require a second queue to do the high-priority state table > update and queue the actual transcoding job; we can't put them in one queue > because a backup of transcode jobs would prevent the high priority job from > running in a timely fashion.) > > A best practices document on future-proofing for multi DC would be pretty > awesome! Maybe factor out some of the stuff from the RfC into a nice dev > doc page... > > -- brion > _______________________________________________ > Wikitech-l mailing list > Wikitech-l@lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
When doing something like that from a read request, there's also the problem for a popular page that there might be lots of views (maybe thousands if the queue is a little backed up) before the job is processed. So if the view triggers the job, and it will only stop triggering inserting the job after the job has been executed, this might cause a large number of useless jobs to be en-queued until one of them is finally executed. -- -bawolff _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
