Re: [Wikitech-l] [Wikivideo-l] Video transcode prioritization changes in progress
Ok, the video scaler queue change is now live: SD and low-resolution video output runs on a new, high-priority queue, with HD video output and transcodes of files over 15 minutes in length run on the old queue. Note that existing queued conversions will continue to run on the low-priority queue until it runs out, but most new uploads from here out will have web-ready output much faster. (Load on the servers may be unbalanced for a bit, and may require further tweaking to best balance responsiveness and throughput, so we'll keep an eye on the graphs.) -- brion On Thu, Feb 9, 2017 at 9:39 AM, Brion Vibber wrote: > We're in progress of deploying a small change to the video scalers[1], > which should improve availability of newly uploaded video and audio files. > > The queue will now be split in two, one which covers low-resolution > conversions for relatively short files, and one which covers long files and > high-resolution conversions. When there's a flood of large uploads, other > new files should still go through the high-priority queue while the large > uploads and HD conversions may back up on the low-priority queue. > > The queue-runner side is updated now (done during the 'puppet SWAT' > deployment window), with the MediaWiki side ready to roll around 19:00 UTC > (11am Pacific time, regular SWAT deployment window). > > [1] https://gerrit.wikimedia.org/r/#/c/336846/ > > -- brion > > ___ > Wikivideo-l mailing list > wikivide...@lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/wikivideo-l > > ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] [Wikivideo-l] Video transcode prioritization changes in progress
Quick update -- I tested some batch-reencoding of missing low-res files and have flooded the high-priority queue. :) Thanks for your patience as we get the balance right. I'll add a throttle to TimedMediaHandler's requeueTranscodes.php so I can "fire-and-forget" on the background jobs without disturbing things... -- brion On Thu, Feb 9, 2017 at 12:00 PM, Brion Vibber wrote: > Ok, the video scaler queue change is now live: SD and low-resolution video > output runs on a new, high-priority queue, with HD video output and > transcodes of files over 15 minutes in length run on the old queue. > > Note that existing queued conversions will continue to run on the > low-priority queue until it runs out, but most new uploads from here out > will have web-ready output much faster. > > (Load on the servers may be unbalanced for a bit, and may require further > tweaking to best balance responsiveness and throughput, so we'll keep an > eye on the graphs.) > > -- brion > > > On Thu, Feb 9, 2017 at 9:39 AM, Brion Vibber > wrote: > >> We're in progress of deploying a small change to the video scalers[1], >> which should improve availability of newly uploaded video and audio files. >> >> The queue will now be split in two, one which covers low-resolution >> conversions for relatively short files, and one which covers long files and >> high-resolution conversions. When there's a flood of large uploads, other >> new files should still go through the high-priority queue while the large >> uploads and HD conversions may back up on the low-priority queue. >> >> The queue-runner side is updated now (done during the 'puppet SWAT' >> deployment window), with the MediaWiki side ready to roll around 19:00 UTC >> (11am Pacific time, regular SWAT deployment window). >> >> [1] https://gerrit.wikimedia.org/r/#/c/336846/ >> >> -- brion >> >> ___ >> Wikivideo-l mailing list >> wikivide...@lists.wikimedia.org >> https://lists.wikimedia.org/mailman/listinfo/wikivideo-l >> >> > > ___ > Wikivideo-l mailing list > wikivide...@lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/wikivideo-l > > ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] [Wikivideo-l] Video transcode prioritization changes in progress
Should people refrain from uploading videos at this time? On February 9, 2017 at 5:01:51 PM, Brion Vibber (bvib...@wikimedia.org) wrote: Quick update -- I tested some batch-reencoding of missing low-res files and have flooded the high-priority queue. :) Thanks for your patience as we get the balance right. I'll add a throttle to TimedMediaHandler's requeueTranscodes.php so I can "fire-and-forget" on the background jobs without disturbing things... -- brion On Thu, Feb 9, 2017 at 12:00 PM, Brion Vibber wrote: > Ok, the video scaler queue change is now live: SD and low-resolution video > output runs on a new, high-priority queue, with HD video output and > transcodes of files over 15 minutes in length run on the old queue. > > Note that existing queued conversions will continue to run on the > low-priority queue until it runs out, but most new uploads from here out > will have web-ready output much faster. > > (Load on the servers may be unbalanced for a bit, and may require further > tweaking to best balance responsiveness and throughput, so we'll keep an > eye on the graphs.) > > -- brion > > > On Thu, Feb 9, 2017 at 9:39 AM, Brion Vibber > wrote: > >> We're in progress of deploying a small change to the video scalers[1], >> which should improve availability of newly uploaded video and audio files. >> >> The queue will now be split in two, one which covers low-resolution >> conversions for relatively short files, and one which covers long files and >> high-resolution conversions. When there's a flood of large uploads, other >> new files should still go through the high-priority queue while the large >> uploads and HD conversions may back up on the low-priority queue. >> >> The queue-runner side is updated now (done during the 'puppet SWAT' >> deployment window), with the MediaWiki side ready to roll around 19:00 UTC >> (11am Pacific time, regular SWAT deployment window). >> >> [1] https://gerrit.wikimedia.org/r/#/c/336846/ >> >> -- brion >> >> ___ >> Wikivideo-l mailing list >> wikivide...@lists.wikimedia.org >> https://lists.wikimedia.org/mailman/listinfo/wikivideo-l >> >> > > ___ > Wikivideo-l mailing list > wikivide...@lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/wikivideo-l > > ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] [Wikivideo-l] Video transcode prioritization changes in progress
Looks like the high-prio queue is back to normal, all is well. :) Please continue uploading. -- brion On Thu, Feb 9, 2017 at 5:02 PM, James Hare wrote: > Should people refrain from uploading videos at this time? > > > On February 9, 2017 at 5:01:51 PM, Brion Vibber (bvib...@wikimedia.org) > wrote: > > Quick update -- I tested some batch-reencoding of missing low-res files and > have flooded the high-priority queue. :) Thanks for your patience as we get > the balance right. > > I'll add a throttle to TimedMediaHandler's requeueTranscodes.php so I can > "fire-and-forget" on the background jobs without disturbing things... > > -- brion > > On Thu, Feb 9, 2017 at 12:00 PM, Brion Vibber > wrote: > > > Ok, the video scaler queue change is now live: SD and low-resolution > video > > output runs on a new, high-priority queue, with HD video output and > > transcodes of files over 15 minutes in length run on the old queue. > > > > Note that existing queued conversions will continue to run on the > > low-priority queue until it runs out, but most new uploads from here out > > will have web-ready output much faster. > > > > (Load on the servers may be unbalanced for a bit, and may require further > > tweaking to best balance responsiveness and throughput, so we'll keep an > > eye on the graphs.) > > > > -- brion > > > > > > On Thu, Feb 9, 2017 at 9:39 AM, Brion Vibber > > wrote: > > > >> We're in progress of deploying a small change to the video scalers[1], > >> which should improve availability of newly uploaded video and audio > files. > >> > >> The queue will now be split in two, one which covers low-resolution > >> conversions for relatively short files, and one which covers long files > and > >> high-resolution conversions. When there's a flood of large uploads, > other > >> new files should still go through the high-priority queue while the > large > >> uploads and HD conversions may back up on the low-priority queue. > >> > >> The queue-runner side is updated now (done during the 'puppet SWAT' > >> deployment window), with the MediaWiki side ready to roll around 19:00 > UTC > >> (11am Pacific time, regular SWAT deployment window). > >> > >> [1] https://gerrit.wikimedia.org/r/#/c/336846/ > >> > >> -- brion > >> > >> ___ > >> Wikivideo-l mailing list > >> wikivide...@lists.wikimedia.org > >> https://lists.wikimedia.org/mailman/listinfo/wikivideo-l > >> > >> > > > > ___ > > Wikivideo-l mailing list > > wikivide...@lists.wikimedia.org > > https://lists.wikimedia.org/mailman/listinfo/wikivideo-l > > > > > ___ > Wikitech-l mailing list > Wikitech-l@lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/wikitech-l > ___ > Wikitech-l mailing list > Wikitech-l@lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/wikitech-l > ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l