Brion VIBBER has uploaded a new change for review. (
https://gerrit.wikimedia.org/r/331662 )
Change subject: TMH job queue split into low and high priority
......................................................................
TMH job queue split into low and high priority
This is a first step to retooling the queue management for TMH transcode.
Lower resolutions and shorter files are now transcoded via one queue
('webVideoTranscodePrioritized' jobs) while high resolutions and long
files are transcoded in a second (the existing 'webVideoTranscode' job)
Beware that in production, this may require setting up the new queue
to send to the scalers!
Does not yet address relative prioritization after upload; each queue
can still bog down if it gets behind.
The thresholds can be configured via global vars:
$wgTmhPriorityResolutionThreshold = 480; // pixels height
$wgTmhPriorityLengthThreshold = 900; // seconds
Bug: T155098
Change-Id: I3d834adddfa68e91bd5369246724cac2360f0de6
---
M TimedMediaHandler.hooks.php
M TimedMediaHandler.php
M WebVideoTranscode/WebVideoTranscode.php
M WebVideoTranscode/WebVideoTranscodeJob.php
4 files changed, 29 insertions(+), 1 deletion(-)
git pull
ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/TimedMediaHandler
refs/changes/62/331662/1
diff --git a/TimedMediaHandler.hooks.php b/TimedMediaHandler.hooks.php
index 016ae73..1a1d46e 100644
--- a/TimedMediaHandler.hooks.php
+++ b/TimedMediaHandler.hooks.php
@@ -261,6 +261,8 @@
// Add transcode job class:
$wgJobClasses['webVideoTranscode'] = 'WebVideoTranscodeJob';
+ // Same class with different queue priority:
+ $wgJobClasses['webVideoTranscodePrioritized'] =
'WebVideoTranscodeJob';
// Transcode jobs must be explicitly requested from the job
queue:
$wgJobTypesExcludedFromDefaultQueue[] = 'webVideoTranscode';
diff --git a/TimedMediaHandler.php b/TimedMediaHandler.php
index 15e348a..bbf1ad5 100644
--- a/TimedMediaHandler.php
+++ b/TimedMediaHandler.php
@@ -244,6 +244,15 @@
$wgFileExtensions = array_merge( $wgFileExtensions, $wgTmhFileExtensions );
+// Transcode resolutions higher than this will run in the low-priority queue
+// This'll give us SD transcodes as fast as possible, then do HD later.
+$wgTmhPriorityResolutionThreshold = 480;
+
+// Transcodes of files longer than this (seconds) will run in the low-priority
+// queue; defaults to 15 minutes.
+// This'll mean long videos won't flood the high-priority queue.
+$wgTmhPriorityLengthThreshold = 900;
+
// Timed Media Handler AutoLoad Classes:
$wgAutoloadClasses['TimedMediaHandler'] =
"$timedMediaDir/TimedMediaHandler_body.php";
$wgAutoloadClasses['TimedMediaHandlerHooks'] =
"$timedMediaDir/TimedMediaHandler.hooks.php";
diff --git a/WebVideoTranscode/WebVideoTranscode.php
b/WebVideoTranscode/WebVideoTranscode.php
index 0ada0c4..1628ec5 100644
--- a/WebVideoTranscode/WebVideoTranscode.php
+++ b/WebVideoTranscode/WebVideoTranscode.php
@@ -1153,6 +1153,8 @@
* @param $transcodeKey String transcode key
*/
public static function updateJobQueue( &$file, $transcodeKey ) {
+ global $wgTmhPriorityResolutionThreshold,
$wgTmhPriorityLengthThreshold;
+
$fileName = $file->getTitle()->getDbKey();
$db = $file->repo->getMasterDB();
@@ -1182,9 +1184,19 @@
return;
}
+ // Set the priority
+ $transcodeHeight = 0;
+ $matches = [];
+ if ( preg_match( '/^(\d+)p/', $transcodeKey, $matches )
) {
+ $transcodeHeight = intval( $matches[0] );
+ }
+ $prioritized = ( $transcodeHeight <=
$wgTmhPriorityResolutionThreshold )
+ && ( $file->getLength() <=
$wgTmhPriorityLengthThreshold );
+
$job = new WebVideoTranscodeJob( $file->getTitle(), [
'transcodeMode' => 'derivative',
'transcodeKey' => $transcodeKey,
+ 'prioritized' => $prioritized
] );
try {
diff --git a/WebVideoTranscode/WebVideoTranscodeJob.php
b/WebVideoTranscode/WebVideoTranscodeJob.php
index 980a66a..32c43b9 100644
--- a/WebVideoTranscode/WebVideoTranscodeJob.php
+++ b/WebVideoTranscode/WebVideoTranscodeJob.php
@@ -26,7 +26,12 @@
public $file;
public function __construct( $title, $params, $id = 0 ) {
- parent::__construct( 'webVideoTranscode', $title, $params, $id
);
+ if ( isset( $params['prioritized'] ) && $params['prioritized'])
{
+ $command = 'webVideoTranscodePrioritized';
+ } else {
+ $command = 'webVideoTranscode';
+ }
+ parent::__construct( $command, $title, $params, $id );
$this->removeDuplicates = true;
}
--
To view, visit https://gerrit.wikimedia.org/r/331662
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I3d834adddfa68e91bd5369246724cac2360f0de6
Gerrit-PatchSet: 1
Gerrit-Project: mediawiki/extensions/TimedMediaHandler
Gerrit-Branch: master
Gerrit-Owner: Brion VIBBER <br...@wikimedia.org>
_______________________________________________
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
