Brion VIBBER has uploaded a new change for review. (
https://gerrit.wikimedia.org/r/331668 )
Change subject: Split TMH transcode queue into two for prioritization
......................................................................
Split TMH transcode queue into two for prioritization
Prep for I3d834ad: creates a second queue for high-priority transcodes,
which includes only SD resolutions and videos less than 15 minutes long
by default.
HD resolutions and longer videos go in the existing queue.
This should allow for more responsivity to new uploads when there's a
flood of uploads, though can still get bogged down if there's too much.
Later, the jobs themselves will be improved to do even better selection
of what to work on, but using these same queues.
Needs to be live or the high-priority jobs will be sent to regular
job queue runners instead of video scalers, leading to failures if
ffmpeg etc is missing.
Bug: T155098
Change-Id: I83e5dab7dfb7be7a6df1a516701ef534601d530f
---
M hieradata/hosts/mw1168.yaml
M hieradata/hosts/mw1169.yaml
M hieradata/labs/deployment-prep/host/deployment-tmh01.yaml
M hieradata/role/common/mediawiki/videoscaler.yaml
M modules/mediawiki/manifests/jobrunner.pp
M modules/mediawiki/templates/jobrunner/jobrunner.conf.erb
6 files changed, 16 insertions(+), 4 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/68/331668/1
diff --git a/hieradata/hosts/mw1168.yaml b/hieradata/hosts/mw1168.yaml
index 08ebfe2..dd089e1 100644
--- a/hieradata/hosts/mw1168.yaml
+++ b/hieradata/hosts/mw1168.yaml
@@ -1,4 +1,5 @@
-mediawiki::jobrunner::runners_transcode: 16
+mediawiki::jobrunner::runners_transcode: 4
+mediawiki::jobrunner::runners_transcode_prioritized: 12
hhvm::extra::fcgi:
hhvm:
server:
diff --git a/hieradata/hosts/mw1169.yaml b/hieradata/hosts/mw1169.yaml
index 08ebfe2..dd089e1 100644
--- a/hieradata/hosts/mw1169.yaml
+++ b/hieradata/hosts/mw1169.yaml
@@ -1,4 +1,5 @@
-mediawiki::jobrunner::runners_transcode: 16
+mediawiki::jobrunner::runners_transcode: 4
+mediawiki::jobrunner::runners_transcode_prioritized: 12
hhvm::extra::fcgi:
hhvm:
server:
diff --git a/hieradata/labs/deployment-prep/host/deployment-tmh01.yaml
b/hieradata/labs/deployment-prep/host/deployment-tmh01.yaml
index 6ef6b62..5436415 100644
--- a/hieradata/labs/deployment-prep/host/deployment-tmh01.yaml
+++ b/hieradata/labs/deployment-prep/host/deployment-tmh01.yaml
@@ -2,4 +2,5 @@
mediawiki::jobrunner::runners_gwt: 0
mediawiki::jobrunner::runners_translate: 0
mediawiki::jobrunner::runners_transcode: 2
+mediawiki::jobrunner::runners_transcode_prioritized: 2
mediawiki::jobrunner::runners_upload: 0
diff --git a/hieradata/role/common/mediawiki/videoscaler.yaml
b/hieradata/role/common/mediawiki/videoscaler.yaml
index c579f22..01cc154 100644
--- a/hieradata/role/common/mediawiki/videoscaler.yaml
+++ b/hieradata/role/common/mediawiki/videoscaler.yaml
@@ -2,7 +2,8 @@
admin::groups:
- deployment
- perf-roots
-mediawiki::jobrunner::runners_transcode: 8
+mediawiki::jobrunner::runners_transcode: 4
+mediawiki::jobrunner::runners_transcode_prioritized: 4
nutcracker::verbosity: "4"
hhvm::extra::fcgi:
hhvm:
diff --git a/modules/mediawiki/manifests/jobrunner.pp
b/modules/mediawiki/manifests/jobrunner.pp
index ebe0db0..8f1f133 100644
--- a/modules/mediawiki/manifests/jobrunner.pp
+++ b/modules/mediawiki/manifests/jobrunner.pp
@@ -11,6 +11,7 @@
$runners_upload = 0,
$runners_gwt = 0,
$runners_transcode = 0,
+ $runners_transcode_prioritized = 0,
$runners_translate = 0,
$statsd_server = undef,
$port = 9005,
diff --git a/modules/mediawiki/templates/jobrunner/jobrunner.conf.erb
b/modules/mediawiki/templates/jobrunner/jobrunner.conf.erb
index d39865c..23bcbd6 100644
--- a/modules/mediawiki/templates/jobrunner/jobrunner.conf.erb
+++ b/modules/mediawiki/templates/jobrunner/jobrunner.conf.erb
@@ -17,7 +17,8 @@
"gwtoolsetUploadMediafileJob",
"gwtoolsetUploadMetadataJob",
"uploadFromUrl",
- "webVideoTranscode"
+ "webVideoTranscode",
+ "webVideoTranscodePrioritized",
],
"low-priority": [
"cirrusSearchLinksUpdate",
@@ -46,6 +47,12 @@
"webVideoTranscode"
]
},
+ "transcode_prioritized": {
+ "runners": <%= @runners_transcode_prioritized %>,
+ "include": [
+ "webVideoTranscodePrioritized"
+ ]
+ },
"translate": {
"runners": <%= @runners_translate %>,
"include": [
--
To view, visit https://gerrit.wikimedia.org/r/331668
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I83e5dab7dfb7be7a6df1a516701ef534601d530f
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Brion VIBBER <br...@wikimedia.org>
_______________________________________________
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
