ArielGlenn has submitted this change and it was merged. (
https://gerrit.wikimedia.org/r/346037 )
Change subject: add new config options for dumps of big wikis
......................................................................
add new config options for dumps of big wikis
we pregenerate page ranges for lists of jobs and do automatic retry
of jobs that fail; the new config settings are used for this
Change-Id: Ia5c7f29f41e1d5c294f2f79a1fe345da89baa8e6
---
M modules/snapshot/manifests/dumps/configs.pp
M modules/snapshot/templates/dumps/wikidump.conf.erb
2 files changed, 16 insertions(+), 0 deletions(-)
Approvals:
ArielGlenn: Looks good to me, approved
jenkins-bot: Verified
diff --git a/modules/snapshot/manifests/dumps/configs.pp
b/modules/snapshot/manifests/dumps/configs.pp
index f169e3c..04ee6ae 100644
--- a/modules/snapshot/manifests/dumps/configs.pp
+++ b/modules/snapshot/manifests/dumps/configs.pp
@@ -21,6 +21,10 @@
keep => '8',
chunksEnabled => '1',
recombineHistory => '0',
+ revsPerJob => '1500000',
+ retryWait => '30',
+ maxRetries => '3',
+ revsMargin => '100',
wikis => {
ruwiki => {
pagesPerChunkHistory => '311181,1142420,1627923,3122803',
diff --git a/modules/snapshot/templates/dumps/wikidump.conf.erb
b/modules/snapshot/templates/dumps/wikidump.conf.erb
index dc239aa..2d72cfc 100644
--- a/modules/snapshot/templates/dumps/wikidump.conf.erb
+++ b/modules/snapshot/templates/dumps/wikidump.conf.erb
@@ -58,6 +58,18 @@
<% if @config[@configtype].has_key?('checkpointTime') -%>
checkpointTime=<%= @config[@configtype]['checkpointTime'] %>
<% end -%>
+<% if @config[@configtype].has_key?('revsPerJob') -%>
+revsPerJob=<%= @config[@configtype]['revsPerJob'] %>
+<% end -%>
+<% if @config[@configtype].has_key?('retryWait') -%>
+retryWait=<%= @config[@configtype]['retryWait'] %>
+<% end -%>
+<% if @config[@configtype].has_key?('maxRetries') -%>
+maxRetries=<%= @config[@configtype]['maxRetries'] %>
+<% end -%>
+<% if @config[@configtype].has_key?('revsMargin') -%>
+revsMargin=<%= @config[@configtype]['revsMargin'] %>
+<% end -%>
[otherformats]
multistream=1
--
To view, visit https://gerrit.wikimedia.org/r/346037
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Ia5c7f29f41e1d5c294f2f79a1fe345da89baa8e6
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: ArielGlenn <ar...@wikimedia.org>
Gerrit-Reviewer: ArielGlenn <ar...@wikimedia.org>
Gerrit-Reviewer: Giuseppe Lavagetto <glavage...@wikimedia.org>
Gerrit-Reviewer: jenkins-bot <>
_______________________________________________
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
