[MediaWiki-commits] [Gerrit] operations/puppet[production]: enable use of 7z files for page content dump prefetch

[ArielGlenn (Code Review)]

ArielGlenn has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/404437 )

Change subject: enable use of 7z files for page content dump prefetch
..


enable use of 7z files for page content dump prefetch

Bug: T179267
Change-Id: Ia55cd97f9e0552fca12368655bc733b5db2ad619
---
M modules/snapshot/templates/dumps/wikidump.conf.erb
1 file changed, 3 insertions(+), 0 deletions(-)

Approvals:
  ArielGlenn: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/snapshot/templates/dumps/wikidump.conf.erb 
b/modules/snapshot/templates/dumps/wikidump.conf.erb
index dee0ff7..9a6273d 100644
--- a/modules/snapshot/templates/dumps/wikidump.conf.erb
+++ b/modules/snapshot/templates/dumps/wikidump.conf.erb
@@ -73,6 +73,9 @@
 [otherformats]
 multistream=1
 
+[misc]
+sevenzipprefetch=1
+
 [stubs]
 orderrevs=0
 minpages=1

-- 
To view, visit https://gerrit.wikimedia.org/r/404437
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ia55cd97f9e0552fca12368655bc733b5db2ad619
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: ArielGlenn 
Gerrit-Reviewer: ArielGlenn 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: enable use of 7z files for page content dump prefetch

[ArielGlenn (Code Review)]

ArielGlenn has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/404437 )

Change subject: enable use of 7z files for page content dump prefetch
..

enable use of 7z files for page content dump prefetch

Bug: T179267
Change-Id: Ia55cd97f9e0552fca12368655bc733b5db2ad619
---
M modules/snapshot/templates/dumps/wikidump.conf.erb
1 file changed, 3 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/37/404437/1

diff --git a/modules/snapshot/templates/dumps/wikidump.conf.erb 
b/modules/snapshot/templates/dumps/wikidump.conf.erb
index dee0ff7..9a6273d 100644
--- a/modules/snapshot/templates/dumps/wikidump.conf.erb
+++ b/modules/snapshot/templates/dumps/wikidump.conf.erb
@@ -73,6 +73,9 @@
 [otherformats]
 multistream=1
 
+[misc]
+sevenzipprefetch=1
+
 [stubs]
 orderrevs=0
 minpages=1

-- 
To view, visit https://gerrit.wikimedia.org/r/404437
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ia55cd97f9e0552fca12368655bc733b5db2ad619
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: ArielGlenn 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/dumps[master]: permit use of 7zip compressed files for prefetch

[ArielGlenn (Code Review)]

ArielGlenn has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/399753 )

Change subject: permit use of 7zip compressed files for prefetch
..


permit use of 7zip compressed files for prefetch

Bug: T179267
Change-Id: I14d4636c78d81a9bfbf04f7f4c218875fcb870dc
---
M xmldumps-backup/defaults.conf
M xmldumps-backup/dumps/WikiDump.py
M xmldumps-backup/dumps/xmlcontentjobs.py
3 files changed, 73 insertions(+), 42 deletions(-)

Approvals:
  ArielGlenn: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/xmldumps-backup/defaults.conf b/xmldumps-backup/defaults.conf
index 222cb2d..109fad6 100644
--- a/xmldumps-backup/defaults.conf
+++ b/xmldumps-backup/defaults.conf
@@ -80,4 +80,5 @@
 maxrevs=5
 
 [misc]
-fixeddumporder=0
\ No newline at end of file
+fixeddumporder=0
+sevenzipprefetch=0
diff --git a/xmldumps-backup/dumps/WikiDump.py 
b/xmldumps-backup/dumps/WikiDump.py
index 005f858..77b86f0 100644
--- a/xmldumps-backup/dumps/WikiDump.py
+++ b/xmldumps-backup/dumps/WikiDump.py
@@ -316,6 +316,11 @@
 self.conf.add_section('wiki')
 self.wiki_dir = self.get_opt_for_proj_or_default("wiki", "dir", 0)
 
+if not self.conf.has_section('misc'):
+self.conf.add_section('misc')
+self.sevenzip_prefetch = self.get_opt_in_overrides_or_default("misc", 
"sevenzipprefetch", 0)
+self.sevenzip_prefetch = int(self.sevenzip_prefetch, 0)
+
 def db_latest_status(self):
 '''
 return list of tuples for each wiki:
diff --git a/xmldumps-backup/dumps/xmlcontentjobs.py 
b/xmldumps-backup/dumps/xmlcontentjobs.py
index 2e21232..7b7956a 100644
--- a/xmldumps-backup/dumps/xmlcontentjobs.py
+++ b/xmldumps-backup/dumps/xmlcontentjobs.py
@@ -198,6 +198,55 @@
 pagerange['end'] = None
 return pagerange
 
+def _find_prefetch_files_from_run(self, runner, date, jobinfo,
+  pagerange, file_ext):
+"""
+for a given wiki and date, see if there are dump content
+files lying about that can be used for prefetch to the
+current job, with the given file extension (might be bz2s
+or 7zs or whatever) for the given range of pages
+"""
+dfnames = get_checkpt_files(
+runner.dump_dir, [jobinfo['dumpname']], self.jobinfo['ftype'],
+file_ext, date, parts=None)
+possible_prefetch_dfnames = self.get_relevant_prefetch_dfnames(
+dfnames, pagerange, date, runner)
+if len(possible_prefetch_dfnames):
+return possible_prefetch_dfnames
+
+# ok, let's check for file parts instead, from any run
+# (may not conform to our numbering for this job)
+dfnames = get_reg_files(
+runner.dump_dir, [jobinfo['dumpname']], jobinfo['ftype'],
+file_ext, date, parts=True)
+possible_prefetch_dfnames = self.get_relevant_prefetch_dfnames(
+dfnames, pagerange, date, runner)
+if len(possible_prefetch_dfnames):
+return possible_prefetch_dfnames
+
+# last shot, get output file that contains all the pages, if there is 
one
+dfnames = get_reg_files(
+runner.dump_dir, [jobinfo['dumpname']],
+jobinfo['ftype'], file_ext, date, parts=False)
+# there is only one, don't bother to check for relevance :-P
+possible_prefetch_dfnames = dfnames
+dfnames = []
+for prefetch_dfname in possible_prefetch_dfnames:
+if runner.wiki.is_private():
+possible_path = 
runner.dump_dir.filename_private_path(prefetch_dfname, date)
+else:
+possible_path = 
runner.dump_dir.filename_public_path(prefetch_dfname, date)
+size = os.path.getsize(possible_path)
+if size < 7:
+runner.debug("small %d-byte prefetch dump at %s, skipping" % (
+size, possible_path))
+continue
+else:
+dfnames.append(prefetch_dfname)
+if len(dfnames):
+return dfnames
+return None
+
 def _find_previous_dump(self, runner, partnum=None):
 """
 this finds the content file or files from the first previous 
successful dump
@@ -226,46 +275,15 @@
 runner.debug("skipping incomplete or failed dump for prefetch 
date %s" % date)
 continue
 
-# first check if there are checkpoint files from this run we can 
use
-dfnames = get_checkpt_files(
-runner.dump_dir, [self.jobinfo['dumpname']], 
self.jobinfo['ftype'],
-self.jobinfo['fext'], date, parts=None)
-possible_prefetch_dfnames = self.get_relevant_prefetch_dfnames(
-dfnames, pagerange, date, runner)
-if len(possible_prefetch_dfnames):
-return 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: don't send mail on dumps failures from labs

[ArielGlenn (Code Review)]

ArielGlenn has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/403176 )

Change subject: don't send mail on dumps failures from labs
..


don't send mail on dumps failures from labs

It was fun to see that it works, but that's enough of that.

Change-Id: I49ab4139e576faa78abd7459fb45b4bc17e11b7b
---
M modules/snapshot/manifests/dumps/configs.pp
M modules/snapshot/templates/dumps/wikidump.conf.erb
2 files changed, 3 insertions(+), 1 deletion(-)

Approvals:
  ArielGlenn: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/snapshot/manifests/dumps/configs.pp 
b/modules/snapshot/manifests/dumps/configs.pp
index 6cafb9e..f3b27c6 100644
--- a/modules/snapshot/manifests/dumps/configs.pp
+++ b/modules/snapshot/manifests/dumps/configs.pp
@@ -21,6 +21,7 @@
 skipdblist=> "${dblistsdir}/skip.dblist",
 keep  => '10',
 chunksEnabled => '0',
+adminmail => 'ops-du...@wikimedia.org',
 },
 big => {
 dblist=> "${dblistsdir}/bigwikis.dblist",
@@ -149,6 +150,7 @@
 skipdblist=> "${dblistsdir}/skip-labs.dblist",
 keep  => '2',
 chunksEnabled => '0',
+adminmail => 'nomail',
 },
 big => {
 dblist=> "${dblistsdir}/bigwikis-labs.dblist",
diff --git a/modules/snapshot/templates/dumps/wikidump.conf.erb 
b/modules/snapshot/templates/dumps/wikidump.conf.erb
index fe04686..dee0ff7 100644
--- a/modules/snapshot/templates/dumps/wikidump.conf.erb
+++ b/modules/snapshot/templates/dumps/wikidump.conf.erb
@@ -23,7 +23,7 @@
 fileperms=0644
 
 [reporting]
-adminmail=ops-du...@wikimedia.org
+adminmail=<%= @config[@configtype]['global']['adminmail'] %>
 mailfrom=r...@wikimedia.org
 smtpserver=<%= @mail_smarthost[0] %>
 # 15 minutes is long enough to decide a lock is expired, right?

-- 
To view, visit https://gerrit.wikimedia.org/r/403176
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I49ab4139e576faa78abd7459fb45b4bc17e11b7b
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: ArielGlenn 
Gerrit-Reviewer: ArielGlenn 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: don't send mail on dumps failures from labs

[ArielGlenn (Code Review)]

ArielGlenn has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/403176 )

Change subject: don't send mail on dumps failures from labs
..

don't send mail on dumps failures from labs

It was fun to see that it works, but that's enough of that.

Change-Id: I49ab4139e576faa78abd7459fb45b4bc17e11b7b
---
M modules/snapshot/manifests/dumps/configs.pp
M modules/snapshot/templates/dumps/wikidump.conf.erb
2 files changed, 3 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/76/403176/1

diff --git a/modules/snapshot/manifests/dumps/configs.pp 
b/modules/snapshot/manifests/dumps/configs.pp
index 6cafb9e..f3b27c6 100644
--- a/modules/snapshot/manifests/dumps/configs.pp
+++ b/modules/snapshot/manifests/dumps/configs.pp
@@ -21,6 +21,7 @@
 skipdblist=> "${dblistsdir}/skip.dblist",
 keep  => '10',
 chunksEnabled => '0',
+adminmail => 'ops-du...@wikimedia.org',
 },
 big => {
 dblist=> "${dblistsdir}/bigwikis.dblist",
@@ -149,6 +150,7 @@
 skipdblist=> "${dblistsdir}/skip-labs.dblist",
 keep  => '2',
 chunksEnabled => '0',
+adminmail => 'nomail',
 },
 big => {
 dblist=> "${dblistsdir}/bigwikis-labs.dblist",
diff --git a/modules/snapshot/templates/dumps/wikidump.conf.erb 
b/modules/snapshot/templates/dumps/wikidump.conf.erb
index fe04686..2278733 100644
--- a/modules/snapshot/templates/dumps/wikidump.conf.erb
+++ b/modules/snapshot/templates/dumps/wikidump.conf.erb
@@ -23,7 +23,7 @@
 fileperms=0644
 
 [reporting]
-adminmail=ops-du...@wikimedia.org
+adminmail=<%= @config[@configfype]['global']['adminmail'] %>
 mailfrom=r...@wikimedia.org
 smtpserver=<%= @mail_smarthost[0] %>
 # 15 minutes is long enough to decide a lock is expired, right?

-- 
To view, visit https://gerrit.wikimedia.org/r/403176
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I49ab4139e576faa78abd7459fb45b4bc17e11b7b
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: ArielGlenn 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/mediawiki-config[master]: load ActiveAbtract extension explicitly so class autoloading...

[ArielGlenn (Code Review)]

ArielGlenn has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/403114 )

Change subject: load ActiveAbtract extension explicitly so class autoloading 
works
..

load ActiveAbtract extension explicitly so class autoloading works

Needed for abstracts dumps, we only load it for php from the cli,
since that's the only time we want it.

Bug: T184177
Change-Id: I8da76b4f03afccc553da586084600885eda5c866
---
M wmf-config/CommonSettings.php
1 file changed, 4 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/mediawiki-config 
refs/changes/14/403114/1

diff --git a/wmf-config/CommonSettings.php b/wmf-config/CommonSettings.php
index 7c6c058..1a83fd6 100644
--- a/wmf-config/CommonSettings.php
+++ b/wmf-config/CommonSettings.php
@@ -3627,6 +3627,10 @@
wfLoadExtension( 'ReadingLists' );
 }
 
+if ( PHP_SAPI === 'cli' ) {
+   wfLoadExtension( 'ActiveAbstract' );
+}
+
 if ( $wmfRealm === 'labs' ) {
require "$wmfConfigDir/CommonSettings-labs.php";
 }

-- 
To view, visit https://gerrit.wikimedia.org/r/403114
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I8da76b4f03afccc553da586084600885eda5c866
Gerrit-PatchSet: 1
Gerrit-Project: operations/mediawiki-config
Gerrit-Branch: master
Gerrit-Owner: ArielGlenn 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: make role::beta::mediawiki into a profile

[ArielGlenn (Code Review)]

ArielGlenn has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/402803 )

Change subject: make role::beta::mediawiki into a profile
..

make role::beta::mediawiki into a profile

It should be included on all the beta mediawiki hosts, so it should
be a profile that goes in the appropriate role.  Make the current
role a shellincluding the profile; when all instances are converted
over to use the profile, the role can go away.

Change-Id: I53e6cf0b93698a415adb0f3b8d7503f9106e3ba5
---
A modules/profile/manifests/beta/mediawiki.pp
M modules/role/manifests/beta/mediawiki.pp
2 files changed, 19 insertions(+), 7 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/03/402803/1

diff --git a/modules/profile/manifests/beta/mediawiki.pp 
b/modules/profile/manifests/beta/mediawiki.pp
new file mode 100644
index 000..35ec787
--- /dev/null
+++ b/modules/profile/manifests/beta/mediawiki.pp
@@ -0,0 +1,15 @@
+# == Class profile::beta::mediawiki
+#
+# Allow mwdeploy to login from scap deployment host. Adds an exception in
+# /etc/security/access.conf to work around labs-specific restrictions
+#
+# filtertags: labs-project-deployment-prep
+class profile::beta::mediawiki {
+include ::profile::base::firewall
+
+$ips = 
join($network::constants::special_hosts[$::realm]['deployment_hosts'], ' ')
+security::access::config { 'scap-allow-mwdeploy':
+content  => "+ : mwdeploy : ${ips}\n",
+priority => 60,
+}
+}
diff --git a/modules/role/manifests/beta/mediawiki.pp 
b/modules/role/manifests/beta/mediawiki.pp
index f3ebc88..1a058e4 100644
--- a/modules/role/manifests/beta/mediawiki.pp
+++ b/modules/role/manifests/beta/mediawiki.pp
@@ -2,14 +2,11 @@
 #
 # Allow mwdeploy to login from scap deployment host. Adds an exception in
 # /etc/security/access.conf to work around labs-specific restrictions
+# Please consider adding profile::beta::mediawiki to your instance
+# roles instead, it does the same thing.  This role is here only for
+# backwards compatibility.
 #
 # filtertags: labs-project-deployment-prep
 class role::beta::mediawiki {
-include ::base::firewall
-
-$ips = 
join($network::constants::special_hosts[$::realm]['deployment_hosts'], ' ')
-security::access::config { 'scap-allow-mwdeploy':
-content  => "+ : mwdeploy : ${ips}\n",
-priority => 60,
-}
+include ::profile::beta::mediawiki
 }

-- 
To view, visit https://gerrit.wikimedia.org/r/402803
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I53e6cf0b93698a415adb0f3b8d7503f9106e3ba5
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: ArielGlenn 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: apt: Do not use experimental on stretch

[ArielGlenn (Code Review)]

ArielGlenn has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/402432 )

Change subject: apt: Do not use experimental on stretch
..


apt: Do not use experimental on stretch

Bug: T184239
Change-Id: I82c76be493e7483c5945014b454a4097fa6e
---
M modules/apt/manifests/init.pp
1 file changed, 1 insertion(+), 1 deletion(-)

Approvals:
  Alex Monk: Looks good to me, but someone else must approve
  Muehlenhoff: Looks good to me, but someone else must approve
  ArielGlenn: Looks good to me, approved
  Hashar: Looks good to me, but someone else must approve
  jenkins-bot: Verified
  Dzahn: Looks good to me, but someone else must approve



diff --git a/modules/apt/manifests/init.pp b/modules/apt/manifests/init.pp
index ba3badf..ed35d23 100644
--- a/modules/apt/manifests/init.pp
+++ b/modules/apt/manifests/init.pp
@@ -135,7 +135,7 @@
 default => absent,
 }
 
-if $::operatingsystem == 'Debian' {
+if os_version('debian <= jessie') {
 apt::repository { 'wikimedia-experimental':
 ensure => $use_experimental_ensure,
 uri=> 'http://apt.wikimedia.org/wikimedia',

-- 
To view, visit https://gerrit.wikimedia.org/r/402432
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I82c76be493e7483c5945014b454a4097fa6e
Gerrit-PatchSet: 4
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Paladox 
Gerrit-Reviewer: Alex Monk 
Gerrit-Reviewer: ArielGlenn 
Gerrit-Reviewer: Dzahn 
Gerrit-Reviewer: Hashar 
Gerrit-Reviewer: Muehlenhoff 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: fix up dblist and related config setting of bigwikis for dum...

[ArielGlenn (Code Review)]

ArielGlenn has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/402657 )

Change subject: fix up dblist and related config setting of bigwikis for dumps 
in labs
..


fix up dblist and related config setting of bigwikis for dumps in labs

Change-Id: I2bc623b1228296ac338dc2a8a54e1ab87c694277
---
M modules/snapshot/manifests/dumps/configs.pp
M modules/snapshot/manifests/dumps/dblists.pp
2 files changed, 9 insertions(+), 1 deletion(-)

Approvals:
  ArielGlenn: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/snapshot/manifests/dumps/configs.pp 
b/modules/snapshot/manifests/dumps/configs.pp
index 17e9585..6cafb9e 100644
--- a/modules/snapshot/manifests/dumps/configs.pp
+++ b/modules/snapshot/manifests/dumps/configs.pp
@@ -151,7 +151,7 @@
 chunksEnabled => '0',
 },
 big => {
-dblist=> "${dblistsdir}/labs_bigwikis.dblist",
+dblist=> "${dblistsdir}/bigwikis-labs.dblist",
 closedlist=> "${apachedir}/dblists/closed-labs.dblist",
 flowlist  => 
"${apachedir}/dblists/flow_only_labs.dblist",
 skipdblist=> "${dblistsdir}/skipmonitor.dblist",
diff --git a/modules/snapshot/manifests/dumps/dblists.pp 
b/modules/snapshot/manifests/dumps/dblists.pp
index 885c428..f6951e4 100644
--- a/modules/snapshot/manifests/dumps/dblists.pp
+++ b/modules/snapshot/manifests/dumps/dblists.pp
@@ -51,6 +51,14 @@
 group   => 'root',
 content => "${bigwikis_dblist}\n",
 }
+file { "${dblistsdir}/bigwikis-labs.dblist":
+ensure  => 'present',
+path=> "${dblistsdir}/bigwikis-labs.dblist",
+mode=> '0644',
+owner   => 'root',
+group   => 'root',
+content => "${labs_bigwikis_dblist}\n",
+}
 file { "${dblistsdir}/skip.dblist":
 ensure  => 'present',
 path=> "${dblistsdir}/skip.dblist",

-- 
To view, visit https://gerrit.wikimedia.org/r/402657
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I2bc623b1228296ac338dc2a8a54e1ab87c694277
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: ArielGlenn 
Gerrit-Reviewer: ArielGlenn 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: fix up dblist and related config setting of bigwikis for dum...

[ArielGlenn (Code Review)]

ArielGlenn has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/402657 )

Change subject: fix up dblist and related config setting of bigwikis for dumps 
in labs
..

fix up dblist and related config setting of bigwikis for dumps in labs

Change-Id: I2bc623b1228296ac338dc2a8a54e1ab87c694277
---
M modules/snapshot/manifests/dumps/configs.pp
M modules/snapshot/manifests/dumps/dblists.pp
2 files changed, 9 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/57/402657/1

diff --git a/modules/snapshot/manifests/dumps/configs.pp 
b/modules/snapshot/manifests/dumps/configs.pp
index 17e9585..6cafb9e 100644
--- a/modules/snapshot/manifests/dumps/configs.pp
+++ b/modules/snapshot/manifests/dumps/configs.pp
@@ -151,7 +151,7 @@
 chunksEnabled => '0',
 },
 big => {
-dblist=> "${dblistsdir}/labs_bigwikis.dblist",
+dblist=> "${dblistsdir}/bigwikis-labs.dblist",
 closedlist=> "${apachedir}/dblists/closed-labs.dblist",
 flowlist  => 
"${apachedir}/dblists/flow_only_labs.dblist",
 skipdblist=> "${dblistsdir}/skipmonitor.dblist",
diff --git a/modules/snapshot/manifests/dumps/dblists.pp 
b/modules/snapshot/manifests/dumps/dblists.pp
index 885c428..f6951e4 100644
--- a/modules/snapshot/manifests/dumps/dblists.pp
+++ b/modules/snapshot/manifests/dumps/dblists.pp
@@ -51,6 +51,14 @@
 group   => 'root',
 content => "${bigwikis_dblist}\n",
 }
+file { "${dblistsdir}/bigwikis-labs.dblist":
+ensure  => 'present',
+path=> "${dblistsdir}/bigwikis-labs.dblist",
+mode=> '0644',
+owner   => 'root',
+group   => 'root',
+content => "${labs_bigwikis_dblist}\n",
+}
 file { "${dblistsdir}/skip.dblist":
 ensure  => 'present',
 path=> "${dblistsdir}/skip.dblist",

-- 
To view, visit https://gerrit.wikimedia.org/r/402657
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I2bc623b1228296ac338dc2a8a54e1ab87c694277
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: ArielGlenn 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: add dumps config settings stanzas for labs testing

[ArielGlenn (Code Review)]

ArielGlenn has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/402633 )

Change subject: add dumps config settings stanzas for labs testing
..


add dumps config settings stanzas for labs testing

Change-Id: Idf94b66efd1ceb3b3aace3fd56022b086626aaf0
---
M modules/snapshot/manifests/dumps/configs.pp
M modules/snapshot/manifests/dumps/dblists.pp
2 files changed, 71 insertions(+), 1 deletion(-)

Approvals:
  ArielGlenn: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/snapshot/manifests/dumps/configs.pp 
b/modules/snapshot/manifests/dumps/configs.pp
index bd510ec..17e9585 100644
--- a/modules/snapshot/manifests/dumps/configs.pp
+++ b/modules/snapshot/manifests/dumps/configs.pp
@@ -136,7 +136,55 @@
 },
 media => {
 skipdblist=> 
"${dblistsdir}/skipmonitor.dblist,${dblistsdir}/globalusage.dblist",
-}
+},
+},
+}
+
+$labsconfig = {
+allwikis => {
+global => {
+dblist=> "${apachedir}/dblists/all-labs.dblist",
+closedlist=> "${apachedir}/dblists/closed-labs.dblist",
+flowlist  => "${apachedir}/dblists/flow_only_labs.dblist",
+skipdblist=> "${dblistsdir}/skip-labs.dblist",
+keep  => '2',
+chunksEnabled => '0',
+},
+big => {
+dblist=> "${dblistsdir}/labs_bigwikis.dblist",
+closedlist=> "${apachedir}/dblists/closed-labs.dblist",
+flowlist  => 
"${apachedir}/dblists/flow_only_labs.dblist",
+skipdblist=> "${dblistsdir}/skipmonitor.dblist",
+keep  => '2',
+chunksEnabled => '1',
+chunksForAbstract => '4',
+chunksForPagelogs => '4',
+checkpointTime=> '720',
+recombineHistory  => '0',
+revsPerJob=> '7',
+retryWait => '30',
+maxRetries=> '3',
+revsMargin=> '20',
+fixeddumporder=> '1',
+wikis => {
+enwiki => {
+pagesPerChunkHistory  => '2,4,7,8',
+pagesPerChunkAbstract => '4000',
+},
+simplewiki => {
+pagesPerChunkHistory  => '2,5,9,14',
+pagesPerChunkAbstract => '6',
+},
+wikidatawiki => {
+pagesPerChunkHistory  => '54,3,2,3',
+pagesPerChunkAbstract => '25000',
+},
+},
+},
+en => {},
+wd => {},
+monitor => {},
+media => {},
 },
 }
 
@@ -150,6 +198,15 @@
 tempdir=> $tempdir,
 }
 
+# for xml/sql dumps testing in beta
+snapshot::dumps::wikiconf { 'wikidump.conf.labs':
+configtype => 'allwikis',
+config => $labsconfig,
+publicdir  => $publicdir,
+privatedir => $privatedir,
+tempdir=> $tempdir,
+}
+
 file { "${confsdir}/table_jobs.yaml":
 ensure => 'present',
 path   => "${confsdir}/table_jobs.yaml",
diff --git a/modules/snapshot/manifests/dumps/dblists.pp 
b/modules/snapshot/manifests/dumps/dblists.pp
index d28843e..885c428 100644
--- a/modules/snapshot/manifests/dumps/dblists.pp
+++ b/modules/snapshot/manifests/dumps/dblists.pp
@@ -10,11 +10,16 @@
 'zhwiki']
 $bigwikis_dblist = join($bigwikis, "\n")
 
+# for testing in deployment-prep
+$labs_bigwikis = ['enwiki', 'simplewiki', 'wikidatawiki']
+$labs_bigwikis_dblist = join($labs_bigwikis, "\n")
+
 # labswiki(s) can't be dumped from snapshot hosts
 $excludewikis = ['labswiki', 'labtestwiki']
 $excludewikis_dblist = join($excludewikis, "\n")
 
 $skip_dblist = 
"${enwiki_dblist}\n${wikidatawiki_dblist}\n${bigwikis_dblist}\n${excludewikis_dblist}"
+$skip_labs_dblist = $labs_bigwikis_dblist
 
 $skipnone_dblist = ''
 
@@ -54,6 +59,14 @@
 group   => 'root',
 content => "${skip_dblist}\n",
 }
+file { "${dblistsdir}/skip-labs.dblist":
+ensure  => 'present',
+path=> "${dblistsdir}/skip-labs.dblist",
+mode=> '0644',
+owner   => 'root',
+group   => 'root',
+content => "${skip_labs_dblist}\n",
+}
 file { "${dblistsdir}/skipmonitor.dblist":
 ensure  => 'present',
 path=> "${dblistsdir}/skipmonitor.dblist",

-- 
To view, visit https://gerrit.wikimedia.org/r/402633
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

[MediaWiki-commits] [Gerrit] operations/puppet[production]: add dumps config settings stanzas for labs testing

[ArielGlenn (Code Review)]

ArielGlenn has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/402633 )

Change subject: add dumps config settings stanzas for labs testing
..

add dumps config settings stanzas for labs testing

Change-Id: Idf94b66efd1ceb3b3aace3fd56022b086626aaf0
---
M modules/snapshot/manifests/dumps/configs.pp
M modules/snapshot/manifests/dumps/dblists.pp
M modules/snapshot/templates/dumps/wikidump.conf.erb
3 files changed, 68 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/33/402633/1

diff --git a/modules/snapshot/manifests/dumps/configs.pp 
b/modules/snapshot/manifests/dumps/configs.pp
index bd510ec..cfad2ad 100644
--- a/modules/snapshot/manifests/dumps/configs.pp
+++ b/modules/snapshot/manifests/dumps/configs.pp
@@ -137,6 +137,45 @@
 media => {
 skipdblist=> 
"${dblistsdir}/skipmonitor.dblist,${dblistsdir}/globalusage.dblist",
 }
+labs_small => {
+dblist=> "${apachedir}/dblists/all-labs.dblist",
+closedlist=> "${apachedir}/dblists/closed-labs.dblist",
+flowlist  => "${apachedir}/dblists/flow_only_labs.dblist",
+skipdblist=> "${dblistsdir}/skip-labs.dblist",
+keep  => '2',
+chunksEnabled => '0',
+},
+labs_big => {
+dblist=> "${dblistsdir}/labs_bigwikis.dblist",
+closedlist=> "${apachedir}/dblists/closed-labs.dblist",
+flowlist  => 
"${apachedir}/dblists/flow_only_labs.dblist",
+skipdblist=> "${dblistsdir}/skipmonitor.dblist",
+keep  => '2',
+chunksEnabled => '1',
+chunksForAbstract => '4',
+chunksForPagelogs => '4',
+checkpointTime=> '720',
+recombineHistory  => '0',
+revsPerJob=> '7',
+retryWait => '30',
+maxRetries=> '3',
+revsMargin=> '20',
+fixeddumporder=> '1',
+wikis => {
+enwiki => {
+pagesPerChunkHistory  => '2,4,7,8',
+pagesPerChunkAbstract => '4000',
+},
+simplewiki => {
+pagesPerChunkHistory  => '2,5,9,14',
+pagesPerChunkAbstract => '6',
+},
+wikidatawikiwiki => {
+pagesPerChunkHistory  => '54,3,2,3',
+pagesPerChunkAbstract => '25000',
+},
+},
+},
 },
 }
 
diff --git a/modules/snapshot/manifests/dumps/dblists.pp 
b/modules/snapshot/manifests/dumps/dblists.pp
index d28843e..3dcb1b9 100644
--- a/modules/snapshot/manifests/dumps/dblists.pp
+++ b/modules/snapshot/manifests/dumps/dblists.pp
@@ -10,11 +10,16 @@
 'zhwiki']
 $bigwikis_dblist = join($bigwikis, "\n")
 
+# for testing in deployment-prep
+$labs_bigwikis = ['dewiki', 'simplewiki', 'wikidatawiki']
+$labs_bigwikis_dblist = join($bigwikis, "\n")
+
 # labswiki(s) can't be dumped from snapshot hosts
 $excludewikis = ['labswiki', 'labtestwiki']
 $excludewikis_dblist = join($excludewikis, "\n")
 
 $skip_dblist = 
"${enwiki_dblist}\n${wikidatawiki_dblist}\n${bigwikis_dblist}\n${excludewikis_dblist}"
+$skip_labs_dblist = "${labs_bigwikis_dblist}"
 
 $skipnone_dblist = ''
 
@@ -54,6 +59,14 @@
 group   => 'root',
 content => "${skip_dblist}\n",
 }
+file { "${dblistsdir}/skip-labs.dblist":
+ensure  => 'present',
+path=> "${dblistsdir}/skip-labs.dblist",
+mode=> '0644',
+owner   => 'root',
+group   => 'root',
+content => "${skip_labs_dblist}\n",
+}
 file { "${dblistsdir}/skipmonitor.dblist":
 ensure  => 'present',
 path=> "${dblistsdir}/skipmonitor.dblist",
diff --git a/modules/snapshot/templates/dumps/wikidump.conf.erb 
b/modules/snapshot/templates/dumps/wikidump.conf.erb
index fe04686..d893c99 100644
--- a/modules/snapshot/templates/dumps/wikidump.conf.erb
+++ b/modules/snapshot/templates/dumps/wikidump.conf.erb
@@ -118,6 +118,22 @@
 <% end -%>
 <% end -%>
 
+[labssmall]
+# generic settings for small wikis in labs
+<% @config[@configtype]['labs_small'].keys.sort.each do |bigsetting| -%>
+<% if bigsetting != 'wikis' -%>
+<%= bigsetting %>=<%= @config[@configtype]['labs_small'][bigsetting] %>
+<% end -%>
+<% end -%>
+
+[labsbig]
+# generic settings for big wikis in labs
+<% @config[@configtype]['labs_big'].keys.sort.each do |bigsetting| -%>
+<% if bigsetting != 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: snapshot testbed role for deployment-prep

[ArielGlenn (Code Review)]

ArielGlenn has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/402576 )

Change subject: snapshot testbed role for deployment-prep
..


snapshot testbed role for deployment-prep

One caveat: it requires application of a second role,
role::beta::mediawiki, as do all instances in eployment-prep
that have mediawiki scapped to them.

Change-Id: I513f4edaa7decebe01702b4f587860952c2b1d81
---
A modules/role/manifests/dumps/generation/worker/beta_testbed.pp
1 file changed, 23 insertions(+), 0 deletions(-)

Approvals:
  ArielGlenn: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/role/manifests/dumps/generation/worker/beta_testbed.pp 
b/modules/role/manifests/dumps/generation/worker/beta_testbed.pp
new file mode 100644
index 000..09cfed2
--- /dev/null
+++ b/modules/role/manifests/dumps/generation/worker/beta_testbed.pp
@@ -0,0 +1,23 @@
+# == Class role::dumps::generation::worker::beta_testbed
+#
+# Set up a dumps snapshot instance as a testbed.
+#
+# Because this class uses mediawiki, the class
+# role::beta::mediawiki must be applied to the instance
+# as well.  The role can't be included in this class for
+# style reasons.
+#
+# You should be using at least a m1.medium instance,
+# giving some extra lvm space locally mounted on
+# which to write the dumps.
+#
+# filtertags: labs-project-deployment-prep
+class role::dumps::generation::worker::beta_testbed {
+include standard
+
+include profile::dumps::generation::worker::common
+
+system::role { 'dumps::generation::worker::beta_testbed':
+description => 'beta testbed for dumps of XML/SQL wiki content',
+}
+}

-- 
To view, visit https://gerrit.wikimedia.org/r/402576
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I513f4edaa7decebe01702b4f587860952c2b1d81
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: ArielGlenn 
Gerrit-Reviewer: ArielGlenn 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: snapshot testbed role for deployment-prep

[ArielGlenn (Code Review)]

ArielGlenn has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/402576 )

Change subject: snapshot testbed role for deployment-prep
..

snapshot testbed role for deployment-prep

One caveat: it requires application of a second role,
role::beta::mediawiki, as do all instances in eployment-prep
that have mediawiki scapped to them.

Change-Id: I513f4edaa7decebe01702b4f587860952c2b1d81
---
A modules/role/manifests/dumps/generation/worker/beta_testbed.pp
1 file changed, 23 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/76/402576/1

diff --git a/modules/role/manifests/dumps/generation/worker/beta_testbed.pp 
b/modules/role/manifests/dumps/generation/worker/beta_testbed.pp
new file mode 100644
index 000..09cfed2
--- /dev/null
+++ b/modules/role/manifests/dumps/generation/worker/beta_testbed.pp
@@ -0,0 +1,23 @@
+# == Class role::dumps::generation::worker::beta_testbed
+#
+# Set up a dumps snapshot instance as a testbed.
+#
+# Because this class uses mediawiki, the class
+# role::beta::mediawiki must be applied to the instance
+# as well.  The role can't be included in this class for
+# style reasons.
+#
+# You should be using at least a m1.medium instance,
+# giving some extra lvm space locally mounted on
+# which to write the dumps.
+#
+# filtertags: labs-project-deployment-prep
+class role::dumps::generation::worker::beta_testbed {
+include standard
+
+include profile::dumps::generation::worker::common
+
+system::role { 'dumps::generation::worker::beta_testbed':
+description => 'beta testbed for dumps of XML/SQL wiki content',
+}
+}

-- 
To view, visit https://gerrit.wikimedia.org/r/402576
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I513f4edaa7decebe01702b4f587860952c2b1d81
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: ArielGlenn 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: restructure the snapshot dumps data directory setup

[ArielGlenn (Code Review)]

ArielGlenn has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/402574 )

Change subject: restructure the snapshot dumps data directory setup
..


restructure the snapshot dumps data directory setup

Account for nfs mounts, labs lvm setups, or local partitions.
Toss the attempted labs data lvm profile created earlier today,
now there's labs specific code in a common profile (eww) but
that's the breaks.

Change-Id: I02cb6353edc49b2bd4cd0e21d36c9a2e1c309456
---
M hieradata/common.yaml
M modules/profile/manifests/dumps/generation/worker/common.pp
D modules/profile/manifests/dumps/generation/worker/labs_localdisk.pp
A modules/snapshot/manifests/dumps/datamount.pp
D modules/snapshot/manifests/dumps/nfsmount.pp
5 files changed, 47 insertions(+), 46 deletions(-)

Approvals:
  ArielGlenn: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/hieradata/common.yaml b/hieradata/common.yaml
index 71a9a87..24678c8 100644
--- a/hieradata/common.yaml
+++ b/hieradata/common.yaml
@@ -350,6 +350,7 @@
 external:
   - odysseus.ip6.fi.muni.cz
   - poincare.acc.umu.se
+dumps_datadir_mount_type: nfs
 dumps_nfs_server: dumpsdata1001.eqiad.wmnet
 dumps_managed_subdirs: []
 
diff --git a/modules/profile/manifests/dumps/generation/worker/common.pp 
b/modules/profile/manifests/dumps/generation/worker/common.pp
index b1d9150..ac5e011 100644
--- a/modules/profile/manifests/dumps/generation/worker/common.pp
+++ b/modules/profile/manifests/dumps/generation/worker/common.pp
@@ -1,6 +1,7 @@
 class profile::dumps::generation::worker::common(
 $nfs_server = hiera('dumps_nfs_server'),
 $managed_subdirs = hiera('dumps_managed_subdirs'),
+$datadir_mount_type = hiera('dumps_datadir_mount_type'),
 $php = hiera('snapshot::dumps::php'),
 ) {
 # mw packages and dependencies
@@ -12,13 +13,15 @@
 
 class { '::dumpsuser': }
 
-snapshot::dumps::nfsmount { 'dumpsdatamount':
+snapshot::dumps::datamount { 'dumpsdatamount':
 mountpoint  => $xmldumpsmount,
+mount_type  => $datadir_mount_type,
 server  => $nfs_server,
 managed_subdirs => $managed_subdirs,
 user=> 'dumpsgen',
 group   => 'dumpsgen',
 }
+
 # dataset server config files,
 # stages files, dblists, html templates
 class { '::snapshot::dumps::dirs':
diff --git 
a/modules/profile/manifests/dumps/generation/worker/labs_localdisk.pp 
b/modules/profile/manifests/dumps/generation/worker/labs_localdisk.pp
deleted file mode 100644
index 447fb65..000
--- a/modules/profile/manifests/dumps/generation/worker/labs_localdisk.pp
+++ /dev/null
@@ -1,10 +0,0 @@
-# Provide space for writing dumps to local filesystem
-#
-# filtertags: labs-common
-class profile::dumps::generation::worker::labs_localdisk {
-$xmldumpsmount = '/mnt/dumpsdata'
-
-labs_lvm::volume { 'data-local-disk':
-mountat => $xmldumpsmount,
-}
-}
diff --git a/modules/snapshot/manifests/dumps/datamount.pp 
b/modules/snapshot/manifests/dumps/datamount.pp
new file mode 100644
index 000..42adc14
--- /dev/null
+++ b/modules/snapshot/manifests/dumps/datamount.pp
@@ -0,0 +1,42 @@
+define snapshot::dumps::datamount(
+$mountpoint = undef,
+$mount_type = undef,
+$server = undef,
+$managed_subdirs = [],
+$user = undef,
+$group = undef,
+) {
+if ($mount_type == 'local' or $mount_type == 'nfs') {
+file { [ $mountpoint ]:
+ensure => 'directory',
+}
+}
+elsif ($mount_type == 'labslvm') {
+labs_lvm::volume { 'data-local-disk':
+mountat => $mountpoint,
+}
+}
+
+if ($mount_type == 'nfs') {
+require_package('nfs-common')
+
+mount { $mountpoint:
+ensure   => 'mounted',
+device   => "${server}:/data",
+fstype   => 'nfs',
+name => $mountpoint,
+options  => 'bg,hard,tcp,rsize=8192,wsize=8192,intr,nfsvers=3',
+require  => File[$mountpoint],
+remounts => false,
+}
+}
+
+if ($managed_subdirs) {
+file { $managed_subdirs:
+ensure => 'directory',
+mode   => '0755',
+owner  => $user,
+group  => $group,
+}
+}
+}
diff --git a/modules/snapshot/manifests/dumps/nfsmount.pp 
b/modules/snapshot/manifests/dumps/nfsmount.pp
deleted file mode 100644
index 86c7f21..000
--- a/modules/snapshot/manifests/dumps/nfsmount.pp
+++ /dev/null
@@ -1,35 +0,0 @@
-define snapshot::dumps::nfsmount(
-$mountpoint = undef,
-$server = undef,
-$managed_subdirs = [],
-$user = undef,
-$group = undef,
-) {
-require_package('nfs-common')
-
-file { [ $mountpoint ]:
-ensure => 'directory',
-}
-
-if (defined('$server') and $server != '') {
-mount { $mountpoint:
-ensure   => 'mounted',
-   

[MediaWiki-commits] [Gerrit] operations/puppet[production]: restructure the snapshot dumps data directory setup

[ArielGlenn (Code Review)]

ArielGlenn has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/402574 )

Change subject: restructure the snapshot dumps data directory setup
..

restructure the snapshot dumps data directory setup

Account for nfs mounts, labs lvm setups, or local partitions.
Toss the attempted labs data lvm profile created earlier today,
now there's labs specific code in a common profile (eww) but
that's the breaks.

Change-Id: I02cb6353edc49b2bd4cd0e21d36c9a2e1c309456
---
M hieradata/common.yaml
M modules/profile/manifests/dumps/generation/worker/common.pp
D modules/profile/manifests/dumps/generation/worker/labs_localdisk.pp
A modules/snapshot/manifests/dumps/datamount.pp
D modules/snapshot/manifests/dumps/nfsmount.pp
5 files changed, 47 insertions(+), 46 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/74/402574/1

diff --git a/hieradata/common.yaml b/hieradata/common.yaml
index 71a9a87..24678c8 100644
--- a/hieradata/common.yaml
+++ b/hieradata/common.yaml
@@ -350,6 +350,7 @@
 external:
   - odysseus.ip6.fi.muni.cz
   - poincare.acc.umu.se
+dumps_datadir_mount_type: nfs
 dumps_nfs_server: dumpsdata1001.eqiad.wmnet
 dumps_managed_subdirs: []
 
diff --git a/modules/profile/manifests/dumps/generation/worker/common.pp 
b/modules/profile/manifests/dumps/generation/worker/common.pp
index b1d9150..ac5e011 100644
--- a/modules/profile/manifests/dumps/generation/worker/common.pp
+++ b/modules/profile/manifests/dumps/generation/worker/common.pp
@@ -1,6 +1,7 @@
 class profile::dumps::generation::worker::common(
 $nfs_server = hiera('dumps_nfs_server'),
 $managed_subdirs = hiera('dumps_managed_subdirs'),
+$datadir_mount_type = hiera('dumps_datadir_mount_type'),
 $php = hiera('snapshot::dumps::php'),
 ) {
 # mw packages and dependencies
@@ -12,13 +13,15 @@
 
 class { '::dumpsuser': }
 
-snapshot::dumps::nfsmount { 'dumpsdatamount':
+snapshot::dumps::datamount { 'dumpsdatamount':
 mountpoint  => $xmldumpsmount,
+mount_type  => $datadir_mount_type,
 server  => $nfs_server,
 managed_subdirs => $managed_subdirs,
 user=> 'dumpsgen',
 group   => 'dumpsgen',
 }
+
 # dataset server config files,
 # stages files, dblists, html templates
 class { '::snapshot::dumps::dirs':
diff --git 
a/modules/profile/manifests/dumps/generation/worker/labs_localdisk.pp 
b/modules/profile/manifests/dumps/generation/worker/labs_localdisk.pp
deleted file mode 100644
index 447fb65..000
--- a/modules/profile/manifests/dumps/generation/worker/labs_localdisk.pp
+++ /dev/null
@@ -1,10 +0,0 @@
-# Provide space for writing dumps to local filesystem
-#
-# filtertags: labs-common
-class profile::dumps::generation::worker::labs_localdisk {
-$xmldumpsmount = '/mnt/dumpsdata'
-
-labs_lvm::volume { 'data-local-disk':
-mountat => $xmldumpsmount,
-}
-}
diff --git a/modules/snapshot/manifests/dumps/datamount.pp 
b/modules/snapshot/manifests/dumps/datamount.pp
new file mode 100644
index 000..374bf6e
--- /dev/null
+++ b/modules/snapshot/manifests/dumps/datamount.pp
@@ -0,0 +1,42 @@
+define snapshot::dumps::datamount(
+$mountpoint = undef,
+$mount_type = undef,
+$server = undef,
+$managed_subdirs = [],
+$user = undef,
+$group = undef,
+) {
+if ($mount_type == 'local' or $mount_type == 'nfs') {
+file { [ $mountpoint ]:
+ensure => 'directory',
+}
+}
+elsif ($mount_type == 'labslvm') {
+labs_lvm::volume { 'data-local-disk':
+mountat => $mountpoint,
+}
+}
+
+if ($mount_type == 'nfs') {
+require_package('nfs-common')
+
+mount { $mountpoint:
+ensure   => 'mounted',
+device   => "${server}:/data",
+fstype   => 'nfs',
+name => $mountpoint,
+options  => 'bg,hard,tcp,rsize=8192,wsize=8192,intr,nfsvers=3',
+require  => File[$mountpoint],
+remounts => false,
+}
+}
+
+if (defined($managed_subdirs) and $managed_subdirs) {
+file { $managed_subdirs:
+ensure => 'directory',
+mode   => '0755',
+owner  => $user,
+group  => $group,
+}
+}
+}
diff --git a/modules/snapshot/manifests/dumps/nfsmount.pp 
b/modules/snapshot/manifests/dumps/nfsmount.pp
deleted file mode 100644
index 86c7f21..000
--- a/modules/snapshot/manifests/dumps/nfsmount.pp
+++ /dev/null
@@ -1,35 +0,0 @@
-define snapshot::dumps::nfsmount(
-$mountpoint = undef,
-$server = undef,
-$managed_subdirs = [],
-$user = undef,
-$group = undef,
-) {
-require_package('nfs-common')
-
-file { [ $mountpoint ]:
-ensure => 'directory',
-}
-
-if (defined('$server') and $server != '') {
-mount { $mountpoint:
-  

[MediaWiki-commits] [Gerrit] operations/puppet[production]: make the snapshot labs lvm stuff specific to a snapshot profile

[ArielGlenn (Code Review)]

ArielGlenn has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/402569 )

Change subject: make the snapshot labs lvm stuff specific to a snapshot profile
..


make the snapshot labs lvm stuff specific to a snapshot profile

Didn't like the arrangement in previous commit much, this is clearer
Also use the standard mount point (the previous commit would have
required an additional symlink, no need for that)

Change-Id: Ibcb6eb56226321e57209380e17197dbf50bea59b
---
A modules/profile/manifests/dumps/generation/worker/labs_localdisk.pp
D modules/profile/manifests/labs/lvm/data.pp
2 files changed, 10 insertions(+), 8 deletions(-)

Approvals:
  ArielGlenn: Looks good to me, approved
  jenkins-bot: Verified



diff --git 
a/modules/profile/manifests/dumps/generation/worker/labs_localdisk.pp 
b/modules/profile/manifests/dumps/generation/worker/labs_localdisk.pp
new file mode 100644
index 000..447fb65
--- /dev/null
+++ b/modules/profile/manifests/dumps/generation/worker/labs_localdisk.pp
@@ -0,0 +1,10 @@
+# Provide space for writing dumps to local filesystem
+#
+# filtertags: labs-common
+class profile::dumps::generation::worker::labs_localdisk {
+$xmldumpsmount = '/mnt/dumpsdata'
+
+labs_lvm::volume { 'data-local-disk':
+mountat => $xmldumpsmount,
+}
+}
diff --git a/modules/profile/manifests/labs/lvm/data.pp 
b/modules/profile/manifests/labs/lvm/data.pp
deleted file mode 100644
index f1e04ff..000
--- a/modules/profile/manifests/labs/lvm/data.pp
+++ /dev/null
@@ -1,8 +0,0 @@
-# Allocate all of the instance's extra space as /data
-#
-# filtertags: labs-common
-class profile::labs::lvm::data {
-labs_lvm::volume { 'data-local-disk':
-mountat => '/data',
-}
-}

-- 
To view, visit https://gerrit.wikimedia.org/r/402569
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ibcb6eb56226321e57209380e17197dbf50bea59b
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: ArielGlenn 
Gerrit-Reviewer: ArielGlenn 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: make the snapshot labs lvm stuff specific to a snapshot profile

[ArielGlenn (Code Review)]

ArielGlenn has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/402569 )

Change subject: make the snapshot labs lvm stuff specific to a snapshot profile
..

make the snapshot labs lvm stuff specific to a snapshot profile

Didn't like the arrangement in previous commit much, this is clearer
Also use the standard mount point (the previous commit would have
required an additional symlink, no need for that)

Change-Id: Ibcb6eb56226321e57209380e17197dbf50bea59b
---
A modules/profile/manifests/dumps/generation/worker/labs_localdisk.pp
D modules/profile/manifests/labs/lvm/data.pp
2 files changed, 10 insertions(+), 8 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/69/402569/1

diff --git 
a/modules/profile/manifests/dumps/generation/worker/labs_localdisk.pp 
b/modules/profile/manifests/dumps/generation/worker/labs_localdisk.pp
new file mode 100644
index 000..447fb65
--- /dev/null
+++ b/modules/profile/manifests/dumps/generation/worker/labs_localdisk.pp
@@ -0,0 +1,10 @@
+# Provide space for writing dumps to local filesystem
+#
+# filtertags: labs-common
+class profile::dumps::generation::worker::labs_localdisk {
+$xmldumpsmount = '/mnt/dumpsdata'
+
+labs_lvm::volume { 'data-local-disk':
+mountat => $xmldumpsmount,
+}
+}
diff --git a/modules/profile/manifests/labs/lvm/data.pp 
b/modules/profile/manifests/labs/lvm/data.pp
deleted file mode 100644
index f1e04ff..000
--- a/modules/profile/manifests/labs/lvm/data.pp
+++ /dev/null
@@ -1,8 +0,0 @@
-# Allocate all of the instance's extra space as /data
-#
-# filtertags: labs-common
-class profile::labs::lvm::data {
-labs_lvm::volume { 'data-local-disk':
-mountat => '/data',
-}
-}

-- 
To view, visit https://gerrit.wikimedia.org/r/402569
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ibcb6eb56226321e57209380e17197dbf50bea59b
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: ArielGlenn 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: add labs lvm setup for dumps snapshot instances without nfs ...

[ArielGlenn (Code Review)]

ArielGlenn has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/402489 )

Change subject: add labs lvm setup for dumps snapshot instances without nfs 
filesystems
..


add labs lvm setup for dumps snapshot instances without nfs filesystems

This will give us space to write dumps for testing.

Change-Id: I80426a5972f6d6743ba304e6a7b157b01bd83d73
---
A modules/profile/manifests/labs/lvm/data.pp
1 file changed, 8 insertions(+), 0 deletions(-)

Approvals:
  ArielGlenn: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/profile/manifests/labs/lvm/data.pp 
b/modules/profile/manifests/labs/lvm/data.pp
new file mode 100644
index 000..f1e04ff
--- /dev/null
+++ b/modules/profile/manifests/labs/lvm/data.pp
@@ -0,0 +1,8 @@
+# Allocate all of the instance's extra space as /data
+#
+# filtertags: labs-common
+class profile::labs::lvm::data {
+labs_lvm::volume { 'data-local-disk':
+mountat => '/data',
+}
+}

-- 
To view, visit https://gerrit.wikimedia.org/r/402489
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I80426a5972f6d6743ba304e6a7b157b01bd83d73
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: ArielGlenn 
Gerrit-Reviewer: ArielGlenn 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: add labs lvm setup for dumps snapshot instances without nfs ...

[ArielGlenn (Code Review)]

ArielGlenn has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/402489 )

Change subject: add labs lvm setup for dumps snapshot instances without nfs 
filesystems
..

add labs lvm setup for dumps snapshot instances without nfs filesystems

This will give us space to write dumps for testing.

Change-Id: I80426a5972f6d6743ba304e6a7b157b01bd83d73
---
A modules/profile/manifests/labs/lvm/data.pp
1 file changed, 8 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/89/402489/1

diff --git a/modules/profile/manifests/labs/lvm/data.pp 
b/modules/profile/manifests/labs/lvm/data.pp
new file mode 100644
index 000..f1e04ff
--- /dev/null
+++ b/modules/profile/manifests/labs/lvm/data.pp
@@ -0,0 +1,8 @@
+# Allocate all of the instance's extra space as /data
+#
+# filtertags: labs-common
+class profile::labs::lvm::data {
+labs_lvm::volume { 'data-local-disk':
+mountat => '/data',
+}
+}

-- 
To view, visit https://gerrit.wikimedia.org/r/402489
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I80426a5972f6d6743ba304e6a7b157b01bd83d73
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: ArielGlenn 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...scap[master]: add explicit name of dumps key id for scap

[ArielGlenn (Code Review)]

ArielGlenn has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/402447 )

Change subject: add explicit name of dumps key id for scap
..


add explicit name of dumps key id for scap

Change-Id: I4224f3e7c16eeed75b97ace9e3465f64361f685c
---
M scap.cfg
1 file changed, 1 insertion(+), 0 deletions(-)

Approvals:
  Thcipriani: Looks good to me, but someone else must approve
  ArielGlenn: Verified; Looks good to me, approved



diff --git a/scap.cfg b/scap.cfg
index ac1f466..86373e1 100644
--- a/scap.cfg
+++ b/scap.cfg
@@ -6,3 +6,4 @@
 
 [deployment-prep.eqiad.wmflabs]
 dsh_targets: dumps_targets_betacluster
+keyholder_key: dumpsdeploy

-- 
To view, visit https://gerrit.wikimedia.org/r/402447
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I4224f3e7c16eeed75b97ace9e3465f64361f685c
Gerrit-PatchSet: 1
Gerrit-Project: operations/dumps/scap
Gerrit-Branch: master
Gerrit-Owner: ArielGlenn 
Gerrit-Reviewer: ArielGlenn 
Gerrit-Reviewer: Thcipriani 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...scap[master]: add explicit name of dumps key id for scap

[ArielGlenn (Code Review)]

ArielGlenn has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/402447 )

Change subject: add explicit name of dumps key id for scap
..

add explicit name of dumps key id for scap

Change-Id: I4224f3e7c16eeed75b97ace9e3465f64361f685c
---
M scap.cfg
1 file changed, 1 insertion(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/dumps/scap 
refs/changes/47/402447/1

diff --git a/scap.cfg b/scap.cfg
index ac1f466..86373e1 100644
--- a/scap.cfg
+++ b/scap.cfg
@@ -6,3 +6,4 @@
 
 [deployment-prep.eqiad.wmflabs]
 dsh_targets: dumps_targets_betacluster
+keyholder_key: dumpsdeploy

-- 
To view, visit https://gerrit.wikimedia.org/r/402447
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I4224f3e7c16eeed75b97ace9e3465f64361f685c
Gerrit-PatchSet: 1
Gerrit-Project: operations/dumps/scap
Gerrit-Branch: master
Gerrit-Owner: ArielGlenn 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] labs/private[master]: add scap keys for dumpsdeploy for beta

[ArielGlenn (Code Review)]

ArielGlenn has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/402426 )

Change subject: add scap keys for dumpsdeploy for beta
..


add scap keys for dumpsdeploy for beta

Change-Id: I8ea442cc38dbfc652e6222b431a52866a93def76
---
M modules/secret/secrets/keyholder/dumpsdeploy
M modules/secret/secrets/keyholder/dumpsdeploy.pub
2 files changed, 53 insertions(+), 4 deletions(-)

Approvals:
  ArielGlenn: Verified; Looks good to me, approved



diff --git a/modules/secret/secrets/keyholder/dumpsdeploy 
b/modules/secret/secrets/keyholder/dumpsdeploy
index 2a6330d..023d96a 100644
--- a/modules/secret/secrets/keyholder/dumpsdeploy
+++ b/modules/secret/secrets/keyholder/dumpsdeploy
@@ -1,4 +1,54 @@
 -BEGIN RSA PRIVATE KEY-
-SNAKEOIL
--END RSA PRIVATE KEY-
+Proc-Type: 4,ENCRYPTED
+DEK-Info: AES-128-CBC,B163082F7412A16197AF66B3CDA3A759
 
+hu/Vq0O69xjFzl+AGPU16Z0nt0iNYIcwKK8cENbciebv+CLDmbVigifBr3uSlle2
+d6d+oGVDDePNVMSHYz64M4P72WfthFkPNOb4Q8QND3oLExLWBnUlV+RIZm2LDN28
+hl7ZfJnTW5+OIuPIuZxkJD6MINqUc2++2cK+NTZQG9mO0qHf4NFxAKBRGD3hk1cW
+obPHe3i7Z3igJgvjxBwWmuJEnKzOwvbK+J+tRS51O3OARKNh8W+AzIZ1hEILG9qR
+0dIrZcQS9IgER8Xc1Rv6AGARQvaqc9fyMDVRPFmZwr1+bTY7edl7tT1Hsxntl0c0
+O3GzhBrU64Fdhf2tO+Sm9J0nrdB9jtklhrZKaGzprm/uvIKxaYCAX6UBzueMJ7cy
+aucD0yVeeeK616a/SWGVtX3TKY+1j9dZDPRrOuPDFvPd1t/qqnCFMWcqQxo3p7AT
+SOJmBIT/GoH08Y63jOUqB9+UShGlXVlO4JmkxICAN2r8D78s7WfOYUz3cW/NYDmm
+gUb+cpjqs0nlU2IhGvymn20utgASGxV2beu0vMIJPZfvz5xnJ1L/BMJZ3lyKRMU3
+GMDCZxYaiKYq8dp+UPVGpINpAX9VMRQvuSINwad9SsDDsa9G4EiZ0QNisb0EuUx3
+/UDPvJ36iXXbMvSIYChSODvsUngHJQ4ElM2c9TqX0HsaKLaIUkq9Y9pcCUHMSPfv
+WGwiVyNW336fNHPigeCSB4jeVyfWQl1r5JFXUK+W6G16GY8x5TPTm/xVBhZjpQbL
+kZI1Z/F56pCwzZG0OrJI173/pXyufE+Hvv12PpZuOuakbAKLqIk3PfAJlUVYXfzT
+cIm85UBqNBhKQ5JDGlVVISbXEOnozFL2THlvvOO/wVQeHDZUmy+xnTLdUynXMv74
+V1u6EJZRcOzbWZiydCmDProLAPXp3xsjDEE0RTIxLyTVKVLAfwiv9J/YncN6rNIZ
+t/pdn7SrLmKvFYYVzGlVuRwzSS/mEjV9CpufemXg5VvZ5iK50XdhBMOOWqtf+pKT
+pK62y7SQuICldzxnGoZgGQUOwuHTSIYZymZ+MeRtYMHd4BbSGCFlEotMwHxYPHd4
+BFvmfI3IDq065k1MHUXKmcCUg37Y4UGiY9EwF0iYmtoIWtTIpojd3qmI//fkZimN
+jFcPvKvj5KDi7ONeYCYSHw3Kghr4rzKv44+WtAU4Ea+Pskzmx3fCCjNs45mnqFT2
+79krk+7544C0fF67AjnY7J9gYAsqVbrlkegvgG9JWx2QLDkxutQhaJQNgjbioozF
+7ymg+8SOSdpELfMqXKbnXSaRcBP5PuOBoJc6SFx6PNSUVYtiuJzvvzVsrnHyqzyx
+r7C4jFZ4U5b9lR9povEJmDWby/OBgHndhARiLfSW15vBuC8cjvV0hjnaf8SlAWPi
+fvL+ObFMBKqagfc1h2KOHubY9qcrNYH0/jSrSYnAijDO8zBfoaBjR5xGpHMlFRFy
+QYwr9SS8yzvFhDDfI1O/znk1I+J0hOWk1LD4XRKzfBmNI61HGsvJmPW3uFoAVBm6
+PGZDLQksYgOBtDSAgEkNjdmrNn3B+rv8nsWnPkX00Q6rxlLCcE/NeDxI0nihRmPm
+YRI2DI6j2qw1StKbF3T7xO+3sPpt507HoE597vHPRAykZ0IlbVzW4sGFmVEEZCP7
+gKBIa3yVd38zVtN6JWzObJfOaMnQnlKhpv5ApbQJM4GiTmsc2Whuv55yHZOi3XmP
+1gWdZ9HgcUomK54hgYuZ5Do/eIq1whCMG+9PJYbE3VtXOHoxZQBE9CRjjHdEOrUX
+DZ3EGM9RAOjMVMamAyABX67OSgVsL8F1sKozw9LsbmQ1YpT0hne1PlIC/Nj3EbcW
+P6F/DpNYfprP05f9487CEsYxRYFAmHXXVXenJp0BaIqiHErhlIsRWUgUeOtIFwq3
+SzE2CYqEiJa0yqG7/92voI830aE1ixLVaBUaiSae8JSt3o+qSHFVgXmr2N0F4+b+
+yEzdvX+b5PNwS2UL9OaYZrjveZQRgFx090gEtcYeTMOmFPqV/wu5iAP2+XR+9ChZ
+DP/MuKhFnF0sGBS3yS9aChEi/oxNjRPSJSGm/NRuBmE7KSBQqzfPjyIIqMAxdsDC
+4zfrEvixyQ0V9IYIDfxOZp+LM3SgTMgApIV1gefAtPZo0l4m4sFLqQvkO1xMYhrT
+JjitzSUoKC9UxPbn3287gPtUvcxeb07DdXNYhcTIzFQX/lgo637T6GBikwdndwbB
+FNn2sBt/aKPk3+2p3ciVpY0E/IoZw8PlV6ddByAVJ5owFKUKSu5h6bHZ8FzeRefS
++ZRHPO6zQ/BHPhnk1J+H1h72Tbl3X0yx6U8kAX93TvIAG33aUsZpcrwWPIaN9icy
+NtyPl63GMv440io+hhxHV0e/CNjeqIPJh0SDICoOCWi8uB+QQtkjPi6RPqZQwWr1
+U+y8a6tzGDPSiL1VHFsqnYNipHNP+y5EPBOY6EQ5JfVL/fzVfmucXj80ID728QzD
+UwW/Pw7vZzIsqSsjDhbDYFpueYM8aFmEIDFKM3v0dcWXk3rGX8Hn7exy1aP7Oz7b
+Y+vNfiTXoGeQNADgmM9Yu6CUHUeP+P7pKmyA7fAr8OjT1svnKadkuYsB0CHOYpCd
+EAVe6XARznFBfW87lqY1skAnlCjk8hpt0EZOpLYLk4blofD+DTRMrfOyrtQ36KN5
+A8KPYeTbc2s3IeT1pXhfULcIuK55SCh/gPIBl4GN/FTelsKLzWaJcGboL55q08kp
+nvrEvzINHIZ9KixQh7vu62UMZMjRE4nYvJFoKHahEpeP4TpWEiJzwYca2/Sx4yzB
+POD8t2CfeQnl9NfIUUHPlPNSxRmyyelxSPeyirU8W4/lFVH/h97pAuG4uXTO19Uz
+vF/admj5x8fkHl2T6JRXjpDhYV1sVuWangrurYTEXOY3DrHyq2q/yU6GTl9Iehr9
+5J+8tiK3hpbVRDph0vgxaYacZPBhRXpsQ/pFDZLoqgIspa3BvTbUtqX43I6X4V8s
+1CnivZN9mFSWydsc0JbPQj7cbGkUgwy99YVUr+qdm415ECzP8LlPk9Q/MC8wbNs/
+S2d43Or2LNQITe7z+yVDaRzrAh1eCACEcoyUZivunnZRaN1VoyffoukKZGxFyKP4
+-END RSA PRIVATE KEY-
diff --git a/modules/secret/secrets/keyholder/dumpsdeploy.pub 
b/modules/secret/secrets/keyholder/dumpsdeploy.pub
index 96c961c..1fce40b 100644
--- a/modules/secret/secrets/keyholder/dumpsdeploy.pub
+++ b/modules/secret/secrets/keyholder/dumpsdeploy.pub
@@ -1,2 +1 @@
-ssh-rsa 
XXX 
dumpsdeploy
-
+ssh-rsa 

[MediaWiki-commits] [Gerrit] labs/private[master]: add scap keys for dumpsdeploy for beta

[ArielGlenn (Code Review)]

ArielGlenn has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/402426 )

Change subject: add scap keys for dumpsdeploy for beta
..

add scap keys for dumpsdeploy for beta

Change-Id: I8ea442cc38dbfc652e6222b431a52866a93def76
---
M modules/secret/secrets/keyholder/dumpsdeploy
M modules/secret/secrets/keyholder/dumpsdeploy.pub
2 files changed, 53 insertions(+), 4 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/labs/private 
refs/changes/26/402426/1

diff --git a/modules/secret/secrets/keyholder/dumpsdeploy 
b/modules/secret/secrets/keyholder/dumpsdeploy
index 2a6330d..023d96a 100644
--- a/modules/secret/secrets/keyholder/dumpsdeploy
+++ b/modules/secret/secrets/keyholder/dumpsdeploy
@@ -1,4 +1,54 @@
 -BEGIN RSA PRIVATE KEY-
-SNAKEOIL
--END RSA PRIVATE KEY-
+Proc-Type: 4,ENCRYPTED
+DEK-Info: AES-128-CBC,B163082F7412A16197AF66B3CDA3A759
 
+hu/Vq0O69xjFzl+AGPU16Z0nt0iNYIcwKK8cENbciebv+CLDmbVigifBr3uSlle2
+d6d+oGVDDePNVMSHYz64M4P72WfthFkPNOb4Q8QND3oLExLWBnUlV+RIZm2LDN28
+hl7ZfJnTW5+OIuPIuZxkJD6MINqUc2++2cK+NTZQG9mO0qHf4NFxAKBRGD3hk1cW
+obPHe3i7Z3igJgvjxBwWmuJEnKzOwvbK+J+tRS51O3OARKNh8W+AzIZ1hEILG9qR
+0dIrZcQS9IgER8Xc1Rv6AGARQvaqc9fyMDVRPFmZwr1+bTY7edl7tT1Hsxntl0c0
+O3GzhBrU64Fdhf2tO+Sm9J0nrdB9jtklhrZKaGzprm/uvIKxaYCAX6UBzueMJ7cy
+aucD0yVeeeK616a/SWGVtX3TKY+1j9dZDPRrOuPDFvPd1t/qqnCFMWcqQxo3p7AT
+SOJmBIT/GoH08Y63jOUqB9+UShGlXVlO4JmkxICAN2r8D78s7WfOYUz3cW/NYDmm
+gUb+cpjqs0nlU2IhGvymn20utgASGxV2beu0vMIJPZfvz5xnJ1L/BMJZ3lyKRMU3
+GMDCZxYaiKYq8dp+UPVGpINpAX9VMRQvuSINwad9SsDDsa9G4EiZ0QNisb0EuUx3
+/UDPvJ36iXXbMvSIYChSODvsUngHJQ4ElM2c9TqX0HsaKLaIUkq9Y9pcCUHMSPfv
+WGwiVyNW336fNHPigeCSB4jeVyfWQl1r5JFXUK+W6G16GY8x5TPTm/xVBhZjpQbL
+kZI1Z/F56pCwzZG0OrJI173/pXyufE+Hvv12PpZuOuakbAKLqIk3PfAJlUVYXfzT
+cIm85UBqNBhKQ5JDGlVVISbXEOnozFL2THlvvOO/wVQeHDZUmy+xnTLdUynXMv74
+V1u6EJZRcOzbWZiydCmDProLAPXp3xsjDEE0RTIxLyTVKVLAfwiv9J/YncN6rNIZ
+t/pdn7SrLmKvFYYVzGlVuRwzSS/mEjV9CpufemXg5VvZ5iK50XdhBMOOWqtf+pKT
+pK62y7SQuICldzxnGoZgGQUOwuHTSIYZymZ+MeRtYMHd4BbSGCFlEotMwHxYPHd4
+BFvmfI3IDq065k1MHUXKmcCUg37Y4UGiY9EwF0iYmtoIWtTIpojd3qmI//fkZimN
+jFcPvKvj5KDi7ONeYCYSHw3Kghr4rzKv44+WtAU4Ea+Pskzmx3fCCjNs45mnqFT2
+79krk+7544C0fF67AjnY7J9gYAsqVbrlkegvgG9JWx2QLDkxutQhaJQNgjbioozF
+7ymg+8SOSdpELfMqXKbnXSaRcBP5PuOBoJc6SFx6PNSUVYtiuJzvvzVsrnHyqzyx
+r7C4jFZ4U5b9lR9povEJmDWby/OBgHndhARiLfSW15vBuC8cjvV0hjnaf8SlAWPi
+fvL+ObFMBKqagfc1h2KOHubY9qcrNYH0/jSrSYnAijDO8zBfoaBjR5xGpHMlFRFy
+QYwr9SS8yzvFhDDfI1O/znk1I+J0hOWk1LD4XRKzfBmNI61HGsvJmPW3uFoAVBm6
+PGZDLQksYgOBtDSAgEkNjdmrNn3B+rv8nsWnPkX00Q6rxlLCcE/NeDxI0nihRmPm
+YRI2DI6j2qw1StKbF3T7xO+3sPpt507HoE597vHPRAykZ0IlbVzW4sGFmVEEZCP7
+gKBIa3yVd38zVtN6JWzObJfOaMnQnlKhpv5ApbQJM4GiTmsc2Whuv55yHZOi3XmP
+1gWdZ9HgcUomK54hgYuZ5Do/eIq1whCMG+9PJYbE3VtXOHoxZQBE9CRjjHdEOrUX
+DZ3EGM9RAOjMVMamAyABX67OSgVsL8F1sKozw9LsbmQ1YpT0hne1PlIC/Nj3EbcW
+P6F/DpNYfprP05f9487CEsYxRYFAmHXXVXenJp0BaIqiHErhlIsRWUgUeOtIFwq3
+SzE2CYqEiJa0yqG7/92voI830aE1ixLVaBUaiSae8JSt3o+qSHFVgXmr2N0F4+b+
+yEzdvX+b5PNwS2UL9OaYZrjveZQRgFx090gEtcYeTMOmFPqV/wu5iAP2+XR+9ChZ
+DP/MuKhFnF0sGBS3yS9aChEi/oxNjRPSJSGm/NRuBmE7KSBQqzfPjyIIqMAxdsDC
+4zfrEvixyQ0V9IYIDfxOZp+LM3SgTMgApIV1gefAtPZo0l4m4sFLqQvkO1xMYhrT
+JjitzSUoKC9UxPbn3287gPtUvcxeb07DdXNYhcTIzFQX/lgo637T6GBikwdndwbB
+FNn2sBt/aKPk3+2p3ciVpY0E/IoZw8PlV6ddByAVJ5owFKUKSu5h6bHZ8FzeRefS
++ZRHPO6zQ/BHPhnk1J+H1h72Tbl3X0yx6U8kAX93TvIAG33aUsZpcrwWPIaN9icy
+NtyPl63GMv440io+hhxHV0e/CNjeqIPJh0SDICoOCWi8uB+QQtkjPi6RPqZQwWr1
+U+y8a6tzGDPSiL1VHFsqnYNipHNP+y5EPBOY6EQ5JfVL/fzVfmucXj80ID728QzD
+UwW/Pw7vZzIsqSsjDhbDYFpueYM8aFmEIDFKM3v0dcWXk3rGX8Hn7exy1aP7Oz7b
+Y+vNfiTXoGeQNADgmM9Yu6CUHUeP+P7pKmyA7fAr8OjT1svnKadkuYsB0CHOYpCd
+EAVe6XARznFBfW87lqY1skAnlCjk8hpt0EZOpLYLk4blofD+DTRMrfOyrtQ36KN5
+A8KPYeTbc2s3IeT1pXhfULcIuK55SCh/gPIBl4GN/FTelsKLzWaJcGboL55q08kp
+nvrEvzINHIZ9KixQh7vu62UMZMjRE4nYvJFoKHahEpeP4TpWEiJzwYca2/Sx4yzB
+POD8t2CfeQnl9NfIUUHPlPNSxRmyyelxSPeyirU8W4/lFVH/h97pAuG4uXTO19Uz
+vF/admj5x8fkHl2T6JRXjpDhYV1sVuWangrurYTEXOY3DrHyq2q/yU6GTl9Iehr9
+5J+8tiK3hpbVRDph0vgxaYacZPBhRXpsQ/pFDZLoqgIspa3BvTbUtqX43I6X4V8s
+1CnivZN9mFSWydsc0JbPQj7cbGkUgwy99YVUr+qdm415ECzP8LlPk9Q/MC8wbNs/
+S2d43Or2LNQITe7z+yVDaRzrAh1eCACEcoyUZivunnZRaN1VoyffoukKZGxFyKP4
+-END RSA PRIVATE KEY-
diff --git a/modules/secret/secrets/keyholder/dumpsdeploy.pub 
b/modules/secret/secrets/keyholder/dumpsdeploy.pub
index 96c961c..1fce40b 100644
--- a/modules/secret/secrets/keyholder/dumpsdeploy.pub
+++ b/modules/secret/secrets/keyholder/dumpsdeploy.pub
@@ -1,2 +1 @@
-ssh-rsa 
XXX 
dumpsdeploy
-
+ssh-rsa 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: snapshot hosts: empty nfs server name means no mount

[ArielGlenn (Code Review)]

ArielGlenn has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/402322 )

Change subject: snapshot hosts: empty nfs server name means no mount
..


snapshot hosts: empty nfs server name means no mount

Change-Id: Iccf4de539955253e200f42d3c231f95014240705
---
M modules/snapshot/manifests/dumps/nfsmount.pp
1 file changed, 1 insertion(+), 1 deletion(-)

Approvals:
  ArielGlenn: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/snapshot/manifests/dumps/nfsmount.pp 
b/modules/snapshot/manifests/dumps/nfsmount.pp
index 92b910e..86c7f21 100644
--- a/modules/snapshot/manifests/dumps/nfsmount.pp
+++ b/modules/snapshot/manifests/dumps/nfsmount.pp
@@ -11,7 +11,7 @@
 ensure => 'directory',
 }
 
-if (defined('$server')) {
+if (defined('$server') and $server != '') {
 mount { $mountpoint:
 ensure   => 'mounted',
 device   => "${server}:/data",

-- 
To view, visit https://gerrit.wikimedia.org/r/402322
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Iccf4de539955253e200f42d3c231f95014240705
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: ArielGlenn 
Gerrit-Reviewer: ArielGlenn 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: snapshot hosts: empty nfs server name means no mount

[ArielGlenn (Code Review)]

ArielGlenn has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/402322 )

Change subject: snapshot hosts: empty nfs server name means no mount
..

snapshot hosts: empty nfs server name means no mount

Change-Id: Iccf4de539955253e200f42d3c231f95014240705
---
M modules/snapshot/manifests/dumps/nfsmount.pp
1 file changed, 1 insertion(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/22/402322/1

diff --git a/modules/snapshot/manifests/dumps/nfsmount.pp 
b/modules/snapshot/manifests/dumps/nfsmount.pp
index 92b910e..86c7f21 100644
--- a/modules/snapshot/manifests/dumps/nfsmount.pp
+++ b/modules/snapshot/manifests/dumps/nfsmount.pp
@@ -11,7 +11,7 @@
 ensure => 'directory',
 }
 
-if (defined('$server')) {
+if (defined('$server') and $server != '') {
 mount { $mountpoint:
 ensure   => 'mounted',
 device   => "${server}:/data",

-- 
To view, visit https://gerrit.wikimedia.org/r/402322
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Iccf4de539955253e200f42d3c231f95014240705
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: ArielGlenn 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: use strict var syntax in snapshot/dumps modules

[ArielGlenn (Code Review)]

ArielGlenn has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/402029 )

Change subject: use strict var syntax in snapshot/dumps modules
..


use strict var syntax in snapshot/dumps modules

Change-Id: I67a0f82a5030902d62c87f9cfcf80be98770b174
---
M modules/snapshot/manifests/dumps/nfsmount.pp
1 file changed, 1 insertion(+), 1 deletion(-)

Approvals:
  ArielGlenn: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/snapshot/manifests/dumps/nfsmount.pp 
b/modules/snapshot/manifests/dumps/nfsmount.pp
index 31f2027..92b910e 100644
--- a/modules/snapshot/manifests/dumps/nfsmount.pp
+++ b/modules/snapshot/manifests/dumps/nfsmount.pp
@@ -11,7 +11,7 @@
 ensure => 'directory',
 }
 
-if ($server != undef) {
+if (defined('$server')) {
 mount { $mountpoint:
 ensure   => 'mounted',
 device   => "${server}:/data",

-- 
To view, visit https://gerrit.wikimedia.org/r/402029
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I67a0f82a5030902d62c87f9cfcf80be98770b174
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: ArielGlenn 
Gerrit-Reviewer: ArielGlenn 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: use strict var syntax in snapshot/dumps modules

[ArielGlenn (Code Review)]

ArielGlenn has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/402029 )

Change subject: use strict var syntax in snapshot/dumps modules
..

use strict var syntax in snapshot/dumps modules

Change-Id: I67a0f82a5030902d62c87f9cfcf80be98770b174
---
M modules/snapshot/manifests/dumps/nfsmount.pp
1 file changed, 1 insertion(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/29/402029/1

diff --git a/modules/snapshot/manifests/dumps/nfsmount.pp 
b/modules/snapshot/manifests/dumps/nfsmount.pp
index 31f2027..931c551 100644
--- a/modules/snapshot/manifests/dumps/nfsmount.pp
+++ b/modules/snapshot/manifests/dumps/nfsmount.pp
@@ -11,7 +11,7 @@
 ensure => 'directory',
 }
 
-if ($server != undef) {
+if (defined($server)) {
 mount { $mountpoint:
 ensure   => 'mounted',
 device   => "${server}:/data",

-- 
To view, visit https://gerrit.wikimedia.org/r/402029
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I67a0f82a5030902d62c87f9cfcf80be98770b174
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: ArielGlenn 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: add dumps repo source to beta scap, add snapshot to beta mw ...

[ArielGlenn (Code Review)]

ArielGlenn has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/400237 )

Change subject: add dumps repo source to beta scap, add snapshot to beta mw scap
..


add dumps repo source to beta scap, add snapshot to beta mw scap

Change-Id: I70e254246cbc8485c9ed572475a8c9f3f4442e26
---
M hieradata/labs/deployment-prep/common.yaml
1 file changed, 9 insertions(+), 0 deletions(-)

Approvals:
  Thcipriani: Looks good to me, but someone else must approve
  ArielGlenn: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/hieradata/labs/deployment-prep/common.yaml 
b/hieradata/labs/deployment-prep/common.yaml
index 5bad6f5..7700115 100644
--- a/hieradata/labs/deployment-prep/common.yaml
+++ b/hieradata/labs/deployment-prep/common.yaml
@@ -203,6 +203,7 @@
 - deployment-tin.deployment-prep.eqiad.wmflabs
 - deployment-mira.deployment-prep.eqiad.wmflabs
 - deployment-videoscaler01.deployment-prep.eqiad.wmflabs
+- deployment-snapshot01.deployment-prep.eqiad.wmflabs
 mediawiki-appserver-canaries:
 hosts:
 - deployment-mediawiki04.deployment-prep.eqiad.wmflabs
@@ -289,6 +290,10 @@
 trusted_groups:
   - deploy-service
 
+  dumpsdeploy:
+trusted_groups:
+  - project-%{::labsproject}
+
 # deployment-prep scap::source declarations.  These are created
 # by the role deployment::server.  Each source listed here
 # will be cloned on the scap deploy server.
@@ -348,6 +353,10 @@
   scholarships/scholarships:
 repository: wikimedia/wikimania-scholarships
 
+  # Xml/sql dumps
+  dumps/dumps:
+repository: operations/dumps
+
 prometheus_nodes:
   - deployment-prometheus01.deployment-prep.eqiad.wmflabs
 profile::recommendation_api::wdqs_uri: http://wdqs-test.wmflabs.org

-- 
To view, visit https://gerrit.wikimedia.org/r/400237
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I70e254246cbc8485c9ed572475a8c9f3f4442e26
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: ArielGlenn 
Gerrit-Reviewer: ArielGlenn 
Gerrit-Reviewer: Thcipriani 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...scap[master]: add wmflabs config for dumps scap

[ArielGlenn (Code Review)]

ArielGlenn has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/400598 )

Change subject: add wmflabs config for dumps scap
..


add wmflabs config for dumps scap

Change-Id: I5b2b4fa35987f64cb74811ec39a98faf35752477
---
A dumps_targets_betacluster
M scap.cfg
2 files changed, 5 insertions(+), 0 deletions(-)

Approvals:
  ArielGlenn: Verified; Looks good to me, approved



diff --git a/dumps_targets_betacluster b/dumps_targets_betacluster
new file mode 100644
index 000..2419e87
--- /dev/null
+++ b/dumps_targets_betacluster
@@ -0,0 +1,2 @@
+deployment-snapshot01.deployment-prep.eqiad.wmflabs
+
diff --git a/scap.cfg b/scap.cfg
index 99348a9..ac1f466 100644
--- a/scap.cfg
+++ b/scap.cfg
@@ -3,3 +3,6 @@
 git_deploy_dir: /srv/deployment
 ssh_user: dumpsgen
 dsh_targets: dumps_targets
+
+[deployment-prep.eqiad.wmflabs]
+dsh_targets: dumps_targets_betacluster

-- 
To view, visit https://gerrit.wikimedia.org/r/400598
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I5b2b4fa35987f64cb74811ec39a98faf35752477
Gerrit-PatchSet: 2
Gerrit-Project: operations/dumps/scap
Gerrit-Branch: master
Gerrit-Owner: ArielGlenn 
Gerrit-Reviewer: ArielGlenn 
Gerrit-Reviewer: Thcipriani 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/dumps[ariel]: prep-dumps-deploy full pylint and pep8 cleanup

[ArielGlenn (Code Review)]

ArielGlenn has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/280101 )

Change subject: prep-dumps-deploy full pylint and pep8 cleanup
..


prep-dumps-deploy full pylint and pep8 cleanup

Change-Id: I1d80b9d4679e6e51ca2ce9df5e9360a3c8ff04e5
---
M deployment/obsolete/prep-dumps-deploy.py
1 file changed, 106 insertions(+), 76 deletions(-)

Approvals:
  ArielGlenn: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/deployment/obsolete/prep-dumps-deploy.py 
b/deployment/obsolete/prep-dumps-deploy.py
index e81edd0..8e393dc 100644
--- a/deployment/obsolete/prep-dumps-deploy.py
+++ b/deployment/obsolete/prep-dumps-deploy.py
@@ -1,56 +1,75 @@
-import os, sys, time, shutil, re
-# this script copies the most recent files into a deployment directory with 
the current date
-# run on fenari/bastion host as root or fail.
+"""
+this script copies the most recent files into
+ a deployment directory with the current date
+run on fenari/bastion host as root or fail.
+"""
 
-basedir = "/home/wikipedia/downloadserver/snapshothosts/dumps"
-deploy = "deploy" # subdir where deployment trees are stored by date
-confs = "confs" # subdir where config files live
+import os
+import sys
+import time
+import shutil
+import re
+
+
+BASEDIR = "/home/wikipedia/downloadserver/snapshothosts/dumps"
+DEPLOY = "deploy"  # subdir where deployment trees are stored by date
+CONFS = "confs"# subdir where config files live
+MONTHNAMES = ["jan", "feb", "mar", "apr", "may", "jun",
+  "jul", "aug", "sep", "oct", "nov", "dec"]
+
 
 class Error(Exception):
 """Base class for exceptions in this module."""
 pass
 
-def dateToDigits(dateString):
-if '-' not in dateString:
+
+def date_to_digits(date_string):
+"""convert date string in form mon-dd- to tuple
+of year, monthnum, day and return it, or None
+on error"""
+if '-' not in date_string:
 return None
-month, day, year  = dateString.split('-', 2)
+month, day, year = date_string.split('-', 2)
 if not month.isdigit():
-if not month in monthNames:
+if month not in MONTHNAMES:
 return None
 else:
-month = int(monthNames.index(month)) + 1
+month = int(MONTHNAMES.index(month)) + 1
 day = int(day)
-return "%s%02d%02d" % ( year, month, day)
+return "%s%02d%02d" % (year, month, day)
 
-def getLatestDeployDate(deploydir):
+
+def get_latest_depl_date(deploydir):
 try:
 subdirs = os.listdir(deploydir)
 except:
 sys.stderr.write("Failed to read contents of %s\n" % deploydir)
 raise
-deployDates = {}
-for d in subdirs:
-if not os.path.isdir(os.path.join(deploydir, d)):
+deploy_dates = {}
+for dname in subdirs:
+if not os.path.isdir(os.path.join(deploydir, dname)):
 continue
 # expect mon-dd-
-canonicalDirName = dateToDigits(d)
-if not canonicalDirName:
+canonical_dirname = date_to_digits(dname)
+if not canonical_dirname:
 continue
-deployDates[canonicalDirName] = d
-if not len(deployDates.keys()):
+deploy_dates[canonical_dirname] = dname
+if not len(deploy_dates.keys()):
 return None
-dates = deployDates.keys()
-dates.sort(reverse = True)
-return deployDates[dates[0]]
+dates = deploy_dates.keys()
+dates.sort(reverse=True)
+return deploy_dates[dates[0]]
+
 
 # modified from the python copytree implementation
-def doCopy(sourcedir, targetdir):
+def do_copy(sourcedir, targetdir):
 """only for regular files, symlinks, dirs:
 will attempt to remove files/symlinks in target dir that
 are to be copied from the source dir, leaving any other contents
 of target dir tree in place"""
 names = os.listdir(sourcedir)
 
+errors = []
 if not os.path.isdir(targetdir):
 # if this fails we want to give up on the spot since
 # there will be no target directory to receive the contents
@@ -58,10 +77,9 @@
 os.makedirs(targetdir)
 # fixme is this the right set of errors?
 except (IOError, os.error), why:
-errors.append((sourcepath, targetpath, str(why)))
+errors.append((sourcedir, targetdir, str(why)))
 raise Error(errors)
-
-errors = []
+
 for name in names:
 sourcepath = os.path.join(sourcedir, name)
 targetpath = os.path.join(targetdir, name)
@@ -79,14 +97,14 @@
 linkto = os.readlink(sourcepath)
 os.symlink(linkto, targetpath)
 elif os.path.isdir(sourcepath):
-doCopy(sourcepath, targetpath)
+do_copy(sourcepath, targetpath)
 elif os.path.isfile(sourcepath):
 shutil.copy2(sourcepath, targetpath)
 else:
 errors.append("refusingto remove %s, 

[MediaWiki-commits] [Gerrit] operations/dumps[ariel]: hackdeploy pylint cleanup: invalid names, indentation, docst...

[ArielGlenn (Code Review)]

ArielGlenn has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/280100 )

Change subject: hackdeploy pylint cleanup: invalid names, indentation, 
docstrings mostly
..


hackdeploy pylint cleanup: invalid names, indentation, docstrings mostly

Change-Id: I408e068831e263779d808e14c622144dfbf5a424
---
M deployment/obsolete/hackdeploy/hack-deploy.py
1 file changed, 286 insertions(+), 188 deletions(-)

Approvals:
  ArielGlenn: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/deployment/obsolete/hackdeploy/hack-deploy.py 
b/deployment/obsolete/hackdeploy/hack-deploy.py
index 476b5ff..ac9e1e8 100755
--- a/deployment/obsolete/hackdeploy/hack-deploy.py
+++ b/deployment/obsolete/hackdeploy/hack-deploy.py
@@ -1,15 +1,83 @@
 import os
 import sys
-import salt.client
-import salt.cli.cp
-import salt.utils
 import time
 import hashlib
 import re
-from salt.exceptions import SaltInvocationError
 import runpy
+import salt.client
+import salt.cli.cp
+import salt.utils
+from salt.exceptions import SaltInvocationError
 
 # todo: test salt cmd_expandminions
+
+
+def condition_kwarg(arg, kwarg):
+'''
+Return a single arg structure for caller to use
+'''
+if isinstance(kwarg, dict):
+kw_ = []
+for key, val in kwarg.items():
+kw_.append('{0}={1}'.format(key, val))
+return list(arg) + kw_
+return arg
+
+
+def get_file_md5s(dirname, files, callback=None):
+'''
+given list of filenames in a directory,
+return a list of [md5, base filename]
+'''
+output = []
+for fname in files:
+md5out = None
+try:
+md5out = hashlib.md5(open(os.path.join(
+dirname, fname)).read()).hexdigest().strip()
+except Exception:
+md5out = None
+if not md5out:
+sys.stderr.write("failed to get md5 of %s\n" % fname)
+return None
+if callback is not None:
+output.append([md5out, callback(fname)])
+else:
+output.append([md5out, fname])
+return output
+
+
+def get_md5s_ok_count(text):
+'''
+given output from md5sum -c -w on a list of files,
+return the number of files for which the result is 'OK'
+'''
+return len([line for line in text.split('\n')
+if line.endswith(': OK')])
+
+
+def check_date(date):
+'''
+check format of user specified date (mname-dd-) and return it
+or return today's date in that format if no user date is specified
+'''
+if date is None:
+print "No date specified, using today's date"
+date = time.strftime("%b-%d-%Y", time.gmtime(time.time()))
+return date[0].lower() + date[1:]
+else:
+# check the user's date for sanity
+date_regexp = ('^(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)'
+   '-[0-9][0-9]-20[0-9][0-9]$')
+if not re.match(date, date_regexp):
+usage(None, "Bad format for datestring; expecting mon-dd-,"
+  " example: mar-12-2012")
+return date
+
+
+def get_tmp_filename(filename):
+'''use standard format for name of all temp files'''
+return filename + "_tmp"
 
 
 class LocalClientPlus(salt.client.LocalClient):
@@ -18,17 +86,6 @@
 list of known minions that match the specified expression,
 and for copying file content to a newly created remote file
 '''
-
-def condition_kwarg(self, arg, kwarg):
-'''
-Return a single arg structure for caller to use
-'''
-if isinstance(kwarg, dict):
-kw_ = []
-for key, val in kwarg.items():
-kw_.append('{0}={1}'.format(key, val))
-return list(arg) + kw_
-return arg
 
 def cmd_expandminions(self, tgt, fun, arg=(), timeout=None,
   expr_form='glob', ret='',
@@ -43,7 +100,7 @@
   salt "$deployhosts" -v --out raw test.ping |
   grep '{' | mawk -F"'" '{ print $2 }'
 '''
-arg = self.condition_kwarg(arg, kwarg)
+arg = condition_kwarg(arg, kwarg)
 pub_data = self.run_job(tgt, fun, arg, expr_form, ret,
 timeout, **kwargs)
 
@@ -77,69 +134,53 @@
 expr_form='glob')
 
 
-class Dirdate(object):
-'''
-handle date strings in directory names, used for
-prep/staging area primarily
-'''
-def __init__(self, date=None):
-self.date = self.check_date(date)
-
-def check_date(self, date):
-if date is None:
-print "No date specified, using today's date"
-date = time.strftime("%b-%d-%Y", time.gmtime(time.time()))
-return date[0].lower() + date[1:]
-else:
-# check the user's date for sanity
-date_regexp = ('^(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)'
-  

[MediaWiki-commits] [Gerrit] operations/dumps[ariel]: pep8 for rsyncmedia script

[ArielGlenn (Code Review)]

ArielGlenn has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/401051 )

Change subject: pep8 for rsyncmedia script
..


pep8 for rsyncmedia script

Change-Id: I85ca822d84ef2466218d6ba830e01178922ad931
---
M xmldumps-backup/unused/rsyncmedia.py
1 file changed, 174 insertions(+), 153 deletions(-)

Approvals:
  ArielGlenn: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/xmldumps-backup/unused/rsyncmedia.py 
b/xmldumps-backup/unused/rsyncmedia.py
index 36c608a..b95fe9d 100644
--- a/xmldumps-backup/unused/rsyncmedia.py
+++ b/xmldumps-backup/unused/rsyncmedia.py
@@ -1,50 +1,56 @@
-import os, sys, getopt, subprocess
+import os
+import sys
+import getopt
 from subprocess import Popen, PIPE
 
+
+def make_path(dir_list):
+dirs = filter(None, dir_list)
+if len(dirs) == 0:
+return None
+elif len(dirs) == 1:
+# this is ok even with 'None'
+return dirs[0]
+else:
+return os.path.join(*dirs)
+
+
 class Rsyncer(object):
-def __init__(self, rsyncHost, remoteBaseDir, outputDir, verbose, dryrun):
-self.rsyncHost = rsyncHost
-self.remoteBaseDir = remoteBaseDir
-self.outputDir = outputDir
+def __init__(self, rsync_host, remote_base_dir, output_dir, verbose, 
dryrun):
+self.rsync_host = rsync_host
+self.remote_base_dir = remote_base_dir
+self.output_dir = output_dir
 self.verbose = verbose
 self.dryrun = dryrun
-self.dirList = []
+self.dir_list = []
 
-def makePath(self, dirList):
-dirs = filter(None, dirList)
-if len(dirs) == 0:
-return None
-elif len(dirs) == 1:
-# this is ok even with 'None'
-return dirs[0]
-else:
-return os.path.join(*dirs)
+def do_rsync(self, files_to_do, get_dir_list=False):
 
-def doRsync(self, filesToDo, localPath, getDirList = False):
-localdir = self.makePath([ self.outputDir, localPath ])
-
-command = [ "rsync", "-rltDp" ]
-if getDirList:
-if filesToDo:
-filesToDoList = filesToDo.split('\n')
-if len(filesToDoList) > 1:
-sys.stderr.write("refusing to generate wanted dir list for 
multiple toplevel dirs %s\n" % filesToDo)
+command = ["rsync", "-rltDp"]
+if get_dir_list:
+if files_to_do:
+files_to_do_list = files_to_do.split('\n')
+if len(files_to_do_list) > 1:
+sys.stderr.write("refusing to generate wanted "
+ "dir list for multiple toplevel dirs %s\n"
+ % files_to_do)
 return
-# we want the first level of hash dirs (to see what exists, so 
we can request only those)
+# we want the first level of hash dirs (to see what
+# exists, so we can request only those)
 # but we don't want anything below that.
-excludeLevels = 3 + filesToDoList[0].count('/')
-excludeString = "/*" * excludeLevels
-command.extend([ "-f", "- "+ excludeString ])
-command.extend([ "--list-only" ])
-dryrunSaved = self.dryrun
+exclude_levels = 3 + files_to_do_list[0].count('/')
+exclude_string = "/*" * exclude_levels
+command.extend(["-f", "- " + exclude_string])
+command.extend(["--list-only"])
+dryrun_saved = self.dryrun
 self.dryrun = False  # we don't actually change anything with 
--list-only so run it
-if filesToDo:
-command.extend([ "--files-from", "-" ])
-if rsyncHost:
-command.extend([self.rsyncHost + "::" + self.remoteBaseDir, 
self.outputDir ])
+if files_to_do:
+command.extend(["--files-from", "-"])
+if self.rsync_host:
+command.extend([self.rsync_host + "::" + self.remote_base_dir, 
self.output_dir])
 else:
 # "remote" dir is accessible as a local filesystem
-command.extend([ self.remoteBaseDir, self.outputDir ])
+command.extend([self.remote_base_dir, self.output_dir])
 
 # 23 = Partial transfer due to error
 # 24 = Partial transfer due to vanished source files
@@ -54,35 +60,37 @@
 # some of each type of error on every single run, log things
 # but don't bail
 
-if (getDirList):
-result, output = self.dirList = self.doCommand(command, filesToDo, 
[23, 24], displayOutput = False)
+if get_dir_list:
+result_unused, output = self.dir_list = self.do_command(
+command, files_to_do, [23, 24], display_output=False)
 else:
-result, output = self.doCommand(command, filesToDo, [23, 24])
-if 

[MediaWiki-commits] [Gerrit] operations/dumps[ariel]: pep8 for rsyncmedia script

[ArielGlenn (Code Review)]

ArielGlenn has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/401051 )

Change subject: pep8 for rsyncmedia script
..

pep8 for rsyncmedia script

Change-Id: I85ca822d84ef2466218d6ba830e01178922ad931
---
M xmldumps-backup/unused/rsyncmedia.py
1 file changed, 174 insertions(+), 153 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/dumps 
refs/changes/51/401051/1

diff --git a/xmldumps-backup/unused/rsyncmedia.py 
b/xmldumps-backup/unused/rsyncmedia.py
index 36c608a..b95fe9d 100644
--- a/xmldumps-backup/unused/rsyncmedia.py
+++ b/xmldumps-backup/unused/rsyncmedia.py
@@ -1,50 +1,56 @@
-import os, sys, getopt, subprocess
+import os
+import sys
+import getopt
 from subprocess import Popen, PIPE
 
+
+def make_path(dir_list):
+dirs = filter(None, dir_list)
+if len(dirs) == 0:
+return None
+elif len(dirs) == 1:
+# this is ok even with 'None'
+return dirs[0]
+else:
+return os.path.join(*dirs)
+
+
 class Rsyncer(object):
-def __init__(self, rsyncHost, remoteBaseDir, outputDir, verbose, dryrun):
-self.rsyncHost = rsyncHost
-self.remoteBaseDir = remoteBaseDir
-self.outputDir = outputDir
+def __init__(self, rsync_host, remote_base_dir, output_dir, verbose, 
dryrun):
+self.rsync_host = rsync_host
+self.remote_base_dir = remote_base_dir
+self.output_dir = output_dir
 self.verbose = verbose
 self.dryrun = dryrun
-self.dirList = []
+self.dir_list = []
 
-def makePath(self, dirList):
-dirs = filter(None, dirList)
-if len(dirs) == 0:
-return None
-elif len(dirs) == 1:
-# this is ok even with 'None'
-return dirs[0]
-else:
-return os.path.join(*dirs)
+def do_rsync(self, files_to_do, get_dir_list=False):
 
-def doRsync(self, filesToDo, localPath, getDirList = False):
-localdir = self.makePath([ self.outputDir, localPath ])
-
-command = [ "rsync", "-rltDp" ]
-if getDirList:
-if filesToDo:
-filesToDoList = filesToDo.split('\n')
-if len(filesToDoList) > 1:
-sys.stderr.write("refusing to generate wanted dir list for 
multiple toplevel dirs %s\n" % filesToDo)
+command = ["rsync", "-rltDp"]
+if get_dir_list:
+if files_to_do:
+files_to_do_list = files_to_do.split('\n')
+if len(files_to_do_list) > 1:
+sys.stderr.write("refusing to generate wanted "
+ "dir list for multiple toplevel dirs %s\n"
+ % files_to_do)
 return
-# we want the first level of hash dirs (to see what exists, so 
we can request only those)
+# we want the first level of hash dirs (to see what
+# exists, so we can request only those)
 # but we don't want anything below that.
-excludeLevels = 3 + filesToDoList[0].count('/')
-excludeString = "/*" * excludeLevels
-command.extend([ "-f", "- "+ excludeString ])
-command.extend([ "--list-only" ])
-dryrunSaved = self.dryrun
+exclude_levels = 3 + files_to_do_list[0].count('/')
+exclude_string = "/*" * exclude_levels
+command.extend(["-f", "- " + exclude_string])
+command.extend(["--list-only"])
+dryrun_saved = self.dryrun
 self.dryrun = False  # we don't actually change anything with 
--list-only so run it
-if filesToDo:
-command.extend([ "--files-from", "-" ])
-if rsyncHost:
-command.extend([self.rsyncHost + "::" + self.remoteBaseDir, 
self.outputDir ])
+if files_to_do:
+command.extend(["--files-from", "-"])
+if self.rsync_host:
+command.extend([self.rsync_host + "::" + self.remote_base_dir, 
self.output_dir])
 else:
 # "remote" dir is accessible as a local filesystem
-command.extend([ self.remoteBaseDir, self.outputDir ])
+command.extend([self.remote_base_dir, self.output_dir])
 
 # 23 = Partial transfer due to error
 # 24 = Partial transfer due to vanished source files
@@ -54,35 +60,37 @@
 # some of each type of error on every single run, log things
 # but don't bail
 
-if (getDirList):
-result, output = self.dirList = self.doCommand(command, filesToDo, 
[23, 24], displayOutput = False)
+if get_dir_list:
+result_unused, output = self.dir_list = self.do_command(
+command, files_to_do, [23, 24], display_output=False)
 else:
-result, output = self.doCommand(command, filesToDo, [23, 24])
-if 

[MediaWiki-commits] [Gerrit] operations/dumps[ariel]: toy offline reader: pylint and pep8

[ArielGlenn (Code Review)]

ArielGlenn has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/280109 )

Change subject: toy offline reader: pylint and pep8
..


toy offline reader: pylint and pep8

Change-Id: Ib0bbf9c3bb1db1a8029a7a7a3e660e82433634b7
---
M toys/bz2multistream/INSTALL.txt
M toys/bz2multistream/wikiarticles.py
M toys/bz2multistream/writetoc.py
3 files changed, 755 insertions(+), 675 deletions(-)

Approvals:
  ArielGlenn: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/toys/bz2multistream/INSTALL.txt b/toys/bz2multistream/INSTALL.txt
index 733c686..1d955ef 100644
--- a/toys/bz2multistream/INSTALL.txt
+++ b/toys/bz2multistream/INSTALL.txt
@@ -23,7 +23,7 @@
 (or you can use the helper script sort.sh for this step)
 
 cat elwiki-pages-multistream-index-sorted.txt | \
-python ./writetoc.py --field 3 --separator ':' --indexfile 
elwiki-pages-multistream-toc.txt
+python ./writetoc.py --field 3 --separator ':' --tocfile 
elwiki-pages-multistream-toc.txt
 
 After the above, you can optionally create a configuration file 
"wikiarticles.conf".
 See the wikiarticles.conf.sample for more information.  An example 
configuration file
diff --git a/toys/bz2multistream/wikiarticles.py 
b/toys/bz2multistream/wikiarticles.py
index 6ebc5b0..27d1e99 100644
--- a/toys/bz2multistream/wikiarticles.py
+++ b/toys/bz2multistream/wikiarticles.py
@@ -1,209 +1,446 @@
-import getopt, os, sys, re, codecs, bz2, ConfigParser
+import getopt
+import os
+import sys
+import re
+import bz2
+import ConfigParser
+
+
+def get_choice_from_batch(titles, start, batch_size):
+'''
+display titles from start to start + batchsize, with count in front
+ask the caller for a title number or an action
+actions may be Q (quit), N (next batch), B (previous batch), R (redisplay)
+if caller enters nothing, treat that as default (R)
+if caller enters something else, whine and treat that as default (R) too
+arguments:
+titles -- full list of titles
+start  -- display from this point in the list
+batch_size -- how many titles to display
+returns a tuple of (action, title number) where one or the other of these
+may be None
+'''
+
+# yay python, it will silently ignore the fact that you requested
+# more things in the list than exist. (no this is not sarcasm)
+count = start
+for line in titles[start:start+batch_size]:
+print "%s) %s" % (count+1, WATitleMunger.un_normalize_title(line))
+count += 1
+print
+print("Enter number of choice, or Q/N/P/R to "
+  "quit/next page/prev page/redisplay page (default R): "),
+choice = sys.stdin.readline()
+choice = choice.strip()
+if not choice:
+choice = 'R'
+if choice.isdigit():
+num = int(choice)
+if num < 1 or num > len(titles):
+print "Bad number given."
+return "R", None
+return None, num
+else:
+choice = choice.capitalize()
+if choice in ['N', 'P', 'Q', 'R']:
+return choice, None
+else:
+print "Bad choice given."
+return "R", None
+
+
+def process_action(action, start, batch_size, total):
+'''
+given a caller action,
+update title list display pointer to the appropriate position
+arguments:
+action -- Q (quit), N (next batch), P (prev batch), or anything else
+start  -- title list display pointer, a batch of titles from the list
+  will be displayed starting from this number
+batch_size -- how many titles are dispayed in a batch
+total  -- total titles in the list
+returns: updated title list display pointer, or exits at user request
+(action Q)
+note that any action other than Q/N/P will result in the default R 
(redisplay
+current batch of titles) which means no change, return existing value.  
This
+includes the None action.
+'''
+if action == 'N' or action == 'n':
+if start + batch_size < total:
+start += batch_size
+else:
+print "End of list reached."
+elif action == 'P' or action == 'p':
+if start > batch_size:
+start = start - batch_size
+else:
+print "Beginning of list reached."
+elif action == 'Q' or action == 'q':
+print "Exiting at user's request."
+sys.exit(0)
+return start
+
+
+def get_user_title_choice(title_hash):
+'''
+show a numbered list of page titles on stdout and read the
+caller's choice on stdin
+I guess this is a poor person's pager
+arguments:
+titleHash -- hash of page titles and their offsets into the xml file
+returns: the offset into the xml file for the title selected
+'''
+titles = title_hash.keys()
+titles.sort()
+total = len(titles)
+
+choice = None
+start = 0
+batch_size = 30
+print "Multiple titles 

[MediaWiki-commits] [Gerrit] operations/dumps[ariel]: thumbDateAnalysis full pylint and pep8

[ArielGlenn (Code Review)]

ArielGlenn has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/280104 )

Change subject: thumbDateAnalysis full pylint and pep8
..


thumbDateAnalysis full pylint and pep8

Change-Id: I81310c994c52f841eb3d93491a6b129efad07564
---
M tools/thumbs/crunchinglogs/datascripts/thumbDateAnalysis.py
1 file changed, 75 insertions(+), 66 deletions(-)

Approvals:
  ArielGlenn: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/tools/thumbs/crunchinglogs/datascripts/thumbDateAnalysis.py 
b/tools/thumbs/crunchinglogs/datascripts/thumbDateAnalysis.py
index 5576798..4f0c1bf 100644
--- a/tools/thumbs/crunchinglogs/datascripts/thumbDateAnalysis.py
+++ b/tools/thumbs/crunchinglogs/datascripts/thumbDateAnalysis.py
@@ -1,35 +1,42 @@
 # -*- coding: utf-8  -*-
 
-import os
 import re
 import sys
-import time
 import getopt
 
+
 def usage(message=None):
-print "Usage: %s [--sdate=date --edate=date --created [filename]" % 
sys.argv[0]
-print "sdate: start date for which to print stats, default: earliest date 
in file "
-print "edate: end date for which to print stats, default: latest date in 
file"
-print "created: show only the number of files and sizes on the date the 
first thumb"
-print "was created (presumably the date the image itself was first 
uploaded)"
-print ""
-print "Date format for sdate and edate: -mm-dd"
-print ""
-print "If no filename is specified, input is read from stdin"
-print
-print "Format of input file: (sample line)"
-print "2011-10-29  01:57:51   100311   
Festiwal_Słowian_i_Wikingów_2009_121.jpg/640px-Festiwal_Słowian_i_Wikingów_2009_121.jpg"
-print "date in -mm-dd format, time in hh:mm::ss format, size in bytes, 
thumb directory/thumb filename"
+if message is not None:
+sys.stderr.write(message + "\n")
+usage_message = """
+Usage: thumbDateAnalysis.py [--sdate=date --edate=date --created [filename]
+
+  --sdate:   start date for which to print stats, default: earliest date in 
file
+  --edate:   end date for which to print stats, default: latest date in file
+  --created: show only the number of files and sizes on the date the first 
thumb
+ was created (presumably the date the image itself was first 
uploaded)
+
+Date format for sdate and edate: -mm-dd
+
+If no filename is specified, input is read from stdin.
+
+Format of input file: (sample line)
+
+2011-10-29  01:57:51   100311   
Festiwal_Słowian_i_Wikingów_2009_121.jpg/640px-Festiwal_Słowian_i_Wikingów_2009_121.jpg
+date in -mm-dd format, time in hh:mm::ss format, size in bytes, thumb 
directory/thumb filename
+"""
+sys.stderr.write(usage_message)
 sys.exit(1)
 
-if __name__ == "__main__":
+
+def do_main():
 sdate = None
 edate = None
 created = False
 try:
-(options, remainder) = getopt.gnu_getopt(sys.argv[1:], "",
- [ 'sdate=', 'edate=', 
'created' ])
-except:
+(options, remainder) = getopt.gnu_getopt(
+sys.argv[1:], "", ['sdate=', 'edate=', 'created'])
+except Exception:
 usage("Unknown option specified")
 
 for (opt, val) in options:
@@ -41,71 +48,73 @@
 created = True
 
 dateexp = re.compile(r"^\d{4}-\d{2}-\d{2}$")
-for d in filter(None, [ sdate, edate ]):
-if not dateexp.match(d):
+for date in filter(None, [sdate, edate]):
+if not dateexp.match(date):
 usage("Bad date format.")
 
 if len(remainder) == 1:
-inputFile = remainder[0]
-fHandle = open(inputFile,"r")
+input_file = remainder[0]
+f_handle = open(input_file, "r")
 elif len(remainder) == 0:
-fHandle = sys.stdin
+f_handle = sys.stdin
 else:
 usage("Too many arguments.")
 
-lastDirName = None
-numFilesSameDate = 0
-byteCountSameDate = 0
-fileCounts = {}
-byteCounts = {}
-for line in fHandle:
+last_dir_name = None
+file_counts = {}
+byte_counts = {}
+for line in f_handle:
 try:
-( fDate, fTime, fSize, path ) = line.rstrip().split()
-except:
+(f_date, ftime_unused, f_size, path) = line.rstrip().split()
+except Exception:
 print >> sys.stderr, "skipping badly formatted line: ", 
line.rstrip()
 continue
-( dirName, fName ) = path.split('/',2)
-if not lastDirName:
-lastDirName = dirName
-if dirName != lastDirName:
-# should just print the number of files for every date sorted by 
date order, plus the dir name of course"
-if (sdate and (fDate >= sdate)) or not sdate:
-if (edate and (fDate <= edate)) or not edate:
+(dir_name, fname_unused) = path.split('/', 2)
+if not last_dir_name:
+last_dir_name = dir_name
+if dir_name != 

[MediaWiki-commits] [Gerrit] operations/dumps[ariel]: fixup-interwikis: full pylint and pep8

[ArielGlenn (Code Review)]

ArielGlenn has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/280103 )

Change subject: fixup-interwikis: full pylint and pep8
..


fixup-interwikis: full pylint and pep8

Change-Id: I22fd2d5141c6276f5b6017af48ed0fe65847ba4a
---
M tools/interwikis/fixup-interwikis.py
1 file changed, 256 insertions(+), 195 deletions(-)

Approvals:
  ArielGlenn: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/tools/interwikis/fixup-interwikis.py 
b/tools/interwikis/fixup-interwikis.py
index b43e2ef..b9a3f53 100644
--- a/tools/interwikis/fixup-interwikis.py
+++ b/tools/interwikis/fixup-interwikis.py
@@ -1,138 +1,170 @@
-import os, re, sys, time, getopt, cdb, urllib
+import re
+import sys
+import getopt
+import urllib
 from os.path import exists
+import cdb
+
 
 class IWCdbUpdater(object):
-def __init__(self, wikiDbName, wikiTablePrefix, cdbFile, siteType, 
wikiLangCode, dryrun, verbose):
-self.wikiDbName = wikiDbName
-self.wikiTablePrefix = wikiTablePrefix
-self.cdbFile = cdbFile
-self.newCdbFile = cdbFile + ".new"
-self.siteType = siteType
-self.wikiLangCode = wikiLangCode
+def __init__(self, wiki_db_name, wiki_table_prefix, cdb_file,
+ site_type, wiki_lang_code, dryrun, verbose):
+self.wiki_db_name = wiki_db_name
+self.wiki_table_prefix = wiki_table_prefix
+self.cdb_file = cdb_file
+self.new_cdb_file = cdb_file + ".new"
+self.site_type = site_type
+self.wiki_lang_code = wiki_lang_code
 self.dryrun = dryrun
 self.verbose = verbose
 
-self.wikiName = self.getWikiName()
+self.wiki_name = self.get_wiki_name()
 
 # if we can't find it, try to download it
-if not exists(self.cdbFile):
+if not exists(self.cdb_file):
 if self.dryrun:
-sys.stderr.write("No such file %s, would download Wikimedia 
interwiki cdb file\n" % self.cdbFile)
+sys.stderr.write("No such file %s, would download "
+ "Wikimedia interwiki cdb file\n" % 
self.cdb_file)
 elif self.verbose:
-sys.stderr.write("No such file %s, downloading Wikimedia 
interwiki cdb file\n" % self.cdbFile)
+sys.stderr.write("No such file %s, downloading "
+ "Wikimedia interwiki cdb file\n" % 
self.cdb_file)
 url = "https://noc.wikimedia.org/conf/interwiki.cdb;
-urllib.urlretrieve(url, self.cdbFile)
+urllib.urlretrieve(url, self.cdb_file)
 
-self.oldcdbfd = cdb.init(self.cdbFile)
+self.oldcdbfd = cdb.init(self.cdb_file)
 self.newcdbfd = None
+self.update_these = {}
 
-def getWikiName(self):
-if self.wikiTablePrefix:
-return("%s-%s" % (self.wikiDbName, self.wikiTablePrefix))
+def get_wiki_name(self):
+'''
+return the wiki name including table prefix if needed
+'''
+if self.wiki_table_prefix:
+return "%s-%s" % (self.wiki_db_name, self.wiki_table_prefix)
 else:
-return self.wikiDbName
+return self.wiki_db_name
 
 @staticmethod
-def getKnownSiteTypesDict():
-return { "wikibooks": "b", "wikimedia": "chapter", "wikidata": "d", 
"wikinews": 'n', "wikiquote": 'q', "wikisource": "s", "wikiversity": 'v', 
"wikivoyage": "voy", "wiki": 'w', "wiktionary": "wikt" }
+def get_known_site_types_dict():
+'''
+return wiki types along with their abbreviations
+'''
+return {"wikibooks": "b", "wikimedia": "chapter",
+"wikidata": "d", "wikinews": 'n', "wikiquote": 'q',
+"wikisource": "s", "wikiversity": 'v',
+"wikivoyage": "voy", "wiki": 'w', "wiktionary": "wikt"}
 
 @staticmethod
-def getKnownSiteTypes():
-return IWCdbUpdater.getKnownSiteTypesDict().keys()
+def get_known_site_types():
+'''
+return the list of known wiki types
+'''
+return IWCdbUpdater.get_known_site_types_dict().keys()
 
 @staticmethod
-def getAbbrevs():
-return IWCdbUpdater.getKnownSiteTypesDict().values()
+def get_abbrevs():
+'''
+return the list of abbreviations of known wiki types
+'''
+return IWCdbUpdater.get_known_site_types_dict().values()
 
 @staticmethod
-def getAbbrevFromSiteType(siteType):
-return IWCdbUpdater.getKnownSiteTypesDict()[siteType]
+def get_abbrev_from_site_type(site_type):
+'''
+given a wiki type, return its abbreviation
+'''
+return IWCdbUpdater.get_known_site_types_dict()[site_type]
 
 @staticmethod
-def getSiteUrl(langCode, siteType):
-if siteType == 'wiki':
+def get_site_url(lang_code, site_type):
+'''
+given the language code 

[MediaWiki-commits] [Gerrit] operations/dumps[ariel]: archiveloader full pylint and pep8

[ArielGlenn (Code Review)]

ArielGlenn has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/280102 )

Change subject: archiveloader full pylint and pep8
..


archiveloader full pylint and pep8

A lot of code cleanup, breaking up the too-long module into
reasonable sized files, removinging redundant code, compacting
long sequences of if statements, etc.

Change-Id: I67526d9b154d2b7a5aa9e38fb8561916aaefb454
---
M tools/archive.org/README
A tools/archive.org/archivelib/__init__.py
A tools/archive.org/archivelib/config.py
A tools/archive.org/archivelib/curlargs.py
A tools/archive.org/archivelib/error.py
A tools/archive.org/archivelib/html_utils.py
A tools/archive.org/archivelib/sitematrix.py
A tools/archive.org/archivelib/uploader.py
A tools/archive.org/archivelib/urls.py
A tools/archive.org/archivelib/xml_utils.py
M tools/archive.org/archiveuploader.py
11 files changed, 1,204 insertions(+), 1,025 deletions(-)

Approvals:
  ArielGlenn: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/tools/archive.org/README b/tools/archive.org/README
index dcd0ffa..054377b 100644
--- a/tools/archive.org/README
+++ b/tools/archive.org/README
@@ -1,30 +1,30 @@
 This is the archiveuploader script which we use to upload dumps to archive.org
-via their S3-style api.  
+via their S3-style api.
 
-Notes: 
+Notes:
 
 This is used only for dumps.  It does things like try to determine the language
-of the project dumped by polling the en wikipedia SiteMatrix. 
+of the project dumped by polling the en wikipedia SiteMatrix.
 
 Setup:
 
 Create a config file.  It should contain the access and secret key needed for
-access to the archive.org api, the url to the SiteMatrix for your projects, 
+access to the archive.org api, the url to the SiteMatrix for your projects,
 and the full path to the name of the file where the SiteMatrix information
 will be cached.   See archiveuploader.conf.sample for an example.
-If you don't need multiple config files, put it in the default place: 
+If you don't need multiple config files, put it in the default place:
 "archiveuploader.conf.sample" in the working directory of the script.
 
 Make sure curl is installed on your system and a pointer to its location
 is also in the config file.
 
-If you are not uploading to WMF items (buckets)... i.e. you are 
-an individual user uploading to some other bucket, you'll need to 
+If you are not uploading to WMF items (buckets)... i.e. you are
+an individual user uploading to some other bucket, you'll need to
 add to the configuration file a pointer to the license covering
 the content of your files, the creator of the dumps, and the download
 location of the dumps.
 
-By default the dbname of your project is assumed to be the same 
+By default the dbname of your project is assumed to be the same
 as the itemname that will appear in S3 urls.  If you don't want
 this to be the case, you can specify a format string in the config
 file, putting %%s in the string in the spot where the dbname would go.
@@ -37,7 +37,7 @@
 
 Files to be uploaded:
 
-We use this uploading a full directory of dumps tarred up into a 
+We use this uploading a full directory of dumps tarred up into a
 single file, with a name like elwiktionary-20060703.tar
 Obviously this only works for smaller projects.
 
@@ -52,12 +52,12 @@
 
 Set up a tarball of the dumped tables and xml files of a project for
 a given date. Run the script without arguments to see a detailed
-help message about its invocation.  
+help message about its invocation.
 
-If you have put all of your auth information in the config file, 
-you should be able to create the item with the command 
-  python archiveuploader.py --createitem dbnamehere 
-to create the initial item.  The dbname should be the actual dbname of the 
+If you have put all of your auth information in the config file,
+you should be able to create the item with the command
+  python archiveuploader.py --createitem dbnamehere
+to create the initial item.  The dbname should be the actual dbname of the
 project with the dump you'll be uploading.  *The item name is created from
 the dbname using the itemnameformat entry in the config file.*
 
@@ -65,26 +65,26 @@
 wind up in a todo queue which you can check here (must log in via
 web interface, no xml or json output available either):
   http://www.archive.org/catalog.php?justme=1
-Completed jobs are listed here: 
+Completed jobs are listed here:
   http://www.archive.org/catalog.php?history=1=1=1
 
-If the metadata loks wrong, you can try to tweak it by altering
+If the metadata looks wrong, you can try to tweak it by altering
 the config file settings, and then update the item by
-  python archiveuploader.py --updateitem dbnamehere 
+  python archiveuploader.py --action updateitem --itemname dbnamehere
 
 Now you're ready to add objects (files) to the item (bucket).
-You can see what the script thinks it should do: 
-  python 

[MediaWiki-commits] [Gerrit] operations/dumps[master]: ability to do xmlpageslogging several pieces at a time in pa...

[ArielGlenn (Code Review)]

ArielGlenn has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/394857 )

Change subject: ability to do xmlpageslogging several pieces at a time in 
parallel
..


ability to do xmlpageslogging several pieces at a time in parallel

Bug: T181935
Change-Id: Icef0aa23c363d7fa4d3b09074571f02a9ed2d3c6
---
M xmldumps-backup/defaults.conf
M xmldumps-backup/dumps/WikiDump.py
M xmldumps-backup/dumps/recombinejobs.py
M xmldumps-backup/dumps/runner.py
M xmldumps-backup/dumps/runnerutils.py
M xmldumps-backup/dumps/utils.py
M xmldumps-backup/dumps/xmljobs.py
M xmldumps-backup/xmlstreams.py
8 files changed, 157 insertions(+), 15 deletions(-)

Approvals:
  ArielGlenn: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/xmldumps-backup/defaults.conf b/xmldumps-backup/defaults.conf
index e95ac47..222cb2d 100644
--- a/xmldumps-backup/defaults.conf
+++ b/xmldumps-backup/defaults.conf
@@ -60,6 +60,8 @@
 pagesPerChunkHistory=0
 revsPerChunkHistory=0
 pagesPerChunkAbstract=0
+chunksForPagelogs=0
+logitemsPerPagelogs=0
 jobsperbatch=""
 revsPerJob=100
 maxRetries=0
diff --git a/xmldumps-backup/dumps/WikiDump.py 
b/xmldumps-backup/dumps/WikiDump.py
index 8ba3838..005f858 100644
--- a/xmldumps-backup/dumps/WikiDump.py
+++ b/xmldumps-backup/dumps/WikiDump.py
@@ -282,6 +282,10 @@
 "chunks", "chunksForAbstract", 0)
 self.pages_per_filepart_abstract = self.get_opt_for_proj_or_default(
 "chunks", "pagesPerChunkAbstract", 0)
+self.numparts_for_pagelogs = self.get_opt_for_proj_or_default(
+"chunks", "chunksForPagelogs", 0)
+self.logitems_per_filepart_pagelogs = self.get_opt_for_proj_or_default(
+"chunks", "logitemsPerPagelogs", 0)
 self.recombine_history = self.get_opt_for_proj_or_default(
 "chunks", "recombineHistory", 1)
 self.checkpoint_time = self.get_opt_for_proj_or_default(
diff --git a/xmldumps-backup/dumps/recombinejobs.py 
b/xmldumps-backup/dumps/recombinejobs.py
index 6d1f0a1..929f23a 100644
--- a/xmldumps-backup/dumps/recombinejobs.py
+++ b/xmldumps-backup/dumps/recombinejobs.py
@@ -329,3 +329,56 @@
 error = result
 if error:
 raise BackupError("error recombining abstract dump files")
+
+
+class RecombineXmlLoggingDump(RecombineDump):
+def __init__(self, name, desc, item_for_recombine):
+# no partnum_todo, no parts generally (False, False), even though 
input may have it
+self.item_for_recombine = item_for_recombine
+self._prerequisite_items = [self.item_for_recombine]
+super(RecombineXmlLoggingDump, self).__init__(name, desc)
+
+def get_filetype(self):
+return self.item_for_recombine.get_filetype()
+
+def get_file_ext(self):
+return self.item_for_recombine.get_file_ext()
+
+def get_dumpname(self):
+return self.item_for_recombine.get_dumpname()
+
+def build_command(self, runner, to_recombine_dfnames, output_dfname):
+input_dfnames = []
+for in_dfname in to_recombine_dfnames:
+if in_dfname.dumpname == output_dfname.dumpname:
+input_dfnames.append(in_dfname)
+if not len(input_dfnames):
+self.set_status("failed")
+raise BackupError("No input files for %s found" % self.name())
+if not exists(runner.wiki.config.gzip):
+raise BackupError("gzip command %s not found" % 
runner.wiki.config.gzip)
+compression_command = "%s > " % runner.wiki.config.gzip
+uncompression_command = ["%s" % runner.wiki.config.gzip, "-dc"]
+recombine_command_string = self.build_recombine_command_string(
+runner, input_dfnames, output_dfname, compression_command,
+uncompression_command)
+recombine_command = [recombine_command_string]
+recombine_pipeline = [recombine_command]
+series = [recombine_pipeline]
+return series
+
+def run(self, runner):
+error = 0
+to_recombine_dfnames = 
self.item_for_recombine.list_outfiles_for_input(runner.dump_dir)
+output_dfnames = self.list_outfiles_for_build_command(runner.dump_dir)
+for output_dfname in output_dfnames:
+command_series = self.build_command(runner, to_recombine_dfnames, 
output_dfname)
+self.setup_command_info(runner, command_series, [output_dfname])
+result, broken = runner.run_command(
+[command_series], callback_timed=self.progress_callback,
+callback_timed_arg=runner, shell=True,
+callback_on_completion=self.command_completion_callback)
+if result:
+error = result
+if error:
+raise BackupError("error recombining log event files")
diff --git a/xmldumps-backup/dumps/runner.py b/xmldumps-backup/dumps/runner.py
index 662f581..e0dfaca 100644
--- 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: enable pagelogs to be dumped by several processes in parallel

[ArielGlenn (Code Review)]

ArielGlenn has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/399589 )

Change subject: enable pagelogs to be dumped by several processes in parallel
..


enable pagelogs to be dumped by several processes in parallel

Bug: T181935
Change-Id: Ib89de0b72aaa0fe73534bb318b54efcd50414192
---
M modules/snapshot/manifests/dumps/configs.pp
1 file changed, 6 insertions(+), 2 deletions(-)

Approvals:
  ArielGlenn: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/snapshot/manifests/dumps/configs.pp 
b/modules/snapshot/manifests/dumps/configs.pp
index 9140a54..bd510ec 100644
--- a/modules/snapshot/manifests/dumps/configs.pp
+++ b/modules/snapshot/manifests/dumps/configs.pp
@@ -28,6 +28,7 @@
 keep  => '8',
 chunksEnabled => '1',
 chunksForAbstract => '4',
+chunksForPagelogs => '4',
 checkpointTime=> '720',
 recombineHistory  => '0',
 revsPerJob=> '150',
@@ -89,10 +90,11 @@
 en => {
 dblist=> "${dblistsdir}/enwiki.dblist",
 skipdblist=> "${dblistsdir}/skipnone.dblist",
-jobsperbatch  => 'xmlstubsdump=9,abstractsdump=9',
+jobsperbatch  => 
'xmlstubsdump=9,abstractsdump=9,xmlpagelogsdump=9',
 keep  => '7',
 chunksEnabled => '1',
 chunksForAbstract => '27',
+chunksForPagelogs => '27',
 recombineHistory  => '0',
 checkpointTime=> '720',
 revsPerJob=> '150',
@@ -109,10 +111,11 @@
 wd => {
 dblist=> "${dblistsdir}/wikidatawiki.dblist",
 skipdblist=> "${dblistsdir}/skipnone.dblist",
-jobsperbatch  => 'xmlstubsdump=9,abstractsdump=9',
+jobsperbatch  => 
'xmlstubsdump=9,abstractsdump=9,xmlpagelogsdump=9',
 keep  => '7',
 chunksEnabled => '1',
 chunksForAbstract => '27',
+chunksForPagelogs => '27',
 recombineHistory  => '0',
 checkpointTime=> '720',
 revsPerJob=> '150',
@@ -138,6 +141,7 @@
 }
 
 # for xml/sql dumps running on dumpsdata host
+# as well as misc dumps via various cron jobs
 snapshot::dumps::wikiconf { 'wikidump.conf.dumps':
 configtype => 'allwikis',
 config => $config,

-- 
To view, visit https://gerrit.wikimedia.org/r/399589
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ib89de0b72aaa0fe73534bb318b54efcd50414192
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: ArielGlenn 
Gerrit-Reviewer: ArielGlenn 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: make all misc dump jobs plus xml dumps config use hiera sett...

[ArielGlenn (Code Review)]

ArielGlenn has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/400692 )

Change subject: make all misc dump jobs plus xml dumps config use hiera setting 
for php
..


make all misc dump jobs plus xml dumps config use hiera setting for php

This lets us choose hhvm or some php version per host, and lets
us use different versions in labs as well

Change-Id: I4c8a768be76db30501790e5d31af8ca9cc1c4f52
---
M hieradata/hosts/snapshot1001.yaml
M hieradata/hosts/snapshot1005.yaml
M hieradata/hosts/snapshot1006.yaml
M hieradata/hosts/snapshot1007.yaml
M modules/profile/manifests/dumps/generation/worker/common.pp
M modules/snapshot/files/cron/dump-global-blocks.sh
M modules/snapshot/files/cron/dumpcategoriesrdf.sh
M modules/snapshot/files/cron/dumpcirrussearch.sh
M modules/snapshot/files/cron/dumpwikidatajson.sh
M modules/snapshot/files/cron/dumpwikidatardf.sh
M modules/snapshot/files/cron/wikidatadumps-shared.sh
M modules/snapshot/manifests/dumps.pp
M modules/snapshot/manifests/dumps/configs.pp
M modules/snapshot/templates/dumps/wikidump.conf.erb
14 files changed, 34 insertions(+), 22 deletions(-)

Approvals:
  ArielGlenn: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/hieradata/hosts/snapshot1001.yaml 
b/hieradata/hosts/snapshot1001.yaml
index d27cca6..bc337cb 100644
--- a/hieradata/hosts/snapshot1001.yaml
+++ b/hieradata/hosts/snapshot1001.yaml
@@ -1 +1,2 @@
 snapshot::dumps::runtype: regular
+snapshot::dumps::php: /usr/bin/php5
diff --git a/hieradata/hosts/snapshot1005.yaml 
b/hieradata/hosts/snapshot1005.yaml
index c34955f..716a988 100644
--- a/hieradata/hosts/snapshot1005.yaml
+++ b/hieradata/hosts/snapshot1005.yaml
@@ -1 +1,2 @@
 snapshot::dumps::runtype: enwiki
+snapshot::dumps::php: /usr/bin/php5
diff --git a/hieradata/hosts/snapshot1006.yaml 
b/hieradata/hosts/snapshot1006.yaml
index ea234d3..70efdd8 100644
--- a/hieradata/hosts/snapshot1006.yaml
+++ b/hieradata/hosts/snapshot1006.yaml
@@ -1 +1,2 @@
 snapshot::dumps::runtype: wikidatawiki
+snapshot::dumps::php: /usr/bin/php5
diff --git a/hieradata/hosts/snapshot1007.yaml 
b/hieradata/hosts/snapshot1007.yaml
index d27cca6..bc337cb 100644
--- a/hieradata/hosts/snapshot1007.yaml
+++ b/hieradata/hosts/snapshot1007.yaml
@@ -1 +1,2 @@
 snapshot::dumps::runtype: regular
+snapshot::dumps::php: /usr/bin/php5
diff --git a/modules/profile/manifests/dumps/generation/worker/common.pp 
b/modules/profile/manifests/dumps/generation/worker/common.pp
index 940a017..b1d9150 100644
--- a/modules/profile/manifests/dumps/generation/worker/common.pp
+++ b/modules/profile/manifests/dumps/generation/worker/common.pp
@@ -1,6 +1,7 @@
 class profile::dumps::generation::worker::common(
 $nfs_server = hiera('dumps_nfs_server'),
 $managed_subdirs = hiera('dumps_managed_subdirs'),
+$php = hiera('snapshot::dumps::php'),
 ) {
 # mw packages and dependencies
 require ::profile::mediawiki::scap_proxy
@@ -29,7 +30,7 @@
 cronsdir   =>  "${xmldumpsmount}/otherdumps",
 apachedir  => '/srv/mediawiki',
 }
-class { '::snapshot::dumps': }
+class { '::snapshot::dumps': php => $php}
 
 # scap3 deployment of dump scripts
 scap::target { 'dumps/dumps':
diff --git a/modules/snapshot/files/cron/dump-global-blocks.sh 
b/modules/snapshot/files/cron/dump-global-blocks.sh
index c3d2baf..bc0a553 100644
--- a/modules/snapshot/files/cron/dump-global-blocks.sh
+++ b/modules/snapshot/files/cron/dump-global-blocks.sh
@@ -11,7 +11,7 @@
 
 multiversionscript="${apachedir}/multiversion/MWScript.php"
 if [ -e "$multiversionscript" ]; then
-host=`php -q "$multiversionscript" 
extensions/CentralAuth/maintenance/getCentralAuthDBInfo.php --wiki="aawiki"` || 
(echo $host >& 2; host="")
+host=`$php -q "$multiversionscript" 
extensions/CentralAuth/maintenance/getCentralAuthDBInfo.php --wiki="aawiki"` || 
(echo $host >& 2; host="")
 fi
 if [ -z "$host" ]; then
 echo "can't locate db server for centralauth, exiting." >& 2
@@ -25,7 +25,7 @@
 
 multiversionscript="${apachedir}/multiversion/MWScript.php"
 if [ -e "$multiversionscript" ]; then
-db_user=`echo 'echo $wgDBadminuser;' | php "$multiversionscript" 
eval.php aawiki`
+db_user=`echo 'echo $wgDBadminuser;' | $php "$multiversionscript" 
eval.php aawiki`
 fi
 if [ -z "$db_user" ]; then
 echo "can't get db user name, exiting." >& 2
@@ -39,7 +39,7 @@
 
 multiversionscript="${apachedir}/multiversion/MWScript.php"
 if [ -e "$multiversionscript" ]; then
-db_pass=`echo 'echo $wgDBadminpassword;' | php "$multiversionscript" 
eval.php aawiki`
+db_pass=`echo 'echo $wgDBadminpassword;' | $php "$multiversionscript" 
eval.php aawiki`
 fi
 if [ -z "$db_pass" ]; then
 echo "can't get db password, exiting." >& 2
@@ -97,11 +97,12 @@
 fi
 done
 
-args="tools:gzip,mysqldump"

[MediaWiki-commits] [Gerrit] operations/puppet[production]: make all misc dump jobs plus xml dumps config use hiera sett...

[ArielGlenn (Code Review)]

ArielGlenn has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/400692 )

Change subject: make all misc dump jobs plus xml dumps config use hiera setting 
for php
..

make all misc dump jobs plus xml dumps config use hiera setting for php

This lets us choose hhvm or some php version per host, and lets
us use different versions in labs as well

Change-Id: I4c8a768be76db30501790e5d31af8ca9cc1c4f52
---
M hieradata/hosts/snapshot1001.yaml
M hieradata/hosts/snapshot1005.yaml
M hieradata/hosts/snapshot1006.yaml
M hieradata/hosts/snapshot1007.yaml
M modules/profile/manifests/dumps/generation/worker/common.pp
M modules/snapshot/files/cron/dump-global-blocks.sh
M modules/snapshot/files/cron/dumpcategoriesrdf.sh
M modules/snapshot/files/cron/dumpcirrussearch.sh
M modules/snapshot/files/cron/dumpwikidatajson.sh
M modules/snapshot/files/cron/dumpwikidatardf.sh
M modules/snapshot/files/cron/wikidatadumps-shared.sh
M modules/snapshot/manifests/dumps.pp
M modules/snapshot/manifests/dumps/configs.pp
M modules/snapshot/templates/dumps/wikidump.conf.erb
14 files changed, 36 insertions(+), 21 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/92/400692/1

diff --git a/hieradata/hosts/snapshot1001.yaml 
b/hieradata/hosts/snapshot1001.yaml
index d27cca6..9015ce4 100644
--- a/hieradata/hosts/snapshot1001.yaml
+++ b/hieradata/hosts/snapshot1001.yaml
@@ -1 +1,2 @@
 snapshot::dumps::runtype: regular
+snapshot::dumps::php: php5
diff --git a/hieradata/hosts/snapshot1005.yaml 
b/hieradata/hosts/snapshot1005.yaml
index c34955f..f51a605 100644
--- a/hieradata/hosts/snapshot1005.yaml
+++ b/hieradata/hosts/snapshot1005.yaml
@@ -1 +1,2 @@
 snapshot::dumps::runtype: enwiki
+snapshot::dumps::php: php5
diff --git a/hieradata/hosts/snapshot1006.yaml 
b/hieradata/hosts/snapshot1006.yaml
index ea234d3..0480659 100644
--- a/hieradata/hosts/snapshot1006.yaml
+++ b/hieradata/hosts/snapshot1006.yaml
@@ -1 +1,2 @@
 snapshot::dumps::runtype: wikidatawiki
+snapshot::dumps::php: php5
diff --git a/hieradata/hosts/snapshot1007.yaml 
b/hieradata/hosts/snapshot1007.yaml
index d27cca6..9015ce4 100644
--- a/hieradata/hosts/snapshot1007.yaml
+++ b/hieradata/hosts/snapshot1007.yaml
@@ -1 +1,2 @@
 snapshot::dumps::runtype: regular
+snapshot::dumps::php: php5
diff --git a/modules/profile/manifests/dumps/generation/worker/common.pp 
b/modules/profile/manifests/dumps/generation/worker/common.pp
index 940a017..b1d9150 100644
--- a/modules/profile/manifests/dumps/generation/worker/common.pp
+++ b/modules/profile/manifests/dumps/generation/worker/common.pp
@@ -1,6 +1,7 @@
 class profile::dumps::generation::worker::common(
 $nfs_server = hiera('dumps_nfs_server'),
 $managed_subdirs = hiera('dumps_managed_subdirs'),
+$php = hiera('snapshot::dumps::php'),
 ) {
 # mw packages and dependencies
 require ::profile::mediawiki::scap_proxy
@@ -29,7 +30,7 @@
 cronsdir   =>  "${xmldumpsmount}/otherdumps",
 apachedir  => '/srv/mediawiki',
 }
-class { '::snapshot::dumps': }
+class { '::snapshot::dumps': php => $php}
 
 # scap3 deployment of dump scripts
 scap::target { 'dumps/dumps':
diff --git a/modules/snapshot/files/cron/dump-global-blocks.sh 
b/modules/snapshot/files/cron/dump-global-blocks.sh
index c3d2baf..16d2cef 100644
--- a/modules/snapshot/files/cron/dump-global-blocks.sh
+++ b/modules/snapshot/files/cron/dump-global-blocks.sh
@@ -6,12 +6,16 @@
 
 source /usr/local/etc/dump_functions.sh
 
+args="tools:php"
+results=`python "${repodir}/getconfigvals.py" --configfile "$configfile" 
--args "$args"`
+php=`getsetting "$results" "tools" "php"` || exit 1
+
 get_db_host() {
 apachedir=$1
 
 multiversionscript="${apachedir}/multiversion/MWScript.php"
 if [ -e "$multiversionscript" ]; then
-host=`php -q "$multiversionscript" 
extensions/CentralAuth/maintenance/getCentralAuthDBInfo.php --wiki="aawiki"` || 
(echo $host >& 2; host="")
+host=`$php -q "$multiversionscript" 
extensions/CentralAuth/maintenance/getCentralAuthDBInfo.php --wiki="aawiki"` || 
(echo $host >& 2; host="")
 fi
 if [ -z "$host" ]; then
 echo "can't locate db server for centralauth, exiting." >& 2
@@ -25,7 +29,7 @@
 
 multiversionscript="${apachedir}/multiversion/MWScript.php"
 if [ -e "$multiversionscript" ]; then
-db_user=`echo 'echo $wgDBadminuser;' | php "$multiversionscript" 
eval.php aawiki`
+db_user=`echo 'echo $wgDBadminuser;' | $php "$multiversionscript" 
eval.php aawiki`
 fi
 if [ -z "$db_user" ]; then
 echo "can't get db user name, exiting." >& 2
@@ -39,7 +43,7 @@
 
 multiversionscript="${apachedir}/multiversion/MWScript.php"
 if [ -e "$multiversionscript" ]; then
-db_pass=`echo 'echo $wgDBadminpassword;' | php "$multiversionscript" 
eval.php aawiki`
+db_pass=`echo 'echo $wgDBadminpassword;' | $php 

[MediaWiki-commits] [Gerrit] operations...scap[master]: add wmflabs config for dumps scap

[ArielGlenn (Code Review)]

ArielGlenn has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/400598 )

Change subject: add wmflabs config for dumps scap
..

add wmflabs config for dumps scap

Change-Id: I5b2b4fa35987f64cb74811ec39a98faf35752477
---
A dumps_targets_betacluster
M scap.cfg
2 files changed, 5 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/dumps/scap 
refs/changes/98/400598/1

diff --git a/dumps_targets_betacluster b/dumps_targets_betacluster
new file mode 100644
index 000..5e0ee0d
--- /dev/null
+++ b/dumps_targets_betacluster
@@ -0,0 +1,2 @@
+deployment-snapshot01.deployment-prep.eqiad.wmnet
+
diff --git a/scap.cfg b/scap.cfg
index 99348a9..ac1f466 100644
--- a/scap.cfg
+++ b/scap.cfg
@@ -3,3 +3,6 @@
 git_deploy_dir: /srv/deployment
 ssh_user: dumpsgen
 dsh_targets: dumps_targets
+
+[deployment-prep.eqiad.wmflabs]
+dsh_targets: dumps_targets_betacluster

-- 
To view, visit https://gerrit.wikimedia.org/r/400598
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I5b2b4fa35987f64cb74811ec39a98faf35752477
Gerrit-PatchSet: 1
Gerrit-Project: operations/dumps/scap
Gerrit-Branch: master
Gerrit-Owner: ArielGlenn 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...mwbzutils[master]: update for stretch build of same source

[ArielGlenn (Code Review)]

ArielGlenn has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/400590 )

Change subject: update for stretch build of same source
..


update for stretch build of same source

Change-Id: Iaea42d868f92ade983a45f72a467cdf5bb5e822d
---
M debian/changelog
M debian/compat
M debian/control
3 files changed, 8 insertions(+), 2 deletions(-)

Approvals:
  ArielGlenn: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/debian/changelog b/debian/changelog
index 14a10eb..e56d7ff 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,9 @@
+mwbzutils (0.0.7~wmf-2+stretch) stretch-wikimedia; urgency=medium
+
+  * Rebuild for stretch
+
+ -- Ariel T. Glenn   Thu, 28 Dec 2017 17:42:45 +0200
+
 mwbzutils (0.0.7~wmf1+trusty) trusty-wikimedia; urgency=medium
 
   * bz2compress output files with names that end in .bz2.otherext
diff --git a/debian/compat b/debian/compat
index 45a4fb7..ec63514 100644
--- a/debian/compat
+++ b/debian/compat
@@ -1 +1 @@
-8
+9
diff --git a/debian/control b/debian/control
index 1d385d0..e47fc72 100644
--- a/debian/control
+++ b/debian/control
@@ -2,7 +2,7 @@
 Section: misc
 Priority: extra
 Maintainer: Ariel T. Glenn 
-Build-Depends: debhelper (>= 8.0.0), libbz2-1.0, zlib1g
+Build-Depends: debhelper (>= 9.0.0), libbz2-1.0, zlib1g
 Standards-Version: 3.9.2
 Homepage: https://www.mediawiki.org/wiki/Manual:Mwbzutils
 Vcs-Git: git clone https://gerrit.wikimedia.org/r/operations/dumps/mwbzutils

-- 
To view, visit https://gerrit.wikimedia.org/r/400590
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Iaea42d868f92ade983a45f72a467cdf5bb5e822d
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/mwbzutils
Gerrit-Branch: master
Gerrit-Owner: ArielGlenn 
Gerrit-Reviewer: ArielGlenn 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...mwbzutils[master]: update for stretch build of same source

[ArielGlenn (Code Review)]

ArielGlenn has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/400590 )

Change subject: update for stretch build of same source
..

update for stretch build of same source

Change-Id: Iaea42d868f92ade983a45f72a467cdf5bb5e822d
---
M debian/changelog
M debian/compat
M debian/control
3 files changed, 8 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/mwbzutils 
refs/changes/90/400590/1

diff --git a/debian/changelog b/debian/changelog
index 14a10eb..e56d7ff 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,9 @@
+mwbzutils (0.0.7~wmf-2+stretch) stretch-wikimedia; urgency=medium
+
+  * Rebuild for stretch
+
+ -- Ariel T. Glenn   Thu, 28 Dec 2017 17:42:45 +0200
+
 mwbzutils (0.0.7~wmf1+trusty) trusty-wikimedia; urgency=medium
 
   * bz2compress output files with names that end in .bz2.otherext
diff --git a/debian/compat b/debian/compat
index 45a4fb7..ec63514 100644
--- a/debian/compat
+++ b/debian/compat
@@ -1 +1 @@
-8
+9
diff --git a/debian/control b/debian/control
index 1d385d0..e47fc72 100644
--- a/debian/control
+++ b/debian/control
@@ -2,7 +2,7 @@
 Section: misc
 Priority: extra
 Maintainer: Ariel T. Glenn 
-Build-Depends: debhelper (>= 8.0.0), libbz2-1.0, zlib1g
+Build-Depends: debhelper (>= 9.0.0), libbz2-1.0, zlib1g
 Standards-Version: 3.9.2
 Homepage: https://www.mediawiki.org/wiki/Manual:Mwbzutils
 Vcs-Git: git clone https://gerrit.wikimedia.org/r/operations/dumps/mwbzutils

-- 
To view, visit https://gerrit.wikimedia.org/r/400590
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Iaea42d868f92ade983a45f72a467cdf5bb5e822d
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/mwbzutils
Gerrit-Branch: master
Gerrit-Owner: ArielGlenn 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: get rid of redundant code in dumps nfs server manifests

[ArielGlenn (Code Review)]

ArielGlenn has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/400405 )

Change subject: get rid of redundant code in dumps nfs server manifests
..


get rid of redundant code in dumps nfs server manifests

Change-Id: I902227b96e6f2e315fefaf23eba8b9c7861fdace
---
R modules/profile/manifests/dumps/generation/server/common.pp
M modules/profile/manifests/dumps/generation/server/primary.pp
M modules/role/manifests/dumps/generation/server/fallback.pp
3 files changed, 6 insertions(+), 20 deletions(-)

Approvals:
  ArielGlenn: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/profile/manifests/dumps/generation/server/fallback.pp 
b/modules/profile/manifests/dumps/generation/server/common.pp
similarity index 91%
rename from modules/profile/manifests/dumps/generation/server/fallback.pp
rename to modules/profile/manifests/dumps/generation/server/common.pp
index d247e91..e0fee61 100644
--- a/modules/profile/manifests/dumps/generation/server/fallback.pp
+++ b/modules/profile/manifests/dumps/generation/server/common.pp
@@ -1,4 +1,4 @@
-class profile::dumps::generation::server::fallback(
+class profile::dumps::generation::server::common(
 $datadir = hiera('profile::dumps::basedatadir'),
 $xmldumpsdir = hiera('profile::dumps::xmldumpspublicdir'),
 $miscdatasetsdir = hiera('profile::dumps::miscdumpsdir'),
diff --git a/modules/profile/manifests/dumps/generation/server/primary.pp 
b/modules/profile/manifests/dumps/generation/server/primary.pp
index 7c5b98c..bb88970 100644
--- a/modules/profile/manifests/dumps/generation/server/primary.pp
+++ b/modules/profile/manifests/dumps/generation/server/primary.pp
@@ -1,24 +1,10 @@
-class profile::dumps::generation::server::primary(
-$datadir = hiera('profile::dumps::basedatadir'),
-$xmldumpsdir = hiera('profile::dumps::xmldumpspublicdir'),
-$miscdatasetsdir = hiera('profile::dumps::miscdumpsdir'),
-$dumpstempdir = hiera('profile::dumps::dumpstempdir'),
-) {
-class { '::dumpsuser': }
-
-class { '::dumps::generation::server::dirs':
-datadir => $datadir,
-xmldumpsdir => $xmldumpsdir,
-tempdir => $dumpstempdir,
-miscdatasetsdir => $miscdatasetsdir,
-user=> $dumpsuser::user,
-group   => $dumpsuser::group,
-}
+class profile::dumps::generation::server::primary {
+require profile::dumps::generation::server::common
 
 class { '::dumps::generation::server::rsyncer':
-xmldumpsdir=> $xmldumpsdir,
+xmldumpsdir=> 
$profile::dumps::generation::server::common::xmldumpsdir,
 xmlremotedirs  => 
'dumpsdata1002.eqiad.wmnet::data/xmldatadumps/public/,dataset1001.wikimedia.org::data/xmldatadumps/public/',
-miscdumpsdir   => $miscdatasetsdir,
+miscdumpsdir   => 
$profile::dumps::generation::server::common::miscdatasetsdir,
 miscremotedirs => 
'dumpsdata1002.eqiad.wmnet::data/otherdumps/,dataset1001.wikimedia.org::data/xmldatadumps/public/other/',
 }
 }
diff --git a/modules/role/manifests/dumps/generation/server/fallback.pp 
b/modules/role/manifests/dumps/generation/server/fallback.pp
index 36440bb..4ae4988 100644
--- a/modules/role/manifests/dumps/generation/server/fallback.pp
+++ b/modules/role/manifests/dumps/generation/server/fallback.pp
@@ -3,7 +3,7 @@
 
 include ::standard
 include ::profile::base::firewall
-include ::profile::dumps::generation::server::fallback
+include ::profile::dumps::generation::server::common
 include ::profile::dumps::generation::server::rsync
 include ::profile::dumps::rsyncer_peer
 include ::profile::dumps::nfs

-- 
To view, visit https://gerrit.wikimedia.org/r/400405
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I902227b96e6f2e315fefaf23eba8b9c7861fdace
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: ArielGlenn 
Gerrit-Reviewer: ArielGlenn 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: get rid of redundant code in dumps nfs server manifests

[ArielGlenn (Code Review)]

ArielGlenn has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/400405 )

Change subject: get rid of redundant code in dumps nfs server manifests
..

get rid of redundant code in dumps nfs server manifests

Change-Id: I902227b96e6f2e315fefaf23eba8b9c7861fdace
---
R modules/profile/manifests/dumps/generation/server/common.pp
M modules/profile/manifests/dumps/generation/server/primary.pp
M modules/role/manifests/dumps/generation/server/fallback.pp
3 files changed, 6 insertions(+), 20 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/05/400405/1

diff --git a/modules/profile/manifests/dumps/generation/server/fallback.pp 
b/modules/profile/manifests/dumps/generation/server/common.pp
similarity index 91%
rename from modules/profile/manifests/dumps/generation/server/fallback.pp
rename to modules/profile/manifests/dumps/generation/server/common.pp
index d247e91..e0fee61 100644
--- a/modules/profile/manifests/dumps/generation/server/fallback.pp
+++ b/modules/profile/manifests/dumps/generation/server/common.pp
@@ -1,4 +1,4 @@
-class profile::dumps::generation::server::fallback(
+class profile::dumps::generation::server::common(
 $datadir = hiera('profile::dumps::basedatadir'),
 $xmldumpsdir = hiera('profile::dumps::xmldumpspublicdir'),
 $miscdatasetsdir = hiera('profile::dumps::miscdumpsdir'),
diff --git a/modules/profile/manifests/dumps/generation/server/primary.pp 
b/modules/profile/manifests/dumps/generation/server/primary.pp
index 7c5b98c..bb88970 100644
--- a/modules/profile/manifests/dumps/generation/server/primary.pp
+++ b/modules/profile/manifests/dumps/generation/server/primary.pp
@@ -1,24 +1,10 @@
-class profile::dumps::generation::server::primary(
-$datadir = hiera('profile::dumps::basedatadir'),
-$xmldumpsdir = hiera('profile::dumps::xmldumpspublicdir'),
-$miscdatasetsdir = hiera('profile::dumps::miscdumpsdir'),
-$dumpstempdir = hiera('profile::dumps::dumpstempdir'),
-) {
-class { '::dumpsuser': }
-
-class { '::dumps::generation::server::dirs':
-datadir => $datadir,
-xmldumpsdir => $xmldumpsdir,
-tempdir => $dumpstempdir,
-miscdatasetsdir => $miscdatasetsdir,
-user=> $dumpsuser::user,
-group   => $dumpsuser::group,
-}
+class profile::dumps::generation::server::primary {
+require profile::dumps::generation::server::common
 
 class { '::dumps::generation::server::rsyncer':
-xmldumpsdir=> $xmldumpsdir,
+xmldumpsdir=> 
$profile::dumps::generation::server::common::xmldumpsdir,
 xmlremotedirs  => 
'dumpsdata1002.eqiad.wmnet::data/xmldatadumps/public/,dataset1001.wikimedia.org::data/xmldatadumps/public/',
-miscdumpsdir   => $miscdatasetsdir,
+miscdumpsdir   => 
$profile::dumps::generation::server::common::miscdatasetsdir,
 miscremotedirs => 
'dumpsdata1002.eqiad.wmnet::data/otherdumps/,dataset1001.wikimedia.org::data/xmldatadumps/public/other/',
 }
 }
diff --git a/modules/role/manifests/dumps/generation/server/fallback.pp 
b/modules/role/manifests/dumps/generation/server/fallback.pp
index 36440bb..4ae4988 100644
--- a/modules/role/manifests/dumps/generation/server/fallback.pp
+++ b/modules/role/manifests/dumps/generation/server/fallback.pp
@@ -3,7 +3,7 @@
 
 include ::standard
 include ::profile::base::firewall
-include ::profile::dumps::generation::server::fallback
+include ::profile::dumps::generation::server::common
 include ::profile::dumps::generation::server::rsync
 include ::profile::dumps::rsyncer_peer
 include ::profile::dumps::nfs

-- 
To view, visit https://gerrit.wikimedia.org/r/400405
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I902227b96e6f2e315fefaf23eba8b9c7861fdace
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: ArielGlenn 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: get rid of redundant code in dumps web server manifests

[ArielGlenn (Code Review)]

ArielGlenn has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/400403 )

Change subject: get rid of redundant code in dumps web server manifests
..


get rid of redundant code in dumps web server manifests

Change-Id: I7bfc4e5e7d762965abf3aed7cea58e0ebb846f98
---
D modules/dumps/manifests/web/xmldumps_active.pp
M modules/profile/manifests/dumps/web/xmldumps_active.pp
R modules/profile/manifests/dumps/web/xmldumps_common.pp
M modules/role/manifests/dumps/web/xmldumps_active.pp
M modules/role/manifests/dumps/web/xmldumps_fallback.pp
5 files changed, 19 insertions(+), 62 deletions(-)

Approvals:
  ArielGlenn: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/dumps/manifests/web/xmldumps_active.pp 
b/modules/dumps/manifests/web/xmldumps_active.pp
deleted file mode 100644
index 7476d2e..000
--- a/modules/dumps/manifests/web/xmldumps_active.pp
+++ /dev/null
@@ -1,28 +0,0 @@
-class dumps::web::xmldumps_active(
-$do_acme  = true,
-$datadir  = undef,
-$xmldumpsdir  = undef,
-$miscdatasetsdir  = undef,
-$logs_dest= undef,
-$htmldumps_server = undef,
-$xmldumps_server  = undef,
-$webuser  = undef,
-$webgroup = undef,
-) {
-# active web server
-class {'::dumps::web::xmldumps':
-do_acme  => $do_acme,
-datadir  => $datadir,
-xmldumpsdir  => $xmldumpsdir,
-miscdatasetsdir  => $miscdatasetsdir,
-htmldumps_server => $htmldumps_server,
-xmldumps_server  => $xmldumps_server,
-webuser  => $webuser,
-webgroup => $webgroup,
-}
-
-# only the active web server should be syncing nginx logs
-class {'::dumps::web::rsync::nginxlogs':
-dest   => $logs_dest,
-}
-}
diff --git a/modules/profile/manifests/dumps/web/xmldumps_active.pp 
b/modules/profile/manifests/dumps/web/xmldumps_active.pp
index fd9783b..a600316 100644
--- a/modules/profile/manifests/dumps/web/xmldumps_active.pp
+++ b/modules/profile/manifests/dumps/web/xmldumps_active.pp
@@ -1,32 +1,18 @@
-class profile::dumps::web::xmldumps_active(
-$do_acme = hiera('do_acme'),
-$datadir = hiera('profile::dumps::basedatadir'),
-$xmldumpsdir = hiera('profile::dumps::xmldumpspublicdir'),
-$miscdatasetsdir = hiera('profile::dumps::miscdumpsdir'),
-) {
-interface::add_ip6_mapped { 'main': }
+class profile::dumps::web::xmldumps_active {
+require profile::dumps::web::xmldumps_common
 
-require profile::dumps::web::nginx
-class { '::dumpsuser': }
-
-class {'::dumps::web::xmldumps_active':
-do_acme  => $do_acme,
-datadir  => $datadir,
-xmldumpsdir  => $xmldumpsdir,
-miscdatasetsdir  => $miscdatasetsdir,
-logs_dest=> 
'stat1005.eqiad.wmnet::srv/log/webrequest/archive/dumps.wikimedia.org/',
-htmldumps_server => 'francium.eqiad.wmnet',
-xmldumps_server  => 'dumps.wikimedia.org',
-webuser  => 'dumpsgen',
-webgroup => 'dumpsgen',
+# copy dumps web server logs to stat host
+class {'::dumps::web::rsync::nginxlogs':
+dest => 
'stat1005.eqiad.wmnet::srv/log/webrequest/archive/dumps.wikimedia.org/',
 }
+
 # copy dumps and other datasets to fallback host(s) and to labs
 class {'::dumps::copying::peers':
 desthost => 'ms1001.wikimedia.org',
 }
 class {'::dumps::copying::labs':
 labhost => 'labstore1003.eqiad.wmnet',
-xmldumpsdir => $xmldumpsdir,
-miscdatasetsdir => $miscdatasetsdir,
+xmldumpsdir => $profile::dumps::web::xmldumps_common::xmldumpsdir,
+miscdatasetsdir => 
$profile::dumps::web::xmldumps_common::miscdatasetsdir,
 }
 }
diff --git a/modules/profile/manifests/dumps/web/xmldumps_fallback.pp 
b/modules/profile/manifests/dumps/web/xmldumps_common.pp
similarity index 68%
rename from modules/profile/manifests/dumps/web/xmldumps_fallback.pp
rename to modules/profile/manifests/dumps/web/xmldumps_common.pp
index 0978a57..5fa5f7d 100644
--- a/modules/profile/manifests/dumps/web/xmldumps_fallback.pp
+++ b/modules/profile/manifests/dumps/web/xmldumps_common.pp
@@ -1,4 +1,4 @@
-class profile::dumps::web::xmldumps_fallback(
+class profile::dumps::web::xmldumps_common(
 $do_acme = hiera('do_acme'),
 $datadir = hiera('profile::dumps::basedatadir'),
 $xmldumpsdir = hiera('profile::dumps::xmldumpspublicdir'),
@@ -7,6 +7,14 @@
 interface::add_ip6_mapped { 'main': }
 
 require profile::dumps::web::nginx
+
+# better here once than copy-pasted into multiple roles.
+require profile::dumps::nfs
+require profile::dumps::web::rsync_server
+require profile::dumps::web::dumpstatusfiles_sync
+require profile::dumps::web::cleanup
+require profile::dumps::web::cleanup_miscdatasets
+
 class { 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: get rid of redundant code in dumps web server manifests

[ArielGlenn (Code Review)]

ArielGlenn has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/400403 )

Change subject: get rid of redundant code in dumps web server manifests
..

get rid of redundant code in dumps web server manifests

Change-Id: I7bfc4e5e7d762965abf3aed7cea58e0ebb846f98
---
D modules/dumps/manifests/web/xmldumps_active.pp
M modules/profile/manifests/dumps/web/xmldumps_active.pp
R modules/profile/manifests/dumps/web/xmldumps_common.pp
M modules/role/manifests/dumps/web/xmldumps_active.pp
M modules/role/manifests/dumps/web/xmldumps_fallback.pp
5 files changed, 19 insertions(+), 62 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/03/400403/1

diff --git a/modules/dumps/manifests/web/xmldumps_active.pp 
b/modules/dumps/manifests/web/xmldumps_active.pp
deleted file mode 100644
index 7476d2e..000
--- a/modules/dumps/manifests/web/xmldumps_active.pp
+++ /dev/null
@@ -1,28 +0,0 @@
-class dumps::web::xmldumps_active(
-$do_acme  = true,
-$datadir  = undef,
-$xmldumpsdir  = undef,
-$miscdatasetsdir  = undef,
-$logs_dest= undef,
-$htmldumps_server = undef,
-$xmldumps_server  = undef,
-$webuser  = undef,
-$webgroup = undef,
-) {
-# active web server
-class {'::dumps::web::xmldumps':
-do_acme  => $do_acme,
-datadir  => $datadir,
-xmldumpsdir  => $xmldumpsdir,
-miscdatasetsdir  => $miscdatasetsdir,
-htmldumps_server => $htmldumps_server,
-xmldumps_server  => $xmldumps_server,
-webuser  => $webuser,
-webgroup => $webgroup,
-}
-
-# only the active web server should be syncing nginx logs
-class {'::dumps::web::rsync::nginxlogs':
-dest   => $logs_dest,
-}
-}
diff --git a/modules/profile/manifests/dumps/web/xmldumps_active.pp 
b/modules/profile/manifests/dumps/web/xmldumps_active.pp
index fd9783b..a600316 100644
--- a/modules/profile/manifests/dumps/web/xmldumps_active.pp
+++ b/modules/profile/manifests/dumps/web/xmldumps_active.pp
@@ -1,32 +1,18 @@
-class profile::dumps::web::xmldumps_active(
-$do_acme = hiera('do_acme'),
-$datadir = hiera('profile::dumps::basedatadir'),
-$xmldumpsdir = hiera('profile::dumps::xmldumpspublicdir'),
-$miscdatasetsdir = hiera('profile::dumps::miscdumpsdir'),
-) {
-interface::add_ip6_mapped { 'main': }
+class profile::dumps::web::xmldumps_active {
+require profile::dumps::web::xmldumps_common
 
-require profile::dumps::web::nginx
-class { '::dumpsuser': }
-
-class {'::dumps::web::xmldumps_active':
-do_acme  => $do_acme,
-datadir  => $datadir,
-xmldumpsdir  => $xmldumpsdir,
-miscdatasetsdir  => $miscdatasetsdir,
-logs_dest=> 
'stat1005.eqiad.wmnet::srv/log/webrequest/archive/dumps.wikimedia.org/',
-htmldumps_server => 'francium.eqiad.wmnet',
-xmldumps_server  => 'dumps.wikimedia.org',
-webuser  => 'dumpsgen',
-webgroup => 'dumpsgen',
+# copy dumps web server logs to stat host
+class {'::dumps::web::rsync::nginxlogs':
+dest => 
'stat1005.eqiad.wmnet::srv/log/webrequest/archive/dumps.wikimedia.org/',
 }
+
 # copy dumps and other datasets to fallback host(s) and to labs
 class {'::dumps::copying::peers':
 desthost => 'ms1001.wikimedia.org',
 }
 class {'::dumps::copying::labs':
 labhost => 'labstore1003.eqiad.wmnet',
-xmldumpsdir => $xmldumpsdir,
-miscdatasetsdir => $miscdatasetsdir,
+xmldumpsdir => $profile::dumps::web::xmldumps_common::xmldumpsdir,
+miscdatasetsdir => 
$profile::dumps::web::xmldumps_common::miscdatasetsdir,
 }
 }
diff --git a/modules/profile/manifests/dumps/web/xmldumps_fallback.pp 
b/modules/profile/manifests/dumps/web/xmldumps_common.pp
similarity index 68%
rename from modules/profile/manifests/dumps/web/xmldumps_fallback.pp
rename to modules/profile/manifests/dumps/web/xmldumps_common.pp
index 0978a57..5fa5f7d 100644
--- a/modules/profile/manifests/dumps/web/xmldumps_fallback.pp
+++ b/modules/profile/manifests/dumps/web/xmldumps_common.pp
@@ -1,4 +1,4 @@
-class profile::dumps::web::xmldumps_fallback(
+class profile::dumps::web::xmldumps_common(
 $do_acme = hiera('do_acme'),
 $datadir = hiera('profile::dumps::basedatadir'),
 $xmldumpsdir = hiera('profile::dumps::xmldumpspublicdir'),
@@ -7,6 +7,14 @@
 interface::add_ip6_mapped { 'main': }
 
 require profile::dumps::web::nginx
+
+# better here once than copy-pasted into multiple roles.
+require profile::dumps::nfs
+require profile::dumps::web::rsync_server
+require profile::dumps::web::dumpstatusfiles_sync
+require profile::dumps::web::cleanup
+require profile::dumps::web::cleanup_miscdatasets
+
 class { 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: move ipv6 setup for dump web servers to the appropriate prof...

[ArielGlenn (Code Review)]

ArielGlenn has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/400394 )

Change subject: move ipv6 setup for dump web servers to the appropriate profiles
..


move ipv6 setup for dump web servers to the appropriate profiles

Change-Id: I774bd42eda1b04bcd5c86e13f62bfd2a42301108
---
M manifests/site.pp
M modules/profile/manifests/dumps/web/xmldumps_active.pp
M modules/profile/manifests/dumps/web/xmldumps_fallback.pp
3 files changed, 4 insertions(+), 6 deletions(-)

Approvals:
  ArielGlenn: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/manifests/site.pp b/manifests/site.pp
index d19f7e5..df1b8a0 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -290,10 +290,7 @@
 }
 
 node 'dataset1001.wikimedia.org' {
-
 role(dumps::web::xmldumps_active)
-
-interface::add_ip6_mapped { 'main': }
 }
 
 # MariaDB 10
@@ -1445,10 +1442,7 @@
 
 # also see dataset1001
 node 'ms1001.wikimedia.org' {
-
 role(dumps::web::xmldumps_fallback)
-
-interface::add_ip6_mapped { 'main': }
 }
 
 node 'ms1002.eqiad.wmnet' {
diff --git a/modules/profile/manifests/dumps/web/xmldumps_active.pp 
b/modules/profile/manifests/dumps/web/xmldumps_active.pp
index b785a63..fd9783b 100644
--- a/modules/profile/manifests/dumps/web/xmldumps_active.pp
+++ b/modules/profile/manifests/dumps/web/xmldumps_active.pp
@@ -4,6 +4,8 @@
 $xmldumpsdir = hiera('profile::dumps::xmldumpspublicdir'),
 $miscdatasetsdir = hiera('profile::dumps::miscdumpsdir'),
 ) {
+interface::add_ip6_mapped { 'main': }
+
 require profile::dumps::web::nginx
 class { '::dumpsuser': }
 
diff --git a/modules/profile/manifests/dumps/web/xmldumps_fallback.pp 
b/modules/profile/manifests/dumps/web/xmldumps_fallback.pp
index cf00edd..0978a57 100644
--- a/modules/profile/manifests/dumps/web/xmldumps_fallback.pp
+++ b/modules/profile/manifests/dumps/web/xmldumps_fallback.pp
@@ -4,6 +4,8 @@
 $xmldumpsdir = hiera('profile::dumps::xmldumpspublicdir'),
 $miscdatasetsdir = hiera('profile::dumps::miscdumpsdir'),
 ) {
+interface::add_ip6_mapped { 'main': }
+
 require profile::dumps::web::nginx
 class { '::dumpsuser': }
 

-- 
To view, visit https://gerrit.wikimedia.org/r/400394
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I774bd42eda1b04bcd5c86e13f62bfd2a42301108
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: ArielGlenn 
Gerrit-Reviewer: ArielGlenn 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: move ipv6 setup for dump web servers to the appropriate prof...

[ArielGlenn (Code Review)]

ArielGlenn has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/400394 )

Change subject: move ipv6 setup for dump web servers to the appropriate profiles
..

move ipv6 setup for dump web servers to the appropriate profiles

Change-Id: I774bd42eda1b04bcd5c86e13f62bfd2a42301108
---
M manifests/site.pp
M modules/profile/manifests/dumps/web/xmldumps_active.pp
M modules/profile/manifests/dumps/web/xmldumps_fallback.pp
3 files changed, 4 insertions(+), 6 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/94/400394/1

diff --git a/manifests/site.pp b/manifests/site.pp
index d19f7e5..df1b8a0 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -290,10 +290,7 @@
 }
 
 node 'dataset1001.wikimedia.org' {
-
 role(dumps::web::xmldumps_active)
-
-interface::add_ip6_mapped { 'main': }
 }
 
 # MariaDB 10
@@ -1445,10 +1442,7 @@
 
 # also see dataset1001
 node 'ms1001.wikimedia.org' {
-
 role(dumps::web::xmldumps_fallback)
-
-interface::add_ip6_mapped { 'main': }
 }
 
 node 'ms1002.eqiad.wmnet' {
diff --git a/modules/profile/manifests/dumps/web/xmldumps_active.pp 
b/modules/profile/manifests/dumps/web/xmldumps_active.pp
index b785a63..fd9783b 100644
--- a/modules/profile/manifests/dumps/web/xmldumps_active.pp
+++ b/modules/profile/manifests/dumps/web/xmldumps_active.pp
@@ -4,6 +4,8 @@
 $xmldumpsdir = hiera('profile::dumps::xmldumpspublicdir'),
 $miscdatasetsdir = hiera('profile::dumps::miscdumpsdir'),
 ) {
+interface::add_ip6_mapped { 'main': }
+
 require profile::dumps::web::nginx
 class { '::dumpsuser': }
 
diff --git a/modules/profile/manifests/dumps/web/xmldumps_fallback.pp 
b/modules/profile/manifests/dumps/web/xmldumps_fallback.pp
index cf00edd..0978a57 100644
--- a/modules/profile/manifests/dumps/web/xmldumps_fallback.pp
+++ b/modules/profile/manifests/dumps/web/xmldumps_fallback.pp
@@ -4,6 +4,8 @@
 $xmldumpsdir = hiera('profile::dumps::xmldumpspublicdir'),
 $miscdatasetsdir = hiera('profile::dumps::miscdumpsdir'),
 ) {
+interface::add_ip6_mapped { 'main': }
+
 require profile::dumps::web::nginx
 class { '::dumpsuser': }
 

-- 
To view, visit https://gerrit.wikimedia.org/r/400394
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I774bd42eda1b04bcd5c86e13f62bfd2a42301108
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: ArielGlenn 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: create a profile for nginx-extras package for dumps

[ArielGlenn (Code Review)]

ArielGlenn has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/400391 )

Change subject: create a profile for nginx-extras package for dumps
..


create a profile for nginx-extras package for dumps

this lets us move the nginx class decl out of the dumps module

Change-Id: Ifd686141fb0b8392bd888d36976153341e279131
---
M modules/dumps/manifests/web/htmldumps.pp
M modules/dumps/manifests/web/xmldumps.pp
M modules/profile/manifests/dumps/web/htmldumps.pp
A modules/profile/manifests/dumps/web/nginx.pp
M modules/profile/manifests/dumps/web/xmldumps_active.pp
M modules/profile/manifests/dumps/web/xmldumps_fallback.pp
6 files changed, 10 insertions(+), 10 deletions(-)

Approvals:
  ArielGlenn: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/dumps/manifests/web/htmldumps.pp 
b/modules/dumps/manifests/web/htmldumps.pp
index adf6229..d5e95ce 100644
--- a/modules/dumps/manifests/web/htmldumps.pp
+++ b/modules/dumps/manifests/web/htmldumps.pp
@@ -22,12 +22,6 @@
 group  => 'htmldumps-admin',
 }
 
-# don't know if we want bw limits etc so let's slap 'extras'
-# on there
-class { '::nginx':
-variant => 'extras',
-}
-
 nginx::site { 'htmldumps':
 content => template('dumps/web/htmldumps/nginx.conf.erb'),
 notify  => Service['nginx'],
diff --git a/modules/dumps/manifests/web/xmldumps.pp 
b/modules/dumps/manifests/web/xmldumps.pp
index 7c7e87e..b4b9a72 100644
--- a/modules/dumps/manifests/web/xmldumps.pp
+++ b/modules/dumps/manifests/web/xmldumps.pp
@@ -17,10 +17,6 @@
 webgroup=> $webgroup,
 }
 
-class { '::nginx':
-variant => 'extras',
-}
-
 $ssl_settings = ssl_ciphersuite('nginx', 'mid', true)
 
 letsencrypt::cert::integrated { 'dumps':
diff --git a/modules/profile/manifests/dumps/web/htmldumps.pp 
b/modules/profile/manifests/dumps/web/htmldumps.pp
index 88d29f0..800eff2 100644
--- a/modules/profile/manifests/dumps/web/htmldumps.pp
+++ b/modules/profile/manifests/dumps/web/htmldumps.pp
@@ -1,4 +1,6 @@
 class profile::dumps::web::htmldumps {
+require profile::dumps::web::nginx
+
 class {'::dumps::web::htmldumps': htmldumps_server => 
'francium.eqiad.wmnet'}
 
 ferm::service { 'html_dumps_http':
diff --git a/modules/profile/manifests/dumps/web/nginx.pp 
b/modules/profile/manifests/dumps/web/nginx.pp
new file mode 100644
index 000..df94d06
--- /dev/null
+++ b/modules/profile/manifests/dumps/web/nginx.pp
@@ -0,0 +1,6 @@
+class profile::dumps::web::nginx {
+# includes module for bandwidth limits
+class { '::nginx':
+variant => 'extras',
+}
+}
diff --git a/modules/profile/manifests/dumps/web/xmldumps_active.pp 
b/modules/profile/manifests/dumps/web/xmldumps_active.pp
index d7d1260..b785a63 100644
--- a/modules/profile/manifests/dumps/web/xmldumps_active.pp
+++ b/modules/profile/manifests/dumps/web/xmldumps_active.pp
@@ -4,6 +4,7 @@
 $xmldumpsdir = hiera('profile::dumps::xmldumpspublicdir'),
 $miscdatasetsdir = hiera('profile::dumps::miscdumpsdir'),
 ) {
+require profile::dumps::web::nginx
 class { '::dumpsuser': }
 
 class {'::dumps::web::xmldumps_active':
diff --git a/modules/profile/manifests/dumps/web/xmldumps_fallback.pp 
b/modules/profile/manifests/dumps/web/xmldumps_fallback.pp
index dbe5396..cf00edd 100644
--- a/modules/profile/manifests/dumps/web/xmldumps_fallback.pp
+++ b/modules/profile/manifests/dumps/web/xmldumps_fallback.pp
@@ -4,6 +4,7 @@
 $xmldumpsdir = hiera('profile::dumps::xmldumpspublicdir'),
 $miscdatasetsdir = hiera('profile::dumps::miscdumpsdir'),
 ) {
+require profile::dumps::web::nginx
 class { '::dumpsuser': }
 
 class {'::dumps::web::xmldumps':

-- 
To view, visit https://gerrit.wikimedia.org/r/400391
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ifd686141fb0b8392bd888d36976153341e279131
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: ArielGlenn 
Gerrit-Reviewer: ArielGlenn 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: create a profile for nginx-extras package for dumps

[ArielGlenn (Code Review)]

ArielGlenn has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/400391 )

Change subject: create a profile for nginx-extras package for dumps
..

create a profile for nginx-extras package for dumps

this lets us move the nginx class decl out of the dumps module

Change-Id: Ifd686141fb0b8392bd888d36976153341e279131
---
M modules/dumps/manifests/web/htmldumps.pp
M modules/dumps/manifests/web/xmldumps.pp
M modules/profile/manifests/dumps/web/htmldumps.pp
A modules/profile/manifests/dumps/web/nginx.pp
M modules/profile/manifests/dumps/web/xmldumps_active.pp
M modules/profile/manifests/dumps/web/xmldumps_fallback.pp
6 files changed, 10 insertions(+), 10 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/91/400391/1

diff --git a/modules/dumps/manifests/web/htmldumps.pp 
b/modules/dumps/manifests/web/htmldumps.pp
index adf6229..d5e95ce 100644
--- a/modules/dumps/manifests/web/htmldumps.pp
+++ b/modules/dumps/manifests/web/htmldumps.pp
@@ -22,12 +22,6 @@
 group  => 'htmldumps-admin',
 }
 
-# don't know if we want bw limits etc so let's slap 'extras'
-# on there
-class { '::nginx':
-variant => 'extras',
-}
-
 nginx::site { 'htmldumps':
 content => template('dumps/web/htmldumps/nginx.conf.erb'),
 notify  => Service['nginx'],
diff --git a/modules/dumps/manifests/web/xmldumps.pp 
b/modules/dumps/manifests/web/xmldumps.pp
index 7c7e87e..b4b9a72 100644
--- a/modules/dumps/manifests/web/xmldumps.pp
+++ b/modules/dumps/manifests/web/xmldumps.pp
@@ -17,10 +17,6 @@
 webgroup=> $webgroup,
 }
 
-class { '::nginx':
-variant => 'extras',
-}
-
 $ssl_settings = ssl_ciphersuite('nginx', 'mid', true)
 
 letsencrypt::cert::integrated { 'dumps':
diff --git a/modules/profile/manifests/dumps/web/htmldumps.pp 
b/modules/profile/manifests/dumps/web/htmldumps.pp
index 88d29f0..800eff2 100644
--- a/modules/profile/manifests/dumps/web/htmldumps.pp
+++ b/modules/profile/manifests/dumps/web/htmldumps.pp
@@ -1,4 +1,6 @@
 class profile::dumps::web::htmldumps {
+require profile::dumps::web::nginx
+
 class {'::dumps::web::htmldumps': htmldumps_server => 
'francium.eqiad.wmnet'}
 
 ferm::service { 'html_dumps_http':
diff --git a/modules/profile/manifests/dumps/web/nginx.pp 
b/modules/profile/manifests/dumps/web/nginx.pp
new file mode 100644
index 000..df94d06
--- /dev/null
+++ b/modules/profile/manifests/dumps/web/nginx.pp
@@ -0,0 +1,6 @@
+class profile::dumps::web::nginx {
+# includes module for bandwidth limits
+class { '::nginx':
+variant => 'extras',
+}
+}
diff --git a/modules/profile/manifests/dumps/web/xmldumps_active.pp 
b/modules/profile/manifests/dumps/web/xmldumps_active.pp
index d7d1260..b785a63 100644
--- a/modules/profile/manifests/dumps/web/xmldumps_active.pp
+++ b/modules/profile/manifests/dumps/web/xmldumps_active.pp
@@ -4,6 +4,7 @@
 $xmldumpsdir = hiera('profile::dumps::xmldumpspublicdir'),
 $miscdatasetsdir = hiera('profile::dumps::miscdumpsdir'),
 ) {
+require profile::dumps::web::nginx
 class { '::dumpsuser': }
 
 class {'::dumps::web::xmldumps_active':
diff --git a/modules/profile/manifests/dumps/web/xmldumps_fallback.pp 
b/modules/profile/manifests/dumps/web/xmldumps_fallback.pp
index dbe5396..cf00edd 100644
--- a/modules/profile/manifests/dumps/web/xmldumps_fallback.pp
+++ b/modules/profile/manifests/dumps/web/xmldumps_fallback.pp
@@ -4,6 +4,7 @@
 $xmldumpsdir = hiera('profile::dumps::xmldumpspublicdir'),
 $miscdatasetsdir = hiera('profile::dumps::miscdumpsdir'),
 ) {
+require profile::dumps::web::nginx
 class { '::dumpsuser': }
 
 class {'::dumps::web::xmldumps':

-- 
To view, visit https://gerrit.wikimedia.org/r/400391
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ifd686141fb0b8392bd888d36976153341e279131
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: ArielGlenn 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: allow dumps nfs server to be configured without clients if n...

[ArielGlenn (Code Review)]

ArielGlenn has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/400387 )

Change subject: allow dumps nfs server to be configured without clients if 
needed
..


allow dumps nfs server to be configured without clients if needed

Change-Id: Icd88705b91ba7d2e040876584d40a5ca53b736a5
---
M modules/profile/manifests/dumps/nfs.pp
1 file changed, 2 insertions(+), 0 deletions(-)

Approvals:
  ArielGlenn: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/profile/manifests/dumps/nfs.pp 
b/modules/profile/manifests/dumps/nfs.pp
index bc8..8965a46 100644
--- a/modules/profile/manifests/dumps/nfs.pp
+++ b/modules/profile/manifests/dumps/nfs.pp
@@ -11,6 +11,8 @@
 $clients = {'generation' => pick($clients_all['snapshots'], [])}
 } elsif ($clients_wanted == 'public') {
 $clients = {'public' => pick($clients_all['other'], [])}
+} else {
+$clients = {}
 }
 
 $lockd_udp   = '32768'

-- 
To view, visit https://gerrit.wikimedia.org/r/400387
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Icd88705b91ba7d2e040876584d40a5ca53b736a5
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: ArielGlenn 
Gerrit-Reviewer: ArielGlenn 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: allow dumps nfs server to be configured without clients if n...

[ArielGlenn (Code Review)]

ArielGlenn has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/400387 )

Change subject: allow dumps nfs server to be configured without clients if 
needed
..

allow dumps nfs server to be configured without clients if needed

Change-Id: Icd88705b91ba7d2e040876584d40a5ca53b736a5
---
M modules/profile/manifests/dumps/nfs.pp
1 file changed, 2 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/87/400387/1

diff --git a/modules/profile/manifests/dumps/nfs.pp 
b/modules/profile/manifests/dumps/nfs.pp
index bc8..8965a46 100644
--- a/modules/profile/manifests/dumps/nfs.pp
+++ b/modules/profile/manifests/dumps/nfs.pp
@@ -11,6 +11,8 @@
 $clients = {'generation' => pick($clients_all['snapshots'], [])}
 } elsif ($clients_wanted == 'public') {
 $clients = {'public' => pick($clients_all['other'], [])}
+} else {
+$clients = {}
 }
 
 $lockd_udp   = '32768'

-- 
To view, visit https://gerrit.wikimedia.org/r/400387
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Icd88705b91ba7d2e040876584d40a5ca53b736a5
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: ArielGlenn 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: don't export dumps web server filesystems to snapshots, they...

[ArielGlenn (Code Review)]

ArielGlenn has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/400386 )

Change subject: don't export dumps web server filesystems to snapshots, they 
don't use it
..


don't export dumps web server filesystems to snapshots, they don't use it

Change-Id: I53fa364b414fe17fd86362b8cd6caebc7bdbfe5d
---
M hieradata/hosts/dataset1001.yaml
M hieradata/hosts/ms1001.yaml
2 files changed, 2 insertions(+), 2 deletions(-)

Approvals:
  ArielGlenn: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/hieradata/hosts/dataset1001.yaml b/hieradata/hosts/dataset1001.yaml
index 06d47e5..7977066 100644
--- a/hieradata/hosts/dataset1001.yaml
+++ b/hieradata/hosts/dataset1001.yaml
@@ -5,7 +5,7 @@
 profile::dumps::xmldumpspublicdir: '/data/xmldatadumps/public'
 profile::dumps::dumpstempdir: '/data/xmldatadumps/temp'
 
-profile::dumps::nfs::clients_wanted: 'all'
+profile::dumps::nfs::clients_wanted: 'public'
 
 profile::dumps::rsyncer:
   dumps_user: 'dumpsgen'
diff --git a/hieradata/hosts/ms1001.yaml b/hieradata/hosts/ms1001.yaml
index aeb0581..49305e5 100644
--- a/hieradata/hosts/ms1001.yaml
+++ b/hieradata/hosts/ms1001.yaml
@@ -6,7 +6,7 @@
 profile::dumps::xmldumpspublicdir: '/data/xmldatadumps/public'
 profile::dumps::dumpstempdir: '/data/xmldatadumps/temp'
 
-profile::dumps::nfs::clients_wanted: 'all'
+profile::dumps::nfs::clients_wanted: 'public'
 
 profile::dumps::rsyncer:
   dumps_user: 'dumpsgen'

-- 
To view, visit https://gerrit.wikimedia.org/r/400386
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I53fa364b414fe17fd86362b8cd6caebc7bdbfe5d
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: ArielGlenn 
Gerrit-Reviewer: ArielGlenn 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: don't export dumps web server filesystems to snapshots, they...

[ArielGlenn (Code Review)]

ArielGlenn has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/400386 )

Change subject: don't export dumps web server filesystems to snapshots, they 
don't use it
..

don't export dumps web server filesystems to snapshots, they don't use it

Change-Id: I53fa364b414fe17fd86362b8cd6caebc7bdbfe5d
---
M hieradata/hosts/dataset1001.yaml
M hieradata/hosts/ms1001.yaml
2 files changed, 2 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/86/400386/1

diff --git a/hieradata/hosts/dataset1001.yaml b/hieradata/hosts/dataset1001.yaml
index 06d47e5..7977066 100644
--- a/hieradata/hosts/dataset1001.yaml
+++ b/hieradata/hosts/dataset1001.yaml
@@ -5,7 +5,7 @@
 profile::dumps::xmldumpspublicdir: '/data/xmldatadumps/public'
 profile::dumps::dumpstempdir: '/data/xmldatadumps/temp'
 
-profile::dumps::nfs::clients_wanted: 'all'
+profile::dumps::nfs::clients_wanted: 'public'
 
 profile::dumps::rsyncer:
   dumps_user: 'dumpsgen'
diff --git a/hieradata/hosts/ms1001.yaml b/hieradata/hosts/ms1001.yaml
index aeb0581..49305e5 100644
--- a/hieradata/hosts/ms1001.yaml
+++ b/hieradata/hosts/ms1001.yaml
@@ -6,7 +6,7 @@
 profile::dumps::xmldumpspublicdir: '/data/xmldatadumps/public'
 profile::dumps::dumpstempdir: '/data/xmldatadumps/temp'
 
-profile::dumps::nfs::clients_wanted: 'all'
+profile::dumps::nfs::clients_wanted: 'public'
 
 profile::dumps::rsyncer:
   dumps_user: 'dumpsgen'

-- 
To view, visit https://gerrit.wikimedia.org/r/400386
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I53fa364b414fe17fd86362b8cd6caebc7bdbfe5d
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: ArielGlenn 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: move ferm rules for nfs out from dumps module to a profile

[ArielGlenn (Code Review)]

ArielGlenn has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/400244 )

Change subject: move ferm rules for nfs out from dumps module to a profile
..


move ferm rules for nfs out from dumps module to a profile

Change-Id: I68c06f7a0e52713a2b4a5c3916f1e5694d2f05bd
---
M hieradata/hosts/dataset1001.yaml
M hieradata/hosts/dumpsdata1001.yaml
M hieradata/hosts/dumpsdata1002.yaml
M hieradata/hosts/ms1001.yaml
M modules/dumps/manifests/nfs.pp
A modules/profile/manifests/dumps/nfs.pp
D modules/profile/manifests/dumps/nfs/all.pp
D modules/profile/manifests/dumps/nfs/generation.pp
D modules/profile/manifests/dumps/nfs/public.pp
M modules/role/manifests/dumps/generation/server/fallback.pp
M modules/role/manifests/dumps/generation/server/primary.pp
M modules/role/manifests/dumps/web/xmldumps_active.pp
M modules/role/manifests/dumps/web/xmldumps_fallback.pp
13 files changed, 93 insertions(+), 122 deletions(-)

Approvals:
  ArielGlenn: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/hieradata/hosts/dataset1001.yaml b/hieradata/hosts/dataset1001.yaml
index 7bd4d09..06d47e5 100644
--- a/hieradata/hosts/dataset1001.yaml
+++ b/hieradata/hosts/dataset1001.yaml
@@ -5,6 +5,8 @@
 profile::dumps::xmldumpspublicdir: '/data/xmldatadumps/public'
 profile::dumps::dumpstempdir: '/data/xmldatadumps/temp'
 
+profile::dumps::nfs::clients_wanted: 'all'
+
 profile::dumps::rsyncer:
   dumps_user: 'dumpsgen'
   dumps_group: 'dumpsgen'
diff --git a/hieradata/hosts/dumpsdata1001.yaml 
b/hieradata/hosts/dumpsdata1001.yaml
index f18774a..622f655 100644
--- a/hieradata/hosts/dumpsdata1001.yaml
+++ b/hieradata/hosts/dumpsdata1001.yaml
@@ -5,3 +5,5 @@
 
 profile::dumps::cleanup::isreplica: false
 profile::dumps::cleanup::labscopy: false
+
+profile::dumps::nfs::clients_wanted: 'generation'
diff --git a/hieradata/hosts/dumpsdata1002.yaml 
b/hieradata/hosts/dumpsdata1002.yaml
index 717b3d0..7ba77da 100644
--- a/hieradata/hosts/dumpsdata1002.yaml
+++ b/hieradata/hosts/dumpsdata1002.yaml
@@ -9,6 +9,8 @@
 profile::dumps::cleanup::isreplica: false
 profile::dumps::cleanup::labscopy: false
 
+profile::dumps::nfs::clients_wanted: 'generation'
+
 profile::dumps::rsyncer:
   dumps_user: 'dumpsgen'
   dumps_group: 'dumpsgen'
diff --git a/hieradata/hosts/ms1001.yaml b/hieradata/hosts/ms1001.yaml
index 1781b14..aeb0581 100644
--- a/hieradata/hosts/ms1001.yaml
+++ b/hieradata/hosts/ms1001.yaml
@@ -6,6 +6,8 @@
 profile::dumps::xmldumpspublicdir: '/data/xmldatadumps/public'
 profile::dumps::dumpstempdir: '/data/xmldatadumps/temp'
 
+profile::dumps::nfs::clients_wanted: 'all'
+
 profile::dumps::rsyncer:
   dumps_user: 'dumpsgen'
   dumps_group: 'dumpsgen'
diff --git a/modules/dumps/manifests/nfs.pp b/modules/dumps/manifests/nfs.pp
index 66770aa..9be6b46 100644
--- a/modules/dumps/manifests/nfs.pp
+++ b/modules/dumps/manifests/nfs.pp
@@ -1,12 +1,11 @@
 class dumps::nfs(
 $clients = undef,
-$statd_port = undef,
-$statd_out = undef,
+$path = undef,
 $lockd_udp = undef,
 $lockd_tcp = undef,
 $mountd_port = undef,
-$path = undef,
-$portmapper_port = undef,
+$statd_port = undef,
+$statd_out = undef,
 ) {
 file { '/etc/exports':
 mode=> '0444',
@@ -45,50 +44,6 @@
 
 kmod::options { 'lockd':
 options => "nlm_udpport=${lockd_udp} nlm_tcpport=${lockd_tcp}",
-}
-
-include ::network::constants
-
-ferm::service { 'dumps_nfs':
-proto  => 'tcp',
-port   => '2049',
-srange => '$PRODUCTION_NETWORKS',
-}
-
-ferm::service { 'nfs_rpc_mountd':
-proto  => 'tcp',
-port   => $mountd_port,
-srange => '$PRODUCTION_NETWORKS',
-}
-
-ferm::service { 'nfs_rpc_statd':
-proto  => 'tcp',
-port   => $statd_port,
-srange => '$PRODUCTION_NETWORKS',
-}
-
-ferm::service { 'nfs_portmapper_udp':
-proto  => 'udp',
-port   => $portmapper_port,
-srange => '$PRODUCTION_NETWORKS',
-}
-
-ferm::service { 'nfs_portmapper_tcp':
-proto  => 'tcp',
-port   => $portmapper_port,
-srange => '$PRODUCTION_NETWORKS',
-}
-
-ferm::service { 'nfs_lockd_udp':
-proto  => 'udp',
-port   => $lockd_udp,
-srange => '$PRODUCTION_NETWORKS',
-}
-
-ferm::service { 'nfs_lockd_tcp':
-proto  => 'tcp',
-port   => $lockd_tcp,
-srange => '$PRODUCTION_NETWORKS',
 }
 
 monitoring::service { 'nfs':
diff --git a/modules/profile/manifests/dumps/nfs.pp 
b/modules/profile/manifests/dumps/nfs.pp
new file mode 100644
index 000..bc8
--- /dev/null
+++ b/modules/profile/manifests/dumps/nfs.pp
@@ -0,0 +1,78 @@
+class profile::dumps::nfs(
+$clients_all = hiera('dumps_nfs_clients'),
+$clients_wanted = hiera('profile::dumps::nfs::clients_wanted'),
+) {
+$path= '/data'
+
+if ($clients_wanted == 'all') {
+

[MediaWiki-commits] [Gerrit] operations/puppet[production]: move ferm rules for nfs out from dumps module to a profile

[ArielGlenn (Code Review)]

ArielGlenn has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/400244 )

Change subject: move ferm rules for nfs out from dumps module to a profile
..

move ferm rules for nfs out from dumps module to a profile

Change-Id: I68c06f7a0e52713a2b4a5c3916f1e5694d2f05bd
---
M modules/dumps/manifests/nfs.pp
M modules/profile/manifests/dumps/nfs/all.pp
A modules/profile/manifests/dumps/nfs/ferm.pp
M modules/profile/manifests/dumps/nfs/generation.pp
M modules/profile/manifests/dumps/nfs/public.pp
5 files changed, 59 insertions(+), 87 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/44/400244/1

diff --git a/modules/dumps/manifests/nfs.pp b/modules/dumps/manifests/nfs.pp
index 66770aa..6a98ce3 100644
--- a/modules/dumps/manifests/nfs.pp
+++ b/modules/dumps/manifests/nfs.pp
@@ -1,12 +1,6 @@
 class dumps::nfs(
 $clients = undef,
-$statd_port = undef,
-$statd_out = undef,
-$lockd_udp = undef,
-$lockd_tcp = undef,
-$mountd_port = undef,
 $path = undef,
-$portmapper_port = undef,
 ) {
 file { '/etc/exports':
 mode=> '0444',
@@ -44,51 +38,7 @@
 }
 
 kmod::options { 'lockd':
-options => "nlm_udpport=${lockd_udp} nlm_tcpport=${lockd_tcp}",
-}
-
-include ::network::constants
-
-ferm::service { 'dumps_nfs':
-proto  => 'tcp',
-port   => '2049',
-srange => '$PRODUCTION_NETWORKS',
-}
-
-ferm::service { 'nfs_rpc_mountd':
-proto  => 'tcp',
-port   => $mountd_port,
-srange => '$PRODUCTION_NETWORKS',
-}
-
-ferm::service { 'nfs_rpc_statd':
-proto  => 'tcp',
-port   => $statd_port,
-srange => '$PRODUCTION_NETWORKS',
-}
-
-ferm::service { 'nfs_portmapper_udp':
-proto  => 'udp',
-port   => $portmapper_port,
-srange => '$PRODUCTION_NETWORKS',
-}
-
-ferm::service { 'nfs_portmapper_tcp':
-proto  => 'tcp',
-port   => $portmapper_port,
-srange => '$PRODUCTION_NETWORKS',
-}
-
-ferm::service { 'nfs_lockd_udp':
-proto  => 'udp',
-port   => $lockd_udp,
-srange => '$PRODUCTION_NETWORKS',
-}
-
-ferm::service { 'nfs_lockd_tcp':
-proto  => 'tcp',
-port   => $lockd_tcp,
-srange => '$PRODUCTION_NETWORKS',
+options => "nlm_udpport=32768 nlm_tcpport=32769",
 }
 
 monitoring::service { 'nfs':
diff --git a/modules/profile/manifests/dumps/nfs/all.pp 
b/modules/profile/manifests/dumps/nfs/all.pp
index 8cfb2e1..aeadcfb 100644
--- a/modules/profile/manifests/dumps/nfs/all.pp
+++ b/modules/profile/manifests/dumps/nfs/all.pp
@@ -1,24 +1,14 @@
 class profile::dumps::nfs::all(
 $clients_all = hiera('dumps_nfs_clients'),
 ) {
-$mountd_port = '32767'
-$statd_port  = '32765'
-$statd_out   = '32766'
-$portmapper_port = '111'
-$lockd_udp   = '32768'
-$lockd_tcp   = '32769'
+require ::profile::dumps::nfs::ferm
+
 $path= '/data'
 $clients = {'generation' => pick($clients_all['snapshots'], []),
 'public' => pick($clients_all['other'], [])}
 
 class { '::dumps::nfs':
 clients => $clients,
-statd_port  => $statd_port,
-statd_out   => $statd_out,
-lockd_udp   => $lockd_udp,
-lockd_tcp   => $lockd_tcp,
-mountd_port => $mountd_port,
-portmapper_port => $portmapper_port,
 path=> $path,
 }
 }
diff --git a/modules/profile/manifests/dumps/nfs/ferm.pp 
b/modules/profile/manifests/dumps/nfs/ferm.pp
new file mode 100644
index 000..cf0cf21
--- /dev/null
+++ b/modules/profile/manifests/dumps/nfs/ferm.pp
@@ -0,0 +1,52 @@
+class profile::dumps::nfs::ferm {
+include ::network::constants
+
+$mountd_port = '32767'
+$statd_port  = '32765'
+$statd_out   = '32766'
+$portmapper_port = '111'
+
+ferm::service { 'dumps_nfs':
+proto  => 'tcp',
+port   => '2049',
+srange => '$PRODUCTION_NETWORKS',
+}
+
+ferm::service { 'nfs_rpc_mountd':
+proto  => 'tcp',
+port   => $mountd_port,
+srange => '$PRODUCTION_NETWORKS',
+}
+
+ferm::service { 'nfs_rpc_statd':
+proto  => 'tcp',
+port   => $statd_port,
+srange => '$PRODUCTION_NETWORKS',
+}
+
+ferm::service { 'nfs_portmapper_udp':
+proto  => 'udp',
+port   => $portmapper_port,
+srange => '$PRODUCTION_NETWORKS',
+}
+
+ferm::service { 'nfs_portmapper_tcp':
+proto  => 'tcp',
+port   => $portmapper_port,
+srange => '$PRODUCTION_NETWORKS',
+}
+
+ferm::service { 'nfs_lockd_udp':
+proto  => 'udp',
+port   => '32768',
+srange => '$PRODUCTION_NETWORKS',
+}
+
+ferm::service { 'nfs_lockd_tcp':
+  

[MediaWiki-commits] [Gerrit] operations/puppet[production]: add dumps repo source to beta scap, add snapshot to beta mw ...

[ArielGlenn (Code Review)]

ArielGlenn has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/400237 )

Change subject: add dumps repo source to beta scap, add snapshot to beta mw scap
..

add dumps repo source to beta scap, add snapshot to beta mw scap

Change-Id: I70e254246cbc8485c9ed572475a8c9f3f4442e26
---
M hieradata/labs/deployment-prep/common.yaml
1 file changed, 9 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/37/400237/1

diff --git a/hieradata/labs/deployment-prep/common.yaml 
b/hieradata/labs/deployment-prep/common.yaml
index f2c54ce..10116a0 100644
--- a/hieradata/labs/deployment-prep/common.yaml
+++ b/hieradata/labs/deployment-prep/common.yaml
@@ -198,6 +198,7 @@
 - deployment-tin.deployment-prep.eqiad.wmflabs
 - deployment-mira.deployment-prep.eqiad.wmflabs
 - deployment-videoscaler01.deployment-prep.eqiad.wmflabs
+- deployment-snapshot01.deployment-prep.eqiad.wmflabs
 mediawiki-appserver-canaries:
 hosts:
 - deployment-mediawiki04.deployment-prep.eqiad.wmflabs
@@ -284,6 +285,10 @@
 trusted_groups:
   - deploy-service
 
+  dumpsdeploy:
+trusted_groups:
+  - project-%{::labsproject}
+
 # deployment-prep scap::source declarations.  These are created
 # by the role deployment::server.  Each source listed here
 # will be cloned on the scap deploy server.
@@ -343,6 +348,10 @@
   scholarships/scholarships:
 repository: wikimedia/wikimania-scholarships
 
+  # Xml/sql dumps
+  dumps/dumps:
+repository: operations/dumps
+
 prometheus_nodes:
   - deployment-prometheus01.deployment-prep.eqiad.wmflabs
 profile::recommendation_api::wdqs_uri: http://wdqs-test.wmflabs.org

-- 
To view, visit https://gerrit.wikimedia.org/r/400237
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I70e254246cbc8485c9ed572475a8c9f3f4442e26
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: ArielGlenn 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: manage directories locally on dumps workers if there is no n...

[ArielGlenn (Code Review)]

ArielGlenn has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/400233 )

Change subject: manage directories locally on dumps workers if there is no nfs 
server
..


manage directories locally on dumps workers if there is no nfs server

Change-Id: I7949196fdaacb71633206019119ff734a5626cfb
---
M hieradata/common.yaml
M modules/profile/manifests/dumps/generation/worker/common.pp
M modules/snapshot/manifests/dumps/nfsmount.pp
3 files changed, 21 insertions(+), 3 deletions(-)

Approvals:
  ArielGlenn: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/hieradata/common.yaml b/hieradata/common.yaml
index 821bdc2..2cfc1b1 100644
--- a/hieradata/common.yaml
+++ b/hieradata/common.yaml
@@ -351,6 +351,7 @@
   - odysseus.ip6.fi.muni.cz
   - poincare.acc.umu.se
 dumps_nfs_server: dumpsdata1001.eqiad.wmnet
+dumps_managed_subdirs: []
 
 # Schemas names that match this regex
 # will not be produced to the eventlogging-valid-mixed
diff --git a/modules/profile/manifests/dumps/generation/worker/common.pp 
b/modules/profile/manifests/dumps/generation/worker/common.pp
index 11855e1..940a017 100644
--- a/modules/profile/manifests/dumps/generation/worker/common.pp
+++ b/modules/profile/manifests/dumps/generation/worker/common.pp
@@ -1,5 +1,6 @@
 class profile::dumps::generation::worker::common(
-$nfs_server = hiera('dumps_nfs_server')
+$nfs_server = hiera('dumps_nfs_server'),
+$managed_subdirs = hiera('dumps_managed_subdirs'),
 ) {
 # mw packages and dependencies
 require ::profile::mediawiki::scap_proxy
@@ -11,8 +12,11 @@
 class { '::dumpsuser': }
 
 snapshot::dumps::nfsmount { 'dumpsdatamount':
-mountpoint => $xmldumpsmount,
-server => $nfs_server,
+mountpoint  => $xmldumpsmount,
+server  => $nfs_server,
+managed_subdirs => $managed_subdirs,
+user=> 'dumpsgen',
+group   => 'dumpsgen',
 }
 # dataset server config files,
 # stages files, dblists, html templates
diff --git a/modules/snapshot/manifests/dumps/nfsmount.pp 
b/modules/snapshot/manifests/dumps/nfsmount.pp
index bbb1765..31f2027 100644
--- a/modules/snapshot/manifests/dumps/nfsmount.pp
+++ b/modules/snapshot/manifests/dumps/nfsmount.pp
@@ -1,6 +1,9 @@
 define snapshot::dumps::nfsmount(
 $mountpoint = undef,
 $server = undef,
+$managed_subdirs = [],
+$user = undef,
+$group = undef,
 ) {
 require_package('nfs-common')
 
@@ -19,4 +22,14 @@
 remounts => false,
 }
 }
+else {
+# manage some directories that the nfs server
+# server would otherwise take care of for us
+file { $managed_subdirs:
+ensure => 'directory',
+mode   => '0755',
+owner  => $user,
+group  => $group,
+}
+}
 }

-- 
To view, visit https://gerrit.wikimedia.org/r/400233
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I7949196fdaacb71633206019119ff734a5626cfb
Gerrit-PatchSet: 5
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: ArielGlenn 
Gerrit-Reviewer: ArielGlenn 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: manage directories locally on dumps workers if there is no n...

[ArielGlenn (Code Review)]

ArielGlenn has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/400233 )

Change subject: manage directories locally on dumps workers if there is no nfs 
server
..

manage directories locally on dumps workers if there is no nfs server

Change-Id: I7949196fdaacb71633206019119ff734a5626cfb
---
M hieradata/common.yaml
M modules/profile/manifests/dumps/generation/worker/common.pp
M modules/snapshot/manifests/dumps/nfsmount.pp
3 files changed, 20 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/33/400233/1

diff --git a/hieradata/common.yaml b/hieradata/common.yaml
index 821bdc2..2cfc1b1 100644
--- a/hieradata/common.yaml
+++ b/hieradata/common.yaml
@@ -351,6 +351,7 @@
   - odysseus.ip6.fi.muni.cz
   - poincare.acc.umu.se
 dumps_nfs_server: dumpsdata1001.eqiad.wmnet
+dumps_managed_subdirs: []
 
 # Schemas names that match this regex
 # will not be produced to the eventlogging-valid-mixed
diff --git a/modules/profile/manifests/dumps/generation/worker/common.pp 
b/modules/profile/manifests/dumps/generation/worker/common.pp
index 11855e1..8dd1ae6 100644
--- a/modules/profile/manifests/dumps/generation/worker/common.pp
+++ b/modules/profile/manifests/dumps/generation/worker/common.pp
@@ -1,5 +1,6 @@
 class profile::dumps::generation::worker::common(
 $nfs_server = hiera('dumps_nfs_server')
+$managed_dumpsdirs = hiera('dumps_managed_subdirs')
 ) {
 # mw packages and dependencies
 require ::profile::mediawiki::scap_proxy
@@ -11,8 +12,11 @@
 class { '::dumpsuser': }
 
 snapshot::dumps::nfsmount { 'dumpsdatamount':
-mountpoint => $xmldumpsmount,
-server => $nfs_server,
+mountpoint  => $xmldumpsmount,
+server  => $nfs_server,
+managed_subdirs => $dumps_managed_subdirs,
+user=> 'dumpsgen'
+group   => 'dumpsgen',
 }
 # dataset server config files,
 # stages files, dblists, html templates
diff --git a/modules/snapshot/manifests/dumps/nfsmount.pp 
b/modules/snapshot/manifests/dumps/nfsmount.pp
index bbb1765..b15a9ab 100644
--- a/modules/snapshot/manifests/dumps/nfsmount.pp
+++ b/modules/snapshot/manifests/dumps/nfsmount.pp
@@ -1,6 +1,9 @@
 define snapshot::dumps::nfsmount(
 $mountpoint = undef,
 $server = undef,
+$managed_dumps_subdirs = [];
+$user = undef,
+$group = undef,
 ) {
 require_package('nfs-common')
 
@@ -19,4 +22,14 @@
 remounts => false,
 }
 }
+else {
+# manage some directories that the nfs server
+# server would otherwise take care of for us
+file { $managed_dumps_subdirs:
+ensure => 'directory',
+mode   => '0755',
+owner  => $user,
+group  => $group,
+}
+}
 }

-- 
To view, visit https://gerrit.wikimedia.org/r/400233
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I7949196fdaacb71633206019119ff734a5626cfb
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: ArielGlenn 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: move dumps nfs server name into hiera param for worker profile

[ArielGlenn (Code Review)]

ArielGlenn has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/400231 )

Change subject: move dumps nfs server name into hiera param for worker profile
..


move dumps nfs server name into hiera param for worker profile

also permit configurations with no external nfs server
(useful in deployment-prep)

Change-Id: I36b0c9b08ae0d8e0924e07dcb4a2dcf68867389f
---
M hieradata/common.yaml
M modules/profile/manifests/dumps/generation/worker/common.pp
M modules/snapshot/manifests/dumps/nfsmount.pp
3 files changed, 13 insertions(+), 9 deletions(-)

Approvals:
  ArielGlenn: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/hieradata/common.yaml b/hieradata/common.yaml
index f55e552..821bdc2 100644
--- a/hieradata/common.yaml
+++ b/hieradata/common.yaml
@@ -350,6 +350,7 @@
 external:
   - odysseus.ip6.fi.muni.cz
   - poincare.acc.umu.se
+dumps_nfs_server: dumpsdata1001.eqiad.wmnet
 
 # Schemas names that match this regex
 # will not be produced to the eventlogging-valid-mixed
diff --git a/modules/profile/manifests/dumps/generation/worker/common.pp 
b/modules/profile/manifests/dumps/generation/worker/common.pp
index f26eb30..11855e1 100644
--- a/modules/profile/manifests/dumps/generation/worker/common.pp
+++ b/modules/profile/manifests/dumps/generation/worker/common.pp
@@ -1,4 +1,5 @@
 class profile::dumps::generation::worker::common(
+$nfs_server = hiera('dumps_nfs_server')
 ) {
 # mw packages and dependencies
 require ::profile::mediawiki::scap_proxy
@@ -11,7 +12,7 @@
 
 snapshot::dumps::nfsmount { 'dumpsdatamount':
 mountpoint => $xmldumpsmount,
-server => 'dumpsdata1001.eqiad.wmnet',
+server => $nfs_server,
 }
 # dataset server config files,
 # stages files, dblists, html templates
diff --git a/modules/snapshot/manifests/dumps/nfsmount.pp 
b/modules/snapshot/manifests/dumps/nfsmount.pp
index 94f76a8..bbb1765 100644
--- a/modules/snapshot/manifests/dumps/nfsmount.pp
+++ b/modules/snapshot/manifests/dumps/nfsmount.pp
@@ -8,13 +8,15 @@
 ensure => 'directory',
 }
 
-mount { $mountpoint:
-ensure   => 'mounted',
-device   => "${server}:/data",
-fstype   => 'nfs',
-name => $mountpoint,
-options  => 'bg,hard,tcp,rsize=8192,wsize=8192,intr,nfsvers=3',
-require  => File[$mountpoint],
-remounts => false,
+if ($server != undef) {
+mount { $mountpoint:
+ensure   => 'mounted',
+device   => "${server}:/data",
+fstype   => 'nfs',
+name => $mountpoint,
+options  => 'bg,hard,tcp,rsize=8192,wsize=8192,intr,nfsvers=3',
+require  => File[$mountpoint],
+remounts => false,
+}
 }
 }

-- 
To view, visit https://gerrit.wikimedia.org/r/400231
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I36b0c9b08ae0d8e0924e07dcb4a2dcf68867389f
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: ArielGlenn 
Gerrit-Reviewer: ArielGlenn 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: move dumps nfs server name into hiera param for worker profile

[ArielGlenn (Code Review)]

ArielGlenn has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/400231 )

Change subject: move dumps nfs server name into hiera param for worker profile
..

move dumps nfs server name into hiera param for worker profile

also permit configurations with no external nfs server
(useful in deployment-prep)

Change-Id: I36b0c9b08ae0d8e0924e07dcb4a2dcf68867389f
---
M hieradata/common.yaml
M modules/profile/manifests/dumps/generation/worker/common.pp
M modules/snapshot/manifests/dumps/nfsmount.pp
3 files changed, 13 insertions(+), 9 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/31/400231/1

diff --git a/hieradata/common.yaml b/hieradata/common.yaml
index f55e552..255de2f 100644
--- a/hieradata/common.yaml
+++ b/hieradata/common.yaml
@@ -350,6 +350,7 @@
 external:
   - odysseus.ip6.fi.muni.cz
   - poincare.acc.umu.se
+dumps:nfs_server: dumpsdata1001.eqiad.wmnet
 
 # Schemas names that match this regex
 # will not be produced to the eventlogging-valid-mixed
diff --git a/modules/profile/manifests/dumps/generation/worker/common.pp 
b/modules/profile/manifests/dumps/generation/worker/common.pp
index f26eb30..dbb09cd 100644
--- a/modules/profile/manifests/dumps/generation/worker/common.pp
+++ b/modules/profile/manifests/dumps/generation/worker/common.pp
@@ -1,4 +1,5 @@
 class profile::dumps::generation::worker::common(
+$nfs_server = hiera('dumps::nfs_server')
 ) {
 # mw packages and dependencies
 require ::profile::mediawiki::scap_proxy
@@ -11,7 +12,7 @@
 
 snapshot::dumps::nfsmount { 'dumpsdatamount':
 mountpoint => $xmldumpsmount,
-server => 'dumpsdata1001.eqiad.wmnet',
+server => $nfs_server,
 }
 # dataset server config files,
 # stages files, dblists, html templates
diff --git a/modules/snapshot/manifests/dumps/nfsmount.pp 
b/modules/snapshot/manifests/dumps/nfsmount.pp
index 94f76a8..bbb1765 100644
--- a/modules/snapshot/manifests/dumps/nfsmount.pp
+++ b/modules/snapshot/manifests/dumps/nfsmount.pp
@@ -8,13 +8,15 @@
 ensure => 'directory',
 }
 
-mount { $mountpoint:
-ensure   => 'mounted',
-device   => "${server}:/data",
-fstype   => 'nfs',
-name => $mountpoint,
-options  => 'bg,hard,tcp,rsize=8192,wsize=8192,intr,nfsvers=3',
-require  => File[$mountpoint],
-remounts => false,
+if ($server != undef) {
+mount { $mountpoint:
+ensure   => 'mounted',
+device   => "${server}:/data",
+fstype   => 'nfs',
+name => $mountpoint,
+options  => 'bg,hard,tcp,rsize=8192,wsize=8192,intr,nfsvers=3',
+require  => File[$mountpoint],
+remounts => false,
+}
 }
 }

-- 
To view, visit https://gerrit.wikimedia.org/r/400231
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I36b0c9b08ae0d8e0924e07dcb4a2dcf68867389f
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: ArielGlenn 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/dumps[master]: make reporting of file sizes for dump steps in progress work...

[ArielGlenn (Code Review)]

ArielGlenn has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/400226 )

Change subject: make reporting of file sizes for dump steps in progress work 
again
..


make reporting of file sizes for dump steps in progress work again

This was probably broken when we introduced writing files with
a special extension and then moving them into place once the
run completes successfully.

Bug: T183694
Change-Id: Ib4bc9f6a41f31431e5642a3f7f7415bd2de38ea8
---
M xmldumps-backup/dumps/apijobs.py
M xmldumps-backup/dumps/fileutils.py
M xmldumps-backup/dumps/flowjob.py
M xmldumps-backup/dumps/jobs.py
M xmldumps-backup/dumps/recombinejobs.py
M xmldumps-backup/dumps/recompressjobs.py
M xmldumps-backup/dumps/runnerutils.py
M xmldumps-backup/dumps/tablesjobs.py
M xmldumps-backup/dumps/xmlcontentjobs.py
M xmldumps-backup/dumps/xmljobs.py
10 files changed, 51 insertions(+), 33 deletions(-)

Approvals:
  ArielGlenn: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/xmldumps-backup/dumps/apijobs.py b/xmldumps-backup/dumps/apijobs.py
index 3991add..534ca99 100644
--- a/xmldumps-backup/dumps/apijobs.py
+++ b/xmldumps-backup/dumps/apijobs.py
@@ -1,5 +1,6 @@
 import time
 from dumps.exceptions import BackupError
+from dumps.fileutils import DumpFilename
 from dumps.jobs import Dump
 
 
@@ -30,11 +31,11 @@
 commands = self.build_command(runner)
 if runner.wiki.is_private():
 command_series = runner.get_save_command_series(
-commands, self.get_inprogress_name(
+commands, DumpFilename.get_inprogress_name(
 runner.dump_dir.filename_private_path(output_dfname)))
 else:
 command_series = runner.get_save_command_series(
-commands, self.get_inprogress_name(
+commands, DumpFilename.get_inprogress_name(
 runner.dump_dir.filename_public_path(output_dfname)))
 self.setup_command_info(runner, command_series, [output_dfname])
 
diff --git a/xmldumps-backup/dumps/fileutils.py 
b/xmldumps-backup/dumps/fileutils.py
index a264ece..e6b4b67 100644
--- a/xmldumps-backup/dumps/fileutils.py
+++ b/xmldumps-backup/dumps/fileutils.py
@@ -165,6 +165,8 @@
 partnum_int   part number as int
 """
 
+INPROG = ".inprog"  # extension for dump output files that are in progress 
(not fully written)
+
 @staticmethod
 def make_checkpoint_string(first_page_id, last_page_id):
 if first_page_id is not None and last_page_id is not None:
@@ -172,6 +174,10 @@
 else:
 return None
 
+@staticmethod
+def get_inprogress_name(filename):
+return filename + DumpFilename.INPROG
+
 def __init__(self, wiki, date=None, dump_name=None, filetype=None,
  ext=None, partnum=None, checkpoint=None, temp=False):
 """Constructor.  Arguments: the dump name as it should appear in the 
filename,
diff --git a/xmldumps-backup/dumps/flowjob.py b/xmldumps-backup/dumps/flowjob.py
index b875101..f1de495 100644
--- a/xmldumps-backup/dumps/flowjob.py
+++ b/xmldumps-backup/dumps/flowjob.py
@@ -5,6 +5,7 @@
 import os
 from dumps.exceptions import BackupError
 from dumps.utils import MultiVersion
+from dumps.fileutils import DumpFilename
 from dumps.jobs import Dump
 
 
@@ -45,7 +46,7 @@
 command.extend(script_command)
 command.extend(["--wiki=%s" % runner.db_name,
 "--current", "--report=1000",
-"--output=bzip2:%s" % 
self.get_inprogress_name(flow_output_fpath)])
+"--output=bzip2:%s" % 
DumpFilename.get_inprogress_name(flow_output_fpath)])
 if self.history:
 command.append("--full")
 pipeline = [command]
diff --git a/xmldumps-backup/dumps/jobs.py b/xmldumps-backup/dumps/jobs.py
index 8ebbbf1..256056b 100644
--- a/xmldumps-backup/dumps/jobs.py
+++ b/xmldumps-backup/dumps/jobs.py
@@ -54,8 +54,6 @@
 
 
 class Dump(object):
-INPROG = ".inprog"  # extension for dump output files that are in progress 
(not fully written)
-
 def __init__(self, name, desc, verbose=False):
 self._desc = desc
 self.verbose = verbose
@@ -84,14 +82,12 @@
 if not hasattr(self, '_parts'):
 self._parts = False
 
-def get_inprogress_name(self, filename):
-return filename + self.INPROG
-
 def setup_command_info(self, runner, command_series, output_dfnames, 
output_dir=None):
 command_info = {}
 command_info['runner'] = runner
 command_info['series'] = command_series
-command_info['output_files'] = [dfname.filename + self.INPROG for 
dfname in output_dfnames]
+command_info['output_files'] = [dfname.filename + DumpFilename.INPROG
+for dfname in output_dfnames]
 if output_dir is not None:
 

[MediaWiki-commits] [Gerrit] operations/dumps[master]: make reporting of file sizes for dump steps in progress work...

[ArielGlenn (Code Review)]

ArielGlenn has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/400226 )

Change subject: make reporting of file sizes for dump steps in progress work 
again
..

make reporting of file sizes for dump steps in progress work again

This was probably broken when we introduced writing files with
a special extension and then moving them into place once the
run completes successfully.

Bug: T183694
Change-Id: Ib4bc9f6a41f31431e5642a3f7f7415bd2de38ea8
---
M xmldumps-backup/dumps/apijobs.py
M xmldumps-backup/dumps/fileutils.py
M xmldumps-backup/dumps/flowjob.py
M xmldumps-backup/dumps/jobs.py
M xmldumps-backup/dumps/recombinejobs.py
M xmldumps-backup/dumps/recompressjobs.py
M xmldumps-backup/dumps/runnerutils.py
M xmldumps-backup/dumps/tablesjobs.py
M xmldumps-backup/dumps/xmlcontentjobs.py
M xmldumps-backup/dumps/xmljobs.py
10 files changed, 51 insertions(+), 33 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/dumps 
refs/changes/26/400226/1

diff --git a/xmldumps-backup/dumps/apijobs.py b/xmldumps-backup/dumps/apijobs.py
index 3991add..534ca99 100644
--- a/xmldumps-backup/dumps/apijobs.py
+++ b/xmldumps-backup/dumps/apijobs.py
@@ -1,5 +1,6 @@
 import time
 from dumps.exceptions import BackupError
+from dumps.fileutils import DumpFilename
 from dumps.jobs import Dump
 
 
@@ -30,11 +31,11 @@
 commands = self.build_command(runner)
 if runner.wiki.is_private():
 command_series = runner.get_save_command_series(
-commands, self.get_inprogress_name(
+commands, DumpFilename.get_inprogress_name(
 runner.dump_dir.filename_private_path(output_dfname)))
 else:
 command_series = runner.get_save_command_series(
-commands, self.get_inprogress_name(
+commands, DumpFilename.get_inprogress_name(
 runner.dump_dir.filename_public_path(output_dfname)))
 self.setup_command_info(runner, command_series, [output_dfname])
 
diff --git a/xmldumps-backup/dumps/fileutils.py 
b/xmldumps-backup/dumps/fileutils.py
index a264ece..e6b4b67 100644
--- a/xmldumps-backup/dumps/fileutils.py
+++ b/xmldumps-backup/dumps/fileutils.py
@@ -165,6 +165,8 @@
 partnum_int   part number as int
 """
 
+INPROG = ".inprog"  # extension for dump output files that are in progress 
(not fully written)
+
 @staticmethod
 def make_checkpoint_string(first_page_id, last_page_id):
 if first_page_id is not None and last_page_id is not None:
@@ -172,6 +174,10 @@
 else:
 return None
 
+@staticmethod
+def get_inprogress_name(filename):
+return filename + DumpFilename.INPROG
+
 def __init__(self, wiki, date=None, dump_name=None, filetype=None,
  ext=None, partnum=None, checkpoint=None, temp=False):
 """Constructor.  Arguments: the dump name as it should appear in the 
filename,
diff --git a/xmldumps-backup/dumps/flowjob.py b/xmldumps-backup/dumps/flowjob.py
index b875101..f1de495 100644
--- a/xmldumps-backup/dumps/flowjob.py
+++ b/xmldumps-backup/dumps/flowjob.py
@@ -5,6 +5,7 @@
 import os
 from dumps.exceptions import BackupError
 from dumps.utils import MultiVersion
+from dumps.fileutils import DumpFilename
 from dumps.jobs import Dump
 
 
@@ -45,7 +46,7 @@
 command.extend(script_command)
 command.extend(["--wiki=%s" % runner.db_name,
 "--current", "--report=1000",
-"--output=bzip2:%s" % 
self.get_inprogress_name(flow_output_fpath)])
+"--output=bzip2:%s" % 
DumpFilename.get_inprogress_name(flow_output_fpath)])
 if self.history:
 command.append("--full")
 pipeline = [command]
diff --git a/xmldumps-backup/dumps/jobs.py b/xmldumps-backup/dumps/jobs.py
index 8ebbbf1..256056b 100644
--- a/xmldumps-backup/dumps/jobs.py
+++ b/xmldumps-backup/dumps/jobs.py
@@ -54,8 +54,6 @@
 
 
 class Dump(object):
-INPROG = ".inprog"  # extension for dump output files that are in progress 
(not fully written)
-
 def __init__(self, name, desc, verbose=False):
 self._desc = desc
 self.verbose = verbose
@@ -84,14 +82,12 @@
 if not hasattr(self, '_parts'):
 self._parts = False
 
-def get_inprogress_name(self, filename):
-return filename + self.INPROG
-
 def setup_command_info(self, runner, command_series, output_dfnames, 
output_dir=None):
 command_info = {}
 command_info['runner'] = runner
 command_info['series'] = command_series
-command_info['output_files'] = [dfname.filename + self.INPROG for 
dfname in output_dfnames]
+command_info['output_files'] = [dfname.filename + DumpFilename.INPROG
+for dfname in output_dfnames]
 if output_dir is not None:
 

[MediaWiki-commits] [Gerrit] mediawiki/core[master]: make 7zip wrapper usable for dumping text revisions again

[ArielGlenn (Code Review)]

ArielGlenn has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/399947 )

Change subject: make 7zip wrapper usable for dumping text revisions again
..

make 7zip wrapper usable for dumping text revisions again

Change-Id: Ic9c5ad284b542c15f4ab48b5459f00e188a4301a
---
M maintenance/dumpTextPass.php
1 file changed, 1 insertion(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/mediawiki/core 
refs/changes/47/399947/1

diff --git a/maintenance/dumpTextPass.php b/maintenance/dumpTextPass.php
index 2b79b54..0011088 100644
--- a/maintenance/dumpTextPass.php
+++ b/maintenance/dumpTextPass.php
@@ -25,6 +25,7 @@
  */
 
 require_once __DIR__ . '/backup.inc';
++require_once __DIR__ . '/7zip.inc';
 require_once __DIR__ . '/../includes/export/WikiExporter.php';
 
 use Wikimedia\Rdbms\IMaintainableDatabase;

-- 
To view, visit https://gerrit.wikimedia.org/r/399947
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ic9c5ad284b542c15f4ab48b5459f00e188a4301a
Gerrit-PatchSet: 1
Gerrit-Project: mediawiki/core
Gerrit-Branch: master
Gerrit-Owner: ArielGlenn 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: fix up one more time the xml misc dumps dir for rsyncs

[ArielGlenn (Code Review)]

ArielGlenn has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/399835 )

Change subject: fix up one more time the xml misc dumps dir for rsyncs
..


fix up one more time the xml misc dumps dir for rsyncs

Change-Id: I9e8e736d855c10350619a0dff86edfcb1e6c1c04
---
M modules/dumps/files/generation/rsync-to-peers.sh
1 file changed, 1 insertion(+), 1 deletion(-)

Approvals:
  ArielGlenn: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/dumps/files/generation/rsync-to-peers.sh 
b/modules/dumps/files/generation/rsync-to-peers.sh
index 49abbe9..3845197 100644
--- a/modules/dumps/files/generation/rsync-to-peers.sh
+++ b/modules/dumps/files/generation/rsync-to-peers.sh
@@ -113,7 +113,7 @@
 
 # rsync of xml/sql dumps for public wikis
 for dest in $xmlremotedirs_list; do
-/usr/bin/rsync -a  --contimeout=600 --timeout=600 --exclude='**bad/' 
--exclude='**save/' --exclude='**not/' --exclude='**temp/' --exclude='**tmp/' 
--exclude='*.inprog'  --exclude='*.html' --exclude='*.txt' --exclude='*.json' 
${xmldumpsdir}/public/*wik* "$dest" > /dev/null 2>&1
+/usr/bin/rsync -a  --contimeout=600 --timeout=600 --exclude='**bad/' 
--exclude='**save/' --exclude='**not/' --exclude='**temp/' --exclude='**tmp/' 
--exclude='*.inprog'  --exclude='*.html' --exclude='*.txt' --exclude='*.json' 
${xmldumpsdir}/*wik* "$dest" > /dev/null 2>&1
 
# send statusfiles tarball over last, remote can unpack it when it 
notices the arrival
# this way, content of status and html files always reflects dump 
output already

-- 
To view, visit https://gerrit.wikimedia.org/r/399835
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I9e8e736d855c10350619a0dff86edfcb1e6c1c04
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: ArielGlenn 
Gerrit-Reviewer: ArielGlenn 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: fix up one more time the xml misc dumps dir for rsyncs

[ArielGlenn (Code Review)]

ArielGlenn has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/399835 )

Change subject: fix up one more time the xml misc dumps dir for rsyncs
..

fix up one more time the xml misc dumps dir for rsyncs

Change-Id: I9e8e736d855c10350619a0dff86edfcb1e6c1c04
---
M modules/dumps/files/generation/rsync-to-peers.sh
1 file changed, 1 insertion(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/35/399835/1

diff --git a/modules/dumps/files/generation/rsync-to-peers.sh 
b/modules/dumps/files/generation/rsync-to-peers.sh
index 49abbe9..3845197 100644
--- a/modules/dumps/files/generation/rsync-to-peers.sh
+++ b/modules/dumps/files/generation/rsync-to-peers.sh
@@ -113,7 +113,7 @@
 
 # rsync of xml/sql dumps for public wikis
 for dest in $xmlremotedirs_list; do
-/usr/bin/rsync -a  --contimeout=600 --timeout=600 --exclude='**bad/' 
--exclude='**save/' --exclude='**not/' --exclude='**temp/' --exclude='**tmp/' 
--exclude='*.inprog'  --exclude='*.html' --exclude='*.txt' --exclude='*.json' 
${xmldumpsdir}/public/*wik* "$dest" > /dev/null 2>&1
+/usr/bin/rsync -a  --contimeout=600 --timeout=600 --exclude='**bad/' 
--exclude='**save/' --exclude='**not/' --exclude='**temp/' --exclude='**tmp/' 
--exclude='*.inprog'  --exclude='*.html' --exclude='*.txt' --exclude='*.json' 
${xmldumpsdir}/*wik* "$dest" > /dev/null 2>&1
 
# send statusfiles tarball over last, remote can unpack it when it 
notices the arrival
# this way, content of status and html files always reflects dump 
output already

-- 
To view, visit https://gerrit.wikimedia.org/r/399835
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I9e8e736d855c10350619a0dff86edfcb1e6c1c04
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: ArielGlenn 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/dumps[master]: permit use of 7zip compressed files for prefetch

[ArielGlenn (Code Review)]

ArielGlenn has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/399753 )

Change subject: permit use of 7zip compressed files for prefetch
..

permit use of 7zip compressed files for prefetch

[WIP] first draft, incomplete, standard disclaimers apply

Bug: T179267
Change-Id: I14d4636c78d81a9bfbf04f7f4c218875fcb870dc
---
M xmldumps-backup/defaults.conf
M xmldumps-backup/dumps/WikiDump.py
M xmldumps-backup/dumps/xmlcontentjobs.py
3 files changed, 73 insertions(+), 42 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/dumps 
refs/changes/53/399753/1

diff --git a/xmldumps-backup/defaults.conf b/xmldumps-backup/defaults.conf
index e95ac47..e0a2c65 100644
--- a/xmldumps-backup/defaults.conf
+++ b/xmldumps-backup/defaults.conf
@@ -78,4 +78,5 @@
 maxrevs=5
 
 [misc]
-fixeddumporder=0
\ No newline at end of file
+fixeddumporder=0
+sevenzipprefetch=0
diff --git a/xmldumps-backup/dumps/WikiDump.py 
b/xmldumps-backup/dumps/WikiDump.py
index 8ba3838..8fd276b 100644
--- a/xmldumps-backup/dumps/WikiDump.py
+++ b/xmldumps-backup/dumps/WikiDump.py
@@ -312,6 +312,11 @@
 self.conf.add_section('wiki')
 self.wiki_dir = self.get_opt_for_proj_or_default("wiki", "dir", 0)
 
+if not self.conf.has_section('misc'):
+self.conf.add_section('misc')
+self.sevenzip_prefetch = self.get_opt_in_overrides_or_default("misc", 
"sevenzipprefetch", 0)
+self.sevenzip_prefetch = int(self.sevenzipprefetch, 0)
+
 def db_latest_status(self):
 '''
 return list of tuples for each wiki:
diff --git a/xmldumps-backup/dumps/xmlcontentjobs.py 
b/xmldumps-backup/dumps/xmlcontentjobs.py
index b0db10d..bfcd229 100644
--- a/xmldumps-backup/dumps/xmlcontentjobs.py
+++ b/xmldumps-backup/dumps/xmlcontentjobs.py
@@ -198,6 +198,55 @@
 pagerange['end'] = None
 return pagerange
 
+def _find_prefetch_files_from_run(self, runner, date, jobinfo,
+  pagerange, file_ext):
+"""
+for a given wiki and date, see if there are dump content
+files lying about that can be used for prefetch to the
+current job, with the given file extension (might be bz2s
+or 7zs or whatever) for the given range of pages
+"""
+dfnames = get_checkpt_files(
+runner.dump_dir, [jobinfo['dumpname']], self.jobinfo['ftype'],
+file_ext, date, parts=None)
+possible_prefetch_dfnames = self.get_relevant_prefetch_dfnames(
+dfnames, pagerange, date, runner)
+if len(possible_prefetch_dfnames):
+return possible_prefetch_dfnames
+
+# ok, let's check for file parts instead, from any run
+# (may not conform to our numbering for this job)
+dfnames = get_reg_files(
+runner.dump_dir, [jobinfo['dumpname']], jobinfo['ftype'],
+file_ext, date, parts=True)
+possible_prefetch_dfnames = self.get_relevant_prefetch_dfnames(
+dfnames, pagerange, date, runner)
+if len(possible_prefetch_dfnames):
+return possible_prefetch_dfnames
+
+# last shot, get output file that contains all the pages, if there is 
one
+dfnames = get_reg_files(
+runner.dump_dir, [jobinfo['dumpname']],
+jobinfo['ftype'], file_ext, date, parts=False)
+# there is only one, don't bother to check for relevance :-P
+possible_prefetch_dfnames = dfnames
+dfnames = []
+for prefetch_dfname in possible_prefetch_dfnames:
+if runner.wiki.is_private():
+possible_path = 
runner.dump_dir.filename_private_path(prefetch_dfname, date)
+else:
+possible_path = 
runner.dump_dir.filename_public_path(prefetch_dfname, date)
+size = os.path.getsize(possible_path)
+if size < 7:
+runner.debug("small %d-byte prefetch dump at %s, skipping" % (
+size, possible_path))
+continue
+else:
+dfnames.append(prefetch_dfname)
+if len(dfnames):
+return dfnames
+return None
+
 def _find_previous_dump(self, runner, partnum=None):
 """
 this finds the content file or files from the first previous 
successful dump
@@ -226,46 +275,15 @@
 runner.debug("skipping incomplete or failed dump for prefetch 
date %s" % date)
 continue
 
-# first check if there are checkpoint files from this run we can 
use
-dfnames = get_checkpt_files(
-runner.dump_dir, [self.jobinfo['dumpname']], 
self.jobinfo['ftype'],
-self.jobinfo['fext'], date, parts=None)
-possible_prefetch_dfnames = self.get_relevant_prefetch_dfnames(
-dfnames, pagerange, date, runner)
-if 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: enable pagelogs to be dumped by several processes in parallel

[ArielGlenn (Code Review)]

ArielGlenn has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/399589 )

Change subject: enable pagelogs to be dumped by several processes in parallel
..

enable pagelogs to be dumped by several processes in parallel

Bug: T181935
Change-Id: Ib89de0b72aaa0fe73534bb318b54efcd50414192
---
M modules/snapshot/manifests/dumps/configs.pp
1 file changed, 6 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/89/399589/1

diff --git a/modules/snapshot/manifests/dumps/configs.pp 
b/modules/snapshot/manifests/dumps/configs.pp
index 36ce427..72c128f 100644
--- a/modules/snapshot/manifests/dumps/configs.pp
+++ b/modules/snapshot/manifests/dumps/configs.pp
@@ -26,6 +26,7 @@
 keep  => '8',
 chunksEnabled => '1',
 chunksForAbstract => '4',
+chunksForPagelogs => '4',
 checkpointTime=> '720',
 recombineHistory  => '0',
 revsPerJob=> '150',
@@ -87,10 +88,11 @@
 en => {
 dblist=> "${dblistsdir}/enwiki.dblist",
 skipdblist=> "${dblistsdir}/skipnone.dblist",
-jobsperbatch  => 'xmlstubsdump=9,abstractsdump=9',
+jobsperbatch  => 
'xmlstubsdump=9,abstractsdump=9,xmlpagelogsdump=9',
 keep  => '7',
 chunksEnabled => '1',
 chunksForAbstract => '27',
+chunksForPagelogs => '27',
 recombineHistory  => '0',
 checkpointTime=> '720',
 revsPerJob=> '150',
@@ -107,10 +109,11 @@
 wd => {
 dblist=> "${dblistsdir}/wikidatawiki.dblist",
 skipdblist=> "${dblistsdir}/skipnone.dblist",
-jobsperbatch  => 'xmlstubsdump=9,abstractsdump=9',
+jobsperbatch  => 
'xmlstubsdump=9,abstractsdump=9,xmlpagelogsdump=9',
 keep  => '7',
 chunksEnabled => '1',
 chunksForAbstract => '27',
+chunksForPagelogs => '27',
 recombineHistory  => '0',
 checkpointTime=> '720',
 revsPerJob=> '150',
@@ -136,6 +139,7 @@
 }
 
 # for xml/sql dumps running on dumpsdata host
+# as well as misc dumps via various cron jobs
 snapshot::dumps::wikiconf { 'wikidump.conf.dumps':
 configtype => 'allwikis',
 config => $config,

-- 
To view, visit https://gerrit.wikimedia.org/r/399589
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ib89de0b72aaa0fe73534bb318b54efcd50414192
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: ArielGlenn 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: when creating lists of dump files for rsync, don't bail on b...

[ArielGlenn (Code Review)]

ArielGlenn has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/399392 )

Change subject: when creating lists of dump files for rsync, don't bail on 
bogus error
..


when creating lists of dump files for rsync, don't bail on bogus error

The "some files failes to be transferred" thing is always a red
herring, and bailing doesn't win us anything, the list is already
written out by then.

Change-Id: Ifb74bf437605c32cbb0d0851d999df2a0e9c45cd
---
M modules/dumps/files/web/list-last-n-good-dumps.py
1 file changed, 4 insertions(+), 0 deletions(-)

Approvals:
  ArielGlenn: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/dumps/files/web/list-last-n-good-dumps.py 
b/modules/dumps/files/web/list-last-n-good-dumps.py
index 7b50990..85b04ad 100755
--- a/modules/dumps/files/web/list-last-n-good-dumps.py
+++ b/modules/dumps/files/web/list-last-n-good-dumps.py
@@ -418,6 +418,10 @@
 # output will be None, we can ignore it
 dummy_output, error = proc.communicate()
 if proc.returncode:
+if 'some files/attrs were not transferred' in error:
+# this can be files that are being rewritten at
+# the moment, we don't care.
+return
 raise DumpListError(
 "command '" + command_string +
 ("' failed with return code %s " % proc.returncode) +

-- 
To view, visit https://gerrit.wikimedia.org/r/399392
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ifb74bf437605c32cbb0d0851d999df2a0e9c45cd
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: ArielGlenn 
Gerrit-Reviewer: ArielGlenn 
Gerrit-Reviewer: Volans 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: when creating lists of dump files for rsync, don't bail on b...

[ArielGlenn (Code Review)]

ArielGlenn has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/399392 )

Change subject: when creating lists of dump files for rsync, don't bail on 
bogus error
..

when creating lists of dump files for rsync, don't bail on bogus error

The "some files failes to be transferred" thing is always a red
herring, and bailing doesn't win us anything, the list is already
written out by then.

Change-Id: Ifb74bf437605c32cbb0d0851d999df2a0e9c45cd
---
M modules/dumps/files/web/list-last-n-good-dumps.py
1 file changed, 4 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/92/399392/1

diff --git a/modules/dumps/files/web/list-last-n-good-dumps.py 
b/modules/dumps/files/web/list-last-n-good-dumps.py
index 7b50990..85b04ad 100755
--- a/modules/dumps/files/web/list-last-n-good-dumps.py
+++ b/modules/dumps/files/web/list-last-n-good-dumps.py
@@ -418,6 +418,10 @@
 # output will be None, we can ignore it
 dummy_output, error = proc.communicate()
 if proc.returncode:
+if 'some files/attrs were not transferred' in error:
+# this can be files that are being rewritten at
+# the moment, we don't care.
+return
 raise DumpListError(
 "command '" + command_string +
 ("' failed with return code %s " % proc.returncode) +

-- 
To view, visit https://gerrit.wikimedia.org/r/399392
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ifb74bf437605c32cbb0d0851d999df2a0e9c45cd
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: ArielGlenn 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: fix up dumps cleanup paths for labstore1003

[ArielGlenn (Code Review)]

ArielGlenn has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/399385 )

Change subject: fix up dumps cleanup paths for labstore1003
..


fix up dumps cleanup paths for labstore1003

Change-Id: I414d00cb9d450c82c8a97036171e5efa17717f09
---
M hieradata/hosts/labstore1003.yaml
1 file changed, 3 insertions(+), 3 deletions(-)

Approvals:
  ArielGlenn: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/hieradata/hosts/labstore1003.yaml 
b/hieradata/hosts/labstore1003.yaml
index 7e709a4..ecb9ed8 100644
--- a/hieradata/hosts/labstore1003.yaml
+++ b/hieradata/hosts/labstore1003.yaml
@@ -1,8 +1,8 @@
 profile::dumps::cleanup::isreplica: false
 profile::dumps::cleanup::labscopy: true
-profile::dumps::xmldumpspublicdir: '/srv/dumps/xmldatadumps/public'
-profile::dumps::miscdumpsdir: '/srv/dumps/xmldatadumps'
-profile::dumps::dumpstempdir: '/srv/dumps/xmldatadumps/temp'
+profile::dumps::xmldumpspublicdir: '/srv/dumps/public'
+profile::dumps::miscdumpsdir: '/srv/dumps'
+profile::dumps::dumpstempdir: '/srv/dumps/temp'
 
 profile::base::domain_search:
   - wikimedia.org

-- 
To view, visit https://gerrit.wikimedia.org/r/399385
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I414d00cb9d450c82c8a97036171e5efa17717f09
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: ArielGlenn 
Gerrit-Reviewer: ArielGlenn 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: fix up dumps cleanup paths for labstore1003

[ArielGlenn (Code Review)]

ArielGlenn has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/399385 )

Change subject: fix up dumps cleanup paths for labstore1003
..

fix up dumps cleanup paths for labstore1003

Change-Id: I414d00cb9d450c82c8a97036171e5efa17717f09
---
M hieradata/hosts/labstore1003.yaml
1 file changed, 3 insertions(+), 3 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/85/399385/1

diff --git a/hieradata/hosts/labstore1003.yaml 
b/hieradata/hosts/labstore1003.yaml
index 7e709a4..ecb9ed8 100644
--- a/hieradata/hosts/labstore1003.yaml
+++ b/hieradata/hosts/labstore1003.yaml
@@ -1,8 +1,8 @@
 profile::dumps::cleanup::isreplica: false
 profile::dumps::cleanup::labscopy: true
-profile::dumps::xmldumpspublicdir: '/srv/dumps/xmldatadumps/public'
-profile::dumps::miscdumpsdir: '/srv/dumps/xmldatadumps'
-profile::dumps::dumpstempdir: '/srv/dumps/xmldatadumps/temp'
+profile::dumps::xmldumpspublicdir: '/srv/dumps/public'
+profile::dumps::miscdumpsdir: '/srv/dumps'
+profile::dumps::dumpstempdir: '/srv/dumps/temp'
 
 profile::base::domain_search:
   - wikimedia.org

-- 
To view, visit https://gerrit.wikimedia.org/r/399385
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I414d00cb9d450c82c8a97036171e5efa17717f09
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: ArielGlenn 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: move one more hardcoded path from dumps profiles to hiera

[ArielGlenn (Code Review)]

ArielGlenn has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/399358 )

Change subject: move one more hardcoded path from dumps profiles to hiera
..


move one more hardcoded path from dumps profiles to hiera

Change-Id: I879097770ace26798127e742b602cd14834fcb3e
---
M hieradata/hosts/dataset1001.yaml
M hieradata/hosts/dumpsdata1001.yaml
M hieradata/hosts/dumpsdata1002.yaml
M hieradata/hosts/ms1001.yaml
M modules/profile/manifests/dumps/generation/server/fallback.pp
M modules/profile/manifests/dumps/generation/server/primary.pp
M modules/profile/manifests/dumps/web/xmldumps_active.pp
M modules/profile/manifests/dumps/web/xmldumps_fallback.pp
8 files changed, 12 insertions(+), 4 deletions(-)

Approvals:
  ArielGlenn: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/hieradata/hosts/dataset1001.yaml b/hieradata/hosts/dataset1001.yaml
index 7029586..7bd4d09 100644
--- a/hieradata/hosts/dataset1001.yaml
+++ b/hieradata/hosts/dataset1001.yaml
@@ -1,5 +1,6 @@
 profile::dumps::cleanup::isreplica: true
 profile::dumps::cleanup::labscopy: false
+profile::dumps::basedatadir: '/data/xmldatadumps'
 profile::dumps::miscdumpsdir: '/data/xmldatadumps/public/other'
 profile::dumps::xmldumpspublicdir: '/data/xmldatadumps/public'
 profile::dumps::dumpstempdir: '/data/xmldatadumps/temp'
diff --git a/hieradata/hosts/dumpsdata1001.yaml 
b/hieradata/hosts/dumpsdata1001.yaml
index 511cd68..f18774a 100644
--- a/hieradata/hosts/dumpsdata1001.yaml
+++ b/hieradata/hosts/dumpsdata1001.yaml
@@ -1,3 +1,4 @@
+profile::dumps::basedatadir: '/data/xmldatadumps'
 profile::dumps::miscdumpsdir: '/data/otherdumps'
 profile::dumps::xmldumpspublicdir: '/data/xmldatadumps/public'
 profile::dumps::dumpstempdir: '/data/xmldatadumps/temp'
diff --git a/hieradata/hosts/dumpsdata1002.yaml 
b/hieradata/hosts/dumpsdata1002.yaml
index 9855fee..717b3d0 100644
--- a/hieradata/hosts/dumpsdata1002.yaml
+++ b/hieradata/hosts/dumpsdata1002.yaml
@@ -1,3 +1,4 @@
+profile::dumps::basedatadir: '/data/xmldatadumps'
 profile::dumps::miscdumpsdir: '/data/otherdumps'
 profile::dumps::xmldumpspublicdir: '/data/xmldatadumps/public'
 profile::dumps::dumpstempdir: '/data/xmldatadumps/temp'
diff --git a/hieradata/hosts/ms1001.yaml b/hieradata/hosts/ms1001.yaml
index 267b0b1..1781b14 100644
--- a/hieradata/hosts/ms1001.yaml
+++ b/hieradata/hosts/ms1001.yaml
@@ -1,6 +1,7 @@
 profile::dumps::cleanup::isreplica: true
 profile::dumps::cleanup::labscopy: false
 
+profile::dumps::basedatadir: '/data/xmldatadumps'
 profile::dumps::miscdumpsdir: '/data/xmldatadumps/public/other'
 profile::dumps::xmldumpspublicdir: '/data/xmldatadumps/public'
 profile::dumps::dumpstempdir: '/data/xmldatadumps/temp'
diff --git a/modules/profile/manifests/dumps/generation/server/fallback.pp 
b/modules/profile/manifests/dumps/generation/server/fallback.pp
index 53a98f7..d247e91 100644
--- a/modules/profile/manifests/dumps/generation/server/fallback.pp
+++ b/modules/profile/manifests/dumps/generation/server/fallback.pp
@@ -1,4 +1,5 @@
 class profile::dumps::generation::server::fallback(
+$datadir = hiera('profile::dumps::basedatadir'),
 $xmldumpsdir = hiera('profile::dumps::xmldumpspublicdir'),
 $miscdatasetsdir = hiera('profile::dumps::miscdumpsdir'),
 $dumpstempdir = hiera('profile::dumps::dumpstempdir'),
@@ -6,7 +7,7 @@
 class { '::dumpsuser': }
 
 class { '::dumps::generation::server::dirs':
-datadir => '/data/xmldatadumps',
+datadir => $datadir,
 xmldumpsdir => $xmldumpsdir,
 tempdir => $dumpstempdir,
 miscdatasetsdir => $miscdatasetsdir,
diff --git a/modules/profile/manifests/dumps/generation/server/primary.pp 
b/modules/profile/manifests/dumps/generation/server/primary.pp
index 6ad0dc6..7c5b98c 100644
--- a/modules/profile/manifests/dumps/generation/server/primary.pp
+++ b/modules/profile/manifests/dumps/generation/server/primary.pp
@@ -1,4 +1,5 @@
 class profile::dumps::generation::server::primary(
+$datadir = hiera('profile::dumps::basedatadir'),
 $xmldumpsdir = hiera('profile::dumps::xmldumpspublicdir'),
 $miscdatasetsdir = hiera('profile::dumps::miscdumpsdir'),
 $dumpstempdir = hiera('profile::dumps::dumpstempdir'),
@@ -6,7 +7,7 @@
 class { '::dumpsuser': }
 
 class { '::dumps::generation::server::dirs':
-datadir => '/data/xmldatadumps',
+datadir => $datadir,
 xmldumpsdir => $xmldumpsdir,
 tempdir => $dumpstempdir,
 miscdatasetsdir => $miscdatasetsdir,
diff --git a/modules/profile/manifests/dumps/web/xmldumps_active.pp 
b/modules/profile/manifests/dumps/web/xmldumps_active.pp
index 252fc6b..d7d1260 100644
--- a/modules/profile/manifests/dumps/web/xmldumps_active.pp
+++ b/modules/profile/manifests/dumps/web/xmldumps_active.pp
@@ -1,5 +1,6 @@
 class profile::dumps::web::xmldumps_active(
 $do_acme = 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: move one more hardcoded path from dumps profiles to hiera

[ArielGlenn (Code Review)]

ArielGlenn has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/399358 )

Change subject: move one more hardcoded path from dumps profiles to hiera
..

move one more hardcoded path from dumps profiles to hiera

Change-Id: I879097770ace26798127e742b602cd14834fcb3e
---
M hieradata/hosts/dataset1001.yaml
M hieradata/hosts/dumpsdata1001.yaml
M hieradata/hosts/dumpsdata1002.yaml
M hieradata/hosts/ms1001.yaml
M modules/profile/manifests/dumps/generation/server/fallback.pp
M modules/profile/manifests/dumps/generation/server/primary.pp
M modules/profile/manifests/dumps/web/xmldumps_active.pp
M modules/profile/manifests/dumps/web/xmldumps_fallback.pp
8 files changed, 12 insertions(+), 4 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/58/399358/1

diff --git a/hieradata/hosts/dataset1001.yaml b/hieradata/hosts/dataset1001.yaml
index 7029586..7bd4d09 100644
--- a/hieradata/hosts/dataset1001.yaml
+++ b/hieradata/hosts/dataset1001.yaml
@@ -1,5 +1,6 @@
 profile::dumps::cleanup::isreplica: true
 profile::dumps::cleanup::labscopy: false
+profile::dumps::basedatadir: '/data/xmldatadumps'
 profile::dumps::miscdumpsdir: '/data/xmldatadumps/public/other'
 profile::dumps::xmldumpspublicdir: '/data/xmldatadumps/public'
 profile::dumps::dumpstempdir: '/data/xmldatadumps/temp'
diff --git a/hieradata/hosts/dumpsdata1001.yaml 
b/hieradata/hosts/dumpsdata1001.yaml
index 511cd68..f18774a 100644
--- a/hieradata/hosts/dumpsdata1001.yaml
+++ b/hieradata/hosts/dumpsdata1001.yaml
@@ -1,3 +1,4 @@
+profile::dumps::basedatadir: '/data/xmldatadumps'
 profile::dumps::miscdumpsdir: '/data/otherdumps'
 profile::dumps::xmldumpspublicdir: '/data/xmldatadumps/public'
 profile::dumps::dumpstempdir: '/data/xmldatadumps/temp'
diff --git a/hieradata/hosts/dumpsdata1002.yaml 
b/hieradata/hosts/dumpsdata1002.yaml
index 9855fee..717b3d0 100644
--- a/hieradata/hosts/dumpsdata1002.yaml
+++ b/hieradata/hosts/dumpsdata1002.yaml
@@ -1,3 +1,4 @@
+profile::dumps::basedatadir: '/data/xmldatadumps'
 profile::dumps::miscdumpsdir: '/data/otherdumps'
 profile::dumps::xmldumpspublicdir: '/data/xmldatadumps/public'
 profile::dumps::dumpstempdir: '/data/xmldatadumps/temp'
diff --git a/hieradata/hosts/ms1001.yaml b/hieradata/hosts/ms1001.yaml
index 267b0b1..1781b14 100644
--- a/hieradata/hosts/ms1001.yaml
+++ b/hieradata/hosts/ms1001.yaml
@@ -1,6 +1,7 @@
 profile::dumps::cleanup::isreplica: true
 profile::dumps::cleanup::labscopy: false
 
+profile::dumps::basedatadir: '/data/xmldatadumps'
 profile::dumps::miscdumpsdir: '/data/xmldatadumps/public/other'
 profile::dumps::xmldumpspublicdir: '/data/xmldatadumps/public'
 profile::dumps::dumpstempdir: '/data/xmldatadumps/temp'
diff --git a/modules/profile/manifests/dumps/generation/server/fallback.pp 
b/modules/profile/manifests/dumps/generation/server/fallback.pp
index 53a98f7..d247e91 100644
--- a/modules/profile/manifests/dumps/generation/server/fallback.pp
+++ b/modules/profile/manifests/dumps/generation/server/fallback.pp
@@ -1,4 +1,5 @@
 class profile::dumps::generation::server::fallback(
+$datadir = hiera('profile::dumps::basedatadir'),
 $xmldumpsdir = hiera('profile::dumps::xmldumpspublicdir'),
 $miscdatasetsdir = hiera('profile::dumps::miscdumpsdir'),
 $dumpstempdir = hiera('profile::dumps::dumpstempdir'),
@@ -6,7 +7,7 @@
 class { '::dumpsuser': }
 
 class { '::dumps::generation::server::dirs':
-datadir => '/data/xmldatadumps',
+datadir => $datadir,
 xmldumpsdir => $xmldumpsdir,
 tempdir => $dumpstempdir,
 miscdatasetsdir => $miscdatasetsdir,
diff --git a/modules/profile/manifests/dumps/generation/server/primary.pp 
b/modules/profile/manifests/dumps/generation/server/primary.pp
index 6ad0dc6..7c5b98c 100644
--- a/modules/profile/manifests/dumps/generation/server/primary.pp
+++ b/modules/profile/manifests/dumps/generation/server/primary.pp
@@ -1,4 +1,5 @@
 class profile::dumps::generation::server::primary(
+$datadir = hiera('profile::dumps::basedatadir'),
 $xmldumpsdir = hiera('profile::dumps::xmldumpspublicdir'),
 $miscdatasetsdir = hiera('profile::dumps::miscdumpsdir'),
 $dumpstempdir = hiera('profile::dumps::dumpstempdir'),
@@ -6,7 +7,7 @@
 class { '::dumpsuser': }
 
 class { '::dumps::generation::server::dirs':
-datadir => '/data/xmldatadumps',
+datadir => $datadir,
 xmldumpsdir => $xmldumpsdir,
 tempdir => $dumpstempdir,
 miscdatasetsdir => $miscdatasetsdir,
diff --git a/modules/profile/manifests/dumps/web/xmldumps_active.pp 
b/modules/profile/manifests/dumps/web/xmldumps_active.pp
index 252fc6b..d7d1260 100644
--- a/modules/profile/manifests/dumps/web/xmldumps_active.pp
+++ b/modules/profile/manifests/dumps/web/xmldumps_active.pp
@@ -1,5 +1,6 @@
 class profile::dumps::web::xmldumps_active(
 $do_acme 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: use hieras settings for a few more hardcoded paths in dumps ...

[ArielGlenn (Code Review)]

ArielGlenn has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/399356 )

Change subject: use hieras settings for a few more hardcoded paths in dumps 
profiles
..

use hieras settings for a few more hardcoded paths in dumps profiles

Change-Id: I7d9830c8040695e47723e5a69ede136de8b2458d
---
M modules/profile/manifests/dumps/generation/server/fallback.pp
M modules/profile/manifests/dumps/generation/server/primary.pp
M modules/profile/manifests/dumps/web/xmldumps_active.pp
M modules/profile/manifests/dumps/web/xmldumps_fallback.pp
4 files changed, 25 insertions(+), 15 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/56/399356/1

diff --git a/modules/profile/manifests/dumps/generation/server/fallback.pp 
b/modules/profile/manifests/dumps/generation/server/fallback.pp
index 7f9b9cf..53a98f7 100644
--- a/modules/profile/manifests/dumps/generation/server/fallback.pp
+++ b/modules/profile/manifests/dumps/generation/server/fallback.pp
@@ -1,11 +1,15 @@
-class profile::dumps::generation::server::fallback {
+class profile::dumps::generation::server::fallback(
+$xmldumpsdir = hiera('profile::dumps::xmldumpspublicdir'),
+$miscdatasetsdir = hiera('profile::dumps::miscdumpsdir'),
+$dumpstempdir = hiera('profile::dumps::dumpstempdir'),
+) {
 class { '::dumpsuser': }
 
 class { '::dumps::generation::server::dirs':
 datadir => '/data/xmldatadumps',
-xmldumpsdir => '/data/xmldatadumps/public',
-tempdir => '/data/xmldatadumps/temp',
-miscdatasetsdir => '/data/otherdumps',
+xmldumpsdir => $xmldumpsdir,
+tempdir => $dumpstempdir,
+miscdatasetsdir => $miscdatasetsdir,
 user=> $dumpsuser::user,
 group   => $dumpsuser::group,
 }
diff --git a/modules/profile/manifests/dumps/generation/server/primary.pp 
b/modules/profile/manifests/dumps/generation/server/primary.pp
index a6ad506..6ad0dc6 100644
--- a/modules/profile/manifests/dumps/generation/server/primary.pp
+++ b/modules/profile/manifests/dumps/generation/server/primary.pp
@@ -1,19 +1,23 @@
-class profile::dumps::generation::server::primary {
+class profile::dumps::generation::server::primary(
+$xmldumpsdir = hiera('profile::dumps::xmldumpspublicdir'),
+$miscdatasetsdir = hiera('profile::dumps::miscdumpsdir'),
+$dumpstempdir = hiera('profile::dumps::dumpstempdir'),
+) {
 class { '::dumpsuser': }
 
 class { '::dumps::generation::server::dirs':
 datadir => '/data/xmldatadumps',
-xmldumpsdir => '/data/xmldatadumps/public',
-tempdir => '/data/xmldatadumps/temp',
-miscdatasetsdir => '/data/otherdumps',
+xmldumpsdir => $xmldumpsdir,
+tempdir => $dumpstempdir,
+miscdatasetsdir => $miscdatasetsdir,
 user=> $dumpsuser::user,
 group   => $dumpsuser::group,
 }
 
 class { '::dumps::generation::server::rsyncer':
-xmldumpsdir=> '/data/xmldatadumps/public',
+xmldumpsdir=> $xmldumpsdir,
 xmlremotedirs  => 
'dumpsdata1002.eqiad.wmnet::data/xmldatadumps/public/,dataset1001.wikimedia.org::data/xmldatadumps/public/',
-miscdumpsdir   => '/data/otherdumps',
+miscdumpsdir   => $miscdatasetsdir,
 miscremotedirs => 
'dumpsdata1002.eqiad.wmnet::data/otherdumps/,dataset1001.wikimedia.org::data/xmldatadumps/public/other/',
 }
 }
diff --git a/modules/profile/manifests/dumps/web/xmldumps_active.pp 
b/modules/profile/manifests/dumps/web/xmldumps_active.pp
index 6eff666..252fc6b 100644
--- a/modules/profile/manifests/dumps/web/xmldumps_active.pp
+++ b/modules/profile/manifests/dumps/web/xmldumps_active.pp
@@ -1,15 +1,15 @@
 class profile::dumps::web::xmldumps_active(
 $do_acme = hiera('do_acme'),
+$xmldumpsdir = hiera('profile::dumps::xmldumpspublicdir'),
+$miscdatasetsdir = hiera('profile::dumps::miscdumpsdir'),
 ) {
 class { '::dumpsuser': }
-$xmldumpsdir = '/data/xmldatadumps/public'
-$miscdatasetsdir = '/data/xmldatadumps/public/other'
 
 class {'::dumps::web::xmldumps_active':
 do_acme  => $do_acme,
 datadir  => '/data/xmldatadumps',
 xmldumpsdir  => $xmldumpsdir,
-miscdatasetsdir  => '/data/xmldatadumps/public/other',
+miscdatasetsdir  => $miscdatasetsdir,
 logs_dest=> 
'stat1005.eqiad.wmnet::srv/log/webrequest/archive/dumps.wikimedia.org/',
 htmldumps_server => 'francium.eqiad.wmnet',
 xmldumps_server  => 'dumps.wikimedia.org',
diff --git a/modules/profile/manifests/dumps/web/xmldumps_fallback.pp 
b/modules/profile/manifests/dumps/web/xmldumps_fallback.pp
index bb58fa9..08b9d14 100644
--- a/modules/profile/manifests/dumps/web/xmldumps_fallback.pp
+++ 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: use hieras settings for a few more hardcoded paths in dumps ...

[ArielGlenn (Code Review)]

ArielGlenn has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/399356 )

Change subject: use hieras settings for a few more hardcoded paths in dumps 
profiles
..


use hieras settings for a few more hardcoded paths in dumps profiles

Change-Id: I7d9830c8040695e47723e5a69ede136de8b2458d
---
M modules/profile/manifests/dumps/generation/server/fallback.pp
M modules/profile/manifests/dumps/generation/server/primary.pp
M modules/profile/manifests/dumps/web/xmldumps_active.pp
M modules/profile/manifests/dumps/web/xmldumps_fallback.pp
4 files changed, 25 insertions(+), 15 deletions(-)

Approvals:
  ArielGlenn: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/profile/manifests/dumps/generation/server/fallback.pp 
b/modules/profile/manifests/dumps/generation/server/fallback.pp
index 7f9b9cf..53a98f7 100644
--- a/modules/profile/manifests/dumps/generation/server/fallback.pp
+++ b/modules/profile/manifests/dumps/generation/server/fallback.pp
@@ -1,11 +1,15 @@
-class profile::dumps::generation::server::fallback {
+class profile::dumps::generation::server::fallback(
+$xmldumpsdir = hiera('profile::dumps::xmldumpspublicdir'),
+$miscdatasetsdir = hiera('profile::dumps::miscdumpsdir'),
+$dumpstempdir = hiera('profile::dumps::dumpstempdir'),
+) {
 class { '::dumpsuser': }
 
 class { '::dumps::generation::server::dirs':
 datadir => '/data/xmldatadumps',
-xmldumpsdir => '/data/xmldatadumps/public',
-tempdir => '/data/xmldatadumps/temp',
-miscdatasetsdir => '/data/otherdumps',
+xmldumpsdir => $xmldumpsdir,
+tempdir => $dumpstempdir,
+miscdatasetsdir => $miscdatasetsdir,
 user=> $dumpsuser::user,
 group   => $dumpsuser::group,
 }
diff --git a/modules/profile/manifests/dumps/generation/server/primary.pp 
b/modules/profile/manifests/dumps/generation/server/primary.pp
index a6ad506..6ad0dc6 100644
--- a/modules/profile/manifests/dumps/generation/server/primary.pp
+++ b/modules/profile/manifests/dumps/generation/server/primary.pp
@@ -1,19 +1,23 @@
-class profile::dumps::generation::server::primary {
+class profile::dumps::generation::server::primary(
+$xmldumpsdir = hiera('profile::dumps::xmldumpspublicdir'),
+$miscdatasetsdir = hiera('profile::dumps::miscdumpsdir'),
+$dumpstempdir = hiera('profile::dumps::dumpstempdir'),
+) {
 class { '::dumpsuser': }
 
 class { '::dumps::generation::server::dirs':
 datadir => '/data/xmldatadumps',
-xmldumpsdir => '/data/xmldatadumps/public',
-tempdir => '/data/xmldatadumps/temp',
-miscdatasetsdir => '/data/otherdumps',
+xmldumpsdir => $xmldumpsdir,
+tempdir => $dumpstempdir,
+miscdatasetsdir => $miscdatasetsdir,
 user=> $dumpsuser::user,
 group   => $dumpsuser::group,
 }
 
 class { '::dumps::generation::server::rsyncer':
-xmldumpsdir=> '/data/xmldatadumps/public',
+xmldumpsdir=> $xmldumpsdir,
 xmlremotedirs  => 
'dumpsdata1002.eqiad.wmnet::data/xmldatadumps/public/,dataset1001.wikimedia.org::data/xmldatadumps/public/',
-miscdumpsdir   => '/data/otherdumps',
+miscdumpsdir   => $miscdatasetsdir,
 miscremotedirs => 
'dumpsdata1002.eqiad.wmnet::data/otherdumps/,dataset1001.wikimedia.org::data/xmldatadumps/public/other/',
 }
 }
diff --git a/modules/profile/manifests/dumps/web/xmldumps_active.pp 
b/modules/profile/manifests/dumps/web/xmldumps_active.pp
index 6eff666..252fc6b 100644
--- a/modules/profile/manifests/dumps/web/xmldumps_active.pp
+++ b/modules/profile/manifests/dumps/web/xmldumps_active.pp
@@ -1,15 +1,15 @@
 class profile::dumps::web::xmldumps_active(
 $do_acme = hiera('do_acme'),
+$xmldumpsdir = hiera('profile::dumps::xmldumpspublicdir'),
+$miscdatasetsdir = hiera('profile::dumps::miscdumpsdir'),
 ) {
 class { '::dumpsuser': }
-$xmldumpsdir = '/data/xmldatadumps/public'
-$miscdatasetsdir = '/data/xmldatadumps/public/other'
 
 class {'::dumps::web::xmldumps_active':
 do_acme  => $do_acme,
 datadir  => '/data/xmldatadumps',
 xmldumpsdir  => $xmldumpsdir,
-miscdatasetsdir  => '/data/xmldatadumps/public/other',
+miscdatasetsdir  => $miscdatasetsdir,
 logs_dest=> 
'stat1005.eqiad.wmnet::srv/log/webrequest/archive/dumps.wikimedia.org/',
 htmldumps_server => 'francium.eqiad.wmnet',
 xmldumps_server  => 'dumps.wikimedia.org',
diff --git a/modules/profile/manifests/dumps/web/xmldumps_fallback.pp 
b/modules/profile/manifests/dumps/web/xmldumps_fallback.pp
index bb58fa9..08b9d14 100644
--- a/modules/profile/manifests/dumps/web/xmldumps_fallback.pp
+++ b/modules/profile/manifests/dumps/web/xmldumps_fallback.pp

[MediaWiki-commits] [Gerrit] operations/puppet[production]: enable dumps of big wikis to run in a fixed order

[ArielGlenn (Code Review)]

ArielGlenn has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/399158 )

Change subject: enable dumps of big wikis to run in a fixed order
..


enable dumps of big wikis to run in a fixed order

Start dewiki first, it takes several days to complete the revision
content history dumps.

Change-Id: I167a6648a369202bf8234961674364245edf96cf
---
M modules/snapshot/manifests/dumps/configs.pp
M modules/snapshot/manifests/dumps/dblists.pp
2 files changed, 4 insertions(+), 3 deletions(-)

Approvals:
  ArielGlenn: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/snapshot/manifests/dumps/configs.pp 
b/modules/snapshot/manifests/dumps/configs.pp
index f1a2de1..36ce427 100644
--- a/modules/snapshot/manifests/dumps/configs.pp
+++ b/modules/snapshot/manifests/dumps/configs.pp
@@ -32,6 +32,7 @@
 retryWait => '30',
 maxRetries=> '3',
 revsMargin=> '100',
+fixeddumporder=> '1',
 wikis => {
 ruwiki => {
 pagesPerChunkHistory  => 
'311181,1142420,1627923,3122803',
diff --git a/modules/snapshot/manifests/dumps/dblists.pp 
b/modules/snapshot/manifests/dumps/dblists.pp
index 1dcb6c7..d28843e 100644
--- a/modules/snapshot/manifests/dumps/dblists.pp
+++ b/modules/snapshot/manifests/dumps/dblists.pp
@@ -5,9 +5,9 @@
 $wikidatawiki = ['wikidatawiki']
 $wikidatawiki_dblist = join($wikidatawiki, "\n")
 
-$bigwikis = ['dewiki', 'eswiki', 'frwiki', 'itwiki', 'jawiki',
-'metawiki', 'nlwiki', 'plwiki', 'ptwiki', 'ruwiki', 
'commonswiki',
-'wikidatawiki', 'zhwiki']
+$bigwikis = ['dewiki', 'commonswiki', 'frwiki', 'eswiki', 'itwiki', 
'jawiki',
+'metawiki', 'nlwiki', 'plwiki', 'ptwiki', 'ruwiki',
+'zhwiki']
 $bigwikis_dblist = join($bigwikis, "\n")
 
 # labswiki(s) can't be dumped from snapshot hosts

-- 
To view, visit https://gerrit.wikimedia.org/r/399158
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I167a6648a369202bf8234961674364245edf96cf
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: ArielGlenn 
Gerrit-Reviewer: ArielGlenn 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: add dumpsgen user to labstore1003 for dumps cron cleanup job

[ArielGlenn (Code Review)]

ArielGlenn has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/399261 )

Change subject: add dumpsgen user to labstore1003 for dumps cron cleanup job
..


add dumpsgen user to labstore1003 for dumps cron cleanup job

Bug: T154798
Change-Id: I4f28c977b8c956365f6ea534cff29ec606016437
---
A modules/profile/manifests/dumps/web/dumpsuser.pp
M modules/role/manifests/labs/nfs/misc.pp
2 files changed, 4 insertions(+), 0 deletions(-)

Approvals:
  ArielGlenn: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/profile/manifests/dumps/web/dumpsuser.pp 
b/modules/profile/manifests/dumps/web/dumpsuser.pp
new file mode 100644
index 000..6c71de7
--- /dev/null
+++ b/modules/profile/manifests/dumps/web/dumpsuser.pp
@@ -0,0 +1,3 @@
+class profile::dumps::web::dumpsuser {
+class { '::dumpsuser': }
+}
diff --git a/modules/role/manifests/labs/nfs/misc.pp 
b/modules/role/manifests/labs/nfs/misc.pp
index 29e8014..47972b3 100644
--- a/modules/role/manifests/labs/nfs/misc.pp
+++ b/modules/role/manifests/labs/nfs/misc.pp
@@ -27,6 +27,7 @@
 # servers, they have cleanups already added
 # in the role
 include ::profile::dumps::web::cleanup
+include ::profile::dumps::web::dumpsuser
 
 rsync::server::module { 'pagecounts':
 path=> '/srv/dumps/pagecounts',

-- 
To view, visit https://gerrit.wikimedia.org/r/399261
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I4f28c977b8c956365f6ea534cff29ec606016437
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: ArielGlenn 
Gerrit-Reviewer: ArielGlenn 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: add dumpsgen user to labstore1003 for dumps cron cleanup job

[ArielGlenn (Code Review)]

ArielGlenn has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/399261 )

Change subject: add dumpsgen user to labstore1003 for dumps cron cleanup job
..

add dumpsgen user to labstore1003 for dumps cron cleanup job

Bug: T154798
Change-Id: I4f28c977b8c956365f6ea534cff29ec606016437
---
A modules/profile/manifests/dumps/web/dumpsuser.pp
M modules/role/manifests/labs/nfs/misc.pp
2 files changed, 4 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/61/399261/1

diff --git a/modules/profile/manifests/dumps/web/dumpsuser.pp 
b/modules/profile/manifests/dumps/web/dumpsuser.pp
new file mode 100644
index 000..6c71de7
--- /dev/null
+++ b/modules/profile/manifests/dumps/web/dumpsuser.pp
@@ -0,0 +1,3 @@
+class profile::dumps::web::dumpsuser {
+class { '::dumpsuser': }
+}
diff --git a/modules/role/manifests/labs/nfs/misc.pp 
b/modules/role/manifests/labs/nfs/misc.pp
index 29e8014..47972b3 100644
--- a/modules/role/manifests/labs/nfs/misc.pp
+++ b/modules/role/manifests/labs/nfs/misc.pp
@@ -27,6 +27,7 @@
 # servers, they have cleanups already added
 # in the role
 include ::profile::dumps::web::cleanup
+include ::profile::dumps::web::dumpsuser
 
 rsync::server::module { 'pagecounts':
 path=> '/srv/dumps/pagecounts',

-- 
To view, visit https://gerrit.wikimedia.org/r/399261
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I4f28c977b8c956365f6ea534cff29ec606016437
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: ArielGlenn 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: dataset1001 rsync to labs of dumps can now use explicit incl...

[ArielGlenn (Code Review)]

ArielGlenn has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/336204 )

Change subject: dataset1001 rsync to labs of dumps can now use explicit 
inclusion list
..


dataset1001 rsync to labs of dumps can now use explicit inclusion list

We can invoke rsync directly with the right include/exclude args.
Remove the complicated python script for copying, no longer needed.

We must clean up old dumps manually now, because we can't rsync --delete,
so include manifests for that.

The copy to labstore1003, including the hiera setting for this config,
"labscopy", will go away once labstore1006,7 are in full production.

While we're here, clean up var names etc in the rsync script. Yeah it's
a one-off with an expiration date but still.

Bug: T154798
Change-Id: Id23a1952f3664941877ee9242bc3d11743fff964
---
M hieradata/hosts/dataset1001.yaml
M hieradata/hosts/dumpsdata1001.yaml
M hieradata/hosts/dumpsdata1002.yaml
M hieradata/hosts/labstore1003.yaml
M hieradata/hosts/labstore1006.yaml
M hieradata/hosts/labstore1007.yaml
M hieradata/hosts/ms1001.yaml
D modules/dumps/files/copying/wmfdumpsmirror.py
M modules/dumps/manifests/copying/labs.pp
M modules/dumps/manifests/web/cleanup.pp
M modules/dumps/manifests/web/cleanups/miscdumps.pp
M modules/dumps/manifests/web/cleanups/xmldumps.pp
M modules/dumps/templates/copying/labs-rsync-cron.sh.erb
M modules/profile/manifests/dumps/web/cleanup.pp
M modules/role/manifests/labs/nfs/misc.pp
15 files changed, 62 insertions(+), 826 deletions(-)

Approvals:
  ArielGlenn: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/hieradata/hosts/dataset1001.yaml b/hieradata/hosts/dataset1001.yaml
index 155aa9d..7029586 100644
--- a/hieradata/hosts/dataset1001.yaml
+++ b/hieradata/hosts/dataset1001.yaml
@@ -1,4 +1,5 @@
 profile::dumps::cleanup::isreplica: true
+profile::dumps::cleanup::labscopy: false
 profile::dumps::miscdumpsdir: '/data/xmldatadumps/public/other'
 profile::dumps::xmldumpspublicdir: '/data/xmldatadumps/public'
 profile::dumps::dumpstempdir: '/data/xmldatadumps/temp'
diff --git a/hieradata/hosts/dumpsdata1001.yaml 
b/hieradata/hosts/dumpsdata1001.yaml
index 03707e5..511cd68 100644
--- a/hieradata/hosts/dumpsdata1001.yaml
+++ b/hieradata/hosts/dumpsdata1001.yaml
@@ -3,3 +3,4 @@
 profile::dumps::dumpstempdir: '/data/xmldatadumps/temp'
 
 profile::dumps::cleanup::isreplica: false
+profile::dumps::cleanup::labscopy: false
diff --git a/hieradata/hosts/dumpsdata1002.yaml 
b/hieradata/hosts/dumpsdata1002.yaml
index 171deca..9855fee 100644
--- a/hieradata/hosts/dumpsdata1002.yaml
+++ b/hieradata/hosts/dumpsdata1002.yaml
@@ -6,6 +6,7 @@
 # we configure cleanups of old files there the
 # same way we do the active generating host
 profile::dumps::cleanup::isreplica: false
+profile::dumps::cleanup::labscopy: false
 
 profile::dumps::rsyncer:
   dumps_user: 'dumpsgen'
diff --git a/hieradata/hosts/labstore1003.yaml 
b/hieradata/hosts/labstore1003.yaml
index 44b71c6..7e709a4 100644
--- a/hieradata/hosts/labstore1003.yaml
+++ b/hieradata/hosts/labstore1003.yaml
@@ -1,3 +1,9 @@
+profile::dumps::cleanup::isreplica: false
+profile::dumps::cleanup::labscopy: true
+profile::dumps::xmldumpspublicdir: '/srv/dumps/xmldatadumps/public'
+profile::dumps::miscdumpsdir: '/srv/dumps/xmldatadumps'
+profile::dumps::dumpstempdir: '/srv/dumps/xmldatadumps/temp'
+
 profile::base::domain_search:
   - wikimedia.org
   - eqiad.wmnet
diff --git a/hieradata/hosts/labstore1006.yaml 
b/hieradata/hosts/labstore1006.yaml
index fcb31f2..476d6cd 100644
--- a/hieradata/hosts/labstore1006.yaml
+++ b/hieradata/hosts/labstore1006.yaml
@@ -3,6 +3,7 @@
 profile::dumps::dumpstempdir: '/srv/dumps/xmldatadumps/temp'
 
 profile::dumps::cleanup::isreplica: true
+profile::dumps::cleanup::labscopy: false
 
 profile::dumps::rsyncer:
   dumps_user: 'dumpsgen'
diff --git a/hieradata/hosts/labstore1007.yaml 
b/hieradata/hosts/labstore1007.yaml
index fcb31f2..476d6cd 100644
--- a/hieradata/hosts/labstore1007.yaml
+++ b/hieradata/hosts/labstore1007.yaml
@@ -3,6 +3,7 @@
 profile::dumps::dumpstempdir: '/srv/dumps/xmldatadumps/temp'
 
 profile::dumps::cleanup::isreplica: true
+profile::dumps::cleanup::labscopy: false
 
 profile::dumps::rsyncer:
   dumps_user: 'dumpsgen'
diff --git a/hieradata/hosts/ms1001.yaml b/hieradata/hosts/ms1001.yaml
index 9d6f50f..267b0b1 100644
--- a/hieradata/hosts/ms1001.yaml
+++ b/hieradata/hosts/ms1001.yaml
@@ -1,4 +1,6 @@
 profile::dumps::cleanup::isreplica: true
+profile::dumps::cleanup::labscopy: false
+
 profile::dumps::miscdumpsdir: '/data/xmldatadumps/public/other'
 profile::dumps::xmldumpspublicdir: '/data/xmldatadumps/public'
 profile::dumps::dumpstempdir: '/data/xmldatadumps/temp'
diff --git a/modules/dumps/files/copying/wmfdumpsmirror.py 
b/modules/dumps/files/copying/wmfdumpsmirror.py
deleted file mode 100644
index 93d53bc..000
--- a/modules/dumps/files/copying/wmfdumpsmirror.py
+++ /dev/null

[MediaWiki-commits] [Gerrit] operations/puppet[production]: enable dumps of big wikis to run in a fixed order

[ArielGlenn (Code Review)]

ArielGlenn has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/399158 )

Change subject: enable dumps of big wikis to run in a fixed order
..

enable dumps of big wikis to run in a fixed order

Start dewiki first, it takes several days to complete the revision
content history dumps.

Change-Id: I167a6648a369202bf8234961674364245edf96cf
---
M modules/snapshot/manifests/dumps/configs.pp
M modules/snapshot/manifests/dumps/dblists.pp
2 files changed, 4 insertions(+), 3 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/58/399158/1

diff --git a/modules/snapshot/manifests/dumps/configs.pp 
b/modules/snapshot/manifests/dumps/configs.pp
index f1a2de1..36ce427 100644
--- a/modules/snapshot/manifests/dumps/configs.pp
+++ b/modules/snapshot/manifests/dumps/configs.pp
@@ -32,6 +32,7 @@
 retryWait => '30',
 maxRetries=> '3',
 revsMargin=> '100',
+fixeddumporder=> '1',
 wikis => {
 ruwiki => {
 pagesPerChunkHistory  => 
'311181,1142420,1627923,3122803',
diff --git a/modules/snapshot/manifests/dumps/dblists.pp 
b/modules/snapshot/manifests/dumps/dblists.pp
index 1dcb6c7..d28843e 100644
--- a/modules/snapshot/manifests/dumps/dblists.pp
+++ b/modules/snapshot/manifests/dumps/dblists.pp
@@ -5,9 +5,9 @@
 $wikidatawiki = ['wikidatawiki']
 $wikidatawiki_dblist = join($wikidatawiki, "\n")
 
-$bigwikis = ['dewiki', 'eswiki', 'frwiki', 'itwiki', 'jawiki',
-'metawiki', 'nlwiki', 'plwiki', 'ptwiki', 'ruwiki', 
'commonswiki',
-'wikidatawiki', 'zhwiki']
+$bigwikis = ['dewiki', 'commonswiki', 'frwiki', 'eswiki', 'itwiki', 
'jawiki',
+'metawiki', 'nlwiki', 'plwiki', 'ptwiki', 'ruwiki',
+'zhwiki']
 $bigwikis_dblist = join($bigwikis, "\n")
 
 # labswiki(s) can't be dumped from snapshot hosts

-- 
To view, visit https://gerrit.wikimedia.org/r/399158
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I167a6648a369202bf8234961674364245edf96cf
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: ArielGlenn 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/dumps[master]: config setting to permit a list of wikis to be dumped in a s...

[ArielGlenn (Code Review)]

ArielGlenn has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/398861 )

Change subject: config setting to permit a list of wikis to be dumped in a 
specific order
..


config setting to permit a list of wikis to be dumped in a specific order

We don't want these for all wikis but it can be useful for small lists.

Example the big wikis that dump via 4 processes at a time.  Some of these
take days longer than others; if we start those first, multiple other wikis
will run to completion on other cores, while the first ones chug along.
If the order is by longet to dump, these big wikis often wind up
starting near the end ofthe list, and run along by themselves after
everything else is completed.

We can't really just expand the configs of these big slow wikis
so they use a pile more processors at once, that's not ok for the db
servers. (enwiki, wikidatawiki, yes, but not the rest.) So, move them
to the front of the queue.

Change-Id: I494ed57363b1ddfe0e10be0aed25facb7ca8a364
---
M xmldumps-backup/defaults.conf
M xmldumps-backup/doc/README.config
M xmldumps-backup/dumps/WikiDump.py
M xmldumps-backup/dumps/utils.py
M xmldumps-backup/worker.py
5 files changed, 44 insertions(+), 9 deletions(-)

Approvals:
  ArielGlenn: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/xmldumps-backup/defaults.conf b/xmldumps-backup/defaults.conf
index d17cbbb..e95ac47 100644
--- a/xmldumps-backup/defaults.conf
+++ b/xmldumps-backup/defaults.conf
@@ -76,3 +76,6 @@
 orderrevs=0
 minpages=1
 maxrevs=5
+
+[misc]
+fixeddumporder=0
\ No newline at end of file
diff --git a/xmldumps-backup/doc/README.config 
b/xmldumps-backup/doc/README.config
index d4d76f4..6ccd705 100644
--- a/xmldumps-backup/doc/README.config
+++ b/xmldumps-backup/doc/README.config
@@ -245,6 +245,15 @@
 The above options do not have to be specified in the config file,
 since default values are provided.
 
+=== Misc (i.e.: [misc])
+fixed_dump_order -- set this to a non-zero integer to enable dumps
+of wikis in the specified db list to be dumped
+in the order listed
+   Default value: 0 (wiki dumped longest ago goes first)
+
+The above options do not have to be specified in the config file,
+since default values are provided.
+
 === Per-wiki configuration
 The following settings may be overriden for specific wikis by specifying
 their name (the name of the db in the database) as a section header,
diff --git a/xmldumps-backup/dumps/WikiDump.py 
b/xmldumps-backup/dumps/WikiDump.py
index 80115ec..8ba3838 100644
--- a/xmldumps-backup/dumps/WikiDump.py
+++ b/xmldumps-backup/dumps/WikiDump.py
@@ -171,9 +171,8 @@
 globals like entries in 'wiki' or 'output' that can
 be overriden by a specific named section
 """
-self.db_list = MiscUtils.db_list(self.get_opt_in_overrides_or_default(
-"wiki", "dblist", 0))
-
+self.db_list_unsorted = 
MiscUtils.db_list(self.get_opt_in_overrides_or_default(
+"wiki", "dblist", 0), nosort=True)
 # permit comma-separated list of files so that eg some script
 # can skip all private and/or closed wikis in addition to some
 # other exclusion list
@@ -191,7 +190,9 @@
 self.apijobs = self.get_opt_in_overrides_or_default(
 "wiki", "apijobs", 0)
 
-self.db_list = list(set(self.db_list) - set(self.skip_db_list))
+self.db_list_unsorted = [dbname for dbname in self.db_list_unsorted
+ if dbname not in self.skip_db_list]
+self.db_list = sorted(self.db_list_unsorted)
 
 if not self.conf.has_section('output'):
 self.conf.add_section('output')
@@ -206,6 +207,11 @@
 self.fileperms = self.get_opt_in_overrides_or_default("output", 
"fileperms", 0)
 self.fileperms = int(self.fileperms, 0)
 
+if not self.conf.has_section('misc'):
+self.conf.add_section('misc')
+self.fixed_dump_order = self.get_opt_in_overrides_or_default("misc", 
"fixeddumporder", 0)
+self.fixed_dump_order = int(self.fixed_dump_order, 0)
+
 def parse_conffile_globally(self):
 
 if not self.conf.has_section('database'):
diff --git a/xmldumps-backup/dumps/utils.py b/xmldumps-backup/dumps/utils.py
index c782931..9c2f040 100644
--- a/xmldumps-backup/dumps/utils.py
+++ b/xmldumps-backup/dumps/utils.py
@@ -17,7 +17,7 @@
 
 class MiscUtils(object):
 @staticmethod
-def db_list(path):
+def db_list(path, nosort=False):
 """Read database list from a file"""
 if not path:
 return []
@@ -28,7 +28,8 @@
 if line != "":
 dbs.append(line)
 infhandle.close()
-dbs = sorted(dbs)
+if not nosort:
+dbs = sorted(dbs)
 return dbs
 
 @staticmethod
diff --git a/xmldumps-backup/worker.py b/xmldumps-backup/worker.py

[MediaWiki-commits] [Gerrit] operations/puppet[production]: apachedir is available to dumps cron jobs via a bash script, ...

[ArielGlenn (Code Review)]

ArielGlenn has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/398106 )

Change subject: apachedir is available to dumps cron jobs via a bash script, 
use it
..


apachedir is available to dumps cron jobs via a bash script, use it

Don't do the extra fork out to get and verify the path from the
dumps config file

Change-Id: I00dd50bfb6a2d7d6b2f11ec30ac1f951c874a1b2
---
M modules/snapshot/files/cron/dump-global-blocks.sh
M modules/snapshot/files/cron/dumpcategoriesrdf.sh
M modules/snapshot/files/cron/dumpcirrussearch.sh
M modules/snapshot/files/cron/dumpcontentxlation.sh
M modules/snapshot/files/cron/wikidatadumps-shared.sh
5 files changed, 13 insertions(+), 18 deletions(-)

Approvals:
  ArielGlenn: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/snapshot/files/cron/dump-global-blocks.sh 
b/modules/snapshot/files/cron/dump-global-blocks.sh
index 4f5416a..c3d2baf 100644
--- a/modules/snapshot/files/cron/dump-global-blocks.sh
+++ b/modules/snapshot/files/cron/dump-global-blocks.sh
@@ -97,14 +97,13 @@
 fi
 done
 
-args="wiki:dir;tools:gzip,mysqldump"
+args="tools:gzip,mysqldump"
 results=`python "${repodir}/getconfigvals.py" --configfile "$configfile" 
--args "$args"`
 
-apachedir=`getsetting "$results" "wiki" "dir"` || exit 1
 gzip=`getsetting "$results" "tools" "gzip"` || exit 1
 mysqldump=`getsetting "$results" "tools" "mysqldump"` || exit 1
 
-for settingname in "apachedir" "gzip" "mysqldump"; do
+for settingname in "gzip" "mysqldump"; do
 checkval "$settingname" "${!settingname}"
 done
 
diff --git a/modules/snapshot/files/cron/dumpcategoriesrdf.sh 
b/modules/snapshot/files/cron/dumpcategoriesrdf.sh
index d55a692..898d088 100755
--- a/modules/snapshot/files/cron/dumpcategoriesrdf.sh
+++ b/modules/snapshot/files/cron/dumpcategoriesrdf.sh
@@ -52,14 +52,13 @@
exit 1
 fi
 
-args="wiki:dir,privatelist;tools:gzip"
+args="wiki:privatelist;tools:gzip"
 results=`python "${repodir}/getconfigvals.py" --configfile "$configFile" 
--args "$args"`
 
-deployDir=`getsetting "$results" "wiki" "dir"` || exit 1
 privateList=`getsetting "$results" "wiki" "privatelist"` || exit 1
 gzip=`getsetting "$results" "tools" "gzip"` || exit 1
 
-for settingname in "deployDir" "gzip" "privateList"; do
+for settingname in "gzip" "privateList"; do
 checkval "$settingname" "${!settingname}"
 done
 
@@ -67,7 +66,7 @@
 targetDirBase="${cronsdir}/categoriesrdf"
 targetDir="${targetDirBase}/${today}"
 timestampsDir="${targetDirBase}/lastdump"
-multiVersionScript="${deployDir}/multiversion/MWScript.php"
+multiVersionScript="${apachedir}/multiversion/MWScript.php"
 
 # create todays folder
 if [ "$dryrun" == "true" ]; then
diff --git a/modules/snapshot/files/cron/dumpcirrussearch.sh 
b/modules/snapshot/files/cron/dumpcirrussearch.sh
index 9a7eeb6..9731e23 100644
--- a/modules/snapshot/files/cron/dumpcirrussearch.sh
+++ b/modules/snapshot/files/cron/dumpcirrussearch.sh
@@ -40,22 +40,21 @@
exit 1
 fi
 
-args="wiki:dir,dblist,privatelist;tools:gzip"
+args="wiki:dblist,privatelist;tools:gzip"
 results=`python "${repodir}/getconfigvals.py" --configfile "$configFile" 
--args "$args"`
 
-deployDir=`getsetting "$results" "wiki" "dir"` || exit 1
 allList=`getsetting "$results" "wiki" "dblist"` || exit 1
 privateList=`getsetting "$results" "wiki" "privatelist"` || exit 1
 gzip=`getsetting "$results" "tools" "gzip"` || exit 1
 
-for settingname in "deployDir" "allList" "privateList" "gzip"; do
+for settingname in "allList" "privateList" "gzip"; do
 checkval "$settingname" "${!settingname}"
 done
 
 today=$(date +'%Y%m%d')
 targetDirBase="${cronsdir}/cirrussearch"
 targetDir="$targetDirBase/$today"
-multiVersionScript="$deployDir/multiversion/MWScript.php"
+multiVersionScript="${apachedir}/multiversion/MWScript.php"
 
 # create todays folder
 if [ "$dryrun" == "true" ]; then
diff --git a/modules/snapshot/files/cron/dumpcontentxlation.sh 
b/modules/snapshot/files/cron/dumpcontentxlation.sh
index c580a7e..13191bf 100644
--- a/modules/snapshot/files/cron/dumpcontentxlation.sh
+++ b/modules/snapshot/files/cron/dumpcontentxlation.sh
@@ -59,13 +59,12 @@
 # Get config settings
 #
 
-args="wiki:dir;tools:php"
+args="tools:php"
 results=`python "${repodir}/getconfigvals.py" --configfile "$configfile" 
--args "$args"`
 
-apachedir=`getsetting "$results" "wiki" "dir"`
 php=`getsetting "$results" "tools" "php"`
 
-for settingname in "apachedir" "php"; do
+for settingname in "php"; do
 checkval "$settingname" "${!settingname}"
 done
 
diff --git a/modules/snapshot/files/cron/wikidatadumps-shared.sh 
b/modules/snapshot/files/cron/wikidatadumps-shared.sh
index 68d9b33..1dac803 100644
--- a/modules/snapshot/files/cron/wikidatadumps-shared.sh
+++ b/modules/snapshot/files/cron/wikidatadumps-shared.sh
@@ -14,20 +14,19 @@
 today=`date +'%Y%m%d'`
 daysToKeep=70
 
-args="wiki:dir;output:temp"

[MediaWiki-commits] [Gerrit] operations/puppet[production]: clean up directory setup manifests for dumps nfs and web ser...

[ArielGlenn (Code Review)]

ArielGlenn has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/398095 )

Change subject: clean up directory setup manifests for dumps nfs and web servers
..


clean up directory setup manifests for dumps nfs and web servers

There are some dirs common to both; move them out into a separate
manifest.  Label everything else so it's clear which directories
are declared where and why.

Change-Id: I8542b5d48afa5914173a26dda37a7fcbd2c9d86a
---
M modules/dumps/manifests/generation/server/dirs.pp
A modules/dumps/manifests/server_dirs.pp
M modules/dumps/manifests/web/dirs.pp
3 files changed, 35 insertions(+), 41 deletions(-)

Approvals:
  ArielGlenn: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/dumps/manifests/generation/server/dirs.pp 
b/modules/dumps/manifests/generation/server/dirs.pp
index 1c29f28..0765360 100644
--- a/modules/dumps/manifests/generation/server/dirs.pp
+++ b/modules/dumps/manifests/generation/server/dirs.pp
@@ -6,6 +6,14 @@
 $user= undef,
 $group   = undef,
 ) {
+class {'dumps::server_dirs':
+datadir => $datadir,
+xmldumpsdir => $xmldumpsdir,
+miscdatasetsdir => $miscdatasetsdir,
+user=> $user,
+group   => $group,
+}
+
 # Directories where dumps of any type are generated
 # This list is not for one-off directories, nor for
 # directories with incoming rsyncs of datasets
@@ -16,26 +24,23 @@
 $medialistsdir= "${miscdatasetsdir}/imageinfo"
 $incrsdir = "${miscdatasetsdir}/incr"
 $mediatitlesdir   = "${miscdatasetsdir}/mediatitles"
-$othermiscdir = "${miscdatasetsdir}/misc"
 $pagetitlesdir= "${miscdatasetsdir}/pagetitles"
-$othertestfilesdir= "${miscdatasetsdir}/testfiles"
 $otherwikibasedir = "${miscdatasetsdir}/wikibase"
 $otherwikibasewikidatadir = "${miscdatasetsdir}/wikibase/wikidatawiki"
 $otherwikidatadir = "${miscdatasetsdir}/wikidata"
 
-# top level directories for various dumps/datasets
-file { [ $datadir, $xmldumpsdir, $miscdatasetsdir, $tempdir ]:
+# top level directories for various dumps/datasets, on generation hosts 
only
+file { $tempdir:
 ensure => 'directory',
 mode   => '0755',
 owner  => $user,
 group  => $group,
 }
 
-# subdirs for various dumps
+# subdirs for various generated dumps
 file { [ $cirrussearchdir, $xlationdir, $categoriesrdfdir,
 $globalblocksdir, $medialistsdir, $incrsdir,
-$mediatitlesdir, $othermiscdir, $pagetitlesdir,
-$othertestfilesdir ]:
+$mediatitlesdir, $pagetitlesdir]:
 
 ensure => 'directory',
 mode   => '0755',
diff --git a/modules/dumps/manifests/server_dirs.pp 
b/modules/dumps/manifests/server_dirs.pp
new file mode 100644
index 000..7b0b3d64
--- /dev/null
+++ b/modules/dumps/manifests/server_dirs.pp
@@ -0,0 +1,15 @@
+class dumps::server_dirs(
+$datadir = undef,
+$xmldumpsdir = undef,
+$miscdatasetsdir = undef,
+$user= undef,
+$group   = undef,
+) {
+# top level directories for various dumps/datasets
+file { [$datadir, $xmldumpsdir, $miscdatasetsdir]:
+ensure => 'directory',
+mode   => '0755',
+owner  => $user,
+group  => $group,
+}
+}
diff --git a/modules/dumps/manifests/web/dirs.pp 
b/modules/dumps/manifests/web/dirs.pp
index beb4cac..f691999 100644
--- a/modules/dumps/manifests/web/dirs.pp
+++ b/modules/dumps/manifests/web/dirs.pp
@@ -5,46 +5,20 @@
 $user = undef,
 $group = undef,
 ) {
-# Please note that this is incomplete, but new directories
-# should be defined in puppet (here).
+class {'dumps::server_dirs':
+datadir => $datadir,
+xmldumpsdir => $xmldumpsdir,
+miscdatasetsdir => $miscdatasetsdir,
+user=> $user,
+group   => $group,
+}
+
 $analyticsdir = "${miscdatasetsdir}/analytics"
 $othermiscdir = "${miscdatasetsdir}/misc"
 $othertestfilesdir= "${miscdatasetsdir}/testfiles"
 $miscdatasetsdir_wikidata_legacy = "${miscdatasetsdir}/wikidata"
 $miscdatasetsdir_wikibase= "${miscdatasetsdir}/wikibase/"
 $relative_wikidatawiki= 'other/wikibase/wikidatawiki'
-$xlationdir   = "${miscdatasetsdir}/contenttranslation"
-$cirrussearchdir  = "${miscdatasetsdir}/cirrussearch"
-$medialistsdir= "${miscdatasetsdir}/imageinfo"
-$pagetitlesdir= "${miscdatasetsdir}/pagetitles"
-$mediatitlesdir   = "${miscdatasetsdir}/mediatitles"
-$categoriesrdf= "${miscdatasetsdir}/categoriesrdf"
-
-# top level dir
-file { 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: rename 'otherdir' in the dumps modules

[ArielGlenn (Code Review)]

ArielGlenn has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/398034 )

Change subject: rename 'otherdir' in the dumps modules
..


rename 'otherdir' in the dumps modules

On some hosts the path for misc dumps and datasets ends in .../other/
and on some hosts not, and it's a horrible name anyways. Other what?
we leave the paths as they are, but change the var names everywhere
to something with meaning.

Also remove more hardcoded paths in the manifests.

Change-Id: I5472a464c2db2335c6e692ff1053c754889ab18d
---
M modules/dumps/files/web/html/analytics_index.html
M modules/dumps/manifests/copying/labs.pp
M modules/dumps/manifests/generation/server/dirs.pp
M modules/dumps/manifests/rsync/media.pp
M modules/dumps/manifests/rsync/pagecounts_ez.pp
M modules/dumps/manifests/rsync/phab_dump.pp
M modules/dumps/manifests/rsync/public.pp
M modules/dumps/manifests/rsync/slowparse_logs.pp
M modules/dumps/manifests/web/dirs.pp
M modules/dumps/manifests/web/fetches/kiwix.pp
M modules/dumps/manifests/web/fetches/stats.pp
M modules/dumps/manifests/web/fetches/wikitech_dumps.pp
M modules/dumps/manifests/web/html.pp
M modules/dumps/manifests/web/xmldumps.pp
M modules/dumps/manifests/web/xmldumps_active.pp
M modules/dumps/templates/copying/labs-rsync-cron.sh.erb
M modules/dumps/templates/rsync/rsyncd.conf.dumps_to_public.erb
M modules/dumps/templates/rsync/rsyncd.conf.media.erb
M modules/dumps/templates/rsync/rsyncd.conf.pagecounts_ez.erb
M modules/dumps/templates/rsync/rsyncd.conf.phab_dump.erb
M modules/dumps/templates/rsync/rsyncd.conf.slowparse_logs.erb
M modules/profile/manifests/dumps/fetcher.pp
M modules/profile/manifests/dumps/generation/server/fallback.pp
M modules/profile/manifests/dumps/generation/server/primary.pp
M modules/profile/manifests/dumps/rsyncer.pp
M modules/profile/manifests/dumps/web/xmldumps_active.pp
M modules/profile/manifests/dumps/web/xmldumps_fallback.pp
27 files changed, 136 insertions(+), 132 deletions(-)

Approvals:
  ArielGlenn: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/dumps/files/web/html/analytics_index.html 
b/modules/dumps/files/web/html/analytics_index.html
index 2f1f6df..7f9ba98 100644
--- a/modules/dumps/files/web/html/analytics_index.html
+++ b/modules/dumps/files/web/html/analytics_index.html
@@ -43,7 +43,10 @@
 
 
 
-Clickstream: (referer, resource) pairs 
extracted from the request logs of Wikipedia. Please visit the https://meta.wikimedia.org/wiki/Research:Wikipedia_clickstream;>Clickstream
 mediawiki page for detailed information, and the https://figshare.com/articles/Wikipedia_Clickstream/1305770;>Clickstream 
figshare page for correctly reference this dataset. Available as:
+Clickstream: (referer, resource) pairs 
extracted from the request logs of Wikipedia. Please visit
+   the https://meta.wikimedia.org/wiki/Research:Wikipedia_clickstream;>Clickstream
 mediawiki page for detailed
+   information, and the https://figshare.com/articles/Wikipedia_Clickstream/1305770;>Clickstream 
figshare page for
+   correctly reference this dataset. Available as:
 
 Monthly generated 
clickstream for wikipedia in English, Russian, German, Spanish and 
Japanese.
 
diff --git a/modules/dumps/manifests/copying/labs.pp 
b/modules/dumps/manifests/copying/labs.pp
index 738803e..2d7061c 100644
--- a/modules/dumps/manifests/copying/labs.pp
+++ b/modules/dumps/manifests/copying/labs.pp
@@ -1,7 +1,7 @@
 class dumps::copying::labs(
 $labhost = undef,
 $xmldumpsdir = undef,
-$otherdir = undef,
+$miscdatasetsdir = undef,
 ) {
 file { '/usr/local/bin/wmfdumpsmirror.py':
 ensure => 'present',
@@ -20,7 +20,7 @@
 user=> 'root',
 minute  => '50',
 hour=> '3',
-command => "/usr/local/sbin/labs-rsync-cron.sh ${labhost} 
${xmldumpsdir} ${otherdir}",
+command => "/usr/local/sbin/labs-rsync-cron.sh ${labhost} 
${xmldumpsdir} ${miscdatasetsdir}",
 environment => 'MAILTO=ops-du...@wikimedia.org',
 require => File['/usr/local/bin/wmfdumpsmirror.py',
 '/usr/local/sbin/labs-rsync-cron.sh'],
diff --git a/modules/dumps/manifests/generation/server/dirs.pp 
b/modules/dumps/manifests/generation/server/dirs.pp
index f6608d7..1c29f28 100644
--- a/modules/dumps/manifests/generation/server/dirs.pp
+++ b/modules/dumps/manifests/generation/server/dirs.pp
@@ -1,30 +1,30 @@
 class dumps::generation::server::dirs(
-$datadir = undef,
-$xmldumpsdir = undef,
-$tempdir = undef,
-$otherdir= undef,
-$user= undef,
-$group   = undef,
+$datadir = undef,
+$xmldumpsdir = undef,
+$tempdir = undef,
+$miscdatasetsdir = undef,
+$user= 

[MediaWiki-commits] [Gerrit] operations/dumps[master]: config setting to permit a list of wikis to be dumped in a s...

[ArielGlenn (Code Review)]

ArielGlenn has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/398861 )

Change subject: config setting to permit a list of wikis to be dumped in a 
specific order
..

config setting to permit a list of wikis to be dumped in a specific order

We don't want these for all wikis but it can be useful for small lists.

Example the big wikis that dump via 4 processes at a time.  Some of these
take days longer than others; if we start those first, multiple other wikis
will run to completion on other cores, while the first ones chug along.
If the order is by longet to dump, these big wikis often wind up
starting near the end ofthe list, and run along by themselves after
everything else is completed.

We can't really just expand the configs of these big slow wikis
so they use a pile more processors at once, that's not ok for the db
servers. (enwiki, wikidatawiki, yes, but not the rest.) So, move them
to the front of the queue.

Change-Id: I494ed57363b1ddfe0e10be0aed25facb7ca8a364
---
M xmldumps-backup/defaults.conf
M xmldumps-backup/doc/README.config
M xmldumps-backup/dumps/WikiDump.py
M xmldumps-backup/dumps/utils.py
M xmldumps-backup/worker.py
5 files changed, 43 insertions(+), 9 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/dumps 
refs/changes/61/398861/1

diff --git a/xmldumps-backup/defaults.conf b/xmldumps-backup/defaults.conf
index d17cbbb..e95ac47 100644
--- a/xmldumps-backup/defaults.conf
+++ b/xmldumps-backup/defaults.conf
@@ -76,3 +76,6 @@
 orderrevs=0
 minpages=1
 maxrevs=5
+
+[misc]
+fixeddumporder=0
\ No newline at end of file
diff --git a/xmldumps-backup/doc/README.config 
b/xmldumps-backup/doc/README.config
index d4d76f4..6ccd705 100644
--- a/xmldumps-backup/doc/README.config
+++ b/xmldumps-backup/doc/README.config
@@ -245,6 +245,15 @@
 The above options do not have to be specified in the config file,
 since default values are provided.
 
+=== Misc (i.e.: [misc])
+fixed_dump_order -- set this to a non-zero integer to enable dumps
+of wikis in the specified db list to be dumped
+in the order listed
+   Default value: 0 (wiki dumped longest ago goes first)
+
+The above options do not have to be specified in the config file,
+since default values are provided.
+
 === Per-wiki configuration
 The following settings may be overriden for specific wikis by specifying
 their name (the name of the db in the database) as a section header,
diff --git a/xmldumps-backup/dumps/WikiDump.py 
b/xmldumps-backup/dumps/WikiDump.py
index 80115ec..6aeca6a 100644
--- a/xmldumps-backup/dumps/WikiDump.py
+++ b/xmldumps-backup/dumps/WikiDump.py
@@ -171,9 +171,8 @@
 globals like entries in 'wiki' or 'output' that can
 be overriden by a specific named section
 """
-self.db_list = MiscUtils.db_list(self.get_opt_in_overrides_or_default(
-"wiki", "dblist", 0))
-
+self.db_list_unsorted = 
MiscUtils.db_list(self.get_opt_in_overrides_or_default(
+"wiki", "dblist", 0), nosort=True)
 # permit comma-separated list of files so that eg some script
 # can skip all private and/or closed wikis in addition to some
 # other exclusion list
@@ -191,7 +190,8 @@
 self.apijobs = self.get_opt_in_overrides_or_default(
 "wiki", "apijobs", 0)
 
-self.db_list = list(set(self.db_list) - set(self.skip_db_list))
+self.db_list_unsorted = [dbname for dbname in self.db_list_unsorted if 
dbname not in self.skip_db_list]
+self.db_list = sorted(self.db_list_unsorted)
 
 if not self.conf.has_section('output'):
 self.conf.add_section('output')
@@ -206,6 +206,11 @@
 self.fileperms = self.get_opt_in_overrides_or_default("output", 
"fileperms", 0)
 self.fileperms = int(self.fileperms, 0)
 
+if not self.conf.has_section('misc'):
+self.conf.add_section('misc')
+self.fixed_dump_order = self.get_opt_in_overrides_or_default("misc", 
"fixeddumporder", 0)
+self.fixed_dump_order = int(self.fixed_dump_order, 0)
+
 def parse_conffile_globally(self):
 
 if not self.conf.has_section('database'):
diff --git a/xmldumps-backup/dumps/utils.py b/xmldumps-backup/dumps/utils.py
index c782931..9c2f040 100644
--- a/xmldumps-backup/dumps/utils.py
+++ b/xmldumps-backup/dumps/utils.py
@@ -17,7 +17,7 @@
 
 class MiscUtils(object):
 @staticmethod
-def db_list(path):
+def db_list(path, nosort=False):
 """Read database list from a file"""
 if not path:
 return []
@@ -28,7 +28,8 @@
 if line != "":
 dbs.append(line)
 infhandle.close()
-dbs = sorted(dbs)
+if not nosort:
+dbs = sorted(dbs)
 return dbs
 
 @staticmethod
diff --git a/xmldumps-backup/worker.py b/xmldumps-backup/worker.py
index 507de7c..9478452 100644

[MediaWiki-commits] [Gerrit] operations/dumps[master]: Revert "use cat to recombine gzipped files together"

[ArielGlenn (Code Review)]

ArielGlenn has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/398637 )

Change subject: Revert "use cat to recombine gzipped files together"
..


Revert "use cat to recombine gzipped files together"

This reverts commit faf7de86d3dd98b4e5bd59809c5938050fbec42d.

A nice idea but we would wind up with multiple site headers in the concatenated 
file, changing the contents.

Change-Id: I7da9a95c0c817b529f40ce024235c1afbe85600f
---
M xmldumps-backup/dumps/recombinejobs.py
1 file changed, 9 insertions(+), 9 deletions(-)

Approvals:
  ArielGlenn: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/xmldumps-backup/dumps/recombinejobs.py 
b/xmldumps-backup/dumps/recombinejobs.py
index ce5e580..12c49e5 100644
--- a/xmldumps-backup/dumps/recombinejobs.py
+++ b/xmldumps-backup/dumps/recombinejobs.py
@@ -129,11 +129,11 @@
 if not len(input_dfnames):
 self.set_status("failed")
 raise BackupError("No input files for %s found" % self.name())
-if not exists(runner.wiki.config.cat):
-raise BackupError("cat command %s not found" % 
runner.wiki.config.cat)
-compression_command = runner.wiki.config.cat
-compression_command = "%s > " % runner.wiki.config.cat
-uncompression_command = ["%s" % runner.wiki.config.cat]
+if not exists(runner.wiki.config.gzip):
+raise BackupError("gzip command %s not found" % 
runner.wiki.config.gzip)
+compression_command = runner.wiki.config.gzip
+compression_command = "%s > " % runner.wiki.config.gzip
+uncompression_command = ["%s" % runner.wiki.config.gzip, "-dc"]
 recombine_command_string = self.build_recombine_command_string(
 runner, input_dfnames, output_dfname, compression_command, 
uncompression_command)
 recombine_command = [recombine_command_string]
@@ -301,10 +301,10 @@
 if not len(input_dfnames):
 self.set_status("failed")
 raise BackupError("No input files for %s found" % self.name())
-if not exists(runner.wiki.config.cat):
-raise BackupError("cat command %s not found" % 
runner.wiki.config.cat)
-compression_command = "%s > " % runner.wiki.config.cat
-uncompression_command = ["%s" % runner.wiki.config.cat]
+if not exists(runner.wiki.config.gzip):
+raise BackupError("gzip command %s not found" % 
runner.wiki.config.gzip)
+compression_command = "%s > " % runner.wiki.config.gzip
+uncompression_command = ["%s" % runner.wiki.config.gzip, "-dc"]
 recombine_command_string = self.build_recombine_command_string(
 runner, input_dfnames, output_dfname, compression_command,
 uncompression_command, "")

-- 
To view, visit https://gerrit.wikimedia.org/r/398637
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I7da9a95c0c817b529f40ce024235c1afbe85600f
Gerrit-PatchSet: 2
Gerrit-Project: operations/dumps
Gerrit-Branch: master
Gerrit-Owner: ArielGlenn 
Gerrit-Reviewer: ArielGlenn 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/dumps[master]: Revert "use cat to recombine gzipped files together"

[ArielGlenn (Code Review)]

Hello jenkins-bot,

I'd like you to do a code review.  Please visit

https://gerrit.wikimedia.org/r/398637

to review the following change.


Change subject: Revert "use cat to recombine gzipped files together"
..

Revert "use cat to recombine gzipped files together"

This reverts commit faf7de86d3dd98b4e5bd59809c5938050fbec42d.

A nice idea but we woudl wind up with multiple site headers in the concatenated 
file, changing the contents.

Change-Id: I7da9a95c0c817b529f40ce024235c1afbe85600f
---
M xmldumps-backup/dumps/recombinejobs.py
1 file changed, 9 insertions(+), 9 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/dumps 
refs/changes/37/398637/1

diff --git a/xmldumps-backup/dumps/recombinejobs.py 
b/xmldumps-backup/dumps/recombinejobs.py
index ce5e580..12c49e5 100644
--- a/xmldumps-backup/dumps/recombinejobs.py
+++ b/xmldumps-backup/dumps/recombinejobs.py
@@ -129,11 +129,11 @@
 if not len(input_dfnames):
 self.set_status("failed")
 raise BackupError("No input files for %s found" % self.name())
-if not exists(runner.wiki.config.cat):
-raise BackupError("cat command %s not found" % 
runner.wiki.config.cat)
-compression_command = runner.wiki.config.cat
-compression_command = "%s > " % runner.wiki.config.cat
-uncompression_command = ["%s" % runner.wiki.config.cat]
+if not exists(runner.wiki.config.gzip):
+raise BackupError("gzip command %s not found" % 
runner.wiki.config.gzip)
+compression_command = runner.wiki.config.gzip
+compression_command = "%s > " % runner.wiki.config.gzip
+uncompression_command = ["%s" % runner.wiki.config.gzip, "-dc"]
 recombine_command_string = self.build_recombine_command_string(
 runner, input_dfnames, output_dfname, compression_command, 
uncompression_command)
 recombine_command = [recombine_command_string]
@@ -301,10 +301,10 @@
 if not len(input_dfnames):
 self.set_status("failed")
 raise BackupError("No input files for %s found" % self.name())
-if not exists(runner.wiki.config.cat):
-raise BackupError("cat command %s not found" % 
runner.wiki.config.cat)
-compression_command = "%s > " % runner.wiki.config.cat
-uncompression_command = ["%s" % runner.wiki.config.cat]
+if not exists(runner.wiki.config.gzip):
+raise BackupError("gzip command %s not found" % 
runner.wiki.config.gzip)
+compression_command = "%s > " % runner.wiki.config.gzip
+uncompression_command = ["%s" % runner.wiki.config.gzip, "-dc"]
 recombine_command_string = self.build_recombine_command_string(
 runner, input_dfnames, output_dfname, compression_command,
 uncompression_command, "")

-- 
To view, visit https://gerrit.wikimedia.org/r/398637
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I7da9a95c0c817b529f40ce024235c1afbe85600f
Gerrit-PatchSet: 1
Gerrit-Project: operations/dumps
Gerrit-Branch: master
Gerrit-Owner: ArielGlenn 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/dumps[master]: use cat to recombine gzipped files together

[ArielGlenn (Code Review)]

ArielGlenn has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/398634 )

Change subject: use cat to recombine gzipped files together
..


use cat to recombine gzipped files together

Most if not all tools should already support uncompressing such files.

Bug: T182572
Change-Id: I650ceb1caf5963a1b5eb6a955fcf19a14096e1c6
---
M xmldumps-backup/dumps/recombinejobs.py
1 file changed, 9 insertions(+), 9 deletions(-)

Approvals:
  ArielGlenn: Looks good to me, approved



diff --git a/xmldumps-backup/dumps/recombinejobs.py 
b/xmldumps-backup/dumps/recombinejobs.py
index 12c49e5..ce5e580 100644
--- a/xmldumps-backup/dumps/recombinejobs.py
+++ b/xmldumps-backup/dumps/recombinejobs.py
@@ -129,11 +129,11 @@
 if not len(input_dfnames):
 self.set_status("failed")
 raise BackupError("No input files for %s found" % self.name())
-if not exists(runner.wiki.config.gzip):
-raise BackupError("gzip command %s not found" % 
runner.wiki.config.gzip)
-compression_command = runner.wiki.config.gzip
-compression_command = "%s > " % runner.wiki.config.gzip
-uncompression_command = ["%s" % runner.wiki.config.gzip, "-dc"]
+if not exists(runner.wiki.config.cat):
+raise BackupError("cat command %s not found" % 
runner.wiki.config.cat)
+compression_command = runner.wiki.config.cat
+compression_command = "%s > " % runner.wiki.config.cat
+uncompression_command = ["%s" % runner.wiki.config.cat]
 recombine_command_string = self.build_recombine_command_string(
 runner, input_dfnames, output_dfname, compression_command, 
uncompression_command)
 recombine_command = [recombine_command_string]
@@ -301,10 +301,10 @@
 if not len(input_dfnames):
 self.set_status("failed")
 raise BackupError("No input files for %s found" % self.name())
-if not exists(runner.wiki.config.gzip):
-raise BackupError("gzip command %s not found" % 
runner.wiki.config.gzip)
-compression_command = "%s > " % runner.wiki.config.gzip
-uncompression_command = ["%s" % runner.wiki.config.gzip, "-dc"]
+if not exists(runner.wiki.config.cat):
+raise BackupError("cat command %s not found" % 
runner.wiki.config.cat)
+compression_command = "%s > " % runner.wiki.config.cat
+uncompression_command = ["%s" % runner.wiki.config.cat]
 recombine_command_string = self.build_recombine_command_string(
 runner, input_dfnames, output_dfname, compression_command,
 uncompression_command, "")

-- 
To view, visit https://gerrit.wikimedia.org/r/398634
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I650ceb1caf5963a1b5eb6a955fcf19a14096e1c6
Gerrit-PatchSet: 1
Gerrit-Project: operations/dumps
Gerrit-Branch: master
Gerrit-Owner: ArielGlenn 
Gerrit-Reviewer: ArielGlenn 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/dumps[master]: use cat to recombine gzipped files together

[ArielGlenn (Code Review)]

ArielGlenn has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/398634 )

Change subject: use cat to recombine gzipped files together
..

use cat to recombine gzipped files together

Most if not all tools should already support uncompressing such files.

Bug: T182572
Change-Id: I650ceb1caf5963a1b5eb6a955fcf19a14096e1c6
---
M xmldumps-backup/dumps/recombinejobs.py
1 file changed, 9 insertions(+), 9 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/dumps 
refs/changes/34/398634/1

diff --git a/xmldumps-backup/dumps/recombinejobs.py 
b/xmldumps-backup/dumps/recombinejobs.py
index 12c49e5..ce5e580 100644
--- a/xmldumps-backup/dumps/recombinejobs.py
+++ b/xmldumps-backup/dumps/recombinejobs.py
@@ -129,11 +129,11 @@
 if not len(input_dfnames):
 self.set_status("failed")
 raise BackupError("No input files for %s found" % self.name())
-if not exists(runner.wiki.config.gzip):
-raise BackupError("gzip command %s not found" % 
runner.wiki.config.gzip)
-compression_command = runner.wiki.config.gzip
-compression_command = "%s > " % runner.wiki.config.gzip
-uncompression_command = ["%s" % runner.wiki.config.gzip, "-dc"]
+if not exists(runner.wiki.config.cat):
+raise BackupError("cat command %s not found" % 
runner.wiki.config.cat)
+compression_command = runner.wiki.config.cat
+compression_command = "%s > " % runner.wiki.config.cat
+uncompression_command = ["%s" % runner.wiki.config.cat]
 recombine_command_string = self.build_recombine_command_string(
 runner, input_dfnames, output_dfname, compression_command, 
uncompression_command)
 recombine_command = [recombine_command_string]
@@ -301,10 +301,10 @@
 if not len(input_dfnames):
 self.set_status("failed")
 raise BackupError("No input files for %s found" % self.name())
-if not exists(runner.wiki.config.gzip):
-raise BackupError("gzip command %s not found" % 
runner.wiki.config.gzip)
-compression_command = "%s > " % runner.wiki.config.gzip
-uncompression_command = ["%s" % runner.wiki.config.gzip, "-dc"]
+if not exists(runner.wiki.config.cat):
+raise BackupError("cat command %s not found" % 
runner.wiki.config.cat)
+compression_command = "%s > " % runner.wiki.config.cat
+uncompression_command = ["%s" % runner.wiki.config.cat]
 recombine_command_string = self.build_recombine_command_string(
 runner, input_dfnames, output_dfname, compression_command,
 uncompression_command, "")

-- 
To view, visit https://gerrit.wikimedia.org/r/398634
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I650ceb1caf5963a1b5eb6a955fcf19a14096e1c6
Gerrit-PatchSet: 1
Gerrit-Project: operations/dumps
Gerrit-Branch: master
Gerrit-Owner: ArielGlenn 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: apachedir is available to dumps cron jobs via a bash script, ...

[ArielGlenn (Code Review)]

ArielGlenn has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/398106 )

Change subject: apachedir is available to dumps cron jobs via a bash script, 
use it
..

apachedir is available to dumps cron jobs via a bash script, use it

Don't do the extra fork out to get and verify the path from the
dumps config file

Change-Id: I00dd50bfb6a2d7d6b2f11ec30ac1f951c874a1b2
---
M modules/snapshot/files/cron/dump-global-blocks.sh
M modules/snapshot/files/cron/dumpcategoriesrdf.sh
M modules/snapshot/files/cron/dumpcirrussearch.sh
M modules/snapshot/files/cron/dumpcontentxlation.sh
M modules/snapshot/files/cron/wikidatadumps-shared.sh
5 files changed, 13 insertions(+), 18 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/06/398106/1

diff --git a/modules/snapshot/files/cron/dump-global-blocks.sh 
b/modules/snapshot/files/cron/dump-global-blocks.sh
index 4f5416a..c3d2baf 100644
--- a/modules/snapshot/files/cron/dump-global-blocks.sh
+++ b/modules/snapshot/files/cron/dump-global-blocks.sh
@@ -97,14 +97,13 @@
 fi
 done
 
-args="wiki:dir;tools:gzip,mysqldump"
+args="tools:gzip,mysqldump"
 results=`python "${repodir}/getconfigvals.py" --configfile "$configfile" 
--args "$args"`
 
-apachedir=`getsetting "$results" "wiki" "dir"` || exit 1
 gzip=`getsetting "$results" "tools" "gzip"` || exit 1
 mysqldump=`getsetting "$results" "tools" "mysqldump"` || exit 1
 
-for settingname in "apachedir" "gzip" "mysqldump"; do
+for settingname in "gzip" "mysqldump"; do
 checkval "$settingname" "${!settingname}"
 done
 
diff --git a/modules/snapshot/files/cron/dumpcategoriesrdf.sh 
b/modules/snapshot/files/cron/dumpcategoriesrdf.sh
index d55a692..898d088 100755
--- a/modules/snapshot/files/cron/dumpcategoriesrdf.sh
+++ b/modules/snapshot/files/cron/dumpcategoriesrdf.sh
@@ -52,14 +52,13 @@
exit 1
 fi
 
-args="wiki:dir,privatelist;tools:gzip"
+args="wiki:privatelist;tools:gzip"
 results=`python "${repodir}/getconfigvals.py" --configfile "$configFile" 
--args "$args"`
 
-deployDir=`getsetting "$results" "wiki" "dir"` || exit 1
 privateList=`getsetting "$results" "wiki" "privatelist"` || exit 1
 gzip=`getsetting "$results" "tools" "gzip"` || exit 1
 
-for settingname in "deployDir" "gzip" "privateList"; do
+for settingname in "gzip" "privateList"; do
 checkval "$settingname" "${!settingname}"
 done
 
@@ -67,7 +66,7 @@
 targetDirBase="${cronsdir}/categoriesrdf"
 targetDir="${targetDirBase}/${today}"
 timestampsDir="${targetDirBase}/lastdump"
-multiVersionScript="${deployDir}/multiversion/MWScript.php"
+multiVersionScript="${apachedir}/multiversion/MWScript.php"
 
 # create todays folder
 if [ "$dryrun" == "true" ]; then
diff --git a/modules/snapshot/files/cron/dumpcirrussearch.sh 
b/modules/snapshot/files/cron/dumpcirrussearch.sh
index 9a7eeb6..9731e23 100644
--- a/modules/snapshot/files/cron/dumpcirrussearch.sh
+++ b/modules/snapshot/files/cron/dumpcirrussearch.sh
@@ -40,22 +40,21 @@
exit 1
 fi
 
-args="wiki:dir,dblist,privatelist;tools:gzip"
+args="wiki:dblist,privatelist;tools:gzip"
 results=`python "${repodir}/getconfigvals.py" --configfile "$configFile" 
--args "$args"`
 
-deployDir=`getsetting "$results" "wiki" "dir"` || exit 1
 allList=`getsetting "$results" "wiki" "dblist"` || exit 1
 privateList=`getsetting "$results" "wiki" "privatelist"` || exit 1
 gzip=`getsetting "$results" "tools" "gzip"` || exit 1
 
-for settingname in "deployDir" "allList" "privateList" "gzip"; do
+for settingname in "allList" "privateList" "gzip"; do
 checkval "$settingname" "${!settingname}"
 done
 
 today=$(date +'%Y%m%d')
 targetDirBase="${cronsdir}/cirrussearch"
 targetDir="$targetDirBase/$today"
-multiVersionScript="$deployDir/multiversion/MWScript.php"
+multiVersionScript="${apachedir}/multiversion/MWScript.php"
 
 # create todays folder
 if [ "$dryrun" == "true" ]; then
diff --git a/modules/snapshot/files/cron/dumpcontentxlation.sh 
b/modules/snapshot/files/cron/dumpcontentxlation.sh
index c580a7e..13191bf 100644
--- a/modules/snapshot/files/cron/dumpcontentxlation.sh
+++ b/modules/snapshot/files/cron/dumpcontentxlation.sh
@@ -59,13 +59,12 @@
 # Get config settings
 #
 
-args="wiki:dir;tools:php"
+args="tools:php"
 results=`python "${repodir}/getconfigvals.py" --configfile "$configfile" 
--args "$args"`
 
-apachedir=`getsetting "$results" "wiki" "dir"`
 php=`getsetting "$results" "tools" "php"`
 
-for settingname in "apachedir" "php"; do
+for settingname in "php"; do
 checkval "$settingname" "${!settingname}"
 done
 
diff --git a/modules/snapshot/files/cron/wikidatadumps-shared.sh 
b/modules/snapshot/files/cron/wikidatadumps-shared.sh
index 68d9b33..1dac803 100644
--- a/modules/snapshot/files/cron/wikidatadumps-shared.sh
+++ b/modules/snapshot/files/cron/wikidatadumps-shared.sh
@@ -14,20 +14,19 @@
 today=`date +'%Y%m%d'`
 daysToKeep=70
 
-args="wiki:dir;output:temp"

[MediaWiki-commits] [Gerrit] operations/puppet[production]: clean up directory setup manifests for dumps nfs and web ser...

[ArielGlenn (Code Review)]

ArielGlenn has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/398095 )

Change subject: clean up directory setup manifests for dumps nfs and web servers
..

clean up directory setup manifests for dumps nfs and web servers

There are some dirs common to both; move them out into a separate
manifest.  Label everything else so it's clear which directories
are declared where and why.

Change-Id: I8542b5d48afa5914173a26dda37a7fcbd2c9d86a
---
M modules/dumps/manifests/generation/server/dirs.pp
A modules/dumps/manifests/server_dirs.pp
M modules/dumps/manifests/web/dirs.pp
3 files changed, 35 insertions(+), 41 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/95/398095/1

diff --git a/modules/dumps/manifests/generation/server/dirs.pp 
b/modules/dumps/manifests/generation/server/dirs.pp
index 1c29f28..0765360 100644
--- a/modules/dumps/manifests/generation/server/dirs.pp
+++ b/modules/dumps/manifests/generation/server/dirs.pp
@@ -6,6 +6,14 @@
 $user= undef,
 $group   = undef,
 ) {
+class {'dumps::server_dirs':
+datadir => $datadir,
+xmldumpsdir => $xmldumpsdir,
+miscdatasetsdir => $miscdatasetsdir,
+user=> $user,
+group   => $group,
+}
+
 # Directories where dumps of any type are generated
 # This list is not for one-off directories, nor for
 # directories with incoming rsyncs of datasets
@@ -16,26 +24,23 @@
 $medialistsdir= "${miscdatasetsdir}/imageinfo"
 $incrsdir = "${miscdatasetsdir}/incr"
 $mediatitlesdir   = "${miscdatasetsdir}/mediatitles"
-$othermiscdir = "${miscdatasetsdir}/misc"
 $pagetitlesdir= "${miscdatasetsdir}/pagetitles"
-$othertestfilesdir= "${miscdatasetsdir}/testfiles"
 $otherwikibasedir = "${miscdatasetsdir}/wikibase"
 $otherwikibasewikidatadir = "${miscdatasetsdir}/wikibase/wikidatawiki"
 $otherwikidatadir = "${miscdatasetsdir}/wikidata"
 
-# top level directories for various dumps/datasets
-file { [ $datadir, $xmldumpsdir, $miscdatasetsdir, $tempdir ]:
+# top level directories for various dumps/datasets, on generation hosts 
only
+file { $tempdir:
 ensure => 'directory',
 mode   => '0755',
 owner  => $user,
 group  => $group,
 }
 
-# subdirs for various dumps
+# subdirs for various generated dumps
 file { [ $cirrussearchdir, $xlationdir, $categoriesrdfdir,
 $globalblocksdir, $medialistsdir, $incrsdir,
-$mediatitlesdir, $othermiscdir, $pagetitlesdir,
-$othertestfilesdir ]:
+$mediatitlesdir, $pagetitlesdir]:
 
 ensure => 'directory',
 mode   => '0755',
diff --git a/modules/dumps/manifests/server_dirs.pp 
b/modules/dumps/manifests/server_dirs.pp
new file mode 100644
index 000..7b0b3d64
--- /dev/null
+++ b/modules/dumps/manifests/server_dirs.pp
@@ -0,0 +1,15 @@
+class dumps::server_dirs(
+$datadir = undef,
+$xmldumpsdir = undef,
+$miscdatasetsdir = undef,
+$user= undef,
+$group   = undef,
+) {
+# top level directories for various dumps/datasets
+file { [$datadir, $xmldumpsdir, $miscdatasetsdir]:
+ensure => 'directory',
+mode   => '0755',
+owner  => $user,
+group  => $group,
+}
+}
diff --git a/modules/dumps/manifests/web/dirs.pp 
b/modules/dumps/manifests/web/dirs.pp
index beb4cac..41f5cd1 100644
--- a/modules/dumps/manifests/web/dirs.pp
+++ b/modules/dumps/manifests/web/dirs.pp
@@ -5,46 +5,20 @@
 $user = undef,
 $group = undef,
 ) {
-# Please note that this is incomplete, but new directories
-# should be defined in puppet (here).
+class {'dumps::server_dirs':
+datadir = $datadir,
+xmldumpsdir = $xmldumpsdir,
+miscdatasetsdir = $miscdatasetsdir,
+user=> $user,
+group   => $group,
+}
+
 $analyticsdir = "${miscdatasetsdir}/analytics"
 $othermiscdir = "${miscdatasetsdir}/misc"
 $othertestfilesdir= "${miscdatasetsdir}/testfiles"
 $miscdatasetsdir_wikidata_legacy = "${miscdatasetsdir}/wikidata"
 $miscdatasetsdir_wikibase= "${miscdatasetsdir}/wikibase/"
 $relative_wikidatawiki= 'other/wikibase/wikidatawiki'
-$xlationdir   = "${miscdatasetsdir}/contenttranslation"
-$cirrussearchdir  = "${miscdatasetsdir}/cirrussearch"
-$medialistsdir= "${miscdatasetsdir}/imageinfo"
-$pagetitlesdir= "${miscdatasetsdir}/pagetitles"
-$mediatitlesdir   = "${miscdatasetsdir}/mediatitles"
-$categoriesrdf= "${miscdatasetsdir}/categoriesrdf"
-
-# top level dir
-file { 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: rename 'otherdir' in the dumps modules

[ArielGlenn (Code Review)]

ArielGlenn has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/398034 )

Change subject: rename 'otherdir' in the dumps modules
..

rename 'otherdir' in the dumps modules

On some hosts the path for misc dumps and datasets ends in .../other/
and on some hosts not, and it's a horrible name anyways. Other what?
we leave the paths as they are, but change the var names everywhere
to something with meaning.

Also remove one more hardcoded path in the manifests, no longer used.

Change-Id: I5472a464c2db2335c6e692ff1053c754889ab18d
---
M modules/dumps/manifests/copying/labs.pp
M modules/dumps/manifests/generation/server/dirs.pp
M modules/dumps/manifests/rsync/media.pp
M modules/dumps/manifests/rsync/pagecounts_ez.pp
M modules/dumps/manifests/rsync/phab_dump.pp
M modules/dumps/manifests/rsync/public.pp
M modules/dumps/manifests/rsync/slowparse_logs.pp
M modules/dumps/manifests/web/dirs.pp
M modules/dumps/manifests/web/fetches/kiwix.pp
M modules/dumps/manifests/web/fetches/stats.pp
M modules/dumps/manifests/web/html.pp
M modules/dumps/manifests/web/xmldumps.pp
M modules/dumps/manifests/web/xmldumps_active.pp
M modules/dumps/templates/copying/labs-rsync-cron.sh.erb
M modules/dumps/templates/rsync/rsyncd.conf.dumps_to_public.erb
M modules/dumps/templates/rsync/rsyncd.conf.media.erb
M modules/dumps/templates/rsync/rsyncd.conf.pagecounts_ez.erb
M modules/dumps/templates/rsync/rsyncd.conf.phab_dump.erb
M modules/dumps/templates/rsync/rsyncd.conf.slowparse_logs.erb
19 files changed, 82 insertions(+), 83 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/34/398034/1

diff --git a/modules/dumps/manifests/copying/labs.pp 
b/modules/dumps/manifests/copying/labs.pp
index 738803e..2d7061c 100644
--- a/modules/dumps/manifests/copying/labs.pp
+++ b/modules/dumps/manifests/copying/labs.pp
@@ -1,7 +1,7 @@
 class dumps::copying::labs(
 $labhost = undef,
 $xmldumpsdir = undef,
-$otherdir = undef,
+$miscdatasetsdir = undef,
 ) {
 file { '/usr/local/bin/wmfdumpsmirror.py':
 ensure => 'present',
@@ -20,7 +20,7 @@
 user=> 'root',
 minute  => '50',
 hour=> '3',
-command => "/usr/local/sbin/labs-rsync-cron.sh ${labhost} 
${xmldumpsdir} ${otherdir}",
+command => "/usr/local/sbin/labs-rsync-cron.sh ${labhost} 
${xmldumpsdir} ${miscdatasetsdir}",
 environment => 'MAILTO=ops-du...@wikimedia.org',
 require => File['/usr/local/bin/wmfdumpsmirror.py',
 '/usr/local/sbin/labs-rsync-cron.sh'],
diff --git a/modules/dumps/manifests/generation/server/dirs.pp 
b/modules/dumps/manifests/generation/server/dirs.pp
index f6608d7..1c29f28 100644
--- a/modules/dumps/manifests/generation/server/dirs.pp
+++ b/modules/dumps/manifests/generation/server/dirs.pp
@@ -1,30 +1,30 @@
 class dumps::generation::server::dirs(
-$datadir = undef,
-$xmldumpsdir = undef,
-$tempdir = undef,
-$otherdir= undef,
-$user= undef,
-$group   = undef,
+$datadir = undef,
+$xmldumpsdir = undef,
+$tempdir = undef,
+$miscdatasetsdir = undef,
+$user= undef,
+$group   = undef,
 ) {
 # Directories where dumps of any type are generated
 # This list is not for one-off directories, nor for
 # directories with incoming rsyncs of datasets
-$cirrussearchdir  = "${otherdir}/cirrussearch"
-$xlationdir   = "${otherdir}/contenttranslation"
-$categoriesrdfdir = "${otherdir}/categoriesrdf"
-$globalblocksdir  = "${otherdir}/globalblocks"
-$medialistsdir= "${otherdir}/imageinfo"
-$incrsdir = "${otherdir}/incr"
-$mediatitlesdir   = "${otherdir}/mediatitles"
-$othermiscdir = "${otherdir}/misc"
-$pagetitlesdir= "${otherdir}/pagetitles"
-$othertestfilesdir= "${otherdir}/testfiles"
-$otherwikibasedir = "${otherdir}/wikibase"
-$otherwikibasewikidatadir = "${otherdir}/wikibase/wikidatawiki"
-$otherwikidatadir = "${otherdir}/wikidata"
+$cirrussearchdir  = "${miscdatasetsdir}/cirrussearch"
+$xlationdir   = "${miscdatasetsdir}/contenttranslation"
+$categoriesrdfdir = "${miscdatasetsdir}/categoriesrdf"
+$globalblocksdir  = "${miscdatasetsdir}/globalblocks"
+$medialistsdir= "${miscdatasetsdir}/imageinfo"
+$incrsdir = "${miscdatasetsdir}/incr"
+$mediatitlesdir   = "${miscdatasetsdir}/mediatitles"
+$othermiscdir = "${miscdatasetsdir}/misc"
+$pagetitlesdir= "${miscdatasetsdir}/pagetitles"
+$othertestfilesdir= 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: clean up all references to a 'public dumps dir' on web/nfs d...

[ArielGlenn (Code Review)]

ArielGlenn has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/397806 )

Change subject: clean up all references to a 'public dumps dir' on web/nfs 
dumps servers
..


clean up all references to a 'public dumps dir' on web/nfs dumps servers

These hosts should have no notion of what a public dumps directory
is (as opposed to private files), they should only have a path for
xml dumps and one for misc/other dumps.

Change-Id: Ib3d75f53712c80bb2355a8aa8d0434502ae93eab
---
M modules/dumps/files/generation/rsync-to-peers.sh
M modules/dumps/files/web/list-last-good-dumps.sh
M modules/dumps/files/web/list-last-n-good-dumps.py
M modules/dumps/files/web/unpack-dumpstatusfiles.sh
M modules/dumps/manifests/copying/labs.pp
M modules/dumps/manifests/generation/server/dirs.pp
M modules/dumps/manifests/rsync/public.pp
M modules/dumps/manifests/web/cleanup.pp
M modules/dumps/manifests/web/cleanups/xmldumps.pp
M modules/dumps/manifests/web/dirs.pp
M modules/dumps/manifests/web/fetches/kiwix.pp
M modules/dumps/manifests/web/html.pp
M modules/dumps/manifests/web/xmldumps.pp
M modules/dumps/manifests/web/xmldumps_active.pp
M modules/dumps/templates/rsync/rsyncd.conf.dumps_to_public.erb
M modules/dumps/templates/web/xmldumps/nginx.conf.erb
M modules/profile/manifests/dumps/fetcher.pp
M modules/profile/manifests/dumps/generation/server/fallback.pp
M modules/profile/manifests/dumps/generation/server/primary.pp
M modules/profile/manifests/dumps/rsyncer.pp
M modules/profile/manifests/dumps/web/cleanup.pp
M modules/profile/manifests/dumps/web/dumpstatusfiles_sync.pp
M modules/profile/manifests/dumps/web/xmldumps_active.pp
M modules/profile/manifests/dumps/web/xmldumps_fallback.pp
24 files changed, 123 insertions(+), 117 deletions(-)

Approvals:
  ArielGlenn: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/dumps/files/generation/rsync-to-peers.sh 
b/modules/dumps/files/generation/rsync-to-peers.sh
index d5b8b4e..49abbe9 100644
--- a/modules/dumps/files/generation/rsync-to-peers.sh
+++ b/modules/dumps/files/generation/rsync-to-peers.sh
@@ -24,7 +24,7 @@
 
 Example:
 
- $0 --xmldumpsdir /data/xmldatadumps \\
+ $0 --xmldumpsdir /data/xmldatadumps/public \\
--xmlremotedirs 
dumpsdata1002.eqiad.wmnet::data/xmldatadumps/public/,dumpsdata1003.eqiad.wmnet::data/xmldatadumps/public/
 \\
--miscdumpsdir /data/otherdumps \\
--miscremotedirs 
dumpsdata1002.eqiad.wmnet::data/otherdumps/,dumpsdata1003.eqiad.wmnet::data/otherdumps/
@@ -34,7 +34,7 @@
 
 make_statusfiles_tarball() {
 # make tarball of all xml/sql dumps status and html files
-tarballpath="${xmldumpsdir}/public/dumpstatusfiles.tar"
+tarballpath="${xmldumpsdir}/dumpstatusfiles.tar"
 tarballpathgz="${tarballpath}.gz"
 
 # Only pick up the html/json/txt files from the latest run; even if it's
@@ -43,18 +43,18 @@
 # the new run started, unless there are 0 minutes between end of
 # one dump run across all wikis and start of the next (in which case
 #  we are cutting things WAY too close with the runs)
-latestwiki=$( cd "${xmldumpsdir}/public"; ls -td *wik* | head -1 )
+latestwiki=$( cd "$xmldumpsdir"; ls -td *wik* | head -1 )
 
  rm -f "$tarballpathgz"
 
 # dirname is MMDD, i.e. 8 digits. ignore all other directories.
-latestrun=$( cd "${xmldumpsdir}/public/${latestwiki}" ; ls -d 
[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9] | sort | tail -1 )
+latestrun=$( cd "${xmldumpsdir}/${latestwiki}" ; ls -d 
[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9] | sort | tail -1 )
 if [ -n "$latestrun" ]; then
 
# top-level index files first
-( cd "$xmldumpsdir"; /bin/tar cfp "$tarballpath" public/*html 
public/*json )
-# add per-wiki files next: ( cd /data/xmldatadumps; /usr/bin/find 
public/ -maxdepth 3 -regextype sed -regex ".*/20171120/.*\(json\|html\|txt\)" )
-( cd "$xmldumpsdir"; /usr/bin/find "public/" -maxdepth 3 -regextype 
sed -regex ".*/${latestrun}/.*\.\(json\|html\|txt\)" | /usr/bin/xargs -s 
1048576 /bin/tar rfp "$tarballpath" )
+( cd "$xmldumpsdir"; /bin/tar cfp "$tarballpath" *html *json )
+# add per-wiki files next: ( cd /data/xmldatadumps/public; 
/usr/bin/find . -maxdepth 3 -regextype sed -regex 
".*/20171120/.*\(json\|html\|txt\)" )
+( cd "$xmldumpsdir"; /usr/bin/find "." -maxdepth 3 -regextype sed 
-regex ".*/${latestrun}/.*\.\(json\|html\|txt\)" | /usr/bin/xargs -s 1048576 
/bin/tar rfp "$tarballpath" )
 
 # if no files found, there will be no tarball created either
if [ -f "$tarballpath" ]; then
diff --git a/modules/dumps/files/web/list-last-good-dumps.sh 
b/modules/dumps/files/web/list-last-good-dumps.sh
index f7ca326..2aa6d51 100644
--- a/modules/dumps/files/web/list-last-good-dumps.sh
+++ b/modules/dumps/files/web/list-last-good-dumps.sh
@@ -15,7 +15,7 @@
 
   --xmldumpsdir   path to root of xml/sql dumps tree
 
-Example:  $0 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: clean up all references to a 'public dumps dir' on web/nfs d...

[ArielGlenn (Code Review)]

ArielGlenn has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/397806 )

Change subject: clean up all references to a 'public dumps dir' on web/nfs 
dumps servers
..

clean up all references to a 'public dumps dir' on web/nfs dumps servers

These hosts should have no notion of what a public dumps directory
is (as opposed to private files), they should only have a path for
xml dumps and one for misc/other dumps.

Change-Id: Ib3d75f53712c80bb2355a8aa8d0434502ae93eab
---
M modules/dumps/files/generation/rsync-to-peers.sh
M modules/dumps/files/web/list-last-n-good-dumps.py
M modules/dumps/manifests/copying/labs.pp
M modules/dumps/manifests/generation/server/dirs.pp
M modules/dumps/manifests/rsync/public.pp
M modules/dumps/manifests/web/cleanup.pp
M modules/dumps/manifests/web/cleanups/xmldumps.pp
M modules/dumps/manifests/web/dirs.pp
M modules/dumps/manifests/web/fetches/kiwix.pp
M modules/dumps/manifests/web/html.pp
M modules/dumps/manifests/web/xmldumps.pp
M modules/dumps/manifests/web/xmldumps_active.pp
M modules/dumps/templates/rsync/rsyncd.conf.dumps_to_public.erb
M modules/dumps/templates/web/xmldumps/nginx.conf.erb
M modules/profile/manifests/dumps/fetcher.pp
M modules/profile/manifests/dumps/generation/server/fallback.pp
M modules/profile/manifests/dumps/generation/server/primary.pp
M modules/profile/manifests/dumps/rsyncer.pp
M modules/profile/manifests/dumps/web/cleanup.pp
M modules/profile/manifests/dumps/web/xmldumps_active.pp
M modules/profile/manifests/dumps/web/xmldumps_fallback.pp
21 files changed, 98 insertions(+), 92 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/06/397806/1

diff --git a/modules/dumps/files/generation/rsync-to-peers.sh 
b/modules/dumps/files/generation/rsync-to-peers.sh
index d5b8b4e..49abbe9 100644
--- a/modules/dumps/files/generation/rsync-to-peers.sh
+++ b/modules/dumps/files/generation/rsync-to-peers.sh
@@ -24,7 +24,7 @@
 
 Example:
 
- $0 --xmldumpsdir /data/xmldatadumps \\
+ $0 --xmldumpsdir /data/xmldatadumps/public \\
--xmlremotedirs 
dumpsdata1002.eqiad.wmnet::data/xmldatadumps/public/,dumpsdata1003.eqiad.wmnet::data/xmldatadumps/public/
 \\
--miscdumpsdir /data/otherdumps \\
--miscremotedirs 
dumpsdata1002.eqiad.wmnet::data/otherdumps/,dumpsdata1003.eqiad.wmnet::data/otherdumps/
@@ -34,7 +34,7 @@
 
 make_statusfiles_tarball() {
 # make tarball of all xml/sql dumps status and html files
-tarballpath="${xmldumpsdir}/public/dumpstatusfiles.tar"
+tarballpath="${xmldumpsdir}/dumpstatusfiles.tar"
 tarballpathgz="${tarballpath}.gz"
 
 # Only pick up the html/json/txt files from the latest run; even if it's
@@ -43,18 +43,18 @@
 # the new run started, unless there are 0 minutes between end of
 # one dump run across all wikis and start of the next (in which case
 #  we are cutting things WAY too close with the runs)
-latestwiki=$( cd "${xmldumpsdir}/public"; ls -td *wik* | head -1 )
+latestwiki=$( cd "$xmldumpsdir"; ls -td *wik* | head -1 )
 
  rm -f "$tarballpathgz"
 
 # dirname is MMDD, i.e. 8 digits. ignore all other directories.
-latestrun=$( cd "${xmldumpsdir}/public/${latestwiki}" ; ls -d 
[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9] | sort | tail -1 )
+latestrun=$( cd "${xmldumpsdir}/${latestwiki}" ; ls -d 
[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9] | sort | tail -1 )
 if [ -n "$latestrun" ]; then
 
# top-level index files first
-( cd "$xmldumpsdir"; /bin/tar cfp "$tarballpath" public/*html 
public/*json )
-# add per-wiki files next: ( cd /data/xmldatadumps; /usr/bin/find 
public/ -maxdepth 3 -regextype sed -regex ".*/20171120/.*\(json\|html\|txt\)" )
-( cd "$xmldumpsdir"; /usr/bin/find "public/" -maxdepth 3 -regextype 
sed -regex ".*/${latestrun}/.*\.\(json\|html\|txt\)" | /usr/bin/xargs -s 
1048576 /bin/tar rfp "$tarballpath" )
+( cd "$xmldumpsdir"; /bin/tar cfp "$tarballpath" *html *json )
+# add per-wiki files next: ( cd /data/xmldatadumps/public; 
/usr/bin/find . -maxdepth 3 -regextype sed -regex 
".*/20171120/.*\(json\|html\|txt\)" )
+( cd "$xmldumpsdir"; /usr/bin/find "." -maxdepth 3 -regextype sed 
-regex ".*/${latestrun}/.*\.\(json\|html\|txt\)" | /usr/bin/xargs -s 1048576 
/bin/tar rfp "$tarballpath" )
 
 # if no files found, there will be no tarball created either
if [ -f "$tarballpath" ]; then
diff --git a/modules/dumps/files/web/list-last-n-good-dumps.py 
b/modules/dumps/files/web/list-last-n-good-dumps.py
index 46070e5..c8e33b7 100755
--- a/modules/dumps/files/web/list-last-n-good-dumps.py
+++ b/modules/dumps/files/web/list-last-n-good-dumps.py
@@ -116,9 +116,8 @@
   'filelist': 'file_list_templ',
   'rsynclist': 'rsync_incl_templ'}
 
-def get_abs_pubdirpath(self, name):
-"""return full path to the location of public dumps,
-

[MediaWiki-commits] [Gerrit] operations/puppet[production]: fix up for script that lists last good dumps for mirrors

[ArielGlenn (Code Review)]

ArielGlenn has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/397790 )

Change subject: fix up for script that lists last good dumps for mirrors
..


fix up for script that lists last good dumps for mirrors

Did not handle properly the case where a wiki had never been dumped

Change-Id: Id03a41d384926220209c1e779ad6c6dcf8654d8f
---
M modules/dumps/files/web/list-last-n-good-dumps.py
1 file changed, 1 insertion(+), 1 deletion(-)

Approvals:
  ArielGlenn: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/dumps/files/web/list-last-n-good-dumps.py 
b/modules/dumps/files/web/list-last-n-good-dumps.py
index 6d7e80e..46070e5 100755
--- a/modules/dumps/files/web/list-last-n-good-dumps.py
+++ b/modules/dumps/files/web/list-last-n-good-dumps.py
@@ -206,7 +206,7 @@
 dir_first = get_first_dir(dirs, dir_to_check)
 if not dir_first:
 # never dumped
-return dirs_reported
+return dirs_reported, None
 
 text = get_dir_status(os.path.join(dir_to_check, dir_first))
 if "in-progress" in text:

-- 
To view, visit https://gerrit.wikimedia.org/r/397790
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Id03a41d384926220209c1e779ad6c6dcf8654d8f
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: ArielGlenn 
Gerrit-Reviewer: ArielGlenn 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: fix up for script that lists last good dumps for mirrors

[ArielGlenn (Code Review)]

ArielGlenn has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/397790 )

Change subject: fix up for script that lists last good dumps for mirrors
..

fix up for script that lists last good dumps for mirrors

Did not handle properly the case where a wiki had never been dumped

Change-Id: Id03a41d384926220209c1e779ad6c6dcf8654d8f
---
M modules/dumps/files/web/list-last-n-good-dumps.py
1 file changed, 1 insertion(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/90/397790/1

diff --git a/modules/dumps/files/web/list-last-n-good-dumps.py 
b/modules/dumps/files/web/list-last-n-good-dumps.py
index 6d7e80e..46070e5 100755
--- a/modules/dumps/files/web/list-last-n-good-dumps.py
+++ b/modules/dumps/files/web/list-last-n-good-dumps.py
@@ -206,7 +206,7 @@
 dir_first = get_first_dir(dirs, dir_to_check)
 if not dir_first:
 # never dumped
-return dirs_reported
+return dirs_reported, None
 
 text = get_dir_status(os.path.join(dir_to_check, dir_first))
 if "in-progress" in text:

-- 
To view, visit https://gerrit.wikimedia.org/r/397790
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Id03a41d384926220209c1e779ad6c6dcf8654d8f
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: ArielGlenn 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits