[MediaWiki-commits] [Gerrit] operations/puppet[production]: role::mariadb::misc::eventlogging: remove the readonly const...

[Elukey (Code Review)]

Elukey has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/365007 )

Change subject: role::mariadb::misc::eventlogging: remove the readonly 
constraint for slaves
..

role::mariadb::misc::eventlogging: remove the readonly constraint for slaves

Bug: T156933
Change-Id: I09841c14103e3ed09b68de2e9158b7f5e9a14d17
---
M modules/role/manifests/mariadb/misc/eventlogging.pp
1 file changed, 7 insertions(+), 5 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/07/365007/1

diff --git a/modules/role/manifests/mariadb/misc/eventlogging.pp 
b/modules/role/manifests/mariadb/misc/eventlogging.pp
index 82394e5..98bd2de 100644
--- a/modules/role/manifests/mariadb/misc/eventlogging.pp
+++ b/modules/role/manifests/mariadb/misc/eventlogging.pp
@@ -30,16 +30,18 @@
 include mariadb::packages_wmf
 include mariadb::service
 
-$read_only = $master ? {
-true  => 0,
-false => 1,
-}
 
+# History context: there used to be a distinction between
+# EL master and slaves, namely that only the master was not
+# in read only mode. The Analytics team removed this constraint
+# before deploying the eventlogging_cleaner script (T156933),
+# that needed to DELETE/UPDATE rows on the job database without
+# running as root for obvious reasons.
 class { 'mariadb::config':
 config=> 'role/mariadb/mysqld_config/eventlogging.my.cnf.erb',
 datadir   => '/srv/sqldata',
 tmpdir=> '/srv/tmp',
-read_only => $read_only,
+read_only => 0,
 ssl   => 'puppet-cert',
 p_s   => 'off',
 binlog_format => 'MIXED',

-- 
To view, visit https://gerrit.wikimedia.org/r/365007
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I09841c14103e3ed09b68de2e9158b7f5e9a14d17
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: eventlogging_cleaner.py: set default loglevel for the main l...

[Elukey (Code Review)]

Elukey has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/364961 )

Change subject: eventlogging_cleaner.py: set default loglevel for the main 
logger
..


eventlogging_cleaner.py: set default loglevel for the main logger

This change is related to https://gerrit.wikimedia.org/r/#/c/364956.
Not setting the default logger loglevel stops any stdout output.

Bug: T156933
Change-Id: Id36303a3ae7789962004ded357da339e470490c7
---
M modules/role/files/mariadb/eventlogging_cleaner.py
1 file changed, 1 insertion(+), 0 deletions(-)

Approvals:
  Elukey: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/role/files/mariadb/eventlogging_cleaner.py 
b/modules/role/files/mariadb/eventlogging_cleaner.py
index a94f9cf..634a74f 100644
--- a/modules/role/files/mariadb/eventlogging_cleaner.py
+++ b/modules/role/files/mariadb/eventlogging_cleaner.py
@@ -436,6 +436,7 @@
 
 log.addHandler(stdout_h)
 log.addHandler(stderr_h)
+log.setLevel(logging.DEBUG)
 
 # Args basic checks
 if args.no_whitelist and args.whitelist:

-- 
To view, visit https://gerrit.wikimedia.org/r/364961
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Id36303a3ae7789962004ded357da339e470490c7
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: Mforns 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: eventlogging_cleaner.py: set default loglevel for the main l...

[Elukey (Code Review)]

Elukey has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/364961 )

Change subject: eventlogging_cleaner.py: set default loglevel for the main 
logger
..

eventlogging_cleaner.py: set default loglevel for the main logger

This change is related to https://gerrit.wikimedia.org/r/#/c/364956.
Not setting the default logger loglevel stops any stdout output.

Bug: T156933
Change-Id: Id36303a3ae7789962004ded357da339e470490c7
---
M modules/role/files/mariadb/eventlogging_cleaner.py
1 file changed, 1 insertion(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/61/364961/1

diff --git a/modules/role/files/mariadb/eventlogging_cleaner.py 
b/modules/role/files/mariadb/eventlogging_cleaner.py
index a94f9cf..634a74f 100644
--- a/modules/role/files/mariadb/eventlogging_cleaner.py
+++ b/modules/role/files/mariadb/eventlogging_cleaner.py
@@ -436,6 +436,7 @@
 
 log.addHandler(stdout_h)
 log.addHandler(stderr_h)
+log.setLevel(logging.DEBUG)
 
 # Args basic checks
 if args.no_whitelist and args.whitelist:

-- 
To view, visit https://gerrit.wikimedia.org/r/364961
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Id36303a3ae7789962004ded357da339e470490c7
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: eventlogging_cleaner.py: split logs between stdout/stderr$

[Elukey (Code Review)]

Elukey has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/364956 )

Change subject: eventlogging_cleaner.py: split logs between stdout/stderr$
..


eventlogging_cleaner.py: split logs between stdout/stderr$

Separate the logging levels to redirect ERRORs only to stderr.$
The script will probably run as cron and it will be easier$
with the new configuration to get email notifications.$

Bug: T156933$
Change-Id: Ib78c09992181b6d95a0c90c7168b50fb7edb8215
---
M modules/role/files/mariadb/eventlogging_cleaner.py
1 file changed, 19 insertions(+), 16 deletions(-)

Approvals:
  Elukey: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/role/files/mariadb/eventlogging_cleaner.py 
b/modules/role/files/mariadb/eventlogging_cleaner.py
index 4baf78b..a94f9cf 100644
--- a/modules/role/files/mariadb/eventlogging_cleaner.py
+++ b/modules/role/files/mariadb/eventlogging_cleaner.py
@@ -60,6 +60,15 @@
 log = logging.getLogger(__name__)
 
 
+class MaxLevelFilter(logging.Filter):
+
+def __init__(self, level):
+self.level = level
+
+def filter(self, record):
+return record.levelno < self.level
+
+
 class Database(object):
 
 def __init__(self, db_host, db_name, db_user, db_password=None,
@@ -402,9 +411,6 @@
 ' (default: 91)')
 parser.add_argument('--dry-run', dest='dry_run', action='store_true',
 help='Only print sql commands without executing them')
-parser.add_argument('--logfile', dest='logfile', default=None,
-help='Redirect the script\'s output to a file rather '
- 'than stdout')
 parser.add_argument('--batch-size', dest='batch_size', default=1000, 
type=int,
 help='Maximum number of DB rows to update/delete in 
one go.'
  ' (default: 1000)')
@@ -419,20 +425,17 @@
  'option is preferred). Default: /etc/my.cnf')
 args = parser.parse_args()
 
-log_format = ('%(levelname)s: line %(lineno)d: %(message)s')
+log_format = logging.Formatter('%(levelname)s: line %(lineno)d: 
%(message)s')
+stdout_h = logging.StreamHandler(sys.stdout)
+stdout_h.addFilter(MaxLevelFilter(logging.WARNING))
+stdout_h.setFormatter(log_format)
 
-if args.logfile:
-logging.basicConfig(
-filename=args.logfile,
-level=logging.INFO,
-format=log_format
-)
-else:
-logging.basicConfig(
-stream=sys.stdout,
-level=logging.INFO,
-format=log_format
-)
+stderr_h = logging.StreamHandler(sys.stderr)
+stderr_h.setLevel(logging.ERROR)
+stderr_h.setFormatter(log_format)
+
+log.addHandler(stdout_h)
+log.addHandler(stderr_h)
 
 # Args basic checks
 if args.no_whitelist and args.whitelist:

-- 
To view, visit https://gerrit.wikimedia.org/r/364956
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ib78c09992181b6d95a0c90c7168b50fb7edb8215
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: Mforns 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: eventlogging_cleaner.py: separate logs betweem stdout/stderr$

[Elukey (Code Review)]

Elukey has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/364956 )

Change subject: eventlogging_cleaner.py: separate logs betweem stdout/stderr$
..

eventlogging_cleaner.py: separate logs betweem stdout/stderr$

Separate the logging levels to emit ERRORs only to stderr.$
The script will probably run as cron and it will be easier$
with the new configuration to get error notifications via email.$

Bug: T156933$
Change-Id: Ib78c09992181b6d95a0c90c7168b50fb7edb8215
---
M modules/role/files/mariadb/eventlogging_cleaner.py
1 file changed, 19 insertions(+), 16 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/56/364956/1

diff --git a/modules/role/files/mariadb/eventlogging_cleaner.py 
b/modules/role/files/mariadb/eventlogging_cleaner.py
index 4baf78b..a94f9cf 100644
--- a/modules/role/files/mariadb/eventlogging_cleaner.py
+++ b/modules/role/files/mariadb/eventlogging_cleaner.py
@@ -60,6 +60,15 @@
 log = logging.getLogger(__name__)
 
 
+class MaxLevelFilter(logging.Filter):
+
+def __init__(self, level):
+self.level = level
+
+def filter(self, record):
+return record.levelno < self.level
+
+
 class Database(object):
 
 def __init__(self, db_host, db_name, db_user, db_password=None,
@@ -402,9 +411,6 @@
 ' (default: 91)')
 parser.add_argument('--dry-run', dest='dry_run', action='store_true',
 help='Only print sql commands without executing them')
-parser.add_argument('--logfile', dest='logfile', default=None,
-help='Redirect the script\'s output to a file rather '
- 'than stdout')
 parser.add_argument('--batch-size', dest='batch_size', default=1000, 
type=int,
 help='Maximum number of DB rows to update/delete in 
one go.'
  ' (default: 1000)')
@@ -419,20 +425,17 @@
  'option is preferred). Default: /etc/my.cnf')
 args = parser.parse_args()
 
-log_format = ('%(levelname)s: line %(lineno)d: %(message)s')
+log_format = logging.Formatter('%(levelname)s: line %(lineno)d: 
%(message)s')
+stdout_h = logging.StreamHandler(sys.stdout)
+stdout_h.addFilter(MaxLevelFilter(logging.WARNING))
+stdout_h.setFormatter(log_format)
 
-if args.logfile:
-logging.basicConfig(
-filename=args.logfile,
-level=logging.INFO,
-format=log_format
-)
-else:
-logging.basicConfig(
-stream=sys.stdout,
-level=logging.INFO,
-format=log_format
-)
+stderr_h = logging.StreamHandler(sys.stderr)
+stderr_h.setLevel(logging.ERROR)
+stderr_h.setFormatter(log_format)
+
+log.addHandler(stdout_h)
+log.addHandler(stderr_h)
 
 # Args basic checks
 if args.no_whitelist and args.whitelist:

-- 
To view, visit https://gerrit.wikimedia.org/r/364956
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ib78c09992181b6d95a0c90c7168b50fb7edb8215
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: eventlogging_purging_whitelist.tsv: remove unnecessary schemas

[Elukey (Code Review)]

Elukey has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/364949 )

Change subject: eventlogging_purging_whitelist.tsv: remove unnecessary schemas
..


eventlogging_purging_whitelist.tsv: remove unnecessary schemas

The eventlogging_cleaner.py script fails to execute on db1047 due
to these tables being present in the whitelist but not in the log
database. This error condition should alert analytics about typos
while redacting the whitelist, but in this case it means something
different: one table is an old one that has already been dropped,
meanwhile the other one is too new and it has received no events
yet (that means no table created in the db yet too).

Bug: T156933
Change-Id: I3cf3570ef0a9f8f1e95b7dbc657236aa0a139e90
---
M modules/role/files/mariadb/eventlogging_purging_whitelist.tsv
1 file changed, 0 insertions(+), 18 deletions(-)

Approvals:
  Elukey: Verified; Looks good to me, approved



diff --git a/modules/role/files/mariadb/eventlogging_purging_whitelist.tsv 
b/modules/role/files/mariadb/eventlogging_purging_whitelist.tsv
index 62d6f0c..65f3b3e 100644
--- a/modules/role/files/mariadb/eventlogging_purging_whitelist.tsv
+++ b/modules/role/files/mariadb/eventlogging_purging_whitelist.tsv
@@ -214,16 +214,6 @@
 GuidedTour isTruncated
 GuidedTour webHost
 GuidedTour wiki
-MediaWikiInstallPingback   webHost
-MediaWikiInstallPingback   wiki
-MediaWikiInstallPingback   event_MediaWiki
-MediaWikiInstallPingback   event_OS
-MediaWikiInstallPingback   event_PHP
-MediaWikiInstallPingback   event_arch
-MediaWikiInstallPingback   event_curl
-MediaWikiInstallPingback   event_database
-MediaWikiInstallPingback   event_machine
-MediaWikiInstallPingback   event_webServer
 MobileOptionsTracking  clientValidated
 MobileOptionsTracking  event_action
 MobileOptionsTracking  event_alpha
@@ -951,14 +941,6 @@
 UploadWizardTutorialActionsevent_username
 UploadWizardTutorialActionsevent_language
 UploadWizardTutorialActionsevent_action
-UploadWizardUploadActions  webHost
-UploadWizardUploadActions  wiki
-UploadWizardUploadActions  event_username
-UploadWizardUploadActions  event_language
-UploadWizardUploadActions  event_filetype
-UploadWizardUploadActions  event_error
-UploadWizardUploadActions  event_numuploads
-UploadWizardUploadActions  event_action
 MobileWikiAppFeed  userAgent
 MobileWikiAppFeed  webHost
 MobileWikiAppFeed  wiki

-- 
To view, visit https://gerrit.wikimedia.org/r/364949
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I3cf3570ef0a9f8f1e95b7dbc657236aa0a139e90
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: Mforns 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: eventlogging_purging_whitelist.tsv: remove unnecessary schemas

[Elukey (Code Review)]

Elukey has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/364949 )

Change subject: eventlogging_purging_whitelist.tsv: remove unnecessary schemas
..

eventlogging_purging_whitelist.tsv: remove unnecessary schemas

The eventlogging_cleaner.py script fails to execute on db1047 due
to these tables being present in the whitelist but not in the log
database. This error condition should alert analytics about typos
while redacting the whitelist, but in this case it means something
different: one table is an old one that has already been dropped,
meanwhile the other one is too new and it has received no events
yet (that means no table created in the db yet too).

Bug: T156933
Change-Id: I3cf3570ef0a9f8f1e95b7dbc657236aa0a139e90
---
M modules/role/files/mariadb/eventlogging_purging_whitelist.tsv
1 file changed, 0 insertions(+), 18 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/49/364949/1

diff --git a/modules/role/files/mariadb/eventlogging_purging_whitelist.tsv 
b/modules/role/files/mariadb/eventlogging_purging_whitelist.tsv
index 62d6f0c..65f3b3e 100644
--- a/modules/role/files/mariadb/eventlogging_purging_whitelist.tsv
+++ b/modules/role/files/mariadb/eventlogging_purging_whitelist.tsv
@@ -214,16 +214,6 @@
 GuidedTour isTruncated
 GuidedTour webHost
 GuidedTour wiki
-MediaWikiInstallPingback   webHost
-MediaWikiInstallPingback   wiki
-MediaWikiInstallPingback   event_MediaWiki
-MediaWikiInstallPingback   event_OS
-MediaWikiInstallPingback   event_PHP
-MediaWikiInstallPingback   event_arch
-MediaWikiInstallPingback   event_curl
-MediaWikiInstallPingback   event_database
-MediaWikiInstallPingback   event_machine
-MediaWikiInstallPingback   event_webServer
 MobileOptionsTracking  clientValidated
 MobileOptionsTracking  event_action
 MobileOptionsTracking  event_alpha
@@ -951,14 +941,6 @@
 UploadWizardTutorialActionsevent_username
 UploadWizardTutorialActionsevent_language
 UploadWizardTutorialActionsevent_action
-UploadWizardUploadActions  webHost
-UploadWizardUploadActions  wiki
-UploadWizardUploadActions  event_username
-UploadWizardUploadActions  event_language
-UploadWizardUploadActions  event_filetype
-UploadWizardUploadActions  event_error
-UploadWizardUploadActions  event_numuploads
-UploadWizardUploadActions  event_action
 MobileWikiAppFeed  userAgent
 MobileWikiAppFeed  webHost
 MobileWikiAppFeed  wiki

-- 
To view, visit https://gerrit.wikimedia.org/r/364949
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I3cf3570ef0a9f8f1e95b7dbc657236aa0a139e90
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: eventlogging_cleaner.py: configure configparser

[Elukey (Code Review)]

Elukey has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/364743 )

Change subject: eventlogging_cleaner.py: configure configparser
..


eventlogging_cleaner.py: configure configparser

ConfigParser needs to allow duplicate/empty values while reading
a my.cnf config to avoid unnecessary runtime exceptions.

Change-Id: I13904b55c3ba1dae9473af4f22a6dba30c1822fe
---
M modules/role/files/mariadb/eventlogging_cleaner.py
1 file changed, 5 insertions(+), 1 deletion(-)

Approvals:
  Elukey: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/role/files/mariadb/eventlogging_cleaner.py 
b/modules/role/files/mariadb/eventlogging_cleaner.py
index 76e3f44..4baf78b 100644
--- a/modules/role/files/mariadb/eventlogging_cleaner.py
+++ b/modules/role/files/mariadb/eventlogging_cleaner.py
@@ -490,7 +490,11 @@
 whitelist = {}
 
 # Parse the db my.cnf config file
-config = configparser.ConfigParser()
+# my.cn may contain duplicate entries within the same section
+# (like multiple plugin-load) and also empty statements (not followed 
by
+# by any '=') so configparser needs to be relaxed a bit to avoid
+# unnecessary runtime exceptions.
+config = configparser.ConfigParser(strict=False, allow_no_value=True)
 config.read(args.my_cnf)
 
 # Priority to the local unix socket, default to username/password

-- 
To view, visit https://gerrit.wikimedia.org/r/364743
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I13904b55c3ba1dae9473af4f22a6dba30c1822fe
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: Mforns 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: eventlogging_cleaner.py: configure configparser

[Elukey (Code Review)]

Elukey has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/364743 )

Change subject: eventlogging_cleaner.py: configure configparser
..

eventlogging_cleaner.py: configure configparser

ConfigParser needs to allow duplicate/empty values while reading
a my.cnf config to avoid unnecessary runtime exceptions.

Change-Id: I13904b55c3ba1dae9473af4f22a6dba30c1822fe
---
M modules/role/files/mariadb/eventlogging_cleaner.py
1 file changed, 5 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/43/364743/1

diff --git a/modules/role/files/mariadb/eventlogging_cleaner.py 
b/modules/role/files/mariadb/eventlogging_cleaner.py
index 76e3f44..4baf78b 100644
--- a/modules/role/files/mariadb/eventlogging_cleaner.py
+++ b/modules/role/files/mariadb/eventlogging_cleaner.py
@@ -490,7 +490,11 @@
 whitelist = {}
 
 # Parse the db my.cnf config file
-config = configparser.ConfigParser()
+# my.cn may contain duplicate entries within the same section
+# (like multiple plugin-load) and also empty statements (not followed 
by
+# by any '=') so configparser needs to be relaxed a bit to avoid
+# unnecessary runtime exceptions.
+config = configparser.ConfigParser(strict=False, allow_no_value=True)
 config.read(args.my_cnf)
 
 # Priority to the local unix socket, default to username/password

-- 
To view, visit https://gerrit.wikimedia.org/r/364743
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I13904b55c3ba1dae9473af4f22a6dba30c1822fe
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics/refinery[master]: Add din.wikipedia to whitelist

[Elukey (Code Review)]

Elukey has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/364720 )

Change subject: Add din.wikipedia to whitelist
..


Add din.wikipedia to whitelist

Change-Id: I6f00d14dd98aa7473506a67ddc7e3b867844d588
---
M static_data/pageview/whitelist/whitelist.tsv
1 file changed, 1 insertion(+), 0 deletions(-)

Approvals:
  Elukey: Verified; Looks good to me, approved



diff --git a/static_data/pageview/whitelist/whitelist.tsv 
b/static_data/pageview/whitelist/whitelist.tsv
index 37b5de0..6758778 100644
--- a/static_data/pageview/whitelist/whitelist.tsv
+++ b/static_data/pageview/whitelist/whitelist.tsv
@@ -168,6 +168,7 @@
 projectde.wikiversity  2015-10-12 00:00:00
 projectde.wikivoyage   2015-10-12 00:00:00
 projectde.wiktionary   2015-10-12 00:00:00
+projectdin.wikipedia   2017-07-12 00:00:00
 projectdiq.wikipedia   2015-10-12 00:00:00
 projectdk.wikimedia2016-11-01 00:00:00
 projectdsb.wikipedia   2015-10-12 00:00:00

-- 
To view, visit https://gerrit.wikimedia.org/r/364720
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I6f00d14dd98aa7473506a67ddc7e3b867844d588
Gerrit-PatchSet: 2
Gerrit-Project: analytics/refinery
Gerrit-Branch: master
Gerrit-Owner: Fdans 
Gerrit-Reviewer: Elukey 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: eventlogging_cleaner.py: fix some runtime issues

[Elukey (Code Review)]

Elukey has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/364701 )

Change subject: eventlogging_cleaner.py: fix some runtime issues
..


eventlogging_cleaner.py: fix some runtime issues

After I tried to run the script on db1047 a couple of issues arose:
- the dbport should not be passed in case the unix_socket is used.
- os.getlogin() does not get the effective uid, and in [1] it is
  suggested to use pwd instead.

[1]: https://docs.python.org/3/library/os.html

Change-Id: Ifb384cf6cade5c7c9f008c4b6949cbce1103dcff
---
M modules/role/files/mariadb/eventlogging_cleaner.py
1 file changed, 5 insertions(+), 2 deletions(-)

Approvals:
  Elukey: Verified; Looks good to me, approved



diff --git a/modules/role/files/mariadb/eventlogging_cleaner.py 
b/modules/role/files/mariadb/eventlogging_cleaner.py
index 58eee6d..76e3f44 100644
--- a/modules/role/files/mariadb/eventlogging_cleaner.py
+++ b/modules/role/files/mariadb/eventlogging_cleaner.py
@@ -39,6 +39,7 @@
 import csv
 import logging
 import os
+import pwd
 import re
 import sys
 import time
@@ -495,8 +496,9 @@
 # Priority to the local unix socket, default to username/password
 try:
 unix_socket = config.get('client', 'socket')
-db_user = os.getlogin()
+db_user = pwd.getpwuid(os.getuid())[0]
 db_password = None
+db_port = None
 except configparser.NoOptionError as e:
 log.info(
 "No local unix socket configured for myql, default to 
username/password"
@@ -504,12 +506,13 @@
 unix_socket = None
 db_user = config.get('client', 'user')
 db_password = config.get('client', 'password')
+db_port = args.dbport
 
 # Connect to the database in localhost (no other option
 # available). This is a design choice to simplify auth
 # and to restrict the actions taken to the local db only.
 database = Database('localhost', args.dbname, db_user, 
db_password=db_password,
-db_port=args.dbport, unix_socket=unix_socket)
+db_port=db_port, unix_socket=unix_socket)
 
 # Apply the retention policy to each table
 tables = database.get_all_tables()

-- 
To view, visit https://gerrit.wikimedia.org/r/364701
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ifb384cf6cade5c7c9f008c4b6949cbce1103dcff
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: Mforns 
Gerrit-Reviewer: Volans 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: eventlogging_cleaner.py: fix some runtime issues

[Elukey (Code Review)]

Elukey has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/364701 )

Change subject: eventlogging_cleaner.py: fix some runtime issues
..

eventlogging_cleaner.py: fix some runtime issues

After I tried to run the script on db1047 a couple of issues arose:
- the dbport should not be passed in case the unix_socket is used.
- os.getlogin() does not get the effective uid, and in [1] it is
  suggested to use pwd instead.

[1]: https://docs.python.org/3/library/os.html

Change-Id: Ifb384cf6cade5c7c9f008c4b6949cbce1103dcff
---
M modules/role/files/mariadb/eventlogging_cleaner.py
1 file changed, 5 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/01/364701/1

diff --git a/modules/role/files/mariadb/eventlogging_cleaner.py 
b/modules/role/files/mariadb/eventlogging_cleaner.py
index 58eee6d..76e3f44 100644
--- a/modules/role/files/mariadb/eventlogging_cleaner.py
+++ b/modules/role/files/mariadb/eventlogging_cleaner.py
@@ -39,6 +39,7 @@
 import csv
 import logging
 import os
+import pwd
 import re
 import sys
 import time
@@ -495,8 +496,9 @@
 # Priority to the local unix socket, default to username/password
 try:
 unix_socket = config.get('client', 'socket')
-db_user = os.getlogin()
+db_user = pwd.getpwuid(os.getuid())[0]
 db_password = None
+db_port = None
 except configparser.NoOptionError as e:
 log.info(
 "No local unix socket configured for myql, default to 
username/password"
@@ -504,12 +506,13 @@
 unix_socket = None
 db_user = config.get('client', 'user')
 db_password = config.get('client', 'password')
+db_port = args.dbport
 
 # Connect to the database in localhost (no other option
 # available). This is a design choice to simplify auth
 # and to restrict the actions taken to the local db only.
 database = Database('localhost', args.dbname, db_user, 
db_password=db_password,
-db_port=args.dbport, unix_socket=unix_socket)
+db_port=db_port, unix_socket=unix_socket)
 
 # Apply the retention policy to each table
 tables = database.get_all_tables()

-- 
To view, visit https://gerrit.wikimedia.org/r/364701
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ifb384cf6cade5c7c9f008c4b6949cbce1103dcff
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: role::mariadb::analytics::custom_repl_slave: correct owner o...

[Elukey (Code Review)]

Elukey has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/364696 )

Change subject: role::mariadb::analytics::custom_repl_slave: correct owner of 
el cleaner
..


role::mariadb::analytics::custom_repl_slave: correct owner of el cleaner

The eventlogging_cleaner cleaner script needs to be ran by the
eventlogcleaner user in order to authenticate to mysql via local
unix socket.

Bug: T170118
Change-Id: Ib86722b2ff72d8fdc7c735ec803c9d8152d5601b
---
M modules/role/manifests/mariadb.pp
1 file changed, 3 insertions(+), 3 deletions(-)

Approvals:
  Elukey: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/role/manifests/mariadb.pp 
b/modules/role/manifests/mariadb.pp
index 8cc2d35..f22bac9 100644
--- a/modules/role/manifests/mariadb.pp
+++ b/modules/role/manifests/mariadb.pp
@@ -229,9 +229,9 @@
 
 file { '/usr/local/bin/eventlogging_cleaner':
 ensure  => present,
-owner   => 'root',
-group   => 'root',
-mode=> '0500',
+owner   => 'eventlogcleaner',
+group   => 'eventlog',
+mode=> '0550',
 source  => 'puppet:///modules/role/mariadb/eventlogging_cleaner.py',
 require => Package['python3-pymysql'],
 }

-- 
To view, visit https://gerrit.wikimedia.org/r/364696
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ib86722b2ff72d8fdc7c735ec803c9d8152d5601b
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: role::mariadb::analytics::custom_repl_slave: correct owner o...

[Elukey (Code Review)]

Elukey has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/364696 )

Change subject: role::mariadb::analytics::custom_repl_slave: correct owner of 
el cleaner
..

role::mariadb::analytics::custom_repl_slave: correct owner of el cleaner

The eventlogging_cleaner cleaner script needs to be ran by the
eventlogcleaner user in order to authenticate to mysql via local
unix socket.

Bug: T170118
Change-Id: Ib86722b2ff72d8fdc7c735ec803c9d8152d5601b
---
M modules/role/manifests/mariadb.pp
1 file changed, 3 insertions(+), 3 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/96/364696/1

diff --git a/modules/role/manifests/mariadb.pp 
b/modules/role/manifests/mariadb.pp
index 8cc2d35..f22bac9 100644
--- a/modules/role/manifests/mariadb.pp
+++ b/modules/role/manifests/mariadb.pp
@@ -229,9 +229,9 @@
 
 file { '/usr/local/bin/eventlogging_cleaner':
 ensure  => present,
-owner   => 'root',
-group   => 'root',
-mode=> '0500',
+owner   => 'eventlogcleaner',
+group   => 'eventlog',
+mode=> '0550',
 source  => 'puppet:///modules/role/mariadb/eventlogging_cleaner.py',
 require => Package['python3-pymysql'],
 }

-- 
To view, visit https://gerrit.wikimedia.org/r/364696
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ib86722b2ff72d8fdc7c735ec803c9d8152d5601b
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: role::mariadb::analytics::custom_repl_slave: fix typo in whi...

[Elukey (Code Review)]

Elukey has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/364692 )

Change subject: role::mariadb::analytics::custom_repl_slave: fix typo in 
whitelist path
..


role::mariadb::analytics::custom_repl_slave: fix typo in whitelist path

Bug: T170118
Change-Id: Iccfd83a6a7155962f4d9be92399baab75ba6aafe
---
M modules/role/manifests/mariadb.pp
1 file changed, 1 insertion(+), 1 deletion(-)

Approvals:
  Elukey: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/role/manifests/mariadb.pp 
b/modules/role/manifests/mariadb.pp
index be11559..8cc2d35 100644
--- a/modules/role/manifests/mariadb.pp
+++ b/modules/role/manifests/mariadb.pp
@@ -248,7 +248,7 @@
 owner   => 'root',
 group   => 'root',
 mode=> '0444',
-source  => 'puppet:///modules/role/mariadb/whitelist.tsv',
+source  => 
'puppet:///modules/role/mariadb/eventlogging_purging_whitelist.tsv',
 require => File['/etc/eventlogging'],
 }
 

-- 
To view, visit https://gerrit.wikimedia.org/r/364692
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Iccfd83a6a7155962f4d9be92399baab75ba6aafe
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: role::mariadb::analytics::custom_repl_slave: fix typo in whi...

[Elukey (Code Review)]

Elukey has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/364692 )

Change subject: role::mariadb::analytics::custom_repl_slave: fix typo in 
whitelist path
..

role::mariadb::analytics::custom_repl_slave: fix typo in whitelist path

Bug: T170118
Change-Id: Iccfd83a6a7155962f4d9be92399baab75ba6aafe
---
M modules/role/manifests/mariadb.pp
1 file changed, 1 insertion(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/92/364692/1

diff --git a/modules/role/manifests/mariadb.pp 
b/modules/role/manifests/mariadb.pp
index be11559..8cc2d35 100644
--- a/modules/role/manifests/mariadb.pp
+++ b/modules/role/manifests/mariadb.pp
@@ -248,7 +248,7 @@
 owner   => 'root',
 group   => 'root',
 mode=> '0444',
-source  => 'puppet:///modules/role/mariadb/whitelist.tsv',
+source  => 
'puppet:///modules/role/mariadb/eventlogging_purging_whitelist.tsv',
 require => File['/etc/eventlogging'],
 }
 

-- 
To view, visit https://gerrit.wikimedia.org/r/364692
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Iccfd83a6a7155962f4d9be92399baab75ba6aafe
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: role::mariadb::analytics::custom_repl_slave: deploy the EL w...

[Elukey (Code Review)]

Elukey has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/364691 )

Change subject: role::mariadb::analytics::custom_repl_slave: deploy the EL 
whitelist
..


role::mariadb::analytics::custom_repl_slave: deploy the EL whitelist

The new eventlogging_cleaner.py script needs a whitelist of tables
of the log database to know what data to sanitize and what simply
to delete.

Related gerrit change: https://gerrit.wikimedia.org/r/#/c/356383

Bug: T108850
Change-Id: I79a3a8d8e30587b25bc9755fdddbcd679e8b3ab0
---
M modules/role/manifests/mariadb.pp
1 file changed, 16 insertions(+), 0 deletions(-)

Approvals:
  Elukey: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/role/manifests/mariadb.pp 
b/modules/role/manifests/mariadb.pp
index ca83e8c..be11559 100644
--- a/modules/role/manifests/mariadb.pp
+++ b/modules/role/manifests/mariadb.pp
@@ -236,6 +236,22 @@
 require => Package['python3-pymysql'],
 }
 
+file { '/etc/eventlogging':
+ensure => 'directory',
+owner  => 'root',
+group  => 'root',
+mode   => '0755',
+}
+
+file { '/etc/eventlogging/whitelist.tsv':
+ensure  => 'present',
+owner   => 'root',
+group   => 'root',
+mode=> '0444',
+source  => 'puppet:///modules/role/mariadb/whitelist.tsv',
+require => File['/etc/eventlogging'],
+}
+
 file { '/usr/local/bin/eventlogging_sync.sh':
 ensure => present,
 owner  => 'root',

-- 
To view, visit https://gerrit.wikimedia.org/r/364691
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I79a3a8d8e30587b25bc9755fdddbcd679e8b3ab0
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: Jcrespo 
Gerrit-Reviewer: Marostegui 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: role::mariadb::analytics::custom_repl_slave: deploy the EL w...

[Elukey (Code Review)]

Elukey has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/364691 )

Change subject: role::mariadb::analytics::custom_repl_slave: deploy the EL 
whitelist
..

role::mariadb::analytics::custom_repl_slave: deploy the EL whitelist

The new eventlogging_cleaner.py script needs a whitelist of tables
of the log database to know what data to sanitize and what simply
to delete.

Related gerrit change: https://gerrit.wikimedia.org/r/#/c/356383

Bug: T108850
Change-Id: I79a3a8d8e30587b25bc9755fdddbcd679e8b3ab0
---
M modules/role/manifests/mariadb.pp
1 file changed, 16 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/91/364691/1

diff --git a/modules/role/manifests/mariadb.pp 
b/modules/role/manifests/mariadb.pp
index ca83e8c..be11559 100644
--- a/modules/role/manifests/mariadb.pp
+++ b/modules/role/manifests/mariadb.pp
@@ -236,6 +236,22 @@
 require => Package['python3-pymysql'],
 }
 
+file { '/etc/eventlogging':
+ensure => 'directory',
+owner  => 'root',
+group  => 'root',
+mode   => '0755',
+}
+
+file { '/etc/eventlogging/whitelist.tsv':
+ensure  => 'present',
+owner   => 'root',
+group   => 'root',
+mode=> '0444',
+source  => 'puppet:///modules/role/mariadb/whitelist.tsv',
+require => File['/etc/eventlogging'],
+}
+
 file { '/usr/local/bin/eventlogging_sync.sh':
 ensure => present,
 owner  => 'root',

-- 
To view, visit https://gerrit.wikimedia.org/r/364691
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I79a3a8d8e30587b25bc9755fdddbcd679e8b3ab0
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: role::mariadb::analytics::custom_repl_slave: add eventloggin...

[Elukey (Code Review)]

Elukey has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/356383 )

Change subject: role::mariadb::analytics::custom_repl_slave: add 
eventlogging_cleaner.py
..


role::mariadb::analytics::custom_repl_slave: add eventlogging_cleaner.py

This script implements the purge/sanitization policies outlined in
T108850. Unit tests have been added at the bottom of the file with
instructions about how to run them properly.

Bug: T108850
Change-Id: I33e312cd39a9860c895897fcb90ed23820ca4dff
---
A modules/role/files/mariadb/eventlogging_cleaner.py
A modules/role/files/mariadb/eventlogging_purging_whitelist.tsv
M modules/role/manifests/mariadb.pp
3 files changed, 1,858 insertions(+), 0 deletions(-)

Approvals:
  Elukey: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/role/files/mariadb/eventlogging_cleaner.py 
b/modules/role/files/mariadb/eventlogging_cleaner.py
new file mode 100644
index 000..58eee6d
--- /dev/null
+++ b/modules/role/files/mariadb/eventlogging_cleaner.py
@@ -0,0 +1,820 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+
+"""
+This script enforces the Analytics data retention guidelines outlined in:
+https://wikitech.wikimedia.org/wiki/Analytics/Systems/EventLogging/Data_retention_and_auto-purging
+
+The script reads a whitelist (TSV file) with the following format for each 
line:
+
+Tablename\tfield
+Tablename\tfield2
+Tablename2\tfield_bla
+[...]
+
+The script works in the following way: for each table in the EventLogging 
database,
+it looks for any reference of it in the whitelist. If none is found, it means 
that
+there is no interest of preserving any kind of non-sensitive data, therefore
+the retention policy is applied simply deleting all the rows matching the time
+delta provided in input. If one or more reference is found, it means that some
+fields of a given table need to be preserved for historical analytics, 
therefore
+the script will execute update commands to set to NULL all the non-whitelisted 
fields
+belonging to rows matching the time delta provided in input.
+
+Important notes:
+1) The script is meant to run on the same host in which the database that needs
+   to be cleaned is running. The script will try basic authentication
+   if any of DB username/password are provided by the user as my.cnf 
configuration
+   file (the conf file needs to have a [client] section with 'user' and 
'password').
+2) If a table is listed in the whitelist, then some of its fields are 
automatically
+   added to it (see COMMON_PERSISTENT_FIELDS). This ensures that important 
fields
+   like timestamp or primary keys are preserved.
+3) The script runs updates/deletes in batches to avoid blocking the database 
for too
+   long creating contention with other write operations (like inserts).
+"""
+
+import argparse
+import collections
+import configparser
+import csv
+import logging
+import os
+import re
+import sys
+import time
+import unittest
+import uuid
+
+from datetime import datetime, timedelta
+from unittest.mock import MagicMock, Mock, call, patch
+
+import pymysql
+
+DATE_FORMAT = '%Y%m%d%H%M%S'
+
+# Fields that are always present due to the EventLogging Capsule.
+# These ones are automatically whitelisted due to their importance.
+COMMON_PERSISTENT_FIELDS = ('id', 'uuid', 'timestamp')
+
+log = logging.getLogger(__name__)
+
+
+class Database(object):
+
+def __init__(self, db_host, db_name, db_user, db_password=None,
+ db_port=None, unix_socket=None):
+self.db_host = db_host
+self.db_name = db_name
+
+if (db_password is not None or db_port is not None) and unix_socket is 
not None:
+raise RuntimeError(
+"Can not create a database connection. Specify either db_port 
and db_password "
+"or unix_socket. You can not specify both at the same time."
+)
+
+self.connection = pymysql.connect(
+host=db_host,
+port=db_port,
+db=db_name,
+user=db_user,
+password=db_password,
+unix_socket=unix_socket,
+autocommit=True,
+charset='utf8',
+use_unicode=True,
+)
+
+def execute(self, command, params=None, dry_run=False):
+"""
+Sends a single sql command to the server instance,
+returns metadata about the execution and the resulting data.
+"""
+result = {
+"query": command,
+"args": params,
+"host": self.db_host,
+"database": self.db_name,
+}
+if dry_run:
+log.info(
+"(DRY-RUN) Executing command: %s with params: %s", command, 
params
+)
+result.update({
+"success": True,
+"fields": [],
+"rows": [],
+"numrows": 0,
+})
+return 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: role::mariadb::analytics::custom_repl_slave: add EventLoggin...

[Elukey (Code Review)]

Elukey has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/364412 )

Change subject: role::mariadb::analytics::custom_repl_slave: add EventLogging 
cleaner user
..


role::mariadb::analytics::custom_repl_slave: add EventLogging cleaner user

This user should allow the upcoming eventlogging_cleaner.py script
to authenticate via unix socket domain in localhost to the log
database. The mysql grants are not automatically deployed, they will
need to be added manually to mariadb.

Bug: T170118
Change-Id: I39925ad25eafdce076f8f44ec09865b3c900b4ef
---
M modules/role/manifests/mariadb.pp
M modules/role/templates/mariadb/grants/production-m4.sql.erb
2 files changed, 21 insertions(+), 0 deletions(-)

Approvals:
  Marostegui: Looks good to me, but someone else must approve
  Elukey: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/role/manifests/mariadb.pp 
b/modules/role/manifests/mariadb.pp
index 32bdc0f..fcec982 100644
--- a/modules/role/manifests/mariadb.pp
+++ b/modules/role/manifests/mariadb.pp
@@ -210,6 +210,21 @@
 # Only 'replicate' this many rows at a time.
 $batch_size  = 1000
 
+group { 'eventlog':
+ensure => 'present',
+system => true,
+}
+
+user { 'eventlogcleaner':
+gid=> 'eventlog',
+shell  => '/bin/false',
+home   => '/nonexistent',
+comment=> 'EventLogging cleaner user',
+system => true,
+managehome => false,
+require=> Group['eventlog'],
+}
+
 file { '/usr/local/bin/eventlogging_sync.sh':
 ensure => present,
 owner  => 'root',
diff --git a/modules/role/templates/mariadb/grants/production-m4.sql.erb 
b/modules/role/templates/mariadb/grants/production-m4.sql.erb
index d92395a..dddcf01 100644
--- a/modules/role/templates/mariadb/grants/production-m4.sql.erb
+++ b/modules/role/templates/mariadb/grants/production-m4.sql.erb
@@ -23,3 +23,9 @@
 
 GRANT ALTER, CREATE, CREATE TEMPORARY TABLES, DELETE, DROP, INDEX, INSERT, 
LOCK TABLES, SELECT, UPDATE
 ON `log`.* TO 'eventlog'@'10.64.32.158';
+
+
+-- eventlog script responsible to sanitize/delete sensitive data over time
+
+GRANT ALTER, SELECT, UPDATE
+ON `log`.* TO 'eventlogcleaner'@'localhost' IDENTIFIED VIA unix_socket;
\ No newline at end of file

-- 
To view, visit https://gerrit.wikimedia.org/r/364412
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I39925ad25eafdce076f8f44ec09865b3c900b4ef
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: Jcrespo 
Gerrit-Reviewer: Marostegui 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: role::mariadb::analytics::custom_repl_slave: add EventLoggin...

[Elukey (Code Review)]

Elukey has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/364412 )

Change subject: role::mariadb::analytics::custom_repl_slave: add EventLogging 
cleaner user
..

role::mariadb::analytics::custom_repl_slave: add EventLogging cleaner user

This user should allow the upcoming eventlogging_cleaner.py script
to authenticate via unix socket domain in localhost to the log
database. The mysql grants are not automatically deployed, they will
need to be added manually to mariadb.

Bug: T170118
Change-Id: I39925ad25eafdce076f8f44ec09865b3c900b4ef
---
M modules/role/manifests/mariadb.pp
M modules/role/templates/mariadb/grants/production-m4.sql.erb
2 files changed, 20 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/12/364412/1

diff --git a/modules/role/manifests/mariadb.pp 
b/modules/role/manifests/mariadb.pp
index 32bdc0f..ea08d81 100644
--- a/modules/role/manifests/mariadb.pp
+++ b/modules/role/manifests/mariadb.pp
@@ -210,6 +210,21 @@
 # Only 'replicate' this many rows at a time.
 $batch_size  = 1000
 
+group { 'eventlog':
+ensure  => 'present',
+system  => true,
+}
+
+user { 'eventlogcleaner':
+gid=> 'eventlog',
+shell  => '/bin/false',
+home   => '/nonexistent',
+comment=> 'EventLogging cleaner user',
+system => true,
+managehome => false,
+require=> Group['eventlog'],
+}
+
 file { '/usr/local/bin/eventlogging_sync.sh':
 ensure => present,
 owner  => 'root',
diff --git a/modules/role/templates/mariadb/grants/production-m4.sql.erb 
b/modules/role/templates/mariadb/grants/production-m4.sql.erb
index d92395a..ffed382 100644
--- a/modules/role/templates/mariadb/grants/production-m4.sql.erb
+++ b/modules/role/templates/mariadb/grants/production-m4.sql.erb
@@ -23,3 +23,8 @@
 
 GRANT ALTER, CREATE, CREATE TEMPORARY TABLES, DELETE, DROP, INDEX, INSERT, 
LOCK TABLES, SELECT, UPDATE
 ON `log`.* TO 'eventlog'@'10.64.32.158';
+
+
+-- eventlog script responsible to sanitize/delete sensitive data over time
+
+GRANT ALTER, SELECT, UPDATE ON `log`.* TO 'eventlogcleaner'@'localhost';
\ No newline at end of file

-- 
To view, visit https://gerrit.wikimedia.org/r/364412
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I39925ad25eafdce076f8f44ec09865b3c900b4ef
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: role::piwik::server: add regular bacula backups

[Elukey (Code Review)]

Elukey has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/364195 )

Change subject: role::piwik::server: add regular bacula backups
..


role::piwik::server: add regular bacula backups

Bug: T164073
Change-Id: Id35e45c1fa848c634b6dd893927b139276c66f35
---
A modules/profile/manifests/piwik/backup.pp
M modules/role/manifests/piwik/server.pp
2 files changed, 50 insertions(+), 1 deletion(-)

Approvals:
  Alexandros Kosiaris: Looks good to me, but someone else must approve
  Elukey: Verified; Looks good to me, approved



diff --git a/modules/profile/manifests/piwik/backup.pp 
b/modules/profile/manifests/piwik/backup.pp
new file mode 100644
index 000..6183267
--- /dev/null
+++ b/modules/profile/manifests/piwik/backup.pp
@@ -0,0 +1,48 @@
+# == Class: profile::piwik::backup
+#
+# Periodical backups of the Piwik database tables using Bacula.
+#
+class profile::piwik::backup (
+$backup_username= hiera('profile::piwik::backup::username'),
+$backup_password= hiera('profile::piwik::backup::password'),
+$retention_days = 14,
+) {
+include ::profile::backup::host
+
+file { '/srv/backups':
+ensure => directory,
+owner  => 'root',
+group  => 'root',
+mode   => '0600', # implicitly 0700 for dirs
+}
+
+file { '/etc/mysql/conf.d/dumps.cnf':
+ensure  => present,
+owner   => 'root',
+group   => 'root',
+mode=> '0400',
+content => 
"[client]\nuser=${backup_username}\npassword=${backup_password}\n",
+}
+
+cron { 'delete_old_backups':
+command => "/usr/bin/find /srv/backup -mtime +${retention_days} -exec 
/bin/rm {} \\;",
+weekday => 0,
+hour=> 0,
+minute  => 0,
+user=> 'root',
+}
+
+backup::mysqlset {'piwik':
+xtrabackup   => false,
+per_db   => true,
+innodb_only  => true,
+binlog   => false,
+slave=> false,
+local_dump_dir   => '/srv/backups',
+password_file=> '/etc/mysql/conf.d/dumps.cnf',
+method   => 'predump',
+mysql_binary => '/usr/bin/mysql',
+mysqldump_binary => '/usr/bin/mysqldump',
+jobdefaults  => 
"Weekly-${profile::backup::host::day}-${profile::backup::host::pool}",
+}
+}
diff --git a/modules/role/manifests/piwik/server.pp 
b/modules/role/manifests/piwik/server.pp
index 2d4bbfe..61dee0d 100644
--- a/modules/role/manifests/piwik/server.pp
+++ b/modules/role/manifests/piwik/server.pp
@@ -4,9 +4,10 @@
 include ::standard
 include ::profile::piwik::webserver
 include ::profile::piwik::instance
+# override profile::backup::enable to disable regular backups
+include ::profile::piwik::backup
 
 # TODO - puppetization of mysql instance
-# Ref: T159136
 require_package('mysql-server')
 
 system::role { 'piwik::server':

-- 
To view, visit https://gerrit.wikimedia.org/r/364195
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Id35e45c1fa848c634b6dd893927b139276c66f35
Gerrit-PatchSet: 5
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 
Gerrit-Reviewer: Alexandros Kosiaris 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: redis::monitoring::nrpe_instance: set retry_interval to 2 mins

[Elukey (Code Review)]

Elukey has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/363791 )

Change subject: redis::monitoring::nrpe_instance: set retry_interval to 2 mins
..


redis::monitoring::nrpe_instance: set retry_interval to 2 mins

By default this check is retried 3 times with a retry_interval of 1 min.
When a slave restarts and issue a SYNC to the master it will take
a couple of minutes for it to bootstrap correctly (sync with the master
and load data in memory).
This new setting should remove daily false positives due to Redis
instance reboots (and similar use cases).

Change-Id: Ie05d8a093389169098db0cb307c968a5284fa5fd
---
M modules/redis/manifests/monitoring/nrpe_instance.pp
1 file changed, 5 insertions(+), 4 deletions(-)

Approvals:
  Alexandros Kosiaris: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/redis/manifests/monitoring/nrpe_instance.pp 
b/modules/redis/manifests/monitoring/nrpe_instance.pp
index b1cae23..e5554b8 100644
--- a/modules/redis/manifests/monitoring/nrpe_instance.pp
+++ b/modules/redis/manifests/monitoring/nrpe_instance.pp
@@ -3,9 +3,10 @@
 $port = $title
 $cmd = $::redis::monitoring::nrpe::nrpe_command
 nrpe::monitor_service { "redis_status_on_port_${port}":
-ensure=> present,
-description   => "Check health of redis instance on ${port}",
-nrpe_command  => "/usr/bin/sudo ${cmd} ${port} ${replica_warning} 
${replica_critical}",
-contact_group => 'admins',
+ensure => present,
+description=> "Check health of redis instance on ${port}",
+nrpe_command   => "/usr/bin/sudo ${cmd} ${port} ${replica_warning} 
${replica_critical}",
+contact_group  => 'admins',
+retry_interval => 2,
 }
 }

-- 
To view, visit https://gerrit.wikimedia.org/r/363791
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ie05d8a093389169098db0cb307c968a5284fa5fd
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 
Gerrit-Reviewer: Alexandros Kosiaris 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] labs/private[master]: Change Piwik's backup user/pass namespace to improve consist...

[Elukey (Code Review)]

Elukey has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/364204 )

Change subject: Change Piwik's backup user/pass namespace to improve consistency
..


Change Piwik's backup user/pass namespace to improve consistency

Change-Id: Id3a57770933d1f2ff29c0d74075349512e0d7a79
---
M hieradata/role/common/piwik/server.yaml
1 file changed, 2 insertions(+), 2 deletions(-)

Approvals:
  Elukey: Verified; Looks good to me, approved



diff --git a/hieradata/role/common/piwik/server.yaml 
b/hieradata/role/common/piwik/server.yaml
index 04b04c6..10cbfa9 100644
--- a/hieradata/role/common/piwik/server.yaml
+++ b/hieradata/role/common/piwik/server.yaml
@@ -2,5 +2,5 @@
 profile::piwik::admin_username: "admin_username"
 profile::piwik::admin_password: "admin_pass"
 profile::piwik::password_salt: "pass_sal"
-profile::piwik::backup_username: "backup_username"
-profile::piwik::backup_password: "pass_backup"
+profile::piwik::backup::username: "backup_username"
+profile::piwik::backup::password: "pass_backup"

-- 
To view, visit https://gerrit.wikimedia.org/r/364204
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Id3a57770933d1f2ff29c0d74075349512e0d7a79
Gerrit-PatchSet: 2
Gerrit-Project: labs/private
Gerrit-Branch: master
Gerrit-Owner: Elukey 
Gerrit-Reviewer: Elukey 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] labs/private[master]: Change Piwik's backup user/pass namespace to improve consist...

[Elukey (Code Review)]

Elukey has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/364204 )

Change subject: Change Piwik's backup user/pass namespace to improve consistency
..

Change Piwik's backup user/pass namespace to improve consistency

Change-Id: Id3a57770933d1f2ff29c0d74075349512e0d7a79
---
M hieradata/role/common/piwik/server.yaml
1 file changed, 2 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/labs/private 
refs/changes/04/364204/1

diff --git a/hieradata/role/common/piwik/server.yaml 
b/hieradata/role/common/piwik/server.yaml
index 04b04c6..4389c92 100644
--- a/hieradata/role/common/piwik/server.yaml
+++ b/hieradata/role/common/piwik/server.yaml
@@ -2,5 +2,5 @@
 profile::piwik::admin_username: "admin_username"
 profile::piwik::admin_password: "admin_pass"
 profile::piwik::password_salt: "pass_sal"
-profile::piwik::backup_username: "backup_username"
-profile::piwik::backup_password: "pass_backup"
+profile::piwik::backup::backup_username: "backup_username"
+profile::piwik::backup::backup_password: "pass_backup"

-- 
To view, visit https://gerrit.wikimedia.org/r/364204
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Id3a57770933d1f2ff29c0d74075349512e0d7a79
Gerrit-PatchSet: 1
Gerrit-Project: labs/private
Gerrit-Branch: master
Gerrit-Owner: Elukey 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: role::analytics_cluster::hadoop::master: add icinga check fo...

[Elukey (Code Review)]

Elukey has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/363307 )

Change subject: role::analytics_cluster::hadoop::master: add icinga check for 
HDFS topology
..


role::analytics_cluster::hadoop::master: add icinga check for HDFS topology

Bug: T163909
Change-Id: I6f1c259eb872abcca60d96add2bfa89a0546a14d
---
A modules/role/files/analytics_cluster/hadoop/check_hdfs_topology
M modules/role/manifests/analytics_cluster/hadoop/master.pp
2 files changed, 31 insertions(+), 2 deletions(-)

Approvals:
  Ottomata: Looks good to me, but someone else must approve
  Elukey: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/role/files/analytics_cluster/hadoop/check_hdfs_topology 
b/modules/role/files/analytics_cluster/hadoop/check_hdfs_topology
new file mode 100644
index 000..8654076
--- /dev/null
+++ b/modules/role/files/analytics_cluster/hadoop/check_hdfs_topology
@@ -0,0 +1,10 @@
+#!/bin/bash
+
+# The HDFS topology should not contain any host in the 'default' rack.
+sudo -u hdfs hdfs dfsadmin -printTopology | grep -q 'Rack: default'
+
+if [ $? -eq 1 ]; then
+echo "OK"
+else
+echo "CRITICAL: There is at least one node in the default rack."
+fi
\ No newline at end of file
diff --git a/modules/role/manifests/analytics_cluster/hadoop/master.pp 
b/modules/role/manifests/analytics_cluster/hadoop/master.pp
index 2ec2da4..4e8cce6 100644
--- a/modules/role/manifests/analytics_cluster/hadoop/master.pp
+++ b/modules/role/manifests/analytics_cluster/hadoop/master.pp
@@ -46,6 +46,14 @@
 require => Class['cdh::hadoop::master'],
 }
 
+file { '/usr/local/lib/nagios/plugins/check_hdfs_topology':
+ensure => present,
+source => 
'puppet:///modules/role/analytics_cluster/hadoop/check_hdfs_topology',
+mode   => '0555',
+owner  => 'root',
+group  => 'root',
+}
+
 # Include icinga alerts if production realm.
 if $::realm == 'production' {
 # Icinga process alerts for NameNode, ResourceManager and HistoryServer
@@ -76,10 +84,21 @@
 require   => Class['cdh::hadoop::master'],
 }
 
-# Allow nagios to run the check_hdfs_active_namenode as hdfs user.
+# Allow nagios to run some scripts as hdfs user.
 sudo::user { 'nagios-check_hdfs_active_namenode':
 user   => 'nagios',
-privileges => ['ALL = NOPASSWD: 
/usr/local/bin/check_hdfs_active_namenode'],
+privileges => [
+'ALL = NOPASSWD: /usr/local/bin/check_hdfs_active_namenode',
+'ALL = NOPASSWD: 
/usr/local/lib/nagios/plugins/check_hdfs_topology',
+],
+}
+# Alert if the HDFS topology shows any inconsistency.
+nrpe::monitor_service { 'check_hdfs_topology':
+description=> 'HDFS topology check',
+nrpe_command   => '/usr/bin/sudo 
/usr/local/lib/nagios/plugins/check_hdfs_topology',
+check_interval => 30,
+retries=> 2,
+require=> 
File['/usr/local/lib/nagios/plugins/check_hdfs_topology'],
 }
 # Alert if there is no active NameNode
 nrpe::monitor_service { 'hadoop-hdfs-active-namenode':

-- 
To view, visit https://gerrit.wikimedia.org/r/363307
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I6f1c259eb872abcca60d96add2bfa89a0546a14d
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: Ottomata 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Use parallelism to sqoop large tables

[Elukey (Code Review)]

Elukey has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/363846 )

Change subject: Use parallelism to sqoop large tables
..


Use parallelism to sqoop large tables

Related Change: I4b6c496d77f855d5d6f2025297c94dcc39de3a37 should be
merged and deployed first

Bug: T169782
Change-Id: I5f87d558f05b09c54dc87319036f36ef01043e3a
---
M modules/role/manifests/analytics_cluster/refinery/job/sqoop_mediawiki.pp
1 file changed, 4 insertions(+), 1 deletion(-)

Approvals:
  Mforns: Looks good to me, but someone else must approve
  Elukey: Verified; Looks good to me, approved



diff --git 
a/modules/role/manifests/analytics_cluster/refinery/job/sqoop_mediawiki.pp 
b/modules/role/manifests/analytics_cluster/refinery/job/sqoop_mediawiki.pp
index 296fb21..8653733 100644
--- a/modules/role/manifests/analytics_cluster/refinery/job/sqoop_mediawiki.pp
+++ b/modules/role/manifests/analytics_cluster/refinery/job/sqoop_mediawiki.pp
@@ -23,10 +23,13 @@
 $log_file = 
"${::role::analytics_cluster::refinery::log_dir}/sqoop-mediawiki.log"
 # number of parallel processors to use when sqooping (querying MySQL)
 $num_processors   = 3
+# number of sqoop mappers to use, but only for tables on big wiki
+$num_mappers  = 4
+# pre-compiled set of java classes for sqoop's convenience
 $orm_jar_file = 
"${::role::analytics_cluster::refinery::path}/artifacts/mediawiki-tables-sqoop-orm.jar"
 
 cron { 'refinery-sqoop-mediawiki':
-command  => "${env} && /usr/bin/python3 
${role::analytics_cluster::refinery::path}/bin/sqoop-mediawiki-tables 
--job-name sqoop-mediawiki-monthly-$(/bin/date --date=\"$(/bin/date 
+\\%Y-\\%m-15) -1 month\" +'\\%Y-\\%m') --labsdb --jdbc-host ${db_host} 
--output-dir ${$output_directory} --wiki-file  ${wiki_file} --jar-file 
${orm_jar_file} --user ${db_user} --password-file ${db_password_file} 
--timestamp \$(/bin/date '+\\%Y\\%m0100') --snapshot \$(/bin/date 
--date=\"$(/bin/date +\\%Y-\\%m-15) -1 month\" +'\\%Y-\\%m') -k 
${num_processors} >> ${log_file} 2>&1",
+command  => "${env} && /usr/bin/python3 
${role::analytics_cluster::refinery::path}/bin/sqoop-mediawiki-tables 
--job-name sqoop-mediawiki-monthly-$(/bin/date --date=\"$(/bin/date 
+\\%Y-\\%m-15) -1 month\" +'\\%Y-\\%m') --labsdb --jdbc-host ${db_host} 
--output-dir ${$output_directory} --wiki-file  ${wiki_file} --jar-file 
${orm_jar_file} --user ${db_user} --password-file ${db_password_file} 
--timestamp \$(/bin/date '+\\%Y\\%m0100') --snapshot \$(/bin/date 
--date=\"$(/bin/date +\\%Y-\\%m-15) -1 month\" +'\\%Y-\\%m') --mappers 
${num_mappers} --processors ${num_processors} >> ${log_file} 2>&1",
 user => 'hdfs',
 minute   => '0',
 hour => '0',

-- 
To view, visit https://gerrit.wikimedia.org/r/363846
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I5f87d558f05b09c54dc87319036f36ef01043e3a
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Milimetric 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: Mforns 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] labs/private[master]: Add fake Piwik backup user/password

[Elukey (Code Review)]

Elukey has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/364196 )

Change subject: Add fake Piwik backup user/password
..


Add fake Piwik backup user/password

Change-Id: I692114c3351074d405404fe8618effe49960aea7
---
M hieradata/role/common/piwik/server.yaml
1 file changed, 2 insertions(+), 0 deletions(-)

Approvals:
  Elukey: Verified; Looks good to me, approved



diff --git a/hieradata/role/common/piwik/server.yaml 
b/hieradata/role/common/piwik/server.yaml
index b06fdaf..04b04c6 100644
--- a/hieradata/role/common/piwik/server.yaml
+++ b/hieradata/role/common/piwik/server.yaml
@@ -2,3 +2,5 @@
 profile::piwik::admin_username: "admin_username"
 profile::piwik::admin_password: "admin_pass"
 profile::piwik::password_salt: "pass_sal"
+profile::piwik::backup_username: "backup_username"
+profile::piwik::backup_password: "pass_backup"

-- 
To view, visit https://gerrit.wikimedia.org/r/364196
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I692114c3351074d405404fe8618effe49960aea7
Gerrit-PatchSet: 1
Gerrit-Project: labs/private
Gerrit-Branch: master
Gerrit-Owner: Elukey 
Gerrit-Reviewer: Elukey 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] labs/private[master]: Add fake Piwik backup user/password

[Elukey (Code Review)]

Elukey has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/364196 )

Change subject: Add fake Piwik backup user/password
..

Add fake Piwik backup user/password

Change-Id: I692114c3351074d405404fe8618effe49960aea7
---
M hieradata/role/common/piwik/server.yaml
1 file changed, 2 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/labs/private 
refs/changes/96/364196/1

diff --git a/hieradata/role/common/piwik/server.yaml 
b/hieradata/role/common/piwik/server.yaml
index b06fdaf..04b04c6 100644
--- a/hieradata/role/common/piwik/server.yaml
+++ b/hieradata/role/common/piwik/server.yaml
@@ -2,3 +2,5 @@
 profile::piwik::admin_username: "admin_username"
 profile::piwik::admin_password: "admin_pass"
 profile::piwik::password_salt: "pass_sal"
+profile::piwik::backup_username: "backup_username"
+profile::piwik::backup_password: "pass_backup"

-- 
To view, visit https://gerrit.wikimedia.org/r/364196
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I692114c3351074d405404fe8618effe49960aea7
Gerrit-PatchSet: 1
Gerrit-Project: labs/private
Gerrit-Branch: master
Gerrit-Owner: Elukey 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: role::piwik::server: add regular bacula backups

[Elukey (Code Review)]

Elukey has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/364195 )

Change subject: role::piwik::server: add regular bacula backups
..

role::piwik::server: add regular bacula backups

Bug: T164073
Change-Id: Id35e45c1fa848c634b6dd893927b139276c66f35
---
A modules/profile/manifests/piwik/backup.pp
M modules/role/manifests/piwik/server.pp
2 files changed, 41 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/95/364195/1

diff --git a/modules/profile/manifests/piwik/backup.pp 
b/modules/profile/manifests/piwik/backup.pp
new file mode 100644
index 000..c9909bb
--- /dev/null
+++ b/modules/profile/manifests/piwik/backup.pp
@@ -0,0 +1,39 @@
+# == Class: profile::piwik::backup
+#
+# Periodical backups of the Piwik database tables using Bacula.
+#
+class profile::piwik::backup (
+$backup_username= hiera('profile::piwik::backup_username'),
+$backup_password= hiera('profile::piwik::backup_password'),
+) {
+include ::profile::backup::host
+
+file { '/srv/backups':
+ensure => directory,
+owner  => 'root',
+group  => 'root',
+mode   => '0600', # implicitly 0700 for dirs
+}
+
+file { '/etc/mysql/conf.d/dumps.cnf':
+ensure  => present,
+owner   => 'root',
+group   => 'root',
+mode=> '0400',
+content => 
"[client]\nuser=${backup_username}\npassword=${backup_password}\n",
+}
+
+backup::mysqlset {'piwik':
+xtrabackup   => false,
+per_db   => true,
+innodb_only  => true,
+binlog   => false,
+slave=> false,
+local_dump_dir   => '/srv/backups',
+password_file=> '/etc/mysql/conf.d/dumps.cnf',
+method   => 'predump',
+mysql_binary => '/usr/bin/mysql',
+mysqldump_binary => '/usr/bin/mysqldump',
+jobdefaults  => 
"Weekly-${profile::backup::host::day}-${profile::backup::host::pool}",
+}
+}
diff --git a/modules/role/manifests/piwik/server.pp 
b/modules/role/manifests/piwik/server.pp
index 2d4bbfe..61dee0d 100644
--- a/modules/role/manifests/piwik/server.pp
+++ b/modules/role/manifests/piwik/server.pp
@@ -4,9 +4,10 @@
 include ::standard
 include ::profile::piwik::webserver
 include ::profile::piwik::instance
+# override profile::backup::enable to disable regular backups
+include ::profile::piwik::backup
 
 # TODO - puppetization of mysql instance
-# Ref: T159136
 require_package('mysql-server')
 
 system::role { 'piwik::server':

-- 
To view, visit https://gerrit.wikimedia.org/r/364195
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Id35e45c1fa848c634b6dd893927b139276c66f35
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: use 'require_package' for stats packages including python-yaml

[Elukey (Code Review)]

Elukey has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/363382 )

Change subject: use 'require_package' for stats packages including python-yaml
..


use 'require_package' for stats packages including python-yaml

python-yaml is required by prometheus node_puppet_agent, and
ensure_packages doesn't apparently play nice with require_package.

Change-Id: I93d971fc91ddd481abe9fcf8a00183c3e66bae5d
---
M modules/statistics/manifests/packages.pp
1 file changed, 6 insertions(+), 6 deletions(-)

Approvals:
  Elukey: Looks good to me, approved
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, but someone else must approve
  Dzahn: Looks good to me, but someone else must approve



diff --git a/modules/statistics/manifests/packages.pp 
b/modules/statistics/manifests/packages.pp
index 38683ea..39248ca 100644
--- a/modules/statistics/manifests/packages.pp
+++ b/modules/statistics/manifests/packages.pp
@@ -6,7 +6,7 @@
 include ::geoip
 include ::imagemagick::install
 
-ensure_packages([
+require_package([
 'emacs',
 'mc',
 'zip',
@@ -45,7 +45,7 @@
 ])
 
 # Python packages
-ensure_packages ([
+require_package ([
 'python-geoip',
 'libapache2-mod-python',
 'python-mysqldb',
@@ -70,13 +70,13 @@
 
 # This is a custom package and currently not available on jessie, don't 
install on jessie for now
 if os_version('ubuntu >= trusty') {
-ensure_packages([
+require_package([
 'python-pygeoip', # For geo-encoding IP addresses
 ])
 }
 
 # FORTRAN packages (T89414)
-ensure_packages([
+require_package([
 'gfortran',# GNU Fortran 95 compiler
 'gfortran-4.8',# Requested by bearloga (Mikhail); see T147682 and 
http://stackoverflow.com/a/36034866/1091835 for more info
 'liblapack-dev',   # FORTRAN library of linear algebra routines
@@ -84,7 +84,7 @@
 ])
 
 # Plotting packages
-ensure_packages([
+require_package([
 'ploticus',
 'libploticus0',
 'libcairo2',
@@ -119,7 +119,7 @@
 # spell checker/dictionary packages for research (halfak)
 # T99030 - for machine learning and natural language processing
 # T121011 - for vandalism detection
-ensure_packages([
+require_package([
 'enchant', # generic spell checking library (uses myspell as 
backend)
 'aspell-id',   # Indonesian dictionary for GNU aspell
 'hunspell-vi', # Vietnamese dictionary for hunspell

-- 
To view, visit https://gerrit.wikimedia.org/r/363382
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I93d971fc91ddd481abe9fcf8a00183c3e66bae5d
Gerrit-PatchSet: 5
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: ArielGlenn 
Gerrit-Reviewer: ArielGlenn 
Gerrit-Reviewer: Dzahn 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Set ladsgroup as absented user

[Elukey (Code Review)]

Elukey has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/364104 )

Change subject: Set ladsgroup as absented user
..


Set ladsgroup as absented user

Follow up of: https://gerrit.wikimedia.org/r/#/c/364102

Change-Id: I858a489b02e50a1e0d719f32459e553d1d996ae1
---
M modules/admin/data/data.yaml
1 file changed, 1 insertion(+), 1 deletion(-)

Approvals:
  Elukey: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/admin/data/data.yaml b/modules/admin/data/data.yaml
index d7402c8..fcd9d4e 100644
--- a/modules/admin/data/data.yaml
+++ b/modules/admin/data/data.yaml
@@ -7,7 +7,7 @@
   johnflewis, marc, jkrauska, akumar, mnoushad, spage, tnegrin, 
msyed, kleduc,
   manybubbles, haithams, jzerebecki, ashwinpp, ironholds, robla, 
asherman,
   laner, declerambaul, srijan, junikowski, krenair, springle, 
jhobs, vbaranetsky,
-  yurik, bsimmers, bcohn, hjiang, tomasz, csteipp, zareen, 
jgirault]
+  yurik, bsimmers, bcohn, hjiang, tomasz, csteipp, zareen, 
jgirault, ladsgroup]
   absent_ldap:
 description: meta group for absented users which had privileged LDAP 
access in the past
 members: [siddharth11, adavenport]

-- 
To view, visit https://gerrit.wikimedia.org/r/364104
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I858a489b02e50a1e0d719f32459e553d1d996ae1
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Set ladsgroup as absented user

[Elukey (Code Review)]

Elukey has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/364104 )

Change subject: Set ladsgroup as absented user
..

Set ladsgroup as absented user

Follow up of: https://gerrit.wikimedia.org/r/#/c/364102

Change-Id: I858a489b02e50a1e0d719f32459e553d1d996ae1
---
M modules/admin/data/data.yaml
1 file changed, 1 insertion(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/04/364104/1

diff --git a/modules/admin/data/data.yaml b/modules/admin/data/data.yaml
index d7402c8..fcd9d4e 100644
--- a/modules/admin/data/data.yaml
+++ b/modules/admin/data/data.yaml
@@ -7,7 +7,7 @@
   johnflewis, marc, jkrauska, akumar, mnoushad, spage, tnegrin, 
msyed, kleduc,
   manybubbles, haithams, jzerebecki, ashwinpp, ironholds, robla, 
asherman,
   laner, declerambaul, srijan, junikowski, krenair, springle, 
jhobs, vbaranetsky,
-  yurik, bsimmers, bcohn, hjiang, tomasz, csteipp, zareen, 
jgirault]
+  yurik, bsimmers, bcohn, hjiang, tomasz, csteipp, zareen, 
jgirault, ladsgroup]
   absent_ldap:
 description: meta group for absented users which had privileged LDAP 
access in the past
 members: [siddharth11, adavenport]

-- 
To view, visit https://gerrit.wikimedia.org/r/364104
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I858a489b02e50a1e0d719f32459e553d1d996ae1
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Remove ladsgroup from production access

[Elukey (Code Review)]

Elukey has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/364102 )

Change subject: Remove ladsgroup from production access
..


Remove ladsgroup from production access

Change-Id: I5ffe3ba7acc92de3835d0657a7b486b317267b9e
---
M modules/admin/data/data.yaml
1 file changed, 7 insertions(+), 9 deletions(-)

Approvals:
  Elukey: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/admin/data/data.yaml b/modules/admin/data/data.yaml
index dd0a5c9..d7402c8 100644
--- a/modules/admin/data/data.yaml
+++ b/modules/admin/data/data.yaml
@@ -61,7 +61,7 @@
   tgr, phuedx, ejegg, twentyafterfour, legoktm, catrope,
   mobrovac, nuria, thcipriani, joal, eevans, mforns, dpatrick, 
dcausse,
   bsitzmann, mholloway-shell, dduvall, dereckson,
-  ladsgroup, bawolff, zfilipin, addshore, niharika29, ppchelko, 
pmiazga,
+  bawolff, zfilipin, addshore, niharika29, ppchelko, pmiazga,
   musikanimal]
 privileges: ['ALL = (www-data,apache,mwdeploy,l10nupdate) NOPASSWD: ALL',
  'ALL = NOPASSWD: /sbin/restart hhvm',
@@ -200,7 +200,7 @@
   deskana, dduvall, nettrom, mforns, jkatz, ebernhardson,
   mlitn, tbayer, joal, kartik, nikerabbit, pcoombe,
   neilpquinn-wmf, maxsem, jminor, dpatrick, atgomez, dstrine,
-  ladsgroup, ovasileva]
+  ovasileva]
   statistics-admins:
 posix_name: stats
 description: access files created by stats user cron jobs
@@ -241,7 +241,7 @@
   bmansurov, atgomez, amire80, kartik, pcoombe, foks, mpany, 
jsamra,
   jdlrobson, chelsyx, ovasileva, mtizzoni, panisson, paolotti, 
ciro,
   psinger, flemmerich, mkroetzsch, akrausetud, filippo, nschaaf, 
pmiazga, faidon,
-  piccardi, fdans, eevans, anomie, demon, ladsgroup, musikanimal, 
joewalsh,
+  piccardi, fdans, eevans, anomie, demon, musikanimal, joewalsh,
   kaldari, goransm, ema, dworley]
   analytics-admins:
 gid: 732
@@ -411,7 +411,7 @@
 gid: 763
 description: Service deploy users
 members: [bd808, eevans, gwicke, mobrovac, ppchelko, demon, 
twentyafterfour, thcipriani, dduvall,
-  bsitzmann, mholloway-shell, kartik, halfak, ladsgroup, 
smalyshev, hoo,
+  bsitzmann, mholloway-shell, kartik, halfak, smalyshev, hoo,
   ssastry, cscott, arlolra, jdlrobson, maxsem]
 privileges: []
 
@@ -541,7 +541,7 @@
   ores-admin:
 description: ORES admins
 gid: 782
-members: [awight, halfak, ladsgroup]
+members: [awight, halfak]
 privileges: ['ALL = NOPASSWD: /usr/sbin/service uwsgi-ores *',
  'ALL = NOPASSWD: /usr/sbin/service celery-ores-worker *']
   sectools-roots:
@@ -989,13 +989,11 @@
 uid: 2008
 email: ttij...@wikimedia.org
   ladsgroup:
-ensure: present
+ensure: absent
 gid: 500
 name: ladsgroup
 realname: Amir Sarabadani
-ssh_keys:
-   - ssh-rsa 
B3NzaC1yc2EDAQABAAACAQC4VTfdeTSboPh/af2zT6ov13A7kSygXEspRgq3/d2BWcNFvwIIpgtpCnHzA6YPvuhrjksw34jk3OvuuQoTyWYfuwql8Wu8O9f32WD4dhrKhB8Rchxj6rrncBrGS5k0HeLr+HMrknVSSXGH60GAFNRGYLCGA+zBXLAja+xROTipIRHOF0SLlsfpY04HTrEtKcKdj8n9BnGpbT7vFp+AmXeSHg9DwRn6yvem3x1ghBFdfg+XwmIqsV4wR2b7GLX2Y6+ei53657v2VxVsORdfn4yV7FcjeVYnwNxMOVMfYcFCz+1LU9ROPDw5sCqyDPMtyQPxExiprKNT9UHkUj7aqE61o4AF1vSb9AEdUpjGgiMl+Dr8Sd+T2sNOq5uFxtWSfuNqgGHY+nOYPmSHsxthwXc7CH7x/eBDefUZwTdwX8Vf/FYOAyiviknCzwLoAbkQwRjOwFZVqJMwRwRyMToKODTTgOM0s3g/5P8n5AYwTLt1I6dLeC3nju3lywyeWb6DkFOB48mg561AU1soTnSKffse7AtJ5Ye3GFEBKxBCRJUpc7EVesOhjTdcmBWYeSSmwxHCjiCact4tyk8ZiLuRqd3EZ0KFT/6W5d4csrJUwBDNWo+7M69gVIMn/zDCy3ZViGMsqPFfXSrpKw8k0WtPBTKdHFPJ0KoLERctzkrNkfBTVw==
 Ladsgroup @ prod
-   - ssh-rsa 
B3NzaC1yc2EDAQABAAABAQCY5EyIRrtuxDgh8SOkTQl8tYAu5ZOGMO0MA9RiqMb6za8ossDmKCwD5JKwooYmT8yFA5JnvyOC0adsCCA+enylrX6r6FK0ur+dekfCGF/dsyxSJU9aw6WwVr2BThrfPjzbMR1IuN4I0t134fYE0VUvibod1kGoxSGYdWvqEFf+v7b2MqWtU+xASBFEbrx2knqVBjyy6+O8sUikfKX+vyCSxiQa2GE66LM8SuX+0JnhlDNuXleQzdFpJ9novw/+/1KqOJOg3i8/Jb1/AEMf/eChombWdjNccGtPueRyJAXEl2tCuLvt9Vf7PBZaoxhAV0Bo0GcwRVBwvsyaNDuVabLJ
+ssh_keys: []
 uid: 3182
 email: amir.tafreshi_...@wikimedia.de
   manybubbles:

-- 
To view, visit https://gerrit.wikimedia.org/r/364102
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I5ffe3ba7acc92de3835d0657a7b486b317267b9e
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Remove ladsgroup from production access

[Elukey (Code Review)]

Elukey has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/364102 )

Change subject: Remove ladsgroup from production access
..

Remove ladsgroup from production access

Change-Id: I5ffe3ba7acc92de3835d0657a7b486b317267b9e
---
M modules/admin/data/data.yaml
1 file changed, 7 insertions(+), 9 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/02/364102/1

diff --git a/modules/admin/data/data.yaml b/modules/admin/data/data.yaml
index dd0a5c9..d7402c8 100644
--- a/modules/admin/data/data.yaml
+++ b/modules/admin/data/data.yaml
@@ -61,7 +61,7 @@
   tgr, phuedx, ejegg, twentyafterfour, legoktm, catrope,
   mobrovac, nuria, thcipriani, joal, eevans, mforns, dpatrick, 
dcausse,
   bsitzmann, mholloway-shell, dduvall, dereckson,
-  ladsgroup, bawolff, zfilipin, addshore, niharika29, ppchelko, 
pmiazga,
+  bawolff, zfilipin, addshore, niharika29, ppchelko, pmiazga,
   musikanimal]
 privileges: ['ALL = (www-data,apache,mwdeploy,l10nupdate) NOPASSWD: ALL',
  'ALL = NOPASSWD: /sbin/restart hhvm',
@@ -200,7 +200,7 @@
   deskana, dduvall, nettrom, mforns, jkatz, ebernhardson,
   mlitn, tbayer, joal, kartik, nikerabbit, pcoombe,
   neilpquinn-wmf, maxsem, jminor, dpatrick, atgomez, dstrine,
-  ladsgroup, ovasileva]
+  ovasileva]
   statistics-admins:
 posix_name: stats
 description: access files created by stats user cron jobs
@@ -241,7 +241,7 @@
   bmansurov, atgomez, amire80, kartik, pcoombe, foks, mpany, 
jsamra,
   jdlrobson, chelsyx, ovasileva, mtizzoni, panisson, paolotti, 
ciro,
   psinger, flemmerich, mkroetzsch, akrausetud, filippo, nschaaf, 
pmiazga, faidon,
-  piccardi, fdans, eevans, anomie, demon, ladsgroup, musikanimal, 
joewalsh,
+  piccardi, fdans, eevans, anomie, demon, musikanimal, joewalsh,
   kaldari, goransm, ema, dworley]
   analytics-admins:
 gid: 732
@@ -411,7 +411,7 @@
 gid: 763
 description: Service deploy users
 members: [bd808, eevans, gwicke, mobrovac, ppchelko, demon, 
twentyafterfour, thcipriani, dduvall,
-  bsitzmann, mholloway-shell, kartik, halfak, ladsgroup, 
smalyshev, hoo,
+  bsitzmann, mholloway-shell, kartik, halfak, smalyshev, hoo,
   ssastry, cscott, arlolra, jdlrobson, maxsem]
 privileges: []
 
@@ -541,7 +541,7 @@
   ores-admin:
 description: ORES admins
 gid: 782
-members: [awight, halfak, ladsgroup]
+members: [awight, halfak]
 privileges: ['ALL = NOPASSWD: /usr/sbin/service uwsgi-ores *',
  'ALL = NOPASSWD: /usr/sbin/service celery-ores-worker *']
   sectools-roots:
@@ -989,13 +989,11 @@
 uid: 2008
 email: ttij...@wikimedia.org
   ladsgroup:
-ensure: present
+ensure: absent
 gid: 500
 name: ladsgroup
 realname: Amir Sarabadani
-ssh_keys:
-   - ssh-rsa 
B3NzaC1yc2EDAQABAAACAQC4VTfdeTSboPh/af2zT6ov13A7kSygXEspRgq3/d2BWcNFvwIIpgtpCnHzA6YPvuhrjksw34jk3OvuuQoTyWYfuwql8Wu8O9f32WD4dhrKhB8Rchxj6rrncBrGS5k0HeLr+HMrknVSSXGH60GAFNRGYLCGA+zBXLAja+xROTipIRHOF0SLlsfpY04HTrEtKcKdj8n9BnGpbT7vFp+AmXeSHg9DwRn6yvem3x1ghBFdfg+XwmIqsV4wR2b7GLX2Y6+ei53657v2VxVsORdfn4yV7FcjeVYnwNxMOVMfYcFCz+1LU9ROPDw5sCqyDPMtyQPxExiprKNT9UHkUj7aqE61o4AF1vSb9AEdUpjGgiMl+Dr8Sd+T2sNOq5uFxtWSfuNqgGHY+nOYPmSHsxthwXc7CH7x/eBDefUZwTdwX8Vf/FYOAyiviknCzwLoAbkQwRjOwFZVqJMwRwRyMToKODTTgOM0s3g/5P8n5AYwTLt1I6dLeC3nju3lywyeWb6DkFOB48mg561AU1soTnSKffse7AtJ5Ye3GFEBKxBCRJUpc7EVesOhjTdcmBWYeSSmwxHCjiCact4tyk8ZiLuRqd3EZ0KFT/6W5d4csrJUwBDNWo+7M69gVIMn/zDCy3ZViGMsqPFfXSrpKw8k0WtPBTKdHFPJ0KoLERctzkrNkfBTVw==
 Ladsgroup @ prod
-   - ssh-rsa 
B3NzaC1yc2EDAQABAAABAQCY5EyIRrtuxDgh8SOkTQl8tYAu5ZOGMO0MA9RiqMb6za8ossDmKCwD5JKwooYmT8yFA5JnvyOC0adsCCA+enylrX6r6FK0ur+dekfCGF/dsyxSJU9aw6WwVr2BThrfPjzbMR1IuN4I0t134fYE0VUvibod1kGoxSGYdWvqEFf+v7b2MqWtU+xASBFEbrx2knqVBjyy6+O8sUikfKX+vyCSxiQa2GE66LM8SuX+0JnhlDNuXleQzdFpJ9novw/+/1KqOJOg3i8/Jb1/AEMf/eChombWdjNccGtPueRyJAXEl2tCuLvt9Vf7PBZaoxhAV0Bo0GcwRVBwvsyaNDuVabLJ
+ssh_keys: []
 uid: 3182
 email: amir.tafreshi_...@wikimedia.de
   manybubbles:

-- 
To view, visit https://gerrit.wikimedia.org/r/364102
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I5ffe3ba7acc92de3835d0657a7b486b317267b9e
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: redis::monitoring::nrpe_instance: set retry_interval to 60s

[Elukey (Code Review)]

Elukey has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/363791 )

Change subject: redis::monitoring::nrpe_instance: set retry_interval to 60s
..

redis::monitoring::nrpe_instance: set retry_interval to 60s

By default this check is retried 3 times with a retry_interval of 1s.
When a slave restarts and issue a SYNC to the master it will take
a couple of minutes for it to bootstrap correctly (sync with the master
and load data in memory).
This new setting should remove daily false positives due to Redis
instance reboots (and similar use cases). As far as I am aware
there is no point of checking a replication delay 3 times in a row
in the timespan of few seconds.

Change-Id: Ie05d8a093389169098db0cb307c968a5284fa5fd
---
M modules/redis/manifests/monitoring/nrpe_instance.pp
1 file changed, 5 insertions(+), 4 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/91/363791/1

diff --git a/modules/redis/manifests/monitoring/nrpe_instance.pp 
b/modules/redis/manifests/monitoring/nrpe_instance.pp
index b1cae23..254f1d6 100644
--- a/modules/redis/manifests/monitoring/nrpe_instance.pp
+++ b/modules/redis/manifests/monitoring/nrpe_instance.pp
@@ -3,9 +3,10 @@
 $port = $title
 $cmd = $::redis::monitoring::nrpe::nrpe_command
 nrpe::monitor_service { "redis_status_on_port_${port}":
-ensure=> present,
-description   => "Check health of redis instance on ${port}",
-nrpe_command  => "/usr/bin/sudo ${cmd} ${port} ${replica_warning} 
${replica_critical}",
-contact_group => 'admins',
+ensure => present,
+description=> "Check health of redis instance on ${port}",
+nrpe_command   => "/usr/bin/sudo ${cmd} ${port} ${replica_warning} 
${replica_critical}",
+contact_group  => 'admins',
+retry_interval => 60,
 }
 }

-- 
To view, visit https://gerrit.wikimedia.org/r/363791
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ie05d8a093389169098db0cb307c968a5284fa5fd
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/dns[master]: Temporary depool ulsfo for network issues

[Elukey (Code Review)]

Elukey has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/363598 )

Change subject: Temporary depool ulsfo for network issues
..

Temporary depool ulsfo for network issues

Change-Id: I764dba773cb2ea788ad33efa49e1a15123d0cd13
---
M admin_state
1 file changed, 1 insertion(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/dns 
refs/changes/98/363598/1

diff --git a/admin_state b/admin_state
index 54ee433..21fb1cc 100644
--- a/admin_state
+++ b/admin_state
@@ -72,3 +72,4 @@
 ### v T164610 - semi-permanent to avoid confusion until T164609
 geoip/misc-addrs/ulsfo => DOWN
 ### ^ T164610 - semi-permanent to avoid confusion until T164609
+geoip/generic-map/ulsfo => DOWN

-- 
To view, visit https://gerrit.wikimedia.org/r/363598
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I764dba773cb2ea788ad33efa49e1a15123d0cd13
Gerrit-PatchSet: 1
Gerrit-Project: operations/dns
Gerrit-Branch: master
Gerrit-Owner: Elukey 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: role::analytics_cluster::hadoop::master: add icinga check fo...

[Elukey (Code Review)]

Elukey has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/363307 )

Change subject: role::analytics_cluster::hadoop::master: add icinga check for 
HDFS topology
..

role::analytics_cluster::hadoop::master: add icinga check for HDFS topology

Bug: T163909
Change-Id: I6f1c259eb872abcca60d96add2bfa89a0546a14d
---
A modules/role/files/analytics_cluster/hadoop/check_hdfs_topology
M modules/role/manifests/analytics_cluster/hadoop/master.pp
2 files changed, 31 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/07/363307/1

diff --git a/modules/role/files/analytics_cluster/hadoop/check_hdfs_topology 
b/modules/role/files/analytics_cluster/hadoop/check_hdfs_topology
new file mode 100644
index 000..919d806
--- /dev/null
+++ b/modules/role/files/analytics_cluster/hadoop/check_hdfs_topology
@@ -0,0 +1,10 @@
+#!/bin/bash
+
+# The HDFS topology should not contain any host in the 'default' rack.
+sudo -u hdfs dfsadmin -printTopology | grep -q 'Rack: default'
+
+if [ $? -eq 1 ]; then
+echo "OK"
+else
+echo "CRITICAL: There is at least one node in the default rack."
+fi
\ No newline at end of file
diff --git a/modules/role/manifests/analytics_cluster/hadoop/master.pp 
b/modules/role/manifests/analytics_cluster/hadoop/master.pp
index 2ec2da4..4e8cce6 100644
--- a/modules/role/manifests/analytics_cluster/hadoop/master.pp
+++ b/modules/role/manifests/analytics_cluster/hadoop/master.pp
@@ -46,6 +46,14 @@
 require => Class['cdh::hadoop::master'],
 }
 
+file { '/usr/local/lib/nagios/plugins/check_hdfs_topology':
+ensure => present,
+source => 
'puppet:///modules/role/analytics_cluster/hadoop/check_hdfs_topology',
+mode   => '0555',
+owner  => 'root',
+group  => 'root',
+}
+
 # Include icinga alerts if production realm.
 if $::realm == 'production' {
 # Icinga process alerts for NameNode, ResourceManager and HistoryServer
@@ -76,10 +84,21 @@
 require   => Class['cdh::hadoop::master'],
 }
 
-# Allow nagios to run the check_hdfs_active_namenode as hdfs user.
+# Allow nagios to run some scripts as hdfs user.
 sudo::user { 'nagios-check_hdfs_active_namenode':
 user   => 'nagios',
-privileges => ['ALL = NOPASSWD: 
/usr/local/bin/check_hdfs_active_namenode'],
+privileges => [
+'ALL = NOPASSWD: /usr/local/bin/check_hdfs_active_namenode',
+'ALL = NOPASSWD: 
/usr/local/lib/nagios/plugins/check_hdfs_topology',
+],
+}
+# Alert if the HDFS topology shows any inconsistency.
+nrpe::monitor_service { 'check_hdfs_topology':
+description=> 'HDFS topology check',
+nrpe_command   => '/usr/bin/sudo 
/usr/local/lib/nagios/plugins/check_hdfs_topology',
+check_interval => 30,
+retries=> 2,
+require=> 
File['/usr/local/lib/nagios/plugins/check_hdfs_topology'],
 }
 # Alert if there is no active NameNode
 nrpe::monitor_service { 'hadoop-hdfs-active-namenode':

-- 
To view, visit https://gerrit.wikimedia.org/r/363307
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I6f1c259eb872abcca60d96add2bfa89a0546a14d
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: role::analytics_cluster::refinery::job::data_drop: drop old ...

[Elukey (Code Review)]

Elukey has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/362148 )

Change subject: role::analytics_cluster::refinery::job::data_drop: drop old 
druid data
..


role::analytics_cluster::refinery::job::data_drop: drop old druid data

We put sampled webrequest data in druid for fast analysis.
This patch provides a cron job deleting data from druid deep-storage
after 60 days to enforce our data rentention policy.

Bug: T168614
Change-Id: I7770655b8f2b98fad751a90aee6e8b771a0851b5
---
M modules/role/manifests/analytics_cluster/refinery/job/data_drop.pp
1 file changed, 17 insertions(+), 4 deletions(-)

Approvals:
  Elukey: Looks good to me, approved
  jenkins-bot: Verified
  Nuria: Looks good to me, but someone else must approve



diff --git a/modules/role/manifests/analytics_cluster/refinery/job/data_drop.pp 
b/modules/role/manifests/analytics_cluster/refinery/job/data_drop.pp
index f7b3c6a..a9283d3 100644
--- a/modules/role/manifests/analytics_cluster/refinery/job/data_drop.pp
+++ b/modules/role/manifests/analytics_cluster/refinery/job/data_drop.pp
@@ -5,13 +5,17 @@
 class role::analytics_cluster::refinery::job::data_drop {
 require ::role::analytics_cluster::refinery
 
-$webrequest_log_file = 
"${role::analytics_cluster::refinery::log_dir}/drop-webrequest-partitions.log"
-$eventlogging_log_file   = 
"${role::analytics_cluster::refinery::log_dir}/drop-eventlogging-partitions.log"
-$wdqs_extract_log_file   = 
"${role::analytics_cluster::refinery::log_dir}/drop-wdqs-extract-partitions.log"
-$mediawiki_log_file  = 
"${role::analytics_cluster::refinery::log_dir}/drop-mediawiki-log-partitions.log"
+$webrequest_log_file   = 
"${role::analytics_cluster::refinery::log_dir}/drop-webrequest-partitions.log"
+$eventlogging_log_file = 
"${role::analytics_cluster::refinery::log_dir}/drop-eventlogging-partitions.log"
+$wdqs_extract_log_file = 
"${role::analytics_cluster::refinery::log_dir}/drop-wdqs-extract-partitions.log"
+$mediawiki_log_file= 
"${role::analytics_cluster::refinery::log_dir}/drop-mediawiki-log-partitions.log"
+$druid_webrequest_log_file = 
"${role::analytics_cluster::refinery::log_dir}/drop-druid-webrequest.log"
 
 # Shortcut var to DRY up cron commands.
 $env = "export 
PYTHONPATH=\${PYTHONPATH}:${role::analytics_cluster::refinery::path}/python"
+
+# Send an email to analytics in case of failure
+$mail_to = 'analytics-ale...@wikimedia.org'
 
 # Keep this many days of raw webrequest data.
 $raw_retention_days = 31
@@ -63,4 +67,13 @@
 minute  => '25',
 hour=> '*/4',
 }
+# keep this many days of druid webrequest sampled
+$druid_webrequest_sampled_retention_days = 60
+cron {'refinery-drop-webrequest-sampled-druid':
+command => "${env} && 
${role::analytics_cluster::refinery::path}/bin/refinery-drop-druid-deep-storage-data
 -d ${druid_webrequest_sampled_retention_days} webrequest >> 
${druid_webrequest_log_file}",
+environment => "MAILTO=${mail_to}",
+user=> 'hdfs',
+minute  => '15',
+hour=> '5'
+}
 }

-- 
To view, visit https://gerrit.wikimedia.org/r/362148
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I7770655b8f2b98fad751a90aee6e8b771a0851b5
Gerrit-PatchSet: 9
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Joal 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: Joal 
Gerrit-Reviewer: Nuria 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: role::analytics_cluster::hadoop::master: add more monitors t...

[Elukey (Code Review)]

Elukey has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/363154 )

Change subject: role::analytics_cluster::hadoop::master: add more monitors to 
HDFS metrics
..


role::analytics_cluster::hadoop::master: add more monitors to HDFS metrics

Bug: T163908
Change-Id: I22910c9e216b3af5211a7c2d7053e75bd4fc745f
---
M modules/role/manifests/analytics_cluster/hadoop/master.pp
1 file changed, 33 insertions(+), 0 deletions(-)

Approvals:
  Elukey: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/role/manifests/analytics_cluster/hadoop/master.pp 
b/modules/role/manifests/analytics_cluster/hadoop/master.pp
index c078465..2ec2da4 100644
--- a/modules/role/manifests/analytics_cluster/hadoop/master.pp
+++ b/modules/role/manifests/analytics_cluster/hadoop/master.pp
@@ -92,6 +92,39 @@
 ],
 }
 
+# Alert if the HDFS space consumption raises above a safe threshold.
+monitoring::graphite_threshold { 'hadoop-hdfs-percent-used':
+description   => 'HDFS capacity used percentage',
+metric=> 
"Hadoop.NameNode.${::hostname}_eqiad_wmnet_9980.Hadoop.NameNode.NameNodeInfo.PercentUsed.mean",
+from  => '30min',
+warning   => 70,
+critical  => 80,
+percentage=> '60',
+contact_group => 'analytics',
+}
+
+# Alert in case of HDFS currupted or missing blocks. In the ideal state
+# these values should always be 0.
+monitoring::graphite_threshold { 'hadoop-hdfs-corrupt-blocks':
+description   => 'HDFS missing blocks',
+metric=> 
"Hadoop.NameNode.${::hostname}_eqiad_wmnet_9980.Hadoop.NameNode.FSNamesystem.CorruptBlocks.mean",
+from  => '30min',
+warning   => 2,
+critical  => 5,
+percentage=> '60',
+contact_group => 'analytics',
+}
+
+monitoring::graphite_threshold { 'hadoop-hdfs-missing-blocks':
+description   => 'HDFS corrupted blocks',
+metric=> 
"Hadoop.NameNode.${::hostname}_eqiad_wmnet_9980.Hadoop.NameNode.FSNamesystem.MissingBlocks.mean",
+from  => '180min',
+warning   => 2,
+critical  => 5,
+percentage=> '60',
+contact_group => 'analytics',
+}
+
 # Java heap space used alerts.
 # The goal is to get alarms for long running memory leaks like T153951.
 # Only include heap size alerts if heap size is configured.

-- 
To view, visit https://gerrit.wikimedia.org/r/363154
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I22910c9e216b3af5211a7c2d7053e75bd4fc745f
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: Joal 
Gerrit-Reviewer: Ottomata 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: role::analytics_cluster::hadoop::master: add more monitors t...

[Elukey (Code Review)]

Elukey has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/363154 )

Change subject: role::analytics_cluster::hadoop::master: add more monitors to 
HDFS metrics
..

role::analytics_cluster::hadoop::master: add more monitors to HDFS metrics

Bug: T163908
Change-Id: I22910c9e216b3af5211a7c2d7053e75bd4fc745f
---
M modules/role/manifests/analytics_cluster/hadoop/master.pp
1 file changed, 33 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/54/363154/1

diff --git a/modules/role/manifests/analytics_cluster/hadoop/master.pp 
b/modules/role/manifests/analytics_cluster/hadoop/master.pp
index c078465..00c5990 100644
--- a/modules/role/manifests/analytics_cluster/hadoop/master.pp
+++ b/modules/role/manifests/analytics_cluster/hadoop/master.pp
@@ -92,6 +92,39 @@
 ],
 }
 
+# Alert if the HDFS space consumption raises above a safe threshold.
+monitoring::graphite_threshold { 'hadoop-hdfs-percent-used':
+description   => 'HDFS capacity used percentage',
+metric=> 
"Hadoop.NameNode.${::hostname}_eqiad_wmnet_9980.Hadoop.NameNode.NameNodeInfo.PercentUsed.mean",
+from  => '30min',
+warning   => 70,
+critical  => 80,
+percentage=> '60',
+contact_group => 'analytics',
+}
+
+# Alert in case of HDFS currupted or missing blocks. In the ideal state
+# these values should always be 0.
+monitoring::graphite_threshold { 'hadoop-hdfs-corrupt-blocks':
+description   => 'HDFS missing blocks',
+metric=> 
"Hadoop.NameNode.${::hostname}_eqiad_wmnet_9980.Hadoop.NameNode.NameNodeInfo.PercentUsed.mean",
+from  => '30min',
+warning   => 70,
+critical  => 80,
+percentage=> '60',
+contact_group => 'analytics',
+}
+
+monitoring::graphite_threshold { 'hadoop-hdfs-missing-blocks':
+description   => 'HDFS corrupted blocks',
+metric=> 
"Hadoop.NameNode.${::hostname}_eqiad_wmnet_9980.Hadoop.NameNode.FSNamesystem.MissingBlocks.mean",
+from  => '180min',
+warning   => 2,
+critical  => 5,
+percentage=> '60',
+contact_group => 'analytics',
+}
+
 # Java heap space used alerts.
 # The goal is to get alarms for long running memory leaks like T153951.
 # Only include heap size alerts if heap size is configured.

-- 
To view, visit https://gerrit.wikimedia.org/r/363154
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I22910c9e216b3af5211a7c2d7053e75bd4fc745f
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Set stat1005's pxe boot option to stretch

[Elukey (Code Review)]

Elukey has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/362975 )

Change subject: Set stat1005's pxe boot option to stretch
..


Set stat1005's pxe boot option to stretch

Bug: T165368
Change-Id: I1274e209fabd5943a20af7b4cc99830aa72ca5ef
---
M modules/install_server/files/dhcpd/linux-host-entries.ttyS1-115200
1 file changed, 2 insertions(+), 0 deletions(-)

Approvals:
  Elukey: Verified; Looks good to me, approved



diff --git a/modules/install_server/files/dhcpd/linux-host-entries.ttyS1-115200 
b/modules/install_server/files/dhcpd/linux-host-entries.ttyS1-115200
index 881eb15..4140ba9 100644
--- a/modules/install_server/files/dhcpd/linux-host-entries.ttyS1-115200
+++ b/modules/install_server/files/dhcpd/linux-host-entries.ttyS1-115200
@@ -6026,6 +6026,8 @@
 host stat1005 {
 hardware ethernet 18:66:DA:FA:0B:C4;
 fixed-address stat1005.eqiad.wmnet;
+option pxelinux.pathprefix "stretch-installer/";
+filename "stretch-installer/debian-installer/amd64/pxelinux.0";
 }
 
 host stat1006 {

-- 
To view, visit https://gerrit.wikimedia.org/r/362975
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I1274e209fabd5943a20af7b4cc99830aa72ca5ef
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 
Gerrit-Reviewer: Elukey 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Set stat1005's pxe boot option to stretch

[Elukey (Code Review)]

Elukey has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/362975 )

Change subject: Set stat1005's pxe boot option to stretch
..

Set stat1005's pxe boot option to stretch

Bug: T165368
Change-Id: I1274e209fabd5943a20af7b4cc99830aa72ca5ef
---
M modules/install_server/files/dhcpd/linux-host-entries.ttyS1-115200
1 file changed, 2 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/75/362975/1

diff --git a/modules/install_server/files/dhcpd/linux-host-entries.ttyS1-115200 
b/modules/install_server/files/dhcpd/linux-host-entries.ttyS1-115200
index 881eb15..4140ba9 100644
--- a/modules/install_server/files/dhcpd/linux-host-entries.ttyS1-115200
+++ b/modules/install_server/files/dhcpd/linux-host-entries.ttyS1-115200
@@ -6026,6 +6026,8 @@
 host stat1005 {
 hardware ethernet 18:66:DA:FA:0B:C4;
 fixed-address stat1005.eqiad.wmnet;
+option pxelinux.pathprefix "stretch-installer/";
+filename "stretch-installer/debian-installer/amd64/pxelinux.0";
 }
 
 host stat1006 {

-- 
To view, visit https://gerrit.wikimedia.org/r/362975
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I1274e209fabd5943a20af7b4cc99830aa72ca5ef
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Update kafkatee module sha to the latest change.

[Elukey (Code Review)]

Elukey has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/362387 )

Change subject: Update kafkatee module sha to the latest change.
..


Update kafkatee module sha to the latest change.

Change-Id: I1b6af2e2e0aace01bd70de54d8f8693af1b4f41d
Ref: https://gerrit.wikimedia.org/r/#/c/362382/
---
M modules/kafkatee
1 file changed, 1 insertion(+), 1 deletion(-)

Approvals:
  Elukey: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/kafkatee b/modules/kafkatee
index 9789ffc..2fba2f6 16
--- a/modules/kafkatee
+++ b/modules/kafkatee
@@ -1 +1 @@
-Subproject commit 9789ffc4a98bddaab2be460ccd1a1ecc7886bcda
+Subproject commit 2fba2f624b0965ba4614c11738f42814fd72762f

-- 
To view, visit https://gerrit.wikimedia.org/r/362387
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I1b6af2e2e0aace01bd70de54d8f8693af1b4f41d
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Update kafkatee module sha to the latest change.

[Elukey (Code Review)]

Elukey has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/362387 )

Change subject: Update kafkatee module sha to the latest change.
..

Update kafkatee module sha to the latest change.

Change-Id: I1b6af2e2e0aace01bd70de54d8f8693af1b4f41d
Ref: https://gerrit.wikimedia.org/r/#/c/362382/
---
M modules/kafkatee
1 file changed, 1 insertion(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/87/362387/1

diff --git a/modules/kafkatee b/modules/kafkatee
index 9789ffc..2fba2f6 16
--- a/modules/kafkatee
+++ b/modules/kafkatee
@@ -1 +1 @@
-Subproject commit 9789ffc4a98bddaab2be460ccd1a1ecc7886bcda
+Subproject commit 2fba2f624b0965ba4614c11738f42814fd72762f

-- 
To view, visit https://gerrit.wikimedia.org/r/362387
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I1b6af2e2e0aace01bd70de54d8f8693af1b4f41d
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...kafkatee[master]: Replace 'invoke-rc.d' with 'service' in logrotate config

[Elukey (Code Review)]

Elukey has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/362382 )

Change subject: Replace 'invoke-rc.d' with 'service' in logrotate config
..


Replace 'invoke-rc.d' with 'service' in logrotate config

This should allow to reload kafkatee even if no init.d script
is provided (for example when only a systemd unit is provided).

Bug: T151748
Change-Id: I356cf5094cf35886df89b67774b2aac4cb1eeb4f
---
M files/kafkatee_logrotate
1 file changed, 2 insertions(+), 2 deletions(-)

Approvals:
  Elukey: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/files/kafkatee_logrotate b/files/kafkatee_logrotate
index d5cf51e..499bbe1 100644
--- a/files/kafkatee_logrotate
+++ b/files/kafkatee_logrotate
@@ -20,6 +20,6 @@
   compress
   delaycompress
   postrotate
-invoke-rc.d kafkatee reload >/dev/null
+service kafkatee reload
   endscript
-}
\ No newline at end of file
+}

-- 
To view, visit https://gerrit.wikimedia.org/r/362382
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I356cf5094cf35886df89b67774b2aac4cb1eeb4f
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet/kafkatee
Gerrit-Branch: master
Gerrit-Owner: Elukey 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...kafkatee[master]: Replace 'invoke-rc.d' with 'service' in logrotate config

[Elukey (Code Review)]

Elukey has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/362382 )

Change subject: Replace 'invoke-rc.d' with 'service' in logrotate config
..

Replace 'invoke-rc.d' with 'service' in logrotate config

This should allow to reload kafkatee even if no init.d script
is provided (for example when only a systemd unit is provided).

Bug: T151748
Change-Id: I356cf5094cf35886df89b67774b2aac4cb1eeb4f
---
M files/kafkatee_logrotate
1 file changed, 2 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet/kafkatee 
refs/changes/82/362382/1

diff --git a/files/kafkatee_logrotate b/files/kafkatee_logrotate
index d5cf51e..499bbe1 100644
--- a/files/kafkatee_logrotate
+++ b/files/kafkatee_logrotate
@@ -20,6 +20,6 @@
   compress
   delaycompress
   postrotate
-invoke-rc.d kafkatee reload >/dev/null
+service kafkatee reload
   endscript
-}
\ No newline at end of file
+}

-- 
To view, visit https://gerrit.wikimedia.org/r/362382
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I356cf5094cf35886df89b67774b2aac4cb1eeb4f
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet/kafkatee
Gerrit-Branch: master
Gerrit-Owner: Elukey 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics/kafkatee[master]: Release version 0.1.6-1

[Elukey (Code Review)]

Elukey has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/362357 )

Change subject: Release version 0.1.6-1
..


Release version 0.1.6-1

Change-Id: If8f0b16a17cd828658e2ca68c9311daa15996dae
---
M debian/changelog
1 file changed, 7 insertions(+), 0 deletions(-)

Approvals:
  Elukey: Verified; Looks good to me, approved



diff --git a/debian/changelog b/debian/changelog
index 9554d09..83328d9 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,10 @@
+kafkatee (0.1.6-1) unstable; urgency=medium
+
+  * Remove logrotate and syslog configuration (will be handled by
+external config management like puppet).
+
+ -- Luca Toscano   Fri, 30 Jun 2017 12:13:00 +0200
+
 kafkatee (0.1.5-1) unstable; urgency=medium
 
   * Unblock signals in children processes, fixes cleanup of shell pipelines.

-- 
To view, visit https://gerrit.wikimedia.org/r/362357
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: If8f0b16a17cd828658e2ca68c9311daa15996dae
Gerrit-PatchSet: 1
Gerrit-Project: analytics/kafkatee
Gerrit-Branch: master
Gerrit-Owner: Elukey 
Gerrit-Reviewer: Elukey 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics/kafkatee[master]: Release version 0.1.6-1

[Elukey (Code Review)]

Elukey has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/362357 )

Change subject: Release version 0.1.6-1
..

Release version 0.1.6-1

Change-Id: If8f0b16a17cd828658e2ca68c9311daa15996dae
---
M debian/changelog
1 file changed, 7 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/analytics/kafkatee 
refs/changes/57/362357/1

diff --git a/debian/changelog b/debian/changelog
index 9554d09..83328d9 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,10 @@
+kafkatee (0.1.6-1) unstable; urgency=medium
+
+  * Remove logrotate and syslog configuration (will be handled by
+external config management like puppet).
+
+ -- Luca Toscano   Fri, 30 Jun 2017 12:13:00 +0200
+
 kafkatee (0.1.5-1) unstable; urgency=medium
 
   * Unblock signals in children processes, fixes cleanup of shell pipelines.

-- 
To view, visit https://gerrit.wikimedia.org/r/362357
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: If8f0b16a17cd828658e2ca68c9311daa15996dae
Gerrit-PatchSet: 1
Gerrit-Project: analytics/kafkatee
Gerrit-Branch: master
Gerrit-Owner: Elukey 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics/kafkatee[master]: Revert "Revert "Remove logrotate and syslog configuration""

[Elukey (Code Review)]

Hello Ottomata, Filippo Giunchedi, Volans,

I'd like you to do a code review.  Please visit

https://gerrit.wikimedia.org/r/362356

to review the following change.


Change subject: Revert "Revert "Remove logrotate and syslog configuration""
..

Revert "Revert "Remove logrotate and syslog configuration""

The master branch is apparently the one tracking the last debian changes, we 
don't really need a separate (confusing) branch other than master.

This reverts commit e381b6400711426711400cb24b158cbf0fc5d658.

Change-Id: I32bd2b991c12d27a78bf362644c643a37240ec84
---
D debian/75-kafkatee.conf
M debian/kafkatee.install
D debian/kafkatee.logrotate
M debian/kafkatee.postinst
4 files changed, 0 insertions(+), 29 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/analytics/kafkatee 
refs/changes/56/362356/1

diff --git a/debian/75-kafkatee.conf b/debian/75-kafkatee.conf
deleted file mode 100644
index d9c29ce..000
--- a/debian/75-kafkatee.conf
+++ /dev/null
@@ -1,2 +0,0 @@
-# rsyslogd kafkatee config.
-if $programname == 'kafkatee' then /var/log/kafkatee.log
diff --git a/debian/kafkatee.install b/debian/kafkatee.install
index ef17cfb..eb3166b 100644
--- a/debian/kafkatee.install
+++ b/debian/kafkatee.install
@@ -1,4 +1,3 @@
 kafkateeusr/bin
 debian/etc/kafkatee.confetc
 debian/etc/kafkatee.d/READMEetc/kafkatee.d
-debian/75-kafkatee.conf etc/rsyslog.d
diff --git a/debian/kafkatee.logrotate b/debian/kafkatee.logrotate
deleted file mode 100644
index 3e284d7..000
--- a/debian/kafkatee.logrotate
+++ /dev/null
@@ -1,23 +0,0 @@
-/var/log/kafkatee.log {
-  daily
-  rotate 7
-  create 0664 syslog adm
-  missingok
-  compress
-  delaycompress
-  postrotate
-service rsyslog reload
-  endscript
-}
-
-/var/cache/kafkatee/kafkatee.stats.json {
-  weekly
-  rotate 4
-  create 0644 kafkatee root
-  missingok
-  compress
-  delaycompress
-  postrotate
-service kafkatee reload
-  endscript
-}
diff --git a/debian/kafkatee.postinst b/debian/kafkatee.postinst
index 4900121..1b3033d 100755
--- a/debian/kafkatee.postinst
+++ b/debian/kafkatee.postinst
@@ -24,9 +24,6 @@
 chown -R kafkatee:kafkatee /var/cache/kafkatee
 chown -R kafkatee:kafkatee /var/run/kafkatee
 
-# force rsyslog to create /var/log/kafkatee.log on install.
-service rsyslog restart
-
 echo \
 "NOTE: Default kafkatee config files have been installed at:
 /etc/kafkatee.conf

-- 
To view, visit https://gerrit.wikimedia.org/r/362356
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I32bd2b991c12d27a78bf362644c643a37240ec84
Gerrit-PatchSet: 1
Gerrit-Project: analytics/kafkatee
Gerrit-Branch: master
Gerrit-Owner: Elukey 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: Ottomata 
Gerrit-Reviewer: Volans 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics/kafkatee[master]: Revert "Revert "Remove logrotate and syslog configuration""

[Elukey (Code Review)]

Elukey has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/362356 )

Change subject: Revert "Revert "Remove logrotate and syslog configuration""
..


Revert "Revert "Remove logrotate and syslog configuration""

The master branch is apparently the one tracking the last debian changes, we 
don't really need a separate (confusing) branch other than master.

This reverts commit e381b6400711426711400cb24b158cbf0fc5d658.

Change-Id: I32bd2b991c12d27a78bf362644c643a37240ec84
---
D debian/75-kafkatee.conf
M debian/kafkatee.install
D debian/kafkatee.logrotate
M debian/kafkatee.postinst
4 files changed, 0 insertions(+), 29 deletions(-)

Approvals:
  Elukey: Verified; Looks good to me, approved



diff --git a/debian/75-kafkatee.conf b/debian/75-kafkatee.conf
deleted file mode 100644
index d9c29ce..000
--- a/debian/75-kafkatee.conf
+++ /dev/null
@@ -1,2 +0,0 @@
-# rsyslogd kafkatee config.
-if $programname == 'kafkatee' then /var/log/kafkatee.log
diff --git a/debian/kafkatee.install b/debian/kafkatee.install
index ef17cfb..eb3166b 100644
--- a/debian/kafkatee.install
+++ b/debian/kafkatee.install
@@ -1,4 +1,3 @@
 kafkateeusr/bin
 debian/etc/kafkatee.confetc
 debian/etc/kafkatee.d/READMEetc/kafkatee.d
-debian/75-kafkatee.conf etc/rsyslog.d
diff --git a/debian/kafkatee.logrotate b/debian/kafkatee.logrotate
deleted file mode 100644
index 3e284d7..000
--- a/debian/kafkatee.logrotate
+++ /dev/null
@@ -1,23 +0,0 @@
-/var/log/kafkatee.log {
-  daily
-  rotate 7
-  create 0664 syslog adm
-  missingok
-  compress
-  delaycompress
-  postrotate
-service rsyslog reload
-  endscript
-}
-
-/var/cache/kafkatee/kafkatee.stats.json {
-  weekly
-  rotate 4
-  create 0644 kafkatee root
-  missingok
-  compress
-  delaycompress
-  postrotate
-service kafkatee reload
-  endscript
-}
diff --git a/debian/kafkatee.postinst b/debian/kafkatee.postinst
index 4900121..1b3033d 100755
--- a/debian/kafkatee.postinst
+++ b/debian/kafkatee.postinst
@@ -24,9 +24,6 @@
 chown -R kafkatee:kafkatee /var/cache/kafkatee
 chown -R kafkatee:kafkatee /var/run/kafkatee
 
-# force rsyslog to create /var/log/kafkatee.log on install.
-service rsyslog restart
-
 echo \
 "NOTE: Default kafkatee config files have been installed at:
 /etc/kafkatee.conf

-- 
To view, visit https://gerrit.wikimedia.org/r/362356
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I32bd2b991c12d27a78bf362644c643a37240ec84
Gerrit-PatchSet: 1
Gerrit-Project: analytics/kafkatee
Gerrit-Branch: master
Gerrit-Owner: Elukey 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: Ottomata 
Gerrit-Reviewer: Volans 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics/kafkatee[master]: Revert "Remove logrotate and syslog configuration"

[Elukey (Code Review)]

Hello Ottomata, Filippo Giunchedi, Volans,

I'd like you to do a code review.  Please visit

https://gerrit.wikimedia.org/r/362354

to review the following change.


Change subject: Revert "Remove logrotate and syslog configuration"
..

Revert "Remove logrotate and syslog configuration"

Merged in master branch, the correct one is the debian one.

This reverts commit 7fcf79c733812801229d55939afdc13d6d53b2af.

Change-Id: Ie65d8726e2ac45e2d989f2d5a3b95d3cd526e8cd
---
A debian/75-kafkatee.conf
M debian/kafkatee.install
A debian/kafkatee.logrotate
M debian/kafkatee.postinst
4 files changed, 29 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/analytics/kafkatee 
refs/changes/54/362354/1

diff --git a/debian/75-kafkatee.conf b/debian/75-kafkatee.conf
new file mode 100644
index 000..d9c29ce
--- /dev/null
+++ b/debian/75-kafkatee.conf
@@ -0,0 +1,2 @@
+# rsyslogd kafkatee config.
+if $programname == 'kafkatee' then /var/log/kafkatee.log
diff --git a/debian/kafkatee.install b/debian/kafkatee.install
index eb3166b..ef17cfb 100644
--- a/debian/kafkatee.install
+++ b/debian/kafkatee.install
@@ -1,3 +1,4 @@
 kafkateeusr/bin
 debian/etc/kafkatee.confetc
 debian/etc/kafkatee.d/READMEetc/kafkatee.d
+debian/75-kafkatee.conf etc/rsyslog.d
diff --git a/debian/kafkatee.logrotate b/debian/kafkatee.logrotate
new file mode 100644
index 000..3e284d7
--- /dev/null
+++ b/debian/kafkatee.logrotate
@@ -0,0 +1,23 @@
+/var/log/kafkatee.log {
+  daily
+  rotate 7
+  create 0664 syslog adm
+  missingok
+  compress
+  delaycompress
+  postrotate
+service rsyslog reload
+  endscript
+}
+
+/var/cache/kafkatee/kafkatee.stats.json {
+  weekly
+  rotate 4
+  create 0644 kafkatee root
+  missingok
+  compress
+  delaycompress
+  postrotate
+service kafkatee reload
+  endscript
+}
diff --git a/debian/kafkatee.postinst b/debian/kafkatee.postinst
index 1b3033d..4900121 100755
--- a/debian/kafkatee.postinst
+++ b/debian/kafkatee.postinst
@@ -24,6 +24,9 @@
 chown -R kafkatee:kafkatee /var/cache/kafkatee
 chown -R kafkatee:kafkatee /var/run/kafkatee
 
+# force rsyslog to create /var/log/kafkatee.log on install.
+service rsyslog restart
+
 echo \
 "NOTE: Default kafkatee config files have been installed at:
 /etc/kafkatee.conf

-- 
To view, visit https://gerrit.wikimedia.org/r/362354
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ie65d8726e2ac45e2d989f2d5a3b95d3cd526e8cd
Gerrit-PatchSet: 1
Gerrit-Project: analytics/kafkatee
Gerrit-Branch: master
Gerrit-Owner: Elukey 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: Ottomata 
Gerrit-Reviewer: Volans 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics/kafkatee[master]: Revert "Remove logrotate and syslog configuration"

[Elukey (Code Review)]

Elukey has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/362354 )

Change subject: Revert "Remove logrotate and syslog configuration"
..


Revert "Remove logrotate and syslog configuration"

Merged in master branch, the correct one is the debian one.

This reverts commit 7fcf79c733812801229d55939afdc13d6d53b2af.

Change-Id: Ie65d8726e2ac45e2d989f2d5a3b95d3cd526e8cd
---
A debian/75-kafkatee.conf
M debian/kafkatee.install
A debian/kafkatee.logrotate
M debian/kafkatee.postinst
4 files changed, 29 insertions(+), 0 deletions(-)

Approvals:
  Elukey: Verified; Looks good to me, approved



diff --git a/debian/75-kafkatee.conf b/debian/75-kafkatee.conf
new file mode 100644
index 000..d9c29ce
--- /dev/null
+++ b/debian/75-kafkatee.conf
@@ -0,0 +1,2 @@
+# rsyslogd kafkatee config.
+if $programname == 'kafkatee' then /var/log/kafkatee.log
diff --git a/debian/kafkatee.install b/debian/kafkatee.install
index eb3166b..ef17cfb 100644
--- a/debian/kafkatee.install
+++ b/debian/kafkatee.install
@@ -1,3 +1,4 @@
 kafkateeusr/bin
 debian/etc/kafkatee.confetc
 debian/etc/kafkatee.d/READMEetc/kafkatee.d
+debian/75-kafkatee.conf etc/rsyslog.d
diff --git a/debian/kafkatee.logrotate b/debian/kafkatee.logrotate
new file mode 100644
index 000..3e284d7
--- /dev/null
+++ b/debian/kafkatee.logrotate
@@ -0,0 +1,23 @@
+/var/log/kafkatee.log {
+  daily
+  rotate 7
+  create 0664 syslog adm
+  missingok
+  compress
+  delaycompress
+  postrotate
+service rsyslog reload
+  endscript
+}
+
+/var/cache/kafkatee/kafkatee.stats.json {
+  weekly
+  rotate 4
+  create 0644 kafkatee root
+  missingok
+  compress
+  delaycompress
+  postrotate
+service kafkatee reload
+  endscript
+}
diff --git a/debian/kafkatee.postinst b/debian/kafkatee.postinst
index 1b3033d..4900121 100755
--- a/debian/kafkatee.postinst
+++ b/debian/kafkatee.postinst
@@ -24,6 +24,9 @@
 chown -R kafkatee:kafkatee /var/cache/kafkatee
 chown -R kafkatee:kafkatee /var/run/kafkatee
 
+# force rsyslog to create /var/log/kafkatee.log on install.
+service rsyslog restart
+
 echo \
 "NOTE: Default kafkatee config files have been installed at:
 /etc/kafkatee.conf

-- 
To view, visit https://gerrit.wikimedia.org/r/362354
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ie65d8726e2ac45e2d989f2d5a3b95d3cd526e8cd
Gerrit-PatchSet: 1
Gerrit-Project: analytics/kafkatee
Gerrit-Branch: master
Gerrit-Owner: Elukey 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: Ottomata 
Gerrit-Reviewer: Volans 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics/kafkatee[master]: Remove logrotate and syslog configuration

[Elukey (Code Review)]

Elukey has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/354223 )

Change subject: Remove logrotate and syslog configuration
..


Remove logrotate and syslog configuration

This will allow us to easily deploy logrotate/syslog
configurations via puppet without the risk of overlapping
with the package defaults.

Bug: T151748
Change-Id: I41b0a0be339701a682d3af49e6fc64e4dc74958e
---
D debian/75-kafkatee.conf
M debian/kafkatee.install
D debian/kafkatee.logrotate
M debian/kafkatee.postinst
4 files changed, 0 insertions(+), 29 deletions(-)

Approvals:
  Ottomata: Looks good to me, but someone else must approve
  Elukey: Verified; Looks good to me, approved
  Filippo Giunchedi: Looks good to me, but someone else must approve



diff --git a/debian/75-kafkatee.conf b/debian/75-kafkatee.conf
deleted file mode 100644
index d9c29ce..000
--- a/debian/75-kafkatee.conf
+++ /dev/null
@@ -1,2 +0,0 @@
-# rsyslogd kafkatee config.
-if $programname == 'kafkatee' then /var/log/kafkatee.log
diff --git a/debian/kafkatee.install b/debian/kafkatee.install
index ef17cfb..eb3166b 100644
--- a/debian/kafkatee.install
+++ b/debian/kafkatee.install
@@ -1,4 +1,3 @@
 kafkateeusr/bin
 debian/etc/kafkatee.confetc
 debian/etc/kafkatee.d/READMEetc/kafkatee.d
-debian/75-kafkatee.conf etc/rsyslog.d
diff --git a/debian/kafkatee.logrotate b/debian/kafkatee.logrotate
deleted file mode 100644
index 3e284d7..000
--- a/debian/kafkatee.logrotate
+++ /dev/null
@@ -1,23 +0,0 @@
-/var/log/kafkatee.log {
-  daily
-  rotate 7
-  create 0664 syslog adm
-  missingok
-  compress
-  delaycompress
-  postrotate
-service rsyslog reload
-  endscript
-}
-
-/var/cache/kafkatee/kafkatee.stats.json {
-  weekly
-  rotate 4
-  create 0644 kafkatee root
-  missingok
-  compress
-  delaycompress
-  postrotate
-service kafkatee reload
-  endscript
-}
diff --git a/debian/kafkatee.postinst b/debian/kafkatee.postinst
index 4900121..1b3033d 100755
--- a/debian/kafkatee.postinst
+++ b/debian/kafkatee.postinst
@@ -24,9 +24,6 @@
 chown -R kafkatee:kafkatee /var/cache/kafkatee
 chown -R kafkatee:kafkatee /var/run/kafkatee
 
-# force rsyslog to create /var/log/kafkatee.log on install.
-service rsyslog restart
-
 echo \
 "NOTE: Default kafkatee config files have been installed at:
 /etc/kafkatee.conf

-- 
To view, visit https://gerrit.wikimedia.org/r/354223
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I41b0a0be339701a682d3af49e6fc64e4dc74958e
Gerrit-PatchSet: 2
Gerrit-Project: analytics/kafkatee
Gerrit-Branch: master
Gerrit-Owner: Elukey 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: Ottomata 
Gerrit-Reviewer: Volans 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: graphite: lower down eventstreams whisper files retention

[Elukey (Code Review)]

Elukey has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/361818 )

Change subject: graphite: lower down eventstreams whisper files retention
..


graphite: lower down eventstreams whisper files retention

This change will ensure that we'll not get regular alerts
for disk space consumption for graphite1001.

Bug: T160644
Change-Id: I969098b6148dc237cfa1aeae1832b2efde690e97
---
M modules/role/manifests/graphite/production.pp
1 file changed, 1 insertion(+), 1 deletion(-)

Approvals:
  Elukey: Looks good to me, approved
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, but someone else must approve



diff --git a/modules/role/manifests/graphite/production.pp 
b/modules/role/manifests/graphite/production.pp
index 5c09589..453fceb 100644
--- a/modules/role/manifests/graphite/production.pp
+++ b/modules/role/manifests/graphite/production.pp
@@ -38,7 +38,7 @@
 # Cleanup eventstreams rdkafka stale data - T160644
 graphite::whisper_cleanup { 'graphite-eventstreams':
 directory => "${storage_dir}/whisper/eventstreams/rdkafka",
-keep_days => 15,
+keep_days => 10,
 }
 
 $graphite_hosts = [

-- 
To view, visit https://gerrit.wikimedia.org/r/361818
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I969098b6148dc237cfa1aeae1832b2efde690e97
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: Ottomata 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: graphite: lower down eventstreams whisper files retention

[Elukey (Code Review)]

Elukey has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/361818 )

Change subject: graphite: lower down eventstreams whisper files retention
..

graphite: lower down eventstreams whisper files retention

This change will ensure that we'll not get regular alerts
for disk space consumption for graphite1001.

Bug: T160644
Change-Id: I969098b6148dc237cfa1aeae1832b2efde690e97
---
M modules/role/manifests/graphite/production.pp
1 file changed, 1 insertion(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/18/361818/1

diff --git a/modules/role/manifests/graphite/production.pp 
b/modules/role/manifests/graphite/production.pp
index 5c09589..453fceb 100644
--- a/modules/role/manifests/graphite/production.pp
+++ b/modules/role/manifests/graphite/production.pp
@@ -38,7 +38,7 @@
 # Cleanup eventstreams rdkafka stale data - T160644
 graphite::whisper_cleanup { 'graphite-eventstreams':
 directory => "${storage_dir}/whisper/eventstreams/rdkafka",
-keep_days => 15,
+keep_days => 10,
 }
 
 $graphite_hosts = [

-- 
To view, visit https://gerrit.wikimedia.org/r/361818
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I969098b6148dc237cfa1aeae1832b2efde690e97
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: contint: PHP ext build dependencies on Nodepool

[Elukey (Code Review)]

Elukey has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/342635 )

Change subject: contint: PHP ext build dependencies on Nodepool
..


contint: PHP ext build dependencies on Nodepool

Our PHP extensions luasandbox and wikidiff2 have build dependencies that
are listed in contint::packages::labs.  That class is intentionally not
applied on the Nodepool instances with the intent of eventually getting
rid of it.

Move the dependencies to contint::php to make them available to Nodepool
instances.  Will let us migrate the Jenkins jobs that compile
extensions.

Also add libtidy-dev for mediawiki/php/tidy extension.

Bug: T134381
Change-Id: I209a544397843914b6b4f9ea36ec8c2e406125ed
---
M modules/contint/manifests/packages/labs.pp
M modules/contint/manifests/packages/php.pp
2 files changed, 10 insertions(+), 3 deletions(-)

Approvals:
  Elukey: Looks good to me, approved
  jenkins-bot: Verified
  Dzahn: Looks good to me, but someone else must approve



diff --git a/modules/contint/manifests/packages/labs.pp 
b/modules/contint/manifests/packages/labs.pp
index b325c28..1cf4ce4 100644
--- a/modules/contint/manifests/packages/labs.pp
+++ b/modules/contint/manifests/packages/labs.pp
@@ -46,11 +46,8 @@
 'pep8',
 'python-simplejson',  # For mw/ext/Translate among others
 
-'luajit',
 'libevent-dev',  # PoolCounter daemon
-'liblua5.1-0-dev',
 'g++',
-'libthai-dev',
 
 'python-sphinx',  # python documentation
 ]:
diff --git a/modules/contint/manifests/packages/php.pp 
b/modules/contint/manifests/packages/php.pp
index 47bbb32..c90da5e 100644
--- a/modules/contint/manifests/packages/php.pp
+++ b/modules/contint/manifests/packages/php.pp
@@ -73,4 +73,14 @@
 }
 }
 
+# PHP Extensions dependencies (mediawiki/php/*.git)
+package { [
+'libthai-dev',  # wikidiff2
+'luajit',   # luasandbox
+'liblua5.1.0-dev',  # luasandbox
+'libtidy-dev',  # tidy
+]:
+ensure => present
+}
+
 }

-- 
To view, visit https://gerrit.wikimedia.org/r/342635
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I209a544397843914b6b4f9ea36ec8c2e406125ed
Gerrit-PatchSet: 5
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Hashar 
Gerrit-Reviewer: Dzahn 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: Hashar 
Gerrit-Reviewer: Muehlenhoff 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Enable/deploy cassandra-metrics-collector 4.0.1

[Elukey (Code Review)]

Elukey has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/361478 )

Change subject: Enable/deploy cassandra-metrics-collector 4.0.1
..


Enable/deploy cassandra-metrics-collector 4.0.1

Bug: T164274
Change-Id: I3bc32e4c15a070875814c451a824daf688a23058
---
M modules/cassandra/manifests/metrics.pp
1 file changed, 1 insertion(+), 1 deletion(-)

Approvals:
  Eevans: Looks good to me, but someone else must approve
  Elukey: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/cassandra/manifests/metrics.pp 
b/modules/cassandra/manifests/metrics.pp
index cd2c4f7..ba0c477 100644
--- a/modules/cassandra/manifests/metrics.pp
+++ b/modules/cassandra/manifests/metrics.pp
@@ -54,7 +54,7 @@
 } elsif $target_cassandra_version == '2.2' {
 $collector_version = '3.1.4-20170427.001104-1'
 } else {
-$collector_version = '4.0.0'
+$collector_version = '4.0.1'
 }
 
 package { 'cassandra/metrics-collector':

-- 
To view, visit https://gerrit.wikimedia.org/r/361478
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I3bc32e4c15a070875814c451a824daf688a23058
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Eevans 
Gerrit-Reviewer: Dzahn 
Gerrit-Reviewer: Eevans 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics/refinery[master]: Add kbp.wikipedia to the pageview whitelist

[Elukey (Code Review)]

Elukey has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/361033 )

Change subject: Add kbp.wikipedia to the pageview whitelist
..

Add kbp.wikipedia to the pageview whitelist

Change-Id: I4b231ae0658eaea12f75f67a76e19bf75a6e4e17
---
M static_data/pageview/whitelist/whitelist.tsv
1 file changed, 1 insertion(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/analytics/refinery 
refs/changes/33/361033/1

diff --git a/static_data/pageview/whitelist/whitelist.tsv 
b/static_data/pageview/whitelist/whitelist.tsv
index 8f6c4f2..37b5de0 100644
--- a/static_data/pageview/whitelist/whitelist.tsv
+++ b/static_data/pageview/whitelist/whitelist.tsv
@@ -376,6 +376,7 @@
 projectka.wikiquote2015-10-12 00:00:00
 projectka.wiktionary   2015-10-12 00:00:00
 projectkbd.wikipedia   2015-10-12 00:00:00
+projectkbp.wikipedia   2017-06-22 00:00:00
 projectkg.wikipedia2015-10-12 00:00:00
 projectki.wikipedia2015-10-12 00:00:00
 projectkj.wikipedia2015-10-12 00:00:00

-- 
To view, visit https://gerrit.wikimedia.org/r/361033
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I4b231ae0658eaea12f75f67a76e19bf75a6e4e17
Gerrit-PatchSet: 1
Gerrit-Project: analytics/refinery
Gerrit-Branch: master
Gerrit-Owner: Elukey 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: role::zookeeper: refactor to multiple profiles

[Elukey (Code Review)]

Elukey has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/354449 )

Change subject: role::zookeeper: refactor to multiple profiles
..


role::zookeeper: refactor to multiple profiles

The zookeeper_cluster_name variable has been moved
from the top level hieradata/eqiad,codfw yaml files
to each profile that uses it, namely most of the
Analytics infrastructure. To avoid namespace clashing
the Hadoop hiera variable has been renamed, waiting
for a better solution when all the Hadoop code will be
moved to profiles as well. Some old roles have been
deleted in favor of a better reuse of the new profile
classes.

Bug: T114815
Change-Id: If511394ac4a80287dd44f0c89d9f29fcf9477ad1
---
M hieradata/codfw.yaml
D hieradata/common/zookeeper.yaml
M hieradata/eqiad.yaml
D hieradata/hosts/druid1001.yaml
D hieradata/hosts/druid1002.yaml
D hieradata/hosts/druid1003.yaml
M hieradata/labs.yaml
M hieradata/role/codfw/configcluster.yaml
D hieradata/role/codfw/zookeeper/server.yaml
A hieradata/role/common/analytics_cluster/client.yaml
M hieradata/role/common/analytics_cluster/druid/worker.yaml
A hieradata/role/common/analytics_cluster/druid/zookeeper.yaml
A hieradata/role/common/analytics_cluster/hadoop/balancer.yaml
A hieradata/role/common/analytics_cluster/hadoop/client.yaml
M hieradata/role/common/analytics_cluster/hadoop/master.yaml
M hieradata/role/common/analytics_cluster/hadoop/standby.yaml
M hieradata/role/common/analytics_cluster/hadoop/worker.yaml
A hieradata/role/common/analytics_cluster/hive/client.yaml
M hieradata/role/common/analytics_cluster/hue.yaml
A hieradata/role/common/analytics_cluster/oozie/client.yaml
A hieradata/role/common/analytics_cluster/refinery.yaml
M hieradata/role/common/configcluster.yaml
M hieradata/role/eqiad/configcluster.yaml
D hieradata/role/eqiad/zookeeper/server.yaml
M manifests/site.pp
R modules/profile/manifests/zookeeper/server.pp
A modules/role/manifests/analytics_cluster/druid/zookeeper.pp
M modules/role/manifests/analytics_cluster/hadoop/client.pp
M modules/role/manifests/analytics_cluster/hive/client.pp
M modules/role/manifests/configcluster.pp
D modules/role/manifests/zookeeper/client.pp
31 files changed, 104 insertions(+), 135 deletions(-)

Approvals:
  Alexandros Kosiaris: Looks good to me, but someone else must approve
  Elukey: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/hieradata/codfw.yaml b/hieradata/codfw.yaml
index 079f800..922c685 100644
--- a/hieradata/codfw.yaml
+++ b/hieradata/codfw.yaml
@@ -20,12 +20,6 @@
 
 ganglia_aggregators: install2002.wikimedia.org:10649
 
-# Default zookeeper cluster to use in codfw.
-# The cluster config is in common.yaml in zookeeper_clusters.
-zookeeper_cluster_name: 'main-codfw'
-# Page if a zookeeper server in this cluster goes down.
-zookeeper_cluster_is_critical: true
-
 #
 # LABS
 #
diff --git a/hieradata/common/zookeeper.yaml b/hieradata/common/zookeeper.yaml
deleted file mode 100644
index 391dfe6..000
--- a/hieradata/common/zookeeper.yaml
+++ /dev/null
@@ -1,7 +0,0 @@
-# Max number of connections per IP for Zookeeper
-zookeeper::max_client_connections: 1024
-
-# Default tick_time is 2000ms, this should allow a max
-# of 16 seconds of latency for Zookeeper client sessions.
-# See comments in role::kafka::analytics::broker for more info.
-zookeeper::sync_limit: 8
diff --git a/hieradata/eqiad.yaml b/hieradata/eqiad.yaml
index 0b83525..61da1b2 100644
--- a/hieradata/eqiad.yaml
+++ b/hieradata/eqiad.yaml
@@ -64,12 +64,6 @@
 
 labs_certmanager_hostname: "labservices1001.wikimedia.org"
 
-# Default zookeeper cluster to use in eqiad.
-# The cluster config is in common.yaml in zookeeper_clusters.
-zookeeper_cluster_name: 'main-eqiad'
-# Page if a zookeeper server in this cluster goes down.
-zookeeper_cluster_is_critical: true
-
 #
 # Labs
 #
diff --git a/hieradata/hosts/druid1001.yaml b/hieradata/hosts/druid1001.yaml
deleted file mode 100644
index 4f22a10..000
--- a/hieradata/hosts/druid1001.yaml
+++ /dev/null
@@ -1,5 +0,0 @@
-# Druid nodes get their own Zookeeper cluster to isolate them
-# from the production ones.
-zookeeper_cluster_name: druid-eqiad
-# Don't page if a zookeeper server in this cluster goes down.
-zookeeper_cluster_is_critical: false
\ No newline at end of file
diff --git a/hieradata/hosts/druid1002.yaml b/hieradata/hosts/druid1002.yaml
deleted file mode 100644
index d2a9521..000
--- a/hieradata/hosts/druid1002.yaml
+++ /dev/null
@@ -1,5 +0,0 @@
-# Druid nodes get their own Zookeeper cluster to isolate them
-# from the production ones.
-zookeeper_cluster_name: druid-eqiad
-# Don't page if a zookeeper server in this cluster goes down.
-zookeeper_cluster_is_critical: false
diff --git a/hieradata/hosts/druid1003.yaml b/hieradata/hosts/druid1003.yaml
deleted file mode 100644
index d2a9521..000
--- a/hieradata/hosts/druid1003.yaml
+++ /dev/null
@@ -1,5 +0,0 @@
-# Druid nodes get their own Zookeeper 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: pivot: rename unique devices daily endpoint

[Elukey (Code Review)]

Elukey has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/358389 )

Change subject: pivot: rename unique devices daily endpoint
..


pivot: rename unique devices daily endpoint

The last_access_uniques dataset has been renamed to
unique_devices_per_domain, in order to also support
unique_devices_project_wide. This patch renames the
pivot endpoint accordingly.

Bug: T167043
Change-Id: I3dfb72791c92536a7e4c80536c7c4961836d54be
---
M modules/pivot/templates/config.yaml.erb
1 file changed, 2 insertions(+), 2 deletions(-)

Approvals:
  Elukey: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/pivot/templates/config.yaml.erb 
b/modules/pivot/templates/config.yaml.erb
index 84d6e13..00127d5 100644
--- a/modules/pivot/templates/config.yaml.erb
+++ b/modules/pivot/templates/config.yaml.erb
@@ -175,8 +175,8 @@
 introspection: autofill-all
 
   - name: unique-devices-per-domain-daily
-title: Unique Devices Daily
-description: From beginning of 2016 - Daily resolution
+title: Unique Devices Per Domain Daily
+description: From beginning of 2016 - Daily resolution - Per Domain
 clusterName: druid
 source: unique-devices-per-domain-daily
 # The refresh rule describes how often the data cube looks for new data.

-- 
To view, visit https://gerrit.wikimedia.org/r/358389
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I3dfb72791c92536a7e4c80536c7c4961836d54be
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Joal 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: pivot: rename unique devices daily endpoint

[Elukey (Code Review)]

Elukey has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/358386 )

Change subject: pivot: rename unique devices daily endpoint
..


pivot: rename unique devices daily endpoint

The last_access_uniques dataset has been renamed to
unique_devices_per_domain, in order to also support
unique_devices_project_wide. This patch renames the
pivot endpoint accordingly.

Bug: T167043
Change-Id: If2b94f7989849b25d15d206e707aaa453e5acd60
---
M modules/pivot/templates/config.yaml.erb
1 file changed, 2 insertions(+), 2 deletions(-)

Approvals:
  Elukey: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/pivot/templates/config.yaml.erb 
b/modules/pivot/templates/config.yaml.erb
index cae68a1..84d6e13 100644
--- a/modules/pivot/templates/config.yaml.erb
+++ b/modules/pivot/templates/config.yaml.erb
@@ -174,11 +174,11 @@
 # * autofill-all - (default) Introspect the datasource, automatically 
generate dimensions and measures
 introspection: autofill-all
 
-  - name: unique-devices-daily
+  - name: unique-devices-per-domain-daily
 title: Unique Devices Daily
 description: From beginning of 2016 - Daily resolution
 clusterName: druid
-source: unique-devices-daily
+source: unique-devices-per-domain-daily
 # The refresh rule describes how often the data cube looks for new data.
 # Default: 'query'/PT1M (every minute)
 refreshRule:

-- 
To view, visit https://gerrit.wikimedia.org/r/358386
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: If2b94f7989849b25d15d206e707aaa453e5acd60
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Joal 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: hhvm: force rsyslog config to create log files with www-data...

[Elukey (Code Review)]

Elukey has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/358318 )

Change subject: hhvm: force rsyslog config to create log files with www-data 
perms
..


hhvm: force rsyslog config to create log files with www-data perms

The current configuration assigns to the hhvm logrotate's config
the duty of creating the hhvm error log file with 644 www-data:www-data
permissions. In all the situations in which rsyslog needs to push
a log to /var/log/hhvm/error.log and the file is not there (most
common case if after reimage) then it will use its default perms,
namely root:adm. This causes problems when the file needs to be
removed by logrotate after 90 days, like cronspam.

Bug: T146464
Change-Id: I93d4b79bd10ef4eb60eacb879ed9e4bdaa22f08e
---
D modules/hhvm/files/hhvm.rsyslog.conf
M modules/hhvm/manifests/init.pp
A modules/hhvm/templates/hhvm.rsyslog.conf.erb
3 files changed, 7 insertions(+), 4 deletions(-)

Approvals:
  Faidon Liambotis: Looks good to me, but someone else must approve
  Ema: Looks good to me, but someone else must approve
  Elukey: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/hhvm/files/hhvm.rsyslog.conf 
b/modules/hhvm/files/hhvm.rsyslog.conf
deleted file mode 100644
index 09ebda4..000
--- a/modules/hhvm/files/hhvm.rsyslog.conf
+++ /dev/null
@@ -1,3 +0,0 @@
-# rsyslogd(8) configuration file for HHVM.
-# This file is managed by Puppet.
-:programname, startswith, "hhvm" /var/log/hhvm/error.log
diff --git a/modules/hhvm/manifests/init.pp b/modules/hhvm/manifests/init.pp
index 17e3eef..255a65f 100644
--- a/modules/hhvm/manifests/init.pp
+++ b/modules/hhvm/manifests/init.pp
@@ -277,7 +277,7 @@
 ## Run-time data and logging
 
 rsyslog::conf { 'hhvm':
-source   => 'puppet:///modules/hhvm/hhvm.rsyslog.conf',
+content  => template('hhvm/hhvm.rsyslog.conf.erb'),
 priority => 20,
 require  => File['/etc/logrotate.d/hhvm'],
 before   => Service['hhvm'],
diff --git a/modules/hhvm/templates/hhvm.rsyslog.conf.erb 
b/modules/hhvm/templates/hhvm.rsyslog.conf.erb
new file mode 100644
index 000..3e4251e
--- /dev/null
+++ b/modules/hhvm/templates/hhvm.rsyslog.conf.erb
@@ -0,0 +1,6 @@
+# rsyslogd(8) configuration file for HHVM.
+# This file is managed by Puppet.
+if ($programname startswith 'hhvm') then {
+action(type="omfile" file="/var/log/hhvm/error.log" FileCreateMode="0640" 
FileOwner="<%= @user %>" FileGroup="<%= @group %>")
+stop
+}
\ No newline at end of file

-- 
To view, visit https://gerrit.wikimedia.org/r/358318
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I93d4b79bd10ef4eb60eacb879ed9e4bdaa22f08e
Gerrit-PatchSet: 5
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: Faidon Liambotis 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: Muehlenhoff 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: hhvm: force rsyslog config to create log files with www-data...

[Elukey (Code Review)]

Elukey has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/358318 )

Change subject: hhvm: force rsyslog config to create log files with www-data 
perms
..

hhvm: force rsyslog config to create log files with www-data perms

Bug: T146464
Change-Id: I93d4b79bd10ef4eb60eacb879ed9e4bdaa22f08e
---
M modules/hhvm/files/hhvm.rsyslog.conf
1 file changed, 5 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/18/358318/1

diff --git a/modules/hhvm/files/hhvm.rsyslog.conf 
b/modules/hhvm/files/hhvm.rsyslog.conf
index 09ebda4..6be733b 100644
--- a/modules/hhvm/files/hhvm.rsyslog.conf
+++ b/modules/hhvm/files/hhvm.rsyslog.conf
@@ -1,3 +1,7 @@
 # rsyslogd(8) configuration file for HHVM.
 # This file is managed by Puppet.
-:programname, startswith, "hhvm" /var/log/hhvm/error.log
+
+if ($programname startswith == 'hhvm') then {
+action(type="omfile" file="/var/log/hhvm/error.log" FileCreateMode="0640" 
FileOwner="<%= @user %>" FileGroup="<%= @group %>")
+stop
+}
\ No newline at end of file

-- 
To view, visit https://gerrit.wikimedia.org/r/358318
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I93d4b79bd10ef4eb60eacb879ed9e4bdaa22f08e
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: beta: profile::cassandra::allow_analytics: false

[Elukey (Code Review)]

Elukey has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/357344 )

Change subject: beta: profile::cassandra::allow_analytics: false
..


beta: profile::cassandra::allow_analytics: false

deployment-aqs{01,02,03} fail puppet with:

Could not find data item profile::cassandra::allow_analytics in any
Hiera data file

The setting has been introduced in 6da30838ab. It is solely in the hiera
'role' hierarchy, however it is not looked up on deployment-prep.

Change-Id: Ia134b06da6200e4a25da4e8fe5676bdf9f3550b0
---
M hieradata/labs/deployment-prep/common.yaml
1 file changed, 1 insertion(+), 0 deletions(-)

Approvals:
  Elukey: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/hieradata/labs/deployment-prep/common.yaml 
b/hieradata/labs/deployment-prep/common.yaml
index 8f39148..0ab1324 100644
--- a/hieradata/labs/deployment-prep/common.yaml
+++ b/hieradata/labs/deployment-prep/common.yaml
@@ -124,6 +124,7 @@
   - deployment-restbase02.deployment-prep.eqiad.wmflabs
 # Cassandra (profile::cassandra)
 graphite_host: labmon1001.eqiad.wmnet
+profile::cassandra::allow_analytics: false
 profile::cassandra::metrics_whitelist: false
 profile::cassandra::metrics_blacklist:
 - .*\.metrics\.Table\..*$

-- 
To view, visit https://gerrit.wikimedia.org/r/357344
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ia134b06da6200e4a25da4e8fe5676bdf9f3550b0
Gerrit-PatchSet: 4
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Hashar 
Gerrit-Reviewer: Eevans 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: Ottomata 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Bump Debian Jessie zookeeper version to 3.4.5+dfsg-2+deb8u2

[Elukey (Code Review)]

Elukey has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/357775 )

Change subject: Bump Debian Jessie zookeeper version to 3.4.5+dfsg-2+deb8u2
..


Bump Debian Jessie zookeeper version to 3.4.5+dfsg-2+deb8u2

This workaround is needed to upgrade zookeper but it is likely
to be changed in the near future for a more maintainable solution
that doesn't require to state a specific version in puppet.

Change-Id: I1cf8ea083d4df0baaf5895bf90840a442679c2da
---
M hieradata/hosts/conf2002.yaml
M modules/role/manifests/zookeeper/client.pp
2 files changed, 2 insertions(+), 3 deletions(-)

Approvals:
  Muehlenhoff: Looks good to me, but someone else must approve
  Elukey: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/hieradata/hosts/conf2002.yaml b/hieradata/hosts/conf2002.yaml
index fa2c163..051c82f 100644
--- a/hieradata/hosts/conf2002.yaml
+++ b/hieradata/hosts/conf2002.yaml
@@ -1,2 +1 @@
-profile::etcd::replication::active: true
-zookeeper_version: 3.4.5+dfsg-2+deb8u2
\ No newline at end of file
+profile::etcd::replication::active: true
\ No newline at end of file
diff --git a/modules/role/manifests/zookeeper/client.pp 
b/modules/role/manifests/zookeeper/client.pp
index f3e54c4..c164f29 100644
--- a/modules/role/manifests/zookeeper/client.pp
+++ b/modules/role/manifests/zookeeper/client.pp
@@ -41,7 +41,7 @@
 
 $version = hiera('zookeeper_version',
 $::lsbdistcodename ? {
-'jessie'  => '3.4.5+dfsg-2',
+'jessie'  => '3.4.5+dfsg-2+deb8u2',
 'trusty'  => '3.4.5+dfsg-1',
 }
 )

-- 
To view, visit https://gerrit.wikimedia.org/r/357775
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I1cf8ea083d4df0baaf5895bf90840a442679c2da
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: Muehlenhoff 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Bump Debian Jessie zookeeper version to 3.4.5+dfsg-2+deb8u2

[Elukey (Code Review)]

Elukey has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/357775 )

Change subject: Bump Debian Jessie zookeeper version to 3.4.5+dfsg-2+deb8u2
..

Bump Debian Jessie zookeeper version to 3.4.5+dfsg-2+deb8u2

This workaround is needed to upgrade zookeper but it is likely
to be changed in the near future for a more maintainable solution
that doesn't require to state a specific version in puppet.

Change-Id: I1cf8ea083d4df0baaf5895bf90840a442679c2da
---
M hieradata/hosts/conf2002.yaml
M modules/role/manifests/zookeeper/client.pp
2 files changed, 2 insertions(+), 3 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/75/357775/1

diff --git a/hieradata/hosts/conf2002.yaml b/hieradata/hosts/conf2002.yaml
index fa2c163..051c82f 100644
--- a/hieradata/hosts/conf2002.yaml
+++ b/hieradata/hosts/conf2002.yaml
@@ -1,2 +1 @@
-profile::etcd::replication::active: true
-zookeeper_version: 3.4.5+dfsg-2+deb8u2
\ No newline at end of file
+profile::etcd::replication::active: true
\ No newline at end of file
diff --git a/modules/role/manifests/zookeeper/client.pp 
b/modules/role/manifests/zookeeper/client.pp
index f3e54c4..c164f29 100644
--- a/modules/role/manifests/zookeeper/client.pp
+++ b/modules/role/manifests/zookeeper/client.pp
@@ -41,7 +41,7 @@
 
 $version = hiera('zookeeper_version',
 $::lsbdistcodename ? {
-'jessie'  => '3.4.5+dfsg-2',
+'jessie'  => '3.4.5+dfsg-2+deb8u2',
 'trusty'  => '3.4.5+dfsg-1',
 }
 )

-- 
To view, visit https://gerrit.wikimedia.org/r/357775
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I1cf8ea083d4df0baaf5895bf90840a442679c2da
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Remove webrequest_maps topic from Camus configuration

[Elukey (Code Review)]

Elukey has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/357768 )

Change subject: Remove webrequest_maps topic from Camus configuration
..


Remove webrequest_maps topic from Camus configuration

The cache maps cluster has been deprecated and Camus
configuration needs to be updated accordingly.
Camus is the tool that Analytics uses to pull data from
Kafka and store it to HDFS for later refinement/processing.

Change-Id: I9384578e96625385467d112ab255331b8a538257
---
M modules/camus/templates/webrequest.erb
1 file changed, 1 insertion(+), 1 deletion(-)

Approvals:
  Elukey: Looks good to me, approved
  Joal: Looks good to me, but someone else must approve
  jenkins-bot: Verified



diff --git a/modules/camus/templates/webrequest.erb 
b/modules/camus/templates/webrequest.erb
index 5c87a03..dc3c501 100644
--- a/modules/camus/templates/webrequest.erb
+++ b/modules/camus/templates/webrequest.erb
@@ -71,7 +71,7 @@
 kafka.blacklist.topics=
 
 # These are the kafka topics camus brings to HDFS
-kafka.whitelist.topics=webrequest_maps,webrequest_text,webrequest_upload,webrequest_misc
+kafka.whitelist.topics=webrequest_text,webrequest_upload,webrequest_misc
 
 # Name of the client as seen by kafka
 kafka.client.name=camus-webrequest-00

-- 
To view, visit https://gerrit.wikimedia.org/r/357768
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I9384578e96625385467d112ab255331b8a538257
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: Joal 
Gerrit-Reviewer: Ottomata 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] analytics/refinery[master]: Remove any trace of the maps cluster

[Elukey (Code Review)]

Elukey has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/357769 )

Change subject: Remove any trace of the maps cluster
..

Remove any trace of the maps cluster

The cache::maps cluster has been merged to the upload one,
this commit takes care of cleaning up all the maps-related
configurations that are not needed anymore.

Change-Id: Ie64ef1bba9b2d83e195f23c8b83b483e5c2f77d1
---
M bin/refinery-dump-status-webrequest-partitions
M oozie/webrequest/datasets.xml
M oozie/webrequest/datasets_raw.xml
M oozie/webrequest/load/bundle.xml
4 files changed, 4 insertions(+), 43 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/analytics/refinery 
refs/changes/69/357769/1

diff --git a/bin/refinery-dump-status-webrequest-partitions 
b/bin/refinery-dump-status-webrequest-partitions
index 5603845..89e25bd 100755
--- a/bin/refinery-dump-status-webrequest-partitions
+++ b/bin/refinery-dump-status-webrequest-partitions
@@ -73,8 +73,8 @@
 }
 
 add_dataset "mediacounts" "daily" "   full  | top1000 |"
-add_dataset "raw_webrequest" "hourly" " maps| misc| text   
 |upload   |"
-add_dataset "webrequest" "hourly" "  maps  |  misc  |  text  | upload |"
+add_dataset "raw_webrequest" "hourly" "  misc| text|upload   |"
+add_dataset "webrequest" "hourly" "  misc  |  text  | upload |"
 add_dataset "pageview" "hourly" "  hourly  |"
 add_dataset "projectview" "hourly" "   hourly|"
 
@@ -414,7 +414,7 @@
 
 local DATE_HDFS_PADDED="$(date --utc -d "$DATE" +'%Y/%m/%d/%H')"
 
-for SOURCE in maps misc text upload
+for SOURCE in misc text upload
 do
 log_no_lf ""
 dump_dataset_raw_webrequest_partition "$DATE_HDFS_PADDED" "$SOURCE"
@@ -428,7 +428,7 @@
 local DATE_DIRS_REL="$(date --utc -d "$DATE" 
+'year=%Y/month=%m/day=%d/hour=%H')"
 DATE_DIRS_REL="${DATE_DIRS_REL//=0/=}"
 
-for SOURCE in maps misc text upload
+for SOURCE in misc text upload
 do
 local STATUS="X"
 
SUCCESS_FILE_ABS="$WEBREQUEST_DATA_DIR_ABS/webrequest_source=$SOURCE/$DATE_DIRS_REL/_SUCCESS"
diff --git a/oozie/webrequest/datasets.xml b/oozie/webrequest/datasets.xml
index 3925db3..ea67dc6 100644
--- a/oozie/webrequest/datasets.xml
+++ b/oozie/webrequest/datasets.xml
@@ -28,14 +28,6 @@
 at “${...}” as input for the second EL level. There, the variables hold
 their expected values, and we can start unpadding them.
 -->
-
-
${webrequest_data_directory}/webrequest_source=maps/year=${YEAR}/month=${"$"}{MONTH
 + 0}/day=${"$"}{DAY + 0}/hour=${"$"}{HOUR + 0}
-_SUCCESS
-
-
 
-
-
${webrequest_raw_data_directory}/webrequest_maps/hourly/${YEAR}/${MONTH}/${DAY}/${HOUR}
-_IMPORTED
-
 
-
-
${webrequest_raw_data_directory}/webrequest_maps/hourly/${YEAR}/${MONTH}/${DAY}/${HOUR}
-_PARTITIONED
-
 
-
-
${webrequest_raw_data_directory}/webrequest_maps/hourly/${YEAR}/${MONTH}/${DAY}/${HOUR}
-_SUCCESS
-
 sla_alert_contact
 
 
-
-${coordinator_file}
-
-
-webrequest_source
-maps
-
-
-
-
 
 ${coordinator_file}
 

-- 
To view, visit https://gerrit.wikimedia.org/r/357769
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ie64ef1bba9b2d83e195f23c8b83b483e5c2f77d1
Gerrit-PatchSet: 1
Gerrit-Project: analytics/refinery
Gerrit-Branch: master
Gerrit-Owner: Elukey 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Remove webrequest_maps topic from Camus configuration

[Elukey (Code Review)]

Elukey has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/357768 )

Change subject: Remove webrequest_maps topic from Camus configuration
..

Remove webrequest_maps topic from Camus configuration

The cache maps cluster has been deprecated and Camus
configuration needs to be updated accordingly.
Camus is the tool that Analytics uses to pull data from
Kafka and store it to HDFS for later refinement/processing.

Change-Id: I9384578e96625385467d112ab255331b8a538257
---
M modules/camus/templates/webrequest.erb
1 file changed, 1 insertion(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/68/357768/1

diff --git a/modules/camus/templates/webrequest.erb 
b/modules/camus/templates/webrequest.erb
index 5c87a03..dc3c501 100644
--- a/modules/camus/templates/webrequest.erb
+++ b/modules/camus/templates/webrequest.erb
@@ -71,7 +71,7 @@
 kafka.blacklist.topics=
 
 # These are the kafka topics camus brings to HDFS
-kafka.whitelist.topics=webrequest_maps,webrequest_text,webrequest_upload,webrequest_misc
+kafka.whitelist.topics=webrequest_text,webrequest_upload,webrequest_misc
 
 # Name of the client as seen by kafka
 kafka.client.name=camus-webrequest-00

-- 
To view, visit https://gerrit.wikimedia.org/r/357768
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I9384578e96625385467d112ab255331b8a538257
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Fix cassandra's jmx_prometheus_javaagent config path for dep...

[Elukey (Code Review)]

Elukey has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/357649 )

Change subject: Fix cassandra's jmx_prometheus_javaagent config path for 
deployment-prep
..


Fix cassandra's jmx_prometheus_javaagent config path for deployment-prep

Bug: T167222
Change-Id: I280dc541aa8bce3810ad7098101435c92286aa21
---
M hieradata/labs/deployment-prep/common.yaml
1 file changed, 1 insertion(+), 1 deletion(-)

Approvals:
  Elukey: Verified; Looks good to me, approved



diff --git a/hieradata/labs/deployment-prep/common.yaml 
b/hieradata/labs/deployment-prep/common.yaml
index 9fa0ee2..8f39148 100644
--- a/hieradata/labs/deployment-prep/common.yaml
+++ b/hieradata/labs/deployment-prep/common.yaml
@@ -137,7 +137,7 @@
   logstash_host: deployment-logstash2.deployment-prep.eqiad.wmflabs
   target_version: '2.2'
   additional_jvm_opts:
-- 
'-javaagent:/srv/deployment/prometheus/jmx_exporter/lib/jmx_prometheus_javaagent-0.8-20170117.190412-1.jar=7800:/etc/cassandra/jmx_exporter.yaml'
+- 
'-javaagent:/srv/deployment/prometheus/jmx_exporter/lib/jmx_prometheus_javaagent-0.8-20170117.190412-1.jar=7800:/etc/cassandra/prometheus_jmx_exporter.yaml'
 # end cassandra
 "udp2log_aggregator": deployment-fluorine02.deployment-prep.eqiad.wmflabs:8420
 "logstash_syslog": deployment-logstash2.deployment-prep.eqiad.wmflabs:10514

-- 
To view, visit https://gerrit.wikimedia.org/r/357649
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I280dc541aa8bce3810ad7098101435c92286aa21
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: Giuseppe Lavagetto 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Fix cassandra's jmx_prometheus_javaagent config path for dep...

[Elukey (Code Review)]

Elukey has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/357649 )

Change subject: Fix cassandra's jmx_prometheus_javaagent config path for 
deployment-prep
..

Fix cassandra's jmx_prometheus_javaagent config path for deployment-prep

Bug: T167222
Change-Id: I280dc541aa8bce3810ad7098101435c92286aa21
---
M hieradata/labs/deployment-prep/common.yaml
1 file changed, 1 insertion(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/49/357649/1

diff --git a/hieradata/labs/deployment-prep/common.yaml 
b/hieradata/labs/deployment-prep/common.yaml
index 9fa0ee2..8f39148 100644
--- a/hieradata/labs/deployment-prep/common.yaml
+++ b/hieradata/labs/deployment-prep/common.yaml
@@ -137,7 +137,7 @@
   logstash_host: deployment-logstash2.deployment-prep.eqiad.wmflabs
   target_version: '2.2'
   additional_jvm_opts:
-- 
'-javaagent:/srv/deployment/prometheus/jmx_exporter/lib/jmx_prometheus_javaagent-0.8-20170117.190412-1.jar=7800:/etc/cassandra/jmx_exporter.yaml'
+- 
'-javaagent:/srv/deployment/prometheus/jmx_exporter/lib/jmx_prometheus_javaagent-0.8-20170117.190412-1.jar=7800:/etc/cassandra/prometheus_jmx_exporter.yaml'
 # end cassandra
 "udp2log_aggregator": deployment-fluorine02.deployment-prep.eqiad.wmflabs:8420
 "logstash_syslog": deployment-logstash2.deployment-prep.eqiad.wmflabs:10514

-- 
To view, visit https://gerrit.wikimedia.org/r/357649
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I280dc541aa8bce3810ad7098101435c92286aa21
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Fix cassandra's jmx_prometheus_javaagent jar path for deploy...

[Elukey (Code Review)]

Elukey has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/357646 )

Change subject: Fix cassandra's jmx_prometheus_javaagent jar path for 
deployment-prep
..


Fix cassandra's jmx_prometheus_javaagent jar path for deployment-prep

Bug: T167222
Change-Id: I31a7d04c74c5599e3f25866a6e0073067d567f66
---
M hieradata/labs/deployment-prep/common.yaml
1 file changed, 1 insertion(+), 1 deletion(-)

Approvals:
  Eevans: Looks good to me, but someone else must approve
  Elukey: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/hieradata/labs/deployment-prep/common.yaml 
b/hieradata/labs/deployment-prep/common.yaml
index eb6e2c1..9fa0ee2 100644
--- a/hieradata/labs/deployment-prep/common.yaml
+++ b/hieradata/labs/deployment-prep/common.yaml
@@ -137,7 +137,7 @@
   logstash_host: deployment-logstash2.deployment-prep.eqiad.wmflabs
   target_version: '2.2'
   additional_jvm_opts:
-- 
'-javaagent:/srv/deployment/prometheus/jmx_exporter/lib/jmx_prometheus_javaagent-0.8-SNAPSHOT.jar=7800:/etc/cassandra/jmx_exporter.yaml'
+- 
'-javaagent:/srv/deployment/prometheus/jmx_exporter/lib/jmx_prometheus_javaagent-0.8-20170117.190412-1.jar=7800:/etc/cassandra/jmx_exporter.yaml'
 # end cassandra
 "udp2log_aggregator": deployment-fluorine02.deployment-prep.eqiad.wmflabs:8420
 "logstash_syslog": deployment-logstash2.deployment-prep.eqiad.wmflabs:10514

-- 
To view, visit https://gerrit.wikimedia.org/r/357646
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I31a7d04c74c5599e3f25866a6e0073067d567f66
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 
Gerrit-Reviewer: Eevans 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Fix cassandra's jmx_prometheus_javaagent jar path for deploy...

[Elukey (Code Review)]

Elukey has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/357646 )

Change subject: Fix cassandra's jmx_prometheus_javaagent jar path for 
deployment-prep
..

Fix cassandra's jmx_prometheus_javaagent jar path for deployment-prep

Bug: T167222
Change-Id: I31a7d04c74c5599e3f25866a6e0073067d567f66
---
M hieradata/labs/deployment-prep/common.yaml
1 file changed, 1 insertion(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/46/357646/1

diff --git a/hieradata/labs/deployment-prep/common.yaml 
b/hieradata/labs/deployment-prep/common.yaml
index eb6e2c1..9fa0ee2 100644
--- a/hieradata/labs/deployment-prep/common.yaml
+++ b/hieradata/labs/deployment-prep/common.yaml
@@ -137,7 +137,7 @@
   logstash_host: deployment-logstash2.deployment-prep.eqiad.wmflabs
   target_version: '2.2'
   additional_jvm_opts:
-- 
'-javaagent:/srv/deployment/prometheus/jmx_exporter/lib/jmx_prometheus_javaagent-0.8-SNAPSHOT.jar=7800:/etc/cassandra/jmx_exporter.yaml'
+- 
'-javaagent:/srv/deployment/prometheus/jmx_exporter/lib/jmx_prometheus_javaagent-0.8-20170117.190412-1.jar=7800:/etc/cassandra/jmx_exporter.yaml'
 # end cassandra
 "udp2log_aggregator": deployment-fluorine02.deployment-prep.eqiad.wmflabs:8420
 "logstash_syslog": deployment-logstash2.deployment-prep.eqiad.wmflabs:10514

-- 
To view, visit https://gerrit.wikimedia.org/r/357646
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I31a7d04c74c5599e3f25866a6e0073067d567f66
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Test new zookeeper version on conf2002

[Elukey (Code Review)]

Elukey has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/357590 )

Change subject: Test new zookeeper version on conf2002
..


Test new zookeeper version on conf2002

The zookeeper client role needs to pick up a zookeeper_version
value to avoid conflicts/overrides with the Cloudera repos.
This commit is only temporary to allow proper testing on conf2002.

Change-Id: Ibb0f246c08d3f95e5ad2540d8fdd37e6a2178e72
---
M hieradata/hosts/conf2002.yaml
1 file changed, 1 insertion(+), 0 deletions(-)

Approvals:
  Muehlenhoff: Looks good to me, but someone else must approve
  Elukey: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/hieradata/hosts/conf2002.yaml b/hieradata/hosts/conf2002.yaml
index fb3165d..fa2c163 100644
--- a/hieradata/hosts/conf2002.yaml
+++ b/hieradata/hosts/conf2002.yaml
@@ -1 +1,2 @@
 profile::etcd::replication::active: true
+zookeeper_version: 3.4.5+dfsg-2+deb8u2
\ No newline at end of file

-- 
To view, visit https://gerrit.wikimedia.org/r/357590
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ibb0f246c08d3f95e5ad2540d8fdd37e6a2178e72
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: Muehlenhoff 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Test new zookeeper version on conf2002

[Elukey (Code Review)]

Elukey has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/357590 )

Change subject: Test new zookeeper version on conf2002
..

Test new zookeeper version on conf2002

The zookeeper client role needs to pick up a zookeeper_version
value to avoid conflicts/overrides with the Cloudera repos.
This commit is only temporary to allow proper testing on conf2002.

Change-Id: Ibb0f246c08d3f95e5ad2540d8fdd37e6a2178e72
---
M hieradata/hosts/conf2002.yaml
1 file changed, 1 insertion(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/90/357590/1

diff --git a/hieradata/hosts/conf2002.yaml b/hieradata/hosts/conf2002.yaml
index fb3165d..fa2c163 100644
--- a/hieradata/hosts/conf2002.yaml
+++ b/hieradata/hosts/conf2002.yaml
@@ -1 +1,2 @@
 profile::etcd::replication::active: true
+zookeeper_version: 3.4.5+dfsg-2+deb8u2
\ No newline at end of file

-- 
To view, visit https://gerrit.wikimedia.org/r/357590
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ibb0f246c08d3f95e5ad2540d8fdd37e6a2178e72
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Bump zookeeper client version to 3.4.5+dfsg-2+deb8u2

[Elukey (Code Review)]

Elukey has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/357567 )

Change subject: Bump zookeeper client version to 3.4.5+dfsg-2+deb8u2
..

Bump zookeeper client version to 3.4.5+dfsg-2+deb8u2

Change-Id: Iac4c8e3c4a348065ee68113b19ca3ba36afdaef1
---
M modules/role/manifests/zookeeper/client.pp
1 file changed, 1 insertion(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/67/357567/1

diff --git a/modules/role/manifests/zookeeper/client.pp 
b/modules/role/manifests/zookeeper/client.pp
index f3e54c4..c164f29 100644
--- a/modules/role/manifests/zookeeper/client.pp
+++ b/modules/role/manifests/zookeeper/client.pp
@@ -41,7 +41,7 @@
 
 $version = hiera('zookeeper_version',
 $::lsbdistcodename ? {
-'jessie'  => '3.4.5+dfsg-2',
+'jessie'  => '3.4.5+dfsg-2+deb8u2',
 'trusty'  => '3.4.5+dfsg-1',
 }
 )

-- 
To view, visit https://gerrit.wikimedia.org/r/357567
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Iac4c8e3c4a348065ee68113b19ca3ba36afdaef1
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Delete unused role/common/analytics/hadoop configs

[Elukey (Code Review)]

Elukey has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/357418 )

Change subject: Delete unused role/common/analytics/hadoop configs
..


Delete unused role/common/analytics/hadoop configs

Change-Id: I7a6f44e9bb9936fc563dd5439a93cd09f102be18
---
D hieradata/role/common/analytics/hadoop/master.yaml
D hieradata/role/common/analytics/hadoop/standby.yaml
D hieradata/role/common/analytics/hadoop/worker.yaml
D hieradata/role/common/analytics/hue.yaml
D hieradata/role/common/analytics/mysql/meta.yaml
D hieradata/role/common/analytics/spark/standalone/worker.yaml
6 files changed, 0 insertions(+), 51 deletions(-)

Approvals:
  Ottomata: Looks good to me, but someone else must approve
  Elukey: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/hieradata/role/common/analytics/hadoop/master.yaml 
b/hieradata/role/common/analytics/hadoop/master.yaml
deleted file mode 100644
index a8fdc0e..000
--- a/hieradata/role/common/analytics/hadoop/master.yaml
+++ /dev/null
@@ -1,18 +0,0 @@
-nagios_group: analytics_eqiad
-cluster: analytics
-admin::groups:
-  # Users in these groups need shell accounts on the Hadoop NameNodes
-  # so that HDFS file perms for these users work properly.
-  # These groupsshould also be added to cdh/hadoop/users.yaml as the
-  # groups parameter to ensure that their hdfs homedirs get created.
-  - analytics-users
-  - analytics-privatedata-users
-  - analytics-admins
-  # role::analytics_cluster::users create system users for different
-  # shared analytics cluster file owership.  Certain user groups
-  # are allowed to sudo -u to these users in order to launch
-  # regular Hadoop jobs that don't run as real people users.
-  - analytics-search-users
-debdeploy::grains:
-  debdeploy-hadoop-master:
-value: standard
diff --git a/hieradata/role/common/analytics/hadoop/standby.yaml 
b/hieradata/role/common/analytics/hadoop/standby.yaml
deleted file mode 100644
index aecf358..000
--- a/hieradata/role/common/analytics/hadoop/standby.yaml
+++ /dev/null
@@ -1,13 +0,0 @@
-nagios_group: analytics_eqiad
-cluster: analytics
-admin::groups:
-  - analytics-users
-  - analytics-privatedata-users
-  - analytics-admins
-  # elasticsearch::analytics creates the analytics-search user and group
-  # that analytics-search-users are allowed to sudo to.  This is used
-  # for deploying files to HDFS.
-  - analytics-search-users
-debdeploy::grains:
-  debdeploy-hadoop-standby:
-value: standard
diff --git a/hieradata/role/common/analytics/hadoop/worker.yaml 
b/hieradata/role/common/analytics/hadoop/worker.yaml
deleted file mode 100644
index 40ce529..000
--- a/hieradata/role/common/analytics/hadoop/worker.yaml
+++ /dev/null
@@ -1,11 +0,0 @@
-nagios_group: analytics_eqiad
-cluster: analytics
-admin::groups:
-  - analytics-admins
-debdeploy::grains:
-  debdeploy-hadoop-worker:
-value: standard
-
-# Analytics worker disks are large.  We will install a custom
-# NRPE check for them, so the base module's should ignore them.
-profile::base::check_disk_options: -w 6% -c 3% -l -e -A -i 
"/var/lib/hadoop/data"
diff --git a/hieradata/role/common/analytics/hue.yaml 
b/hieradata/role/common/analytics/hue.yaml
deleted file mode 100644
index f3c3b72..000
--- a/hieradata/role/common/analytics/hue.yaml
+++ /dev/null
@@ -1,3 +0,0 @@
-debdeploy::grains:
-  debdeploy-hue:
-value: standard
diff --git a/hieradata/role/common/analytics/mysql/meta.yaml 
b/hieradata/role/common/analytics/mysql/meta.yaml
deleted file mode 100644
index 1c3255a..000
--- a/hieradata/role/common/analytics/mysql/meta.yaml
+++ /dev/null
@@ -1,3 +0,0 @@
-debdeploy::grains:
-  debdeploy-mysql-analytics:
-value: standard
diff --git a/hieradata/role/common/analytics/spark/standalone/worker.yaml 
b/hieradata/role/common/analytics/spark/standalone/worker.yaml
deleted file mode 100644
index 4eb118d..000
--- a/hieradata/role/common/analytics/spark/standalone/worker.yaml
+++ /dev/null
@@ -1,3 +0,0 @@
-debdeploy::grains:
-  debdeploy-spark-worker:
-value: standard

-- 
To view, visit https://gerrit.wikimedia.org/r/357418
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I7a6f44e9bb9936fc563dd5439a93cd09f102be18
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: Ottomata 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Delete unused role/common/analytics/hadoop configs

[Elukey (Code Review)]

Elukey has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/357418 )

Change subject: Delete unused role/common/analytics/hadoop configs
..

Delete unused role/common/analytics/hadoop configs

Change-Id: I7a6f44e9bb9936fc563dd5439a93cd09f102be18
---
D hieradata/role/common/analytics/hadoop/master.yaml
D hieradata/role/common/analytics/hadoop/standby.yaml
D hieradata/role/common/analytics/hadoop/worker.yaml
D hieradata/role/common/analytics/hue.yaml
D hieradata/role/common/analytics/mysql/meta.yaml
D hieradata/role/common/analytics/spark/standalone/worker.yaml
6 files changed, 0 insertions(+), 51 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/18/357418/1

diff --git a/hieradata/role/common/analytics/hadoop/master.yaml 
b/hieradata/role/common/analytics/hadoop/master.yaml
deleted file mode 100644
index a8fdc0e..000
--- a/hieradata/role/common/analytics/hadoop/master.yaml
+++ /dev/null
@@ -1,18 +0,0 @@
-nagios_group: analytics_eqiad
-cluster: analytics
-admin::groups:
-  # Users in these groups need shell accounts on the Hadoop NameNodes
-  # so that HDFS file perms for these users work properly.
-  # These groupsshould also be added to cdh/hadoop/users.yaml as the
-  # groups parameter to ensure that their hdfs homedirs get created.
-  - analytics-users
-  - analytics-privatedata-users
-  - analytics-admins
-  # role::analytics_cluster::users create system users for different
-  # shared analytics cluster file owership.  Certain user groups
-  # are allowed to sudo -u to these users in order to launch
-  # regular Hadoop jobs that don't run as real people users.
-  - analytics-search-users
-debdeploy::grains:
-  debdeploy-hadoop-master:
-value: standard
diff --git a/hieradata/role/common/analytics/hadoop/standby.yaml 
b/hieradata/role/common/analytics/hadoop/standby.yaml
deleted file mode 100644
index aecf358..000
--- a/hieradata/role/common/analytics/hadoop/standby.yaml
+++ /dev/null
@@ -1,13 +0,0 @@
-nagios_group: analytics_eqiad
-cluster: analytics
-admin::groups:
-  - analytics-users
-  - analytics-privatedata-users
-  - analytics-admins
-  # elasticsearch::analytics creates the analytics-search user and group
-  # that analytics-search-users are allowed to sudo to.  This is used
-  # for deploying files to HDFS.
-  - analytics-search-users
-debdeploy::grains:
-  debdeploy-hadoop-standby:
-value: standard
diff --git a/hieradata/role/common/analytics/hadoop/worker.yaml 
b/hieradata/role/common/analytics/hadoop/worker.yaml
deleted file mode 100644
index 40ce529..000
--- a/hieradata/role/common/analytics/hadoop/worker.yaml
+++ /dev/null
@@ -1,11 +0,0 @@
-nagios_group: analytics_eqiad
-cluster: analytics
-admin::groups:
-  - analytics-admins
-debdeploy::grains:
-  debdeploy-hadoop-worker:
-value: standard
-
-# Analytics worker disks are large.  We will install a custom
-# NRPE check for them, so the base module's should ignore them.
-profile::base::check_disk_options: -w 6% -c 3% -l -e -A -i 
"/var/lib/hadoop/data"
diff --git a/hieradata/role/common/analytics/hue.yaml 
b/hieradata/role/common/analytics/hue.yaml
deleted file mode 100644
index f3c3b72..000
--- a/hieradata/role/common/analytics/hue.yaml
+++ /dev/null
@@ -1,3 +0,0 @@
-debdeploy::grains:
-  debdeploy-hue:
-value: standard
diff --git a/hieradata/role/common/analytics/mysql/meta.yaml 
b/hieradata/role/common/analytics/mysql/meta.yaml
deleted file mode 100644
index 1c3255a..000
--- a/hieradata/role/common/analytics/mysql/meta.yaml
+++ /dev/null
@@ -1,3 +0,0 @@
-debdeploy::grains:
-  debdeploy-mysql-analytics:
-value: standard
diff --git a/hieradata/role/common/analytics/spark/standalone/worker.yaml 
b/hieradata/role/common/analytics/spark/standalone/worker.yaml
deleted file mode 100644
index 4eb118d..000
--- a/hieradata/role/common/analytics/spark/standalone/worker.yaml
+++ /dev/null
@@ -1,3 +0,0 @@
-debdeploy::grains:
-  debdeploy-spark-worker:
-value: standard

-- 
To view, visit https://gerrit.wikimedia.org/r/357418
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I7a6f44e9bb9936fc563dd5439a93cd09f102be18
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Set profile::base::check_raid_policy to 'WriteBack' for hado...

[Elukey (Code Review)]

Elukey has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/357403 )

Change subject: Set profile::base::check_raid_policy to 'WriteBack' for hadoop 
workers
..


Set profile::base::check_raid_policy to 'WriteBack' for hadoop workers

Bug: T166140
Change-Id: Id47f7c50eec1b7bd9de178d54d74a5bb76c9c1c7
---
M hieradata/role/common/analytics_cluster/hadoop/worker.yaml
1 file changed, 1 insertion(+), 0 deletions(-)

Approvals:
  Elukey: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/hieradata/role/common/analytics_cluster/hadoop/worker.yaml 
b/hieradata/role/common/analytics_cluster/hadoop/worker.yaml
index 0add55e..83b8dc7 100644
--- a/hieradata/role/common/analytics_cluster/hadoop/worker.yaml
+++ b/hieradata/role/common/analytics_cluster/hadoop/worker.yaml
@@ -9,3 +9,4 @@
 # Analytics worker disks are large.  We will install a custom
 # NRPE check for them, so the base module's should ignore them.
 profile::base::check_disk_options: '-w 6% -c 3% -l -e -A -i 
"/var/lib/hadoop/data"'
+profile::base::check_raid_policy: 'WriteBack'
\ No newline at end of file

-- 
To view, visit https://gerrit.wikimedia.org/r/357403
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Id47f7c50eec1b7bd9de178d54d74a5bb76c9c1c7
Gerrit-PatchSet: 7
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: Jcrespo 
Gerrit-Reviewer: Ottomata 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Set profile::base::check_raid_policy to 'WriteBack' for hado...

[Elukey (Code Review)]

Elukey has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/357403 )

Change subject: Set profile::base::check_raid_policy to 'WriteBack' for hadoop 
workers
..

Set profile::base::check_raid_policy to 'WriteBack' for hadoop workers

Bug: T166140
Change-Id: Id47f7c50eec1b7bd9de178d54d74a5bb76c9c1c7
---
M hieradata/regex.yaml
1 file changed, 3 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/03/357403/1

diff --git a/hieradata/regex.yaml b/hieradata/regex.yaml
index 4f9f056..6ee8e79 100644
--- a/hieradata/regex.yaml
+++ b/hieradata/regex.yaml
@@ -336,3 +336,6 @@
 db_like_raid_policy:
   __regex: !ruby/regexp 
/^(db|dbstore|es|pc|labsdb)[12]\d\d\d\.(eqiad|codfw)\.wmnet$/
   profile::base::check_raid_policy: 'WriteBack'
+analytics_workers_raid_policy:
+  __regex: !ruby/regexp /^(analytics)[1][23456]\d\d\.eqiad\.wmnet$/
+  profile::base::check_raid_policy: 'WriteBack'

-- 
To view, visit https://gerrit.wikimedia.org/r/357403
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Id47f7c50eec1b7bd9de178d54d74a5bb76c9c1c7
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Disable role::analytics_cluster::refinery::job::guard

[Elukey (Code Review)]

Elukey has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/357372 )

Change subject: Disable role::analytics_cluster::refinery::job::guard
..


Disable role::analytics_cluster::refinery::job::guard

The script is currently not working properly and generating
periodical spam.

Bug: T166937
Change-Id: Ib795eeba0b69d61bdbb3560087652b19fa6b3fd4
---
M manifests/site.pp
1 file changed, 2 insertions(+), 1 deletion(-)

Approvals:
  Elukey: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/manifests/site.pp b/manifests/site.pp
index 37cb7c5..94cefdf 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -2508,7 +2508,8 @@
 # webrequest partitions faultyness.
 analytics_cluster::refinery::job::data_check,
 # Include analytics/refinery/source guard checks
-analytics_cluster::refinery::job::guard,
+# Disabled due to T166937
+# analytics_cluster::refinery::job::guard,
 
 # Set up a read only rsync module to allow access
 # to public data generated by the Analytics Cluster.

-- 
To view, visit https://gerrit.wikimedia.org/r/357372
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ib795eeba0b69d61bdbb3560087652b19fa6b3fd4
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Disable role::analytics_cluster::refinery::job::guard

[Elukey (Code Review)]

Elukey has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/357372 )

Change subject: Disable role::analytics_cluster::refinery::job::guard
..

Disable role::analytics_cluster::refinery::job::guard

The script is currently not working properly and generating
periodical spam.

Bug: T166937
Change-Id: Ib795eeba0b69d61bdbb3560087652b19fa6b3fd4
---
M manifests/site.pp
1 file changed, 2 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/72/357372/1

diff --git a/manifests/site.pp b/manifests/site.pp
index 37cb7c5..94cefdf 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -2508,7 +2508,8 @@
 # webrequest partitions faultyness.
 analytics_cluster::refinery::job::data_check,
 # Include analytics/refinery/source guard checks
-analytics_cluster::refinery::job::guard,
+# Disabled due to T166937
+# analytics_cluster::refinery::job::guard,
 
 # Set up a read only rsync module to allow access
 # to public data generated by the Analytics Cluster.

-- 
To view, visit https://gerrit.wikimedia.org/r/357372
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ib795eeba0b69d61bdbb3560087652b19fa6b3fd4
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Correct pageview_hourly loading scheme on pivot home

[Elukey (Code Review)]

Elukey has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/357315 )

Change subject: Correct pageview_hourly loading scheme on pivot home
..


Correct pageview_hourly loading scheme on pivot home

Bug:T167068
Change-Id: I4fd6f60344c27bda086f937b85d793036e300e62
---
M modules/pivot/templates/config.yaml.erb
1 file changed, 1 insertion(+), 1 deletion(-)

Approvals:
  Elukey: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/pivot/templates/config.yaml.erb 
b/modules/pivot/templates/config.yaml.erb
index 71c4157..cae68a1 100644
--- a/modules/pivot/templates/config.yaml.erb
+++ b/modules/pivot/templates/config.yaml.erb
@@ -112,7 +112,7 @@
 
   - name: pageviews-hourly
 title: Pageviews Hourly
-description: Large dataset - Last 3 months - Hourly resolution - Daily 
loading
+description: Large dataset - Last 3 months - Hourly resolution - Hourly 
loading
 clusterName: druid
 source: pageviews-hourly
 # The refresh rule describes how often the data cube looks for new data.

-- 
To view, visit https://gerrit.wikimedia.org/r/357315
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I4fd6f60344c27bda086f937b85d793036e300e62
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Nuria 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: Joal 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Add webrequest dataset to pivot configuration

[Elukey (Code Review)]

Elukey has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/357191 )

Change subject: Add webrequest dataset to pivot configuration
..


Add webrequest dataset to pivot configuration

Sampled webrequest is now loaded in Druid. This patch updates
pivot configuration to show the dataset with a proper name and
coments.

Bug: T166967
Change-Id: I8d1df2de39e56c93695414171bb24f4d6dec45b8
---
M modules/pivot/templates/config.yaml.erb
1 file changed, 31 insertions(+), 0 deletions(-)

Approvals:
  Elukey: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/pivot/templates/config.yaml.erb 
b/modules/pivot/templates/config.yaml.erb
index 57afaa1..71c4157 100644
--- a/modules/pivot/templates/config.yaml.erb
+++ b/modules/pivot/templates/config.yaml.erb
@@ -206,4 +206,35 @@
 # * autofill-all - (default) Introspect the datasource, automatically 
generate dimensions and measures
 introspection: autofill-all
 
+  - name: webrequest
+title: Webrequest
+description: Last 7 days of webrequests sampled to 1/128
+clusterName: druid
+source: webrequest
+# The refresh rule describes how often the data cube looks for new data.
+# Default: 'query'/PT1M (every minute)
+refreshRule:
+  rule: query
+
+#defaultTimezone: Etc/UTC
+
+#defaultDuration: P1D
+
+#defaultSortMeasure: view_count
+
+# The names of measures that are selected by default
+#defaultSelectedMeasures: []
+
+# The names of dimensions that are pinned by default (in order that they 
will appear in the pin bar)
+
+#defaultPinnedDimensions: []
+
+# How the dataset should be introspected
+# possible options are:
+# * none - Do not do any introspection, take what is written in the config 
as the rule of law.
+# * no-autofill - Introspect the datasource but do not automatically 
generate dimensions or measures
+# * autofill-dimensions-only - Introspect the datasource, automatically 
generate dimensions only
+# * autofill-measures-only - Introspect the datasource, automatically 
generate measures only
+# * autofill-all - (default) Introspect the datasource, automatically 
generate dimensions and measures
+introspection: autofill-all
 

-- 
To view, visit https://gerrit.wikimedia.org/r/357191
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I8d1df2de39e56c93695414171bb24f4d6dec45b8
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Joal 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] labs/private[master]: Revert "Add zookeeper.yaml to hieradata common"

[Elukey (Code Review)]

Elukey has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/356556 )

Change subject: Revert "Add zookeeper.yaml to hieradata common"
..


Revert "Add zookeeper.yaml to hieradata common"

This reverts commit fa5a1f54c79be453401a57726b083b73f2b9a5eb.

Change-Id: Ic1e94fda810e027254ed4e7325f331ae65404962
---
D hieradata/common/zookeeper.yaml
1 file changed, 0 insertions(+), 7 deletions(-)

Approvals:
  Elukey: Verified; Looks good to me, approved



diff --git a/hieradata/common/zookeeper.yaml b/hieradata/common/zookeeper.yaml
deleted file mode 100644
index 5a231a2..000
--- a/hieradata/common/zookeeper.yaml
+++ /dev/null
@@ -1,7 +0,0 @@
-# Max number of connections per IP for Zookeeper
-profile::zookeeper::max_client_connections: 1024
-
-# Default tick_time is 2000ms, this should allow a max
-# of 16 seconds of latency for Zookeeper client sessions.
-# See comments in role::kafka::analytics::broker for more info.
-profile::zookeeper::sync_limit: 8

-- 
To view, visit https://gerrit.wikimedia.org/r/356556
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ic1e94fda810e027254ed4e7325f331ae65404962
Gerrit-PatchSet: 1
Gerrit-Project: labs/private
Gerrit-Branch: master
Gerrit-Owner: Elukey 
Gerrit-Reviewer: Elukey 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] labs/private[master]: Revert "Add zookeeper.yaml to hieradata common"

[Elukey (Code Review)]

Elukey has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/356556 )

Change subject: Revert "Add zookeeper.yaml to hieradata common"
..

Revert "Add zookeeper.yaml to hieradata common"

This reverts commit fa5a1f54c79be453401a57726b083b73f2b9a5eb.

Change-Id: Ic1e94fda810e027254ed4e7325f331ae65404962
---
D hieradata/common/zookeeper.yaml
1 file changed, 0 insertions(+), 7 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/labs/private 
refs/changes/56/356556/1

diff --git a/hieradata/common/zookeeper.yaml b/hieradata/common/zookeeper.yaml
deleted file mode 100644
index 5a231a2..000
--- a/hieradata/common/zookeeper.yaml
+++ /dev/null
@@ -1,7 +0,0 @@
-# Max number of connections per IP for Zookeeper
-profile::zookeeper::max_client_connections: 1024
-
-# Default tick_time is 2000ms, this should allow a max
-# of 16 seconds of latency for Zookeeper client sessions.
-# See comments in role::kafka::analytics::broker for more info.
-profile::zookeeper::sync_limit: 8

-- 
To view, visit https://gerrit.wikimedia.org/r/356556
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ic1e94fda810e027254ed4e7325f331ae65404962
Gerrit-PatchSet: 1
Gerrit-Project: labs/private
Gerrit-Branch: master
Gerrit-Owner: Elukey 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: role::mariadb::analytics::custom_repl_slave: add eventloggin...

[Elukey (Code Review)]

Elukey has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/356383 )

Change subject: role::mariadb::analytics::custom_repl_slave: add 
eventlogging_cleaner.py
..

role::mariadb::analytics::custom_repl_slave: add eventlogging_cleaner.py

This script implements the purge/sanitization policies outlined in
T108850. Unit tests have been added at the bottom of the file with
instructions about how to run them properly.

Bug: T108850
Change-Id: I33e312cd39a9860c895897fcb90ed23820ca4dff
---
A modules/role/files/mariadb/eventlogging_cleaner.py
M modules/role/manifests/mariadb.pp
2 files changed, 629 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/83/356383/1

diff --git a/modules/role/files/mariadb/eventlogging_cleaner.py 
b/modules/role/files/mariadb/eventlogging_cleaner.py
new file mode 100644
index 000..4feb2bc
--- /dev/null
+++ b/modules/role/files/mariadb/eventlogging_cleaner.py
@@ -0,0 +1,621 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+"""
+This script enforces the Analytics data retention guidelines outlined in:
+https://wikitech.wikimedia.org/wiki/Analytics/Systems/EventLogging/Data_retention_and_auto-purging
+
+The script reads a whitelist (TSV file) with the following format for each 
line:
+
+Tablename\tfield
+Tablename\tfield2
+Tablename2\tfield_bla
+[...]
+
+The script work in the following way: for each table in the EventLogging 
database,
+it looks for any reference of it in the whitelist. If none is found, it means 
that
+there is no interest of preserving any kind of non-sensitive data, therefore
+the retention policy is applied simply deleting all the rows matching the time
+delta provided in input. If one or more reference is found, it means that some
+fields of a given table needs to be preserved for historical analytics, 
therefore
+the script will execute updates commands to set to NULL all the 
non-whitelisted fields
+belonging to rows matching the time delta provided in input.
+
+Important notes:
+1) The script is meant to run on the same host in which the database that needs
+   to be cleaned is running. The auth scheme by default tries to connect to the
+   local mysql unix socket that exposes the plugin_auth feature. In this way 
there
+   is no need to provide username/passwords. The script will try basic 
authentication
+   if any of DB username/password are provided by the user to ease the 
development
+   of fixes/new-features, but it is not meant to be used in production.
+2) If a table is listed in the whitelist, then some of its fields are 
automatically
+   added to it (see COMMON_PERSISTENT_FIELDS). This ensures that important 
fields
+   like timestamp or primary keys are preserved.
+3) The script runs updates/deletes in batches to avoid blocking the database 
for too
+   long creating contention with other write operations (like inserts).
+"""
+
+import argparse
+import collections
+import ConfigParser
+import csv
+from datetime import datetime, timedelta
+import logging
+import os
+import pymysql
+import re
+import sys
+import unittest
+import uuid
+
+from mock import MagicMock, Mock, call, patch
+
+DATE_FORMAT = '%Y%m%d%H%M%S'
+BATCH_SIZE = 1000
+
+# Fields that are always present due to the EventLogging Capsule.
+# These ones are automatically whitelisted due to their importance.
+COMMON_PERSISTENT_FIELDS = ['id', 'uuid', 'timestamp']
+
+log = logging.getLogger(__name__)
+
+
+class Database(object):
+
+def __init__(self, db_host, db_port, db_name, db_user, db_password):
+self.db_host = db_host
+self.db_name = db_name
+self.db_port = db_port
+self.log = logging.getLogger(self.__class__.__name__)
+
+self.connection = pymysql.connect(
+host=db_host,
+port=db_port,
+db=db_name,
+user=db_user,
+password=db_password,
+autocommit=True,
+charset='utf8',
+use_unicode=True
+)
+
+def execute(self, command, dry_run=False):
+"""
+Sends a single sql command to the server instance,
+returns metadata about the execution and the resulting data.
+"""
+cursor = self.connection.cursor()
+result = {
+"query": command,
+"host": self.db_host,
+"port": self.db_port,
+"database": self.db_name
+}
+try:
+if dry_run:
+self.log.info((
+"We will *NOT* execute \"{}\" on {}:{}/{} because "
+"this is a dry run."
+).format(command, self.db_host, self.db_port, self.db_name))
+result.update({
+"success": True,
+"fields": [],
+"rows": [],
+"numrows": 0
+})
+return result
+

[MediaWiki-commits] [Gerrit] labs/private[master]: Add zookeeper.yaml to hieradata common

[Elukey (Code Review)]

Elukey has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/356042 )

Change subject: Add zookeeper.yaml to hieradata common
..


Add zookeeper.yaml to hieradata common

Needed to test https://gerrit.wikimedia.org/r/#/c/354449

Change-Id: I2d58eac8268355a21ccf7a75306617d7b3be283b
---
A hieradata/common/zookeeper.yaml
1 file changed, 7 insertions(+), 0 deletions(-)

Approvals:
  Elukey: Verified; Looks good to me, approved



diff --git a/hieradata/common/zookeeper.yaml b/hieradata/common/zookeeper.yaml
new file mode 100644
index 000..5a231a2
--- /dev/null
+++ b/hieradata/common/zookeeper.yaml
@@ -0,0 +1,7 @@
+# Max number of connections per IP for Zookeeper
+profile::zookeeper::max_client_connections: 1024
+
+# Default tick_time is 2000ms, this should allow a max
+# of 16 seconds of latency for Zookeeper client sessions.
+# See comments in role::kafka::analytics::broker for more info.
+profile::zookeeper::sync_limit: 8

-- 
To view, visit https://gerrit.wikimedia.org/r/356042
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I2d58eac8268355a21ccf7a75306617d7b3be283b
Gerrit-PatchSet: 1
Gerrit-Project: labs/private
Gerrit-Branch: master
Gerrit-Owner: Elukey 
Gerrit-Reviewer: Elukey 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] labs/private[master]: Add zookeeper.yaml to hieradata common

[Elukey (Code Review)]

Elukey has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/356042 )

Change subject: Add zookeeper.yaml to hieradata common
..

Add zookeeper.yaml to hieradata common

Needed to test https://gerrit.wikimedia.org/r/#/c/354449

Change-Id: I2d58eac8268355a21ccf7a75306617d7b3be283b
---
A hieradata/common/zookeeper.yaml
1 file changed, 7 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/labs/private 
refs/changes/42/356042/1

diff --git a/hieradata/common/zookeeper.yaml b/hieradata/common/zookeeper.yaml
new file mode 100644
index 000..5a231a2
--- /dev/null
+++ b/hieradata/common/zookeeper.yaml
@@ -0,0 +1,7 @@
+# Max number of connections per IP for Zookeeper
+profile::zookeeper::max_client_connections: 1024
+
+# Default tick_time is 2000ms, this should allow a max
+# of 16 seconds of latency for Zookeeper client sessions.
+# See comments in role::kafka::analytics::broker for more info.
+profile::zookeeper::sync_limit: 8

-- 
To view, visit https://gerrit.wikimedia.org/r/356042
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I2d58eac8268355a21ccf7a75306617d7b3be283b
Gerrit-PatchSet: 1
Gerrit-Project: labs/private
Gerrit-Branch: master
Gerrit-Owner: Elukey 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: role::aqs: use profile::cassandra

[Elukey (Code Review)]

Elukey has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/354107 )

Change subject: role::aqs: use profile::cassandra
..


role::aqs: use profile::cassandra

Also add a switch to profile::cassandra to allow opening connections to
the analytics network in case of need, and fix handling of the case we
have no TLS encryption.

Change-Id: I727779606ff6c83f95e2dd308df3fa34f14e3f77
---
M hieradata/hosts/aqs1004.yaml
M hieradata/hosts/aqs1005.yaml
M hieradata/hosts/aqs1006.yaml
M hieradata/hosts/aqs1007.yaml
M hieradata/hosts/aqs1008.yaml
M hieradata/hosts/aqs1009.yaml
M hieradata/role/common/aqs.yaml
M hieradata/role/common/restbase/dev_cluster.yaml
M hieradata/role/common/restbase/production.yaml
M hieradata/role/common/restbase/test_cluster.yaml
M modules/profile/manifests/cassandra.pp
M modules/role/manifests/aqs.pp
12 files changed, 99 insertions(+), 190 deletions(-)

Approvals:
  Elukey: Looks good to me, approved
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, but someone else must approve



diff --git a/hieradata/hosts/aqs1004.yaml b/hieradata/hosts/aqs1004.yaml
index 3d2168d..5af54e2 100644
--- a/hieradata/hosts/aqs1004.yaml
+++ b/hieradata/hosts/aqs1004.yaml
@@ -1,15 +1,3 @@
 # Analytics Query Service - rack1 instances
 
-cassandra::rack: 'rack1'
-
-cassandra::instances:
-  a:
-jmx_port: 7189
-listen_address: 10.64.0.126 #aqs1004-a.eqiad.wmnet
-rpc_address: 10.64.0.126
-rpc_interface: eth0
-  b:
-jmx_port: 7190
-listen_address: 10.64.0.127 #aqs1004-b.eqiad.wmnet
-rpc_address: 10.64.0.127
-rpc_interface: eth0
+profile::cassandra::rack: 'rack1'
diff --git a/hieradata/hosts/aqs1005.yaml b/hieradata/hosts/aqs1005.yaml
index a0f7435..cab90eb 100644
--- a/hieradata/hosts/aqs1005.yaml
+++ b/hieradata/hosts/aqs1005.yaml
@@ -1,15 +1,3 @@
 # Analytics Query Service - rack2 instances
 
-cassandra::rack: 'rack2'
-
-cassandra::instances:
-  a:
-jmx_port: 7189
-listen_address: 10.64.32.189 #aqs1005-a.eqiad.wmnet
-rpc_address: 10.64.32.189
-rpc_interface: eth0
-  b:
-jmx_port: 7190
-listen_address: 10.64.32.190 #aqs1005-b.eqiad.wmnet
-rpc_address: 10.64.32.190
-rpc_interface: eth0
\ No newline at end of file
+profile::cassandra::rack: 'rack2'
diff --git a/hieradata/hosts/aqs1006.yaml b/hieradata/hosts/aqs1006.yaml
index 1484eec..ea5174f 100644
--- a/hieradata/hosts/aqs1006.yaml
+++ b/hieradata/hosts/aqs1006.yaml
@@ -1,15 +1,3 @@
 # Analytics Query Service - rack3 instances
 
-cassandra::rack: 'rack3'
-
-cassandra::instances:
-  a:
-jmx_port: 7189
-listen_address: 10.64.48.148 #aqs1006-a.eqiad.wmnet
-rpc_address: 10.64.48.148
-rpc_interface: eth0
-  b:
-jmx_port: 7190
-listen_address: 10.64.48.149 #aqs1006-b.eqiad.wmnet
-rpc_address: 10.64.48.149
-rpc_interface: eth0
\ No newline at end of file
+profile::cassandra::rack: 'rack3'
diff --git a/hieradata/hosts/aqs1007.yaml b/hieradata/hosts/aqs1007.yaml
index 40795bf..5af54e2 100644
--- a/hieradata/hosts/aqs1007.yaml
+++ b/hieradata/hosts/aqs1007.yaml
@@ -1,15 +1,3 @@
 # Analytics Query Service - rack1 instances
 
-cassandra::rack: 'rack1'
-
-cassandra::instances:
-  a:
-jmx_port: 7189
-listen_address: 10.64.0.213 #aqs1007-a.eqiad.wmnet
-rpc_address: 10.64.0.213
-rpc_interface: eth0
-  b:
-jmx_port: 7190
-listen_address: 10.64.0.237 #aqs1007-b.eqiad.wmnet
-rpc_address: 10.64.0.237
-rpc_interface: eth0
+profile::cassandra::rack: 'rack1'
diff --git a/hieradata/hosts/aqs1008.yaml b/hieradata/hosts/aqs1008.yaml
index eb5e3b9..cab90eb 100644
--- a/hieradata/hosts/aqs1008.yaml
+++ b/hieradata/hosts/aqs1008.yaml
@@ -1,15 +1,3 @@
 # Analytics Query Service - rack2 instances
 
-cassandra::rack: 'rack2'
-
-cassandra::instances:
-  a:
-jmx_port: 7189
-listen_address: 10.64.16.74 #aqs1008-a.eqiad.wmnet
-rpc_address: 10.64.16.74
-rpc_interface: eth0
-  b:
-jmx_port: 7190
-listen_address: 10.64.16.78 #aqs1008-b.eqiad.wmnet
-rpc_address: 10.64.16.78
-rpc_interface: eth0
+profile::cassandra::rack: 'rack2'
diff --git a/hieradata/hosts/aqs1009.yaml b/hieradata/hosts/aqs1009.yaml
index 660a401..ea5174f 100644
--- a/hieradata/hosts/aqs1009.yaml
+++ b/hieradata/hosts/aqs1009.yaml
@@ -1,15 +1,3 @@
 # Analytics Query Service - rack3 instances
 
-cassandra::rack: 'rack3'
-
-cassandra::instances:
-  a:
-jmx_port: 7189
-listen_address: 10.64.48.122 #aqs1009-a.eqiad.wmnet
-rpc_address: 10.64.48.122
-rpc_interface: eth0
-  b:
-jmx_port: 7190
-listen_address: 10.64.48.123 #aqs1009-b.eqiad.wmnet
-rpc_address: 10.64.48.123
-rpc_interface: eth0
\ No newline at end of file
+profile::cassandra::rack: 'rack3'
diff --git a/hieradata/role/common/aqs.yaml b/hieradata/role/common/aqs.yaml
index 6dc8cf4..281582d 100644
--- a/hieradata/role/common/aqs.yaml
+++ b/hieradata/role/common/aqs.yaml
@@ -1,4 +1,7 @@
 # 

[MediaWiki-commits] [Gerrit] operations...analytics-eventlogging-maintenance[master]: [WIP] Add the eventlogging_cleaner script and base package

[Elukey (Code Review)]

Elukey has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/355604 )

Change subject: [WIP] Add the eventlogging_cleaner script and base package
..

[WIP] Add the eventlogging_cleaner script and base package

Bug: T156933
Change-Id: Ia428f39832f0b22c586c3ee7484953ae93fbc8d3
---
A AUTHORS
A LICENSE.txt
A README.md
A eventlogging_cleaner/__init__.py
A eventlogging_cleaner/eventlogging_cleaner.py
A setup.py
6 files changed, 396 insertions(+), 0 deletions(-)


  git pull 
ssh://gerrit.wikimedia.org:29418/operations/software/analytics-eventlogging-maintenance
 refs/changes/04/355604/1

diff --git a/AUTHORS b/AUTHORS
new file mode 100644
index 000..cbafc5d
--- /dev/null
+++ b/AUTHORS
@@ -0,0 +1,2 @@
+Luca Toscano 
+Marcel Ruiz Forns 
diff --git a/LICENSE.txt b/LICENSE.txt
new file mode 100644
index 000..c7a22c6
--- /dev/null
+++ b/LICENSE.txt
@@ -0,0 +1,15 @@
+Copyright (c) 2016-17 Wikimedia Foundation Inc.
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program.  If not, see .
+
diff --git a/README.md b/README.md
new file mode 100644
index 000..0b162f8
--- /dev/null
+++ b/README.md
@@ -0,0 +1,26 @@
+analytics-eventlogging-maintenance documentation
+=
+
+TODO
+
+Installation
+
+
+This software is known to work correctly with python 2.7 and 3.5
+
+From Source
+~~~
+
+.. code:: bash
+
+$ python setup.py install
+
+Usage
+-
+
+TODO
+
+Limitations
+---
+
+TODO
diff --git a/eventlogging_cleaner/__init__.py b/eventlogging_cleaner/__init__.py
new file mode 100644
index 000..e69de29
--- /dev/null
+++ b/eventlogging_cleaner/__init__.py
diff --git a/eventlogging_cleaner/eventlogging_cleaner.py 
b/eventlogging_cleaner/eventlogging_cleaner.py
new file mode 100644
index 000..04abdd7
--- /dev/null
+++ b/eventlogging_cleaner/eventlogging_cleaner.py
@@ -0,0 +1,319 @@
+import argparse
+import collections
+import csv
+from datetime import datetime, timedelta
+import logging
+import os
+import pymysql
+import re
+import sys
+
+
+DATE_FORMAT = '%Y%m%d%H%M%S'
+BATCH_SIZE = 1000
+
+# Fields that are always present due to the EventLogging Capsule.
+# These ones are automatically whitelisted due to their importance.
+COMMON_PERSISTENT_FIELDS = ['id', 'uuid', 'timestamp']
+
+log = logging.getLogger(__name__)
+
+
+class Database(object):
+
+def __init__(self, db_host, db_port, db_name):
+self.db_host = db_host
+self.db_name = db_name
+self.db_port = db_port
+self.log = logging.getLogger(self.__class__.__name__)
+self.connection = pymysql.connect(
+host=db_host,
+port=db_port,
+db=db_name,
+user='root',
+password='root',
+autocommit=True,
+charset='utf8',
+use_unicode=True
+)
+
+def execute(self, command, commit=False, dry_run=False):
+"""
+Sends a single sql command to the server instance,
+returns metadata about the execution and the resulting data.
+"""
+cursor = self.connection.cursor()
+result = {
+"query": command,
+"host": self.db_host,
+"port": self.db_port,
+"database": self.db_name
+}
+try:
+if dry_run:
+self.log.info((
+"We will *NOT* execute \"{}\" on {}:{}/{} because "
+"this is a dry run."
+).format(command, self.db_host, self.db_port, self.db_name))
+result.update({
+"success": True,
+"fields": [],
+"rows": [],
+"numrows": 0
+})
+return result
+else:
+log.info("Executing command: " + command)
+cursor.execute(command)
+
+fields = None
+rows = None
+if cursor.rowcount > 0:
+rows = cursor.fetchall()
+fields = [] if not cursor.description else tuple([x[0] for x 
in cursor.description])
+numrows = cursor.rowcount
+cursor.close()
+
+result.update({
+"success": True,
+"fields": fields,
+"rows": rows,
+  

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Remove any reference of mc1001->mc1018 for decom

[Elukey (Code Review)]

Elukey has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/354453 )

Change subject: Remove any reference of mc1001->mc1018 for decom
..

Remove any reference of mc1001->mc1018 for decom

Bug: T164341
Change-Id: Ib704791b5bf2149f7db31fc056835786bafdd9d6
---
M manifests/site.pp
M modules/install_server/files/dhcpd/linux-host-entries.ttyS1-115200
2 files changed, 0 insertions(+), 95 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/53/354453/1

diff --git a/manifests/site.pp b/manifests/site.pp
index 807d5b9..6a398fb 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1811,10 +1811,6 @@
 role(maps::server, maps::slave)
 }
 
-node /^mc10(0[1-9]|1[0-8])\.eqiad\.wmnet/ {
-role(spare::system)
-}
-
 node /^mc10(19|2[0-9]|3[0-6])\.eqiad\.wmnet/ {
 role(mediawiki::memcached)
 }
diff --git a/modules/install_server/files/dhcpd/linux-host-entries.ttyS1-115200 
b/modules/install_server/files/dhcpd/linux-host-entries.ttyS1-115200
index 2461d17..716e5ea 100644
--- a/modules/install_server/files/dhcpd/linux-host-entries.ttyS1-115200
+++ b/modules/install_server/files/dhcpd/linux-host-entries.ttyS1-115200
@@ -3093,97 +3093,6 @@
 fixed-address maps-test2004.codfw.wmnet;
 }
 
-host mc1001 {
-hardware ethernet 00:1b:21:7a:88:0b;
-fixed-address mc1001.eqiad.wmnet;
-}
-
-host mc1002 {
-hardware ethernet 00:1b:21:70:8d:4f;
-fixed-address mc1002.eqiad.wmnet;
-}
-
-host mc1003 {
-hardware ethernet 00:1b:21:70:8d:41;
-fixed-address mc1003.eqiad.wmnet;
-}
-
-host mc1004 {
-hardware ethernet 00:1b:21:7a:88:9b;
-fixed-address mc1004.eqiad.wmnet;
-}
-
-host mc1005 {
-hardware ethernet 00:1b:21:7a:88:61;
-fixed-address mc1005.eqiad.wmnet;
-}
-
-host mc1006 {
-hardware ethernet 00:1b:21:70:8d:e3;
-fixed-address mc1006.eqiad.wmnet;
-}
-
-host mc1007 {
-hardware ethernet 00:1b:21:70:8d:45;
-fixed-address mc1007.eqiad.wmnet;
-}
-
-host mc1008 {
-hardware ethernet 00:1b:21:70:8d:3f;
-fixed-address mc1008.eqiad.wmnet;
-}
-
-
-host mc1009 {
-hardware ethernet 90:e2:ba:18:95:64;
-fixed-address mc1009.eqiad.wmnet;
-}
-
-host mc1010 {
-hardware ethernet 90:e2:ba:1d:48:e4;
-fixed-address mc1010.eqiad.wmnet;
-}
-
-host mc1011 {
-hardware ethernet 90:e2:ba:18:93:9c;
-fixed-address mc1011.eqiad.wmnet;
-}
-
-host mc1012 {
-hardware ethernet 90:e2:ba:18:97:30;
-fixed-address mc1012.eqiad.wmnet;
-}
-
-host mc1013 {
-hardware ethernet 90:e2:ba:18:92:4c;
-fixed-address mc1013.eqiad.wmnet;
-}
-
-host mc1014 {
-hardware ethernet 90:e2:ba:18:96:58;
-fixed-address mc1014.eqiad.wmnet;
-}
-
-host mc1015 {
-hardware ethernet 90:e2:ba:18:96:7c;
-fixed-address mc1015.eqiad.wmnet;
-}
-
-host mc1016 {
-hardware ethernet 90:e2:ba:18:91:88;
-fixed-address mc1016.eqiad.wmnet;
-}
-
-host mc1017 {
-hardware ethernet 00:10:18:ef:fe:10;
-fixed-address mc1017.eqiad.wmnet;
-}
-
-host mc1018 {
-hardware ethernet 00:10:18:ef:f6:d0;
-fixed-address mc1018.eqiad.wmnet;
-}
-
 host mc1019 {
 hardware ethernet 1c:98:ec:29:27:04;
 fixed-address mc1019.eqiad.wmnet;

-- 
To view, visit https://gerrit.wikimedia.org/r/354453
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ib704791b5bf2149f7db31fc056835786bafdd9d6
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: [WIP] Refactor zookeeper roles to profiles

[Elukey (Code Review)]

Elukey has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/354449 )

Change subject: [WIP] Refactor zookeeper roles to profiles
..

[WIP] Refactor zookeeper roles to profiles

Change-Id: If511394ac4a80287dd44f0c89d9f29fcf9477ad1
---
M hieradata/common/zookeeper.yaml
A modules/profile/manifests/zookeeper/client.pp
A modules/profile/manifests/zookeeper/server.pp
M modules/role/manifests/zookeeper/client.pp
M modules/role/manifests/zookeeper/server.pp
5 files changed, 106 insertions(+), 143 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/49/354449/1

diff --git a/hieradata/common/zookeeper.yaml b/hieradata/common/zookeeper.yaml
index 391dfe6..5a231a2 100644
--- a/hieradata/common/zookeeper.yaml
+++ b/hieradata/common/zookeeper.yaml
@@ -1,7 +1,7 @@
 # Max number of connections per IP for Zookeeper
-zookeeper::max_client_connections: 1024
+profile::zookeeper::max_client_connections: 1024
 
 # Default tick_time is 2000ms, this should allow a max
 # of 16 seconds of latency for Zookeeper client sessions.
 # See comments in role::kafka::analytics::broker for more info.
-zookeeper::sync_limit: 8
+profile::zookeeper::sync_limit: 8
diff --git a/modules/profile/manifests/zookeeper/client.pp 
b/modules/profile/manifests/zookeeper/client.pp
new file mode 100644
index 000..58c4c04
--- /dev/null
+++ b/modules/profile/manifests/zookeeper/client.pp
@@ -0,0 +1,20 @@
+# == Class profile::zookeeper::client
+#
+class profile::zookeeper::client(
+$clusters = hiera('zookeeper_clusters'),
+$cluster_name = hiera('profile::zookeeper::zookeeper_cluster_name'),
+$version  = hiera('profile::zookeeper::zookeeper_version'),
+$sync_limit   = hiera('profile::zookeeper::sync_limit'),
+) {
+require_package('openjdk-7-jdk')
+
+class { '::zookeeper':
+hosts  => $clusters[$cluster_name]['hosts'],
+version=> $version,
+sync_limit => $sync_limit,
+}
+
+system::role { 'profile::zookeeper::client':
+description => "${cluster_name} Cluster Zookeeper Client"
+}
+}
\ No newline at end of file
diff --git a/modules/profile/manifests/zookeeper/server.pp 
b/modules/profile/manifests/zookeeper/server.pp
new file mode 100644
index 000..6b4774f
--- /dev/null
+++ b/modules/profile/manifests/zookeeper/server.pp
@@ -0,0 +1,81 @@
+# == Class profile::zookeeper::server
+#
+# zookeeper_cluster_name in hiera will be used to make jmxtrans
+# properly prefix zookeeper statsd (and graphite) metrics.
+#
+# filtertags: labs-project-deployment-prep labs-project-analytics
+class profile::zookeeper::server (
+$cluster_name   = 
hiera('profile::zookeeper::zookeeper_cluster_name'),
+$is_critical= hiera('profile::zookeeper::is_critical'),
+$max_client_connections = 
hiera('profile::zookeeper::max_client_connections'),
+$statsd_host= hiera('statsd'),
+$jmxtrans_port  = hiera('profile::zookeeper::jmxtrans::jmx_port'),
+) {
+class { '::zookeeper::server':
+# If zookeeper runs in environments where JAVA_TOOL_OPTIONS is defined,
+# (like all the analytics hosts after T128295)
+# the zkCleanup.sh script will cause cronspam to root@ due to
+# message like the following to stderr:
+# 'Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8'
+# There seems to be no elegant way to avoid the JVM spam,
+# so until somebody finds a better way we redirect stdout to /dev/null
+# and we filter out JAVA_TOOL_OPTIONS messages from stderr.
+cleanup_script_args => '-n 10 2>&1 > /dev/null | grep -v 
JAVA_TOOL_OPTIONS',
+java_opts   => '-Xms1g -Xmx1g',
+}
+
+system::role { 'profile::zookeeper::server':
+description => "${cluster_name} Cluster Zookeeper Server"
+}
+
+ferm::service { 'zookeeper':
+proto  => 'tcp',
+# Zookeeper client, protocol ports
+port   => '(2181 2182 2183)',
+srange => '$DOMAIN_NETWORKS',
+}
+
+$group_prefix = "zookeeper.cluster.${cluster_name}."
+# Use jmxtrans for sending metrics to ganglia
+class { 'zookeeper::jmxtrans':
+group_prefix => $group_prefix,
+statsd   => $statsd_host,
+}
+
+if $is_critical {
+# Alert if Zookeeper Server is not running.
+nrpe::monitor_service { 'zookeeper':
+description  => 'Zookeeper Server',
+nrpe_command => '/usr/lib/nagios/plugins/check_procs -c 1:1 -C 
java -a "org.apache.zookeeper.server.quorum.QuorumPeerMain 
/etc/zookeeper/conf/zoo.cfg"',
+critical => $is_critical,
+}
+
+# jmxtrans statsd writer emits fqdns in keys
+# by substituting '.' with '_' and suffixing the jmx port.
+$graphite_broker_key = 
regsubst("${::fqdn}_${::zookeeper::jmxtrans::jmx_port}", '\.', '_', 'G')
+
+# Alert if 

[MediaWiki-commits] [Gerrit] analytics/kafkatee[master]: Remove logrotate and syslog configuration

[Elukey (Code Review)]

Elukey has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/354223 )

Change subject: Remove logrotate and syslog configuration
..

Remove logrotate and syslog configuration

This will allow us to easily deploy logrotate/syslog
configurations via puppet without the risk of overlapping
with the package defaults.

Bug: T151748
Change-Id: I41b0a0be339701a682d3af49e6fc64e4dc74958e
---
D debian/75-kafkatee.conf
M debian/kafkatee.install
D debian/kafkatee.logrotate
M debian/kafkatee.postinst
4 files changed, 0 insertions(+), 29 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/analytics/kafkatee 
refs/changes/23/354223/1

diff --git a/debian/75-kafkatee.conf b/debian/75-kafkatee.conf
deleted file mode 100644
index d9c29ce..000
--- a/debian/75-kafkatee.conf
+++ /dev/null
@@ -1,2 +0,0 @@
-# rsyslogd kafkatee config.
-if $programname == 'kafkatee' then /var/log/kafkatee.log
diff --git a/debian/kafkatee.install b/debian/kafkatee.install
index ef17cfb..eb3166b 100644
--- a/debian/kafkatee.install
+++ b/debian/kafkatee.install
@@ -1,4 +1,3 @@
 kafkateeusr/bin
 debian/etc/kafkatee.confetc
 debian/etc/kafkatee.d/READMEetc/kafkatee.d
-debian/75-kafkatee.conf etc/rsyslog.d
diff --git a/debian/kafkatee.logrotate b/debian/kafkatee.logrotate
deleted file mode 100644
index 3e284d7..000
--- a/debian/kafkatee.logrotate
+++ /dev/null
@@ -1,23 +0,0 @@
-/var/log/kafkatee.log {
-  daily
-  rotate 7
-  create 0664 syslog adm
-  missingok
-  compress
-  delaycompress
-  postrotate
-service rsyslog reload
-  endscript
-}
-
-/var/cache/kafkatee/kafkatee.stats.json {
-  weekly
-  rotate 4
-  create 0644 kafkatee root
-  missingok
-  compress
-  delaycompress
-  postrotate
-service kafkatee reload
-  endscript
-}
diff --git a/debian/kafkatee.postinst b/debian/kafkatee.postinst
index 4900121..1b3033d 100755
--- a/debian/kafkatee.postinst
+++ b/debian/kafkatee.postinst
@@ -24,9 +24,6 @@
 chown -R kafkatee:kafkatee /var/cache/kafkatee
 chown -R kafkatee:kafkatee /var/run/kafkatee
 
-# force rsyslog to create /var/log/kafkatee.log on install.
-service rsyslog restart
-
 echo \
 "NOTE: Default kafkatee config files have been installed at:
 /etc/kafkatee.conf

-- 
To view, visit https://gerrit.wikimedia.org/r/354223
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I41b0a0be339701a682d3af49e6fc64e4dc74958e
Gerrit-PatchSet: 1
Gerrit-Project: analytics/kafkatee
Gerrit-Branch: master
Gerrit-Owner: Elukey 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Fix MediaWiki centralauth errors graphite alarm

[Elukey (Code Review)]

Elukey has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/354184 )

Change subject: Fix MediaWiki centralauth errors graphite alarm
..

Fix MediaWiki centralauth errors graphite alarm

Small fix for the metric to monitor (sum should be sumSeries)

Change-Id: I75ff7c9daf230d79af21562ef3bcf15fe1a55a24
---
M modules/role/manifests/graphite/alerts.pp
1 file changed, 1 insertion(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/84/354184/1

diff --git a/modules/role/manifests/graphite/alerts.pp 
b/modules/role/manifests/graphite/alerts.pp
index 44a0171..3983b04 100644
--- a/modules/role/manifests/graphite/alerts.pp
+++ b/modules/role/manifests/graphite/alerts.pp
@@ -79,7 +79,7 @@
 # Monitor MediaWiki CentralAuth bad tokens
 monitoring::graphite_threshold { 'mediawiki_centralauth_errors':
 description => 'MediaWiki centralauth errors',
-metric  => 'sum(MediaWiki.centralauth.centrallogin_errors.*.rate)',
+metric  => 
'sumSeries(MediaWiki.centralauth.centrallogin_errors.*.rate)',
 warning => 0.5,
 critical=> 1,
 from=> '15min',

-- 
To view, visit https://gerrit.wikimedia.org/r/354184
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I75ff7c9daf230d79af21562ef3bcf15fe1a55a24
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Fix MediaWiki centralauth errors graphite alarm

[Elukey (Code Review)]

Elukey has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/354184 )

Change subject: Fix MediaWiki centralauth errors graphite alarm
..


Fix MediaWiki centralauth errors graphite alarm

Small fix for the metric to monitor (sum should be sumSeries)

Change-Id: I75ff7c9daf230d79af21562ef3bcf15fe1a55a24
---
M modules/role/manifests/graphite/alerts.pp
1 file changed, 1 insertion(+), 1 deletion(-)

Approvals:
  Elukey: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/role/manifests/graphite/alerts.pp 
b/modules/role/manifests/graphite/alerts.pp
index 44a0171..3983b04 100644
--- a/modules/role/manifests/graphite/alerts.pp
+++ b/modules/role/manifests/graphite/alerts.pp
@@ -79,7 +79,7 @@
 # Monitor MediaWiki CentralAuth bad tokens
 monitoring::graphite_threshold { 'mediawiki_centralauth_errors':
 description => 'MediaWiki centralauth errors',
-metric  => 'sum(MediaWiki.centralauth.centrallogin_errors.*.rate)',
+metric  => 
'sumSeries(MediaWiki.centralauth.centrallogin_errors.*.rate)',
 warning => 0.5,
 critical=> 1,
 from=> '15min',

-- 
To view, visit https://gerrit.wikimedia.org/r/354184
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I75ff7c9daf230d79af21562ef3bcf15fe1a55a24
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/mediawiki-config[master]: Re-enable persistent connection to Redis for jobrunners in labs

[Elukey (Code Review)]

Elukey has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/353247 )

Change subject: Re-enable persistent connection to Redis for jobrunners in labs
..

Re-enable persistent connection to Redis for jobrunners in labs

Related to https://gerrit.wikimedia.org/r/#/c/351854

Bug: T125735
Change-Id: I7d628f3756c14099e1ac17877a4f658316022ec9
---
M wmf-config/jobqueue-labs.php
1 file changed, 1 insertion(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/mediawiki-config 
refs/changes/47/353247/1

diff --git a/wmf-config/jobqueue-labs.php b/wmf-config/jobqueue-labs.php
index be38320..58a43e4 100644
--- a/wmf-config/jobqueue-labs.php
+++ b/wmf-config/jobqueue-labs.php
@@ -11,6 +11,7 @@
'redisConfig' => [
'connectTimeout' => 1,
'password' => $wmgRedisPassword,
+   'persistent' => defined( 'MEDIAWIKI_JOB_RUNNER' )
],
'daemonized' => true
 ];

-- 
To view, visit https://gerrit.wikimedia.org/r/353247
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I7d628f3756c14099e1ac17877a4f658316022ec9
Gerrit-PatchSet: 1
Gerrit-Project: operations/mediawiki-config
Gerrit-Branch: master
Gerrit-Owner: Elukey 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Fix logrotate config for analytics1003 to avoid cronspam

[Elukey (Code Review)]

Elukey has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/353014 )

Change subject: Fix logrotate config for analytics1003 to avoid cronspam
..


Fix logrotate config for analytics1003 to avoid cronspam

Bug: T132324
Change-Id: Id0b668eb3b36067ecb3b4c83d23651650dfa25f2
---
M modules/role/files/analytics_cluster/hadoop/hadoop_hdfs.logrotate
1 file changed, 1 insertion(+), 0 deletions(-)

Approvals:
  Elukey: Verified; Looks good to me, approved



diff --git a/modules/role/files/analytics_cluster/hadoop/hadoop_hdfs.logrotate 
b/modules/role/files/analytics_cluster/hadoop/hadoop_hdfs.logrotate
index ea1072c..2d2a943 100644
--- a/modules/role/files/analytics_cluster/hadoop/hadoop_hdfs.logrotate
+++ b/modules/role/files/analytics_cluster/hadoop/hadoop_hdfs.logrotate
@@ -3,6 +3,7 @@
 /var/log/hadoop-hdfs/balancer.log {
   weekly
   rotate 4
+  su root hadoop
   missingok
   notifempty
   nocreate

-- 
To view, visit https://gerrit.wikimedia.org/r/353014
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Id0b668eb3b36067ecb3b4c83d23651650dfa25f2
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Fix logrotate config for analytics1003 to avoid cronspam

[Elukey (Code Review)]

Elukey has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/353014 )

Change subject: Fix logrotate config for analytics1003 to avoid cronspam
..

Fix logrotate config for analytics1003 to avoid cronspam

Bug: T132324
Change-Id: Id0b668eb3b36067ecb3b4c83d23651650dfa25f2
---
M modules/role/files/analytics_cluster/hadoop/hadoop_hdfs.logrotate
1 file changed, 1 insertion(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/14/353014/1

diff --git a/modules/role/files/analytics_cluster/hadoop/hadoop_hdfs.logrotate 
b/modules/role/files/analytics_cluster/hadoop/hadoop_hdfs.logrotate
index ea1072c..2d2a943 100644
--- a/modules/role/files/analytics_cluster/hadoop/hadoop_hdfs.logrotate
+++ b/modules/role/files/analytics_cluster/hadoop/hadoop_hdfs.logrotate
@@ -3,6 +3,7 @@
 /var/log/hadoop-hdfs/balancer.log {
   weekly
   rotate 4
+  su root hadoop
   missingok
   notifempty
   nocreate

-- 
To view, visit https://gerrit.wikimedia.org/r/353014
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Id0b668eb3b36067ecb3b4c83d23651650dfa25f2
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits