Faidon Liambotis has submitted this change and it was merged.
Change subject: ssh: fix completely broken host key collection
......................................................................
ssh: fix completely broken host key collection
SSH host key collection basically exhibits the "Naginator" problem: it's
extremely slow, due to needing to parse the whole file N+1 times. Thus,
we are only supposed to collect on every run only on bastions, and on
the rest of the fleet a few times a day (the infamous "position of the
moon").
I say "supposed" because:
- This seems to be only enabled on every run for tin and bast1001, not
other bastions.
- It does does not in fact run at all on *any* other host (and thus the
potm logic is unused), as the class is never included. Many hosts have
their ssh_known_hosts with a Oct 18, 2013 mtime, which is when
36c7f7d90c551755fee90234ad1c4abf8e32cb4e got merged and broke this.
This patch adds an "sshknowngen", copied over from "naggen2" and
modified extensively. It then replaces the ssh_known_hosts generation
with a File resource and a generate() statement. Finally, it
(re-)enables it for every host on the fleet, unconditionally (no potm,
should be fast enough by now).
A diff between bast1001's ssh_known_hosts and sshknowngen's output
showed no functional differences and the current logic is quite broken
already, so this should be safe.
Change-Id: Ieaede0c9356c46661ddb3c236da4c5f0c3ce350a
---
M manifests/site.pp
A modules/puppetmaster/files/sshknowngen
A modules/puppetmaster/manifests/generators.pp
D modules/puppetmaster/manifests/naggen2.pp
M modules/puppetmaster/manifests/scripts.pp
M modules/ssh/manifests/client.pp
D modules/ssh/manifests/hostkeys-collect.pp
7 files changed, 150 insertions(+), 34 deletions(-)
Approvals:
Faidon Liambotis: Looks good to me, approved
jenkins-bot: Verified
diff --git a/manifests/site.pp b/manifests/site.pp
index cb9e514..ef5e079 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -246,7 +246,6 @@
include standard
include subversion::client
include dsh
- include ssh::hostkeys-collect
class { 'nfs::netapp::home':
mountpoint => '/srv/home_pmtpa',
mount_site => 'pmtpa',
@@ -2359,7 +2358,6 @@
include role::deployment::server
include mysql
include role::labsdb::manager
- include ssh::hostkeys-collect
include role::releases::upload
interface::add_ip6_mapped { 'main':
diff --git a/modules/puppetmaster/files/sshknowngen
b/modules/puppetmaster/files/sshknowngen
new file mode 100755
index 0000000..915f0dc
--- /dev/null
+++ b/modules/puppetmaster/files/sshknowngen
@@ -0,0 +1,115 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+import os
+import sys
+# python 3 compatibility
+try:
+ import ConfigParser as configparser
+except ImportError:
+ import configparser
+
+import sqlalchemy
+from collections import defaultdict
+import argparse
+import logging
+from logging.handlers import SysLogHandler
+
+
+class SshKnownHostsGen(object):
+ query = """
+SELECT resources.title as title,
+ GROUP_CONCAT(CONCAT(param_names.name, "\t", param_values.value)
+ SEPARATOR "\n") AS service_content
+FROM param_values
+ JOIN param_names ON param_names.id = param_values.param_name_id
+ JOIN resources ON param_values.resource_id = resources.id
+WHERE restype = 'Sshkey'
+GROUP BY resources.id ORDER BY resources.title ASC"""
+
+ def load_config(self, configfile):
+ self.config = configparser.SafeConfigParser()
+ self.config.read(configfile)
+ self.dsn = "{}://{}:{}@{}:3306/puppet".format(
+ self.config.get('master', 'dbadapter'),
+ self.config.get('master', 'dbuser'),
+ self.config.get('master', 'dbpassword'),
+ self.config.get('master', 'dbserver')
+ )
+
+ def __init__(self, configfile, debug):
+ self.log = logging.getLogger('sshknowngen')
+ self.log.debug('Loading configfile %s', configfile)
+ self.load_config(configfile)
+ self.db_engine = sqlalchemy.create_engine(
+ self.dsn,
+ echo=debug
+ )
+
+ def _query(self):
+ connection = self.db_engine.connect()
+ connection.execute('set group_concat_max_len = @@max_allowed_packet')
+ res = connection.execute(self.query)
+ connection.close()
+ return res
+
+ def render(self):
+ try:
+ for entity in self._query():
+ self.log.debug('Working on resource %s', entity['title'])
+ attrs = defaultdict(list)
+ for restuple in entity['service_content'].split("\n"):
+ (k, v) = restuple.split("\t")
+ attrs[k].append(v)
+
+ if attrs['ensure'][0] != 'present':
+ continue
+
+ hostname = entity['title']
+ aliases = [
+ a
+ for a in attrs['host_aliases']
+ if not a.startswith('---')
+ ]
+ host_part = ",".join([hostname] + aliases)
+ (keytype, key) = (attrs['type'][0], attrs['key'][0])
+
+ line = '%s %s %s' % (host_part, keytype, key)
+ yield line
+ except Exception as e:
+ self.log.exception(
+ 'Could not generate output for resource Sshkey')
+ sys.exit(30)
+
+
+def main():
+ parser = argparse.ArgumentParser()
+ parser.add_argument(
+ '--configfile', '-c', dest='configfile',
default='/etc/puppet/puppet.conf')
+ parser.add_argument('--debug', action='store_true', default=False)
+ args = parser.parse_args()
+
+ log_format='%(name)s: %(levelname)s - %(message)s'
+ log = logging.getLogger('sshknowngen')
+
+ if not args.debug:
+ #if normal mode, log to syslog
+ log.setLevel(logging.INFO)
+ log.propagate = False
+ handler = SysLogHandler(
+ address='/dev/log',
+ facility=SysLogHandler.LOG_LOCAL3)
+ formatter = logging.Formatter(fmt=log_format)
+ handler.setFormatter(formatter)
+ log.addHandler(handler)
+ else:
+ #if debug mode, print to stderr
+ logging.basicConfig(level=logging.DEBUG, format=log_format)
+
+ log.info('Generating output for resource Sshkey')
+ n = SshKnownHostsGen(args.configfile, args.debug)
+ for entity in n.render():
+ print(entity)
+ log.info('Run completed')
+
+if __name__ == '__main__':
+ main()
diff --git a/modules/puppetmaster/manifests/generators.pp
b/modules/puppetmaster/manifests/generators.pp
new file mode 100644
index 0000000..412b3bb
--- /dev/null
+++ b/modules/puppetmaster/manifests/generators.pp
@@ -0,0 +1,23 @@
+class puppetmaster::generators($ensure = 'present'){
+
+ $packages = ['python-jinja2', 'python-mysqldb', 'python-sqlalchemy']
+ require_package($packages)
+
+ file {'/usr/local/bin/naggen2':
+ ensure => 'present',
+ owner => 'root',
+ group => 'root',
+ mode => '0555',
+ source => 'puppet:///modules/puppetmaster/naggen2',
+ require => Package[$packages]
+ }
+
+ file {'/usr/local/bin/sshknowngen':
+ ensure => 'present',
+ owner => 'root',
+ group => 'root',
+ mode => '0555',
+ source => 'puppet:///modules/puppetmaster/sshknowngen',
+ require => Package[$packages]
+ }
+}
diff --git a/modules/puppetmaster/manifests/naggen2.pp
b/modules/puppetmaster/manifests/naggen2.pp
deleted file mode 100644
index 5476009..0000000
--- a/modules/puppetmaster/manifests/naggen2.pp
+++ /dev/null
@@ -1,14 +0,0 @@
-class puppetmaster::naggen2($ensure = 'present'){
-
- $packages = ['python-jinja2', 'python-mysqldb', 'python-sqlalchemy']
- require_package($packages)
-
- file {'/usr/local/bin/naggen2':
- ensure => 'present',
- owner => 'root',
- group => 'root',
- mode => '0555',
- source => 'puppet:///modules/puppetmaster/naggen2',
- require => Package[$packages]
- }
-}
diff --git a/modules/puppetmaster/manifests/scripts.pp
b/modules/puppetmaster/manifests/scripts.pp
index 82027c0..514431b 100644
--- a/modules/puppetmaster/manifests/scripts.pp
+++ b/modules/puppetmaster/manifests/scripts.pp
@@ -13,7 +13,7 @@
$keep_reports_minutes = 960, # 16 hours
) {
- require puppetmaster::naggen2
+ require puppetmaster::generators
file {'/usr/local/bin/uuid-generator':
ensure => absent,
diff --git a/modules/ssh/manifests/client.pp b/modules/ssh/manifests/client.pp
index 11c21ea..87f3582 100644
--- a/modules/ssh/manifests/client.pp
+++ b/modules/ssh/manifests/client.pp
@@ -1,5 +1,14 @@
class ssh::client {
- package { "openssh-client":
- ensure => latest
+ package { 'openssh-client':
+ ensure => latest,
}
+
+ file { '/etc/ssh/ssh_known_hosts':
+ content => generate('/usr/local/bin/sshknowngen'),
+ backup => false,
+ owner => 'root',
+ group => 'root',
+ mode => '0644',
+ }
+
}
diff --git a/modules/ssh/manifests/hostkeys-collect.pp
b/modules/ssh/manifests/hostkeys-collect.pp
deleted file mode 100644
index e52a1b3..0000000
--- a/modules/ssh/manifests/hostkeys-collect.pp
+++ /dev/null
@@ -1,15 +0,0 @@
-class ssh::hostkeys-collect {
- # Do this about twice a day
- $potm = inline_template('<%= srand ; (rand(25) == 5).to_s.capitalize -%>')
- if $hostname == "tin" or $hostname == "bast1001" or $hostname == "mira" or
$potm == "True" {
- notice("Collecting SSH host keys on ${hostname}.")
-
- # install all collected SSH host keys
- Sshkey <<| |>>
-
- # clean up unmanaged host keys
- resources { 'sshkey':
- purge => true,
- }
- }
-}
--
To view, visit https://gerrit.wikimedia.org/r/210926
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Ieaede0c9356c46661ddb3c236da4c5f0c3ce350a
Gerrit-PatchSet: 4
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Faidon Liambotis <fai...@wikimedia.org>
Gerrit-Reviewer: Alexandros Kosiaris <akosia...@wikimedia.org>
Gerrit-Reviewer: BBlack <bbl...@wikimedia.org>
Gerrit-Reviewer: Faidon Liambotis <fai...@wikimedia.org>
Gerrit-Reviewer: Giuseppe Lavagetto <glavage...@wikimedia.org>
Gerrit-Reviewer: jenkins-bot <>
_______________________________________________
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
