[MediaWiki-commits] [Gerrit] operations/puppet[production]: Generate stats for monthly package upgrade activity
Muehlenhoff has submitted this change and it was merged. Change subject: Generate stats for monthly package upgrade activity .. Generate stats for monthly package upgrade activity Displays the amount of packages that have been updated or installed on a system per month (this also accounts for installed packages since some upgrades may install a new binary package). This operates on apt's log files, so this only displays the update activity until the last reimage Passing the parameter --top10 also displays the ten packages which have been upgraded most often. This will be used to collect the amount of package upgrades as a KPI. This doesn't match low level package upgrades done "dpkg -i", but that's negligable for our use case. Bug: T116742 Change-Id: I9e5c82d05b1cef445eff293222c12b8d70bcd9b1 --- A modules/base/files/apt-upgrade-activity.py M modules/base/manifests/debdeploy.pp 2 files changed, 69 insertions(+), 0 deletions(-) Approvals: Muehlenhoff: Looks good to me, approved Volans: Looks good to me, but someone else must approve jenkins-bot: Verified diff --git a/modules/base/files/apt-upgrade-activity.py b/modules/base/files/apt-upgrade-activity.py new file mode 100644 index 000..258 --- /dev/null +++ b/modules/base/files/apt-upgrade-activity.py @@ -0,0 +1,61 @@ +#! /usr/bin/python +# -*- coding: utf-8 -*- + +""" +Displays the amount of packages that have been updated or installed on +a system per month (this also accounts for installed packages since some +upgrades may install a new binary package). This operates on apt's log +files, so this only displays the update activity until the last reimage +Passing the parameter --top10 also displays the ten packages which have +been upgraded most often. +""" + +import sys +import os +import gzip +import glob +import re +import collections +import dateutil.parser + +if os.geteuid() != 0: +print "needs to be run as root" +sys.exit(1) + +history_files = glob.glob("/var/log/apt/history*") + +months = collections.defaultdict(int) +package_update_count = collections.defaultdict(int) + +for history_file in history_files: +if history_file.endswith(".gz"): +f = gzip.open(history_file, "r") +else: +f = open(history_file, "r") + +for i in f.readlines(): +if not i or i == "\n": +continue +header, value = i.split(":", 1) +if header == "Start-Date": +month = dateutil.parser.parse(value).date().strftime("%Y-%m") +elif header in ("Upgrade", "Install"): +# Strip the information on the versions which were upgraded (noted in brackets): +updated_packages = 0 +for package in re.sub('\(.*?\)', '', value.strip()).split(","): +updated_packages += 1 +package_update_count[package.strip()] += 1 +months[month] += updated_packages + +for i in sorted(months): +print i, months[i] + +if len(sys.argv) > 1 and sys.argv[1] == "--top10": +print +package_amount = len(package_update_count) +cnt = 0 + +for key, value in sorted(package_update_count.iteritems(), key=lambda(k, v): (v, k)): +cnt += 1 +if package_amount - 10 - cnt < 0: +print "%s: %s" % (key, value) diff --git a/modules/base/manifests/debdeploy.pp b/modules/base/manifests/debdeploy.pp index ffa260d..2fd5e8b 100644 --- a/modules/base/manifests/debdeploy.pp +++ b/modules/base/manifests/debdeploy.pp @@ -14,4 +14,12 @@ if $grains != {} { create_resources(salt::grain, $grains) } + +file { '/usr/local/bin/apt-upgrade-activity': +ensure => present, +source => 'puppet:///modules/base/apt-upgrade-activity.py', +owner => 'root', +group => 'root', +mode => '0555', +} } -- To view, visit https://gerrit.wikimedia.org/r/303531 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I9e5c82d05b1cef445eff293222c12b8d70bcd9b1 Gerrit-PatchSet: 6 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: MuehlenhoffGerrit-Reviewer: Filippo Giunchedi Gerrit-Reviewer: Gehel Gerrit-Reviewer: Muehlenhoff Gerrit-Reviewer: Volans Gerrit-Reviewer: jenkins-bot <> ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Generate stats for monthly package upgrade activity
Muehlenhoff has uploaded a new change for review. https://gerrit.wikimedia.org/r/303531 Change subject: Generate stats for monthly package upgrade activity .. Generate stats for monthly package upgrade activity Displays the amount of packages that have been updated or installed on a system per month (this also accounts for installed packages since some upgrades may install a new binary package). This operates on apt's log files, so this only displays the update activity until the last reimage Passing the parameter --top10 also displays the ten packages which have been upgraded most often. This will be used to collect the amount of package upgrades as a KPI. This doesn't match low level package upgrades done "dpkg -i", but that's negligable for our use case. Bug: T116742 Change-Id: I9e5c82d05b1cef445eff293222c12b8d70bcd9b1 --- A modules/base/files/apt-upgrade-activity M modules/base/manifests/debdeploy.pp 2 files changed, 69 insertions(+), 0 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet refs/changes/31/303531/1 diff --git a/modules/base/files/apt-upgrade-activity b/modules/base/files/apt-upgrade-activity new file mode 100644 index 000..bb23483 --- /dev/null +++ b/modules/base/files/apt-upgrade-activity @@ -0,0 +1,61 @@ +#! /usr/bin/python +# -*- coding: utf-8 -*- + +# Displays the amount of packages that have been updated or installed on +# a system per month (this also accounts for installed packages since some +# upgrades may install a new binary package). This operates on apt's log +# files, so this only displays the update activity until the last reimage +# Passing the parameter --top10 also displays the ten packages which have +# been upgraded most often. + +import sys, os, gzip, glob, re + +if os.geteuid() != 0: +print "needs to be run as root" +sys.exit(1) + +history_files = glob.glob("/var/log/apt/history*") + +months = {} +package_update_count = {} + +for history_file in history_files: +if history_file.endswith(".gz"): +f = gzip.open(history_file, "r") +else: +f = open(history_file, "r") + +upgrade = () + +for i in f.readlines(): +header = i.split(":", 1)[0] +if header == "Start-Date": +month = i.split(":", 1)[1].strip().split(" ")[0][0:7] +elif header == "Upgrade" or header == "Install": +# Strip the information on the versions which were upgraded (noted in brackets): +updated_packages = 0 +for package in re.sub('\(.*?\)', '', i.split(":", 1)[1].strip()).split(","): +updated_packages += 1 + +try: +package_update_count[package.strip()] += 1 +except KeyError: +package_update_count[package.strip()] = 1 + +try: +months[month] += updated_packages +except KeyError: +months[month] = updated_packages + +for i in sorted(months): +print i, months[i] + +if len(sys.argv) > 1 and sys.argv[1] == "--top10": +print +package_amount = len(package_update_count) +cnt = 0 + +for key, value in sorted(package_update_count.iteritems(), key=lambda(k,v): (v,k)): +cnt += 1 +if package_amount-10-cnt < 0: +print "%s: %s" % (key, value) diff --git a/modules/base/manifests/debdeploy.pp b/modules/base/manifests/debdeploy.pp index ffa260d..bf0436e 100644 --- a/modules/base/manifests/debdeploy.pp +++ b/modules/base/manifests/debdeploy.pp @@ -14,4 +14,12 @@ if $grains != {} { create_resources(salt::grain, $grains) } + +file { '/usr/local/bin/apt-upgrade-activity': +ensure => present, +source => 'puppet:///modules/base/apt-upgrade-activity', +owner => 'root', +group => 'root', +mode=> '0555', +} } -- To view, visit https://gerrit.wikimedia.org/r/303531 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I9e5c82d05b1cef445eff293222c12b8d70bcd9b1 Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Muehlenhoff___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits