ArielGlenn has submitted this change and it was merged.
Change subject: clean up names and paths, remove last runpy calls, use more yaml
......................................................................
clean up names and paths, remove last runpy calls, use more yaml
* allhosts.py becomes yaml, make sure all references to it use right name
* split up modules into those that will go into a package 'clouseau'
which will be an audit library, and those that will go along with
the retention package which will be for data retention auditing
* remove a bunch of hardcoded paths and use 'confdir' instead
* cp.sh install placeholder til we can build real packages
* export and import rules as yaml
add file describing configuration
rename some conf files and paths to suck less
remember that cp.get_file will be expensive and change everything to use
cp.recv with a single common file, no grains in the filename; this
means one message gets sent out with all the content for a cluster,
instead of one message per host
send only store rules for ignoring files/dirs to minions, instead
of all of them; this will make the message size much smaller
remove old code left over from shipping rules to minions via compressed
json (get_perhost_rules_as_json et al)
fix up conflicting options in rulestore script
shuffle around config file names and method names some more
* naming is hard
* we have four levels of files that might have entries to ignore
- global: these entries are ignored on all hosts
- perhost: these entries are listed by host and ignored
only on the particular host, a flat text file suitable
for deployment by e.g. puppet
- fromstore: these entries are listed in files per host
generated at export
tried to make all names match the above so it's clear for the reader
tossed one more function left over from rules fed to stdin etc
Change-Id: Ib4e1565232184aafcc591bb3254362f82e0db993
---
A dataretention/CONFIG.txt
A dataretention/cp.sh
M dataretention/data_auditor.py
M dataretention/retention/cli.py
M dataretention/retention/cliutils.py
M dataretention/retention/completion.py
M dataretention/retention/config.py
R dataretention/retention/global_ignored.yaml
M dataretention/retention/ignores.py
M dataretention/retention/localexaminer.py
M dataretention/retention/localfileaudit.py
M dataretention/retention/localhomeaudit.py
M dataretention/retention/locallogaudit.py
M dataretention/retention/localusercfgrabber.py
M dataretention/retention/remoteexaminer.py
M dataretention/retention/remotefileauditor.py
M dataretention/retention/remotehomeauditor.py
M dataretention/retention/remotelogauditor.py
M dataretention/retention/remoteusercfgrabber.py
M dataretention/retention/retentionaudit.py
M dataretention/retention/rule.py
M dataretention/retention/ruleutils.py
M dataretention/retention/runner.py
M dataretention/retention/saltclientplus.py
M dataretention/rulestore.py
25 files changed, 457 insertions(+), 551 deletions(-)
Approvals:
ArielGlenn: Verified; Looks good to me, approved
diff --git a/dataretention/CONFIG.txt b/dataretention/CONFIG.txt
new file mode 100644
index 0000000..716e263
--- /dev/null
+++ b/dataretention/CONFIG.txt
@@ -0,0 +1,43 @@
+The data_auditor script requires the following config files:
+
+config.yaml -- general configuration settings
+allhosts_ignored.yaml -- list of files/dirs/extentions
+ that will be ignored on all hosts
+perhost_ignored.yaml -- list of ignored dirs and files
+ per host (in addition to any rules
+ that might be stored in the sqlite db,
+ for which see below)
+
+These should be in yaml format and stored in the config directory,
+/srv/audits/retention/configs by default, on the salt master
+and all minions.
+
+In addition it maintains an sqlite db for storing the status rules
+for files and directories for each host, by default located
+in /etc/data_retention/dataretention_rules.sq3 on the salt
+master.
+
+Text files generated from the sq3 database are stored in
+files named <fqdn>_store.conf for each host, on the salt master,
+by default in the directory /etc/data_retention/data_retention.d
+These files can be used to regenerate the sq3 database if
+something should happen to it.
+
+Yaml files also generated from the sq3 database are stored in
+files named <fqdn>_store_good.conf for each host, on the salt master,
+by default in the directory /srv/audits/retention/configs/fromstore
+These files should be synced out to all salt minions every time
+they are updated, which would be every time that the rules for
+'good' files/dirs are changed on any host, via the interactive
+mode to data_auditor or via the rulestore script. If the
+salt-cp feature of data_auditor is enabled, this will be
+done automatically by the script via salt-cp; you should only
+rely on this method if the size of the files is less than 100K.
+However, the directory /srv/audits/retention/configs/fromstore
+must already exist on the minions.
+
+
+
+
+
+
diff --git a/dataretention/cp.sh b/dataretention/cp.sh
new file mode 100644
index 0000000..21f74be
--- /dev/null
+++ b/dataretention/cp.sh
@@ -0,0 +1,29 @@
+basedir="/srv/audits/retention"
+scriptsdir="${basedir}/scripts"
+libdir="${scriptsdir}/retention"
+pythondir="/usr/lib/python2.7/site-packages"
+pythonlibdir="${pythondir}/clouseau/retention"
+configsdir="${basedir}/configs"
+saltmodsdir="/srv/salt/_modules"
+
+mkdir -p "$pythonlibdir"
+
+libfiles="__init__.py cliutils.py completion.py \
+ config.py fileinfo.py fileutils.py ignores.py \
+ magic.py rule.py ruleutils.py saltclientplus.py status.py utils.py \
+ locallogaudit.py localfileaudit.py localhomeaudit.py \
+ localusercfgrabber.py localexaminer.py"
+cd retention; cp $libfiles "$pythonlibdir"; cd ..
+
+files="__init__.py cli.py remoteexaminer.py runner.py \
+ remotefileauditor.py remotelogauditor.py \
+ remotehomeauditor.py fileutils.py \
+ remoteusercfgrabber.py"
+cd retention; cp $files "$libdir"; cd ..
+
+cp data_auditor.py rulestore.py data_auditor_run.sh "$scriptsdir"
+cd retention; cp config.yaml global_ignored.yaml perhost_ignored.yaml
"$configsdir"; cd ..
+cd retention; cp retentionaudit.py "$saltmodsdir"; cd ..
+
+
+
diff --git a/dataretention/data_auditor.py b/dataretention/data_auditor.py
index a2b355b..c9083a9 100644
--- a/dataretention/data_auditor.py
+++ b/dataretention/data_auditor.py
@@ -2,7 +2,6 @@
import getopt
from retention.cli import CommandLine
-#from retention.auditor import HomesAuditor
from retention.remotefileauditor import RemoteFilesAuditor
from retention.remotelogauditor import RemoteLogsAuditor
from retention.remotehomeauditor import RemoteHomesAuditor
@@ -40,7 +39,7 @@
'logs' or 'homes'; this may not be specified with
the 'info' option.
confdir (-d) -- path to dir where ignores.yaml is located
- default: /srv/salt/audits/retention/configs
+ default: /srv/audits/retention/configs
target (-t) -- for local runs, this must be 'localhost' or '127.0.1'
for remote hosts, this should be a host expression
recognizable by salt, in the following format:
@@ -110,7 +109,7 @@
def main():
hosts_expr = None
audit_type = None
- confdir = '/srv/salt/audits/retention/configs'
+ confdir = '/srv/audits/retention/configs'
files_to_check = None
prettyprint = False
show_sample_content = False
@@ -252,7 +251,7 @@
timeout, maxfiles, store_filepath,
verbose)
report, ignored = logsaudit.audit_hosts()
if interactive:
- cmdline = CommandLine(store_filepath, timeout, audit_type,
hosts_expr)
+ cmdline = CommandLine(confdir, store_filepath, timeout,
audit_type, hosts_expr)
cmdline.run(report, ignored)
elif audit_type == 'root':
@@ -264,7 +263,7 @@
timeout, maxfiles, store_filepath,
verbose)
report, ignored = filesaudit.audit_hosts()
if interactive:
- cmdline = CommandLine(store_filepath, timeout, audit_type,
hosts_expr)
+ cmdline = CommandLine(confdir, store_filepath, timeout,
audit_type, hosts_expr)
cmdline.run(report, ignored)
elif audit_type == 'homes':
@@ -276,7 +275,7 @@
timeout, maxfiles, store_filepath,
verbose)
report, ignored = homesaudit.audit_hosts()
if interactive:
- cmdline = CommandLine(store_filepath, timeout, audit_type,
hosts_expr)
+ cmdline = CommandLine(confdir, store_filepath, timeout,
audit_type, hosts_expr)
cmdline.run(report, ignored)
if __name__ == '__main__':
diff --git a/dataretention/retention/cli.py b/dataretention/retention/cli.py
index e31e450..04d29fb 100644
--- a/dataretention/retention/cli.py
+++ b/dataretention/retention/cli.py
@@ -5,22 +5,20 @@
import readline
import traceback
-from retention.status import Status
-from retention.rule import RuleStore
+from clouseau.retention.status import Status
+from clouseau.retention.rule import RuleStore
import retention.remotefileauditor
-from retention.locallogaudit import LocalLogsAuditor
-from retention.fileinfo import FileInfo
-import retention.utils
-from retention.utils import JsonHelper
-import retention.config
+from clouseau.retention.locallogaudit import LocalLogsAuditor
+from clouseau.retention.fileinfo import FileInfo
+from clouseau.retention.utils import JsonHelper
+import clouseau.retention.config
from retention.remoteexaminer import RemoteDirExaminer, RemoteFileExaminer
-import retention.fileutils
-import retention.ruleutils
-import retention.cliutils
-from retention.ignores import Ignores
+import clouseau.retention.ruleutils
+import clouseau.retention.cliutils
+from clouseau.retention.ignores import Ignores
from retention.remoteusercfgrabber import RemoteUserCfGrabber
-import retention.ignores
-from retention.completion import Completion
+import clouseau.retention.ignores
+from clouseau.retention.completion import Completion
class CurrentEnv(object):
@@ -167,7 +165,7 @@
if num_pages == 1:
break
- page = retention.cliutils.show_pager(page, num_items, num_per_page)
+ page = clouseau.retention.cliutils.show_pager(page, num_items,
num_per_page)
if page is None:
return
elif page == num_pages:
@@ -190,7 +188,8 @@
# todo: down and up should check you really are (descending,
# ascending path)
- def __init__(self, store_filepath, timeout, audit_type, hosts_expr=None):
+ def __init__(self, confdir, store_filepath, timeout, audit_type,
hosts_expr=None):
+ self.confdir = confdir
self.cdb = RuleStore(store_filepath)
self.cdb.store_db_init(None)
self.timeout = timeout
@@ -202,7 +201,7 @@
self.today = time.strftime("%Y%m%d", time.gmtime())
self.basedir = None
self.prompt = None
- retention.cliutils.init_readline_hist()
+ clouseau.retention.cliutils.init_readline_hist()
# this is arbitrary, can tweak it later
# how many levels down we keep in our list of
# top-level dirs from which the user can start
@@ -217,16 +216,16 @@
self.ignored = None
self.local_ignores = None
- self.ignores = Ignores(self.cdb)
- self.ignores.get_perhost_from_rules()
+ self.ignores = Ignores(self.confdir, self.cdb)
+ self.ignores.get_ignores_from_rules_for_hosts()
self.dircontents = CurrentDirContents(self.timeout)
self.cenv = CurrentEnv()
self.cmpl = Completion(self.dircontents, self.cenv,
self.max_depth_top_level)
- retention.config.set_up_conf()
+ clouseau.retention.config.set_up_conf(self.confdir)
def do_one_host(self, host, report):
self.set_host(host)
- self.ignores.get_perhost_from_rules([host])
+ self.ignores.get_ignores_from_rules_for_hosts([host])
if host not in report:
dirs_problem = None
@@ -239,7 +238,7 @@
elif len(self.cenv.problem_dirs) == 0 and len(self.cenv.skipped_dirs)
== 0:
print "No problem dirs and no skipped dirs on this host"
else:
- dirs_problem_to_depth = [retention.cliutils.get_path_prefix(
+ dirs_problem_to_depth =
[clouseau.retention.cliutils.get_path_prefix(
d, self.max_depth_top_level)
for d in dirs_problem]
dirs_skipped = [s for s in dirs_skipped
@@ -254,7 +253,7 @@
elif dir_todo not in relevant_dirs:
print "Please choose one of the following directories:"
# fixme another arbitrary setting
- retention.cliutils.print_columns(relevant_dirs, 5)
+ clouseau.retention.cliutils.print_columns(relevant_dirs, 5)
else:
self.basedir = None
self.cenv.cwdir = None
@@ -276,9 +275,9 @@
print "exiting at user request"
break
else:
- local_ign = RemoteUserCfGrabber(host_todo, self.timeout,
self.audit_type)
+ local_ign = RemoteUserCfGrabber(host_todo, self.timeout,
self.audit_type, self.confdir)
self.local_ignores = local_ign.run(True)
- local_ignored_dirs, local_ignored_files =
retention.ignores.process_local_ignores(
+ local_ignored_dirs, local_ignored_files =
clouseau.retention.ignores.process_local_ignores(
self.local_ignores, self.ignored)
self.do_one_host(host_todo, report)
@@ -352,7 +351,7 @@
return contents
def get_basedir_from_path(self, path):
- for location in retention.config.cf[self.locations]:
+ for location in clouseau.retention.config.cf[self.locations]:
if path == location or path.startswith(location + os.path.sep):
return location
# fixme is this really the right fallback? check it
@@ -364,36 +363,36 @@
path = LocalLogsAuditor.normalize(path)
if entrytype == 'file':
- if retention.ignores.file_is_ignored(path, basedir, self.ignored):
+ if clouseau.retention.ignores.file_is_ignored(path, basedir,
self.ignored):
return False
# check perhost file
if self.cenv.host in self.ignores.perhost_ignores:
- if retention.ignores.file_is_ignored(
+ if clouseau.retention.ignores.file_is_ignored(
path, basedir,
self.ignores.perhost_ignores[self.cenv.host]):
return False
# check perhost rules
if self.cenv.host in self.ignores.perhost_ignores_from_rules:
- if retention.ignores.file_is_ignored(
+ if clouseau.retention.ignores.file_is_ignored(
path, basedir,
self.ignores.perhost_ignores_from_rules[self.cenv.host]):
return False
elif entrytype == 'dir':
- if retention.ignores.dir_is_ignored(path, self.ignored):
+ if clouseau.retention.ignores.dir_is_ignored(path, self.ignored):
return False
# check perhost file
if self.cenv.host in self.ignores.perhost_ignores:
- if retention.ignores.dir_is_ignored(
+ if clouseau.retention.ignores.dir_is_ignored(
path, self.ignores.perhost_ignores[self.cenv.host]):
return False
# check perhost rules
if self.cenv.host in self.ignores.perhost_ignores_from_rules:
- if retention.ignores.dir_is_ignored(
+ if clouseau.retention.ignores.dir_is_ignored(
path,
self.ignores.perhost_ignores_from_rules[self.cenv.host]):
return False
else:
@@ -451,7 +450,7 @@
print 'skipping %s, not in current dir listing' % entry
print self.dircontents.entries_dict
continue
- filetype = retention.ruleutils.entrytype_to_text(
+ filetype = clouseau.retention.ruleutils.entrytype_to_text(
self.dircontents.entries_dict[entry]['type'])
if filetype == 'link':
print 'No need to mark', file_expr, 'links are always skipped'
@@ -460,7 +459,7 @@
print 'Not a dir or regular file, no need to mark, skipping'
continue
status = Status.text_to_status('good')
- retention.ruleutils.do_add_rule(self.cdb, file_expr, filetype,
status, self.cenv.host)
+ clouseau.retention.ruleutils.do_add_rule(self.cdb, file_expr,
filetype, status, self.cenv.host)
return True
def do_rule(self, command):
@@ -497,14 +496,14 @@
path = os.path.join(self.cenv.cwdir, path)
if path[-1] == os.path.sep:
path = path[:-1]
- filetype = retention.ruleutils.text_to_entrytype('dir')
+ filetype =
clouseau.retention.ruleutils.text_to_entrytype('dir')
else:
- filetype = retention.ruleutils.text_to_entrytype('file')
+ filetype =
clouseau.retention.ruleutils.text_to_entrytype('file')
- retention.ruleutils.do_add_rule(self.cdb, path, filetype, status,
self.cenv.host)
+ clouseau.retention.ruleutils.do_add_rule(self.cdb, path, filetype,
status, self.cenv.host)
# update the ignores list since we have a new rule
self.ignores.perhost_ignores_from_rules = {}
- self.ignores.get_perhost_from_rules([self.cenv.host])
+ self.ignores.get_ignores_from_rules_for_hosts([self.cenv.host])
return True
elif command == 'S' or command == 's':
default = Status.text_to_status('problem')
@@ -529,19 +528,19 @@
if prefix == "":
prefix = "/"
if status == 'a' or status == 'A':
- retention.ruleutils.show_rules(self.cdb, self.cenv.host,
prefix=prefix)
+ clouseau.retention.ruleutils.show_rules(self.cdb,
self.cenv.host, prefix=prefix)
return True
elif status[0].upper() in Status.STATUSES:
- retention.ruleutils.show_rules(self.cdb, self.cenv.host,
status[0].upper(),
+ clouseau.retention.ruleutils.show_rules(self.cdb,
self.cenv.host, status[0].upper(),
prefix=prefix)
return True
elif command == 'D' or command == 'd':
self.dircontents.get(self.cenv.host, self.cenv.cwdir, self.batchno)
- retention.ruleutils.get_rules_for_path(self.cdb, self.cenv.cwdir,
self.cenv.host)
+ clouseau.retention.ruleutils.get_rules_for_path(self.cdb,
self.cenv.cwdir, self.cenv.host)
return True
elif command == 'C' or command == 'c':
self.dircontents.get(self.cenv.host, self.cenv.cwdir, self.batchno)
- retention.ruleutils.get_rules_for_entries(self.cdb,
self.cenv.cwdir,
+ clouseau.retention.ruleutils.get_rules_for_entries(self.cdb,
self.cenv.cwdir,
self.dircontents.entries_dict,
self.cenv.host)
return True
@@ -557,10 +556,10 @@
path = os.path.join(self.cenv.cwdir, path)
if path[-1] == os.path.sep:
path = path[:-1]
- retention.ruleutils.do_remove_rule(self.cdb, path, self.cenv.host)
+ clouseau.retention.ruleutils.do_remove_rule(self.cdb, path,
self.cenv.host)
# update the ignores list since we removed a rule
self.ignores.perhost_ignores_from_rules = {}
- self.ignores.get_perhost_from_rules([self.cenv.host])
+ self.ignores.get_ignores_from_rules_for_hosts([self.cenv.host])
return True
elif command == 'I' or command == 'i':
readline.set_completer(None)
@@ -568,10 +567,10 @@
rules_path = rules_path.strip()
if rules_path == '':
return True
- if not retention.cliutils.check_rules_path(rules_path):
+ if not clouseau.retention.cliutils.check_rules_path(rules_path):
print "bad rules file path specified, aborting"
else:
- retention.ruleutils.import_rules(self.cdb, rules_path,
self.cenv.host)
+ clouseau.retention.ruleutils.import_rules(self.cdb,
rules_path, self.cenv.host)
return True
elif command == 'E' or command == 'e':
readline.set_completer(None)
@@ -579,16 +578,16 @@
rules_path = rules_path.strip()
if rules_path == '':
return True
- if not retention.cliutils.check_rules_path(rules_path):
+ if not clouseau.retention.cliutils.check_rules_path(rules_path):
print "bad rules file path specified, aborting"
else:
- retention.ruleutils.export_rules(self.cdb, rules_path,
self.cenv.host)
+ clouseau.retention.ruleutils.export_rules(self.cdb,
rules_path, self.cenv.host)
return True
elif command == 'Q' or command == 'q':
print "quitting this level"
return None
else:
- retention.cliutils.show_help('rule')
+ clouseau.retention.cliutils.show_help('rule')
return True
def do_file_contents(self):
@@ -688,7 +687,7 @@
print "quitting this level"
return None
else:
- retention.cliutils.show_help('examine')
+ clouseau.retention.cliutils.show_help('examine')
return True
def do_top(self, command, dir_path):
@@ -723,7 +722,7 @@
elif command == 'Q' or command == 'q':
return None
else:
- retention.cliutils.show_help('top')
+ clouseau.retention.cliutils.show_help('top')
return True
def do_command(self, command, level, dir_path):
@@ -741,14 +740,14 @@
if command in Status.STATUSES:
# this option is invoked on a directory so
# type is dir every time
- retention.ruleutils.do_add_rule(self.cdb, dir_path,
-
retention.ruleutils.text_to_entrytype('dir'),
+ clouseau.retention.ruleutils.do_add_rule(self.cdb, dir_path,
+
clouseau.retention.ruleutils.text_to_entrytype('dir'),
command, self.cenv.host)
return None
elif command == 'Q' or command == 'q':
return None
else:
- retention.cliutils.show_help(level)
+ clouseau.retention.cliutils.show_help(level)
return True
elif level == 'examine':
return self.do_examine(command)
diff --git a/dataretention/retention/cliutils.py
b/dataretention/retention/cliutils.py
index e5089ff..0b3808c 100644
--- a/dataretention/retention/cliutils.py
+++ b/dataretention/retention/cliutils.py
@@ -3,11 +3,6 @@
import readline
import atexit
-import retention.remotefileauditor
-import retention.utils
-import retention.fileutils
-import retention.ruleutils
-
def init_readline_hist():
readline.parse_and_bind("tab: complete")
diff --git a/dataretention/retention/completion.py
b/dataretention/retention/completion.py
index 133becc..7478165 100644
--- a/dataretention/retention/completion.py
+++ b/dataretention/retention/completion.py
@@ -1,12 +1,7 @@
import os
-import sys
import readline
-import retention.remotefileauditor
-import retention.utils
-import retention.fileutils
-import retention.ruleutils
-import retention.cliutils
+import clouseau.retention.cliutils
class Completion(object):
'''
@@ -52,11 +47,11 @@
return host_todo
else:
print "Please choose one of the following hosts:"
- retention.cliutils.print_columns(self.cenv.hostlist, 4)
+ clouseau.retention.cliutils.print_columns(self.cenv.hostlist,
4)
def dir_completion(self, text, state):
if self.cenv.cwdir is None:
- dirs_problem_to_depth = [retention.cliutils.get_path_prefix(
+ dirs_problem_to_depth =
[clouseau.retention.cliutils.get_path_prefix(
d, self.max_depth_top_level) for d in self.cenv.problem_dirs]
dirs_skipped = [s for s in self.cenv.skipped_dirs
if s not in dirs_problem_to_depth]
diff --git a/dataretention/retention/config.py
b/dataretention/retention/config.py
index ea589b1..4336547 100644
--- a/dataretention/retention/config.py
+++ b/dataretention/retention/config.py
@@ -1,18 +1,19 @@
import os
import salt.utils.yamlloader
-cf = {}
+cf = None
-def set_up_conf():
+def set_up_conf(confdir):
global cf
- if cf:
+ if cf is not None:
return
- print "INFO: about to parse config yaml"
- if os.path.exists('/srv/salt/audits/retention/configs/config.yaml'):
+ configfile = os.path.join(confdir, 'config.yaml')
+ if os.path.exists(configfile):
+ cf = {}
try:
- contents =
open('/srv/salt/audits/retention/configs/config.yaml').read()
+ contents = open(configfile).read()
yamlcontents = salt.utils.yamlloader.load(contents,
Loader=salt.utils.yamlloader.SaltYamlSafeLoader)
# fixme do I need this or will a direct assign get it?
for key in yamlcontents:
diff --git a/dataretention/retention/ignored.yaml
b/dataretention/retention/global_ignored.yaml
similarity index 100%
rename from dataretention/retention/ignored.yaml
rename to dataretention/retention/global_ignored.yaml
diff --git a/dataretention/retention/ignores.py
b/dataretention/retention/ignores.py
index 2284655..8098ae1 100644
--- a/dataretention/retention/ignores.py
+++ b/dataretention/retention/ignores.py
@@ -1,14 +1,38 @@
import os
import sys
-import runpy
import salt.client
import salt.utils.yamlloader
-from retention.status import Status
-import retention.utils
-import retention.fileutils
-import retention.ruleutils
-import retention.config
+from clouseau.retention.status import Status
+import clouseau.retention.utils
+import clouseau.retention.fileutils
+import clouseau.retention.ruleutils
+import clouseau.retention.config
+
+def prep_good_rules_tosend(dirname, hosts):
+ '''
+ on the master:
+
+ prepare a dict of file paths vs contents of good
+ rules, for use by salt cp.recv
+ (sending these files to the minions)
+
+ this method expects the good rules to have
+ previously been exported from the rules store
+ '''
+ results = {}
+ if not hosts:
+ return results
+ for host in hosts:
+ path = os.path.join(dirname, host + '_store_good.py')
+ if os.path.exists(path):
+ try:
+ results[path] = open(path).read()
+ except:
+ # if we can't get the file contents for some reason,
+ # make sure this file isn't in the dict
+ results.pop(path, None)
+ return results
def expand_ignored_dirs(basedir, ignored):
'''
@@ -52,7 +76,7 @@
os.path.dirname(dirname), ignored)
if dirname in expanded_dirs:
return True
- if retention.fileutils.wildcard_matches(dirname, wildcard_dirs):
+ if clouseau.retention.fileutils.wildcard_matches(dirname, wildcard_dirs):
return True
return False
@@ -67,15 +91,15 @@
basename = os.path.basename(fname)
if 'prefixes' in ignored:
- if retention.fileutils.startswith(basename, ignored['prefixes']):
+ if clouseau.retention.fileutils.startswith(basename,
ignored['prefixes']):
return True
if 'extensions' in ignored:
if '*' in ignored['extensions']:
- if retention.fileutils.endswith(basename,
ignored['extensions']['*']):
+ if clouseau.retention.fileutils.endswith(basename,
ignored['extensions']['*']):
return True
if basedir in ignored['extensions']:
- if retention.fileutils.endswith(
+ if clouseau.retention.fileutils.endswith(
basename, ignored['extensions'][basedir]):
return True
@@ -83,35 +107,31 @@
if basename in ignored['files']:
return True
if '*' in ignored['files']:
- if retention.fileutils.endswith(basename, ignored['files']['*']):
+ if clouseau.retention.fileutils.endswith(basename,
ignored['files']['*']):
return True
if '/' in ignored['files']:
if fname in ignored['files']['/']:
return True
- if retention.fileutils.wildcard_matches(
+ if clouseau.retention.fileutils.wildcard_matches(
fname, [w for w in ignored['files']['/'] if '*' in w]):
return True
if basedir in ignored['files']:
- if retention.fileutils.endswith(basename,
ignored['files'][basedir]):
+ if clouseau.retention.fileutils.endswith(basename,
ignored['files'][basedir]):
return True
return False
-def get_home_dirs(locations):
+def get_home_dirs(confdir, locations):
'''
get a list of home directories where the root location(s) for home are
specified in the Config class (see 'home_locations'), by reading
these root location dirs and grabbing all subdirectory names from them
'''
- retention.config.set_up_conf()
+ clouseau.retention.config.set_up_conf(confdir)
home_dirs = []
-# filep =
open('/home/ariel/src/wmf/git-ops-software/software/dataretention/retention/junk',
'w+')
-# filep.write('INFO: ' + ','.join(dir('retention.config')))
-# filep.close()
-# print 'INFO:', dir('retention.config')
- for location in retention.config.cf[locations]:
+ for location in clouseau.retention.config.cf[locations]:
if not os.path.isdir(location):
continue
home_dirs.extend([os.path.join(location, d)
@@ -119,14 +139,14 @@
if os.path.isdir(os.path.join(location, d))])
return home_dirs
-def get_local_ignores(locations):
+def get_local_ignores(confdir, locations):
'''
read a list of absolute paths from /home/blah/.data_retention
for all blah. Dirs are specified by op sep at the end ('/')
and files without.
'''
local_ignores = {}
- home_dirs = get_home_dirs(locations)
+ home_dirs = get_home_dirs(confdir, locations)
for hdir in home_dirs:
local_ignores[hdir] = []
if os.path.exists(os.path.join(hdir, ".data_retention")):
@@ -185,21 +205,21 @@
on a given host
'''
- def __init__(self, cdb):
+ def __init__(self, confdir, cdb):
+ self.confdir = confdir
self.cdb = cdb
- self.perhost_rules_from_file = None
if cdb is not None:
self.hosts = self.cdb.store_db_list_all_hosts()
else:
self.hosts = None
+ self.perhost_ignores_from_file = None
self.perhost_ignores = {}
- self.perhost_ignores_from_rules = {}
- self.perhost_rules_from_store = {}
- self.get_perhost_cf_from_file()
+ self.ignores_from_rules = {}
+ self.process_perhost_ignores_from_file()
self.ignored = {}
- def set_up_ignored(self, confdir, ignore_also=None):
+ def set_up_global_ignored(self, confdir, ignore_also=None):
'''
collect up initial list of files/dirs to skip during audit
'''
@@ -210,7 +230,7 @@
self.ignored['extensions'] = {}
if confdir is not None:
- configfile = os.path.join(confdir, 'ignored.yaml')
+ configfile = os.path.join(confdir, 'global_ignored.yaml')
if os.path.exists(configfile):
try:
contents = open(configfile).read()
@@ -239,94 +259,107 @@
self.ignored['files']['/'] = []
self.ignored['files']['/'].append(path)
- def get_perhost_from_rules(self, hosts=None):
+ def get_ignores_from_rules_for_hosts(self, hosts=None):
if hosts == None:
hosts = self.hosts
for host in hosts:
- self.perhost_rules_from_store = retention.ruleutils.get_rules(
+ local_rules_from_store_db = clouseau.retention.ruleutils.get_rules(
self.cdb, host, Status.text_to_status('good'))
- if self.perhost_rules_from_store is not None:
- if host not in self.perhost_ignores_from_rules:
- self.perhost_ignores_from_rules[host] = {}
- self.perhost_ignores_from_rules[host]['dirs'] = {}
- self.perhost_ignores_from_rules[host]['dirs']['/'] = []
- self.perhost_ignores_from_rules[host]['files'] = {}
- self.perhost_ignores_from_rules[host]['files']['/'] = []
+ if local_rules_from_store_db is not None:
+ if host not in self.ignores_from_rules:
+ self.ignores_from_rules[host] = {}
+ self.ignores_from_rules[host]['dirs'] = {}
+ self.ignores_from_rules[host]['dirs']['/'] = []
+ self.ignores_from_rules[host]['files'] = {}
+ self.ignores_from_rules[host]['files']['/'] = []
- if (self.perhost_rules_from_file is not None and
- 'ignored_dirs' in self.perhost_rules_from_file and
- host in self.perhost_rules_from_file['ignored_dirs']):
- for path in
self.perhost_rules_from_file['ignored_dirs'][host]:
+ if (self.perhost_ignores_from_file is not None and
+ 'ignored_dirs' in self.perhost_ignores_from_file and
+ host in
self.perhost_ignores_from_file['ignored_dirs']):
+ for path in
self.perhost_ignores_from_file['ignored_dirs'][host]:
if (path.startswith('/') and
- path not in
self.perhost_ignores_from_rules[host][
+ path not in self.ignores_from_rules[host][
'dirs']['/']):
if path[-1] == '/':
path = path[:-1]
- self.perhost_ignores_from_rules[host][
+ self.ignores_from_rules[host][
'dirs']['/'].append(path)
- if (self.perhost_rules_from_file is not None and
- 'ignored_files' in self.perhost_rules_from_file and
- host in self.perhost_rules_from_file['ignored_files']):
- for path in
self.perhost_rules_from_file['ignored_files'][host]:
+ if (self.perhost_ignores_from_file is not None and
+ 'ignored_files' in self.perhost_ignores_from_file and
+ host in
self.perhost_ignores_from_file['ignored_files']):
+ for path in
self.perhost_ignores_from_file['ignored_files'][host]:
if (path.startswith('/') and
- path not in self.perhost_ignores_from_rules[
+ path not in self.ignores_from_rules[
host]['files']['/']):
-
self.perhost_ignores_from_rules[host]['files']['/'].append(path)
+
self.ignores_from_rules[host]['files']['/'].append(path)
- def get_perhost_cf_from_file(self):
- if os.path.exists('audit_files_perhost_config.py'):
+ def get_perhost_ignores_from_file(self):
+ '''
+ get the lists of files and dirs to be ignored,
+ from the perhost_ignored file, for all hosts
+ in file
+ '''
+ perhost_ignores = None
+ configfile = os.path.join(self.confdir, 'perhost_ignored.yaml')
+ if os.path.exists(configfile):
try:
- self.perhost_rules_from_file = runpy.run_path(
- 'audit_files_perhost_config.py')['perhostcf']
+ contents = open(configfile).read()
+ yamlcontents = salt.utils.yamlloader.load(contents,
Loader=salt.utils.yamlloader.SaltYamlSafeLoader)
+ perhost_ignores = yamlcontents['perhostcf']
except:
- self.perhost_rules_from_file = None
+ perhost_ignores = None
+ return perhost_ignores
- if self.perhost_rules_from_file is None:
+ def process_perhost_ignores_from_file(self):
+ '''
+ add to ignored dirs and files lists the entries
+ we get from the perhost_ignored file for each
+ host in file
+ '''
+ self.perhost_ignores_from_file = self.get_perhost_ignores_from_file()
+ if self.perhost_ignores_from_file is None:
return
- if 'ignored_dirs' in self.perhost_rules_from_file:
- for host in self.perhost_rules_from_file['ignored_dirs']:
+ if 'ignored_dirs' in self.perhost_ignores_from_file:
+ for host in self.perhost_ignores_from_file['ignored_dirs']:
if host not in self.perhost_ignores:
self.perhost_ignores[host] = {}
self.perhost_ignores[host]['dirs'] = {}
self.perhost_ignores[host]['dirs']['/'] = [
(lambda path: path[:-1] if path[-1] == '/'
else path)(p)
- for p in self.perhost_rules_from_file[
+ for p in self.perhost_ignores_from_file[
'ignored_dirs'][host]]
- if 'ignored_files' in self.perhost_rules_from_file:
- for host in self.perhost_rules_from_file['ignored_files']:
+ if 'ignored_files' in self.perhost_ignores_from_file:
+ for host in self.perhost_ignores_from_file['ignored_files']:
if host not in self.perhost_ignores:
self.perhost_ignores[host] = {}
self.perhost_ignores[host]['files'] = {}
self.perhost_ignores[host]['files']['/'] = (
- self.perhost_rules_from_file['ignored_files'][host])
+ self.perhost_ignores_from_file['ignored_files'][host])
- def add_perhost_rules_to_ignored(self, host):
+ def add_rules_to_ignored(self, rules):
'''
- add dirs/files to be skipped during audit based
- on rules in the rule store db
+ add dirs/files to be ignored, based on rules
+ from the rulestore passed in as an arg
'''
if '/' not in self.ignored['dirs']:
self.ignored['dirs']['/'] = []
if '/' not in self.ignored['files']:
self.ignored['files']['/'] = []
- if host not in self.perhost_rules_from_store:
+
+ if 'good' not in rules:
return
- for rule in self.perhost_rules_from_store[host]:
- path = os.path.join(rule['basedir'], rule['name'])
- if rule['status'] == 'good':
- if retention.ruleutils.entrytype_to_text(rule['type']) ==
'dir':
- if path not in self.ignored['dirs']['/']:
- self.ignored['dirs']['/'].append(path)
- elif retention.ruleutils.entrytype_to_text(rule['type']) ==
'file':
- if path not in self.ignored['files']['/']:
- self.ignored['files']['/'].append(path)
- else:
- # some other random type, don't care
- continue
+ for path in rules['good']:
+ if path.endswith('/'):
+ path = path[:-1]
+ if path not in self.ignored['dirs']['/']:
+ self.ignored['dirs']['/'].append(path)
+ else:
+ if path not in self.ignored['files']['/']:
+ self.ignored['files']['/'].append(path)
def show_ignored(self, basedirs):
sys.stderr.write(
diff --git a/dataretention/retention/localexaminer.py
b/dataretention/retention/localexaminer.py
index 976c038..718b246 100644
--- a/dataretention/retention/localexaminer.py
+++ b/dataretention/retention/localexaminer.py
@@ -3,9 +3,8 @@
import json
import logging
-import retention.utils
-from retention.utils import JsonHelper
-from retention.fileinfo import FileInfo, EntryInfo
+from clouseau.retention.utils import JsonHelper
+from clouseau.retention.fileinfo import FileInfo, EntryInfo
log = logging.getLogger(__name__)
diff --git a/dataretention/retention/localfileaudit.py
b/dataretention/retention/localfileaudit.py
index fcd57c4..c8ae28e 100644
--- a/dataretention/retention/localfileaudit.py
+++ b/dataretention/retention/localfileaudit.py
@@ -2,19 +2,18 @@
import sys
import time
import socket
-import runpy
import stat
import locale
+import salt.utils.yamlloader
-import retention.utils
-import retention.magic
-from retention.rule import Rule
-import retention.config
-from retention.fileinfo import FileInfo
-import retention.fileutils
-import retention.ruleutils
-from retention.ignores import Ignores
-import retention.ignores
+import clouseau.retention.magic
+from clouseau.retention.rule import Rule
+import clouseau.retention.config
+from clouseau.retention.fileinfo import FileInfo
+import clouseau.retention.fileutils
+from clouseau.retention.ignores import Ignores
+import clouseau.retention.ignores
+
class LocalFilesAuditor(object):
'''
@@ -27,6 +26,7 @@
timeout=60, maxfiles=None):
'''
audit_type: type of audit e.g. 'logs', 'homes'
+ confdir: dir path where yaml config files are kept
show_content: show the first line or so from problematic files
dirsizes: show only directories which have too many files to
audit properly, don't report on files at all
@@ -48,6 +48,7 @@
'''
self.audit_type = audit_type
+ self.confdir = confdir
self.locations = audit_type + "_locations"
self.show_sample_content = show_content
self.dirsizes = dirsizes
@@ -63,21 +64,20 @@
self.ignore_also = self.ignore_also.split(',')
self.timeout = timeout
- self.ignored = {}
- self.ignores = Ignores(None)
- self.ignores.set_up_ignored(confdir)
+ self.ignores = Ignores(self.confdir, None)
+ self.ignores.set_up_global_ignored(self.confdir)
self.hostname = socket.getfqdn()
- retention.config.set_up_conf()
- self.cutoff = retention.config.cf['cutoff']
+ clouseau.retention.config.set_up_conf(self.confdir)
+ self.cutoff = clouseau.retention.config.cf['cutoff']
- self.perhost_rules_from_store = None
- self.perhost_rules_from_file = None
- self.set_up_perhost_rules()
+ self.local_rules_from_store_exported = None
+ self.perhost_ignores_from_file = None
+ self.set_up_ignored()
self.today = time.time()
- self.magic = retention.magic.magic_open(retention.magic.MAGIC_NONE)
+ self.magic =
clouseau.retention.magic.magic_open(clouseau.retention.magic.MAGIC_NONE)
self.magic.load()
self.summary = None
self.display_from_dict = FileInfo.display_from_dict
@@ -114,26 +114,31 @@
self.dirs_to_check = [d.rstrip(os.path.sep) for d in check_list
if d.startswith(os.sep)]
- def set_up_perhost_rules(self):
- self.perhost_rules_from_store = runpy.run_path(
- '/srv/audits/retention/configs/%s_store.cf' %
self.hostname)['rules']
- self.perhost_rules_from_file = runpy.run_path(
- '/srv/audits/retention/configs/allhosts_file.cf')['perhostcf']
+ def set_up_ignored(self):
+ fromstore_file = os.path.join(self.confdir, 'fromstore', self.hostname
+ "_store_good.py")
+ if os.path.exists(fromstore_file):
+ contents = open(fromstore_file).read()
+ self.local_rules_from_store_exported = salt.utils.yamlloader.load(
+ contents, Loader=salt.utils.yamlloader.SaltYamlSafeLoader)
+ else:
+ self.local_rules_from_store_exported = None
- if self.perhost_rules_from_store is not None:
- self.ignores.add_perhost_rules_to_ignored(self.hostname)
+ self.perhost_ignores_from_file =
self.ignores.get_perhost_ignores_from_file()
- if (self.perhost_rules_from_file is not None and
- 'ignored_dirs' in self.perhost_rules_from_file):
+ if self.local_rules_from_store_exported is not None:
+
self.ignores.add_rules_to_ignored(self.local_rules_from_store_exported)
+
+ if (self.perhost_ignores_from_file is not None and
+ 'ignored_dirs' in self.perhost_ignores_from_file):
if '/' not in self.ignores.ignored['dirs']:
self.ignores.ignored['dirs']['/'] = []
- if self.hostname in self.perhost_rules_from_file['ignored_dirs']:
- for path in self.perhost_rules_from_file[
+ if self.hostname in self.perhost_ignores_from_file['ignored_dirs']:
+ for path in self.perhost_ignores_from_file[
'ignored_dirs'][self.hostname]:
if path.startswith('/'):
self.ignores.ignored['dirs']['/'].append(path)
- if '*' in self.perhost_rules_from_file['ignored_dirs']:
- for path in self.perhost_rules_from_file[
+ if '*' in self.perhost_ignores_from_file['ignored_dirs']:
+ for path in self.perhost_ignores_from_file[
'ignored_dirs'][self.hostname]:
if path.startswith('/'):
self.ignores.ignored['dirs']['/'].append(path)
@@ -156,7 +161,7 @@
'''
fname = self.normalize(fname)
- if retention.ignores.file_is_ignored(fname, basedir,
self.ignores.ignored):
+ if clouseau.retention.ignores.file_is_ignored(fname, basedir,
self.ignores.ignored):
return False
if (self.filenames_to_check is not None and
@@ -168,9 +173,9 @@
def get_subdirs_to_do(self, dirname, dirname_depth, todo):
locale.setlocale(locale.LC_ALL, '')
- if retention.ignores.dir_is_ignored(dirname, self.ignores.ignored):
+ if clouseau.retention.ignores.dir_is_ignored(dirname,
self.ignores.ignored):
return todo
- if retention.fileutils.dir_is_wrong_type(dirname):
+ if clouseau.retention.fileutils.dir_is_wrong_type(dirname):
return todo
if self.depth < dirname_depth:
@@ -180,7 +185,7 @@
todo[dirname_depth] = []
if self.dirs_to_check is not None:
- if retention.fileutils.subdir_check(dirname, self.dirs_to_check):
+ if clouseau.retention.fileutils.subdir_check(dirname,
self.dirs_to_check):
todo[dirname_depth].append(dirname)
else:
todo[dirname_depth].append(dirname)
@@ -192,7 +197,7 @@
dirs = [os.path.join(dirname, d)
for d in os.listdir(dirname)]
if self.dirs_to_check is not None:
- dirs = [d for d in dirs if retention.fileutils.dirtree_check(
+ dirs = [d for d in dirs if
clouseau.retention.fileutils.dirtree_check(
d, self.dirs_to_check)]
for dname in dirs:
@@ -201,7 +206,7 @@
def get_dirs_to_do(self, dirname):
if (self.dirs_to_check is not None and
- not retention.fileutils.dirtree_check(dirname,
self.dirs_to_check)):
+ not clouseau.retention.fileutils.dirtree_check(dirname,
self.dirs_to_check)):
return {}
todo = {}
@@ -279,10 +284,10 @@
results: the result set
'''
if self.dirs_to_check is not None:
- if not retention.fileutils.dirtree_check(subdirpath,
self.dirs_to_check):
+ if not clouseau.retention.fileutils.dirtree_check(subdirpath,
self.dirs_to_check):
return
- if retention.ignores.dir_is_ignored(subdirpath, self.ignores.ignored):
+ if clouseau.retention.ignores.dir_is_ignored(subdirpath,
self.ignores.ignored):
return True
count = 0
@@ -310,16 +315,16 @@
# cutoff won't be in our list
temp_results = []
for base, paths, files in self.walk_nolinks(subdirpath):
- expanded_dirs, wildcard_dirs =
retention.ignores.expand_ignored_dirs(
+ expanded_dirs, wildcard_dirs =
clouseau.retention.ignores.expand_ignored_dirs(
base, self.ignores.ignored)
if self.dirs_to_check is not None:
paths[:] = [p for p in paths
- if
retention.fileutils.dirtree_check(os.path.join(base, p),
+ if
clouseau.retention.fileutils.dirtree_check(os.path.join(base, p),
self.dirs_to_check)]
paths[:] = [p for p in paths if
- (not retention.fileutils.startswithpath(os.path.join(
+ (not
clouseau.retention.fileutils.startswithpath(os.path.join(
base, p), expanded_dirs) and
- not retention.fileutils.wildcard_matches(os.path.join(
+ not
clouseau.retention.fileutils.wildcard_matches(os.path.join(
base, p), wildcard_dirs, exact=False))]
count = self.process_files_from_path(location, base, files,
count, temp_results,
@@ -331,7 +336,7 @@
def find_all_files(self):
results = []
- for location in retention.config.cf[self.locations]:
+ for location in clouseau.retention.config.cf[self.locations]:
dirs_to_do = self.get_dirs_to_do(location)
if location.count(os.path.sep) >= self.depth + 1:
# do the run at least once
@@ -356,16 +361,12 @@
% (os.path.sep.join(fields[:self.depth + 1]), self.MAX_FILES))
def do_local_audit(self):
- open_files = retention.fileutils.get_open_files()
+ open_files = clouseau.retention.fileutils.get_open_files()
all_files = {}
files = self.find_all_files()
- count = 0
for (f, st) in files:
- if count < 10:
- print "got", f, st
- count += 1
all_files[f] = FileInfo(f, self.magic, st)
all_files[f].load_file_info(self.today, self.cutoff, open_files)
@@ -377,8 +378,8 @@
for fname in all_files]) + 2
for fname in all_files_sorted:
- if (not retention.fileutils.contains(all_files[fname].filetype,
-
retention.config.cf['ignored_types'])
+ if (not
clouseau.retention.fileutils.contains(all_files[fname].filetype,
+
clouseau.retention.config.cf['ignored_types'])
and not all_files[fname].is_empty):
result.append(all_files[fname].format_output(
self.show_sample_content, False,
diff --git a/dataretention/retention/localhomeaudit.py
b/dataretention/retention/localhomeaudit.py
index 17867db..a6ebc9c 100644
--- a/dataretention/retention/localhomeaudit.py
+++ b/dataretention/retention/localhomeaudit.py
@@ -1,9 +1,7 @@
import sys
-import retention.utils
-import retention.magic
-from retention.localfileaudit import LocalFilesAuditor
-import retention.ignores
+from clouseau.retention.localfileaudit import LocalFilesAuditor
+import clouseau.retention.ignores
class LocalHomesAuditor(LocalFilesAuditor):
'''
@@ -28,6 +26,6 @@
self.homes_owners = {}
# FIXME where are these ever used???
- local_ignores = retention.ignores.get_local_ignores(self.locations)
- local_ignored_dirs, local_ignored_files =
retention.ignores.process_local_ignores(
+ local_ignores =
clouseau.retention.ignores.get_local_ignores(self.confdir, self.locations)
+ local_ignored_dirs, local_ignored_files =
clouseau.retention.ignores.process_local_ignores(
local_ignores, self.ignores.ignored)
diff --git a/dataretention/retention/locallogaudit.py
b/dataretention/retention/locallogaudit.py
index 9852218..774a2e6 100644
--- a/dataretention/retention/locallogaudit.py
+++ b/dataretention/retention/locallogaudit.py
@@ -1,13 +1,11 @@
import os
-import sys
import glob
-import retention.utils
-import retention.magic
-import retention.config
-from retention.fileinfo import LogInfo, LogUtils
-from retention.localfileaudit import LocalFilesAuditor
-import retention.fileutils
+import clouseau.retention.utils
+import clouseau.retention.config
+from clouseau.retention.fileinfo import LogInfo, LogUtils
+from clouseau.retention.localfileaudit import LocalFilesAuditor
+import clouseau.retention.fileutils
class LocalLogsAuditor(LocalFilesAuditor):
def __init__(self, audit_type, confdir=None,
@@ -24,8 +22,9 @@
self.oldest_only = oldest
self.show_system_logs = show_system_logs
if self.show_system_logs:
- self.ignored['files'].pop("/var/log")
+ self.ignores.ignored['files'].pop("/var/log")
self.display_from_dict = LogInfo.display_from_dict
+ clouseau.retention.config.set_up_conf(self.confdir)
@staticmethod
def get_rotated_freq(rotated):
@@ -55,10 +54,9 @@
keep = None
return keep
- @staticmethod
- def parse_logrotate_contents(contents,
+ def parse_logrotate_contents(self, contents,
default_freq='-', default_keep='-'):
- retention.config.set_up_conf()
+ clouseau.retention.config.set_up_conf(self.confdir)
lines = contents.split('\n')
state = 'want_lbracket'
logs = {}
@@ -83,7 +81,7 @@
continue
if '*' in line:
log_group.extend(glob.glob(
- os.path.join(retention.config.cf['rotate_basedir'],
line)))
+
os.path.join(clouseau.retention.config.cf['rotate_basedir'], line)))
else:
log_group.append(line)
elif state == 'want_rbracket':
@@ -108,8 +106,7 @@
return logs
def get_logrotate_defaults(self):
- retention.config.set_up_conf()
- contents = open(retention.config.cf['rotate_mainconf']).read()
+ contents = open(clouseau.retention.config.cf['rotate_mainconf']).read()
lines = contents.split('\n')
skip = False
freq = '-'
@@ -142,16 +139,15 @@
gather all names of log files from logrotate
config files
'''
- retention.config.set_up_conf()
rotated_logs = {}
default_freq, default_keep = self.get_logrotate_defaults()
- rotated_logs.update(LocalLogsAuditor.parse_logrotate_contents(
- open(retention.config.cf['rotate_mainconf']).read(),
+ rotated_logs.update(self.parse_logrotate_contents(
+ open(clouseau.retention.config.cf['rotate_mainconf']).read(),
default_freq, default_keep))
- for fname in os.listdir(retention.config.cf['rotate_basedir']):
- pathname = os.path.join(retention.config.cf['rotate_basedir'],
fname)
+ for fname in
os.listdir(clouseau.retention.config.cf['rotate_basedir']):
+ pathname =
os.path.join(clouseau.retention.config.cf['rotate_basedir'], fname)
if os.path.isfile(pathname):
- rotated_logs.update(LocalLogsAuditor.parse_logrotate_contents(
+ rotated_logs.update(self.parse_logrotate_contents(
open(pathname).read(), default_freq, default_keep))
return rotated_logs
@@ -160,9 +156,8 @@
check how long mysql logs are kept around
'''
# note that I also see my.cnf.s3 and we don't check those (yet)
- retention.config.set_up_conf()
output = ''
- for filename in retention.config.cf['mysqlconf']:
+ for filename in clouseau.retention.config.cf['mysqlconf']:
found = False
try:
contents = open(filename).read()
@@ -197,15 +192,15 @@
# add these files to ignore list; a one line report on
# mysql log expiry configuration is sufficient
- if datadir not in self.ignored['files']:
- self.ignored['files'][datadir] = ignore_these
+ if datadir not in self.ignores.ignored['files']:
+ self.ignores.ignored['files'][datadir] = ignore_these
else:
- self.ignored['files'][datadir].extend(ignore_these)
+
self.ignores.ignored['files'][datadir].extend(ignore_these)
# skip the subdirectories in here, they will be full of
mysql dbs
- if datadir not in self.ignored['dirs']:
- self.ignored['files'][datadir] = ['*']
+ if datadir not in self.ignores.ignored['dirs']:
+ self.ignores.ignored['files'][datadir] = ['*']
else:
- self.ignored['files'][datadir].append('*')
+ self.ignores.ignored['files'][datadir].append('*')
if line.startswith('expire_logs_days'):
fields = line.split('=', 1)
@@ -215,7 +210,7 @@
if not fields[1].isdigit():
continue
found = True
- if int(fields[1]) > retention.config.cf['cutoff']/86400:
+ if int(fields[1]) >
clouseau.retention.config.cf['cutoff']/86400:
if output:
output = output + '\n'
output = output + ('WARNING: some mysql logs expired
after %s days in %s'
@@ -231,13 +226,12 @@
note that no summary report is done for a single host,
for logs we summarize across hosts
'''
- retention.config.set_up_conf()
mysql_issues = self.check_mysqlconf()
result = []
if mysql_issues:
result.append(mysql_issues)
- open_files = retention.fileutils.get_open_files()
+ open_files = clouseau.retention.fileutils.get_open_files()
rotated = self.find_rotated_logs()
all_files = {}
@@ -261,8 +255,8 @@
for fname in all_files]) + 2
for fname in all_files_sorted:
- if retention.fileutils.contains(all_files[fname].filetype,
-
retention.config.cf['ignored_types']):
+ if clouseau.retention.fileutils.contains(all_files[fname].filetype,
+
clouseau.retention.config.cf['ignored_types']):
continue
if (self.oldest_only and
diff --git a/dataretention/retention/localusercfgrabber.py
b/dataretention/retention/localusercfgrabber.py
index 1b67a7e..fb8fe78 100644
--- a/dataretention/retention/localusercfgrabber.py
+++ b/dataretention/retention/localusercfgrabber.py
@@ -1,12 +1,13 @@
import json
-import retention.ignores
+import clouseau.retention.ignores
class LocalUserCfGrabber(object):
'''
retrieval and display dirs / files listed as to
be ignored in per-user lists on local host
'''
- def __init__(self, timeout, audit_type='homes'):
+ def __init__(self, confdir, timeout, audit_type='homes'):
+ self.confdir = confdir
self.timeout = timeout
self.audit_type = audit_type
self.locations = audit_type + "_locations"
@@ -22,7 +23,7 @@
local_ignores = {}
- local_ignores = retention.ignores.get_local_ignores(self.locations)
+ local_ignores =
clouseau.retention.ignores.get_local_ignores(self.confdir, self.locations)
output = json.dumps(local_ignores)
print output
return output
diff --git a/dataretention/retention/remoteexaminer.py
b/dataretention/retention/remoteexaminer.py
index d14c3de..4ae22b9 100644
--- a/dataretention/retention/remoteexaminer.py
+++ b/dataretention/retention/remoteexaminer.py
@@ -2,9 +2,8 @@
import logging
from salt.client import LocalClient
-import retention.utils
-from retention.utils import JsonHelper
-from retention.fileinfo import EntryInfo
+from clouseau.retention.utils import JsonHelper
+from clouseau.retention.fileinfo import EntryInfo
log = logging.getLogger(__name__)
diff --git a/dataretention/retention/remotefileauditor.py
b/dataretention/retention/remotefileauditor.py
index f7f38a1..b5e074e 100644
--- a/dataretention/retention/remotefileauditor.py
+++ b/dataretention/retention/remotefileauditor.py
@@ -3,19 +3,17 @@
import time
import json
import socket
-import runpy
-import retention.utils
-import retention.magic
-from retention.status import Status
-from retention.saltclientplus import LocalClientPlus
-from retention.rule import Rule, RuleStore
-import retention.config
-from retention.fileinfo import FileInfo
-from retention.utils import JsonHelper
+import clouseau.retention.magic
+from clouseau.retention.status import Status
+from clouseau.retention.saltclientplus import LocalClientPlus
+from clouseau.retention.rule import RuleStore
+import clouseau.retention.config
+from clouseau.retention.fileinfo import FileInfo
+from clouseau.retention.utils import JsonHelper
from retention.runner import Runner
-import retention.ruleutils
-from retention.ignores import Ignores
+import clouseau.retention.ruleutils
+from clouseau.retention.ignores import Ignores
def get_dirs_toexamine(host_report):
@@ -127,8 +125,8 @@
# need this for locally running jobs
self.hostname = socket.getfqdn()
- retention.config.set_up_conf()
- self.cutoff = retention.config.cf['cutoff']
+ clouseau.retention.config.set_up_conf(confdir)
+ self.cutoff = clouseau.retention.config.cf['cutoff']
client = LocalClientPlus()
hosts, expr_type = Runner.get_hosts_expr_type(self.hosts_expr)
@@ -137,29 +135,19 @@
self.set_up_max_files(maxfiles)
- self.perhost_raw = None
- if
os.path.exists('/srv/audits/retention/scripts/audit_files_perhost_config.py'):
- try:
- self.perhost_rules_from_file = runpy.run_path(
-
'/srv/audits/retention/scripts/audit_files_perhost_config.py')['perhostcf']
- self.perhost_raw = open(
-
'/srv/audits/retention/scripts/audit_files_perhost_config.py').read()
- except:
- pass
-
- self.write_rules_for_minion()
-
self.cdb = RuleStore(self.store_filepath)
self.cdb.store_db_init(self.expanded_hosts)
self.set_up_and_export_rule_store()
- self.ignores = Ignores(self.cdb)
- self.ignores.set_up_ignored(self.confdir, self.ignore_also)
+ self.ignores = Ignores(self.confdir, self.cdb)
+ self.ignores.set_up_global_ignored(self.confdir, self.ignore_also)
if self.verbose:
- self.ignores.show_ignored(retention.config.cf[self.locations])
+
self.ignores.show_ignored(clouseau.retention.config.cf[self.locations])
+
+ self.perhost_ignores_from_file =
self.ignores.get_perhost_ignores_from_file()
self.today = time.time()
- self.magic = retention.magic.magic_open(retention.magic.MAGIC_NONE)
+ self.magic =
clouseau.retention.magic.magic_open(clouseau.retention.magic.MAGIC_NONE)
self.magic.load()
self.summary = None
self.display_from_dict = FileInfo.display_from_dict
@@ -177,7 +165,9 @@
def set_up_runner(self):
- self.runner = Runner(self.hosts_expr,
+ self.runner = Runner(self.confdir,
+ self.store_filepath,
+ self.hosts_expr,
self.expanded_hosts,
self.audit_type,
self.get_audit_args(),
@@ -207,79 +197,16 @@
def set_up_and_export_rule_store(self):
hosts = self.cdb.store_db_list_all_hosts()
- where_to_put = os.path.join(os.path.dirname(self.store_filepath),
- "data_retention.d")
- if not os.path.isdir(where_to_put):
- os.makedirs(where_to_put, 0755)
+ destdir = os.path.join(os.path.dirname(self.store_filepath),
+ "data_retention.d")
+ if not os.path.isdir(destdir):
+ os.makedirs(destdir, 0755)
for host in hosts:
- nicepath = os.path.join(where_to_put, host + ".conf")
- retention.ruleutils.export_rules(self.cdb, nicepath, host)
-
- def get_perhost_rules_as_json(self):
- '''
- this reads from the data_retention.d directory files for the minions
- on which the audit will be run, converts each host's rules to json
- strings, and returns a hash of rules where keys are the hostname and
- values are the list of rules on that host
- '''
- where_to_get = os.path.join(os.path.dirname(self.store_filepath),
- "data_retention.d")
- if not os.path.isdir(where_to_get):
- os.mkdir(where_to_get, 0755)
- # really? or just read each file and be done with it?
- # also I would like to check the syntax cause paranoid.
- rules = {}
- self.cdb = RuleStore(self.store_filepath)
- self.cdb.store_db_init(self.expanded_hosts)
- for host in self.expanded_hosts:
- rules[host] = []
- nicepath = os.path.join(where_to_get, host + ".conf")
- if os.path.exists(nicepath):
- dir_rules = None
- try:
- text = open(nicepath)
- exec(text)
- except:
- continue
- if dir_rules is not None:
- for status in Status.status_cf:
- if status in dir_rules:
- for entry in dir_rules[status]:
- if entry[0] != os.path.sep:
- print ("WARNING: relative path in rule,"
- "skipping:", entry)
- continue
- if entry[-1] == os.path.sep:
- entry = entry[:-1]
- entry_type =
retention.ruleutils.text_to_entrytype('dir')
- else:
- entry_type =
retention.ruleutils.text_to_entrytype('file')
- rule = retention.ruleutils.get_rule_as_json(
- entry, entry_type, status)
- rules[host].append(rule)
- return rules
-
- def write_perhost_rules_normal_code(self, indent):
- rules = self.get_perhost_rules_as_json()
-
- for host in rules:
- rulescode = "rules = {}\n\n"
- rulescode += "rules['%s'] = [\n" % host
- rulescode += (indent +
- (",\n%s" % (indent + indent)).join(rules[host]) +
"\n")
- rulescode += "]\n"
-
- with open("/srv/salt/audits/retention/configs/%s_store.py" % host,
"w+") as fp:
- fp.write(rulescode)
- fp.close()
-
- def write_rules_for_minion(self):
- indent = " "
- self.write_perhost_rules_normal_code(indent)
- if self.perhost_raw is not None:
- with open("/srv/salt/audits/retention/configs/allhosts_file.py",
"w+") as fp:
- fp.write(self.perhost_raw)
- fp.close()
+ all_destpath = os.path.join(destdir, host + "_store.py")
+ clouseau.retention.ruleutils.export_rules(self.cdb, all_destpath,
host)
+ good_destpath = os.path.join(destdir, host + "_store_good.py")
+ clouseau.retention.ruleutils.export_rules(self.cdb, good_destpath,
host,
+
Status.text_to_status('good'))
def normalize(self, fname):
'''
@@ -426,7 +353,7 @@
hostlist = report.keys()
for host in hostlist:
try:
- problem_rules = retention.ruleutils.get_rules(self.cdb, host,
Status.text_to_status('problem'))
+ problem_rules =
clouseau.retention.ruleutils.get_rules(self.cdb, host,
Status.text_to_status('problem'))
except:
print 'WARNING: problem retrieving problem rules for host',
host
problem_rules = None
@@ -439,8 +366,8 @@
if dirs_problem is not None:
dirs_problem = list(set(dirs_problem))
for dirname in dirs_problem:
- retention.ruleutils.do_add_rule(self.cdb, dirname,
-
retention.ruleutils.text_to_entrytype('dir'),
+ clouseau.retention.ruleutils.do_add_rule(self.cdb, dirname,
+
clouseau.retention.ruleutils.text_to_entrytype('dir'),
Status.text_to_status('problem'), host)
if dirs_skipped is not None:
@@ -449,6 +376,6 @@
if dirname in dirs_problem or dirname in existing_problems:
# problem report overrides 'too many to audit'
continue
- retention.ruleutils.do_add_rule(self.cdb, dirname,
-
retention.ruleutils.text_to_entrytype('dir'),
+ clouseau.retention.ruleutils.do_add_rule(self.cdb, dirname,
+
clouseau.retention.ruleutils.text_to_entrytype('dir'),
Status.text_to_status('unreviewed'), host)
diff --git a/dataretention/retention/remotehomeauditor.py
b/dataretention/retention/remotehomeauditor.py
index 0adb819..9b2b1b2 100644
--- a/dataretention/retention/remotehomeauditor.py
+++ b/dataretention/retention/remotehomeauditor.py
@@ -1,8 +1,6 @@
import os
import sys
-import retention.utils
-import retention.magic
from retention.remotefileauditor import RemoteFilesAuditor
diff --git a/dataretention/retention/remotelogauditor.py
b/dataretention/retention/remotelogauditor.py
index 83fafda..03ab392 100644
--- a/dataretention/retention/remotelogauditor.py
+++ b/dataretention/retention/remotelogauditor.py
@@ -1,10 +1,8 @@
import sys
import json
-import retention.utils
-import retention.magic
-from retention.fileinfo import LogInfo
-from retention.utils import JsonHelper
+from clouseau.retention.fileinfo import LogInfo
+from clouseau.retention.utils import JsonHelper
from retention.remotefileauditor import RemoteFilesAuditor
diff --git a/dataretention/retention/remoteusercfgrabber.py
b/dataretention/retention/remoteusercfgrabber.py
index 7f31af3..121f928 100644
--- a/dataretention/retention/remoteusercfgrabber.py
+++ b/dataretention/retention/remoteusercfgrabber.py
@@ -2,13 +2,7 @@
import salt.client
import salt.utils.yamlloader
-import retention.remotefileauditor
-import retention.utils
-from retention.utils import JsonHelper
-import retention.fileutils
-import retention.ruleutils
-import retention.cliutils
-import retention.config
+from clouseau.retention.utils import JsonHelper
class RemoteUserCfGrabber(object):
@@ -16,8 +10,9 @@
retrieval and display dirs / files listed as to
be ignored in per-user lists on remote host
'''
- def __init__(self, host, timeout, audit_type):
+ def __init__(self, host, timeout, audit_type, confdir):
self.host = host
+ self.confdir = confdir
self.timeout = timeout
self.audit_type = audit_type
self.locations = audit_type + "_locations"
@@ -34,7 +29,7 @@
local_ignores = {}
client = salt.client.LocalClient()
- module_args = [self.timeout, self.audit_type]
+ module_args = [self.confdir, self.timeout, self.audit_type]
result = client.cmd([self.host], "retentionaudit.retrieve_usercfs",
module_args, expr_form='list',
diff --git a/dataretention/retention/retentionaudit.py
b/dataretention/retention/retentionaudit.py
index 87bf64d..2405c5a 100644
--- a/dataretention/retention/retentionaudit.py
+++ b/dataretention/retention/retentionaudit.py
@@ -3,11 +3,11 @@
log = logging.getLogger(__name__)
-from retention.localfileaudit import LocalFilesAuditor
-from retention.locallogaudit import LocalLogsAuditor
-from retention.localhomeaudit import LocalHomesAuditor
-from retention.localexaminer import LocalFileExaminer, LocalDirExaminer
-from retention.localusercfgrabber import LocalUserCfGrabber
+from clouseau.retention.localfileaudit import LocalFilesAuditor
+from clouseau.retention.locallogaudit import LocalLogsAuditor
+from clouseau.retention.localhomeaudit import LocalHomesAuditor
+from clouseau.retention.localexaminer import LocalFileExaminer,
LocalDirExaminer
+from clouseau.retention.localusercfgrabber import LocalUserCfGrabber
def fileaudit_host(confdir,show_content, dirsizes, depth,
to_check, ignore_also, timeout,
@@ -56,7 +56,7 @@
result = dexaminer.run()
return result
-def retrieve_usercfs(timeout, audit_type):
- ucfsretriever = LocalUserCfGrabber(timeout, audit_type)
+def retrieve_usercfs(confdir, timeout, audit_type):
+ ucfsretriever = LocalUserCfGrabber(confdir, timeout, audit_type)
result = ucfsretriever.run()
return result
diff --git a/dataretention/retention/rule.py b/dataretention/retention/rule.py
index c46e670..840a204 100644
--- a/dataretention/retention/rule.py
+++ b/dataretention/retention/rule.py
@@ -4,8 +4,8 @@
import json
import traceback
import sqlite3
-from retention.saltclientplus import LocalClientPlus
-from retention.status import Status
+from clouseau.retention.saltclientplus import LocalClientPlus
+from clouseau.retention.status import Status
def to_unicode(param):
'''
diff --git a/dataretention/retention/ruleutils.py
b/dataretention/retention/ruleutils.py
index b23086f..a2e6eb6 100644
--- a/dataretention/retention/ruleutils.py
+++ b/dataretention/retention/ruleutils.py
@@ -2,9 +2,13 @@
import sys
import json
import traceback
-from retention.status import Status
-import retention.rule
-from retention.rule import Rule, RuleStore
+from clouseau.retention.status import Status
+import clouseau.retention.rule
+from clouseau.retention.rule import Rule, RuleStore
+import salt.utils.yamlloader
+from salt.utils.yamldumper import SafeOrderedDumper
+import yaml
+
def get_rules_for_entries(cdb, path, path_entries, host, quiet=False):
rules = get_rules_for_path(cdb, path, host, True)
@@ -23,19 +27,6 @@
for rule in uniq_sorted:
print rule
return uniq_sorted
-
-def format_rules_for_export(rules_list, indent_count):
- if len(rules_list) == 0:
- return "[]"
-
- spaces = " " * 4
- indent = spaces * indent_count
- return ("[\n" + indent + spaces +
- (",\n" + indent + spaces).join(
- ["'" + rule['path'].replace("'", r"\'") + "'"
- for rule in rules_list]
- )
- + "\n" + indent + "]")
def import_rule_list(cdb, entries, status, host):
'''
@@ -64,20 +55,6 @@
sys.stderr.write("Couldn't add rule for %s to rule store\n" %
entry)
-def import_handle_status(line):
- '''
- see if the line passed is a status def line
- returns status found (if any) and next state
- '''
- for stat in Status.status_cf:
- result = Status.status_cf[stat][1].match(line)
- if result is not None:
- if "]" in result.group(0):
- return None, Rule.STATE_EXPECT_STATUS
- else:
- return stat, Rule.STATE_EXPECT_ENTRIES
- return None, None
-
def import_rules(cdb, rules_path, host):
# we don't toss all existing rules, these get merged into
# the rules already in the rules store
@@ -85,8 +62,9 @@
# it is possible to bork the list of files by deliberately
# including a file/dir with a newline in the name; this will
# just mean that your rule doesn't cover the files/dirs you want.
+
try:
- rules_text = open(rules_path).read()
+ contents = open(rules_path).read()
except:
exc_type, exc_value, exc_traceback = sys.exc_info()
sys.stderr.write(repr(traceback.format_exception(
@@ -94,70 +72,13 @@
sys.stderr.write("Couldn't read rules from %s.\n" % rules_path)
return
- lines = rules_text.split("\n")
- state = Rule.STATE_START
- rules = {}
- active = None
- for line in lines:
- if Rule.comment_expr.match(line) or Rule.blank_expr.match(line):
- continue
- elif state == Rule.STATE_START:
- if not Rule.first_line_expected.match(line):
- print "unexpected line in rules file, wanted "
- print "'dir_rules = ...', aborting:"
- print line
- return
- else:
- state = Rule.STATE_EXPECT_STATUS
- elif state == Rule.STATE_EXPECT_STATUS:
- if Rule.last_line_expected.match(line):
- # done parsing file
- break
- active, state = import_handle_status(line)
- if state == Rule.STATE_EXPECT_STATUS:
- continue
- elif state == Rule.STATE_EXPECT_ENTRIES:
- rules[active] = []
- elif state is None:
- # not a status with empty list, not a status
- # expecting entries on following lines, bail
- print "unexpected line in rules file, aborting:"
- print line
- return
- elif state == Rule.STATE_EXPECT_ENTRIES:
- if Rule.entry_expr.match(line):
- result = Rule.entry_expr.match(line)
- rules[active].append(result.group(1))
- elif Rule.end_entries_expr.match(line):
- active = None
- state = Rule.STATE_EXPECT_STATUS
- else:
- active, state = import_handle_status(line)
- if state == Rule.STATE_EXPECT_STATUS:
- # end of entries with crap syntax, we forgive
- continue
- elif state == Rule.STATE_EXPECT_ENTRIES:
- # found a status line with empty list.
- # so end of these entries ayways
- state = Rule.STATE_EXPECT_STATUS
- continue
- elif state is None:
- # not an entry, not a status, not end of entries
- print "unexpected line in rules file, wanted entry, "
- print "status or entry end marker, aborting:"
- print line
- return
- else:
- print "unexpected line in rules file, aborting:"
- print line
- return
-
+ yaml_contents = salt.utils.yamlloader.load(contents,
Loader=salt.utils.yamlloader.SaltYamlSafeLoader)
for status in Status.status_cf:
- if status in rules:
+ if status in yaml_contents:
import_rule_list(
- cdb, rules[status],
+ cdb, yaml_contents[status],
Status.status_cf[status][0], host)
-
+
def do_remove_rule(cdb, path, host):
cdb.store_db_delete({'basedir': os.path.dirname(path),
'name': os.path.basename(path)},
@@ -191,8 +112,6 @@
return path
def export_rules(cdb, rules_path, host, status=None):
- # would be nice to be able to only export some rules. whatever
-
rules = get_rules(cdb, host, status)
sorted_rules = {}
for stext in Status.STATUS_TEXTS:
@@ -200,19 +119,20 @@
for rule in rules:
if rule['status'] in Status.STATUS_TEXTS:
rule['path'] = normalize_path(rule['path'], rule['type'])
- sorted_rules[rule['status']].append(rule)
+ sorted_rules[rule['status']].append(rule['path'])
else:
continue
- output = "dir_rules = {\n"
+ rules_by_status = {}
for status in Status.STATUS_TEXTS:
- output += " '%s': %s,\n" % (
- status, format_rules_for_export(sorted_rules[status], 2))
- output += "}\n"
+ rules_by_status[status] = sorted_rules[status]
try:
- filep = open(rules_path, "w")
- filep.write("# -*- coding: utf-8 -*-\n")
- filep.write(output)
+ filep = open(rules_path, "w+")
+ contents = yaml.dump(rules_by_status,
+ line_break='\n',
+ default_flow_style=False,
+ Dumper=SafeOrderedDumper)
+ filep.write(contents)
filep.close()
except:
exc_type, exc_value, exc_traceback = sys.exc_info()
@@ -235,8 +155,8 @@
def row_to_rule(row):
# ('/home/ariel/wmf/security', '/home/ariel/wmf/security/openjdk6', 'D',
'G')
(basedir, name, entrytype, status) = row
- basedir = retention.rule.from_unicode(basedir)
- name = retention.rule.from_unicode(name)
+ basedir = clouseau.retention.rule.from_unicode(basedir)
+ name = clouseau.retention.rule.from_unicode(name)
rule = {'path': os.path.join(basedir, name),
'type': entrytype_to_text(entrytype),
'status': Status.status_to_text(status)}
@@ -377,10 +297,3 @@
prefix = os.path.join(prefix, field)
prefixes.append(prefix)
return prefixes
-
-def get_rule_as_json(path, ptype, status):
- rule = {'basedir': os.path.dirname(path),
- 'name': os.path.basename(path),
- 'type': ptype,
- 'status': status}
- return json.dumps(rule)
diff --git a/dataretention/retention/runner.py
b/dataretention/retention/runner.py
index 8b6919e..c80bb05 100644
--- a/dataretention/retention/runner.py
+++ b/dataretention/retention/runner.py
@@ -1,19 +1,20 @@
+import os
import sys
-sys.path.append('/srv/audits/retention/scripts/')
-
-from retention.saltclientplus import LocalClientPlus
-import retention.config
+from clouseau.retention.saltclientplus import LocalClientPlus
+import clouseau.retention.config
class Runner(object):
'''
Manage running current script remotely via salt on one or more hosts
'''
- def __init__(self, hosts_expr, expanded_hosts,
+ def __init__(self, confdir, store_filepath, hosts_expr, expanded_hosts,
audit_type, auditor_args,
show_sample_content=False, to_check=None,
timeout=30, verbose=False):
+ self.confdir = confdir
+ self.store_filepath = store_filepath
self.hosts_expr = hosts_expr
self.expanded_hosts = expanded_hosts
self.hosts, self.hosts_expr_type = Runner.get_hosts_expr_type(
@@ -24,7 +25,7 @@
self.to_check = to_check
self.timeout = timeout
self.verbose = verbose
- retention.config.set_up_conf()
+ clouseau.retention.config.set_up_conf(self.confdir)
def get_auditfunction_name(self):
if self.audit_type == 'root':
@@ -50,9 +51,9 @@
# fixme instead of this we call the right salt module based on the
# audit type and with the self.auditmodule_args which is a list
- hostbatches = [self.expanded_hosts[i: i +
retention.config.cf['batchsize']]
+ hostbatches = [self.expanded_hosts[i: i +
clouseau.retention.config.cf['batchsize']]
for i in range(0, len(self.expanded_hosts),
- retention.config.cf['batchsize'])]
+
clouseau.retention.config.cf['batchsize'])]
result = {}
for hosts in hostbatches:
@@ -60,24 +61,12 @@
sys.stderr.write("INFO: running on hosts\n")
sys.stderr.write(','.join(hosts) + '\n')
- # try to work around a likely race condition in zmq/salt
- # time.sleep(5)
-
- # step one: have them pick up the config files
- # fixme only copy if exists, check returns
- new_result = client.cmd_full_return(hosts, 'cp.get_file',
-
['salt://audits/retention/configs/{{grains.fqdn}}_store.py',
-
"/srv/audits/retention/configs/{{grains.fqdn}}_store.cf",
- 'template=jinja'],
expr_form='list')
- # fixme only copy if exists, check returns
- # fixme this content should be ordered by host instead of by
ignore-list type
- # and split into separate files just as the previous files are,
and actually be in one file
- # with one copy total per client
- new_result = client.cmd_full_return(hosts, 'cp.get_file',
-
['salt://audits/retention/configs/allhosts_file.py',
-
"/srv/audits/retention/configs/allhosts_file.cf",
- 'template=jinja'],
expr_form='list')
-
+ path = os.path.join(os.path.dirname(self.store_filepath),
+ "data_retention.d")
+ contents = clouseau.retention.ignores.prep_good_rules_tosend(path,
hosts)
+ if contents:
+ new_result = client.cmd_full_return(hosts, 'cp.recv',
[contents, os.path.join(self.confdir, 'fromstore')],
+ expr_form='list')
# step two: run the appropriate salt audit module function
new_result = client.cmd(hosts, "retentionaudit.%s" %
self.get_auditfunction_name(), self.auditmodule_args,
expr_form='list', timeout=self.timeout)
diff --git a/dataretention/retention/saltclientplus.py
b/dataretention/retention/saltclientplus.py
index fc41761..98b6a75 100644
--- a/dataretention/retention/saltclientplus.py
+++ b/dataretention/retention/saltclientplus.py
@@ -39,6 +39,7 @@
timeout, **kwargs)
if not job:
+ print "WARNING: failed to get any valid minions from", tgt
return []
else:
time.sleep(3)
diff --git a/dataretention/rulestore.py b/dataretention/rulestore.py
index 515acf0..9c9a7c6 100644
--- a/dataretention/rulestore.py
+++ b/dataretention/rulestore.py
@@ -6,16 +6,16 @@
import sys
import getopt
-from retention.saltclientplus import LocalClientPlus
-import retention.utils
-import retention.ruleutils
-from retention.rule import Rule, RuleStore
-from retention.status import Status
+from clouseau.retention.saltclientplus import LocalClientPlus
+import clouseau.retention.utils
+import clouseau.retention.ruleutils
+from clouseau.retention.rule import Rule, RuleStore
+from clouseau.retention.status import Status
def usage(message=None):
if message:
sys.stderr.write(message + "\n")
- usage_message = """Usage: rulestore.py --host <hostname>
+ usage_message = """Usage: rulestore.py --hosts <hostexpr>
--action <action> [--path <path>] [--status <status>]
[--rulestore <rulestore-path>]
[--dryrun] [--help]
@@ -39,14 +39,14 @@
All options may also be specified in their short form
i.e. -a instead of --action, by using the first letter
- of the option.
+ of the option, except for --host which has the abbrev H.
"""
sys.stderr.write(usage_message)
sys.exit(1)
-def check_args(host, action, status):
- if host is None:
- usage("Mandatory 'host' argument not specified")
+def check_args(hosts, action, status):
+ if hosts is None:
+ usage("Mandatory 'hosts' argument not specified")
if action is None:
usage("Mandatory 'action' argument not specified")
@@ -64,7 +64,7 @@
if path and path[-1] == os.path.sep:
path = path[:-1]
for host in hosts:
- retention.ruleutils.show_rules(cdb, host, status, prefix=path)
+ clouseau.retention.ruleutils.show_rules(cdb, host, status,
prefix=path)
elif action == 'delete':
if path and path[-1] == os.path.sep:
@@ -75,13 +75,13 @@
print "would remove rule for %s in %s" % (path, hosts)
else:
for host in hosts:
- retention.ruleutils.do_remove_rule(cdb, path, host)
+ clouseau.retention.ruleutils.do_remove_rule(cdb, path,
host)
elif status:
if dryrun:
print "would remove rules for status %s in %s" % (status,
hosts)
else:
for host in hosts:
- retention.ruleutils.do_remove_rules(cdb, status, host)
+ clouseau.retention.ruleutils.do_remove_rules(cdb, status,
host)
elif action == 'add':
if status is None:
@@ -90,20 +90,20 @@
usage('path must be specified to add a rule')
if path[-1] == os.path.sep:
- rtype = retention.ruleutils.text_to_entrytype('dir')
+ rtype = clouseau.retention.ruleutils.text_to_entrytype('dir')
path = path[:-1]
else:
- rtype = retention.ruleutils.text_to_entrytype('file')
+ rtype = clouseau.retention.ruleutils.text_to_entrytype('file')
if dryrun:
print "would add rule for %s in %s with status %s of type %s" % (
hosts, path, status, rtype)
for host in hosts:
- retention.ruleutils.do_add_rule(cdb, path, rtype, status, host)
+ clouseau.retention.ruleutils.do_add_rule(cdb, path, rtype, status,
host)
def main():
- host = None
+ hosts = None
action = None
path = None
status = None
@@ -112,16 +112,16 @@
try:
(options, remainder) = getopt.gnu_getopt(
- sys.argv[1:], "h:a:p:s:r:dh",
- ["host=", "action=", "path=",
+ sys.argv[1:], "H:a:p:s:r:dh",
+ ["hosts=", "action=", "path=",
"status=", "dryrun", "help"])
except getopt.GetoptError as err:
usage("Unknown option specified: " + str(err))
for (opt, val) in options:
- if opt in ["-h", "--host"]:
- host = val
+ if opt in ["-H", "--hosts"]:
+ hosts = val
elif opt in ["-a", "--action"]:
action = val
elif opt in ["-p", "--path"]:
@@ -140,7 +140,7 @@
if len(remainder) > 0:
usage("Unknown option specified: <%s>" % remainder[0])
- check_args(host, action, status)
+ check_args(hosts, action, status)
if not os.path.exists(store_filepath):
usage('no such rulestore at %s' % store_filepath)
@@ -148,13 +148,12 @@
cdb = RuleStore(store_filepath)
cdb.store_db_init(None)
- hosts, htype = retention.utils.get_hosts_expr_type(host)
-
+ hosts, htype = clouseau.retention.utils.get_hosts_expr_type(hosts)
# if we are given one host, check that the host has a table or whine
if htype == 'glob' and '*' not in hosts:
- if not retention.ruleutils.check_host_table_exists(cdb, host):
- usage('no such host in rule store, %s' % host)
- elif htype == 'grain':
+ if not clouseau.retention.ruleutils.check_host_table_exists(cdb,
hosts):
+ usage('no such host in rule store, %s' % hosts)
+ if htype == 'grain' or htype == 'glob':
client = LocalClientPlus()
hosts = client.cmd_expandminions(hosts, "test.ping", expr_form=htype)
do_action(cdb, action, hosts, status, path, dryrun)
--
To view, visit https://gerrit.wikimedia.org/r/233468
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Ib4e1565232184aafcc591bb3254362f82e0db993
Gerrit-PatchSet: 2
Gerrit-Project: operations/software
Gerrit-Branch: master
Gerrit-Owner: ArielGlenn <[email protected]>
Gerrit-Reviewer: ArielGlenn <[email protected]>
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
