Re: [Tutor] My first real python project: LOLLERSKATES
Tracy R Reed wrote: Questions/problems/TODO's: This is a fairly simple structured programming implementation. No OO. Should I be using some classes somewhere? There doesn't seem to be any need for it. The config file is just a module which I import which causes all of my configs to become globals. Globals are bad. Is there a better way or would just about anything else be overkill? A singleton config class or something? Overkill? Instead of from lollerskates_config import * try import lollerskates_config as config or some other abbreviated name. Then instead of using e.g. 'macros' in your program use 'config.macros'. This has the advantage that it is clear where macros is defined and the config variables are not global to your module. IMO singletons are greatly overused in general; in Python a module is a unique namespace. I have several loops in this code for processing the logfiles. I once tried to convert these for loops to list comprehensions and totally confused myself and backed out the change (yeay svn!). Is there any benefit to list comprehensions in this case? I don't see any loops that lend themselves to list comprehensions. A list comp is a shorthand way to build a new list from an existing list. You don't do that. I would kinda like to play with unit tests. Not sure how I would construct unit tests for this. And again, perhaps overkill. But some people tell me you should write the unit tests before you even begin coding and code until the tests are satisfied. So even on a small project you would have tests. I run into a couple nasty bugs I created which caused the script to never return anything from the logfile (so you don't immediately realize something is broken) where I thought It sure would be nice to have a test to catch that if it ever happens again. Yep. Have you looked at the unit test module? Many of your functions are unit-testable though it will be easier if you write them so they don't rely on external state (i.e. globals) (this is one reason globals are evil). For example replace_tokens() could be tested, it would be easier if macros was a parameter instead of taken from the global state. process_line() would be easier to test if it didn't rely on the global events list, but either took the list as a parameter or returned the line or None and let the caller deal with it. Or maybe you do want a class that can hold the shared state... One more note: if re.match(^$,line): could just be if not line: Overall it looks pretty clean and well-written. Kent ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
[Tutor] My first real python project: LOLLERSKATES
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I have been a sysadmin for quite a while now and used to do a lot of perl. I haven't done much programming at all in the last couple of years but have been meaning to dump perl for python anyhow (for the usual reasons) and have finally gotten around to doing it. The first itch that I wanted to scratch was something to help me find anomalies in our system logs. We have a lot of servers and need some automation for watching over them including logfiles. I have used logwatch and logcheck but they are both too complicated for what I really need not to mention practically unmaintained these days. So I decided to write my own. Features I wanted were: 1. A basic grep -v sort of functionality on the logfile where I have a file full of regexes/lines to be ignored that are applied to the logfiles to filter everything uninteresting out and whatever is left gets emailed to the admin. 2. A sort of macro system where I can substitute nasty long commonly used regexes in my filter/ignore file with something more easily read and typed. 3. Simplicity. Logcheck had several different levels of logfile events which it could report back on and if you wanted something ignored you had to put it in the right config file for whatever level it was popping up in. This drove me nuts because I often got the wrong one. Logwatch tries to analyze your logs and provide various summaries and statistics and other things I don't care about. I just want to see the interesting lines from the logfile. My code seems to have all of the above and a silly name as well: http://ultraviolet.org/Members/treed/lollerskates It has a list of logfiles to look at, a list of regexes to compare to each line to know what is uninteresting and to filter out/ignore, and emails anything left to the admin, and a simple macro facility. I would appreciate it if anyone interested would download and critique my code. Am I doing anything terribly un-pythonic? Questions/problems/TODO's: This is a fairly simple structured programming implementation. No OO. Should I be using some classes somewhere? The config file is just a module which I import which causes all of my configs to become globals. Globals are bad. Is there a better way or would just about anything else be overkill? A singleton config class or something? Overkill? I have several loops in this code for processing the logfiles. I once tried to convert these for loops to list comprehensions and totally confused myself and backed out the change (yeay svn!). Is there any benefit to list comprehensions in this case? I would kinda like to play with unit tests. Not sure how I would construct unit tests for this. And again, perhaps overkill. But some people tell me you should write the unit tests before you even begin coding and code until the tests are satisfied. So even on a small project you would have tests. I run into a couple nasty bugs I created which caused the script to never return anything from the logfile (so you don't immediately realize something is broken) where I thought It sure would be nice to have a test to catch that if it ever happens again. Stuff tends to build up in the ignore file. If a line is typo'd and never gets used/matched you will never know. If a service is no longer running that we had lines in the ignore file for they will be there forever unused. I don't like this sort of clutter. I am really tempted to somehow make it keep track of when a rule is matched and if it doesn't match in a whole month email the admin because it doesn't need to be there. I am wondering if this is needless complexity or something useful. I think it could be useful. If so, how to save the info on when each line was last matched? I am thinking maybe of just having a dictionary keyed on the compiled regex object and the value being a tuple of the uncompiled regex string from the ignore file and a date string containing when it was last matched and pickling this data structure to disk to be read in each start of the program. Comments? I am currently pushing this code to around 20 systems with cfengine using just one ignore file for them all. If I use the scheme mentioned in the previous paragraph to alert me of unused rules in the ignore file I will get tons of such messages from machines not running a service that other machines of mine are. I guess that means perhaps I should split my ignore file out into ignore.hostname.conf or something like that so there is a separate one for each host. Anyhow, I appreciate any feedback on this potentially very handy sysadmin tool. Thanks in advance! - -- Tracy R Reed http://ultraviolet.org -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.5 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org iD8DBQFFObPK9PIYKZYVAq0RAiflAJ9jprJgGnNRXkB+nKsljFUsAUGnFwCgiJg/ M3U24dNDtH+hAgSh7kZ40hQ= =nxQS -END PGP SIGNATURE- ___ Tutor maillist -
Re: [Tutor] My first real python project: LOLLERSKATES
All I can say to this... LOLLERSKATES =DOn 10/20/06, Tracy R Reed [EMAIL PROTECTED] wrote: -BEGIN PGP SIGNED MESSAGE-Hash: SHA1I have been a sysadmin for quite a while now and used to do a lot of perl. I haven't done much programming at all in the last couple of yearsbut have been meaning to dump perl for python anyhow (for the usualreasons) and have finally gotten around to doing it.The first itch that I wanted to scratch was something to help me find anomalies in our system logs. We have a lot of servers and need someautomation for watching over them including logfiles. I have usedlogwatch and logcheck but they are both too complicated for what Ireally need not to mention practically unmaintained these days. So I decided to write my own.Features I wanted were:1. A basic grep -v sort of functionality on the logfile where I have afile full of regexes/lines to be ignored that are applied to thelogfiles to filter everything uninteresting out and whatever is left gets emailed to the admin.2. A sort of macro system where I can substitute nasty long commonlyused regexes in my filter/ignore file with something more easily readand typed.3. Simplicity. Logcheck had several different levels of logfile events which it could report back on and if you wanted something ignored youhad to put it in the right config file for whatever level it was poppingup in. This drove me nuts because I often got the wrong one. Logwatch tries to analyze your logs and provide various summaries and statisticsand other things I don't care about. I just want to see the interestinglines from the logfile.My code seems to have all of the above and a silly name as well: http://ultraviolet.org/Members/treed/lollerskatesIt has a list of logfiles to look at, a list of regexes to compare toeach line to know what is uninteresting and to filter out/ignore, and emails anything left to the admin, and a simple macro facility.I would appreciate it if anyone interested would download and critiquemy code. Am I doing anything terribly un-pythonic?Questions/problems/TODO's: This is a fairly simple structured programming implementation. No OO.Should I be using some classes somewhere?The config file is just a module which I import which causes all of myconfigs to become globals. Globals are bad. Is there a better way or would just about anything else be overkill? A singleton config class orsomething? Overkill?I have several loops in this code for processing the logfiles. I oncetried to convert these for loops to list comprehensions and totally confused myself and backed out the change (yeay svn!). Is there anybenefit to list comprehensions in this case?I would kinda like to play with unit tests. Not sure how I wouldconstruct unit tests for this. And again, perhaps overkill. But some people tell me you should write the unit tests before you even begincoding and code until the tests are satisfied. So even on a smallproject you would have tests. I run into a couple nasty bugs I createdwhich caused the script to never return anything from the logfile (so you don't immediately realize something is broken) where I thought Itsure would be nice to have a test to catch that if it ever happens again.Stuff tends to build up in the ignore file. If a line is typo'd and never gets used/matched you will never know. If a service is no longerrunning that we had lines in the ignore file for they will be thereforever unused. I don't like this sort of clutter. I am really tempted to somehow make it keep track of when a rule is matched and if itdoesn't match in a whole month email the admin because it doesn't needto be there. I am wondering if this is needless complexity or somethinguseful. I think it could be useful. If so, how to save the info on when each line was last matched? I am thinking maybe of just having adictionary keyed on the compiled regex object and the value being atuple of the uncompiled regex string from the ignore file and a datestring containing when it was last matched and pickling this data structure to disk to be read in each start of the program. Comments?I am currently pushing this code to around 20 systems with cfengineusing just one ignore file for them all. If I use the scheme mentioned in the previous paragraph to alert me of unused rules in the ignore file I will get tons of such messages from machines not running a servicethat other machines of mine are. I guess that means perhaps I should split my ignore file out into ignore.hostname.conf or something likethat so there is a separate one for each host.Anyhow, I appreciate any feedback on this potentially very handysysadmin tool. Thanks in advance! - --Tracy R Reedhttp://ultraviolet.org-BEGIN PGP SIGNATURE-Version: GnuPG v1.4.5 (GNU/Linux)Comment: Using GnuPG with Fedora - http://enigmail.mozdev.orgiD8DBQFFObPK9PIYKZYVAq0RAiflAJ9jprJgGnNRXkB+nKsljFUsAUGnFwCgiJg/M3U24dNDtH+hAgSh7kZ40hQ==nxQS-END PGP SIGNATURE-___ Tutor
