Movement on this got pushed back in favor of other work, but I think the end result is a very useful tool.
http://cdn.tabtonic.com/ttanon-1.1.zip It's straightforward to use and when I get around to putting up a web page up for it I'll have downloads for Mac/Linux as well. There is an option for automatically replacing usernames. It will grab the usernames as it processes the logs, then process them a second time once all the usernames have been identified. It will do search/replace terms from a CSV file containing two columns that must be named "TERM" and "REPLACEMENT" It will also simply censor any words identified in a plain text file (replace any words contained in the file with "*"s) When it replaces user names, it replaces them with something with the same length - so "Demo" would be replaced with "u002" whereas "Mary Manager" would be replaced with "user00000034". This is to keep the structure of the file intact for whatever log analysis tool you want to use. It will do single file replacement or process all files within a given directory. Options can be saved and loaded into the GUI. The GUI will automatically load the last saved configuration on load. The GUI will also give you the command line syntax for you if you just want to run a batch file to process a directory. You can run this from a thumb drive. (A JRE is necessary, but those are installed everywhere these days) Either double click "anonymize.exe" or run the command line version: usage: anoncmd [options] -c <file> configuration file -help print this message and exit -i <file or directory> input file or directory -o <file or directory> output file or directory -p Replace discovered email addresses, phone numbers, and social security numbers with "*"s -r <mapfile.csv> Automatically Replace Users -t <termfile.csv> file containing private terms (CSV file with two columns: TERM, REPLACEMENT) -w <wordfile.txt> Censor words (separated by spaces or carriage retuns) On Fri, May 31, 2013 at 4:32 AM, Peter Joran <ppjo...@gmail.com> wrote: > ** > Very nice! > Thanks again Steve > Pete > > > On Thu, May 30, 2013 at 1:49 PM, Steve Kallestad <st...@tabtonic.com>wrote: > >> ** >> Yes on both counts Peter. >> >> I'm also thinking that it should optionally grab login names and >> auto-replace them with "user001", "user002" and have an options file or gui >> so you don't have to remember the command line. >> >> It's in progress... >> >> >> On Thu, May 30, 2013 at 4:36 AM, Peter Joran <ppjo...@gmail.com> wrote: >> >>> ** >>> The ARSList is simply the best!! >>> Steve, many thanks for what you have done! I never expected anyone to >>> write something to solve the issue. Your time and effort is greatly >>> appreciated! >>> >>> At the risk of being a cad, how easy would it be to allow the user to >>> determine what the replacement string should be? For example, replace >>> 1.1.1.1 with AR Server, 2.2.2.2 with Mid-tier server, 3.3.3.3 with Database >>> Server, Demo with Admin account, jfrost with Approver account, etc. >>> Meaningful replacement strings would help support to maintain context when >>> reviewing the logs. Also, can the utility be made to run on all files in a >>> folder? >>> >>> Many thanks again >>> Pete >>> >>> >>> On Wed, May 29, 2013 at 8:42 PM, Steve Kallestad <st...@tabtonic.com>wrote: >>> >>>> ** >>>> I was going to give some technical advice since I couldn't find any >>>> standalone utility like this, but I decided instead to just write one for >>>> you. >>>> >>>> It's pretty straight forward - it needs an input file, an output file, >>>> and a private terms file. >>>> >>>> The private terms file is a file containing words to be censored. They >>>> can be space separated, line separated or both. >>>> >>>> It will read the input file, replace all private terms with "*"s and it >>>> will also replace phone numbers, social security numbers, and email >>>> addresses with "*"s. I couldn't find one of those 2GB log files to test >>>> with, but it ran through a 20MB file in about 6 seconds. >>>> >>>> I shared it on Google Drive: >>>> >>>> https://docs.google.com/file/d/0ByjmQ_f5-wyKcEhqUTgyTDNLdE0/edit >>>> >>>> It requires java to be installed. Other than that, just drop it >>>> anywhere on your windows box and run it from the command line. >>>> >>>> Let me know if you have any issues or if you have any feedback on >>>> changes that would be useful. >>>> >>>> Thanks, >>>> Steve >>>> >>>> >>>> On Wed, May 29, 2013 at 1:35 PM, Francois Seegers < >>>> franco...@blueturtle.co.za> wrote: >>>> >>>>> ** >>>>> >>>>> Hi Pete, >>>>> >>>>> >>>>> >>>>> If you are running v8 and up you might want to use the Atrium >>>>> Integrator transformations running scripts on the files…just a thought… >>>>> >>>>> >>>>> >>>>> Regards >>>>> >>>>> Francois >>>>> >>>>> >>>>> >>>>> *From:* Action Request System discussion list(ARSList) [mailto: >>>>> arslist@ARSLIST.ORG] *On Behalf Of *Peter Joran >>>>> *Sent:* Wednesday, May 29, 2013 10:32 PM >>>>> *To:* arslist@ARSLIST.ORG >>>>> *Subject:* OT:Log scrubber question >>>>> >>>>> >>>>> >>>>> ** >>>>> >>>>> All >>>>> >>>>> I have a need to clean log files before sending them to BMC support. >>>>> I'm looking for a tool that will allow me to create multiple search and >>>>> replace configurations (rules) that can be used at any time. Not having to >>>>> create search and replace every time would be a huge time saver. I also >>>>> need the rules to be persistent and editable as well as being able to >>>>> apply >>>>> the search and replace function to a single file or all files in a >>>>> directory. >>>>> >>>>> For example, apply search and replace Rule 1 to file x or apply search >>>>> and replace Rule 1 to all files in folder y. >>>>> >>>>> >>>>> Rule 1: >>>>> >>>>> Find Replace With (could be 1 or 100 replacement items) >>>>> >>>>> 1 a >>>>> >>>>> 2 b >>>>> >>>>> 3 c >>>>> >>>>> Rule 2: >>>>> >>>>> tom Jones >>>>> >>>>> david Parker >>>>> >>>>> etc >>>>> >>>>> Anyone know of such a tool? >>>>> >>>>> >>>>> >>>>> Many thanks >>>>> >>>>> Pete >>>>> >>>>> _ARSlist: "Where the Answers Are" and have been for 20 years_ >>>>> >>>>> ------------------------------ >>>>> >>>>> Blue Turtle Technologies (Pty) Limited | Reg. no.: 2003/002610/07 | >>>>> http://www.blueturtle.co.za >>>>> Gauteng : Tel: +27 (0)11 206 5600 | Fax: +27 (0)86 208 0237 | >>>>> Midridge Office Estate, International Business Gateway, cnr New Road & >>>>> Sixth Street, Midrand, 1685 | P O Box 31331, Kyalami, 1684 >>>>> Western Cape: Tel: +27 (0)87 721 1874 | Fax: +27 (0)21 552 7764 | >>>>> Unit E6, Century Square, Heron Crescent, Century City, Cape Town, 7446 >>>>> >>>>> DISCLAIMER: This email and any files transmitted with it are >>>>> confidential and are intended solely for the use of the individual or >>>>> entity to whom they are addressed. This communication represents the >>>>> originator's personal views and opinions, which do not necessarily reflect >>>>> those of Blue Turtle Technologies (Pty) Ltd. If you are not the original >>>>> recipient or the person responsible for delivering the email to the >>>>> intended recipient, be advised that you have received this email in error, >>>>> and that any use, dissemination, forwarding, printing, or copying of this >>>>> email is strictly prohibited. If you received this email in error, please >>>>> immediately notify the sender. Thank you. >>>>> _ARSlist: "Where the Answers Are" and have been for 20 years_ >>>>> >>>> >>>> _ARSlist: "Where the Answers Are" and have been for 20 years_ >>>> >>> >>> _ARSlist: "Where the Answers Are" and have been for 20 years_ >>> >> >> _ARSlist: "Where the Answers Are" and have been for 20 years_ > > > _ARSlist: "Where the Answers Are" and have been for 20 years_ > _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org "Where the Answers Are, and have been for 20 years"
