Re: OT:Log scrubber question
Movement on this got pushed back in favor of other work, but I think the end result is a very useful tool. http://cdn.tabtonic.com/ttanon-1.1.zip It's straightforward to use and when I get around to putting up a web page up for it I'll have downloads for Mac/Linux as well. There is an option for automatically replacing usernames. It will grab the usernames as it processes the logs, then process them a second time once all the usernames have been identified. It will do search/replace terms from a CSV file containing two columns that must be named TERM and REPLACEMENT It will also simply censor any words identified in a plain text file (replace any words contained in the file with *s) When it replaces user names, it replaces them with something with the same length - so Demo would be replaced with u002 whereas Mary Manager would be replaced with user0034. This is to keep the structure of the file intact for whatever log analysis tool you want to use. It will do single file replacement or process all files within a given directory. Options can be saved and loaded into the GUI. The GUI will automatically load the last saved configuration on load. The GUI will also give you the command line syntax for you if you just want to run a batch file to process a directory. You can run this from a thumb drive. (A JRE is necessary, but those are installed everywhere these days) Either double click anonymize.exe or run the command line version: usage: anoncmd [options] -c fileconfiguration file -helpprint this message and exit -i file or directory input file or directory -o file or directory output file or directory -p Replace discovered email addresses, phone numbers, and social security numbers with *s -r mapfile.csv Automatically Replace Users -t termfile.csvfile containing private terms (CSV file with two columns: TERM, REPLACEMENT) -w wordfile.txtCensor words (separated by spaces or carriage retuns) On Fri, May 31, 2013 at 4:32 AM, Peter Joran ppjo...@gmail.com wrote: ** Very nice! Thanks again Steve Pete On Thu, May 30, 2013 at 1:49 PM, Steve Kallestad st...@tabtonic.comwrote: ** Yes on both counts Peter. I'm also thinking that it should optionally grab login names and auto-replace them with user001, user002 and have an options file or gui so you don't have to remember the command line. It's in progress... On Thu, May 30, 2013 at 4:36 AM, Peter Joran ppjo...@gmail.com wrote: ** The ARSList is simply the best!! Steve, many thanks for what you have done! I never expected anyone to write something to solve the issue. Your time and effort is greatly appreciated! At the risk of being a cad, how easy would it be to allow the user to determine what the replacement string should be? For example, replace 1.1.1.1 with AR Server, 2.2.2.2 with Mid-tier server, 3.3.3.3 with Database Server, Demo with Admin account, jfrost with Approver account, etc. Meaningful replacement strings would help support to maintain context when reviewing the logs. Also, can the utility be made to run on all files in a folder? Many thanks again Pete On Wed, May 29, 2013 at 8:42 PM, Steve Kallestad st...@tabtonic.comwrote: ** I was going to give some technical advice since I couldn't find any standalone utility like this, but I decided instead to just write one for you. It's pretty straight forward - it needs an input file, an output file, and a private terms file. The private terms file is a file containing words to be censored. They can be space separated, line separated or both. It will read the input file, replace all private terms with *s and it will also replace phone numbers, social security numbers, and email addresses with *s. I couldn't find one of those 2GB log files to test with, but it ran through a 20MB file in about 6 seconds. I shared it on Google Drive: https://docs.google.com/file/d/0ByjmQ_f5-wyKcEhqUTgyTDNLdE0/edit It requires java to be installed. Other than that, just drop it anywhere on your windows box and run it from the command line. Let me know if you have any issues or if you have any feedback on changes that would be useful. Thanks, Steve On Wed, May 29, 2013 at 1:35 PM, Francois Seegers franco...@blueturtle.co.za wrote: ** Hi Pete, If you are running v8 and up you might want to use the Atrium Integrator transformations running scripts on the files…just a thought… Regards Francois *From:* Action Request System discussion list(ARSList) [mailto: arslist@ARSLIST.ORG] *On Behalf Of *Peter Joran *Sent:* Wednesday, May 29, 2013 10:32 PM *To:* arslist@ARSLIST.ORG *Subject:* OT:Log scrubber question ** All I have a need to clean log files before sending them to BMC support. I'm looking for a tool that will allow me to create multiple
Re: OT:Log scrubber question
Very nice! Thanks again Steve Pete On Thu, May 30, 2013 at 1:49 PM, Steve Kallestad st...@tabtonic.com wrote: ** Yes on both counts Peter. I'm also thinking that it should optionally grab login names and auto-replace them with user001, user002 and have an options file or gui so you don't have to remember the command line. It's in progress... On Thu, May 30, 2013 at 4:36 AM, Peter Joran ppjo...@gmail.com wrote: ** The ARSList is simply the best!! Steve, many thanks for what you have done! I never expected anyone to write something to solve the issue. Your time and effort is greatly appreciated! At the risk of being a cad, how easy would it be to allow the user to determine what the replacement string should be? For example, replace 1.1.1.1 with AR Server, 2.2.2.2 with Mid-tier server, 3.3.3.3 with Database Server, Demo with Admin account, jfrost with Approver account, etc. Meaningful replacement strings would help support to maintain context when reviewing the logs. Also, can the utility be made to run on all files in a folder? Many thanks again Pete On Wed, May 29, 2013 at 8:42 PM, Steve Kallestad st...@tabtonic.comwrote: ** I was going to give some technical advice since I couldn't find any standalone utility like this, but I decided instead to just write one for you. It's pretty straight forward - it needs an input file, an output file, and a private terms file. The private terms file is a file containing words to be censored. They can be space separated, line separated or both. It will read the input file, replace all private terms with *s and it will also replace phone numbers, social security numbers, and email addresses with *s. I couldn't find one of those 2GB log files to test with, but it ran through a 20MB file in about 6 seconds. I shared it on Google Drive: https://docs.google.com/file/d/0ByjmQ_f5-wyKcEhqUTgyTDNLdE0/edit It requires java to be installed. Other than that, just drop it anywhere on your windows box and run it from the command line. Let me know if you have any issues or if you have any feedback on changes that would be useful. Thanks, Steve On Wed, May 29, 2013 at 1:35 PM, Francois Seegers franco...@blueturtle.co.za wrote: ** Hi Pete, If you are running v8 and up you might want to use the Atrium Integrator transformations running scripts on the files…just a thought… Regards Francois *From:* Action Request System discussion list(ARSList) [mailto: arslist@ARSLIST.ORG] *On Behalf Of *Peter Joran *Sent:* Wednesday, May 29, 2013 10:32 PM *To:* arslist@ARSLIST.ORG *Subject:* OT:Log scrubber question ** All I have a need to clean log files before sending them to BMC support. I'm looking for a tool that will allow me to create multiple search and replace configurations (rules) that can be used at any time. Not having to create search and replace every time would be a huge time saver. I also need the rules to be persistent and editable as well as being able to apply the search and replace function to a single file or all files in a directory. For example, apply search and replace Rule 1 to file x or apply search and replace Rule 1 to all files in folder y. Rule 1: Find Replace With (could be 1 or 100 replacement items) 1 a 2 b 3 c Rule 2: tom Jones david Parker etc Anyone know of such a tool? Many thanks Pete _ARSlist: Where the Answers Are and have been for 20 years_ -- Blue Turtle Technologies (Pty) Limited | Reg. no.: 2003/002610/07 | http://www.blueturtle.co.za Gauteng : Tel: +27 (0)11 206 5600 | Fax: +27 (0)86 208 0237 | Midridge Office Estate, International Business Gateway, cnr New Road Sixth Street, Midrand, 1685 | P O Box 31331, Kyalami, 1684 Western Cape: Tel: +27 (0)87 721 1874 | Fax: +27 (0)21 552 7764 | Unit E6, Century Square, Heron Crescent, Century City, Cape Town, 7446 DISCLAIMER: This email and any files transmitted with it are confidential and are intended solely for the use of the individual or entity to whom they are addressed. This communication represents the originator's personal views and opinions, which do not necessarily reflect those of Blue Turtle Technologies (Pty) Ltd. If you are not the original recipient or the person responsible for delivering the email to the intended recipient, be advised that you have received this email in error, and that any use, dissemination, forwarding, printing, or copying of this email is strictly prohibited. If you received this email in error, please immediately notify the sender. Thank you. _ARSlist: Where the Answers Are and have been for 20 years_ _ARSlist: Where the Answers Are and have been for 20 years_ _ARSlist: Where the Answers Are and have been for 20 years_ _ARSlist: Where the Answers Are and have been for 20 years_
Re: OT:Log scrubber question
The ARSList is simply the best!! Steve, many thanks for what you have done! I never expected anyone to write something to solve the issue. Your time and effort is greatly appreciated! At the risk of being a cad, how easy would it be to allow the user to determine what the replacement string should be? For example, replace 1.1.1.1 with AR Server, 2.2.2.2 with Mid-tier server, 3.3.3.3 with Database Server, Demo with Admin account, jfrost with Approver account, etc. Meaningful replacement strings would help support to maintain context when reviewing the logs. Also, can the utility be made to run on all files in a folder? Many thanks again Pete On Wed, May 29, 2013 at 8:42 PM, Steve Kallestad st...@tabtonic.com wrote: ** I was going to give some technical advice since I couldn't find any standalone utility like this, but I decided instead to just write one for you. It's pretty straight forward - it needs an input file, an output file, and a private terms file. The private terms file is a file containing words to be censored. They can be space separated, line separated or both. It will read the input file, replace all private terms with *s and it will also replace phone numbers, social security numbers, and email addresses with *s. I couldn't find one of those 2GB log files to test with, but it ran through a 20MB file in about 6 seconds. I shared it on Google Drive: https://docs.google.com/file/d/0ByjmQ_f5-wyKcEhqUTgyTDNLdE0/edit It requires java to be installed. Other than that, just drop it anywhere on your windows box and run it from the command line. Let me know if you have any issues or if you have any feedback on changes that would be useful. Thanks, Steve On Wed, May 29, 2013 at 1:35 PM, Francois Seegers franco...@blueturtle.co.za wrote: ** Hi Pete, If you are running v8 and up you might want to use the Atrium Integrator transformations running scripts on the files…just a thought… Regards Francois *From:* Action Request System discussion list(ARSList) [mailto: arslist@ARSLIST.ORG] *On Behalf Of *Peter Joran *Sent:* Wednesday, May 29, 2013 10:32 PM *To:* arslist@ARSLIST.ORG *Subject:* OT:Log scrubber question ** All I have a need to clean log files before sending them to BMC support. I'm looking for a tool that will allow me to create multiple search and replace configurations (rules) that can be used at any time. Not having to create search and replace every time would be a huge time saver. I also need the rules to be persistent and editable as well as being able to apply the search and replace function to a single file or all files in a directory. For example, apply search and replace Rule 1 to file x or apply search and replace Rule 1 to all files in folder y. Rule 1: Find Replace With (could be 1 or 100 replacement items) 1 a 2 b 3 c Rule 2: tom Jones david Parker etc Anyone know of such a tool? Many thanks Pete _ARSlist: Where the Answers Are and have been for 20 years_ -- Blue Turtle Technologies (Pty) Limited | Reg. no.: 2003/002610/07 | http://www.blueturtle.co.za Gauteng : Tel: +27 (0)11 206 5600 | Fax: +27 (0)86 208 0237 | Midridge Office Estate, International Business Gateway, cnr New Road Sixth Street, Midrand, 1685 | P O Box 31331, Kyalami, 1684 Western Cape: Tel: +27 (0)87 721 1874 | Fax: +27 (0)21 552 7764 | Unit E6, Century Square, Heron Crescent, Century City, Cape Town, 7446 DISCLAIMER: This email and any files transmitted with it are confidential and are intended solely for the use of the individual or entity to whom they are addressed. This communication represents the originator's personal views and opinions, which do not necessarily reflect those of Blue Turtle Technologies (Pty) Ltd. If you are not the original recipient or the person responsible for delivering the email to the intended recipient, be advised that you have received this email in error, and that any use, dissemination, forwarding, printing, or copying of this email is strictly prohibited. If you received this email in error, please immediately notify the sender. Thank you. _ARSlist: Where the Answers Are and have been for 20 years_ _ARSlist: Where the Answers Are and have been for 20 years_ ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Where the Answers Are, and have been for 20 years
Re: OT:Log scrubber question
Yes on both counts Peter. I'm also thinking that it should optionally grab login names and auto-replace them with user001, user002 and have an options file or gui so you don't have to remember the command line. It's in progress... On Thu, May 30, 2013 at 4:36 AM, Peter Joran ppjo...@gmail.com wrote: ** The ARSList is simply the best!! Steve, many thanks for what you have done! I never expected anyone to write something to solve the issue. Your time and effort is greatly appreciated! At the risk of being a cad, how easy would it be to allow the user to determine what the replacement string should be? For example, replace 1.1.1.1 with AR Server, 2.2.2.2 with Mid-tier server, 3.3.3.3 with Database Server, Demo with Admin account, jfrost with Approver account, etc. Meaningful replacement strings would help support to maintain context when reviewing the logs. Also, can the utility be made to run on all files in a folder? Many thanks again Pete On Wed, May 29, 2013 at 8:42 PM, Steve Kallestad st...@tabtonic.comwrote: ** I was going to give some technical advice since I couldn't find any standalone utility like this, but I decided instead to just write one for you. It's pretty straight forward - it needs an input file, an output file, and a private terms file. The private terms file is a file containing words to be censored. They can be space separated, line separated or both. It will read the input file, replace all private terms with *s and it will also replace phone numbers, social security numbers, and email addresses with *s. I couldn't find one of those 2GB log files to test with, but it ran through a 20MB file in about 6 seconds. I shared it on Google Drive: https://docs.google.com/file/d/0ByjmQ_f5-wyKcEhqUTgyTDNLdE0/edit It requires java to be installed. Other than that, just drop it anywhere on your windows box and run it from the command line. Let me know if you have any issues or if you have any feedback on changes that would be useful. Thanks, Steve On Wed, May 29, 2013 at 1:35 PM, Francois Seegers franco...@blueturtle.co.za wrote: ** Hi Pete, If you are running v8 and up you might want to use the Atrium Integrator transformations running scripts on the files…just a thought… Regards Francois *From:* Action Request System discussion list(ARSList) [mailto: arslist@ARSLIST.ORG] *On Behalf Of *Peter Joran *Sent:* Wednesday, May 29, 2013 10:32 PM *To:* arslist@ARSLIST.ORG *Subject:* OT:Log scrubber question ** All I have a need to clean log files before sending them to BMC support. I'm looking for a tool that will allow me to create multiple search and replace configurations (rules) that can be used at any time. Not having to create search and replace every time would be a huge time saver. I also need the rules to be persistent and editable as well as being able to apply the search and replace function to a single file or all files in a directory. For example, apply search and replace Rule 1 to file x or apply search and replace Rule 1 to all files in folder y. Rule 1: Find Replace With (could be 1 or 100 replacement items) 1 a 2 b 3 c Rule 2: tom Jones david Parker etc Anyone know of such a tool? Many thanks Pete _ARSlist: Where the Answers Are and have been for 20 years_ -- Blue Turtle Technologies (Pty) Limited | Reg. no.: 2003/002610/07 | http://www.blueturtle.co.za Gauteng : Tel: +27 (0)11 206 5600 | Fax: +27 (0)86 208 0237 | Midridge Office Estate, International Business Gateway, cnr New Road Sixth Street, Midrand, 1685 | P O Box 31331, Kyalami, 1684 Western Cape: Tel: +27 (0)87 721 1874 | Fax: +27 (0)21 552 7764 | Unit E6, Century Square, Heron Crescent, Century City, Cape Town, 7446 DISCLAIMER: This email and any files transmitted with it are confidential and are intended solely for the use of the individual or entity to whom they are addressed. This communication represents the originator's personal views and opinions, which do not necessarily reflect those of Blue Turtle Technologies (Pty) Ltd. If you are not the original recipient or the person responsible for delivering the email to the intended recipient, be advised that you have received this email in error, and that any use, dissemination, forwarding, printing, or copying of this email is strictly prohibited. If you received this email in error, please immediately notify the sender. Thank you. _ARSlist: Where the Answers Are and have been for 20 years_ _ARSlist: Where the Answers Are and have been for 20 years_ _ARSlist: Where the Answers Are and have been for 20 years_ ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Where the Answers Are, and have been for 20 years
OT:Log scrubber question
All I have a need to clean log files before sending them to BMC support. I'm looking for a tool that will allow me to create multiple search and replace configurations (rules) that can be used at any time. Not having to create search and replace every time would be a huge time saver. I also need the rules to be persistent and editable as well as being able to apply the search and replace function to a single file or all files in a directory. For example, apply search and replace Rule 1 to file x or apply search and replace Rule 1 to all files in folder y. Rule 1: Find Replace With (could be 1 or 100 replacement items) 1 a 2 b 3 c Rule 2: tom Jones david Parker etc Anyone know of such a tool? Many thanks Pete ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Where the Answers Are, and have been for 20 years
Re: OT:Log scrubber question
Hi Pete, If you are running v8 and up you might want to use the Atrium Integrator transformations running scripts on the files...just a thought... Regards Francois From: Action Request System discussion list(ARSList) [mailto:arslist@ARSLIST.ORG] On Behalf Of Peter Joran Sent: Wednesday, May 29, 2013 10:32 PM To: arslist@ARSLIST.ORG Subject: OT:Log scrubber question ** All I have a need to clean log files before sending them to BMC support. I'm looking for a tool that will allow me to create multiple search and replace configurations (rules) that can be used at any time. Not having to create search and replace every time would be a huge time saver. I also need the rules to be persistent and editable as well as being able to apply the search and replace function to a single file or all files in a directory. For example, apply search and replace Rule 1 to file x or apply search and replace Rule 1 to all files in folder y. Rule 1: Find Replace With (could be 1 or 100 replacement items) 1 a 2 b 3 c Rule 2: tom Jones david Parker etc Anyone know of such a tool? Many thanks Pete _ARSlist: Where the Answers Are and have been for 20 years_ Blue Turtle Technologies (Pty) Limited | Reg. no.: 2003/002610/07 | http://www.blueturtle.co.za Gauteng : Tel: +27 (0)11 206 5600 | Fax: +27 (0)86 208 0237 | Midridge Office Estate, International Business Gateway, cnr New Road Sixth Street, Midrand, 1685 | P O Box 31331, Kyalami, 1684 Western Cape: Tel: +27 (0)87 721 1874 | Fax: +27 (0)21 552 7764 | Unit E6, Century Square, Heron Crescent, Century City, Cape Town, 7446 DISCLAIMER: This email and any files transmitted with it are confidential and are intended solely for the use of the individual or entity to whom they are addressed. This communication represents the originator's personal views and opinions, which do not necessarily reflect those of Blue Turtle Technologies (Pty) Ltd. If you are not the original recipient or the person responsible for delivering the email to the intended recipient, be advised that you have received this email in error, and that any use, dissemination, forwarding, printing, or copying of this email is strictly prohibited. If you received this email in error, please immediately notify the sender. Thank you. ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Where the Answers Are, and have been for 20 years
Re: OT:Log scrubber question
I was going to give some technical advice since I couldn't find any standalone utility like this, but I decided instead to just write one for you. It's pretty straight forward - it needs an input file, an output file, and a private terms file. The private terms file is a file containing words to be censored. They can be space separated, line separated or both. It will read the input file, replace all private terms with *s and it will also replace phone numbers, social security numbers, and email addresses with *s. I couldn't find one of those 2GB log files to test with, but it ran through a 20MB file in about 6 seconds. I shared it on Google Drive: https://docs.google.com/file/d/0ByjmQ_f5-wyKcEhqUTgyTDNLdE0/edit It requires java to be installed. Other than that, just drop it anywhere on your windows box and run it from the command line. Let me know if you have any issues or if you have any feedback on changes that would be useful. Thanks, Steve On Wed, May 29, 2013 at 1:35 PM, Francois Seegers franco...@blueturtle.co.za wrote: ** Hi Pete, If you are running v8 and up you might want to use the Atrium Integrator transformations running scripts on the files…just a thought… Regards Francois *From:* Action Request System discussion list(ARSList) [mailto: arslist@ARSLIST.ORG] *On Behalf Of *Peter Joran *Sent:* Wednesday, May 29, 2013 10:32 PM *To:* arslist@ARSLIST.ORG *Subject:* OT:Log scrubber question ** All I have a need to clean log files before sending them to BMC support. I'm looking for a tool that will allow me to create multiple search and replace configurations (rules) that can be used at any time. Not having to create search and replace every time would be a huge time saver. I also need the rules to be persistent and editable as well as being able to apply the search and replace function to a single file or all files in a directory. For example, apply search and replace Rule 1 to file x or apply search and replace Rule 1 to all files in folder y. Rule 1: Find Replace With (could be 1 or 100 replacement items) 1 a 2 b 3 c Rule 2: tom Jones david Parker etc Anyone know of such a tool? Many thanks Pete _ARSlist: Where the Answers Are and have been for 20 years_ -- Blue Turtle Technologies (Pty) Limited | Reg. no.: 2003/002610/07 | http://www.blueturtle.co.za Gauteng : Tel: +27 (0)11 206 5600 | Fax: +27 (0)86 208 0237 | Midridge Office Estate, International Business Gateway, cnr New Road Sixth Street, Midrand, 1685 | P O Box 31331, Kyalami, 1684 Western Cape: Tel: +27 (0)87 721 1874 | Fax: +27 (0)21 552 7764 | Unit E6, Century Square, Heron Crescent, Century City, Cape Town, 7446 DISCLAIMER: This email and any files transmitted with it are confidential and are intended solely for the use of the individual or entity to whom they are addressed. This communication represents the originator's personal views and opinions, which do not necessarily reflect those of Blue Turtle Technologies (Pty) Ltd. If you are not the original recipient or the person responsible for delivering the email to the intended recipient, be advised that you have received this email in error, and that any use, dissemination, forwarding, printing, or copying of this email is strictly prohibited. If you received this email in error, please immediately notify the sender. Thank you. _ARSlist: Where the Answers Are and have been for 20 years_ ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Where the Answers Are, and have been for 20 years
