On 07/09/2014 09:15 AM, leam hall wrote: > I could use some feedback on a thought process. > > Situation: Lots machines (3-4 digit count), lots of scripts on each > machine (100-500). The scripts are identified by a set alphanumeric > scheme and the scripts will either pass, fail, or fix the issue the > script refers to. > > Planned output is one file per machine in the following format: > > servername 2017-07-09 scriptID Pass > > The idea is that the relevant outputs can be collated into one larger > file for searching, or input into a database. > > What are your thoughts on this? Ideally others will use this so I'm > trying to think outside my own habits/needs. Feel free to bounce ideas > around, you probably have ideas I haven't even begun to ponder.
Some questions to consider: * How critical will it be that you collect the output of every script every time it runs? * How likely will it be that you will have to scale this past the initial environment? This might happen in machine count, script count, or script run frequency. * How likely will it be that you will have to change the format of the message? * Do you have any security issues to consider (i.e. do messages have to be signed, will the database have to be opened to the world)? * What kind of queries do you expect to ask of the data? How often do you expect to run them? With this many scripts and systems, I would consider using some kind of message bus rather than having the scripts (or even an agent running on the clients) write directly to the database. You could be looking at hundreds or thousands of simultaneous connections, and most databases are not designed for that kind of use. Instead, have the scripts output data into the message bus, and have a pool of consumer processes that read messages from the bus and into the database. The consumer processes would start up a connection to the database, prepare one or more statements, and use the cached statement handles for every message they consume. This is /much/ more efficient than setting up a fresh TCP connection, getting the query planner to parse a statement, insert/commit a single row, and then tear everything apart. Depending on your database technology, using stored procedures can be much more efficient than ad-hoc statements, and will definitely be more secure for non-authenticated connections. Skylar _______________________________________________ Discuss mailing list [email protected] https://lists.lopsa.org/cgi-bin/mailman/listinfo/discuss This list provided by the League of Professional System Administrators http://lopsa.org/
