On 02/06/2014 01:38 AM, John P. Rouillard wrote: > > In message > <ca+pmdhkbk9j94rchwafonh9w07v-ktyky2uahbq5q+kudbm...@mail.gmail.com> , > Edward Gleeck writes: > >> Is there a way for sec to handle dynamically generated log files in a >> directory. Our application would often move a log file from a directory and >> create new ones (sort of the way log rotate can be configured to run). I >> was thinking more like sec --input=/dir/*.log which monitor the directory > > That is a valid input specification, but it only evaluates the *.log > on startup or via a restart signal (SIGHUP or SIGABRT, see the man > page for details). > >> for new files that are being generated. If SEC does not support this, how >> can you use SEC in this scenario? > >>From an intro coursebook I wrote a few years back: > > ============ > > Some processes create multiple log files that are date stamped or > otherwise uniquely named. SEC however wants a static name to be able > to follow the log file. The following rule allows you to have SEC > follow files where the most recent log file has a unique name. > > This will work only if the period between creating new files is longer > than 2 minutes. The basic idea is: > > * have sec follow a single input file (called current) > * identify newest of the log files > * see if the inode of that file is the same as current file > * if different, hard link file to current > * use a Calendar rule that fires every minute to check/create link > > This rule identifies the most recent file in the log directory and > hard links it to the file named current in the working directory of > the SEC process. The directory holding the file current is writable by > the user running the SEC process while the parent directory is not > writable to the SEC user. > > #****v* calendar/relink_current > # SYNOPSIS > # relink_current - created hard link to newest log file in the parent > directory > # DESCRIPTION > # Every minute find the most recently written log file in the parent directory > # and obtain its inode number. Obtain the inode number of the file current > # (which must exist). If they have the same inode number, exit status 0. If > # they don't force current to be linked to the most recently written log file. > #****** > type = calendar > time = * * * * * > desc = relink current file > action = shellcmd (/bin/sh -c 'lastfile=`ls -tr .. | grep '\\.log\$' | tail > -n 1`; [ `stat --printf "%i" current` -eq `stat --printf "%i" ../$lastfile` ] > || ln -f ../$lastfile current') > > The shellcmd identifies the last log file by listing the parent > directory with the files sorted by date. The grep limits the file list > to log files. The tail then selects just the last file. The test > command uses stat to see if the inodes for the current and last file > are the same. If they are not, the current file is forcibly replaced > with a link to the newest log file. > > Since SEC follows the current file until there is no more input it > will process the old log file to the end. Then when there is no more > data, it detects that the current file has changed and starts > processing it from the beginning. No events should be lost, but the > processing of data in the new log file can be delayed by up to a > minute. > > ======================== > > IIRC if you use a symbolic link in place of the hard link and the > target file disapears, and then the link is changed, sec takes longer > to recognize the change, so the hard link is your best bet.
Just a small remark -- the symbolic link indeed incurs a small processing overhead, but this is only related to the fact that it is not directly referring to i-node. When a symbolic link is re-created and pointed to an input file which does not exist yet, SEC sees it as if the previous input file had been deleted. In other words, if symbolic link 'input' points to 'log1' and 'log2' does not exist, then the command ln -sf log2 input has the same effect as removing regular input file. When this condition is observed, SEC will simply continue to keep 'log1' open. If at some point in time 'log2' is created, it is recognized the same way as the re-creation of previously deleted regular input file. When SEC observes this condition, it will open 'log2' and process it from the beginning. The solution that John proposed helps to address all scenarios, since it does not care at which time intervals input files might be created and handles all irregular time frames. However, there is one specific scenario where symbolic link might actually be helpful. If you know that new input files are always created at specific time moments, you could set up a Calendar rule for taking action at these specific moments. For example, if input files are created on daily basis, you could have the following rule: type=Calendar time=0 0 * * * desc=set up a link for new log file action=shellcmd ln -sf log-`date +"%%Y-%%m-%%d"` input This rule would point 'input' to 'log-YYYY-MM-DD' at the beginning of each day (at 00:00). The symbolic link is helpful here, because it sets up the link even if log-YYYY-MM-DD has not yet been created for the given day (if that is the case, then SEC will keep the 'log-' for previous day open, until the new file appears). One extra thing to note is that if you would like to pass % signs to shell, they should be written as %%, since otherwise %Y, %m and %d would be taken for action list variables. Pre-2.6.X versions of SEC did not substitute non-existing action list variables, but this caused confusion and therefore newer versions replace them with empty strings (just as majority of unix shells and other tools do). kind regards, risto > > -- > -- rouilj > John Rouillard > =========================================================================== > My employers don't acknowledge my existence much less my opinions. > > ------------------------------------------------------------------------------ > Managing the Performance of Cloud-Based Applications > Take advantage of what the Cloud has to offer - Avoid Common Pitfalls. > Read the Whitepaper. > http://pubads.g.doubleclick.net/gampad/clk?id=121051231&iu=/4140/ostg.clktrk > _______________________________________________ > Simple-evcorr-users mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users > > ------------------------------------------------------------------------------ Managing the Performance of Cloud-Based Applications Take advantage of what the Cloud has to offer - Avoid Common Pitfalls. Read the Whitepaper. http://pubads.g.doubleclick.net/gampad/clk?id=121051231&iu=/4140/ostg.clktrk _______________________________________________ Simple-evcorr-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
