John, As usual, thanks for such a well thought out response! I will follow up with this if/when my proposal to management goes to a lab phase.
Thanks SO much for your help! ______________________________________________________________ Clayton Dukes ______________________________________________________________ On Tue, Aug 10, 2010 at 2:19 PM, John P. Rouillard <[email protected]>wrote: > > In message <[email protected]>, > Clayton Dukes writes: > >We would like to set up our servers in a high availability architecture > >where 2 servers would send syslog messages to two receiving servers. > >I have proposed that we utilize SEC on the two receiving servers as > depicted > >in the image below to monitor for any duplicate messages and discard the > >duplicate. This would ensure delivery in the event that one of the servers > >were to fail. Can SEC provide this? > > You can see a slight discussion on redundancy (a few paragraphs) in > > http://www.cs.umb.edu/~rouilj/sec/sec_paper_full.pdf > > section 4.4. In that I recommend a heartbeat senario so only one SEC > has its rules enabled. I did test that setup at one point and it > worked pretty well (although it did take about 35 seconds worst case > for the heartbeat (sent every 30 seconds) to fail, be detected and the > second SEC instance to come on line. > > The ruleset: > http://www.cs.umb.edu/~rouilj/sec/rulesets/01report_myself_only.sr > > may be of some use. It's meant to use my parallel rules framework, but > the basic idea is: > > if the context filter_MYHOST.example.com exists, capture all the > inbound events and signal to all the rest of the rules to ignore > the event by creating the EVENT_PROCESSED context. (This could be > done using the jump commands in newer SEC's much more easily.) > > in the redundant SEC, create the context filter_MYHOST.example.com > when the heartbeat comes in with a lifetime of heartbeat interval > + 1 second. > > the heartbeat event could be sent over a shared file (nfs...), > netcat etc. On the primary/master set up a command that fires > every heartbeat interval and writes the heartbeat event to the > transfer mechanism. (IF the hearbeat should be less than a > minute, you can use a calendar rule to create contexts with > appropriate lifetimes say: 15, 30, 45 seconds that generate the > heartbeat message when the expire). > > I could see this being adapted so that rather then heartbeat's being > sent you send the actual events and turn them into short lived > contexts. You would want to use an input channel that has a context > associated with it. E.G. on the SEC command line > > -input=/tmp/heartbeats=heartbeats > > and place this rule first: > > type = single > ptype = regexp > pattern = (.*) > context = heartbeats > action = create $1 2 > > to create a context that can be matched against an event for 2 > seconds. This only looks at events arriving over the heartbeat > channel, and it consumes them so they are not passed on. Then the > following rule should be applied to every event comming from > syslog-ng: > > type = single > ptype = regexp > pattern = (.*) > context = $1 > action = none > > would capture and suppress the event from propigating through the > redundant sec. However this assumes a couple of things: > > the master gets the event to the redundat system before the message > arrives at the redundant system. If not the message gets emitted > twice, once from each sec. > > that the delay between the two systems seeing the message is < 2. > if the redundant system gets the message 3 seconds later it will > process it. > > However one thing that I never got working right was duplicating the > state of the two SECS so that they would have the same internal state > and if one failed in the middle of a threshold count the redundant > node would pick up counting where the first one died. > > Also in your senario it looks like you want multiple masters not just > a master/redundant backup, which is very tricky and I can sort of see > how I would do it, but there are a lot of timing and other details to > deal with, and I am not sure I would try doing that. > > If you can afford to lose a few messages that are in transit it would > be easier. > > Once other thing that could be done is to have syslog-ng feed a > (custom written) queue that preserves inter-event timing. Then have > the queue delay feeding the event by heartbeat seconds to the > redundant sec. Think of the 4 second delay in broadcasting. > > If the heartbeat fails during the queue delay time, all events over > the heartbeat delay time are stored in the queue and ready for > replay. You still have the lack of internal SEC state to deal with, > but this should preserve all the events. > > -- > -- rouilj > John Rouillard > =========================================================================== > My employers don't acknowledge my existence much less my opinions. >
------------------------------------------------------------------------------ This SF.net email is sponsored by Make an app they can't live without Enter the BlackBerry Developer Challenge http://p.sf.net/sfu/RIM-dev2dev
_______________________________________________ Simple-evcorr-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
