On Fri, 25 Oct 2013, Rainer Gerhards wrote:
On Fri, Oct 25, 2013 at 1:07 PM, David Lang <[email protected]> wrote:
the one functionality that is really problematic with the old global
variables is doing math on them (counter functionality)
any set operation is counter-intutive.
If you have e.g.
set $/prevtag = $tag
if $tag == "this" and $prevtag == "that" then ...
you get far from the expected result.
no matter what we do with global/state variables, this will be problematic
because the two log messages may end up being processed by different threads,
the first log message being at the end of a batch for thread 1 while the second
log message being at the beginning of a batch processed by thread 2 and therefor
be processed first.
SIMD can be thought of as effectively having a thread per message spawned for
each statement for every message in the batch. They aren't OS threads, but the
resulting race conditions are similar (just that the 'odd' thing happens
regularly)
But even without SIMD, threading and batching will break the same use cases,
just not as consistantly.
I think a statement along the lines of the following is going to be needed in
any case, and with such expectations for state variables set (as opposed to
global variables, which as you say bring special expectations), more
restrictions may not be needed.
Rsyslog is makes very heavy use of threads and out-of-order log processing for
performance. This results in two major limitations for the use of state
variables
Changes to state variables may not be visible when processing log messages
that arrive 'shortly' after the log message that triggers the change, or they
may be visible when processing log messages that arrived 'shortly' before the
log message that triggers the change. On systems handling hundreds of thousands
of logs per second, 'shortly' can be +- a few thousand log messages depending on
the configuration, generally still within a small fraction of a second.
State variables may be changed multiple times during the processing of a
single log message as the result of the processing other log messages.
This means that State variables are not suitable for several use cases
Log Correlation (if message A is followed immediatly by message B)
Counters ($/x = $/x + 1)
State variables are useful for things that change infrequently, and where the
results of the changes do not have to take effect immediately. Such uses
include:
setting per-system values at startup that never change
toggling outputs (if debuglog then set $/debug=1; if $/debug==1
then <action>)
redirecting outputs (if specialmessage then set $/filename=re_extract())
you could implement the entire shadow system as you outlined and narrow the
problem a little bit, but without eliminating threads and batching, I don't
think you can eliminate the problems.
So I think that rather than trying to "make global variables work", the better
answer is to rename them, and implement a separate set of counter related
functions.
David Lang
Rainer
I am thinking that if we remove that capability (or at least, strongly
discourage it, something like "rsyslog makes very heavy use of out-of-order
execution, and as a result, doing math with global variables will lead to
very unintuitive results"), the rest of things can stay as-is. My article
didn't describe doing math on them. We can log a warning if someone does
math on a global variable (rate limited, or only the first time per
execution/thread)
yes, it will change some of the other documentation, but only the examples
in them
David Lang
______________________________**_________________
rsyslog mailing list
http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
