On Fri, Oct 25, 2013 at 12:34 PM, David Lang <[email protected]> wrote: > On Fri, 25 Oct 2013, Rainer Gerhards wrote: > > On Fri, Oct 25, 2013 at 10:42 AM, David Lang <[email protected]> wrote: >> >> On Fri, 25 Oct 2013, Rainer Gerhards wrote: >>> >>> Hi all, >>> >>>> >>>> I thought out the details of what I have on my mind and think the >>>> solution >>>> can work and support all known use cases. I've also managed to write it >>>> down this morning: >>>> >>>> http://blog.gerhards.net/2013/****10/a-proposal-for-rsyslog-**<http://blog.gerhards.net/2013/**10/a-proposal-for-rsyslog-**> >>>> state-variables.html<http://**blog.gerhards.net/2013/10/a-** >>>> proposal-for-rsyslog-state-**variables.html<http://blog.gerhards.net/2013/10/a-proposal-for-rsyslog-state-variables.html> >>>> > >>>> >>>> >>>> I would appreciate if you could check it and see how the spec can be >>>> technically broken or identify use cases which it will not be able to >>>> handle. >>>> >>>> >>> >>> One case I don't think it can handle is the current work that mmcount is >>> doing. >>> >>> As I understand mmcount, it creates a counter for each appname that it >>> sees, and uses that to count how many times that appname has been seen. >>> >>> the fact that you can use $/a!b lets you have the variables without >>> having >>> the conflict with anything else, >>> >>> but you end up with no way to know what variables exist >>> >>> you can't output the entire set of counts without being able to use $/a >>> >>> you can't even lookup the count for the current message's appname without >>> being able to use domething like $a/{$appname} >>> >>> >>> you are right, that use case breaks. Thanks! I'll review mmcount in more >> depth now and see how it is actually used. >> >> The ability to support subtrees can be added, but will be relatively >> expensive. >> > > I don't think you need normal full subtree support, you could get away > with a function like export($/a) to return the tree, this could be with > atomic locking if needed. > > I think the hardest thing to do (although possibly the most valuable in > the long run) is the concept of $[/!.]a!{$b} to allow you to use a variable > as part of the name of another variable.
right - but we need to think a bit of what we want to do with the *config* file script. This was never intended to morph into a real programming language with an underlying syslog library ;) Beside this general consideration, I agree (and tend to think that this doesn't yet go totally over the edge). Indirection may even not so hard to implement -- but I want to stay focussed, I can't do this now as a side-activity in any case (and global vars tell a bit about doing something quickly as a side-activity ;)). > > depending on how large the table of global variables ends up, I wonder if >>> it would be easier (and because of the simpler design, possibly even >>> faster) to just make a shadow copy of all global variables at the start >>> of >>> the message processing, (I'm thinking the RCU read-copy-update) mechanism >>> may be a good fit for this) >>> >>> >> I don't think so, as with each of this "copies" I would need to update >> work flags (like modifiable) for all variables. Also, I need to know which >> one is actually shadowed, so that would be a new flag (currently, it is >> just a failed lookup on the shadow dictionary). >> > > my thought is that you essentially shadow everything. each variable would > have a flag that in the master copy is modifiable. > > for each message, you would > > grab a pointer to the 'current' master table. > > if all you do is read variables, you read them from the copy you have a > pointer to. > > when you do an update, you copy the entire table, modify your copy and it > becomes the new master copy (with a different pointer), any other thread > has a pointer to the old copy and keeps using it (making the old master > become the 'shadow' version for all the other threads). once all threads > have finished a batch, old copies can be garbage collected (how to track > this is a different discussion that I will leave out for the moment for > simplicity) > That won't work, as it would not permit in-batch math. Assume 10 messages and one counter var (currently at 1). Each message would update it's own shadow, going from 1 to 2. So after batch execution, each of these shadows would have the value 2, whereas it would need to be 2, 3, 4, 5, ..., 11 for the respective messages. So updates must actually be made to a global state. > > atomic operations would have to lock the pointer to the current master > table, check to see if it matches the pointer the thread is using, if so, > continue normally, if not, it would have to update the variable based on > the current master, not it's 'shadow' copy. > > > My assumption (I should >> have spelled this out) is also that we have a very low number of state >> variable updates, and even a low number of read accesses to them. >> > > I think this assumption is incorrect. I think there are four distinct > use-cases > > 1. no global variables in use > > no updates, no reads. (obviously :-) > > 2. global variables used for configuration/path type capabilities > > few, if any updates, extremely frequent reads. > > 3a. counters for reporting > > extremely frequent updates, infrequent reads. > > 3b. counters for load balancing or sequence numbers > > extremely frequent updates and reads. > > > sorry, my wording implied another thing. I actually did not mean the number of update opertions, but the number of variables involved. > the RCU approach works extremely well for cases #1 and #2, but not for #3a > and #3b > > I'm actually thinking that we should possibly split the two use cases. > > State Variables ($/) would be intended for infrequent updates (case #2) > and could be handled very well with the RCU style mechanism that I'm > talking about. If you use them for counters, performance will be poor and > you have no atomic operations > > Counter Variables would only be accessed through function calls > > atomic_set(counter='varname' value=number) > sets value, returns oldvalue > > atomic_add(counter='varname' step=number) > modifies value, returns newvalue > > atomic_report(counter='**varname') > returns value > > Counter variables would not be shadowed, they would be true global > variables, and can be modified in unexpected ways between accesses. if you > want to use the value of a counter for something, you save it's value into > a local/message variable. Because these do not look or act like normal > variables, there isn't the expectation that you can do math and multiple > actions on them. > > A general note: I have done already very large changes during the refactoring. Thus, I would like to keep whatever I do to as simple as possible, to reduce any further bug potential. We can always optimize later on. Also, I really have hard time constraints (getting harder each day...). The simpler the idea, the faster I can code it up. I may become a question between "I can do it" and "I put it on the TODO list and finally disable global vars without replacement". So let's try to stay focussed on the simple solution. Optimizations can always happen later, even month later. Just think that I orignally wanted to improve imfile for those frequently requested use cases (much more frequently requested than state variables!). This is already dead for the next time, and I am not very happy with that. So let's try to get to a very simple, but *working* solution. As a side-note, I am not sure if I can work on any of this next week, so I had preferred to finish some things today, but I guess that's no longer possible (and not so since yesterday, so it probably looked bad since mid-week...). Rainer _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
