I have now written down a large part of the new calling interface specification. Find it at:
http://www.rsyslog.com/doc-dev_oplugins.html This document is updated as I continue to work on the topic. It does not yet talk about the string quadruple, mostly because this is not yet relevant (and I am not sure if it is relevant at all at the rsyslog core level, I think it is mostly for the individual plugin). Even more importantly, I have done a first proof of concept implementation both inside the rsyslog core as well as the PostgreSQL output module. All of this is contained in the multi-dequeue git branch. From a performance perspective, the code should now be very much at the performance level we can reach with this technology (only expecting another increase if we use prepared statements or the native API). So for performance-testing, this version should be good. Note that it is VITAL that the action queue runs in any non-direct mode, so an $ActionQueueType statement must be given for the action in question, otherwise you do not see any effect of the change. Also, this is far from production code: it works if all goes well and nothing to challenging happens. If, for example, the connection to the database server is lost, it will not abort, but lose those messages that were in the transaction before the problem occurred. Numerous other failure cases exist. So this version is purely for evaluating how the interface looks and how it can be used in code (the ompgsql is the example to look at). Feedback, of course, is appreciated. I'll continue to work on the final solution, a process that involves a lot of thinking but far less coding. So don't wonder if only occasional git updates happen ;) Rainer > -----Original Message----- > From: [email protected] [mailto:rsyslog- > [email protected]] On Behalf Of [email protected] > Sent: Friday, April 24, 2009 10:06 AM > To: rsyslog-users > Subject: Re: [rsyslog] [PERFORM] performance for high-volume > loginsertion(fwd) > > On Fri, 24 Apr 2009, Rainer Gerhards wrote: > > >> -----Original Message----- > >> From: [email protected] [mailto:rsyslog- > >> [email protected]] On Behalf Of [email protected] > >> > >> On Fri, 24 Apr 2009, Rainer Gerhards wrote: > >> > >>>> -----Original Message----- > >>>> From: [email protected] [mailto:rsyslog- > >>>> [email protected]] On Behalf Of [email protected] > >>>> > >>>> On Fri, 24 Apr 2009, Rainer Gerhards wrote: > >>>> > >>>>>> -----Original Message----- > >>>>>> From: [email protected] [mailto:rsyslog- > >>>>>> [email protected]] On Behalf Of [email protected] > >>>>> > >>>>>> that logic can work for every module that needs it, so this > >>>> shouldn't > >>>>>> be > >>>>>> an issue.. I was mixing up the API with the need for a config > >>>> variable. > >>>>>> > >>>>>> David Lang > >>>>> > >>>>> Lol, our messages crossed. Still the question is if the > >> stringbuilder > >>>> should > >>>>> support this mode. If so, we need to make deep changes to the way > >> the > >>>>> property replacer works, thus I am hesitant to do this without > real > >>>> needs > >>>>> (not just because of the work to be done, but also because of the > >>>> extra > >>>>> complexity [read: bugs, performance] it introduces). > >>>> > >>>> if the string builder is not in the core, the output module can do > >> the > >>>> work for 'mid' as/if needed. > >>>> > >>>> table it for now. > >>>> > >>>> food for thought (which may affect the decision when it happens), > >> there > >>>> is > >>>> benifit in doing the database equivalent of dynafiles (inserting > >> into > >>> > >>> Just to avoid misunderstanding: today, this is easy to acomplish - > >> you just > >>> need to use a property replacer expression inside the template > string > >> like > >>> this "insert into syslog%hostname% ...". > >> > >> correct (as long as you didn't need to create the tables) > >> > >>> That, of course, will not work any longer if we go for prepared > >> statements > >>> (indeed another subtlety). I'd still expect that it works with the > >> begin... > >>> insert* ... end exec calls. > >> > >> it will work with the begin;insert;end approach > >> > >> for prepared statements you would need to prepare one for each > >> destination > >> (similar to creating/opening files), and if you ended up doing copy > or > >> multi-value inserts you would need to do seperate ones for different > >> destinations. > >> > >> but in any case, issues for another day. > > > > Yep - and not only because of time zones ;) > > > > I think it is vital to get the calling interface ready (we had quite > some > > progress already :)). The layers below it are important (to help > crafting he > > interface), but some issues I think do not need to be explored in > full detail > > at this level. > > > > The bottom line, I think, is that this is a database-specific problem > and > > other plugins probably do not have any such issue. So, we could > handle that > > in a database abstraction, but not the general calling interface > (assuming > > that the calling interface provides the necessary plumbing). > > > > Please all let me know if I am overlooking something. > > sounds good to me. as you can tell I will speak up if I think you > missed > something :-) > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
