On Thu, 2 Apr 2009, Rainer Gerhards wrote:
> Just a partial response (but quick ;))
>
>> I confess to being a bit confused as to why the existing output module
>> interface wasn't readily extending to batching,
>
> That would be the real solution (and David Lang suggested it long ago). The
> only problem is it takes quite some effort, as we need to make sure we do not
> lose messages along that way.
for those who are interested, what I proposed was to shift completely away
from the idea that the output module processes a fixed number of records,
and instead have a loop something like the following.
while (events)
if (# events > N)
grab first N events
else
grab all events
create sql string
insert to database
mark the events grabbed as written
with the create sql string being something like the following perlish code
$sql=$header.join($mid,@events).$footer;
so you could say
$header='insert into table logs values ('
$mid = '),('
$footer=);
and if you pass it three events you get
insert into table logs values (msg1),(msg2),(msg3);
five values you would get
insert into table logs values (msg1),(msg2),(msg3),(msg4),(msg5);
I was not concerned about the command parsing time, due to the fact that
if it takes a little longer, it just means that there are more events in
the queue for the next pass to handle. there could reach a point where you
have so many events that it matters, but since this process could easily
insert hundreds or thousands of messages in one statement the overhead is
pretty low
> It is still on my agenda, but without a sponsor I fear it'll stay there
> for quite a while.
still hoping
> In most cases, rsyslog is simply too fast to see a bottleneck.
true
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com
