I think we should have at least a stub module in each language in the main
repository. All the stubs could do the same 'write to file', but they would give
people a place to start
Actually, I think we need to do three modules in every language
read from file
write to file
message modification that gets the message and passes it back.
while we could just use pipes and stdin/stdout, I've been thinking about this a
bit more over the last week or so, and think we should really go ahead and be
more formal about this
parsing messages is hard, formatting messages to be parsed is error prone, and
both tend to be slow, and when working with text messages, tend to accumulate a
lot of cruft to work around corner cases.
Also, as Rainer said above, we need to have the ability to get feedback from
these other languages, and I'm thinking we probably need to be able to send
command-type messages (we just got a HUP, close and reopen your inputs/outputs)
I think we should look at creating new immultilang, ommultilang, and mmmultilang
modules for rsyslog that use either google protocol buffers or apache thrift to
do the message passing (ideally both would be available since there are people
who prefer both)
What I'm currently thinking of is that we should have the ability to include
configuration for these external modules in the rsyslog config (and the modules
can do whatever they want for external configs), when the module is initialized,
rsyslog passes the config data to the module and doesn't consider it good until
it gets a reply saying that it's ready. This could also include handshaking to
coordinate capabilities (does this module have the ability to process batches of
messages, if so, how large?, can it do async message processing where messages
may be processed out of order?, other things) I'm not saying that rsyslog needs
to support such advanced features immediately, and some may never be supported,
but there needs to be a way to find out what is and isn't supported by a module.
Yes, this will be more work initially, and in the meantime we can go ahead and
use omprog (I'm not sure we need improg since programs can just write to
/dev/log or logger), but I think the long-term results will be much better.
thoughts?
David Lang
On Wed, 22 Jan 2014, Boylan, James wrote:
Hmm. That's a tougher question. On the one hand I can see the benefit to it.
But on the other hand, is it the best choice for location... One could argue
that processing scripts like that should be in their own repo since they are
not directly linked to the Rsyslog codebase.
-- James
-----Original Message-----
From: rsyslog-boun...@lists.adiscon.com
[mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of Rainer Gerhards
Sent: Wednesday, January 22, 2014 9:31 AM
To: rsyslog-users
Subject: Re: [rsyslog] non-C output plugins
sorry for being so imprecise...
What I meant to ask is if we should place such non-C plugins into the main
rsyslog repo. I see lot's of good in this (assuming the contributor is happy
with that, of course), but that means that changes can only go into them via
PR, patches...
Rainer
On Wed, Jan 22, 2014 at 4:07 PM, Boylan, James <james.boy...@orbitz.com>wrote:
It seems to me that if it's either added as a modification to omprog
or a new ompipe then having it added to master as a default disabled
option should work. Less maintenance overhead and is available for use.
-- James
-- Sent from my mobile --
----- Reply message -----
From: "Rainer Gerhards" <rgerha...@hq.adiscon.com>
To: "rsyslog-users" <rsyslog@lists.adiscon.com>
Subject: [rsyslog] non-C output plugins
Date: Wed, Jan 22, 2014 9:04 AM
On Wed, Jan 22, 2014 at 4:02 PM, Rainer Gerhards
<rgerha...@hq.adiscon.com>wrote:
On Wed, Jan 22, 2014 at 3:59 PM, Boylan, James
<james.boy...@orbitz.com
wrote:
I agree with everyone that omprog should be a good solution for this.
I'm
going to be doing some tests in the near future with omprog so I'll
be looking at Radu's post as well. :)
Mostly I think using a stdout pipe is a good way of handling this.
It is similar to how Hadoop handles non-Java mapreduce jobs as well.
A key thing to note is that a pipe is a quite efficient
inter-process communication facility and also includes some built-in
flow control (so
the
reading end of the pipe can push back the writer). Of course, it
requires additional context switches, but that should not be too bad.
An interesting thought is to write a simple write-to-file script and
measure it's performance vs. omfile. That should be an indication of
the actual pipe overhead, especially if /dev/null is written to...
Better even: a native C program, so that we don't measure language
runtime overhead...
Question in this regard: should all of this go into the main repo?
There are pros and cons...
Rainer
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is
a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our
control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
