On Thu, 23 Jan 2014, Radu Gheorghe wrote:
Hi David,
I had the same thoughts about writing to /dev/log instead of having a new
improg module. Two comments here, though:
- omprog does a nice job of babysitting the program (starting it,
restarting it if it crashes). With /dev/log, you have to manage the daemon
this is fine for a small input program, but not for a large one, we should
probably support both.
1. immultilangprog that manages the app like omprog does.
2. immultilang that listens for connections.
- logger is typically truncating messages at 1K
short term this isn't a huge problem (it's also not that hard to write a
replacement for logger without this limit, in any language)
Whatever we do for "inputs in other languages" I think we need to be able
to simply get the received message as it is (ie: don't require syslog
format) and parse it later, or add timestamp, or...
And (don't throw sticks or onions!) support multiline messages.
long term I agree.
and with a protocol buffer/thrift interface, everything you are talking about
should be trivial.
/dev/log and logger are just the short-term work-around so that we don't waste
effort writing two sets of rsyslog input modules. If the input is going to be
an ascii stream, they can do this today with /dev/log or nc to localhost 514. If
we are going to do structured/binary data, let's do this right and not reinvent
the wheel multiple times.
David Lang
2014/1/22 David Lang <da...@lang.hm>
I think we should have at least a stub module in each language in the main
repository. All the stubs could do the same 'write to file', but they would
give people a place to start
Actually, I think we need to do three modules in every language
read from file
write to file
message modification that gets the message and passes it back.
while we could just use pipes and stdin/stdout, I've been thinking about
this a bit more over the last week or so, and think we should really go
ahead and be more formal about this
parsing messages is hard, formatting messages to be parsed is error prone,
and both tend to be slow, and when working with text messages, tend to
accumulate a lot of cruft to work around corner cases.
Also, as Rainer said above, we need to have the ability to get feedback
from these other languages, and I'm thinking we probably need to be able to
send command-type messages (we just got a HUP, close and reopen your
inputs/outputs)
I think we should look at creating new immultilang, ommultilang, and
mmmultilang modules for rsyslog that use either google protocol buffers or
apache thrift to do the message passing (ideally both would be available
since there are people who prefer both)
What I'm currently thinking of is that we should have the ability to
include configuration for these external modules in the rsyslog config (and
the modules can do whatever they want for external configs), when the
module is initialized, rsyslog passes the config data to the module and
doesn't consider it good until it gets a reply saying that it's ready. This
could also include handshaking to coordinate capabilities (does this module
have the ability to process batches of messages, if so, how large?, can it
do async message processing where messages may be processed out of order?,
other things) I'm not saying that rsyslog needs to support such advanced
features immediately, and some may never be supported, but there needs to
be a way to find out what is and isn't supported by a module.
Yes, this will be more work initially, and in the meantime we can go ahead
and use omprog (I'm not sure we need improg since programs can just write
to /dev/log or logger), but I think the long-term results will be much
better.
thoughts?
David Lang
On Wed, 22 Jan 2014, Boylan, James wrote:
Hmm. That's a tougher question. On the one hand I can see the benefit to
it. But on the other hand, is it the best choice for location... One could
argue that processing scripts like that should be in their own repo since
they are not directly linked to the Rsyslog codebase.
-- James
-----Original Message-----
From: rsyslog-boun...@lists.adiscon.com [mailto:rsyslog-bounces@lists.
adiscon.com] On Behalf Of Rainer Gerhards
Sent: Wednesday, January 22, 2014 9:31 AM
To: rsyslog-users
Subject: Re: [rsyslog] non-C output plugins
sorry for being so imprecise...
What I meant to ask is if we should place such non-C plugins into the
main rsyslog repo. I see lot's of good in this (assuming the contributor is
happy with that, of course), but that means that changes can only go into
them via PR, patches...
Rainer
On Wed, Jan 22, 2014 at 4:07 PM, Boylan, James <james.boy...@orbitz.com>
wrote:
It seems to me that if it's either added as a modification to omprog
or a new ompipe then having it added to master as a default disabled
option should work. Less maintenance overhead and is available for use.
-- James
-- Sent from my mobile --
----- Reply message -----
From: "Rainer Gerhards" <rgerha...@hq.adiscon.com>
To: "rsyslog-users" <rsyslog@lists.adiscon.com>
Subject: [rsyslog] non-C output plugins
Date: Wed, Jan 22, 2014 9:04 AM
On Wed, Jan 22, 2014 at 4:02 PM, Rainer Gerhards
<rgerha...@hq.adiscon.com>wrote:
On Wed, Jan 22, 2014 at 3:59 PM, Boylan, James
<james.boy...@orbitz.com
wrote:
I agree with everyone that omprog should be a good solution for this.
I'm
going to be doing some tests in the near future with omprog so I'll
be looking at Radu's post as well. :)
Mostly I think using a stdout pipe is a good way of handling this.
It is similar to how Hadoop handles non-Java mapreduce jobs as well.
A key thing to note is that a pipe is a quite efficient
inter-process communication facility and also includes some built-in
flow control (so
the
reading end of the pipe can push back the writer). Of course, it
requires additional context switches, but that should not be too bad.
An interesting thought is to write a simple write-to-file script and
measure it's performance vs. omfile. That should be an indication of
the actual pipe overhead, especially if /dev/null is written to...
Better even: a native C program, so that we don't measure language
runtime overhead...
Question in this regard: should all of this go into the main repo?
There are pros and cons...
Rainer
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL:
This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites
beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
