Hi David, I had the same thoughts about writing to /dev/log instead of having a new improg module. Two comments here, though: - omprog does a nice job of babysitting the program (starting it, restarting it if it crashes). With /dev/log, you have to manage the daemon - logger is typically truncating messages at 1K
Whatever we do for "inputs in other languages" I think we need to be able to simply get the received message as it is (ie: don't require syslog format) and parse it later, or add timestamp, or... And (don't throw sticks or onions!) support multiline messages. 2014/1/22 David Lang <da...@lang.hm> > I think we should have at least a stub module in each language in the main > repository. All the stubs could do the same 'write to file', but they would > give people a place to start > > Actually, I think we need to do three modules in every language > > read from file > write to file > message modification that gets the message and passes it back. > > while we could just use pipes and stdin/stdout, I've been thinking about > this a bit more over the last week or so, and think we should really go > ahead and be more formal about this > > parsing messages is hard, formatting messages to be parsed is error prone, > and both tend to be slow, and when working with text messages, tend to > accumulate a lot of cruft to work around corner cases. > > Also, as Rainer said above, we need to have the ability to get feedback > from these other languages, and I'm thinking we probably need to be able to > send command-type messages (we just got a HUP, close and reopen your > inputs/outputs) > > I think we should look at creating new immultilang, ommultilang, and > mmmultilang modules for rsyslog that use either google protocol buffers or > apache thrift to do the message passing (ideally both would be available > since there are people who prefer both) > > What I'm currently thinking of is that we should have the ability to > include configuration for these external modules in the rsyslog config (and > the modules can do whatever they want for external configs), when the > module is initialized, rsyslog passes the config data to the module and > doesn't consider it good until it gets a reply saying that it's ready. This > could also include handshaking to coordinate capabilities (does this module > have the ability to process batches of messages, if so, how large?, can it > do async message processing where messages may be processed out of order?, > other things) I'm not saying that rsyslog needs to support such advanced > features immediately, and some may never be supported, but there needs to > be a way to find out what is and isn't supported by a module. > > Yes, this will be more work initially, and in the meantime we can go ahead > and use omprog (I'm not sure we need improg since programs can just write > to /dev/log or logger), but I think the long-term results will be much > better. > > thoughts? > > David Lang > > > > On Wed, 22 Jan 2014, Boylan, James wrote: > > Hmm. That's a tougher question. On the one hand I can see the benefit to >> it. But on the other hand, is it the best choice for location... One could >> argue that processing scripts like that should be in their own repo since >> they are not directly linked to the Rsyslog codebase. >> >> -- James >> >> >> -----Original Message----- >> From: rsyslog-boun...@lists.adiscon.com [mailto:rsyslog-bounces@lists. >> adiscon.com] On Behalf Of Rainer Gerhards >> Sent: Wednesday, January 22, 2014 9:31 AM >> To: rsyslog-users >> Subject: Re: [rsyslog] non-C output plugins >> >> sorry for being so imprecise... >> >> What I meant to ask is if we should place such non-C plugins into the >> main rsyslog repo. I see lot's of good in this (assuming the contributor is >> happy with that, of course), but that means that changes can only go into >> them via PR, patches... >> >> Rainer >> >> >> On Wed, Jan 22, 2014 at 4:07 PM, Boylan, James <james.boy...@orbitz.com> >> wrote: >> >> It seems to me that if it's either added as a modification to omprog >>> or a new ompipe then having it added to master as a default disabled >>> option should work. Less maintenance overhead and is available for use. >>> >>> -- James >>> -- Sent from my mobile -- >>> >>> ----- Reply message ----- >>> From: "Rainer Gerhards" <rgerha...@hq.adiscon.com> >>> To: "rsyslog-users" <rsyslog@lists.adiscon.com> >>> Subject: [rsyslog] non-C output plugins >>> Date: Wed, Jan 22, 2014 9:04 AM >>> >>> On Wed, Jan 22, 2014 at 4:02 PM, Rainer Gerhards >>> <rgerha...@hq.adiscon.com>wrote: >>> >>> On Wed, Jan 22, 2014 at 3:59 PM, Boylan, James >>>> <james.boy...@orbitz.com >>>> wrote: >>>> >>>> I agree with everyone that omprog should be a good solution for this. >>>>> >>>> I'm >>> >>>> going to be doing some tests in the near future with omprog so I'll >>>>> be looking at Radu's post as well. :) >>>>> >>>>> Mostly I think using a stdout pipe is a good way of handling this. >>>>> It is similar to how Hadoop handles non-Java mapreduce jobs as well. >>>>> >>>>> >>>>> A key thing to note is that a pipe is a quite efficient >>>> inter-process communication facility and also includes some built-in >>>> flow control (so >>>> >>> the >>> >>>> reading end of the pipe can push back the writer). Of course, it >>>> requires additional context switches, but that should not be too bad. >>>> >>>> An interesting thought is to write a simple write-to-file script and >>>> measure it's performance vs. omfile. That should be an indication of >>>> the actual pipe overhead, especially if /dev/null is written to... >>>> >>>> >>>> Better even: a native C program, so that we don't measure language >>> runtime overhead... >>> >>> Question in this regard: should all of this go into the main repo? >>> There are pros and cons... >>> >>> Rainer >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE >>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of >>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE >>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of >>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> >>> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: >> This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites >> beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE >> THAT. >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> >> _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
