I was thinking more in terms of rsyslog making outgoing REST requests than accepting incoming.
However - we wrote one version of our zeromq out module for rsyslog that used a zeromq ROUTER socket that allowed zeromq DEALER sockets to make requests for batches of messages. It's functionality I'm thinking about adding back in when we move the zeromq modules to the new v8 interface because it worked very well for us. -----Original Message----- From: rsyslog-boun...@lists.adiscon.com [mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of Boylan, James Sent: Monday, December 16, 2013 6:47 AM To: rsyslog-users Subject: Re: [rsyslog] Modules in other programming languages? Omelasticsearch uses libcurl for pushing to elasticsearch, not for accepting pull requests of any sort. Not sure it would really help anyone thinking about 'omrest' not to mention as Rainer said, the entire Rsyslog application isn't designed from a pull model on the output module end. It would take significant core work to implement something reliable with good performance. -- James ----- Reply message ----- From: "Brian Knox" <bri...@talksum.com> To: "rsyslog-users" <rsyslog@lists.adiscon.com> Subject: [rsyslog] Modules in other programming languages? Date: Mon, Dec 16, 2013 5:39 AM I believe the output module for elastic search might be a good place to start looking for anyone interested in writing an "omrest" module? If I recall correctly the elastic search output uses libcurl. Brian -----Original Message----- From: rsyslog-boun...@lists.adiscon.com [mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of Rainer Gerhards Sent: Sunday, December 15, 2013 4:59 AM To: rsyslog-users Subject: Re: [rsyslog] Modules in other programming languages? On Sun, Dec 15, 2013 at 9:36 AM, Radu Gheorghe <radu0gheor...@gmail.com>wrote: > Just my 2 cents here: > a lot earlier I came with a proposal go give a REST API that would > basically enable external applications to get messages from a rsyslog > queue: > http://bugzilla.adiscon.com/show_bug.cgi?id=482 > > With omrest, one should be able to use any programming language to > pull messages from rsyslog. For example, one could write a Kafka > publisher (in any language) that would pull messages from rsyslog and publish > to Kafka. > > yeah, but it is *considerably* more work to be done. Don't say I don't like, but 24hrs is really biting here. It's *a lot* of work (maybe a month of intense work or more). No short term solution. I assume this is better than omprog because AFAIK with omprog piping to the > STDIN of a binary there's a tiny OS buffer (a pipe or something? this > is iffy territory for me) that may get full and you may lose messages > if the other app isn't fast enough. That, or you need to implement > queues in your external program. Which is duplicate work, queues are already > in rsyslog. > With omrest (hypothetically), if you need more performance, you just > need to spawn more threads/processes to pull from the queue and push wherever. > Assuming you have the hardware. > Nope, that's wrong. If the pipe gets full, the OS blocks the writer (rsyslog). That's it. So it's a very simple any easy to use interface. Of course, it currently lacks features, like to ability to convey error and suspensions states back to rsyslog. But that's something that could be fixed with reasonable time. > > On the input side, one can already write connectors in any language. > Just make the thing push to any input rsyslog supports. For most > use-cases, rsyslog should pull from that input fast enough to avoid any > issues. > > Now the only problem with omrest is that it needs to be implemented :) > Which bumps into the 24h problem of people [who can actually do it]. > > hehe, ok, you saw that ;) Again, it's far from trivial. I think I alread wrote that, but you need to change the rsyslog core, as it does not yet provide queues that natively support pull mode. Thinking about that, it's probably more a quite big refactoring in the 2 to 4 month effort ballpark (about 15 to 20% of rsyslog core code need to be changed if I am not totally wrong now and if done decently). At least it's the same effort as the v8 changes done in the past weeks -- they were somewhat simpler to do. Rainer > Best regards, > Radu > > 2013/12/15 Otis Gospodnetic <otis.gospodne...@gmail.com> > > > Hi, > > > > Thanks for the info. > > I was asking because having the ability to write ims and oms in > > different languages would open a lot of opportunities. This is one > > of those "enablement" things. I understand writing modules in other > > languages may mean those using such modules may hurt performance, > > but some people need certain functionality more than performance. > > > > Take omkafka example from the other day. If there were a way to > > write an om in Java it's be trivial for a lot of Java developers out > > there to contribute omkafka. > > > > If omprog enables development of the ecosystem, it sounds like > > something > to > > point out clearly somewhere and nurture that a bit. I do see > > http://www.rsyslog.com/doc/omprog.html because somebody shared a > > link, > but > > I don't see that on http://www.rsyslog.com/doc or on > > http://www.rsyslog.com/doc/dev_oplugins.html or in the new README. > > > > Coincidentally, I just came across Fluentd's instructions for > > writing plugins, which could serve as guidance: > > http://docs.fluentd.org/articles/plugin-development . Nice, clean, > > well structured, not a lot of prose... > > > > Otis > > -- > > Performance Monitoring * Log Analytics * Search Analytics Solr & > > Elasticsearch Support * http://sematext.com/ > > > > > > On Sat, Dec 14, 2013 at 1:39 PM, David Lang <da...@lang.hm> wrote: > > > > > On Sat, 14 Dec 2013, RB wrote: > > > > > > On Sat, Dec 14, 2013 at 5:24 AM, Rainer Gerhards > > >> <rgerha...@hq.adiscon.com> wrote: > > >> > > >>> well, technically it's for sure possible, it's just another of > > >>> these > > 24h > > >>> things. Technically, it's a question of interface, and insofar > > >>> of > which > > >>> types of modules. Obviously, these will be slower, and how slow > > >>> is another interface/effort question. > > >>> > > >>> Thinking about this, one could probably also claim the answer is > "yes, > > >>> you > > >>> can write OUTPUT modules in any language", it's just a doc > > >>> issue. In fact, omprog can be used as an interface here. It's > > >>> actually not even a bad interface... > > >>> > > >>> Again, something learned ;) > > >>> > > >> > > >> Probably the cheapest (implementation) "binding" for rsyslog > > >> would be a system() like call. Execute the subprogram with > > >> /bin/sh -c and communicate with structured messages on STDIO. > > >> > > > > > > a real module binding would be far more complex. It would allow > > > the > > module > > > (in whatever language) access to the rsyslog queues and other data > > > structures. This is possible, but not easy by any means. > > > > > > One big problem is that currently rsyslog does all this work in a > > threaded > > > environment. It may make sense in v9 or v10 to shift from a > > > default shared-everything threading model to a explicit shared > > > memory > > multiprocess > > > model. At that point having one of the processes use a different > language > > > would not be that hard. > > > > > > But in the current threaded model, having one thread run a > > > different language would be very, very hard. > > > > > > The other issue here is performance. Rsyslog goes to a LOT of > > > effort to > > be > > > fast. Some of the things that have made very noticable diffences > > > in performance in rsyslog are things that seem like they should be > > > very > > minor. > > > Think about these things and then think about what would be > > > involved to define interfaces in a multi-language safe way. > > > > > > things that have resulted in noticable speedups have been: > > > > > > removing gettimeofday() calls. > > > > > > it used to be that rsyslog recorded when a message arrived, when > > > it > was > > > put on the main queue, when it was moved to an action queue, when > > > it > was > > > pulled from the action queue, and when it was delivered > > > > > > now, high performance users configure rsyslog so that it only > > > does > one > > > gettimeofday() call per hundred (ot thousand) messages that arrive > > > and > > use > > > that one time for every message > > > > > > string modules > > > > > > it used to be that the default template (<%pri%>%timestamp% > %hostname% > > > %syslogtag%%msg%) was interpreted by the rsyslog engine for every > message > > > that was output > > > > > > now string modules written in C create these strings rather than > > > interpreting the template. This resulted in a double-digit % > performance > > > improvement > > > > > > With optimizations like these in use, changing things to allow for > > > a module written in a different language to have access to the > > > rsyslog internals as would be needed for a high-performance > > > interface seems > like > > it > > > will probably end up hurting the rsyslog performance overall. > > > > > > > > > That being said, I am very much in favor of multi-process with > > > explicit sharing rather than multi-threaded with implicit sharing, > > > but getting > all > > > the interfaces correct and fast would be a VERY hard task. > > > > > > David Lang > > > > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com/professional-services/ > > > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE > > > WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > > > you DON'T LIKE THAT. > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE > > WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > > you DON'T LIKE THAT. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE > WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
