Re: [rsyslog] Modules in other programming languages?
I was thinking that omrest would be a module that made outbound http requests to send messages; not as something that waited for incoming http requests for messages. So, in my mind it was something far more similar to the current elastic search module, which pushes messages out. So - more of an outgoing http request to an external rest api, rather than something providing a rest api to make calls into. -Original Message- From: rsyslog-boun...@lists.adiscon.com [mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of David Lang Sent: Monday, December 16, 2013 1:57 PM To: rsyslog-users Subject: Re: [rsyslog] Modules in other programming languages? On Mon, 16 Dec 2013, Brian Knox wrote: I believe the output module for elastic search might be a good place to start looking for anyone interested in writing an omrest module? If I recall correctly the elastic search output uses libcurl. not really. currently every om* module consists of code that is executed by a rsyslog worker thread that is passed a list of messages and acts on each message. omelasticsearch pushes messages via libcurl for omrest you would need to change the entire paradigm of how an om* module would work. Instead of being code invoked by a worker thread that's invoking many other om* code as well on a given message, the omrelp module would need to listen for a connection from the outside, and when it receives a request, it would need to retrieve messages from the queue, and the worker threads would need to leave the messages on the queue. You should see by now that this is a really ugly thing to talk about implementing. It's almost a complete re-write of the rsyslog core to support this. The other approach is to have omrelp maintain it's own queue of messages and knowledge of who should be asking for messages, timeouts for messages that aren't asked for, etc. At that point, omrelp's interface to rsyslog is straightforward, it's just the omrelp queue and interface stuff that gets really 'interesting' and people who want omrelp should work on writing some code that will perform all the relp functions that you want to support but just accepts new messages on stdin (which would let it be driven by omprog for now), and then as you get it running and find it useful the input portion could be changed to make it into a 'real' rsyslog om* module. Depending on what you do for your queue, you may be able to use a different language to handle adding things to the queue and pulling things from the queue. This would make it easy to have some rsyslog C code that adds things to the queueu. David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Modules in other programming languages?
That would probably be better called 'omhttp' as 'omrest' paints the picture of a REST interface for accessing into Rsyslog, not outputting http post calls to a destination. -- James -Original Message- From: rsyslog-boun...@lists.adiscon.com [mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of Brian Knox Sent: Tuesday, December 17, 2013 4:59 AM To: rsyslog-users Subject: Re: [rsyslog] Modules in other programming languages? I was thinking that omrest would be a module that made outbound http requests to send messages; not as something that waited for incoming http requests for messages. So, in my mind it was something far more similar to the current elastic search module, which pushes messages out. So - more of an outgoing http request to an external rest api, rather than something providing a rest api to make calls into. -Original Message- From: rsyslog-boun...@lists.adiscon.com [mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of David Lang Sent: Monday, December 16, 2013 1:57 PM To: rsyslog-users Subject: Re: [rsyslog] Modules in other programming languages? On Mon, 16 Dec 2013, Brian Knox wrote: I believe the output module for elastic search might be a good place to start looking for anyone interested in writing an omrest module? If I recall correctly the elastic search output uses libcurl. not really. currently every om* module consists of code that is executed by a rsyslog worker thread that is passed a list of messages and acts on each message. omelasticsearch pushes messages via libcurl for omrest you would need to change the entire paradigm of how an om* module would work. Instead of being code invoked by a worker thread that's invoking many other om* code as well on a given message, the omrelp module would need to listen for a connection from the outside, and when it receives a request, it would need to retrieve messages from the queue, and the worker threads would need to leave the messages on the queue. You should see by now that this is a really ugly thing to talk about implementing. It's almost a complete re-write of the rsyslog core to support this. The other approach is to have omrelp maintain it's own queue of messages and knowledge of who should be asking for messages, timeouts for messages that aren't asked for, etc. At that point, omrelp's interface to rsyslog is straightforward, it's just the omrelp queue and interface stuff that gets really 'interesting' and people who want omrelp should work on writing some code that will perform all the relp functions that you want to support but just accepts new messages on stdin (which would let it be driven by omprog for now), and then as you get it running and find it useful the input portion could be changed to make it into a 'real' rsyslog om* module. Depending on what you do for your queue, you may be able to use a different language to handle adding things to the queue and pulling things from the queue. This would make it easy to have some rsyslog C code that adds things to the queueu. David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Modules in other programming languages?
I agree that omhttp would be a better name. Note - I'm not signing up for that one quite yet - my first priority is going to be moving the omzmq3 plugins to the rsyslog v8 plugin api, and moving them from zeromq3 to zeromq4. Brian -Original Message- From: rsyslog-boun...@lists.adiscon.com [mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of Boylan, James Sent: Tuesday, December 17, 2013 7:09 AM To: rsyslog-users Subject: Re: [rsyslog] Modules in other programming languages? That would probably be better called 'omhttp' as 'omrest' paints the picture of a REST interface for accessing into Rsyslog, not outputting http post calls to a destination. -- James -Original Message- From: rsyslog-boun...@lists.adiscon.com [mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of Brian Knox Sent: Tuesday, December 17, 2013 4:59 AM To: rsyslog-users Subject: Re: [rsyslog] Modules in other programming languages? I was thinking that omrest would be a module that made outbound http requests to send messages; not as something that waited for incoming http requests for messages. So, in my mind it was something far more similar to the current elastic search module, which pushes messages out. So - more of an outgoing http request to an external rest api, rather than something providing a rest api to make calls into. -Original Message- From: rsyslog-boun...@lists.adiscon.com [mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of David Lang Sent: Monday, December 16, 2013 1:57 PM To: rsyslog-users Subject: Re: [rsyslog] Modules in other programming languages? On Mon, 16 Dec 2013, Brian Knox wrote: I believe the output module for elastic search might be a good place to start looking for anyone interested in writing an omrest module? If I recall correctly the elastic search output uses libcurl. not really. currently every om* module consists of code that is executed by a rsyslog worker thread that is passed a list of messages and acts on each message. omelasticsearch pushes messages via libcurl for omrest you would need to change the entire paradigm of how an om* module would work. Instead of being code invoked by a worker thread that's invoking many other om* code as well on a given message, the omrelp module would need to listen for a connection from the outside, and when it receives a request, it would need to retrieve messages from the queue, and the worker threads would need to leave the messages on the queue. You should see by now that this is a really ugly thing to talk about implementing. It's almost a complete re-write of the rsyslog core to support this. The other approach is to have omrelp maintain it's own queue of messages and knowledge of who should be asking for messages, timeouts for messages that aren't asked for, etc. At that point, omrelp's interface to rsyslog is straightforward, it's just the omrelp queue and interface stuff that gets really 'interesting' and people who want omrelp should work on writing some code that will perform all the relp functions that you want to support but just accepts new messages on stdin (which would let it be driven by omprog for now), and then as you get it running and find it useful the input portion could be changed to make it into a 'real' rsyslog om* module. Depending on what you do for your queue, you may be able to use a different language to handle adding things to the queue and pulling things from the queue. This would make it easy to have some rsyslog C code that adds things to the queueu. David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL
Re: [rsyslog] Modules in other programming languages?
The prior suggestions for omrest were specifically for a module that created a pull mechanism for getting logs rather than having the logs pushed to the destination at the speed that they arrive. One of the justifications was avoiding overloading log destinations as they would be in control of the rate that they got data. David Lang On Tue, 17 Dec 2013, Brian Knox wrote: I was thinking that omrest would be a module that made outbound http requests to send messages; not as something that waited for incoming http requests for messages. So, in my mind it was something far more similar to the current elastic search module, which pushes messages out. So - more of an outgoing http request to an external rest api, rather than something providing a rest api to make calls into. -Original Message- From: rsyslog-boun...@lists.adiscon.com [mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of David Lang Sent: Monday, December 16, 2013 1:57 PM To: rsyslog-users Subject: Re: [rsyslog] Modules in other programming languages? On Mon, 16 Dec 2013, Brian Knox wrote: I believe the output module for elastic search might be a good place to start looking for anyone interested in writing an omrest module? If I recall correctly the elastic search output uses libcurl. not really. currently every om* module consists of code that is executed by a rsyslog worker thread that is passed a list of messages and acts on each message. omelasticsearch pushes messages via libcurl for omrest you would need to change the entire paradigm of how an om* module would work. Instead of being code invoked by a worker thread that's invoking many other om* code as well on a given message, the omrelp module would need to listen for a connection from the outside, and when it receives a request, it would need to retrieve messages from the queue, and the worker threads would need to leave the messages on the queue. You should see by now that this is a really ugly thing to talk about implementing. It's almost a complete re-write of the rsyslog core to support this. The other approach is to have omrelp maintain it's own queue of messages and knowledge of who should be asking for messages, timeouts for messages that aren't asked for, etc. At that point, omrelp's interface to rsyslog is straightforward, it's just the omrelp queue and interface stuff that gets really 'interesting' and people who want omrelp should work on writing some code that will perform all the relp functions that you want to support but just accepts new messages on stdin (which would let it be driven by omprog for now), and then as you get it running and find it useful the input portion could be changed to make it into a 'real' rsyslog om* module. Depending on what you do for your queue, you may be able to use a different language to handle adding things to the queue and pulling things from the queue. This would make it easy to have some rsyslog C code that adds things to the queueu. David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Modules in other programming languages?
FYI: I have something cooking in the back of my mind that may enable pull plugins on top of the existing infrastructure. It's too weak an idea right now to elaborate (especially as not much time is left before I leave), but I wanted to share this thought. I'll let it ripen over vacation. If it could work out, it would be great if someone would come up and write something that could utilize that interface. Non-C would be no problem. Rainer On Tue, Dec 17, 2013 at 6:20 PM, David Lang da...@lang.hm wrote: The prior suggestions for omrest were specifically for a module that created a pull mechanism for getting logs rather than having the logs pushed to the destination at the speed that they arrive. One of the justifications was avoiding overloading log destinations as they would be in control of the rate that they got data. David Lang On Tue, 17 Dec 2013, Brian Knox wrote: I was thinking that omrest would be a module that made outbound http requests to send messages; not as something that waited for incoming http requests for messages. So, in my mind it was something far more similar to the current elastic search module, which pushes messages out. So - more of an outgoing http request to an external rest api, rather than something providing a rest api to make calls into. -Original Message- From: rsyslog-boun...@lists.adiscon.com [mailto:rsyslog-bounces@lists. adiscon.com] On Behalf Of David Lang Sent: Monday, December 16, 2013 1:57 PM To: rsyslog-users Subject: Re: [rsyslog] Modules in other programming languages? On Mon, 16 Dec 2013, Brian Knox wrote: I believe the output module for elastic search might be a good place to start looking for anyone interested in writing an omrest module? If I recall correctly the elastic search output uses libcurl. not really. currently every om* module consists of code that is executed by a rsyslog worker thread that is passed a list of messages and acts on each message. omelasticsearch pushes messages via libcurl for omrest you would need to change the entire paradigm of how an om* module would work. Instead of being code invoked by a worker thread that's invoking many other om* code as well on a given message, the omrelp module would need to listen for a connection from the outside, and when it receives a request, it would need to retrieve messages from the queue, and the worker threads would need to leave the messages on the queue. You should see by now that this is a really ugly thing to talk about implementing. It's almost a complete re-write of the rsyslog core to support this. The other approach is to have omrelp maintain it's own queue of messages and knowledge of who should be asking for messages, timeouts for messages that aren't asked for, etc. At that point, omrelp's interface to rsyslog is straightforward, it's just the omrelp queue and interface stuff that gets really 'interesting' and people who want omrelp should work on writing some code that will perform all the relp functions that you want to support but just accepts new messages on stdin (which would let it be driven by omprog for now), and then as you get it running and find it useful the input portion could be changed to make it into a 'real' rsyslog om* module. Depending on what you do for your queue, you may be able to use a different language to handle adding things to the queue and pulling things from the queue. This would make it easy to have some rsyslog C code that adds things to the queueu. David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL
Re: [rsyslog] Modules in other programming languages?
17.12.2013 21:20, David Lang: The prior suggestions for omrest were specifically for a module that created a pull mechanism for getting logs rather than having the logs pushed to the destination at the speed that they arrive. One of the justifications was avoiding overloading log destinations as they would be in control of the rate that they got data. Rate control is equally possible with push-based model, say, with TCP, RELP or omprog. One real advantage of pull model is that it does not need to do retries when receiving side is busy or non-operational. Hence, it decreases delays significantly. It is even more valuable given out current discussion of problems with RELP and suspension. But if these problems were mitigated, pull mode could be less of a value. Pull mode requires a model different from current worker threads with output modules; it is more like as input modules, which run own threads (while this is not strictly obligatory). -- Pavel Levshin David Lang On Tue, 17 Dec 2013, Brian Knox wrote: I was thinking that omrest would be a module that made outbound http requests to send messages; not as something that waited for incoming http requests for messages. So, in my mind it was something far more similar to the current elastic search module, which pushes messages out. So - more of an outgoing http request to an external rest api, rather than something providing a rest api to make calls into. -Original Message- From: rsyslog-boun...@lists.adiscon.com [mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of David Lang Sent: Monday, December 16, 2013 1:57 PM To: rsyslog-users Subject: Re: [rsyslog] Modules in other programming languages? On Mon, 16 Dec 2013, Brian Knox wrote: I believe the output module for elastic search might be a good place to start looking for anyone interested in writing an omrest module? If I recall correctly the elastic search output uses libcurl. not really. currently every om* module consists of code that is executed by a rsyslog worker thread that is passed a list of messages and acts on each message. omelasticsearch pushes messages via libcurl for omrest you would need to change the entire paradigm of how an om* module would work. Instead of being code invoked by a worker thread that's invoking many other om* code as well on a given message, the omrelp module would need to listen for a connection from the outside, and when it receives a request, it would need to retrieve messages from the queue, and the worker threads would need to leave the messages on the queue. You should see by now that this is a really ugly thing to talk about implementing. It's almost a complete re-write of the rsyslog core to support this. The other approach is to have omrelp maintain it's own queue of messages and knowledge of who should be asking for messages, timeouts for messages that aren't asked for, etc. At that point, omrelp's interface to rsyslog is straightforward, it's just the omrelp queue and interface stuff that gets really 'interesting' and people who want omrelp should work on writing some code that will perform all the relp functions that you want to support but just accepts new messages on stdin (which would let it be driven by omprog for now), and then as you get it running and find it useful the input portion could be changed to make it into a 'real' rsyslog om* module. Depending on what you do for your queue, you may be able to use a different language to handle adding things to the queue and pulling things from the queue. This would make it easy to have some rsyslog C code that adds things to the queueu. David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http
Re: [rsyslog] Modules in other programming languages?
I believe the output module for elastic search might be a good place to start looking for anyone interested in writing an omrest module? If I recall correctly the elastic search output uses libcurl. Brian -Original Message- From: rsyslog-boun...@lists.adiscon.com [mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of Rainer Gerhards Sent: Sunday, December 15, 2013 4:59 AM To: rsyslog-users Subject: Re: [rsyslog] Modules in other programming languages? On Sun, Dec 15, 2013 at 9:36 AM, Radu Gheorghe radu0gheor...@gmail.comwrote: Just my 2 cents here: a lot earlier I came with a proposal go give a REST API that would basically enable external applications to get messages from a rsyslog queue: http://bugzilla.adiscon.com/show_bug.cgi?id=482 With omrest, one should be able to use any programming language to pull messages from rsyslog. For example, one could write a Kafka publisher (in any language) that would pull messages from rsyslog and publish to Kafka. yeah, but it is *considerably* more work to be done. Don't say I don't like, but 24hrs is really biting here. It's *a lot* of work (maybe a month of intense work or more). No short term solution. I assume this is better than omprog because AFAIK with omprog piping to the STDIN of a binary there's a tiny OS buffer (a pipe or something? this is iffy territory for me) that may get full and you may lose messages if the other app isn't fast enough. That, or you need to implement queues in your external program. Which is duplicate work, queues are already in rsyslog. With omrest (hypothetically), if you need more performance, you just need to spawn more threads/processes to pull from the queue and push wherever. Assuming you have the hardware. Nope, that's wrong. If the pipe gets full, the OS blocks the writer (rsyslog). That's it. So it's a very simple any easy to use interface. Of course, it currently lacks features, like to ability to convey error and suspensions states back to rsyslog. But that's something that could be fixed with reasonable time. On the input side, one can already write connectors in any language. Just make the thing push to any input rsyslog supports. For most use-cases, rsyslog should pull from that input fast enough to avoid any issues. Now the only problem with omrest is that it needs to be implemented :) Which bumps into the 24h problem of people [who can actually do it]. hehe, ok, you saw that ;) Again, it's far from trivial. I think I alread wrote that, but you need to change the rsyslog core, as it does not yet provide queues that natively support pull mode. Thinking about that, it's probably more a quite big refactoring in the 2 to 4 month effort ballpark (about 15 to 20% of rsyslog core code need to be changed if I am not totally wrong now and if done decently). At least it's the same effort as the v8 changes done in the past weeks -- they were somewhat simpler to do. Rainer Best regards, Radu 2013/12/15 Otis Gospodnetic otis.gospodne...@gmail.com Hi, Thanks for the info. I was asking because having the ability to write ims and oms in different languages would open a lot of opportunities. This is one of those enablement things. I understand writing modules in other languages may mean those using such modules may hurt performance, but some people need certain functionality more than performance. Take omkafka example from the other day. If there were a way to write an om in Java it's be trivial for a lot of Java developers out there to contribute omkafka. If omprog enables development of the ecosystem, it sounds like something to point out clearly somewhere and nurture that a bit. I do see http://www.rsyslog.com/doc/omprog.html because somebody shared a link, but I don't see that on http://www.rsyslog.com/doc or on http://www.rsyslog.com/doc/dev_oplugins.html or in the new README. Coincidentally, I just came across Fluentd's instructions for writing plugins, which could serve as guidance: http://docs.fluentd.org/articles/plugin-development . Nice, clean, well structured, not a lot of prose... Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr Elasticsearch Support * http://sematext.com/ On Sat, Dec 14, 2013 at 1:39 PM, David Lang da...@lang.hm wrote: On Sat, 14 Dec 2013, RB wrote: On Sat, Dec 14, 2013 at 5:24 AM, Rainer Gerhards rgerha...@hq.adiscon.com wrote: well, technically it's for sure possible, it's just another of these 24h things. Technically, it's a question of interface, and insofar of which types of modules. Obviously, these will be slower, and how slow is another interface/effort question. Thinking about this, one could probably also claim the answer is yes, you can write OUTPUT modules in any language, it's just a doc issue. In fact, omprog can be used as an interface here
Re: [rsyslog] Modules in other programming languages?
Omelasticsearch uses libcurl for pushing to elasticsearch, not for accepting pull requests of any sort. Not sure it would really help anyone thinking about 'omrest' not to mention as Rainer said, the entire Rsyslog application isn't designed from a pull model on the output module end. It would take significant core work to implement something reliable with good performance. -- James - Reply message - From: Brian Knox bri...@talksum.com To: rsyslog-users rsyslog@lists.adiscon.com Subject: [rsyslog] Modules in other programming languages? Date: Mon, Dec 16, 2013 5:39 AM I believe the output module for elastic search might be a good place to start looking for anyone interested in writing an omrest module? If I recall correctly the elastic search output uses libcurl. Brian -Original Message- From: rsyslog-boun...@lists.adiscon.com [mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of Rainer Gerhards Sent: Sunday, December 15, 2013 4:59 AM To: rsyslog-users Subject: Re: [rsyslog] Modules in other programming languages? On Sun, Dec 15, 2013 at 9:36 AM, Radu Gheorghe radu0gheor...@gmail.comwrote: Just my 2 cents here: a lot earlier I came with a proposal go give a REST API that would basically enable external applications to get messages from a rsyslog queue: http://bugzilla.adiscon.com/show_bug.cgi?id=482 With omrest, one should be able to use any programming language to pull messages from rsyslog. For example, one could write a Kafka publisher (in any language) that would pull messages from rsyslog and publish to Kafka. yeah, but it is *considerably* more work to be done. Don't say I don't like, but 24hrs is really biting here. It's *a lot* of work (maybe a month of intense work or more). No short term solution. I assume this is better than omprog because AFAIK with omprog piping to the STDIN of a binary there's a tiny OS buffer (a pipe or something? this is iffy territory for me) that may get full and you may lose messages if the other app isn't fast enough. That, or you need to implement queues in your external program. Which is duplicate work, queues are already in rsyslog. With omrest (hypothetically), if you need more performance, you just need to spawn more threads/processes to pull from the queue and push wherever. Assuming you have the hardware. Nope, that's wrong. If the pipe gets full, the OS blocks the writer (rsyslog). That's it. So it's a very simple any easy to use interface. Of course, it currently lacks features, like to ability to convey error and suspensions states back to rsyslog. But that's something that could be fixed with reasonable time. On the input side, one can already write connectors in any language. Just make the thing push to any input rsyslog supports. For most use-cases, rsyslog should pull from that input fast enough to avoid any issues. Now the only problem with omrest is that it needs to be implemented :) Which bumps into the 24h problem of people [who can actually do it]. hehe, ok, you saw that ;) Again, it's far from trivial. I think I alread wrote that, but you need to change the rsyslog core, as it does not yet provide queues that natively support pull mode. Thinking about that, it's probably more a quite big refactoring in the 2 to 4 month effort ballpark (about 15 to 20% of rsyslog core code need to be changed if I am not totally wrong now and if done decently). At least it's the same effort as the v8 changes done in the past weeks -- they were somewhat simpler to do. Rainer Best regards, Radu 2013/12/15 Otis Gospodnetic otis.gospodne...@gmail.com Hi, Thanks for the info. I was asking because having the ability to write ims and oms in different languages would open a lot of opportunities. This is one of those enablement things. I understand writing modules in other languages may mean those using such modules may hurt performance, but some people need certain functionality more than performance. Take omkafka example from the other day. If there were a way to write an om in Java it's be trivial for a lot of Java developers out there to contribute omkafka. If omprog enables development of the ecosystem, it sounds like something to point out clearly somewhere and nurture that a bit. I do see http://www.rsyslog.com/doc/omprog.html because somebody shared a link, but I don't see that on http://www.rsyslog.com/doc or on http://www.rsyslog.com/doc/dev_oplugins.html or in the new README. Coincidentally, I just came across Fluentd's instructions for writing plugins, which could serve as guidance: http://docs.fluentd.org/articles/plugin-development . Nice, clean, well structured, not a lot of prose... Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr Elasticsearch Support * http://sematext.com/ On Sat, Dec 14, 2013 at 1:39 PM, David Lang da...@lang.hm wrote: On Sat, 14 Dec 2013, RB wrote
Re: [rsyslog] Modules in other programming languages?
Yeah, I'd abandon this quest for now. Unless someone really needs it and is willing to put in the effort. Until then, omprog FTW! 2013/12/16 Boylan, James james.boy...@orbitz.com Omelasticsearch uses libcurl for pushing to elasticsearch, not for accepting pull requests of any sort. Not sure it would really help anyone thinking about 'omrest' not to mention as Rainer said, the entire Rsyslog application isn't designed from a pull model on the output module end. It would take significant core work to implement something reliable with good performance. -- James - Reply message - From: Brian Knox bri...@talksum.com To: rsyslog-users rsyslog@lists.adiscon.com Subject: [rsyslog] Modules in other programming languages? Date: Mon, Dec 16, 2013 5:39 AM I believe the output module for elastic search might be a good place to start looking for anyone interested in writing an omrest module? If I recall correctly the elastic search output uses libcurl. Brian -Original Message- From: rsyslog-boun...@lists.adiscon.com [mailto: rsyslog-boun...@lists.adiscon.com] On Behalf Of Rainer Gerhards Sent: Sunday, December 15, 2013 4:59 AM To: rsyslog-users Subject: Re: [rsyslog] Modules in other programming languages? On Sun, Dec 15, 2013 at 9:36 AM, Radu Gheorghe radu0gheor...@gmail.com wrote: Just my 2 cents here: a lot earlier I came with a proposal go give a REST API that would basically enable external applications to get messages from a rsyslog queue: http://bugzilla.adiscon.com/show_bug.cgi?id=482 With omrest, one should be able to use any programming language to pull messages from rsyslog. For example, one could write a Kafka publisher (in any language) that would pull messages from rsyslog and publish to Kafka. yeah, but it is *considerably* more work to be done. Don't say I don't like, but 24hrs is really biting here. It's *a lot* of work (maybe a month of intense work or more). No short term solution. I assume this is better than omprog because AFAIK with omprog piping to the STDIN of a binary there's a tiny OS buffer (a pipe or something? this is iffy territory for me) that may get full and you may lose messages if the other app isn't fast enough. That, or you need to implement queues in your external program. Which is duplicate work, queues are already in rsyslog. With omrest (hypothetically), if you need more performance, you just need to spawn more threads/processes to pull from the queue and push wherever. Assuming you have the hardware. Nope, that's wrong. If the pipe gets full, the OS blocks the writer (rsyslog). That's it. So it's a very simple any easy to use interface. Of course, it currently lacks features, like to ability to convey error and suspensions states back to rsyslog. But that's something that could be fixed with reasonable time. On the input side, one can already write connectors in any language. Just make the thing push to any input rsyslog supports. For most use-cases, rsyslog should pull from that input fast enough to avoid any issues. Now the only problem with omrest is that it needs to be implemented :) Which bumps into the 24h problem of people [who can actually do it]. hehe, ok, you saw that ;) Again, it's far from trivial. I think I alread wrote that, but you need to change the rsyslog core, as it does not yet provide queues that natively support pull mode. Thinking about that, it's probably more a quite big refactoring in the 2 to 4 month effort ballpark (about 15 to 20% of rsyslog core code need to be changed if I am not totally wrong now and if done decently). At least it's the same effort as the v8 changes done in the past weeks -- they were somewhat simpler to do. Rainer Best regards, Radu 2013/12/15 Otis Gospodnetic otis.gospodne...@gmail.com Hi, Thanks for the info. I was asking because having the ability to write ims and oms in different languages would open a lot of opportunities. This is one of those enablement things. I understand writing modules in other languages may mean those using such modules may hurt performance, but some people need certain functionality more than performance. Take omkafka example from the other day. If there were a way to write an om in Java it's be trivial for a lot of Java developers out there to contribute omkafka. If omprog enables development of the ecosystem, it sounds like something to point out clearly somewhere and nurture that a bit. I do see http://www.rsyslog.com/doc/omprog.html because somebody shared a link, but I don't see that on http://www.rsyslog.com/doc or on http://www.rsyslog.com/doc/dev_oplugins.html or in the new README. Coincidentally, I just came across Fluentd's instructions for writing plugins, which could serve as guidance: http://docs.fluentd.org/articles/plugin-development . Nice, clean, well
Re: [rsyslog] Modules in other programming languages?
I was thinking more in terms of rsyslog making outgoing REST requests than accepting incoming. However - we wrote one version of our zeromq out module for rsyslog that used a zeromq ROUTER socket that allowed zeromq DEALER sockets to make requests for batches of messages. It's functionality I'm thinking about adding back in when we move the zeromq modules to the new v8 interface because it worked very well for us. -Original Message- From: rsyslog-boun...@lists.adiscon.com [mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of Boylan, James Sent: Monday, December 16, 2013 6:47 AM To: rsyslog-users Subject: Re: [rsyslog] Modules in other programming languages? Omelasticsearch uses libcurl for pushing to elasticsearch, not for accepting pull requests of any sort. Not sure it would really help anyone thinking about 'omrest' not to mention as Rainer said, the entire Rsyslog application isn't designed from a pull model on the output module end. It would take significant core work to implement something reliable with good performance. -- James - Reply message - From: Brian Knox bri...@talksum.com To: rsyslog-users rsyslog@lists.adiscon.com Subject: [rsyslog] Modules in other programming languages? Date: Mon, Dec 16, 2013 5:39 AM I believe the output module for elastic search might be a good place to start looking for anyone interested in writing an omrest module? If I recall correctly the elastic search output uses libcurl. Brian -Original Message- From: rsyslog-boun...@lists.adiscon.com [mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of Rainer Gerhards Sent: Sunday, December 15, 2013 4:59 AM To: rsyslog-users Subject: Re: [rsyslog] Modules in other programming languages? On Sun, Dec 15, 2013 at 9:36 AM, Radu Gheorghe radu0gheor...@gmail.comwrote: Just my 2 cents here: a lot earlier I came with a proposal go give a REST API that would basically enable external applications to get messages from a rsyslog queue: http://bugzilla.adiscon.com/show_bug.cgi?id=482 With omrest, one should be able to use any programming language to pull messages from rsyslog. For example, one could write a Kafka publisher (in any language) that would pull messages from rsyslog and publish to Kafka. yeah, but it is *considerably* more work to be done. Don't say I don't like, but 24hrs is really biting here. It's *a lot* of work (maybe a month of intense work or more). No short term solution. I assume this is better than omprog because AFAIK with omprog piping to the STDIN of a binary there's a tiny OS buffer (a pipe or something? this is iffy territory for me) that may get full and you may lose messages if the other app isn't fast enough. That, or you need to implement queues in your external program. Which is duplicate work, queues are already in rsyslog. With omrest (hypothetically), if you need more performance, you just need to spawn more threads/processes to pull from the queue and push wherever. Assuming you have the hardware. Nope, that's wrong. If the pipe gets full, the OS blocks the writer (rsyslog). That's it. So it's a very simple any easy to use interface. Of course, it currently lacks features, like to ability to convey error and suspensions states back to rsyslog. But that's something that could be fixed with reasonable time. On the input side, one can already write connectors in any language. Just make the thing push to any input rsyslog supports. For most use-cases, rsyslog should pull from that input fast enough to avoid any issues. Now the only problem with omrest is that it needs to be implemented :) Which bumps into the 24h problem of people [who can actually do it]. hehe, ok, you saw that ;) Again, it's far from trivial. I think I alread wrote that, but you need to change the rsyslog core, as it does not yet provide queues that natively support pull mode. Thinking about that, it's probably more a quite big refactoring in the 2 to 4 month effort ballpark (about 15 to 20% of rsyslog core code need to be changed if I am not totally wrong now and if done decently). At least it's the same effort as the v8 changes done in the past weeks -- they were somewhat simpler to do. Rainer Best regards, Radu 2013/12/15 Otis Gospodnetic otis.gospodne...@gmail.com Hi, Thanks for the info. I was asking because having the ability to write ims and oms in different languages would open a lot of opportunities. This is one of those enablement things. I understand writing modules in other languages may mean those using such modules may hurt performance, but some people need certain functionality more than performance. Take omkafka example from the other day. If there were a way to write an om in Java it's be trivial for a lot of Java developers out there to contribute omkafka. If omprog enables development of the ecosystem, it sounds like something to point out clearly
Re: [rsyslog] Modules in other programming languages?
On Mon, 16 Dec 2013, Radu Gheorghe wrote: Some quick comments: - with the hypothetical omrest, rsyslog doesn't have to keep the message until everyone gets it. We can make it so we assume there's one consumer only, and once the message is acknowledged, it's gone. If you need to fan out, you can have multiple actions (with their own action queues) what if you have more than one thing accessing the rest interface? how should rsyslog know how many clients are going to try and pull? - but it sounds like omrest isn't a good idea (for now), because omprog already does most of what omrest would do (except it's push vs pull) the big difference is that the client would have to do the queuing instead of leaving it to rsyslog. Dealing with failure conditions of the client is something that can get interesting. - about the particular case of omkafka, we have a publish-subscribe architecture. Kind of like *MQ, if I understand correctly. So you can't make Kafka pull, you have to push. So we either have omkafka, or we use omprog with a script that pushes to Kafka, or we have an independent service that receives syslog and can push to Kafka. AFAIK the likes of Logstash, Flume and Fluentd can already play that role, so we don't need to reinvent the wheel: https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem omkafka is probably a much better thing to have than an omprog script for performance reasons (an om* interface can insert messages in batches, which is usually MUCH faster) having to setup a JVM and separate app like Flume and others is a whole lot of work and seriously complicates your system. David Lang 2013/12/16 David Lang da...@lang.hm On Sun, 15 Dec 2013, Otis Gospodnetic wrote: Hi, On Sun, Dec 15, 2013 at 3:36 AM, Radu Gheorghe radu0gheor...@gmail.com wrote: Just my 2 cents here: a lot earlier I came with a proposal go give a REST API that would basically enable external applications to get messages from a rsyslog queue: http://bugzilla.adiscon.com/show_bug.cgi?id=482 With omrest, one should be able to use any programming language to pull messages from rsyslog. For example, one could write a Kafka publisher (in any language) that would pull messages from rsyslog and publish to Kafka. I assume this is better than omprog because AFAIK with omprog piping to the STDIN of a binary there's a tiny OS buffer (a pipe or something? this is iffy territory for me) that may get full and you may lose messages if the other app isn't fast enough. That, or you need to implement queues in your external program. Which is duplicate work, queues are already in rsyslog. With omrest (hypothetically), if you need more performance, you just need to spawn more threads/processes to pull from the queue and push wherever. Assuming you have the hardware. I like the omrest idea because I like the ability for systems to pull at their own speed and to upgrade more freely. the problem is that rsyslog is stuck holding on to the messages until all possible clients have pulled the message. This is not a really good idea. If the above issue were in Github I'd click the Watch button immediately to get notified of any comments or activity around it but I'd have to create yet another account in somebody's Bugzilla, so I won't. (just trying to illustrate the thinking that I bet many people go through in similar situations). Re omprog, does rsyslog launch the referenced binary for each message or just launches the process once and keeps feeding it via stdin? once and then feeds multiple messages (and relaunches it if it dies) I'm no Linux/C programmer, so maybe this makes no sense, but would using sendfile be appropriate here? not really. sendfile is appropriate when you have a file that you are sending as-is with no modification to it. David Lang Thanks, Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr Elasticsearch Support * http://sematext.com/ On the input side, one can already write connectors in any language. Just make the thing push to any input rsyslog supports. For most use-cases, rsyslog should pull from that input fast enough to avoid any issues. Now the only problem with omrest is that it needs to be implemented :) Which bumps into the 24h problem of people [who can actually do it]. Best regards, Radu 2013/12/15 Otis Gospodnetic otis.gospodne...@gmail.com Hi, Thanks for the info. I was asking because having the ability to write ims and oms in different languages would open a lot of opportunities. This is one of those enablement things. I understand writing modules in other languages may mean those using such modules may hurt performance, but some people need certain functionality more than performance. Take omkafka example from the other day. If there were a way to write an om in Java it's be trivial for a lot of Java developers out there to contribute omkafka. If omprog enables development of the ecosystem, it sounds like
Re: [rsyslog] Modules in other programming languages?
On Mon, 16 Dec 2013, Brian Knox wrote: I believe the output module for elastic search might be a good place to start looking for anyone interested in writing an omrest module? If I recall correctly the elastic search output uses libcurl. not really. currently every om* module consists of code that is executed by a rsyslog worker thread that is passed a list of messages and acts on each message. omelasticsearch pushes messages via libcurl for omrest you would need to change the entire paradigm of how an om* module would work. Instead of being code invoked by a worker thread that's invoking many other om* code as well on a given message, the omrelp module would need to listen for a connection from the outside, and when it receives a request, it would need to retrieve messages from the queue, and the worker threads would need to leave the messages on the queue. You should see by now that this is a really ugly thing to talk about implementing. It's almost a complete re-write of the rsyslog core to support this. The other approach is to have omrelp maintain it's own queue of messages and knowledge of who should be asking for messages, timeouts for messages that aren't asked for, etc. At that point, omrelp's interface to rsyslog is straightforward, it's just the omrelp queue and interface stuff that gets really 'interesting' and people who want omrelp should work on writing some code that will perform all the relp functions that you want to support but just accepts new messages on stdin (which would let it be driven by omprog for now), and then as you get it running and find it useful the input portion could be changed to make it into a 'real' rsyslog om* module. Depending on what you do for your queue, you may be able to use a different language to handle adding things to the queue and pulling things from the queue. This would make it easy to have some rsyslog C code that adds things to the queueu. David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Modules in other programming languages?
Just my 2 cents here: a lot earlier I came with a proposal go give a REST API that would basically enable external applications to get messages from a rsyslog queue: http://bugzilla.adiscon.com/show_bug.cgi?id=482 With omrest, one should be able to use any programming language to pull messages from rsyslog. For example, one could write a Kafka publisher (in any language) that would pull messages from rsyslog and publish to Kafka. I assume this is better than omprog because AFAIK with omprog piping to the STDIN of a binary there's a tiny OS buffer (a pipe or something? this is iffy territory for me) that may get full and you may lose messages if the other app isn't fast enough. That, or you need to implement queues in your external program. Which is duplicate work, queues are already in rsyslog. With omrest (hypothetically), if you need more performance, you just need to spawn more threads/processes to pull from the queue and push wherever. Assuming you have the hardware. On the input side, one can already write connectors in any language. Just make the thing push to any input rsyslog supports. For most use-cases, rsyslog should pull from that input fast enough to avoid any issues. Now the only problem with omrest is that it needs to be implemented :) Which bumps into the 24h problem of people [who can actually do it]. Best regards, Radu 2013/12/15 Otis Gospodnetic otis.gospodne...@gmail.com Hi, Thanks for the info. I was asking because having the ability to write ims and oms in different languages would open a lot of opportunities. This is one of those enablement things. I understand writing modules in other languages may mean those using such modules may hurt performance, but some people need certain functionality more than performance. Take omkafka example from the other day. If there were a way to write an om in Java it's be trivial for a lot of Java developers out there to contribute omkafka. If omprog enables development of the ecosystem, it sounds like something to point out clearly somewhere and nurture that a bit. I do see http://www.rsyslog.com/doc/omprog.html because somebody shared a link, but I don't see that on http://www.rsyslog.com/doc or on http://www.rsyslog.com/doc/dev_oplugins.html or in the new README. Coincidentally, I just came across Fluentd's instructions for writing plugins, which could serve as guidance: http://docs.fluentd.org/articles/plugin-development . Nice, clean, well structured, not a lot of prose... Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr Elasticsearch Support * http://sematext.com/ On Sat, Dec 14, 2013 at 1:39 PM, David Lang da...@lang.hm wrote: On Sat, 14 Dec 2013, RB wrote: On Sat, Dec 14, 2013 at 5:24 AM, Rainer Gerhards rgerha...@hq.adiscon.com wrote: well, technically it's for sure possible, it's just another of these 24h things. Technically, it's a question of interface, and insofar of which types of modules. Obviously, these will be slower, and how slow is another interface/effort question. Thinking about this, one could probably also claim the answer is yes, you can write OUTPUT modules in any language, it's just a doc issue. In fact, omprog can be used as an interface here. It's actually not even a bad interface... Again, something learned ;) Probably the cheapest (implementation) binding for rsyslog would be a system() like call. Execute the subprogram with /bin/sh -c and communicate with structured messages on STDIO. a real module binding would be far more complex. It would allow the module (in whatever language) access to the rsyslog queues and other data structures. This is possible, but not easy by any means. One big problem is that currently rsyslog does all this work in a threaded environment. It may make sense in v9 or v10 to shift from a default shared-everything threading model to a explicit shared memory multiprocess model. At that point having one of the processes use a different language would not be that hard. But in the current threaded model, having one thread run a different language would be very, very hard. The other issue here is performance. Rsyslog goes to a LOT of effort to be fast. Some of the things that have made very noticable diffences in performance in rsyslog are things that seem like they should be very minor. Think about these things and then think about what would be involved to define interfaces in a multi-language safe way. things that have resulted in noticable speedups have been: removing gettimeofday() calls. it used to be that rsyslog recorded when a message arrived, when it was put on the main queue, when it was moved to an action queue, when it was pulled from the action queue, and when it was delivered now, high performance users configure rsyslog so that it only does one gettimeofday() call per hundred (ot thousand) messages
Re: [rsyslog] Modules in other programming languages?
On Sun, Dec 15, 2013 at 9:36 AM, Radu Gheorghe radu0gheor...@gmail.comwrote: Just my 2 cents here: a lot earlier I came with a proposal go give a REST API that would basically enable external applications to get messages from a rsyslog queue: http://bugzilla.adiscon.com/show_bug.cgi?id=482 With omrest, one should be able to use any programming language to pull messages from rsyslog. For example, one could write a Kafka publisher (in any language) that would pull messages from rsyslog and publish to Kafka. yeah, but it is *considerably* more work to be done. Don't say I don't like, but 24hrs is really biting here. It's *a lot* of work (maybe a month of intense work or more). No short term solution. I assume this is better than omprog because AFAIK with omprog piping to the STDIN of a binary there's a tiny OS buffer (a pipe or something? this is iffy territory for me) that may get full and you may lose messages if the other app isn't fast enough. That, or you need to implement queues in your external program. Which is duplicate work, queues are already in rsyslog. With omrest (hypothetically), if you need more performance, you just need to spawn more threads/processes to pull from the queue and push wherever. Assuming you have the hardware. Nope, that's wrong. If the pipe gets full, the OS blocks the writer (rsyslog). That's it. So it's a very simple any easy to use interface. Of course, it currently lacks features, like to ability to convey error and suspensions states back to rsyslog. But that's something that could be fixed with reasonable time. On the input side, one can already write connectors in any language. Just make the thing push to any input rsyslog supports. For most use-cases, rsyslog should pull from that input fast enough to avoid any issues. Now the only problem with omrest is that it needs to be implemented :) Which bumps into the 24h problem of people [who can actually do it]. hehe, ok, you saw that ;) Again, it's far from trivial. I think I alread wrote that, but you need to change the rsyslog core, as it does not yet provide queues that natively support pull mode. Thinking about that, it's probably more a quite big refactoring in the 2 to 4 month effort ballpark (about 15 to 20% of rsyslog core code need to be changed if I am not totally wrong now and if done decently). At least it's the same effort as the v8 changes done in the past weeks -- they were somewhat simpler to do. Rainer Best regards, Radu 2013/12/15 Otis Gospodnetic otis.gospodne...@gmail.com Hi, Thanks for the info. I was asking because having the ability to write ims and oms in different languages would open a lot of opportunities. This is one of those enablement things. I understand writing modules in other languages may mean those using such modules may hurt performance, but some people need certain functionality more than performance. Take omkafka example from the other day. If there were a way to write an om in Java it's be trivial for a lot of Java developers out there to contribute omkafka. If omprog enables development of the ecosystem, it sounds like something to point out clearly somewhere and nurture that a bit. I do see http://www.rsyslog.com/doc/omprog.html because somebody shared a link, but I don't see that on http://www.rsyslog.com/doc or on http://www.rsyslog.com/doc/dev_oplugins.html or in the new README. Coincidentally, I just came across Fluentd's instructions for writing plugins, which could serve as guidance: http://docs.fluentd.org/articles/plugin-development . Nice, clean, well structured, not a lot of prose... Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr Elasticsearch Support * http://sematext.com/ On Sat, Dec 14, 2013 at 1:39 PM, David Lang da...@lang.hm wrote: On Sat, 14 Dec 2013, RB wrote: On Sat, Dec 14, 2013 at 5:24 AM, Rainer Gerhards rgerha...@hq.adiscon.com wrote: well, technically it's for sure possible, it's just another of these 24h things. Technically, it's a question of interface, and insofar of which types of modules. Obviously, these will be slower, and how slow is another interface/effort question. Thinking about this, one could probably also claim the answer is yes, you can write OUTPUT modules in any language, it's just a doc issue. In fact, omprog can be used as an interface here. It's actually not even a bad interface... Again, something learned ;) Probably the cheapest (implementation) binding for rsyslog would be a system() like call. Execute the subprogram with /bin/sh -c and communicate with structured messages on STDIO. a real module binding would be far more complex. It would allow the module (in whatever language) access to the rsyslog queues and other data structures. This is possible, but not easy by any means. One big problem is that
Re: [rsyslog] Modules in other programming languages?
On Sun, Dec 15, 2013 at 7:42 AM, Otis Gospodnetic otis.gospodne...@gmail.com wrote: Hi, Thanks for the info. I was asking because having the ability to write ims and oms in different languages would open a lot of opportunities. This is one of those enablement things. I understand writing modules in other languages may mean those using such modules may hurt performance, but some people need certain functionality more than performance. yeah, I agree. And if it becomes popular, that would be an indication that investing to write a native C implementation makes sense. So it would actually be good to do PoC for less-requested things. Take omkafka example from the other day. If there were a way to write an om in Java it's be trivial for a lot of Java developers out there to contribute omkafka. If it's trivial, is there somebody who would like to do it? That would also enable us to see limits into which the current omprog implementation runs -- and get those fixed. As I said, everything is already in place... If omprog enables development of the ecosystem, it sounds like something to point out clearly somewhere and nurture that a bit. I do see http://www.rsyslog.com/doc/omprog.html because somebody shared a link, but I don't see that on http://www.rsyslog.com/doc or on http://www.rsyslog.com/doc/dev_oplugins.html or in the new README. You are right. I frankly admit that it never before occured to me to think of this as an interface to other languages. I am pretty focussed on performance, and everything that's not fast enough I don't (didn't) consider as a solution. Again, there are many more talents required inside a project than software engineering, and rsyslog is missing lots of other talents. I am glad for everyone who brings in some of the missing ones! Coincidentally, I just came across Fluentd's instructions for writing plugins, which could serve as guidance: http://docs.fluentd.org/articles/plugin-development . Nice, clean, well structured, not a lot of prose... Will look at it. But again, I am not the greatest writer. While I can write small code, not a lot of prose poses a problem to me in doc. I always think of the remote border cases that could happen... That's one of the reasons why I *really* think that the doc project is a separate entity with separate folks working on it... Thanks again for all the help, Rainer Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr Elasticsearch Support * http://sematext.com/ On Sat, Dec 14, 2013 at 1:39 PM, David Lang da...@lang.hm wrote: On Sat, 14 Dec 2013, RB wrote: On Sat, Dec 14, 2013 at 5:24 AM, Rainer Gerhards rgerha...@hq.adiscon.com wrote: well, technically it's for sure possible, it's just another of these 24h things. Technically, it's a question of interface, and insofar of which types of modules. Obviously, these will be slower, and how slow is another interface/effort question. Thinking about this, one could probably also claim the answer is yes, you can write OUTPUT modules in any language, it's just a doc issue. In fact, omprog can be used as an interface here. It's actually not even a bad interface... Again, something learned ;) Probably the cheapest (implementation) binding for rsyslog would be a system() like call. Execute the subprogram with /bin/sh -c and communicate with structured messages on STDIO. a real module binding would be far more complex. It would allow the module (in whatever language) access to the rsyslog queues and other data structures. This is possible, but not easy by any means. One big problem is that currently rsyslog does all this work in a threaded environment. It may make sense in v9 or v10 to shift from a default shared-everything threading model to a explicit shared memory multiprocess model. At that point having one of the processes use a different language would not be that hard. But in the current threaded model, having one thread run a different language would be very, very hard. The other issue here is performance. Rsyslog goes to a LOT of effort to be fast. Some of the things that have made very noticable diffences in performance in rsyslog are things that seem like they should be very minor. Think about these things and then think about what would be involved to define interfaces in a multi-language safe way. things that have resulted in noticable speedups have been: removing gettimeofday() calls. it used to be that rsyslog recorded when a message arrived, when it was put on the main queue, when it was moved to an action queue, when it was pulled from the action queue, and when it was delivered now, high performance users configure rsyslog so that it only does one gettimeofday() call per hundred (ot thousand) messages that arrive and use that one time for every message string modules it used to be that the
Re: [rsyslog] Modules in other programming languages?
Hi Rainer, Thanks for your comments. I didn't know about the OS blocking part. That's essential, I think. For failed connects from the external program to the destination, there can be workarounds in the program itself (wait until connection can be established again, re-queue the message in rsyslog by sending it back to its input... maybe other options) So, while the omrest idea sounds pretty elegant and all, it's much more work than it's worth, it seems. I think omprog can be a really good interface, if you don't need absolute performance. I'm thinking it might be useful to make official connectors for other languages. For example, some module (thinking about Python now, but the idea should work everywhere) that would read a batch of JSONs from the command line and put them in some small internal queue. Then, there could be an interface that will allow programs using this module to: a) change those messages and do whatever, like change the date b) pull a batch and push it somewhere Such modules won't even be rsyslog-specific. We'd only have to document how to use templates in rsyslog to send such JSONs via omprog. Anyway, this is just an idea I wanted to share. I'll let it bake some more. And maybe others can comment and improve the idea, or replace it with a better one. 2013/12/15 Rainer Gerhards rgerha...@hq.adiscon.com On Sun, Dec 15, 2013 at 9:36 AM, Radu Gheorghe radu0gheor...@gmail.com wrote: Just my 2 cents here: a lot earlier I came with a proposal go give a REST API that would basically enable external applications to get messages from a rsyslog queue: http://bugzilla.adiscon.com/show_bug.cgi?id=482 With omrest, one should be able to use any programming language to pull messages from rsyslog. For example, one could write a Kafka publisher (in any language) that would pull messages from rsyslog and publish to Kafka. yeah, but it is *considerably* more work to be done. Don't say I don't like, but 24hrs is really biting here. It's *a lot* of work (maybe a month of intense work or more). No short term solution. I assume this is better than omprog because AFAIK with omprog piping to the STDIN of a binary there's a tiny OS buffer (a pipe or something? this is iffy territory for me) that may get full and you may lose messages if the other app isn't fast enough. That, or you need to implement queues in your external program. Which is duplicate work, queues are already in rsyslog. With omrest (hypothetically), if you need more performance, you just need to spawn more threads/processes to pull from the queue and push wherever. Assuming you have the hardware. Nope, that's wrong. If the pipe gets full, the OS blocks the writer (rsyslog). That's it. So it's a very simple any easy to use interface. Of course, it currently lacks features, like to ability to convey error and suspensions states back to rsyslog. But that's something that could be fixed with reasonable time. On the input side, one can already write connectors in any language. Just make the thing push to any input rsyslog supports. For most use-cases, rsyslog should pull from that input fast enough to avoid any issues. Now the only problem with omrest is that it needs to be implemented :) Which bumps into the 24h problem of people [who can actually do it]. hehe, ok, you saw that ;) Again, it's far from trivial. I think I alread wrote that, but you need to change the rsyslog core, as it does not yet provide queues that natively support pull mode. Thinking about that, it's probably more a quite big refactoring in the 2 to 4 month effort ballpark (about 15 to 20% of rsyslog core code need to be changed if I am not totally wrong now and if done decently). At least it's the same effort as the v8 changes done in the past weeks -- they were somewhat simpler to do. Rainer Best regards, Radu 2013/12/15 Otis Gospodnetic otis.gospodne...@gmail.com Hi, Thanks for the info. I was asking because having the ability to write ims and oms in different languages would open a lot of opportunities. This is one of those enablement things. I understand writing modules in other languages may mean those using such modules may hurt performance, but some people need certain functionality more than performance. Take omkafka example from the other day. If there were a way to write an om in Java it's be trivial for a lot of Java developers out there to contribute omkafka. If omprog enables development of the ecosystem, it sounds like something to point out clearly somewhere and nurture that a bit. I do see http://www.rsyslog.com/doc/omprog.html because somebody shared a link, but I don't see that on http://www.rsyslog.com/doc or on http://www.rsyslog.com/doc/dev_oplugins.html or in the new README. Coincidentally, I just came across Fluentd's instructions for writing plugins, which could serve as
Re: [rsyslog] Modules in other programming languages?
15.12.2013 14:36, Radu Gheorghe: I'm thinking it might be useful to make official connectors for other languages. For example, some module (thinking about Python now, but the idea should work everywhere) that would read a batch of JSONs from the command line and put them in some small internal queue. Then, there could be an interface that will allow programs using this module to: a) change those messages and do whatever, like change the date b) pull a batch and push it somewhere All this can be done even with current module interfaces, but there is not enough demand. Personally, I've considered writing such an interface to Python or Perl, but now all I needed now works without it. For example, there is a Python interface module from FreeRADIUS, which essentially has the same model as in rsyslog: messages, queues, modules. https://github.com/FreeRADIUS/freeradius-server/tree/master/src/modules/rlm_python It is fairly simple... But you need to write a module for each language you want to support. -- Pavel Levshin ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Modules in other programming languages?
Hi Pavel, Thanks for the update. That was exactly my point, that it works with what currently is in rsyslog. So whenever there's timeneed, implementing something like that makes sense. I just brought up the idea of a place to put some commons that might be re-used for other output modules. 2013/12/15 Pavel Levshin pa...@levshin.spb.ru 15.12.2013 14:36, Radu Gheorghe: I'm thinking it might be useful to make official connectors for other languages. For example, some module (thinking about Python now, but the idea should work everywhere) that would read a batch of JSONs from the command line and put them in some small internal queue. Then, there could be an interface that will allow programs using this module to: a) change those messages and do whatever, like change the date b) pull a batch and push it somewhere All this can be done even with current module interfaces, but there is not enough demand. Personally, I've considered writing such an interface to Python or Perl, but now all I needed now works without it. For example, there is a Python interface module from FreeRADIUS, which essentially has the same model as in rsyslog: messages, queues, modules. https://github.com/FreeRADIUS/freeradius-server/tree/master/ src/modules/rlm_python It is fairly simple... But you need to write a module for each language you want to support. -- Pavel Levshin ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Modules in other programming languages?
Hi, On Sun, Dec 15, 2013 at 3:36 AM, Radu Gheorghe radu0gheor...@gmail.comwrote: Just my 2 cents here: a lot earlier I came with a proposal go give a REST API that would basically enable external applications to get messages from a rsyslog queue: http://bugzilla.adiscon.com/show_bug.cgi?id=482 With omrest, one should be able to use any programming language to pull messages from rsyslog. For example, one could write a Kafka publisher (in any language) that would pull messages from rsyslog and publish to Kafka. I assume this is better than omprog because AFAIK with omprog piping to the STDIN of a binary there's a tiny OS buffer (a pipe or something? this is iffy territory for me) that may get full and you may lose messages if the other app isn't fast enough. That, or you need to implement queues in your external program. Which is duplicate work, queues are already in rsyslog. With omrest (hypothetically), if you need more performance, you just need to spawn more threads/processes to pull from the queue and push wherever. Assuming you have the hardware. I like the omrest idea because I like the ability for systems to pull at their own speed and to upgrade more freely. If the above issue were in Github I'd click the Watch button immediately to get notified of any comments or activity around it but I'd have to create yet another account in somebody's Bugzilla, so I won't. (just trying to illustrate the thinking that I bet many people go through in similar situations). Re omprog, does rsyslog launch the referenced binary for each message or just launches the process once and keeps feeding it via stdin? I'm no Linux/C programmer, so maybe this makes no sense, but would using sendfile be appropriate here? Thanks, Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr Elasticsearch Support * http://sematext.com/ On the input side, one can already write connectors in any language. Just make the thing push to any input rsyslog supports. For most use-cases, rsyslog should pull from that input fast enough to avoid any issues. Now the only problem with omrest is that it needs to be implemented :) Which bumps into the 24h problem of people [who can actually do it]. Best regards, Radu 2013/12/15 Otis Gospodnetic otis.gospodne...@gmail.com Hi, Thanks for the info. I was asking because having the ability to write ims and oms in different languages would open a lot of opportunities. This is one of those enablement things. I understand writing modules in other languages may mean those using such modules may hurt performance, but some people need certain functionality more than performance. Take omkafka example from the other day. If there were a way to write an om in Java it's be trivial for a lot of Java developers out there to contribute omkafka. If omprog enables development of the ecosystem, it sounds like something to point out clearly somewhere and nurture that a bit. I do see http://www.rsyslog.com/doc/omprog.html because somebody shared a link, but I don't see that on http://www.rsyslog.com/doc or on http://www.rsyslog.com/doc/dev_oplugins.html or in the new README. Coincidentally, I just came across Fluentd's instructions for writing plugins, which could serve as guidance: http://docs.fluentd.org/articles/plugin-development . Nice, clean, well structured, not a lot of prose... Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr Elasticsearch Support * http://sematext.com/ On Sat, Dec 14, 2013 at 1:39 PM, David Lang da...@lang.hm wrote: On Sat, 14 Dec 2013, RB wrote: On Sat, Dec 14, 2013 at 5:24 AM, Rainer Gerhards rgerha...@hq.adiscon.com wrote: well, technically it's for sure possible, it's just another of these 24h things. Technically, it's a question of interface, and insofar of which types of modules. Obviously, these will be slower, and how slow is another interface/effort question. Thinking about this, one could probably also claim the answer is yes, you can write OUTPUT modules in any language, it's just a doc issue. In fact, omprog can be used as an interface here. It's actually not even a bad interface... Again, something learned ;) Probably the cheapest (implementation) binding for rsyslog would be a system() like call. Execute the subprogram with /bin/sh -c and communicate with structured messages on STDIO. a real module binding would be far more complex. It would allow the module (in whatever language) access to the rsyslog queues and other data structures. This is possible, but not easy by any means. One big problem is that currently rsyslog does all this work in a threaded environment. It may make sense in v9 or v10 to shift from a default shared-everything threading model to a explicit shared memory multiprocess model. At that point having
Re: [rsyslog] Modules in other programming languages?
On 12/15/2013 02:36 AM, Radu Gheorghe wrote: Hi Rainer, Thanks for your comments. I didn't know about the OS blocking part. That's essential, I think. For failed connects from the external program to the destination, there can be workarounds in the program itself (wait until connection can be established again, re-queue the message in rsyslog by sending it back to its input... maybe other options) So, while the omrest idea sounds pretty elegant and all, it's much more work than it's worth, it seems. I think omprog can be a really good interface, if you don't need absolute performance. I'm thinking it might be useful to make official connectors for other languages. For example, some module (thinking about Python now, but the idea should work everywhere) that would read a batch of JSONs from the command line and put them in some small internal queue. Then, there could be an interface that will allow programs using this module to: a) change those messages and do whatever, like change the date b) pull a batch and push it somewhere Such modules won't even be rsyslog-specific. We'd only have to document how to use templates in rsyslog to send such JSONs via omprog. Anyway, this is just an idea I wanted to share. I'll let it bake some more. And maybe others can comment and improve the idea, or replace it with a better one. when we're talking at this level (omrest, omprog) it might be easier for other programs to implement syslog or RELP protocols and use omfwd, im/omrelp modules. Seems easier and cleaner... (more efficient that omprog) I mean implementing a rest client that talks to omrest is basically same as implementing syslog or RELP client. Perhaps implementing few sample clients (that speak syslog and RELP protocols) in few languages would help people to interface with rsyslog? Specially RELP (lot of other logging systems already support syslog protocol but I didn't find any that would support RELP) as a specific example related to recent discussions, it might be a better idea to implement syslog/RELP input module for kafka rather than omkafka for rsyslog... thoughts? (not suggesting somebody should go out and write those example client, just thinking... I am not at the stage where I would work on it yet since we're just trying to make rsyslog work reliably with amazon load balancer but we'll likely do something like that to be able to implement some on the fly streaming processing for monitoring and alarms etc. (unless we use something that's already implemented of course)) erik 2013/12/15 Rainer Gerhards rgerha...@hq.adiscon.com On Sun, Dec 15, 2013 at 9:36 AM, Radu Gheorghe radu0gheor...@gmail.com wrote: Just my 2 cents here: a lot earlier I came with a proposal go give a REST API that would basically enable external applications to get messages from a rsyslog queue: http://bugzilla.adiscon.com/show_bug.cgi?id=482 With omrest, one should be able to use any programming language to pull messages from rsyslog. For example, one could write a Kafka publisher (in any language) that would pull messages from rsyslog and publish to Kafka. yeah, but it is *considerably* more work to be done. Don't say I don't like, but 24hrs is really biting here. It's *a lot* of work (maybe a month of intense work or more). No short term solution. I assume this is better than omprog because AFAIK with omprog piping to the STDIN of a binary there's a tiny OS buffer (a pipe or something? this is iffy territory for me) that may get full and you may lose messages if the other app isn't fast enough. That, or you need to implement queues in your external program. Which is duplicate work, queues are already in rsyslog. With omrest (hypothetically), if you need more performance, you just need to spawn more threads/processes to pull from the queue and push wherever. Assuming you have the hardware. Nope, that's wrong. If the pipe gets full, the OS blocks the writer (rsyslog). That's it. So it's a very simple any easy to use interface. Of course, it currently lacks features, like to ability to convey error and suspensions states back to rsyslog. But that's something that could be fixed with reasonable time. On the input side, one can already write connectors in any language. Just make the thing push to any input rsyslog supports. For most use-cases, rsyslog should pull from that input fast enough to avoid any issues. Now the only problem with omrest is that it needs to be implemented :) Which bumps into the 24h problem of people [who can actually do it]. hehe, ok, you saw that ;) Again, it's far from trivial. I think I alread wrote that, but you need to change the rsyslog core, as it does not yet provide queues that natively support pull mode. Thinking about that, it's probably more a quite big refactoring in the 2 to 4 month effort ballpark (about 15 to 20% of rsyslog core code need to be changed if I am not totally wrong now and if done decently). At least
Re: [rsyslog] Modules in other programming languages?
On Sun, 15 Dec 2013, Otis Gospodnetic wrote: Hi, On Sun, Dec 15, 2013 at 3:36 AM, Radu Gheorghe radu0gheor...@gmail.comwrote: Just my 2 cents here: a lot earlier I came with a proposal go give a REST API that would basically enable external applications to get messages from a rsyslog queue: http://bugzilla.adiscon.com/show_bug.cgi?id=482 With omrest, one should be able to use any programming language to pull messages from rsyslog. For example, one could write a Kafka publisher (in any language) that would pull messages from rsyslog and publish to Kafka. I assume this is better than omprog because AFAIK with omprog piping to the STDIN of a binary there's a tiny OS buffer (a pipe or something? this is iffy territory for me) that may get full and you may lose messages if the other app isn't fast enough. That, or you need to implement queues in your external program. Which is duplicate work, queues are already in rsyslog. With omrest (hypothetically), if you need more performance, you just need to spawn more threads/processes to pull from the queue and push wherever. Assuming you have the hardware. I like the omrest idea because I like the ability for systems to pull at their own speed and to upgrade more freely. the problem is that rsyslog is stuck holding on to the messages until all possible clients have pulled the message. This is not a really good idea. If the above issue were in Github I'd click the Watch button immediately to get notified of any comments or activity around it but I'd have to create yet another account in somebody's Bugzilla, so I won't. (just trying to illustrate the thinking that I bet many people go through in similar situations). Re omprog, does rsyslog launch the referenced binary for each message or just launches the process once and keeps feeding it via stdin? once and then feeds multiple messages (and relaunches it if it dies) I'm no Linux/C programmer, so maybe this makes no sense, but would using sendfile be appropriate here? not really. sendfile is appropriate when you have a file that you are sending as-is with no modification to it. David Lang Thanks, Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr Elasticsearch Support * http://sematext.com/ On the input side, one can already write connectors in any language. Just make the thing push to any input rsyslog supports. For most use-cases, rsyslog should pull from that input fast enough to avoid any issues. Now the only problem with omrest is that it needs to be implemented :) Which bumps into the 24h problem of people [who can actually do it]. Best regards, Radu 2013/12/15 Otis Gospodnetic otis.gospodne...@gmail.com Hi, Thanks for the info. I was asking because having the ability to write ims and oms in different languages would open a lot of opportunities. This is one of those enablement things. I understand writing modules in other languages may mean those using such modules may hurt performance, but some people need certain functionality more than performance. Take omkafka example from the other day. If there were a way to write an om in Java it's be trivial for a lot of Java developers out there to contribute omkafka. If omprog enables development of the ecosystem, it sounds like something to point out clearly somewhere and nurture that a bit. I do see http://www.rsyslog.com/doc/omprog.html because somebody shared a link, but I don't see that on http://www.rsyslog.com/doc or on http://www.rsyslog.com/doc/dev_oplugins.html or in the new README. Coincidentally, I just came across Fluentd's instructions for writing plugins, which could serve as guidance: http://docs.fluentd.org/articles/plugin-development . Nice, clean, well structured, not a lot of prose... Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr Elasticsearch Support * http://sematext.com/ On Sat, Dec 14, 2013 at 1:39 PM, David Lang da...@lang.hm wrote: On Sat, 14 Dec 2013, RB wrote: On Sat, Dec 14, 2013 at 5:24 AM, Rainer Gerhards rgerha...@hq.adiscon.com wrote: well, technically it's for sure possible, it's just another of these 24h things. Technically, it's a question of interface, and insofar of which types of modules. Obviously, these will be slower, and how slow is another interface/effort question. Thinking about this, one could probably also claim the answer is yes, you can write OUTPUT modules in any language, it's just a doc issue. In fact, omprog can be used as an interface here. It's actually not even a bad interface... Again, something learned ;) Probably the cheapest (implementation) binding for rsyslog would be a system() like call. Execute the subprogram with /bin/sh -c and communicate with structured messages on STDIO. a real module binding would be far more complex. It would allow the module (in whatever language) access to the rsyslog queues and other data structures. This is possible, but not easy by any means. One big
Re: [rsyslog] Modules in other programming languages?
On Sat, Dec 14, 2013 at 5:40 AM, Otis Gospodnetic otis.gospodne...@gmail.com wrote: Hi, Is there any support in Rsyslog for writing modules in something other than C? Not yet. Someone could probably builds some mappings to other languages, I wouldn't object that (as usual). Rainer Thanks, Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr Elasticsearch Support * http://sematext.com/ ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Modules in other programming languages?
2013/12/14 Rainer Gerhards rgerha...@hq.adiscon.com On Sat, Dec 14, 2013 at 5:40 AM, Otis Gospodnetic otis.gospodne...@gmail.com wrote: Hi, Is there any support in Rsyslog for writing modules in something other than C? Not yet. Someone could probably builds some mappings to other languages, I wouldn't object that (as usual). Some mappings? What do you mean? ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Modules in other programming languages?
Sent from phone, thus brief. Am 14.12.2013 12:38 schrieb Radu Gheorghe radu0gheor...@gmail.com: 2013/12/14 Rainer Gerhards rgerha...@hq.adiscon.com On Sat, Dec 14, 2013 at 5:40 AM, Otis Gospodnetic otis.gospodne...@gmail.com wrote: Hi, Is there any support in Rsyslog for writing modules in something other than C? Not yet. Someone could probably builds some mappings to other languages, I wouldn't object that (as usual). Some mappings? What do you mean? Let me rephrase: whatever it takes to make this work. Obviously, I've no real idea about that ;) Rainer ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Modules in other programming languages?
A, OK. Because when I saw Otis' mail I wondered how it's technically possible. 2013/12/14 Rainer Gerhards rgerha...@hq.adiscon.com Sent from phone, thus brief. Am 14.12.2013 12:38 schrieb Radu Gheorghe radu0gheor...@gmail.com: 2013/12/14 Rainer Gerhards rgerha...@hq.adiscon.com On Sat, Dec 14, 2013 at 5:40 AM, Otis Gospodnetic otis.gospodne...@gmail.com wrote: Hi, Is there any support in Rsyslog for writing modules in something other than C? Not yet. Someone could probably builds some mappings to other languages, I wouldn't object that (as usual). Some mappings? What do you mean? Let me rephrase: whatever it takes to make this work. Obviously, I've no real idea about that ;) Rainer ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Modules in other programming languages?
On Sat, Dec 14, 2013 at 12:51 PM, Radu Gheorghe radu0gheor...@gmail.comwrote: A, OK. Because when I saw Otis' mail I wondered how it's technically possible. well, technically it's for sure possible, it's just another of these 24h things. Technically, it's a question of interface, and insofar of which types of modules. Obviously, these will be slower, and how slow is another interface/effort question. Thinking about this, one could probably also claim the answer is yes, you can write OUTPUT modules in any language, it's just a doc issue. In fact, omprog can be used as an interface here. It's actually not even a bad interface... Again, something learned ;) Rainer 2013/12/14 Rainer Gerhards rgerha...@hq.adiscon.com Sent from phone, thus brief. Am 14.12.2013 12:38 schrieb Radu Gheorghe radu0gheor...@gmail.com: 2013/12/14 Rainer Gerhards rgerha...@hq.adiscon.com On Sat, Dec 14, 2013 at 5:40 AM, Otis Gospodnetic otis.gospodne...@gmail.com wrote: Hi, Is there any support in Rsyslog for writing modules in something other than C? Not yet. Someone could probably builds some mappings to other languages, I wouldn't object that (as usual). Some mappings? What do you mean? Let me rephrase: whatever it takes to make this work. Obviously, I've no real idea about that ;) Rainer ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Modules in other programming languages?
On Sat, Dec 14, 2013 at 5:24 AM, Rainer Gerhards rgerha...@hq.adiscon.com wrote: well, technically it's for sure possible, it's just another of these 24h things. Technically, it's a question of interface, and insofar of which types of modules. Obviously, these will be slower, and how slow is another interface/effort question. Thinking about this, one could probably also claim the answer is yes, you can write OUTPUT modules in any language, it's just a doc issue. In fact, omprog can be used as an interface here. It's actually not even a bad interface... Again, something learned ;) Probably the cheapest (implementation) binding for rsyslog would be a system() like call. Execute the subprogram with /bin/sh -c and communicate with structured messages on STDIO. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Modules in other programming languages?
On Sat, Dec 14, 2013 at 2:51 PM, RB aoz@gmail.com wrote: On Sat, Dec 14, 2013 at 5:24 AM, Rainer Gerhards rgerha...@hq.adiscon.com wrote: well, technically it's for sure possible, it's just another of these 24h things. Technically, it's a question of interface, and insofar of which types of modules. Obviously, these will be slower, and how slow is another interface/effort question. Thinking about this, one could probably also claim the answer is yes, you can write OUTPUT modules in any language, it's just a doc issue. In fact, omprog can be used as an interface here. It's actually not even a bad interface... Again, something learned ;) Probably the cheapest (implementation) binding for rsyslog would be a system() like call. Execute the subprogram with /bin/sh -c and communicate with structured messages on STDIO. that's exactly what omprog does: http://www.rsyslog.com/doc/omprog.html It's, however, a seldom-used output and thus it may require some tweaking if it gets seriously used. Note that it offers full template support, so the output script can accept messages in almost all formats (inclusing JSON). Rainer ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Modules in other programming languages?
On Sat, 14 Dec 2013, Rainer Gerhards wrote: On Sat, Dec 14, 2013 at 5:40 AM, Otis Gospodnetic otis.gospodne...@gmail.com wrote: Hi, Is there any support in Rsyslog for writing modules in something other than C? Not yet. Someone could probably builds some mappings to other languages, I wouldn't object that (as usual). well, the module interface would need to finish settling down a bit from the current V8 changes first. :-) David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Modules in other programming languages?
On Sat, 14 Dec 2013, RB wrote: On Sat, Dec 14, 2013 at 5:24 AM, Rainer Gerhards rgerha...@hq.adiscon.com wrote: well, technically it's for sure possible, it's just another of these 24h things. Technically, it's a question of interface, and insofar of which types of modules. Obviously, these will be slower, and how slow is another interface/effort question. Thinking about this, one could probably also claim the answer is yes, you can write OUTPUT modules in any language, it's just a doc issue. In fact, omprog can be used as an interface here. It's actually not even a bad interface... Again, something learned ;) Probably the cheapest (implementation) binding for rsyslog would be a system() like call. Execute the subprogram with /bin/sh -c and communicate with structured messages on STDIO. a real module binding would be far more complex. It would allow the module (in whatever language) access to the rsyslog queues and other data structures. This is possible, but not easy by any means. One big problem is that currently rsyslog does all this work in a threaded environment. It may make sense in v9 or v10 to shift from a default shared-everything threading model to a explicit shared memory multiprocess model. At that point having one of the processes use a different language would not be that hard. But in the current threaded model, having one thread run a different language would be very, very hard. The other issue here is performance. Rsyslog goes to a LOT of effort to be fast. Some of the things that have made very noticable diffences in performance in rsyslog are things that seem like they should be very minor. Think about these things and then think about what would be involved to define interfaces in a multi-language safe way. things that have resulted in noticable speedups have been: removing gettimeofday() calls. it used to be that rsyslog recorded when a message arrived, when it was put on the main queue, when it was moved to an action queue, when it was pulled from the action queue, and when it was delivered now, high performance users configure rsyslog so that it only does one gettimeofday() call per hundred (ot thousand) messages that arrive and use that one time for every message string modules it used to be that the default template (%pri%%timestamp% %hostname% %syslogtag%%msg%) was interpreted by the rsyslog engine for every message that was output now string modules written in C create these strings rather than interpreting the template. This resulted in a double-digit % performance improvement With optimizations like these in use, changing things to allow for a module written in a different language to have access to the rsyslog internals as would be needed for a high-performance interface seems like it will probably end up hurting the rsyslog performance overall. That being said, I am very much in favor of multi-process with explicit sharing rather than multi-threaded with implicit sharing, but getting all the interfaces correct and fast would be a VERY hard task. David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Modules in other programming languages?
Hi, Thanks for the info. I was asking because having the ability to write ims and oms in different languages would open a lot of opportunities. This is one of those enablement things. I understand writing modules in other languages may mean those using such modules may hurt performance, but some people need certain functionality more than performance. Take omkafka example from the other day. If there were a way to write an om in Java it's be trivial for a lot of Java developers out there to contribute omkafka. If omprog enables development of the ecosystem, it sounds like something to point out clearly somewhere and nurture that a bit. I do see http://www.rsyslog.com/doc/omprog.html because somebody shared a link, but I don't see that on http://www.rsyslog.com/doc or on http://www.rsyslog.com/doc/dev_oplugins.html or in the new README. Coincidentally, I just came across Fluentd's instructions for writing plugins, which could serve as guidance: http://docs.fluentd.org/articles/plugin-development . Nice, clean, well structured, not a lot of prose... Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr Elasticsearch Support * http://sematext.com/ On Sat, Dec 14, 2013 at 1:39 PM, David Lang da...@lang.hm wrote: On Sat, 14 Dec 2013, RB wrote: On Sat, Dec 14, 2013 at 5:24 AM, Rainer Gerhards rgerha...@hq.adiscon.com wrote: well, technically it's for sure possible, it's just another of these 24h things. Technically, it's a question of interface, and insofar of which types of modules. Obviously, these will be slower, and how slow is another interface/effort question. Thinking about this, one could probably also claim the answer is yes, you can write OUTPUT modules in any language, it's just a doc issue. In fact, omprog can be used as an interface here. It's actually not even a bad interface... Again, something learned ;) Probably the cheapest (implementation) binding for rsyslog would be a system() like call. Execute the subprogram with /bin/sh -c and communicate with structured messages on STDIO. a real module binding would be far more complex. It would allow the module (in whatever language) access to the rsyslog queues and other data structures. This is possible, but not easy by any means. One big problem is that currently rsyslog does all this work in a threaded environment. It may make sense in v9 or v10 to shift from a default shared-everything threading model to a explicit shared memory multiprocess model. At that point having one of the processes use a different language would not be that hard. But in the current threaded model, having one thread run a different language would be very, very hard. The other issue here is performance. Rsyslog goes to a LOT of effort to be fast. Some of the things that have made very noticable diffences in performance in rsyslog are things that seem like they should be very minor. Think about these things and then think about what would be involved to define interfaces in a multi-language safe way. things that have resulted in noticable speedups have been: removing gettimeofday() calls. it used to be that rsyslog recorded when a message arrived, when it was put on the main queue, when it was moved to an action queue, when it was pulled from the action queue, and when it was delivered now, high performance users configure rsyslog so that it only does one gettimeofday() call per hundred (ot thousand) messages that arrive and use that one time for every message string modules it used to be that the default template (%pri%%timestamp% %hostname% %syslogtag%%msg%) was interpreted by the rsyslog engine for every message that was output now string modules written in C create these strings rather than interpreting the template. This resulted in a double-digit % performance improvement With optimizations like these in use, changing things to allow for a module written in a different language to have access to the rsyslog internals as would be needed for a high-performance interface seems like it will probably end up hurting the rsyslog performance overall. That being said, I am very much in favor of multi-process with explicit sharing rather than multi-threaded with implicit sharing, but getting all the interfaces correct and fast would be a VERY hard task. David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow
