17.12.2013 21:20, David Lang:
The prior suggestions for omrest were specifically for a module that
created a pull mechanism for getting logs rather than having the logs
pushed to the destination at the speed that they arrive.
One of the justifications was avoiding overloading log destinations as
they would be in control of the rate that they got data.
Rate control is equally possible with push-based model, say, with TCP,
RELP or omprog. One real advantage of pull model is that it does not
need to do retries when receiving side is busy or non-operational.
Hence, it decreases delays significantly. It is even more valuable given
out current discussion of problems with RELP and suspension. But if
these problems were mitigated, pull mode could be less of a value.
Pull mode requires a model different from current worker threads with
output modules; it is more like as input modules, which run own threads
(while this is not strictly obligatory).
--
Pavel Levshin
David Lang
On Tue, 17 Dec 2013, Brian Knox wrote:
I was thinking that omrest would be a module that made outbound http
requests to send messages; not as something that waited for incoming
http requests for messages. So, in my mind it was something far more
similar to the current elastic search module, which pushes messages out.
So - more of an outgoing http request to an external rest api, rather
than something providing a rest api to make calls into.
-----Original Message-----
From: rsyslog-boun...@lists.adiscon.com
[mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of David Lang
Sent: Monday, December 16, 2013 1:57 PM
To: rsyslog-users
Subject: Re: [rsyslog] Modules in other programming languages?
On Mon, 16 Dec 2013, Brian Knox wrote:
I believe the output module for elastic search might be a good place
to start looking for anyone interested in writing an "omrest" module?
If I recall correctly the elastic search output uses libcurl.
not really.
currently every om* module consists of code that is executed by a
rsyslog worker thread that is passed a list of messages and acts on
each message.
omelasticsearch pushes messages via libcurl
for omrest you would need to change the entire paradigm of how an om*
module would work.
Instead of being code invoked by a worker thread that's invoking many
other
om* code as well on a given message, the omrelp module would need to
listen for a connection from the outside, and when it receives a
request, it would need to retrieve messages from the queue, and the
worker threads would need to leave the messages on the queue.
You should see by now that this is a really ugly thing to talk about
implementing. It's almost a complete re-write of the rsyslog core to
support this.
The other approach is to have omrelp maintain it's own queue of
messages and knowledge of who should be asking for messages, timeouts
for messages that aren't asked for, etc. At that point, omrelp's
interface to rsyslog is straightforward, it's just the omrelp queue
and interface stuff that gets really 'interesting' and people who
want omrelp should work on writing some code that will perform all
the relp functions that you want to support but just accepts new
messages on stdin (which would let it be driven by omprog for now),
and then as you get it running and find it useful the input portion
could be changed to make it into a 'real' rsyslog om* module.
Depending on what you do for your queue, you may be able to use a
different language to handle adding things to the queue and pulling
things from the queue. This would make it easy to have some rsyslog C
code that adds things to the queueu.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
