On Wed, 13 Jan 2016, Ralph Caraveo wrote:
Hello,
I'm hoping the group can provide some guidance around a requirement we have
to have transactional support around having an Rsyslog OM module that
writes to Kafka.
What we'd like to do, is leverage consuming data from Rsyslog and posting
to Kafka however it looks like the OMKafka module doesn't currently support
transactions when posting to Kafka in the event of an error. So, if we
write to Kafka, and an error occurs, it sounds like we lose the log-line
for that particular Action item?
Additionally, we've looked at writing our own Kafka Produce using the
OMProg style where we consume off of STDIN and then connect to kafka and
produce data. Unfortunately this approach also doesn't allow us to
communicate back to Rsyslog that a failure has happened in the event of an
error.
We tried to deal with this by writing to a fallback text file, and this
works great when there are errors with Kafka, but if the process dies
between receiving from STDIN and before writing to Kafka, we can still
potentially lose messages.
Additionally, I noticed the OMHIREDIS client does support transactions, so
it sounds like we want the design of OMHIREDIS (where it utilizes
transactions) but with that support in OMKafka.
I'm just looking for a recommendation on a way forward from the group. If
it makes more sense to enhance OMKafka perhaps we can contribute a change
to make it transactional...beyond that i'm not sure what a solid solution
is.
It makes a lot of sense to change omkafka to handle batches/transactions as it
should be significantly more efficcient to process the messages.
When you get an error in a batch, you should roll back the entire batch and
rsyslog will try again with a subset of that batch (continuing until the batch
holds one message, the one with an error in it)
The issue is what to do at that point.
I don't remember if we try to differentiate between 'I had some sort of problem'
and 'this message is bad and can never be processed', so you may end up needing
to set your retries to something larger than 1, but not infinate so that it will
give up after a few tries (you want _some_ retries to handle network hiccups,
server reboots, etc) remember that unless you have a secondary queue (action or
ruleset queue) all log processing is going to pause while the retries happen.
you will then need to use the error handling in the rsyslog.conf file to have a
statement that only executes if there was an error on the omkafka action and
have it write to a file or do whatever you choose, otherwise the message will be
'lost' as far as omkafka is concerned (but anything else will process it
normally)
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
