Hello, I'm hoping the group can provide some guidance around a requirement we have to have transactional support around having an Rsyslog OM module that writes to Kafka.
What we'd like to do, is leverage consuming data from Rsyslog and posting to Kafka however it looks like the OMKafka module doesn't currently support transactions when posting to Kafka in the event of an error. So, if we write to Kafka, and an error occurs, it sounds like we lose the log-line for that particular Action item? Additionally, we've looked at writing our own Kafka Produce using the OMProg style where we consume off of STDIN and then connect to kafka and produce data. Unfortunately this approach also doesn't allow us to communicate back to Rsyslog that a failure has happened in the event of an error. We tried to deal with this by writing to a fallback text file, and this works great when there are errors with Kafka, but if the process dies between receiving from STDIN and before writing to Kafka, we can still potentially lose messages. Additionally, I noticed the OMHIREDIS client does support transactions, so it sounds like we want the design of OMHIREDIS (where it utilizes transactions) but with that support in OMKafka. I'm just looking for a recommendation on a way forward from the group. If it makes more sense to enhance OMKafka perhaps we can contribute a change to make it transactional...beyond that i'm not sure what a solid solution is. Thanks for anyone's time around this! Ralph @deckarep _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
