Hello, (I apologize if this got sent twice but I may have sent it too soon before the mailing list registration process was completed.)
I'm hoping the group can provide some guidance around a requirement we have to have transactional support around having an Rsyslog OM module that writes to Kafka. What we'd like to do, is leverage consuming data from Rsyslog and posting to Kafka however it looks like the OMKafka module doesn't currently support transactions when posting to Kafka in the event of an error. So, if we write to Kafka, and an error occurs, it looks like we lose the log-line for that particular Action item unless we write it to a fallback log file. Additionally, we've looked at writing our own Kafka Producer using the OMProg style where we consume off of STDIN and then connect to kafka and produce data. Unfortunately this approach also doesn't allow us to communicate back to Rsyslog that a failure has happened in the event of an error. We tried to deal with this by writing to a fallback text file, and this works great when there are errors with Kafka, but if the process dies between receiving from STDIN and before writing to Kafka, we can still potentially lose messages. Additionally, I noticed that the OMHIREDIS client does support transactions, so it sounds like we want the design of OMHIREDIS (where it utilizes transactions) but with that support in OMKafka. I'm just looking for a recommendation on a way forward from the group. If it makes more sense to enhance OMKafka to have transaction support or if we try to go down the path of adding some type of transaction support to the Omprog module which would allow us to continue using our custom Kafka producer of which is written in Go. Thanks for anyone's time around this! Ralph @deckarep _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
