We're seeing a strange scenario where omkafka completely stops producing messages, even though logs are still being written. I enabled stats ( statistics.interval.ms=60000) and noticed the the queue size which was hovering near the max of 500000 suddenly (within 1 minute) dropped to 0. That seems unlikely, but I don't know where to look for further debugging information on this. Here's a snippet from the omkafka statistics output that shows the ruleset queue. The drop to 0 occurs at 16:49:48.
Mon Sep 24 16:45:48 2018: ruleset_kafka_soak: origin=core.queue size=433692 enqueued=378979590 full=0 discarded.full=0 discarded.nf=0 maxqsize=500000 Mon Sep 24 16:46:48 2018: ruleset_kafka_soak: origin=core.queue size=450602 enqueued=381037332 full=0 discarded.full=0 discarded.nf=0 maxqsize=500000 Mon Sep 24 16:47:48 2018: ruleset_kafka_soak: origin=core.queue size=470304 enqueued=383389707 full=0 discarded.full=0 discarded.nf=0 maxqsize=500000 Mon Sep 24 16:48:48 2018: ruleset_kafka_soak: origin=core.queue size=494598 enqueued=385575664 full=0 discarded.full=0 discarded.nf=0 maxqsize=500000 Mon Sep 24 16:49:48 2018: ruleset_kafka_soak: origin=core.queue size=0 enqueued=386380346 full=0 discarded.full=0 discarded.nf=0 maxqsize=500000 Mon Sep 24 16:50:48 2018: ruleset_kafka_soak: origin=core.queue size=0 enqueued=386380346 full=0 discarded.full=0 discarded.nf=0 maxqsize=500000 Mon Sep 24 16:51:48 2018: ruleset_kafka_soak: origin=core.queue size=0 enqueued=386380346 full=0 discarded.full=0 discarded.nf=0 maxqsize=500000 Mon Sep 24 16:52:48 2018: ruleset_kafka_soak: origin=core.queue size=0 enqueued=386380346 full=0 discarded.full=0 discarded.nf=0 maxqsize=500000 Plus the queue size never goes back above 0, even though logs are still being written, and the actions are not suspended either. -- Peter Cullen Principal Software Engineer Reservoir Labs [email protected] Office: (212) 780-0527 x160 _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
