Should I create an issue for omkafka on github? -----Original Message----- From: rsyslog [mailto:[email protected]] On Behalf Of Jigar Rathod Sent: Friday, February 01, 2019 3:17 PM To: 'rsyslog-users' Subject: Re: [rsyslog] Issues with omkafka
Prerequisite: rsyslog is reading a file that has 1 million lines. For simplicity, each line can be a line number itself. Kafka broker is up & running. Keep queue size small i.e. 50,000. Scenario: Run rsyslog for few minutes. It will start sending some messages. Suddenly stop kafka broker. Observation: Content from Imfile-state will be updating continuously. Curr_offs will keep increasing. When curr_offs stops increasing; start kafka broker. You will notice that no messages would be published to kafka. My theory: Imfile and omkafka are not in sync. Omkafka prints success message whenever it sends a message successfully. Omkafka is aware of what's going on. However, imfile is not aware of anything. It will continue to write to queue continuously. This may be the reason why curr_offs will keep increasing. When it stops increasing; this means imfile has done its job. No more data to read; so it will not put anything in the queue. I assume if the queue size was huge then this would not have happened. But, producer cannot rely upon that. Is there a way to tell imfile to stop reading the file; as soon as omkafka detects that kafka broker is not working ? -----Original Message----- From: rsyslog [mailto:[email protected]] On Behalf Of Jigar Rathod Sent: Friday, February 01, 2019 1:52 PM To: 'rsyslog-users' Subject: Re: [rsyslog] Issues with omkafka Well, scenario is reproducible. I am running Kafka_2.11-2.0.0 on a single machine. While producer is producing the message; stop the Kafka service. Expected outcome: Producer would resend messages that were not sent earlier Current outcome: Producer does not resend messages when Kafka broker comes back alive. I will also send you debug log. It will take me some time moving the logs to external network. -----Original Message----- From: rsyslog [mailto:[email protected]] On Behalf Of Rainer Gerhards Sent: Friday, February 01, 2019 1:14 PM To: rsyslog-users Subject: Re: [rsyslog] Issues with omkafka strange - this means we probably need a debug log showcasing the problem: https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.rsyslog.com%2Fdoc%2Fv8-stable%2Ftroubleshooting%2Fhowtodebug.html%3Fhighlight%3Ddebug%2520log&data=01%7C01%7Cjrathod%40interactivebrokers.com%7C04686a4982234fdf83b408d6888263f9%7C7abd04ef837d48e69ba869d84f65a110%7C0&sdata=ljVCMZaXvTUrCwPg3GGmFQvmHvSjSzyKQVjS60i816I%3D&reserved=0 Rainer El vie., 1 feb. 2019 a las 18:56, Jigar Rathod (<[email protected]>) escribió: > > It's rsyslogd 8.40.0 with lirdkafka-0.11.5. > > -----Original Message----- > From: rsyslog [mailto:[email protected]] On Behalf Of > Rainer Gerhards > Sent: Friday, February 01, 2019 12:46 PM > To: rsyslog-users > Subject: Re: [rsyslog] Issues with omkafka > > El vie., 1 feb. 2019 a las 18:41, Jigar Rathod > (<[email protected]>) escribió: > > > > Thanks for quick response. > > > > The config specified in the mail is not a verbatim config. > > I should have mentioned that I generalized the configuration by specifying > > variables such as $CUSTOMLOCATION, $PATH_TO_FILE, $KAFKA_BROKER. Actual > > configuration does not have this variables. It has specific filepath > > /broker url. > > Which version is it? rsyslogd -v will tell. > > Rainer > _______________________________________________ > rsyslog mailing list > https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists. > adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&data=01%7C01%7Cjrathod% > 40interactivebrokers.com%7C49b99d858def41a6ebb808d688711107%7C7abd04ef > 837d48e69ba869d84f65a110%7C0&sdata=KgOqRgKsnlHZ9JnMzk4MejGoNgv1V%2 > FneKqOglAKCArk%3D&reserved=0 > https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.rs > yslog.com%2Fprofessional-services%2F&data=01%7C01%7Cjrathod%40inte > ractivebrokers.com%7C49b99d858def41a6ebb808d688711107%7C7abd04ef837d48 > e69ba869d84f65a110%7C0&sdata=OpV9xI%2BCgMI59RGaFG3tLWFbQKPsX2X%2BD > bPJGlQzc0I%3D&reserved=0 What's up with rsyslog? Follow > https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitt > er.com%2Frgerhards&data=01%7C01%7Cjrathod%40interactivebrokers.com > %7C49b99d858def41a6ebb808d688711107%7C7abd04ef837d48e69ba869d84f65a110 > %7C0&sdata=Y6rYDRwuYzMkFVO6%2F%2BbS2jUbVkjpXBlwxcieriJT9w0%3D& > reserved=0 NOTE WELL: This is a PUBLIC mailing list, posts are > ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO > NOT POST if you DON'T LIKE THAT. > > _______________________________________________ > rsyslog mailing list > https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists. > adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&data=01%7C01%7Cjrathod% > 40interactivebrokers.com%7C49b99d858def41a6ebb808d688711107%7C7abd04ef > 837d48e69ba869d84f65a110%7C0&sdata=KgOqRgKsnlHZ9JnMzk4MejGoNgv1V%2 > FneKqOglAKCArk%3D&reserved=0 > https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.rs > yslog.com%2Fprofessional-services%2F&data=01%7C01%7Cjrathod%40inte > ractivebrokers.com%7C49b99d858def41a6ebb808d688711107%7C7abd04ef837d48 > e69ba869d84f65a110%7C0&sdata=OpV9xI%2BCgMI59RGaFG3tLWFbQKPsX2X%2BD > bPJGlQzc0I%3D&reserved=0 What's up with rsyslog? Follow > https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitt > er.com%2Frgerhards&data=01%7C01%7Cjrathod%40interactivebrokers.com > %7C49b99d858def41a6ebb808d688711107%7C7abd04ef837d48e69ba869d84f65a110 > %7C0&sdata=Y6rYDRwuYzMkFVO6%2F%2BbS2jUbVkjpXBlwxcieriJT9w0%3D& > reserved=0 NOTE WELL: This is a PUBLIC mailing list, posts are > ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO > NOT POST if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&data=01%7C01%7Cjrathod%40interactivebrokers.com%7C04686a4982234fdf83b408d6888263f9%7C7abd04ef837d48e69ba869d84f65a110%7C0&sdata=%2BjvppADq2BKIsbwT8u4jBJmfIT8VljXmjpdwKfhJR2E%3D&reserved=0 https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.rsyslog.com%2Fprofessional-services%2F&data=01%7C01%7Cjrathod%40interactivebrokers.com%7C04686a4982234fdf83b408d6888263f9%7C7abd04ef837d48e69ba869d84f65a110%7C0&sdata=p8HTfDNBWJcftbR2Z8gCbLv62dSRA1uGEZF76UQgzWs%3D&reserved=0 What's up with rsyslog? Follow https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Frgerhards&data=01%7C01%7Cjrathod%40interactivebrokers.com%7C04686a4982234fdf83b408d6888263f9%7C7abd04ef837d48e69ba869d84f65a110%7C0&sdata=88A3sy1dT7K9BjtcOczzgfAWE5puTZLWqJd%2FkeF1tMc%3D&reserved=0 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&data=01%7C01%7Cjrathod%40interactivebrokers.com%7C04686a4982234fdf83b408d6888263f9%7C7abd04ef837d48e69ba869d84f65a110%7C0&sdata=%2BjvppADq2BKIsbwT8u4jBJmfIT8VljXmjpdwKfhJR2E%3D&reserved=0 https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.rsyslog.com%2Fprofessional-services%2F&data=01%7C01%7Cjrathod%40interactivebrokers.com%7C04686a4982234fdf83b408d6888263f9%7C7abd04ef837d48e69ba869d84f65a110%7C0&sdata=p8HTfDNBWJcftbR2Z8gCbLv62dSRA1uGEZF76UQgzWs%3D&reserved=0 What's up with rsyslog? Follow https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Frgerhards&data=01%7C01%7Cjrathod%40interactivebrokers.com%7C04686a4982234fdf83b408d6888263f9%7C7abd04ef837d48e69ba869d84f65a110%7C0&sdata=88A3sy1dT7K9BjtcOczzgfAWE5puTZLWqJd%2FkeF1tMc%3D&reserved=0 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&data=01%7C01%7Cjrathod%40interactivebrokers.com%7C04686a4982234fdf83b408d6888263f9%7C7abd04ef837d48e69ba869d84f65a110%7C0&sdata=%2BjvppADq2BKIsbwT8u4jBJmfIT8VljXmjpdwKfhJR2E%3D&reserved=0 https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.rsyslog.com%2Fprofessional-services%2F&data=01%7C01%7Cjrathod%40interactivebrokers.com%7C04686a4982234fdf83b408d6888263f9%7C7abd04ef837d48e69ba869d84f65a110%7C0&sdata=p8HTfDNBWJcftbR2Z8gCbLv62dSRA1uGEZF76UQgzWs%3D&reserved=0 What's up with rsyslog? Follow https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Frgerhards&data=01%7C01%7Cjrathod%40interactivebrokers.com%7C04686a4982234fdf83b408d6888263f9%7C7abd04ef837d48e69ba869d84f65a110%7C0&sdata=88A3sy1dT7K9BjtcOczzgfAWE5puTZLWqJd%2FkeF1tMc%3D&reserved=0 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
