[rsyslog] rsyslog going down frequently
I am sending data to elasticsearch via rsyslog. I have been successful in that. But from yesterday, rsyslog is going down. There was no problem in elasticsearch server, but rsyslog went down a couple of times, alphauser@AlphaServer:~$ dmesg | grep -A1 -B1 "Killed process" | grep "rsyslog" [3829358.159352] Out of memory: Kill process 8340 (rsyslogd) score 621 or sacrifice child [3829358.183579] Killed process 8340 (rsyslogd) total-vm:1577504kB, anon-rss:1064096kB, file-rss:0kB [3829358.495953] init: rsyslog main process (8340) killed by KILL signal [3829822.288212] Out of memory: Kill process 10452 (rsyslogd) score 436 or sacrifice child [3829822.304238] Killed process 10452 (rsyslogd) total-vm:1315360kB, anon-rss:747232kB, file-rss:0kB [3829822.518360] init: rsyslog main process (10452) killed by KILL signal [3830138.346746] Out of memory: Kill process 11990 (rsyslogd) score 377 or sacrifice child [3830138.369427] Killed process 11990 (rsyslogd) total-vm:1184288kB, anon-rss:646832kB, file-rss:0kB [3830138.748289] init: rsyslog main process (11990) killed by KILL signal [3830608.796438] Out of memory: Kill process 13230 (rsyslogd) score 620 or sacrifice child [3830608.810256] Killed process 13230 (rsyslogd) total-vm:1643040kB, anon-rss:1062824kB, file-rss:0kB [3830609.161384] init: rsyslog main process (13230) killed by KILL signal [3830848.524858] Out of memory: Kill process 15113 (rsyslogd) score 290 or sacrifice child [3830848.538968] Killed process 15113 (rsyslogd) total-vm:1053216kB, anon-rss:497632kB, file-rss:0kB [3830848.700492] init: rsyslog main process (15113) killed by KILL signal [3831401.272656] Out of memory: Kill process 16116 (rsyslogd) score 621 or sacrifice child [3831401.287363] Killed process 16116 (rsyslogd) total-vm:1577504kB, anon-rss:1064088kB, file-rss:0kB [3831401.784038] init: rsyslog main process (16116) killed by KILL signal It is telling me that it is going out of memory. I used rsyslog because it is light and fast, but even this is getting a large OOM score which I did not expect. What is the possible reason ? How can I avoid it ? -- View this message in context: http://rsyslog-users.1305293.n2.nabble.com/rsyslog-going-down-frequently-tp7592527.html Sent from the rsyslog-users mailing list archive at Nabble.com. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] rsyslog going down frequently
impstats will show yo how many queues you have, and you can figure it out how much memory they are taking. Each queue you use, increases the memory footprint. On 29/06/17 11:12, Luv via rsyslog wrote: I am sending data to elasticsearch via rsyslog. I have been successful in that. But from yesterday, rsyslog is going down. There was no problem in elasticsearch server, but rsyslog went down a couple of times, alphauser@AlphaServer:~$ dmesg | grep -A1 -B1 "Killed process" | grep "rsyslog" [3829358.159352] Out of memory: Kill process 8340 (rsyslogd) score 621 or sacrifice child [3829358.183579] Killed process 8340 (rsyslogd) total-vm:1577504kB, anon-rss:1064096kB, file-rss:0kB [3829358.495953] init: rsyslog main process (8340) killed by KILL signal [3829822.288212] Out of memory: Kill process 10452 (rsyslogd) score 436 or sacrifice child [3829822.304238] Killed process 10452 (rsyslogd) total-vm:1315360kB, anon-rss:747232kB, file-rss:0kB [3829822.518360] init: rsyslog main process (10452) killed by KILL signal [3830138.346746] Out of memory: Kill process 11990 (rsyslogd) score 377 or sacrifice child [3830138.369427] Killed process 11990 (rsyslogd) total-vm:1184288kB, anon-rss:646832kB, file-rss:0kB [3830138.748289] init: rsyslog main process (11990) killed by KILL signal [3830608.796438] Out of memory: Kill process 13230 (rsyslogd) score 620 or sacrifice child [3830608.810256] Killed process 13230 (rsyslogd) total-vm:1643040kB, anon-rss:1062824kB, file-rss:0kB [3830609.161384] init: rsyslog main process (13230) killed by KILL signal [3830848.524858] Out of memory: Kill process 15113 (rsyslogd) score 290 or sacrifice child [3830848.538968] Killed process 15113 (rsyslogd) total-vm:1053216kB, anon-rss:497632kB, file-rss:0kB [3830848.700492] init: rsyslog main process (15113) killed by KILL signal [3831401.272656] Out of memory: Kill process 16116 (rsyslogd) score 621 or sacrifice child [3831401.287363] Killed process 16116 (rsyslogd) total-vm:1577504kB, anon-rss:1064088kB, file-rss:0kB [3831401.784038] init: rsyslog main process (16116) killed by KILL signal It is telling me that it is going out of memory. I used rsyslog because it is light and fast, but even this is getting a large OOM score which I did not expect. What is the possible reason ? How can I avoid it ? -- View this message in context: http://rsyslog-users.1305293.n2.nabble.com/rsyslog-going-down-frequently-tp7592527.html Sent from the rsyslog-users mailing list archive at Nabble.com. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] rsyslog going down frequently
2017-06-29 11:25 GMT+02:00 mostolog--- via rsyslog : > impstats will show yo how many queues you have, and you can figure it out > how much memory they are taking. > > Each queue you use, increases the memory footprint. +1 >From past experience I would bet that Elastic is too slow, causing queue buildup. Large queues are permitted by conf and so you end up with what you say. As mostolog says very correctly, impstats will show you this. Rainer > > > > On 29/06/17 11:12, Luv via rsyslog wrote: >> >> I am sending data to elasticsearch via rsyslog. >> >> I have been successful in that. But from yesterday, rsyslog is going down. >> There was no problem in elasticsearch server, but rsyslog went down a >> couple >> of times, >> >> alphauser@AlphaServer:~$ dmesg | grep -A1 -B1 "Killed process" | grep >> "rsyslog" >> [3829358.159352] Out of memory: Kill process 8340 (rsyslogd) score 621 or >> sacrifice child >> [3829358.183579] Killed process 8340 (rsyslogd) total-vm:1577504kB, >> anon-rss:1064096kB, file-rss:0kB >> [3829358.495953] init: rsyslog main process (8340) killed by KILL signal >> [3829822.288212] Out of memory: Kill process 10452 (rsyslogd) score 436 or >> sacrifice child >> [3829822.304238] Killed process 10452 (rsyslogd) total-vm:1315360kB, >> anon-rss:747232kB, file-rss:0kB >> [3829822.518360] init: rsyslog main process (10452) killed by KILL signal >> [3830138.346746] Out of memory: Kill process 11990 (rsyslogd) score 377 or >> sacrifice child >> [3830138.369427] Killed process 11990 (rsyslogd) total-vm:1184288kB, >> anon-rss:646832kB, file-rss:0kB >> [3830138.748289] init: rsyslog main process (11990) killed by KILL signal >> [3830608.796438] Out of memory: Kill process 13230 (rsyslogd) score 620 or >> sacrifice child >> [3830608.810256] Killed process 13230 (rsyslogd) total-vm:1643040kB, >> anon-rss:1062824kB, file-rss:0kB >> [3830609.161384] init: rsyslog main process (13230) killed by KILL signal >> [3830848.524858] Out of memory: Kill process 15113 (rsyslogd) score 290 or >> sacrifice child >> [3830848.538968] Killed process 15113 (rsyslogd) total-vm:1053216kB, >> anon-rss:497632kB, file-rss:0kB >> [3830848.700492] init: rsyslog main process (15113) killed by KILL signal >> [3831401.272656] Out of memory: Kill process 16116 (rsyslogd) score 621 or >> sacrifice child >> [3831401.287363] Killed process 16116 (rsyslogd) total-vm:1577504kB, >> anon-rss:1064088kB, file-rss:0kB >> [3831401.784038] init: rsyslog main process (16116) killed by KILL signal >> >> >> It is telling me that it is going out of memory. I used rsyslog because it >> is light and fast, but even this is getting a large OOM score which I did >> not expect. >> What is the possible reason ? How can I avoid it ? >> >> >> >> -- >> View this message in context: >> http://rsyslog-users.1305293.n2.nabble.com/rsyslog-going-down-frequently-tp7592527.html >> Sent from the rsyslog-users mailing list archive at Nabble.com. >> ___ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >> LIKE THAT. > > > ___ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] rsyslog going down frequently
Yeah thats fine. But I did not make any use of queues. This is one of the actions I was usng, action(type="omelasticsearch" template="all-json-nginx" # use the template defined earlier searchIndex="django-logs-alpha" searchType="django" server="xx.xx.xx.xx" serverport="9200" uid="elastic" pwd="x" bulkmode="on" # use the bulk API action.resumeretrycount="-1" # retry indefinitely if Elasticsearch is unreachable ) So, you see no queues implemented. -- View this message in context: http://rsyslog-users.1305293.n2.nabble.com/rsyslog-going-down-frequently-tp7592527p7592530.html Sent from the rsyslog-users mailing list archive at Nabble.com. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] rsyslog going down frequently
2017-06-29 12:31 GMT+02:00 Luv via rsyslog : > Yeah thats fine. > > But I did not make any use of queues. This is one of the actions I was usng, > > action(type="omelasticsearch" > template="all-json-nginx" # use the template defined earlier > searchIndex="django-logs-alpha" > searchType="django" > server="xx.xx.xx.xx" > serverport="9200" > uid="elastic" > pwd="x" > bulkmode="on" # use the bulk API > action.resumeretrycount="-1" # retry indefinitely if Elasticsearch is > unreachable > ) > > > So, you see no queues implemented. > there still is the main queue... We also see other things in pstats, so if you want to get this solved, I think providing the stats is the way to go. Rainer > > > -- > View this message in context: > http://rsyslog-users.1305293.n2.nabble.com/rsyslog-going-down-frequently-tp7592527p7592530.html > Sent from the rsyslog-users mailing list archive at Nabble.com. > ___ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] rsyslog going down frequently
there still is the main queue... We also see other things in pstats, so if you want to get this solved, I think providing the stats is the way to go. we will also need to see the full config, not just snippets of it. David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
