Re: [rsyslog] pre-releease of rsyslog 08.18.0 available
Hi, OK, when I fixed the PR which added faketime_common.sh I misunderstood your suggestion to call "diag.sh exit" (I replaced my simple "exit 0" call with "diag.sh exit" so we are cleaning up the configs, that's why the testconf.conf file is missing later, but didn't really exit the test). Fixed via https://github.com/rsyslog/rsyslog/pull/964 -Thomas ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] pre-releease of rsyslog 08.18.0 available
Hi, yep, I dropped the workaround an no failures yet. However I tried to run test suite on a non-complaint year 2038 system and seeing test failures I cannot explain at the moment: > ***SUBTEST: check 2038-01-01 > rsyslogd started with pid 22465 > starting run 1 > Sending 1 messages. > ^M0001 messages sent > runtime: 0.000 > End of tcpflood Run > imdiag[13500]: mainqueue empty > Skipping further tests because system doesn't support year 2038 ... > --- > ***SUBTEST: check 2040-01-01 > ERROR: config file 'testconf.conf' not found! > connect(): Connection refused > connect() failed > error in trying to open connection i=0 > error opening connections > error during tcpflood! see rsyslog.out.log.save for what was written > cp: cannot stat ‘rsyslog.out.log’: No such file or directory > FAIL timegenerated-uxtimestamp.sh (exit status: 1) > > > > [...] > ***SUBTEST: check 2038-01-01 > rsyslogd started with pid 22872 > starting run 1 > Sending 1 messages. > ^M0001 messages sent > runtime: 0.000 > End of tcpflood Run > imdiag[13500]: mainqueue empty > Skipping further tests because system doesn't support year 2038 ... > --- > ***SUBTEST: check 2038-12-31 > ERROR: config file 'testconf.conf' not found! > connect(): Connection refused > connect() failed > error in trying to open connection i=0 > error opening connections > error during tcpflood! see rsyslog.out.log.save for what was written > cp: cannot stat ‘rsyslog.out.log’: No such file or directory > FAIL timegenerated-dateordinal.sh (exit status: 1) I am wondering about seeing another subtest after "Skipping further tests" and about "file 'testconf.conf' not found" error. But haven't looked in this yet. -Thomas ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] retry if output module returns RS_RET_SUSPENDED
Hello Rainer, what's your general position on this? Do you think dropping messages is better than duplicating em? On Fri, Apr 1, 2016 at 7:03 AM, Rainer Gerhards wrote: > 2016-04-01 15:58 GMT+02:00 Kane Kim : > > Bump, Rainer, what's your thoughts on what I've posted above?^ > > one thing after another, please ;) I did not yet have time to dig > deeper into that (but much of the work I did was kind of prep work). I > hope to be able to look some time next week into it. > > Rainer > > > Also, it seems I understand why rsyslog would drop messages, but for our > > use case it's better to have duplicates than drop them (we want at least > > once semantics), is it possible to configure rsyslog that way (or add > this > > functionality)? > > > > Thanks! > > > > On Tue, Mar 29, 2016 at 10:46 AM, Kane Kim > wrote: > > > >> Updated config to: > >> module(load="omrelp") > >> local0.* action(type="omrelp" > >> target="10.0.0.1" > >> port="2514" > >> name="debug_relp" > >> queue.filename="relp_queue" > >> queue.maxdiskspace="1G" > >> queue.saveonshutdown="on" > >> queue.spoolDirectory="/var/spool/rsyslog" > >> queue.type="Disk" > >> action.resumeRetryCount="-1" > >> action.resumeInterval="1" > >> ) > >> > >> Still message is lost: > >> > >> 3489.658349703:imuxsock.c : Message from UNIX socket: #3 > >> 3489.658367582:imuxsock.c : main Q: qqueueAdd: entry added, size now > >> log 1, phys 1 entries > >> 3489.658372561:imuxsock.c : main Q: EnqueueMsg advised worker start > >> 3489.658374985:imuxsock.c : imuxsock calling select, active > >> file descriptors (max 3): 3 > >> 3489.658384201:main Q:Reg/w0 : wti 0x23c8ba0: worker awoke from idle > >> processing > >> 3489.658386792:main Q:Reg/w0 : DeleteProcessedBatch: we deleted 0 > objects > >> and enqueued 0 objects > >> 3489.658388548:main Q:Reg/w0 : doDeleteBatch: delete batch from store, > >> new sizes: log 1, phys 1 > >> 3489.658391356:main Q:Reg/w0 : processBATCH: batch of 1 elements must > be > >> processed > >> 3489.658392786:main Q:Reg/w0 : processBATCH: next msg 0: <134>Mar 29 > >> 17:44:49 logger: test > >> 3489.658395027:main Q:Reg/w0 : PRIFILT 'local0.*' > >> 3489.658397540:main Q:Reg/w0 : pmask: X X X X X X X X X X > >> X X X X X X FF X X X X X X X X X > >> 3489.658413235:main Q:Reg/w0 : PRIFILT condition result is 1 > >> 3489.658414686:main Q:Reg/w0 : ACTION 0 > [omrelp:action(type="omrelp" > >> ...)] > >> 3489.658417648:main Q:Reg/w0 : executing action 0 > >> 3489.658419520:main Q:Reg/w0 : action 'debug_relp': called, logging to > >> omrelp (susp 0/0, direct q 0) > >> 3489.658433462:main Q:Reg/w0 : strm 0x23b74c0: file -1(relp_queue) > flush, > >> buflen 410 > >> 3489.658435496:main Q:Reg/w0 : strmPhysWrite, stream 0x23b74c0, len 410 > >> 3489.658484243:main Q:Reg/w0 : file > >> '/var/spool/rsyslog/relp_queue.0001' opened as #5 with mode 384 > >> 3489.658487347:main Q:Reg/w0 : strm 0x23b74c0: opened file > >> '/var/spool/rsyslog/relp_queue.0001' for WRITE as 5 > >> 3489.658500498:main Q:Reg/w0 : strm 0x23b74c0: file 5 write wrote 410 > >> bytes > >> 3489.658502751:main Q:Reg/w0 : debug_relp queue: write wrote 410 octets > >> to disk, queue disk size now 410 octets, EnqOnly:0 > >> 3489.658504568:main Q:Reg/w0 : debug_relp queue: qqueueAdd: entry > added, > >> size now log 1, phys 1 entries > >> 3489.658506705:main Q:Reg/w0 : debug_relp queue:Reg: high activity - > >> starting 1 additional worker thread(s). > >> 3489.658528951:main Q:Reg/w0 : debug_relp queue:Reg: started with state > >> 0, num workers now 1 > >> 3489.658530689:main Q:Reg/w0 : debug_relp queue: EnqueueMsg advised > >> worker start > >> 3489.658532183:main Q:Reg/w0 : action 'debug_relp': set suspended state > >> to 0 > >> 3489.658533672:main Q:Reg/w0 : PRIFILT > >> '*.info;mail.none;authpriv.none;cron.none' > >> 3489.658536164:main Q:Reg/w0 : pmask: 7F 7F X 7F 7F 7F 7F 7F 7F X > >> X 7F 7F 7F 7F 7F 7F 7F 7F 7F 7F 7F 7F 7F 7F 7F > >> 3489.658552240:main Q:Reg/w0 : PRIFILT condition result is 1 > >> 3489.658553393:main Q:Reg/w0 : ACTION 1 > >> [builtin:omfile:/var/log/messages] > >> 3489.658556301:main Q:Reg/w0 : executing action 1 > >> 3489.658557878:main Q:Reg/w0 : action 'action 1': called, logging to > >> builtin:omfile (susp 0/0, direct q 1) > >> 3489.658561265:main Q:Reg/w0 : action 'action 1': is transactional - > >> executing in commit phase > >> 3489.658563696:main Q:Reg/w0 : Action 1 transitioned to state: itx > >> 3489.658564902:main Q:Reg/w0 : action 'action 1': set suspended state > to 0 > >> 3489.658566053:main Q:Reg/w0 : PRIFILT 'authpriv.*' > >> 3489.658568233:main Q:Reg/w0 : pmask: X X X X X X X X X X > >> FF X X X X X X X X X X X X X X X > >> 3489.658582888:main Q:Reg/w0 : PRIFILT condition result is 0 > >> 3489.658584058:main Q:Reg/w0 : PRIFILT 'mail.*' > >> 3489.658586243:main Q:Reg/
Re: [rsyslog] Rsyslog exiting on signal 15
I'm saying it should be kill based on the intuition that it can't be handled. So it is the quickest way to free memory. Not sure though, need to check code. On Apr 19, 2016 1:23 AM, "David Lang" wrote: > I'm not sure. I know I have seen some cases where rsyslog has exited > saying it was signal 15 when it was really SIGHUP that it received when > things are corrupted. > > I know that it's preferred to let a program exit with SIGTERM rather than > SIGKILL if at all possible, and the behavior of the OOM handler has changed > over time, so it may vary depending on what you are running. > > David Lang > > On Tue, 19 Apr 2016, singh.janmejay wrote: > > Won't oom be a sigkill? >> >> You can determine which process issued term (syscall is kill). >> >> Match it with time of death to find the culprit. >> On Apr 19, 2016 12:32 AM, "Alec Swan" wrote: >> >> I checked dmesg output and didn't find anything related to rsyslog being >> killed even though I found that other programs were killed at completely >> different times due to OOM. >> >> I will be looking forward to 8.18 release to fix these. >> >> Thanks, >> >> Alec >> >> On Mon, Apr 18, 2016 at 11:50 AM, David Lang wrote: >> >> signal 15 is SIGTERM, which tells rsyslog to shutdown gracefully. Check to >>> see if you are hitting OOM or something like that (check dmesg) >>> >>> I have seen this happen when things get corrupted and a log rotation >>> happens. 8.14 was still using json-c which we found has some nasty race >>> conditions in it that can cause these sorts of corruptions (this is why >>> libfastjson was forked from it) >>> >>> David Lang >>> >>> >>> On Mon, 18 Apr 2016, Alec Swan wrote: >>> >>> Hi there, >>> I've been seeing the following stack trace in /var/log/messages. It seems like something is trying to shut down rsyslog causing it to leave its queues in a bad state. The log line preceding rsyslog termination with signal 15 happens 14 minutes prior and doesn't give me any information on who tried to terminate rsyslog. dmesg doesn't show rsyslog being killed either. Any ideas how to track down what kills rsyslog? Apr 17 15:57:37 m0058601 dhclient[771]: bound to ... Apr 17 16:11:58 m0058601 rsyslogd: [origin software="rsyslogd" swVersion="8.14.0" x-pid="2408" x-info="http://www.rsyslog.com";] exiting on signal 15. Apr 17 16:12:05 m0058601 rsyslogd: [origin software="rsyslogd" swVersion="8.14.0" x-pid="10955" x-info="http://www.rsyslog.com";] start Apr 17 16:12:05 m0058601 rsyslogd-2040: fatal error on disk queue 'omelasticsearch-event-indexer-http-request.log queue[DA]', emergency switch to direct mode [v8.14.0 try http://www.rsyslog.com/e/2040 ] Apr 17 16:12:05 m0058601 rsyslogd-2040: fatal error on disk queue 'omelasticsearch-syslog queue[DA]', emergency switch to direct mode [v8.14.0 try http://www.rsyslog.com/e/2040 ] Apr 17 16:12:04 m0058601 rsyslogd-2221: module 'imuxsock' already in this config, cannot be added [v8.14.0 try http://www.rsyslog.com/e/2221 ] Apr 17 16:12:04 m0058601 rsyslogd-2221: module 'imklog' already in this config, cannot be added [v8.14.0 try http://www.rsyslog.com/e/2221 ] Thanks, Alec ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> >>> ___ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of >> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >> LIKE THAT. >> ___ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> >> __
Re: [rsyslog] pre-releease of rsyslog 08.18.0 available
Thx! Was that with or without the "10 second wait patch"? Sent from phone, thus brief. Am 18.04.2016 21:57 schrieb "Thomas D." : > Hi, > > RC passed build tests with test suite on Gentoo x64 and x86. > > Thanks! > > > -Thomas > ___ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] pre-releease of rsyslog 08.18.0 available
Hi, RC passed build tests with test suite on Gentoo x64 and x86. Thanks! -Thomas ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Rsyslog exiting on signal 15
I'm not sure. I know I have seen some cases where rsyslog has exited saying it was signal 15 when it was really SIGHUP that it received when things are corrupted. I know that it's preferred to let a program exit with SIGTERM rather than SIGKILL if at all possible, and the behavior of the OOM handler has changed over time, so it may vary depending on what you are running. David Lang On Tue, 19 Apr 2016, singh.janmejay wrote: Won't oom be a sigkill? You can determine which process issued term (syscall is kill). Match it with time of death to find the culprit. On Apr 19, 2016 12:32 AM, "Alec Swan" wrote: I checked dmesg output and didn't find anything related to rsyslog being killed even though I found that other programs were killed at completely different times due to OOM. I will be looking forward to 8.18 release to fix these. Thanks, Alec On Mon, Apr 18, 2016 at 11:50 AM, David Lang wrote: signal 15 is SIGTERM, which tells rsyslog to shutdown gracefully. Check to see if you are hitting OOM or something like that (check dmesg) I have seen this happen when things get corrupted and a log rotation happens. 8.14 was still using json-c which we found has some nasty race conditions in it that can cause these sorts of corruptions (this is why libfastjson was forked from it) David Lang On Mon, 18 Apr 2016, Alec Swan wrote: Hi there, I've been seeing the following stack trace in /var/log/messages. It seems like something is trying to shut down rsyslog causing it to leave its queues in a bad state. The log line preceding rsyslog termination with signal 15 happens 14 minutes prior and doesn't give me any information on who tried to terminate rsyslog. dmesg doesn't show rsyslog being killed either. Any ideas how to track down what kills rsyslog? Apr 17 15:57:37 m0058601 dhclient[771]: bound to ... Apr 17 16:11:58 m0058601 rsyslogd: [origin software="rsyslogd" swVersion="8.14.0" x-pid="2408" x-info="http://www.rsyslog.com";] exiting on signal 15. Apr 17 16:12:05 m0058601 rsyslogd: [origin software="rsyslogd" swVersion="8.14.0" x-pid="10955" x-info="http://www.rsyslog.com";] start Apr 17 16:12:05 m0058601 rsyslogd-2040: fatal error on disk queue 'omelasticsearch-event-indexer-http-request.log queue[DA]', emergency switch to direct mode [v8.14.0 try http://www.rsyslog.com/e/2040 ] Apr 17 16:12:05 m0058601 rsyslogd-2040: fatal error on disk queue 'omelasticsearch-syslog queue[DA]', emergency switch to direct mode [v8.14.0 try http://www.rsyslog.com/e/2040 ] Apr 17 16:12:04 m0058601 rsyslogd-2221: module 'imuxsock' already in this config, cannot be added [v8.14.0 try http://www.rsyslog.com/e/2221 ] Apr 17 16:12:04 m0058601 rsyslogd-2221: module 'imklog' already in this config, cannot be added [v8.14.0 try http://www.rsyslog.com/e/2221 ] Thanks, Alec ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Rsyslog exiting on signal 15
Won't oom be a sigkill? You can determine which process issued term (syscall is kill). Match it with time of death to find the culprit. On Apr 19, 2016 12:32 AM, "Alec Swan" wrote: I checked dmesg output and didn't find anything related to rsyslog being killed even though I found that other programs were killed at completely different times due to OOM. I will be looking forward to 8.18 release to fix these. Thanks, Alec On Mon, Apr 18, 2016 at 11:50 AM, David Lang wrote: > signal 15 is SIGTERM, which tells rsyslog to shutdown gracefully. Check to > see if you are hitting OOM or something like that (check dmesg) > > I have seen this happen when things get corrupted and a log rotation > happens. 8.14 was still using json-c which we found has some nasty race > conditions in it that can cause these sorts of corruptions (this is why > libfastjson was forked from it) > > David Lang > > > On Mon, 18 Apr 2016, Alec Swan wrote: > > Hi there, >> >> I've been seeing the following stack trace in /var/log/messages. It seems >> like something is trying to shut down rsyslog causing it to leave its >> queues in a bad state. The log line preceding rsyslog termination with >> signal 15 happens 14 minutes prior and doesn't give me any information on >> who tried to terminate rsyslog. dmesg doesn't show rsyslog being killed >> either. Any ideas how to track down what kills rsyslog? >> >> Apr 17 15:57:37 m0058601 dhclient[771]: bound to ... >> Apr 17 16:11:58 m0058601 rsyslogd: [origin software="rsyslogd" >> swVersion="8.14.0" x-pid="2408" x-info="http://www.rsyslog.com";] exiting >> on >> signal 15. >> Apr 17 16:12:05 m0058601 rsyslogd: [origin software="rsyslogd" >> swVersion="8.14.0" x-pid="10955" x-info="http://www.rsyslog.com";] start >> Apr 17 16:12:05 m0058601 rsyslogd-2040: fatal error on disk queue >> 'omelasticsearch-event-indexer-http-request.log queue[DA]', emergency >> switch to direct mode [v8.14.0 try http://www.rsyslog.com/e/2040 ] >> Apr 17 16:12:05 m0058601 rsyslogd-2040: fatal error on disk queue >> 'omelasticsearch-syslog queue[DA]', emergency switch to direct mode >> [v8.14.0 try http://www.rsyslog.com/e/2040 ] >> Apr 17 16:12:04 m0058601 rsyslogd-2221: module 'imuxsock' already in this >> config, cannot be added [v8.14.0 try http://www.rsyslog.com/e/2221 ] >> Apr 17 16:12:04 m0058601 rsyslogd-2221: module 'imklog' already in this >> config, cannot be added [v8.14.0 try http://www.rsyslog.com/e/2221 ] >> >> >> Thanks, >> >> Alec >> ___ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> >> ___ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Rsyslog exiting on signal 15
I checked dmesg output and didn't find anything related to rsyslog being killed even though I found that other programs were killed at completely different times due to OOM. I will be looking forward to 8.18 release to fix these. Thanks, Alec On Mon, Apr 18, 2016 at 11:50 AM, David Lang wrote: > signal 15 is SIGTERM, which tells rsyslog to shutdown gracefully. Check to > see if you are hitting OOM or something like that (check dmesg) > > I have seen this happen when things get corrupted and a log rotation > happens. 8.14 was still using json-c which we found has some nasty race > conditions in it that can cause these sorts of corruptions (this is why > libfastjson was forked from it) > > David Lang > > > On Mon, 18 Apr 2016, Alec Swan wrote: > > Hi there, >> >> I've been seeing the following stack trace in /var/log/messages. It seems >> like something is trying to shut down rsyslog causing it to leave its >> queues in a bad state. The log line preceding rsyslog termination with >> signal 15 happens 14 minutes prior and doesn't give me any information on >> who tried to terminate rsyslog. dmesg doesn't show rsyslog being killed >> either. Any ideas how to track down what kills rsyslog? >> >> Apr 17 15:57:37 m0058601 dhclient[771]: bound to ... >> Apr 17 16:11:58 m0058601 rsyslogd: [origin software="rsyslogd" >> swVersion="8.14.0" x-pid="2408" x-info="http://www.rsyslog.com";] exiting >> on >> signal 15. >> Apr 17 16:12:05 m0058601 rsyslogd: [origin software="rsyslogd" >> swVersion="8.14.0" x-pid="10955" x-info="http://www.rsyslog.com";] start >> Apr 17 16:12:05 m0058601 rsyslogd-2040: fatal error on disk queue >> 'omelasticsearch-event-indexer-http-request.log queue[DA]', emergency >> switch to direct mode [v8.14.0 try http://www.rsyslog.com/e/2040 ] >> Apr 17 16:12:05 m0058601 rsyslogd-2040: fatal error on disk queue >> 'omelasticsearch-syslog queue[DA]', emergency switch to direct mode >> [v8.14.0 try http://www.rsyslog.com/e/2040 ] >> Apr 17 16:12:04 m0058601 rsyslogd-2221: module 'imuxsock' already in this >> config, cannot be added [v8.14.0 try http://www.rsyslog.com/e/2221 ] >> Apr 17 16:12:04 m0058601 rsyslogd-2221: module 'imklog' already in this >> config, cannot be added [v8.14.0 try http://www.rsyslog.com/e/2221 ] >> >> >> Thanks, >> >> Alec >> ___ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> >> ___ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] pre-releease of rsyslog 08.18.0 available
Yay! I'll try doing a local build. Brian On Mon, Apr 18, 2016 at 8:59 AM, Rainer Gerhards wrote: > Hi all, > > I have just placed pre-release tarballs for tomorrow's upcoming > rsyslog 8.18.0 release on the server. > > * http://www.rsyslog.com/files/download/rsyslog/rc/rsyslog-8.18.0.tar.gz > > That tarbal is most probably the final one. > > It would be appreciated if a build could be tried and any problems be > reported. > > Note that I originally intended to do a new release (0.99.3) of > libfastjson at the same time. However, there are some spots I am > ironing out, and so I have postponed that. The 8.18.0 codebase works > with 0.99.2 and will most probably be able to support 0.99.3, when > available, without any changes to rsyslog. > > Thanks, > Rainer > ___ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Rsyslog exiting on signal 15
signal 15 is SIGTERM, which tells rsyslog to shutdown gracefully. Check to see if you are hitting OOM or something like that (check dmesg) I have seen this happen when things get corrupted and a log rotation happens. 8.14 was still using json-c which we found has some nasty race conditions in it that can cause these sorts of corruptions (this is why libfastjson was forked from it) David Lang On Mon, 18 Apr 2016, Alec Swan wrote: Hi there, I've been seeing the following stack trace in /var/log/messages. It seems like something is trying to shut down rsyslog causing it to leave its queues in a bad state. The log line preceding rsyslog termination with signal 15 happens 14 minutes prior and doesn't give me any information on who tried to terminate rsyslog. dmesg doesn't show rsyslog being killed either. Any ideas how to track down what kills rsyslog? Apr 17 15:57:37 m0058601 dhclient[771]: bound to ... Apr 17 16:11:58 m0058601 rsyslogd: [origin software="rsyslogd" swVersion="8.14.0" x-pid="2408" x-info="http://www.rsyslog.com";] exiting on signal 15. Apr 17 16:12:05 m0058601 rsyslogd: [origin software="rsyslogd" swVersion="8.14.0" x-pid="10955" x-info="http://www.rsyslog.com";] start Apr 17 16:12:05 m0058601 rsyslogd-2040: fatal error on disk queue 'omelasticsearch-event-indexer-http-request.log queue[DA]', emergency switch to direct mode [v8.14.0 try http://www.rsyslog.com/e/2040 ] Apr 17 16:12:05 m0058601 rsyslogd-2040: fatal error on disk queue 'omelasticsearch-syslog queue[DA]', emergency switch to direct mode [v8.14.0 try http://www.rsyslog.com/e/2040 ] Apr 17 16:12:04 m0058601 rsyslogd-2221: module 'imuxsock' already in this config, cannot be added [v8.14.0 try http://www.rsyslog.com/e/2221 ] Apr 17 16:12:04 m0058601 rsyslogd-2221: module 'imklog' already in this config, cannot be added [v8.14.0 try http://www.rsyslog.com/e/2221 ] Thanks, Alec ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] pre-releease of rsyslog 08.18.0 available
2016-04-18 19:34 GMT+02:00 David Lang : > Just a note that the cron jobs to generate PPAs are failing due to the > cleanup changes that require a PPA variable be defined in the config There are a couple of issues, not all related to the cleanup. I noticed that last friday when I began with release preps. In a nutshell, the archive was not properly populated for a couple of weeks, which also made part of the extended integration tests not being carried out. One of them is a issue with libfastjson. I am cleaning up the archive, but it will probably take the week to become it fully functional again. I'll also think about better integration tests. Rainer > > David Lang > > > On Mon, 18 Apr 2016, Rainer Gerhards wrote: > >> Hi all, >> >> I have just placed pre-release tarballs for tomorrow's upcoming >> rsyslog 8.18.0 release on the server. >> >> * http://www.rsyslog.com/files/download/rsyslog/rc/rsyslog-8.18.0.tar.gz >> >> That tarbal is most probably the final one. >> >> It would be appreciated if a build could be tried and any problems be >> reported. >> >> Note that I originally intended to do a new release (0.99.3) of >> libfastjson at the same time. However, there are some spots I am >> ironing out, and so I have postponed that. The 8.18.0 codebase works >> with 0.99.2 and will most probably be able to support 0.99.3, when >> available, without any changes to rsyslog. >> >> Thanks, >> Rainer >> ___ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >> LIKE THAT. >> > ___ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] pre-releease of rsyslog 08.18.0 available
Just a note that the cron jobs to generate PPAs are failing due to the cleanup changes that require a PPA variable be defined in the config David Lang On Mon, 18 Apr 2016, Rainer Gerhards wrote: Hi all, I have just placed pre-release tarballs for tomorrow's upcoming rsyslog 8.18.0 release on the server. * http://www.rsyslog.com/files/download/rsyslog/rc/rsyslog-8.18.0.tar.gz That tarbal is most probably the final one. It would be appreciated if a build could be tried and any problems be reported. Note that I originally intended to do a new release (0.99.3) of libfastjson at the same time. However, there are some spots I am ironing out, and so I have postponed that. The 8.18.0 codebase works with 0.99.2 and will most probably be able to support 0.99.3, when available, without any changes to rsyslog. Thanks, Rainer ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
[rsyslog] Rsyslog exiting on signal 15
Hi there, I've been seeing the following stack trace in /var/log/messages. It seems like something is trying to shut down rsyslog causing it to leave its queues in a bad state. The log line preceding rsyslog termination with signal 15 happens 14 minutes prior and doesn't give me any information on who tried to terminate rsyslog. dmesg doesn't show rsyslog being killed either. Any ideas how to track down what kills rsyslog? Apr 17 15:57:37 m0058601 dhclient[771]: bound to ... Apr 17 16:11:58 m0058601 rsyslogd: [origin software="rsyslogd" swVersion="8.14.0" x-pid="2408" x-info="http://www.rsyslog.com";] exiting on signal 15. Apr 17 16:12:05 m0058601 rsyslogd: [origin software="rsyslogd" swVersion="8.14.0" x-pid="10955" x-info="http://www.rsyslog.com";] start Apr 17 16:12:05 m0058601 rsyslogd-2040: fatal error on disk queue 'omelasticsearch-event-indexer-http-request.log queue[DA]', emergency switch to direct mode [v8.14.0 try http://www.rsyslog.com/e/2040 ] Apr 17 16:12:05 m0058601 rsyslogd-2040: fatal error on disk queue 'omelasticsearch-syslog queue[DA]', emergency switch to direct mode [v8.14.0 try http://www.rsyslog.com/e/2040 ] Apr 17 16:12:04 m0058601 rsyslogd-2221: module 'imuxsock' already in this config, cannot be added [v8.14.0 try http://www.rsyslog.com/e/2221 ] Apr 17 16:12:04 m0058601 rsyslogd-2221: module 'imklog' already in this config, cannot be added [v8.14.0 try http://www.rsyslog.com/e/2221 ] Thanks, Alec ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
[rsyslog] pre-releease of rsyslog 08.18.0 available
Hi all, I have just placed pre-release tarballs for tomorrow's upcoming rsyslog 8.18.0 release on the server. * http://www.rsyslog.com/files/download/rsyslog/rc/rsyslog-8.18.0.tar.gz That tarbal is most probably the final one. It would be appreciated if a build could be tried and any problems be reported. Note that I originally intended to do a new release (0.99.3) of libfastjson at the same time. However, there are some spots I am ironing out, and so I have postponed that. The 8.18.0 codebase works with 0.99.2 and will most probably be able to support 0.99.3, when available, without any changes to rsyslog. Thanks, Rainer ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
