Re: Recent stable: bsnmpd eats up memory and cpu
On Tue, Jul 05, 2016 at 10:38:23AM +0200, Mark Huizer wrote: > On Mon, May 02, 2016 at 12:01:07AM +0200, Wolfgang Zenker wrote: > > Hi, > > > > after updating some 10-STABLE systems a few days ago, I noticed that on > > two of those systems bsnmpd started to use up a lot of cpu time, and the > > available memory shrinked until rendering the system unusable. Killing > > bsnmpd stops the cpu usage but does not free up memory. > > Both affected systems are amd64, one having moved from r297555 to > > r298723, the other from r297555 to r298722. Another amd64 system > > that went from r297555 to r298722 appears to be not affected. > > The two affected systems are on an internal LAN segment and there > > is currently no application connecting to snmp on those machines. > > > > What would be useful debugging data to collect in this case? > > I saw a resembling thing here after upgrading to 10-stable yesterday. > I noticed bsnmp running at 30 to 40% cpu continuously, resembling what I > found here: > > https://forums.freebsd.org/threads/37045/ Sorry, I meant https://forums.freebsd.org/threads/56235/ Greetings Mark -- Nice testing in little China... ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Recent stable: bsnmpd eats up memory and cpu
On Mon, May 02, 2016 at 12:01:07AM +0200, Wolfgang Zenker wrote: > Hi, > > after updating some 10-STABLE systems a few days ago, I noticed that on > two of those systems bsnmpd started to use up a lot of cpu time, and the > available memory shrinked until rendering the system unusable. Killing > bsnmpd stops the cpu usage but does not free up memory. > Both affected systems are amd64, one having moved from r297555 to > r298723, the other from r297555 to r298722. Another amd64 system > that went from r297555 to r298722 appears to be not affected. > The two affected systems are on an internal LAN segment and there > is currently no application connecting to snmp on those machines. > > What would be useful debugging data to collect in this case? I saw a resembling thing here after upgrading to 10-stable yesterday. I noticed bsnmp running at 30 to 40% cpu continuously, resembling what I found here: https://forums.freebsd.org/threads/37045/ After ktrace-ing the process I noticed it was doing sysctl stuff all the time and a more close look showed that it was freaking out on the CD device without a medium. So I inserted a CD in the device, and bsnmp went back to normal. "Interesting". So far I didn't notice memory issues. Mark -- Nice testing in little China... ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Recent stable: bsnmpd eats up memory and cpu
On Mon, 2 May 2016 09:10+0200, Trond Endrestøl wrote: > On Mon, 2 May 2016 00:01+0200, Wolfgang Zenker wrote: > > > Hi, > > > > after updating some 10-STABLE systems a few days ago, I noticed that on > > two of those systems bsnmpd started to use up a lot of cpu time, and the > > available memory shrinked until rendering the system unusable. Killing > > bsnmpd stops the cpu usage but does not free up memory. > > Both affected systems are amd64, one having moved from r297555 to > > r298723, the other from r297555 to r298722. Another amd64 system > > that went from r297555 to r298722 appears to be not affected. > > The two affected systems are on an internal LAN segment and there > > is currently no application connecting to snmp on those machines. > > > > What would be useful debugging data to collect in this case? > > I believe I've seen the very same on my systems. All of them got > updated last Friday due to the recent NTP fix. Prior to last Friday, > they all ran stable/10 from early March, r296648-ish. Neither of them > run bsnmpd, but they offer a lot of network services. > > Three of my i386 systems each with 1 GiB of memory ran out of swap > space, Sunday afternoon. > > This night a mail server running i386 with 4 GiB of memory died while > handling mail. From the messages I could glean on /dev/ttyvb (due to > custom logging) before rebooting, is that it's all networking related. > > SpamAssassin and syslogd on the mail server managed to transmit these > lines to the central log host before dying: > > May 2 00:05:17 [HOSTNAME] spamc[63613]: connect to spamd on ::1 > failed, retrying (#1 of 3): Connection refused > May 2 00:05:17 [HOSTNAME] spamc[63613]: connect to spamd on > 127.0.0.1 failed, retrying (#1 of 3): Connection refused > May 2 00:05:18 [HOSTNAME] spamc[63613]: connect to spamd on ::1 > failed, retrying (#2 of 3): Connection refused > May 2 00:05:18 [HOSTNAME] spamc[63613]: connect to spamd on > 127.0.0.1 failed, retrying (#2 of 3): Connection refused > May 2 00:05:19 [HOSTNAME] spamc[63613]: connect to spamd on ::1 > failed, retrying (#3 of 3): Connection refused > May 2 00:05:19 [HOSTNAME] spamc[63613]: connect to spamd on > 127.0.0.1 failed, retrying (#3 of 3): Connection refused > May 2 00:05:19 [HOSTNAME] spamc[63613]: connection attempt to > spamd aborted after 3 retries > > May 2 00:52:17 [HOSTNAME] sm-mta[63740]: u41Mp86h063740: Milter > (spamassassin): error creating socket: No buffer space available > May 2 00:52:17 [HOSTNAME] sm-mta[63739]: u41Mp8r9063739: Milter > (spamassassin): error creating socket: No buffer space available > May 2 00:52:17 [HOSTNAME] sm-mta[63740]: u41Mp86h063740: Milter > (spamassassin): to error state > May 2 00:52:17 [HOSTNAME] sm-mta[63739]: u41Mp8r9063739: Milter > (spamassassin): to error state > > All of the amd64 systems with 4 GiB or 8 GiB of memory are apparently > unaffected. > > Maybe it's time to convert the remaining i386 systems to amd64 > systems, and add some memory while I'm at it. > > The bug is either in the kernel or in libc, or both. You might want to try out the patch created by Mark Johnston: http://lists.freebsd.org/pipermail/freebsd-current/2016-May/061015.html I'm in the process of testing this patch on one i386 stable/10 system and on one amd64 stable/10 system. -- +---++ | Vennlig hilsen, | Best regards, | | Trond Endrestøl, | Trond Endrestøl, | | IT-ansvarlig, | System administrator, | | Fagskolen Innlandet, | Gjøvik Technical College, Norway, | | tlf. mob. 952 62 567, | Cellular...: +47 952 62 567, | | sentralbord 61 14 54 00. | Switchboard: +47 61 14 54 00. | +---++ ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Recent stable: bsnmpd eats up memory and cpu
On Mon, 2 May 2016 00:01+0200, Wolfgang Zenker wrote: > Hi, > > after updating some 10-STABLE systems a few days ago, I noticed that on > two of those systems bsnmpd started to use up a lot of cpu time, and the > available memory shrinked until rendering the system unusable. Killing > bsnmpd stops the cpu usage but does not free up memory. > Both affected systems are amd64, one having moved from r297555 to > r298723, the other from r297555 to r298722. Another amd64 system > that went from r297555 to r298722 appears to be not affected. > The two affected systems are on an internal LAN segment and there > is currently no application connecting to snmp on those machines. > > What would be useful debugging data to collect in this case? I believe I've seen the very same on my systems. All of them got updated last Friday due to the recent NTP fix. Prior to last Friday, they all ran stable/10 from early March, r296648-ish. Neither of them run bsnmpd, but they offer a lot of network services. Three of my i386 systems each with 1 GiB of memory ran out of swap space, Sunday afternoon. This night a mail server running i386 with 4 GiB of memory died while handling mail. From the messages I could glean on /dev/ttyvb (due to custom logging) before rebooting, is that it's all networking related. SpamAssassin and syslogd on the mail server managed to transmit these lines to the central log host before dying: May 2 00:05:17 [HOSTNAME] spamc[63613]: connect to spamd on ::1 failed, retrying (#1 of 3): Connection refused May 2 00:05:17 [HOSTNAME] spamc[63613]: connect to spamd on 127.0.0.1 failed, retrying (#1 of 3): Connection refused May 2 00:05:18 [HOSTNAME] spamc[63613]: connect to spamd on ::1 failed, retrying (#2 of 3): Connection refused May 2 00:05:18 [HOSTNAME] spamc[63613]: connect to spamd on 127.0.0.1 failed, retrying (#2 of 3): Connection refused May 2 00:05:19 [HOSTNAME] spamc[63613]: connect to spamd on ::1 failed, retrying (#3 of 3): Connection refused May 2 00:05:19 [HOSTNAME] spamc[63613]: connect to spamd on 127.0.0.1 failed, retrying (#3 of 3): Connection refused May 2 00:05:19 [HOSTNAME] spamc[63613]: connection attempt to spamd aborted after 3 retries May 2 00:52:17 [HOSTNAME] sm-mta[63740]: u41Mp86h063740: Milter (spamassassin): error creating socket: No buffer space available May 2 00:52:17 [HOSTNAME] sm-mta[63739]: u41Mp8r9063739: Milter (spamassassin): error creating socket: No buffer space available May 2 00:52:17 [HOSTNAME] sm-mta[63740]: u41Mp86h063740: Milter (spamassassin): to error state May 2 00:52:17 [HOSTNAME] sm-mta[63739]: u41Mp8r9063739: Milter (spamassassin): to error state All of the amd64 systems with 4 GiB or 8 GiB of memory are apparently unaffected. Maybe it's time to convert the remaining i386 systems to amd64 systems, and add some memory while I'm at it. The bug is either in the kernel or in libc, or both. -- +---++ | Vennlig hilsen, | Best regards, | | Trond Endrestøl, | Trond Endrestøl, | | IT-ansvarlig, | System administrator, | | Fagskolen Innlandet, | Gjøvik Technical College, Norway, | | tlf. mob. 952 62 567, | Cellular...: +47 952 62 567, | | sentralbord 61 14 54 00. | Switchboard: +47 61 14 54 00. | +---++ ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Recent stable: bsnmpd eats up memory and cpu
Hi, after updating some 10-STABLE systems a few days ago, I noticed that on two of those systems bsnmpd started to use up a lot of cpu time, and the available memory shrinked until rendering the system unusable. Killing bsnmpd stops the cpu usage but does not free up memory. Both affected systems are amd64, one having moved from r297555 to r298723, the other from r297555 to r298722. Another amd64 system that went from r297555 to r298722 appears to be not affected. The two affected systems are on an internal LAN segment and there is currently no application connecting to snmp on those machines. What would be useful debugging data to collect in this case? Wolfgang ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
