Hi guys, First of all, thanks to share your thoughts about this issue. I think it’s really important to find out a solution for this issue together.
I can see two behaviors related, but for me the root cause is the same: 1- mpd5 process stuck with umtxn flag 2- system crash I’ve tested recently on FreeBSD 10.3 and FreeBSD-11-RC3. I’ve tried all suggested tunings with no success. My environment is: - About 430 clients connected (but i can add more) - Using ZFS - igb NICs. - Generic kernel Two days ago i updated my system to FreeBSD 11-RELEASE-p1 and after this my system seems stable for almost 3 days. No crashes anymore. I need more days to feel confident if something has changed. But anyway, my crashes before happened every day. If it crashs again i’ll apply Donald recommendation and let you guys know. Let’s keep in touch, to try to at last fix it. Thanks. On Wed, Oct 12, 2016 at 8:24 PM, Donald Baud via freebsd-net < freebsd-net@freebsd.org> wrote: > On 10/12/16 3:24 PM, Zaphod Beeblebrox wrote: > > While my mp5 servers are possibly less busy (I havn't had common crashes), >> I have noticed a "group" of problems. >> >> 1. The carrier dropping communication (ie: fiber cut or l2 switch >> breakage) of the L2TP streams can leave mpd5 in a state where it will not >> die and will not destroy interfaces (requires reboot to clear). >> > I've encountered that once on 10.3 and I had tweaked some sysctl values > while monitoring : > > vmstat -z | head -1; vmstat -z | grep -i netgraph > > you might want to search other people's experience with the following > values: > # net.graph.maxdgram #this is set in /etc/sysctl.conf > # net.graph.recvspace #this is set in /etc/sysctl.conf > # net.graph.maxdata #this is set in /boot/loader.conf > # net.graph.maxalloc #this is set in /boot/loader.conf > > I'll leave others to comment on what's best to set as values with their > experience on FreeBSD10.3. > In my case, as I had explained, one of the recipes that worked for me is > to comment out and leave those kernel values to their default. > > I've read in mpd5 mailing list some saying that FreeBSD-11 have had > upgrades on the netgraph modules. > I am now using FreeBSD-11 and It looks like I don't need any of the kernel > tweaks that I've described. > > Also, may I suggest you troubleshoot the fiber-cut or L2 switch breakage > by playing with some ipfw values to simulate a fiber-cut.: > ex: ipfw add 100 deny ip from 10.10.10.10 to me > >> 2. There are race conditions between quagga and mpd5 for adding/dropping >> routes. >> > While troubleshooting the crashes of the mpd5, I have removed net/quagga > and installed net/bird instead. > I am now using net/bird I've written a little howto to get you started > with net/bird > see: https://forums.freebsd.org/threads/56988/ > > 3. if A is a pppoe client and B is the mpd5 server, A cannot access TCP >> services on B. It can access tcp services _beyond_ B, but not on B. (there >> is a ticket open for this). >> >> On Wed, Oct 12, 2016 at 10:51 AM, Donald Baud via freebsd-net < >> freebsd-net@freebsd.org <mailto:freebsd-net@freebsd.org>> wrote: >> >> >> On 10/12/16 1:13 AM, Julian Elischer wrote: >> >> On 11/10/2016 8:56 PM, Donald Baud via freebsd-net wrote: >> >> I've been plagued with these =daily= panics until I tried >> the following recipes and the server has been up for 30 >> days so far: >> >> Normally I should expermient more to see which one of the >> receipes is really the fix, but I'm just glad that the >> server is stable for now. >> >> >> this is really great information. >> It makes debugging a lot more possible. >> I know it is a hard question, but do you have a way to >> simulate this workload? >> >> I have no real way to simulate this kind of workload >> >> >> Sadly, I don't have a way to simulate the workload but I am very >> interested to help fix these crashes since as Cassiano said, this >> makes mpd5/freebsd useless for pppoe/l2tp termination. >> >> At this point, I would suggest that Cassiano and Андрей confirm >> that they don't get panics when they apply the recipes that I am >> using. >> >> I am still running many other cisco-vpdn gateways that I would >> convert into mpd5/freebsd but my plan was stalled with the daily >> crashes. >> I'll wait a couple of weeks to be sure that my recipes are a valid >> workaround before converting my remaining cisco gateways to mpd5. >> >> -Dbaud >> >> >> >> recipe-1: Don't let mpd5 start automatically when server >> boots: >> i.e. in: /etc/rc.conf >> mpd5_enable="NO" >> and wait about 5 minutes after server boots then issue: >> /usr/local/etc/rc.d/mpd5 onestart >> >> >> recipe-2: recompile the kernel with the NETGRAPH_DEBUG option: >> options NETGRAPH >> options NETGRAPH_DEBUG >> options NETGRAPH_KSOCKET >> options NETGRAPH_L2TP >> options NETGRAPH_SOCKET >> options NETGRAPH_TEE >> options NETGRAPH_VJC >> options NETGRAPH_PPP >> options NETGRAPH_IFACE >> options NETGRAPH_MPPC_COMPRESSION >> options NETGRAPH_MPPC_ENCRYPTION >> options NETGRAPH_TCPMSS >> options IPFIREWALL >> >> recipe-3: recompile the kernel and disable the IPv6 and >> SCTP options: >> nooptions INET6 >> nooptions SCTP >> >> recipe-4: Don't use any of the sysctl optimizations >> in other words I commented out all values in sysctl.conf: >> # net.graph.maxdgram=20480 (this is the default) >> # net.graph.recvspace=20480 (this is the default) >> >> recipe-5: Don't use any of the loader.conf optimizations >> in other words I commented out all values in loader.conf >> # net.graph.maxdata=4096 (this is the default) >> # net.graph.maxalloc=4096 (this is the default) >> >> ================================ >> In my case, I had the panics with 10.3 and 11-PRERELEASE >> 11.0-PRERELEASE FreeBSD 11.0-PRERELEASE #2 r305587 >> >> With those recipes, I have been running without any crash >> for a month and counting. Thats' 300 l2tp tunnels and >> 1400 l2tp sessions generating 700Mbit/s. >> >> >> -DBaud >> >> >> On Tuesday, October 11, 2016 7:30 AM, Cassiano Peixoto >> <peixotocassi...@gmail.com >> <mailto:peixotocassi...@gmail.com>> wrote: >> Hi, >> >> There are many users complaining about this: >> >> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=186114 >> <https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=186114> >> >> I've been dealing with this issue for one year with no >> solution. mpd5 as >> pppoe server on FreeBSD is useless with this bug. >> >> I really would like to see it working again, i think it's >> quite important >> to both project and many users. >> >> Thanks. >> >> On Tue, Oct 11, 2016 at 3:24 AM, Eugene Grosbein >> <eu...@grosbein.net <mailto:eu...@grosbein.net>> wrote: >> >> 11.10.2016 11:02, Андрей Леушкин пишет: >> >> Hello. I have problem with "FreeBSD nas >> 10.3-RELEASE FreeBSD 10.3-RELEASE >> #0: Fri Oct 7 21:12:56 YEKT 2016 >> nas@nas:/usr/obj/usr/src/sys/nasv3 >> amd64" >> >> Kernel panic is repeated at intervals of 2-3 days. >> At first I thought that >> the problem is in the hardware, but the problem >> did not go away after >> replacing the server platform. >> >> Coredumps and more info on link >> https://drive.google.com/open? >> id=0BxciMy2q7ZjTTkIxem9wTE1tM2M >> <https://drive.google.com/open >> ?id=0BxciMy2q7ZjTTkIxem9wTE1tM2M> >> >> Sorry for my english. >> I'll wait for an answer. >> >> This is known and long-stanging problem in the FreeBSD >> network stack. >> It shows up when you have lots of network interfaced >> created/removed >> frequently >> like in your case of Network Access Server (PPtP, >> PPPoE etc). >> >> Generally, people run into this problem using mpd5 >> network daemon. >> mpd5 uses NETGRAPH kernel subsystem to process traffic and >> if an interface disappears (f.e., ,user disconnected) >> while kernel still processes traffic obtained from >> this interface, it >> panices. >> >> There were lots of reports of this problem. Noone >> seems to be working on >> it at the moment. >> You should fill a PR using Bugzilla and attach your >> logs to it. >> >> Eugene Grosbein >> >> > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org" > _______________________________________________ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
