Re: xl driver again? Re: mbuf leakage on 4.1.1-STABLE
In message <[EMAIL PROTECTED]>, Bill Paul writes: > The xl driver never holds more than 128 mbuf clusters in the receive ring. > Whenever a new packet comes in, it sends one of the mbufs out and replaces > it with a new one. IT DOESN'T HOLD ONTO THEM. It will however complain if > it can't allocate a replacement mbuf. What it will do is allocate them very > fast, and sometimes mbufs do get held inside the kernel for too long a > time. It seems to be worse in cases where you have a lot of UDP traffic > or a large number of open TCP connections. My only suggestion for now > is to add more mbufs by bumping NMBCLUSTERS. Is there a reason why the kernel might hold on to mbufs for "too long a time" (I take that to mean longer than it should)? Regards, Phone: (250)387-8437 Cy Schubert Fax: (250)387-5766 Team Leader, Sun/DEC Team Internet: [EMAIL PROTECTED] Open Systems Group, ITSD, ISTA Province of BC To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-stable" in the body of the message
Re: mbuf leakage on 4.1.1-STABLE
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Fri, 13 Oct 2000, David Greenman wrote: ... :Perhaps I should have spoken up earlier, but none of these reports has : indicated a problem. The "92%" above is the percent of buffers that have been : allocated and are currently in-use. The system allocates more when the pool : runs out. The % in use is essentially a useless number that shouldn't even : be reported because it just causes confusion about what it means. Sounds like a chance for a FAQ entry? I know I got caught by that originally till I sat down and thought about it for a while, might be nice to have. :) : -DG * Matt Heckaman - mailto:[EMAIL PROTECTED] http://www.lucida.qc.ca/ * * GPG fingerprint - A9BC F3A8 278E 22F2 9BDA BFCF 74C3 2D31 C035 5390 * -BEGIN PGP SIGNATURE- Version: GnuPG v1.0.3 (FreeBSD) Comment: http://www.lucida.qc.ca/pgp iD8DBQE55tQJdMMtMcA1U5ARAr2VAJsHeHGh7t8K6UYbG5ApYG/h9ElZjQCffYY4 zGYXNcARIHhJx8F20rq5w2M= =5VGI -END PGP SIGNATURE- To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-stable" in the body of the message
Re: mbuf leakage on 4.1.1-STABLE
>"Kuriyama, Kent K Mr (CPF N651KK)" wrote: >> >> Chris, >> >> Your email prompted me to look at mbuf utilization on a 4.1.1-STABLE box >> that is currently not in production. >> >> outside# netstat -m >> 130/160/7168 mbufs in use (current/peak/max): >> 129 mbufs allocated to data >> 1 mbufs allocated to packet headers >> 128/136/1792 mbuf clusters in use (current/peak/max) >> 312 Kbytes allocated to network (92% in use) >> ^^ Perhaps I should have spoken up earlier, but none of these reports has indicated a problem. The "92%" above is the percent of buffers that have been allocated and are currently in-use. The system allocates more when the pool runs out. The % in use is essentially a useless number that shouldn't even be reported because it just causes confusion about what it means. -DG David Greenman Co-founder, The FreeBSD Project - http://www.freebsd.org President, TeraSolutions, Inc. - http://www.terasolutions.com Pave the road of life with opportunities. To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-stable" in the body of the message
Re: mbuf leakage on 4.1.1-STABLE
"Kuriyama, Kent K Mr (CPF N651KK)" wrote: > > Chris, > > Your email prompted me to look at mbuf utilization on a 4.1.1-STABLE box > that is currently not in production. > > outside# netstat -m > 130/160/7168 mbufs in use (current/peak/max): > 129 mbufs allocated to data > 1 mbufs allocated to packet headers > 128/136/1792 mbuf clusters in use (current/peak/max) > 312 Kbytes allocated to network (92% in use) > ^^ > 0 requests for memory denied > 0 requests for memory delayed > 0 calls to protocol drain routines > outside# uptime > 4:32AM up 1 day, 14:01, 1 user, load averages: 0.00, 0.00, 0.00 > > I don't know whether to be concerned about the 92% utilization since the > number of bytes allocated seems low. The machine has never crashed but then > it has never served as a server. > > Is this kind of mbuf utilization expected? > > Kent Kuriyama > SPAWAR Sys Ctr San Diego D424, CINCPACFLT N671KK > [EMAIL PROTECTED], 808-471-4125 Mine looks like the following: A box with heavy traffic : FreeBSD 3.4-RELEASE Average traffic exceeds 4GB per day. # netstat -m 867/1120 mbufs in use: 800 mbufs allocated to data 67 mbufs allocated to packet headers 463/656/1024 mbuf clusters in use (current/peak/max) 1452 Kbytes allocated to network (71% in use) 0 requests for memory denied 0 requests for memory delayed 0 calls to protocol drain routines # uptime 9:28AM up 206 days, 19:30, 1 user, load averages: 0.00, 0.00, 0.00 And a box with very light traffic : FreeBSD 4.1-2828-STABLE Average traffic less than 20MB per day. $ netstat -m 207/448/4096 mbufs in use (current/peak/max): 157 mbufs allocated to data 50 mbufs allocated to packet headers 117/254/1024 mbuf clusters in use (current/peak/max) 620 Kbytes allocated to network (46% in use) 0 requests for memory denied 0 requests for memory delayed 0 calls to protocol drain routines $ uptime 9:21AM up 17 days, 17:09, 4 users, load averages: 2.02, 0.97, 0.50 The worrying thing is that the box with light traffic has had to be rebooted because it stopped talking on the network. This happens expecially quickly if I am using NFS a lot. To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-stable" in the body of the message
RE: mbuf leakage on 4.1.1-STABLE
On Thu, 12 Oct 2000, Matt Heckaman wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > I thought I would through this into the mix: > > Server, NOT in production yet: 4.1.1-RELEASE: > > matt[beta]:~> uptime;netstat -m > 10:40AM up 16 days, 1:42, 2 users, load averages: 0.00, 0.00, 0.00 > 132/352/131072 mbufs in use (current/peak/max): > 130 mbufs allocated to data > 2 mbufs allocated to packet headers > 128/316/32768 mbuf clusters in use (current/peak/max) > 720 Kbytes allocated to network (40% in use) > 0 requests for memory denied > 0 requests for memory delayed > 0 calls to protocol drain routines > > Heavy Use Workstation, 4.1.1-RELEASE: > > 10:42AM up 16 days, 49 mins, 9 users, load averages: 0.28, 0.20, 0.17 > 693/1712/131072 mbufs in use (current/peak/max): > 132 mbufs allocated to data > 561 mbufs allocated to packet headers > 131/1410/32768 mbuf clusters in use (current/peak/max) > 3248 Kbytes allocated to network (13% in use) > 0 requests for memory denied > 0 requests for memory delayed > 0 calls to protocol drain routines > > So what you're showing below looks pretty normal. Note: I KNOW that I have > a small leak on epsilon. It's from wmbiff, which holds up descriptors like > you would believe. Gotta shut it down every couple of weeks to clear it > out, it's quite funny. Just to give a roug idea: > > root[epsilon]:~# lsof -p 98037 | wc -l > 3225 behanna@topperwein> netstat -m 211/272/8192 mbufs in use (current/peak/max): 181 mbufs allocated to data 30 mbufs allocated to packet headers 175/182/2048 mbuf clusters in use (current/peak/max) 432 Kbytes allocated to network (93% in use) 0 requests for memory denied 0 requests for memory delayed 0 calls to protocol drain routines behanna@topperwein> uptime 11:11AM up 4:42, 6 users, load averages: 0.06, 0.02, 0.00 This is a box sitting on the end of a cable modem in my basement office. it doesn't see a large network load, although I'm blocking about 900 spam attempts per day (the same two bozos with forged addresses keep retrying every five minutes, and I keep 550-ing them. Now, it could be that my ISP is retrying to send these messages, which wouldn't surprise me :-( ). By tonight, I expect to see around 500 mbuf clusters in use, and a comparable number of mbufs in use. By late tomorrow, it will be time to reboot. :-( -- Chris BeHanna Software Engineer (at yourfit.com) [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-stable" in the body of the message
RE: mbuf leakage on 4.1.1-STABLE
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I thought I would through this into the mix: Server, NOT in production yet: 4.1.1-RELEASE: matt[beta]:~> uptime;netstat -m 10:40AM up 16 days, 1:42, 2 users, load averages: 0.00, 0.00, 0.00 132/352/131072 mbufs in use (current/peak/max): 130 mbufs allocated to data 2 mbufs allocated to packet headers 128/316/32768 mbuf clusters in use (current/peak/max) 720 Kbytes allocated to network (40% in use) 0 requests for memory denied 0 requests for memory delayed 0 calls to protocol drain routines Heavy Use Workstation, 4.1.1-RELEASE: 10:42AM up 16 days, 49 mins, 9 users, load averages: 0.28, 0.20, 0.17 693/1712/131072 mbufs in use (current/peak/max): 132 mbufs allocated to data 561 mbufs allocated to packet headers 131/1410/32768 mbuf clusters in use (current/peak/max) 3248 Kbytes allocated to network (13% in use) 0 requests for memory denied 0 requests for memory delayed 0 calls to protocol drain routines So what you're showing below looks pretty normal. Note: I KNOW that I have a small leak on epsilon. It's from wmbiff, which holds up descriptors like you would believe. Gotta shut it down every couple of weeks to clear it out, it's quite funny. Just to give a roug idea: root[epsilon]:~# lsof -p 98037 | wc -l 3225 Oh well. :) On Thu, 12 Oct 2000, Kuriyama, Kent K Mr (CPF N651KK) wrote: : Chris, : : Your email prompted me to look at mbuf utilization on a 4.1.1-STABLE box : that is currently not in production. : : outside# netstat -m : 130/160/7168 mbufs in use (current/peak/max): : 129 mbufs allocated to data : 1 mbufs allocated to packet headers : 128/136/1792 mbuf clusters in use (current/peak/max) : 312 Kbytes allocated to network (92% in use) : ^^ : 0 requests for memory denied : 0 requests for memory delayed : 0 calls to protocol drain routines : outside# uptime : 4:32AM up 1 day, 14:01, 1 user, load averages: 0.00, 0.00, 0.00 : : I don't know whether to be concerned about the 92% utilization since the : number of bytes allocated seems low. The machine has never crashed but then : it has never served as a server. : : Is this kind of mbuf utilization expected? : : Kent Kuriyama : SPAWAR Sys Ctr San Diego D424, CINCPACFLT N671KK : [EMAIL PROTECTED], 808-471-4125 * Matt Heckaman - mailto:[EMAIL PROTECTED] http://www.lucida.qc.ca/ * * GPG fingerprint - A9BC F3A8 278E 22F2 9BDA BFCF 74C3 2D31 C035 5390 * -BEGIN PGP SIGNATURE- Version: GnuPG v1.0.3 (FreeBSD) Comment: http://www.lucida.qc.ca/pgp iD8DBQE55c6DdMMtMcA1U5ARArwgAKCS4RiGBzFXwcwQN6TD5yUsVrt6QQCgrt/8 YFTb8kW6FEArAv0N9qS0FZ0= =kZM5 -END PGP SIGNATURE- To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-stable" in the body of the message
RE: mbuf leakage on 4.1.1-STABLE
Chris, Your email prompted me to look at mbuf utilization on a 4.1.1-STABLE box that is currently not in production. outside# netstat -m 130/160/7168 mbufs in use (current/peak/max): 129 mbufs allocated to data 1 mbufs allocated to packet headers 128/136/1792 mbuf clusters in use (current/peak/max) 312 Kbytes allocated to network (92% in use) ^^ 0 requests for memory denied 0 requests for memory delayed 0 calls to protocol drain routines outside# uptime 4:32AM up 1 day, 14:01, 1 user, load averages: 0.00, 0.00, 0.00 I don't know whether to be concerned about the 92% utilization since the number of bytes allocated seems low. The machine has never crashed but then it has never served as a server. Is this kind of mbuf utilization expected? Kent Kuriyama SPAWAR Sys Ctr San Diego D424, CINCPACFLT N671KK [EMAIL PROTECTED], 808-471-4125 -Original Message- From: Chris BeHanna [mailto:[EMAIL PROTECTED]] Sent: Thursday, October 12, 2000 01:15 AM To: FreeBSD-Stable Subject: mbuf leakage on 4.1.1-STABLE I posted a week or two ago about my machine wedging every couple of days, and seeing RX packet drop messages due to no memory available in /var/log/messages. Specifically, the messages were "dhclient: send_packet: No buffer space available /kernel: xl0: no memory for rx list -- packet dropped!" repeated thousands of times. Sean O'Connell pointed me in the right direction, stating that this looked like an mbuf starvation problem. I've been checking my system constantly via netstat -m, and it looks like I'm leaking mbufs (mbufs in use and mbuf clusters in use increase until they hit their limit, then the machine freezes, waiting, I suppose, for an mbuf to become available). I've taken an interim measure of doubling the number of NMBCLUSTERS in my kernel, but that just puts off the inevitable. I end up rebooting when I'm at 80% mbuf cluster usage, so that I can avoid having to fsck my disks when the machine wedges and I have to hit the button. A friend of mine suggested that I instrument mbuf.h and uipc_mbuf.c so that I could see where all of the allocs and frees occur. I've looked through these files, and it's not immediately obvious to me just how I'd instrument them to do that (e.g., __FILE__ and __LINE__ obviously can't be used). For reference, I'm running 4.1.1-STABLE, cvsup'ed early on October 4th or late on October 3rd, and my NIC is a 3Com 3C905B. I've been seeing this problem for some time now, but it's gotten a lot worse recently, and I'm given to understand that it could be just about anywhere in the protocol stack, not necessarily in the xl driver. I am willing to do some work to debug this, but never having been a kernel hacker, I need a little bit of guidance. Help! My Linux-using co-workers are really giving me the business over this! Thanks, -- Chris BeHanna Software Engineer (at yourfit.com) [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-stable" in the body of the message To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-stable" in the body of the message
