On Tue, 7 Mar 2006, Steve Fairhead wrote:
One of my production machines (3.8-stable) has suddenly started
panicing every couple of hours. I found out that the culprit is smbd,
eating through memory like there's no tomorrow (approx. 10Mb /
minute! ). Can't figure out what has triggered it, nothing changed on
the machine lately and there is only one active w2k client, writing a
2.5kB file every 15 seconds or so.
I'd be glad of any assistance, even pointing out any stupid mistakes I
have made, because this is driving me nuts.
I ran into something very similar recently. In my case I eventually
discovered that one user was writing to a folder containing 22,000 files.
Avoiding this folder has entirely solved the problem. (Or at least worked
around it.)
FWIW, the Samba logs were helpful only inasmuch as they pointed me to the
user who was "causing" the problem. I had to sit down and watch her operate
to find out what she was doing...
Perhaps (indeed probably) not relevant to your problem, but might give you
some ideas. If you're writing a file every 15s, perhaps your problem is
related to mine.
Steve
http://www.fivetrees.com
Have a similar problem, though mine appears when moving files between
various directories when either has a large number of files in it.
Finally decided to delve into the Samba code to see if I could track
down the cause. I believe I tracked it to code that scans the
directories prior to the file rename operation. The code that scans
the directories does some odd things (like hammer on the system lib
call: telldir() which appears to leak memory like a sieve if
unaccompanied by a matching seekdir() -- approximately 16 bytes per
file per directory scanned). This can add up to signifant loss if
the directory has, say, 10,000 files or more in it, especially if
the directory is scanned multiple times per operation and repeatedly
over time.
I'm at a bit of a loss how to proceed from here. Given the state and
conventions of *BSD-ish directory library calls, Samba isn't scanning
directories in a very memory efficient manner -- at least in the
case of OpenBSD. But the directory traversals and possible
dependencies on the scanning methods could spider badly to fix it
properly and reliably within Samba. (ie, I believe that to fix the
issue in Samba "properly" is a heck of a lot of work and effort,
but I'm also not exactly a Samba expert/developer either)
On the other hand, the fact that the system library call telldir() can
leak as badly as it does probably isn't a good thing either as
outlined here:
http://mail-index.netbsd.org/netbsd-bugs/2004/02/05/0008.html
It would appear that at least in OpenBSD 3.8-release, the library
implementation suffers similar potental issues. I have no idea if
the patch proposed in the URL above ever made it into NetBSD, since
I don't run NetBSD anywhere; however, the patch looks promising.
Changing the implementation of telldir() and related functions
would likely fix this particular memory leak in Samba as well, though
there may be underlying OS/userland issues about which I am unaware.
I guess the bottom line here is that I can see that if you have a
process writing into a directory that contains a LOT of files, Samba
(or your client, or both) may be scanning the directory prior to any
write, and possibly multiple times. If that's the case, the
telldir() issue will likely affect you as well. While reducing the
number of files in the directory in question won't stop the leak, it
may significantly slow it ...
I suppose this also assumes that your problem is related to the
one I am seeing, and that my preliminary analysis is correct.
Hope this helps,
- Paul
