Bug#520794: samba: smbd memory usage always increase - memory leak
On Fri, May 01, 2009 at 07:45:47AM +0200, Christian Perrier wrote: Quoting Peter Cordes (pe...@cordes.ca): It doesn't leak memory anymore, but I'm seeing smbd panics every few days. Well, ae you sure that this is related to the memory leak fix?? No. I didn't see any panics in the week I was running the leaky Debian binary package. Maybe that was just random chance, since I have only seen 4 panics since Apr 14th, when I installed the self-compiled version. Sat Apr 18 14:04:32 2009 Tue Apr 21 16:06:47 2009 Fri Apr 24 17:39:27 2009 Wed Apr 29 18:06:36 2009 (So probably while samba was actually in use from one of the client machines, judging from the times.) Even if the binary package was ok, it's possible my build environment introduced a problem. I built the packages on the same fully up-to-date Lenny that I'm running Samba on, though. Using dpkg-buildpackage -us -uc -b, IIRC. It's possible I would have seen those samba panics if I'd just compiled from source without applying the patch. My machine doesn't have enough memory to test for long with the leak happening, so I can't think of any great solutions. I can install the -dbg package, though, and maybe get a useful backtrace. -- #define X(x,y) x##y Peter Cordes ; e-mail: X(pe...@cor , des.ca) The gods confound the man who first found out how to distinguish the hours! Confound him, too, who in this place set up a sundial, to cut and hack my day so wretchedly into small pieces! -- Plautus, 200 BC -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#520794: [Pkg-samba-maint] Bug#520794: samba: smbd memory usage always increase - memory leak
Quoting Peter Cordes (pe...@cordes.ca): I didn't see any panics in the week I was running the leaky Debian binary package. Maybe that was just random chance, since I have only seen 4 panics since Apr 14th, when I installed the self-compiled version. Sat Apr 18 14:04:32 2009 Tue Apr 21 16:06:47 2009 Fri Apr 24 17:39:27 2009 Wed Apr 29 18:06:36 2009 (So probably while samba was actually in use from one of the client machines, judging from the times.) Even if the binary package was ok, it's possible my build environment introduced a problem. I built the packages on the same fully up-to-date Lenny that I'm running Samba on, though. Using dpkg-buildpackage -us -uc -b, IIRC. It's possible I would have seen those samba panics if I'd just compiled from source without applying the patch. My machine doesn't have enough memory to test for long with the leak happening, so I can't think of any great solutions. I can install the -dbg package, though, and maybe get a useful backtrace. Packages with the memory leak fix have been uploaded to stable-proposed-updates yesterday. See http://wiki.debian.org/StableProposedUpdates for instructions about how to use it, but please be aware that such packages have not been officially accepted in stable, yet. They are planned for the next stable point release. signature.asc Description: Digital signature
Bug#520794: [Pkg-samba-maint] Bug#520794: samba: smbd memory usage always increase - memory leak
Christian Perrier bubu...@debian.org wrote: Quoting Peter Cordes (pe...@cordes.ca): I compiled Samba packages for myself with bug_520794.patch (thanks Christian). I'll keep an eye on it to see if it's still leaking. ping me in a week if I forget to update this. Any news about this ? I wasn't subscribed to the bug (I thought that would happen automatically after sending mail to n...@b.d.o), so I didn't get that email. I have subscribed now, though, so you can just reply to the bug. It doesn't leak memory anymore, but I'm seeing smbd panics every few days. Here's a sample from my /var/log/samba/log.smbd: [2009/04/29 17:38:20, 0] printing/print_cups.c:cups_connect(68) Unable to connect to CUPS server localhost:631 - Connection refused (these CUPS messages happen at this frequency all the time. I guess I should turn that off whatever wants it, since I don't run CUPS on this machine.) [2009/04/29 17:38:20, 0] printing/print_cups.c:cups_connect(68) Unable to connect to CUPS server localhost:631 - Connection refused [2009/04/29 17:50:29, 0] printing/print_cups.c:cups_connect(68) Unable to connect to CUPS server localhost:631 - Connection refused [2009/04/29 17:50:29, 0] printing/print_cups.c:cups_connect(68) Unable to connect to CUPS server localhost:631 - Connection refused [2009/04/29 18:03:17, 0] printing/print_cups.c:cups_connect(68) Unable to connect to CUPS server localhost:631 - Connection refused [2009/04/29 18:03:17, 0] printing/print_cups.c:cups_connect(68) Unable to connect to CUPS server localhost:631 - Connection refused [2009/04/29 18:06:28, 0] lib/util_tdb.c:tdb_wrap_log(886) tdb(/var/lib/samba/passdb.tdb): tdb_transaction_commit: failed to setup recovery data [2009/04/29 18:06:28, 0] passdb/pdb_tdb.c:tdb_update_sam(1193) Could not commit transaction [2009/04/29 18:06:28, 0] lib/util.c:smb_panic(1663) PANIC (pid 14712): transaction_cancel failed [2009/04/29 18:06:28, 0] lib/util.c:log_stack_trace(1767) BACKTRACE: 0 stack frames: [2009/04/29 18:06:28, 0] lib/util.c:smb_panic(1668) smb_panic(): calling panic action [/usr/share/samba/panic-action 14712] [2009/04/29 18:06:36, 0] lib/util.c:smb_panic(1676) smb_panic(): action returned status 0 [2009/04/29 18:06:36, 0] lib/fault.c:dump_core(201) dumping core in /var/log/samba/cores/smbd [2009/04/29 18:06:38, 0] printing/print_cups.c:cups_connect(68) All the Samba crashes generate the same series of log file lines. all with 0 stack frames of backtrace :( The panic-action script sends email with this backtrace (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) [Thread debugging using libthread_db enabled] [New Thread 0xb79f06d0 (LWP 21277)] (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) 0xb7eeb424 in __kernel_vsyscall () #0 0xb7eeb424 in __kernel_vsyscall () #1 0xb7b2d6f3 in waitpid () from /lib/i686/cmov/libc.so.6 #2 0xb7acb46b in ?? () from /lib/i686/cmov/libc.so.6 #3 0xb7cb74ad in system () from /lib/i686/cmov/libpthread.so.0 #4 0x08201ef9 in smb_panic () #5 0x081d3a5b in ?? () #6 0x084f2c97 in ?? () #7 0x0008 in ?? () #8 0x0852d3b9 in ?? () #9 0x0852dceb in ?? () #10 0x04a9 in ?? () #11 0x1db3b1f8 in ?? () #12 0xbfc055a8 in ?? () #13 0x081dfc3b in memcache_flush () #14 0x081d3a85 in ?? () #15 0x0003 in ?? () #16 0x091f9ad0 in ?? () #17 0xbfc055c8 in ?? () #18 0x081b46af in pdb_update_sam_account () Backtrace stopped: frame did not save the PC My smb.conf is not very modern either. It still uses security=share, because I set it up that way 6 years ago, and haven't wanted to re-do my Windows home network since then. (file attached) It's not the parent daemon that panics, and Windows clients seem to reconnect ok. Anyway, my users haven't noticed any actual service interruptions. -- #define X(x,y) x##y Peter Cordes ; e-mail: X(pe...@cor , des.ca) The gods confound the man who first found out how to distinguish the hours! Confound him, too, who in this place set up a sundial, to cut and hack my day so wretchedly into small pieces! -- Plautus, 200 BC [global] # Do something
Bug#520794: [Pkg-samba-maint] Bug#520794: Bug#520794: samba: smbd memory usage always increase - memory leak
Quoting Peter Cordes (pe...@cordes.ca): Christian Perrier bubu...@debian.org wrote: Quoting Peter Cordes (pe...@cordes.ca): I compiled Samba packages for myself with bug_520794.patch (thanks Christian). I'll keep an eye on it to see if it's still leaking. ping me in a week if I forget to update this. Any news about this ? I wasn't subscribed to the bug (I thought that would happen automatically after sending mail to n...@b.d.o), so I didn't get that email. I have subscribed now, though, so you can just reply to the bug. It doesn't leak memory anymore, but I'm seeing smbd panics every few days. Well, ae you sure that this is related to the memory leak fix ? signature.asc Description: Digital signature
Bug#520794: [Pkg-samba-maint] Bug#520794: samba: smbd memory usage always increase - memory leak
Quoting Massimiliano Ferrero (m.ferr...@midhgard.it): Christian Perrier ha scritto: Quoting Peter Cordes (pe...@cordes.ca): I compiled Samba packages for myself with bug_520794.patch (thanks Christian). I'll keep an eye on it to see if it's still leaking. ping me in a week if I forget to update this. Any news about this ? I sent you a mail on 8/4, did you received that one? There was a memory It might have fallen up in s spam trap. Sorry for this. graph attached For me the patch has resolved the issue and there is no sign of instability or any regression. I think the patch should be included in the next secutiry update. Could you resend the mail with the graph to the BTS: 520...@bugs.debian.org ? Many thanks in advance. signature.asc Description: Digital signature
Bug#520794: [Pkg-samba-maint] Bug#520794: samba: smbd memory usage always increase - memory leak
Quoting Peter Cordes (pe...@cordes.ca): I was seeing memory leaks in Samba on my home server. (simple setup with 1 win2k client.) I was wondering why my machine was feeling slow, and I saw that smbd was using all my RAM: 300MB RSS, 600MB virtual size, on my 512MB PIII 500MHz. And my small swap partition was mostly full. This is after only a week of having Samba running, and this machine normally goes for months without reboots. This is a showstopper bug for me, so I'm heavily in favour of updating stable. Are there configurations where it doesn't leak, or are modern servers supposed to have so much RAM and swap that it doesn't matter? The latter is IMHO not a good enough argument to justify leaving it unfixed. I compiled Samba packages for myself with bug_520794.patch (thanks Christian). I'll keep an eye on it to see if it's still leaking. ping me in a week if I forget to update this. Any news about this ? signature.asc Description: Digital signature
Bug#520794: [Pkg-samba-maint] Bug#520794: samba: smbd memory usage always increase - memory leak
On Mon, Mar 23, 2009 at 06:26:49PM +0100, Christian Perrier wrote: Quoting Massimiliano Ferrero (m.ferr...@midhgard.it): if you think it's necessary in the next days I could compile a modified 3.2.5-4 with Volker patch, test on this customer and verify that the problem disappears. That would be even more convincing that this bug is worth fixing *and* not risky, which is important to decide whether or not we fix it for lenny (this is a matter of interpretation whether it's important enough to warrant an update to lenny and the (always non null though apparently very very low) risk that we could introduce other bugs with that fix. I was seeing memory leaks in Samba on my home server. (simple setup with 1 win2k client.) I was wondering why my machine was feeling slow, and I saw that smbd was using all my RAM: 300MB RSS, 600MB virtual size, on my 512MB PIII 500MHz. And my small swap partition was mostly full. This is after only a week of having Samba running, and this machine normally goes for months without reboots. This is a showstopper bug for me, so I'm heavily in favour of updating stable. Are there configurations where it doesn't leak, or are modern servers supposed to have so much RAM and swap that it doesn't matter? The latter is IMHO not a good enough argument to justify leaving it unfixed. I compiled Samba packages for myself with bug_520794.patch (thanks Christian). I'll keep an eye on it to see if it's still leaking. ping me in a week if I forget to update this. -- #define X(x,y) x##y Peter Cordes ; e-mail: X(pe...@cor , des.ca) The gods confound the man who first found out how to distinguish the hours! Confound him, too, who in this place set up a sundial, to cut and hack my day so wretchedly into small pieces! -- Plautus, 200 BC signature.asc Description: Digital signature
Bug#520794: [Pkg-samba-maint] Bug#520794: samba: smbd memory usage always increase - memory leak
Quoting Peter Cordes (pe...@cordes.ca): Are there configurations where it doesn't leak, or are modern servers supposed to have so much RAM and swap that it doesn't matter? The At least on my home server, it doesn't leak (genuine lenny server with samba 2:3.2.5-4lenny2) latter is IMHO not a good enough argument to justify leaving it unfixed. Of course. I compiled Samba packages for myself with bug_520794.patch (thanks Christian). I'll keep an eye on it to see if it's still leaking. ping me in a week if I forget to update this. Yes. Having confirmation that this patch fixes the memory leak would definitely be a good argument to include it in an update for stable. signature.asc Description: Digital signature
Bug#520794: [Pkg-samba-maint] Bug#520794: samba: smbd memory usage always increase - memory leak
tags 520794 patch
thanks
Quoting Volker Lendecke (volker.lende...@sernet.de):
On Mon, Mar 23, 2009 at 07:10:51AM +0100, Christian Perrier wrote:
Volker (Lendecke, CC'ed), you were the one fixing this. Do you have
the patch you used handy somewhere so that we see if we can backport
it to Debian's 3.2.5?
http://git.samba.org/?p=samba.git;a=commitdiff;h=cc70e0f88328e3641008
OK, thanks Volker.
For reference in the Debian BTS, here's the patch that applies to
3.2.5 and could then be applied in the next point release.
Steve, any objection to include that one in the next point release?
Goal: fix memory leak in vfs_full_audit
Fixes: #520794
Status wrt upstream: Fixed in 3.2.8
Author: Volker Lendecke v...@samba.org
Index: lenny/source/modules/vfs_full_audit.c
===
--- lenny.orig/source/modules/vfs_full_audit.c
+++ lenny/source/modules/vfs_full_audit.c
@@ -701,6 +701,7 @@
static char *audit_prefix(TALLOC_CTX *ctx, connection_struct *conn)
{
char *prefix = NULL;
+ char *result;
prefix = talloc_strdup(ctx,
lp_parm_const_string(SNUM(conn), full_audit,
@@ -708,12 +709,14 @@
if (!prefix) {
return NULL;
}
- return talloc_sub_advanced(ctx,
+ result = talloc_sub_advanced(ctx,
lp_servicename(SNUM(conn)), conn-user,
conn-connectpath, conn-gid,
get_current_username(),
current_user_info.domain,
prefix);
+ TALLOC_FREE(prefix);
+ return result;
}
static bool log_success(vfs_handle_struct *handle, vfs_op_type op)
signature.asc
Description: Digital signature
Bug#520794: [Pkg-samba-maint] Bug#520794: samba: smbd memory usage always increase - memory leak
On Mon, Mar 23, 2009 at 11:22:10AM +0100, Christian Perrier wrote: tags 520794 patch thanks Quoting Volker Lendecke (volker.lende...@sernet.de): On Mon, Mar 23, 2009 at 07:10:51AM +0100, Christian Perrier wrote: Volker (Lendecke, CC'ed), you were the one fixing this. Do you have the patch you used handy somewhere so that we see if we can backport it to Debian's 3.2.5? http://git.samba.org/?p=samba.git;a=commitdiff;h=cc70e0f88328e3641008 OK, thanks Volker. For reference in the Debian BTS, here's the patch that applies to 3.2.5 and could then be applied in the next point release. Steve, any objection to include that one in the next point release? I wouldn't have considered this critical enough to include in a stable update, but if you want to do the work, I have no objections. Cheers, -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. Ubuntu Developerhttp://www.debian.org/ slanga...@ubuntu.com vor...@debian.org -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#520794: [Pkg-samba-maint] Bug#520794: samba: smbd memory usage always increase - memory leak
Quoting Massimiliano Ferrero (m.ferr...@midhgard.it): if you think it's necessary in the next days I could compile a modified 3.2.5-4 with Volker patch, test on this customer and verify that the problem disappears. That would be even more convincing that this bug is worth fixing *and* not risky, which is important to decide whether or not we fix it for lenny (this is a matter of interpretation whether it's important enough to warrant an update to lenny and the (always non null though apparently very very low) risk that we could introduce other bugs with that fix. -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#520794: samba: smbd memory usage always increase - memory leak
Package: samba Version: 2:3.2.5-4 Severity: important Short description: after upgrade from etch to lenny we are experiencing a memory leak in smbd processes. We have a production system made of two samba server, pdc and bdc for a domain; user backend is in openldap, managed through ldap-account-manager The pdc acts as file server for about 50 clients, clients are mixed operating systems: there are still about 10 Win95/98, about 35 Windows XP, one Windows 2000 server and three Windows 2003 servers, almost all into the domain. Until two weeks ago the two samba server were on debian etch and the system was stable, no problems reported, we had upgraded to etch around september 2007, so the system has been stable for at least one and a half year. We use munin to monitor server memory and other counters: we discovered that since samba upgrade to 3.2.5 memory usage has started to increase linearly. Before memory usage was constant. There is one pc with an application active 24/7 and the smbd correspondig to this process is using about 350 MB ram and increasing ps auxn|grep smbd 1000 7296 0.0 0.0 3144 764 pts/1S+ 20:04 0:00 grep smbd 0 8528 0.0 0.3 16668 3772 ?Ss Mar18 0:19 /usr/sbin/smbd -D 0 8534 0.0 0.2 16360 2084 ?SMar18 0:01 /usr/sbin/smbd -D 1000 8535 2.7 33.7 362000 349932 ? SMar18 191:14 /usr/sbin/smbd -D 0 9251 0.0 0.4 17616 4176 ?SMar18 0:04 /usr/sbin/smbd -D 0 9415 0.0 1.3 25740 14252 ?SMar18 4:47 /usr/sbin/smbd -D 10024 13280 0.0 0.6 18520 6992 ?SMar20 0:07 /usr/sbin/smbd -D 0 16733 0.0 0.5 18252 5864 ?SMar18 0:10 /usr/sbin/smbd -D 0 23226 0.0 0.6 18376 6596 ?SMar18 0:07 /usr/sbin/smbd -D The fourth process is the one that is using 350 MB ram The process has used 191 minutes of cpu, this is consistent with the client application being active for about one week without shutdown. I suppose we do not see other processes in such a situation because there are no other clients running 24 hours a day. We are using vfs_full_audit, I read in mainstream samba changelog for 3.2.8: * Fix a bad memleak in vfs_full_audit. Has this bug been fixed in 3.2.5-4 for lenny? Could we be experiencing this bug? If so is it possible to backport the relevant patch to lenny source? If it is necessary to verify if the bug is this one I can disable vfs_full_audit and run the server in this condition for a week. Thanks for your help Best regards Massimiliano Ferrero Here is /etc/samba/smb.conf [global] workgroup = DOMAIN netbios name = PDC server string = %h (Linux Server) encrypt passwords = true lanman auth = Yes passdb backend = ldapsam:ldap://127.0.0.1/ ldap admin dn = cn=admin,dc=domain,dc=com ldap suffix = dc=domain,dc=com ldap group suffix = ou=Groups ldap user suffix = ou=Users ldap machine suffix = ou=Computers ldap ssl = No ldap passwd sync = Yes unix password sync = Yes passwd program = /usr/bin/passwd %u add machine script = /usr/local/sbin/smbldap-useradd -w %u add user script = /usr/local/sbin/smbldap-useradd -m %u ldap delete dn = Yes #delete user script = /usr/local/sbin/smbldap-userdel %u add machine script = /usr/local/sbin/smbldap-useradd -w %u add group script = /usr/local/sbin/smbldap-groupadd -p %g #delete group script = /usr/local/sbin/smbldap-groupdel %g add user to group script = /usr/local/sbin/smbldap-groupmod -m %u %g delete user from group script = /usr/local/sbin/smbldap-groupmod -x %u %g set primary group script = /usr/local/sbin/smbldap-usermod -g %g %u Dos charset = 850 Unix charset = ISO8859-1 syslog = 0 log level = 0 log file = /var/log/samba/log.client max log size = 1000 unix extensions = Yes logon script = scripts\logon.bat logon path = preferred master = Yes domain master = Yes domain logons = Yes os level = 85 dns proxy = No wins support = Yes vfs objects = full_audit full_audit:prefix = %u|%I full_audit:success = chdir chmod chmod_acl chown close connect disconnect fchmod fchmod_acl fchown mkdir open opendir read rename rmdir sendfile unlink write
