On 2010-08-26, at 18:42, Jagga Soorma wrote: > I am still running into this issue on some nodes: > > client109: ll_obdo_cache 0 152914489 208 19 1 : tunables > 120 60 8 : slabdata 0 8048131 0 > client102: ll_obdo_cache 0 308526883 208 19 1 : tunables > 120 60 8 : slabdata 0 16238257 0 > > How can I calculate how much memory this is holding on to.
If you do "head -1 /proc/slabinfo" it reports the column descriptions. The "slabdata" will section reports numslabs=16238257, and pagesperslab=1, so tis is 16238257 pages of memory, or about 64GB of RAM on client102. Ouch. > My system shows a lot of memory that is being used up but none of the jobs > are using that much memory. Also, these clients are running a smp sles 11 > kernel but I can't find any /sys/kernel/slab directory. > > Linux client102 2.6.27.29-0.1-default #1 SMP 2009-08-15 17:53:59 +0200 x86_64 > x86_64 x86_64 GNU/Linux > > What makes you say that this does not look like a lustre memory leak? I > thought all the ll_* objects in slabinfo are lustre related? It's true that the ll_obdo_cache objects are allocated by Lustre, but the above data shows 0 of those objects in use, so the kernel _should_ be freeing the unused slab objects. This particular data type (obdo) is only ever in use temporarily during system calls on the client, and should never be allocated for a long time. For some reason the kernel is not freeing the empty slab pages. That is the responsibility of the kernel, and not Lustre. > To me it looks like lustre is holding on to this memory but I don't know > much about lustre internals. > > Also, memused on these systems are: > > client102: 2353666940 > client109: 2421645924 This shows that Lustre is actively using about 2.4GB of memory allocations. It is not tracking the 64GB of memory in the obdo_cache slab, because it has freed that memory (even though the kernel has not freed those pages). > Any help would be greatly appreciated. The only suggestion I have is that if you unmount Lustre and unload the modules (lustre_rmmod) it will free up this memory. Otherwise, searching for problems with the slab cache on this kernel may turn up something. > On Wed, May 19, 2010 at 8:08 AM, Dmitry Zogin <dmitry.zogu...@oracle.com> > wrote: > Hello Jagga, > > I checked the data, and indeed this does not look like a lustre memory leak, > rather than a slab fragmentation, which assumes there might be a kernel issue > here. From the slabinfo (I only keep three first columns here): > > > name <active_objs> <num_objs> > ll_obdo_cache 0 452282156 208 > > means that there are no active objects, but the memory pages are not released > back from slab allocator to the free pool (the num value is huge). That looks > like a slab fragmentation - you can get more description at > http://kerneltrap.org/Linux/Slab_Defragmentation > > Checking your mails, I wonder if this only happens on clients which have > SLES11 installed? As the RAM size is around 192Gb, I assume they are NUMA > systems? > If so, SLES11 has defrag_ratio tunables in /sys/kernel/slab/xxx > From the source of get_any_partial() > > #ifdef CONFIG_NUMA > > /* > * The defrag ratio allows a configuration of the tradeoffs between > * inter node defragmentation and node local allocations. A lower > * defrag_ratio increases the tendency to do local allocations > * instead of attempting to obtain partial slabs from other nodes. > * > * If the defrag_ratio is set to 0 then kmalloc() always > * returns node local objects. If the ratio is higher then kmalloc() > * may return off node objects because partial slabs are obtained > * from other nodes and filled up. > * > * If /sys/kernel/slab/xx/defrag_ratio is set to 100 (which makes > * defrag_ratio = 1000) then every (well almost) allocation will > * first attempt to defrag slab caches on other nodes. This means > * scanning over all nodes to look for partial slabs which may be > * expensive if we do it every time we are trying to find a slab > * with available objects. > */ > > Could you please verify that your clients have defrag_ratio tunable and try > to use various values? > It looks like the value of 100 should be the best, unless there is a bug, > then may be even 0 gets the desired result? > > Best regards, > Dmitry > > > Jagga Soorma wrote: >> Hi Johann, >> >> I am actually using 1.8.1 and not 1.8.2: >> >> # rpm -qa | grep -i lustre >> lustre-client-1.8.1.1-2.6.27.29_0.1_lustre.1.8.1.1_default >> lustre-client-modules-1.8.1.1-2.6.27.29_0.1_lustre.1.8.1.1_default >> >> My kernel version on the SLES 11 clients is: >> # uname -r >> 2.6.27.29-0.1-default >> >> My kernel version on the RHEL 5.3 mds/oss servers is: >> # uname -r >> 2.6.18-128.7.1.el5_lustre.1.8.1.1 >> >> Please let me know if you need any further information. I am still trying >> to get the user to help me run his app so that I can run the leak finder >> script to capture more information. >> >> Regards, >> -Simran >> >> On Tue, Apr 27, 2010 at 7:20 AM, Johann Lombardi <joh...@sun.com> wrote: >> Hi, >> >> On Tue, Apr 20, 2010 at 09:08:25AM -0700, Jagga Soorma wrote: >> > Thanks for your response.* I will try to run the leak-finder script and >> > hopefully it will point us in the right direction.* This only seems to be >> > happening on some of my clients: >> >> Could you please tell us what kernel you use on the client side? >> >> > client104: ll_obdo_cache********* 0 433506280*** 208** 19*** 1 : >> > tunables* >> > 120** 60*** 8 : slabdata***** 0 22816120***** 0 >> > client116: ll_obdo_cache********* 0 457366746*** 208** 19*** 1 : >> > tunables* >> > 120** 60*** 8 : slabdata***** 0 24071934***** 0 >> > client113: ll_obdo_cache********* 0 456778867*** 208** 19*** 1 : >> > tunables* >> > 120** 60*** 8 : slabdata***** 0 24040993***** 0 >> > client106: ll_obdo_cache********* 0 456372267*** 208** 19*** 1 : >> > tunables* >> > 120** 60*** 8 : slabdata***** 0 24019593***** 0 >> > client115: ll_obdo_cache********* 0 449929310*** 208** 19*** 1 : >> > tunables* >> > 120** 60*** 8 : slabdata***** 0 23680490***** 0 >> > client101: ll_obdo_cache********* 0 454318101*** 208** 19*** 1 : >> > tunables* >> > 120** 60*** 8 : slabdata***** 0 23911479***** 0 >> > -- >> > >> > Hopefully this should help.* Not sure which application might be causing >> > the leaks.* Currently R is the only app that users seem to be using >> > heavily on these clients.* Will let you know what I find. >> >> Tommi Tervo has filed a bugzilla ticket for this issue, see >> https://bugzilla.lustre.org/show_bug.cgi?id=22701 >> >> Could you please add a comment to this ticket to describe the >> behavior of the application "R" (fork many threads, write to >> many files, use direct i/o, ...)? >> >> Cheers, >> Johann >> >> >> _______________________________________________ >> Lustre-discuss mailing list >> >> Lustre-discuss@lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-discuss >> >> >> > > Cheers, Andreas -- Andreas Dilger Lustre Technical Lead Oracle Corporation Canada Inc. _______________________________________________ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss