I tried to change swapiness from 0 to 95 but it did not have any impact on the system overhead.
r. On Thursday 22. August 2013 15.38.37 Dragseth Roy Einar wrote: > No, I cannot detect any swap activity on the system. > > r. > > On Thursday 22. August 2013 09.21.33 you wrote: > > Is this slowdown due to increased swap activity? If "yes", then try > > lowering the "swappiness" value. This will sacrifice buffer cache space > > to > > lower swap activity. > > > > Take a look at http://en.wikipedia.org/wiki/Swappiness. > > > > Roger S. > > > > On 08/22/2013 08:51 AM, Roy Dragseth wrote: > > > We have just discovered that a large buffer cache generated from > > > traversing a lustre file system will cause a significant system overhead > > > for applications with high memory demands. We have seen a 50% slowdown > > > or worse for applications. Even High Performance Linpack, that have no > > > file IO whatsoever is affected. The only remedy seems to be to empty > > > the > > > buffer cache from memory by running "echo 3 > /proc/sys/vm/drop_caches" > > > > > > Any hints on how to improve the situation is greatly appreciated. > > > > > > > > > System setup: > > > Client: Dual socket Sandy Bridge, with 32GB ram and infiniband > > > connection > > > to lustre server. CentOS 6.4, with kernel 2.6.32-358.11.1.el6.x86_64 > > > and > > > lustre v2.1.6 rpms downloaded from whamcloud download site. > > > > > > Lustre: 1 MDS and 4 OSS running Lustre 2.1.3 (also from whamcloud site). > > > Each OSS has 12 OST, total 1.1 PB storage. > > > > > > How to reproduce: > > > > > > Traverse the lustre file system until the buffer cache is large enough. > > > In our case we run > > > > > > find . -print0 -type f | xargs -0 cat > /dev/null > > > > > > on the client until the buffer cache reaches ~15-20GB. (The lustre file > > > system has lots of small files so this takes up to an hour.) > > > > > > Kill the find process and start a single node parallel application, we > > > use > > > HPL (high performance linpack). We run on all 16 cores on the system > > > with 1GB ram per core (a normal run should complete in appr. 150 > > > seconds.) The system monitoring shows a 10-20% system cpu overhead and > > > the HPL run takes more than 200 secs. After running "echo 3 > > > > /proc/sys/vm/drop_caches" the system performance goes back to normal > > > with > > > a run time at 150 secs. > > > > > > I've created an infographic from our ganglia graphs for the above > > > scenario. > > > > > > https://dl.dropboxusercontent.com/u/23468442/misc/lustre_bc_overhead.png > > > > > > Attached is an excerpt from perf top indicating that the kernel routine > > > taking the most time is _spin_lock_irqsave if that means anything to > > > anyone. > > > > > > > > > Things tested: > > > > > > It does not seem to matter if we mount lustre over infiniband or > > > ethernet. > > > > > > Filling the buffer cache with files from an NFS filesystem does not > > > degrade > > > performance. > > > > > > Filling the buffer cache with one large file does not give degraded > > > performance. (tested with iozone) > > > > > > > > > Again, any hints on how to proceed is greatly appreciated. > > > > > > > > > Best regards, > > > Roy. > > > > > > > > > > > > _______________________________________________ > > > Lustre-discuss mailing list > > > Lustre-discuss@lists.lustre.org > > > http://lists.lustre.org/mailman/listinfo/lustre-discuss -- The Computer Center, University of Tromsø, N-9037 TROMSØ Norway. phone:+47 77 64 41 07, fax:+47 77 64 41 00 Roy Dragseth, Team Leader, High Performance Computing Direct call: +47 77 64 62 56. email: roy.drags...@uit.no _______________________________________________ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
