What is the underlying disk, did that hardware/RAID config change when you switched hardware? The 'still busy' message is a bug, may be fixed in 1.8.5 cliffw
On Sat, Apr 2, 2011 at 1:01 AM, Thomas Roth <t.r...@gsi.de> wrote: > Hi all, > > we are suffering from a sever metadata performance degradation on our 1.8.4 > cluster and are pretty clueless. > - We moved the MDT to a new hardware, since the old one was failing > - We increased the size of the MDT with 'resize2fs' (+ mounted it and saw > all the files) > - We found the performance of the new mds dreadful > - We restarted the MDT on the old hardware with the failed RAID controller > replaced, but without doing anything with OSS or clients > The machine crashed three minutes after recovery was over > - Moved back to the new hardware, but the system was now pretty messed up: > persistent "still busy with N RPCs" and some "going back to sleep" > messages (by the way, there is no way to find out what these RPCs are, and > how to kill them? Of course I wouldn't mind switching off some clients or > even rebooting some OSS if I only new which ones...) > - Shut down the entire cluster, writeconf, restart without any client > mounts - worked fine > - Mounted Lustre and tried to "ls" a directory with 100 files: takes > several minutes(!) > - Being patient and then trying the same on a second client: takes > msecs. > > I have done complete shutdowns before, lastly to upgrade from 1.6 to 1.8, > then without writeconf and without performance loss. Before to change the > IPs of all servers (moving into a subnet), with writeconf, but without > recollection of the metadata behavior afterwards. > It is clear that after writeconf some information has to be regenerated, > but this is really extreme - also normal? > > The MDT now behaves more like an xrootd master which makes first contact to > its file servers and has to read in the entire database (would be nice to > have in Lustre to regenerate the MDT in case of desaster ;-) ). > Which caches are being filled now when I ls through the cluster? May I > expect the MDT to explode once it has learned about a certain percentage of > the > system? ;-) I mean, we have 100 mio files now and the current MDT hardware > has just 32GB memory... > In any case this is not the Lustre behavior we are used to. > > Thanks for any hints, > Thomas > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss@lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss > -- cliffw Support Guy WhamCloud, Inc. www.whamcloud.com
_______________________________________________ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss