Do you have the Lustre read caching feature enabled? I think it should be on by default, but you might want to check. If the files are only 20 KB, then I would think the Lustre OSS nodes could keep them in memory most of the time to speed up access (unless of course this is a metadata bottleneck as Oliver suggested.) Do your OSS nodes have a lot of memory? Do you know what your typical memory usage is on the OSS nodes?
-- Rick Mohr Senior HPC System Administrator National Institute for Computational Sciences http://www.nics.tennessee.edu > On Jul 28, 2016, at 10:19 PM, Riccardo Veraldi > <riccardo.vera...@cnaf.infn.it> wrote: > > Hello, > > I have a lustre cluster on rhel7, 6 OSS each of them has 3 OSTs and 1 MDS. > > I am using lustre on ZFS. > While write performances are excellent also on smaller files, I find there is > a drop down in performance > on reading 20KB files. Performance can go as low as 200MB/sec or even less. > I am talking about random reads and random stride reads. > I did the following to try to improve things: > • disabled lnet debug messages: > • sysctl -w lnet.debug=0 > • increased dirty cache > • lctl set_param osc.lutrefs\*.max_dirty_mb=256 > • increased number of RPC in flight > • for i in `ls > /proc/fs/lustre/osc/lustrefs-OST00*/max_rpcs_in_flight`; do echo 32 > $i; done > it did not improve reading 20KB file performances. > I have to say in advance I did not set up any striping because I will have no > more than 6 concurrent reads and writes, > so striping is not that much important for me. > Here the problem is that one single random read of a 20KB file is around > 190MB/s and this is really disappointing. > I am open to any suggestion. > I made sure it is not a ZFS problem, on the OSSs ZFS is performing like a > charm locally. > thank you > > > Riccardo > > > _______________________________________________ > lustre-discuss mailing list > lustre-discuss@lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org _______________________________________________ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
