On Thu, Oct 19, 2017 at 10:43:19AM -0500, Mark Guz wrote: > Hi, > > I'm running a PoC orangefs setup, and have read through all of the > previous posts on performance. We are running 2.9.6 on Redhat 7.4. The > clients are using the kernel interface > > I'm running currently on 1 Power 750 server as host (with 8 dual meta/data > servers running). The clients are a mix of Intel and PPC64 systems all > interconnected by Infiniband DDR cards in Connected mode. > > The storage backend is a 4G FC attached ssd chassis with 20 250 Gig SSD > cards (not regular drives) in, with 8 are assigned to meta and 8 are > assigned to data > > The network tests good. 7ish Gb/s with no retries or errors. We are using > bmi_tcp. > > I can get great performance for large files as expected but when > performing small file actions the performance is significantly poorer. > > For example. I can untar linux-4.13.3.tar.xz locally on the filesystem > in 14seconds while on the orangefs it takes 10mins > > I can see the performance difference when playing with stripe sizes etc > when copying monolithic files, but there seems to be a wall that gets hit > when there is a lot of metadata activity. > > I can see how the need for network back and forth could impact performance > but is it reasonable to see a 42x performance drop in such cases? > > Also if I then try and compile the kernel on the orangefs it takes well > over 2hours and most of the time is spent io waiting. Compiling locally > takes about 20mins on the same server. > > I've tried running over multiple hosts, running some meta only and some > data only servers, i've tried running with just 1 meta and 1 data server. > I've applied all the system level optimizations I've found and just cannot > reasonably speed up the untar operation. > > Maybe it's a client thing? there's not much i can see that's configurable > about the client though. > > It seems to me that i should be able to get better performance, even on > small file operations, but I'm kinda stumped. > > Am I just chasing unicorns or is it possible to get usable performance for > this sort of file activity? (untaring, compiling etc etc)
There's no magic switch that'll make it just as performant as a local filesystem, but have you tried running the newer kernel? RHEL 7 uses kernel 3.10. OrangeFS was imported upstream in kernel 4.6. The newest kernel 4.13 incorporates many client-side caching improvements (and more are in the pipeline), which should be a big work on metadata-heavy workloads. With the upstream kernel module, it is not necessary to build the kernel (--with-kernel, make kmod, make kmod_install) module within the OrangeFS tree. The CONFIG_ORANGEFS_FS option must be set to m or y in the kernel .config file. Also see https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/filesystems/orangefs.txt for more details. There are many improvements in the upstream code, but notably, there is now a kernel-side attribute cache. There kernel-side caches are set to 50 ms by default, but may be configured by echo 100 > /sys/fs/orangefs/getattr_timeout_msecs echo 100 > /sys/fs/orangefs/dcache_timeout_msecs where 100 means 100 ms. However certain operations will require a round-trip to the server regardless of the cache time. There is also the acache (attribute cache) which operates within the client-core. The advantage of the kernel caches is that they do not even require a context-switch into the client-core. Martin _______________________________________________ Pvfs2-users mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
