Re: [Gluster-users] GlusterFS 3.0.2 small file read performance benchmark
Another thing that makes me wonder is the read-subvolume setting: > volume afr > type cluster/replicate > ... > option read-subvolume node2 > ... > end-volume So even if we play around and set this to the local node or some remote node respectively, it won't gain any performance for small files. Looks like the whole bottleneck for small files is meta-data and the global namespace lookup. It would be really great if all of this could be cached within io-cache, only falling back to a namespace query (and probably locking) if something wants to write to the file, or if the result is longer than cache-timeout seconds in the cache. So even if the file has been renamed, is unlinked, has changed permissions / metadata - simply take the version of the io-cache until it's invalidated. At least that is what I would expect the io-cache to do. This will introduce a discrepancy between the cached file version and the real version in the global namespace, but isn't that what one would expect from caching...? Note that the cache-size was in all tests on all nodes 1024MB, and the whole set of test-data was ~240MB. Add some meta-data and it's probably at 250MB. In addition, cache-timeout was 60 seconds, while the whole test took around 40 seconds. So *all* of the read-only test could have been served completely by the io-cache... or am I mistaken here? I'm trying to understand the poor performance, because network latency should be eliminated by the cache. Could some Gluster-Dev please elaborate a bit on that one? Best Regards, John ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] GlusterFS 3.0.2 small file read performance benchmark
After reading the mail again I'm under the impression that I didn't make it clear enough: We don't have a pure read-only, but mostly read-only workload. This is the reason why we've tried GlusterFS with AFR, so we can have a multi-master read/write filesystem with a persitent copy on each node. If we wouldn't need write access every here and then, we could have gone with plain copies of the data. Now another idea is the following, based on the fact that the local ext4 filesystem + VFS cache is *much* faster: > GlusterFS with populated IO-Cache: > real0m38.576s > user0m3.356s > sys 0m6.076s # Work directly on the back-end (this is read-only...) $ cd /mnt/brick/test/glusterfs/data # Ext4 without VFS Cache: $ echo 3 > /proc/sys/vm/drop_caches $ for ((i=0;i<100;i++)); do tar cf - . > /dev/null & done; time wait real0m1.598s user0m2.136s sys 0m3.696s # Ext4 with VFS Cache: $ for ((i=0;i<100;i++)); do tar cf - . > /dev/null & done; time wait real0m1.312s user0m2.264s sys 0m3.256s So the idea now is to bind-mount the backend filesystem *read-only* and use it for all read operations. For all write operations, use the GlusterFS mountpoint which provides locking etc. (This implies some sort of Read/Write splitting, but we can do that...) The downside is that the backend read operations won't make use of the GlusterFS on-demand self-healing. But since 99% of our read-only files are "write once, read a lot of times..." -- this could work out. After a node failure, a simple "ls -lR" should self-heal everything and the backend is fine too. The chance to read a broken file is very low? Any comments on this idea? Is there something else that could go wrong by using the backend in a pure read-only fashion that I've missed? Any ideas why the GlusterFS performance/io-cache translator with a cache-timeout of 60 is still so slow? Is there any way to *really* cache meta and filedata on GlusterFS _without_ hitting the network and thus getting very poor small file performance introduced by network latency? Are there any plans to implement support for FS-Cache [1] (CacheFS, Cachefiles), shipped with recent Linux kernels? Or to improve io-cache likewise? [1] http://people.redhat.com/steved/fscache/docs/FS-Cache.pdf Lots of questions... :) Best regards, John ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] GlusterFS 3.0.2 small file read performance benchmark
> Some final words: > > - don't add performance translators blindly (!) > - always test with a similar workload you will use in production > - never go and copy+paste a volume spec, then moan about bad performance > - don't rely on "glusterfs-volgen", it gives you just a starting point! > - less translators == less overhead > - read documentation for all options of all translators and get an idea: > http://www.gluster.com/community/documentation/index.php/Translators > (some stuff is still undocumented, but this is open source... so have a > look) > Thanks to share your results. During my tests I found somewhat similar, although I didn't went into details like you with GlusterFS, I was more interested in tunning my RAID controller and ext3 filesystem. Also note - has been reported on the list before - that with the default, glusterfs-volgen created configuration 'ls -lR' does NOT re-sync the content between AFR nodes (at least not with CentOS 5.4 x86_64). 'ls -lR' works without the translators. Seems for me it's safer and faster to use a stripped down config. "Your mileage may vary" ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] GlusterFS 3.0.2 small file read performance benchmark
Greetings, in contrast to some performance tips regarding small file *read* performance, I want to share these results. The test is rather simple but yields some very remarkable results: 400% improved read performance by simply dropping some of the so called "performance translators"! Please note that this test resembles a simplified version of our workload, which is more or less sequential, read-only small file serving with an average of 100 concurrent clients. (We use GlusterFS as a flat-file backend to a cluster of webservers, which is hit only after missing some caches in a more sophisticated caching infrastructure on top of it) The test-setup is a 3 node AFR cluster, with server+client on each one, single process model (one volfile, the local volume is attached to within the same process to save overhead), connected via 1 Gbit Ethernet. This way each node can continue to operate on it's own, even if the whole internal network for GlusterFS is down. We used commodity hardware for the test. Each node is identical: - Intel Core i7 - 12G RAM - 500GB filesystem - 1 Gbit NIC dedicated for GlusterFS Software: - Linux 2.6.32.8 - GlusterFS 3.0.2 - FUSE inited with protocol versions: glusterfs 7.13 kernel 7.13 - Filesystem / Storage Backend: - LVM2 on top of software RAID 1 - ext4 with noatime I will paste the configurations inline, so people can comment on them. /etc/fstab: - /dev/data/test /mnt/brick/test ext4noatime 0 2 /etc/glusterfs/test.vol /mnt/glusterfs/test glusterfs noauto,noatime,log-level=NORMAL,log-file=/var/log/glusterfs/test.log 0 0 - *** Please note: this is the final configuration with the best results. All translators are numbered to make the explanation easier later on. Unused translators are commented out... The volume spec is identical on all nodes, except that the bind-address option in the server volume [*4*] is adjusted. *** /etc/glusterfs/test.vol - # Sat Feb 27 16:53:00 CET 2010 John Feuerstein # # Single Process Model with AFR (Automatic File Replication). ## ## Storage backend ## # # POSIX STORAGE [*1*] # volume posix type storage/posix option directory /mnt/brick/test/glusterfs end-volume # # POSIX LOCKS [*2*] # #volume locks volume brick type features/locks subvolumes posix end-volume ## ## Performance translators (server side) ## # # IO-Threads [*3*] # #volume brick # type performance/io-threads # subvolumes locks # option thread-count 8 #end-volume ### End of performance translators # # TCP/IP server [*4*] # volume server type protocol/server subvolumes brick option transport-type tcp option transport.socket.bind-address 10.1.0.1 # FIXME option transport.socket.listen-port 820 option transport.socket.nodelay on option auth.addr.brick.allow 127.0.0.1,10.1.0.1,10.1.0.2,10.1.0.3 end-volume # # TCP/IP clients [*5*] # volume node1 type protocol/client option remote-subvolume brick option transport-type tcp/client option remote-host 10.1.0.1 option remote-port 820 option transport.socket.nodelay on end-volume volume node2 type protocol/client option remote-subvolume brick option transport-type tcp/client option remote-host 10.1.0.2 option remote-port 820 option transport.socket.nodelay on end-volume volume node3 type protocol/client option remote-subvolume brick option transport-type tcp/client option remote-host 10.1.0.3 option remote-port 820 option transport.socket.nodelay on end-volume # # Automatic File Replication Translator (AFR) [*6*] # # NOTE: "node3" is the primary metadata node, so this one *must* # be listed first in all volume specs! Also, node3 is the global # favorite-child with the definite file version if any conflict # arises while self-healing... # volume afr type cluster/replicate subvolumes node3 node1 node2 option read-subvolume node2 option favorite-child node3 end-volume ## ## Performance translators (client side) ## # # IO-Threads [*7*] # #volume client-threads-1 # type performance/io-threads # subvolumes afr # option thread-count 8 #end-volume # # Write-Behind [*8*] # volume wb type performance/write-behind subvolumes afr option cache-size 4MB end-volume # # Read-Ahead [*9*] # #volume ra # type performance/read-ahead # subvolumes wb # option page-count 2 #end-volume # # IO-Cache [*10*] # volume cache type performance/io-cache subvolumes wb option cache-size 1024MB option cache-timeout 60 end-volume # # Quick-Read for small files [*11*] # #volume qr # type performance/quick-read # subvolumes cache # option cache-timeout 60 #end-volume # # Metadata prefetch [*12*] # #volume sp # type performance/stat-prefetch # subvolumes qr #end-volume # # IO-Threads [*13*] # #volume client-threads-2