I would highly stress, regardless of whatever solution you choose - make sure you test actual workload performance before going all-in.
In my testing, performance (esp. iops and latency) decreased as I added bricks and additional nodes. Since you have many spindles now, I would encourage you to test your workload up to and including the total brick count you ultimately expect. RAID level and whether it’s md, zfs, or hardware isn’t likely to make as significant of a performance impact as Gluster and its various clients will. Test failure scenarios and performance characteristics during impairment events thoroughly. Make sure heals happen as you expect, including final contents of files modified during an impairment. If you have many small files or directories that will be accessed concurrently, make sure to stress that behavior in your testing. Gluster can be great for targeting availability and distribution at low software cost, and I would say as of today at the expense of performance, but as with any scale-out NAS there are limitations and some surprises along the path. Good hunting, -t > On Jul 4, 2016, at 10:44 AM, Gandalf Corvotempesta > <gandalf.corvotempe...@gmail.com> wrote: > > 2016-07-04 19:35 GMT+02:00 Russell Purinton <russell.purin...@gmail.com>: >> For 3 servers with 12 disks each, I would do Hardware RAID0 (or madam if you >> don’t have a RAID card) of 3 disks. So four 3-disk RAID0’s per server. > > 3 servers is just to start. We plan to use 5 server in shorter time > and up to 15 on production. > >> I would set them up as Replica 3 Arbiter 1 >> >> server1:/brickA server2:/brickC server3:/brickA >> server1:/brickB server2:/brickD server3:/brickB >> server2:/brickA server3:/brickC server1:/brickA >> server2:/brickB server3:/brickD server1:/brickB >> server3:/brickA server1:/brickC server2:/brickA >> server3:/brickB server1:/brickD server2:/brickB >> >> The benefit of this is that you can lose an entire server node (12 disks) >> and all of your data is still accessible. And you get the same space as if >> they were all in a RAID10. >> >> If you lose any disk, the entire 3 disk brick will need to be healed from >> the replica. I have 20GbE on each server so it doesn’t take long. It >> copied 20TB in about 18 hours once. > > So, any disk failure would me at least 6TB to be recovered via > network. This mean an high network utilization and as long gluster > doesn't have a dedicated network for replica, > this can slow down client access. > _______________________________________________ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users _______________________________________________ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
