Hello all, I'm setting up GlusterFS on 2 hw w/ same configuration, 8 hdds. This deployment will grow later on.
I'm undecided between these different configurations and am seeing comments or advice from more experienced users of GlusterFS. Here is the summary of 3 options: 1. 1 brick per host, Gluster "distributed" volumes, internal redundancy at brick level 2. 1 brick per drive, Gluster "distributed replicated" volumes, no internal redundancy 3. 1 brick per host, Gluster "distributed replicated" volumes, no internal redundancy I don't know how the performance of these different configurations would compare. The workload is HPC and ML-training, data-heavy, metadata-light. # 1 brick per host, simplified cluster management, higher blast-radius having 1 brick per host (/data/bricks/hdd0) where each brick is a ZFS raid10 of 8 hdd. Pros: * I know ZFS raid10 performs very well. * simpler management of Gluster at the Host-brick level. * using Gluster in "distributed" mode, no replication (is this a pro?) * don't need to worry about GlusterFS performance with "distributed replicated" * future hosts can have raidz2 + hot spares or whatnot Cons: * large blast radius, if a zfs volume goes bad or node goes bad, I lose data. * not using "distributed replicated" (is this a con?) * I can't use hosts without internal redundancy later on? # 1 brick per hard disk, fine grained device management on Gluster, smaller blast-radius. Having 1 brick per drive (/data/bricks/hddN for 1 to X drives on box), each brick would still use ZFS. Pros: * 1 drive blast radius, the ideal. * GlusterFS w/ distributed replicated * no complicated host-fault management or runbook, I can use hosts with low availability Cons: * distributed replicated performance vs zfs raid10 * managing on gluster at the disk level can be more time consuming * managing disk spares and replacements w/ gluster # 1 brick per host, coarse grained brick management on Gluster, Gluster for replication. having 1 brick per host (/data/bricks/hdd0) where each brick is a ZFS raid0 of 8 hdd. Pros: * I know ZFS raid0 performs very well. * simpler management of Gluster at the Host-brick level. * using Gluster for redundancy (is this a pro?) * system can use lower redundancy hosts Cons: * large blast radius, replicating a whole host takes longer. Any comments or advice is welcome Thanks in advance, -- Miguel Filipe
________ Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users