subject:"Re\: \[Gluster\-users\] Gluster Performance \- 12 Gbps SSDs and 10 Gbps NIC"

Re: [Gluster-users] Gluster Performance - 12 Gbps SSDs and 10 Gbps NIC

2023-12-14 Thread Gilberto Ferreira

Thanks for the advice.
---
Gilberto Nunes Ferreira
(47) 99676-7530 - Whatsapp / Telegram






Em qui., 14 de dez. de 2023 às 09:54, Strahil Nikolov 
escreveu:

> Hi Gilberto,
>
>
> Have you checked
> https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/chap-configuring_red_hat_storage_for_enhancing_performance
>  ?
>
> I think that you will need to test the virt profile as the settings will
> prevent some bad situations - especially VM live migration.
> You should also consider sharding which can reduce healing time but also
> makes your life more difficult if you need to access the disks of the VMs.
>
> I think that client.event-thread , server.event-thread and
> performance.io-thread-count can be tuned in your case. Consider setting ip
> a VM using the gluster volume as backing store and run the tests inside the
> VM to simulate real workload (best is to run a DB, webserver, etc inside
> a VM).
>
> Best Regards,
> Strahil Nikolov
>
>
>
>
>
>
>
>
> On Wednesday, December 13, 2023, 2:34 PM, Gilberto Ferreira <
> gilberto.nune...@gmail.com> wrote:
>
> Hi all
> Aravinda, usually I set this in two server env and never get split brain:
> gluster vol set VMS cluster.heal-timeout 5
> gluster vol heal VMS enable
> gluster vol set VMS cluster.quorum-reads false
> gluster vol set VMS cluster.quorum-count 1
> gluster vol set VMS network.ping-timeout 2
> gluster vol set VMS cluster.favorite-child-policy mtime
> gluster vol heal VMS granular-entry-heal enable
> gluster vol set VMS cluster.data-self-heal-algorithm full
> gluster vol set VMS features.shard on
>
> Strahil, in general, I get 0,06ms with 1G dedicated NIC.
> My env are very simple, using Proxmox + QEMU/KVM, with 3 or 5 VM.
>
>
> ---
> Gilberto Nunes Ferreira
> (47) 99676-7530 - Whatsapp / Telegram
>
>
>
>
>
>
> Em qua., 13 de dez. de 2023 às 06:08, Strahil Nikolov <
> hunter86...@yahoo.com> escreveu:
>
> Hi Aravinda,
>
> Based on the output it’s a ‘replica 3 arbiter 1’ type.
>
> Gilberto,
> What’s the latency between the nodes ?
>
> Best Regards,
> Strahil Nikolov
>
>
>
> On Wednesday, December 13, 2023, 7:36 AM, Aravinda 
> wrote:
>
> Only Replica 2 or Distributed Gluster volumes can be created with two
> servers. High chance of split brain with Replica 2 compared to Replica 3
> volume.
>
> For NFS Ganesha, no issue exporting the volume even if only one server is
> available. Run NFS Ganesha servers in Gluster server nodes and NFS clients
> from the network can connect to any NFS Ganesha server.
>
> You can use Haproxy + Keepalived (or any other load balancer) if high
> availability required for the NFS Ganesha connections (Ex: If a server node
> goes down, then nfs client can connect to other NFS ganesha server node).
>
> --
> Aravinda
> Kadalu Technologies
>
>
>
>  On Wed, 13 Dec 2023 01:42:11 +0530 *Gilberto Ferreira
> >* wrote ---
>
> Ah that's nice.
> Somebody knows this can be achieved with two servers?
>
> ---
> Gilberto Nunes Ferreira
> (47) 99676-7530 - Whatsapp / Telegram
>
>
>
>
>
>
>
> Em ter., 12 de dez. de 2023 às 17:08, Danny 
> escreveu:
>
> 
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://meet.google.com/cpu-eiue-hvk
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
> Wow, HUGE improvement with NFS-Ganesha!
>
>
> sudo dnf -y install glusterfs-ganesha
> sudo vim /etc/ganesha/ganesha.conf
>
> NFS_CORE_PARAM {
> mount_path_pseudo = true;
> Protocols = 3,4;
> }
> EXPORT_DEFAULTS {
> Access_Type = RW;
> }
>
> LOG {
> Default_Log_Level = WARN;
> }
>
> EXPORT{
> Export_Id = 1 ; # Export ID unique to each export
> Path = "/data"; # Path of the volume to be exported
>
> FSAL {
> name = GLUSTER;
> hostname = "localhost"; # IP of one of the nodes in the trusted
> pool
> volume = "data";# Volume name. Eg: "test_volume"
> }
>
> Access_type = RW;   # Access permissions
> Squash = No_root_squash;# To enable/disable root squashing
> Disable_ACL = TRUE; # To enable/disable ACL
> Pseudo = "/data";   # NFSv4 pseudo path for this export
> Protocols = "3","4" ;   # NFS protocols supported
> Transports = "UDP","TCP" ;  # Transport protocols supported
> SecType = "sys";# Security flavors supported
> }
>
>
> sudo systemctl enable --now nfs-ganesha
> sudo vim /etc/fstab
>
> localhost:/data /data nfs
> defaults,_netdev  0 0
>
>
> sudo systemctl daemon-reload
> sudo mount -a
>
> fio --name=test --filename=/data/wow --size=1G --readwrite=write
>
> Run status group 0 (all jobs):
>   WRITE: bw=2246MiB/s (2355MB/s), 2246MiB/s-2246MiB/s (2355MB/s-2355MB/s),
> io=1024MiB (1074MB), run=456-456msec
>
> Yeah 2355MB/s is much better than the original 115MB/s
>
> So in the end, I guess FUSE isn't the

Re: [Gluster-users] Gluster Performance - 12 Gbps SSDs and 10 Gbps NIC

2023-12-14 Thread Strahil Nikolov

Hi Gilberto,

Have you checked 
https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/chap-configuring_red_hat_storage_for_enhancing_performance
 ?
I think that you will need to test the virt profile as the settings will 
prevent some bad situations - especially VM live migration.You should also 
consider sharding which can reduce healing time but also makes your life more 
difficult if you need to access the disks of the VMs.
I think that client.event-thread , server.event-thread and 
performance.io-thread-count can be tuned in your case. Consider setting ip a VM 
using the gluster volume as backing store and run the tests inside the VM to 
simulate real workload (best is to run a DB, webserver, etc inside a VM).
Best Regards,Strahil Nikolov 








On Wednesday, December 13, 2023, 2:34 PM, Gilberto Ferreira 
 wrote:

Hi allAravinda, usually I set this in two server env and never get split 
brain:gluster vol set VMS cluster.heal-timeout 5
gluster vol heal VMS enable
gluster vol set VMS cluster.quorum-reads false
gluster vol set VMS cluster.quorum-count 1
gluster vol set VMS network.ping-timeout 2
gluster vol set VMS cluster.favorite-child-policy mtime
gluster vol heal VMS granular-entry-heal enable
gluster vol set VMS cluster.data-self-heal-algorithm full
gluster vol set VMS features.shard on
Strahil, in general, I get 0,06ms with 1G dedicated NIC.My env are very simple, 
using Proxmox + QEMU/KVM, with 3 or 5 VM.

---Gilberto Nunes Ferreira(47) 99676-7530 - Whatsapp / Telegram











Em qua., 13 de dez. de 2023 às 06:08, Strahil Nikolov  
escreveu:

Hi Aravinda,
Based on the output it’s a ‘replica 3 arbiter 1’ type.
Gilberto,What’s the latency between the nodes ?
Best Regards,Strahil Nikolov 




On Wednesday, December 13, 2023, 7:36 AM, Aravinda  wrote:

Only Replica 2 or Distributed Gluster volumes can be created with two servers. 
High chance of split brain with Replica 2 compared to Replica 3 volume.

For NFS Ganesha, no issue exporting the volume even if only one server is 
available. Run NFS Ganesha servers in Gluster server nodes and NFS clients from 
the network can connect to any NFS Ganesha server.

You can use Haproxy + Keepalived (or any other load balancer) if high 
availability required for the NFS Ganesha connections (Ex: If a server node 
goes down, then nfs client can connect to other NFS ganesha server node).

--Aravinda
Kadalu Technologies


 On Wed, 13 Dec 2023 01:42:11 +0530 Gilberto Ferreira 
 wrote ---


Ah that's nice.Somebody knows this can be achieved with two servers?

---
Gilberto Nunes Ferreira
(47) 99676-7530 - Whatsapp / Telegram












Em ter., 12 de dez. de 2023 às 17:08, Danny  
escreveu:


 
 
 
Community Meeting Calendar: 
 
Schedule - 
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC 
Bridge: https://meet.google.com/cpu-eiue-hvk 
Gluster-users mailing list 
Gluster-users@gluster.org 
https://lists.gluster.org/mailman/listinfo/gluster-users 

Wow, HUGE improvement with NFS-Ganesha!


sudo dnf -y install glusterfs-ganesha
sudo vim /etc/ganesha/ganesha.conf

NFS_CORE_PARAM {
mount_path_pseudo = true;
Protocols = 3,4;
}
EXPORT_DEFAULTS {
Access_Type = RW;
}
 
LOG {
Default_Log_Level = WARN;
}
 
EXPORT{
Export_Id = 1 ; # Export ID unique to each export
Path = "/data"; # Path of the volume to be exported
 
FSAL {
name = GLUSTER;
hostname = "localhost"; # IP of one of the nodes in the trusted pool
volume = "data";    # Volume name. Eg: "test_volume"
}
 
Access_type = RW;   # Access permissions
Squash = No_root_squash;    # To enable/disable root squashing
Disable_ACL = TRUE; # To enable/disable ACL
Pseudo = "/data";   # NFSv4 pseudo path for this export
Protocols = "3","4" ;   # NFS protocols supported
Transports = "UDP","TCP" ;  # Transport protocols supported
SecType = "sys";    # Security flavors supported
}


sudo systemctl enable --now nfs-ganesha
sudo vim /etc/fstab 

localhost:/data /data nfs    defaults,_netdev   
   0 0


sudo systemctl daemon-reload
sudo mount -a

fio --name=test --filename=/data/wow --size=1G --readwrite=write

Run status group 0 (all jobs):
  WRITE: bw=2246MiB/s (2355MB/s), 2246MiB/s-2246MiB/s (2355MB/s-2355MB/s), 
io=1024MiB (1074MB), run=456-456msec
Yeah 2355MB/s is much better than the original 115MB/s

So in the end, I guess FUSE isn't the best choice. 

On Tue, Dec 12, 2023 at 3:00 PM Gilberto Ferreira  
wrote:

Fuse there some overhead.Take a look at libgfapi:
https://staged-gluster-docs.readthedocs.io/en/release3.7.0beta1/Features/libgfapi/

I know this doc somehow is out of date, but could be a hint


---
Gilberto Nunes Ferreira
(47) 99676-7530 - Whatsapp / Telegram












Em ter., 12 de dez. de 2023 às 16:29, Danny  
escreveu:

Nope, not a caching thing. I've tried multiple different types of fio tests,

Re: [Gluster-users] Gluster Performance - 12 Gbps SSDs and 10 Gbps NIC

2023-12-13 Thread Gilberto Ferreira

Hi all
Aravinda, usually I set this in two server env and never get split brain:
gluster vol set VMS cluster.heal-timeout 5
gluster vol heal VMS enable
gluster vol set VMS cluster.quorum-reads false
gluster vol set VMS cluster.quorum-count 1
gluster vol set VMS network.ping-timeout 2
gluster vol set VMS cluster.favorite-child-policy mtime
gluster vol heal VMS granular-entry-heal enable
gluster vol set VMS cluster.data-self-heal-algorithm full
gluster vol set VMS features.shard on

Strahil, in general, I get 0,06ms with 1G dedicated NIC.
My env are very simple, using Proxmox + QEMU/KVM, with 3 or 5 VM.


---
Gilberto Nunes Ferreira
(47) 99676-7530 - Whatsapp / Telegram






Em qua., 13 de dez. de 2023 às 06:08, Strahil Nikolov 
escreveu:

> Hi Aravinda,
>
> Based on the output it’s a ‘replica 3 arbiter 1’ type.
>
> Gilberto,
> What’s the latency between the nodes ?
>
> Best Regards,
> Strahil Nikolov
>
>
>
> On Wednesday, December 13, 2023, 7:36 AM, Aravinda 
> wrote:
>
> Only Replica 2 or Distributed Gluster volumes can be created with two
> servers. High chance of split brain with Replica 2 compared to Replica 3
> volume.
>
> For NFS Ganesha, no issue exporting the volume even if only one server is
> available. Run NFS Ganesha servers in Gluster server nodes and NFS clients
> from the network can connect to any NFS Ganesha server.
>
> You can use Haproxy + Keepalived (or any other load balancer) if high
> availability required for the NFS Ganesha connections (Ex: If a server node
> goes down, then nfs client can connect to other NFS ganesha server node).
>
> --
> Aravinda
> Kadalu Technologies
>
>
>
>  On Wed, 13 Dec 2023 01:42:11 +0530 *Gilberto Ferreira
> >* wrote ---
>
> Ah that's nice.
> Somebody knows this can be achieved with two servers?
>
> ---
> Gilberto Nunes Ferreira
> (47) 99676-7530 - Whatsapp / Telegram
>
>
>
>
>
>
>
> Em ter., 12 de dez. de 2023 às 17:08, Danny 
> escreveu:
>
> 
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://meet.google.com/cpu-eiue-hvk
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
> Wow, HUGE improvement with NFS-Ganesha!
>
>
> sudo dnf -y install glusterfs-ganesha
> sudo vim /etc/ganesha/ganesha.conf
>
> NFS_CORE_PARAM {
> mount_path_pseudo = true;
> Protocols = 3,4;
> }
> EXPORT_DEFAULTS {
> Access_Type = RW;
> }
>
> LOG {
> Default_Log_Level = WARN;
> }
>
> EXPORT{
> Export_Id = 1 ; # Export ID unique to each export
> Path = "/data"; # Path of the volume to be exported
>
> FSAL {
> name = GLUSTER;
> hostname = "localhost"; # IP of one of the nodes in the trusted
> pool
> volume = "data";# Volume name. Eg: "test_volume"
> }
>
> Access_type = RW;   # Access permissions
> Squash = No_root_squash;# To enable/disable root squashing
> Disable_ACL = TRUE; # To enable/disable ACL
> Pseudo = "/data";   # NFSv4 pseudo path for this export
> Protocols = "3","4" ;   # NFS protocols supported
> Transports = "UDP","TCP" ;  # Transport protocols supported
> SecType = "sys";# Security flavors supported
> }
>
>
> sudo systemctl enable --now nfs-ganesha
> sudo vim /etc/fstab
>
> localhost:/data /data nfs
> defaults,_netdev  0 0
>
>
> sudo systemctl daemon-reload
> sudo mount -a
>
> fio --name=test --filename=/data/wow --size=1G --readwrite=write
>
> Run status group 0 (all jobs):
>   WRITE: bw=2246MiB/s (2355MB/s), 2246MiB/s-2246MiB/s (2355MB/s-2355MB/s),
> io=1024MiB (1074MB), run=456-456msec
>
> Yeah 2355MB/s is much better than the original 115MB/s
>
> So in the end, I guess FUSE isn't the best choice.
>
> On Tue, Dec 12, 2023 at 3:00 PM Gilberto Ferreira <
> gilberto.nune...@gmail.com> wrote:
>
> Fuse there some overhead.
> Take a look at libgfapi:
>
> https://staged-gluster-docs.readthedocs.io/en/release3.7.0beta1/Features/libgfapi/
>
> I know this doc somehow is out of date, but could be a hint
>
>
> ---
> Gilberto Nunes Ferreira
> (47) 99676-7530 - Whatsapp / Telegram
>
>
>
>
>
>
>
> Em ter., 12 de dez. de 2023 às 16:29, Danny 
> escreveu:
>
> Nope, not a caching thing. I've tried multiple different types of fio
> tests, all produce the same results. Gbps when hitting the disks locally,
> slow MB\s when hitting the Gluster FUSE mount.
>
> I've been reading up on glustr-ganesha, and will give that a try.
>
> On Tue, Dec 12, 2023 at 1:58 PM Ramon Selga  wrote:
>
> Dismiss my first question: you have SAS 12Gbps SSDs  Sorry!
>
> El 12/12/23 a les 19:52, Ramon Selga ha escrit:
>
> May ask you which kind of disks you have in this setup? rotational, ssd
> SAS/SATA, nvme?
>
> Is there a RAID controller with writeback caching?
>
> It seems to me your fio test on local brick has a unclear result due to
> some caching.
>
> Try something like (you can

Re: [Gluster-users] Gluster Performance - 12 Gbps SSDs and 10 Gbps NIC

2023-12-13 Thread Strahil Nikolov

Hi Aravinda,
Based on the output it’s a ‘replica 3 arbiter 1’ type.
Gilberto,What’s the latency between the nodes ?
Best Regards,Strahil Nikolov 




On Wednesday, December 13, 2023, 7:36 AM, Aravinda  wrote:

Only Replica 2 or Distributed Gluster volumes can be created with two servers. 
High chance of split brain with Replica 2 compared to Replica 3 volume.

For NFS Ganesha, no issue exporting the volume even if only one server is 
available. Run NFS Ganesha servers in Gluster server nodes and NFS clients from 
the network can connect to any NFS Ganesha server.

You can use Haproxy + Keepalived (or any other load balancer) if high 
availability required for the NFS Ganesha connections (Ex: If a server node 
goes down, then nfs client can connect to other NFS ganesha server node).

--Aravinda
Kadalu Technologies


 On Wed, 13 Dec 2023 01:42:11 +0530 Gilberto Ferreira 
 wrote ---


Ah that's nice.Somebody knows this can be achieved with two servers?

---
Gilberto Nunes Ferreira
(47) 99676-7530 - Whatsapp / Telegram












Em ter., 12 de dez. de 2023 às 17:08, Danny  
escreveu:


 
 
 
Community Meeting Calendar: 
 
Schedule - 
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC 
Bridge: https://meet.google.com/cpu-eiue-hvk 
Gluster-users mailing list 
Gluster-users@gluster.org 
https://lists.gluster.org/mailman/listinfo/gluster-users 

Wow, HUGE improvement with NFS-Ganesha!


sudo dnf -y install glusterfs-ganesha
sudo vim /etc/ganesha/ganesha.conf

NFS_CORE_PARAM {
mount_path_pseudo = true;
Protocols = 3,4;
}
EXPORT_DEFAULTS {
Access_Type = RW;
}
 
LOG {
Default_Log_Level = WARN;
}
 
EXPORT{
Export_Id = 1 ; # Export ID unique to each export
Path = "/data"; # Path of the volume to be exported
 
FSAL {
name = GLUSTER;
hostname = "localhost"; # IP of one of the nodes in the trusted pool
volume = "data";    # Volume name. Eg: "test_volume"
}
 
Access_type = RW;   # Access permissions
Squash = No_root_squash;    # To enable/disable root squashing
Disable_ACL = TRUE; # To enable/disable ACL
Pseudo = "/data";   # NFSv4 pseudo path for this export
Protocols = "3","4" ;   # NFS protocols supported
Transports = "UDP","TCP" ;  # Transport protocols supported
SecType = "sys";    # Security flavors supported
}


sudo systemctl enable --now nfs-ganesha
sudo vim /etc/fstab 

localhost:/data /data nfs    defaults,_netdev   
   0 0


sudo systemctl daemon-reload
sudo mount -a

fio --name=test --filename=/data/wow --size=1G --readwrite=write

Run status group 0 (all jobs):
  WRITE: bw=2246MiB/s (2355MB/s), 2246MiB/s-2246MiB/s (2355MB/s-2355MB/s), 
io=1024MiB (1074MB), run=456-456msec
Yeah 2355MB/s is much better than the original 115MB/s

So in the end, I guess FUSE isn't the best choice. 

On Tue, Dec 12, 2023 at 3:00 PM Gilberto Ferreira  
wrote:

Fuse there some overhead.Take a look at libgfapi:
https://staged-gluster-docs.readthedocs.io/en/release3.7.0beta1/Features/libgfapi/

I know this doc somehow is out of date, but could be a hint


---
Gilberto Nunes Ferreira
(47) 99676-7530 - Whatsapp / Telegram












Em ter., 12 de dez. de 2023 às 16:29, Danny  
escreveu:

Nope, not a caching thing. I've tried multiple different types of fio tests, 
all produce the same results. Gbps when hitting the disks locally, slow MB\s 
when hitting the Gluster FUSE mount.

I've been reading up on glustr-ganesha, and will give that a try.

On Tue, Dec 12, 2023 at 1:58 PM Ramon Selga  wrote:

Dismiss my first question: you have SAS 12Gbps SSDs  Sorry!
 
 El 12/12/23 a les 19:52, Ramon Selga ha escrit:

May ask you which kind of disks you have in this setup? rotational, ssd 
SAS/SATA, nvme?
 
 Is there a RAID controller with writeback caching?
 
 It seems to me your fio test on local brick has a unclear result due to some 
caching.
 
 Try something like (you can consider to increase test file size depending of 
your caching memory) :
 
 fio --size=16G --name=test --filename=/gluster/data/brick/wow --bs=1M 
--nrfiles=1 --direct=1 --sync=0 --randrepeat=0 --rw=write --refill_buffers 
--end_fsync=1 --iodepth=200 --ioengine=libaio
 
 Also remember a replica 3 arbiter 1 volume writes synchronously to two data 
bricks, halving throughput of your network backend.
 
 Try similar fio on gluster mount but I hardly see more than 300MB/s writing 
sequentially on only one fuse mount even with nvme backend. On the other side, 
with 4 to 6 clients, you can easily reach 1.5GB/s of aggregate throughput 
 
 To start, I think is better to try with default parameters for your replica 
volume.
 
 Best regards!
 
 Ramon
 
  
 El 12/12/23 a les 19:10, Danny ha escrit:

Sorry, I noticed that too after I posted, so I instantly upgraded to 10. Issue 
remains. 

On Tue, Dec 12, 2023 at 1:09 PM Gilberto Ferreira  
wrote:

I strongly suggest you update to version 10 or higher. 
 It's

Re: [Gluster-users] Gluster Performance - 12 Gbps SSDs and 10 Gbps NIC

2023-12-12 Thread Aravinda

Only Replica 2 or Distributed Gluster volumes can be created with two servers. 
High chance of split brain with Replica 2 compared to Replica 3 volume.



For NFS Ganesha, no issue exporting the volume even if only one server is 
available. Run NFS Ganesha servers in Gluster server nodes and NFS clients from 
the network can connect to any NFS Ganesha server.



You can use Haproxy + Keepalived (or any other load balancer) if high 
availability required for the NFS Ganesha connections (Ex: If a server node 
goes down, then nfs client can connect to other NFS ganesha server node).



--
Aravinda

Kadalu Technologies







 On Wed, 13 Dec 2023 01:42:11 +0530 Gilberto Ferreira 
 wrote ---



Ah that's nice.Somebody knows this can be achieved with two servers?

---

Gilberto Nunes Ferreira


(47) 99676-7530 - Whatsapp / Telegram






















Em ter., 12 de dez. de 2023 às 17:08, Danny 
 escreveu:





 
 
 
Community Meeting Calendar: 
 
Schedule - 
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC 
Bridge: https://meet.google.com/cpu-eiue-hvk 
Gluster-users mailing list 
mailto:Gluster-users@gluster.org 
https://lists.gluster.org/mailman/listinfo/gluster-users 

Wow, HUGE improvement with NFS-Ganesha!





sudo dnf -y install glusterfs-ganesha

sudo vim /etc/ganesha/ganesha.conf





NFS_CORE_PARAM {

mount_path_pseudo = true;

Protocols = 3,4;

}

EXPORT_DEFAULTS {

Access_Type = RW;

}

 

LOG {

Default_Log_Level = WARN;

}

 

EXPORT{

Export_Id = 1 ; # Export ID unique to each export

Path = "/data"; # Path of the volume to be exported

 

FSAL {

name = GLUSTER;

hostname = "localhost"; # IP of one of the nodes in the trusted pool

volume = "data";    # Volume name. Eg: "test_volume"

}

 

Access_type = RW;   # Access permissions

Squash = No_root_squash;    # To enable/disable root squashing

Disable_ACL = TRUE; # To enable/disable ACL

Pseudo = "/data";   # NFSv4 pseudo path for this export

Protocols = "3","4" ;   # NFS protocols supported

Transports = "UDP","TCP" ;  # Transport protocols supported

SecType = "sys";    # Security flavors supported

}







sudo systemctl enable --now nfs-ganesha



sudo vim /etc/fstab 



localhost:/data /data nfs    defaults,_netdev   
   0 0






sudo systemctl daemon-reload

sudo mount -a





fio --name=test --filename=/data/wow --size=1G --readwrite=write




Run status group 0 (all jobs):
  WRITE: bw=2246MiB/s (2355MB/s), 2246MiB/s-2246MiB/s (2355MB/s-2355MB/s), 
io=1024MiB (1074MB), run=456-456msec


Yeah 
2355MB/s is much better than the original 115MB/s



So in the end, I guess FUSE isn't the best choice. 




On Tue, Dec 12, 2023 at 3:00 PM Gilberto Ferreira 
 wrote:

Fuse there some overhead.Take a look at libgfapi:

https://staged-gluster-docs.readthedocs.io/en/release3.7.0beta1/Features/libgfapi/



I know this doc somehow is out of date, but could be a hint




---

Gilberto Nunes Ferreira


(47) 99676-7530 - Whatsapp / Telegram






















Em ter., 12 de dez. de 2023 às 16:29, Danny 
 escreveu:

Nope, not a caching thing. I've tried multiple different types of fio tests, 
all produce the same results. Gbps when hitting the disks locally, slow MB\s 
when hitting the Gluster FUSE mount.



I've been reading up on glustr-ganesha, and will give that a try.



On Tue, Dec 12, 2023 at 1:58 PM Ramon Selga  
wrote:

Dismiss my first question: you have SAS
  12Gbps SSDs  Sorry!
 
 El 12/12/23 a les 19:52, Ramon Selga ha
  escrit:

May ask you which kind of disks you have
in this setup? rotational, ssd SAS/SATA, nvme?
 
 Is there a RAID controller with writeback caching?
 
 It seems to me your fio test on local brick has a unclear result
due to some caching.
 
 Try something like (you can consider to increase test file size
depending of your caching memory) :
 
 fio --size=16G --name=test --filename=/gluster/data/brick/wow
--bs=1M --nrfiles=1 --direct=1 --sync=0 --randrepeat=0
--rw=write --refill_buffers --end_fsync=1 --iodepth=200
--ioengine=libaio
 
 Also remember a replica 3 arbiter 1 volume writes
  synchronously to two data bricks, halving throughput of your
  network backend.
 
 Try similar fio on gluster mount but I hardly see more than
  300MB/s writing sequentially on only one fuse mount even with nvme
  backend. On the other side, with 4 to 6 clients, you can easily
  reach 1.5GB/s of aggregate throughput 
 
 To start, I think is better to try with default parameters for
  your replica volume.
 
 Best regards!
 
 Ramon
 
  
 El 12/12/23 a les 19:10, Danny ha
escrit:

Sorry, I noticed that too after I posted, so I

Re: [Gluster-users] Gluster Performance - 12 Gbps SSDs and 10 Gbps NIC

2023-12-12 Thread Gilberto Ferreira

Ah that's nice.
Somebody knows this can be achieved with two servers?

---
Gilberto Nunes Ferreira
(47) 99676-7530 - Whatsapp / Telegram






Em ter., 12 de dez. de 2023 às 17:08, Danny 
escreveu:

> Wow, HUGE improvement with NFS-Ganesha!
>
> sudo dnf -y install glusterfs-ganesha
> sudo vim /etc/ganesha/ganesha.conf
>
> NFS_CORE_PARAM {
> mount_path_pseudo = true;
> Protocols = 3,4;
> }
> EXPORT_DEFAULTS {
> Access_Type = RW;
> }
>
> LOG {
> Default_Log_Level = WARN;
> }
>
> EXPORT{
> Export_Id = 1 ; # Export ID unique to each export
> Path = "/data"; # Path of the volume to be exported
>
> FSAL {
> name = GLUSTER;
> hostname = "localhost"; # IP of one of the nodes in the trusted
> pool
> volume = "data";# Volume name. Eg: "test_volume"
> }
>
> Access_type = RW;   # Access permissions
> Squash = No_root_squash;# To enable/disable root squashing
> Disable_ACL = TRUE; # To enable/disable ACL
> Pseudo = "/data";   # NFSv4 pseudo path for this export
> Protocols = "3","4" ;   # NFS protocols supported
> Transports = "UDP","TCP" ;  # Transport protocols supported
> SecType = "sys";# Security flavors supported
> }
>
>
> sudo systemctl enable --now nfs-ganesha
> sudo vim /etc/fstab
>
> localhost:/data /data nfs
> defaults,_netdev  0 0
>
> sudo systemctl daemon-reload
> sudo mount -a
>
> fio --name=test --filename=/data/wow --size=1G --readwrite=write
>
> Run status group 0 (all jobs):
>   WRITE: bw=2246MiB/s (2355MB/s), 2246MiB/s-2246MiB/s (2355MB/s-2355MB/s),
> io=1024MiB (1074MB), run=456-456msec
>
> Yeah 2355MB/s is much better than the original 115MB/s
>
> So in the end, I guess FUSE isn't the best choice.
>
> On Tue, Dec 12, 2023 at 3:00 PM Gilberto Ferreira <
> gilberto.nune...@gmail.com> wrote:
>
>> Fuse there some overhead.
>> Take a look at libgfapi:
>>
>> https://staged-gluster-docs.readthedocs.io/en/release3.7.0beta1/Features/libgfapi/
>>
>> I know this doc somehow is out of date, but could be a hint
>>
>>
>> ---
>> Gilberto Nunes Ferreira
>> (47) 99676-7530 - Whatsapp / Telegram
>>
>>
>>
>>
>>
>>
>> Em ter., 12 de dez. de 2023 às 16:29, Danny 
>> escreveu:
>>
>>> Nope, not a caching thing. I've tried multiple different types of fio
>>> tests, all produce the same results. Gbps when hitting the disks locally,
>>> slow MB\s when hitting the Gluster FUSE mount.
>>>
>>> I've been reading up on glustr-ganesha, and will give that a try.
>>>
>>> On Tue, Dec 12, 2023 at 1:58 PM Ramon Selga 
>>> wrote:
>>>
 Dismiss my first question: you have SAS 12Gbps SSDs  Sorry!

 El 12/12/23 a les 19:52, Ramon Selga ha escrit:

 May ask you which kind of disks you have in this setup? rotational, ssd
 SAS/SATA, nvme?

 Is there a RAID controller with writeback caching?

 It seems to me your fio test on local brick has a unclear result due to
 some caching.

 Try something like (you can consider to increase test file size
 depending of your caching memory) :

 fio --size=16G --name=test --filename=/gluster/data/brick/wow --bs=1M
 --nrfiles=1 --direct=1 --sync=0 --randrepeat=0 --rw=write --refill_buffers
 --end_fsync=1 --iodepth=200 --ioengine=libaio

 Also remember a replica 3 arbiter 1 volume writes synchronously to two
 data bricks, halving throughput of your network backend.

 Try similar fio on gluster mount but I hardly see more than 300MB/s
 writing sequentially on only one fuse mount even with nvme backend. On the
 other side, with 4 to 6 clients, you can easily reach 1.5GB/s of aggregate
 throughput

 To start, I think is better to try with default parameters for your
 replica volume.

 Best regards!

 Ramon


 El 12/12/23 a les 19:10, Danny ha escrit:

 Sorry, I noticed that too after I posted, so I instantly upgraded to
 10. Issue remains.

 On Tue, Dec 12, 2023 at 1:09 PM Gilberto Ferreira <
 gilberto.nune...@gmail.com> wrote:

> I strongly suggest you update to version 10 or higher.
> It's come with significant improvement regarding performance.
> ---
> Gilberto Nunes Ferreira
> (47) 99676-7530 - Whatsapp / Telegram
>
>
>
>
>
>
> Em ter., 12 de dez. de 2023 às 13:03, Danny <
> dbray925+glus...@gmail.com> escreveu:
>
>> MTU is already 9000, and as you can see from the IPERF results, I've
>> got a nice, fast connection between the nodes.
>>
>> On Tue, Dec 12, 2023 at 9:49 AM Strahil Nikolov <
>> hunter86...@yahoo.com> wrote:
>>
>>> Hi,
>>>
>>> Let’s try the simple things:
>>>
>>> Check if you can use MTU9000 and if it’s possible, set it on the
>>> Bond Slaves and the bond devices:
>>>  ping GLUSTER_PEER -c 10 -M do -s 8972
>>>
>>> Then try to follow up

Re: [Gluster-users] Gluster Performance - 12 Gbps SSDs and 10 Gbps NIC

2023-12-12 Thread Danny

Wow, HUGE improvement with NFS-Ganesha!

sudo dnf -y install glusterfs-ganesha
sudo vim /etc/ganesha/ganesha.conf

NFS_CORE_PARAM {
mount_path_pseudo = true;
Protocols = 3,4;
}
EXPORT_DEFAULTS {
Access_Type = RW;
}

LOG {
Default_Log_Level = WARN;
}

EXPORT{
Export_Id = 1 ; # Export ID unique to each export
Path = "/data"; # Path of the volume to be exported

FSAL {
name = GLUSTER;
hostname = "localhost"; # IP of one of the nodes in the trusted pool
volume = "data";# Volume name. Eg: "test_volume"
}

Access_type = RW;   # Access permissions
Squash = No_root_squash;# To enable/disable root squashing
Disable_ACL = TRUE; # To enable/disable ACL
Pseudo = "/data";   # NFSv4 pseudo path for this export
Protocols = "3","4" ;   # NFS protocols supported
Transports = "UDP","TCP" ;  # Transport protocols supported
SecType = "sys";# Security flavors supported
}


sudo systemctl enable --now nfs-ganesha
sudo vim /etc/fstab

localhost:/data /data nfs
defaults,_netdev  0 0

sudo systemctl daemon-reload
sudo mount -a

fio --name=test --filename=/data/wow --size=1G --readwrite=write

Run status group 0 (all jobs):
  WRITE: bw=2246MiB/s (2355MB/s), 2246MiB/s-2246MiB/s (2355MB/s-2355MB/s),
io=1024MiB (1074MB), run=456-456msec

Yeah 2355MB/s is much better than the original 115MB/s

So in the end, I guess FUSE isn't the best choice.

On Tue, Dec 12, 2023 at 3:00 PM Gilberto Ferreira <
gilberto.nune...@gmail.com> wrote:

> Fuse there some overhead.
> Take a look at libgfapi:
>
> https://staged-gluster-docs.readthedocs.io/en/release3.7.0beta1/Features/libgfapi/
>
> I know this doc somehow is out of date, but could be a hint
>
>
> ---
> Gilberto Nunes Ferreira
> (47) 99676-7530 - Whatsapp / Telegram
>
>
>
>
>
>
> Em ter., 12 de dez. de 2023 às 16:29, Danny 
> escreveu:
>
>> Nope, not a caching thing. I've tried multiple different types of fio
>> tests, all produce the same results. Gbps when hitting the disks locally,
>> slow MB\s when hitting the Gluster FUSE mount.
>>
>> I've been reading up on glustr-ganesha, and will give that a try.
>>
>> On Tue, Dec 12, 2023 at 1:58 PM Ramon Selga 
>> wrote:
>>
>>> Dismiss my first question: you have SAS 12Gbps SSDs  Sorry!
>>>
>>> El 12/12/23 a les 19:52, Ramon Selga ha escrit:
>>>
>>> May ask you which kind of disks you have in this setup? rotational, ssd
>>> SAS/SATA, nvme?
>>>
>>> Is there a RAID controller with writeback caching?
>>>
>>> It seems to me your fio test on local brick has a unclear result due to
>>> some caching.
>>>
>>> Try something like (you can consider to increase test file size
>>> depending of your caching memory) :
>>>
>>> fio --size=16G --name=test --filename=/gluster/data/brick/wow --bs=1M
>>> --nrfiles=1 --direct=1 --sync=0 --randrepeat=0 --rw=write --refill_buffers
>>> --end_fsync=1 --iodepth=200 --ioengine=libaio
>>>
>>> Also remember a replica 3 arbiter 1 volume writes synchronously to two
>>> data bricks, halving throughput of your network backend.
>>>
>>> Try similar fio on gluster mount but I hardly see more than 300MB/s
>>> writing sequentially on only one fuse mount even with nvme backend. On the
>>> other side, with 4 to 6 clients, you can easily reach 1.5GB/s of aggregate
>>> throughput
>>>
>>> To start, I think is better to try with default parameters for your
>>> replica volume.
>>>
>>> Best regards!
>>>
>>> Ramon
>>>
>>>
>>> El 12/12/23 a les 19:10, Danny ha escrit:
>>>
>>> Sorry, I noticed that too after I posted, so I instantly upgraded to 10.
>>> Issue remains.
>>>
>>> On Tue, Dec 12, 2023 at 1:09 PM Gilberto Ferreira <
>>> gilberto.nune...@gmail.com> wrote:
>>>
 I strongly suggest you update to version 10 or higher.
 It's come with significant improvement regarding performance.
 ---
 Gilberto Nunes Ferreira
 (47) 99676-7530 - Whatsapp / Telegram






 Em ter., 12 de dez. de 2023 às 13:03, Danny 
 escreveu:

> MTU is already 9000, and as you can see from the IPERF results, I've
> got a nice, fast connection between the nodes.
>
> On Tue, Dec 12, 2023 at 9:49 AM Strahil Nikolov 
> wrote:
>
>> Hi,
>>
>> Let’s try the simple things:
>>
>> Check if you can use MTU9000 and if it’s possible, set it on the Bond
>> Slaves and the bond devices:
>>  ping GLUSTER_PEER -c 10 -M do -s 8972
>>
>> Then try to follow up the recommendations from
>> https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/chap-configuring_red_hat_storage_for_enhancing_performance
>>
>>
>>
>> Best Regards,
>> Strahil Nikolov
>>
>> On Monday, December 11, 2023, 3:32 PM, Danny <
>> dbray925+glus...@gmail.com> wrote:
>>
>> Hello list, I'm hoping someone can let me know what setting I missed.
>>
>>

Re: [Gluster-users] Gluster Performance - 12 Gbps SSDs and 10 Gbps NIC

2023-12-12 Thread Gilberto Ferreira

Fuse there some overhead.
Take a look at libgfapi:
https://staged-gluster-docs.readthedocs.io/en/release3.7.0beta1/Features/libgfapi/

I know this doc somehow is out of date, but could be a hint


---
Gilberto Nunes Ferreira
(47) 99676-7530 - Whatsapp / Telegram






Em ter., 12 de dez. de 2023 às 16:29, Danny 
escreveu:

> Nope, not a caching thing. I've tried multiple different types of fio
> tests, all produce the same results. Gbps when hitting the disks locally,
> slow MB\s when hitting the Gluster FUSE mount.
>
> I've been reading up on glustr-ganesha, and will give that a try.
>
> On Tue, Dec 12, 2023 at 1:58 PM Ramon Selga  wrote:
>
>> Dismiss my first question: you have SAS 12Gbps SSDs  Sorry!
>>
>> El 12/12/23 a les 19:52, Ramon Selga ha escrit:
>>
>> May ask you which kind of disks you have in this setup? rotational, ssd
>> SAS/SATA, nvme?
>>
>> Is there a RAID controller with writeback caching?
>>
>> It seems to me your fio test on local brick has a unclear result due to
>> some caching.
>>
>> Try something like (you can consider to increase test file size depending
>> of your caching memory) :
>>
>> fio --size=16G --name=test --filename=/gluster/data/brick/wow --bs=1M
>> --nrfiles=1 --direct=1 --sync=0 --randrepeat=0 --rw=write --refill_buffers
>> --end_fsync=1 --iodepth=200 --ioengine=libaio
>>
>> Also remember a replica 3 arbiter 1 volume writes synchronously to two
>> data bricks, halving throughput of your network backend.
>>
>> Try similar fio on gluster mount but I hardly see more than 300MB/s
>> writing sequentially on only one fuse mount even with nvme backend. On the
>> other side, with 4 to 6 clients, you can easily reach 1.5GB/s of aggregate
>> throughput
>>
>> To start, I think is better to try with default parameters for your
>> replica volume.
>>
>> Best regards!
>>
>> Ramon
>>
>>
>> El 12/12/23 a les 19:10, Danny ha escrit:
>>
>> Sorry, I noticed that too after I posted, so I instantly upgraded to 10.
>> Issue remains.
>>
>> On Tue, Dec 12, 2023 at 1:09 PM Gilberto Ferreira <
>> gilberto.nune...@gmail.com> wrote:
>>
>>> I strongly suggest you update to version 10 or higher.
>>> It's come with significant improvement regarding performance.
>>> ---
>>> Gilberto Nunes Ferreira
>>> (47) 99676-7530 - Whatsapp / Telegram
>>>
>>>
>>>
>>>
>>>
>>>
>>> Em ter., 12 de dez. de 2023 às 13:03, Danny 
>>> escreveu:
>>>
 MTU is already 9000, and as you can see from the IPERF results, I've
 got a nice, fast connection between the nodes.

 On Tue, Dec 12, 2023 at 9:49 AM Strahil Nikolov 
 wrote:

> Hi,
>
> Let’s try the simple things:
>
> Check if you can use MTU9000 and if it’s possible, set it on the Bond
> Slaves and the bond devices:
>  ping GLUSTER_PEER -c 10 -M do -s 8972
>
> Then try to follow up the recommendations from
> https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/chap-configuring_red_hat_storage_for_enhancing_performance
>
>
>
> Best Regards,
> Strahil Nikolov
>
> On Monday, December 11, 2023, 3:32 PM, Danny <
> dbray925+glus...@gmail.com> wrote:
>
> Hello list, I'm hoping someone can let me know what setting I missed.
>
> Hardware:
> Dell R650 servers, Dual 24 Core Xeon 2.8 GHz, 1 TB RAM
> 8x SSD s Negotiated Speed 12 Gbps
> PERC H755 Controller - RAID 6
> Created virtual "data" disk from the above 8 SSD drives, for a ~20 TB
> /dev/sdb
>
> OS:
> CentOS Stream
> kernel-4.18.0-526.el8.x86_64
> glusterfs-7.9-1.el8.x86_64
>
> IPERF Test between nodes:
> [ ID] Interval   Transfer Bitrate Retr
> [  5]   0.00-10.00  sec  11.5 GBytes  9.90 Gbits/sec0
> sender
> [  5]   0.00-10.04  sec  11.5 GBytes  9.86 Gbits/sec
>  receiver
>
> All good there. ~10 Gbps, as expected.
>
> LVM Install:
> export DISK="/dev/sdb"
> sudo parted --script $DISK "mklabel gpt"
> sudo parted --script $DISK "mkpart primary 0% 100%"
> sudo parted --script $DISK "set 1 lvm on"
> sudo pvcreate --dataalignment 128K /dev/sdb1
> sudo vgcreate --physicalextentsize 128K gfs_vg /dev/sdb1
> sudo lvcreate -L 16G -n gfs_pool_meta gfs_vg
> sudo lvcreate -l 95%FREE -n gfs_pool gfs_vg
> sudo lvconvert --chunksize 1280K --thinpool gfs_vg/gfs_pool
> --poolmetadata gfs_vg/gfs_pool_meta
> sudo lvchange --zero n gfs_vg/gfs_pool
> sudo lvcreate -V 19.5TiB --thinpool gfs_vg/gfs_pool -n gfs_lv
> sudo mkfs.xfs -f -i size=512 -n size=8192 -d su=128k,sw=10
> /dev/mapper/gfs_vg-gfs_lv
> sudo vim /etc/fstab
> /dev/mapper/gfs_vg-gfs_lv   /gluster/data/brick   xfs
> rw,inode64,noatime,nouuid 0 0
>
> sudo systemctl daemon-reload && sudo mount -a
> fio --name=test --filename=/gluster/data/brick/wow --size=1G
> --readwrite=write
>
> Run status group 0 (all jobs):
>   WRITE: bw=2081MiB/s

Re: [Gluster-users] Gluster Performance - 12 Gbps SSDs and 10 Gbps NIC

2023-12-12 Thread Danny

Nope, not a caching thing. I've tried multiple different types of fio
tests, all produce the same results. Gbps when hitting the disks locally,
slow MB\s when hitting the Gluster FUSE mount.

I've been reading up on glustr-ganesha, and will give that a try.

On Tue, Dec 12, 2023 at 1:58 PM Ramon Selga  wrote:

> Dismiss my first question: you have SAS 12Gbps SSDs  Sorry!
>
> El 12/12/23 a les 19:52, Ramon Selga ha escrit:
>
> May ask you which kind of disks you have in this setup? rotational, ssd
> SAS/SATA, nvme?
>
> Is there a RAID controller with writeback caching?
>
> It seems to me your fio test on local brick has a unclear result due to
> some caching.
>
> Try something like (you can consider to increase test file size depending
> of your caching memory) :
>
> fio --size=16G --name=test --filename=/gluster/data/brick/wow --bs=1M
> --nrfiles=1 --direct=1 --sync=0 --randrepeat=0 --rw=write --refill_buffers
> --end_fsync=1 --iodepth=200 --ioengine=libaio
>
> Also remember a replica 3 arbiter 1 volume writes synchronously to two
> data bricks, halving throughput of your network backend.
>
> Try similar fio on gluster mount but I hardly see more than 300MB/s
> writing sequentially on only one fuse mount even with nvme backend. On the
> other side, with 4 to 6 clients, you can easily reach 1.5GB/s of aggregate
> throughput
>
> To start, I think is better to try with default parameters for your
> replica volume.
>
> Best regards!
>
> Ramon
>
>
> El 12/12/23 a les 19:10, Danny ha escrit:
>
> Sorry, I noticed that too after I posted, so I instantly upgraded to 10.
> Issue remains.
>
> On Tue, Dec 12, 2023 at 1:09 PM Gilberto Ferreira <
> gilberto.nune...@gmail.com> wrote:
>
>> I strongly suggest you update to version 10 or higher.
>> It's come with significant improvement regarding performance.
>> ---
>> Gilberto Nunes Ferreira
>> (47) 99676-7530 - Whatsapp / Telegram
>>
>>
>>
>>
>>
>>
>> Em ter., 12 de dez. de 2023 às 13:03, Danny 
>> escreveu:
>>
>>> MTU is already 9000, and as you can see from the IPERF results, I've got
>>> a nice, fast connection between the nodes.
>>>
>>> On Tue, Dec 12, 2023 at 9:49 AM Strahil Nikolov 
>>> wrote:
>>>
 Hi,

 Let’s try the simple things:

 Check if you can use MTU9000 and if it’s possible, set it on the Bond
 Slaves and the bond devices:
  ping GLUSTER_PEER -c 10 -M do -s 8972

 Then try to follow up the recommendations from
 https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/chap-configuring_red_hat_storage_for_enhancing_performance



 Best Regards,
 Strahil Nikolov

 On Monday, December 11, 2023, 3:32 PM, Danny <
 dbray925+glus...@gmail.com> wrote:

 Hello list, I'm hoping someone can let me know what setting I missed.

 Hardware:
 Dell R650 servers, Dual 24 Core Xeon 2.8 GHz, 1 TB RAM
 8x SSD s Negotiated Speed 12 Gbps
 PERC H755 Controller - RAID 6
 Created virtual "data" disk from the above 8 SSD drives, for a ~20 TB
 /dev/sdb

 OS:
 CentOS Stream
 kernel-4.18.0-526.el8.x86_64
 glusterfs-7.9-1.el8.x86_64

 IPERF Test between nodes:
 [ ID] Interval   Transfer Bitrate Retr
 [  5]   0.00-10.00  sec  11.5 GBytes  9.90 Gbits/sec0
 sender
 [  5]   0.00-10.04  sec  11.5 GBytes  9.86 Gbits/sec
  receiver

 All good there. ~10 Gbps, as expected.

 LVM Install:
 export DISK="/dev/sdb"
 sudo parted --script $DISK "mklabel gpt"
 sudo parted --script $DISK "mkpart primary 0% 100%"
 sudo parted --script $DISK "set 1 lvm on"
 sudo pvcreate --dataalignment 128K /dev/sdb1
 sudo vgcreate --physicalextentsize 128K gfs_vg /dev/sdb1
 sudo lvcreate -L 16G -n gfs_pool_meta gfs_vg
 sudo lvcreate -l 95%FREE -n gfs_pool gfs_vg
 sudo lvconvert --chunksize 1280K --thinpool gfs_vg/gfs_pool
 --poolmetadata gfs_vg/gfs_pool_meta
 sudo lvchange --zero n gfs_vg/gfs_pool
 sudo lvcreate -V 19.5TiB --thinpool gfs_vg/gfs_pool -n gfs_lv
 sudo mkfs.xfs -f -i size=512 -n size=8192 -d su=128k,sw=10
 /dev/mapper/gfs_vg-gfs_lv
 sudo vim /etc/fstab
 /dev/mapper/gfs_vg-gfs_lv   /gluster/data/brick   xfs
 rw,inode64,noatime,nouuid 0 0

 sudo systemctl daemon-reload && sudo mount -a
 fio --name=test --filename=/gluster/data/brick/wow --size=1G
 --readwrite=write

 Run status group 0 (all jobs):
   WRITE: bw=2081MiB/s (2182MB/s), 2081MiB/s-2081MiB/s
 (2182MB/s-2182MB/s), io=1024MiB (1074MB), run=492-492msec

 All good there. 2182MB/s =~ 17.5 Gbps. Nice!


 Gluster install:
 export NODE1='10.54.95.123'
 export NODE2='10.54.95.124'
 export NODE3='10.54.95.125'
 sudo gluster peer probe $NODE2
 sudo gluster peer probe $NODE3
 sudo gluster volume create data replica 3 arbiter 1
 $NODE1:/gluster/data/brick $NODE2:/gluster/data/brick

Re: [Gluster-users] Gluster Performance - 12 Gbps SSDs and 10 Gbps NIC

2023-12-12 Thread Ramon Selga


Dismiss my first question: you have SAS 12Gbps SSDs  Sorry!

El 12/12/23 a les 19:52, Ramon Selga ha escrit:
May ask you which kind of disks you have in this setup? rotational, ssd 
SAS/SATA, nvme?


Is there a RAID controller with writeback caching?

It seems to me your fio test on local brick has a unclear result due to some 
caching.


Try something like (you can consider to increase test file size depending of 
your caching memory) :


fio --size=16G --name=test --filename=/gluster/data/brick/wow --bs=1M 
--nrfiles=1 --direct=1 --sync=0 --randrepeat=0 --rw=write --refill_buffers 
--end_fsync=1 --iodepth=200 --ioengine=libaio


Also remember a replica 3 arbiter 1 volume writes synchronously to two data 
bricks, halving throughput of your network backend.


Try similar fio on gluster mount but I hardly see more than 300MB/s writing 
sequentially on only one fuse mount even with nvme backend. On the other side, 
with 4 to 6 clients, you can easily reach 1.5GB/s of aggregate throughput


To start, I think is better to try with default parameters for your replica 
volume.


Best regards!

Ramon


El 12/12/23 a les 19:10, Danny ha escrit:
Sorry, I noticed that too after I posted, so I instantly upgraded to 10. 
Issue remains.


On Tue, Dec 12, 2023 at 1:09 PM Gilberto Ferreira 
 wrote:


I strongly suggest you update to version 10 or higher.
It's come with significant improvement regarding performance.
---
Gilberto Nunes Ferreira
(47) 99676-7530 - Whatsapp / Telegram






Em ter., 12 de dez. de 2023 às 13:03, Danny mailto:dbray925%2bglus...@gmail.com>> escreveu:

MTU is already 9000, and as you can see from the IPERF results, I've
got a nice, fast connection between the nodes.

On Tue, Dec 12, 2023 at 9:49 AM Strahil Nikolov
 wrote:

Hi,

Let’s try the simple things:

Check if you can use MTU9000 and if it’s possible, set it on the
Bond Slaves and the bond devices:
 ping GLUSTER_PEER -c 10 -M do -s 8972

Then try to follow up the recommendations from

https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/chap-configuring_red_hat_storage_for_enhancing_performance



Best Regards,
Strahil Nikolov

On Monday, December 11, 2023, 3:32 PM, Danny
mailto:dbray925%2bglus...@gmail.com>> wrote:

Hello list, I'm hoping someone can let me know what setting I
missed.

Hardware:
Dell R650 servers, Dual 24 Core Xeon 2.8 GHz, 1 TB RAM
8x SSD s Negotiated Speed 12 Gbps
PERC H755 Controller - RAID 6
Created virtual "data" disk from the above 8 SSD drives, for
a ~20 TB /dev/sdb

OS:
CentOS Stream
kernel-4.18.0-526.el8.x86_64
glusterfs-7.9-1.el8.x86_64

IPERF Test between nodes:
[ ID] Interval           Transfer Bitrate         Retr
[  5]   0.00-10.00  sec  11.5 GBytes  9.90 Gbits/sec    0    
        sender
[  5]   0.00-10.04  sec  11.5 GBytes  9.86 Gbits/sec  receiver

All good there. ~10 Gbps, as expected.

LVM Install:
export DISK="/dev/sdb"
sudo parted --script $DISK "mklabel gpt"
sudo parted --script $DISK "mkpart primary 0% 100%"
sudo parted --script $DISK "set 1 lvm on"
sudo pvcreate --dataalignment 128K /dev/sdb1
sudo vgcreate --physicalextentsize 128K gfs_vg /dev/sdb1
sudo lvcreate -L 16G -n gfs_pool_meta gfs_vg
sudo lvcreate -l 95%FREE -n gfs_pool gfs_vg
sudo lvconvert --chunksize 1280K --thinpool gfs_vg/gfs_pool
--poolmetadata gfs_vg/gfs_pool_meta
sudo lvchange --zero n gfs_vg/gfs_pool
sudo lvcreate -V 19.5TiB --thinpool gfs_vg/gfs_pool -n gfs_lv
sudo mkfs.xfs -f -i size=512 -n size=8192 -d su=128k,sw=10
/dev/mapper/gfs_vg-gfs_lv
sudo vim /etc/fstab
/dev/mapper/gfs_vg-gfs_lv /gluster/data/brick   xfs
rw,inode64,noatime,nouuid 0 0

sudo systemctl daemon-reload && sudo mount -a
fio --name=test --filename=/gluster/data/brick/wow --size=1G
--readwrite=write

Run status group 0 (all jobs):
  WRITE: bw=2081MiB/s (2182MB/s), 2081MiB/s-2081MiB/s
(2182MB/s-2182MB/s), io=1024MiB (1074MB), run=492-492msec

All good there. 2182MB/s =~ 17.5 Gbps. Nice!


Gluster install:
export NODE1='10.54.95.123'
export NODE2='10.54.95.124'
export NODE3='10.54.95.125'

Re: [Gluster-users] Gluster Performance - 12 Gbps SSDs and 10 Gbps NIC

2023-12-12 Thread Ramon Selga

May ask you which kind of disks you have in this setup? rotational, ssd 
SAS/SATA, nvme?


Is there a RAID controller with writeback caching?

It seems to me your fio test on local brick has a unclear result due to some 
caching.


Try something like (you can consider to increase test file size depending of 
your caching memory) :


fio --size=16G --name=test --filename=/gluster/data/brick/wow --bs=1M 
--nrfiles=1 --direct=1 --sync=0 --randrepeat=0 --rw=write --refill_buffers 
--end_fsync=1 --iodepth=200 --ioengine=libaio


Also remember a replica 3 arbiter 1 volume writes synchronously to two data 
bricks, halving throughput of your network backend.


Try similar fio on gluster mount but I hardly see more than 300MB/s writing 
sequentially on only one fuse mount even with nvme backend. On the other side, 
with 4 to 6 clients, you can easily reach 1.5GB/s of aggregate throughput


To start, I think is better to try with default parameters for your replica 
volume.

Best regards!

Ramon


El 12/12/23 a les 19:10, Danny ha escrit:
Sorry, I noticed that too after I posted, so I instantly upgraded to 10. Issue 
remains.


On Tue, Dec 12, 2023 at 1:09 PM Gilberto Ferreira  
wrote:


I strongly suggest you update to version 10 or higher.
It's come with significant improvement regarding performance.
---
Gilberto Nunes Ferreira
(47) 99676-7530 - Whatsapp / Telegram






Em ter., 12 de dez. de 2023 às 13:03, Danny mailto:dbray925%2bglus...@gmail.com>> escreveu:

MTU is already 9000, and as you can see from the IPERF results, I've
got a nice, fast connection between the nodes.

On Tue, Dec 12, 2023 at 9:49 AM Strahil Nikolov
 wrote:

Hi,

Let’s try the simple things:

Check if you can use MTU9000 and if it’s possible, set it on the
Bond Slaves and the bond devices:
 ping GLUSTER_PEER -c 10 -M do -s 8972

Then try to follow up the recommendations from

https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/chap-configuring_red_hat_storage_for_enhancing_performance



Best Regards,
Strahil Nikolov

On Monday, December 11, 2023, 3:32 PM, Danny
mailto:dbray925%2bglus...@gmail.com>>
wrote:

Hello list, I'm hoping someone can let me know what setting I
missed.

Hardware:
Dell R650 servers, Dual 24 Core Xeon 2.8 GHz, 1 TB RAM
8x SSD s Negotiated Speed 12 Gbps
PERC H755 Controller - RAID 6
Created virtual "data" disk from the above 8 SSD drives, for a
~20 TB /dev/sdb

OS:
CentOS Stream
kernel-4.18.0-526.el8.x86_64
glusterfs-7.9-1.el8.x86_64

IPERF Test between nodes:
[ ID] Interval           Transfer Bitrate         Retr
[  5]   0.00-10.00  sec  11.5 GBytes  9.90 Gbits/sec    0    
        sender
[  5]   0.00-10.04  sec  11.5 GBytes  9.86 Gbits/sec          
       receiver

All good there. ~10 Gbps, as expected.

LVM Install:
export DISK="/dev/sdb"
sudo parted --script $DISK "mklabel gpt"
sudo parted --script $DISK "mkpart primary 0% 100%"
sudo parted --script $DISK "set 1 lvm on"
sudo pvcreate --dataalignment 128K /dev/sdb1
sudo vgcreate --physicalextentsize 128K gfs_vg /dev/sdb1
sudo lvcreate -L 16G -n gfs_pool_meta gfs_vg
sudo lvcreate -l 95%FREE -n gfs_pool gfs_vg
sudo lvconvert --chunksize 1280K --thinpool gfs_vg/gfs_pool
--poolmetadata gfs_vg/gfs_pool_meta
sudo lvchange --zero n gfs_vg/gfs_pool
sudo lvcreate -V 19.5TiB --thinpool gfs_vg/gfs_pool -n gfs_lv
sudo mkfs.xfs -f -i size=512 -n size=8192 -d su=128k,sw=10
/dev/mapper/gfs_vg-gfs_lv
sudo vim /etc/fstab
/dev/mapper/gfs_vg-gfs_lv /gluster/data/brick   xfs
rw,inode64,noatime,nouuid 0 0

sudo systemctl daemon-reload && sudo mount -a
fio --name=test --filename=/gluster/data/brick/wow --size=1G
--readwrite=write

Run status group 0 (all jobs):
  WRITE: bw=2081MiB/s (2182MB/s), 2081MiB/s-2081MiB/s
(2182MB/s-2182MB/s), io=1024MiB (1074MB), run=492-492msec

All good there. 2182MB/s =~ 17.5 Gbps. Nice!


Gluster install:
export NODE1='10.54.95.123'
export NODE2='10.54.95.124'
export NODE3='10.54.95.125'
sudo gluster peer probe $NODE2
sudo

Re: [Gluster-users] Gluster Performance - 12 Gbps SSDs and 10 Gbps NIC

2023-12-12 Thread Danny

Sorry, I noticed that too after I posted, so I instantly upgraded to 10.
Issue remains.

On Tue, Dec 12, 2023 at 1:09 PM Gilberto Ferreira <
gilberto.nune...@gmail.com> wrote:

> I strongly suggest you update to version 10 or higher.
> It's come with significant improvement regarding performance.
> ---
> Gilberto Nunes Ferreira
> (47) 99676-7530 - Whatsapp / Telegram
>
>
>
>
>
>
> Em ter., 12 de dez. de 2023 às 13:03, Danny 
> escreveu:
>
>> MTU is already 9000, and as you can see from the IPERF results, I've got
>> a nice, fast connection between the nodes.
>>
>> On Tue, Dec 12, 2023 at 9:49 AM Strahil Nikolov 
>> wrote:
>>
>>> Hi,
>>>
>>> Let’s try the simple things:
>>>
>>> Check if you can use MTU9000 and if it’s possible, set it on the Bond
>>> Slaves and the bond devices:
>>>  ping GLUSTER_PEER -c 10 -M do -s 8972
>>>
>>> Then try to follow up the recommendations from
>>> https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/chap-configuring_red_hat_storage_for_enhancing_performance
>>>
>>>
>>>
>>> Best Regards,
>>> Strahil Nikolov
>>>
>>> On Monday, December 11, 2023, 3:32 PM, Danny 
>>> wrote:
>>>
>>> Hello list, I'm hoping someone can let me know what setting I missed.
>>>
>>> Hardware:
>>> Dell R650 servers, Dual 24 Core Xeon 2.8 GHz, 1 TB RAM
>>> 8x SSD s Negotiated Speed 12 Gbps
>>> PERC H755 Controller - RAID 6
>>> Created virtual "data" disk from the above 8 SSD drives, for a ~20 TB
>>> /dev/sdb
>>>
>>> OS:
>>> CentOS Stream
>>> kernel-4.18.0-526.el8.x86_64
>>> glusterfs-7.9-1.el8.x86_64
>>>
>>> IPERF Test between nodes:
>>> [ ID] Interval   Transfer Bitrate Retr
>>> [  5]   0.00-10.00  sec  11.5 GBytes  9.90 Gbits/sec0
>>> sender
>>> [  5]   0.00-10.04  sec  11.5 GBytes  9.86 Gbits/sec
>>>  receiver
>>>
>>> All good there. ~10 Gbps, as expected.
>>>
>>> LVM Install:
>>> export DISK="/dev/sdb"
>>> sudo parted --script $DISK "mklabel gpt"
>>> sudo parted --script $DISK "mkpart primary 0% 100%"
>>> sudo parted --script $DISK "set 1 lvm on"
>>> sudo pvcreate --dataalignment 128K /dev/sdb1
>>> sudo vgcreate --physicalextentsize 128K gfs_vg /dev/sdb1
>>> sudo lvcreate -L 16G -n gfs_pool_meta gfs_vg
>>> sudo lvcreate -l 95%FREE -n gfs_pool gfs_vg
>>> sudo lvconvert --chunksize 1280K --thinpool gfs_vg/gfs_pool
>>> --poolmetadata gfs_vg/gfs_pool_meta
>>> sudo lvchange --zero n gfs_vg/gfs_pool
>>> sudo lvcreate -V 19.5TiB --thinpool gfs_vg/gfs_pool -n gfs_lv
>>> sudo mkfs.xfs -f -i size=512 -n size=8192 -d su=128k,sw=10
>>> /dev/mapper/gfs_vg-gfs_lv
>>> sudo vim /etc/fstab
>>> /dev/mapper/gfs_vg-gfs_lv   /gluster/data/brick   xfs
>>> rw,inode64,noatime,nouuid 0 0
>>>
>>> sudo systemctl daemon-reload && sudo mount -a
>>> fio --name=test --filename=/gluster/data/brick/wow --size=1G
>>> --readwrite=write
>>>
>>> Run status group 0 (all jobs):
>>>   WRITE: bw=2081MiB/s (2182MB/s), 2081MiB/s-2081MiB/s
>>> (2182MB/s-2182MB/s), io=1024MiB (1074MB), run=492-492msec
>>>
>>> All good there. 2182MB/s =~ 17.5 Gbps. Nice!
>>>
>>>
>>> Gluster install:
>>> export NODE1='10.54.95.123'
>>> export NODE2='10.54.95.124'
>>> export NODE3='10.54.95.125'
>>> sudo gluster peer probe $NODE2
>>> sudo gluster peer probe $NODE3
>>> sudo gluster volume create data replica 3 arbiter 1
>>> $NODE1:/gluster/data/brick $NODE2:/gluster/data/brick
>>> $NODE3:/gluster/data/brick force
>>> sudo gluster volume set data network.ping-timeout 5
>>> sudo gluster volume set data performance.client-io-threads on
>>> sudo gluster volume set data group metadata-cache
>>> sudo gluster volume start data
>>> sudo gluster volume info all
>>>
>>> Volume Name: data
>>> Type: Replicate
>>> Volume ID: b52b5212-82c8-4b1a-8db3-52468bc0226e
>>> Status: Started
>>> Snapshot Count: 0
>>> Number of Bricks: 1 x (2 + 1) = 3
>>> Transport-type: tcp
>>> Bricks:
>>> Brick1: 10.54.95.123:/gluster/data/brick
>>> Brick2: 10.54.95.124:/gluster/data/brick
>>> Brick3: 10.54.95.125:/gluster/data/brick (arbiter)
>>> Options Reconfigured:
>>> network.inode-lru-limit: 20
>>> performance.md-cache-timeout: 600
>>> performance.cache-invalidation: on
>>> performance.stat-prefetch: on
>>> features.cache-invalidation-timeout: 600
>>> features.cache-invalidation: on
>>> network.ping-timeout: 5
>>> transport.address-family: inet
>>> storage.fips-mode-rchecksum: on
>>> nfs.disable: on
>>> performance.client-io-threads: on
>>>
>>> sudo vim /etc/fstab
>>> localhost:/data /data glusterfs
>>> defaults,_netdev  0 0
>>>
>>> sudo systemctl daemon-reload && sudo mount -a
>>> fio --name=test --filename=/data/wow --size=1G --readwrite=write
>>>
>>> Run status group 0 (all jobs):
>>>   WRITE: bw=109MiB/s (115MB/s), 109MiB/s-109MiB/s (115MB/s-115MB/s),
>>> io=1024MiB (1074MB), run=9366-9366msec
>>>
>>> Oh no, what's wrong? From 2182MB/s down to only 115MB/s? What am I
>>> missing? I'm not expecting the above ~17 Gbps, but I'm thinking it should
>>> at least be close(r) to ~10 Gbps.
>>>
>>>

Re: [Gluster-users] Gluster Performance - 12 Gbps SSDs and 10 Gbps NIC

2023-12-12 Thread Gilberto Ferreira

I strongly suggest you update to version 10 or higher.
It's come with significant improvement regarding performance.
---
Gilberto Nunes Ferreira
(47) 99676-7530 - Whatsapp / Telegram






Em ter., 12 de dez. de 2023 às 13:03, Danny 
escreveu:

> MTU is already 9000, and as you can see from the IPERF results, I've got a
> nice, fast connection between the nodes.
>
> On Tue, Dec 12, 2023 at 9:49 AM Strahil Nikolov 
> wrote:
>
>> Hi,
>>
>> Let’s try the simple things:
>>
>> Check if you can use MTU9000 and if it’s possible, set it on the Bond
>> Slaves and the bond devices:
>>  ping GLUSTER_PEER -c 10 -M do -s 8972
>>
>> Then try to follow up the recommendations from
>> https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/chap-configuring_red_hat_storage_for_enhancing_performance
>>
>>
>>
>> Best Regards,
>> Strahil Nikolov
>>
>> On Monday, December 11, 2023, 3:32 PM, Danny 
>> wrote:
>>
>> Hello list, I'm hoping someone can let me know what setting I missed.
>>
>> Hardware:
>> Dell R650 servers, Dual 24 Core Xeon 2.8 GHz, 1 TB RAM
>> 8x SSD s Negotiated Speed 12 Gbps
>> PERC H755 Controller - RAID 6
>> Created virtual "data" disk from the above 8 SSD drives, for a ~20 TB
>> /dev/sdb
>>
>> OS:
>> CentOS Stream
>> kernel-4.18.0-526.el8.x86_64
>> glusterfs-7.9-1.el8.x86_64
>>
>> IPERF Test between nodes:
>> [ ID] Interval   Transfer Bitrate Retr
>> [  5]   0.00-10.00  sec  11.5 GBytes  9.90 Gbits/sec0
>> sender
>> [  5]   0.00-10.04  sec  11.5 GBytes  9.86 Gbits/sec
>>  receiver
>>
>> All good there. ~10 Gbps, as expected.
>>
>> LVM Install:
>> export DISK="/dev/sdb"
>> sudo parted --script $DISK "mklabel gpt"
>> sudo parted --script $DISK "mkpart primary 0% 100%"
>> sudo parted --script $DISK "set 1 lvm on"
>> sudo pvcreate --dataalignment 128K /dev/sdb1
>> sudo vgcreate --physicalextentsize 128K gfs_vg /dev/sdb1
>> sudo lvcreate -L 16G -n gfs_pool_meta gfs_vg
>> sudo lvcreate -l 95%FREE -n gfs_pool gfs_vg
>> sudo lvconvert --chunksize 1280K --thinpool gfs_vg/gfs_pool
>> --poolmetadata gfs_vg/gfs_pool_meta
>> sudo lvchange --zero n gfs_vg/gfs_pool
>> sudo lvcreate -V 19.5TiB --thinpool gfs_vg/gfs_pool -n gfs_lv
>> sudo mkfs.xfs -f -i size=512 -n size=8192 -d su=128k,sw=10
>> /dev/mapper/gfs_vg-gfs_lv
>> sudo vim /etc/fstab
>> /dev/mapper/gfs_vg-gfs_lv   /gluster/data/brick   xfs
>> rw,inode64,noatime,nouuid 0 0
>>
>> sudo systemctl daemon-reload && sudo mount -a
>> fio --name=test --filename=/gluster/data/brick/wow --size=1G
>> --readwrite=write
>>
>> Run status group 0 (all jobs):
>>   WRITE: bw=2081MiB/s (2182MB/s), 2081MiB/s-2081MiB/s
>> (2182MB/s-2182MB/s), io=1024MiB (1074MB), run=492-492msec
>>
>> All good there. 2182MB/s =~ 17.5 Gbps. Nice!
>>
>>
>> Gluster install:
>> export NODE1='10.54.95.123'
>> export NODE2='10.54.95.124'
>> export NODE3='10.54.95.125'
>> sudo gluster peer probe $NODE2
>> sudo gluster peer probe $NODE3
>> sudo gluster volume create data replica 3 arbiter 1
>> $NODE1:/gluster/data/brick $NODE2:/gluster/data/brick
>> $NODE3:/gluster/data/brick force
>> sudo gluster volume set data network.ping-timeout 5
>> sudo gluster volume set data performance.client-io-threads on
>> sudo gluster volume set data group metadata-cache
>> sudo gluster volume start data
>> sudo gluster volume info all
>>
>> Volume Name: data
>> Type: Replicate
>> Volume ID: b52b5212-82c8-4b1a-8db3-52468bc0226e
>> Status: Started
>> Snapshot Count: 0
>> Number of Bricks: 1 x (2 + 1) = 3
>> Transport-type: tcp
>> Bricks:
>> Brick1: 10.54.95.123:/gluster/data/brick
>> Brick2: 10.54.95.124:/gluster/data/brick
>> Brick3: 10.54.95.125:/gluster/data/brick (arbiter)
>> Options Reconfigured:
>> network.inode-lru-limit: 20
>> performance.md-cache-timeout: 600
>> performance.cache-invalidation: on
>> performance.stat-prefetch: on
>> features.cache-invalidation-timeout: 600
>> features.cache-invalidation: on
>> network.ping-timeout: 5
>> transport.address-family: inet
>> storage.fips-mode-rchecksum: on
>> nfs.disable: on
>> performance.client-io-threads: on
>>
>> sudo vim /etc/fstab
>> localhost:/data /data glusterfs
>> defaults,_netdev  0 0
>>
>> sudo systemctl daemon-reload && sudo mount -a
>> fio --name=test --filename=/data/wow --size=1G --readwrite=write
>>
>> Run status group 0 (all jobs):
>>   WRITE: bw=109MiB/s (115MB/s), 109MiB/s-109MiB/s (115MB/s-115MB/s),
>> io=1024MiB (1074MB), run=9366-9366msec
>>
>> Oh no, what's wrong? From 2182MB/s down to only 115MB/s? What am I
>> missing? I'm not expecting the above ~17 Gbps, but I'm thinking it should
>> at least be close(r) to ~10 Gbps.
>>
>> Any suggestions?
>> 
>>
>>
>>
>> Community Meeting Calendar:
>>
>> Schedule -
>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>> Bridge: https://meet.google.com/cpu-eiue-hvk
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>
>> 
>
>
>
>

Re: [Gluster-users] Gluster Performance - 12 Gbps SSDs and 10 Gbps NIC

2023-12-12 Thread Danny

 MTU is already 9000, and as you can see from the IPERF results, I've got a
nice, fast connection between the nodes.

On Tue, Dec 12, 2023 at 9:49 AM Strahil Nikolov 
wrote:

> Hi,
>
> Let’s try the simple things:
>
> Check if you can use MTU9000 and if it’s possible, set it on the Bond
> Slaves and the bond devices:
>  ping GLUSTER_PEER -c 10 -M do -s 8972
>
> Then try to follow up the recommendations from
> https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/chap-configuring_red_hat_storage_for_enhancing_performance
>
>
>
> Best Regards,
> Strahil Nikolov
>
> On Monday, December 11, 2023, 3:32 PM, Danny 
> wrote:
>
> Hello list, I'm hoping someone can let me know what setting I missed.
>
> Hardware:
> Dell R650 servers, Dual 24 Core Xeon 2.8 GHz, 1 TB RAM
> 8x SSD s Negotiated Speed 12 Gbps
> PERC H755 Controller - RAID 6
> Created virtual "data" disk from the above 8 SSD drives, for a ~20 TB
> /dev/sdb
>
> OS:
> CentOS Stream
> kernel-4.18.0-526.el8.x86_64
> glusterfs-7.9-1.el8.x86_64
>
> IPERF Test between nodes:
> [ ID] Interval   Transfer Bitrate Retr
> [  5]   0.00-10.00  sec  11.5 GBytes  9.90 Gbits/sec0
> sender
> [  5]   0.00-10.04  sec  11.5 GBytes  9.86 Gbits/sec
>  receiver
>
> All good there. ~10 Gbps, as expected.
>
> LVM Install:
> export DISK="/dev/sdb"
> sudo parted --script $DISK "mklabel gpt"
> sudo parted --script $DISK "mkpart primary 0% 100%"
> sudo parted --script $DISK "set 1 lvm on"
> sudo pvcreate --dataalignment 128K /dev/sdb1
> sudo vgcreate --physicalextentsize 128K gfs_vg /dev/sdb1
> sudo lvcreate -L 16G -n gfs_pool_meta gfs_vg
> sudo lvcreate -l 95%FREE -n gfs_pool gfs_vg
> sudo lvconvert --chunksize 1280K --thinpool gfs_vg/gfs_pool --poolmetadata
> gfs_vg/gfs_pool_meta
> sudo lvchange --zero n gfs_vg/gfs_pool
> sudo lvcreate -V 19.5TiB --thinpool gfs_vg/gfs_pool -n gfs_lv
> sudo mkfs.xfs -f -i size=512 -n size=8192 -d su=128k,sw=10
> /dev/mapper/gfs_vg-gfs_lv
> sudo vim /etc/fstab
> /dev/mapper/gfs_vg-gfs_lv   /gluster/data/brick   xfs
> rw,inode64,noatime,nouuid 0 0
>
> sudo systemctl daemon-reload && sudo mount -a
> fio --name=test --filename=/gluster/data/brick/wow --size=1G
> --readwrite=write
>
> Run status group 0 (all jobs):
>   WRITE: bw=2081MiB/s (2182MB/s), 2081MiB/s-2081MiB/s (2182MB/s-2182MB/s),
> io=1024MiB (1074MB), run=492-492msec
>
> All good there. 2182MB/s =~ 17.5 Gbps. Nice!
>
>
> Gluster install:
> export NODE1='10.54.95.123'
> export NODE2='10.54.95.124'
> export NODE3='10.54.95.125'
> sudo gluster peer probe $NODE2
> sudo gluster peer probe $NODE3
> sudo gluster volume create data replica 3 arbiter 1
> $NODE1:/gluster/data/brick $NODE2:/gluster/data/brick
> $NODE3:/gluster/data/brick force
> sudo gluster volume set data network.ping-timeout 5
> sudo gluster volume set data performance.client-io-threads on
> sudo gluster volume set data group metadata-cache
> sudo gluster volume start data
> sudo gluster volume info all
>
> Volume Name: data
> Type: Replicate
> Volume ID: b52b5212-82c8-4b1a-8db3-52468bc0226e
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 1 x (2 + 1) = 3
> Transport-type: tcp
> Bricks:
> Brick1: 10.54.95.123:/gluster/data/brick
> Brick2: 10.54.95.124:/gluster/data/brick
> Brick3: 10.54.95.125:/gluster/data/brick (arbiter)
> Options Reconfigured:
> network.inode-lru-limit: 20
> performance.md-cache-timeout: 600
> performance.cache-invalidation: on
> performance.stat-prefetch: on
> features.cache-invalidation-timeout: 600
> features.cache-invalidation: on
> network.ping-timeout: 5
> transport.address-family: inet
> storage.fips-mode-rchecksum: on
> nfs.disable: on
> performance.client-io-threads: on
>
> sudo vim /etc/fstab
> localhost:/data /data glusterfs
> defaults,_netdev  0 0
>
> sudo systemctl daemon-reload && sudo mount -a
> fio --name=test --filename=/data/wow --size=1G --readwrite=write
>
> Run status group 0 (all jobs):
>   WRITE: bw=109MiB/s (115MB/s), 109MiB/s-109MiB/s (115MB/s-115MB/s),
> io=1024MiB (1074MB), run=9366-9366msec
>
> Oh no, what's wrong? From 2182MB/s down to only 115MB/s? What am I
> missing? I'm not expecting the above ~17 Gbps, but I'm thinking it should
> at least be close(r) to ~10 Gbps.
>
> Any suggestions?
> 
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://meet.google.com/cpu-eiue-hvk
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
>




Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster Performance - 12 Gbps SSDs and 10 Gbps NIC

2023-12-12 Thread Strahil Nikolov

Hi,
Let’s try the simple things:
Check if you can use MTU9000 and if it’s possible, set it on the Bond Slaves 
and the bond devices: ping GLUSTER_PEER -c 10 -M do -s 8972
Then try to follow up the recommendations from 
https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/chap-configuring_red_hat_storage_for_enhancing_performance
 

Best Regards,Strahil Nikolov 


On Monday, December 11, 2023, 3:32 PM, Danny  wrote:

Hello list, I'm hoping someone can let me know what setting I missed.
Hardware:Dell R650 servers, Dual 24 Core Xeon 2.8 GHz, 1 TB RAM
8x SSD s Negotiated Speed 12 GbpsPERC H755 Controller - RAID 6 
Created virtual "data" disk from the above 8 SSD drives, for a ~20 TB /dev/sdb

OS:CentOS Streamkernel-4.18.0-526.el8.x86_64glusterfs-7.9-1.el8.x86_64
IPERF Test between nodes:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  11.5 GBytes  9.90 Gbits/sec    0             sender
[  5]   0.00-10.04  sec  11.5 GBytes  9.86 Gbits/sec                  receiver

All good there. ~10 Gbps, as expected.

LVM Install:export DISK="/dev/sdb"
sudo parted --script $DISK "mklabel gpt"
sudo parted --script $DISK "mkpart primary 0% 100%"
sudo parted --script $DISK "set 1 lvm on"sudo pvcreate --dataalignment 128K 
/dev/sdb1
sudo vgcreate --physicalextentsize 128K gfs_vg /dev/sdb1
sudo lvcreate -L 16G -n gfs_pool_meta gfs_vg
sudo lvcreate -l 95%FREE -n gfs_pool gfs_vg
sudo lvconvert --chunksize 1280K --thinpool gfs_vg/gfs_pool --poolmetadata 
gfs_vg/gfs_pool_meta
sudo lvchange --zero n gfs_vg/gfs_pool
sudo lvcreate -V 19.5TiB --thinpool gfs_vg/gfs_pool -n gfs_lv
sudo mkfs.xfs -f -i size=512 -n size=8192 -d su=128k,sw=10 
/dev/mapper/gfs_vg-gfs_lv
sudo vim /etc/fstab/dev/mapper/gfs_vg-gfs_lv   /gluster/data/brick   xfs       
rw,inode64,noatime,nouuid 0 0
sudo systemctl daemon-reload && sudo mount -a
fio --name=test --filename=/gluster/data/brick/wow --size=1G --readwrite=write

Run status group 0 (all jobs):
  WRITE: bw=2081MiB/s (2182MB/s), 2081MiB/s-2081MiB/s (2182MB/s-2182MB/s), 
io=1024MiB (1074MB), run=492-492msec

All good there. 2182MB/s =~ 17.5 Gbps. Nice!


Gluster install:export NODE1='10.54.95.123'
export NODE2='10.54.95.124'
export NODE3='10.54.95.125'
sudo gluster peer probe $NODE2
sudo gluster peer probe $NODE3
sudo gluster volume create data replica 3 arbiter 1 $NODE1:/gluster/data/brick 
$NODE2:/gluster/data/brick $NODE3:/gluster/data/brick force
sudo gluster volume set data network.ping-timeout 5
sudo gluster volume set data performance.client-io-threads on
sudo gluster volume set data group metadata-cache
sudo gluster volume start data
sudo gluster volume info all

Volume Name: data
Type: Replicate
Volume ID: b52b5212-82c8-4b1a-8db3-52468bc0226e
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x (2 + 1) = 3
Transport-type: tcp
Bricks:
Brick1: 10.54.95.123:/gluster/data/brick
Brick2: 10.54.95.124:/gluster/data/brick
Brick3: 10.54.95.125:/gluster/data/brick (arbiter)
Options Reconfigured:
network.inode-lru-limit: 20
performance.md-cache-timeout: 600
performance.cache-invalidation: on
performance.stat-prefetch: on
features.cache-invalidation-timeout: 600
features.cache-invalidation: on
network.ping-timeout: 5
transport.address-family: inet
storage.fips-mode-rchecksum: on
nfs.disable: on
performance.client-io-threads: on
sudo vim /etc/fstab
localhost:/data             /data                 glusterfs defaults,_netdev    
  0 0
sudo systemctl daemon-reload && sudo mount -afio --name=test 
--filename=/data/wow --size=1G --readwrite=write
Run status group 0 (all jobs):
  WRITE: bw=109MiB/s (115MB/s), 109MiB/s-109MiB/s (115MB/s-115MB/s), io=1024MiB 
(1074MB), run=9366-9366msec
Oh no, what's wrong? From 2182MB/s down to only 115MB/s? What am I missing? I'm 
not expecting the above ~17 Gbps, but I'm thinking it should at least be 
close(r) to ~10 Gbps. 

Any suggestions?



Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users







Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster Performance - 12 Gbps SSDs and 10 Gbps NIC

Re: [Gluster-users] Gluster Performance - 12 Gbps SSDs and 10 Gbps NIC

Re: [Gluster-users] Gluster Performance - 12 Gbps SSDs and 10 Gbps NIC

Re: [Gluster-users] Gluster Performance - 12 Gbps SSDs and 10 Gbps NIC

Re: [Gluster-users] Gluster Performance - 12 Gbps SSDs and 10 Gbps NIC

Re: [Gluster-users] Gluster Performance - 12 Gbps SSDs and 10 Gbps NIC

Re: [Gluster-users] Gluster Performance - 12 Gbps SSDs and 10 Gbps NIC

Re: [Gluster-users] Gluster Performance - 12 Gbps SSDs and 10 Gbps NIC

Re: [Gluster-users] Gluster Performance - 12 Gbps SSDs and 10 Gbps NIC

Re: [Gluster-users] Gluster Performance - 12 Gbps SSDs and 10 Gbps NIC

Re: [Gluster-users] Gluster Performance - 12 Gbps SSDs and 10 Gbps NIC

Re: [Gluster-users] Gluster Performance - 12 Gbps SSDs and 10 Gbps NIC

Re: [Gluster-users] Gluster Performance - 12 Gbps SSDs and 10 Gbps NIC

Re: [Gluster-users] Gluster Performance - 12 Gbps SSDs and 10 Gbps NIC

Re: [Gluster-users] Gluster Performance - 12 Gbps SSDs and 10 Gbps NIC

15 matches

Site Navigation

Mail list logo

Footer information