Hi all,
I would like to better understand gluster performance and how to profile/analyze them.

I set up two old test machine each with a quad-core i7 CPU, 8 GB RAM and 4x 5400 RPM disks in software RAID 10. OS is CentOS 8.1 and I am using Gluster 6.7. To avoid being limited by the mechanical HDD, I created a small replica 2 volume using /dev/shm/gluster for both bricks.

Testing using fio with 4k random write and fsync=1, I see the following:
- write to /dev/shm: almost 300K IOPs (expected, as I am writing in memory);
- write to /mnt/fuse (where the volume is mounted): 250 IOPs;
- as above, but something I get as low as 20-30 IOPs for minutes.

Writing to /mnt/fuse without fsync=1 I get about 5500 IOPs, without the strange (and very low) drops. In this case, I seems limited by CPU because both glusterd and glusterfsd are near/above 100% core utilization.

Further details and volume info can be found below [1]

So, I have some questions:
- why performance are so low with fsync?
- why do I have so low IOPs (20/30) for minutes?
- what is capping the non-fsync test?
- why both glusterd and glusterfd are so CPU intensive? I can understand glusterfd itself requiring more CPU, but glusterd should only manage the other processes and send the volume information to asking client, right?

Thanks.

[1] [root@blackhole gluster]# gluster volume info tv0
Volume Name: tv0
Type: Replicate
Volume ID: 96534afe-bfde-4d60-a94d-379278ab45c4
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: singularity:/dev/shm/gluster
Brick2: blackhole:/dev/shm/gluster
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off

[root@blackhole tmp]# mount | grep glusterfs
localhost:tv0 on /mnt/fuse type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)

[root@blackhole glusterfs]# fio --name=test --filename=/mnt/fuse/test.img --size=256M --rw=randwrite --fsync=1 test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [w(1)][11.6%][r=0KiB/s,w=976KiB/s][r=0,w=244 IOPS][eta 04m:19s]

[root@blackhole tmp]# fio --name=test --filename=/mnt/fuse/test.img --size=256M --rw=randwrite --fsync=1 test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [w(1)][1.8%][r=0KiB/s,w=112KiB/s][r=0,w=28 IOPS][eta 38m:16s]

[root@blackhole tmp]# fio --name=test --filename=/mnt/fuse/test.img --size=256M --rw=randwrite test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=21.3MiB/s][r=0,w=5462 IOPS][eta 00m:00s]

[root@blackhole gluster]# gluster volume heal tv0 info
Brick singularity:/dev/shm/gluster
Status: Connected
Number of entries: 0
Brick blackhole:/dev/shm/gluster
Status: Connected
Number of entries: 0

--
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.da...@assyoma.it - i...@assyoma.it
GPG public key ID: FF5F32A8
________

Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Reply via email to