Re: [Gluster-users] Gluster-induced load averages of 50-200

[Darrell Budic]

I’ve had issues like that while playing with different healing config, too many 
shd threads will drive load higher than it needs to be and keep the boxes busy. 
I’ve had better luck with

cluster.shd-max-threads: 1
cluster.shd-wait-qlength: 1024

than 8 & 1. It does heal a little faster with 8/10k than 1/1k, but it keeps 
the servers too busy to do anything else. In a hyper converged install, I could 
see it causing problems for the VMs and keeping healing from effectively 
happening (it would cause disk read issues for me even with a separate server 
cluster).

Also, what version are you running? If it’s 3.12, I’m finding that .9-.11 have 
a client side memory leak, just havn’t gotten everything together enough to 
post about it. .8 seems ok, and .6 is definitely ok though, flat memory use 
from both, steadily increasing from .9 and .11 clients.

Good luck!

  -Darrell
> From: Jim Kusznir 
> Subject: [Gluster-users] Gluster-induced load averages of 50-200
> Date: July 6, 2018 at 11:43:21 AM CDT
> To: gluster-user
> 
> Hello:
> 
> I have a hyperconverged ovirt cluster with 3 replica 2+arb and one replica 3 
> volume.  My replica 3 volume has been having lots of problems for a while now.
> 
> First, after a week or two, I notice a lot of RAM usage (half of my physical 
> ram, plus lots of swap) for the replica 3 gluster process.  The amount of ram 
> in use varies by machine, but they all have had the problem show up (one at a 
> time).  Often the OOM killer kills it off before I catch it.
> 
> This last time, I caught it.  I migrated all the machines off the problem 
> server, and rebooted it.  That was yesterday (thurs at 9am).  Shortly after 
> coming back up, ALL 3 of my servers are showing very high load averages (cpu 
> % as reported by top is fairly low, althoguh iowait will spike to 25%).  The 
> one process that is using the most CPU is the glusterfsd process for the 
> replica 3 volume.  It is consistant on all 3 machines, and it loads down the 
> machine and IO enough to be causing issues with the VMs running on it (even 
> IO to a seprate disk on the same machine) .  
> 
> I watched network I/O and I can see occasional spikes to 50MB/s (gig link), 
> but its not consistant, and often spends considerable time near zero or in 
> kB/s range.  the repica 3 has its own dedicated gig network.
> 
> As of last night I still had 4 unhealed items for the server I rebooted.  
> This AM, I'm down to 1, but its still there, and its still causing 
> considerable performance issues.  On a server, if I stop that glusterfsd 
> process, then the loadavg returns to normal quickly.
> 
> HELP...what do I do?  Lately it seems that all my outages and performace 
> issues have been glusterfsd taking down my servers that they are running on.  
> Is gluster ready for production, or did I misunderstand?  My reliability has 
> been abismal on my vm cluster, and I still have 3 VMs that won't boot anymore 
> due to disk corruption caused by my last gluster outage (which was the high 
> RAM until it crashed on the server due to kernel OOM killer).
> 
> Here's some info on the affected gluster volume:
> Volume Name: data-hdd
> Type: Replicate
> Volume ID: d342a3ab-16f3-49f0-bbcf-f788be8ac5f1
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 1 x 3 = 3
> Transport-type: tcp
> Bricks:
> Brick1: 172.172.1.11:/gluster/brick3/data-hdd
> Brick2: 172.172.1.12:/gluster/brick3/data-hdd
> Brick3: 172.172.1.13:/gluster/brick3/data-hdd
> Options Reconfigured:
> diagnostics.count-fop-hits: on
> diagnostics.latency-measurement: on
> transport.address-family: inet
> performance.readdir-ahead: on
> geo-replication.indexing: on
> geo-replication.ignore-pid-check: on
> changelog.changelog: on
> performance.quick-read: off
> performance.read-ahead: off
> performance.io-cache: off
> performance.stat-prefetch: off
> performance.low-prio-threads: 32
> network.remote-dio: enable
> cluster.eager-lock: off
> cluster.quorum-type: auto
> cluster.server-quorum-type: server
> cluster.data-self-heal-algorithm: full
> cluster.locking-scheme: granular
> cluster.shd-max-threads: 8
> cluster.shd-wait-qlength: 1
> features.shard: on
> user.cifs: off
> storage.owner-uid: 36
> storage.owner-gid: 36
> server.allow-insecure: on
> network.ping-timeout: 30
> 
> ---
> 
> [root@ovirt3 ~]# gluster volume profile data-hdd info
> Brick: 172.172.1.13:/gluster/brick3/data-hdd
> 
> Cumulative Stats:
>Block Size:256b+ 512b+
> 1024b+
>  No. of Reads:   188488  6126 
> 17234
> No. of Writes:   80331655  
> 3899
> 
>Block Size:   2048b+4096b+
> 8192b+
>  No. of Reads:17228   1371069
> 265013
> No. of Writes: 6365  26617689  
> 15776028

[Gluster-users] Updated Gluster Releases

[Amye Scavarda]

*The Gluster community has released an out-of-normal-cadence release for
Gluster 3.12, and 4.1 that resolves a CVE[1]. A privilege escalation flaw
was found.Glusterfs is vulnerable to privilege escalation on gluster server
nodes. An authenticated gluster client via TLS could use gluster cli with
--remote-host command to add it self to trusted storage pool and perform
privileged gluster operations like adding other machines to trusted storage
pool, start, stop, and delete volumes. Installing the updated packages and
restarting gluster services on gluster brick hosts, will help prevent the
security issue. Further information can be found at NVD[2].Our
recommendation is to upgrade to these new releases:
https://download.gluster.org/pub/gluster/glusterfs/3.12/3.12.11/
https://download.gluster.org/pub/gluster/glusterfs/4.0/4.1.1/
 [1]
http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-10841
 [2]
https://nvd.nist.gov/vuln/detail/CVE-2018-10841
 *


-- 
Amye Scavarda | a...@redhat.com | Gluster Community Lead
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Gluster-induced load averages of 50-200

[Jim Kusznir]

Hello:

I have a hyperconverged ovirt cluster with 3 replica 2+arb and one replica
3 volume.  My replica 3 volume has been having lots of problems for a while
now.

First, after a week or two, I notice a lot of RAM usage (half of my
physical ram, plus lots of swap) for the replica 3 gluster process.  The
amount of ram in use varies by machine, but they all have had the problem
show up (one at a time).  Often the OOM killer kills it off before I catch
it.

This last time, I caught it.  I migrated all the machines off the problem
server, and rebooted it.  That was yesterday (thurs at 9am).  Shortly after
coming back up, ALL 3 of my servers are showing very high load averages
(cpu % as reported by top is fairly low, althoguh iowait will spike to
25%).  The one process that is using the most CPU is the glusterfsd process
for the replica 3 volume.  It is consistant on all 3 machines, and it loads
down the machine and IO enough to be causing issues with the VMs running on
it (even IO to a seprate disk on the same machine) .

I watched network I/O and I can see occasional spikes to 50MB/s (gig link),
but its not consistant, and often spends considerable time near zero or in
kB/s range.  the repica 3 has its own dedicated gig network.

As of last night I still had 4 unhealed items for the server I rebooted.
This AM, I'm down to 1, but its still there, and its still causing
considerable performance issues.  On a server, if I stop that glusterfsd
process, then the loadavg returns to normal quickly.

HELP...what do I do?  Lately it seems that all my outages and performace
issues have been glusterfsd taking down my servers that they are running
on.  Is gluster ready for production, or did I misunderstand?  My
reliability has been abismal on my vm cluster, and I still have 3 VMs that
won't boot anymore due to disk corruption caused by my last gluster outage
(which was the high RAM until it crashed on the server due to kernel OOM
killer).

Here's some info on the affected gluster volume:
Volume Name: data-hdd
Type: Replicate
Volume ID: d342a3ab-16f3-49f0-bbcf-f788be8ac5f1
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 172.172.1.11:/gluster/brick3/data-hdd
Brick2: 172.172.1.12:/gluster/brick3/data-hdd
Brick3: 172.172.1.13:/gluster/brick3/data-hdd
Options Reconfigured:
diagnostics.count-fop-hits: on
diagnostics.latency-measurement: on
transport.address-family: inet
performance.readdir-ahead: on
geo-replication.indexing: on
geo-replication.ignore-pid-check: on
changelog.changelog: on
performance.quick-read: off
performance.read-ahead: off
performance.io-cache: off
performance.stat-prefetch: off
performance.low-prio-threads: 32
network.remote-dio: enable
cluster.eager-lock: off
cluster.quorum-type: auto
cluster.server-quorum-type: server
cluster.data-self-heal-algorithm: full
cluster.locking-scheme: granular
cluster.shd-max-threads: 8
cluster.shd-wait-qlength: 1
features.shard: on
user.cifs: off
storage.owner-uid: 36
storage.owner-gid: 36
server.allow-insecure: on
network.ping-timeout: 30

---

[root@ovirt3 ~]# gluster volume profile data-hdd info
Brick: 172.172.1.13:/gluster/brick3/data-hdd

Cumulative Stats:
   Block Size:256b+ 512b+
1024b+
 No. of Reads:   188488  6126
 17234
No. of Writes:   80331655
3899

   Block Size:   2048b+4096b+
8192b+
 No. of Reads:17228   1371069
265013
No. of Writes: 6365  26617689
15776028

   Block Size:  16384b+   32768b+
 65536b+
 No. of Reads:   350572636120
189538
No. of Writes:  4475250   1557330
604455

   Block Size: 131072b+  262144b+
 No. of Reads:  6442414 0
No. of Writes:  519496413
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
 Fop
 -   ---   ---   ---   

  0.00   0.00 us   0.00 us   0.00 us 63
FORGET
  0.00   0.00 us   0.00 us   0.00 us   9356
 RELEASE
  0.00   0.00 us   0.00 us   0.00 us 113830
RELEASEDIR
  0.00 765.33 us 102.00 us2091.00 us  3
 FGETXATTR
  0.00 352.90 us 263.00 us1397.00 us 31
TRUNCATE
  0.001094.05 us 360.00 us5643.00 us 21
 MKDIR
  0.00 647.01 us 260.00 us4059.00 us 83
 MKNOD
  0.00 899.64 us  68.00 us   27531.00 us105
SETXATTR
  0.00 139.19 us  26.00 us4619.00 us740
 ENTRYLK
  0.002121.54 us1250.00 us9726.00 us 63
RENAME
  0.00 254.93 us  18.00 us  169316.00 us