On Sun, Feb 13, 2022 at 08:00:29PM -0800, Paul B. Henson wrote:
> I'm still trying to figure out why my servers sometimes get into a state
> where queries requesting the memberOf attribute take an exceedingly long
So one of my servers got into this state again:
Total DISK READ : 89.60 M/s | Total DISK WRITE : 241.97 K/s
[0/9406]
Actual DISK READ: 91.61 M/s | Actual DISK WRITE: 140.50 K/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
430373 be/4 ldap 34.88 M/s 0.00 B/s 0.00 % 96.44 % slapd -d 0 -h
ldap:/// ~dapi:/// -u ldap -g ldap
430064 be/4 ldap 45.04 M/s 0.00 B/s 0.00 % 94.93 % slapd -d 0 -h
ldap:/// ~dapi:/// -u ldap -g ldap
430069 be/4 ldap 6.34 M/s 0.00 B/s 0.00 % 8.22 % slapd -d 0 -h
ldap:/// ~dapi:/// -u ldap -g ldap
There's plenty of free memory:
ldap-02 ~ # free -m
total used free shared buff/cache available
Mem: 3901 418 113 0 3368 3255
Swap: 2047 763 1284
Just for giggles, I removed all swap:
total used free shared buff/cache available
Mem: 3901 730 102 1 3068 2949
Swap: 0 0 0
The problem immediately went away. Didn't restart slapd, didn't do anything,
other than remove all swap and force it to use the plethora of memory
it had available. Disk read went back to virtually 0, response time went
back to subsecond.
I updated vm.swappiness to 1 (it defaulted to 30) and added swap back, I'm
going to see what happens.
Have no idea what's causing it, but it seems somehow the system and slapd
get into a state where doing memberof queries make it read lmdb pages
from disk rather than keeping them in memory?
