Hi Thomas, Sorry it took me so long to get back to you.
I think the problem is that your slapd.conf uses LDAP Sync replication and not delta-syncrepl. I missed that at first because you have an accesslog database configured, so I assumed you were using delta-syncrepl, but your syncrepl consumers are actually not configured for it. I created a basic provider-consumer setup (configuration files attached) and the memory usage was low and stable. When I changed from delta-syncrepl back to LDAP Sync, the memory usage climbed quickly. I reproduced the same behaviour with recent (2.4.36) code from upstream, so it's not a Debian-specific problem. As per slapd.conf(5), the change to switch from delta-syncrepl back to LDAP Sync is to remove the logbase, logfilter, and syncdata attributes from the syncrepl configuration. Reversing that change, I think it might help if you added those three attributes to your syncrepl configurations. See the OpenLDAP admin guide and slapd.conf(5) for details. Your syncprov overlay also doesn't have a syncprov-checkpoint directive, which is optional but useful in case of a crash or unclean shutdown. The reason LDAP Sync is not efficient for your use case is that it transfers entire entries and not just changed attributes. So, if you add or remove a member from a group, the entire list of members for that group has to be transferred, which is of course pretty bad when your groups have a lot of members. Delta-syncrepl is better because it only transfers changed attributes. Does this help?
slapd-consumer.conf
Description: Binary data
slapd-provider.conf
Description: Binary data