[389-users] Re: autosizing the cache

[Sergei Gerasenko]

Hi William,

> On Mar 19, 2018, at 9:18 PM, William Brown  wrote:
> 
> yeah, dncachesize is manual. But I think dncachesize is per backend,
> not part of cn=config,cn=ldbm plugin. 

Yes, I see one for changelog and one for userRoot.

Here’s the data:

> dbscan -f /var/lib/dirsrv/slapd--XXX/db/userRoot/id2entry.db -t 200 | 
> grep -c rdn:
12212

> dbmon.sh
Tue Mar 20 02:16:04 UTC 2018
dbcachefree 432873472 free% 80.629 roevicts 0 hit% 99 pagein 1267 pageout 23191
changelog:ent 616509293897  100.0  4359.2
changelog:dn29262999793332  100.070.6
 userroot:ent  117776408341275   98.4  8594.6
 userroot:dn   117772998722450  100.0   108.5

> ls -lh /var/lib/dirsrv/slapd--XXX/db/userRoot
total 121M
-rw--- 1 dirsrv dirsrv  16K Mar 19 14:41 aci.db
-rw--- 1 dirsrv dirsrv 744K Mar 19 21:04 ancestorid.db
-rw--- 1 dirsrv dirsrv  16K Mar 19 14:41 automountkey.db
-rw--- 1 dirsrv dirsrv 5.7M Mar 19 18:39 cn.db
-rw--- 1 dirsrv dirsrv   51 Mar 19 14:44 DBVERSION
-rw--- 1 dirsrv dirsrv 296K Mar 19 14:41 displayname.db
-rw--- 1 dirsrv dirsrv 4.7M Mar 19 21:04 entryrdn.db
-rw--- 1 dirsrv dirsrv 144K Mar 20 02:23 entryusn.db
-rw--- 1 dirsrv dirsrv 592K Mar 19 18:15 fqdn.db
-rw--- 1 dirsrv dirsrv  56K Mar 19 14:41 gidnumber.db
-rw--- 1 dirsrv dirsrv 176K Mar 19 14:41 givenName.db
-rw--- 1 dirsrv dirsrv  74M Mar 20 02:23 id2entry.db
-rw--- 1 dirsrv dirsrv  16K Mar 19 14:44 ipaallowedtarget.db
-rw--- 1 dirsrv dirsrv  16K Mar 19 14:44 ipaassignedidview.db
-rw--- 1 dirsrv dirsrv  16K Mar 19 18:15 ipakrbprincipalalias.db
-rw--- 1 dirsrv dirsrv  16K Mar 19 14:44 ipalocation.db
-rw--- 1 dirsrv dirsrv  16K Mar 19 14:44 ipaMemberCa.db
-rw--- 1 dirsrv dirsrv  16K Mar 19 14:41 ipaMemberCertProfile.db
-rw--- 1 dirsrv dirsrv 112K Mar 19 14:41 ipasudorunas.db
-rw--- 1 dirsrv dirsrv  16K Mar 19 14:44 ipasudorunasgroup.db
-rw--- 1 dirsrv dirsrv  16K Mar 19 14:44 ipatokenradiusconfiglink.db
-rw--- 1 dirsrv dirsrv 952K Mar 19 21:04 ipauniqueid.db
-rw--- 1 dirsrv dirsrv 2.8M Mar 19 18:15 krbCanonicalName.db
-rw--- 1 dirsrv dirsrv 7.2M Mar 19 18:15 krbPrincipalName.db
-rw--- 1 dirsrv dirsrv  16K Mar 19 14:44 macAddress.db
-rw--- 1 dirsrv dirsrv 416K Mar 19 14:41 mail.db
-rw--- 1 dirsrv dirsrv 8.1M Mar 19 18:15 managedby.db
-rw--- 1 dirsrv dirsrv  16K Mar 19 14:44 manager.db
-rw--- 1 dirsrv dirsrv 264K Mar 19 21:04 memberallowcmd.db
-rw--- 1 dirsrv dirsrv 3.3M Mar 19 19:09 member.db
-rw--- 1 dirsrv dirsrv  16K Mar 19 14:44 memberdenycmd.db
-rw--- 1 dirsrv dirsrv 1.7M Mar 19 14:41 memberHost.db
-rw--- 1 dirsrv dirsrv 2.0M Mar 19 19:09 memberOf.db
-rw--- 1 dirsrv dirsrv 128K Mar 19 14:41 memberservice.db
-rw--- 1 dirsrv dirsrv 472K Mar 19 14:41 memberUser.db
-rw--- 1 dirsrv dirsrv  48K Mar 19 20:12 nscpEntryDN.db
-rw--- 1 dirsrv dirsrv  16K Mar 19 14:41 nsds5ReplConflict.db
-rw--- 1 dirsrv dirsrv  32K Mar 19 20:12 nsTombstoneCSN.db
-rw--- 1 dirsrv dirsrv 920K Mar 19 21:04 nsuniqueid.db
-rw--- 1 dirsrv dirsrv  16K Mar 19 21:04 numsubordinates.db
-rw--- 1 dirsrv dirsrv 1.3M Mar 19 21:04 objectclass.db
-rw--- 1 dirsrv dirsrv  16K Mar 19 14:41 ou.db
-rw--- 1 dirsrv dirsrv  16K Mar 19 14:44 owner.db
-rw--- 1 dirsrv dirsrv 176K Mar 19 21:04 parentid.db
-rw--- 1 dirsrv dirsrv  16K Mar 19 14:44 secretary.db
-rw--- 1 dirsrv dirsrv  16K Mar 19 14:44 seeAlso.db
-rw--- 1 dirsrv dirsrv 200K Mar 19 14:41 sn.db
-rw--- 1 dirsrv dirsrv  16K Mar 19 14:44 sourcehost.db
-rw--- 1 dirsrv dirsrv  16K Mar 19 18:39 telephoneNumber.db
-rw--- 1 dirsrv dirsrv  96K Mar 19 14:41 title.db
-rw--- 1 dirsrv dirsrv 184K Mar 19 14:41 uid.db
-rw--- 1 dirsrv dirsrv  56K Mar 19 14:41 uidnumber.db
-rw--- 1 dirsrv dirsrv  16K Mar 19 14:44 uniquemember.db
-rw--- 1 dirsrv dirsrv 4.4M Mar 19 18:15 userCertificate.db

Thank you!
  Sergei___
389-users mailing list -- 389-users@lists.fedoraproject.org
To unsubscribe send an email to 389-users-le...@lists.fedoraproject.org

[389-users] Re: autosizing the cache

[William Brown]

On Mon, 2018-03-19 at 20:58 -0500, Sergei Gerasenko wrote:
> > Dogtag is Java/Tomcat. It's well known for consuming large volumes
> > of
> > ram!
> 
> Ah, I thought it was something besides that :)

Nope, just that :) 

> 
> > Sure, sounds reasonable to me - I'd want to see your database sizes
> > to
> > make a complete assesment, but it seems pretty reasonable to me. 
> 
> I will get that for you. I’ve used dbmon.sh on a live system and
> things looked pretty well (>=92%). You would like to see the number
> of entries in the db, right?
> 

Yeah I think that should do it. It would be good to know the filesizes
in the backend too. 

> > Additionally, check that cn=config,cn=ldbm ... is actually
> > d'B'cachesize not d’N'cachesize.
> 
> Now I’m confused a bit. I thought the dncachesize was what was
> supposed to be set manually while everything else was autosized.
> Sorry for being slow on this.

yeah, dncachesize is manual. But I think dncachesize is per backend,
not part of cn=config,cn=ldbm plugin. 

> ___
> 389-users mailing list -- 389-users@lists.fedoraproject.org
> To unsubscribe send an email to 389-users-leave@lists.fedoraproject.o
> rg
-- 
Thanks,

William Brown
___
389-users mailing list -- 389-users@lists.fedoraproject.org
To unsubscribe send an email to 389-users-le...@lists.fedoraproject.org

[389-users] Re: autosizing the cache

[Sergei Gerasenko]

> Dogtag is Java/Tomcat. It's well known for consuming large volumes of
> ram!

Ah, I thought it was something besides that :)

> Sure, sounds reasonable to me - I'd want to see your database sizes to
> make a complete assesment, but it seems pretty reasonable to me. 

I will get that for you. I’ve used dbmon.sh on a live system and things looked 
pretty well (>=92%). You would like to see the number of entries in the db, 
right?

> Additionally, check that cn=config,cn=ldbm ... is actually
> d'B'cachesize not d’N'cachesize.

Now I’m confused a bit. I thought the dncachesize was what was supposed to be 
set manually while everything else was autosized. Sorry for being slow on this.
___
389-users mailing list -- 389-users@lists.fedoraproject.org
To unsubscribe send an email to 389-users-le...@lists.fedoraproject.org

[389-users] Re: autosizing the cache

[William Brown]

On Sun, 2018-03-18 at 21:57 -0500, Sergei Gerasenko wrote:
> Thank you for the detailed response, William. That’s great info. You
> mentioned FreeIPA in passing and that’s actually what I use 389-ds
> for. You mentioned dogtag eating memory. You mean it has a memory
> leak or some other memory mismanagement issue? 

Dogtag is Java/Tomcat. It's well known for consuming large volumes of
ram!

I actually wrote the autotuning system specifically for FreeIPA since
it historically did no DS tuning. So I'm glad that you are finding it
useful! 

> 
> I have 125G of RAM on my systems. So, I can allocate quite a bit of
> RAM to the caches. My plan is to set nsslapd-cache-autosize to 20%
> and set the dncache size to 5G for all three backends. Does that
> sound reasonable to you? These are dedicated FreeIPA machines, so
> it’s the main consumer of the memory.

Sure, sounds reasonable to me - I'd want to see your database sizes to
make a complete assesment, but it seems pretty reasonable to me. 

> 
> Also, can I set the dncachesize once at the cn=config,cn=ldbm
> database,cn=plugins,cn=config level and have all the other backends
> inherit that? Would I have to remove that parameter from the other
> backends for it to become inherited?

Sadly there is no method to inherit cache sizes.

Additionally, check that cn=config,cn=ldbm ... is actually
d'B'cachesize not d'N'cachesize.

Hope that helps! 

> 
> Thanks again, William!
>   Sergei
>  
> > On Mar 18, 2018, at 8:06 PM, William Brown  > u> wrote:
> > 
> > On Tue, 2018-03-13 at 21:10 -0500, Sergei Gerasenko wrote:
> > > Hi William,
> > > 
> > > With autosizing on I configured the changelog backend:
> > > 
> > > dn: cn=changelog,cn=ldbm database,cn=plugins,cn=config
> > > cn: changelog
> > > objectClass: top
> > > objectClass: extensibleObject
> > > objectClass: nsBackendInstance
> > > nsslapd-suffix: cn=changelog
> > > nsslapd-cachesize: -1
> > > nsslapd-cachememsize: 8858370048
> > > nsslapd-readonly: off
> > > nsslapd-require-index: off
> > > nsslapd-directory: /var/lib/dirsrv/slapd-CNVR-NET/db/changelog
> > > nsslapd-dncachememsize: 30
> > > 
> > > Here’s the ldif:
> > > 
> > > dn: cn=changelog,cn=ldbm database,cn=plugins,cn=config
> > > changetype: modify
> > > replace: nsslapd-dncachememsize
> > > nsslapd-dncachememsize: 30
> > > -
> > > replace: nsslapd-cachememsize
> > > nsslapd-cachememsize: 30
> > > 
> > > The server refused to change nsslapd-cachememsize because of
> > > autosizing but nsslapd-dncachememsize did get changed. So it
> > > seems
> > > that I can still control nsslapd-dncachememsize? When I restart
> > > 389-
> > > ds, the 3G persists as well.
> > > 
> > > Does that make sense?
> > 
> > Yes it does.
> > 
> > > 
> > > Thank you for a quick response!
> > 
> > At the current point in time, dncachememsize is NOT controlled by
> > autosizing. There is some ideas in my head about how to improve
> > this,
> > but the issue is legacy. We have one control: autosize percentage.
> > But
> > this doesn't really represent a complete picture and story about
> > how we
> > size our backends and data.
> > 
> > We have to continue to support the current variables, so I can't
> > easily
> > change this from it's current form. 
> > 
> > *IF* I was given unlimited power (I never will be ;) ) I would
> > probably
> > make the interface something like:
> > 
> > 
> > 
> > backend-foo:
> > autosize-max-memory: 
> > entrycache-slice: A%
> > dncache-slice: B%
> > dbcache-slice: C%
> > ndncache-slice: D%
> > 
> > So how would this work? 
> > 
> > You have autosize max that is the "total ram" we want to use. Lets
> > say
> > 2GB. Each backend would be given it's own "limit". We'd try to
> > detect
> > some sane values on your system of course, but we can only do so
> > much
> > :) So say ... each backend gets 5-10% of system ram limit. (We have
> > to
> > be super conservative due to dogtag in freeipa which eats ram).
> > 
> > Then we slice that up. So entrycache-slice + dncache-slice +
> > dbcache-
> > slice + ndncache-slice = 100%. So by default we might do something
> > like:
> > 
> > entrycache-slice: 75
> > dbcache-slice: 10
> > dncache-slice: 10
> > ndncache-slice: 5
> > 
> > So this on a 1GB backend translates to:
> > 
> > 750MB of entrycache
> > 100MB of dncache
> > 100MB of dbcache
> > 50MB of ndncache
> > 
> > of course some tests would be needed to find the right slice values
> > by
> > default.
> > 
> > 
> > Then it's as simple as "if you want more from your backend" just up
> > the
> > one memlimit value and "everything" increases in a correct
> > proportional
> > rate. Rather than thinking about how to hand tune everything, we
> > make
> > informed design decisions, you just say "here's mem max". You have
> > control to say each backend gets a different value (say FreeIPA you
> > may
> > want 4GB to userRoot, 1GB to CA system backend). you still have
> > lots of
> > flexibility as an admin, but you don't have