Hello,
thanks for digging in it! As I understand, this is somehow first about reducing the size of used memory, not clearly identifying the leak itself. The duplication done per process was to avoid races if local changes has to be performed (e.g., by libssl internally on handling traffic), but maybe with the libssl3.x and new threading approach is no longer needed, if ever was necessary. What I would suggest is to make it configurable for now, so in case behaviour becomes unstable on high load, one can switch between modes. Cheers, Daniel On 14.05.24 10:36, Xenofon Karamanos via sr-dev wrote: > Hello all, > > I am currently looking into issue > https://github.com/kamailio/kamailio/issues/3823 > <https://github.com/kamailio/kamailio/issues/3823> regarding the > tls.reload and the constant increase in memory usage when it's called. > > I tried to look something up related to CRL and related functions for > what causing it but nothing to obvious. The only culprit i could find > was this function call in SSL_verify_load_locations in load_crl() > <https://github.com/kamailio/kamailio/blob/9c2705b566103f4fad0cad63bc1e36ba3cca621f/src/modules/tls/tls_domain.c#L674-L679>. > But the same exact function is also called in load_ca_list() with no > such behaviour. > > Next thing i notice, is that on the first rpc call of tls.reload, > there is a "significant" memory allocation. > What i mean by this is that when a fresh kamailio (5.8) is started up > (with -m 7, for low shared memory availability to try and replicate > the issue and children=2) and then check free shared memory usage > using `kamcmd core.shmmem` is around 2196096 (bytes i am assuming) for > my system. > > On the first call of tls.reload, it drops to 539248. That's a heavy > drop so i started digging why on so much allocation is caused. When > loading a tls configuration, ksr_fix_domain > <https://github.com/kamailio/kamailio/blob/9c2705b566103f4fad0cad63bc1e36ba3cca621f/src/modules/tls/tls_domain.c#L1071> > allocates > a new SSL_CTX per process. From what i am seeing this SSL_CTX, is > basically the same for each one, with same configuration without any > difference on settings. > > So, what i tried is to allocated it once (see tls_reload branch in my > repo > <https://github.com/kamailio/kamailio/compare/master...xkaraman:kamailio:tls_reload>), > and just share this context to all of the process and the memory used > is way less and also the unbounding memory seems to fixed or at least > it's slowed that's barely visible. > > What are your thoughts on these? Is this something we can done? Is the > assumption that tls config will and be the same for every process or > can one somehow differentiate from another? > > Thanks, > Xenofon > > Some numbers for anyone interested: > > *Before the patch 5.8 branch* and with settings such as -m 7 (shared > memory) and children=2 (with children=8 it doesn't even start, failed > to load cyphers in SSL_CTX): > Fresh kamailio start shared memory: total: 7340032 > free: 2196096 > > After one tls.reload: free: 539248 > After 100 tls.reload: free: 483776 > > > *After patch on master* (kamailio -m 7 and children=2) (with > children=8 it starts normally): > Fresh kamailio start shared memory: total: 7340032 > free: 3772000 > > After one tls.reload: free: 3690192 > After 100 tls.reload: free: 3681056 > > <https://github.com/kamailio/kamailio/compare/master...xkaraman:kamailio:tls_reload> > > Comparing kamailio:master...xkaraman:tls_reload · kamailio/kamailio > <https://github.com/kamailio/kamailio/compare/master...xkaraman:kamailio:tls_reload> > Kamailio - The Open Source SIP Server for large VoIP and real-time > communication platforms - - Comparing > kamailio:master...xkaraman:tls_reload · kamailio/kamailio > github.com > > > <https://github.com/xkaraman/kamailio/blob/9c2705b566103f4fad0cad63bc1e36ba3cca621f/src/modules/tls/tls_domain.c#L1077> > > kamailio/src/modules/tls/tls_domain.c at > 9c2705b566103f4fad0cad63bc1e36ba3cca621f · xkaraman/kamailio > <https://github.com/xkaraman/kamailio/blob/9c2705b566103f4fad0cad63bc1e36ba3cca621f/src/modules/tls/tls_domain.c#L1077> > Kamailio - The Open Source SIP Server for large VoIP and real-time > communication platforms - - xkaraman/kamailio > github.com > > > <https://github.com/kamailio/kamailio/blob/9c2705b566103f4fad0cad63bc1e36ba3cca621f/src/modules/tls/tls_domain.c#L674-L679> > > kamailio/src/modules/tls/tls_domain.c at > 9c2705b566103f4fad0cad63bc1e36ba3cca621f · kamailio/kamailio > <https://github.com/kamailio/kamailio/blob/9c2705b566103f4fad0cad63bc1e36ba3cca621f/src/modules/tls/tls_domain.c#L674-L679> > Kamailio - The Open Source SIP Server for large VoIP and real-time > communication platforms - - kamailio/kamailio > github.com > > > <https://github.com/kamailio/kamailio/issues/3823> > > Memory usage increases everytime tls.reload is executed · Issue #3823 > · kamailio/kamailio <https://github.com/kamailio/kamailio/issues/3823> > Description We are using Kamailio 5.7.4 on Debian 12 (from > http://deb.kamailio.org/kamailio57) with rtpengine as an Edgeproxy for > our clients. The instance terminates SIP/TLS (with Cliencertificate... > github.com > > > > _______________________________________________ > Kamailio (SER) - Development Mailing List > To unsubscribe send an email to sr-dev-le...@lists.kamailio.org -- Daniel-Constantin Mierla (@ asipto.com) twitter.com/miconda -- linkedin.com/in/miconda Kamailio Consultancy, Training and Development Services -- asipto.com
_______________________________________________ Kamailio (SER) - Development Mailing List To unsubscribe send an email to sr-dev-le...@lists.kamailio.org
