This probably went unnoticed until now.
On Mon, Apr 22, 2019 at 10:45 AM Zhou, Cynthia (NSB - CN/Hangzhou) < cynthia.z...@nokia-sbell.com> wrote: > Why there is no bio_free called in ssl_teardown_connection then? > > > > cynthia > > > > *From:* Milind Changire <mchan...@redhat.com> > *Sent:* Monday, April 22, 2019 10:21 AM > *To:* Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.z...@nokia-sbell.com> > *Cc:* Atin Mukherjee <amukh...@redhat.com>; gluster-devel@gluster.org > *Subject:* Re: [Gluster-devel] glusterfsd memory leak issue found after > enable ssl > > > > After patch 22334 <https://review.gluster.org/c/glusterfs/+/22334>, the > priv->ssl_ctx is now maintained per socket connection and is no longer > shared. > > So you might want to SSL_free(priv->ssl_ctx) as well and set priv->ssl_ctx > to NULL. > > > > There might be some strings that are duplicated (gf_strdup()) via the > socket_init() code path. Please take a look at those as well. > > > > Sorry about that. I missed it. > > > > > > On Mon, Apr 22, 2019 at 7:25 AM Zhou, Cynthia (NSB - CN/Hangzhou) < > cynthia.z...@nokia-sbell.com> wrote: > > > > Hi, > > From my code study it seems priv->ssl_ssl is not properly released, I made > a patch and the glusterfsd memory leak is alleviated with my patch, but > some otherwhere is still leaking, I have no clue about the other leak > points. > > > > --- a/rpc/rpc-transport/socket/src/socket.c > > +++ b/rpc/rpc-transport/socket/src/socket.c > > @@ -1019,7 +1019,16 @@ static void __socket_reset(rpc_transport_t *this) { > > memset(&priv->incoming, 0, sizeof(priv->incoming)); > > event_unregister_close(this->ctx->event_pool, priv->sock, priv->idx); > > - > > + if(priv->use_ssl&& priv->ssl_ssl) > > + { > > + gf_log(this->name, GF_LOG_TRACE, > > + "clear and reset for socket(%d), free ssl ", > > + priv->sock); > > + SSL_shutdown(priv->ssl_ssl); > > + SSL_clear(priv->ssl_ssl); > > + SSL_free(priv->ssl_ssl); > > + priv->ssl_ssl = NULL; > > + } > > priv->sock = -1; > > priv->idx = -1; > > priv->connected = -1; > > @@ -4238,6 +4250,16 @@ void fini(rpc_transport_t *this) { > > pthread_mutex_destroy(&priv->out_lock); > > pthread_mutex_destroy(&priv->cond_lock); > > pthread_cond_destroy(&priv->cond); > > + if(priv->use_ssl&& priv->ssl_ssl) > > + { > > + gf_log(this->name, GF_LOG_TRACE, > > + "clear and reset for socket(%d), free ssl ", > > + priv->sock); > > + SSL_shutdown(priv->ssl_ssl); > > + SSL_clear(priv->ssl_ssl); > > + SSL_free(priv->ssl_ssl); > > + priv->ssl_ssl = NULL; > > + } > > if (priv->ssl_private_key) { > > GF_FREE(priv->ssl_private_key); > > } > > > > > > *From:* Zhou, Cynthia (NSB - CN/Hangzhou) > *Sent:* Thursday, April 18, 2019 5:31 PM > *To:* 'Atin Mukherjee' <amukh...@redhat.com> > *Cc:* 'Raghavendra Gowdappa' <rgowd...@redhat.com>; ' > gluster-devel@gluster.org' <gluster-devel@gluster.org> > *Subject:* RE: [Gluster-devel] glusterfsd memory leak issue found after > enable ssl > > > > We scan it use memory-leak tool, there are following prints. We doubt some > open ssl lib malloc is is not properly freed by glusterfs code. > > er+0x2af [*libglusterfs.so.0.0.1]\n\t\tstart_thread+0xda* [*libpthread-2.27.so > <http://libpthread-2.27.so>*]' > *13580* bytes in 175 allocations from stack > b'CRYPTO_malloc+0x58 [*libcrypto.so.1.0.2p*]' > *232904* bytes in 14 allocations from stack > b'CRYPTO_malloc+0x58 [*libcrypto.so.1.0.2p]\n\t\t[unknown* > ]' > [15:41:56] Top 10 stacks with outstanding allocations: > *8792* bytes in 14 allocations from stack > b'CRYPTO_malloc+0x58 [*libcrypto.so.1.0.2p]\n\t\t[unknown* > ]' > *9408* bytes in 42 allocations from stack > b'CRYPTO_realloc+0x4d [*libcrypto.so.1.0.2p*]' > *9723* bytes in 14 allocations from stack > b'CRYPTO_malloc+0x58 [*libcrypto.so.1.0.2p]\n\t\t[unknown* > ]' > *10696* bytes in 21 allocations from stack > b'CRYPTO_malloc+0x58 [*libcrypto.so.1.0.2p]\n\t\t[unknown* > ]' > *11319* bytes in 602 allocations from stack > b'CRYPTO_malloc+0x58 [*libcrypto.so.1.0.2p]\n\t\t[unknown* > ]' > *11431* bytes in 518 allocations from stack > b'CRYPTO_malloc+0x58 [*libcrypto.so.1.0.2p]\n\t\t[unknown* > ]' > *11704* bytes in 371 allocations from stack > b'CRYPTO_malloc+0x58 [*libcrypto.so.1.0.2p]\n\t\t[unknown* > ]' > > > > > > cynthia > > *From:* Zhou, Cynthia (NSB - CN/Hangzhou) > *Sent:* Thursday, April 18, 2019 5:25 PM > *To:* 'Atin Mukherjee' <amukh...@redhat.com> > *Cc:* Raghavendra Gowdappa <rgowd...@redhat.com>; > gluster-devel@gluster.org > *Subject:* RE: [Gluster-devel] glusterfsd memory leak issue found after > enable ssl > > > > I’ve test on glusterfs3.12.15 and glusterfs5.5 all have this issue, after > enable tls ssl socket, when execute gluster v heal <vol-name> info, will > trigger glfshel to connect glusterfsd process, and cause glusterfsd process > memory leak. Could you please try in your env? > > > > cynthia > > > > > > *From:* Atin Mukherjee <amukh...@redhat.com> > *Sent:* Thursday, April 18, 2019 1:19 PM > *To:* Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.z...@nokia-sbell.com> > *Cc:* Raghavendra Gowdappa <rgowd...@redhat.com>; > gluster-devel@gluster.org > *Subject:* Re: [Gluster-devel] glusterfsd memory leak issue found after > enable ssl > > > > > > > > On Wed, 17 Apr 2019 at 10:53, Zhou, Cynthia (NSB - CN/Hangzhou) < > cynthia.z...@nokia-sbell.com> wrote: > > Hi, > > In my recent test, I found that there are very severe glusterfsd memory > leak when enable socket ssl option > > > > What gluster version are you testing? Would you be able to continue your > investigation and share the root cause? > > > > -- > > - Atin (atinm) > > _______________________________________________ > Gluster-devel mailing list > Gluster-devel@gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-devel > > > > -- > > Milind > -- Milind
_______________________________________________ Gluster-devel mailing list Gluster-devel@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-devel
