Ok, I will post it later. cynthia
From: Raghavendra Gowdappa <rgowd...@redhat.com> Sent: Monday, April 22, 2019 10:09 AM To: Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.z...@nokia-sbell.com> Cc: Atin Mukherjee <amukh...@redhat.com>; gluster-devel@gluster.org Subject: Re: [Gluster-devel] glusterfsd memory leak issue found after enable ssl On Mon, Apr 22, 2019 at 7:18 AM Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.z...@nokia-sbell.com<mailto:cynthia.z...@nokia-sbell.com>> wrote: Hi, From my code study it seems priv->ssl_ssl is not properly released, I made a patch and the glusterfsd memory leak is alleviated with my patch, This is a legitimate leak. Can you post a patch to gerrit? but some otherwhere is still leaking, I have no clue about the other leak points. --- a/rpc/rpc-transport/socket/src/socket.c +++ b/rpc/rpc-transport/socket/src/socket.c @@ -1019,7 +1019,16 @@ static void __socket_reset(rpc_transport_t *this) { memset(&priv->incoming, 0, sizeof(priv->incoming)); event_unregister_close(this->ctx->event_pool, priv->sock, priv->idx); - + if(priv->use_ssl&& priv->ssl_ssl) + { + gf_log(this->name, GF_LOG_TRACE, + "clear and reset for socket(%d), free ssl ", + priv->sock); + SSL_shutdown(priv->ssl_ssl); + SSL_clear(priv->ssl_ssl); + SSL_free(priv->ssl_ssl); + priv->ssl_ssl = NULL; + } priv->sock = -1; priv->idx = -1; priv->connected = -1; @@ -4238,6 +4250,16 @@ void fini(rpc_transport_t *this) { pthread_mutex_destroy(&priv->out_lock); pthread_mutex_destroy(&priv->cond_lock); pthread_cond_destroy(&priv->cond); + if(priv->use_ssl&& priv->ssl_ssl) + { + gf_log(this->name, GF_LOG_TRACE, + "clear and reset for socket(%d), free ssl ", + priv->sock); + SSL_shutdown(priv->ssl_ssl); + SSL_clear(priv->ssl_ssl); + SSL_free(priv->ssl_ssl); + priv->ssl_ssl = NULL; + } if (priv->ssl_private_key) { GF_FREE(priv->ssl_private_key); } From: Zhou, Cynthia (NSB - CN/Hangzhou) Sent: Thursday, April 18, 2019 5:31 PM To: 'Atin Mukherjee' <amukh...@redhat.com<mailto:amukh...@redhat.com>> Cc: 'Raghavendra Gowdappa' <rgowd...@redhat.com<mailto:rgowd...@redhat.com>>; 'gluster-devel@gluster.org<mailto:gluster-devel@gluster.org>' <gluster-devel@gluster.org<mailto:gluster-devel@gluster.org>> Subject: RE: [Gluster-devel] glusterfsd memory leak issue found after enable ssl We scan it use memory-leak tool, there are following prints. We doubt some open ssl lib malloc is is not properly freed by glusterfs code. er+0x2af [libglusterfs.so.0.0.1]\n\t\tstart_thread+0xda [libpthread-2.27.so<http://libpthread-2.27.so>]' 13580 bytes in 175 allocations from stack b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]' 232904 bytes in 14 allocations from stack b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]' [15:41:56] Top 10 stacks with outstanding allocations: 8792 bytes in 14 allocations from stack b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]' 9408 bytes in 42 allocations from stack b'CRYPTO_realloc+0x4d [libcrypto.so.1.0.2p]' 9723 bytes in 14 allocations from stack b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]' 10696 bytes in 21 allocations from stack b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]' 11319 bytes in 602 allocations from stack b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]' 11431 bytes in 518 allocations from stack b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]' 11704 bytes in 371 allocations from stack b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]' cynthia From: Zhou, Cynthia (NSB - CN/Hangzhou) Sent: Thursday, April 18, 2019 5:25 PM To: 'Atin Mukherjee' <amukh...@redhat.com<mailto:amukh...@redhat.com>> Cc: Raghavendra Gowdappa <rgowd...@redhat.com<mailto:rgowd...@redhat.com>>; gluster-devel@gluster.org<mailto:gluster-devel@gluster.org> Subject: RE: [Gluster-devel] glusterfsd memory leak issue found after enable ssl I’ve test on glusterfs3.12.15 and glusterfs5.5 all have this issue, after enable tls ssl socket, when execute gluster v heal <vol-name> info, will trigger glfshel to connect glusterfsd process, and cause glusterfsd process memory leak. Could you please try in your env? cynthia From: Atin Mukherjee <amukh...@redhat.com<mailto:amukh...@redhat.com>> Sent: Thursday, April 18, 2019 1:19 PM To: Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.z...@nokia-sbell.com<mailto:cynthia.z...@nokia-sbell.com>> Cc: Raghavendra Gowdappa <rgowd...@redhat.com<mailto:rgowd...@redhat.com>>; gluster-devel@gluster.org<mailto:gluster-devel@gluster.org> Subject: Re: [Gluster-devel] glusterfsd memory leak issue found after enable ssl On Wed, 17 Apr 2019 at 10:53, Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.z...@nokia-sbell.com<mailto:cynthia.z...@nokia-sbell.com>> wrote: Hi, In my recent test, I found that there are very severe glusterfsd memory leak when enable socket ssl option What gluster version are you testing? Would you be able to continue your investigation and share the root cause? -- - Atin (atinm)
_______________________________________________ Gluster-devel mailing list Gluster-devel@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-devel
