Matt, Richard, I have not created any patch. Instead I am using s_server app provided from Openssl. Below is the server side command run
openssl s_server -accept <port> -key <custom-key> -cert <custom-certificate signed by custom CA> -CAfile <custom self signed CA> -verify <custom self signed CA> On the client side I am running script below #/bin/sh while [ 1 ] do openssl s_client -connect <server-ip>:<server-port> -cert <custom client certificate not trusted by server> -key <custom client certificate key> &>/dev/null done Running the client once and printing the results showed the cipher selected was ECDHE cipher. I believe there is inherent leak in the ECDHE side of openssl. I find that the server started leaking was leaking. Check the Resident memory and the CPU. Resident memory increases after every 3 second. When I had stopped the client traffic the resident memory did not increase however it did not returned back to what it was earlier. Hope this will help to reproduce the issue. top - 10:16:37 up 46 days, 1:29, 2 users, load average: 0.03, 0.06, 0.08 Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie Cpu(s): 0.6%us, 0.1%sy, 0.0%ni, 99.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 3924288k total, 3757716k used, 166572k free, 78200k buffers Swap: 8388604k total, 999556k used, 7389048k free, 219736k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 27303 root 20 0 39908 3528 2740 S 0.0 0.1 0:00.00 openssl top - 10:16:40 up 46 days, 1:29, 2 users, load average: 0.03, 0.06, 0.08 Tasks: 1 total, 1 running, 0 sleeping, 0 stopped, 0 zombie Cpu(s): 1.6%us, 0.9%sy, 0.0%ni, 97.4%id, 0.0%wa, 0.0%hi, 0.1%si, 0.0%st Mem: 3924288k total, 3757856k used, 166432k free, 78208k buffers Swap: 8388604k total, 999556k used, 7389048k free, 219744k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 27303 root 20 0 39908 3532 2740 R 4.3 0.1 0:00.13 openssl top - 10:16:43 up 46 days, 1:29, 2 users, load average: 0.03, 0.06, 0.08 Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie Cpu(s): 2.6%us, 0.8%sy, 0.1%ni, 96.2%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%st Mem: 3924288k total, 3757980k used, 166308k free, 78232k buffers Swap: 8388604k total, 999556k used, 7389048k free, 219744k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 27303 root 20 0 39908 3544 2740 S 7.0 0.1 0:00.34 openssl top - 10:16:46 up 46 days, 1:30, 2 users, load average: 0.03, 0.06, 0.08 Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie Cpu(s): 2.1%us, 1.1%sy, 0.0%ni, 96.5%id, 0.0%wa, 0.0%hi, 0.2%si, 0.0%st Mem: 3924288k total, 3757964k used, 166324k free, 78248k buffers Swap: 8388604k total, 999556k used, 7389048k free, 219732k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 27303 root 20 0 39908 3560 2740 S 6.7 0.1 0:00.54 openssl top - 10:16:49 up 46 days, 1:30, 2 users, load average: 0.02, 0.06, 0.08 Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie Cpu(s): 2.7%us, 1.1%sy, 0.0%ni, 96.0%id, 0.0%wa, 0.0%hi, 0.2%si, 0.0%st Mem: 3924288k total, 3758212k used, 166076k free, 78264k buffers Swap: 8388604k total, 999556k used, 7389048k free, 219744k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 27303 root 20 0 39908 3572 2740 S 8.0 0.1 0:00.78 openssl top - 10:16:52 up 46 days, 1:30, 2 users, load average: 0.02, 0.06, 0.08 Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie Cpu(s): 3.2%us, 1.2%sy, 0.0%ni, 94.8%id, 0.1%wa, 0.0%hi, 0.6%si, 0.0%st Mem: 3924288k total, 3758212k used, 166076k free, 78288k buffers Swap: 8388604k total, 999556k used, 7389048k free, 219732k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 27303 root 20 0 39908 3592 2740 S 11.0 0.1 0:01.11 openssl top - 10:16:55 up 46 days, 1:30, 2 users, load average: 0.02, 0.06, 0.08 Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie Cpu(s): 3.3%us, 1.3%sy, 0.1%ni, 94.7%id, 0.0%wa, 0.0%hi, 0.6%si, 0.0%st Mem: 3924288k total, 3758584k used, 165704k free, 78288k buffers Swap: 8388604k total, 999556k used, 7389048k free, 219744k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 27303 root 20 0 39908 3608 2740 S 11.3 0.1 0:01.45 openssl top - 10:16:58 up 46 days, 1:30, 2 users, load average: 0.02, 0.06, 0.08 Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie Cpu(s): 3.3%us, 1.6%sy, 0.0%ni, 94.7%id, 0.0%wa, 0.0%hi, 0.4%si, 0.0%st Mem: 3924288k total, 3758864k used, 165424k free, 78296k buffers Swap: 8388604k total, 999556k used, 7389048k free, 219744k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 27303 root 20 0 39908 3628 2740 S 11.3 0.1 0:01.79 openssl top - 10:17:01 up 46 days, 1:30, 2 users, load average: 0.02, 0.06, 0.08 Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie Cpu(s): 3.4%us, 1.5%sy, 0.0%ni, 94.7%id, 0.0%wa, 0.0%hi, 0.4%si, 0.0%st Mem: 3924288k total, 3758956k used, 165332k free, 78312k buffers Swap: 8388604k total, 999556k used, 7389048k free, 219740k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 27303 root 20 0 40016 3648 2740 S 10.7 0.1 0:02.11 openssl top - 10:17:04 up 46 days, 1:30, 2 users, load average: 0.02, 0.06, 0.07 Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie Cpu(s): 3.9%us, 1.9%sy, 0.0%ni, 91.9%id, 1.6%wa, 0.0%hi, 0.7%si, 0.0%st Mem: 3924288k total, 3759708k used, 164580k free, 78328k buffers Swap: 8388604k total, 999556k used, 7389048k free, 219916k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 27303 root 20 0 40016 3668 2740 S 11.0 0.1 0:02.44 openssl top - 10:17:07 up 46 days, 1:30, 2 users, load average: 0.02, 0.06, 0.07 Tasks: 1 total, 1 running, 0 sleeping, 0 stopped, 0 zombie Cpu(s): 3.4%us, 1.2%sy, 0.1%ni, 93.8%id, 0.0%wa, 0.0%hi, 1.4%si, 0.0%st Mem: 3924288k total, 3759336k used, 164952k free, 78360k buffers Swap: 8388604k total, 999556k used, 7389048k free, 219900k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 27303 root 20 0 40016 3688 2740 R 10.7 0.1 0:02.76 openssl top - 10:17:10 up 46 days, 1:30, 2 users, load average: 0.02, 0.06, 0.07 Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie Cpu(s): 3.2%us, 1.6%sy, 0.0%ni, 94.2%id, 0.0%wa, 0.0%hi, 1.0%si, 0.0%st Mem: 3924288k total, 3758584k used, 165704k free, 78360k buffers Swap: 8388604k total, 999556k used, 7389048k free, 219916k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 27303 root 20 0 40016 3708 2740 S 11.3 0.1 0:03.10 openssl top - 10:17:13 up 46 days, 1:30, 2 users, load average: 0.02, 0.06, 0.07 Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie Cpu(s): 3.6%us, 1.1%sy, 0.0%ni, 94.5%id, 0.0%wa, 0.0%hi, 0.8%si, 0.0%st Mem: 3924288k total, 3758476k used, 165812k free, 78376k buffers Swap: 8388604k total, 999556k used, 7389048k free, 219920k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 27303 root 20 0 40016 3728 2740 S 10.7 0.1 0:03.42 openssl top - 10:17:16 up 46 days, 1:30, 2 users, load average: 0.02, 0.06, 0.07 Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie Cpu(s): 3.8%us, 1.3%sy, 0.0%ni, 93.3%id, 0.0%wa, 0.0%hi, 1.7%si, 0.0%st Mem: 3924288k total, 3759824k used, 164464k free, 78384k buffers Swap: 8388604k total, 999556k used, 7389048k free, 219920k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 27303 root 20 0 40124 3748 2740 S 11.3 0.1 0:03.76 openssl --------------- top - 10:23:10 up 46 days, 1:36, 2 users, load average: 0.03, 0.05, 0.06 Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie Cpu(s): 3.6%us, 1.3%sy, 0.1%ni, 94.4%id, 0.0%wa, 0.0%hi, 0.6%si, 0.0%st Mem: 3924288k total, 3756048k used, 168240k free, 75888k buffers Swap: 8388604k total, 998972k used, 7389632k free, 216576k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 27303 root 20 0 42392 6048 2740 S 10.7 0.2 0:41.95 openssl top - 10:23:13 up 46 days, 1:36, 2 users, load average: 0.02, 0.04, 0.06 Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie Cpu(s): 3.2%us, 1.3%sy, 0.0%ni, 94.8%id, 0.0%wa, 0.0%hi, 0.6%si, 0.0%st Mem: 3924288k total, 3756064k used, 168224k free, 75904k buffers Swap: 8388604k total, 998972k used, 7389632k free, 216572k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 27303 root 20 0 42392 6068 2740 S 10.7 0.2 0:42.27 openssl top - 10:23:16 up 46 days, 1:36, 2 users, load average: 0.02, 0.04, 0.06 Tasks: 1 total, 1 running, 0 sleeping, 0 stopped, 0 zombie Cpu(s): 3.2%us, 1.3%sy, 0.1%ni, 94.9%id, 0.0%wa, 0.0%hi, 0.4%si, 0.0%st Mem: 3924288k total, 3756188k used, 168100k free, 75912k buffers Swap: 8388604k total, 998972k used, 7389632k free, 216572k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 27303 root 20 0 42392 6088 2740 R 11.0 0.2 0:42.60 openssl top - 10:23:19 up 46 days, 1:36, 2 users, load average: 0.02, 0.04, 0.06 Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie Cpu(s): 3.1%us, 1.2%sy, 1.6%ni, 93.5%id, 0.0%wa, 0.0%hi, 0.6%si, 0.0%st Mem: 3924288k total, 3756528k used, 167760k free, 75920k buffers Swap: 8388604k total, 998972k used, 7389632k free, 216584k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 27303 root 20 0 42392 6108 2740 S 10.7 0.2 0:42.92 openssl top - 10:23:22 up 46 days, 1:36, 2 users, load average: 0.02, 0.04, 0.06 Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie Cpu(s): 11.1%us, 4.6%sy, 20.2%ni, 63.6%id, 0.0%wa, 0.0%hi, 0.6%si, 0.0%st Mem: 3924288k total, 3756924k used, 167364k free, 75952k buffers Swap: 8388604k total, 998972k used, 7389632k free, 216600k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 27303 root 20 0 42500 6124 2740 S 10.3 0.2 0:43.23 openssl Thanks Darshan -----Original Message----- From: openssl-dev [mailto:openssl-dev-boun...@openssl.org] On Behalf Of Mody, Darshan (Darshan) Sent: Thursday, March 23, 2017 7:40 PM To: openssl-dev@openssl.org Cc: Bahr, William G (Bill) Subject: Re: [openssl-dev] Memory leak in application when we use ECDH Thanks Richard and Matt, I will patch it and send the patch. It will take me couple of days. Regards Darshan -----Original Message----- From: openssl-dev [mailto:openssl-dev-boun...@openssl.org] On Behalf Of Richard Levitte Sent: Thursday, March 23, 2017 7:31 PM To: openssl-dev@openssl.org Subject: Re: [openssl-dev] Memory leak in application when we use ECDH I think that Matt is asking for example code that exhibits this leak. You could patch apps/s_server.c with your callback, or ssl/ssltest.c, and give us that patch. The reason is that we can't know what assumptions you're going with in your callback or application, so if we code an example together, it will be with Our conditions, not yours, and therefore a pretty bad method to figure this out. Cheers, Richard In message <25d2ec755404b4409f263ac6d050febb2a107...@az-ffexmb03.global.avaya.com> on Thu, 23 Mar 2017 13:47:10 +0000, "Mody, Darshan (Darshan)" <darshanm...@avaya.com> said: darshanmody> Matt, darshanmody> darshanmody> Below is the scenario. darshanmody> darshanmody> 1. Have server open a listen socket which always validates the client certificate and chain. darshanmody> 2. On server support ECDHE using callback. Ensure the EC_KEY passed to openssl from app is cleaned up by the app. darshanmody> 3. Connect client with certificates that server does not trust. darshanmody> 4. The connections from client to server fails darshanmody> darshanmody> In course of time the app running the server has been leaking. Even after accounting for the EC_KEY passed by the server app to openssl we find there seems to be leak. Further investigation on the core dumps generated from the server app shows that it has the certificates from the client saved. darshanmody> darshanmody> Hope this helps darshanmody> darshanmody> Thanks darshanmody> Darshan darshanmody> darshanmody> -----Original Message----- darshanmody> From: openssl-dev [mailto:openssl-dev-boun...@openssl.org] darshanmody> On Behalf Of Matt Caswell darshanmody> Sent: Thursday, March 23, 2017 6:55 PM darshanmody> To: openssl-dev@openssl.org darshanmody> Subject: Re: [openssl-dev] Memory leak in application when darshanmody> we use ECDH darshanmody> darshanmody> darshanmody> darshanmody> On 23/03/17 13:19, Mody, Darshan (Darshan) wrote: darshanmody> > Can you further elaborate? darshanmody> > darshanmody> > What we did is to create a TLS connection and with darshanmody> > invalid certificates from the client and server on darshanmody> > verification would reject the certificate. The cipher darshanmody> > negotiated was ECDHE cipher between client and server. darshanmody> > darshanmody> > This was done with load (multiple while 1 script trying darshanmody> > to connect to server using invalid certificates and in darshanmody> > course of time the memory was increasing). darshanmody> darshanmody> Without being able to recreate the problem its going to be very difficult/impossible for us to fix it (assuming the problem is in OpenSSl itself). We would need some simple reproducer code that demonstrates the problem occurring. darshanmody> darshanmody> Matt darshanmody> darshanmody> darshanmody> > darshanmody> > Thanks Darshan darshanmody> > darshanmody> > -----Original Message----- From: openssl-dev darshanmody> > [mailto:openssl-dev-boun...@openssl.org] On Behalf Of darshanmody> > Matt Caswell darshanmody> > Sent: Thursday, March 23, 2017 4:09 PM To: darshanmody> > openssl-dev@openssl.org darshanmody> > Subject: Re: [openssl-dev] Memory leak in application darshanmody> > when we use ECDH darshanmody> > darshanmody> > darshanmody> > darshanmody> > On 23/03/17 10:13, Mody, Darshan (Darshan) wrote: darshanmody> >> Matt, darshanmody> >> darshanmody> >> Even after accounting for the EC_KEY we still observe some leak. darshanmody> >> The leak started after we started using supporting EC darshanmody> >> with callback SSL_set_tmp_ecdh_callback(). darshanmody> >> darshanmody> >> The core dump shows the string data of the far-end certificates. darshanmody> >> I cannot pin point the code in openssl with this regard. darshanmody> > darshanmody> > Are you able to create a simple reproducer demonstrating darshanmody> > the problem with the callback? darshanmody> > darshanmody> > Matt darshanmody> > darshanmody> -- darshanmody> openssl-dev mailing list darshanmody> To unsubscribe: darshanmody> https://urldefense.proofpoint.com/v2/url?u=https-3A__mta.op darshanmody> enssl.org_mailman_listinfo_openssl-2Ddev&d=DwICAg&c=BFpWQw8 darshanmody> bsuKpl1SgiZH64Q&r=bsEULbVnjelD7InzgsegHBEbtXzaIDagy9EuEhJrK darshanmody> fQ&m=VbrRgO8PZIVkFM4PjeK7TEgKDHnbLu_QfbyqRhmvx8I&s=u0cR7sQf darshanmody> _Zz8FoCnrzgLc3drBSR8Ou1qDUyxV8z1xYQ&e= darshanmody> -- darshanmody> openssl-dev mailing list darshanmody> To unsubscribe: darshanmody> https://urldefense.proofpoint.com/v2/url?u=https-3A__mta.op darshanmody> enssl.org_mailman_listinfo_openssl-2Ddev&d=DwICAg&c=BFpWQw8 darshanmody> bsuKpl1SgiZH64Q&r=bsEULbVnjelD7InzgsegHBEbtXzaIDagy9EuEhJrK darshanmody> fQ&m=OtZlUFiavvOVqXL900IST85y3pZLikUdEgekBIIyZoI&s=3T5xlm8q darshanmody> 92-eP1ItbDzGOU972l4wFrkJUgLrBNR4Qx8&e= darshanmody> -- openssl-dev mailing list To unsubscribe: https://urldefense.proofpoint.com/v2/url?u=https-3A__mta.openssl.org_mailman_listinfo_openssl-2Ddev&d=DwICAg&c=BFpWQw8bsuKpl1SgiZH64Q&r=bsEULbVnjelD7InzgsegHBEbtXzaIDagy9EuEhJrKfQ&m=OtZlUFiavvOVqXL900IST85y3pZLikUdEgekBIIyZoI&s=3T5xlm8q92-eP1ItbDzGOU972l4wFrkJUgLrBNR4Qx8&e= -- openssl-dev mailing list To unsubscribe: https://urldefense.proofpoint.com/v2/url?u=https-3A__mta.openssl.org_mailman_listinfo_openssl-2Ddev&d=DwICAg&c=BFpWQw8bsuKpl1SgiZH64Q&r=bsEULbVnjelD7InzgsegHBEbtXzaIDagy9EuEhJrKfQ&m=aM8Nc6bSJzLs-HdERDikYMeQK6uUqvjYvWnfqtvREM4&s=5DECvTKmFHf9urfCwMDaDnk14k9fM_XDkpl9X5r5aJA&e= -- openssl-dev mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev