Dear Gert, after having been able to spend a smooth operating time of the around 3400
permanent connected O-VPN clients (after having attached memory in form of a 10Gb swap partition), we just recognized a hickout, where all clients got disconnected. Unfortunately, after around 3270 disconnected clients, memory was'nt freed at all: 3270 /etc/openvpn/cgwvpn/persipoff top - 10:31:58 up 31 days, 20:10, 2 users, load average: 0.78, 0.70, 0.69 Tasks: 71 total, 2 running, 69 sleeping, 0 stopped, 0 zombie Cpu(s): 0.3%us, 23.2%sy, 0.0%ni, 50.0%id, 0.0%wa, 14.4%hi, 0.0%si, 12.0%st Mem: 7629920k total, 5073576k used, 2556344k free, 159504k buffers Swap: 10485756k total, 0k used, 10485756k free, 192536k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 22810 root 20 0 4481m 4.2g 1996 S 62.5 58.1 5102:38 openvpn 12619 root 20 0 8 4 0 R 3.0 0.0 0:00.09 openvpn 12222 root 20 0 15228 1216 932 R 0.3 0.0 0:00.02 top Unfortunately, this is not something, we had expected. We have not applied the patch yet, which brings back the "2.1" behavior. freundlich grĂ¼sst, kind regards, _________________________ _________________________ _________________________ _________________________ _________________________ ________________ Arno Odermatt Phone +41414454053 arno.oderm...@ch.schindler.com Schindler Elevator Ltd. | Corporate Research & Development | CRD/CO-BAT Zugerstr. 13 | CH-6030 Ebikon, Switzerland http://www.schindler.com _________________________ _________________________ _________________________ _________________________ _________________________ ________________ Please consider your environment. Schindler supports sustainable urban development with safe, reliable and ecologically sound mobility solutions. From: Gert Doering <g...@greenie.muc.de> To: Brad Zhang <hebei5...@gmail.com>, Cc: openvpn-devel@lists.sourceforge.net List-Post: openvpn-devel@lists.sourceforge.net Date: 23.10.2013 16:20 Subject: Re: [Openvpn-devel] Does openvpn 2.3.2 has memory leak? Hi, On Sun, Oct 06, 2013 at 09:21:46PM +0800, Brad Zhang wrote: > Do you have some idea about this issue? Looking forward to your reply. Coming back to *this* thread, after some debugging in the other thread with Arno, I think what I can say so far is that we're observing two different things: - the first renegotiation brings a big jump in memory, but this is to be expected, because the new keying material needs extra memory, and the memory for the old key is not released right away (it seems to be released at the next renegotiation, or at client disconnect - I've seen VSZ/RSS go down(!) when clients disconnect, so that seems to work right). This is the "big" jump in memory consumption which is actually killing Arno's setup, because there is not enough virtual memory available to hold all that is needed for 3000 clients (but that can be fixed by increasing max-memory limits if there is enough RAM). - each further renegotiation leaks "a few kbyte" of memory per client, which should also be fully returned when the client disconnects - this is the issue I discovered earlier, with the GC handling of the per-client environment set, and I'll send a patch for that "soon" (as soon as I have fully understood the code involved). Now, what I do not really understand is why the memory consumption on the 3rd, 4th, 5th renegotiation seems to slow down - there seems to be some additional memory fragmentation involved in the 2nd+3rd renegotiation, which leads to memory re-use later on. Or so. Testing this with "reneg-sec 60" is actually quite interesting... here's *disconnect* of a client: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 19892 0.0 0.0 7052 4304 pts/7 SN+ 17:48 0:01 openvpn ../serv USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 19892 0.0 0.0 6300 3740 pts/7 SN+ 17:48 0:01 openvpn ../serv ... so you can see that memory is returned to the OS. (I think this will not be visible if many clients are connected as the free()ed memory will be "somewhere in the middle" - but disconnecting *all* clients should also show memory being returned) Now going to stare more at the code... :-) gert -- USENET is *not* the non-clickable part of WWW! //www.muc.de/~gert/ Gert Doering - Munich, Germany g...@greenie.muc.de fax: +49-89-35655025 g...@net.informatik.tu-muenchen.de [attachment "attxgxs6.dat" deleted by Arno Odermatt/R&D/SCH/SCHINDLER] ------------------------------------------------------------------------------ October Webinars: Code for Performance Free Intel webinars can help you accelerate application performance. Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from the latest Intel processors and coprocessors. See abstracts and register > http://pubads.g.doubleclick.net/gampad/clk?id=60135991&iu=/4140/ostg.clktrk _________________________ ______________________ Openvpn-devel mailing list Openvpn-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/openvpn-devel ****************************************************** Notice: The information contained in this message is intended only for use of the individual(s) named above and may contain confidential, proprietary or legally privileged information. No confidentiality or privilege is waived or lost by any mistransmission. If you are not the intended recipient of this message you are hereby notified that you must not use, disseminate , copy it in any form or take any action in reliance of it. If you have received this message in error please delete it and any copies of it and notify the sender immediately. *******************************************************