Re: [Openvpn-devel] [Openvpn-users] OpenVPN memory usage stats
Hi, came across an old posting today, by chance - cc'ing openvpn-devel, because that's really more of a -devel question... On Thu, May 18, 2017 at 02:47:37PM -0700, Yury Shefer wrote: > Here is what I noticed: > 1) when I hit max-sessions and the daemon no longer accepts new > connections (by logging "new incoming connection would exceed maximum > number of clients (16200)"), the memory usage by the process > continuously increases (by ~30 MB per minute). For 16K connections > OpenVPN will consume around 3-3.2Gb of RAM and then, in a period of > ~2-3 hours, it will consume the rest of VM RAM (8 GB total) and gets > killed by the kernel. This looks like we do not properly clean up memory when refusing a new session (every new session needs memory, which is expected, but if we do not actually accept the session, memory needs to be returned). I guess that you see quite a number of incoming connections on top of the allowed limit, and thus the "continuous increase". [..] > 2) run the process with valgrind (valgrind --leak-check=yes) but it > reported no errors. Logging with verb 9 with such amount of sessions > is a bit unrealistic. Valgrind might not notice because our memory management is... quite a bit advanced... Can you reproduce this with a much lower limit, like "max-sessions 5"? gert -- "If was one thing all people took for granted, was conviction that if you feed honest figures into a computer, honest figures come out. Never doubted it myself till I met a computer with a sense of humor." Robert A. Heinlein, The Moon is a Harsh Mistress Gert Doering - Munich, Germany g...@greenie.muc.de signature.asc Description: PGP signature -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot___ Openvpn-devel mailing list Openvpn-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/openvpn-devel
Re: [Openvpn-devel] [Openvpn-users] OpenVPN memory usage
Hi David, David Sommerseth wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 20/04/11 17:25, Jan Just Keijser wrote: Hi *, copying in the openvpn-devel list as they might be interested in this memory usage analysis as well Ralf Hildebrandt wrote: * Ralf Hildebrandt: * Fredrik Kers : I measure the memory usage by checking the VmRSS (Resident Set Size) in /proc/[pid]/status before and after the clients connect (about 1000 of them). So it's the same method (because that's what top and htop do). I'm using a 64 bit system, maybe that will cause OpenVPN to consume more memory per client then your setup Yes, that could be. It's 32Bit here. pmap -d pid_of_openvpn is showing: # pmap -d 1607 1607: /usr/sbin/openvpn --writepid /var/run/openvpn.server.pid --syslog ovpn-server --cd /etc/openvpn --config /etc/openvpn/server.conf Address Kbytes Mode Offset DeviceMapping 08048000 500 r-x-- 068:5 openvpn 080c5000 4 rwx-- 0007d000 068:5 openvpn 080c6000 24 rwx-- 000:0 [ anon ] * 08b51000 41396 rwx-- 000:0 [ anon ] b727 132 rwx-- 000:0 [ anon ] ... bfef8000 132 rw--- 000:0 [ stack ] mapped: 46240Kwriteable/private: 42304Kshared: 0K The line starting with * shows the biggest chunk of memory in use. I'm running openvpn 2.1.4 on a 64bit system (CentSO 5.5 with openssl 0.9.8e). when the first client connects the openvpn server gets "hit" by ~ 600 Kb (as seen using pmap). The size of the 'hit' most likely depends on the encryption cipher used. I recompiled openvpn using ./configure --with-mem-check=valgrind and then ran the server again using valgrind --tool=massif .../openvpn this results in a nice memory report which you can view using ms_print massif.out.22218 turns out that most memory is allocated by the openssl libraries : CRYPTO_malloc the memory trace is too large to show here, but there's tons of stuff in that 'ms_print' output. Note that compression was NOT used in this example, apparently the openssl libs call deflateInit internally? Timestep 49 is when the client connects: you see the memory usage jump from 327,520 bytes to 934,688. ntime(i) total(B) useful-heap(B) extra-heap(B)stacks(B) 48 36,072,833 327,520 276,10251,4180 49 36,341,762 934,688 883,12651,5620 94.48% (883,126B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc. Again, thanks a lot for wonderful reports. It is easy to shoot OpenSSL here. But I'm not going to do that just yet. There are some work in progress of modularising the SSL layer in OpenVPN (if patches are cleaned up in time and get a decent review, we might have this in OpenVPN 2.3 already). This work will also add support for PolarSSL. When we see those patches, it might be easier to see if there's something which can be improved in OpenVPN or if it is something which is completely into the hands of OpenSSL. So running this test with the PolarSSL module instead will be very interesting. I generally think that the crypto context needs to stay in memory for some time, especially in UDP mode, to avoid re-negotiating of the connections in case of connectivity issues between client and server. But when the session is defined as closed by the OpenVPN server, this memory pool should be released. For TCP mode, the session is closed (afair) when the client explicitly closes the connection. In UDP mode it's a timeout, unless - --explicit-exit is used. This freeing of context memory needs to be verified in the code. for the record: I did not mean to "shoot OpenSSL" here; I use OpenSSL a lot for my regular work and I know how difficult it is to get it to behave correctly when it comes to memory usage. The reason I started digging was that I added a very dumb 'printf' statement every time gc_alloc or ALLOC_OBJ was called (in openvpn buffer.c) ; I then grabbed the output after a client connected and added up all the mallocs : this did not even add up to 75 KB (and I'm probably double counting too). So then I went on a hunt to find out what _IS_ causing the 600 KB hit, which led to the ms_print report. I'm not too worried about a 600 KB per client hit for an OpenVPN server - it means a 1,000 clients will eat up less than 1 GB of RAM . Unless you're running on tiny hardware (dd-wrt etc) this should not be an issue, but I would not recommend to connect a 1,000 clients to a single dd-wrt box anyway. It will be interesting to see if PolarSSL is
Re: [Openvpn-devel] [Openvpn-users] OpenVPN memory usage
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 20/04/11 17:25, Jan Just Keijser wrote: > Hi *, > > copying in the openvpn-devel list as they might be interested in this memory > usage analysis as well > > Ralf Hildebrandt wrote: >> * Ralf Hildebrandt: >>> * Fredrik Kers : I measure the memory usage by checking the VmRSS (Resident Set Size) in /proc/[pid]/status before and after the clients connect (about 1000 of them). >>> So it's the same method (because that's what top and htop do). >>> I'm using a 64 bit system, maybe that will cause OpenVPN to consume more memory per client then your setup >>> Yes, that could be. It's 32Bit here. >> >> pmap -d pid_of_openvpn >> >> is showing: >> >> # pmap -d 1607 >> 1607: /usr/sbin/openvpn --writepid /var/run/openvpn.server.pid --syslog >> ovpn-server --cd /etc/openvpn --config /etc/openvpn/server.conf >> Address Kbytes Mode Offset DeviceMapping >> 08048000 500 r-x-- 068:5 openvpn >> 080c5000 4 rwx-- 0007d000 068:5 openvpn >> 080c6000 24 rwx-- 000:0 [ anon ] >> * 08b51000 41396 rwx-- 000:0 [ anon ] >> b727 132 rwx-- 000:0 [ anon ] >> ... >> bfef8000 132 rw--- 000:0 [ stack ] >> mapped: 46240Kwriteable/private: 42304Kshared: 0K >> >> The line starting with * shows the biggest chunk of memory in use. >> > > I'm running openvpn 2.1.4 on a 64bit system (CentSO 5.5 with openssl 0.9.8e). > when the first client connects the openvpn server > gets "hit" by ~ 600 Kb (as seen using pmap). The size of the 'hit' most > likely depends on the encryption cipher used. > > I recompiled openvpn using > ./configure --with-mem-check=valgrind > and then ran the server again using > valgrind --tool=massif .../openvpn > > this results in a nice memory report which you can view using > ms_print massif.out.22218 > > turns out that most memory is allocated by the openssl libraries : > CRYPTO_malloc > the memory trace is too large to show here, but there's tons of stuff in that > 'ms_print' output. > Note that compression was NOT used in this example, apparently the openssl > libs call deflateInit internally? > > Timestep 49 is when the client connects: you see the memory usage jump from > 327,520 bytes to 934,688. > > > ntime(i) total(B) useful-heap(B) extra-heap(B) > stacks(B) > > 48 36,072,833 327,520 276,10251,418> 0 > 49 36,341,762 934,688 883,12651,562> 0 > 94.48% (883,126B) (heap allocation functions) malloc/new/new[], --alloc-fns, > etc. Again, thanks a lot for wonderful reports. It is easy to shoot OpenSSL here. But I'm not going to do that just yet. There are some work in progress of modularising the SSL layer in OpenVPN (if patches are cleaned up in time and get a decent review, we might have this in OpenVPN 2.3 already). This work will also add support for PolarSSL. When we see those patches, it might be easier to see if there's something which can be improved in OpenVPN or if it is something which is completely into the hands of OpenSSL. So running this test with the PolarSSL module instead will be very interesting. I generally think that the crypto context needs to stay in memory for some time, especially in UDP mode, to avoid re-negotiating of the connections in case of connectivity issues between client and server. But when the session is defined as closed by the OpenVPN server, this memory pool should be released. For TCP mode, the session is closed (afair) when the client explicitly closes the connection. In UDP mode it's a timeout, unless - --explicit-exit is used. This freeing of context memory needs to be verified in the code. kind regards, David Sommerseth -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/ iEYEARECAAYFAk2vFPIACgkQDC186MBRfrrjYACdFgsI9bBKA/Mp5OZ3uwvuwEH8 knwAn3TMjSl2nsiKVxJRNadLh+RAB8mY =9D/e -END PGP SIGNATURE-
Re: [Openvpn-devel] [Openvpn-users] OpenVPN memory usage
Hi *, copying in the openvpn-devel list as they might be interested in this memory usage analysis as well Ralf Hildebrandt wrote: > * Ralf Hildebrandt: >> * Fredrik Kers : >>> I measure the memory usage by checking the VmRSS (Resident Set Size) in >>> /proc/[pid]/status before and after the clients connect (about 1000 of >>> them). >> So it's the same method (because that's what top and htop do). >> >>> I'm using a 64 bit system, maybe that will cause OpenVPN to consume >>> more memory per client then your setup >> Yes, that could be. It's 32Bit here. > > pmap -d pid_of_openvpn > > is showing: > > # pmap -d 1607 > 1607: /usr/sbin/openvpn --writepid /var/run/openvpn.server.pid --syslog > ovpn-server --cd /etc/openvpn --config /etc/openvpn/server.conf > Address Kbytes Mode Offset DeviceMapping > 08048000 500 r-x-- 068:5 openvpn > 080c5000 4 rwx-- 0007d000 068:5 openvpn > 080c6000 24 rwx-- 000:0 [ anon ] > * 08b51000 41396 rwx-- 000:0 [ anon ] > b727 132 rwx-- 000:0 [ anon ] > ... > bfef8000 132 rw--- 000:0 [ stack ] > mapped: 46240Kwriteable/private: 42304Kshared: 0K > > The line starting with * shows the biggest chunk of memory in use. > I'm running openvpn 2.1.4 on a 64bit system (CentSO 5.5 with openssl 0.9.8e). when the first client connects the openvpn server gets "hit" by ~ 600 Kb (as seen using pmap). The size of the 'hit' most likely depends on the encryption cipher used. I recompiled openvpn using ./configure --with-mem-check=valgrind and then ran the server again using valgrind --tool=massif .../openvpn this results in a nice memory report which you can view using ms_print massif.out.22218 turns out that most memory is allocated by the openssl libraries : CRYPTO_malloc the memory trace is too large to show here, but there's tons of stuff in that 'ms_print' output. Note that compression was NOT used in this example, apparently the openssl libs call deflateInit internally? Timestep 49 is when the client connects: you see the memory usage jump from 327,520 bytes to 934,688. ntime(i) total(B) useful-heap(B) extra-heap(B)stacks(B) 48 36,072,833 327,520 276,10251,4180 49 36,341,762 934,688 883,12651,5620 94.48% (883,126B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc. ->82.41% (770,295B) 0x3858CDAD90: CRYPTO_malloc (in /lib64/libcrypto.so.0.9.8e) | ->62.91% (588,016B) 0x3858CBFF34: ??? (in /lib64/libcrypto.so.0.9.8e) | | ->14.02% (131,072B) 0x38578064B3: deflateInit2_ (in /usr/lib64/libz.so.1.2.3) | | | ->14.02% (131,072B) 0x3857806622: deflateInit_ (in /usr/lib64/libz.so.1.2.3) | | | ->14.02% (131,072B) 0x3858CBFECA: ??? (in /lib64/libcrypto.so.0.9.8e) | | | ->14.02% (131,072B) 0x3858CBFA96: COMP_CTX_new (in /lib64/libcrypto.so.0.9.8e) | | | ->07.01% (65,536B) 0x385E025CDF: tls1_change_cipher_state (in /lib64/libssl.so.0.9.8e) | | | | ->07.01% (65,536B) 0x385E020492: ssl3_do_change_cipher_spec (in /lib64/libssl.so.0.9.8e) | | | | ->07.01% (65,536B) 0x385E021952: ssl3_read_bytes (in /lib64/libssl.so.0.9.8e) | | | | ->07.01% (65,536B) 0x385E0222AE: ssl3_get_message (in /lib64/libssl.so.0.9.8e) | | | | ->07.01% (65,536B) 0x385E022B1B: ssl3_get_finished (in /lib64/libssl.so.0.9.8e) | | | | ->07.01% (65,536B) 0x385E01A740: ssl3_accept (in /lib64/libssl.so.0.9.8e) | | | | ->07.01% (65,536B) 0x385E021883: ssl3_read_bytes (in /lib64/libssl.so.0.9.8e) | | | | ->07.01% (65,536B) 0x385E01E55F: ??? (in /lib64/libssl.so.0.9.8e) | | | | ->07.01% (65,536B) 0x385E038F3F: ??? (in /lib64/libssl.so.0.9.8e) | | | | ->07.01% (65,536B) 0x3858C7678D: BIO_read (in /lib64/libcrypto.so.0.9.8e) | | | | ->07.01% (65,536B) 0x46F7A3: bio_read (ssl.c:2058) | | | | ->07.01% (65,536B) 0x46FA8F: key_state_read_plaintext (ssl.c:2148) | | | | ->07.01% (65,536B) 0x474859: tls_process (ssl.c:4075) | | | | ->07.01% (65,536B) 0x475420: tls_multi_process (ssl.c:4303) | | | | ->07.01% (65,536B) 0x411B4C: check_tls_dowork (forward.c:93) | | | | ->07.01% (65,536B) 0x41571F: check_tls (forward-inline.h:41) | | | | ->07.01% (65,536B) 0x415663: pre_select (forward.c:1304) | | | | ->07.01% (65,536B)