Re: [Openvpn-devel] [Openvpn-users] OpenVPN memory usage

2011-04-20 Thread Jan Just Keijser 

Hi David,

David Sommerseth wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 20/04/11 17:25, Jan Just Keijser wrote:
  

Hi *,

copying in the openvpn-devel list as they might be interested in this memory 
usage analysis as well

Ralf Hildebrandt wrote:


* Ralf Hildebrandt :
  

* Fredrik Kers :


I measure the memory usage by checking the VmRSS (Resident Set Size) in
/proc/[pid]/status before and after the clients connect (about 1000 of
them).
  

So it's the same method (because that's what top and htop do).



I'm using a 64 bit system, maybe that will cause OpenVPN to consume
more memory per client then your setup
  

Yes, that could be. It's 32Bit here.

pmap -d pid_of_openvpn 


is showing:

# pmap -d 1607
1607:   /usr/sbin/openvpn --writepid /var/run/openvpn.server.pid --syslog 
ovpn-server --cd /etc/openvpn --config /etc/openvpn/server.conf
Address   Kbytes Mode  Offset   DeviceMapping
08048000 500 r-x--  068:5 openvpn
080c5000   4 rwx-- 0007d000 068:5 openvpn
080c6000  24 rwx--  000:0   [ anon ]
* 08b51000   41396 rwx--  000:0   [ anon ]
b727 132 rwx--  000:0   [ anon ]
...
bfef8000 132 rw---  000:0   [ stack ]
mapped: 46240Kwriteable/private: 42304Kshared: 0K

The line starting with * shows the biggest chunk of memory in use.

  

I'm running openvpn 2.1.4 on a 64bit system (CentSO 5.5 with openssl 0.9.8e). 
when the first client connects the openvpn server
gets "hit" by ~ 600 Kb (as seen using pmap). The size of the 'hit' most likely 
depends on the encryption cipher used.

I recompiled openvpn using
  ./configure --with-mem-check=valgrind
and then ran the server again using
  valgrind --tool=massif .../openvpn

this results in a nice memory report which you can view using
  ms_print massif.out.22218

turns out that most memory is allocated by the openssl libraries : CRYPTO_malloc
the memory trace is too large to show here, but there's tons of stuff in that 
'ms_print' output.
Note that compression was NOT used in this example, apparently the openssl libs 
call deflateInit internally?

Timestep 49 is when the client connects: you see the memory usage jump from 
327,520 bytes to 934,688.


  ntime(i) total(B)   useful-heap(B) extra-heap(B)stacks(B)

 48 36,072,833  327,520  276,10251,4180
 49 36,341,762  934,688  883,12651,5620
94.48% (883,126B) (heap allocation functions) malloc/new/new[], --alloc-fns, 
etc.



Again, thanks a lot for wonderful reports.  It is easy to shoot OpenSSL
here.  But I'm not going to do that just yet.  There are some work in
progress of modularising the SSL layer in OpenVPN (if patches are cleaned
up in time and get a decent review, we might have this in OpenVPN 2.3
already).  This work will also add support for PolarSSL.

When we see those patches, it might be easier to see if there's something
which can be improved in OpenVPN or if it is something which is completely
into the hands of OpenSSL.  So running this test with the PolarSSL module
instead will be very interesting.

I generally think that the crypto context needs to stay in memory for some
time, especially in UDP mode, to avoid re-negotiating of the connections in
case of connectivity issues between client and server.  But when the
session is defined as closed by the OpenVPN server, this memory pool should
be released.  For TCP mode, the session is closed (afair) when the client
explicitly closes the connection.  In UDP mode it's a timeout, unless
- --explicit-exit is used.  This freeing of context memory needs to be
verified in the code.
  
for the record: I did not mean to "shoot OpenSSL"  here; I use OpenSSL a 
lot for my regular work and I know how difficult it is to get it to 
behave correctly when it comes to memory usage.
The reason I started digging was that I added a very dumb 'printf' 
statement every time gc_alloc or ALLOC_OBJ was called (in openvpn 
buffer.c) ; I then grabbed the output after a client connected and added 
up all the mallocs : this did not even add up to 75 KB (and I'm probably 
double counting too). So then I went on a hunt to find out what _IS_ 
causing the 600 KB hit, which led to the ms_print report.


I'm not too worried about a 600 KB per client hit for an OpenVPN server 
- it means a 1,000 clients will eat up less than 1 GB of RAM . Unless 
you're running on tiny hardware (dd-wrt etc) this should not be an 
issue, but I would not recommend to connect a 1,000 clients to a single 
dd-wrt box anyway.


It will be interesting to see if PolarSSL is

Re: [Openvpn-devel] [Openvpn-users] OpenVPN memory usage

2011-04-20 Thread David Sommerseth 
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 20/04/11 17:25, Jan Just Keijser wrote:
> Hi *,
> 
> copying in the openvpn-devel list as they might be interested in this memory 
> usage analysis as well
> 
> Ralf Hildebrandt wrote:
>> * Ralf Hildebrandt :
>>> * Fredrik Kers :
 I measure the memory usage by checking the VmRSS (Resident Set Size) in
 /proc/[pid]/status before and after the clients connect (about 1000 of
 them).
>>> So it's the same method (because that's what top and htop do).
>>>
 I'm using a 64 bit system, maybe that will cause OpenVPN to consume
 more memory per client then your setup
>>> Yes, that could be. It's 32Bit here.
>>
>> pmap -d pid_of_openvpn 
>>
>> is showing:
>>
>> # pmap -d 1607
>> 1607:   /usr/sbin/openvpn --writepid /var/run/openvpn.server.pid --syslog 
>> ovpn-server --cd /etc/openvpn --config /etc/openvpn/server.conf
>> Address   Kbytes Mode  Offset   DeviceMapping
>> 08048000 500 r-x--  068:5 openvpn
>> 080c5000   4 rwx-- 0007d000 068:5 openvpn
>> 080c6000  24 rwx--  000:0   [ anon ]
>> * 08b51000   41396 rwx--  000:0   [ anon ]
>> b727 132 rwx--  000:0   [ anon ]
>> ...
>> bfef8000 132 rw---  000:0   [ stack ]
>> mapped: 46240Kwriteable/private: 42304Kshared: 0K
>>
>> The line starting with * shows the biggest chunk of memory in use.
>>
> 
> I'm running openvpn 2.1.4 on a 64bit system (CentSO 5.5 with openssl 0.9.8e). 
> when the first client connects the openvpn server
> gets "hit" by ~ 600 Kb (as seen using pmap). The size of the 'hit' most 
> likely depends on the encryption cipher used.
> 
> I recompiled openvpn using
>   ./configure --with-mem-check=valgrind
> and then ran the server again using
>   valgrind --tool=massif .../openvpn
> 
> this results in a nice memory report which you can view using
>   ms_print massif.out.22218
> 
> turns out that most memory is allocated by the openssl libraries : 
> CRYPTO_malloc
> the memory trace is too large to show here, but there's tons of stuff in that 
> 'ms_print' output.
> Note that compression was NOT used in this example, apparently the openssl 
> libs call deflateInit internally?
> 
> Timestep 49 is when the client connects: you see the memory usage jump from 
> 327,520 bytes to 934,688.
> 
> 
>   ntime(i) total(B)   useful-heap(B) extra-heap(B)
> stacks(B)
> 
>  48 36,072,833  327,520  276,10251,418> 0
>  49 36,341,762  934,688  883,12651,562> 0
> 94.48% (883,126B) (heap allocation functions) malloc/new/new[], --alloc-fns, 
> etc.

Again, thanks a lot for wonderful reports.  It is easy to shoot OpenSSL
here.  But I'm not going to do that just yet.  There are some work in
progress of modularising the SSL layer in OpenVPN (if patches are cleaned
up in time and get a decent review, we might have this in OpenVPN 2.3
already).  This work will also add support for PolarSSL.

When we see those patches, it might be easier to see if there's something
which can be improved in OpenVPN or if it is something which is completely
into the hands of OpenSSL.  So running this test with the PolarSSL module
instead will be very interesting.

I generally think that the crypto context needs to stay in memory for some
time, especially in UDP mode, to avoid re-negotiating of the connections in
case of connectivity issues between client and server.  But when the
session is defined as closed by the OpenVPN server, this memory pool should
be released.  For TCP mode, the session is closed (afair) when the client
explicitly closes the connection.  In UDP mode it's a timeout, unless
- --explicit-exit is used.  This freeing of context memory needs to be
verified in the code.


kind regards,

David Sommerseth
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/

iEYEARECAAYFAk2vFPIACgkQDC186MBRfrrjYACdFgsI9bBKA/Mp5OZ3uwvuwEH8
knwAn3TMjSl2nsiKVxJRNadLh+RAB8mY
=9D/e
-END PGP SIGNATURE-

Re: [Openvpn-devel] [Openvpn-users] OpenVPN memory usage

2011-04-20 Thread Jan Just Keijser 
Hi *,

copying in the openvpn-devel list as they might be interested in this memory 
usage analysis as well

Ralf Hildebrandt wrote:
> * Ralf Hildebrandt :
>> * Fredrik Kers :
>>> I measure the memory usage by checking the VmRSS (Resident Set Size) in
>>> /proc/[pid]/status before and after the clients connect (about 1000 of
>>> them).
>> So it's the same method (because that's what top and htop do).
>>
>>> I'm using a 64 bit system, maybe that will cause OpenVPN to consume
>>> more memory per client then your setup
>> Yes, that could be. It's 32Bit here.
> 
> pmap -d pid_of_openvpn 
> 
> is showing:
> 
> # pmap -d 1607
> 1607:   /usr/sbin/openvpn --writepid /var/run/openvpn.server.pid --syslog 
> ovpn-server --cd /etc/openvpn --config /etc/openvpn/server.conf
> Address   Kbytes Mode  Offset   DeviceMapping
> 08048000 500 r-x--  068:5 openvpn
> 080c5000   4 rwx-- 0007d000 068:5 openvpn
> 080c6000  24 rwx--  000:0   [ anon ]
> * 08b51000   41396 rwx--  000:0   [ anon ]
> b727 132 rwx--  000:0   [ anon ]
> ...
> bfef8000 132 rw---  000:0   [ stack ]
> mapped: 46240Kwriteable/private: 42304Kshared: 0K
> 
> The line starting with * shows the biggest chunk of memory in use.
> 

I'm running openvpn 2.1.4 on a 64bit system (CentSO 5.5 with openssl 0.9.8e). 
when the first client connects the openvpn server
gets "hit" by ~ 600 Kb (as seen using pmap). The size of the 'hit' most likely 
depends on the encryption cipher used.

I recompiled openvpn using
  ./configure --with-mem-check=valgrind
and then ran the server again using
  valgrind --tool=massif .../openvpn

this results in a nice memory report which you can view using
  ms_print massif.out.22218

turns out that most memory is allocated by the openssl libraries : CRYPTO_malloc
the memory trace is too large to show here, but there's tons of stuff in that 
'ms_print' output.
Note that compression was NOT used in this example, apparently the openssl libs 
call deflateInit internally?

Timestep 49 is when the client connects: you see the memory usage jump from 
327,520 bytes to 934,688.


  ntime(i) total(B)   useful-heap(B) extra-heap(B)stacks(B)

 48 36,072,833  327,520  276,10251,4180
 49 36,341,762  934,688  883,12651,5620
94.48% (883,126B) (heap allocation functions) malloc/new/new[], --alloc-fns, 
etc.
->82.41% (770,295B) 0x3858CDAD90: CRYPTO_malloc (in /lib64/libcrypto.so.0.9.8e)
| ->62.91% (588,016B) 0x3858CBFF34: ??? (in /lib64/libcrypto.so.0.9.8e)
| | ->14.02% (131,072B) 0x38578064B3: deflateInit2_ (in 
/usr/lib64/libz.so.1.2.3)
| | | ->14.02% (131,072B) 0x3857806622: deflateInit_ (in 
/usr/lib64/libz.so.1.2.3)
| | |   ->14.02% (131,072B) 0x3858CBFECA: ??? (in /lib64/libcrypto.so.0.9.8e)
| | | ->14.02% (131,072B) 0x3858CBFA96: COMP_CTX_new (in 
/lib64/libcrypto.so.0.9.8e)
| | |   ->07.01% (65,536B) 0x385E025CDF: tls1_change_cipher_state (in 
/lib64/libssl.so.0.9.8e)
| | |   | ->07.01% (65,536B) 0x385E020492: ssl3_do_change_cipher_spec (in 
/lib64/libssl.so.0.9.8e)
| | |   |   ->07.01% (65,536B) 0x385E021952: ssl3_read_bytes (in 
/lib64/libssl.so.0.9.8e)
| | |   | ->07.01% (65,536B) 0x385E0222AE: ssl3_get_message (in 
/lib64/libssl.so.0.9.8e)
| | |   |   ->07.01% (65,536B) 0x385E022B1B: ssl3_get_finished (in 
/lib64/libssl.so.0.9.8e)
| | |   | ->07.01% (65,536B) 0x385E01A740: ssl3_accept (in 
/lib64/libssl.so.0.9.8e)
| | |   |   ->07.01% (65,536B) 0x385E021883: ssl3_read_bytes (in 
/lib64/libssl.so.0.9.8e)
| | |   | ->07.01% (65,536B) 0x385E01E55F: ??? (in 
/lib64/libssl.so.0.9.8e)
| | |   |   ->07.01% (65,536B) 0x385E038F3F: ??? (in 
/lib64/libssl.so.0.9.8e)
| | |   | ->07.01% (65,536B) 0x3858C7678D: BIO_read (in 
/lib64/libcrypto.so.0.9.8e)
| | |   |   ->07.01% (65,536B) 0x46F7A3: bio_read 
(ssl.c:2058)
| | |   | ->07.01% (65,536B) 0x46FA8F: 
key_state_read_plaintext (ssl.c:2148)
| | |   |   ->07.01% (65,536B) 0x474859: tls_process 
(ssl.c:4075)
| | |   | ->07.01% (65,536B) 0x475420: 
tls_multi_process (ssl.c:4303)
| | |   |   ->07.01% (65,536B) 0x411B4C: 
check_tls_dowork (forward.c:93)
| | |   | ->07.01% (65,536B) 0x41571F: 
check_tls (forward-inline.h:41)
| | |   |   ->07.01% (65,536B) 0x415663: 
pre_select (forward.c:1304)
| | |   | ->07.01% (65,536B)