Re: [Dovecot] v2.1 memory usage
On 2012-11-12 2:31 PM, Ed W li...@wildgooses.com wrote: Openssl is a *massive* project and I'm unsure that gnutls is much smaller... We should assume that both are quite scary from a security point of view. Licensing is the main thing which divides them, gnutls is stated as GPL compatible (however, the nominal incompatibility of openssl seems difficult to understand?) OpenVPN integrated with PolarSSL and got Dutch government official approval for the combined package. I think elsewhere it's stated that openssl would not have been approved because something like the codebase was too large to inspect and sign off http://polarssl.org/news?item=0132 I haven't worked with PolarSSL, so no idea, but it's massively smaller codebase is likely attractive if you are the kind of person who actually *does* security audits on the software you run in secure situations. Openssl is just a complete swiss army knife of tools! Very interesting... I'd certainly be interested in switching to it, as long as the most common tools are supported Is it considered a drop in replacement for OpenSSL? -- Best regards, Charles
Re: [Dovecot] v2.1 memory usage
On 2012-11-12 20:44, Timo Sirainen wrote: On 12.11.2012, at 6.13, Daniel L. Miller wrote: The tiny bit of Googling I've done tells me GnuTLS seems to be a more standards-compliant implementation, and MAY be safer than OpenSSL. However, as OpenSSL is the de-facto standard used by most Linux programs, acceptance of GnuTLS is quite limited. I've been intrigued by what I've read about it, and took a quick look at enabling support in Dovecot for GnuTLS directly - but while it didn't seem overly heavy at first glance the fact that Timo doesn't want to do it tells me I'm underestimating the complexity. I already once wrote GnuTLS support for Dovecot, but GnuTLS changed its APIs since then and it was probably originally already buggy. I think the only somewhat special APIs that Dovecot needs nowadays are related to reading cert/keys from memory instead of from files. If GnuTLS can do that, I don't think there's anything special in supporting it. Although it might be a bit complex to make it work properly asynchronously. istream-openssl was a bit annoying in that way (all the data read from the fd must be parsed and decoded all the way through to the SSL istream, regardless of any max buffer limits). A while ago, I started working on GnuTLS support for Dovecot. While I didn't finish the implementation due to time constraints (the abstract API of lib-ssl-iostream is made for OpenSSL and you have to emulate some things), I think it would be possible to add GnuTLS support and with GnuTLS it would be possible to load X.509 certificates from memory. Regards, Matthias-Christian
Re: [Dovecot] v2.1 memory usage
On 12/11/2012 04:13, Daniel L. Miller wrote: The tiny bit of Googling I've done tells me GnuTLS seems to be a more standards-compliant implementation, and MAY be safer than OpenSSL. However, as OpenSSL is the de-facto standard used by most Linux programs, acceptance of GnuTLS is quite limited. I've been intrigued by what I've read about it, and took a quick look at enabling support in Dovecot for GnuTLS directly - but while it didn't seem overly heavy at first glance the fact that Timo doesn't want to do it tells me I'm underestimating the complexity. Openssl is a *massive* project and I'm unsure that gnutls is much smaller... We should assume that both are quite scary from a security point of view. Licensing is the main thing which divides them, gnutls is stated as GPL compatible (however, the nominal incompatibility of openssl seems difficult to understand?) OpenVPN integrated with PolarSSL and got Dutch government official approval for the combined package. I think elsewhere it's stated that openssl would not have been approved because something like the codebase was too large to inspect and sign off http://polarssl.org/news?item=0132 I haven't worked with PolarSSL, so no idea, but it's massively smaller codebase is likely attractive if you are the kind of person who actually *does* security audits on the software you run in secure situations. Openssl is just a complete swiss army knife of tools! Ed W
Re: [Dovecot] v2.1 memory usage
On 12.11.2012, at 6.13, Daniel L. Miller wrote: where is the problem with openssl? I don't know what the problem is - I just know that I've heard from a number of developers (including the Postfix Dovecot developers) that they don't like OpenSSL - but while GnuTLS looks interesting they aren't interested in working on the interface - though they're willing to accept patches. (My full apologies right now if Timo or Wietse are offended by my speaking out of turn). OpenSSL documentation is very bad. Its API has some annoying missing features. For example you can load certificates from a directory or a file but not from anything else like from a string in memory. I had to copypaste a few functions from OpenSSL code just to be able to do them. The tiny bit of Googling I've done tells me GnuTLS seems to be a more standards-compliant implementation, and MAY be safer than OpenSSL. However, as OpenSSL is the de-facto standard used by most Linux programs, acceptance of GnuTLS is quite limited. I've been intrigued by what I've read about it, and took a quick look at enabling support in Dovecot for GnuTLS directly - but while it didn't seem overly heavy at first glance the fact that Timo doesn't want to do it tells me I'm underestimating the complexity. I already once wrote GnuTLS support for Dovecot, but GnuTLS changed its APIs since then and it was probably originally already buggy. I think the only somewhat special APIs that Dovecot needs nowadays are related to reading cert/keys from memory instead of from files. If GnuTLS can do that, I don't think there's anything special in supporting it. Although it might be a bit complex to make it work properly asynchronously. istream-openssl was a bit annoying in that way (all the data read from the fd must be parsed and decoded all the way through to the SSL istream, regardless of any max buffer limits).
Re: [Dovecot] v2.1 memory usage
On 11/6/2012 12:30 PM, Timo Sirainen wrote: On 6.11.2012, at 17.26, Ed W wrote: On 05/11/2012 23:22, Timo Sirainen wrote: On Mon, 2012-11-05 at 23:40 +0200, Timo Sirainen wrote: This also provides a nice abstraction to OpenSSL, making it again possible to implement other backends like GnuTLS or NSS. (Except login process code doesn't use lib-ssl-iostream yet.) Does libtomcrypt implement enough? It doesn't do SSL, which is all Dovecot cares about. Can the GnuTLS OpenSSL compatibility layer be used safely? -- Daniel
Re: [Dovecot] v2.1 memory usage
Am 12.11.2012 02:11, schrieb Daniel L. Miller: On 11/6/2012 12:30 PM, Timo Sirainen wrote: On 6.11.2012, at 17.26, Ed W wrote: On 05/11/2012 23:22, Timo Sirainen wrote: On Mon, 2012-11-05 at 23:40 +0200, Timo Sirainen wrote: This also provides a nice abstraction to OpenSSL, making it again possible to implement other backends like GnuTLS or NSS. (Except login process code doesn't use lib-ssl-iostream yet.) Does libtomcrypt implement enough? It doesn't do SSL, which is all Dovecot cares about. Can the GnuTLS OpenSSL compatibility layer be used safely? where is the problem with openssl? please leave us in peace with gnuTLS and see how it affects OpenVAS / Greenbone Sceurity Assistant on distributions like Fedora the whole year what about config compatibility like ssl_cipher_list = ALL:!LOW:!MEDIUM:!SSLv2:!MD5:!aNULL:!eNUL:!ADH:!AESGCM:!EXP:HIGH signature.asc Description: OpenPGP digital signature
Re: [Dovecot] v2.1 memory usage
On 2012-11-11 17:20, Reindl Harald wrote: Am 12.11.2012 02:11, schrieb Daniel L. Miller: On 11/6/2012 12:30 PM, Timo Sirainen wrote: On 6.11.2012, at 17.26, Ed W wrote: On 05/11/2012 23:22, Timo Sirainen wrote: On Mon, 2012-11-05 at 23:40 +0200, Timo Sirainen wrote: This also provides a nice abstraction to OpenSSL, making it again possible to implement other backends like GnuTLS or NSS. (Except login process code doesn't use lib-ssl-iostream yet.) Does libtomcrypt implement enough? It doesn't do SSL, which is all Dovecot cares about. Can the GnuTLS OpenSSL compatibility layer be used safely? where is the problem with openssl? I don't know what the problem is - I just know that I've heard from a number of developers (including the Postfix Dovecot developers) that they don't like OpenSSL - but while GnuTLS looks interesting they aren't interested in working on the interface - though they're willing to accept patches. (My full apologies right now if Timo or Wietse are offended by my speaking out of turn). I'm no security expert, but I do know that OpenSSL has had issues with version compatiblity. I had a very troubled time during an OpenSSL/Postfix upgrade that left me non-functional until I found the exact version pairings required. The tiny bit of Googling I've done tells me GnuTLS seems to be a more standards-compliant implementation, and MAY be safer than OpenSSL. However, as OpenSSL is the de-facto standard used by most Linux programs, acceptance of GnuTLS is quite limited. I've been intrigued by what I've read about it, and took a quick look at enabling support in Dovecot for GnuTLS directly - but while it didn't seem overly heavy at first glance the fact that Timo doesn't want to do it tells me I'm underestimating the complexity. -- Daniel
Re: [Dovecot] v2.1 memory usage
On 05/11/2012 23:22, Timo Sirainen wrote: On Mon, 2012-11-05 at 23:40 +0200, Timo Sirainen wrote: Anyway, looks like Dovecot can't link OpenSSL to imap/pop3 processes without wasting a ton of memory. In v2.2 I already moved imapc/pop3c backend code to plugins to avoid this. Looks like similar ugliness is needed for other features/backends also that may end up using SSL code. (We were wondering with Stephan what to do about his new HTTP library code that added support for SSL. It would be nice to keep it in the core libdovecot.so, but not if it links with SSL. So looks like we'll need some kind of a http-ssl plugin that is loaded only when needed.) Implemented it a bit easier way that also gets rid of imapc/pop3c plugins and simplifies other things: lib-ssl-iostream now loads OpenSSL dynamically: http://hg.dovecot.org/dovecot-2.2/rev/68d21f872fd7 This also provides a nice abstraction to OpenSSL, making it again possible to implement other backends like GnuTLS or NSS. (Except login process code doesn't use lib-ssl-iostream yet.) Does libtomcrypt implement enough? Ed
Re: [Dovecot] v2.1 memory usage
On 6.11.2012, at 17.26, Ed W wrote: On 05/11/2012 23:22, Timo Sirainen wrote: On Mon, 2012-11-05 at 23:40 +0200, Timo Sirainen wrote: Anyway, looks like Dovecot can't link OpenSSL to imap/pop3 processes without wasting a ton of memory. In v2.2 I already moved imapc/pop3c backend code to plugins to avoid this. Looks like similar ugliness is needed for other features/backends also that may end up using SSL code. (We were wondering with Stephan what to do about his new HTTP library code that added support for SSL. It would be nice to keep it in the core libdovecot.so, but not if it links with SSL. So looks like we'll need some kind of a http-ssl plugin that is loaded only when needed.) Implemented it a bit easier way that also gets rid of imapc/pop3c plugins and simplifies other things: lib-ssl-iostream now loads OpenSSL dynamically: http://hg.dovecot.org/dovecot-2.2/rev/68d21f872fd7 This also provides a nice abstraction to OpenSSL, making it again possible to implement other backends like GnuTLS or NSS. (Except login process code doesn't use lib-ssl-iostream yet.) Does libtomcrypt implement enough? It doesn't do SSL, which is all Dovecot cares about.
Re: [Dovecot] v2.1 memory usage
On Tue, 2012-09-11 at 21:43 +0300, Timo Sirainen wrote: v2.1's imap/pop3 processes are linked with OpenSSL by default, because of imapc/pop3c storages. I wonder if you notice any actual (system-global) memory usage reduction when compiling without those and you have thousands of imap processes? ./configure --with-storages=mdbox sdbox maildir mbox cydir I guess I could try it myself also but maybe someone else wanting to reduce memory usage is quicker? :) Related to that, you can look at where imap processes' memory goes with: http://dovecot.org/tools/linux-maps-parse-imap.pl I don't know how much memory those (writable!) file-backed memory areas actually use per-process. Maybe nothing, maybe something.. http://dovecot.org/tmp/proctest.sh can be used to easily test this. Looks like simply linking a binary with -lcrypto increases each process's memory usage by something like: * RHEL5: 50 kB * RHEL6/Debian/Ubuntu: 150-200 kB * Solaris: 100 kB * OpenBSD: 20 kB * OSX: 0 kB Looks like it's the linker that uses up all of this memory. I'm not sure why exactly though, I couldn't create a test library that had the same effect. I also tried imap binary that was statically built vs. linked with libdovecot*.so, and the difference was pretty small (.so used maybe 10-20k more). Anyway, looks like Dovecot can't link OpenSSL to imap/pop3 processes without wasting a ton of memory. In v2.2 I already moved imapc/pop3c backend code to plugins to avoid this. Looks like similar ugliness is needed for other features/backends also that may end up using SSL code. (We were wondering with Stephan what to do about his new HTTP library code that added support for SSL. It would be nice to keep it in the core libdovecot.so, but not if it links with SSL. So looks like we'll need some kind of a http-ssl plugin that is loaded only when needed.)
Re: [Dovecot] v2.1 memory usage
On Mon, 2012-11-05 at 23:40 +0200, Timo Sirainen wrote: Anyway, looks like Dovecot can't link OpenSSL to imap/pop3 processes without wasting a ton of memory. In v2.2 I already moved imapc/pop3c backend code to plugins to avoid this. Looks like similar ugliness is needed for other features/backends also that may end up using SSL code. (We were wondering with Stephan what to do about his new HTTP library code that added support for SSL. It would be nice to keep it in the core libdovecot.so, but not if it links with SSL. So looks like we'll need some kind of a http-ssl plugin that is loaded only when needed.) Implemented it a bit easier way that also gets rid of imapc/pop3c plugins and simplifies other things: lib-ssl-iostream now loads OpenSSL dynamically: http://hg.dovecot.org/dovecot-2.2/rev/68d21f872fd7 This also provides a nice abstraction to OpenSSL, making it again possible to implement other backends like GnuTLS or NSS. (Except login process code doesn't use lib-ssl-iostream yet.)
Re: [Dovecot] v2.1 memory usage
Only have 650 imap clients this weekend, have to give it a try during a weekday. Total(MB) Avg(kB) Name 2 4 /lib/ld-2.11.1.so 2 4 /lib/libbz2.so.1.0.4 2 4 /lib/libc-2.11.1.so 63 96 /lib/libcrypto.so.0.9.8 2 4 /lib/libdl-2.11.1.so 0 0 /lib/libnss_dns-2.11.1.so 0 0 /lib/libnss_files-2.11.1.so 2 4 /lib/libpthread-2.11.1.so 0 0 /lib/libresolv-2.11.1.so 2 4 /lib/librt-2.11.1.so 13 20 /lib/libssl.so.0.9.8 2 4 /lib/libz.so.1.2.3.3 2 4 /usr/lib/dovecot/imap 10 16 /usr/lib/dovecot/libdovecot-storage.so.0.0.0 2 4 /usr/lib/dovecot/libdovecot.so.0.0.0 2 4 /usr/lib/dovecot/modules/lib10_quota_plugin.so 2 4 /usr/lib/dovecot/modules/lib11_imap_quota_plugin.so 2 4 /usr/lib/dovecot/modules/lib20_zlib_plugin.so 2 4 /usr/lib/dovecot/modules/lib30_imap_zlib_plugin.so 435 661 [anonymous] 640 973 [heap] 55 84 [stack] 1253 58352 TOTAL Quoting Timo Sirainen t...@iki.fi: v2.1's imap/pop3 processes are linked with OpenSSL by default, because of imapc/pop3c storages. I wonder if you notice any actual (system-global) memory usage reduction when compiling without those and you have thousands of imap processes? ./configure --with-storages=mdbox sdbox maildir mbox cydir I guess I could try it myself also but maybe someone else wanting to reduce memory usage is quicker? :) Related to that, you can look at where imap processes' memory goes with: http://dovecot.org/tools/linux-maps-parse-imap.pl I don't know how much memory those (writable!) file-backed memory areas actually use per-process. Maybe nothing, maybe something..
[Dovecot] v2.1 memory usage
v2.1's imap/pop3 processes are linked with OpenSSL by default, because of imapc/pop3c storages. I wonder if you notice any actual (system-global) memory usage reduction when compiling without those and you have thousands of imap processes? ./configure --with-storages=mdbox sdbox maildir mbox cydir I guess I could try it myself also but maybe someone else wanting to reduce memory usage is quicker? :) Related to that, you can look at where imap processes' memory goes with: http://dovecot.org/tools/linux-maps-parse-imap.pl I don't know how much memory those (writable!) file-backed memory areas actually use per-process. Maybe nothing, maybe something..
