Re: Memory usage on relays
On Thu, Jan 28, 2010 at 11:51:05AM -0700, John Brooks wrote: > On Sun, Jan 17, 2010 at 9:29 PM, John Brooks wrote: > > Thanks for the prompt replies. I've got a custom-built Tor with > > openssl 1.0.0-beta4 running now; i'll keep an eye on it and see if > > things run better that way. > > > > I'll let you know in a few days how things are going with this new setup. > > As a belated update/conclusion to this thread, I've had Tor running > with openssl 1.0.0-beta5 for a bit over 7 days now. Current memory > usage is 133MB res, 155MB virtual, with 2107 open FDs. This is a very > significant improvement over the 750MB I had before, with the same > uptime. The SSL buffers do seem to have been the problem. I've updated the FAQ entry to reflect this tip along with others: https://www.torproject.org/faq#RelayMemory Let me know if anybody runs across more good suggestions. :) --Roger *** To unsubscribe, send an e-mail to majord...@torproject.org with unsubscribe or-talkin the body. http://archives.seul.org/or/talk/
Re: Memory usage on relays
On Sun, Jan 17, 2010 at 9:29 PM, John Brooks wrote: > Thanks for the prompt replies. I've got a custom-built Tor with > openssl 1.0.0-beta4 running now; i'll keep an eye on it and see if > things run better that way. > > I'll let you know in a few days how things are going with this new setup. > As a belated update/conclusion to this thread, I've had Tor running with openssl 1.0.0-beta5 for a bit over 7 days now. Current memory usage is 133MB res, 155MB virtual, with 2107 open FDs. This is a very significant improvement over the 750MB I had before, with the same uptime. The SSL buffers do seem to have been the problem. Thanks to everyone who helped. - John *** To unsubscribe, send an e-mail to majord...@torproject.org with unsubscribe or-talkin the body. http://archives.seul.org/or/talk/
Re: Memory usage on relays
Scott Bennett wrote: > On Tue, 19 Jan 2010 10:18:46 +0100 Olaf Selke > wrote: >> LD_PRELOAD"/foo/bar/libssl.so /foo/bar/libcrypto.so" >> in /etc/init.d/tor? > > Yup. It looks to be missing an equal sign. :) What?! ;-) You are right, it's a copy&paste error. Temporarily using an icc compiled openssl lib /etc/init.d/tor reads: LD_PRELOAD="/home/icc/tmp/openssl-0.9.8k/libssl.so /home/icc/tmp/openssl-0.9.8k/libcrypto.so" start-stop-daemon --start --quiet --oknodo \ [...] Olaf *** To unsubscribe, send an e-mail to majord...@torproject.org with unsubscribe or-talkin the body. http://archives.seul.org/or/talk/
Re: Memory usage on relays
Hi Olaf, On Tue, 19 Jan 2010 10:18:46 +0100 Olaf Selke wrote: >Nn6eumtr wrote: >> Binaries are staticly linked so that someone can't substitute a >> replacement library. Otherwise you can replace the library or set >> LDPRELOAD to implement a variety of attacks. > >can you give an example what's wrong with >LD_PRELOAD"/foo/bar/libssl.so /foo/bar/libcrypto.so" >in /etc/init.d/tor? Yup. It looks to be missing an equal sign. :) Scott Bennett, Comm. ASMELG, CFIAG ** * Internet: bennett at cs.niu.edu * ** * "A well regulated and disciplined militia, is at all times a good * * objection to the introduction of that bane of all free governments * * -- a standing army." * *-- Gov. John Hancock, New York Journal, 28 January 1790 * ** *** To unsubscribe, send an e-mail to majord...@torproject.org with unsubscribe or-talkin the body. http://archives.seul.org/or/talk/
Re: Memory usage on relays
On Tue, Jan 19, 2010 at 4:18 AM, Olaf Selke wrote: > Nn6eumtr wrote: >> Binaries are staticly linked so that someone can't substitute a >> replacement library. Otherwise you can replace the library or set >> LDPRELOAD to implement a variety of attacks. > > can you give an example what's wrong with > LD_PRELOAD"/foo/bar/libssl.so /foo/bar/libcrypto.so" > in /etc/init.d/tor? > > That's how I enable special openssl versions on Debian. The failure mode is if you somehow wind up in a position where an adversary is in control of your environment; they could set LD_PRELOAD or LD_PATH to whatever they wanted. Personally, I'm not convinced that this is a reason not to dynamically link. Most attacks that would give somebody write access to your environment would let them subvert your system in ways that don't require dynamic linking. (That is, If the attacker can run arbitrary shell commands, put stuff in your ~/.profile, or mess with your shell process's memory, then they're in a great position whether your binaries are static or not.) I'm not alone in thinking this: there are some pretty paranoid applications out there (gnupg and openssh for example) that are happy to use the dynamic linker. yrs, -- Nick *** To unsubscribe, send an e-mail to majord...@torproject.org with unsubscribe or-talkin the body. http://archives.seul.org/or/talk/
Re: Memory usage on relays
Nn6eumtr wrote: > Binaries are staticly linked so that someone can't substitute a > replacement library. Otherwise you can replace the library or set > LDPRELOAD to implement a variety of attacks. can you give an example what's wrong with LD_PRELOAD"/foo/bar/libssl.so /foo/bar/libcrypto.so" in /etc/init.d/tor? That's how I enable special openssl versions on Debian. Olaf *** To unsubscribe, send an e-mail to majord...@torproject.org with unsubscribe or-talkin the body. http://archives.seul.org/or/talk/
Re: Memory usage on relays
Binaries are staticly linked so that someone can't substitute a replacement library. Otherwise you can replace the library or set LDPRELOAD to implement a variety of attacks. On Jan 18, 2010, at 2:28 PM, Nick Mathewson wrote: On Sun, Jan 17, 2010 at 11:29 PM, John Brooks > wrote: [...] As a vaguely related sidenote, is it intentional that openssl is statically linked? I would expect that Tor more than anything would want to benefit from security updates as quickly as possible, and most package managers / people won't rebuild it after an openssl update. Seems a bit dangerous. I was able to confirm that I was running with the right version, though, by adding the following right under Tor's version notice: Tor links against openssl dynamically for me, at least. Let us know if there's some more magic we need to do in src/or/Makefile.am to make it dynamically linke for others. I'm not sure openssl builds shared libraries by default, though: could that be the problem. -- Nick *** To unsubscribe, send an e-mail to majord...@torproject.org with unsubscribe or-talkin the body. http://archives.seul.org/or/talk/ *** To unsubscribe, send an e-mail to majord...@torproject.org with unsubscribe or-talkin the body. http://archives.seul.org/or/talk/
Re: Memory usage on relays
On Sun, Jan 17, 2010 at 11:29 PM, John Brooks wrote: [...] > As a vaguely related sidenote, is it intentional that openssl is > statically linked? I would expect that Tor more than anything would > want to benefit from security updates as quickly as possible, and most > package managers / people won't rebuild it after an openssl update. > Seems a bit dangerous. I was able to confirm that I was running with > the right version, though, by adding the following right under Tor's > version notice: Tor links against openssl dynamically for me, at least. Let us know if there's some more magic we need to do in src/or/Makefile.am to make it dynamically linke for others. I'm not sure openssl builds shared libraries by default, though: could that be the problem. -- Nick *** To unsubscribe, send an e-mail to majord...@torproject.org with unsubscribe or-talkin the body. http://archives.seul.org/or/talk/
Re: Memory usage on relays
John Brooks wrote: > This topic has been addressed before, but then the answer was often to > use/wait for 0.2.1.x or switch to another allocator. "./configure --enable-openbsd-malloc" solved this memory leak issue on my Debian box. Currently tor is using 354m resident memory and 20k open tcp sessions. Olaf *** To unsubscribe, send an e-mail to majord...@torproject.org with unsubscribe or-talkin the body. http://archives.seul.org/or/talk/
Re: Memory usage on relays
On Sun, Jan 17, 2010 at 8:31 PM, Nick Mathewson wrote: > On Sun, Jan 17, 2010 at 9:36 PM, Roger Dingledine wrote: >> Nick wrote an OpenSSL patch to not waste so much memory in its internal >> buffers. See item #3 on >> http://archives.seul.org/or/dev/Jun-2008/msg1.html >> >> That said, I don't know what the current state of the patch is, or where >> you can get a copy. Nick? > > It's in recent versions of OpenSSL (recent as in the 1.0.0 beta versions.) > > If you would rather try patching an older version of OpenSSL yourself, try out > http://freehaven.net/~nickm/openssl_mem/openssl-mem-patch-v17.txt > I have no idea whether it applies cleanly (or at all) to older versions. Thanks for the prompt replies. I've got a custom-built Tor with openssl 1.0.0-beta4 running now; i'll keep an eye on it and see if things run better that way. As a vaguely related sidenote, is it intentional that openssl is statically linked? I would expect that Tor more than anything would want to benefit from security updates as quickly as possible, and most package managers / people won't rebuild it after an openssl update. Seems a bit dangerous. I was able to confirm that I was running with the right version, though, by adding the following right under Tor's version notice: log(LOG_NOTICE, LD_GENERAL, "Built for %s, using %s", OPENSSL_VERSION_TEXT, SSLeay_version(SSLEAY_VERSION)); That may be worth adding, to make which version is being used visible, especially if it's going to be statically linked. I'll let you know in a few days how things are going with this new setup. - John *** To unsubscribe, send an e-mail to majord...@torproject.org with unsubscribe or-talkin the body. http://archives.seul.org/or/talk/
Re: Memory usage on relays
On Sun, Jan 17, 2010 at 9:36 PM, Roger Dingledine wrote: > On Sun, Jan 17, 2010 at 06:41:03PM -0700, John Brooks wrote: >> I run a reasonably fast (500KB/s) node with Guard+Fast+Stable, so it's >> a popular destination. It runs at bandwidth capacity at all times. The >> only problem with this is the massive memory usage that results; at >> the moment, Tor has 748MB res usage, with almost 7 days of uptime. >> Generally it escalates at a rate of 100-200MB per day after a restart, >> and tops out around this number. My understanding is that most of that >> memory usage is related to the open connections; socket buffers, SSL >> buffers, etc. At the moment (according to /proc/x/fd), Tor has 5,364 >> open connections. > > Nick wrote an OpenSSL patch to not waste so much memory in its internal > buffers. See item #3 on > http://archives.seul.org/or/dev/Jun-2008/msg1.html > > I ran a super-fast Tor relay recently that held 15000 TLS connections > open. That's 550MB of ram wasted inside openssl. > > That said, I don't know what the current state of the patch is, or where > you can get a copy. Nick? It's in recent versions of OpenSSL (recent as in the 1.0.0 beta versions.) If you would rather try patching an older version of OpenSSL yourself, try out http://freehaven.net/~nickm/openssl_mem/openssl-mem-patch-v17.txt I have no idea whether it applies cleanly (or at all) to older versions. hth, -- Nick *** To unsubscribe, send an e-mail to majord...@torproject.org with unsubscribe or-talkin the body. http://archives.seul.org/or/talk/
Re: Memory usage on relays
On Sun, Jan 17, 2010 at 06:41:03PM -0700, John Brooks wrote: > I run a reasonably fast (500KB/s) node with Guard+Fast+Stable, so it's > a popular destination. It runs at bandwidth capacity at all times. The > only problem with this is the massive memory usage that results; at > the moment, Tor has 748MB res usage, with almost 7 days of uptime. > Generally it escalates at a rate of 100-200MB per day after a restart, > and tops out around this number. My understanding is that most of that > memory usage is related to the open connections; socket buffers, SSL > buffers, etc. At the moment (according to /proc/x/fd), Tor has 5,364 > open connections. Nick wrote an OpenSSL patch to not waste so much memory in its internal buffers. See item #3 on http://archives.seul.org/or/dev/Jun-2008/msg1.html I ran a super-fast Tor relay recently that held 15000 TLS connections open. That's 550MB of ram wasted inside openssl. That said, I don't know what the current state of the patch is, or where you can get a copy. Nick? --Roger *** To unsubscribe, send an e-mail to majord...@torproject.org with unsubscribe or-talkin the body. http://archives.seul.org/or/talk/
Memory usage on relays
This topic has been addressed before, but then the answer was often to use/wait for 0.2.1.x or switch to another allocator. I'm thinking there must be a real solution to this somewhere. I run a reasonably fast (500KB/s) node with Guard+Fast+Stable, so it's a popular destination. It runs at bandwidth capacity at all times. The only problem with this is the massive memory usage that results; at the moment, Tor has 748MB res usage, with almost 7 days of uptime. Generally it escalates at a rate of 100-200MB per day after a restart, and tops out around this number. My understanding is that most of that memory usage is related to the open connections; socket buffers, SSL buffers, etc. At the moment (according to /proc/x/fd), Tor has 5,364 open connections. Short of limiting available FDs, which might harm the performance of the node, what can I do to lower memory usage? It's currently running the Debian testing build, 0.2.1.20-2, and openssl 0.9.8g-15+lenny6. I'm not against doing custom builds of Tor or OpenSSL if it will help. I did have similar problems on my previous machine running Gentoo, where I had tried different the different allocator configurations, and that had little or no effect. Somebody else has to have come across this problem and some sort of solution; I can't have Tor taking up half the available memory on my system. Suggestions would be very appreciated. Thanks, - John Brooks *** To unsubscribe, send an e-mail to majord...@torproject.org with unsubscribe or-talkin the body. http://archives.seul.org/or/talk/
