Re: Reduced Tor Traffic [was: Re: peculiar server...]
Wouldn't the throughput of your server depend on the nodes up and downstream of it? On Sep 9, 2008, at 8:02 PM, Lucky Green wrote: Roger Dingledine wrote: On Tue, Sep 09, 2008 at 05:15:15AM -0500, Scott Bennett wrote: [...] That brings us back to something I've already posted on OR- TALK, namely, the apparent slowdown in tor traffic that has reduced the traffic through my tor server by at least 30% and, judging from the reduced peaks shown for a lot of the high-volume servers listed on the torstatus page, the tor network at large. We're working on plans to start gathering more methodical data about how the network has run and is running, with the goal of being able to answer questions like this more usefully. I am very much looking forward to more diagnostic instrumentation in the Tor network. I am seeing a 30% difference in the traffic through basically identical servers that are neither bandwidth nor CPU limited with identical uptimes. Something about the path selection appears to lead to favor one server over another. Also interesting to me is the overall reduced amount of traffic over the last few months that I have been seeing with my middleman nodes. The most likely explanation is that the overall Tor network capacity is exit node bound and that middleman nodes have grown disproportionately over time. Still, it sure would be nice to be able to perform rigorous analysis on the network. --Lucky
Re: Reduced Tor Traffic [was: Re: peculiar server...]
On Tue, Sep 9, 2008 at 8:02 PM, Lucky Green <[EMAIL PROTECTED]> wrote: [snip] > Also interesting to me is the overall reduced amount of traffic over the > last few months that I have been seeing with my middleman nodes. The > most likely explanation is that the overall Tor network capacity is exit > node bound and that middleman nodes have grown disproportionately over > time. [snip] Considering the implications of exit-node operation, I don't expect that pattern to end any time soon. Perhaps it would be reasonable to do more promotion of hidden services, OnionCat, exit-enclaves, and other non-exit limited services to make good use of what is likely to be an excess of middleman capacity, and help drive more tor adoption?
Reduced Tor Traffic [was: Re: peculiar server...]
Roger Dingledine wrote: > On Tue, Sep 09, 2008 at 05:15:15AM -0500, Scott Bennett wrote: > > [...] >> That brings us back to something I've already posted on OR-TALK, namely, >> the apparent slowdown in tor traffic that has reduced the traffic through my >> tor server by at least 30% and, judging from the reduced peaks shown for a >> lot >> of the high-volume servers listed on the torstatus page, the tor network at >> large. >> > > We're working on plans to start gathering more methodical data about > how the network has run and is running, with the goal of being able to > answer questions like this more usefully. > I am very much looking forward to more diagnostic instrumentation in the Tor network. I am seeing a 30% difference in the traffic through basically identical servers that are neither bandwidth nor CPU limited with identical uptimes. Something about the path selection appears to lead to favor one server over another. Also interesting to me is the overall reduced amount of traffic over the last few months that I have been seeing with my middleman nodes. The most likely explanation is that the overall Tor network capacity is exit node bound and that middleman nodes have grown disproportionately over time. Still, it sure would be nice to be able to perform rigorous analysis on the network. --Lucky
Re: peculiar server "bandwidth" posted by server "mnl" and possible new type of attack
Kasimir Gabert wrote: > > Would you mind looking over the new > router detail page and seeing if it looks reasonable to you? You can > view it at > http://trunk.torstatus.kgprog.com/router_detail.php?FP=795513a52e5155af5e36937d5a7c76d3bf20d0c4 Sir, Yes Sir! It appears to be slighly below my mrtg interface reporting but it's definitely more realistic than peak bandwidth reporting from the well known tns stats sites. Keep you posted... regards Olaf
Re: Ports 465/587 in exit policy (was Re: Update to default exit policy)
F. Fox([EMAIL PROTECTED])@Sun, Sep 07, 2008 at 06:27:08PM -0700: > Bill Weiss wrote: > (snip) > > My Tor node runs a medium-load mail server as well, and I've never been > > blacklisted for spam stuff [1]. That seems like a decent indication of it > > not causing problems given how rabid the anti-spam people can get. > > > > 1: I've gotten blacklisted twice by SORBS for "virus" activities, which > > were people using IRC (for bad things, I assume) via my node. That > > doesn't count. > > > > I've gotten on some DNSBL list, which basically keeps me off of several > IRC networks. The catch is: I'm running a middleman-only node! Ugh, yes. I pretty much can't SSH from my shell server (/ Tor server / mail server / etc) because of that. The kicker is, I don't allow most IRC traffic out. It's really time to buy a new IP or two. -- Bill Weiss There is no 'patch' for stupidity. -- SQLSecurity.com
Re: peculiar server "bandwidth" posted by server "mnl" and possible new type of attack
On Tue, Sep 9, 2008 at 8:10 AM, Olaf Selke <[EMAIL PROTECTED]> wrote: > Scott Bennett wrote: >> >> Nearly 49 MB/s seems a bit of a stretch. The server's operator sent me >> a note saying that the server is attached to the 1 GB/s campus backbone net, >> but it is attached via a 100 Mb/s router, so the reported data rate is four >> to five times the rate physically possible due to the router's limitation. >> The server, according to its operator, is running on a 2.6 GHz P4, and its >> descriptor says the machine is running LINUX. Based upon postings quite a >> while back from blutmagie's operator and from a few other operators of very >> high-data-rate servers, it seems to me that a 2.6 GHz P4 (Northwood?) running >> LINUX would not be capable of handling a load eight to ten times that of >> blutmagie, regardless of its network connection's capacity. > > blutmagie tor node is running on a pair of the old Prestonia P4 NetBurst Xeon > DP > 3200MHz processors. Over the last four weeks mrtg monitoring is showing an > average interface throughput of 32 MBit/s in and 33 MBit/s out. Throughput is > limited by cpu power rather than by available network bandwidth. Since Tor > doesn't scale very well with the number of cores, one core is loaded with > 100%, > leaving the other three cores almost idle. Compiling the openssl library with > Intel's C compiler icc improved performance by about 20-25% compared with gcc > (compiling tor with icc doesn't change very much). That's the reason > blutmagie's > observed data rate increased from about 5500 to nearly 7000 KByte/s some > weeks ago. > > regards Olaf > Hello Olaf, For the upcoming version of TorStatus, I followed Roger's suggestion of calculating the observed bandwidth using the read and write history instead of the given observed bandwidth rate. After doing this, and what seems to follow the graphs relatively accurately, your bandwidth rate has dropped to 3463.39 KB/s. I'm using a linear moving average to calculate the bandwidth. Would you mind looking over the new router detail page and seeing if it looks reasonable to you? You can view it at http://trunk.torstatus.kgprog.com/router_detail.php?FP=795513a52e5155af5e36937d5a7c76d3bf20d0c4 Also, with regards to mnl, it is down now but I can remember that when it was running I noticed how it was the lead by a large margin on the current TorStatus page, but was at something like 200 on the trunk page. It seems like it never received the amount of traffic that it could handle, or something similar. Kasimir -- Kasimir Gabert
Re: peculiar server "bandwidth" posted by server "mnl" and possible new type of attack
Scott Bennett wrote: > > Nearly 49 MB/s seems a bit of a stretch. The server's operator sent me > a note saying that the server is attached to the 1 GB/s campus backbone net, > but it is attached via a 100 Mb/s router, so the reported data rate is four > to five times the rate physically possible due to the router's limitation. > The server, according to its operator, is running on a 2.6 GHz P4, and its > descriptor says the machine is running LINUX. Based upon postings quite a > while back from blutmagie's operator and from a few other operators of very > high-data-rate servers, it seems to me that a 2.6 GHz P4 (Northwood?) running > LINUX would not be capable of handling a load eight to ten times that of > blutmagie, regardless of its network connection's capacity. blutmagie tor node is running on a pair of the old Prestonia P4 NetBurst Xeon DP 3200MHz processors. Over the last four weeks mrtg monitoring is showing an average interface throughput of 32 MBit/s in and 33 MBit/s out. Throughput is limited by cpu power rather than by available network bandwidth. Since Tor doesn't scale very well with the number of cores, one core is loaded with 100%, leaving the other three cores almost idle. Compiling the openssl library with Intel's C compiler icc improved performance by about 20-25% compared with gcc (compiling tor with icc doesn't change very much). That's the reason blutmagie's observed data rate increased from about 5500 to nearly 7000 KByte/s some weeks ago. regards Olaf
Re: peculiar server "bandwidth" posted by server "mnl" and possible new type of attack
On Tue, 9 Sep 2008 06:42:02 -0400 Roger Dingledine <[EMAIL PROTECTED]> wrote: >On Tue, Sep 09, 2008 at 05:15:15AM -0500, Scott Bennett wrote: >> bandwidth 5242880 10485760 52239166 >> ---> ~48.8 MB/s (!) > >Wow. Nice! :) > >And, as you say, unlikely to be true. > >> All of the above leads me to suspect two things. One is that there may >> be some bug triggered only under exceedingly odd conditions that leads to >> reporting of data rates grossly out of line with reality. > >Yep. Probably an integer underflow somewhere. That would certainly be nice. I hope it's so easy. > >> The second is that when a server claims such disproportionately high >> capacity, the overall performance of the tor server network can be >> compromised. > >There has been quite a bit of work on this one, at least. > >First, clients use the min of the bandwidthrate and the measured >bandwidth, so in the above "5242880 10485760 52239166" example clients >will load-balance based on 5242880. So this bug doesn't have any real >effect, at least in this case. > >Also, clients cap the number they'll believe at 10MB, so an evil relay >can only sucker people so much. Whew. Thank you so much, Roger! That's exactly the kind of thing I wanted for reassurance. However...hmmm...how much disruption could, say, a dozen i486's on 10 Mb/s ethernet connections claiming to be 5 MB/s servers cause? That ought to be fairly cheap, well within the coffee budget of an NSA office, for example. > >There are research designs for doing community-measured bandwidth >numbers, which would make it much harder for an evil relay to sucker >people. See for example http://freehaven.net/anonbib/#snader08 >Unfortunately, all of these designs have downsides currently, so they're >not ready for deployment yet. > >> That brings us back to something I've already posted on OR-TALK, namely, >> the apparent slowdown in tor traffic that has reduced the traffic through my >> tor server by at least 30% and, judging from the reduced peaks shown for a >> lot >> of the high-volume servers listed on the torstatus page, the tor network at >> large. > >We're working on plans to start gathering more methodical data about >how the network has run and is running, with the goal of being able to >answer questions like this more usefully. Sounds good. I'll look forward to that. > >Another example I'd love to investigate is the apparent feedback effect -- >once a relay happens to have a high-volume flow through it, it tends to >continue to advertise high bandwidth numbers for weeks after, whereas if >it doesn't start with any high-volume flows, it never seems to attract >them later. Yes, indeed. That is something I've posted about twice recently and gotten no responses. Actual changes in the physical capacity of a server and its connection to the Internet are relatively rare events, most of which would involve a restart of the tor server anyway. The published actual usage (a.k.a. observed "bandwidth" [jeez, what a misappropriated term that is]) is used as a means of determining what that physical limit may be, which is then used by clients everywhere. Ideally, a sequence of such reports from a server should constitute a successive approximation that converges on the actual physical limitation. However, in tor's current method of reporting reduced values for reduced *usage* and having those reduced values then understood to represent *capacity*, the series never converges. So instead of seeing a series of reported usages that increases asymptotically toward the capacity, tor sees fictionally varying *capacities*. The feedback you mention would be fine if not permitted to create nonlinear responses. This problem could be eliminated by having tor always publish the greater of the last published value and the highest value from the reporting period (currently the previous 24 hours, right?). A second potential issue under the current implementation is that the default sampling rate is 1.333.../day, so if there are shorter term, but repetitive, variations (e.g., daily fluctuations), tor has no possibility of detecting and recognizing them. 1.333.../day gives a Nyquist frequency of .666.../day, so any normal oscillations in traffic volume that happen faster than that (i.e., with a period shorter than 1.333... days = 32 hours) cannot be detected, but *would* be aliased in the measured data, thereby bollixing any mechanism put into tor to adapt to periodic changes. This situation also argues in favor of a linearization like what I proposed above. OTOH, this aspect might be handled differently. For example, a separate, more frequent measurement of peak data rate could be made. In essence, the 15-minute totals reported in the old descriptors and now in the extra-info documents could be used that way, though they would be up to 18 hours out of date. They aren't quite the same as 10-second
Re: peculiar server "bandwidth" posted by server "mnl" and possible new type of attack
On Tue, Sep 09, 2008 at 05:15:15AM -0500, Scott Bennett wrote: > bandwidth 5242880 10485760 52239166 > ---> ~48.8 MB/s (!) Wow. Nice! :) And, as you say, unlikely to be true. > All of the above leads me to suspect two things. One is that there may > be some bug triggered only under exceedingly odd conditions that leads to > reporting of data rates grossly out of line with reality. Yep. Probably an integer underflow somewhere. > The second is that when a server claims such disproportionately high > capacity, the overall performance of the tor server network can be > compromised. There has been quite a bit of work on this one, at least. First, clients use the min of the bandwidthrate and the measured bandwidth, so in the above "5242880 10485760 52239166" example clients will load-balance based on 5242880. So this bug doesn't have any real effect, at least in this case. Also, clients cap the number they'll believe at 10MB, so an evil relay can only sucker people so much. There are research designs for doing community-measured bandwidth numbers, which would make it much harder for an evil relay to sucker people. See for example http://freehaven.net/anonbib/#snader08 Unfortunately, all of these designs have downsides currently, so they're not ready for deployment yet. > That brings us back to something I've already posted on OR-TALK, namely, > the apparent slowdown in tor traffic that has reduced the traffic through my > tor server by at least 30% and, judging from the reduced peaks shown for a lot > of the high-volume servers listed on the torstatus page, the tor network at > large. We're working on plans to start gathering more methodical data about how the network has run and is running, with the goal of being able to answer questions like this more usefully. Another example I'd love to investigate is the apparent feedback effect -- once a relay happens to have a high-volume flow through it, it tends to continue to advertise high bandwidth numbers for weeks after, whereas if it doesn't start with any high-volume flows, it never seems to attract them later. My guess is that the current slowdown is due to a combination of fewer file-sharing users (which, if true, will alas probably pass) and Tor's not-as-optimal-as-it-should-be load balancing algorithm. In particular, our load balancing algorithm produces high variance. Or said more normally, the speed you get is wildly unpredictable, sometimes really good and sometimes really awful -- and we need to cut down on the frequency of 'really awful'. So, stay tuned, but be patient. I'm hoping that we'll have that "metrics and measurements" plan ramped up sometime in early 2009 -- with people paid to focus on it, even. :) In the meantime, perhaps we can get some more intuition about this particular slowdown. For example, does it happen for 0.2.0.x relays and not 0.1.2.x relays? Thanks, --Roger
peculiar server "bandwidth" posted by server "mnl" and possible new type of attack
Over the last several days, a server nicknamed "mnl" has been posting
descriptor updates bearing highly suspect data rate information. I have
contacted the person at the contact-info address, so he is aware that the
observed data rate (misnamed "bandwidth") reported in mnl's server descriptor
is probably inaccurate by a factor of 10 or greater. His server has been
up for over a month, and IIRC, is usually a relay with a very high data rate
and therefore to be considered a significant benefit to the tor network's
overall performance.
However, mnl has lately been claiming to have had ten-second peak rates
on the order of eight to ten times the peak rates typically published by
blutmagie, the extraordinarily high-rate server that is nearly always a big
chunk of the tor network capacity's foundation. Here is the latest of mnl's
descriptors from my cached-descriptors file:
router mnl 159.149.71.27 9001 0 9030
platform Tor 0.2.0.30 (r15956) on Linux i686
opt protocols Link 1 2 Circuit 1
published 2008-09-08 14:41:50
opt fingerprint ABD3 8668 D3F4 76F5 0232 FEC0 B6DB 6550 EA43 EDD0
uptime 2781643
bandwidth 5242880 10485760 52239166
---> ~48.8 MB/s (!)
opt extra-info-digest 791494258CFA222ABCA7CD60D41CBFA108275E36
onion-key
-BEGIN RSA PUBLIC KEY-
MIGJAoGBAMZ7P0H3/DUwp/L1+eLx9CZlq8ZYh0vmcI4fA/5IzL4xld0/zOUGBXgI
QVv9qKaLOWB5ljn/LH/cyW2zW/mfJAURyZd00ffbsmA76QEJuJldzgOMQVrVZQwA
+QZCMh2+CgyvHhyyim9gFDsNHbkJQFbdX9hW3//sAMI0Oc4Lp+T7AgMBAAE=
-END RSA PUBLIC KEY-
signing-key
-BEGIN RSA PUBLIC KEY-
MIGJAoGBANo7EyYEVcDchLG30pHczPvjC9nB3pwn7CMo2vsa1uuXmvi5Kil1W6go
KhQDvbowfe2TpjMRMxCZ2o02+dmRUQrgwCVZeKqlIbAfbMUpxuB6wCl352nNK6uT
v6cgGNd5cd5aXZN1TVPkVlohi8h9cekDjrWhUu0lIz4GftOQviGxAgMBAAE=
-END RSA PUBLIC KEY-
contact [EMAIL PROTECTED]
reject *:*
router-signature
-BEGIN SIGNATURE-
YBaKbNNJXSVVuqGTlLRwAAdp1GHdRpGCQwiM5j6ws2Z2W13nbPHUk7PyKJ/3pMEl
EYeVtP1P9G+Hc0ItoC2CmdREDnz/R5I9M9uNG+bHdD1lea/oyQRD6WXMczDFwrvs
54XpWr/MRUXyN0mF5hq8AtHU3NcL6Do4kXXnO9/6m+8=
-END SIGNATURE-
Nearly 49 MB/s seems a bit of a stretch. The server's operator sent me
a note saying that the server is attached to the 1 GB/s campus backbone net,
but it is attached via a 100 Mb/s router, so the reported data rate is four
to five times the rate physically possible due to the router's limitation.
The server, according to its operator, is running on a 2.6 GHz P4, and its
descriptor says the machine is running LINUX. Based upon postings quite a
while back from blutmagie's operator and from a few other operators of very
high-data-rate servers, it seems to me that a 2.6 GHz P4 (Northwood?) running
LINUX would not be capable of handling a load eight to ten times that of
blutmagie, regardless of its network connection's capacity.
All of the above leads me to suspect two things. One is that there may
be some bug triggered only under exceedingly odd conditions that leads to
reporting of data rates grossly out of line with reality.
The second is that when a server claims such disproportionately high
capacity, the overall performance of the tor server network can be compromised.
This would happen due to the statistical method used for selecting nodes to
be ussed to construct a circuit, which is partially based upon reported, recent
(i.e., in the preceding 24 hours), peak data rates. In other words, a server
reporting wildly high data rates distorts the PDF, such that an inordinately
high fraction of circuit routes worldwide will include that server, regardless
of whether that server can actually keep up with that workload. If circuits
fail or fail to be built as a result of the overallocated errant server being
in those circuits' routes, the effect is merely to reduce to networkwide total
capacity of the tor network.
That brings us back to something I've already posted on OR-TALK, namely,
the apparent slowdown in tor traffic that has reduced the traffic through my
tor server by at least 30% and, judging from the reduced peaks shown for a lot
of the high-volume servers listed on the torstatus page, the tor network at
large. If this is actually what has been going on, then not only should the
bug be tracked down and killed ASAP, it serves as a call to rethink the method
of circuit route selection to find ways to prevent a reduction-in-throughput
attack that could be made by almost any creep by setting up a corrupted relay.
(mnl is not even an exit.)
(deep breath) I want to state right now that I do not in any way
whatsoever suspect mnl's operator of any nefarious activity. I believe that
he is at least as perplexed over his server's behavior as I am, especially
given other information he provided about events over the weekend. I do not
have a clue what sort of bug in tor could cause a server to begin reporting
exaggerated data rates, but the majority of bugs fixed by Roger, Nick, et al.
are ones that I would never have imagined either. Somethi
