Re: [tor-relays] BWAUTH weightings too volatile. . .twitchy
At 02:43 6/6/2015 -0400, starlight.201...@binnacle.cx wrote: I'm back to complain further about erratic bandwidth authority behavior, previously. . . MYSTERY SOLVED!!! Of course one should always RTFS (read the fine specification) when trying to understand all things Tor. First, scratch my previous time- averaging suggestion as the current implementation incorporates a sophisticated time averaging feedback algorithm borrowed from industrial control systems. Turns out the problem of sudden, sharp, schizophrenic consensus jumps is a boundary artifact. My relay is hovering right around the 12% threshold between the fastest and second fastest groups of relays (out of four groups). These two groups operate as separate statistical domains, and the algorithm is complex enough that it would be shocking if the measurement of any one relay came out the same when moved from one group to the other group. Looks to me like the relay is bouncing back-and-forth between the 0-12% band and the 12-35% band. This happens independently for each of the four BWauths. The 0-12% band assigned weight for the relay is anywhere from 130 to 200% of the weight calculated in the 12-35% band. Group-hopping explains why the individual BWauth weights tend to jump dramatically and suddenly. The median-weight consensus selection adds an additional element of randomness. Assigned-weight moves smoothly within either group, but is discontinuous when shifting from one group to the next. Is interesting that 'moria1' consistently reports much higher weights than the three other BWauths. Might be that one of the algorithm parameters is set to a different value for this authority. ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] BWAUTH weightings too volatile. . .twitchy
Mon Jun 8 05:25:55 UTC 2015 UBSAN seems expensive and doesn't seem it would run other than test, but I didn't work on it long and am not 100% certain. Was trying ASAN extra stack checking at the time, which may have been the culprit. As I said in my previous email, if you're running releases before 0.2.6.6 (2015), Tor won't run under UBSAN due to issues that have been fixed in subsequent releases. Had a need to run ASAN+UBSAN and it works perfectly fine for me with version 0.2.4.27. Required a minor patch to convert a single common variable 'incoming_queue' to extern. Performance hit was not as bad as I thought from the earlier try, but bad enough that I limited the instrumented modules to only what's necessary. ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] BWAUTH weightings too volatile. . .twitchy
A straightforward improvement to BWauth measurement crossed my mind. Seems likely part of the volatile, bipolar measurement issue is overfast feedback of weighting increases and the increased traffic that results. For example, a BWauth measures 8 MByte/sec of bandwidth day one and increases the assigned score to 20k. The relay's weight attracts a pile-on of new traffic and now by day three the relay measures 2 Mbyte/sec of available bandwidth due to the presence a huge amount of traffic, and the BWauth crashed the assigned value back to perhaps 10k. Thus the weight of the relay swinges wildly between two extremes. Solution is for BWauths to time- average several days of measurements, probably with a decaying weight for older samples. Ten days of samples with the oldest four assigned declining weights comes to mind as a place to start, though of course the number of days and weighting parameters should be easily adjusted. This will result in gradual shifting of BW weights assigned to relays with an equilibrium outcome rather than wild swings. Will also compensate for random sample timing where a BWauth may test a relay at a busy time on one day and a light load time the next day. Probably a downside threshold should exist and trigger the resetting of the accumulated data points to address relays that fail or deteriorate rapidly. ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] BWAUTH weightings too volatile. . .twitchy
Date: Sat, 06 Jun 2015 14:37:01 -0400 From: starlight.201...@binnacle.cx At 04:12 6/7/2015 +1000, teor wrote: Please let me know how you go - the 0.2.6.x series should also be relatively ASAN and UBSAN clean, as Tor has been tested with them since late 2014. I've run 0.2.4.x and 0.2.5.x with ASAN live in production with no problems when the relay had less bandwidth. Performance hit is something like 30% extra CPU. Also had it on libssl.so and libevent.so, but was too expensive to run on libcrypto.so. UBSAN seems expense and doesn't seem it would run other than test, but I didn't work on it long and am not 100% certain. Was trying ASAN extra stack checking at the time, which may have been the culprit. As I said in my previous email, if you're running releases before 0.2.6.6 (2015), Tor won't run under UBSAN due to issues that have been fixed in subsequent releases. If you're running on architectures other than x86_64, Tor won't run under UBSAN due to a known issue in the donna C code. teor teor2345 at gmail dot com pgp 0xABFED1AC https://gist.github.com/teor2345/d033b8ce0a99adbc89c5 teor at blah dot im OTR D5BE4EC2 255D7585 F3874930 DB130265 7C9EBBC7 signature.asc Description: Message signed with OpenPGP using GPGMail ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
[tor-relays] BWAUTH weightings too volatile. . .twitchy
I'm back to complain further about erratic bandwidth authority behavior, previously [tor-relays] BWauth kookiness https://lists.torproject.org/pipermail/tor-relays/2015-May/007039.html Allowing that BWauths are in a bit of flux, with tor26 replaced by maatuska and moria1 dropping the GuardFraction calculation, the bandwidth calculations evidence wildly erratic swings. Specifically my relay, which is perfectly stable, reliable, fast (9.375 Mbyte/sec) has been assigned a jaggedly random series of consensus weights. https://atlas.torproject.org/#details/4F0DB7E687FC7C0AE55C8F243DA8B0EB27FBF1F2 earlier, fairly sane *gabelmoo Bandwidth=7701 Measured=9960 tor26 Bandwidth=7701 Measured=9340 moria1Bandwidth=7701 Measured=18000 GuardFraction=66 longclaw Bandwidth=7701 Measured=12800 later, bit high, nutty gablemoo Bandwidth=9375 Measured=17100 moria1 Bandwidth=9375 Measured=77900 GuardFraction=75 *longclaw Bandwidth=9375 Measured=23000 now, sane but undervalued gablemoo Bandwidth=8925 Measured=14900 maatuska Bandwidth=8925 Measured=17200 moria1 Bandwidth=8925 Measured=5330 *longclaw Bandwidth=8925 Measured=7440 moria1 here is downright schizophrenic but other BWauths regularly double and halve the bandwidth value they assign. Graph shows it a more vividly. What the graphs and numbers do not show is that the effective difference between the consensus values of ~7000 and ~23000 is staggering. At the low end of this range the relay shows roughly 2500 active circuits and a average load factor of about 20% of actual bandwidth, while an assignment of 23000 results in almost 8000 circuits and a load factor of more like 50% (both per Blutemagie.de). My point is that BWauths should not be arbitrarily flipping stable, well run relays from the top to the bottom of this steeply sloped sweet-spot of the weighting curve. Perhaps the sweet-spot range has moved over the last couple of years as the price of bandwidth has dropped and faster connections become more prevalent, and this has been overlooked in the algorithm. Regardless, it seems BWauth measurement should be more nuanced in this particular range, such that relays are not slammed constantly between rather popular and a bit boring irrespective of their actual available capacity. One reason this vexes is that I would like to see how well the relay runs with Address Sanitizer active. ASAN provides obvious benefits w/r/t security, but entails a performance trade-off. With the BWauths throwing darts, eyes closed, when choosing weighting, it's impossible to gauge the performance impact of various adjustments. In the bigger picture view, erratic BWauth weighting can't be adding clarity to the performance, capacity and utilization situation. ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] BWAUTH weightings too volatile. . .twitchy
Date: Sat, 06 Jun 2015 02:43:09 -0400 From: starlight.201...@binnacle.cx … One reason this vexes is that I would like to see how well the relay runs with Address Sanitizer active. ASAN provides obvious benefits w/r/t security, but entails a performance trade-off. With the BWauths throwing darts, eyes closed, when choosing weighting, it's impossible to gauge the performance impact of various adjustments. … Good timing! Tor 0.2.7.1-alpha on x86_64 is currently Address Sanitizer and Undefined Behavior Sanitizer clean. I've just submitted a branch with instructions for building, running, and testing Tor with ASAN and UBSAN. https://trac.torproject.org/projects/tor/ticket/15817#comment:8 The following known issues exist: Two of the tests deliberately invoke undefined / illegal behavior, the instructions provide a blacklist file and an environmental variable to exempt them from ASAN/UBSAN. Architectures without (x86?) 64-bit assembler use donna C code that left-shifts 1 bits into and past the sign bit of signed integers. Until https://trac.torproject.org/projects/tor/ticket/13538 is resolved, this particular file can be exempted from UBSAN. Please let me know how you go - the 0.2.6.x series should also be relatively ASAN and UBSAN clean, as Tor has been tested with them since late 2014. teor teor2345 at gmail dot com pgp 0xABFED1AC https://gist.github.com/teor2345/d033b8ce0a99adbc89c5 teor at blah dot im OTR D5BE4EC2 255D7585 F3874930 DB130265 7C9EBBC7 signature.asc Description: Message signed with OpenPGP using GPGMail ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] BWAUTH weightings too volatile. . .twitchy
At 04:12 6/7/2015 +1000, teor wrote: Please let me know how you go - the 0.2.6.x series should also be relatively ASAN and UBSAN clean, as Tor has been tested with them since late 2014. I've run 0.2.4.x and 0.2.5.x with ASAN live in production with no problems when the relay had less bandwidth. Performance hit is something like 30% extra CPU. Also had it on libssl.so and libevent.so, but was too expensive to run on libcrypto.so. UBSAN seems expense and doesn't seem it would run other than test, but I didn't work on it long and am not 100% certain. Was trying ASAN extra stack checking at the time, which may have been the culprit. Did crash out with a good analysis report the day Heartbleed was announced due to someone scanning all the nodes for the vuln. After a major bandwidth increase, have held off waiting for the bandwidth weight to stabilize before trying it. However the with the insanity of the BWauths, I haven't done much with it. Ran it awhile back and the consensus took a modest hit, but the numbers are so unreliable I couldn't tell if it was ASAN overhead or plain randomness. The idea is not to use ASAN for a one off test, but to run it 100% in production so that an exploit showing up out-of-the blue might be caught out. Later this year Intel is releasing new CPUs with hardware support for new UBSAN checks. That's something I want to try. ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
