Re: [tor-bugs] #29772 [Metrics/Website]: Plot nearly worst-case bandwidth when downloading from [public|onion] server

[Tor Bug Tracker & Wiki]

#29772: Plot nearly worst-case bandwidth when downloading from [public|onion]
server
-+--
 Reporter:  karsten  |  Owner:  metrics-team
 Type:  enhancement  | Status:  new
 Priority:  Medium   |  Milestone:
Component:  Metrics/Website  |Version:
 Severity:  Normal   | Resolution:
 Keywords:  scalability  |  Actual Points:
Parent ID:  #29507   | Points:
 Reviewer:   |Sponsor:
-+--
Changes (by karsten):

 * parent:   => #29507


Comment:

 Adding this new graph is part of the larger task to evaluate existing
 OnionPerf data regarding worst-case performance.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #29772 [Metrics/Website]: Plot nearly worst-case bandwidth when downloading from [public|onion] server

[Tor Bug Tracker & Wiki]

#29772: Plot nearly worst-case bandwidth when downloading from [public|onion]
server
-+--
 Reporter:  karsten  |  Owner:  metrics-team
 Type:  enhancement  | Status:  new
 Priority:  Medium   |  Milestone:
Component:  Metrics/Website  |Version:
 Severity:  Normal   | Resolution:
 Keywords:  scalability  |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by karsten):

 Replying to [comment:8 irl]:
 > I think for now, moving forward with what we have is OK.

 Great!

 > We might want to add a note to the description when we add this graph
 that this is an experimental one, subject to change without notice, or
 even disappearing without notice. I don't know how many plots we might be
 asked to make but we shouldn't be introducing them all with the same
 change procedures as we have for our other graphs just yet.

 Good point. We don't have good procedures in place for graphs. We do have
 a policy of announcing changes to the per-graph CSV files two weeks in
 advance. Having something like that, plus a good plan for archiving old
 graphs, would be good. Will think more about this!

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #29772 [Metrics/Website]: Plot nearly worst-case bandwidth when downloading from [public|onion] server

[Tor Bug Tracker & Wiki]

#29772: Plot nearly worst-case bandwidth when downloading from [public|onion]
server
-+--
 Reporter:  karsten  |  Owner:  metrics-team
 Type:  enhancement  | Status:  new
 Priority:  Medium   |  Milestone:
Component:  Metrics/Website  |Version:
 Severity:  Normal   | Resolution:
 Keywords:  scalability  |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--
Changes (by irl):

 * status:  needs_review => new


Comment:

 I think for now, moving forward with what we have is OK.

 We might want to add a note to the description when we add this graph that
 this is an experimental one, subject to change without notice, or even
 disappearing without notice. I don't know how many plots we might be asked
 to make but we shouldn't be introducing them all with the same change
 procedures as we have for our other graphs just yet.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #29772 [Metrics/Website]: Plot nearly worst-case bandwidth when downloading from [public|onion] server

[Tor Bug Tracker & Wiki]

#29772: Plot nearly worst-case bandwidth when downloading from [public|onion]
server
-+--
 Reporter:  karsten  |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics/Website  |Version:
 Severity:  Normal   | Resolution:
 Keywords:  scalability  |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--
Changes (by karsten):

 * status:  needs_revision => needs_review


Comment:

 I should start this comment by saying that I'm not a statistician. In case
 of doubt what I'm saying below, please go re-read this first sentence! :)

 I agree with you that the bandwidth plot works better than the latency
 plot. We're excluding very few bandwidth numbers as outliers as compared
 to the number of latency numbers that we're throwing out.

 However, I don't think that a 4-day moving average would fix this. As you
 can see in the boxplots I posted here last week, medians and quartiles are
 relatively stable over the days, and those values are what we're using to
 figure out if another value is excluded as outlier. After all, we have
 around 144 latency values per day and public/onion service. So, even if we
 considered 4 days (or even more) at a time, our threshold for excluding
 values as outliers would not change much. Of course, implementing such a
 moving average wouldn't be trivial to do, with all the missing data that
 we have to handle.

 I think the issue is that the way we're excluding outliers is based on the
 assumption that our data is normally distributed. This works okay for
 bandwidth, which is obviously not 100% correct, because there's no
 negative bandwidth, but which is apparently close enough. It doesn't work
 very well for latencies, because there's some heavy-tailed distribution at
 work that we don't know, and not all the values we're excluding are really
 outliers.

 Another reason could be that we're looking at the smallest bandwidth
 values, which are at the ''head'' of the distribution, and at the largest
 latency values, which are the heavy ''tail''.

 However, my suggestion is to ignore all this and make the plots as you
 suggested earlier and as I plotted them last week. Two reasons:

  1. Boxplots are understood by many people, and if we say that we're
 plotting the five values from boxplots, people will understand what we're
 doing.

  2. We need a baseline, even if it's not 100% correct in a
 mathematical/statistical sense. If our way to exclude outliers is flawed,
 it will be flawed for past measurements as well as for future
 measurements, in the exact same way.

 Regarding your rocket analogy: it's certainly not just distance between
 relays that we're seeing here. We're also seeing overfull queues keeping
 received cells waiting for crypto and forwarding to the next relay. But
 this is fine, we want to know how long it takes to send something over the
 circuit and get back a response.

 So, my suggestion would be to move forward with what we have. What do you
 think?

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #29772 [Metrics/Website]: Plot nearly worst-case bandwidth when downloading from [public|onion] server

[Tor Bug Tracker & Wiki]

#29772: Plot nearly worst-case bandwidth when downloading from [public|onion]
server
-+
 Reporter:  karsten  |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_revision
 Priority:  Medium   |  Milestone:
Component:  Metrics/Website  |Version:
 Severity:  Normal   | Resolution:
 Keywords:  scalability  |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+
Changes (by irl):

 * status:  needs_review => needs_revision


Comment:

 For the bandwidths, I think that plotting the minimum of the minor
 outliers is OK, we're not excluding many measurements there and we can see
 what the nearly-worst-case is.

 I think for latency, we have to accept that Tor as it currently operates
 is going to have wildly varying latency depending on the path you choose.
 There is currently no way of selecting a "low-latency" path and as we
 increase relay diversity we're going to see these latencies go up. In a
 way, higher latency may indicate greater network diversity.

 For bandwidth, choosing one high bandwidth server compared to another
 isn't going to affect the measurement result. When the network stays the
 same, we are likely to choose a similar set of relays throughout the day
 (or at least, the same consensus weight distribution). For latency, there
 is no consideration so we could be picking relays all over the place, or
 could be picking them all close together.

 The absolute worst cases though, I think we are not hitting that often.
 Plotting the latency per-day may be the wrong approach because we may not
 have enough data points to accurately portray the true latency users
 experience. Perhaps we need a 4-day moving average, at which point some of
 those major outliers are becoming minor outliers and we can plot the
 maximum minor outliers.

 As a data point, when we see a 2000ms latency, that is long enough to get
 a packet through optic fiber to the moon from the earth (not including the
 time to run the fiber, probably with special-purpose rockets). There might
 be some old routing/switching equipment near to relays that is causing
 impact there because this can't just be distance between relays.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #29772 [Metrics/Website]: Plot nearly worst-case bandwidth when downloading from [public|onion] server

[Tor Bug Tracker & Wiki]

#29772: Plot nearly worst-case bandwidth when downloading from [public|onion]
server
-+--
 Reporter:  karsten  |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics/Website  |Version:
 Severity:  Normal   | Resolution:
 Keywords:  scalability  |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--
Changes (by karsten):

 * status:  needs_revision => needs_review


Comment:

 Great idea! I made some new graphs and annotated them by hand. Please find
 them attached. And please take another look. Thanks!

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #29772 [Metrics/Website]: Plot nearly worst-case bandwidth when downloading from [public|onion] server

[Tor Bug Tracker & Wiki]

#29772: Plot nearly worst-case bandwidth when downloading from [public|onion]
server
-+
 Reporter:  karsten  |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_revision
 Priority:  Medium   |  Milestone:
Component:  Metrics/Website  |Version:
 Severity:  Normal   | Resolution:
 Keywords:  scalability  |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+
Changes (by karsten):

 * Attachment "onionperf-boxplots-annotated-2019-04-17.pdf" added.


--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #29772 [Metrics/Website]: Plot nearly worst-case bandwidth when downloading from [public|onion] server

[Tor Bug Tracker & Wiki]

#29772: Plot nearly worst-case bandwidth when downloading from [public|onion]
server
-+
 Reporter:  karsten  |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_revision
 Priority:  Medium   |  Milestone:
Component:  Metrics/Website  |Version:
 Severity:  Normal   | Resolution:
 Keywords:  scalability  |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+
Changes (by irl):

 * status:  needs_review => needs_revision


Comment:

 I'm not sure that 1st percentile is the right way to do this. Can we
 instead exclude minor/major outliers, those that are slower than 1st
 quartile minus 1.5/3x interquartile range, and then taken the minimum? How
 does this change the way the plot looks?

 I don't think we should make a public graph on Tor Metrics for it, but can
 you also do a box plot for a month of measurements so I can understand
 just how variable the results are? I don't think I've done that before.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #29772 [Metrics/Website]: Plot nearly worst-case bandwidth when downloading from [public|onion] server

[Tor Bug Tracker & Wiki]

#29772: Plot nearly worst-case bandwidth when downloading from [public|onion]
server
-+--
 Reporter:  karsten  |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics/Website  |Version:
 Severity:  Normal   | Resolution:
 Keywords:  scalability  |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--
Changes (by gaba):

 * keywords:   => scalability


--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #29772 [Metrics/Website]: Plot nearly worst-case bandwidth when downloading from [public|onion] server

[Tor Bug Tracker & Wiki]

#29772: Plot nearly worst-case bandwidth when downloading from [public|onion]
server
-+--
 Reporter:  karsten  |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics/Website  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--
Changes (by karsten):

 * status:  new => needs_review


--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #29772 [Metrics/Website]: Plot nearly worst-case bandwidth when downloading from [public|onion] server

[Tor Bug Tracker & Wiki]

#29772: Plot nearly worst-case bandwidth when downloading from [public|onion]
server
-+--
 Reporter:  karsten  |  Owner:  metrics-team
 Type:  enhancement  | Status:  new
 Priority:  Medium   |  Milestone:
Component:  Metrics/Website  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by karsten):

 [[Image(onionperf-bandwidth-public.png, 600px)]]

 [[Image(onionperf-bandwidth-onion.png, 600px)]]

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #29772 [Metrics/Website]: Plot nearly worst-case bandwidth when downloading from [public|onion] server

[Tor Bug Tracker & Wiki]

#29772: Plot nearly worst-case bandwidth when downloading from [public|onion]
server
-+--
 Reporter:  karsten  |  Owner:  metrics-team
 Type:  enhancement  | Status:  new
 Priority:  Medium   |  Milestone:
Component:  Metrics/Website  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--
Changes (by karsten):

 * Attachment "onionperf-bandwidth-onion.png" added.


--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #29772 [Metrics/Website]: Plot nearly worst-case bandwidth when downloading from [public|onion] server

[Tor Bug Tracker & Wiki]

#29772: Plot nearly worst-case bandwidth when downloading from [public|onion]
server
-+--
 Reporter:  karsten  |  Owner:  metrics-team
 Type:  enhancement  | Status:  new
 Priority:  Medium   |  Milestone:
Component:  Metrics/Website  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--
Changes (by karsten):

 * Attachment "onionperf-bandwidth-public.png" added.


--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs