Re: [tor-bugs] #25924 [Metrics/Statistics]: Improve execution time of onion service statistics module

[Tor Bug Tracker & Wiki]

#25924: Improve execution time of onion service statistics module
+-
 Reporter:  karsten |  Owner:  karsten
 Type:  enhancement | Status:  closed
 Priority:  Medium  |  Milestone:
Component:  Metrics/Statistics  |Version:
 Severity:  Normal  | Resolution:  fixed
 Keywords:  |  Actual Points:
Parent ID:  | Points:
 Reviewer:  |Sponsor:
+-

Comment (by karsten):

 It's deployed now. Last execution times of daily update runs in hours are:
 13.6, 14.4, 17.8, 15.8, '''5.4'''. Those 5.4 hours are from the first
 execution running this branch. Yay!

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #25924 [Metrics/Statistics]: Improve execution time of onion service statistics module

[Tor Bug Tracker & Wiki]

#25924: Improve execution time of onion service statistics module
+-
 Reporter:  karsten |  Owner:  karsten
 Type:  enhancement | Status:  closed
 Priority:  Medium  |  Milestone:
Component:  Metrics/Statistics  |Version:
 Severity:  Normal  | Resolution:  fixed
 Keywords:  |  Actual Points:
Parent ID:  | Points:
 Reviewer:  |Sponsor:
+-
Changes (by karsten):

 * status:  merge_ready => closed
 * resolution:   => fixed


Comment:

 Merged to master. Will make a backup and deploy either tonight or tomorrow
 morning before the next daily update starts. Thanks for checking! Closing.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #25924 [Metrics/Statistics]: Improve execution time of onion service statistics module

[Tor Bug Tracker & Wiki]

#25924: Improve execution time of onion service statistics module
+-
 Reporter:  karsten |  Owner:  karsten
 Type:  enhancement | Status:  merge_ready
 Priority:  Medium  |  Milestone:
Component:  Metrics/Statistics  |Version:
 Severity:  Normal  | Resolution:
 Keywords:  |  Actual Points:
Parent ID:  | Points:
 Reviewer:  |Sponsor:
+-
Changes (by irl):

 * status:  needs_review => merge_ready


Comment:

 We got lucky here with this low hanging fruit. Next time will be harder.

 LGTM.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #25924 [Metrics/Statistics]: Improve execution time of onion service statistics module

[Tor Bug Tracker & Wiki]

#25924: Improve execution time of onion service statistics module
+--
 Reporter:  karsten |  Owner:  karsten
 Type:  enhancement | Status:  needs_review
 Priority:  Medium  |  Milestone:
Component:  Metrics/Statistics  |Version:
 Severity:  Normal  | Resolution:
 Keywords:  |  Actual Points:
Parent ID:  | Points:
 Reviewer:  |Sponsor:
+--
Changes (by karsten):

 * status:  accepted => needs_review


Comment:

 Alright, I think I know what's going on. The reason why runtime keeps
 getting higher is that we're re-processing an ever increasing number of
 onion service statistics reported by relays for which we computed a
 network fraction of zero for having observed these statistics. Of course,
 it does make sense not to extrapolate these numbers, because we cannot
 divide by zero. But we should remember this fact and not attempt to
 extrapolate statistics in all future executions. That's also what we do
 with statistics reported by relays with non-zero network fractions: we
 notice that we did extrapolate these before and skip them. This is the
 reason why runtime keeps getting higher and higher over time. The fix is
 simply to write extrapolated numbers for all reported statistics.

 Please review [https://gitweb.torproject.org/user/karsten/metrics-
 web.git/commit/?h=task-25924=ed4f75df1a246afdc86b2c38f95a31e6a37a5d33
 commit ed4f75d in my task-25924 branch]. I did test this branch locally,
 but before deploying it we should make fresh a backup of
 `work/modules/hidserv/`, just in case.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #25924 [Metrics/Statistics]: Improve execution time of onion service statistics module

[Tor Bug Tracker & Wiki]

#25924: Improve execution time of onion service statistics module
+--
 Reporter:  karsten |  Owner:  karsten
 Type:  enhancement | Status:  accepted
 Priority:  Medium  |  Milestone:
Component:  Metrics/Statistics  |Version:
 Severity:  Normal  | Resolution:
 Keywords:  |  Actual Points:
Parent ID:  | Points:
 Reviewer:  |Sponsor:
+--
Changes (by karsten):

 * status:  new => accepted
 * owner:  metrics-team => karsten


Comment:

 I just noticed that execution time of this module went up to 7.5 hours on
 average during the last week. This is too much. Looking into this now.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #25924 [Metrics/Statistics]: Improve execution time of onion service statistics module

[Tor Bug Tracker & Wiki]

#25924: Improve execution time of onion service statistics module
+--
 Reporter:  karsten |  Owner:  metrics-team
 Type:  enhancement | Status:  new
 Priority:  Medium  |  Milestone:
Component:  Metrics/Statistics  |Version:
 Severity:  Normal  | Resolution:
 Keywords:  |  Actual Points:
Parent ID:  | Points:
 Reviewer:  |Sponsor:
+--
Changes (by gaba):

 * priority:  High => Medium


--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #25924 [Metrics/Statistics]: Improve execution time of onion service statistics module

[Tor Bug Tracker & Wiki]

#25924: Improve execution time of onion service statistics module
+--
 Reporter:  karsten |  Owner:  metrics-team
 Type:  enhancement | Status:  new
 Priority:  High|  Milestone:
Component:  Metrics/Statistics  |Version:
 Severity:  Normal  | Resolution:
 Keywords:  |  Actual Points:
Parent ID:  | Points:
 Reviewer:  |Sponsor:
+--
Changes (by karsten):

 * priority:  Low => High


Comment:

 Maybe this has just become more urgent.

 https://lists.torproject.org/pipermail/metrics-
 team/2018-December/000971.html

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #25924 [Metrics/Statistics]: Improve execution time of onion service statistics module

[Tor Bug Tracker & Wiki]

#25924: Improve execution time of onion service statistics module
+--
 Reporter:  karsten |  Owner:  metrics-team
 Type:  enhancement | Status:  new
 Priority:  Low |  Milestone:
Component:  Metrics/Statistics  |Version:
 Severity:  Normal  | Resolution:
 Keywords:  |  Actual Points:
Parent ID:  | Points:
 Reviewer:  |Sponsor:
+--

Comment (by karsten):

 [[Image(metrics-modules-2016-01-07.png, 600px)]]
 [[Image(metrics-modules-2018-04-26.png, 600px)]]

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #25924 [Metrics/Statistics]: Improve execution time of onion service statistics module

[Tor Bug Tracker & Wiki]

#25924: Improve execution time of onion service statistics module
+--
 Reporter:  karsten |  Owner:  metrics-team
 Type:  enhancement | Status:  new
 Priority:  Low |  Milestone:
Component:  Metrics/Statistics  |Version:
 Severity:  Normal  | Resolution:
 Keywords:  |  Actual Points:
Parent ID:  | Points:
 Reviewer:  |Sponsor:
+--
Changes (by karsten):

 * Attachment "metrics-modules-2018-04-26.png" added.


--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #25924 [Metrics/Statistics]: Improve execution time of onion service statistics module

[Tor Bug Tracker & Wiki]

#25924: Improve execution time of onion service statistics module
+--
 Reporter:  karsten |  Owner:  metrics-team
 Type:  enhancement | Status:  new
 Priority:  Low |  Milestone:
Component:  Metrics/Statistics  |Version:
 Severity:  Normal  | Resolution:
 Keywords:  |  Actual Points:
Parent ID:  | Points:
 Reviewer:  |Sponsor:
+--
Changes (by karsten):

 * Attachment "metrics-modules-2016-01-07.png" added.


--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

[tor-bugs] #25924 [Metrics/Statistics]: Improve execution time of onion service statistics module

[Tor Bug Tracker & Wiki]

#25924: Improve execution time of onion service statistics module
+--
 Reporter:  karsten |  Owner:  metrics-team
 Type:  enhancement | Status:  new
 Priority:  Low |  Milestone:
Component:  Metrics/Statistics  |Version:
 Severity:  Normal  |   Keywords:
Actual Points:  |  Parent ID:
   Points:  |   Reviewer:
  Sponsor:  |
+--
 Two years ago, in February 2016, we
 [https://lists.torproject.org/pipermail/tor-dev/2016-February/010408.html
 noticed] that some of our back-end modules had really long execution
 times. Back then we made improvements to two of these modules. I'm going
 to post a graph of execution times in the comments.

 Looks like we'll have to do it again, and this time work even harder on
 improving execution times. The onion service statistics module again takes
 roughly 2 hours to complete, with a clear trend. I assume that we'll reach
 execution times of 3 hours by the end of 2018 and 5 to 6 hours by the end
 of 2019.

 I'm going to post another graph with recent execution times. (Note that we
 don't have logs for just the onion service statistics module, but only for
 that module plus the previous module together; but that other module is
 still relatively fast, contributing with just a few minutes execution time
 to the graph.)

 I don't have concrete suggestions for improvements. And it's still early
 enough to try to get this work funded before we dive into it. That's why
 I'm setting priority to low. But it's clear that we'll have to do
 something here in 2019.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs