[tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  defect   | Status:  new
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   |   Keywords:
Actual Points:   |  Parent ID:
   Points:   |   Reviewer:
  Sponsor:   |
-+--
 Hello,
 I have finished deploying a test of OnionPerf on OTF Cloud.
 It can be accessed at: https://199.119.112.144/ (certificate is self
 signed atm)

 Now that we can look at the generated files, we can start thinking and
 planning about how we want to consume them.
 I think this is a good moment to decide if we want to include an option to
 generate a different format within OnionPerf, or we want to write a script
 to parse the files in a format we can use.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  new
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--
Changes (by iwakeh):

 * type:  defect => enhancement


Comment:

 Adding a link from our mail conversation to the first analysis:
 https://github.com/hiromipaw/onionperf-notebook

 Changed ticket type to 'enhancement'.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  new
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by hiro):

 Hi,

 So I have updated the wiki for onionperf:
 
https://trac.torproject.org/projects/tor/wiki/org/operations/Infrastructure/onionperf
 Now we have onionperf-one which is in DC, United States, and onionperf-eu
 which is in Amsterdam.
 I am finishing installing also the tpo instance as I will need some
 libraries installed there first. I will let you know once this is
 completed

 My notebook can be accessed on:
 https://github.com/hiromipaw/onionperf-notebook
 And the results can be displayed from https://github.com/hiromipaw
 /onionperf-notebook/blob/master/transfer_times.html which is basically the
 quick-n-dirty test I did to make sure measurements made some sense.

 I could try to make onionperf measurements available for collector, or
 maybe you prefer me to do something else instead. I haven't honestly
 touched Java in a very long time ;)

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  new
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by karsten):

 hiro, I'm having a difficult time reading your results.  Can you put them
 on GitHub Pages or export some images and attach them here?  And can you
 add explanations what your results show and why you consider them close
 enough to Torperf data that we can switch over?  That would be awesome!

 Regarding the CollecTor integration, I started hacking on that yesterday
 evening.  No need for you to touch Java just for this. :)

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  new
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by hiro):

 Hi Karsten, sure I will generate some annotated images.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  new
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by hiro):

 I have started this board: https://github.com/hiromipaw/onionperf-
 notebook/blob/master/board.md
 I will be collecting graphs for all the measurements provided by
 onionperf. At the moment I only have the 50KiB download time but I will be
 adding more measurements before our meeting tomorrow.

 My idea up to this point has been that if I do not find something that it
 is completely unreasonable when comparing w/ Torperf I assume the
 measurements (and my analysis) can be considered as accurate.

 Talk more tomorrow at the meeting.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  new
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by hiro):

 Hi,

 I have discovered I was making an error in plotting the time differences.
 In fact I was just plotting the microseconds difference :(
 Now I have fixed this and it looks more accurate:
 https://github.com/hiromipaw/onionperf-notebook/blob/master/board.md

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  new
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--
Changes (by teor):

 * cc: teor (added)


--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  new
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by karsten):

 Looks good to me, thanks!  I'd say let's go ahead and include these
 measurements on Metrics, and if we later find out that there are
 measurement issues, we can still take broken measurements out.  But if
 nobody looks at the data, nobody will spot any issues.

 So, seems like the missing piece is the CollecTor integration.  I'll
 continue working on that.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  new
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by karsten):

 hiro, can you rename the sources to `op-$cc` with `$cc` being the country
 code where the instance is running in as discussed at the last team
 meeting?

 And can you check why https://37.218.247.40/ is still empty?

 And what's the URL for the third instance, or did you not set that up yet?

 Thanks!

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  new
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by karsten):

 Oh, and can you look into getting (Let's Encrypt) certificates, maybe
 using `op-$cc.$something` as domain names?  Otherwise we'll first have to
 look into ways to tell CollecTor which (self-signed) certificate to expect
 on each OnionPerf instance, which might be possible but which is probably
 not trivial.  Thanks!

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  new
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by hiro):

 Hi Karsten,
 I have renamed the op-us and op-nd ones already. Also renamed the files
 too. I will have left to rename the hostname fields in the already
 generated log. Doing that at the moment.
 Regarding the logs I am checking why these are generated but not copied
 over. Maybe there is a mismatch in directories. I am checking that.
 Regarding op-tpo I am working to have the webserver running.
 Also installing let's encrypt right now :)

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  new
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by hiro):

 Also do you think we could have these running on op-$cc.torproject.org ?

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  new
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by hiro):

 Update: I have also replaced the REMOTEHOSTNAME in the tpf files.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  new
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by hiro):

 Let's encrypt is running. New domain names are:

 - op-us.onionperf.torproject.net

 - op-nl.onionperf.torproject.net

 Working on tpo setup.

 For OFT Cloud we have a new location available: Hong Kong. Are we
 interested in setting this up?

 Regarding the tpf files not showing up in the directory. Logs are
 generated but files are not "packed" into tpf at midnight. I am trying to
 investigate why this is happening.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  new
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by karsten):

 Thanks for configuring domain names and Let's Encrypt!  Downloading using
 Java works just fine now!

 However, I just looked at one of the .tpf files and found a couple of
 onion addresses:

 {{{
 $ grep START op-us-1048576-2017-03-07.tpf | sed 's/.*ENDPOINTREMOTE=\([^
 ]*\) .*/\1/' | sort | uniq -c
   19 199-119-112-144.i95.net:199.119.112.144:8080
   17 5rk7srcqqtesjxpo.onion:0.0.0.0:8080
1 ggxgcnswjxwjcqoe.onion:0.0.0.0:8080
7 ysfuzafp5bis.onion:0.0.0.0:8080
 }}}

 Any idea what's up with those?  Could it be that this instance is
 configured to make more requests than it should?  Can you maybe paste the
 configuration here?

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--
Changes (by karsten):

 * status:  new => needs_review


Comment:

 Please review [https://gitweb.torproject.org/karsten/metrics-
 db.git/log/?h=task-21272 my task-21272 branch] with a commit that
 downloads .tpf files from OnionPerf instances.  Still quite rough around
 the edges, but passed an initial test by successfully downloading .tpf
 files from https://op-us.onionperf.torproject.net/.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by iwakeh):

 From reading the git diffs, I have a few questions and suggestions:

 Does this code reflect the transition period of processing both
 torperf and onionperf files?  Is that the reason for introducing
 the empty property with the meaning of 'nothing to download'?

 Surely tests will be added for the new functionality of `Configuration`'s
 `getStringArray` and `getStringArrayArray` methods?
 I assume current code that uses these methods relies on receiving a non-
 empty
 array or double array.  It needs to be verified that we don't run
 into trouble in the existing code using these methods.

 All storing of files should be done by the persist-mechanism, i.e.,
 a `o.t.c.persist.TpfPersistence` class needs to be introduced (this would
 shorten `TorperfDownloader` quite a bit and prevent another
 re-implementation of this functionality).
 And, this is a prerequisite for also syncing these files with
 CollecTor's sync-mechanism, which was left out because we knew
 a Torperf replacement is coming.

 It might be useful to put the Configuration's getUrlArray method
 to work.  The old Torperf code wasn't modernized, b/c we knew
 it will be replaced.  Now, it might be better to have
 OnionPerfHosts as Url array property.  This way the url-property
 checking in the downloader class is not necessary anymore as
 `Configuration`
 already provides it.
 Why is the host-nick-name needed and could not be replaced by the hostname
 itself?  (For torperf it is used for further configuration.)  Or, is this
 one of the rough edges and note yet available?
 If so, maybe have a different configuration approach instead of the
 hybrid-String-Url array?
 It would be great to make the configuration simpler to read
 and edit and keep allmost all property-checking code in the
 `Configuration` class.  For example, an url-array and a string-array-array
 property, the latter for the configuration similar to torperf.
 The only check left for the TorperfDownloader would be to match
 length of these two array- properties, but there might be other
 good approaches.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by karsten):

 Replying to [comment:17 iwakeh]:
 > From reading the git diffs, I have a few questions and suggestions:

 Wow, quick review there.  Thanks!

 > Does this code reflect the transition period of processing both
 > torperf and onionperf files?  Is that the reason for introducing
 > the empty property with the meaning of 'nothing to download'?

 Huh, good point.  What I had in mind that we could use this code to 0) run
 the same configuration as we do now, 1) start collecting OnionPerf files,
 and soon after 2) stop collecting Torperf files.  But we could achieve the
 same by deploying this code when we do 1) and deploy another patch for 2).
 I can undo that change.

 > Surely tests will be added for the new functionality of
 `Configuration`'s
 > `getStringArray` and `getStringArrayArray` methods?
 > I assume current code that uses these methods relies on receiving a non-
 empty
 > array or double array.  It needs to be verified that we don't run
 > into trouble in the existing code using these methods.

 Right.  Let me undo this part of the change.

 > All storing of files should be done by the persist-mechanism, i.e.,
 > a `o.t.c.persist.TpfPersistence` class needs to be introduced (this
 would
 > shorten `TorperfDownloader` quite a bit and prevent another
 > re-implementation of this functionality).
 > And, this is a prerequisite for also syncing these files with
 > CollecTor's sync-mechanism, which was left out because we knew
 > a Torperf replacement is coming.

 Hmm, is this something you might be able to do?  I'd really want us to
 deploy this code before Amsterdam, so that we can discuss next steps
 there.  Otherwise, how about we defer this part until after Amsterdam?

 > It might be useful to put the Configuration's getUrlArray method
 > to work.  The old Torperf code wasn't modernized, b/c we knew
 > it will be replaced.  Now, it might be better to have
 > OnionPerfHosts as Url array property.  This way the url-property
 > checking in the downloader class is not necessary anymore as
 `Configuration`
 > already provides it.
 > Why is the host-nick-name needed and could not be replaced by the
 hostname
 > itself?  (For torperf it is used for further configuration.)  Or, is
 this
 > one of the rough edges and note yet available?
 > If so, maybe have a different configuration approach instead of the
 hybrid-String-Url array?
 > It would be great to make the configuration simpler to read
 > and edit and keep allmost all property-checking code in the
 > `Configuration` class.  For example, an url-array and a string-array-
 array
 > property, the latter for the configuration similar to torperf.
 > The only check left for the TorperfDownloader would be to match
 > length of these two array- properties, but there might be other
 > good approaches.

 Well, my idea was that we should use the configured source name to make
 sure that an OnionPerf host doesn't send us measurement results for other
 sources, either accidentally or on purpose.  I'm not sure how we could use
 the hostname there if we want to stick to the
 [https://collector.torproject.org/recent/torperf/ current naming schema of
 .tpf files].  But it could be that I overlooked something.

 However, if we want to keep this, what we could do is add a new method
 `Configuration.getStringUrlMap()` that returns a (sorted) map of Strings
 to URLs.  Again, would you want to write such a thing?

 Thanks again!

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by iwakeh):

 Only two short questions about naming scheme and host url:

 Currently the beginning of the host url is reflected in the file name.
 Why can't we just use the beginning of the host url, i.e., the op-$cc part
 for checking the filenames?

 Is the goal now to just download all available measurements regarding file
 size?

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by karsten):

 Replying to [comment:19 iwakeh]:
 > Only two short questions about naming scheme and host url:
 >
 > Currently the beginning of the host url is reflected in the file name.
 > Why can't we just use the beginning of the host url, i.e., the op-$cc
 part
 > for checking the filenames?

 Sure, we could do that.  And we could extend that if we need to.  Works
 for me.

 > Is the goal now to just download all available measurements regarding
 file size?

 Hmm, I guess the main reason for specifying file sizes explicitly in
 `TorperfFilesLines` was that we didn't parse a directory listing, so we
 had to know which file sizes exist.  But here we do know which file sizes
 are measured, so the only reason for specifying those would be to exclude
 sizes we don't recognize.  Do we have to do that?  I'd say as long as the
 file sizes stated in the .tpf file name matches the `FILESIZE` value
 contained in that file, we'll take what we get.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by iwakeh):

 Replying to [comment:20 karsten]:
 > Replying to [comment:19 iwakeh]:
 > > Only two short questions about naming scheme and host url:
 > >
 > > Currently the beginning of the host url is reflected in the file name.
 > > Why can't we just use the beginning of the host url, i.e., the op-$cc
 part
 > > for checking the filenames?
 >
 > Sure, we could do that.  And we could extend that if we need to.  Works
 for me.

 So, all paved for using `getUrlArray`, isn't it?

 >
 > > Is the goal now to just download all available measurements regarding
 file size?
 >
 > Hmm, I guess the main reason for specifying file sizes explicitly in
 `TorperfFilesLines` was that we didn't parse a directory listing, so we
 had to know which file sizes exist.  But here we do know which file sizes
 are measured, so the only reason for specifying those would be to exclude
 sizes we don't recognize.  Do we have to do that?  I'd say as long as the
 file sizes stated in the .tpf file name matches the `FILESIZE` value
 contained in that file, we'll take what we get.

 I think it's fine, to reduce the configuration.
 The verification of filename vs. FILESIZE value should be done by
 descriptor, if we feel the need to introduce it.

 (more about your other comment:18 tomorrow)

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by hiro):

 Hi Karsten,
 I just wanted to point out that there is no configuration option in onion
 perf. The measurements are started by the onionperf process. Something we
 could do is investigating how to provide a small config. I could dig
 through the code and see what Rob thinks. What do you think?

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by karsten):

 iwakeh, please find my updated task-21272 branch with some tweaks as
 discussed above.

 hiro, I see.  I'll take a closer look at the results and get back to you
 on this ticket.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by karsten):

 hiro, I just looked a bit more at the data and found a few differences
 that we can work with.  For example, OnionPerf randomly picks a file size
 (with different weights, though) every five minutes whereas Torperf had
 three separate schedules for each file size, which is okay.  Also,
 OnionPerf alternates between direct download and download via onion
 address, which is different from Torperf, but which is okay, too.

 However, there's one difference that we should fix: the direct downloads
 go to port 8080, not to port 80.  The issue is that the set of exits
 permitting port 8080 may be different from the set permitting port 80, so
 this might impact measurement results.  As far as I can see there are two
 ways to do this: either run OnionPerf on port 80 (which may have security
 implications), or configure a firewall rule to forward port 80 to 8080.  I
 assume we'll want to do the latter.  In that case we'll have to use
 OnionPerf's options `--tgen-listen-port` and `--tgen-connect-port` to tell
 it where to listen (8080) and where to connect (80).

 Would you want to try making that change?

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by iwakeh):

 Replying to [comment:23 karsten] and what's left from [comment:18]:
 > iwakeh, please find my updated task-21272 branch with some tweaks as
 discussed above.

 Thanks, for these changes!

 Couldn't `downloadFromOnionPerfHost` do some of the filename checking
 before
 calling `downloadAndParseOnionPerfTpfFile`?

 About the older code:
 Could we just also make the change for #20514 now?
 In addition, there are quite a few places where try-with resources
 or some of `Files`' methods would prevent unclosed readers/writers.
 Still this older code needs even more work, sigh.  Maybe, after Amsterdam?

 And, regarding the descriptor parsing don't the checks inside
 [https://gitweb.torproject.org/karsten/metrics-
 
db.git/tree/src/main/java/org/torproject/collector/torperf/TorperfDownloader.java?h=task-21272&id=3318eb8ca769392cca1a6ddc3344c43eba62da91#n797
 this loop] belong into descriptor/metrics-lib itself?
 (that might be a new ticket, though)

 I can take a look at the Persistence topic at some point.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by karsten):

 Replying to [comment:25 iwakeh]:
 > Replying to [comment:23 karsten] and what's left from [comment:18]:
 > > iwakeh, please find my updated task-21272 branch with some tweaks as
 discussed above.
 >
 > Thanks, for these changes!
 >
 > Couldn't `downloadFromOnionPerfHost` do some of the filename checking
 before
 > calling `downloadAndParseOnionPerfTpfFile`?

 Well, that wouldn't change functionality but would be a simple
 refactoring, right?  What's the goal there?  Make methods more testable or
 easier to read or something else?  In any case, would you want to suggest
 new methods, and I'll move around code?  Or do you want to work on a
 patch?

 > About the older code:
 > Could we just also make the change for #20514 now?
 > In addition, there are quite a few places where try-with resources
 > or some of `Files`' methods would prevent unclosed readers/writers.
 > Still this older code needs even more work, sigh.  Maybe, after
 Amsterdam?

 No need to spend time on this.  Let's just remove the Torperf code in a
 few weeks when we're certain that the OnionPerfs can take over.

 > And, regarding the descriptor parsing don't the checks inside
 [https://gitweb.torproject.org/karsten/metrics-
 
db.git/tree/src/main/java/org/torproject/collector/torperf/TorperfDownloader.java?h=task-21272&id=3318eb8ca769392cca1a6ddc3344c43eba62da91#n797
 this loop] belong into descriptor/metrics-lib itself?
 > (that might be a new ticket, though)

 Well, if we moved that code to metrics-lib, users wouldn't be able to read
 Torperf results from anything else than the originally named .tpf file.
 We usually avoid dependencies on file names if we can.  This case is a bit
 different, because we're archiving .tpf files, and we should be certain
 that they contain what they say.  I'd say that's specific to the CollecTor
 case though and cannot be generalized in metrics-lib.

 Note that we could have picked a different approach by parsing descriptors
 from files and appending them to new files with file names taken from
 descriptor contents.  The result might be the exact same output, or it
 could be a file with fewer descriptors or descriptors in a different
 order, etc.  But I felt it's easier to verify .tpf file contents and
 either take them or leave them.

 > I can take a look at the Persistence topic at some point.

 Sounds good!  The last paragraph above might be relevant here.  Hope this
 works with the persistence classes.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by iwakeh):

 Replying to [comment:26 karsten]:
 > Replying to [comment:25 iwakeh]:
 > > Replying to [comment:23 karsten] and what's left from [comment:18]:
 > > > iwakeh, please find my updated task-21272 branch with some tweaks as
 discussed above.
 > >
 > > Thanks, for these changes!
 > >
 > > Couldn't `downloadFromOnionPerfHost` do some of the filename checking
 before
 > > calling `downloadAndParseOnionPerfTpfFile`?
 >
 > Well, that wouldn't change functionality but would be a simple
 refactoring, right?  What's the goal there?  Make methods more testable or
 easier to read or something else?  In any case, would you want to suggest
 new methods, and I'll move around code?  Or do you want to work on a
 patch?

 No, I intended to avoid the [https://gitweb.torproject.org/karsten
 /metrics-
 
db.git/tree/src/main/java/org/torproject/collector/torperf/TorperfDownloader.java?h=task-21272&id=3318eb8ca769392cca1a6ddc3344c43eba62da91#n750
 superfluous parsing] of the URL for each file and avoid download when the
 filename doesn't make sense.
 No refactoring.

 >
 > > About the older code:
 > > Could we just also make the change for #20514 now?
 > > In addition, there are quite a few places where try-with resources
 > > or some of `Files`' methods would prevent unclosed readers/writers.
 > > Still this older code needs even more work, sigh.  Maybe, after
 Amsterdam?
 >
 > No need to spend time on this.  Let's just remove the Torperf code in a
 few weeks when we're certain that the OnionPerfs can take over.

 That's fine, too.

 >
 > > And, regarding the descriptor parsing don't the checks inside
 [https://gitweb.torproject.org/karsten/metrics-
 
db.git/tree/src/main/java/org/torproject/collector/torperf/TorperfDownloader.java?h=task-21272&id=3318eb8ca769392cca1a6ddc3344c43eba62da91#n797
 this loop] belong into descriptor/metrics-lib itself?
 > > (that might be a new ticket, though)
 >
 > Well, if we moved that code to metrics-lib, users wouldn't be able to
 read Torperf results from anything else than the originally named .tpf
 file.  We usually avoid dependencies on file names if we can.  This case
 is a bit different, because we're archiving .tpf files, and we should be
 certain that they contain what they say.  I'd say that's specific to the
 CollecTor case though and cannot be generalized in metrics-lib.

 There could be a boolean parameter for strict checking?
 But, I didn't mean to hijack this ticket.  I can just write that down on a
 list on my desk ;-)

 >
 > Note that we could have picked a different approach by parsing
 descriptors from files and appending them to new files with file names
 taken from descriptor contents.  The result might be the exact same
 output, or it could be a file with fewer descriptors or descriptors in a
 different order, etc.  But I felt it's easier to verify .tpf file contents
 and either take them or leave them.

 Hmm, ok, good to know.

 >
 > > I can take a look at the Persistence topic at some point.
 >
 > Sounds good!  The last paragraph above might be relevant here.  Hope
 this works with the persistence classes.

 Yes, I noticed this.  We'll see.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by karsten):

 Replying to [comment:27 iwakeh]:
 > Replying to [comment:26 karsten]:
 > > Replying to [comment:25 iwakeh]:
 > > > Couldn't `downloadFromOnionPerfHost` do some of the filename
 checking before
 > > > calling `downloadAndParseOnionPerfTpfFile`?
 > >
 > > Well, that wouldn't change functionality but would be a simple
 refactoring, right?  What's the goal there?  Make methods more testable or
 easier to read or something else?  In any case, would you want to suggest
 new methods, and I'll move around code?  Or do you want to work on a
 patch?
 >
 > No, I intended to avoid the [https://gitweb.torproject.org/karsten
 /metrics-
 
db.git/tree/src/main/java/org/torproject/collector/torperf/TorperfDownloader.java?h=task-21272&id=3318eb8ca769392cca1a6ddc3344c43eba62da91#n750
 superfluous parsing] of the URL for each file and avoid download when the
 filename doesn't make sense.
 > No refactoring.

 Hmm, I'm a bit lost what you mean here.  We only download if the filename
 makes sense.  But this is discussion has become very theoretical.  Want to
 provide a patch? :D

 > > Well, if we moved that code to metrics-lib, users wouldn't be able to
 read Torperf results from anything else than the originally named .tpf
 file.  We usually avoid dependencies on file names if we can.  This case
 is a bit different, because we're archiving .tpf files, and we should be
 certain that they contain what they say.  I'd say that's specific to the
 CollecTor case though and cannot be generalized in metrics-lib.
 >
 > There could be a boolean parameter for strict checking?
 > But, I didn't mean to hijack this ticket.  I can just write that down on
 a list on my desk ;-)

 Sounds like a new ticket.  But I'm not yet sold on the idea.  If it's
 something that only CollecTor needs as provider of .tpf files and none of
 the consumers, then we shouldn't add make it part of metrics-lib.  Of
 course, if there's a plausible use case for consumers, happy to
 reconsider.  But, new (metrics-lib) ticket?

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by iwakeh):

 Sure, I'll add a patch.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by hiro):

 Replying to [comment:24 karsten]:
 > hiro, I just looked a bit more at the data and found a few differences
 that we can work with.  For example, OnionPerf randomly picks a file size
 (with different weights, though) every five minutes whereas Torperf had
 three separate schedules for each file size, which is okay.  Also,
 OnionPerf alternates between direct download and download via onion
 address, which is different from Torperf, but which is okay, too.
 >
 > However, there's one difference that we should fix: the direct downloads
 go to port 8080, not to port 80.  The issue is that the set of exits
 permitting port 8080 may be different from the set permitting port 80, so
 this might impact measurement results.  As far as I can see there are two
 ways to do this: either run OnionPerf on port 80 (which may have security
 implications), or configure a firewall rule to forward port 80 to 8080.  I
 assume we'll want to do the latter.  In that case we'll have to use
 OnionPerf's options `--tgen-listen-port` and `--tgen-connect-port` to tell
 it where to listen (8080) and where to connect (80).
 >
 > Would you want to try making that change?

 Hi Karsten sure I can do that.
 In the meantime I have fixed the logrotation issue on op-nl. Hopefully our
 tpo instance should also be ready. I will let you know if everything is
 working correctly on https://onionperf.torproject.org

 More soon.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by hiro):

 Hi Karsten,
 I have setup a proxy forward for 8080 to 80 for op-nl. I'll check if
 everything is ok and eventually will set it up also for op-us and op.tpo.
 More soon.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by karsten):

 Thanks, hiro!  Curious to see how that goes.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by hiro):

 Proxy forward up for op-nl and op-us

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by hiro):

 Also measurements are up on tpo: https://onionperf.torproject.org/
 Still some finishing touches, but looks good.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by iwakeh):

 Replying to [comment:29 iwakeh]:
 > Sure, I'll add a patch.

 This
 
[https://gitweb.torproject.org/user/iwakeh/collector.git/commit/?h=task-21272-2&id=883d0d8a6634c82e2097a404ca6049cfd2c3de19
 change] is indeed shorter than its description.

 (Just noticed, I accidentally, slipped in a filename length readability
 tweak, too.)

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by karsten):

 iwakeh, patch looks good, pushed to [https://gitweb.torproject.org/karsten
 /metrics-db.git/log/?h=task-21272 my task-21272 branch].  Thanks!  Do you
 want to change anything else, or can I squash and merge?

 hiro: now looking at new data...

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by karsten):

 hiro, I looked at the data and have two questions:

  - It seems that the .tpf files produced by OnionPerf don't show whether
 connections were made to port 80 (and forwarded by firewall rules) or were
 made to port 8080.  But we should only use data from measurements to port
 80, or we'll compare two different measurements.  Can you delete older
 measurements from before you changed that?

  - The OnionPerf source name (`papillare`) and DNS name (`onionperf`) of
 `https://onionperf.torproject.org/papillare-5242880-2017-03-12.tpf` don't
 match, which is something we implicitly require on CollecTor (which you
 couldn't know).  Ideally, we'd stick with the naming scheme of the other
 instances and rename both to `op-de`.  That's probably easiest to
 understand for users who will see these source names on Tor Metrics and
 would rightfully ask what a papillare is.  Can you change the name of both
 OnionPerf instance and in the DNS entry?

 Thanks!

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by hiro):

 Hi Karsten,
 Sure I will delete all the old measurements starting from the day before
 yesterday. Regarding papillare, the host and dns names where the bits that
 I was missing. Will have that changed. :)
 -Silvia

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by hiro):

 Update:
 I have changed the onionperf source name on papillare as onionperf. I was
 chatting w the tpa team and they would prefer not having to change the dns
 to op-de. If we can stick with onionperf as service name and subdomain we
 are good, otherwise I'll bump them again.
 Hopefully the ports are correct now.
 Will finish deleting old files when we will check the new results
 tomorrow, to compare.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by iwakeh):

 Replying to [comment:36 karsten]:
 > iwakeh, patch looks good, pushed to
 [https://gitweb.torproject.org/karsten/metrics-db.git/log/?h=task-21272 my
 task-21272 branch].  Thanks!  Do you want to change anything else, or can
 I squash and merge?
 >

 So far, this seems ready for merge.  Tests pass and the download works.

 The persistence topic from comment:18 and before, and also the later sync-
 addition has a separate ticket #21759.

 Let's move any further things related to CollecTor java code to #21760,
 which already lists the open final move to onionperf and the tests etc.

 The test run showed empty files
 https://op-us.onionperf.torproject.net/op-us-5242880-2017-03-15.tpf (size
 0)
 on the us source.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by karsten):

 Thanks for looking again, and thanks for opening those new tickets.  I
 just squashed and merged to master, together with a change log entry.

 I noticed the empty file, too.  It's yet unclear whether it will go away
 tonight.  If not, we should handle that case less loudly.

 I'll keep this ticket open until deployment is complete, with data being
 collected by CollecTor.  But let's wait at least until tomorrow to see
 what the three op-xx instances produce.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by karsten):

 I took another look at the data and compared it to Torperf data.  Please
 find the attached graph.  I have two remaining questions before adding the
 new data to CollecTor:

  - It looks like op-nl is quite a bit faster than the other OnionPerf and
 Torperf instances.  One measurement only took 80 milliseconds from making
 the request until receiving the last byte.  Is this realistic?  Or does
 OnionPerf take any shortcuts that the other instances don't take?

 {{{
 @type torperf 1.0
 BUILDTIMES=0.079237061,0.089141693,0.10895096 CIRC_ID=5284
 CONNECT=1489499550.79 DATACOMPLETE=1489499550.87 DATAPERC0=1489499550.86
 DATAPERC10=1489499550.86 DATAPERC100=1489499550.87
 DATAPERC20=1489499550.86 DATAPERC30=1489499550.86 DATAPERC40=1489499550.86
 DATAPERC50=1489499550.87 DATAPERC60=1489499550.87 DATAPERC70=1489499550.87
 DATAPERC80=1489499550.87 DATAPERC90=1489499550.87
 DATAREQUEST=1489499550.82 DATARESPONSE=1489499550.84 DIDTIMEOUT=0
 ENDPOINTLOCAL=localhost:127.0.0.1:60802
 ENDPOINTPROXY=localhost:127.0.0.1:29128
 ENDPOINTREMOTE=37.218.247.40:37.218.247.40:8080 FILESIZE=51200
 HOSTNAMELOCAL=op-nl HOSTNAMEREMOTE=op-nl LAUNCH=1489499310.22
 NEGOTIATE=1489499550.79
 
PATH=$A1EF24A8E85E440F215A5C296790636C62F43A2A,$150F2AD7D0FE7715ECAA642D4CD4BCBF95E8BD0D,$5CECC5C30ACC4B3DE462792323967087CC53D947
 QUANTILE=0.8 READBYTES=51269 REQUEST=1489499550.80 RESPONSE=1489499550.82
 SOCKET=1489499550.79 SOURCE=op-nl START=1489499550.79 TIMEOUT=1500
 USED_AT=1489499550.87 USED_BY=10787 WRITEBYTES=54
 }}}

  - The IP address of op-hk gets resolved to the Netherlands, though the
 measurements look very different from the op-nl host.  Can we confirm that
 the host is really located in Hong Kong and that it's just the IP-to-
 country resolution that is behind?

 If we have answers to these questions, I'd say let's make the data
 available.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--
Changes (by robgjansen):

 * cc: rob.g.jansen@… (added)


Comment:

 Replying to [comment:42 karsten]:
 > I took another look at the data and compared it to Torperf data.  Please
 find the attached graph.  I have two remaining questions before adding the
 new data to CollecTor:
 >
 >  - It looks like op-nl is quite a bit faster than the other OnionPerf
 and Torperf instances.  One measurement only took 80 milliseconds from
 making the request until receiving the last byte.  Is this realistic?  Or
 does OnionPerf take any shortcuts that the other instances don't take?
 >

 Looking at the relays chosen for that path, is looks like the full path
 is:
 
Netherlands(client)-->France(guard)-->Germany(middle)-->France(exit)-->Netherlands(server)

 This means the latency on each of those links is about 10 milliseconds.
 That seems feasible to me, given how close those countries are and that
 the machines are probably well connected to the backbone. The server can
 send all ~100 cells at once and it will likely travel through Tor in one
 piece, so I don't think that would cause any or much delay.

 Since most of the Tor relay bandwidth is in Europe, more of the circuits
 of a client in Europe would not leave the continent compared to clients in
 other regions. I'm not surprised if an OnionPerf client in Europe would
 trend faster than other countries on average. I wonder if either but not
 both of the two TorPerf nodes are in Europe, as we could potentially use
 the data they collect to test my hypothesis.

 Also, there is more information about that circuit in particular and the
 download process in general in the json.xz files that are also dumped to
 the data directory. I added some notes to the elements there quite some
 time ago to help us understand what is available:

 https://github.com/robgjansen/onionperf/blob/master/README_JSON.md

 Finally, I have been running my OnionPerf instance since April 2016, and
 would like to contribute to Tor metrics. Is that possible? I think it
 would be great if you could import the data that I have been collecting if
 it's valid (someone should double check that it's valid first, and I'm
 happy to do so). If there is something about my setup that you would
 prefer that I change before accepting data from my instance, please let me
 know. Also, if you want all instances to be run by TPI, let me know that
 too so I can stop paying for the VPS.

 http://onionperf.robgjansen.com:8081/

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by karsten):

 Quick and only partial answer: I think it would be great to add your data
 to CollecTor and Tor Metrics.  Here's one thing that we'd need to change
 though:

 We currently require that the subdomain of the OnionPerf source
 ("onionperf" in your case) must match the OnionPerf source name
 ("phantomtrain" in your case).  The reason is that we want to avoid a case
 where one instance produces data with the source name of another instance,
 which would mess up things quickly.  You could probably work around this
 by adding yet another subdomain in front of your current domain.  But
 let's first discuss which name you should pick.

 Another requirement is that we pick source names in a way that Tor Metrics
 users are not confused by seeing them listed on
 https://metrics.torproject.org/torperf.html.  It was not very smart to
 pick "torperf" as one name there, because all three are Torperf instances,
 and the names "moria" and "siv" don't have much meaning, either.  That's
 why we picked "op-us" for "OnionPerf instance located in the U.S." etc.
 for the current instances.  Obviously, that scheme does not scale either,
 with your instance being hosted in the U.S., too.  But "onionperf" is just
 too generic.  "phantomtrain" might work, because it's just a name.  What
 do you think?  Still time to do it right this time. :)

 Thanks!  (Will reply to the other parts later today.)

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by hiro):

 Replying to [comment:42 karsten]:
 > I took another look at the data and compared it to Torperf data.  Please
 find the attached graph.  I have two remaining questions before adding the
 new data to CollecTor:
 >
 >  - It looks like op-nl is quite a bit faster than the other OnionPerf
 and Torperf instances.  One measurement only took 80 milliseconds from
 making the request until receiving the last byte.  Is this realistic?  Or
 does OnionPerf take any shortcuts that the other instances don't take?
 >
 > {{{
 > @type torperf 1.0
 > BUILDTIMES=0.079237061,0.089141693,0.10895096 CIRC_ID=5284
 CONNECT=1489499550.79 DATACOMPLETE=1489499550.87 DATAPERC0=1489499550.86
 DATAPERC10=1489499550.86 DATAPERC100=1489499550.87
 DATAPERC20=1489499550.86 DATAPERC30=1489499550.86 DATAPERC40=1489499550.86
 DATAPERC50=1489499550.87 DATAPERC60=1489499550.87 DATAPERC70=1489499550.87
 DATAPERC80=1489499550.87 DATAPERC90=1489499550.87
 DATAREQUEST=1489499550.82 DATARESPONSE=1489499550.84 DIDTIMEOUT=0
 ENDPOINTLOCAL=localhost:127.0.0.1:60802
 ENDPOINTPROXY=localhost:127.0.0.1:29128
 ENDPOINTREMOTE=37.218.247.40:37.218.247.40:8080 FILESIZE=51200
 HOSTNAMELOCAL=op-nl HOSTNAMEREMOTE=op-nl LAUNCH=1489499310.22
 NEGOTIATE=1489499550.79
 
PATH=$A1EF24A8E85E440F215A5C296790636C62F43A2A,$150F2AD7D0FE7715ECAA642D4CD4BCBF95E8BD0D,$5CECC5C30ACC4B3DE462792323967087CC53D947
 QUANTILE=0.8 READBYTES=51269 REQUEST=1489499550.80 RESPONSE=1489499550.82
 SOCKET=1489499550.79 SOURCE=op-nl START=1489499550.79 TIMEOUT=1500
 USED_AT=1489499550.87 USED_BY=10787 WRITEBYTES=54
 > }}}
 >
 >  - The IP address of op-hk gets resolved to the Netherlands, though the
 measurements look very different from the op-nl host.  Can we confirm that
 the host is really located in Hong Kong and that it's just the IP-to-
 country resolution that is behind?
 >
 > If we have answers to these questions, I'd say let's make the data
 available.

 Hi Karsten, I can have a chat w/ the people at Greenhost and find out
 about your questions. I was wondering that too actually.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by karsten):

 robgjansen, thanks for the response on those very fast measurements.
 Sounds plausible to me.  I'll add add op-us and op-nl to CollecTor
 tomorrow afternoon.  And I'll op-hk as soon as we're confident where it's
 located and your instance as soon as we have resolved the naming part.
 Thanks!

 hiro, it would be great if you could chat with Greenhost people about the
 location.  Just keep it running, and once we know better where it is we
 can add its data to CollecTor.  Thanks!

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by robgjansen):

 For naming, it seems like the current scheme works fine for now. Mine
 would be 'op-ca' since it appears that mine is located in Canada:
 https://geoiptool.com/en/?ip=167.114.171.3

 But yeah, you may run into collisions if you plan to run more instances.
 Some other schemes I can think of are:
 - IP address: 'op-ca-167.114.171.3'
 - AS number: 'op-ca-as1234'
 - Simple increment: 'op-ca1', 'op-ca2', etc.

 I like tying it to an IP or AS number, because it is more precise. The
 name is longer though, and if you move the instance, you would need to
 rotate the name (which may be a good thing anyway since it moved network
 locations).

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by robgjansen):

 I would have no problem updating the data hosted by my instance before you
 import it.

 I looked through my data a bit. I started out running my instance in a
 different network location (216.17.99.183) before moving it to a new VPS
 at 167.114.171.3. It appears that I was running the TGen server listening
 on port 216.17.99.183: before, and now it is listening on
 167.114.171.3:8080. So, connections from Tor will request these ports.

 I think you are trying to get your instances to run a TGen server at port
 80, right? Is that a hard requirement? If so, my existing data is useless.
 I wrote a quick script (attached in case it's useful) to help us
 understand how much exit bandwidth by consensus weight allows exit to
 various ports:

 {{{
 167.114.171.3:80 is allowed by 83.1011269711 percent of exit bandwidth
 167.114.171.3:443 is allowed by 85.7649427029 percent of exit bandwidth
 167.114.171.3: is allowed by 78.7485340684 percent of exit bandwidth
 167.114.171.3:8080 is allowed by 77.0591360016 percent of exit bandwidth
 167.114.171.3:8443 is allowed by 76.3254015495 percent of exit bandwidth
 216.17.99.183:80 is allowed by 83.1011269711 percent of exit bandwidth
 216.17.99.183:443 is allowed by 85.7649427029 percent of exit bandwidth
 216.17.99.183: is allowed by 78.7485340684 percent of exit bandwidth
 216.17.99.183:8080 is allowed by 77.0591360016 percent of exit bandwidth
 216.17.99.183:8443 is allowed by 76.3254015495 percent of exit bandwidth
 }}}

 I think that the difference between port  or 8080 and 80 is minor.

 Let me know how to proceed. If you want to use my existing data, perhaps I
 should use a different name for each of the data sets that were gathered
 in different locations? It looks like 216.17.99.183 is in California, so
 that would be another 'op-us-XXX' name.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by karsten):

 robgjansen, thanks for the great suggestions above!  I think we can make
 this work.  Here are my suggestions:

  - ''Source names'': Maybe we should just pick a naming scheme that works
 for the moment without expecting that it will scale in the future.  Your
 example of an instance moving from the U.S. to Canada is a great example.
 Sure, you could split the data into two sets and pick different names.
 But if we open up contributions to other community members, which I think
 we should, we'll see similar changes happening, and we can't expect other
 operators to get the naming right.

My suggestion for the source name of your instance would be to just
 stick with 'phantomtrain'.  The only change that you'd have to make though
 is make the data (also) available under
 http://phantomtrain.robgjansen.com:8081/.  The reason is that we're
 currently parsing the first part of the domain name and comparing that to
 the beginning of file names and source names contained in the data.  If
 this is not an easy change for you, we can look into making CollecTor a
 little more flexible.  But maybe it's just a simple configuration change
 for you.

  - ''Local IP address'': Even if we pick source names that don't indicate
 locations, we could resolve instance IP addresses to country codes and AS
 numbers when displaying results on Tor Metrics.  But therefore we'll have
 to find out the public IP address and include it in the data.  I'm yet
 unsure how we'd do that.  I mean, "endpoint_remote" contains a host name
 and IP address, but only for direct downloads, but this information is
 also relevant for onion services.

Hmm, I don't really have a good suggestion for this.  Do you have an
 idea?

  - ''Ports 80, , and 8080'': I agree that the difference is minimal,
 and we probably don't even have to mention this on Tor Metrics.  But still
 I'd like to preserve this information somewhere in the data.  Because who
 knows if somebody wants to run an OnionPerf instance with a rather unusual
 port.

I wonder if we can include both `--tgen-listen-port` and `--tgen-
 connect-port` in the output?  Maybe we'll have to add a new field
 "endpoint_connect" with remote host name, remote address, and connect
 port?  Or maybe there's a better way to include this information, possibly
 even related to the previous change where we include the local IP address?

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by karsten):

 Quick update: the two OnionPerf instances op-nl and op-us are now
 available on [https://collector.torproject.org/recent/torperf/ CollecTor]
 and [https://metrics.torproject.org/torperf.html Tor Metrics].

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--
Changes (by robgjansen):

 * cc: rob.g.jansen@… (removed)
 * cc: robgjansen (added)


Comment:

 I made the easy config update, so my data is now also available here:
 ​http://phantomtrain.robgjansen.com:8081/

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by robgjansen):

 For IPs and ports, here is the related data for a direct download:
 {{{
 HOSTNAMELOCAL=onionperf
 HOSTNAMEREMOTE=onionperf
 ENDPOINTLOCAL=localhost:127.0.0.1:55998
 ENDPOINTPROXY=localhost:127.0.0.1:45640
 ENDPOINTREMOTE=onionperf.robgjansen.com:167.114.171.3:8080
 }}}

 And here is what I see for a .onion download:
 {{{
 HOSTNAMELOCAL=onionperf
 HOSTNAMEREMOTE=onionperf
 ENDPOINTLOCAL=localhost:127.0.0.1:45664
 ENDPOINTPROXY=localhost:127.0.0.1:45640
 ENDPOINTREMOTE=ih7hhuuppsy5wysu.onion:0.0.0.0:8080
 }}}

 Notice that the server listen port that was used for this download, 8080,
 is available in both cases. This will always be the same place that the
 client connects, otherwise the download won't happen.
   If my OnionPerf instance runs a server on `--tgen-listen-port=8080` and
 I instruct the client to connect to someone else's OnionPerf server at
 `--tgen-connect-port=443`, then 443 is the port that will show up in the
 download data. So I don't think we need to worry about the `--tgen-*-port`
 options.

 For IPs and hostnames, we have:
 -
 
[https://github.com/shadow/shadow/blob/46368f34f4ee8f602be226c4c16a8a6d82cd9a70/src/plugin
 /shadow-plugin-tgen/shd-tgen-transfer.c#L964 HOSTNAMELOCAL and
 HOSTNAMEREMOTE contain the result of `gethostname`] on the client and the
 server end, respectively (in onionperf, the client and server run on the
 same machine).
 - The name, IP, and port for each of the 3 ENDPOINT data items runs the
 same code:
   -
 
[https://github.com/shadow/shadow/blob/46368f34f4ee8f602be226c4c16a8a6d82cd9a70/src/plugin
 /shadow-plugin-tgen/shd-tgen-peer.c#L79 the hostname contains the result
 of `getnameinfo`],
   -
 
[https://github.com/shadow/shadow/blob/46368f34f4ee8f602be226c4c16a8a6d82cd9a70/src/plugin
 /shadow-plugin-tgen/shd-tgen-peer.c#L53 the IP contains the result of
 `getaddrinfo`].
   -
 
[https://github.com/shadow/shadow/blob/46368f34f4ee8f602be226c4c16a8a6d82cd9a70/src/plugin
 /shadow-plugin-tgen/shd-tgen-peer.c#L245 the ports are in host order].

 I'm not sure that we should change any of this...

 This is tricky, since OnionPerf could be run by a client that sits behind
 a firewall and only contributes .onion downloads, or that connects to
 someone else's OnionPerf server. These clients probably don't have FQDNs.
 They of course will have some kind of public IP address, but that won't be
 discoverable by the server end if the downloads are done over Tor. And the
 IP address assigned to it's local interface may not be the public-facing
 IP address.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by hiro):

 Hi,
 I have updated measurements that were failing on https://op-
 us.onionperf.torproject.net and sent Rob a pull request that should fix
 the issue: https://github.com/robgjansen/onionperf/pull/27
 Also I got in contact with the people from Greenhost so that we can find
 out if they route the traffic for that hong kong instance in a "funny
 way".
 More soon.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by hiro):

 Also, answer from Greenhost regarding our hk onionperf instance:


 Hi Silvia,

 The IP's are from us, so registered on name of Greenhost, so whois will
 say they are dutch, however, they are just routed/announced directly in
 HK.

 (actually, using Whois/GeoIP would not give a clear determination where
 the IP's actually are hosted, triangulating whould be a better approach)

 Just use mtr/traceroute to determine the path it takes.

 Regards,

 Mart

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by robgjansen):

 Replying to [comment:53 hiro]:
 > I have updated measurements that were failing on https://op-
 us.onionperf.torproject.net and sent Rob a pull request that should fix
 the issue: https://github.com/robgjansen/onionperf/pull/27

 Merged. Thanks!

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by karsten):

 hiro, I just made op-hk measurements available on CollecTor and Tor
 Metrics.  Thanks!

 robgjansen, thanks for making your data available under the new URL.  I
 can download the files just fine, but I ran into problems with some files
 containing measurements from other dates than what the file name
 indicates.  I sent you details via email.

 Regarding the local IP address I see how this might be problematic.  We
 should figure out something there though, because measurement results
 highly depend on the location, and we need to include that somehow.

 Regarding ports, are you certain that the `--tgen-connect-port` will be
 included and not the `--tgen-listen-port`?  I believe hiro changed all
 op-* instances to download from port 80, but where would I find that port
 80 in the [https://collector.torproject.org/recent/torperf/ .tpf files]?

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by robgjansen):

 Replying to [comment:56 karsten]:
 > Regarding ports, are you certain that the `--tgen-connect-port` will be
 included and not the `--tgen-listen-port`?  I believe hiro changed all
 op-* instances to download from port 80, but where would I find that port
 80 in the [https://collector.torproject.org/recent/torperf/ .tpf files]?

 It looks like your recent torperf files are not including all of the keys
 that OnionPerf produces. Here is an example of the output produced by my
 node:
 http://phantomtrain.robgjansen.com:8081/phantomtrain-5242880-2017-03-19.tpf

 I think either your data is coming from a Torperf instance, or the output
 of OnionPerf is being filtered to remove some of the key=value entries.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by karsten):

 Ah, that directory contains .tpf files from Torperf instances (`torperf-*`
 and `siv*`) as well as OnionPerf instances (`op-*`).  The latter should
 contain all keys that OnionPerf produces.  Mind checking again?

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by robgjansen):

 Replying to [comment:58 karsten]:
 > Ah, that directory contains .tpf files from Torperf instances
 (`torperf-*` and `siv*`) as well as OnionPerf instances (`op-*`).  The
 latter should contain all keys that OnionPerf produces.  Mind checking
 again?

 Ahh, ok. I'm now looking at `op-us-5242880-2017-03-21.tpf`. The data
 element I think you are looking for is:
 {{{
 ENDPOINTREMOTE=199-119-112-144.i95.net:199.119.112.144:8080
 }}}

 I think that 8080 is the port that the tgen client requests Tor to connect
 to.

 I see the problem now. From the code, it appears that this port (8080 in
 this case) is the `--tgen-connect-port`, i.e., the port that the tgen
 client asks Tor to connect to. This not necessarily be the same as the
 `--tgen-listen-port`, since the remote server machine could have firewall
 rules set up to forward requests on various ports to the actual listening
 port.

 So, I wonder which of the following is correct:
 - hiro is using `--tgen-connect-port=8080` and `--tgen-listen-port=8080`
 - hiro is using `--tgen-connect-port=8080` and `--tgen-listen-port=80` and
 forwarding requests on 8080 over to the server running on 80
 - my conclusion after reading the code is incorrect.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by hiro):

 Please note that I am using the nginx as reverse proxy, so even though
 tgen is running on port 8080 this is being redirected to 80 externally. Is
 this correct?

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by robgjansen):

 I just realized, the easiest way to find out where you are instructing
 your client to connect is by looking inside the `onionperf-data/tgen-
 client/tgen.graphml.xml` file in your installation. Here is mine:

 {{{
  
 
   ...
   ih7hhuuppsy5wysu.onion:8080,167.114.171.3:8080
   ...
 
 }}}

 Notice that my client is connecting to port 8080. What port is listed in
 your client tgen file, hiro?

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by hiro):

 Hi,
 I have updated the configuration. Now the tgen server would be on 8080 and
 the client on 80. Please let me know if this is correct. I will check the
 new measurements tomorrow morning.

 More soon.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by robgjansen):

 According to [http://199.119.112.144:8081/op-us-1048576-2017-03-26.tpf a
 .tpf file  from 2017-03-26], the client is now correctly requesting port
 80 for non-onion downloads. (It is still requesting 8080 for onion
 downloads, but that should not affect affect path selection for onion
 service circuits.)

 Unfortunately, it looks like all of the downloads are now timing out,
 since the data shows `DIDTIMEOUT=1` for all entries in the .tpf file. More
 debugging is needed.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by hiro):

 Ok I think what is missing is a port forwarding so that 8080 is forwarded
 to 80.
 More soon.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by hiro):

 This is now implemented after changing the supported kernel in all 3
 instances. I am seeing successful transfers. It is probably a good idea to
 clean up old logs with no transfers and wrong ports in the measurements.
 I'll do this starting from tomorrow. Please let me know if you prefer
 otherwise.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by karsten):

 Can you trigger generation of .tpf files without having to wait for the
 daily cronjob to do that?  Then we could check if it's really using port
 80 and telling us about it.

 Removing old data once we're sure it's buggy is probably a good idea.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by hiro):

 Here is an example:

 @type torperf 1.0
 CONNECT=1490960319.10 DATACOMPLETE=1490960321.08 DATAPERC0=1490960319.56
 DATAPERC10=1490960319.98 DATAPERC100=1490960321.08
 DATAPERC20=1490960320.19 DATAPERC30=1490960320.36 DATAPERC40=1490960320.46
 DATAPERC50=1490960320.56 DATAPERC60=1490960320.68 DATAPERC70=1490960320.80
 DATAPERC80=1490960320.92 DATAPERC90=1490960321.04
 DATAREQUEST=1490960319.31 DATARESPONSE=1490960319.52 DIDTIMEOUT=0
 ENDPOINTLOCAL=localhost:127.0.0.1:46700
 ENDPOINTPROXY=localhost:127.0.0.1:33806
 ENDPOINTREMOTE=37.218.247.40:37.218.247.40:80 FILESIZE=1048576
 HOSTNAMELOCAL=op-nl HOSTNAMEREMOTE=op-nl LAUNCH=0.0
 NEGOTIATE=1490960319.10 READBYTES=1048642 REQUEST=1490960319.10
 RESPONSE=1490960319.31 SOCKET=1490960319.10 SOURCE=op-nl
 START=1490960319.10 WRITEBYTES=53

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by hiro):

 list of file on all VMs now updated.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by karsten):

 H, it looks like the direct downloads are fine now, but all onion
 service requests time out.  Ugh.

 Maybe OnionPerf tries to connect to port 80 for onion service requests,
 too, and there's no onion service and no local port forwarding.

 robgjansen, any ideas how to fix/work around this?

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by robgjansen):

 Replying to [comment:69 karsten]:

 > robgjansen, any ideas how to fix/work around this?

 It looks like that was a bug:
 
https://github.com/robgjansen/onionperf/commit/e3ea7db4d8b1c3f2f410aa85326177495a7215b1

 I pushed a fix to the OnionPerf repo on GitHub. @hiro Testing would be
 appreciated ;-)

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by hiro):

 Attaching new tpf files. Please note that I pulled the updates and
 reinstalled around 12pm.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by robgjansen):

 That's great, it looks like both onion and regular downloads are working
 correctly. But...

 {{{
 SOURCEADDRESS=unknown
 }}}

 it looks like my new feature to add the IP address of the measurement
 machine needs more testing on my end.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by karsten):

 Uhm, is it just me, or do these files still contain lots of timed out
 requests to onion services?

 @type torperf 1.0
 CONNECT=1491478019.16 DATACOMPLETE=0.0 DATAPERC0=1491478139.78
 DATAPERC10=0.0 DATAPERC20=0.0 DATAPERC30=0.0 DATAPERC40=0.0 DATAPERC50=0.0
 DATAPERC60=0.0 DATAPERC70=0.0 DATAPERC80=0.0 DATAPERC90=0.0
 DATAREQUEST=0.0 DATARESPONSE=0.0 DIDTIMEOUT=1
 ENDPOINTLOCAL=localhost:127.0.0.1:52680
 ENDPOINTPROXY=localhost:127.0.0.1:30850
 ENDPOINTREMOTE=7va4ssp7wgszolfj.onion:0.0.0.0:8080 FILESIZE=51200
 HOSTNAMELOCAL=op-nl HOSTNAMEREMOTE=(null) LAUNCH=0.0
 NEGOTIATE=1491478019.16 READBYTES=0 REQUEST=1491478019.16
 RESPONSE=1491478139.78 SOCKET=1491478019.16 SOURCE=op-nl
 SOURCEADDRESS=unknown START=1491478019.16 WRITEBYTES=0

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by hiro):

 Yes, I see the timeouts again in all the instances. Attaching results
 generated just now.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by hiro):

 I am noticing that every time onionperf tries to connect to a remote
 hostname on port 8080 it fails:
 @type torperf 1.0
 CONNECT=1491496019.19 DATACOMPLETE=0.0 DATAPERC0=1491496020.86
 DATAPERC10=0.0 DATAPERC20=0.0 DATAPERC30=0.0 DATAPERC40=0.0 DATAPERC50=0.0
 DATAPERC60=0.0 DATAPERC70=0.0 DATAPERC80=0.0 DATAPERC90=0.0
 DATAREQUEST=0.0 DATARESPONSE=0.0 DIDTIMEOUT=1
 ENDPOINTLOCAL=localhost:127.0.0.1:53300
 ENDPOINTPROXY=localhost:127.0.0.1:30850
 ENDPOINTREMOTE=7va4ssp7wgszolfj.onion:0.0.0.0:8080 FILESIZE=51200
 HOSTNAMELOCAL=op-nl HOSTNAMEREMOTE=(null) LAUNCH=0.0
 NEGOTIATE=1491496019.19 READBYTES=0 REQUEST=1491496019.19
 RESPONSE=1491496020.86 SOCKET=1491496019.19 SOURCE=op-nl
 SOURCEADDRESS=unknown START=1491496019.19 WRITEBYTES=0

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by robgjansen):

 - issue 1 - onion service downloads failing

   It should be connecting to `.onion:80` instead of `.onion:8080`. I found
 the bug, fixed it, tested it, and push the fix
 
[https://github.com/robgjansen/onionperf/commit/803650bb2cc3f8dec720ab39983a618848f76556
 to the onionperf master branch].

 - issue 2 - `SOURCEADDRESS=unknown`

   hiro, I need some help from you to figure out why this is not working
 (it works for me in my tests). Can you run the following and let me know
 the results:
 {{{
 source venv/bin/activate
 $ python
 >>> from onionperf import util
 >>> util.get_ip_address()
 }}}

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by hiro):

 Hi Rob,
 Updated the VMs so issue 1 should be live now.

 Regarding issue 2:


 $ python
 >>> from onionperf import util
 >>> util.get_ip_address()
 '37.218.247.40'

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by robgjansen):

 Hmm, OK, so it seems like OnionPerf should be producing correct
 SOURCEADDRESS lines. Are you running the `onionperf analyze` mode by some
 external script to regenerate the `.tpf` files and ignore the ones that
 OnionPerf produces? If so, you need to use something like
 {{{
 onionperf analyze --address=37.218.247.40 --name=op-nl --date=2017-04-10
 -t --tgen ... --torctl ...
 }}}
 This will ensure that the correct SOURCE and SOURCEADDRESS appear in the
 `.tpf.` output files, and that only downloads for the given date are
 included.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by hiro):

 I am letting onionperf generate the files each day together with the log
 rotation. But when I generate the tpf in the midlde of the day to test I
 am just running:
 onionperf analyze --torperf --tgen tgen-client/onionperf.tgen.log --torctl
 tor-client/onionperf.tor.log -p /home/cloud/onionperf/onionperf-data

 Which is why you might see the source address empty in this case. If you
 check a tpf generated at midnight though, the source address is present,
 e.g.: https://op-nl.onionperf.torproject.net/op-nl-1048576-2017-04-09.tpf

 Attaching newly generated measurements just now to check if it's
 connecting to the right port.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by hiro):

 I can confirm we do not have timeouts anymore. I have also deleted old
 corrupted measurements.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by karsten):

 hiro, I can confirm that .tpf files op-??-*-2017-04-11.tpf and later look
 good.  I'm attaching a graph comparing measurements from 2017-04-12 with
 Torperf data.  Can you please delete files from 2017-04-10?  Those contain
 timeouts and connections to port 8080.  Once that's done, I'll delete
 files from CollecTor.

 Moving forward, I think the following tasks remain:
  - Include data from Rob's phantomtrain.  I'm not sure where we are with
 this.
  - Add irl's instance op-ab, after changing its DNS to op-ab.something and
 possibly after switching that to port 80 using the same iptables trick
 that we use, but not necessarily.
  - Explain to ln5 how to set up an instance, maybe op-se, as replacement
 for the Torperf instance siv, ideally on port 80 with the same iptables
 trick that we use.
  - Turn off onionperf.torproject.org to reduce the complexity of our setup
 a little bit.
  - (Anything else?)

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by hiro):

 Hi Karsten,

 I have actually already have got to chat with ln5 and irl regarding old-
 torperf and op-ab.

 - Regarding old-torperf VM I have chatted with ln5 already and explained
 the setup last week. Haven't had opportunity to chat again.
 - Will ping irl again (last week he was a bit busy w something else) and
 see if the VM is ready so I can request a tpn subdomain for that VM.

 Regarding our VM onionperf.tpo, I have requested the shut down.

 Something else I was working on is a way to orchestrate deployments and
 managements of the VMs, but this will go into
 
https://trac.torproject.org/projects/tor/wiki/org/operations/Infrastructure/onionperf
 and will not affect the measurements.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--
Changes (by irl):

 * cc: irl@… (added)


Comment:

 Hiro - the IP address for op-ab will remain stable, I have a few /24s
 there to play with and I try to never change an address while a system is
 running. Some machines have had the same address over 20 years now (though
 the hardware has changed). It is set to 139.133.232.234. It also has an
 IPv6 address (native connectivity) but I'm not sure if Onionperf makes any
 use of this.

 I would like to add the port forwarding next week, if you could let me
 know what the firewall rules are that are required.

 For orchestration, I would love it to be Salt based as I'm learning this
 anyway.

 If you can ping me once there is a domain set up, I'll renew the lets
 encrypt cert to include the new domain.

 Thanks,

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by hiro):

 Hi irl,
 subdomain requested. Yes I have looked at salt in the last few days and
 was thinking to do it since we are starting to have more machines.

 The firewall rules I used are:

 iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j REDIRECT --to-
 port 8080
 iptables -t nat -A PREROUTING -i eth0 -p udp --dport 80 -j REDIRECT --to-
 port 8080

 Let me know if is there anything else :)

 Also I'll ping you once the subdomain is setup.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by robgjansen):

 Replying to [comment:81 karsten]:
 > Moving forward, I think the following tasks remain:
 >  - Include data from Rob's phantomtrain.  I'm not sure where we are with
 this.

 I added one last date filter to make sure that downloads that started on a
 day previous to the day that it finished are not included in the results;
 this is now
 
[https://github.com/robgjansen/onionperf/commit/8b1abb1795cd34390e1fa1f73f1d98219f58fe1b
 pushed to the master branch on GitHub].

 I regenerated my tpf files and I think the downloads are properly filtered
 now. The data I have been collecting is archived here, separated by
 measurement location (let me know if you want me to move the files for
 some reason):
   - http://phantomtrain.robgjansen.com/archive-216.17.99.183/
   - http://phantomtrain.robgjansen.com/archive-167.114.171.3/

 I also finished migrating my onionperf instance as I previously mentioned
 I would, and the new data will be available here:
   - http://phantomtrain.robgjansen.com

 I am planning on replacing the twisted web server with
 [https://caddyserver.com/ caddy], which will automatically get me a
 [https://letsencrypt.org/ let's encrypt] cert and host the files over
 https. Once that is set up, the links above should automatically get
 redirected the the https versions. You may want to wait until I have that
 set up to fetch the data. I'll be collecting measurements in the meantime
 :)

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by robgjansen):

 https://phantomtrain.robgjansen.com is now encrypted via a let's encrypt
 cert.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by irl):

 Replying to [comment:84 hiro]:
 > iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j REDIRECT
 --to-port 8080
 > iptables -t nat -A PREROUTING -i eth0 -p udp --dport 80 -j REDIRECT
 --to-port 8080

 Why do we need UDP rules? Should I be allowing UDP traffic through the
 firewall? (Currently it's blocked).

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by robgjansen):

 Replying to [comment:87 irl]:
 > Replying to [comment:84 hiro]:
 > > iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j REDIRECT
 --to-port 8080
 > > iptables -t nat -A PREROUTING -i eth0 -p udp --dport 80 -j REDIRECT
 --to-port 8080
 >
 > Why do we need UDP rules? Should I be allowing UDP traffic through the
 firewall? (Currently it's blocked).

 OnionPerf only uses TCP; no need to allow UDP ports. (I am only forwarding
 TCP ports on my instance and it's working fine.)

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by hiro):

 Hi,
 While I was testing I was afraid I had overlooked something and I made
 sure all possible traffic on port 80 was being redirected. So yes you are
 right that udp rule shouldn't be there.

 Regarding the latest updates. I have "refreshed" all three instances us,
 nl and hk.

 Regarding op-ab.onionperf.torproject.net subdomain that's now active.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by irl):

 Replying to [comment:88 robgjansen]:
 > OnionPerf only uses TCP; no need to allow UDP ports. (I am only
 forwarding TCP ports on my instance and it's working fine.)

 Ok cool, as I suspected. Thanks for clarifying.

 Replying to [comment:89 hiro]:
 > Regarding op-ab.onionperf.torproject.net subdomain that's now active.

 The cert has been refreshed to include this name.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by karsten):

 So, I deleted measurements from op-hk, op-nl, and op-us until 2017-04-10
 from CollecTor and Tor Metrics.  Looks good so far.

 I also noticed that the last remaining Torperf instance had an issue that
 I cannot fix easily, so I decided to remove it, too, and add the OnionPerf
 instance running on that host in the near future.  But as stated above
 somewhere, removing the last Torperf instance requires removing the code
 from CollecTor that downloads and merges Torperf files.  Please find
 [https://gitweb.torproject.org/karsten/metrics-db.git/log/?h=task-21272-2
 my branch task-21272-2] with a patch.

 Adding other OnionPerf hosts is on my list for the next days, or more
 realistically next week.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by iwakeh):

 Replying to [comment:91 karsten]:
 > So, I deleted measurements from op-hk, op-nl, and op-us until 2017-04-10
 from CollecTor and Tor Metrics.  Looks good so far.
 >
 > I also noticed that the last remaining Torperf instance had an issue
 that I cannot fix easily, so I decided to remove it, too, and add the
 OnionPerf instance running on that host in the near future.  But as stated
 above somewhere, removing the last Torperf instance requires removing the
 code from CollecTor that downloads and merges Torperf files.  Please find
 [https://gitweb.torproject.org/karsten/metrics-db.git/log/?h=task-21272-2
 my branch task-21272-2] with a patch.
 >

 Looks fine; all tests and checks pass.

 I added three commits
 [https://gitweb.torproject.org/user/iwakeh/collector.git/log/?h=task-21272-2
 here]; the latter two rename all 'torperf' occurrences and the package to
 'onionperf'; the final one also renames the properties.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by karsten):

 Thanks for looking!  And thanks for those three patches, all merged.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by irl):

 Looking at the .tpf output files from op-ab, it only seems to be testing
 Onion services and isn't doing any testing of the clearnet tgen instance,
 unless I'm misinterpreting the results.

 Any ideas what might have gone wrong?

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by robgjansen):

 The new onion data on the metrics performance page looks great! What's the
 status of adding the archived data and the current data from
 https://phantomtrain.robgjansen.com? Once that is done, we should have
 historical data going back to April 2016 :)

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by iwakeh):

 When testing collector I noticed that the op-{hk,nl,us} hosts all provide
 still data since 2017/04/11 , which currently is no problem.  But, at some
 point in future a deletion policy might be needed (CollecTor will supply
 historical data).  Or, is there one in place?

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21272 [Metrics]: Onionperf deployment

[Tor Bug Tracker & Wiki]

#21272: Onionperf deployment
-+--
 Reporter:  hiro |  Owner:  metrics-team
 Type:  enhancement  | Status:  closed
 Priority:  Medium   |  Milestone:
Component:  Metrics  |Version:
 Severity:  Normal   | Resolution:  fixed
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--
Changes (by karsten):

 * status:  needs_review => closed
 * resolution:   => fixed


Comment:

 I'm closing this ticket now. We have three OnionPerf instances deployed
 and integrated into CollecTor and Tor Metrics.

 Regarding irl's op-ab instance, we should discuss any remaining issues
 either at team meetings or on a new ticket, possibly even on GitHub where
 OnionPerf is developed.

 Regarding robgjansen's phantomtrain instance, we discussed putting that
 data on Tor Metrics under Research, but that discussion happens via email.

 Regarding iwakeh's question about a deletion policy, that discussion
 should probably happen on GitHub, too.

 However, nothing important remains for ''deploying'' OnionPerf. Closing.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs