Re: [tor-bugs] #28320 [Metrics/CollecTor]: Rewrite CollecTor relaydescs module using Stem/txtorcon

[Tor Bug Tracker & Wiki]

#28320: Rewrite CollecTor relaydescs module using Stem/txtorcon
---+---
 Reporter:  karsten|  Owner:  irl
 Type:  task   | Status:  closed
 Priority:  Medium |  Milestone:
Component:  Metrics/CollecTor  |Version:
 Severity:  Normal | Resolution:  fixed
 Keywords: |  Actual Points:
Parent ID: | Points:
 Reviewer: |Sponsor:  Sponsor13
---+---
Changes (by gaba):

 * status:  accepted => closed
 * resolution:   => fixed


Comment:

 Yeah!

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #28320 [Metrics/CollecTor]: Rewrite CollecTor relaydescs module using Stem/txtorcon

[Tor Bug Tracker & Wiki]

#28320: Rewrite CollecTor relaydescs module using Stem/txtorcon
---+---
 Reporter:  karsten|  Owner:  irl
 Type:  task   | Status:  accepted
 Priority:  Medium |  Milestone:
Component:  Metrics/CollecTor  |Version:
 Severity:  Normal | Resolution:
 Keywords: |  Actual Points:
Parent ID: | Points:
 Reviewer: |Sponsor:  Sponsor13
---+---

Comment (by irl):

 The tech report is published and source code on GitHub. Only remaining
 task is to set up a test instance with the new code somewhere.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #28320 [Metrics/CollecTor]: Rewrite CollecTor relaydescs module using Stem/txtorcon

[Tor Bug Tracker & Wiki]

#28320: Rewrite CollecTor relaydescs module using Stem/txtorcon
---+---
 Reporter:  karsten|  Owner:  irl
 Type:  task   | Status:  accepted
 Priority:  Medium |  Milestone:
Component:  Metrics/CollecTor  |Version:
 Severity:  Normal | Resolution:
 Keywords: |  Actual Points:
Parent ID: | Points:
 Reviewer: |Sponsor:  Sponsor13
---+---
Changes (by irl):

 * status:  new => accepted
 * owner:  metrics-team => irl


--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #28320 [Metrics/CollecTor]: Rewrite CollecTor relaydescs module using Stem/txtorcon

[Tor Bug Tracker & Wiki]

#28320: Rewrite CollecTor relaydescs module using Stem/txtorcon
---+--
 Reporter:  karsten|  Owner:  metrics-team
 Type:  task   | Status:  new
 Priority:  Medium |  Milestone:
Component:  Metrics/CollecTor  |Version:
 Severity:  Normal | Resolution:
 Keywords: |  Actual Points:
Parent ID: | Points:
 Reviewer: |Sponsor:  Sponsor13
---+--

Comment (by atagar):

 Gotcha! When you guys know what you want just let me know. I'd be happy to
 whip up a prototype if you'd like since the asks here thus far sound
 pretty simple.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #28320 [Metrics/CollecTor]: Rewrite CollecTor relaydescs module using Stem/txtorcon

[Tor Bug Tracker & Wiki]

#28320: Rewrite CollecTor relaydescs module using Stem/txtorcon
---+--
 Reporter:  karsten|  Owner:  metrics-team
 Type:  task   | Status:  new
 Priority:  Medium |  Milestone:
Component:  Metrics/CollecTor  |Version:
 Severity:  Normal | Resolution:
 Keywords: |  Actual Points:
Parent ID: | Points:
 Reviewer: |Sponsor:  Sponsor13
---+--

Comment (by karsten):

 It's true that Stem/txtorcon might not be of primary interest for this
 prototype. But if we later want to build upon this prototype to rewrite
 the rest of CollecTor's relaydescs module, Stem's descriptor parsing and
 validation capabilities will be quite useful. I'll leave it up to irl to
 decide whether he wants to use Stem or txtorcon. This discussion will also
 be more useful as soon as the requirements are written down. Stay tuned!

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #28320 [Metrics/CollecTor]: Rewrite CollecTor relaydescs module using Stem/txtorcon

[Tor Bug Tracker & Wiki]

#28320: Rewrite CollecTor relaydescs module using Stem/txtorcon
---+--
 Reporter:  karsten|  Owner:  metrics-team
 Type:  task   | Status:  new
 Priority:  Medium |  Milestone:
Component:  Metrics/CollecTor  |Version:
 Severity:  Normal | Resolution:
 Keywords: |  Actual Points:
Parent ID: | Points:
 Reviewer: |Sponsor:  Sponsor13
---+--

Comment (by atagar):

 Hi Karsten. This sounds an awful lot like DocTor (download descriptors on
 an hourly basis and check a series of characteristics).

 That said, honestly I'm unsure you need either stem or txorcon for this.
 If all you want is to download descriptors won't cron and curl do the
 trick? Stem's benefit is that it **parses** descriptors and can download
 leveraging directory mirrors. As for txtorcon, I'm unaware of any
 descriptor capabilities it provides (meejah can correct me wrong but it's
 solely a twisted control port controller).

 Would you mind further describing what you're hoping for one of these to
 provide? If you'd like a simple example of downloading descriptors
 [https://stem.torproject.org/tutorials/mirror_mirror_on_the_wall.html
 #where-can-i-get-the-current-descriptors our tutorials],
 [https://stem.torproject.org/tutorials/examples/download_descriptor.html
 demos], and
 [https://gitweb.torproject.org/doctor.git/tree/descriptor_checker.py
 doctor] have examples.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

[tor-bugs] #28320 [Metrics/CollecTor]: Rewrite CollecTor relaydescs module using Stem/txtorcon

[Tor Bug Tracker & Wiki]

#28320: Rewrite CollecTor relaydescs module using Stem/txtorcon
---+--
 Reporter:  karsten|  Owner:  metrics-team
 Type:  task   | Status:  new
 Priority:  Medium |  Milestone:
Component:  Metrics/CollecTor  |Version:
 Severity:  Normal |   Keywords:
Actual Points: |  Parent ID:
   Points: |   Reviewer:
  Sponsor:  Sponsor13  |
---+--
 The CollecTor service collects and archives data from various nodes and
 services in the public Tor network. Internally, it consists of several
 modules that are running in the background following a pre-defined
 schedule. These modules either download data from other hosts or process
 data that has been copied from other hosts to the local file system. The
 processed data is then provided via a locally running static web server.

 CollecTor is written in Java. It uses several APIs either provided in the
 JDK or in third-party libraries. For example, it uses
 `java.util.concurrent` for scheduling. However, it does not use a specific
 framework for batch processing. That is why it has to solve challenges
 like the following on its own:

  - Scheduling: Make sure modules are running, say, once per hour; avoid
 overlapping runs.
  - Dependencies: Make sure that module runs don't interfere with each
 other; one module writes newly obtained files to disk, another tars them
 up, yet another writes an index file and provides that to external
 applications.
  - Shutdowns: Handle externally triggered shutdowns gracefully and make
 sure the service resumes operation after reboot, without missing data.

 These are just a few examples, and CollecTor does not resolve all of them
 in the best way possible. It also feels like somebody must have solved
 these challenges before. We should find out, and the best way is probably
 to try it out in practice.

 In Mexico City we decided to evaluate existing batch processing frameworks
 by rewriting the CollecTor relaydescs module using Python with Stem or
 txtorcon. It should be sufficient to make it work for at least consensuses
 and server descriptors as initial proof of concept. Other descriptor types
 can follow later, if we decide to switch from Java to Python for
 CollecTor.

 The first steps are to write down requirements and possible Python
 libraries for the batch-processing parts.

 We're done with this task when we have a working prototype of CollecTor in
 Python that fetches consensuses and server descriptors from the directory
 authorities.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs