Re: [tor-bugs] #27359 [Core Tor/Tor]: Store family lists more efficiently

[Tor Bug Tracker & Wiki]

#27359: Store family lists more efficiently
-+-
 Reporter:  nickm|  Owner:  nickm
 Type:  defect   | Status:  closed
 Priority:  Medium   |  Milestone:  Tor:
 |  0.4.0.x-final
Component:  Core Tor/Tor |Version:
 Severity:  Normal   | Resolution:
 Keywords:  035-roadmap-master, 035-triaged- |  implemented
  in-20180711|  Actual Points:  3
Parent ID:  #27243   | Points:
 Reviewer:  teor |Sponsor:
 |  Sponsor8-can
-+-
Changes (by nickm):

 * status:  merge_ready => closed
 * resolution:   => implemented
 * actualpoints:  2 => 3


--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #27359 [Core Tor/Tor]: Store family lists more efficiently

[Tor Bug Tracker & Wiki]

#27359: Store family lists more efficiently
-+-
 Reporter:  nickm|  Owner:  nickm
 Type:  defect   | Status:
 |  merge_ready
 Priority:  Medium   |  Milestone:  Tor:
 |  0.4.0.x-final
Component:  Core Tor/Tor |Version:
 Severity:  Normal   | Resolution:
 Keywords:  035-roadmap-master, 035-triaged- |  Actual Points:  2
  in-20180711|
Parent ID:  #27243   | Points:
 Reviewer:  teor |Sponsor:
 |  Sponsor8-can
-+-

Comment (by nickm):

 Thanks! Squashed and merged!

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #27359 [Core Tor/Tor]: Store family lists more efficiently

[Tor Bug Tracker & Wiki]

#27359: Store family lists more efficiently
-+-
 Reporter:  nickm|  Owner:  nickm
 Type:  defect   | Status:
 |  merge_ready
 Priority:  Medium   |  Milestone:  Tor:
 |  0.4.0.x-final
Component:  Core Tor/Tor |Version:
 Severity:  Normal   | Resolution:
 Keywords:  035-roadmap-master, 035-triaged- |  Actual Points:  2
  in-20180711|
Parent ID:  #27243   | Points:
 Reviewer:  teor |Sponsor:
 |  Sponsor8-can
-+-
Changes (by teor):

 * status:  needs_review => merge_ready


Comment:

 Looks good to me.

 Let's defer the routerinfo / microdesc decision until later.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #27359 [Core Tor/Tor]: Store family lists more efficiently

[Tor Bug Tracker & Wiki]

#27359: Store family lists more efficiently
-+-
 Reporter:  nickm|  Owner:  nickm
 Type:  defect   | Status:
 |  needs_review
 Priority:  Medium   |  Milestone:  Tor:
 |  0.4.0.x-final
Component:  Core Tor/Tor |Version:
 Severity:  Normal   | Resolution:
 Keywords:  035-roadmap-master, 035-triaged- |  Actual Points:  2
  in-20180711|
Parent ID:  #27243   | Points:
 Reviewer:  teor |Sponsor:
 |  Sponsor8-can
-+-
Changes (by nickm):

 * status:  needs_revision => needs_review


Comment:

 I've resolved the bug, and added a bunch of tests.  Do you like the test
 coverage now?

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #27359 [Core Tor/Tor]: Store family lists more efficiently

[Tor Bug Tracker & Wiki]

#27359: Store family lists more efficiently
-+-
 Reporter:  nickm|  Owner:  nickm
 Type:  defect   | Status:
 |  needs_revision
 Priority:  Medium   |  Milestone:  Tor:
 |  0.3.6.x-final
Component:  Core Tor/Tor |Version:
 Severity:  Normal   | Resolution:
 Keywords:  035-roadmap-master, 035-triaged- |  Actual Points:  2
  in-20180711|
Parent ID:  #27243   | Points:
 Reviewer:  teor |Sponsor:
 |  Sponsor8-can
-+-
Changes (by teor):

 * status:  needs_review => needs_revision


Comment:

 I did a review, looks good, except for a case-comparison bug.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #27359 [Core Tor/Tor]: Store family lists more efficiently

[Tor Bug Tracker & Wiki]

#27359: Store family lists more efficiently
-+-
 Reporter:  nickm|  Owner:  nickm
 Type:  defect   | Status:
 |  needs_review
 Priority:  Medium   |  Milestone:  Tor:
 |  0.3.6.x-final
Component:  Core Tor/Tor |Version:
 Severity:  Normal   | Resolution:
 Keywords:  035-roadmap-master, 035-triaged- |  Actual Points:  2
  in-20180711|
Parent ID:  #27243   | Points:
 Reviewer:   |Sponsor:
 |  Sponsor8-can
-+-

Comment (by teor):

 Replying to [comment:9 nickm]:
 > > 1. Also do the same canonicalization in ns consensus documents
 >
 > fwiw, I don't think ns consensus documents include family lines?

 Yes, you're right, family only appears in descriptors.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #27359 [Core Tor/Tor]: Store family lists more efficiently

[Tor Bug Tracker & Wiki]

#27359: Store family lists more efficiently
-+-
 Reporter:  nickm|  Owner:  nickm
 Type:  defect   | Status:
 |  needs_review
 Priority:  Medium   |  Milestone:  Tor:
 |  0.3.6.x-final
Component:  Core Tor/Tor |Version:
 Severity:  Normal   | Resolution:
 Keywords:  035-roadmap-master, 035-triaged- |  Actual Points:  2
  in-20180711|
Parent ID:  #27243   | Points:
 Reviewer:   |Sponsor:
 |  Sponsor8-can
-+-
Changes (by nickm):

 * status:  accepted => needs_review
 * actualpoints:   => 2


Comment:

 See branch `ticket27359_v2`, with PR at
 https://github.com/torproject/tor/pull/463

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #27359 [Core Tor/Tor]: Store family lists more efficiently

[Tor Bug Tracker & Wiki]

#27359: Store family lists more efficiently
-+-
 Reporter:  nickm|  Owner:  nickm
 Type:  defect   | Status:
 |  accepted
 Priority:  Medium   |  Milestone:  Tor:
 |  0.3.6.x-final
Component:  Core Tor/Tor |Version:
 Severity:  Normal   | Resolution:
 Keywords:  035-roadmap-master, 035-triaged- |  Actual Points:
  in-20180711|
Parent ID:  #27243   | Points:
 Reviewer:   |Sponsor:
 |  Sponsor8-can
-+-

Comment (by nickm):

 > 1. Also do the same canonicalization in ns consensus documents

 fwiw, I don't think ns consensus documents include family lines?

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #27359 [Core Tor/Tor]: Store family lists more efficiently

[Tor Bug Tracker & Wiki]

#27359: Store family lists more efficiently
-+-
 Reporter:  nickm|  Owner:  nickm
 Type:  defect   | Status:
 |  accepted
 Priority:  Medium   |  Milestone:  Tor:
 |  0.3.6.x-final
Component:  Core Tor/Tor |Version:
 Severity:  Normal   | Resolution:
 Keywords:  035-roadmap-master, 035-triaged- |  Actual Points:
  in-20180711|
Parent ID:  #27243   | Points:
 Reviewer:   |Sponsor:
 |  Sponsor8-can
-+-

Comment (by teor):

 Replying to [comment:6 nickm]:
 > ...
 > 4. Add a new consensus method to canonicalize family lists by adding
 each node's own identity and sorting the entries.

 Looking at the available microdescs, we should also:
 * uppercase the hex ids

 It's also worth noting;:
 * chutney doesn't set MyFamily, so we'll need to test carefully
 * this change likely causes:
   * a slight increase in the size of descriptors on disk (from the extra
 ids), offset by removing =nick and malformed hex id, and
   * a slight decrease of compressed descriptor size, because the sorted
 lines are identical

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #27359 [Core Tor/Tor]: Store family lists more efficiently

[Tor Bug Tracker & Wiki]

#27359: Store family lists more efficiently
-+-
 Reporter:  nickm|  Owner:  nickm
 Type:  defect   | Status:
 |  accepted
 Priority:  Medium   |  Milestone:  Tor:
 |  0.3.6.x-final
Component:  Core Tor/Tor |Version:
 Severity:  Normal   | Resolution:
 Keywords:  035-roadmap-master, 035-triaged- |  Actual Points:
  in-20180711|
Parent ID:  #27243   | Points:
 Reviewer:   |Sponsor:
 |  Sponsor8-can
-+-

Comment (by nickm):

 For future work (TODO, make tickets)
   1. Also do the same canonicalization in ns consensus documents
   2. Deprecate $foo=bar and $foo~bar
   3. Relays should warn if they are about to publish an invalid hexid in
 their family line.
   4. Relays should canonicalize their own family lines

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #27359 [Core Tor/Tor]: Store family lists more efficiently

[Tor Bug Tracker & Wiki]

#27359: Store family lists more efficiently
-+-
 Reporter:  nickm|  Owner:  nickm
 Type:  defect   | Status:
 |  accepted
 Priority:  Medium   |  Milestone:  Tor:
 |  0.3.6.x-final
Component:  Core Tor/Tor |Version:
 Severity:  Normal   | Resolution:
 Keywords:  035-roadmap-master, 035-triaged- |  Actual Points:
  in-20180711|
Parent ID:  #27243   | Points:
 Reviewer:   |Sponsor:
 |  Sponsor8-can
-+-

Comment (by nickm):

 After conversation with teor, here's what we're thinking:

 1. Ignore any malformed family entries
 2. Treat $hexid=nickname as if it just said $hexid.
 3. Don't use this trick outside of parsed microdescriptors, since that's
 what clients need.
 4. Add a new consensus method to canonicalize family lists by adding each
 node's own identity and sorting the entries.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #27359 [Core Tor/Tor]: Store family lists more efficiently

[Tor Bug Tracker & Wiki]

#27359: Store family lists more efficiently
-+-
 Reporter:  nickm|  Owner:  nickm
 Type:  defect   | Status:
 |  accepted
 Priority:  Medium   |  Milestone:  Tor:
 |  0.3.6.x-final
Component:  Core Tor/Tor |Version:
 Severity:  Normal   | Resolution:
 Keywords:  035-roadmap-master, 035-triaged- |  Actual Points:
  in-20180711|
Parent ID:  #27243   | Points:
 Reviewer:   |Sponsor:
 |  Sponsor8-can
-+-

Comment (by nickm):

 So the idea of the scheme here is to encode families in a more compact
 representation: roughly, as a sorted array of 21-byte elements, where each
 element is a one byte tag, and 20 bytes of either a sha1 ID or a nickname.
 These objects are reference-counted, so that they can be shared by relays
 with the same family members.

 To optimize the number of family objects that are shared by multiple
 relays, I want to add each relay's hex ID to its own family -- doing this
 will make all the relays in the same family have the same encoded family.
 (It's harmless to consider a relay a member of its own family.)

 I've run into a few issues with this, and I'm not sure which of them need
 to be solved pre-merge, and how.

   1) A small number of relays specify invalid hex IDs in their family
 lines -- typically, with the wrong number of hex digits.

   2) A small number of relays use the $hexid=nickname syntax, which can't
 be encoded in the manner above, and which is probably not what they want
 anyway.

   3) Authorities need to process family lines exactly as they are
 received, and rely on a lossless encoding of their inputs.

   4) When parsing a microdescriptor, we do not know which relay it is
 associated with, so we can't easily add its own ID to its family when
 parsing it.  (Further, it is in theory possible for two misconfigured
 relays to wind up with the same microdescriptor, I think.)

 I'm not sure quite what to do about each of these issues.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #27359 [Core Tor/Tor]: Store family lists more efficiently

[Tor Bug Tracker & Wiki]

#27359: Store family lists more efficiently
-+-
 Reporter:  nickm|  Owner:  nickm
 Type:  defect   | Status:
 |  accepted
 Priority:  Medium   |  Milestone:  Tor:
 |  0.3.6.x-final
Component:  Core Tor/Tor |Version:
 Severity:  Normal   | Resolution:
 Keywords:  035-roadmap-master, 035-triaged- |  Actual Points:
  in-20180711|
Parent ID:  #27243   | Points:
 Reviewer:   |Sponsor:
 |  Sponsor8-can
-+-
Changes (by nickm):

 * status:  new => accepted
 * owner:  (none) => nickm


--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #27359 [Core Tor/Tor]: Store family lists more efficiently

[Tor Bug Tracker & Wiki]

#27359: Store family lists more efficiently
-+-
 Reporter:  nickm|  Owner:  (none)
 Type:  defect   | Status:  new
 Priority:  Medium   |  Milestone:  Tor:
 |  0.3.6.x-final
Component:  Core Tor/Tor |Version:
 Severity:  Normal   | Resolution:
 Keywords:  035-roadmap-master, 035-triaged- |  Actual Points:
  in-20180711|
Parent ID:  #27243   | Points:
 Reviewer:   |Sponsor:
-+-
Changes (by nickm):

 * milestone:  Tor: 0.3.5.x-final => Tor: 0.3.6.x-final


--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #27359 [Core Tor/Tor]: Store family lists more efficiently

[Tor Bug Tracker & Wiki]

#27359: Store family lists more efficiently
-+-
 Reporter:  nickm|  Owner:  (none)
 Type:  defect   | Status:  new
 Priority:  Medium   |  Milestone:  Tor:
 |  0.3.5.x-final
Component:  Core Tor/Tor |Version:
 Severity:  Normal   | Resolution:
 Keywords:  035-roadmap-master, 035-triaged- |  Actual Points:
  in-20180711|
Parent ID:  #27243   | Points:
 Reviewer:   |Sponsor:
-+-

Comment (by nickm):

 I've got an untested, undocumented, unintegrated sketch design in a branch
 called `ticket27359`.  It needs to get tested, documented, and integrated
 with microdesc. (maybe someday it could get integrated with routerdesc --
 but that wouldn't work for authorities yet.)

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs