Re: [tor-bugs] #21693 [Core Tor/Tor]: prop224: HS descriptors do wasteful double-base64 encoding

[Tor Bug Tracker & Wiki]

#21693: prop224: HS descriptors do wasteful double-base64 encoding
--+
 Reporter:  asn   |  Owner:  asn
 Type:  task  | Status:  closed
 Priority:  Medium|  Milestone:  Tor:
  |  unspecified
Component:  Core Tor/Tor  |Version:
 Severity:  Normal| Resolution:  wontfix
 Keywords:  tor-hs, prop224, 031-stretch  |  Actual Points:
Parent ID:| Points:  4
 Reviewer:|Sponsor:  SponsorR-can
--+
Changes (by teor):

 * status:  assigned => closed
 * resolution:   => wontfix


Comment:

 We deployed code that relies on this encoding, so I don't think we can fix
 this issue without an onion service ersion upgrade.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21693 [Core Tor/Tor]: prop224 HS descriptors do wasteful double-base64 encoding

[Tor Bug Tracker & Wiki]

#21693: prop224 HS descriptors do wasteful double-base64 encoding
+
 Reporter:  asn |  Owner:
 Type:  task| Status:  new
 Priority:  Medium  |  Milestone:  Tor: 0.3.1.x-final
Component:  Core Tor/Tor|Version:
 Severity:  Normal  | Resolution:
 Keywords:  tor-hs prop224  |  Actual Points:
Parent ID:  #21334  | Points:  4
 Reviewer:  |Sponsor:  SponsorR-can
+

Comment (by teor):

 I'm not sure I understand the issue here: what stops you replacing
 `base64(encrypt(inner_layer))` with `encrypt(inner_layer)`?
 Is there some requirement that it be plain-text?

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21693 [Core Tor/Tor]: prop224 HS descriptors do wasteful double-base64 encoding

[Tor Bug Tracker & Wiki]

#21693: prop224 HS descriptors do wasteful double-base64 encoding
+
 Reporter:  asn |  Owner:
 Type:  task| Status:  new
 Priority:  Medium  |  Milestone:  Tor: 0.3.1.x-final
Component:  Core Tor/Tor|Version:
 Severity:  Normal  | Resolution:
 Keywords:  tor-hs prop224  |  Actual Points:
Parent ID:  #21334  | Points:  4
 Reviewer:  |Sponsor:  SponsorR-can
+

Comment (by asn):

 Replying to [comment:1 teor]:
 > I'm not sure I understand the issue here: what stops you replacing
 `base64(encrypt(inner_layer))` with `encrypt(inner_layer)`?
 > Is there some requirement that it be plain-text?

 Hmm, I think it's because we use the routerparse API (`tokenize_string()`
 etc.)  to parse each layer, and the routerparse API mainly works on NUL-
 terminated strings.

 Maybe we could play with the `start` and `end` arguments of
 `tokenize_string()` to tell it where it should stop parsing due to binary
 data. In that case, I think we would need some sort of separator between
 the ASCII part of the descriptor layer and the binary part of the
 descriptor layer.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21693 [Core Tor/Tor]: prop224 HS descriptors do wasteful double-base64 encoding

[Tor Bug Tracker & Wiki]

#21693: prop224 HS descriptors do wasteful double-base64 encoding
+
 Reporter:  asn |  Owner:
 Type:  task| Status:  new
 Priority:  Medium  |  Milestone:  Tor: 0.3.1.x-final
Component:  Core Tor/Tor|Version:
 Severity:  Normal  | Resolution:
 Keywords:  tor-hs prop224  |  Actual Points:
Parent ID:  | Points:  4
 Reviewer:  |Sponsor:  SponsorR-can
+
Changes (by nickm):

 * parent:  #21334 =>


--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21693 [Core Tor/Tor]: prop224 HS descriptors do wasteful double-base64 encoding

[Tor Bug Tracker & Wiki]

#21693: prop224 HS descriptors do wasteful double-base64 encoding
+
 Reporter:  asn |  Owner:
 Type:  task| Status:  new
 Priority:  Medium  |  Milestone:  Tor: 0.3.1.x-final
Component:  Core Tor/Tor|Version:
 Severity:  Normal  | Resolution:
 Keywords:  tor-hs prop224  |  Actual Points:
Parent ID:  | Points:  4
 Reviewer:  |Sponsor:  SponsorR-can
+

Comment (by asn):

 A further point of complication here is that we apply NUL padding (up to
 nearest multiple of 10k bytes) on the superencrypted section to hide
 metadata about client auth details and intro points. So it's more like:

 `middle_layer = b64(encrypt(client_auth_data + b64(encrypt(inner_layer)) +
 nul_padding))`
 `outer_layer = header + middle_layer.`

 So unfortunately it's not as simple as replacing
 `b64(encrypt(inner_layer))` with `encrypt(inner_layer)` since then the
 binary ciphertext gets mangled with the NUL padding... :(

 I guess this means we need some sort of frame on the binary data that
 specifies the length of `encrypt(inner_layer)`, so that the decoding side
 can separate the ciphertext from the padding.

 In my experience, these sort of frames need careful consideration due to
 all sorts of weird padding-oracle type of stuff... Will think some more,
 but this might be a reasonable topic for amsterdam as well...

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21693 [Core Tor/Tor]: prop224: HS descriptors do wasteful double-base64 encoding

[Tor Bug Tracker & Wiki]

#21693: prop224: HS descriptors do wasteful double-base64 encoding
-+
 Reporter:  asn  |  Owner:  asn
 Type:  task | Status:  assigned
 Priority:  Medium   |  Milestone:  Tor: 0.3.1.x-final
Component:  Core Tor/Tor |Version:
 Severity:  Normal   | Resolution:
 Keywords:  tor-hs, prop224  |  Actual Points:
Parent ID:   | Points:  4
 Reviewer:   |Sponsor:  SponsorR-can
-+

Comment (by nickm):

 Something that would be almost as good: whenever we are going to encrypt,
 first compress, and then encrypt.  That would mitigate most of the
 trouble.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21693 [Core Tor/Tor]: prop224: HS descriptors do wasteful double-base64 encoding

[Tor Bug Tracker & Wiki]

#21693: prop224: HS descriptors do wasteful double-base64 encoding
-+
 Reporter:  asn  |  Owner:  asn
 Type:  task | Status:  assigned
 Priority:  Medium   |  Milestone:  Tor: 0.3.1.x-final
Component:  Core Tor/Tor |Version:
 Severity:  Normal   | Resolution:
 Keywords:  tor-hs, prop224  |  Actual Points:
Parent ID:   | Points:  4
 Reviewer:   |Sponsor:  SponsorR-can
-+

Comment (by asn):

 Replying to [comment:6 nickm]:
 > Something that would be almost as good: whenever we are going to
 encrypt, first compress, and then encrypt.  That would mitigate most of
 the trouble.

 Nice suggestion. Let's see how that could work out.

 As discussed before here is how the descriptor layers work:

 `middle_layer = b64(encrypt(client_auth_data + b64(encrypt(inner_layer)) +
 nul_padding))`
 `outer_layer = header + middle_layer`

 There are two encrypts here. We should probably '''not''' compress the
 plaintext of the middle layer, as that includes the NUL padding, which
 would basically get annihilated by the compression and lose all of its
 metadata hiding properties (we are using the NUL padding to hide the
 number of intro points and client auth details).

 However, compressing the `inner_layer` before `b64(encrypt(inner_layer))`
 could make sense. The inner layer is basically a small header with a bunch
 of intro points. It's usually about 3k bytes (for 3-4 intro points), and
 it compresses down to about 2k bytes using zlib (based on some testing I
 just did).

 So by compressing the inner layer before encrypting we can save about 1k
 bytes (which is also about the cost of double-base64 encoding).

 Do you think we should do this?
 Is the zlib API a PITA to use, or do you think it's gonna be a
 straightforward patch?

 

 That said, the above change will '''not''' change the final size of the HS
 descriptor because of the padding that gets applied on the layer above.

 If we wanted to also reduce the size of the final descriptor we could
 relax the padding logic from padding to 10k bytes (total desc size: 14k
 bytes), to padding to 8k bytes or so (total desc size: 11.5k bytes). I
 think that might be OK to do, if we feel that having 14k bytes descriptors
 is a stupid thing.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21693 [Core Tor/Tor]: prop224: HS descriptors do wasteful double-base64 encoding

[Tor Bug Tracker & Wiki]

#21693: prop224: HS descriptors do wasteful double-base64 encoding
-+
 Reporter:  asn  |  Owner:  asn
 Type:  task | Status:  assigned
 Priority:  Medium   |  Milestone:  Tor: 0.3.1.x-final
Component:  Core Tor/Tor |Version:
 Severity:  Normal   | Resolution:
 Keywords:  tor-hs, prop224  |  Actual Points:
Parent ID:   | Points:  4
 Reviewer:   |Sponsor:  SponsorR-can
-+

Comment (by asn):

 WRT the zlib API, I looked at the test `test_buffers_zlib_impl()` and it
 does not seem super hard. It seems to require to require a `buf_t`, the
 use of `write_to_buf_zlib()`, and then extracting the compressed data from
 the `buf_t` into a `uint8_t`. Uncompressing seems even easier, just use
 `tor_gzip_uncompress()`.

 Maybe I should give this a try for the inner layer.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21693 [Core Tor/Tor]: prop224: HS descriptors do wasteful double-base64 encoding

[Tor Bug Tracker & Wiki]

#21693: prop224: HS descriptors do wasteful double-base64 encoding
-+
 Reporter:  asn  |  Owner:  asn
 Type:  task | Status:  assigned
 Priority:  Medium   |  Milestone:  Tor: 0.3.1.x-final
Component:  Core Tor/Tor |Version:
 Severity:  Normal   | Resolution:
 Keywords:  tor-hs, prop224  |  Actual Points:
Parent ID:   | Points:  4
 Reviewer:   |Sponsor:  SponsorR-can
-+

Comment (by nickm):

 It's even easier than that  -- you can use tor_gzip_compress and
 tor_gzip_uncompress. No need to involve bufs.

 And what about doing it like this --
 {{{
   outer-layer = header + middle_layer
   middle-layer = b64(encrypt(compressed-part + nul-padding))
   compressed-part = gzip(client_auth_data +
 b64(encrypt(gzip(inner_layer
 }}}

 This keeps the length-hiding property of the nul padding, but allows more
 client_auth_data.

 Thoughts?

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21693 [Core Tor/Tor]: prop224: HS descriptors do wasteful double-base64 encoding

[Tor Bug Tracker & Wiki]

#21693: prop224: HS descriptors do wasteful double-base64 encoding
-+
 Reporter:  asn  |  Owner:  asn
 Type:  task | Status:  assigned
 Priority:  Medium   |  Milestone:  Tor: 0.3.1.x-final
Component:  Core Tor/Tor |Version:
 Severity:  Normal   | Resolution:
 Keywords:  tor-hs, prop224  |  Actual Points:
Parent ID:   | Points:  4
 Reviewer:   |Sponsor:  SponsorR-can
-+

Comment (by dgoulet):

 That scheme looks sane.

 I'm curious on how much we'll win here to be honest. `inner_layer` is a
 part we can save a bit of space but it will be from ~3k to ~2k... (most
 common case). And then `client_auth_data` is pretty small (16 lines of
 small amount of text) so maybe 1k to 800B?

 If HSDir were serving descriptor compressed that is `gzip(outer-layer)`,
 we would go from 14k to ~11k for a client to download. (Same for uploading
 a desc for a service). So I'm guessing we can bring that 11k to maybe ~10k
 with compressing more inside the descriptor (inner-layer)?

 The above implies two things. First, we need to change how HSDir are
 serving descriptors which is fine because we could just make that "if
 request comes in with URL.z, send it compressed". And we make client fetch
 it that way by default. Second, we need to implement the
 compression/encryption changes for the descriptor encoding/decoding in
 hs_descriptor.c which is a bit involving considering testing (maybe a day
 of work).

 That being said and to be honest, I'm slightly unconvinced that all this
 is needed. Going between ~10k to ~14k for the common case, it is very
 little for something you "should" download every ~18h-24h. Nathan in AMS
 was telling me that the concern with mobile for instance is not much the
 bandwidth nowadays but rather the battery life which we are going to kill
 more with multiple gzip instead of download extra 3k *I think*.
 (speculation)

 I'm missing some data here on how much going from ~3k (legacy code) to
 ~14k will affect people around the world... :S

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21693 [Core Tor/Tor]: prop224: HS descriptors do wasteful double-base64 encoding

[Tor Bug Tracker & Wiki]

#21693: prop224: HS descriptors do wasteful double-base64 encoding
--+
 Reporter:  asn   |  Owner:  asn
 Type:  task  | Status:  assigned
 Priority:  Medium|  Milestone:  Tor:
  |  0.3.1.x-final
Component:  Core Tor/Tor  |Version:
 Severity:  Normal| Resolution:
 Keywords:  tor-hs, prop224, 031-stretch  |  Actual Points:
Parent ID:| Points:  4
 Reviewer:|Sponsor:  SponsorR-can
--+
Changes (by asn):

 * keywords:  tor-hs, prop224 => tor-hs, prop224, 031-stretch


--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21693 [Core Tor/Tor]: prop224: HS descriptors do wasteful double-base64 encoding

[Tor Bug Tracker & Wiki]

#21693: prop224: HS descriptors do wasteful double-base64 encoding
--+
 Reporter:  asn   |  Owner:  asn
 Type:  task  | Status:  assigned
 Priority:  Medium|  Milestone:  Tor:
  |  0.3.2.x-final
Component:  Core Tor/Tor  |Version:
 Severity:  Normal| Resolution:
 Keywords:  tor-hs, prop224, 031-stretch  |  Actual Points:
Parent ID:| Points:  4
 Reviewer:|Sponsor:  SponsorR-can
--+
Changes (by asn):

 * milestone:  Tor: 0.3.1.x-final => Tor: 0.3.2.x-final


Comment:

 Deferring to 0.3.2 for now.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21693 [Core Tor/Tor]: prop224: HS descriptors do wasteful double-base64 encoding

[Tor Bug Tracker & Wiki]

#21693: prop224: HS descriptors do wasteful double-base64 encoding
--+
 Reporter:  asn   |  Owner:  asn
 Type:  task  | Status:  assigned
 Priority:  Medium|  Milestone:  Tor:
  |  0.3.3.x-final
Component:  Core Tor/Tor  |Version:
 Severity:  Normal| Resolution:
 Keywords:  tor-hs, prop224, 031-stretch  |  Actual Points:
Parent ID:| Points:  4
 Reviewer:|Sponsor:  SponsorR-can
--+
Changes (by dgoulet):

 * milestone:  Tor: 0.3.2.x-final => Tor: 0.3.3.x-final


--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21693 [Core Tor/Tor]: prop224: HS descriptors do wasteful double-base64 encoding

[Tor Bug Tracker & Wiki]

#21693: prop224: HS descriptors do wasteful double-base64 encoding
--+
 Reporter:  asn   |  Owner:  asn
 Type:  task  | Status:  assigned
 Priority:  Medium|  Milestone:  Tor:
  |  unspecified
Component:  Core Tor/Tor  |Version:
 Severity:  Normal| Resolution:
 Keywords:  tor-hs, prop224, 031-stretch  |  Actual Points:
Parent ID:| Points:  4
 Reviewer:|Sponsor:  SponsorR-can
--+
Changes (by dgoulet):

 * milestone:  Tor: 0.3.3.x-final => Tor: unspecified


--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #21693 [Core Tor/Tor]: prop224: HS descriptors do wasteful double-base64 encoding (was: prop224 HS descriptors do wasteful double-base64 encoding)

[Tor Bug Tracker & Wiki]

#21693: prop224: HS descriptors do wasteful double-base64 encoding
-+
 Reporter:  asn  |  Owner:  asn
 Type:  task | Status:  assigned
 Priority:  Medium   |  Milestone:  Tor: 0.3.1.x-final
Component:  Core Tor/Tor |Version:
 Severity:  Normal   | Resolution:
 Keywords:  tor-hs, prop224  |  Actual Points:
Parent ID:   | Points:  4
 Reviewer:   |Sponsor:  SponsorR-can
-+
Changes (by dgoulet):

 * keywords:  tor-hs prop224 => tor-hs, prop224
 * owner:   => asn
 * status:  new => assigned


--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs