Re: [tor-dev] Proposal 350: A phased plan to remove TAP onion keys

[Nick Mathewson]

And also: If you like, you can see a rendered version of this proposal
at https://spec.torproject.org/proposals/350-remove-tap.html .
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Proposal 350: A phased plan to remove TAP onion keys

[Nick Mathewson]

```
Filename: 350-remove-tap.md
Title: A phased plan to remove TAP onion keys
Author: Nick Mathewson
Created: 31 May 2024
Status: Open
```

## Introduction

Back in [proposal 216], we introduced the `ntor` circuit extension handshake.
It replaced the older `TAP` handshake, which was badly designed,
and dependent on insecure key lengths (RSA1024, DH1024).

With the [final shutdown of v2 onion services][hsv2-deprecation],
there are no longer any supported users of TAP anywhere in the Tor protocols.
Anecdotally, a relay operator reports that fewer than
one handshake in 300,000 is currently TAP.
(Such handshakes are presumably coming from long-obsolete clients.)

Nonetheless, we continue to bear burdens from TAP support.
For example:
  - TAP keys compose a significant (but useless!) portion
of directory traffic.
  - The TAP handshake requires cryptographic primitives
used nowhere else in the Tor protocols.

Now that we are implementing [relays in Arti],
the time is ripe to remove TAP.
(The only alternative is to add a useless TAP implementation in Arti,
which would be a waste of time.)

This document outlines a plan to completely remove the TAP handshake,
and its associated keys, from the Tor protocols.

This is, in some respects, a modernized version of [proposal 245].

## Migration plan

Here is the plan in summary;
we'll discuss each phase below.

- Phase 1: Remove TAP dependencies
  - Item 1: Stop accepting TAP circuit requests.
  - Item 2: Make TAP keys optional in directory documents.
  - Item 3: Publish dummy TAP keys.
- Phase 2: After everybody has updated
  - Item 1: Allow TAP-free routerdescs at the authorities
  - Item 2: Generate TAP-free microdescriptors
- Phase 3: Stop publishing dummy TAP keys.

Phase 1 can begin immediately.

Phase 2 can begin once all supported clients and relays have upgraded
to run versions with the changes made in Phase 1.

Phase 3 can begin once all authorities have made the changes
described in phase 2.


### Phase 1, Item 1: Stop accepting TAP circuit requests.

(All items in phase 1 can happen in parallel.)

Immediately, Tor relays should stop accepting TAP requests.
This includes all CREATE cells (not CREATE2),
and any CREATE2 cell whose type is TAP (0x).

When receiving such a request,
relays should respond with DESTROY.
Relays MAY just drop the request entirely, however,
if they find that they are getting too many requests.

Such relays must stop reporting `Relay=1`
among their supported protocol versions.
(This protocol version is not currently required or recommended.)

> If this proposal is accepted,
> we should clarify the protocol version specification
> to say that `Relay=1` specifically denotes TAP.


### Phase 1, Item 2: Make TAP keys optional in directory documents.

(All items in phase 1 can happen in parallel.)

In C tor and Arti, we should make the `onion-key` entry
and the `onion-key-crosscert` entry optional.
(If either is present, the other still must be present.)

When we do this, we should also modify the authority code
to reject any descriptors that do not have these fields.
(This will be needed so that existing Tor instances do not break.)

In the microdescriptor documents format, we should make
the _object_ of the `onion-key` element optional.
(We do not want to make the element itself optional,
since it is used to delimit microdescriptors.)

We use new protocol version flags (Desc=X, Routerdesc=Y)
to note the ability to parse these documents.

### Phase 1, Item 3: Publish dummy TAP keys

(All items in phase 1 can happen in parallel.)

Even after step 2 is done,
many clients and relays on the network
will still require TAP keys to be present in directory documents.
Therefore, we can't remove those keys right away.

Relays, therefore, must put _some_ kind of RSA key
into their `onion-key` field.

I'll present three designs on what relays should do.
We should pick one.

>  Option 1 (conservative)
>
> Maybe, we should say that a relay
> should generate a TAP key, generate an onion-key-crosscert,
> and then discard the private component of the key.

>  Option 2 (low-effort)
>
> In C tor, we can have relays simply proceed as they do now,
> maintaining TAP private keys and generating crosscerts.
>
> This has little real risk beyond what is described in Option 1.

>  Option 3 (nutty)
>
> We _could_ generate a global, shared RSA1024 private key,
> to be used only for generating onion-key-crosscerts
> and placing into the onion-key field of a descriptor.
>
> We would say that relays publishing this key MUST NOT
> actually handle any TAP requests.
>
> The advantage of this approach over Option 1
> would be that we'd see gains in our directory traffic
> immediately, since all identical onion keys would be
> highly compressible.
>
> The downside here is that any client TAP requests
> sent to such a relay would be decryptable by anybody,
> wh

Re: [tor-dev] Key Blinding Secrets

[Nick Mathewson]

On Tue, Apr 30, 2024 at 8:07 AM Bellebaum, Thomas
 wrote:
>
> Hello everyone,
>
> I am a researcher currently looking into different schemes for what you call 
> Keyblinding in the rendevouz spec.

Hello and welcome!

> https://spec.torproject.org/rend-spec/keyblinding-scheme.html
>
> I noticed that your description there mentiones a secret `s` to be hashed 
> into the blinding factor, and have a few questions about it:
>
> 1. Is this secret currently being used / intended to be used? If so, how?

Nope, nothing is using it or setting it right now.

> 2. What kinds of security (formally or informally) would you expect from 
> using a secret in the derivation process? For example, do you just require 
> that someone without `s` cannot look up the service, or is this also meant as 
> a way of ensuring that HSDir nodes cannot find correlations between services 
> and descriptors (amounting to some sort of additional censorship resistance)?

So, I worked on this design more than 10 years ago, and I am not 100%
sure I remember what we originally had in mind for `s`.

That said, I think my expectation would have been that somebody
without `s`  should not be able to look up the onion service, connect
to the onion service, *or* link services and descriptors, or link
descriptors to one another.  I don't know if we ever relied on that
latter piece though.

The reason we never built it (IIRC) is that having `KP_hs_id` public
but keeping `s` secret didn't actually achieve anything that couldn't
be achieved just as easily by keeping KP_hs_id secret.

best wishes,
-- 
Nick
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Timers in Arti?

[Nick Mathewson]

On Tue, Jan 9, 2024 at 12:58 PM Micah Elizabeth Scott
 wrote:
>
> Ah. If you want to run an onion service you'll need to keep at least a
> couple circuits open continuously for the introduction points. I'm not
> sure where you would meaningfully find time to deep sleep in that
> scenario. There will be ongoing obligations from the wifi/wan and tcp
> stacks. You need a continuous TCP connection to the guard, and multiple
> circuits that are not discarded as idle. Incoming traffic on those
> circuits need to be addressed quickly or clients won't be able to connect.
>
> If we were really optimizing for a low power mobile onion service
> platform we'd have a different way to facilitate introductions without a
> continuously open set of circuits, but that would also be much more
> abuse-prone.
>
> -beth

Hm. Do you know, is it possible to make network-driven wakeup events
relatively prompt?  (Like, within a second or so if a TCP stream we're
waiting on wakes up). If so, then onion services have a decent chance
of being made to work.

As for the original question about timers, it would be good to know if
the variance between scheduled wakeup and actual wakeup can be
bounded, or if there's any way to mark a timer as  high-priority vs
low-priority or something.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Proposal 347: Domain separation for certificate signing keys

[Nick Mathewson]

To see this rendered, go to
https://spec.torproject.org/proposals/347-domain-separation.html

```
Filename: 347-domain-separation.md
Title: Domain separation for certificate signing keys
Author: Nick Mathewson
Created: 19 Oct 2023
Status: Open
```

## Our goal

We'd like to be able to use the "family key" from proposal 321 as a
general purpose signing key, to authenticate other things than the
membership of a family.  For example, we might want to have a
challenge/response mechanism where the challenger says, "If you want
to log in as the owner of the account corresponding to this family,
sign the following challenge with your key.  Or we might want to
have a message authentication scheme where an operator can
sign a message in a way that proves key ownership.

We _might_ also like to use relay identity keys or onion service
identitiy keys for the same purpose.

## The problem

When we're using a signing key for two purposes, it's important
to perform some kind of domain separation so that documents
signed for one purpose can't be mistaken for documents signed for the
other.

For example, in the challenge/response example, it would be bad if
the challenger could provide a challenge string that would cause the
signer to inadvertently authenticate an incorrect family.

These keys are currently used in some places with no
personalization. Their signature format is as described in
`cert-spec.txt`, which says:

>  The signature is created by signing all the fields in
>  the certificate up until "SIGNATURE" (that is, signing
>  sizeof(ed25519_cert) - 64 bytes).



## One solution

This one is pretty easy: we would extend `cert-spec` as follows.


> ### Using signing keys for other purposes.
>
> Other protocols may introduce other uses for the signing keys
> in these certificates other than those specified here.  When
> they do, they MUST ensure that the documents being signed
> cannot be confused with the certificate bodies of this document.
>
> In some existing cases in the Tor protocols, we achieve this by
> specifying an ASCII prefix string that must be prepended to the
> other protocol's signed object before it is signed.
>
> For future protocols, we recommend that this be done by
> specifying that the signing key is to be used to sign a
> cSHAKE digest (or other secure customizable digest)
> of the other protocol's signed object, using
> a customization string unique to the other protocol.

We would also make this amendment:

> ### Future versions of this specification
>
> In order to maintain the domain separation that currently exists
> between the signatures on these certificates and other documents
> signed with the same keys, it suffices (for now!) that these
> certificates begin with the version byte `[01]`, whereas
> the other documents are in printable ASCII, which never
> includes `[01]`.
>
> Future versions of this specification SHOULD move to using
> an ed25519-prehash construction, using a customizable hash
> with built-in domain separation.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Proposal 346: Clarifying and extending the use of protocol versioning

[Nick Mathewson]

Note: we've updated our specs website!  You can read a rendered
version of this proposal at
https://spec.torproject.org/proposals/346-protovers-again.html .

```
Filename: 346-protovers-again.md
Title: Clarifying and extending the use of protocol versioning
Author: Nick Mathewson
Created: 19 Oct 2023
Status: Open
```

# Introduction

In proposal 264, we introduced "subprotocol versions" as a way to
independently version different pieces of the Tor protocols, and
communicate which parts of the Tor protocols are supported,
recommended, and required.

Here we clarify the semantics of individual subprotocol versions, and
describe more ways to use and negotiate them.

# Semantics: Protocol versions are feature flags

One issue we left unclarified previously is the relationship between
two different versions of the same subprotocol.  That is, if we know
the semantics of (say) `Handshake=7`, can we infer anything about a
relay that supports `Handshake=8`?  In particular, can we infer that
it supports all of the same features implied by `Handshake=7`?  If we
want to know "does this relay support some feature supported by
`Handshake=7`", must we check whether it supports `Handshake=7`, or
should we check `Handshake=x for any x≥7`?

In this proposal, we settle the question as follows: subprotocol
versions are flags.  They do not have any necessary semantic
relationship between them.

We reject the `≥` interpretation for several reasons:
  * It's tricky to implement.
  * It prevents us from ever removing a feature.
  * It requires us to implement features in the same order across
all Tor versions.

## ...but sometimes a flag _is_ a version!

There _are_ some places in our protocol (notably: directory authority
consensus methods, and channel protocol versions) where there _is_ a
semantic relationship between version numbers.  Specifically: "higher
numbers are already better".  When parties need to pick a one of
_these_ versions, they always pick the highest version number
supported by enough of them.

When this kind of _real_ version intersects with the "subprotocol
versions" system, we use the same numbers:

 * `Link` subprotocols correspond one-to-one with the version numbers
   sent in a VERSIONS cell.
 * `Microdesc` and `Cons` subprotocols correspond to a _subset_ of
   the version numbers of consensus methods.

## How to document subprotocol versions

When describing a subprotocol, we should be clear what relationship,
if any, exists between its versions and any versions negotiated
elsewhere in the specifications.

Unless otherwise documented, all versions can be in use _at the same
time_: if only one can exist at once (on a single circuit, a single
document, etc), this must be documented.

> Implication: This means that we must say (for example) that you
> can't use Link=4 and Link=5 on the same channel.

# Negotiating protocol versions in circuit handshakes.

Here we describe a way for a client to opt into features as part of
its circuit handshake, in order to avoid proliferating negotiating
extensions.

## Binary-encoding protocol versions.

We assign a one-byte encoding for each protocol version number,
ordered in the same way as in tor-spec.

| Protocol | Id  |
| ---  | --- |
| Link | 0  |
| LinkAuth | 1  |
| Relay| 2  |
| DirCache | 3  |
| HSDir| 4  |
| HSIntro  | 5  |
| HSRend   | 6  |
| Desc | 7  |
| MicroDesc| 8  |
| Cons | 9  |
| Padding  | 10 |
| FlowCtrl | 11 |
| Conflux  | 12 |
| Datagram | 13 |

> Note: This is the same encoding used in the [walking onions
> proposal][prop323].  It takes its order from the ordering of
> protocol versions in
> [tor-spec][subprotocol-versioning]
> and matches up with the values defined in for `protocol_type_t` in C
> tor's `protover.h`.

## Requesting an opt-in circuit feature

When a client wants to request a given set of features, it sends an
`ntor_v3` extension containing:

```
struct subproto_request {
  struct req[..]; // up to end of extension
}

struct req {
  u8 protocol_id;
  u8 protovol_version;
}
```

> Note 1: The above format does not include any parameters for each
> req.  Thus, if we're negotiating an extension that requires a client-
> supplied parameter, it may not be appropriate to use this
> request format.
>
> Note 2: This proposal does not include any relay extension
> acknowledging support.  In the case of individual subprotocols, we could
> later say "If this subprotocol is in use, the relay MUST also send
> extension foo".
>
> Note 3: The existence of this extension does not preclude the later
> addition of other extensions to negotiate featuress differently, or
> to do anything else.

Each `req` entry corresponds to a single subprotocol version. A client
MUST NOT send any `req` entry unless:
  * That subprotocol version is advertised by the relay,
  * OR that subprotocol version is listed as required f

Re: [tor-dev] Two features that would help load-balanced bridges

[Nick Mathewson]

On Wed, May 24, 2023 at 9:10 PM David Fifield  wrote:
>
> Linus Nordberg and I will have a paper at FOCI this summer on the
> special way we run tor on the Snowflake bridges to permit better
> scaling. It discusses the two workarounds from the post below, namely a
> shim for predictable ExtORPort auth, and disabling onion key rotation.
> This setup has been in place on Snowflake bridges since January 2022.
> About 2.5% of Tor users (all users, not just bridge users) access Tor
> using Snowflake, so it's not a niche use case even if it's just us.
>
> One of the reviewers asked if there was a chance changes might be made
> in tor that make our workarounds unnecessary. Is there anything to say
> to this question? Might tor get a feature to control ExtORPort
> authentication or onion key rotation, or is something that's planned to
> stay as it is in favor of Arti? (Arti will probably remove the need for
> the load-balanced multi-tor configuration, which will also remove the
> need to disable onion key rotation, though better control over ExtORPort
> auth could still be useful for running server PTs that are not child
> processes of arti.)

I can't speak definitively about the plans for the C tor
implementation. My understanding is that they are hoping to keep new
features there to a minimum, so I don't know if this would be
something they can do.

In Arti, we're a lot more open to that kind of stuff. We're not
planning to start a relay implementation till next year at the
earliest, but please remind us when we start working on ExtOrPort and
related things and we can try to figure out a design that works.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Is Arti expected to have better multi-CPU support than C-tor?

[Nick Mathewson]

On Tue, Mar 7, 2023 at 4:07 PM David Fifield  wrote:

> Linus Nordberg and I are preparing a submission for FOCI about the
> special way we run tor on the Snowflake bridge. We run many tor
> processes with the same identity and onion keys, because otherwise tor
> being limited to one CPU would be the main bottleneck.
>
> I'm writing to fact-check a claim about Arti and how we hope the current
> complicated procedure will not be needed in the future:
>
> The first and most important bottleneck to overcome is the
> single-threaded nature of the Tor implementation.² A single Tor
> process is limited to one CPU core: once Tor hits 100% CPU, the
> performance of the bridge is capped, no matter the speed of the
> network connection or the number of CPU cores
>
> ²We expect that Arti, the in-progress reimplementation of Tor,
> will be natively multi-threaded, and remove this primary
> complication.
>
> Is this correct? Is a relay that uses a future version of Arti expected
> to be able to use all its CPU resources?
>
>
Yes, that's right.  There is no "main thread" in Arti; it's written in an
asynchronous task-oriented style, and we use a runtime written in Rust
(Tokio by default, but we abstract them so you can swap them out) to
schedule tasks across multiple threads.

That said, we have spent approximately zero time so far tuning this
multithreading, and I'd be surprised if it scales perfectly the first
time.  Our first opportunity to show off here will be when we get onion
service support later in this year.

cheers,
-- 
Nick
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] The future of Tor client software?

[Nick Mathewson]

On Mon, Feb 13, 2023 at 9:40 AM Nick Mathewson  wrote:

>
>
> Right now, we're starting a working group of interested people to talk
> about getting all of these APIs right. You can see an initial thread at
> https://forum.torproject.net/t/defining-an-interface-to-arti/6432 with
> links to different design sketches; pretty soon we hope that there will be
> a new subforum to work on the issue and discuss more.  I'll follow up with
> a link once there is (unless somebody else does).
>
>
>
The forum section is at : https://forum.torproject.net/c/arti-development/30
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] The future of Tor client software?

[Nick Mathewson]

On Mon, Feb 13, 2023 at 4:20 AM Steven Engler  wrote:

> Hi tor-dev,
>
> In the past, there were generally two options for supporting Tor in an
> application.
>
> 1. Require the user to install the tor daemon (ex: apt install tor, Tor
> Expert Bundle, etc) and configure the application to use its SOCKS proxy.
>
> 2. Bundle the tor binary with the application (ex: Tor Browser) and have
> the application use the app-specific tor process as the SOCKS proxy.
>
> I'm not clear on how this changes in an Arti world. Arti currently has a
> Rust Tor client library for applications, and a CLI application to run a
> SOCKS proxy. Is there any plan to offer an Arti daemon (ex: a systemd
> service) for clients like with the current tor package? In a future
> where Arti replaces the Tor client and relay code, or when Arti is
> recommended for all client-related use cases, will there continue to be
> a Tor daemon with client support?
>

Hi, Steven!

We're looking at a timeline more or less like this:

In the short term, both implementations will coexist.

In the medium term,  we will deprecate the C Tor client implementation.
This will not be until some time after Arti is a *complete* tor client with
support for onion services, with a *complete *embedding/FFI/RPC solution.

In the long term, we intend that Arti will replace the C tor implementation
completely.  This means we'll have to implement relays and directory
authorities with Arti.  (We won't be starting this for a year at least, and
it will probably take some while to achieve.)[1]


We intend that Arti will be usable in multiple deployment environments,
including as a daemon that they talk to over an RPC-like protocol
(analogous to the current control port) and as an embedded library.  We
want applications written in most any reasonable language to be able to use
Arti, and to be more or less agnostic about whether they're doing it with
an in-process library or over an RPC channel.  Of course, this involves a
lot of API work, and there will be a lot of design and prototyping to do.
Our plan is to have this part ready for experimental use this year.

Right now, we're starting a working group of interested people to talk
about getting all of these APIs right. You can see an initial thread at
https://forum.torproject.net/t/defining-an-interface-to-arti/6432 with
links to different design sketches; pretty soon we hope that there will be
a new subforum to work on the issue and discuss more.  I'll follow up with
a link once there is (unless somebody else does).



[1]  (A note for the anxious: We don't plan to stop C Tor support
immediately when any of these phases is done, but we do plan to do it after
a reasonable time for migration and making sure that the migration issues
are solved.  We haven't committed to a schedule for this. Whatever we _do_
decide on will definitely be annoyingly long from some points of view, and
annoyingly short from others.)

best wishes,
-- 
Nick
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Proposal 342: Decouple hs_interval and SRV lifetime

[Nick Mathewson]

On Tue, Jan 24, 2023 at 9:02 AM David Goulet  wrote:

> On 23 Jan (13:31:49), Nick Mathewson wrote:
> > On Tue, Jan 10, 2023 at 8:22 AM Nick Mathewson 
> wrote:
> >
> > > ```
> > > Filename: 342-decouple-hs-interval.md
> > > Title: Decoupling hs_interval and SRV lifetime
> > > Author: Nick Mathewson
> > > Created: 9 January 2023
> > > Status: Draft
> > > ```
> > >
> > > # Motivation and introduction
> > >
> > >
> > I think there's another issue to address here too: the offset from the
> Unix
> > Epoch at which the first Time Period begins.  According to rend-spec-v3,
> >
> > "we want our time periods to start at 12:00UTC every day, so
> > we subtract a "rotation time offset" of 12*60 minutes from the number of
> > minutes since the epoch, before dividing by the time period (effectively
> > making "our" epoch start at Jan 1, 1970 12:00UTC)."
> >
> > But this isn't exactly what the C Tor implementation does.  In
> > `hs_get_time_period_num(),` it defines the offset as
> > `sr_state_get_phase_duration()`, which is tied to the voting interval and
> > the constant SHARED_RANDOM_N_ROUNDS (which is 12).
> >
> > David, do you have any thoughts on the right solution here?  Some options
> > are:
> >   * We could document the current behavior.
> >   * We could add a consensus parameter for the time period offset.
> >   * We could define the time period offset as exactly 12 hours in all
> > cases. (I guess this would break test networks though?)
> >   * Something else?
>
> My intuition here would be to document current behavior. This shared random
> ritual is tied to the voting protocol (interval) because it has these
> commit +
> reveal phases thus using the voting interval between phase rounds is
> foundational.
>
> And so, I would keep those two tied and simply document.
>
> What we can think of is to add consensus parameters for how many rounds per
> phase instead of these hardcoded values in our C-tor code but unclear to me
> what it would give us in the long run. But for the interval, I would keep
> the
> voting one. We get TestingNetwork for free also with this.
>
> Whatever we do, the very important piece here is that we can't end up with
> a
> protocol taking more time than our MinUptimeHidServDirectoryV2 value
> (minimum
> uptime for a relay before becoming an HSDir).
>
> And so let say one day we change the voting interval to every 4 hours then
> we
> end up with 24 rounds of voting to complete the commit + reveal phases
> meaning
> a total of 96 hours (which is current value of MinUptimeHidServDirectoryV2)
> thus borderline no good.
>
> (Maybe something to keep in mind for arti-relay authorities to check
> on...).
>
> Hope this help with your question?
>
>
Sure!  It sounds to me that we should then change rend-spec to say
something like this:

 "We want our time periods to start at a regular offset from the SRV voting
schedule, so
we subtract a "rotation time offset" of 12 voting periods from the number of
minutes since the epoch, before dividing by the time period (effectively
making "our" epoch start at Jan 1, 1970 12:00UTC when the voting period is
1 hour.)"

How would that be?

-- 
Nick
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Proposal 342: Decouple hs_interval and SRV lifetime

[Nick Mathewson]

On Tue, Jan 10, 2023 at 8:22 AM Nick Mathewson  wrote:

> ```
> Filename: 342-decouple-hs-interval.md
> Title: Decoupling hs_interval and SRV lifetime
> Author: Nick Mathewson
> Created: 9 January 2023
> Status: Draft
> ```
>
> # Motivation and introduction
>
>
I think there's another issue to address here too: the offset from the Unix
Epoch at which the first Time Period begins.  According to rend-spec-v3,

"we want our time periods to start at 12:00UTC every day, so
we subtract a "rotation time offset" of 12*60 minutes from the number of
minutes since the epoch, before dividing by the time period (effectively
making "our" epoch start at Jan 1, 1970 12:00UTC)."

But this isn't exactly what the C Tor implementation does.  In
`hs_get_time_period_num(),` it defines the offset as
`sr_state_get_phase_duration()`, which is tied to the voting interval and
the constant SHARED_RANDOM_N_ROUNDS (which is 12).

David, do you have any thoughts on the right solution here?  Some options
are:
  * We could document the current behavior.
  * We could add a consensus parameter for the time period offset.
  * We could define the time period offset as exactly 12 hours in all
cases. (I guess this would break test networks though?)
  * Something else?

best wishes,
-- 
Nick
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Proposal 342: Decouple hs_interval and SRV lifetime

[Nick Mathewson]

On Tue, Jan 10, 2023 at 8:41 AM Holmes Wilson  wrote:

> Apologies if I missed this in the proposal, but what are the benefits of
> this beyond decoupling? Will onion service users see benefits? Or is it
> just about simplifying the code?
>

There's no immediate benefit to users; it's entirely about making the code
more well-defined in the case where hs_interval is constant.

I'm attempting to do this now since we're starting to write onion services
in Arti, and it'll be easier if there are fewer ambiguities in the spec.

cheers,
-- 
Nick
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Proposal 342: Decouple hs_interval and SRV lifetime

[Nick Mathewson]

```
Filename: 342-decouple-hs-interval.md
Title: Decoupling hs_interval and SRV lifetime
Author: Nick Mathewson
Created: 9 January 2023
Status: Draft
```

# Motivation and introduction

Tor uses shared random values (SRVs) in the consensus to determine
positions of relays within a hash ring.  Which shared random value is to
be used for a given time period depends upon the time at which that
shared random value became valid.

But right now, the consensus voting period is closely tied to the shared
random value voting cycle: and clients need to understand both of these
in order to determine when a shared random value became current.

This creates tight coupling between:
   * The voting schedule
   * The SRV liveness schedule
   * The hsdir_interval parameter that determines the length of the
 an HSDIR index

To decouple these values, this proposal describes a forward compatible
change to how Tor reports SRVs in consensuses, and how Tor decides which
hash ring to use when.


## Reporting SRV timestamps

In consensus documents, parties should begin to accept
`shared-rand-*-value` lines with an additional argument, in the format
of an IsoTimeNospace timestamp (like "1985-10-26T00:00:00").  When
present, this timestamp indicates the time at which the given shared
random value first became the "current" SRV.

Additionally, we define a new consensus method that adds these
timestamps to the consensus.

We specify that, in the absence of such a timestamp, parties are to
assume that the `shared-rand-current-value` SRV became "current" at the
first 00:00 UTC on the UTC day of the consensus's valid-after timestamp,
and that the `shard-rand-previous-value` SRV became "current" at 00:00
UTC on the previous UTC day.


## Generalizing HSDir index scheduling.

Under the current HSDir design, there is one SRV for each time period,
and one time period for which each SRV is in use.  Decoupling
`hsdir_interval` from 24 hours will require that we change this notion
slightly.

We therefore propose this set of generalized directory behavior rules,
which should be equivalent to the current rules under current
parameters.

The calculation of time periods remains the same (see `rend-spec-v3.txt`
section `[TIME PERIODS]`).

A single SRV is associated with each time period: specifically, the SRV
that was "current" at the start of the time period.

There is a separate hash ring associated with each time period and its
SRV.

Whenever fetching an onion service descriptor, the client uses the hash
ring for the time period that contains the start of the liveness
interval of the current consensus.  Call this the "Consensus" time period.

Whenever uploading an onion service descriptor, the service uses _two or
three_ hash rings:
  * The "consensus" time period (see above).
  * The immediately preceding time period, if the SRV to calculate that
hash ring is available in the consensus.
  * The immediately following time period, if the SRV to calculate that
hash ring is available in the consensus.

(Under the current parameters, where `hsdir_interval = SRV_interval`,
there will never be more than two possible time periods for which the
service can qualify.)

## Migration

We declare that, for at least the lifetime of the C tor client, we will
not make any changes to the voting interval, the SRV interval, or the
`hsdir_interval`.  As such, we do not need to prioritize implementing
these changes in the C client: we can make them in Arti only.

## Issues left unsolved

There are likely other lingering issues that would come up if we try to
change the voting interval.  This proposal does not attempt to solve
them.

This proposal does not attempt to add flexibility to the SRV voting
algorithm itself.

Changing `hsdir_interval` would create a flag day where everybody using
old and new values of `hsdir_interval` would get different hash
rings. We do not try to solve that here.

## Acknowledgments

Thanks to David Goulet for explaining all of this stuff to me!
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Proposal 341: A better algorithm for out-of-sockets eviction

[Nick Mathewson]

```
Filename: 341-better-oos.md
Title: A better algorithm for out-of-sockets eviction
Author: Nick Mathewson
Created: 25 July 2022
Status: Open
```

# Introduction

Our existing algorithm for handling an out-of-sockets condition needs
improvement.  It only handles sockets used for OR connections, and
prioritizes those with more circuits.  Because of these weaknesses, the
algorithm is trivial to circumvent, and it's disabled by default with
`DisableOOSCheck`.

Here we propose a new algorithm for choosing which connections to close
when we're out of sockets.  In summary, the new algorithm works by
deciding which kinds of connections we have "too many" of, and then by
closing excess connections of each kind.  The algorithm for selecting
connections of each kind is different.



# Intuitions behind the algorithm below

We want to keep a healthy mix of connections running; favoring one kind
of connection over another gives the attacker a fine way to starve the
disfavored connections by making a bunch of the favored kind.

The correct mix of connections depends on the type of service we are
providing.  Everywhere _except_ authorities, for example, inbound
directory connections are perfectly fine to close, since nothing in our
protocol actually generates them.

In general, we would prefer to close DirPort connections, then Exit
connections, then OR connections.

The priority with which to close connections is different depending on
the connection type.  "Age of connection" or "number of circuits" may be
a fine metric for how truly used an OR connection is, but for a DirPort
connection, high age is suspicious.

# The algorithm

Define a "candidate" connection as one that has a socket, and is either
an exit stream, an _inbound_ directory stream, or an OR connection.

(Note that OR connections can be from clients, relays, or bridges. Note
that ordinary relays should not get directory streams that use sockets,
since clients always use `BEGIN_DIR` to create tunneled directory
streams.)

In all of the following, treat subtraction as saturating at zero.  In
other words, when you see "A - B" below, read it as "MAX(A-B, 0)".

## Phase 1: Deciding how many connections to close

When we are find that we are low on sockets, we pick a number of sockets
that we want to close according to our existing algorithm.  (That is, we
try to close 1/4 of our maximum sockets if we have reached our upper
limit, or 1/10 of our maximum sockets if we have encountered a failure
from socket(2).)  Call this `N_CLOSE`.

Then we decide which sockets to target based on this algorithm.

1. Consider the total number of sockets used for exit streams
   (`N_EXIT`), the total number used for _inbound_ directory streams
   (`N_DIR`), and the total number used for OR connections (`N_OR`).
   (In these calculations, we exclude connections that are already
   marked to be closed.)  Call the total `N_CONN = N_DIR + N_OR +
   N_EXIT`.  Define `N_RETAIN = N_CONN - N_CLOSE`.

2. Compute how many connections of each type are "in excess". First,
   calculate our target proportions:

* If we are an authority, let `T_DIR` = 1.  Otherwise set `T_DIR = 0.1`.
* If we are an exit or we are running an onion service, let `T_EXIT =
  2`. Otherwise let `T_EXIT = 0.1`.
* Let `T_OR` = 1.

   > TODO: Should those numbers be consensus parameters?

   These numbers define the relative proportions of connections that
   we would be willing to retain retain in our final mix.  Compute a
   number of _excess_ connections of each type by calculating.

   ```
   T_TOTAL = T_OR + T_DIR + T_EXIT.
   EXCESS_DIR   = N_DIR  - N_RETAIN * (T_DIR  / T_TOTAL)
   EXCESS_EXIT  = N_EXIT - N_RETAIN * (T_EXIT / T_TOTAL)
   EXCESS_OR= N_OR   - N_RETAIN * (T_OR   / T_TOTAL)
   ```

3. Finally, divide N_CLOSE among the different types of excess
   connections, assigning first to excess directory connections, then
   excess exit connections, and finally to excess OR connections.

   ```
   CLOSE_DIR = MIN(EXCESS_DIR, N_CLOSE)
   N_CLOSE := N_CLOSE - CLOSE_DIR
   CLOSE_EXIT = MIN(EXCESS_EXIT, N_CLOSE)
   N_CLOSE := N_CLOSE - CLOSE_EXIT
   CLOSE_OR = MIN(EXCESS_OR, N_CLOSE)
   ```

We will try to close `CLOSE_DIR` directory connections, `CLOSE_EXIT`
exit connections, and `CLOSE_OR` OR connections.

## Phase 2: Closing directory connections

We want to close a certain number of directory connections.  To select
our targets, we sort first by the number of directory connections from
a similar address (see "similar address" below) and then by their age,
preferring to close the oldest ones first.

> This approach defeats "many requests from the same address" and "Open
> a connection and hold it open, and do so from many addresses".  It
> doesn't do such a great job with defeating "open and close frequently
> and do so on many addresses."

> Note that fal

Re: [tor-dev] Shrink Tor binary size

[Nick Mathewson]

On Mon, Jun 27, 2022 at 5:03 AM Sergey Ponomarev  wrote:

> I’m working on a firmware for routers based on OpenWrt  and it needs
> Tor out of the box for NAT punching i.e. SSH and Web admin access. It
> will expose a Single Onion Service i.e. not "hidden" with just 3 hop
> for a better performance.
>

Hi, Sergey! This sounds like a neat project.

The Tor is already ported to OpenWrt but its binary is more than 2mb.
> For 16mb routers this is not critical but a lot and I need to keep
> space.
> So is it possible to compile Tor without some features?
> As far as I understood from autoconf I can disable Relay functionality.
> But maybe I can also disable some admin api parts and SOCKS proxy.
> Also OpenWrt out of the box has WolfSSL so is it possible to compile
> tor with it instead of OpenSSL?
>
>
So, you've seen the options  `--disable-module-relay` and
`--disable-module-dirauth`, and those are the only ones there are at the
moment.  We don't support WolfSSL, but you could try it out (it supposedly
has an OpenSSL compatibility layer?) and see if it works.


> Another problem is files sizes inside of /var/lib/tor/:
>  20,442 cached-certs
> 2,303,443 cached-microdesc-consensus
> 8,725,598 cached-microdescs.new
>3,531 state
>
> I think that all these files may be minimized. E.g. cached-certs
> contains keys in PEM format which is just Base64 form of DER.
> So switching to plain DER certs can significantly reduce size.
>

That isn't likely to happen at this point in the lifecycle of the current
directory system; your best bet here is to look in to some kind of
compressing filesystem.  You _might_ be able to save a little space by
tinkering with the definition of TOLERATE_MICRODESC_AGE and setting it to 3
days instead of 7, but I don't know how well that would work.

I do want us to move to a world where we need a lot less storage to run a
client, but that will require an implementation of the "Walking Onions"
design [1], which probably won't be possible until we've got a full relay
and authority implementation in Arti [2].

[1]
https://gitlab.torproject.org/dgoulet/torspec/-/blob/master/proposals/323-walking-onions-full.md
[2] https://arti.torproject.org


> As a side question: is it possible to make a small relay proxy that
> can work part time when I sleep?
> Imagine that each router already has a Tor. This is potentially
> thousands of relay nodes and all of them have a motivation to support
> the network which they are using themselves.
>
> But users don’t want to lose bandwidth. I see that I can set some
> traffic or bandwidth limits.
> But maybe I can write some script that will enable or disable the
> relay by schedule in the evening.
> As far I know relays must be always online so this will make the relay
> unstable and it won’t be used by Tor.
> Is it technically possible?
>

It's a cool idea, but it's not something we support with our current
design.  To deliver good performance to users, we need relays to have good,
fast, reliable connections.  Having a relay that drops off the network once
a day, or that changes its capabilities too frequently, doesn't really work
so well (as I understand it).  This is an area where others understand the
design space better, though, so maybe somebody will figure something out.

cheers,
-- 
Nick
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Proposal 340: Packed and fragmented relay messages

[Nick Mathewson]

```
Filename: 340-packed-and-fragmented.md
Title: Packed and fragmented relay messages
Author: Nick Mathewson
Created: 31 May 2022
Status: Open
```

# Introduction

Tor sends long-distance messages on circuits via _relay cells_.  The
current relay cell format allows one _relay message_ (e.g., "BEGIN" or
"DATA" or "END") per relay cell. We want to relax this 1:1 requirement,
between messages and cells, for two reasons:

 * To support relay messages that are longer than the current 498-byte
   limit.  Applications would include wider handshake messages for
   postquantum crypto, UDP messages, and SNIP transfer in walking
   onions.

 * To transmit small messages more efficiently.  Several message types
   (notably `SENDME`, `XON`, `XOFF`, and several types from
   [proposal 329](./329-traffic-splitting.txt)) are much smaller than
   the relay cell size, and could be sent comparatively often.

In this proposal, we describe a way to decouple relay cells from relay
messages.  Relay messages can now be packed into multiple cells or split
across multiple cells.

This proposal combines ideas from
[proposal 319](./319-wide-everything.md) (fragmentation) and
[proposal 325](./325-packed-relay-cells.md) (packed cells).  It requires
[ntor v3](./332-ntor-v3-with-extra-data.md) and prepares for
[next-generation relay cryptography](./308-counter-galois-onion).

## A preliminary change: Relay encryption, version 1.5

We are fairly sure that, whatever we do for our next batch of relay
cryptography, we will want to increase the size of the data used to
authenticate relay cells to 128 bits.  (Currently it uses a 4-byte tag
plus 2 bytes of zeros.)

To avoid proliferating formats, I'm going to suggest that we make the
other changes in this proposal changes concurrently with a change in our
relay cryptography, so that we do not have too many incompatible cell
formats going on at the same time.

The new format for a decrypted relay _cell_ will be:

   recognized [2 bytes]
   digest [14 bytes]
   body   [509 - 16 = 493 bytes]

"Digest" and "recognized" are computed as before; the only difference
is that they occur _before_ the rest of the cell, and that "digest" is
truncated to 14 bytes instead of 4.

If we are lucky, we won't have to build this encryption at all, and we
can just move to some version of GCM-UIV or other RPRP that reserves 16
bytes for an authentication tag or similar cryptographic object.

## New relay message packing

Relay _messages_ now follow the following format:

  Header
command   u8
lengthu16
stream_id u16
  Body
data  u8[length]

We require that "command" is never 0.

Messages can be split across relay cells; multiple messages can occur in
a single relay cell.  We enforce the following rules:

  * Headers may not be split across cells.
  * If a 0 byte follows any relay message, there are no more messages in
that cell.
  * A relay cell may not be "empty": it must have at least some part of
some message.

Unless specified elsewhere, **all** message types may be packed, and
**all** message types may be fragmented.

Every command has an associated maximum length for its messages.  If not
specified elsewhere, the maximum length for every message is 498 bytes
(for legacy reasons).

Receivers MUST validate that headers are well-formed and have valid
lengths while handling the cell in which the header is encoded.  If the
header is invalid, the receiver must destroy the circuit.

### Some examples


## Negotiation

This message format requires a new `Relay` subprotocol version to
indicate support.  If a client wants to indicate support for this
format, it sends the following extension as part of its `ntor3`
handshake:

   RELAY_PROTOCOL
 versionu8

The `version` field is the `Relay` subprotocol version that the client
wants to use. The relay must send back the same extension in its ntor3
handshake to acknowledge support.

## Migration

We add a consensus parameter, "streamed-relay-messages", with default
value 0, minimum value 0, and maximum value 1.

If this value is 0, then clients will not (by default) negotiate this
relay protocol.  If it is 1, then clients will negotiate it when relays
support it.

For testing, clients can override this setting.  Once enough relays
support this proposal, we'll change the consensus parameter to 1.
Later, we'll change the default to 1 as well.

## Packing decisions

We specify the following greedy algorithm for making decisions about
fragmentation and packing.  Other algorithms are possible, but this one
is fairly simple, and using it will help avoid distinguishability
issues:

Whenever a client or relay is about to send a cell that would leave
at least 32 bytes unused in a relay cell, it checks to see whether there
is any pending data to be sent in the same circuit (in a data cell).  If
there is, then it adds a DATA message to the end

Re: [tor-dev] Proposal 338: Use an 8-byte handshake in NETINFO cells

[Nick Mathewson]

On Mon, Mar 14, 2022 at 3:18 PM Jim Newsome  wrote:

> On 3/14/22 11:44, Nick Mathewson wrote:
>
> 
>
>  > Currently Tor relays use a 4-byte timestamp (in seconds since the Unix
>  > epoch) in their NETINFO cells.  Notoriously, such a timestamp will
>  > overflow on 19 January 2038.
>  >
>  > Let's get ahead of the problem and squash this issue now, by expanding
>  > the timestamp to 8 bytes. (8 bytes worth of seconds will be long enough
>  > to outlast the Earth's sun.)
>
> 
>
> With all those extra bits, would there be any value to using a more
> granular time measure? e.g. microseconds?
>


I don't think so, necessarily.  We aren't doing full NTP here; in fact,
we're just trying to detect clock skew that's big enough to break Tor.
(Like, on the order of hours.)  I don't think we'll get anything useful
out of sub-second observations.


If not, would it be worth saving some bytes and only expanding to 5
> bytes? (I know; it *feels* wrong, but I can't think of much real downside)
>

 Hm, possibly.  One downside is that 5-byte decoder/encoder functions
aren't exactly common, so we'd require everybody to build one.  (Or to do
something like
  u8 ts_high;
  u32 ts_low;
)

Since the extra 3 bytes are only used once per connection attempt, I'm
pretty comfortable letting them be useless until 36812 CE or whenever.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Proposal 338: Use an 8-byte handshake in NETINFO cells

[Nick Mathewson]

```
Filename: 338-netinfo-y2038.md
Title: Use an 8-byte timestamp in NETINFO cells
Author: Nick Mathewson
Created: 2022-03-14
Status: Open
```

# Introduction

Currently Tor relays use a 4-byte timestamp (in seconds since the Unix
epoch) in their NETINFO cells.  Notoriously, such a timestamp will
overflow on 19 January 2038.

Let's get ahead of the problem and squash this issue now, by expanding
the timestamp to 8 bytes. (8 bytes worth of seconds will be long enough
to outlast the Earth's sun.)

# Proposed change

I propose adding a new link protocol version.  (The next one in
sequence, as of this writing, is version 6.)

I propose that we change the text of tor-spec section 4.5 from:

```
  TIME   (Timestamp) [4 bytes]
```

to

```
 TIME   (Timestamp) [4 or 8 bytes *]
```

and specify that this field is 4 bytes wide on link protocols 1-5, but 8
bytes wide on link protocols 6 and beyond.

# Rejected alternatives

Our protocol specifies that parties MUST ignore extra data at the end of
cells. Therefore we _could_ add additional data at the end of the
NETINFO cell, and use that to store the high 4 bytes of the timestamp
without having to increase the link protocol version number.  I propose
that we don't do that: it's ugly.

As another alternative, we could declare that parties must interpret the
timestamp such that its high 4 bytes place it as close as possible to
their current time.  I'm rejecting this kludge because it would give
confusing results in the too-common case where clients have their clocks
mis-set to Jan 1, 1970.

# Impacts on our implementations

Arti won't be able to implement this change until it supports connection
padding (as required by link protocol 5), which is currently planned for
the next Arti milestone (1.0.0, scheduled for this fall).

If we think that that's a problem, or if we want to have support for
implementations without connection padding in the future, we should
reconsider this plan so that connection padding support is independent
from 8-byte timestamps.

# Other timestamps in Tor

I've done a cursory search of our protocols to see if we have any other
instances of the Y2038 problem.

There is a 4-byte timestamp in `cert-spec`, but that one is an unsigned
integer counting _hours_ since the Unix epoch, which will keep it from
wrapping around till 478756 C.E. (The rollover date of "10136 CE"
reported in `cert-spec` is wrong, and seems to be based on the
misapprehension that the counter is in *minutes*.)

The v2 onion service protocol has 4-byte timestamps, but it is
thoroughly deprecated and unsupported.

I couldn't find any other 4-byte timestamps, but that is no guarantee:
others should look for them too.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] proposal 328: consensus parameters

[Nick Mathewson]

On Wed, Nov 17, 2021 at 4:39 PM nusenu  wrote:

> Hi,
>
>
> https://gitlab.torproject.org/tpo/core/torspec/-/blob/main/proposals/328-relay-overload-report.md
> > For DNS timeouts, the X and Y are consensus parameters
> > (overload_dns_timeout_scale_percent and overload_dns_timeout_period_secs)
> > defined in param-spec.txt.
>
> should one be able to find these parameters
> (overload_dns_timeout_scale_percent and overload_dns_timeout_period_secs)
> on
> https://consensus-health.torproject.org/#consensusparams
> or in
> https://collector.torproject.org/recent/relay-descriptors/consensuses/
>
> or is this not yet deployed?
> What are the values of these parameters currently?
>
>
Every network parameter has a default value; IIUC, unless at least three
authorities are voting for some value, everybody will use the default.
It's normal for the authorities not to vote for a different value if they
all think the default is reasonable.

The default for these parameters is in param-spec.txt at
https://gitlab.torproject.org/tpo/core/torspec/-/blob/main/param-spec.txt#L194

cheers,
-- 
Nick
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Arti report 8: October 14 through October 27

[Nick Mathewson]

# Arti Report 7: October 14 through October 27

# Towards Arti 0.0.1

On Monday November 1, we're planning to put out Arti version 0.0.1.  Our goal
is to include all the features that are needed for reasonable security, and a
top-level API that isn't _so_ far from completely wrong.  Right now, it looks
like we're in good shape for that timeline.

Since the last update, we've fixed a lot of smaller issues, including:
* running out of files on OSX
* making it harder to accidentally leak DNS requests
* lots of API refactoring
* restored the ability for multiple Arti instances to share a single set of
local directories
* easier setup
* easier API for stream isolatoin
* more reliable tests for asynchronous timeout-based logic
* and more!

We're now working on documentation and usability.  There's now an example
program in
[`crates/arti-client/examples`](https://gitlab.torproject.org/tpo/core/arti/-/blob/main/crates/arti-client/examples/hyper.rs)
that uses [`hyper`](https://hyper.rs/) to download an HTTP document, and
we're adding examples elsewhere in our top-level API documentation.

# Followup from our guard implementation

Our work on guards (completed in the last round) has kicked off a new batch
of guard research.  As with other parts of the Tor protocols,
implementing our specification in Rust has exposed some weaknesses or
confusing points in our design.  In response to issues that we found,
we've opened a few issues and written a couple of proposals to improve
our code.

For example, see:
* [Proposal 
336](https://gitlab.torproject.org/tpo/core/torspec/-/blob/main/proposals/336-randomize-guard-retries.md),
  to randomize guard retry timing.
* [Proposal 
337](https://gitlab.torproject.org/tpo/core/torspec/-/blob/main/proposals/337-simpler-guard-usability.md),
  to describe a simpler formulation of our guard behavior.
* [torspec#67](https://gitlab.torproject.org/tpo/core/torspec/-/issues/67),
  to treat guards as usable more eagerly.
* [torspec#68](https://gitlab.torproject.org/tpo/core/torspec/-/issues/68),
  to improve our behavior when our primary guards are down.


# What's next

Once the next release is out, we'll turn our attention to Arti 0.1.0.
For that release, our priority target is beta testing and experimental
embedding.  We'll be working to improve performance and to stabilize a
set of stable APIs to provide a wider set of functionality for
applications that want to embed Arti in different scenarios.

We'll need your help, though: we will need feedback from everybody who's
interested in using our code, in order to make sure that we're providing
solid, usable interfaces that do what you need.  Please try it out, and
let us know what's bad or missing!

We're hoping to get Arti 0.1.0 released some time in March of 2022, with
multiple smaller update releases between now and then.  At our bi-weekly
meeting next Wednesday, we'll be prioritizing tasks for the next set of
releases, and refining our roadmap for the coming months.  Why not join
us?  Information is on the
[meeting pad](https://pad.riseup.net/p/arti-meeting-pad-keep).


# Thanks!

Thanks to Jani Monoses and Dimitris Apostolou for their patches in this
release!

And special thanks and welcome to eta, our new team member!  She's
started with Tor last Tuesday, and has hit the ground running, already
fixing substantial bugs and refactoring a bunch of tricky code.  It's
great to watch the pace and quality of our development improve as more
people join the team!
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Proposal 337: A simpler way to decide, "Is this guard usable?"

[Nick Mathewson]

```
Filename: 337-simpler-guard-usability.md
Title: A simpler way to decide, "Is this guard usable?"
Author: Nick Mathewson
Created: 2021-10-22
Status: Open
```

# Introduction

The current `guard-spec` describes a mechanism for how to behave when
our primary guards are unreachable, and we don't know which other guards
are reachable.  This proposal describes a simpler method, currently
implemented in [Arti](https://gitlab.torproject.org/tpo/core/arti/).

(Note that this method might not actually give different results: its
only advantage is that it is much simpler to implement.)

## The task at hand

For illustration, we'll assume that our primary guards are P1, P2, and
P3, and our subsequent guards (in preference order) are G1, G2, G3, and
so on.  The status of each guard is Reachable (we think we can connect
to it), Unreachable (we think it's down), or Unknown (we haven't tried
it recently).

The question becomes, "What should we do when P1, P2, and P3 are
Unreachable, and G1, G2, ... are all Unknown"?

In this circumstance, we _could_ say that we only build circuits to G1,
wait for them to succeed or fail, and only try G2 if we see that the
circuits to G1 have failed completely.  But that delays in the case that
G1 is down.

Instead, the first time we get a circuit request, we try to build one
circuit to G1.  On the next circuit request, if the circuit to G1 isn't
done yet, we launch a circuit to G2 instead.  The next request (if the
G1 and G2 circuits are still pending) goes to G3, and so on.  But
(here's the critical part!) we don't actually _use_ the circuit to G2
unless the circuit to G1 fails, and we don't actually _use_ the circuit
to G3 unless the circuits to G1 and G2 both fail.

This approach causes Tor clients to check the status of multiple
possible guards in parallel, while not actually _using_ any guard until
we're sure that all the guards we'd rather use are down.

## The current algorithm and its drawbacks

For the current algorithm, see `guard-spec` section 4.9: circuits are
exploratory if they are not using a primary guard.  If such an
exploratory circuit is `waiting_for_better_guard`, then we advance it
(or not) depending on the status of all other _circuits_ using guards that
we'd rather be using.

In other words, the current algorithm is described in terms of actions
to take with given circuits.

For Arti (and for other modular Tor implementations), however, this
algorithm is a bit of a pain: it introduces dependencies between the
guard code and the circuit handling code, requiring each one to mess
with the other.

# Proposal

I suggest that we describe an alternative algorithm for handing circuits
to non-primary guards, to be used in preference to the current
algorithm.  Unlike the existing approach, it isolates the guard logic a
bit better from the circuit logic.

## Handling exploratory circuits

When all primary guards are Unreachable, we need to try non-primary
guards.  We select the first such guard (in preference order) that is
neither Unreachable nor Pending.  Whenever we give out such a guard, if
the guard's status is Unknown, then we call that guard "Pending" until
the attempt to use it succeeds or fails.  We remember when the guard
became Pending.

> Aside: None of the above is a change from our existing specification.

After completing a circuit, the implementation must check whether
its guard is usable.  A guard is usable according to these rules:

Primary guards are always usable.

Non-primary guards are usable for a given circuit if every guard earlier
in the preference list is either unsuitable for that circuit
(e.g. because of family restrictions), or marked as Unreachable, or has
been pending for at least `{NONPRIMARY_GUARD_CONNECT_TIMEOUT}`.

Non-primary guards are unusable for a given circuit if some guard earlier
in the preference list is suitable for the circuit _and_ Reachable.

Non-primary guards are unusable if they have not become usable after
`{NONPRIMARY_GUARD_IDLE_TIMEOUT}` seconds.

If a circuit's guard is neither usable nor unusable immediately, the
circuit is not discarded; instead, it is kept (but not used) until it
becomes usable or unusable.

> I am not 100% sure whether this description produces the same behavior
> as the current guard-spec, but it is simpler to describe, and has
> proven to be simpler to implement.

## Implications for program design.

(This entire section is implementation detail to explain why this is a
simplification from the previous algorithm. It is for explanatory
purposes only and is not part of the spec.)

With this algorithm, we cut down the interaction between the guard code
and the circuit code considerably, but we do not remove it entirely.
Instead, there remains (in Arti terms) a pair of communication channels
between the circuit manager and the guard manager:

 * Whenever a guard is given to the circuit manager, the circuit manager
   receives the write end of a single-use 

[tor-dev] Proposal 336: Randomized schedule for guard retries

[Nick Mathewson]

```
Filename: 336-randomize-guard-retries.md
Title: Randomized schedule for guard retries
Author: Nick Mathewson
Created: 2021-10-22
Status: Open
```

# Introduction

When we notice that a guard isn't working, we don't mark it as retriable
until a certain interval has passed.  Currently, these intervals are
fixed, as described in the documentation for `GUARDS_RETRY_SCHED` in
`guard-spec` appendix A.1.  Here we propose using a randomized retry
interval instead, based on the same decorrelated-jitter algorithm we use
for directory retries.

The upside of this approach is that it makes our behavior in
the presence of an unreliable network a bit harder for an attacker to
predict. It also means that if a guard goes down for a while, its
clients will notice that it is up at staggered times, rather than
probing it in lock-step.

The downside of this approach is that we can, if we get unlucky
enough, completely fail to notice that a preferred guard is online when
we would otherwise have noticed sooner.

Note that when a guard is marked retriable, it isn't necessarily retried
immediately.  Instead, its status is changed from "Unreachable" to
"Unknown", which will cause it to get retried.

For reference, our previous schedule was:

```
   {param:PRIMARY_GUARDS_RETRY_SCHED}
  -- every 10 minutes for the first six hours,
  -- every 90 minutes for the next 90 hours,
  -- every 4 hours for the next 3 days,
  -- every 9 hours thereafter.

   {param:GUARDS_RETRY_SCHED} --
  -- every hour for the first six hours,
  -- every 4 hours for the next 90 hours,
  -- every 18 hours for the next 3 days,
  -- every 36 hours thereafter.
```

# The new algorithm

We re-use the decorrelated-jitter algorithm from `dir-spec` section 5.5.
The specific formula used to compute the 'i+1'th delay is:

```
Delay_{i+1} = MIN(cap, random_between(lower_bound, upper_bound))
where upper_bound = MAX(lower_bound+1, Delay_i * 3)
  lower_bound = MAX(1, base_delay).
```

For primary guards, we set base_delay to 30 seconds and cap to 6 hours.

For non-primary guards, we set base_delay to 10 minutes and cap to 36
hours.

(These parameters were selected by simulating the results of using them
until they looked "a bit more aggressive" than the current algorithm, but
not too much.)

The average behavior for the new primary schedule is:

```
First 1.0 hours: 10.14283 attempts. (Avg delay 4m 47.41s)
First 6.0 hours: 19.02377 attempts. (Avg delay 15m 36.95s)
First 96.0 hours: 56.11173 attempts. (Avg delay 1h 40m 3.13s)
First 168.0 hours: 83.67091 attempts. (Avg delay 1h 58m 43.16s)
Steady state: 2h 36m 44.63s between attempts.
```

The average behavior for the new non-primary schedule is:

```
First 1.0 hours: 3.08069 attempts. (Avg delay 14m 26.08s)
First 6.0 hours: 8.1473 attempts. (Avg delay 35m 25.27s)
First 96.0 hours: 22.57442 attempts. (Avg delay 3h 49m 32.16s)
First 168.0 hours: 29.02873 attempts. (Avg delay 5h 27m 2.36s)
Steady state: 11h 15m 28.47s between attempts.
```
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Proposal 335: An authority-only design for MiddleOnly

[Nick Mathewson]

On Sun, Oct 10, 2021 at 12:39 PM nusenu  wrote:
>
> Hi,
>
> has this proposal any implications wrt to making the MiddleOnly
> feature available to clients (without requiring DA actions)?
>
> With ExcludeExitNodes/ExitNodes + ExcludeGuardNodes [1]
> tor clients basically can get the "MiddleOnly" feature without DA actions,
> but ExcludeGuardNodes [1] is not there yet. This is why I'm asking.
>
> [1] https://gitlab.torproject.org/tpo/core/tor/-/merge_requests/436

I don't think it would especially help or prevent a
MiddleOnly/ExcludeGuard feature for clients.

> > ## Generating votes
> >
> > When voting for a relay with the `MiddleOnly` flag, an authority
> > should set all flags indicating that a relay is unusable for a
> > particular purpose, and against all flags indicating that the relay
> > is usable for a particular position.
>
> This part was not clear to me, but if I ignore it, the rest makes sense.
> It is not clear because it says an authority should set _all_ flags to 
> indicate
> that a relay is unusable for a particular purpose.

Ooops, that should say that the authority should vote for all "this
relay is unusable" flags, and against all "this relay is usable for a
non-middle purpose" flags.  I'll edit.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Arti report 7: September 30 through October 13

[Nick Mathewson]

# Arti Report 7: September 30 through October 13

## Guards are working!

Finally, at long last, we've got Guard nodes working in Arti!

When we first made our roadmap, we set our first milestone as "Arti
has all the features necessary for basic anonymity."  Now that
Guards are implemented, all of those features are done!

Does this mean that Arti is now production-ready for anonymous use?
No, not yet.  Just because we no longer have _missing_ privacy
features doesn't mean that those privacy features are actually
implemented correctly.  We'll need a lot more testing, debugging,
and experimentation.  Also, it's very possible we've forgotten
something.  We'll need careful auditing to be sure.

### Interesting guard issues

There are more issues to fix and features to add to the guard
implementation down the road.  For a current list, have a look at
[this 
comment](https://gitlab.torproject.org/tpo/core/arti/-/issues/58#note_2754904),
which lists them.

## What's next

Over the next couple of weeks, we'll be working on smaller issues,
bugfixes, and miscellaneous cleanups for an upcoming "0.0.1"
release, targeting November 1.

You can see our latest progress
[here]((https://gitlab.torproject.org/tpo/core/arti/-/milestones/6).

## Thanks!

Thanks to Jani Monoses for progress towards a WASM port, and Trinity
Pointard for updates to dependencies.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] How do Ed25519 relay IDs look like?

[Nick Mathewson]

On Sun, Oct 10, 2021 at 5:13 PM nusenu  wrote:
>
> > On Tue, Aug 4, 2020 at 6:41 PM nusenu  wrote:
> >>
> >> nusenu:
> >>> I'll wait until you (Tor developers) decided on the final naming and 
> >>> format
> >>
> >> Is there any interest to move this topic forward to come to some decision
> >> in the near future? (before the end of the month)
> >
> > I don't think that'd be too hard.
> >
> >> Here is a short summary of what opinions I observed for this topic (naming 
> >> and format
> >> for Ed25519 identities) so far:
> >>
> >> Naming proposals for relay Ed25519 identities:
> >> 
> >>
> >> 'v2 fingerprints' (Damian)
> >>
> >> "ed25519 identity" or even just "identity" (nickm)
> >>
> >>
> >> Output format the Ed25519 relay IDs:
> >> 
> >>
> >> base64 - 43 characters long (nickm)
> >>this is problematic due to the "/" sign (Damian)
> >> hex - 64 characters long (Damian)
> >>"/" is problematic for DirPort urls, GETINFO commands, etc (Damian)
> >>  isn't there urlencoding for URLs? (nusenu)
> >> base64urlsafe - 43 characters long (nusenu)
> >>
> >> I hope we can agree to use the same format in all places.
> >>
> >> How does the decision process looks like in general in the Tor Project?
> >
> > I think right now Tor uses unpadded base64 in most internal formats,
> > but it doesn't actually use those in the user interface anywhere, so
> > we could just use base64urlsafe (per rfc4648 section 5) for the user
> > interface.
> >
> > I would be fine with standardizing that for our API, but I'd want to
> > write a proposal for it first.  It wouldn't have to be long.  We'd
> > want to describe other places where we currently use regular base64
> > for 256-bit keys, and say whether we should/shouldn't accept and emit
> > url-safe identifiers there instead.
> >
> > We should specify that there are no spaces, that the padding "="
> > characters are removed, and that even though the format as given can
> > handle 43*6==258 bits, the last two bits must be set to 0, since these
> > are only 256-bit identifiers.
> >
> > We should also _probably_ specify some canonical encoding for a pair of 
> > keys.
> >
>
> I've come to the conclusion that since people are used so much to the fact
> that relay ID's (RSA) never were case sensitive, ed25519 should not
> be case sensitive either.
>
> So I'd propose to use base32 without padding.
> That would make it 52 chars long.
>
> Any opinions?
>

Hm.  Every time we've displayed Ed25519 fingerprints so far, we've
used base64.  I'm not sure that changing it will actually save more
confusion than it causes.

yrs,
-- 
Nick
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] ed25519_master_id_public_key -> ed25519 id

[Nick Mathewson]

On Sun, Oct 10, 2021 at 8:44 AM nusenu  wrote:
>
> Hi,
>
> given a relay's ed25519_master_id_public_key
> file, is there a simple way to generate the
> 43 chars long ed25519 identity string (also found in fingerprint-ed25519)?
>

Yes, there is!

First you verify that the file is really 64 bytes long, and that the
first 32 bytes of the file are really "== ed25519v1-public: type0
==\0\0\0".

Having done that, you base64-encode the second 32 bytes of the file,
with no "=" padding.

I've attached a lazy little python script.

cheers,
-- 
Nick
#!/usr/bin/python

import sys
import binascii

# requires python 3
assert sys.version_info >= (3,0,0)

try:
s = open(sys.argv[1], "rb").read()
except IndexError:
print("Syntax: {} ".format(sys.argv[0]))
sys.exit(1)

header = b"== ed25519v1-public: type0 ==\0\0\0"

if len(s) != 64 and s[:32] != header:
print("This wasn't an ed25519_master_id_public_key file.")
sys.exit(1)
else:
print(binascii.b2a_base64(s[32:]).decode("ascii").strip().replace("=",""))
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Proposal 335: An authority-only design for MiddleOnly

[Nick Mathewson]

```
Filename: 335-middle-only-redux.md
Title: An authority-only design for MiddleOnly
Author: Nick Mathewson
Created: 2021-10-08
Status: Open
```

# Introduction

This proposal describes an alternative design for a `MiddleOnly`
flag.  Instead of making changes at the client level, it adds a
little increased complexity at the directory authority's voting
process.  In return for that complexity, this design will work
without additional changes required from Tor clients.

For additional motivation and discussion see proposal 334 by Neel
Chauhan, and the related discussions on tor-dev.

# Protocol changes

## Generating votes

When voting for a relay with the `MiddleOnly` flag, an authority
should set all flags indicating that a relay is unusable for a
particular purpose, and against all flags indicating that the relay
is usable for a particular position.

These flags SHOULD be set in a vote whenever `MiddleOnly` is
present, and only when the authority is configured to vote on the
`BadExit` flag.

  * `BadExit`

These flags SHOULD be cleared in a vote whenever `MiddleOnly` is
present.

  * `Exit`
  * `Guard`
  * `HSDir`
  * `V2Dir`

## Computing a consensus

This proposal will introduce a new consensus method (probably 32).
Whenever computing a consensus using that consensus method or later,
authorities post-process the set of flags that appear in the
consensus after flag voting takes place, by applying the same rule
as above.

That is, with this consensus method, the authorities first compute
the presence or absence of each flag on each relay as usual.  Then,
if the `MiddleOnly` flag is present, the authorities set `BadExit`,
and clear `Exit`, `Guard`, `HSDir`, and `V2Dir`.

# Configuring authorities

We'll need a means for configuring which relays will receive this
flag.  For now, we'll just reuse the same mechanism as
`AuthDirReject` and `AuthDirBadExit`: a set of torrc configuration
lines listing relays by address.  We'll call this
`AuthDirMiddleOnly`.

We'll also add an `AuthDirListsMiddleOnly` option to turn on or off
voting on this option at all.

# Notes on safety and migration

Under this design, the MiddleOnly option becomes useful immediately,
since authorities that use it will stop voting for certain
additional options for MiddleOnly relays without waiting for the
other authorities.

We don't need to worry about a single authority setting MiddleOnly
unilaterally for all relays, since the MiddleOnly flag will have no
special effect until most authorities have upgraded to the new
consensus method.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] (Short) Arti Report 6: September 15 through September 30

[Nick Mathewson]

# Arti Report 6: September 15 through September 30

## Activities since our last report

Hello!  Lots of code since last time, but not so much to say about it.  I'm
chugging ahead on the Guard nodes implementation, but it isn't done yet.
Most of the code is there, but I need to write tests, chase down a few spec
issues, fix a couple of things that "don't feel right", and integrate it all
with tor-circmgr.

You can track progress on the guard implementation at
.


## Thanks!

Thanks to Jani Monoses for refactoring, Daniel Eades for cleanup, and Trinity
Pointard for OSX reproducible builds and integration testing (!).
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Proposal 332: Ntor protocol with extra data, version 3.

[Nick Mathewson]

On Fri, Jul 16, 2021 at 8:31 AM Ian Goldberg  wrote:
 [...]
> But this post from Trevor also made me realize a bigger issue with the
> protocol Nick proposed:
>
> If you want the protocol to work with Walking Onions, it needs to be
> *post-specified peer*.  That is, contrary to:
>
> > The client knows:
> >   * B: a public "onion key" for S
>
> The client will in fact _not_ know B in advance in a Walking Onions
> setting, but rather will learn it at the end of the handshake.  The
> protocol Nick specified does in fact use B in the first message, unlike
> the current ntor handshake, which just sends KEYID(B) in the first flow,
> but it's not part of the math, or indeed as far as I can see, used for
> anything at all in Section 5.1.4 of tor-spec.txt, and so can be easily
> removed (and replaced with B being sent by the server) for Walking
> Onions.

Well, in the current deployment, the client only knows one B for each
server at a time, and if the server responds with a B that the client
_doesn't_ recognize, the client won't have any idea whether or not B
is really authentic. (Priving that B is authentic is a big part of
what Walking Onions has to solve.)

Walking Onions is a big enough set of protocol changes that I'm
comfortable requiring a subsequent handshake bump when we get there.
Who knows, we may want another one between now and then if we decide
to go hybrid-PQ or use Noise or something :)

cheers,
-- 
Nick
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Arti report 5: August 18 through September 15

[Nick Mathewson]

# Arti Report 5: August 4 through September 15

## Activities since our last report

I'm back, with updates from the last month!  We've spent a lot of time over
the last on cleaning up technical debt issues in our code that had
accumulated as I wrote it.  There are more tests and warnings enabled than
before, and we've changed some new internal APIs to make them unable to
panic.  We're also using the [`derive_builder`][derive_builder] tool to throw
out a bunch of boilerplate code.

When I wrote last month, I'd just finished up the implementation for circuit
build timeout inference.  As followup work to that, I worked with Mike Perry
to collect all of the missing specification issues that I'd found while
cloning that feature, and [roll them back][torspec-mr-40] into our
`path-spec.txt` document.  Mike'sw found a couple of mistakes in my
implementation that I was able to resolve—fortunately, by removing code from
the Arti timeout inference code.



## Getting a head start on API design

API design will be the focus of our 0.1.0 release, but we still want to get
it "as right as we can" in the upcoming 0.0.1 version.

With that in mind, we've started enumerating the various kinds of programs
and environments that we need to support down the road, and [working on a list
of example programs][ticket-164] that we might be able to ship with Arti.
(My own experience so far is that writing example code is just about the best
way to find API deficiencies.)


## Uplifting new features from C Tor

For our work on congestion control, we're getting a new
[circuit extension handshake][prop332].  Going forward, as we add features to
C Tor, we're going to try to add them to Arti at around the same time.  With
that in mind, we have the new handshake implemented (but not enabled): we
used its implementation to validate our specifications double-check
our test vectors.

## Towards guard support

Our largest remaining priority for Arti 0.0.1 is full support for
[guard nodes][guard-spec].  (In Tor, a "guard node" is a relay that a client
decides to use as the first hop for all of its circuits for a long time.
Using guard nodes improves resistance against many kinds of attacks based on
hostile relays.)

We've started laying the groundwork here: we now have support for all of the
guard-related consensus parameters, and we've reworked our relay selection
code to allow picking multiple weighted random items without replacement.
We've also added some previously unneeded lookup-by-ID features to network
directory code.

(Fun fact: the Tor implementation of mapping IDs to relays has been
historically ugly, since we don't really know a relay's ed25519 identity
until we have a microdescriptor for it.  In C, this has led to a reasonably
messy pile of dangling pointer bugs…which Rust won't even let us write!
Also, we've made better design choices in Arti about our directory objects,
so we don't have do support all of the strange mutability that our C code had
to try to handle.)

We're now in the midst of coding up a guard implementation.  The tricky part
here is that our guard logic affects circuit building selection (since
circuit paths must begin with a guard), but circuit construction affects
guard selection.  We're experimenting with different designs to try to keep
the complexity here under control.



# Thanks to awesome volunteers!

Thanks to Jani Monoses for migrating Arti to the tracing crate and adding
journald support; to Robin Leander Schröder for cleaning up some of our
`unwrap_or()` code; to S0AndS0 for work enabling the `unwrap_used` Clippy
lint; to Daniel Eades for work cleaning up Clippy lints in our test modules,
and to Trinity Pointard for work on stream isolation and reproducible builds!


[ticket-164]: https://gitlab.torproject.org/tpo/core/arti/-/issues/164
[prop-332]: 
https://gitlab.torproject.org/tpo/core/torspec/-/blob/main/proposals/332-ntor-v3-with-extra-data.md
[derive_builder]: https://crates.io/crates/derive_builder
[torspec-mr-40]:
https://gitlab.torproject.org/tpo/core/torspec/-/merge_requests/40
[guard-spec]: 
https://gitlab.torproject.org/tpo/core/torspec/-/blob/main/guard-spec.txt
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] No arti report this week.

[Nick Mathewson]

Hi!

Usually I do Arti reports every other Wednesday, but this week I'm on
vacation.  I'll send the next report on September 15, so that the
reports can be offset from the Arti meetings.

cheers,
-- 
Nick
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Arti report 4: August 4 through August 18

[Nick Mathewson]

# Arti Report 3: August 4 through August 18


At last we're done with circuit timeout inference!  See the last weeks
update for an overview of what this was, and why it was hard. As of last
Friday, we've finally got an Arti implementation that builds testing
circuits as appropriate, infers reasonable timeouts, and remembers those
timeouts across runs.

As a side-benefit to this work, we've found a number of points where we
need to clarify the Tor specification to be more precise about correct
behavior for a Tor client.  (For example, how to break ties when
deciding on the most populous histogram bins.) I'm taking some time to
circle back and fix those in the Tor specs.

Here is the first recorded persistent state from an Arti implementation,
lightly reformatted:

version = 1
histogram = [[415, 1], [495, 2], [515, 1], [535, 1], [545, 1],
[555, 1], [625, 1], [655, 1], [665, 1], [785, 1], [835, 1],
[865, 2], [885, 3], [955, 1], [965, 1], [975, 2], [985, 1],
[1005, 2], [1025, 1], [1035, 2], [1055, 3], [1065, 1],
[1075, 1], [1085, 2], [1095, 1], [1115, 2], [1135, 2],
[1145, 2], [1175, 1], [1185, 1], [1195, 2], [1205, 3],
[1215, 1], [1235, 2], [1245, 6], [1255, 1], [1265, 2],
[1285, 2], [1295, 3], [1305, 1], [1315, 1], [1355, 2],
[1375, 2], [1385, 2], [1405, 2], [1435, 1], [1455, 1],
[1465, 1], [1495, 1], [1505, 1], [1525, 1], [1535, 1],
[1555, 1], [1565, 1], [1605, 1], [1625, 1], [1665, 1],
[1765, 1], [1785, 1], [1825, 1], [1965, 1], [2165, 1],
[2495, 1], [3025, 1], [3855, 1], [4265, 1], [4795, 1],
[4935, 1], [5435, 1], [6825, 1]]
current_timeout = 1592
abandoned_circs = 0
successful_circs = 20

## Other work

We've continued to review and merge volunteer patches. Thanks,
everybody! In response to a patch documenting a number of possible
panics, I've taken a pass through the code to reduce the number of
functions that could panic.

I've also been spending some time improving test coverage: while doing
so, I found an incorrect behavior when checking whether two IPv6
addresses were on the same network.  (Tests are great and need to be
everywhere, apparently.)

## Coming up

Our current priorities are:

   * Starting API design work.  So far we've been feeling our way
 forward, and it's time to get more systematic.

   * Improving our APIs and internal functionality for stream isolation,
 to support a variety of usecases more like what can be done with
 current Tor.

   * Kicking off the work on a Guard implementation.  This is the last
 key feature we need for a basic level of privacy.

   * Lots more interviewing job candidates.

Let's see how far we get in the next two weeks!
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] This week's arti meeting rescheduled: 18 August, 1600 UTC

[Nick Mathewson]

Hi!

Because of a scheduling conflict, our biweekly Arti meeting is
rescheduled (for this Wednesday only).  The time this week will be at
18 August, 1600 UTC, at https://tor.meet.coop/gab-sis-ldw-xad .

This week we'll be brainstorming API ideas, use cases, and user
stories.  Don't worry if you can't make it: we record our meetings and
put them online on Tor's youtube channel.  (And this won't be your
last chance to influence the API; it's just an initial brainstorming
session.)

cheers,
-- 
Nick
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Arti Report 3: July 21 through August 4

[Nick Mathewson]

Still quiet here for the last couple of weeks.  I've been moving ahead on
circuit timeout inference, and reviewing _lots_ of volunteer-submitted code
(and job applications).

The quiet part is coming to an end: Two of the engineers who have been on
vacation or leave are back, and we're all trying to ramp up.  We're also
hoping to be scheduling interviews with prospective co-workers soon, so
that's taking up some of our time.. but once it's done, it should accelerate
our pace a bit.

## Circuit timeout inference logic

I had hoped to be done with this by now, but it turns out that this code
touches on several key architectural issues that we'll want to make sure we
get right in the future.  I'll touch on them a bit here.

### Persistent state

The Tor protocol requires clients to keep persistent state across
invocations in order to achieve good performance and privacy.  Two
notable cases of this are in the circuit management code: client need to
remember circuit timeout data, and need to remember guard selection
data.

The C Tor client records these items in the same ad-hoc format used by
Tor's configuration code.  But for Rust, using serde seems like a better
fit.

I want to maintain the feature that Arti has now where you can run
multiple processes with the same state directory.  That could get
tricky in the long run, but I think it's a design feature we need if
we're going to keep Arti flexible enough for general use.

My first implementation here is going to be bare-bones, but I'm trying
to design it to grow and get replaced down the line.

### Inter-module event notification

Frequently one module needs to notify another when some event occurs.
In the circuit-timeout case, we need to notify the circuit manager
whenever the network parameters have changed.

We don't want to have the circuit manager probing the directory manager,
since that's a higher-level module.  And we don't want it probing the
`NetDir` object it gets whenever it's time to build a circuit, since
that would be a weird side-effect.

The best I've been able to come up with for now is to allow the
directory manager to expose a `Stream` of events for directory changes,
and to have the `TorClient` object use events from that stream to alert
the circuit manager to possibly changed network parameters.  I think
this should scale to other kinds of inter-module events in the future,
though I do have a slight concern that it makes `TorClient` harder to
reproduce with lower-level crates than it would be otherwise.  (There's
probably no avoiding that at this stage, though.)

### Non-user-initiated background tasks

The circuit timeout code sometimes needs to build testing circuits in
order to gather data, and the guard code does too.  If we code this in a
sloppy way, we'll wind up with three separate systems that all build circuits
for different reasons at different times.  With any luck, we can summarize
the triggers from all of these circuit types into a single system.

## Coming up

We've been working on refinements to the API in `arti-tor-client` to try to
simplify the API as much as possible while still exposing the full back-end
functionality.  Everybody should expect a lot of unstability here.

I am still hoping very much that the next time I write one of these reports,
we'll be done with circuit timeout inference, and I can talk about API work
and early progress on guards! :)
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Arti development report: July 7 through July 21

[Nick Mathewson]

# Arti Report 2: July 7 through July 21

It's been a quiet couple of weeks.  Most of the team has been on
vacation or leave for the last two weeks, so the Arti hacking has
been down to me (nick) for this period.

Our last report went out on the same day that our [announcement]
went public, and since then we've had an influx of attention and
volunteers interest.  Thanks to everybody who's been trying out the
code!  Special thanks to `trinity-1686a` for a
[stream-isolation] implementation, `lyuyuan` for
work on [RESOLVE and RESOLVE_PTR][resolve], and everybody else who's
been submitting patches!

## Circuit timeout inference

My own work for this period has focused on [circuit timeout
inference][cti].  (I decided to start with this because it's part of the
Tor protocol that I've worked with the least in the past.)

The approach is kind of complicated: in order to adjust our timeout, we
need to observe how long circuits actually take to
complete... including the circuits that we don't use because they
take longer than our current timeout timeout. So we have to keep
building circuits for a while even after they time out, so that we
can observe how long it takes them to finish.  Then we use an
estimator based on a Pareto distribution, and use that to determine
our actual cutoffs.

The core logic is now [implemented in Arti][pareto-code], and we
just have a few more pieces ([persistence][ticket-143],
[directory notification][ticket-144], and
[testing circuits][ticket-145]) to cover before we can call it done.

## Open problems

Our timeout tests rely on using a simulated view of time.  (Nobody
wants to wait 10 seconds for a simulated 10-second timeout!)  For
that, we have [testing code][rt-mock-time] to override the
asynchronous runtime's view of the passage of time.

But we have a bug somewhere that makes these tests unreliable.  The
[WaitFor] future, which is supposed to run a another future to
completion while advancing the current (simulated) time, doesn't
seem to be reliable.  As implemented, it advances the current time
even when the future it's testing _isn't_ blocked on time, and
sometimes advances time much too far while the tested future makes
no progress.  I've kludged it up by making it yield a lot and
advance time in tiny increments, but that's just a workaround.

Does anybody like [debugging Futures code][ticket-149]?

## Next steps

In the next two weeks I hope we can wrap up the outstanding issues above
in circuit timeouts, including the ones that require some
architectural decisions.

I'm also hoping that we can make progress on [guards]: that's the
last remaining really big security feature we have to do for
our 0.0.1 milestone.  Once that's done, we can move on to polishing
and infrastructure issues for the rest of the time we have for this
milestone.

Finally, we're going to be expanding our team!  Please have a look
at our [job-opening], and share it with anybody you know who is
interested in Rust and Tor, and who might be interested in a job.

Next update in two weeks!

[announcement]: https://blog.torproject.org/announcing-arti
[cti]: 
https://gitlab.torproject.org/tpo/core/torspec/-/blob/master/path-spec.txt#L364
[cti]: 
https://gitlab.torproject.org/tpo/core/torspec/-/blob/master/guard-spec.txt
[job-opening]: https://www.torproject.org/about/jobs/rust-dev/
[pareto-code]: 
https://gitlab.torproject.org/tpo/core/arti/-/blob/main/tor-circmgr/src/timeouts/pareto.rs
[resolve]: https://gitlab.torproject.org/tpo/core/arti/-/merge_requests/39
[rt-mock-time]:
https://gitlab.torproject.org/tpo/core/arti/-/blob/main/tor-rtmock/src/time.rs
[stream-isolation]:
https://gitlab.torproject.org/tpo/core/arti/-/merge_requests/38
[ticket-143]: https://gitlab.torproject.org/tpo/core/arti/-/issues/143
[ticket-144]: https://gitlab.torproject.org/tpo/core/arti/-/issues/144
[ticket-145]: https://gitlab.torproject.org/tpo/core/arti/-/issues/145
[ticket-149]: https://gitlab.torproject.org/tpo/core/arti/-/issues/149
[WaitFor]: 
https://gitlab.torproject.org/tpo/core/arti/-/blob/main/tor-rtmock/src/sleep_runtime.rs#L123
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Proposal 332: Ntor protocol with extra data, version 3.

[Nick Mathewson]

On Mon, Jul 12, 2021 at 3:04 PM Ian Goldberg  wrote:
>
> On Mon, Jul 12, 2021 at 12:01:47PM -0400, Nick Mathewson wrote:
> > Both parties know that they used the same verification string; if
> > they did not, they do not learn what the verification string was.
> > (This feature is required for HS handshakes.)
>
> I'm not sure the protocol you specify has this feature as written.  For
> example, if the verification string has low entropy, the server could
> brute-force the client's verification string (using the MAC to check its
> guess).  This is unlike, say, OTR's SMP or a PAKE, in which each online
> execution of the protocol allows the server just one guess.
>
> But perhaps you don't actually need the property in as strong a form as
> you wrote it, since the HS handshake application has high-entropy
> secrets?

Oh yes, you are right, of course.

Can you suggest a way to phrase this property that encompasses what
the protocol actually does provide?

best wishes,
-- 
Nick
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Proposal 332: Ntor protocol with extra data, version 3.

[Nick Mathewson]

```
Filename: 332-ntor-v3-with-extra-data.md
Title: Ntor protocol with extra data, version 3.
Author: Nick Mathewson
Created: 12 July 2021
Status: Open
```

# Overview

The ntor handshake is our current protocol for circuit
establishment.

So far we have two variants of the ntor handshake in use: the "ntor
v1" that we use for everyday circuit extension (see `tor-spec.txt`)
and the "hs-ntor" that we use for v3 onion service handshake (see
`rend-spec-v3.txt`).  This document defines a third version of ntor,
adapting the improvements from hs-ntor for use in regular circuit
establishment.

These improvements include:

 * Support for sending additional encrypted and authenticated
   protocol-setup handshake data as part of the ntor handshake.  (The
   information sent from the client to the relay does not receive
   forward secrecy.)

 * Support for using an external shared secret that both parties must
   know in order to complete the handshake.  (In the HS handshake, this
   is the subcredential.  We don't use it for circuit extension, but in
   theory we could.)

 * Providing a single specification that can, in the future, be used
   both for circuit extension _and_ HS introduction.

# The improved protocol: an abstract view

Given a client "C" that wants to construct a circuit to a
relay "S":

The client knows:
  * B: a public "onion key" for S
  * ID: an identity for S, represented as a fixed-length
byte string.
  * CM: a message that it wants to send to S as part of the
handshake.
  * An optional "verification" string.

The relay knows:
  * A set of [(b,B)...] "onion key" keypairs.  One of them is
"current", the others are outdated, but still valid.
  * ID: Its own identity.
  * A function for computing a server message SM, based on a given
client message.
  * An optional "verification" string. This must match the "verification"
string from the client.

Both parties have a strong source of randomness.

Given this information, the client computes a "client handshake"
and sends it to the relay.

The relay then uses its information plus the client handshake to see
if the incoming message is valid; if it is, then it computes a
"server handshake" to send in reply.

The client processes the server handshake, and either succeeds or fails.

At this point, the client and the relay both have access to:
  * CM (the message the client sent)
  * SM (the message the relay sent)
  * KS (a shared byte stream of arbitrary length, used to compute
keys to be used elsewhere in the protocol).

Additionally, the client knows that CM was sent _only_ to the relay
whose public onion key is B, and that KS is shared _only_ with that
relay.

The relay does not know which client participated in the handshake,
but it does know that CM came from the same client that generated
the key X, and that SM and KS were shared _only_ with that client.

Both parties know that CM, SM, and KS were shared correctly, or not
at all.

Both parties know that they used the same verification string; if
they did not, they do not learn what the verification string was.
(This feature is required for HS handshakes.)

# The handshake in detail

## Notation

We use the following notation:

  * `|` -- concatenation
  * `"..."` -- a byte string, with no terminating NUL.
  * `ENCAP(s)` -- an encapsulation function.  We define this
 as `htonll(len(s)) | s`.  (Note that `len(ENCAP(s)) = len(s) + 8`).
  * `PARTITION(s, n1, n2, n3, ...)` -- a function that partitions a
 bytestring `s` into chunks of length `n1`, `n2`, `n3`, and so
 on. Extra data is put into a final chunk.  If `s` is not long
 enough, the function fails.

We require the following crypto operations:

  * `KDF(s,t)` -- a tweakable key derivation function, returning a
 keystream of arbitrary length.
  * `H(s,t)` -- a tweakable hash function of output length
 `DIGEST_LEN`.
  * `MAC(k, msg, t)` -- a tweakable message-authentication-code function,
 of output length `MAC_LEN`.
  * `EXP(pk,sk)` -- our Diffie Hellman group operation, taking a
 public key of length `PUB_KEY_LEN`.
  * `KEYGEN()` -- our Diffie-Hellman keypair generation algorithm,
returning a (secret-key,public-key) pair.
  * `ENC(k, m)` -- a stream cipher with key of length `ENC_KEY_LEN`.
`DEC(k, m)` is its inverse.

Parameters:

  * `PROTOID` -- a short protocol identifier
  * `t_*` -- a set of "tweak" strings, used to derive distinct
hashes from a single hash function.
  * `ID_LEN` -- the length of an identity key that uniquely identifies
a relay.

Given our cryptographic operations and a set of tweak strings, we
define:

```
H_foo(s) = H(s, t_foo)
MAC_foo(k, msg) = MAC(k, msg, t_foo)
KDF_foo(s) = KDF(s, t_foo)
```

See Appendix A.1 below for a set of instantiations for these operations
and constants.

## Client operation, phase 1

[tor-dev] Arti development report: 23 June through July 7

[Nick Mathewson]

# Arti Report 1: 23 June through July 7

We started with a project kickoff meeting.  We're planning to continue
meeting every 2 weeks, making our meetings increasingly public, and
posting them on youtube.

We wrote up an announcement to explain what Arti is and why we're doing
it.  The [announcement](https://blog.torproject.org/announcing-arti)
goes live on July 8; it's a good place to start if you're not already
familiar with the project, or if you're looking for more helpful links.

To get ready for future development, and check the state of our packages
we put out an [0.0.0 release](https://crates.io/crates/arti) and
uploaded it to crates.io.  We found a couple of issues while doing so
([#129][ticket-129], [#130][ticket-130]).

We want to switch as much new-feature development to Arti as we can.
With that in mind, we're developing a set of guidelines for discouraging
features in C -- especially features with the potential to destabilize
our code. (See [#128][ticket-128].)  We're trying a bottom-up approach
here, starting by identifying what kinds of features we've done recently
that maybe we should skip in the future.

We had an excellent meeting with developers on various Zcash projects,
to gather requirements, set expectations, and open lines of
communication.  Based on their suggestions, we opened a set of new
tickets to work on.  ([#132][ticket-132] [#133][ticket-133]
[#134][ticket-134] [#135][ticket-135] [#136][ticket-136]
[#137][ticket-137] [#138][ticket-138] [#139][ticket-139]
[#140][ticket-140])

In terms of new code written, things have been a bit slow: we're still
ramping up, and a lot of folks take vacations in the summer.  We
[revised our asynchronous runtime backend][ticket-129] based on issues
we found while uploading our packages.  More recently, I've started
working on an [implementation][ticket-57] of Tor's
[circuit timeout inference][cti] code, which is necessary for good
performance and security.

## Next steps

In our next couple of weeks, I hope we get a first cut at our circuit
timeout code finished and working, and figure out a reasonable initial
solution for our [persistence needs][ticket-59].  We'll also be ramping
up on implementations for [guard nodes][ticket-58] and
[stream isolation][ticket-73], both of which are necessary for security.


[ticket-57]: https://gitlab.torproject.org/tpo/core/arti/-/issues/57
[ticket-58]: https://gitlab.torproject.org/tpo/core/arti/-/issues/58
[ticket-59]: https://gitlab.torproject.org/tpo/core/arti/-/issues/59
[ticket-73]: https://gitlab.torproject.org/tpo/core/arti/-/issues/73
[ticket-128]: https://gitlab.torproject.org/tpo/core/arti/-/issues/128
[ticket-129]: https://gitlab.torproject.org/tpo/core/arti/-/issues/129
[ticket-130]: https://gitlab.torproject.org/tpo/core/arti/-/issues/130
[ticket-132]: https://gitlab.torproject.org/tpo/core/arti/-/issues/132
[ticket-133]: https://gitlab.torproject.org/tpo/core/arti/-/issues/133
[ticket-134]: https://gitlab.torproject.org/tpo/core/arti/-/issues/134
[ticket-135]: https://gitlab.torproject.org/tpo/core/arti/-/issues/135
[ticket-136]: https://gitlab.torproject.org/tpo/core/arti/-/issues/136
[ticket-137]: https://gitlab.torproject.org/tpo/core/arti/-/issues/137
[ticket-138]: https://gitlab.torproject.org/tpo/core/arti/-/issues/138
[ticket-139]: https://gitlab.torproject.org/tpo/core/arti/-/issues/139
[ticket-140]: https://gitlab.torproject.org/tpo/core/arti/-/issues/140
[cti]: 
https://gitlab.torproject.org/tpo/core/torspec/-/blob/master/path-spec.txt#L364
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Proposal 330: Modernizing authority contact entries

[Nick Mathewson]

(I'm using 330 for this proposal, since 329 is reserved for conflux.)

```
Filename: 330-authority-contact.md
Title: Modernizing authority contact entries
Author: Nick Mathewson
Created: ---
Status: Draft
```

This proposal suggests changes to interfaces used to describe a
directory authority, to better support load balancing and
denial-of-service resistance.

(In an appendix, it also suggests an improvement to the description of
authority identity keys, to avoid a deprecated cryptographic algorithm.)

# Background

There are, broadly, three good reasons to make a directory request to a Tor
directory authority:

   - As a relay, to publish a new descriptor.
   - As another authority, to perform part of the voting and consensus
 protocol.
   - As a relay, to fetch a consensus or a set of (micro)descriptors.

There are some more reasons that are OK-ish:
   - as a bandwidth authority or similar related tool running under the
 auspices of an authority.
   - as a metrics tool, to fetch directory information.
   - As a liveness checking tool, to make sure the authorities are running.

There are also a number of bad reasons to make a directory request to a
Tor directory authority.

   - As a client, to download directory information.  (_Clients should
 instead use a directory guard, or a fallback directory if
 they don't know any directory information at all._)
   - As a tor-related application, to download directory information.
 (_Such applications should instead run a tor client, which can
 maintain an up-to-date directory much more efficiently._)


Currently, Tor provides two mechanisms for downloading and uploading directory
information: the DirPort, and the BeginDir command.  A DirPort is an
HTTP port on which directory information is served.  The BeginDir
command is a relay command that is used to send an HTTP stream directly
over a Tor circuit.

Historically, we used DirPort for all directory requests.  Later, when
we needed encrypted or anonymous directory requests, we moved to a
"Begin-over-tor" approach, and then to BeginDir.  We still use the
DirPort directly, however, when relays are connecting to authorities to
publish descriptors or download fresh directories.  We also use it for
voting.

This proposal suggests that instead of having only a single DirPort,
authorities should be able to expose a separate contact point for each
supported interaction above.  By separating these contact points, we can
impose separate access controls and rate limits on each, to improve the
robustness of the consensus voting process.

Eventually, separate contact points will allow us do even more: we'll be
able to have separate implementations for the upload and download
components of the authorities, and keep the voting component mostly
offline.

# Adding contact points to authorities

Currently, for each directory authority, we ship an authority entry.
For example, the entry describing tor26 is:

"tor26 orport=443 "
  "v3ident=14C131DFC5C6F93646BE72FA1401C02A8DF2E8B4 "
  "ipv6=[2001:858:2:2:aabb:0:563b:1526]:443 "
  "86.59.21.38:80 847B 1F85 0344 D787 6491 A548 92F9 0493 4E4E B85D",

We extend these lines with optional contact point elements as follows:

   - `upload=http://IP:port/`  A location to publish router descriptors.
   - `download=http://IP:port/`  A location to use for caches when fetching
 router descriptors.
   - `vote=http://IP:port/` A location to use for authorities when voting.

Each of these contact point elements can appear more than once.  If it does,
then it describes multiple valid contact points for a given purpose;
implementations MAY use any of the contact point elements that they recognize
for a given authority.

Implementations SHOULD ignore url schemas that they do not recognize, and
SHOULD ignore hostnames addresses that appear in the place of the IP elements
above. (This will make it easier for us to extend these lists in the future.)

If there is no contact point element for a given type, then implementations
should fall back to using the main IPv4 addr:port, and/or the IPv6 addr:port
if available.

As an extra rule: If more than one authority lists the same upload
point, then uploading a descriptor to that upload point counts as having
uploaded it to all of those authorities.  (This rule will allow multiple
authorities to share an upload point in the future, if they decide to do
so.  We do not need a corresponding rules for voting or downloading,
since every authority participates in voting directly, and since there
is no notion of "downloading from each authority.")

# Authority-side configuration

We add a few flags to DirPort configuration, indicating what kind of requests
are acceptable.

   - `no-voting`
   - `no-download`
   - `no-upload`

These flags remove a given set of possible operations from a given
DirPort.  So for example, an authority might say:

 DirPort 9030

[tor-dev] Proposed changes to Tor long-term-support (LTS) policy

[Nick Mathewson]

Hi, all!

I've been working on a proposed change to Tor's LTS policies. I've run
it by a few people already, and now I'm posting it here for wider
comment.

(summary: If we decide to do this, we will still be able to do LTS
releases, but we will backport fewer things to them, and we will make
fewer promises about how well they will work on the network.)


=
# Background and summary:

I'm proposing a change to Tor's long-term support (LTS) policies.

For reference, our current policy is described at
https://gitlab.torproject.org/tpo/core/team/-/wikis/NetworkTeam/CoreTorReleases

We've been struggling with our LTS policies for a while.  In brief: we
backport too many fixes, and we promise too much support for LTS
releases.

The developers don't like it, because the amount of things that we keep
trying to fix in our LTS releases keeps us working on old crufty code
for a long time.

Many packagers don't like it, because they have a policy of auditing
security backports, and we backport too much to our LTS releases for
them to audit carefully.

And our network maintenance group doesn't like it, because our
commitment to supporting very old protocol versions keeps us from
implementing performance and security improvements on a rapid schedule,
unless we backport those changes to the LTS releases.

Therefore, we're going to propose these changes:
   - That once a release becomes LTS-only, its code no longer gets
 anything but security patches (narrowly defined), and minimal
 patches to keep it working on the network.

   - We will no longer guarantee that an LTS-only release will work (or
 work well) on the mainline Tor network for its entire LTS
 lifetime.  We'll try to deliver this if we can, but it won't
 be a definite guarantee.

# In more detail

We propose the following release statuses:

   - Development.  (Every series starts out in this state as an alpha.)

   - Stable.  (Once a series is officially 'ready', we call it stable.)

   - Old-stable.  (Every supported stable release, except the most
 recent one, is in this state.)

   - Long-term support only. (Any LTS release, once a newer release
 has become old-stable.  Only certain releases will get LTS support.)

Every series starts out in "development".  Once it's officially
ready, we call it "stable".  All stable releases besides the most
recent one are "old-stable".

Allowed in all releases:
   - Updates to authorities list
   - Updates to fallbackdirs list
   - Updates to geoip database

LTS-only (any LTS release, once an newer release is oldstable):
   - Only two kinds of changes are allowed:
   - Security fixes, narrowly defined. (See below for a definition.)
   - _Simple_ patches that keep the release functional on the
 network.
   - Relays are not guaranteed to be supported on the network,
 although we'll try not to remove them gratuitously.
   - Clients and onion services are not guaranteed to work on the
 network, although we'll try not to break them gratuitously.

In other words, with an LTS release there will be no guarantee that the
software works on the network.  The promise is that we will keep it
working on the network when we can do so with simple low-risk patches,
and that _if_ it works, we will fix security problems in it.

Oldstable (All stable releases besides the most recent stable release):
  - Stability fixes are also allowed.
  - Relays will be supported on the network.
  - Clients and onion services will be supported on the network.
  - Dirauths may be supported.

Stable (The single most recent stable release):
  - All fixes are allowed.
  - Relays will be supported on the network.
  - Clients and onion services will be supported on the network.
  - Dirauths will be supported.

Development:
  - All fixes are allowed.
  - Relays will be supported on the network.
  - Clients and onion services will be supported on the network.
  - Dirauths will be supported.

==

What is a security fix?
  - It is a _bugfix_ that resolves a vulnerability.  _Features_ that
make Tor more private, anonymous, or more secure won't count.

==

The LTS policy above will apply to 0.3.5 _starting with 0.3.5.14_, since
we've already made backports that will appear in 0.3.5.13.

We have already committed to making 0.3.5 an LTS release until Feb 1,
2022.

We also now commit to making 0.4.5 an LTS release until _at least_ Feb
15, 2023. Whether we continue to do this LTS for longer will depend on
our experiences with this new policy.

==

So, any proposed amendments to this?

best wishes,
-- 
Nick
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Has Core Tor Development Slowed? Or Are We Moving To Rust/arti?

[Nick Mathewson]

On Fri, Jan 8, 2021 at 7:43 PM Neel Chauhan  wrote:
>
> Hi tor-dev@,
>
> I hope you all had a great holiday season.
>
> I noticed that Core Tor development, especially with the tor daemon, has
> slowed down significantly since the COVID crisis.
>
> Sorry if I have been less active in the Tor community as opposed to the
> past. I've been more focused on FreeBSD Ports as of now, but I still
> want to post at least the occasional Tor patch here and there.
>
> Core Tor/Tor hasn't seen an update since December 21, even when we are
> already a week into 2021.

Briefly:

We've been on vacation for a while.  We're still developing Tor.

-- 
Nick
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Arti (was Re: ARTI)

[Nick Mathewson]

On Fri, Dec 11, 2020 at 12:31 PM Zaphoid  wrote:
>
> Greetings Tor-Dev,
>
> I've watched Nick's presentation on State of the Onion and have read the "Tor 
> in 2021" blog post searching for more details on A Rust Tor Implementation 
> (ARTI). I think Rust is a fascinating language and looking to further my 
> knowledge of it and Tor's internals as they are developed.

Hi, Zaphoid, and thanks for the email!

The official repository is at
https://gitlab.torproject.org/tpo/core/arti/ ; most of the info on it
is there.

> I understand that some Rust implementation is available in the source 
> repository. What I am looking for is a road-map or planned outcome. Could 
> anyone advise if the intent is to completely re-write the Tor client in Rust? 
> Or, is the project's goal to rewrite specific components?

[tl;dr: All these plans are tentative and subject to change. I would
_like_ to eventually replace all of our C with Rust, but that is going
to take a long time, and will depend on resources.  For now, we're
going to continue supporting the C code, and we plan to do so at least
until the Rust code is ready to replace it. There is no timeline.]


Okay, so first: this is still an experimental project, so I can't make
promises. I don't know how it's going to go, and I'm still learning
Rust myself.

Also, Tor's plans are always, and forever, at least a little flexible
and amorphous.  Our only _permanent_ plan is that we will defy all
obstacles to use whatever resources we have to bring stronger privacy
to the world. Other plans are always subsidiary to that, and dependent
on our resources. [Please donate to our end-of-year fundraising
campaign if you can, everybody.]

With those disclaimers aside: I do _want_ to eventually replace the C
version of Tor as a client, for most users, with a Rust
implementation.  Eventually, if that works out, I'd like relays and
authorities to be written in Rust too.

The motivation here is not just for the additional safety and
performance that Rust can bring us.  For me, the most important reason
to move towards Rust is maintainability.  Rust gives us tools that
make it much easier to write a maintainable, well-structured,
well-tested set of programs, and to develop those programs in the
future.  I'm confident that a Rust implementation of Tor would still
be comprehensible and maintainable by a different set of programmers
in 20 years; I don't have the same confidence about a C
implementation.

Of course, that won't be easy.  (I know. I've been working on the C
version since 2002, so it would be a bit shocking if the Rust version
replaced it right away.)  We have to choose where to spend our time,
and we can't walk away from the C while it's still our best
implementation.  We'll be supporting the C version of Tor for the
future, I think -- probably for quite a few years.  It's possible that
at some point the two implementations will converge, but that might be
tricky: the C implementation's structure is not really conducive to
having parts of it replaced.

So where are we now, and what's the roadmap?

Right now, if you know how to download and compile Rust programs, you
can build Arti.  It can run as a SOCKS proxy and use the Tor network.
I wouldn't recommend it for serious use yet: it doesn't support onion
services, guards, bridges, and a lot of other security features that
Tor provides, and it's still pretty early in development.  As they
say, "When it breaks, you get to keep both pieces."

Over the coming months, I want to work to close the gap between Arti
and C Tor.  There's an approximate list of the stuff we'll need to do
in order to get there in the TODO file in the Arti repository.  (No
guarantees that I haven't forgotten anything)  How fast we can make
progress on this work is dependent on how much of our time we can
spend on it, and how many people can help us out.  So if anybody here
is interested in Rust, why not have a look at the codebase, and see
what you think?




(Also, there's no need to name any of my software in all-caps.  It's
just "arti" for now, or "Arti" if you really like to capitalize proper
nouns.)

best wishes,
-- 
Nick
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Proposal 328: Make Relays Report When They Are Overloaded

[Nick Mathewson]

On Wed, Dec 9, 2020 at 10:10 AM David Goulet  wrote:
>
> On 07 Dec (15:36:53), Ian Goldberg wrote:
> > On Mon, Dec 07, 2020 at 03:06:43PM -0500, David Goulet wrote:
> > > Greetings,
> > >
> > > Attached is a proposal from Mike Perry and I. Merge requsest is here:
> > >
> > > https://gitlab.torproject.org/tpo/core/torspec/-/merge_requests/22
> > >
> > > Cheers!
> > > David
> >
> > Nice!
> >
> > Is there a way to distinguish "not overloaded" from "does not support
> > this extension"?  (Ideally in a better way than checking the tor release
> > version number and inferring when support was added?)
>
> Gd point.
>
> So in theory, we have protocol version[1] in order to differentiate relays but
> I do not believe such a change would be a wise thing to use a new "Desc="
> since tor will just ignore the unknown fields.
>
> The other reason for that is that "tor functionalities" as in to function
> properly won't depend on that descriptor field so it is a bit a stretch to
> justify this as a new protocol version :S ...
>
> So yeah, either one looks at the tor version or "you don't see it, not
> overloaded" which is ofc a lie until the entire network migrates. We expect
> our sbws (bandwidth scanner tool) to be the main customer for this.

We could add a new element to this proposal, something like "I will
tell you about overload information", or "I am not overloaded" or
something like that?

peace,
-- 
Nick
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Update to Proposal 316: FlashFlow

[Nick Mathewson]

Hi!  We had a meeting about FlashFlow today, including several of the
authors.  Here are the notes we wound up with for ideas and next
straps.

Easy changes:
* Just use a PRNG; assume we can make them arbitrarily fast.
(example candidates: chacha8, shake128.)
* Use relay identities as the identifiers for measurers, so that
we won't need a novel authentication scheme.
* We can't call the list of measurer IDs a "network parameter",
since technically speaking network parameters have to be integers.  It
will have to be a different part of the consensus header.
* Make sure that all of the declared ranges for network parameters
are as wide as they could possibly be; making these parameters take a
wider range is hard to change later.

Trickier but straightforward:
* Describe how to avoid collisions with multiple coordinators
   - idea: exactly how it's specified in the paper ;) but a
simplier idea ...
   - idea: coord 1 measurers on day 1, c2 on d2, ... etc. for all
coords, then repeat
* Describe how to aggregate all background measurements over the
full 30 seconds, and how to use that data.  (This may lower accuracy a
little, but makes some kinds of the analysis harder.) Idea: relay
reports *once* at end of measurement the total amount of bg traffic
and the coord simply divides that by the length of the measurement to
have a per-second average.
* Mention whether relays should reserve sockets in case they get measured

More thinking may be needed:
*  Summarize ideas for how multiple coordinators don't have to
share full schedules with one another. Possibly divide up the network
by days? [e.g., Coordinator 1 measures nodes in set X on Monday]
* Would it work if we declare a maximum measurement fraction (eg
75% of bandwidth) but measurers only use that fraction in a few
measurements once in a while, and mostly they do less (eg 10% of
bandwidth).
* Discuss migration: how do we use this data when not all relays
support being measured in this way?


In terms of implementation:
- identify the python parts that are different to sbws, create sbws
subpackages "ff measurer" and "ff coordinator" and add a config option
to run in 1 mode or other, to do not have yet another code base to
maintain

In terms of deployment:
- we currently don't have any automatic way to ensure net is still
"working properly", only some mostly-manual ways and some one-time
experiments. This has caused some relay operators to do not be happy
and some quite time to figure out the problem and solve it

In terms of coordination:
We're deploying sbws only 1 dirauth at a time and trying to ensure net
is still "working properly".
If we deploy ff, before we have deployed sbws in all bwauths and
ensure net is still "working properly", will be hard to see what is an
sbws bug or ff one or both


FlashFlow, the python code for coordinator, measurer, etc.
https://gitlab.torproject.org/pastly/flashflow

The rendered documentation for/from the above https://flashflow.pastly.xyz/

Tor repo with branch https://gitlab.torproject.org/pastly/tor/-/tree/ff-v2

The ticket with the concerning graphs attributable to "Rob's speedtest thing"
https://trac.torproject.org/projects/tor/ticket/33076
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] new home for trac/CoreTorReleases?

[Nick Mathewson]

On Thu, Sep 17, 2020 at 4:50 PM nusenu  wrote:
>
> Hi,
>
> in the past
> https://trac.torproject.org/projects/tor/wiki/org/teams/NetworkTeam/CoreTorReleases
>
> used to be the canonical place for tor EoL information.
> Since Trac is no longer used: Is there a new home for this kind of 
> information?

https://gitlab.torproject.org/tpo/core/team/-/wikis/NetworkTeam/CoreTorReleases
is the new place.

> @Nick:
> In the past, before rejecting eol versions you asked about
> the current state on the tor network, the following new table
> will now provide you with a list of operators (more precisely ContactInfos)
> by clicking on the CW column you can get it sorted by consensus weight
> to find the biggest affected operators running eol releases:
>
> https://nusenu.github.io/OrNetStats/#tor-version-distribution-relays
> https://nusenu.github.io/OrNetStats/#end-of-life-relays-share

Thanks, this is quite helpful!  Is it okay if I add links to these
from the wiki page?

-- 
Nick
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] proposal for "tor-relay" well-known URI

[Nick Mathewson]

On Mon, Aug 17, 2020 at 11:19 AM David Goulet  wrote:
>
> On 14 Aug (12:17:50), nusenu wrote:
> > Hi,
> >
> > I'm submitting this as a proposal according to:
> > https://github.com/torproject/torspec/blob/master/proposals/001-process.txt
> >
> > I made a PR request for you:
> > https://github.com/torproject/torspec/pull/129/files
>
> Thanks nusenu!
>
> FYI, the merge request to tor-spec.git is here:
> https://gitlab.torproject.org/tpo/core/torspec/-/merge_requests/3
>
> Cheers!
> David
>

This is now proposal 326.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Update to Proposal 316: FlashFlow

[Nick Mathewson]

On Fri, Sep 4, 2020 at 8:26 AM Matt Traudt  wrote:
>
> Hi all
>
> I polished up the FlashFlow proposal based on the feedback provided by
> Teor, Nick, and Mike. I converted it to markdown, and pasted the new
> text at the bottom of this email. The updated proposal is also in my
> fork of torspec is on gitlab [0]; the branch with the changes is
> "flashflow-revision".

Merged to torspec.

-- 
Nick
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Can "ExcludeNodes" be used multiple times in torrc?

[Nick Mathewson]

On Mon, Aug 10, 2020 at 6:13 PM nusenu  wrote:
>
> Hi,
>
> it is not clear to me whether ExcludeNodes and other similar options 
> (ExcludeExitNodes)
> can be used multiple times in a torrc file.
>
> Are all of the lines merged like multiple "MyFamily" lines or
> how does it behave?

No, only one is recognized.

If you try to start tor with ExcludeNodes set twice, it should say
something like:

Aug 11 17:14:36.928 [warn] Option 'ExcludeNodes' used more than once;
all but the last value will be ignored.

I'd welcome patches to clarify this in the documentation, or to allow
this option to repeat.

-- 
Nick
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] How do Ed25519 relay IDs look like?

[Nick Mathewson]

On Tue, Aug 4, 2020 at 6:41 PM nusenu  wrote:
>
> nusenu:
> > I'll wait until you (Tor developers) decided on the final naming and format
>
> Is there any interest to move this topic forward to come to some decision
> in the near future? (before the end of the month)

I don't think that'd be too hard.

> Here is a short summary of what opinions I observed for this topic (naming 
> and format
> for Ed25519 identities) so far:
>
> Naming proposals for relay Ed25519 identities:
> 
>
> 'v2 fingerprints' (Damian)
>
> "ed25519 identity" or even just "identity" (nickm)
>
>
> Output format the Ed25519 relay IDs:
> 
>
> base64 - 43 characters long (nickm)
>   this is problematic due to the "/" sign (Damian)
> hex - 64 characters long (Damian)
>   "/" is problematic for DirPort urls, GETINFO commands, etc (Damian)
> isn't there urlencoding for URLs? (nusenu)
> base64urlsafe - 43 characters long (nusenu)
>
> I hope we can agree to use the same format in all places.
>
> How does the decision process looks like in general in the Tor Project?

I think right now Tor uses unpadded base64 in most internal formats,
but it doesn't actually use those in the user interface anywhere, so
we could just use base64urlsafe (per rfc4648 section 5) for the user
interface.

I would be fine with standardizing that for our API, but I'd want to
write a proposal for it first.  It wouldn't have to be long.  We'd
want to describe other places where we currently use regular base64
for 256-bit keys, and say whether we should/shouldn't accept and emit
url-safe identifiers there instead.

We should specify that there are no spaces, that the padding "="
characters are removed, and that even though the format as given can
handle 43*6==258 bits, the last two bits must be set to 0, since these
are only 256-bit identifiers.

We should also _probably_ specify some canonical encoding for a pair of keys.

-- 
Nick
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Safe Alternative Uses of Onion Service Keys

[Nick Mathewson]

but it's trivial to verify the certificate if you know what the

On Mon, Aug 10, 2020 at 9:00 AM Nick Mathewson  wrote:
>
> On Wed, Jul 29, 2020 at 1:15 AM Matthew Finkel  wrote:
> >
> > Hello everyone,
>
> Hi, Matt!
>
> There's a part of this that I'm still trying to figure out:
>
> > The safest usage of the long-term keys for alternative purposes I see
> > appears to be by deriving a (fixed/deterministic) blinded key pair using
> > the same scheme that Tor uses, and signing/verification simply follow
> > the same process as Tor, except the derived keys need not rotate
> > periodically (is this true?). The derived key should be used for
> > certifying a freshly generated ed25519 key, which is used in the
> > application protocol. For example, if I want to use a key for code
> > signing such that it is bound to my onion service key, then I could
> > derive a certifying key by following Tor's derivation scheme, by
> > substituting:
> >
> >   BLIND_STRING = "Derive temporary signing key" | INT_1(0)
> >   N = "key-blind" | INT_8(period-number) | INT_8(period_length)
> >
> > with
> >
> >   BLIND_STRING = "Derive code signing key" | INT_1(0)
> >   N = "code-sigining-key-blind" | "v0" | "-MM-DD" |  
> > INT_8(validity_period)
>
> In the case of v3 onion services, 'period-number' comes from the
> current time, and 'period-length' comes from the consensus, so it's
> easy for the client to know what parameters to use when deriving the
> key.
>
> But how is the party that relies on the derived key supposed to know
> what values were used for "-MM-DD" and "validity period" in this
> case?  It seems like those two values would need to be shipped along
> with the key, which could make for logistical issues.
>
> I guess in the case of an X.509 certificate, we could use the
> validAfter  and validUntil fields to set the  parameters -- though
> it's a little weird to have to look at _just_ those fields to see what
> the signing key is supposed to be.
>
> Maybe, for an X.509 cert, we could have
>
>   N = "x509 onion key derivation" | H(unsigned-certificate)
>
> where `unsigned-certificate` is a DER-encoded TBSCertificate (the part
> of the certificate that's signed).  That way, we get a separate
> blinded key for each certificate.
>
>  [...]
> > The above process seems like a lot to ask from application developers.
> > Can we make it easier for them?
> >
> > Open questions:
> >
> >  1) Going back to the long-term secret key, can LH and RH be used
> > directly in EdDSA without reducing the security and unlinkability of
> > the blinded keys?
>
> From a practical point of view: Forcing the signer to keep LH and RH
> online reduces the security of the protocol IMO; the current protocol
> is designed so that the onion service doesn't need to keep its
> long-term identity key online.
>
> Also note that if we care about keeping the primary identity key
> offline, LH' and RH' shouldn't be used directly!  Although it is not
> possible to derive a long-term public key from a blinded public key,
> it IS possible to derive a long-term _private_ key from a blined
> private key.  That's why in the v3 onion service system, blinded
> private keys aren't kept for any longer than needed in order to
> certify a short-term randomly generated signing key.
>
> >  2) Should other use cases of the long-term keys only derive distinct
> > (blinded) keys, instead of using the long-term keys directly?
>
> IMO yes.
>
> >  3) If other use cases should only use derived keys, then is there an
> > alternative derivation scheme for when unlinkability between derived
> > keys is not needed (without reducing the security properties of the
> > onion service blinded keys), and is allowing linkability
> > useful/worthwhile?
> >
> >  4) Is the above example derivation scheme safe if different
> > applications tweak the above prefix strings in similar ways?
> >
> >  5) Should Tor simply derive one blinded key that can be used by all
> > alternative applications? Is that safe?
>
> Yes, but only if that key will never ever need to rotate.
>
> Here's a suggestion -- what if we come up with a single profile for
> this kind of key derivation, to be used in X.509.  Along with it, we
> could have a nice small library that can generate and verify the
> "root" certificates (the ones signed by a blinded key).
>
> What important parts of the application space would be left unaddressed by 
> that?

Like, here'

Re: [tor-dev] Safe Alternative Uses of Onion Service Keys

[Nick Mathewson]

On Wed, Jul 29, 2020 at 1:15 AM Matthew Finkel  wrote:
>
> Hello everyone,

Hi, Matt!

There's a part of this that I'm still trying to figure out:

> The safest usage of the long-term keys for alternative purposes I see
> appears to be by deriving a (fixed/deterministic) blinded key pair using
> the same scheme that Tor uses, and signing/verification simply follow
> the same process as Tor, except the derived keys need not rotate
> periodically (is this true?). The derived key should be used for
> certifying a freshly generated ed25519 key, which is used in the
> application protocol. For example, if I want to use a key for code
> signing such that it is bound to my onion service key, then I could
> derive a certifying key by following Tor's derivation scheme, by
> substituting:
>
>   BLIND_STRING = "Derive temporary signing key" | INT_1(0)
>   N = "key-blind" | INT_8(period-number) | INT_8(period_length)
>
> with
>
>   BLIND_STRING = "Derive code signing key" | INT_1(0)
>   N = "code-sigining-key-blind" | "v0" | "-MM-DD" |  
> INT_8(validity_period)

In the case of v3 onion services, 'period-number' comes from the
current time, and 'period-length' comes from the consensus, so it's
easy for the client to know what parameters to use when deriving the
key.

But how is the party that relies on the derived key supposed to know
what values were used for "-MM-DD" and "validity period" in this
case?  It seems like those two values would need to be shipped along
with the key, which could make for logistical issues.

I guess in the case of an X.509 certificate, we could use the
validAfter  and validUntil fields to set the  parameters -- though
it's a little weird to have to look at _just_ those fields to see what
the signing key is supposed to be.

Maybe, for an X.509 cert, we could have

  N = "x509 onion key derivation" | H(unsigned-certificate)

where `unsigned-certificate` is a DER-encoded TBSCertificate (the part
of the certificate that's signed).  That way, we get a separate
blinded key for each certificate.

 [...]
> The above process seems like a lot to ask from application developers.
> Can we make it easier for them?
>
> Open questions:
>
>  1) Going back to the long-term secret key, can LH and RH be used
> directly in EdDSA without reducing the security and unlinkability of
> the blinded keys?

>From a practical point of view: Forcing the signer to keep LH and RH
online reduces the security of the protocol IMO; the current protocol
is designed so that the onion service doesn't need to keep its
long-term identity key online.

Also note that if we care about keeping the primary identity key
offline, LH' and RH' shouldn't be used directly!  Although it is not
possible to derive a long-term public key from a blinded public key,
it IS possible to derive a long-term _private_ key from a blined
private key.  That's why in the v3 onion service system, blinded
private keys aren't kept for any longer than needed in order to
certify a short-term randomly generated signing key.

>  2) Should other use cases of the long-term keys only derive distinct
> (blinded) keys, instead of using the long-term keys directly?

IMO yes.

>  3) If other use cases should only use derived keys, then is there an
> alternative derivation scheme for when unlinkability between derived
> keys is not needed (without reducing the security properties of the
> onion service blinded keys), and is allowing linkability
> useful/worthwhile?
>
>  4) Is the above example derivation scheme safe if different
> applications tweak the above prefix strings in similar ways?
>
>  5) Should Tor simply derive one blinded key that can be used by all
> alternative applications? Is that safe?

Yes, but only if that key will never ever need to rotate.

Here's a suggestion -- what if we come up with a single profile for
this kind of key derivation, to be used in X.509.  Along with it, we
could have a nice small library that can generate and verify the
"root" certificates (the ones signed by a blinded key).

What important parts of the application space would be left unaddressed by that?

-- 
Nick
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] How do Ed25519 relay IDs look like?

[Nick Mathewson]

On Sat, Aug 1, 2020 at 6:10 AM nusenu  wrote:
>
> nusenu:
> >> The only question that came up was: Will there be two types of relay 
> >> fingerprints
> >> in the future (Ed25519)?
> >
> > I assume the correct proposal for the Ed25519 keys is this:
> > https://gitweb.torproject.org/torspec.git/tree/proposals/220-ecc-id-keys.txt
> >
> > I'm wondering what kind of format is used for a relay's Ed25519 ID in tor?
> >
> > The spec says base64:
> >
> >>When an ed25519 signature is present, there MAY be a 
> >> "master-key-ed25519"
> >>element containing the base64 encoded ed25519 master key as a single
> >>argument.  If it is present, it MUST match the identity key in
> >>the certificate.
> >
> > examples:
> > grep master-key-ed 2020-07-28-19-05-00-server-descriptors |head -2
> >
> > master-key-ed25519 clT/2GWmTY/qU5TBGaudAIjOUUxUdKhMY/Q5riK6G2E
> > master-key-ed25519 qDI9PbwtiKzpR9phLnWI99uimdwNW8+l9c7hDoWV9dQ
> >
> > Is this the canonical format you use when referring to a relay's Ed25519 
> > identity?
>
> I looked at what stem does in this area [1].
> It uses the more accurate name "ed25519_master_key" instead of Ed25519 ID
> and contains the above mentioned base64 encoded Ed25519 public master key
> so I assume this is the canonical format since I didn't see any other 
> representation.

I'd like to use "ed25519 identity" or even just "identity" here going
forward.  While it might make sense to use other names when describing
it in relation to other keys, when talking about the relay, it is an
identity key.

The base64-encoded form is the best one we have; whenever we output a
key, we use that format.

> > What command does a relay operator need to run to find out
> > his relay's Ed25519 ID on the command line?
>
> base64 encoding (parts of) the ed25519_master_id_public_key
> file, provides the same output as in master-key-ed25519 descriptor lines
> but I didn't find a spec for that key file to confirm the try and error 
> approach
> or a tor command to simply output the ed25519_master_key public key in base64 
> format.

I'd like to add such a command, as well as support for using ed25519
keys in more places in the UI and the control API.  I'm not going to
have time for a while, though, but if anybody would be interested in
hacking this together, I can point to some of the places in the code
you'd need to change.

best wishes,
-- 
Nick
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Proposal 325: Packed relay cells: saving space on small commands

[Nick Mathewson]

On Fri, Jul 10, 2020 at 2:07 PM Ian Goldberg  wrote:
>
> On Fri, Jul 10, 2020 at 01:52:00PM -0400, Nick Mathewson wrote:
> > After receiving a packed relay cell, the relay know that the client
>
> typo: "know" -> "knows"

thanks!

> > struct relay_header {
> >u1 stream_id_included; // Is the stream_id included?
> >u6 relay_command; // as before
> >u9 relay_data_len; // as before
> >u8 optional_stream_id[]; // 0 bytes or two bytes.
> > }
> >
> > Alternatively, you can view the first three fields as a 16-bit
> > value, computed as:
> >
> > (stream_id_included<<15) | (relay_command << 9) | (relay_data_len).
>
> Where everything is big-endian, both at the byte and bit level?  (Is
> that specified at some higher level in the specs?  I forget.)

Early in the spec, we say that we always use network byte order. I
don't think we ever actually spec a bit order.

> > If the optional_stream_id field is not present, then the default
> > value for the stream_id is computed as follows.  We use stream_id 0
> > for any command that doesn't take a stream ID.  For commands that
> > _do_ take a steam_id, we use whichever nonzero stream_id appeared
> > last in this cell.
>
> Do you mean "last in this cell" as in "the one closest to the end of the
> cell" or as in "the one that appeared closest to, but before, this relay
> command header"?

Ah; the latter. Maybe "most recently" would be more clear?

yrs,
-- 
Nick
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Proposal 325: Packed relay cells: saving space on small commands

[Nick Mathewson]

```
Filename: 325-packed-relay-cells.md
Title: Packed relay cells: saving space on small commands
Author: Nick Mathewson
Created: 10 July 2020
Status: Draft
```

# Introduction

In proposal 319 I suggested a way to fragment long commands across
multiple RELAY cells.  In this proposal, I suggest a new format for
RELAY cells that can be used to pack multiple relay commands into a
single cell.

Why would we want to do this?  As we move towards improved
congestion-control and flow-control algorithms, we might not want to
use an entire 498-byte relay payload just to send a one-byte
flow-control message.

We already have some cases where we'd benefit from this feature.
For example, when we send SENDME messages, END cells, or BEGIN_DIR
cells, most of the cell body is wasted with padding.

As a side benefit, packing cells in this way may make the job of the
traffic analyst a little more tricky, as cell contents become less
predictable.

# The basic design

Let's use the term "Relay Message" to mean the kind of thing that a
relay cell used to hold.  Thus, this proposal is about packing
multiple "Relay Messages" in to a cell.

I'll use "Packed relay cell" to mean a relay cell in this new
format, that supports multiple messages.

I'll use "client" to mean the initiator of a circuit, and "relay" to
refer to the parties through who a circuit is created.  Note that
each "relay" (as used here) may be the "client" on circuits of its own.

When a relay supports relay message packing, it advertises the fact
using a new Relay protocol version.  Clients must opt-in to using
this protocol version (see XXX below) before they can send any
packed relay cells, and before the relay will send them any packed
relay cells.

When packed cells are in use, multiple cell messages can be
concatenated in a single relay cell.

Only some relay commands are supported for relay cell packing,
listed here:
  - SENDME
  - DROP
  - DATA
  - BEGIN
  - BEGIN_DIR
  - END
  - CONNECTED
  - PADDING_NEGOTIATE
  - PADDING_NEGOTIATED

If any relay message with a relay command _not_ listed above appears
in a packed relay cell with another relay message, then the
receiving party MUST tear down the circuit.

(Note that relay cell fragments (proposal 319) are not supported for
packing.)

The command byte "0" is now used to explicitly indicate "end of
cell".  If the byte "0" appears after a relay message, the rest of
the cell MUST be ignored.

When generating RELAY cells, implementations SHOULD (as they do
today) fill in the unused bytes with four 0-valued bytes, followed by
a sequence of random bytes up to the end of the cell.  If there are
fewer than 4 unused bytes at the end of the cell, those unused bytes
should all be filled with 0-valued bytes.

# Negotiation and migration

After receiving a packed relay cell, the relay know that the client
supports this proposal: Relays SHOULD send packed relay
cells on any circuit on which they have received a packed relay
cell.  Relays MUST NOT send packed relay cells otherwise.

Clients, in turn, MAY send packed relay cells to any relay whose
"Relay" subprotocol version indicates that it supports this
protocol.  To avoid fingerprinting, this client behavior should
controlled with a tristate (1/0/auto) torrc configuration value,
with the default set to use a consensus parameter.

The parameter is:

"relay-cell-packing"

Boolean: if 1, clients should send packed relay cells.
(Min: 0, Max 1, Default: 0)

To handle migration, first the parameter should be set to 0 and the
configuration setting should be "auto".  To test the feature, individual
clients can set the tristate to "1".

Once enough clients have support for the parameter, the parameter
can be set to 1.


# A new relay message format

(This section is optional and should be considered separately; we
may decide it is too complex.)

Currently, every relay message uses 5 bytes of header to hold a
relay command, a length field, and a stream ID.  This is wasteful:
the stream ID is often redundant, and the top 7 bits of the length
field are always zero.

I propose a new relay message format, described here (with `ux`
denoting an x-bit bitfield).  This format is 2 bytes or 4 bytes,
depending on its first bit.

struct relay_header {
   u1 stream_id_included; // Is the stream_id included?
   u6 relay_command; // as before
   u9 relay_data_len; // as before
   u8 optional_stream_id[]; // 0 bytes or two bytes.
}

Alternatively, you can view the first three fields as a 16-bit
value, computed as:

(stream_id_included<<15) | (relay_command << 9) | (relay_data_len).

If the optional_stream_id field is not present, then the default
value for the stream_id is computed as follows.  We use stream_id 0
for any command that doesn't take a stream ID.  For com

Re: [tor-dev] Implementing an embeddable C++ Tor client, advice requested

[Nick Mathewson]

On Fri, Jun 19, 2020 at 3:59 PM The Paranoia Project
 wrote:
>
> Hello everyone, I'm a long time user of Tor, first time poster here.
>
> Over the last few months, I have been working on a light weight C++ client 
> only implementation of the Tor protocol, intended to be used as an embedded 
> library in other applications. It is now at the stage where it can complete 
> bootstrapping, build circuits, as well as connect to services / host hidden 
> services (v3). I have been building this primarily off the spec documents 
> (which have generally been extremely helpful), and well as the assistance of 
> stepping through the official Tor implementation when needed for 
> troubleshooting and to confirm specifics. Before I release this to the wider 
> world, I'd like to confirm a few points that may not be explicitly stated in 
> the specs.

Hello, P!  It'll be neat to have a look at this when it comes out;
building a Tor implementation is a lot of work.

> 1. In general, what are the things to look out for when implementing the Tor 
> protocol beyond "making it work" and validating all data (signatures, 
> timestamps, etc)? One thing I'm concerned about is the risk of fingerprinting 
> where the spec does not completely specify behaviour, e.g. the order in which 
> link specifiers are passed in an extend cell, exact criteria for when 
> circuits are explicitly destroyed etc. (I'm very excited to see the proposals 
> around CBOR on this point which would help greatly with knowing that a 
> canonical data representation was used).

There are a lot of these, I'm afraid, and they're not all perfectly
documented.  Some of the trickier ones are about "being kind to the
network" -- not making too many circuits, not over-using resources
when idle, and so on.  These are under-documented, but I believe Roger
has talked a few times about starting a document to collect these.
Roger, do you remember if this ever went anywhere, and produced a
draft or something?

For your first few versions, I'd suggest that your best bet is to
label your software loudly as experimental and alpha, since there will
almost certainly be ways to distinguish your software and surprising
bugs. Trying to be completely indistinguishable is probably
impossible, due to timing issues at least -- about the best you can do
is try to avoid easy ways to passively distinguish your software.

In general, I wouldn't mind taking patches to enhance the specs by
describing a preferred behavior whenever there is more than on
possible behavior.

> 2. When it comes to bootstrapping, the official implementation appears to 
> favour accessing directories via plaintext HTTP rather than connecting on the 
> OR port and using create fast / begin dir. What is the motivation for using 
> the plaintext option (and for that matter, having a plaintext http service 
> open at all)?. While the OR will learn just as much about the client 
> regardless, it seems like the default plaintext access to directory 
> information unnecessarily gives away details of how clients engage with the 
> Tor network to third parties.

That isn't right.  It's preferred for clients to download directory
material over the ORPort via begindir.  Plaintext DirPorts are
supposed to be used by relays and authorities only.

What part of the spec or the implementation says that the plaintext
dirport should be preferred?  I'd like to correct that.

> 3. When using bridges and in particular pluggable transports, how is the 
> client intended to safely bootstrap in the cold start case where it does not 
> know up front which bridge/relay it will be connected to (e.g. when using 
> Snowflake)? The RSA identity can be accepted in blind faith based on the Tor 
> handshake, and it's then possible to get the full details with create fast / 
> begin dir, but how does a client know that it has been connected to a bridge 
> that is "blessed" by the Tor network rather than a MITM actor?

Bridge addresses and identities need to be discovered out-of-band, by
some means like bridgedb.torproject.org, personal communication, or
bundling with software.  The provenance of this information is the
only way to tell whether you're getting a likely-to-be-good bridge or
likely-to-be-run-by-your-enemy bridge.

> 4. Finally, if anyone reading has been involved with or close to the 
> development of other unofficial Tor implementations, what are the lessons 
> learned on this front? I'm aware of among others Orchid (updated last in 
> 2016), node-Tor (does not implement ECC) and torpy (does not implement hidden 
> services v3). What makes these fail / stall?

I'll let developers answer here -- part of the issue is that it can be
hard to maintain feature parity over time.

For a longer list of implementations, see
https://gitlab.torproject.org/legacy/trac/-/wikis/doc/ListOfTorImplementations
[warning -- wiki migration in progress].

best wishes,
-- 
Nick
___
tor-dev mailing list

Re: [tor-dev] Reproducing #33598 "chutney does not fail on some SOCKS errors"

[Nick Mathewson]

On Mon, Jun 15, 2020 at 8:53 AM c  wrote:
>
> I asked on ticket https://bugs.torproject.org/33598 how to reliably
> reproduce SOCKS errors so that it would be easier to determine when the
> underlying issue is properly fixed, rather than running into the
> possibility of race conditions (as the current workaround depends on
> "waiting long enough" for connections to succeed most of the time).
>
> In #tor-dev I believe it was arma who pointed me toward the right
> direction, but that still leaves me with unresolved questions. It was
> suggested that I can attempt SOCKS connections to an invalid host/port
> versus a valid one, in order to have a reliable failure case for
> testing. But in the context of Chutney I believe we only want to
> attempt local connections, correct? So either attempting connection to
> 127.0.0.0/8 on a known-closed port, or perhaps more simply 0.0.0.0 on
> any port, would be a reliable case to use, correct?

This seems plausible, though 0.0.0.0:x will just be the local computer as well.

Maybe you could try routing to a known-unassigned address, or one that
Tor simply won't support, like the one from RFC ?  For IPv4 I
think maybe a multicast or known-unassigned prefix might have similar
results.

> Also from my comment on #33598:
>
> > Assuming workaround is at Traffic.py:441? I see the timeout was
> > adjusted in 95ce144c which has more changes than just that line.

Hm, that might be where the "5.0" is coming from, yeah.

> and
>
> > will decreasing the timeout back to 0.2 be enough to encourage
> > failure?

So the way that timeout works, I think, is that it controls how long
asyncore will go without any events before it exits.  The while loop
around asyncore.loop() is what makes it retry over and over.

I think the problem here might be that the while loop keeps going
until the time reaches "end", or until "self.tests.all_done()".  This
might mean that we need to adjust the socks handling code instead, so
it detects socks refusals and treats them as the test being "done",
but failed.

In theory, the code handles this in Source.collect_incoming_data(),
where it looks for a socks response, and compares it to an expected
value.  But I guess chutney is restarting the connection again, or not
treating this as a test failure?

> Last question: I looked for a bit, but where is Chutney actually
> initiating SOCKS connections to Tor during tests? I still find it hard
> to follow especially when I am going in blind for much of this.

It's in Traffic.py, Source.handle_connect(), in this line:

self.push(socks_cmd(self.server))

best wishes,
-- 
Nick
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] #32888 IPv6 and 'Address' option

[Nick Mathewson]

On Sun, Jun 14, 2020 at 6:42 AM c  wrote:
>
> Trac  describes
> that "we should also log [for] IPv4 and IPv6: the Address torrc option"
> and as tor(1) states, 'Address' takes only an IPv4 address and port.
>
> So what is suggested here? Make 'Address' support IPv6 to be consistent
> with other options ('ORPort', 'SocksPort', 'TransPort', 'DNSPort', and
> the like all support multiple declarations per torrc, both IPv4 and
> IPv6)? If we will have IPv6-only relays in the future, this option
> would make quite a bit of sense.

I think what you're looking for here is section 3.2.1 of proposal 312,
which covers how Tor should determine its IPv6 addresses.  The plan
described there is to allow Address to appear up to twice.

Link:

https://gitweb.torproject.org/torspec.git/tree/proposals/312-relay-auto-ipv6-addr.txt#n245

> Since I'm on the topic of #32888, I went ahead and pushed the IPv6
> logging portion to  and
> have tested with 'ORPort [::1]:9001' and it seems to log successfully.
> Not a big change, so I expect it to be an easy merge.

Sounds good!  I'll take a look at it soon.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Walking onions specifications: Project wrap-up

[Nick Mathewson]

On a grant from the zcash foundation, I've been working on a
full specification for the Walking Onions design.  This is the last update
for the spec project.

My previous updates are linked below:

 Week 1:
   formats, preliminaries, git repositories, binary diffs,
   metaformat decisions, and Merkle Tree trickery.

   https://lists.torproject.org/pipermail/tor-dev/2020-March/014178.html

 Week 2:
   Specifying details of SNIP and ENDIVE formats, in CDDL.

   https://lists.torproject.org/pipermail/tor-dev/2020-March/014181.html

 Week 3:
   Expanding ENDIVEs into SNIPs.

   https://lists.torproject.org/pipermail/tor-dev/2020-March/014194.html

Week 4:
   Voting (part 1) and extending circuits.

   https://lists.torproject.org/pipermail/tor-dev/2020-March/014207.html

Week 5:
   Voting (part 2)

   https://lists.torproject.org/pipermail/tor-dev/2020-April/014218.html

Week 6:
   Editing, voting rules, and client behavior (part 1)

   https://lists.torproject.org/pipermail/tor-dev/2020-April/014222.html

Week 7:
   Exit policies: How do they work?

   https://lists.torproject.org/pipermail/tor-dev/2020-April/014232.html

Week 8:
   Onion services, relay honesty, migration and families

   https://lists.torproject.org/pipermail/tor-dev/2020-April/014255.html

Week 9:
   (There was no week 9)

Week 10:
   Pushing towards completion

   https://lists.torproject.org/pipermail/tor-dev/2020-May/014281.html

== Since the last update

The last couple weeks of the walking onions project were mostly the
"fiddly bits" left over from previous work.  I had to edit for
consistency and clarity, move text around, add specifications for
missing details, and so on.  There were parts of the voting design
that didn't actually work as written, and needed to get tweaked
around.

Index voting was tricky for a few reasons: notably, index weights
are a complex function of bandwidths, which themselves are voted on.
I had hoped that we could move the algorithms we use for weighting
out of the consensus algorithm and into the authorities, but for
now, I think we're stuck with having them as yet another thing
authorities need to implement identically.  At least the new consensus
system itself is extensible, so we have a clear path forward to a
better consensus approach for indices (if we ever figure one out).

I also needed to figure out handling for VoterCerts: I had done the
first version of the spec for how they should appear in parameters
documents before I actually figured out how voting would work, and
the two approaches weren't compatible.  This took some revision, but
I think it should work out now.

Similarly, I had known that we'd need to use the "detached
signatures" mechanism from the existing consensus protocol, but the
Walking Onions spec didn't actually allow for this.  Fortunately,
this was an easy chance to make.

And overall, I went through a pretty large number of places in the
document where there were "" comments indicating things I had to
come back to later.

With that, the first version Walking Onions proposal was ready to go
out as [PROP323].  You can see a rendered version of it over at
[PROP323-RENDERED], though there will be a better URL for that in
the future.

I also put out the remaining side related proposals ([PROP321] for
families, and [PROP322] for link specifiers for directory ports).

== Project recap

When I first described Walking Onions [PROP300], I had only the
general outlines of a design for a more efficient Tor network: a
couple of tricks to allow clients to continue building secure paths
through the network without having to know a list of relays on the
network.  But I had known there would be remaining problems to
solve, and listed them in that proposal.

Thanks to this grant, I've been able to flesh out Walking Onions to
a specified design that I think we could actually build.  In doing
so, I ran into the typical specification curse, where each change
led to a need for other changes.  Notably:

  * The need for a new kind of directory-like structure led to the
need for a more flexible and parsable metadocument format, and a
more generic voting algorithm for authorities.

  * When designing the new meta-format, I found a better approach
for handling expiration and clock synchronization between
clients and relays that should allow us to be a little more lax
with skew, while better resisting attacks based on serving
obsolete information.

  * The search for a reasonable solution to exit policies led to
a hybrid extraction/designed approach for clustering exit ports
by their correlatedness, then designing exit policies around
semantic similarity in port types.

  * When enumerating the various ways that clients need to be able
to request SNIPs and circuits, I found a need for a span-based
SNIP query request, which I hadn't previously known that we'd
need.  This simplifies some of our logic.

  * (and more; please see previous updates.)

The 

Re: [tor-dev] Proposal 316: FlashFlow: A Secure Speed Test for Tor (Parent Proposal)

[Nick Mathewson]

On Thu, Apr 23, 2020 at 2:48 PM Matt Traudt  wrote:
>

Hi!  I've got some comments on the FlashFlow proposal; I'll start with
the ones that I think are most important, so that we can try to get
them out of the way.

First off, I'm concerned about the approach where measurers get to
consume a certain amount of bandwidth, with only a set fraction left
to devote to the background traffic.  It seems like a hostile set of
measurers could use this authority to introduce traffic patterns on
the network to assist in traffic analysis.  In general, having regular
scheduled and visible changes in relay capacity seem to me like they'd
help out traffic analysis a good deal.

Second, the "MSM_BG" information type also seems like a serious
traffic analysis risk.  It is, literally, telling the measurers a
report of how much traffic was sent each second on other connections.
Previously we decided that a much coarser summary than this was too
much information to publish in bandwidth-history lines, and I'm
worried not to see any analysis here.

{In both of the above cases we might say, "well, an attacker could do
that anyway!"  But to get the traffic information, an attacker would
need to compromise the upstream connection, and to introduce traffic
spikes the attacker would need to risk detection.  This proposal as
written would make both of these traffic analysis opportunities an
expected part of the infrastructure, which seems not-so-good to me.}

Third, I don't understand why we're using cell crypto here but we
aren't using RELAY cells or (apparently?) circuits.  Since TLS is
already in play, we'll already be measuring the relays' encryption
performance.  But if we do decide that cell crypto is needed, then
it's way easier to get that crypto happening if there are circuits
involved.  I think there's been some discussion of that on IRC; I'd
suggest that we try to make that work if we can.

Fourth, this approach to authenticating echo cell contents seems
needlessly complicated.  Instead of using random contents and
remembering a fraction of cells, it would make more sense for
measurers to use a keyed pseudorandom stream function to generate the
cells, and to verify the contents of all the cells as they come back
in.  (AES128-CTR and ChaCha8 and SHAKE128 all have nice properties
here.)

Fifth, using IP addresses for identification is NOT something we do on
the production network.  I think we should authenticate measurers by
identity key, not by IPv4 address (as is happening here, unless I
misunderstand.)

yrs,
-- 
Nick
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Moving key material out of the main tor process

[Nick Mathewson]

On Wed, May 20, 2020 at 11:46 AM Linus Nordberg  wrote:
>
> Hi,
>
> tl;dr How to move key material out of tor?
>
> ## The idea of a vault component
>
> ahf and others in the network team have been discussing the
> possibility of a "vault" component in tor, for moving private keys out
> of the tor process. Separating secret key material from the code
> handling data from the network seems valuable and providing a
> component making different implementations "pluggable" would allow for
> anyone to use their favourite technology without touching the tor
> code base. Examples are local trusted execution environments like Intel
> SGX and Arm TrustZone and various HSM's and security keys/tokens.
>
> One way of implementing this would be to define a protocol, for a
> vault component to talk to a daemon running on the same host as tor,
> over some IPC mechanism. This protocol would allow tor to request a
> signature over a hash, or a document, in a certain key. Whether the
> daemon has access to the key material or has to forward the request to
> a separate device or hardware component is irrelevant to the protocol
> and the vault component.
>
> Even if the design focuses on signatures it should probably take
> encryption and decryption into account, to be added later.


Hi!

I'm also +1 on this idea, and I think you're correct to hint that the
"vault" or "separate daemon" part doesn't have to be a tor-specific
tool.  Moving responsibilities out of the core Tor process should help
us make it harder for keys to leak.

One issue with the ssh-agent protocol as I see it is that it isn't
originally designed for decryption or for high-volume usage.  If we
want to support those in the future, we'll need to make sure that we
have an extension path for them in whatever vault tool we're using.

yrs,
-- 
Nick
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Proposal 322: Extending link specifiers to include the directory port

[Nick Mathewson]

```
Filename: 322-dirport-linkspec.md
Title: Extending link specifiers to include the directory port
Author: Nick Mathewson
Created: 27 May 2020
Status: Open
```

## Motivation

Directory ports remain the only way to contact a (non-bridge) Tor
relay that isn't expressible as a Link Specifier.  We haven't
specified a link specifier of this kind so far, since it isn't a way
to contact a relay to create a channel.

But authorities still expose directory ports, and encourage relays
to use them preferentially for uploading and downloading.  And with
Walking Onions, it would be convenient to try to make every kind of
"address" a link specifier -- we'd like want authorities to be able
to specify a list of link specifiers that can be used to contact
them for uploads and downloads.

> It is possible that after revision, Walking Onions won't need a way
> to specify this information.  If so, this proposal should be moved
> to "Reserve" status as generally unuseful.

## Proposal

We reserve a new link specifier type "dir-url", for use with the
directory system.  This is a variable-length link specifier, containing
a URL prefix.  The only currently supported URL schema is "http://;.
Implementations SHOULD ignore unrecognized schemas.  IPv4 and IPv6
addresses MAY be used directory; hostnames are also allowed.
Implementations MAY ignore hostnames and only use raw addresses.

The URL prefix includes everything through the string "tor" in the
directory hierarchy.

A dir-url link specifier SHOULD NOT appear in an EXTEND cell;
implementations SHOULD reject them if they do appear.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Proposal 321: Better performance and usability for the MyFamily option (v2)

[Nick Mathewson]

```
Filename: 321-happy-families.md
Title: Better performance and usability for the MyFamily option (v2)
Author: Nick Mathewson
Created: 27 May 2020
Status: Open
```

## Problem statement.

The current family mechanism allows well-behaved relays to
identify that they all belong to the same 'family', and should
not be used in the same circuits.

Right now, families work by having every family member list every
other family member in its server descriptor.  This winds up using
O(n^2) space in microdescriptors and server descriptors. (For RAM,
we can de-duplicate families which sometimes helps.)  Adding or
removing a server from the family requires all the other servers to
change their torrc settings.

The is growth in size is not just a theoretical problem. Family
declarations currently make up a little over 55% of the
microdescriptors in the directory--around 24% after compression.
The largest family has around 270 members.  With Walking Onions, 270
members times a 160-bit hashed identifier leads to over 5 kilobytes
per SNIP, which is much greater than we'd want to use.

This is an updated version of proposal 242.  It differs by clarifying
requirements and providing a more detailed migration plan.

## Design overview.

In this design, every family has a master ed25519 "family key".  A node
is in the family iff its server descriptor includes a certificate of its
ed25519 identity key with the family key.  The certificate
format the one in the tor-certs.txt spec; we would allocate a new
certificate type for this usage.  These certificates would need to
include the signing key in the appropriate extension.

Note that because server descriptors are signed with the node's
ed25519 signing key, this creates a bidirectional relationship
between the two keys, so that nodes can't be put in families without
their consent.

## Changes to router descriptors

We add a new entry to server descriptors:

"family-cert" NL
"-BEGIN FAMILY CERT-" NL
cert
"-END FAMILY CERT-".

This entry contains a base64-encoded certificate as described
above.  It may appear any number of times; authorities MAY reject
descriptors that include it more than three times.

## Changes to microdescriptors

We add a new entry to microdescriptors: `family-keys`.

This line contains one or more space-separated strings describing
families to which the node belongs.  These strings MUST be sorted in
lexicographic order.  Clients MUST NOT depend on any particular property
of these strings.

## Changes to voting algorithm

We allocate a new consensus method number for voting on these keys.

When generating microdescriptors using a suitable consensus method,
the authorities include a "family-keys" line if the underlying
server descriptor contains any valid family-cert lines.  For each
valid family-cert in the server descriptor, they add a
base-64-encoded string of that family-cert's signing key.

> See also "deriving family lines from family-keys?" below for an
> interesting but more difficult extension mechanism that I would
> not recommend.

## Relay configuration

There are several ways that we could configure relays to let them
include family certificates in their descriptors.

The easiest would be putting the private family key on each relay,
so that the relays could generate their own certificates.  This is
easy to configure, but slightly risky: if the private key is
compromised on any relay, anybody can claim membership in the
family.  That isn't so very bad, however -- all the relays would
need to do in this event would be to move to a new private family
key.

A more orthodox method would be to keep the private key somewhere
offline, and using it to generate a certificate for each relay in
the family as needed.  These certificates should be made with
long-enough lifetimes, and relays should warn when they are going to
expire soon.

## Changes to relay behavior

Each relay should track which other relays they have seen using the
same family-key as itself.  When generating a router descriptor,
each relay should list all of these relays on the legacy 'family'
line.  This keeps the "family" lines up-to-date with "family-keys"
lines for compliant relays.

Relays should continue listing relays in their family lines if they
have seen a relay with that identity using the same family-key at
any time in the last 7 days.

The presence of this line should be configured by a network
parameter, `derive-family-line`.

Relays whose family lines do not stay at least mostly in sync with
their family keys should be marked invalid by the authorities.

## Client behavior

Clients should treat node A and node B as belonging to the same
family if ANY of these is true:

* The client has descriptors for A and B, and A's descriptor lists B
  in its family line, and B's descriptor lists A in its family line.

* Client A has descriptors for A and B, and they b

Re: [tor-dev] Proposal 320: Removing TAP usage from v2 onion services

[Nick Mathewson]

On Wed, May 13, 2020 at 10:09 AM David Goulet  wrote:
>
> On 11 May (16:47:53), Nick Mathewson wrote:
>
> Hello!
>
> > ```
> > Filename: 320-tap-out-again.md
> > Title: Removing TAP usage from v2 onion services
> > Author: Nick Mathewson
> > Created: 11 May 2020
> > Status: Open
> > ```
> >
> > (This proposal is part of the Walking Onions spec project.  It updates
> > proposal 245.)
> >
> > # Removing TAP from v2 onion services
> >
> > As we implement walking onions, we're faced with a problem: what to do
> > with TAP keys?  They are bulky and insecure, and not used for anything
> > besides v2 onion services.  Keeping them in SNIPs would consume
> > bandwidth, and keeping them indirectly would consume complexity.  It
> > would be nicer to remove TAP keys entirely.
> >
> > But although v2 onion services are obsolescent and their
> > cryptographic parameters are disturbing, we do not want to drop
> > support for them as part of the Walking Onions migration.  If we did
> > so, then we would force some users to choose between Walking Onions
> > and v2 onion services, which we do not want to do.
>
> I haven't read the entire proposal so I won't comment on its technical aspect.
> I was reading and got here and that made me very uncertain about the whole
> proposal itself.
>
> I will propose that we revisit the overall idea of changing v2 here.
>
> I personally think this is the wrong approach. Onion services v2 should be
> deprecated as in removed from the network instead of being offered as a choice
> to the users.
>
> We haven't properly done a deprecation path yet for v2 primarly due to our
> lack of time to do so. But at this point in time, where the network is 100%
> composed of relays supporting v3 now (which took 3+ years to get there), it is
> time for v2 to not be presented as a choice anymore.
>
> It is a codebase that is barely maintained, no new features are being added to
> it and thus moving it to ntor means another at least 3 years of network
> migration. This would mean a major new feature in that deprecated code base...
>
> So thus, I personally will argue that moving v2 to ntor is really not the
> right thing to do. Onion service v2 are, at this point in time, _dangerous_
> choice for the users.

Hi, David!  I'm sympathetic to this point of view: I certainly don't
like carrying around old code either.

If we do decide to finally deprecate v2 onion services, that would be
a significant maintenance burden reduced for us, but we'd have to
handle the transition carefully.  Unlike all the other migrations
we've done, there isn't a drop-in path to get the same functionality
or keep the same identities with v3 onion services.  (And the problem
is that there _can't_ be: the identities are strongly tied to
80-bit-truncated-SHA1 and RSA-1024, and the lack of key blinding makes
them enumerable.)


The main reason I wrote this proposal is this: Any deprecation will
probably cause a few users to stick with the old versions of the code
for as long as they still work on the network, even if those versions
become unsupported and insecure.  (After all, people who listen to our
advice about what is secure and what isn't have already stopped using
v2 onion services.) .

Is it time to start this deprecation?  If so we need to start working
on a timeline, and I agree with Teor that we'd need to figure out how
that timeline would work with any walking onions timeline.

One possible role for this proposal is to be kept in reserve, in case
somebody feels so strongly that they want v2 services to work that
they want to maintain them themselves, or pay for somebody else to do
it.  If so, we can indicate this proposal as "the right way to keep v2
services working without TAP", make it clear that we don't plan to
implement it, and move along.


There are other ways to keep TAP working with walking onions, but they
seem kludgey too, and would also require implementation changes for v2
onion services.

What do others think?

-- 
Nick
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Proposal 320: Removing TAP usage from v2 onion services

[Nick Mathewson]

On Mon, May 11, 2020 at 8:52 PM teor  wrote:
>
> Hi Nick,
>
> > On 12 May 2020, at 06:48, Nick Mathewson  wrote:
> >
> > ## Migration steps
> >
> > First, we implement all of the above, but set it to be disabled by
> > default.  We use torrc fields to selectively enable them for testing
> > purposes, to make sure they work.
>
> Can you expand on the testing plan here?
>
> One of the risks with multi-year migration projects is that unrelated
> changes break the migration, and we don't notice.
>
> For example, you might need to create a chutney network for each
> stage, and run them on every pull request and merge. In our current
> CI, that's 30-60 seconds of extra time per network, or 2-4 extra
> minutes of CI time.
>
> If you need to test different combinations of features for each stage,
> let's try to do that in the same networks. Otherwise, the testing matrix
> expands out very quickly.

I agree here think that the right approach here is to test for the
various ways that we expect the network to exist at a time.  The
trickiest stage of the migration will be the one where some services
support ntor keys and some don't, some clients do and some don't.  If
we add a chutney network for that case specifically and make sure that
all clients can reach all services, we should be fine here.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Proposal 319: RELAY_FRAGMENT cells

[Nick Mathewson]

On Thu, May 14, 2020 at 3:15 PM David Goulet  wrote:
>
> On 11 May (16:47:24), Nick Mathewson wrote:
 [...]
> > # Onion service concerns.
> >
> > We allocate a new extension for use in the ESTABLISH_INTRO by onion 
> > services,
> > to indicate that they can receive a wide INTRODUCE2 cell.  This extension
> > contains:
> >
> > struct wide_intro2_ok {
> >   u16 max_len;
> > }
> >
> > We allocate a new extension for use in the `ESTABLISH_RENDEZVOUS`
> > cell, to indicate acceptance of wide `RENDEZVOUS2` cells.  This
> > extension contains:
> >
> > struct wide_rend2_ok {
> >   u16 max_len;
> > }
> >
> > (Note that `ESTABLISH_RENDEZVOUS` cells do not currently have a an
> > extension mechanism.  They should be extended to use the same
> > extension format as `ESTABLISH_INTRO` cells, with extensions placed
> > after the rendezvous cookie.)
>
> Why would a client need to announce wide cells in the ESTABLISH phase as
> opposed to using protover "Relay=N" ?

This is not for announcing support of wide cells -- this is for
reporting a setting for how wide fragmented cells should be.

> The maximum length of a fragmented cell is capped to 2^16 (u16) so we don't
> really need the establish process to inform us of the maximum expected length
> but rather use the max_len in the first fragment?

This all comes back to an earlier part of the proposal:

Not all lengths up to 65535 are valid lengths for a fragmented
cell.  Any length under 499 bytes SHOULD cause the circuit
to close, since that could fit into a non-fragmented RELAY cell.
Parties SHOULD enforce maximum lengths for cell types that
they understand.

In other words, I'm imagining that there is a maximum length for each
cell type that is much shorter than 65535, even though we're using two
bytes for the length field.

The extension in the establish_intro cell is to tell the intro point
the longest introduce1 cell that it should accept;  this extension in
the establish_rend cell is to tell the rendezvous point the longest
rendezvous1 cell that it should accept.

Another way we could do this would be with a set of network parameters
to describe the maximum length of each fragmented cell.  Do you think
that would be simpler?

(I can't quite remember why I specified it this way in the first place.)

> Furthermore, ESTABLISH_INTRO has extensions (only 1 as of today) so they could
> also be fragments themselves and thus I'm not sure I see the point of having
> two different ways of "expecting" fragments for the ESTABLISH_* cells and the
> INTRO/RENDEZVOUS cells?

The difference thing here is that everybody can tell which protocols
that a relay supports, but there is no automatic way to tell which
protocols an onion service or client supports.  Since
INTRODUCE2/RENDEZVOUS2 cells are handled by these clients, they need
to get opted into by the relays.

(I'm not sure I understood the question completely.)

> > # Compatibility
> >
> > This proposal will require the allocation of a new 'Relay' protocol version,
> > to indicate understanding of the RELAY_FRAGMENTED command.
>
> Here is a thought about a DoS vector. Here goes:
>
> As an upper limit of 65KB total fragment size, it represents ~126 cells in
> total so I could basically send *125* cells and then stop which will put in
> memory a bit more than 64KB and it will stay there until the last fragment is
> received.
>
> And then I do that on 1000 different circuits bringing the total count in
> memory to 64GB. All stuck there, all "waiting" for the last fragment.
>
> Our OOM would kick in killing circuits but it just seems to me a very easy way
> to continously kick the OOM of a _service_ which is pretty bad side channel.

A few responses here:

First, we shouldn't allow 65535-byte fragmented cells.   The actual
maximum length should be something more like 1024 or 4096 bytes.

Second, we should make sure that when we are reassembling cells, we
use the same buf_t buffers that we use for other stuff.  Our buffers
are timestamped, so we can tell which buffer has had data stalling for
the longest, and we should use that to make sure we're killing off the
right circuits preferentially.

Third, fragments should only be allowed at an onion service for
INTRODUCE2, and those should only come one at a time from each
introduction point, so the number that it's reassembling at the time
will be limited by the number of intro circuits it has open.  It'll be
the the intro points that have to be keeping a bunch of cells in
assembly at once, and be ready to kill off circuits that dawdle too
long.

Does this make more sense?  If so I'll try to clarify it in the proposal.
-- 
Nick
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Chutney code refactoring

[Nick Mathewson]

On Thu, May 14, 2020 at 7:04 PM c  wrote:
>
> On Sat, 18 Apr 2020 11:02:03 +
> c  wrote:
>
> > I came across Node.specialize() which does not seem to be called
> > elsewhere, and I cannot guess at its purpose.
>
> I ran vulture (a static analyzer for dead code), trimmed the output to
> omit things I know were being used (some functions/attributes are used
> solely in Python 2), and found these unused names:
>
> lib/chutney/Templating.py:280: unused function 'getUpdateTime' (60% 
> confidence)
> lib/chutney/TorNet.py:439: unused function 'specialize' (60% confidence)
> lib/chutney/TorNet.py:658: unused function '_getFreeVars' (60% confidence)
> lib/chutney/TorNet.py:1232: unused function 'isBootstrapped' (60% confidence)
> lib/chutney/TorNet.py:1799: unused function 'isInExpectedDirInfoDocs' (60% 
> confidence)
> lib/chutney/TorNet.py:2151: unused function 'configure' (60% confidence)
> lib/chutney/TorNet.py:2191: unused function 'restart' (60% confidence)
> lib/chutney/TorNet.py:2286: unused function 'wait_for_bootstrap' (60% 
> confidence)
> lib/chutney/Traffic.py:335: unused attribute 'am_closing' (60% confidence)
> lib/chutney/Traffic.py:345: unused attribute 'am_closing' (60% confidence)
> lib/chutney/Traffic.py:400: unused attribute 'pending_close' (60% confidence)
> lib/chutney/Traffic.py:406: unused attribute 'dot_repetitions' (60% 
> confidence)
>
> Aside from isBootstrapped() which we discussed previously and are
> likely going to use, is there any code that stands out as unnecessary
> or dead?

Hm. Of these:

isInExpectedDirInfoDocs looks like it might once have done something useful.

configure and restart and wait_for_bootstrap are all in use; they are
invoked by name, from the command line, at the end of runConfigFile,
where it says `return getattr(network, verb)()`.

I think the rest are likely to be unused.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Walking onions: week 10 update

[Nick Mathewson]

Walking Onions: week 10 update

 On our current grant from the zcash foundation, I'm working on a
full specification for the Walking Onions design.  I'm going to try to
send out these updates once a week.

My previous updates are linked below:

 Week 1:
   formats, preliminaries, git repositories, binary diffs,
   metaformat decisions, and Merkle Tree trickery.

   https://lists.torproject.org/pipermail/tor-dev/2020-March/014178.html

 Week 2:
   Specifying details of SNIP and ENDIVE formats, in CDDL.

   https://lists.torproject.org/pipermail/tor-dev/2020-March/014181.html

 Week 3:
   Expanding ENDIVEs into SNIPs.

   https://lists.torproject.org/pipermail/tor-dev/2020-March/014194.html

Week 4:
   Voting (part 1) and extending circuits.

   https://lists.torproject.org/pipermail/tor-dev/2020-March/014207.html

Week 5:
   Voting (part 2)

   https://lists.torproject.org/pipermail/tor-dev/2020-April/014218.html

Week 6:
   Editing, voting rules, and client behavior (part 1)

   https://lists.torproject.org/pipermail/tor-dev/2020-April/014222.html

Week 7:
   Exit policies: How do they work?

   https://lists.torproject.org/pipermail/tor-dev/2020-April/014232.html

Week 8:
   Onion services, relay honesty, migration and families

   https://lists.torproject.org/pipermail/tor-dev/2020-April/014255.html

Week 9:
   (There was no week 9)

== Now with an introduction

I left the Introduction section of the proposal till late in the
game; now I have it in place as a new section 1:



== Proposals into the pipeline

I've also revised all of the "other proposals" that were logically
separate from the rest of Walking Onions.  Three of them were ready
to go to tor-dev, so I've added them as:

 318-limit-protovers.md Limit protover values to 0-63.
 319-wide-everything.md RELAY_FRAGMENT cells
 320-tap-out-again.md   Removing TAP usage from v2 onion services

== In the home stretch

I've got till the end of the month to wrap up this proposal, so
let's see where we are.  We have a minimal viable proposal, I
believe: if we were to sit down and start implementing, we would
probably get most of the way based on what we have now.

There are a few additional things I'd like to wrap up before I call
the proposal done, however:

* I've got all the necessary machinery written up for generating and
  calculating indices, but before I actually move ahead, I should
  fill in the details in sections 2 through 4.

* One of the original goals of Walking Onions was to permit us to
  move to alternative topologies where not every relay connects to
  every other relay.  That's not something that we can include in
  this proposal, since the research here is quite preliminary, but
  I still want to prepare for it by making sure that clients can
  handle getting SNIPs that say "this SNIP is only valid when given
  by a relay of type X".

* I should take another pass through the CDDL format to see whether
  there are any opportunities to save a bunch of bytes.  I should
  also fill in a couple of example SNIPs to see how big they are
  with this proposal.

* There are places all over the proposal where I have marked
  uncertain points with .  I should go through them and resolve
  them all before I call the proposal done.


[INTRO] 
https://github.com/nmathewson/walking-onions-wip/blob/master/specs/01-intro.md
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Proposal 320: Removing TAP usage from v2 onion services

[Nick Mathewson]

On Mon, May 11, 2020 at 5:58 PM Ian Goldberg  wrote:
>
> On Mon, May 11, 2020 at 04:47:53PM -0400, Nick Mathewson wrote:
> > ## INTRODUCE cells, RENDEZVOUS cells, and ntor.
> >
> > We allow clients to specify the rendezvous point's ntor key in the
> > INTRODUCE2 cell instead of the TAP key.  To do this, the client
> > simply sets KLEN to 32, and includes the ntor key for the relay.
> >
> > Clients should only use ntor keys in this way if the network parameter
> > "hsv2-client-rend-ntor" is set to 1, and if the entry "allow-rend-ntor"
> > is present in the onion service descriptor.
> >
> > Services should only advertise "allow-rend-ntor" in this way if the
> > network parameter "hsv2-service-rend-ntor" is set to 1.
>
> It should be stronger, right? A service that does not advertise
> allow-rend-ntor (because hsv2-service-rend-tor is unset) MUST reject an
> ntor key, even if the service actually does support it?  Otherwise a
> client could simply try it even if support is not advertised?

Ah yes, you're right.

-- 
Nick
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Proposal 319: RELAY_FRAGMENT cells

[Nick Mathewson]

```
Filename: 319-wide-everything.md
Title: RELAY_FRAGMENT cells
Author: Nick Mathewson
Created: 11 May 2020
Status: Open
```

(This proposal is part of the Walking Onions spec project.)

# Introduction

Proposal 249 described a system for `CREATE` cells to become wider, in order to
accommodate hybrid crypto.  And in order to send those cell bodies across
circuits, it described a way to split `CREATE` cells into multiple `EXTEND`
cells.

But there are other cell types that can need to be wider too. For
example, `INTRODUCE` and `RENDEZVOUS` cells also contain key material
used for a handshake: if handshakes need to grow larger, then so do
these cells.

This proposal describes an encoding for arbitrary "wide" relay cells,
that can be used to send a wide variant of anything.

To be clear, although this proposal describes a way that all relay
cells can become "wide", I do not propose that wide cells should
actually be _allowed_ for all relay cell types.

# Proposal

We add a new relay cell type: `RELAY_FRAGMENT`.  This cell type contains part
of another relay cell.  A `RELAY_FRAGEMENT` cell can either introduce a new
fragmented cell, or can continue one that is already in progress.

The format of a RELAY_FRAGMENT body is one of the following:

// First body in a series
struct fragment_begin {
   // What relay_command is in use for the underlying cell?
   u8 relay_command;
   // What will the total length of the cell be once it is reassembled?
   u16 total_len;
   // Bytes for the cell body
   u8 body[];
}

// all other cells.
struct fragment_continued {
   // More bytes for the cell body.
   u8 body[];
}

To send a fragmented cell, first a party sends a RELAY_FRAGMENT cell
containing a "fragment_begin" payload.  This payload describes the total
length of the cell, the relay command

Fragmented cells other than the last one in sequence MUST be sent full of
as much data as possible.  Parties SHOULD close a circuit if they receive a
non-full fragmented cell that is not the last fragment in a sequence.

Fragmented cells MUST NOT be interleaved with other relay cells on a circuit,
other than cells used for flow control. (Currently, this is only SENDME
cells.)  If any party receives any cell on a circuit, other than a flow
control cell or a RELAY_FRAGEMENT cell, before the fragmented cell is
complete, than it SHOULD close the circuit.

Parties MUST NOT send extra data in fragmented cells beyond the amount given
in the first 'total_len' field.

Not every relay command may be sent in a fragmented cell.  In this proposal,
we allow the following cell types to be fragmented: EXTEND2, EXTENDED2,
INTRODUCE1, INTRODUCE2, RENDEZVOUS.  Any party receiving a command that they
believe should not be fragmented should close the circuit.

Not all lengths up to 65535 are valid lengths for a fragmented cell.  Any
length under 499 bytes SHOULD cause the circuit to close, since that could
fit into a non-fragmented RELAY cell.  Parties SHOULD enforce maximum lengths
for cell types that they understand.

All `RELAY_FRAGMENT` cells for the fragmented cell must have the
same Stream ID.  (For those cells allowed above, the Stream ID is
always zero.)  Implementations SHOULD close a circuit if they
receive fragments with mismatched Stream ID.

# Onion service concerns.

We allocate a new extension for use in the ESTABLISH_INTRO by onion services,
to indicate that they can receive a wide INTRODUCE2 cell.  This extension
contains:

struct wide_intro2_ok {
  u16 max_len;
}

We allocate a new extension for use in the `ESTABLISH_RENDEZVOUS`
cell, to indicate acceptance of wide `RENDEZVOUS2` cells.  This
extension contains:

struct wide_rend2_ok {
  u16 max_len;
}

(Note that `ESTABLISH_RENDEZVOUS` cells do not currently have a an
extension mechanism.  They should be extended to use the same
extension format as `ESTABLISH_INTRO` cells, with extensions placed
after the rendezvous cookie.)

# Handling RELAY_EARLY

The first fragment of each EXTEND cell should be tagged with `RELAY_EARLY`.
The remaining fragments should not.  Relays should accept `EXTEND` cells if and
only if their _first_ fragment is tagged with `RELAY_EARLY`.

> Rationale: We could allow any fragment to be tagged, but that would give
> hostile guards an opportunity to move RELAY_EARLY tags around and build a
> covert channel.  But if we later move to a relay encryption method that
> lets us authenticate RELAY_EARLY, we could then require only that _any_
> fragment has RELAY_EARLY set.

# Compatibility

This proposal will require the allocation of a new 'Relay' protocol version,
to indicate understanding of the RELAY_FRAGMENTED command.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Proposal 320: Removing TAP usage from v2 onion services

[Nick Mathewson]

```
Filename: 320-tap-out-again.md
Title: Removing TAP usage from v2 onion services
Author: Nick Mathewson
Created: 11 May 2020
Status: Open
```

(This proposal is part of the Walking Onions spec project.  It updates
proposal 245.)

# Removing TAP from v2 onion services

As we implement walking onions, we're faced with a problem: what to do
with TAP keys?  They are bulky and insecure, and not used for anything
besides v2 onion services.  Keeping them in SNIPs would consume
bandwidth, and keeping them indirectly would consume complexity.  It
would be nicer to remove TAP keys entirely.

But although v2 onion services are obsolescent and their
cryptographic parameters are disturbing, we do not want to drop
support for them as part of the Walking Onions migration.  If we did
so, then we would force some users to choose between Walking Onions
and v2 onion services, which we do not want to do.

Instead, we describe here a phased plan to replace TAP in v2 onion
services with ntor.  This change improves the forward secrecy of
_some_ of the session keys used with v2 onion services, but does not
improve their authentication, which is strongly tied to truncated
SHA1 hashes of RSA1024 keys.

Implementing this change is more complex than similar changes
elsewhere in the Tor protocol, since we do not want clients or
services to leak whether they have support for this proposal, until
support is widespread enough that revealing it is no longer a
privacy risk.

## Ntor keys, link specifiers, and SNIPs in v2 descriptors.

We define these entries that may appear in v2 onion service
descriptors, once per introduction point.

"identity-ed25519"
"ntor-onion-key"

   [at most once each per intro point.]

   These values are in the same format as and follow the same
   rules as their equivalents in router descriptors.

"link-specifiers"

   [at most once per introduction point]

   This value is the same as the link specifiers in a v3 onion
   service descriptor, and follows the same rules.

Services should not include any of these fields unless a new network
parameter, "hsv2-intro-updated" is set to 1. Clients should not parse
these fields or use them unless "hsv2-use-intro-updated" is set to 1.

We define a new field that can be used for hsv2 descriptors with
walking onions:

"snip"
[at most once]

This value is the same as the snip field introduced to a v3
onion service descriptor by proposal (XXX) and follows the
same rules.

Services should not include this field unless a new network parameter,
"hsv2-intro-snip" is set to 1. Clients should not parse this field or use it
unless the parameter "hsv2-use-intro-snip" is set to 1.

Additionally, relays SHOULD omit the following legacy intro point
parameters when a new network parameter, "hsv2-intro-legacy" is set
to 0: "ip-address", "onion-port", and "onion-key". Clients should
treat them as optional when "hsv2-tolerate-no-legacy" is set to 1.

## INTRODUCE cells, RENDEZVOUS cells, and ntor.

We allow clients to specify the rendezvous point's ntor key in the
INTRODUCE2 cell instead of the TAP key.  To do this, the client
simply sets KLEN to 32, and includes the ntor key for the relay.

Clients should only use ntor keys in this way if the network parameter
"hsv2-client-rend-ntor" is set to 1, and if the entry "allow-rend-ntor"
is present in the onion service descriptor.

Services should only advertise "allow-rend-ntor" in this way if the
network parameter "hsv2-service-rend-ntor" is set to 1.

## Migration steps

First, we implement all of the above, but set it to be disabled by
default.  We use torrc fields to selectively enable them for testing
purposes, to make sure they work.

Once all non-LTS versions of Tor without support for this proposal are
obsolete, we can safely enable "hsv2-client-rend-ntor",
"hsv2-service-rend-ntor", "hsv2-intro-updated", and
"hsv2-use-intro-updated".

Once all non-LTS versions of Tor without support for walking onions
are obsolete, we can safely enable "hsv2-intro-snip",
"hsv2-use-intro-snip", and "hsv2-tolerate-no-legacy".

Once all non-LTS versions of Tor without support for _both_ of the
above implementations are finally obsolete, we can finally set
"hsv2-intro-legacy" to 0.

## Future work

There is a final TAP-like protocol used for v2 hidden services: the
client uses RSA1024 and DH1024 to send information about the
rendezvous point and to start negotiating the session key to be used
for end-to-end encryption.

In theory we could get a benefit to forward secrecy by using ntor
instead of TAP here, but we would get not corresponding benefit for
authentication, since authentication is still ultimately tied to
HSv2's scary RSA102

[tor-dev] Proposal 318: Limit protover values to 0-63

[Nick Mathewson]

```
Filename: 318-limit-protovers.md
Title: Limit protover values to 0-63.
Author: Nick Mathewson
Created: 11 May 2020
Status: Open
```

# Limit protover values to 0-63.

I propose that we no longer accept protover values higher than 63,
so that they can all fit nicely into 64-bit fields.

(This proposal is part of the Walking Onions spec project.)

## Motivation

Doing this will simplify our implementations and our protocols.
Right now, an efficient protover implementation needs to use ranges
to represent possible protocol versions, and needs workarounds to
prevent an attacker from constructing a protover line that would
consume too much memory.  With Walking Onions, we need lists of
protocol versions to be represented in an extremely compact format,
which also would benefit from a limited set of possible versions.

I believe that we will lose nothing by making this
change. Currently, after nearly two decades of Tor development
and 3.5 years of experiences with protovers in production, we have
no protocol version high than 5.

Even if we did someday need to implement higher protocol
versions, we could simply add a new subprotocol name instead.  For
example, instead of "HSIntro=64", we could say "HSIntro2=1".

## Migration

Immediately, authorities should begin rejecting relays with protocol
versions above 63.  (There are no such relays in the consensus right
now.)

Once this change is deployed to a majority of authorities, we can
remove support in other Tor environments for protocol versions
above 63.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Proposal 315: Updating the list of fields required in directory documents

[Nick Mathewson]

On Thu, Apr 23, 2020 at 5:26 PM teor  wrote:
>
> Hi Nick,
>
> This proposal is missing the "bridge" case.
>
> Bridges are more complicated, because we have at least
> 3 kinds of bridges:
> * bridges distributed by BridgeDB
> * bridges distributed with apps (such as Tor Browser)
> * private bridges
>
> Bridge option transitions are also more complicated, because clients
> download bridge descriptors directly from their configured bridges.

Thank you!

I think that the transition isn't too bad, since the partitioning that
a bridge _could_ do by maintaining an obsolete descriptor format is
limited in its impact, since the bridge already (typically) sees the
client's IP address.  So the only difference is that we need to be a
little more careful about when we start  to require the fields, since
bridges sometimes  lag the versions supported by the rest of the
network.

I've update the proposal with these paragraphs:

   Bridge relays have their descriptors processed by clients
   without necessarily passing through authorities.
   We can make fields mandatory in bridge descriptors once we
   can be confident that no bridge lacking them will actually
   connect to the network-- or that all such bridges are safe
   to stop using.

   For bridges, when a field becomes required, it will take some
   time before all clients require that field.  This would create a
   partitioning opportunity, but partitioning at the first-hop
   position is not so strong: the bridge already knows the client's
   IP, which is a much better identifier than the client's Tor
   version.

cheers,
-- 
Nick
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] (No walking onions update for last week)

[Nick Mathewson]

Hi all!  Sending a short note that I won't be doing a walking onions
update from the past week -- my time went to follow-on tasks for
personnel transitions, and I didn't make much progress to speak of on
the walking onions specs.  I hope to get more done this week, and send
an update then.

-- 
Nick
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Walking onions status update: week 8

[Nick Mathewson]

Walking Onions: week 8 update

 On our current grant from the zcash foundation, I'm working on a
full specification for the Walking Onions design.  I'm going to try to
send out these updates once a week.

My previous updates are linked below:

 Week 1:
   formats, preliminaries, git repositories, binary diffs,
   metaformat decisions, and Merkle Tree trickery.

   https://lists.torproject.org/pipermail/tor-dev/2020-March/014178.html

 Week 2:
   Specifying details of SNIP and ENDIVE formats, in CDDL.

   https://lists.torproject.org/pipermail/tor-dev/2020-March/014181.html

 Week 3:
   Expanding ENDIVEs into SNIPs.

   https://lists.torproject.org/pipermail/tor-dev/2020-March/014194.html

Week 4:
   Voting (part 1) and extending circuits.

   https://lists.torproject.org/pipermail/tor-dev/2020-March/014207.html

Week 5:
   Voting (part 2)

   https://lists.torproject.org/pipermail/tor-dev/2020-April/014218.html

Week 6:
   Editing, voting rules, and client behavior (part 1)

   https://lists.torproject.org/pipermail/tor-dev/2020-April/014222.html

Week 7:
   Exit policies: How do they work?

   https://lists.torproject.org/pipermail/tor-dev/2020-April/014232.html

== Onion services: surprisingly easy! (but not?)

This week I started with a look at onion services and how to make
them work with Walking Onions [ONIONSPEC].  Most of my initial
thought had gone into the problem of finding and using HSDirs, and
so specifying this part went smoothly.  I already havd a link
specifier for asking for a span of several entries in an ENDIVE, and
I already had a notion of truncating index values based on a network
parameter.

The harder part came when I wanted to talk about how to specify
introduction points and rendezvous points by their SNIPs.  The hard
part here was managing the migration path.  To avoid fingerprinting,
when we make changes to the onion service protocol of this kind, we
need to give a lot of onion services and clients the opportunity to
start supporting the new protocol all at once, and then to stop
supporting the old one all at once.  This requires more network
parameters, somewhat carefully specified.  I'm hoping we can turn
this into a more general pattern.

And the most irritating part is when I ran into the issue of
supporting HSv2 handshakes.  Were it not for those, we wouldn't have
to keep TAP keys around any more -- and TAP keys would bloat SNIPs
to a pretty huge degree.  The best answer here seems to be migrating
our legacy HSv2 services so they can use ntor onion keys for intro
and rendezvous points.  That would also need to be done in a phased
way.  [TAP-OUT-AGAIN].

The alternative way to keep HSv2 working with walking onions would
be to allow TAP keys to be distributed as part of the SNIP system.
That seems like a fair amount of work, though, and unlike using
ntor, it wouldn't have any positive benefit.

((Yes, HSv2 is deprecated.  But I don't want to give people who
still use it a reason to avoid the walking onions transition.))

== Relay honesty and ENDIVE swapping

One of the issues with our current directory system is that multiple
consensus network documents are valid at the same time.  In
practice, clients will accept fairly old documents, to avoid running
into trouble if the authorities have failed to vote.  But this
client behavior gives hostile directory guards an opportunity to
select which of several recent documents to serve.

Walking Onions would make this problem worse, since every relay,
when it serves a SNIP, is serving part of a directory that it
knows a client is about to use.  If we did nothing to mitigate the
problem, then hostile relays could mount an "ENDIVE swapping" attack
where they look at a client's choice of routing index, and then see
whether any live ENDIVE contains a SNIP that the attacker likes at
that position.

In [HONESTY] I described a few responses to this problem.  The first
is that we should limit the rate at which the index layouts
change. The second is that all parties that look up SNIPs through a
relay should verify that the "published" times from each relay are
monotonically increasing.

The third response seems like the most useful to me: we would
recognize that under ordinary circumstances, the permissible
lifetime for an ENDIVE would be much shorter than it would be under
emergency (consensus failed) circumstances.  That is, if everything
is going well, each relay should get a new ENDIVE every hour or so.
Only if there is no new ENDIVE should clients accept SNIPs that
are extremely old.

With this in mind, I'm proposing a rule that if a client sees _any_
valid SNIP published at time `T` or later, it should not accept any
SNIP published before time `T-Delta`.  (Delta is a network
parameter.)  With this rule in place, clients will insist on fresh
SNIPs when the network is working well, but will tolerate old ones
if the authorities are failing to reach consensus for a while.

== More questions of migration

One hard step in any proposal is 

Re: [tor-dev] Support for full DNS resolution and DNSSEC validation

[Nick Mathewson]

On Sun, Apr 26, 2020 at 4:32 PM Christian Hofer  wrote:
>
> Hi there,
>
> I have a proposal regarding DNS name resolution.
>
> Ticket: https://trac.torproject.org/projects/tor/ticket/34004
> Proposal:
> https://trac.torproject.org/projects/tor/attachment/ticket/34004/317-secure-dns-name-resolution.txt
> Implementation: https://github.com/torproject/tor/pull/1869
>
> All functioniality is behind the DNSResolver feature flag, so don't
> forget to activate it before you start testing.
>
> Please let me know what you think.

Added to the torspec repository as proposal 317.  This is cool stuff!
I'm behind in proposal review but I'll try to catch up soon.  It's
good to see other people with good questions too, here and on the
ticket.

peace,
-- 
Nick
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Proposal XXX: FlashFlow: A Secure Speed Test for Tor (Parent Proposal)

[Nick Mathewson]

Added as proposal 316.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Proposal 314: Allow Markdown for proposal format.

[Nick Mathewson]

```
Filename: 314-allow-markdown-proposals.md
Title: Allow Markdown for proposal format.
Author: Nick Mathewson
Created: 23 April 2020
Status: Open
```

# Introduction

This document proposes a change in our proposal format: to allow
Markdown.

## Motivation

Many people, particularly researchers, have found it difficult to
write text in the format that we prefer.  Moreover, we have often
wanted to add more formatting in proposals, and found it nontrivial
to do so.

Markdown is an emerging "standard" (albeit not actually a
standardized one), and we're using it in several other places.  It
seems like a natural fit for our purposes here.

# Details

We should pick a particular Markdown dialect.  "CommonMark" seems like a
good choice, since it's the basis of what github and gitlab use.

We should also pick a particular tool to use for validating Markdown
proposals.

We should continue to allow text proposals.

We should continue to require headers for our proposals, and do so
using the format at the head of this document: wrapping the headers
inside triple backticks.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Proposal 315: Updating the list of fields required in directory documents

[Nick Mathewson]

Filename: 315-update-dir-required-fields.txt
Title: Updating the list of fields required in directory documents
Author: Nick Mathewson
Created: 23 April 2020
Status: Open

1. Introduction

   When we add a new field to a directory document, we must at first
   describe it as "optional", since older Tor implementations will
   not generate it.  When those implementations are obsolete and
   unsupported, however, we can safely describe those fields as
   "required", since they are always included in practice.

   Making fields required is not just a matter of bookkeeping: it
   helps prevent bugs in two ways.  First, it simplifies our code.
   Second, it makes our code's requirements match our assumptions
   about the network.

   Here I'll describe a general policy for making fields required
   when LTS versions become unsupported, and include a list of
   fields that should become required today.

   This document does not require to us to make all optional fields
   required -- only those which we intend that all Tor instances
   should always generate and expect.

   When we speak of making a field "required", we are talking about
   describing it as "required" in dir-spec.txt, so that any document
   missing that field is no longer considered well-formed.

2. When fields should become required

   We have three relevant kinds of directory documents: those
   generated by relays, those generated by authorities, and those
   generated by onion services.

   Relays generate extrainfo documents and routerdesc documents.
   For these, we can safely make a field required when it is always
   generated by all relay versions that the authorities allow to
   join the network.  To avoid partitioning, authorities should
   start requiring the field before any relays or clients do.

   (If a relay field indicates the presence of a now-required
   feature, then instead of making the field mandatory, we may
   change the semantics so that the field is assumed to be
   present. Later we can remove the option.)

   Authorities generate authority certificates, votes, consensus
   documents, and microdescriptors.  For these, we can safely make a
   field required once all authorities are generating it, and we are
   confident that we do not plan to downgrade those authorities.

   Onion services generate service descriptors.  Because of the risk
   of partitioning attacks, we should not make features in service
   descriptors required without a phased process, described in the
   following section.

2.1. Phased addition of onion service descriptor changes

   Phase one: we add client and service support for the new field,
   but have this support disabled by default. By default, services
   should not generate the new field, and clients should not parse
   it when it is present.  This behavior is controlled by a pair of
   network parameters.  (If the feature is at all complex, the
   network parameters should describe a _minimum version_ that
   should enable the feature, so that we can later enable it only in
   the versions where the feature is not buggy.)

   During this phase, we can manually override the defaults on
   particular clients and services to test the new field.

   Phase two: authorities use the network parameters to enable the
   client support and the service support.  They should only do this
   once enough clients and services have upgraded to a version that
   supports the feature.

   Phase three: once all versions that support the feature are
   obsolete and unsupported, the feature may be marked as required
   in the specifications, and the network parameters ignored.

   Phase four: once all versions that used the network parameters
   are obsolete and unsupported, authorities may stop including
   those parameters in their votes.

3. Directory fields that should become required.

   These fields in router descriptors should become required:
  * identity-ed25519
  * master-key-ed25519
  * onion-key-crosscert
  * ntor-onion-key
  * ntor-onion-key-crosscert
  * router-sig-ed25519
  * proto

   These fields in router descriptors should become "assumed present":
  * hidden-service-dir

   These fields in extra-info documents should become required:
  * identity-ed25519
  * router-sig-ed25519

   The following fields in microdescriptors should become
   required:
  * ntor-onion-key

   The following fields in votes and consensus documents should
   become required:
  * pr
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Chutney code refactoring

[Nick Mathewson]

On Fri, Apr 17, 2020 at 10:03 PM c  wrote:
>
> On Fri, 17 Apr 2020 18:01:42 -0400
> Nick Mathewson  wrote:
>
>
> > Though I think this is likely to be an ongoing change that we see
> > over time, since
>
> Was this sentence supposed to be longer?

Yeah, I'm afraid I do that sometimes.  I start composing a sentence,
then think of something I want to write somewhere else, and then get
distracted by a third sentence I ought to start.  Usually, I come back
and finish all the sentences.  But sometimes I miss one or get
distracted.

I think what I meant to say here was that we're going to need to keep
refining the API as we work on implementation, since experience with
an API can often show its deficiencies in a way that isn't apparent in
advance.

best wishes,
-- 
Nick
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Walking Onions: week 7 update

[Nick Mathewson]

Walking Onions: week 7 update

 On our current grant from the zcash foundation, I'm working on a
full specification for the Walking Onions design.  I'm going to try to
send out these updates once a week.

My previous updates are linked below:

 Week 1:
   formats, preliminaries, git repositories, binary diffs,
   metaformat decisions, and Merkle Tree trickery.

   https://lists.torproject.org/pipermail/tor-dev/2020-March/014178.html

 Week 2:
   Specifying details of SNIP and ENDIVE formats, in CDDL.

   https://lists.torproject.org/pipermail/tor-dev/2020-March/014181.html

 Week 3:
   Expanding ENDIVEs into SNIPs.

   https://lists.torproject.org/pipermail/tor-dev/2020-March/014194.html

Week 4:
   Voting (part 1) and extending circuits.

   https://lists.torproject.org/pipermail/tor-dev/2020-March/014207.html

Week 5:
   Voting (part 2)

   https://lists.torproject.org/pipermail/tor-dev/2020-April/014218.html

Week 6:
   Editing, voting rules, and client behavior (part 1)

   https://lists.torproject.org/pipermail/tor-dev/2020-April/014222.html


== Clustering exit ports

This week I tried to figure out how exits work with walking onions.
A naive approach might have 65535 different routing indices in the
ENDIVE (one for each TCP port), but that would give horrible
performance: it would make the expanded ENDIVEs absolutely huge.
Instead, we can cluster ports into "classes" based on which are
always supported or not supported together.  (For example, on the
day I scanned the directory, every relay supporting any port in the
range 49002-5 supported all of them.)  There are about 200 such
classes on the current Tor network.

But 200 exit indices is still probably too much: it seems like we'll
need to cluster exit ports together in a more lossy way.  For
example, not every exit that supports port 80 supports port 443 or
vice versa.  But around 98% of them do -- so it we wouldn't be
losing too much by using one exit index for "443 and 80", and not
listing the 2% of exits that only support one of those ports.

I wrote a short program [EXIT-ANALYSIS] to experiment with
clustering strategies here, and settled on a greedy algorithm to get
down to a smaller number of port classes by iteratively combining
whichever pair of classes will cause us to lose the fewest number of
(relay:port) possibilities.  We can get down to 16 classes and lose
only 0.07% of all the relay:port possibilities; we can get down to 8
classes and lose only 0.2%.

That's not the whole story, though.  Some of what we're losing would
be things we'd miss.  When the algorithm gives us 16 classes, it
loses 75% of port 1, 70% of port 23, and so on.  Clearly not all
ports are equivalent, and we should tune this approach to take that
into account.  We should also consider the bandwidth impact of these
changes, and retool the algorithm so that it minimizes the
bandwidth-per-port combination we have.

We'd also likely want to hand-edit these port lists
so that they make more sense semantically on their own.  That way,
we could take Teor's suggestion from last month and try to give exit
relays the options to opt in to or out of human-intelligible
partitions.

Further, we could save some complexity by requiring dual-stack exits
to support the same ports on IPv4 and IPv6.  That way we wouldn't
need to worry about divergent sets of port classes.

== So what do we do with these clusters?

If a client wants to connect to port 25, and uses a port-25-only
index, that leaks the client's intention to the middle node.  That's
not so good.  Instead, in the tech report [WHITEPAPER], one or both
of Chelsea Komlo and Ian Goldberg came up with a mechanism called
Delegated Verifiable Selection (DVS).  With DVS, when the client
sends a BEGIN cell, the relay can respond that it does not support
the requested port, along with a SNIP from a relay that does.  To
prevent the relay from choosing an arbitrary SNIP, the BEGIN cell
would have to include an auxiliary index value to be used when DVS
is in place, and the END cell would need room for a SNIP.

(Alternatively, clients could just make a 3-hop circuit and ask it
for SNIPs to be used as exits on some other circuit.)

== Voting on policy sets

These policy clusters (and a few other things!) are too complex for
the voting operations I had before.  Fortunately, it's safe to treat
them as opaque.

I've added a new voting operation to allow authorities to vote on a
bstr that is then decoded and treated as a cbor object (if it is valid
and well-formed).  It should be useful in other places too.

== Migrating port classes

Migrating from one set of port classes to another is nontrivial.
Clients who have the old root document will expect the indices to
correspond to one partition of ports, whereas clients with the new
root document will expect another partition.  To solve this, I've
designed the system so that two sets of exit indices can exist
concurrently while clients are moving to the newer partition of
ports.

Re: [tor-dev] Chutney code refactoring

[Nick Mathewson]

On Wed, Apr 15, 2020 at 11:45 PM c  wrote:

Skipping over some design stuff that seems reasonable.

 [...]
> Here is what I have been considering lately, with the code overall. I
> hope that laying it out in points will help both myself and others try
> to divy up what could be improved about the code, work on it piecewise,
> and make sure nothing is reimplemented incorrectly. I know I'm new to
> the code and thus have not looked into it thoroughly enough to
> understand everything about it, but again, something in the back of my
> head is telling me that if it's this difficult to understand now, then
> it can stand to be changed. So please take my criticism lightly and
> only where it applies. I could be wrong about some things as well. But,
> as a testing suite, this is the place we absolutely need to be sure we
> get right, because with a hard-to-reason testing suite, we open
> ourselves up to bugs within the tests themselves.

Agreed totally.

> - Documentation especially for methods that the tests are supposed to
>   call. In `Node` class, there is a comment "Users are expected to call
>   these" with `__init__` and three other functions declared. `__init__`
>   is fairly self-explanatory but the other three could stand to have
>   the same docstrings that we are giving other functions. I think this
>   should be my first task, before trying to mess with anything that
>   moves. That way, we have a clearer overview of the testsuite's API,
>   and then we can work the implementation around that.

This seems plausible, though I would caution that we shouldn't expect
to find a great separation between the external and internal parts of
chutney right now.  It's been built up piece by piece over time
without a wholly consistent vision, so it probably doesn't have the
best isolation between its layers.

This documentation project would also IMO benefit from a high-level or
module-level overviews on what things exactly chutney does.  Some of
its tasks are well-documented, but others are less so.

If you want to work on this it would be helpful to maybe start by
listing (here or elsewhere) some places where you *don't* feel like
you could (or would want to) write documentation: those would be a
good target for devs who _have_ worked on Chutney before.

> - Once it's clear to all of us (meaning, not only the core Chutney
>   devs, but also outsiders such as myself) what the current interface
>   does, we can see the strengths of that interface, maybe some design
>   flaws if we deem any to exist, and some improvements we can make with
>   the interface. What I mean is: here is the place to make any
>   necessary adjustments to the API before we dive into implementation.

+1.  Though I think this is likely to be an ongoing change that we see
over time, since

> - Now we visit the implementation. By this point I (and other non-core
>   developers) should have a better understanding on what the tests
>   *should* be doing, that we can then proceed to verify the correctness
>   of the tests' implementations themselves. Here's where I would like
>   to revisit cleaning up the code so that we make use of functions to
>   abstract and "atomize" certain aspects of the code (like the
>   currently-unused function isBoostrapped), where we should introduce
>   more self-documenting names (as suggested, replace getAttribute names
>   with isAttribute names for when we want to return bool values), et
>   cetera.

Yes.  I think there are other steps that could help at this stage too,
including:
  * More tests for the various parts of chutney, and refactoring to
make them more testable.
  * Making chutney comply with one or more of the python style
checking tools that are out there.

> I'm outlining what *I* think the approach should be, and in hopes that
> everything else will line up and make more sense for us all. I do want
> to make sure I know what the code should be doing, because while I
> could inspect the code line by line and go through the logic branches
> myself, I'm going to be perfectly honest and say it gives me a
> headache :) I think it's much easier to reason about with
> compartmentalised, self-documenting (and docstringed) components, than
> to try to step through the code as it runs and take it in as one giant
> piece.
>
> Because, we're testing a complex piece of software (Tor). Tor itself
> can do a lot of things that can throw us off especially if we're less
> familiar with its implementation, yet. And that throws off our
> understanding of how Chutney is supposed to work, as well. A lot of the
> tests currently seem to be very conditional -- they pass or fail with
> little rhyme or reason, which is of course the best type of issues to
> debug :P -- so I don't see the test results helping me as well as
> refactoring the testsuite first.

This is an good vision for how Chutney should go; I really support this idea.

> Again, let me know if I have anything wrong, made any unjust
> assumptions, showed 

Re: [tor-dev] Choosing a valid Circuit ID for an OR connection

[Nick Mathewson]

On Tue, Apr 14, 2020 at 9:13 AM Eli Vakrat  wrote:
>
> Hello to the TOR dev team!
>
> My name is Eli, I'm a high school student from Israel and I'm currently 
> trying to implement a TOR Client in Python.
> Currently, my project is configured so that the python client (OP) has its 
> guard node set as my local machine (which is running a downloaded version of 
> TOR). I do this for debugging purposes so that if I send a malformed cell as 
> the implemented client, I can read the debug log that the OR generates and 
> see what I did wrong.
>
> As I am writing this, I have successfully been able to make a v3 "in 
> protocol" handshake by sending the proper VERSIONS and NETINFO cells that the 
> OR has accepted.
>
> Right now, I am working on getting the CREATE cell to work (with the TAP 
> handshake), and when I send the cell to the OR It sends a DESTROY cell with 
> the following message in the debug log:
>
> Apr 14 12:24:51.166 [warn] Received create cell with unexpected circ_id 1. 
> Closing.
>
> To my understanding, the problem here is that the circuit id I have chosen is 
> not valid.
>
> I read from the tor spec that there is a range of acceptable circuit id's for 
> each relay and it can be found in the long term 1024 bit RSA "signing key".
>
> My question to you all is, how can I extract that range for the circuit ID's 
> (along with any other relevant information) from the signing key?
> In other words, what is the "signing key" made up of, and how can I get the 
> information that its made of?
>
> The bottom line is I need to find out what valid circuit ID's I can send to 
> OR's.
>
> I am having a lot of fun doing this project so far and I hope to hear back 
> from anyone who has an answer :)

Hi!  This looks like a cool project.

So, it was only the earlier versions of Tor that chose circuit IDs
based on RSA keys.  In link protocol version 4 or higher (as
negotiated by the VERSIONS cell), you can use any circuit ID that
starts with a 1 bit:

   "In link protocol version 4 or higher, whichever node initiated the
   connection sets its MSB to 1, and whichever node didn't initiate the
   connection sets its MSB to 0."

Hoping this helps,
-- 
Nick
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Walking onions status update: week 6 notes

[Nick Mathewson]

Walking Onions: week 6 update

 On our current grant from the zcash foundation, I'm working on a
full specification for the Walking Onions design.  I'm going to try
to send out these updates once a week.

My previous updates are linked below:

 Week 1:
   formats, preliminaries, git repositories, binary diffs,
   metaformat decisions, and Merkle Tree trickery.

   https://lists.torproject.org/pipermail/tor-dev/2020-March/014178.html

 Week 2:
   Specifying details of SNIP and ENDIVE formats, in CDDL.

   https://lists.torproject.org/pipermail/tor-dev/2020-March/014181.html

 Week 3:
   Expanding ENDIVEs into SNIPs.

   https://lists.torproject.org/pipermail/tor-dev/2020-March/014194.html

Week 4:
   Voting (part 1) and extending circuits.

   https://lists.torproject.org/pipermail/tor-dev/2020-March/014207.html

Week 5:
   Voting (part 2)

   https://lists.torproject.org/pipermail/tor-dev/2020-April/014218.html


This week I did a pass through all of the proposal that's written so
far, trying to revise it for consistency and to look for unexpected
gaps. Then I wrote a first take on a section about client behavior,
and outlined sections about how to keep onions services working.

== Fixing inconsistencies

There were, predictably, some places where I had wrote things in an
introduction to a section, and then turn around and did them
differently later in the section.  I tried to make those consistent
when I could.

For one example, I think Microdescs should now be embedded in votes.
They were separate before, for space reasons, but now that we have
defined support for compression and diffs in voting, we don't need
to keep votes as small as we did before.  But when I went to explain
voting, I didn't actually give a place to put in microdescriptors.
This is fixed now.

== Example rules, and changes to voting.

Thanks to Teor's help and feedback on previous iterations of voting,
the voting operations [VOTING] are now more consistent, and
specified more consistently.  I had to describe a "sorting"
operation for use with voting, since CBOR's default "canonical"
ordering is not what we actually want semantically.

To try to make sure voting could do what we need, I wrote up an
example VotingRules object in a new appendix [EXRULES].  This turned
up a few impedance mismatches between the available voting rules,
the SNIP format, and the ENDIVE format.  I fixed some of these, and
marked others with "XXX" strings to come back and fix on the next
pass through voting.

== More index and extension protocol work

I've found a need for a new link specifier (LS) for use with onion
services.  Unlike a regular routing index LS, this kind includes a
routing index and a count of how many relays to skip after that
position.  This lets a client say, for example, "Give me the third
relay after the one whose HDirIndex is 0xbf9630".  (In order to
answer this query, the extending relay needs to return not only the
SNIP for the third relay, but all SNIPs between the first relay and
that relay, so that the client can trust the answer.)

This kind of LS can be used with the new NIL handshake if the client
_only_ wants to fetch the descriptors; the details are in the
circuit-protocol section [EXTENDING].

== Client operations, part 1

In a new section 06, I've sketched out solutions for guards,
bridges, bootstrapping, and several kinds of path restriction
[CLIENTS]. In doing so I've turned up a possible need for new
IP-based and key-based indices; I've added those to a new appendix
where I'm listing the different indices I think we'll need
[INDEXLIST].

== Outlines of work to come

I've added outlines for a new section 7 about making onion services
work, and a new section 8 about tracking some new ways that relays
might have to misbehave.  I've also added quick sketches for a pair
of additional proposals we'll need: one for a better family
notation, and one for finally migrating away from the last vestiges
of TAP.

But none of those are quite next: In the coming week, if time
permits, I want to get exit policies figured out.  They are the
biggest missing piece of section 06 right now, and seem to me to be
the biggest remaining gap in the specification.

== Fun facts:

At 109KiB, this is now the longest Tor proposal ever.  I will need
to come up with a different fun fact next week.


== References:

[CLIENTS] 
https://github.com/nmathewson/walking-onions-wip/blob/master/specs/06-clients-behavior.md

[EXTENDING] 
https://github.com/nmathewson/walking-onions-wip/blob/master/specs/05-extending-circuits.md

[EXRULES] 
https://github.com/nmathewson/walking-onions-wip/blob/master/specs/AF-example-rules.md

[INDEXLIST] 
https://github.com/nmathewson/walking-onions-wip/blob/master/specs/AG-index-list.md

[VOTING] 
https://github.com/nmathewson/walking-onions-wip/blob/master/specs/03-voting-and-authorities.md
___
tor-dev mailing list
tor-dev@lists.torproject.org

[tor-dev] Walking onions status update: week 5 notes

[Nick Mathewson]

Walking Onions: week 5 update

 On our current grant from the zcash foundation, I'm working on a
full specification for the Walking Onions design.  I'm going to try to
send out these updates once a week.

My previous updates are linked below:

 Week 1:
   formats, preliminaries, git repositories, binary diffs,
   metaformat decisions, and Merkle Tree trickery.

   https://lists.torproject.org/pipermail/tor-dev/2020-March/014178.html

 Week 2:
   Specifying details of SNIP and ENDIVE formats, in CDDL.

   https://lists.torproject.org/pipermail/tor-dev/2020-March/014181.html

 Week 3:
   Expanding ENDIVEs into SNIPs.

   https://lists.torproject.org/pipermail/tor-dev/2020-March/014194.html

Week 4:
   Voting (part 1) and extending circuits.

   https://lists.torproject.org/pipermail/tor-dev/2020-March/014207.html


This week found me still rather distracted by fallout from current
events and preparations for more isolation at home.  I didn't get a
complete rewrite done.  But I do, however, think I figured out voting.

== The big idea for voting

Our current system for voting on relay properties is fairly piecemeal:
every property has its own independent specification in dir-spec.txt
[DIRSPEC], which can get kind of cumbersome.

Every time we have a new property to specify, if it can't be defined as
a flag, we need to define a new consensus method that puts it in the
consensus or the microdescriptors.  Until authorities have upgraded to
support the new method, they can't sign consensuses generated with it,
since they don't know how those consensuses are generated.

We can solve both problems by defining "vote tabulation" as a set of
parameterized operations operating over a set of CBOR inputs.
Individual operations can be simple, like "Take the median of all
integer votes", or progressively more complex.

What's more, we can specify these voting operations themselves as data,
which gives us more advantages: First, that we no longer need
independent English-language specs for each field in the consensus--and
second, that we can introduce new voting rules or change the existing
ones without adding new code.  As long as a quorum of authorities agree
about the rules for voting on some field, the other authorities can
safely follow those rules, and include the new field in the consensus
ENDIVE and SNIPs.  We would only need to introduce new consensus methods
when we wanted to introduce support for an entirely new operation.

After voting is done, some transformations are needed to transform the
objects it produces.  SNIP signatures need to be signed, indexes built,
and so on.  I've left this unspecified for now, since I am likely to
return to the SNIP and ENDIVE formats again in the coming weeks, and
I'll want to tweak the transformation quite a lot.

This is not something I would want to implement in C.  Rust would be a
better fit -- or some functional language.  Fortunately, only
authorities need to follow this algorithm.

I have initial specifications written for these operations that I
believe are sufficient (or nearly so?) for all the things that we do in
voting now.  These are in the draft voting spec as it stands [VOTING].
I expect that they are full of errors, but I wouldn't be too concerned
right now; I'm going to need to do another pass over them, but I want to
return to voting a bit later (see below) once I have more SNIP uses
specified.

== Backward compatibility in voting

There are two separate backward compatibility issues in voting.

The first issue is easier: how to operate when we aren't sure whether
all the authorities have upgraded to support this new voting format.
For this purpose, I'm including legacy votes as part of the new vote
format, signatures and all.  This will make the new votes bigger until
all the authorities have definitely upgraded, but other changes (voting
diffs and compression) should offset this risk.  We can stop sending
legacy votes when everybody has upgraded.

(I thought about having an "implicit version" of each legacy vote
derived from the new vote, but that seemed riskier, and like it would
involve lots more new code of dubious benefit.)

The second issue is a bit harder: for clients that haven't upgraded to
walking onions, we'll need to generate legacy consensus documents for a
some time.  This doesn't need to follow exactly the same rules as
consensus voting uses today:  instead we should have legacy consensus
documents derived from the CBOR consensus, so that we only need to
maintain one voting algorithm long term.

I've left this part unspecified for now too; I'll be comping back to it
after a round of revisions and SNIP usage design.

== Next steps

I plan to spend the first part of week going over what I've written so
far and trying to make it correct and consistent, and to fill in the
baps that I have.  After that, I'm planning to write the example sections
for how circuit extension works.  If time remains, this is when I
finally move on to section 6 of the 

[tor-dev] Walking onions status update: week 4 notes

[Nick Mathewson]

Walking Onions: week 4 update

 On our current grant from the zcash foundation, I'm working on a
full specification for the Walking Onions design.  I'm going to try to
send out these updates once a week.

My previous updates are linked below:

 Week 1:
   formats, preliminaries, git repositories, binary diffs,
   metaformat decisions, and Merkle Tree trickery.

   https://lists.torproject.org/pipermail/tor-dev/2020-March/014178.html

 Week 2:
   Specifying details of SNIP and ENDIVE formats, in CDDL.

   https://lists.torproject.org/pipermail/tor-dev/2020-March/014181.html

 Week 3:
   Expanding ENDIVEs into SNIPs.

   https://lists.torproject.org/pipermail/tor-dev/2020-March/014194.html


This week was pretty rough for me; the impact of being isolated at home
is starting to kick in, and my kid is still trying to find a schedule
that works for him.  I got a lot less done this week than I had planned.

That said, I _did_ get some good design done, in two areas: voting, and
circuit extension.

== Voting

I'm trying to generalize our voting algorithms to minimize the need
for new consensus formats in the future.  I want most new fields to be
self-describing in terms of how to tabulate votes for them.  Trying to
work this out has taken a bunch of energy, though and I'm not quite sure
I'm doing it right.  The work here is unfinished.

I still think it's reasonable to encode existing votes and new votes in
a new cbor-based structure.  By using compression, a little redundancy
in votes shouldn't hurt us too badly -- though that's something I need
to benchmark eventually before we build this.

I've come up with a way for voters to use diffs and compression when
_uploading_ votes: first, the voter asks for a summary of what
mechanisms are supported by the recipient and which older votes it has.
Next, the voter can use that information to upload a vote diff with an
appropriate compression format.

Right now I believe that I have most of the fields for the votes figured
out, but I need to clean them up a _lot_.  SNIP bodies can work
similarly to how microdescriptors work now, if we like -- or they could
involve individual votes on input fields.  I think I'm going to write up
what I can here, and then revisit SNIP bodies once I've worked some on
section 6 (the "fancy" client behavior).

While working on voting, I found a need for various changes in the
already written sections.  For one example, the previous description
of the ENDIVE format included the SNIP signatures as part of the
signed ENDIVE.  But we can't actually generate that as part our
voting protocol: the authorities can't sign SNIPs until they know
the complete ENDIVE contents, and so putting the SNIP signatures in
the signed part of the ENDIVE would add an extra round of voting and
signatures.  Instead, I've moved the SNIP signatures (and related
stuff) into the signature header of the ENDIVE.

I've also worked somewhat on the so-called "root document" holding
the network configuration parameters.  I'm hoping that this document
can have an effective lifetime of weeks or months for clients, and
that clients can download only the parts that they need.  The format
is in section 02, but needs more description.

The work-in-progress voting writeup is in [VOTING], but it is not so
close to done. I think it will need a lot of writing before it's
ready for even preliminary review; it doesn't make sense as it is,
and it contradicts itself in places.

== Extending circuits

There are a few wrinkles here that we hadn't actually specified in
earlier walking onions work!

When one relay is told to extending a circuit to a target relay by
index, the extending relay actually needs to include part of a SNIP for
that second relay: otherwise, the extending relay won't know which onion
key the client expects to receive for it.  We don't want to send an
entire SNIP -- only the parts that the client will expect to match.

In addition to wide EXTEND cells, we'll also need wide RENDEZVOUS and
INTRODUCE1 and INTRODUCE2 and TRUNCATED cells.  All together, that means
that having a general "cell fragmentation" mechanism is probably
warranted; I've started a separate proposal for that [WIDE_EVERYTHING].

We're going to need a general extension format for CREATE and EXTEND
cells, to signal other information, like SNIPs, requests for SNIPs, and
so on.  We also need a way to ask for SNIPs without actually performing
a handshake.  I've tried to fit these all together in a fairly logical
way in [EXTENDING], but there's more to write here, including examples.

== Fun facts

The walking onion proposal is now the second-longest that we have
ever had, behind only proposal 224 (v3 hidden services).

== Next steps

In this coming week I plan to try to wrap up section 3 on voting and
section 5 on extending circuits.  I'm going to go back to the start
of section 2 and start revising all I've written so far for
consistency.

Time permitting, I want to sketch out all the knowns and unknowns
for 

Re: [tor-dev] Walking onions status update: week 3 notes

[Nick Mathewson]

On Fri, Mar 20, 2020 at 5:09 PM teor  wrote:
>
> Hi Nick,
>
> On 21 Mar 2020, at 05:38, Nick Mathewson  wrote:
>
> Walking Onions: week 3 update
>
> As you might recall, the SNIPs need to be nice and small, but they
> need to be completely self-contained.  The ENDIVE, on the other
> hand, needs to be diff-friendly, and compressible.  Relays need to
> derive all the SNIPs from the ENDIVE.  The ENDIVE and SNIP formats
> that I worked on during last week [SNIPFMT] was designed to meet these
> criteria.
>
> This week, I wrote up the algorithms to extract signed SNIPs from an
> ENDIVE.  This is in section 4 [RECONST] of the specs in my
> work-in-progress repository [GITREPO] This was mainly straightforward
> consequence from the work I did last week, but there were a few
> things that turned up.
>
> One trickier aspect of this work is that we want to move most of the
> decision-making to the authorities, and have the relays simply do an
> expansion algorithm.  Updating all the relays would be slow, so relays
> need to be able to handle new index types and data types without
> fully understanding them.  That means that we need to specify an
> algorithm that can work flexibly with many future kinds of data.
>
> Another tricky aspect is that the data which the relays need to put
> into SNIPs all needs to be signed, so all of that data needs to come
> out the same when relays calculate it as the when the authorities
> calculate it. Thus, all the algorithms for building Merkle trees and
> indices and SNIPRouterData bodies need to be deterministic.
>
> I think that I got all of this relatively right in [RECONST], but
> there are probably some mistakes hiding in there.
>
> I'm hoping that there is a cleaner alternative to the weighted-index
> reconstruction algorithm -- the one that's there right now puts a
> lot of requirements on the input data, in order to avoid
> floating-point operations.
>
>
> There's a shorter encoding for Raw:
>
> for each [i, pos1, pos2] in index_ranges:
>   w = pos2 - pos1
>   j  = the index of pos1 among all sorted pos1s
>   new_encoding = [i, j, w]

This will work when w is an integer that we can work on additively,
but not so well when we're talking about something more like a hash
ring.  I think it makes sense to have two different possible encodings
here, though.  I'm calling this one "RawNumeric.

> [i, j, w] is an efficient encoding if index_ranges is sparse compared
> with ENDIVERouterData, because:
> * j has the same cardinality as I
> * w is smaller than pos1 and pos2
>
> If index_ranges is dense, there may be a more efficient encoding:
>   add missing i with weight 0
>   drop j
> With this encoding, you can drop a few of the constraints.


> There's a faster calculation and more efficient encoding for
> Weighted, because there's a common factor in each POS()
> calculation:
> (1 << 32) / total_weight
>
> If we remove that factor, we end up with a much simpler algorithm,
> that's also more flexible:
>
> Algorithm: Expanding a "Weighted" indexspec.
>
>
> Each weighted indexspec also has a multiplier, which may vary
>
> between indexspecs.
>
>
> Let total_weight = SUM(indexspec.index_weights)
>
> Verify total_weight * multiplier <= UINT64_MAX.
>
>
> Let total_so_far = 0.
>
> Let result_idx = {} (an empty mapping).
>
> Define POS(b) = b * multiplier
>
>
> For 0 <= i < LEN(indexspec.indexweights):
>
>Let w = indexspec.indexweights[i].
>
>Let lo = POS(total_so_far).
>
>Let total_so_far = total_so_far + w.
>
>Let hi = POS(total_so_far) - 1.
>
>Append (lo, hi) => i to result_idx.
>
>
> Return result_idx.

I'm not so sure about this one -- see the note below about the total value.

> If multiplier is large, then the weights can be quite small.
>
> (Small weights sacrifice some accuracy.) And if the multiplier
>
> is 1, you effectively have a "Raw" index.
>
>
> If you make those changes, you should be able to use a similar
> process to expand all the different index types. (After multiplying,
> truncating, or hashing, you either end up with a delta, or an
> absolute position. You can turn deltas into absolute positions,
> and then feed them all into the same base algorithm.)
>
> There are also a few things I think might be bugs:
>
> Was there meant to be a constraint that the Weighted total is
> UINT64_MAX? Or close to UINT64_MAX?

The actual intention is that it has to be " = UINT32_MAX", not" <=
UINT32_MAX".  Remember that the the client picks its next relay by
specifying a random index value in range 0..UINT32_MAX, and it expects
that the relay to that it

Re: [tor-dev] Walking Onions status update: week 2 notes

[Nick Mathewson]

On Fri, Mar 20, 2020 at 12:38 PM teor  wrote:
>
> Hi Nick,
>
> > On 20 Mar 2020, at 23:01, Nick Mathewson  wrote:
> >
> > On Wed, Mar 18, 2020 at 11:21 AM teor  wrote:
> >>
> >>> On 14 Mar 2020, at 14:44, teor  wrote:
> >>>
> >>>> * As I work, I'm identifying other issues in tor that stand in
> >>>>   the way of a good efficient walking onion implementation that
> >>>>   will require other follow-up work.  This week I ran into a
> >>>>   need for non-TAP-based v2 hidden services, and a need for a
> >>>>   more efficient family encoding.  I'm keeping track of these
> >>>>   in my outline file.
> >>
> >> Here's another issue you might want to consider:
> >>
> >> Currently, new relays get in the consensus as soon as:
> >>  * they post their descriptors, and
> >>  * a majority of authorities can contact their ORPorts.
> >>
> >> That means clients and relays waste a whole bunch of bandwidth
> >> downloading consensus info and descriptors for relays with very
> >> low weights.
> >>
> >> Instead, we could have two documents:
> >
> > Thanks for this! This dovetails nicely with some of the voting design
> > work I'm up to right now.
>
> It would be great to have a protocol that doesn't depend on:
> * time synchronisation
> * big documents
> * one-shot updates
> * absolute consistency

Hi!  I think that _some_ of these changes are orthogonal to the shifts
we'll need for walking onions, but some of them do dovetail nicely.
I'm going to keep this email as a note to myself, so that I can
double-check whether any of these items are precluded by choices I'm
making in the design space.

> We've already made vote timing a bit more robust in the tor master
> branch, by ignoring late votes:
> https://trac.torproject.org/projects/tor/ticket/4631
>
> Here's a few other tweaks that might help:
>
> Tor halves the consensus interval when there is no fresh consensus.
> But changing the interval makes tor's code much more complex.
> Instead, let's have a fixed consensus interval. And make it long
> enough for efficiency, but short enough to recover from a failed
> consensus.

This makes sense; I'd forgotten that we did this.

> Let's support vote diffs, as well as consensus diffs. Vote diffs
> don't help when posting votes. But when requesting votes, authorities
> can include hashes of votes they already have. That way, authorities
> that are under heavy load are more likely to get all the votes.

Actually, we *can* do them when posting votes; I've sketched out a
design here in the work-in-progress voting spec.


> We could increase the time that authorities have to fetch votes, and
> make them retry fetches every few minutes.
>
> We could do consistency checks on smaller shards, so that a
> consistency failure in any one document does not break the entire
> consensus.

This seems like something we could do as a followup here? It does seem
like it could get pretty complex.

> We could create a shard for each supported consensus method (like we
> do microdescriptors). That way, a consistency failure in any one
> consensus method does not break the entire consensus.

Hm, interesting.  This would involve generating multiple consensuses
with different methods?  It sounds orthogonal to me, but quite
possibly a good idea.  But we'd have to withhold signatures on older
consensus methods until we're sure that the newer ones are failing --
and I worry about attacks where we _cause_ a newer consensus method to
be inconsistent specifically so we can force an older one.

I also worry about mix-and-match attacks on the different shards.

> We could make shards valid for a longer time, so that if the
> replacement shard does not reach consensus, the older one is used.
>
> Then, the final documents are a combination of all the consistent
> shards, using the highest consistent consensus method. (Much like
> the current microdesc consensus.)
>
> Once we've made some of those changes, then some other changes
> become plausible:
>
> Let's make votes valid for exactly 2 voting periods, and use the
> latest available vote from each authority.

Hmmm. So this doesn't help us much in the case when an authority goes
down -- it just means that its vote counts for an extra hour more than
it would otherwise.  That would buy us an hour of extra consensus time
-- but it would only be a big improvement if the authority is
-regularly- failing to upload a vote.  For that case, I think it might
make more sense  to address the reliability problem directly instead
of trying to work around it.

> [...]

I dig the idea of dropping time synchronization if we can; let's look
into this more.  It seems like something we can do orthogonally to
walking onions, though?


-- 
Nick
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Walking onions status update: week 3 notes

[Nick Mathewson]

Walking Onions: week 3 update

 On our current grant from the zcash foundation, I'm working on a
full specification for the Walking Onions design.  I'm going to try to
send out these updates once a week.

My previous updates are linked below:

 Week 1:
   formats, preliminaries, git repositories, binary diffs,
   metaformat decisions, and Merkle Tree trickery.

   https://lists.torproject.org/pipermail/tor-dev/2020-March/014178.html

 Week 2:
   Specifying details of SNIP and ENDIVE formats, in CDDL.

   https://lists.torproject.org/pipermail/tor-dev/2020-March/014181.html

You might like to check out those updates, and their references,
if this update doesn't make sense to you.

===

As you might recall, the SNIPs need to be nice and small, but they
need to be completely self-contained.  The ENDIVE, on the other
hand, needs to be diff-friendly, and compressible.  Relays need to
derive all the SNIPs from the ENDIVE.  The ENDIVE and SNIP formats
that I worked on during last week [SNIPFMT] was designed to meet these
criteria.

This week, I wrote up the algorithms to extract signed SNIPs from an
ENDIVE.  This is in section 4 [RECONST] of the specs in my
work-in-progress repository [GITREPO] This was mainly straightforward
consequence from the work I did last week, but there were a few
things that turned up.

One trickier aspect of this work is that we want to move most of the
decision-making to the authorities, and have the relays simply do an
expansion algorithm.  Updating all the relays would be slow, so relays
need to be able to handle new index types and data types without
fully understanding them.  That means that we need to specify an
algorithm that can work flexibly with many future kinds of data.

Another tricky aspect is that the data which the relays need to put
into SNIPs all needs to be signed, so all of that data needs to come
out the same when relays calculate it as the when the authorities
calculate it. Thus, all the algorithms for building Merkle trees and
indices and SNIPRouterData bodies need to be deterministic.

I think that I got all of this relatively right in [RECONST], but
there are probably some mistakes hiding in there.

I'm hoping that there is a cleaner alternative to the weighted-index
reconstruction algorithm -- the one that's there right now puts a
lot of requirements on the input data, in order to avoid
floating-point operations.

===

After working on ENDIVE expansion, I moved on to the problem of
authorities voting.  There's a lot of uncertainty here and decisions
I'll need to make.

The biggest decision is how the new voting process for ENDIVEs
should relate to the current voting process for consensus documents.
The choices seem to be:

   * Have two processes with different kinds of votes.
   * Expand the information that goes into current votes, and
 generate ENDIVEs as a new kind of consensus from the current
 voting process.
   * Design the ENDIVE voting process so that it can also be used to
 generate current-style consensuses.

Using separate processes might be simplest, but it does carry risk
for duplicated code, and for the two voting processes reaching
divergent results.  It would also be potentially bad if an authority
could vote differently in each process.

The second approach -- where ENDIVEs are another kind of consensus
generated from current-style voting documents -- has some charm to
it, but it perpetuates our current consensus-voting system, and
makes us specify a second encoding of CBOR objects as text, if we
want to vote on them meaningfully.  This is a bit nasty, since the
standard-specified text encoding of cbor is not a bijection, and
doesn't mesh too well with our old directory format.

Therefore the third approach seems a little better to me now.  In
it, we'd specify our votes as a CBOR-based format loosely based on
the current vote format, and describe a way to extract from them the
same data that is currently used for our voting algorithm.  This
would let us encode CBOR within the document naturally, while not
having to throw away too much of our existing voting specification
and voting code.

I've got a few more ideas here, though I don't have any specs yet.
I've been collecting the ideas in a notes file [VNOTES] so I don't
forget them.

===

While working on voting notes, I found some areas where the ENDIVE
and SNIP formats might be better.

First, I hadn't thought about authority certificates.  Those should
go in, so that we can have the ENDIVE-signing keys be signed
themselves with the authorities' long-lived identities.

Second, there seem to be some opportunities for transmitting indices
in a simpler way.  Instead of assigning a weight to each router
separately, for example, we could derive one weighted index from
another weighted index by re-weighting it, as we do with the
"bandwidth-weights" values in the current consensus.

Teor also made some good suggestions for improving the diff format;
I incorporated one and made a note 

Re: [tor-dev] Walking Onions status update: week 2 notes

[Nick Mathewson]

On Wed, Mar 18, 2020 at 11:21 AM teor  wrote:
>
> Hi Nick,
>
> > On 14 Mar 2020, at 14:44, teor  wrote:
> >
> >>  * As I work, I'm identifying other issues in tor that stand in
> >>the way of a good efficient walking onion implementation that
> >>will require other follow-up work.  This week I ran into a
> >>need for non-TAP-based v2 hidden services, and a need for a
> >>more efficient family encoding.  I'm keeping track of these
> >>in my outline file.
>
> Here's another issue you might want to consider:
>
> Currently, new relays get in the consensus as soon as:
>   * they post their descriptors, and
>   * a majority of authorities can contact their ORPorts.
>
> That means clients and relays waste a whole bunch of bandwidth
> downloading consensus info and descriptors for relays with very
> low weights.
>
> Instead, we could have two documents:

Thanks for this! This dovetails nicely with some of the voting design
work I'm up to right now.

-- 
Nick
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Walking Onions status update: week 2 notes

[Nick Mathewson]

On Sat, Mar 14, 2020 at 12:44 AM teor  wrote:
>
> Hi Nick,
>
> I'm interested in following along with Walking Onions, but I might
> drop out when the relay IPv6 work gets busy.
>
> I'm not sure how you'd like feedback, so I'm going to try to put it
> in emails, or in pull requests.
>
> (I made one comment on a git commit in walking-onions-wip, but
> I'm not sure if you see those, so I'll repeat it here.)

Thanks, this is really helpful!  I missed the repository comments, and
I'll probably miss some more.


> > On 14 Mar 2020, at 03:52, Nick Mathewson  wrote:
> >
> > This week, I worked specifying the nitty-gritty of the SNIP and
> > ENDIVE document formats.  I used the CBOR meta-format [CBOR] to
> > build them, and the CDDL specification language [CDDL] to specify
> > what they should contain.
> >
> > As before, I've been working in a git repository at [GITHUB]; you
> > can see the document I've been focusing on this week at
> > [SNIPFMT].  (That's the thing to read if you want to send me
> > patches for my grammar.)
>
> I'm not sure if you've got to exit ports yet, but here's one possible
> way to partition ports:
> * choose large partitions so that all exits support all ports in the
>   partition
> * choose smaller categories so that most exits support most ports
>   in the partition
> * ignore small partitions, they're bad for client privacy anyway
>
> For example, you might end up with:
> * web (80 & 443)
> * interactive (SSH, IRC, etc.)
> * bulk (torrent, etc.)
> * default exit policy
> * reduced exit policy
>
> I'm not sure if we will want separate categories for IPv4-only
> and dual-stack policies. We can probably ignore IPv6-only
> policies for the moment, but we should think about them in
> future.

Interesting!  Yeah, something like this might work.  I've added this
to my notes.

Also, there's some interesting ideas about handling exit policies in
the whitepaper's section 6.  I don't know if

> > There were a few neat things to do here:
> >
> >   * I had to define SNIPs so that clients and relays can be
> > mostly agnostic about whether we're using a merkle tree or a
> > bunch of signatures.
> >
> >   * I had to define a binary diff format so that relays can keep
> > on downloading diffs between ENDIVE documents. (Clients don't
> > download ENDIVEs).  I did a quick prototype of how to output
> > this format, using python's difflib.
>
> Can we make the OrigBytesCmdId use start and length?
> length may be shorter than end, and it will never be longer.

Good idea; done.

> If we are doing chunk-based encoding, we could make start
> relative to the last position in the original file. But that would
> mean no back-tracking, which means we can't use some
> more sophisticated diff algorithms.

Well, we could allow signed integers.  I'm making a note to look into
whether this would help much.

[...]
> If the issue is having multiple valid ENDIVEs, then authorities could
> also put a cap on the number of concurrently valid ENDIVEs.
>
> There are two simple schemes to implement a cap:
> * set a longer interval for rebuilding all ENDIVEs
>   (the cap is the rebuild interval, divided by the validity interval)
> * refuse to sign a new SNIP for a relay that's rapidly changing
>   (or equivalently, leave that relay out of the next ENDIVE)
>
> Both these schemes also limit the amount of bandwidth used
> for a relay that's rapidly changing details.

Interesting idea; I think in the case of the first one, we'd be giving
up something important, but I don't know how much so.  The second one
might actually help with our network stability, though.

[...]
>
> Do "tricky restrictions" include the IP subnet restriction (avoid
> relays in the same IPv4 /16 and IPv6 /32) ?

I'm thinking of _all_ tricky restrictions, including but not limited
to IP subnets, families, user settings, and


> What about a heterogenous IPv4 / IPv6 network, where
> IPv4-only relays can't connect to IPv6-only relays?

This one would fit more into "alternative topologies", but I think the
design can handle that.  (See proposal 300 section 3.9.)

The way it would work is, you put IPv4-only relays into group A,
dual-stack relays in group B, and IPv6 relays into group C.

Then you give them different successor lists, so that A has successor
in A and B, C has successors in B and C, and B can have all
successors.

> If we do decide to add IPv6-only relays, we'll probably add
> them in this order:
> * IPv6-only bridges (needs dual-stack bridge guards / middles?)
> * IPv6-only exits (needs dual-stack middles)
> * IPv6-only guards (needs dual-stack middles)
> * IPv6-only middles (needs dual-stack or IPv6-only

[tor-dev] Fwd: Upcoming Tor security releases to fix a denial-of-service issue

[Nick Mathewson]

-- Forwarded message -
From: Nick Mathewson 
Date: Mon, Mar 16, 2020 at 1:25 PM
Subject: Upcoming Tor security releases to fix a denial-of-service issue
To: 


Hello!

Some time this week, we currently plan to put out a set of security
updates for all supported versions of Tor.  These releases will fix a
pair of denial-of-service bugs: one that we are classifying at "low"
severity, and one that we are classifying at "high" severity.

Our recommendation will be for everybody, including relays and
clients, to upgrade once packages are available for their platforms.
Although these vulnerabilities are "only" denial-of-service issues,
any denial-of-service attack against Tor could be leveraged by an
attacker to aid in a traffic analysis attack.

To the best of our knowledge, these vulnerabilities are not being
exploited in the wild.

Currently supported release series are 0.3.5, 0.4.1, 0.4.2, and 0.4.3
(alpha).  If you have not yet upgraded to one of those, the time to do
so is soon.

For our policy and process for handing security issues, please see:
https://trac.torproject.org/projects/tor/wiki/org/teams/NetworkTeam/SecurityPolicy

best wishes,
--
Nick
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Walking Onions status update: week 2 notes

[Nick Mathewson]

Walking onions -- week 2 update

Hi!  On our current grant from the zcash foundation, I'm working on a
full specification for the Walking Onions design.  I'm going to try to
send these out thee updates once a week, in case anybody is interested.

My previous updates are linked below:

 Week 1:
   formats, preliminaries, git repositories, binary diffs,
   metaformat decisions, and Merkle Tree trickery.

   https://lists.torproject.org/pipermail/tor-dev/2020-March/014178.html

You might like to have a look at that update, and its references,
if this update doesn't make sense to you.

===

This week, I worked specifying the nitty-gritty of the SNIP and
ENDIVE document formats.  I used the CBOR meta-format [CBOR] to
build them, and the CDDL specification language [CDDL] to specify
what they should contain.

As before, I've been working in a git repository at [GITHUB]; you
can see the document I've been focusing on this week at
[SNIPFMT].  (That's the thing to read if you want to send me
patches for my grammar.)

There were a few neat things to do here:

   * I had to define SNIPs so that clients and relays can be
 mostly agnostic about whether we're using a merkle tree or a
 bunch of signatures.

   * I had to define a binary diff format so that relays can keep
 on downloading diffs between ENDIVE documents. (Clients don't
 download ENDIVEs).  I did a quick prototype of how to output
 this format, using python's difflib.

   * To make ENDIVE diffs as efficient as possible, it's important
 not to transmit data that changes in every ENDIVE.  To this
 end, I've specified ENDIVEs so that the most volatile parts
 (Merkle trees and index ranges) are recomputed on the relay
 side.  I still need to specify how these re-computations work,
 but I'm pretty sure I got the formats right.

 Doing this calculation should save relays a bunch of
 bandwidth each hour, but cost some implementation complexity.
 I'm going to have to come back to this choice going forward
 to see whether it's worth it.

   * Some object types are naturally extensible, some aren't.  I've
 tried to err on the size of letting us expand important
 things in the future, and using maps (key->value mappings)
 for object that are particularly important.

 In CBOR, small integers are encoded with a little less space
 than small strings.  To that end, I'm specifying the use of
 small integers for dictionary keys that need to be encoded
 briefly, and strings for non-tor and experimental extensions.

   * This is a fine opportunity to re-think how we handle document
 liveness.  Right now, consensus directories have an official
 liveness interval on them, but parties that rely on
 consensuses tolerate larger variance than is specified in the
 consensus.  Instead of that approach, the usable lifetime of
 each object is now specified in the object, and is ultimately
 controlled by the authorities.  This gives the directory
 authorities more ability to work around network tolerance
 issues.

 Having large lifetime tolerances in the context of walking
 onions is a little risky: it opens us up to an attack where
 a hostile relay holds multiple ENDIVEs, and decides which one
 to use when responding to a request.  I think we can address this
 attack, however, by making sure that SNIPs have a published
 time in them, and that this time moves monotonically forward.

   * As I work, I'm identifying other issues in tor that stand in
 the way of a good efficient walking onion implementation that
 will require other follow-up work.  This week I ran into a
 need for non-TAP-based v2 hidden services, and a need for a
 more efficient family encoding.  I'm keeping track of these
 in my outline file.

Fun fact: In number of bytes, the walking onions proposal is now
the 9th-longest proposal in the Tor proposal repository.  And it's
still growing!


Next week, I'm planning to specify ENDIVE reconstruction, circuit
extension, and maybe start on a specification for voting.


[CBOR] RFC 7049: "Concise Binary Object Representation (CBOR)"
https://tools.ietf.org/html/rfc7049b

[CDDL] RFC 8610: "Concise Data Definition Language (CDDL): A
Notational Convention to Express Concise Binary Object
Representation (CBOR) and JSON Data Structures"
https://tools.ietf.org/html/rfc8610

[GITREPO]  https://github.com/nmathewson/walking-onions-wip

[SNIPFMT] 
https://github.com/nmathewson/walking-onions-wip/blob/master/specs/02-endives-and-snips.md
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Walking Onions status update: week 1 notes

[Nick Mathewson]

Hi!  On our current grant from the zcash foundation, I'm working
on a full specification for the Walking Onions design.

If you haven't read it, Walking Onions is a set of ideas meant to
transition Tor away from its current dependency on a directory
system, to improve scaling in the long term, and performance of
low-bandwidth clients in the short term.

I'm going assume in the rest of my email that you already have a
decent idea of how Tor works, and of the Walking Onions proposal.
If you need a Walking Onions refresher, see proposal 300 [PROP300]
and the whitepaper that Chelsea Komlo, Ian Goldberg, and I have
been working on [WHITEPAPER].

The current scope for this work is to try to solve all the open
design questions for Walking Onions, and to write a set of
specifications we could use to build the system.

I'm aiming to have the initial versions of these specs wrapped up
in April.  Rather than dumping everything onto the mailing list at
once, I'm going to try working in the open, sending weekly updates
about my work.

Design and specification are a lot of fun for me; I hope you'll
have fun too reading along.

This past week, I started by writing:

   - An outline of the set of specifications we'd need to have
 before we could implement walking onions. (OUTLINE0.txt)

   - A list of areas of uncertainty that we need to solve before
 or during writing the specs. (UNCERTAINTY.txt)

   - A list of related problems that we might wind up solving
 along with the walking onion specs (OPPORTUNITIES.txt)

   - A rough grouping of initial tasks, which should eventually
 turn into a schedule. (PLAN.txt)

You can find all these documents in a working git repository
[GITREPO] that I've been making for this purpose, if you like.  I
don't intend for this repository to be a permanent reference or to
be necessarily useful for anybody but me: everything important
about it should eventually wind up in specifications and on this
mailing list.

==

The biggest area of uncertainty was to pick a meta-format for
ENDIVEs and SNIPs.  I looked at our existing directory
meta-format, along with protobufs, messagepack, and a whole bunch
of other systems.  I also looked into what kind of efficiency we'd
get by just using a hand-tooled non-extensible binary format, so
that I could see what kind of overhead we were looking at.

The issue here is that I think we don't want to be using a
text-based metaformat for SNIPs: we need SNIPs to be as small as
possible, and our current text-based directory metaformat only
compresses well when we're using huge amounts of it at once.

But if we're doing a binary format, we probably shouldn't
hand-roll it.  Trunnel lets us handle our existing formats safely,
but those formats aren't so efficient in practice, and they aren't
too standardized.  If we use a standardized binary format, then we
get encoding, decoding, debugging dumps, optimization, and lots of
other stuff "for free".

After looking at a pretty huge variety of formats, I think that
our best bet is something called "CBOR", standardized as RFC 7049
[CBOR].  It is substantially based on MessagePack, but with a
number of improvements.  Here are some of CBOR's advantages:

+ It's optimized first for allowing encoding/decoding to be
  done with small, simple code -- and second, for having a
  small binary format.  IMO it succeeds at both: the
  metaformat is simple, and there are decoder implementations
  in less than 1kb of binary size.

+ You can think of its data model as a "binary json"; it's
  very flexible. (yes I know about the other binary jsons)

+ It's standardized, and there are lots of implementations of
  it, in lots of different languages.

+ It has a data definition language [CDDL], which we don't
  actually need to implement, but which we can use to help
  write our specs and validate our implementation.

+ It has a textual encoding for debugging, and a one-way
  mapping to json.

Here are some of its disadvantages:

- It doesn't have a built-in object mapping like protobufs
  does.

- It allows multiple encodings for a single piece of
  data. (The CBOR RFC explains how to make a canonical
  encoding.)

- Our diff story gets slightly more complex when we have a
  binary format.

(I'd be happy to talk about other possibilities, but please make sure you
know about the bikeshed first. [BIKESHED])

==

Doing diffs from one ENDIVE to the next will be helpful if we want
to keep relay bandwidth on the small side.  We'll need binary
diffs for this.  Fortunately, I think that shouldn't be too hard
to do with our existing diff code: instead of doing line-by-line
diffs, we'll just do CBOR-item-by-item diffs, and encode them in a
byte-oriented way.  I've got a prototype half-written to make sure
this is right.  The simplicity of CBOR makes this pretty easy.

==

There are a few ways to authenticate SNIPs.  It's looking like
right now the 

Re: [tor-dev] Proposal 313: Relay IPv6 Statistics

[Nick Mathewson]

On Mon, Feb 10, 2020 at 1:37 AM teor  wrote:

Hi, Teor!  This proposal looks good and thorough to me.  I have only a
couple of questions on section 3:

> 3. Logging IPv6 Relays in the Consensus
>
>We propose that relays (and bridges) log:
>  * the number of relays, and
>  * the consensus weight fraction of relays,
>in the consensus that:
>  * have an IPv6 ORPort,
>  * support IPv6 reachability checks, and
>  * support IPv6 clients.

I don't understand the motivation behind doing this in the Tor code,
since it's not something that relay operators need to know about or
take action on.  To me, it seems more like something do do as part of
metrics than in Tor per se.

 [...]
>"Usable Guards" have the Guard flag, but do not have the Exit flag. If the
>Guard also has the BadExit flag, the Exit flag should be ignored.

It seems to me that this rule should depend on the Wgd
bandwidth-weights value ("Weight for Guard+Exit-flagged nodes in the
guard Position"), right?  (Right now that is zero, and I don't expect
it to change.)

See also Karsten's response here for more information.
-- 
Nick
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Proposal 312: Automatic Relay IPv6 Addresses

[Nick Mathewson]

On Wed, Jan 29, 2020 at 9:04 AM teor  wrote:

Hello again!  This looks like another fine proposal.  I'm leaving
comments inline, and clipping sections that I'm not commenting on.


>
> Filename: 312-relay-auto-ipv6-addr.txt
> Title: Tor Relays Automatically Find Their IPv6 Address
> Author: teor
> Created: 28-January-2020
> Status: Draft
> Ticket: #33073
>
> 0. Abstract
>
>We propose that Tor relays (and bridges) should automatically find their
>IPv6 address, and use it to publish an IPv6 ORPort. For some relays to find
>their IPv6 address, they may need to fetch some directory documents from
>directory authorities over IPv6. (For anonymity reasons, bridges are unable
>to fetch directory documents over IPv6, until clients start to do so.)
>
> 1. Introduction
>
>Tor relays (and bridges) currently find their IPv4 address, and use it as
>their ORPort and DirPort address when publishing their descriptor. But
>relays and bridges do not automatically find their IPv6 address.

At the beginning of this document, we should be a bit more clear about
which address specifically we're trying to find.  If we wanted _some_
address, or if NAT and firewalls didn't exist, we could just open a
socket, call getsockname(), and be done with it.  What we are looking
for specifically is an address that we can advertise to the rest of
the world in our server descriptor.  [I know you know this, but we
should say so.]


 [...]
> 3. Finding Relay IPv6 Addresses
>
>We propose that tor relays (and bridges) automatically find their IPv6
>address, and use it to publish an IPv6 ORPort.
>
>For some relays to find their IPv6 address, they may need to fetch some
>directory documents from directory authorities over IPv6. (For anonymity
>reasons, bridges are unable to fetch directory documents over IPv6, until
>clients start to do so.)
>
> 3.1. Current Relay IPv4 Address Implementation
>
>Currently, all relays (and bridges) must have an IPv4 address. IPv6
>addresses are optional for relays.
>
>Tor currently tries to find relay IPv4 addresses in this order:
>  1. the Address torrc option
>  2. the address of the hostname (resolved using DNS, if needed)
>  3. a local interface address
> (by making a self-connected socket, if needed)
>  4. an address reported by a directory server (using X-Your-Address-Is)

Any server, or only an authority?  Over any connection, or only an
authenticated one?

 [...]
> 3.2. Finding Relay IPv6 Addresses
>
>We propose that relays (and bridges) try to find their IPv6 address. For
>consistency, we also propose to change the address resolution order for
>IPv4 addresses.
>
>We use the following general principles to choose the order of IP address
>methods:
>  * Explicit is better than Implicit,
>  * Local Information is better than a Remote Dependency,
>  * Trusted is better than Untrusted, and
>  * Reliable is better than Unreliable.
>Within these constraints, we try to find the simplest working design.

We should make sure to be clear about the impact of using an untrusted
source.  Anybody who can fool a relay about its IP can effectively
MITM that relay's incoming connections (traffic patterns only), so
using a non-trusted source can be risky for anonymity.

 [...]
>(Each of these address resolution steps is described in more detail, in its
>own subsection.)
>
>While making these changes, we want to preserve tor's existing behaviour:
>  * resolve Address using the local resolver, if needed,
>  * ignore private addresses on public tor networks, and
>  * when there are multiple valid addresses, choose the first or latest
>address, as appropriate.

Instead of "first or latest" I suggest "first-listed or most recently
received" here, to help non-native speakers.

> 3.2.1. Make the Address torrc Option Support IPv6
 [...]
>It is an error to configure an Address option with a private IPv4 or IPv6
>address, or with a hostname that does not resolve to any publicly routable
>IPv4 or IPv6 addresses.

We should say "on a public network" here -- private addresses are fine
on private networks.

Also, this seems to mean that if the relay's DNS resolver goes down,
the relay should give an error and exit, even if it was already
running.  That seems undesired.

 [...]
> 3.2.2. Use the Advertised ORPort IPv4 and IPv6 Addresses
>
>Next, we propose that relays (and bridges) use the first advertised ORPort
>IPv4 and IPv6 addresses, as configured in their torrc.
>
>The ORPort address may be a hostname. If it is, tor should try to use it to
>resolve an IPv4 and IPv6 address, and open ORPorts on the first available
>IPv4 and IPv6 address. Tor should respect the IPv4Only and IPv6Only port
>flags, if specified. (Tor currently resolves IPv4 addresses in ORPort
>lines. It may not look for an IPv6 address.)
>
>Relays (and bridges) currently 

Re: [tor-dev] Prop 311: Relay IPv6 Reachability

[Nick Mathewson]

On Wed, Jan 29, 2020 at 8:56 AM teor  wrote:

Thanks for the followup, Teor!  I have only one comment on the changes:

> I made these useful features "may not" in:
> https://github.com/torproject/torspec/pull/103/commits/2662c688170696fbed2ba7e4dfa7f33ccf702435

We should avoid "may not" in standards documents because it can be
ambiguous.  "Nick may not eat breakfast" can mean either that I'm not
allowed to eat breakfast, or that it's possible I won't eat breakfast.

I think this is the reason why RFC 2119 does not include "MAY NOT" as
an option.  Let's use "should not" or "or "may refrain from" or "might
not" as appropriate?

best wishes,
-- 
Nick
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Prop 311: Relay IPv6 Reachability

[Nick Mathewson]

On Fri, Jan 24, 2020 at 1:31 AM teor  wrote:
>
> Hi,
>
> Here is an initial draft of Proposal 311: Relay IPv6 Reachability.
>
> This proposal includes:
>   * relay IPv6 ORPort extends, and
>   * relay IPv6 ORPort reachability checks.
>
> This is the first of 3 proposals:
> * Proposal 311: Relay IPv6 Reachability
> * Proposal 312: Automatic Relay IPv6 Addresses
> * Proposal 313: Relay IPv6 Statistics
> (I haven't written the others yet!)
>
> I also want to make some minor changes to Proposal 306, so that bridge
> IPv6 behaviour stays in sync with client IPv6 behaviour.
>
> There are still a few TODO items in the proposal, mostly about Tor's
> current behaviour. If you know the answers, please let me know.
>
> The full text is included below, and it is also available as a GitHub
> pull request:
> https://github.com/torproject/torspec/pull/103
>
> The related tickets are #24404 (proposal) and #24403 (implementation):
> https://trac.torproject.org/projects/tor/ticket/24404
> https://trac.torproject.org/projects/tor/ticket/24403
>
> Please feel free to reply on this list, or via GitHub pull request
> comments.

Hello, Teor!  I hope this reply finds you well.

This proposal seems like a good start: It's solid, readable, and I
think it would work.  There are few places where I have questions or
suggested clarifications, and a few where I think we should consider
other designs as well.

I'm going to only quote the parts I'm commenting on.

> 3.2.1. Making IPv6 ORPort Extend Connections
>
>Relays can make a new connection over IPv6 when:
>  * there is no existing authenticated connection to the requested relay,
>and
>  * the extend cell contains an IPv6 ORPort.

Also we should probably have the condition, "The relay itself supports
IPv6, or thinks that it does"?

>If these conditions are satisfied, and the extend cell also contains an
>IPv4 ORPort, we propose that the relay choose between an IPv4 and an IPv6
>connection at random.
>
>If the extend cell does not contain an IPv4 ORPort, we propose that the
>relay connects over IPv6. (Relays should support IPv6-only extend cells,
>even though they are not used to test relay reachability in this proposal.)
>
>A successful IPv6 connection also requires that:
>  * the requested relay has an IPv6 ORPort.
>But extending relays must not check the consensus for other relays' IPv6
>information. Consensuses may be out of date, particularly when relays are
>doing reachability checks for new IPv6 ORPorts.
>
>See section 3.3.2 for other situations where IPv6 information may be
>incorrect or unavailable.

[...]
> 3.3. Alternative Extend Designs
>
>We briefly mention some potential extend designs, and the reasons that
>they were not used in this proposal.
>
>(Some designs may be proposed for future Tor versions, but are not 
> necessary
>at this time.)

Let me sketch another alternative design here:

Suppose that a relay's extend cell contains the IPv4 address and the
IPv6 address in their _preferred order_.  So if the party generating
the extend cell would prefer an IPv4 connection, it puts the IPv4
addess first; if it would prefer an IPv6 connection, it puts the IPv6
address first.

The relay that receives the extend cell could respond in several ways:
   * One possibility (similar to your 3.2.1) is to choose at random,
with a higher probability given to the first option.
   * One possibility (similar to your 3.3.1) is to try the first, and
then try the second if the first one fails.

I think this scheme has some advantage, in that it lets the
self-testing relay say "please try IPv6 if you can" or "please try
IPv4 if you can" in a reliable way, and lets us migrate from the
current behavior to the 3.3.1 behavior down the road.


[...]
> 4.3. Refuse to Publish Descriptor if IPv6 ORPort is Unreachable
>
>If an IPv6 ORPort reachability check fails, relays (and bridges) should log
>a warning.
>
>In addition, if IPv6ORPort reachability checks are reliable, based on the
>number of relays in the network that support IPv6 extends, relays should
>refuse to publish their descriptor.

Is this the behavior we want?  Another behavior would be to omit our
IPv6 orport from our descriptor if it is not reachable.  That way,
relay operators could configure their relays to just listen on IPv6
unconditionally, and let the relay figure out whether it should
advertise IPv6 or not.  This behavior  seems preferable to me, since
it has less chance of breaking a relay that would otherwise "work".

As a third possibility, we could have an option to say whether we
should publish a descriptor if some of our addresses are not
reachable.

 [...]
> 4.3.2. Add AssumeIPv6Reachable Option
>
>We add an AssumeIPv6Reachable torrc option and consensus parameter.
>
>If IPv6 ORPort checks have bugs that impact the health of the network,
>they can be disabled by setting AssumeIPv6Reachable=1 in the 

Re: [tor-dev] Changes to accessing and using MaxMind's databases

[Nick Mathewson]

On Thu, Jan 9, 2020 at 8:25 AM Karsten Loesing  wrote:
>
> Hi!
>
> When trying to update tor's geoip databases the other day I found that
> MaxMind's GeoLite2 database is not available for download anymore. The
> reason is:
>
> https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-using-geolite2-databases/
>
> We do not have a MaxMind account (yet). As of now, we cannot update
> tor's geoip files nor the files used by Onionoo.
>
> Should we try to get somebody who knows more about licenses and legal
> stuff to review their GeoLite2 EULA and tell us if it's okay for us to
> sign up for a MaxMind account? A possible downside would be that whoever
> wants to verify that we didn't mess with their database when converting
> it to our format would have to sign up for an account, too.
>
> An alternative is to find another, truly open data source than MaxMind
> databases (#25542, #26585). However, this could eat up more time than we
> currently have available, and we should have something ready in a few
> weeks from now. I'm not sure how we would squeeze this into the metrics
> team schedule, so we might need help with this.
>
> Thoughts on the two alternatives? What else did I miss?

There's a debian-legal thread here:

https://lists.debian.org/debian-legal/2020/01/threads.html

From what I can tell, Maxmind's new EULA is not compatible with
redistributing the information as free software, but of course I am
not a lawyer.

-- 
Nick
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Onion DoS: Killing rendezvous circuits over the application layer

[Nick Mathewson]

On Mon, Dec 2, 2019 at 9:16 AM George Kadianakis  wrote:

> However, IMO the right way to do this feature, would be to improve the control
> port code and design so that it doesn't get so overwhelmed by multiple
> events. That said, I'm not sure exactly what kind of changes we would have to
> do to the control port to actually make it a viable option, and it seems to me
> like a pretty big project that serves as a medium-term to long-term solution
> (which we have no resources to pursue right now), whereas the hack of this
> thread is more of a short-term solution.

I think I agree with you here about this part.

A quick question that might help us: _why_ is the control port code
slow in this case?  Is the problem that a bunch of events are queued,
and that keeps the controller from getting events that it needs?  Or
that the control port doesn't listen for incoming commands until the
events are flushed (seems unlikely to me)?  Or that a bunch of events
are queued, and that keeps the controller from getting the ACK for its
close command?Or something else?

For some of these cases, there might be a quick workaround in just
having the controller open two connections: one for bulk and one for
low-latency.

-- 
Nick
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev