Re: [tor-dev] Brainstorming ideas for controller features for improved testing; want feedback
grarpamp writes: > Combine setevents circ and stream > https://trac.torproject.org/projects/tor/ticket/11179 Could this just be done with some simple controller code using the existing API? Something like "carml monitor" plus some grep should get you 90% of the way there... https://carml.readthedocs.org/en/latest/command-monitor.html Sounds like you want this for debugging user problems, though? I guess I'm suggesting that perhaps the controller-side is a good place to do this aggregation, etc instead of more code in Tor, or am I missing something? -- meejah ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] Brainstorming ideas for controller features for improved testing; want feedback
Would like to see some of these implemented. Presume maybe this thread means some funds appeared for HS love and/or the control port. Not sure if it is control port only. Combine setevents circ and stream https://trac.torproject.org/projects/tor/ticket/11179 signal NEWNYM exit bucketing https://trac.torproject.org/projects/tor/ticket/6256 MAPADDRESS for IP ranges (CIDR, etc) https://trac.torproject.org/projects/tor/ticket/3982 Allow controllers to retrieve HS descriptors from Tor https://trac.torproject.org/projects/tor/ticket/3521 ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] Brainstorming ideas for controller features for improved testing; want feedback
Hi, please also consider the 'Better hidden service support on Tor control interface': https://trac.torproject.org/projects/tor/ticket/8993 Looks like what's currently missing is a control port keyword for trac. Cheers, Patrick ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] Brainstorming ideas for controller features for improved testing; want feedback
Hi! Let Tor tell us what relays, bridges, authority [mirrors] think what time it is to be able to implement a distributed time fetcher: https://trac.torproject.org/projects/tor/ticket/6894 Useful in context of 'get independent from host clock time / insecure NTP': https://trac.torproject.org/projects/tor/ticket/8170 Or the more limited proposal: https://trac.torproject.org/projects/tor/ticket/3652 Cheers, Patrick ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] Brainstorming ideas for controller features for improved testing; want feedback
On Fri, Mar 20, 2015 at 11:55 AM, Nick Mathewson wrote:
> Hi! I've got an end-of-month deliverable to flesh out as many good
> ideas here as I can, and I'd appreciate feedback on what kind of
> features it would be good to add to the controller protocol in order
> to better support testing.
>
> More ideas would be most welcome.
Thanks for the comments, all!
Here's the latest version, with more things folded in, and some syntax sketches.
All syntaxes below are examples meant to focus discussion around
what might appear in a spec, not final (ha!) formal specification.
IDEAS
=
1. Step-by-step hidden service connections
Add the ability to create connections to hidden services step by
step, to best
What's necessary here is commands to:
* Establish a rendezvous point on a given circuit.
* Construct and send an introduce1 cell on a given circuit.
* Realize that a rendezvous circuit has been constructed.
ESTABLISH_REND {circuit_num} {rend_token}
SEND_INTRO1 {circuit_num} intro_key={key} authinfo={ nfo}
650 RECEIVED_REND_OK {circuit_num}
2. Send a single cell on a circuit
(TESTING ONLY)
For fuzzing and low-level testing purposes, it would be handy to be
able to send a single cell on a tor circuit.
This might be better to expose via a low-level modular API than via
the control port.
SEND_RAW_RELAY_CELL {circuit_num} LAYER={cpath layer} DATA={hex}
3. Intercept cell by cell on a circuit
(TESTING ONLY)
For fuzzing, testing, and debugging purposes, it might be handy for
a controller to be able to observe data cell by cell on a circuit of
interest.
This might be better to expose via a low-level modular API than via
the control port.
650 RELAY_CELL {circuit_num} LAYER={cpath layer} DATA={hex}
4. Send a single cell on a connection.
(TESTING ONLY)
As 2, but for connections. Note that we might even, for testing,
expose this at a sub-cell level.
SEND_RAW_CELL {conn id} DATA={hex}
5. Intercept all cells on a connection
(TESTING ONLY)
As 2, but for connections.
650 CONN_CELL {conn id} DATA={hex}
6. Plug-in to handle a relay or other command.
Right now, all Tor's features need to be baked into Tor; it's not
easy to write extensions. We could change that by having the
controller able to intersect particular relay or extension commands
and act accordingly. This could be used for prototyping new
features, etc.
HANDLE_EXTENSION {conn=id|circ=id|conn=*|circ=*} {command_num}
650 EXTENSION {conn=id|circ=id} {command_num} DATA
7. Force a given protocol on a given connection
We could add a feature to restrict what protocols can be negotiated
on a given connection we create. This could help us better test our
protocols for interoperatbility.
OPEN_CONN {addr=address...} {protos=1,2,3...} {KEYID1=...}
8. Examine fine-grained connection detail.
There are many data available for a given connection (such as
fine-grained TLS information) that are not currently exposed on the
GETINFO interface. We could make most of this available for testing,
pending security analysis.
GETINFO orconn//all
GETINFO orconn//item
9. Examine cache in detail
In the past we've seen crazy issues with our descriptor caching
code. It might be good to expose for testing information about
where exactly descriptors are stored, what attributes are set on
them, and so on. We could also expose events for cache compaction
and discarded expired descriptors.
This might also (TESTING ONLY) extend to events for access of cache
documents, information about HS Descs, etc, though that's a bit worrisome.
GETINFO cache//keys
GETINFO cache//by-key// ...
raw
annotations
location
650 CACHE_COMPACTED (bytes_saved)
650 CACHE_DISCARDED (item)
10. Fetch literal documents
Currently there's no way for a controller to ask Tor to download a
given descriptor or microdescriptor or networkstatus. That could
change.
DOWNLOAD [{circ=circ_id} {anon=0|1} {cache=router_id}]
11. OOM stats
To resist out-of-memory attacks, we track our memory usage and kill
off circuits as needed when memory gets low. We could expose the
memory thresholds and current sizes via one or more controller
commands.
GETINFO mem-usage/all
GETINFO mem-usage/
650 MEM_USAGE k=v k=v k=v
650 MEM_FREED k=v k=v k=v
12. Timeout values
Tor has a truly huge variety of internal timers to ensure that given
periodic events happen enough; we could expose those, and (TESTING
ONLY) allow controllers to adjust them or trigger their corresponding
events.
GETINFO timer/list
GETINFO timer/expires/
ADJTIMER .
650 TIMER next=
13. Detailed connection debugging info
Current connection events expose only large-scale state changes in
connections; we could instead expose every state transition at the
cell
Re: [tor-dev] Brainstorming ideas for controller features for improved testing; want feedback
Hi! > On 20 Mar 2015, at 16:55, Nick Mathewson wrote: > > IDEAS > = Many great ideas! Here's a few more, mostly just adding a bit to the things you already listed. Some of them make me kind of uneasy because they really are testing only and could be used for much unhappiness on the real network: Expose more per-circuit information to the controller, like bytes sent/received on this circuit, maybe even an event any time new bytes are received (testing only). This could help debug stuff like slow connections, inefficiencies in the protocol, etc. Allow write access to everything in the state file (testing only). Access to all stored cached documents (TESTING ONLY) (thinking about consensus diffs, hs descriptors, etc). Also an event (TESTING ONLY) whenever such a document is accessed/served. Modify state of existing connections at relays (TESTING ONLY), like redirecting to a different host, killing it, etc. I'll add more if I think of anything else. Cheers Sebastian signature.asc Description: Message signed with OpenPGP using GPGMail ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] Brainstorming ideas for controller features for improved testing; want feedback
Michael Rogers writes: > On 20/03/15 15:55, Nick Mathewson wrote: >> 27. Hidden service intropoint changes, desc changes, uploads >> >> Many hidden service transitions currently generate no events. We >> could at minimum generate events for changed inroduction points, >> changed hidden service descriptors, uploading our own HS descriptor. > An HS descriptor upload event would be useful for apps that use hidden > services for p2p connections - we can avoid polling for descriptors if > we know when our own descriptor has been published. +2 on this (and not just debug-mode only, please :) Currently, txtorcon parses INFO logs for the right log message which is obviously pretty fragile and hack-y. It does "work", but still... -- meejah ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] Brainstorming ideas for controller features for improved testing; want feedback
On 20/03/15 15:55, Nick Mathewson wrote: > 27. Hidden service intropoint changes, desc changes, uploads > > Many hidden service transitions currently generate no events. We > could at minimum generate events for changed inroduction points, > changed hidden service descriptors, uploading our own HS descriptor. An HS descriptor upload event would be useful for apps that use hidden services for p2p connections - we can avoid polling for descriptors if we know when our own descriptor has been published. > 32. Forget cached information > > To better test our download logic, it would be helpful to have a way > to drop items from our caches. This too. (I have a patch for purging entries from the HS descriptor cache that I'll bring to the patch workshop one of these weeks...) Cheers, Michael signature.asc Description: OpenPGP digital signature ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] Brainstorming ideas for controller features for improved testing; want feedback
On 20 Mar 2015, at 12:33, Jeff Burdges wrote: > I could imagine an “onion token” variant of ephemeral hidden services in > which the person who initiates the connection does not know what they’re > connecting to, like sending a message to a mailbox. Example : > > Alice wants Bob to send her a message asynchronously by anonymously dropping > it into a numbered mailbox system, but Alice only wants to check one mailbox > for all her contacts, so she does not want Bob to be able to reveal her > mailbox. > > Rough outline : > - Alice gives bob a “token” that contains a bunch of pre-encrypted tor > extends, data, etc. frames and some additional data such as symmetric keys. > Alice goes offline. Actually Alice building this token would presumably involve the mailbox system too since it’s operating as a hidden service itself. > - Bob sends Alice’s mailbox a message by building a circuit to a specified > machine, encrypting each of the frames supplied by Alice for all of his > circuit except the endpoint because Alice already did that encryption, and > sending them. > - These frames continue building a circuit from that endpoint to wherever > Alice wants it to go. > - Bob encrypts his data frames using first the additional data supplied by > Alice so that they can traverse this longer circuit that he only understands, > and then encrypts those for the portion of the circuit he understands. > - Alice logs back in, contacts the mailbox hidden service, and retrieves her > messages, including Bob’s message. > > Optional : > - Amongst the frames Bob needs to use to set up the circuit might be one that > causes re-incryption so that even if an adversary hacked both Bob and the > mailbox system they cannot search the mailbox system for Bob’s message. > > Of course “onion tokens” would not live forever since Alice’s token fails to > describe a valid circuit if any server she selected goes down, but maybe it’s > provide a nice short-term asynchronous delivery options for IM systems like > Ricochet. > https://github.com/ricochet-im/ricochet > > Best, > Jeff ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] Brainstorming ideas for controller features for improved testing; want feedback
I could imagine an “onion token” variant of ephemeral hidden services in which the person who initiates the connection does not know what they’re connecting to, like sending a message to a mailbox. Example : Alice wants Bob to send her a message asynchronously by anonymously dropping it into a numbered mailbox system, but Alice only wants to check one mailbox for all her contacts, so she does not want Bob to be able to reveal her mailbox. Rough outline : - Alice gives bob a “token” that contains a bunch of pre-encrypted tor extends, data, etc. frames and some additional data such as symmetric keys. Alice goes offline. - Bob sends Alice’s mailbox a message by building a circuit to a specified machine, encrypting each of the frames supplied by Alice for all of his circuit except the endpoint because Alice already did that encryption, and sending them. - These frames continue building a circuit from that endpoint to wherever Alice wants it to go. - Bob encrypts his data frames using first the additional data supplied by Alice so that they can traverse this longer circuit that he only understands, and then encrypts those for the portion of the circuit he understands. - Alice logs back in, contacts the mailbox hidden service, and retrieves her messages, including Bob’s message. Optional : - Amongst the frames Bob needs to use to set up the circuit might be one that causes re-incryption so that even if an adversary hacked both Bob and the mailbox system they cannot search the mailbox system for Bob’s message. Of course “onion tokens” would not live forever since Alice’s token fails to describe a valid circuit if any server she selected goes down, but maybe it’s provide a nice short-term asynchronous delivery options for IM systems like Ricochet. https://github.com/ricochet-im/ricochet Best, Jeff On 20 Mar 2015, at 11:55, Nick Mathewson wrote: > Hi! I've got an end-of-month deliverable to flesh out as many good > ideas here as I can, and I'd appreciate feedback on what kind of > features it would be good to add to the controller protocol in order > to better support testing. > > More ideas would be most welcome. > > Yes, some of these ideas are probably foolish or pointless or > half-baked or useless or even dangerous; this is a brainstorming > exercise, not a declaration of intent. The goal right now is to > generate a lot of ideas and thoughts now, and to make decisions about > what to build later. > > > > > IDEAS > = > > > 1. Step-by-step hidden service connections > > Add the ability to create connections to hidden services step by > step, to best > > What's necessary here is commands to: > * Establish a rendezvous point on a given circuit. > * Construct and send an introduce2 cell on a given circuit. > * Realize that a rendezvous circuit has been constructed. > > > 2. Send a single cell on a circuit > > (TESTING ONLY) > > For fuzzing and low-level testing purposes, it would be handy to be > able to send a single cell on a tor circuit. > > This might be better to expose via a low-level modular API than via > the control port. > > 3. Intercept cell by cell on a circuit > > (TESTING ONLY) > > For fuzzing, testing, and debugging purposes, it might be handy for > a controller to be able to observe data cell by cell on a circuit of > interest. > > This might be better to expose via a low-level modular API than via > the control port. > > 4. Send a single cell on a connection. > > (TESTING ONLY) > > As 2, but for connections. Note that we might even, for testing, > expose this at a sub-cell level. > > 5. Intercept all cells on a connection > > (TESTING ONLY) > > As 2, but for connections. > > 6. Plug-in to handle a relay or other command. > > Right now, all Tor's features need to be baked into Tor; it's not > easy to write extensions. We could change that by having the > controller able to intersect particular relay or extension commands > and act accordingly. This could be used for prototyping new > features, etc. > > 7. Force a given protocol on a given connection > > We could add a feature to restrict what protocols can be negotiated > on a given connection we create. This could help us better test our > protocols for interoperatbility. > > 8. Examine fine-grained connection detail. > > There are many data available for a given connection (such as > fine-grained TLS information) that are not currently exposed on the > GETINFO interface. We could make most of this available for testing, > pending security analysis. > > 9. Examine cache in detail > > In the past we've seen crazy issues with our descriptor caching > code. It might be good to expose for testing information about > where exactly descriptors are stored, what attributes are set on > them, and so on. We could also expose events for cache compaction > and discarded expired
[tor-dev] Brainstorming ideas for controller features for improved testing; want feedback
Hi! I've got an end-of-month deliverable to flesh out as many good ideas here as I can, and I'd appreciate feedback on what kind of features it would be good to add to the controller protocol in order to better support testing. More ideas would be most welcome. Yes, some of these ideas are probably foolish or pointless or half-baked or useless or even dangerous; this is a brainstorming exercise, not a declaration of intent. The goal right now is to generate a lot of ideas and thoughts now, and to make decisions about what to build later. IDEAS = 1. Step-by-step hidden service connections Add the ability to create connections to hidden services step by step, to best What's necessary here is commands to: * Establish a rendezvous point on a given circuit. * Construct and send an introduce2 cell on a given circuit. * Realize that a rendezvous circuit has been constructed. 2. Send a single cell on a circuit (TESTING ONLY) For fuzzing and low-level testing purposes, it would be handy to be able to send a single cell on a tor circuit. This might be better to expose via a low-level modular API than via the control port. 3. Intercept cell by cell on a circuit (TESTING ONLY) For fuzzing, testing, and debugging purposes, it might be handy for a controller to be able to observe data cell by cell on a circuit of interest. This might be better to expose via a low-level modular API than via the control port. 4. Send a single cell on a connection. (TESTING ONLY) As 2, but for connections. Note that we might even, for testing, expose this at a sub-cell level. 5. Intercept all cells on a connection (TESTING ONLY) As 2, but for connections. 6. Plug-in to handle a relay or other command. Right now, all Tor's features need to be baked into Tor; it's not easy to write extensions. We could change that by having the controller able to intersect particular relay or extension commands and act accordingly. This could be used for prototyping new features, etc. 7. Force a given protocol on a given connection We could add a feature to restrict what protocols can be negotiated on a given connection we create. This could help us better test our protocols for interoperatbility. 8. Examine fine-grained connection detail. There are many data available for a given connection (such as fine-grained TLS information) that are not currently exposed on the GETINFO interface. We could make most of this available for testing, pending security analysis. 9. Examine cache in detail In the past we've seen crazy issues with our descriptor caching code. It might be good to expose for testing information about where exactly descriptors are stored, what attributes are set on them, and so on. We could also expose events for cache compaction and discarded expired descriptors. 10. Fetch literal documents Currently there's no way for a controller to ask Tor to download a given descriptor or microdescriptor or networkstatus. That could change. 11. OOM stats To resist out-of-memory attacks, we track our memory usage and kill off circuits as needed when memory gets low. We could expose the memory thresholds and current sizes via one or more controller commands. 12. Timeout values Tor has a truly huge variety of internal timers to ensure that given periodic events happen enough; we could expose those, and (TESTING ONLY) allow controllers to adjust them or trigger their corresponding events. 13. Detailed connection debugging info Current connection events expose only large-scale state changes in connections; we could instead expose every state transition at the cell-by-cell handshake level. 14. Detailed circuit debugging info As 13 but for circuits. 15. Halt main loop except for control layer. (TESTING ONLY) For inspection/debugging purposes, it might be clever to have Tor be able to freeze itself, except for the control layer, and let the controller inquire about information. This presents implementation challenges, and is probably not a great idea to do before a _big_ refactoring. 16. Service a single connection (TESTING ONLY) Currently controllers can disable circuit construction or stream attachment, and do them manually. We might also do this for connections, allowing a testing controller to trace what Tor does cell by cell on a single connection. 17. All rephist data There are many data about history and usage in rephist.c (which stands for 'reputation and history!'). We could expose them, to let us better test them. Some of this might be useful for Seth (previously arm) users. Spec: This would use GETINFO extensions, and probably some new events. 18. All ratelim data Sometimes our rate-limiting code can get wonky. It would be great to expose it to Tor con
