[tor-dev] RFC: Trunnel -- a binary parser generator for Tor and more

2014-08-13 Thread Nick Mathewson
Hi!

Back when the Heartbleed vulnerability went public, I thought to
myself: "If we keep generating writing C code to parse binary formats
by hand, the same thing will happen to us sooner or later."

So on and off, I've been working on a little tool to automatically
generate binary encoding and parsing code based on C-like structure
descriptions.  It's a bit like nail[0], but more lightweight: it
doesn't support backtracking, sub-byte alignment, or any of several
other features that Tor doesn't actually want.

You can see a pre-alpha version of the tool (called "Trunnel") in one
of my git repositories [1].  The "doc" subdirectory has a simple
manual [2].

To make sure that it would suit our needs, I wrote up trunnel
specification files for the current Tor protocols, for SOCKS, and for
the new protocols in proposals 220, 224, and 229.  It seems to work
okay.

In case you want to see what the tool does without running it
yourself, you can find the SOCKS+proposal229 specification here [3],
and compare it with the generated header [4] and C implementation [5].

Trunnel is written in Python; my tests have 99% line coverage on the
Python code. The C code which is generated by those tests itself has
99% line coverage from other unit tests, so I'm thinking that it's
reasonably tested.  (Branch coverage is lower, but still pretty high.)

So, please check this out and have a look at it!  My open questions are:

  * Is this a good idea?
  * Is it a good idea to use this in Tor?
  * Are there any tricky bugs left in the generated code?
  * What am I forgetting to think of?

FWIW, my current plan is not to do a blanket replacement of our
existing parse code with Trunnel (yet) but rather to use Trunnel to
generate parsing code for new cell formats that are getting added in
0.2.6 and later.

[0] https://github.com/jbangert/nail
[1]  https://gitweb.torproject.org/user/nickm/trunnel.git .
[2] Also online at http://www.wangafu.net/~nickm/trunnel-manual.html
[3] 
https://gitweb.torproject.org/user/nickm/trunnel.git/blob/HEAD:/examples/socks5.trunnel
[4] http://www.wangafu.net/~nickm/volatile/socks5.h
[5] http://www.wangafu.net/~nickm/volatile/socks5.c

cheers,
-- 
Nick Mathewson
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev


Re: [tor-dev] RFC: Trunnel -- a binary parser generator for Tor and more

2014-10-01 Thread Nick Mathewson
An update:  Trunnel is now at version 1.3.  Its official repository is now
https://gitweb.torproject.org/trunnel.git .  Some code that  it has
generated has been merged into the Tor master branch for the 0.2.6
release series, though that code is not yet in active use.

  More review and comments are always appreciated. :)

On Wed, Aug 13, 2014 at 3:37 PM, Nick Mathewson  wrote:
> Hi!
>
> Back when the Heartbleed vulnerability went public, I thought to
> myself: "If we keep generating writing C code to parse binary formats
> by hand, the same thing will happen to us sooner or later."
>
> So on and off, I've been working on a little tool to automatically
> generate binary encoding and parsing code based on C-like structure
> descriptions.  It's a bit like nail[0], but more lightweight: it
> doesn't support backtracking, sub-byte alignment, or any of several
> other features that Tor doesn't actually want.
>
> You can see a pre-alpha version of the tool (called "Trunnel") in one
> of my git repositories [1].  The "doc" subdirectory has a simple
> manual [2].
>
> To make sure that it would suit our needs, I wrote up trunnel
> specification files for the current Tor protocols, for SOCKS, and for
> the new protocols in proposals 220, 224, and 229.  It seems to work
> okay.
>
> In case you want to see what the tool does without running it
> yourself, you can find the SOCKS+proposal229 specification here [3],
> and compare it with the generated header [4] and C implementation [5].
>
> Trunnel is written in Python; my tests have 99% line coverage on the
> Python code. The C code which is generated by those tests itself has
> 99% line coverage from other unit tests, so I'm thinking that it's
> reasonably tested.  (Branch coverage is lower, but still pretty high.)
>
> So, please check this out and have a look at it!  My open questions are:
>
>   * Is this a good idea?
>   * Is it a good idea to use this in Tor?
>   * Are there any tricky bugs left in the generated code?
>   * What am I forgetting to think of?
>
> FWIW, my current plan is not to do a blanket replacement of our
> existing parse code with Trunnel (yet) but rather to use Trunnel to
> generate parsing code for new cell formats that are getting added in
> 0.2.6 and later.
>
> [0] https://github.com/jbangert/nail
> [1]  https://gitweb.torproject.org/user/nickm/trunnel.git .
> [2] Also online at http://www.wangafu.net/~nickm/trunnel-manual.html
> [3] 
> https://gitweb.torproject.org/user/nickm/trunnel.git/blob/HEAD:/examples/socks5.trunnel
> [4] http://www.wangafu.net/~nickm/volatile/socks5.h
> [5] http://www.wangafu.net/~nickm/volatile/socks5.c
>
> cheers,
> --
> Nick Mathewson
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev