Re: [tor-relays] Testing Golang relay implementation

[grarpamp]

On Tue, Oct 24, 2017 at 12:18 AM, Michael McLoughlin
 wrote:
> I am working on a pure Golang relay implementation.
> https://github.com/mmcloughlin/pearl/

Please add your implementation (and any others known, and their locations,
as may have been mentioned or updated in this thread)
to the list of implementations maintained here...

https://trac.torproject.org/projects/tor/wiki/doc/ListOfTorImplementations

This can help bring participants, review, etc to all.
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] Testing Golang relay implementation

[teor]

On 27 Oct 2017, at 09:00, Roger Dingledine  wrote:

>> On Thu, Oct 26, 2017 at 02:56:03PM -0700, Michael McLoughlin wrote:
>> After another look at the spec, I still believe the descriptor I'm
>> publishing conforms, as was my intention.

Each of the bugs you found are very useful to us, because they help
the next person who tries to implement something that works with Tor.

In particular, we almost forgot this one:

If directory authorities treat the "proto" line as mandatory for non-Tor
implementations, we should probably document this fact here:

https://gitweb.torproject.org/torspec.git/tree/dir-spec.txt#n773

I opened a ticket:

https://trac.torproject.org/projects/tor/ticket/24023#ticket

>> Sorry to have caused all these
>> problems :(
> 
> No, don't apologize! It's great that there are people implementing
> from our specs, and it's great when they find bugs with tools that
> made bad assumptions and expectations. :)

You should see the list of changes we made to tor-spec.txt when
implementing a simple OR protocol client in Python. About half of
the last 20 spec changes were triggered by that project:

https://gitweb.torproject.org/torspec.git/log/tor-spec.txt

And here's the project repository:

https://github.com/teor2345/endosome

That spec is much clearer now!

T___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] Testing Golang relay implementation

[Michael McLoughlin]

Missed a link from my last email: https://github.com/go-tor/gotor

On Thu, Oct 26, 2017 at 2:56 PM, Michael McLoughlin 
wrote:

> After another look at the spec, I still believe the descriptor I'm
> publishing conforms, as was my intention. Sorry to have caused all these
> problems :(
>
> Heads up that there's another (nascent) tor relay implementation in the
> works. I reached out to them to see if they were interested in
> collaborating, but I didn't get a response. It's unclear to me what their
> plans are. However Filippo Valsorda has a strong reputation so it's worth
> keeping an eye on.
>
> Mike
>
> On Thu, Oct 26, 2017 at 12:07 AM, Karsten Loesing 
> wrote:
>
>> On 2017-10-26 00:09, teor wrote:
>> > On 26 Oct 2017, at 06:58, Michael McLoughlin > > > wrote:
>> >> I can easily change the descriptor if necessary?
>> >
>> > As long as it conforms to the spec, it's fine.
>>
>> Agreed. FWIW, the descriptor published by this relay confused Metrics
>> quite a bit. But that's okay, we'll just make Metrics more robust. The
>> good news is that we didn't lose any data in the process.
>>
>> > We should really fuzz descriptor parsers better.
>> > But that's not an appropriate thing to do on the live network, and some
>> > parser code only runs on descriptors on the live network.
>>
>> If somebody wants to generate a bunch of fuzzed descriptors that conform
>> to the spec, I'll happily throw them into a local Metrics instance to
>> see if anything else breaks. I could imagine that Damian would do the
>> same with stem and Philipp with zoossh.
>>
>> All the best,
>> Karsten
>>
>>
>> ___
>> tor-relays mailing list
>> tor-relays@lists.torproject.org
>> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
>>
>>
>
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] Testing Golang relay implementation

[Roger Dingledine]

On Thu, Oct 26, 2017 at 02:56:03PM -0700, Michael McLoughlin wrote:
> After another look at the spec, I still believe the descriptor I'm
> publishing conforms, as was my intention. Sorry to have caused all these
> problems :(

No, don't apologize! It's great that there are people implementing
from our specs, and it's great when they find bugs with tools that
made bad assumptions and expectations. :)

--Roger

___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] Testing Golang relay implementation

[Michael McLoughlin]

After another look at the spec, I still believe the descriptor I'm
publishing conforms, as was my intention. Sorry to have caused all these
problems :(

Heads up that there's another (nascent) tor relay implementation in the
works. I reached out to them to see if they were interested in
collaborating, but I didn't get a response. It's unclear to me what their
plans are. However Filippo Valsorda has a strong reputation so it's worth
keeping an eye on.

Mike

On Thu, Oct 26, 2017 at 12:07 AM, Karsten Loesing 
wrote:

> On 2017-10-26 00:09, teor wrote:
> > On 26 Oct 2017, at 06:58, Michael McLoughlin  > > wrote:
> >> I can easily change the descriptor if necessary?
> >
> > As long as it conforms to the spec, it's fine.
>
> Agreed. FWIW, the descriptor published by this relay confused Metrics
> quite a bit. But that's okay, we'll just make Metrics more robust. The
> good news is that we didn't lose any data in the process.
>
> > We should really fuzz descriptor parsers better.
> > But that's not an appropriate thing to do on the live network, and some
> > parser code only runs on descriptors on the live network.
>
> If somebody wants to generate a bunch of fuzzed descriptors that conform
> to the spec, I'll happily throw them into a local Metrics instance to
> see if anything else breaks. I could imagine that Damian would do the
> same with stem and Philipp with zoossh.
>
> All the best,
> Karsten
>
>
> ___
> tor-relays mailing list
> tor-relays@lists.torproject.org
> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
>
>
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] Testing Golang relay implementation

[Damian Johnson]

> leekspin and maybe stem can produce valid, randomised descriptors.

Yup. Stem now supports descriptor creation (and should have all
capabilities Leekspin did)...

https://stem.torproject.org/tutorials/mirror_mirror_on_the_wall.html#can-i-create-descriptors
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] Testing Golang relay implementation

[teor]

On 26 Oct 2017, at 09:09, teor  wrote:

>> But it did occur to me that other code may assume the "platform" line takes 
>> a certain form.
> 
> It shouldn't, that's what protocol lines are for.
> 
> But any alternate implementation will never be used as a v3 HSDir, because it
> would need to claim to be Tor 0.3.0.8 or later for v3 onion services to use 
> it. This
> is a poor design decision on our part: just like consensus methods, when we 
> break
> a protocol version, we should allocate a new number, and check it.
> (Or we should exclude broken versions from the consensus.)
> 
> I've opened this ticket to fix that:
> https://trac.torproject.org/projects/tor/ticket/23998

For the record, I was wrong about this.

We ignore "Tor" implementations on 0.3.0.7 and below when checking
for the HSDir protover 2. (Which supports v3 onion services.)

So any non-Tor implementation that advertises support for protover 2
will be accepted as a v3 onion service HSDir.

T___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] Testing Golang relay implementation

[teor]

> On 26 Oct 2017, at 18:07, Karsten Loesing  wrote:
> 
> On 2017-10-26 00:09, teor wrote:
>> On 26 Oct 2017, at 06:58, Michael McLoughlin > > wrote:
>>> I can easily change the descriptor if necessary?
>> 
>> As long as it conforms to the spec, it's fine.
> 
> Agreed. FWIW, the descriptor published by this relay confused Metrics
> quite a bit. But that's okay, we'll just make Metrics more robust. The
> good news is that we didn't lose any data in the process.
> 
>> We should really fuzz descriptor parsers better.
>> But that's not an appropriate thing to do on the live network, and some
>> parser code only runs on descriptors on the live network.
> 
> If somebody wants to generate a bunch of fuzzed descriptors that conform
> to the spec, I'll happily throw them into a local Metrics instance to
> see if anything else breaks. I could imagine that Damian would do the
> same with stem and Philipp with zoossh.

leekspin and maybe stem can produce valid, randomised descriptors.

Tor also has a collection of valid and invalid directory documents in:

https://gitweb.torproject.org/fuzzing-corpora.git/tree/

I'm not sure if there are any others.

--
Tim / teor

PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B
ricochet:ekmygaiu4rzgsk6n




signature.asc
Description: Message signed with OpenPGP
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] Testing Golang relay implementation

[Karsten Loesing]

On 2017-10-26 00:09, teor wrote:
> On 26 Oct 2017, at 06:58, Michael McLoughlin  > wrote:
>> I can easily change the descriptor if necessary?
> 
> As long as it conforms to the spec, it's fine.

Agreed. FWIW, the descriptor published by this relay confused Metrics
quite a bit. But that's okay, we'll just make Metrics more robust. The
good news is that we didn't lose any data in the process.

> We should really fuzz descriptor parsers better.
> But that's not an appropriate thing to do on the live network, and some
> parser code only runs on descriptors on the live network.

If somebody wants to generate a bunch of fuzzed descriptors that conform
to the spec, I'll happily throw them into a local Metrics instance to
see if anything else breaks. I could imagine that Damian would do the
same with stem and Philipp with zoossh.

All the best,
Karsten



signature.asc
Description: OpenPGP digital signature
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] Testing Golang relay implementation

[teor]

> On 26 Oct 2017, at 06:58, Michael McLoughlin  wrote:
> 
> ...
> 
> In testing it took a little bit of finagling to get chutney to accept a 
> non-Tor "platform" item in the descriptor. Turned out adding a "proto" line 
> as well is enough.

I think this is a feature of the tor directory authority implementation, not 
chutney.
(Chutney doesn't do anything with platform or proto lines.)
Directory authorities will only synthesise a proto line for platforms that 
claim to be
Tor, because they can't be sure what features non-Tor platforms support.

> But it did occur to me that other code may assume the "platform" line takes a 
> certain form.

It shouldn't, that's what protocol lines are for.

But any alternate implementation will never be used as a v3 HSDir, because it
would need to claim to be Tor 0.3.0.8 or later for v3 onion services to use it. 
This
is a poor design decision on our part: just like consensus methods, when we 
break
a protocol version, we should allocate a new number, and check it.
(Or we should exclude broken versions from the consensus.)

I've opened this ticket to fix that:
https://trac.torproject.org/projects/tor/ticket/23998

> I can easily change the descriptor if necessary?

As long as it conforms to the spec, it's fine.

We should really fuzz descriptor parsers better.
But that's not an appropriate thing to do on the live network, and some
parser code only runs on descriptors on the live network.

T

___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] Testing Golang relay implementation

[Roger Dingledine]

On Tue, Oct 24, 2017 at 10:54:38AM -0700, Michael McLoughlin wrote:
> Yes I am very aware of Tom van der Woerdt's previous work, and I am
> attempting to avoid some of the problems he faced. This implementation is
> pure Go, so I will not have cgo-based issues at least.

Great! Yes, I think the memory bloat was the main issue that stopped Tom.

Also, you should take a look at Alex's talk about his erlang Tor relay
implementation experiences, including lessons learned and why he became
less enthusiastic about maintaining a separate Tor relay implementation:
https://media.ccc.de/v/SHA2017-304-introducing_talla_an_erlang_implementation_of_tor

--Roger

___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] Testing Golang relay implementation

[nusenu]

Michael McLoughlin:
> Is the bug in my descriptor or the parser?

First of all it was a problem for the parser and they hotfixed it.
(I did not verify if your descriptor follows the specification, if it
does you should be fine.)


>>> https://trac.torproject.org/projects/tor/ticket/23981


-- 
https://mastodon.social/@nusenu
twitter: @nusenu_



signature.asc
Description: OpenPGP digital signature
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] Testing Golang relay implementation

[Michael McLoughlin]

Is the bug in my descriptor or the parser?

In testing it took a little bit of finagling to get chutney to accept a
non-Tor "platform" item in the descriptor. Turned out adding a "proto" line
as well is enough. But it did occur to me that other code may assume the
"platform" line takes a certain form.

I can easily change the descriptor if necessary? I've included it below for
reference:

---

router pearl000 35.203.138.1 9001 0 0
signing-key
-BEGIN RSA PUBLIC KEY-
MIGJAoGBAKzTaN4tZGv1kiQWBzeuOk+ovr2LtIURlaVC38j6j/fQuYfuAZX/XvV1
fQr9EVh+T617dh+frt2D0QDuzLUvP3hpgVozW94w+Ib85pUCne03f4rj3QYu5Qtg
GvzShslZI6vgyy0g2jAOGa4jxT/UYAcKE5dQo8CBKA6Qb0P5Joc1AgMBAAE=
-END RSA PUBLIC KEY-
fingerprint 6832 5B4B 1E17 7374 B84D 372F 0304 6351 BEE7 FF6A
onion-key
-BEGIN RSA PUBLIC KEY-
MIGJAoGBANN/gLTe05kWKPSEyYJeknuxQst+cVsVmZrZgIYNXuhPn+3XnWhEc10r
ICa82FkB7hBH6REuW0ugGDc2QLwENmDiaBiFW1LDujEFeVlV8o0VSDwrL3VCPsPL
zC4/zHqR4DmLFXp5V238MKj85Pud04g65piZCIAsy6hiGMDCoGdtAgMBAAE=
-END RSA PUBLIC KEY-
ntor-onion-key ATSN2Q9KwCeRu35agh/ChjX8MsgM/FGFRDUX6o9Sbmk
platform Pearl 0fd5756 on Linux
contact pe...@m15n.org https://github.com/mmcloughlin/pearl
bandwidth 153600 307200 153600
published 2017-10-25 18:00:28
reject *:*
proto Link=4 LinkAuth=1 Relay=2
router-signature
-BEGIN SIGNATURE-
OyY0vQc5n2RYdkrXqfn09HoACJBx7GrBHZMnmNtlX5nJIL9N4eyyPvmxhmuC+A94
dDE0u/6w3nCABikFFLHcKaBAdmYBdxrzk3imfVjzYZazHWWr/se8HxK1jibP186A
8K8bdtMih127CGv3mn+g17uXFTbbuylM7r1xf8NpqRs=
-END SIGNATURE-



On Wed, Oct 25, 2017 at 12:34 PM, D.S. Ljungmark  wrote:

> So it's already finding bugs in other implementations?
>
> That's pretty awesome! ;-)
>
> //Spider
>
> On 25/10/17 21:16, nusenu wrote:
> > https://trac.torproject.org/projects/tor/ticket/23981#comment:9
> > wrote:
> >
> >> Looks like this issue is caused by a server descriptor produced by an
> >>  alternate Tor implementation that identifies itself as `"platform Pearl
> >>  0fd5756 on Linux"`. And that implementation uses a different order of
> >>  elements in its descriptor.
> >
> > :)
> >
> >
> >
> > ___
> > tor-relays mailing list
> > tor-relays@lists.torproject.org
> > https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
> >
>
>
> ___
> tor-relays mailing list
> tor-relays@lists.torproject.org
> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
>
>
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] Testing Golang relay implementation

[D.S. Ljungmark]

So it's already finding bugs in other implementations?

That's pretty awesome! ;-)

//Spider

On 25/10/17 21:16, nusenu wrote:
> https://trac.torproject.org/projects/tor/ticket/23981#comment:9
> wrote:
> 
>> Looks like this issue is caused by a server descriptor produced by an
>>  alternate Tor implementation that identifies itself as `"platform Pearl
>>  0fd5756 on Linux"`. And that implementation uses a different order of
>>  elements in its descriptor.
> 
> :)
> 
> 
> 
> ___
> tor-relays mailing list
> tor-relays@lists.torproject.org
> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
> 



signature.asc
Description: OpenPGP digital signature
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] Testing Golang relay implementation

[nusenu]

https://trac.torproject.org/projects/tor/ticket/23981#comment:9
wrote:

> Looks like this issue is caused by a server descriptor produced by an
>  alternate Tor implementation that identifies itself as `"platform Pearl
>  0fd5756 on Linux"`. And that implementation uses a different order of
>  elements in its descriptor.

:)

-- 
https://mastodon.social/@nusenu
twitter: @nusenu_



signature.asc
Description: OpenPGP digital signature
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] Testing Golang relay implementation

[Michael McLoughlin]

Thanks for the encouragement Nagaev :)

Yes I am very aware of Tom van der Woerdt's previous work, and I am
attempting to avoid some of the problems he faced. This implementation is
pure Go, so I will not have cgo-based issues at least.

As I said the project is still far from complete. It implements a minimal
subset of the protocol. But I'd like to learn by handling real traffic as
soon as I can.





On Tue, Oct 24, 2017 at 6:59 AM, Nagaev Boris  wrote:

> Hey
>
> Thanks for doing that! It came many times to my mind.
>
> There were efforts on implementing Tor relay in Golang before, by
> Tom van der Woerdt:
> https://github.com/tvdw/gotor
> Have you looked at the project?
>
> In his blog post
> https://tvdw.eu/blog/2015/01/24/implementing-a-tor-relay-from-scratch/
> Tom described some issues he faced with during this work and a list of
> lessons learnt.
>
>
>
> On Tue, Oct 24, 2017 at 5:18 AM, Michael McLoughlin
>  wrote:
> > All,
> >
> > I am working on a pure Golang relay implementation.
> >
> > https://github.com/mmcloughlin/pearl/
> >
> > I have thus far been testing locally with chutney
> > (https://gitweb.torproject.org/chutney.git). The project is not
> complete by
> > any stretch, but I believe I am close to the point where it can handle
> Tor
> > network traffic. I expect I can learn *a lot* by running against real
> > traffic rather than my sandboxed environment.
> >
> > I wanted to get in touch with the community before going live. Any advice
> > gratefully received.
> >
> > The implementation identifies itself clearly in the server descriptor,
> both
> > in the "platform" and "contact" items. If people notice any problems with
> > the relay specifically they will be able to contact me.
> >
> > Thanks,
> > Mike
> >
> > ___
> > tor-relays mailing list
> > tor-relays@lists.torproject.org
> > https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
> >
>
>
>
> --
> Best regards,
> Boris Nagaev
> ___
> tor-relays mailing list
> tor-relays@lists.torproject.org
> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
>
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] Testing Golang relay implementation

[Nagaev Boris]

Hey

Thanks for doing that! It came many times to my mind.

There were efforts on implementing Tor relay in Golang before, by
Tom van der Woerdt:
https://github.com/tvdw/gotor
Have you looked at the project?

In his blog post
https://tvdw.eu/blog/2015/01/24/implementing-a-tor-relay-from-scratch/
Tom described some issues he faced with during this work and a list of
lessons learnt.



On Tue, Oct 24, 2017 at 5:18 AM, Michael McLoughlin
 wrote:
> All,
>
> I am working on a pure Golang relay implementation.
>
> https://github.com/mmcloughlin/pearl/
>
> I have thus far been testing locally with chutney
> (https://gitweb.torproject.org/chutney.git). The project is not complete by
> any stretch, but I believe I am close to the point where it can handle Tor
> network traffic. I expect I can learn *a lot* by running against real
> traffic rather than my sandboxed environment.
>
> I wanted to get in touch with the community before going live. Any advice
> gratefully received.
>
> The implementation identifies itself clearly in the server descriptor, both
> in the "platform" and "contact" items. If people notice any problems with
> the relay specifically they will be able to contact me.
>
> Thanks,
> Mike
>
> ___
> tor-relays mailing list
> tor-relays@lists.torproject.org
> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
>



-- 
Best regards,
Boris Nagaev
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

[tor-relays] Testing Golang relay implementation

[Michael McLoughlin]

All,

I am working on a pure Golang relay implementation.

https://github.com/mmcloughlin/pearl/

I have thus far been testing locally with chutney (
https://gitweb.torproject.org/chutney.git). The project is not complete by
any stretch, but I believe I am close to the point where it can handle Tor
network traffic. I expect I can learn *a lot* by running against real
traffic rather than my sandboxed environment.

I wanted to get in touch with the community before going live. Any advice
gratefully received.

The implementation identifies itself clearly in the server descriptor, both
in the "platform" and "contact" items. If people notice any problems with
the relay specifically they will be able to contact me.

Thanks,
Mike
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays