magic-folder development roadmap?

[David Stainton]

Dear Brian Warner,

Would you have time to help us figure out the most suitable
development roadmap for the Tahoe-LAFS Magic-Folder project?

I believe that at this time we need your senior developer experience
to help us make some decisions. Without this kind of leadership we
will continue to be stuck in Zeno's paradox where we make progress but
there is always more to do.

Lately Daira, Meejah and I have been adding minor features and fixing
minor bugs... however I believe that these are not necessarily the
best things to work on right now.

When and if you have time to chat about this project, here are some of
the things we'd like to discuss with you:

We could stick with our current design which would only have conflict
detection for 2 parties. But then we'd have two additional design
flaws IMHO:
 - ambient authority among clients to perform file deletions
 - an "admin" role with excess authority

To fix our conflict detection we could implement Leif's or Zooko's proposal:
https://github.com/tahoe-lafs/tahoe-lafs/blob/2438.magic-folder-stable.8/docs/proposed/magic-folder/multi-party-conflict-detection.rst

Leif's design preserves history and Zooko's design allows for file
deletions. I could see a need for either design... though they have
different use cases. I would like us to implement Leif's design
because fits my use case. If we do this then we'd close these tickets:

https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2582
https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2565
https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2551

Leif's design preserves all file history and thus eliminates the
possibility of clients removing file objects forever. Additionally
Leif's design eliminates the "administrator" and it's excess authority
from our original design by implementing a subscription model and
resolving this ticket:
https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2589


Sincerely,

David
___
tahoe-dev mailing list
tahoe-dev@tahoe-lafs.org
https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

Re: Topics for Tahoe Hack Fest in Berlin?

[David Stainton]

feeling sick today with a cold and so i'm hoping to rest up and
recover tomorrow hopefully.

On Sat, Jan 2, 2016 at 9:06 AM, Daira Hopwood  wrote:
> Frederik Braun wrote:
>
>> > Gottschedstraße 4, [Aufgang] 4
>> > 13357 Berlin
>>
>> When, though? :-)
>
> Today and the next few days. The morning start time is ill-defined, and you
> may have difficulty getting into the building -- from 11:00ish I'll be
> monitoring #tahoe-lafs and we'll try to get you in.
>
>
> ___
> tahoe-dev mailing list
> tahoe-dev@tahoe-lafs.org
> https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev
>
___
tahoe-dev mailing list
tahoe-dev@tahoe-lafs.org
https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

Re: why Magic Folders doesn't work for more than 2 clients

[David Stainton]

yes we can make a git-like design... although we only have to preserve
the history of a single file... not the history of the whole directory
hierarchy.

the other property i think a new design might *need* is to preserve
the identity of the client that created the "snapshot object". the
only notion of identity we have in this case is the write capability
to the client's DMD. Zooko and I briefly chatted about this a few days
ago... and he pointed out that if the new design is able to preserve
the information about which DMD first linked the "snapshot object"
then that should be sufficient to serve as the client identity of the
snapshot.


On Fri, Nov 13, 2015 at 7:08 PM, Raoul Duke  wrote:
> hi,
>
> 1) if the documents always contain the version, there is no problem.
>
> 2) if the document do not contain the version, then the whole thing is
> just screwed from the get-go. you'd have to have a distributed db of
> document-hash->document-version or something.
>
> 4) i think it is not correct to say that versions are linear. they are
> a tree. so the belief that if we only had the version we could tell
> you if it was an edit or not doesn't help the *human* level of using
> things, even if it make "math" sense.
> ___
> tahoe-dev mailing list
> tahoe-dev@tahoe-lafs.org
> https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev
___
tahoe-dev mailing list
tahoe-dev@tahoe-lafs.org
https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

Re: why Magic Folders doesn't work for more than 2 clients

[David Stainton]

actually we didn't finish implementing the two party conflict
detection... but we *could* finish it... but why? i think we should
just solve the harder problem instead.

On Thu, Nov 12, 2015 at 2:34 PM, Freelab initiative
<freelab...@gmail.com> wrote:
> it is very great if magic folder are now working
>
> 2 clients on same file is fair enough
>
> we shall grab the last github to build to test this.
> or we shall take any other build/repo ?
>
>
> thanks
>
> xavier
>
>
> Le dimanche 8 novembre 2015, David Stainton <dstainton...@gmail.com> a écrit
> :
>>
>> Agreed; If possible we'd like to first hear Daira's solution before we
>> send Leif's.
>> I'm also curious to hear Zooko's solution.
>>
>> I've written a proposal document explaining Leif's solution.
>> Hopefully later today Leif will apply more corrections to this document
>> which is almost ready to be reviewed.
>>
>> On Fri, Nov 6, 2015 at 6:55 PM, Zooko Wilcox-O'Hearn <zoo...@gmail.com>
>> wrote:
>> > Dear folks:
>> >
>> > Context: “Magic Folders” is a new layer on top of Tahoe-LAFS which
>> > implements Dropbox-like "auto-sync" behavior. It's super exciting!
>> > Development of it was sponsored by Open Technology Fund. We've
>> > finished implementing it, but now we're noticing some bugs and
>> > limitations, and this is the biggest one I'm aware of right now.
>> >
>> > As David described in this comment:
>> >
>> > https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2551#comment:22
>> >
>> > The current design and implementation of Magic Folders only works if
>> > you have no more than two clients attached to it. By "clients" I mean
>> > here either two devices owned by the same user or two users each with
>> > their own device — Tahoe-LAFS and Magic Folders are unaware of any
>> > distinction between a separate human user with their own device versus
>> > a separate client/node/process/device operated by the same human user.
>> >
>> > The problem is subtle, and I'm writing this letter mostly in order to
>> > cement my own understanding of *why* the current design fails in the
>> > 3-party (or more) case. Also in order to draw Daira's attention to it
>> > and see what sort of fix she would suggest.
>> >
>> > Here is the section of the design doc about the problem of
>> > distinguishing conflicts from overwrites pushed by your peer. We
>> > whimsically named this problem the "Fire Dragons" problem.
>> >
>> >
>> > https://tahoe-lafs.org/trac/tahoe-lafs/browser/docs/proposed/magic-folder/remote-to-local-sync.rst#fire-dragons-distinguishing-conflicts-from-overwrites
>> >
>> > The goal of the Fire Dragon slaying protocol is to make it so that you
>> > can reliably tell whether a given new version submitted to you by your
>> > peer was derived from the most recent version that *you* created, or
>> > if it was derived from a previous version than the most recent version
>> > that you created. If your peer says that it was derived from the most
>> > recent version that you created, then this means your peer is
>> > instructing you to *overwrite* your most recent version with their new
>> > version. If instead your peer says that it was derived from an earlier
>> > version, then this means there is a *conflict*, where you and your
>> > peer simultaneously made changes to the file.
>> >
>> > The way the current Fire Dragon slaying protocol attempts to track
>> > this distinction is by including a slot with each upload called the
>> > "last-downloaded record". When you receive a new version from someone
>> > else, you inspect the last-downloaded record that accompanies that
>> > version, and if it shows that the version they started from was the
>> > most recent version you sent them, then you know it is an overwrite
>> > [*], and if it isn't, then you know it is a conflict.
>> >
>> > Now here's the problem: this design assumes that you are the only
>> > source of downloadable versions that they could have started with! But
>> > if there's a *third* party, and they started with a download from that
>> > third party, then with this design you will assume that they are
>> > starting from an *earlier version from you*, when in fact they are
>> > starting from a version from the third party, which might in turn have
>> > been derived from the most recent version from you. (Or it might not
>> >

Re: update on Tahoe/Tor/Foolscap integration

[David Stainton]

Dear Paul Rabahy,

Thanks for running the public grid so users can test out Tahoe-LAFS...

Btw both you and Zooko misunderstood Brian Warner's e-mail; he doesn't
want to take away the autodetect feature... he just wants it to not be
the default if no listening address is specified... keyword AUTO will
be used to specify autodetect... As Daira pointed out, this makes
"tahoe create-note" slightly more difficult to use.

On the other hand these changes will make Tahoe-LAFS way easier to use
with Tor and other transports (I2p, ipv6 etc) who wish to run storage
nodes at home and need the transport's NAT penetration property.

I think there are political, economic and technical consequences to
the NAT penetration method chosen... so I have this work in progress
document to help explain some of these tradeoffs:

https://github.com/telekommunisten/nat-penetration-tradeoffs

It's far from complete...


Cheers,

David
___
tahoe-dev mailing list
tahoe-dev@tahoe-lafs.org
https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

Re: update on Tahoe/Tor/Foolscap integration

[David Stainton]

First of all, we had several meetings, they take a long time, we talk
about a lot of stuff... it's work. Therefore it seems really unfair to
hear more quasi design discussion/complaining talk from people who
didn't attend the meeting; it disregards our efforts. Perhaps come to
the meetings? Offer solutions?

Secondly, I'm sure that the world's spying agencies would love for us
to deliberate writing this Tor integration even longer by having
confused cross talky design conversations so that we can't possibly
finish the integration before this upcoming Tor developer meeting:
https://trac.torproject.org/projects/tor/wiki/org/meetings/2015SummerDevMeeting

Are you aware when ticket #517 was opened by Jake Appelbaum?
https://tahoe-lafs.org/trac/tahoe-lafs/ticket/517
Opened at 2008-09-18T20:52:23Z so yeah let's please not deliberate
for several more years!

In general Tahoe-LAFS storage nodes cannot serve traffic from coffee
shops because of NAT. This Tor integration at it's core is an
acknowledgement not only that people need anonymity guarantees but
that the Internet is fundamentally broken because of NAT; it violates
the end-to-end principal. We therefore need protocols like Tor onion
services to penetrate NAT via rendezvous points... in other words,
Tahoe-LAFS needs a p2p transport; Tahoe-LAFS isn't a NAT-penetrating
p2p protocol itself. Yawning Angel of the Tor project has said
publicly many times that these NAT devices have very unreliable
forwarding protocols like NAT-PMP:

https://lists.torproject.org/pipermail/tor-dev/2015-July/009155.html

We think that overcoming NAT by way of a rendezvous transport is going
to be way more reliable and widely useable than trying to make use of
things like NAT-PMP. Honestly, I'd rather see the internet just unfuck
itself by way of IPv6 (or whatever next general IP)... but that was
supposed to happen years ago... we aren't holding our breath and we
aren't pretending NAT isn't a problem.

> I don't like this idea for 1 main reason -> My IP address changes. I know

Your IP address of what changes? Of your laptop? Are you trying to
serve files from your laptop? If not then does it matter if your IP
address changes?

> that some people use/see Tahoe-lafs as a client/server protocol, but I see
> it as a p2p protocol. If I need to re-setup my node every time I take my

Not a matter of opinion here... see above; the Internet is broken. We
can write lots of p2p software that doesn't actually work if we don't
think about how to fix the end-to-end principal violations.

> laptop to a coffee shop, that would be very inconvenient. The thing that
> feels the most like a server to me is the introducer, but since I have been
> controlling the public test grid I know that the ip address has changed at
> least 4 times and I lost control of the DNS name at least once.

What do you mean by "feels like a server"?
You lost control of the introducer DNS name?
I'm not sure what point you are trying to make here.

> Final thought, in a p2p environment, auto detecting IP addresses locally
> doesn't work very well anyway because of NAT. As long as their is an
> introducer, I believe that the most accurate way to get your own IP address
> is to ask the introducer what it is.

Um these statements are a bit confusing. Storage servers announce
themselves to the introducer! and if a storage server is behind a NAT
device then it announces an RFC1918 address. Do you want it to remain
broken?

> (Don't get me wrong, I like the idea of supporting TOR and making TOR
> configuration easier.)

nit: Tor not TOR.

> On Mon, Aug 31, 2015 at 8:30 AM, Lukas Pirl  wrote:
>>
>> On 09/01/2015 12:22 AM, Daira Hopwood wrote:
>>>
>>> I am concerned about this being a significant usability regression.
>>
>>
>> I thought so too
>> (but didn't feel I am into that enough to note that).
>>
>> ___
>> tahoe-dev mailing list
>> tahoe-dev@tahoe-lafs.org
>> https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev
>
>
>
> ___
> tahoe-dev mailing list
> tahoe-dev@tahoe-lafs.org
> https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev
>
___
tahoe-dev mailing list
tahoe-dev@tahoe-lafs.org
https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

Re: removing IP-address autodetection, Tor integration

[David Stainton]

Hi Brian Warner and everyone else too,

There's some really good examples of how to use the txtorcon's Tor
hidden service endpoints... that also demonstrate how to properly
determine the generated .onion address from the fired IListeningPort:
https://github.com/meejah/txtorcon/blob/master/examples/launch_tor_endpoint.py
https://github.com/meejah/txtorcon/blob/master/examples/launch_tor_endpoint2.py

whereas these two examples don't use the endpoint abstraction and
instead use TorConfig to configure the tor process with a onion
service:
https://github.com/meejah/txtorcon/blob/master/examples/launch_tor_with_hiddenservice.py
https://github.com/meejah/txtorcon/blob/master/examples/launch_tor_with_simplehttpd.py

I don't think there is any reason for us to not use the client +
server endpoint objects... Even if there are edge cases where the
existing endpoint parser or endpoint constructors are not sufficiently
flexible we should opt to add another class method to the endpoint
object instead of using TorConfig directly.

Meejah has written some excellent documentation here:
http://timaq4ygg2iegci7.onion/txtorcon-protocol.html#txtorcon.TCPHiddenServiceEndpoint

There are three main ways to use this class, and you are encouraged
to use the @classmethod ways of creating instances: system_tor,
global_tor, and private_tor


Cheers,

~david

On Tue, Jul 28, 2015 at 9:23 PM, Daira Hopwood da...@jacaranda.org wrote:
 On 23/07/15 16:54, Chris Kerr wrote:
 Also, perhaps changing the keyword to 'AUTODETECT' rather than 'AUTO' might
 make the configuration easier to understand.

 -1; people are unlikely to find out about AUTO except by looking at the
 docs or by following someone else's advice, and in both those cases it would
 be clear that it refers to auto-detection. Changing it now would prevent 
 writing
 a tahoe.cfg, or instructions for how to write/change one, that would work 
 under
 both new and old versions.

 --
 Daira Hopwood ⚥


 ___
 tahoe-dev mailing list
 tahoe-dev@tahoe-lafs.org
 https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

___
tahoe-dev mailing list
tahoe-dev@tahoe-lafs.org
https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

Re: Rsync with sftp account

[David Stainton]

The inefficiency has to do with the FUSE frontend to Tahoe-LAFS. Just
don't use it... Tahoe-LAFS is not a POSIX compliant filesystem and
cannot efficiently perform random writes.

I suggest using the tahoe backup command to backup files to a tahoe grid.


On Thu, Jul 30, 2015 at 5:10 AM, Freelab initiative
freelab...@gmail.com wrote:


 to my humble observations,

 trying to make a rsync over a sftp frontend to tahoe grid is not working
 very efficiently

 rsync is asking to read so much the tahoe grid
 and tahoe lafs is not a fast file system

 so the rsync at the end is very slow


 maybe there is another way to achieve what you want to do

 2015-07-27 21:46 GMT+02:00 David Stainton dstainton...@gmail.com:

 Why is a question about rsync sent to a list about Tahoe-LAFS?
 Seriously, I fail to see the connection between Tahoe-LAFS and
 rsync... except for the file paradigm... they both deal with file like
 objects... and that's cool and all but what exactly do you want?

 Do you want to use Tahoe-LAFS with rsync? This isn't trivially
 possible. Tahoe-LAFS does not in fact implement a filesystem...
 instead think of it as a key value data store.


 On Mon, Jul 27, 2015 at 12:06 PM, InfoTest infotest...@gmail.com wrote:
  Is it true that rsync doesn't know difference between two directory?
  ___
  tahoe-dev mailing list
  tahoe-dev@tahoe-lafs.org
  https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev
 ___
 tahoe-dev mailing list
 tahoe-dev@tahoe-lafs.org
 https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev


___
tahoe-dev mailing list
tahoe-dev@tahoe-lafs.org
https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

Re: Rsync with sftp account

[David Stainton]

Why is a question about rsync sent to a list about Tahoe-LAFS?
Seriously, I fail to see the connection between Tahoe-LAFS and
rsync... except for the file paradigm... they both deal with file like
objects... and that's cool and all but what exactly do you want?

Do you want to use Tahoe-LAFS with rsync? This isn't trivially
possible. Tahoe-LAFS does not in fact implement a filesystem...
instead think of it as a key value data store.


On Mon, Jul 27, 2015 at 12:06 PM, InfoTest infotest...@gmail.com wrote:
 Is it true that rsync doesn't know difference between two directory?
 ___
 tahoe-dev mailing list
 tahoe-dev@tahoe-lafs.org
 https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev
___
tahoe-dev mailing list
tahoe-dev@tahoe-lafs.org
https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

Re: Need to restore zeroed file

[David Stainton]

Dear Roland,

Don't have the caps? Sorry for your lose. Continue to insist it's
possible to recover? Trolling.

Furthermore... you really must have a low opinion of our security
model if you think you can so easily recover a file without it's
capability. ;-p


Cheers,

David


On Fri, Jul 10, 2015 at 7:29 AM, Roland Haeder rol...@mxchange.org wrote:
 Opps, 16 Bytes, of course.


 ___
 tahoe-dev mailing list
 tahoe-dev@tahoe-lafs.org
 https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

___
tahoe-dev mailing list
tahoe-dev@tahoe-lafs.org
https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

Re: Need to restore zeroed file

[David Stainton]

NO. You cannot try all possibilities. It would defeat the point of
using cryptographic capabilities.
Your data is lost forever unless you possess the cryptographic capability.
I'm sorry for your loss but if you continue to insist you can recover
it then that's like saying you don't believe in cryptography and
mathematics... and time and physics and the universe. People will
think you are trolling and little children will laugh. So don't make
that mistake because just because. oh dear am i ranting? sorry folks.
ok bye!

On Thu, Jul 9, 2015 at 3:10 PM, Roland Haeder rol...@mxchange.org wrote:
 On 07/09/2015 11:33 PM, Daira Hopwood wrote:
 This is unclear; do you mean that a file containing the URI got zeroed?
 I have saved a file through sshfs and all I got was a zero-length file
 in my mount point.


 num is actually the length of the file.
 Ah, okay.


 You won't be able to brute-force either key1 or key2. Unless you have a 
 copy of
 the URI somewhere else, the file contents are lost.

 Well, I *could* try all possible URIs out, but as Ed said, our sun will
 die before I have brute-forced key1*key2*file size ... So I have tried
 another approach, please see my other email.

 And thank you for replying. :)

 Regards,
 Roland


 ___
 tahoe-dev mailing list
 tahoe-dev@tahoe-lafs.org
 https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

___
tahoe-dev mailing list
tahoe-dev@tahoe-lafs.org
https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

Re: Need to restore zeroed file

[David Stainton]

Dear Roland,

I'm sorry to hear about your data loss. It is my understanding that if
you do not possess a cryptographic capability for either ReadOnly or
ReadWrite access to a file, you will NOT be able to recover it. That
is... not with some sort of cryptographic attack.

You might check if any of your Tahoe-LAFS aliases link to the file or
directory that you are trying to recover. In other words, you can
either choose to remember a capability inside or outside of tahoe.



On Thu, Jul 9, 2015 at 2:22 PM, Ed Kapitein e...@kapitein.org wrote:
 On 07/09/15 21:00, Roland Haeder wrote:

 Hello all,

 bad news: One of my files in my storage got zeroed and I don't have the
 old (working) URI or up-to-date backup (only from 2013).

 I have currently written a brute-forcer which currently *should* test
 all possibilities. Sure this needs massive improvement.

 So I need to know something:

 1) Is this URL pattern valid:


 http://127.0.0.1:3456/file/URI%3ACHK%3Akey1%3Akey2%3A3%3A10%3Anum/@@named=/filename

 by:
 key1 = 26 Byte long character consisting of a BASE32 checksum
 key2 = 52 Byte long character consisting of a BASE32 checksum
 num  = Random number from 0 to 9
 filename = The file's name without path

 2) Is z_base_32_alphabet or rfc3548_alphabet taken as an alphabet?

 3) Do I calculate this correctly?

 26 length, each can have 32 different characters (key1)

 This means 1*32^26 =
 1361129467683753853853498429727072845824

 possibilities?

 And

 1852673427797059126777135760139006525652319754650249024631321344126610074238976

 possibilities for key2?

 3) Do I *really* have to brute-force them all to find the correct URI?
 Or can someone (I don't like to paste brute-forcer publicly) improve the
 search algorithm?

 Regards,
 Roland

 Well, without doing the actual math, i think our sun has died before you can
 bruteforce key1*key2*9.
 Do you have the storage index for the lost file? that might give you a
 fighting change to get the content back.

 Kind regards,
 Ed

 ___
 tahoe-dev mailing list
 tahoe-dev@tahoe-lafs.org
 https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev
___
tahoe-dev mailing list
tahoe-dev@tahoe-lafs.org
https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

Re: removing IP-address autodetection, Tor integration

[David Stainton]

Just for the record, it is *very* difficult to get a Tahoe-LAFS client
running on Debian 8 to connect to an onion grid. A Tahoe-LAFS code
contributor recently reports having to modify the source of torsocks
so that it works with Tahoe... !!

I'm going to CCC camp... however it doesn't make much sense to have a
Tahoe-LAFS Onion grid workshop if it's too damn hard for most people
to setup. Even as a Qubes user I feel the pain. I had to create a
Qubes VM using the new Debian 7 AppVM template... and then use apt-get
to install tahoe-lafs and the old torsocks.

From my perspective a large portion of the motivation to create some
form of native Tor+Tahoe-LAFS integration is to make it very easy for
Tor users to use Tahoe-LAFS. The results of our efforts could aid
journalists, activists, document leaking, data exfiltration across
borders... etc.

I gotta snicker when I think; capability model security nerds are all
anarchists and should therefore make their software more accessible to
other anarchist software developers... in this case the core Tor
developers who won't be trying out Tahoe-LAFS until it has native Tor
integration. ;-p

After that's all working... then it might be interesting to discuss
making Tahoe-LAFS more denial-of-service resistant.


On Mon, Jun 29, 2015 at 5:43 PM, David Stainton dstainton...@gmail.com wrote:
 I think per-server connection preferences should be exposed via the
 introducerless mode which you (Brian) mostly implemented long ago but left
 commented out and which David made work in the truckee branch[1]. Speaking of
 which, I really need to bring that up to date with the last 6 months or so of
 Tahoe development... I'll try to work on that in the near future.

 Yes that's right! Brian has already implemented this feature which I
 also would like to use. We can easily change it to be an override
 table so that none of the storage server options are required... and
 the ones specified act as overrides to the introducer acquired storage
 server info.
___
tahoe-dev mailing list
tahoe-dev@tahoe-lafs.org
https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

Re: Fwd: removing IP-address autodetection, Tor integration

[David Stainton]

Dear Meejah,

I appreciate all the work you've put into the txtorcon project. I am
very excited about this Tahoe-LAFS integration.

 Cool, tahoe via Tor :)

free NAT penetration via Tor onions means Tahoe-LAFS will be *much*
more useable... and we can possibly change our thinking of all
Tahoe-LAFS participants as both client and server.

 If txtorcon is missing features needed for this, let me know what you
 need.

hmm yes I'd to take a detailed look and see what this Tor integration
project needs.

 Also, I just merged David's tor: endpoint stuff, which makes txtorcon
 depend on txsocksx now...

yay!

 Anyway aside from epemeral HS... I was thinking that ideally the
 Tahoe-LAFS user would select whether to launch a new tor process or to
 use an existing tor process. Either the location of tor would be known
 or guessed... or the socks port and control port must be known... and
 we'd also want to provide the location of the tor control port cookie
 auth file.

 Strictly speaking, just providing the control-endpoint should be
 sufficient; the client is told where the cookie is during
 authentication and the socks port (ports?) can be queried from
 Tor. e.g. I could add factory methods such as
 TorClientEndpoint.from_connection() to which you'd pass a
 control-protocol connection and get back a TorClientEndpoint ready to
 use it.

Oh yeah that makes sense.

I think Tahoe-LAFS default Tor quickstart settings would include
configuration parameters specifying to use a txtorcon launched tor
process. More advanced users may specify tahoe.cfg parameters to
utilize the tor control port of an existing tor process. Furthermore
we'll set it up to save key material to the client's private directory
by default... but it will also be possible to instead use ephemeral
hidden services if you really want to do so.

The end goal is that it will be super easy to setup Tahoe-LAFS + Tor
on any olde computer-like device behind a NAT and just make it work
right right now with that onion grid introducer FURL your cryptoparty
friend gave you. hmm this makes me think that it might be useful to
have a PAKE based service to help pass around introducer FURLS... and
for other short bits of secret information in general. If only we had
a magical-pond-wormhole type thing to help us ;-p

If the Tahoe-LAFS storage operator uses an existing tor process via
the control port then does the txtorcon remove existing Tor hidden
services when it adds it's own new hidden service? The old control
port API is terrible and hard to use... does txtorcon use the new
ephemeral hidden service control port API? I believe there was some
discussion on the tor-dev mailing list about that a while back and
someone mentioned sending key material through the control port for a
given onion service. I'm just wondering what sorts of options we have
for Tahoe-LAFS + existing tor process. I'd want to make sure that our
release docs make well known the the various deployment issues and
feature caveats.

 [..]
 using our own plugin system... we can instantiate endpoint objects
 however we want; with or without the help of the endpoint parser
 plugin system.

 Perfect, I *believe* the existing hooks (factory methods) for the server
 endpoints will let you choose between a system or launched Tor easily
 (and change its configuration as necessary).

yes I was looking at the API. Looks like we want to use `system_tor`
and `global_tor depending` on what the user specifies in their
tahoe.cfg. I'm thankful for the clear API documentation... because
even though I helped write this API it's been months since I've looked
at it in detail.

https://github.com/meejah/txtorcon/blob/master/txtorcon/endpoints.py#L124-L136


 The client-side of this doesn't exist at all, but I *would* like to add
 similar features as the server-side...

oh yes I see what you mean... and I agree that factory (class) methods
for the client side would be helpful.
___
tahoe-dev mailing list
tahoe-dev@tahoe-lafs.org
https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

Re: removing IP-address autodetection, Tor integration

[David Stainton]

Yes the direct onion services are a very good feature for the Tahoe
+ Tor combo... especially with regards to how most people would like
to use tools like Tahoe-LAFS to transfer largish amounts of data.
Furthermore if/when we release Tahoe+Tor integration I'd like to write
a short little release note about not abusing the Tor network's
resources.

And on a related topic: I hope Tor Project gets the funding they need
to fix the Tor hidden services design to provide stronger anonymity
guarantees.


On Sat, Jun 20, 2015 at 11:12 PM, meejah mee...@meejah.ca wrote:
 Brian Warner war...@lothar.com writes:

 But is that.. useful? Safe? You aren't hiding the server's address.. I
 guess you're making life easier for clients who want to come in via Tor
 (we could make them prefer the onion address, and avoid exit nodes), but
 it'd be slower than the usual tor-to-the-public-IP exit-node style. Who
 would it protect?

 .onion addresses are self-authenticating if this helps?
 Also, Tor has been talking about direct onion services (i.e. non-
 hidden hidden service) and possibly adding support for single-hop
 circuits on the service-side (since the server isn't trying to
 hide). So, 2 fewer hops.

 See: https://trac.torproject.org/projects/tor/ticket/2555
 and: https://lists.torproject.org/pipermail/tor-dev/2015-April/008625.html

 --
 meejah
 ___
 tahoe-dev mailing list
 tahoe-dev@tahoe-lafs.org
 https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev
___
tahoe-dev mailing list
tahoe-dev@tahoe-lafs.org
https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

Re: Fwd: removing IP-address autodetection, Tor integration

[David Stainton]

yay! i'm excited for this native Tor integration project.

The default Tahoe-LAFS+Txtorcon behavior will persist hidden service
key material to a private client config directory... however I'm sure
that ephemeral storage nodes would easily be possible as well;

I envision ephemeral Tahoe-LAFS onion service nodes would be
configured as such via a combination of a tahoe.cfg setting specifying
a ephemeral=True listening endpoint paramater as well as SystemD unit
files that caused the periodic restart of the Tor and Tahoe-LAFS
processes.

Anyway aside from epemeral HS... I was thinking that ideally the
Tahoe-LAFS user would select whether to launch a new tor process or to
use an existing tor process. Either the location of tor would be known
or guessed... or the socks port and control port must be known... and
we'd also want to provide the location of the tor control port cookie
auth file.

Qubes users for instance might want their laptop's instance of
Tahoe-LAFS to run in a separate vm from the vm running tor. Other
users might not care as much and prefer the convenience of
auto-launching tor.

Further I think by default weather the user is using the auto-launched
tor process or a preconfigured tor process client Tor connections
and onion service connections should utilize the same tor process.

We'll be using our own transport plugin system for Foolscap... and
each of these transport plugins will utilize client and server
endpoints. However, we won't be limited by the endpoint options
exposed to the Twisted endpoint parser plugin system... because of
using our own plugin system... we can instantiate endpoint objects
however we want; with or without the help of the endpoint parser
plugin system.
___
tahoe-dev mailing list
tahoe-dev@tahoe-lafs.org
https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

Re: removing IP-address autodetection, Tor integration

[David Stainton]

What would happen if the foolscap transport plugin state directory was
removed but the tahoe.cfg config file remained intact?

In that error case when the Tor-Foolscap plugin is used, the correct
behavior would be to exit with an error telling the user that the
Tahoe-LAFS configuration file expresses the desire to advertise a
particular onion address but the key material to do so is not present.
Too bad!

Should we have some sort of reset-advertised-endpoint command line tool?
It would make it possible to recompute the advertised name based on
new input for a dynamic server endpoint (e.g. tcp:0 or onion:80); and
it writes the resulting new client side endpoint descriptor string to
the tub.location section of the tahoe.cfg.

I'd like to use some of my fun-Fridays to help clean up the truckee
branch so that we can get it merged into trunk.

I do like the idea of a heterogeneous grid even though it will not
have any anonymity for the storage node operators. Users could select
their desired transport for upload or downloading. Furthermore this
multi-introducer feature makes it seem like: grids are sets and using
multiple introducers gets the union of the sets. But then anonymity
policy filters this union...

With the introducerless feature it should be possible for users to
use a variety of grids... and perhaps even a variety of transport
protocols with which to talk to those grids. However from their Tahoe
client gateway's perspective it is talking to a single grid. Would
this even be a desired usage?

Perhaps if there were several ipv4+onion heterogeneous grids... some
of these grid members might be friends with members of another grid.
They could exchange introducer FURLS and use the multi-introducer
feature to learn about all the other grid nodes. After that they each
have the option to use the no-introducer feature to write their
static list of storage servers... servers from the union of 2 or more
grids. I would think that after combining several grids into one very
large grid that you'd want a non-Reed-Solomon erasure encoding scheme
more suited for a different set of trade-offs.

I'm excited at the prospect of Tahoe-LAFS gaining both a flexible API
for storage backend AND network transports; it really seems like we
are giving our users and potential developers to tools to surprise us
with their own creative solutions to problems.


David



On Thu, Jun 18, 2015 at 1:02 PM, Leif Ryge l...@synthesize.us wrote:
 On Thu, Jun 18, 2015 at 12:31:16PM -0700, Brian Warner wrote:
 [snip]

 This all sounds great to me! But there are a few edge cases which shouldn't be
 forgotten:

  * It could be desirable to connect to a grid (possibly of non-onion storage
servers) using Tor to reach all of the servers *except* the user's own
servers, which are reachable via their LAN or VPN.

  * It could be desirable to have a server listen on both an onion address and 
 a
LAN address.

  * It could be desirable to connect to some servers via different addresses
than they are advertising (say, because you know its LAN address).

 OK so maybe these are all variations on the same use case, which happens to be
 how I want to use Tahoe :)

 I think per-server connection preferences should be exposed via the
 introducerless mode which you (Brian) mostly implemented long ago but left
 commented out and which David made work in the truckee branch[1]. Speaking of
 which, I really need to bring that up to date with the last 6 months or so of
 Tahoe development... I'll try to work on that in the near future.

 I'm looking forward to being able to use the i2p grid (which I believe is the
 largest and longest running public tahoe grid) and the onion grid
 simultaneously!

 ~leif

 [1]: https://github.com/leif/tahoe-lafs/blob/truckee/NEWS.rst
 ___
 tahoe-dev mailing list
 tahoe-dev@tahoe-lafs.org
 https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev
___
tahoe-dev mailing list
tahoe-dev@tahoe-lafs.org
https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

Re: Tahoe-LAFS featured in “Why the command-line is not usable”

[David Stainton]

I think this just means that for the people who do not know or like
command line tools, they will use the easier to use interface for
Tahoe-LAFS (which doesn't exist yet).

On Wed, Apr 8, 2015 at 5:14 PM, John jsch...@flowtools.net wrote:
 Long time lurker on this project and I'm hoping it succeeds.

 Hmm, sorry, how can anyone claim to be an expert reviewer if you don't
 know how to do command line?

 GUI isn't the answer for everything. Just because you don't know how to do
 command line functions doesn't make command line bad.

 --John

 On 04/08/2015 10:40 AM, Tony Arcieri wrote:

 That looks like a pretty old version of Tahoe. It doesn't even have the
 redesigned status page I did.

 On Wed, Apr 8, 2015 at 9:32 AM, Zooko Wilcox-OHearn
 zo...@leastauthority.com wrote:

 Folks:

 You might be interested in this video in which Gus The Usability
 Hacker added annotations on top of Taylor's instructional video about
 how to use Tahoe-LAFS:

 http://gandre.ws/blog/blog/2015/04/07/why-the-command-line-is-not-usable/

 Please don't angrily lash out at Gus about this! We asked her to help
 by telling us her opinion of Tahoe's usability. I hope that we'll get
 into a virtuous feedback loop with her in which we make more changes
 to Tahoe (hopefully for the better!) and she gives us more ideas about
 usability design issues. You folks being polite and encouraging to her
 would probably help.

 The bit about Lovecraftian horror is pretty funny.

 Regards,

 Zooko Wilcox-O'Hearn

 Founder, CEO, and Customer Support Rep
 https://LeastAuthority.com — Freedom matters.
 ___
 tahoe-dev mailing list
 tahoe-dev@tahoe-lafs.org
 https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev




 --
 Tony Arcieri


 ___
 tahoe-dev mailing list
 tahoe-dev@tahoe-lafs.org
 https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev



 ___
 tahoe-dev mailing list
 tahoe-dev@tahoe-lafs.org
 https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

___
tahoe-dev mailing list
tahoe-dev@tahoe-lafs.org
https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

Re: GSoC 2015 to overcome NAT limitations?

[David Stainton]

 Perhaps I did not express myself very well. I want to store several TB on
 Tahoe-LAFS nodes behind NATs. This is not a super-expensive high-tech exascale
 project; just a few Raspberry Pis with external hard drives, distributed
 around my friends' houses, total cost well under US$1000.
 Your onion grid may well be small, but if I try to use Tor as a NAT-piercing
 technology then it means using on the order of 100 TB of bandwidth on the Tor
 network (connections to a .onion server go through six intermediate nodes;
 multiply that by the redundancy setting and add a bit more to account for
 downloading the data again and you have a factor of 20+ times the data being
 stored). I think using 100 TB of volunteers' bandwidth counts as leeching.

Agreed.
___
tahoe-dev mailing list
tahoe-dev@tahoe-lafs.org
https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

Re: GSoC 2015 to overcome NAT limitations?

[David Stainton]

Your comment about leeching bandwidth is judgmental and incorrect. I
store very small files on our Tahoe-LAFS onion grid. I recently was at
the Tor developer meeting in Valencia, Spain and we (me and Leif Ryge)
explained to some of the core Tor developers how this Tahoe-LAFS onion
grid works... and none of them raised this concern.

Furthermore, if my Foolscap Twisted endpoints patch is merge then
we'll be able to host heterogeneous Tahoe-LAFS grid; that is a storage
grid accessible by more than one transport protocol.

You might have also pointed out (but maybe you forgot?) that
Tahoe-LAFS uploads duplicate shares... therefore to not leech Tor
network bandwidth one could upload via TCP and download via Tor... if
using a heterogeneous grid and this behavior was approved by your
threat model.


Sincerely,

David

On Fri, Feb 27, 2015 at 7:06 AM, Chris Kerr chris.k...@mykolab.ch wrote:
 Using Tahoe-LAFS over Tor will drastically reduce data transfer speeds
 versus a direct connection, not to mention leeching bandwidth from other Tor
 users.



 For me, usability from behind a NAT is an essential feature. I have had
 moderate success using PageKite but things would be much easier if there was
 only one package to install and configure. If this is not something that
 Tahoe-LAFS can do itself then integrating with external tools would also
 help. Also, if NAT piercing turns out to be too difficult, connection
 forwarding super-helper-nodes would also help, albeit with increased
 bandwidth costs.



 On Wednesday 25 Feb 2015 15:51:14 David Stainton wrote:

 Hi Lukas,



 I'm very interested to see what the tahoe-devs say to this idea of

 participating in GSoC.



 With regards to NAT... it seems that they decided a while back that

 this problem was outside the scope of what Tahoe-LAFS should solve...

 but maybe they are interested? I can tell you that there are various

 people such as myself who use Tahoe-LAFS with Tor and Tor hidden

 services... we therefore get NAT penetration for free. I don't know

 much about I2p but I think they may have NAT penetration as well.



 Cheers,



 David



 On Wed, Feb 25, 2015 at 4:20 PM, Lukas Pirl tahoe-...@lukas-pirl.de
 wrote:

  Hello Tahoe-People!

 

  What if I'd be interested in contributing to Tahoe-LAFS in the context

  of this year's Google Summer of Code¹?

 

  What is interesting for me is the firewall/NAT/hole punching topic. From

  what I recognize, running Tahoe behind some NAT is relatively popular

  (e.g. at home). From my own experiences, I can say that the peer-to-peer

  idea is somewhat destroyed if one runs most of the nodes behind a NAT.

 

  Well, there are five days left for you as an organization to apply for

  this year's Summer of Code. Tahoe can definitely profit!

 

  Further, I have a (early) question about the mentoring. Do you want the

  mentor to be a Tahoe core developer (what definitely would make perfect

  sense) or am I supposed/can I ask one of my folks in preference? And if

  so, is it sufficient for mentors to be wizards in Python and to have

  joined the GSoC already or should they have made contributions to Tahoe

  already?

 

  I'd be happy to hear from you!

 

  Kind regards,

 

  Lukas

 

  ¹ https://www.google-melange.com/gsoc/homepage/google/gsoc2015

  ___

  tahoe-dev mailing list

  tahoe-dev@tahoe-lafs.org

  https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev



 ___

 tahoe-dev mailing list

 tahoe-dev@tahoe-lafs.org

 https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev


 ___
 tahoe-dev mailing list
 tahoe-dev@tahoe-lafs.org
 https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

___
tahoe-dev mailing list
tahoe-dev@tahoe-lafs.org
https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

Re: GSoC 2015 to overcome NAT limitations?

[David Stainton]

Yes... me too. Unfortunately large patches like mine probably seem
like a daunting code review task that nobody has had time for.

On Wed, Feb 25, 2015 at 5:13 PM, Lukas Pirl tahoe-...@lukas-pirl.de wrote:
 On 02/25/2015 05:28 PM, David Stainton wrote:
 I don't know much about IPv6 but the core Tahoe-LAFS dev team can
 decide to be compatible with both IPv6 and IPv4 anytime they want to
 review and merge my foolscap Twisted endpoints branch which would
 allow the use arbitrary network transports (transports that have
 Twisted client and server endpoints).

 Yes, I saw that there are patches to support IPv6 (and others
 transports, respectively) and I hope that those get merged soon.

 lp
___
tahoe-dev mailing list
tahoe-dev@tahoe-lafs.org
https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

Re: NAT traversal

[David Stainton]

Tor hidden services get you NAT whole punching for free... so just use
Tor hidden services!!! =-|

On Sun, Dec 21, 2014 at 11:13 AM, Lukas Pirl tahoe-...@lukas-pirl.de wrote:
 On 12/20/2014 08:15 PM, Shu Lin wrote:
 if Tahoe team has any intention to put effort in this area

 Well, I am not the team but I'd appreciate UPnP/STUN/… very much!
 Setting up a grid with nodes mostly behind NATs actually lowers the
 initial pleasure when starting with Tahoe…

 lp
 ___
 tahoe-dev mailing list
 tahoe-dev@tahoe-lafs.org
 https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev
___
tahoe-dev mailing list
tahoe-dev@tahoe-lafs.org
https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

Re: Tahoe on AWS

[David Stainton]

Hi Sameer,

You need to set (in the node section) the tub.location configuration
parameter.
Here's the details:
https://tahoe-lafs.org/trac/tahoe-lafs/browser/docs/configuration.rst

And... if you are in San Francisco then come to by Tahoe-LAFS
cryptoparty tomorrow night at noisebridge!
https://www.cryptoparty.in/sanfrancisco

Cheers,

David Stainton


On Fri, Dec 19, 2014 at 3:21 AM, Sameer Verma sve...@sfsu.edu wrote:
 Hi,

 I'm trying to set up a system of 1 introducer and 10 storage nodes on
 AWS, plus one client on my laptop. The introducer sets up, but when I
 run it, the introducer.furl always reflects the internal IP that AWS
 assigns and not the public IP. Is there a way around this?

 Also, apart from port 3456 TCP, what ports do I need to open on the
 introducer and the storage nodes?

 cheers,
 Sameer
 ___
 tahoe-dev mailing list
 tahoe-dev@tahoe-lafs.org
 https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev
___
tahoe-dev mailing list
tahoe-dev@tahoe-lafs.org
https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

Re: IMAP back end

[David Stainton]

Dear Lukas,

I dare say this does not belong in the Tahoe-LAFS trac ticket system.
You can write all sorts of applications that use Tahoe-LAFS... good
luck with your software project!


David

On Sat, Nov 1, 2014 at 12:12 PM, Lukas Pirl tahoe-...@lukas-pirl.de wrote:
 Hi everyone!

 I just tried to create a ticket that proposes the enhancement of
 implementing an IMAP back end to enable user to make use of proven,
 existing infrastructures.

 Since there not much to explain, the ticked was rated as Spam and
 rejected - so may be someone with higher privileges could create such a
 ticket?

 I am not familiar with the Tahoe back end API or the details of the IMAP
 protocol but I'd expect that the two would fit together somehow.

 Best,

 Lukas
 ___
 tahoe-dev mailing list
 tahoe-dev@tahoe-lafs.org
 https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev
___
tahoe-dev mailing list
tahoe-dev@tahoe-lafs.org
https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

Re: How to let these ports listen only on one device?

[David Stainton]

I. Do. Not. Understand-why-you-didn't-try-my-suggestion-out-=-=-(((
That is the expected behavior when tub.port is set to a port number.
In your case you must set tub.port to a Twisted strport/endpoint
string like this:

tub.port = tcp:interface=127.0.0.1:7789

Now... I hope you are not discouraged from using Tahoe-LAFS!
Please continue to ask your questions and I will help you...
however I think for the time being your problem is solved! Enjoy!

Cheers,

David


On Sun, Sep 14, 2014 at 11:28 AM, Roland Haeder rol...@mxchange.org wrote:
 On 09/13/2014 11:36 PM, David Stainton wrote:
 https://tahoe-lafs.org/trac/tahoe-lafs/browser/docs/configuration.rst#overall-node-configuration

 tub.port must be set to something like: tcp:interface=127.0.0.1:7789


 These were my settings:

 tub.port = 8097
 tub.location = 192.168.2.4:8097,127.0.0.1:8097

 Still it is listening to 0.0.0.0:8097

___
tahoe-dev mailing list
tahoe-dev@tahoe-lafs.org
https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

Re: How to let these ports listen only on one device?

[David Stainton]

oopsy... sorry for my hasty reply.
I'm glad it worked out or you! yay!

Cheers!

David

On Tue, Sep 16, 2014 at 8:35 PM, Roland Haeder rol...@mxchange.org wrote:
 On 09/13/2014 11:36 PM, David Stainton wrote:
 https://tahoe-lafs.org/trac/tahoe-lafs/browser/docs/configuration.rst#overall-node-configuration

 tub.port must be set to something like: tcp:interface=127.0.0.1:7789

 tub.port = tcp:8097:interface=192.168.x.y
 tub.location = tcp:interface=192.168.x.y

 This finally worked. Thank you for helping. :)


___
tahoe-dev mailing list
tahoe-dev@tahoe-lafs.org
https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

Re: Tahoe for P2P networks

[David Stainton]

Greetings!

You. Must. Read!!!
http://www.laser.dist.unige.it/Repository/IPI-1011/FileSystems/TahoeDFS.pdf

What does simulating a Peer-to-Peer network? mean?
I have a feeling if you read the above paper it would help you understand 
Tahoe-LAFS.

I2P AND Tor (as in Tor hidden services) can penetrate network address 
translation routers.
Tahoe-LAFS works well with both I2P and Tor.

Hey I2P developers... we are on the same team here... and both Tor and I2P are 
part of
the Tahoe-LAFS anonymity roadmap! Please offer up your corrections and pull 
requests if
you have time to review these 2 documents.

Soon I hope I'll be finished with the native Tor + I2P integration:
https://github.com/david415/tahoe-lafs/blob/tor-i2p-documentation/docs/anonymity-roadmap.rst
https://github.com/david415/tahoe-lafs/blob/tor-i2p-documentation/docs/anonymity-configuration.rst

Note that I am applying document driven development here... so that config doc 
describes features
that are not yet implemented... But this section will tell you how you can use 
Torsocks and Tor Hidden Services with current and older versions of Tahoe-LAFS:

https://github.com/david415/tahoe-lafs/blob/tor-i2p-documentation/docs/anonymity-configuration.rst#torsocks-the-old-way-of-configuring-tahoe-lafs-to-use-tor

Using that configuration you can definitely penetrate NAT devices all day long 
as you connect to your storage servers via their Tor Hidden Service onion 
addresses.

By the way the truckee feature branch has got lots of excellent feature 
additions for Tahoe-LAFS including
multiple introducers and introducer-less feature where clients can specify 
storage server information directly
in their tahoe.cfg file :

https://github.com/leif/tahoe-lafs/tree/truckee

Hopefully we'll get this branch merged upstream soon.


Cheers!

David



signature.asc
Description: Digital signature
___
tahoe-dev mailing list
tahoe-dev@tahoe-lafs.org
https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

Re: How to let these ports listen only on one device?

[David Stainton]

https://tahoe-lafs.org/trac/tahoe-lafs/browser/docs/configuration.rst#overall-node-configuration

tub.port must be set to something like: tcp:interface=127.0.0.1:7789

On Sat, Sep 13, 2014 at 9:22 PM, Roland Haeder rol...@mxchange.org wrote:
 Hello together,

 I run Tahoe-LAFS on my local network on all my computers (because of
 their storage capacity). This also includes Laptops (Debian, Gentoo)
 with WiFi cards.

 I then run batman-adv to let the laptops mesh a little. I have now found
 1 Tahoe-LAFS port open to 0.0.0.0 which means they are accessible from
 outside through my WiFi cards.

 Of course, I can turn them off but then I cannot mesh. :)

 So here are the ports:

 laptop ~ # netstat -lnp|grep 0 0.0.0.0:
 tcp0  0 0.0.0.0:80970.0.0.0:*
 LISTEN  12239/python2.7
 laptop ~ #

 Is there a way to only let it open to my LAN (cable) network, e.g. a
 tweak in config file?

 Best regards,
 Roland




 ___
 tahoe-dev mailing list
 tahoe-dev@tahoe-lafs.org
 https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

___
tahoe-dev mailing list
tahoe-dev@tahoe-lafs.org
https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

Re: Tahoe-LAFS cryptoparty at Noisebridge (San Francisco, CA)

[David Stainton]

Tahoe-LAFS Cryptoparty at Noisebridge at 8pm September 20th, 2014.

Here's my posting to the noisebridge-discuss list:
https://www.noisebridge.net/pipermail/noisebridge-discuss/2014-September/045270.html


On Mon, Sep 8, 2014 at 7:08 PM, Zancas Dearana
zan...@leastauthority.com wrote:
 I hope to attend virtually.

 On Sat, Sep 6, 2014 at 12:10 PM, David Stainton dstainton...@gmail.com
 wrote:


 Dear Tahoe-LAFS enthusiasts,

 I am organizing a Tahoe-LAFS cryptoparty at Noisebridge in the next coming
 weeks
 and I'd be delighted if some of you would attend. The explicit goal of
 this cryptoparty
 is to show participants an alternative to the inferior mainstream cloud
 storage...
 and create a collectively operated Tahoe-LAFS grid which is only
 accessible via Tor Hidden Services.


 https://www.noisebridge.net/pipermail/noisebridge-discuss/2014-September/045259.html

 https://www.noisebridge.net/pipermail/noisebridge-discuss/2014-August/045127.html

 We have not decided upon a date for this event yet.


 Cheers,

 David Stainton
 contact info →  https://www.lumiere.net/~mrdavid/contact.txt


 ___
 tahoe-dev mailing list
 tahoe-dev@tahoe-lafs.org
 https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev




 --
 -- ظ
___
tahoe-dev mailing list
tahoe-dev@tahoe-lafs.org
https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

Tahoe-LAFS cryptoparty at Noisebridge (San Francisco, CA)

[David Stainton]

Dear Tahoe-LAFS enthusiasts,

I am organizing a Tahoe-LAFS cryptoparty at Noisebridge in the next coming weeks
and I'd be delighted if some of you would attend. The explicit goal of this 
cryptoparty
is to show participants an alternative to the inferior mainstream cloud 
storage...
and create a collectively operated Tahoe-LAFS grid which is only accessible via 
Tor Hidden Services.

https://www.noisebridge.net/pipermail/noisebridge-discuss/2014-September/045259.html
https://www.noisebridge.net/pipermail/noisebridge-discuss/2014-August/045127.html

We have not decided upon a date for this event yet.


Cheers,

David Stainton
contact info →  https://www.lumiere.net/~mrdavid/contact.txt



signature.asc
Description: Digital signature
___
tahoe-dev mailing list
tahoe-dev@tahoe-lafs.org
https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

Foolscap endpoints code review

[David Stainton]

Dear Warner,

I have without much trouble broken out the Foolscap client side
endpoint code into it's own branch:

https://github.com/david415/foolscap/tree/endpoint_client1

All Tahoe-LAFS and Foolscap unit tests pass.
Is this OK? Is this commit a small enough unit of code?

This is definitely not the ideal case you mentioned where there are 3
phases of introducing a change... I believe that would take a lot more
effort... but I'd be willing to do it if that is the only way to get
this code merged.


Sincerely,

David
___
tahoe-dev mailing list
tahoe-dev@tahoe-lafs.org
https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

Re: Foolscap endpoints code review

[David Stainton]

I just added another changeset containing unit tests
(maybe we should squash them together)

Here's the branch containing the server side endpoint code.
This is a fully backwards compatible preparatory step on the way to
endpoint agnosticism. Tub's ``getPortnum`` requires the endpoint to be TCP.

https://github.com/david415/foolscap/tree/endpoint_server1


On Sun, Aug 31, 2014 at 6:19 PM, David Stainton dstainton...@gmail.com wrote:
 Dear Warner,

 I have without much trouble broken out the Foolscap client side
 endpoint code into it's own branch:

 https://github.com/david415/foolscap/tree/endpoint_client1

 All Tahoe-LAFS and Foolscap unit tests pass.
 Is this OK? Is this commit a small enough unit of code?

 This is definitely not the ideal case you mentioned where there are 3
 phases of introducing a change... I believe that would take a lot more
 effort... but I'd be willing to do it if that is the only way to get
 this code merged.


 Sincerely,

 David
___
tahoe-dev mailing list
tahoe-dev@tahoe-lafs.org
https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

Fwd: serialization of ListeningPort

[David Stainton]

I posted this to the twisted mailing list a few weeks ago... and I
wonder if any Tahoe devs have thoughts about this.
We essentially need this core twisted api change in order for Foolscap
to be fully endpoint agnostic on the server side.
I think what I need to do next is to write some code and post it on a
new twisted trac ticket...

-- Forwarded message --
From: David Stainton dstainton...@gmail.com
Date: Thu, Jul 3, 2014 at 8:16 AM
Subject: serialization of ListeningPort
To: twisted-pyt...@twistedmatrix.com




Greetings,

I wanted to see what people think of this before attempting to open a
Twisted trac ticket and submitting code patches.

I think Twisted servers need a way to derive a client endpoint
descriptor string from a listeningPort. This approximates an idiom
commonly used in Tahoe-LAFS/Foolscap:


serverEndpoint = serverFromString(tcp:interface=127.0.0.1:0)
listeningPortDeferred = serverEndpoint.listen(myFactory)

def doStuffWithPort(listeningPort):
address = listeningPort.getHost()

myHost = address.host
myPort = address.port
clientEndpointString = tcp:%s:%s % (myHost, myPort)

# announce clientEndpointString to various clients
# ...

listeningPortDeferred.addCallback(doStuffWithPort)


In this case the server needs to announce a client endpoint string
to various clients... the server has enough information to construct
the endpoint string after the listening port has been created. However
the construction of the endpoint string is not done in an endpoint
agnostic manner... and this code will therefore break when used with
Unix domain socket endpoints for instance.

One way to accomplish endpoint agnosticism would be to have a global
function called
getClientEndpointStringFromPort or something like that. This
function would take an IListeningPort object as it's only argument and
return a client endpoint string which can be used to connect to that very
listening port.

Under the hood it should find a plugin that can
serialize the listening port objects into a client endpoint
string. This plugin can be the same class as the client endpoint
parser, since it has the correct prefix attribute with which to
construct the client endpoint string.

I am basically saying that each server endpoint type needs a
small amount of unique code in order to serialize into a client
endpoint descriptor string... because there are various IAddress
implementations with different attributes.

Cheers,

David


signature.asc
Description: PGP signature
___
tahoe-dev mailing list
tahoe-dev@tahoe-lafs.org
https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

[Tails-dev] Tahoe-LAFS persistence

[David Stainton]

-- Forwarded message --
From: intrigeri intrig...@boum.org
Date: Thu, Jun 5, 2014 at 12:23 AM
Subject: Re: [Tails-dev] Tahoe-LAFS persistence
To: The Tails public development discussion list tails-...@boum.org
Cc: tahoe-dev@tahoe-lafs.org, David Stainton dstainton...@gmail.com


Hi,

[Snipping large chunks of discussion about how various kinds of
downloads are advertised on the Tahoe-LAFS website, as I fail to see
what it has to do with Tails. It might be because I've not catched up
with the rest of this thread yet.]

Zooko Wilcox-OHearn wrote (03 Jun 2014 18:18:16 GMT) :
 So my question for Tails devs (this is where my ignorance begins):
 does Tails come with an SFTP client?

Yes.

The first hit I get when searching our website for SFTP is:
https://tails.boum.org/doc/first_steps/introduction_to_gnome_and_the_tails_desktop/index.en.html#index4h1

Additionally, on any relevant Debian system, the command-line sftp
client is shipped as part of the openssh-client package, that is
installed in Tails.

 Does it come with a Bittorrent client?

No: https://tails.boum.org/support/faq/#index22h2

 Can we make Tahoe-LAFS be the third thing next to those two things?

I'm not sure I get what you mean by next to those two things.

Cheers,
--
  intrigeri
  | GnuPG key @ https://gaffer.ptitcanardnoir.org/intrigeri/intrigeri.asc
  | OTR fingerprint @ https://gaffer.ptitcanardnoir.org/intrigeri/otr.asc
___
tahoe-dev mailing list
tahoe-dev@tahoe-lafs.org
https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

[Tails-dev] Tahoe-LAFS persistence

[David Stainton]

-- Forwarded message --
From: intrigeri intrig...@boum.org
Date: Thu, Jun 5, 2014 at 3:03 PM
Subject: Re: [Tails-dev] Tahoe-LAFS persistence
To: David Stainton dstainton...@gmail.com
Cc: The Tails public development discussion list tails-...@boum.org,
tahoe-dev tahoe-dev@tahoe-lafs.org


Hi,

(Disclaimer: I still have not read this full thread.)

David Stainton wrote (05 Jun 2014 13:28:21 GMT) :
 I think what Zooko is suggesting is that the Tahoe-LAFS debian
 package be included in the Tails releases.

Thanks for clarifying.

 Now that we have debian packages and a maintainer

... which is great!

 is there any reason not to?

If someone properly integrates Tahoe-LAFS withing Tails (including
patching tails-persistence-setup, design doc, and whatever nobody has
thought of yet), then I'm happy.

I'm yet to see a branch that implements this, and that we can check
out and test. Also see
https://tails.boum.org/contribute/merge_policy/, particularly the
Documentation is not optional section.

I'm wondering whether, maybe, the best first step would be to add
a page about Tahoe-LAFS in our documentation, that gives an overview
of the pre-requisites (Tails installed with Tails Installer,
persistent volume configured) and needed additional steps (I guess:
have tahoe-lafs installed at every boot with the Additional software
packages feature, make the right directories persistent, use it),
pointing to the relevant documentation pages.

Then, it's easy for anyone interested to try out, and while early
testers give it a try, you can work on integrating Tahoe-Lafs within
Tails, which is now the main blocker to install the software by
default, from my PoV. And the branch that does the integration already
has the user doc ready, it just needs to drop the bullet point about
adding tahoe-lafs to the list of Additional software packages.

But if someone feels bold enough to try and do it all in one single
iteration, well, I'm happy too.

Cheers,
--
  intrigeri
  | GnuPG key @ https://gaffer.ptitcanardnoir.org/intrigeri/intrigeri.asc
  | OTR fingerprint @ https://gaffer.ptitcanardnoir.org/intrigeri/otr.asc
___
tahoe-dev mailing list
tahoe-dev@tahoe-lafs.org
https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

Fwd: [Tails-dev] Tahoe-LAFS persistence

[David Stainton]

-- Forwarded message --
From: intrigeri intrig...@boum.org
Date: Thu, Jun 5, 2014 at 6:07 PM
Subject: Re: [Tails-dev] Tahoe-LAFS persistence
To: David Stainton dstainton...@gmail.com
Cc: The Tails public development discussion list tails-...@boum.org,
tahoe-dev tahoe-dev@tahoe-lafs.org


Hi,

David Stainton wrote (05 Jun 2014 16:58:52 GMT) :
 Since you didn't read the entire thread... I want to make it
 explicitly clear that there is most definitely not one proper
 integration design for Tails and Tahoe-LAFS... AND Tahoe-LAFS
 cannot help with persistence... wrong tool for the job.

This part I got, as Zooko made it clear a while ago on the ticket.
Thanks for explaining anyway :)

When I'm talking of persistence in my last email, it was about making
the Tahoe-LAFS configuration persistent, *not* about using Tahoe-LAFS
to host the persistent volume.

 There could be another track to follow... that might be better for the
 first iteration:

 Do the minimal amount of work possible to get Tails to ship Tahoe-LAFS...
 No fancy backup system. Just Tahoe-LAFS + a GUI + user docs... to help
 users configure their Tahoe client.

 What do you think of this?

Looks good. Still, will need to persist the Tahoe-LAFS configuration.
Hence my proposal.

 Then, it's easy for anyone interested to try out, and while early
 testers give it a try, you can work on integrating Tahoe-Lafs within
 Tails, which is now the main blocker to install the software by
 default, from my PoV. And the branch that does the integration already
 has the user doc ready, it just needs to drop the bullet point about
 adding tahoe-lafs to the list of Additional software packages.

 OK... clearly am confused and do not understand. My apologies for
 misunderstanding you :
 It sounds like you are saying that writing docs for non-existent
 software would make it easy for existent software to be used.

Have you actually tried installing Tahoe-LAFS on Tails, in a way that
it's re-installed automatically on every boot, and you don't have to
reconfigure it every time you start Tails? If you haven't, please do.
I suspect this will make my previous email much clearer :)

Cheers,
--
  intrigeri
  | GnuPG key @ https://gaffer.ptitcanardnoir.org/intrigeri/intrigeri.asc
  | OTR fingerprint @ https://gaffer.ptitcanardnoir.org/intrigeri/otr.asc
___
tahoe-dev mailing list
tahoe-dev@tahoe-lafs.org
https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

Re: [Tails-dev] Tahoe-LAFS persistence

[David Stainton]

 1. the Tahoe-LAFS debian package:

 This part is done. Great!

 Congratulation :)
 Tahoe documentation could mention this:
 https://tahoe-lafs.org/trac/tahoe-lafs/browser/docs/quickstart.rst

OK... I volunteer to update
https://tahoe-lafs.org/trac/tahoe-lafs/browser/docs/quickstart.rst
to mention install Tahoe-LAFS using the Debian package.

 2. Tails persistent volume assistant feature additions:

 It's good to have a concrete integration proposition! Looks good to me
 (understandable and usable), but i'm curious about other people's
 opinions. And... would that not be better to have, like you propose, 3
 columns, but the third being Tahoe, so users can choose to persist
 localy, OR on Tahoe, OR both - by selecting one column, the other, or
 both. But... Tahoe column should be grey if Tahoe has not been configured.

Since Tahoe-LAFS is not a posix compliant filesystem...
we cannot easily create a persistent volume that only
stores data on a Tahoe grid. There is an ugly FUSE hack
but it is extremely ineffient.

So there should be three options per persistent file-set:
1. do not persist
2. persist to local media
3. persist to local media AND a Tahoe-LAFS grid

For the use case where you only want to store the data in
the Tahoe grid... then simply use the Tahoe commandline
tools to upload the file(s).

 And also, who volunteers to hack the persistent volume assistant?

I volunteer to hack the persistent volume assistant.
I would certainly help out with documentation as well.

 3. periodic Tahoe-LAFS backup scheduler
 I suggest to raise the idea in Tahoe mailing list... once it exists we
 can see if it's interesting for Tails :)

I agree! Daira from the Tahoe-LAFS dev team pointed me to the relevant
Tahoe trac tickets. I will be carefully reviewing these... and chatting with the
Tahoe-LAFS dev team when I need advice/assistance/ideas.


 4. Tahoe-LAFS backup GUI applet

I'll be opening a Tahoe-LAFS trac ticket about this one soon.
Of course it will have to be written for the same desktop environment
that Tails uses.
I agree that this is lower priority than hacking the persistent volume assistant
and writing a Tahoe-LAFS client configuration assistant. I will work
on those first.


Onward!

David
___
tahoe-dev mailing list
tahoe-dev@tahoe-lafs.org
https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

Fwd: [Tails-dev] Tahoe-LAFS persistence

[David Stainton]

-- Forwarded message --
From: BitingBird bitingb...@riseup.net
Date: Fri, May 30, 2014 at 2:06 AM
Subject: Re: [Tails-dev] Tahoe-LAFS persistence
To: The Tails public development discussion list tails-...@boum.org


David Stainton:
 Hi,
 [...]
 1. the Tahoe-LAFS debian package:

 This part is done. Great!

Congratulation :)
Tahoe documentation could mention this:
https://tahoe-lafs.org/trac/tahoe-lafs/browser/docs/quickstart.rst

 2. Tails persistent volume assistant feature additions:

 Right now the Tails persistent volume assistant has a user configurable
 list of persistence futures which correspond to sets of files that can
 be persisted.
 The user can selects which to persist or not persist.
 For each of these items there should be a third option: persist to
 Tahoe-LAFS grid and persist to local volume.

It's good to have a concrete integration proposition! Looks good to me
(understandable and usable), but i'm curious about other people's
opinions. And... would that not be better to have, like you propose, 3
columns, but the third being Tahoe, so users can choose to persist
localy, OR on Tahoe, OR both - by selecting one column, the other, or
both. But... Tahoe column should be grey if Tahoe has not been configured.

And also, who volunteers to hack the persistent volume assistant?

 The persistent volume assistant should also prompt the user for some
 information on how to configure Tahoe-LAFS.

Somebody wants to write the text? I looked at Tahoe documentation, and
it seemed that there is no GUI for that.
https://tahoe-lafs.org/trac/tahoe-lafs/browser/docs/running.rst looks
complete, if not very user-friendly, so linking to that could be a good
start.

Once Tahoe persistence is implemented, documentation:
https://tails.boum.org/doc/about/features/index.en.html should mention
Tahoe, probably in the Encryption  Privacy part.
https://tails.boum.org/contribute/design/#index44h3 should mention it,
right after 3.6.17 Persistence feature. Maybe there should be a
sub-ticket from #6227 about the documentation.


 3. periodic Tahoe-LAFS backup scheduler

 This daemon could be part of Tahoe-LAFS... there is nothing Tails
 specific about it.
 [...]

I suggest to raise the idea in Tahoe mailing list... once it exists we
can see if it's interesting for Tails :)

 4. Tahoe-LAFS backup GUI applet
 [...]

Idem, probably interesting ideas for Tahoe developers.

But from what I read from the documentation, I suggest a GUI to setup
Tahoe client first. As it is, it's quite a geek tool, IMO.

 Tahoe-LAFS runs just fine on Tails.

That's a good start :)

Cheers,

 BitingBird
___
Tails-dev mailing list
tails-...@boum.org
https://mailman.boum.org/listinfo/tails-dev
To unsubscribe from this list, send an empty email to
tails-dev-unsubscr...@boum.org.
___
tahoe-dev mailing list
tahoe-dev@tahoe-lafs.org
https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

Fwd: mail server design

[David Stainton]

Hey Michael,

This reminds me of a conversation that came up a few days ago.
If you are going to write the client AND the server then why use SMTP at all?
Why not just use pond (https://pond.imperialviolet.org/)?


David


On Wed, May 28, 2014 at 11:50 PM, Michael Samuel m...@miknet.net wrote:
 Hi,

 I've drawn up a design for a mail server, which uses Tahoe-LAFS as a
 storage medium.

 The server part of this is relatively easy - getting clients right is
 almost certainly the hard part!

 A current brain dump of my design is here:
 https://gist.github.com/therealmik/4cd07c8025757d0d85e3

 Feedback is more than welcome - I don't expect to start coding for a few 
 weeks.

 I'm not 100% familiar with LAFS semantics or API yet, so I expect
 parts of my design to be completely wrong/stupid.

 My project goals are:
 - Implement Inbound SMTP server
 - Implement Outbound SMTP server
 - Implement IMAP server
 - Implement Android sync client

 If there was a way to implement race-free directory changes, I'd
 implement this as just Maildir.  So if
 anyone can think of a high-level abstraction to achieve this I'd love
 to know about it.

 Regards,
   Michael
 ___
 tahoe-dev mailing list
 tahoe-dev@tahoe-lafs.org
 https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev
___
tahoe-dev mailing list
tahoe-dev@tahoe-lafs.org
https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

Re: Accepted tahoe-lafs 1.10.0-2 (source all)

[David Stainton]

I volunteer!


On Tue, May 20, 2014 at 6:36 PM, Zooko Wilcox-OHearn
zo...@leastauthority.com wrote:
 On Sat, May 3, 2014 at 12:51 AM, David Stainton dstainton...@gmail.com 
 wrote:
 Sweet! Does this help Tahoe-LAFS get into the next release of TAILS?

 Well, according to https://labs.riseup.net/code/issues/6227#note-39 ,
 Tails developer intrigeri wrote Anyone interested in pushing this
 further, see comment 20 on this very ticket :). Comment 20 is
 https://labs.riseup.net/code/issues/6227#note-20 , which is my
 description of how Tahoe-LAFS could be useful for Tails users.

 So I think that means that the next step is for some Tails users to
 install Tahoe-LAFS and attempt to use it in the ways described in
 Comment 20, and then post a comment on
 https://labs.riseup.net/code/issues/6227 saying that they used it and
 whether it worked for them.

 Any volunteers?

 Regards,

 Zooko Wilcox-O'Hearn

 Founder, CEO, and Customer Support Rep
 https://LeastAuthority.com
 Freedom matters.
 ___
 tahoe-dev mailing list
 tahoe-dev@tahoe-lafs.org
 https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev
___
tahoe-dev mailing list
tahoe-dev@tahoe-lafs.org
https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

Re: tor hidden service endpoint designs

[David Stainton]

 I also don't mind doing a 0.9.x release with this in it as a preview
 feature, even if it's not completely-fully-baked -- especially if having
 a version of this on PyPI helps your Tahoe dev. Is this something you
 need?

Thanks! That's very nice of you to offer swift code merges.

The Tahoe-LAFS Tor integration is a rather large project...
and I will be thinking about these code deployment issues soon when I finish
the server side port of Foolscap to twisted endpoints.

I'm not sure what the txsocksx author thinks about my pull request:
https://github.com/habnabit/txsocksx/pull/8

It's been a whole week without word... so I'll e-mail him soon and ask
what's up.
Maybe he'd appreciate me writing endpoint parsers for all his socks endpoints...


Thanks!
David
___
tahoe-dev mailing list
tahoe-dev@tahoe-lafs.org
https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

tor hidden service endpoint designs

[David Stainton]

Dear Meejah,

I'm replying by e-mail because it might be better than a conversation
inside a github pull request.

Excellent idea about not launching tor all the time! We were just
talking about that. My friend Leif Ryge and I were discussing possible
designs for this tor hidden service endpoint... and what we realized
is that the Tahoe-LAFS python process will have many endpoints it will
listen on and many endpoints it will connect to. In this case we
really want these features:

- Each Twisted process (in this case Tahoe-LAFS) would use 1 (one)
instance of Tor for all it's client and server communication. We don't
want txsocksx to have to be dependent on txtorcon... however the
txsocksx client tor endpoint could *optionally* read a txtorcon module
variable specifying the tor socks port. If that fails then the tor
client endpoint would try to find the system tor socks port.

- The first time a python process calls the `listen` method on a
hidden service endpoint, the endpoint object will either use the
system tor control port OR it will launch it's own tor process.
(great... because we can reuse your new code for this...) Which ever
it chooses it will then record this information in txtorcon module
variables... so that if the same python process later creates more
hidden service endpoints they will all use the tor process that was
chosen by the first `listen` method call. This last part about
tracking a single tor proc is not implemented yet... but Leif seemed
close to finishing a patch for that. Once this is done then I will
make the necessary changes to the txsocksx tor client endpoint.

With this design we would not really need to specify the control port.
We should probably still have it as an option.. however it isn't needed because
if we try a few ports where the system control port might be and we
cannot connect to it then in that case we should start tor with it's
control port set to an unused port.

By the way... a bit about design patterns I'm using with twisted endpoints:
In Foolscap and Tahoe-LAFS we cannot special case any usage of
endpoint objects. Foolscap currently treats all endpoint objects in
the exact same way... and in this sense is completely decoupled from
the underlying wire protocol; Foolscap is totally decoupled from Tor.
Tahoe-LAFS will have interesting Tor-friendly features like converting
non-tor endpoint descriptor strings into tor endpoint descriptor
strings... when running in Tor-only mode for instance. But
Tahoe-LAFS in this case still should be totally decoupled from
launching Tor AND decoupled from the twisted endpoints api; that is
the job of Foolscap. However... Tahoe-LAFS only knows about endpoint
descriptor strings... and this makes it's job very simple. Each
component in the system has it's own simple responsibilities... this
is why I wrote the parsers and why I cannot use extra additional
public methods calling into an endpoint object.


Sincerely,

David
___
tahoe-dev mailing list
tahoe-dev@tahoe-lafs.org
https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

Re: tor hidden service endpoint designs

[David Stainton]

Hi Meejah,

Briefly changing subject to the IListeningPort interface:

http://foolscap.lothar.com/trac/ticket/203

please read trac comment #19 ---
http://foolscap.lothar.com/trac/ticket/203#comment:19

Reading Foolscap trac ticket 203 caused me to think about the exact
object implementing IListeningPort interface which is fired by the Tor
Hidden Service endpoint `listen` method. A common use case for TCP
servers is to create an endpoint object with port=0; which means
randomly assigned port. Then when IListeningPort is fired you grab
your host-port tuple from the object implementing IAddress returned
from the getHost method of the object implementing IListeningPort. You
may want to pass this information to other clients in the network so
that they can connect to your server. In my opinion the Tor Hidden
Service endpoint should not return the TCP specific IAddress object...
but instead it's own.. with only the needed information (onion and
port).

The twisted documentation specifies that the getHost method of the
IListeningPort should return an IAddress object. If we take a close
look at how various twisted endpoints work... we see that they have
their own objects which implement IListeningPort and IAddress where
appropriate. I'm working on a simple changeset which will cause the
tor hidden service endpoint `listen` method to fire a Tor Onion
address specific object which implements the IListeningPort
interface... and whose getHost method will return a proper Tor Onion
address specific object which implements the IAddress interface.

My unfinished branch implementing these interfaces is here:
https://github.com/david415/txtorcon/tree/endpoint_parser_plugin-rewrite3_tor_onion_address

 Now I'm a little confused: doesn't the txsocksx client-endpoint do TCP
 over Tor connections? That is, client-side connections? It shouldn't
 really care about which Tor instance it's using (except, of course, that
 you don't want to launch lots of them), right?
 I guess maybe you mean you want launch if 0 tors, re-use if 1 logic
 here too?

No... not exactly.

In my previous e-mail I was slightly wrong and unclear when I
mentioned a couple of things...
1. A Tahoe-LAFS introducer or storage node does listen on multiple
ports... but only for onion-grids it will only need one Tor Hidden
Service onion address. Maybe it will be desirable to have a second
onion for the diagnostic web port listener...
2. I did not mean to imply that the txsocksx tor client endpoint
should launch tor. It should not. What it currently does is try a list
of local tcp ports that tor might have it's socks port listening. 9050
is the default tor socks port and 9150 the tor browser bundle default
socks port:
socks_ports_to_try = [9050, 9150]

What I would like to do is have the txsocksx tor client endpoint
optionally (if txtorcon is installed) check for a
txtorcon-launched-tor socks port and add it to the begining of
`socks_ports_to_try`... so that it tries the txtorcon tor socks port
first. If fail then retry with the next port in the list. Repeat until
`socks_ports_to_try` list is consumed.

This should be good enough. Users (developers using txsocksx and
txtorcon) have 3 options:
1. launch their own tor proc using your launch_tor api call directly
2. launch tor by calling `listen` method on a Tor Hidden Service endpoint object
3. there system tor will be used if listening on default ports

In cases 1 and 2 the launching of tor could cause a txtorcon module
attribute to record the socks port and control port. These port
numbers can then be used later by txsocksx tor client endpoints and
txtorcon tor hidden service endpoints. Some applications may want many
outgoing tor connections, some may want many hidden services and some
may want both. In this way we can accommodate all these uses
efficiently.

 One thing I need to bear in mind a little as the library author is
 that I know some people will want a private tor no matter what API
 (which is of course still possible; just use txtorcon.launch_tor).

Yes... you are right that it should have this option.

 Perhaps providing a txtorcon API for tell me all the tors you have
 launched so far would be a good thing. This would also be useful for
 use-cases where you launch lots of tor instances (e.g. things like
 Chutney, which launched a tor test network).

Yeah I agree that txtorcon should accommodate this use-case.
Let's try to clearly work out a good way to do this while still
implementing the above mentioned logic for also ensuring we have only
one instance of tor per python process. I think one instance of Tor
per python process should be the simple default way...

 I think we've already rejected overloading controlPort= for two
 different use-cases, right?

Yeah... Tahoe-LAFS and Foolscap will never need to mess with the controlPort.
Most uses of the endpoint class will not ever need to set the control
port... but someone out there
might want the ability to set it... so we should give it to them but

Foolscap endpoints: status update

[David Stainton]

Greetings,

My Foolscap branch is now backwards compatible with Foolscap 0.6.4...
What I mean is: all of the Tahoe-LAFS unit tests pass when using my
branch of Foolscap. All of the Foolscap unit tests pass too.

I was hanging out with Leif working on Foolscap today... and when I
was thinking about how to make the new endpoints Foolscap branch
compatible with the upstream Foolscap I said aloud: What would
Zooko do? He'd write unit tests to check for backwards compatibility
before writing the code!

So... that is what I did. I first wrote all the tests as they should
be... made sure they all failed and then gradually fixed the code
until all the tests passed. To me this is a new and interesting way to
approach software engineering.

https://github.com/david415/foolscap/commit/f3b7c9b21355741c90dd1e415b51750410a51e5d

The tahoe unit tests outputted this at the end:

Ran 1139 tests in 349.623s
PASSED (skips=5, expectedFailures=3, successes=1131)

Some of the comments in these Foolscap changes need to be modified or
removed... Other than that it should now be ready for code review.
It will be nice to not have so many intermediate commits... so at some
point I'll have to squash/rebase.

What will I be working on next?

- I'll merge in Str4d's Foolscap server side endpoints patch and make
sure all the Foolscap and Tahoe-LAFS unit tests pass.

- Finish creating tor_only mode with Tahoe.cfg configuration option etc.

- Make sure client, introducer and storage nodes in an onion grid
configuration in tor_only mode... works correctly.

- Write Tahoe-LAFS unit tests for tor_only mode?

- Continue working on the txtorconf Tor Hidden Service endpoint/parser...
Leif had an excellent design suggestion... that he didn't quite have
enough time to write code for today.


Sincerely,

David
___
tahoe-dev mailing list
tahoe-dev@tahoe-lafs.org
https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

Foolscap endpoint port: request for preliminary code review

[David Stainton]

Greetings,

I believe the Foolscap-endpoints-port is at least ready for code review.
pull request: https://github.com/warner/foolscap/pull/18/files

So... after code review and perhaps code cleanup... I can
squash/rebase if needed to clean up the git history a bit: Str4d's
server side foolscap endpoints patch was committed by me because I
could not find a git commit id to cherry-pick... we can fix this
later.

In case you were wondering about this branch of Foolscap:
- Yes it passes all the unit tests.
- Yes it is backwards compatible with the old and new style connection
hints... while it also works with valid twisted endpoint descriptors.

I am close to having a branch of Tahoe-LAFS pass all the unit tests
when using this Foolscap branch:
https://github.com/david415/tahoe-lafs/tree/ticket517-unittests

So far I found that a Tahoe-LAFS unit test helped me to find a very
subtle bug in foolscap which allowed unicode to appear in a connection
hint. Fixed.

I guess in theory until all Tahoe-LAFS unit tests pass... it is
possible that there still may be some very subtle bugs in Foolscap.


Sincerely,

David
___
tahoe-dev mailing list
tahoe-dev@tahoe-lafs.org
https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

Re: Foolscap endpoint port: request for preliminary code review

[David Stainton]

I talked with Leif today.
Now I am going to work on making my foolscap branch fully compatible
with the current release of Tahoe.
All the Foolscap and Tahoe unit tests should pass at that point...

Cheers,

David


On Fri, May 9, 2014 at 9:19 AM, David Stainton dstainton...@gmail.com wrote:
 Greetings,

 I believe the Foolscap-endpoints-port is at least ready for code review.
 pull request: https://github.com/warner/foolscap/pull/18/files

 So... after code review and perhaps code cleanup... I can
 squash/rebase if needed to clean up the git history a bit: Str4d's
 server side foolscap endpoints patch was committed by me because I
 could not find a git commit id to cherry-pick... we can fix this
 later.

 In case you were wondering about this branch of Foolscap:
 - Yes it passes all the unit tests.
 - Yes it is backwards compatible with the old and new style connection
 hints... while it also works with valid twisted endpoint descriptors.

 I am close to having a branch of Tahoe-LAFS pass all the unit tests
 when using this Foolscap branch:
 https://github.com/david415/tahoe-lafs/tree/ticket517-unittests

 So far I found that a Tahoe-LAFS unit test helped me to find a very
 subtle bug in foolscap which allowed unicode to appear in a connection
 hint. Fixed.

 I guess in theory until all Tahoe-LAFS unit tests pass... it is
 possible that there still may be some very subtle bugs in Foolscap.


 Sincerely,

 David
___
tahoe-dev mailing list
tahoe-dev@tahoe-lafs.org
https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

Re: [Twisted-Python] tor hidden service endpoint parser returns a deferred

[David Stainton]

 The endpoint constructor should do _construction_.  The endpoint parser
 should do _parsing_.  The task of parsing is of taking strings and producing
 meaningful values.

Ah yeah I see what you mean. OK... I changed it:
https://github.com/david415/txtorcon/commit/1e96d550c40bef1be1b45c3c975c0da0bf6a2949

 You may want to have the endpoint parser actually live in a separate method
 in txtorcon rather than putting it into the plugin itself, and leave the
 plugin simply the task of plugging in, but the parsing method should not be
 called __init__.

Done.

 (Also, don't use asserts.  This is probably indicative of a weakness in the
 parsing API we've provided, but you should likely raise something like
 UsageError.)

OK... I changed that part too.

 What exactly is 'usewithtor'?

 usewithtor pip install twisted

It's torsocks. It performs an LD_PRELOAD hack to force programs to use tor.
Here's some torsocks info:

https://trac.torproject.org/projects/tor/wiki/doc/torsocks
https://gitweb.torproject.org/torsocks.git
https://lists.torproject.org/pipermail/tor-dev/2014-April/006649.html

Torsocks is useful... but a terrible hack... and this reminds me:

I'm hoping that the effort to make Tahoe-LAFS more Tor-friendly will
result in well documented
solutions. I would hope that this would make it easier for other
developers to make their twisted apps more Tor-friendly as well.
I do not think we have all the solution details worked out yet...

We'd like to stop using torsocks with Tahoe-LAFS and instead use
twisted endpoints...
which is why I ported the Foolscap client side to use twisted endpoints.
(str4d already ported the Foolscap server side to use endpoints)

https://tahoe-lafs.org/trac/tahoe-lafs/ticket/517

This trac ticket discusses various ideas to make Tahoe more Tor friendly...
including this endpoint related idea that I've been thinking about
lately in trac comment 16
(https://tahoe-lafs.org/trac/tahoe-lafs/ticket/517#comment:16) by leif
:

My current thought is that Tahoe should have an always_use_tor
option which causes all furls to be rewritten as tor: endpoints before
being passed to foolscap. If this option is not enabled, tor
connections will be made if a tor endpoint is used, but not otherwise.
So, users of mixed tor/non-tor grids can leave it disabled if they
prefer faster connections to non-tor nodes.

This is necessary for a tahoe client wanting to use a public grid over
tor... because the grid's introducer node is going to
announce tcp endpoints not tor endpoints.

However I was thinking that the only_tor mode for tahoe (mentioned
here https://tahoe-lafs.org/trac/tahoe-lafs/ticket/517#comment:15 )
could be implemented in tahoe... it could filter endpoint descriptors
by calling clientFromString or serverFromString and then checking the
object type or is this a terrible way to do it? I'm not sure. The
thing is... tahoe wouldn't normally
know anything about endpoints because it uses Foolscap which could use
endpoints (it does in my branch
https://github.com/david415/foolscap/tree/endpoint_descriptors_server2).

 Don't build eggs.  Why are you building eggs?  Eggs are unfortunately a huge
 problem magnet.  I don't think 'pip install' builds eggs; I don't manually
 run setup.py anywhere any more because I assume pip understands these
 concerns better than I do.

Ah OK yeah it works just fine when installed with pip.

 Figure out a way to hook the _build_ step specifically so you can run
 getPlugins() to generate the dropin cache before zipping up your zip file.
 Twisted plugins actually do work within zip files, but nobody has bothered
 working out the setuptools incantation to get them to generate at the
 appropriate time for building an egg.  This is extra tricky because you only
 want to generate this dropin.cache file if the dropin is going to live
 inside an egg; otherwise, it would conflict with the communal dropin.cache
 file generated at install time.  This option will be the most confusing and
 frustrating for you, but I hope you choose it, because it would be GREAT to
 have a Right Way™ to address this problem in a Twisted plugin project's
 setup.py :).

OK... I'll try to get it working that way... but for now a pip install
seems to work fine.

 zip_safe flag not set; analyzing archive contents...


 It's probably best to set zip_safe one way or the other though, even if you
 want to set it to 'true'.  Letting setuptools guess like this will,
 unsurprisingly, get you unpredictable results.
___
tahoe-dev mailing list
tahoe-dev@tahoe-lafs.org
https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

Re: [Twisted-Python] tor hidden service endpoint parser returns a deferred

[David Stainton]

OK... I decided that txtorcon's endpoint constructor should only take
string arguments besides the reactor arg...
because serverFromString passes only str args to the endpoint constructor.

I wrote the endpoint parser to be as simple as possible.
The endpoint's `listen` method now handles the tor configuration and
launch logic... and returns a deferred which will
fire with an IListenPort.

The pip method definitely works... my code works but prints traceback
with this error:
Unexpected error while writing cache file
Do you as well?

Here's my endpoint parser plugin:
https://github.com/david415/txtorcon/blob/endpoint_parser_plugin-rewrite3/twisted/plugins/txtorcon_endpoint_parser.py

Here's how I install it:
cd ~/projects/virtualenv-1.11.5
./virtualenv.py ~/virtenv-txtorcon
. ~/virtenv-txtorcon/bin/activate
usewithtor pip install twisted
cd ~/projects
git clone https://github.com/david415/txtorcon.git
cd txtorcon
git checkout endpoint_parser_plugin-rewrite3
python setup.py install


Running the example code:
cd examples
python launch_tor_endpoint.py
Unexpected error while writing cache file
Traceback (most recent call last):
  File 
/home/human/virtenv-txtorcon/local/lib/python2.7/site-packages/twisted/internet/endpoints.py,
line 1465, in serverFromString
return _serverFromStringLegacy(reactor, description, _NO_DEFAULT)
  File 
/home/human/virtenv-txtorcon/local/lib/python2.7/site-packages/twisted/internet/endpoints.py,
line 1393, in _serverFromStringLegacy
nameOrPlugin, args, kw = _parseServer(description, None, default)
  File 
/home/human/virtenv-txtorcon/local/lib/python2.7/site-packages/twisted/internet/endpoints.py,
line 1380, in _parseServer
for plugin in getPlugins(IStreamServerEndpointStringParser):
  File 
/home/human/virtenv-txtorcon/local/lib/python2.7/site-packages/twisted/plugin.py,
line 209, in getPlugins
allDropins = getCache(package)
--- exception caught here ---
  File 
/home/human/virtenv-txtorcon/local/lib/python2.7/site-packages/twisted/plugin.py,
line 181, in getCache
dropinPath.setContent(pickle.dumps(dropinDotCache))
exceptions.AttributeError: 'ZipPath' object has no attribute 'setContent'
I have set up a hidden service, advertised at: class
'twisted.internet.tcp.Port' of twisted.web.server.Site on 55141
http://mqrcnytlnh4xynmh.onion:80
locally listening on IPv4Address(TCP, '127.0.0.1', 55141)

and txtorcon's setup.py output:
python setup.py install
WARNING: not using PyPi over SSH!
running install
running bdist_egg
running egg_info
writing requirements to txtorcon.egg-info/requires.txt
writing txtorcon.egg-info/PKG-INFO
writing top-level names to txtorcon.egg-info/top_level.txt
writing dependency_links to txtorcon.egg-info/dependency_links.txt
package init file 'twisted/plugins/__init__.py' not found (or not a
regular file)
reading manifest file 'txtorcon.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'scripts/*.py'
warning: no previously-included files found matching 'docs/_static/*~'
warning: no previously-included files found matching 'examples/*~'
writing manifest file 'txtorcon.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/twisted
creating build/bdist.linux-x86_64/egg/twisted/plugins
copying build/lib.linux-x86_64-2.7/twisted/plugins/txtorcon_endpoint_parser.py
- build/bdist.linux-x86_64/egg/twisted/plugins
creating build/bdist.linux-x86_64/egg/txtorcon
copying build/lib.linux-x86_64-2.7/txtorcon/stream.py -
build/bdist.linux-x86_64/egg/txtorcon
copying build/lib.linux-x86_64-2.7/txtorcon/log.py -
build/bdist.linux-x86_64/egg/txtorcon
copying build/lib.linux-x86_64-2.7/txtorcon/torconfig.py -
build/bdist.linux-x86_64/egg/txtorcon
copying build/lib.linux-x86_64-2.7/txtorcon/router.py -
build/bdist.linux-x86_64/egg/txtorcon
copying build/lib.linux-x86_64-2.7/txtorcon/torcontrolprotocol.py -
build/bdist.linux-x86_64/egg/txtorcon
copying build/lib.linux-x86_64-2.7/txtorcon/circuit.py -
build/bdist.linux-x86_64/egg/txtorcon
copying build/lib.linux-x86_64-2.7/txtorcon/addrmap.py -
build/bdist.linux-x86_64/egg/txtorcon
copying build/lib.linux-x86_64-2.7/txtorcon/interface.py -
build/bdist.linux-x86_64/egg/txtorcon
copying build/lib.linux-x86_64-2.7/txtorcon/spaghetti.py -
build/bdist.linux-x86_64/egg/txtorcon
copying build/lib.linux-x86_64-2.7/txtorcon/torstate.py -
build/bdist.linux-x86_64/egg/txtorcon
copying build/lib.linux-x86_64-2.7/txtorcon/torinfo.py -
build/bdist.linux-x86_64/egg/txtorcon
copying build/lib.linux-x86_64-2.7/txtorcon/__init__.py -
build/bdist.linux-x86_64/egg/txtorcon
copying build/lib.linux-x86_64-2.7/txtorcon/util.py -
build/bdist.linux-x86_64/egg/txtorcon
byte-compiling 
build/bdist.linux-x86_64/egg/twisted/plugins/txtorcon_endpoint_parser.py
to txtorcon_endpoint_parser.pyc
byte-compiling build/bdist.linux-x86_64/egg/txtorcon/stream.py to stream.pyc

tor hidden service endpoint parser returns a deferred

[David Stainton]

Greetings,

Today I wrote an endpoint parser for Meejah's txtorcon Tor Hidden
Service endpoint.

Now that we have foolscap ported to twisted endpoints (branch needs
review but I should probably rebase first) we'd like to be able to
simply specify an onion endpoint descriptor in Tahoe-LAFS
configuration... that way foolscap receives the descriptor and uses
serverFromString to generate the endpoint object.

This will work fine for the txsocksx tor client endpoint parser I
wrote... However the txtorcon Tor Hidden Service endpoint setup
requires a deferred to fire once the tor process is started... This
means that the endpoint parser needs to return a deferred. But this
breaks the interface!... meaning that foolscap or any other api using
this onion endpoint parser will have to special case the situation
where serverFromString returns a deferred.

Terrible! Or do you think this is not so terrible?
I am curious to know your thoughts and if any of you have alternate
solutions... since I am obviously doing it wrong.

Here's my txtorcon onion endpoint parser:
https://github.com/david415/txtorcon/blob/endpoint_parser_plugin/examples/hiddenservice_endpoint_parser.py

And here's some example code that uses it:
https://github.com/david415/txtorcon/blob/endpoint_parser_plugin/examples/launch_tor_endpoint.py

By the way (just in case you don't know)... you can test this parser
by placing it in the twisted/plugins directory... here's how I do it:

cd projects/virtualenv-1.11.1/
./virtualenv.py ~/virtenv-endpoints-test
. ~/virtenv-endpoints-test/bin/activate
usewithtor pip install twisted
cd ~/projects
git clone https://github.com/david415/txtorcon.git
# defaults to my endpoint_parser_plugin branch
cd txtorcon
usewithtor python setup.py install
cd examples # CWD is now /home/human/projects/txtorcon/examples
cp hiddenservice_endpoint_parser.py
/home/human/virtenv-endpoints-test/lib/python2.7/site-packages/twisted/plugins
# and then run the example code that uses this endpoint parser
python launch_tor_endpoint.py



Cheers,

David
___
tahoe-dev mailing list
tahoe-dev@tahoe-lafs.org
https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

Re: adding endpoint plugins

[David Stainton]

For those of you who don't also follow the twisted mailing list... JP
helped me figure this issue with loading plugins.
As it turns out I had to instantiate the endpoint parser object.
A nice one line fix... and my tor client endpoint parser plugin was
loading fine:
https://github.com/david415/txsocksx.git



On Wed, Apr 30, 2014 at 1:42 PM, David Stainton dstainton...@gmail.com wrote:
 Greetings,

 I wrote a simple IStreamClientEndpointStringParser for using the
 SOCKS5ClientEndpoint with Tor:
 https://gist.github.com/david415/26a4ed59078d2e27376f

 I modified a txsocksx code example for testing the new endpoint descriptor:
 https://gist.github.com/david415/7c6040117319cc3b0230

 ...but I am not sure how to register it such that calls to
 clientFromString can return this endpoint when the descriptor prefix
 matches.

 This doc:
 https://twistedmatrix.com/documents/13.2.0/api/twisted.internet.interfaces.IStreamClientEndpointStringParser.html
 mentions placing the plugin in the twisted.plugins package... however this 
 doc:
 http://twistedsphinx.funsize.net/projects/core/howto/plugin.html
 suggests that I can also set the PYTHONPATH to a directory:

 if a directory which has been added to sys.path (typically by
 adding it to the PYTHONPATH environment variable) contains a directory
 named twisted/plugins/ , each .py file in that directory will be
 loaded as a source of plugins.

 I'm using twisted 13.2.0 in a virtualenv... and I've tried setting
 PYTHONPATH to a twisted/plugins directory no avail. I must be doing
 something wrong here but after scouring the docs and twisted source
 code... I'm just not sure.

 This effort is part of solving for Tahoe-LAFS trac ticket 517:
 make tahoe Tor- and I2P-friendly
 https://tahoe-lafs.org/trac/tahoe-lafs/ticket/517

 Thanks to Glyph and JP's help I was able to write a foolscap client
 side patch to use twisted endpoints. See foolscap trac ticket 203:
 http://foolscap.lothar.com/trac/ticket/203#comment:36

 str4d has already written the foolscap server side endpoint patch...
 Therefore it seems to me the next steps are to create parsers for
 endpoints we are interested in supporting... and get Tahoe-LAFS to
 register them so that serverFromString and clientFromString will
 return the expected endpoint objects.

 Cheers!

 David
___
tahoe-dev mailing list
tahoe-dev@tahoe-lafs.org
https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

Re: [Twisted-Python] tor hidden service endpoint parser returns a deferred

[David Stainton]

Ah yeah... twisted endpoints that support udp...
Perhaps when Yawning Angel finishes Lightweight Obfuscated Datagram
Protocol (LODP) there will be interesting things we can do with
this... and a more immediate perceived need for twisted datagram
endpoint interfaces.


On Fri, May 2, 2014 at 9:16 PM, Glyph Lefkowitz gl...@twistedmatrix.com wrote:
 On May 2, 2014, at 1:02 PM, meejah mee...@meejah.ca wrote:

 Another point for this is that someday tor will have UDP support...


 There's a reason that the existing interfaces all have stream in their
 names :-).

 The main reason we didn't do datagram at the same time is that there's a
 pretty clear client/server division in all stream endpoints (one side has to
 initiate, the other side has to respond) but given the possibilities for
 multicast / fan-out / fan-in with datagram sockets, just do what UDP does
 didn't seem obviously correct.

 I'd love it if someone with an interest would make some datagram endpoint
 interfaces though.

 -glyph

___
tahoe-dev mailing list
tahoe-dev@tahoe-lafs.org
https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

Re: [Twisted-Python] tor hidden service endpoint parser returns a deferred

[David Stainton]

Thanks. Sure thing. Yes I have every intention of contributing
endpoint parsers to txtorcon and txsocksx for use with tor.
Both of these could make it much easier for developers to make their
applications tor friendly.

The endpoints/endpoint parsers api is brilliant! When the tor
client/server endpoint parsers are complete then Tahoe-LAFS for
instance can simply use these endpoints without any code changes at
all. Of course I'm not saying that is all that is required for
Tahoe-LAFS to be tor friendly... we have detailed other
considerations in trac tickets:
https://tahoe-lafs.org/trac/tahoe-lafs/ticket/517
http://foolscap.lothar.com/trac/ticket/203

I'm starting to think about the deployment issues... Perhaps someone
on the list knows of the correct setup.py (distutils/setuputils?)
procedure to deploy twisted plugins? I shall have to research this
soon. For now my hacky development/testing flow works OK.


On Fri, May 2, 2014 at 9:18 PM, Glyph Lefkowitz gl...@twistedmatrix.com wrote:

 On May 2, 2014, at 6:59 AM, David Stainton dstainton...@gmail.com wrote:

 Today I wrote an endpoint parser for Meejah's txtorcon Tor Hidden
 Service endpoint.


 This is fantastic.  Basically this is _exactly_ the use-case that we had in
 mind when designing the endpoint parser plugin APIs.

 Thanks so much for doing it.

 (I hope you'll contribute it to txtorcon when you're done...)

 -glyph

___
tahoe-dev mailing list
tahoe-dev@tahoe-lafs.org
https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

Re: Accepted tahoe-lafs 1.10.0-2 (source all)

[David Stainton]

Sweet! Does this help Tahoe-LAFS get into the next release of TAILS?

On Fri, May 2, 2014 at 10:11 PM, Daira Hopwood da...@jacaranda.org wrote:
 Woo!

  Original Message 
 From: Ramakrishnan Muthukrishnan rkrish...@debian.org
 To: debian-devel-chan...@lists.debian.org
 Subject: Accepted tahoe-lafs 1.10.0-2 (source all)
 Date: Fri, 02 May 2014 18:49:17 +

 Format: 1.8
 Date: Fri, 02 May 2014 23:47:07 +0530
 Source: tahoe-lafs
 Binary: tahoe-lafs
 Architecture: source all
 Version: 1.10.0-2
 Distribution: unstable
 Urgency: medium
 Maintainer: Bert Agaz berta...@ptitcanardnoir.org
 Changed-By: Ramakrishnan Muthukrishnan rkrish...@debian.org
 Description:
  tahoe-lafs - Secure distributed filesystem
 Closes: 735940
 Changes:
  tahoe-lafs (1.10.0-2) unstable; urgency=medium
  .
[Ramakrishnan Muthukrishnan]
* Include unminified sources for d3 and jquery. (Closes: #735940)
* New co-maintainer (Ramakrishnan Muthukrishnan).
 Checksums-Sha1:
  a8e674aa20a98eb91a1e0cabba39a8f6d81a18dd 2024 tahoe-lafs_1.10.0-2.dsc
  14fcfa63f44f87d81d840c05216491e7acc328cd 157344 
 tahoe-lafs_1.10.0-2.debian.tar.xz
  0549ef956b15ab1080abb6af26b256ca2bfdada5 1055748 tahoe-lafs_1.10.0-2_all.deb
 Checksums-Sha256:
  b9afdc2f0fad835222f2efe072ec73c4571a899fe3673411aa4e029f8f5c6ad5 2024 
 tahoe-lafs_1.10.0-2.dsc
  0e8dc50dcf2afc3e7eae924b39891655185f31fb06f03d9dd02a4f7676954ca0 157344
 tahoe-lafs_1.10.0-2.debian.tar.xz
  06b57f6a220c608c875cda3fa5d938878ca3c3c4eb52a021d0ce25e2871bf605 1055748
 tahoe-lafs_1.10.0-2_all.deb
 Files:
  df3321b7ab35329579d57c9afacb5714 1055748 utils optional 
 tahoe-lafs_1.10.0-2_all.deb
  28867b477f92a0d9b95c7c648b211013 2024 utils optional tahoe-lafs_1.10.0-2.dsc
  7e64ec4b36110791fe894b0618f5ef7b 157344 utils optional 
 tahoe-lafs_1.10.0-2.debian.tar.xz



 ___
 tahoe-dev mailing list
 tahoe-dev@tahoe-lafs.org
 https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

___
tahoe-dev mailing list
tahoe-dev@tahoe-lafs.org
https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

adding endpoint plugins

[David Stainton]

Greetings,

I wrote a simple IStreamClientEndpointStringParser for using the
SOCKS5ClientEndpoint with Tor:
https://gist.github.com/david415/26a4ed59078d2e27376f

I modified a txsocksx code example for testing the new endpoint descriptor:
https://gist.github.com/david415/7c6040117319cc3b0230

...but I am not sure how to register it such that calls to
clientFromString can return this endpoint when the descriptor prefix
matches.

This doc:
https://twistedmatrix.com/documents/13.2.0/api/twisted.internet.interfaces.IStreamClientEndpointStringParser.html
mentions placing the plugin in the twisted.plugins package... however this doc:
http://twistedsphinx.funsize.net/projects/core/howto/plugin.html
suggests that I can also set the PYTHONPATH to a directory:

if a directory which has been added to sys.path (typically by
adding it to the PYTHONPATH environment variable) contains a directory
named twisted/plugins/ , each .py file in that directory will be
loaded as a source of plugins.

I'm using twisted 13.2.0 in a virtualenv... and I've tried setting
PYTHONPATH to a twisted/plugins directory no avail. I must be doing
something wrong here but after scouring the docs and twisted source
code... I'm just not sure.

This effort is part of solving for Tahoe-LAFS trac ticket 517:
make tahoe Tor- and I2P-friendly
https://tahoe-lafs.org/trac/tahoe-lafs/ticket/517

Thanks to Glyph and JP's help I was able to write a foolscap client
side patch to use twisted endpoints. See foolscap trac ticket 203:
http://foolscap.lothar.com/trac/ticket/203#comment:36

str4d has already written the foolscap server side endpoint patch...
Therefore it seems to me the next steps are to create parsers for
endpoints we are interested in supporting... and get Tahoe-LAFS to
register them so that serverFromString and clientFromString will
return the expected endpoint objects.

Cheers!

David
___
tahoe-dev mailing list
tahoe-dev@tahoe-lafs.org
https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

Re: If the machine run the introducer down,what can i do

[David Stainton]

Hi Zooko,

Yes... a once a week meeting sounds good and doable.
I reside in Berlin... and I am usually awake during CET daylight hours.

I would like to suggest we meet at 17:00 UTC because it would be
convenient for me as I would simply stay
in my office a bit longer and chat with you two.

22:00 UTC is a bit late for work nights... so I would suggest 21:00
UTC or earlier... unless it happens to
be the weekend or friday night.

I'm not particular about which day we meet as I currently do not have
much of a weekly schedule to follow other than the square 9-6 mon-fri
routine.


Cheers!

David



On Mon, Mar 17, 2014 at 3:30 PM, Zooko Wilcox-OHearn
zo...@leastauthority.com wrote:
 Hey dstainton415 and str4d:

 Is there an hour per week we could all get together on IRC and/or a
 videoconferencing and/or screen-sharing tool and work on this? Str4d
 and I had previously agreed on Monday 22:00Z, but that hasn't really
 come together yet, and also I'm not sure if DST [*] changes Str4d's
 availability for that.

 Regards,

 Zooko

 [*] Fuck DST!
___
tahoe-dev mailing list
tahoe-dev@tahoe-lafs.org
https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev