[tor-dev] Tor2web 2.0 is live!

[Arturo Filastò]

Hi all,

We are glad to announce the release of the new tor2web software.

For those of you who are not aware of what tor2web is let us give you a
brief description. The goal of tor2web is that of promoting the use of
Tor Hidden Services
(https://www.torproject.org/docs/hidden-services.html.en). Hidden
Services allow people to run TCP based services without disclosing the
identity/location of their server. In the specific they allow people to
anonymously publish content to the web. Also, since you are being
reached trough the Tor network, you are not required to have a static ip
address or purchase a domain. This lowers the entry barrier to content
publishing and protect the content publisher from retaliation and Denial
of Service attacks.

The problem though is that Hidden Services are usually only accessible
by installing a Tor client
(https://www.torproject.org/projects/torbrowser.html.en). Tor2web
creates a transport, by acting as a web proxy, between the internet and
the Tor network. This means that anonymous publishers are able to reach
a much wider audience. The user visiting a website though tor2web is
always advised to install a Tor client as by doing so he will protect
his identity and leverage Hidden Services end-to-end encryption.

This version of tor2web (called tor2web 2.0) is based on glype PHP web
proxy (http://www.glype.com) and it is by no means the definitive
solution. We are currently working on a new design that will be able to
withstand other "attacks" that are currently possible.

What we have implemented is:
* A clear disclaimer warning the user that the content is not being
served directly from the server, but it comes from the Tor network
* Contact forms for abuse complaints and to report broken websites
* Transparent rewriting of URLs into the tor2web form (i.e.
so4rmjdiwmqjosxz.onion become so4rmjdiwmqjosxz.tor2web.org)
* Blocklists to allow a tor2web node maintainer to block particular
websites, the blocklists are stored in md5 format so the node maintainer
does not need to store potentially illegal site lists.

At this current stage we would like the community to stand-up and help
us by:
* Finding security and functional bugs in the existing implementation
* Volounteering to run new tor2web servers:
   In this first stage we are looking for reliable systems, run or
endorsed by trustworthy organizations involved in anonymity and privacy
research and development.

For the new release the goals that we wish to further pursue are:
* Distribute responsibility across multiple actors
* Minimize the probability of takedown of a tor2web node

If you want further information on the tor2web project visit:
Wiki for new developments: http://wiki.tor2web.org/
Tor2web original website: http://www.tor2web.org
Github: https://github.com/globaleaks/tor2web-2.0
Mailing List: tor2web-t...@lists.tor2web.org on http://bit.ly/pxFwNS .
IRC: irc.oftc.net #tor2web

Have a nice day,
Some Random GlobaLeaks Contributors

Please spread across the anonimity communities and mailing lists

___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Future direction for torperf

[Arturo Filastò]

I have some requirements for torperf. I would like it to be able to give
me some useful data about tor2web improvements.

We have made a list of the things we think we are interesting in taking
note of: https://github.com/globaleaks/tor2web-2.0/issues/2.

- Art.

On 9/29/11 9:16 PM, Thomas S. Benjamin wrote:
> Dear all,
> 
> I am working on a re-design of torperf that will use a central python
> script to automate initiation and control of torperf experiments.
> 
> I have a very rough draft that provides the most basic level of
> functionality in tomb/torperf.git branch 3280, and have ticket #3280
> as wanting review.
> 
> The code needs substantial improvement, but I would love feedback on
> my general approach before I spend a lot of time on grooming this
> specific version.
> 
> If you use torperf please consider giving me feedback on the direction
> in which I am heading with this.  I want to make sure that I am
> designing something that will work for other people, rather than just
> for my own experiments.
> 
> I have chosen not to merge into a branch of torperf.git because my
> working style involves lots of frequent verbose commits, which would
> be distracting for most people.  I will roll together the many commits
> before merging into torperf.git
> 

___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Pluggable Transport through SOCKS proxy

[Arturo Filastò]

Following up on the discussion at the Tor dev meeting with Nick this
is the proposal for the Pluggable Transport through SOCKS proxies.

You can also find it committed on github:
https://github.com/hellais/torspec/commit/f0a89c8ded1971199fec6ffd0fe7e40562274ced

Filename: xxx-pluggable-transports-through-proxy.txt
Title: Pluggable Transport through SOCKS proxy
Author: Arturo Filastò
Created: 28 February 2012
Status: Draft

Overview

  Tor introduced Pluggable Transports in proposal 180 Pluggable Transports
  for circumvention.

  The problem is that Tor currently cannot use a Pluggable Transport proxy
  and a SOCKS proxy at the same time. This has been noticed by users in
  #5195, where Tor would be failing saying "Unacceptable option value:
  You have configured more than one proxy type".


Trivia

  This comes from a discussion that came up with Nick and I promised
  to write a proposal for it if I wanted to hear what he had to say.
  Nick spoke and I am writing this proposal.

Acknowledgements

  Most of the credit goes to Nick Mathewson for the main idea and
  the rest of it goes to George Kadianakis for helping me out in writing
  it.

Motivation

  After looking at some options we decided to go for this solution solution
  since it guarantees backwards compatibility and is not particularly
  costly to implement.

Design overview

  When Tor is configured to use both a Pluggable Transport proxy and SOCKS
  proxy it should delegate the proxying to the pluggable transport proxy.

  This can be achieved by setting the environment variables for the SOCKS
  proxy to that specified inside of the torrc.

  When the pluggable transport proxy starts it will first read the environment
  variables and if it detects that it should be using a SOCKS proxy make
  all it's traffic go through it. Once the pluggable transport proxy has 
successfully
  established a connection to the SOCKS proxy it should notify Tor of it's
  success or failure.
  When both the SOCKS and the PluggableTransport directives are set Tor
  should set the environemnt variable start the pluggabletransport proxy and 
wait
  for it to report back on the SOCKS proxy status. If the pluggable transport
  reports back a failure or it does not report back at all (maybe because
  it is an outdated version), Tor should notify the user of the failure
  and exit with an error.


  The environment variables can also contain the credentials for accessing
  the proxy.

Specifications: Tor Pluggable Transport communication

  When Tor detects a SOCKS proxy directive and a Pluggable Transport
  proxy directive it sets the environment variable:

"TOR_PT_PROXY" -- This is the address of the proxy to be used by
the pluggable transport proxy. It is in the format:
://:@:
ex. socks5://tor:test1234@198.51.100.1:8000
socks4a://198.51.100.2:8001


  If the pluggable transport proxy detects that the TOR_PT_PROXY environment
  variable is set it attempts connecting to it. On successs it will
  write to stdout (as specified in 180-pluggable-transport.txt)
  PROXY true. On failure it should write PROXY-ERROR .

  If Tor does not read any PROXY line or it reads a PROXY-ERROR line
  and it is configured to use both SOCKS and PluggableTransport it should
  exit with error.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Pluggable Transport through SOCKS proxy

[Arturo Filastò]

On Feb 28, 2012, at 5:38 PM, Robert Ransom wrote:

> On 2012-02-29, Arturo Filastò  wrote:
> 
>>  When Tor is configured to use both a Pluggable Transport proxy and SOCKS
>>  proxy it should delegate the proxying to the pluggable transport proxy.
>> 
>>  This can be achieved by setting the environment variables for the SOCKS
>>  proxy to that specified inside of the torrc.
>> 
>>  When the pluggable transport proxy starts it will first read the
>> environment
>>  variables and if it detects that it should be using a SOCKS proxy make
>>  all it's traffic go through it. Once the pluggable transport proxy has
>> successfully
>>  established a connection to the SOCKS proxy it should notify Tor of it's
>>  success or failure.
>>  When both the SOCKS and the PluggableTransport directives are set Tor
>>  should set the environemnt variable start the pluggabletransport proxy and
>> wait
>>  for it to report back on the SOCKS proxy status. If the pluggable
>> transport
>>  reports back a failure or it does not report back at all (maybe because
>>  it is an outdated version), Tor should notify the user of the failure
>>  and exit with an error.
> 
> That's not very nice.  At a minimum, Vidalia users will never be able
> to use the GUI to recover from setting such a configuration.  (Users
> can put Tor into such a configuration using the GUI, by configuring
> Tor to use a proxy while a managed transport which does not support
> one is specified in the torrc.)
> 
> 
>> Specifications: Tor Pluggable Transport communication
>> 
>>  When Tor detects a SOCKS proxy directive and a Pluggable Transport
>>  proxy directive it sets the environment variable:
>> 
>>"TOR_PT_PROXY" -- This is the address of the proxy to be used by
>>the pluggable transport proxy. It is in the format:
>>://:@:
>>ex. socks5://tor:test1234@198.51.100.1:8000
>>socks4a://198.51.100.2:8001
> 
> What does Tor send if a SOCKS username or password contains ':', '@', or '\0'?
> 

Well I think no username and password should contain that characters, 
especially the
last. If that is not the case I am sure an approach to deal with this problem 
has been
found by many others in the past. I would just take a look at their solution.

> How does Tor specify an HTTP proxy?
> 

   HTTPProxy host[:port]
   Tor will make all its directory requests through this host:port (or 
host:80 if port is not specified), rather than connecting directly to any 
directory
   servers.


> How does Tor specify an HTTP/HTTPS proxy (i.e. an HTTP proxy which
> supports the CONNECT method)?
> 

   HTTPSProxy host[:port]
   Tor will make all its OR (SSL) connections through this host:port 
(or host:443 if port is not specified), via HTTP CONNECT rather than connecting
   directly to servers. You may want to set FascistFirewall to restrict 
the set of ports you might try to connect to, if your HTTPS proxy only allows
   connecting to certain ports.


> How does Tor pass proxy settings to a managed transport after it has
> started?  (If it can't, then you'll have to either (a) break all OR
> connections through that transport by stopping and restarting it or
> (b) remember to not use that instance of the transport again, and
> launch and start using another instance of the same transport for new
> OR connections with the same managed transport specified.  (a) is
> easier to implement, but not nice.)
> 

via the environment variable TOR_PT_PROXY. This means of communication
is documented inside of proposal 180.

> 
>>  If the pluggable transport proxy detects that the TOR_PT_PROXY environment
>>  variable is set it attempts connecting to it. On successs it will
>>  write to stdout (as specified in 180-pluggable-transport.txt)
>>  PROXY true. On failure it should write PROXY-ERROR .
> 
> What kinds of failures lead to a PROXY-ERROR response?
> 

That the proxy server in unreachable, that the authentication has failed for 
example.

>> 
>>  If Tor does not read any PROXY line or it reads a PROXY-ERROR line
>>  and it is configured to use both SOCKS and PluggableTransport it should
>>  exit with error.
> 
> Are managed transports not permitted to report to Tor that they have
> had a non-fatal error while attempting to connect to a proxy?
> 

Yes that is what is done with PROXY-ERROR. This means of communication
between Tor and pluggable transports is documented better inside of proposal 
180.

- Art.


___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Deployability of Python software.

[Arturo Filastò]

We were discussing last night with George about deployability of python
application on multiple platforms. 

In particular how it would work out if there were to be a python port of 
obfsproxy
and we wanted to have it deployed inside of the Tor Browser bundle.

The issues that he said were raised in other discussions with Nick and Roger
are mainly the following:

- How do we get a good Windows binary of the Software?
- How do we keep the size down to an acceptable level?
- What kind of performance drawbacks would we be experiencing?
- Is it even secure to do crypto in python?

I will try and address these issues as they are something that I ran into
also while designing AWAF (Anonymous Web Application Framework):
http://wiki.globaleaks.org/index.php/Awaf and 
https://piratenpad.de/p/AnonymousWebApplicationFramework

For packaging python software on Windows and OSX, what is generally done is
shipped a precompiled python interpreter and bundle everything up with a nice 
bow.

This technique is already quite tested in real world applications: an example 
that I
particularly like is Tucan Manager (http://www.tucaneando.com/development.html).

This application is basically a download manager written in python and gtk. 
The final size of the packaged software is 20MB. If you remove gtk this size 
goes down
to around 10MB.

What they are using to bundle up the application for Windows is py2exe and 
py2app for
OSX.

Another very widely used solution for packaging python applications in 
PyInstaller and
that is probably the solution I would recommend. Quite a few open source 
software
uses it already:
http://www.pyinstaller.org/wiki/ProjectsUsingPyInstaller

George also mentioned to me pypy, though I don't think pypy is ready for 
building shippable
application just yet.

The issue of size is something that we should come to an agreement on what is 
acceptable.
What is the maximum size that we are comfortable with shipping? We are already 
shipping
a TBB that has 25 MB of QT libraries in it, I don't think a 13 MB Python 
interpreter is going to
be killer.

With respect to performance I don't think it is particularly an issue. Python 
is pretty fast and if
it is not fast enough for what needs to be done you can always rewrite the code 
in C and
integrate that piece of application logic as a python binding.

By talking to some of the core python developers my understanding is that there 
is a way of 
securely storing keys in memory and wiping that memory region in python. It 
involves using
bytearray. We you override a cell in a byte array you are not simply 
dereferencing the pointer
to the python struct, you are actually overwriting that portion of memory.
I think I might write a blog post about this and illustrate what other python 
crypto software is
using to solve this problem (PyCrypto etc.).

In conclusion having a python interpreter shipped as part of Tor would allow 
developers of
anonymity related software to integrate their "Tor add-ons" into a Tor bundle 
easily. I am thinking
of for example making a Tor IRCD bundle, a Tor HTTPD bundle, etc.

What do you think?

- Art.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Deployability of Python software.

[Arturo Filastò]

On 3/7/12 2:24 AM, intrigeri wrote:
> Hi,
>
> Fabio Pietrosanti (naif) wrote (07 Mar 2012 08:24:50 GMT) :
>> So two activists for example would be able to have a redundant,
>> anonymous, 0-maintenance, easy-to-be-setup web application server.
> This rings a bell:
>
>   https://www.torproject.org/getinvolved/volunteer.html.en#tailsServer
>   https://tails.boum.org/todo/server_edition/ 
>
> I'm sorry I did not read this thread, so this may be totally OT.
>
This is quite OT and I think naif brought the discussion a bit off
of the main discussion at hand (:P). Though while we are at it you may
be interested in checking out Anonymous Web Application Framework which
does exactly what you are describing for web sites.
https://piratenpad.de/p/AnonymousWebApplicationFramework

This will a GSoC project for either GlobaLeaks (if we get in) or Tor.


- Art.

___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Implement JSONP interface for check.torproject.org

[Arturo Filastò]

Resurrecting a thread from the grave!

>> I have made a patch to check.torproject.org to expose a JSONP interface
>> that would allow people to have the user check client side if (s)he is
>> using Tor.
>>
>> This would allow people to embed a badge on their website
>> (privacybadge.html) that congratulates the user of using Tor or warns
>> him of non Tor usage with a link to torproject.org.
>>
>> I can imagine privacy advocates having this deployed on their websites
>> or systems that engourage users to connect to them anonymously.
>>
>> Compared to what check.torproject.org does at the moment the risk does
>> not change, it is erogating exactly the same service, just making it
>> more useful and flexible.
>>
>> Basically what it does is check if the ip doing the connection is
>> connected through Tor. The web service will reply with a JSON encoded
>> array that can be loaded from the user and display in the browser a nice
>> looking badge.
>>

Since I noticed that check.tpo was removed from the front page I was
thinking it would be a good idea to bring back up the topic of migrating
check.torproject.org to a JSONP based system.

Such a system would also allow to have the "JSONP check nodes" distributed
across multiple machines (avoiding the single point of failure that check
currently is) and the client side software could be embedded inside of
TBB directly.

People could further promote the usage of Tor by placing an "Anonymity"
badge on their website.

A person wishing to setup such a node needs to simply install TorBel
and a python based web app that runs this JSONP system.

My threat model for this is very lax, so I don't see any purpose in
bad actors telling a client when he is not using Tor that he is using it.
If check.tpo tells the user is not using Tor it already means that TBB
failed, the purpose of it is just to provide visual feedback to the user
that all is did went well.

>> I still need to finish the styling of the badge to contain links to
>> torproject.org and generally make it cooler.
>>
>> Also, the check.torproject repo should be moved to svn.
>>
> Isn't it already in svn? Shouldn't we move it to git?
>
If check is moved to git and you think it is a good idea I can start
working on this.


- Art.

___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Self publishing over Tor Hidden Services

[Arturo Filastò]

Tor Hidden Services are great, though their impact is grossly limited by the
fact that they are not at all easy to deploy. Systems such as Tor2web allow
people that decide to publish anonymously to be reachable by anybody not
using
a Tor client.

For dealing with the usability aspect of Tor Hidden Services, this GSoC I am
going to be mentoring APAF: Anonymous Python Application Framework. The goal
is to give easy to use tools for people to do self publishing.

This is a basic description of the project:

1. Overview
  Tor Hidden Services are underused compared to their potential, the goal
  of APAF is to provide an easy system to allow network related python
  application developers to build their software in a way that it runs as
  a Tor Hidden Service (Tor HS).
  The framework will allow developers to easily build .exe, .app, statically
  linked linux binaries that contain the python interpreter and the Tor
daemon.
  This will allow the end user to easily start running that service on their
  machine, by simply downloading a package. This is similar to what is
done with the
  Tor Browser Bundle (TBB).

2. Motivation
  One of the reasons for which Tor HS are not used that much is that
there is
  no simple way for an application developer to ship their application
with a
  Tor binary and automatically configure a Tor HS.
  This leads to users not being able to easily run Tor Hidden Services
on their
  desktop machines limiting the diffusion of HSs.
  An example use case is a person that wishes to run a temporary chat
server on
  their home machine. With APAF a chat server developer could package such a
  python application and the end user will be able to run it by
downloading a package
  and executing it.

3. What is built?
  APAF compiles all the dependencies for all the target systems.
  The software that will come bundled with it are:
  * the Python interpreter (cpython bundled with PyInstaller:
http://www.pyinstaller.org/)
  * Tor
  * The desired python dependecies (computed with PyInstaller)
  The build system must be configurable and extensible.
  It should allow easy bundling of third party applications such as
p7zip, gpg, etc
 as APAF modules, in order to let the project grow with new functionalities.
 
  The output of the build process will be:
  - Win32: MyApplication.exe
  - OSX: MyApplication.app (inside an Application.dmg container)
  - Linux: Deb build or statically linked binary
  The buildsystem should download the latest release of Tor for the
appropriate platform
  and extract the required files into the build structure, in order to
be packaged within the application.
  Note: Another possibility is that it could build Tor from source for
the desired target
  platforms, but this may require some additional effort.

4. What happens when I start APAF?
  When APAF starts the user running it is presented with a splash screen
that
  displays the startup progress. The image in the splash screen should
be customizable
  by the application developer.
  Another option would be to start the system browser and point it to
  http://127.0.0.1:/ and display the bootstrap process inside
of the bundled
  web based UI.
  At first launch APAF will show a startup splash screen with a progress
bar describing
  application startup event informations, optionally displaying an image.
  Then the system browser will be started to let the user access APAF
UI, that
  will provide a wizard for bootstrapping the setup of the Tor Hidden
Service.
 If the APAF application is already running by clicking on it, it will
just start the browser
to open directly the APAF UI.
  By default APAF will come with a web application that is used for
administering
  and checking on the status of the running Tor HS. It should provide
functionality
  the following functionality:
  * Check the current status of the Tor HS (it's hostname and port mapping)
  * Start and stop tor Hidden Service
  * API to add/remove new Tor Hidden service mapping
  * Select from the list of bundled applications the ones to run
  * Test it's reachability from the Tor network (by doing a request over
Tor to it's .onion address)
  * Configure Tor (User Interface to edit torrc)
   * Close Awaf

5. Web Applications
  One of the first applications that will be used as an example for APAF
will be
  a simple python web application. The application will simply serve to
the client
  static files.
  The basic scaffolding that this web application provides should allow
developers
  to build their own web application based on this example.
  The application will be written using TornadoWeb
(http://www.tornadoweb.org/).

6. Security Features
  Outbound Connection Torrification
  -
  The framework must provide support to automatically torify all or
specific outbound connection.
  The entire python application framework (Tornadoweb) should be
forbidden to make any outbound
  connections directly, it should not leak out of the Tor network.
  

[tor-dev] Improving Tor Hidden Services

[Arturo Filastò]

Setting aside the issue related with usability there are also some
interesting
improvements that can be made to make Tor HS more performant.

I will summarize here the ideas that have been brought forward along
with some
that are not detailed anywhere and would like to see more interest in.

I would suggest to start collecting all the information regarded to Tor HS
improvements on this wiki page:
https://trac.torproject.org/projects/tor/wiki/org/roadmaps/Tor/HiddenServices.

With respect to what is already on that page I got some feedback from
rransom
on those two items on IRC, but I did not note them down. It would be
good if you
were to summarize the critiques here or on the wiki page.

Also there are a set of proposals that are related to Tor HS
improvements that
have been abandoned for some time and I believe it would be useful to
summarize
them inside of that wiki page.

The proposals are:

#121
Filename: 121-hidden-service-authentication.txt
Title: Hidden Service Authentication
https://gitweb.torproject.org/torspec.git/blob/HEAD:/proposals/121-hidden-service-authentication.txt

#142
Filename: 142-combine-intro-and-rend-points.txt
Title: Combine Introduction and Rendezvous Points
https://gitweb.torproject.org/torspec.git/blob/HEAD:/proposals/142-combine-intro-and-rend-points.txt

#143
Filename: 143-distributed-storage-improvements.txt
Title: Improvements of Distributed Storage for Tor Hidden Service
Descriptors
https://gitweb.torproject.org/torspec.git/blob/HEAD:/proposals/143-distributed-storage-improvements.txt

#155
Filename: 155-four-hidden-service-improvements.txt
Title: Four Improvements of Hidden Service Performance
https://gitweb.torproject.org/torspec.git/blob/HEAD:/proposals/155-four-hidden-service-improvements.txt

#194
Filename: 194-mnemonic-urls.txt
Title: Mnemonic .onion URLs
https://gitweb.torproject.org/torspec.git/blob/HEAD:/proposals/194-mnemonic-urls.txt

and also this inside of the ideas, that is loosely related to #194, but
instead of offering
an encoding it offers a petname system:

Filename: xxx-onion-nyms.txt
Title: .onion nym system
https://gitweb.torproject.org/torspec.git/blob/HEAD:/proposals/ideas/xxx-onion-nyms.txt

The single most important thing I believe is needed in Tor Hidden
Service is Encrypted services.
These can be seen, in a way, as the reverse of Tor2web mode. It allows
people to publish Hidden Services
with no anonymity, but have the Tor end-to-end encryption and
performance improvements.
I see these to be the future of what was previously done, poorly, with
Tor Exit Enclaves. One that
wishes to have an end-to-end encrypted tunnel from Tor clients can run
an encrypted service and have
a reduced number of hops from the IP and RP.

Roger started writing up a spec on this and it can be found here:

Filename: xxx-encrypted-services.txt
Title: Encrypted services as a replacement to exit enclaving
https://gitweb.torproject.org/torspec.git/blob/HEAD:/proposals/ideas/xxx-encrypted-services.txt


- Art.




___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Implement JSONP interface for check.torproject.org

[Arturo Filastò]

On 3/23/12 4:34 PM, Robert Ransom wrote:
> On 2012-03-23, Arturo Filastò  wrote:
>
>> Since I noticed that check.tpo was removed from the front page I was
>> thinking it would be a good idea to bring back up the topic of migrating
>> check.torproject.org to a JSONP based system.
> JSONP gives the party which is expected to provide a piece of data the
> ability to run arbitrary JavaScript code in the security context of
> the website which requested the data.  The Tor Project should never
> put itself in a position to have that level of control over other
> parties' websites.

If this is a concern, and I don't think it is since Tor Project
already has the ability to get users to run arbitrary code when
they first start their browser, it could be managed by having the
badge loaded on third party websites inside of an IFRAME.

This would mean that the execution of anything is relative to that
IFRAME.

An alternative to using the JSONP object would be to do a XHR with
"Access-Control-Allow-Origin: *". This is only supported since firefox
3.5, but I don't think it would be an issue for TBB.

A XHR does not lead to any code execution and all that the rogue node
can do is tell the client that he is not running Tor.

>> Such a system would also allow to have the "JSONP check nodes" distributed
>> across multiple machines (avoiding the single point of failure that check
>> currently is) and the client side software could be embedded inside of
>> TBB directly.
>>
>> People could further promote the usage of Tor by placing an "Anonymity"
>> badge on their website.
>>
>> A person wishing to setup such a node needs to simply install TorBel
>> and a python based web app that runs this JSONP system.
>>
>> My threat model for this is very lax, so I don't see any purpose in
>> bad actors telling a client when he is not using Tor that he is using it.
>> If check.tpo tells the user is not using Tor it already means that TBB
>> failed, the purpose of it is just to provide visual feedback to the user
>> that all is did went well.
> check.torproject.org is the only service which can warn Tor users that
> a security upgrade is available for the Tor Browser Bundle.

Good point. I had not considered this aspect. Though wouldn't this
be replaced by thandy in the future?

Are we sure the best way to inform users of updates is through
check.tpo?

> It is also accessed by every Tor Browser Bundle as the first page
> shown after the user uses the ‘New Identity’ Torbutton command; any
> party which can impersonate check.torproject.org can plant
> user-tracking cookies in every TBB user's browser.

With the XHR solution this would not be an issue anymore.

> check.torproject.org cannot ever be run by untrusted parties, and
> cannot ever use a JSONP service provided by untrusted parties.
>

I disagree. If we properly define what the threat model is
I am sure we can figure out a way to make a solution that fits
it.

The overall question is, currently check.tpo is a centralized
single point of failure. Can we do better? Is there a way to
run a distributed infrastructure of this kind?

>> If check is moved to git and you think it is a good idea I can start
>> working on this.
> It is a more horrible idea now than it was the first time you proposed
> it.
>
Heh, I appreciate your comments, although you are often a bit rough
:P.


- Art.

___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Alternatives to Tor Exit Enclaves

[Arturo Filastò]

The purpose of Exit Enclaves was to allow people running a website to
make Tor users
access it without ever leaving the Tor network. This leads to the
clients having end-to-end
encryption with the target destination.

Even in previous version this had some issues, one of which was the fact
that at the first
connection the user would not be accessing the destination over a Tor
circuit if the destination
was provided in a hostname format (and not IP).

The current stable version of Tor (0.2.2.x) still supports Exit
enclaves. The new versions of Tor
(> 0.2.3.x) use a new descriptor format (microdescriptors) allow relays
to specify an Exit Enclave
policy, but clients will not use it, therefore voiding the purpose of
exit enclaving.

I believe there is the need for something similar to Tor Exit Enclaving
and the closest thing I see
fit these requirements are Tor Encrypted Services
(https://gitweb.torproject.org/torspec.git/blob/HEAD:/proposals/ideas/xxx-encrypted-services.txt).

Encrypted Services (EC), are basically regular Tor Hidden Services, that
do not provide anonymity
for the server and gain a better performance because of this (they have
a one-hop circuit
to the RV and IP).

The problem with making Encrypted services work to replace Exit Enclaves
is that the client needs
to have a way to understand that their destination is running also as an
Encrypted Service.

In this very high level overview I don't go into very much detail of how
this system will actually
work, but I hope it will prompt some discussion on the matter.

I think this can be achieved mainly in 3 ways.

1) The client already knows all of the EC's
2) The client looks up if a destination is an EC when trying to connect
to it
3) The final hop looks up if the destination is an EC

These all have some drawbacks:

In 1) the client needs to download the full list of EC's, therefore if
the number of EC's get's
very large it will take clients much more to bootstrap and they will
need to store more data.
The good thing of this though is that the speed of connections would be
as fast as they
are at the moment as it does not require any extra connections.

In 2) the clients needs to complete an extra round-trip for every
connection. I don't think
this is a valid solution as it would degrade the quality of connections
for every user.

In 3) the final hop would do along side a normal A lookup for hostnames
a CNAME lookup (
or another special field). If it finds that such a lookup returns a
.onion address instead of
returning a RELAY_CONNECTED cell it will return a ENCRYPTED_SERVICE cell
containing
the .onion address of the target ES.

The client will then cache this address and connect to it.

This approach adds a little bit of overhead (since two DNS lookups need
to be made),
but it is still faster than 2).

It suffers from the issue of the exit node could spoof the .onion
address and redirect
the user to a malicious .onion address. This is quite a tough problem
that I am still
unsure how it could be solved. If we have support for DNSSec this issue
could be mitigated.

I would love some feedback on this topic.


- Art.




___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Alternatives to Tor Exit Enclaves

[Arturo Filastò]

On 4/18/12 5:33 PM, Andrew Clausen wrote:
> Do .exit addresses already do what you had in mind?  For example, if
> you add "AllowDotExit 1" to your torrc, you can type an address like
> this
>
>
No, .exit notation is a bad idea because it allows people
to force you to exit through a particular exit node of their
choosing. For example I can place a  tag on a website
and de-anonymize every user by getting them to go through my
address.

- Art.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Alternatives to Tor Exit Enclaves

[Arturo Filastò]

On 4/19/12 3:42 AM, Andrew Lewman wrote:
> On Wed, 18 Apr 2012 17:28:42 -0400
> Arturo Filastò  wrote:
>
>> The current stable version of Tor (0.2.2.x) still supports Exit
>> enclaves. The new versions of Tor
>> (> 0.2.3.x) use a new descriptor format (microdescriptors) allow
>> relays to specify an Exit Enclave
>> policy, but clients will not use it, therefore voiding the purpose of
>> exit enclaving.
> Is there a plan to re-enable clients to use exit enclaves?

The overall consensus that I got from discussing this in IRC is
that Exit Enclaves where never thought of very well and they already
did not have all the properties that we wanted.

Re-enabling clients to use exit enclaves requires changes to the
microdescriptor format, and since they have issues we should not
encourage their use.


- Art.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Arturo's June Status Report

[Arturo Filastò]

# Summary

I worked mainly on documentation and specification tasks related to 
OONI. Did a bit of travelling and a bit of coding too.


# Documentation/Design related

The wiki is starting to look good. Most of the basic analysis required 
for moving on to implementing
OONI is in place. We still need to have some more discussion on a few 
design choices, but the overall

concept is in place.

In particular we have finished the analysis of Tests.

I created templates for the analysis of Censorship detection tools 
(https://trac.torproject.org/projects/tor/wiki/doc/OONI/CensorshipDetectionTools/Template)


did the analysis of Neubot: 
https://trac.torproject.org/projects/tor/wiki/doc/OONI/CensorshipDetectionTools/NeuBot
and of Herdict: 
https://trac.torproject.org/projects/tor/wiki/doc/OONI/CensorshipDetectionTools/Herdict


We created the methodology for specifying and writing OONI probe tests:
https://trac.torproject.org/projects/tor/wiki/doc/OONI/TestWritingMethodology

created a module view and a component and connector view and specified 
the overall
architecture of OONI: 
https://trac.torproject.org/projects/tor/wiki/doc/OONI/Architecture


I started defining the API for the backend system in particular the 
reporting mechanism:

https://trac.torproject.org/projects/tor/wiki/doc/OONI/Backend.

In the past days had some discussions with jake and isis if we should be 
using HTTP for reporting.
Jake suggested that we use rsync instead of HTTP. I will detail in a 
following mail the pro and cotra

of one solution over the other.

We finished specifying the basic set of Test for OONI here: 
https://trac.torproject.org/projects/tor/wiki/doc/OONI/Tests


The data collection scheme has been defined here: 
https://trac.torproject.org/projects/tor/wiki/doc/OONI/DataCollection.


I got a few good tips from Nick on the topic of design and specification 
of software. What I took from it is that in the end the design that you 
make will not be ultimate and you will end up changing it. For this 
reason it's good to start implementing some of the tests and that will 
allow you to understand what is the abstraction that fits all of them.
He used a very nice analogy, it's like having a set of points in N 
dimensional space and by designing an abstraction for it you are trying 
to find an hypercube that fits all of them (I may have a bit 
re-elaborated this).


# Coding

Implemented an HTTP utility library for writing HTTP based OONI tests 
and implemented a few of the using

such library.

Wrote a OONI scapy twisted library that replaces scapys send and receive 
functions with some based on

twisted. Wrote an OONI Test protocol for it.

Implemented based on OONI a test written by phw that does packet 
mutation to bissect censorship fingerprints.


Misc. code refactoring.

# Travel/Advocacy

Travelled to Madrid to do a talk on Tor and GlobaLeaks. Met there with 
Jillian York.


Went to hackmeeting in l'Aquila and did a talk about Tor and censorship 
( http://www.slideshare.net/hellais/tor-censorship-2012-ooni). I believe 
this talk went very well and everybody was very excited about Tor at the 
end of it.


# What I plan to do next

Work on drawing these N dots, by implementing test for OONI. I will 
finish one of the first fully client server tests based on OONI called 
b0wser (the server part is almost done). Figure out what should be done 
with the backend and start having a test backend deployment on ooni.nu.


- Art.

___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Onionoo in Python

[Arturo Filastò]

On 07/10/2012 05:36 PM, Norman Danner wrote:
Based on a quick look, it seems like Cyclone provides a slightly nicer 
way to specify how to handle the various requests than does a plain 
Twisted web application.  Are there any other advantages to using 
Cyclone as opposed to plain Twisted?


To me, there is a trade-off:

Cyclone+Twisted:  slightly nicer way to write the web application.  
More dependencies for Onionoo.


Plain Twisted:  not quite as nice a way to write the application.  
Fewer dependencies for Onionoo.




Maintaining a REST API in pure twisted is *very* painful. The syntax for 
defining the API in twisted
is very hackish and unclean. Using cyclone makes your code more 
structured and cleaner.
The hackiness is very clear if you are interested in doing things like 
matching to a certain functionality
based on a regexp. Cyclone makes this easy as it's based on tornado that 
was designed with the creation

of an API for friendfeed in mind.

Also, the documentation for Cyclone seems...minimal.  It might be 
straightforward for someone used to using Tornado, but that doesn't 
describe us over here...


The documentation of cyclone is the documentation of tornado. Cyclone is 
basically just a fork of it

so anything that you are used to doing in tornado you can do in cyclone.


- Art.

___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] [OONI] Designing the OONI Backend (OONIB). RESTful API vs rsynch

[Arturo Filastò]

I would like to follow up on the discussion we had in Florence on some
design choices behind OONIB.

In particular the most controversy was around using HTTP or rsync.

Before discussion the pro and contra about one choice over the other it
would be useful to frame
what are exactly the requirements for OONIB.

# What is OONIB

OONIB is the backend of OONI. It will run mainly on one centralized
machine and may in
a later stage run distributed across multiple ones. Currently we have
not though of how
to make it scale to being distributed, so we will look at it as if it
were running only
on one central machine.

It will be responsible for:

* Reporting
  a) Collecting reports from OONIProbes. Such reports are the results of
tests.
  b) Collecting reports from third party censorship measurement tools
(e.x. Bismark, NeuBot, etc.)

* Assistance in test running
  Certain OONI Tests require to have a backend component (e.x. b0wser).
On OONIB we will have
  the serverside component that will assist us in running the test.
 
  Note: Certain tests require the server to make connections to the
client. This means that the
  client will need to request the server to probe them.

* Control Channel Signaling
  This is required for making some measurements to verify that the data
received by the backend
  specific to the test matches with the one sent by the client.
 
# What properties we would like it to have
note: these are not ordered.

* Efficient even over high latency networks.

* Ease of integration for third party developers.

* Expandable to support requirements of new tests we develop.

* Anonymous

* Secure

# HTTP and rsych comparison
note: I will not deal with the security aspects of OONIB. We will
suppose to have
an encrypted and authenticated transport (this can be TLS, Tor Hidden
Services,
etc.)

## Rsync:

Pro:

* It supports good compression algorithms

* It's efficient and supports resume

* It does integrity checking on the uploaded files

Contra:

* It's designed only for copying files, this means we can't implement
any more advanced
API like logic. [*]

* It's not supported by many languages (for example in python we only
have an implementation
of the rsync algorithm, not of the protocol [1])

* It's not as commonly used by other application developers that have
similar requirements.

* Painful to do sanitization of the data sent by clients.

* Does not allow bidirectional communication (Request-Response pattern)


[*] I would like to be able to create a session ID for a specific test
and be able to reference
such test ID when interacting with the Test helpers. rsync is one way, I
push data to the server,
but the server cannot signal me back with some data. This largely
impeeds it's usefulness as an
API interface.


## HTTP:
note: I am not necessarily talking only about HTTP, we could use any
other protocol with similar
properties (e.x. SPDY). I will discuss HTTP because it is the one that I
am most familiar with,
but don't

Pro:

* Industry standard for exposing APIs

* Supported natively in most programming languages

* Well understood protocol

* Implementation of sanitization of passed data can be done more easily

* Allows bidirectional communication

* Good support in twisted (what we use as a language for OONIB)

Contra:

* Compression is not enabled by default (we can use gzip compression
with HTTP 1.1), and no compression for
  headers.

* No resume support (this can be implemented on top of HTTP, we could
even implement the rsyc algorithm
on top of HTTP).

* No support for deltas (we can use rsych protocol over HTTP if we
really need this).

I feel like we are a bit comparing apples and oranges here and I don't
see why we could not use
rsync algorithm on top of HTTP. Anyways I would like to get some
feedback as to what we should use
for something that should have the above described properties.

Thoughts?

- Art.

[1] https://github.com/isislovecruft/pyrsync
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] [OONI] Designing the OONI Backend (OONIB). RESTful API vs rsynch

[Arturo Filastò]

On 7/16/12 2:15 AM, Ondrej Mikle wrote:
> On 07/15/2012 02:56 PM, Arturo Filastò wrote:
>> I would like to follow up on the discussion we had in Florence on some
>> design choices behind OONIB.
>>
>> In particular the most controversy was around using HTTP or rsync.
> [...]
>
>> # What properties we would like it to have
>> note: these are not ordered.
>>
>> * Efficient even over high latency networks.
>>
>> * Ease of integration for third party developers.
>>
>> * Expandable to support requirements of new tests we develop.
>>
>> * Anonymous
>>
>> * Secure
> Even though you will probably not end up using this, it may be a good idea to
> know that it exists:
>
> ZeroC Ice - http://www.zeroc.com/ice.html

There are a bunch of very fancy and nice libraries out there to do RPC like
things that support a variety of languages.

Another one I am very fond of is ZeroMQ, but I think we should not start
worrying
about supporting such advanced and scalable libraries at this time.

This is the reason why I stated at the beginning of the requirements
that we should
"so we will look at it as if it were running only on one central machine".

The scalability issues will be dealt with once we have properly defined
the problem
scope. Making the node communication rely on Zero(C|MQ) can be something
that we integrate
without having to require clients to change their behavior.

I think the overall feeling from the responses is that going for
something like an HTTP
RESTful API is what we are looking for.

HTTP is a well understood technology and I have quite some experience in
designing and
building RESTful APIs based on twisted (cyclone) and this is also what
is used in other [1]
projects [2] belonging [3] to the [4] Tor community [5].

Moreover by looking at how the reporting systems of other network
measurements tools
worked [5] we found that almost all of them used an HTTP API to collect
reports [6][7][8][9].
I see this as an indication that such a strategy is the best practice.

For the time being we should go for something simple like this and once
we encounter major
scalability/performance bottlenecks we can quantify them and figure out
what the best
path to a solution may be.

If you were the developer of a censorship detection tool would you like
to have to report
to anything that is not a RESTful HTTPs API?

- Art.

[1] https://github.com/mmaker/APAF
[2] https://github.com/gsathya/pyonionoo
[3] https://github.com/globaleaks/Tor2web-3.0
[4] https://github.com/globaleaks/GLBackend
[5]
https://trac.torproject.org/projects/tor/wiki/doc/OONI/CensorshipDetectionTools
[6]
https://trac.torproject.org/projects/tor/wiki/doc/OONI/CensorshipDetectionTools/Herdict#Howdoesthereportingsystemwork
[7]
https://trac.torproject.org/projects/tor/wiki/doc/OONI/CensorshipDetectionTools/Netalyzr#Howdoesthereportingsystemwork
[8]
https://trac.torproject.org/projects/tor/wiki/doc/OONI/CensorshipDetectionTools/NeuBot#Reportingsystem
[9]
https://trac.torproject.org/projects/tor/wiki/doc/OONI/CensorshipDetectionTools/ProjectBismark#Reportingsystem


___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] [ooniprobe-dev] Summary of OONI hackfest

[Arturo Filastò]

We had a mini hackfest at noisebridge on OONI and this thread is to 
summarize what we discussed.


mct: feel free to add things from your notes that are not detailed here.

One of the problems that we mostly focused our attention on was how to 
detect censorship on HTTP when the user is presented with a block page. 
The reason for focusing on this is that censorship over HTTP is the most 
prevalent form.


# Detecting censorship in web sites

Detecting that the presented web page is not the expected site, but is a 
censorship page turns out to be a non trivial task, especially when you 
are dealing with web pages that have dynamic content. This problem is 
detailed in this trac ticket: 
https://trac.torproject.org/projects/tor/ticket/6180.


We ended up dividing the possible approaches into two macro categories: 
Statistical and Heuristic [1]


One of the properties we would like our classifier to have is that that 
a user should be able to detect that the presented page is the block 
page by just having a copy of OONI and some data that can be given to 
them over a non censored channel (sneakernet for example).


## DOMClass, a eigenvalue based statistical classifier

What we ended up implementing is a classifier that considers the DOM 
structure of the web page. We can easily build such a database of the 
DOM structure features of the websites we are interested in analyzing 
and ship these to users.


This is how the algorithm works:

  This classifier uses the DOM structure of a website to determine 
how similar

  the two sites are.
  The procedure we use is the following:
  * First we parse all the DOM tree of the web page and we 
build a list of
TAG parent child relationships (ex. 
 =>

(html, a), (a, b), (html, c)).

  * We then use this information to build a matrix (M) where 
m[i][j] = P(of
transitioning from tag[i] to tag[j]). If tag[i] does not 
exists P() = 0.

Note: M is a square matrix that is number_of_tags wide.

  * We then calculate the eigenvectors (v_i) and eigenvalues 
(e) of M.


  * The corelation between page A and B is given via this formula:
correlation = dot_product(e_A, e_B), where e_A and e_B are
resepectively the eigenvalues for the probability matrix A 
and the

probability matrix B.

You will note that the Matrix we are computing eigenvalues for somewhat 
resembles a Markov model for transitions between DOM element X to Y.


This algorithm appears to work pretty well even on highly dynamic web 
sites (such as the homepage of news.google.com).


Problems with this algorithm:

* It does not take into account the global position of DOM elements or 
how deeply nested they are. ( 
(a,b),(b,a),(a,b) is equivalent to  
(a,b), (b,a), (a,b))


Looking into other possible solutions to this problem I took a look at 
algorithms that are used to compute graph similarity.


This problem could be solved by using an algorithm that calculates the 
maximum isomorphic subgraph, but this problem is NP-hard. [2]
I am unsure if the computation effort to use algorithms along these 
lines is worth while.


I can't seem to find the notes on the rest of the discussion, perhaps 
mct will integrate with this discussion.


- Art.

[1] https://trac.torproject.org/projects/tor/ticket/6180#comment:3
[2] 
http://en.wikipedia.org/wiki/Maximum_common_subgraph_isomorphism_problem 

___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Automating Bridge Reachability Testing (#6414)

[Arturo Filastò]

On Fri, Oct 12, 2012 at 10:48 PM, Isis  wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA512
>
> Hi Karsten!
>
> Oh sheesh. I did not see it...I will have to figure out why. That is slightly
> worrying.
>
>
> 2) Arturo redesigned the OONI testing framework API again to use a
> completely different structure, which was supposed to be backwards
> compatible and turned out not to be (though I believe that my recent OONI
> commits fixed that).
>However, I have been fighting the framework already, because the main
> scripts in OONI (/ooni/oonicli.py and /ooni/ooniprobe.py) control the
> reactor, and also expect static iterations through single test and single
> control functions for each asset (an asset in this case would be one
> bridge address).

What do you mean exactly by the fact that your tests control the
reactor? The twisted reactor is an event loop and it should be started
only once and stopped only once. Can you show me the piece of code
that you wrote that "controls it"?
There is a proper way to do more or less anything, but some things
require gotchas.

> The bridge testing is rather dynamic (I would like it to
> be able to evaluate an approximate danger level to running the next test)
> and so the framework is kind of troublesome. Also, because the framework
> handles calling the reactor (in Twisted, the reactor is a sort of event
> scheduler), and it also expects a rather linear progression of
> defer.Deferreds (in Twisted, those are standin objects which execute
> callbacks when they get results from some previous deferred/callback), it
> would be nicer if I had full control of these myself without needing to
> hack around the parent scripts.

The twisted reactor is not like a scheduler. It is an event loop. This
means that you register deferred with it (promises that your function
will eventually return) and when you fire the event for the fact that
your operation has concluded it will run the callbacks.
This pattern is common to (I think) all event loops (like libevent for example).

The neat thing though is that the new API actually supports
"automatically" blocking operations transparently. This means that if
you run a test that is blocking and you don't want to fiddle around
with twisted deferreds you can do it none the less.
Check out the documentation for it https://ooni.readthedocs.org and
for more details on what is possible you can do most of what is
possible with twisted trial:
http://twistedmatrix.com/documents/current/api/twisted.trial.html.

If you give me some examples of where you are encountering
difficulties I can help you out.

> I think it's wise that OONI deals with
> these things for the testwriter in most cases, because the testwriter
> shouldn't be expected to be an expert in using Twisted. However, I also
> think that, in the long term, OONI shouldn't prohibit people who know what
> they are doing or are doing odd things from being able to do so.

Being a framework I believe this is not the ideal path. A framework is
made to impose on you certain design decisions that you do not have to
make. There is a right way to use such frameworks and if you don't use
them as they are designed you will obviously have some issues.

Could you provide me with examples of what you are trying to do and
you are not able to do or experiencing problems with?

Whatever you need to do should be possible and if it is not possible
this functionality should be exposed to whoever will use OONI in the
future.

>As a result, I've decided (for now), to use bits are parts of the OONI
> code before the recent refactoring, and later (after the deliverable) I
> will work on adding flags to OONI to give the test script full control of
> the reactor and deferreds, as well as evaluating whether or not the bridge
> test is even compatible with the new API. I do not want to get caught up
> in dealing with this right now, I just want to have it all working and
> deployable in a way that I know will work.

I still do not understand exactly what you mean by "full control of
the reactor". I am very keen in learning what you are trying to do
though and I bet there is a way to do it, since the new OONI is just
an extension of twisted trial unittesting framework to support inputs
and reporting with the OONI format.

- Art.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Brainstorming a Tor censorship analysis tool

[Arturo Filastò]

On Dec 18, 2012, at 8:07 PM, Philipp Winter  wrote:

> Hi there,
> 
> Deliverable 6 for sponsor Z says:
> 
>> 6. Start a tool that a censored developer can run to discover why their Tor 
>> is
>> failing to connect: brainstorm a list of "things to check", and sort them by
>> how useful they'd be to check / how hard they'd be to build. (#7137)
> 
> The deliverable is due on Feb. 28, 2013 so we should get started.
> 
> Some background about the deliverable:
> The reason for this project is that debugging possible censorship events is
> tedious right now. We often have no access to machines in censoring countries
> and we are dependent on users creating packet dumps for us. This tool should
> speed up and automate this process to some extent. Censored users should run 
> it
> and the tool should then collect data which should then somehow reach us.
> 
> I created the following wiki page which should contain all the necessary
> information:
> https://censorshipwiki.torproject.org/TorCensorshipAnalyzer
> 
> Please add/modify stuff and share your opinion. Since there is quite some
> overlap with OONI, it would be great if the OONI people could give feedback.
> 

I believe you should be using ooniprobe to build a the tests you are interested
in building, or you may at least be interested in looking at our code to see 
how to do the things you are interested in doing.

The main points where ooniprobe would be of use to you (now) are:

# Standard reporting format

All ooniprobe tests share a common base format depending on the test template 
your test is based on.

I recommend you look at the Test Writing tutorial to get an idea of how this
looks like:
https://ooni.torproject.org/docs/writing_tests.html

# Collection of packet captures

When you run an ooniprobe test and you have set your ooniprobe.conf file to 
"includepcap: true" then you will collect a full pcap of what has happened on 
the probes network during the test run.

Note: This requires the test to be run as root and will include *all* the 
network
traffic during the testing session (i.e. if the user is looking at their 
favorite 
kitten website while running the test, such data will be in the pcap)

# Collection of packet captures specific to the sent and received packets

When you run a ooniprobe test that inherits from the scapy test template 
(https://ooni.torproject.org/docs/api/ooni.templates.html#module-ooni.templates.scapyt)
 
the packets sent and received (i.e. that are answers to the packet(s) sent) 
will be 
captured.

When configured to not include the probe IP address, source IP of sent packets 
and dst IP of received packets is replaced with 127.0.0.1. (warning: if the IP 
address of the probe is present in some other parts of the packet it will not 
get 
stripped, for example if it's present in the ICMP citation)

# Reporting system

Currently we only support collection of YAML formatted reports (that means not 
.pcap files) and only via Tor Hidden Services.

Extending it to support reporting via HTTP(s) should be trivial and is a 
feature 
that we have already received a request for.

Adding support for collecting also .pcaps also probably does not require that 
much 
amount of time and is something that will happen in the near future.

# Things to come

ooniprobe will soon expose a HTTP based API that binds to localhost that can 
then
be (optionally) exposed as a Tor Hidden Service. Such API will allow 
researchers to
connect to a probe and run some tests and will allow us to build a JS/HTML5 
client
interface to allow users to select which tests to run and monitor the status of 
running
tests.

More details here:
https://ooni.torproject.org/docs/architecture.html#ooniprobe-api

For a birds-eye view of the project see:
https://ooni.torproject.org/docs/architecture.html

Even if you do not end up using ooniprobe for developing your system today, I 
highly encourage you to use the libraries that we are using so that in the 
future we can find a way to integrate code from each others projects.

The main libraries that we are using are:

* Twisted http://twistedmatrix.com
* Scapy http://www.secdev.org/projects/scapy/
* txtorcon https://github.com/meejah/txtorcon


~ Art.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Twisted-based Tor client performance measurement tool

[Arturo Filastò]

On Jan 22, 2013, at 8:32 AM, meejah  wrote:

>> 3) periodically run one or more tests which can be:
>> 3.1) an HTTP GET request over Tor to its own web server,
>> 3.2) an HTTP POST request to measure upload speed,
>> 3.3) a GET or POST request to a locally running hidden service,
> 
> You'd need https://github.com/ln5/twisted-socks for a SOCKS client for
> this (looks like V4 only?). There are some other ones floating around
> out there, too, but nothing in core Twisted (as far as I
> recall). Ah, like https://twistedmatrix.com/trac/ticket/1330
> 

What I am currently using as a SOCKS Client these days is this one:
https://github.com/hellais/txsocksx.

It's got decent unittest coverage and some examples on how to use it.

It supports v4 and v5.

We've been using it quite heavily in ooniprobe and it appears to be doing it's
job well.

~ A.


___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] [ooni] transparent proxy detection

[Arturo Filastò]

On Mar 6, 2013, at 3:58 PM, Sam Smith  wrote:

> 
> Hey all,
> 

Hello :)

> I'm looking at detection and identification of some transparent proxies (of 
> different kinds). Has anyone already got tests that effectively but 
> generically merge manipulation/http_host and experimental/squid.py ?
> 
> 

I am not sure what you mean exactly by "effectively but generically merge 
manipulation/http_host and experimental/squid.py". 

There is a currently unmerged branch of mine that add some features to the HTTP 
Host test. Specifically it tests for some HTTP Proxy bypassing techniques 
(prepending a \n to the GET method and postfixing the Host header field with a 
\t), it checks if filtering is happening also on subdomains, if it's done based 
on fuzzy matching.
You can find it here: https://github.com/TheTorProject/ooni-probe/pull/51 (this 
is the branch: https://github.com/hellais/ooni-probe/tree/experimental-tests).

The tests that are most relevant to your endeavor are:

* nettests/manipulation/http_invalid_request_line.py (for example bluecoat 
transparent HTTP proxies are know to crash with some, but not all of these 
requests)

* nettests/manipulation/http_host.py, with the added patches in my 
experimental-tests branch it will also attempt some filter bypassing techniques 
that can be useful to understand what the device in question is.

* nettests/manipulation/http_header_field_manipulation.py, when pointing this 
towards a oonib test helper you will understand if the device in question is 
adding extra HTTP Headers or performing some other HTTP header mangling.


> 
> Also, I'm testing on a debian (6.0) VM running in a parallels 8 on OS X.  
> Ooni successfully runs tests once , and then requires a debian reboot in 
> order to run again (especially http_host, it was less repeatable with 
> others). I have no idea what's going on, and it'll be a pain to debug, but 
> it's the sort of thing someone's probably interested in for other purposes ;)
> 


That is quite peculiar behavior. What exactly is it that leads you to need to 
reboot? Are you seeing some errors after the first run?

What version of ooniprobe are you running?

Feel free to drop by on IRC: irc.oftc.net #ooni.

~ Art.

___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] [ooni] proposed specification for ooni backend component

[Arturo Filastò]

Greetings!

It would be great to get some feedback on the specification of oonib that is
the backend component of ooniprobe.

As later said, not everything described here is implemented, but this document
details what oonib should become.

Moreover if there is something that you believe we should pay special attention
to in when implementing this it would be of great use to know.

If you wish to modify this spec directly you can send us a pull request on 
github:

https://github.com/TheTorProject/ooni-spec

-

# oonib specification

* version: 0.1
* date: 2013-03-04
* author: Arturo Filastò

This document aims at providing a functional specification of oonib. At the
time of writing this document not all parts are fully implemented, though the
application interface to oonib is.

# 1.0 System overview

oonib is the backend component of ooni. It is responsible for:

  * Collecting the results of tests from ooni-probes (Collector).

  * Exposing a set of services that are needed for allowing ooni-probes to
perform their tests (Test Helpers).

# 2.0 Collector

## 2.1 System overview

The oonib collector exposes a JSON RPC like HTTP interface and allows probes to
submit the results of their measurements. Once probe measurement is complete
the test result is published as an ooni test report in the YAML format.

The oonib collector shall be exposed as a Tor Hidden Service and as a HTTPS
service.

## 2.2 Theat model

The collector shall provide end-to-end between the probe and the oonib
collector.

A malicious actor should not be able to update test results that they have not
created.

It is outside of the scope of the oonib collector to provide blocking
resistance or to conceal to a passive network observer the fact that they are
communicating to a collector.
Such properties are to be provided by other software components, for example
using Tor or obfsproxy.

## 2.3 Test result submission interface

Unless otherwise stated all of the network operations below can be performed
either via HTTPS or HTTPO (HTTP over Tor Hidden Service).

Note: we will eventually want to migrate over to using YAML instead of JSON as
a data exchange format. Not doing so adds unnecessary overhead in including
YAML data inside of JSON data.

### 2.3.1 Create a new report

When a probe starts a test they will *create* a new report with the oonib
collector backend.
The HTTP request it performs is:

`POST /report`

{

 'software_name':
`string` the name of the software that is creating a report (ex. 
"ooni-probe")

 'software_version':
`string` the version of the software creating the report (ex. 
"0.0.10-beta")

 'probe_asn':
`string` the Authonomous System Number of the network the test is
  related to prefixed by "AS" (ex. "AS1234")

 'test_name':
`string` the name of the test performing the network measurement. In
  the case of ooni-probe this is the test filename without the ".py"
  extension.

 'test_version':
`string` the version of the test peforming the network measurement.

 'content':
(optional) `string` it is optionally possible to create a report with
  already some data inside of it.

 'probe_ip':
(optional) `string` the IP Address of the ooniprobe client. When the
  test requires a test_helper the probe should inform oonib of it's IP
  address. We need to know this since we are not sure if the probe is
  accessing the report collector via Tor or not.
 }

The server will respond with a report identifier that will then allow the probe
to update the report and the version of the backend software like follows:

{

  'backend_version':
`string` containing the version of the backend

  'report_id':
`string` report identifier of the format detailed below.

  'test_helper_address':
`string` the address of a test helper for the requested test.

}

The report identifier allows the probe to update the report and it will be
contructed as follows:

  ISO 8601 timestamp + '_' + probe ASN + '_' + 50 mixed lowercase uppercase 
characters

A report identifier can be for example:

  1912-06-23T101234Z_AS1234_ihvhaeZDcBpDwYTbmdyqZSLCDNuJSQuoOdJMciiQWwAWUKJwmR

Note:
The report identifier should be at least 256 bits and generated by means of a
CSPRNG.

Client implementation notes:
Probes should not expect the report identifier to be in a particular format as
the report id may be changed in the future.

### 2.3.2 Update a report

Once the probe has a report ID they will be able to add test related content to
the report by referencing it by id:

`PUT /report`

{

'report_id':
  `string` the report identifier

'conten

Re: [tor-dev] Atlas/Compass GSoC 2013

[Arturo Filastò]

On Apr 23, 2013, at 11:26 PM, Newk  wrote:

> Hi, 
> 
> I'm Nikola, 1st year student in the university in Skopje, Macedonia. Around 6 
> months ago I've put a tor relay at home - one of only two running in the 
> whole country (the other one is placed in the local hackerspace, also the 
> only one in the country). Being interested in internet anonymity and network 
> security, I've started reading about how tor works and how I can make a 
> contribution in the project.
> 
> I would like to work on Atlas and/or Compass (even Vidalia or Arm), because 
> as a tor relay owner, I've met some points I would like to improve. 
> 
> Any thoughts?
> 
> Thanks,
> Nikola
> 
> P.S.
> As a late blooming activist, I would like to contribute to this project even 
> if I don't be selected for this years GSoC program. The reason being that in 
> couple of years or less, you will see my country as a place where democracy 
> quietly ceased to exist. And tor will be the only way reaching to the 
> outside. On this note, I'd like to add that my goal is not only contributing 
> by improving tor, but also making a shout to the free-minded 


Hi Nikola,

Good to hear your interest in the Tor GSoC. 

Regarding Atlas and Compass there is a project that we have not put up as a 
GSoC, but if you are interested gsathya or me could probably mentor you.

The most pressing topic regarding Atlas and Compass is getting the two 
platforms merged. The relevant ticket is this one: 
https://trac.torproject.org/projects/tor/ticket/6682.

What is your skillset? Are you familiar with Angular.js or other relevant web 
technologies?

You can find me and gsathya on IRC on the channels #tor-dev and #tor on 
irc.oftc.net.

~ Art.

___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] GSOC - Internet Censorship Virtual Machine Based Simulator

[Arturo Filastò]

On May 1, 2013, at 1:59 AM, Claudiu Perta  wrote:

> Hi Tor team,

Greetings Claudiu!

>  
> I'm Claudiu, a second year PhD student in CS at Sapienza University (Rome, 
> Italy) and I would like to work on your GSOC idea to build a VB-based 
> simulator for OONI. My main research topics are network neutrality and 
> broadband measurement and I'm currently working on a project on traffic 
> shaping detection. I also partecipated to last year GSOC when I developed in 
> Python a server selection mechanism for M-Lab 
> (https://code.google.com/p/m-lab/source/browse/?repo=ns).
> 

This is very relevant to the OONI areas of research.

> I've worked in the past with Netkit 
> (http://wiki.netkit.org/index.php/Main_Page) and I think it meets all the 
> requirements of this GSOC project.  Netkit is an open source project,based on 
> user-mode linux[1], a linux kernel that can be executed as a user  process on 
> a standard linux box. In particular, each virtual machine in Netkit can be 
> configured to play the role of a regular host, of a router, or even of a 
> switch.  For this project, each test could be set up as a ‘netkit lab’, which 
> is a set of preconfigured virtual machines that can be started and halted 
> together. For example, one virtual machine running ooniprobe could be 
> connected to the external world through a second virtual machine that 
> implements various censorship policies, specific to each test.
> 

Netkit does seem like a good choice for doing this. Making this work on the 
basis of Netkit would probably be a matter of writing a set of NetML 
descriptions for the various censorships we are interested in simulating.

BTW I am currently in Rome and would be up for meeting up someday to perhaps 
discuss this in person further or hack on it together.

~ Art.

___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Local onionoo cache

[Arturo Filastò]

On May 13, 2013, at 7:15 PM, David Fifield  wrote:

> Thank you for taking a look.
> 
> On Mon, May 13, 2013 at 08:58:27AM +0200, Karsten Loesing wrote:
>> I'm not sure, but Tor2web might do something similar.  From Onionoo's
>> project page: "Tor2web is a web proxy to Tor Hidden Services. It uses
>> Onionoo to get the list of currently running Tor Exits to detect if the
>> client is a Tor user and if so redirect them to the .onion address."
> 
> If I read this right, Tor2web is doing it not only for exits, but for
> all relays:
> 
> https://github.com/globaleaks/Tor2web-3.0/blob/c6e26b35e83fd897f9c4f9cb6787eb0132d8a9d0/tor2web/utils/lists.py#L230
> 


Yeah we are doing this because downloading all the exit lists is too much. If 
there was a better way to do this we would be very useful to us.

We only need the exit list, but we consider any relay as a possible exit. This 
means that stuff like a relay that exits  through a different address will not 
be detected as coming from Tor (this is something torbel accounts for).

These are the relevant tickets on the topic:

https://trac.torproject.org/projects/tor/ticket/6488

https://github.com/globaleaks/Tor2web-3.0/issues/10

https://github.com/globaleaks/Tor2web-3.0/issues/85

~ Art.


___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Censorship Simulator Virtual Machine GSoC

[Arturo Filastò]

Hi Johannes,

I wanted to introduce you and your GSoC project to the list.

The project is about building a virtual machine based censorship simulator. The 
goal is to have a way to test if the ooniprobe tests are working properly by 
simulating the network of a censor.

Our goal is to actually run the software that a censor would be running (e.x. 
squid, bluecoat proxy sg va, etc.) and detect it's presence in an as real as 
possible environment.

Johannes and Klaudiu analyzed a bit of the possible technological solutions to 
making this here: 
https://trac.torproject.org/projects/tor/wiki/doc/OONI/CensorshipSimulator.

I think that the outcome is that we will be using Vagrant, meaning that this 
simulator could then be run on OSX, Linux and Windows.

The kind of networks we will be creating are those where:

* There is a transparent HTTP proxy
* There is NAT
* Certain sites are blocked thanks to HTTP headers
* RST packets are sent when connecting to certain hosts
* DNS based censorship is present

If you have some ideas of things you would like to see in this, add ideas to 
this ticket: https://trac.torproject.org/projects/tor/ticket/7233, or create a 
page in the trac wiki nested under 
https://trac.torproject.org/projects/tor/wiki/doc/OONI/CensorshipSimulator.

~ Art.

___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Running Atlas Application

[Arturo Filastò]

On Jun 13, 2013, at 5:22 PM, abhiram  wrote:

> Hello,
> 
> I am trying to get the Atlas application to run. After installing the 
> required python packages, I keep getting this error.
> 
> Traceback (most recent call last):
>   File "run.py", line 34, in 
> from tornado.util import b, bytes_type, import_object, ObjectDict
> ImportError: cannot import name b
> 
> Its seems like a version problem with tornado. If so, what version should I 
> be using.
> 
> I am running python 2.7.3 and tornado 3.0.3.
> 


>From the looks of it that appears to be an incompatibility with the new 
>version of tornado.

It is not required, though, to run atlas. All that that script does is serve 
the static files that are in the atlas repo.

We should actually replace that script with a simple python script that does 
something like:

python -m SimpleHTTPServer 8080

Would you like to hack that together and submit a patch for it?


~ Art.

___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] ooniprobe 1.0.2 released

[Arturo Filastò]

We are pleased to announce that ooniprobe 1.0.2 has been released.

It includes a series of bugfixes and improvements including:

* Add ooniprobe manpage.

* Fix various security issues raised by the least authority audit.

* Add a test that checks for Tor bridge reachability.

* Record the IP address of the exit node being used in torified requests.

* Captive portal test now uses the ooni-probe test templates.

* Have better test naming consistency.

You may download it from pypi here:
https://pypi.python.org/pypi/ooniprobe/1.0.2

Soon the debian package for it will also be updated.

Have fun,

~ Art.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Rescheduling of next OONI development meeting

[Arturo Filastò]

The next weekly OONI development meeting is supposed to happen on the
15th of August.

Since the this day is holiday in a lot of countries I am going to
propose we skip next weeks meeting and move it to the week after: the
22nd of August.

The time will be the same 5 PM CET (3 PM UTC).

The upcoming weeks the meeting should happen every Friday as usual.

Happy holidays,

~ Art.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Feedback on new OONI test deck format

[Arturo Filastò]

Hi all,

We have been discussing lately some changes that we would like to make
to the ooni-probe test deck format [1] and would like to have some
feedback on what we have come up with.

For those of you not familiar with ooni-probe, a test deck is basically
a way of telling it "Run this list of OONI tests with these inputs and
by the way be sure you also set these options properly when doing so".

The previous deck format involved a lot of boilerplate and did not make
it possible to bake inputs into the deck itself. This new format is
supposed to overcome some of the limitations of the old design and we
hope that a major redesign will not be needed in the near future.

The specification can be found in git [3], but to facilitate commenting
I am also going to paste it into this email. All feedback is greatly
appreciated.

-- BEGIN SPEC --

# Test deck specification

* version: 0.1.0
* date: 2014-08-13
* author: Alejandro López (kudrom), Arturo Filastò

# 0. Terminology
Analyst: The person who writes a deck.

Collector: A machine running the ooni-backend that collects the reports
generated by the execution of ooni-probe.

Nettest: A test whose execution by ooniprobe creates a report sent later
to the
collector if provided.

Ooni: Open observatory of network interference.

Ooni-probe: The client side of ooni used to execute a set of nettests.

Tester: The person who executes ooni-probe.

YAML: A human-readable data serialization format.

# 1. Rationale

To ease the execution of nettests, the ooni developers came up with the
idea of
a container that would allow a tester to easily execute a bunch of
configured
nettests.That container was called a deck.

This way, an analyst interested in a particular behaviour would write a
deck;
then, she would distribute that deck to every tester interested in the
ongoing
analysis. Finally, the tester would execute ooni-probe with that deck to
properly create and send the nettests' reports.

Unfortunately, right now both the deck format and ooni-probe don't allow
that
level of automation. The way we want to solve this situation is by writing a
new spec for the deck format that would allow us to write, with great
confidence, the given functionality to ooni-probe. The document that you're
reading is that spec.

# 2. Goals

Allow an analyst to reuse well written and tested nettests in an easy way.

Allow a tester to execute easily a battery of nettests with a complex
configuration.

Grant privacy of the tester.

# 3. The data format

Every deck is a yaml file composed of two major sections: the header and the
body.

The header is a dictionary that provides all the shared and global
configuration of every nettest included in the deck. Its main purpose is to
reduce boilerplate by letting the analyst express common behaviour in one
section instead of in every nettest execution. The ooni-probe options
allowed
in this section are:

1. collector: Address of the collector of test results.
2. bouncer: Address of the bouncer for test helpers.
3. annotations: Annotate the report with a key:value[, key:value] format.
4. no-collector: Disable the collector. (FLAG)
5. no-geoip: Disable the geoip support. (FLAG)

Every option of the header section is a dictionary's key except that labeled
with a FLAG, which are members of a list called flags. So for example a
valid
header section is the following:

```
header:
  collector: 'http://localhost'
  annotations:
  key1: value1
  key2: value2
  flags:
  - no-collector
```

The deck header can also contain metadata associated to the test deck. The
possible fields are:

1. name: The name of the test deck.
2. description: A short description of the test deck.
3. author: The author of the test deck.
4. version: A version number for the test deck.
5. requires-root: A flag to indicate that the test deck requires root to
run. (FLAG)
6. requires-tor: A flag to indicate that the test deck requires tor to
run. (FLAG)

The header may also be omitted.

The body is a list composed of one element per nettest execution. Every
nettest
execution is a dictionary composed of the following three keys:

1. nettest: name of the nettest to execute (MANDATORY)
2. local_options: local_options of the test execution (OPTIONAL)
3. global_options: global options of the test execution (OPTIONAL)

In the same way that with the header, every option can be a member of a flag
list if that options doesn't have any arguments. A valid body can be:

```
body:
- nettest: manipulation/http_request
  local_options:
url: 'http://torproject.org'
  global_options:
collector: 'http://localhost'
flags:
- no-geoip
- nettest: manipulation/captiveportal
```

All file paths must be relative and they must start with "deck/" if they are
referring to files contained inside of the test deck. They must start with
"http(o|s)://" if they are referring to files to be downloaded via Tor.

All other f

[tor-dev] ooniprobe 1.1.0 is out!

[Arturo Filastò]

Hello beautiful humans!

We are very pleased to announce that a new version of ooniprobe is now out.

In this new release of ooniprobe we have added a new command line tool
for listing the reports that have not been published to a collector and
that allows the probe operator to choose which ones they would like to
upload.

We have also made some privacy improvements to the reports (we will
sanitize all things that may look like file paths) and added metadata
associated with the maxmind database being used by the probe operator.

Here is a more detailed list of what has been done:

* Annotate on disk which reports we have submitted and which ones we
  have not:
  https://trac.torproject.org/projects/tor/ticket/11860

* Add tool called oonireport for publishing unpublished ooniprobe
  reports to a
  collector: https://trac.torproject.org/projects/tor/ticket/11862

* Probe Report does not leak filepaths anymore:
  https://trac.torproject.org/projects/tor/ticket/12706

* Reports now include version information about the maxmind database
  being used: https://trac.torproject.org/projects/tor/ticket/12771

* We now do integrity checks on the ooniprobe.conf file so that we
  don't start the tool if the config file is missing some settings
  or is not consistent:
  https://trac.torproject.org/projects/tor/ticket/11983
  (thanks to Alejandro López (kudrom))

* Improvements have been made to the sniffer subsystem
  (thanks to Alejandro López (kudrom))

* Fix the multi protocol traceroute test.
  https://trac.torproject.org/projects/tor/ticket/12883

Minor bug fixes:

* Fix dns_spoof test (by kudrom)
  https://trac.torproject.org/projects/tor/ticket/12486

* ooni might not look at requiresTor:
  https://trac.torproject.org/projects/tor/ticket/11858

* ooni spits out gobs of tracebacks if Tor is not running and the OONI
  config says it will be:
  https://trac.torproject.org/projects/tor/ticket/11859

* The README for ooni-probe should mention the bugtracker and repository
  https://trac.torproject.org/projects/tor/ticket/11980

For the moment you can download the new version from pypi here:
https://pypi.python.org/pypi/ooniprobe.

Soon a there will also be a new debian package for it (thank to Lunar^).

As always feedback is greatly appreciated so please submit any bugs you
may find to our bug tracker:
https://trac.torproject.org/projects/tor/newticket?component=Ooni

Have fun!

~ Art.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Reminder: OONI dev meetings Friday 15:00 UTC (17:00 CET)

[Arturo Filastò]

Hello Oonitarians,

This is a reminder that the weekly dev meeting will be happening on the
#ooni channel on irc.oftc.net this Friday (tomorrow and every following
Friday) at 15:00 UTC (17:00 CET).

Everyone is invited.

~ Art.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] oonibackend 1.1.0 released

[Arturo Filastò]

Hello Oonitarians,

I am pleased to announce that a new version of oonibackend has now been
released.

Here are some of the changes introduced in the new version:

* Make changes to the bouncer API to make it aware of the policy
  of collectors.

* Improve the bouncer API to make it more RESTful.

* Add test helper that can be used to discover the DNS resolver being
  used by the probe.

* Code coverage and unittesting improvements.

* Fix compatibility with latest txtorcon versions.

Starting from this version we will also be uploading the package to
pypi, hence you will be able to install oonibackend with:

pip install oonibackend


https://pypi.python.org/pypi/oonibackend

Have fun,

~ Art.

___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Rescheduling of OONI dev meeting of next week

[Arturo Filastò]

Hello Oonitarians,

Next week (on Friday 12th at 17:00 CET) we were supposed to have our
weekly ooni dev meeting. Some of, though, will be at the OTF summit in
DC and will therefore not be able to attend.

After discussing this at todays weekly meeting we concluded to ask for
feedback on whether we should skip next week and postpone it to next
Friday 19th or if we should do it another day in the week of the 15th.

If you have a strong opinion on doing either of these please let us
know, if not it will by default be postponed to Friday 19th 17:00 CET.

Have fun,

~ Art.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Reminder OONI dev meeting happening tomorrow

[Arturo Filastò]

Hi Oonitarians,

This is a reminder to inform you that tomorrow (Friday 19th of
September) there will the weekly dev meeting.

It happens on IRC (irc.oftc.net) on the channel #ooni at 17:00 CET
(15:00 UTC).

Everyone is invited!

See you tomorrow!

~ Art.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] ooniprobe 1.2.0 and oonibackend 1.1.4 are out!

[Arturo Filastò]

Greetings!

We are very pleased to announce the release of new versions of ooniprobe
and oonibackend.

One of the most interesting new features that are now part of ooniprobe
is the ability to generate test decks for the country you are in a way
that is much easier than before.

As a matter of fact to start contributing useful measurements it's just
a mater of 5 minutes of setup:
https://pypi.python.org/pypi/ooniprobe#ooni-in-5-minutes

Here are some more details of what this release includes:

ooniprobe
=

* Introduce a new tool for generating ooniprobe test decks called
oonideckgen.

* Introduce a new tool for updating resources used for geoip lookup and
deck generation.

* Add support for policy aware bouncing in the client.
  https://trac.torproject.org/projects/tor/ticket/12579


* Various improvements to the bridge_reachability test
 (enable better tor logging and also log obfsproxy)

* Fix backward compatibility with twisted 13.1 and add regression tests
for this.
https://trac.torproject.org/projects/tor/ticket/13139


oonibackend
===

* Add support for specifying the report archive path from config file

* Write tor notice level logs to file

* Fix bug that lead test helpers to not being started

The new packages have been uploaded to pypy and can be found at the
following links:

https://pypi.python.org/pypi/ooniprobe
https://pypi.python.org/pypi/oonibackend

Soon ooniprobe will also be available inside of debian testing thanks to
Lunar^.

Have fun,

~ Art.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Schedule change for ooni dev meeting at 19:00 CET on Thursday October 2 2014

[Arturo Filastò]

Hello Oonitarians,

This is a reminder to inform you that the weekly dev meeting will happen
today at 19:00 CET Thursday October 2.

Note: That the schedule has changed from Friday to Thursday. Adjust your
calendaring software accordingly.

See you!

~ Art.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] [tor-relays] French Flag

[Arturo Filastò]

On 10/3/14, 12:22 PM, Karsten Loesing wrote:
> [Please don't cross-post.  Removed tor-relays@.]
> 
> On 02/10/14 17:00, Sebastian Urbach wrote:
>> Hi,
>>
>> I saw that a lot of services around tor are using a bright blue in the
>> french flag. It should be a dark blue like the one used in the Dutch
>> flag for example ...
> 
> Though I think we shouldn't start modifying flag images ourselves.  The
> better approach would be to report this problem to whoever made these
> flags, ask them to get the colors right, and upgrade.


The source of all flags for Atlas is: http://flag-sprites.com/.

Perhaps it's worth trying to contact them.

~ Art.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] On the visualization of OONI bridge reachability data

[Arturo Filastò]

On 10/6/14, 6:28 PM, Matthew Finkel wrote:
> On Sat, Oct 04, 2014 at 06:27:22PM -0700, M. C. McGrath wrote:
>> These were a few possibilities for visualization that we came up with
>> at the OTF summit (I can send the full notes from that discussion if
>> everyone is okay with it):

Is this something that is different from what is on this pad:
https://pad.riseup.net/p/bridgereachability?

If so please do!

>> - - Timelines (by protocol, pool, country)
>> - - Pie charts for above
>> - - Timeline/graph of time it takes to block bridge from when added to
>> TBB (github parser)
> 
> Similar to the next one, I wonder if showing a map cooresponding to
> this data would also help. At t0, zero countries block the built-in
> bridges, at t1 = only China blocks, at t2 = China + Iran block, at t3 =
> China + Iran + Syria block, t4 = t3 + Turkey, etc. I'm thinking this
> would be nice in addition to the timeline which George sketched (where
> some of the time points are clickable and update the map). I don't
> actually know how difficult this is to make.
> 

I like this idea, though having both the map and the timeline will take
up quite a bit of screen real estate. I think that both of these are
useful graphs to have and linking the two into one giant one probably
does not require that amount of effort so I would go for it.

>> - - Geographic breakdown by region (if enough data points) Could be
>> similar to this map of % of internet users who use Tor by country
>> https://transparencytoolkit.org/tormap.html

[...]

> 
> But, it would also be really cool if we can create a map like this
> based on the reachability of bridges per country per protocol and
> maybe, in addition, color-code/denote how the ISPs/country are
> interfering with the connection (e.g. throttling, DNS cache
> poisoning, IP addr/port blocking).

This would indeed be very cool. A problem is that it's quite hard to
make a statement as to which protocol is working especially in cases
like China where the blocking does not happen immediately.

What we can do however is have something like bubbles over every country
that show the percentage of bridges of every category that we have
detected as "not working" in the country at that given time and if "not
working" means that "Tor cannot bootstrap to 100%", "the connection
attempt failed" or "the connection was reset".

>> - - At what point in the tor bootstrapping does it fail (may be
>> difficult to determine, especially anonymized)?
> 
> Yes, but there's already a risk to running ooni-probe (at least right
> now, hopefully this will change in time). We will eventually need
> probes running in most countries if we want a good understanding of
> what network interference is taking place and who is affected.
> 

I don't think it's an issue to publish at what point Tor bootstrap
failed as it doesn't give away any particularly personally identifiable
information. Also keep in mind that at this stage all of the
measurements are being conducted from machines that we have rented and
operated ourselves so privacy of the probe operator is not much of a
problem.

>> - - In all visualizations, compare with control (filter, line break,
>> plot alongside, etc)
>>
>> And the variables we thought would be relevant to visualizations:
>> Protocol
>> Pool
>> Country (and region)- Iran, China, Netherlands (control)
>> Time it takes to be blocked
>> Point in bootstrap where it fails
>> Classify the bridges by commercial/residential connection
>> Time we started scanning the bridge from where
>>
> 
> Maybe latency measurements per protocol? Initially, I'm thinking
> "the time is takes to download a consensus from the bridge" but
> there are many variables that may affect this. Anyone have a better
> idea?
> 
> I think this mostly covers it. The only addition can think of right
> now is comparing different control countries against each other (and
> different ISPs within the control countries). Maybe we'll find
> something interesting.
> 

I was more thinking of something like "downloading a resource of [10k,
100k, 1M] from a fixed location" so that we don't have the variable of
the consensus size and can use this as a benchmark.

What I am looking for is patterns that can be symptoms of throttling of
encrypted/tor traffic.

>> It should be relatively simple to make rough versions of a lot of
>> visualizations to see what works once we have a parser/converter that
>> will generate JSONs (or similar) from OONI output that include the
>> variables listed above.
>>
> 
> Is someone already working on this? I'm not really volunteering, merely
> curious if this is in progress. :)
> 

I have written such scripts, but have not yet published them since I
still need to finish cleaning them up.

The kind of data that they end up generating looks something like this:
http://arturo.filasto.net/vizPlayground/bridge_rearchability.csv

>> Are there any other variables that would be particularly helpful to
>> track or visualize? And are there 

Re: [tor-dev] On the visualization of OONI bridge reachability data

[Arturo Filastò]

On 10/7/14, 6:18 AM, M. C. McGrath wrote:
>> The kind of data that they end up generating looks something like 
>> this: 
>> http://arturo.filasto.net/vizPlayground/bridge_rearchability.csv
> 
> Nice- though the link seems to be dead for me (I get a 404). But it is
> great that you are working on this as having this even in a rough form
> makes it possible to get started on visualizations.

Yeah sorry I mistyped.

The correct link is:
http://arturo.filasto.net/vizPlayground/bridge_reachability.csv

~ Art.




___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] On the visualization of OONI bridge reachability data

[Arturo Filastò]

On 10/7/14, 8:56 PM, Nima Fatemi wrote:
> This is awesome. one quick question tho. isn't it better to hash the
> fingerprints?

Yes you are right.
The fingerprints are now hashed.

~ Art.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] ooniprobe v1.2.2 is out

[Arturo Filastò]

Hi all,

This mail is to inform you that a new version of ooniprobe has just been
released.

Here is the changelog:

v1.2.2 (Fri, 17 Oct 2014)
-

Who said friday 17th is only bad luck?

* Add two new report entry keys test_start_time and test_runtime

* Fix bug that lead to ooniresources not working properly

It is now available for download from pypi:

https://pypi.python.org/pypi/ooniprobe

and soon will be available for download in debian.

Have fun!

~ Arturo
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] OONI hackfest summary

[Arturo Filastò]

>From October 24th to 26th the OONI team gathered in Berlin for a
hackfest. Around 20 people ended up showing up and although most of them
were seasoned Oonitarians some fairly new people joined us that I hope
will become part of the growing OONI community.

The scope of the hackfest was that of data analytics and visualization
with special focus on the Tor bridge reachability study we are currently
doing.

# Bridge reachability study

The goal of this study [1] is that of answering some questions
concerning the blocking of Tor bridges [2] and pluggable transport [3]
enabled bridges in the countries of China, Iran, Russia and Ukraine
(test vantage points).

To establish a baseline to eliminate the cases in which the bridge is
marked as blocked, while it is in fact just offline, we measure also
from a vantage point located in the Netherlands.
For every test vantage point we perform two types of measurements:

* A Bridge reachability measurement [4][5] that attempts to build a tor
circuit using the bridge in question

* A TCP connect measurement [6][7] that simply does a TCP connect to the
bridge IP and port

We run both of the measurements to further debug the reason why the
blocking is happening, may this be due to a TCP RST or direct IP
blocking or tor malfunction.

So far this study has been running for a little less than 1 month.

# OONI data pipeline

In order to produce the aggregate data needed to build visualizations we
have built a data pipeline [8]

This consists of a series of operations that are done to the raw reports
in order to strip out sensitive information and place the collected data
into a database.

The nice thing is that the data pipeline we have designed is not
specific to this study, but can and will be in the future expanded to
export data needed to visualize also the other types of measurements
done by OONI.

The data pipeline is comprised of 3 steps (or states, depending on how
you want to look at it).
When the data is submitted to a OONI collector it is synchronized with
the aggregator.
This is a central machine responsible for running all the data
processing tasks, storing the collected data in a database and hosting a
public interface to the sanitised reports. Since all the steps are
independent from one another it is not necessary that they run on the
machine, but it may also be more distributed.

Once the data is on the aggregator machine it is said to be in the RAW
state. The sanitise task is then run on the RAW data to remove sensitive
information and strip out some superfluous information. A RAW copy of
every report is also stored in a private compressed archive for future
reference.
Once the data is sanitised it is said to tbe in SANITISED state. At this
point a import task is run on the data to place it inside of a database.
The SANITISED reports are then place in a directory that is publicly
exposed to the internet to allow people to download also a copy of the
YAML reports.

At this point is is possible to run any export task that performs
queries on the database and produces as output some documents to be used
in the data visualizations (think JSON, CSV, etc.).

# The OONI hackfest

The first day of the hackfest was spent going over the scope of the
project we would be working on in the following days as well as working
in groups that were interested in tacking the design of one aspect of
the problem.

Sticky notes were plentiful and helped us have a clear vision of what
lied ahead of us.

By the end of the first day we had clear what were the set of tasks that
were needed to achieve our goals and which teams would be responsible
for doing what.

The second day was almost entirely dedicated to hacking and everybody
had a task to complete that was either completed by the end of the day
or sooner. Some people even completed their initially assigned task
before the end of the day and came back asking for more!

By the end of the second day we had a real data set to hand over to the
visualization team, to start producing some pretty graphs based on real
data.

We decided that the first visualization we wanted to do should be kept
as simple as possible and be something that we could also use to debug
the data we had collected. It should tell us which bridges were working
when and it should present the information in a way that would highlight
the country involved and the pluggable transport type.

A prototype of it can be seen here:

http://reports.ooni.nu/analytics/bridge_reachability/timeline/

The code for this visualization can be found here:
https://github.com/Shidash/OONI-Bridge-Reachability-Timeline

# Next steps

* Write scripts for generating the bridge_db.json document based on the
data that is given to us from the bridge db team
https://trac.torproject.org/projects/tor/ticket/13570

* Align the dates in the visual timeline
https://trac.torproject.org/projects/tor/ticket/13639

* Better tokenising for bridges so that bridges that have the same
  fingerprint, but 

[tor-dev] OONI weekly dev meeting reminder

[Arturo Filastò]

Hello Oonitarians,

This is reminder that today (Thursday 6th of November 2014) there will
be the weekly ooni dev meeting in #ooni on irc.oftc.net.

The time will be:

18:00 UTC
19:00 CET
13:00 EST
12:00 CST
10:00:00 PST

You are all invited to attend!

~ Arturo
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] OONI weekly dev meeting reminder and possible rotating schedule for next meetings

[Arturo Filastò]

Hello Oonitarians,

This is reminder that today (Thursday 13th of November 2014) there will
be the weekly ooni dev meeting in #ooni on irc.oftc.net.

The time will be:

18:00 UTC
19:00 CET
13:00 EST
12:00 CST
10:00:00 PST

I take this opportunity to also propose a solution that was brought up
during last weeks meeting. Some people have issues attending the meeting
on some days, while others have issues with other days. It was suggested
that we start having a rotating schedule that changes every week so that
everybody will be able to at least attend 1 meeting every month.

What do you think?

~ Arturo
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Next OONI weekly meetings

[Arturo Filastò]

Hello Oonitarians,

Sorry for not sending this email sooner, but this week has been a bit
hectic.

A couple of people have said that they cannot attend the weekly meeting
on thursday at 18:00, because it conflicts with another recurring
appointment they have.

For this reason I am going to suggest we reschedule the next OONI
meetings to a date that works for everybody.

Monday and Tuesdays seem to be the best option, so I am going to propose
we skip next weeks meeting and instead opt for holding next weeks one on
either a Monday or Tuesday at 18:00 UTC, 19:00 CET, 13:00 EST, 12:00
CST, 10:00:00 PST.

How does this sound for everybody?

I personally would prefer Monday over Tuesday, but if you would instead
prefer Tuesday please send a reply to the ooni-dev mailing list.

If nobody replies I am going to assume every Monday works for you all
and the next meeting will be on Monday 24th of November.

Have fun!

~ Arturo

___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Next OONI weekly meetings

[Arturo Filastò]

My bad for not confirming this and not sending out a reminder to Mondays
(today's) meeting.

If you are on IRC and have something to discuss we can do it now, if not
next meeting will be December 1st at 18:00 UTC (19:00 CET).

I will be traveling at that time so I am not yet certain I will be able
to attend. I will nonetheless send out a reminder about it and if it
does not work out to do it that date the next one will be on December 8th.

~ Arturo

On 11/19/14, 7:37 PM, Aaron Gibson wrote:
> On 2014-11-19 11:20, Arturo Filastò wrote:
>> Hello Oonitarians,
>>
>> Sorry for not sending this email sooner, but this week has been a bit
>> hectic.
>>
>> A couple of people have said that they cannot attend the weekly meeting
>> on thursday at 18:00, because it conflicts with another recurring
>> appointment they have.
>>
>> For this reason I am going to suggest we reschedule the next OONI
>> meetings to a date that works for everybody.
>>
>> Monday and Tuesdays seem to be the best option, so I am going to propose
>> we skip next weeks meeting and instead opt for holding next weeks one on
>> either a Monday or Tuesday at 18:00 UTC, 19:00 CET, 13:00 EST, 12:00
>> CST, 10:00:00 PST.
>>
>> How does this sound for everybody?
>>
>> I personally would prefer Monday over Tuesday, but if you would instead
>> prefer Tuesday please send a reply to the ooni-dev mailing list.
>>
>> If nobody replies I am going to assume every Monday works for you all
>> and the next meeting will be on Monday 24th of November.
> 
> My preference is also Monday.
> 
> --Aaron
>>
>> Have fun!
>>
>> ~ Arturo
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Reminder for weekly OONI dev meeting Monday 18:00 UTC (19:00 CET)

[Arturo Filastò]

Hi,

This is a reminder that today there will be the weekly OONI meeting.

As I am writing this I am on a plane heading towards London and will be
working on a GlobaLeaks workshop in some office there.

The schedule for the workshop is not very well defined so I am not 100%
sure I will manage to be online at the above defined time.

If I am not there please start without me and I will be sure to read the
backlog and write feedback asynchronously.

~ Arturo
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Reminder for weekly OONI dev meeting Monday 2014-12-15 18:00 UTC (19:00 CET)

[Arturo Filastò]

Hi,

This is a reminder that today there will be the weekly OONI meeting.

It will happen as usual on the #ooni channel on irc.oftc.net at 18:00
UTC (19:00 CET, 13:00 EST, 10:00 PST).

Everybody is welcome to join us and bring their questions and feedback.

See you later,

~ Arturo

___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Tor bridge reachability timeline being updated?

[Arturo Filastò]

Hi David,

Thanks for your interest!

On 12/16/14, 3:18 AM, David Fifield wrote:
> Arturo,
> 
> I really like http://reports.ooni.nu/analytics/bridge_reachability/timeline/.
> Is it still being updated? Are updates going to a different place?
> 

The visualization is currently not being updated with live data, because
there are a couple of bugs in the visualization that I requested by
fixed before making it live.
We are however collecting measurements from all those vantage points, so
it's not like we are loosing data, it's just that I haven't had the time
to write a cronjob to update the graphs with the data we collect.

Choke Point Project is working on fixing some of the above mentioned
bugs starting December 19th and will, if all goes well, be done by the
end of this year.

> What does it take to add obfs4 support? As obfs4 is about to be
> introduced, it would be great to have reachability measurements from a
> variety of vantage points from the very beginning.
> 

We actually have a ticket already filed about adding it. It would really
be a pretty simple thing to do, it's just that nobody has sat down to do it.

https://trac.torproject.org/projects/tor/ticket/13597


~ Arturo
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Reminder for weekly OONI dev meeting Monday 2014-12-22 18:00 UTC (19:00 CET)

[Arturo Filastò]

Hello Oonitarians,

This is a reminder that today there will be the weekly OONI meeting.

It will happen as usual on the #ooni channel on irc.oftc.net at 18:00
UTC (19:00 CET, 13:00 EST, 10:00 PST).

Everybody is welcome to join us and bring their questions and feedback.

See you later,

~ Arturo
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Reminder for weekly OONI dev meeting Monday 2015-01-05 18:00 UTC (19:00 CET)

[Arturo Filastò]

Hello Oonitarians,

This is a reminder that today there will be the weekly OONI meeting.

It will happen as usual on the #ooni channel on irc.oftc.net at 18:00
UTC (19:00 CET, 13:00 EST, 10:00 PST).

Everybody is welcome to join us and bring their questions and feedback.

See you later,

~ Arturo
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] FTBFS for ooniprobe on Fedora 21

[Arturo Filastò]

On 1/8/15 1:21 AM, Jacek Wielemborek wrote:
> Hello,
> 
> Below is the log. Am I missing some package or something?
> 

[ snip ]

> 
> pcap_ex.c:18:23: fatal error: pcap-int.h: No such file or directory
> 
>  # include 
> 
>^
> 
> compilation terminated.
> 
> error: command 'gcc' failed with exit status 1
> 

[ snip ]

> [1:18:05][~]$ yum provides pcap-int.h
> Loaded plugins: auto-update-debuginfo, langpacks
> (cut)
> No matches found
> 

>From the looks of it you are missing the libpcap-dev package. I believe
in fedora it is called libpcap-devel.

If you still encounter other issues you should reach us on the ooni-dev
mailing list (ooni-...@lists.torproject.org) or try asking around on IRC
#ooni irc.oftc.net.

~ Arturo


___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Request to reschedule ooni dev meeting to Tuesday 13th

[Arturo Filastò]

Hi,

I have to study for an exam on Tuesday so it would be ideal for me if we
could move the next ooni dev meeting to Tuesday.

Does that work for those interested in attending?

I would suggest we do it at the same time (19:00 CET).

~ Arturo
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Request to reschedule ooni dev meeting to Tuesday 13th

[Arturo Filastò]

From IRC it seems like it's ok to move the meeting.

So unless something else comes up in the next hours it's going to happen
tomorrow at 19:00 CET.

I will send out a reminder for the meeting later today.

~ Arturo

On 1/11/15 7:16 PM, Arturo Filastò wrote:
> Hi,
> 
> I have to study for an exam on Tuesday so it would be ideal for me if we
> could move the next ooni dev meeting to Tuesday.
> 
> Does that work for those interested in attending?
> 
> I would suggest we do it at the same time (19:00 CET).
> 
> ~ Arturo
> 
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Reminder for ooni weekly meeting today at 19:00 CET

[Arturo Filastò]

This is to remind you that today Tuesday 13th of January at 19:00 CET
(18:00 UTC) there will be the weekly ooni dev meeting.

The channel is #ooni, the network is irc.oftc.net.

See you there.

~ Arturo


___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Event at NEXA center and next OONI dev meeting

[Arturo Filastò]

Hello Oonitarians!

We skipped this weeks meeting due to a lot of us being busy with an
event at the NEXA center.

It was a very interesting opportunity to meet a lot of great network
measurement and network neutrality researchers and discuss possible
areas of collaboration.

I will give a brief summary of some of the most relevant things WRT OONI.

# Libight hackfest

On the first day we did a hackfest on libight. At the end of it we were
able to produce a build of libight for iOS [1] as well as a mockup of
the GUI [2].

We also finished the integration of the SOCKS client [3] that will allow
us to use tor from libight.

Overall we are quite close to having a working prototype for iOS and
soon also for Android.

# NNTools meeting

Enrico Gregori and Valerio Luconi from Italian National Research Council
presented their work on Portolan, that is trying to map BGP
interconnections with traceroutes. They have developed a desktop and
mobile application that is very cool and works well!

Later we discussed about possible collaborations, in particular
deploying their tests as part of our raspberry pi deployment.

For the full schedule of events see: http://nexa.polito.it/nntools2015

Regarding the next developer meeting I am going to suggest we do it as
usual on Monday at 19:00 CET (18:00 UTC).

That's it, until next week!

Have fun!

~ Arturo


[1] https://github.com/alemela/libight_iOS
[2] https://people.torproject.org/~art/ooni/mockups/ooni-ight-assets.zip
[3] https://github.com/TheTorProject/libight/pull/71
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Reminder for weekly OONI dev meeting Monday 2015-01-26 18:00 UTC (19:00 CET)

[Arturo Filastò]

Hello Oonitarians,

This is a reminder that today there will be the weekly OONI meeting.

It will happen as usual on the #ooni channel on irc.oftc.net at 18:00
UTC (19:00 CET, 13:00 EST, 10:00 PST).

Everybody is welcome to join us and bring their questions and feedback.

See you later,

~ Arturo

___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Let's come up with the roadmap for the future of OONI

[Arturo Filastò]

Hello Oonitarians and Divisionists,

I would love to have your feedback on what you believe to be the most
important topics for the future of OONI.

I have made a list of what I believe are all possible and interesting
tasks to perform, but we can't do them all and for sure we can't do them
all at once.

For this reason it would be very useful if you could express a vote from
1-5:

1: Nah, this is boring and pointless
2: Not really super important, I would give priority to other stuff
3: Useful, I would do it
4: This would be awesome
5: Epic!

Feel free to also expand these topics with questions and feedback (or
new ones). At the end of the list I will give you a table to cast your vote.


# Get daily OONI measurements from 50 countries

This means focusing on getting a large and diverse reliable OONI
userbase. It
does not simply mean to get at least 1 measurement in 50 countries, but
it means
either establishing relationships with trusted parties in 50 countries, or
expanding our userbase to a point where we have at least 1 measurement
per day
for every test per country (from the same network).

Ways of achieving this are:

  * Rent a VPS in X number of countries
  * Running the adopt an ooniprobe program (see below for more details)
  * Establish an agreement with operators in 50 countries to host the
probing
infrastructure (as previously stated)

# Develop OONI tests for censorship circumvention tools

This involves devising a methodology for testing the reliability of
censorship
circumvention tools in various countries.
It means testing censorship circumvention tool both open
source and propertairy.

A cursory list of the protocols/tools we could be interested includes,
but is
not limited to:

* Tor
* VPN
* Web proxies (hidemyass, etc.)
* SSH tunnel
* Freegate
* Psiphon
* Ultrasurf
* alkasir

If you think we should be testing some other tools too, please add to
this list.

Useful resources:
http://cyber.law.harvard.edu/publications/2011/2011_Circumvention_Tool_Evaluation
http://cyber.law.harvard.edu/publications/2010/Circumvention_Tool_Usage


# Develop scheme for orchestrating ooni-probes

This means coming up with a protocol that allows an OONI test developer to
schedule a measurement to be run with a certain input they decide on a
set of
probes in country X.

Obviously security considerations need to be taken into account and
access will
be in the initial stage only limited to a very restricted set of OONI
developers that will be made public.

# Implement data analytics and visualization for OONI tests

We have a bunch of data and we would like to give meaning to it.
This would involve writing tools for querying the data in the database and
extract useful analytical information from it.

Based on this data we can then start looking at historical OONI data and
provide some sample visually supported reports.

This is a list of tests we should develop analytics for:

  * HTTP requests
  * DNS Consistency
  * DNS Injection
  * TCP Connect
  * HTTP Invalid Request Line
  * HTTP Header Field Manipulation
  * Multi protocol port traceroute

# Implement pub-sub system for ooni collectors

Currently OONI collectors (the things you send your ooniprobe measurement
results to) keep in sync thanks to a bunch of shell scripts and cronjobs.
To have more real time data it would be useful to have a pub-sub
mechanism that
allows the pipeline to subscribe to all the collectors and the
collectors will
then publish the collected reports to it, as soon as they are submitted.

This will allow the OONI data to go through the data pipeline much faster
(instead of ~2 hours, perhaps just some minutes or even less potentially).

# Reach production quality ooni rasperry-pi (beagle-board) images

This involves implementing what is specified in the lepidopter
specification:
https://github.com/anadahz/lepidopter/blob/master/specification.md

We should then provide scripts for building the image yourself or how to
download and burn it to an SD card on Windows, OSX, Linux (with
screenshots).

As a bonus we could also offer shipping of pre-made raspberry pi images
already
burn to an SD card, similar to what is done with rasbpian images.

# Promote and further develop OONI on mobile (Android, iOS)

This involves improving the GUI of OONI on mobile and getting it into the
Google Play store and the Apple App store.

We should also work on making it easier for developers of existing iOS and
Android apps to add internet measurement capabilities to their app by
linking
to libight.

# Do research based on OONI

This would involve doing some research on internet censorship based on
OONI probe or on internet measurement in general and publishing them in
peer reviewed venues.

# Publish monthly reports about the status of internet censorship in a
country

This would be sort of like a monthly e-zine, where every month we
analyse the
status of internet censorship in a given country.
It should be backed by OONI data, but the core of it should

[tor-dev] OONI weekly dev meeting today Monday 23rd of February 18:00 UTC

[Arturo Filastò]

Hello Oonitarians,

We have skipped the last two weekly dev meetings because of various
necessities, but I am going to suggest we try and make it today.

There are a lot of interesting things to discuss and think about together.

These are the topics that I have in mind, feel free to append to this
list if you think that there is something else:

* Discuss the outcome of the "Future of OONI" thread:
https://lists.torproject.org/pipermail/ooni-dev/2015-February/000246.html

* Updates on the migration of the pipeline

* Discuss how to get 1.2.3 released on debian

* Updates on libight and iOS integration

* Finishing the bridge reachability study

The meeting will happen as usual on the #ooni channel on irc.oftc.net at
18:00 UTC (19:00 CET, 13:00 EST, 10:00 PST).

See you soon!

~ Arturo

___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] OONI weekly dev meeting today Monday 9th of March 18:00 UTC

[Arturo Filastò]

Hello Oonitarians,

As discussed during the last meeting we did not gather this week because
of the OpenITP festival.

It was a very fruitful and interesting event, that we should update
people that could not attend about.

We shall therefore gather at the usual 18:00 UTC (19:00 CET, 13:00 EST,
10:00 PST) on the #ooni channel on irc.oftc.net.

See you soon!

~ Arturo

___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Summer of Privacy application, Censorship Analyzer

[Arturo Filastò]

 




On April 15, 2015 at 6:01:15 PM, Philipp Winter 
(p...@nymity.ch(mailto:p...@nymity.ch)) wrote:
> On Tue, Apr 14, 2015 at 11:56:12AM +0200, Miquel Llobet wrote:
> > As far as coding goes, I played a bit with OONI (did a scan, turns out I'm
> > clean :-) ). and built it from source. What bugs to you recommend to work
> > on as a start? Ideally I can write a patch before the submission is due to
> > attest my skills.
> >
> > Let me know if you have any suggestion or advice on pursuing this project.
>

Hi Miguel,


Thanks for your interest in OONI!

To have a brids-eye view of all the OONI tickets the link Philip gave you is 
indeed the best.

Let me also highlight some things that would need to be done that have not yet 
made their way into trac (mea culpa):

It would be good to have a Tor test that specifically checks for reachability, 
via a TCP Connect scan, of:

1) Tor Directory authorities

2) Tor relays

We are contributing to the citizenlab test-lists 
(https://github.com/citizenlab/test-lists(https://github.com/citizenlab/test-lists/pull/9)/)
 to add support for new input lists we would like to scan.

In particular you should look at my feature/services pull request: 
https://github.com/citizenlab/test-lists/pull/9 and the ticket on that 
repository that describes the motivation behind it:
https://github.com/citizenlab/test-lists/issues/6

What in particular would need to be done is:

* Implement 2 new services called tor/directory_authorities and tor/relays. As 
an example you can see how I implemented the services/tor/bridges here: 
https://github.com/hellais/test-lists/blob/feature/services/lib/lists/services/tor/bridges.py

* Implement a new ooni-probe test that will:

1) To a TCP connect scan to the Tor Directory autorities AND/OR Tor relays. See 
the tcp_connect test to see how this can be done: 
https://gitweb.torproject.org/ooni-probe.git/tree/ooni/nettests/blocking/tcp_connect.py

2) Do a full Tor connection by running the actual Tor binary. See the bridge 
reachability test to see how this can be done: 
https://gitweb.torproject.org/ooni-probe.git/tree/ooni/nettests/blocking/bridge_reachability.py

That said I would say the first thing you do is open some tickets to do the 
above mentioned tasks.

For the ooni-probe test please file that as a child ticket of this ticket:
https://trac.torproject.org/projects/tor/ticket/15170


> Also, do you have some experience with asynchronous programming in
> general, and Twisted in particular? I'm asking because several students
> have tried to work on this project in the past and nobody has delivered
> much code so far. I think that many had issues with learning
> asynchronous programming and getting familiar with OONI's code base.
>


I agree with philip that it would be ideal if you had experience with Twisted 
and async programming, though I think that for some of the above mentioned 
tasks that is not a must.

I am always very happy to help you out and discuss ideas more synchronously on 
either jabber: hell...@jabber.ccc.de(mailto:hell...@jabber.ccc.de) or IRC: 
#ooni irc.oftc.net(http://irc.oftc.net).

Have fun!

~ Arturo



___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] ooniprobe depending on a badly written lib (Re: FTBFS for ooniprobe on Fedora 21)

[Arturo Filastò]

On May 18, 2015 at 4:19:09 PM, Jacek Wielemborek (d33...@gmail.com) wrote:

I do have libpcap-devel installed, it's just this pcap-int.h header that 
is both missing and not to be found in any Fedora package: 

$ rpm -qa | grep pcap-dev 
libpcap-devel-1.6.2-1.fc21.x86_64 

Note the following tickets: 

https://bugzilla.redhat.com/show_bug.cgi?id=449387 
https://code.google.com/p/pypcap/issues/detail?id=45 

It looks like this wasn't included on purpose - pypcap is using 
libpcap's private interface. Note that according to Google Code, this 
package had no releases or SVN updates since 2010. 


Hi Jacek,

From where did you get pypcap?

The version on pypi (installable with pip install pypap) should be my fork of 
it: https://github.com/hellais/pypcap that includes a workaround for that 
problem:

https://github.com/hellais/pypcap/blob/master/pcap_ex.c#L17

Try installing that version of it and if it fails please open a ticket on:

https://github.com/hellais/pypcap/issues

~ Arturo


___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev