date:20160429

[tor-commits] [stem/master] Drop nickname from fallback directories

2016-04-29 Thread atagar

commit 2bc6282179705a58e4ed114b0895c2ab2d83ab01
Author: Damian Johnson 
Date:   Fri Apr 29 11:44:08 2016 -0700

Drop nickname from fallback directories

Tor commit 8b90d45 complicates how fallback_dirs.inc is generated. Now that 
it
no longer has hand-managed headers we don't have the nicknames. Dropping 
this
attribute from the class.
---
 cache_fallback_directories.py|   1 -
 stem/descriptor/fallback_directories.cfg | 408 ++-
 stem/descriptor/remote.py|  75 +++---
 test/unit/descriptor/remote.py   |  18 +-
 4 files changed, 323 insertions(+), 179 deletions(-)

diff --git a/cache_fallback_directories.py b/cache_fallback_directories.py
index 138f767..21d9fcf 100755
--- a/cache_fallback_directories.py
+++ b/cache_fallback_directories.py
@@ -59,6 +59,5 @@ if __name__ == '__main__':
 conf.set('%s.address' % fingerprint, directory.address)
 conf.set('%s.or_port' % fingerprint, str(directory.or_port))
 conf.set('%s.dir_port' % fingerprint, str(directory.dir_port))
-conf.set('%s.nickname' % fingerprint, directory.nickname)
 
   conf.save(stem.descriptor.remote.CACHE_PATH)
diff --git a/stem/descriptor/fallback_directories.cfg 
b/stem/descriptor/fallback_directories.cfg
index d420fb5..50ad27b 100644
--- a/stem/descriptor/fallback_directories.cfg
+++ b/stem/descriptor/fallback_directories.cfg
@@ -1,130 +1,302 @@
-tor_commit e2202146d16af22502fd4166ac926fefa6528dff
-stem_commit a96ec510539f2918bf93c50d63ce68e919f71209
-8FA37B93397015B2BC5A525C908485260BE9F422.address 178.254.44.135
-8FA37B93397015B2BC5A525C908485260BE9F422.or_port 9001
-8FA37B93397015B2BC5A525C908485260BE9F422.dir_port 9030
-8FA37B93397015B2BC5A525C908485260BE9F422.nickname Doedel22
-855BC2DABE24C861CD887DB9B2E950424B49FC34.address 84.219.173.60
-855BC2DABE24C861CD887DB9B2E950424B49FC34.or_port 443
-855BC2DABE24C861CD887DB9B2E950424B49FC34.dir_port 9030
-855BC2DABE24C861CD887DB9B2E950424B49FC34.nickname Logforme
+tor_commit 1fd4340f827065485f8ce3fd03c5573f89880893
+stem_commit b5661f160a393984fd99b2cb67438ee4eb76e3a9
+9504CB22EEB25D344DE63CB7A6F2C46F895C3686.address 46.101.102.71
+9504CB22EEB25D344DE63CB7A6F2C46F895C3686.or_port 443
+9504CB22EEB25D344DE63CB7A6F2C46F895C3686.dir_port 80
+823AA81E277F366505545522CEDC2F529CE4DC3F.address 192.160.102.164
+823AA81E277F366505545522CEDC2F529CE4DC3F.or_port 9001
+823AA81E277F366505545522CEDC2F529CE4DC3F.dir_port 80
+AEA43CB1E47BE5F8051711B2BF01683DB1568E05.address 37.187.7.74
+AEA43CB1E47BE5F8051711B2BF01683DB1568E05.or_port 443
+AEA43CB1E47BE5F8051711B2BF01683DB1568E05.dir_port 80
+86C281AD135058238D7A337D546C902BE8505DDE.address 185.96.88.29
+86C281AD135058238D7A337D546C902BE8505DDE.or_port 443
+86C281AD135058238D7A337D546C902BE8505DDE.dir_port 80
+8664DC892540F3C789DB37008236C096C871734D.address 163.172.138.22
+8664DC892540F3C789DB37008236C096C871734D.or_port 443
+8664DC892540F3C789DB37008236C096C871734D.dir_port 80
+387B065A38E4DAA16D9D41C2964ECBC4B31D30FF.address 62.210.92.11
+387B065A38E4DAA16D9D41C2964ECBC4B31D30FF.or_port 9101
+387B065A38E4DAA16D9D41C2964ECBC4B31D30FF.dir_port 9130
+DE134FC8E5CC4EC8A5DE66934E70AC9D70267197.address 195.191.233.221
+DE134FC8E5CC4EC8A5DE66934E70AC9D70267197.or_port 443
+DE134FC8E5CC4EC8A5DE66934E70AC9D70267197.dir_port 80
+5714542DCBEE1DD9864824723638FD44B2122CEA.address 109.163.234.9
+5714542DCBEE1DD9864824723638FD44B2122CEA.or_port 443
+5714542DCBEE1DD9864824723638FD44B2122CEA.dir_port 80
+50586E25BE067FD1F739998550EDDCB1A14CA5B2.address 212.51.134.123
+50586E25BE067FD1F739998550EDDCB1A14CA5B2.or_port 9001
+50586E25BE067FD1F739998550EDDCB1A14CA5B2.dir_port 9030
+BCEDF6C193AA687AE471B8A22EBF6BC57C2D285E.address 198.96.155.3
+BCEDF6C193AA687AE471B8A22EBF6BC57C2D285E.or_port 5001
+BCEDF6C193AA687AE471B8A22EBF6BC57C2D285E.dir_port 8080
+F65E0196C94DFFF48AFBF2F5F9E3E19AAE583FD0.address 94.242.246.23
+F65E0196C94DFFF48AFBF2F5F9E3E19AAE583FD0.or_port 9001
+F65E0196C94DFFF48AFBF2F5F9E3E19AAE583FD0.dir_port 443
+616081EC829593AF4232550DE6FFAA1D75B37A90.address 95.128.43.164
+616081EC829593AF4232550DE6FFAA1D75B37A90.or_port 443
+616081EC829593AF4232550DE6FFAA1D75B37A90.dir_port 80
+58ED9C9C35E433EE58764D62892B4FFD518A3CD0.address 185.21.100.50
+58ED9C9C35E433EE58764D62892B4FFD518A3CD0.or_port 9001
+58ED9C9C35E433EE58764D62892B4FFD518A3CD0.dir_port 9030
+35E8B344F661F4F2E68B17648F35798B44672D7E.address 146.0.32.144
+35E8B344F661F4F2E68B17648F35798B44672D7E.or_port 9001
+35E8B344F661F4F2E68B17648F35798B44672D7E.dir_port 9030
+14419131033443AE6E21DA82B0D307F7CAE42BDB.address 144.76.14.145
+14419131033443AE6E21DA82B0D307F7CAE42BDB.or_port 143
+14419131033443AE6E21DA82B0D307F7CAE42BDB.dir_port 110
+79E169B25E4C7CE99584F6ED06F379478F23E2B8.address 62.210.129.246
+79E169B25E4C7CE99584F6ED06F379478F23E2B8.or_port 443
+79E169B25E4C7CE99584F6ED06F379478F23E2B8.dir_port 80
+FD1871854BFC06D7B02F10742073069F0528B5CC.address 192.187.124.98

[tor-commits] [translation/tails-misc_completed] Update translations for tails-misc_completed

2016-04-29 Thread translation

commit 8e669088ed74663f925f5babe9f98fdc8be31331
Author: Translation commit bot 
Date:   Fri Apr 29 17:15:55 2016 +

Update translations for tails-misc_completed
---
 ca.po | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/ca.po b/ca.po
index ea18b08..6c2aebf 100644
--- a/ca.po
+++ b/ca.po
@@ -14,7 +14,7 @@ msgstr ""
 "Project-Id-Version: The Tor Project\n"
 "Report-Msgid-Bugs-To: \n"
 "POT-Creation-Date: 2016-04-25 14:02+0200\n"
-"PO-Revision-Date: 2016-04-28 17:44+\n"
+"PO-Revision-Date: 2016-04-29 16:58+\n"
 "Last-Translator: laia_\n"
 "Language-Team: Catalan 
(http://www.transifex.com/otf/torproject/language/ca/)\n"
 "MIME-Version: 1.0\n"

___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [translation/tails-misc] Update translations for tails-misc

2016-04-29 Thread translation

commit dd3970621e23f7a33a8aadf9345de9cf80156cb4
Author: Translation commit bot 
Date:   Fri Apr 29 17:15:52 2016 +

Update translations for tails-misc
---
 ca.po | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/ca.po b/ca.po
index ea18b08..6c2aebf 100644
--- a/ca.po
+++ b/ca.po
@@ -14,7 +14,7 @@ msgstr ""
 "Project-Id-Version: The Tor Project\n"
 "Report-Msgid-Bugs-To: \n"
 "POT-Creation-Date: 2016-04-25 14:02+0200\n"
-"PO-Revision-Date: 2016-04-28 17:44+\n"
+"PO-Revision-Date: 2016-04-29 16:58+\n"
 "Last-Translator: laia_\n"
 "Language-Team: Catalan 
(http://www.transifex.com/otf/torproject/language/ca/)\n"
 "MIME-Version: 1.0\n"

___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [translation/https_everywhere_completed] Update translations for https_everywhere_completed

2016-04-29 Thread translation

commit fc09bf861695c1971df194dbd0eb5d10efac57f5
Author: Translation commit bot 
Date:   Fri Apr 29 17:15:25 2016 +

Update translations for https_everywhere_completed
---
 ca/https-everywhere.dtd | 20 ++--
 ca/ssl-observatory.dtd  | 10 +-
 2 files changed, 15 insertions(+), 15 deletions(-)

diff --git a/ca/https-everywhere.dtd b/ca/https-everywhere.dtd
index bc91115..cddb1cf 100644
--- a/ca/https-everywhere.dtd
+++ b/ca/https-everywhere.dtd
@@ -6,8 +6,8 @@
 
 
 
-
-
+
+
 
 
 
@@ -39,18 +39,18 @@
 
 
 
-
+
 
 
-
-
+
+
 
 
-
-
+
+
 
-
-
-
+
+
+
 
 
diff --git a/ca/ssl-observatory.dtd b/ca/ssl-observatory.dtd
index d4eb580..dc44b2e 100644
--- a/ca/ssl-observatory.dtd
+++ b/ca/ssl-observatory.dtd
@@ -27,16 +27,16 @@ xarxa corporativa molt intrusiva.">
 "Ãs segur, excepte si utilitzeu una xarxa corporativa amb noms secrets 
d'intranet.">
 
 
+"Envia i comprova els certificats firmats per CAs d'arrel no estÃ ndard.">
 
 
+"Ãs segur (i una bona idea) permetre aquesta opciÃ³, excepte si utilitzeu una 
xarxa corporativa molt intrusiva o un antivirus Kaspersky que monitoreja el 
navegador amb un servidor intermediari TLS i un Certificat d'Autoritat d'arrel 
privada. Si ho permeteu en aquesta xarxa, aquesta opciÃ³ pot publicar 
evidÃ¨ncies de quins dominis https:// sÃ³n visitats a travÃ©s d'aquest servidor 
intermediari, pel seu certificat Ãºnic. En aquest cas, Ã©s millor deixar-ho 
igual.">
 
-
+
 
+"Comprova els certificats fent servir Tor per a l'anonimat (requereix Tor)">
 
+"Aquesta opciÃ³ requereix que Tor estigui instalÂ·lat i en funcionament">
 
 

___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [translation/torcheck_completed] Update translations for torcheck_completed

2016-04-29 Thread translation

commit 43e5a3409c38a4fec39da13d83035971bd6f2f98
Author: Translation commit bot 
Date:   Fri Apr 29 17:15:12 2016 +

Update translations for torcheck_completed
---
 ca/torcheck.po | 22 +++---
 1 file changed, 11 insertions(+), 11 deletions(-)

diff --git a/ca/torcheck.po b/ca/torcheck.po
index feee344..5fbc174 100644
--- a/ca/torcheck.po
+++ b/ca/torcheck.po
@@ -6,14 +6,14 @@
 # Bartges90, 2015
 # David Anglada , 2014
 # Humbert , 2014
-# laia_, 2014-2015
+# laia_, 2014-2016
 # Pau SellÃ©s i Garcia , 2013
 # Toni Hermoso Pulido , 2012
 msgid ""
 msgstr ""
 "Project-Id-Version: The Tor Project\n"
 "POT-Creation-Date: 2012-02-16 20:28+PDT\n"
-"PO-Revision-Date: 2016-03-21 16:27+\n"
+"PO-Revision-Date: 2016-04-29 16:52+\n"
 "Last-Translator: laia_\n"
 "Language-Team: Catalan 
(http://www.transifex.com/otf/torproject/language/ca/)\n"
 "MIME-Version: 1.0\n"
@@ -33,7 +33,7 @@ msgid ""
 msgstr "Visiteu el https://www.torproject.org/\\\;>lloc web de 
Tor per obtenir mÃ©s informaciÃ³ de com utilitzar Tor de manera segura. Ara 
teniu l'oportunitat de navegar per Internet anÃ²nimament."
 
 msgid "There is a security update available for Tor Browser."
-msgstr "Hi ha una actualitzaciÃ³ de seguretat disponible per al navegador Tor"
+msgstr "Hi ha una actualitzaciÃ³ de seguretat disponible per al Navegador Tor"
 
 msgid ""
 "https://www.torproject.org/download/download-easy.html\;>Click "
@@ -41,14 +41,14 @@ msgid ""
 msgstr "https://www.torproject.org/download/download-easy.html\\\;>Cliqueu 
aquÃ per anar a la pÃ gina de baixades"
 
 msgid "Sorry. You are not using Tor."
-msgstr "Ho sentim. No esteu utilitzant Tor."
+msgstr "No esteu utilitzant Tor."
 
 msgid ""
 "If you are attempting to use a Tor client, please refer to the https://www.torproject.org/\;>Tor website and specifically the instructions for "
 "configuring your Tor client."
-msgstr "Si esteu intentant utilitzar el client  Tor, visiteu el https://www.torproject.org/\\\;>lloc web de Tor i especÃficament 
les https://www.torproject.org/docs/faq#DoesntWork\\\;>instruccions 
per a configurar el client Tor."
+msgstr "Si esteu intentant utilitzar el client Tor, visiteu el https://www.torproject.org/\\\;>lloc web de Tor i especÃficament 
les https://www.torproject.org/docs/faq#DoesntWork\\\;>instruccions 
per a configurar el client Tor."
 
 msgid "Sorry, your query failed or an unexpected response was received."
 msgstr "La consulta ha fallat o s'ha rebut una resposta no esperada."
@@ -56,19 +56,19 @@ msgstr "La consulta ha fallat o s'ha rebut una resposta no 
esperada."
 msgid ""
 "A temporary service outage prevents us from determining if your source IP "
 "address is a https://www.torproject.org/\;>Tor node."
-msgstr "Una falla temporal del servei no ens permet determinar si la adreÃ§a 
IP d'origen Ã©s un node https://www.torproject.org/\\\;>Tor."
+msgstr "Una falla temporal del servei no ens permet determinar si l'adreÃ§a IP 
d'origen Ã©s un node https://www.torproject.org/\\\;>Tor."
 
 msgid "Your IP address appears to be: "
-msgstr "La vostra adreÃ§a IP sembla que Ã©s:"
+msgstr "Sembla que la vostra adreÃ§a IP sigui:"
 
 msgid "Are you using Tor?"
 msgstr "Esteu usant Tor?"
 
 msgid "This page is also available in the following languages:"
-msgstr "PÃ gina tambÃ© disponible en aquests idiomes:"
+msgstr "Aquesta pÃ gina tambÃ© estÃ  disponible en aquests idiomes:"
 
 msgid "For more information about this exit relay, see:"
-msgstr "Visiteu la segÃ¼ent pÃ gina per a mÃ©s informaciÃ³ sobre el trÃ fic de 
sortida:"
+msgstr "Per a mÃ©s informaciÃ³ sobre el trÃ fic de sortida visiteu:"
 
 msgid ""
 "The Tor Project is a US 501(c)(3) non-profit dedicated to the research, "
@@ -82,7 +82,7 @@ msgid "Go"
 msgstr "VÃ©s-hi"
 
 msgid "Short User Manual"
-msgstr "Manual d'usuari curt"
+msgstr "Manual d'usuari abreujat"
 
 msgid "Donate to Support Tor"
 msgstr "Donatiu per donar suport a Tor"
@@ -91,7 +91,7 @@ msgid "Tor Q Site"
 msgstr "Lloc de Q de Tor"
 
 msgid "Volunteer"
-msgstr "Voluntari"
+msgstr "Fes-te voluntari"
 
 msgid "JavaScript is enabled."
 msgstr "JavaScript estÃ  activat."

___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [translation/https_everywhere] Update translations for https_everywhere

2016-04-29 Thread translation

commit 9524b67f908419ec53b5f1d6dd9973d174384c21
Author: Translation commit bot 
Date:   Fri Apr 29 17:15:20 2016 +

Update translations for https_everywhere
---
 ca/https-everywhere.dtd | 20 ++--
 ca/ssl-observatory.dtd  | 10 +-
 2 files changed, 15 insertions(+), 15 deletions(-)

diff --git a/ca/https-everywhere.dtd b/ca/https-everywhere.dtd
index bc91115..cddb1cf 100644
--- a/ca/https-everywhere.dtd
+++ b/ca/https-everywhere.dtd
@@ -6,8 +6,8 @@
 
 
 
-
-
+
+
 
 
 
@@ -39,18 +39,18 @@
 
 
 
-
+
 
 
-
-
+
+
 
 
-
-
+
+
 
-
-
-
+
+
+
 
 
diff --git a/ca/ssl-observatory.dtd b/ca/ssl-observatory.dtd
index d4eb580..dc44b2e 100644
--- a/ca/ssl-observatory.dtd
+++ b/ca/ssl-observatory.dtd
@@ -27,16 +27,16 @@ xarxa corporativa molt intrusiva.">
 "Ãs segur, excepte si utilitzeu una xarxa corporativa amb noms secrets 
d'intranet.">
 
 
+"Envia i comprova els certificats firmats per CAs d'arrel no estÃ ndard.">
 
 
+"Ãs segur (i una bona idea) permetre aquesta opciÃ³, excepte si utilitzeu una 
xarxa corporativa molt intrusiva o un antivirus Kaspersky que monitoreja el 
navegador amb un servidor intermediari TLS i un Certificat d'Autoritat d'arrel 
privada. Si ho permeteu en aquesta xarxa, aquesta opciÃ³ pot publicar 
evidÃ¨ncies de quins dominis https:// sÃ³n visitats a travÃ©s d'aquest servidor 
intermediari, pel seu certificat Ãºnic. En aquest cas, Ã©s millor deixar-ho 
igual.">
 
-
+
 
+"Comprova els certificats fent servir Tor per a l'anonimat (requereix Tor)">
 
+"Aquesta opciÃ³ requereix que Tor estigui instalÂ·lat i en funcionament">
 
 

___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [translation/torcheck] Update translations for torcheck

2016-04-29 Thread translation

commit a2674dc2a96bf93dca1aa6ad3483e6095dfa4a3a
Author: Translation commit bot 
Date:   Fri Apr 29 17:15:09 2016 +

Update translations for torcheck
---
 ca/torcheck.po | 22 +++---
 1 file changed, 11 insertions(+), 11 deletions(-)

diff --git a/ca/torcheck.po b/ca/torcheck.po
index feee344..5fbc174 100644
--- a/ca/torcheck.po
+++ b/ca/torcheck.po
@@ -6,14 +6,14 @@
 # Bartges90, 2015
 # David Anglada , 2014
 # Humbert , 2014
-# laia_, 2014-2015
+# laia_, 2014-2016
 # Pau SellÃ©s i Garcia , 2013
 # Toni Hermoso Pulido , 2012
 msgid ""
 msgstr ""
 "Project-Id-Version: The Tor Project\n"
 "POT-Creation-Date: 2012-02-16 20:28+PDT\n"
-"PO-Revision-Date: 2016-03-21 16:27+\n"
+"PO-Revision-Date: 2016-04-29 16:52+\n"
 "Last-Translator: laia_\n"
 "Language-Team: Catalan 
(http://www.transifex.com/otf/torproject/language/ca/)\n"
 "MIME-Version: 1.0\n"
@@ -33,7 +33,7 @@ msgid ""
 msgstr "Visiteu el https://www.torproject.org/\\\;>lloc web de 
Tor per obtenir mÃ©s informaciÃ³ de com utilitzar Tor de manera segura. Ara 
teniu l'oportunitat de navegar per Internet anÃ²nimament."
 
 msgid "There is a security update available for Tor Browser."
-msgstr "Hi ha una actualitzaciÃ³ de seguretat disponible per al navegador Tor"
+msgstr "Hi ha una actualitzaciÃ³ de seguretat disponible per al Navegador Tor"
 
 msgid ""
 "https://www.torproject.org/download/download-easy.html\;>Click "
@@ -41,14 +41,14 @@ msgid ""
 msgstr "https://www.torproject.org/download/download-easy.html\\\;>Cliqueu 
aquÃ per anar a la pÃ gina de baixades"
 
 msgid "Sorry. You are not using Tor."
-msgstr "Ho sentim. No esteu utilitzant Tor."
+msgstr "No esteu utilitzant Tor."
 
 msgid ""
 "If you are attempting to use a Tor client, please refer to the https://www.torproject.org/\;>Tor website and specifically the instructions for "
 "configuring your Tor client."
-msgstr "Si esteu intentant utilitzar el client  Tor, visiteu el https://www.torproject.org/\\\;>lloc web de Tor i especÃficament 
les https://www.torproject.org/docs/faq#DoesntWork\\\;>instruccions 
per a configurar el client Tor."
+msgstr "Si esteu intentant utilitzar el client Tor, visiteu el https://www.torproject.org/\\\;>lloc web de Tor i especÃficament 
les https://www.torproject.org/docs/faq#DoesntWork\\\;>instruccions 
per a configurar el client Tor."
 
 msgid "Sorry, your query failed or an unexpected response was received."
 msgstr "La consulta ha fallat o s'ha rebut una resposta no esperada."
@@ -56,19 +56,19 @@ msgstr "La consulta ha fallat o s'ha rebut una resposta no 
esperada."
 msgid ""
 "A temporary service outage prevents us from determining if your source IP "
 "address is a https://www.torproject.org/\;>Tor node."
-msgstr "Una falla temporal del servei no ens permet determinar si la adreÃ§a 
IP d'origen Ã©s un node https://www.torproject.org/\\\;>Tor."
+msgstr "Una falla temporal del servei no ens permet determinar si l'adreÃ§a IP 
d'origen Ã©s un node https://www.torproject.org/\\\;>Tor."
 
 msgid "Your IP address appears to be: "
-msgstr "La vostra adreÃ§a IP sembla que Ã©s:"
+msgstr "Sembla que la vostra adreÃ§a IP sigui:"
 
 msgid "Are you using Tor?"
 msgstr "Esteu usant Tor?"
 
 msgid "This page is also available in the following languages:"
-msgstr "PÃ gina tambÃ© disponible en aquests idiomes:"
+msgstr "Aquesta pÃ gina tambÃ© estÃ  disponible en aquests idiomes:"
 
 msgid "For more information about this exit relay, see:"
-msgstr "Visiteu la segÃ¼ent pÃ gina per a mÃ©s informaciÃ³ sobre el trÃ fic de 
sortida:"
+msgstr "Per a mÃ©s informaciÃ³ sobre el trÃ fic de sortida visiteu:"
 
 msgid ""
 "The Tor Project is a US 501(c)(3) non-profit dedicated to the research, "
@@ -82,7 +82,7 @@ msgid "Go"
 msgstr "VÃ©s-hi"
 
 msgid "Short User Manual"
-msgstr "Manual d'usuari curt"
+msgstr "Manual d'usuari abreujat"
 
 msgid "Donate to Support Tor"
 msgstr "Donatiu per donar suport a Tor"
@@ -91,7 +91,7 @@ msgid "Tor Q Site"
 msgstr "Lloc de Q de Tor"
 
 msgid "Volunteer"
-msgstr "Voluntari"
+msgstr "Fes-te voluntari"
 
 msgid "JavaScript is enabled."
 msgstr "JavaScript estÃ  activat."

___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [translation/tails-persistence-setup] Update translations for tails-persistence-setup

2016-04-29 Thread translation

commit 26364c14d1b52986351317d51462e440db308057
Author: Translation commit bot 
Date:   Fri Apr 29 16:45:23 2016 +

Update translations for tails-persistence-setup
---
 ca/ca.po | 10 +-
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/ca/ca.po b/ca/ca.po
index 74592d1..a39b898 100644
--- a/ca/ca.po
+++ b/ca/ca.po
@@ -5,14 +5,14 @@
 # Translators:
 # Eloi GarcÃa i Fargas, 2014
 # Humbert , 2014
-# laia_, 2014-2015
+# laia_, 2014-2016
 msgid ""
 msgstr ""
 "Project-Id-Version: The Tor Project\n"
 "Report-Msgid-Bugs-To: Tails developers \n"
 "POT-Creation-Date: 2016-04-24 16:40+0200\n"
-"PO-Revision-Date: 2016-04-25 08:33+\n"
-"Last-Translator: carolyn \n"
+"PO-Revision-Date: 2016-04-29 16:42+\n"
+"Last-Translator: laia_\n"
 "Language-Team: Catalan 
(http://www.transifex.com/otf/torproject/language/ca/)\n"
 "MIME-Version: 1.0\n"
 "Content-Type: text/plain; charset=UTF-8\n"
@@ -118,11 +118,11 @@ msgstr "Llistes descarregades per APT"
 
 #: ../lib/Tails/Persistence/Configuration/Presets.pm:168
 msgid "Mumble server"
-msgstr ""
+msgstr "Servidor Mumble"
 
 #: ../lib/Tails/Persistence/Configuration/Presets.pm:170
 msgid "Mumble server configuration and its Tor Hidden Service address"
-msgstr ""
+msgstr "ConfiguraciÃ³ del servidor Mumble i la seva adreÃ§a de Tor Hidden 
Service"
 
 #: ../lib/Tails/Persistence/Configuration/Presets.pm:178
 msgid "Dotfiles"

___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [translation/tails-persistence-setup_completed] Update translations for tails-persistence-setup_completed

2016-04-29 Thread translation

commit 8fd6dff3fc2a7f5d6fadf80817547c158a27e37b
Author: Translation commit bot 
Date:   Fri Apr 29 16:45:26 2016 +

Update translations for tails-persistence-setup_completed
---
 ca/ca.po | 16 
 1 file changed, 12 insertions(+), 4 deletions(-)

diff --git a/ca/ca.po b/ca/ca.po
index 12862c4..a39b898 100644
--- a/ca/ca.po
+++ b/ca/ca.po
@@ -5,13 +5,13 @@
 # Translators:
 # Eloi GarcÃa i Fargas, 2014
 # Humbert , 2014
-# laia_, 2014-2015
+# laia_, 2014-2016
 msgid ""
 msgstr ""
 "Project-Id-Version: The Tor Project\n"
 "Report-Msgid-Bugs-To: Tails developers \n"
-"POT-Creation-Date: 2016-01-25 17:34+0100\n"
-"PO-Revision-Date: 2016-03-21 16:32+\n"
+"POT-Creation-Date: 2016-04-24 16:40+0200\n"
+"PO-Revision-Date: 2016-04-29 16:42+\n"
 "Last-Translator: laia_\n"
 "Language-Team: Catalan 
(http://www.transifex.com/otf/torproject/language/ca/)\n"
 "MIME-Version: 1.0\n"
@@ -117,10 +117,18 @@ msgid "Lists downloaded by APT"
 msgstr "Llistes descarregades per APT"
 
 #: ../lib/Tails/Persistence/Configuration/Presets.pm:168
+msgid "Mumble server"
+msgstr "Servidor Mumble"
+
+#: ../lib/Tails/Persistence/Configuration/Presets.pm:170
+msgid "Mumble server configuration and its Tor Hidden Service address"
+msgstr "ConfiguraciÃ³ del servidor Mumble i la seva adreÃ§a de Tor Hidden 
Service"
+
+#: ../lib/Tails/Persistence/Configuration/Presets.pm:178
 msgid "Dotfiles"
 msgstr "Dotfiles"
 
-#: ../lib/Tails/Persistence/Configuration/Presets.pm:170
+#: ../lib/Tails/Persistence/Configuration/Presets.pm:180
 msgid ""
 "Symlink into $HOME every file or directory found in the `dotfiles' directory"
 msgstr "Symlink a $HOME tot arxiu o carpeta trobat al directori 'dotfiles'"

___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [translation/tor-messenger-conversationsproperties] Update translations for tor-messenger-conversationsproperties

2016-04-29 Thread translation

commit e7c1ce85ff2c7ab7accd7b0163d681eb6b88a5e9
Author: Translation commit bot 
Date:   Fri Apr 29 16:16:21 2016 +

Update translations for tor-messenger-conversationsproperties
---
 ca/conversations.properties | 26 +-
 1 file changed, 13 insertions(+), 13 deletions(-)

diff --git a/ca/conversations.properties b/ca/conversations.properties
index b5079dc..98c40dc 100644
--- a/ca/conversations.properties
+++ b/ca/conversations.properties
@@ -26,46 +26,46 @@ statusChangedFromUnknownWithStatusText=%1$S Ã©s %2$S: %3$S.
 # special case of the previous 2 strings for when an account has just
 # been reconnected, so the status is now known. These 2 strings should not
 # mislead the user into thinking the person's status has just changed.
-statusKnown=Your account has been reconnected (%1$S is %2$S).
-statusKnownWithStatusText=Your account has been reconnected (%1$S is %2$S: 
%3$S).
+statusKnown=El vostre compte s'ha tornat a connectar (%1$S is %2$S).
+statusKnownWithStatusText=El vostre compte s'ha tornat a connectar (%1$S is 
%2$S: %3$S).
 # LOCALIZATION NOTE (statusUnknown):
 #  %S is the display name of the contact.
-statusUnknown=Your account is disconnected (the status of %S is no longer 
known).
+statusUnknown=El vostre compte estÃ  desconnectat (l'estat de %S ja no es 
coneix)
 
-accountDisconnected=La seva conta estÃ  desconnectada.
-accountReconnected=Your account has been reconnected.
+accountDisconnected=El vostre compte estÃ  desconnectar.
+accountReconnected=El vostre compte s'ha tornat a connectar.
 
 # LOCALIZATION NOTE (autoReply):
 #  %S is replaced by the text of a message that was sent as an automatic reply.
-autoReply=Auto-reply - %S
+autoReply=Autoresposta - %S 
 
 # LOCALIZATION NOTE (noTopic):
 # Displayed instead of the topic when no topic is set.
-noTopic=No topic message for this room.
+noTopic=No hi ha tema per aquest canal.
 
 # LOCALIZATION NOTE (topicSet):
 #  %1$S is the conversation name, %2$S is the topic.
-topicSet=The topic for %1$S is: %2$S.
+topicSet=El tema per a  %1$S Ã©s: %2$S.
 # LOCALIZATION NOTE (topicNotSet):
 #  %S is the conversation name.
-topicNotSet=There is no topic for %S.
+topicNotSet=No hi ha tema per a %S.
 # LOCALIZATION NOTE (topicChanged):
 #  %1$S is the user who changed the topic, %2$S is the new topic.
-topicChanged=%1$S has changed the topic to: %2$S.
+topicChanged=%1$S ha canviat el tema a: %2$S.
 # LOCALIZATION NOTE (topicCleared):
 #  %1$S is the user who cleared the topic.
-topicCleared=%1$S has cleared the topic.
+topicCleared=%1$S ha esborrat el tema.
 
 # LOCALIZATION NOTE (nickSet):
 #   This is displayed as a system message when a participant changes his/her
 #   nickname in a conversation.
 #   %1$S is the old nick.
 #   %2$S is the new nick.
-nickSet=%1$S is now known as %2$S.
+nickSet=%1$S ara Ã©s %2$S.
 # LOCALIZATION NOTE (nickSet.you):
 #   This is displayed as a system message when your nickname is changed.
 #   %S is your new nick.
-nickSet.you=You are now known as %S.
+nickSet.you=Ara sÃ³u %S.
 
 # LOCALIZATION NOTE (messenger.conversations.selections.ellipsis):
 #  ellipsis is used when copying a part of a message to show that the message 
was cut

___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [translation/tor-messenger-conversationsproperties_completed] Update translations for tor-messenger-conversationsproperties_completed

2016-04-29 Thread translation

commit f19bdf9749d8a5530eaf23daf64a0a4731374dde
Author: Translation commit bot 
Date:   Fri Apr 29 16:16:24 2016 +

Update translations for tor-messenger-conversationsproperties_completed
---
 ca/conversations.properties | 80 +
 1 file changed, 80 insertions(+)

diff --git a/ca/conversations.properties b/ca/conversations.properties
new file mode 100644
index 000..98c40dc
--- /dev/null
+++ b/ca/conversations.properties
@@ -0,0 +1,80 @@
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+# LOCALIZATION NOTE (targetChanged):
+#  %1$S is the new conversation title (display name of the new target),
+#  %2$S is the protocol name used for the new target.
+targetChanged=La conversa continuarÃ  amb %1$S, fent servir %2$S.
+
+# LOCALIZATION NOTE (statusChanged):
+#  %1$S is the display name of the contact.
+#  %2$S is the new status type (a value from status.properties).
+statusChanged=%1$S ara Ã©s %2$S.
+# LOCALIZATION NOTE (statusChangedWithStatusText):
+#  %1$S is the display name of the contact.
+#  %2$S is the new status type (a value from status.properties).
+#  %3$S is the status text (eg. "I'm currently away from the computer").
+statusChangedWithStatusText=%1$S ara Ã©s %2$S: %3$S.
+# LOCALIZATION NOTE (statusChangedFromUnknown[WithStatusText]):
+#  special case of the previous 2 strings for when the status was
+#  previously unknown. These 2 strings should not mislead the user
+#  into thinking the person's status has just changed.
+statusChangedFromUnknown=%1$S Ã©s %2$S.
+statusChangedFromUnknownWithStatusText=%1$S Ã©s %2$S: %3$S.
+# LOCALIZATION NOTE (statusKnown[WithStatusText]):
+# special case of the previous 2 strings for when an account has just
+# been reconnected, so the status is now known. These 2 strings should not
+# mislead the user into thinking the person's status has just changed.
+statusKnown=El vostre compte s'ha tornat a connectar (%1$S is %2$S).
+statusKnownWithStatusText=El vostre compte s'ha tornat a connectar (%1$S is 
%2$S: %3$S).
+# LOCALIZATION NOTE (statusUnknown):
+#  %S is the display name of the contact.
+statusUnknown=El vostre compte estÃ  desconnectat (l'estat de %S ja no es 
coneix)
+
+accountDisconnected=El vostre compte estÃ  desconnectar.
+accountReconnected=El vostre compte s'ha tornat a connectar.
+
+# LOCALIZATION NOTE (autoReply):
+#  %S is replaced by the text of a message that was sent as an automatic reply.
+autoReply=Autoresposta - %S 
+
+# LOCALIZATION NOTE (noTopic):
+# Displayed instead of the topic when no topic is set.
+noTopic=No hi ha tema per aquest canal.
+
+# LOCALIZATION NOTE (topicSet):
+#  %1$S is the conversation name, %2$S is the topic.
+topicSet=El tema per a  %1$S Ã©s: %2$S.
+# LOCALIZATION NOTE (topicNotSet):
+#  %S is the conversation name.
+topicNotSet=No hi ha tema per a %S.
+# LOCALIZATION NOTE (topicChanged):
+#  %1$S is the user who changed the topic, %2$S is the new topic.
+topicChanged=%1$S ha canviat el tema a: %2$S.
+# LOCALIZATION NOTE (topicCleared):
+#  %1$S is the user who cleared the topic.
+topicCleared=%1$S ha esborrat el tema.
+
+# LOCALIZATION NOTE (nickSet):
+#   This is displayed as a system message when a participant changes his/her
+#   nickname in a conversation.
+#   %1$S is the old nick.
+#   %2$S is the new nick.
+nickSet=%1$S ara Ã©s %2$S.
+# LOCALIZATION NOTE (nickSet.you):
+#   This is displayed as a system message when your nickname is changed.
+#   %S is your new nick.
+nickSet.you=Ara sÃ³u %S.
+
+# LOCALIZATION NOTE (messenger.conversations.selections.ellipsis):
+#  ellipsis is used when copying a part of a message to show that the message 
was cut
+messenger.conversations.selections.ellipsis=[...]
+
+# LOCALIZATION NOTE 
(messenger.conversations.selections.{system,content,action}MessagesTemplate):
+#  These 3 templates are used to format selected messages before copying them.
+#  Do not translate the texts between % characters, but feel free to adjust
+#  whitespace and separators to make them fit your locale.
+messenger.conversations.selections.systemMessagesTemplate=%time% - %message%
+messenger.conversations.selections.contentMessagesTemplate=%time% - %sender%: 
%message%
+messenger.conversations.selections.actionMessagesTemplate=%time% * %sender% 
%message%

___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [translation/tor-messenger-ircproperties] Update translations for tor-messenger-ircproperties

2016-04-29 Thread translation

commit 4ba76786e70eb42e31ace8e0b08d3216618e7a31
Author: Translation commit bot 
Date:   Fri Apr 29 16:16:35 2016 +

Update translations for tor-messenger-ircproperties
---
 ca/irc.properties | 80 +++
 1 file changed, 40 insertions(+), 40 deletions(-)

diff --git a/ca/irc.properties b/ca/irc.properties
index 6152d27..04d1227 100644
--- a/ca/irc.properties
+++ b/ca/irc.properties
@@ -120,83 +120,83 @@ message.alreadyInChannel=%1$S ja Ã©s a %2$S.
 #%S is the nickname of the user who was summoned.
 message.summoned=%S ha sigut cridat.
 #%S is the nickname of the user whose WHOIS information follows this 
message.
-message.whois=WHOIS information for %S:
+message.whois=InformaciÃ³ WHOIS per a %S:
 #%1$S is the nickname of the (offline) user whose WHOWAS information 
follows this message.
-message.whowas=%1$S is offline. WHOWAS information for %1$S:
+message.whowas=%1$S estÃ  fora de lÃnia. InformaciÃ³ WHOWAS per a %1$S:
 #%1$S is the entry description (from tooltip.*), %2$S is its value.
 message.whoisEntry=\\ua0\\ua0\\ua0\\ua0%1$S: %2$S
 #%S is the nickname that is not known to the server.
-message.unknownNick=%S is an unknown nickname.
+message.unknownNick=%S Ã©s un usuari desconnegut.
 #%1$S is the nickname of the user who changed the mode and %2$S is the new
 #channel key (password).
-message.channelKeyAdded=%1$S changed the channel password to %2$S.
-message.channelKeyRemoved=%S removed the channel password.
+message.channelKeyAdded=%1$S ha canviat la contrasenya del canal a %2$S.
+message.channelKeyRemoved=%S ha tret la contrasenya del canal.
 #This will be followed by a list of ban masks.
-message.banMasks=Users connected from the following locations are banned from 
%S:
-message.noBanMasks=There are no banned locations for %S.
-message.banMaskAdded=Users connected from locations matching %1$S have been 
banned by %2$S.
-message.banMaskRemoved=Users connected from locations matching %1$S are no 
longer banned by %2$S.
+message.banMasks=Els usuaris connectats des de les ubicacions segÃ¼ents estan 
bloquejats a %S:
+message.noBanMasks=No hi ha ubicacions bloquejades per a %s.
+message.banMaskAdded=Els usuaris connectats des de les ubicacions de %1$S han 
estat bloquejats per %2$S.
+message.banMaskRemoved=Els usuaris conectats des de les ubicacions de %1$S ja 
no estan bloquejats per %2$S.
 # LOCALIZATION NOTE (message.ping): Semi-colon list of plural forms.
 #  See: http://developer.mozilla.org/en/docs/Localization_and_Plurals
 #   %1$S is the nickname of the user or the server that was pinged.
 #   #2 is the delay (in milliseconds).
-message.ping=Ping reply from %1$S in #2 millisecond.;Ping reply from %1$S in 
#2 milliseconds.
+message.ping=la resposta ping de %1$S en #2 milÂ·lisegon.;La resposta ping de 
%1$S en #2 milÂ·lisegons.
 
 
 # LOCALIZATION NOTE (error.*):
 #These are shown as error messages in the conversation or server tab.
 #%S is the channel name.
-error.noChannel=There is no channel: %S.
-error.tooManyChannels=Cannot join %S; you've joined too many channels.
+error.noChannel=No hi ha cap canal: %S.
+error.tooManyChannels=No es pot entrar a %S: Has entrat a massa canals. 
 #%1$S is your new nick, %2$S is the kill message from the server.
-error.nickCollision=Nick already in use, changing nick to %1$S [%2$S].
-error.erroneousNickname=%S is not an allowed nickname.
-error.banned=You are banned from this server.
-error.bannedSoon=You will soon be banned from this server.
-error.mode.wrongUser=You cannot change modes for other users.
+error.nickCollision=Aquest sobrenom ja s'estÃ  usant, canviant el sobrenom a 
%1$S [%2$S].
+error.erroneousNickname=%S no Ã©s un sobrenom permÃ¨s.
+error.banned=Esteu bloquejat d'aquest servidor. 
+error.bannedSoon=Aviat estareu bloquejat d'aquest servidor.
+error.mode.wrongUser=No podeu canviar els modes dels altres usuaris.
 #%S is the nickname or channel name that isn't available.
 error.noSuchNick=%S no estÃ  en lÃnia
-error.wasNoSuchNick=There was no nickname: %S
-error.noSuchChannel=There is no channel: %S.
-error.unavailable=%S is temporarily unavailable.
+error.wasNoSuchNick=No hi havia cap sobrenom: %S
+error.noSuchChannel=No hi ha cap canal: %S.
+error.unavailable=%S no estÃ  disponible ara mateix.
 #%S is the channel name.
-error.channelBanned=You have been banned from %S.
-error.cannotSendToChannel=You cannot send messages to %S.
-error.channelFull=The channel %S is full.
-error.inviteOnly=You must be invited to join %S.
-error.nonUniqueTarget=%S is not a unique user@host or shortname or you have 
tried to join too many channels at once.
-error.notChannelOp=You are not a channel operator on %S.
-error.notChannelOwner=You are not a channel owner of %S.
-error.wrongKey=Cannot join %S, invalid channel password.
-error.sendMessageFailed=An error occurred while sending your last message. 
Please try again once the

[tor-commits] [translation/tor-messenger-ircproperties_completed] Update translations for tor-messenger-ircproperties_completed

2016-04-29 Thread translation

commit ba0afa573da971f1c5b38d7d5662e8429a4001f0
Author: Translation commit bot 
Date:   Fri Apr 29 16:16:37 2016 +

Update translations for tor-messenger-ircproperties_completed
---
 ca/irc.properties | 208 ++
 1 file changed, 208 insertions(+)

diff --git a/ca/irc.properties b/ca/irc.properties
new file mode 100644
index 000..04d1227
--- /dev/null
+++ b/ca/irc.properties
@@ -0,0 +1,208 @@
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+# LOCALIZATION NOTE (irc.usernameHint):
+#  This is displayed inside the accountUsernameInfoWithDescription
+#  string defined in imAccounts.properties when the user is
+#  configuring an IRC account.
+irc.usernameHint=Sobrenom
+
+# LOCALIZATION NOTE (connection.error.*):
+#   These will show in the account manager if the account is
+#   disconnected because of an error.
+connection.error.lost=S'ha perdut la connexiÃ³ amb el servidor
+connection.error.timeOut=La connexiÃ³ ha esgotat el temps.
+connection.error.invalidUsername=%S no Ã©s un nom d'usuari permÃ¨s
+connection.error.invalidPassword= La contrasenya de servidor no Ã©s vÃ lida
+connection.error.passwordRequired=Es requereix contrasenya
+
+# LOCALIZATION NOTE (joinChat.*):
+#   These show up on the join chat menu. An underscore is for the access key.
+joinChat.channel=_Canal
+joinChat.password=_Constrasenya
+
+# LOCALIZATION NOTE (options.*):
+#   These are the protocol specific options shown in the account manager and
+#   account wizard windows.
+options.server=Servidor
+options.port=Port
+options.ssl=Utilitza SSL
+options.encoding=Joc de carÃ cters
+options.quitMessage=Surt del missatge
+options.partMessage=Divideix el missatge
+options.showServerTab=Mostra  els missatges del servidor
+options.alternateNicks=Alterna els sobrenoms
+
+# LOCALIZATION NOTE (ctcp.version):
+#   %1$S is the nickname of the user whose version was requested.
+#   %2$S is the version response from the client.
+ctcp.version=%1$S estÃ  usant "%2$S".
+# LOCALIZATION NOTE (ctcp.time):
+#   %1$S is the nickname of the user whose time was requested.
+#   %2$S is the time response.
+ctcp.time=El temps per a  %1$S Ã©s %2$S.
+
+# LOCALZIATION NOTE (command.*):
+#   These are the help messages for each command, the %S is the command name
+#   Each command first gives the parameter it accepts and then a description of
+#   the command.
+command.action=%S action to perform: Duu a terme una acciÃ³.
+command.ctcp=%S nick msg: Envia un missatge CTCP a l'usuari.
+command.chanserv=%S command: Envia una ordre al ChanServ.
+command.deop=%S nick1[,nick2]*: Treu l'estat d'operador de 
canal a algÃº. Cal ser un operador de canal per fer-ho.
+command.devoice=%S nick1[,nick2]*: Treu l'estat de veu al 
canal a algÃº, evitant que pugui parlar al canal si aquest estÃ  moderat (+m). 
Cal ser un operador de canal per fer-ho.
+command.invite2=%S nick[ nick]* [channel]: Convideu un 
o mÃ©s usuaris a unir-se al canal, o uniu-vos al canal especificat.
+command.join=%S room1[ key1][,room2[ key2]]*: 
Entreu a un o mÃ©s canals, proveint de manera opcional la clau del canal si Ã©s 
necessÃ ria.
+command.kick=%S nick [message]: Expulseu algÃº del canal. Cal 
ser un operador de canal per fer-ho. 
+command.list=%S: Mostra una llista de sales de xat de la xarxa. Compte, alguns 
servidors poden desconectar-vos per fer-ho. 
+command.memoserv=%S command: Envia una ordre a MemoServ. 
+command.modeUser=%S (+|-)new mode [nick]: Configura el mode 
d'usuari.
+command.modeChannel=%S channel[ (+|-)new mode 
[parameter][,parameter]*]: Configura els modes del canal.
+command.msg=%S nick message: Envia un missatge privat a un 
usuari (en contraposiciÃ³ al canal).
+command.nick=%S new nickname: Canvia el nom d'usuari.
+command.nickserv=%S command: Envia una ordre al NickServ.
+command.notice=%S target message: Envia una notificaciÃ³ a un 
usuari o a un canal.
+command.op=%S nick1[,nick2]*: Otorga l'estat d'operador de 
canal a algÃº. Cal ser un operador de canal per fer-ho. 
+command.operserv=%S command: Envia una ordre a l'OperServ.
+command.part=%S [message]: Marxa del canal actual amb un missatge opcional.
+command.ping=%S [nick]: Pregunta a l'usuari quant lag tÃ© (o al 
servidor, si no s'ha especificat usuari).
+command.quit=%S message: Desconnecta del servidor amb un missatge 
opcional.
+command.quote=%S command: Envia una ordre al servidor.
+command.time=%S: Mostra l'hora actual al servidor IRC.
+command.topic=%S [new topic]: Canvia el tema del canal.
+command.umode=%S (+|-)new mode: Especifica o treu un mode d'usuari.
+command.version=%S nick: Demana la versiÃ³ del client d'un usuari.
+command.voice=%S nick1[,nick2]*: DÃ³na l'estat de veu al canal 
a un usuari. Cal ser un operador de canal per fer-ho.
+command.whois2=%S [nick]: Mostra

[tor-commits] [translation/tails-perl5lib] Update translations for tails-perl5lib

2016-04-29 Thread translation

commit 61a73f6aedde439fa7cefab259168589615c4ff8
Author: Translation commit bot 
Date:   Fri Apr 29 15:45:58 2016 +

Update translations for tails-perl5lib
---
 ca.po | 12 ++--
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/ca.po b/ca.po
index 2cb0588..91a2541 100644
--- a/ca.po
+++ b/ca.po
@@ -4,14 +4,14 @@
 # 
 # Translators:
 # Carlos Marin , 2016
-# laia_, 2014-2015
+# laia_, 2014-2016
 msgid ""
 msgstr ""
 "Project-Id-Version: The Tor Project\n"
 "Report-Msgid-Bugs-To: Tails developers \n"
-"POT-Creation-Date: 2016-01-25 16:59+0100\n"
-"PO-Revision-Date: 2016-03-21 16:27+\n"
-"Last-Translator: Carlos Marin \n"
+"POT-Creation-Date: 2016-04-24 16:36+0200\n"
+"PO-Revision-Date: 2016-04-29 15:37+\n"
+"Last-Translator: laia_\n"
 "Language-Team: Catalan 
(http://www.transifex.com/otf/torproject/language/ca/)\n"
 "MIME-Version: 1.0\n"
 "Content-Type: text/plain; charset=UTF-8\n"
@@ -27,10 +27,10 @@ msgstr "Error"
 msgid ""
 "The device Tails is running from cannot be found. Maybe you used the `toram'"
 " option?"
-msgstr "El dispositiu des d'on Tails estÃ  funcionant no s'ha trobat. Potser 
heu usat l'opciÃ³ \"toram\"?"
+msgstr "El dispositiu des d'on s'executa el Tails no s'ha trobat. Heu fet 
servir l'opciÃ³ Toram?"
 
 #: ../lib/Tails/RunningSystem.pm:192
 msgid ""
 "The drive Tails is running from cannot be found. Maybe you used the `toram' "
 "option?"
-msgstr ""
+msgstr "La unitat des d'on s'executa el Tails no s'ha trobat. Heu fet servir 
l'opciÃ³ Toram?"

___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [translation/tor-messenger-ircproperties] Update translations for tor-messenger-ircproperties

2016-04-29 Thread translation

commit c5f56439de78c7a0bbc2385c9afa1f8bcc1ac285
Author: Translation commit bot 
Date:   Fri Apr 29 15:46:37 2016 +

Update translations for tor-messenger-ircproperties
---
 ca/irc.properties | 52 ++--
 1 file changed, 26 insertions(+), 26 deletions(-)

diff --git a/ca/irc.properties b/ca/irc.properties
index 55aa9eb..6152d27 100644
--- a/ca/irc.properties
+++ b/ca/irc.properties
@@ -15,7 +15,7 @@ connection.error.lost=S'ha perdut la connexiÃ³ amb el servidor
 connection.error.timeOut=La connexiÃ³ ha esgotat el temps.
 connection.error.invalidUsername=%S no Ã©s un nom d'usuari permÃ¨s
 connection.error.invalidPassword= La contrasenya de servidor no Ã©s vÃ lida
-connection.error.passwordRequired=Contrasenya requerida
+connection.error.passwordRequired=Es requereix contrasenya
 
 # LOCALIZATION NOTE (joinChat.*):
 #   These show up on the join chat menu. An underscore is for the access key.
@@ -31,8 +31,8 @@ options.ssl=Utilitza SSL
 options.encoding=Joc de carÃ cters
 options.quitMessage=Surt del missatge
 options.partMessage=Divideix el missatge
-options.showServerTab=Mostra missatges des del servidor
-options.alternateNicks=Alterna sobrenoms
+options.showServerTab=Mostra  els missatges del servidor
+options.alternateNicks=Alterna els sobrenoms
 
 # LOCALIZATION NOTE (ctcp.version):
 #   %1$S is the nickname of the user whose version was requested.
@@ -47,7 +47,7 @@ ctcp.time=El temps per a  %1$S Ã©s %2$S.
 #   These are the help messages for each command, the %S is the command name
 #   Each command first gives the parameter it accepts and then a description of
 #   the command.
-command.action=%S action to perform: Duu a terme una acciÃ³
+command.action=%S action to perform: Duu a terme una acciÃ³.
 command.ctcp=%S nick msg: Envia un missatge CTCP a l'usuari.
 command.chanserv=%S command: Envia una ordre al ChanServ.
 command.deop=%S nick1[,nick2]*: Treu l'estat d'operador de 
canal a algÃº. Cal ser un operador de canal per fer-ho.
@@ -57,10 +57,10 @@ command.join=%S room1[ 
key1][,room2[ key2]]*: En
 command.kick=%S nick [message]: Expulseu algÃº del canal. Cal 
ser un operador de canal per fer-ho. 
 command.list=%S: Mostra una llista de sales de xat de la xarxa. Compte, alguns 
servidors poden desconectar-vos per fer-ho. 
 command.memoserv=%S command: Envia una ordre a MemoServ. 
-command.modeUser=%S (+|-)new mode [nick]: Configura el mode 
d'usuari
+command.modeUser=%S (+|-)new mode [nick]: Configura el mode 
d'usuari.
 command.modeChannel=%S channel[ (+|-)new mode 
[parameter][,parameter]*]: Configura els modes del canal.
-command.msg=%S nick message: Envia un missatge privat a un 
usuari (en contraposiciÃ³ al canal)
-command.nick=%S new nickname: Canvia el teu nom d'usuari
+command.msg=%S nick message: Envia un missatge privat a un 
usuari (en contraposiciÃ³ al canal).
+command.nick=%S new nickname: Canvia el nom d'usuari.
 command.nickserv=%S command: Envia una ordre al NickServ.
 command.notice=%S target message: Envia una notificaciÃ³ a un 
usuari o a un canal.
 command.op=%S nick1[,nick2]*: Otorga l'estat d'operador de 
canal a algÃº. Cal ser un operador de canal per fer-ho. 
@@ -71,54 +71,54 @@ command.quit=%S message: Desconnecta del servidor 
amb un missatge opcion
 command.quote=%S command: Envia una ordre al servidor.
 command.time=%S: Mostra l'hora actual al servidor IRC.
 command.topic=%S [new topic]: Canvia el tema del canal.
-command.umode=%S (+|-)new mode: Set or unset a user mode.
-command.version=%S nick: Request the version of a user's client.
-command.voice=%S nick1[,nick2]*: Grant channel voice status to 
someone. You must be a channel operator to do this.
-command.whois2=%S [nick]: Get information on a user.
+command.umode=%S (+|-)new mode: Especifica o treu un mode d'usuari.
+command.version=%S nick: Demana la versiÃ³ del client d'un usuari.
+command.voice=%S nick1[,nick2]*: DÃ³na l'estat de veu al canal 
a un usuari. Cal ser un operador de canal per fer-ho.
+command.whois2=%S [nick]: Mostra informaciÃ³ d'un usuari.
 
 # LOCALIZATION NOTE (message.*):
 #These are shown as system messages in the conversation.
 #%1$S is the nick and %2$S is the nick and host of the user who joined.
-message.join=%1$S [%2$S] entered the room.
-message.rejoined=You have rejoined the room.
+message.join=%1$S [%2$S] ha entrat al canal.
+message.rejoined=Has tornat a entrar al canal.
 #%1$S is the nick of who kicked you.
 #%2$S is message.kicked.reason, if a kick message was given.
-message.kicked.you=You have been kicked by %1$S%2$S.
+message.kicked.you=Has estat expulsat per %1$S%2$S.
 #%1$S is the nick that is kicked, %2$S the nick of the person who kicked
 #%1$S. %3$S is message.kicked.reason, if a kick message was given.
-message.kicked=%1$S has been kicked by %2$S%3$S.
+message.kicked=%1$S ha estat expulsat per %2$S%3$S.
 #%S is the kick message
 message.kicked.reason=: %S
 #%1$S is

[tor-commits] [translation/tails-perl5lib_completed] Update translations for tails-perl5lib_completed

2016-04-29 Thread translation

commit b2ec0141762420f6a46a4bcdae118b3fb989ada7
Author: Translation commit bot 
Date:   Fri Apr 29 15:46:02 2016 +

Update translations for tails-perl5lib_completed
---
 ca.po | 25 ++---
 1 file changed, 18 insertions(+), 7 deletions(-)

diff --git a/ca.po b/ca.po
index 5547ad0..91a2541 100644
--- a/ca.po
+++ b/ca.po
@@ -3,23 +3,34 @@
 # This file is distributed under the same license as the PACKAGE package.
 # 
 # Translators:
-# laia_ , 2014-2015
+# Carlos Marin , 2016
+# laia_, 2014-2016
 msgid ""
 msgstr ""
 "Project-Id-Version: The Tor Project\n"
 "Report-Msgid-Bugs-To: Tails developers \n"
-"POT-Creation-Date: 2014-06-24 11:37+0200\n"
-"PO-Revision-Date: 2015-04-12 02:08+\n"
-"Last-Translator: laia_ \n"
-"Language-Team: Catalan 
(http://www.transifex.com/projects/p/torproject/language/ca/)\n"
+"POT-Creation-Date: 2016-04-24 16:36+0200\n"
+"PO-Revision-Date: 2016-04-29 15:37+\n"
+"Last-Translator: laia_\n"
+"Language-Team: Catalan 
(http://www.transifex.com/otf/torproject/language/ca/)\n"
 "MIME-Version: 1.0\n"
 "Content-Type: text/plain; charset=UTF-8\n"
 "Content-Transfer-Encoding: 8bit\n"
 "Language: ca\n"
 "Plural-Forms: nplurals=2; plural=(n != 1);\n"
 
-#: ../lib/Tails/RunningSystem.pm:153
+#: ../lib/Tails/RunningSystem.pm:159
+msgid "Error"
+msgstr "Error"
+
+#: ../lib/Tails/RunningSystem.pm:161
 msgid ""
 "The device Tails is running from cannot be found. Maybe you used the `toram'"
 " option?"
-msgstr "El dispositiu des d'on Tails estÃ  funcionant no s'ha trobat. Potser 
heu usat l'opciÃ³ \"toram\"?"
+msgstr "El dispositiu des d'on s'executa el Tails no s'ha trobat. Heu fet 
servir l'opciÃ³ Toram?"
+
+#: ../lib/Tails/RunningSystem.pm:192
+msgid ""
+"The drive Tails is running from cannot be found. Maybe you used the `toram' "
+"option?"
+msgstr "La unitat des d'on s'executa el Tails no s'ha trobat. Heu fet servir 
l'opciÃ³ Toram?"

___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [translation/tor-launcher-network-settings_completed] Update translations for tor-launcher-network-settings_completed

2016-04-29 Thread translation

commit b6d538d53d1c7f74fbbbc86e36eaa827990374fd
Author: Translation commit bot 
Date:   Fri Apr 29 14:45:46 2016 +

Update translations for tor-launcher-network-settings_completed
---
 ca/network-settings.dtd | 59 +++--
 1 file changed, 33 insertions(+), 26 deletions(-)

diff --git a/ca/network-settings.dtd b/ca/network-settings.dtd
index 8ef50cf..bff8120 100644
--- a/ca/network-settings.dtd
+++ b/ca/network-settings.dtd
@@ -11,59 +11,66 @@
 
 
 
-
-
-
+
+
+
 
-
+
 
-
+
 
 
-
+
 
-
-
+
+
+
 
 
-
-
+
+
+
 
 
 
 
 
+
+
+
+
 
 
 
-
-
+
+
 
-
+
 
 
 
 
 
 
-
+
 
 
-
+
+
 
-
-
+
+
 
 
-
-
-
-
+
+
+
+
 
-https://bridges.torproject.org;>
-
-
+https://bridges.torproject.org;>
+
+
 https://www.riseup.net, 
https://mail.google.com, o https://mail.yahoo.com;>
-
-
+
+

___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [translation/tor-launcher-network-settings] Update translations for tor-launcher-network-settings

2016-04-29 Thread translation

commit 0eebda0e3b042fa0ccb973dc9fc154d6009f457b
Author: Translation commit bot 
Date:   Fri Apr 29 14:45:42 2016 +

Update translations for tor-launcher-network-settings
---
 ca/network-settings.dtd | 64 -
 1 file changed, 32 insertions(+), 32 deletions(-)

diff --git a/ca/network-settings.dtd b/ca/network-settings.dtd
index 9ebf4d2..bff8120 100644
--- a/ca/network-settings.dtd
+++ b/ca/network-settings.dtd
@@ -11,66 +11,66 @@
 
 
 
-
-
-
+
+
+
 
-
+
 
-
+
 
 
-
+
 
-
-
-
+
+
+
 
 
-
-
-
+
+
+
 
 
 
 
 
-
+
 
-
-
+
+
 
 
 
-
-
+
+
 
-
+
 
 
 
 
 
 
-
+
 
 
-
-
+
+
 
-
-
+
+
 
 
-
-
-
-
+
+
+
+
 
-https://bridges.torproject.org;>
-
-
+https://bridges.torproject.org;>
+
+
 https://www.riseup.net, 
https://mail.google.com, o https://mail.yahoo.com;>
-
-
+
+

___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [translation/https_everywhere] Update translations for https_everywhere

2016-04-29 Thread translation

commit fe8b7c0c7ecb05f6c184d3d257d97d9da1c0b10b
Author: Translation commit bot 
Date:   Fri Apr 29 11:45:16 2016 +

Update translations for https_everywhere
---
 ja/https-everywhere.dtd | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/ja/https-everywhere.dtd b/ja/https-everywhere.dtd
index 1ddd984..0e196fa 100644
--- a/ja/https-everywhere.dtd
+++ b/ja/https-everywhere.dtd
@@ -48,7 +48,7 @@
 
 
 
-
+
 
 
 

___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [collector/master] Prefix entries in .gitignore with slash.

2016-04-29 Thread karsten

commit b41a71cd70d69e9847d4b3907015cc3217844a6a
Author: Karsten Loesing 
Date:   Fri Apr 29 00:14:43 2016 +0200

Prefix entries in .gitignore with slash.

We really only want to ignore files and directories in the current
directory, not files and directories in subdirectories.
---
 .gitignore | 26 +-
 1 file changed, 13 insertions(+), 13 deletions(-)

diff --git a/.gitignore b/.gitignore
index ec941d0..a31fb1c 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,14 +1,14 @@
-.classpath
-.project
-classes/
-config
-data/
-in/
-out/
-log/
-rsync/
-stats/
-tarballs/
-generated
-lib
+/.classpath
+/.project
+/classes/
+/config
+/data/
+/in/
+/out/
+/log/
+/rsync/
+/stats/
+/tarballs/
+/generated
+/lib
 

___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] Check to ensure response_body is either a str or unicode before sanitising it (#485)

2016-04-29 Thread art

commit 4598237c15331cf97fb99b53593b99e8fb182ee7
Author: Arturo FilastÃ² 
Date:   Thu Apr 28 12:10:15 2016 +0200

Check to ensure response_body is either a str or unicode before sanitising 
it (#485)
---
 ooni/templates/httpt.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/ooni/templates/httpt.py b/ooni/templates/httpt.py
index fd6c8ab..e608f8f 100644
--- a/ooni/templates/httpt.py
+++ b/ooni/templates/httpt.py
@@ -172,7 +172,8 @@ class HTTPTest(NetTestCase):
 else:
 response_body = ''
 # Attempt to redact the IP address of the probe from the responses
-if config.privacy.includeip is False and config.probe_ip.address 
is not None:
+if (config.privacy.includeip is False and config.probe_ip.address 
is not None and
+(isinstance(response_body, str) or 
isinstance(response_body, unicode))):
 response_body = response_body.replace(config.probe_ip.address, 
"[REDACTED]")
 session['response'] = {
 'headers': _representHeaders(response.headers),



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] Use the official scapy repository instead of the dev fork

2016-04-29 Thread art

commit 8af20fbfad44ef87564dbb4c421d539f4aa76016
Author: Arturo FilastÃ² 
Date:   Fri Apr 29 11:26:28 2016 +0200

Use the official scapy repository instead of the dev fork
---
 requirements.txt | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/requirements.txt b/requirements.txt
index 7aeaabc..7107d9e 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -11,7 +11,7 @@ pyOpenSSL>=0.13
 geoip
 txtorcon>=0.7
 txsocksx>=0.0.2
-scapy-real>=2.2.0-dev
+scapy>=2.2.0
 pypcap>=1.1
 service-identity
 pydumbnet



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] Properly represent also the packets that are sent

2016-04-29 Thread art

commit 8215d8807b3f6c43c766baf4a04f454d83038eea
Author: Arturo FilastÃ² 
Date:   Fri Apr 29 10:51:08 2016 +0200

Properly represent also the packets that are sent
---
 ooni/templates/scapyt.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/ooni/templates/scapyt.py b/ooni/templates/scapyt.py
index f4b6280..2bb9139 100644
--- a/ooni/templates/scapyt.py
+++ b/ooni/templates/scapyt.py
@@ -148,7 +148,7 @@ class BaseScapyTest(NetTestCase):
 
 scapySender.stopSending()
 for sent_packet in packets:
-self.report['sent_packets'].append(sent_packet)
+self.report['sent_packets'].append(_representPacket(sent_packet))
 
 
 ScapyTest = BaseScapyTest



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] Update Changelog and put a release date on it.

2016-04-29 Thread art

commit 3ca8cb86c732fef3d3c27221b0025ad953465e70
Author: Arturo FilastÃ² 
Date:   Tue Apr 26 23:22:39 2016 +0200

Update Changelog and put a release date on it.
---
 ChangeLog.rst | 7 ++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/ChangeLog.rst b/ChangeLog.rst
index 47851e1..82d04cd 100644
--- a/ChangeLog.rst
+++ b/ChangeLog.rst
@@ -1,7 +1,7 @@
 Changelog
 =
 
-v1.4.0 (XXX, XX Apr 2016)
+v1.4.0 (Wed, 27 Apr 2016)
 -
 
 * Support for reporting using JSON
@@ -12,11 +12,16 @@ v1.4.0 (XXX, XX Apr 2016)
 
 * OpenVPN censorship circumvention test
 
+* Add test for vanilla Tor
+
 * Support for disabling reporting to disk
 
 * Improvements to HTTP response body decoding (includes fix that lead to empty
   bodies being misrepresented)
 
+* Attempt to scrub the probe IP address from the body of HTTP responses
+
+
 v1.3.2 (Fri, 20 Nov 2015)
 -
 



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] Bump to version 1.4.2

2016-04-29 Thread art

commit 4ddac3ecf1aa4e797214cbbcbb6a7930adc0c0f8
Author: Arturo FilastÃ² 
Date:   Fri Apr 29 11:28:22 2016 +0200

Bump to version 1.4.2
---
 ChangeLog.rst| 8 
 ooni/__init__.py | 2 +-
 2 files changed, 9 insertions(+), 1 deletion(-)

diff --git a/ChangeLog.rst b/ChangeLog.rst
index c24d46d..56dd061 100644
--- a/ChangeLog.rst
+++ b/ChangeLog.rst
@@ -1,6 +1,14 @@
 Changelog
 =
 
+v1.4.2 (Fri, 29 Apr 2016)
+-
+
+* Hotfix for bug in serialising binary response bodies
+
+* Use the most recent scapy version
+
+
 v1.4.1 (Wed, 27 Apr 2016)
 -
 
diff --git a/ooni/__init__.py b/ooni/__init__.py
index cbf84b7..8e05068 100644
--- a/ooni/__init__.py
+++ b/ooni/__init__.py
@@ -1,7 +1,7 @@
 # -*- encoding: utf-8 -*-
 
 __author__ = "Open Observatory of Network Interference"
-__version__ = "1.4.1"
+__version__ = "1.4.2"
 
 __all__ = ['config', 'inputunit', 'kit',
'lib', 'nettest', 'oonicli', 'report', 'reporter',

___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] Improve the installation instructions for ooniprobe

2016-04-29 Thread art

commit 652fe233c4b74011841420d947485be6779c0733
Author: Arturo FilastÃ² 
Date:   Wed Apr 27 13:21:15 2016 +0200

Improve the installation instructions for ooniprobe
---
 README.rst | 9 +
 1 file changed, 9 insertions(+)

diff --git a/README.rst b/README.rst
index a6dcb2c..b42d755 100644
--- a/README.rst
+++ b/README.rst
@@ -107,10 +107,19 @@ Make sure you have installed the following dependencies:
   * libffi-dev
   * tor (>=0.2.5.1 to run all the tor related tests)
 
+On debian based systems this can generally be done by running::
+
+sudo apt-get install -y build-essential libdumbnet-dev libpcap-dev 
libgeoip-dev libffi-dev python-dev python-pip
+
 Then you should be able to install ooniprobe by running::
 
+sudo pip install ooniprobe
+
+or as a user to install inside a virtualenv::
+
 pip install ooniprobe
 
+
 Other platforms (with Vagrant)
 --
 



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] Bump to version 1.4.0

2016-04-29 Thread art

commit 943dbec8d344be7b02fb7f6e8ae6f9e0fc6fcbe1
Author: Arturo FilastÃ² 
Date:   Wed Apr 27 19:48:05 2016 +0200

Bump to version 1.4.0
---
 ooni/__init__.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/ooni/__init__.py b/ooni/__init__.py
index a9c6112..5ccd2da 100644
--- a/ooni/__init__.py
+++ b/ooni/__init__.py
@@ -1,7 +1,7 @@
 # -*- encoding: utf-8 -*-
 
 __author__ = "Open Observatory of Network Interference"
-__version__ = "1.3.2"
+__version__ = "1.4.0"
 
 __all__ = ['config', 'inputunit', 'kit',
'lib', 'nettest', 'oonicli', 'report', 'reporter',



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] The upload of the release failed and hence need to re-issue upload

2016-04-29 Thread art

commit bff00ebbf32730f0d2a44a41e40a7c5e3b0027c0
Author: Arturo FilastÃ² 
Date:   Wed Apr 27 20:27:24 2016 +0200

The upload of the release failed and hence need to re-issue upload
---
 ChangeLog.rst| 6 ++
 ooni/__init__.py | 2 +-
 2 files changed, 7 insertions(+), 1 deletion(-)

diff --git a/ChangeLog.rst b/ChangeLog.rst
index 471e282..c24d46d 100644
--- a/ChangeLog.rst
+++ b/ChangeLog.rst
@@ -1,6 +1,12 @@
 Changelog
 =
 
+v1.4.1 (Wed, 27 Apr 2016)
+-
+
+* Fix problem with uploading of release
+
+
 v1.4.0 (Wed, 27 Apr 2016)
 -
 
diff --git a/ooni/__init__.py b/ooni/__init__.py
index 5ccd2da..cbf84b7 100644
--- a/ooni/__init__.py
+++ b/ooni/__init__.py
@@ -1,7 +1,7 @@
 # -*- encoding: utf-8 -*-
 
 __author__ = "Open Observatory of Network Interference"
-__version__ = "1.4.0"
+__version__ = "1.4.1"
 
 __all__ = ['config', 'inputunit', 'kit',
'lib', 'nettest', 'oonicli', 'report', 'reporter',



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] Use the process template in the netalyzr test

2016-04-29 Thread art

commit 51c9b63c9cbf4499f43ec497e883afbfee8388c8
Author: Arturo FilastÃ² 
Date:   Wed Apr 27 12:07:02 2016 +0200

Use the process template in the netalyzr test

* Clean up text of psiphon test
---
 ooni/nettests/third_party/netalyzr.py | 77 +++
 ooni/nettests/third_party/psiphon.py  |  2 +-
 2 files changed, 43 insertions(+), 36 deletions(-)

diff --git a/ooni/nettests/third_party/netalyzr.py 
b/ooni/nettests/third_party/netalyzr.py
index 3154af7..ee5c477 100644
--- a/ooni/nettests/third_party/netalyzr.py
+++ b/ooni/nettests/third_party/netalyzr.py
@@ -6,55 +6,62 @@
 #   Arturo "hellais" FilastÃ² 
 # :licence: see LICENSE
 
-from ooni import nettest
-from ooni.utils import log
 import time
 import os
+import distutils.spawn
+
+from twisted.python import usage
 from twisted.internet import reactor, threads
 
-class NetalyzrWrapperTest(nettest.NetTestCase):
-name = "NetalyzrWrapper"
-requiresRoot = False
-requiresTor = False
+from ooni.templates import process
+from ooni.utils import log
 
-def setUp(self):
-cwd = os.path.abspath(os.path.join(os.path.abspath(__file__), '..'))
+class JavaNotInstalled(Exception):
+pass
 
-# XXX set the output directory to something more uniform
-outputdir = os.path.join(cwd, '..', '..')
+class CouldNotFindNetalyzrCli(Exception):
+pass
 
-program_path = os.path.join(cwd, 'NetalyzrCLI.jar')
-program = "java -jar %s -d" % program_path
+class UsageOptions(usage.Options):
+optParameters = [
+['clipath', 'p', None, 'Specify the path to NetalyzrCLI.jar (can be '
+   'downloaded from '
+   
'http://netalyzr.icsi.berkeley.edu/NetalyzrCLI.jar)']
+]
 
-test_token = time.asctime(time.gmtime()).replace(" ", "_").strip()
+class NetalyzrWrapperTest(process.ProcessTest):
+name = "NetalyzrWrapper"
+description = "A wrapper around the Netalyzr java command line client"
+author = "Jacob Appelbaum "
+
+requiredOptions = ['clipath']
 
-self.output_file = os.path.join(outputdir,
-"NetalyzrCLI_" + test_token + ".out")
-self.output_file.strip()
-self.run_me = program + " 2>&1 >> " + self.output_file
+usageOptions = UsageOptions
+requiresRoot = False
+requiresTor = False
 
-def blocking_call(self):
-try:
-result = threads.blockingCallFromThread(reactor, os.system, 
self.run_me) 
-except:
-log.debug("Netalyzr had an error, please see the log file: %s" % 
self.output_file)
-finally:
-self.clean_up()
+timeout = 300
+
+def requirements(self):
+if not distutils.spawn.find_executable("java"):
+raise JavaNotInstalled("Java is not installed.")
+
+def setUp(self):
+if not os.path.exists(self.localOptions['clipath']):
+raise CouldNotFindNetalyzrCli("Could not find NetalyzrCLI.jar at 
{}".format(self.localOptions['clipath']))
 
-def clean_up(self):
-self.report['netalyzr_report'] = self.output_file
-log.debug("finished running NetalzrWrapper")
-log.debug("Please check %s for Netalyzr output" % self.output_file)
+self.command = [
+distutils.spawn.find_executable("java"),
+"-jar",
+"{}".format(self.localOptions['clipath']),
+"-d"
+]
 
 def test_run_netalyzr(self):
 """
 This test simply wraps netalyzr and runs it from command line
 """
 log.msg("Running NetalyzrWrapper (this will take some time, be 
patient)")
-log.debug("with command '%s'" % self.run_me)
-# XXX we probably want to use a processprotocol here to obtain the
-# stdout from Netalyzr. This would allows us to visualize progress
-# (currently there is no progress because the stdout of os.system is
-# trapped by twisted) and to include the link to the netalyzr report
-# directly in the OONI report, perhaps even downloading it.
-reactor.callInThread(self.blocking_call)
+log.debug("with command '%s'" % self.command)
+self.d = self.run(self.command, env=os.environ, usePTY=1)
+return self.d
diff --git a/ooni/nettests/third_party/psiphon.py 
b/ooni/nettests/third_party/psiphon.py
index d4eb18f..9e50832 100644
--- a/ooni/nettests/third_party/psiphon.py
+++ b/ooni/nettests/third_party/psiphon.py
@@ -31,7 +31,7 @@ class PsiphonTest(httpt.HTTPTest,  process.ProcessTest):
 """
 
 name = "Psiphon Test"
-description = ("Bootstraps Psiphon and"
+description = ("Bootstraps Psiphon and "
"does a HTTP GET for the specified URL")
 author = "juga"
 version = "0.1.0"



___
tor-commits mailing list
tor-commits@lists.torproject.org

[tor-commits] [ooni-probe/master] Use a correct humans.txt in the example for lantern

2016-04-29 Thread art

commit f058349ee84de84c6ce491e02d5e022858693796
Author: Arturo FilastÃ² 
Date:   Tue Apr 26 23:19:51 2016 +0200

Use a correct humans.txt in the example for lantern
---
 docs/source/tests/lantern.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/docs/source/tests/lantern.rst b/docs/source/tests/lantern.rst
index 9f3ee09..7be4d49 100644
--- a/docs/source/tests/lantern.rst
+++ b/docs/source/tests/lantern.rst
@@ -34,7 +34,7 @@ How to run the test
 For advanced usages you may also configure a different URL and expected body
 for the response with the `--url` and `--expected-body` command line options.
 
-`ooniprobe nettests/third_party/lantern.py --url 
http://www.github.com/humans.txt --expected-body '/* TEAM */'`
+`ooniprobe nettests/third_party/lantern.py --url 
http://humanstxt.org/humans.txt --expected-body '/* TEAM */'`
 
 Sample report
 =



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] Merge pull request #456 from TheTorProject/feature/json_reports

2016-04-29 Thread art

commit 5252b00b29f8c9570ecb9533f4577c0a5b69f06f
Merge: a0fc5e6 dd58c40
Author: Simone Basso 
Date:   Mon Apr 4 18:41:52 2016 +0200

Merge pull request #456 from TheTorProject/feature/json_reports

Feature/json reports

 .travis.yml|   4 -
 ooni/deckgen/processors/citizenlab_test_lists.py   |   5 +
 ooni/director.py   |  11 +-
 ooni/errors.py |   3 +
 ooni/nettest.py|  17 +--
 ooni/nettests/blocking/dns_consistency.py  |  29 +++--
 ooni/nettests/blocking/http_requests.py|  11 +-
 ooni/nettests/experimental/daphne.py   | 121 ++
 ooni/nettests/manipulation/captiveportal.py|   2 +-
 ooni/nettests/manipulation/daphne.py   | 121 --
 ooni/nettests/manipulation/http_host.py|   4 +-
 .../manipulation/http_invalid_request_line.py  |   7 +-
 ooni/oonibclient.py|   3 -
 ooni/oonicli.py|  50 
 ooni/otime.py  |  79 +---
 ooni/report/tool.py|   4 +-
 ooni/reporter.py   | 136 ++---
 ooni/tasks.py  |   5 +-
 ooni/templates/dnst.py | 107 +++-
 ooni/templates/httpt.py|  50 ++--
 ooni/templates/scapyt.py   |  12 +-
 ooni/templates/tcpt.py |   1 +
 ooni/tests/test_director.py|   6 +-
 ooni/tests/test_oonicli.py |  17 +--
 ooni/tests/test_otime.py   |  14 ---
 ooni/tests/test_reporter.py|  23 +++-
 ooni/tests/test_templates.py   |  44 ++-
 ooni/tests/test_trueheaders.py |  12 +-
 ooni/tests/test_utils.py   |  10 +-
 ooni/utils/__init__.py |  13 +-
 ooni/utils/log.py  |   2 +-
 ooni/utils/trueheaders.py  |   2 +-
 requirements.txt   |   2 +-
 33 files changed, 526 insertions(+), 401 deletions(-)



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] Add codename for ooniprobe release

2016-04-29 Thread art

commit 28ba3237b782a95202008ba1aea98bc608295401
Author: Arturo FilastÃ² 
Date:   Wed Apr 27 19:46:46 2016 +0200

Add codename for ooniprobe release
---
 ChangeLog.rst | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/ChangeLog.rst b/ChangeLog.rst
index 82d04cd..471e282 100644
--- a/ChangeLog.rst
+++ b/ChangeLog.rst
@@ -4,6 +4,8 @@ Changelog
 v1.4.0 (Wed, 27 Apr 2016)
 -
 
+codename: Under the Sea
+
 * Support for reporting using JSON
 
 * Support for running ooniprobe with a message queue providing URLs to test



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] Add --collector option to oonideckgen

2016-04-29 Thread art

commit 88cfa90a7b76f2c0ff6cc3253459fce4d0790cd2
Author: Arturo FilastÃ² 
Date:   Tue Apr 26 16:33:43 2016 +0200

Add --collector option to oonideckgen

* Remove unused variable in ooniprobe
* Add description to requirements.txt

* Remove printing of probe IP
---
 Vagrantfile |  3 ++-
 ooni/deckgen/cli.py | 49 +++--
 ooni/geoip.py   |  5 ++---
 ooni/oonicli.py |  1 -
 requirements.txt|  4 
 5 files changed, 35 insertions(+), 27 deletions(-)

diff --git a/Vagrantfile b/Vagrantfile
index 3eeb54c..59b2751 100644
--- a/Vagrantfile
+++ b/Vagrantfile
@@ -65,7 +65,9 @@ Vagrant.configure("2") do |config|
   backend.vm.synced_folder "../ooni-backend", "/data/ooni-backend"
   backend.vm.provision :shell, :inline => $setup_oonibackend
 end
+  end
 
+  if File.directory?("../ooni-backend")
 config.vm.provision "shell", inline: <<-EOF
   echo "To run oonibackend:"
   echo "1. vagrant ssh backend"
@@ -73,7 +75,6 @@ Vagrant.configure("2") do |config|
   echo "3. cd /data/ooni-backend"
   echo "4. sudo ./bin/oonib -c /etc/oonibackend.conf"
 EOF
-
   end
 
   config.vm.provision "shell", inline: <<-EOF
diff --git a/ooni/deckgen/cli.py b/ooni/deckgen/cli.py
index 664842e..c2ebfd8 100644
--- a/ooni/deckgen/cli.py
+++ b/ooni/deckgen/cli.py
@@ -22,15 +22,13 @@ class Options(usage.Options):
 """ % sys.argv[0]
 
 optParameters = [
-["country-code", "c",
- None,
+["country-code", "c", None,
  "Specify the two letter country code for which we should "
- "generate the deck."
- ],
-["output", "o",
- None,
- "Specify the directory where to write output."
- ]
+ "generate the deck."],
+["collector", None, None, "Specify a custom collector to use when "
+  "submitting reports"],
+["output", "o", None,
+ "Specify the directory where to write output."]
 ]
 
 def opt_version(self):
@@ -41,33 +39,40 @@ class Options(usage.Options):
 class Deck(object):
 _base_entry = {
 "options": {
+"test_file": None,
+"subargs": [],
+"annotations": None,
+
 "collector": None,
-"help": 0,
-"logfile": None,
-"no-default-reporter": 0,
-"parallelism": None,
-"pcapfile": None,
+# XXX setting this is currently not supported
+"bouncer": None,
+
 "reportfile": None,
-"resume": 0,
-"testdeck": None
+
+"no-collector": 0,
+"no-geoip": 0,
+"no-yamloo": 0,
+"verbose": 0
 }
 }
 
-def __init__(self):
-self.deck = []
+def __init__(self, collector=None):
+self.deck_entries = []
+self.collector = collector
 
 def add_test(self, test_file, subargs=[]):
 deck_entry = copy.deepcopy(self._base_entry)
+deck_entry['options']['collector'] = self.collector
 deck_entry['options']['test_file'] = test_file
 deck_entry['options']['subargs'] = subargs
-self.deck.append(deck_entry)
+self.deck_entries.append(deck_entry)
 
 def pprint(self):
-print yaml.safe_dump(self.deck)
+print yaml.safe_dump(self.deck_entries)
 
 def write_to_file(self, filename):
 with open(filename, "w+") as f:
-f.write(yaml.safe_dump(self.deck))
+f.write(yaml.safe_dump(self.deck_entries))
 
 
 def generate_deck(options):
@@ -89,11 +94,11 @@ def generate_deck(options):
 options['output']
 )
 
-deck = Deck()
+deck = Deck(collector=options['collector'])
 deck.add_test('manipulation/http_invalid_request_line')
 deck.add_test('manipulation/http_header_field_manipulation')
 
-   if url_list_country is not None:
+if url_list_country is not None:
 deck.add_test('blocking/http_requests', ['-f', url_list_country])
 deck.add_test('blocking/http_requests', ['-f', url_list_global])
 
diff --git a/ooni/geoip.py b/ooni/geoip.py
index 417c7d5..dcc799e 100644
--- a/ooni/geoip.py
+++ b/ooni/geoip.py
@@ -6,7 +6,6 @@ import random
 from hashlib import sha256
 
 from twisted.web import client, http_headers
-from ooni.utils.net import hasRawSocketPermission
 
 client._HTTP11ClientFactory.noisy = False
 
@@ -183,7 +182,7 @@ class ProbeIP(object):
 else:
 try:
 yield self.askTor()
-log.msg("Found your IP via Tor %s" % self.address)
+log.msg("Found your IP via Tor")
 self.resolveGeodata()
 defer.returnValue(self.address)
 except errors.TorStateNotFound:
@@ -193,7 +192,7 @@ class ProbeIP(object):
 
 try:
 yield self.askGeoIPService()
-log.msg("Found your IP via a GeoIP

[tor-commits] [ooni-probe/master] Update references to ooni logo

2016-04-29 Thread art

commit 44028962566098be17af0ecb5ccb2c3fa3858fba
Author: Arturo FilastÃ² 
Date:   Wed Apr 27 19:53:43 2016 +0200

Update references to ooni logo

* Use the png instead of the svg for the mascot image
---
 README.rst | 2 +-
 setup.py   | 9 -
 2 files changed, 9 insertions(+), 2 deletions(-)

diff --git a/README.rst b/README.rst
index b42d755..ef9e0ba 100644
--- a/README.rst
+++ b/README.rst
@@ -9,7 +9,7 @@ ooniprobe: a network interference detection tool
 
 ___
 
-.. image:: https://ooni.torproject.org/images/ooni-header-mascot.svg
+.. image:: https://ooni.torproject.org/images/ooni-header-mascot.png
 :target: https:://ooni.torproject.org/
 
 OONI, the Open Observatory of Network Interference, is a global observation
diff --git a/setup.py b/setup.py
index a70df0c..da0b967 100644
--- a/setup.py
+++ b/setup.py
@@ -10,7 +10,7 @@ ooniprobe: a network interference detection tool
 
 ___
 
-.. image:: https://ooni.torproject.org/theme/img/ooni-logo.png
+.. image:: https://ooni.torproject.org/images/ooni-header-mascot.png
 :target: https:://ooni.torproject.org/
 
 OONI, the Open Observatory of Network Interference, is a global observation
@@ -53,6 +53,13 @@ To install ooniprobe you will need the following 
dependencies:
 * libffi-dev
 * libdumbnet-dev
 
+
+On debian based systems this can generally be done by running:
+
+.. code:: bash
+
+sudo apt-get install -y build-essential libdumbnet-dev libpcap-dev 
libgeoip-dev libffi-dev python-dev python-pip
+
 When you got them run:
 
 .. code:: bash



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] Merge branch 'feature/fix_cct_tests' into release/1.4.0

2016-04-29 Thread art

commit 397ed5506d97a17cb4d05852af06ebf3674c638b
Merge: b2dbd38 bcc2be4
Author: Arturo FilastÃ² 
Date:   Tue Apr 26 23:17:01 2016 +0200

Merge branch 'feature/fix_cct_tests' into release/1.4.0

* feature/fix_cct_tests:
  Fix bug in parsing of options for Lantern
  Add support for setting the url and expected-body in tests
  Re-create the command inside the test method.
  Update user documentation for psiphon and lantern
  When GeoIP lookup is disabled it's not possible to perform sanitisations
  Always use the same URL when testing psiphon
  Add support for stripping potential inclusion of probe IPs in reports
  Refactor lantern test to use the ProcessTest template
  Add test for Vanilla Tor

 docs/source/tests/lantern.rst |  23 ---
 docs/source/tests/psiphon.rst |  24 ---
 ooni/nettests/blocking/vanilla_tor.py | 105 +
 ooni/nettests/third_party/lantern.py  | 122 ++
 ooni/nettests/third_party/psiphon.py  |  50 --
 ooni/templates/httpt.py   |   8 ++-
 ooni/templates/process.py |   9 ++-
 ooni/utils/net.py |   3 +
 8 files changed, 245 insertions(+), 99 deletions(-)

diff --cc ooni/nettests/third_party/psiphon.py
index 2633f60,2b18b18..d4eb18f
--- a/ooni/nettests/third_party/psiphon.py
+++ b/ooni/nettests/third_party/psiphon.py
@@@ -30,11 -31,11 +31,11 @@@ class PsiphonTest(httpt.HTTPTest,  proc
  """
  
  name = "Psiphon Test"
 -description = "Bootstraps Psiphon and \
 -does a HTTP GET for the specified URL"
 +description = ("Bootstraps Psiphon and"
 +   "does a HTTP GET for the specified URL")
  author = "juga"
- version = "0.0.1"
- timeout = 20
+ version = "0.1.0"
+ timeout = 120
  usageOptions = UsageOptions
  
  def _setUp(self):
diff --cc ooni/templates/httpt.py
index edab3fa,12995bb..fd6c8ab
--- a/ooni/templates/httpt.py
+++ b/ooni/templates/httpt.py
@@@ -163,22 -131,21 +163,26 @@@ class HTTPTest(NetTestCase)
  'url': request['url'],
  'method': request['method'],
  'tor': request['tor']
--}
++},
++'response': None
  }
  if response:
 +if self.localOptions.get('withoutbody', 0) is 0:
 +response_body = _representBody(response_body)
 +else:
 +response_body = ''
+ # Attempt to redact the IP address of the probe from the responses
+ if config.privacy.includeip is False and config.probe_ip.address 
is not None:
+ response_body = 
response_body.replace(config.probe_ip.address, "[REDACTED]")
 -request_response['response'] = {
 -'headers': list(response.headers.getAllRawHeaders()),
 -'body': response_body if self.localOptions.get('withoutbody', 
0) == 0 else '',
 +session['response'] = {
 +'headers': _representHeaders(response.headers),
 +'body': response_body,
  'code': response.code
- }
+ }
 +session['failure'] = None
  if failure_string:
 -request_response['failure'] = failure_string
 -
 -self.report['requests'].append(request_response)
 +session['failure'] = failure_string
 +self.report['requests'].append(session)
  
  def _processResponseBody(self, response_body, request, response, 
body_processor):
  log.debug("Processing response body")



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] Remove references to the geoip_data_dir since it's not respected

2016-04-29 Thread art

commit c79a087b5926e36962290ce1fb76a4301bddd49c
Author: Arturo FilastÃ² 
Date:   Sat Apr 23 19:04:14 2016 +0200

Remove references to the geoip_data_dir since it's not respected
---
 data/ooniprobe.conf.sample   | 1 -
 ooni/deckgen/processors/namebench_dns_servers.py | 3 +--
 ooni/geoip.py| 6 +++---
 ooni/oonicli.py  | 1 +
 ooni/tests/test_oonicli.py   | 1 -
 5 files changed, 5 insertions(+), 7 deletions(-)

diff --git a/data/ooniprobe.conf.sample b/data/ooniprobe.conf.sample
index 7aaacd7..5f3f17d 100644
--- a/data/ooniprobe.conf.sample
+++ b/data/ooniprobe.conf.sample
@@ -23,7 +23,6 @@ reports:
 pcap: null
 collector: null
 advanced:
-geoip_data_dir: /usr/share/GeoIP
 debug: false
 # enable if auto detection fails
 #tor_binary: /usr/sbin/tor
diff --git a/ooni/deckgen/processors/namebench_dns_servers.py 
b/ooni/deckgen/processors/namebench_dns_servers.py
index f35ca93..9855611 100644
--- a/ooni/deckgen/processors/namebench_dns_servers.py
+++ b/ooni/deckgen/processors/namebench_dns_servers.py
@@ -8,7 +8,6 @@ from ooni.settings import config
 class GeoIPDB(object):
 _borg = {}
 country = None
-asn = None
 
 def __init__(self):
 self.__dict__ = self._borg
@@ -25,7 +24,7 @@ class GeoIPDB(object):
 def generate_country_input(country_code, dst):
 
 csv_file = config.get_data_file_path("resources/"
-   "namebench-dns-servers.csv")
+ "namebench-dns-servers.csv")
 
 filename = os.path.join(dst, "dns-server-%s.txt" % country_code)
 fw = open(filename, "w")
diff --git a/ooni/geoip.py b/ooni/geoip.py
index 83f3648..6179844 100644
--- a/ooni/geoip.py
+++ b/ooni/geoip.py
@@ -37,9 +37,9 @@ def IPToLocation(ipaddr):
 location = {'city': None, 'countrycode': 'ZZ', 'asn': 'AS0'}
 
 def error():
-log.err("Could not find GeoIP data file in %s."
-"Try running ooniresources --update-geoip or"
-" edit your ooniprobe.conf" % config.advanced.geoip_data_dir)
+log.err("Could not find GeoIP data file in data directories."
+"Try running ooniresources or"
+" edit your ooniprobe.conf")
 
 try:
 country_dat = GeoIP(country_file)
diff --git a/ooni/oonicli.py b/ooni/oonicli.py
index 5bc2124..8e7d1f1 100644
--- a/ooni/oonicli.py
+++ b/ooni/oonicli.py
@@ -169,6 +169,7 @@ def director_startup_other_failures(failure):
 
 def setupGlobalOptions(logging, start_tor, check_incoherences):
 global_options = parseOptions()
+
 config.global_options = global_options
 config.set_paths()
 config.initialize_ooni_home()
diff --git a/ooni/tests/test_oonicli.py b/ooni/tests/test_oonicli.py
index c57205c..c3b179a 100644
--- a/ooni/tests/test_oonicli.py
+++ b/ooni/tests/test_oonicli.py
@@ -39,7 +39,6 @@ reports:
 pcap: null
 collector: null
 advanced:
-geoip_data_dir: /usr/share/GeoIP
 debug: false
 interface: auto
 start_tor: false



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] Print messages about how to use the boxes after they have both been deployed.

2016-04-29 Thread art

commit bc416be172786b7e58e5ad2d5da4ed09f2c452d3
Author: Arturo FilastÃ² 
Date:   Tue Apr 26 16:09:08 2016 +0200

Print messages about how to use the boxes after they have both been 
deployed.
---
 Vagrantfile | 27 ---
 1 file changed, 16 insertions(+), 11 deletions(-)

diff --git a/Vagrantfile b/Vagrantfile
index e61bf68..3eeb54c 100644
--- a/Vagrantfile
+++ b/Vagrantfile
@@ -17,11 +17,6 @@ apt-get install -y build-essential libdumbnet-dev 
libpcap-dev libgeoip-dev libff
 cd /data/ooni-probe
 python setup.py install
 
-echo "Now:"
-echo "1. vagrant ssh probe"
-echo "2. oonideckgen"
-echo "3. ooniprobe -i deck-*/*.deck"
-
 SCRIPT
 
 $setup_oonibackend =

[tor-commits] [ooni-probe/master] Detect when the control response contains a blockpage from cloudflare

2016-04-29 Thread art

commit 210ec234e1d60be752b44e1b2136b69659de6eec
Author: Arturo FilastÃ² 
Date:   Sun Apr 24 21:31:52 2016 +0200

Detect when the control response contains a blockpage from cloudflare
---
 ooni/nettests/blocking/http_requests.py | 12 +---
 1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/ooni/nettests/blocking/http_requests.py 
b/ooni/nettests/blocking/http_requests.py
index abc7efc..24d95cc 100644
--- a/ooni/nettests/blocking/http_requests.py
+++ b/ooni/nettests/blocking/http_requests.py
@@ -35,10 +35,10 @@ class HTTPRequestsTest(httpt.HTTPTest):
 lengths match.
 """
 name = "HTTP Requests"
-description = "Performs a HTTP GET request over Tor and one over the " \
-  "local network and compares the two results."
+description = ("Performs a HTTP GET request over Tor and one over the "
+  "local network and compares the two results.")
 author = "Arturo FilastÃ²"
-version = "0.2.4"
+version = "0.2.5"
 
 usageOptions = UsageOptions
 
@@ -78,6 +78,7 @@ class HTTPRequestsTest(httpt.HTTPTest):
 self.report['factor'] = float(self.factor)
 self.report['headers_diff'] = None
 self.report['headers_match'] = None
+self.report['control_cloudflare'] = None
 
 self.headers = {'User-Agent': [random.choice(userAgents)]}
 
@@ -145,6 +146,11 @@ class HTTPRequestsTest(httpt.HTTPTest):
 if experiment and control:
 if hasattr(experiment, 'body') and hasattr(control, 'body') \
 and experiment.body and control.body:
+self.report['control_cloudflare'] = False
+if 'Attention Required! | CloudFlare' in control.body:
+log.msg("The control body contains a blockpage from "
+"cloudflare. This will skew our results.")
+self.report['control_cloudflare'] = True
 self.compare_body_lengths(len(control.body),
   len(experiment.body))
 if hasattr(experiment, 'headers') and hasattr(control, 'headers') \



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] Update documentation to reflect changes in oonideckgen

2016-04-29 Thread art

commit b2dbd382c030c80cf09fcefe52ce5a1116bf0159
Author: Arturo FilastÃ² 
Date:   Tue Apr 26 21:42:26 2016 +0200

Update documentation to reflect changes in oonideckgen
---
 README.rst | 11 ++-
 1 file changed, 2 insertions(+), 9 deletions(-)

diff --git a/README.rst b/README.rst
index 384f71b..f137b88 100644
--- a/README.rst
+++ b/README.rst
@@ -58,7 +58,6 @@ To install it from the current master run::
 Then run::
 
 mkdir my_decks
-sudo ooniresources --update-inputs --update-geoip
 oonideckgen -o my_decks/
 
 **BUG** Note:
@@ -174,17 +173,11 @@ To generate decks you will have to update the input 
resources of ooniprobe.
 
 This can be done with::
 
-ooniresources --update-inputs
+ooniresources
 
 If you get a permission error, you may have to run the command as root or
 change the ooniprobe data directory inside of `ooniprobe.conf`.
 
-On some platforms, for example debian contrib, you will not get all the geoip
-related files needed. In that case it is possible to manually download them
-with ``ooniresources``::
-
-ooniresources --update-geoip
-
 Generating decks
 
 
@@ -195,7 +188,7 @@ you can do so (be sure to have updated the input resources 
first) by running::
 
 oonideckgen --country-code IT --output ~/
 
-You will now have in your home a folder called `deck-it`, containing the ooni
+You will now have in your home a folder called `deck-YOUR_COUNTRY_CODE`, 
containing the ooni
 deck (ends with .deck) and the inputs.
 Note: that you should not move the `deck-*` directory once it has been
 generated as the paths to the inputs referenced by the test in the deck are



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] Change the order in which decks get generated by default

2016-04-29 Thread art

commit 6d49b3fbbbebf824c106f2257d6c774839cd6703
Author: Arturo FilastÃ² 
Date:   Tue Apr 26 15:46:45 2016 +0200

Change the order in which decks get generated by default

* Give priority to country specific lists

* Give priority to faster tests

* Run dns consistency for last
---
 ooni/deckgen/cli.py | 18 --
 1 file changed, 8 insertions(+), 10 deletions(-)

diff --git a/ooni/deckgen/cli.py b/ooni/deckgen/cli.py
index b1a6b87..664842e 100644
--- a/ooni/deckgen/cli.py
+++ b/ooni/deckgen/cli.py
@@ -71,17 +71,15 @@ class Deck(object):
 
 
 def generate_deck(options):
-
+url_list_country = None
 try:
 url_list_country = citizenlab_test_lists.generate_country_input(
 options['country-code'],
 options['output']
 )
-
 except Exception:
 print "Could not generate country specific url list"
 print "We will just use the global one."
-url_list_country = None
 
 url_list_global = citizenlab_test_lists.generate_global_input(
 options['output']
@@ -92,19 +90,19 @@ def generate_deck(options):
 )
 
 deck = Deck()
-# deck.add_test('manipulation/http_host', ['-f', 'somefile.txt'])
+deck.add_test('manipulation/http_invalid_request_line')
+deck.add_test('manipulation/http_header_field_manipulation')
+
+   if url_list_country is not None:
+deck.add_test('blocking/http_requests', ['-f', url_list_country])
 deck.add_test('blocking/http_requests', ['-f', url_list_global])
-deck.add_test('blocking/dns_consistency',
-  ['-f', url_list_global, '-T', dns_servers])
 
 if url_list_country is not None:
 deck.add_test('blocking/dns_consistency',
   ['-f', url_list_country, '-T', dns_servers])
-deck.add_test('blocking/http_requests', ['-f', url_list_country])
+deck.add_test('blocking/dns_consistency',
+  ['-f', url_list_global, '-T', dns_servers])
 
-deck.add_test('manipulation/http_invalid_request_line')
-deck.add_test('manipulation/http_header_field_manipulation')
-# deck.add_test('manipulation/traceroute')
 if config.advanced.debug:
 deck.pprint()
 deck_filename = os.path.join(options['output'],



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] Run ooniresources without any command line options in the setup.py

2016-04-29 Thread art

commit a4244fcb6eeb402a606ab99fec6ac1a15e0d8960
Author: Arturo FilastÃ² 
Date:   Sat Apr 23 20:44:37 2016 +0200

Run ooniresources without any command line options in the setup.py

* Print some progress when running oonireport
* Use print function instead of old print
---
 ooni/report/tool.py | 23 ++-
 setup.py|  2 +-
 2 files changed, 15 insertions(+), 10 deletions(-)

diff --git a/ooni/report/tool.py b/ooni/report/tool.py
index 5946c42..5c7bcb2 100644
--- a/ooni/report/tool.py
+++ b/ooni/report/tool.py
@@ -1,4 +1,6 @@
+from __future__ import print_function
 import yaml
+import sys
 
 from twisted.internet import defer
 
@@ -49,9 +51,12 @@ def upload(report_file, collector=None, bouncer=None):
 report_id = yield oonib_reporter.createReport()
 report.header['report_id'] = report_id
 yield oonib_report_log.created(report_file, collector, report_id)
+log.msg("Writing report entries")
 for entry in report:
 yield oonib_reporter.writeReportEntry(entry)
-log.msg("Closing report.")
+sys.stdout.write('.')
+sys.stdout.flush()
+log.msg("Closing report")
 yield oonib_reporter.finish()
 yield oonib_report_log.closed(report_file)
 
@@ -68,24 +73,24 @@ def upload_all(collector=None, bouncer=None):
 
 
 def print_report(report_file, value):
-print "* %s" % report_file
-print "  %s" % value['created_at']
+print("* %s" % report_file)
+print("  %s" % value['created_at'])
 
 
 def status():
 oonib_report_log = OONIBReportLog()
 
-print "Reports to be uploaded"
-print "--"
+print("Reports to be uploaded")
+print("--")
 for report_file, value in oonib_report_log.reports_to_upload:
 print_report(report_file, value)
 
-print "Reports in progress"
-print "---"
+print("Reports in progress")
+print("---")
 for report_file, value in oonib_report_log.reports_in_progress:
 print_report(report_file, value)
 
-print "Incomplete reports"
-print "--"
+print("Incomplete reports")
+print("--")
 for report_file, value in oonib_report_log.reports_incomplete:
 print_report(report_file, value)
diff --git a/setup.py b/setup.py
index 388de5f..1fd9a7b 100644
--- a/setup.py
+++ b/setup.py
@@ -148,7 +148,7 @@ class install(_st_install):
 def ooniresources(self):
 ooniresources = find_executable("ooniresources")
 from subprocess import Popen
-process = Popen([ooniresources, '--update-inputs', '--update-geoip'],
+process = Popen([ooniresources],
 stdout=sys.stdout.fileno(), stderr=sys.stderr.fileno())
 process.wait()
 



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] Always output the same deck filename

2016-04-29 Thread art

commit 97cd370e7349e6f74471f8ebb6beedb900197551
Author: Arturo FilastÃ² 
Date:   Wed Apr 27 12:14:18 2016 +0200

Always output the same deck filename
---
 README.rst  | 2 +-
 ooni/deckgen/cli.py | 3 +--
 2 files changed, 2 insertions(+), 3 deletions(-)

diff --git a/README.rst b/README.rst
index f137b88..a6dcb2c 100644
--- a/README.rst
+++ b/README.rst
@@ -188,7 +188,7 @@ you can do so (be sure to have updated the input resources 
first) by running::
 
 oonideckgen --country-code IT --output ~/
 
-You will now have in your home a folder called `deck-YOUR_COUNTRY_CODE`, 
containing the ooni
+You will now have in your home a folder called `deck-it`, containing the ooni
 deck (ends with .deck) and the inputs.
 Note: that you should not move the `deck-*` directory once it has been
 generated as the paths to the inputs referenced by the test in the deck are
diff --git a/ooni/deckgen/cli.py b/ooni/deckgen/cli.py
index c2ebfd8..2e0e976 100644
--- a/ooni/deckgen/cli.py
+++ b/ooni/deckgen/cli.py
@@ -111,8 +111,7 @@ def generate_deck(options):
 if config.advanced.debug:
 deck.pprint()
 deck_filename = os.path.join(options['output'],
- "%s-%s-user.deck" % (__version__,
-  options['country-code']))
+ "default-user.deck")
 deck.write_to_file(deck_filename)
 print "Deck written to %s" % deck_filename
 print "Run ooniprobe like so:"



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] Improve readability of the ooniprobe -s command

2016-04-29 Thread art

commit 33e86a4a4ae96053812685c9f663db261703e449
Author: Arturo FilastÃ² 
Date:   Sun Apr 24 19:13:45 2016 +0200

Improve readability of the ooniprobe -s command

* Remove various pieces of dead code
---
 data/ooniprobe.conf.sample   |  2 --
 ooni/director.py |  2 +-
 ooni/geoip.py| 20 
 ooni/nettests/third_party/psiphon.py |  4 ++--
 ooni/oonicli.py  | 30 +++---
 ooni/tests/test_oonicli.py   |  1 -
 ooni/utils/log.py|  2 --
 7 files changed, 26 insertions(+), 35 deletions(-)

diff --git a/data/ooniprobe.conf.sample b/data/ooniprobe.conf.sample
index 5f3f17d..edc3f86 100644
--- a/data/ooniprobe.conf.sample
+++ b/data/ooniprobe.conf.sample
@@ -12,8 +12,6 @@ privacy:
 includeasn: true
 # Should we include the country as reported by GeoIP in the report?
 includecountry: true
-# Should we include the city as reported by GeoIP in the report?
-includecity: false
 # Should we collect a full packet capture on the client?
 includepcap: false
 reports:
diff --git a/ooni/director.py b/ooni/director.py
index ab8da65..a60b91e 100644
--- a/ooni/director.py
+++ b/ooni/director.py
@@ -61,7 +61,7 @@ class Director(object):
 
 _scheduledTests = 0
 # Only list NetTests belonging to these categories
-categories = ['blocking', 'manipulation']
+categories = ['blocking', 'manipulation', 'third_party']
 
 def __init__(self):
 self.activeNetTests = []
diff --git a/ooni/geoip.py b/ooni/geoip.py
index 6179844..417c7d5 100644
--- a/ooni/geoip.py
+++ b/ooni/geoip.py
@@ -171,8 +171,6 @@ class ProbeIP(object):
 self.geodata['ip'] = self.address
 if not config.privacy.includeasn:
 self.geodata['asn'] = 'AS0'
-if not config.privacy.includecity:
-self.geodata['city'] = None
 if not config.privacy.includecountry:
 self.geodata['countrycode'] = 'ZZ'
 if not config.privacy.includeip:
@@ -194,16 +192,6 @@ class ProbeIP(object):
 log.msg("Unable to lookup the probe IP via Tor.")
 
 try:
-yield self.askTraceroute()
-log.msg("Found your IP via Traceroute %s" % self.address)
-self.resolveGeodata()
-defer.returnValue(self.address)
-except errors.InsufficientPrivileges:
-log.debug("Cannot determine the probe IP address with a 
traceroute, becase of insufficient priviledges")
-except:
-log.msg("Unable to lookup the probe IP via traceroute")
-
-try:
 yield self.askGeoIPService()
 log.msg("Found your IP via a GeoIP service: %s" % self.address)
 self.resolveGeodata()
@@ -230,14 +218,6 @@ class ProbeIP(object):
 if not self.address:
 raise errors.ProbeIPUnknown
 
-def askTraceroute(self):
-"""
-Perform a UDP traceroute to determine the probes IP address.
-"""
-if not hasRawSocketPermission():
-raise errors.InsufficientPrivileges
-raise NotImplemented
-
 def askTor(self):
 """
 Obtain the probes IP address by asking the Tor Control port via GET 
INFO
diff --git a/ooni/nettests/third_party/psiphon.py 
b/ooni/nettests/third_party/psiphon.py
index bfdec70..2633f60 100644
--- a/ooni/nettests/third_party/psiphon.py
+++ b/ooni/nettests/third_party/psiphon.py
@@ -30,8 +30,8 @@ class PsiphonTest(httpt.HTTPTest,  process.ProcessTest):
 """
 
 name = "Psiphon Test"
-description = "Bootstraps Psiphon and \
-does a HTTP GET for the specified URL"
+description = ("Bootstraps Psiphon and"
+   "does a HTTP GET for the specified URL")
 author = "juga"
 version = "0.0.1"
 timeout = 20
diff --git a/ooni/oonicli.py b/ooni/oonicli.py
index 8e7d1f1..3be64a9 100644
--- a/ooni/oonicli.py
+++ b/ooni/oonicli.py
@@ -4,6 +4,7 @@ import os
 import json
 import yaml
 import random
+import textwrap
 import urlparse
 
 from twisted.python import usage
@@ -24,7 +25,6 @@ class Options(usage.Options):
 " files listed on the command line.")
 
 optFlags = [["help", "h"],
-["resume", "r"],
 ["no-collector", "n", "disable writing to collector"],
 ["no-yamloo", "N", "disable writing to YAML file"],
 ["no-geoip", "g"],
@@ -342,12 +342,28 @@ def runWithDirector(logging=True, start_tor=True, 
check_incoherences=True):
 from ooni.director import Director
 director = Director()
 if global_options['list']:
-print "# Installed nettests"
-for net_test_id, net_test in director.getNetTests().items():
-print "* %s (%s/%s)" % (net_test['name'],
-net_test['category'],
-

[tor-commits] [ooni-probe/master] Drop dependencies that are for sub-modules of ooni

2016-04-29 Thread art

commit f098d7048e9394d0ad2f624be153b8d802ab4e61
Author: Arturo FilastÃ² 
Date:   Fri Apr 8 16:10:48 2016 +0200

Drop dependencies that are for sub-modules of ooni
---
 requirements.txt | 6 +-
 1 file changed, 1 insertion(+), 5 deletions(-)

diff --git a/requirements.txt b/requirements.txt
index 28e5f78..fc6f629 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,3 +1,4 @@
+pyasn1>=0.1.8
 PyYAML>=3.10
 Twisted>=13.2.0
 ipaddr>=2.1.10
@@ -5,13 +6,8 @@ pyOpenSSL>=0.13
 geoip
 txtorcon>=0.7
 txsocksx>=0.0.2
-parsley>=1.1
 scapy-real>=2.2.0-dev
 pypcap>=1.1
 service-identity
 pydumbnet
-pyasn1>=0.1.8
-pyasn1-modules
-characteristic
 zope.interface
-cryptography



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] Update manpages for ooniprobe, ooniresources, oonideckgen and oonireport.

2016-04-29 Thread art

commit a6897988fb96bae8816dd1bd7973a6e1781757f1
Author: Arturo FilastÃ² 
Date:   Tue Apr 26 21:40:32 2016 +0200

Update manpages for ooniprobe, ooniresources, oonideckgen and oonireport.

* Update versions of oonideckgen and ooniresources
* Clean up the help text of ooniprobe options
---
 data/oonideckgen.1 |  7 +-
 data/ooniprobe.1   | 18 
 data/oonireport.1  |  4 ++--
 data/ooniresources.1   | 22 ---
 ooni/deckgen/__init__.py   |  2 +-
 ooni/oonicli.py| 54 +++---
 ooni/resources/__init__.py |  2 +-
 7 files changed, 65 insertions(+), 44 deletions(-)

diff --git a/data/oonideckgen.1 b/data/oonideckgen.1
index f808165..e2c8fed 100644
--- a/data/oonideckgen.1
+++ b/data/oonideckgen.1
@@ -35,6 +35,8 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
 .B oonideckgen
 .RB [ --version ]
 .RB [ --help ]
+.RB [ --collector
+.IR httpo://collector-address.onion ]
 .RB [ \-c
 .IR country-code]
 .RB [ \-o
@@ -56,6 +58,9 @@ Display this help and exit.
 .BR \-\^c " or " \-\-country-code
 Specify the country code for which a test deck should be geneated
 .TP
+.BR \-\-collector
+Specify the address of the collector to use for submitting reports.
+.TP
 .BR \-\^o " or " \-\-output
 Specify in which directory the deck should be written.
 .TP
@@ -81,5 +86,5 @@ In order to run deckgen you must have first have run 
ooniresources --update-inpu
 .SH AUTHOR
 The Tor Project
 .SH COPYRIGHT
-2014, The Tor Project
+2016, The Tor Project
 .
diff --git a/data/ooniprobe.1 b/data/ooniprobe.1
index cf3f593..5fa47b4 100644
--- a/data/ooniprobe.1
+++ b/data/ooniprobe.1
@@ -1,6 +1,5 @@
-.\" Man page generated from reStructuredText.
 .
-.TH "ooniprobe" "1" "April 29, 2014" "1.0.2" "ooniprobe"
+.TH "ooniprobe" "1" "April 26, 2016" "1.4.0" "ooniprobe"
 .SH NAME
 ooniprobe - a network censorship measurement tool.
 .
@@ -33,7 +32,7 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
 
 .SH SYNOPSIS
 .B ooniprobe
-.RB [ \-hnsp ]
+.RB [ \-hnspgN ]
 .RB [ --version ]
 .RB [ --spew ]
 .RB [ \-o
@@ -52,6 +51,8 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
 .IR configfile ]
 .RB [ \-d
 .IR datadir ]
+.RB [ \-a
+.IR annotations]
 .I "test_name"
 
 .SH DESCRIPTION
@@ -69,6 +70,12 @@ Display this help and exit.
 .BR \-\^n " or " \-\-no\-collector
 Disable reporting the net test results to an oonib collector.
 .TP
+.BR \-\^N " or " \-\-no\-yamloo
+Disable writing report file to disk.
+.TP
+.BR \-\^g " or " \-\-no\-geoip
+Disable performing geoip lookup.
+.TP
 .BR \-\^s " or " \-\-list
 List all of the available net tests.
 .TP
@@ -103,6 +110,9 @@ Specify a path to the ooniprobe configuration file.
 .BR \-\^d " or " \-\-datadir
 Specify a path to the ooniprobe data directory
 .TP
+.BR \-\^a " or " \-\-annotations
+Annotate the report with a key:value[, key:value] format.
+.TP
 .BR \-\-spew
 Print an insanely verbose log of everything that happens.
 Useful when debugging freezes or locks in complex code.
@@ -1096,5 +1106,5 @@ require for a researcher to assess what is the output of 
a test.
 .SH AUTHOR
 The Tor Project
 .SH COPYRIGHT
-2014, The Tor Project
+2016, The Tor Project
 .
diff --git a/data/oonireport.1 b/data/oonireport.1
index 7910e67..8a91009 100644
--- a/data/oonireport.1
+++ b/data/oonireport.1
@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH "oonireport" "1" "June 26, 2014" "1.0.2" "oonireport"
+.TH "oonireport" "1" "April 26, 2016" "0.1.0" "oonireport"
 .SH NAME
 oonireport - tool for managing ooniprobe reports.
 .
@@ -96,5 +96,5 @@ of the collector did not support the specified test.
 .SH AUTHOR
 The Tor Project
 .SH COPYRIGHT
-2014, The Tor Project
+2016, The Tor Project
 .
diff --git a/data/ooniresources.1 b/data/ooniresources.1
index 5cd66cf..1e16822 100644
--- a/data/ooniresources.1
+++ b/data/ooniresources.1
@@ -1,8 +1,8 @@
 .\" Man page generated from reStructuredText.
 .
-.TH "ooniresources" "1" "October 1, 2014" "1.1.4" "ooniresources"
+.TH "ooniresources" "1" "April 26, 2016" "0.2.0" "ooniresources"
 .SH NAME
-ooniresources - tool for generating ooniprobe test decks
+ooniresources - tool for updating resources for OONI tests
 .
 .nr rst2man-indent-level 0
 .
@@ -35,14 +35,15 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
 .B ooniresources
 .RB [ --version ]
 .RB [ --help ]
-.RB [ --update-inputs ]
-.RB [ --update-geoip ]
 
 .SH DESCRIPTION
 .sp
 ooniresources will update the resources that are needed for the functioning of
 ooniprobe and related tools.
 
+By invoking this command with no arguments the inputs and GeoIP data files
+shall be updated.
+
 .SH OPTIONS
 
 .TP
@@ -50,11 +51,11 @@ ooniprobe and related tools.
 Display this help and exit.
 .TP
 .BR \-\-update\-inputs
-This will update the inputs that are used for generating test decks with
+(DEPRECATED) This will update the inputs that are used for

[tor-commits] [ooni-probe/master] Join ooniprobe and ooniprobequeue into one command

2016-04-29 Thread art

commit 1192ca0d0fd7c7b4aa5cb593fe711eb175d5c880
Author: Arturo FilastÃ² 
Date:   Tue Apr 26 23:33:14 2016 +0200

Join ooniprobe and ooniprobequeue into one command
---
 bin/ooniprobe  | 19 +--
 bin/ooniprobequeue | 16 
 ooni/oonicli.py| 21 +++--
 ooni/tests/test_oonicli.py |  5 +++--
 setup.py   |  3 +--
 5 files changed, 28 insertions(+), 36 deletions(-)

diff --git a/bin/ooniprobe b/bin/ooniprobe
index 4832d91..ad97553 100755
--- a/bin/ooniprobe
+++ b/bin/ooniprobe
@@ -1,9 +1,24 @@
 #!/usr/bin/env python
+import sys
+
 from twisted.internet import reactor
 
-from ooni.oonicli import runWithDirector
-d = runWithDirector()
+from ooni.oonicli import setupGlobalOptions
+from ooni.oonicli import runWithDaemonDirector, runWithDirector
+
+exit_code=0
+
+global_options = setupGlobalOptions(logging=True, start_tor=True,
+check_incoherences=True)
+if global_options['queue']:
+d = runWithDaemonDirector(global_options)
+else:
+d = runWithDirector(global_options)
 @d.addBoth
 def cb(result):
+global exit_code
+if result is not None:
+exit_code=1
 reactor.stop()
 reactor.run()
+sys.exit(exit_code)
diff --git a/bin/ooniprobequeue b/bin/ooniprobequeue
deleted file mode 100755
index 88359e0..000
--- a/bin/ooniprobequeue
+++ /dev/null
@@ -1,16 +0,0 @@
-#!/usr/bin/env python
-import sys
-from twisted.internet import reactor
-
-code=0
-
-from ooni.oonicli import runWithDaemonDirector
-d = runWithDaemonDirector()
-@d.addBoth
-def cb(result):
-global code
-if result is not None:
-code=1
-reactor.stop()
-reactor.run()
-sys.exit(code)
diff --git a/ooni/oonicli.py b/ooni/oonicli.py
index 09936b6..1ffe467 100644
--- a/ooni/oonicli.py
+++ b/ooni/oonicli.py
@@ -205,6 +205,7 @@ def setupGlobalOptions(logging, start_tor, 
check_incoherences):
 log.err("Insufficient Privileges to capture packets."
 " See ooniprobe.conf privacy.includepcap")
 sys.exit(2)
+global_options['check_incoherences'] = check_incoherences
 return global_options
 
 def setupAnnotations(global_options):
@@ -294,14 +295,13 @@ def createDeck(global_options, url=None):
 return deck
 
 
-def runTestWithDirector(director, global_options, url=None,
-start_tor=True, check_incoherences=True):
+def runTestWithDirector(director, global_options, url=None, start_tor=True):
 deck = createDeck(global_options, url=url)
 
 start_tor |= deck.requiresTor
 
 d = director.start(start_tor=start_tor,
-   check_incoherences=check_incoherences)
+   check_incoherences=global_options['check_incoherences'])
 
 def setup_nettest(_):
 try:
@@ -341,13 +341,11 @@ def runTestWithDirector(director, global_options, 
url=None,
 d.addErrback(director_startup_other_failures)
 return d
 
-def runWithDirector(logging=True, start_tor=True, check_incoherences=True):
+def runWithDirector(global_options):
 """
 Instance the director, parse command line options and start an ooniprobe
 test!
 """
-global_options = setupGlobalOptions(logging, start_tor, check_incoherences)
-
 from ooni.director import Director
 director = Director()
 if global_options['list']:
@@ -398,14 +396,13 @@ def runWithDirector(logging=True, start_tor=True, 
check_incoherences=True):
 
 return runTestWithDirector(director=director,
start_tor=start_tor,
-   global_options=global_options,
-   check_incoherences=check_incoherences)
+   global_options=global_options)
 
 
 # this variant version of runWithDirector splits the process in two,
 # allowing a single director instance to be reused with multiple decks.
 
-def runWithDaemonDirector(logging=True, start_tor=True, 
check_incoherences=True):
+def runWithDaemonDirector(global_options):
 """
 Instance the director, parse command line options and start an ooniprobe
 test!
@@ -421,9 +418,6 @@ def runWithDaemonDirector(logging=True, start_tor=True, 
check_incoherences=True)
 print "Install with \"pip install pika\"."
 sys.exit(7)
 
-
-global_options = setupGlobalOptions(logging, start_tor, check_incoherences)
-
 director = Director()
 
 if global_options.get('annotations') is not None:
@@ -464,8 +458,7 @@ def runWithDaemonDirector(logging=True, start_tor=True, 
check_incoherences=True)
 d = runTestWithDirector(director=director,
 start_tor=start_tor,
 global_options=global_options,
-url=data['url'].encode('utf8'),
-check_incoherences=check_incoherences)
+

[tor-commits] [ooni-probe/master] Specify a more recent version of setuptools

2016-04-29 Thread art

commit df7070bfdeab4060476437ab079e251d150e0b9a
Author: Arturo FilastÃ² 
Date:   Wed Apr 27 12:49:29 2016 +0200

Specify a more recent version of setuptools

* cryptography -> depends on -> cffi -> ipaddress -> setuptools >=11.3
* Python dependencies are hell
---
 requirements.txt | 1 +
 1 file changed, 1 insertion(+)

diff --git a/requirements.txt b/requirements.txt
index 229504b..7aeaabc 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,6 +1,7 @@
 # These are dependencies of ooniprobe depedencies that require specific
 # versions.
 pyasn1>=0.1.8
+setuptools>=11.3
 
 # These are direct dependencies of ooniprobe.
 PyYAML>=3.10



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] Update changelog for version 1.4.0

2016-04-29 Thread art

commit 2f8a8a3cfcbab9c019f613d2588b281549788f39
Author: Arturo FilastÃ² 
Date:   Fri Apr 22 12:52:08 2016 +0200

Update changelog for version 1.4.0
---
 ChangeLog.rst | 16 
 1 file changed, 16 insertions(+)

diff --git a/ChangeLog.rst b/ChangeLog.rst
index 79cb883..47851e1 100644
--- a/ChangeLog.rst
+++ b/ChangeLog.rst
@@ -1,6 +1,22 @@
 Changelog
 =
 
+v1.4.0 (XXX, XX Apr 2016)
+-
+
+* Support for reporting using JSON
+
+* Support for running ooniprobe with a message queue providing URLs to test
+
+* Psiphon censorship circumvention test
+
+* OpenVPN censorship circumvention test
+
+* Support for disabling reporting to disk
+
+* Improvements to HTTP response body decoding (includes fix that lead to empty
+  bodies being misrepresented)
+
 v1.3.2 (Fri, 20 Nov 2015)
 -
 



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] Remove hacks module and patch reduce_ex

2016-04-29 Thread art

commit 55e55a55a0365c3e4958482e315add4f0090243a
Author: Arturo FilastÃ² 
Date:   Sat Apr 23 19:03:49 2016 +0200

Remove hacks module and patch reduce_ex

* Delete benchmarks txt file
---
 bin/ooniprobe   |  7 --
 bin/ooniprobequeue  |  5 -
 ooni/kit/benchmarks.txt | 35 -
 ooni/utils/hacks.py | 60 -
 4 files changed, 107 deletions(-)

diff --git a/bin/ooniprobe b/bin/ooniprobe
index 31da086..4832d91 100755
--- a/bin/ooniprobe
+++ b/bin/ooniprobe
@@ -1,13 +1,6 @@
 #!/usr/bin/env python
-import copy_reg
 from twisted.internet import reactor
 
-# This is a hack to overcome a bug in python
-from ooni.utils.hacks import patched_reduce_ex
-copy_reg._reduce_ex = patched_reduce_ex
-
-# from ooni.oonicli import run
-# run()
 from ooni.oonicli import runWithDirector
 d = runWithDirector()
 @d.addBoth
diff --git a/bin/ooniprobequeue b/bin/ooniprobequeue
index f81f156..88359e0 100755
--- a/bin/ooniprobequeue
+++ b/bin/ooniprobequeue
@@ -1,12 +1,7 @@
 #!/usr/bin/env python
 import sys
-import copy_reg
 from twisted.internet import reactor
 
-# This is a hack to overcome a bug in python
-from ooni.utils.hacks import patched_reduce_ex
-copy_reg._reduce_ex = patched_reduce_ex
-
 code=0
 
 from ooni.oonicli import runWithDaemonDirector
diff --git a/ooni/kit/benchmarks.txt b/ooni/kit/benchmarks.txt
deleted file mode 100644
index c2070e5..000
--- a/ooni/kit/benchmarks.txt
+++ /dev/null
@@ -1,35 +0,0 @@
-Read file B
-done in 0.74356508255
-Read file A
-done in 0.94336104393
-Computing prob matrix
-done in 0.0432229042053
-Computing eigenvalues
-done in 0.00188422203064
-Corelation: 0.99079331
-
-
-Read file B
-done in 0.946599960327
-Read file A
-done in 0.909075975418
-Computing prob matrix
-done in 0.0541450977325
-Computing eigenvalues
-done in 0.00259518623352
-Computing prob matrix B
-done in 0.060467004776
-Computing eigen B
-done in 0.00102496147156
-Computing correlation
-done in 0.000158071517944
-Corelation: 0.99079331
-
-this was with:
-683 filea.txt
-678 fileb.txt
-
-diff file* | wc -l
-283
-
-
diff --git a/ooni/utils/hacks.py b/ooni/utils/hacks.py
deleted file mode 100644
index 7ec3d0f..000
--- a/ooni/utils/hacks.py
+++ /dev/null
@@ -1,60 +0,0 @@
-# When some software has issues and we need to fix it in a
-# hackish way, we put it in here. This one day will be empty.
-
-import copy_reg
-from twisted.web.client import SchemeNotSupported
-
-from txsocksx.http import SOCKS5Agent as SOCKS5AgentOriginal
-
-
-def patched_reduce_ex(self, proto):
-"""
-This is a hack to overcome a bug in one of pythons core functions. It is
-located inside of copy_reg and is called _reduce_ex.
-
-Some background on the issue can be found here:
-
-
http://stackoverflow.com/questions/569754/how-to-tell-for-which-object-attribute-pickle
-http://stackoverflow.com/questions/2049849/why-cant-i-pickle-this-object
-
-There was also an open bug on the pyyaml trac repo, but it got closed 
because
-they could not reproduce.
-http://pyyaml.org/ticket/190
-
-It turned out to be easier to patch the python core library than to monkey
-patch yaml.
-
-XXX see if there is a better way. sigh...
-"""
-_HEAPTYPE = 1 << 9
-assert proto < 2
-for base in self.__class__.__mro__:
-if hasattr(base, '__flags__') and not base.__flags__ & _HEAPTYPE:
-break
-else:
-base = object  # not really reachable
-if base is object:
-state = None
-elif base is int:
-state = None
-else:
-if base is self.__class__:
-raise TypeError("can't pickle %s objects" % base.__name__)
-state = base(self)
-args = (self.__class__, base, state)
-try:
-getstate = self.__getstate__
-except AttributeError:
-if getattr(self, "__slots__", None):
-raise TypeError("a class that defines __slots__ without "
-"defining __getstate__ cannot be pickled")
-try:
-dict = self.__dict__
-except AttributeError:
-dict = None
-else:
-dict = getstate()
-if dict:
-return copy_reg._reconstructor, args, dict
-else:
-return copy_reg._reconstructor, args



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] Use debian/jessie instead of precise

2016-04-29 Thread art

commit 223ba4803e17b25bc4d1a54bafc5b51896a2c14c
Author: Arturo FilastÃ² 
Date:   Tue Apr 26 15:45:22 2016 +0200

Use debian/jessie instead of precise

* Make the setup of ooni-backend conditional and do it in a another machine
---
 Vagrantfile | 85 -
 1 file changed, 61 insertions(+), 24 deletions(-)

diff --git a/Vagrantfile b/Vagrantfile
index 16a6ba0..e61bf68 100644
--- a/Vagrantfile
+++ b/Vagrantfile
@@ -1,22 +1,8 @@
 # -*- mode: ruby -*-
 # vi: set ft=ruby :
 
-Vagrant.configure("2") do |config|
-  # All Vagrant configuration is done here. The most common configuration
-  # options are documented and commented below. For a complete reference,
-  # please see the online documentation at vagrantup.com.
-  config.vm.box = "precise32"
-  config.vm.box_url = "http://files.vagrantup.com/precise32.box;
-
-  config.vm.synced_folder ".", "/ooni"
-  # Place the ooni-backend source code in ../ooni-backend to sync it with the 
vagrant instance
-  config.vm.synced_folder "../ooni-backend", "/backend"
-
-end
-
-$script =

[tor-commits] [ooni-probe/master] Print a helpful message when run with no arguments (inciting to run oonideckgen)

2016-04-29 Thread art

commit 96dad3f578e0797aee4992fc79b25ea8af2b9414
Author: Arturo FilastÃ² 
Date:   Sat Apr 23 15:48:09 2016 +0200

Print a helpful message when run with no arguments (inciting to run 
oonideckgen)

* Optimise a bit the importing of modules
---
 ooni/oonicli.py | 32 +++-
 1 file changed, 19 insertions(+), 13 deletions(-)

diff --git a/ooni/oonicli.py b/ooni/oonicli.py
index 63312b5..5bc2124 100644
--- a/ooni/oonicli.py
+++ b/ooni/oonicli.py
@@ -7,20 +7,11 @@ import random
 import urlparse
 
 from twisted.python import usage
-from twisted.python.util import spewer
-from twisted.internet import defer, reactor, protocol
+from twisted.internet import defer
 
 from ooni import errors, __version__
-
 from ooni.settings import config
-from ooni.director import Director
-from ooni.deck import Deck, nettest_to_path
-from ooni.nettest import NetTestLoader
-
 from ooni.utils import log
-from ooni.utils.net import hasRawSocketPermission
-
-
 
 class LifetimeExceeded(Exception): pass
 
@@ -30,7 +21,7 @@ class Options(usage.Options):
 
 longdesc = ("ooniprobe loads and executes a suite or a set of suites of"
 " network tests. These are loaded from modules, packages and"
-" files listed on the command line")
+" files listed on the command line.")
 
 optFlags = [["help", "h"],
 ["resume", "r"],
@@ -71,11 +62,21 @@ class Options(usage.Options):
 self['test'] = None
 usage.Options.__init__(self)
 
+def getUsage(self, width=None):
+return super(Options, self).getUsage(width) + """
+To get started you may want to run:
+
+$ oonideckgen
+
+This will tell you how to run ooniprobe :)
+"""
+
 def opt_spew(self):
 """
 Print an insanely verbose log of everything that happens.  Useful
 when debugging freezes or locks in complex code.
 """
+from twisted.python.util import spewer
 sys.settrace(spewer)
 
 def opt_version(self):
@@ -186,6 +187,7 @@ def setupGlobalOptions(logging, start_tor, 
check_incoherences):
 log.start(global_options['logfile'])
 
 if config.privacy.includepcap:
+from ooni.utils.net import hasRawSocketPermission
 if hasRawSocketPermission():
 from ooni.utils.txscapy import ScapyFactory
 config.scapyFactory = ScapyFactory(config.advanced.interface)
@@ -223,6 +225,9 @@ def setupCollector(global_options, collector_address):
 
 
 def createDeck(global_options, url=None):
+from ooni.nettest import NetTestLoader
+from ooni.deck import Deck, nettest_to_path
+
 if url:
 log.msg("Creating deck for: %s" % (url))
 
@@ -331,9 +336,9 @@ def runWithDirector(logging=True, start_tor=True, 
check_incoherences=True):
 Instance the director, parse command line options and start an ooniprobe
 test!
 """
-
 global_options = setupGlobalOptions(logging, start_tor, check_incoherences)
 
+from ooni.director import Director
 director = Director()
 if global_options['list']:
 print "# Installed nettests"
@@ -379,7 +384,8 @@ def runWithDaemonDirector(logging=True, start_tor=True, 
check_incoherences=True)
 Instance the director, parse command line options and start an ooniprobe
 test!
 """
-
+from twisted.internet import reactor, protocol
+from ooni.director import Director
 try:
 import pika
 from pika import exceptions



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] Add support for downloading resources from mirror on github.

2016-04-29 Thread art

commit 6a6e0ac21e9ee20db47df37a4fc9d4fb4e3be794
Author: Arturo FilastÃ² 
Date:   Sat Apr 23 15:20:02 2016 +0200

Add support for downloading resources from mirror on github.

* Download resources when they are missing from oonideckgen
* Remove dependency on GeoLiteCity database (16MB is just too much)
---
 ooni/deckgen/cli.py  | 32 +++--
 ooni/deckgen/processors/citizenlab_test_lists.py |  6 +-
 ooni/geoip.py| 11 ---
 ooni/resources/__init__.py   | 59 +---
 ooni/resources/cli.py| 37 --
 ooni/resources/update.py | 86 ++--
 ooni/tests/test_geoip.py |  1 -
 7 files changed, 93 insertions(+), 139 deletions(-)

diff --git a/ooni/deckgen/cli.py b/ooni/deckgen/cli.py
index 1f618b7..b1a6b87 100644
--- a/ooni/deckgen/cli.py
+++ b/ooni/deckgen/cli.py
@@ -13,8 +13,9 @@ from ooni.geoip import ProbeIP
 from ooni.settings import config
 
 from ooni.deckgen import __version__
-from ooni.resources import inputs
-
+from ooni.deckgen.processors import citizenlab_test_lists
+from ooni.deckgen.processors import namebench_dns_servers
+from ooni.resources.update import download_resources
 
 class Options(usage.Options):
 synopsis = """%s [options]
@@ -70,11 +71,9 @@ class Deck(object):
 
 
 def generate_deck(options):
-dns_servers_processor = inputs['namebench-dns-servers.csv']['processor']
-url_lists_processor = inputs['citizenlab-test-lists.zip']['processor']
 
 try:
-url_list_country = url_lists_processor.generate_country_input(
+url_list_country = citizenlab_test_lists.generate_country_input(
 options['country-code'],
 options['output']
 )
@@ -84,10 +83,10 @@ def generate_deck(options):
 print "We will just use the global one."
 url_list_country = None
 
-url_list_global = url_lists_processor.generate_global_input(
+url_list_global = citizenlab_test_lists.generate_global_input(
 options['output']
 )
-dns_servers = dns_servers_processor.generate_country_input(
+dns_servers = namebench_dns_servers.generate_country_input(
 options['country-code'],
 options['output']
 )
@@ -124,6 +123,20 @@ def get_user_country_code():
 yield probe_ip.lookup()
 defer.returnValue(probe_ip.geodata['countrycode'])
 
+def resources_up_to_date():
+if config.get_data_file_path("GeoIP/GeoIP.dat") is None:
+return False
+
+if config.get_data_file_path("resources/"
+ "namebench-dns-servers.csv") is None:
+return False
+
+if config.get_data_file_path("resources/"
+ "citizenlab-test-lists/"
+ "global.csv") is None:
+return False
+
+return True
 
 @defer.inlineCallbacks
 def run():
@@ -135,6 +148,11 @@ def run():
 print options
 sys.exit(1)
 
+if not resources_up_to_date():
+print("Resources for running ooniprobe are not up to date.")
+print("Will update them now.")
+yield download_resources()
+
 if not options['output']:
 options['output'] = os.getcwd()
 
diff --git a/ooni/deckgen/processors/citizenlab_test_lists.py 
b/ooni/deckgen/processors/citizenlab_test_lists.py
index 086b550..8a660cf 100644
--- a/ooni/deckgen/processors/citizenlab_test_lists.py
+++ b/ooni/deckgen/processors/citizenlab_test_lists.py
@@ -28,10 +28,9 @@ def generate_country_input(country_code, dst):
 
 input_list = config.get_data_file_path("resources/"
"citizenlab-test-lists/"
-   "test-lists-master/lists/"
+ country_code + ".csv")
 
-if not os.path.exists(input_list):
+if not input_list:
 raise Exception("Could not find list for country %s" % country_code)
 
 load_input(input_list, filename)
@@ -44,12 +43,11 @@ def generate_global_input(dst):
 
 input_list = config.get_data_file_path("resources/"
"citizenlab-test-lists/"
-   "test-lists-master/lists/"
"global.csv")
 
 if not input_list:
 print("Could not find the global input list")
-print("Perhaps you should run ooniresources --update-inputs")
+print("Perhaps you should run ooniresources")
 raise Exception("Could not find the global input list")
 
 load_input(input_list, filename)
diff --git a/ooni/geoip.py b/ooni/geoip.py
index 11800b6..83f3648 100644
--- a/ooni/geoip.py
+++ b/ooni/geoip.py
@@ -31,7 +31,6 @@ class GeoIPDataFilesNotFound(Exception):
 def IPToLocation(ipaddr):
 from ooni.settings import config
 
-city_file =

[tor-commits] [ooni-probe/master] fix the dependency on pyasn1 (#470)

2016-04-29 Thread art

commit f0b6332693cd7ea149d2b43780e133d9489a1379
Author: Veggie Monster 
Date:   Tue Apr 26 09:44:51 2016 -0400

fix the dependency on pyasn1 (#470)
---
 requirements.txt | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/requirements.txt b/requirements.txt
index 0143b58..28e5f78 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -10,7 +10,7 @@ scapy-real>=2.2.0-dev
 pypcap>=1.1
 service-identity
 pydumbnet
-pyasn1
+pyasn1>=0.1.8
 pyasn1-modules
 characteristic
 zope.interface



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] Explicitly test twisted version 15.5.0

2016-04-29 Thread art

commit 52bb445c881d111b19c3a8f365b435dc8bb889b8
Author: Arturo FilastÃ² 
Date:   Fri Feb 5 20:54:34 2016 +0100

Explicitly test twisted version 15.5.0
---
 .travis.yml | 1 +
 1 file changed, 1 insertion(+)

diff --git a/.travis.yml b/.travis.yml
index acda409..e3033f6 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -10,6 +10,7 @@ env:
   - TWISTED=Twisted==15.2.1
   - TWISTED=Twisted==15.3.0
   - TWISTED=Twisted==15.4.0
+  - TWISTED=Twisted==15.5.0
   - TWISTED=Twisted
 before_install:
   - sudo apt-get update



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] [hotfix] fix _representBody with commit lost in rebase

2016-04-29 Thread art

commit d916e0144553cc2f655c358bcf6a2fb270ae7e0e
Author: Arturo FilastÃ² 
Date:   Mon Apr 11 17:10:13 2016 +0200

[hotfix] fix _representBody with commit lost in rebase
---
 ooni/templates/httpt.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/ooni/templates/httpt.py b/ooni/templates/httpt.py
index b9ca2e4..6ca486a 100644
--- a/ooni/templates/httpt.py
+++ b/ooni/templates/httpt.py
@@ -139,6 +139,7 @@ class HTTPTest(NetTestCase):
 body = body.replace('\0', '')
 except UnicodeDecodeError:
 body = base64Dict(body)
+return body
 
 log.debug("Adding %s to report" % request)
 request_headers = TrueHeaders(request['headers'])



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] [bugfix] Critical bug that sometimes lead to incorrect body content

2016-04-29 Thread art

commit 0bfb58cd9d2647e6defa3bcdf0dbd7a9e2e16189
Author: Arturo FilastÃ² 
Date:   Thu Apr 14 22:44:47 2016 +0200

[bugfix] Critical bug that sometimes lead to incorrect body content
---
 ooni/utils/trueheaders.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/ooni/utils/trueheaders.py b/ooni/utils/trueheaders.py
index b7671b5..d5aa206 100644
--- a/ooni/utils/trueheaders.py
+++ b/ooni/utils/trueheaders.py
@@ -103,7 +103,7 @@ class HTTPClientParser(_newclient.HTTPClientParser):
 self._partialHeader = None
 
 def headerReceived(self, name, value):
-if self.isConnectionControlHeader(name):
+if self.isConnectionControlHeader(name.lower()):
 headers = self.connHeaders
 else:
 headers = self.headers



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] OONI Control Interface

2016-04-29 Thread art

commit 754ce68db5e11e0d9b6df7fbc5e7b898489af061
Author: Poly 
Date:   Tue Apr 5 15:14:25 2016 +

OONI Control Interface

control interface initial commit



small updates to spec



formatting



added deletion to results API



transition deck start to POST



added netttest support and more



spelling mistakes :P



added more error code specs



more error codes



more error codes 2



completed error codes



s/deck/decks/g



added link to API spec



fixed link
---
 docs/source/architecture.rst  | 128 +---
 docs/source/control_interface.rst | 429 ++
 2 files changed, 435 insertions(+), 122 deletions(-)

diff --git a/docs/source/architecture.rst b/docs/source/architecture.rst
index d72a75c..5e7d253 100644
--- a/docs/source/architecture.rst
+++ b/docs/source/architecture.rst
@@ -179,128 +179,12 @@ Here we describe how an ooniprobe run should look like:
report file and/or the collector will be updated.
 
 
-ooniprobe API
--
-
-Note: This is currently not under active development, but we are looking for
-people interested in hacking on it!
-
-The goals of the ooniprobe API is that of allowing applications to interact
-with an ooniprobe.
-
-Such API will be exposed as an HTTP service that communicates with applications
-thanks to JSON formatted messages.
-
-When ooniprobe is launched it will start as a daemon. Through the HTTP based
-API it will be possible to `start tests`_, `stop tests`_ and `monitor test
-progress`_.
-
-By having it use HTTP it will be possible to decouple the problem of running
-tests from that of controlling them.
-You can think of it as the Tor Control port of ooniprobe.
-
-The kinds of clients that will be interested in using such API are:
-
-  * The ooniprobe *command line interface*
-
-  * The ooniprobe *HTML5/JS web application*
-
-  * Any *third party* tool that is interested in running ooniprobe tests with
-custom arguments.
-
-Towards a RPC like interface
-
-
-Such API will allow ooniprobes to expose such API as a Tor Hidden
-Service (if the user wishes to do so). This will allow people running ooniprobe
-to give the .onion address of the probe to a censorship researcher and they
-will then be able to run tests from the network vantage point of the probe.
-
-Draft API specification
-
-
-Through the ooniprobe API it will be possible to `start tests`_, `stop tests`_ 
and `monitor test
-progress`_.
-
-List tests
-..
-
-`GET /test`
-
-Shall return the list of available tests as an array.
-
-This is how a response looks like
-::
-
-  [{'id': 'http_requests',
-'name': 'HTTP Requests Test',
-'description': 'This test perform a HTTP GET request for the / resource 
over the test network and over Tor',
-'type': [ 'blocking' ],
-'version': '0.1',
-'arguments': {
-  'urllist': 'Specify the list of URLs to be used for the test'
-}
-  }]
-
-*type* may be either **blocking** or **manipulation**.
-
-Start tests
-...
-
-
-`POST /test//start`
-
-Is used to start a test with the specified test_id.
-
-Inside of the request you will specify the arguments supported by the test
-
-This is how a request could look like
-::
-  {
-'urllist':
-  ['http://google.com/', 'http://torproject.org/']
-  }
-
-The server will then respond with the test object
-::
-  {
-'status': 'running',
-'percentage': 0,
-'current_input': 'http://google.com/',
-'urllist':
-  ['http://google.com/', 'http://torproject.org/']
-  }
-
-
-Stop tests
-...
-
-`POST /test//stop`
-
-This will terminate the execution of the test with the specified test_id.
-
-The request may optionally contain a reason for stopping the test such as
-::
-  {
-'reason': 'some reason'
-  }
-
-Monitor test progress
-.
-
-`GET /test/`
-
-Will return the status of a test
-
-Like so for example
-::
-  {
-'status': 'running',
-'percentage': 0,
-'current_input': 'http://google.com/',
-'urllist':
-  ['http://google.com/', 'http://torproject.org/']
-  }
+OONIprobe Control Interface
+---
+.. XXX update this section once interface is implemented.
+The ooniprobe client provides a rich and simple JSON-based interface for 
+control over HTTP. While the implementation of this interface is currently
+a work in progress, the specification may be found `here 
`_.
 
 
 Implementation status
diff --git a/docs/source/control_interface.rst 
b/docs/source/control_interface.rst
new file mode 100644
index 000..0104e4d
--- /dev/null
+++ b/docs/source/control_interface.rst
@@ -0,0 +1,429 @@
+
+HTTP Control Interface Specification

[tor-commits] [ooni-probe/master] Add test for Vanilla Tor

2016-04-29 Thread art

commit 8b7adcae3bc80d3a844b60aacde5b6d5f8a0a2ae
Author: Arturo FilastÃ² 
Date:   Fri Mar 11 16:06:21 2016 +0100

Add test for Vanilla Tor
---
 ooni/nettests/blocking/vanilla_tor.py | 105 ++
 1 file changed, 105 insertions(+)

diff --git a/ooni/nettests/blocking/vanilla_tor.py 
b/ooni/nettests/blocking/vanilla_tor.py
new file mode 100644
index 000..257f21a
--- /dev/null
+++ b/ooni/nettests/blocking/vanilla_tor.py
@@ -0,0 +1,105 @@
+# -*- encoding: utf-8 -*-
+import os
+import tempfile
+import shutil
+
+from twisted.python import usage
+from twisted.internet import reactor, error
+
+import txtorcon
+
+from ooni.utils import log, onion, net
+from ooni import nettest
+
+
+class TorIsNotInstalled(Exception):
+pass
+
+
+class UsageOptions(usage.Options):
+optParameters = [
+['timeout', 't', 200,
+ 'Specify the timeout after which to consider '
+ 'the Tor bootstrapping process to have failed'], ]
+
+
+class VanillaTor(nettest.NetTestCase):
+name = "Vanilla Tor"
+description = "A test for checking if vanilla Tor connections work"
+author = "Arturo FilastÃ²"
+version = "0.1.0"
+
+usageOptions = UsageOptions
+
+def requirements(self):
+if not onion.find_tor_binary():
+raise TorIsNotInstalled(
+"For instructions on installing Tor see: "
+"https://www.torproject.org/download/download;)
+
+def setUp(self):
+self.tor_progress = 0
+self.timeout = int(self.localOptions['timeout'])
+
+fd, self.tor_logfile = tempfile.mkstemp()
+os.close(fd)
+self.tor_datadir = tempfile.mkdtemp()
+
+self.report['error'] = None
+self.report['success'] = None
+self.report['timeout'] = self.timeout
+self.report['transport_name'] = 'vanilla'
+self.report['tor_version'] = str(onion.tor_details['version'])
+self.report['tor_progress'] = 0
+self.report['tor_progress_tag'] = None
+self.report['tor_progress_summary'] = None
+self.report['tor_log'] = None
+
+def test_full_tor_connection(self):
+config = txtorcon.TorConfig()
+config.ControlPort = net.randomFreePort()
+config.SocksPort = net.randomFreePort()
+config.DataDirectory = self.tor_datadir
+log.msg(
+"Connecting to tor %s" %
+(onion.tor_details['version']))
+
+config.log = ['notice stdout', 'notice file %s' % self.tor_logfile]
+config.save()
+
+def updates(prog, tag, summary):
+log.msg("Progress is at: %s%%" % (prog))
+self.report['tor_progress'] = int(prog)
+self.report['tor_progress_tag'] = tag
+self.report['tor_progress_summary'] = summary
+
+d = txtorcon.launch_tor(config, reactor, timeout=self.timeout,
+progress_updates=updates)
+
+@d.addCallback
+def setup_complete(proto):
+try:
+proto.transport.signalProcess('TERM')
+except error.ProcessExitedAlready:
+proto.transport.loseConnection()
+log.msg("Successfully connected to Tor")
+self.report['success'] = True
+
+@d.addErrback
+def setup_failed(failure):
+log.msg("Failed to connect to Tor")
+self.report['success'] = False
+self.report['error'] = 'timeout-reached'
+return
+
+@d.addCallback
+def write_log(_):
+with open(self.tor_logfile) as f:
+self.report['tor_log'] = f.read()
+os.remove(self.tor_logfile)
+try:
+shutil.rmtree(self.tor_datadir)
+except:
+pass
+
+return d



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] Add support for detecting the charset of the HTML response body via the meta tag

2016-04-29 Thread art

commit 37564aa2876ba18516bb28e0e8eec7b489eea1c3
Author: Arturo FilastÃ² 
Date:   Thu Apr 14 17:16:46 2016 +0200

Add support for detecting the charset of the HTML response body via the 
meta tag
---
 ooni/templates/httpt.py  | 39 ++-
 ooni/tests/test_templates.py | 10 ++
 2 files changed, 36 insertions(+), 13 deletions(-)

diff --git a/ooni/templates/httpt.py b/ooni/templates/httpt.py
index 6ca486a..edab3fa 100644
--- a/ooni/templates/httpt.py
+++ b/ooni/templates/httpt.py
@@ -1,3 +1,4 @@
+import re
 import random
 
 from twisted.internet import defer
@@ -16,6 +17,7 @@ from ooni.utils.net import BodyReceiver, StringProducer, 
userAgents
 from ooni.utils.trueheaders import TrueHeaders
 from ooni.errors import handleAllFailures
 
+META_CHARSET_REGEXP = 
re.compile(']*?charset\s*=[\s"\']*([^\s"\'/>]*)')
 
 class InvalidSocksProxyOption(Exception):
 pass
@@ -35,6 +37,30 @@ class StreamListener(StreamListenerMixin):
 except:
 log.err("Tor Exit ip detection failed")
 
+
+
+def _representBody(body):
+# XXX perhaps add support for decoding gzip in the future.
+body = body.replace('\0', '')
+decoded = False
+charsets = ['ascii', 'utf-8']
+
+# If we are able to detect the charset of body from the meta tag
+# try to decode using that one first
+charset = META_CHARSET_REGEXP.search(body, re.IGNORECASE)
+if charset:
+charsets.insert(0, charset.group(1))
+for encoding in charsets:
+try:
+body = unicode(body, encoding)
+decoded = True
+break
+except UnicodeDecodeError:
+pass
+if not decoded:
+body = base64Dict(body)
+return body
+
 class HTTPTest(NetTestCase):
 """
 A utility class for dealing with HTTP based testing. It provides methods to
@@ -128,19 +154,6 @@ class HTTPTest(NetTestCase):
 represented_headers[name] = value[0]
 return represented_headers
 
-def _representBody(body):
-# XXX perhaps add support for decoding gzip in the future.
-try:
-body = unicode(body, 'ascii')
-body = body.replace('\0', '')
-except UnicodeDecodeError:
-try:
-body = unicode(body, 'utf-8')
-body = body.replace('\0', '')
-except UnicodeDecodeError:
-body = base64Dict(body)
-return body
-
 log.debug("Adding %s to report" % request)
 request_headers = TrueHeaders(request['headers'])
 session = {
diff --git a/ooni/tests/test_templates.py b/ooni/tests/test_templates.py
index bf05e56..931e052 100644
--- a/ooni/tests/test_templates.py
+++ b/ooni/tests/test_templates.py
@@ -44,6 +44,16 @@ class TestHTTPT(unittest.TestCase):
 yield self.assertFailure(http_test.doRequest('http://invaliddomain/'), 
DNSLookupError)
 assert http_test.report['requests'][0]['failure'] == 'dns_lookup_error'
 
+def test_charset_detection(self):
+no_charset_html = """
+http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd;>
+
+
+Foo
+"""
+with_charset_html = no_charset_html + '\n'
+self.assertEqual(httpt.META_CHARSET_REGEXP.search(no_charset_html), 
None)
+
self.assertEqual(httpt.META_CHARSET_REGEXP.search(with_charset_html).group(1), 
'iso-8859-1')
 
 class TestDNST(unittest.TestCase):
 def test_represent_answer_a(self):



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] Fix bug in converting failure to string

2016-04-29 Thread art

commit 250d86d5247463447f3abf0607c1c6bf0997c9a8
Author: Arturo FilastÃ² 
Date:   Thu Apr 14 22:42:49 2016 +0200

Fix bug in converting failure to string
---
 ooni/errors.py | 8 +++-
 1 file changed, 3 insertions(+), 5 deletions(-)

diff --git a/ooni/errors.py b/ooni/errors.py
index e523de0..f98a09f 100644
--- a/ooni/errors.py
+++ b/ooni/errors.py
@@ -78,11 +78,9 @@ def failureToString(failure):
 
 for failure_type, failure_string in known_failures:
 if isinstance(failure.value, failure_type):
-if failure_string:
-return failure_string
-else:
-# Failure without a corresponding failure message
-return 'unknown_failure %s' % str(failure.value)
+return failure_string
+# Failure without a corresponding failure message
+return 'unknown_failure %s' % str(failure.value)
 
 class DirectorException(Exception):
 pass



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] [hotfix] Ensure the _setUp() method gets called only once for every input

2016-04-29 Thread art

commit a0fc5e6f4ef76cd935839f08cf9f3db962d95a99
Author: Arturo FilastÃ² 
Date:   Fri Apr 1 19:52:25 2016 +0200

[hotfix] Ensure the _setUp() method gets called only once for every input
---
 ooni/nettest.py | 3 +++
 ooni/tasks.py   | 1 -
 ooni/templates/httpt.py | 4 +---
 3 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/ooni/nettest.py b/ooni/nettest.py
index 848d958..43ae23b 100644
--- a/ooni/nettest.py
+++ b/ooni/nettest.py
@@ -525,6 +525,7 @@ class NetTest(object):
 for input in test_class.inputs:
 measurements = []
 test_instance = test_class()
+test_instance._setUp()
 test_instance.summary = self.summary
 for method in test_methods:
 log.debug("Running %s %s" % (test_class, method))
@@ -645,6 +646,7 @@ class NetTestCase(object):
 def _setUp(self):
 """
 This is the internal setup method to be overwritten by templates.
+It gets called once for every input.
 """
 self.report = {}
 self.inputs = None
@@ -659,6 +661,7 @@ class NetTestCase(object):
 def setUp(self):
 """
 Place here your logic to be executed when the test is being setup.
+It gets called once every test method + input.
 """
 pass
 
diff --git a/ooni/tasks.py b/ooni/tasks.py
index 3f331a2..bd8cbc9 100644
--- a/ooni/tasks.py
+++ b/ooni/tasks.py
@@ -110,7 +110,6 @@ class Measurement(TaskWithTimeout):
 """
 self.testInstance = test_instance
 self.testInstance.input = test_input
-self.testInstance._setUp()
 if not hasattr(self.testInstance, '_start_time'):
 self.testInstance._start_time = time.time()
 
diff --git a/ooni/templates/httpt.py b/ooni/templates/httpt.py
index 3702e86..2f1136c 100644
--- a/ooni/templates/httpt.py
+++ b/ooni/templates/httpt.py
@@ -141,7 +141,6 @@ class HTTPTest(NetTestCase):
 }
 if failure_string:
 request_response['failure'] = failure_string
-
 self.report['requests'].append(request_response)
 
 def _processResponseBody(self, response_body, request, response, 
body_processor):
@@ -312,8 +311,7 @@ class HTTPTest(NetTestCase):
 log.debug("Randomizing user agent")
 self.randomize_useragent(request)
 
-if 'requests' not in self.report:
-self.report['requests'] = []
+self.report['requests'] = self.report.get('requests', [])
 
 # If we have a request body payload, set the request body to such
 # content



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] When GeoIP lookup is disabled it's not possible to perform sanitisations

2016-04-29 Thread art

commit 81d05bdd15bf68c271cac5b0cde257372f7b4892
Author: Arturo FilastÃ² 
Date:   Tue Mar 15 14:48:06 2016 +0100

When GeoIP lookup is disabled it's not possible to perform sanitisations
---
 ooni/templates/httpt.py   | 2 +-
 ooni/templates/process.py | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/ooni/templates/httpt.py b/ooni/templates/httpt.py
index d0822f2..12995bb 100644
--- a/ooni/templates/httpt.py
+++ b/ooni/templates/httpt.py
@@ -135,7 +135,7 @@ class HTTPTest(NetTestCase):
 }
 if response:
 # Attempt to redact the IP address of the probe from the responses
-if config.privacy.includeip is False:
+if config.privacy.includeip is False and config.probe_ip.address 
is not None:
 response_body = response_body.replace(config.probe_ip.address, 
"[REDACTED]")
 request_response['response'] = {
 'headers': list(response.headers.getAllRawHeaders()),
diff --git a/ooni/templates/process.py b/ooni/templates/process.py
index 42a4256..faf0a66 100644
--- a/ooni/templates/process.py
+++ b/ooni/templates/process.py
@@ -108,7 +108,7 @@ class ProcessTest(NetTestCase):
 self.report['commands'] = []
 
 # Attempt to redact the IP address of the probe from the standard 
output
-if config.privacy.includeip is False:
+if config.privacy.includeip is False and config.probe_ip.address is 
not None:
 result['stdout'] = 
result['stdout'].replace(config.probe_ip.address, "[REDACTED]")
 result['stderr'] = 
result['stderr'].replace(config.probe_ip.address, "[REDACTED]")
 



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] Refactor lantern test to use the ProcessTest template

2016-04-29 Thread art

commit c1ff7516d35b7f76fb6638ae2bce04db5c4b3864
Author: Arturo FilastÃ² 
Date:   Fri Mar 11 17:55:16 2016 +0100

Refactor lantern test to use the ProcessTest template

* Improve blocking detection
---
 ooni/nettests/third_party/lantern.py | 98 ++--
 1 file changed, 38 insertions(+), 60 deletions(-)

diff --git a/ooni/nettests/third_party/lantern.py 
b/ooni/nettests/third_party/lantern.py
index 1f1b3b0..187e210 100644
--- a/ooni/nettests/third_party/lantern.py
+++ b/ooni/nettests/third_party/lantern.py
@@ -1,47 +1,13 @@
+import os
 from twisted.internet import defer, reactor
 from twisted.internet.endpoints import TCP4ClientEndpoint
-from twisted.python import usage
 from twisted.web.client import ProxyAgent, readBody
 from ooni.templates.process import ProcessTest, ProcessDirector
 from ooni.utils import log
 from ooni.errors import handleAllFailures
-import os.path
-from os import getenv
-
-class UsageOptions(usage.Options):
-optParameters = [
-['url', 'u', None, 'Specify a single URL to test.'],]
-
-class LanternProcessDirector(ProcessDirector):
-"""
-This Process Director monitors Lantern during its
-bootstrap and fires a callback if bootstrap is
-successful or an errback if it fails to bootstrap
-before timing out.
-"""
-
-def __init__(self, d, timeout=None):
-self.d = d
-self.stderr = ""
-self.stdout = ""
-self.finished = None
-self.timeout = timeout
-self.stdin = None
-self.timer = None
-self.exit_reason = None
-self.bootstrapped = defer.Deferred()
-
-def outReceived(self, data):
-self.stdout += data
-# output received, see if we have bootstrapped
-if not self.bootstrapped.called and "Connected to proxy on localhost" 
in self.stdout:
-log.debug("Bootstrap Detected")
-self.cancelTimer()
-self.bootstrapped.callback("bootstrapped")
-
+import distutils.spawn
 
 class LanternTest(ProcessTest):
-
 """
 This class tests Lantern (https://getlantern.org).
 
@@ -54,47 +20,59 @@ class LanternTest(ProcessTest):
 """
 
 name = "Lantern Circumvention Tool Test"
-description = "Bootstraps Lantern and does a HTTP GET for the specified 
URL"
+description = "Bootstraps Lantern, connects to a URL and verifies if it 
contains the expected input"
 author = "Aaron Gibson"
-version = "0.0.1"
-timeout = 20
-usageOptions = UsageOptions
-requiredOptions = ['url']
+version = "0.1.0"
+timeout = 120
 
 def setUp(self):
-self.command = ["lantern", "--headless"]
-self.d = defer.Deferred()
-self.processDirector = LanternProcessDirector(self.d, 
timeout=self.timeout)
-self.d.addCallback(self.processEnded, self.command)
-if self.localOptions['url']:
-self.url = self.localOptions['url']
-
-def runLantern(self):
-paths = filter(os.path.exists,[os.path.join(os.path.expanduser(x), 
self.command[0]) for x in getenv('PATH').split(':')])
-log.debug("Spawning Lantern")
-reactor.spawnProcess(self.processDirector, paths[0], self.command)
+self.command = [distutils.spawn.find_executable("lantern"), 
"--headless"]
+self.bootstrapped = defer.Deferred()
+self.exited = False
+self.url = 'http://www.google.com/humans.txt'
+
+def stop(self, reason=None):
+if not self.exited:
+self.processDirector.close()
+self.processDirector.transport.signalProcess('TERM')
+self.exited = True
+
+def handleRead(self, stdout=None, stderr=None):
+"""
+This is called with each chunk of data from stdout and stderr.
+"""
+if not self.bootstrapped.called and "Successfully dialed via" in 
self.processDirector.stdout:
+log.msg("Lantern connection successful")
+self.processDirector.cancelTimer()
+self.bootstrapped.callback("bootstrapped")
 
 def test_lantern_circumvent(self):
-proxyEndpoint=TCP4ClientEndpoint(reactor, '127.0.0.1', 8787)
-agent = ProxyAgent(proxyEndpoint, reactor)
-
 def addResultToReport(result):
 self.report['body'] = result
-self.report['success'] = True
+if result.startswith('Google is built by a large'):
+log.msg("Got the HTTP response body I expected!")
+self.report['success'] = True
+else:
+self.report['success'] = False
 
 def addFailureToReport(failure):
+log.err("Failed to connect to lantern")
+log.failure(failure)
 self.report['failure'] = handleAllFailures(failure)
 self.report['success'] = False
 
 def doRequest(noreason):
-log.debug("Doing HTTP request via Lantern (127.0.0.1:8787) for %s" 
% self.url)
+proxyEndpoint =

[tor-commits] [ooni-probe/master] Update user documentation for psiphon and lantern

2016-04-29 Thread art

commit c6964f5fb8ddcc3cec5d3eeebb28a07bd9ab0c7b
Author: Arturo FilastÃ² 
Date:   Tue Mar 15 17:53:39 2016 +0100

Update user documentation for psiphon and lantern
---
 docs/source/tests/lantern.rst | 17 +
 docs/source/tests/psiphon.rst | 17 -
 2 files changed, 17 insertions(+), 17 deletions(-)

diff --git a/docs/source/tests/lantern.rst b/docs/source/tests/lantern.rst
index 0903c3a..9f3af2f 100644
--- a/docs/source/tests/lantern.rst
+++ b/docs/source/tests/lantern.rst
@@ -17,22 +17,25 @@ Description
 ===
 
 This test launches Lantern in --headless mode, and parses output to determine
-if it has bootstrapped.  After bootstrap, it fetches the URL supplied by the
---url option using Lanterns http proxy interface listening on 127.0.0.1.8787.
+if it has bootstrapped.  After bootstrap, it fetches a URL using Lanterns HTTP
+proxy interface listening on 127.0.0.1.8787 and checks to see if the response
+body matches the expected result.
+As a URL for testing we use http://www.google.com/humans.txt and look for the
+string "Google is built by a large" in the response body.
 
 The specific string used to determine bootstrap from Lantern output in version
-"2.0.10" is "client (http) proxy at" from standard output.
+"2.0.10" is "Successfully dialed via" from standard output.
 
 How to run the test
 ===
 
-`ooniprobe nettests/third_party/lantern.py -u http://`
+`ooniprobe nettests/third_party/lantern.py`
 
 Sample report
 =
 
 From running:
-`ooniprobe nettests/third_party/lantern.py -u http://www.google.com`
+`ooniprobe nettests/third_party/lantern.py`
 
 ::
 
@@ -50,9 +53,7 @@ test_name: lantern_circumvention_tool_test
 test_version: 0.0.1
 ...
 ---
-body: "\n301 Moved\n301 Moved\nThe document 
has moved\n\
-  http://www.google.com/\;>here.\r\n\r\n"
+body: "Google is built by a large team of engineers, designers, researchers, 
robots, and others in many different sites across the globe. It is updated 
continuously, and built with more tools and technologies than we can shake a 
stick at. If you'd like to help us out, see google.com/careers."
 bootstrapped: true
 input: null
 lantern --headless: {exit_reason: process_done, stderr: '', stdout: ''}
diff --git a/docs/source/tests/psiphon.rst b/docs/source/tests/psiphon.rst
index b05f1f0..3934bcb 100644
--- a/docs/source/tests/psiphon.rst
+++ b/docs/source/tests/psiphon.rst
@@ -16,7 +16,12 @@ Details
 Description
 ===
 
-This test first check that the Psiphon path exists, then launches Psiphon and 
parses output to determine if it has bootstrapped. After bootstrap, it fetches 
google.com (or other URL specified by the --url argument) using Psiphons SOCKS 
proxy listening on 127.0.0.1:1080 (or otherwise specified by the --socksproxy 
argument).
+This test first check that the Psiphon path exists, then launches Psiphon and
+parses output to determine if it has bootstrapped. After bootstrap, it fetches
+`http://www.google.com/humans.txt` using Psiphons SOCKS
+proxy listening on 127.0.0.1:1080 (or otherwise specified by the --socksproxy
+argument).
+It will then check to see if the response body contains the string: "Google is 
built by a large"
 
 The specific string used to determine bootstrap from Psiphon output in version
 "0.0.1" is "Press Ctrl-C to terminate." from standard output.
@@ -26,10 +31,6 @@ How to run the test
 
 `ooniprobe third_party/psiphon`
 
-To test Psiphon with a different URL run:
-
-`ooniprobe third_party/psiphon -u http:///`
-
 To test Psiphon when it is installed in a different path other than the user 
home:
 
 `ooniprobe third_party/psiphon -p `
@@ -49,7 +50,7 @@ To run Psiphon manually, it must be run inside of the proper 
directory:
 Sample report
 =
 
-`ooniprobe third_party/psiphon -u http://google.com`
+`ooniprobe third_party/psiphon`
 
 ::
 
@@ -87,9 +88,7 @@ Sample report
 tor: {is_tor: false}
 url: http://google.com
   response:
-body: "\n301 Moved\n301 Moved\nThe 
document has\
-  \ moved\nhttp://www.google.com/\;>here.\r\n\r\n"
+body: "Google is built by a large team of engineers, designers, 
researchers, robots, and others in many different sites across the globe. It is 
updated continuously, and built with more tools and technologies than we can 
shake a stick at. If you'd like to help us out, see google.com/careers."
 code: 301
 headers:
 - - Content-Length



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] Move daphne to the experimental directory

2016-04-29 Thread art

commit 03a820cbb53d1c9de114b0760ce58da0fdbce8df
Author: Arturo FilastÃ² 
Date:   Thu Mar 24 13:40:27 2016 +0100

Move daphne to the experimental directory

In it's current incarnation it's very hard to use and we should consider it 
experimental
---
 ooni/nettests/experimental/daphne.py | 121 +++
 ooni/nettests/manipulation/daphne.py | 121 ---
 2 files changed, 121 insertions(+), 121 deletions(-)

diff --git a/ooni/nettests/experimental/daphne.py 
b/ooni/nettests/experimental/daphne.py
new file mode 100644
index 000..72b5c8f
--- /dev/null
+++ b/ooni/nettests/experimental/daphne.py
@@ -0,0 +1,121 @@
+# -*- encoding: utf-8 -*-
+from twisted.python import usage
+from twisted.internet import protocol, endpoints, reactor
+
+from ooni import nettest
+from ooni.kit import daphn3
+from ooni.utils import log
+
+class Daphn3ClientProtocol(daphn3.Daphn3Protocol):
+def nextStep(self):
+log.debug("Moving on to next step in the state walk")
+self.current_data_received = 0
+if self.current_step >= (len(self.steps) - 1):
+log.msg("Reached the end of the state machine")
+log.msg("Censorship fingerprint bisected!")
+step_idx, mutation_idx = self.factory.mutation
+log.msg("step_idx: %s | mutation_id: %s" % (step_idx, 
mutation_idx))
+#self.transport.loseConnection()
+if self.report:
+self.report['mutation_idx'] = mutation_idx
+self.report['step_idx'] = step_idx
+self.d.callback(None)
+return
+else:
+self.current_step += 1
+if self._current_step_role() == self.role:
+# We need to send more data because we are again responsible for
+# doing so.
+self.sendPayload()
+
+
+class Daphn3ClientFactory(protocol.ClientFactory):
+protocol = daphn3.Daphn3Protocol
+mutation = [0,0]
+steps = None
+
+def buildProtocol(self, addr):
+p = self.protocol()
+p.steps = self.steps
+p.factory = self
+return p
+
+def startedConnecting(self, connector):
+log.msg("Started connecting %s" % connector)
+
+def clientConnectionFailed(self, reason, connector):
+log.err("We failed connecting the the OONIB")
+log.err("Cannot perform test. Perhaps it got blocked?")
+log.err("Please report this to tor-assista...@torproject.org")
+
+def clientConnectionLost(self, reason, connector):
+log.err("Daphn3 client connection lost")
+print reason
+
+class daphn3Args(usage.Options):
+optParameters = [
+ ['host', 'h', '127.0.0.1', 'Target Hostname'],
+ ['port', 'p', 57003, 'Target port number']]
+
+optFlags = [['pcap', 'c', 'Specify that the input file is a pcap file'],
+['yaml', 'y', 'Specify that the input file is a YAML file 
(default)']]
+
+class daphn3Test(nettest.NetTestCase):
+
+name = "Daphn3"
+description = "Bisects the censors fingerprint by mutating the given input 
packets."
+usageOptions = daphn3Args
+inputFile = ['file', 'f', None, 
+'Specify the pcap or YAML file to be used as input to the test']
+
+#requiredOptions = ['file']
+requiresRoot = False
+requiresTor = False
+steps = None
+
+def inputProcessor(self, filename):
+"""
+step_idx is the step in the packet exchange
+ex.
+[.X.] are packets sent by a client or a server
+
+client:  [.1.][.3.] [.4.]
+server: [.2.] [.5.]
+
+mutation_idx: is the sub index of the packet as in the byte of the
+packet at the step_idx that is to be mutated
+
+"""
+if self.localOptions['pcap']:
+daphn3Steps = daphn3.read_pcap(filename)
+else:
+daphn3Steps = daphn3.read_yaml(filename)
+log.debug("Loaded these steps %s" % daphn3Steps)
+yield daphn3Steps
+
+def test_daphn3(self):
+host = self.localOptions['host']
+port = int(self.localOptions['port'])
+
+def failure(failure):
+log.msg("Failed to connect")
+self.report['censored'] = True
+self.report['mutation'] = 0
+raise Exception("Error in connection, perhaps the backend is 
censored")
+return
+
+def success(protocol):
+log.msg("Successfully connected")
+protocol.sendPayload()
+return protocol.d
+
+log.msg("Connecting to %s:%s" % (host, port))
+endpoint = endpoints.TCP4ClientEndpoint(reactor, host, port)
+daphn3_factory = Daphn3ClientFactory()
+daphn3_factory.steps = self.input
+daphn3_factory.report = self.report
+d = endpoint.connect(daphn3_factory)
+d.addErrback(failure)
+d.addCallback(success)
+

[tor-commits] [ooni-probe/master] Remove dead/unnecessary code from otime

2016-04-29 Thread art

commit 7528c07975edbd45776086ccaf329e0aed923d35
Author: Arturo FilastÃ² 
Date:   Tue Mar 22 17:33:12 2016 +0100

Remove dead/unnecessary code from otime

* Minor naming changes to functions
---
 ooni/nettest.py  |  2 +-
 ooni/otime.py| 80 +++-
 ooni/tasks.py|  2 +-
 ooni/tests/test_otime.py | 14 -
 ooni/utils/log.py|  2 +-
 5 files changed, 8 insertions(+), 92 deletions(-)

diff --git a/ooni/nettest.py b/ooni/nettest.py
index 9c5e171..6cd9f57 100644
--- a/ooni/nettest.py
+++ b/ooni/nettest.py
@@ -206,7 +206,7 @@ class NetTestLoader(object):
 
 options = sanitize_options(self.options)
 self.testDetails = {
-'test_start_time': otime.UTCTimestampNow(),
+'test_start_time': otime.timestampNowLongUTC(),
 'probe_asn': config.probe_ip.geodata['asn'],
 'probe_cc': config.probe_ip.geodata['countrycode'],
 'probe_ip': config.probe_ip.geodata['ip'],
diff --git a/ooni/otime.py b/ooni/otime.py
index 37ada02..ffa95d0 100644
--- a/ooni/otime.py
+++ b/ooni/otime.py
@@ -1,17 +1,4 @@
-from datetime import datetime, timedelta, tzinfo
-
-class UTC(tzinfo):
-"""UTC"""
-ZERO = timedelta(0)
-
-def utcoffset(self, dt):
-return self.ZERO
-
-def tzname(self, dt):
-return "UTC"
-
-def dst(self, dt):
-return self.ZERO
+from datetime import datetime
 
 def prettyDateNow():
 """
@@ -19,72 +6,15 @@ def prettyDateNow():
 """
 return datetime.now().ctime()
 
-def utcPrettyDateNow():
+def prettyDateNowUTC():
 """
 Returns a good looking string for utc time.
 """
 return datetime.utcnow().ctime()
 
-class InvalidTimestampFormat(Exception):
-pass
-
-def fromTimestamp(s):
-"""
-Converts a string that is output from the timestamp function back to a
-datetime object
-
-Args:
-s (str): a ISO8601 formatted string.
-ex. 1912-06-23T101234Z"
-
-Note: we currently only support parsing strings that are generated from the
-timestamp function and have no intention in supporting the full 
standard.
-"""
-try:
-date_part, time_part = s.split('T')
-hours, minutes, seconds = time_part[:2], time_part[2:4], time_part[4:6]
-year, month, day = date_part.split('-')
-except:
-raise InvalidTimestampFormat(s)
-
-return datetime(int(year), int(month), int(day), int(hours), int(minutes),
-int(seconds))
-
-def timestamp(t=None):
+def timestampNowLongUTC():
 """
-The timestamp for ooni reports follows ISO 8601 in
-UTC time format.
-We do not inlcude ':' and include seconds.
-
-Example:
-
-if the current date is "10:12:34 AM, June 23 1912" (datetime(1912, 6,
-23, 10, 12, 34))
-
-the timestamp will be:
-
-   "1912-06-23T101234Z"
-
-Args:
-t (datetime): a datetime object representing the
-time to be represented (*MUST* be expressed
-in UTC).
-
-If not specified will default to the current time
-in UTC.
+Returns a timestamp in the format of %Y-%m-%d %H:%M:%S in Universal Time
+Coordinates.
 """
-if not t:
-t = datetime.utcnow()
-ISO8601 = "%Y-%m-%dT%H%M%SZ"
-return t.strftime(ISO8601)
-
-
-def epochToTimestamp(seconds):
-return timestamp(datetime.fromtimestamp(seconds, UTC()))
-
-
-def epochToUTC(seconds):
-return float(datetime.utcfromtimestamp(seconds).strftime("%s"))
-
-def UTCTimestampNow():
 return datetime.utcnow().strftime("%Y-%m-%d %H:%M:%S")
diff --git a/ooni/tasks.py b/ooni/tasks.py
index 381902a..ac128d6 100644
--- a/ooni/tasks.py
+++ b/ooni/tasks.py
@@ -140,7 +140,7 @@ class Measurement(TaskWithTimeout):
 
 def run(self):
 if 'measurement_start_time' not in self.testInstance.report.keys():
-self.testInstance.report['measurement_start_time'] = 
otime.UTCTimestampNow()
+self.testInstance.report['measurement_start_time'] = 
otime.timestampNowLongUTC()
 return self.netTestMethod()
 
 
diff --git a/ooni/tests/test_otime.py b/ooni/tests/test_otime.py
deleted file mode 100644
index f3997eb..000
--- a/ooni/tests/test_otime.py
+++ /dev/null
@@ -1,14 +0,0 @@
-import unittest
-from datetime import datetime
-from ooni import otime
-
-test_date = datetime(2002, 6, 26, 22, 45, 49)
-
-
-class TestOtime(unittest.TestCase):
-def test_timestamp(self):
-self.assertEqual(otime.timestamp(test_date), "2002-06-26T224549Z")
-
-def test_fromTimestamp(self):
-time_stamp = otime.timestamp(test_date)
-self.assertEqual(test_date, otime.fromTimestamp(time_stamp))
diff --git a/ooni/utils/log.py b/ooni/utils/log.py
index beba78f..d3036f0 100644
--- a/ooni/utils/log.py
+++ b/ooni/utils/log.py
@@ -64,7 +64,7 @@ class OONILogger(object):
 
 txlog.msg("Starting %s on %s (%s UTC)" %

[tor-commits] [ooni-probe/master] Add checks for input file and url in http_requests test

2016-04-29 Thread art

commit 510adaf6c473fb289aca6973b662f0a6cb2a53c9
Author: Arturo FilastÃ² 
Date:   Thu Mar 24 13:50:53 2016 +0100

Add checks for input file and url in http_requests test
---
 ooni/nettests/blocking/http_requests.py | 8 
 1 file changed, 8 insertions(+)

diff --git a/ooni/nettests/blocking/http_requests.py 
b/ooni/nettests/blocking/http_requests.py
index e00b4c6..e2c9671 100644
--- a/ooni/nettests/blocking/http_requests.py
+++ b/ooni/nettests/blocking/http_requests.py
@@ -11,6 +11,8 @@ from ooni.utils.net import userAgents
 from ooni.templates import httpt
 from ooni.errors import failureToString
 
+class MissingInput(Exception):
+pass
 
 class UsageOptions(usage.Options):
 optParameters = [
@@ -50,6 +52,12 @@ class HTTPRequestsTest(httpt.HTTPTest):
 control_body_length = None
 experiment_body_length = None
 
+def requirements(self):
+if not self.localOptions['url'] and \
+not self.localOptions['file']:
+raise MissingInput("You did not specify either a URL with -u "
+   "or an input file with -f")
+
 def setUp(self):
 """
 Check for inputs.



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] hotfix: remove unneeded calls for check_for_censorship in http_host test

2016-04-29 Thread art

commit 7405cacf858399ac7a18bc228cf8662c62a347cf
Author: Arturo FilastÃ² 
Date:   Thu Mar 17 21:00:27 2016 +0100

hotfix: remove unneeded calls for check_for_censorship in http_host test
---
 ooni/nettests/manipulation/http_host.py | 2 --
 1 file changed, 2 deletions(-)

diff --git a/ooni/nettests/manipulation/http_host.py 
b/ooni/nettests/manipulation/http_host.py
index d0387cb..cab4d47 100644
--- a/ooni/nettests/manipulation/http_host.py
+++ b/ooni/nettests/manipulation/http_host.py
@@ -68,14 +68,12 @@ class HTTPHost(httpt.HTTPTest):
 if not body.startswith("{"):
 log.msg("This does not appear to be JSON")
 self.report['transparent_http_proxy'] = True
-self.check_for_censorship(body)
 return
 try:
 content = json.loads(body)
 except:
 log.msg("The json does not parse, this is not what we expected")
 self.report['transparent_http_proxy'] = True
-self.check_for_censorship(body)
 return
 
 # We base the determination of the presence of a transparent HTTP



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] Add check for input file in http_host test

2016-04-29 Thread art

commit 3683957390dca24ff0d29ae40b7ecea06cec29d1
Author: Arturo FilastÃ² 
Date:   Thu Mar 24 13:25:54 2016 +0100

Add check for input file in http_host test
---
 ooni/nettests/manipulation/http_host.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/ooni/nettests/manipulation/http_host.py 
b/ooni/nettests/manipulation/http_host.py
index d0387cb..a38e3cc 100644
--- a/ooni/nettests/manipulation/http_host.py
+++ b/ooni/nettests/manipulation/http_host.py
@@ -21,7 +21,7 @@ class UsageOptions(usage.Options):
 optParameters = [['backend', 'b', None,
   'URL of the test backend to use. Should be \
   listening on port 80 and be a \
-  HTTPReturnJSONHeadersHelper'],
+HTTPReturnJSONHeadersHelper (ex. http://1.1.1.1)'],
  ['content', 'c', None, 'The file to read \
 from containing the content of a block page']]
 
@@ -50,7 +50,7 @@ class HTTPHost(httpt.HTTPTest):
  'List of hostnames to test for censorship']
 
 requiredTestHelpers = {'backend': 'http-return-json-headers'}
-requiredOptions = ['backend']
+requiredOptions = ['backend', 'file']
 requiresTor = False
 requiresRoot = False
 



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] Add unittests for the DNS test template and representing of answers

2016-04-29 Thread art

commit c0d2cc0d4fb087035ae57d2730f82b121b427eca
Author: Arturo FilastÃ² 
Date:   Thu Mar 24 16:42:31 2016 +0100

Add unittests for the DNS test template and representing of answers
---
 ooni/tests/test_templates.py | 44 ++--
 1 file changed, 42 insertions(+), 2 deletions(-)

diff --git a/ooni/tests/test_templates.py b/ooni/tests/test_templates.py
index cbd378c..bf05e56 100644
--- a/ooni/tests/test_templates.py
+++ b/ooni/tests/test_templates.py
@@ -1,9 +1,11 @@
-from ooni.templates import httpt
+from ooni.templates import httpt, dnst
 
+from twisted.names import dns
 from twisted.internet.error import DNSLookupError
-from twisted.internet import reactor, defer
+from twisted.internet import reactor, defer, base
 from twisted.trial import unittest
 
+base.DelayedCall.debug = True
 
 class TestHTTPT(unittest.TestCase):
 def setUp(self):
@@ -41,3 +43,41 @@ class TestHTTPT(unittest.TestCase):
 http_test._setUp()
 yield self.assertFailure(http_test.doRequest('http://invaliddomain/'), 
DNSLookupError)
 assert http_test.report['requests'][0]['failure'] == 'dns_lookup_error'
+
+
+class TestDNST(unittest.TestCase):
+def test_represent_answer_a(self):
+a_record = dns.RRHeader(payload=dns.Record_A(address="1.1.1.1"),
+type=dns.A)
+self.assertEqual(dnst.representAnswer(a_record),
+ {'ipv4': '1.1.1.1', 'answer_type': 'A'})
+
+def test_represent_answer_ptr(self):
+ptr_record = dns.RRHeader(payload=dns.Record_PTR(name="example.com"),
+  type=dns.PTR)
+self.assertEqual(dnst.representAnswer(ptr_record),
+ {'hostname': 'example.com', 'answer_type': 'PTR'})
+
+def test_represent_answer_soa(self):
+ptr_record = dns.RRHeader(payload=dns.Record_SOA(mname='example.com',
+ 
rname='foo.example.com'),
+  type=dns.SOA)
+represented_answer = {}
+represented_answer['ttl'] = None
+represented_answer['answer_type'] = 'SOA'
+represented_answer['hostname'] = 'example.com'
+represented_answer['responsible_name'] = 'foo.example.com'
+represented_answer['serial_number'] = 0
+represented_answer['refresh_interval'] = 0
+represented_answer['retry_interval'] = 0
+represented_answer['minimum_ttl'] = 0
+represented_answer['expiration_limit'] = 0
+self.assertEqual(dnst.representAnswer(ptr_record),
+ represented_answer)
+
+@defer.inlineCallbacks
+def test_perform_a_lookup(self):
+dns_test = dnst.DNSTest()
+dns_test._setUp()
+result = yield dns_test.performALookup('example.com', 
dns_server=('8.8.8.8', 53))
+self.assertEqual(result, ['93.184.216.34'])



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] [Refactoring] oonicli logic

2016-04-29 Thread art

commit 18385338507bf4c3f5959e0de68a3ad33f291353
Author: Arturo FilastÃ² 
Date:   Wed Mar 30 14:08:01 2016 +0200

[Refactoring] oonicli logic
---
 ooni/oonicli.py | 37 -
 1 file changed, 16 insertions(+), 21 deletions(-)

diff --git a/ooni/oonicli.py b/ooni/oonicli.py
index 9509a18..63312b5 100644
--- a/ooni/oonicli.py
+++ b/ooni/oonicli.py
@@ -209,25 +209,22 @@ def setupAnnotations(global_options):
 global_options["annotations"] = annotations
 return annotations
 
-def setupCollector(global_options, net_test_loader):
-collector = None
-if not global_options['no-collector']:
-if global_options['collector']:
-collector = global_options['collector']
-elif 'collector' in config.reports \
-and config.reports['collector']:
-collector = config.reports['collector']
-elif net_test_loader.collector:
-collector = net_test_loader.collector
-
-if collector and collector.startswith('httpo:') \
+def setupCollector(global_options, collector_address):
+if global_options['collector']:
+collector_address = global_options['collector']
+elif 'collector' in config.reports \
+and config.reports['collector']:
+collector_address = config.reports['collector']
+
+if collector_address.startswith('httpo:') \
 and (not (config.tor_state or config.tor.socks_port)):
 raise errors.TorNotRunning
-return collector
+return collector_address
 
 
 def createDeck(global_options, url=None):
-log.msg("Creating deck for: %s" % (url))
+if url:
+log.msg("Creating deck for: %s" % (url))
 
 if global_options['no-yamloo']:
 log.msg("Will not write to a yamloo report file")
@@ -311,18 +308,16 @@ def runTestWithDirector(director, global_options, 
url=None,
 # If a collector is not specified in the deck, or the
 # deck is a singleton, the default collector set in
 # ooniprobe.conf will be used
-
-collector = setupCollector(global_options, net_test_loader)
-
-if collector and collector.startswith('httpo:') \
-and (not (config.tor_state or config.tor.socks_port)):
-raise errors.TorNotRunning
+collector_address = None
+if not global_options['no-collector']:
+collector_address = setupCollector(global_options,
+   net_test_loader.collector)
 
 net_test_loader.annotations = global_options['annotations']
 
 yield director.startNetTest(net_test_loader,
 global_options['reportfile'],
-collector,
+collector_address,
 global_options['no-yamloo'])
 
 d.addCallback(setup_nettest)



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] Fix bug in parsing of options for Lantern

2016-04-29 Thread art

commit bcc2be41fb5b3a0ce45e5dce81718f677fbf0af5
Author: Arturo FilastÃ² 
Date:   Wed Mar 16 20:10:12 2016 +0100

Fix bug in parsing of options for Lantern

* Add check to assert if lantern is installed in test requirements
---
 ooni/nettests/third_party/lantern.py | 8 
 1 file changed, 8 insertions(+)

diff --git a/ooni/nettests/third_party/lantern.py 
b/ooni/nettests/third_party/lantern.py
index da19566..5018ccb 100644
--- a/ooni/nettests/third_party/lantern.py
+++ b/ooni/nettests/third_party/lantern.py
@@ -10,6 +10,9 @@ from ooni.templates.process import ProcessTest, 
ProcessDirector
 from ooni.utils import log, net
 from ooni.errors import handleAllFailures
 
+class LanternNotInstalled(Exception):
+pass
+
 class UsageOptions(usage.Options):
 optParameters = [
 ['url', 'u', net.GOOGLE_HUMANS[0],
@@ -35,6 +38,11 @@ class LanternTest(ProcessTest):
 author = "Aaron Gibson"
 version = "0.1.0"
 timeout = 120
+usageOptions = UsageOptions
+
+def requirements(self):
+if not distutils.spawn.find_executable("lantern"):
+raise LanternNotInstalled('lantern is not installed')
 
 def setUp(self):
 self.report['body'] = None



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] Fix bug that lead to the probe_asn and probe_cc not being set

2016-04-29 Thread art

commit 29e63ed80eb9298e3ce86f7d4582b2c6ee909afe
Author: Arturo FilastÃ² 
Date:   Wed Mar 30 14:07:22 2016 +0200

Fix bug that lead to the probe_asn and probe_cc not being set
---
 ooni/director.py | 6 +-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/ooni/director.py b/ooni/director.py
index 4aa5fba..ab8da65 100644
--- a/ooni/director.py
+++ b/ooni/director.py
@@ -241,12 +241,16 @@ class Director(object):
 net_test_loader:
 an instance of :class:ooni.nettest.NetTestLoader
 """
+# Here we set the test details again since the geoip lookups may
+# not have already been done and probe_asn and probe_ip
+# are not set.
+net_test_loader.setTestDetails()
+
 if self.allTestsDone.called:
 self.allTestsDone = defer.Deferred()
 
 if config.privacy.includepcap:
 self.startSniffing(net_test_loader.testDetails)
-
 report = Report(net_test_loader.testDetails, report_filename,
 self.reportEntryManager, collector_address,
 no_yamloo)



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] Set the measurement_start_time closer to the test method call to improve accuracy

2016-04-29 Thread art

commit cb718ade5beaf546bb87375f4a79bf16bd15956a
Author: Arturo FilastÃ² 
Date:   Thu Mar 17 19:44:49 2016 +0100

Set the measurement_start_time closer to the test method call to improve 
accuracy
---
 ooni/tasks.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/ooni/tasks.py b/ooni/tasks.py
index b5a15e0..381902a 100644
--- a/ooni/tasks.py
+++ b/ooni/tasks.py
@@ -116,8 +116,6 @@ class Measurement(TaskWithTimeout):
 
 if 'input' not in self.testInstance.report.keys():
 self.testInstance.report['input'] = test_input
-if 'measurement_start_time' not in self.testInstance.report.keys():
-self.testInstance.report['measurement_start_time'] = 
otime.UTCTimestampNow()
 
 self.testInstance.setUp()
 
@@ -141,6 +139,8 @@ class Measurement(TaskWithTimeout):
 pass
 
 def run(self):
+if 'measurement_start_time' not in self.testInstance.report.keys():
+self.testInstance.report['measurement_start_time'] = 
otime.UTCTimestampNow()
 return self.netTestMethod()
 
 



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] Add test for YAML based reporter

2016-04-29 Thread art

commit 1e9f2316859825343ae2ae2f28b6fd5ff2b4266b
Author: Arturo FilastÃ² 
Date:   Sat Mar 26 19:48:00 2016 +0100

Add test for YAML based reporter
---
 ooni/tests/test_reporter.py | 14 ++
 1 file changed, 14 insertions(+)

diff --git a/ooni/tests/test_reporter.py b/ooni/tests/test_reporter.py
index 94192ee..9774449 100644
--- a/ooni/tests/test_reporter.py
+++ b/ooni/tests/test_reporter.py
@@ -36,6 +36,13 @@ oonib_new_report_message = {
 'supported_formats': ["yaml", "json"]
 }
 
+# This is used for testing legacy collectors
+oonib_new_report_yaml_message = {
+'report_id': "20140129T202038Z_AS0_" + "A" * 50,
+'backend_version': "1.0"
+}
+
+
 oonib_generic_error_message = {
 'error': 'generic-error'
 }
@@ -103,6 +110,13 @@ class TestOONIBReporter(unittest.TestCase):
 yield self.oonib_reporter.writeReportEntry(req)
 assert self.oonib_reporter.agent.request.called
 
+@defer.inlineCallbacks
+def test_write_report_entry_in_yaml(self):
+self.mock_response = oonib_new_report_yaml_message
+yield self.oonib_reporter.createReport()
+req = {'content': 'something'}
+yield self.oonib_reporter.writeReportEntry(req)
+assert self.oonib_reporter.agent.request.called
 
 class TestOONIBReportLog(unittest.TestCase):
 



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] Fix serialisation of SOA queries and cast set to list

2016-04-29 Thread art

commit 07749637dd9d4d8b4c5f9ff9137a1dd9d2cafbd8
Author: Arturo FilastÃ² 
Date:   Thu Mar 24 13:37:41 2016 +0100

Fix serialisation of SOA queries and cast set to list
---
 ooni/nettests/manipulation/captiveportal.py | 2 +-
 ooni/templates/dnst.py  | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/ooni/nettests/manipulation/captiveportal.py 
b/ooni/nettests/manipulation/captiveportal.py
index f700973..b1cbba7 100644
--- a/ooni/nettests/manipulation/captiveportal.py
+++ b/ooni/nettests/manipulation/captiveportal.py
@@ -426,7 +426,7 @@ class CaptivePortal(httpt.HTTPTest, dnst.DNSTest):
 gmatch, google_dns_result = yield self.compare_random_hostnames(3, 10)
 ret = {
 'result': gmatch,
-'addresses': google_dns_result
+'addresses': list(google_dns_result)
 }
 
 if gmatch:
diff --git a/ooni/templates/dnst.py b/ooni/templates/dnst.py
index 9ee7508..b96fdec 100644
--- a/ooni/templates/dnst.py
+++ b/ooni/templates/dnst.py
@@ -72,8 +72,8 @@ def representAnswer(answer):
 }
 if answer_type is 'SOA':
 represented_answer['ttl'] = answer.payload.ttl
-represented_answer['hostname'] = answer.payload.mname
-represented_answer['responsible_name'] = answer.payload.rname
+represented_answer['hostname'] = answer.payload.mname.name
+represented_answer['responsible_name'] = answer.payload.rname.name
 represented_answer['serial_number'] = answer.payload.serial
 represented_answer['refresh_interval'] = answer.payload.refresh
 represented_answer['retry_interval'] = answer.payload.retry



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] Re-create the command inside the test method.

2016-04-29 Thread art

commit d368c895b9fda39d3422ea78e567876b1b1294dc
Author: Arturo FilastÃ² 
Date:   Wed Mar 16 14:41:45 2016 +0100

Re-create the command inside the test method.

If it's not done inside the test method when it fails and is re-scheduled
it will not re-run the setUp and hence it will have already been deleted.
---
 ooni/nettests/third_party/psiphon.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/ooni/nettests/third_party/psiphon.py 
b/ooni/nettests/third_party/psiphon.py
index 7c1462b..575002b 100644
--- a/ooni/nettests/third_party/psiphon.py
+++ b/ooni/nettests/third_party/psiphon.py
@@ -58,6 +58,7 @@ class PsiphonTest(httpt.HTTPTest,  process.ProcessTest):
 getenv('HOME'), 
'psiphon-circumvention-system/pyclient/pyclient')
 log.debug('psiphon path: %s' % self.psiphonpath)
 
+def createCommand(self):
 # psi_client.py can not be run directly because the paths in the
 # code are relative, so it'll fail to execute from this test
 x = """
@@ -68,7 +69,7 @@ connect(False)
 f.write(x)
 f.close()
 self.command = [sys.executable, f.name]
-log.debug('command: %s' % ''.join(self.command))
+log.debug('command: %s' % ' '.join(self.command))
 
 def handleRead(self, stdout, stderr):
 if 'Press Ctrl-C to terminate.' in self.processDirector.stdout:
@@ -81,6 +82,7 @@ connect(False)
 
 def test_psiphon(self):
 log.debug('PsiphonTest.test_psiphon')
+self.createCommand()
 
 self.report['bootstrapped_success'] = None
 self.report['request_success'] = None



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] Implement measurement_start_time and test_start_time in client

2016-04-29 Thread art

commit 58db73a6d8c7d5296262cfa8ea2ce6e987ef724e
Author: Arturo FilastÃ² 
Date:   Thu Mar 17 15:49:08 2016 +0100

Implement measurement_start_time and test_start_time in client
---
 ooni/nettest.py | 11 ++-
 ooni/otime.py   |  3 +++
 ooni/reporter.py|  4 +++-
 ooni/tasks.py   |  5 ++---
 ooni/tests/test_director.py |  6 ++
 ooni/tests/test_reporter.py |  2 +-
 ooni/tests/test_utils.py| 10 +-
 ooni/utils/__init__.py  |  6 +++---
 8 files changed, 25 insertions(+), 22 deletions(-)

diff --git a/ooni/nettest.py b/ooni/nettest.py
index 9bdc6d9..9c5e171 100644
--- a/ooni/nettest.py
+++ b/ooni/nettest.py
@@ -197,8 +197,7 @@ class NetTestLoader(object):
 
 return input_files
 
-@property
-def testDetails(self):
+def setTestDetails(self):
 from ooni import __version__ as software_version
 
 input_file_hashes = []
@@ -206,8 +205,8 @@ class NetTestLoader(object):
 input_file_hashes.append(input_file['hash'])
 
 options = sanitize_options(self.options)
-test_details = {
-'start_time': otime.epochToUTC(time.time()),
+self.testDetails = {
+'test_start_time': otime.UTCTimestampNow(),
 'probe_asn': config.probe_ip.geodata['asn'],
 'probe_cc': config.probe_ip.geodata['countrycode'],
 'probe_ip': config.probe_ip.geodata['ip'],
@@ -223,7 +222,6 @@ class NetTestLoader(object):
 'annotations': self.annotations,
 'data_format_version': '0.2.0'
 }
-return test_details
 
 def _parseNetTestOptions(self, klass):
 """
@@ -351,6 +349,9 @@ class NetTestLoader(object):
 
 if options:
 klass.localOptions = options
+# XXX this class all needs to be refactored and this is kind of a
+# hack.
+self.setTestDetails()
 
 test_instance = klass()
 if test_instance.requiresRoot and not hasRawSocketPermission():
diff --git a/ooni/otime.py b/ooni/otime.py
index 1406fbf..37ada02 100644
--- a/ooni/otime.py
+++ b/ooni/otime.py
@@ -85,3 +85,6 @@ def epochToTimestamp(seconds):
 
 def epochToUTC(seconds):
 return float(datetime.utcfromtimestamp(seconds).strftime("%s"))
+
+def UTCTimestampNow():
+return datetime.utcnow().strftime("%Y-%m-%d %H:%M:%S")
diff --git a/ooni/reporter.py b/ooni/reporter.py
index d9d147c..d772ae9 100644
--- a/ooni/reporter.py
+++ b/ooni/reporter.py
@@ -267,6 +267,7 @@ class OONIBReporter(OReporter):
 'input': entry.testInstance.report.pop('input', None),
 'id': str(uuid.uuid4()),
 'test_start_time': 
entry.testInstance.report.pop('test_start_time', None),
+'measurement_start_time': 
entry.testInstance.report.pop('measurement_start_time', None),
 'test_runtime': 
entry.testInstance.report.pop('test_runtime', None),
 'test_keys': entry.testInstance.report
 }
@@ -275,6 +276,7 @@ class OONIBReporter(OReporter):
 'input': entry.pop('input', None),
 'id': str(uuid.uuid4()),
 'test_start_time': entry.pop('test_start_time', None),
+'measurement_start_time': 
entry.testInstance.report.pop('measurement_start_time', None),
 'test_runtime': entry.pop('test_runtime', None),
 'test_keys': entry
 }
@@ -352,7 +354,7 @@ class OONIBReporter(OReporter):
 'probe_cc': self.testDetails['probe_cc'],
 'test_name': self.testDetails['test_name'],
 'test_version': self.testDetails['test_version'],
-'start_time': self.testDetails['start_time'],
+'test_start_time': self.testDetails['test_start_time'],
 'input_hashes': self.testDetails['input_hashes'],
 'data_format_version': self.testDetails['data_format_version'],
 'format': 'json'
diff --git a/ooni/tasks.py b/ooni/tasks.py
index 3f331a2..b5a15e0 100644
--- a/ooni/tasks.py
+++ b/ooni/tasks.py
@@ -116,9 +116,8 @@ class Measurement(TaskWithTimeout):
 
 if 'input' not in self.testInstance.report.keys():
 self.testInstance.report['input'] = test_input
-if 'test_start_time' not in self.testInstance.report.keys():
-start_time = otime.epochToUTC(self.testInstance._start_time)
-self.testInstance.report['test_start_time'] = start_time
+if 'measurement_start_time' not in self.testInstance.report.keys():
+self.testInstance.report['measurement_start_time'] = 
otime.UTCTimestampNow()
 
 self.testInstance.setUp()
 
diff --git a/ooni/tests/test_director.py b/ooni/tests/test_director.py
index 38e8481..61d504c 100644
--- a/ooni/tests/test_director.py
+++ b/ooni/tests/test_director.py
@@ -1,5 +1,3 @@
-import time

[tor-commits] [ooni-probe/master] Add support for setting the url and expected-body in tests

2016-04-29 Thread art

commit c17f564789b85bc9baa9dc1c64e77d8ff070aebd
Author: Arturo FilastÃ² 
Date:   Wed Mar 16 15:06:59 2016 +0100

Add support for setting the url and expected-body in tests

* When these are changed from their default values a flag is set to false
---
 docs/source/tests/lantern.rst|  6 ++
 docs/source/tests/psiphon.rst|  7 +++
 ooni/nettests/third_party/lantern.py | 30 ++
 ooni/nettests/third_party/psiphon.py | 29 +++--
 ooni/utils/net.py|  3 +++
 5 files changed, 61 insertions(+), 14 deletions(-)

diff --git a/docs/source/tests/lantern.rst b/docs/source/tests/lantern.rst
index 9f3af2f..9f3ee09 100644
--- a/docs/source/tests/lantern.rst
+++ b/docs/source/tests/lantern.rst
@@ -31,6 +31,11 @@ How to run the test
 
 `ooniprobe nettests/third_party/lantern.py`
 
+For advanced usages you may also configure a different URL and expected body
+for the response with the `--url` and `--expected-body` command line options.
+
+`ooniprobe nettests/third_party/lantern.py --url 
http://www.github.com/humans.txt --expected-body '/* TEAM */'`
+
 Sample report
 =
 
@@ -55,6 +60,7 @@ test_version: 0.0.1
 ---
 body: "Google is built by a large team of engineers, designers, researchers, 
robots, and others in many different sites across the globe. It is updated 
continuously, and built with more tools and technologies than we can shake a 
stick at. If you'd like to help us out, see google.com/careers."
 bootstrapped: true
+default_configuration: true
 input: null
 lantern --headless: {exit_reason: process_done, stderr: '', stdout: ''}
 ```
diff --git a/docs/source/tests/psiphon.rst b/docs/source/tests/psiphon.rst
index 3934bcb..3499501 100644
--- a/docs/source/tests/psiphon.rst
+++ b/docs/source/tests/psiphon.rst
@@ -35,6 +35,12 @@ To test Psiphon when it is installed in a different path 
other than the user hom
 
 `ooniprobe third_party/psiphon -p `
 
+For advanced usages you may also configure a different URL and expected body
+for the response with the `--url` and `--expected-body` command line options.
+
+`ooniprobe third_party/psiphon --url http://www.github.com/humans.txt 
--expected-body '/* TEAM */'`
+
+
 How to install Psiphon
 ===
 
@@ -80,6 +86,7 @@ Sample report
 agent: agent
 input: null
 psiphon_installed: true
+default_configuration: true
 requests:
 - request:
 body: null
diff --git a/ooni/nettests/third_party/lantern.py 
b/ooni/nettests/third_party/lantern.py
index 187e210..da19566 100644
--- a/ooni/nettests/third_party/lantern.py
+++ b/ooni/nettests/third_party/lantern.py
@@ -1,11 +1,22 @@
 import os
+import distutils.spawn
+
 from twisted.internet import defer, reactor
 from twisted.internet.endpoints import TCP4ClientEndpoint
 from twisted.web.client import ProxyAgent, readBody
+from twisted.python import usage
+
 from ooni.templates.process import ProcessTest, ProcessDirector
-from ooni.utils import log
+from ooni.utils import log, net
 from ooni.errors import handleAllFailures
-import distutils.spawn
+
+class UsageOptions(usage.Options):
+optParameters = [
+['url', 'u', net.GOOGLE_HUMANS[0],
+'Specify the URL to fetch over lantern (default: 
http://www.google.com/humans.txt).'],
+['expected-body', 'e', net.GOOGLE_HUMANS[1],
+'Specify the beginning of the expected body in the response 
(default: ' + net.GOOGLE_HUMANS[1] + ')']
+]
 
 class LanternTest(ProcessTest):
 """
@@ -26,10 +37,21 @@ class LanternTest(ProcessTest):
 timeout = 120
 
 def setUp(self):
+self.report['body'] = None
+self.report['failure'] = None
+self.report['success'] = None
+self.report['default_configuration'] = True
+
 self.command = [distutils.spawn.find_executable("lantern"), 
"--headless"]
 self.bootstrapped = defer.Deferred()
 self.exited = False
-self.url = 'http://www.google.com/humans.txt'
+
+self.url = self.localOptions['url']
+if self.url != net.GOOGLE_HUMANS[0]:
+self.report['default_configuration'] = False
+
+if self.localOptions['expected-body'] != net.GOOGLE_HUMANS[1]:
+self.report['default_configuration'] = False
 
 def stop(self, reason=None):
 if not self.exited:
@@ -49,7 +71,7 @@ class LanternTest(ProcessTest):
 def test_lantern_circumvent(self):
 def addResultToReport(result):
 self.report['body'] = result
-if result.startswith('Google is built by a large'):
+if result.startswith(self.localOptions['expected-body']):
 log.msg("Got the HTTP response body I expected!")
 self.report['success'] = True
 else:
diff --git a/ooni/nettests/third_party/psiphon.py 
b/ooni/nettests/third_party/psiphon.py
index 575002b..2b18b18 100644
--- a/ooni/nettests/third_party/psiphon.py

[tor-commits] [ooni-probe/master] Merge branch 'master' into feature/json_reports

2016-04-29 Thread art

commit dd58c40cb351b4d9d1b33fbef08919a8c225fad7
Merge: 1838533 a0fc5e6
Author: Simone Basso 
Date:   Mon Apr 4 17:58:05 2016 +0200

Merge branch 'master' into feature/json_reports

 README.rst  | 41 ++---
 docs/source/tests/psiphon.rst   |  4 ++--
 ooni/nettest.py |  3 +++
 ooni/nettests/manipulation/http_host.py |  2 --
 ooni/tasks.py   |  1 -
 ooni/templates/httpt.py |  4 +---
 6 files changed, 13 insertions(+), 42 deletions(-)

diff --cc ooni/templates/httpt.py
index d2cc6e1,2f1136c..b9ca2e4
--- a/ooni/templates/httpt.py
+++ b/ooni/templates/httpt.py
@@@ -152,20 -134,14 +152,19 @@@ class HTTPTest(NetTestCase)
  }
  }
  if response:
 -request_response['response'] = {
 -'headers': list(response.headers.getAllRawHeaders()),
 -'body': response_body if self.localOptions.get('withoutbody', 
0) == 0 else '',
 +if self.localOptions.get('withoutbody', 0) is 0:
 +response_body = _representBody(response_body)
 +else:
 +response_body = ''
 +session['response'] = {
 +'headers': _representHeaders(response.headers),
 +'body': response_body,
  'code': response.code
 -}
 +}
 +session['failure'] = None
  if failure_string:
 -request_response['failure'] = failure_string
 -self.report['requests'].append(request_response)
 +session['failure'] = failure_string
- 
 +self.report['requests'].append(session)
  
  def _processResponseBody(self, response_body, request, response, 
body_processor):
  log.debug("Processing response body")



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] Implement unittest to test for bug spotted by @bassosimone

2016-04-29 Thread art

commit 48435739f1bdd6631065a8ef41e5c51605955b2a
Author: Arturo FilastÃ² 
Date:   Mon Mar 21 16:54:18 2016 +0100

Implement unittest to test for bug spotted by @bassosimone

* Fix bug in serializing of JSON
---
 ooni/reporter.py| 2 +-
 ooni/tests/test_reporter.py | 3 ++-
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/ooni/reporter.py b/ooni/reporter.py
index d772ae9..2ee8468 100644
--- a/ooni/reporter.py
+++ b/ooni/reporter.py
@@ -276,7 +276,7 @@ class OONIBReporter(OReporter):
 'input': entry.pop('input', None),
 'id': str(uuid.uuid4()),
 'test_start_time': entry.pop('test_start_time', None),
-'measurement_start_time': 
entry.testInstance.report.pop('measurement_start_time', None),
+'measurement_start_time': 
entry.pop('measurement_start_time', None),
 'test_runtime': entry.pop('test_runtime', None),
 'test_keys': entry
 }
diff --git a/ooni/tests/test_reporter.py b/ooni/tests/test_reporter.py
index 5adb4af..94192ee 100644
--- a/ooni/tests/test_reporter.py
+++ b/ooni/tests/test_reporter.py
@@ -32,7 +32,8 @@ test_details = {
 
 oonib_new_report_message = {
 'report_id': "20140129T202038Z_AS0_" + "A" * 50,
-'backend_version': "1.0"
+'backend_version': "1.0",
+'supported_formats': ["yaml", "json"]
 }
 
 oonib_generic_error_message = {



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] Set the input to be the specified URL in the http_requests test

2016-04-29 Thread art

commit 76298b89ba341380ec4d1ab8e640eb7a4a9ca531
Author: Arturo FilastÃ² 
Date:   Wed Mar 30 14:07:43 2016 +0200

Set the input to be the specified URL in the http_requests test
---
 ooni/nettests/blocking/http_requests.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/ooni/nettests/blocking/http_requests.py 
b/ooni/nettests/blocking/http_requests.py
index e2c9671..abc7efc 100644
--- a/ooni/nettests/blocking/http_requests.py
+++ b/ooni/nettests/blocking/http_requests.py
@@ -68,6 +68,7 @@ class HTTPRequestsTest(httpt.HTTPTest):
 self.url = self.localOptions['url']
 else:
 raise Exception("No input specified")
+self.report['input'] = self.url
 
 self.factor = self.localOptions['factor']
 self.report['control_failure'] = None



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] Add support for stripping potential inclusion of probe IPs in reports

2016-04-29 Thread art

commit ee2fa7b4093117bd618af03382809ade4e9832a0
Author: Arturo FilastÃ² 
Date:   Fri Mar 11 17:57:10 2016 +0100

Add support for stripping potential inclusion of probe IPs in reports
---
 ooni/templates/httpt.py   | 3 +++
 ooni/templates/process.py | 9 -
 2 files changed, 11 insertions(+), 1 deletion(-)

diff --git a/ooni/templates/httpt.py b/ooni/templates/httpt.py
index 3702e86..d0822f2 100644
--- a/ooni/templates/httpt.py
+++ b/ooni/templates/httpt.py
@@ -134,6 +134,9 @@ class HTTPTest(NetTestCase):
 }
 }
 if response:
+# Attempt to redact the IP address of the probe from the responses
+if config.privacy.includeip is False:
+response_body = response_body.replace(config.probe_ip.address, 
"[REDACTED]")
 request_response['response'] = {
 'headers': list(response.headers.getAllRawHeaders()),
 'body': response_body if self.localOptions.get('withoutbody', 
0) == 0 else '',
diff --git a/ooni/templates/process.py b/ooni/templates/process.py
index 82457dc..42a4256 100644
--- a/ooni/templates/process.py
+++ b/ooni/templates/process.py
@@ -1,5 +1,6 @@
 from twisted.internet import protocol, defer, reactor
 
+from ooni.settings import config
 from ooni.nettest import NetTestCase
 from ooni.utils import log
 
@@ -66,7 +67,7 @@ class ProcessDirector(protocol.ProcessProtocol):
 self.stderr += data
 if self.shouldClose():
 self.close("condition_met")
-self.handlRead(None,  data)
+self.handleRead(None,  data)
 
 
 def inConnectionLost(self):
@@ -105,6 +106,12 @@ class ProcessTest(NetTestCase):
 log.debug("Finished %s: %s" % (command, result))
 if not isinstance(self.report.get('commands'), list):
 self.report['commands'] = []
+
+# Attempt to redact the IP address of the probe from the standard 
output
+if config.privacy.includeip is False:
+result['stdout'] = 
result['stdout'].replace(config.probe_ip.address, "[REDACTED]")
+result['stderr'] = 
result['stderr'].replace(config.probe_ip.address, "[REDACTED]")
+
 self.report['commands'].append({
 'command_name': ' '.join(command),
 'command_stdout': result['stdout'],



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] Improvements to the output of the http_invalid_request_line test

2016-04-29 Thread art

commit da4ad2f16e3bd7d87f8d66714452a2746518147c
Author: Arturo FilastÃ² 
Date:   Thu Feb 4 16:26:44 2016 +0100

Improvements to the output of the http_invalid_request_line test
---
 ooni/nettests/manipulation/http_invalid_request_line.py | 3 +++
 ooni/templates/httpt.py | 4 ++--
 ooni/templates/tcpt.py  | 1 +
 3 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/ooni/nettests/manipulation/http_invalid_request_line.py 
b/ooni/nettests/manipulation/http_invalid_request_line.py
index b6ce2da..f81d953 100644
--- a/ooni/nettests/manipulation/http_invalid_request_line.py
+++ b/ooni/nettests/manipulation/http_invalid_request_line.py
@@ -45,8 +45,11 @@ class HTTPInvalidRequestLine(tcpt.TCPTest):
 def check_for_manipulation(self, response, payload):
 log.debug("Checking if %s == %s" % (response, payload))
 if response != payload:
+log.msg("Detected manipulation!")
+log.msg(response)
 self.report['tampering'] = True
 else:
+log.msg("No manipulation detected.")
 self.report['tampering'] = self.report['tampering'] | False
 
 def test_random_invalid_method(self):
diff --git a/ooni/templates/httpt.py b/ooni/templates/httpt.py
index 15630d5..a49c39b 100644
--- a/ooni/templates/httpt.py
+++ b/ooni/templates/httpt.py
@@ -347,9 +347,9 @@ class HTTPTest(NetTestCase):
 
 def errback(failure, request):
 if request['tor']['is_tor']:
-log.err("Error performing torified request: %s" % 
request['url'])
+log.err("Error performing torified HTTP request: %s" % 
request['url'])
 else:
-log.err("Error performing request: %s" % request['url'])
+log.err("Error performing HTTP request: %s" % request['url'])
 failure_string = handleAllFailures(failure)
 self.addToReport(request, failure_string=failure_string)
 return failure
diff --git a/ooni/templates/tcpt.py b/ooni/templates/tcpt.py
index bdedebc..5e1d21a 100644
--- a/ooni/templates/tcpt.py
+++ b/ooni/templates/tcpt.py
@@ -44,6 +44,7 @@ class TCPSender(protocol.Protocol):
 self.transport.write(payload)
 
 class TCPSenderFactory(protocol.Factory):
+noisy = False
 def buildProtocol(self, addr):
 return TCPSender()
 



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] Remove obsolete install method on Debian/Ubuntu

2016-04-29 Thread art

commit f73a26aa1c2d62a2e2ebe0c335ac29da4d4c4702
Author: anadahz 
Date:   Fri Mar 11 00:12:38 2016 -0300

Remove obsolete install method on Debian/Ubuntu

Currently the ooniprobe packages offered in Debian and Ubuntu are old,
unavailable in some releases or very old version. Removing obsolete
install instructions until the ooniprobe packages in the repositories
are available and up-to-date
---
 README.rst | 32 
 1 file changed, 32 deletions(-)

diff --git a/README.rst b/README.rst
index 59c7afa..6234b62 100644
--- a/README.rst
+++ b/README.rst
@@ -47,15 +47,6 @@ probe is a liability for you, please be aware of this risk.
 OONI in 5 minutes
 =
 
-On debian testing or unstable::
-
-sudo apt-get install ooniprobe
-
-If you are running debian stable you can get it from backports via::
-
-sudo sh -c 'echo "deb http://http.debian.net/debian jessie-backports main" 
>> /etc/apt/sources.list'
-sudo apt-get update && sudo apt-get install ooniprobe
-
 On unix systems::
 
 sudo pip install ooniprobe
@@ -94,29 +85,6 @@ Run this command to automatically update your crontab::
 Installation
 
 
-Debian based systems
-
-
-If you are running Debian testing or Debian unstable you can install ooniprobe
-simply with::
-
-apt-get install ooniprobe
-
-If you are running Debian stable you can get it from backports via::
-
-sudo sh -c 'echo "deb http://http.debian.net/debian wheezy-backports main" 
>> /etc/apt/sources.list'
-sudo apt-get update && sudo apt-get install ooniprobe
-
-If you are running Ubuntu 14.04 LTS, 14.10 or 15.04 you can install it from 
the PPA
-(https://launchpad.net/~irl/+archive/ubuntu/ooni/)::
-
-sudo add-apt-repository ppa:irl/ooni
-sudo apt-get update && sudo apt-get install ooniprobe
-
-You will be warned that the packages are unauthenticated. This is due to the
-PPA not being signed and is normal behaviour. If you would prefer to verify the
-integrity of the package, use our private Debian repository below.
-
 Mac OS X
 
 



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] Added a simple tutorial on how to write test helpers.

2016-04-29 Thread art

commit 2b1bfac9b7a9d52af9f36f25d19430b8935fac4c
Author: seamus tuohy 
Date:   Mon Jan 18 14:24:59 2016 -0500

Added a simple tutorial on how to write test helpers.
---
 docs/source/writing_test_helpers.rst | 137 +++
 1 file changed, 137 insertions(+)

diff --git a/docs/source/writing_test_helpers.rst 
b/docs/source/writing_test_helpers.rst
new file mode 100644
index 000..2206115
--- /dev/null
+++ b/docs/source/writing_test_helpers.rst
@@ -0,0 +1,137 @@
+Writing Test Helpers
+
+
+OONI test helpers are...
+
+Writing a Censorship Directionality Test Helper
+--
+
+Our goal is to write an OONI test helper that helps an OONI-probe client 
determine the directionality of keyword censorship it has detected. To do this 
our helper will receive "encoded" data from an OONI-probe client, decode that 
data into a text string, and send the OONI-probe client a confirmation packet 
and a second packet containing the decoded string.
+
+The OONI-backend code-base has many concise examples of test-helpers that you 
can build off to create your own. For this tutorial I used the `TCP echo 
test-helper 
`
 as my guide.
+
+Following this tutorial requires basic knowledge of event-driven programming 
(specifically 'Twisted'). You will be more than ready to build and implement a 
test-helper after reading though one or two `tutorials online. 
`
+
+Creating the test helper
+--
+
+OONI-backend keeps all the test-helpers in the `oonib/testhelpers directory 
` 
Each individual test helper is a twisted service. Most of the current 
test-helpers consists of a twisted Factory and a twisted Protocol defined in 
the test helpers directory and a `stock Twisted Server 
`
 that is defined in the backend code. We will follow this model in the tutorial.
+
+Because of how simple this example test-helper is the job of our test-helper 
factory is merely to deploy a single instance of our protocol each time it's 
buildProtocol method is called. Because we have our factory inheret from the 
base `Factory object 
`
 we merely have to define its ``protocol`` property to point to our protocol.::
+
+class TCPDirectionalityTestHelper(Factory):
+"""
+A test helper for checking for directionality of censorship
+"""
+protocol = TCPDirectionalityTestProtocol
+
+
+The protocol for this helper needs to do two things. First, upon receiving 
encoded data it needs to send the OONI-probe client back confirmation that the 
data has been received. Second, it needs to send the decoded data back to the 
OONI-probe client. Because we are extending the `Protocol object 
`
 we can rewrite its ``dataReceived`` method which is called and passed data 
whenever it is received.::
+
+
+class TCPDirectionalityTestProtocol(Protocol):
+"""Takes encoded packet data, decodes it, and then sends it back.
+
+This protocol sends two packets in response to an encoded packet.
+It first sends a confirmation packet, and then follows with a packet
+containing the decoded data requested from the test.
+"""
+def dataReceived(self, data):
+# send back receipt of the packet
+self.transport.write(data)
+# send back the decoded term to test against.
+original_string = 
data.decode("rot13").decode("unicode-escape").encode("utf-8")
+self.transport.write(original_string)
+
+
+In order to make this test-helper slightly more flexible we will be allowing 
the backend to determine the encoding within their config file. To this end we 
will have to retreive the encoding from the config file.::
+
+
+def dataReceived(self, data):
+# send back receipt of the packet
+self.transport.write(data)
+
+# Get the encoding from the config or fallback to rot13
+if config.helpers['tcp-directionality'].encoding:
+tcp_dir_encoding = 
config.helpers['tcp-directionality'].encoding
+else:
+tcp_dir_encoding = "rot13"
+
+# send back the decoded term to test against.
+original_string = 
data.decode("rot13").decode("unicode-escape").encode("utf-8")
+self.transport.write(original_string)
+
+
+With this added we have completed the base of simple test-helper and now just 
have to integrate it into the rest of the backend.
+
+

[tor-commits] [ooni-probe/master] Merge pull request #465 from TheTorProject/fix/venv-doc

2016-04-29 Thread art

commit e2baf90930a5ce290021e29f8a35935969f24e1b
Merge: 9bc3006 d76970b
Author: Arturo FilastÃ² 
Date:   Tue Mar 15 15:13:41 2016 +0100

Merge pull request #465 from TheTorProject/fix/venv-doc

Fix install doc on virtualenv setup

 README.rst | 9 +++--
 1 file changed, 7 insertions(+), 2 deletions(-)



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] Update the tests to be consistent with the new data formats

2016-04-29 Thread art

commit 9bd24ee947194b0f5713b46e845106210e7163c9
Author: Arturo FilastÃ² 
Date:   Thu Jan 28 13:09:36 2016 +0100

Update the tests to be consistent with the new data formats
---
 ooni/nettests/blocking/dns_consistency.py | 29 +++
 ooni/reporter.py  | 50 ++-
 ooni/templates/dnst.py| 83 ++-
 ooni/templates/httpt.py   | 41 +++
 ooni/templates/scapyt.py  | 12 +++--
 ooni/tests/test_trueheaders.py| 12 ++---
 ooni/utils/__init__.py|  7 +++
 ooni/utils/trueheaders.py |  2 +-
 8 files changed, 157 insertions(+), 79 deletions(-)

diff --git a/ooni/nettests/blocking/dns_consistency.py 
b/ooni/nettests/blocking/dns_consistency.py
index 88e5d5e..93a2d65 100644
--- a/ooni/nettests/blocking/dns_consistency.py
+++ b/ooni/nettests/blocking/dns_consistency.py
@@ -39,7 +39,7 @@ class DNSConsistencyTest(dnst.DNSTest):
 name = "DNS Consistency"
 description = "Checks to see if the DNS responses from a "\
   "set of DNS resolvers are consistent."
-version = "0.6"
+version = "0.7.0"
 authors = "Arturo FilastÃ², Isis Lovecruft"
 
 inputFile = ['file', 'f', None,
@@ -110,7 +110,11 @@ class DNSConsistencyTest(dnst.DNSTest):
 log.msg("Doing the test lookups on %s" % self.input)
 hostname = self.input
 
-self.report['tampering'] = {}
+self.report['successful'] = []
+self.report['failures'] = []
+self.report['inconsistent'] = []
+
+self.report['errors'] = {}
 
 try:
 control_answers = yield self.performALookup(hostname,
@@ -121,11 +125,11 @@ class DNSConsistencyTest(dnst.DNSTest):
 "Got no response from control DNS server %s:%d, "
 "perhaps the DNS resolver is down?" %
 self.control_dns_server)
-self.report['tampering'][
+self.report['errors'][
 "%s:%d" %
 self.control_dns_server] = 'no_answer'
 except:
-self.report['tampering'][
+self.report['errors'][
 "%s:%d" %
 self.control_dns_server] = 'error'
 control_answers = None
@@ -139,12 +143,14 @@ class DNSConsistencyTest(dnst.DNSTest):
test_dns_server)
 except Exception:
 log.err("Problem performing the DNS lookup")
-self.report['tampering'][test_resolver] = 'dns_lookup_error'
+self.report['errors'][test_resolver] = 'dns_lookup_error'
+self.report['failures'].append(test_resolver)
 continue
 
 if not experiment_answers:
 log.err("Got no response, perhaps the DNS resolver is down?")
-self.report['tampering'][test_resolver] = 'no_answer'
+self.report['errors'][test_resolver] = 'no_answer'
+self.report['failures'].append(test_resolver)
 continue
 else:
 log.debug(
@@ -165,12 +171,13 @@ class DNSConsistencyTest(dnst.DNSTest):
 
 if not control_answers:
 log.msg("Skipping control resolver comparison")
-self.report['tampering'][test_resolver] = None
+self.report['errors'][test_resolver] = None
 
 elif set(experiment_answers) & set(control_answers):
 lookup_details()
 log.msg("tampering: false")
-self.report['tampering'][test_resolver] = False
+self.report['errors'][test_resolver] = False
+self.report['successful'].append(test_resolver)
 else:
 log.msg("Trying to do reverse lookup")
 experiment_reverse = yield 
self.performPTRLookup(experiment_answers[0],
@@ -182,12 +189,14 @@ class DNSConsistencyTest(dnst.DNSTest):
 log.msg("Further testing has eliminated false positives")
 lookup_details()
 log.msg("tampering: reverse_match")
-self.report['tampering'][test_resolver] = 'reverse_match'
+self.report['errors'][test_resolver] = 'reverse_match'
+self.report['successful'].append(test_resolver)
 else:
 log.msg("Reverse lookups do not match")
 lookup_details()
 log.msg("tampering: true")
-self.report['tampering'][test_resolver] = True
+self.report['errors'][test_resolver] = True
+self.report['inconsistent'].append(test_resolver)
 
 def inputProcessor(self, filename=None):
 """
diff --git a/ooni/reporter.py b/ooni/reporter.py
index db6f7c1..3ea1b10

[tor-commits] [ooni-probe/master] Adapt unittests to changes in the reporting

2016-04-29 Thread art

commit 54529569c99ba670ff1135a3219de0ae19186d40
Author: Arturo FilastÃ² 
Date:   Sat Jan 30 22:24:51 2016 +0100

Adapt unittests to changes in the reporting
---
 ooni/tests/test_reporter.py | 5 -
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/ooni/tests/test_reporter.py b/ooni/tests/test_reporter.py
index 454fbef..f1f7c3a 100644
--- a/ooni/tests/test_reporter.py
+++ b/ooni/tests/test_reporter.py
@@ -25,11 +25,12 @@ test_details = {
 'software_version': '1.0',
 'input_hashes': [],
 'probe_asn': 'AS0',
+'probe_cc': 'ZZ',
 'start_time': time.time()
 }
 
 oonib_new_report_message = {
-'report_id': "2014-01-29T202038Z_AS0_" + "A" * 50,
+'report_id': "20140129T202038Z_AS0_" + "A" * 50,
 'backend_version': "1.0"
 }
 
@@ -94,6 +95,8 @@ class TestOONIBReporter(unittest.TestCase):
 
 @defer.inlineCallbacks
 def test_write_report_entry(self):
+self.mock_response = oonib_new_report_message
+yield self.oonib_reporter.createReport()
 req = {'content': 'something'}
 yield self.oonib_reporter.writeReportEntry(req)
 assert self.oonib_reporter.agent.request.called



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] Handle more DNS exceptions

2016-04-29 Thread art

commit 2b5c77f44c6c25aec9979d931aa72375377c120f
Author: Arturo FilastÃ² 
Date:   Thu Feb 4 16:25:54 2016 +0100

Handle more DNS exceptions
---
 ooni/errors.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/ooni/errors.py b/ooni/errors.py
index 3b327f7..e523de0 100644
--- a/ooni/errors.py
+++ b/ooni/errors.py
@@ -5,6 +5,7 @@ from twisted.web.error import Error
 
 from twisted.internet.error import ConnectionRefusedError, TCPTimedOutError
 from twisted.internet.error import DNSLookupError, ConnectError, ConnectionLost
+from twisted.names.error import DNSNameError, DNSServerError
 from twisted.internet.error import TimeoutError as GenericTimeoutError
 from twisted.internet.error import ProcessDone, ConnectionDone
 
@@ -26,6 +27,8 @@ known_failures = [
 (CancelledError, 'task_timed_out'),
 (gaierror, 'address_family_not_supported_error'),
 (DNSLookupError, 'dns_lookup_error'),
+(DNSNameError, 'dns_name_error'),
+(DNSServerError, 'dns_server_failure'),
 (TCPTimedOutError, 'tcp_timed_out_error'),
 (ResponseNeverReceived, 'response_never_received'),
 (DeferTimeoutError, 'deferred_timeout_error'),



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] Fixes to the DNS test template

2016-04-29 Thread art

commit b13cf26fed638a0de17d87ff1f4910fe98e1a135
Author: Arturo FilastÃ² 
Date:   Mon Feb 1 17:05:10 2016 +0100

Fixes to the DNS test template
---
 ooni/templates/dnst.py | 30 +-
 1 file changed, 17 insertions(+), 13 deletions(-)

diff --git a/ooni/templates/dnst.py b/ooni/templates/dnst.py
index e8ae189..9ee7508 100644
--- a/ooni/templates/dnst.py
+++ b/ooni/templates/dnst.py
@@ -180,16 +180,16 @@ class DNSTest(NetTestCase):
 answers = []
 addrs = []
 for answer in msg:
-if answer.type is dnsType:
-if dnsType is dns.SOA:
-addr = (answer.name.name,answer.payload.serial)
-elif dnsType in [dns.NS,dns.PTR]:
-addr = answer.payload.name.name
-elif dnsType is dns.A:
-addr = answer.payload.dottedQuad()
-else:
-addr = None
-addrs.append(addr)
+addr = None
+if answer.type is dns.SOA:
+addr = (answer.name.name,answer.payload.serial)
+elif answer.type in [dns.NS, dns.PTR, dns.CNAME]:
+addr = answer.payload.name.name
+elif answer.type is dns.A:
+addr = answer.payload.dottedQuad()
+else:
+log.debug("Unidentified answer %s" % answer)
+addrs.append(addr)
 answers.append(representAnswer(answer))
 
 if dns_type == 'SOA':
@@ -225,9 +225,13 @@ class DNSTest(NetTestCase):
 def addToReport(self, query, resolver=None, query_type=None,
 answers=None, failure=None):
 log.debug("Adding %s to report)" % query)
-result = {}
-result['resolver_hostname'] = resolver[0]
-result['resolver_port'] = resolver[1]
+result = {
+'resolver_hostname': None,
+'resolver_port': None
+}
+if resolver is not None and len(resolver) == 2:
+result['resolver_hostname'] = resolver[0]
+result['resolver_port'] = resolver[1]
 result['query_type'] = query_type
 result['hostname'] = str(query[0].name)
 result['failure'] = None



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] Refactor error handling to avoid inconsistencies

2016-04-29 Thread art

commit 29119b673f2880eedd47b0531d58bb8e7425dea8
Author: srvetus 
Date:   Thu Sep 24 18:04:53 2015 +0200

Refactor error handling to avoid inconsistencies

The error handling code in handleAllFailures() and failureToString()
is inconsistent and does not correctly trap all of the Failures for
which there are error messages defined.

The revised code places the Exception and error message in a single
place to avoid inconsistencies arising in future.
---
 ooni/errors.py | 153 ++---
 1 file changed, 38 insertions(+), 115 deletions(-)

diff --git a/ooni/errors.py b/ooni/errors.py
index cb18fe9..5751baf 100644
--- a/ooni/errors.py
+++ b/ooni/errors.py
@@ -20,39 +20,44 @@ from txsocksx.errors import TTLExpired, CommandNotSupported
 
 from socket import gaierror
 
+known_failures = {
+ConnectionRefusedError: 'connection_refused_error',
+ConnectionLost: 'connection_lost_error',
+ConnectError: 'connect_error',
+CancelledError: 'task_timed_out',
+gaierror: 'address_family_not_supported_error',
+DNSLookupError: 'dns_lookup_error',
+TCPTimedOutError: 'tcp_timed_out_error',
+ResponseNeverReceived: 'response_never_received',
+DeferTimeoutError: 'deferred_timeout_error',
+GenericTimeoutError: 'generic_timeout_error',
+SOCKSError: 'socks_error',
+MethodsNotAcceptedError: 'socks_methods_not_supported',
+AddressNotSupported: 'socks_address_not_supported',
+NetworkUnreachable: 'socks_network_unreachable',
+ConnectionError: 'socks_connect_error',
+ConnectionLostEarly: 'socks_connection_lost_early',
+ConnectionNotAllowed: 'socks_connection_not_allowed',
+NoAcceptableMethods: 'socks_no_acceptable_methods',
+ServerFailure: 'socks_server_failure',
+HostUnreachable: 'socks_host_unreachable',
+ConnectionRefused: 'socks_connection_refused',
+TTLExpired: 'socks_ttl_expired',
+CommandNotSupported: 'socks_command_not_supported',
+ProcessDone: 'process_done',
+ConnectionDone: 'connection_done',
+}
 
 def handleAllFailures(failure):
 """
-Here we make sure to trap all the failures that are supported by the
-failureToString function and we return the the string that represents the
-failure.
+Trap all the known Failures and we return a string that
+represents the failure. Any unknown Failures will be reraised and
+returned by failure.trap().
 """
-failure.trap(
-ConnectionRefusedError,
-gaierror,
-DNSLookupError,
-TCPTimedOutError,
-ResponseNeverReceived,
-DeferTimeoutError,
-GenericTimeoutError,
-SOCKSError,
-MethodsNotAcceptedError,
-AddressNotSupported,
-ConnectionError,
-NetworkUnreachable,
-ConnectionLostEarly,
-ConnectionNotAllowed,
-NoAcceptableMethods,
-ServerFailure,
-HostUnreachable,
-ConnectionRefused,
-TTLExpired,
-CommandNotSupported,
-ConnectError,
-ConnectionLost,
-CancelledError, 
-ConnectionDone)
 
+# TODO: Should this function actually handle ALL failures and
+#   not just the failures configured in this list.
+failure.trap(*known_failures.keys())
 return failureToString(failure)
 
 
@@ -69,95 +74,13 @@ def failureToString(failure):
 
 A string representing the HTTP response error message.
 """
-string = None
-if isinstance(failure.value, ConnectionRefusedError):
-# log.err("Connection refused.")
-string = 'connection_refused_error'
 
-elif isinstance(failure.value, ConnectionLost):
-# log.err("Connection lost.")
-string = 'connection_lost_error'
-
-elif isinstance(failure.value, ConnectError):
-# log.err("Connect error.")
-string = 'connect_error'
-
-elif isinstance(failure.value, gaierror):
-# log.err("Address family for hostname not supported")
-string = 'address_family_not_supported_error'
-
-elif isinstance(failure.value, DNSLookupError):
-# log.err("DNS lookup failure")
-string = 'dns_lookup_error'
-
-elif isinstance(failure.value, TCPTimedOutError):
-# log.err("TCP Timed Out Error")
-string = 'tcp_timed_out_error'
-
-elif isinstance(failure.value, ResponseNeverReceived):
-# log.err("Response Never Received")
-string = 'response_never_received'
-
-elif isinstance(failure.value, DeferTimeoutError):
-# log.err("Deferred Timeout Error")
-string = 'deferred_timeout_error'
-
-elif isinstance(failure.value, GenericTimeoutError):
-# log.err("Time Out Error")
-string = 'generic_timeout_error'
-
-elif isinstance(failure.value, ServerFailure):
-# log.err("SOCKS error: ServerFailure")
-string = 'socks_server_failure'
-
-elif isinstance(failure.value,

[tor-commits] [ooni-probe/master] Package virtualenvwrapper not needed

2016-04-29 Thread art

commit d76970b926c5ae7a2607e596c715cb56f6621ea3
Author: anadahz 
Date:   Tue Mar 15 10:33:37 2016 -0300

Package virtualenvwrapper not needed

Removing virtualenvwrapper packages requirement.
As spotted by @hellais
---
 README.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/README.rst b/README.rst
index 58c8376..384f71b 100644
--- a/README.rst
+++ b/README.rst
@@ -346,7 +346,7 @@ Setting up development environment
 On Debian based systems a development environment can be setup as follows: 
(prerequisites include build essentials, python-dev, and tor; for tor see 
https://www.torproject.org/docs/debian.html.en)::
 
 
-sudo apt-get install python-pip python-virtualenv virtualenv 
virtualenvwrapper
+sudo apt-get install python-pip python-virtualenv virtualenv
 sudo apt-get install libgeoip-dev libffi-dev libdumbnet-dev libssl-dev 
libpcap-dev
 git clone https://github.com/TheTorProject/ooni-probe
 cd ooni-probe



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] Fixes to unittests

2016-04-29 Thread art

commit ef5e3cbcd1c3a64288f404c3182d15311cb78ddf
Author: Arturo FilastÃ² 
Date:   Fri Feb 5 20:03:52 2016 +0100

Fixes to unittests
---
 ooni/tests/test_oonicli.py  | 9 +
 ooni/tests/test_reporter.py | 3 ++-
 2 files changed, 3 insertions(+), 9 deletions(-)

diff --git a/ooni/tests/test_oonicli.py b/ooni/tests/test_oonicli.py
index 564f9e6..c57205c 100644
--- a/ooni/tests/test_oonicli.py
+++ b/ooni/tests/test_oonicli.py
@@ -158,14 +158,7 @@ class TestRunDirector(ConfigTestCase):
 assert 'agent' in entry
 assert 'requests' in entry
 assert 'socksproxy' in entry
-assert 'tampering' in entry
-assert 'header_field_name' in entry['tampering']
-assert 'header_field_number' in entry['tampering']
-assert 'header_field_value' in entry['tampering']
-assert 'header_name_capitalization' in entry['tampering']
-assert 'header_name_diff' in entry['tampering']
-assert 'request_line_capitalization' in entry['tampering']
-assert 'total' in entry['tampering']
+assert 'errors' in entry
 
 yield self.run_helper('manipulation/http_header_field_manipulation',
   ['-b', 'http://4.15.35.157:80'],
diff --git a/ooni/tests/test_reporter.py b/ooni/tests/test_reporter.py
index f1f7c3a..4b99584 100644
--- a/ooni/tests/test_reporter.py
+++ b/ooni/tests/test_reporter.py
@@ -26,7 +26,8 @@ test_details = {
 'input_hashes': [],
 'probe_asn': 'AS0',
 'probe_cc': 'ZZ',
-'start_time': time.time()
+'start_time': time.time(),
+'data_format_version': '0.2.0'
 }
 
 oonib_new_report_message = {



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] fixes and enhancements to psipshon install script

2016-04-29 Thread art

commit 955de5407b0ae3c4766ad9b48b99418ef2280a78
Author: juga0 
Date:   Tue Nov 10 17:35:09 2015 +

fixes and enhancements to psipshon install script

* don't clone repository if the directory already exists
* eliminate duplicated lines copying the server file
* don't compile ssh client if the file already exists
* fix syntax of line checking if a virtualenv is activated
* fix psiphon directory path
* remove x flag
* remove commented lines for running psiphon
* replace source command by dot
* add variables for psiphon repo url and ooni virtualenv path
* fix identation
---
 scripts/psiphon_install.sh | 89 +++---
 1 file changed, 44 insertions(+), 45 deletions(-)

diff --git a/scripts/psiphon_install.sh b/scripts/psiphon_install.sh
index 208aa83..e5b2a78 100755
--- a/scripts/psiphon_install.sh
+++ b/scripts/psiphon_install.sh
@@ -3,13 +3,15 @@
 #FIXME remove x
 set -ex
 
-PSIPHON_PATH=$HOME/test
+PSIPHON_PATH=$HOME
 PSIPHON_PYCLIENT_PATH=$PSIPHON_PATH/psiphon-circumvention-system/pyclient
+PSIPHON_REPO_URL=https://bitbucket.org/psiphon/psiphon-circumvention-system
+OONI_VIRTUALENV_PATH=$HOME/.virtualenvs/ooniprobe
 
 mkdir -p $PSIPHON_PATH
 
 command_exists() {
-   command -v "$@" > /dev/null 2>&1
+  command -v "$@" > /dev/null 2>&1
 }
 
 user="$(id -un 2>/dev/null || true)"
@@ -17,55 +19,59 @@ user="$(id -un 2>/dev/null || true)"
 sh_c='sh -c'
 
 if [ "$user" != 'root' ]; then
-   if command_exists sudo; then
-   sh_c='sudo sh -c -E'
-   elif command_exists su; then
-   sh_c='su -c --preserve-environment'
-   else
-   echo >&2 'Error: this installer needs the ability to run 
commands as root.'
-   echo >&2 'We are unable to find either "sudo" or "su" available 
to make this happen.'
-   exit 1
-   fi
+  if command_exists sudo; then
+sh_c='sudo sh -c -E'
+  elif command_exists su; then
+sh_c='su -c --preserve-environment'
+  else
+echo >&2 'Error: this installer needs the ability to run commands as root.'
+echo >&2 'We are unable to find either "sudo" or "su" available to make 
this happen.'
+exit 1
+  fi
 fi
 
 echo "installing dependencies"
 $sh_c "apt-get -y install zlib1g-dev libssl-dev"
 
 if [ -z "command_exists hg" ]; then
-$sh_c "apt-get -y install mercurial"
+  $sh_c "apt-get -y install mercurial"
 fi
 
 echo "cloning psiphon repository"
 cd $PSIPHON_PATH
-hg clone https://bitbucket.org/psiphon/psiphon-circumvention-system
-echo "psiphon repository cloned"
+if [ ! -d "psiphon-circumvention-system" ]; then
+  hg clone $PSIPHON_REPO_URL
+  echo "psiphon repository cloned"
+fi
 
 # optional, compile their ssh
-echo "compiling psiphon ssh"
-cd psiphon-circumvention-system/Server/3rdParty/openssh-5.9p1/
-./configure
-make
-mv ssh ../../../pyclient/
-make clean
-echo "psiphon ssh compiled"
+if [ ! -f "$PSIPHON_PYCLIENT_PATH/ssh" ]; then
+echo "compiling psiphon ssh"
+cd psiphon-circumvention-system/Server/3rdParty/openssh-5.9p1/
+./configure
+make
+mv ssh ../../../pyclient/
+make clean
+echo "psiphon ssh compiled"
+fi
 
 # check if we are in a virtualenv, create it otherwise
 echo "checking virtualenv"
-if [ python -c 'import sys; print hasattr(sys, "real_prefix")'  = "False"];then
-# we are not in a virtualenv
-# create a virtualenv
-# FIXME: assuming debian version will have secure pip/virtualenv
-if [ -z "command_exists virtualenv" ]; then
-$sh_c "apt-get -y install python-virtualenv"
-fi
-if [ ! -f $HOME/.virtualenvs/ooniprobe/bin/activate ]; then
-  # Set up the virtual environment
-  mkdir -p $HOME/.virtualenvs
-  virtualenv $HOME/.virtualenvs/ooniprobe
-  source $HOME/.virtualenvs/ooniprobe/bin/activate
-else
-  source $HOME/.virtualenvs/ooniprobe/bin/activate
-fi
+if [ `python -c 'import sys; print hasattr(sys, "real_prefix")'` = "False" ]; 
then
+  # not in a virtualenv
+  # create a virtualenv
+  # FIXME: assuming debian version will have secure pip/virtualenv
+  if [ -z "command_exists virtualenv" ]; then
+$sh_c "apt-get -y install python-virtualenv"
+  fi
+  if [ ! -f $OONI_VIRTUALENV_PATH/bin/activate ]; then
+# Set up the virtual environment
+mkdir -p $HOME/.virtualenvs
+virtualenv $OONI_VIRTUALENV_PATH
+. $OONI_VIRTUALENV_PATH/bin/activate
+  else
+. $OONI_VIRTUALENV_PATH/bin/activate
+  fi
 fi
 echo "virtualenv activated"
 
@@ -83,7 +89,7 @@ import json
 
 # Delete 'server_list' if exists
 if os.path.exists("server_list"):
-# os.remove("server_list")
+   # os.remove("server_list")
 # os.rename("server_list", "server_list")
 pass
 else:
@@ -102,15 +108,8 @@ EOF
 chmod +x psi_generate_dat.py
 ./psi_generate_dat.py
 echo "servers data file created"
-chmod +x psi_generate_dat.py
-./psi_generate_dat.py
-echo "servers data file created"
 mv psi_client.dat $PSIPHON_PYCLIENT_PATH
-

[tor-commits] [ooni-probe/master] Fix invalid references to http_requests test

2016-04-29 Thread art

commit 6e4af441525a6b96858eb5e517a12f133a49f2e1
Author: anadahz 
Date:   Mon Mar 14 20:40:24 2016 -0300

Fix invalid references to http_requests test
---
 docs/source/tests/psiphon.rst | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/docs/source/tests/psiphon.rst b/docs/source/tests/psiphon.rst
index 5050b79..b05f1f0 100644
--- a/docs/source/tests/psiphon.rst
+++ b/docs/source/tests/psiphon.rst
@@ -32,7 +32,7 @@ To test Psiphon with a different URL run:
 
 To test Psiphon when it is installed in a different path other than the user 
home:
 
-`ooniprobe blocking/http_requests -p `
+`ooniprobe third_party/psiphon -p `
 
 How to install Psiphon
 ===
@@ -49,7 +49,7 @@ To run Psiphon manually, it must be run inside of the proper 
directory:
 Sample report
 =
 
-`ooniprobe blocking/http_requests -f example_inputs/url_lists_file.txt`
+`ooniprobe third_party/psiphon -u http://google.com`
 
 ::
 



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] Integrate withoutbody optflag lost in rebase

2016-04-29 Thread art

commit 31c3a81dba626d3defed46213c7e6d6bfde632d6
Author: Arturo FilastÃ² 
Date:   Fri Feb 5 17:47:29 2016 +0100

Integrate withoutbody optflag lost in rebase
---
 ooni/nettests/blocking/http_requests.py | 2 +-
 ooni/templates/httpt.py | 6 +-
 2 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/ooni/nettests/blocking/http_requests.py 
b/ooni/nettests/blocking/http_requests.py
index f21b041..e00b4c6 100644
--- a/ooni/nettests/blocking/http_requests.py
+++ b/ooni/nettests/blocking/http_requests.py
@@ -18,7 +18,7 @@ class UsageOptions(usage.Options):
 ['factor', 'f', 0.8,
  'What factor should be used for triggering censorship (0.8 == 80%)']]
 optFlags = [
-["withoutbody","B", "don't include HTTP response body inside of the 
report"],
+["withoutbody", "B", "don't include HTTP response body inside of the 
report"],
 ]
 
 
diff --git a/ooni/templates/httpt.py b/ooni/templates/httpt.py
index a49c39b..3a36b18 100644
--- a/ooni/templates/httpt.py
+++ b/ooni/templates/httpt.py
@@ -151,9 +151,13 @@ class HTTPTest(NetTestCase):
 }
 }
 if response:
+if self.localOptions.get('withoutbody', 0) is 0:
+response_body = _representBody(response_body)
+else:
+response_body = ''
 session['response'] = {
 'headers': _representHeaders(response.headers),
-'body': _representBody(response_body),
+'body': response_body,
 'code': response.code
 }
 session['failure'] = None



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] Add psiphon install script

2016-04-29 Thread art

commit 75b88d7801fae016153e365600c3a6e649b613d0
Author: juga0 
Date:   Sun Sep 20 21:09:06 2015 +

Add psiphon install script
---
 scripts/psiphon_install.sh | 86 ++
 1 file changed, 86 insertions(+)

diff --git a/scripts/psiphon_install.sh b/scripts/psiphon_install.sh
new file mode 100755
index 000..1b6837d
--- /dev/null
+++ b/scripts/psiphon_install.sh
@@ -0,0 +1,86 @@
+#!/bin/sh
+
+#FIXME remove x
+set -ex
+
+PSIPHON_PATH=$HOME/test
+PSIPHON_PYCLIENT_PATH=$PSIPHON_PATH/psiphon-circumvention-system/pyclient
+
+echo "installing dependencies"
+# FIXME: check if hg command exits and if not bail out?
+sudo apt-get install -y mercurial
+echo "cloning psiphon repository"
+cd $PSIPHON_PATH
+hg clone https://bitbucket.org/psiphon/psiphon-circumvention-system
+echo "psiphon repository cloned"
+
+# optional, compile their ssh
+echo "compiling psiphon ssh"
+cd psiphon-circumvention-system/Server/3rdParty/openssh-5.9p1/
+./configure
+make
+mv ssh ../../../pyclient/
+make clean
+echo "psiphon ssh compiled"
+
+# check if we are in a virtualenv, create it otherwise
+echo "checking virtualenv"
+if [ python -c 'import sys; print hasattr(sys, "real_prefix")'  = "False"];then
+# we are not in a virtualenv
+# create a virtualenv
+# FIXME: assuming debian version will have secure pip/virtualenv
+sudo apt-get -y install python-virtualenv
+
+if [ ! -f $HOME/.virtualenvs/ooniprobe/bin/activate ]; then
+  # Set up the virtual environment
+  mkdir -p $HOME/.virtualenvs
+  virtualenv $HOME/.virtualenvs/ooniprobe
+  source $HOME/.virtualenvs/ooniprobe/bin/activate
+else
+  source $HOME/.virtualenvs/ooniprobe/bin/activate
+fi
+fi
+echo "virtualenv activated"
+
+# create psi_client.dat
+echo "creating servers data file"
+echo "installing dependencies to create servers data file"
+pip install -v --timeout 60  wget
+cd /tmp
+cat < psi_generate_dat.py
+#!/usr/bin/env python
+
+import wget
+import os
+import json
+
+# Delete 'server_list' if exists
+if os.path.exists("server_list"):
+# os.remove("server_list")
+# os.rename("server_list", "server_list")
+pass
+else:
+# Download 'server_list'
+url ="https://psiphon3.com/server_list;
+wget.download(url)
+
+# convert server_list to psi_client.dat
+dat = {}
+dat["propagation_channel_id"] = ""
+dat["sponsor_id"] = ""
+dat["servers"] = json.load(open('server_list'))['data'].split()
+json.dump(dat, open('psi_client.dat', 'w'))
+EOF
+
+chmod +x psi_generate_dat.py 
+./psi_generate_dat.py
+echo "serers data file created"
+mv psi_client.dat $PSIPHON_PYCLIENT_PATH
+
+
+echo "[+] Installing all of the Python dependency requirements with pip in 
your virtualenv!";
+pip install -v --timeout 60  jsonpickle pexpect
+
+# run psiphon
+# cd $PSIPHON_PYCLIENT_PATH
+# python psi_client.py



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] Fix install doc on virtualenv setup

2016-04-29 Thread art

commit 8a8500ab97a2eba292f93f05d129b9f627e59c6e
Author: anadahz 
Date:   Mon Mar 14 19:09:06 2016 -0300

Fix install doc on virtualenv setup
---
 README.rst | 7 ++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/README.rst b/README.rst
index 6234b62..58c8376 100644
--- a/README.rst
+++ b/README.rst
@@ -350,7 +350,12 @@ On Debian based systems a development environment can be 
setup as follows: (prer
 sudo apt-get install libgeoip-dev libffi-dev libdumbnet-dev libssl-dev 
libpcap-dev
 git clone https://github.com/TheTorProject/ooni-probe
 cd ooni-probe
-mkvirtualenv ooniprobe  # . ~/.virtualenvs/ooniprobe/bin/activate to 
access later
+virtualenv venv
+`virtualenv venv` will create a folder in the current directory which will
+contain the Python executable files, and a copy of the pip library which you 
can
+use to install other packages. To begin using the virtual environment, it needs
+to be activated:
+source venv/bin/activate
 pip install -r requirements.txt
 pip install -r requirements-dev.txt
 python setup.py install



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [ooni-probe/master] Merge pull request #462 from TheTorProject/fix/install-doc

2016-04-29 Thread art

commit e1c428dbb24b81edccd67bb38188ccfcd756f9a0
Merge: 3a4d9ef f73a26a
Author: Arturo FilastÃ² 
Date:   Fri Mar 11 16:05:48 2016 +0100

Merge pull request #462 from TheTorProject/fix/install-doc

Remove obsolete install method on Debian/Ubuntu

 README.rst | 32 
 1 file changed, 32 deletions(-)



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

1 2 3 >

1 - 100 of 200 matches

Mail list logo