Re: [Tails-dev] Progress report on the automated test suite

[intrigeri]

hi,

anonym wrote (13 Nov 2012 13:01:41 GMT) :
> 03/11/12 09:55, intrigeri wrote:
>>> that starts on the guest when the boot parameter
>>> "autotest_never_use_me" is present on the kernel cmdline.
>> 
>> "autotest_never_use_me" looks to me like "(speaking to) autotest:
>> never use me". What about "backdoor_for_autotest"?

> I'm not sure I want to mention the word "backdoor". Sure, I do it in the
> remote shell server script, but then it's mentioned in a context where
> sane people should have no reason to be worried.

Why should the boot parameter name be mentioned in contexts where this
does not apply? (Not a rhetorical question.)

>>> Saving/restoring VM snapshots
>>> =
>>> [...]
>> 
>> For both features, to reply on the 'how "ethical" they are in the
>> context of test-driven development' topic, I'd need a concrete example
>> of how this would be used in practice.

> I'm sure there are specific/concrete situations where this is not a good
> idea. I was more interested  if you saw any fundamental flaws with this
> approach since it's a step away from black box testing (same applies to
> the remote shell).

Sorry, I'm not in a mood to think about fundamental flaws without
examples. Anyhow, I'll try to contribute a bit, hoping that helps.

So, I think that:

1. There are serious shortcomings that come with these features.
   Every time one cheats and uses them, one should know what they are
   actually *not* testing, and think if/how that could be tested.
2. Trying at all costs to totally avoid to use these features is
   probably not a good use of our time.
3. Generally, I like to exercise systems under test at different
   levels, and the remote shell feature is probably the most efficient
   way to zoom-in and run something like "unit tests". This may not
   totally replace more zoomed-out, behavioral testing, though.

That's all I feel I can answer, on a general level.

I'd rather be pointed to cases when it looks much easier or even
needed to resort to such hacks, and then, discuss whether that is
acceptable on a case by case basis. But that may happen later, once
actual reasons to use these features arise.

Cheers,
-- 
  intrigeri
  | GnuPG key @ https://gaffer.ptitcanardnoir.org/intrigeri/intrigeri.asc
  | OTR fingerprint @ https://gaffer.ptitcanardnoir.org/intrigeri/otr.asc
___
tails-dev mailing list
tails-dev@boum.org
https://mailman.boum.org/listinfo/tails-dev

Re: [Tails-dev] Progress report on the automated test suite

[anonym]

03/11/12 17:18, Ague Mill wrote:
> anonym:
>> Next I'd like to announce that the automated test suite, in its current
>> unfinished state, actually has found its very first Tails bug. Here's
>> the cucumber output of when it was found:
>>
>> [...]
>> And all Internet traffic has only flowed through Tor
>>   # cucumber/iceweasel/step_definitions/torified_browsing.rb:66
>>
>>   The following IPv6 hosts were contacted:
>>   ff02::1
>>   Full network capture available at: [...censored...]
>> There were network leaks! (RuntimeError) [...]
>>
>> In other words, our firewall leaks link-local IPv6 broadcasts even
>> though it should block everything IPv6 (right?). This is promising (not
>> that Tails has this particular bug, but that the test suite found it)!
> 
> I did not run the code itself, but are you sure that these packets came
> from Tails and not from the host system?

Unless there's a bug in tcpdump, yes, I'm sure. tcpdump is configured
with the filter "src host ${IP} or src host ${IP6}", where IP* = the
guest running Tails.

>> Saving/restoring VM snapshots
>> =
>>
>> This is how I intend to use it for a given feature:
>>
>>   Background:
>> Given I restore the background snapshot if it exists
>> [ ... "real" background steps ... ]
>> And I save the background snapshot if it does not exist
>>
>>   [ ... Scenarios ... ]
> 
> Those lines feel like noise: they are an implementation detail and
> should not appear in the scenarios.
> 
> Cucumber offer tags and hooks that should be usable to achieve something
> similar while keeping the scenarios as lean as possible. See:
>  and
> 

I looked at them but couldn't find anything that I thought suited. I
need one hook that runs exactly before the background, and one that runs
exactly after.

>> An issue with restoring past state like this is that our Tor's circuit
>> state may get out-of-sync with the circuit state of the relays they use.
>> For instance, I ran 10 tests that restored to the same post-background
>> state and all but the first two failed to fetch a web page. Then I ran
>> 10 tests where I do the following after each snapshot restore:
>>
>>   1. Stop Tor.
>>   2. Sync time from host to guest.
>>   3. Start Tor.
>>
>> And then all 10 tests succeeded, so it seems resetting Tor like this is
>> highly necessary.
> 
> Indeed, as restoring from a snapshot is likely to break all existing TCP
> connections. Have you tried to see if a SIGHUP sent to Tor is sufficient?

>From previous experiences (e.g. "tordate") Tor doesn't behave well if it
experience a time jump, even if given a HUP, so in order to avoid future
mysterious failed tests due to Tor being confused, I thought restarting
it was for the best.

> Side note: your `try_for` function is very unidiomatic Ruby.
> I suggest you have a look at the part about blocks on
> ,
> and the `yield` and `block_given?` methods.

Thanks, I'll look into it (again)! I actually used `yield` at first, but
didn't get it to work, whereas anonymous functions worked fine. I
suspect those issues might have been due to an unrelated problem (the
stock 'timeout' module in Ruby, which I used at first, can lockup if
syscall get involved, apparently).

Cheers!

___
tails-dev mailing list
tails-dev@boum.org
https://mailman.boum.org/listinfo/tails-dev

Re: [Tails-dev] Progress report on the automated test suite

[anonym]

03/11/12 09:55, intrigeri wrote:
>> Next I'd like to announce that the automated test suite, in its
>> current unfinished state, actually has found its very first Tails
>> bug. [...] In other words, our firewall leaks link-local IPv6
>> broadcasts even though it should block everything IPv6 (right?).
> 
> Ouch.
> 
> WAN hat on: please report it (ticket + email) separately so that it
> does not get lost in the middle of this "report on the automated test
> suite" thread.

For the ticket, see todo/really_block_ipv6.

>> I'd like to present the last two with a bit more depth and hear your
>> opinions, especially w.r.t. the fact that they alter Tails or "cheat" in
>> the testing process, so I wonder how "ethical" they are in the context
>> of test-driven development.
> 
>> Running arbitrary commands inside the guest VM
>> ==
> 
>> This is very valuable as it makes many tests that would be truly
>> awkward to do with sikuli into something trivial. libvirt doesn't
>> seem to have something like VirtualBox' `vboxmanage guestcontrol
>> execute` (provided by the VirtualBox guest additions), so
>> I implemented a simple remote shell (read: a backdoor (listening on
>> port 1337 + firewall exception) so expect havoc on the Tails forum!)
>> that starts on the guest when the boot parameter
>> "autotest_never_use_me" is present on the kernel cmdline.
> 
> "autotest_never_use_me" looks to me like "(speaking to) autotest:
> never use me". What about "backdoor_for_autotest"?

I'm not sure I want to mention the word "backdoor". Sure, I do it in the
remote shell server script, but then it's mentioned in a context where
sane people should have no reason to be worried. Makes sense? Other
naming suggestions?

>> Saving/restoring VM snapshots
>> =
>> [...]
> 
> For both features, to reply on the 'how "ethical" they are in the
> context of test-driven development' topic, I'd need a concrete example
> of how this would be used in practice.

I'm sure there are specific/concrete situations where this is not a good
idea. I was more interested  if you saw any fundamental flaws with this
approach since it's a step away from black box testing (same applies to
the remote shell).

Cheers!

___
tails-dev mailing list
tails-dev@boum.org
https://mailman.boum.org/listinfo/tails-dev

Re: [Tails-dev] Progress report on the automated test suite

[Ague Mill]

anonym:
> I'd like to start by saying that I think the work bertagaz did on the
> jruby + sikuli + cucumber combo is really great. He called it PoC, but
> after having worked on it for some time now I say it's definitely fit
> for the task at hand.

Great. :)
 
> Next I'd like to announce that the automated test suite, in its current
> unfinished state, actually has found its very first Tails bug. Here's
> the cucumber output of when it was found:
> 
> [...]
> And all Internet traffic has only flowed through Tor
>   # cucumber/iceweasel/step_definitions/torified_browsing.rb:66
> 
>   The following IPv6 hosts were contacted:
>   ff02::1
>   Full network capture available at: [...censored...]
> There were network leaks! (RuntimeError) [...]
> 
> In other words, our firewall leaks link-local IPv6 broadcasts even
> though it should block everything IPv6 (right?). This is promising (not
> that Tails has this particular bug, but that the test suite found it)!

I did not run the code itself, but are you sure that these packets came
from Tails and not from the host system?
 
> Saving/restoring VM snapshots
> =
> 
> This is how I intend to use it for a given feature:
> 
>   Background:
> Given I restore the background snapshot if it exists
> [ ... "real" background steps ... ]
> And I save the background snapshot if it does not exist
> 
>   [ ... Scenarios ... ]

Those lines feel like noise: they are an implementation detail and
should not appear in the scenarios.

Cucumber offer tags and hooks that should be usable to achieve something
similar while keeping the scenarios as lean as possible. See:
 and


> An issue with restoring past state like this is that our Tor's circuit
> state may get out-of-sync with the circuit state of the relays they use.
> For instance, I ran 10 tests that restored to the same post-background
> state and all but the first two failed to fetch a web page. Then I ran
> 10 tests where I do the following after each snapshot restore:
> 
>   1. Stop Tor.
>   2. Sync time from host to guest.
>   3. Start Tor.
> 
> And then all 10 tests succeeded, so it seems resetting Tor like this is
> highly necessary.

Indeed, as restoring from a snapshot is likely to break all existing TCP
connections. Have you tried to see if a SIGHUP sent to Tor is sufficient?
 


Side note: your `try_for` function is very unidiomatic Ruby.
I suggest you have a look at the part about blocks on
,
and the `yield` and `block_given?` methods.

-- 
Ague


pgp4LnUB9l9Af.pgp
Description: PGP signature
___
tails-dev mailing list
tails-dev@boum.org
https://mailman.boum.org/listinfo/tails-dev

Re: [Tails-dev] Progress report on the automated test suite

[intrigeri]

hi,

anonym wrote (02 Nov 2012 16:42:31 GMT) :
> I'd like to start by saying that I think the work bertagaz did on
> the jruby + sikuli + cucumber combo is really great. He called it
> PoC, but after having worked on it for some time now I say it's
> definitely fit for the task at hand.

Great!

> Next I'd like to announce that the automated test suite, in its
> current unfinished state, actually has found its very first Tails
> bug. [...] In other words, our firewall leaks link-local IPv6
> broadcasts even though it should block everything IPv6 (right?).

Ouch.

WAN hat on: please report it (ticket + email) separately so that it
does not get lost in the middle of this "report on the automated test
suite" thread.

For this reason, I'll refrain from replying here.

> This is promising (not that Tails has this particular bug, but that
> the test suite found it)!

Clearly!

> The framework currently has the following test primitives and other
> interesting features so far: [...]

Awesome.

> I'd like to present the last two with a bit more depth and hear your
> opinions, especially w.r.t. the fact that they alter Tails or "cheat" in
> the testing process, so I wonder how "ethical" they are in the context
> of test-driven development.

> Running arbitrary commands inside the guest VM
> ==

> This is very valuable as it makes many tests that would be truly
> awkward to do with sikuli into something trivial. libvirt doesn't
> seem to have something like VirtualBox' `vboxmanage guestcontrol
> execute` (provided by the VirtualBox guest additions), so
> I implemented a simple remote shell (read: a backdoor (listening on
> port 1337 + firewall exception) so expect havoc on the Tails forum!)
> that starts on the guest when the boot parameter
> "autotest_never_use_me" is present on the kernel cmdline.

"autotest_never_use_me" looks to me like "(speaking to) autotest:
never use me". What about "backdoor_for_autotest"?

> If it's not there, the script that implements the remote shell
> server is removed from the filesystem. Of course, it is still
> accessible from the read-only fs, but it will refuse to start if the
> boot parameter isn't present. My hope is that all these failsafes
> will make sense to all but our most "conspiranoid" users. :)

Looks good.

> Saving/restoring VM snapshots
> =
> [...]

For both features, to reply on the 'how "ethical" they are in the
context of test-driven development' topic, I'd need a concrete example
of how this would be used in practice.

Cheers!
-- 
  intrigeri
  | GnuPG key @ https://gaffer.ptitcanardnoir.org/intrigeri/intrigeri.asc
  | OTR fingerprint @ https://gaffer.ptitcanardnoir.org/intrigeri/otr.asc
___
tails-dev mailing list
tails-dev@boum.org
https://mailman.boum.org/listinfo/tails-dev

[Tails-dev] Progress report on the automated test suite

[anonym]

Hi,

I'd like to start by saying that I think the work bertagaz did on the
jruby + sikuli + cucumber combo is really great. He called it PoC, but
after having worked on it for some time now I say it's definitely fit
for the task at hand.

Next I'd like to announce that the automated test suite, in its current
unfinished state, actually has found its very first Tails bug. Here's
the cucumber output of when it was found:

[...]
And all Internet traffic has only flowed through Tor
  # cucumber/iceweasel/step_definitions/torified_browsing.rb:66

  The following IPv6 hosts were contacted:
  ff02::1
  Full network capture available at: [...censored...]
There were network leaks! (RuntimeError) [...]

In other words, our firewall leaks link-local IPv6 broadcasts even
though it should block everything IPv6 (right?). This is promising (not
that Tails has this particular bug, but that the test suite found it)!

The framework currently has the following test primitives and other
interesting features so far:

* Sikuli stuff, e.g. waiting for an image (or text via OCR) to appear
  on the screen, and perform arbitrary mouse and keyboard actions.

* Capturing all of the guest's network traffic and determining whether
  the Tails firewall leaked anything. On failure the capture is
  preserved so manual inspection is possible.

* Take screenshots. It's currently unused, but I think it could be
  useful to save a screenshot upon test failure. If we want to go
  really crazy it would be trivial to add support for recording videos
  using e.g. `ffmpeg -f x11grab` to record the whole DISPLAY that
  virt-viewer + sikuli runs in, which could be helpful for post-failure
  analysis.

* Running arbitrary commands inside the guest VM.

* Saving/restoring VM snapshots (not yet pushed to git)

I'd like to present the last two with a bit more depth and hear your
opinions, especially w.r.t. the fact that they alter Tails or "cheat" in
the testing process, so I wonder how "ethical" they are in the context
of test-driven development.

Running arbitrary commands inside the guest VM
==

This is very valuable as it makes many tests that would be truly awkward
to do with sikuli into something trivial. libvirt doesn't seem to have
something like VirtualBox' `vboxmanage guestcontrol execute` (provided
by the VirtualBox guest additions), so I implemented a simple remote
shell (read: a backdoor (listening on port 1337 + firewall exception) so
expect havoc on the Tails forum!) that starts on the guest when the boot
parameter "autotest_never_use_me" is present on the kernel cmdline. If
it's not there, the script that implements the remote shell server is
removed from the filesystem. Of course, it is still accessible from the
read-only fs, but it will refuse to start if the boot parameter isn't
present. My hope is that all these failsafes will make sense to all but
our most "conspiranoid" users. :)

Saving/restoring VM snapshots
=

This is how I intend to use it for a given feature:

  Background:
Given I restore the background snapshot if it exists
[ ... "real" background steps ... ]
And I save the background snapshot if it does not exist

  [ ... Scenarios ... ]

So, for the first scenario all background steps are run normally, and in
the last one the VM state is saved. For all remaining scenarios the VM
state is restored in the first step, and all remaining steps, including
the "real" ones, just pass immediately (i.e. each "real" step starts
with "pass if we restored from a snapshot"). I suppose common background
snapshots can be shared between different features too to improve the
testing speed.

While saving/restoring snapshots greatly improves the speed of the test
suite, this wasn't the main reason for me implementing it. Without it we
would hammer the Tor authorities (once per scenario) since each Tails
boot is a full Tor bootstrap, which clearly isn't acceptable. In the end
we will have *many* scenarios in total, and if we're gonna run the test
suite for each automatic build at some point... well, you get the picture.

An issue with restoring past state like this is that our Tor's circuit
state may get out-of-sync with the circuit state of the relays they use.
For instance, I ran 10 tests that restored to the same post-background
state and all but the first two failed to fetch a web page. Then I ran
10 tests where I do the following after each snapshot restore:

  1. Stop Tor.
  2. Sync time from host to guest.
  3. Start Tor.

And then all 10 tests succeeded, so it seems resetting Tor like this is
highly necessary.

That's all.

Cheers!

___
tails-dev mailing list
tails-dev@boum.org
https://mailman.boum.org/listinfo/tails-dev