Re: [Tails-dev] [review'n'merge:1.1] #6559, #7062, #6275: test/6559-adapt-test-suite-for-Wheezy
Merged, thanks! ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] [review'n'merge:1.1] #6559, #7062, #6275: test/6559-adapt-test-suite-for-Wheezy
13/05/14 22:12, intrigeri wrote: anonym wrote (13 May 2014 15:53:17 GMT) : My suggestion would be that we don't use the emergency shitdown, and simply sends a `halt`, which we did before, and which was much more reliable. I know you want us to the same ways we except our users to use, and I agree, but well... can't we just make a dedicated test for the emergency shutdown instead? Yes, it would be good to have the USB feature a bit more robust, while not dropping a useful test = fine with me, please go ahead. See commit fc95510. 2. Scenario: Booting Tails from a USB drive upgraded from DVD with persistence enabled # features/usb_install.feature:182 [...] And the boot device has safe access rights # features/step_definitions/usb.rb:326 And the expected persistent files are present in the filesystem # features/step_definitions/usb.rb:423 Could not find expected file in persistent directory /etc/NetworkManager/system-connections (RuntimeError) Same in Booting Tails from a USB drive upgraded from USB with persistence enabled and Booting a USB drive upgraded from ISO with persistence enabled. And on next run, I cannot reproduce this. Weird. It's notable that that particular persistence preset's directory was changed in feature/wheezy. What was your --old-iso when you ran the first test vs the second? Perhaps it's not so notable since that's the first preset that will be tested, so probably all of them were not there. A build from the devel branch, from a few days earlier. Could it have been a Wheezy-based image from before the persistence preset was changed? IIRC, this change was made a while ago, and I don't think I have any such ISO anymore. Sorry, then I have no idea. 4. Scenario: Memory erasure on an old computer # features/erase_memory.fe [...] And I shutdown and wait for Tails to finish wiping the memory # features/step_definitions/erase_memory.rb:164 Then I find very few patterns in the guest's memory # features/step_definitions/erase_memory.rb:140 Pattern coverage: 0.314% (11 MiB) 0.314% of the memory is filled with the pattern, but less than 0.250% was expected (RuntimeError) I got this once out of two tries. Is it an acceptable drawback of how the test suite works, or a real problem? To me it just shows that with the particular kernel (or whatever) that we happen to use now require us to bump the highly arbitrary 0.25% to 0.5%, perhaps. Without some more rigorous guideline to what we think is acceptable, arbitrary is what we've got. What do you think? Fair enough. Done in commit bf1795b. Maybe we want a low-priority Research ticket to look at this later, at least to document that we have an issue here? Filed as #7313. Cheers! ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] [review'n'merge:1.1] #6559, #7062, #6275: test/6559-adapt-test-suite-for-Wheezy
hi, anonym wrote (27 May 2014 15:06:03 GMT) : 13/05/14 22:48, intrigeri wrote: anonym wrote (13 May 2014 15:54:22 GMT) : Regarding the removal of test suite steps that look at notifications: [...] I've brought these checks back in commits bd095dd and 7b6d3d6. You really nailed it this time. :) Running the test suite now uncovered that the start and stop notifications aren't shown for the Unsafe Web Browser. Apparently running `notify-send` as root (as the script does) shows nothing in Wheezy, so I switched to `tails-notify-user`. Hopefully you don't mind that I fixed this in this branch and not a dedicated one. :) Any way, I'm worried that Wheezy's default notification timeout of 5 seconds (which BTW is not a bug [1]) may end up causing us problems since Sikuli certainly may need more than that to find the image, in particular on slow hardware. We'll see. That bug tells me it's not considered to be a bug in GNOME Shell, but the implication that it's not a bug in Fallback mode isn't obvious to me. Anyway, we'll see :) cryptsetup changed [1] what it does when you try to luksOpen an already unlocked device in version 1.2. [...] Thanks for the explanation. I've reviewed the branch, read your other email in this thread, and everything seems alright to me. I have assigned the tickets back to me, added a tiny improvement (ed7908), and merged into experimental. I'm now going to run the test suite, and will report back later today. Cheers! -- intrigeri | GnuPG key @ https://gaffer.ptitcanardnoir.org/intrigeri/intrigeri.asc | OTR fingerprint @ https://gaffer.ptitcanardnoir.org/intrigeri/otr.asc ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] [review'n'merge:1.1] #6559, #7062, #6275: test/6559-adapt-test-suite-for-Wheezy
11/05/14 13:50, intrigeri wrote: Hi, intrigeri wrote (10 May 2014 15:22:54 GMT) : Test suite run in progress, I'll inspect results and report back later (likely tomorrow). Here we go! I think my first failure may be a blocker, as it's a regression. For the others, I don't know. So, the failures I have seen are: 1. I got this failure once out of two tries in the Writing files to a read/write-enabled persistent partition scenario, once out of two tries in the Writing files to a read/write-enabled persistent partition with the old Tails USB installation scenario: And I completely shutdown Tails# features/step_definitions/common_steps.rb:398 Then only the expected files should persist on USB drive current # features/step_definitions/usb.rb:433 FindFailed: can not find TailsEmergencyShutdownHalt.png on the screen. Looking at the video, it seems as if TailsEmergencyShutdownHalt.png was never clicked. There's a notification left on screen at this time, so I wonder if it may be interfering somehow (but perhaps I'm only trying to find justifications for my doubt wrt. dropping the notification handling code ;) -- in any case, that's a regression, since I have never seen this before. Note that that scenario does the And all notifications have disappeared step. The notification killing I removed in commit 13d0ae9 doesn't affect usb_install.feature in any way. I think this just shows that the all notifications have disappeared step is problematic (and ). That step will immediately complete once it sees no notifications, so it really depends on being run after the last notification is has been shown... I really don't know how to improve that step. My suggestion would be that we don't use the emergency shitdown, and simply sends a `halt`, which we did before, and which was much more reliable. I know you want us to the same ways we except our users to use, and I agree, but well... can't we just make a dedicated test for the emergency shutdown instead? 2. Scenario: Booting Tails from a USB drive upgraded from DVD with persistence enabled # features/usb_install.feature:182 [...] And the boot device has safe access rights # features/step_definitions/usb.rb:326 And the expected persistent files are present in the filesystem # features/step_definitions/usb.rb:423 Could not find expected file in persistent directory /etc/NetworkManager/system-connections (RuntimeError) Same in Booting Tails from a USB drive upgraded from USB with persistence enabled and Booting a USB drive upgraded from ISO with persistence enabled. And on next run, I cannot reproduce this. Weird. It's notable that that particular persistence preset's directory was changed in feature/wheezy. What was your --old-iso when you ran the first test vs the second? Could it have been a Wheezy-based image from before the persistence preset was changed? 3. Scenario: Iceweasel should not have any plugins enabled# features/torified_browsing.feature:26 When I run iceweasel # features/step_definitions/common_steps.rb:340 And Iceweasel has started and is not loading a web page # features/step_definitions/common_steps.rb:247 And I open the address about:plugins in Iceweasel # features/step_definitions/torified_browsing.rb:7 Then I see IceweaselNoPlugins.png after at most 60 seconds # features/step_definitions/common_steps.rb:302 FindFailed: can not find IceweaselNoPlugins.png on the screen. I've seen this once out of three tries, without --keep-snapshots. Unfortunately, I have no screenshot nor video anymore. I just had a look (with --view) at what's actually happening in these scenarios. I noticed that the Iceweasel window, after appearing, is minimized and than maximized, which is strange. It would also explain the randomness of this error as the following key presses surely can be lost while that's in motion. The reason is that we end up doing two @screen.wait_and_click(IceweaselRunning.png, ...) and IceweaselRunning.png was changed from being the title to the task icon in commit 13dce7f (introduced in the wheezy branch). The `wait_and_click` was introduced to ensure focus of the window, so it has to be e.g. the title. This is fixed now. 4. Scenario: Memory erasure on an old computer # features/erase_memory.fe [...] And I shutdown and wait for Tails to finish wiping the memory # features/step_definitions/erase_memory.rb:164 Then I find very few patterns in the guest's memory # features/step_definitions/erase_memory.rb:140 Pattern coverage: 0.314% (11 MiB) 0.314% of the memory is filled with the pattern, but less
Re: [Tails-dev] [review'n'merge:1.1] #6559, #7062, #6275: test/6559-adapt-test-suite-for-Wheezy
anonym wrote (13 May 2014 15:53:17 GMT) : My suggestion would be that we don't use the emergency shitdown, and simply sends a `halt`, which we did before, and which was much more reliable. I know you want us to the same ways we except our users to use, and I agree, but well... can't we just make a dedicated test for the emergency shutdown instead? Yes, it would be good to have the USB feature a bit more robust, while not dropping a useful test = fine with me, please go ahead. 2. Scenario: Booting Tails from a USB drive upgraded from DVD with persistence enabled # features/usb_install.feature:182 [...] And the boot device has safe access rights # features/step_definitions/usb.rb:326 And the expected persistent files are present in the filesystem # features/step_definitions/usb.rb:423 Could not find expected file in persistent directory /etc/NetworkManager/system-connections (RuntimeError) Same in Booting Tails from a USB drive upgraded from USB with persistence enabled and Booting a USB drive upgraded from ISO with persistence enabled. And on next run, I cannot reproduce this. Weird. It's notable that that particular persistence preset's directory was changed in feature/wheezy. What was your --old-iso when you ran the first test vs the second? A build from the devel branch, from a few days earlier. Could it have been a Wheezy-based image from before the persistence preset was changed? IIRC, this change was made a while ago, and I don't think I have any such ISO anymore. 3. Scenario: Iceweasel should not have any plugins enabled# features/torified_browsing.feature:26 [...] This is fixed now. Congrats! \o/ 4. Scenario: Memory erasure on an old computer # features/erase_memory.fe [...] And I shutdown and wait for Tails to finish wiping the memory # features/step_definitions/erase_memory.rb:164 Then I find very few patterns in the guest's memory # features/step_definitions/erase_memory.rb:140 Pattern coverage: 0.314% (11 MiB) 0.314% of the memory is filled with the pattern, but less than 0.250% was expected (RuntimeError) I got this once out of two tries. Is it an acceptable drawback of how the test suite works, or a real problem? To me it just shows that with the particular kernel (or whatever) that we happen to use now require us to bump the highly arbitrary 0.25% to 0.5%, perhaps. Without some more rigorous guideline to what we think is acceptable, arbitrary is what we've got. What do you think? Fair enough. Maybe we want a low-priority Research ticket to look at this later, at least to document that we have an issue here? Cheers, -- intrigeri | GnuPG key @ https://gaffer.ptitcanardnoir.org/intrigeri/intrigeri.asc | OTR fingerprint @ https://gaffer.ptitcanardnoir.org/intrigeri/otr.asc ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] [review'n'merge:1.1] #6559, #7062, #6275: test/6559-adapt-test-suite-for-Wheezy
Hi, intrigeri wrote (10 May 2014 15:22:54 GMT) : Test suite run in progress, I'll inspect results and report back later (likely tomorrow). Here we go! I think my first failure may be a blocker, as it's a regression. For the others, I don't know. So, the failures I have seen are: 1. I got this failure once out of two tries in the Writing files to a read/write-enabled persistent partition scenario, once out of two tries in the Writing files to a read/write-enabled persistent partition with the old Tails USB installation scenario: And I completely shutdown Tails# features/step_definitions/common_steps.rb:398 Then only the expected files should persist on USB drive current # features/step_definitions/usb.rb:433 FindFailed: can not find TailsEmergencyShutdownHalt.png on the screen. Looking at the video, it seems as if TailsEmergencyShutdownHalt.png was never clicked. There's a notification left on screen at this time, so I wonder if it may be interfering somehow (but perhaps I'm only trying to find justifications for my doubt wrt. dropping the notification handling code ;) -- in any case, that's a regression, since I have never seen this before. 2. Scenario: Booting Tails from a USB drive upgraded from DVD with persistence enabled # features/usb_install.feature:182 [...] And the boot device has safe access rights # features/step_definitions/usb.rb:326 And the expected persistent files are present in the filesystem # features/step_definitions/usb.rb:423 Could not find expected file in persistent directory /etc/NetworkManager/system-connections (RuntimeError) Same in Booting Tails from a USB drive upgraded from USB with persistence enabled and Booting a USB drive upgraded from ISO with persistence enabled. And on next run, I cannot reproduce this. Weird. 3. Scenario: Iceweasel should not have any plugins enabled# features/torified_browsing.feature:26 When I run iceweasel # features/step_definitions/common_steps.rb:340 And Iceweasel has started and is not loading a web page # features/step_definitions/common_steps.rb:247 And I open the address about:plugins in Iceweasel # features/step_definitions/torified_browsing.rb:7 Then I see IceweaselNoPlugins.png after at most 60 seconds # features/step_definitions/common_steps.rb:302 FindFailed: can not find IceweaselNoPlugins.png on the screen. I've seen this once out of three tries, without --keep-snapshots. Unfortunately, I have no screenshot nor video anymore. 4. Scenario: Memory erasure on an old computer # features/erase_memory.fe [...] And I shutdown and wait for Tails to finish wiping the memory # features/step_definitions/erase_memory.rb:164 Then I find very few patterns in the guest's memory # features/step_definitions/erase_memory.rb:140 Pattern coverage: 0.314% (11 MiB) 0.314% of the memory is filled with the pattern, but less than 0.250% was expected (RuntimeError) I got this once out of two tries. Is it an acceptable drawback of how the test suite works, or a real problem? Cheers, -- intrigeri | GnuPG key @ https://gaffer.ptitcanardnoir.org/intrigeri/intrigeri.asc | OTR fingerprint @ https://gaffer.ptitcanardnoir.org/intrigeri/otr.asc ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] [review'n'merge:1.1] #6559, #7062, #6275: test/6559-adapt-test-suite-for-Wheezy
Hi, anonym wrote (08 May 2014 15:52:25 GMT) : Tickets: https://labs.riseup.net/code/issues/6559 https://labs.riseup.net/code/issues/7062 https://labs.riseup.net/code/issues/6275 Branch: test/6559-adapt-test-suite-for-Wheezy The the computer boots Tails step fails for me: it does not detect TailsBootSplash.png. It seems that the syslinux menu is quite smaller than before (in --view mode, it takes about 2 thirds of the screen), presumably since the move to QXL. Needless to say, it prevents from running 99% of the test suite without fixing it myself = reassigning to anonym. Cheers! -- intrigeri | GnuPG key @ https://gaffer.ptitcanardnoir.org/intrigeri/intrigeri.asc | OTR fingerprint @ https://gaffer.ptitcanardnoir.org/intrigeri/otr.asc ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
