Re: Fedora 11 Test Day survey
1. How did you find out about Fedora Test Days? fedora-devel-list 2. Was sufficient documentation available to help you participate in a Fedora Test Day? If not, what did you find missing or in need of improvement? mostly, see examples: good: Test Day:2009-03-26 Nouveau /sbin/lspci -d 10de: | grep -iq VGA echo Join Nouveau Fedora Test Day || echo No nVidia graphics hardware found. bad: Test Day:2009-04-09 UEFI FIXME|How to identify if your system supports UEFI? This text was there for some time and than it just disappeared without any answer. 3. Did you encounter any obstacles preventing participation in Fedora test Days? How might they have been avoided? Did you discover any workaround? let's have live cd for all test days (where it makes sense) 4. Were you able to locate and download installation media for testing? Did it function as expected? yes 5. What follow-up actions do you expect after the Test Day? Are your expectations currently being met? imo it's ok now 6. Would you participate again in future Fedora Test Days? yes 7. Do you have any more general comments or any suggestions for improving future test days? start them earlier in rawhide development phase. If tens of bugs are found three weeks before freeze, we can hardly expect they are fixed in time. -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Fedora 11 Test Day survey
On Wed June 3 2009, James Laska wrote: 1. How did you find out about Fedora Test Days? devel mailing list 2. Was sufficient documentation available to help you participate in a Fedora Test Day? If not, what did you find missing or in need of improvement? Yes. 3. Did you encounter any obstacles preventing participation in Fedora test Days? How might they have been avoided? Did you discover any workaround? The live iso images failed to be converted to live usb media on F10 with a strange looking boot menu. It could have beend avoided with testing this before starting the test day. 4. Were you able to locate and download installation media for testing? Did it function as expected? Yes. No, see 3. Also I would have expected testing instructions on the live media to perform the tests without the need to setup an internet connection on the testing machine. Also some bug report info gathering scripts would have been nice, that e.g. collect all the data required by the packages maintainers for a bug report in some file on the usb medium. E.g. the Xorg people wanted a lot of information that I did not have anymore after I rebooted to my stable F10 system and reported the bug. Also when the test crashes the Xserver, it is very annoying to get to the log files, because they are deleted after a reboot. Also the media was not signed in a way that one can easily verify that it was not tampered with. E.g. the official Fedora releases are normally signed with a gpg key that can be obtained via the SSH secured CVS repository. 5. What follow-up actions do you expect after the Test Day? Are your expectations currently being met? The reported bugs should be fixed soon. 6. Would you participate again in future Fedora Test Days? Yes. 7. Do you have any more general comments or any suggestions for improving future test days? It would be nice to automate the tests that need to be run, e.g. just boot the media and then the tests are automatically started and the user is asked for input in case something cannot be detected automatically. This is especially useful for testing hardware support, because the required testing actions are mostly simple, e.g. boot with this option, play this video, ... Regards Till signature.asc Description: This is a digitally signed message part. -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Fedora 11 Test Day survey
On Thu, 2009-06-04 at 10:08 +0200, Michal Hlavinka wrote: 2. Was sufficient documentation available to help you participate in a Fedora Test Day? If not, what did you find missing or in need of improvement? mostly, see examples: good: Test Day:2009-03-26 Nouveau /sbin/lspci -d 10de: | grep -iq VGA echo Join Nouveau Fedora Test Day || echo No nVidia graphics hardware found. bad: Test Day:2009-04-09 UEFI FIXME|How to identify if your system supports UEFI? This text was there for some time and than it just disappeared without any answer. The UEFI one was kind of a special case. The only people who actually have appropriate hardware either work for BIOS companies, hardware manufacturers, or in the testing department of companies like RH, and everyone who has UEFI hardware certainly knows they do. :) We intentionally under-promoted that test day because it was really only of interest to a small and very definitely defined circle of people. We should, however, have made that rather a lot clearer on the Wiki page itself, for people who'd got into the habit of just looking at each week's Test Day as it came. You're right there. 7. Do you have any more general comments or any suggestions for improving future test days? start them earlier in rawhide development phase. If tens of bugs are found three weeks before freeze, we can hardly expect they are fixed in time. Scheduling is a tricky point, yep: too late in the cycle and you can't get fixes in, too early and the feature isn't done being written yet. We're trying to think of ways to optimize scheduling as far as we can. Thanks for your feedback! -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org http://www.happyassassin.net -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Fedora 11 Test Day survey
On Thu, 2009-06-04 at 14:16 +0200, Till Maas wrote: 3. Did you encounter any obstacles preventing participation in Fedora test Days? How might they have been avoided? Did you discover any workaround? The live iso images failed to be converted to live usb media on F10 with a strange looking boot menu. It could have beend avoided with testing this before starting the test day. This sounds like the somewhat-infamous problem with f10's USB creator not working for F11 images. Later in the F11 cycle, we did have that issue neatly identified and pinned down, and we started adding a box out to test days explaining it. See the nouveau page, for e.g.: https://fedoraproject.org/wiki/Test_Day:2009-03-26_Nouveau#Live_Image note the blue box. The explanation could have been a bit more descriptive and emphatic, though, I suppose. 4. Were you able to locate and download installation media for testing? Did it function as expected? Yes. No, see 3. Also I would have expected testing instructions on the live media to perform the tests without the need to setup an internet connection on the testing machine. That's a good point - in some situations it's hard to get an internet connection on a live image. So we should try as hard as we can to include all necessary material on the image itself. Also some bug report info gathering scripts would have been nice, that e.g. collect all the data required by the packages maintainers for a bug report in some file on the usb medium. E.g. the Xorg people wanted a lot of information that I did not have anymore after I rebooted to my stable F10 system and reported the bug. Also when the test crashes the Xserver, it is very annoying to get to the log files, because they are deleted after a reboot. This one's a bit tricky - on the one hand it'd be nice, on the other it'd be a substantial development effort for each test day and they already take quite a lot of effort to organize. Perhaps, however, we could come up with a general-purpose script which could be used for X test days, as the procedures tend to be very similar for different drivers and different releases. We'll certainly look into that for the X test days for F12. Thanks for your feedback! -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org http://www.happyassassin.net -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Fedora 11 Test Day survey
On Wed, Jun 3, 2009 at 9:10 AM, James Laska jla...@redhat.com wrote: Greetings, The Fedora QA team would like your feedback on Fedora 11 Test Days. You may have seen Adam Williamson's planet post [1] kicking off Fedora 12 Test Day planning. We're interested in identifying areas for improvement to increase participation and improve effectiveness. Please take 10-15 minutes to answer any/all of the questions below. You may reply to the mailing list, or send feedback directly to me. Your responses to this survey are instrumental in making Fedora 12 Test Days successful. Many thanks to Chris Ward for his help in getting things moving with the survey questions! === 1. How did you find out about Fedora Test Days? 2. Was sufficient documentation available to help you participate in a Fedora Test Day? If not, what did you find missing or in need of improvement? 3. Did you encounter any obstacles preventing participation in Fedora test Days? How might they have been avoided? Did you discover any workaround? 4. Were you able to locate and download installation media for testing? Did it function as expected? 5. What follow-up actions do you expect after the Test Day? Are your expectations currently being met? 6. Would you participate again in future Fedora Test Days? 7. Do you have any more general comments or any suggestions for improving future test days? === Thanks, James [1] http://www.happyassassin.net/2009/06/02/whats-goin-on-f12-test-days/ -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list I'll answer these in order. 1. I got lucky when I looked in the mailing list. 2. Yes there was good enough docs for the test days to help me participate. 3. My biggest obstacle with test days was that they were not planned early enough. Most of the test days seemed to be planned less than 48 hours ahead of time. If test days were planned better, I could actually participate more. 4. Yes, I was able to download them. No, the media didn't work. It generally hung the computer, but that's not the fault of the test days. 5. I would expect a recap of the testing efforts so that Fedora people could analyze what the issues were, track them, and fix them. I suppose they are. The mailing list enabled them to do this, but there was no formal method of doing it. 6. If they were planned better, then maybe I would be able to set aside time to do them. I would like to participate in future Test Days. 7. Set up a reporting center just for Test Day feedback. Using the wiki is definitely not good enough. Additionally, Do not limit the test days to people subscribed to the mailing list. Take a page from Mozilla's books and announce those test days to the world. Unfortunately, to do that, test days need to be planned better. Hopefully this feedback helps :) -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Fedora 11 Test Day survey
James Laska wrote: 1. How did you find out about Fedora Test Days? Mailing list posting. 2. Was sufficient documentation available to help you participate in a Fedora Test Day? If not, what did you find missing or in need of improvement? Yes, I found everything I needed on the corresponding wiki page. 3. Did you encounter any obstacles preventing participation in Fedora test Days? How might they have been avoided? Did you discover any workaround? Time. Test days are sometimes not announced early enough for me, or I do not have them marked on my calendar so I forget about them. 4. Were you able to locate and download installation media for testing? Did it function as expected? Yes. Yes. 5. What follow-up actions do you expect after the Test Day? Are your expectations currently being met? I expected an analysis of the data received either by a mailing list post or an update on the wiki page. I saw neither and thought my data was just thrown into the wind. My expectations were not met. 6. Would you participate again in future Fedora Test Days? Yes. 7. Do you have any more general comments or any suggestions for improving future test days? Please get the Fedora calendar server going. I'd love to subscribe Thunderbird/Lightning to the QA calendar. People would be able to know about and participate in test days (or any QA event) without a mailing list subscription or a 24/7 IRC connection as it seems some things are discussed solely on IRC. -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Fedora 11 Test Day survey
Thanks for the feedback! On Wed, 2009-06-03 at 20:16 +0530, Rahul Sundaram wrote: 5. What follow-up actions do you expect after the Test Day? Are your expectations currently being met? Yes. Although I was hoping there would be a test day for Ext4. I was too, but there were some schedule conflicts which kept it from happening on the QA side. In the end the only test day topic with focus on ext4 was around changing the anaconda default filesystem to ext4 (https://fedoraproject.org/wiki/QA/Test_Days/2009-02-05). Thanks, James signature.asc Description: This is a digitally signed message part -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Fedora 11 Test Day survey
James Laska wrote: 1. How did you find out about Fedora Test Days? fedora-devel-announce 2. Was sufficient documentation available to help you participate in a Fedora Test Day? If not, what did you find missing or in need of improvement? The instructions were sufficient. 3. Did you encounter any obstacles preventing participation in Fedora test Days? How might they have been avoided? Did you discover any workaround? I found out about the Intel graphics test day too late to be able to participate on the right day. I first had to create a FAS account as I hadn't yet taken all the steps to become a packager. I got side-tracked by an incorrect error message in FAS and had some problems before I could report that. Then I had to create a Smolt profile. SmoltGUI crashed but I could work around the crash by changing the locale. Smolt using two kinds of UUIDs caused some confusion. I could eventually go through the test cases for Intel graphics a week after the actual test day. See also the next question: 4. Were you able to locate and download installation media for testing? Did it function as expected? Live CD images were linked from the wiki pages. I had no problems downloading them. I eventually managed to make a working live USB stick from the one for the Intel test day, after I transferred it to my work computer where I had Fedora 10 and could install the latest Syslinux from Rawhide. It wasn't possible to do this on Fedora 9. The live USB stick I made from the CD image for the Nvidia test day was more dead than live. It wouldn't boot, so I couldn't participate in that test day. 5. What follow-up actions do you expect after the Test Day? Are your expectations currently being met? I expected that someone would attempt to fix the bugs I reported. No such attempts have been mentioned in Bugzilla so far. I suppose I'll se whether they've been fixed when I upgrade to Fedora 11. Perhaps there would have been more interest in my reports if I had submitted them during the actual test day. 6. Would you participate again in future Fedora Test Days? If they cover some functionality that's particularly important to me or some less than common hardware that I have. Björn Persson signature.asc Description: This is a digitally signed message part. -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Fedora 11 Test Day survey
1. How did you find out about Fedora Test Days? Planet.fp.o 2. Was sufficient documentation available to help you participate in a Fedora Test Day? If not, what did you find missing or in need of improvement? Documentation was excellent. 3. Did you encounter any obstacles preventing participation in Fedora test Days? How might they have been avoided? Did you discover any workaround? None - only took part on the nouveau day though. 4. Were you able to locate and download installation media for testing? Did it function as expected? Yes. 5. What follow-up actions do you expect after the Test Day? Are your expectations currently being met? Bugs fixed. Possibly a summary of how the previous test day helped before talking about the next one? 6. Would you participate again in future Fedora Test Days? Yes, dependent on barrier to entry. 7. Do you have any more general comments or any suggestions for improving future test days? Just that I'm very glad they're happening. Well done. -- Christopher Brown -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
