Re: [Sugar-devel] Test protocol for Resume vs. Start new
On Wed, Jan 27, 2010 at 13:17, Wade Brainerd wad...@gmail.com wrote: If possible, it might be helpful to record the students completing this test, and then post the results to the wiki (as links to YouTube etc). We video tape focus testing sessions all the time in the video game industry. You just have to be careful about interpreting the results, and it does take some time to go through watching them all. But it reveals a lot of hesitations, failed attempts, etc. that don't come out in a survey. Anyway, either way I agree that this test would be helpful in determining the best home view behavior. This may be relevant: http://mairin.wordpress.com/2009/08/26/open-source-portable-usability-testing-lab/ http://mairin.wordpress.com/2009/10/17/open-source-portable-usability-testing-lab-part-2-the-parts/ Regards, Tomeu Wade On Mon, Jan 25, 2010 at 11:35 AM, Christian Marc Schmidt christianm...@gmail.com wrote: Thanks Gary, I'll have a look at this. I'm worried that conducting tests with novice users might be inconclusive, given that we are testing schoolchildren. But I am also not an educator. I'd be happy to work with someone with an educational background to put something together that might be suitable as a test for new users. For now, though, I'd suggest we stick to our plan to see where it gets us. Christian On Mon, Jan 25, 2010 at 11:20 AM, Gary C Martin g...@garycmartin.com wrote: Hi Christian, On 23 Jan 2010, at 15:01, Christian Marc Schmidt wrote: Hi all: For our meeting at 10:30pm EST today, I put together a draft of a test protocol (in email body below). Please look it over and feel free to edit or add to. Thanks, Christian -- Sugar Home View Concept Testing Draft 01/23/10 Goal: Evaluate the Home view for Resume activity and Start new activity functionality. Protocol: Alternate between two builds, one per student: The current build [Build A], which makes Resume the default action, and a test build [Build B] making Start new the default action. Duration: 5 mins per student Student Profile: Should be a current user (not a first-time user). Age and grade should be recorded, along with duration of overall Sugar experience. 1. Begin at Home screen after boot up 2. Test cognitive model • Describe what you see in this view. • What are the symbols in the ring? • Describe what each symbol in the ring is. • Do you know why the symbols are different colors? • What activity did you last work on? 3. Test pathways [Alternate between two pathways, one per student] Pathway 1: • Show how you would go back to the activity you last worked on. Describe what you are doing. [Student shows pathway to resume activity.] • Now, show what you would do if you wanted to start a new [Write/Draw/Browse, etc.; should be of the same kind] activity. Describe what you are doing. [Student shows pathway to start a new activity of the same kind.] • Now, show what you would do if you wanted to start a new [Write/Draw/Browse, etc.; should be a different activity than the current one] activity. [Student shows pathway to start a new activity different from the current activity.] Pathway 2: • Show how you would start a new activity. Describe what you are doing. [Student opens a new activity.] • Now, show what you would do if you wanted to go back to the activity you last worked on. Describe what you are doing. [Student shows pathway to resume a previous activity.] • Now, show what you would do if you wanted to start a new [Write/Draw/Browse, etc.; should be of the same kind] activity. Describe what you are doing.[Student shows pathway to start a new activity of the same kind.] 4. General questions If you could change anything about Sugar, what would make it easier for you to go back to a previous activity? What would make it easier for you to start a new activity? Is there anything else about Sugar that you would change? I know we've drifted from a usability study model, to a focus group model for practical classroom reasons – but I though Jakob Nielson's latest article was a timely piece worth a quick read through given we are going to be getting feedback from existing Sugar users. Summary: It's more difficult to conduct usability studies with experienced users than with novices, and the improvements are usually smaller. Still, improving expert performance is often worth the effort. http://www.useit.com/alertbox/experienced-users.html Regards, --Gary -- anyth...@christianmarcschmidt.com 917/ 575 0013 http://www.christianmarcschmidt.com http://www.linkedin.com/in/christianmarcschmidt http://twitter.com/cms_ ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Test protocol for Resume vs. Start new
Hi Christian, On 23 Jan 2010, at 15:01, Christian Marc Schmidt wrote: Hi all: For our meeting at 10:30pm EST today, I put together a draft of a test protocol (in email body below). Please look it over and feel free to edit or add to. Thanks, Christian -- Sugar Home View Concept Testing Draft 01/23/10 Goal: Evaluate the Home view for Resume activity and Start new activity functionality. Protocol: Alternate between two builds, one per student: The current build [Build A], which makes Resume the default action, and a test build [Build B] making Start new the default action. Duration: 5 mins per student Student Profile: Should be a current user (not a first-time user). Age and grade should be recorded, along with duration of overall Sugar experience. 1. Begin at Home screen after boot up 2. Test cognitive model • Describe what you see in this view. • What are the symbols in the ring? • Describe what each symbol in the ring is. • Do you know why the symbols are different colors? • What activity did you last work on? 3. Test pathways [Alternate between two pathways, one per student] Pathway 1: • Show how you would go back to the activity you last worked on. Describe what you are doing. [Student shows pathway to resume activity.] • Now, show what you would do if you wanted to start a new [Write/Draw/Browse, etc.; should be of the same kind] activity. Describe what you are doing. [Student shows pathway to start a new activity of the same kind.] • Now, show what you would do if you wanted to start a new [Write/Draw/Browse, etc.; should be a different activity than the current one] activity. [Student shows pathway to start a new activity different from the current activity.] Pathway 2: • Show how you would start a new activity. Describe what you are doing. [Student opens a new activity.] • Now, show what you would do if you wanted to go back to the activity you last worked on. Describe what you are doing. [Student shows pathway to resume a previous activity.] • Now, show what you would do if you wanted to start a new [Write/Draw/Browse, etc.; should be of the same kind] activity. Describe what you are doing.[Student shows pathway to start a new activity of the same kind.] 4. General questions If you could change anything about Sugar, what would make it easier for you to go back to a previous activity? What would make it easier for you to start a new activity? Is there anything else about Sugar that you would change? I know we've drifted from a usability study model, to a focus group model for practical classroom reasons – but I though Jakob Nielson's latest article was a timely piece worth a quick read through given we are going to be getting feedback from existing Sugar users. Summary: It's more difficult to conduct usability studies with experienced users than with novices, and the improvements are usually smaller. Still, improving expert performance is often worth the effort. http://www.useit.com/alertbox/experienced-users.html Regards, --Gary ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Test protocol for Resume vs. Start new
Thanks Gary, I'll have a look at this. I'm worried that conducting tests with novice users might be inconclusive, given that we are testing schoolchildren. But I am also not an educator. I'd be happy to work with someone with an educational background to put something together that might be suitable as a test for new users. For now, though, I'd suggest we stick to our plan to see where it gets us. Christian On Mon, Jan 25, 2010 at 11:20 AM, Gary C Martin g...@garycmartin.comwrote: Hi Christian, On 23 Jan 2010, at 15:01, Christian Marc Schmidt wrote: Hi all: For our meeting at 10:30pm EST today, I put together a draft of a test protocol (in email body below). Please look it over and feel free to edit or add to. Thanks, Christian -- Sugar Home View Concept Testing Draft 01/23/10 Goal: Evaluate the Home view for Resume activity and Start new activity functionality. Protocol: Alternate between two builds, one per student: The current build [Build A], which makes Resume the default action, and a test build [Build B] making Start new the default action. Duration: 5 mins per student Student Profile: Should be a current user (not a first-time user). Age and grade should be recorded, along with duration of overall Sugar experience. 1. Begin at Home screen after boot up 2. Test cognitive model • Describe what you see in this view. • What are the symbols in the ring? • Describe what each symbol in the ring is. • Do you know why the symbols are different colors? • What activity did you last work on? 3. Test pathways [Alternate between two pathways, one per student] Pathway 1: • Show how you would go back to the activity you last worked on. Describe what you are doing. [Student shows pathway to resume activity.] • Now, show what you would do if you wanted to start a new [Write/Draw/Browse, etc.; should be of the same kind] activity. Describe what you are doing. [Student shows pathway to start a new activity of the same kind.] • Now, show what you would do if you wanted to start a new [Write/Draw/Browse, etc.; should be a different activity than the current one] activity. [Student shows pathway to start a new activity different from the current activity.] Pathway 2: • Show how you would start a new activity. Describe what you are doing. [Student opens a new activity.] • Now, show what you would do if you wanted to go back to the activity you last worked on. Describe what you are doing. [Student shows pathway to resume a previous activity.] • Now, show what you would do if you wanted to start a new [Write/Draw/Browse, etc.; should be of the same kind] activity. Describe what you are doing.[Student shows pathway to start a new activity of the same kind.] 4. General questions If you could change anything about Sugar, what would make it easier for you to go back to a previous activity? What would make it easier for you to start a new activity? Is there anything else about Sugar that you would change? I know we've drifted from a usability study model, to a focus group model for practical classroom reasons – but I though Jakob Nielson's latest article was a timely piece worth a quick read through given we are going to be getting feedback from existing Sugar users. Summary: It's more difficult to conduct usability studies with experienced users than with novices, and the improvements are usually smaller. Still, improving expert performance is often worth the effort. http://www.useit.com/alertbox/experienced-users.html Regards, --Gary -- anyth...@christianmarcschmidt.com 917/ 575 0013 http://www.christianmarcschmidt.com http://www.linkedin.com/in/christianmarcschmidt http://twitter.com/cms_ ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
[Sugar-devel] Test protocol for Resume vs. Start new
Hi all: For our meeting at 10:30pm EST today, I put together a draft of a test protocol (in email body below). Please look it over and feel free to edit or add to. Thanks, Christian -- * Sugar Home View Concept Testing* Draft 01/23/10 *Goal*: Evaluate the Home view for Resume activity and Start new activity functionality. *Protocol*: Alternate between two builds, one per student: The current build [Build A], which makes Resume the default action, and a test build [Build B] making Start new the default action. *Duration*: 5 mins per student *Student Profile*: Should be a current user (not a first-time user). Age and grade should be recorded, along with duration of overall Sugar experience. 1. Begin at Home screen after boot up 2. Test cognitive model 1. Describe what you see in this view. 2. What are the symbols in the ring? 3. Describe what each symbol in the ring is. 4. Do you know why the symbols are different colors? 5. What activity did you last work on? 3. Test pathways [Alternate between two pathways, one per student] Pathway 1: 1. Show how you would go back to the activity you last worked on. Describe what you are doing. [Student shows pathway to resume activity.] 2. Now, show what you would do if you wanted to start a new [Write/Draw/Browse, etc.; should be of the same kind] activity. Describe what you are doing. [Student shows pathway to start a new activity of the same kind.] 3. Now, show what you would do if you wanted to start a new [Write/Draw/Browse, etc.; should be a different activity than the current one] activity. [Student shows pathway to start a new activity different from the current activity.] Pathway 2: 1. Show how you would start a new activity. Describe what you are doing. [Student opens a new activity.] 2. Now, show what you would do if you wanted to go back to the activity you last worked on. Describe what you are doing. [Student shows pathway to resume a previous activity.] 3. Now, show what you would do if you wanted to start a new [Write/Draw/Browse, etc.; should be of the same kind] activity. Describe what you are doing.[Student shows pathway to start a new activity of the same kind.] 4. General questions If you could change anything about Sugar, what would make it easier for you to go back to a previous activity? What would make it easier for you to start a new activity? Is there anything else about Sugar that you would change? ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
