Re: [Sugar-devel] Test protocol for "Resume" vs. "Start new"
On Wed, Jan 27, 2010 at 13:17, Wade Brainerd wrote: > If possible, it might be helpful to record the students completing > this test, and then post the results to the wiki (as links to YouTube > etc). > > We video tape focus testing sessions all the time in the video game > industry. You just have to be careful about interpreting the > results, and it does take some time to go through watching them all. > But it reveals a lot of hesitations, failed attempts, etc. that don't > come out in a survey. > > Anyway, either way I agree that this test would be helpful in > determining the best home view behavior. This may be relevant: http://mairin.wordpress.com/2009/08/26/open-source-portable-usability-testing-lab/ http://mairin.wordpress.com/2009/10/17/open-source-portable-usability-testing-lab-part-2-the-parts/ Regards, Tomeu > Wade > > > On Mon, Jan 25, 2010 at 11:35 AM, Christian Marc Schmidt > wrote: >> Thanks Gary, I'll have a look at this. I'm worried that conducting tests >> with novice users might be inconclusive, given that we are testing >> schoolchildren. But I am also not an educator. I'd be happy to work with >> someone with an educational background to put something together that might >> be suitable as a test for new users. >> >> For now, though, I'd suggest we stick to our plan to see where it gets us. >> >> Christian >> On Mon, Jan 25, 2010 at 11:20 AM, Gary C Martin >> wrote: >>> >>> Hi Christian, >>> >>> On 23 Jan 2010, at 15:01, Christian Marc Schmidt wrote: >>> >>> > Hi all: >>> > >>> > For our meeting at 10:30pm EST today, I put together a draft of a test >>> > protocol (in email body below). Please look it over and feel free to edit >>> > or >>> > add to. Thanks, >>> > >>> > Christian >>> > >>> > -- >>> > Sugar Home View Concept Testing >>> > Draft 01/23/10 >>> > >>> > Goal: Evaluate the Home view for "Resume activity" and "Start new >>> > activity" functionality. >>> > Protocol: Alternate between two builds, one per student: The current >>> > build [Build A], which makes "Resume" the default action, and a test build >>> > [Build B] making "Start new" the default action. >>> > Duration: 5 mins per student >>> > Student Profile: Should be a current user (not a first-time user). Age >>> > and grade should be recorded, along with duration of overall Sugar >>> > experience. >>> > >>> > 1. Begin at Home screen after boot up >>> > >>> > 2. Test cognitive model >>> > • "Describe what you see in this view." >>> > • "What are the symbols in the ring?" >>> > • "Describe what each symbol in the ring is." >>> > • "Do you know why the symbols are different colors?" >>> > • "What activity did you last work on?" >>> > 3. Test pathways >>> > [Alternate between two pathways, one per student] >>> > >>> > Pathway 1: >>> > • "Show how you would go back to the activity you last worked on. >>> > Describe what you are doing." [Student shows pathway to resume activity.] >>> > • "Now, show what you would do if you wanted to start a new >>> > [Write/Draw/Browse, etc.; should be of the same kind] activity. Describe >>> > what you are doing." [Student shows pathway to start a new activity of the >>> > same kind.] >>> > • "Now, show what you would do if you wanted to start a new >>> > [Write/Draw/Browse, etc.; should be a different activity than the current >>> > one] activity." [Student shows pathway to start a new activity different >>> > from the current activity.] >>> > Pathway 2: >>> > • "Show how you would start a new activity. Describe what you are >>> > doing." [Student opens a new activity.] >>> > • "Now, show what you would do if you wanted to go back to the >>> > activity you last worked on. Describe what you are doing." [Student shows >>> > pathway to resume a previous activity.] >>> > • "Now, show what you would do if you wanted to start a new >>> > [Write/Draw/Browse, etc.; should be of the same kind] activity. Describe >>> > what you are doing."[Student shows pathway to start a new activity of the >>> > same kind.] >>> > 4. General questions >>> > "If you could change anything about Sugar, what would make it easier for >>> > you to go back to a previous activity?" >>> > "What would make it easier for you to start a new activity?" >>> > "Is there anything else about Sugar that you would change?" >>> >>> I know we've drifted from a usability study model, to a focus group model >>> for practical classroom reasons – but I though Jakob Nielson's latest >>> article was a timely piece worth a quick read through given we are going to >>> be getting feedback from existing Sugar users. >>> >>> "Summary: It's more difficult to conduct usability studies with >>> experienced users than with novices, and the improvements are usually >>> smaller. Still, improving expert performance is often worth the effort." >>> >>> http://www.useit.com/alertbox/experienced-users.html >>> >>> Regards, >>> --Gary >> >> >> -- >> anyth...@christianmarcsch
Re: [Sugar-devel] Test protocol for "Resume" vs. "Start new"
If possible, it might be helpful to record the students completing this test, and then post the results to the wiki (as links to YouTube etc). We video tape focus testing sessions all the time in the video game industry. You just have to be careful about interpreting the results, and it does take some time to go through watching them all. But it reveals a lot of hesitations, failed attempts, etc. that don't come out in a survey. Anyway, either way I agree that this test would be helpful in determining the best home view behavior. Wade On Mon, Jan 25, 2010 at 11:35 AM, Christian Marc Schmidt wrote: > Thanks Gary, I'll have a look at this. I'm worried that conducting tests > with novice users might be inconclusive, given that we are testing > schoolchildren. But I am also not an educator. I'd be happy to work with > someone with an educational background to put something together that might > be suitable as a test for new users. > > For now, though, I'd suggest we stick to our plan to see where it gets us. > > Christian > On Mon, Jan 25, 2010 at 11:20 AM, Gary C Martin > wrote: >> >> Hi Christian, >> >> On 23 Jan 2010, at 15:01, Christian Marc Schmidt wrote: >> >> > Hi all: >> > >> > For our meeting at 10:30pm EST today, I put together a draft of a test >> > protocol (in email body below). Please look it over and feel free to edit >> > or >> > add to. Thanks, >> > >> > Christian >> > >> > -- >> > Sugar Home View Concept Testing >> > Draft 01/23/10 >> > >> > Goal: Evaluate the Home view for "Resume activity" and "Start new >> > activity" functionality. >> > Protocol: Alternate between two builds, one per student: The current >> > build [Build A], which makes "Resume" the default action, and a test build >> > [Build B] making "Start new" the default action. >> > Duration: 5 mins per student >> > Student Profile: Should be a current user (not a first-time user). Age >> > and grade should be recorded, along with duration of overall Sugar >> > experience. >> > >> > 1. Begin at Home screen after boot up >> > >> > 2. Test cognitive model >> > • "Describe what you see in this view." >> > • "What are the symbols in the ring?" >> > • "Describe what each symbol in the ring is." >> > • "Do you know why the symbols are different colors?" >> > • "What activity did you last work on?" >> > 3. Test pathways >> > [Alternate between two pathways, one per student] >> > >> > Pathway 1: >> > • "Show how you would go back to the activity you last worked on. >> > Describe what you are doing." [Student shows pathway to resume activity.] >> > • "Now, show what you would do if you wanted to start a new >> > [Write/Draw/Browse, etc.; should be of the same kind] activity. Describe >> > what you are doing." [Student shows pathway to start a new activity of the >> > same kind.] >> > • "Now, show what you would do if you wanted to start a new >> > [Write/Draw/Browse, etc.; should be a different activity than the current >> > one] activity." [Student shows pathway to start a new activity different >> > from the current activity.] >> > Pathway 2: >> > • "Show how you would start a new activity. Describe what you are >> > doing." [Student opens a new activity.] >> > • "Now, show what you would do if you wanted to go back to the >> > activity you last worked on. Describe what you are doing." [Student shows >> > pathway to resume a previous activity.] >> > • "Now, show what you would do if you wanted to start a new >> > [Write/Draw/Browse, etc.; should be of the same kind] activity. Describe >> > what you are doing."[Student shows pathway to start a new activity of the >> > same kind.] >> > 4. General questions >> > "If you could change anything about Sugar, what would make it easier for >> > you to go back to a previous activity?" >> > "What would make it easier for you to start a new activity?" >> > "Is there anything else about Sugar that you would change?" >> >> I know we've drifted from a usability study model, to a focus group model >> for practical classroom reasons – but I though Jakob Nielson's latest >> article was a timely piece worth a quick read through given we are going to >> be getting feedback from existing Sugar users. >> >> "Summary: It's more difficult to conduct usability studies with >> experienced users than with novices, and the improvements are usually >> smaller. Still, improving expert performance is often worth the effort." >> >> http://www.useit.com/alertbox/experienced-users.html >> >> Regards, >> --Gary > > > -- > anyth...@christianmarcschmidt.com > 917/ 575 0013 > > http://www.christianmarcschmidt.com > http://www.linkedin.com/in/christianmarcschmidt > http://twitter.com/cms_ > > ___ > Sugar-devel mailing list > Sugar-devel@lists.sugarlabs.org > http://lists.sugarlabs.org/listinfo/sugar-devel > > ___ Sugar-devel mailing list Sugar-devel@lists.su
Re: [Sugar-devel] Test protocol for "Resume" vs. "Start new"
Thanks Gary, I'll have a look at this. I'm worried that conducting tests with novice users might be inconclusive, given that we are testing schoolchildren. But I am also not an educator. I'd be happy to work with someone with an educational background to put something together that might be suitable as a test for new users. For now, though, I'd suggest we stick to our plan to see where it gets us. Christian On Mon, Jan 25, 2010 at 11:20 AM, Gary C Martin wrote: > Hi Christian, > > On 23 Jan 2010, at 15:01, Christian Marc Schmidt wrote: > > > Hi all: > > > > For our meeting at 10:30pm EST today, I put together a draft of a test > protocol (in email body below). Please look it over and feel free to edit or > add to. Thanks, > > > > Christian > > > > -- > > Sugar Home View Concept Testing > > Draft 01/23/10 > > > > Goal: Evaluate the Home view for "Resume activity" and "Start new > activity" functionality. > > Protocol: Alternate between two builds, one per student: The current > build [Build A], which makes "Resume" the default action, and a test build > [Build B] making "Start new" the default action. > > Duration: 5 mins per student > > Student Profile: Should be a current user (not a first-time user). Age > and grade should be recorded, along with duration of overall Sugar > experience. > > > > 1. Begin at Home screen after boot up > > > > 2. Test cognitive model > > • "Describe what you see in this view." > > • "What are the symbols in the ring?" > > • "Describe what each symbol in the ring is." > > • "Do you know why the symbols are different colors?" > > • "What activity did you last work on?" > > 3. Test pathways > > [Alternate between two pathways, one per student] > > > > Pathway 1: > > • "Show how you would go back to the activity you last worked on. > Describe what you are doing." [Student shows pathway to resume activity.] > > • "Now, show what you would do if you wanted to start a new > [Write/Draw/Browse, etc.; should be of the same kind] activity. Describe > what you are doing." [Student shows pathway to start a new activity of the > same kind.] > > • "Now, show what you would do if you wanted to start a new > [Write/Draw/Browse, etc.; should be a different activity than the current > one] activity." [Student shows pathway to start a new activity different > from the current activity.] > > Pathway 2: > > • "Show how you would start a new activity. Describe what you are > doing." [Student opens a new activity.] > > • "Now, show what you would do if you wanted to go back to the > activity you last worked on. Describe what you are doing." [Student shows > pathway to resume a previous activity.] > > • "Now, show what you would do if you wanted to start a new > [Write/Draw/Browse, etc.; should be of the same kind] activity. Describe > what you are doing."[Student shows pathway to start a new activity of the > same kind.] > > 4. General questions > > "If you could change anything about Sugar, what would make it easier for > you to go back to a previous activity?" > > "What would make it easier for you to start a new activity?" > > "Is there anything else about Sugar that you would change?" > > I know we've drifted from a usability study model, to a focus group model > for practical classroom reasons – but I though Jakob Nielson's latest > article was a timely piece worth a quick read through given we are going to > be getting feedback from existing Sugar users. > > "Summary: It's more difficult to conduct usability studies with experienced > users than with novices, and the improvements are usually smaller. Still, > improving expert performance is often worth the effort." > >http://www.useit.com/alertbox/experienced-users.html > > Regards, > --Gary -- anyth...@christianmarcschmidt.com 917/ 575 0013 http://www.christianmarcschmidt.com http://www.linkedin.com/in/christianmarcschmidt http://twitter.com/cms_ ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Test protocol for "Resume" vs. "Start new"
Hi Christian, On 23 Jan 2010, at 15:01, Christian Marc Schmidt wrote: > Hi all: > > For our meeting at 10:30pm EST today, I put together a draft of a test > protocol (in email body below). Please look it over and feel free to edit or > add to. Thanks, > > Christian > > -- > Sugar Home View Concept Testing > Draft 01/23/10 > > Goal: Evaluate the Home view for "Resume activity" and "Start new activity" > functionality. > Protocol: Alternate between two builds, one per student: The current build > [Build A], which makes "Resume" the default action, and a test build [Build > B] making "Start new" the default action. > Duration: 5 mins per student > Student Profile: Should be a current user (not a first-time user). Age and > grade should be recorded, along with duration of overall Sugar experience. > > 1. Begin at Home screen after boot up > > 2. Test cognitive model > • "Describe what you see in this view." > • "What are the symbols in the ring?" > • "Describe what each symbol in the ring is." > • "Do you know why the symbols are different colors?" > • "What activity did you last work on?" > 3. Test pathways > [Alternate between two pathways, one per student] > > Pathway 1: > • "Show how you would go back to the activity you last worked on. > Describe what you are doing." [Student shows pathway to resume activity.] > • "Now, show what you would do if you wanted to start a new > [Write/Draw/Browse, etc.; should be of the same kind] activity. Describe what > you are doing." [Student shows pathway to start a new activity of the same > kind.] > • "Now, show what you would do if you wanted to start a new > [Write/Draw/Browse, etc.; should be a different activity than the current > one] activity." [Student shows pathway to start a new activity different from > the current activity.] > Pathway 2: > • "Show how you would start a new activity. Describe what you are > doing." [Student opens a new activity.] > • "Now, show what you would do if you wanted to go back to the activity > you last worked on. Describe what you are doing." [Student shows pathway to > resume a previous activity.] > • "Now, show what you would do if you wanted to start a new > [Write/Draw/Browse, etc.; should be of the same kind] activity. Describe what > you are doing."[Student shows pathway to start a new activity of the same > kind.] > 4. General questions > "If you could change anything about Sugar, what would make it easier for you > to go back to a previous activity?" > "What would make it easier for you to start a new activity?" > "Is there anything else about Sugar that you would change?" I know we've drifted from a usability study model, to a focus group model for practical classroom reasons – but I though Jakob Nielson's latest article was a timely piece worth a quick read through given we are going to be getting feedback from existing Sugar users. "Summary: It's more difficult to conduct usability studies with experienced users than with novices, and the improvements are usually smaller. Still, improving expert performance is often worth the effort." http://www.useit.com/alertbox/experienced-users.html Regards, --Gary ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
