But, we started. This motivated those of us who decided we had to do this the right way to actually run more user tests. Because we thought maybe we could overwhelm them with facts, and that will overcome this reliance on votes of committees. So, Chris Doerr, Wallace Judd -- I think Wallace maybe worked at Xerox before this, I can't remember -- these are two people that got involved with user testing the Filer, which we think of as the Finder today. And they ran some tests, and came up with all sorts of comments. I've underlined a couple here. Like there needs to be a way to select multiple icons at once. Well, they weren't icons yet even. Ways to manipulate multiple things at once.
But basically, after making some comments, they said, "You know, this is a good idea." Their own personal experience with testing was that you hired a testing firm, you spent a lot of money, three months later you came back with test results, and you did some things, and you did it again. You know, it took forever. Or, they would take a book and give it to a subject. Have the user read the manual, try out the manual, videotape the whole thing. And she said, "You know, this wasn't bad, just sitting down next to the user and having a dialog with the user, trying to get the user talking, and express what they're doing, is not a bad way to do it." So that was good, that we got the people doing the user testing to adopt this much simpler and cheaper paradigm. We tape-recorded the sessions, and they found that useful. They went back and listened to the tapes. However, the tapes were hard to hear, because the people in the room were making noise. So they asked the observers to be quieter. But one problem, she said, was that we're only testing Apple employees. We would get Apple employees the day they took the job, the day they started work. We'd get them in orientation. The person running orientation would say, "Well, after orientation you can go straight to work, or you can go be a user test subject, and, hey, you can get out of work for a few more hours," so we got a lot of people to come over and be user test subjects. Almost all of them had very little computer experience, because although Apple would have liked to hire people with computer experience in those days, there weren't very many. Here's a typical schedule. We weren't trying to do two an hour any more. One person at a time. One each day. Nine o'clock, one o'clock, three o'clock, much more civilized. And we had a checklist that the tester went through, before each session: Have you reminded the subject of whatever -- I've no idea, whatever they're supposed to remember, that's on the list. Is there coffee available? Do you have a blank questionnaire ready for the person? Because we had them fill out a questionnaire at the end. Do you have the checklist ready? Have you rewound the tape? Have we rewound the tape for this, the talk today? Have you rebooted? Because it crashed if you didn't. Are the cue cards in the testing room, etcetera. After each session did you cover up the Lisa again, because the room had a window, and at night you could see in the window, and see the secret Lisa machine. Did you close the blinds, for a double measure? Etcetera, etcetera. So we had a lot of things to remind the tester about. That's a good idea if you ever run user tests, have a checklist. Now Wallace, the guy who received the last memo, wrote this memo about some testing results. He said he spent about an hour a subject, which was pretty typical. That's the way I like to do testing myself. And he said, "Well, some functions still unclear; in particular, seven subjects out of" -- how many? It doesn't say, but not a lot, not very many -- "most of the subjects had trouble with Undo Last Command. They'd never seen such a thing in a program before, and they just didn't quite understand what it meant." Then there were other ones they understood, but they didn't feel comfortable with somehow. One was Print a Copy, one was Mail a Copy. Well, I believe the reason was that we didn't have printers yet on the machine, and we didn't have e-mail. In fact, we never had e-mail. [Laughter] So, we're probably just showing these, and not really -- couldn't use them, so of course they didn't understand them. Graphics editor user testing. This was LisaDraw, which became Mac Draw later, implemented by the same guy actually, Mark Cutter. And we wanted to test various things. Is this confusing, is that confusing? What happens if you try to use rulers at the same time as cross hairs? Is that too many things on the screen? Considering all the widgets that are there today in a Photo Shop or something, why were we worried? There was even worry about whether people could use a graphics program, because most people can't draw. So they wouldn't use a graphics program. We also wanted to know the error rates. And so a lot of counting was being done, and looking at tapes to see how many times people made errors, and try to reduce the error rate. So we were trying to be sort of scientific, even though we were on this headlong schedule. You may have noticed that six months have already gone by and we haven't shipped. In fact, it's about a year at this point. Okay, so June '81, Pat Marriott writes this memo about validating the prototypes, and Trip Hawkins at this point is at his wit's end. He says, "Aren't we done with all these decisions yet? Let's get on with it! Why are we still talking?" Here are some of the issues. Should the arrows point apart, away from each other, or towards each other? Because, after all, the text moves in the direction that's opposite of the arrow direction, we could flip the arrows and make them point to each other. And I wrote a comment here, "Ahh, we can wait until September to test this. No hurry." I wasn't in synch with Product Marketing, who wanted all the decisions to be made, because I knew that changing this was a matter of about three minutes of programming. So what was the hurry? I'd rather work or harder issues, like getting the software to fit in memory, or something. And there were some questions about clicking in folders -- oh, by the way, when you see folder here, we used to call windows folders. I forgot to mention that earlier; it might have been confusing here. Every window is a folder, even a document window was a folder. Not just the ones on the desktop. And we wanted to do some tests on that. [Inaudible comments from audience.] Okay, October '81. We took Chris's advice to heart -- Chris Doerr's -- and we did outside testing. 24 people who did not work for Apple were done in a 1-way mirror room in Daly City. They didn't know what company it was that was showing it. A blind test. The psychologists had their day. We got to do this right. This was the thing that, there was no way we were going to do this, too expensive, takes too much time, but by this point Product Marketing was going, "Well, maybe we'd better test this on real people." We wanted to know whether they could read the manual, and install the hardware, and get the thing running, by themselves. And it turned out that it was 10 minutes, on average, that a person could, from the time they opened the box, until the time they get set up and running something. We thought that was pretty good. Those of you who were at Apple later know that on the Mac, it was even easier, at the beginning, but as the Mac got more and more developed, that number went up. And when that number started getting about an hour, Jean-Louis Gassee got extremely upset, and he ordered new user tests, and they ran user tests about that, and got the time back down to 10 minutes again. But there were some bad years in there. It was very hard to open a Mac box and figure how to set it up. Two thirds of the users say they felt comfortable using the product as a result of the first experience. And about half of the people who were -- half of the third that were uncomfortable were people who happened to run into crashes and so on. Which probably affected -- they were trying to beat the clock on setting up the system. It was kind of frustrating. And we had a questionnaire they had to fill out. This is the result of the questionnaire, so those numbers are how many people had that answer. And they didn't think it would be that difficult to set up. Before, they started ... [few words missing from tape] ... after they did it, they found that it was very easy. -- LisaList is sponsored by <http://lowendmac.com/> and... Shop buy.com and save. <http://lowendmac.com/ad/buy.com.html> Support Low End Mac <http://lowendmac.com/lists/support.html> LisaList info: <http://lowendmac.com/lists/lisa.html> --> AOL users, remove "mailto:"; Send list messages to: <mailto:lisalist@mail.maclaunch.com> To unsubscribe, email: <mailto:[EMAIL PROTECTED]> For digest mode, email: <mailto:[EMAIL PROTECTED]> Subscription questions: <mailto:[EMAIL PROTECTED]> Archive: <http://www.mail-archive.com/lisalist%40mail.maclaunch.com/> iPod Accessories for Less at 1-800-iPOD.COM Fast Delivery, Low Price, Good Deal www.1800ipod.com
