Hello Marat, You were asking about using a numerical scale, eg Likert, for users to assess CDs.
I think the short answer is, Try it - and don't forget to tell us what happens. If you want a longer answer, though, I have to say that there seem to be problems. (That's not to say they can't solved, or that you shouldn't have a go.) The major problem is that when a user says "this system is very viscous" (or whatever), their assessment will depend on their mental model. So, for exmaple, changing the spelling of all instances of US 'center' to British 'centre' would quite likely be seen as a single mental operation, and if the user had to do each one individually it would count as high viscosity: whereas changing every _second_ instance would probably strike the user as pretty weird, and it's much less likely that the user would take that into account in assessing viscosity. Nevertheless one can imagine situations where one might want to do something like that. Yes, I know it sounds implausible - but on one occasion I had to produce a passage in which three characters were speaking at once and this was shown by alternating words and phrases, in different type faces, and I found that my word processor was quite unequal to the task. For me, that was a high viscosity moment. So the assessment of viscosity is not just a measure of the system in isolation: it depends on the match between what the system offers and what the user would like. And with fast-evolving technologies, such as mobile phones and video, we can observe a continual shift in what is offered, as users become more experienced and demand more of their system. A further problem is that a system may be very uneven - it might be very easy to use in most respects, but really difficult in one other respect. Some users may give prominence to that one source of difficulty, while others may give prominence to the more numerous easy bits. You could probably get some useful information by asking your respondents to give illustrations as well as numbers, so that you can see whether different users are giving prominence to the same issues. Of course these difficulties are by no means unique to questions about CDs. Psychometrics has a long history of grappling with very similar problems, as I expect you know, and there are various techniques for assessing reliability of questionnaires which you could deploy (Cronbach's Alpha is popular). Validity is harder. I hope that helps with the main question. Do say if it doesn't. On your subsidiary question, whether to go notation-specific or not, I'm simply not sure. Maybe someone else has some ideas? Whatever you do, do please give feedback. These questions are relevant to many others, and a body of experience is what we need. Finally, I had a swift look at your website. It looks really interesting. Good luck, Thomas Green Marat Boshernitsan said: >I have a system that I developed using the Cognitive Dimensions >framework to guide the design of the interface and of the interaction. >I am now performing a user evaluation of this systems and I would like >to include a Cognitive Dimensions questionnaire. I have found a few >descriptions of such user-based CD evaluation in the CD bibliography >(http://www.cl.cam.ac.uk/~afb21/CognitiveDimensions/CDbibliography.html). > >At least one of the previous evaluations (Clarke 2001) notes that it is >difficult to generate numerical data from the user's responses. So my >question is whether it makes sense to use a numerical scale (e.g. Likert >scale) instead of (or in addition to) free-form responses? What about >using a notation-neutral questionnaire (Blackwell & Green 2000) vs. a >notation-specific one, such as the one in Kadoda (2000)? Would one make >more sense with a numerical scale than the other? > >Thanks! > >Marat. > > >---------------------------------------------------------------------- >PPIG Discuss List ([email protected]) >Discuss admin: http://limitlessmail.net/mailman/listinfo/discuss >Announce admin: http://limitlessmail.net/mailman/listinfo/announce >PPIG Discuss archive: http://www.mail-archive.com/discuss%40ppig.org/ 27 Allerton Park, Leeds LS7 4ND +44-(0)-113-226-6687 http://homepage.ntlworld.com/greenery/ ---------------------------------------------------------------------- PPIG Discuss List ([email protected]) Discuss admin: http://limitlessmail.net/mailman/listinfo/discuss Announce admin: http://limitlessmail.net/mailman/listinfo/announce PPIG Discuss archive: http://www.mail-archive.com/discuss%40ppig.org/
