Will we have any chance to ship both, and randomly select (at startup
time??) between the two dictionaries? Alternatively, could we ship a series
of dev builds, and alternate use of old an new dictionaries.
The bottom line IMO is that when running experiments, you need the closest
to apples-apples comparisons. Comparisons across different user groups
(beta vs stable?), and different time periods (last months, vs next month),
tend to include too much noise. Also note that experiments often have "long
lasting impact," by which I mean that IF the dictionary is "problematic,"
then users will proceed into the future to avoid using it, and this will
persist even after we correct the problems.
Do you have any way of supporting and (without asking the user) providing
new vs old dictionaries?
Thanks,
Jim
2009/10/28 Evan Martin <e...@chromium.org>
> 2009/10/28 Hironori Bono (坊野 博典) <hb...@chromium.org>:
> > Even though this is still a random thought, I would personally like to
> > use chromium to evaluate the new dictionaries: i.e. uploading the new
> > dictionaries to our dictionary server, changing the chromium code to
> > use the updated ones, asking users to compare the new dictionaries
> > with the old ones and give us their feedbacks (*1). If users like the
> > new dictionaries, we would like to release the new ones. Otherwise we
> > will keep the old ones.
>
> In the web search world when we made changes like these, we'd try to
> measure it without users giving explicit feedback. For example: give
> some users the new dictionary, and others the old one. Log which
> entry index from the dictionary suggestion list people are frequently
> choosing, then compare the aggregate counts between the two sets of
> users. (Maybe call the "Add to Dictionary" menu option -1.) For a
> better dictionary, I'd expect people to use the earlier entries from
> the suggestion list more frequently, and the "Add to Dictionary"
> option less frequently.
>
> This can be done with our existing histogram framework and I believe
> Jim wrote a framework within that for doing these sorts of
> experiments.
>
> It still might be worth soliciting feedback from users directly. For
> example, if the new dictionary is missing a common word the above
> measure would get a high count of "Add to Dictionary", and maybe users
> could tell us about this. But in general, you get higher quality data
> when you involve more users, and a spreadsheet will be limited to
> people who understand the English instructions.
>
--~--~---------~--~----~------------~-------~--~----~
Chromium Developers mailing list: chromium-dev@googlegroups.com
View archives, change email options, or unsubscribe:
http://groups.google.com/group/chromium-dev
-~----------~----~----~----~------~----~------~--~---
