Re: [WikimediaMobile] [Apps] How good is our search? Here's some data!
I noticed many bilingual friends search in their native language on English Wikipedia. It would be interesting to look closely at search terms and see if they are mispellings, different languages or just our search failing them. On Wed, Dec 3, 2014 at 11:01 PM, Amir E. Aharoni amir.ahar...@mail.huji.ac.il wrote: Thanks. This is not directly related, but some time I'd love to see data about interlanguage links tapping on all media - iOS, Android, MobileFrontend. -- Amir Elisha Aharoni · אָמִיר אֱלִישָׁע אַהֲרוֹנִי http://aharoni.wordpress.com “We're living in pieces, I want to live in peace.” – T. Moore 2014-12-04 4:15 GMT+02:00 Dan Garry dga...@wikimedia.org: We've now got some preliminary data in about how effective our search is in mobile apps. Note that this is only half a day's data, and it's sampled at 1%. High-level metrics: Number of searching sessions which end by tapping on a result: 91% Percentage of queries that give 0 results: 19% Mean queries per searching session: 3.8 Median user-perceived time taken to retrieve search results: 486ms 90th percentile of user-perceived time taken: 898ms My quick take-homes from this: 91% clickthrough seems pretty good. 19% of queries giving no results seems really bad. The combination of the above two means users are generally finding what they need, but that it's a struggle to do it. User-perceived performance seems pretty good; 90% of our users get search results within a second! What I think our next steps are: Push out our new search improvements and see how these baselines change. Try to tackle the 0 results problem, which seems to be more of a problem than clickthrough. Spreadsheet containing my queries and processed data: https://docs.google.com/a/wikimedia.org/spreadsheets/d/1syLgNygAS7Prxxg7RTIvK3vxwyVG-ZkjxtK0LSbU76w/edit Dan -- Dan Garry Associate Product Manager, Mobile Apps Wikimedia Foundation ___ Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l ___ Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l ___ Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
[WikimediaMobile] [Apps] How good is our search? Here's some data!
We've now got some preliminary data in about how effective our search is in mobile apps. Note that this is only half a day's data, and it's sampled at 1%. High-level metrics: - *Number of searching sessions which end by tapping on a result:* 91% - *Percentage of queries that give 0 results: *19% - *Mean queries per searching session:* 3.8 - *Median user-perceived time taken to retrieve search results: *486ms - *90th percentile of user-perceived time taken:* 898ms My quick take-homes from this: - 91% clickthrough seems pretty good. - 19% of queries giving no results seems really bad. - The combination of the above two means users are generally finding what they need, but that it's a struggle to do it. - User-perceived performance seems pretty good; 90% of our users get search results within a second! What I think our next steps are: - Push out our new search improvements and see how these baselines change. - Try to tackle the 0 results problem, which seems to be more of a problem than clickthrough. Spreadsheet containing my queries and processed data: https://docs.google.com/a/wikimedia.org/spreadsheets/d/1syLgNygAS7Prxxg7RTIvK3vxwyVG-ZkjxtK0LSbU76w/edit Dan -- Dan Garry Associate Product Manager, Mobile Apps Wikimedia Foundation ___ Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Re: [WikimediaMobile] [Apps] How good is our search? Here's some data!
Since presently all app searches are prefix, I'm not too surprised by 19% - I seem to misspell at least that percentage typing on phone screens, and most misspellings result in zero prefix results. But yes, really excited about supplementing small/null prefix results sets with full text results as suggested by Jared. We'll see that 19% really drop. On Dec 3, 2014, at 6:27 PM, Tomasz Finc tf...@wikimedia.org wrote: On Wed, Dec 3, 2014 at 6:15 PM, Dan Garry dga...@wikimedia.org wrote: 19% of queries giving no results seems really bad. Fixing this needs to be our priority across all projects. It doesn't matter what we fix later down the pipeline if this number is as high as it is. Let's get this as a daily/hourly number so that we can see live stats and then discuss changes/improvements. Great work getting these --tomasz ___ Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l ___ Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Re: [WikimediaMobile] [Apps] How good is our search? Here's some data!
Nemo, Prefix search doesn't offer suggestions. So, 0 then. Bernd On Wed, Dec 3, 2014 at 11:28 PM, Federico Leva (Nemo) nemow...@gmail.com wrote: Dan Garry, 04/12/2014 03:15: # *Percentage of queries that give 0 results: *19% How many of these offered a did you mean alternative? Can typos on mobile increase the error rate? Nemo P.s.: Someone add this number next to the low-hanging fruit hypothesis claims. ;-) ___ Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l ___ Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Re: [WikimediaMobile] [Apps] How good is our search? Here's some data!
Thanks. This is not directly related, but some time I'd love to see data about interlanguage links tapping on all media - iOS, Android, MobileFrontend. -- Amir Elisha Aharoni · אָמִיר אֱלִישָׁע אַהֲרוֹנִי http://aharoni.wordpress.com “We're living in pieces, I want to live in peace.” – T. Moore 2014-12-04 4:15 GMT+02:00 Dan Garry dga...@wikimedia.org: We've now got some preliminary data in about how effective our search is in mobile apps. Note that this is only half a day's data, and it's sampled at 1%. High-level metrics: - *Number of searching sessions which end by tapping on a result:* 91% - *Percentage of queries that give 0 results: *19% - *Mean queries per searching session:* 3.8 - *Median user-perceived time taken to retrieve search results: *486ms - *90th percentile of user-perceived time taken:* 898ms My quick take-homes from this: - 91% clickthrough seems pretty good. - 19% of queries giving no results seems really bad. - The combination of the above two means users are generally finding what they need, but that it's a struggle to do it. - User-perceived performance seems pretty good; 90% of our users get search results within a second! What I think our next steps are: - Push out our new search improvements and see how these baselines change. - Try to tackle the 0 results problem, which seems to be more of a problem than clickthrough. Spreadsheet containing my queries and processed data: https://docs.google.com/a/wikimedia.org/spreadsheets/d/1syLgNygAS7Prxxg7RTIvK3vxwyVG-ZkjxtK0LSbU76w/edit Dan -- Dan Garry Associate Product Manager, Mobile Apps Wikimedia Foundation ___ Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l ___ Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
