Re: [WikimediaMobile] [Wikimedia-search] Fwd: "Morelike" suggestions - the results are in!
This is fantastic work to see! For future work around this stuff, can I ask that you keep us in the loop? When you build a feature like this, users get more things - and search gets more queries, some that succeed and some but fail. But for this email at the tail-end of the test we wouldn't know this was happening, and it has the potential to mess with some of our core KPIs. On 29 July 2015 at 16:01, Dmitry Brant wrote: > moving to mobile-l, and cc Search & Discovery. > > -- Forwarded message -- > From: Dmitry Brant > Date: Wed, Jul 29, 2015 at 3:38 PM > Subject: "Morelike" suggestions - the results are in! > To: Internal communication for WMF Reading team > > > > Hi all, > > For the last few weeks, we've had an A/B test in the Android app where we > measure user engagement with the "read more" suggestions that we show at the > bottom of each article. We display three suggestions for further reading, > based on either (A) a plain full-text search query based on the title of the > current article, or (B) a query using the "morelike" feature in > CirrusSearch. > > And the winner is... (perhaps not entirely surprisingly) "morelike"! Users > who saw suggestions based on "morelike" were over 20% more likely to click > on one of the suggestions. > > Here's a quick analysis and chart of the data from the last 10 days: > > https://docs.google.com/spreadsheets/d/1BFsrAcPgexQyNVemmJ3k3IX5rtPvJ_5vdYOyGgS5R6Y/edit?usp=sharing > > > -Dmitry > > > > ___ > Wikimedia-search mailing list > wikimedia-sea...@lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/wikimedia-search > -- Oliver Keyes Research Analyst Wikimedia Foundation ___ Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Re: [WikimediaMobile] [Wikimedia-search] Fwd: "Morelike" suggestions - the results are in!
Will surely take note for next time. Thanks! Now that we've got the results from this test, we will soon update the app to use *only* morelike suggestions, which means you'll be getting twice as many morelike queries from the app as before. (We'll let you know when that's released) On Wed, Jul 29, 2015 at 4:17 PM, Oliver Keyes wrote: > This is fantastic work to see! > > For future work around this stuff, can I ask that you keep us in the > loop? When you build a feature like this, users get more things - and > search gets more queries, some that succeed and some but fail. But for > this email at the tail-end of the test we wouldn't know this was > happening, and it has the potential to mess with some of our core > KPIs. > > On 29 July 2015 at 16:01, Dmitry Brant wrote: > > moving to mobile-l, and cc Search & Discovery. > > > > -- Forwarded message -- > > From: Dmitry Brant > > Date: Wed, Jul 29, 2015 at 3:38 PM > > Subject: "Morelike" suggestions - the results are in! > > To: Internal communication for WMF Reading team > > > > > > > > Hi all, > > > > For the last few weeks, we've had an A/B test in the Android app where we > > measure user engagement with the "read more" suggestions that we show at > the > > bottom of each article. We display three suggestions for further reading, > > based on either (A) a plain full-text search query based on the title of > the > > current article, or (B) a query using the "morelike" feature in > > CirrusSearch. > > > > And the winner is... (perhaps not entirely surprisingly) "morelike"! > Users > > who saw suggestions based on "morelike" were over 20% more likely to > click > > on one of the suggestions. > > > > Here's a quick analysis and chart of the data from the last 10 days: > > > > > https://docs.google.com/spreadsheets/d/1BFsrAcPgexQyNVemmJ3k3IX5rtPvJ_5vdYOyGgS5R6Y/edit?usp=sharing > > > > > > -Dmitry > > > > > > > > ___ > > Wikimedia-search mailing list > > wikimedia-sea...@lists.wikimedia.org > > https://lists.wikimedia.org/mailman/listinfo/wikimedia-search > > > > > > -- > Oliver Keyes > Research Analyst > Wikimedia Foundation > > ___ > Mobile-l mailing list > Mobile-l@lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/mobile-l > ___ Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Re: [WikimediaMobile] [Wikimedia-search] Fwd: "Morelike" suggestions - the results are in!
I'm late to this, but great news! I just want to commend those involved for a great example of cross-pollination between the web team and the app team. This is all the more impressive, given that the backend service was designed for A by the growth team(?), web is using it for B, and the app is using it for C. Communication! -J On Wed, Jul 29, 2015 at 2:23 PM, Dmitry Brant wrote: > Will surely take note for next time. Thanks! > Now that we've got the results from this test, we will soon update the app > to use *only* morelike suggestions, which means you'll be getting twice as > many morelike queries from the app as before. (We'll let you know when > that's released) > > > On Wed, Jul 29, 2015 at 4:17 PM, Oliver Keyes > wrote: > >> This is fantastic work to see! >> >> For future work around this stuff, can I ask that you keep us in the >> loop? When you build a feature like this, users get more things - and >> search gets more queries, some that succeed and some but fail. But for >> this email at the tail-end of the test we wouldn't know this was >> happening, and it has the potential to mess with some of our core >> KPIs. >> >> On 29 July 2015 at 16:01, Dmitry Brant wrote: >> > moving to mobile-l, and cc Search & Discovery. >> > >> > -- Forwarded message -- >> > From: Dmitry Brant >> > Date: Wed, Jul 29, 2015 at 3:38 PM >> > Subject: "Morelike" suggestions - the results are in! >> > To: Internal communication for WMF Reading team >> > >> > >> > >> > Hi all, >> > >> > For the last few weeks, we've had an A/B test in the Android app where >> we >> > measure user engagement with the "read more" suggestions that we show >> at the >> > bottom of each article. We display three suggestions for further >> reading, >> > based on either (A) a plain full-text search query based on the title >> of the >> > current article, or (B) a query using the "morelike" feature in >> > CirrusSearch. >> > >> > And the winner is... (perhaps not entirely surprisingly) "morelike"! >> Users >> > who saw suggestions based on "morelike" were over 20% more likely to >> click >> > on one of the suggestions. >> > >> > Here's a quick analysis and chart of the data from the last 10 days: >> > >> > >> https://docs.google.com/spreadsheets/d/1BFsrAcPgexQyNVemmJ3k3IX5rtPvJ_5vdYOyGgS5R6Y/edit?usp=sharing >> > >> > >> > -Dmitry >> > >> > >> > >> > ___ >> > Wikimedia-search mailing list >> > wikimedia-sea...@lists.wikimedia.org >> > https://lists.wikimedia.org/mailman/listinfo/wikimedia-search >> > >> >> >> >> -- >> Oliver Keyes >> Research Analyst >> Wikimedia Foundation >> >> ___ >> Mobile-l mailing list >> Mobile-l@lists.wikimedia.org >> https://lists.wikimedia.org/mailman/listinfo/mobile-l >> > > > ___ > Mobile-l mailing list > Mobile-l@lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/mobile-l > > ___ Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Re: [WikimediaMobile] [Wikimedia-search] Fwd: "Morelike" suggestions - the results are in!
On 4 August 2015 at 14:22, Jon Katz wrote: > > given that the backend service was designed for A by the growth team(?) > *cough* Discovery! ;-) Dan -- Dan Garry Lead Product Manager, Discovery Wikimedia Foundation ___ Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
