Re: [WikimediaMobile] Results from similar articles A/B test

2017-05-04 Thread Corey Floyd 
+ JMo

On Fri, Feb 26, 2016 at 4:19 AM David Causse  wrote:

> Thanks!
>
> yes this is not exactly what we expected :(
> I guess it was too good to be true: reduce latency and improve quality at
> the same time :)
> On our side I'd say that perf was the main issue, Erik added a cache at
> the backend-end level which seems to have a good impact.
> Morelike queries are still routed to the new datacenter in dallas to
> reduce stress on eqiad. We could maybe try to reroute them to eqiad and see
> if caching is sufficient?
> If it's the case I'd say that we don't need to run any A/B test.
>
> Random questions:
> Is it possible to analyze the correlation between the chosen article and
> the presence of an image?
> Rescoring options are slightly different for enwiki, is it possible to
> have the detail for e.g. enwiki/frwiki/dewiki?
>
> If it does not require huge effort on your side I'd say that you could run
> another A/B test by disabling boostLinks, you just have to add
> cirrusBoostLinks=no to api URL.
>
> Thank you
>
>
> Le 26/02/2016 00:33, Erik Bernhardson a écrit :
>
> ouch,  that is not at all the result we were hoping for. Just goes to show
> why we have to test these things and not just take a few examples that
> perform badly in one set and look to do a better job with some different
> options. Thanks for putting this together!
>
>
> On Thu, Feb 25, 2016 at 3:06 PM, Dmitry Brant 
> wrote:
>
>> Hello all,
>>
>> As mentioned previously, the current version of the Android app contains
>> an A/B test where it presents "read more" suggestions to the user, based on
>> (a) the standard "morelike" query, or (b) the new "opening_text" query.
>>
>> Here are the results from the last ~10 days of the test[0]:
>> - The clickthrough rate using the default morelike query is (and has
>> been) around 15%.
>> - With the new opening_text query, the clickthrough rate decreases to
>> about 12%:
>>
>>
>>
>> Therefore, it seems that the new query has a nontrivial negative effect
>> on CTR :(
>> We'll plan on removing this test in the next release of the app, but
>> we'll be happy to plug in a different or updated query, if it will be of
>> further use to Discovery.
>>
>>
>> [0]
>> https://docs.google.com/a/wikimedia.org/spreadsheets/d/1BFsrAcPgexQyNVemmJ3k3IX5rtPvJ_5vdYOyGgS5R6Y/edit?usp=sharing
>> (queries embedded as comments in the headers)
>>
>> --
>> Dmitry Brant
>> Senior Software Engineer / Product Owner (Android)
>> Wikimedia Foundation
>> https://www.mediawiki.org/wiki/Wikimedia_mobile_engineering
>>
>>
>> ___
>> Mobile-l mailing list
>> Mobile-l@lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/mobile-l
>>
>>
>
>
> ___
> Mobile-l mailing 
> listMobile-l@lists.wikimedia.orghttps://lists.wikimedia.org/mailman/listinfo/mobile-l
>
>
> ___
> Mobile-l mailing list
> Mobile-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/mobile-l
>
-- 
Corey Floyd
Engineering Manager
Reading
Wikimedia Foundation
cfl...@wikimedia.org
___
Mobile-l mailing list
Mobile-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mobile-l

Re: [WikimediaMobile] Results from similar articles A/B test

2016-02-26 Thread David Causse 

Thanks!

yes this is not exactly what we expected :(
I guess it was too good to be true: reduce latency and improve quality 
at the same time :)
On our side I'd say that perf was the main issue, Erik added a cache at 
the backend-end level which seems to have a good impact.
Morelike queries are still routed to the new datacenter in dallas to 
reduce stress on eqiad. We could maybe try to reroute them to eqiad and 
see if caching is sufficient?

If it's the case I'd say that we don't need to run any A/B test.

Random questions:
Is it possible to analyze the correlation between the chosen article and 
the presence of an image?
Rescoring options are slightly different for enwiki, is it possible to 
have the detail for e.g. enwiki/frwiki/dewiki?


If it does not require huge effort on your side I'd say that you could 
run another A/B test by disabling boostLinks, you just have to add 
cirrusBoostLinks=no to api URL.


Thank you

Le 26/02/2016 00:33, Erik Bernhardson a écrit :
ouch,  that is not at all the result we were hoping for. Just goes to 
show why we have to test these things and not just take a few examples 
that perform badly in one set and look to do a better job with some 
different options. Thanks for putting this together!



On Thu, Feb 25, 2016 at 3:06 PM, Dmitry Brant > wrote:


Hello all,

As mentioned previously, the current version of the Android app
contains an A/B test where it presents "read more" suggestions to
the user, based on (a) the standard "morelike" query, or (b) the
new "opening_text" query.

Here are the results from the last ~10 days of the test[0]:
- The clickthrough rate using the default morelike query is (and
has been) around 15%.
- With the new opening_text query, the clickthrough rate decreases
to about 12%:

Inline image 1

Therefore, it seems that the new query has a nontrivial negative
effect on CTR :(
We'll plan on removing this test in the next release of the app,
but we'll be happy to plug in a different or updated query, if it
will be of further use to Discovery.


[0]

https://docs.google.com/a/wikimedia.org/spreadsheets/d/1BFsrAcPgexQyNVemmJ3k3IX5rtPvJ_5vdYOyGgS5R6Y/edit?usp=sharing
(queries embedded as comments in the headers)

-- 
Dmitry Brant

Senior Software Engineer / Product Owner (Android)
Wikimedia Foundation
https://www.mediawiki.org/wiki/Wikimedia_mobile_engineering


___
Mobile-l mailing list
Mobile-l@lists.wikimedia.org 
https://lists.wikimedia.org/mailman/listinfo/mobile-l




___
Mobile-l mailing list
Mobile-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mobile-l


___
Mobile-l mailing list
Mobile-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mobile-l

Re: [WikimediaMobile] Results from similar articles A/B test

2016-02-25 Thread Erik Bernhardson 
ouch,  that is not at all the result we were hoping for. Just goes to show
why we have to test these things and not just take a few examples that
perform badly in one set and look to do a better job with some different
options. Thanks for putting this together!


On Thu, Feb 25, 2016 at 3:06 PM, Dmitry Brant  wrote:

> Hello all,
>
> As mentioned previously, the current version of the Android app contains
> an A/B test where it presents "read more" suggestions to the user, based on
> (a) the standard "morelike" query, or (b) the new "opening_text" query.
>
> Here are the results from the last ~10 days of the test[0]:
> - The clickthrough rate using the default morelike query is (and has been)
> around 15%.
> - With the new opening_text query, the clickthrough rate decreases to
> about 12%:
>
> [image: Inline image 1]
>
> Therefore, it seems that the new query has a nontrivial negative effect on
> CTR :(
> We'll plan on removing this test in the next release of the app, but we'll
> be happy to plug in a different or updated query, if it will be of further
> use to Discovery.
>
>
> [0]
> https://docs.google.com/a/wikimedia.org/spreadsheets/d/1BFsrAcPgexQyNVemmJ3k3IX5rtPvJ_5vdYOyGgS5R6Y/edit?usp=sharing
> (queries embedded as comments in the headers)
>
> --
> Dmitry Brant
> Senior Software Engineer / Product Owner (Android)
> Wikimedia Foundation
> https://www.mediawiki.org/wiki/Wikimedia_mobile_engineering
>
>
> ___
> Mobile-l mailing list
> Mobile-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/mobile-l
>
>
___
Mobile-l mailing list
Mobile-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mobile-l

[WikimediaMobile] Results from similar articles A/B test

2016-02-25 Thread Dmitry Brant 
Hello all,

As mentioned previously, the current version of the Android app contains an
A/B test where it presents "read more" suggestions to the user, based on
(a) the standard "morelike" query, or (b) the new "opening_text" query.

Here are the results from the last ~10 days of the test[0]:
- The clickthrough rate using the default morelike query is (and has been)
around 15%.
- With the new opening_text query, the clickthrough rate decreases to about
12%:

[image: Inline image 1]

Therefore, it seems that the new query has a nontrivial negative effect on
CTR :(
We'll plan on removing this test in the next release of the app, but we'll
be happy to plug in a different or updated query, if it will be of further
use to Discovery.


[0]
https://docs.google.com/a/wikimedia.org/spreadsheets/d/1BFsrAcPgexQyNVemmJ3k3IX5rtPvJ_5vdYOyGgS5R6Y/edit?usp=sharing
(queries embedded as comments in the headers)

-- 
Dmitry Brant
Senior Software Engineer / Product Owner (Android)
Wikimedia Foundation
https://www.mediawiki.org/wiki/Wikimedia_mobile_engineering
___
Mobile-l mailing list
Mobile-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mobile-l