Re: [WikimediaMobile] [reading-wmf] Browse Hypothesis Results

[S Page]

On Sun, Oct 11, 2015 at 11:32 AM, Jon Katz  wrote:

>
> [Most-relevant category tag display experiment] was intended to lie in
> contrast with our current category pages, which are alphabetical and not
> really intended for human browsing:
> https://en.wikipedia.org/wiki/Category:American_male_film_actors
>

A category page is more an index to search or skim. But we could experiment
with making them more engaging for human browsing. Just as the search
results dropdown in mobile shows a page thumbnail and description along
with each title, we could show the same thumbnail and description with
every title on a category page. It could be a button [Enhance title
display].

That would leave categories alphabetized instead of most relevant, but
perhaps more engaging.

Cheers,
--
=S Page  WMF Tech writer
___
Mobile-l mailing list
Mobile-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mobile-l

Re: [WikimediaMobile] [reading-wmf] Browse Hypothesis Results

[Jon Katz]

Thanks, Joaquin!

On Tue, Oct 13, 2015 at 4:32 AM, Joaquin Oltra Hernandez <
jhernan...@wikimedia.org> wrote:

> Thanks a lot for the detailed report Jon.
>
> I've parsed it and posted it to
> https://www.mediawiki.org/wiki/Reading/Web/Projects/Categories_Browse so
> that can keep it more accessible than the mailing list archive
> .
>
> Any help with formatting or text corrections would be appreciated.
>
>
> On Sun, Oct 11, 2015 at 8:32 PM, Jon Katz  wrote:
>
>> Hi Team,
>> I just wanted to update you on the results of something we internally
>> referred to as the '*browse' *prototype.
>> TLDR: as implemented the mobile 'browse by category' test did not drive
>> significant engagement.  In fact, as implemented, it seemed inferior to
>> blue links.  However, we started with a very rough and low-impact
>> prototype, so a few tweaks would give us more definitive results.
>>
>> Here is the doc from which I am pasting from below:
>> https://docs.google.com/document/d/1Mqw-awAcp01IcLhHPsHmWsqaAyK1l2-w_LMDtizyFQ4/edit?usp=sharing
>>
>> Questions/comments welcome!
>> Best,
>>
>> J
>>
>>
>> Browse Prototype Results
>>
>>
>> 
>>
>> Intro
>> 
>>
>> Process
>> 
>>
>> Results
>> 
>>
>> Blue links in general
>> 
>>
>> Category tags
>> 
>>
>> Conclusion and Next Steps
>> 
>>
>> Process
>> 
>>
>> Do people want to browse by categories?
>> 
>> 
>>
>>
>> Intro
>>
>> As outlined in this doc
>> ,
>> the concept is a tag that allows readers to navigate WP via categories that
>> are meaningful and populated in order of 'significance' (as determined by
>> user input).  The hypothesis:
>>
>>-
>>
>>users will want to navigate by category if there are fewer, more
>>meaningful categories per page and those category pages showed the most
>>‘notable’ members first.
>>
>> Again, see the full doc
>> 
>> to understand the premise.
>>
>> Process
>>
>> The first step was to validate: do users want to navigate via category?
>> So we built a very lightweight prototype on mobile web, en wikipedia
>> (stable, not beta) using hardcoded config variables, in the following
>> categories ( ~4000 pages).  Here we did not look into sub-categories
>> with one exception (see T94732 
>> for details).  There was also an error and 2 of the categories did not have
>> tags implemented (struck through, below)
>>
>> Category
>>
>> Pagecount
>>
>> NBA All Stars
>>
>> 400
>>
>> American Politicians
>>
>> 818
>>
>> Object-Oriented Programming Languages
>>
>> 164
>>
>> European States
>>
>> 24
>>
>> American Female Pop Singers
>>
>> 326
>>
>> American drama television series
>>
>> 1048
>>
>> Modern Painters
>>
>> 983
>>
>> Landmarks in San Francisco, California
>>
>> 270
>>
>>
>>
>> Here is how it appeared on the Alcatraz page
>>
>>
>> When the user clicked the tag, they were taken to a gather-like
>> collection based on manually estimated relevance
>>
>> (sorry cropped shot)
>>
>>
>>
>>
>> The category pages were designed to show the most relevant (as deemed by
>> me) to the broadest audience, first. Here is the ordering:
>> https://docs.google.com/spreadsheets/d/12xLXQsH1zcg6E8lDuSonumZNdBvfaBuHOS1a1TCASK4/edit#gid=0
>>
>> This was intended to lie in contrast with our current category pages,
>> which are alphabetical and not really intended for human browsing:
>> https://en.wikipedia.org/wiki/Category:American_male_film_actors
>>
>>
>> We primarily measured a few things:
>>
>>
>>-
>>
>>when a tag was seen by a user
>>-
>>
>>when a tag was clicked on by a user
>>-
>>
>>when a page in the new ‘category view’ was clicked on by a user
>>
>>
>> As a side effort, I looked to see if overall referrals from pages with
>> tags went up--this was a timed intervention rather than an a/b test and
>> given the click-thru on the tags, the impact 

Re: [WikimediaMobile] [reading-wmf] Browse Hypothesis Results

[Brian Gerstle]

Great experiment!  A couple questions/comments:

   1. The % clickthrough per category shows SF Landmarks at 120%. Is that
   correct, and if so, what does it mean?
   2. As a big believer in the power of categories as a driver for
   engagement, I would love to see more variations of this experiment w/
   different placements, in a feed, different categories, add'n of portals, as
   a FTUE, etc. (likely to have a great deal of overlap w/ cascade D: deep
   dive educational experience)
   3. Also loved the win/needs-improvement breakdown at the end

Again, nice work!

On Tue, Oct 13, 2015 at 11:23 AM, Jon Katz  wrote:

> Thanks, Joaquin!
>
> On Tue, Oct 13, 2015 at 4:32 AM, Joaquin Oltra Hernandez <
> jhernan...@wikimedia.org> wrote:
>
>> Thanks a lot for the detailed report Jon.
>>
>> I've parsed it and posted it to
>> https://www.mediawiki.org/wiki/Reading/Web/Projects/Categories_Browse so
>> that can keep it more accessible than the mailing list archive
>> 
>> .
>>
>> Any help with formatting or text corrections would be appreciated.
>>
>>
>> On Sun, Oct 11, 2015 at 8:32 PM, Jon Katz  wrote:
>>
>>> Hi Team,
>>> I just wanted to update you on the results of something we internally
>>> referred to as the '*browse' *prototype.
>>> TLDR: as implemented the mobile 'browse by category' test did not drive
>>> significant engagement.  In fact, as implemented, it seemed inferior to
>>> blue links.  However, we started with a very rough and low-impact
>>> prototype, so a few tweaks would give us more definitive results.
>>>
>>> Here is the doc from which I am pasting from below:
>>> https://docs.google.com/document/d/1Mqw-awAcp01IcLhHPsHmWsqaAyK1l2-w_LMDtizyFQ4/edit?usp=sharing
>>>
>>> Questions/comments welcome!
>>> Best,
>>>
>>> J
>>>
>>>
>>> Browse Prototype Results
>>>
>>>
>>> 
>>>
>>> Intro
>>> 
>>>
>>> Process
>>> 
>>>
>>> Results
>>> 
>>>
>>> Blue links in general
>>> 
>>>
>>> Category tags
>>> 
>>>
>>> Conclusion and Next Steps
>>> 
>>>
>>> Process
>>> 
>>>
>>> Do people want to browse by categories?
>>> 
>>> 
>>>
>>>
>>> Intro
>>>
>>> As outlined in this doc
>>> ,
>>> the concept is a tag that allows readers to navigate WP via categories that
>>> are meaningful and populated in order of 'significance' (as determined by
>>> user input).  The hypothesis:
>>>
>>>-
>>>
>>>users will want to navigate by category if there are fewer, more
>>>meaningful categories per page and those category pages showed the most
>>>‘notable’ members first.
>>>
>>> Again, see the full doc
>>> 
>>> to understand the premise.
>>>
>>> Process
>>>
>>> The first step was to validate: do users want to navigate via category?
>>> So we built a very lightweight prototype on mobile web, en wikipedia
>>> (stable, not beta) using hardcoded config variables, in the following
>>> categories ( ~4000 pages).  Here we did not look into sub-categories
>>> with one exception (see T94732
>>>  for details).  There was
>>> also an error and 2 of the categories did not have tags implemented (struck
>>> through, below)
>>>
>>> Category
>>>
>>> Pagecount
>>>
>>> NBA All Stars
>>>
>>> 400
>>>
>>> American Politicians
>>>
>>> 818
>>>
>>> Object-Oriented Programming Languages
>>>
>>> 164
>>>
>>> European States
>>>
>>> 24
>>>
>>> American Female Pop Singers
>>>
>>> 326
>>>
>>> American drama television series
>>>
>>> 1048
>>>
>>> Modern Painters
>>>
>>> 983
>>>
>>> Landmarks in San Francisco, California
>>>
>>> 270
>>>
>>>
>>>
>>> Here is how it appeared on the Alcatraz page
>>>
>>>
>>> When the user clicked the tag, they were taken to a gather-like
>>> collection based on manually estimated relevance
>>>
>>> (sorry cropped shot)
>>>
>>>
>>>
>>>
>>> The category pages were designed to show the most relevant (as deemed by
>>> me) 

Re: [WikimediaMobile] [reading-wmf] Browse Hypothesis Results

[Jon Robson]

Jon thanks so much for writing this up and thanks Joaquin for putting it on
the wiki (you beat me to it :))

I'm a little confused to the meaning of "tag impression" -  does it mean
they rendered or that the user saw them?

If the latter...
https://www.mediawiki.org/wiki/Reading/Web/Projects/Categories_Browse#/media/File:Browse-Views_and_clickthrough.png
suggests that 50% of the time a reader saw a tag (.45% of the time) (there
was a tag impression) they clicked on them (0.18%). Am I misunderstanding
what tag impression means?

Is this a fair summary?:
"When the tags were __visible__ to the user, 30-50% of the time they
clicked through to the "category page". On this page only 40% of visits
resulted in a click to an article in the list.

To me this is significant traffic... especially if made more visible.
I personally would expect normal link traffic to be higher and for this to
be a new source of engagement.

Can we clarify this on the wiki page and in this thread as I fear I'm
misunderstanding something..?

Other questions:
* What was the number of clicks to tags per visit? (were being opening new
tabs or clicking on one?)

On Tue, Oct 13, 2015 at 8:36 AM, Brian Gerstle 
wrote:

> Great experiment!  A couple questions/comments:
>
>1. The % clickthrough per category shows SF Landmarks at 120%. Is that
>correct, and if so, what does it mean?
>2. As a big believer in the power of categories as a driver for
>engagement, I would love to see more variations of this experiment w/
>different placements, in a feed, different categories, add'n of portals, as
>a FTUE, etc. (likely to have a great deal of overlap w/ cascade D: deep
>dive educational experience)
>3. Also loved the win/needs-improvement breakdown at the end
>
> Again, nice work!
>
> On Tue, Oct 13, 2015 at 11:23 AM, Jon Katz  wrote:
>
>> Thanks, Joaquin!
>>
>> On Tue, Oct 13, 2015 at 4:32 AM, Joaquin Oltra Hernandez <
>> jhernan...@wikimedia.org> wrote:
>>
>>> Thanks a lot for the detailed report Jon.
>>>
>>> I've parsed it and posted it to
>>> https://www.mediawiki.org/wiki/Reading/Web/Projects/Categories_Browse so
>>> that can keep it more accessible than the mailing list archive
>>> 
>>> .
>>>
>>> Any help with formatting or text corrections would be appreciated.
>>>
>>>
>>> On Sun, Oct 11, 2015 at 8:32 PM, Jon Katz  wrote:
>>>
 Hi Team,
 I just wanted to update you on the results of something we internally
 referred to as the '*browse' *prototype.
 TLDR: as implemented the mobile 'browse by category' test did not drive
 significant engagement.  In fact, as implemented, it seemed inferior to
 blue links.  However, we started with a very rough and low-impact
 prototype, so a few tweaks would give us more definitive results.

 Here is the doc from which I am pasting from below:
 https://docs.google.com/document/d/1Mqw-awAcp01IcLhHPsHmWsqaAyK1l2-w_LMDtizyFQ4/edit?usp=sharing

 Questions/comments welcome!
 Best,

 J


 Browse Prototype Results


 

 Intro
 

 Process
 

 Results
 

 Blue links in general
 

 Category tags
 

 Conclusion and Next Steps
 

 Process
 

 Do people want to browse by categories?
 
 


 Intro

 As outlined in this doc
 ,
 the concept is a tag that allows readers to navigate WP via categories that
 are meaningful and populated in order of 'significance' (as determined by
 user input).  The hypothesis:

-

users will want to navigate by category if there are fewer, more
meaningful categories per page and those category pages showed the most
‘notable’ members first.

 Again, see the full doc
 

Re: [WikimediaMobile] [reading-wmf] Browse Hypothesis Results

[Jon Katz]

Thanks for thinking through! Responses inline
On Tue, Oct 13, 2015 at 8:36 AM, Brian Gerstle 
wrote:

> Great experiment!  A couple questions/comments:
>
>1. The % clickthrough per category shows SF Landmarks at 120%. Is that
>correct, and if so, what does it mean?
>
> It means that for every 10 times the SF Landmarks list was visited from a
tag, there were 12 clicks on to other articles (this is possible because a
user can visit 2 or more articles from the page by clicking on the 'back'
button in their browser)

>
>1. As a big believer in the power of categories as a driver for
>engagement, I would love to see more variations of this experiment w/
>different placements, in a feed, different categories, add'n of portals, as
>a FTUE, etc. (likely to have a great deal of overlap w/ cascade D: deep
>dive educational experience)
>
> Me too!  it is very hard to learn anything from a first stab like this

>
>1. Also loved the win/needs-improvement breakdown at the end
>
> Thanks!


>
> On Tue, Oct 13, 2015 at 11:23 AM, Jon Katz  wrote:
>
>> Thanks, Joaquin!
>>
>> On Tue, Oct 13, 2015 at 4:32 AM, Joaquin Oltra Hernandez <
>> jhernan...@wikimedia.org> wrote:
>>
>>> Thanks a lot for the detailed report Jon.
>>>
>>> I've parsed it and posted it to
>>> https://www.mediawiki.org/wiki/Reading/Web/Projects/Categories_Browse so
>>> that can keep it more accessible than the mailing list archive
>>> 
>>> .
>>>
>>> Any help with formatting or text corrections would be appreciated.
>>>
>>>
>>> On Sun, Oct 11, 2015 at 8:32 PM, Jon Katz  wrote:
>>>
 Hi Team,
 I just wanted to update you on the results of something we internally
 referred to as the '*browse' *prototype.
 TLDR: as implemented the mobile 'browse by category' test did not drive
 significant engagement.  In fact, as implemented, it seemed inferior to
 blue links.  However, we started with a very rough and low-impact
 prototype, so a few tweaks would give us more definitive results.

 Here is the doc from which I am pasting from below:
 https://docs.google.com/document/d/1Mqw-awAcp01IcLhHPsHmWsqaAyK1l2-w_LMDtizyFQ4/edit?usp=sharing

 Questions/comments welcome!
 Best,

 J


 Browse Prototype Results


 

 Intro
 

 Process
 

 Results
 

 Blue links in general
 

 Category tags
 

 Conclusion and Next Steps
 

 Process
 

 Do people want to browse by categories?
 
 


 Intro

 As outlined in this doc
 ,
 the concept is a tag that allows readers to navigate WP via categories that
 are meaningful and populated in order of 'significance' (as determined by
 user input).  The hypothesis:

-

users will want to navigate by category if there are fewer, more
meaningful categories per page and those category pages showed the most
‘notable’ members first.

 Again, see the full doc
 
 to understand the premise.

 Process

 The first step was to validate: do users want to navigate via
 category?  So we built a very lightweight prototype on mobile web, en
 wikipedia (stable, not beta) using hardcoded config variables, in the
 following categories ( ~4000 pages).  Here we did not look into
 sub-categories with one exception (see T94732
  for details).  There was
 also an error and 2 of the categories did not have tags implemented (struck
 through, below)

 Category

 Pagecount

 NBA All Stars

 400

 American Politicians

 818

 Object-Oriented 

Re: [WikimediaMobile] [reading-wmf] Browse Hypothesis Results

[Jon Katz]

Thanks for thinking through this, Jon.  I think I made something
unclear--responses inline.  In green to make reading easier:

On Tue, Oct 13, 2015 at 8:58 AM, Jon Robson  wrote:

> Jon thanks so much for writing this up and thanks Joaquin for putting it
> on the wiki (you beat me to it :))
>
> I'm a little confused to the meaning of "tag impression" -  does it mean
> they rendered or that the user saw them?
>

User saw them, or, more specifically, appeared on screen.


> If the latter...
>
> https://www.mediawiki.org/wiki/Reading/Web/Projects/Categories_Browse#/media/File:Browse-Views_and_clickthrough.png
> suggests that 50% of the time a reader saw a tag (.45% of the time) (there
> was a tag impression) they clicked on them (0.18%). Am I misunderstanding
> what tag impression means?
>

Sorry, this graph is a little confusing because it has 2 axes.  The red
(impressions) corresponds to the right axis and the blue (% click-through)
is the left- axis.  So, the way to read it is that there were ~50,000
impressions a day, and the click-thru rate was .18%.  This means that for
every ~1000 times a tag appeared to a user, ~2 clicks occurred.  The
question of what % of pageviews resulted in an 'impression' is a really
good one - I think it could be easily looked up using the pageview_hourly
table.

>
> Is this a fair summary?:
> "When the tags were __visible__ to the user, 30-50% of the time they
> clicked through to the "category page". On this page only 40% of visits
> resulted in a click to an article in the list.
>
No--for reasons explained above.

>
> To me this is significant traffic... especially if made more visible.
> I personally would expect normal link traffic to be higher and for this to
> be a new source of engagement.
>
Yes, it would be hard to get higher click-through than our current
engagement model, but anything additive represents not only massive
numbers, but an opportunity to have our users increasingly think of us as a
place to learn (rather than a simple question and answer)


>
> Can we clarify this on the wiki page and in this thread as I fear I'm
> misunderstanding something..?
>

Yes--will add clarity to the graph on wiki.  It might be awhile, however,
before I have the brain and time to execute the query your question brought
up of "what % of total pages lead to the tag being shown"?  This will just
as easily be answered by the read more data you will be implementing.


>
> Other questions:
> * What was the number of clicks to tags per visit? (were being opening new
> tabs or clicking on one?)
>
I don't understand this question.  I know we can't track 'visits'.
___
Mobile-l mailing list
Mobile-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mobile-l