Jon thanks so much for writing this up and thanks Joaquin for putting it on the wiki (you beat me to it :))
I'm a little confused to the meaning of "tag impression" - does it mean they rendered or that the user saw them? If the latter... https://www.mediawiki.org/wiki/Reading/Web/Projects/Categories_Browse#/media/File:Browse-Views_and_clickthrough.png suggests that 50% of the time a reader saw a tag (.45% of the time) (there was a tag impression) they clicked on them (0.18%). Am I misunderstanding what tag impression means? Is this a fair summary?: "When the tags were __visible__ to the user, 30-50% of the time they clicked through to the "category page". On this page only 40% of visits resulted in a click to an article in the list. To me this is significant traffic... especially if made more visible. I personally would expect normal link traffic to be higher and for this to be a new source of engagement. Can we clarify this on the wiki page and in this thread as I fear I'm misunderstanding something..? Other questions: * What was the number of clicks to tags per visit? (were being opening new tabs or clicking on one?) On Tue, Oct 13, 2015 at 8:36 AM, Brian Gerstle <bgers...@wikimedia.org> wrote: > Great experiment! A couple questions/comments: > > 1. The % clickthrough per category shows SF Landmarks at 120%. Is that > correct, and if so, what does it mean? > 2. As a big believer in the power of categories as a driver for > engagement, I would love to see more variations of this experiment w/ > different placements, in a feed, different categories, add'n of portals, as > a FTUE, etc. (likely to have a great deal of overlap w/ cascade D: deep > dive educational experience) > 3. Also loved the win/needs-improvement breakdown at the end > > Again, nice work! > > On Tue, Oct 13, 2015 at 11:23 AM, Jon Katz <jk...@wikimedia.org> wrote: > >> Thanks, Joaquin! >> >> On Tue, Oct 13, 2015 at 4:32 AM, Joaquin Oltra Hernandez < >> jhernan...@wikimedia.org> wrote: >> >>> Thanks a lot for the detailed report Jon. >>> >>> I've parsed it and posted it to >>> https://www.mediawiki.org/wiki/Reading/Web/Projects/Categories_Browse so >>> that can keep it more accessible than the mailing list archive >>> <https://lists.wikimedia.org/pipermail/mobile-l/2015-October/009827.html> >>> . >>> >>> Any help with formatting or text corrections would be appreciated. >>> >>> >>> On Sun, Oct 11, 2015 at 8:32 PM, Jon Katz <jk...@wikimedia.org> wrote: >>> >>>> Hi Team, >>>> I just wanted to update you on the results of something we internally >>>> referred to as the '*browse' *prototype. >>>> TLDR: as implemented the mobile 'browse by category' test did not drive >>>> significant engagement. In fact, as implemented, it seemed inferior to >>>> blue links. However, we started with a very rough and low-impact >>>> prototype, so a few tweaks would give us more definitive results. >>>> >>>> Here is the doc from which I am pasting from below: >>>> https://docs.google.com/document/d/1Mqw-awAcp01IcLhHPsHmWsqaAyK1l2-w_LMDtizyFQ4/edit?usp=sharing >>>> >>>> Questions/comments welcome! >>>> Best, >>>> >>>> J >>>> >>>> >>>> Browse Prototype Results >>>> >>>> >>>> >>>> >>>> Intro >>>> <https://docs.google.com/document/d/1Mqw-awAcp01IcLhHPsHmWsqaAyK1l2-w_LMDtizyFQ4/edit#heading=h.6s40inyan02p> >>>> >>>> Process >>>> <https://docs.google.com/document/d/1Mqw-awAcp01IcLhHPsHmWsqaAyK1l2-w_LMDtizyFQ4/edit#heading=h.d5x661n72t7d> >>>> >>>> Results >>>> <https://docs.google.com/document/d/1Mqw-awAcp01IcLhHPsHmWsqaAyK1l2-w_LMDtizyFQ4/edit#heading=h.naqxa4etwhl4> >>>> >>>> Blue links in general >>>> <https://docs.google.com/document/d/1Mqw-awAcp01IcLhHPsHmWsqaAyK1l2-w_LMDtizyFQ4/edit#heading=h.8nn07h675j3o> >>>> >>>> Category tags >>>> <https://docs.google.com/document/d/1Mqw-awAcp01IcLhHPsHmWsqaAyK1l2-w_LMDtizyFQ4/edit#heading=h.gagragojxpiz> >>>> >>>> Conclusion and Next Steps >>>> <https://docs.google.com/document/d/1Mqw-awAcp01IcLhHPsHmWsqaAyK1l2-w_LMDtizyFQ4/edit#heading=h.z3p82tg8enr> >>>> >>>> Process >>>> <https://docs.google.com/document/d/1Mqw-awAcp01IcLhHPsHmWsqaAyK1l2-w_LMDtizyFQ4/edit#heading=h.ocqtfqhf8n0t> >>>> >>>> Do people want to browse by categories? >>>> <https://docs.google.com/document/d/1Mqw-awAcp01IcLhHPsHmWsqaAyK1l2-w_LMDtizyFQ4/edit#heading=h.9ksw2zvt8q19> >>>> >>>> >>>> >>>> Intro >>>> >>>> As outlined in this doc >>>> <https://docs.google.com/presentation/d/1ZssE8G0P5WVg8XmkBTi5G3n4OdLHPFGWZDZFW5_DSS0/edit?usp=sharing>, >>>> the concept is a tag that allows readers to navigate WP via categories that >>>> are meaningful and populated in order of 'significance' (as determined by >>>> user input). The hypothesis: >>>> >>>> - >>>> >>>> users will want to navigate by category if there are fewer, more >>>> meaningful categories per page and those category pages showed the most >>>> ‘notable’ members first. >>>> >>>> Again, see the full doc >>>> <https://docs.google.com/presentation/d/1ZssE8G0P5WVg8XmkBTi5G3n4OdLHPFGWZDZFW5_DSS0/edit?usp=sharing> >>>> to understand the premise. >>>> >>>> Process >>>> >>>> The first step was to validate: do users want to navigate via >>>> category? So we built a very lightweight prototype on mobile web, en >>>> wikipedia (stable, not beta) using hardcoded config variables, in the >>>> following categories ( ~4000 pages). Here we did not look into >>>> sub-categories with one exception (see T94732 >>>> <https://phabricator.wikimedia.org/T94732> for details). There was >>>> also an error and 2 of the categories did not have tags implemented (struck >>>> through, below) >>>> >>>> Category >>>> >>>> Pagecount >>>> >>>> NBA All Stars >>>> >>>> 400 >>>> >>>> American Politicians >>>> >>>> 818 >>>> >>>> Object-Oriented Programming Languages >>>> >>>> 164 >>>> >>>> European States >>>> >>>> 24 >>>> >>>> American Female Pop Singers >>>> >>>> 326 >>>> >>>> American drama television series >>>> >>>> 1048 >>>> >>>> Modern Painters >>>> >>>> 983 >>>> >>>> Landmarks in San Francisco, California >>>> >>>> 270 >>>> >>>> >>>> >>>> Here is how it appeared on the Alcatraz page >>>> >>>> >>>> When the user clicked the tag, they were taken to a gather-like >>>> collection based on manually estimated relevance >>>> >>>> (sorry cropped shot) >>>> >>>> >>>> >>>> >>>> The category pages were designed to show the most relevant (as deemed >>>> by me) to the broadest audience, first. Here is the ordering: >>>> https://docs.google.com/spreadsheets/d/12xLXQsH1zcg6E8lDuSonumZNdBvfaBuHOS1a1TCASK4/edit#gid=0 >>>> >>>> This was intended to lie in contrast with our current category pages, >>>> which are alphabetical and not really intended for human browsing: >>>> https://en.wikipedia.org/wiki/Category:American_male_film_actors >>>> >>>> >>>> We primarily measured a few things: >>>> >>>> >>>> - >>>> >>>> when a tag was seen by a user >>>> - >>>> >>>> when a tag was clicked on by a user >>>> - >>>> >>>> when a page in the new ‘category view’ was clicked on by a user >>>> >>>> >>>> As a side effort, I looked to see if overall referrals from pages with >>>> tags went up--this was a timed intervention rather than an a/b test and >>>> given the click-thru on the tags, the impact would have been negligible >>>> anyway. This was confirmed by some very noisy results. >>>> >>>> >>>> Results >>>> Blue links in general >>>> >>>> One benefit of the side study mentioned in the previous paragraph is >>>> that I was able to generate a table that looked at the pages in question >>>> before we started the test that shows a ratio of total pageviews/pageviews >>>> referred by a page (estimate of how many links were opened from that >>>> page). Though it is literally just for 0-1 GMT, 6/29/15, now that we have >>>> the pageview hourly table, a more robust analysis can tell us how >>>> categories differ in this regard: >>>> >>>> >>>> Category >>>> >>>> links clicked >>>> >>>> #pvs >>>> >>>> clicks/pvs >>>> >>>> Category:20th-centuryAmericanpoliticians >>>> >>>> 761 >>>> >>>> 1243 >>>> >>>> 61% >>>> >>>> Category:Americandramatelevisionseries >>>> >>>> 5981 >>>> >>>> 8844 >>>> >>>> 68% >>>> >>>> Category:Americanfemalepopsingers >>>> >>>> 2502 >>>> >>>> 4280 >>>> >>>> 58% >>>> >>>> Category:LandmarksinSanFrancisco, >>>> >>>> 104 >>>> >>>> 287 >>>> >>>> 36% >>>> >>>> Category:Modernpainters >>>> >>>> 136 >>>> >>>> 369 >>>> >>>> 37% >>>> >>>> Category:NationalBasketballAssociationAll-Stars >>>> >>>> 1908 >>>> >>>> 3341 >>>> >>>> 57% >>>> >>>> Category:Object-orientedprogramminglanguages >>>> >>>> 48 >>>> >>>> 181 >>>> >>>> 27% >>>> >>>> Category:WesternEurope >>>> >>>> 657 >>>> >>>> 1221 >>>> >>>> 54% >>>> >>>> Grand Total >>>> >>>> 12099 >>>> >>>> 19766 >>>> >>>> 50% >>>> >>>> >>>> You can see here that for pages in the category ‘Landmarks in San >>>> Francisco’, if there are 10 pageviews, 5.4 clicks to other pages are >>>> generated on average. >>>> >>>> I do not have the original queries for this handy, but can dig them up >>>> if you’re really interested. >>>> >>>> Category tags >>>> >>>> Full data and queries here: >>>> https://docs.google.com/a/wikimedia.org/spreadsheets/d/1vD3DopxGyeh9FQsuTQDMo6f5y43Yoy5gnJQqKn9hEQg/edit?usp=sharing >>>> >>>> The tags themselves generated an average click-through rate of .18%. >>>> Given the overall click thru rate on the pages estimated above ~50%, this >>>> single tag is not driving anything significant. Furthermore, given Leila >>>> and Bob’s paper suggest that this is performing no better than a >>>> mid-article click--given the mobile web sections are collapsed, I would >>>> need to understand more about their method to know just how to interpret >>>> their results against our mobile-web only implementation. Furthermore, our >>>> click through rate used the number of times the tag appeared on screen as >>>> the denominator, whereas their research looked at overall pageviews. >>>> >>>> >>>> This being noted, the tag was implemented to be as obscure as possible >>>> to establish a baseline. Furthermore, any feature like this would probably >>>> be different in the following ways: >>>> >>>> - >>>> >>>> each page would be in 1-4 tag groups (as opposed to just 1) >>>> - >>>> >>>> each page would be tagged, creating the expectation on the part of >>>> the user that this was something to look for >>>> - >>>> >>>> presumably the categories could be implemented as a menu item as >>>> opposed to being buried at the bottom of the page (and competing with >>>> features like read more. >>>> - >>>> >>>> Using the learnings from ‘read more’ tags with images or buttons >>>> would likely fare much better. >>>> >>>> >>>> The follow graph shows: >>>> >>>> - >>>> >>>> number of impressions on the right axis >>>> - >>>> >>>> click-thru-rate on the left-axis. >>>> >>>> >>>> >>>> When you look at click through rates on the ‘category’ pages >>>> themselves, you see that they average at 41% (Chart below) Meaning that >>>> for every 10 times a user visited a category page, there were 4.1 clicks to >>>> one of those pages as a result. >>>> >>>> >>>> Here is the same broken up by category: >>>> >>>> >>>> Each ‘category’ page here had at least 400 visits, and you can see that >>>> the interest seems to vary dramatically across categories. It is worth >>>> noting that the top three categories here are the ones with the fewest >>>> entities. Each list, however, was capped at ~50 articles, so it is unclear >>>> what might be causing this effect, if it is real. >>>> >>>> As mentioned above, the average article page has an overall click rate >>>> of 50%. So this page of categories did not have the click-through rate that >>>> a page has. However, this page had summaries of each of the pages, so it >>>> could be that users were generating value beyond what a blue link would >>>> provide. A live-user test of Gather collections, from whom this format was >>>> borrowed, suggested that the format used up too much vertical space on each >>>> article and was hard to flip through. Shortening the amount of text or >>>> image space might be something to try to make the page more useful >>>> >>>> >>>> Conclusion and Next StepsProcess >>>> >>>> - >>>> >>>> This was the first time I am aware of that we ran a live prototype >>>> and learn something without building a scalable solution. Win >>>> - >>>> >>>> Developer time was estimated at 1 FTE for 2 weeks (by pheudx), but >>>> the chronological time for pushing to stable took a quarter. Room >>>> for improvement >>>> - >>>> >>>> The time to analysis was almost 2 quarters, due to a lack of data >>>> analysis support (I ran the initial analysis within 2 weeks of launch, >>>> during paternity leave, but was unable to go back and get it ready to >>>> distribute for 3 months). Room for improvement--possibly solved by >>>> additional Data Analyst. >>>> >>>> >>>> This experiment was not designed to answer questions definitively in >>>> one round, but with the understanding that multiple iterations would allow >>>> us to fully answer our questions. >>>> >>>> The long turn-around time, particularly around analysis and >>>> communication, meant that tweaking a variable to test the conclusions or >>>> the new questions that arosee below will involve a whole lot more work and >>>> effort than if we had been able to explore modifications within a few weeks >>>> of the initial launch. >>>> >>>> >>>> Do people want to browse by categories? >>>> >>>> Category tags at the bottom of the mobile web page in a dull gray >>>> background that lead to manually curated categories are not a killer >>>> feature :) >>>> >>>> I would be reluctant to say that this means users are not interested in >>>> browsing by category, however. For instance, it is likely that >>>> >>>> - >>>> >>>> users did not notice the tag, even if it appeared on screen >>>> - >>>> >>>> users are accustomed to our current category tags on desktop and >>>> not interested in that experience >>>> - >>>> >>>> users who did like the tag were unlikely to find another page that >>>> had it--there was no feedback mechanism by which the improved category >>>> page >>>> would drive additional tag interactions >>>> - >>>> >>>> the browse experience created was not ideal >>>> >>>> >>>> >>>> >>>> If we decide to pursue what is currently termed “cascade c: update ux”, >>>> I would like to proceed with more tests in this arena, by altering the >>>> appearance and position of the tags, and by improving the flow of the >>>> ‘category’ pages. If we choose a different strategy, hopefully other teams >>>> can build off of what was learned here. >>>> >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> reading-wmf mailing list >>>> reading-...@lists.wikimedia.org >>>> https://lists.wikimedia.org/mailman/listinfo/reading-wmf >>>> >>>> >>> >>> _______________________________________________ >>> Mobile-l mailing list >>> Mobile-l@lists.wikimedia.org >>> https://lists.wikimedia.org/mailman/listinfo/mobile-l >>> >>> >> >> _______________________________________________ >> Mobile-l mailing list >> Mobile-l@lists.wikimedia.org >> https://lists.wikimedia.org/mailman/listinfo/mobile-l >> >> > > > -- > EN Wikipedia user page: https://en.wikipedia.org/wiki/User:Brian.gerstle > IRC: bgerstle > > _______________________________________________ > Mobile-l mailing list > Mobile-l@lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/mobile-l > >
_______________________________________________ Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
