On Mon, Jul 4, 2016 at 11:58 AM, Jon Robson <jrob...@wikimedia.org> wrote: > Certainly, I hear more often from people I talk to that one of their big > gripes with the mobile site is inability to use find in page when sections > are collapsed. I know at least 3 people who open all sections just to use > this function. Maybe I'm missing something, but based on your results it > seems we should be moving towards expanded sections by default as soon as > possible. There certainly doesn't seem anything negative here. If not what > would you suggest as next steps before doing that. > >> * Readers in the test group (sections expanded) tend to stay longer on the >> page > This seems like a good result to me. > > * Readers in the test group tend to spend more time reading, and less > time navigating >> Again this seems a good result for sections expanded. I'm not entirely >> sure what you mean by less time navigating though - is this less navigating >> to another page or to another section (the latter seems a given if sections >> are already expanded) See the aforementioned page on Meta for definitions, or the linked Phabricator task for the nitty-gritty details including the exact queries used. The posting here was basically just the TLDR of the summary ;) > > * Readers in the test group tend to scroll more sections into view > than readers in the control group open > > again this seems to be a good result. > > * Readers in the test group tend to stay shorter on the page than > readers using the Android Wikipedia app (which offers a TOC for easier > navigation, something not yet available in the mobile web test group) > > This doesn't seem like a fair comparison unless you compare readers outside > the test group to the Android app and talk about the delta. Again, see the Meta page for a discussion of limitations of this comparisons. And I would say that the test condition was clearly still more comparable to the Android app reading experience (because both are showing the full content initially) than the control group. > > On Tue, Jun 21, 2016 at 1:38 PM, Tilman Bayer <tba...@wikimedia.org> wrote: >> >> On Tue, Jun 21, 2016 at 3:17 AM, Justin Ormont <justin.orm...@gmail.com> >> wrote: >> > The chosen metrics are interesting ones as the sensitivity is high for >> > this >> > experiment though they aren't inherently positive or negative, hence the >> > mentioned ambiguity. >> Yes, so as mentioned in the writeup, readers in the test group could >> of course be spending more time on the page simply because they need >> longer to navigate to the desired part. But the second result >> addresses this particular concern. >> > Do you track metrics which rather closely track user >> > satisfaction? Perhaps a metric like distinct daily page views per user, >> > or >> > days active per week. >> The experiment was designed to also measure the number of pages viewed >> per browser session. Sadly, it turned out afterwards during data >> analysis that that part of the instrumentation had been buggy >> (different page views by the same reader in the same session were >> sometimes in the sampled group, sometimes not), so we don't have this >> data. We took a lesson from this and have since been testing other new >> instrumentations more thoroughly before deployment. >> >> >> > Breaking down the metrics by page length >> > (short/med/long) could give interesting results. >> Yes, that's doable in principle, although it would require some extra >> effort to join the EventLogging table with a list of page lengths. The >> schema is here BTW in case you want to suggest concrete queries: >> https://meta.wikimedia.org/wiki/Schema:MobileWebSectionUsage >> >> > >> > The semi-collapsed sections mentioned by Joaquin Oltra Hernandez sounds >> > quite useful. Perhaps the sections could be auto-collapsed or >> > semi-collapsed >> > for longer pages but short pages could remain fully expanded. >> > >> > --justin >> > >> > On Tue, Jun 21, 2016 at 1:45 AM, Joaquin Oltra Hernandez >> > <jhernan...@wikimedia.org> wrote: >> >> >> >> Great research, thanks for sharing! >> >> >> >> I'm looking forward to further diving into more subjective nuance, like >> >> the usefulness of each model for different reading use cases (quick >> >> fact >> >> checking vs exploratory learning for example). >> >> >> >> At some point I saw POC mocks of a mix between expanded and collapsed, >> >> where the section was collapsed, but it showed a small summary of the >> >> section below the title, like a teaser. It would be very interesting to >> >> test >> >> that kind of design too and see how it fares with the other two. >> >> >> >> >> >> >> >> On Tue, Jun 21, 2016 at 10:21 AM, Tilman Bayer <tba...@wikimedia.org> >> >> wrote: >> >>> >> >>> Hi, >> >>> >> >>> as most on this list will be aware, on the mobile web version of >> >>> Wikipedia, all top-level sections below the lead section are currently >> >>> shown collapsed on initial view. Users can tap on a section heading to >> >>> show the content, and to collapse it again. >> >>> To examine the tradeoffs of this solution and inform future product >> >>> decisions, we ran an experiment where 0.05% of mobile web users were >> >>> shown all pages with every section expanded on initial load, >> >>> instrumented alongside a control group of 0.05% that kept seeing the >> >>> standard view where all sections all initially collapsed. >> >>> >> >>> A high-level summary of results is now available at >> >>> >> >>> >> >>> https://meta.wikimedia.org/wiki/Research:Collapsed_vs_uncollapsed_section_view_on_mobile_web >> >>> . In particular: >> >>> >> >>> * Readers in the test group (sections expanded) tend to stay longer on >> >>> the page >> >>> * Readers in the test group tend to spend more time reading, and less >> >>> time navigating >> >>> * Readers in the test group tend to scroll more sections into view >> >>> than readers in the control group open >> >>> * Readers in the test group tend to stay shorter on the page than >> >>> readers using the Android Wikipedia app (which offers a TOC for easier >> >>> navigation, something not yet available in the mobile web test group) >> >>> >> >>> Comments and questions are welcome, feel free to use the talk page for >> >>> them too. >> >>> >> >>> Note that this experiment only measured some aspects, and that the >> >>> results don't yet allow the unambiguous conclusion that it would be >> >>> better to switch to the uncollapsed view. That said, they certainly >> >>> suggest that such a change should be considered. It is being planned >> >>> to examine this question further with some user testing sessions. >> >>> >> >>> (As an experiment, I've taken the opportunity to write this up this >> >>> analysis as a page in the research namespace on Meta, instead of on >> >>> Phabricator or in form of an email as done on other occasions. >> >>> Feedback on the format is welcome too.) >> >>> -- >> >>> Tilman Bayer >> >>> Senior Analyst >> >>> Wikimedia Foundation >> >>> IRC (Freenode): HaeB >> >>> >> >> >> >> >> >> _______________________________________________ >> >> Mobile-l mailing list >> >> Mobile-l@lists.wikimedia.org >> >> https://lists.wikimedia.org/mailman/listinfo/mobile-l >> >> >> > >> > >> > _______________________________________________ >> > Mobile-l mailing list >> > Mobile-l@lists.wikimedia.org >> > https://lists.wikimedia.org/mailman/listinfo/mobile-l >> > >> >> >> >> -- >> Tilman Bayer >> Senior Analyst >> Wikimedia Foundation >> IRC (Freenode): HaeB >> >> _______________________________________________ >> Mobile-l mailing list >> Mobile-l@lists.wikimedia.org >> https://lists.wikimedia.org/mailman/listinfo/mobile-l > >
-- Tilman Bayer Senior Analyst Wikimedia Foundation IRC (Freenode): HaeB _______________________________________________ Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
