Hi Toby, Everyone, thanks for this update. I think the approach you describe, boosting stats.grok.se with hardware support while developing the "official", scalable WMF solution, is quite sensible in this situation.
I do understand having just too many things on your plate to deal with, and I also know the difficulties inherent to dealing with large amounts of data in a scalable fashion. No one expects you to "press a button" and view data appears as if by magic; dedicated developer time, steady progress with the occasional update, support of Henrik's current interim solution, and maybe some rough time estimates for milestones would be what I (and people from the GLAM community) would hope for. Thanks, Magnus On Wed, Feb 19, 2014 at 11:14 PM, Toby Negrin <tneg...@wikimedia.org> wrote: > Hi Everyone, > > Many of you have read Magnus' post on his > blog<http://magnusmanske.de/wordpress/?p=173>. > I've commented on his blog and I wanted to repost here and address Magnus' > concerns directly. > > First of all, I'm sorry that we let Magnus and other folks down on the > page view APIs -- we made some commitments late last year that we weren't > able to meet. Not only that, these failures echoed previous points of > frustration with the Foundation. > > I do want to note that we actively support the infrastructure that feeds > data to stats.grok.se. We've fixed a number of issues with that pipeline, > most recently last week. We understand the importance of this data to the > community. > > The page view API project has been challenging for a number of reasons -- > the size of the data, the fact that definitions of page views have not been > updated to stay in line with the changing traffic (mobile, bots, API > requests, etc) and the challenges in aggregating various aliases. We've > needed to revisit our definitions of page views in order to get this right > as well as design and build a global architecture for collecting these and > other metrics. In addition, we've tried to do this with a perspective of > privacy and respect for our users. > > To this end, we presented an approach to measuring page views in MediaWiki > at FOSDEM in January and have made progress towards our new infrastructure > by deploying middleware delivering unsampled page view data from mobile > devices from our globally distributed datacenters to our compute cluster > for analysis. > > However, these initiatives are complex and will take several months to > complete at the earliest. In the meantime, we're working with Henrik to > scale up stats.grok.se. > > I also want to call out that the Analytics team has been supporting a wide > range of users and stakeholders during the year. We've developed > WikiMetrics, a tool for measuring editor productivity that is used by WMF > program evaluation and community members; provided dashboards and support > for Wikipedia Zero, our program to partner with our mobile partners to > enable mobile Wikipedia access free from data charges; and supported > product teams and researchers both inside and outside of the foundation. > > We've been prioritizing and working on these projects as our resources > allow and it's important to understand that the team has not been idle. > While we've done a less than stellar job in communicating our progress to > the community, information on what we've been doing is available via our > planning > pages <https://www.mediawiki.org/wiki/Analytics/Prioritization_Planning> on > mediawiki. In the future, we will be more proactive in communicating with > the community regarding our goals and projects. > > -Toby > > _______________________________________________ > Analytics mailing list > Analytics@lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/analytics > > -- undefined
_______________________________________________ Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
