[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

Manuel closed this task as "Resolved". TASK DETAIL https://phabricator.wikimedia.org/T282563 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GoranSMilovanovic, Manuel Cc: Esh77, Pablo, Mohammed_Sadat_WMDE, Tobi_WMDE_SW, MGerlach, awight,

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

GoranSMilovanovic added a comment. @Jan_Dittrich Do we need this ticket anymore? TASK DETAIL https://phabricator.wikimedia.org/T282563 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GoranSMilovanovic Cc: Esh77, Pablo, Mohammed_Sadat_WMDE,

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

Maintenance_bot removed a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T282563 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GoranSMilovanovic, Maintenance_bot Cc: Esh77, Pablo, Mohammed_Sadat_WMDE, Tobi_WMDE_SW,

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

gerritbot added a comment. Change 735736 **merged** by GoranSMilovanovic: [analytics/wmde/WD/WikidataAdHocAnalytics@master] T282563 https://gerrit.wikimedia.org/r/735736 TASK DETAIL https://phabricator.wikimedia.org/T282563 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

gerritbot added a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T282563 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GoranSMilovanovic, gerritbot Cc: Esh77, Pablo, Mohammed_Sadat_WMDE, Tobi_WMDE_SW, MGerlach, awight,

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

gerritbot added a comment. Change 735736 had a related patch set uploaded (by GoranSMilovanovic; author: GoranSMilovanovic): [analytics/wmde/WD/WikidataAdHocAnalytics@master] T282563 https://gerrit.wikimedia.org/r/735736 TASK DETAIL https://phabricator.wikimedia.org/T282563

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

GoranSMilovanovic added a comment. @MGerlach @Jan_Dittrich I have used XGBoost to train a `leave` vs `stay` binary classifier to our data. I did not go into elaborated cross-validation, used only a single train and single test dataset, downsampling by a huge factor (because of a

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

GoranSMilovanovic added a comment. @MGerlach @Jan_Dittrich It is a power-law (and thus Lindy) after all: "H0: data IS generated from a power law distribution; H1: data IS NOT generated from a power law distribution." (from: Fitting Heavy Tailed Distributions: The

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

GoranSMilovanovic added a comment. @MGerlach The authors of the paper that you have cited in T282563#7419722 use a similar - if not the same - approach to feature engineering for the prediction task as I have used in T282563#7251679

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

Jan_Dittrich added a comment. sure sure, go ahead :) TASK DETAIL https://phabricator.wikimedia.org/T282563 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GoranSMilovanovic, Jan_Dittrich Cc: Esh77, Pablo, Mohammed_Sadat_WMDE, Tobi_WMDE_SW,

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

GoranSMilovanovic added a comment. @Jan_Dittrich Sound great, but I think @MGerlach and I would like to add some modeling efforts to see if we can predict if a users stays or not. TASK DETAIL https://phabricator.wikimedia.org/T282563 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

Jan_Dittrich added a comment. What about: **Proposal title:** Wikidata user retention over time? **Session type:** Lightning or short, I guess? **Abstract:** People leave online communities after some time. However, the likelyhood that a particular user leaves the project is

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

GoranSMilovanovic added a comment. @MGerlach @Jan_Dittrich We also need to decide on the following in order to submit our session proposal to WikidataCon 2021: - Proposal title - Session type (please take a look at the submission

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

GoranSMilovanovic added a subscriber: Esh77. GoranSMilovanovic added a comment. @Esh77 @MGerlach @Jan_Dittrich Martin and Jan: thank you for your readiness to present our findings on Wikidata User Retention in the WikidataCon 2021 Education & science track (see WikidataCon 2021 program

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

MGerlach added a comment. @GoranSMilovanovic regarding the prediction model, a recent paper from this year's ISCW-conference might be very interesting (e.g. which features they use and to compare prediction-performance): **Learning to Predict the Departure Dynamics of Wikidata Editors

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

Manuel added a project: Wikidata Analytics. TASK DETAIL https://phabricator.wikimedia.org/T282563 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GoranSMilovanovic, Manuel Cc: Pablo, Mohammed_Sadat_WMDE, Tobi_WMDE_SW, MGerlach, awight, WMDE-leszek,

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

Maintenance_bot removed a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T282563 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GoranSMilovanovic, Maintenance_bot Cc: Pablo, Mohammed_Sadat_WMDE, Tobi_WMDE_SW, MGerlach,

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

GoranSMilovanovic added a comment. Here's the ETL code . I will add modeling and power law estimation as soon as I complete all additional steps as suggested. TASK DETAIL

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

gerritbot added a comment. Change 709690 **merged** by GoranSMilovanovic: [analytics/wmde/WD/WikidataAdHocAnalytics@master] T282563 https://gerrit.wikimedia.org/r/709690 TASK DETAIL https://phabricator.wikimedia.org/T282563 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

gerritbot added a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T282563 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GoranSMilovanovic, gerritbot Cc: Pablo, Mohammed_Sadat_WMDE, Tobi_WMDE_SW, MGerlach, awight,

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

gerritbot added a comment. Change 709690 had a related patch set uploaded (by GoranSMilovanovic; author: GoranSMilovanovic): [analytics/wmde/WD/WikidataAdHocAnalytics@master] T282563 https://gerrit.wikimedia.org/r/709690 TASK DETAIL https://phabricator.wikimedia.org/T282563

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

GoranSMilovanovic added a comment. @MGerlach First of all, thank you very much for the insights that you have provided. **On Power Laws and Lindy:** > One possible path out of this is to slightly change the question. Instead of asking whether the data is perfectly described by

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

MGerlach added a comment. In T282563#7252149 , @awight wrote: > In T282563#7251679 , @GoranSMilovanovic wrote: > >> Anyways, following a series of cross-validations and tricks to

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

MGerlach added a comment. @GoranSMilovanovic In T282563#7250712 , @GoranSMilovanovic wrote: > @Jan_Dittrich **Do we really find a Lindy effect in the Wikidata acount age distribution?** > > **Assumption.** As demonstrated in

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

GoranSMilovanovic added a comment. @awight First of all, I might have missed to mention that the outcome variable (i.e. what we are predicting) is **"stay"**, not "leave". My bad. > I'm unsure whether "positive" here means the classifier identifies a person who will leave or stay,

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

awight added a comment. In T282563#7251679 , @GoranSMilovanovic wrote: > Anyways, following a series of cross-validations and tricks to account for a highly imbalanced dataset, one Random Forrest classifier was able to predict leave vs

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

Jan_Dittrich added a comment. Thanks, super interesting! Some things are beyond my data-skills, thus, I look forward to feedback from other people! TASK DETAIL https://phabricator.wikimedia.org/T282563 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

GoranSMilovanovic added a comment. @Jan_Dittrich @awight @Lydia_Pintscher @Manuel @Tobi_WMDE_SW Probably of interest to all of you, because we have a quite interesting - and potentially very useful - outcome here. As a side kick to this ticket, I have trained a Random Forest

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

GoranSMilovanovic added a comment. @Jan_Dittrich **Do we really find a Lindy effect in the Wikidata acount age distribution?** **Assumption.** As demonstrated in Eliazar, Iddo (November 2017). "Lindy's Law". Physica A: Statistical Mechanics and Its Applications. 486: 797–805, if the

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

GoranSMilovanovic added a comment. @Jan_Dittrich @awight Finally, as of > ... user behavior on talk pages F34570923: 07_RevisionTalkNamespacesVSLeftWikidata.png but please take into your considerations that the distributions

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

GoranSMilovanovic added a comment. @Jan_Dittrich @awight In reference to T282563#7186386 and T282563#7226336 : - I have used a fresh dataset, relying on the `2021-06` snapshot of

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

GoranSMilovanovic added a comment. - Re-work on a fresh dataset (the `2021-06` snapshot of the `wmf.mediawiki_history` table) is underway; - Reporting: until tonight (hopefully); - @Jan_Dittrich I will be getting in touch via e-mail about the research/paper part later during the day.

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

Jan_Dittrich added a comment. Seems also relevant: https://eprints.whiterose.ac.uk/140352/1/evolution-wikidata-editors.pdf "The evolution of power and standard Wikidata editors: comparing editing behavior over time to predict lifespan and volume of edits" TASK DETAIL

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

awight added a comment. I really like where this is going. Maybe also look for patterns in the 94% who have dropped off, for example any variables that negatively correlate with longetivity. TASK DETAIL https://phabricator.wikimedia.org/T282563 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

GoranSMilovanovic added a comment. @Jan_Dittrich Following our 20210630 discussion: **Additional questions** - for those ~ 6% who are still with us: can we find any interesting patterns - the distribution of the length of their periods of inactivity - the distribution of

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

Jan_Dittrich updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T282563 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GoranSMilovanovic, Jan_Dittrich Cc: MGerlach, awight, WMDE-leszek, Manuel, Lydia_Pintscher, Aklapper,

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

Jan_Dittrich updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T282563 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GoranSMilovanovic, Jan_Dittrich Cc: MGerlach, awight, WMDE-leszek, Manuel, Lydia_Pintscher, Aklapper,

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

GoranSMilovanovic added a comment. @Jan_Dittrich **Please disregard all previous findings**. The following is based on: - the definition of editor inactivity in T282563#7124389 , - and the two important corrections in the

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

GoranSMilovanovic added a comment. @Jan_Dittrich Happening now: - the incorporation of the new inactivity criterion mentioned in T282563#7124389 (thanks @MGerlach), and - checking the completeness of my technical procedures in

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

Jan_Dittrich added a comment. From: "How Long Do Wikipedia Editors Keep Active?" > …specifically, we consider an editor to be“dead” or inactive if he did not make any edit for a certain period of time. Here we set the threshold of inactivity to be5 months, since it reflects WMF’s

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

Jan_Dittrich added a subscriber: MGerlach. Jan_Dittrich added a comment. Some new pointers (via @MGerlach ): - Understanding Editor Drop-off - How long do Wikipedia Editors

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

GoranSMilovanovic added a comment. @Jan_Dittrich @awight - I need to re-adjust the regular expression for editor reactivations as suggested by Adam in T282563#7110253 now > I am beginning to think that we might need to readjust

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

Jan_Dittrich added a comment. > I am beginning to think that we might need to readjust the definition of when we consider the editor to has left Wikidata as I have proposed it. What do you think? Any ideas? Would probably make sense to adjust it, but I have currently no good idea how.

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

Jan_Dittrich added a comment. >> Also, I have a vague and anecdotal memory that Wikidata has a lot of users who are semi- or fully-automated bots but are not registered as such. Is this the case? If so, is there some other heuristic like overly rapid editing that we can use to filter out

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

GoranSMilovanovic added a comment. @awight > If the zero reactivations category includes anyone for whom the 1+0+1+ regex doesn't match, doesn't this also include active editors who simply have a history like, 000111..., and who are still active? Bravo... Will take a look at it and

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

awight added a comment. This is looking great! I mean, it's a discouraging phenomenon but a promising analysis :-) I have a question about how to interpret the "0" reactivations category, which is stated above to be "a vast majority of editors [who] never reactivate following one

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

GoranSMilovanovic added a comment. @Jan_Dittrich > For people who become active editors again, it would be interesting to understand the patterns: Do they leave for a year and start again? (Like parents taking a baby break) Do they stop for a month and continue? (maybe they were sick

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

GoranSMilovanovic added a comment. @Jan_Dittrich This is also interesting: higher the number of reactivations in editing behavior - higher the probability to leave Wikidata. F34466163: NumReactivations_ProbabilityLeave.png TASK DETAIL

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

GoranSMilovanovic added a comment. @Jan_Dittrich This might also help, a larger version of the chart in T282563#7107757 with the number of editors on each level of activity (x-axis) included. Of course we get to observe fewer editors with

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

GoranSMilovanovic added a comment. @Jan_Dittrich > Question: What is the relation between past length of participation in the Wikidata community and the likelihood to stop participating? @Jan_Dittrich I had to impose some definitions in order to be able to precisely formulate the

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

GoranSMilovanovic added a comment. @WMDE-leszek Thank you. Don't worry, I will request the repo: we need one for this kind of one-shot tasks anyways. @awight @Jan_Dittrich The following is based on `395,680` Wikidata editors and following the corrections as suggested by @awight in

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

WMDE-leszek added a comment. > A Gerrit repo could take weeks I have o opinion whether it would be an overkill or not, but for what it could be worth, I can create gerrit repos when needed, and hopefully can do it in the same working day as I am requested. TASK DETAIL

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

awight added a comment. >> Can you share more about the query that produced reactivations.csv? I can't tell from the information provided what counts as a "period of inactivity". > > R code. If you still would like me to share it with you I will open a Gerrit repo for this ticket.

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

GoranSMilovanovic added a comment. @awight > This line can be removed, > event_user_id != 0 Indeed. > why would we need to check the historical column? If the user was classified as a bot at a time but now is not, shouldn't we respect the updated classification? Because

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

awight added a comment. This line can be removed, > event_user_id != 0 Anonymous users are already filtered out with `event_user_is_anonymous = FALSE`, and anyway `event_user_id` is set to `null` rather than `0` for anonymous (or revision-deleted) users. I think the bot

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

GoranSMilovanovic added a comment. @Jan_Dittrich To answer the following, simple question: > How high is the likelihood to become an active editor again? please find a dataset attached: `userId` is a fake (but unique) Wikidata user ID, `reactivationsN` is the number of the

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

GoranSMilovanovic added a comment. @Jan_Dittrich Please find the analytics dataset attached. Columns: - **userId**: the anonymized Wikidata user Id - **registrationYM**: the `-MM` timestamp of user registration on Wikidata - **revisionYM**: the `-MM` timestamp of

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

GoranSMilovanovic added projects: WMDE-Analytics-Engineering, User-GoranSMilovanovic. TASK DETAIL https://phabricator.wikimedia.org/T282563 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GoranSMilovanovic Cc: Manuel, Lydia_Pintscher, Aklapper,

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

Jan_Dittrich updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T282563 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GoranSMilovanovic, Jan_Dittrich Cc: Manuel, Lydia_Pintscher, Aklapper, Jan_Dittrich, Invadibot,

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

Jan_Dittrich renamed this task from "User Retention Wikidata: Exploring the resons for patterns in the 2021 Wikidata Community Survey" to "User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey". TASK DETAIL