[Wikidata-bugs] [Maniphest] T277551: [Curious Facts] improvements to issue descriptions

2021-06-10 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. @amy_rc - Part 1 (the so called `m1` anomalies): solved; issue descriptions fixed in accordance w. T277551#7137941 <https://phabricator.wikimedia.org/T277551#7137941> - Note: changes will not be visible on the dashboard before the full system

[Wikidata-bugs] [Maniphest] T283575: Wikidata Analytics: codebase modularization

2021-06-09 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. Current status: > funtcions that work with SPARQL/GAS programs against WDQS; - Develop an R function to query SPARQL/GAS against WDQS constrained by N (arbitrary) attempts to guard against timeout errors. TASK DETAIL ht

[Wikidata-bugs] [Maniphest] T277551: [Curious Facts] improvements to issue descriptions

2021-06-08 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. @amy_rc The suggestions given in T277551#7137941 <https://phabricator.wikimedia.org/T277551#7137941> imply a full system update. I am currently implementing the changes and running the update on the fly to speed up the process. Necessarily c

[Wikidata-bugs] [Maniphest] T277564: [Curious Facts] take separators into account for single value constraints

2021-06-07 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. @amy_rc Thank you. Shall we close this ticket then? TASK DETAIL https://phabricator.wikimedia.org/T277564 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GoranSMilovanovic Cc: amy_rc, WMDE-leszek, Aklapper

[Wikidata-bugs] [Maniphest] T277551: [Curious Facts] improvements to issue descriptions

2021-06-07 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. @amy_rc No problem, I am on it. TASK DETAIL https://phabricator.wikimedia.org/T277551 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GoranSMilovanovic Cc: WMDE-leszek, amy_rc, GoranSMilovanovic, Aklapper

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

2021-06-02 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. @Jan_Dittrich Happening now: - the incorporation of the new inactivity criterion mentioned in T282563#7124389 <https://phabricator.wikimedia.org/T282563#7124389> (thanks @MGerlach), and - checking the completeness of my technical proc

[Wikidata-bugs] [Maniphest] T277551: [Curious Facts] improvements to issue descriptions

2021-06-01 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. @amy_rc @WMDE-leszek > Yes. @Lydia_Pintscher and I tested it after reading T277551#7069216 <https://phabricator.wikimedia.org/T277551#7069216>. We then made the corresponding notes. You must have missed something in that case because

[Wikidata-bugs] [Maniphest] T277564: [Curious Facts] take separators into account for single value constraints

2021-06-01 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. @Lydia_Pintscher Please check it out now: https://wikidata-analytics.wmcloud.org/app/WikidataAnalytics TASK DETAIL https://phabricator.wikimedia.org/T277564 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences

[Wikidata-bugs] [Maniphest] T283465: movie and TV connections finder

2021-05-27 Thread GoranSMilovanovic
GoranSMilovanovic removed projects: User-GoranSMilovanovic, WMDE-Analytics-Engineering. TASK DETAIL https://phabricator.wikimedia.org/T283465 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GoranSMilovanovic Cc: amy_rc, WMDE-leszek

[Wikidata-bugs] [Maniphest] T283466: topic overlap between Wikipedia language versions

2021-05-27 Thread GoranSMilovanovic
GoranSMilovanovic removed projects: User-GoranSMilovanovic, WMDE-Analytics-Engineering. TASK DETAIL https://phabricator.wikimedia.org/T283466 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GoranSMilovanovic Cc: amy_rc, WMDE-leszek

[Wikidata-bugs] [Maniphest] T283466: topic overlap between Wikipedia language versions

2021-05-25 Thread GoranSMilovanovic
GoranSMilovanovic removed GoranSMilovanovic as the assignee of this task. TASK DETAIL https://phabricator.wikimedia.org/T283466 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GoranSMilovanovic Cc: amy_rc, WMDE-leszek, GoranSMilovanovic, EpicPupper

[Wikidata-bugs] [Maniphest] T283465: movie and TV connections finder

2021-05-25 Thread GoranSMilovanovic
GoranSMilovanovic removed GoranSMilovanovic as the assignee of this task. TASK DETAIL https://phabricator.wikimedia.org/T283465 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GoranSMilovanovic Cc: amy_rc, WMDE-leszek, GoranSMilovanovic, Aklapper

[Wikidata-bugs] [Maniphest] T283575: Wikidata Analytics: codebase modularization

2021-05-25 Thread GoranSMilovanovic
GoranSMilovanovic created this task. GoranSMilovanovic added projects: User-GoranSMilovanovic, WMDE-Analytics-Engineering, Wikidata. TASK DESCRIPTION There is still some modularization that should happen in the Wikidata Analytics core codebase: - functions that work with the Wikibase API

[Wikidata-bugs] [Maniphest] T283574: Wikidata Analytics Data Products: DockerHub/Deployment

2021-05-25 Thread GoranSMilovanovic
GoranSMilovanovic created this task. GoranSMilovanovic added projects: User-GoranSMilovanovic, WMDE-Analytics-Engineering, Wikidata. TASK DESCRIPTION All Wikidata Analytics Data Products <https://wikidata-analytics.wmcloud.org/app/WikidataAnalytics> (dashboards, apps, and r

[Wikidata-bugs] [Maniphest] T283571: Automation of large Wikidata Analytics updates

2021-05-25 Thread GoranSMilovanovic
GoranSMilovanovic created this task. GoranSMilovanovic added projects: User-GoranSMilovanovic, WMDE-Analytics-Engineering, Wikidata. TASK DESCRIPTION Some Wikidata Analytics systems run large and risky (in terms of how much of the Analytics Cluster/Clients resources the use) updates

[Wikidata-bugs] [Maniphest] T283570: Impose the tidyverse style for R code wherever possible

2021-05-25 Thread GoranSMilovanovic
GoranSMilovanovic created this task. GoranSMilovanovic added projects: User-GoranSMilovanovic, WMDE-Analytics-Engineering, Wikidata. TASK DESCRIPTION - The Wikidata Analytics core codebase is now already mature, but - it was under constant development since March 2017 with no formal coding

[Wikidata-bugs] [Maniphest] T283568: Wikidata Analytics Core Codebase Maintainance

2021-05-25 Thread GoranSMilovanovic
GoranSMilovanovic created this task. GoranSMilovanovic added projects: User-GoranSMilovanovic, WMDE-Analytics-Engineering, Wikidata. Restricted Application added a subscriber: Aklapper. TASK DESCRIPTION An umbrella ticket for Wikidata Analytics Core Codebase Maintainance: https://github.com

[Wikidata-bugs] [Maniphest] T283465: movie and TV connections finder

2021-05-25 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. It should not be too difficult to have an RStudio Shiny app developed for this. Tech stack: - a bit of Pyspark/R for the clustering part, and then probably - plain SPARQL against WDQS for everything else directly from the dashboard. TASK DETAIL

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

2021-05-25 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. @Jan_Dittrich @awight - I need to re-adjust the regular expression for editor reactivations as suggested by Adam in T282563#7110253 <https://phabricator.wikimedia.org/T282563#7110253> now > I am beginning to think that we might need to

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

2021-05-25 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. @awight > If the zero reactivations category includes anyone for whom the 1+0+1+ regex doesn't match, doesn't this also include active editors who simply have a history like, 000111..., and who are still active? Bravo... Will take a l

[Wikidata-bugs] [Maniphest] T283466: topic overlap between Wikipedia language versions

2021-05-25 Thread GoranSMilovanovic
GoranSMilovanovic added a subscriber: WMDE-leszek. GoranSMilovanovic added a comment. @Lydia_Pintscher @Manuel @WMDE-leszek Before we proceed with this, please take a look at our WDCM Sitelinks Dashboard <https://wikidata-analytics.wmcloud.org/app/WDCM_SitelinksDashboard>:

[Wikidata-bugs] [Maniphest] T283465: movie and TV connections finder

2021-05-25 Thread GoranSMilovanovic
GoranSMilovanovic claimed this task. GoranSMilovanovic added projects: WMDE-Analytics-Engineering, User-GoranSMilovanovic. GoranSMilovanovic added subscribers: GoranSMilovanovic, WMDE-leszek. TASK DETAIL https://phabricator.wikimedia.org/T283465 EMAIL PREFERENCES https

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

2021-05-24 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. @Jan_Dittrich > For people who become active editors again, it would be interesting to understand the patterns: Do they leave for a year and start again? (Like parents taking a baby break) Do they stop for a month and continue? (maybe they were s

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

2021-05-24 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. @Jan_Dittrich This is also interesting: higher the number of reactivations in editing behavior - higher the probability to leave Wikidata. F34466163: NumReactivations_ProbabilityLeave.png <https://phabricator.wikimedia.org/F34466163> TASK

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

2021-05-24 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. @Jan_Dittrich This might also help, a larger version of the chart in T282563#7107757 <https://phabricator.wikimedia.org/T282563#7107757> with the number of editors on each level of activity (x-axis) included. Of course we get to observe fewer e

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

2021-05-24 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. @Jan_Dittrich > Question: What is the relation between past length of participation in the Wikidata community and the likelihood to stop participating? @Jan_Dittrich I had to impose some definitions in order to be able to precisely formul

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

2021-05-24 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. @WMDE-leszek Thank you. Don't worry, I will request the repo: we need one for this kind of one-shot tasks anyways. @awight @Jan_Dittrich The following is based on `395,680` Wikidata editors and following the corrections as suggested by @awight

[Wikidata-bugs] [Maniphest] T283466: topic overlap between Wikipedia language versions

2021-05-24 Thread GoranSMilovanovic
GoranSMilovanovic added projects: WMDE-Analytics-Engineering, User-GoranSMilovanovic. TASK DETAIL https://phabricator.wikimedia.org/T283466 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GoranSMilovanovic Cc: GoranSMilovanovic, EpicPupper, Manuel

[Wikidata-bugs] [Maniphest] T283466: topic overlap between Wikipedia language versions

2021-05-24 Thread GoranSMilovanovic
GoranSMilovanovic claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T283466 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GoranSMilovanovic Cc: GoranSMilovanovic, EpicPupper, Manuel, Aklapper, Lydia_Pintscher, Invadibot, maantietaja

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

2021-05-22 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. @awight > This line can be removed, > event_user_id != 0 Indeed. > why would we need to check the historical column? If the user was classified as a bot at a time but now is not, shouldn't we respect the updated classification?

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

2021-05-21 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. @Jan_Dittrich To answer the following, simple question: > How high is the likelihood to become an active editor again? please find a dataset attached: `userId` is a fake (but unique) Wikidata user ID, `reactivationsN` is the num

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

2021-05-17 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. @Jan_Dittrich Please find the analytics dataset attached. Columns: - **userId**: the anonymized Wikidata user Id - **registrationYM**: the `-MM` timestamp of user registration on Wikidata - **revisionYM**: the `-MM` timestamp

[Wikidata-bugs] [Maniphest] T277551: [Curious Facts] improvements to issue descriptions

2021-05-17 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. @amy_rc Ok, I will take a look. TASK DETAIL https://phabricator.wikimedia.org/T277551 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GoranSMilovanovic Cc: WMDE-leszek, amy_rc, GoranSMilovanovic, Aklapper

[Wikidata-bugs] [Maniphest] T277551: [Curious Facts] improvements to issue descriptions

2021-05-17 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. > Only the ISSUE DESCRIPTION topic is related to this ticket. The others are already done. @amy_rc I understand that, but did you (or anyone else) check the system following the changes announced in T277551#7069216 <https://phabricator.wikimed

[Wikidata-bugs] [Maniphest] T277551: [Curious Facts] improvements to issue descriptions

2021-05-17 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. @amy_rc @WMDE-leszek Hey do the changes in T277551#7091927 <https://phabricator.wikimedia.org/T277551#7091927> and discussed in the doc <https://docs.google.com/document/d/1o36ljsJ9jMNInkVu-vIF_qLyyr-Djm8IjI9KzXmoQmE/edit#> precede

[Wikidata-bugs] [Maniphest] T259105: Qurator: Data about Current Events

2021-05-13 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. @Maria_WMDE > ... items were included in the list even though they were edited by less than three editors Well there were discussions whether to display all recently edited items if it happens that none was observed to have been edited by

[Wikidata-bugs] [Maniphest] T277564: [Curious Facts] take separators into account for single value constraints

2021-05-12 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. @Lydia_Pintscher > Does this help clarify it? I think I understand completelly what you are saying. Thank you. > Does your system take this into account or only the separators from ISNN? Nope, I have obviosuly solved a simplified p

[Wikidata-bugs] [Maniphest] T282563: User Retention Wikidata: A model for "participating since" patterns in the 2021 Wikidata Community Survey

2021-05-11 Thread GoranSMilovanovic
GoranSMilovanovic added projects: WMDE-Analytics-Engineering, User-GoranSMilovanovic. TASK DETAIL https://phabricator.wikimedia.org/T282563 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GoranSMilovanovic Cc: Manuel, Lydia_Pintscher, Aklapper

[Wikidata-bugs] [Maniphest] T277551: [Curious Facts] improvements to issue descriptions

2021-05-07 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. @Lydia_Pintscher @WMDE-leszek Ok; the generic fix for issue descriptions is now in place; the system is back online from Wikidata Analytics <https://wikidata-analytics.wmcloud.org/app/WikidataAnalytics>. TASK DETAIL https://phabricator.wikimed

[Wikidata-bugs] [Maniphest] T277551: [Curious Facts] improvements to issue descriptions

2021-05-06 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. @Lydia_Pintscher @WMDE-leszek - Ok, I have figured out the problem. - Implementing the fix now. It might take some time. TASK DETAIL https://phabricator.wikimedia.org/T277551 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel

[Wikidata-bugs] [Maniphest] T277551: [Curious Facts] improvements to issue descriptions

2021-05-06 Thread GoranSMilovanovic
GoranSMilovanovic added a subscriber: WMDE-leszek. GoranSMilovanovic added a comment. Ok @Lydia_Pintscher @WMDE-leszek - following the latest update of the Curious Facts system I am facing a serious production side problem related to the `{data.table}` package; - the Curious Facts

[Wikidata-bugs] [Maniphest] T277551: [Curious Facts] improvements to issue descriptions

2021-05-05 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. @Lydia_Pintscher - I will now re-run parts of the update to see what could have gone wrong with some of the issue descriptions. TASK DETAIL https://phabricator.wikimedia.org/T277551 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings

[Wikidata-bugs] [Maniphest] T277551: [Curious Facts] improvements to issue descriptions

2021-05-05 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. @Lydia_Pintscher - The update is now complete, but it seems like the changes in descriptions did not apply correctly to all reported anomalies. - Inspecting now. TASK DETAIL https://phabricator.wikimedia.org/T277551 EMAIL PREFERENCES https

[Wikidata-bugs] [Maniphest] T281063: Wikidata Concepts Monitor: some datasets are empty

2021-05-05 Thread GoranSMilovanovic
GoranSMilovanovic closed this task as "Resolved". GoranSMilovanovic added a comment. @MisterSynergy I will close this task now. Please re-open or file a new ticket altogether if you encounter any similar problems. Again, thanks for catching this. TASK DETA

[Wikidata-bugs] [Maniphest] T277551: [Curious Facts] improvements to issue descriptions

2021-05-03 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. @Lydia_Pintscher - A generic fix was applied across all types issue descriptions and they should change as soon as the lengthy update procedure finishes; - then we can evaluate them and see if they need further improvement; - I will ping here

[Wikidata-bugs] [Maniphest] T277564: [Curious Facts] take separators into account for single value constraints

2021-05-01 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. @Lydia_Pintscher Done. The following separator values (as defined here <https://www.wikidata.org/wiki/Property:P236#P236$a4e1524c-4191-7ca4-b275-b6096ec05cba>) where taken into account and any item/property pair which makes use of them was filter

[Wikidata-bugs] [Maniphest] T281316: WDCM_Sqoop_Clients.R fails from stat1004 (again)

2021-04-30 Thread GoranSMilovanovic
GoranSMilovanovic closed this task as "Resolved". GoranSMilovanovic added a comment. Ok. In any case the fix to this script is easy if anything similar happens again. I will begin to monitor the Sqoop runs more closely. Thank you @elukey ! TASK DETAIL https://phabricator.wik

[Wikidata-bugs] [Maniphest] T281063: Wikidata Concepts Monitor: some datasets are empty

2021-04-30 Thread GoranSMilovanovic
GoranSMilovanovic closed subtask T281316: WDCM_Sqoop_Clients.R fails from stat1004 (again) as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T281063 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GoranSMilovanovic Cc: elukey, WMDE-leszek

[Wikidata-bugs] [Maniphest] T281316: WDCM_Sqoop_Clients.R fails from stat1004 (again)

2021-04-30 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. @elukey Thank you. I was thinking along the following lines: - if due to any updates, upgrades, or other changes, this turns out to be a persistent problem, - then is there a way to ask from some Analytics Client if `org.mariadb.jdbc.Driver` should

[Wikidata-bugs] [Maniphest] T281554: tool to make it easy to classify unclassified Items

2021-04-30 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. @Lydia_Pintscher I understand, we are just looking for a tool to expose such items to the community better. Got it. TASK DETAIL https://phabricator.wikimedia.org/T281554 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel

[Wikidata-bugs] [Maniphest] T281554: tool to make it easy to classify unclassified Items

2021-04-30 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. @Lydia_Pintscher I think we should first (1) assess those items without classifying properties and (2) see if (at least on the average) there is enough data about them to consider (3) the development of a recommendation system for their classification. Let

[Wikidata-bugs] [Maniphest] T277558: [Curious Facts] remove missing image message

2021-04-30 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. @Lydia_Pintscher Fixed: Curious Facts dashboard <https://wikidata-analytics.wmcloud.org/app/CuriousFacts>. TASK DETAIL https://phabricator.wikimedia.org/T277558 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailprefe

[Wikidata-bugs] [Maniphest] T277554: [Curious Facts] reduce Qurator branding

2021-04-30 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. Changes are now in effect from Wikidata Analytics <https://wikidata-analytics.wmcloud.org>. @Lydia_Pintscher Please let me know if there is anything else in relation to this ticket. Thank you. TASK DETAIL https://phabricator.wikimedia.org/T

[Wikidata-bugs] [Maniphest] T281063: Wikidata Concepts Monitor: some datasets are empty

2021-04-30 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. @MisterSynergy > your wikidata-analytics.wmcloud.org seems to be down currently, but I am accessing the topItems.csv file directly anyways Yes, some changes in production were just deployed (in relation to T277554 <

[Wikidata-bugs] [Maniphest] T277554: [Curious Facts] reduce Qurator branding

2021-04-30 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. All Qurator brending is now removed. Next step: fix Wikidata Analytics <https://wikidata-analytics.wmcloud.org/app/WikidataAnalytics> links for the Curious Facts dashboard. TASK DETAIL https://phabricator.wikimedia.org/T277554 EMAIL PREFE

[Wikidata-bugs] [Maniphest] T277554: [Curious Facts] reduce Qurator branding

2021-04-30 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. @Lydia_Pintscher > The URL and HTML title still mention Qurator for me. The URL is now: https://wikidata-analytics.wmcloud.org/app/CuriousFacts The change is not yet in effect in the Wikidata Analytics Portal - working on it right now.

[Wikidata-bugs] [Maniphest] T281063: Wikidata Concepts Monitor: some datasets are empty

2021-04-30 Thread GoranSMilovanovic
GoranSMilovanovic lowered the priority of this task from "High" to "Low". TASK DETAIL https://phabricator.wikimedia.org/T281063 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GoranSMilovanovic Cc: elukey, WMDE-leszek, Goran

[Wikidata-bugs] [Maniphest] T281063: Wikidata Concepts Monitor: some datasets are empty

2021-04-30 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. @MisterSynergy I have checked the datasets <https://wikidata-analytics.wmcloud.org/app_direct/WikidataAnalytics/datasets.html> in Wikidata Analytics and everything seems to be in place now. TASK DETAIL https://phabricator.wikimedia.org/T281063

[Wikidata-bugs] [Maniphest] T281063: Wikidata Concepts Monitor: some datasets are empty

2021-04-29 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. @MisterSynergy The WDCM system update should be in place now. Please let me know if the datasets <https://wikidata-analytics.wmcloud.org/app_direct/WikidataAnalytics/datasets.html> that you need are now complete. I apologize f

[Wikidata-bugs] [Maniphest] T281316: WDCM_Sqoop_Clients.R fails from stat1004 (again)

2021-04-29 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. @WMDE-leszek @elukey I would like to learn from this. The following argument to `/usr/bin/sqoop` > --driver org.mariadb.jdbc.Driver seems to have been causing us trouble for some time already in WDCM_Sqoop_Clients.R <https://gith

[Wikidata-bugs] [Maniphest] T281316: WDCM_Sqoop_Clients.R fails from stat1004 (again)

2021-04-29 Thread GoranSMilovanovic
GoranSMilovanovic reopened this task as "Open". GoranSMilovanovic added a comment. @elukey Let's take a close look at this, if you agree. TASK DETAIL https://phabricator.wikimedia.org/T281316 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/

[Wikidata-bugs] [Maniphest] T281063: Wikidata Concepts Monitor: some datasets are empty

2021-04-29 Thread GoranSMilovanovic
GoranSMilovanovic reopened subtask T281316: WDCM_Sqoop_Clients.R fails from stat1004 (again) as Open. TASK DETAIL https://phabricator.wikimedia.org/T281063 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GoranSMilovanovic Cc: elukey, WMDE-leszek

[Wikidata-bugs] [Maniphest] T281063: Wikidata Concepts Monitor: some datasets are empty

2021-04-29 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. Current status: - WDCM system update completed; - monitoring the Wikidata Analytics now, TASK DETAIL https://phabricator.wikimedia.org/T281063 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences

[Wikidata-bugs] [Maniphest] T281063: Wikidata Concepts Monitor: some datasets are empty

2021-04-29 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. Current status: - WDCM system update: complete; - monitoring WDCM datasets and dashboards now. TASK DETAIL https://phabricator.wikimedia.org/T281063 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences

[Wikidata-bugs] [Maniphest] T281063: Wikidata Concepts Monitor: some datasets are empty

2021-04-28 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. WDCM system update, current status - Collect module (SPARQL/GAS) completed; - ETL module (Spark) completed; - ML module is now running. TASK DETAIL https://phabricator.wikimedia.org/T281063 EMAIL PREFERENCES https://phabricator.wikimedia.org

[Wikidata-bugs] [Maniphest] T281063: Wikidata Concepts Monitor: some datasets are empty

2021-04-28 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. Current status: - `WDCM_Sqoop_Clients.R` update is completed; - running a manual update of the WDCM system now. TASK DETAIL https://phabricator.wikimedia.org/T281063 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel

[Wikidata-bugs] [Maniphest] T281316: WDCM_Sqoop_Clients.R fails from stat1004 (again)

2021-04-28 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. @elukey No worries. Let me know if you need any external tests performed. TASK DETAIL https://phabricator.wikimedia.org/T281316 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GoranSMilovanovic Cc: elukey

[Wikidata-bugs] [Maniphest] T281063: Wikidata Concepts Monitor: some datasets are empty

2021-04-28 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. Current status: - monitoring the `WDCM_Sqoop_Clients.R` update, - all looking good; - it might take 10 - 12 hours for this procedure to complete. TASK DETAIL https://phabricator.wikimedia.org/T281063 EMAIL PREFERENCES https

[Wikidata-bugs] [Maniphest] T281063: Wikidata Concepts Monitor: some datasets are empty

2021-04-28 Thread GoranSMilovanovic
GoranSMilovanovic added a subscriber: elukey. GoranSMilovanovic added a comment. The problem will be handled here and in T281316 <https://phabricator.wikimedia.org/T281316> (where we already have the solution thanks to @elukey). The following T281063#7037642

[Wikidata-bugs] [Maniphest] T281316: WDCM_Sqoop_Clients.R fails from stat1004 (again)

2021-04-28 Thread GoranSMilovanovic
GoranSMilovanovic lowered the priority of this task from "High" to "Low". TASK DETAIL https://phabricator.wikimedia.org/T281316 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GoranSMilovanovic Cc: elukey, Lydia_Pintscher, Mis

[Wikidata-bugs] [Maniphest] T281316: WDCM_Sqoop_Clients.R fails from stat1004 (again)

2021-04-28 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. @elukey > As a quick workaround you should be able to unblock your queries just removing --driver org.mariadb.jdbc.Driver, can you try to see if it works? That would do, thank you Luca! I will keep the ticket open just in case, until the upd

[Wikidata-bugs] [Maniphest] T281316: WDCM_Sqoop_Clients.R fails from stat1004 (again)

2021-04-28 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. @elukey Thank your for a prompt response, Luca! > The first error is unrelated, it is due to the fact that you have created the /tmp/wmde directory as analytics-privatedata (and by default permissions allow only user + group, not others) and y

[Wikidata-bugs] [Maniphest] T281316: WDCM_Sqoop_Clients.R fails from stat1004 (again)

2021-04-27 Thread GoranSMilovanovic
GoranSMilovanovic triaged this task as "High" priority. TASK DETAIL https://phabricator.wikimedia.org/T281316 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GoranSMilovanovic Cc: Lydia_Pintscher, MisterSynergy, Aklapper, Manuel, W

[Wikidata-bugs] [Maniphest] T281063: Wikidata Concepts Monitor: some datasets are empty

2021-04-27 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. @MisterSynergy Thank you for catching this. > re-run the script for fresh datasets > ensure that it won't put empty datasets there in the future again Sure, but first we need to find out about the cause of this failure. I suspect that it

[Wikidata-bugs] [Maniphest] T281063: Wikidata Concepts Monitor: some datasets are empty

2021-04-27 Thread GoranSMilovanovic
GoranSMilovanovic claimed this task. GoranSMilovanovic added a project: User-GoranSMilovanovic. GoranSMilovanovic triaged this task as "High" priority. GoranSMilovanovic added a subscriber: WMDE-leszek. TASK DETAIL https://phabricator.wikimedia.org/T281063 EMAIL PREFERENC

[Wikidata-bugs] [Maniphest] T277554: [Curious Facts] reduce Qurator branding

2021-04-21 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. @Lydia_Pintscher The following changes: > The URL and HTML title still mention Qurator for me. ask for a re-production of the Shiny/{Golem} dashboard. I am on it. TASK DETAIL https://phabricator.wikimedia.org/T277554 EMAIL PREFEREN

[Wikidata-bugs] [Maniphest] T278698: metrics: number of translations made possible through Lexemes as of Jan 1st 2021

2021-04-11 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. @Lydia_Pintscher @Lea_WMDE @WMDE-leszek The **"translation" connections** part is now also completed (by parsing the XML dump): F34349129: lexemeTranslations_B.csv <https://phabricator.wikimedia.org/F34349129> The results ar

[Wikidata-bugs] [Maniphest] T278698: metrics: number of translations made possible through Lexemes as of Jan 1st 2021

2021-04-10 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. @Lydia_Pintscher @Lea_WMDE @WMDE-leszek The **"Item for this Sense" connections** is completed (by parsing the XML dump): F34322039: senseItemFrame_A.csv <https://phabricator.wikimedia.org/F34322039> The results are

[Wikidata-bugs] [Maniphest] T278698: metrics: number of translations made possible through Lexemes as of Jan 1st 2021

2021-04-09 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. @WMDE-leszek @Lydia_Pintscher Thank you for your suggestions. Please let me first try to accomplish this by relying on the approach described in T278698#6986358 <https://phabricator.wikimedia.org/T278698#6986358>: it seems doable and I have a

[Wikidata-bugs] [Maniphest] T278698: metrics: number of translations made possible through Lexemes as of Jan 1st 2021

2021-04-08 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. @Lydia_Pintscher @Lea_WMDE @WMDE-leszek The data that you are looking for are **extremely** difficult to obtain. The only way that works - or at least the only one that I was able to discover - is to parse the revisions from the Mediawiki wikitext

[Wikidata-bugs] [Maniphest] T278698: metrics: number of translations made possible through Lexemes as of Jan 1st 2021

2021-04-06 Thread GoranSMilovanovic
GoranSMilovanovic claimed this task. GoranSMilovanovic added a project: User-GoranSMilovanovic. TASK DETAIL https://phabricator.wikimedia.org/T278698 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GoranSMilovanovic Cc: Aklapper, GoranSMilovanovic

[Wikidata-bugs] [Maniphest] T272192: Migrate to new Wikidata Analytics

2021-03-30 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. @diego Diego I am really sorry but as I have already mentioned the migration took some two months, and was publicly announced. TASK DETAIL https://phabricator.wikimedia.org/T272192 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel

[Wikidata-bugs] [Maniphest] T277556: [Curious Facts] give explanation for timestamp

2021-03-30 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. Deployed: https://wikidata-analytics.wmcloud.org/app/QURATOR_CuriousFacts TASK DETAIL https://phabricator.wikimedia.org/T277556 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GoranSMilovanovic Cc

[Wikidata-bugs] [Maniphest] T277554: [Curious Facts] reduce Qurator branding

2021-03-30 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. Deployed: https://wikidata-analytics.wmcloud.org/app/QURATOR_CuriousFacts TASK DETAIL https://phabricator.wikimedia.org/T277554 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GoranSMilovanovic Cc

[Wikidata-bugs] [Maniphest] T277558: [Curious Facts] remove missing image message

2021-03-30 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. Deployed: https://wikidata-analytics.wmcloud.org/app/QURATOR_CuriousFacts TASK DETAIL https://phabricator.wikimedia.org/T277558 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GoranSMilovanovic Cc

[Wikidata-bugs] [Maniphest] T277558: [Curious Facts] remove missing image message

2021-03-30 Thread GoranSMilovanovic
GoranSMilovanovic claimed this task. GoranSMilovanovic added a project: User-GoranSMilovanovic. TASK DETAIL https://phabricator.wikimedia.org/T277558 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GoranSMilovanovic Cc: GoranSMilovanovic, Aklapper

[Wikidata-bugs] [Maniphest] T277556: [Curious Facts] give explanation for timestamp

2021-03-30 Thread GoranSMilovanovic
GoranSMilovanovic claimed this task. GoranSMilovanovic added a project: User-GoranSMilovanovic. TASK DETAIL https://phabricator.wikimedia.org/T277556 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GoranSMilovanovic Cc: GoranSMilovanovic, Aklapper

[Wikidata-bugs] [Maniphest] T277554: [Curious Facts] reduce Qurator branding

2021-03-30 Thread GoranSMilovanovic
GoranSMilovanovic claimed this task. GoranSMilovanovic added a project: User-GoranSMilovanovic. TASK DETAIL https://phabricator.wikimedia.org/T277554 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GoranSMilovanovic Cc: GoranSMilovanovic, Aklapper

[Wikidata-bugs] [Maniphest] T277551: [Curious Facts] improvements to issue descriptions

2021-03-30 Thread GoranSMilovanovic
GoranSMilovanovic claimed this task. GoranSMilovanovic added a project: User-GoranSMilovanovic. TASK DETAIL https://phabricator.wikimedia.org/T277551 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GoranSMilovanovic Cc: GoranSMilovanovic, Aklapper

[Wikidata-bugs] [Maniphest] T277564: [Curious Facts] take separators into account for single value constraints

2021-03-30 Thread GoranSMilovanovic
GoranSMilovanovic claimed this task. GoranSMilovanovic added a project: User-GoranSMilovanovic. TASK DETAIL https://phabricator.wikimedia.org/T277564 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GoranSMilovanovic Cc: Aklapper, Lydia_Pintscher

[Wikidata-bugs] [Maniphest] T277551: [Curious Facts] improvements to issue descriptions

2021-03-30 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. @Lydia_Pintscher The issue descriptions are generated in the final phase of the knowledge extraction modules in the Qurator Curious Facts system. We are talking R code. That makes anyone's intervention there quite complicated - unless I do it. My

[Wikidata-bugs] [Maniphest] T277558: [Curious Facts] remove missing image message

2021-03-30 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. - Done. - To be deployed soon. TASK DETAIL https://phabricator.wikimedia.org/T277558 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GoranSMilovanovic Cc: GoranSMilovanovic, Aklapper, Lydia_Pintscher

[Wikidata-bugs] [Maniphest] T277556: [Curious Facts] give explanation for timestamp

2021-03-30 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. - Done. - To be deployed soon. TASK DETAIL https://phabricator.wikimedia.org/T277556 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GoranSMilovanovic Cc: GoranSMilovanovic, Aklapper, Lydia_Pintscher

[Wikidata-bugs] [Maniphest] T277554: [Curious Facts] reduce Qurator branding

2021-03-30 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. - Done. - To be deployed soon. TASK DETAIL https://phabricator.wikimedia.org/T277554 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GoranSMilovanovic Cc: GoranSMilovanovic, Aklapper, Lydia_Pintscher

[Wikidata-bugs] [Maniphest] T272192: Migrate to new Wikidata Analytics

2021-03-30 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. @diego The transition process took almost two months, I think. During the transition process the old dashboards pointed clearly to their new counterparts... The old proxy is also down and the respective CloudVPS instance is being repurposed for other

[Wikidata-bugs] [Maniphest] T259105: Qurator: Data about Current Events

2021-03-25 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. @Lydia_Pintscher Fixed: see dashboard <https://wikidata-analytics.wmcloud.org/app/QURATOR_CurrentEvents>. TASK DETAIL https://phabricator.wikimedia.org/T259105 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailprefe

[Wikidata-bugs] [Maniphest] T270109: Hoover Inequality Score Data Retrival and Calculation

2021-03-21 Thread GoranSMilovanovic
GoranSMilovanovic closed this task as "Resolved". GoranSMilovanovic added a comment. We're in production: https://wikidata-analytics.wmcloud.org/app/WikidataAnalytics Closing as resolved. TASK DETAIL https://phabricator.wikimedia.org/T270109 EMAIL PREFERENC

[Wikidata-bugs] [Maniphest] T270109: Hoover Inequality Score Data Retrival and Calculation

2021-03-21 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. - In production on test server: http://datakolektiv.org/app/WD_Inequality - Deploying to Wikidata Analytics <https://wikidata-analytics.wmcloud.org/> now. TASK DETAIL https://phabricator.wikimedia.org/T270109 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T270109: Hoover Inequality Score Data Retrival and Calculation

2021-03-16 Thread GoranSMilovanovic
GoranSMilovanovic reopened this task as "Open". GoranSMilovanovic added a comment. @Lydia_Pintscher Re-opened; the system is (1) not in production state, and (2) not included to Wikidata Analytics yet. It should not take more than several hours to complete (1, 2)

[Wikidata-bugs] [Maniphest] T270109: Hoover Inequality Score Data Retrival and Calculation

2021-03-14 Thread GoranSMilovanovic
GoranSMilovanovic added a comment. @Lydia_Pintscher @Jan_Dittrich @WMDE-leszek Please help me clear my backlog a bit: this analytics dashboard, for example, needs a review before I productionize it. Thank you. TASK DETAIL https://phabricator.wikimedia.org/T270109 EMAIL PREFERENCES

<    1   2   3   4   5   6   7   8   9   >