[Wikidata-bugs] [Maniphest] [Commented On] T154601: Grafana: "wikidata-datamodel-terms" doesn't update anymore

Addshore added a comment. @Lydia_Pintscher with all of the changes this is starting to get toward something that it might make sense to have @GoranSMilovanovic tackle. TASK DETAIL https://phabricator.wikimedia.org/T154601 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T154601: Grafana: "wikidata-datamodel-terms" doesn't update anymore

Nuria added a comment. Then I suggest something like: https://wmcs-edits.wmflabs.org/#wmcs-edits this dashboard is public and powered by data that is extracted from hive. TASK DETAIL https://phabricator.wikimedia.org/T154601 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T154601: Grafana: "wikidata-datamodel-terms" doesn't update anymore

Lydia_Pintscher added a comment. In T154601#6269648 , @Addshore wrote: > @Lydia_Pintscher would it be okay for these dashboards to move to superset? (not public) As @Bugreporter said the editors should have access to this

[Wikidata-bugs] [Maniphest] [Commented On] T154601: Grafana: "wikidata-datamodel-terms" doesn't update anymore

Nuria added a comment. @Addshore you can query wikidata_entity from the SQL lab tb in superset. Superset is not public however so to access you need public authentication https://superset.wikimedia.org TASK DETAIL https://phabricator.wikimedia.org/T154601 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T154601: Grafana: "wikidata-datamodel-terms" doesn't update anymore

Addshore added a comment. We need queries like this: SELECT count(lang), lang FROM wmf.wikidata_entity LATERAL VIEW explode(labels) t AS lang, text WHERE snapshot='2020-03-02' GROUP BY lang LIMIT 100 So briefly chatting with WMF Analytics. > 7:33 PM

[Wikidata-bugs] [Maniphest] [Commented On] T154601: Grafana: "wikidata-datamodel-terms" doesn't update anymore

GoranSMilovanovic added a comment. @Ladsgroup @Addshore Do you need any help around this thing? TASK DETAIL https://phabricator.wikimedia.org/T154601 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GoranSMilovanovic Cc: Lucas_Werkmeister_WMDE,

[Wikidata-bugs] [Maniphest] [Commented On] T154601: Grafana: "wikidata-datamodel-terms" doesn't update anymore

Ladsgroup added a comment. In T154601#6202952 , @Addshore wrote: > In T154601#6202918 , @Lucas_Werkmeister_WMDE wrote: > >> That said, I tried to run the queries against stat1007.

[Wikidata-bugs] [Maniphest] [Commented On] T154601: Grafana: "wikidata-datamodel-terms" doesn't update anymore

Addshore added a comment. In T154601#6202918 , @Lucas_Werkmeister_WMDE wrote: > That said, I tried to run the queries against stat1007. The properties query completes in 18 seconds, that’s fine. But the items query just kept running,

[Wikidata-bugs] [Maniphest] [Commented On] T154601: Grafana: "wikidata-datamodel-terms" doesn't update anymore

Lucas_Werkmeister_WMDE added a comment. The above change implements Amir’s suggestion, I think – stick to the usual analytics-wmde-scripts methods, just with a different SQL query. The PHP code isn’t tested at all yet because I don’t know how to test it. That said, I tried to run the

[Wikidata-bugs] [Maniphest] [Commented On] T154601: Grafana: "wikidata-datamodel-terms" doesn't update anymore

gerritbot added a comment. Change 603557 had a related patch set uploaded (by Lucas Werkmeister (WMDE); owner: Lucas Werkmeister (WMDE)): [analytics/wmde/scripts@master] Fix and reenable terms_by_language script https://gerrit.wikimedia.org/r/603557 TASK DETAIL

[Wikidata-bugs] [Maniphest] [Commented On] T154601: Grafana: "wikidata-datamodel-terms" doesn't update anymore

Ladsgroup added a comment. I won't be around but also I want to mention I'm not sure reportupdater would be a great idea here. We are not using the codebase, reportupdater is not under active development (44 commits according to github), its yaml files are scattered everywhere, we have our

[Wikidata-bugs] [Maniphest] [Commented On] T154601: Grafana: "wikidata-datamodel-terms" doesn't update anymore

Addshore added a comment. Moving back to TODO and removing my face as I haven't started writing any code yet and won't be on camp from next week. TASK DETAIL https://phabricator.wikimedia.org/T154601 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/

[Wikidata-bugs] [Maniphest] [Commented On] T154601: Grafana: "wikidata-datamodel-terms" doesn't update anymore

Addshore added a comment. A tech task, but I think this mainly needs to be in the product prioritization list TASK DETAIL https://phabricator.wikimedia.org/T154601 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Addshore Cc: Bugreporter,

[Wikidata-bugs] [Maniphest] [Commented On] T154601: Grafana: "wikidata-datamodel-terms" doesn't update anymore

Addshore added a comment. This (and the rest of these analytics scripts) should be re written to use https://wikitech.wikimedia.org/wiki/Analytics/Systems/Reportupdater#The_graphite_section TASK DETAIL https://phabricator.wikimedia.org/T154601 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T154601: Grafana: "wikidata-datamodel-terms" doesn't update anymore

Addshore added a comment. In T154601#5985576 , @Bugreporter wrote: > What's the status of this after wb_terms is removed? It still will not update and the data collection will still need to be re written. TASK DETAIL

[Wikidata-bugs] [Maniphest] [Commented On] T154601: Grafana: "wikidata-datamodel-terms" doesn't update anymore

Bugreporter added a comment. What's the status of this after wb_terms is removed? TASK DETAIL https://phabricator.wikimedia.org/T154601 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Bugreporter Cc: Bugreporter, WMDE-leszek, Aleksey_WMDE,

[Wikidata-bugs] [Maniphest] [Commented On] T154601: Grafana: "wikidata-datamodel-terms" doesn't update anymore

GoranSMilovanovic added a comment. Ok. Someone please ping me know when the relevant metrics are instantiated in Graphite. @Addshore As of the comments in respect to T154601#4275121 (wb_terms -> Big Data), well, obviously because it makes more sense to work with a proper Big Data solution than to

[Wikidata-bugs] [Maniphest] [Commented On] T154601: Grafana: "wikidata-datamodel-terms" doesn't update anymore

Addshore added a comment. In T154601#4275121, @GoranSMilovanovic wrote: @Addshore After reconsidering this, I have to state openly that I am against relying on JSON dumps as the only source of data. Why? @WMDE-leszek @Aleksey_WMDE will also be interested to hear, I guess. Well: Given the

[Wikidata-bugs] [Maniphest] [Commented On] T154601: Grafana: "wikidata-datamodel-terms" doesn't update anymore

GoranSMilovanovic added a comment. @Lydia_Pintscher @Addshore @WMDE-leszek @Aleksey_WMDE Putting aside the question of the afterlife of the SQL wb_terms table for now: is there anything that can be done to fix this Dashboard anytime soon, or we wait for a new data engineering solution for the

[Wikidata-bugs] [Maniphest] [Commented On] T154601: Grafana: "wikidata-datamodel-terms" doesn't update anymore

GoranSMilovanovic added a comment. @Lydia_Pintscher I'm on it.TASK DETAILhttps://phabricator.wikimedia.org/T154601EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: GoranSMilovanovicCc: Ivanhercaz, VIGNERON, Lydia_Pintscher, GoranSMilovanovic, gerritbot,

[Wikidata-bugs] [Maniphest] [Commented On] T154601: Grafana: "wikidata-datamodel-terms" doesn't update anymore

gerritbot added a comment. Change 330657 merged by jenkins-bot: Disable temrs_by_language in cron https://gerrit.wikimedia.org/r/330657TASK DETAILhttps://phabricator.wikimedia.org/T154601EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: gerritbotCc: gerritbot,

[Wikidata-bugs] [Maniphest] [Commented On] T154601: Grafana: "wikidata-datamodel-terms" doesn't update anymore

gerritbot added a comment. Change 330656 merged by Addshore: Disable temrs_by_language in cron https://gerrit.wikimedia.org/r/330656TASK DETAILhttps://phabricator.wikimedia.org/T154601EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: gerritbotCc: gerritbot,

[Wikidata-bugs] [Maniphest] [Commented On] T154601: Grafana: "wikidata-datamodel-terms" doesn't update anymore

gerritbot added a comment. Change 330656 had a related patch set uploaded (by Addshore): Disable temrs_by_language in cron https://gerrit.wikimedia.org/r/330656TASK DETAILhttps://phabricator.wikimedia.org/T154601EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T154601: Grafana: "wikidata-datamodel-terms" doesn't update anymore

gerritbot added a comment. Change 330657 had a related patch set uploaded (by Addshore): Disable temrs_by_language in cron https://gerrit.wikimedia.org/r/330657TASK DETAILhttps://phabricator.wikimedia.org/T154601EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T154601: Grafana: "wikidata-datamodel-terms" doesn't update anymore

Addshore added a comment. PHP Fatal error: Uncaught exception 'RuntimeException' with message 'Something went wrong with the db query' in /a/analytics-wmde/src/scripts/src/wikidata/datamodel/terms_by_language.php:24 Stack trace: #0

[Wikidata-bugs] [Maniphest] [Commented On] T154601: Grafana: "wikidata-datamodel-terms" doesn't update anymore

Addshore added a comment. It looks like this might be too much for the DB to handle and this should be moved to reading from the dump (weekly)TASK DETAILhttps://phabricator.wikimedia.org/T154601EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: AddshoreCc: