[Wikidata] EveryBookItsReader problem with banned books in Wikidata

[Nuria Ferran Ferrer]

Hi!

We are working in the international campaign of this April "Every Book its 
Reader" https://meta.wikimedia.org/wiki/EveryBooksItsReader_2023 and we wanted 
to create or modify articles about banned books.

We realised that from all the different classes that are related to banned 
books, we think that only Classe Index Librorum Prohibitorum 
(http://www.wikidata.org/entity/Q22765) provides the especific works (books) 
that have really been banned. For example if we query https://w.wiki/6Uwy and 
we take for example the 1st result Paradise Lost 
(http://www.wikidata.org/entity/Q28754) we find the statement part of Index 
Librorum Prohibitorum. Most of the other classes what they do is providing 
links to lists in Wikipedia about banned books. We think that some work needs 
to be done in order to clarify the components of Wikidata ontologies about 
banned books. And we think there is the need to transfer data from Wikipedia 
(for example one list about banned books like this 
https://de.wikipedia.org/wiki/Liste_von_Autoren,_deren_Werke_auf_dem_Index_librorum_prohibitorum_standen)
 into Wikidata and make sure that the class banned books is provided.

Is there anybody that will like to work with this challenge?
Thanks,



Núria Ferran Ferrer

Professora de la Facultat d'Informació i Mitjans Audiovisuals (FIMA)

Departament de Biblioteconomia, Documentació i Comunicació Audiovisual

Centre de Recerca en Informació, Comunicació i Cultura (CRICC)

Comissió d'Igualtat de la Facultat

Universitat de Barcelona (UB)

Melcior de Palau, 140
08014 Barcelona


IP Projecte de Recerca: Women

Co-directora de la revista BiD

[cid:45b7b38b-84fa-41cf-9186-ab62c33ba3bc]





Aquest missatge, i els fitxers adjunts que hi pugui haver, pot contenir 
informació confidencial o protegida legalment i s'adreça exclusivament a la 
persona o entitat destinatària. Si no consteu com a destinatari final o no 
teniu l'encàrrec de rebre'l, no esteu autoritzat a llegir-lo, retenir-lo, 
modificar-lo, distribuir-lo, copiar-lo ni a revelar-ne el contingut. Si l'heu 
rebut per error, informeu-ne el remitent i elimineu del sistema tant el 
missatge com els fitxers adjunts que hi pugui haver.

Este mensaje, y los ficheros adjuntos que pueda incluir, puede contener 
información confidencial o legalmente protegida y está exclusivamente dirigido 
a la persona o entidad destinataria. Si usted no consta como destinatario final 
ni es la persona encargada de recibirlo, no está autorizado a leerlo, 
retenerlo, modificarlo, distribuirlo o copiarlo, ni a revelar su contenido. Si 
lo ha recibido por error, informe de ello al remitente y elimine del sistema 
tanto el mensaje como los ficheros adjuntos que pueda contener.

This email message and any attachments it carries may contain confidential or 
legally protected material and are intended solely for the individual or 
organization to whom they are addressed. If you are not the intended recipient 
of this message or the person responsible for processing it, then you are not 
authorized to read, save, modify, send, copy or disclose any part of it. If you 
have received the message by mistake, please inform the sender of this and 
eliminate the message and any attachments it carries from your account.
___
Wikidata mailing list -- wikidata@lists.wikimedia.org
Public archives at 
https://lists.wikimedia.org/hyperkitty/list/wikidata@lists.wikimedia.org/message/XH6XP2TQS57KVNGRCJJQQ7HCLJAFTTHX/
To unsubscribe send an email to wikidata-le...@lists.wikimedia.org

[Wikidata] Re: Fwd: Upcoming Wikidata & Wikibase office hours on Wednesday, November 9th 2022 at 17:00 UTC (18:00 Berlin) in the Wikidata Telegram group

[Nuria Ferran Ferrer]

Dear Lea,
do you have a calendar of the office hours for this month?
We are thinking of presenting a proposal for the WMF Research Grant and it will 
help us to present our project and receive feedback.
Thanks!
Núria


Núria Ferran Ferrer

Professora de la Facultat d'Informació i Mitjans Audiovisuals (FIMA)

Departament de Biblioteconomia, Documentació i Comunicació Audiovisual

Centre de Recerca en Informació, Comunicació i Cultura (CRICC)

Comissió d'Igualtat de la Facultat

Universitat de Barcelona (UB)

Melcior de Palau, 140
08014 Barcelona


IP Projecte de Recerca: Women

Co-directora de la revista BiD

[cid:5414144c-85be-4a04-a956-2e476ace0f70]




De: Léa Lacroix 
Enviat el: dilluns, 7 de novembre de 2022 12:53
Per a: wikidata@lists.wikimedia.org ; 
wikibas...@lists.wikimedia.org ; 
wikibase-cl...@lists.wikimedia.org 
Tema: [Wikidata] Fwd: Upcoming Wikidata & Wikibase office hours on Wednesday, 
November 9th 2022 at 17:00 UTC (18:00 Berlin) in the Wikidata Telegram group

Hello all,

As a reminder, the Wikidata and Wikibase office hours will take place on this 
Wednesday, November 9th 2022 at 17:00 UTC (18:00 Berlin time) in the Wikidata 
Telegram group.

The notes of these discussions will be transferred 
onwiki as usual.

Best,
--
Léa Lacroix
Community Engagement & Events Consultant
Contractor for Wikimedia Deutschland e.V.

-- Forwarded message -
From: Mohammed Sadat Abdulai 
mailto:mohammed.abdu...@wikimedia.de>>
Date: Mon, 24 Oct 2022, 10:24
Subject: [Wikibase] Upcoming Wikidata & Wikibase office hours on Wednesday, 
November 9th 2022 at 17:00 UTC (18:00 Berlin) in the Wikidata Telegram group
To: Discussion list for the Wikidata project 
mailto:wikidata@lists.wikimedia.org>>, Wikibase 
Community User Group 
mailto:wikibas...@lists.wikimedia.org>>, 
Wikibase-Cloud mailing list 
mailto:wikibase-cl...@lists.wikimedia.org>>


(apologies for cross-posting)

Hi everyone,

The next Wikidata+Wikibase office 
hours will take 
place on Wednesday, November 9th 2022 at 17:00 UTC (18:00 Berlin) in the 
Wikidata Telegram group.

The Wikidata and Wikibase office hours are online events where the development 
team presents what we have been working on over the past quarter, and the 
community is welcome to ask questions and discuss important issues related to 
the development of Wikidata and Wikibase.

I am looking forward to seeing you all.

--
Mohammed Sadat
Community Communications Manager for Wikidata



Aquest missatge, i els fitxers adjunts que hi pugui haver, pot contenir 
informació confidencial o protegida legalment i s'adreça exclusivament a la 
persona o entitat destinatària. Si no consteu com a destinatari final o no 
teniu l'encàrrec de rebre'l, no esteu autoritzat a llegir-lo, retenir-lo, 
modificar-lo, distribuir-lo, copiar-lo ni a revelar-ne el contingut. Si l'heu 
rebut per error, informeu-ne el remitent i elimineu del sistema tant el 
missatge com els fitxers adjunts que hi pugui haver.

Este mensaje, y los ficheros adjuntos que pueda incluir, puede contener 
información confidencial o legalmente protegida y está exclusivamente dirigido 
a la persona o entidad destinataria. Si usted no consta como destinatario final 
ni es la persona encargada de recibirlo, no está autorizado a leerlo, 
retenerlo, modificarlo, distribuirlo o copiarlo, ni a revelar su contenido. Si 
lo ha recibido por error, informe de ello al remitente y elimine del sistema 
tanto el mensaje como los ficheros adjuntos que pueda contener.

This email message and any attachments it carries may contain confidential or 
legally protected material and are intended solely for the individual or 
organization to whom they are addressed. If you are not the intended recipient 
of this message or the person responsible for processing it, then you are not 
authorized to read, save, modify, send, copy or disclose any part of it. If you 
have received the message by mistake, please inform the sender of this and 
eliminate the message and any attachments it carries from your account.
___
Wikidata mailing list -- wikidata@lists.wikimedia.org
Public archives at 
https://lists.wikimedia.org/hyperkitty/list/wikidata@lists.wikimedia.org/message/6JOQOWQY3XUQZQXR4MBILBTWOV6I5PDJ/
To unsubscribe send an email to wikidata-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T249654: Categorize different types of Wikidata re-use within Wikimedia projects

[Nuria]

Nuria added a comment.


  And, forgot to say, THIS IS SUPER USEFUL, thanks!

TASK DETAIL
  https://phabricator.wikimedia.org/T249654

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Isaac, Nuria
Cc: Akuckartz, calbon, Addshore, Lydia_Pintscher, Nuria, MGerlach, 
GoranSMilovanovic, Isaac, Liuxinyu970226, darthmon_wmde, Nandana, Abdeaitali, 
Lahi, Gq86, QZanden, LawExplorer, Avner, _jensen, rosalieper, Scott_WUaS, 
Wikidata-bugs, aude, Capt_Swing, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T249654: Categorize different types of Wikidata re-use within Wikimedia projects

[Nuria]

Nuria added a comment.


  > No transclusion: in line with current estimates, 41% of articles had no 
templates that support transclusion whatsoever
  
  This is "overall articles for all projects", correct?
  
  > Overlapping with the other categories, 54% of articles additionally use 
Wikidata descriptions in Search on the mobile apps
  
  How is this calculated?
  
  Do you have your selects to group by "importance" using   wbc_entity_usage?

TASK DETAIL
  https://phabricator.wikimedia.org/T249654

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Isaac, Nuria
Cc: Akuckartz, calbon, Addshore, Lydia_Pintscher, Nuria, MGerlach, 
GoranSMilovanovic, Isaac, Liuxinyu970226, darthmon_wmde, Nandana, Abdeaitali, 
Lahi, Gq86, QZanden, LawExplorer, Avner, _jensen, rosalieper, Scott_WUaS, 
Wikidata-bugs, aude, Capt_Swing, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T249654: Categorize different types of Wikidata re-use within Wikimedia projects

[Nuria]

Nuria added a subscriber: calbon.
Nuria added a comment.


  cc @calbon so he is aware of this reserach

TASK DETAIL
  https://phabricator.wikimedia.org/T249654

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Isaac, Nuria
Cc: calbon, Addshore, Lydia_Pintscher, Nuria, MGerlach, GoranSMilovanovic, 
Isaac, Liuxinyu970226, Akuckartz, darthmon_wmde, Nandana, Abdeaitali, Lahi, 
Gq86, QZanden, LawExplorer, Avner, _jensen, rosalieper, Scott_WUaS, 
Wikidata-bugs, aude, Capt_Swing, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T154601: Grafana: "wikidata-datamodel-terms" doesn't update anymore

[Nuria]

Nuria added a comment.


  Then I suggest something like: https://wmcs-edits.wmflabs.org/#wmcs-edits 
this dashboard is public and powered by data that is extracted from hive.

TASK DETAIL
  https://phabricator.wikimedia.org/T154601

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Addshore, Nuria
Cc: Nuria, Lucas_Werkmeister_WMDE, Ladsgroup, Bugreporter, WMDE-leszek, 
Aleksey_WMDE, Ivanhercaz, VIGNERON, Lydia_Pintscher, GoranSMilovanovic, 
gerritbot, Addshore, Sjoerddebruin, Aklapper, Alter-paule, Beast1978, Un1tY, 
Hook696, Iflorez, darthmon_wmde, Kent7301, alaa_wmde, joker88john, CucyNoiD, 
Nandana, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
QZanden, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, 
Jonas, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T154601: Grafana: "wikidata-datamodel-terms" doesn't update anymore

[Nuria]

Nuria added a comment.


  @Addshore you can query wikidata_entity from the SQL lab tb in superset. 
Superset is not public however  so to access you need public authentication 
https://superset.wikimedia.org

TASK DETAIL
  https://phabricator.wikimedia.org/T154601

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Addshore, Nuria
Cc: Nuria, Lucas_Werkmeister_WMDE, Ladsgroup, Bugreporter, WMDE-leszek, 
Aleksey_WMDE, Ivanhercaz, VIGNERON, Lydia_Pintscher, GoranSMilovanovic, 
gerritbot, Addshore, Sjoerddebruin, Aklapper, Alter-paule, Beast1978, Un1tY, 
Hook696, Iflorez, darthmon_wmde, Kent7301, alaa_wmde, joker88john, CucyNoiD, 
Nandana, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
QZanden, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, 
Jonas, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T253753: Increase retention for mediawiki.revision-create on the kafka jumbo cluster

[Nuria]

Nuria added a comment.


  Given that retention is not on puppet is this a setting that is communicated 
to a new node when it joins the cluster by the leader of the partition or 
similar?

TASK DETAIL
  https://phabricator.wikimedia.org/T253753

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata, Nuria
Cc: Nuria, JAllemandou, Ottomata, dcausse, Aklapper, CBogen, 4748kitoko, 
darthmon_wmde, Nandana, Namenlos314, Akovalyov, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, terrrydactyl, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Unblock] T244590: EPIC: Rework the WDQS updater as an event driven application

[Nuria]

Nuria closed subtask T253753: Increase retention for mediawiki.revision-create 
on the kafka jumbo cluster as Resolved.

TASK DETAIL
  https://phabricator.wikimedia.org/T244590

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Nuria
Cc: revi, Mholloway, Ladsgroup, Multichill, darthmon_wmde, Iamamz3, Smalyshev, 
Ottomata, JAllemandou, Aklapper, Zbyszko, Gehel, dcausse, NavinRizwi, CBogen, 
DannyS712, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Dinoguy1000, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Closed] T253753: Increase retention for mediawiki.revision-create on the kafka jumbo cluster

[Nuria]

Nuria closed this task as "Resolved".

TASK DETAIL
  https://phabricator.wikimedia.org/T253753

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata, Nuria
Cc: Nuria, JAllemandou, Ottomata, dcausse, Aklapper, CBogen, 4748kitoko, 
darthmon_wmde, Nandana, Namenlos314, Akovalyov, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, terrrydactyl, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Declined] T59379: Wikistats for Wikidata lists several bots as normal users

[Nuria]

Nuria closed this task as "Declined".
Restricted Application removed a subscriber: Liuxinyu970226.

TASK DETAIL
  https://phabricator.wikimedia.org/T59379

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Nuria
Cc: Addshore, Aklapper, QChris, ezachte, 4748kitoko, Saimongoltini, 
darthmon_wmde, Jsc39, Nandana, Akovalyov, Lahi, Gq86, Bsandipan, 
GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, TheDaveRoss, 
Scott_WUaS, JAllemandou, terrrydactyl, Wikidata-bugs, aude, jeblad, Nemo_bis, 
Mbch331, jeremyb, Liuxinyu970226
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Unblock] T169798: Create UDFs for analyzing SPARQL queries

[Nuria]

Nuria closed subtask T164020: Use spark to split webrequest on tags as 
Declined.

TASK DETAIL
  https://phabricator.wikimedia.org/T169798

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Nuria
Cc: PokestarFan, gerritbot, AndrewSu, Nuria, Aklapper, Smalyshev, 
darthmon_wmde, ET4Eva, Nandana, Lahi, Gq86, Darkminds3113, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Avner, Gehel, _jensen, rosalieper, Scott_WUaS, Jonas, FloNight, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Closed] T236895: ArticlePlaceholder dashboard stopped tracking page views

[Nuria]

Nuria closed this task as "Resolved".

TASK DETAIL
  https://phabricator.wikimedia.org/T236895

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ladsgroup, Nuria
Cc: mforns, Milimetric, Ladsgroup, Nuria, JAllemandou, elukey, Addshore, 
Aklapper, Lydia_Pintscher, Alter-paule, Hazizibinmahdi, Beast1978, Un1tY, 
4748kitoko, Hook696, Daryl-TTMG, RomaAmorRoma, E.S.A-Sheild, Iflorez, 
darthmon_wmde, alaa_wmde, Meekrab2012, joker88john, CucyNoiD, Nandana, 
NebulousIris, Akovalyov, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, 
Adrian1985, Cpaulf30, Lahi, Gq86, Af420, Darkminds3113, Bsandipan, Lordiis, 
GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, cmadeo, 
LawExplorer, WSH1906, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, 
Jonas, terrrydactyl, Wikidata-bugs, aude, jayvdb, Ricordisamoa, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T247058: Deployment strategy and hardware requirement for new Flink based WDQS updater

[Nuria]

Nuria added a comment.


  I think it will be very helpful to have a design document for this service so 
we are all in the same page of what the flink install would do (as there are 
other projects currently evaluating flink as well). Can we get  a google doc 
that goes over the design proposed with design diagrams and such?

TASK DETAIL
  https://phabricator.wikimedia.org/T247058

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Nuria
Cc: Nuria, Addshore, Milimetric, JAllemandou, Ottomata, Pchelolo, Joe, 
Aklapper, dcausse, Zbyszko, Gehel, 4748kitoko, darthmon_wmde, Legado_Shulgin, 
Nandana, Akovalyov, Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, 
merbst, LawExplorer, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
Wong128hk, terrrydactyl, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, Manybubbles, faidon, Mbch331, Rxy, Jay8g, fgiunchedi, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Closed] T209655: Copy Wikidata dumps to HDFS + parquet

[Nuria]

Nuria closed this task as "Resolved".

TASK DETAIL
  https://phabricator.wikimedia.org/T209655

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: JAllemandou, Nuria
Cc: Isaac, Groceryheist, MGerlach, WMDE-leszek, abian, leila, Ottomata, Nuria, 
GoranSMilovanovic, Addshore, JAllemandou, bmansurov, Beast1978, Un1tY, 
4748kitoko, Hook696, Daryl-TTMG, RomaAmorRoma, E.S.A-Sheild, darthmon_wmde, 
Meekrab2012, joker88john, CucyNoiD, Nandana, NebulousIris, Akovalyov, Gaboe420, 
Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, 
Af420, Darkminds3113, Bsandipan, Lordiis, Adik2382, Th3d3v1ls, Ramalepe, 
Liugev6, QZanden, LawExplorer, WSH1906, Lewizho99, Maathavan, _jensen, 
rosalieper, Scott_WUaS, terrrydactyl, Wikidata-bugs, aude, Capt_Swing, Mbch331, 
jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Unblock] T209655: Copy Wikidata dumps to HDFS + parquet

[Nuria]

Nuria closed subtask T243832: Fix hdfs-rsync`prune-empty-dirs` feature as 
Resolved.

TASK DETAIL
  https://phabricator.wikimedia.org/T209655

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: JAllemandou, Nuria
Cc: Isaac, Groceryheist, MGerlach, WMDE-leszek, abian, leila, Ottomata, Nuria, 
GoranSMilovanovic, Addshore, JAllemandou, bmansurov, Beast1978, Un1tY, 
4748kitoko, Hook696, Daryl-TTMG, RomaAmorRoma, E.S.A-Sheild, darthmon_wmde, 
Meekrab2012, joker88john, CucyNoiD, Nandana, NebulousIris, Akovalyov, Gaboe420, 
Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, 
Af420, Darkminds3113, Bsandipan, Lordiis, Adik2382, Th3d3v1ls, Ramalepe, 
Liugev6, QZanden, LawExplorer, WSH1906, Lewizho99, Maathavan, _jensen, 
rosalieper, Scott_WUaS, terrrydactyl, Wikidata-bugs, aude, Capt_Swing, Mbch331, 
jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T174981: Add pageviews total counts to WDQS

[Nuria]

Nuria added a comment.


  I think before talking about bytes you need a use case, what is the use case 
here? As we mentioned earlier the GLAM folks care about human pageviews (real 
eye balls)  on media files and pages, both cases are (and will be better) 
satisfied by existing analytics APIs.  What is the use for this request?

TASK DETAIL
  https://phabricator.wikimedia.org/T174981

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Nuria
Cc: Gehel, Zache, Nuria, Tagishsimon, Elya, christophbraun, Bovlb, thalhamm, 
EBernhardson, Esc3300, Lydia_Pintscher, Yair_rand, Smalyshev, Aklapper, Yurik, 
4748kitoko, darthmon_wmde, ET4Eva, Nandana, Akovalyov, Lahi, Gq86, 
Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, 
merbst, LawExplorer, Avner, _jensen, rosalieper, Scott_WUaS, Jonas, FloNight, 
Xmlizer, JAllemandou, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, Manybubbles, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Closed] T239565: Create reportupdater reports that execute SDC requests

[Nuria]

Nuria closed this task as "Resolved".

TASK DETAIL
  https://phabricator.wikimedia.org/T239565

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Milimetric, Nuria
Cc: Abit, Ramsey-WMF, kzimmerman, Addshore, matthiasmullie, gsingers, 
Mayakp.wiki, Ladsgroup, nettrom_WMF, Cparle, Nuria, Milimetric, mpopov, 
4748kitoko, darthmon_wmde, Nandana, JKSTNK, Akovalyov, Lahi, PDrouin-WMF, Gq86, 
E1presidente, Anooprao, SandraF_WMF, GoranSMilovanovic, QZanden, Tramullas, 
Acer, LawExplorer, Salgo60, Silverfish, _jensen, rosalieper, Scott_WUaS, 
Susannaanas, JAllemandou, Jane023, terrrydactyl, Wikidata-bugs, Base, aude, 
Ricordisamoa, Wesalius, Lydia_Pintscher, Fabrice_Florin, Raymond, 
Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Unblock] T238878: Data about how many file pages on Commons contain at least one structured data element

[Nuria]

Nuria closed subtask T239565: Create reportupdater reports that execute SDC 
requests  as Resolved.

TASK DETAIL
  https://phabricator.wikimedia.org/T238878

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Nuria
Cc: Milimetric, Cparle, nettrom_WMF, Ladsgroup, daniel, Mayakp.wiki, gsingers, 
matthiasmullie, Addshore, kzimmerman, mpopov, Ramsey-WMF, Abit, Nuria, 
4748kitoko, darthmon_wmde, Nandana, JKSTNK, Akovalyov, Lahi, PDrouin-WMF, Gq86, 
E1presidente, Anooprao, SandraF_WMF, GoranSMilovanovic, QZanden, Tramullas, 
Acer, LawExplorer, Salgo60, Silverfish, _jensen, rosalieper, Scott_WUaS, 
Susannaanas, JAllemandou, Jane023, terrrydactyl, Wikidata-bugs, Base, aude, 
Ricordisamoa, Wesalius, Lydia_Pintscher, Fabrice_Florin, Raymond, 
Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T174981: Add pageviews total counts to WDQS

[Nuria]

Nuria added a comment.


  Please see: 
https://stats.wikimedia.org/v2/#/wikidata.org/reading/total-page-views/normal|bar|2-year|agent~user*spider|monthly

TASK DETAIL
  https://phabricator.wikimedia.org/T174981

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Nuria
Cc: Nuria, Tagishsimon, Elya, christophbraun, Bovlb, thalhamm, EBernhardson, 
Esc3300, Lydia_Pintscher, Yair_rand, Smalyshev, Aklapper, Yurik, 4748kitoko, 
darthmon_wmde, ET4Eva, Nandana, Akovalyov, Lahi, Gq86, Darkminds3113, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Avner, Gehel, _jensen, rosalieper, Scott_WUaS, Jonas, FloNight, 
Xmlizer, JAllemandou, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, Manybubbles, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T174981: Add pageviews total counts to WDQS

[Nuria]

Nuria added a comment.


  @christophbraun I think it would help to start a ticket describing your use 
case in detail. Have in mind that pageviews (defined as content consumed by 
humans) do not really "apply" to wikidata items. The bulk of the activity on 
the site around http requests has a lot to do with bots creating/consuming 
content. The percentage of bots requests is about 70/80%

TASK DETAIL
  https://phabricator.wikimedia.org/T174981

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Nuria
Cc: Nuria, Tagishsimon, Elya, christophbraun, Bovlb, thalhamm, EBernhardson, 
Esc3300, Lydia_Pintscher, Yair_rand, Smalyshev, Aklapper, Yurik, 4748kitoko, 
darthmon_wmde, ET4Eva, Nandana, Akovalyov, Lahi, Gq86, Darkminds3113, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Avner, Gehel, _jensen, rosalieper, Scott_WUaS, Jonas, FloNight, 
Xmlizer, JAllemandou, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, Manybubbles, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T174981: Add pageviews total counts to WDQS

[Nuria]

Nuria added a comment.


  Updating WDQS (a relational query engine) with metadata about pageviews (per 
definition a timeseries) seems not the best idea from a data modeling 
standpoint.  The GLAM use case is much better served by an API that returns 
pageviews across time, I would put the engineering effort on building such API.

TASK DETAIL
  https://phabricator.wikimedia.org/T174981

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Nuria
Cc: Nuria, Tagishsimon, Elya, christophbraun, Bovlb, thalhamm, EBernhardson, 
Esc3300, Lydia_Pintscher, Yair_rand, Smalyshev, Aklapper, Yurik, 4748kitoko, 
darthmon_wmde, ET4Eva, Nandana, Akovalyov, Lahi, Gq86, Darkminds3113, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Avner, Gehel, _jensen, rosalieper, Scott_WUaS, Jonas, FloNight, 
Xmlizer, JAllemandou, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, Manybubbles, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T238878: Data about how many file pages on Commons contain at least one structured data element

[Nuria]

Nuria added a comment.


  > I'd like a clear explanation as the multiple threads have become 
complicated to resolve objectively.
  
  Indeed.
  @Milimetric we just want to have data for commons, for wikidata there are 
plenty metrics available that also measure usage from this table, see: 
https://grafana.wikimedia.org/d/00176/wikidata-entity-usage-project?orgId=1=17=now-5y=now

TASK DETAIL
  https://phabricator.wikimedia.org/T238878

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Nuria
Cc: Milimetric, Cparle, nettrom_WMF, Ladsgroup, daniel, Mayakp.wiki, gsingers, 
matthiasmullie, Addshore, kzimmerman, mpopov, Ramsey-WMF, Abit, Nuria, 
4748kitoko, darthmon_wmde, Nandana, JKSTNK, Akovalyov, Lahi, PDrouin-WMF, Gq86, 
E1presidente, Anooprao, SandraF_WMF, GoranSMilovanovic, QZanden, Tramullas, 
Acer, LawExplorer, Salgo60, Silverfish, _jensen, rosalieper, Scott_WUaS, 
Susannaanas, JAllemandou, Jane023, terrrydactyl, Wikidata-bugs, Base, aude, 
Ricordisamoa, Wesalius, Lydia_Pintscher, Fabrice_Florin, Raymond, 
Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T238878: Data about how many file pages on Commons contain at least one structured data element

[Nuria]

Nuria added a comment.


  @Addshore @matthiasmullie Can we document on 
https://www.mediawiki.org/wiki/Wikibase/Schema/wbc_entity_usage what the "M" in 
wbc_entity_usage  stands for?

TASK DETAIL
  https://phabricator.wikimedia.org/T238878

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Nuria
Cc: Milimetric, Cparle, nettrom_WMF, Ladsgroup, daniel, Mayakp.wiki, gsingers, 
matthiasmullie, Addshore, kzimmerman, mpopov, Ramsey-WMF, Abit, Nuria, 
4748kitoko, darthmon_wmde, Nandana, JKSTNK, Akovalyov, Lahi, PDrouin-WMF, Gq86, 
E1presidente, Anooprao, SandraF_WMF, GoranSMilovanovic, QZanden, Tramullas, 
Acer, LawExplorer, Salgo60, Silverfish, _jensen, rosalieper, Scott_WUaS, 
Susannaanas, JAllemandou, Jane023, terrrydactyl, Wikidata-bugs, Base, aude, 
Ricordisamoa, Wesalius, Lydia_Pintscher, Fabrice_Florin, Raymond, 
Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

[Nuria]

Nuria added a comment.


  > We would still like productionized reports for (3). If that is still 
possible, I would love to discuss it more :)
  
  Please coordinate with #product-analytics 
<https://phabricator.wikimedia.org/tag/product-analytics/>  on those.
  
  I found yet another ticket about SDC metrics in which is apparent that there 
is duplicated work happening. Let's please try to avoid that going forward.
  https://phabricator.wikimedia.org/T231952

TASK DETAIL
  https://phabricator.wikimedia.org/T239565

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Milimetric, Nuria
Cc: Abit, Ramsey-WMF, kzimmerman, Addshore, matthiasmullie, gsingers, 
Mayakp.wiki, Ladsgroup, nettrom_WMF, Cparle, Nuria, Milimetric, mpopov, 
4748kitoko, darthmon_wmde, DannyS712, Nandana, JKSTNK, Akovalyov, Lahi, 
PDrouin-WMF, Gq86, E1presidente, Anooprao, SandraF_WMF, GoranSMilovanovic, 
QZanden, Tramullas, Acer, LawExplorer, Salgo60, Silverfish, _jensen, 
rosalieper, Scott_WUaS, Susannaanas, JAllemandou, Jane023, terrrydactyl, 
Wikidata-bugs, Base, aude, Ricordisamoa, Wesalius, Lydia_Pintscher, 
Fabrice_Florin, Raymond, Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

[Nuria]

Nuria added a comment.


  @Abit:  Sorry it was not clear. This below is the request you send to 
analytics couple weeks ago via e-mail, as I mentioned then we rather work on 
requests via phab tickets that via e-mail.
  
  "
  From: Amanda Bittaker 
  Date: Tue, Nov 19, 2019 at 1:30 AM
  Subject: Analytics for structured data elements on Commons
  To: Joseph Allemandou 
  Cc: Ramsey Isler 
  
  We're wondering how many file pages on Commons contain at least one 
structured data element (statement or caption), and we hear you are the person 
to ask.
  
  Is this something you can help us set up analytics for?  If it wouldn't be 
too much effort, there are other structured data numbers it would be nice to 
have, but most urgently we just need to be able to regularly check the number 
of files with structured data.  (We have been using the search results, but 
search indexing is proving unreliable.)   "
  
  > Matthias is working on (1) in another ticket.
  
  Can you link that ticket so we do not duplicate efforts?

TASK DETAIL
  https://phabricator.wikimedia.org/T239565

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Milimetric, Nuria
Cc: Abit, Ramsey-WMF, kzimmerman, Addshore, matthiasmullie, gsingers, 
Mayakp.wiki, Ladsgroup, nettrom_WMF, Cparle, Nuria, Milimetric, mpopov, 
4748kitoko, darthmon_wmde, DannyS712, Nandana, JKSTNK, Akovalyov, Lahi, 
PDrouin-WMF, Gq86, E1presidente, Anooprao, SandraF_WMF, GoranSMilovanovic, 
QZanden, Tramullas, Acer, LawExplorer, Salgo60, Silverfish, _jensen, 
rosalieper, Scott_WUaS, Susannaanas, JAllemandou, Jane023, terrrydactyl, 
Wikidata-bugs, Base, aude, Ricordisamoa, Wesalius, Lydia_Pintscher, 
Fabrice_Florin, Raymond, Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

[Nuria]

Nuria added a comment.


  > So, it seems we have 2 completely separate definitions of "structured data":
  
  Well, the intent of these numbers is not to measure "structured data usage" 
nor to define that concept.  The intent is to measure the impact of the 
structure data on commons project per @Abit initial request couple weeks ago.
  
  With that objective in mind I do not see the Wikidata usage on commons to be 
of relevance as that usage it is not directly tied to the work the team working 
on "structure data on commons" is been doing. Please do let me know if this is 
not  a correct statement.
  
  As a side note I think the true interesting metric is the rate of growth of 
the result of the query below. Following rate of growth of usage of structured 
data on commons growths seems much more relevant to track impact than the 
absolute numbers we have been talking about to date.
  
  SELECT DISTINCT page_id
FROM page
INNER JOIN slots ON slot_revision_id = page_latest
INNER JOIN content ON slot_content_id = content_id AND content_size > 
122
INNER JOIN slot_roles ON role_id = slot_role_id AND role_name = 
'mediainfo'

TASK DETAIL
  https://phabricator.wikimedia.org/T239565

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Milimetric, Nuria
Cc: Abit, Ramsey-WMF, kzimmerman, Addshore, matthiasmullie, gsingers, 
Mayakp.wiki, Ladsgroup, nettrom_WMF, Cparle, Nuria, Milimetric, mpopov, 
4748kitoko, darthmon_wmde, DannyS712, Nandana, JKSTNK, Akovalyov, Lahi, 
PDrouin-WMF, Gq86, E1presidente, Anooprao, SandraF_WMF, GoranSMilovanovic, 
QZanden, Tramullas, Acer, LawExplorer, Salgo60, Silverfish, _jensen, 
rosalieper, Scott_WUaS, Susannaanas, JAllemandou, Jane023, terrrydactyl, 
Wikidata-bugs, Base, aude, Ricordisamoa, Wesalius, Lydia_Pintscher, 
Fabrice_Florin, Raymond, Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T238878: Data about how many file pages on Commons contain at least one structured data element

[Nuria]

Nuria added a comment.


  I see 7.9 million wikidata items on that table and 1936 mediawinfo items.

TASK DETAIL
  https://phabricator.wikimedia.org/T238878

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Nuria
Cc: Milimetric, Cparle, nettrom_WMF, Ladsgroup, daniel, Mayakp.wiki, gsingers, 
matthiasmullie, Addshore, kzimmerman, mpopov, Ramsey-WMF, Abit, Nuria, 
4748kitoko, darthmon_wmde, DannyS712, Nandana, JKSTNK, Akovalyov, Lahi, 
PDrouin-WMF, Gq86, E1presidente, Anooprao, SandraF_WMF, GoranSMilovanovic, 
QZanden, Tramullas, Acer, LawExplorer, Salgo60, Silverfish, _jensen, 
rosalieper, Scott_WUaS, Susannaanas, JAllemandou, Jane023, terrrydactyl, 
Wikidata-bugs, Base, aude, Ricordisamoa, Wesalius, Lydia_Pintscher, 
Fabrice_Florin, Raymond, Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T238878: Data about how many file pages on Commons contain at least one structured data element

[Nuria]

Nuria added a comment.


  > but by and large, it's currently wikidata entities being pulled in via Lua
  
  It is exclusively wikidata items at this time correct?

TASK DETAIL
  https://phabricator.wikimedia.org/T238878

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Nuria
Cc: Milimetric, Cparle, nettrom_WMF, Ladsgroup, daniel, Mayakp.wiki, gsingers, 
matthiasmullie, Addshore, kzimmerman, mpopov, Ramsey-WMF, Abit, Nuria, 
4748kitoko, darthmon_wmde, DannyS712, Nandana, JKSTNK, Akovalyov, Lahi, 
PDrouin-WMF, Gq86, E1presidente, Anooprao, SandraF_WMF, GoranSMilovanovic, 
QZanden, Tramullas, Acer, LawExplorer, Salgo60, Silverfish, _jensen, 
rosalieper, Scott_WUaS, Susannaanas, JAllemandou, Jane023, terrrydactyl, 
Wikidata-bugs, Base, aude, Ricordisamoa, Wesalius, Lydia_Pintscher, 
Fabrice_Florin, Raymond, Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T238878: Data about how many file pages on Commons contain at least one structured data element

[Nuria]

Nuria added a comment.


  Select for @mpopov to look at
  
select count(distinct eu_page_id)  from mediawiki_page as P JOIN 
mediawiki_wbc_entity_usage as W ON (W.eu_page_id = P.page_id and 
W.wiki_db=P.wiki_db and W.snapshot=P.snapshot)  where W.wiki_db="commonswiki" 
and W.snapshot="2019-11" and page_namespace = 6  and eu_aspect RLIKE '^L' and 
page_is_redirect=0;
  
  @mpopov

TASK DETAIL
  https://phabricator.wikimedia.org/T238878

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Nuria
Cc: Milimetric, Cparle, nettrom_WMF, Ladsgroup, daniel, Mayakp.wiki, gsingers, 
matthiasmullie, Addshore, kzimmerman, mpopov, Ramsey-WMF, Abit, Nuria, 
4748kitoko, darthmon_wmde, DannyS712, Nandana, JKSTNK, Akovalyov, Lahi, 
PDrouin-WMF, Gq86, E1presidente, Anooprao, SandraF_WMF, GoranSMilovanovic, 
QZanden, Tramullas, Acer, LawExplorer, Salgo60, Silverfish, _jensen, 
rosalieper, Scott_WUaS, Susannaanas, JAllemandou, Jane023, terrrydactyl, 
Wikidata-bugs, Base, aude, Ricordisamoa, Wesalius, Lydia_Pintscher, 
Fabrice_Florin, Raymond, Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

[Nuria]

Nuria added a comment.


  Please see my comment on https://phabricator.wikimedia.org/T238878#5726624 
Seems like the 7.9 million items are from contributions of wikidata alone.

TASK DETAIL
  https://phabricator.wikimedia.org/T239565

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Milimetric, Nuria
Cc: Abit, Ramsey-WMF, kzimmerman, Addshore, matthiasmullie, gsingers, 
Mayakp.wiki, Ladsgroup, nettrom_WMF, Cparle, Nuria, Milimetric, mpopov, 
4748kitoko, darthmon_wmde, DannyS712, Nandana, JKSTNK, Akovalyov, Lahi, 
PDrouin-WMF, Gq86, E1presidente, Anooprao, SandraF_WMF, GoranSMilovanovic, 
QZanden, Tramullas, Acer, LawExplorer, Salgo60, Silverfish, _jensen, 
rosalieper, Scott_WUaS, Susannaanas, JAllemandou, Jane023, terrrydactyl, 
Wikidata-bugs, Base, aude, Ricordisamoa, Wesalius, Lydia_Pintscher, 
Fabrice_Florin, Raymond, Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T238878: Data about how many file pages on Commons contain at least one structured data element

[Nuria]

Nuria added a comment.


  After looking at this for a bit with @Ladsgroup and @mpopov  (cc @Abit ) 
  A commons page can have data from wikidata and the wikibase instance on 
commons (called WikibaseMediaInfo, MediaInfor for short).
  
  wbc_entity_usage will not have any data from structured data on commons that 
is kept on WikibaseMediaInfo. The data on that table only pertains to wikidata 
items used by "wiki clients". "wiki clients" are users of iwkidata's data , 
like eswiki.  So,  to find "how many file pages on commons contain at least 1 
structured data element" looking at slots is sufficient. That should be 
inclusive of all commons files. That number is ~3 million.
  
  Now, there are pages in commons that use, for example, wikidata info boxes, 
like: https://commons.wikimedia.org/wiki/Category:Lucas_Werkmeister
  The pages using wikidata items on commons can be tracked with 
wbc_entity_usage and that number is 7.9 million as of today. Now, to be clear, 
the 7.9 pertains entirely to a different set of infrastructure, namely 
wikidata's.  None of those 7.9 structure data items represent any info on the 
WikibaseMediaInfo instance.
  
  Ping @matthiasmullie to asses this info

TASK DETAIL
  https://phabricator.wikimedia.org/T238878

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Nuria
Cc: Milimetric, Cparle, nettrom_WMF, Ladsgroup, daniel, Mayakp.wiki, gsingers, 
matthiasmullie, Addshore, kzimmerman, mpopov, Ramsey-WMF, Abit, Nuria, 
4748kitoko, darthmon_wmde, DannyS712, Nandana, JKSTNK, Akovalyov, Lahi, 
PDrouin-WMF, Gq86, E1presidente, Anooprao, SandraF_WMF, GoranSMilovanovic, 
QZanden, Tramullas, Acer, LawExplorer, Salgo60, Silverfish, _jensen, 
rosalieper, Scott_WUaS, Susannaanas, JAllemandou, Jane023, terrrydactyl, 
Wikidata-bugs, Base, aude, Ricordisamoa, Wesalius, Lydia_Pintscher, 
Fabrice_Florin, Raymond, Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

[Nuria]

Nuria added a comment.


  Adding here public pdf with structure data on commons grant proposal: 
https://upload.wikimedia.org/wikipedia/foundation/f/f0/Public_Copy_-_Structured_Data_on_Commons_Proposal.pdf

TASK DETAIL
  https://phabricator.wikimedia.org/T239565

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Milimetric, Nuria
Cc: Abit, Ramsey-WMF, kzimmerman, Addshore, matthiasmullie, gsingers, 
Mayakp.wiki, Ladsgroup, nettrom_WMF, Cparle, Nuria, Milimetric, mpopov, 
4748kitoko, darthmon_wmde, DannyS712, Nandana, JKSTNK, Akovalyov, Lahi, 
PDrouin-WMF, Gq86, E1presidente, Anooprao, SandraF_WMF, GoranSMilovanovic, 
QZanden, Tramullas, Acer, LawExplorer, Salgo60, Silverfish, _jensen, 
rosalieper, Scott_WUaS, Susannaanas, JAllemandou, Jane023, terrrydactyl, 
Wikidata-bugs, Base, aude, Ricordisamoa, Wesalius, Lydia_Pintscher, 
Fabrice_Florin, Raymond, Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

[Nuria]

Nuria added a comment.


  @Abit: The queries that report  the 7.8 million include , per @matthiasmullie 
comment both Wikidata Items and Mediainfo items. We can help calculate the 
percentage of each  but from numbers thus far it seems that of those 7.8M items 
more than half are Wikidata items. We can help further quantify that 7M number 
so we have  a precise definition of what we are reporting.

TASK DETAIL
  https://phabricator.wikimedia.org/T239565

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Milimetric, Nuria
Cc: Abit, Ramsey-WMF, kzimmerman, Addshore, matthiasmullie, gsingers, 
Mayakp.wiki, Ladsgroup, nettrom_WMF, Cparle, Nuria, Milimetric, mpopov, 
4748kitoko, darthmon_wmde, DannyS712, Nandana, JKSTNK, Akovalyov, Lahi, 
PDrouin-WMF, Gq86, E1presidente, Anooprao, SandraF_WMF, GoranSMilovanovic, 
QZanden, Tramullas, Acer, LawExplorer, Salgo60, Silverfish, _jensen, 
rosalieper, Scott_WUaS, Susannaanas, JAllemandou, Jane023, terrrydactyl, 
Wikidata-bugs, Base, aude, Ricordisamoa, Wesalius, Lydia_Pintscher, 
Fabrice_Florin, Raymond, Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

[Nuria]

Nuria added a comment.


  @abit: Numbers about SDC will be reported in the platform evolution slides.

TASK DETAIL
  https://phabricator.wikimedia.org/T239565

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Milimetric, Nuria
Cc: Abit, Ramsey-WMF, kzimmerman, Addshore, matthiasmullie, gsingers, 
Mayakp.wiki, Ladsgroup, nettrom_WMF, Cparle, Nuria, Milimetric, mpopov, 
4748kitoko, darthmon_wmde, DannyS712, Nandana, JKSTNK, Akovalyov, Lahi, 
PDrouin-WMF, Gq86, E1presidente, Anooprao, SandraF_WMF, GoranSMilovanovic, 
QZanden, Tramullas, Acer, LawExplorer, Salgo60, Silverfish, _jensen, 
rosalieper, Scott_WUaS, Susannaanas, JAllemandou, Jane023, terrrydactyl, 
Wikidata-bugs, Base, aude, Ricordisamoa, Wesalius, Lydia_Pintscher, 
Fabrice_Florin, Raymond, Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

[Nuria]

Nuria added a comment.


  It looks like we are going to have to report this number on the tunning 
session so taking back my comment above, let's proceed.

TASK DETAIL
  https://phabricator.wikimedia.org/T239565

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Milimetric, Nuria
Cc: Abit, Ramsey-WMF, kzimmerman, Addshore, matthiasmullie, gsingers, 
Mayakp.wiki, Ladsgroup, nettrom_WMF, Cparle, Nuria, Milimetric, mpopov, 
4748kitoko, darthmon_wmde, DannyS712, Nandana, JKSTNK, Akovalyov, Lahi, 
PDrouin-WMF, Gq86, E1presidente, Anooprao, SandraF_WMF, GoranSMilovanovic, 
QZanden, Tramullas, Acer, LawExplorer, Salgo60, Silverfish, _jensen, 
rosalieper, Scott_WUaS, Susannaanas, JAllemandou, Jane023, terrrydactyl, 
Wikidata-bugs, Base, aude, Ricordisamoa, Wesalius, Lydia_Pintscher, 
Fabrice_Florin, Raymond, Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

[Nuria]

Nuria added a comment.


  Let's pause this work as it turns out as there is a parallel effort happening 
 , @Abit to create a ticket for ongoing work

TASK DETAIL
  https://phabricator.wikimedia.org/T239565

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Milimetric, Nuria
Cc: Abit, Ramsey-WMF, kzimmerman, Addshore, matthiasmullie, gsingers, 
Mayakp.wiki, Ladsgroup, nettrom_WMF, Cparle, Nuria, Milimetric, mpopov, 
4748kitoko, darthmon_wmde, DannyS712, Nandana, JKSTNK, Akovalyov, Lahi, 
PDrouin-WMF, Gq86, E1presidente, Anooprao, SandraF_WMF, GoranSMilovanovic, 
QZanden, Tramullas, Acer, LawExplorer, Salgo60, Silverfish, _jensen, 
rosalieper, Scott_WUaS, Susannaanas, JAllemandou, Jane023, terrrydactyl, 
Wikidata-bugs, Base, aude, Ricordisamoa, Wesalius, Lydia_Pintscher, 
Fabrice_Florin, Raymond, Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

[Nuria]

Nuria added a comment.


  Some alternatives: superset can source data from other places than druid and 
we have couple dashboards on top of some tables in staging. This might not be 
the best option as reportupdater produces tsvs rather than inserting data into 
staging again. Druid is good for cubes but this is a simple metric that has 1 
dimension so with  the tsvs in mind would be easy to have a dashiki dashboard 
that would look similar to: 
https://analytics.wikimedia.org/dashboards/browsers/#all-sites-by-os
  
  The dashboard above reports "ratios per browser", if we have 2 metrics:1 ) 
"total_number_of_files" and another with 2) 
total_number_of_files_with_sdc_data/total_number_of_files

TASK DETAIL
  https://phabricator.wikimedia.org/T239565

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Milimetric, Nuria
Cc: Abit, Ramsey-WMF, kzimmerman, Addshore, matthiasmullie, gsingers, 
Mayakp.wiki, Ladsgroup, nettrom_WMF, Cparle, Nuria, Milimetric, mpopov, 
4748kitoko, darthmon_wmde, DannyS712, Nandana, JKSTNK, Akovalyov, Lahi, 
PDrouin-WMF, Gq86, E1presidente, Anooprao, SandraF_WMF, GoranSMilovanovic, 
QZanden, Tramullas, Acer, LawExplorer, Salgo60, Silverfish, _jensen, 
rosalieper, Scott_WUaS, Susannaanas, JAllemandou, Jane023, terrrydactyl, 
Wikidata-bugs, Base, aude, Ricordisamoa, Wesalius, Lydia_Pintscher, 
Fabrice_Florin, Raymond, Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Created] T239565: Create reportupdater reports that execute SDC requests

[Nuria]

Nuria created this task.
Nuria added projects: Analytics, Wikidata, SDC General, Product-Analytics, 
Analytics-Kanban.

TASK DESCRIPTION
  These queries: https://phabricator.wikimedia.org/T238878#5692516 should be 
productionized as reportupdater reports either from data from the replicas or 
from tables newly scooped hadoop
  
  Assigning to @Milimetric to get this work started, please coordinate with 
@mpopov

TASK DETAIL
  https://phabricator.wikimedia.org/T239565

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Milimetric, Nuria
Cc: Abit, Ramsey-WMF, kzimmerman, Addshore, matthiasmullie, gsingers, 
Mayakp.wiki, Ladsgroup, nettrom_WMF, Cparle, Nuria, Milimetric, mpopov, 
4748kitoko, darthmon_wmde, DannyS712, Nandana, JKSTNK, Akovalyov, Lahi, 
PDrouin-WMF, Gq86, E1presidente, Anooprao, SandraF_WMF, GoranSMilovanovic, 
QZanden, Tramullas, Acer, LawExplorer, Salgo60, Silverfish, _jensen, 
rosalieper, Scott_WUaS, Susannaanas, JAllemandou, Jane023, terrrydactyl, 
Wikidata-bugs, Base, aude, Ricordisamoa, Wesalius, Lydia_Pintscher, 
Fabrice_Florin, Raymond, Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T238878: Data about how many file pages on Commons contain at least one structured data element

[Nuria]

Nuria added a comment.


  Can someone explain why the content table is as large as the revision table? 
I though these tables  were not used much other than commons but content table 
is big in dewiki for example

TASK DETAIL
  https://phabricator.wikimedia.org/T238878

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Nuria
Cc: Cparle, nettrom_WMF, Ladsgroup, daniel, Mayakp.wiki, gsingers, 
matthiasmullie, Addshore, kzimmerman, mpopov, Ramsey-WMF, Abit, Nuria, 
4748kitoko, darthmon_wmde, DannyS712, Nandana, JKSTNK, Akovalyov, Lahi, 
PDrouin-WMF, Gq86, E1presidente, Anooprao, SandraF_WMF, GoranSMilovanovic, 
QZanden, Tramullas, Acer, LawExplorer, Salgo60, Silverfish, _jensen, 
rosalieper, Scott_WUaS, Susannaanas, JAllemandou, Jane023, terrrydactyl, 
Wikidata-bugs, Base, aude, Ricordisamoa, Wesalius, Lydia_Pintscher, 
Fabrice_Florin, Raymond, Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Updated] T238878: Data about how many file pages on Commons contain at least one structured data element

[Nuria]

Nuria added a subtask: T239127: Import slots/slots_roles  and  
wikibase.wbc_entity_usage through scoop .

TASK DETAIL
  https://phabricator.wikimedia.org/T238878

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Nuria
Cc: Cparle, nettrom_WMF, Ladsgroup, daniel, Mayakp.wiki, gsingers, 
matthiasmullie, Addshore, kzimmerman, mpopov, Ramsey-WMF, Abit, Nuria, 
4748kitoko, darthmon_wmde, DannyS712, Nandana, JKSTNK, Akovalyov, Lahi, 
PDrouin-WMF, Gq86, E1presidente, Anooprao, SandraF_WMF, GoranSMilovanovic, 
QZanden, Tramullas, Acer, LawExplorer, Salgo60, Silverfish, _jensen, 
rosalieper, Scott_WUaS, Susannaanas, JAllemandou, Jane023, terrrydactyl, 
Wikidata-bugs, Base, aude, Ricordisamoa, Wesalius, Lydia_Pintscher, 
Fabrice_Florin, Raymond, Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T238878: Data about how many file pages on Commons contain at least one structured data element

[Nuria]

Nuria added a comment.


  We will be scooping the slot tables to the cluster so these queries can run 
in parallel.

TASK DETAIL
  https://phabricator.wikimedia.org/T238878

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Nuria
Cc: Cparle, nettrom_WMF, Ladsgroup, daniel, Mayakp.wiki, gsingers, 
matthiasmullie, Addshore, kzimmerman, mpopov, Ramsey-WMF, Abit, Nuria, 
4748kitoko, darthmon_wmde, DannyS712, Nandana, JKSTNK, Akovalyov, Lahi, 
PDrouin-WMF, Gq86, E1presidente, Anooprao, SandraF_WMF, GoranSMilovanovic, 
QZanden, Tramullas, Acer, LawExplorer, Salgo60, Silverfish, _jensen, 
rosalieper, Scott_WUaS, Susannaanas, JAllemandou, Jane023, terrrydactyl, 
Wikidata-bugs, Base, aude, Ricordisamoa, Wesalius, Lydia_Pintscher, 
Fabrice_Florin, Raymond, Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T238878: Data about how many file pages on Commons contain at least one structured data element

[Nuria]

Nuria added a comment.


  So, per my comment above, I think the number of items is actually smaller 
than the one @Addshore has computed but more wise folks can correct me if I am 
wrong.

TASK DETAIL
  https://phabricator.wikimedia.org/T238878

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Nuria
Cc: matthiasmullie, Addshore, kzimmerman, mpopov, Ramsey-WMF, Abit, Nuria, 
4748kitoko, darthmon_wmde, DannyS712, Nandana, JKSTNK, Akovalyov, Lahi, 
PDrouin-WMF, Gq86, E1presidente, Cparle, Anooprao, SandraF_WMF, 
GoranSMilovanovic, QZanden, Tramullas, Acer, LawExplorer, Salgo60, Silverfish, 
_jensen, rosalieper, Scott_WUaS, Susannaanas, JAllemandou, Jane023, 
terrrydactyl, Wikidata-bugs, Base, aude, Ricordisamoa, Wesalius, 
Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T238878: Data about how many file pages on Commons contain at least one structured data element

[Nuria]

Nuria added a comment.


  @Addshore : disclaimer: I know next to nothing about this but how are you 
taking into account that the revision is the last one for the page? That is, a 
page might have had a structured data item  in a prior revision and from its 
most current revision that structured data item is removed? In your select * i 
think* you are counting revisions with slots that contain data but if that 
revision is a past one for the page it should not be counted, as we only care 
about  structured data present in the page right now, correct? Hopefully this  
makes sense, I think you are discounting deleted "slots" but are counting 
"past" revisions for the page where those slots where "alive" as if they 
pertained to the count.

TASK DETAIL
  https://phabricator.wikimedia.org/T238878

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Nuria
Cc: matthiasmullie, Addshore, kzimmerman, mpopov, Ramsey-WMF, Abit, Nuria, 
4748kitoko, darthmon_wmde, DannyS712, Nandana, JKSTNK, Akovalyov, Lahi, 
PDrouin-WMF, Gq86, E1presidente, Cparle, Anooprao, SandraF_WMF, 
GoranSMilovanovic, QZanden, Tramullas, Acer, LawExplorer, Salgo60, Silverfish, 
_jensen, rosalieper, Scott_WUaS, Susannaanas, JAllemandou, Jane023, 
terrrydactyl, Wikidata-bugs, Base, aude, Ricordisamoa, Wesalius, 
Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Edited] T238878: Data about how many file pages on Commons contain at least one structured data element

[Nuria]

Nuria updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T238878

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Nuria
Cc: Addshore, kzimmerman, mpopov, Ramsey-WMF, Abit, Nuria, 4748kitoko, 
darthmon_wmde, DannyS712, Nandana, JKSTNK, Akovalyov, Lahi, PDrouin-WMF, Gq86, 
E1presidente, Cparle, Anooprao, SandraF_WMF, GoranSMilovanovic, QZanden, 
Tramullas, Acer, LawExplorer, Salgo60, Silverfish, _jensen, rosalieper, 
Scott_WUaS, Susannaanas, JAllemandou, Jane023, terrrydactyl, Wikidata-bugs, 
Base, matthiasmullie, aude, Ricordisamoa, Wesalius, Lydia_Pintscher, 
Fabrice_Florin, Raymond, Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Changed Subscribers] T238878: Data about how many file pages on Commons contain at least one structured data element

[Nuria]

Nuria added a subscriber: Addshore.
Nuria added a comment.


  I think @Addshore had some information to add here.

TASK DETAIL
  https://phabricator.wikimedia.org/T238878

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Nuria
Cc: Addshore, kzimmerman, mpopov, Ramsey-WMF, Abit, Nuria, 4748kitoko, 
darthmon_wmde, DannyS712, Nandana, JKSTNK, Akovalyov, Lahi, PDrouin-WMF, Gq86, 
E1presidente, Cparle, Anooprao, SandraF_WMF, GoranSMilovanovic, QZanden, 
Tramullas, Acer, LawExplorer, Salgo60, Silverfish, _jensen, rosalieper, 
Scott_WUaS, Susannaanas, JAllemandou, Jane023, terrrydactyl, Wikidata-bugs, 
Base, matthiasmullie, aude, Ricordisamoa, Wesalius, Lydia_Pintscher, 
Fabrice_Florin, Raymond, Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Updated] T238878: Data about how many file pages on Commons contain at least one structured data element

[Nuria]

Nuria added a comment.


  Per T220525 <https://phabricator.wikimedia.org/T220525> it looks like none of 
the xml dump files that provide content for analytics contain any data about 
structure data in commons files.
  
  Proposed changes (that I am not sure got implemented) to change the dumps 
format to include this info are here: 
https://www.mediawiki.org/wiki/Requests_for_comment/Schema_update_for_multiple_content_objects_per_revision_(MCR)_in_XML_dumps#Schema
  
  Dumps are on version 0.10, the proposal linked is for (I think) version 0.11
  
  Summing up: there is no way to search actual page content to get the 
structure data of the page at this time from the dumps.

TASK DETAIL
  https://phabricator.wikimedia.org/T238878

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Nuria
Cc: kzimmerman, mpopov, Ramsey-WMF, Abit, Nuria, 4748kitoko, darthmon_wmde, 
DannyS712, Nandana, JKSTNK, Akovalyov, Lahi, PDrouin-WMF, Gq86, E1presidente, 
Cparle, Anooprao, SandraF_WMF, GoranSMilovanovic, QZanden, Tramullas, Acer, 
LawExplorer, Salgo60, Silverfish, _jensen, rosalieper, Scott_WUaS, Susannaanas, 
JAllemandou, Jane023, terrrydactyl, Wikidata-bugs, Base, matthiasmullie, aude, 
Ricordisamoa, Wesalius, Lydia_Pintscher, Fabrice_Florin, Raymond, 
Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Edited] T238878: Data about how many file pages on Commons contain at least one structured data element

[Nuria]

Nuria updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T238878

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Nuria
Cc: kzimmerman, mpopov, Ramsey-WMF, Abit, Nuria, 4748kitoko, darthmon_wmde, 
DannyS712, Nandana, JKSTNK, Akovalyov, Lahi, PDrouin-WMF, Gq86, E1presidente, 
Cparle, Anooprao, SandraF_WMF, GoranSMilovanovic, QZanden, Tramullas, Acer, 
LawExplorer, Salgo60, Silverfish, _jensen, rosalieper, Scott_WUaS, Susannaanas, 
JAllemandou, Jane023, terrrydactyl, Wikidata-bugs, Base, matthiasmullie, aude, 
Ricordisamoa, Wesalius, Lydia_Pintscher, Fabrice_Florin, Raymond, 
Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Created] T238878: Data about how many file pages on Commons contain at least one structured data element

[Nuria]

Nuria created this task.
Nuria added projects: Analytics, SDC General, Product-Analytics.
Restricted Application added a project: Wikidata.

TASK DESCRIPTION


TASK DETAIL
  https://phabricator.wikimedia.org/T238878

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Nuria
Cc: kzimmerman, mpopov, Ramsey-WMF, Abit, Nuria, 4748kitoko, darthmon_wmde, 
DannyS712, Nandana, JKSTNK, Akovalyov, Lahi, PDrouin-WMF, Gq86, E1presidente, 
Cparle, Anooprao, SandraF_WMF, GoranSMilovanovic, QZanden, Tramullas, Acer, 
LawExplorer, Salgo60, Silverfish, _jensen, rosalieper, Scott_WUaS, Susannaanas, 
JAllemandou, Jane023, terrrydactyl, Wikidata-bugs, Base, matthiasmullie, aude, 
Ricordisamoa, Wesalius, Lydia_Pintscher, Fabrice_Florin, Raymond, 
Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T238878: Data about how many file pages on Commons contain at least one structured data element

[Nuria]

Nuria added a comment.


  The work done by @mpopov
  
  The wbc_entity_usage table is supposed to hold info on Wikidata usage for the 
pages For example, here's a random file I added some structured data to a few 
days ago: 
https://commons.wikimedia.org/wiki/File:P%C3%B3voa_de_Varzim_-i---i-_(25379025808).jpg
  
  When you look for it the commonswiki replica, it has a page ID of 68860692. 
Looking for it in the wbc_entity_usage table we only see that it has a caption 
in English, which I added at basically the same time as several statements:
  
  Screen Shot 2019-11-19 at 5.58.57 PM.png
  eu_aspect column does have other values like "O" (statements) and "D" (not 
documented, but from a brief investigation looks like it's specifically for 
linking categories on Commons to Wikidata Q-items). There are some records of 
files with "O" aspects (as the MW page notes, it can refer to a variety to 
entity usages but typically it's statements) but then it gets weird because the 
language of the label isn't recorded and there's a bunch of seemingly 
unnecessary info? Take for example the MediaWiki DB data for 
https://commons.wikimedia.org/wiki/File:Jodrell_Bank_Mark_II_5.jpg
  
  Screen Shot 2019-11-19 at 6.36.04 PM.png
  Screen Shot 2019-11-19 at 6.36.33 PM.png
  Woof! That's…not great. So, uh, clearly there's something funky going on with 
the Wikibase client extension? Or maybe that's data that was recorded by an 
earlier version of the extension before it knew to append language codes to 
labels? I don't know enough about the nitty-gritty there, so these are just 
vaguely educated guesses.
  
  So, with the caveat that we're dealing with imperfect data and that some data 
is definitely missing, here are SOME lower bounds on files which have 
structured data:
  
  SELECT
  
COUNT(DISTINCT eu_page_id) AS n_files_with_statements,
COUNT(1) AS n_statements
  
  FROM wbc_entity_usage
  INNER JOIN page ON wbc_entity_usage.eu_page_id = page.page_id
  WHERE page_namespace = 6
  
AND eu_aspect = 'O';
  
  n_files_with_statements: 5 212 680
  n_statements: 8 677 888
  
  Files with labels:
  
  SELECT
  
COUNT(DISTINCT eu_page_id) AS n_files_with_captions
  
  FROM wbc_entity_usage
  INNER JOIN page ON wbc_entity_usage.eu_page_id = page.page_id
  WHERE page_namespace = 6
  
AND eu_aspect RLIKE '^L';
  
  n_files_with_captions: 7 857 520
  
  Files with captions and/or statements:
  
  SELECT
  
COUNT(DISTINCT eu_page_id) AS n_files_with_structured_data
  
  FROM wbc_entity_usage
  INNER JOIN page ON wbc_entity_usage.eu_page_id = page.page_id
  WHERE page_namespace = 6
  
AND (eu_aspect RLIKE '^L' OR eu_aspect = 'O');
  
  n_files_with_structured_data: 7 861 887
  
  Total files (at the moment I ran these queries, so we have a record of % w/ 
structured data):
  
  SELECT COUNT(1) AS n_files_total
  FROM page WHERE page_namespace = 6;
  
  n_files_total: 58 698 512
  
  Thanks for reading! Feel free to correct me if I made any mistakes with the 
queries or my thought process.

TASK DETAIL
  https://phabricator.wikimedia.org/T238878

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Nuria
Cc: kzimmerman, mpopov, Ramsey-WMF, Abit, Nuria, 4748kitoko, darthmon_wmde, 
DannyS712, Nandana, JKSTNK, Akovalyov, Lahi, PDrouin-WMF, Gq86, E1presidente, 
Cparle, Anooprao, SandraF_WMF, GoranSMilovanovic, QZanden, Tramullas, Acer, 
LawExplorer, Salgo60, Silverfish, _jensen, rosalieper, Scott_WUaS, Susannaanas, 
JAllemandou, Jane023, terrrydactyl, Wikidata-bugs, Base, matthiasmullie, aude, 
Ricordisamoa, Wesalius, Lydia_Pintscher, Fabrice_Florin, Raymond, 
Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Updated] T199121: RFC: Spec for representing multiple content objects per revision (MCR) in XML dumps

[Nuria]

Nuria added a comment.
Restricted Application added a project: Structured-Data-Backlog.


  I see this ticket is resolved but the dumps on commons have version  
version="0.10" since from this ticket i gather that the dumps that contain 
those slots are version=11 , are those being produced?

TASK DETAIL
  https://phabricator.wikimedia.org/T199121

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn, Nuria
Cc: Nuria, mako, FaFlo, Halfak, vrandezo, Denny, kchapman, tstarling, awight, 
JAllemandou, hoo, pmiazga, Nemo_bis, brion, Tgr, Aklapper, Fjalapeno, 
ArielGlenn, daniel, darthmon_wmde, DannyS712, Nandana, kostajh, Lahi, Gq86, 
Ramsey-WMF, GoranSMilovanovic, RazeSoldier, Lunewa, QZanden, V4switch, 
LawExplorer, JJMC89, _jensen, rosalieper, Agabi10, D3r1ck01, Scott_WUaS, Izno, 
SBisson, Wong128hk, gnosygnu, Perhelion, Wikidata-bugs, matthiasmullie, aude, 
GWicke, Bawolff, jayvdb, fbstj, Fabrice_Florin, santhosh, Jdforrester-WMF, 
Ladsgroup, Matanya, Mbch331, Rxy, Jay8g, Ltrlg, bd808, Legoktm
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T236895: ArticlePlaceholder dashboard stopped tracking page views

[Nuria]

Nuria added a comment.


  yes, you can use 
https://github.com/wikimedia/analytics-refinery-source/blob/master/refinery-hive/src/main/java/org/wikimedia/analytics/refinery/hive/GetHostPropertiesUDF.java
 to get the "project/family"

TASK DETAIL
  https://phabricator.wikimedia.org/T236895

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Nuria
Cc: Nuria, JAllemandou, elukey, Addshore, Aklapper, Lydia_Pintscher, 
4748kitoko, darthmon_wmde, DannyS712, Nandana, Akovalyov, Lahi, Gq86, 
GoranSMilovanovic, QZanden, cmadeo, LawExplorer, _jensen, rosalieper, Jonas, 
terrrydactyl, Wikidata-bugs, aude, jayvdb, Ricordisamoa, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T236895: ArticlePlaceholder dashboard stopped tracking page views

[Nuria]

Nuria added a comment.


  So this query needs to remove the is_pageview=true line: 
https://github.com/wikimedia/analytics-refinery-source/blob/master/refinery-job/src/main/scala/org/wikimedia/analytics/refinery/job/WikidataArticlePlaceholderMetrics.scala#L90

TASK DETAIL
  https://phabricator.wikimedia.org/T236895

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Nuria
Cc: Nuria, JAllemandou, elukey, Addshore, Aklapper, Lydia_Pintscher, 
4748kitoko, darthmon_wmde, DannyS712, Nandana, Akovalyov, Lahi, Gq86, 
GoranSMilovanovic, QZanden, cmadeo, LawExplorer, _jensen, rosalieper, Jonas, 
terrrydactyl, Wikidata-bugs, aude, jayvdb, Ricordisamoa, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T236895: ArticlePlaceholder dashboard stopped tracking page views

[Nuria]

Nuria added a comment.


  Ya, =1 to joseph, Special:blah urls (other than Special:Search) should not 
have been counted as pageviews and since  a fix on July they no longer are.

TASK DETAIL
  https://phabricator.wikimedia.org/T236895

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Nuria
Cc: Nuria, JAllemandou, elukey, Addshore, Aklapper, Lydia_Pintscher, 
4748kitoko, darthmon_wmde, DannyS712, Nandana, Akovalyov, Lahi, Gq86, 
GoranSMilovanovic, QZanden, cmadeo, LawExplorer, _jensen, rosalieper, Jonas, 
terrrydactyl, Wikidata-bugs, aude, jayvdb, Ricordisamoa, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Edited] T215413: Image Classification Working Group

[Nuria]

Nuria updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T215413

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Miriam, Nuria
Cc: Mholloway, Ottomata, Jheald, Cirdan, MoritzMuehlenhoff, CDanis, akosiaris, 
SandraF_WMF, Fuzheado, PDrouin-WMF, Krenair, d.astrikov, JoeWalsh, Nirzar, 
dcausse, fgiunchedi, JAllemandou, leila, Capt_Swing, mpopov, Nuria, DarTar, 
Halfak, Gilles, EBernhardson, dr0ptp4kt, Harej, MusikAnimal, Abit, elukey, 
diego, Cparle, Ramsey-WMF, Miriam, Isaac, 4748kitoko, darthmon_wmde, DannyS712, 
Nandana, JKSTNK, Akovalyov, Lahi, Gq86, E1presidente, Anooprao, 
GoranSMilovanovic, QZanden, EBjune, Tramullas, Acer, V4switch, LawExplorer, 
Salgo60, Avner, Silverfish, _jensen, rosalieper, Susannaanas, Wong128hk, 
Jane023, terrrydactyl, Wikidata-bugs, Base, matthiasmullie, aude, Ricordisamoa, 
Wesalius, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Matanya, 
Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Unblock] T215413: Image Classification Working Group

[Nuria]

Nuria closed subtask T148843: Remove computational bottlenecks in stats machine 
via adding a GPU that can be used to train ML models as Resolved.

TASK DETAIL
  https://phabricator.wikimedia.org/T215413

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Miriam, Nuria
Cc: Mholloway, Ottomata, Jheald, Cirdan, MoritzMuehlenhoff, CDanis, akosiaris, 
SandraF_WMF, Fuzheado, PDrouin-WMF, Krenair, d.astrikov, JoeWalsh, Nirzar, 
dcausse, fgiunchedi, JAllemandou, leila, Capt_Swing, mpopov, Nuria, DarTar, 
Halfak, Gilles, EBernhardson, dr0ptp4kt, Harej, MusikAnimal, Abit, elukey, 
diego, Cparle, Ramsey-WMF, Miriam, Isaac, 4748kitoko, darthmon_wmde, DannyS712, 
Nandana, JKSTNK, Akovalyov, Lahi, Gq86, E1presidente, Anooprao, 
GoranSMilovanovic, QZanden, EBjune, Tramullas, Acer, V4switch, LawExplorer, 
Salgo60, Avner, Silverfish, _jensen, rosalieper, Susannaanas, Wong128hk, 
Jane023, terrrydactyl, Wikidata-bugs, Base, matthiasmullie, aude, Ricordisamoa, 
Wesalius, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Matanya, 
Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Unblock] T208567: Count Wikidata page views per page type

[Nuria]

Nuria closed subtask T227905: Public Data Review Needed as Resolved.

TASK DETAIL
  https://phabricator.wikimedia.org/T208567

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: GoranSMilovanovic, Nuria
Cc: GoranSMilovanovic, Aklapper, WMDE-leszek, Lea_WMDE, darthmon_wmde, Nandana, 
Sario528, Lahi, Gq86, QZanden, LawExplorer, _jensen, rosalieper, Wikidata-bugs, 
aude, Lydia_Pintscher, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Retitled] T221921: Provision search endpoint for SDC. Requirements from Product Team.

[Nuria]

Nuria renamed this task from "Provision sparql endpoint for SDC. Requirements 
from Product Team." to "Provision search endpoint for SDC. Requirements from 
Product Team.".

TASK DETAIL
  https://phabricator.wikimedia.org/T221921

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Nuria
Cc: Smalyshev, Aklapper, Nuria, alaa_wmde, Nandana, JKSTNK, Lahi, PDrouin-WMF, 
Gq86, E1presidente, Ramsey-WMF, Cparle, Anooprao, SandraF_WMF, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, Tramullas, Acer, 
merbst, LawExplorer, Salgo60, Silverfish, Poyekhali, _jensen, rosalieper, 
Jonas, Xmlizer, Susannaanas, Wong128hk, Jane023, jkroll, Wikidata-bugs, 
Jdouglas, Base, matthiasmullie, aude, Tobias1984, El_Grafo, Dinoguy1000, 
Manybubbles, Ricordisamoa, Wesalius, Lydia_Pintscher, Fabrice_Florin, Raymond, 
Steinsplitter, Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Created] T221921: Provision sparql endpoint for SDC. Requirements from Product Team.

[Nuria]

Nuria created this task.
Nuria added projects: Wikidata, Commons, SDC General, Wikidata-Query-Service.

TASK DESCRIPTION


TASK DETAIL
  https://phabricator.wikimedia.org/T221921

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Nuria
Cc: Smalyshev, Aklapper, Nuria, alaa_wmde, Nandana, JKSTNK, Lahi, PDrouin-WMF, 
Gq86, E1presidente, Ramsey-WMF, Cparle, Anooprao, SandraF_WMF, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, Tramullas, Acer, 
merbst, LawExplorer, Silverfish, Poyekhali, _jensen, rosalieper, Jonas, 
Xmlizer, Susannaanas, Wong128hk, Jane023, jkroll, Wikidata-bugs, Jdouglas, 
Base, matthiasmullie, aude, Tobias1984, El_Grafo, Dinoguy1000, Manybubbles, 
Ricordisamoa, Wesalius, Lydia_Pintscher, Fabrice_Florin, Raymond, 
Steinsplitter, Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T209655: Copy Wikidata dumps to HDFs

[Nuria]

Nuria added a comment.


  @abian : this is still not happening on a recurrent schedule yet.

TASK DETAIL
  https://phabricator.wikimedia.org/T209655

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Nuria
Cc: abian, leila, Ottomata, Nuria, GoranSMilovanovic, Addshore, JAllemandou, 
bmansurov, alaa_wmde, Nandana, Akovalyov, Lahi, Gq86, QZanden, LawExplorer, 
Avner, _jensen, rosalieper, Wikidata-bugs, aude, Capt_Swing, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Changed Subscribers] T220823: Use ElasticSearch for bulk Wikidata entity term lookup

[Nuria]

Nuria added subscribers: Fjalapeno, Nuria.
Nuria added a comment.


  pinging @Fjalapeno from your comments the other day I understand Wikidata is 
going to use cassandra for these use cases at the end?  cc @Addshore

TASK DETAIL
  https://phabricator.wikimedia.org/T220823

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Nuria
Cc: Nuria, Fjalapeno, EBernhardson, dcausse, alaa_wmde, Addshore, Aklapper, 
Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, EBjune, LawExplorer, _jensen, 
rosalieper, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Unblock] T145712: Statement counts from pageprops do not match actual ones ( wikibase:statements and wikibase:sitelinks )

[Nuria]

Nuria closed subtask T161731: Create reliable change stream for specific wiki 
as Resolved.

TASK DETAIL
  https://phabricator.wikimedia.org/T145712

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Nuria
Cc: Lucas_Werkmeister_WMDE, Liuxinyu970226, Sjoerddebruin, MisterSynergy, Tgr, 
MZMcBride, aaron, Multichill, hoo, Esc3300, Pasleim, daniel, Aklapper, 
Smalyshev, alaa_wmde, ET4Eva, Nandana, Lahi, Gq86, Darkminds3113, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Avner, Gehel, _jensen, 
rosalieper, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Closed] T161731: Create reliable change stream for specific wiki

[Nuria]

Nuria closed this task as "Resolved".

TASK DETAIL
  https://phabricator.wikimedia.org/T161731

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata, Nuria
Cc: gerritbot, JAllemandou, Pchelolo, Ladsgroup, Nuria, Anomie, Aklapper, 
Smalyshev, alaa_wmde, holger.knust, joker88john, ET4Eva, CucyNoiD, Nandana, 
NebulousIris, Akovalyov, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, 
Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Xinbenlv, Vacio, Darkminds3113, 
Bsandipan, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, 
Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, merbst, LawExplorer, WSH1906, 
Avner, Lewizho99, Maathavan, Gehel, _jensen, rosalieper, Jonas, FloNight, 
Xmlizer, Nirmos, mobrovac, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
GWicke, Manybubbles, Mbch331, Krenair, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T217324: Have a more fine-grained history for property values on item pages

[Nuria]

Nuria added a comment.


  You would need a reconstruction that is property-aware, the current one knows 
only about pages and revisions. So, with different parameters for what the 
reconstruction is doing yes, possible.

TASK DETAIL
  https://phabricator.wikimedia.org/T217324

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Nuria
Cc: Harej, Nuria, Ogoorcs, Rachmat04, Amire80, Aklapper, Nandana, Lahi, Gq86, 
GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Wikidata-bugs, 
aude, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T216701: Wikidata Query Service should have a proper high level error handler

[Nuria]

Nuria added a comment.


  @Smalyshev ah, i see what you mean now but I am still of the opinion that the 
user should report the query that failed. On our end we can run it and retrieve 
the stack trace. Our 500 page could include helpful link to phabricator to 
report query that failed.
  Maybe I am missing something here and there are situations in which this 
wouldn't work but a world-wide accessible trace seems a problem security wise.

TASK DETAIL
  https://phabricator.wikimedia.org/T216701

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Gehel, Nuria
Cc: Nuria, hoo, csteipp, dpatrick, Lucas_Werkmeister_WMDE, abian, Mvolz, 
gerritbot, Smalyshev, Gehel, Aklapper, CucyNoiD, Nandana, NebulousIris, 
Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, 
Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, 
Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, merbst, LawExplorer, 
Lewizho99, Maathavan, _jensen, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, 
aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T216701: Wikidata Query Service should have a proper high level error handler

[Nuria]

Nuria added a comment.


  @Smalyshev if we configure the error logger to print requests and stack 
traces (however deep)  we can have alarming on them which would give us a 
measure of errors (maybe we already have this). Relying on users to report 
stack traces does not seem like it would give you a true measure of errors 
happening, right?

TASK DETAIL
  https://phabricator.wikimedia.org/T216701

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Gehel, Nuria
Cc: Nuria, hoo, csteipp, dpatrick, Lucas_Werkmeister_WMDE, abian, Mvolz, 
gerritbot, Smalyshev, Gehel, Aklapper, CucyNoiD, Nandana, NebulousIris, 
Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, 
Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, 
Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, merbst, LawExplorer, 
Lewizho99, Maathavan, _jensen, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, 
aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T215967: Add keyword for filtering based on captions in specific language

[Nuria]

Nuria added a comment.
@Ramsey-WMF Could we possibly get a bit more structured use cases?

Are those documented somewhere besides this ticket so we can see how this use case fits on the big picture? Is there any UI that goes with this case?TASK DETAILhttps://phabricator.wikimedia.org/T215967EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: EBernhardson, NuriaCc: Nuria, Abit, Cparle, Ramsey-WMF, Aklapper, JKSTNK, Lahi, PDrouin-WMF, E1presidente, Anooprao, SandraF_WMF, EBjune, Tramullas, Acer, V4switch, Silverfish, Susannaanas, Wong128hk, Jane023, Wikidata-bugs, Base, matthiasmullie, Ricordisamoa, Wesalius, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Matanya___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Triaged] T215616: Improve interlingual links across wikis through Wikidata IDs

[Nuria]

Nuria moved this task from Incoming to Smart Tools for Better Data on the Analytics board.Nuria triaged this task as "High" priority.
TASK DETAILhttps://phabricator.wikimedia.org/T215616WORKBOARDhttps://phabricator.wikimedia.org/project/board/11/EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: NuriaCc: Nuria, JAllemandou, diego, Nandana, Akovalyov, Banyek, AndyTan, Rayssa-, Lahi, Gq86, GoranSMilovanovic, QZanden, Marostegui, LawExplorer, Avner, Minhnv-2809, _jensen, Luke081515, Wikidata-bugs, aude, Capt_Swing, Dinoguy1000, Mbch331, Jay8g, Krenair, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T189744: Add hints parameter to wbsearchentities

[Nuria]

Nuria added a comment.
Not actively working on this now.TASK DETAILhttps://phabricator.wikimedia.org/T189744EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Smalyshev, NuriaCc: Nuria, Jonas, EBernhardson, gerritbot, Lydia_Pintscher, daniel, Aklapper, Smalyshev, ET4Eva, CucyNoiD, Nandana, NebulousIris, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, LawExplorer, Avner, Lewizho99, Maathavan, Gehel, _jensen, FloNight, Wikidata-bugs, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T214706: How to surface link changes as a stream?

[Nuria]

Nuria added a comment.
@Samwalton9 we still need to see if urls are url encoded or not and hook publishing to one of the mediawiki events (I think @bmansurov is doing this with @Pchelolo .help?) Once events are flowing and looking OK they can be set to be published to the outside world.TASK DETAILhttps://phabricator.wikimedia.org/T214706EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: NuriaCc: EvanProdromou, Mvolz, Lydia_Pintscher, Addshore, gerritbot, Pchelolo, Nuria, mobrovac, Samwalton9, Ottomata, ssastry, Smalyshev, bearND, bmansurov, holger.knust, CucyNoiD, Nandana, NebulousIris, Akovalyov, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Avner, Lewizho99, Maathavan, _jensen, Agabi10, Eevans, JAllemandou, Jrf, Hardikj, Wikidata-bugs, aude, GWicke, Capt_Swing, Dinoguy1000, Arlolra, Jdforrester-WMF, Mbch331, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Raised Priority] T214706: How to surface link changes as a stream?

[Nuria]

Nuria moved this task from Incoming to Radar on the Analytics board.Nuria raised the priority of this task from "Normal" to "Needs Triage".
TASK DETAILhttps://phabricator.wikimedia.org/T214706WORKBOARDhttps://phabricator.wikimedia.org/project/board/11/EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: NuriaCc: gerritbot, Pchelolo, Nuria, mobrovac, Samwalton9, Ottomata, ssastry, Smalyshev, bearND, bmansurov, holger.knust, EvanProdromou, CucyNoiD, Nandana, NebulousIris, Akovalyov, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Avner, Lewizho99, Maathavan, _jensen, Agabi10, Eevans, JAllemandou, Jrf, Hardikj, Wikidata-bugs, aude, GWicke, Capt_Swing, Dinoguy1000, Arlolra, Jdforrester-WMF, Mbch331, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T214706: How to surface link changes as a stream?

[Nuria]

Nuria added a comment.
@bmansurov I think you need to consider also couple more things: a list of links can be very lengthy, do we have a limit for how much this field should occupy? Are links url encoded? (we probably want them to be so).TASK DETAILhttps://phabricator.wikimedia.org/T214706EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: NuriaCc: gerritbot, Pchelolo, Nuria, mobrovac, Samwalton9, Ottomata, ssastry, Smalyshev, bearND, bmansurov, holger.knust, EvanProdromou, CucyNoiD, Nandana, NebulousIris, Akovalyov, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Avner, Lewizho99, Maathavan, _jensen, Agabi10, Eevans, JAllemandou, Jrf, Hardikj, Wikidata-bugs, aude, GWicke, Capt_Swing, Dinoguy1000, Arlolra, Jdforrester-WMF, Mbch331, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T214706: How to surface link changes as a stream?

[Nuria]

Nuria added a comment.
@bmansurov ah I think I understand what you meant, now sorry: if mediawiki cannot generate the diff  you are interested on at the time the page is edited you need to consume an event that happens later in the chain, ya, makes sense.TASK DETAILhttps://phabricator.wikimedia.org/T214706EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: NuriaCc: Nuria, mobrovac, Samwalton9, Ottomata, ssastry, Smalyshev, bearND, bmansurov, holger.knust, EvanProdromou, Nandana, Akovalyov, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Avner, _jensen, Agabi10, Eevans, JAllemandou, Jrf, Hardikj, Wikidata-bugs, aude, GWicke, Capt_Swing, Dinoguy1000, Arlolra, Jdforrester-WMF, Mbch331, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T214706: How to surface link changes as a stream?

[Nuria]

Nuria added a comment.
Clarifying: ChnageProp consumes EventBus data  just like EventStreams consumes EventBus data. So you cannot "use" changeprop rather you will be sending events to EventBus (soon to be called EventGate) and consuming them from elsewhere and in turn exposing them to the world.

Previously EventLogging was used for this purpose, but it didn't fully materalize. Is EventLogging still the right solution?

I think eventlogging should not be needed. It seems that this can be done from mediawiki  directly.

Should the RecentChanges event stream be expanded to include links?

Probably not, makes sense to create an event specifically for this. RecentChanges is used to note an edit event happened, it does not concern itself with content of the event and seems a bad idea to bloat event schema to do so. See https://github.com/wikimedia/mediawiki-event-schemas/blob/master/jsonschema/mediawiki/recentchange/2.yaml

See resources below:

Events can be published directly from mediawiki. The event streams we have for RecentChanges and similar do not use eventlogging but rather events are sent directly from mediawiki's backend when they occur. Schemas for those events can be found here: https://github.com/wikimedia/mediawiki-event-schemas/tree/master/jsonschema/mediawiki

Also please see: https://www.mediawiki.org/wiki/Wikimedia_Technology/Annual_Plans/FY2019/TEC2:_Modern_Event_Platform

And: https://github.com/wikimedia/eventgate

We are happy to help you CR your changes as needed be, just let us know if you need a meeting to come up with a plan on the steps needed to accomplish what this ticket wants to do. I would start drafting the schema of the event you are interested in.TASK DETAILhttps://phabricator.wikimedia.org/T214706EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: NuriaCc: Nuria, mobrovac, Samwalton9, Ottomata, ssastry, Smalyshev, bearND, bmansurov, holger.knust, Nandana, Akovalyov, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Avner, _jensen, Eevans, JAllemandou, Jrf, Hardikj, Wikidata-bugs, aude, GWicke, Capt_Swing, Dinoguy1000, Arlolra, Jdforrester-WMF, Mbch331, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T209655: Copy Wikidata dumps to HDFs

[Nuria]

Nuria added a comment.
Having missed most of goals this quarter due to our mw woes i think this might need to be moved to next quarter (q4?)TASK DETAILhttps://phabricator.wikimedia.org/T209655EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: NuriaCc: Ottomata, Nuria, GoranSMilovanovic, Addshore, JAllemandou, bmansurov, Nandana, Akovalyov, Lahi, Gq86, QZanden, LawExplorer, Avner, _jensen, D3r1ck01, Wikidata-bugs, aude, Capt_Swing, Mbch331, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Updated] T193728: Address concerns about perceived legal uncertainty of Wikidata

[Nuria]

Nuria removed a project: Analytics-Legal.
TASK DETAILhttps://phabricator.wikimedia.org/T193728EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: NuriaCc: ChristianKl, Alsee, Aklapper, Huji, ArthurPSmith, SimonPoole, Scott_WorldUnivAndSch, Micru, lisong, Lofhi, Nemo_bis, TomT0m, jrbs, EgonWillighagen, sarojdhakal, Agabi10, NMaia, Simon_Villeneuve, Jarekt, Rspeer, OhKayeSierra, AndrewSu, Mateusz_Konieczny, Maxlath, Glrx, Realworldobject, Ltrlg, Papapep, Tgr, Ayack, Gnom1, MichaelMaggs, MisterSynergy, Pasleim, Cirdan, 0x010C, Sylvain_WMFr, Denny, Ivanhercaz, Pintoch, Lydia_Pintscher, Lea_Lacroix_WMDE, Psychoslave, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, D3r1ck01, ZhouZ, Mpaulson, Wikidata-bugs, aude, jayvdb, Slaporte, Mbch331, Jay8g___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Changed Subscribers] T209031: Not able to scoop comment table in labs for mediawiki reconstruction process

[Nuria]

Nuria added subscribers: tstarling, bd808.Nuria added a comment.
Pinging @bd808 and @Fjalapeno and @tstarling per above comment.TASK DETAILhttps://phabricator.wikimedia.org/T209031EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: NuriaCc: bd808, tstarling, Fjalapeno, Cloud-Services, jcrespo, Marostegui, mforns, faidon, Milimetric, Krenair, elukey, Nuria, Aklapper, JAllemandou, Akovalyov, Banyek, AndyTan, Killertrap, Zylc, Giuliamocci, 1978Gage2001, Chicocvenancio, Tbscho, Minhnv-2809, JJMC89, srodlund, Luke081515, mys_721tx, Gryllida, scfc, Jay8g, jeremyb, chasemp___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T199517: Investigate June Unique devices increase of 170% for wikidata

[Nuria]

Nuria added a comment.
Added annotation for this event to wikidata unique devices data on wikistats: http://localhost:5000/dist/#/wikidata.org/reading/unique-devices/normal|line|All|~totalTASK DETAILhttps://phabricator.wikimedia.org/T199517EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: NuriaCc: Tbayer, Nuria, Aklapper, Lydia_Pintscher, JAllemandou, Addshore, Nandana, Akovalyov, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Wikidata-bugs, aude, Mbch331, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T204415: Query stats dashboard not updating

[Nuria]

Nuria added a comment.
Assigned to @mpopov Again, our apologies that the data sources are hardcoded like this. As I mentioned on our meeting  abetter path to go forward would be using the tags for wdqs to identify the requests: https://github.com/wikimedia/analytics-refinery-source/blob/master/refinery-core/src/main/java/org/wikimedia/analytics/refinery/core/webrequest/tag/WDQSTagger.javaTASK DETAILhttps://phabricator.wikimedia.org/T204415EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: mpopov, NuriaCc: Ottomata, elukey, Nuria, mpopov, chelsyx, Aklapper, Addshore, Smalyshev, Lydia_Pintscher, Akovalyov, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Jonas, Xmlizer, JAllemandou, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Reassigned] T204415: Query stats dashboard not updating

[Nuria]

Nuria reassigned this task from Nuria to mpopov.
TASK DETAILhttps://phabricator.wikimedia.org/T204415EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: mpopov, NuriaCc: Ottomata, elukey, Nuria, mpopov, chelsyx, Aklapper, Addshore, Smalyshev, Lydia_Pintscher, Akovalyov, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Jonas, Xmlizer, JAllemandou, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T204415: Query stats dashboard not updating

[Nuria]

Nuria added a comment.
Misc is no longer in service, all requests have been migrated to 'text'TASK DETAILhttps://phabricator.wikimedia.org/T204415EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: NuriaCc: Nuria, mpopov, chelsyx, Aklapper, Addshore, Smalyshev, Lydia_Pintscher, Akovalyov, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Jonas, Xmlizer, JAllemandou, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Closed] T191022: Add Wikidata website extract oozie job

[Nuria]

Nuria closed this task as "Resolved".
TASK DETAILhttps://phabricator.wikimedia.org/T191022EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Jonas, NuriaCc: Smalyshev, Nuria, gerritbot, JAllemandou, Jonas, Aklapper, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, lisong, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Maathavan, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Updated] T199517: Investigate June Unique devices increase of 170% for wikidata

[Nuria]

Nuria added a parent task: T138207: [Open question] Improve bot identification at scale.
TASK DETAILhttps://phabricator.wikimedia.org/T199517EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Addshore, NuriaCc: Nuria, Aklapper, Lydia_Pintscher, JAllemandou, Addshore, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Wikidata-bugs, aude, Mbch331, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Reopened] T199517: Investigate June Unique devices increase of 170% for wikidata

[Nuria]

Nuria reopened this task as "Stalled".
TASK DETAILhttps://phabricator.wikimedia.org/T199517EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Addshore, NuriaCc: Nuria, Aklapper, Lydia_Pintscher, JAllemandou, Addshore, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Wikidata-bugs, aude, Mbch331, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T199517: Investigate June Unique devices increase of 170% for wikidata

[Nuria]

Nuria added a comment.
yes , please, I listed issue on dataset page: https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Traffic/Unique_Devices#Changes_and_Known_Problems_with_Dataset
 We do not yet have annotations in wikistats (we will at the end of quarter) but when we do this is a good one to list. Moving ticket to bot work.TASK DETAILhttps://phabricator.wikimedia.org/T199517EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Addshore, NuriaCc: Nuria, Aklapper, Lydia_Pintscher, JAllemandou, Addshore, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Wikidata-bugs, aude, Mbch331, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T199517: Investigate June Unique devices increase of 170% for wikidata

[Nuria]

Nuria added a comment.
Bot did not accepted cookies, user agent was changing slightly, in 1000 records when this event is happening 995 are part of event and of those about 200 are unqiue user agents. Still the IP is teh same and the volumes of requests so high that I am wondering how these requests did not get throttled. Will look at throttling limits.TASK DETAILhttps://phabricator.wikimedia.org/T199517EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Addshore, NuriaCc: Nuria, Aklapper, Lydia_Pintscher, JAllemandou, Addshore, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Wikidata-bugs, aude, Mbch331, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T199517: Investigate June Unique devices increase of 170% for wikidata

[Nuria]

Nuria added a comment.
F23734550: Screen Shot 2018-07-13 at 12.43.07 PM.png

It coincides with a spike of pageviews from thailand, that seems like a bot accessing teh desktop size, will investigate  a bit as to whether this bot was accepting cookies.TASK DETAILhttps://phabricator.wikimedia.org/T199517EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Addshore, NuriaCc: Nuria, Aklapper, Lydia_Pintscher, JAllemandou, Addshore, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Wikidata-bugs, aude, Mbch331, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T161731: Create reliable change stream for specific wiki

[Nuria]

Nuria added a comment.
Ping @Smalyshev now that you have a reliable stream on the new kafka cluster (that supports time-based consumption)  is there any other blockers on your end ?TASK DETAILhttps://phabricator.wikimedia.org/T161731EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Ottomata, NuriaCc: gerritbot, JAllemandou, Pchelolo, Ladsgroup, Nuria, Anomie, Aklapper, Smalyshev, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Vacio, Darkminds3113, Bsandipan, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, merbst, LawExplorer, Avner, Lewizho99, Maathavan, Gehel, Jonas, FloNight, Xmlizer, Nirmos, mobrovac, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Manybubbles, Mbch331, Krenair, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Unblock] T161731: Create reliable change stream for specific wiki

[Nuria]

Nuria closed subtask T187296: Increase kafka event retention to 31 as "Resolved".
TASK DETAILhttps://phabricator.wikimedia.org/T161731EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Ottomata, NuriaCc: gerritbot, JAllemandou, Pchelolo, Ladsgroup, Nuria, Anomie, Aklapper, Smalyshev, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Vacio, Darkminds3113, Bsandipan, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, merbst, LawExplorer, Avner, Lewizho99, Maathavan, Gehel, Jonas, FloNight, Xmlizer, Nirmos, mobrovac, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Manybubbles, Mbch331, Krenair, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Closed] T187296: Increase kafka event retention to 31

[Nuria]

Nuria closed this task as "Resolved".
TASK DETAILhttps://phabricator.wikimedia.org/T187296EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Ottomata, NuriaCc: mforns, elukey, Ottomata, Aklapper, Nuria, Ladsgroup, Pchelolo, JAllemandou, Smalyshev, Lahi, Gq86, Vacio, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Avner, Gehel, Jonas, FloNight, Xmlizer, Eevans, Nirmos, mobrovac, jkroll, Hardikj, Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Manybubbles, Mbch331, Krenair, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T191022: Add Wikidata website extract oozie job

[Nuria]

Nuria added a comment.
@Jonas: do you want all requests to www.wikidata.org to be included, correct?  Do you care about request to wikidata query service or anything else about the request at hand?TASK DETAILhttps://phabricator.wikimedia.org/T191022EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Jonas, NuriaCc: Nuria, gerritbot, JAllemandou, Jonas, Aklapper, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Lordiis, GoranSMilovanovic, lisong, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Maathavan, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Changed Subscribers] T143819: Data request for logs from SparQL interface at query.wikidata.org

[Nuria]

Nuria added a subscriber: JAllemandou.Nuria added a comment.
I think notes look good.

@mforns main point that I missed is that we probably also want to remove geolocation from dataset #1, I see that from your sumup you did.

Remaining item is sanitization of sparql queries and on that I think we have to trust your expertise. As in any system any non parseable queries should be removed cause -as we have seen before- bad queries might contain someone's credit card number (for real). From your notes you are also removing non parseable queries so, good again.  I think also grouping user agent should be of use and not a privacy concern (as long as you only include broad categories. For example, we do not want to include: "user agent of the lunux distro only 3 people in the world have access to", this case is covered by you removing long tail of browsers with less than 10.000 requests.

Let's go ahead and start working on this, oozie/spark will be the way to go, since you already have tags on webrequest data you can probably run the job that will create this data once a day as you are only quering a subset of webrequest table ? (cc @JAllemandou to confirm). Here are some spark examples that might be of use to know how to generally approach the problem: https://github.com/wikimedia/analytics-refinery-source/blob/master/refinery-job/src/main/scala/org/wikimedia/analytics/refinery/job/ClickstreamBuilder.scalaTASK DETAILhttps://phabricator.wikimedia.org/T143819EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: NuriaCc: JAllemandou, mpopov, mforns, PokestarFan, Nuria, Lydia_Pintscher, mkroetzsch, leila, debt, thiemowmde, Jonas, Smalyshev, AndrewSu, Aklapper, I9606, Lahi, Gq86, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Avner, Gehel, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T174519: [epic] SDoC: Determine baseline for metrics

[Nuria]

Nuria added a comment.
Nice! Thank you for documenting.TASK DETAILhttps://phabricator.wikimedia.org/T174519EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: NuriaCc: Nuria, Liuxinyu970226, Capt_Swing, Ramsey-WMF, SandraF_WMF, Abit, chelsyx, mpopov, debt, Aklapper, Lahi, PDrouin-WMF, Gq86, E1presidente, GoranSMilovanovic, QZanden, EBjune, Tramullas, Acer, LawExplorer, Avner, Gehel, FloNight, Susannaanas, Aschroet, Jane023, Wikidata-bugs, PKM, Base, matthiasmullie, aude, Ricordisamoa, Fabrice_Florin, Raymond, Steinsplitter, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T143819: Data request for logs from SparQL interface at query.wikidata.org

[Nuria]

Nuria added a comment.
@Smalyshev We like to default to public if possible, the more eyes on the data the more useful it can be.TASK DETAILhttps://phabricator.wikimedia.org/T143819EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: NuriaCc: mforns, PokestarFan, Nuria, Lydia_Pintscher, mkroetzsch, leila, debt, thiemowmde, Jonas, Smalyshev, AndrewSu, Aklapper, I9606, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Avner, Gehel, FloNight, Xmlizer, JAllemandou, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Changed Subscribers] T143819: Data request for logs from SparQL interface at query.wikidata.org

[Nuria]

Nuria added a subscriber: mforns.Nuria added a comment.
@Smalyshev: Take a look at information we keep on pageview hourly, for long time keeping we need to remove PII and we neither store detail timestamps or sessionIds as we want to avoid session reconstruction precisely. So probably if we round timestamp and remove sessionId your proposal for dattaset #1 is safe to keep long term  (cc @mforns for anything I might be missing)

The proposal for dataset #2 looks good

I think  with the mentioned adjustments both datasets would be useful for public consumption.TASK DETAILhttps://phabricator.wikimedia.org/T143819EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: NuriaCc: mforns, PokestarFan, Nuria, Lydia_Pintscher, mkroetzsch, leila, debt, thiemowmde, Jonas, Smalyshev, AndrewSu, Aklapper, I9606, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, Avner, Gehel, FloNight, Xmlizer, JAllemandou, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T161731: Create reliable change stream for specific wiki

[Nuria]

Nuria added a comment.
@Smalyshev Please, 45 minutes with me and @Ottomata would do?TASK DETAILhttps://phabricator.wikimedia.org/T161731EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Ottomata, NuriaCc: gerritbot, JAllemandou, Pchelolo, Ladsgroup, Nuria, Anomie, Aklapper, Smalyshev, Cpaulf30, Lahi, Gq86, Baloch007, Vacio, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, merbst, Avner, Lewizho99, Maathavan, Gehel, Jonas, FloNight, Xmlizer, Nirmos, mobrovac, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Manybubbles, Mbch331, Krenair, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T161731: Create reliable change stream for specific wiki

[Nuria]

Nuria added a comment.
@Smalyshev Ok, we aim to have the cluster handling all prod traffic by end of next  quarter, until then it will be mirroing data which i think should be sufficient for you to get started in the wdqs consumer? Correct me if I am wrong.TASK DETAILhttps://phabricator.wikimedia.org/T161731EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Ottomata, NuriaCc: gerritbot, JAllemandou, Pchelolo, Ladsgroup, Nuria, Anomie, Aklapper, Smalyshev, Cpaulf30, Lahi, Gq86, Baloch007, Vacio, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, merbst, Avner, Lewizho99, Maathavan, Gehel, Jonas, FloNight, Xmlizer, Nirmos, mobrovac, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Manybubbles, Mbch331, Krenair, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T161731: Create reliable change stream for specific wiki

[Nuria]

Nuria added a comment.
Nice,  Can @Smalyshev check whether consuming from these topics as set would work for his purposes?TASK DETAILhttps://phabricator.wikimedia.org/T161731EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Ottomata, NuriaCc: gerritbot, JAllemandou, Pchelolo, Ladsgroup, Nuria, Anomie, Aklapper, Smalyshev, Lahi, Gq86, Baloch007, Vacio, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, merbst, Avner, Lewizho99, Maathavan, Gehel, Jonas, FloNight, Xmlizer, Nirmos, mobrovac, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Manybubbles, Mbch331, Krenair, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T161731: Create reliable change stream for specific wiki

[Nuria]

Nuria added a comment.
I got same doing:

/home/otto/kafkacat -Q  -b kafka-jumbo1003.eqiad.wmnet -t eqiad.mediawiki.revision-create:0:1512687299  -Xdebug=allTASK DETAILhttps://phabricator.wikimedia.org/T161731EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Ottomata, NuriaCc: JAllemandou, Pchelolo, Ladsgroup, Nuria, Anomie, Aklapper, Smalyshev, Lahi, Gq86, Vacio, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, Avner, Gehel, Jonas, FloNight, Xmlizer, Nirmos, mobrovac, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Manybubbles, Mbch331, Krenair, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T161731: Create reliable change stream for specific wiki

[Nuria]

Nuria added a comment.
@Ottomata Could @Smalyshev do a test on consuming from the new cluster though with teh understanding it is not yet productionized to make sure it fits the use cases?TASK DETAILhttps://phabricator.wikimedia.org/T161731EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Ottomata, NuriaCc: Ladsgroup, Nuria, Anomie, Aklapper, Smalyshev, Lahi, Gq86, Vacio, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, Avner, Gehel, Jonas, FloNight, Xmlizer, Nirmos, JAllemandou, mobrovac, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Manybubbles, Mbch331, Krenair, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs