[Wikidata-bugs] [Maniphest] T294133: Expose rdf-streaming-updater.mutation content through EventStreams

[dcausse]

dcausse added a parent task: T330521: Make WDQS update stream public.

TASK DETAIL
  https://phabricator.wikimedia.org/T294133

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: RBrounley_WMF, Harej, Ottomata, Aklapper, dcausse, Mohamed-Awnallah, 
Astuthiodit_1, AWesterinen, EChetty, lbowmaker, BTullis, karapayneWMDE, 
Invadibot, MPhamWMF, Ywats0ns, maantietaja, Chlod, CBogen, ItamarWMDE, 
Akuckartz, 4748kitoko, holger.knust, Nandana, Namenlos314, Akovalyov, Lahi, 
Gq86, Xinbenlv, Vacio, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, 
EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
Nirmos, JAllemandou, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, GWicke, Manybubbles, Mbch331, Krenair, jeremyb
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294133: Expose rdf-streaming-updater.mutation content through EventStreams

[Gehel]

Gehel merged a task: T330521: Make WDQS update stream public.
Gehel added subscribers: Sj, danshick-wmde, tfmorris, Jheald, Gehel, 
KingsleyIdehen.

TASK DETAIL
  https://phabricator.wikimedia.org/T294133

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Gehel
Cc: KingsleyIdehen, Gehel, Jheald, tfmorris, danshick-wmde, Sj, RBrounley_WMF, 
Harej, Ottomata, Aklapper, dcausse, Mohamed-Awnallah, Astuthiodit_1, 
AWesterinen, EChetty, lbowmaker, BTullis, karapayneWMDE, Invadibot, MPhamWMF, 
Ywats0ns, maantietaja, Chlod, CBogen, ItamarWMDE, Akuckartz, 4748kitoko, 
holger.knust, Nandana, Namenlos314, Akovalyov, Lahi, Gq86, Xinbenlv, Vacio, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Nirmos, 
JAllemandou, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
GWicke, Manybubbles, Mbch331, Krenair, jeremyb
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294133: Expose rdf-streaming-updater.mutation content through EventStreams

[Gehel]

Gehel added a comment.


  Note that there is additional context in T330521 
.

TASK DETAIL
  https://phabricator.wikimedia.org/T294133

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Gehel
Cc: KingsleyIdehen, Gehel, Jheald, tfmorris, danshick-wmde, Sj, RBrounley_WMF, 
Harej, Ottomata, Aklapper, dcausse, Mohamed-Awnallah, Astuthiodit_1, 
AWesterinen, EChetty, lbowmaker, BTullis, karapayneWMDE, Invadibot, MPhamWMF, 
Ywats0ns, maantietaja, Chlod, CBogen, ItamarWMDE, Akuckartz, 4748kitoko, 
holger.knust, Nandana, Namenlos314, Akovalyov, Lahi, Gq86, Xinbenlv, Vacio, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Nirmos, 
JAllemandou, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
GWicke, Manybubbles, Mbch331, Krenair, jeremyb
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294133: Expose rdf-streaming-updater.mutation content through EventStreams

[Sj]

Sj added a subscriber: MPhamWMF.
Sj added a comment.


  Thanks @Gehel for catching the duplicate.
  
  Suggested edits for this (merged) issue:
  
  - The merged issue should be for **making public the data WDQS uses for live 
updates**.  A specific implementation such as "**...using EventStreams**" could 
be part of the discussion, or even a child task (trying that out as a way to 
make the data public).
  - @MPhamWMF  set T330521  as high 
priority, and previously set this issue as high priority.  If there is 
disagreement about that, it would be helpful to write a few words about that.  
Lowering the barrier to mirroring is critical for those maintaining live 
mirrors, and less so for those only working on WD and WDQS core. But committing 
to make this work could open avenues to share WDQS loads which would affect 
core performance and disaster planning (the context in which this came up 
recently).

TASK DETAIL
  https://phabricator.wikimedia.org/T294133

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Sj
Cc: MPhamWMF, KingsleyIdehen, Gehel, Jheald, tfmorris, danshick-wmde, Sj, 
RBrounley_WMF, Harej, Ottomata, Aklapper, dcausse, Mohamed-Awnallah, 
Astuthiodit_1, AWesterinen, EChetty, lbowmaker, BTullis, karapayneWMDE, 
Invadibot, Ywats0ns, maantietaja, Chlod, CBogen, ItamarWMDE, Akuckartz, 
4748kitoko, holger.knust, Nandana, Namenlos314, Akovalyov, Lahi, Gq86, 
Xinbenlv, Vacio, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, 
merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Nirmos, 
JAllemandou, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
GWicke, Manybubbles, Mbch331, Krenair, jeremyb
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294133: Expose rdf-streaming-updater.mutation content through EventStreams

[dcausse]

dcausse created this task.
dcausse added a project: Wikidata-Query-Service.
Restricted Application added a subscriber: Aklapper.

TASK DESCRIPTION
  As a consumer of the wikidata content I want to be able to have access the 
same RDF data the WMF WDQS servers  use to perform their live updates so that I 
can keep my own replica of the wikidata query service (or another RDF store) up 
to date more easily.
  
  A solution might be to use the EventStreams 
 service.
  
  Note on the stream:
  It was decided to go fully active/active for the flink application powering 
the WDQS updater. Which means the complete stream of changes is available in 
both topics:
  
  - eqiad.rdf-streaming-updater.mutation
  - codfw.rdf-streaming-updater.mutation
  
  It is sligthly different to what we currently see in our topic topology where 
if you want to have a complete view of the data you need to consume both 
eqiad.topic and codfw.topic. Here you must consume only one.
  
  AC:
  
  - RDF data is exposed through EventStreams
  - A java client is offered for third parties to use with store comptatible 
with the SPARQL 1.1 Update operations 


TASK DETAIL
  https://phabricator.wikimedia.org/T294133

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: Aklapper, dcausse, MPhamWMF, CBogen, Namenlos314, Gq86, 
Lucas_Werkmeister_WMDE, EBjune, merbst, Jonas, Xmlizer, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294133: Expose rdf-streaming-updater.mutation content through EventStreams

[dcausse]

dcausse updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T294133

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: Aklapper, dcausse, MPhamWMF, CBogen, Namenlos314, Gq86, 
Lucas_Werkmeister_WMDE, EBjune, merbst, Jonas, Xmlizer, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294133: Expose rdf-streaming-updater.mutation content through EventStreams

[dcausse]

dcausse added a subscriber: Ottomata.
dcausse added a project: EventStreams.
dcausse updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T294133

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: Ottomata, Aklapper, dcausse, MPhamWMF, RBrounley_WMF, CBogen, Namenlos314, 
Gq86, Xinbenlv, Vacio, Lucas_Werkmeister_WMDE, EBjune, merbst, Jonas, Xmlizer, 
Nirmos, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Krenair
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294133: Expose rdf-streaming-updater.mutation content through EventStreams

[Maintenance_bot]

Maintenance_bot added a project: Wikidata.

TASK DETAIL
  https://phabricator.wikimedia.org/T294133

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Maintenance_bot
Cc: Ottomata, Aklapper, dcausse, Invadibot, MPhamWMF, maantietaja, 
RBrounley_WMF, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Xinbenlv, 
Vacio, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Nirmos, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, Krenair
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294133: Expose rdf-streaming-updater.mutation content through EventStreams

[Ottomata]

Ottomata added projects: Analytics, Event-Platform.
Ottomata added a comment.


  > Here you must consume only one.
  
  Should we expose both?
  
  We'll need to declare these streams in EventStreammConfig, likely each as a 
distinct stream overriding the list of topics that make up the stream.
  
  Then we can edit EventStreams helmfile configs to expose those streams.

TASK DETAIL
  https://phabricator.wikimedia.org/T294133

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: Ottomata, Aklapper, dcausse, EChetty, Invadibot, MPhamWMF, maantietaja, 
RBrounley_WMF, CBogen, Akuckartz, 4748kitoko, holger.knust, Nandana, 
Namenlos314, Akovalyov, Lahi, Gq86, Xinbenlv, Vacio, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, Nirmos, JAllemandou, terrrydactyl, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Manybubbles, Mbch331, 
Krenair, jeremyb
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294133: Expose rdf-streaming-updater.mutation content through EventStreams

[Ottomata]

Ottomata added a comment.


  Oh, we'll also want to add create event schema and add it to the schema repo.

TASK DETAIL
  https://phabricator.wikimedia.org/T294133

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: Ottomata, Aklapper, dcausse, EChetty, Invadibot, MPhamWMF, maantietaja, 
RBrounley_WMF, CBogen, Akuckartz, 4748kitoko, holger.knust, Nandana, 
Namenlos314, Akovalyov, Lahi, Gq86, Xinbenlv, Vacio, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, Nirmos, JAllemandou, terrrydactyl, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Manybubbles, Mbch331, 
Krenair, jeremyb
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294133: Expose rdf-streaming-updater.mutation content through EventStreams

[MPhamWMF]

MPhamWMF triaged this task as "High" priority.

TASK DETAIL
  https://phabricator.wikimedia.org/T294133

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: MPhamWMF
Cc: Ottomata, Aklapper, dcausse, EChetty, Invadibot, MPhamWMF, maantietaja, 
RBrounley_WMF, CBogen, Akuckartz, 4748kitoko, holger.knust, Nandana, 
Namenlos314, Akovalyov, Lahi, Gq86, Xinbenlv, Vacio, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, Nirmos, JAllemandou, terrrydactyl, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Manybubbles, Mbch331, 
Krenair, jeremyb
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294133: Expose rdf-streaming-updater.mutation content through EventStreams

[odimitrijevic]

odimitrijevic moved this task from Incoming to Event Platform on the Analytics 
board.
odimitrijevic lowered the priority of this task from "High" to "Medium".

TASK DETAIL
  https://phabricator.wikimedia.org/T294133

WORKBOARD
  https://phabricator.wikimedia.org/project/board/11/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: odimitrijevic
Cc: Ottomata, Aklapper, dcausse, EChetty, Invadibot, MPhamWMF, maantietaja, 
RBrounley_WMF, CBogen, Akuckartz, 4748kitoko, holger.knust, Nandana, 
Namenlos314, Akovalyov, Lahi, Gq86, Xinbenlv, Vacio, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, Nirmos, JAllemandou, terrrydactyl, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Manybubbles, Mbch331, 
Krenair, jeremyb
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294133: Expose rdf-streaming-updater.mutation content through EventStreams

[RBrounley_WMF]

Restricted Application added a project: Data-Engineering.

TASK DETAIL
  https://phabricator.wikimedia.org/T294133

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RBrounley_WMF
Cc: RBrounley_WMF, Harej, Ottomata, Aklapper, dcausse, EChetty, Invadibot, 
MPhamWMF, maantietaja, CBogen, Akuckartz, 4748kitoko, holger.knust, Nandana, 
Namenlos314, Akovalyov, Lahi, Gq86, Xinbenlv, Vacio, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, Nirmos, JAllemandou, terrrydactyl, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Manybubbles, Mbch331, 
Krenair, jeremyb
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294133: Expose rdf-streaming-updater.mutation content through EventStreams

[Ottomata]

Ottomata added a comment.


  > Here you must consume only one.
  
  Actually, this is curious.  These are really distinct streams.  We probably 
should have named them differently.  We can't just pick 
eqiad.rdf-streaming-updater.mutation when the user connects to EventStreams in 
eqiad, because the next time around they might connect to EventStreams in 
codfw, depending on how they are routed.
  
  So, I think either we expose only one of these topics, OR, we expose them 
both but as different distinct streams, and the user has to pick one.

TASK DETAIL
  https://phabricator.wikimedia.org/T294133

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: RBrounley_WMF, Harej, Ottomata, Aklapper, dcausse, EChetty, Invadibot, 
MPhamWMF, maantietaja, CBogen, Akuckartz, 4748kitoko, holger.knust, Nandana, 
Namenlos314, Akovalyov, Lahi, Gq86, Xinbenlv, Vacio, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, Nirmos, JAllemandou, terrrydactyl, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Manybubbles, Mbch331, 
Krenair, jeremyb
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294133: Expose rdf-streaming-updater.mutation content through EventStreams

[Ottomata]

Ottomata added a comment.


  (Oh, past me said this already... :p)

TASK DETAIL
  https://phabricator.wikimedia.org/T294133

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: RBrounley_WMF, Harej, Ottomata, Aklapper, dcausse, EChetty, Invadibot, 
MPhamWMF, maantietaja, CBogen, Akuckartz, 4748kitoko, holger.knust, Nandana, 
Namenlos314, Akovalyov, Lahi, Gq86, Xinbenlv, Vacio, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, Nirmos, JAllemandou, terrrydactyl, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Manybubbles, Mbch331, 
Krenair, jeremyb
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294133: Expose rdf-streaming-updater.mutation content through EventStreams

[JArguello-WMF]

JArguello-WMF added a project: Data Engineering and Event Platform Team.

TASK DETAIL
  https://phabricator.wikimedia.org/T294133

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: JArguello-WMF
Cc: MPhamWMF, KingsleyIdehen, Gehel, Jheald, tfmorris, Sj, RBrounley_WMF, 
Harej, Ottomata, Aklapper, dcausse, Mohamed-Awnallah, Astuthiodit_1, 
AWesterinen, EChetty, lbowmaker, BTullis, karapayneWMDE, Invadibot, Ywats0ns, 
maantietaja, Chlod, ItamarWMDE, Akuckartz, 4748kitoko, holger.knust, Nandana, 
Namenlos314, Akovalyov, Lahi, Gq86, Xinbenlv, Vacio, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, Nirmos, JAllemandou, terrrydactyl, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Manybubbles, Mbch331, 
Krenair, jeremyb
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294133: Expose rdf-streaming-updater.mutation content through EventStreams

[lbowmaker]

lbowmaker removed a project: Data Engineering and Event Platform Team.
lbowmaker moved this task from Data Eng/Event Platform to Icebox (not 
considered in current quarter) on the Data-Engineering board.

TASK DETAIL
  https://phabricator.wikimedia.org/T294133

WORKBOARD
  https://phabricator.wikimedia.org/project/board/5462/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: lbowmaker
Cc: bking, gmodena, MPhamWMF, KingsleyIdehen, Gehel, Jheald, tfmorris, Sj, 
RBrounley_WMF, Harej, Ottomata, Aklapper, dcausse, Danny_Benjafield_WMDE, 
Mohamed-Awnallah, Astuthiodit_1, AWesterinen, EChetty, lbowmaker, BTullis, 
karapayneWMDE, Invadibot, Ywats0ns, maantietaja, Chlod, ItamarWMDE, Akuckartz, 
4748kitoko, holger.knust, Nandana, Namenlos314, Akovalyov, Lahi, Gq86, 
Xinbenlv, Vacio, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, 
merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Nirmos, 
JAllemandou, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
GWicke, Manybubbles, Mbch331, Krenair, jeremyb
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294133: Expose rdf-streaming-updater.mutation content through EventStreams

[Hannah_Bast]

Hannah_Bast added a comment.


  I just looked into https://github.com/wikimedia/wikidata-query-rdf, which 
provides a tool `runUpdate.sh`. When I run it for a Blazegraph instance with 
exactly one triple of the form ` 
 "2024-02-11T05:42Z"^^xsd:dateTime`, it will 
continuously update the instance with all changes since that date. I have two 
questions:
  
  1. Which API is this script (or rather the underlying Update.java 
)
 calling to get the updates since a particular date?
  
  2. Apparently, this API is public. From an earlier communication, I got the 
impression that there is no public API for this yet. What did I misunderstand?

TASK DETAIL
  https://phabricator.wikimedia.org/T294133

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Hannah_Bast
Cc: Hannah_Bast, bking, gmodena, MPhamWMF, KingsleyIdehen, Gehel, Jheald, 
tfmorris, Sj, RBrounley_WMF, Harej, Ottomata, Aklapper, dcausse, 
Danny_Benjafield_WMDE, Mohamed-Awnallah, Astuthiodit_1, AWesterinen, EChetty, 
lbowmaker, BTullis, karapayneWMDE, Invadibot, Ywats0ns, maantietaja, Chlod, 
ItamarWMDE, Akuckartz, 4748kitoko, holger.knust, Nandana, Namenlos314, 
Akovalyov, Lahi, Gq86, Xinbenlv, Vacio, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, KimKelting, merbst, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, Nirmos, JAllemandou, terrrydactyl, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Manybubbles, 
Mbch331, Krenair, jeremyb
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294133: Expose rdf-streaming-updater.mutation content through EventStreams

[Harej]

Harej added a comment.


  In T294133#9531815 , 
@Hannah_Bast wrote:
  
  > 1. Which API is this script (or rather the underlying Update.java 
)
 calling to get the updates since a particular date?
  
  It is retrieving updates from https://wikidata.org/wiki/Special:RecentChanges
  
  > 2. Apparently, this API is public. From an earlier communication, I got the 
impression that there is no public API for this yet. What did I misunderstand?
  
  runUpdate.sh does not use this modern updater. It synchronizes with 
Wikidata's live recent changes.

TASK DETAIL
  https://phabricator.wikimedia.org/T294133

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Harej
Cc: Hannah_Bast, bking, gmodena, MPhamWMF, KingsleyIdehen, Gehel, Jheald, 
tfmorris, Sj, RBrounley_WMF, Harej, Ottomata, Aklapper, dcausse, 
Danny_Benjafield_WMDE, Mohamed-Awnallah, Astuthiodit_1, AWesterinen, EChetty, 
lbowmaker, BTullis, karapayneWMDE, Invadibot, Ywats0ns, maantietaja, Chlod, 
ItamarWMDE, Akuckartz, 4748kitoko, holger.knust, Nandana, Namenlos314, 
Akovalyov, Lahi, Gq86, Xinbenlv, Vacio, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, KimKelting, merbst, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, Nirmos, JAllemandou, terrrydactyl, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Manybubbles, 
Mbch331, Krenair, jeremyb
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294133: Expose rdf-streaming-updater.mutation content through EventStreams

[Hannah_Bast]

Hannah_Bast added a comment.


  @Harej Thanks for the quick reply, James! Are you saying that the script is 
scraping + parsing https://www.wikidata.org/wiki/Special:RecentChanges to 
obtain the triples to be added and deleted? Or is there a different way to 
access that page, which gives you the added and deleted triples in a more 
machine-friendly format?

TASK DETAIL
  https://phabricator.wikimedia.org/T294133

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Hannah_Bast
Cc: Hannah_Bast, bking, gmodena, MPhamWMF, KingsleyIdehen, Gehel, Jheald, 
tfmorris, Sj, RBrounley_WMF, Harej, Ottomata, Aklapper, dcausse, 
Danny_Benjafield_WMDE, Mohamed-Awnallah, Astuthiodit_1, AWesterinen, EChetty, 
lbowmaker, BTullis, karapayneWMDE, Invadibot, Ywats0ns, maantietaja, Chlod, 
ItamarWMDE, Akuckartz, 4748kitoko, holger.knust, Nandana, Namenlos314, 
Akovalyov, Lahi, Gq86, Xinbenlv, Vacio, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, KimKelting, merbst, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, Nirmos, JAllemandou, terrrydactyl, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Manybubbles, 
Mbch331, Krenair, jeremyb
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294133: Expose rdf-streaming-updater.mutation content through EventStreams

[Lucas_Werkmeister_WMDE]

Lucas_Werkmeister_WMDE added a comment.


  AFAIK: The “legacy” updater queries the recent changes via the API 
, then gets the RDF for the 
edited entities from Special:EntityData, and compares that to the live data in 
the query service to determine which triples need to be added and removed. 
(This includes some logic to clean up “orphaned” nodes like statements, 
references or full values.) This should work for any Wikibase, but it’s 
inefficient, which is why we no longer use it in production.

TASK DETAIL
  https://phabricator.wikimedia.org/T294133

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lucas_Werkmeister_WMDE
Cc: Lucas_Werkmeister_WMDE, Hannah_Bast, bking, gmodena, MPhamWMF, 
KingsleyIdehen, Gehel, Jheald, tfmorris, Sj, RBrounley_WMF, Harej, Ottomata, 
Aklapper, dcausse, Danny_Benjafield_WMDE, Mohamed-Awnallah, Astuthiodit_1, 
AWesterinen, EChetty, lbowmaker, BTullis, karapayneWMDE, Invadibot, Ywats0ns, 
maantietaja, Chlod, ItamarWMDE, Akuckartz, 4748kitoko, holger.knust, Nandana, 
Namenlos314, Akovalyov, Lahi, Gq86, Xinbenlv, Vacio, GoranSMilovanovic, 
QZanden, EBjune, KimKelting, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, Nirmos, JAllemandou, terrrydactyl, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Manybubbles, Mbch331, 
Krenair, jeremyb
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org