[Wikidata-bugs] [Maniphest] T361935: Adapt the WDQS Streaming Updater to update multiple WDQS subgraphs

[gerritbot]

gerritbot added a project: Patch-For-Review.

TASK DETAIL
  https://phabricator.wikimedia.org/T361935

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: pfischer, gerritbot
Cc: Daniel_Mietchen, dr0ptp4kt, pfischer, dcausse, Aklapper, 
Danny_Benjafield_WMDE, Isabelladantes1983, Themindcoder, Adamm71, S8321414, 
Jersione, Hellket777, LisafBia6531, Astuthiodit_1, 786, Biggs657, 
karapayneWMDE, Invadibot, maantietaja, Juan90264, Alter-paule, Beast1978, 
ItamarWMDE, Un1tY, Akuckartz, Dringsim, Hook696, Kent7301, joker88john, 
CucyNoiD, Nandana, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, 
Bsandipan, GoranSMilovanovic, QZanden, EBjune, KimKelting, LawExplorer, 
Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, 
Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T361935: Adapt the WDQS Streaming Updater to update multiple WDQS subgraphs

[gerritbot]

gerritbot added a comment.


  Change #1033386 had a related patch set uploaded (by Peter Fischer; author: 
Peter Fischer):
  
  [wikidata/query/rdf@master] Introducing SubgraphRuleMatcher
  
  https://gerrit.wikimedia.org/r/1033386

TASK DETAIL
  https://phabricator.wikimedia.org/T361935

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: pfischer, gerritbot
Cc: Daniel_Mietchen, dr0ptp4kt, pfischer, dcausse, Aklapper, 
Danny_Benjafield_WMDE, S8321414, Astuthiodit_1, karapayneWMDE, Invadibot, 
maantietaja, ItamarWMDE, Akuckartz, Dringsim, Nandana, Lahi, Gq86, 
GoranSMilovanovic, QZanden, EBjune, KimKelting, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T361935: Adapt the WDQS Streaming Updater to update multiple WDQS subgraphs

[pfischer]

pfischer claimed this task.
pfischer moved this task from Incoming to In Progress on the Discovery-Search 
(Current work) board.

TASK DETAIL
  https://phabricator.wikimedia.org/T361935

WORKBOARD
  https://phabricator.wikimedia.org/project/board/1227/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: pfischer
Cc: Daniel_Mietchen, dr0ptp4kt, pfischer, dcausse, Aklapper, 
Danny_Benjafield_WMDE, S8321414, Astuthiodit_1, karapayneWMDE, Invadibot, 
maantietaja, ItamarWMDE, Akuckartz, Dringsim, Nandana, Lahi, Gq86, 
GoranSMilovanovic, QZanden, EBjune, KimKelting, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T361935: Adapt the WDQS Streaming Updater to update multiple WDQS subgraphs

[Maintenance_bot]

Maintenance_bot removed a project: Patch-For-Review.

TASK DETAIL
  https://phabricator.wikimedia.org/T361935

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Maintenance_bot
Cc: Daniel_Mietchen, dr0ptp4kt, pfischer, dcausse, Aklapper, 
Danny_Benjafield_WMDE, S8321414, Astuthiodit_1, karapayneWMDE, Invadibot, 
maantietaja, ItamarWMDE, Akuckartz, Dringsim, Nandana, Lahi, Gq86, 
GoranSMilovanovic, QZanden, EBjune, KimKelting, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331, Isabelladantes1983, 
Themindcoder, Adamm71, Jersione, Hellket777, LisafBia6531, 786, Biggs657, 
Juan90264, Alter-paule, Beast1978, Un1tY, Hook696, Kent7301, joker88john, 
CucyNoiD, Gaboe420, Giuliamocci, Cpaulf30, Af420, Bsandipan, Lewizho99, 
Maathavan, Neuronton
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T361935: Adapt the WDQS Streaming Updater to update multiple WDQS subgraphs

[gerritbot]

gerritbot added a comment.


  Change #1018323 **merged** by jenkins-bot:
  
  [wikidata/query/rdf@master] Add a basic model for defining subgraphs
  
  https://gerrit.wikimedia.org/r/1018323

TASK DETAIL
  https://phabricator.wikimedia.org/T361935

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: Daniel_Mietchen, dr0ptp4kt, pfischer, dcausse, Aklapper, 
Danny_Benjafield_WMDE, Isabelladantes1983, Themindcoder, Adamm71, S8321414, 
Jersione, Hellket777, LisafBia6531, Astuthiodit_1, 786, Biggs657, 
karapayneWMDE, Invadibot, maantietaja, Juan90264, Alter-paule, Beast1978, 
ItamarWMDE, Un1tY, Akuckartz, Dringsim, Hook696, Kent7301, joker88john, 
CucyNoiD, Nandana, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, 
Bsandipan, GoranSMilovanovic, QZanden, EBjune, KimKelting, LawExplorer, 
Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, 
Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T361935: Adapt the WDQS Streaming Updater to update multiple WDQS subgraphs

[Gehel]

Gehel moved this task from Incoming to Current work on the 
Wikidata-Query-Service board.
Gehel removed a project: Wikidata-Query-Service.

TASK DETAIL
  https://phabricator.wikimedia.org/T361935

WORKBOARD
  https://phabricator.wikimedia.org/project/board/891/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Gehel
Cc: Daniel_Mietchen, dr0ptp4kt, pfischer, dcausse, Aklapper, 
Danny_Benjafield_WMDE, Isabelladantes1983, Themindcoder, Adamm71, S8321414, 
Jersione, Hellket777, LisafBia6531, Astuthiodit_1, 786, Biggs657, 
karapayneWMDE, Invadibot, maantietaja, Juan90264, Alter-paule, Beast1978, 
ItamarWMDE, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, 
Nandana, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
GoranSMilovanovic, QZanden, EBjune, KimKelting, LawExplorer, Lewizho99, 
Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, Wikidata-bugs, aude, 
Mbch331, AWesterinen, Namenlos314, Lucas_Werkmeister_WMDE, merbst, Jonas, 
Xmlizer, jkroll, Jdouglas, Tobias1984, Manybubbles
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T361935: Adapt the WDQS Streaming Updater to update multiple WDQS subgraphs

[Gehel]

Gehel triaged this task as "High" priority.

TASK DETAIL
  https://phabricator.wikimedia.org/T361935

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Gehel
Cc: Daniel_Mietchen, dr0ptp4kt, pfischer, dcausse, Aklapper, 
Danny_Benjafield_WMDE, Isabelladantes1983, Themindcoder, Adamm71, S8321414, 
Jersione, Hellket777, LisafBia6531, Astuthiodit_1, AWesterinen, 786, Biggs657, 
karapayneWMDE, Invadibot, maantietaja, Juan90264, Alter-paule, Beast1978, 
ItamarWMDE, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, 
Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, 
Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, 
KimKelting, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, 
Neuronton, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T361935: Adapt the WDQS Streaming Updater to update multiple WDQS subgraphs

[gerritbot]

gerritbot added a project: Patch-For-Review.

TASK DETAIL
  https://phabricator.wikimedia.org/T361935

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: Daniel_Mietchen, dr0ptp4kt, pfischer, dcausse, Aklapper, 
Danny_Benjafield_WMDE, Isabelladantes1983, Themindcoder, Adamm71, S8321414, 
Jersione, Hellket777, LisafBia6531, Astuthiodit_1, AWesterinen, 786, Biggs657, 
karapayneWMDE, Invadibot, maantietaja, Juan90264, Alter-paule, Beast1978, 
ItamarWMDE, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, 
Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, 
Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, 
KimKelting, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, 
Neuronton, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T361935: Adapt the WDQS Streaming Updater to update multiple WDQS subgraphs

[gerritbot]

gerritbot added a comment.


  Change #1018323 had a related patch set uploaded (by DCausse; author: 
DCausse):
  
  [wikidata/query/rdf@master] Add a basic model for defining sugraphs
  
  https://gerrit.wikimedia.org/r/1018323

TASK DETAIL
  https://phabricator.wikimedia.org/T361935

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: Daniel_Mietchen, dr0ptp4kt, pfischer, dcausse, Aklapper, 
Danny_Benjafield_WMDE, S8321414, Astuthiodit_1, AWesterinen, karapayneWMDE, 
Invadibot, maantietaja, ItamarWMDE, Akuckartz, Nandana, Namenlos314, Lahi, 
Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, KimKelting, 
merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T361935: Adapt the WDQS Streaming Updater to update multiple WDQS subgraphs

[dcausse]

dcausse updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T361935

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: Daniel_Mietchen, dr0ptp4kt, pfischer, dcausse, Aklapper, 
Danny_Benjafield_WMDE, S8321414, Astuthiodit_1, AWesterinen, karapayneWMDE, 
Invadibot, maantietaja, ItamarWMDE, Akuckartz, Nandana, Namenlos314, Lahi, 
Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, KimKelting, 
merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T361935: Adapt the WDQS Streaming Updater to update multiple WDQS subgraphs

[dcausse]

dcausse updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T361935

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: Daniel_Mietchen, dr0ptp4kt, pfischer, dcausse, Aklapper, 
Danny_Benjafield_WMDE, S8321414, Astuthiodit_1, AWesterinen, karapayneWMDE, 
Invadibot, maantietaja, ItamarWMDE, Akuckartz, Nandana, Namenlos314, Lahi, 
Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, KimKelting, 
merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T361935: Adapt the WDQS Streaming Updater to update multiple WDQS subgraphs

[dr0ptp4kt]

dr0ptp4kt added a project: Discovery-Search (Current work).

TASK DETAIL
  https://phabricator.wikimedia.org/T361935

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dr0ptp4kt
Cc: Daniel_Mietchen, dr0ptp4kt, pfischer, dcausse, Aklapper, 
Danny_Benjafield_WMDE, S8321414, Astuthiodit_1, AWesterinen, karapayneWMDE, 
Invadibot, maantietaja, ItamarWMDE, Akuckartz, Nandana, Namenlos314, Lahi, 
Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, KimKelting, 
merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T361935: Adapt the WDQS Streaming Updater to update multiple WDQS subgraphs

[Maintenance_bot]

Maintenance_bot added a project: Wikidata.

TASK DETAIL
  https://phabricator.wikimedia.org/T361935

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Maintenance_bot
Cc: dr0ptp4kt, pfischer, dcausse, Aklapper, Danny_Benjafield_WMDE, S8321414, 
Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, 
Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, KimKelting, merbst, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T361935: Adapt the WDQS Streaming Updater to update multiple WDQS subgraphs

[dcausse]

dcausse updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T361935

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: dr0ptp4kt, pfischer, dcausse, Aklapper, AWesterinen, Namenlos314, Gq86, 
Lucas_Werkmeister_WMDE, EBjune, KimKelting, merbst, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T361935: Adapt the WDQS Streaming Updater to update multiple WDQS subgraphs

[dcausse]

dcausse added a parent task: T337013: [Epic] Splitting the graph in WDQS.

TASK DETAIL
  https://phabricator.wikimedia.org/T361935

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: dr0ptp4kt, pfischer, dcausse, Aklapper, AWesterinen, Namenlos314, Gq86, 
Lucas_Werkmeister_WMDE, EBjune, KimKelting, merbst, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T361935: Adapt the WDQS Streaming Updater to update multiple WDQS subgraphs

[dcausse]

dcausse created this task.
dcausse added a project: Wikidata-Query-Service.
Restricted Application added a subscriber: Aklapper.

TASK DESCRIPTION
  In order to support updating the subgraphs defined in 
Wikidata:SPARQL_query_service/WDQS_graph_split 
 
the streaming updater must be adapted to produce the right mutations for a 
given subgraph.
  
  General Idea
  
  
  Right after fetching the entity content from `Special:EntityData` and before 
generating the diff a new component will be added to apply the set of rules 
defining the subgraph and will populate a stream per subgraph:
  
  - `rdf-streaming-updater.mutations` will remain and will contain all the 
mutations to update the existing setup
  - `rdf-streaming-updater.mutations-main` will be added and will contain 
mutations related to the `main`
  - `rdf-streaming-updater.mutations-scholarly` will be added and will contain 
mutations related to the `scholarly`
  
  The consumer component on the other end of the stream might require very few 
(no?) modifications and might just work by configuring the kafka topic it 
should consume.
  
  The ticket describes the general case to support an arbitrary number of 
subgraphs, there could be certainly a simpler way to handle this with the 2 
subgraphs special case but it might error prone and less future proof so we 
should attempt to solve the general case. If difficulties are found while 
implementing it we can always reconsider and give up on the general case.
  
  Stubs
  -
  
  //Stubs// are artificial triples (explained in 
WDQS_Split_Refinement#Add_triples_to_help_navigate_between_the_subgraphs 
)
 that will take the following form:
  
wd:Q42 wikibaseqs:subgraph wdsubgraph:main
  
  - `wikibaseqs` (with suggested IRI: `http://wikiba.se/queryservice#`) a new 
namespace to hold the vocabulary of things related to the wdqs codebase and 
will probably be hard-coded there, `subgraph` will be the first and only term 
required at the moment
  - `wdsubgraph` (with suggested IRI: `https://query.wikidata.org/subgraph/`) a 
new namespace to hold the IRIs identifying subgraphs in the scope wikidata 
query service. It will be setup in the `prefixes.json` config file.
  
  Rules
  -
  
  The rules should be extremely simple to apply and will require only the data 
available locally after fetch the entity content. How the rules are expressed 
is up for discussion but could be a simple yaml file:
  
subgraphs:
   - scholarly:
   stream: "rdf-streaming-updater.mutations-scholarly"
   default: block
   rules:
 - "pass ?entity wdt:P31 wd:Q13442814"
 - "pass ?entity rdf:type wikibase:Property"
   stubs_source: true
   stubs_subgraph_uri: "https://query.wikidata.org/subgraph/scholarly;
   - main:
   stream: "rdf-streaming-updater.mutations-main"
   default: pass
   rules:
 - "block ?entity wdt:P31 wd:Q13442814"
   stubs_source: true
   stubs_subgraph_uri: "https://query.wikidata.org/subgraph/main;
   - full:
   stream: "rdf-streaming-udpater.mutations"
   default: pass
   stubs_source: false
  
  - rules are prefixed with `pass` or `block` telling what to do if evaluated 
to true
  - `?entity` will be replaced by the entity URI being updated
  - `[]` means any rdf literal
  
  The rules are applied in order and stop at the first match, if it's a `pass` 
the entity should enter this subraph if it's a block it should not pass. If no 
rule matches then entity uses the `default` setting to decide to either `pass` 
or `block`.
  The `stubs_source` attribute will determine if this graph is OK to be linked 
from a stub triple when an entity is blocked from another subgraph.
  
  Rules Outcome
  -
  
  Applying the rules will only answer the question: //does this entity revision 
belong to this graph?//
  The actual set of mutations to apply may also vary depending of the type of 
MutationOperation 

 to apply.
  
  **Diff**
  
  When a diff is required the rules will be applied from both the previous and 
next revision.
  
  - when an entity enters a subgraph:
- Diff: to remove stubs
- FullImport: the entity is fully imported
  - when an entity leaves a subgraph:
- DeleteItem: the entity is fully delete
- Diff: to add the stubs
  - when an entity stays in the subgraph
- Diff: simple diff
  - when an entity stays outside the subgraph
- Diff: special case for diffing stubs
  
  **FullImport**
  
  - the entity matches a subgraph
-