[Wikidata-bugs] [Maniphest] [Created] T212826: Create dedicated Updater service in Blazegraph

[Smalyshev]

Smalyshev created this task. Smalyshev triaged this task as "Normal" priority. Smalyshev added projects: Wikidata, Wikidata-Query-Service, Performance. Restricted Application added a subscriber: Aklapper.

TASK DESCRIPTION

Right now, Updater sends SPARQL UPDATE queries to Blazegraph in order to update items. Instead, we could create a separate service that would receive:

1. List of items to update (URIs)
2. For each item to update, list of triples associated with the item (as collected by Updater)

The service would then perform the same work as Updater did but without the overhead of creating/parsing SPARQL and sending extra data over the wire. For each item, it would:

1. Fetch the triples that currently belong to the item
   1. Including direct connectors to the item, statement triples, forms/senses (for lexemes) and form statements.
   2. This also collects all reference and value URIs mentioned by any of the statements.
2. Diff the data above against the data received from Updater.
3. The triples that is not in incoming set are deleted.
4. The triples that is in the incoming set but not in result set are inserted
5. The values/references that had been mentioned in the existing data but not in the incoming data are verified to see if anything links to them. If not, they are deleted.

TASK DETAIL

https://phabricator.wikimedia.org/T212826

EMAIL PREFERENCES

https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Smalyshev
Cc: Aklapper, Smalyshev, Nandana, Lahi, Gq86, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Vali.matei, _jensen, D3r1ck01, Jonas, Xmlizer, Volker_E, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Dinoguy1000, Manybubbles, Mbch331, Jay8g

_______________________________________________
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs