Gehel created this task. Gehel added projects: Wikidata-Query-Service, Service-deployment-requests. Restricted Application added a subscriber: Aklapper. Restricted Application added projects: Wikidata, Services, SRE.
TASK DESCRIPTION **Name:** WDQS Streaming Updater **Description:** The WDQS Streaming Updater allows generating a stream of updates for WDQS. This service is a replacement for part of the current updater that is deployed on each WDQS server. It allows consolidating the common part of that update process instead duplicating that processing on each node and improves throughput by 5-10x. **Timeline:** Q4 2021, hopefully May. Discussions have been ongoing for a few month, so while this timeline is short, it is not unexpected. **Point person:** @RKemper as SRE, @dcausse and @Zbyszko as SWE, Search Platform as a team. **Technologies:** - Flink <https://flink.apache.org/>: a stateful stream processing framework (Java based) - Java / JVM - Kafka (both as a consumer and a producer) - Running on k8s, with some direct dependencies (ability to publish config maps) **Request flow diagram:** - a link to a diagram that explains the interaction between your service and any other parts of the operational stack inside the production cluster, such as requests made to MediaWiki, RESTBase, etc. **Misc:** - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Streaming_Updater TASK DETAIL https://phabricator.wikimedia.org/T280579 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Aklapper, Gehel, dcausse, Zbyszko, RKemper, Invadibot, Ramtin0071, MPhamWMF, Devnull, maantietaja, lmata, JMeybohm, Muchiri124, CBogen, Akuckartz, RhinosF1, Legado_Shulgin, ReaperDawn, Nandana, Namenlos314, Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, Eevans, jkroll, Hardikj, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Jdforrester-WMF, Mbch331, Jay8g, fgiunchedi
_______________________________________________ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
