[Wikidata-bugs] [Maniphest] [Claimed] T239908: Extract more metrics from blazegraph sparql update response
Zbyszko claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T239908 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Aklapper, dcausse, darthmon_wmde, DannyS712, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Created] T241213: Organize and improve integration test coverage for WDQS Updater
Zbyszko created this task. Zbyszko added a project: Wikidata-Query-Service. Restricted Application added a subscriber: Aklapper. Restricted Application added a project: Wikidata. TASK DESCRIPTION [ ] Verify test coverage for Updater [ ] Create E2E test [ ] Review previous tests ( remove duplicated cases) [ ] Drop dependencies on real resources in integration tests TASK DETAIL https://phabricator.wikimedia.org/T241213 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Aklapper, Zbyszko, darthmon_wmde, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Claimed] T230588: Wikidata Query Service is swapping items and properties
Zbyszko claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T230588 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Larske, Mathew.onipe, Esc3300, agray, Jakub.klimek, Gehel, bennofs, Magnus, Robby, Mmarx, jeremyb, Lea_Lacroix_WMDE, Manu1400, Lucas_Werkmeister_WMDE, Smalyshev, Moebeus, Aklapper, Fnielsen, darthmon_wmde, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T245637: disable WDQS jump to focus when used in an iframe
Zbyszko added a comment. @Lucas_Werkmeister_WMDE deployment done, can you verify the fix on production? TASK DETAIL https://phabricator.wikimedia.org/T245637 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Zbyszko, dcausse, Lucas_Werkmeister_WMDE, Lydia_Pintscher, Lea_Lacroix_WMDE, Aklapper, Iflorez, darthmon_wmde, alaa_wmde, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Salgo60, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Declined] T231411: Test new Updater service
Zbyszko closed this task as "Declined". Zbyszko added a comment. We're going in the new direction of rewriting updated into a streaming applicaiton. TASK DETAIL https://phabricator.wikimedia.org/T231411 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Igorkim78, Zbyszko Cc: Zbyszko, Lea_Lacroix_WMDE, Gehel, Igorkim78, Aklapper, Daniel_Mietchen, Fnielsen, EgonWillighagen, Abbe98, Smalyshev, Beast1978, Un1tY, Hook696, Daryl-TTMG, RomaAmorRoma, E.S.A-Sheild, darthmon_wmde, Meekrab2012, joker88john, CucyNoiD, Nandana, NebulousIris, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Af420, Darkminds3113, Bsandipan, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, merbst, LawExplorer, Vali.matei, WSH1906, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Volker_E, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Dinoguy1000, Manybubbles, Mbch331, Jay8g ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Unblock] T212826: Create dedicated Updater service in Blazegraph
Zbyszko closed subtask T231411: Test new Updater service as Declined. TASK DETAIL https://phabricator.wikimedia.org/T212826 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Igorkim78, Zbyszko Cc: Gehel, Abbe98, EgonWillighagen, Fnielsen, Daniel_Mietchen, Aklapper, Smalyshev, NavinRizwi, darthmon_wmde, Nandana, Lahi, Gq86, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Vali.matei, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Volker_E, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Dinoguy1000, Manybubbles, Mbch331, Jay8g ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Created] T246417: Create Async I/O Flink task for wikibase communication
Zbyszko created this task. Zbyszko added projects: Wikidata-Query-Service, Wikidata. TASK DESCRIPTION We need a Flink task to communicate with Wikibase API, to get entity data based on revision. Task should be pluggable into pipeline as an Async I/O task. TASK DETAIL https://phabricator.wikimedia.org/T246417 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: dcausse, Gehel, Aklapper, JAllemandou, Ottomata, Smalyshev, Iamamz3, Zbyszko, darthmon_wmde, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Unblock] T244590: EPIC: Rework the WDQS updater as an event driven application
Zbyszko closed subtask T245727: Create a streaming-updater submodule under query/wikidata/rdf as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Iamamz3, Smalyshev, Ottomata, JAllemandou, Aklapper, Zbyszko, Gehel, dcausse, darthmon_wmde, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Dinoguy1000, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Closed] T245727: Create a streaming-updater submodule under query/wikidata/rdf
Zbyszko closed this task as "Resolved". TASK DETAIL https://phabricator.wikimedia.org/T245727 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Gehel, Zbyszko, Aklapper, JAllemandou, Ottomata, Smalyshev, Iamamz3, dcausse, darthmon_wmde, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Edited] T247058: Deployment strategy and hardware requirement for new Flink based WDQS updater
Zbyszko updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T247058 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Aklapper, dcausse, Zbyszko, Gehel, darthmon_wmde, Legado_Shulgin, Nandana, Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Rxy, Jay8g, fgiunchedi ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Edited] T247058: Deployment strategy and hardware requirement for new Flink based WDQS updater
Zbyszko updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T247058 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Aklapper, dcausse, Zbyszko, Gehel, darthmon_wmde, Legado_Shulgin, Nandana, Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Rxy, Jay8g, fgiunchedi ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Created] T243587: Add rights for JenkinsBot for tagging and push to master on wikidata/query/rdf
Zbyszko created this task. Zbyszko added projects: Release-Engineering-Team, Wikidata-Query-Service. Restricted Application added a subscriber: Aklapper. Restricted Application added a project: Wikidata. TASK DESCRIPTION We want to be able to do releases of wikidata/query/rdf. To do that we need JenkinsBot user on gerrit to have correct rights on: https://gerrit.wikimedia.org/r/admin/projects/wikidata/query/rdf,access We need JenkinsBot to have following new rights on that project: - Create Annotated Tag - Push (also, Force Push) TASK DETAIL https://phabricator.wikimedia.org/T243587 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: dcausse, Gehel, Aklapper, Zbyszko, darthmon_wmde, Nandana, NebulousIris, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Liudvikas, Scott_WUaS, Jonas, Xmlizer, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Created] T243603: Create a way to deploy WDQS artifacts to Archiva with Jenkins
Zbyszko created this task. Zbyszko added a project: Wikidata-Query-Service. Restricted Application added a subscriber: Aklapper. Restricted Application added a project: Wikidata. TASK DESCRIPTION We want to have a similar way of deploying WDQS to Archiva as Analytics do here: https://integration.wikimedia.org/ci/job/analytics-refinery-release/m2release/ TASK DETAIL https://phabricator.wikimedia.org/T243603 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Aklapper, Zbyszko, darthmon_wmde, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Claimed] T105427: Need a way for WDQS updater to become aware of suppressed deletes
Zbyszko claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T105427 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: dcausse, Bugreporter, Sjoerddebruin, Krenair, gerritbot, JanZerebecki, Deskana, daniel, Legoktm, Aklapper, Smalyshev, darthmon_wmde, ET4Eva, Nandana, Lahi, Gq86, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Avner, Gehel, _jensen, rosalieper, Scott_WUaS, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Changed Status] T105427: Need a way for WDQS updater to become aware of suppressed deletes
Zbyszko changed the task status from "Open" to "Stalled". Zbyszko added a comment. We are stopping work on this for now, pending architecture change. We have a solution in mind that can make this work automatically - based on the events coming from mediawiki.revision-visibility-change we can refresh affected entities. Once we move to a fully reactive system, we can implement this quite easily. TASK DETAIL https://phabricator.wikimedia.org/T105427 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: dcausse, Bugreporter, Sjoerddebruin, Krenair, gerritbot, JanZerebecki, Deskana, daniel, Legoktm, Aklapper, Smalyshev, darthmon_wmde, ET4Eva, Nandana, Lahi, Gq86, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Avner, Gehel, _jensen, rosalieper, Scott_WUaS, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Edited] T241213: Organize and improve integration test coverage for WDQS Updater
Zbyszko updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T241213 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: dcausse, Aklapper, Zbyszko, darthmon_wmde, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Claimed] T249099: [WDQS Streaming Updater] Error during munging process
Zbyszko claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T249099 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: dcausse, Aklapper, Zbyszko, darthmon_wmde, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T249099: [WDQS Streaming Updater] Error during munging process
Zbyszko added a comment. After adding some additional logging, this is what we see: > 2020-04-08 07:44:07,287 ERROR org.wikidata.query.rdf.updater.ExtractTriplesFunction - Exception thrown for op: FullImport(Property:P1331 <https://phabricator.wikimedia.org/P1331>,2020-04-08T07:42:30Z,1152868984) > org.wikidata.query.rdf.tool.exception.ContainedException: Didn't get a revision id for [] > at org.wikidata.query.rdf.tool.rdf.Munger$MungeOperation.finishCommon(Munger.java:792) > at org.wikidata.query.rdf.tool.rdf.Munger$MungeOperation.munge(Munger.java:398) > at org.wikidata.query.rdf.tool.rdf.Munger.munge(Munger.java:131) > at org.wikidata.query.rdf.updater.ExtractTriplesFunction.munge(ExtractTriplesFunction.scala:53) > at org.wikidata.query.rdf.updater.ExtractTriplesFunction.map(ExtractTriplesFunction.scala:37) > at org.wikidata.query.rdf.updater.ExtractTriplesFunction.map(ExtractTriplesFunction.scala:20) > at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41) > at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:641) > at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:616) > at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:596) > at org.apache.flink.streaming.runtime.tasks.OperatorChain$BroadcastingOutputCollector.collect(OperatorChain.java:711) > at org.apache.flink.streaming.runtime.tasks.OperatorChain$BroadcastingOutputCollector.collect(OperatorChain.java:664) > at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:730) > at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:708) > at org.apache.flink.streaming.api.operators.TimestampedCollector.collect(TimestampedCollector.java:53) > at org.wikidata.query.rdf.updater.RouteIgnoredMutationToSideOutput.processElement(DecideMutationOperationMapFunction.scala:56) > at org.wikidata.query.rdf.updater.RouteIgnoredMutationToSideOutput.processElement(DecideMutationOperationMapFunction.scala:47) > at org.apache.flink.streaming.api.operators.ProcessOperator.processElement(ProcessOperator.java:66) > at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:641) > at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:616) > at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:596) > at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:730) > at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:708) > at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41) > at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:173) > at org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.processElement(StreamTaskNetworkInput.java:151) > at org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.emitNext(StreamTaskNetworkInput.java:128) > at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:69) > at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:311) > at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:187) > at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:487) > at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:470) > at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:707) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:532) > at java.lang.Thread.run(Thread.java:748) From what I see - every single issue with that is related to operation about property. Are we handling them incorrectly? TASK DETAIL https://phabricator.wikimedia.org/T249099 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: dcausse, Aklapper, Zbyszko, darthmon_wmde, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___
[Wikidata-bugs] [Maniphest] [Commented On] T241213: Organize and improve integration test coverage for WDQS Updater
Zbyszko added a comment. Will be done for the streaming updater. TASK DETAIL https://phabricator.wikimedia.org/T241213 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: dcausse, Aklapper, Zbyszko, darthmon_wmde, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Declined] T241213: Organize and improve integration test coverage for WDQS Updater
Zbyszko closed this task as "Declined". Zbyszko updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T241213 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: dcausse, Aklapper, Zbyszko, darthmon_wmde, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Retitled] T248464: Implement ouput format in Streaming Updater
Zbyszko renamed this task from "Implement NTriples format ouput in Streaming Updater" to "Implement ouput format in Streaming Updater". Zbyszko updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T248464 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Aklapper, Zbyszko, darthmon_wmde, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Edited] T248449: Add error handling for Streaming Updater
Zbyszko updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T248449 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Aklapper, Zbyszko, darthmon_wmde, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Retitled] T248450: Monitor Streaming Updater - metrics
Zbyszko renamed this task from "Monitor Streaming Updater Latency" to "Monitor Streaming Updater - metrics ". Zbyszko updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T248450 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Aklapper, Zbyszko, darthmon_wmde, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Edited] T248451: Custom parallelism configuration for Streaming Updater
Zbyszko updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T248451 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Aklapper, Zbyszko, darthmon_wmde, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Created] T249097: [WDQS Streaming Updater] Fix pipeline checkpointing
Zbyszko created this task. Zbyszko added a project: Wikidata-Query-Service. Restricted Application added a subscriber: Aklapper. Restricted Application added a project: Wikidata. TASK DESCRIPTION Early tests showed that checkpointing doesn't work very well, during 4 days run, statistics for checkpointing count look like this: Triggered: 1585In Progress: 1Completed: 16Failed: 1568Restored: 1567 AC: - Checkpointing works correctly TASK DETAIL https://phabricator.wikimedia.org/T249097 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Aklapper, Zbyszko, darthmon_wmde, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Retitled] T248450: [WDQS Streaming Updater] Monitor Streaming Updater - metrics
Zbyszko renamed this task from "Monitor Streaming Updater - metrics " to "[WDQS Streaming Updater] Monitor Streaming Updater - metrics ". Zbyszko claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T248450 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Aklapper, Zbyszko, darthmon_wmde, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Retitled] T248451: [WDQS Streaming Updater] Custom parallelism configuration for Streaming Updater
Zbyszko renamed this task from "Custom parallelism configuration for Streaming Updater" to "[WDQS Streaming Updater] Custom parallelism configuration for Streaming Updater". TASK DETAIL https://phabricator.wikimedia.org/T248451 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Aklapper, Zbyszko, darthmon_wmde, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Retitled] T248464: [WDQS Streaming Updater] Implement ouput format in Streaming Updater
Zbyszko renamed this task from "Implement ouput format in Streaming Updater" to "[WDQS Streaming Updater] Implement ouput format in Streaming Updater". TASK DETAIL https://phabricator.wikimedia.org/T248464 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Aklapper, Zbyszko, darthmon_wmde, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Retitled] T248452: [WDQS Streaming Updater] Deploy and configure Streaming Updater Hadoop YARN
Zbyszko renamed this task from "Deploy and configure Streaming Updater Hadoop YARN" to "[WDQS Streaming Updater] Deploy and configure Streaming Updater Hadoop YARN". TASK DETAIL https://phabricator.wikimedia.org/T248452 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Aklapper, Zbyszko, darthmon_wmde, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Created] T249099: [WDQS Streaming Updater] Error during munging process
Zbyszko created this task. Zbyszko added a project: Wikidata-Query-Service. Restricted Application added a subscriber: Aklapper. Restricted Application added a project: Wikidata. TASK DESCRIPTION Pipeline breaks with a following exception: org.wikidata.query.rdf.tool.exception.ContainedException: Didn't get a revision id for [] at org.wikidata.query.rdf.tool.rdf.Munger$MungeOperation.finishCommon(Munger.java:792) at org.wikidata.query.rdf.tool.rdf.Munger$MungeOperation.munge(Munger.java:398) at org.wikidata.query.rdf.tool.rdf.Munger.munge(Munger.java:131) at org.wikidata.query.rdf.updater.ExtractTriplesFunction.munge(ExtractTriplesFunction.scala:34) at org.wikidata.query.rdf.updater.ExtractTriplesFunction.map(ExtractTriplesFunction.scala:25) at org.wikidata.query.rdf.updater.ExtractTriplesFunction.map(ExtractTriplesFunction.scala:15) at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41) at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:641) at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:616) at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:596) at org.apache.flink.streaming.runtime.tasks.OperatorChain$BroadcastingOutputCollector.collect(OperatorChain.java:711) at org.apache.flink.streaming.runtime.tasks.OperatorChain$BroadcastingOutputCollector.collect(OperatorChain.java:664) at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:730) at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:708) at org.apache.flink.streaming.api.operators.TimestampedCollector.collect(TimestampedCollector.java:53) at org.wikidata.query.rdf.updater.RouteIgnoredMutationToSideOutput.processElement(DecideMutationOperationMapFunction.scala:56) at org.wikidata.query.rdf.updater.RouteIgnoredMutationToSideOutput.processElement(DecideMutationOperationMapFunction.scala:47) at org.apache.flink.streaming.api.operators.ProcessOperator.processElement(ProcessOperator.java:66) at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:641) at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:616) at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:596) at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:730) at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:708) at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41) at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:173) at org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.processElement(StreamTaskNetworkInput.java:151) at org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.emitNext(StreamTaskNetworkInput.java:128) at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:69) at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:311) at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:187) at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:487) at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:470) at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:707) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:532) at java.lang.Thread.run(Thread.java:748) TASK DETAIL https://phabricator.wikimedia.org/T249099 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Aklapper, Zbyszko, darthmon_wmde, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Retitled] T248449: [WDQS Streaming Updater] Add error handling for Streaming Updater
Zbyszko renamed this task from "Add error handling for Streaming Updater" to "[WDQS Streaming Updater] Add error handling for Streaming Updater". TASK DETAIL https://phabricator.wikimedia.org/T248449 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Aklapper, Zbyszko, darthmon_wmde, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T249099: [WDQS Streaming Updater] Error during munging process
Zbyszko added a comment. No, right now exceptions are just uncaught. As for an error itself - triple list doesn't have to be empty. During munging, statement list is being modified and by the time that this exception is thrown, it can be empty. I'm not yet suggesting that this is the case here, though. TASK DETAIL https://phabricator.wikimedia.org/T249099 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: dcausse, Aklapper, Zbyszko, darthmon_wmde, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Created] T248451: Custom parallelism configuration for Streaming Updater
Zbyszko created this task. Zbyszko added a project: Wikidata-Query-Service. Restricted Application added a subscriber: Aklapper. Restricted Application added a project: Wikidata. TASK DESCRIPTION AC: TODO TASK DETAIL https://phabricator.wikimedia.org/T248451 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Aklapper, Zbyszko, darthmon_wmde, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T248450: Monitor Streaming Updater Latency
Zbyszko added a project: Wikidata-Query-Service. Restricted Application added a project: Wikidata. TASK DETAIL https://phabricator.wikimedia.org/T248450 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Aklapper, Zbyszko, darthmon_wmde, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Created] T248452: Deploy and configure Streaming Updater Hadoop YARN
Zbyszko created this task. Zbyszko added a project: Wikidata-Query-Service. Restricted Application added a subscriber: Aklapper. Restricted Application added a project: Wikidata. TASK DESCRIPTION AC: TODO TASK DETAIL https://phabricator.wikimedia.org/T248452 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Aklapper, Zbyszko, darthmon_wmde, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Created] T248449: Add error handling for Streaming Updater
Zbyszko created this task. Zbyszko added a project: Wikidata-Query-Service. Restricted Application added a subscriber: Aklapper. Restricted Application added a project: Wikidata. TASK DESCRIPTION AC: TODO TASK DETAIL https://phabricator.wikimedia.org/T248449 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Aklapper, Zbyszko, darthmon_wmde, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Created] T248464: Implement NTriples format ouput in Streaming Updater
Zbyszko created this task. Zbyszko added a project: Wikidata-Query-Service. Restricted Application added a subscriber: Aklapper. Restricted Application added a project: Wikidata. TASK DESCRIPTION AC: TODO TASK DETAIL https://phabricator.wikimedia.org/T248464 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Aklapper, Zbyszko, darthmon_wmde, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Created] T249500: [WDQS Streaming Updater] Reuse WikibaseRepository
Zbyszko created this task. Zbyszko added a project: Wikidata-Query-Service. Restricted Application added a subscriber: Aklapper. Restricted Application added a project: Wikidata. TASK DESCRIPTION AC: - Streaming Updater uses WikibaseRepositry - WikibaseRepository metrics are reported correctly through Apache Flink Metrics TASK DETAIL https://phabricator.wikimedia.org/T249500 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Aklapper, Zbyszko, darthmon_wmde, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Created] T251096: [WDQS Streaming Updater] Organise module structure
Zbyszko created this task. Zbyszko added a project: Wikidata-Query-Service. Restricted Application added a subscriber: Aklapper. Restricted Application added a project: Wikidata. TASK DESCRIPTION Right now, every functionality related to updating is either in tools/common (old updater + shared code) and in streaming-updater + rdf-spark-tools. We should reorganise the structure with best practices in mind. AC: - shared code is put in separate module and logically named. - tools is no longer a catch-all module. TASK DETAIL https://phabricator.wikimedia.org/T251096 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Aklapper, Zbyszko, darthmon_wmde, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, DannyS712 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T244590: EPIC: Rework the WDQS updater as an event driven application
Zbyszko added a subtask: T251096: [WDQS Streaming Updater] Organise module structure. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: revi, Mholloway, Ladsgroup, Multichill, darthmon_wmde, Iamamz3, Smalyshev, Ottomata, JAllemandou, Aklapper, Zbyszko, Gehel, dcausse, NavinRizwi, CBogen, DannyS712, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Dinoguy1000, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T251096: [WDQS Streaming Updater] Organise module structure
Zbyszko added a parent task: T244590: EPIC: Rework the WDQS updater as an event driven application. TASK DETAIL https://phabricator.wikimedia.org/T251096 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Aklapper, Zbyszko, darthmon_wmde, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Retitled] T251275: [WDQS Streaming Updater] Update blazegraph based on the content present in the streaming updater output kafka stream
Zbyszko renamed this task from "Update blazegraph based on the content present in the streaming updater output kafka stream" to "[WDQS Streaming Updater] Update blazegraph based on the content present in the streaming updater output kafka stream". TASK DETAIL https://phabricator.wikimedia.org/T251275 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: dcausse, Aklapper, darthmon_wmde, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Created] T252504: Smoke test the canary deployment of WDQS
Zbyszko created this task. Zbyszko added a project: Wikidata-Query-Service. Restricted Application added a subscriber: Aklapper. Restricted Application added a project: Wikidata. TASK DESCRIPTION Currently, the deployer, after deploying WDQS to canary server (wdqs1003) has to manually perform smoke test to make sure that deployment went correctly. There is no reason for this step not to happen automatically AC: - After canary deployment, execute /queries/tests.sh on wdqs1003 server TASK DETAIL https://phabricator.wikimedia.org/T252504 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Aklapper, Zbyszko, CBogen, darthmon_wmde, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Created] T252508: Improve visibility of WDQS inaccessability
Zbyszko created this task. Zbyszko added a project: Wikidata-Query-Service. Restricted Application added a subscriber: Aklapper. Restricted Application added a project: Wikidata. TASK DESCRIPTION During last WDQS outage it was difficult to correctly ascertain the impact of it. Current graphs, while show error rates - also include errors that are BAU ones - like rejections because of throttling. Other then eyeballing it or using more complicated statistical methods, it's impossible to determine which errors are because of an outage. Because of this it's hard to asses the general deterioration of service. We need a way (metric and/or graph) to do that. AC: - Provide a way to determine, how many requests were denied because of an outage. TASK DETAIL https://phabricator.wikimedia.org/T252508 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Aklapper, Zbyszko, CBogen, darthmon_wmde, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T252503: Create automatically updated CI test environment
Zbyszko added a project: Sustainability (Incident Prevention). TASK DETAIL https://phabricator.wikimedia.org/T252503 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Aklapper, Zbyszko, CBogen, darthmon_wmde, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, elukey, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, Jay8g ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Closed] T249500: [WDQS Streaming Updater] Reuse WikibaseRepository
Zbyszko closed this task as "Resolved". TASK DETAIL https://phabricator.wikimedia.org/T249500 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Aklapper, Zbyszko, CBogen, darthmon_wmde, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Unblock] T244590: EPIC: Rework the WDQS updater as an event driven application
Zbyszko closed subtask T249500: [WDQS Streaming Updater] Reuse WikibaseRepository as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: revi, Mholloway, Ladsgroup, Multichill, darthmon_wmde, Iamamz3, Smalyshev, Ottomata, JAllemandou, Aklapper, Zbyszko, Gehel, dcausse, NavinRizwi, CBogen, DannyS712, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Dinoguy1000, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Closed] T243292: Fix the munger to support commons RDF dump
Zbyszko closed this task as "Resolved". Zbyszko added a comment. Munger correctly processes commons dump. TASK DETAIL https://phabricator.wikimedia.org/T243292 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Mahir256, Physikerwelt, ArielGlenn, Aklapper, dcausse, CBogen, darthmon_wmde, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Unblock] T221917: Create RDF dump of structured data on Commons
Zbyszko closed subtask T243292: Fix the munger to support commons RDF dump as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T221917 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: nettrom_WMF, Mahir256, dcausse, EBernhardson, Cparle, Abit, Gehel, jleedev, hoo, ArielGlenn, WMDE-leszek, Poyekhali, Steinsplitter, Aklapper, Lydia_Pintscher, Bugreporter, Tgr, Ramsey-WMF, Jarekt, Addshore, Tpt, Salgo60, Lucas_Werkmeister_WMDE, Smalyshev, jannee_e, CBogen, darthmon_wmde, Nandana, Lahi, Gq86, GoranSMilovanovic, Lunewa, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Taiwania_Justo, Scott_WUaS, Jonas, Xmlizer, Ixocactus, Wong128hk, gnosygnu, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, El_Grafo, Dinoguy1000, Manybubbles, Mbch331, Keegan ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Unblock] T251497: Adapt munging process for SDoC
Zbyszko closed subtask T243292: Fix the munger to support commons RDF dump as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T251497 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Aklapper, Gehel, CBogen, darthmon_wmde, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Unblock] T243270: Test commons RDF dumps on sdcquery.wmflabs.org
Zbyszko closed subtask T243292: Fix the munger to support commons RDF dump as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T243270 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: dcausse, Aklapper, CBogen, darthmon_wmde, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Closed] T246417: Create Async I/O Flink task for wikibase communication
Zbyszko closed this task as "Resolved". Zbyszko added a comment. Async I/O has limitations that would complicate the pipeline and in the end it seems it's not actually needed. TASK DETAIL https://phabricator.wikimedia.org/T246417 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: dcausse, Gehel, Aklapper, JAllemandou, Ottomata, Smalyshev, Iamamz3, Zbyszko, CBogen, darthmon_wmde, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Unblock] T244590: EPIC: Rework the WDQS updater as an event driven application
Zbyszko closed subtask T246417: Create Async I/O Flink task for wikibase communication as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: revi, Mholloway, Ladsgroup, Multichill, darthmon_wmde, Iamamz3, Smalyshev, Ottomata, JAllemandou, Aklapper, Zbyszko, Gehel, dcausse, NavinRizwi, CBogen, DannyS712, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Dinoguy1000, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Unblock] T244590: EPIC: Rework the WDQS updater as an event driven application
Zbyszko closed subtask T248450: [WDQS Streaming Updater] Monitor Streaming Updater - metrics as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: revi, Mholloway, Ladsgroup, Multichill, darthmon_wmde, Iamamz3, Smalyshev, Ottomata, JAllemandou, Aklapper, Zbyszko, Gehel, dcausse, NavinRizwi, CBogen, DannyS712, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Dinoguy1000, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Closed] T248450: [WDQS Streaming Updater] Monitor Streaming Updater - metrics
Zbyszko closed this task as "Resolved". Zbyszko added a comment. Metrics are available on grafana.wikimedia.org through Graphite: https://grafana.wikimedia.org/d/_kZ1VGRGk/wdqs-pipeline?orgId=1=1m TASK DETAIL https://phabricator.wikimedia.org/T248450 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Aklapper, Zbyszko, CBogen, darthmon_wmde, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T252504: Smoke test the canary deployment of WDQS
Zbyszko added a project: Sustainability (Incident Prevention). TASK DETAIL https://phabricator.wikimedia.org/T252504 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Aklapper, Zbyszko, CBogen, darthmon_wmde, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, elukey, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, Jay8g ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T252508: Improve visibility of WDQS inaccessability
Zbyszko added a project: Sustainability (Incident Prevention). TASK DETAIL https://phabricator.wikimedia.org/T252508 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Aklapper, Zbyszko, CBogen, darthmon_wmde, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, elukey, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, Jay8g ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T252503: Create automatically updated CI test environment
Zbyszko added a project: Wikidata-Query-Service. Restricted Application added a project: Wikidata. TASK DETAIL https://phabricator.wikimedia.org/T252503 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Aklapper, Zbyszko, CBogen, darthmon_wmde, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Closed] T243603: Create a way to deploy WDQS artifacts to Archiva with Jenkins
Zbyszko closed this task as "Resolved". Zbyszko added a comment. We have a build in Jenkins for that now. TASK DETAIL https://phabricator.wikimedia.org/T243603 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: WMDE-leszek, Aklapper, Zbyszko, CBogen, darthmon_wmde, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T252503: Create automatically updated CI test environment
Zbyszko updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T252503 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Gehel, Aklapper, Zbyszko, CBogen, Akuckartz, darthmon_wmde, Nandana, Namenlos314, jijiki, Klaas_Z4us_V, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, elukey, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, Jay8g ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T261840: Jetty startup logs in /var/log/wdqs
Zbyszko created this task. Zbyszko added a project: Wikidata-Query-Service. Restricted Application added a subscriber: Aklapper. Restricted Application added a project: Wikidata. TASK DESCRIPTION As a WDQS/WCQS maintainer a want to be able to access jetty startup logs from a file in /var/log/wdqs so that I can use those logs in the same way as all of the other ones. AC: - Jetty startup log is written into /var/log/wdqs/ director TASK DETAIL https://phabricator.wikimedia.org/T261840 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Aklapper, Zbyszko, CBogen, Akuckartz, darthmon_wmde, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T261097: WDQS Categories reload is failing on thankyouwiki
Zbyszko added a comment. @RKemper we should be able to retry categories reload after deploying this. TASK DETAIL https://phabricator.wikimedia.org/T261097 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: dcausse, RKemper, Gehel, Aklapper, lmata, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, darthmon_wmde, Kent7301, joker88john, Legado_Shulgin, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Davinaclare77, Qtn1293, Cpaulf30, Techguru.pc, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Zppix, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Rxy, Jay8g, fgiunchedi ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T258240: Refactor Options handling in Streaming Updater
Zbyszko claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T258240 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: dcausse, Aklapper, Zbyszko, CBogen, Akuckartz, darthmon_wmde, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T262265: Provide real-time updates for WCQS
Zbyszko created this task. Zbyszko added a project: Wikidata-Query-Service. Restricted Application added a subscriber: Aklapper. Restricted Application added a project: Wikidata. TASK DESCRIPTION As a user of WCQS I want to have a real-time updates to WCQS so that I can see the changes soon after they are done and without the need of weekly downtime. Proposed solution is to use Streaming Updater with WCQS. This would require: - some changes for Streaming updater consumer(mostly around stream filtering) - probably only configuration one for producer - publicly available patch stream for SDC content (beta lives on horizon, with no access to Jumbo kafka cluster) Note that this would still be a part of beta service - Streaming Updater is not yet ready for production deployment (and neither is WCQS). AC: - WCQS is updated with SDC changes as they happen (with minimal lag) - Data updates generates no downtime TASK DETAIL https://phabricator.wikimedia.org/T262265 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Aklapper, Zbyszko, CBogen, Akuckartz, darthmon_wmde, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T262265: Provide real-time updates for WCQS
Zbyszko updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T262265 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Aklapper, Zbyszko, CBogen, Akuckartz, darthmon_wmde, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T262265: Provide real-time updates for WCQS
Zbyszko added a comment. If we go by the solution of having an additional pipeline for SDC - https://phabricator.wikimedia.org/T262020 should be done first. TASK DETAIL https://phabricator.wikimedia.org/T262265 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Aklapper, Zbyszko, CBogen, Akuckartz, darthmon_wmde, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T262020: Make sure that transaction.id assigned by Flink is unique for concurrent runs of Streaming Updater pipeline
Zbyszko added a comment. If we go by the solution of having an additional pipeline for SDC - https://phabricator.wikimedia.org/T262020 should be done first. TASK DETAIL https://phabricator.wikimedia.org/T262020 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Aklapper, Zbyszko, CBogen, Akuckartz, darthmon_wmde, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T262265: Provide real-time updates for WCQS
Zbyszko added a comment. Just to be own's devil's advocate or to provide alternatives, we can solve both downtime and real-time updates with the old updater. Additionally, we can eliminate the downtime by having two blazegraph instances in an active/standby setup. TASK DETAIL https://phabricator.wikimedia.org/T262265 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Aklapper, Zbyszko, CBogen, Akuckartz, darthmon_wmde, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T260568: [EPIC] Productionize WCQS
Zbyszko added a subtask: T262265: Provide real-time updates for WCQS. TASK DETAIL https://phabricator.wikimedia.org/T260568 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: CBogen, Gehel, Aklapper, NavinRizwi, Akuckartz, darthmon_wmde, DannyS712, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Dinoguy1000, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T262265: Provide real-time updates for WCQS
Zbyszko added a parent task: T260568: [EPIC] Productionize WCQS. TASK DETAIL https://phabricator.wikimedia.org/T262265 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Aklapper, Zbyszko, CBogen, Akuckartz, darthmon_wmde, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T261097: WDQS Categories reload is failing on thankyouwiki
Zbyszko added a comment. I went with the fix for wikidata/query/rdf scripts - it made most sense to me, since issues with a single wiki should block updates for others. Once it's merged, I'll add an entry on that to runbook TASK DETAIL https://phabricator.wikimedia.org/T261097 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: dcausse, RKemper, Gehel, Aklapper, lmata, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, darthmon_wmde, Kent7301, joker88john, Legado_Shulgin, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Davinaclare77, Qtn1293, Cpaulf30, Techguru.pc, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Zppix, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Rxy, Jay8g, fgiunchedi ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T262178: One week after SDC edits the data still shows up in WCQS queries
Zbyszko added a comment. The process we have right now is that we use SDC dumps to reload the data each week. Dumps are made each Sunday, which means, that all the changes made between Aug 30th and Sep 6th will only show up in the dump released on Sep 6th. I pushed the update time from Tuesday to Monday, so that the change is visible faster, but it still dependent on the dump itself. Since the current process takes around 4h, we decided not to do the reload more often. Reload should happen today, so after today the change should be visible. Please note that this is a limitation of the beta service - real time updates will be implemented at some point. TASK DETAIL https://phabricator.wikimedia.org/T262178 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Mahir256, Zbyszko, Aklapper, Jarekt, CBogen, Akuckartz, darthmon_wmde, Nandana, JKSTNK, Namenlos314, Lahi, PDrouin-WMF, Gq86, E1presidente, Ramsey-WMF, Cparle, Anooprao, SandraF_WMF, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, Tramullas, Acer, merbst, LawExplorer, Salgo60, Silverfish, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Susannaanas, Jane023, jkroll, Wikidata-bugs, Jdouglas, Base, matthiasmullie, aude, Tobias1984, Manybubbles, Ricordisamoa, Wesalius, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T262178: One week after SDC edits the data still shows up in WCQS queries
Zbyszko added a comment. The process we have right now is that we use SDC dumps to reload the data each week. Dumps are made each Sunday, which means, that all the changes made between Aug 30th and Sep 6th will only show up in the dump released on Sep 6th. I pushed the update time from Tuesday to Monday, so that the change is visible faster, but it still dependent on the dump itself. Since the current process takes around 4h, we decided not to do the reload more often. Reload should happen today, so after today the change should be visible. Please note that this is a limitation of the beta service - real time updates will be implemented at some point. TASK DETAIL https://phabricator.wikimedia.org/T262178 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Mahir256, Zbyszko, Aklapper, Jarekt, CBogen, Akuckartz, darthmon_wmde, Nandana, JKSTNK, Namenlos314, Lahi, PDrouin-WMF, Gq86, E1presidente, Ramsey-WMF, Cparle, Anooprao, SandraF_WMF, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, Tramullas, Acer, merbst, LawExplorer, Salgo60, Silverfish, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Susannaanas, Jane023, jkroll, Wikidata-bugs, Jdouglas, Base, matthiasmullie, aude, Tobias1984, Manybubbles, Ricordisamoa, Wesalius, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T262828: Near zero downtime Data reload for WCQS
Zbyszko claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T262828 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Gehel, Bugreporter, Aklapper, Zbyszko, CBogen, Akuckartz, darthmon_wmde, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T262828: Near zero downtime Data reload for WCQS
Zbyszko triaged this task as "High" priority. TASK DETAIL https://phabricator.wikimedia.org/T262828 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Gehel, Bugreporter, Aklapper, Zbyszko, CBogen, Akuckartz, darthmon_wmde, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T261119: Architecture review of Flink based WDQS Streaming Updater
Zbyszko claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T261119 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Aklapper, Zbyszko, dcausse, Gehel, CBogen, Akuckartz, darthmon_wmde, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T262828: Near zero downtime Data reload for WCQS
Zbyszko created this task. Zbyszko added a project: Wikidata-Query-Service. Restricted Application added a subscriber: Aklapper. Restricted Application added a project: Wikidata. TASK DESCRIPTION As a user I want a shortest possible downtime for WCQS in case of data reload, so that I can continue using that. Proposed solutions: - Feed the second blazegraph instance, switch journals afterwards - The same but with two namespaces (single instance of Blazegraph) Category reload happens this way already. Switch can also happen via nginx. AC: - data reload doesn't trigger maintenance page - data is available after reload. TASK DETAIL https://phabricator.wikimedia.org/T262828 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Aklapper, Zbyszko, CBogen, Akuckartz, darthmon_wmde, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T262828: Near zero downtime Data reload for WCQS
Zbyszko moved this task from All WDQS-related tasks to Current work on the Wikidata-Query-Service board. Zbyszko added a project: Discovery-Search (Current work). TASK DETAIL https://phabricator.wikimedia.org/T262828 WORKBOARD https://phabricator.wikimedia.org/project/board/891/ EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Aklapper, Zbyszko, CBogen, Akuckartz, darthmon_wmde, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T262020: Make sure that transaction.id assigned by Flink is unique for concurrent runs of Streaming Updater pipeline
Zbyszko added a project: Wikidata-Query-Service. Restricted Application added a project: Wikidata. TASK DETAIL https://phabricator.wikimedia.org/T262020 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Aklapper, Zbyszko, CBogen, Akuckartz, darthmon_wmde, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T261097: WDQS Categories reload is failing on thankyouwiki
Zbyszko claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T261097 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: dcausse, RKemper, Gehel, Aklapper, lmata, CBogen, Akuckartz, darthmon_wmde, Legado_Shulgin, Nandana, Namenlos314, Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Rxy, Jay8g, fgiunchedi ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T262178: One week after SDC edits the data still shows up in WCQS queries
Zbyszko added a comment. Today's reload happened and if this (https://tinyurl.com/y5vd95rm) query is correct, there are no duplicates. @Jarekt, can you confirm? TASK DETAIL https://phabricator.wikimedia.org/T262178 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Mahir256, Zbyszko, Aklapper, Jarekt, CBogen, Akuckartz, darthmon_wmde, Nandana, JKSTNK, Namenlos314, Lahi, PDrouin-WMF, Gq86, E1presidente, Ramsey-WMF, Cparle, Anooprao, SandraF_WMF, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, Tramullas, Acer, merbst, LawExplorer, Salgo60, Silverfish, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Susannaanas, Jane023, jkroll, Wikidata-bugs, Jdouglas, Base, matthiasmullie, aude, Tobias1984, Manybubbles, Ricordisamoa, Wesalius, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T264042: Query service returns weird Commons links when asking for P50 (author) statement value
Zbyszko removed Zbyszko as the assignee of this task. TASK DETAIL https://phabricator.wikimedia.org/T264042 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Zbyszko, CBogen, Lucas_Werkmeister_WMDE, Aklapper, matej_suchanek, Vojtech.dostal, Akuckartz, darthmon_wmde, Nandana, Namenlos314, Lahi, Gq86, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T263855: mvn package fails for wikidata-query-rdf on Mac OS 10.13.6 High Sierra
Zbyszko added a comment. The error you get: Referenced from: /private/var/folders/2t/2g54bjr10830rv00508_y13wgn/T/flink-io-6ec39247-613e-410a-a83f-712f841ce3a8/rocksdb-lib-396871de50f5fa7595c1071b59c34498/librocksdbjni-osx.jnilib (which was built for Mac OS X 10.15) would suggest that the issue s with the macOS version - system update will probably help. Unfortunately, we don't have the equipment to investigate that issue. If update isn't an option, there is some advice on this here - https://github.com/JohnSnowLabs/spark-nlp/issues/652#issuecomment-537036577 - hopefully that will help. TASK DETAIL https://phabricator.wikimedia.org/T263855 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Zbyszko, Seppl2013, Aklapper, CBogen, Akuckartz, darthmon_wmde, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T264042: Query service returns weird Commons links when asking for P50 (author) statement value
Zbyszko claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T264042 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: CBogen, Lucas_Werkmeister_WMDE, Aklapper, matej_suchanek, Vojtech.dostal, Akuckartz, darthmon_wmde, Nandana, Namenlos314, Lahi, Gq86, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T246004: Spike: Can/should Swift be used as Flink checkpoint backend?
Zbyszko added a comment. @fgiunchedi Currently, Flink pipeline resides on the Analytics Hadoop cluster. As for the question whether Flink creates it's containers - I think not, it did complain when there was no container, so I assume it expects one. TASK DETAIL https://phabricator.wikimedia.org/T246004 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: elukey, EBernhardson, JMeybohm, fgiunchedi, CBogen, #analytics, dcausse, Gehel, Zbyszko, Aklapper, JAllemandou, Smalyshev, Iamamz3, Ottomata, NavinRizwi, Akuckartz, darthmon_wmde, DannyS712, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Dinoguy1000, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T256949: The streaming updater should support suppressed deletes
Zbyszko removed a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T256949 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Bugreporter, dcausse, Aklapper, CBogen, Akuckartz, darthmon_wmde, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, Alter-paule, Beast1978, Un1tY, Hook696, Kent7301, joker88john, CucyNoiD, Gaboe420, Giuliamocci, Cpaulf30, Af420, Bsandipan, Lewizho99, Maathavan ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T246004: Spike: Can/should Swift be used as Flink checkpoint backend?
Zbyszko added a comment. I'm fine with the thanos cluster option - we can proceed with that. @Ottomata do you know if thanos swift cluster is accessible from hadoop? TASK DETAIL https://phabricator.wikimedia.org/T246004 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: elukey, EBernhardson, JMeybohm, fgiunchedi, CBogen, #analytics, dcausse, Gehel, Zbyszko, Aklapper, JAllemandou, Smalyshev, Iamamz3, Ottomata, NavinRizwi, Akuckartz, darthmon_wmde, DannyS712, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Dinoguy1000, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T264042: Query service returns weird Commons links when asking for P50 (author) statement value
Zbyszko claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T264042 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Zbyszko, CBogen, Lucas_Werkmeister_WMDE, Aklapper, matej_suchanek, Vojtech.dostal, Akuckartz, darthmon_wmde, Nandana, Namenlos314, Lahi, Gq86, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T265891: Add Wikimedia Commons Query Service endpoint in the WDQS federation whitelist
Zbyszko added a comment. It's currently not possible - as long as Wikimedia Commons Query Service is in beta, OAuth authorization will not work with federation. We want to reevaluate this federation once we productionize WCQS. TASK DETAIL https://phabricator.wikimedia.org/T265891 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Zbyszko, Zache, Aklapper, CBogen, Akuckartz, darthmon_wmde, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T264042: Query service returns weird Commons links when asking for P50 (author) statement value
Zbyszko added a comment. First look at the issue - usual culpruits don't seem to apply here: - Munged dump is correct for one of the affected entitites - query, after loading the data into blazegraph is fine,too Interestingly, every single entity I found with this, was updated on 12 June 2020 (though with correct data and by different users) - I'm not sure if there's a significance here yet or just a freak coincidence. Reload might be in order - currently process works correctly. TASK DETAIL https://phabricator.wikimedia.org/T264042 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Zbyszko, CBogen, Lucas_Werkmeister_WMDE, Aklapper, matej_suchanek, Vojtech.dostal, Akuckartz, darthmon_wmde, Nandana, Namenlos314, Lahi, Gq86, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T264042: Query service returns weird Commons links when asking for P50 (author) statement value
Zbyszko added a comment.
According to this query:
SELECT DISTINCT ?item ?rev ?date WHERE {
{
?st ps:P50|ps:P106|ps:P136|ps:P275|pq:P512|pq:P106
<http://commons.wikimedia.org/wiki/Special:FilePath/>.
?item ?p ?st
}
UNION
{
?item wdt:P50|wdt:P106|wdt:P136|wdt:P275
<http://commons.wikimedia.org/wiki/Special:FilePath/> .
}
?item schema:version ?rev .
?item schema:dateModified ?date .
}
we have 119 entities affected which I will refresh. Results - F32399934:
query (2).json <https://phabricator.wikimedia.org/F32399934>.
TASK DETAIL
https://phabricator.wikimedia.org/T264042
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Zbyszko
Cc: Zbyszko, CBogen, Lucas_Werkmeister_WMDE, Aklapper, matej_suchanek,
Vojtech.dostal, Akuckartz, darthmon_wmde, Nandana, Namenlos314, Lahi, Gq86,
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper,
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984,
Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T264042: Query service returns weird Commons links when asking for P50 (author) statement value
Zbyszko added a comment.
All the entities affected were refreshed and this:
SELECT ?p (COUNT(*) AS ?count) WHERE {
?s ?p <http://commons.wikimedia.org/wiki/Special:FilePath/>.
}
GROUP BY ?p
ORDER BY DESC(?count)
no longer returns any results.
All affected entitites had been updated on 12 June 2020.
@Lucas_Werkmeister_WMDE, are you aware of any issue around that day concerning
wikidata? I can't find anything on wdqs side that could produce that kind of a
localized result.
TASK DETAIL
https://phabricator.wikimedia.org/T264042
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Zbyszko
Cc: Zbyszko, CBogen, Lucas_Werkmeister_WMDE, Aklapper, matej_suchanek,
Vojtech.dostal, Akuckartz, darthmon_wmde, Nandana, Namenlos314, Lahi, Gq86,
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper,
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984,
Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T246004: Spike: Can/should Swift be used as Flink checkpoint backend?
Zbyszko added a comment. We lack precise data for production - we haven't really optimised yet and complete functionality isn't yet ready (it will soon, though). Rarely, we get around 8-9GB checkpoints (when bootstrapping for example), but they do not happen regularly. Normally, checkpoints are incremental and each 30 seconds they drop data of sub-1MB volume. We may change those values, but we'll still use incremental checkpoints. It's hard to tell when it comes to concurrency. It should be something around 20-30 threads writing small amounts of data, but due to the nature of the process, it's hard to know how many of this will be doing that at the same time. TASK DETAIL https://phabricator.wikimedia.org/T246004 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: elukey, EBernhardson, JMeybohm, fgiunchedi, CBogen, #analytics, dcausse, Gehel, Zbyszko, Aklapper, JAllemandou, Smalyshev, Iamamz3, Ottomata, NavinRizwi, Akuckartz, darthmon_wmde, DannyS712, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Dinoguy1000, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T248449: [WDQS Streaming Updater] Add error handling for Streaming Updater
Zbyszko added a comment. In T248449#6411542 <https://phabricator.wikimedia.org/T248449#6411542>, @dcausse wrote: > In T248449#6382230 <https://phabricator.wikimedia.org/T248449#6382230>, @Zbyszko wrote: > >> We need to decide our approach on possible data corruption issues as well. One that comes to mind is a rev create with revid higher than previouis page delete. > > All scenario could happen if events are received out of order and are not necessarily "data corruption", revcreate with revid > a previous page delete may happen if a page-undelete event for this revid is late. Right, I wasn't aware back when writing a comment how exactly undeletes are handled - probably in most case that will be out of order events. I'm wondering how should we treat those situation - if we won't receive some intermediate revision, we simply diff the newer one received and ignore the one out of order. Should we treat page undeletes events the same way? In any case, that's probably out of scope for this ticket. TASK DETAIL https://phabricator.wikimedia.org/T248449 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse, Zbyszko Cc: dcausse, Aklapper, Zbyszko, CBogen, Akuckartz, darthmon_wmde, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T251515: Automate data reload for SPARQL Endpoint for Commons
Zbyszko added a comment. During the first data reload for some reason there data was not restored properly. I couldn't find a root cause of this - I'm doing some small changes to have a better understanding of the issue if it happens again. TASK DETAIL https://phabricator.wikimedia.org/T251515 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: CBogen, Mstyles, dcausse, Zbyszko, Aklapper, Lea_Lacroix_WMDE, Gehel, Alter-paule, Beast1978, Un1tY, Akuckartz, Hook696, darthmon_wmde, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T248449: [WDQS Streaming Updater] Add error handling for Streaming Updater
Zbyszko updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T248449 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: dcausse, Aklapper, Zbyszko, CBogen, Akuckartz, darthmon_wmde, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T251096: [WDQS Streaming Updater] Organise module structure
Zbyszko claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T251096 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Aklapper, Zbyszko, CBogen, Akuckartz, darthmon_wmde, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T248449: [WDQS Streaming Updater] Add error handling for Streaming Updater
Zbyszko updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T248449 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: dcausse, Aklapper, Zbyszko, CBogen, Akuckartz, darthmon_wmde, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T248449: [WDQS Streaming Updater] Add error handling for Streaming Updater
Zbyszko updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T248449 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: dcausse, Aklapper, Zbyszko, CBogen, Akuckartz, darthmon_wmde, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T246004: Spike: Can/should Swift be used as Flink checkpoint backend?
Zbyszko added a comment. @fgiunchedi We estimate we'd need around 500GB of storage for the streaming updater (not accounting for replicas). Our use case is almost always write only (checkpoints are read only on pipeline restarts, which ideally will be done rarely) - but we have a elasticity when it comes to configuration of the checkpoints interval. TASK DETAIL https://phabricator.wikimedia.org/T246004 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: EBernhardson, JMeybohm, fgiunchedi, CBogen, #analytics, dcausse, Gehel, Zbyszko, Aklapper, JAllemandou, Smalyshev, Iamamz3, Ottomata, NavinRizwi, Akuckartz, darthmon_wmde, DannyS712, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Dinoguy1000, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
