[Wikidata-bugs] [Maniphest] [Updated] T252068: WQDS Data Reload

2020-05-06 Thread Gehel
Gehel added projects: Discovery-Search (Current work), Operations. TASK DETAIL https://phabricator.wikimedia.org/T252068 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper, Gehel Cc: RKemper, Aklapper, CBogen, darthmon_wmde, Legado_Shulgin

[Wikidata-bugs] [Maniphest] [Closed] T193466: Consider using WikidataToolkit for testing

2020-05-06 Thread Gehel
Gehel closed this task as "Resolved". Gehel claimed this task. Gehel added a comment. We've instead replace some of the tests with WIremock and are trying to remove the dependency on Wikidata from the tests. TASK DETAIL https://phabricator.wikimedia.org/T193466 EMAIL PREFERENC

[Wikidata-bugs] [Maniphest] [Declined] T200931: Consider support for EventStreams as WDQS changes source

2020-05-06 Thread Gehel
Gehel closed this task as "Declined". Gehel added a comment. We're working instead on the new streaming updater, which mostly replace the need for consuming eventstreams. TASK DETAIL https://phabricator.wikimedia.org/T200931 EMAIL PREFERENCES https://phabricator.wikimedia.or

[Wikidata-bugs] [Maniphest] [Declined] T213191: Some queries causes wdqs-blazegraph on wdqs1006 to crash and restart

2020-05-06 Thread Gehel
Gehel closed this task as "Declined". Gehel added a comment. Restricted Application removed a subscriber: Liuxinyu970226. This is probably a duplicate of T242453 <https://phabricator.wikimedia.org/T242453>. There isn't enough context here to investigate more. TAS

[Wikidata-bugs] [Maniphest] [Commented On] T244341: Stop using blank nodes for encoding SomeValue and OWL constraints in WDQS

2020-05-06 Thread Gehel
Gehel added a comment. In T244341#6098237 <https://phabricator.wikimedia.org/T244341#6098237>, @Pfps wrote: > My view is that fewer breaking changes are to be preferred, and breaking changes in fewer "products" is to be even more preferred. So, again, I wonder why

[Wikidata-bugs] [Maniphest] [Assigned] T237089: Create CQS puppet configs by applying query_service module

2020-05-04 Thread Gehel
Gehel assigned this task to EBernhardson. TASK DETAIL https://phabricator.wikimedia.org/T237089 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson, Gehel Cc: Aklapper, Igorkim78, Gehel, Liuxinyu970226, Mathew.onipe, CBogen, darthmon_wmde

[Wikidata-bugs] [Maniphest] [Assigned] T251489: Validate that we have enough resources on WMCS for a SPARQL Endpoint for Commons

2020-05-04 Thread Gehel
Gehel assigned this task to EBernhardson. TASK DETAIL https://phabricator.wikimedia.org/T251489 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson, Gehel Cc: Gehel, Aklapper, CBogen, darthmon_wmde, Nandana, Lahi, Gq86

[Wikidata-bugs] [Maniphest] [Edited] T251498: Access restriction for SPARQL Endpoint for Commons

2020-04-30 Thread Gehel
Gehel updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T251498 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Aklapper, Gehel, darthmon_wmde, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic

[Wikidata-bugs] [Maniphest] [Commented On] T230588: Wikidata Query Service is swapping items and properties

2020-04-30 Thread Gehel
Gehel added a comment. wdqs1010 is one of our test / admin / maintenance server, but not part of any public server pool. Now that the data is up to date on wdqs1010, we need to replicate it to all servers. We have a new team member starting soon, and that's going to be his first task. So

[Wikidata-bugs] [Maniphest] [Created] T251515: Automate data reload for SPARQL Endpoint for Commons

2020-04-30 Thread Gehel
Gehel created this task. Gehel added projects: Wikidata-Query-Service, Wikidata. TASK DESCRIPTION We can regularly reload data from dumps while waiting for the streaming updates to be ready. This can probably be achieved with a simple bash script and a cron job. Blazegraph needs to be shut

[Wikidata-bugs] [Maniphest] [Created] T251514: UI for SPARQL Endpoint for Commons

2020-04-30 Thread Gehel
Gehel created this task. Gehel added projects: Wikidata-Query-Service, Wikidata. TASK DESCRIPTION It is unclear what changes need to be made to the UI so that it works in the context of SDoC. Some investigation is needed to clarify. TASK DETAIL https://phabricator.wikimedia.org/T251514

[Wikidata-bugs] [Maniphest] [Created] T251500: oAuth authentication for SPARQL Endpoint for Commons

2020-04-30 Thread Gehel
Gehel created this task. Gehel added projects: Wikidata-Query-Service, Wikidata. TASK DESCRIPTION oAuth would allow anyone with an existing account to use SEfC Acceptance Criteria: - SEfC is oAuth protected - anyone with an account can access it TASK DETAIL https

[Wikidata-bugs] [Maniphest] [Created] T251498: Access restriction for SPARQL Endpoint for Commons

2020-04-30 Thread Gehel
Gehel created this task. Gehel added projects: Wikidata-Query-Service, Wikidata. TASK DESCRIPTION TASK DETAIL https://phabricator.wikimedia.org/T251498 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Aklapper, Gehel, darthmon_wmde

[Wikidata-bugs] [Maniphest] [Created] T251499: Minimal authentication for SPARQL Endpoint for Commons

2020-04-30 Thread Gehel
Gehel created this task. Gehel added projects: Wikidata-Query-Service, Wikidata. TASK DESCRIPTION The minimal authentication is to just have a static password. This would help validate the principle, but is obviously not a long term solution. Acceptance Criteria: - SEfC is protected

[Wikidata-bugs] [Maniphest] [Updated] T243292: Fix the munger to support commons RDF dump

2020-04-30 Thread Gehel
Gehel added a parent task: T251497: Adapt munging process for SDoC. TASK DETAIL https://phabricator.wikimedia.org/T243292 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Mahir256, Physikerwelt, ArielGlenn, Aklapper, dcausse, darthmon_wmde

[Wikidata-bugs] [Maniphest] [Updated] T251497: Adapt munging process for SDoC

2020-04-30 Thread Gehel
Gehel added a subtask: T243292: Fix the munger to support commons RDF dump. TASK DETAIL https://phabricator.wikimedia.org/T251497 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Aklapper, Gehel, darthmon_wmde, Nandana, Lahi, Gq86

[Wikidata-bugs] [Maniphest] [Created] T251496: Validate and fix TTL dumps of SDoC

2020-04-30 Thread Gehel
Gehel created this task. Gehel added projects: Wikidata-Query-Service, Wikidata. TASK DESCRIPTION Dumps have been created as part of T221917 <https://phabricator.wikimedia.org/T221917> but have never been used. They should be reviewed and fixed if needed. Acceptance Cr

[Wikidata-bugs] [Maniphest] [Created] T251497: Adapt munging process for SDoC

2020-04-30 Thread Gehel
Gehel created this task. Gehel added projects: Wikidata-Query-Service, Wikidata. TASK DESCRIPTION SDoC TTL dumps are different enough from the Wikidata dumps that we need to adapt the process. The exact adaptation needed need to be discovered. Acceptance criteria: - dumps are munged

[Wikidata-bugs] [Maniphest] [Created] T251491: Enable federation between SPARQL Endpoint for Commons and WDQS

2020-04-30 Thread Gehel
Gehel created this task. Gehel added projects: Wikidata-Query-Service, Wikidata. TASK DESCRIPTION The direction of federation needs to be decided. Acceptance Criteria: - Federated queries between SEfC and WDQS can be executed correctly TASK DETAIL https://phabricator.wikimedia.org

[Wikidata-bugs] [Maniphest] [Created] T251490: Load data into the SPARQL Endpoint for Commons

2020-04-30 Thread Gehel
Gehel created this task. Gehel added projects: Wikidata-Query-Service, Wikidata. TASK DESCRIPTION Acceptance Criteria: - full Commons TTL dump is loaded into SEfC - data can be queried appropriately TASK DETAIL https://phabricator.wikimedia.org/T251490 EMAIL PREFERENCES https

[Wikidata-bugs] [Maniphest] [Created] T251489: Validate that we have enough resources on WMCS for a SPARQL Endpoint for Commons

2020-04-30 Thread Gehel
Gehel created this task. Gehel added projects: Wikidata-Query-Service, Wikidata. Restricted Application added a subscriber: Aklapper. TASK DESCRIPTION With some estimates of the current dump data size, ensure that we have enough resources in WMCS for a new SPARQL Endpoint for Commons. Note

[Wikidata-bugs] [Maniphest] [Updated] T251488: Create minimal SPARQL Endpoint for Commons on WMCS

2020-04-30 Thread Gehel
Gehel added a subtask: T237089: Create CQS puppet configs by applying query_service module. TASK DETAIL https://phabricator.wikimedia.org/T251488 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Aklapper, Gehel, darthmon_wmde, Nandana, Lahi

[Wikidata-bugs] [Maniphest] [Created] T251488: Create minimal SPARQL Endpoint for Commons on WMCS

2020-04-30 Thread Gehel
Gehel created this task. Gehel added a project: Wikidata-Query-Service. Restricted Application added a subscriber: Aklapper. Restricted Application added a project: Wikidata. TASK DESCRIPTION This minimal endpoint has the following constraints: - running on WMCS - data loaded manually

[Wikidata-bugs] [Maniphest] [Updated] T237089: Create CQS puppet configs by applying query_service module

2020-04-30 Thread Gehel
Gehel added a parent task: T251488: Create minimal SPARQL Endpoint for Commons on WMCS. TASK DETAIL https://phabricator.wikimedia.org/T237089 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Aklapper, Igorkim78, Gehel, Liuxinyu970226

[Wikidata-bugs] [Maniphest] [Updated] T237089: Create CQS puppet configs by applying query_service module

2020-04-30 Thread Gehel
Gehel removed a parent task: T232297: Refactor Puppet WDQS module to make it usable for wdqs and cqs. TASK DETAIL https://phabricator.wikimedia.org/T237089 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Aklapper, Igorkim78, Gehel

[Wikidata-bugs] [Maniphest] [Updated] T232297: Refactor Puppet WDQS module to make it usable for wdqs and cqs

2020-04-30 Thread Gehel
Gehel removed a subtask: T237089: Create CQS puppet configs by applying query_service module. TASK DETAIL https://phabricator.wikimedia.org/T232297 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mathew.onipe, Gehel Cc: Liuxinyu970226, Gehel

[Wikidata-bugs] [Maniphest] [Retitled] T141602: [Objective Fiscal 19-20/Q4] (9) Provide a Proof of Concept SPARQL endpoint in support of SDoC project

2020-04-30 Thread Gehel
Gehel renamed this task from "[Objective Fiscal 19-20/Q2] (9) Provide a Proof of Concept SPARQL endpoint in support of SDoC project (stretch)" to "[Objective Fiscal 19-20/Q4] (9) Provide a Proof of Concept SPARQL endpoint in support of SDoC project".

[Wikidata-bugs] [Maniphest] [Declined] T217925: Keep global "last seen revision" map for Updater

2020-04-29 Thread Gehel
Gehel closed this task as "Declined". Gehel added a comment. Will be superseeded by the new streaming updater TASK DETAIL https://phabricator.wikimedia.org/T217925 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Lucas_Werkme

[Wikidata-bugs] [Maniphest] [Closed] T222404: Tasks requiring Blazegraph reload

2020-04-29 Thread Gehel
Gehel closed this task as "Resolved". Gehel claimed this task. Gehel added a comment. After discussion, we'll close this and track specific reloads. TASK DETAIL https://phabricator.wikimedia.org/T222404 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailp

[Wikidata-bugs] [Maniphest] [Merged] T225245: Beta endpoint for Wikidata SPARQL

2020-04-29 Thread Gehel
Gehel merged a task: T230760: Create a service to query result sets of Quarry. Gehel added a subscriber: Bugreporter. TASK DETAIL https://phabricator.wikimedia.org/T225245 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Bugreporter

[Wikidata-bugs] [Maniphest] [Updated] T230760: Create a service to query result sets of Quarry

2020-04-29 Thread Gehel
Gehel closed this task as a duplicate of T225245: Beta endpoint for Wikidata SPARQL. TASK DETAIL https://phabricator.wikimedia.org/T230760 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Aklapper, Bugreporter, darthmon_wmde, Nandana, Lahi

[Wikidata-bugs] [Maniphest] [Declined] T229329: WDQS Updater: java.lang.StringIndexOutOfBoundsException: String index out of range: -8

2020-04-29 Thread Gehel
Gehel closed this task as "Declined". Gehel added a comment. seems related to specific tests, we can't see this happening anymore, we'll reopen if needed. TASK DETAIL https://phabricator.wikimedia.org/T229329 EMAIL PREFERENCES https://phabricator.wikimedia.org/sett

[Wikidata-bugs] [Maniphest] [Commented On] T249540: WDQS Docker Image 0.3.10 fails to load RDF dump: BTree Exception after allocation error (Record exists)

2020-04-29 Thread Gehel
Gehel added a comment. The properties used in production are in puppet <https://github.com/wikimedia/puppet/blob/production/modules/query_service/templates/RWStore.properties.erb>. There have been some work to fix some issues in Blazegraph, but not sure what applies. Updating to th

[Wikidata-bugs] [Maniphest] [Declined] T250556: Investigate if "wdqs-heavy-queries" should point to cluster: /wdqs-internal

2020-04-29 Thread Gehel
Gehel closed this task as "Declined". Gehel added a comment. The goal of the internal cluster is to serve synchronous traffic that needs low latency. So we definitely don't want heavy queries to run there! We might discuss spinning up a dedicated cluster for heavy queries, but in

[Wikidata-bugs] [Maniphest] [Commented On] T251356: Can i execute a Wikidata query service from my iframe code at my web page?

2020-04-29 Thread Gehel
Gehel added a comment. It's not very clear what the issue is without more context. There is some minimal documentation on wiki: https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service/Wikidata_Query_Help/Result_Views#Embed_Mode To help you more, we're going to need error messages

[Wikidata-bugs] [Maniphest] [Updated] T249260: SUPPORT: wikibase update from 1.33 to 1.34 error message elastic search

2020-04-27 Thread Gehel
Gehel removed a project: Discovery-Search (Current work). TASK DETAIL https://phabricator.wikimedia.org/T249260 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: DD063520, Gehel Cc: Addshore, dcausse, Aklapper, DD063520, Samantha_Alipio_WMDE, Iflorez

[Wikidata-bugs] [Maniphest] [Updated] T212933: Optimize SERVICE wikibase:label

2020-04-20 Thread Gehel
Gehel removed a project: Discovery-Search (Current work). TASK DETAIL https://phabricator.wikimedia.org/T212933 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Manu1400, Lucas_Werkmeister_WMDE, abian, Aklapper, darthmon_wmde, Nandana, Lahi

[Wikidata-bugs] [Maniphest] [Unassigned] T237089: Create CQS puppet configs by applying query_service module

2020-04-20 Thread Gehel
Gehel removed Mathew.onipe as the assignee of this task. Gehel edited projects, added Discovery-Search; removed Discovery-Search (Current work). TASK DETAIL https://phabricator.wikimedia.org/T237089 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences

[Wikidata-bugs] [Maniphest] [Updated] T243603: Create a way to deploy WDQS artifacts to Archiva with Jenkins

2020-04-20 Thread Gehel
Gehel added a parent task: T244590: EPIC: Rework the WDQS updater as an event driven application. TASK DETAIL https://phabricator.wikimedia.org/T243603 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko, Gehel Cc: Aklapper, Zbyszko, CBogen

[Wikidata-bugs] [Maniphest] [Updated] T244590: EPIC: Rework the WDQS updater as an event driven application

2020-04-20 Thread Gehel
Gehel added a subtask: T248451: [WDQS Streaming Updater] Custom parallelism configuration for Streaming Updater. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: revi, Mholloway

[Wikidata-bugs] [Maniphest] [Updated] T248450: [WDQS Streaming Updater] Monitor Streaming Updater - metrics

2020-04-20 Thread Gehel
Gehel added a parent task: T244590: EPIC: Rework the WDQS updater as an event driven application. TASK DETAIL https://phabricator.wikimedia.org/T248450 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko, Gehel Cc: Aklapper, Zbyszko, Blissjay007

[Wikidata-bugs] [Maniphest] [Updated] T244590: EPIC: Rework the WDQS updater as an event driven application

2020-04-20 Thread Gehel
Gehel added a subtask: T243603: Create a way to deploy WDQS artifacts to Archiva with Jenkins. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: revi, Mholloway, Ladsgroup, Multichill

[Wikidata-bugs] [Maniphest] [Updated] T248451: [WDQS Streaming Updater] Custom parallelism configuration for Streaming Updater

2020-04-20 Thread Gehel
Gehel added a parent task: T244590: EPIC: Rework the WDQS updater as an event driven application. TASK DETAIL https://phabricator.wikimedia.org/T248451 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Aklapper, Zbyszko, darthmon_wmde

[Wikidata-bugs] [Maniphest] [Updated] T249500: [WDQS Streaming Updater] Reuse WikibaseRepository

2020-04-20 Thread Gehel
Gehel added a parent task: T244590: EPIC: Rework the WDQS updater as an event driven application. TASK DETAIL https://phabricator.wikimedia.org/T249500 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko, Gehel Cc: Aklapper, Zbyszko, Blissjay007

[Wikidata-bugs] [Maniphest] [Updated] T244590: EPIC: Rework the WDQS updater as an event driven application

2020-04-20 Thread Gehel
Gehel added a subtask: T248450: [WDQS Streaming Updater] Monitor Streaming Updater - metrics . TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: revi, Mholloway, Ladsgroup, Multichill

[Wikidata-bugs] [Maniphest] [Updated] T249097: [WDQS Streaming Updater] Fix pipeline checkpointing

2020-04-20 Thread Gehel
Gehel added a parent task: T244590: EPIC: Rework the WDQS updater as an event driven application. TASK DETAIL https://phabricator.wikimedia.org/T249097 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Aklapper, Zbyszko, darthmon_wmde

[Wikidata-bugs] [Maniphest] [Updated] T249099: [WDQS Streaming Updater] Error during munging process

2020-04-20 Thread Gehel
Gehel added a parent task: T244590: EPIC: Rework the WDQS updater as an event driven application. TASK DETAIL https://phabricator.wikimedia.org/T249099 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko, Gehel Cc: dcausse, Aklapper, Zbyszko

[Wikidata-bugs] [Maniphest] [Updated] T244590: EPIC: Rework the WDQS updater as an event driven application

2020-04-20 Thread Gehel
Gehel added a subtask: T249500: [WDQS Streaming Updater] Reuse WikibaseRepository. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: revi, Mholloway, Ladsgroup, Multichill, darthmon_wmde

[Wikidata-bugs] [Maniphest] [Updated] T244590: EPIC: Rework the WDQS updater as an event driven application

2020-04-20 Thread Gehel
Gehel added a subtask: T249097: [WDQS Streaming Updater] Fix pipeline checkpointing. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: revi, Mholloway, Ladsgroup, Multichill

[Wikidata-bugs] [Maniphest] [Updated] T248452: [WDQS Streaming Updater] Deploy and configure Streaming Updater Hadoop YARN

2020-04-20 Thread Gehel
Gehel added a parent task: T244590: EPIC: Rework the WDQS updater as an event driven application. TASK DETAIL https://phabricator.wikimedia.org/T248452 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Aklapper, Zbyszko, darthmon_wmde

[Wikidata-bugs] [Maniphest] [Updated] T244590: EPIC: Rework the WDQS updater as an event driven application

2020-04-20 Thread Gehel
Gehel added a subtask: T248452: [WDQS Streaming Updater] Deploy and configure Streaming Updater Hadoop YARN. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: revi, Mholloway, Ladsgroup

[Wikidata-bugs] [Maniphest] [Updated] T244590: EPIC: Rework the WDQS updater as an event driven application

2020-04-20 Thread Gehel
Gehel added a subtask: T249099: [WDQS Streaming Updater] Error during munging process. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: revi, Mholloway, Ladsgroup, Multichill

[Wikidata-bugs] [Maniphest] [Updated] T248449: [WDQS Streaming Updater] Add error handling for Streaming Updater

2020-04-20 Thread Gehel
Gehel added a parent task: T244590: EPIC: Rework the WDQS updater as an event driven application. TASK DETAIL https://phabricator.wikimedia.org/T248449 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Aklapper, Zbyszko, darthmon_wmde

[Wikidata-bugs] [Maniphest] [Updated] T244590: EPIC: Rework the WDQS updater as an event driven application

2020-04-20 Thread Gehel
Gehel added a subtask: T248449: [WDQS Streaming Updater] Add error handling for Streaming Updater. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: revi, Mholloway, Ladsgroup, Multichill

[Wikidata-bugs] [Maniphest] [Updated] T244590: EPIC: Rework the WDQS updater as an event driven application

2020-04-20 Thread Gehel
Gehel added a subtask: T248464: [WDQS Streaming Updater] Implement ouput format in Streaming Updater. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: revi, Mholloway, Ladsgroup

[Wikidata-bugs] [Maniphest] [Updated] T248464: [WDQS Streaming Updater] Implement ouput format in Streaming Updater

2020-04-20 Thread Gehel
Gehel added a parent task: T244590: EPIC: Rework the WDQS updater as an event driven application. TASK DETAIL https://phabricator.wikimedia.org/T248464 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse, Gehel Cc: Aklapper, Zbyszko, CBogen

[Wikidata-bugs] [Maniphest] [Updated] T241128: EPIC: Reduce the time needed to do the initial WDQS import

2020-04-20 Thread Gehel
Gehel removed a subtask: Unknown Object (Task). TASK DETAIL https://phabricator.wikimedia.org/T241128 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: dcausse, Aklapper, darthmon_wmde, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE

[Wikidata-bugs] [Maniphest] [Commented On] T248308: Analyse a small sample of the most often used query patterns on WDQS

2020-04-16 Thread Gehel
Gehel added a comment. A few additional notes: - There is probably better / more useful information published as part of the new events <https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/event-schemas/+/master/jsonschema/sparql/query/1.0.0.yaml> published directly fro

[Wikidata-bugs] [Maniphest] [Declined] T236663: Create a parallel loader to improve load performance for WDQS / Blazegraph

2020-04-15 Thread Gehel
Gehel closed this task as "Declined". Gehel added a comment. That's going to be part of the upcoming hadoop pipeline TASK DETAIL https://phabricator.wikimedia.org/T236663 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc

[Wikidata-bugs] [Maniphest] [Closed] T237844: Wikidata Query Service grafana board is not tracking current data for several metrics

2020-04-15 Thread Gehel
Gehel closed this task as "Resolved". Gehel claimed this task. Gehel added a comment. Looks like all metrics are now showing. Please reopen if needed (and let us know which metrics were missing :) TASK DETAIL https://phabricator.wikimedia.org/T237844 EMAIL PREFERENC

[Wikidata-bugs] [Maniphest] [Declined] T238002: WDQS Munger should be multi threaded

2020-04-15 Thread Gehel
Gehel closed this task as "Declined". Gehel added a comment. The real solution will come from moving all this processing in hadoop as part of the new streaming updater TASK DETAIL https://phabricator.wikimedia.org/T238002 EMAIL PREFERENCES https://phabricator.wikimedia.or

[Wikidata-bugs] [Maniphest] [Declined] T238013: Improve unit test branch coverage on 1 or 2 classes in WDQS

2020-04-15 Thread Gehel
Gehel closed this task as "Declined". Gehel added a comment. Maryum's onboarding is completed. Further improvements will be done as part of regular operations. TASK DETAIL https://phabricator.wikimedia.org/T238013 EMAIL PREFERENCES https://phabricator.wikimedia.org/sett

[Wikidata-bugs] [Maniphest] [Closed] T238153: puppet breakage in the wikidata-query VPS project

2020-04-15 Thread Gehel
Gehel closed this task as "Resolved". Gehel claimed this task. Gehel added a comment. Puppet is now running correctly TASK DETAIL https://phabricator.wikimedia.org/T238153 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc:

[Wikidata-bugs] [Maniphest] [Declined] T238362: Blazegraph write performance tuning

2020-04-15 Thread Gehel
Gehel closed this task as "Declined". Gehel added a comment. Some investigation and tuning was done by Igor already. Our current higher level understanding is that we don't have a write throughput issue, but a combination of read / writes due to the naive updater implementat

[Wikidata-bugs] [Maniphest] [Unassigned] T105427: Need a way for WDQS updater to become aware of suppressed deletes

2020-04-15 Thread Gehel
Gehel removed Zbyszko as the assignee of this task. Gehel added a subscriber: Zbyszko. TASK DETAIL https://phabricator.wikimedia.org/T105427 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Zbyszko, revi, dcausse, Bugreporter, Sjoerddebruin

[Wikidata-bugs] [Maniphest] [Unblock] T231411: Test new Updater service

2020-04-15 Thread Gehel
Gehel closed subtask T238555: Create endpoint to extract low level data for a list of entity IDs. as Declined. TASK DETAIL https://phabricator.wikimedia.org/T231411 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Igorkim78, Gehel Cc: Zbyszko

[Wikidata-bugs] [Maniphest] [Declined] T238555: Create endpoint to extract low level data for a list of entity IDs.

2020-04-15 Thread Gehel
Gehel closed this task as "Declined". Gehel added a comment. We're moving away from the merging updater in favor for a streaming updater. TASK DETAIL https://phabricator.wikimedia.org/T238555 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailp

[Wikidata-bugs] [Maniphest] [Unblock] T231411: Test new Updater service

2020-04-15 Thread Gehel
Gehel closed subtask T238557: Allow for logging recently updated entities as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T231411 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Igorkim78, Gehel Cc: Zbyszko, Lea_Lacroix_WMDE, Gehel

[Wikidata-bugs] [Maniphest] [Closed] T238557: Allow for logging recently updated entities

2020-04-15 Thread Gehel
Gehel closed this task as "Resolved". Gehel claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T238557 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Daniel_Mietchen, Mathew.onipe, Gehel, dcausse, Igorkim78

[Wikidata-bugs] [Maniphest] [Closed] T239414: Investigate how blank nodes are used and synced between wikibase and wdqs

2020-04-15 Thread Gehel
Gehel closed this task as "Resolved". Gehel claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T239414 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Smalyshev, Lucas_Werkmeister_WMDE, Igorkim78, dcausse

[Wikidata-bugs] [Maniphest] [Closed] T241536: Remove the use of chronology_id in wdqs-updater

2020-04-15 Thread Gehel
Gehel closed this task as "Resolved". TASK DETAIL https://phabricator.wikimedia.org/T241536 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse, Gehel Cc: DannyS712, Addshore, sbassett, Zbyszko, dcausse, Gehel, Aklapper, CBogen, dar

[Wikidata-bugs] [Maniphest] [Updated] T249701: maps/wdqs: traffic to maps2004 dropped by iptables

2020-04-08 Thread Gehel
Gehel added a project: Discovery-Search (Current work). Gehel added a comment. This is temporary during data reload on maps master (T249086 <https://phabricator.wikimedia.org/T249086>). Note that looking at logs, I only found dropped packets from maps200[1-3], not from wdqs2001. TASK

[Wikidata-bugs] [Maniphest] [Updated] T244341: Wikibase RDF dump: stop using blank nodes for encoding SomeValue and OWL constraints

2020-04-08 Thread Gehel
Gehel added a project: Discovery-Search (Current work). TASK DETAIL https://phabricator.wikimedia.org/T244341 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Luitzen, VladimirAlexiev, Lea_Lacroix_WMDE, Jheald, Daniel_Mietchen, mkroetzsch

[Wikidata-bugs] [Maniphest] [Updated] T244590: EPIC: Rework the WDQS updater as an event driven application

2020-04-08 Thread Gehel
Gehel added a project: Discovery-Search (Current work). TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Ladsgroup, Multichill, darthmon_wmde, Iamamz3, Smalyshev, Ottomata, JAllemandou

[Wikidata-bugs] [Maniphest] [Changed Project Column] T246497: WDQS Categories update lag alert

2020-04-08 Thread Gehel
Gehel moved this task from In Progress to To Be Deployed on the Discovery-Search (Current work) board. Gehel added a comment. We are fixing an issue in blazegraph that failed categories update in the next deployment. This should be fixed after the deployment is completed. TASK DETAIL

[Wikidata-bugs] [Maniphest] [Updated] T246497: WDQS Categories update lag alert

2020-04-08 Thread Gehel
Gehel added a project: Discovery-Search (Current work). TASK DETAIL https://phabricator.wikimedia.org/T246497 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: dcausse, ayounsi, Aklapper, CBogen, darthmon_wmde, Legado_Shulgin, Nandana

[Wikidata-bugs] [Maniphest] [Updated] T246343: Service implementation on wdqs200[7-8].codfw.wmnet

2020-04-08 Thread Gehel
Gehel edited projects, added Discovery-Search (Current work); removed Discovery-Search. TASK DETAIL https://phabricator.wikimedia.org/T246343 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: elukey, Aklapper, Gehel, CBogen, darthmon_wmde

[Wikidata-bugs] [Maniphest] [Commented On] T240831: Sometimes Q77104211 shows up in sparql query as having a wrong value for P2397, however item is deleted since December 3rd 2019

2020-04-02 Thread Gehel
Gehel added a comment. Full reimport is in progress. By the look of it <https://grafana.wikimedia.org/d/00489/wikidata-query-service?panelId=8=1=now-12h=now=10s_name=wdqs-test>, it will still take around a week to catch up on lag on wdqs1010 before we can copy the data over to

[Wikidata-bugs] [Maniphest] [Commented On] T230588: Wikidata Query Service is swapping items and properties

2020-04-02 Thread Gehel
Gehel added a comment. Full reimport is in progress. By the look of it <https://grafana.wikimedia.org/d/00489/wikidata-query-service?panelId=8=1=now-12h=now=10s_name=wdqs-test>, it will still take around a week to catch up on lag on wdqs1010 before we can copy the data over to

[Wikidata-bugs] [Maniphest] [Commented On] T249041: Updated URL for the WikiPathways SPARQL endpoint

2020-04-01 Thread Gehel
Gehel added a comment. This is scheduled for deployment with the next deploy window on Monday April 6. TASK DETAIL https://phabricator.wikimedia.org/T249041 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RhinosF1, Gehel Cc: Gehel, RhinosF1

[Wikidata-bugs] [Maniphest] [Commented On] T240831: Sometimes Q77104211 shows up in sparql query as having a wrong value for P2397, however item is deleted since December 3rd 2019

2020-03-19 Thread Gehel
Gehel added a comment. A full reimport is in progress and should solve this issue once completed (or if it does not, then we'll need to dig deeper). Reimport has started last week and is likely to run for another week. It then needs to be propagated to all servers... TASK DETAIL https

[Wikidata-bugs] [Maniphest] [Updated] T85101: create index for each dump

2020-03-18 Thread Gehel
Gehel removed projects: Discovery, Wikidata-Query-Service. TASK DETAIL https://phabricator.wikimedia.org/T85101 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Lazhar, Maxlath, Chaotix63, JanZerebecki, Aklapper, hoo, darthmon_wmde, Nandana

[Wikidata-bugs] [Maniphest] [Declined] T92009: Support more fine-grained date fields than xsd:dateTime

2020-03-18 Thread Gehel
Gehel closed this task as "Declined". Gehel added a comment. Closing this as it is unclear if we still need this feature. This looks more like a design question from the start of the project. Feel free to re-open as needed. TASK DETAIL https://phabricator.wikimedia.org/T92

[Wikidata-bugs] [Maniphest] [Declined] T93488: [Task] Determine which dump parts we want in which files

2020-03-18 Thread Gehel
Gehel closed this task as "Declined". Gehel added a comment. Restricted Application removed a subscriber: Liuxinyu970226. Closing this as it is missing way too much information to be actionable 5 years after the last comment TASK DETAIL https://phabricator.wikimedia.org/T93

[Wikidata-bugs] [Maniphest] [Unblock] T50143: Implement complete RDF mapping for entities (tracking)

2020-03-18 Thread Gehel
Gehel closed subtask T93488: [Task] Determine which dump parts we want in which files as Declined. TASK DETAIL https://phabricator.wikimedia.org/T50143 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: PokestarFan, intracer, Aklapper

[Wikidata-bugs] [Maniphest] [Unblock] T131960: "_" character encoded as %20 in Wikidata URI RDF serialization

2020-03-18 Thread Gehel
Gehel closed subtask T132319: Sitelink URIs should be IRIs as Declined. TASK DETAIL https://phabricator.wikimedia.org/T131960 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Smalyshev, Gehel Cc: PokestarFan, Yurik, gerritbot, Smalyshev, JanZerebecki

[Wikidata-bugs] [Maniphest] [Declined] T132319: Sitelink URIs should be IRIs

2020-03-18 Thread Gehel
Gehel closed this task as "Declined". Gehel added a comment. Since there hasn't been any update since 2016 and the problem still does not seem to be fully understood, let's close this. Feel free to reopen and add more context if it is needed. TASK DETAIL https://phabricator.wik

[Wikidata-bugs] [Maniphest] [Updated] T241128: EPIC: Reduce the time needed to do the initial WDQS import

2020-03-18 Thread Gehel
Gehel added a subtask: Unknown Object (Task). TASK DETAIL https://phabricator.wikimedia.org/T241128 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: dcausse, Aklapper, darthmon_wmde, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE

[Wikidata-bugs] [Maniphest] [Commented On] T221921: Provision search endpoint for SDC. Requirements from Product Team.

2020-03-17 Thread Gehel
Gehel added a comment. Sorry the the misunderstanding here. It was never our intention to not provide a SPARQL endpoint for Commons. But given the trouble we have at the moment with WDQS, we are focusing on stabilising our existing services before adding new ones. With the work we are doing

[Wikidata-bugs] [Maniphest] [Commented On] T247058: Deployment strategy and hardware requirement for new Flink based WDQS updater

2020-03-12 Thread Gehel
Gehel added a comment. And we have a first version of a design document <https://docs.google.com/document/d/1H-iaH5Tktye5rIcLic38FkVGqRBZzXwFhYwsL3nXH78/edit#>. This is still work in progress, feel free to comment! TASK DETAIL https://phabricator.wikimedia.org/T247058 EMAIL PREFE

[Wikidata-bugs] [Maniphest] [Edited] T241128: EPIC: Reduce the time needed to do the initial WDQS import

2020-03-11 Thread Gehel
Gehel updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T241128 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: dcausse, Aklapper, darthmon_wmde, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic

[Wikidata-bugs] [Maniphest] [Commented On] T221921: Provision search endpoint for SDC. Requirements from Product Team.

2020-03-11 Thread Gehel
Gehel added a comment. Some of the use cases described here are already supported by search (wbstatement keywords, etc...). We are not going to work on a new SPARQL endpoint before we have a scaling strategy for the current WDQS. It looks like the remaining use cases described here might

[Wikidata-bugs] [Maniphest] [Edited] T247058: Deployment strategy and hardware requirement for new Flink based WDQS updater

2020-03-06 Thread Gehel
Gehel updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T247058 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Joe, Aklapper, dcausse, Zbyszko, Gehel, darthmon_wmde, Legado_Shulgin, Nandana, Davinaclare77

[Wikidata-bugs] [Maniphest] [Created] T247058: Deployment strategy and hardware requirement for new Flink based WDQS updater

2020-03-06 Thread Gehel
Gehel created this task. Gehel added projects: Wikidata-Query-Service, Wikidata, Operations. Restricted Application added a subscriber: Aklapper. TASK DESCRIPTION Our evaluation and proof of concept around Flink is moving forward. We need to start thinking about a deployment strategy

[Wikidata-bugs] [Maniphest] [Unblock] T246343: Service implementation on wdqs200[7-8].codfw.wmnet

2020-03-05 Thread Gehel
Gehel closed subtask T242301: (Need by: TBD) codfw: rack/setup/install wdqs200[7-8].codfw.wmnet as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T246343 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Aklapper, Gehel, CBogen

[Wikidata-bugs] [Maniphest] [Lowered Priority] T170196: Categorymembers: Join on redirect items causes results to explode.

2020-03-04 Thread Gehel
Gehel lowered the priority of this task from "High" to "Low". Gehel added a comment. Changing priority to "Low" to reflect the reality. This has been opened since 2017 and has not moved since. Bug is opened upstream, we are unlikely to solve it on

[Wikidata-bugs] [Maniphest] [Commented On] T193473: Add HTTPS support to wdqs-internal service

2020-03-04 Thread Gehel
Gehel added a comment. Some work has been done to standardize SSL termination around envoy. I'm not sure if that has been applied to WDQS. We need to check, but this might already have been implemented. TASK DETAIL https://phabricator.wikimedia.org/T193473 EMAIL PREFERENCES https

[Wikidata-bugs] [Maniphest] [Declined] T199228: Define an SLO for Wikidata Query Service public endpoint and communicate it

2020-03-04 Thread Gehel
Gehel closed this task as "Declined". Gehel added a comment. We are in the process of significantly changing the architecture of WDQS. We will address a better definition of what the services are supposed to provide as part of redefining those services. There is already a lot more

[Wikidata-bugs] [Maniphest] [Declined] T207665: Run test queries automatically on wdqs autodeployed servers

2020-03-04 Thread Gehel
Gehel closed this task as "Declined". Gehel added a comment. We still have a manual process to build the packaging, so our auto-deploy is not really automated (or not sufficiently). We might want run tests as part of the scap deployment, but that's a different issue. TASK DETA

[Wikidata-bugs] [Maniphest] [Unblock] T209201: WDQS server/updater performance issues

2020-03-02 Thread Gehel
Gehel closed subtask T212826: Create dedicated Updater service in Blazegraph as Declined. TASK DETAIL https://phabricator.wikimedia.org/T209201 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: EgonWillighagen, Daniel_Mietchen

<    13   14   15   16   17   18   19   20   21   22   >