[Wikidata-bugs] [Maniphest] T349069: Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2024-06-13T16:11:50Z] START - Cookbook sre.wdqs.data-reload reloading wikidata_full on wdqs2023.codfw.wmnet from DumpsSource.HDFS (hdfs:///wmf/discovery/wdqs-reload-cookbook-test-T349069/ using stat1009.eqiad.wmnet) TASK

[Wikidata-bugs] [Maniphest] T349069: Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2024-06-13T16:18:23Z] END (PASS) - Cookbook sre.wdqs.data-reload (exit_code=0) reloading wikidata_full on wdqs2023.codfw.wmnet from DumpsSource.HDFS (hdfs:///wmf/discovery/wdqs-reload-cookbook-test-T349069/ using stat1009.

[Wikidata-bugs] [Maniphest] T349069: Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers

gerritbot added a comment. Change #1042965 **merged** by Ryan Kemper: [operations/puppet@production] wdqs: remove wdqs2023 from the public cluster and enable the updaters https://gerrit.wikimedia.org/r/1042965 TASK DETAIL https://phabricator.wikimedia.org/T349069 EMAIL PREFERENCE

[Wikidata-bugs] [Maniphest] T349069: Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers

gerritbot added a comment. Change #1042965 had a related patch set uploaded (by DCausse; author: DCausse): [operations/puppet@production] wdqs: remove wdqs2023 from the public cluster and enable the updaters https://gerrit.wikimedia.org/r/1042965 TASK DETAIL https://phabricator.w

[Wikidata-bugs] [Maniphest] T349069: Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2024-06-12T21:11:29Z] END (PASS) - Cookbook sre.wdqs.data-reload (exit_code=0) reloading wikidata_full on wdqs2023.codfw.wmnet from DumpsSource.HDFS (hdfs:///wmf/discovery/wdqs-reload-cookbook-test-T349069/ using stat1009.

[Wikidata-bugs] [Maniphest] T349069: Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2024-06-12T21:05:11Z] START - Cookbook sre.wdqs.data-reload reloading wikidata_full on wdqs2023.codfw.wmnet from DumpsSource.HDFS (hdfs:///wmf/discovery/wdqs-reload-cookbook-test-T349069/ using stat1009.eqiad.wmnet) TASK

[Wikidata-bugs] [Maniphest] T349069: Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2024-06-12T18:04:27Z] END (FAIL) - Cookbook sre.wdqs.data-reload (exit_code=99) reloading wikidata_full on wdqs2023.codfw.wmnet from DumpsSource.HDFS (hdfs:///wmf/discovery/wdqs-reload-cookbook-test-T349069/ using stat1009

[Wikidata-bugs] [Maniphest] T349069: Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2024-06-12T17:58:08Z] START - Cookbook sre.wdqs.data-reload reloading wikidata_full on wdqs2023.codfw.wmnet from DumpsSource.HDFS (hdfs:///wmf/discovery/wdqs-reload-cookbook-test-T349069/ using stat1009.eqiad.wmnet) TASK

[Wikidata-bugs] [Maniphest] T349069: Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2024-06-12T17:56:20Z] END (FAIL) - Cookbook sre.wdqs.data-reload (exit_code=99) reloading wikidata_full on wdqs2023.codfw.wmnet from DumpsSource.HDFS (hdfs:///wmf/discovery/wdqs-reload-cookbook-test-T349069/ using stat1009

[Wikidata-bugs] [Maniphest] T349069: Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2024-06-12T17:49:58Z] START - Cookbook sre.wdqs.data-reload reloading wikidata_full on wdqs2023.codfw.wmnet from DumpsSource.HDFS (hdfs:///wmf/discovery/wdqs-reload-cookbook-test-T349069/ using stat1009.eqiad.wmnet) TASK

[Wikidata-bugs] [Maniphest] T349069: Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2024-06-12T13:33:55Z] END (FAIL) - Cookbook sre.wdqs.data-reload (exit_code=99) reloading wikidata_full on wdqs2023.codfw.wmnet from DumpsSource.HDFS (hdfs:///wmf/discovery/wdqs-reload-cookbook-test-T349069/ using stat1009

[Wikidata-bugs] [Maniphest] T349069: Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2024-06-12T13:30:02Z] START - Cookbook sre.wdqs.data-reload reloading wikidata_full on wdqs2023.codfw.wmnet from DumpsSource.HDFS (hdfs:///wmf/discovery/wdqs-reload-cookbook-test-T349069/ using stat1009.eqiad.wmnet) TASK

[Wikidata-bugs] [Maniphest] T349069: Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2024-06-12T08:28:07Z] END (FAIL) - Cookbook sre.wdqs.data-reload (exit_code=99) reloading wikidata_full on wdqs2023.codfw.wmnet from DumpsSource.HDFS (hdfs:///wmf/discovery/wdqs-reload-cookbook-test-T349069/ using stat1009

[Wikidata-bugs] [Maniphest] T349069: Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2024-06-12T08:24:03Z] START - Cookbook sre.wdqs.data-reload reloading wikidata_full on wdqs2023.codfw.wmnet from DumpsSource.HDFS (hdfs:///wmf/discovery/wdqs-reload-cookbook-test-T349069/ using stat1009.eqiad.wmnet) TASK

[Wikidata-bugs] [Maniphest] T349069: Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2024-06-12T08:15:18Z] END (FAIL) - Cookbook sre.wdqs.data-reload (exit_code=99) reloading wikidata_full on wdqs2023.codfw.wmnet from DumpsSource.HDFS (hdfs:///wmf/discovery/wdqs-reload-cookbook-test-T349069/ using stat1009

[Wikidata-bugs] [Maniphest] T349069: Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2024-06-12T08:14:59Z] START - Cookbook sre.wdqs.data-reload reloading wikidata_full on wdqs2023.codfw.wmnet from DumpsSource.HDFS (hdfs:///wmf/discovery/wdqs-reload-cookbook-test-T349069/ using stat1009.eqiad.wmnet) TASK

[Wikidata-bugs] [Maniphest] T349069: Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2024-06-12T08:12:42Z] END (FAIL) - Cookbook sre.wdqs.data-reload (exit_code=99) reloading wikidata_full on wdqs2023.codfw.wmnet from DumpsSource.HDFS (hdfs:///wmf/discovery/wdqs-reload-cookbook-test-T349069/ using stat1009

[Wikidata-bugs] [Maniphest] T349069: Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2024-06-12T08:12:30Z] START - Cookbook sre.wdqs.data-reload reloading wikidata_full on wdqs2023.codfw.wmnet from DumpsSource.HDFS (hdfs:///wmf/discovery/wdqs-reload-cookbook-test-T349069/ using stat1009.eqiad.wmnet) TASK

[Wikidata-bugs] [Maniphest] T349069: Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2024-06-11T22:56:06Z] END (FAIL) - Cookbook sre.wdqs.data-reload (exit_code=99) reloading wikidata_full on wdqs2023.codfw.wmnet from DumpsSource.HDFS (hdfs:///wmf/discovery/wdqs-reload-cookbook-test-T349069/ using stat1009

[Wikidata-bugs] [Maniphest] T349069: Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2024-06-10T20:30:26Z] END (FAIL) - Cookbook sre.wdqs.data-reload (exit_code=99) reloading wikidata_full on wdqs2023.codfw.wmnet from DumpsSource.HDFS (hdfs:///wmf/discovery/wdqs-reload-cookbook-test-T349069/ using stat1009

[Wikidata-bugs] [Maniphest] T349069: Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2024-06-10T20:30:03Z] START - Cookbook sre.wdqs.data-reload reloading wikidata_full on wdqs2023.codfw.wmnet from DumpsSource.HDFS (hdfs:///wmf/discovery/wdqs-reload-cookbook-test-T349069/ using stat1009.eqiad.wmnet) TASK

[Wikidata-bugs] [Maniphest] T349069: Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2024-06-10T19:22:50Z] END (FAIL) - Cookbook sre.wdqs.data-reload (exit_code=99) reloading wikidata_full on wdqs2023.codfw.wmnet from DumpsSource.HDFS (hdfs:///wmf/discovery/wdqs-reload-cookbook-test-T349069/ using stat1009

[Wikidata-bugs] [Maniphest] T349069: Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2024-06-10T19:02:49Z] START - Cookbook sre.wdqs.data-reload reloading wikidata_full on wdqs2023.codfw.wmnet from DumpsSource.HDFS (hdfs:///wmf/discovery/wdqs-reload-cookbook-test-T349069/ using stat1009.eqiad.wmnet) TASK

[Wikidata-bugs] [Maniphest] T349069: Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2024-06-10T19:02:06Z] END (FAIL) - Cookbook sre.wdqs.data-reload (exit_code=99) reloading wikidata_full on wdqs2023.codfw.wmnet from DumpsSource.HDFS (hdfs:///wmf/discovery/wdqs-reload-cookbook-test-T349069/ using stat1009

[Wikidata-bugs] [Maniphest] T349069: Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2024-06-10T18:11:45Z] START - Cookbook sre.wdqs.data-reload reloading wikidata_full on wdqs2023.codfw.wmnet from DumpsSource.HDFS (hdfs:///wmf/discovery/wdqs-reload-cookbook-test-T349069/ using stat1009.eqiad.wmnet) TASK

[Wikidata-bugs] [Maniphest] T349069: Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers

gerritbot added a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T349069 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse, gerritbot Cc: Daniel_Mietchen, JAllemandou, dr0ptp4kt, bking, BTullis, dcausse, Aklapper, Dann

[Wikidata-bugs] [Maniphest] T349069: Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers

gerritbot added a comment. Change #1038904 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper): [operations/cookbooks@master] wdqs.data-reload: fix regex escaping https://gerrit.wikimedia.org/r/1038904 TASK DETAIL https://phabricator.wikimedia.org/T349069 EMAIL

[Wikidata-bugs] [Maniphest] T349069: Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers

Maintenance_bot removed a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T349069 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse, Maintenance_bot Cc: Daniel_Mietchen, JAllemandou, dr0ptp4kt, bking, BTullis, dcausse, A

[Wikidata-bugs] [Maniphest] T349069: Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers

gerritbot added a comment. Change #1031933 **merged** by jenkins-bot: [operations/cookbooks@master] wdqs.data-reload: support HDFS as a source https://gerrit.wikimedia.org/r/1031933 TASK DETAIL https://phabricator.wikimedia.org/T349069 EMAIL PREFERENCES https://phabricator.wikim

[Wikidata-bugs] [Maniphest] T349069: Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers

CodeReviewBot added a comment. dr0ptp4kt merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/682 search: automate graph split and n3 dump generation TASK DETAIL https://phabricator.wikimedia.org/T349069 EMAIL PREFERENCES https://phabricator.wikim

[Wikidata-bugs] [Maniphest] T349069: Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers

gerritbot added a comment. Change #1031933 had a related patch set uploaded (by DCausse; author: DCausse): [operations/cookbooks@master] wdqs.data-reload: support HDFS as a source https://gerrit.wikimedia.org/r/1031933 TASK DETAIL https://phabricator.wikimedia.org/T349069 EMAIL P

[Wikidata-bugs] [Maniphest] T349069: Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers

gerritbot added a comment. Change #1030897 **merged** by jenkins-bot: [wikidata/query/rdf@master] Allow setting the format of dump files https://gerrit.wikimedia.org/r/1030897 TASK DETAIL https://phabricator.wikimedia.org/T349069 EMAIL PREFERENCES https://phabricator.wikimedia.o

[Wikidata-bugs] [Maniphest] T349069: Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers

gerritbot added a comment. Change #1030897 had a related patch set uploaded (by DCausse; author: DCausse): [wikidata/query/rdf@master] Allow setting the format of dump files https://gerrit.wikimedia.org/r/1030897 TASK DETAIL https://phabricator.wikimedia.org/T349069 EMAIL PREFERE

[Wikidata-bugs] [Maniphest] T349069: Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers

CodeReviewBot added a project: Patch-For-Review. CodeReviewBot added a comment. dcausse opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/682 search: automate graph split and n3 dump generation TASK DETAIL https://phabricator.wikimedia.org/T349069

[Wikidata-bugs] [Maniphest] T349069: Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers

JAllemandou added a comment. No objection :) I'd have gone for option 1 as it seems the easiest to maintain, but I agree, it means installing some stuff to the blazegraph machines. TASK DETAIL https://phabricator.wikimedia.org/T349069 EMAIL PREFERENCES https://phabricator.wikimedia.org/

[Wikidata-bugs] [Maniphest] T349069: Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers

dcausse added a comment. 1. Runs hdfs-rsync directly from the blazegraph hosts - requires installing its dependencies - open a holes between blazegraph and the hadoop cluster 2. Schedule hdfs-rsync on a stat machine copying the ttl dumps from hdfs to `/srv/analytics-search/wikibase_p

[Wikidata-bugs] [Maniphest] T349069: Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers

JAllemandou added a comment. I would suggest using the `hdfs-rsync` tool to do this - it requires some setting up with puppet, but it is helpful, through copying only new stuff from folders (see https://github.com/wikimedia/operations-puppet/blob/1c4d67ff19372832484f7551dc49836be5806024/modu

[Wikidata-bugs] [Maniphest] T349069: Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers

dcausse added a comment. Another approach could be to use the `/mnt/hdfs` mountpoint? I have been told that it might not be stable enough but perhaps it's OK for doing a copy? TASK DETAIL https://phabricator.wikimedia.org/T349069 EMAIL PREFERENCES https://phabricator.wikimedia.org/settin

[Wikidata-bugs] [Maniphest] T349069: Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers

bking added a comment. @dcausse It looks like there's plenty of disk space on `/srv`: bking@stat1007:~$ df -h | grep srv /dev/mapper/stat1007--vg-data 7.2T 4.4T 2.5T 65% /srv I don't own the stats servers though. @BTullis are there any other concerns besides disk space we

[Wikidata-bugs] [Maniphest] T349069: Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers

dcausse added a comment. @BTullis @bking I plan to use a cookbook to transfer some data out of hdfs to blazegraph machines, a naive approach I thought about was to use a temp folder somewhere in `/srv` of a stat100x machine and then re-use the transferpy

[Wikidata-bugs] [Maniphest] T349069: Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers

dcausse updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T349069 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse Cc: Daniel_Mietchen, JAllemandou, dr0ptp4kt, bking, BTullis, dcausse, Aklapper, Danny_Benjafield_WMDE,

[Wikidata-bugs] [Maniphest] T349069: Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers

dcausse claimed this task. dcausse moved this task from Incoming to In Progress on the Discovery-Search (Current work) board. TASK DETAIL https://phabricator.wikimedia.org/T349069 WORKBOARD https://phabricator.wikimedia.org/project/board/1227/ EMAIL PREFERENCES https://phabricator.wikimed

[Wikidata-bugs] [Maniphest] T349069: Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers

dcausse added a project: Discovery-Search (Current work). TASK DETAIL https://phabricator.wikimedia.org/T349069 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse Cc: Daniel_Mietchen, JAllemandou, dr0ptp4kt, bking, BTullis, dcausse, Aklapper, Da

[Wikidata-bugs] [Maniphest] T349069: Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers

Gehel closed subtask T350106: Implement a spark job that converts a RDF triples table into a RDF file format as "Resolved". TASK DETAIL https://phabricator.wikimedia.org/T349069 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: JAllemandou, d

[Wikidata-bugs] [Maniphest] T349069: Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers

Gehel removed a project: Discovery-Search (Current work). Gehel moved this task from Quarterly Goals to Watching on the Data-Platform-SRE board. TASK DETAIL https://phabricator.wikimedia.org/T349069 WORKBOARD https://phabricator.wikimedia.org/project/board/6524/ EMAIL PREFERENCES https://

[Wikidata-bugs] [Maniphest] T349069: Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers

Gehel edited parent tasks, added: T350464: Expose SPARQL endpoints with full wikidata data set and with split graph to enable experimentation on federation with a split graph; removed: T337013: [Epic] Splitting the graph in WDQS. TASK DETAIL https://phabricator.wikimedia.org/T349069 EMAIL PRE

[Wikidata-bugs] [Maniphest] T349069: Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers

Gehel moved this task from Incoming to Current work on the Wikidata-Query-Service board. Gehel added a project: Discovery-Search (Current work). TASK DETAIL https://phabricator.wikimedia.org/T349069 WORKBOARD https://phabricator.wikimedia.org/project/board/891/ EMAIL PREFERENCES https://p

[Wikidata-bugs] [Maniphest] T349069: Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers

Gehel triaged this task as "High" priority. TASK DETAIL https://phabricator.wikimedia.org/T349069 WORKBOARD https://phabricator.wikimedia.org/project/board/6524/ EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: dr0ptp4kt, bking, BTullis, d

[Wikidata-bugs] [Maniphest] T349069: Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers

Gehel added a project: Data-Platform-SRE. TASK DETAIL https://phabricator.wikimedia.org/T349069 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: dr0ptp4kt, bking, BTullis, dcausse, Aklapper, Danny_Benjafield_WMDE, Astuthiodit_1, AWesterinen,

[Wikidata-bugs] [Maniphest] T349069: Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers

dcausse created this task. dcausse added projects: Wikidata, Wikidata-Query-Service. Restricted Application added a subscriber: Aklapper. TASK DESCRIPTION As of today the data-reload cookbook does multiple tasks on the wdqs host being reloaded: - copy the dumps from the snapshot machines t