bking added a comment.
Apologies, as I lost track of this ticket.
Repeating the list of steps from the initial message:
1. depool one wcqs2001
2. run the reload script
3. note the date of the dump and position the offset of the wcqs2001 consumer
according to this date
4. start the updater
5. wait for the lag to catch up
6. propagate the journal to other nodes using the data-transfer cookbook
According to the spicerack logs on cumin2022 , I invoked the cookbook as
follows:
./sre/wdqs/data-reload-extended.log:2022-09-20 18:47:32,823 bking 3077967
[DEBUG _cookbook.py:450 in main] Executing cookbook sre.wdqs.data-reload with
args: ['wcqs2001.codfw.wmnet', '--task-id', 'T316236', '--reason', 'reloading
data', '--reuse-download-and-munge', '--depool', '--reload-data=commons',
'--kafka-timestamp=1662854400000']
The final log message appears ~20m later:
./sre/wdqs/data-reload-extended.log:2022-09-20 19:08:23,325 bking 3082992
[INFO clustershell.py:435 in success_nodes] 100.0% (1/1) success ratio (>=
100.0% threshold) for command: 'enable-puppet "r...n2002 - T316236"'.
I don't think this indicates a successful completion. I believe we wanted to
mount via NFS before starting the data reload. Let me check in again on this...
TASK DETAIL
https://phabricator.wikimedia.org/T316236
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: bking
Cc: bking, EBernhardson, HenkvD, Aklapper, dcausse, Jersione, Hellket777,
LisafBia6531, Astuthiodit_1, AWesterinen, 786, Biggs657, karapayneWMDE,
Invadibot, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen,
ItamarWMDE, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD,
Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420,
Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst,
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS,
Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles,
Mbch331
_______________________________________________
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
