[Wikidata-bugs] [Maniphest] T274270: WDQS servers taking up to 30 minutes to reboot

2023-05-15 Thread bking 
bking edited projects, added Discovery-Search; removed Discovery-Search 
(Current work).

TASK DETAIL
  https://phabricator.wikimedia.org/T274270

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: bking
Cc: bking, RKemper, Gehel, Aklapper, Astuthiodit_1, karapayneWMDE, Invadibot, 
MPhamWMF, maantietaja, ItamarWMDE, Akuckartz, Nandana, Lahi, Gq86, 
GoranSMilovanovic, QZanden, EBjune, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T274270: WDQS servers taking up to 30 minutes to reboot

2023-03-16 Thread Gehel 
Gehel edited projects, added Discovery-Search (Current work); removed 
Wikidata-Query-Service.

TASK DETAIL
  https://phabricator.wikimedia.org/T274270

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Gehel
Cc: bking, RKemper, Gehel, Aklapper, Astuthiodit_1, karapayneWMDE, Invadibot, 
MPhamWMF, maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, Lahi, Gq86, 
GoranSMilovanovic, QZanden, EBjune, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Wikidata-bugs, aude, Mbch331, AWesterinen, Namenlos314, 
Lucas_Werkmeister_WMDE, merbst, Jonas, Xmlizer, jkroll, Jdouglas, Tobias1984, 
Manybubbles
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T274270: WDQS servers taking up to 30 minutes to reboot

2023-01-23 Thread Gehel 
Gehel removed a project: Discovery-Search (Current work).

TASK DETAIL
  https://phabricator.wikimedia.org/T274270

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Gehel
Cc: bking, RKemper, Gehel, Aklapper, Astuthiodit_1, AWesterinen, karapayneWMDE, 
Invadibot, MPhamWMF, maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, 
Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, 
EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T274270: WDQS servers taking up to 30 minutes to reboot

2022-12-20 Thread Gehel 
Gehel added a project: Discovery-Search (Current work).

TASK DETAIL
  https://phabricator.wikimedia.org/T274270

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Gehel
Cc: bking, RKemper, Gehel, Aklapper, Astuthiodit_1, AWesterinen, karapayneWMDE, 
Invadibot, MPhamWMF, maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, 
Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, 
EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T274270: WDQS servers taking up to 30 minutes to reboot

2022-07-25 Thread Gehel 
Gehel removed a project: Discovery-Search.

TASK DETAIL
  https://phabricator.wikimedia.org/T274270

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Gehel
Cc: bking, RKemper, Gehel, Aklapper, Astuthiodit_1, AWesterinen, karapayneWMDE, 
Invadibot, MPhamWMF, maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, 
Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, 
EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T274270: WDQS servers taking up to 30 minutes to reboot

2022-03-31 Thread bking 
bking added a comment.


  Unfortunately, the systemd workaround listed above did **not** work. We will 
try adjusting some other unit file values when time permits.

TASK DETAIL
  https://phabricator.wikimedia.org/T274270

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: bking
Cc: bking, RKemper, Gehel, Aklapper, Astuthiodit_1, karapayneWMDE, Invadibot, 
MPhamWMF, maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, Namenlos314, 
Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T274270: WDQS servers taking up to 30 minutes to reboot

2022-03-31 Thread bking 
bking added a comment.


  Correction: both MDRAID and LVM servers have this problem. Both services' 
systemd unit files have the same "Conflicts=shutdown.target" directive. Still 
haven't tried the systemd workaround though, will test that today.

TASK DETAIL
  https://phabricator.wikimedia.org/T274270

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: bking
Cc: bking, RKemper, Gehel, Aklapper, Astuthiodit_1, karapayneWMDE, Invadibot, 
MPhamWMF, maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, Namenlos314, 
Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T274270: WDQS servers taking up to 30 minutes to reboot

2022-03-30 Thread bking 
bking added a comment.


  Another piece of the puzzle, some wdqs hosts use MDRAID for their /srv 
partition, some use LVM  . Working 
assumption is that only the LVM hosts will take forever to reboot.

TASK DETAIL
  https://phabricator.wikimedia.org/T274270

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: bking
Cc: bking, RKemper, Gehel, Aklapper, Astuthiodit_1, karapayneWMDE, Invadibot, 
MPhamWMF, maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, Namenlos314, 
Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T274270: WDQS servers taking up to 30 minutes to reboot

2022-03-29 Thread bking 
bking added a comment.


  Actions tried so far: disabling swap via systemd before rebooting. Worked on 
`wdqs2007`, did not work on `wdqs2002`. Also worth noting is that we had 
previously rebooted `wdqs2007` within the last 30 minutes, so a minor kernel 
update (from 4.19.0-16-amd64 to 4.19.0-20-amd64) or any other reboot-required 
updates could have fixed the issue. It's also possible the system hadn't been 
up long enough to cause any problems. Compare to `wdqs2002` which has been 
running a production workload and has not been rebooted recently.
  
  We will pick this up again tomorrow, attempting the systemd workaround linked 
in my last comment.

TASK DETAIL
  https://phabricator.wikimedia.org/T274270

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: bking
Cc: bking, RKemper, Gehel, Aklapper, Astuthiodit_1, karapayneWMDE, Invadibot, 
MPhamWMF, maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, Namenlos314, 
Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T274270: WDQS servers taking up to 30 minutes to reboot

2022-03-29 Thread bking 
bking added a comment.


  This is still happening, @RKemper  found some interesting links that could 
explain this behavior:
  
  
https://wiki.freedesktop.org/www/Software/systemd/Debugging/#diagnosingshutdownproblems
  
  
https://old.reddit.com/r/archlinux/comments/ba3zec/very_slow_shutdownreboot_fixed/
  
  https://github.com/systemd/systemd/issues/11821#issuecomment-477545885
  
  I think it's worth trying the systemd workaround mentioned in the last github 
thread.

TASK DETAIL
  https://phabricator.wikimedia.org/T274270

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: bking
Cc: bking, RKemper, Gehel, Aklapper, Astuthiodit_1, karapayneWMDE, Invadibot, 
MPhamWMF, maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, Namenlos314, 
Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T274270: WDQS servers taking up to 30 minutes to reboot

2022-02-16 Thread Gehel 
Gehel edited projects, added Discovery-Search; removed Discovery-Search 
(Current work).

TASK DETAIL
  https://phabricator.wikimedia.org/T274270

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Gehel
Cc: RKemper, Gehel, Aklapper, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, 
CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T274270: WDQS servers taking up to 30 minutes to reboot

2021-03-01 Thread MPhamWMF 
MPhamWMF set the point value for this task to "8".

TASK DETAIL
  https://phabricator.wikimedia.org/T274270

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: MPhamWMF
Cc: RKemper, Gehel, Aklapper, MPhamWMF, CBogen, Akuckartz, Nandana, 
Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, 
EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
abian, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T274270: WDQS servers taking up to 30 minutes to reboot

2021-03-01 Thread Gehel 
Gehel moved this task from Operations to Current work on the 
Wikidata-Query-Service board.
Gehel added a project: Discovery-Search (Current work).

TASK DETAIL
  https://phabricator.wikimedia.org/T274270

WORKBOARD
  https://phabricator.wikimedia.org/project/board/891/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Gehel
Cc: RKemper, Gehel, Aklapper, MPhamWMF, CBogen, Akuckartz, Nandana, 
Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, 
EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
abian, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T274270: WDQS servers taking up to 30 minutes to reboot

2021-02-15 Thread Gehel 
Gehel triaged this task as "High" priority.
Gehel moved this task from All WDQS-related tasks to Operations on the 
Wikidata-Query-Service board.

TASK DETAIL
  https://phabricator.wikimedia.org/T274270

WORKBOARD
  https://phabricator.wikimedia.org/project/board/891/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Gehel
Cc: RKemper, Gehel, Aklapper, MPhamWMF, CBogen, Akuckartz, Nandana, 
Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, 
EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T274270: WDQS servers taking up to 30 minutes to reboot

2021-02-09 Thread Gehel 
Gehel updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T274270

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Gehel
Cc: Gehel, Aklapper, MPhamWMF, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, 
Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T274270: WDQS servers taking up to 30 minutes to reboot

2021-02-09 Thread Gehel 
Gehel created this task.
Gehel added a project: Wikidata-Query-Service.
Restricted Application added a subscriber: Aklapper.
Restricted Application added a project: Wikidata.

TASK DESCRIPTION
  As an administrator of WDQS, I want reboots to be fast so that I can run 
cluster wide operations in a reasonable amount of time.
  
  While doing a full cluster restart of WDQS for kernel upgrade, multiple 
servers took at least 30 minutes to reboot. Looking at console, it looks like 
the shutdown is waiting to unmount disks. Stopping blazegaph (both wdqs and 
categories) before the reboot does not have a significant impact on shutdown 
time.
  
  Maybe related logs (wdqs1007:/var/log/syslog):
  
Feb  9 16:21:50 wdqs1007 blkdeactivate[16486]:   [SKIP]: unmount of 
vg0-swap (dm-1) mounted on [SWAP]
Feb  9 16:21:51 wdqs1007 blkdeactivate[16486]:   [UMOUNT]: unmounting 
vg0-srv (dm-2) mounted on /srv... skipping
Feb  9 16:21:51 wdqs1007 blkdeactivate[16486]:   [SKIP]: unmount of 
vg0-root (dm-0) mounted on /
  
  AC:
  
  - wdqs servers can be rebooted in < 5 minutes

TASK DETAIL
  https://phabricator.wikimedia.org/T274270

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Gehel
Cc: Gehel, Aklapper, MPhamWMF, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, 
Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs