Hi, We have a cluster and some nodes are down we tried to set them idle using "scontrol update NodeName=xx State=idle" but they get back to down also we tried to troubleshoot them using SLURM Troubleshooting Guide but unfortunately nodes are still not working.
Here are commands outputs Sinfo output is :sinfo: error: slurm_receive_msg: Zero Bytes were transmitted or received slurm_load_partitions: Zero Bytes were transmitted or received. scontrol ping output is: scontrol: error: slurm_receive_msg: Zero Bytes were transmitted or received Slurmctld(primary/backup) at master/(NULL) are DOWN/DOWN ************************************************************** ** RESTORE SLURMCTLD DAEMON TO SERVICE ** ************************************************************** Regrads Fahad Alzannan CTAM KACST Warning: This message and its attachment, if any, are confidential and may contain information protected by law. If you are not the intended recipient, please contact the sender immediately and delete the message and its attachment, if any. You should not copy the message and its attachment, if any, or disclose its contents to any other person or use it for any purpose. Statements and opinions expressed in this e-mail and its attachment, if any, are those of the sender, and do not necessarily reflect those of King Abdulaziz city for Science and Technology (KACST) in the Kingdom of Saudi Arabia. KACST accepts no liability for any damage caused by this email. ?????: ??? ??????? ??? ????? ?? ?????? (?? ????) ???? ????? ???? ?? ????? ??? ??????? ????? ????? ???????. ??? ?? ??? ????? ?????? ???? ??????? ???? ???? ????? ??????? ???? ?????? ????? ???? ??????? ????????? (?? ????)? ??? ???? ?? ??? ?? ????? ??? ??????? ?? ???????? (?? ????) ?? ?? ??? ????? ?? ????? ?????????? ????? ?? ????????? ??? ???. ????? ??? ???? ??? ??????? ????????? (?? ????) ???? ?? ??? ??????? ???? ???????? ??? ????? ????? ????????? ?????? ???????? ???????? ??????? ????????? ??? ????? ??????? ?? ??????? ?? ??????? ??????? ?? ?? ?? ?????? ??? ??????.
