There is no ntp service running on any of my nodes, and all but this one is working. I haven't heard that ntp is a requirement for slurm, just that the time be synchronized across the cluster. And it is.
On Wed, Apr 15, 2020 at 12:17 PM Carlos Fenoy <mini...@gmail.com> wrote: > I’d check ntp as your encoding time seems odd to me > > On Wed, 15 Apr 2020 at 19:59, Dean Schulze <dean.w.schu...@gmail.com> > wrote: > >> I've installed two new nodes onto my slurm cluster. One node works, but >> the other one complains about an invalid credential for munge. I've >> verified that the munge.key is the same as on all other nodes with >> >> sudo cksum /etc/munge/munge.key >> >> I recopied a munge.key from a node that works. I've verified that munge >> uid and gid are the same on the nodes. The time is in sync on all nodes. >> >> Here is what is in the slurmd.log: >> >> error: Unable to register: Unable to contact slurm controller (connect >> failure) >> error: Munge decode failed: Invalid credential >> ENCODED: Wed Dec 31 17:00:00 1969 >> DECODED: Wed Dec 31 17:00:00 1969 >> error: authentication: Invalid authentication credential >> error: slurm_receive_msg_and_forward: Protocol authentication error >> error: service_connection: slurm_receive_msg: Protocol authentication >> error >> error: Unable to register: Unable to contact slurm controller (connect >> failure) >> >> I've checked in the munged.log and all it says is >> >> Invalid credential >> >> Thanks for your help >> > -- > -- > Carles Fenoy >