ssh returns "Permission denied (gssapi-keyex,gssapi-with-mic)."

[Joseph Areeda]

I can't figure out what causes this error.

I can "fix" it by regenerating the server key on the system I'm trying 
to connect to and restarting sshd but that seems to be temporary as the 
same problem comes back in a week or so.  Rebooting the server does not 
fix it.


Does anyone know what that error means?  I am using ssh not gsissh 
although I do have globus toolkit installed to contact grid computers.


I'm pretty sure it's a misconfiguration on my part but I can't figure 
out what I did or didn't do.


Thanks,

Joe

Re: ssh returns "Permission denied (gssapi-keyex,gssapi-with-mic)."

[Steven Timm]

What does the output of
ssh -vv 

give you?

and what does /var/log/secure say on the server side?

Permission denied could be a number of things (time not in sync,
PAM configuration right, or other stuff.  without knowing the
server and client sshd_config and ssh_config respectively it is hard to 
tell.


Steve Timm


On Wed, 21 Nov 2012, Joseph Areeda wrote:


I can't figure out what causes this error.

I can "fix" it by regenerating the server key on the system I'm trying to 
connect to and restarting sshd but that seems to be temporary as the same 
problem comes back in a week or so.  Rebooting the server does not fix it.


Does anyone know what that error means?  I am using ssh not gsissh although I 
do have globus toolkit installed to contact grid computers.


I'm pretty sure it's a misconfiguration on my part but I can't figure out 
what I did or didn't do.


Thanks,

Joe



--
Steven C. Timm, Ph.D  (630) 840-8525
t...@fnal.gov  http://home.fnal.gov/~timm/
Fermilab Computing Division, Scientific Computing Facilities,
Grid Facilities Department, FermiGrid Services Group, Group Leader.
Lead of FermiCloud project.

Re: ssh returns "Permission denied (gssapi-keyex,gssapi-with-mic)."

[Tam Nguyen]

Hi Joe,
Did you look at the sshd_config file?
I ran into a similar error output but it may not necessarily be the same
issue you're having.  In my case, the sshd_conf file on one of my users
machine was edited and renamed.  I backup that file and copy a default
sshd_config file, then test it.

Good luck.
-T

On Wed, Nov 21, 2012 at 5:16 PM, Joseph Areeda  wrote:

> I can't figure out what causes this error.
>
> I can "fix" it by regenerating the server key on the system I'm trying to
> connect to and restarting sshd but that seems to be temporary as the same
> problem comes back in a week or so.  Rebooting the server does not fix it.
>
> Does anyone know what that error means?  I am using ssh not gsissh
> although I do have globus toolkit installed to contact grid computers.
>
> I'm pretty sure it's a misconfiguration on my part but I can't figure out
> what I did or didn't do.
>
> Thanks,
>
> Joe
>

Re: ssh returns "Permission denied (gssapi-keyex,gssapi-with-mic)."

[Joseph Areeda]

Thank you Tam, and Steven,

I just confirmed that regenerating the keys (ssh-keygen -t dsa -f 
ssh_host_dsa_key && ssh -t rsa -f ssh_host_rsa_key) in /etc/ssh "fixes 
the problem"


So ssh -vv shows me how it's supposed to look.  I'll save that and do a 
diff when it happens again.


As I continue my googling I can report on a few things it's not

Server machine has a fixed ip address and dns/rdns appears working.

Time issue Steven mentioned does not seem to be it, although I may stop 
using pool machines and set up a local ntp server so everybody gets the 
same time.  I can ssh and gsissh to other servers.


Server:
ntpq -p
 remote   refid  st t when poll reach   delay   
offset  jitter

==
*ping-audit-207- .ACTS.   1 u5  128  377   19.867
5.804   1.927
+10504.x.rootbsd 198.30.92.2  2 u  129  128  376   45.146  
-28.571   5.558
+ntp.sunflower.c 132.236.56.250   3 u   77  128  355   63.836  
-14.753   5.360
-ntp2.ResComp.Be 128.32.206.553 u  126  128  377   22.112
7.311   2.022


Client:

ntpq -p
 remote   refid  st t when poll reach   delay   
offset  jitter

==
 64.147.116.229  .ACTS.   1 u   47  1280   13.543
0.567   0.000
*nist1-chi.ustim .ACTS.   1 u   25  128  377  106.619   
14.458   5.896
+name3.glorb.com 69.36.224.15 2 u   64  128  377   88.564  
-27.542   3.631
+131.211.8.244   .PPS.1 u   81  128  377  167.107
3.259   2.340




The only setting I change in sshd_config is to turn off password auth 
but this machine is being brought up behind a firewall and I haven't 
done that yet.  Also if it was a config problem I doubt changing the key 
would fix it, even temporarily.


I will report back with the ssh -vv stuff when it happens again.
At least now I have a chance of figuring out what's going on.

Best,
Joe


On 11/21/2012 02:30 PM, Tam Nguyen wrote:

Hi Joe,
Did you look at the sshd_config file?
I ran into a similar error output but it may not necessarily be the 
same issue you're having.  In my case, the sshd_conf file on one of my 
users machine was edited and renamed.  I backup that file and copy a 
default sshd_config file, then test it.


Good luck.
-T

On Wed, Nov 21, 2012 at 5:16 PM, Joseph Areeda > wrote:


I can't figure out what causes this error.

I can "fix" it by regenerating the server key on the system I'm
trying to connect to and restarting sshd but that seems to be
temporary as the same problem comes back in a week or so.
 Rebooting the server does not fix it.

Does anyone know what that error means?  I am using ssh not gsissh
although I do have globus toolkit installed to contact grid computers.

I'm pretty sure it's a misconfiguration on my part but I can't
figure out what I did or didn't do.

Thanks,

Joe

RE: ssh returns "Permission denied (gssapi-keyex,gssapi-with-mic)."

[Steven C Timm]

Shouldn't need to regenerate the keys.. once you get them generated once they 
should be good for the life of the machine.
Save copies of the keys as they are now and if your system goes bad, do 
differences to see what changed, if anything.

Steve Timm


From: owner-scientific-linux-us...@listserv.fnal.gov 
[mailto:owner-scientific-linux-us...@listserv.fnal.gov] On Behalf Of Joseph 
Areeda
Sent: Wednesday, November 21, 2012 5:46 PM
To: owner-scientific-linux-us...@listserv.fnal.gov
Cc: scientific-linux-users
Subject: Re: ssh returns "Permission denied (gssapi-keyex,gssapi-with-mic)."

Thank you Tam, and Steven,

I just confirmed that regenerating the keys (ssh-keygen -t dsa -f 
ssh_host_dsa_key && ssh -t rsa -f ssh_host_rsa_key) in /etc/ssh "fixes the 
problem"

So ssh -vv shows me how it's supposed to look.  I'll save that and do a diff 
when it happens again.

As I continue my googling I can report on a few things it's not

Server machine has a fixed ip address and dns/rdns appears working.

Time issue Steven mentioned does not seem to be it, although I may stop using 
pool machines and set up a local ntp server so everybody gets the same time.  I 
can ssh and gsissh to other servers.

Server:
ntpq -p

 remote   refid  st t when poll reach   delay   offset  jitter
==
*ping-audit-207- .ACTS.   1 u5  128  377   19.8675.804   1.927
+10504.x.rootbsd 198.30.92.2  2 u  129  128  376   45.146  -28.571   5.558
+ntp.sunflower.c 132.236.56.250   3 u   77  128  355   63.836  -14.753   5.360
-ntp2.ResComp.Be 128.32.206.553 u  126  128  377   22.1127.311   2.022

Client:

ntpq -p
 remote   refid  st t when poll reach   delay   offset  jitter
==
 64.147.116.229  .ACTS.   1 u   47  1280   13.5430.567   0.000
*nist1-chi.ustim .ACTS.   1 u   25  128  377  106.619   14.458   5.896
+name3.glorb.com 69.36.224.15 2 u   64  128  377   88.564  -27.542   3.631
+131.211.8.244   .PPS.1 u   81  128  377  167.1073.259   2.340



The only setting I change in sshd_config is to turn off password auth but this 
machine is being brought up behind a firewall and I haven't done that yet.  
Also if it was a config problem I doubt changing the key would fix it, even 
temporarily.

I will report back with the ssh -vv stuff when it happens again.
At least now I have a chance of figuring out what's going on.

Best,
Joe


On 11/21/2012 02:30 PM, Tam Nguyen wrote:
Hi Joe,
Did you look at the sshd_config file?
I ran into a similar error output but it may not necessarily be the same issue 
you're having.  In my case, the sshd_conf file on one of my users machine was 
edited and renamed.  I backup that file and copy a default sshd_config file, 
then test it.

Good luck.
-T
On Wed, Nov 21, 2012 at 5:16 PM, Joseph Areeda 
mailto:newsre...@areeda.com>> wrote:
I can't figure out what causes this error.

I can "fix" it by regenerating the server key on the system I'm trying to 
connect to and restarting sshd but that seems to be temporary as the same 
problem comes back in a week or so.  Rebooting the server does not fix it.

Does anyone know what that error means?  I am using ssh not gsissh although I 
do have globus toolkit installed to contact grid computers.

I'm pretty sure it's a misconfiguration on my part but I can't figure out what 
I did or didn't do.

Thanks,

Joe

RE: ssh returns "Permission denied (gssapi-keyex,gssapi-with-mic)."

[Paul Robert Marino]

On Nov 21, 2012 7:57 PM, "Paul Robert Marino"  wrote:

> Ok
> To be clear are you using kerberos or not
> If the answer is no and you are just using ssh keys the most common cause
> of this issue is that the useres home directory is group or world readable.
> In the most secure mode which is the default if the useres home and or the
> ~/.ssh directory is has a any thing other than 700 or 500 set as the
> permissions it will reject the public key (the one on the server you are
> trying to connect to) this become obvious with -vvv but not -vv
>  On Nov 21, 2012 7:34 PM, "Steven C Timm"  wrote:
>
>>  Shouldn’t need to regenerate the keys.. once you get them generated
>> once they should be good for the life of the machine.
>>
>> Save copies of the keys as they are now and if your system goes bad, do
>> differences to see what changed, if anything.
>>
>> ** **
>>
>> Steve Timm
>>
>> ** **
>>
>> ** **
>>
>> *From:* owner-scientific-linux-us...@listserv.fnal.gov [mailto:
>> owner-scientific-linux-us...@listserv.fnal.gov] *On Behalf Of *Joseph
>> Areeda
>> *Sent:* Wednesday, November 21, 2012 5:46 PM
>> *To:* owner-scientific-linux-us...@listserv.fnal.gov
>> *Cc:* scientific-linux-users
>> *Subject:* Re: ssh returns "Permission denied
>> (gssapi-keyex,gssapi-with-mic)."
>>
>> ** **
>>
>> Thank you Tam, and Steven,
>>
>> I just confirmed that regenerating the keys (ssh-keygen -t dsa -f
>> ssh_host_dsa_key && ssh -t rsa -f ssh_host_rsa_key) in /etc/ssh "fixes the
>> problem"
>>
>> So ssh -vv shows me how it's supposed to look.  I'll save that and do a
>> diff when it happens again.
>>
>> As I continue my googling I can report on a few things it's not
>>
>> Server machine has a fixed ip address and dns/rdns appears working.
>>
>> Time issue Steven mentioned does not seem to be it, although I may stop
>> using pool machines and set up a local ntp server so everybody gets the
>> same time.  I can ssh and gsissh to other servers.
>>
>> Server:
>> ntpq -p
>>
>> 
>>
>>  remote   refid  st t when poll reach   delay   offset
>> jitter
>>
>> ==
>> *ping-audit-207- .ACTS.   1 u5  128  377   19.8675.804
>> 1.927
>> +10504.x.rootbsd 198.30.92.2  2 u  129  128  376   45.146  -28.571
>> 5.558
>> +ntp.sunflower.c 132.236.56.250   3 u   77  128  355   63.836  -14.753
>> 5.360
>> -ntp2.ResComp.Be 128.32.206.553 u  126  128  377   22.1127.311
>> 2.022
>>
>>
>> Client:
>>
>> 
>>
>> ntpq -p
>>  remote   refid  st t when poll reach   delay   offset
>> jitter
>>
>> ==
>>  64.147.116.229  .ACTS.   1 u   47  1280   13.5430.567
>> 0.000
>> *nist1-chi.ustim .ACTS.   1 u   25  128  377  106.619   14.458
>> 5.896
>> +name3.glorb.com 69.36.224.15 2 u   64  128  377   88.564  -27.542
>> 3.631
>> +131.211.8.244   .PPS.1 u   81  128  377  167.1073.259
>> 2.340
>>
>>
>>
>>
>> The only setting I change in sshd_config is to turn off password auth but
>> this machine is being brought up behind a firewall and I haven't done that
>> yet.  Also if it was a config problem I doubt changing the key would fix
>> it, even temporarily.
>>
>> I will report back with the ssh -vv stuff when it happens again.
>> At least now I have a chance of figuring out what's going on.
>>
>> Best,
>> Joe
>>
>>
>> On 11/21/2012 02:30 PM, Tam Nguyen wrote: 
>>
>> Hi Joe, 
>>
>> Did you look at the sshd_config file?  
>>
>> I ran into a similar error output but it may not necessarily be the same
>> issue you're having.  In my case, the sshd_conf file on one of my users
>> machine was edited and renamed.  I backup that file and copy a default
>> sshd_config file, then test it.  
>>
>> ** **
>>
>> Good luck.
>>
>> -T
>>
>> On Wed, Nov 21, 2012 at 5:16 PM, Joseph Areeda 
>> wrote:
>>
>> I can't figure out what causes this error.
>>
>> I can "fix" it by regenerating the server key on the system I'm trying to
>> connect to and restarting sshd but that seems to be temporary as the same
>> problem comes back in a week or so.  Rebooting the server does not fix it.
>>
>> Does anyone know what that error means?  I am using ssh not gsissh
>> although I do have globus toolkit installed to contact grid computers.
>>
>> I'm pretty sure it's a misconfiguration on my part but I can't figure out
>> what I did or didn't do.
>>
>> Thanks,
>>
>> Joe
>>
>> ** **
>>
>

Re: ssh returns "Permission denied (gssapi-keyex,gssapi-with-mic)."

[Joseph Areeda]

Thank you Paul, Steven and Steve,

I think Kerberos may be the issue.  I do NOT use Kerberos to access this 
machine, I have a lot to learn before I turn that and LDAP on.  But I do 
use it to access several services in our collaboration so the client 
machine often has a valid Kerberos TGT (and probably more often an 
expired ticket).  I think it's worth experimenting with the client in 
different states of Kerberosity (or whatever that word should be).


The user's directory is 755 which is the convention for grid computers 
in our collaboration and the plan is for this machine to be on our soon 
to be delivered cluster.  The .ssh directory is 700.  This doesn't 
change between the working and non-working state.


I tarred the /etc/ssh directory and saved it for next time but wouldn't 
generating new keys make them almost completely different?  Generating 
new keys makes no sense to me either, but it does work.  Well, at least 
it has been the only thing I've done coincident with resolving the 
problem the last 3 times this has happened.


I also save the triple verbose ssh output.

I really appreciate the discussion gentlemen, it helps a lot.

Best,
Joe

On 11/21/2012 04:58 PM, Paul Robert Marino wrote:
On Nov 21, 2012 7:57 PM, "Paul Robert Marino" <mailto:prmari...@gmail.com>> wrote:


Ok
To be clear are you using kerberos or not
If the answer is no and you are just using ssh keys the most
common cause of this issue is that the useres home directory is
group or world readable. In the most secure mode which is the
default if the useres home and or the ~/.ssh directory is has a
any thing other than 700 or 500 set as the permissions it will
reject the public key (the one on the server you are trying to
connect to) this become obvious with -vvv but not -vv

On Nov 21, 2012 7:34 PM, "Steven C Timm" mailto:t...@fnal.gov>> wrote:

Shouldn’t need to regenerate the keys.. once you get them
generated once they should be good for the life of the machine.

Save copies of the keys as they are now and if your system
goes bad, do differences to see what changed, if anything.

Steve Timm

*From:*owner-scientific-linux-us...@listserv.fnal.gov
<mailto:owner-scientific-linux-us...@listserv.fnal.gov>
[mailto:owner-scientific-linux-us...@listserv.fnal.gov
<mailto:owner-scientific-linux-us...@listserv.fnal.gov>] *On
Behalf Of *Joseph Areeda
*Sent:* Wednesday, November 21, 2012 5:46 PM
*To:* owner-scientific-linux-us...@listserv.fnal.gov
<mailto:owner-scientific-linux-us...@listserv.fnal.gov>
    *Cc:* scientific-linux-users
    *Subject:* Re: ssh returns "Permission denied
(gssapi-keyex,gssapi-with-mic)."

Thank you Tam, and Steven,

I just confirmed that regenerating the keys (ssh-keygen -t dsa
-f ssh_host_dsa_key && ssh -t rsa -f ssh_host_rsa_key) in
/etc/ssh "fixes the problem"

So ssh -vv shows me how it's supposed to look.  I'll save that
and do a diff when it happens again.

As I continue my googling I can report on a few things it's not

Server machine has a fixed ip address and dns/rdns appears
working.

Time issue Steven mentioned does not seem to be it, although I
may stop using pool machines and set up a local ntp server so
everybody gets the same time.  I can ssh and gsissh to other
servers.

Server:
ntpq -p

 remote   refid  st t when poll reach  
delay   offset  jitter


==
*ping-audit-207- .ACTS.   1 u5  128  377  
19.8675.804   1.927
+10504.x.rootbsd 198.30.92.2  2 u  129  128  376   45.146 
-28.571   5.558
+ntp.sunflower.c 132.236.56.250   3 u   77  128  355   63.836 
-14.753   5.360

-ntp2.ResComp.Be <http://ntp2.ResComp.Be> 128.32.206.553
 u  126  128  377  
22.1127.311   2.022



Client:

ntpq -p
 remote   refid  st t when poll reach  
delay   offset  jitter


==
 64.147.116.229  .ACTS.   1 u   47  1280  
13.5430.567   0.000
*nist1-chi.ustim .ACTS.   1 u   25  128  377 
106.619   14.458   5.896

+name3.glorb.com <http://name3.glorb.com> 69.36.224.15 2
u   64  128  377   88.564  -27.542   3.631
+131.211.8.244   .PPS.1 u   81  128  377 
167.1073.259   2.340





The only setting I change in sshd_config is to turn off
password auth b

Re: ssh returns "Permission denied (gssapi-keyex,gssapi-with-mic)."

[Alan Bartlett]

On 22 November 2012 01:18, Joseph Areeda  wrote:
>
> The user's directory is 755 which is the convention for grid computers in
> our collaboration and the plan is for this machine to be on our soon to be
> delivered cluster.  The .ssh directory is 700.  This doesn't change between
> the working and non-working state.

Good, you've checked the directory.

Now what about the files within it? Hopefully they are all 600?

Alan.

Re: ssh returns "Permission denied (gssapi-keyex,gssapi-with-mic)."

[Joseph Areeda]

On 11/21/2012 07:08 PM, Alan Bartlett wrote:

On 22 November 2012 01:18, Joseph Areeda  wrote:

The user's directory is 755 which is the convention for grid computers in
our collaboration and the plan is for this machine to be on our soon to be
delivered cluster.  The .ssh directory is 700.  This doesn't change between
the working and non-working state.

Good, you've checked the directory.

Now what about the files within it? Hopefully they are all 600?

Alan.

Alan,

The private keys are all 600 and the public keys are 644.  I keep a few 
different ones for going to different systems.


Joe

Re: ssh returns "Permission denied (gssapi-keyex,gssapi-with-mic)."

[Paul Robert Marino]

Well there is your problem
The users home directory needs to be 700 unless you turn off strict key
checking in the sshd configuration file. Also the public key should be 600
as well.

Making home directories world or group readable isn't a good plan for
collaberation because many applications store sensitive information like
passwords and cached information like session data in the home directory.
instead consider creating group directories an setting the setgid bit on it
so the group permissions are inherited by any files created in the
directories.
Making home directories world or group readable is a lazy solution to an
easily solved problem. Its a common mistake that causes loads of problems
because many application which are written to be secure purposly break when
you do it.
I highly suggest you comeup with a better plan for collaberation than that.
On Nov 21, 2012 11:10 PM, "Joseph Areeda"  wrote:

> On 11/21/2012 07:08 PM, Alan Bartlett wrote:
>
>> On 22 November 2012 01:18, Joseph Areeda  wrote:
>>
>>> The user's directory is 755 which is the convention for grid computers in
>>> our collaboration and the plan is for this machine to be on our soon to
>>> be
>>> delivered cluster.  The .ssh directory is 700.  This doesn't change
>>> between
>>> the working and non-working state.
>>>
>> Good, you've checked the directory.
>>
>> Now what about the files within it? Hopefully they are all 600?
>>
>> Alan.
>>
> Alan,
>
> The private keys are all 600 and the public keys are 644.  I keep a few
> different ones for going to different systems.
>
> Joe
>

Re: ssh returns "Permission denied (gssapi-keyex,gssapi-with-mic)."

[Joseph Areeda]

Thanks for the comments Paul.

I was surprised when I joined the collaboration and saw home directories 
world readable but that decision was made long before I arrived and 
changing it remains above my pay grade.


The reason I doubt that's my current problem is because regenerating the 
server key files works.  I can log in fine today and I haven't changed 
permissions.  I also don't have problem logging into other systems from 
that machine that are [supposed to be] set up the same way.


When it happens again, I will check if changing permissions helps.

Also for the record I waited until my existing Kerberos tickets 
expired.  These are to other services not that machine.  I can log in 
fine with an expired or valid TGT hanging around and after kdestroy.


Happy holidays,
Joe




On 11/22/2012 08:32 AM, Paul Robert Marino wrote:


Well there is your problem
The users home directory needs to be 700 unless you turn off strict 
key checking in the sshd configuration file. Also the public key 
should be 600 as well.


Making home directories world or group readable isn't a good plan for 
collaberation because many applications store sensitive information 
like passwords and cached information like session data in the home 
directory. instead consider creating group directories an setting the 
setgid bit on it so the group permissions are inherited by any files 
created in the directories.
Making home directories world or group readable is a lazy solution to 
an easily solved problem. Its a common mistake that causes loads of 
problems because many application which are written to be secure 
purposly break when you do it.
I highly suggest you comeup with a better plan for collaberation than 
that.


On Nov 21, 2012 11:10 PM, "Joseph Areeda" > wrote:


On 11/21/2012 07:08 PM, Alan Bartlett wrote:

On 22 November 2012 01:18, Joseph Areeda mailto:newsre...@areeda.com>> wrote:

The user's directory is 755 which is the convention for
grid computers in
our collaboration and the plan is for this machine to be
on our soon to be
delivered cluster.  The .ssh directory is 700.  This
doesn't change between
the working and non-working state.

Good, you've checked the directory.

Now what about the files within it? Hopefully they are all 600?

Alan.

Alan,

The private keys are all 600 and the public keys are 644.  I keep
a few different ones for going to different systems.

Joe

Resolved: ssh returns "Permission denied (gssapi-keyex,gssapi-with-mic)."

[Joseph Areeda]

Well this has been a thorn in my side for months but I think I've 
figured it out.  At least I found a plausible reason for it and it's 
been working longer than it has before.


The problem turned out to be I had both gsisshd and sshd running and the 
fix was to use chkconfig to disable it.


The really weird part that made it hard to figure out was that ssh would 
work for days then suddenly stop.  "sudo service sshd restart" would get 
it to work again for a few days.


I had installed the gsi server stuff because we will (hopefully) move to 
that certificate based access soon, not thinking that it would be 
enabled on install.


The take home lesson is think before you install potentially conflicting 
services.


Thanks,
Joe

On 11/21/2012 02:16 PM, Joseph Areeda wrote:

I can't figure out what causes this error.

I can "fix" it by regenerating the server key on the system I'm trying 
to connect to and restarting sshd but that seems to be temporary as 
the same problem comes back in a week or so.  Rebooting the server 
does not fix it.


Does anyone know what that error means?  I am using ssh not gsissh 
although I do have globus toolkit installed to contact grid computers.


I'm pretty sure it's a misconfiguration on my part but I can't figure 
out what I did or didn't do.


Thanks,

Joe