Re: ssh session times out annoyingly fast, why?

[Tom Dial]

On 9/29/20 06:53, Michael Stone wrote:
> On Tue, Sep 29, 2020 at 08:44:18AM -0400, Gene Heskett wrote:
>> This is likely quite true Michael, but it also is only a hint as to how
>> to fix it for the OP.
> 
> It was already fixed, serveraliveinterval/clientaliveinterval is the
> right answer. I guess I can review: these options simply have the client
> & server exchange an encrypted "are you here" message every N seconds to
> prevent the firewall from timing out the connection. tcpkeepalives won't
> do that, as the firewall can see that there is no actual data being
> transferred and may still timeout idle connections.
> 
> All that aside, it's important to be precise about what functionality is
> related to NAT and what functionality is related to firewalling.
> Imprecision about these concepts leads to all sorts of (wrong) ideas
> like "you need NAT to be secure".

+2

I use NAT for convenience, and a firewall (and other measures) for security.

And thank you for stating the distinction clearly; I sort of knew it,
but clarity always is a good thing.

Tom Dial

Re: ssh session times out annoyingly fast, why?

[Michael Stone]

On Tue, Sep 29, 2020 at 11:13:59AM -0400, Stefan Monnier wrote:

In general it's kind of dumb on modern hardware to expire sessions
that are still exchanging TCP keepalives unless you're under extreme
pressure from a DoS attack or somesuch.


Indeed, I'd be *very* surprised if a connection was dropped despite
exchange of TCP keepalives.  It seems much more likely that the
keepalives aren't used by the application (quite common and normal) or
that they get filtered somewhere.


Nope, it's reasonably common on the internet and a complete PITA.


But people rarely get to choose the other end's firewall
configuration, so enter kludges like the ssh protocol keepalives.


According to `man ssh(d)_config` one reason to use SSH's `Clientalive` or
`ServerAlive` is that, contrary to TCP keepalives, it can't be spoofed.


The issue with spoofing is potentially *too much* keeping alive, and if 
you read further that can be relevant if you for some reason need to 
know that an ssh connection has died but (e.g.) a malicious third party 
is using TCP keepalives to prevent ssh from knowing that the other end 
is down. If the problem you're trying to solve is not enough keeping 
alive (that is, your ssh connection is dying) rather than too much 
keeping alive, this reason is irrelevant. The protocol keepalives 
*also* fix the problem of firewalls timing out connections with TCP
keepalives. I don't know why the man page doesn't just say that, maybe 
ideological opposition to accomodating firewall stupidity.

Re: ssh session times out annoyingly fast, why?

[Stefan Monnier]

> In general it's kind of dumb on modern hardware to expire sessions
> that are still exchanging TCP keepalives unless you're under extreme
> pressure from a DoS attack or somesuch.

Indeed, I'd be *very* surprised if a connection was dropped despite
exchange of TCP keepalives.  It seems much more likely that the
keepalives aren't used by the application (quite common and normal) or
that they get filtered somewhere.

> But people rarely get to choose the other end's firewall
> configuration, so enter kludges like the ssh protocol keepalives. 

According to `man ssh(d)_config` one reason to use SSH's `Clientalive` or
`ServerAlive` is that, contrary to TCP keepalives, it can't be spoofed.


Stefan

Re: ssh session times out annoyingly fast, why?

[tomas]

On Tue, Sep 29, 2020 at 10:44:13AM -0400, Michael Stone wrote:
> On Tue, Sep 29, 2020 at 04:34:06PM +0200, to...@tuxteam.de wrote:
> >Setting the socket option to keep alive "fixed" that.
> 
> You were lucky. ssh does that by default, so if ssh sessions are
> getting killed these days it's because the firewall ignores tcp
> keepalives when calculating timeouts. If you're in such an
> environment and can't fix the firewall, then every application needs
> to be written to explicitly
> exchange data when idle to keep connections alive.

It wasn't ssh in this case. It was a (Perl DBI) database connection,
which, by default, is silent on inactivity. So after one hour, the
NAT dropped it.

To set the keepalive option, I had to convince the application
provider to update its (then already paleontological) Perl version
to one in which setting the keepalive socket option was possible.

In the end, that helped.

(I first tried to talk the customer into hitting their data centre
provider with a Thick Ethernet cable, but wasn't successful, alas).

This was anoter long story on its own :)

If the above NAT is killing entries which send keepalives then
a Thick Ethernet cable probably won't help either. That's downright
malicious.

Cheers
 - t


signature.asc
Description: Digital signature

Re: ssh session times out annoyingly fast, why?

[Michael Stone]

On Tue, Sep 29, 2020 at 04:34:06PM +0200, to...@tuxteam.de wrote:

Setting the socket option to keep alive "fixed" that.


You were lucky. ssh does that by default, so if ssh sessions are getting 
killed these days it's because the firewall ignores tcp keepalives when 
calculating timeouts. If you're in such an environment and can't fix the 
firewall, then every application needs to be written to explicitly

exchange data when idle to keep connections alive.

Re: ssh session times out annoyingly fast, why?

[Michael Stone]

On Tue, Sep 29, 2020 at 04:22:32PM +0200, to...@tuxteam.de wrote:

On Tue, Sep 29, 2020 at 08:18:54AM -0400, Michael Stone wrote:

On Wed, Sep 23, 2020 at 03:48:56PM -0400, Greg Wooledge wrote:
>The normal reason people need to use ServerAlive or ClientAlive is NAT.
>If your connection from ssh client to ssh server goes through a NAT
>router, the router may keep track of activity on that connection, and
>drop the translation when it goes idle for 5 minutes or so.  Forcing the
>*Alive packets to happen every few minutes prevents a NAT timeout.

This is a stateful firewall thing, not a NAT thing


That depends on what Greg means by "activity". NAT has to keep a
map of (internal IP, internal local port) to external local port
to do the translation (the so-called "translation table"). Since
it'd grow without bounds whenever one side drops the connection,
it's customary to let NAT table entries to expire after some
inactivity (typical: 1h, but network admins are known to be a
capricious species ;-)

So Greg is probably right. NAT is, in its own way, stateful.


NAT is a special case of a stateful firewall. You can get rid of NAT but 
basically the entire modern internet has stateful firewalls so getting 
rid of NAT won't make the problem at hand go away. The basic connection 
state tables and NAT state tables track basically the same information 
using the same algorithms for session start & stop, have the same issues 
with potentially leaking entries if hosts disappear, and have the same 
strategy of expiring inactive entries. 

In general it's kind of dumb on modern hardware to expire sessions that 
are still exchanging TCP keepalives unless you're under extreme pressure 
from a DoS attack or somesuch. (Modern devices just don't have the 
memory constraints that were an issue 20 years ago and don't need to 
aggressively prune sessions that are actively advertising that they're 
alive.) But people rarely get to choose the other end's firewall 
configuration, so enter kludges like the ssh protocol keepalives. 

Re: ssh session times out annoyingly fast, why?

[tomas]

On Tue, Sep 29, 2020 at 04:22:32PM +0200, to...@tuxteam.de wrote:

Following up on myself: I had exactly this case with an (outsourced)
data centre: they had NATs between different realms (you might ask
"why, oh, why?" and you'd be right). The application server and the
database server were separated by a NAT. To add insult to injury,
ICMP "not reachable" packets were filtered. The database connection
NAT entry timed out from time to time. It took a timeout of several
minutes for the application to notice that and to reconnect.

Lots of hilarity ensued.

Setting the socket option to keep alive "fixed" that.

Cheers
 - t




signature.asc
Description: Digital signature

Re: ssh session times out annoyingly fast, why?

[tomas]

On Tue, Sep 29, 2020 at 08:18:54AM -0400, Michael Stone wrote:
> On Wed, Sep 23, 2020 at 03:48:56PM -0400, Greg Wooledge wrote:
> >The normal reason people need to use ServerAlive or ClientAlive is NAT.
> >If your connection from ssh client to ssh server goes through a NAT
> >router, the router may keep track of activity on that connection, and
> >drop the translation when it goes idle for 5 minutes or so.  Forcing the
> >*Alive packets to happen every few minutes prevents a NAT timeout.
> 
> This is a stateful firewall thing, not a NAT thing

That depends on what Greg means by "activity". NAT has to keep a
map of (internal IP, internal local port) to external local port
to do the translation (the so-called "translation table"). Since
it'd grow without bounds whenever one side drops the connection,
it's customary to let NAT table entries to expire after some
inactivity (typical: 1h, but network admins are known to be a
capricious species ;-)

So Greg is probably right. NAT is, in its own way, stateful.

Cheers
 - t


signature.asc
Description: Digital signature

Re: ssh session times out annoyingly fast, why?

[Michael Stone]

On Tue, Sep 29, 2020 at 08:44:18AM -0400, Gene Heskett wrote:

This is likely quite true Michael, but it also is only a hint as to how
to fix it for the OP.


It was already fixed, serveraliveinterval/clientaliveinterval is the 
right answer. I guess I can review: these options simply have the client 
& server exchange an encrypted "are you here" message every N seconds to 
prevent the firewall from timing out the connection. tcpkeepalives won't 
do that, as the firewall can see that there is no actual data being 
transferred and may still timeout idle connections.


All that aside, it's important to be precise about what functionality is 
related to NAT and what functionality is related to firewalling. 
Imprecision about these concepts leads to all sorts of (wrong) ideas 
like "you need NAT to be secure".

Re: ssh session times out annoyingly fast, why?

[Gene Heskett]

On Tuesday 29 September 2020 08:18:54 Michael Stone wrote:

> On Wed, Sep 23, 2020 at 03:48:56PM -0400, Greg Wooledge wrote:
> >The normal reason people need to use ServerAlive or ClientAlive is
> > NAT. If your connection from ssh client to ssh server goes through a
> > NAT router, the router may keep track of activity on that
> > connection, and drop the translation when it goes idle for 5 minutes
> > or so.  Forcing the *Alive packets to happen every few minutes
> > prevents a NAT timeout.
>
> This is a stateful firewall thing, not a NAT thing

This is likely quite true Michael, but it also is only a hint as to how 
to fix it for the OP. I  maintain 8 to 12 such ssh connections here to 
my othermachines, establishing them at boot time, but all are local 
192.168.xx.xx addresses so not NAT'd going either direction, so I am not 
affected. I would be upset if I was.

Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page 

Re: ssh session times out annoyingly fast, why?

[Michael Stone]

On Wed, Sep 23, 2020 at 03:48:56PM -0400, Greg Wooledge wrote:

The normal reason people need to use ServerAlive or ClientAlive is NAT.
If your connection from ssh client to ssh server goes through a NAT
router, the router may keep track of activity on that connection, and
drop the translation when it goes idle for 5 minutes or so.  Forcing the
*Alive packets to happen every few minutes prevents a NAT timeout.


This is a stateful firewall thing, not a NAT thing

Re: ssh session times out annoyingly fast, why?

[Greg Wooledge]

On Sat, Sep 26, 2020 at 01:16:19PM -0800, Britton Kerin wrote:
> Looks like NAT was the culprit, because top kept it alive.  Internet has bogus
> advice on this one because it suggests ServerAliveInterval 1200 or something
> which I guess is larger than most firewall timeout.
> 
> Thanks for all help good to see debian community still so good.

I use "ServerAliveInterval 300" with my cheap consumer-grade router.

Re: ssh session times out annoyingly fast, why?

[Britton Kerin]

On Wed, Sep 23, 2020 at 11:49 AM Greg Wooledge  wrote:
>
> On Wed, Sep 23, 2020 at 10:30:15AM +0300, Anssi Saari wrote:
> > Britton Kerin  writes:
> > > I'm using ssh from a debian box to a rasberry pi (sorta debian also :).
> > >
> > > For some reason ssh sessions seem to time out pretty quickly.
>
> How quickly, exactly?  What is the actual message/behavior you see when
> it happens?  Are they both on the same LAN, or is there some complexity
> in between them (especially a NAT router)?
>
> > Well, the keepalives themselves can cause a disconnect if the keepalive
> > messages are not reaching the other end due to bad connection for
> > example. Looks like by default in Debian client sends keepalives if
> > server is quiet but server doesn't send keepalives to a client.
>
> The normal reason people need to use ServerAlive or ClientAlive is NAT.
> If your connection from ssh client to ssh server goes through a NAT
> router, the router may keep track of activity on that connection, and
> drop the translation when it goes idle for 5 minutes or so.  Forcing the
> *Alive packets to happen every few minutes prevents a NAT timeout.
>
> If there is no NAT involved, then I agree with the previous suggestion
> that this might be a shell's TMOUT variable.  Are you sitting at a shell
> prompt when the "timeout" occurs?  Does the timeout stop occurring when
> you're inside a text editor, for example?

Looks like NAT was the culprit, because top kept it alive.  Internet has bogus
advice on this one because it suggests ServerAliveInterval 1200 or something
which I guess is larger than most firewall timeout.

Thanks for all help good to see debian community still so good.

Britton

Re: ssh session times out annoyingly fast, why?

[Greg Wooledge]

On Wed, Sep 23, 2020 at 10:30:15AM +0300, Anssi Saari wrote:
> Britton Kerin  writes:
> > I'm using ssh from a debian box to a rasberry pi (sorta debian also :).
> >
> > For some reason ssh sessions seem to time out pretty quickly.

How quickly, exactly?  What is the actual message/behavior you see when
it happens?  Are they both on the same LAN, or is there some complexity
in between them (especially a NAT router)?

> Well, the keepalives themselves can cause a disconnect if the keepalive
> messages are not reaching the other end due to bad connection for
> example. Looks like by default in Debian client sends keepalives if
> server is quiet but server doesn't send keepalives to a client.

The normal reason people need to use ServerAlive or ClientAlive is NAT.
If your connection from ssh client to ssh server goes through a NAT
router, the router may keep track of activity on that connection, and
drop the translation when it goes idle for 5 minutes or so.  Forcing the
*Alive packets to happen every few minutes prevents a NAT timeout.

If there is no NAT involved, then I agree with the previous suggestion
that this might be a shell's TMOUT variable.  Are you sitting at a shell
prompt when the "timeout" occurs?  Does the timeout stop occurring when
you're inside a text editor, for example?

Much more information is needed here.

Re: ssh session times out annoyingly fast, why?

[Anssi Saari]

Britton Kerin  writes:

> I'm using ssh from a debian box to a rasberry pi (sorta debian also :).
>
> For some reason ssh sessions seem to time out pretty quickly.  I've
> tried setting ClientAliveInterval and ClientAliveCountMax and also
> ServerAliveInterval  and ServerAliveCountMax, but it doesn't seem to
> make any difference.  Is there some other setting somewhere that
> affects this?

Well, the keepalives themselves can cause a disconnect if the keepalive
messages are not reaching the other end due to bad connection for
example. Looks like by default in Debian client sends keepalives if
server is quiet but server doesn't send keepalives to a client.

You could try mosh instead of ssh. Or enable some debug printouts with
ssh -v (or -vv or -vvv for max messages) to see if that tells you why
the connection dropped.

Re: ssh session times out annoyingly fast, why?

[Gary Dale]

On 2020-09-21 19:38, Britton Kerin wrote:

I'm using ssh from a debian box to a rasberry pi (sorta debian also :).

For some reason ssh sessions seem to time out pretty quickly.  I've
tried setting ClientAliveInterval and ClientAliveCountMax and also
ServerAliveInterval  and ServerAliveCountMax, but it doesn't seem to
make any difference.  Is there some other setting somewhere that
affects this?

Thanks,
Britton

My money is on a network issue. Lately my connection to a remote server 
seems to lock up quickly while I have a stable connection to a local 
server. Both servers are running Debian/Stable and I haven't fiddled 
with the ssh settings in a long time.

Re: ssh session times out annoyingly fast, why?

[Darac Marjal]

On 22/09/2020 00:38, Britton Kerin wrote:
> I'm using ssh from a debian box to a rasberry pi (sorta debian also :).
>
> For some reason ssh sessions seem to time out pretty quickly.  I've
> tried setting ClientAliveInterval and ClientAliveCountMax and also
> ServerAliveInterval  and ServerAliveCountMax, but it doesn't seem to
> make any difference.  Is there some other setting somewhere that
> affects this?

Have a look at the environmental variable TMOUT. From the bash manpage:

  TMOUT  If set to a value greater than zero, TMOUT is treated as
the default timeout for the read builtin.  The select command terminates
if input does not arrive after TMOUT seconds when input is coming from a
terminal.  In  an interactive  shell,  the  value  is interpreted as the
number of seconds to wait for a line of input after issuing the primary
prompt.  Bash terminates after waiting for that number of seconds if a
complete line of input does not arrive.


>
> Thanks,
> Britton
>



signature.asc
Description: OpenPGP digital signature

Re: ssh session times out annoyingly fast, why?

[Toni Mas Soler]

First, you should be sure it is not a network issue.
You could open a terminal and run, for example, top program. This
avoid any timeout configured. If this does not work, you should follow
for a network issue, otherwise we can see sshd's config file.

Toni Mas

Missatge de Britton Kerin  del dia dt., 22 de
set. 2020 a les 1:38:
>
> I'm using ssh from a debian box to a rasberry pi (sorta debian also :).
>
> For some reason ssh sessions seem to time out pretty quickly.  I've
> tried setting ClientAliveInterval and ClientAliveCountMax and also
> ServerAliveInterval  and ServerAliveCountMax, but it doesn't seem to
> make any difference.  Is there some other setting somewhere that
> affects this?
>
> Thanks,
> Britton
>

Re: ssh session times out annoyingly fast, why?

[Joseph Loo]

have you tried "ssh -Y" option

On 9/21/20 4:38 PM, Britton Kerin wrote:

I'm using ssh from a debian box to a rasberry pi (sorta debian also :).

For some reason ssh sessions seem to time out pretty quickly.  I've
tried setting ClientAliveInterval and ClientAliveCountMax and also
ServerAliveInterval  and ServerAliveCountMax, but it doesn't seem to
make any difference.  Is there some other setting somewhere that
affects this?

Thanks,
Britton



--
Joseph Loo
j...@acm.org