Re: Timeout dead connections
Hi! I still have problems, this is my output from 'who': admin pts/0 02:50 Apr 5 07:24:09 x.x.x.x admin pts/1 00:00 Apr 5 09:39:05 y.y.y.y current time: Fri Apr 5 10:18:27 CEST 2013 shouldn't the first session be timed out? It has not just been idle for 2 h 50 min, the computer is not there any more. So in my opinion, dropbear should have forgotten the connection. Mattias On 2013-04-01 17:01, Matt Johnston wrote: Hi, The attached attached patch against 2013.56 should fix it, or https://secure.ucc.asn.au/hg/dropbear/rev/70811267715c Dropbear wasn't running cleanup handlers when it exited due to the TCP connection being closed. Matt On Thu, Mar 28, 2013 at 07:24:55PM +0800, Matt Johnston wrote: I think that -K on the server should be enough. On the server can you run tcpdump -i eth0 -w cap1.cap port 22, get a ssh session going, pull out the cable, wait 10 minutes, then send me the capture? Could you also check that the Dropbear process for the connection is still running after the connection should have been finished. It's possible that the process is exiting but the session cleanup code isn't working correctly. The whole debug log might give me an idea what's going on. Cheers, Matt On Thu, Mar 28, 2013 at 09:56:02AM +0100, Mattias Walström wrote: Thanks for your responses, all your suggestions imply that you should do something in the client (set keepalive on client end), but shouldn't the server itself be able to decide if a client is dead (can't OpenSSH do this?). If I do the -K 15 -I 20 on the server end only, this will close the connection when the OpenSSH client has not sent any characters in 20s. I expected the keepalive to be two way, that the server got responses on these packages as well, is that not the case? Regards Mattias On Wed, Mar 27, 2013 at 11:24 AM, Mattias Walström mattias.walst...@westermo.se wrote: Hi! I am running dropbear 2013.56, connecting to the server with a PC but not performing a clean close (I pulled my ethernet cable), this caused dropbear to never drop its connection. Looking at the utmp entries, I could see that the connection never got dropped, the utmp entries was kept forever, and running with debug indicates that also. Tried to use -K to send keepalive, but it just keeps sending keepalives to the peer, even it is no longer there, and not possible to reach. Shouldn't the connection be dropped if the keepalive does not reach its destination? I know there is the -I option, but that does not really do what I want, I want the connection to be tear down when the peer is unreachable, not when the user has been idle for a while. Regards Mattias
Re: Timeout dead connections
Are you using -K ? I wouldn't expect it to time out otherwise. As an example I hibernate my computer nightly but connections remain alive in the morning. Cheers, Matt On 2013-04-05 16:25, Mattias Walström wrote: Hi! I still have problems, this is my output from 'who': admin pts/0 02:50 Apr 5 07:24:09 x.x.x.x admin pts/1 00:00 Apr 5 09:39:05 y.y.y.y current time: Fri Apr 5 10:18:27 CEST 2013 shouldn't the first session be timed out? It has not just been idle for 2 h 50 min, the computer is not there any more. So in my opinion, dropbear should have forgotten the connection. Mattias On 2013-04-01 17:01, Matt Johnston wrote: Hi, The attached attached patch against 2013.56 should fix it, or https://secure.ucc.asn.au/hg/dropbear/rev/70811267715c Dropbear wasn't running cleanup handlers when it exited due to the TCP connection being closed. Matt On Thu, Mar 28, 2013 at 07:24:55PM +0800, Matt Johnston wrote: I think that -K on the server should be enough. On the server can you run tcpdump -i eth0 -w cap1.cap port 22, get a ssh session going, pull out the cable, wait 10 minutes, then send me the capture? Could you also check that the Dropbear process for the connection is still running after the connection should have been finished. It's possible that the process is exiting but the session cleanup code isn't working correctly. The whole debug log might give me an idea what's going on. Cheers, Matt On Thu, Mar 28, 2013 at 09:56:02AM +0100, Mattias Walström wrote: Thanks for your responses, all your suggestions imply that you should do something in the client (set keepalive on client end), but shouldn't the server itself be able to decide if a client is dead (can't OpenSSH do this?). If I do the -K 15 -I 20 on the server end only, this will close the connection when the OpenSSH client has not sent any characters in 20s. I expected the keepalive to be two way, that the server got responses on these packages as well, is that not the case? Regards Mattias On Wed, Mar 27, 2013 at 11:24 AM, Mattias Walström mattias.walst...@westermo.se wrote: Hi! I am running dropbear 2013.56, connecting to the server with a PC but not performing a clean close (I pulled my ethernet cable), this caused dropbear to never drop its connection. Looking at the utmp entries, I could see that the connection never got dropped, the utmp entries was kept forever, and running with debug indicates that also. Tried to use -K to send keepalive, but it just keeps sending keepalives to the peer, even it is no longer there, and not possible to reach. Shouldn't the connection be dropped if the keepalive does not reach its destination? I know there is the -I option, but that does not really do what I want, I want the connection to be tear down when the peer is unreachable, not when the user has been idle for a while. Regards Mattias
Re: Timeout dead connections
2013/4/5 Matt Johnston m...@ucc.asn.au: Are you using -K ? I wouldn't expect it to time out otherwise. As an example I hibernate my computer nightly but connections remain alive in the morning. I got same issue with 2012.55-1.3@debian(debian does not have 2013.56 at the moment) without -K switch. Once I not issuing 'exit' command but closing putty window directly, the session leaves alone. Cheers, Matt On 2013-04-05 16:25, Mattias Walström wrote: Hi! I still have problems, this is my output from 'who': admin pts/0 02:50 Apr 5 07:24:09 x.x.x.x admin pts/1 00:00 Apr 5 09:39:05 y.y.y.y current time: Fri Apr 5 10:18:27 CEST 2013 shouldn't the first session be timed out? It has not just been idle for 2 h 50 min, the computer is not there any more. So in my opinion, dropbear should have forgotten the connection. Mattias On 2013-04-01 17:01, Matt Johnston wrote: Hi, The attached attached patch against 2013.56 should fix it, or https://secure.ucc.asn.au/hg/dropbear/rev/70811267715c Dropbear wasn't running cleanup handlers when it exited due to the TCP connection being closed. Matt On Thu, Mar 28, 2013 at 07:24:55PM +0800, Matt Johnston wrote: I think that -K on the server should be enough. On the server can you run tcpdump -i eth0 -w cap1.cap port 22, get a ssh session going, pull out the cable, wait 10 minutes, then send me the capture? Could you also check that the Dropbear process for the connection is still running after the connection should have been finished. It's possible that the process is exiting but the session cleanup code isn't working correctly. The whole debug log might give me an idea what's going on. Cheers, Matt On Thu, Mar 28, 2013 at 09:56:02AM +0100, Mattias Walström wrote: Thanks for your responses, all your suggestions imply that you should do something in the client (set keepalive on client end), but shouldn't the server itself be able to decide if a client is dead (can't OpenSSH do this?). If I do the -K 15 -I 20 on the server end only, this will close the connection when the OpenSSH client has not sent any characters in 20s. I expected the keepalive to be two way, that the server got responses on these packages as well, is that not the case? Regards Mattias On Wed, Mar 27, 2013 at 11:24 AM, Mattias Walström mattias.walst...@westermo.se wrote: Hi! I am running dropbear 2013.56, connecting to the server with a PC but not performing a clean close (I pulled my ethernet cable), this caused dropbear to never drop its connection. Looking at the utmp entries, I could see that the connection never got dropped, the utmp entries was kept forever, and running with debug indicates that also. Tried to use -K to send keepalive, but it just keeps sending keepalives to the peer, even it is no longer there, and not possible to reach. Shouldn't the connection be dropped if the keepalive does not reach its destination? I know there is the -I option, but that does not really do what I want, I want the connection to be tear down when the peer is unreachable, not when the user has been idle for a while. Regards Mattias
Re: Timeout dead connections
2013/4/5 Matt Johnston m...@ucc.asn.au: Roy Tam roy...@gmail.com wrote: I got same issue with 2012.55-1.3@debian(debian does not have 2013.56 at the moment) without -K switch. Once I not issuing 'exit' command but closing putty window directly, the session leaves alone. That sounds exactly like the situation fixed in 2013.56 I tried dpkg-buildpackage -rfakeroot dropbear-2013.56 and installed dropbear_2013.56-0.1_i386.deb restarted dropbear services, open 2 putty connection to that host, and press [X] button on one of them, then run w on another one, it show 2 sessions. In ps auxww, it shows 3 lines of: /usr/sbin/dropbear -d /etc/dropbear/dropbear_dss_host_key -r /etc/dropbear/dropbear_rsa_host_key -p 22 -W 65536 So, the problem still exists. Cheers, Matt Cheers, Matt On 2013-04-05 16:25, Mattias Walström wrote: Hi! I still have problems, this is my output from 'who': admin pts/0 02:50 Apr 5 07:24:09 x.x.x.x admin pts/1 00:00 Apr 5 09:39:05 y.y.y.y current time: Fri Apr 5 10:18:27 CEST 2013 shouldn't the first session be timed out? It has not just been idle for 2 h 50 min, the computer is not there any more. So in my opinion, dropbear should have forgotten the connection. Mattias On 2013-04-01 17:01, Matt Johnston wrote: Hi, The attached attached patch against 2013.56 should fix it, or https://secure.ucc.asn.au/hg/dropbear/rev/70811267715c Dropbear wasn't running cleanup handlers when it exited due to the TCP connection being closed. Matt On Thu, Mar 28, 2013 at 07:24:55PM +0800, Matt Johnston wrote: I think that -K on the server should be enough. On the server can you run tcpdump -i eth0 -w cap1.cap port 22, get a ssh session going, pull out the cable, wait 10 minutes, then send me the capture? Could you also check that the Dropbear process for the connection is still running after the connection should have been finished. It's possible that the process is exiting but the session cleanup code isn't working correctly. The whole debug log might give me an idea what's going on. Cheers, Matt On Thu, Mar 28, 2013 at 09:56:02AM +0100, Mattias Walström wrote: Thanks for your responses, all your suggestions imply that you should do something in the client (set keepalive on client end), but shouldn't the server itself be able to decide if a client is dead (can't OpenSSH do this?). If I do the -K 15 -I 20 on the server end only, this will close the connection when the OpenSSH client has not sent any characters in 20s. I expected the keepalive to be two way, that the server got responses on these packages as well, is that not the case? Regards Mattias On Wed, Mar 27, 2013 at 11:24 AM, Mattias Walström mattias.walst...@westermo.se wrote: Hi! I am running dropbear 2013.56, connecting to the server with a PC but not performing a clean close (I pulled my ethernet cable), this caused dropbear to never drop its connection. Looking at the utmp entries, I could see that the connection never got dropped, the utmp entries was kept forever, and running with debug indicates that also. Tried to use -K to send keepalive, but it just keeps sending keepalives to the peer, even it is no longer there, and not possible to reach. Shouldn't the connection be dropped if the keepalive does not reach its destination? I know there is the -I option, but that does not really do what I want, I want the connection to be tear down when the peer is unreachable, not when the user has been idle for a while. Regards Mattias
Re: Timeout dead connections
I am not using Keepalive. Why is keepalive required? If reading the manual understood that was to make sure firewalls did not close the connection. Shouldn't a TCP socket timeout in maximum 15 minutes by itself? Mattias On 2013-04-05 13:18, Matt Johnston wrote: Are you using -K ? I wouldn't expect it to time out otherwise. As an example I hibernate my computer nightly but connections remain alive in the morning. Cheers, Matt On 2013-04-05 16:25, Mattias Walström wrote: Hi! I still have problems, this is my output from 'who': admin pts/0 02:50 Apr 5 07:24:09 x.x.x.x admin pts/1 00:00 Apr 5 09:39:05 y.y.y.y current time: Fri Apr 5 10:18:27 CEST 2013 shouldn't the first session be timed out? It has not just been idle for 2 h 50 min, the computer is not there any more. So in my opinion, dropbear should have forgotten the connection. Mattias On 2013-04-01 17:01, Matt Johnston wrote: Hi, The attached attached patch against 2013.56 should fix it, or https://secure.ucc.asn.au/hg/dropbear/rev/70811267715c Dropbear wasn't running cleanup handlers when it exited due to the TCP connection being closed. Matt On Thu, Mar 28, 2013 at 07:24:55PM +0800, Matt Johnston wrote: I think that -K on the server should be enough. On the server can you run tcpdump -i eth0 -w cap1.cap port 22, get a ssh session going, pull out the cable, wait 10 minutes, then send me the capture? Could you also check that the Dropbear process for the connection is still running after the connection should have been finished. It's possible that the process is exiting but the session cleanup code isn't working correctly. The whole debug log might give me an idea what's going on. Cheers, Matt On Thu, Mar 28, 2013 at 09:56:02AM +0100, Mattias Walström wrote: Thanks for your responses, all your suggestions imply that you should do something in the client (set keepalive on client end), but shouldn't the server itself be able to decide if a client is dead (can't OpenSSH do this?). If I do the -K 15 -I 20 on the server end only, this will close the connection when the OpenSSH client has not sent any characters in 20s. I expected the keepalive to be two way, that the server got responses on these packages as well, is that not the case? Regards Mattias On Wed, Mar 27, 2013 at 11:24 AM, Mattias Walström mattias.walst...@westermo.se wrote: Hi! I am running dropbear 2013.56, connecting to the server with a PC but not performing a clean close (I pulled my ethernet cable), this caused dropbear to never drop its connection. Looking at the utmp entries, I could see that the connection never got dropped, the utmp entries was kept forever, and running with debug indicates that also. Tried to use -K to send keepalive, but it just keeps sending keepalives to the peer, even it is no longer there, and not possible to reach. Shouldn't the connection be dropped if the keepalive does not reach its destination? I know there is the -I option, but that does not really do what I want, I want the connection to be tear down when the peer is unreachable, not when the user has been idle for a while. Regards Mattias
Re: Timeout dead connections
Actually.. when I looked at it again. My first session has been closed, it took some while, but at last it seems like it was closed. I just read more about TCP timeouts, must have mixed things up. It is rather long, but it seems that dropbear now clean upp after itself. Now in who, I can see that my first entry (made Apr 5 07:24:09) is now removed. admin pts/0 00:00 Apr 5 15:56:06 y.y.y.y admin pts/1 05:39 Apr 5 09:39:05 y.y.y.y admin pts/2 03:05 Apr 5 12:52:47 y.y.y.y Mattias On 2013-04-05 15:52, Mattias Walström wrote: I am not using Keepalive. Why is keepalive required? If reading the manual understood that was to make sure firewalls did not close the connection. Shouldn't a TCP socket timeout in maximum 15 minutes by itself? Mattias On 2013-04-05 13:18, Matt Johnston wrote: Are you using -K ? I wouldn't expect it to time out otherwise. As an example I hibernate my computer nightly but connections remain alive in the morning. Cheers, Matt On 2013-04-05 16:25, Mattias Walström wrote: Hi! I still have problems, this is my output from 'who': admin pts/0 02:50 Apr 5 07:24:09 x.x.x.x admin pts/1 00:00 Apr 5 09:39:05 y.y.y.y current time: Fri Apr 5 10:18:27 CEST 2013 shouldn't the first session be timed out? It has not just been idle for 2 h 50 min, the computer is not there any more. So in my opinion, dropbear should have forgotten the connection. Mattias On 2013-04-01 17:01, Matt Johnston wrote: Hi, The attached attached patch against 2013.56 should fix it, or https://secure.ucc.asn.au/hg/dropbear/rev/70811267715c Dropbear wasn't running cleanup handlers when it exited due to the TCP connection being closed. Matt On Thu, Mar 28, 2013 at 07:24:55PM +0800, Matt Johnston wrote: I think that -K on the server should be enough. On the server can you run tcpdump -i eth0 -w cap1.cap port 22, get a ssh session going, pull out the cable, wait 10 minutes, then send me the capture? Could you also check that the Dropbear process for the connection is still running after the connection should have been finished. It's possible that the process is exiting but the session cleanup code isn't working correctly. The whole debug log might give me an idea what's going on. Cheers, Matt On Thu, Mar 28, 2013 at 09:56:02AM +0100, Mattias Walström wrote: Thanks for your responses, all your suggestions imply that you should do something in the client (set keepalive on client end), but shouldn't the server itself be able to decide if a client is dead (can't OpenSSH do this?). If I do the -K 15 -I 20 on the server end only, this will close the connection when the OpenSSH client has not sent any characters in 20s. I expected the keepalive to be two way, that the server got responses on these packages as well, is that not the case? Regards Mattias On Wed, Mar 27, 2013 at 11:24 AM, Mattias Walström mattias.walst...@westermo.se wrote: Hi! I am running dropbear 2013.56, connecting to the server with a PC but not performing a clean close (I pulled my ethernet cable), this caused dropbear to never drop its connection. Looking at the utmp entries, I could see that the connection never got dropped, the utmp entries was kept forever, and running with debug indicates that also. Tried to use -K to send keepalive, but it just keeps sending keepalives to the peer, even it is no longer there, and not possible to reach. Shouldn't the connection be dropped if the keepalive does not reach its destination? I know there is the -I option, but that does not really do what I want, I want the connection to be tear down when the peer is unreachable, not when the user has been idle for a while. Regards Mattias
Re: Timeout dead connections
Hi, The attached attached patch against 2013.56 should fix it, or https://secure.ucc.asn.au/hg/dropbear/rev/70811267715c Dropbear wasn't running cleanup handlers when it exited due to the TCP connection being closed. Matt On Thu, Mar 28, 2013 at 07:24:55PM +0800, Matt Johnston wrote: I think that -K on the server should be enough. On the server can you run tcpdump -i eth0 -w cap1.cap port 22, get a ssh session going, pull out the cable, wait 10 minutes, then send me the capture? Could you also check that the Dropbear process for the connection is still running after the connection should have been finished. It's possible that the process is exiting but the session cleanup code isn't working correctly. The whole debug log might give me an idea what's going on. Cheers, Matt On Thu, Mar 28, 2013 at 09:56:02AM +0100, Mattias Walström wrote: Thanks for your responses, all your suggestions imply that you should do something in the client (set keepalive on client end), but shouldn't the server itself be able to decide if a client is dead (can't OpenSSH do this?). If I do the -K 15 -I 20 on the server end only, this will close the connection when the OpenSSH client has not sent any characters in 20s. I expected the keepalive to be two way, that the server got responses on these packages as well, is that not the case? Regards Mattias On Wed, Mar 27, 2013 at 11:24 AM, Mattias Walström mattias.walst...@westermo.se wrote: Hi! I am running dropbear 2013.56, connecting to the server with a PC but not performing a clean close (I pulled my ethernet cable), this caused dropbear to never drop its connection. Looking at the utmp entries, I could see that the connection never got dropped, the utmp entries was kept forever, and running with debug indicates that also. Tried to use -K to send keepalive, but it just keeps sending keepalives to the peer, even it is no longer there, and not possible to reach. Shouldn't the connection be dropped if the keepalive does not reach its destination? I know there is the -I option, but that does not really do what I want, I want the connection to be tear down when the peer is unreachable, not when the user has been idle for a while. Regards Mattias
Re: Timeout dead connections
And the patch actually attached here.
On Mon, Apr 01, 2013 at 11:01:42PM +0800, Matt Johnston wrote:
Hi,
The attached attached patch against 2013.56 should fix it, or
https://secure.ucc.asn.au/hg/dropbear/rev/70811267715c
Dropbear wasn't running cleanup handlers when it exited due
to the TCP connection being closed.
Matt
On Thu, Mar 28, 2013 at 07:24:55PM +0800, Matt Johnston wrote:
I think that -K on the server should be enough. On the
server can you run tcpdump -i eth0 -w cap1.cap port 22,
get a ssh session going, pull out the cable, wait 10
minutes, then send me the capture?
Could you also check that the Dropbear process for the
connection is still running after the connection should have
been finished. It's possible that the process is exiting but
the session cleanup code isn't working correctly. The whole
debug log might give me an idea what's going on.
Cheers,
Matt
On Thu, Mar 28, 2013 at 09:56:02AM +0100, Mattias Walström wrote:
Thanks for your responses, all your suggestions imply that you should do
something
in the client (set keepalive on client end), but shouldn't the server
itself be able to
decide if a client is dead (can't OpenSSH do this?).
If I do the -K 15 -I 20 on the server end only, this will close the
connection when
the OpenSSH client has not sent any characters in 20s. I expected the
keepalive to be
two way, that the server got responses on these packages as well, is that
not the case?
Regards
Mattias
On Wed, Mar 27, 2013 at 11:24 AM, Mattias Walström
mattias.walst...@westermo.se wrote:
Hi!
I am running dropbear 2013.56, connecting to the server with a PC but
not performing a clean close (I pulled my ethernet cable), this caused
dropbear to never drop its connection.
Looking at the utmp entries, I could see that the connection never got
dropped,
the utmp entries was kept forever, and running with debug indicates
that
also.
Tried to use -K to send keepalive, but it just keeps sending
keepalives
to the peer,
even it is no longer there, and not possible to reach. Shouldn't
the connection be dropped if the keepalive does not reach its
destination?
I know there is the -I option, but that does not really do what I want,
I want the connection to be tear down when the peer is unreachable, not
when the user has been idle for a while.
Regards
Mattias
diff -r 1b8b2b9d6e94 cli-main.c
--- a/cli-main.c Thu Mar 21 23:29:04 2013 +0800
+++ b/cli-main.c Mon Apr 01 22:59:26 2013 +0800
@@ -98,8 +98,7 @@
}
/* Do the cleanup first, since then the terminal will be reset */
- cli_session_cleanup();
- common_session_cleanup();
+ session_cleanup();
_dropbear_log(LOG_INFO, fmtbuf, param);
diff -r 1b8b2b9d6e94 cli-session.c
--- a/cli-session.c Thu Mar 21 23:29:04 2013 +0800
+++ b/cli-session.c Mon Apr 01 22:59:26 2013 +0800
@@ -41,6 +41,7 @@
static void cli_sessionloop();
static void cli_session_init();
static void cli_finished();
+static void cli_session_cleanup(void);
struct clientsession cli_ses; /* GLOBAL */
@@ -142,6 +143,7 @@
/* For printing remote host closed for the user */
ses.remoteclosed = cli_remoteclosed;
+ ses.extra_session_cleanup = cli_session_cleanup;
ses.buf_match_algo = cli_buf_match_algo;
/* packet handlers */
@@ -278,7 +280,7 @@
}
-void cli_session_cleanup() {
+static void cli_session_cleanup(void) {
if (!sessinitdone) {
return;
@@ -296,8 +298,7 @@
static void cli_finished() {
- cli_session_cleanup();
- common_session_cleanup();
+ session_cleanup();
fprintf(stderr, Connection to %s@%s:%s closed.\n, cli_opts.username,
cli_opts.remotehost, cli_opts.remoteport);
exit(cli_ses.retval);
diff -r 1b8b2b9d6e94 common-session.c
--- a/common-session.c Thu Mar 21 23:29:04 2013 +0800
+++ b/common-session.c Mon Apr 01 22:59:26 2013 +0800
@@ -225,7 +225,7 @@
}
/* clean up a session on exit */
-void common_session_cleanup() {
+void session_cleanup() {
TRACE((enter session_cleanup))
@@ -234,6 +234,10 @@
TRACE((leave session_cleanup: !sessinitdone))
return;
}
+
+ if (ses.extra_session_cleanup) {
+ ses.extra_session_cleanup();
+ }
m_free(ses.session_id);
m_burn(ses.keys, sizeof(struct key_context));
diff -r 1b8b2b9d6e94 session.h
--- a/session.h Thu Mar 21 23:29:04 2013 +0800
+++ b/session.h Mon Apr 01 22:59:26 2013 +0800
@@ -44,7 +44,7 @@
void common_session_init(int sock_in, int sock_out);
void session_loop(void(*loophandler)());
-void common_session_cleanup();
+void session_cleanup();
void session_identification();
void send_msg_ignore();
@@ -58,7 +58,6 @@
/* Client */
void cli_session(int sock_in, int sock_out);
-void cli_session_cleanup();
void cleantext(unsigned char* dirtytext);
/* crypto parameters that are stored individually for transmit and receive */
@@ -178,6 +177,7 @@
void(*remoteclosed)(); /*
Re: Timeout dead connections
Thanks for your responses, all your suggestions imply that you should do something in the client (set keepalive on client end), but shouldn't the server itself be able to decide if a client is dead (can't OpenSSH do this?). If I do the -K 15 -I 20 on the server end only, this will close the connection when the OpenSSH client has not sent any characters in 20s. I expected the keepalive to be two way, that the server got responses on these packages as well, is that not the case? Regards Mattias On 2013-03-27 16:47, Matt Johnston wrote: I thought those were fixed in 0.53 or perhaps 2011.54: 2011.54 - Tuesday 8 November 2011 - Fixed case where -K 1 keepalive for dbclient would cause a SSH_MSG_IGNORE packet to be sent 0.53 - Thurs 24 February 2011 - Make -K (keepalive) and -I (idle timeout) work together sensibly in the client. The idle timeout is no longer reset by SSH_MSG_IGNORE packets. If the network cable has been pulled out, shouldn't the OS send a TCP RST packet eventually after some traffic and close the connection? Cheers, Matt On Wed, Mar 27, 2013 at 11:41:40AM -0400, Fabrizio Bertocci wrote: I remember reporting this problem and sending a patch long time ago (for version 0.52). The problem with the keep-alive (if I remember correctly) was that every time dropbear was sending the keep-alive message, it was also resetting the timeout counter... so dropbear or dbclient never detect the dropped connection. Here is an extract from my old email sent on 9/29/2010: Hope this help, Regards, Fabrizio First Issue: When keep-alive messages are sent, they reset the idle timeout counter. (-I counter). I would expect that SENT messages (in particular keep-alive packets) do not affect the idle timeout... This is in function write_packet() (file packet.c) When a message is written, it stores the current time in both the registers for the last packet transmitted *AND* last packet (for the idle timeout): ses.last_trx_packet_time = time(NULL); ses.last_packet_time = time(NULL); (beside that, this cause two system calls, to read the time, when only one would be needed... just optimizing :) ) This is a little unexpected because I would think that the idle timeout works only on received packets, not about sent packets. Basically if I start dropbear with -I and -K options, the idle timeout will never kick in... because the keepalive will always reset the timer even if the connection is dead. I'm proposing to simply remove the line: ses.last_packet_time = time(NULL); So the idle timeout does not get reset when any packet is sent. Watch out: after this change, the semantic of the argument -I is different than before, as it only consider received packets... but at least it makes more sense. Here is a scenario WITHOUT this modification: 1. Start the server with: dropbear -K 15 -I 20 [...] 2. Start the client with dbclient -K 15 [...] 3. On my device, start a program that sends data over one tunneled port Everything works fine, connection is up and data is exchanging. Now... 4. Unplug my embedded device (the one running dbclient) - The server does not detect the connection is down. Any attempt to access a tunneled port cause the caller to hang. now, after this change, with the same scenario, after I unplug my box, the server detects it after 20 seconds and closes the connection. Second Issue: When a keepalive message is received, the idle timeout timer (for received packets) is NOT updated. I'm referring here to the function 'process_packet()' in file 'process-packet.c'. Here the timer update: ses.last_packet_time = time(NULL); is performed AFTER the first switch where we check for SSH_MSG_IGNORE, SSH_MSG_DEBUG, SSH_MSG_UNIMPLEMENTED, and SSH_MSG_DISCONNECT. So, in few words: although a keep-alive message (that is a message of type SSH_MSG_IGNORE) is correctly ignored, but the timer is not reset. Here is what happen: 1. Start my server again with dropbear -I 20 [...] 2. Start my client with dropbear -K 15 [...] (this time I'm not starting my application to send data over a tunneled port) Without doing anything, the server will close the connection after 20 seconds. No matter if the client have sent the keep-alivemessages... After moving that statement: ses.last_packet_time = time(NULL); BEFORE the first switch(), now a keep-alive message cause the idle timer to reset, and the previous test case works as expected (server does't disconnect). So, in conclusion, as you see, these two small changes are critical for my situation, and I believe they could also benefit others with similar needs. On Wed, Mar 27, 2013 at 11:24 AM, Mattias Walström mattias.walst...@westermo.se wrote: Hi! I am running dropbear 2013.56, connecting to the server with a PC but not performing a clean
Re: Timeout dead connections
I think that -K on the server should be enough. On the server can you run tcpdump -i eth0 -w cap1.cap port 22, get a ssh session going, pull out the cable, wait 10 minutes, then send me the capture? Could you also check that the Dropbear process for the connection is still running after the connection should have been finished. It's possible that the process is exiting but the session cleanup code isn't working correctly. The whole debug log might give me an idea what's going on. Cheers, Matt On Thu, Mar 28, 2013 at 09:56:02AM +0100, Mattias Walström wrote: Thanks for your responses, all your suggestions imply that you should do something in the client (set keepalive on client end), but shouldn't the server itself be able to decide if a client is dead (can't OpenSSH do this?). If I do the -K 15 -I 20 on the server end only, this will close the connection when the OpenSSH client has not sent any characters in 20s. I expected the keepalive to be two way, that the server got responses on these packages as well, is that not the case? Regards Mattias On Wed, Mar 27, 2013 at 11:24 AM, Mattias Walström mattias.walst...@westermo.se wrote: Hi! I am running dropbear 2013.56, connecting to the server with a PC but not performing a clean close (I pulled my ethernet cable), this caused dropbear to never drop its connection. Looking at the utmp entries, I could see that the connection never got dropped, the utmp entries was kept forever, and running with debug indicates that also. Tried to use -K to send keepalive, but it just keeps sending keepalives to the peer, even it is no longer there, and not possible to reach. Shouldn't the connection be dropped if the keepalive does not reach its destination? I know there is the -I option, but that does not really do what I want, I want the connection to be tear down when the peer is unreachable, not when the user has been idle for a while. Regards Mattias
Re: Timeout dead connections
Hi, At the very least if there is traffic on the connection (which -K will ensure) then TCP should timeout and the connection should eventually (a minute or so?) close. Can you get a packet capture with tcpdump? Cheers, Matt On Wed, Mar 27, 2013 at 04:24:27PM +0100, Mattias Walström wrote: Hi! I am running dropbear 2013.56, connecting to the server with a PC but not performing a clean close (I pulled my ethernet cable), this caused dropbear to never drop its connection. Looking at the utmp entries, I could see that the connection never got dropped, the utmp entries was kept forever, and running with debug indicates that also. Tried to use -K to send keepalive, but it just keeps sending keepalives to the peer, even it is no longer there, and not possible to reach. Shouldn't the connection be dropped if the keepalive does not reach its destination? I know there is the -I option, but that does not really do what I want, I want the connection to be tear down when the peer is unreachable, not when the user has been idle for a while. Regards Mattias
Re: Timeout dead connections
I remember reporting this problem and sending a patch long time ago (for version 0.52). The problem with the keep-alive (if I remember correctly) was that every time dropbear was sending the keep-alive message, it was also resetting the timeout counter... so dropbear or dbclient never detect the dropped connection. Here is an extract from my old email sent on 9/29/2010: Hope this help, Regards, Fabrizio First Issue: When keep-alive messages are sent, they reset the idle timeout counter. (-I counter). I would expect that SENT messages (in particular keep-alive packets) do not affect the idle timeout... This is in function write_packet() (file packet.c) When a message is written, it stores the current time in both the registers for the last packet transmitted *AND* last packet (for the idle timeout): ses.last_trx_packet_time = time(NULL); ses.last_packet_time = time(NULL); (beside that, this cause two system calls, to read the time, when only one would be needed... just optimizing :) ) This is a little unexpected because I would think that the idle timeout works only on received packets, not about sent packets. Basically if I start dropbear with -I and -K options, the idle timeout will never kick in... because the keepalive will always reset the timer even if the connection is dead. I'm proposing to simply remove the line: ses.last_packet_time = time(NULL); So the idle timeout does not get reset when any packet is sent. Watch out: after this change, the semantic of the argument -I is different than before, as it only consider received packets... but at least it makes more sense. Here is a scenario WITHOUT this modification: 1. Start the server with: dropbear -K 15 -I 20 [...] 2. Start the client with dbclient -K 15 [...] 3. On my device, start a program that sends data over one tunneled port Everything works fine, connection is up and data is exchanging. Now... 4. Unplug my embedded device (the one running dbclient) - The server does not detect the connection is down. Any attempt to access a tunneled port cause the caller to hang. now, after this change, with the same scenario, after I unplug my box, the server detects it after 20 seconds and closes the connection. Second Issue: When a keepalive message is received, the idle timeout timer (for received packets) is NOT updated. I'm referring here to the function 'process_packet()' in file 'process-packet.c'. Here the timer update: ses.last_packet_time = time(NULL); is performed AFTER the first switch where we check for SSH_MSG_IGNORE, SSH_MSG_DEBUG, SSH_MSG_UNIMPLEMENTED, and SSH_MSG_DISCONNECT. So, in few words: although a keep-alive message (that is a message of type SSH_MSG_IGNORE) is correctly ignored, but the timer is not reset. Here is what happen: 1. Start my server again with dropbear -I 20 [...] 2. Start my client with dropbear -K 15 [...] (this time I'm not starting my application to send data over a tunneled port) Without doing anything, the server will close the connection after 20 seconds. No matter if the client have sent the keep-alivemessages... After moving that statement: ses.last_packet_time = time(NULL); BEFORE the first switch(), now a keep-alive message cause the idle timer to reset, and the previous test case works as expected (server does't disconnect). So, in conclusion, as you see, these two small changes are critical for my situation, and I believe they could also benefit others with similar needs. On Wed, Mar 27, 2013 at 11:24 AM, Mattias Walström mattias.walst...@westermo.se wrote: Hi! I am running dropbear 2013.56, connecting to the server with a PC but not performing a clean close (I pulled my ethernet cable), this caused dropbear to never drop its connection. Looking at the utmp entries, I could see that the connection never got dropped, the utmp entries was kept forever, and running with debug indicates that also. Tried to use -K to send keepalive, but it just keeps sending keepalives to the peer, even it is no longer there, and not possible to reach. Shouldn't the connection be dropped if the keepalive does not reach its destination? I know there is the -I option, but that does not really do what I want, I want the connection to be tear down when the peer is unreachable, not when the user has been idle for a while. Regards Mattias
Re: Timeout dead connections
I thought those were fixed in 0.53 or perhaps 2011.54: 2011.54 - Tuesday 8 November 2011 - Fixed case where -K 1 keepalive for dbclient would cause a SSH_MSG_IGNORE packet to be sent 0.53 - Thurs 24 February 2011 - Make -K (keepalive) and -I (idle timeout) work together sensibly in the client. The idle timeout is no longer reset by SSH_MSG_IGNORE packets. If the network cable has been pulled out, shouldn't the OS send a TCP RST packet eventually after some traffic and close the connection? Cheers, Matt On Wed, Mar 27, 2013 at 11:41:40AM -0400, Fabrizio Bertocci wrote: I remember reporting this problem and sending a patch long time ago (for version 0.52). The problem with the keep-alive (if I remember correctly) was that every time dropbear was sending the keep-alive message, it was also resetting the timeout counter... so dropbear or dbclient never detect the dropped connection. Here is an extract from my old email sent on 9/29/2010: Hope this help, Regards, Fabrizio First Issue: When keep-alive messages are sent, they reset the idle timeout counter. (-I counter). I would expect that SENT messages (in particular keep-alive packets) do not affect the idle timeout... This is in function write_packet() (file packet.c) When a message is written, it stores the current time in both the registers for the last packet transmitted *AND* last packet (for the idle timeout): ses.last_trx_packet_time = time(NULL); ses.last_packet_time = time(NULL); (beside that, this cause two system calls, to read the time, when only one would be needed... just optimizing :) ) This is a little unexpected because I would think that the idle timeout works only on received packets, not about sent packets. Basically if I start dropbear with -I and -K options, the idle timeout will never kick in... because the keepalive will always reset the timer even if the connection is dead. I'm proposing to simply remove the line: ses.last_packet_time = time(NULL); So the idle timeout does not get reset when any packet is sent. Watch out: after this change, the semantic of the argument -I is different than before, as it only consider received packets... but at least it makes more sense. Here is a scenario WITHOUT this modification: 1. Start the server with: dropbear -K 15 -I 20 [...] 2. Start the client with dbclient -K 15 [...] 3. On my device, start a program that sends data over one tunneled port Everything works fine, connection is up and data is exchanging. Now... 4. Unplug my embedded device (the one running dbclient) - The server does not detect the connection is down. Any attempt to access a tunneled port cause the caller to hang. now, after this change, with the same scenario, after I unplug my box, the server detects it after 20 seconds and closes the connection. Second Issue: When a keepalive message is received, the idle timeout timer (for received packets) is NOT updated. I'm referring here to the function 'process_packet()' in file 'process-packet.c'. Here the timer update: ses.last_packet_time = time(NULL); is performed AFTER the first switch where we check for SSH_MSG_IGNORE, SSH_MSG_DEBUG, SSH_MSG_UNIMPLEMENTED, and SSH_MSG_DISCONNECT. So, in few words: although a keep-alive message (that is a message of type SSH_MSG_IGNORE) is correctly ignored, but the timer is not reset. Here is what happen: 1. Start my server again with dropbear -I 20 [...] 2. Start my client with dropbear -K 15 [...] (this time I'm not starting my application to send data over a tunneled port) Without doing anything, the server will close the connection after 20 seconds. No matter if the client have sent the keep-alivemessages... After moving that statement: ses.last_packet_time = time(NULL); BEFORE the first switch(), now a keep-alive message cause the idle timer to reset, and the previous test case works as expected (server does't disconnect). So, in conclusion, as you see, these two small changes are critical for my situation, and I believe they could also benefit others with similar needs. On Wed, Mar 27, 2013 at 11:24 AM, Mattias Walström mattias.walst...@westermo.se wrote: Hi! I am running dropbear 2013.56, connecting to the server with a PC but not performing a clean close (I pulled my ethernet cable), this caused dropbear to never drop its connection. Looking at the utmp entries, I could see that the connection never got dropped, the utmp entries was kept forever, and running with debug indicates that also. Tried to use -K to send keepalive, but it just keeps sending keepalives to the peer, even it is no longer there, and not possible to reach. Shouldn't the connection be dropped if the keepalive
Re: Timeout dead connections
Yep, you're right Matt... the latest version contains those fixes... (the truth is that I'm still working with my patched 0.52 that is rock solid for my usage)... Regards, Fabrizio On Wed, Mar 27, 2013 at 11:47 AM, Matt Johnston m...@ucc.asn.au wrote: I thought those were fixed in 0.53 or perhaps 2011.54: 2011.54 - Tuesday 8 November 2011 - Fixed case where -K 1 keepalive for dbclient would cause a SSH_MSG_IGNORE packet to be sent 0.53 - Thurs 24 February 2011 - Make -K (keepalive) and -I (idle timeout) work together sensibly in the client. The idle timeout is no longer reset by SSH_MSG_IGNORE packets. If the network cable has been pulled out, shouldn't the OS send a TCP RST packet eventually after some traffic and close the connection? Cheers, Matt On Wed, Mar 27, 2013 at 11:41:40AM -0400, Fabrizio Bertocci wrote: I remember reporting this problem and sending a patch long time ago (for version 0.52). The problem with the keep-alive (if I remember correctly) was that every time dropbear was sending the keep-alive message, it was also resetting the timeout counter... so dropbear or dbclient never detect the dropped connection. Here is an extract from my old email sent on 9/29/2010: Hope this help, Regards, Fabrizio First Issue: When keep-alive messages are sent, they reset the idle timeout counter. (-I counter). I would expect that SENT messages (in particular keep-alive packets) do not affect the idle timeout... This is in function write_packet() (file packet.c) When a message is written, it stores the current time in both the registers for the last packet transmitted *AND* last packet (for the idle timeout): ses.last_trx_packet_time = time(NULL); ses.last_packet_time = time(NULL); (beside that, this cause two system calls, to read the time, when only one would be needed... just optimizing :) ) This is a little unexpected because I would think that the idle timeout works only on received packets, not about sent packets. Basically if I start dropbear with -I and -K options, the idle timeout will never kick in... because the keepalive will always reset the timer even if the connection is dead. I'm proposing to simply remove the line: ses.last_packet_time = time(NULL); So the idle timeout does not get reset when any packet is sent. Watch out: after this change, the semantic of the argument -I is different than before, as it only consider received packets... but at least it makes more sense. Here is a scenario WITHOUT this modification: 1. Start the server with: dropbear -K 15 -I 20 [...] 2. Start the client with dbclient -K 15 [...] 3. On my device, start a program that sends data over one tunneled port Everything works fine, connection is up and data is exchanging. Now... 4. Unplug my embedded device (the one running dbclient) - The server does not detect the connection is down. Any attempt to access a tunneled port cause the caller to hang. now, after this change, with the same scenario, after I unplug my box, the server detects it after 20 seconds and closes the connection. Second Issue: When a keepalive message is received, the idle timeout timer (for received packets) is NOT updated. I'm referring here to the function 'process_packet()' in file 'process-packet.c'. Here the timer update: ses.last_packet_time = time(NULL); is performed AFTER the first switch where we check for SSH_MSG_IGNORE, SSH_MSG_DEBUG, SSH_MSG_UNIMPLEMENTED, and SSH_MSG_DISCONNECT. So, in few words: although a keep-alive message (that is a message of type SSH_MSG_IGNORE) is correctly ignored, but the timer is not reset. Here is what happen: 1. Start my server again with dropbear -I 20 [...] 2. Start my client with dropbear -K 15 [...] (this time I'm not starting my application to send data over a tunneled port) Without doing anything, the server will close the connection after 20 seconds. No matter if the client have sent the keep-alivemessages... After moving that statement: ses.last_packet_time = time(NULL); BEFORE the first switch(), now a keep-alive message cause the idle timer to reset, and the previous test case works as expected (server does't disconnect). So, in conclusion, as you see, these two small changes are critical for my situation, and I believe they could also benefit others with similar needs. On Wed, Mar 27, 2013 at 11:24 AM, Mattias Walström mattias.walst...@westermo.se wrote: Hi! I am running dropbear 2013.56, connecting to the server with a PC but not performing a clean close (I pulled my ethernet cable), this caused dropbear to never drop its connection. Looking
