Timeout dead connections
Hi! I am running dropbear 2013.56, connecting to the server with a PC but not performing a clean close (I pulled my ethernet cable), this caused dropbear to never drop its connection. Looking at the utmp entries, I could see that the connection never got dropped, the utmp entries was kept forever, and running with debug indicates that also. Tried to use -K to send keepalive, but it just keeps sending keepalives to the peer, even it is no longer there, and not possible to reach. Shouldn't the connection be dropped if the keepalive does not reach its destination? I know there is the -I option, but that does not really do what I want, I want the connection to be tear down when the peer is unreachable, not when the user has been idle for a while. Regards Mattias
Re: Timeout dead connections
Hi, At the very least if there is traffic on the connection (which -K will ensure) then TCP should timeout and the connection should eventually (a minute or so?) close. Can you get a packet capture with tcpdump? Cheers, Matt On Wed, Mar 27, 2013 at 04:24:27PM +0100, Mattias Walström wrote: Hi! I am running dropbear 2013.56, connecting to the server with a PC but not performing a clean close (I pulled my ethernet cable), this caused dropbear to never drop its connection. Looking at the utmp entries, I could see that the connection never got dropped, the utmp entries was kept forever, and running with debug indicates that also. Tried to use -K to send keepalive, but it just keeps sending keepalives to the peer, even it is no longer there, and not possible to reach. Shouldn't the connection be dropped if the keepalive does not reach its destination? I know there is the -I option, but that does not really do what I want, I want the connection to be tear down when the peer is unreachable, not when the user has been idle for a while. Regards Mattias
Re: Timeout dead connections
I remember reporting this problem and sending a patch long time ago (for version 0.52). The problem with the keep-alive (if I remember correctly) was that every time dropbear was sending the keep-alive message, it was also resetting the timeout counter... so dropbear or dbclient never detect the dropped connection. Here is an extract from my old email sent on 9/29/2010: Hope this help, Regards, Fabrizio First Issue: When keep-alive messages are sent, they reset the idle timeout counter. (-I counter). I would expect that SENT messages (in particular keep-alive packets) do not affect the idle timeout... This is in function write_packet() (file packet.c) When a message is written, it stores the current time in both the registers for the last packet transmitted *AND* last packet (for the idle timeout): ses.last_trx_packet_time = time(NULL); ses.last_packet_time = time(NULL); (beside that, this cause two system calls, to read the time, when only one would be needed... just optimizing :) ) This is a little unexpected because I would think that the idle timeout works only on received packets, not about sent packets. Basically if I start dropbear with -I and -K options, the idle timeout will never kick in... because the keepalive will always reset the timer even if the connection is dead. I'm proposing to simply remove the line: ses.last_packet_time = time(NULL); So the idle timeout does not get reset when any packet is sent. Watch out: after this change, the semantic of the argument -I is different than before, as it only consider received packets... but at least it makes more sense. Here is a scenario WITHOUT this modification: 1. Start the server with: dropbear -K 15 -I 20 [...] 2. Start the client with dbclient -K 15 [...] 3. On my device, start a program that sends data over one tunneled port Everything works fine, connection is up and data is exchanging. Now... 4. Unplug my embedded device (the one running dbclient) - The server does not detect the connection is down. Any attempt to access a tunneled port cause the caller to hang. now, after this change, with the same scenario, after I unplug my box, the server detects it after 20 seconds and closes the connection. Second Issue: When a keepalive message is received, the idle timeout timer (for received packets) is NOT updated. I'm referring here to the function 'process_packet()' in file 'process-packet.c'. Here the timer update: ses.last_packet_time = time(NULL); is performed AFTER the first switch where we check for SSH_MSG_IGNORE, SSH_MSG_DEBUG, SSH_MSG_UNIMPLEMENTED, and SSH_MSG_DISCONNECT. So, in few words: although a keep-alive message (that is a message of type SSH_MSG_IGNORE) is correctly ignored, but the timer is not reset. Here is what happen: 1. Start my server again with dropbear -I 20 [...] 2. Start my client with dropbear -K 15 [...] (this time I'm not starting my application to send data over a tunneled port) Without doing anything, the server will close the connection after 20 seconds. No matter if the client have sent the keep-alivemessages... After moving that statement: ses.last_packet_time = time(NULL); BEFORE the first switch(), now a keep-alive message cause the idle timer to reset, and the previous test case works as expected (server does't disconnect). So, in conclusion, as you see, these two small changes are critical for my situation, and I believe they could also benefit others with similar needs. On Wed, Mar 27, 2013 at 11:24 AM, Mattias Walström mattias.walst...@westermo.se wrote: Hi! I am running dropbear 2013.56, connecting to the server with a PC but not performing a clean close (I pulled my ethernet cable), this caused dropbear to never drop its connection. Looking at the utmp entries, I could see that the connection never got dropped, the utmp entries was kept forever, and running with debug indicates that also. Tried to use -K to send keepalive, but it just keeps sending keepalives to the peer, even it is no longer there, and not possible to reach. Shouldn't the connection be dropped if the keepalive does not reach its destination? I know there is the -I option, but that does not really do what I want, I want the connection to be tear down when the peer is unreachable, not when the user has been idle for a while. Regards Mattias
Re: Timeout dead connections
I thought those were fixed in 0.53 or perhaps 2011.54: 2011.54 - Tuesday 8 November 2011 - Fixed case where -K 1 keepalive for dbclient would cause a SSH_MSG_IGNORE packet to be sent 0.53 - Thurs 24 February 2011 - Make -K (keepalive) and -I (idle timeout) work together sensibly in the client. The idle timeout is no longer reset by SSH_MSG_IGNORE packets. If the network cable has been pulled out, shouldn't the OS send a TCP RST packet eventually after some traffic and close the connection? Cheers, Matt On Wed, Mar 27, 2013 at 11:41:40AM -0400, Fabrizio Bertocci wrote: I remember reporting this problem and sending a patch long time ago (for version 0.52). The problem with the keep-alive (if I remember correctly) was that every time dropbear was sending the keep-alive message, it was also resetting the timeout counter... so dropbear or dbclient never detect the dropped connection. Here is an extract from my old email sent on 9/29/2010: Hope this help, Regards, Fabrizio First Issue: When keep-alive messages are sent, they reset the idle timeout counter. (-I counter). I would expect that SENT messages (in particular keep-alive packets) do not affect the idle timeout... This is in function write_packet() (file packet.c) When a message is written, it stores the current time in both the registers for the last packet transmitted *AND* last packet (for the idle timeout): ses.last_trx_packet_time = time(NULL); ses.last_packet_time = time(NULL); (beside that, this cause two system calls, to read the time, when only one would be needed... just optimizing :) ) This is a little unexpected because I would think that the idle timeout works only on received packets, not about sent packets. Basically if I start dropbear with -I and -K options, the idle timeout will never kick in... because the keepalive will always reset the timer even if the connection is dead. I'm proposing to simply remove the line: ses.last_packet_time = time(NULL); So the idle timeout does not get reset when any packet is sent. Watch out: after this change, the semantic of the argument -I is different than before, as it only consider received packets... but at least it makes more sense. Here is a scenario WITHOUT this modification: 1. Start the server with: dropbear -K 15 -I 20 [...] 2. Start the client with dbclient -K 15 [...] 3. On my device, start a program that sends data over one tunneled port Everything works fine, connection is up and data is exchanging. Now... 4. Unplug my embedded device (the one running dbclient) - The server does not detect the connection is down. Any attempt to access a tunneled port cause the caller to hang. now, after this change, with the same scenario, after I unplug my box, the server detects it after 20 seconds and closes the connection. Second Issue: When a keepalive message is received, the idle timeout timer (for received packets) is NOT updated. I'm referring here to the function 'process_packet()' in file 'process-packet.c'. Here the timer update: ses.last_packet_time = time(NULL); is performed AFTER the first switch where we check for SSH_MSG_IGNORE, SSH_MSG_DEBUG, SSH_MSG_UNIMPLEMENTED, and SSH_MSG_DISCONNECT. So, in few words: although a keep-alive message (that is a message of type SSH_MSG_IGNORE) is correctly ignored, but the timer is not reset. Here is what happen: 1. Start my server again with dropbear -I 20 [...] 2. Start my client with dropbear -K 15 [...] (this time I'm not starting my application to send data over a tunneled port) Without doing anything, the server will close the connection after 20 seconds. No matter if the client have sent the keep-alivemessages... After moving that statement: ses.last_packet_time = time(NULL); BEFORE the first switch(), now a keep-alive message cause the idle timer to reset, and the previous test case works as expected (server does't disconnect). So, in conclusion, as you see, these two small changes are critical for my situation, and I believe they could also benefit others with similar needs. On Wed, Mar 27, 2013 at 11:24 AM, Mattias Walström mattias.walst...@westermo.se wrote: Hi! I am running dropbear 2013.56, connecting to the server with a PC but not performing a clean close (I pulled my ethernet cable), this caused dropbear to never drop its connection. Looking at the utmp entries, I could see that the connection never got dropped, the utmp entries was kept forever, and running with debug indicates that also. Tried to use -K to send keepalive, but it just keeps sending keepalives to the peer, even it is no longer there, and not possible to reach. Shouldn't the connection be dropped if the keepalive
Re: Timeout dead connections
Yep, you're right Matt... the latest version contains those fixes... (the truth is that I'm still working with my patched 0.52 that is rock solid for my usage)... Regards, Fabrizio On Wed, Mar 27, 2013 at 11:47 AM, Matt Johnston m...@ucc.asn.au wrote: I thought those were fixed in 0.53 or perhaps 2011.54: 2011.54 - Tuesday 8 November 2011 - Fixed case where -K 1 keepalive for dbclient would cause a SSH_MSG_IGNORE packet to be sent 0.53 - Thurs 24 February 2011 - Make -K (keepalive) and -I (idle timeout) work together sensibly in the client. The idle timeout is no longer reset by SSH_MSG_IGNORE packets. If the network cable has been pulled out, shouldn't the OS send a TCP RST packet eventually after some traffic and close the connection? Cheers, Matt On Wed, Mar 27, 2013 at 11:41:40AM -0400, Fabrizio Bertocci wrote: I remember reporting this problem and sending a patch long time ago (for version 0.52). The problem with the keep-alive (if I remember correctly) was that every time dropbear was sending the keep-alive message, it was also resetting the timeout counter... so dropbear or dbclient never detect the dropped connection. Here is an extract from my old email sent on 9/29/2010: Hope this help, Regards, Fabrizio First Issue: When keep-alive messages are sent, they reset the idle timeout counter. (-I counter). I would expect that SENT messages (in particular keep-alive packets) do not affect the idle timeout... This is in function write_packet() (file packet.c) When a message is written, it stores the current time in both the registers for the last packet transmitted *AND* last packet (for the idle timeout): ses.last_trx_packet_time = time(NULL); ses.last_packet_time = time(NULL); (beside that, this cause two system calls, to read the time, when only one would be needed... just optimizing :) ) This is a little unexpected because I would think that the idle timeout works only on received packets, not about sent packets. Basically if I start dropbear with -I and -K options, the idle timeout will never kick in... because the keepalive will always reset the timer even if the connection is dead. I'm proposing to simply remove the line: ses.last_packet_time = time(NULL); So the idle timeout does not get reset when any packet is sent. Watch out: after this change, the semantic of the argument -I is different than before, as it only consider received packets... but at least it makes more sense. Here is a scenario WITHOUT this modification: 1. Start the server with: dropbear -K 15 -I 20 [...] 2. Start the client with dbclient -K 15 [...] 3. On my device, start a program that sends data over one tunneled port Everything works fine, connection is up and data is exchanging. Now... 4. Unplug my embedded device (the one running dbclient) - The server does not detect the connection is down. Any attempt to access a tunneled port cause the caller to hang. now, after this change, with the same scenario, after I unplug my box, the server detects it after 20 seconds and closes the connection. Second Issue: When a keepalive message is received, the idle timeout timer (for received packets) is NOT updated. I'm referring here to the function 'process_packet()' in file 'process-packet.c'. Here the timer update: ses.last_packet_time = time(NULL); is performed AFTER the first switch where we check for SSH_MSG_IGNORE, SSH_MSG_DEBUG, SSH_MSG_UNIMPLEMENTED, and SSH_MSG_DISCONNECT. So, in few words: although a keep-alive message (that is a message of type SSH_MSG_IGNORE) is correctly ignored, but the timer is not reset. Here is what happen: 1. Start my server again with dropbear -I 20 [...] 2. Start my client with dropbear -K 15 [...] (this time I'm not starting my application to send data over a tunneled port) Without doing anything, the server will close the connection after 20 seconds. No matter if the client have sent the keep-alivemessages... After moving that statement: ses.last_packet_time = time(NULL); BEFORE the first switch(), now a keep-alive message cause the idle timer to reset, and the previous test case works as expected (server does't disconnect). So, in conclusion, as you see, these two small changes are critical for my situation, and I believe they could also benefit others with similar needs. On Wed, Mar 27, 2013 at 11:24 AM, Mattias Walström mattias.walst...@westermo.se wrote: Hi! I am running dropbear 2013.56, connecting to the server with a PC but not performing a clean close (I pulled my ethernet cable), this caused dropbear to never drop its connection. Looking
