Mutt hangs, Network Issue?
Hello, I have a strange problem that is driving me nuts. Sorry if this is long. Background: I am a long time mutt user. I use it at home. I use it at work. I get a lot of email. I love mutt. The problem is at work. I routinely interact with several servers (Linux and *BSD). I keep terminals with ssh sessions open to all these. On one of these, I keep my personal mail spool (mbox). This is a CentOS Linux server. The mutt version is mutt-1.4.1-12.0.3.el4. So I connect from my Ubuntu Desktop via ssh to the server, and mutt runs locally on the server. I do not use mutt for retrieving mail, just reading mail from the spool. This Mutt hangs/freezes probably 10-15 times a day. It is completely unresponsive. If I open a second ssh session, and kill the mutt process, it generally takes maybe 30 seconds for mutt to let go, and to get a shell prompt back. If I don't manually kill mutt, and just let nature take its course, the ssh session itself will eventually time out and die, and I am dropped back to my local desktop. The aggravation is killing me! I have approximately 12 ssh session opens now, and three to this same server where I use mutt. Many of these stay open for weeks, and sometimes months without issue. The only one that is problematic is the one with mutt. And only when mutt is running. The other ssh sessions to this same server are 100% problem free. This problem only started after we changed our local office router that does NAT, DHCP, packet filtering, etc a few months ago. Other than this one issue, the router is problem free, with approx 20 users connected to it, using it all day long. My suspicion is that there is some background network related stuff that mutt is doing, that is breaking statefulness on the router, and the network connection just hangs. But that is a wild guess. It may turn out to be a network issue, but if so, mutt is the only thing triggering it. I am at a loss of how to troubleshoot this. Any ideas would be hugely appreciated! -- Hal Burgiss
Re: Mutt hangs, Network Issue?
=- Hal Burgiss wrote on Tue 20.May'08 at 9:30:27 -0400 -= The problem is at work. I routinely interact with several servers (Linux and *BSD). I keep terminals with ssh sessions open to all these. On one of these, I keep my personal mail spool (mbox). This is a CentOS Linux server. The mutt version is mutt-1.4.1-12.0.3.el4. {...} This Mutt hangs/freezes probably 10-15 times a day. It is completely unresponsive. If I open a second ssh session, and kill the mutt process, it generally takes maybe 30 seconds for mutt to let go, and to get a shell prompt back. a) try 1.5.18 b) I only know of stale NFS handles freezing mutt beyond recovery. Try with local filesys folder open in mutt all day. If it doesn't hang, it's NFS. -- © Rado S. -- You must provide YOUR effort for your goal! EVERY effort counts: at least to show your attitude. You're responsible for ALL you do: you get what you give.
Re: Mutt hangs, Network Issue?
On Tue, May 20, 2008 at 05:45:47PM +0200, Rado S wrote: a) try 1.5.18 OK, good idea. I am running that now, and should know within an hour or two. b) I only know of stale NFS handles freezing mutt beyond recovery. Try with local filesys folder open in mutt all day. If it doesn't hang, it's NFS. I don't think thats it. There are no nfs mounts involved here. Thanks! -- Hal Burgiss
Re: Mutt hangs, Network Issue?
On Tue, May 20, 2008 at 12:13:34PM -0400, Hal Burgiss wrote: On Tue, May 20, 2008 at 05:45:47PM +0200, Rado S wrote: a) try 1.5.18 OK, good idea. I am running that now, and should know within an hour or two. No go. Still hanging. :( -- Hal Burgiss
Re: Mutt hangs, Network Issue?
This Mutt hangs/freezes probably 10-15 times a day. It is completely unresponsive. If I open a second ssh session, and kill the mutt process, it generally takes maybe 30 seconds for mutt to let go, and to get a shell prompt back. If I don't manually kill mutt, and just So you regain control of mutt's session when mutt dies to a kill signal? Can you also suspend mutt (control-Z) from within the same session, or only from outside? Does mutt freeze while you're actively using it, or only while you're idling? Or both? Is it at all possible that the freezes are related to software flow control signals (normally control-S = stop; control-Q = start; but see `stty`)? This seems unlikely since you said the problem started with the installation of a NATting router, but it's sometimes worth asking. -- -D.[EMAIL PROTECTED]NSITUniversity of Chicago
Re: Mutt hangs, Network Issue?
On Tue, May 20, 2008 at 12:06:28PM -0500, David Champion wrote: So you regain control of mutt's session when mutt dies to a kill signal? Yes. Sometimes quickly. Sometimes after more than 30 seconds or so. Can you also suspend mutt (control-Z) from within the same session, or only from outside? I have not tried suspending. I will next time. Does mutt freeze while you're actively using it, or only while you're idling? Or both? Idling only. And only on the index screen. I use vim as both pager and composer, so maybe that means something. In fact, I have accidently left a partially composed mail open for many hours, and mutt is always functional once I close out the vim/compose aspect. Is it at all possible that the freezes are related to software flow control signals (normally control-S = stop; control-Q = start; but see `stty`)? This seems unlikely since you said the problem started with the installation of a NATting router, but it's sometimes worth asking. I wouldn't think so either. I guess I could dig into that a little more. This is the same system (hardware + software) that worked flawlessly until the new router. The router thing might be co-incidence and a red-herring, but that is when the problem started. Thanks. -- Hal Burgiss
Re: Mutt hangs, Network Issue?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Tuesday, May 20 at 09:30 AM, quoth Hal Burgiss: This Mutt hangs/freezes probably 10-15 times a day. It is completely unresponsive. Try running mutt in debug mode; with luck, we can see what mutt's doing when it dies. ~Kyle - -- He who dares not offend cannot be honest. -- Thomas Paine -BEGIN PGP SIGNATURE- Comment: Thank you for using encryption! iEYEARECAAYFAkgzCd0ACgkQBkIOoMqOI166EgCdFMLsoZA5C17sBDyMqrVVlnt8 4J0AoLZq36WJBS7BnIXNyBljvINQUsZ9 =ptC8 -END PGP SIGNATURE-
Re: Mutt hangs, Network Issue?
On Tue, May 20, 2008 at 12:26:54PM -0500, Kyle Wheeler wrote: Try running mutt in debug mode; with luck, we can see what mutt's doing when it dies. Thanks! I wasn't aware of debug mode, but I'm game. Rebuilt now and running -d 5 now. -- Hal Burgiss DBSInteractive Technical Services
Re: Mutt hangs, Network Issue?
Try enabling SSH keepalives.
Re: Mutt hangs, Network Issue?
On Tue, May 20, 2008 at 06:01:10PM +, Jussi Peltola wrote: Try enabling SSH keepalives. Thanks. I have keepalives enabled on the server end already. -- Hal Burgiss DBSInteractive Technical Services
Re: Mutt hangs, Network Issue?
On Tue, May 20, 2008 at 12:26:54PM -0500, Kyle Wheeler wrote: Try running mutt in debug mode; with luck, we can see what mutt's doing when it dies. All I get is ... Caught signal 15... Exiting. That is from killing the mutt process after its hung. -- Hal Burgiss DBSInteractive Technical Services
Re: Mutt hangs, Network Issue?
On Tue, May 20, 2008 at 12:06:28PM -0500, David Champion wrote: So you regain control of mutt's session when mutt dies to a kill signal? Can you also suspend mutt (control-Z) from within the same session, or only from outside? I've had two opportunites to try control-z. Both times it worked, though there was a delay. On the second attempt, I had tried Control-c a couple of times first, and its possible there was a delayed thing going on. What should a functional control-z tell me? -- Hal Burgiss DBSInteractive Technical Services
Re: Mutt hangs, Network Issue?
On Tue, May 20, 2008 at 01:46:59PM -0700, Gary Johnson wrote: The debug output is sent to a file, not to your terminal. The latest debug file will have the name ~/.muttdebug0. Ach! Thanks. Wasn't expecting that. Now have 5 different ones, and its a little hard to see where the problem might have been. I'll know better next time. -- Hal Burgiss
Re: Mutt hangs, Network Issue?
On 2008-05-20, Hal Burgiss [EMAIL PROTECTED] wrote: On Tue, May 20, 2008 at 01:46:59PM -0700, Gary Johnson wrote: The debug output is sent to a file, not to your terminal. The latest debug file will have the name ~/.muttdebug0. Ach! Thanks. Wasn't expecting that. Now have 5 different ones, and its a little hard to see where the problem might have been. I'll know better next time. I should have mentioned this before, but something you can do to help associate the contents of the debug file with mutt's actions is to open another terminal and in it run tail -f ~/.muttdebug0 Regards, Gary
Re: Mutt hangs, Network Issue?
On Tue, May 20, 2008 at 02:05:28PM -0700, Gary Johnson wrote: I should have mentioned this before, but something you can do to help associate the contents of the debug file with mutt's actions is to open another terminal and in it run tail -f ~/.muttdebug0 Excellent! Its going now. Thanks. -- Hal Burgiss DBSInteractive Technical Services
Re: Mutt hangs, Network Issue?
What should a functional control-z tell me? If control-Z works, your transport (ssh) and terminal are still fundamentally intact, and responsive to low-level traffic and out-of-band signals. (It means that mutt is responsive to signals too, for that matter -- once it receives them.) Does mutt work again once you bring it back to foreground, or does it remain hung? If it hangs, and you type a bunch of stuff followed by control-Z, and then foreground mutt, does all the stuff you typed suddenly appear to execute at once? Does it appear to have already executed long ago? Or is it ignored? This all serves to indicate what stage things are at when mutt appears to be unresponsive. You might also check the debug log during a suspend instead of after a kill; this could help pinpoint more exactly what mutt thought it was up to. If you have gdb installed and you're familiar with it, you might try attaching gdb to the hung mutt process from within another terminal. That is: window 1window 2 -- -- + ssh in- ssh in + run mutt - get mutt's process id from ps for example: 14001 + do stuff, wait for a hang - attach gdb to the mutt process for example: gdb /usr/bin/mutt 14001 - get a backtrace, minimally type: backtrace This will tell you more finely than debug logs will what mutt is doing at an exact moment that it appears hung, but it's quite possible that it's just waiting for input that's simply not arriving. The logs will tell what it *has* been doing up until that, but it might not be anything special. Honestly this has the feel of an ordinary session timeout at the network level, which wouldn't be surprising. But you said that it happens only with mutt, and not even with an idle shell. That's what seems strange to me. -- -D.[EMAIL PROTECTED]NSITUniversity of Chicago
Re: Mutt hangs, Network Issue?
On Tue, May 20, 2008 at 05:09:07PM -0500, David Champion wrote: What should a functional control-z tell me? If control-Z works, your transport (ssh) and terminal are still fundamentally intact, and responsive to low-level traffic and out-of-band signals. (It means that mutt is responsive to signals too, for that matter -- once it receives them.) Does mutt work again once you bring it back to foreground, or does it remain hung? If it hangs, and you type a bunch of stuff followed by control-Z, and then foreground mutt, does all the stuff you typed suddenly appear to execute at once? Does it appear to have already executed long ago? Or is it ignored? I'll have to dig more into this later ... The two occasions I've done this, there was some delay with mutt responding. But once it did, it acted perfectly normal. There have been times when mutt doesn't respond, I'll hit a few keys, and after a while maybe it comes back to life, and maybe it doesn't. But, because I'm hugely impatient, I typically just kill the mutt process, and wait for that to bring back a prompt. BTW, I'm at home now. Looks like mutt died. The ssh session is still up. The last 30 or 40 lines of the debug log are like mutt_index_menu[613]: Got op -1 mutt_index_menu[613]: Got op -1 mutt_index_menu[613]: Got op -1 mutt_index_menu[613]: Got op -1 mutt_index_menu[613]: Got op -1 mutt_index_menu[613]: Got op -1 mutt_index_menu[613]: Got op -1 It just occurred to me, that when I ssh in from home, I don't seem to have this problem. Obviously same version of mutt. Same configs. But I tend not to leave the home sessions open all the time so a little hard to tell. at an exact moment that it appears hung, but it's quite possible that it's just waiting for input that's simply not arriving. The logs will tell what it *has* been doing up until that, but it might not be anything special. Thanks, I may get into this more. Honestly this has the feel of an ordinary session timeout at the network level, which wouldn't be surprising. But you said that it happens only with mutt, and not even with an idle shell. That's what seems strange to me. My feeling all along is that it is more network related than anything. But mutt is somehow triggering something screwy. I am still convinced the router change is one of the pieces of the puzzle. And I never have a problem with other terminals that are opened to the same server. Thanks for the help! -- Hal Burgiss