[OT] Broken sendmail

[Dave Cross]

I've just worked at that at some point on Saturday 22nd March,
my Linux box decided to stop sending emails out.

Emails that I sent to mailing lists on Fri 21st March arrived.
Emails that I send to mailing lists on Sat 22nd March (and subsequently)
didn't.

On 22nd March I installed some new rpms via Redhat's up2date
tool. These were:

nscd-2.3.2-4.80 Sat 22 Mar 2003 03:09:43 PM GMT
glibc-profile-2.3.2-4.80 Sat 22 Mar 2003 03:09:41 PM GMT
glibc-devel-2.3.2-4.80 Sat 22 Mar 2003 10:20:48 AM GMT
glibc-common-2.3.2-4.80 Sat 22 Mar 2003 10:19:28 AM GMT
glibc-2.3.2-4.80 Sat 22 Mar 2003 10:18:33 AM GMT
evolution-1.0.8-11 Sat 22 Mar 2003 10:17:28 AM GMT

But I can't really see how any of those would effect mail (I
use mutt, not Evolution for mail).

If someone could give me a step-by-step guide to debugging email
delivery in RH 8.0 then I'd be very grateful.

Thanks,

Dave...
-- 


"Let me see you make decisions, without your television"
   - Depeche Mode (Stripped)

Re: [OT] Broken sendmail

[Nick Woolley]

I can't tell you what's wrong with your system, or give you a guide to 
debugging email delivery, but I suspect there's something fishy about recent 
versions of glibc, so that *may* be your problem.

I also tried installing evolution recently (the exact version escapes me), 
from Mandrake Cooker using urpmi.  It upgraded my glibc from glibc-2.3.1-6mdk 
to glibc-devel-2.3.1-10mdk.  I subsequentially discovered that I had broken 
MySQL-3.23.52-1.3mdk (it segfaulted on startup), and rpm-4.0.4-19mdk (it did 
too, when I tried to do anything nontrivial, like downgrade my glibc back to 
the original version). 

I don't use mutt, can't comment on that.

After some hair-pulling I finally fixed the problem by uninstalling evolution, 
booting from the Mandrake 9.0 rescue CD and downgrading glibc back to 2.3.1-6 
with the rescue disk's rpm.

More details here:

http://www.mandrakeclub.com/modules.php?op=modload&name=Splatt_Forum&file=viewtopic&topic=5037&forum=11

Nick

Re: [OT] Broken sendmail

[Dirk Koopman]

What MTA?

Did you reboot the machine?

Bear in mind that most MTAs fork to handle some of the process, if any
of the libraries that is currently using have disappeared (they have
been erased and replacements with different inode nos added), then forks
will fail.

I have done more or less exactly what you did, rebooted, it just works.
Mind you I run exim with rpms that I maintain.

On Tue, 2003-04-01 at 23:32, Dave Cross wrote:
> I've just worked at that at some point on Saturday 22nd March,
> my Linux box decided to stop sending emails out.
> 
> Emails that I sent to mailing lists on Fri 21st March arrived.
> Emails that I send to mailing lists on Sat 22nd March (and subsequently)
> didn't.
> 
> On 22nd March I installed some new rpms via Redhat's up2date
> tool. These were:
> 
> nscd-2.3.2-4.80 Sat 22 Mar 2003 03:09:43 PM GMT
> glibc-profile-2.3.2-4.80 Sat 22 Mar 2003 03:09:41 PM GMT
> glibc-devel-2.3.2-4.80 Sat 22 Mar 2003 10:20:48 AM GMT
> glibc-common-2.3.2-4.80 Sat 22 Mar 2003 10:19:28 AM GMT
> glibc-2.3.2-4.80 Sat 22 Mar 2003 10:18:33 AM GMT
> evolution-1.0.8-11 Sat 22 Mar 2003 10:17:28 AM GMT
> 
> But I can't really see how any of those would effect mail (I
> use mutt, not Evolution for mail).
> 
> If someone could give me a step-by-step guide to debugging email
> delivery in RH 8.0 then I'd be very grateful.
> 
> Thanks,
> 
> Dave...
-- 
Please Note: Some Quantum Physics Theories Suggest That When the
Consumer Is Not Directly Observing This Product, It May Cease to
Exist or Will Exist Only in a Vague and Undetermined State.

Re: [OT] Broken sendmail

[Dave Cross]

From: Dirk Koopman <[EMAIL PROTECTED]>
Date: 4/2/03 9:07:11 AM

> What MTA?

It's sendmail.

> Did you reboot the machine?

No. Not yet. I should probably try restarting sendmail first.

> Bear in mind that most MTAs fork to handle some of the 
> process, if any of the libraries that is currently using 
> have disappeared (they have been erased and replacements 
> with different inode nos added), then forks will fail.
>
> I have done more or less exactly what you did, rebooted, 
> it just works. Mind you I run exim with rpms that I 
> maintain.

I made some progress this morning.

"mailq" revealed all 200-odd emails still waiting in the queue
and "sendmail -q" sent them all on their merry way.

Just need to work out why sendmail isn't automatically sending
them at the moment.

Dave...

-- 


"Let me see you make decisions, without your television"
   - Depeche Mode (Stripped)

Re: [OT] Broken sendmail

[Dirk Koopman]

On Wed, 2003-04-02 at 10:16, Dave Cross wrote:
> From: Dirk Koopman <[EMAIL PROTECTED]>
> Date: 4/2/03 9:07:11 AM
> 
> > What MTA?
> 
> It's sendmail.
> 
> > Did you reboot the machine?
> 
> No. Not yet. I should probably try restarting sendmail first.
> 
> > Bear in mind that most MTAs fork to handle some of the 
> > process, if any of the libraries that is currently using 
> > have disappeared (they have been erased and replacements 
> > with different inode nos added), then forks will fail.
> >
> > I have done more or less exactly what you did, rebooted, 
> > it just works. Mind you I run exim with rpms that I 
> > maintain.
> 
> I made some progress this morning.
> 
> "mailq" revealed all 200-odd emails still waiting in the queue
> and "sendmail -q" sent them all on their merry way.

Which simply confirms my conjecture. New (independant) process, uses new
glibc, works.

> Just need to work out why sendmail isn't automatically sending
> them at the moment.

service sendmail restart

But, it would be simpler and better to reboot the machine. You have just
replaced the most fundamental library in the system. In order to get the
benefits of any changes for *all* programs - you must reboot.

Dirk
-- 
Please Note: Some Quantum Physics Theories Suggest That When the
Consumer Is Not Directly Observing This Product, It May Cease to
Exist or Will Exist Only in a Vague and Undetermined State.

Re: [OT] Broken sendmail

[Alex Hudson]

On Wed, Apr 02, 2003 at 01:16:21AM -0800, Dave Cross wrote:
> I made some progress this morning.
> 
> "mailq" revealed all 200-odd emails still waiting in the queue
> and "sendmail -q" sent them all on their merry way.
> 
> Just need to work out why sendmail isn't automatically sending
> them at the moment.

Maybe sendmail is fine - I noticed yesterday (? postcode disc. maybe)
that your posts arrived several hours late, when the list seemed
to be functioning fine. Perhaps you're having bizarre DNS/resolver
issues? That would fit with a glibc upgrade..

Cheers,

Alex.

Re: [OT] Broken sendmail

[Jason Clifford]

On 1 Apr 2003, Dave Cross wrote:

> I've just worked at that at some point on Saturday 22nd March,
> my Linux box decided to stop sending emails out.
> 
> On 22nd March I installed some new rpms via Redhat's up2date
> tool. These were:
> 
> nscd-2.3.2-4.80 Sat 22 Mar 2003 03:09:43 PM GMT
> glibc-profile-2.3.2-4.80 Sat 22 Mar 2003 03:09:41 PM GMT
> glibc-devel-2.3.2-4.80 Sat 22 Mar 2003 10:20:48 AM GMT
> glibc-common-2.3.2-4.80 Sat 22 Mar 2003 10:19:28 AM GMT
> glibc-2.3.2-4.80 Sat 22 Mar 2003 10:18:33 AM GMT
> evolution-1.0.8-11 Sat 22 Mar 2003 10:17:28 AM GMT
> 
> But I can't really see how any of those would effect mail (I
> use mutt, not Evolution for mail).

A glibc update often requires that you restart daemonised applications 
such as sendmail. Have you tried restarting it?

Jason Clifford
-- 
UKFSN.ORG   Finance Free Software while you surf the 'net
http://www.ukfsn.org/   Sign up now

Re: [OT] Broken sendmail

[Merijn Broeren]

Quoting Dave Cross ([EMAIL PROTECTED]):
> 
> No. Not yet. I should probably try restarting sendmail first.
> 
Yes, that'll probably help if you upgraded libc. You need to restart
sshd and other daemons as well.

> Just need to work out why sendmail isn't automatically sending
> them at the moment.
> 
Look in /var/log/mail.log, that should tell you in far too much detail.

Cheers,
-- 
Merijn Broeren | We take risks, we know we take them. Therefore, when things
Software Geek  | come out against us, we have no cause for complaint.
   | - Scott, last journal entry, march 1912

Re: [OT] Broken sendmail

[Dave Cross]

From: Alex Hudson <[EMAIL PROTECTED]>
Date: 4/2/03 9:28:07 AM

On Wed, Apr 02, 2003 at 01:16:21AM -0800, Dave Cross wrote:
>> I made some progress this morning.
>> 
>> "mailq" revealed all 200-odd emails still waiting in the 
>> queue and "sendmail -q" sent them all on their merry way.
>> 
>> Just need to work out why sendmail isn't automatically 
>> sending them at the moment.
>
> Maybe sendmail is fine - I noticed yesterday (? postcode 
> disc. maybe) that your posts arrived several hours late, 
> when the list seemed to be functioning fine. Perhaps 
> you're having bizarre DNS/resolver issues? That would fit 
> with a glibc upgrade..

That was a separate issue. At work I use a webmail system (the
same one that I used to send my plea for help last night) and
for some reason yesterday that decided to hold on to my emails
for several hours before sending them.

Sometimes it sucks to be me :)

Dave...

-- 


"Let me see you make decisions, without your television"
   - Depeche Mode (Stripped)

Re: [OT] Broken sendmail

[Peter Haworth]

On 02 Apr 2003 10:07:11 +0100, Dirk Koopman wrote:
> Bear in mind that most MTAs fork to handle some of the process, if any
> of the libraries that is currently using have disappeared (they have
> been erased and replacements with different inode nos added), then forks
> will fail.

I can't believe this is true.

-- 
Peter Haworth   [EMAIL PROTECTED]
"Democracy is overrated. I think a meritocracy is needed.
 Perhaps measured by Perl competence."
-- Simon Cozens

Re: [OT] Broken sendmail

[Lusercop]

On Wed, Apr 02, 2003 at 10:44:24AM +0100, Peter Haworth wrote:
> On 02 Apr 2003 10:07:11 +0100, Dirk Koopman wrote:
> > Bear in mind that most MTAs fork to handle some of the process, if any
> > of the libraries that is currently using have disappeared (they have
> > been erased and replacements with different inode nos added), then forks
> > will fail.
> I can't believe this is true.

It isn't anymore. Most of them, if they fork() at all, will fork() at the
accept stage. Exim has some magic with re-exec()ing itself at various points
to be able to regain privilege, but this doesn't have to involve forking.

The original sendmail used to fork() to handle the various different
SMTP commands, (which is why pipelining is an extension to SMTP, and why
real RFC821 SMTP *MUST* be lock-step). I don't believe that anything such
as this has been done for a very long time, though. They may well fire off
auxiliary helpers, of course for pipe deliveries or even for actual local
deliveries via, for example mail.local or procmail.

TMTATSNBN also involves lots of small processes, so probably fork()s and
exec()s in many places.

Anyway, getting back to the original point. If this were indeed the case,
then there are two issues with Dirk's original post:
1) the inodes will still exist (they will exist until both their link count
   and their reference count have gone to 0). The unlink() call only
   decrements the former. :-)
2) if this were indeed the case, then I believe that the sendmail -q would
   also have failed.

-- 
Lusercop.net - LARTing Lusers everywhere since 2002

Re: [OT] Broken sendmail

[Shevek]

On 2 Apr 2003, Dirk Koopman wrote:

> On Wed, 2003-04-02 at 10:16, Dave Cross wrote:
> 
> > Just need to work out why sendmail isn't automatically sending
> > them at the moment.
> 
> service sendmail restart
> 
> But, it would be simpler and better to reboot the machine. You have just
> replaced the most fundamental library in the system. In order to get the
> benefits of any changes for *all* programs - you must reboot.

This is not actually true. The change will affect all new processes, and 
frankly the old libc was probably fine. Look in /proc/*/maps if you don't 
believe me.

S.

-- 
Shevekhttp://www.anarres.org/
I am the Borg. http://design.anarres.org/

Re: [OT] Broken sendmail

[Dirk Koopman]

On Wed, 2003-04-02 at 10:57, Lusercop wrote:
> The original sendmail used to fork() to handle the various different
> SMTP commands, (which is why pipelining is an extension to SMTP, and why
> real RFC821 SMTP *MUST* be lock-step). I don't believe that anything such
> as this has been done for a very long time, though. They may well fire off
> auxiliary helpers, of course for pipe deliveries or even for actual local
> deliveries via, for example mail.local or procmail.

Sorry but: have you ever *watched* sendmail send stuff? It forks itself
(as does exim BTW). 

> Anyway, getting back to the original point. If this were indeed the case,
> then there are two issues with Dirk's original post:
> 1) the inodes will still exist (they will exist until both their link count
>and their reference count have gone to 0). The unlink() call only
>decrements the former. :-)

I actually have no idea why daemons fail in the case that 'their' libc
libraries get replaced. All I *do* know is that the symptoms described
occur. *And* that the cure is to restart the daemons or (preferably)
reboot.

I suspect it may be to do with come kind of magic in ELF to cope with
copy on write and address virtualising.

> 2) if this were indeed the case, then I believe that the sendmail -q would
>also have failed.

New process. It uses new versions of libraries. It is effectively
"restarting the daemon" (but just for the queue).

Just reboot, for (your personal) God's sake!

Just what is it with people? What's wrong with rebooting? Is this some
new crime? Is David Blunkett involved???  

Sheesh!

Dirk
-- 
Please Note: Some Quantum Physics Theories Suggest That When the
Consumer Is Not Directly Observing This Product, It May Cease to
Exist or Will Exist Only in a Vague and Undetermined State.

Re: [OT] Broken sendmail

[Shevek]

On Wed, 2 Apr 2003, Peter Haworth wrote:

> On 02 Apr 2003 10:07:11 +0100, Dirk Koopman wrote:
> > Bear in mind that most MTAs fork to handle some of the process, if any
> > of the libraries that is currently using have disappeared (they have
> > been erased and replacements with different inode nos added), then forks
> > will fail.
> 
> I can't believe this is true.

It isn't. The old inode is still there until all processes close the file. 
Read /proc/*/maps, the linker will annotate the old files as (deleted) and 
add version numbers to the mmap()'d space for the new copies.

S.

-- 
Shevekhttp://www.anarres.org/
I am the Borg. http://design.anarres.org/

Re: [OT] Broken sendmail

[Nick Woolley]

On Wednesday 02 Apr 2003 11:43 am, Dirk Koopman wrote:
> I actually have no idea why daemons fail in the case that 'their' libc
> libraries get replaced. All I *do* know is that the symptoms described
> occur. *And* that the cure is to restart the daemons or (preferably)
> reboot.

Of course, you'd think that were so.  However, I didn't notice MySQL had 
broken until I had rebooted, and I found it wasn't there anymore.

I checked my logs for the times of installation and rebooting, so there's no 
mistake: the order is upgrade glibc, shutdown, reboot, mysql fails and rpm 
useless.

It seems daft that upgrading from glibc 2.3.1-6 to 2.3.1-10 should break 
things so seriously, but what ever was wrong in my case, rebooting did not 
sort it out.

Nick