Re: Mails stuck in incoming queue

[Terry Carmen]

> I have multiple almost identically configured postfix servers relaying
> mails
>
> On one of the servers the mails get stuck in incoming queue. All other
> servers are delivering perfectly fine
>
> There are no body / header  checks on this machine that could
> potentially be slowing down pickup. And apparently there are no hardware
> issues
>
> How do I debug the pickup process  ?
>
>

Adding a -v to the end of the "pickup" line your master.cf file and restarting
postfix seems to work although I don't see the -v switch listed in the pickup
man page.

Terry

Re: Mails stuck in incoming queue

[Terry Carmen]

> I have multiple almost identically configured postfix servers relaying
> mails
>
> On one of the servers the mails get stuck in incoming queue. All other
> servers are delivering perfectly fine
>
> There are no body / header  checks on this machine that could
> potentially be slowing down pickup. And apparently there are no hardware
> issues
>
> How do I debug the pickup process  ?
>

PS. Don't forget to "postfix reload"

Terry

Re: Mails stuck in incoming queue

[Victor Duchovni]

On Wed, May 06, 2009 at 09:01:25PM +0530, ram wrote:

> I have multiple almost identically configured postfix servers relaying
> mails 
> 
> On one of the servers the mails get stuck in incoming queue. All other
> servers are delivering perfectly fine 
> 
> There are no body / header  checks on this machine that could
> potentially be slowing down pickup. And apparently there are no hardware
> issues
> 
> How do I debug the pickup process  ?

The "pickup" process is not responsible for moving mail out of the
"incoming" queue. If mail is stuck in "maildrop", then debug "pickup".

http://www.postfix.org/QSHAPE_README.html#queues

-- 
Viktor.

Disclaimer: off-list followups get on-list replies or get ignored.
Please do not ignore the "Reply-To" header.

To unsubscribe from the postfix-users list, visit
http://www.postfix.org/lists.html or click the link below:


If my response solves your problem, the best way to thank me is to not
send an "it worked, thanks" follow-up. If you must respond, please put
"It worked, thanks" in the "Subject" so I can delete these quickly.

Re: Mails stuck in incoming queue

[ram]

On Wed, 2009-05-06 at 14:23 -0400, Victor Duchovni wrote:
> On Wed, May 06, 2009 at 09:01:25PM +0530, ram wrote:
> 
> > I have multiple almost identically configured postfix servers relaying
> > mails 
> > 
> > On one of the servers the mails get stuck in incoming queue. All other
> > servers are delivering perfectly fine 
> > 
> > There are no body / header  checks on this machine that could
> > potentially be slowing down pickup. And apparently there are no hardware
> > issues
> > 
> > How do I debug the pickup process  ?
> 
> The "pickup" process is not responsible for moving mail out of the
> "incoming" queue. If mail is stuck in "maildrop", then debug "pickup".
> 
>   http://www.postfix.org/QSHAPE_README.html#queues
> 


Sorry,  I had not fully read the architecture. If mails are coming from
a remote host , the go first into incoming and then put into active. 

In my case the incoming directory keeps increasing in size( and number
of files)  and the active directory is empty

Why are the mails not put into active ?

Re: Mails stuck in incoming queue

[Victor Duchovni]

On Thu, May 07, 2009 at 10:16:51AM +0530, ram wrote:

> > The "pickup" process is not responsible for moving mail out of the
> > "incoming" queue. If mail is stuck in "maildrop", then debug "pickup".
> > 
> > http://www.postfix.org/QSHAPE_README.html#queues
> 
> Sorry,  I had not fully read the architecture. If mails are coming from
> a remote host , the go first into incoming and then put into active. 
> 
> In my case the incoming directory keeps increasing in size( and number
> of files)  and the active directory is empty
> 
> Why are the mails not put into active ?

Well, I did spend a good chunk of time writing the document:

http://www.postfix.org/QSHAPE_README.html#incoming

 The queue manager scans the incoming queue bringing any new mail into
 the "active" queue if the active queue resource limits have not been
 exceeded. By default, the active queue accommodates at most 2
 messages. Once the active queue message limit is reached, the queue
 manager stops scanning the incoming (and deferred, see below) queue.

 Under normal conditions the incoming queue is nearly empty (has
 only mode 0600 files), with the queue manager able to import new
 messages into the active queue as soon as they become available.

 The incoming queue grows when the message input rate spikes above
 the rate at which the queue manager can import messages into the
 active queue. The main factors slowing down the queue manager are
 disk I/O and lookup queries to the trivial-rewrite service. If
 the queue manager is routinely not keeping up, consider not using
 "slow" lookup services (MySQL, LDAP, ...) for transport lookups
 or speeding up the hosts that provide the lookup service. If the
 problem is I/O starvation, consider striping the queue over more
 disks, faster controllers with a battery write cache, or other
 hardware improvements. At the very least, make sure that the queue
 directory is mounted with the "noatime" option if applicable to
 the underlying filesystem.

If the files are mode 0700, and the active queue is not full, your queue
manager is not doing its job at all, or is totally starved of disk I/O
or is stuck waiting for trivial-rewrite to perform table lookups.

One possiblity not mentioned in QSHAPE_README, is qmgr(8) DoS via a slow
syslogd(8) on a Linux system with an incorrectly defined "mail" facility
log file (uses /path not -/path). This would probably be the best guess
here.

http://www.postfix.org/BASIC_CONFIGURATION_README.html#syslog_howto
http://www.postfix.org/LINUX_README.html

It is fundamentally the same as the disk-starvation, only the cause is
more indirect, because syslogd is starved for disk, and qmgr is blocked
waiting for syslogd. As with direct disk I/O startvation, a single qmgr(8)
is competing against lots of smtpd(8) and cleanup(8) processes, so can't
possibly keep up.

-- 
Viktor.

Disclaimer: off-list followups get on-list replies or get ignored.
Please do not ignore the "Reply-To" header.

To unsubscribe from the postfix-users list, visit
http://www.postfix.org/lists.html or click the link below:


If my response solves your problem, the best way to thank me is to not
send an "it worked, thanks" follow-up. If you must respond, please put
"It worked, thanks" in the "Subject" so I can delete these quickly.

Re: Mails stuck in incoming queue

[ram]

On Thu, 2009-05-07 at 03:22 -0400, Victor Duchovni wrote:
> On Thu, May 07, 2009 at 10:16:51AM +0530, ram wrote:
> 
> > > The "pickup" process is not responsible for moving mail out of the
> > > "incoming" queue. If mail is stuck in "maildrop", then debug "pickup".
> > > 
> > >   http://www.postfix.org/QSHAPE_README.html#queues
> > 
> > Sorry,  I had not fully read the architecture. If mails are coming from
> > a remote host , the go first into incoming and then put into active. 
> > 
> > In my case the incoming directory keeps increasing in size( and number
> > of files)  and the active directory is empty
> > 
> > Why are the mails not put into active ?
> 
> Well, I did spend a good chunk of time writing the document:



Thanks for all the info. Well the high incoming queue is definitely is
due to syslog.  Because I also notice that some of my logs are also
getting dropped. 

Sorry for being OT , but can someone help me find what is wrong with my
syslogd. 


I use sysklogd-1.4.2 ( from default rpm )  on Centos 5.2 ,64bit

Re: Mails stuck in incoming queue

[Ralf Hildebrandt]

* ram :

> Thanks for all the info. Well the high incoming queue is definitely is
> due to syslog.  Because I also notice that some of my logs are also
> getting dropped. 
> 
> Sorry for being OT , but can someone help me find what is wrong with my
> syslogd. 

Show the config.


-- 
Ralf Hildebrandt
Postfix - Einrichtung, Betrieb und Wartung   Tel. +49 (0)30-450 570-155
http://www.computerbeschimpfung.de
"General Failure's Fault. Not Yours."  -Anon.  

Re: Mails stuck in incoming queue

[ram]

On Fri, 2009-05-08 at 09:10 +0200, Ralf Hildebrandt wrote:
> * ram :
> 
> > Thanks for all the info. Well the high incoming queue is definitely is
> > due to syslog.  Because I also notice that some of my logs are also
> > getting dropped. 
> > 
> > Sorry for being OT , but can someone help me find what is wrong with my
> > syslogd. 
> 
> Show the config.
> 
> 

This is my postconf -n output below ( Note: syslog_facility = local6  ) 


I just realized this now 
If I remove entry for  syslog_facility  then mails go fine 
But I need postfix logs on a different file .. so I cant remove this
permanently 





--- postconf -n 

alias_database = hash:/etc/postfix/aliases
alias_maps = hash:/etc/postfix/aliases
alternate_config_directories = /etc/postfix2nd
bounce_size_limit = 5000
canonical_maps = hash:/etc/postfix/canonical
command_directory = /usr/sbin
config_directory = /etc/postfix
daemon_directory = /usr/libexec/postfix
data_directory = /var/lib/postfix
debug_peer_level = 2
default_destination_concurrency_limit = 100
default_process_limit = 700
header_checks = regexp:/etc/postfix/header_checks
html_directory = /usr/share/doc/postfix-2.5.6-documentation/html
mail_owner = postfix
mailq_path = /usr/bin/mailq.postfix
manpage_directory = /usr/share/man
message_size_limit = 5120
mynetworks = /etc/postfix/mynetworks
newaliases_path = /usr/bin/newaliases.postfix
queue_directory = /var/spool/postfix
readme_directory = /usr/share/doc/postfix-2.5.6-documentation/readme
sample_directory = /etc/postfix
sendmail_path = /usr/sbin/sendmail.postfix
setgid_group = postdrop
smtpd_recipient_restrictions=check_recipient_accesshash:/etc/postfix/badrecipients,check_recipient_accessregexp:/etc/postfix/blockedrecipientscheck_sender_accessregexp:/etc/postfix/badsenders,permit_mynetworks,reject
syslog_facility=local6
transport_maps=hash:/etc/postfix/bounce_transport,hash:/etc/postfix/transport1,hash:/etc/postfix/ecm/transport,hash:/etc/postfix/transport
unknown_local_recipient_reject_code=550

Re: Mails stuck in incoming queue

[Ralf Hildebrandt]

* ram :
> 
> On Fri, 2009-05-08 at 09:10 +0200, Ralf Hildebrandt wrote:
> > * ram :
> > 
> > > Thanks for all the info. Well the high incoming queue is definitely is
> > > due to syslog.  Because I also notice that some of my logs are also
> > > getting dropped. 
> > > 
> > > Sorry for being OT , but can someone help me find what is wrong with my
> > > syslogd. 
> > 
> > Show the config.
> > 
> > 
> 
> This is my postconf -n output below ( Note: syslog_facility = local6  ) 

And the syslog.conf?

-- 
Ralf Hildebrandt
Postfix - Einrichtung, Betrieb und Wartung   Tel. +49 (0)30-450 570-155
http://www.computerbeschimpfung.de
The number of the beast - vi vi vi.

Re: Mails stuck in incoming queue

[ram]

On Fri, 2009-05-08 at 11:27 +0200, Ralf Hildebrandt wrote:
> * ram :
> > 
> > On Fri, 2009-05-08 at 09:10 +0200, Ralf Hildebrandt wrote:
> > > * ram :
> > > 
> > > > Thanks for all the info. Well the high incoming queue is definitely is
> > > > due to syslog.  Because I also notice that some of my logs are also
> > > > getting dropped. 
> > > > 
> > > > Sorry for being OT , but can someone help me find what is wrong with my
> > > > syslogd. 
> > > 
> > > Show the config.
> > > 
> > > 
> > 
> > This is my postconf -n output below ( Note: syslog_facility = local6  ) 
> 
> And the syslog.conf?
> 



[r...@lvs4 ~]# cat /etc/syslog.conf
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none/var/log/messages

# The authpriv file has restricted access.
authpriv.*  /var/log/secure

# Log all the mail messages in one place.
mail.*-/var/log/maillog


# Log cron stuff
cron.*  /var/log/cron

# Everybody gets emergency messages
*.emerg *

# Save news errors of level crit and higher in a special file.
uucp,news.crit  /var/log/spooler

# Save boot messages also to boot.log
local7.*/var/log/boot.log
local6.*  -/var/log/lmaillog

Re: Mails stuck in incoming queue

[Ralf Hildebrandt]

* ram :

> # Log anything (except mail) of level info or higher.
> # Don't log private authentication messages!
> *.info;mail.none;authpriv.none;cron.none/var/log/messages

*.info;mail;authpriv;cron;local7;local6.none   /var/log/messages

otherwise /var/log/messages will contain local7. & local6.*!

> # The authpriv file has restricted access.
> authpriv.*  /var/log/secure
> 
> # Log all the mail messages in one place.
> mail.*-/var/log/maillog
> 
> 
> # Log cron stuff
> cron.*  /var/log/cron
> 
> # Everybody gets emergency messages
> *.emerg *
> 
> # Save news errors of level crit and higher in a special file.
> uucp,news.crit  /var/log/spooler
> 
> # Save boot messages also to boot.log
> local7.*/var/log/boot.log
> local6.*  -/var/log/lmaillog
> 
> 

-- 
Ralf Hildebrandt
Postfix - Einrichtung, Betrieb und Wartung   Tel. +49 (0)30-450 570-155
http://www.computerbeschimpfung.de
Redmond WA -- Microsoft announced today that the official release date
for the new operating system "Windows 2000" will be delayed until the
second quarter of 1901. -- seen in Brian Hatch's sig