Re: Mail stuck (Connection Timed-Out)
I was reading about "Defferred queue full of dictionary attack bounces" which I think might be an issue here. So i performed a qshape analysis and I got this: command: qshape deferred | head T 5 10 20 40 80 160 320 640 1280 1280+ TOTAL 583 0 0 0 0 0 1 8 47 25 502 adbaa.org 214 0 0 0 0 0 0 0 00 214 onramp.bz 191 0 0 0 0 0 1 6 26 10 148 unitedimagingpartners.com 62 0 0 0 0 0 0 0 7550 mmvacations.com 26 0 0 0 0 0 0 0 1025 fishwindowcleaning.com 12 0 0 0 0 0 0 0 31 8 warrensouth.com 5 0 0 0 0 0 0 0 10 4 ecodiscoverypark.com 5 0 0 0 0 0 0 1 00 4 pfg.com 4 0 0 0 0 0 0 0 12 1 Luckily the active and and incoming queues aren't showing any signs of backscatter. I'm going to be checking firewall tomorrow to see if there's an issue there. Thanks for your help. Gonzo Fernandez On Jan 30, 2012, at 9:09 PM, Noel Jones wrote: > On 1/30/2012 10:30 PM, Noel Jones wrote: >> On 1/30/2012 6:46 PM, Gonzo Fernandez wrote: >>> Thank you Noel. Our server sends out copies of email confirmations >>> to our clients and if the client decides to make a large order they >>> end up pushing our volume up and we end up getting blocked by their >>> mail server. I seem to be getting connection timed out on a lot of >>> the hosts. I even try to telnet to ip and port 25 but it keeps >>> timing out. I used "grep" to search in /var/log/maillog and I got >>> this. Any ideas? >> >> >> This isn't a postfix problem. If you can't telnet to any client >> port 25, then you have a connectivity problem; maybe your ISP is >> blocking that port, or some firewall has been misconfigured. >> Contact your networking team or your ISP. >> >> If you can't telnet to this one destination port 25, they're >> blocking you. You'll need to contact them to get this resolved. >> >> >> Good luck. >> >> >> >> -- Noel Jones > > > If the destination can't handle the load you're sending it, you can > slow postfix down. Details here: > http://www.postfix.org/QSHAPE_README.html#active_congestion > > Of course, this won't help until they stop blocking you. > > -- Noel Jones
Re: Mail stuck (Connection Timed-Out)
Thank you Noel. Our server sends out copies of email confirmations to our clients and if the client decides to make a large order they end up pushing our volume up and we end up getting blocked by their mail server. I seem to be getting connection timed out on a lot of the hosts. I even try to telnet to ip and port 25 but it keeps timing out. I used "grep" to search in /var/log/maillog and I got this. Any ideas? [root@mx-server ~]# cat /var/log/maillog | grep B0847E8491 Jan 30 08:44:38 mx-server postfix/cleanup[24478]: B0847E8491: message-id=<20120130164438.B0847E8491@mxser...@example.com> Jan 30 08:44:38 mx-server postfix/qmgr[16186]: B0847E8491: from=<>, size=3456, nrcpt=1 (queue active) Jan 30 08:44:38 mx-server postfix/bounce[24473]: 2604BE84D6: sender non-delivery notification: B0847E8491 Jan 30 08:45:01 mx-server postfix/smtp[24278]: B0847E8491: to=, relay=none, delay=23, delays=0.03/0/23/0, dsn=4.4.1, status=deferred (connect to example.com[1.2.3.4]: Connection timed out) Jan 30 09:08:09 mx-server postfix/qmgr[16186]: B0847E8491: from=<>, size=3456, nrcpt=1 (queue active) Jan 30 09:08:32 mx-server postfix/smtp[24522]: B0847E8491: to=, relay=none, delay=1434, delays=1411/0/23/0, dsn=4.4.1, status=deferred (connect to example.com[1.2.3.4]: Connection timed out) Jan 30 09:41:31 mx-server postfix/qmgr[16186]: B0847E8491: from=<>, size=3456, nrcpt=1 (queue active) Jan 30 09:41:52 mx-server postfix/smtp[24793]: B0847E8491: to=, relay=none, delay=3434, delays=3412/0.1/21/0, dsn=4.4.1, status=deferred (connect to example.com[1.2.3.4]: Connection timed out) Jan 30 10:48:09 mx-server postfix/qmgr[16186]: B0847E8491: from=<>, size=3456, nrcpt=1 (queue active) Jan 30 10:48:15 mx-server postfix/smtp[25097]: B0847E8491: to=, relay=none, delay=7417, delays=7411/0.06/5.9/0, dsn=4.4.3, status=deferred (Host or domain name not found. Name service error for name=example.com type=A: Host not found, try again) Jan 30 12:11:30 mx-server postfix/qmgr[16186]: B0847E8491: from=<>, size=3456, nrcpt=1 (queue active) Jan 30 12:11:53 mx-server postfix/smtp[25539]: B0847E8491: to=, relay=none, delay=12435, delays=12411/0.05/23/0, dsn=4.4.1, status=deferred (connect to example.com[1.2.3.4]: Connection timed out) Jan 30 13:22:45 mx-server postfix/qmgr[26236]: B0847E8491: from=<>, size=3456, nrcpt=1 (queue active) Jan 30 13:23:12 mx-server postfix/smtp[26261]: B0847E8491: to=, relay=none, delay=16713, delays=16687/0.56/26/0, dsn=4.4.1, status=deferred (connect to example.com[1.2.3.4]: Connection timed out) Jan 30 13:53:27 mx-server postfix/qmgr[26443]: B0847E8491: from=<>, size=3456, nrcpt=1 (queue active) Jan 30 13:53:55 mx-server postfix/smtp[26593]: B0847E8491: to=, relay=none, delay=18556, delays=18529/6.5/21/0, dsn=4.4.1, status=deferred (connect to example.com[1.2.3.4]: Connection timed out) Jan 30 15:14:54 mx-server postfix/qmgr[27600]: B0847E8491: from=<>, size=3456, nrcpt=1 (queue active) Jan 30 15:15:21 mx-server postfix/smtp[27790]: B0847E8491: to=, relay=none, delay=23443, delays=23416/5.9/21/0, dsn=4.4.1, status=deferred (connect to example.com[1.2.3.4]: Connection timed out) [root@mx-server ~]# telnet 1.2.3.4 25 Trying 1.2.3.4... telnet: connect to address 1.2.3.4: Connection timed out telnet: Unable to connect to remote host: Connection timed out Gonzo Fernandez On Jan 30, 2012, at 3:36 PM, Noel Jones wrote: > On 1/30/2012 5:07 PM, Gonzo Fernandez wrote: >> Hi All, >> >> My relay servers have mail being received but unable to send. When I >> type "mailq" I see: Delivery temporarily suspended….Connection timed >> out. I also noticed this line: >> >> Tarpitting active for [1.2.3.4) >> >> I restarted postfix, flushed mailq and still everything is stuck. >> Now the mail is building up and I don't know what else to do. I'm >> still continuing to work on it but I figure I might as well ask the >> postfix team members. Can anyone help me figure this thing out please? >> >> mailq: >> >> Jan 30 13:53:27 mx-server postfix/qmgr[26443]: BC535E8264: >> from=mailto:m...@example.com>>, size=805, nrcpt=1 >> (queue active) >> Jan 30 13:53:55 mx-server postfix/qmgr[26443]: BC535E8264: >> to=mailto:m...@example.com>>, relay=none, delay=357647, >> delays=357619/28/0/0, dsn=4.4.1, status=deferred (delivery >> temporarily suspended: connect to example.com >> <http://example.com>[1.2.3.4]: Connection timed out) > > > (please post in plain-text only) > (please use example.com rather than real domain names. thanks) > > > Looks as if the destination 1.2.3.4 doesn't like your server. > You'll need to check with them about why. > > One possibility is that you've been flooding them with backscatter > a
Mail stuck (Connection Timed-Out)
Hi All, My relay servers have mail being received but unable to send. When I type "mailq" I see: Delivery temporarily suspended….Connection timed out. I also noticed this line: Tarpitting active for [1.2.3.4) I restarted postfix, flushed mailq and still everything is stuck. Now the mail is building up and I don't know what else to do. I'm still continuing to work on it but I figure I might as well ask the postfix team members. Can anyone help me figure this thing out please? mailq: Jan 30 13:53:27 mx-ca4-01 postfix/qmgr[26443]: BC535E8264: from=, size=805, nrcpt=1 (queue active) Jan 30 13:53:55 mx-ca4-01 postfix/qmgr[26443]: BC535E8264: to=, relay=none, delay=357647, delays=357619/28/0/0, dsn=4.4.1, status=deferred (delivery temporarily suspended: connect to mail.com[1.2.3.4]: Connection timed out) I set this line up in main.cf and it did help a little bit: smtpd_error_sleep_time = 0 Here's my postconf -n: alias_database = hash:/etc/aliases alias_maps = hash:/etc/aliases command_directory = /usr/sbin config_directory = /etc/postfix daemon_directory = /usr/libexec/postfix debug_peer_level = 2 disable_vrfy_command = yes header_checks = regexp:/etc/postfix/header_checks html_directory = no inet_interfaces = all mail_owner = postfix mailq_path = /usr/bin/mailq.postfix manpage_directory = /usr/share/man mydestination = $myhostname, localhost.$mydomain, localhost mynetworks = 1.2.3.0/24, 1.2.3.0/24, 1.2.3.0/24, 1.2.3.0/24, 1.2.3.0/24 newaliases_path = /usr/bin/newaliases.postfix queue_directory = /var/spool/postfix readme_directory = /usr/share/doc/postfix-2.3.3/README_FILES sample_directory = /usr/share/doc/postfix-2.3.3/samples sendmail_path = /usr/sbin/sendmail.postfix setgid_group = postdrop smtpd_error_sleep_time = 0 unknown_local_recipient_reject_code = 550 Gonzo Fernandez
Re: Postfix Not Sending Emails. Timeout on Cleanup socket error.
Hi Wietse, The UID 500 is from our batch email not spam. It constantly sends out emails to our clients on a regular basis. Also, I turned off verbose logging on all options. I originally turned it on to try and figure out why it was getting stuck on the weekends, but now it is off. I will see if that helps this weekend. Thank you. Gonzo Fernandez Network Engineer On Dec 15, 2011, at 5:35 PM, Wietse Venema wrote: > Gonzo Fernandez: >> # egrep 884643E30022 /var/log/maillog >> >> Dec 12 10:08:50 batch-ca4-02 postfix/pickup[26935]: warning: 884643E30022: >> message has been queued for 1 days >> Dec 12 10:08:50 batch-ca4-02 postfix/pickup[26935]: 884643E30022: uid=500 >> from= > > What user acccount has uid=500? Is this your web server? Perhaps > you have an exploitable web application. This is very popular with > spammers. > > To block local submissions from this user, > > /etc/postfix/main.cf: >authorized_submit_users = !username, static:anyone > >> Dec 12 10:08:50 batch-ca4-02 postfix/qmgr[26936]: qmgr_active_feed: >> incoming/884643E30022 > > Please turn off all your verbose logging in master.cf. It makes > Postfix VERY VERY SLOW and makes mail overload problems much worse. > > On the other hand, if your email overload is caused by spam, please > turn on verbose logging on everything, to slow down the delivery > of spam. > > Wietse
Re: Postfix Not Sending Emails. Timeout on Cleanup socket error.
09:19 batch-ca4-02 postfix/cleanup[31691]: 7905B3E3003A: message-id=<20111211120919.7905B3E3003A@batch-ca4-02> Dec 11 04:09:19 batch-ca4-02 postfix/qmgr[28082]: trigger_server_accept_fifo: trigger arrived Dec 11 04:09:19 batch-ca4-02 postfix/qmgr[28082]: master_notify: status 0 Dec 11 04:09:19 batch-ca4-02 postfix/qmgr[28082]: request: 87 (W) Dec 11 04:09:19 batch-ca4-02 postfix/qmgr[28082]: qmgr_scan_start: start incoming queue scan Dec 11 04:09:19 batch-ca4-02 postfix/qmgr[28082]: master_notify: status 1 Dec 11 04:09:19 batch-ca4-02 postfix/qmgr[28082]: qmgr_active_feed: queue incoming Dec 11 04:09:19 batch-ca4-02 postfix/qmgr[28082]: qmgr_active_feed: incoming/7905B3E3003A Dec 11 04:09:19 batch-ca4-02 postfix/qmgr[28082]: qmgr_message_alloc: active 7905B3E3003A Dec 11 04:09:19 batch-ca4-02 postfix/qmgr[28082]: 7905B3E3003A: from=, size=1084, nrcpt=1 (queue active) Dec 11 04:09:19 batch-ca4-02 postfix/qmgr[28082]: start sorted recipient list Dec 11 04:09:19 batch-ca4-02 postfix/qmgr[28082]: qmgr_message_sort: x...@.com Dec 11 04:09:39 batch-ca4-02 postfix/smtpd[31631]: rewrite stream disconnect Dec 11 04:09:39 batch-ca4-02 postfix/smtp[31689]: idle timeout -- exiting Dec 11 04:16:47 batch-ca4-02 postfix/smtpd[31631]: idle timeout -- exiting Dec 11 05:09:19 batch-ca4-02 postfix/cleanup[31691]: warning: timeout on cleanup socket while reading input attribute name Dec 11 05:30:09 batch-ca4-02 postfix/smtpd[5663]: dict_eval: const mail Dec 11 05:31:27 batch-ca4-02 postfix/cleanup[31691]: warning: 8A2993E3003B: read timeout on cleanup socket Dec 11 05:53:51 batch-ca4-02 postfix/smtpd[5663]: dict_eval: const ipv4 I know I originally stated that the email is from the pickup but now I'm not so sure. I'm seeing a lot of postfix/smtp verbose messages. I hope this helps. Gonzo Fernandez Network Engineer On Dec 15, 2011, at 2:07 PM, Lorens Kockum wrote: > On Thu, Dec 15, 2011 at 11:49:35AM -0800, Gonzo Fernandez wrote: >> /var/log/maillog:Dec 14 04:03:07 batch-ca4-02 postfix/sendmail[12280]: >> fatal: root(0): queue file write error >> /var/log/maillog:Dec 15 12:03:07 batch-ca4-02 postfix/postdrop[21744]: >> warning: uid=0: Illegal seek > > In addition to what Wietse asked for in another mail just now, > do look at the output of the commands > > mount > > dmesg > > That should show if there is a hardware problem. > > If it is not, then is it not possible that the 8A2993E3003B you > are worrying about is a locally submitted mail from some system > daemon (like logwatch) that is too big for your configured > system limits? The times on the error messages make me suspect > some kind of cron job. > > See: > > http://www.electrictoolbox.com/logwatch-postfix-sendmail-errors/ > > > http://www.cyberciti.biz/tips/linux-unix-posttfix-mutt-illegal-seek-error.html > > 8A2993E3003B would then be a minor consequence of your main > problem, which is mail freezing up over weekends. > > To diagnose that problem, it would be interesting to grep on a > queue id that was correctly queued, like (from an earlier mail): > > Dec 12 10:08:50 batch-ca4-02 postfix/pickup[26935]: warning: 884643E30022: > message has been queued for 1 days > > Since mails such as those have been correctly queued, there > should be a wealth of information about when and how and why > it's staying in the queue so long. > > HTH >
Re: Postfix Not Sending Emails. Timeout on Cleanup socket error.
Hi guys, So far no luck searching for 8A2993E3003B. :( The system is heavily used. From what I understand this machine relays all mail to our main mx server and we have a lot of email being relayed daily. The strange part is that it works all week up until late Saturday/Sunday in the AM hours. Here are all log files for mail. I have searched all log files for the specific string but no luck. Performing another search for all warning, error, fatal, panic on all previous log files has given me a bit more info. The following is certain errors I have noticed pop up more often than not: # egrep (warning|error|fatal|panic): /var/log/maillog* | more /var/log/maillog:Dec 11 05:09:19 batch-ca4-02 postfix/cleanup[31691]: warning: timeout on cleanup socket while reading input attribute name /var/log/maillog:Dec 11 05:31:27 batch-ca4-02 postfix/cleanup[31691]: warning: 8A2993E3003B: read timeout on cleanup socket /var/log/maillog:Dec 13 12:04:13 batch-ca4-02 postfix/postdrop[15153]: warning: uid=0: Illegal seek /var/log/maillog:Dec 13 04:04:13 batch-ca4-02 postfix/sendmail[15152]: fatal: root(0): queue file write error /var/log/maillog:Dec 14 12:03:07 batch-ca4-02 postfix/postdrop[12285]: warning: uid=0: Illegal seek /var/log/maillog:Dec 14 04:03:07 batch-ca4-02 postfix/sendmail[12280]: fatal: root(0): queue file write error /var/log/maillog:Dec 15 12:03:07 batch-ca4-02 postfix/postdrop[21744]: warning: uid=0: Illegal seek /var/log/maillog:Dec 15 04:03:07 batch-ca4-02 postfix/sendmail[21739]: fatal: root(0): queue file write error /var/log/maillog.1:Dec 4 05:18:29 batch-ca4-02 postfix/cleanup[6555]: warning: timeout on cleanup socket while reading input attribute name /var/log/maillog.1:Dec 4 05:45:13 batch-ca4-02 postfix/cleanup[6555]: warning: 0433D3E3003A: read timeout on cleanup socket /var/log/maillog:Dec 15 12:03:07 batch-ca4-02 postfix/postdrop[21744]: warning: uid=0: Illegal seek /var/log/maillog:Dec 15 04:03:07 batch-ca4-02 postfix/sendmail[21739]: fatal: root(0): queue file write error /var/log/maillog.1:Dec 5 12:03:47 batch-ca4-02 postfix/postdrop[4667]: warning: uid=0: Illegal seek /var/log/maillog.1:Dec 5 04:03:47 batch-ca4-02 postfix/sendmail[4660]: fatal: root(0): queue file write error (I tried searching for 0433D3E3003A as well but to no avail.) # ls -ld /var/log/maillog* -rw--- 1 root root 194889523 Dec 15 10:48 /var/log/maillog -rw--- 1 root root 370337428 Dec 11 04:04 /var/log/maillog.1 -rw--- 1 root root 644886321 Dec 4 04:03 /var/log/maillog.2 -rw--- 1 root root 250479845 Nov 27 04:03 /var/log/maillog.3 -rw--- 1 root root 11139266 Nov 20 04:02 /var/log/maillog.4 I also commented out fallback relay as it was active but left blank in master.cf. Not sure if that has anything to do with it but better safe than sorry. # When relaying mail as backup MX, disable fallback_relay to avoid MX loops relay unix - - n - - smtp # -o fallback_relay= I'm going to wait and see what happens this weekend and when it happens again I'll try to get more info for you. Thanks again for the help. Gonzo Fernandez Network Engineer On Dec 14, 2011, at 11:01 PM, Lorens Kockum wrote: > On Wed, Dec 14, 2011 at 05:43:58PM -0800, Gonzo Fernandez wrote: >> I'm not sure I'm understanding the log file info you would like. I tried >> doing grep search for 8A2993E3003B on all log files under /var/log/* and >> only found the following line to show up: >> >> Dec 11 05:31:27 batch-ca4-02 postfix/cleanup[31691]: warning: 8A2993E3003B: >> read timeout on cleanup socket >> >> Can you please be more specific? Thanks for your patience and help. > > > Your log line is dated 5 AM plus a little. Maybe your log file > got rotated (that usually happens at 4 or 5 AM). The reason > that your grep on all log files did not yield the result is > probably that the rotated log files are zipped. zgrep (at least > my version) behaves nicely when the files aren't zipped, so > > zgrep 8A2993E3003B /var/log/maillog* > > should yield the information Wietse was asking for. If not, an > >ls -ld /var/log/maillog* > > would be useful. > > Is this a very heavily used system?
Re: Postfix Not Sending Emails. Timeout on Cleanup socket error.
I'm not sure I'm understanding the log file info you would like. I tried doing grep search for 8A2993E3003B on all log files under /var/log/* and only found the following line to show up: Dec 11 05:31:27 batch-ca4-02 postfix/cleanup[31691]: warning: 8A2993E3003B: read timeout on cleanup socket Can you please be more specific? Thanks for your patience and help. Gonzo Fernandez Network Engineer On Dec 14, 2011, at 4:12 PM, Wietse Venema wrote: > Gonzo Fernandez: >> Hi Wietse, >> >> I apologize for the confusion. Here is the result of the following command. >> The mail arrived via pickup (local). > > PLEASE show the logfile records that I asked for. > > Wietse > >> # grep 8A2993E3003B /var/log/maillog >> Dec 11 05:31:27 batch-ca4-02 postfix/cleanup[31691]: warning: 8A2993E3003B: >> read timeout on cleanup socket >> >> >> Gonzo Fernandez >> Network Engineer >> >> On Dec 13, 2011, at 12:35 PM, Wietse Venema wrote: >> >>> Gonzo Fernandez: >>>> Dec 11 05:09:19 batch-ca4-02 postfix/cleanup[31691]: warning: timeout on >>>> cleanup socket while reading input attribute name >>>> Dec 11 05:31:27 batch-ca4-02 postfix/cleanup[31691]: warning: >>>> 8A2993E3003B: read timeout on cleanup socket >>> >>> Did the email arrive via smtpd (network), or via pickup (local)? >>> >>> $ grep 8A2993E3003B /the/log/file >>> >>> Wietse >>
Re: Postfix Not Sending Emails. Timeout on Cleanup socket error.
Hi Wietse, I apologize for the confusion. Here is the result of the following command. The mail arrived via pickup (local). # grep 8A2993E3003B /var/log/maillog Dec 11 05:31:27 batch-ca4-02 postfix/cleanup[31691]: warning: 8A2993E3003B: read timeout on cleanup socket Gonzo Fernandez Network Engineer On Dec 13, 2011, at 12:35 PM, Wietse Venema wrote: > Gonzo Fernandez: >> Dec 11 05:09:19 batch-ca4-02 postfix/cleanup[31691]: warning: timeout on >> cleanup socket while reading input attribute name >> Dec 11 05:31:27 batch-ca4-02 postfix/cleanup[31691]: warning: 8A2993E3003B: >> read timeout on cleanup socket > > Did the email arrive via smtpd (network), or via pickup (local)? > > $ grep 8A2993E3003B /the/log/file > > Wietse
Re: Postfix Not Sending Emails. Timeout on Cleanup socket error.
file write error Gonzo Fernandez Network Engineer On Dec 13, 2011, at 12:21 PM, Brian Evans - Postfix List wrote: > egrep '(warning|error|fatal|panic):' /some/log/file | more
Postfix Not Sending Emails. Timeout on Cleanup socket error.
Hi all, I'm not too familiar with postfix but our server seems to stop sending emails usually only on the weekends. The rest of the week it's fine but once it gets to about Saturday/Sunday morning it gets stuck. I've come in usually on Monday and type "mailq" only to find hundreds if not thousand or so emails stuck in the incoming queue. I have to restart the postfix service for all the emails to get sent. I was hoping someone a little more experienced than me can point me in the right direction and explain what's going on as I'm not that familiar with it yet. I'm sure I'm not the first to have this issue and I've searched online but have found nothing specific that could be causing this. This is what "cat /var/log/maillog | grep warning" shows: Dec 11 05:09:19 batch-ca4-02 postfix/cleanup[31691]: warning: timeout on cleanup socket while reading input attribute name Dec 11 05:31:27 batch-ca4-02 postfix/cleanup[31691]: warning: 8A2993E3003B: read timeout on cleanup socket Here is my postconf -n: alias_database = hash:/etc/aliases alias_maps = hash:/etc/aliases command_directory = /usr/sbin config_directory = /etc/postfix daemon_directory = /usr/libexec/postfix debug_peer_level = 2 html_directory = no in_flow_delay = 10s inet_interfaces = localhost mail_owner = postfix mailq_path = /usr/bin/mailq.postfix manpage_directory = /usr/share/man mydestination = $myhostname, localhost.$mydomain, localhost mydomain = usaepay.com myorigin = $mydomain newaliases_path = /usr/bin/newaliases.postfix queue_directory = /var/spool/postfix readme_directory = /usr/share/doc/postfix-2.3.3/README_FILES relayhost = 192.168.x.x sample_directory = /usr/share/doc/postfix-2.3.3/samples sendmail_path = /usr/sbin/sendmail.postfix setgid_group = postdrop unknown_local_recipient_reject_code = 550 Can anyone please help?? If I need to submit more info please let me know. Thank you! Gonzo Fernandez Network Engineer