Greetings all,
I've come across a situation that has me a bit confused and with a system
that is effectively down at the moment. Here is what has occurred thus far:
I've had a LWQ setup running for about 4 months now without issue. Over this
time, I've accumulated 22k email boxes on 8k domains. Last week, I made a
mistake that should've been a temporary issue that has ballooned into a
serious situation. The rcpthosts file was deleted, which, of course, made
the box start to reject email. I rebuilt the rcpthosts list the next
morning, and expected all to be well. Soon after the reload, we began to see
our SMTP service go painfully slow, only allowing a trickle of emails to get
in. So the rcpthosts list going blank and then being rebuilt is not the
problem, but I suspect that due to unrelated misconfiguration, the box was
unprepared to handle the backlog of email due to the 10 hours of downtime(or
it was just coincidence).
I have noticed that if I do a "qmail stop" and then a "qmail start", about
20 successful SMTP connections immediately come in, and then even though
qmail is still running, no more connections can get thru. If I leave it be,
then they will very slowly come in over time. When looking at a "netstat" I
see the following:
[...several more pop3 connections...]
tcp 0 0 mail1.godaddy.com:pop3 66.74.215.231:4542
TIME_WAIT
tcp 0 0 mail1.godaddy.com:pop3 66.74.215.231:4542
TIME_WAIT
tcp 0 0 mail1.godaddy.com:pop3 24.167.206.98:1240
TIME_WAIT
tcp 0 0 mail1.godaddy.com:pop3 65.4.131.26:1247
TIME_WAIT
tcp 0 0 mail1.godaddy.com:pop3 208.191.34.40:3539
TIME_WAIT
tcp 0 0 mail1.godaddy.com:pop3 65.4.131.26:1246
TIME_WAIT
tcp 0 0 mail1.godaddy.com:pop3 65.4.131.26:1245
TIME_WAIT
tcp 0 0 mail1.godaddy.com:pop3 65.11.146.164:1121
TIME_WAIT
tcp 0 0 mail1.godaddy.com:pop3 65.4.131.26:1246
TIME_WAIT
tcp 0 0 mail1.godaddy.com:pop3 65.4.131.26:1245
TIME_WAIT
tcp 0 0 mail1.godaddy.com:pop3 65.11.146.164:1121
TIME_WAIT
tcp 0 0 mail1.godaddy.com:pop3 c327691-a.grnsbrg1:1071
TIME_WAIT
tcp 0 0 mail1.godaddy.com:pop3 65.11.146.164:1118
TIME_WAIT
tcp 0 0 mail1.godaddy.com:pop3 adsl-80-40-124.mia:4278
TIME_WAIT
tcp 0 0 mail1.godaddy.com:pop3 65.24.43.27:2869
TIME_WAIT
tcp 0 0 mail1.godaddy.com:pop3 65.24.43.27:2869
TIME_WAIT
tcp 0 0 mail1.godaddy.com:pop3 64.68.236.161:3033
TIME_WAIT
tcp 0 0 mail1.godaddy.com:pop3 208.191.34.40:3536
TIME_WAIT
tcp 0 0 mail1.godaddy.com:pop3 ool-18b88b07.dyn.o:1951
ESTABLISHED
tcp 0 0 mail1.godaddy.com:pop3 ool-18b88b07.dyn.o:1948
ESTABLISHED
tcp 0 0 mail1.godaddy.com:smtp 208.162.63.130:2439
ESTABLISHED
tcp 0 0 mail1.godaddy.com:smtp 64.211.240.238:37129
ESTABLISHED
tcp 0 0 mail1.godaddy.com:smtp 64.211.240.238:37129
ESTABLISHED
tcp 0 0 mail1.godaddy.com:smtp 208.50.170.31:4152
ESTABLISHED
tcp 0 0 mail1.godaddy.com:smtp 192.147.236.1:4795
ESTABLISHED
tcp 0 0 mail1.godaddy.com:smtp 64.211.240.238:37118
ESTABLISHED
tcp 0 0 mail1.godaddy.com:smtp 64.211.240.238:37106
ESTABLISHED
tcp 0 0 mail1.godaddy.com:smtp 208.50.153.24:54923
ESTABLISHED
tcp 0 0 mail1.godaddy.com:smtp 206.64.128.6:40610
ESTABLISHED
tcp 0 0 mail1.godaddy.com:smtp 206.64.128.6:40610
ESTABLISHED
tcp 0 0 mail1.godaddy.com:smtp 139.76.67.20:42768
ESTABLISHED
tcp 0 0 mail1.godaddy.com:smtp 64.4.9.55:3234
ESTABLISHED
tcp 0 0 mail1.godaddy.com:smtp 207.69.200.157:8125
ESTABLISHED
tcp 0 0 mail1.godaddy.com:smtp 205.184.38.2:44352
ESTABLISHED
tcp 0 0 mail1.godaddy.com:smtp 207.217.120.14:53273
ESTABLISHED
tcp 0 0 mail1.godaddy.com:smtp 210.118.246.250:2886
ESTABLISHED
tcp 0 0 mail1.godaddy.com:smtp 64.211.240.232:13211
ESTABLISHED
tcp 0 0 mail1.godaddy.com:smtp 209.208.202.178:3833
ESTABLISHED
tcp 0 0 mail1.godaddy.com:smtp 208.50.144.69:5865
ESTABLISHED
tcp 0 0 mail1.godaddy.com:smtp 148.233.27.132:4087
ESTABLISHED
tcp 0 0 mail1.godaddy.com:smtp 207.150.192.30:1066
ESTABLISHED
tcp 0 0 mail1.godaddy.com:smtp 208.184.37.231:3667 SYN_RECV
tcp 0 0 mail1.godaddy.com:smtp 65.10.73.142:36399 SYN_RECV
tcp 0 0 mail1.godaddy.com:smtp 148.233.27.132:4087
ESTABLISHED
tcp 0 0 mail1.godaddy.com:smtp 207.150.192.30:1066
ESTABLISHED
tcp 0 0 mail1.godaddy.com:smtp 208.48.26.72:8948 SYN_RECV
tcp 0 0 mail1.godaddy.com:smtp 205.150.6.55:3250 SYN_RECV
tcp 0 0 mail1.godaddy.com:smtp mail1.stofanet.dk:1521 SYN_RECV
tcp 0 0 mail1.godaddy.com:smtp 216.33.156.140:63931 SYN_RECV
tcp 0 0 mail1.godaddy.com:smtp mail-out.chello.n:21159 SYN_RECV
tcp 0 0 mail1.godaddy.com:smtp 216.33.156.140:63933 SYN_RECV
tcp 0 0 mail1.godaddy.com:smtp 216.33.149.107:2532 SYN_RECV
tcp 0 0 mail1.godaddy.com:smtp 207.217.120.14:57736 SYN_RECV
tcp 0 0 mail1.godaddy.com:smtp 208.237.254.8:1189 SYN_RECV
tcp 0 0 mail1.godaddy.com:smtp 207.151.14.107:3166 SYN_RECV
tcp 0 0 mail1.godaddy.com:smtp 64.12.136.161:57745 SYN_RECV
tcp 0 0 mail1.godaddy.com:smtp 204.86.96.11:39465 SYN_RECV
tcp 0 0 mail1.godaddy.com:smtp 200.42.0.148:2089 SYN_RECV
tcp 0 0 mail1.godaddy.com:smtp 65.32.1.39:64871 SYN_RECV
tcp 0 0 mail1.godaddy.com:smtp 208.237.254.8:1190 SYN_RECV
tcp 0 0 mail1.godaddy.com:smtp 209.87.128.98:3944 SYN_RECV
tcp 0 0 mail1.godaddy.com:smtp 207.0.62.253:1169 SYN_RECV
[...several more smtp connections in the SYN_RECV state...]
A few minutes later, I did another "netstat". For brevity, here are the
"established smtp" connections reported this second time:
tcp 0 0 mail1.godaddy.com:smtp 208.162.63.130:2439
ESTABLISHED
tcp 0 0 mail1.godaddy.com:smtp 64.211.240.238:37129
ESTABLISHED
tcp 0 0 mail1.godaddy.com:smtp 208.50.170.31:4152
ESTABLISHED
tcp 0 0 mail1.godaddy.com:smtp 192.147.236.1:4795
ESTABLISHED
tcp 0 0 mail1.godaddy.com:smtp 64.211.240.238:37118
ESTABLISHED
tcp 0 0 mail1.godaddy.com:smtp 64.211.240.238:37106
ESTABLISHED
tcp 0 0 mail1.godaddy.com:smtp 208.50.153.24:54923
ESTABLISHED
tcp 0 0 mail1.godaddy.com:smtp 206.64.128.6:40610
ESTABLISHED
tcp 0 0 mail1.godaddy.com:smtp 139.76.67.20:42768
ESTABLISHED
tcp 0 0 mail1.godaddy.com:smtp 64.4.9.55:3234
ESTABLISHED
tcp 0 0 mail1.godaddy.com:smtp 207.69.200.157:8125
ESTABLISHED
tcp 0 0 mail1.godaddy.com:smtp 205.184.38.2:44352
ESTABLISHED
tcp 0 0 mail1.godaddy.com:smtp 207.217.120.14:53273
ESTABLISHED
tcp 0 0 mail1.godaddy.com:smtp 210.118.246.250:2886
ESTABLISHED
tcp 0 0 mail1.godaddy.com:smtp 212.29.65.201:3937
ESTABLISHED
tcp 0 0 mail1.godaddy.com:smtp 64.37.73.213:41102
ESTABLISHED
tcp 0 0 mail1.godaddy.com:smtp 64.211.240.232:13211
ESTABLISHED
tcp 0 0 mail1.godaddy.com:smtp 209.208.202.178:3833
ESTABLISHED
tcp 0 0 mail1.godaddy.com:smtp 208.50.144.69:5865
ESTABLISHED
tcp 0 0 mail1.godaddy.com:smtp 148.233.27.132:4087
ESTABLISHED
tcp 0 0 mail1.godaddy.com:smtp 207.150.192.30:1066
ESTABLISHED
tcp 0 0 mail1.godaddy.com:smtp 207.150.192.30:1066
ESTABLISHED
What I've noticed is that virtually all of the same connections are still
there. This makes sense given that every time I do a "qmail stop" and then a
"qmail start" I am able to get ~20 more emails delivered, because the first
~20 SMTP servers to connect are served, but then they don't disconnect and
get out of the way for new connections to be made.
So, I _think_ that I'm either wanting to increase the number of established
connections I can make, or I need to reconfigure to make those established
smtp connections disconnect as soon as they're done sending email(or both).
However, you guys are the gurus I'm leaning on for insight, so I am, of
course, open to any ideas as to the cause.
Here is the tcpserver line I am currently using for smtp:
22482 ? S 0:00 /usr/local/bin/tcpserver -v -l
mail1.godaddy.com -P -H -R -x /etc/tcp.smtp.cdb -c 120 -u 503 -g 502 0 smtp
/var/qmail/bin/qmail-smtpd
And a ps -ax | grep qmail:
[root@mail1 qmail]# ps -ax | grep qmail
692 ? S 0:11 /usr/local/bin/multilog t s1000000 n20
/var/log/qmail
29658 ? S 0:00 supervise qmail-smtpd
29660 ? S 0:00 /usr/local/bin/multilog t /var/log/qmail
29800 ? S 0:00 supervise qmail-send
29804 ? S 0:00 qmail-send
29806 ? S 0:00 qmail-lspawn ./Maildir/
29807 ? S 0:00 qmail-rspawn
29808 ? S 0:00 qmail-clean
2334 ? S 0:00 /var/qmail/bin/qmail-popup mail1.godaddy.com
/bin/che
2835 ? S 0:00 /var/qmail/bin/qmail-pop3d Maildir
3555 ? S 0:00 /var/qmail/bin/qmail-popup mail1.godaddy.com
/bin/che
3558 ? S 0:00 /var/qmail/bin/qmail-popup mail1.godaddy.com
/bin/che
3596 ? S 0:00 /var/qmail/bin/qmail-pop3d Maildir
3597 ? S 0:00 /var/qmail/bin/qmail-pop3d Maildir
3674 ? S 0:00 /var/qmail/bin/qmail-popup mail1.godaddy.com
/bin/che
3681 ? S 0:00 /var/qmail/bin/qmail-pop3d Maildir
Any assistance is greatly appreciated. If there is any other information you
need, please let me know.
Thanks,
Matt Hubbard
Go Daddy Software
