Postfix redundancy

2021-03-24 Thread Patrick Chemla

  
  
Hi,


Apparently, searching Google, I still can't find a good solution
  to build a fault tolerant emails platform.


Is there any good solution to synchronize 2 emails servers,
  including incoming mails, having users connected to any of the
  server?


I am looking for such architecture for long time, and from time
  to time the question come again.


Thanks for any idea.


Patrick


  



Re: Host not found?

2020-10-18 Thread Patrick Chemla

  
  
No MX server for client.com:


# nslookup -type=mx client.com
  Server: 8.8.8.8
  Address:    8.8.8.8#53
  
  Non-authoritative answer:
  *** Can't find client.com: No answer




Le 18/10/2020 à 23:16, Richard a
  écrit :


  


  
Date: Sunday, October 18, 2020 16:07:24 -0400
From: Joey J 

Hello all,

I'm trying to understand why this is telling me host not found.
On that same server if I nslookup the ip it does resolve.

Oct 18 16:00:51 mgw postfix/smtpd[24119]: NOQUEUE: reject: RCPT from
unknown[199.5.50.180]: 450 4.7.1 : Helo command
rejected: Host not found; from=
to= proto=ESMTP helo=

  
  

There doesn't appear to be an A or MX record for "br2.vw.com".





  



Re: load balanced emails servers pair

2017-02-09 Thread Patrick Chemla

Thanks all for your answers.

I have at last setup the NAS, and mails are received there.

So I will set the second server and second MTA, and both will receive 
emails.


Next step is to give users access to both servers to retreive emails.

As a load-balancer could help easily for http/https access, how to deal 
with IMAP ports? How to load-balance IMAP ports?


Thanks
Patrick

Le 29/01/2017 à 14:29, rightkicktech.gmail.com a écrit :

A shared storage with glusterfs seems a nice approach.
In this way, it doesn't matter which server receives the mail, as long 
as the MDAs of each server write on the shared storage.


Alex

On January 25, 2017 6:08:59 PM EET, Patrick Domack 
<patric...@patrickdk.com> wrote:


All options, assuming your imap/pop/lmtp are compatable and friendly using 
it.

I know dovecot you should only access a mailstore from one host at a
time, don't just randomly balance things, or it can corrupt the index
files.

Quoting Eero Volotinen <eero.voloti...@iki.fi>:

how about mounting ceph or glusterfs disk to message store?
eero 25.1.2017 5.18 ap. "Patrick Domack"
<patric...@patrickdk.com> kirjoitti:

This would not be a good thing to do, as deleted email
will magically reappear. Using unison to sync it worked
for me, over 10years ago. But these days, just use dsync
part of dovecot, and your life will be happy. Quoting
Patrick Chemla <patrick.che...@perfaction.net>: Hi Wietse,

Of course I thought about such NAS solution, but I
wanted to check if there is a way with 2 separate
disks, with a kind of that could be aware of emails
files changes. Actually, the mail server run onto a
VM, on a big server. I have another big server with
same emails VM, and I just rsync --delete --update
from the first one to the second. So I have a full
image copy every 5 minutes, but only one real MTA. I
will check the NAS option, if there is no other way.
Thanks Patrick Le 24/01/2017 à 13:45, Wietse Venema a
    écrit :

Patrick Chemla:

Hi, I have a running Fedora 24 emails server
using postfix 3.1.3, with courier. I wonder
how to build a pair of MTAs to secure emails
at all time, having 2 servers receiving the
emails, and users could connect to either
server to get emails, maybe on a load balanced
way. Problems are with synchronization when
receiving emails from outside, or emails read,
emails moved,

You need a redundant message store. In pre-cloud
times, people would use a NAS filer with redundant
disks, store email as maildir files (one per
message) and MDAs would mount that store via NFS.
Perhaps that model still works for you. Does
someone have a good guide, howto, doc to achieve
this?

Thanks for help. Patrick





--
Sent from my Android device with K-9 Mail. Please excuse my brevity. 





Re: load balanced emails servers pair

2017-01-24 Thread Patrick Chemla

Hi Wietse,

Of course I thought about such NAS solution, but I wanted to check if 
there is a way with 2 separate disks, with a kind of that could be aware 
of emails files changes.


Actually, the mail server run onto a VM, on a big server. I have another 
big server with same emails VM, and I just rsync --delete --update from 
the first one to the second. So I have a full image copy every 5 
minutes, but only one real MTA.


I will check the NAS option, if there is no other way.

Thanks
Patrick

Le 24/01/2017 à 13:45, Wietse Venema a écrit :

Patrick Chemla:

Hi,

I have a running Fedora 24 emails server using postfix 3.1.3, with courier.

I wonder how to build a pair of MTAs to secure emails at all time,
having 2 servers receiving the emails, and users could connect to either
server to get emails, maybe on a load balanced way.

Problems are with synchronization when receiving emails from outside, or
emails read, emails moved,

You need a redundant message store. In pre-cloud times, people
would use a NAS filer with redundant disks, store email as maildir
files (one per message) and MDAs would mount that store via NFS.
Perhaps that model still works for you.


Does someone have a good guide, howto, doc to achieve this?

Thanks for help.

Patrick







load balanced emails servers pair

2017-01-24 Thread Patrick Chemla

Hi,

I have a running Fedora 24 emails server using postfix 3.1.3, with courier.

I wonder how to build a pair of MTAs to secure emails at all time, 
having 2 servers receiving the emails, and users could connect to either 
server to get emails, maybe on a load balanced way.


Problems are with synchronization when receiving emails from outside, or 
emails read, emails moved,


Does someone have a good guide, howto, doc to achieve this?

Thanks for help.

Patrick




Re: hacker or server problem

2016-11-16 Thread Patrick Chemla

Le 16/11/2016 à 12:38, li...@lazygranch.com a écrit :

On Wed, 16 Nov 2016 02:26:13 -0800
"li...@lazygranch.com" <li...@lazygranch.com> wrote:


On Wed, 16 Nov 2016 11:52:14 +0200
Patrick Chemla <patrick.che...@perfaction.net> wrote:


Le 16/11/2016 à 11:45, li...@lazygranch.com a écrit :

Is this a hack or a server problem. IP was listed in abusedb
about a year ago.


Nov 16 09:14:36 theranch postfix/smtpd[6094]: connect from
unknown[87.236.215.11] Nov 16 09:14:36 theranch
postfix/smtpd[6094]: lost connection after AUTH from
unknown[87.236.215.11] Nov 16 09:14:36 theranch


# bzgrep -e 87.236.215.11 maillog | wc -l
  212

Three lines per hack. Make that 70 attempts. The stats line messes up
the line count.
First entry:Nov 16 09:13:45
Last entry: Nov 16 09:18:00
255 seconds
16.5 attempts a minute


16 Attempts per second, yes this is a hack attempt.

Protect yourself immediatly, even if he will surely need some (hundred 
of) thousands attempts to find a password.


Another problem is that he is taking your bandwith.

Patrick



Re: hacker or server problem

2016-11-16 Thread Patrick Chemla


Le 16/11/2016 à 11:45, li...@lazygranch.com a écrit :

Is this a hack or a server problem. IP was listed in abusedb about a
year ago.


Nov 16 09:14:36 theranch postfix/smtpd[6094]: connect from 
unknown[87.236.215.11]
Nov 16 09:14:36 theranch postfix/smtpd[6094]: lost connection after AUTH from 
unknown[87.236.215.11]
Nov 16 09:14:36 theranch postfix/smtpd[6094]: disconnect from 
unknown[87.236.215.11] ehlo=1 auth=0/1 commands=1/2
Nov 16 09:14:36 theranch postfix/smtpd[6094]: connect from 
unknown[87.236.215.11]
Nov 16 09:14:37 theranch postfix/smtpd[6094]: lost connection after AUTH from 
unknown[87.236.215.11]
Nov 16 09:14:37 theranch postfix/smtpd[6094]: disconnect from 
unknown[87.236.215.11] ehlo=1 auth=0/1 commands=1/2
Nov 16 09:14:37 theranch postfix/smtpd[6094]: connect from 
unknown[87.236.215.11]
Nov 16 09:14:38 theranch postfix/smtpd[6094]: lost connection after AUTH from 
unknown[87.236.215.11]
Nov 16 09:14:38 theranch postfix/smtpd[6094]: disconnect from 
unknown[87.236.215.11] ehlo=1 auth=0/1 commands=1/2
Nov 16 09:14:38 theranch postfix/smtpd[6094]: connect from 
unknown[87.236.215.11]
Nov 16 09:14:39 theranch postfix/smtpd[6094]: lost connection after AUTH from 
unknown[87.236.215.11]
Nov 16 09:14:39 theranch postfix/smtpd[6094]: disconnect from 
unknown[87.236.215.11] ehlo=1 auth=0/1 commands=1/2
Nov 16 09:14:39 theranch postfix/smtpd[6094]: connect from 
unknown[87.236.215.11]
Nov 16 09:14:39 theranch postfix/smtpd[6094]: lost connection after AUTH from 
unknown[87.236.215.11]
Nov 16 09:14:39 theranch postfix/smtpd[6094]: disconnect from 
unknown[87.236.215.11] ehlo=1 auth=0/1 commands=1/2
Nov 16 09:14:40 theranch postfix/smtpd[6094]: connect from 
unknown[87.236.215.11]
Nov 16 09:14:40 theranch postfix/smtpd[6094]: lost connection after AUTH from 
unknown[87.236.215.11]
Nov 16 09:14:40 theranch postfix/smtpd[6094]: disconnect from 
unknown[87.236.215.11] ehlo=1 auth=0/1 commands=1/2
Nov 16 09:18:00 theranch postfix/anvil[6096]: statistics: max connection rate 
70/60s for (smtp:87.236.215.11) at Nov 16 09:14:40
Nov 16 09:18:00 theranch postfix/anvil[6096]: statistics: max connection count 
1 for (smtp:87.236.215.11) at Nov 16 09:13:45
Nov 16 09:18:00 theranch postfix/anvil[6096]: statistics: max cache size 1 at 
Nov 16 09:13:45


Hi,

This is a trace of 6 connections tries from IP 87.236.215.11 with bad 
credential (user/passwd).


Someone is trying to enter your server emails. Call it a hack.

Patrick

www.top-secured.com



Re: relay local domains to a specific server

2010-05-24 Thread Patrick Chemla

Le 23/05/2010 22:03, Wietse Venema a écrit :

  Obviously you have something like...
  
 transport_maps = hash:/etc/postfix/transport
  
  in your main.cf, don't you?
   

This was my problem.

Sorry for wasting your time.

Thanks for help Gian Carlo

Patrick


relay local domains to a specific server

2010-05-23 Thread Patrick Chemla

Hi,

I am managing my emails on 2 Postfix 2.7 servers.

A front smtpd server receives all messages from outside and inside 
users, and a back server handles email boxes for local domains deliveries.


I am trying to send directly messages from the front smtpd to the back 
server without looking to MX from DNS.


So, in the /etc/postfix/transport file, I put lines as follow:

example1.com:[10.0.0.2]
example2.com:[10.0.0.2]
example3.com:[10.0.0.2]
example4.com:[10.0.0.2]

exampleN.com are domains to relay directly to internal server 10.0.0.2 
without looking to DNS (using brackets).


I have run postmap transport and postfix reload.

I made some simple tests puting mails through a telnet to port 25 of the 
front server.


It still lookup for MX for domains exampleN.com and delivers through an 
outside address. If I set this MX to be the smtpd front server, I wonder 
what will happen. At least, not what I want.


I made something wrong, but what?
Thanks for help.
Patrick



Re: relay local domains to a specific server

2010-05-23 Thread Patrick Chemla

Le 23/05/2010 18:20, Wietse Venema a écrit :

Patrick Chemla:
   

Hi,

I am managing my emails on 2 Postfix 2.7 servers.

A front smtpd server receives all messages from outside and inside
users, and a back server handles email boxes for local domains deliveries.

I am trying to send directly messages from the front smtpd to the back
server without looking to MX from DNS.

So, in the /etc/postfix/transport file, I put lines as follow:

example1.com:[10.0.0.2]
example2.com:[10.0.0.2]
example3.com:[10.0.0.2]
example4.com:[10.0.0.2]

exampleN.com are domains to relay directly to internal server 10.0.0.2
without looking to DNS (using brackets).

I have run postmap transport and postfix reload.

I made some simple tests puting mails through a telnet to port 25 of the
front server.

It still lookup for MX for domains exampleN.com and delivers through an
outside address. If I set this MX to be the smtpd front server, I wonder
what will happen. At least, not what I want.

I made something wrong, but what?
 

TO REPORT A PROBLEM see http://www.postfix.org/DEBUG_README.html#mail

TO (UN)SUBSCRIBE see http://www.postfix.org/lists.html

Thank you for using Postfix.
   

Wietse,

Does it mean that I did right and it could be a bug?

Thanks
Patrick



Re: relay local domains to a specific server

2010-05-23 Thread Patrick Chemla

Le 23/05/2010 19:16, Ralf Hildebrandt a écrit :

I made some simple tests puting mails through a telnet to port 25 of
  the front server.
  
  It still lookup for MX for domains exampleN.com and delivers through

  an outside address.
 

How do you know that?

   
I just look at the maillog and I can find the IP address of the external 
server where it tries to deliver.


Patrick



postfix multi-instances and qmail co-existence

2010-05-04 Thread Patrick Chemla

Hi,

I am trying to upgrade smoothly from qmail mono-instance to 
postfix-multi-instances.


When I start the new postfix installation, I get the warning:
postfix/postfix-script: warning: /usr/lib/sendmail and 
/usr/sbin/sendmail differ

postfix/postfix-script: warning: Replace one by a symbolic link to the other

and if I check the files:
ls -l /usr/lib/sendmail /usr/sbin/sendmail
lrwxrwxrwx 1 root root 23 jun 19  2009 /usr/lib/sendmail - 
/var/qmail/bin/sendmail

-rwxr-xr-x 1 root root 581869 avr 23 12:04 /usr/sbin/sendmail

Should I set the 2 linked to the same file ? /usr/sbin/sendmail ? 
/var/qmail/bin/sendmail ?


What will arrive if qmail uses /usr/sbin/sendmail or postfix uses 
/var/qmail/bin/sendmail ?


Whenever, postfix starts, all instances, and I can process messages. Is 
this a correct production environnement?


Thanks for help
Patrick




Re: postfix multi-instances and qmail co-existence

2010-05-04 Thread Patrick Chemla

Le 04/05/2010 10:07, Ralf Hildebrandt a écrit :

  What will arrive if qmail uses /usr/sbin/sendmail or postfix uses
  /var/qmail/bin/sendmail ?
 

? Nothing, you disabled qmail I hope.

   
As I wrote, I am moving smoothly, means that actually the regular 
traffic is still going through the qmail interface/port, and I will move 
traffic depending on each domain I manage.


The idea is to separate domains on different queues, for domains with 
same characteristics of traffic.


Some domains have special settings on DKIM, rate, smtp recipient 
server,... so I need to check each one  before I move completly.


So, should I change the links to sendmail?

Patrick



group message delivery per domain

2010-05-04 Thread Patrick Chemla

Hi,

I wonder if there is a way to group messages deliveries to specific 
domains on a single persistent smtp connexion.


I mean, if in the queue, I have some messages to deliver to one domain, 
I would like to deliver them during one single persistent connexion, 
instead of closing and opening a new connexion for each message.


Some ISPs told me they prefer to receive for example 50 messages to 
different recipients during one connexion, than getting 50 different new 
connexion.


Maybe there is some parameters to manage persistent connexions?

Thanks for help.

Patrick




Re: monitoring sevreal postfix serevrs.

2010-04-27 Thread Patrick Chemla

Le 27/04/2010 10:54, Patrick Ben Koetter a écrit :

* Israel Garciaigalva...@gmail.com:
   

  I have about 20 debian servers send all mail through a loadbalancer
  (haproxy) with 2backend smarthosts which send emails to internet. I
  have pflogsumm running only  on every smarhost. As every smarthost see
  on IP source (haproxy) I can not get email stats from every debian
  server. Questions: How can I get email stats of my servers in this
  scenario?
 

Send log from all nodes to a central log server. Use rsyslogd with RELP to get
reliability. Use rsyslog log templates to cut down on bandwidth usage if
Postfix log is to verbose.
   
I am also running a big number of servers. To avoid heavy load on the 
network and on the central server processing a huge amount of data, I 
wrote a script on each node to extract statistics and send a very small 
amount of data to the central server with exactly what I want to monitor.

Then this one as just to generate graphs. I am using munin.
Patrick


Re: monitoring sevreal postfix serevrs.

2010-04-27 Thread Patrick Chemla

Le 27/04/2010 15:27, Israel Garcia a écrit :

could you share the script with us?

   


You can put whatever you want in the script. I am monitoring different 
parts of the servers.


Extract:

# Identify the server by the last digit of his IP
ip3d=`/sbin/ifconfig| grep Bcast|grep 10.0.0|cut -d: -f2 | cut -d  
-f1|cut -d. -f4`


# Get queue stats from qmail
qmailctl stat|grep queue:|awk -vip=$ip3d '{print qmail a ip,$NF}'  
/tmp/qstat$ip3d


# Get active queue stats from postfix
find /var/spool/postfix/active -type f|wc -l|awk -vip=$ip3d '{print 
postfix a ip,$NF}'  /tmp/qstat$ip3d


# Add here whatever number you want to send to the server : cpu, 
memory, disk space, 


# Send the file to the central. You can  use shared disk. Some of my 
servers are remote, so I use scp.

scp /tmp/qstat$ip3d 10.0.0.252:/tmp/muninqstat

Munin has also very powerful specific scripts, but on large architecture 
it creates a huge amount of traffic and load both on nodes and server to 
update the database. This way, I also use munin, but I split better the 
load  and I minimize connexions.


On the server side, with grep and awk extract from all qstatXXX files 
the data you want to show for each graph, you can generate with whatever 
tool. There I do it with a munin script, it's simple.


Patrick


multi instances and multi interfaces

2010-04-25 Thread Patrick Chemla

Hi,

I am running a smtp relay for different domains, and I want to separate 
all traffic.


I would like to bind for each domain  the same IP address to receive and 
send messages, and it should be for each domain its own public IP address.


I understand that with postfix 2.7.0 multi-instances feature I can 
handle separate inbound queues on one system, so each queue can have its 
own IP address, but what about the outbound messages ?


When users will post messages to relay outside, will the sending process 
bind the output on the same IP interface from where it got the message? 
I wonder it will be routed according to the route table through the 
default interface.


Is there a way to send through separate interfaces ?

Thanks for help
Patrick




routing according to sender domain, not recipient domain

2010-03-22 Thread Patrick Chemla

Hi,

I would like to relay messages to specific MX servers according to 
sender domain, not recipient domain.


I saw the primitive sender_dependent_relayhost_maps but I don't know how 
to use it.


Should I create a list and postmap it and set a line in my main.cf like :

sender_dependent_relayhost_maps = hash:/etc/postfix/senderdomainroutes  ?

Thanks for help

Patrick
*
*


How to manage local blacklist on my postfix relay?

2010-02-18 Thread Patrick Chemla

Hi,

I have a Postfix 2.6 relaying tons of emails to millions email addresses 
and domains.


I have listed tens of thousands of email addresses and domains to which 
I don't want to relay any more.


Is there a way to manage a local blacklist without spamassassin?  
However, up to now I think spamassassin is for local delivery, not relay.


So I have a file of more than  100,000 email addresses and another made 
of bad domains.


I can write scripts in shell, php, perl,

Your help will be welcomed.

Patrick


Re: load balancing among mail servers

2010-02-16 Thread Patrick Chemla

Le 16/02/2010 17:47, donovan jeffrey j a écrit :

DNS round robin is bad, it works but is defective for real load
balancing. The client choose the IP to use, this is random, and
after can use the same ip for a while... this is not random.



Again, I am doing every days exactly what required at the beginning of 
this thread.


I have one Postfix server who simply relays all emails to send to a farm 
of 40 mail sub-servers.


To load balance, I simply use a local DNS who manage a local domain. All 
40 sub-servers are identified as equivalent MX of this local domain. The 
Postfix server just ask the DNS what is the MX of this local domain, and 
get a name /ip from the DNS.


I am very satisfied of this load balancing.

Again, I said before that the statistics show a difference of less than 
2% of the traffic to the sub-server who work the most and the one who 
work the less. It is, as I think, very very well load balanced.


When a sub-server fails, some messages are stuck in the Postfix, but a 
very small percentage. Immediatly after you restart the sub-server, or 
you put it out of the DNS, all messages are processed.


I don't think there is a need for more precision in the load balancing.

I don't think there is a need for keepalive, or any expensive device to 
do it.


Patrick



Re: Huge active queue and system idle, not delivering

2010-01-10 Thread Patrick Chemla

Wietse,

Please try the following, as asked half a week ago:

 postconf -e smtp_connection_cache_on_demand=no
 postfix reload

and report if this makes a difference.
Wietse
 

I have tested this since yesterday night.

I got some problems with Linux per user number of processes limit. I 
fixed it. I also increased some delivery concurrency  figures, and now I 
can see up to 1300 processes delivering emails to the qmail servers.


I had a few minutes shot today at a rate of 6300 emails per minute. I 
ran a full hour at 180,000 emails per hour. The outbound line was saturated.


CPU is about 30% loaded, no Wait I/O, no swap, memory is large.

I think I will reach about 600,000 emails per hour if I fix some timeout 
on the qmails (replace by postfix?). Maybe I could reach 1 million?


The full architecture that I plan will include 2 to 3 clustered postfix 
relays and 50 2nd level qmails(or postfix) delivery servers, each with 3 
to 5 IP addresses, and upgraded outbound internet connection.


With your help, I better understand now the impact of timeout and 
concurrency parameters. In fact, delivery was blocked because postfix 
was trying to reuse connections, so was waiting each email to complete 
to send the next one. Also, because hundreds processes were created at 
start time to manage inbound messages, there were no slots to fork 
processes to deliver messages on the other hand. Same problem caused 
very slow DNS and EHLO, because no available slots to fork.


Of course, if you want me to post my conf, I will with pleasure.

Many thanks to you, to Victor and Stan.

Patrick



Re: Huge active queue and system idle, not delivering

2010-01-10 Thread Patrick Chemla

Le 10/01/2010 23:58, Stan Hoeppner a écrit :

On a technical level I'm happy you got it working.  Just please tell us you're
not sending mass spam with this setup.

--
Stan
   


I have to do it for a customer who send as he said, only opt-in mass 
emails. He has a big blacklisted email database where he keeps all 
unsubscribe messages. He said he has the right filters not to send 
unwanted emails.




Thanks
Patrick



Re: Huge active queue and system idle, not delivering

2010-01-10 Thread Patrick Chemla

Le 11/01/2010 01:13, Wietse Venema a écrit :

Patrick Chemla:
   

Wietse,
 

Please try the following, as asked half a week ago:

  postconf -e smtp_connection_cache_on_demand=no
  postfix reload

and report if this makes a difference.
Wietse

 

I have tested this since yesterday night.

I got some problems with Linux per user number of processes limit. I
fixed it. I also increased some delivery concurrency  figures, and now I
can see up to 1300 processes delivering emails to the qmail servers.

I had a few minutes shot today at a rate of 6300 emails per minute. I
ran a full hour at 180,000 emails per hour. The outbound line was saturated.

CPU is about 30% loaded, no Wait I/O, no swap, memory is large.

I think I will reach about 600,000 emails per hour if I fix some timeout
on the qmails (replace by postfix?). Maybe I could reach 1 million?
 

OK, so you can turn back on that connection caching. Note that
qmail creates and destroys two processes per SMTP session, so
reusing a session is also a win from a CPU resource point of view.



Wietse
   
If I do so, will postfix open more than one connexion to each qmail for 
parallel deliveries?
I am afraid that if we use connection caching this will create a single 
queue on each qmail. As far as I have available resources, I think 
prefer parallel deliveries.


Patrick



Re: Huge active queue and system idle, not delivering

2010-01-10 Thread Patrick Chemla

Le 11/01/2010 09:27, Stan Hoeppner a écrit :

Patrick Chemla put forth on 1/11/2010 1:02 AM:
   

Le 10/01/2010 23:58, Stan Hoeppner a écrit :
 

On a technical level I'm happy you got it working.  Just please tell
us you're
not sending mass spam with this setup.

--
Stan

   

I have to do it for a customer who send as he said, only opt-in mass
emails. He has a big blacklisted email database where he keeps all
unsubscribe messages. He said he has the right filters not to send
unwanted emails.
 

Sigh...  This doesn't pass the sniff test.  I fear we've helped enable the
sending of mass UBE.  Patrick would you mind providing the IP netblock(s) you
will be sending these mass mailings from?  Or provide them to me off list
please?  Thanks.

--
Stan
   
Don't be afraid Stan. They work only on french market, maybe also on 
french people who have a mailbox overseas. You have very very very low 
chance to be concerned.

Patrick



Re: Huge active queue and system idle, not delivering

2010-01-09 Thread Patrick Chemla

Hi,

I will try all your advises, but something still very strange for me:

We see that postfix logs show that ehlo process is very slow through 
postfix but very fast by hand. Even I have recorded through 
tcpdump/WireShark and I can see that messages are sent very very very 
quickly in about 1 second.


But still messages are sent at a rate of a dozen in 10 seconds. That 
means that messages are sent 1 by one.


If connexion to qmail servers are slow, or if qmails are mis-parameted, 
too slow or anything else, When I do netstat -apn |grep :25 I get only a 
few connexions from postfix server to qmail servers. Even if DNS+EHLO 
are slow, and more, because DNS+EHLO seem to be slow, why I don't see 
hundreds TCP connexions ESTABLISHED ?


I expected that postfix will deliver on 30 qmail servers at the same 
time, and should manage hundreds parallel deliveries, hundreds parallel 
connexions. Is there some parameter or some conception rule that refrain 
him to do so?


I expected that postfix will full up his own CPU/memory creating these 
parallel delivery processes or/and will wait after the qmail servers, 
but on all servers at the same time, on multiple connections to each one.


Am I correct ? or I am dreaming of another mail transport package?

Patrick



Re: Huge active queue and system idle, not delivering

2010-01-09 Thread Patrick Chemla

Hi all,

I got these statistics:

Jan  9 19:15:21 postfix postfix/scache[18038]: statistics: start 
interval Jan  9 19:09:03
Jan  9 19:15:21 postfix postfix/scache[18038]: statistics: domain lookup 
hits=110 miss=89 success=55%
Jan  9 19:15:21 postfix postfix/scache[18038]: statistics: address 
lookup hits=0 miss=2492 success=0%
Jan  9 19:15:21 postfix postfix/scache[18038]: statistics: max 
simultaneous domains=1 addresses=4 connection=4



What means miss=89 success=55%, miss=2492 success=0%?

Thanks
Patrick







Re: Huge active queue and system idle, not delivering

2010-01-09 Thread Patrick Chemla

Hi Stan,

Thanks for your interest.

Le 09/01/2010 20:21, Stan Hoeppner a écrit :

Patrick Chemla put forth on 1/9/2010 11:17 AM:
   

Hi all,

I got these statistics:

Jan  9 19:15:21 postfix postfix/scache[18038]: statistics: start
interval Jan  9 19:09:03
Jan  9 19:15:21 postfix postfix/scache[18038]: statistics: domain lookup
hits=110 miss=89 success=55%
Jan  9 19:15:21 postfix postfix/scache[18038]: statistics: address
lookup hits=0 miss=2492 success=0%
Jan  9 19:15:21 postfix postfix/scache[18038]: statistics: max
simultaneous domains=1 addresses=4 connection=4


What means miss=89 success=55%, miss=2492 success=0%?
 

http://www.postfix.com/CONNECTION_CACHE_README.html

   
I wen t there but did not find explanations about miss address lookup or 
miss domain lookup.
While I have 122,000 messages in active queue I still don't understand 
why statistics show max simultaneous domains=1. It should be dozens , or 
hundreds.


Patrick


--
Stan
   




Re: Huge active queue and system idle, not delivering

2010-01-09 Thread Patrick Chemla

Le 09/01/2010 20:54, Stan Hoeppner a écrit :

Patrick Chemla put forth on 1/9/2010 12:37 PM:

   

I wen t there but did not find explanations about miss address lookup or
miss domain lookup.
While I have 122,000 messages in active queue I still don't understand
why statistics show max simultaneous domains=1. It should be dozens , or
hundreds.
 

Those are statistics relating to scache performance.  It tells you how many
domains or addresses were able to be delivered via scache reuse.  I.e. how many
emails Postfix was able to send through an already open SMTP connection to a
given host.

Since all of your qmail hosts are configured identically, and should be able to
relay mail bound for any destination on the internet, you should never see
anything less than ~100% in those statistics, _unless_ there is some other kind
of problem.

   


You mean 100% success?

If your qmail servers are rate limiting via any method, and Postfix is
attempting to send 2000 emails per minute down that one SMTP connection, when
qmail blocks individual deliveries for any reason, those scache failure
statistics will increase.

   
Before I set up the postfix relay to load balance between 30 qmail 
servers, each of them was able to accept in his own queue hundreds 
thousands email. Email were sent by campaigns of thousands balanced on 3 
qmails servers, each one full in CPU/memory working hard to deliver.


Instead of sending each campaign on only 3 qmails, I though that by 
sending each campaign on 30 qmails I will cut each one load by ten and 
speed up deliveries. But now, postfix is retaining the emails in his own 
queue, not pushing the queue down to the qmails.


Postfix server and qmail servers are all about 90%cpu free. only 1 to 9 
connexions exist at a time from postfix to qmails.


This is exactly what I would like to append: Instead of a queue of 
122,000 on postfix, I expect to have each qmail with a queue of 4000.


Qmails did this before I set up postfix.

Patrick


--
Stan
   




Re: Huge active queue and system idle, not delivering

2010-01-08 Thread Patrick Chemla

Le 08/01/2010 03:03, Wietse Venema a écrit :

Patrick Chemla:
   

But the CPU of the box is idle more than 80%. It is clear that it is not a
matter of CPU, nor memory, nor disk. Something in the number of
processes/users/simultaneous tasks is blocking.
 

Indeed, the symptom of blocking is in the third field of
the Postfix delays logging.

The format of the delays=a/b/c/d logging is as follows:

o  a = time from message arrival to last active queue entry

o  b = time from last active queue entry to connection setup

o  c = time in connection setup, including DNS, EHLO and TLS

o  d = time in message transmission

In your case, it takes a minute or more to set up the connection
including DNS lookup and EHLO handshake. That is holding up your mail.

- Check if the qmail servers are responsive (telnet hostname 25).

   
qmail are responsive. I made some arrangements to my DNS. DNS is better 
now, but the connexion is still very slow. I saw this morning c=285.

- Check if your Postfix needs a /var/spool/postfix/etc/resolv.conf
   file, and if that file is consistent with /etc/resolv.conf. If
   Postfix needs /var/spool/postfix/etc/resolv.conf and the file
   is missong or contains a bogus server that will add time to
   your deliveries.

   

Hi Wietse,
How do I know if  Postfix needs a /var/spool/postfix/etc/resolv.conf
directory /var/spool/postfix/etc doesn't exist.



- If they aren't, increase the concurrency on the qmail side.

   

conccurency =100. It's already a large number. I can increase it.

Wietse
   

Thanks
Patrick



Re: Huge active queue and system idle, not delivering

2010-01-08 Thread Patrick Chemla

Le 08/01/2010 00:43, Victor Duchovni a écrit :

On Fri, Jan 08, 2010 at 12:30:34AM +0200, Patrick Chemla wrote:

   

Jan  7 22:02:57 postfix postfix/qmgr[26441]: 5B91F873F6: removed
Jan  7 22:02:57 postfix postfix/smtp[27180]: 375DDD5923:
to=lexoti...@gmail.com, relay=a139.localpc2105.com[10.0.0.139]:25,
conn_use=59, delay=61550, delays=17019/44435/96/0.17, dsn=2.0.0,
status=sent (250 ok 1262894577 qp 12113)
 

This recipient does not match the destination that is clogging the
queue. Is the queue clogged with postmaster notices. I never enable
any postmaster notices, they don't scale.

notify_classes =

   

done, no change.

This said, the 96 seconds of connection setup latency is an obvious and
severe problem. Why on earth does it take 96 seconds to complete a HELO
handshake with a139.localpcc2105.com? You are not going to get much
mail out if each delivery takes 96 seconds...

Is your Postfix server's IP address resolvable on the qmail systems?
   

Should it be? qmail accept all RELAY CLIENT from local network.

Are they doing some sort of pre-banner delay? ...


   

When I do  telnet a139.localpc2105.com 25, I get immediate response.

Jan  7 22:02:58 postfix postfix/smtp[27070]: 7F0F2943B3:
to=gpo...@wanadoo.fr, relay=a70.localpc2105.com[10.0.0.70]:25,
conn_use=10, delay=73795, delays=29264/44481/50/0.21, dsn=2.0.0,
status=sent (250 ok 1262894577 qp 23067)
 

Once again, 50 seconds is severely crippled.

   

When I telnet a70.localpc2105.com 25 I get an immediate response.

I have checked my local DNS. There were some troubles, and I made some 
improvements. I have now 2 local caching DNS respawning fast. All qmail 
servers addresses are in the postfix /etc/hosts to avoid Ip lookup.
I have checked qmails servers, nothing has changed since they were able 
to have a queue of 200,000 messages, but they have now a few hundreds only.
I have calculated average times to complete HELO. All qmails are in the 
same kind of value around 2 minutes. Not any one is better than others. 
Again, each was handling a queue of hundreds thousands before I set up 
the postfix relay to load balance.

I really don't have a clue. I don't know where to look.

Jan  7 22:02:58 postfix postfix/smtp[27050]: 32BB182182:
to=gmarin-jardins-lois...@wanadoo.fr,
relay=a139.localpc2105.com[10.0.0.139]:25, conn_use=48, delay=73799,
delays=29268/44466/65/0.28, dsn=2.0.0, status=sent (250 ok 1262894578 qp
12121)
 

This is enough. Fix this.

   

How I can fix it if it works fine through telnet?

Where are the deliveries to the clogged destination???

   

Sorry, I don't understand this question. Please be clear.

Patrick



Huge active queue and system idle, not delivering

2010-01-07 Thread Patrick Chemla

Hi,

I am running Postfix 2.5.6 on a Fedora 11 Linux system on a hardware 
based Intel  I5/750 Quad Core, 8 Gb memory, 160Gb SSD hard disk.


Incoming messages are entering very fast (500 smtp processes declared) 
and the active queue is actually of 2 millions messages waiting for 
delivery.


The delivery, for all messages should go through a farm of 30 MX servers 
from domain localpc2105.com, on load balancing through DNS resolution. 
DNS server is of course local. All 30 MX servers are running qmail. All 
of them are more than 90% idle. Before I set up my postfix server, email 
were sent directly to the qmail servers, and qmail was running at full 
CPU. So I am sure that qmail can handle much more faster. I have set up 
the postfix server to load balance the load between all the 30 qmail 
servers to avoid situation where some were running at full charge and 
others were not working.


Postfix server, 30 Qmail servers, DNS are on the same 1Gb LAN.

I don't need any messages or clients or recipients control at this 
stage. No anti spam, no anti virus. All this is already done by qmail 
servers, despite my plan is to replace all qmails by postfix.


CPU is more than 85% idle on my postfix I5/750 box, but the outbound 
queue is very very slow.


It seems that something refrain qmgr to work at full range, despite the 
parameters


here is my main.cf file:

queue_directory = /var/spool/postfix
command_directory = /usr/sbin
daemon_directory = /usr/libexec/postfix
data_directory = /var/lib/postfix
mail_owner = postfix
myhostname = postfix.proacti5.net
mydomain = localpc2105.com
inet_interfaces = all
mydestination = $myhostname, localhost.$mydomain, localhost
unknown_local_recipient_reject_code = 550
mynetworks = 172.27.27.0/24, 10.0.0.0/24, 127.0.0.0/24
relayhost = $mydomain
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
local_destination_recipient_limit = 500
local_destination_concurrency_limit = 50
debug_peer_level = 8
debug_peer_list = orange.fr
debugger_command =
 PATH=/bin:/usr/bin:/usr/local/bin:/usr/X11R6/bin
 ddd $daemon_directory/$process_name $process_id  sleep 5
sendmail_path = /usr/sbin/sendmail.postfix
newaliases_path = /usr/bin/newaliases.postfix
mailq_path = /usr/bin/mailq.postfix
setgid_group = postdrop
html_directory = no
manpage_directory = /usr/share/man
sample_directory = /usr/share/doc/postfix-2.5.6/samples
readme_directory = /usr/share/doc/postfix-2.5.6/README_FILES
inet_protocols = all
default_process_limit = 1000
initial_destination_concurrency = 100
transport_initial_destination_concurrency = 100
default_destination_concurrency_failed_cohort_limit = 10
default_destination_recipient_limit = 200
transport_destination_recipient_limit = 100
default_delivery_slot_cost = 30
default_minimum_delivery_slots = 30
default_delivery_slot_discount = 100
smtpd_peername_lookup = no
default_recipient_limit = 200
mailbox_size_limit = 512000
qmgr_message_active_limit = 200
qmgr_message_recipient_limit = 200
default_destination_concurrency_limit = 500
lmtp_destination_concurrency_limit = $default_destination_concurrency_limit
relay_destination_concurrency_limit = $default_destination_concurrency_limit
smtp_destination_concurrency_limit = $default_destination_concurrency_limit
max_use = 1000
mime_nesting_limit = 100
qmgr_fudge_factor = 200
queue_file_attribute_count_limit = 250
smtpd_history_flush_threshold = 100
smtpd_junk_command_limit = 100
smtp_connect_timeout = 10s
smtp_data_done_timeout = 10s
smtp_mail_timeout = 5s


Here is my master.cf file:


service typeprivate
(yes)
unpriv
(yes)
chroot
(yes)
wakeup
(never)
maxproc
(100)
command + args
smtpinetn   -   n   -   -   smtpd
pickup  fifon   -   n   60  1   pickup
cleanup unixn   -   n   -   0   cleanup
qmgrfifon   -   n   30  1   qmgr
tlsmgr  unix-   -   n   1000?   1   tlsmgr
rewrite unix-   -   n   -   -   trivial-rewrite
bounce  unix-   -   n   -   0   bounce
defer   unix-   -   n   -   0   bounce
trace   unix-   -   n   -   0   bounce
verify  unix-   -   n   -   1   verify
flush   unixn   -   n   1000?   0   flush
proxymapunix-   -   n   -   -   proxymap
proxywrite  unix-   -   n   -   1   proxymap
smtpunix-   -   n   -   -   smtp
relay   unix
-o smtp_fallback_relay= -   -   n   -   -   smtp
showq   unixn   -   n   -   -   showq
error   unix-   -   n   -   -   error
retry   unix-   -   n   -   -   error
discard unix-   -   n   -   -   discard
local   unix- 

Re: Huge active queue and system idle, not delivering

2010-01-07 Thread Patrick Chemla

Le 07/01/2010 20:03, Barney Desmond a écrit :

2010/1/8 Patrick Chemlapatrick.che...@perfaction.net
   

Incoming messages are entering very fast (500 smtp processes declared) and
the active queue is actually of 2 millions messages waiting for delivery.
snip
here is my main.cf file:
 

That's some very thorough information, you've provided plenty of
context and clear description, which is great. While I lack sufficient
knowledge to provide thoughts on the bottlenecking, I *do* expect that
people will want to see the output of `postconf -n`, instead of your
main.cf (to ensure we see what postfix actually sees and uses).

   

Here is postconf -n
alias_database = hash:/etc/aliases
alias_maps = hash:/etc/aliases
command_directory = /usr/sbin
config_directory = /etc/postfix
daemon_directory = /usr/libexec/postfix
data_directory = /var/lib/postfix
debug_peer_level = 8
debug_peer_list = orange.fr
default_delivery_slot_cost = 30
default_delivery_slot_discount = 100
default_destination_concurrency_failed_cohort_limit = 10
default_destination_concurrency_limit = 500
default_destination_recipient_limit = 200
default_minimum_delivery_slots = 30
default_process_limit = 1000
default_recipient_limit = 200
html_directory = no
inet_interfaces = all
inet_protocols = all
initial_destination_concurrency = 100
lmtp_destination_concurrency_limit = $default_destination_concurrency_limit
local_destination_concurrency_limit = 50
local_destination_recipient_limit = 500
mail_owner = postfix
mailbox_size_limit = 512000
mailq_path = /usr/bin/mailq.postfix
manpage_directory = /usr/share/man
max_use = 1000
mime_nesting_limit = 100
mydestination = $myhostname, localhost.$mydomain, localhost
mydomain = localpc2105.com
myhostname = postfix.proacti5.net
mynetworks = 172.27.27.0/24, 10.0.0.0/24, 127.0.0.0/24
newaliases_path = /usr/bin/newaliases.postfix
qmgr_fudge_factor = 200
qmgr_message_active_limit = 200
qmgr_message_recipient_limit = 200
queue_directory = /var/spool/postfix
queue_file_attribute_count_limit = 250
readme_directory = /usr/share/doc/postfix-2.5.6/README_FILES
relay_destination_concurrency_limit = $default_destination_concurrency_limit
relayhost = $mydomain
sample_directory = /usr/share/doc/postfix-2.5.6/samples
sendmail_path = /usr/sbin/sendmail.postfix
setgid_group = postdrop
smtp_connect_timeout = 10s
smtp_data_done_timeout = 10s
smtp_destination_concurrency_limit = $default_destination_concurrency_limit
smtp_mail_timeout = 5s
smtpd_history_flush_threshold = 100
smtpd_junk_command_limit = 100
smtpd_peername_lookup = no
unknown_local_recipient_reject_code = 550



Can you clarify what you mean by 500 smtp processes declared? A
sample output from qshape also wouldn't go astray either
(http://www.postfix.org/qshape.1.html).
Here is qshape:  T  5 10 20 40 80  160   320   640 
1280 1280+
 TOTAL 133000  0  0  0  0  0 2470 40538 80844 
7167  1981
wanadoo.fr  61955  0  0  0  0  0 2469 26830 31340 
126056
 orange.fr   4171  0  0  0  0  00  1176  2144  
540   311
 skynet.be   3286  0  0  0  0  00 1  
32840 1
  aliceadsl.fr   3259  0  0  0  0  0054  3169   
2511
   aol.com   3150  0  0  0  0  00  1545  1524   
4041
   free.fr   2138  0  0  0  0  00   453  1561   
8935
sfr.fr840  0  0  0  0  0023   
8161 0
hotmail.fr679  0  0  0  0  00   150   420   
1297
telenet.be658  0  0  0  0  00 0   
6580 0
 gmail.com358  0  0  0  0  00   157   145   
1145
   hotmail.com325  0  0  0  0  0044   220   
2041
   neuf.fr252  0  0  0  0  0062   176   
14 0
9online.fr250  0  0  0  0  00 6   
2440 0
   cegetel.net195  0  0  0  0  0026   
155410
   laposte.net183  0  0  0  0  005193   
1524
  swing.be141  0  0  0  0  00 2   
1390 0
  9business.fr111  0  0  0  0  0023
853 0
sonepar.fr107  0  0  0  0  0033
722 0
axa.fr103  0  0  0  0  0030
671 5


most of the messages stay in the queue for hours.



You're provided some
proportional figures (percentages), but some solid throughput numbers
would be good too. Eg. We're injecting 2 million messages to the
postfix box, we expect to enqueue them in X hrs, but it takes Y hrs,
and they're only leaving the postfix box at Z messages/sec. I see you
said I just found that Postfix could send 1 million emails per hour
when I send less than a half million in 24 hours, but I can't make
sense of 

Re: Huge active queue and system idle, not delivering

2010-01-07 Thread Patrick Chemla

Le 07/01/2010 20:00, Wietse Venema a écrit :

Patrick Chemla:
   

Hi,

I am running Postfix 2.5.6 on a Fedora 11 Linux system on a hardware
based Intel  I5/750 Quad Core, 8 Gb memory, 160Gb SSD hard disk.

Incoming messages are entering very fast (500 smtp processes declared)
and the active queue is actually of 2 millions messages waiting for
delivery.

The delivery, for all messages should go through a farm of 30 MX servers
from domain localpc2105.com, on load balancing through DNS resolution.
DNS server is of course local. All 30 MX servers are running qmail. All
of them are more than 90% idle. Before I set up my postfix server, email
were sent directly to the qmail servers, and qmail was running at full
CPU. So I am sure that qmail can handle much more faster. I have set up
the postfix server to load balance the load between all the 30 qmail
servers to avoid situation where some were running at full charge and
others were not working.
 

http://www.postfix.org/DEBUG_README.html#logging

Wietse
   


Here the logs:

Jan  6 23:12:48 postfix postfix/qmgr[31260]: warning: to turn off these 
warnings specify: qmgr_clog_warn_time = 0
Jan  6 23:19:39 postfix postfix/qmgr[31260]: warning: mail for 
localpc2105.com is using up 461335 of 461335 active queue entries
Jan  6 23:19:39 postfix postfix/qmgr[31260]: warning: you may need to 
increase the main.cf smtp_destination_concurrency_limit from 100
Jan  6 23:19:39 postfix postfix/qmgr[31260]: warning: please avoid 
flushing the whole queue when you have
Jan  6 23:19:39 postfix postfix/qmgr[31260]: warning: lots of deferred 
mail, that is bad for performance
Jan  6 23:19:39 postfix postfix/qmgr[31260]: warning: to turn off these 
warnings specify: qmgr_clog_warn_time = 0
Jan  6 23:24:51 postfix postfix/qmgr[31260]: warning: mail for 
localpc2105.com is using up 461086 of 461086 active queue entries
Jan  6 23:24:51 postfix postfix/qmgr[31260]: warning: you may need to 
increase the main.cf smtp_destination_concurrency_limit from 100
Jan  6 23:24:51 postfix postfix/qmgr[31260]: warning: please avoid 
flushing the whole queue when you have
Jan  6 23:24:51 postfix postfix/qmgr[31260]: warning: lots of deferred 
mail, that is bad for performance
Jan  6 23:24:51 postfix postfix/qmgr[31260]: warning: to turn off these 
warnings specify: qmgr_clog_warn_time = 0
Jan  6 23:29:51 postfix postfix/qmgr[31260]: warning: mail for 
localpc2105.com is using up 460872 of 460872 active queue entries
Jan  6 23:29:51 postfix postfix/qmgr[31260]: warning: you may need to 
increase the main.cf smtp_destination_concurrency_limit from 100
Jan  6 23:29:51 postfix postfix/qmgr[31260]: warning: please avoid 
flushing the whole queue when you have
Jan  6 23:29:51 postfix postfix/qmgr[31260]: warning: lots of deferred 
mail, that is bad for performance
Jan  6 23:29:51 postfix postfix/qmgr[31260]: warning: to turn off these 
warnings specify: qmgr_clog_warn_time = 0
Jan  6 23:35:51 postfix postfix/qmgr[31260]: warning: mail for 
localpc2105.com is using up 460025 of 460025 active queue entries
Jan  6 23:35:51 postfix postfix/qmgr[31260]: warning: you may need to 
increase the main.cf smtp_destination_concurrency_limit from 100
Jan  6 23:35:51 postfix postfix/qmgr[31260]: warning: please avoid 
flushing the whole queue when you have
Jan  6 23:35:51 postfix postfix/qmgr[31260]: warning: lots of deferred 
mail, that is bad for performance
Jan  6 23:35:51 postfix postfix/qmgr[31260]: warning: to turn off these 
warnings specify: qmgr_clog_warn_time = 0
Jan  6 23:40:51 postfix postfix/qmgr[31260]: warning: mail for 
localpc2105.com is using up 460283 of 460283 active queue entries
Jan  6 23:40:51 postfix postfix/qmgr[31260]: warning: you may need to 
increase the main.cf smtp_destination_concurrency_limit from 100
Jan  6 23:40:51 postfix postfix/qmgr[31260]: warning: please avoid 
flushing the whole queue when you have
Jan  6 23:40:51 postfix postfix/qmgr[31260]: warning: lots of deferred 
mail, that is bad for performance
Jan  6 23:40:51 postfix postfix/qmgr[31260]: warning: to turn off these 
warnings specify: qmgr_clog_warn_time = 0
Jan  6 23:47:21 postfix postfix/qmgr[31260]: warning: mail for 
localpc2105.com is using up 459714 of 459714 active queue entries
Jan  6 23:47:21 postfix postfix/qmgr[31260]: warning: you may need to 
increase the main.cf smtp_destination_concurrency_limit from 100
Jan  6 23:47:21 postfix postfix/qmgr[31260]: warning: please avoid 
flushing the whole queue when you have
Jan  6 23:47:21 postfix postfix/qmgr[31260]: warning: lots of deferred 
mail, that is bad for performance
Jan  6 23:47:21 postfix postfix/qmgr[31260]: warning: to turn off these 
warnings specify: qmgr_clog_warn_time = 0
Jan  6 23:52:21 postfix postfix/qmgr[31260]: warning: mail for 
localpc2105.com is using up 459491 of 459491 active queue entries
Jan  6 23:52:21 postfix postfix/qmgr[31260]: warning: you may need to 
increase the main.cf smtp_destination_concurrency_limit from 100
Jan

Re: Huge active queue and system idle, not delivering

2010-01-07 Thread Patrick Chemla

Le 07/01/2010 23:47, Stefan Caunter a écrit :

On Thu, Jan 7, 2010 at 1:25 PM, Patrick Chemla
patrick.che...@perfaction.net  wrote:

   

said I just found that Postfix could send 1 million emails per hour
when I send less than a half million in 24 hours, but I can't make
sense of that, sorry.

   

I have to inject 2 to 4 millions emails to the postfix box in 24 hours, and
I expect to deliver within the same delay.
Actually, I can't deliver more than 500,000 per 24h hours.
 

It could be viewed that half a million delivered in 24 hours is fine.
Are you signing the mail? This can help with delivery rates to the
large webmailer mx destinations.

Stef

   


Half a million is 4 times lower than what we have done with qmail 
servers. Email are signed, but not from Postfix. Postfix must only relay 
mails from clients to local MXs. These local MXs will assume deliveries 
to the outside. Mail queue should be on these MXs, because they are 
dependant on final destinations.

But the CPU of the box is idle more than 80%. It is clear that it is not a
matter of CPU, nor memory, nor disk. Something in the number of
processes/users/simultaneous tasks is blocking.




 




Re: Huge active queue and system idle, not delivering

2010-01-07 Thread Patrick Chemla

Le 07/01/2010 20:37, Victor Duchovni a écrit :

On Thu, Jan 07, 2010 at 08:29:44PM +0200, Patrick Chemla wrote:

   

Here the logs:
 

This is just the qmgr(8) warnings about a clogged queue. Other than
telling us that all the mail is going to localpc2105.com, this
is not very useful. Where are the logs from smtp(8)?

What transport is localpc2105.com destined for? Any earlier
logging about actual delivery attempts for this destination?

   


Victor, thank you for your interest.

Daily logs are huge.

Here is a sample of deliveries:
Jan  7 22:02:57 postfix postfix/qmgr[26441]: 5B91F873F6: removed
Jan  7 22:02:57 postfix postfix/smtp[27180]: 375DDD5923: 
to=lexoti...@gmail.com, relay=a139.localpc2105.com[10.0.0.139]:25, 
conn_use=59, delay=61550, delays=17019/44435/96/0.17, dsn=2.0.0, 
status=sent (250 ok 1262894577 qp 12113)

Jan  7 22:02:57 postfix postfix/qmgr[26441]: 375DDD5923: removed
Jan  7 22:02:58 postfix postfix/smtp[27070]: 7F0F2943B3: 
to=gpo...@wanadoo.fr, relay=a70.localpc2105.com[10.0.0.70]:25, 
conn_use=10, delay=73795, delays=29264/44481/50/0.21, dsn=2.0.0, 
status=sent (250 ok 1262894577 qp 23067)

Jan  7 22:02:58 postfix postfix/qmgr[26441]: 7F0F2943B3: removed
Jan  7 22:02:58 postfix postfix/smtp[27050]: 32BB182182: 
to=gmarin-jardins-lois...@wanadoo.fr, 
relay=a139.localpc2105.com[10.0.0.139]:25, conn_use=48, delay=73799, 
delays=29268/44466/65/0.28, dsn=2.0.0, status=sent (250 ok 1262894578 qp 
12121)

Jan  7 22:02:58 postfix postfix/qmgr[26441]: 32BB182182: removed
Jan  7 22:02:58 postfix postfix/smtp[26758]: 577D6C7F7D: 
to=gerardtremb...@vinsdusiecle.com, 
relay=a139.localpc2105.com[10.0.0.139]:25, conn_use=60, delay=68451, 
delays=23920/44481/50/0.29, dsn=2.0.0, status=sent (250 ok 1262894578 qp 
12122)

Jan  7 22:02:58 postfix postfix/qmgr[26441]: 577D6C7F7D: removed
Jan  7 22:02:58 postfix postfix/smtp[26935]: CDCE074F53: 
to=christian.lebe...@arcelor.com, 
relay=a139.localpc2105.com[10.0.0.139]:25, conn_use=49, delay=104597, 
delays=60065/44421/110/0.3, dsn=2.0.0, status=sent (250 ok 1262894578 qp 
12135)

Jan  7 22:02:58 postfix postfix/qmgr[26441]: CDCE074F53: removed
Jan  7 22:02:58 postfix postfix/smtp[26708]: 4B0B6E77FD: 
to=m...@metaproductique.com, 
relay=a139.localpc2105.com[10.0.0.139]:25, conn_use=61, delay=46137, 
delays=1606/44461/70/0.31, dsn=2.0.0, status=sent (250 ok 1262894578 qp 
12136)

Jan  7 22:02:58 postfix postfix/qmgr[26441]: 4B0B6E77FD: removed
Jan  7 22:02:58 postfix postfix/smtp[26794]: D2CB5DC84C: 
to=secretar...@mairie-charly.fr, 
relay=a70.localpc2105.com[10.0.0.70]:25, conn_use=11, delay=58160, 
delays=13628/44481/50/0.23, dsn=2.0.0, status=sent (250 ok 1262894578 qp 
23076)

Jan  7 22:02:58 postfix postfix/qmgr[26441]: D2CB5DC84C: removed
Jan  7 22:02:58 postfix postfix/smtp[26968]: 1A651E17E0: 
to=davau.br...@orange.fr, relay=a74.localpc2105.com[10.0.0.74]:25, 
conn_use=2, delay=54426, delays=9894/44462/69/0.27, dsn=2.0.0, 
status=sent (250 ok 1262894578 qp 7411)

Jan  7 22:02:58 postfix postfix/qmgr[26441]: 1A651E17E0: removed
Jan  7 22:02:58 postfix postfix/smtp[27037]: 4CCC486B55: 
to=lenaerts.natuurst...@pandora.be, 
relay=a139.localpc2105.com[10.0.0.139]:25, conn_use=50, delay=45538, 
delays=1005/44407/125/0.17, dsn=2.0.0, status=sent (250 ok 1262894578 qp 
12150)

Jan  7 22:02:58 postfix postfix/qmgr[26441]: 4CCC486B55: removed
Jan  7 22:02:58 postfix postfix/smtp[27188]: D130997201: 
to=cont...@afcmecanum.com, relay=a74.localpc2105.com[10.0.0.74]:25, 
conn_use=2, delay=71536, delays=27004/8/84/0.28, dsn=2.0.0, 
status=sent (250 ok 1262894578 qp 7412)

Jan  7 22:02:58 postfix postfix/qmgr[26441]: D130997201: removed
Jan  7 22:02:59 postfix postfix/smtp[27033]: 6BD743906A: 
to=copyboli...@orange.fr, relay=a139.localpc2105.com[10.0.0.139]:25, 
conn_use=62, delay=81473, delays=36941/44467/65/0.24, dsn=2.0.0, 
status=sent (250 ok 1262894579 qp 12157)

Jan  7 22:02:59 postfix postfix/qmgr[26441]: 6BD743906A: removed
Jan  7 22:02:59 postfix postfix/smtp[26793]: 84947C14B2: 
to=wgall...@saemshema.com, relay=a70.localpc2105.com[10.0.0.70]:25, 
conn_use=12, delay=69401, delays=24868/44469/63/0.2, dsn=2.0.0, 
status=sent (250 ok 1262894578 qp 23084)

Jan  7 22:02:59 postfix postfix/qmgr[26441]: 84947C14B2: removed
Jan  7 22:02:59 postfix postfix/smtp[26737]: 6023552F52: 
to=cont...@installation-spa-gard.com, 
relay=a139.localpc2105.com[10.0.0.139]:25, conn_use=51, delay=96132, 
delays=51599/8/84/0.3, dsn=2.0.0, status=sent (250 ok 1262894579 qp 
12158)

Jan  7 22:02:59 postfix postfix/qmgr[26441]: 6023552F52: removed
Jan  7 22:02:59 postfix postfix/smtp[27134]: connect to 
a132.localpc2105.com[10.0.0.132]:25: Connection timed out
Jan  7 22:02:59 postfix postfix/smtp[26717]: 96A447C426: 
to=alain.perignon.aulnaysousb...@reseau.renault.fr, 
relay=a139.localpc2105.com[10.0.0.139]:25, conn_use=63, delay=103800, 
delays=59267/44433/99/0.27, dsn=2.0.0, status=sent (250 ok 1262894579 qp 
12166)

Jan  7 22:02:59 postfix postfix/qmgr[26441