Re: postfix relay - mass mailing - how to properly send messages

[Viktor Dukhovni]

> On Mar 22, 2017, at 3:27 PM, Zalezny Niezalezny 
>  wrote:
> 
> My Mailman server is connected to one of my mail gateways responsible for 
> forwarding messages to the client from the internet. As I see in the logs 
> Postfix from Mailman server sending each message with multiple recipients 
> inside. Is this correct or I need to change here something?

The most efficient, and least disruptive to other non-list traffic, way
to deliver mail to large lists is to avoid forwarding to intermediate
gateways, and have the list server connected directly to the Internet,
sending email directly to the MX hosts of each recipient's domain.

The list manager should emit a single message addressed to all
the recipients, with VERP enabled to disambiguate bounces.

I don't know whether Mailman can do this or not.

The most natural way to create large mailings with Postfix is to
use the ":include:" feature of aliases(5) to expand the subscriber
list to the underlying recipient list, set an "owner-alias" to
direct all bounces to the bounce processing engine and enable VERP
(the "-XV" sendmail(1) option) when injecting the message so that
bounce processing sees the actual failed recipient.

When all the recipients are in a single message, Postfix will
schedule delivery of other messages interspersed with delivery
of recipients of the jumbo message, and thus not starve out
other messages that arrive after delivery of the jumbo message
begins.

If you're routinely sending sending mail to large subscriber
lists, a dedicated outbound MTA just for the list traffic
may be a good idea.

Of course you'll have deliverability issues to deal with at
all the major providers, be prepared to enroll in their
whitelist programs, and work with their support staff to
resolve issues.

-- 
Viktor.

Re: postfix relay - mass mailing - how to properly send messages

[Noel Jones]

On 3/22/2017 2:27 PM, Zalezny Niezalezny wrote:
> Hi,
> 
> I have a short question. How to properly send messages from the big
> mailing lists. For example Mailman list with 50 000 or 100 000
> subscribers. How to do it in the right way with which Postfix settings ?

Postfix default settings are carefully chosen and should give
reliable performance under a wide range of loads and conditions.
Don't change any of the delivery, scheduling, or queueing defaults
unless you have a specific problem to solve.

> 
> My Mailman server is connected to one of my mail gateways
> responsible for forwarding messages to the client from the internet.
> As I see in the logs Postfix from Mailman server sending each
> message with multiple recipients inside. Is this correct or I need
> to change here something ?

Yes, this is correct.  Multiple recipients per message reduces the
load on all servers involved.  Don't change this unless you have
specific problem destinations.  See the postfix documentation.

> 
> I`m not 100% sure if each server will accept messages with some many
> recipients inside. Would it be better to send 1 message with 1
> recipient or not ?
> 
> Maybe somebody has some experience and can help me to tune my
> Postfix settings to speed up delivery process ?

People with experience have written postfix and its documentation.
If you have a specific problem, see the postfix docs.  Specifically,
see the "Problem Solving" section of
http://www.postfix.org/documentation.html



  -- Noel Jones

Re: verify_cache.db: No such file or directory - possible Berkeley DB bug ?

[Wietse Venema]

Maurizio Caloro:
> Hello
> 
> >From time to time I see on mail.log the following error message:
> 
> Mar 22 23:29:43 mail postfix/verify[2206]: close database
> /var/lib/postfix/verify_cache.db: No such file or directory (possible
> Berkeley DB bug)

Indeed, the db->close() operation returns an error code with some
Berkeley DB implementations. Postfix is written with great care,
and it pays attention to functions that report errors. In this case
the data is known to be stored safely, therefore db->close() should
not return errors. But it does, and Postfix warns.

This was once a fatal error, but it was changed into a warning
because the fatal error broke Postfix on some Linux systems.

> Please what I need to do here ?

Complain to your vendor. Postfix is only the messenger.

Wietse

verify_cache.db: No such file or directory - possible Berkeley DB bug ?

[Maurizio Caloro]

Hello

>From time to time I see on mail.log the following error message:

Mar 22 23:29:43 mail postfix/verify[2206]: close database
/var/lib/postfix/verify_cache.db: No such file or directory (possible
Berkeley DB bug)

 

I have see found different answer, but I don't know which in further
pursues.

Please what I need to do here ?

 

root@mail:~# ls -la /var/lib/postfix/verify_cache.db

-rw-r--r-- 1 postfix postfix 8192 Mar 22 23:26
/var/lib/postfix/verify_cache.db

 

root@mail:/etc/postfix# postconf -d | grep mail_version

mail_version = 2.11.3

 

Debian 3.16.39-1+deb8u2 (2017-03-07

 

Regards

Mauri

Re: Name service error for name=* type=AAAA, when it should be IPv4

[Marvin Renich]

* Bastian Blank  [170322 16:27]:
> On Wed, Mar 22, 2017 at 04:11:19PM -0400, Marvin Renich wrote:
> > At 12:30 I edited main.cf to add inet_protocols = ipv4, then restarted
> > Postfix.  I did not reboot or restart any other services, or (knowingly)
> > clear any dns cache.
> 
> - You have by any chance a nameserver != 127.0.0.1 or ::1 in
>   /etc/resolv.conf and it can change during the runtime of the system?
> - You run Postfix daemons chrooted? (check the chroot column in
>   /etc/postfix/master.cf)

Yes to both!  I think that explains it.  resolv.con was copied to the
chroot during a reboot after a power (and therefore network) outage,
when it had a bad value.  The chrooted copy was never updated after the
network came back up.  (The laptop boots much faster than the machine
with the DHCP server!)

Thank you very much!  Now I know how to handle this the next time it
happens (and, in fact, will be expecting it to happen in those
circumstances).

...Marvin

Re: Name service error for name=* type=AAAA, when it should be IPv4

[Marvin Renich]

* Bastian Blank  [170322 15:09]:
> Moin
> 
> On Wed, Mar 22, 2017 at 01:04:36PM -0400, Marvin Renich wrote:
> > Mar 21 14:42:45 basil postfix/smtp[12587]: 3AF35240229: 
> > to=, relay=none, delay=0.47, 
> > delays=0.25/0.22/0/0, dsn=4.4.3, status=deferred (Host or domain name not 
> > found. Name service error for name=florina.renich.org type=: Host not 
> > found, try again)
> 
> I looked that error up and I'm not sure if I found everything.
> 
> - "Host not found, try again" is TRY_AGAIN, according to
>   src/dns/dns_strerror.c.
> - This error is only produced if res_send produced an error, or if the
>   lookup explicitely gets a SERVFAIL.
> 
> So this message should only show up if something is already pretty wrong
> with DNS.  If there is no network it will show up for example.

Thanks for looking at this, Bastian.  Is there any reason you can think
of that res_send would fail for Postfix, but other programs, such as
host, would get a correct response?  For example, if Postfix calls
res_init, which gets a bad value from resolv.conf, like for a temporary
network outage, then Postfix continues to run for days, not requiring
any dns services.  Meanwhile, the temporary outage is short-lived, but
Postfix never gets the word.  Other programs work, but Postfix continues
to get dns failures because res_init was called at a time when
resolv.conf was bad?

...Marvin

Re: Name service error for name=* type=AAAA, when it should be IPv4

[Bastian Blank]

On Wed, Mar 22, 2017 at 04:11:19PM -0400, Marvin Renich wrote:
> At 12:30 I edited main.cf to add inet_protocols = ipv4, then restarted
> Postfix.  I did not reboot or restart any other services, or (knowingly)
> clear any dns cache.

- You have by any chance a nameserver != 127.0.0.1 or ::1 in
  /etc/resolv.conf and it can change during the runtime of the system?
- You run Postfix daemons chrooted? (check the chroot column in
  /etc/postfix/master.cf)

Bastian

-- 
We have found all life forms in the galaxy are capable of superior
development.
-- Kirk, "The Gamesters of Triskelion", stardate 3211.7

Re: Name service error for name=* type=AAAA, when it should be IPv4

[Marvin Renich]

* Noel Jones  [170322 14:38]:
> On 3/22/2017 1:08 PM, Marvin Renich wrote:
> > Thanks, Wietse and Noel.  Once the IPv4 delivery fails for some reason,
> > and Postfix tries IPv6 (which must fail for this relayhost), the message
> > is deferred.  Do subsequent redelivery attempts only try IPv6?  This is
> > what it looked like was happening, even if the failure was caused by
> > something else.
> 
> Each subsequent attempt will try both, but only the last error is
> reported in the queue listing.

That is what I expected.

> > The message sat in the deferred queue overnight, with more than a dozen
> > redelivery attempts, all with the same result, even though during that
> > time I was able to correctly resolve the relayhost's IP.  postqueue -f
> > failed, but changing inet_protocols to ipv4 and restarting Postfix
> > delivered the message immediately.
> 
> If DNS lookups were working, the temporary delivery error was
> probably something other than a DNS lookup failure.  You should be
> able to find these other delivery attempts logged by the smtp
> delivery process.

The mail.log file contains no other intervening messages.  zgrep smtp
/var/log/* reveals nothing else of any interest.  Except for the [snip]
in the middle, the following is the complete mail.log from message
pickup through delivery of my initial message in this thread.

Mar 21 14:42:45 basil postfix/pickup[11818]: 3AF35240229: uid=1001 
from=
Mar 21 14:42:45 basil postfix/cleanup[12584]: 3AF35240229: 
message-id=<20170321184244.cxyiq2u6w53qt...@basil.wdw>
Mar 21 14:42:45 basil postfix/qmgr[3804]: 3AF35240229: from=, 
size=8480, nrcpt=2 (queue active)
Mar 21 14:42:45 basil postfix/smtp[12587]: 3AF35240229: 
to=, relay=none, delay=0.47, 
delays=0.25/0.22/0/0, dsn=4.4.3, status=deferred (Host or domain name not 
found. Name service error for name=florina.renich.org type=: Host not 
found, try again)
Mar 21 14:42:45 basil postfix/smtp[12587]: 3AF35240229: to=, 
relay=none, delay=0.47, delays=0.25/0.22/0/0, dsn=4.4.3, status=deferred (Host 
or domain name not found. Name service error for name=florina.renich.org 
type=: Host not found, try again)
Mar 21 14:52:26 basil postfix/qmgr[3804]: 3AF35240229: from=, 
size=8480, nrcpt=2 (queue active)
Mar 21 14:52:26 basil postfix/smtp[12810]: 3AF35240229: 
to=, relay=none, delay=581, delays=581/0.01/0/0, 
dsn=4.4.3, status=deferred (Host or domain name not found. Name service error 
for name=florina.renich.org type=: Host not found, try again)
Mar 21 14:52:26 basil postfix/smtp[12810]: 3AF35240229: to=, 
relay=none, delay=581, delays=581/0.01/0/0, dsn=4.4.3, status=deferred (Host or 
domain name not found. Name service error for name=florina.renich.org 
type=: Host not found, try again)
Mar 21 15:02:26 basil postfix/qmgr[3804]: 3AF35240229: from=, 
size=8480, nrcpt=2 (queue active)
Mar 21 15:02:26 basil postfix/smtp[13022]: 3AF35240229: 
to=, relay=none, delay=1181, 
delays=1181/0.01/0/0, dsn=4.4.3, status=deferred (Host or domain name not 
found. Name service error for name=florina.renich.org type=: Host not 
found, try again)
Mar 21 15:02:26 basil postfix/smtp[13022]: 3AF35240229: to=, 
relay=none, delay=1181, delays=1181/0.01/0/0, dsn=4.4.3, status=deferred (Host 
or domain name not found. Name service error for name=florina.renich.org 
type=: Host not found, try again)
Mar 21 15:22:26 basil postfix/qmgr[3804]: 3AF35240229: from=, 
size=8480, nrcpt=2 (queue active)
Mar 21 15:22:27 basil postfix/smtp[13538]: 3AF35240229: 
to=, relay=none, delay=2381, 
delays=2381/0.01/0/0, dsn=4.4.3, status=deferred (Host or domain name not 
found. Name service error for name=florina.renich.org type=: Host not 
found, try again)
Mar 21 15:22:27 basil postfix/smtp[13538]: 3AF35240229: to=, 
relay=none, delay=2381, delays=2381/0.01/0/0, dsn=4.4.3, status=deferred (Host 
or domain name not found. Name service error for name=florina.renich.org 
type=: Host not found, try again)
Mar 21 16:02:26 basil postfix/qmgr[3804]: 3AF35240229: from=, 
size=8480, nrcpt=2 (queue active)
Mar 21 16:02:26 basil postfix/smtp[14418]: 3AF35240229: 
to=, relay=none, delay=4781, 
delays=4781/0.01/0/0, dsn=4.4.3, status=deferred (Host or domain name not 
found. Name service error for name=florina.renich.org type=: Host not 
found, try again)
Mar 21 16:02:26 basil postfix/smtp[14418]: 3AF35240229: to=, 
relay=none, delay=4781, delays=4781/0.01/0/0, dsn=4.4.3, status=deferred (Host 
or domain name not found. Name service error for name=florina.renich.org 
type=: Host not found, try again)
Mar 21 16:11:19 basil postfix/qmgr[3804]: 3AF35240229: 

postfix relay - mass mailing - how to properly send messages

[Zalezny Niezalezny]

Hi,

I have a short question. How to properly send messages from the big mailing
lists. For example Mailman list with 50 000 or 100 000 subscribers. How to
do it in the right way with which Postfix settings ?

My Mailman server is connected to one of my mail gateways responsible for
forwarding messages to the client from the internet. As I see in the logs
Postfix from Mailman server sending each message with multiple recipients
inside. Is this correct or I need to change here something ?

I`m not 100% sure if each server will accept messages with some many
recipients inside. Would it be better to send 1 message with 1 recipient or
not ?

Maybe somebody has some experience and can help me to tune my Postfix
settings to speed up delivery process ?


Cheers

Zalezny

Re: Name service error for name=* type=AAAA, when it should be IPv4

[Bastian Blank]

Moin

On Wed, Mar 22, 2017 at 01:04:36PM -0400, Marvin Renich wrote:
> Mar 21 14:42:45 basil postfix/smtp[12587]: 3AF35240229: 
> to=, relay=none, delay=0.47, 
> delays=0.25/0.22/0/0, dsn=4.4.3, status=deferred (Host or domain name not 
> found. Name service error for name=florina.renich.org type=: Host not 
> found, try again)

I looked that error up and I'm not sure if I found everything.

- "Host not found, try again" is TRY_AGAIN, according to
  src/dns/dns_strerror.c.
- This error is only produced if res_send produced an error, or if the
  lookup explicitely gets a SERVFAIL.

So this message should only show up if something is already pretty wrong
with DNS.  If there is no network it will show up for example.

Bastian

-- 
Without followers, evil cannot spread.
-- Spock, "And The Children Shall Lead", stardate 5029.5

Re: Name service error for name=* type=AAAA, when it should be IPv4

[Noel Jones]

On 3/22/2017 1:08 PM, Marvin Renich wrote:
> Thanks, Wietse and Noel.  Once the IPv4 delivery fails for some reason,
> and Postfix tries IPv6 (which must fail for this relayhost), the message
> is deferred.  Do subsequent redelivery attempts only try IPv6?  This is
> what it looked like was happening, even if the failure was caused by
> something else.

Each subsequent attempt will try both, but only the last error is
reported in the queue listing.

> 
> The message sat in the deferred queue overnight, with more than a dozen
> redelivery attempts, all with the same result, even though during that
> time I was able to correctly resolve the relayhost's IP.  postqueue -f
> failed, but changing inet_protocols to ipv4 and restarting Postfix
> delivered the message immediately.

If DNS lookups were working, the temporary delivery error was
probably something other than a DNS lookup failure.  You should be
able to find these other delivery attempts logged by the smtp
delivery process.

> 
> Also, what about my third question:  was the serverfault answer only
> useful if an smtp-ipv4 service is manually added to master.cf?

That answer is useful for a host that delivers some mail via IPv6
but has some destinations that don't work properly on IPv6 -- for
example, a destination that 5xx rejects IPv6 mail, but accepts IPv4
mail. Such workarounds are not required for well-behaved
destinations.  And yes, a master.cf smtp-ipv4 service would be
needed, so it was not a complete answer.



  -- Noel Jones

Re: Name service error for name=* type=AAAA, when it should be IPv4

[Marvin Renich]

* Wietse Venema  [170322 13:14]:
> Marvin Renich:
> > First, why would the DNS query correctly give the ipv4 address on Mar
> > 10, but then fail saying it could not find an ipv6 entry on Mar 21?  I
> > can find no configuration change or program version change that would
> > explain this.
> 
> Previously, the A lookup succeeded. Now, the the A lookup fails,
> then Postfix tries , and that fails, too. Postfix only logs the
> last DNS error.
> 
> So the real question is why did the A lookup fail. You can avoid
> the IPv6 lookup by configuring inet_protocols in main.cf.
> 
>   Wietse

Thanks, Wietse and Noel.  Once the IPv4 delivery fails for some reason,
and Postfix tries IPv6 (which must fail for this relayhost), the message
is deferred.  Do subsequent redelivery attempts only try IPv6?  This is
what it looked like was happening, even if the failure was caused by
something else.

The message sat in the deferred queue overnight, with more than a dozen
redelivery attempts, all with the same result, even though during that
time I was able to correctly resolve the relayhost's IP.  postqueue -f
failed, but changing inet_protocols to ipv4 and restarting Postfix
delivered the message immediately.

Also, what about my third question:  was the serverfault answer only
useful if an smtp-ipv4 service is manually added to master.cf?

Thanks...Marvin

Re: Name service error for name=* type=AAAA, when it should be IPv4

[Noel Jones]

On 3/22/2017 12:13 PM, Wietse Venema wrote:
> Marvin Renich:
>> First, why would the DNS query correctly give the ipv4 address on Mar
>> 10, but then fail saying it could not find an ipv6 entry on Mar 21?  I
>> can find no configuration change or program version change that would
>> explain this.
> 
> Previously, the A lookup succeeded. Now, the the A lookup fails,
> then Postfix tries , and that fails, too. Postfix only logs the
> last DNS error.
> 
> So the real question is why did the A lookup fail. You can avoid
> the IPv6 lookup by configuring inet_protocols in main.cf.
> 
>   Wietse
> 


Postfix only reports the last delivery error, so /any/ temporary
error delivering to the IPv4 address will cause this.  Search the
maillog for prior entries from the smtp delivery process; note they
will not contain the QUEUEID.

If postfix should never use IPv6, disable it with "inet_protocols =
ipv4".  That won't fix the delivery problem, but it will make the
logging clearer.



  -- Noel Jones

Re: bitdefender

[chaouche yacine]

Hello David,
I have no experience with any particular antivirus, but looking at 
/etc/amavisd-new/conf.d/15-av_scanners.conf I can see that bitdefender is 
supported.
Since this is an amavis question you should get better luck asking in the 
amavis list instead.

  -- Yassine.

Re: Problems with lmtp

[chaouche yacine]

All clear ! thanks.
 

On Friday, March 17, 2017 5:19 PM, Thomas Leuxner  wrote:
 

 * chaouche yacine  2017.03.17 14:52:

> Thank you Thomas, so if I understand correctly in Viktor's config dovecot is 
> only used by postfix as a backend to query for valid virtual email addresses ?

Hi Yassine,

one of the benefits of using Dovecot's MDAs besides Sieve, is that they update 
the metadata and indexes upon delivery which improves its IMAP performance. You 
also get a broader choice of mailstorage formats which offer new features 
compared to the older maildir format.

Regards
Thomas


   

Re: Name service error for name=* type=AAAA, when it should be IPv4

[Wietse Venema]

Marvin Renich:
> First, why would the DNS query correctly give the ipv4 address on Mar
> 10, but then fail saying it could not find an ipv6 entry on Mar 21?  I
> can find no configuration change or program version change that would
> explain this.

Previously, the A lookup succeeded. Now, the the A lookup fails,
then Postfix tries , and that fails, too. Postfix only logs the
last DNS error.

So the real question is why did the A lookup fail. You can avoid
the IPv6 lookup by configuring inet_protocols in main.cf.

Wietse

Name service error for name=* type=AAAA, when it should be IPv4

[Marvin Renich]

I'm having a new problem with ipv6.  I'm running Debian (mostly testing
release) on my laptop, with Postfix running simply to allow mutt to send
when I don't have connectivity.  The relevant postconf entries (full
postconf -n below):

inet_protocols = all   (default; not specified in main.cf)
relayhost = [florina.renich.org]:587

On March 10, I have this log entry:

Mar 10 08:59:49 basil postfix/smtp[10878]: C0CFC2401AA: 
to=, 
relay=florina.renich.org[64.150.161.163]:587, delay=1.5, 
delays=0.13/0.04/1.2/0.11, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 
84B16680B0)

That was the last successful delivery through relayhost.  The next
outgoing message was on March 21:

Mar 21 14:42:45 basil postfix/smtp[12587]: 3AF35240229: 
to=, relay=none, delay=0.47, 
delays=0.25/0.22/0/0, dsn=4.4.3, status=deferred (Host or domain name not 
found. Name service error for name=florina.renich.org type=: Host not 
found, try again)

Postfix was last updated on February 5, from Debian postfix 3.1.4-3 to
3.1.4-4.  etckeeper shows that the last changes to Postfix configuration
came from that upgrade on that date (dynamicmaps.cf, makedefs.out, and
/etc/aliases.db).  mail.log shows Postfix was restarted on March 3, 11,
12, and 18.

The DNS information for the relayhost has not changed in a long time,
and it has never had an ipv6 entry (I control the tinydns server that is
authoritative for that domain).  From my laptop, various tools (host,
dig, nslookup, dnsip) have no trouble resolving the relayhost's address,
returning the correct ipv4 address.  I even did a raw dump of the result
from getaddrinfo("florina.renich.org", "587", ...) and got the expected
result.

First, why would the DNS query correctly give the ipv4 address on Mar
10, but then fail saying it could not find an ipv6 entry on Mar 21?  I
can find no configuration change or program version change that would
explain this.

Second, I would expect that with inet_protocols = all that if an ipv6
lookup fails, it would try an ipv4 lookup.  Is that not the case?  Do I
need to explicitly specify which hosts should use ipv4?

Third, I found a serverfault.com post[1] from 2014 where the answer said
to use a transport map with an entry like

example.com smtp-ipv4:[mail.domain.com]

but it did not say anything about creating a service smtp-ipv4 in
master.cf; was this answer just missing that info?  Does the default
Postfix config have that entry, but the Debian package doesn't?  That
same answer said that with inet_protocols=all Postfix should try ipv6
first, then fall back to ipv4.

Since on my laptop, I am always sending through the relayhost, is the
best solution to just set inet_protocols = ipv4?  Is there a better
solution?

This is what I have done for now, and it works, but I have other Postfix
servers, including the one on the relayhost (which has not yet shown any
problem), and as it delivers globally, I would like to understand why my
laptop's Postfix is having trouble.

Thanks...Marvin

[1] http://serverfault.com/questions/577134/postfix-host-or-domain-not-found

# postconf -n
alias_database = hash:/etc/aliases
alias_maps = hash:/etc/aliases
append_dot_mydomain = no
biff = no
home_mailbox = Maildir/
html_directory = /usr/share/doc/postfix/html
inet_interfaces = loopback-only
mailbox_command = procmail -a "$EXTENSION"
mailbox_size_limit = 0
mydestination = basil.localdomain, localhost.localdomain, localhost
myhostname = basil.localdomain
mynetworks = 127.0.0.0/8 [:::127.0.0.0]/104 [::1]/128
myorigin = /etc/mailname
readme_directory = /usr/share/doc/postfix
recipient_delimiter = -
relayhost = [florina.renich.org]:587
smtp_generic_maps = hash:/etc/postfix/generic
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/sasl_password
smtp_sasl_security_options = noanonymous, noplaintext
smtp_sasl_tls_security_options = noanonymous
smtp_tls_security_level = may
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
smtpd_banner = $myhostname ESMTP $mail_name (Debian/GNU)
smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated 
defer_unauth_destination
smtpd_tls_cert_file = /etc/ssl/certs/ssl-cert-snakeoil.pem
smtpd_tls_key_file = /etc/ssl/private/ssl-cert-snakeoil.key
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtpd_use_tls = yes