Re: performance tuning - relay

[Stan Hoeppner]

Christian Purnomo put forth on 6/27/2010 5:50 PM:

 From your questions above, I could see where you're coming from that if
 Server2 has performance problem then it would make sense to see the
 queue built up at Server1.  I can confirm server2 is very underload at
 any time, the server is overspec'ed for what it is intended to do.  I
 can also confirm while those thousands of emails queued up at Server1,
 Server2 was running smooth with  0.1-0.3 load average.  

What piqued my curiosity is why the queue on server2 starting growing, and
rather large at that, _after_ you got the Postfix bottleneck straightened out
on server1.

 We have had server2 for about 4 years now and we have been having this
 issues in the last 1 year where one of our new server happens to be a
 mailling list which sends out thousands of emails to subscribers.
 
 Anyway, Server2 spec is HP DL385G4, 4G RAM, 6 SCSI disks RAID 5 and
 reiserfs.  

I would have thought this hardware would be able to get the mails into the
mailboxen as quickly as server1 could push them over, without the queue
building up as you demonstrated in a previous message.  Email service is
primarily a disk bound application.  IIRC, with the DL385G4 you would have the
Smart Array 6i which is an integrated entry level controller.  Even so, with
128MB of read/write cache and 6x10k(15?)rpm drives on a SCSI 320 bus, even in
a slowish RAID5 configuration, you should easily be able to sync to mailboxen
as many messages as server1 could push over either fast or gigabit ethernet.
This server should be able to sync a few hundred emails to disk per second.
Is the 6i just really horrible at RAID5, or is there something in the software
stack slowing things down?  Were you peaking the disk subsystem when the queue
was building?

 The delivery method on Server2 is maildrop - we use some mailfilter rule
 to drop certain emails to certain folders.  I can understand this is
 adding some overhead for the local delivery on Server2 but this is the
 cost I'm happy to take on.  The queue can build up on Server2 and clear
 up overtime without impacting our primary MX (Server1).

I'm not familiar at all with maildrop as I've never used it.  That said, I
wouldn't think maildrop alone would cause such a bottleneck.  Some versions of
Reiser are known for great speed will lots of small files, at least as far as
delete performance.  However, most versions of Reiser do not do so well with
large files.  Reiser is normally a good performer with maildir, but doesn't do
so well with mbox, especially once the mbox files get large.

Other disk writes?  Is maildrop or any other process you're running creating
extra log stamps per email processed?  I assume you're storing the OS, logs,
mail, everything on that RAID5 volume.  Is this correct?

As you stated, you're not really concerned with queue growth on server2.  I
went through all this simply because I think you're leaving some performance,
maybe quite a bit, on the table WRT server2.  I'm guessing it's in the
OS/software stack and not the hardware.  You may be able to get this box
screaming with simple changes (reduce logging to only what's necessary), and
maybe one or two more major changes (maildir to mbox or vice versa, switching
from Reiser--defunct now anyway--to XFS).  Or a really big change, dumping
Maildrop/Courier for Dovecot/LDA which is quite a bit quicker from everything
I've read.  I say read because I've not used Courier but I have used Dovecot,
and still do.

Sorry if I've wasted your time here.  I just thought I'd point out a few
things just in case you get the urge to poke around on server2 looking for a
little performance boost.

-- 
Stan

Re: performance tuning - relay

[Christian Purnomo]

Hi Stan

Subject: Re: performance tuning - relay
Date: Mon, Jun 28, 2010 at 01:23:15AM -0500
Quoting Stan Hoeppner (s...@hardwarefreak.com):

: What piqued my curiosity is why the queue on server2 starting growing, and
: rather large at that, _after_ you got the Postfix bottleneck straightened out
: on server1.

I was expecting this and don't have a problem with this limitation.  The
maildrop rule is rather long and I knew I would get penalized.  However
delays on local delivery on Server2 has no impact to production so it's
ok.

: I would have thought this hardware would be able to get the mails into the
: mailboxen as quickly as server1 could push them over, without the queue
: building up as you demonstrated in a previous message.  Email service is
: primarily a disk bound application.  IIRC, with the DL385G4 you would have the
: Smart Array 6i which is an integrated entry level controller.  Even so, with
: 128MB of read/write cache and 6x10k(15?)rpm drives on a SCSI 320 bus, even in
: a slowish RAID5 configuration, you should easily be able to sync to mailboxen
: as many messages as server1 could push over either fast or gigabit ethernet.
: This server should be able to sync a few hundred emails to disk per second.
: Is the 6i just really horrible at RAID5, or is there something in the software
: stack slowing things down?  Were you peaking the disk subsystem when the queue
: was building?
: 
: I'm not familiar at all with maildrop as I've never used it.  That said, I
: wouldn't think maildrop alone would cause such a bottleneck.  Some versions of
: Reiser are known for great speed will lots of small files, at least as far as
: delete performance.  However, most versions of Reiser do not do so well with
: large files.  Reiser is normally a good performer with maildir, but doesn't do
: so well with mbox, especially once the mbox files get large.


Maildrop is just procmail for Maildir.  I had to use Maildir
format as there are hundreds of thousands of email to the always_bcc
email on Server2.  

: Other disk writes?  Is maildrop or any other process you're running creating
: extra log stamps per email processed?  I assume you're storing the OS, logs,
: mail, everything on that RAID5 volume.  Is this correct?
: 
: As you stated, you're not really concerned with queue growth on server2.  I
: went through all this simply because I think you're leaving some performance,
: maybe quite a bit, on the table WRT server2.  I'm guessing it's in the
: OS/software stack and not the hardware.  You may be able to get this box
: screaming with simple changes (reduce logging to only what's necessary), and
: maybe one or two more major changes (maildir to mbox or vice versa, switching
: from Reiser--defunct now anyway--to XFS).  Or a really big change, dumping
: Maildrop/Courier for Dovecot/LDA which is quite a bit quicker from everything
: I've read.  I say read because I've not used Courier but I have used Dovecot,
: and still do.

Server2 wasn't my concern, Server1 was :)

The issue as far as I could see Server1 was unable to feed enough email
to Server2, I knew there was a limit somewhere on Server1 that prevented
this.

: Sorry if I've wasted your time here.  I just thought I'd point out a few
: things just in case you get the urge to poke around on server2 looking for a
: little performance boost.


There is no such thing as wasting time here, I am grateful for anyone to
reply to my question.  Thanks *_^

Re: performance tuning - relay

[Daniel V. Reinhardt]

- Original Message 
 From: Stan Hoeppner s...@hardwarefreak.com
 To: postfix-users@postfix.org
 Sent: Mon, June 28, 2010 2:23:15 AM
 Subject: Re: performance tuning - relay
 
 Christian Purnomo put forth on 6/27/2010 5:50 PM:

 From your 
 questions above, I could see where you're coming from that if
 Server2 
 has performance problem then it would make sense to see the
 queue built 
 up at Server1.  I can confirm server2 is very underload at
 any 
 time, the server is overspec'ed for what it is intended to do.  I
 
 can also confirm while those thousands of emails queued up at Server1,
 
 Server2 was running smooth with  0.1-0.3 load average.  

What 
 piqued my curiosity is why the queue on server2 starting growing, and
rather 
 large at that, _after_ you got the Postfix bottleneck straightened out
on 
 server1.

 We have had server2 for about 4 years now and we have been 
 having this
 issues in the last 1 year where one of our new server 
 happens to be a
 mailling list which sends out thousands of emails to 
 subscribers.
 
 Anyway, Server2 spec is HP DL385G4, 4G RAM, 6 SCSI 
 disks RAID 5 and
 reiserfs.  

I would have thought this 
 hardware would be able to get the mails into the
mailboxen as quickly as 
 server1 could push them over, without the queue
building up as you 
 demonstrated in a previous message.  Email service is
primarily a disk 
 bound application.  IIRC, with the DL385G4 you would have the
Smart 
 Array 6i which is an integrated entry level controller.  Even so, 
 with
128MB of read/write cache and 6x10k(15?)rpm drives on a SCSI 320 bus, 
 even in
a slowish RAID5 configuration, you should easily be able to sync to 
 mailboxen
as many messages as server1 could push over either fast or gigabit 
 ethernet.
This server should be able to sync a few hundred emails to disk per 
 second.
Is the 6i just really horrible at RAID5, or is there something in the 
 software
stack slowing things down?  Were you peaking the disk subsystem 
 when the queue
was building?

 The delivery method on Server2 is 
 maildrop - we use some mailfilter rule
 to drop certain emails to certain 
 folders.  I can understand this is
 adding some overhead for the 
 local delivery on Server2 but this is the
 cost I'm happy to take 
 on.  The queue can build up on Server2 and clear
 up overtime 
 without impacting our primary MX (Server1).

I'm not familiar at all with 
 maildrop as I've never used it.  That said, I
wouldn't think maildrop 
 alone would cause such a bottleneck.  Some versions of
Reiser are known 
 for great speed will lots of small files, at least as far as
delete 
 performance.  However, most versions of Reiser do not do so well 
 with
large files.  Reiser is normally a good performer with maildir, but 
 doesn't do
so well with mbox, especially once the mbox files get 
 large.

Other disk writes?  Is maildrop or any other process you're 
 running creating
extra log stamps per email processed?  I assume you're 
 storing the OS, logs,
mail, everything on that RAID5 volume.  Is this 
 correct?

As you stated, you're not really concerned with queue growth on 
 server2.  I
went through all this simply because I think you're leaving 
 some performance,
maybe quite a bit, on the table WRT server2.  I'm 
 guessing it's in the
OS/software stack and not the hardware.  You may be 
 able to get this box
screaming with simple changes (reduce logging to only 
 what's necessary), and
maybe one or two more major changes (maildir to mbox 
 or vice versa, switching
from Reiser--defunct now anyway--to XFS).  Or a 
 really big change, dumping
Maildrop/Courier for Dovecot/LDA which is quite a 
 bit quicker from everything
I've read.  I say read because I've not used 
 Courier but I have used Dovecot,
and still do.

Sorry if I've wasted 
 your time here.  I just thought I'd point out a few
things just in case 
 you get the urge to poke around on server2 looking for a
little performance 
 boost.

-- 
Stan

-

Stan,

Actually you do not need to pay for their mail forwarding services.  I have a 
sever setup to accept email just fine and dandy for a dyndns.org support host, 
and I do not pay anything for it.  I get mail to my system woa.homeip.net just 
fine without paying.  

The paid for services you speak of are for people who want to customize their 
own dyndns settings.

You can send me an email to crypto...@woa.homeip.net and I will receive it, and 
I can send out.  I would suggest you get a dyndns.org account, and do some 
research on it.

I have been using dyndns.org since about 2001 when I first my DSL Connection.


 Daniel Reinhardt
Website: www.cryptodan.com
Email: crypto...@yahoo.com


  

Re: performance tuning - relay

[Daniel V. Reinhardt]

- Original Message 
 From: Daniel V. Reinhardt crypto...@yahoo.com
 To: postfix-users@postfix.org
 Sent: Mon, June 28, 2010 3:32:04 AM
 Subject: Re: performance tuning - relay
 
 

- Original Message 
 From: Stan Hoeppner 
 ymailto=mailto:s...@hardwarefreak.com; 
 href=mailto:s...@hardwarefreak.com;s...@hardwarefreak.com
 To: 
 
 href=mailto:postfix-users@postfix.org;postfix-users@postfix.org
 
 Sent: Mon, June 28, 2010 2:23:15 AM
 Subject: Re: performance tuning - 
 relay
 
 Christian Purnomo put forth on 6/27/2010 5:50 
 PM:

 From your 
 questions above, I could see where you're 
 coming from that if
 Server2 
 has performance problem then it 
 would make sense to see the
 queue built 
 up at Server1.  I 
 can confirm server2 is very underload at
 any 
 time, the server 
 is overspec'ed for what it is intended to do.  I
 
 can also 
 confirm while those thousands of emails queued up at Server1,
 
 
 Server2 was running smooth with  0.1-0.3 load average.  

What 
 
 piqued my curiosity is why the queue on server2 starting growing, 
 and
rather 
 large at that, _after_ you got the Postfix bottleneck 
 straightened out
on 
 server1.

 We have had server2 for 
 about 4 years now and we have been 
 having this
 issues in the 
 last 1 year where one of our new server 
 happens to be a
 
 mailling list which sends out thousands of emails to 
 
 subscribers.
 
 Anyway, Server2 spec is HP DL385G4, 4G RAM, 6 SCSI 
 
 disks RAID 5 and
 reiserfs.  

I would have thought 
 this 
 hardware would be able to get the mails into the
mailboxen as 
 quickly as 
 server1 could push them over, without the queue
building 
 up as you 
 demonstrated in a previous message.  Email service 
 is
primarily a disk 
 bound application.  IIRC, with the DL385G4 
 you would have the
Smart 
 Array 6i which is an integrated entry level 
 controller.  Even so, 
 with
128MB of read/write cache and 
 6x10k(15?)rpm drives on a SCSI 320 bus, 
 even in
a slowish RAID5 
 configuration, you should easily be able to sync to 
 mailboxen
as 
 many messages as server1 could push over either fast or gigabit 
 
 ethernet.
This server should be able to sync a few hundred emails to disk per 
 
 second.
Is the 6i just really horrible at RAID5, or is there 
 something in the 
 software
stack slowing things down?  Were you 
 peaking the disk subsystem 
 when the queue
was building?

 
 The delivery method on Server2 is 
 maildrop - we use some mailfilter 
 rule
 to drop certain emails to certain 
 folders.  I can 
 understand this is
 adding some overhead for the 
 local delivery 
 on Server2 but this is the
 cost I'm happy to take 
 on.  The 
 queue can build up on Server2 and clear
 up overtime 
 without 
 impacting our primary MX (Server1).

I'm not familiar at all with 
 
 maildrop as I've never used it.  That said, I
wouldn't think maildrop 
 
 alone would cause such a bottleneck.  Some versions of
Reiser 
 are known 
 for great speed will lots of small files, at least as far 
 as
delete 
 performance.  However, most versions of Reiser do not 
 do so well 
 with
large files.  Reiser is normally a good 
 performer with maildir, but 
 doesn't do
so well with mbox, especially 
 once the mbox files get 
 large.

Other disk writes?  Is 
 maildrop or any other process you're 
 running creating
extra log 
 stamps per email processed?  I assume you're 
 storing the OS, 
 logs,
mail, everything on that RAID5 volume.  Is this 
 
 correct?

As you stated, you're not really concerned with queue growth on 
 
 server2.  I
went through all this simply because I think you're 
 leaving 
 some performance,
maybe quite a bit, on the table WRT 
 server2.  I'm 
 guessing it's in the
OS/software stack and not 
 the hardware.  You may be 
 able to get this box
screaming with 
 simple changes (reduce logging to only 
 what's necessary), and
maybe 
 one or two more major changes (maildir to mbox 
 or vice versa, 
 switching
from Reiser--defunct now anyway--to XFS).  Or a 
 
 really big change, dumping
Maildrop/Courier for Dovecot/LDA which is quite a 
 
 bit quicker from everything
I've read.  I say read because I've 
 not used 
 Courier but I have used Dovecot,
and still do.

Sorry 
 if I've wasted 
 your time here.  I just thought I'd point out a 
 few
things just in case 
 you get the urge to poke around on server2 
 looking for a
little performance 
 boost.

-- 
 
Stan

-

Stan,

Actually you do not need 
 to pay for their mail forwarding services.  I have a sever setup to accept 
 email just fine and dandy for a dyndns.org support host, and I do not pay 
 anything for it.  I get mail to my system woa.homeip.net just fine without 
 paying.  

The paid for services you speak of are for people who want 
 to customize their own dyndns settings.

You can send me an email to 
 ymailto=mailto:crypto...@woa.homeip.net; 
 href=mailto:crypto...@woa.homeip.net;crypto...@woa.homeip.net and I will 
 receive it, and I can send out.  I would suggest

Re: performance tuning - relay

[Christian Purnomo]

Subject: Re: performance tuning - relay
Date: Fri, Jun 25, 2010 at 06:21:01PM -0500
Quoting Stan Hoeppner (s...@hardwarefreak.com):

: Can you provide some more specs on server2?  IIRC you said you had a multidisk
: RAID array on serv2.  What RAID level and how many disks?  What filesystem?
: Are you running Courier with maildrop or the standalone maildrop with another
: IMAP server?  What filtering, if any, are you doing with maildrop?  Using mbox
: or maildir storage?  IIRC you previously said you're BCC'ing _everything_ into
: a single mailbox (single address) on server2.  Is this correct?
: 
: And, lastly, was server2 in production for any amount of time before these
: problems occurred, prompting your post, or is this a new server that you just
: brought online?
: 
: -- 
: Stan
: 
: 

Hi Stan,

All valid questions...

The real issue AFAIK was Server1 not feeding enough emails to Server2
and hence thousands of emails were queued on Server1 waiting to be
delivered to Server 2 via transport-relay.  This was the major issue I
was trying to solve.

From your questions above, I could see where you're coming from that if
Server2 has performance problem then it would make sense to see the
queue built up at Server1.  I can confirm server2 is very underload at
any time, the server is overspec'ed for what it is intended to do.  I
can also confirm while those thousands of emails queued up at Server1,
Server2 was running smooth with  0.1-0.3 load average.  

We have had server2 for about 4 years now and we have been having this
issues in the last 1 year where one of our new server happens to be a
mailling list which sends out thousands of emails to subscribers.

Anyway, Server2 spec is HP DL385G4, 4G RAM, 6 SCSI disks RAID 5 and
reiserfs.  

The delivery method on Server2 is maildrop - we use some mailfilter rule
to drop certain emails to certain folders.  I can understand this is
adding some overhead for the local delivery on Server2 but this is the
cost I'm happy to take on.  The queue can build up on Server2 and clear
up overtime without impacting our primary MX (Server1).

CP

Re: performance tuning - relay

[Stan Hoeppner]

Christian Purnomo put forth on 6/24/2010 11:33 PM:

 /etc/postfix/transport:
 server2.com:  relay:[10.0.2.73]
 
 /etc/postfix/master.cf:
 relay unix  -   -   n   -   200  smtp
 -o smtp_helo_timeout=3s
 -o smtp_connect_timeout=3s
 -o disable_dns_lookups=yes
 -o fallback_relay=

This was answered by Wietse 4 years ago on this list.  Took me ten seconds to
find it via Google.  Read the entire thread on Neohapsis carefully and you'll
find your answer, which is to remove all this custom stuff and go back to the
defaults.  The first 2 of 4 above are the cause of your immediate problem, as
they are wy too low.  The other two are just unnecessary.  And change
max_proc back to 100.  You're probably not getting close to 100 processes
running anyway.

http://archives.neohapsis.com/archives/postfix/2006-01/thread.html#1866

 Server 2 has the following configurations:
 
 /etc/postfix/master.cf:
 smtp  inet  n   -   -   -   200 smtpd

Change the max process limit back to 100.  If everything else is configured
correctly, you can drain an unbelievable amount of mail with less than 100
smtp/smtpd processes.

 Could you please tell me what I'm missing here? I would like to improve
 the rate that Server1 can relay messages to Server2.

If I may be frank, you missed the fact that you shouldn't mess with the
default settings unless you really know what you're doing.  Custom settings
here would require an extreme scenario.  I don't believe your scenario is
extreme, but rather common.  I'm not pretending to be an expert on this, or to
create the image that _I_ know how/when to customize these settings.  I simply
know when _not_ to.

-- 
Stan

Re: performance tuning - relay

[Christian Purnomo]

HI Stan,

Thanks for your feedback.

I did try google for about an hour before turning to this list, I also
read http://postfix.nctu.edu.tw/TUNING_README.html several times.  It
all starts making some sense after reading it a couple of times today.

This is what I have done so far which works:

Server1 (MX host)

/etc/postfix/transport:
server2.com:relayhigh:[10.0.2.73]

/etc/postfix/main.cf:
relayhigh_destination_concurrency_limit = 150

/etc/postfix/master.cf:
relayhigh unix  -   -   n   -   200 smtp
-o smtp_connect_timeout=1s
-o fallback_relay=


I tried putting the original setting back to original as your per
suggestion, the mail count in the queue was still hovering at 9800 mark
for about 15 minutes, going down at a rate of 10-15 per minute which
was unsustainable.

With the settings above, the queue is now down to 2442 within 20
minutes.   It was at 21,000 mark when I sent my first email below
(nearly 12 hours ago), so the progress has been very minimal until the
change above.  The bottleneck has now switched from Server1 queue to
Server2 queue as server2 uses maildrop for local delivery.

I would take any suggestions - the settings above are based from reading
TUNING_README.html, it's trial and error. 

CP





Subject: Re: performance tuning - relay
Date: Fri, Jun 25, 2010 at 01:53:46AM -0500
Quoting Stan Hoeppner (s...@hardwarefreak.com):

: Christian Purnomo put forth on 6/24/2010 11:33 PM:
: 
:  /etc/postfix/transport:
:  server2.com:relay:[10.0.2.73]
:  
:  /etc/postfix/master.cf:
:  relay unix  -   -   n   -   200  smtp
:  -o smtp_helo_timeout=3s
:  -o smtp_connect_timeout=3s
:  -o disable_dns_lookups=yes
:  -o fallback_relay=
: 
: This was answered by Wietse 4 years ago on this list.  Took me ten seconds to
: find it via Google.  Read the entire thread on Neohapsis carefully and you'll
: find your answer, which is to remove all this custom stuff and go back to the
: defaults.  The first 2 of 4 above are the cause of your immediate problem, as
: they are wy too low.  The other two are just unnecessary.  And change
: max_proc back to 100.  You're probably not getting close to 100 processes
: running anyway.
: 
: http://archives.neohapsis.com/archives/postfix/2006-01/thread.html#1866
: 
:  Server 2 has the following configurations:
:  
:  /etc/postfix/master.cf:
:  smtp  inet  n   -   -   -   200 smtpd
: 
: Change the max process limit back to 100.  If everything else is configured
: correctly, you can drain an unbelievable amount of mail with less than 100
: smtp/smtpd processes.
: 
:  Could you please tell me what I'm missing here? I would like to improve
:  the rate that Server1 can relay messages to Server2.
: 
: If I may be frank, you missed the fact that you shouldn't mess with the
: default settings unless you really know what you're doing.  Custom settings
: here would require an extreme scenario.  I don't believe your scenario is
: extreme, but rather common.  I'm not pretending to be an expert on this, or to
: create the image that _I_ know how/when to customize these settings.  I simply
: know when _not_ to.
: 
: -- 
: Stan

Re: performance tuning - relay

[Victor Duchovni]

On Fri, Jun 25, 2010 at 01:53:46AM -0500, Stan Hoeppner wrote:

 Christian Purnomo put forth on 6/24/2010 11:33 PM:
 
  /etc/postfix/transport:
  server2.com:relay:[10.0.2.73]
  
  /etc/postfix/master.cf:
  relay unix  -   -   n   -   200  smtp
  -o smtp_helo_timeout=3s
  -o smtp_connect_timeout=3s
  -o disable_dns_lookups=yes
  -o fallback_relay=
 
 This was answered by Wietse 4 years ago on this list.  Took me ten seconds to
 find it via Google.  Read the entire thread on Neohapsis carefully and you'll
 find your answer, which is to remove all this custom stuff and go back to the
 defaults.  The first 2 of 4 above are the cause of your immediate problem, as
 they are wy too low.  The other two are just unnecessary.  And change
 max_proc back to 100.  You're probably not getting close to 100 processes
 running anyway.

The connect timeout is actually reasonable for internal
destinations. The helo timeout is a bit light. Both are only useful
if there are multiple internal servers, which seems unlikely given the
disable_dns_lookups=yes.  Why is that  setting there? It became obsolete
with Postfix 2.0 which was released 8 years ago.

The fallback_relay setting is correct, but even better is:

-o smtp_fallback_relay=

because the parameter has been renamed and the fallback_relay name
is a legacy alias, so is not always effective if the underlying real
variable is set in main.cf.

-- 
Viktor.

Re: performance tuning - relay

[Stan Hoeppner]

Christian Purnomo put forth on 6/25/2010 8:01 AM:

 With the settings above, the queue is now down to 2442 within 20
 minutes.   It was at 21,000 mark when I sent my first email below
 (nearly 12 hours ago), so the progress has been very minimal until the
 change above.  The bottleneck has now switched from Server1 queue to
 Server2 queue as server2 uses maildrop for local delivery.

Can you provide some more specs on server2?  IIRC you said you had a multidisk
RAID array on serv2.  What RAID level and how many disks?  What filesystem?
Are you running Courier with maildrop or the standalone maildrop with another
IMAP server?  What filtering, if any, are you doing with maildrop?  Using mbox
or maildir storage?  IIRC you previously said you're BCC'ing _everything_ into
a single mailbox (single address) on server2.  Is this correct?

And, lastly, was server2 in production for any amount of time before these
problems occurred, prompting your post, or is this a new server that you just
brought online?

-- 
Stan

Re: Performance tuning

[Ralf Hildebrandt]

* Brandon Hilkert bhilk...@vt.edu:

 any thoughts on how to mount the ramfs to get a true test of running the  
 queue in memory? 

In /etc/fstab:

/dev/shm  /var/spool/postfix tmpfs 
defaults,size=300m,mode=770,uid=0,gid=0 0 0

-- 
Ralf Hildebrandt
Postfix - Einrichtung, Betrieb und Wartung   Tel. +49 (0)30-450 570-155
http://www.computerbeschimpfung.de
Wenn Unternehmen Lehrstühle spenden, dann frage ich mich, ob die nicht
zu wenig Steuern zahlen, denn früher hätte der Staat davon den
Lehrstuhl gestellt.

Re: Performance tuning

[Wietse Venema]

Brandon Hilkert:
 I was able to mount it to a tmpfs partition. There was no change in 
 throughput with my script on a tmpfs vs ext3 drive.

That depends entirely on how good the script is.

You were talking about sending mail from Exchange though Postfix.

Now you are talking about a home-grown script? In that case we
first need to make sure that the script does a valid measurement.

Wietse

Re: Performance tuning

[Brandon Hilkert]

- Original Message - 
From: Wietse Venema wie...@porcupine.org

To: Brandon Hilkert bhilk...@vt.edu
Cc: postfix-users@postfix.org
Sent: Saturday, March 21, 2009 7:23 AM
Subject: Re: Performance tuning



Brandon Hilkert:

I was able to mount it to a tmpfs partition. There was no change in
throughput with my script on a tmpfs vs ext3 drive.


That depends entirely on how good the script is.

You were talking about sending mail from Exchange though Postfix.

Now you are talking about a home-grown script? In that case we
first need to make sure that the script does a valid measurement.


Exchange was never involved. On monday, I'm going to re-evaulate what the 
script is doing and implement the smtp-source tester to apply the parallel 
processing and see how the tmpfs changes that throughput.




Wietse 

Re: Performance tuning

[Brandon Hilkert]

- Original Message - 
From: Ralf Hildebrandt ralf.hildebra...@charite.de

To: postfix-users@postfix.org
Sent: Saturday, March 21, 2009 5:28 AM
Subject: Re: Performance tuning



* Brandon Hilkert bhilk...@vt.edu:


any thoughts on how to mount the ramfs to get a true test of running the
queue in memory?


In /etc/fstab:

/dev/shm  /var/spool/postfix tmpfs 
defaults,size=300m,mode=770,uid=0,gid=0 0 0




So tmpfs does use the ram ? I as able to get tmpfs to work, but there was 
some notion that it too uses the disk.



--
Ralf Hildebrandt
Postfix - Einrichtung, Betrieb und Wartung   Tel. +49 (0)30-450 
570-155

http://www.computerbeschimpfung.de
Wenn Unternehmen Lehrstühle spenden, dann frage ich mich, ob die nicht
zu wenig Steuern zahlen, denn früher hätte der Staat davon den
Lehrstuhl gestellt. 

Re: Performance tuning

[Terry Carmen]

Brandon Hilkert wrote:



So tmpfs does use the ram ? I as able to get tmpfs to work, but there 
was some notion that it too uses the disk.


It uses RAM, but will swap to disk if it needs to. There's no guarantee 
that it won't cause disk activity.


Terry

Re: Performance tuning

[Ralf Hildebrandt]

* Brandon Hilkert bhilk...@vt.edu:

 In /etc/fstab:

 /dev/shm  /var/spool/postfix tmpfs  
 defaults,size=300m,mode=770,uid=0,gid=0 0 0


 So tmpfs does use the ram ?

Yep

-- 
Ralf Hildebrandt
Postfix - Einrichtung, Betrieb und Wartung   Tel. +49 (0)30-450 570-155
http://www.computerbeschimpfung.de
If JavaScript is walking alone late at night through a bad part of
town with a pocket full of $20 bills, ActiveX is dropping your
trousers in the middle of the yard of a maximum-security prison,
bending over, and yelling 'Come and get it, boys!' 

Re: Performance tuning

[Victor Duchovni]

On Sat, Mar 21, 2009 at 02:08:51AM +, Duane Hill wrote:

 I'm just following this thread because of curiosity.

 tmpfs? Or, do you mean ramfs (like Ralf spoke of). I believe there was a 
 response already made by Wietse with regards to tmpfs that stated:

 tmpfs is backed by the swap file, which is on disk.

 Therefore, I would think you would not see a difference.

This is wrong.

-- 
Viktor.

Disclaimer: off-list followups get on-list replies or get ignored.
Please do not ignore the Reply-To header.

To unsubscribe from the postfix-users list, visit
http://www.postfix.org/lists.html or click the link below:
mailto:majord...@postfix.org?body=unsubscribe%20postfix-users

If my response solves your problem, the best way to thank me is to not
send an it worked, thanks follow-up. If you must respond, please put
It worked, thanks in the Subject so I can delete these quickly.

Re: Performance tuning

[Victor Duchovni]

On Sat, Mar 21, 2009 at 09:05:34AM -0400, Terry Carmen wrote:

 Brandon Hilkert wrote:


 So tmpfs does use the ram ? I as able to get tmpfs to work, but there was 
 some notion that it too uses the disk.

 It uses RAM, but will swap to disk if it needs to. There's no guarantee 
 that it won't cause disk activity.

Irrelevant, with tmpfs, sync() is free. That's enough, and in practice
there is plenty of RAM for benchmarks.

-- 
Viktor.

Disclaimer: off-list followups get on-list replies or get ignored.
Please do not ignore the Reply-To header.

To unsubscribe from the postfix-users list, visit
http://www.postfix.org/lists.html or click the link below:
mailto:majord...@postfix.org?body=unsubscribe%20postfix-users

If my response solves your problem, the best way to thank me is to not
send an it worked, thanks follow-up. If you must respond, please put
It worked, thanks in the Subject so I can delete these quickly.

Re: Performance tuning

[Costin Guşă]

On Thu, Mar 19, 2009 at 10:30 PM, Wietse Venema wie...@porcupine.org wrote:
 Brandon Hilkert:
 We send out a pretty volume of emails right now using a combination
 of SQL and IIS SMTP. We get rates now of about 5,000/min. We're
 looking to not only improve the rates, but incorporate DKIM/Domainkey
 signing into the process. The choice has been made to go with
 postfix along with a queue directory on an XFS file system.

 I'm using postfix as a relay, and having it sign the outgoing
 emails with DKIM. That process was about twice as slow as without
 it. Without DKIM, I'm getting a rate of 700/min.

 I was expecting much better performance out of the box. I realize
 in my tests I'm going from a single host, to another single postfix
 box that is receiving all the emails to one account, so there may
 be bottlenecks that exist in my test setup, that may not be there
 in our production environment.

 I'm really hoping to move away from MS SMTP and don't have tons
 of experience with Linux, so I'm hoping people could give me some
 good ideas about improving outbound performance. The server will
 not be directly exposed to the internet, so security is less of
 a concern as it will be behind several Cisco firewalls. This server
 will not be receiving any email, so nothing needs to be considered
 for that side.

 To improve performance, you need to identify the bottleneck.
 Usually, the file system is the slowest component.

 On Linux systems, fixing this can be as simple as editing the
 syslog.conf file and making sure that the syslog daemon does not
 sync the disk after every logfile record. Something like:

    mail.*      -/var/log/maillog

 Postfix was recently clocked at several thousand messages a second
 while sending mail, but receiving mail is much more expensive since
 each message needs to be made persistent on the local disk.

 You also need to keep in mind that you can't deliver mail as fast
 to ONE single machine as you can deliver to the whole Internet.

 In the one machine case, all the mail is fighting for access to
 the same file system, while with mail into the Internet, you are
 effectively spreading the load over many remote file systems.

        Wietse



also, for benchmarking and testing purposes only, if you have plenty
of ram, you can try to move the entire postfix system into ramfs and
completely disable logging, both on the sending and receiving machine

Re: Performance tuning

[Wietse Venema]

Brandon Hilkert:
 What's the best way to clearly identify that syslog is the issue?

Look in my reply. There is an example.

Wietse

 - Original Message - 
 From: Victor Duchovni victor.ducho...@morganstanley.com
 To: postfix-users@postfix.org
 Sent: Thursday, March 19, 2009 11:12 PM
 Subject: Re: Performance tuning
 
 
  On Thu, Mar 19, 2009 at 09:52:42PM -0400, Brandon Hilkert wrote:
 
  I understand what you mean about sending to one server. I'm going to try
  and setup a few more receiving servers so that I can more accurately
  simulate sending it out to the internet.
 
  Did you at least take time to rule out the syslog bottleneck? This is
  a common problem with stock Linux configurations, where syslog hammers
  the disk so hard that Postfix can't get any I/O done. Throughput as low
  as 10 msgs/sec is strongly suggestive of something like that, or perhaps
  just failure to send in parallel, or insufficient concurrency in output
  processing because all the test messages are routed to the same local(8)
  mailbox.
 
  Linux servers that are 5 years can do 300-400 msgs/sec, when the disk is
  managed by a RAID controller with an 8MB battery cache, and IIRC somewhere
  between 50 and 100 msgs/sec with the cache off.
 
  Start with syslog, then figure out where the messages are accumulating,
  see QSHAPE_README.
 
  -- 
  Viktor.
 
  Disclaimer: off-list followups get on-list replies or get ignored.
  Please do not ignore the Reply-To header.
 
  To unsubscribe from the postfix-users list, visit
  http://www.postfix.org/lists.html or click the link below:
  mailto:majord...@postfix.org?body=unsubscribe%20postfix-users
 
  If my response solves your problem, the best way to thank me is to not
  send an it worked, thanks follow-up. If you must respond, please put
  It worked, thanks in the Subject so I can delete these quickly. 
 
 
 

Re: Performance tuning

[Ralf Hildebrandt]

* Brandon Hilkert bhilk...@vt.edu:

 We send out a pretty volume of emails right now using a combination of
 SQL and IIS SMTP. We get rates now of about 5,000/min. We're looking to
 not only improve the rates, but incorporate DKIM/Domainkey signing into
 the process. The choice has been made to go with postfix along with a
 queue directory on an XFS file system.

You can check if the disk I/O is the bottleneck by simply putting the
queue fs in a RAM disk!
 
 I'm using postfix as a relay, and having it sign the outgoing emails
 with DKIM. That process was about twice as slow as without it. Without
 DKIM, I'm getting a rate of 700/min.

Signing takes time! htop will tell you IO rates and CPU usage...
 
-- 
Ralf Hildebrandt
Postfix - Einrichtung, Betrieb und Wartung   Tel. +49 (0)30-450 570-155
http://www.computerbeschimpfung.de
Windows 95 /n./ 32 bit extensions and a graphical shell for a 16 bit
patch to an 8 bit operating system originally coded for a 4 bit
microprocessor, written by a 2 bit company that can't stand 1 bit of
competition. 

Re: Performance tuning

[Rainer Frey (Inxmail GmbH)]

On Friday 20 March 2009 02:52:42 Brandon Hilkert wrote:

 As I mentioned, we're using the XFS system for the queue, does that provide
 any additional benefit, or would ext3 perform the same? Keep in mind, we
 will be dealing with 1,000,000 piece mailouts during a session. My findings
 were that XFS might handle more of the small html email files better,
 especially if they get backed up in a deferred queue for some reason, is
 this true?

I don't want to make an assumption whether this has anything to do with the 
problem. But although I haven't used XFS for a long time on production 
systems, I remember it was not recommended for fast access to many small 
files, but contrarily for few large files, e.g. media files for streaming. 
Using XFS for a mail spool for performance reason is new to me.

Rainer  

Re: Performance tuning

[Brandon Hilkert]

Is a simple ext3 partition usually the recommend file system?

- Original Message - 
From: Rainer Frey (Inxmail GmbH) rainer.f...@inxmail.de

To: postfix-users@postfix.org
Sent: Friday, March 20, 2009 6:54 AM
Subject: Re: Performance tuning



On Friday 20 March 2009 02:52:42 Brandon Hilkert wrote:

As I mentioned, we're using the XFS system for the queue, does that 
provide

any additional benefit, or would ext3 perform the same? Keep in mind, we
will be dealing with 1,000,000 piece mailouts during a session. My 
findings

were that XFS might handle more of the small html email files better,
especially if they get backed up in a deferred queue for some reason, is
this true?


I don't want to make an assumption whether this has anything to do with 
the

problem. But although I haven't used XFS for a long time on production
systems, I remember it was not recommended for fast access to many small
files, but contrarily for few large files, e.g. media files for streaming.
Using XFS for a mail spool for performance reason is new to me.

Rainer 

Re: Performance tuning

[Ralf Hildebrandt]

* Brandon Hilkert bhilk...@vt.edu:
 Is a simple ext3 partition usually the recommend file system?

Yes

-- 
Ralf Hildebrandt
Postfix - Einrichtung, Betrieb und Wartung   Tel. +49 (0)30-450 570-155
http://www.computerbeschimpfung.de
C makes it easy to shoot yourself in the foot. C++ makes it harder,
but when you do, it blows away your whole leg. -- Bjarne Stroustrup

Re: Performance tuning

[Rainer Frey (Inxmail GmbH)]

On Friday 20 March 2009 12:04:22 Brandon Hilkert wrote:
 Is a simple ext3 partition usually the recommend file system?

Please do not top post.

We use ext3 to have simple, repeatable, clear server setups without surprises 
or pitfalls. Performance is good enough for our needs, so I never actually 
compared performance. My comment on XFS is a vague memory from several years 
back, and not related to mail servers specifically. Actually, I'd expect XFS 
to be fast enough, but I also see no reason to use it in this case.

Probably you have a different problem, but if you have to setup the server 
again, I'd use ext3.

Rainer

Re: Performance tuning

[Brandon Hilkert]

What's the best way to clearly identify that syslog is the issue?

- Original Message - 
From: Victor Duchovni victor.ducho...@morganstanley.com

To: postfix-users@postfix.org
Sent: Thursday, March 19, 2009 11:12 PM
Subject: Re: Performance tuning



On Thu, Mar 19, 2009 at 09:52:42PM -0400, Brandon Hilkert wrote:


I understand what you mean about sending to one server. I'm going to try
and setup a few more receiving servers so that I can more accurately
simulate sending it out to the internet.


Did you at least take time to rule out the syslog bottleneck? This is
a common problem with stock Linux configurations, where syslog hammers
the disk so hard that Postfix can't get any I/O done. Throughput as low
as 10 msgs/sec is strongly suggestive of something like that, or perhaps
just failure to send in parallel, or insufficient concurrency in output
processing because all the test messages are routed to the same local(8)
mailbox.

Linux servers that are 5 years can do 300-400 msgs/sec, when the disk is
managed by a RAID controller with an 8MB battery cache, and IIRC somewhere
between 50 and 100 msgs/sec with the cache off.

Start with syslog, then figure out where the messages are accumulating,
see QSHAPE_README.

--
Viktor.

Disclaimer: off-list followups get on-list replies or get ignored.
Please do not ignore the Reply-To header.

To unsubscribe from the postfix-users list, visit
http://www.postfix.org/lists.html or click the link below:
mailto:majord...@postfix.org?body=unsubscribe%20postfix-users

If my response solves your problem, the best way to thank me is to not
send an it worked, thanks follow-up. If you must respond, please put
It worked, thanks in the Subject so I can delete these quickly. 

Re: Performance tuning

[Brandon Hilkert]

- Original Message - 
From: Ralf Hildebrandt ralf.hildebra...@charite.de

To: postfix-users@postfix.org
Sent: Friday, March 20, 2009 6:52 AM
Subject: Re: Performance tuning



* Brandon Hilkert bhilk...@vt.edu:


We send out a pretty volume of emails right now using a combination of
SQL and IIS SMTP. We get rates now of about 5,000/min. We're looking to
not only improve the rates, but incorporate DKIM/Domainkey signing into
the process. The choice has been made to go with postfix along with a
queue directory on an XFS file system.


You can check if the disk I/O is the bottleneck by simply putting the
queue fs in a RAM disk!



Sorry if this is a stupid question, but how do I go about this. I tried:

mkdir /ram
mount -t ramfs none /ram

and when I send a mail, postfix says there's not enough space in the queue. 
Should I be doing it a different way?


I also put the queue directory back on an ext3 partition and the rates went 
up by about a factor of two.


Also, by default the syslog messages were already set with 
 -/var/log/mail.log. I disabled mail logging all together and found no 
change in rates.


My disk is writing about 3 MB/s which should be well within it's range. I 
would hope even larger, but I would like to work out the ramfs and test for 
sure.




I'm using postfix as a relay, and having it sign the outgoing emails
with DKIM. That process was about twice as slow as without it. Without
DKIM, I'm getting a rate of 700/min.


Signing takes time! htop will tell you IO rates and CPU usage...

--
Ralf Hildebrandt
Postfix - Einrichtung, Betrieb und Wartung   Tel. +49 (0)30-450 
570-155

http://www.computerbeschimpfung.de
Windows 95 /n./ 32 bit extensions and a graphical shell for a 16 bit
patch to an 8 bit operating system originally coded for a 4 bit
microprocessor, written by a 2 bit company that can't stand 1 bit of
competition. 

Re: Performance tuning

[Brandon Hilkert]

- Original Message - 
From: Brandon Hilkert bhilk...@vt.edu

To: postfix-users@postfix.org
Sent: Friday, March 20, 2009 10:48 AM
Subject: Re: Performance tuning


- Original Message - 
From: Ralf Hildebrandt ralf.hildebra...@charite.de

To: postfix-users@postfix.org
Sent: Friday, March 20, 2009 6:52 AM
Subject: Re: Performance tuning



* Brandon Hilkert bhilk...@vt.edu:


We send out a pretty volume of emails right now using a combination of
SQL and IIS SMTP. We get rates now of about 5,000/min. We're looking to
not only improve the rates, but incorporate DKIM/Domainkey signing into
the process. The choice has been made to go with postfix along with a
queue directory on an XFS file system.


You can check if the disk I/O is the bottleneck by simply putting the
queue fs in a RAM disk!



Sorry if this is a stupid question, but how do I go about this. I tried:

mkdir /ram
mount -t ramfs none /ram


I was able to mount to tmpfs and found there was no change in performance.



and when I send a mail, postfix says there's not enough space in the 
queue. Should I be doing it a different way?


I also put the queue directory back on an ext3 partition and the rates 
went up by about a factor of two.


Also, by default the syslog messages were already set with 
 -/var/log/mail.log. I disabled mail logging all together and found no 
change in rates.


My disk is writing about 3 MB/s which should be well within it's range. I 
would hope even larger, but I would like to work out the ramfs and test 
for sure.




I'm using postfix as a relay, and having it sign the outgoing emails
with DKIM. That process was about twice as slow as without it. Without
DKIM, I'm getting a rate of 700/min.


Signing takes time! htop will tell you IO rates and CPU usage...

--
Ralf Hildebrandt
Postfix - Einrichtung, Betrieb und Wartung   Tel. +49 (0)30-450 
570-155

http://www.computerbeschimpfung.de
Windows 95 /n./ 32 bit extensions and a graphical shell for a 16 bit
patch to an 8 bit operating system originally coded for a 4 bit
microprocessor, written by a 2 bit company that can't stand 1 bit of
competition.

Re: Performance tuning

[Noel Jones]

Brandon Hilkert wrote:


I also put the queue directory back on an ext3 partition and the rates 
went up by about a factor of two.


Also, by default the syslog messages were already set with  
-/var/log/mail.log. I disabled mail logging all together and found no 
change in rates.


My disk is writing about 3 MB/s which should be well within it's range. 
I would hope even larger, but I would like to work out the ramfs and 
test for sure.


If your test destination is a single server delivering the 
mail {anywhere}, that's a good candidate for a bottleneck.


For testing, you should set up postfix smtp-sink test program 
included with the postfix source.  The program isn't installed 
by default, but can be found in the src/smtpstone directory of 
the postfix source tree.

http://www.postfix.org/smtp-sink.1.html
Run the smtp-sink program on the test destination server, 
listening on port 25.  Don't run postfix or another mail 
program on that server.


  -- Noel Jones

Re: Performance tuning

[Wietse Venema]

Brandon Hilkert:
 Sorry if this is a stupid question, but how do I go about this. I tried:
 
 mkdir /ram
 mount -t ramfs none /ram
 
 and when I send a mail, postfix says there's not enough space in the queue. 
 Should I be doing it a different way?

Postfix requires that the amount of space is several times larger
than the message size limit 

http://www.postfix.org/postconf.5.html#queue_minfree
http://www.postfix.org/postconf.5.html#message_size_limit

 I also put the queue directory back on an ext3 partition and the rates went 
 up by about a factor of two.

Which confirms that XFS is not the best choice to quickly create/delete
of lots of little files.

 Also, by default the syslog messages were already set with 
   -/var/log/mail.log. I disabled mail logging all together and found no 
 change in rates.

Good.

 My disk is writing about 3 MB/s which should be well within it's range. I 
 would hope even larger, but I would like to work out the ramfs and test for 
 sure.

Bitrates are completely irrelevant.

Email is about creating and deleting lots of tiny files.  

You need to forget everything you learned about web servers, as it
does not apply to email.

Wietse

Re: Performance tuning

[Brandon Hilkert]

- Original Message - 
From: Wietse Venema wie...@porcupine.org

To: Brandon Hilkert bhilk...@vt.edu
Cc: postfix-users@postfix.org
Sent: Friday, March 20, 2009 11:30 AM
Subject: Re: Performance tuning



Brandon Hilkert:

Sorry if this is a stupid question, but how do I go about this. I tried:

mkdir /ram
mount -t ramfs none /ram

and when I send a mail, postfix says there's not enough space in the 
queue.

Should I be doing it a different way?


Postfix requires that the amount of space is several times larger
than the message size limit

http://www.postfix.org/postconf.5.html#queue_minfree
http://www.postfix.org/postconf.5.html#message_size_limit


I was able to get it to mount to tmpfs and it showed no change in 
performance, so that would theoretcially rule out any existing disk issue 
right?




I also put the queue directory back on an ext3 partition and the rates 
went

up by about a factor of two.


Which confirms that XFS is not the best choice to quickly create/delete
of lots of little files.


Also, by default the syslog messages were already set with
  -/var/log/mail.log. I disabled mail logging all together and found no
change in rates.


Good.


My disk is writing about 3 MB/s which should be well within it's range. I
would hope even larger, but I would like to work out the ramfs and test 
for

sure.


Bitrates are completely irrelevant.


what is best to look out and compare?



Email is about creating and deleting lots of tiny files.

You need to forget everything you learned about web servers, as it
does not apply to email.

Wietse 

Re: Performance tuning

[Brandon Hilkert]

- Original Message - 
From: Wietse Venema wie...@porcupine.org

To: Brandon Hilkert bhilk...@vt.edu
Cc: Wietse Venema wie...@porcupine.org; postfix-users@postfix.org
Sent: Friday, March 20, 2009 12:45 PM
Subject: Re: Performance tuning



Brandon Hilkert:

 and when I send a mail, postfix says there's not enough space in the
 queue.
 Should I be doing it a different way?

 Postfix requires that the amount of space is several times larger
 than the message size limit

 http://www.postfix.org/postconf.5.html#queue_minfree
 http://www.postfix.org/postconf.5.html#message_size_limit

I was able to get it to mount to tmpfs and it showed no change in
performance, so that would theoretcially rule out any existing disk issue
right?


tmpfs is backed by the swap file, which is on disk.

 My disk is writing about 3 MB/s which should be well within it's 
 range. I
 would hope even larger, but I would like to work out the ramfs and 
 test

 for
 sure.

 Bitrates are completely irrelevant.

what is best to look out and compare?


When the disk is 100% busy, then it is the bottle neck.  Disks can
be 100% busy jumping around doing very little I/O.

As Noel suggested in earlier email, try running smtp-sink which
does no disk I/O at all. If things are still slow, then the problem
is network or MS-Exchange configuration.


I found the source. How do I build it in Debian? I realize this is probably 
a stupid question, but I don't have tons of experience with Linux.


Exchange hasn't been involved at all. I've been running everything from 
scripts, hoping to zero in on the bottleneck.




Wietse 

Re: Performance tuning

[Victor Duchovni]

On Fri, Mar 20, 2009 at 01:01:55PM -0400, Brandon Hilkert wrote:

 I've been running everything from scripts, hoping to zero in on the
 bottleneck.

How many messages are you sending in parallel in the injector scripts?

SMTP is a high latency half-duplex protocol, and a single injector will
never stress the capacity of the Postfix SMTP server. On a low delay,
high bandwidth network link, you need around 10 parallel injectors to
before you get anywhere near peak server throughput.

-- 
Viktor.

Disclaimer: off-list followups get on-list replies or get ignored.
Please do not ignore the Reply-To header.

To unsubscribe from the postfix-users list, visit
http://www.postfix.org/lists.html or click the link below:
mailto:majord...@postfix.org?body=unsubscribe%20postfix-users

If my response solves your problem, the best way to thank me is to not
send an it worked, thanks follow-up. If you must respond, please put
It worked, thanks in the Subject so I can delete these quickly.

Re: Performance tuning

[Brandon Hilkert]

- Original Message - 
From: Victor Duchovni victor.ducho...@morganstanley.com

To: postfix-users@postfix.org
Sent: Friday, March 20, 2009 1:20 PM
Subject: Re: Performance tuning



On Fri, Mar 20, 2009 at 01:01:55PM -0400, Brandon Hilkert wrote:


I've been running everything from scripts, hoping to zero in on the
bottleneck.


How many messages are you sending in parallel in the injector scripts?


My local script on the postfix machine itself. It's just a single shell 
script in a for loop.


Is there a better way to test?



SMTP is a high latency half-duplex protocol, and a single injector will
never stress the capacity of the Postfix SMTP server. On a low delay,
high bandwidth network link, you need around 10 parallel injectors to
before you get anywhere near peak server throughput.

--
Viktor.

Disclaimer: off-list followups get on-list replies or get ignored.
Please do not ignore the Reply-To header.

To unsubscribe from the postfix-users list, visit
http://www.postfix.org/lists.html or click the link below:
mailto:majord...@postfix.org?body=unsubscribe%20postfix-users

If my response solves your problem, the best way to thank me is to not
send an it worked, thanks follow-up. If you must respond, please put
It worked, thanks in the Subject so I can delete these quickly. 

Re: Performance tuning

[Victor Duchovni]

On Fri, Mar 20, 2009 at 01:32:26PM -0400, Brandon Hilkert wrote:


 - Original Message - From: Victor Duchovni 
 victor.ducho...@morganstanley.com
 To: postfix-users@postfix.org
 Sent: Friday, March 20, 2009 1:20 PM
 Subject: Re: Performance tuning


 On Fri, Mar 20, 2009 at 01:01:55PM -0400, Brandon Hilkert wrote:

 I've been running everything from scripts, hoping to zero in on the
 bottleneck.

 How many messages are you sending in parallel in the injector scripts?

 My local script on the postfix machine itself. It's just a single shell 
 script in a for loop.

 Is there a better way to test?

Yes, of course. Run multiple copies of your script if it
generates particularly representative (of your intended work-load)
content. Otherwise use smtp-source(1) with a suitable number of parallel
streams.

Measurements of single-stream performance just measure single transaction
latency on a lightly loaded server, not peak server throughput.

-- 
Viktor.

Disclaimer: off-list followups get on-list replies or get ignored.
Please do not ignore the Reply-To header.

To unsubscribe from the postfix-users list, visit
http://www.postfix.org/lists.html or click the link below:
mailto:majord...@postfix.org?body=unsubscribe%20postfix-users

If my response solves your problem, the best way to thank me is to not
send an it worked, thanks follow-up. If you must respond, please put
It worked, thanks in the Subject so I can delete these quickly.

Re: Performance tuning

[Victor Duchovni]

On Fri, Mar 20, 2009 at 11:27:27AM -0700, J Sloan wrote:

 For what it's worth, we've found ext3 to be far too slow for our needs.
 The best setup we've found is reiserfs, mounted with noatime and
 notail options -

Lets not start file system wars in this thread. The OP's problem is
largely unrelated to file-system selection. Most Postfix users do just
fine with a variety of file-systems, including ext3.

-- 
Viktor.

Disclaimer: off-list followups get on-list replies or get ignored.
Please do not ignore the Reply-To header.

To unsubscribe from the postfix-users list, visit
http://www.postfix.org/lists.html or click the link below:
mailto:majord...@postfix.org?body=unsubscribe%20postfix-users

If my response solves your problem, the best way to thank me is to not
send an it worked, thanks follow-up. If you must respond, please put
It worked, thanks in the Subject so I can delete these quickly.

Re: Performance tuning

[J Sloan]

For what it's worth, we've found ext3 to be far too slow for our needs.
The best setup we've found is reiserfs, mounted with noatime and
notail options -

Joe

Brandon Hilkert wrote:
 - Original Message - From: Ralf Hildebrandt
 ralf.hildebra...@charite.de
 To: postfix-users@postfix.org
 Sent: Friday, March 20, 2009 6:52 AM
 Subject: Re: Performance tuning


 * Brandon Hilkert bhilk...@vt.edu:

 We send out a pretty volume of emails right now using a combination of
 SQL and IIS SMTP. We get rates now of about 5,000/min. We're looking to
 not only improve the rates, but incorporate DKIM/Domainkey signing into
 the process. The choice has been made to go with postfix along with a
 queue directory on an XFS file system.

 You can check if the disk I/O is the bottleneck by simply putting the
 queue fs in a RAM disk!


 Sorry if this is a stupid question, but how do I go about this. I tried:

 mkdir /ram
 mount -t ramfs none /ram

 and when I send a mail, postfix says there's not enough space in the
 queue. Should I be doing it a different way?

 I also put the queue directory back on an ext3 partition and the rates
 went up by about a factor of two.

 Also, by default the syslog messages were already set with 
 -/var/log/mail.log. I disabled mail logging all together and found no
 change in rates.

 My disk is writing about 3 MB/s which should be well within it's
 range. I would hope even larger, but I would like to work out the
 ramfs and test for sure.


 I'm using postfix as a relay, and having it sign the outgoing emails
 with DKIM. That process was about twice as slow as without it. Without
 DKIM, I'm getting a rate of 700/min.

 Signing takes time! htop will tell you IO rates and CPU usage...

 -- 
 Ralf Hildebrandt
 Postfix - Einrichtung, Betrieb und Wartung   Tel. +49 (0)30-450
 570-155
 http://www.computerbeschimpfung.de
 Windows 95 /n./ 32 bit extensions and a graphical shell for a 16 bit
 patch to an 8 bit operating system originally coded for a 4 bit
 microprocessor, written by a 2 bit company that can't stand 1 bit of
 competition. 

Re: Performance tuning

[Wietse Venema]

Brandon Hilkert:
  what is best to look out and compare?
 
  When the disk is 100% busy, then it is the bottle neck.  Disks can
  be 100% busy jumping around doing very little I/O.
 
  As Noel suggested in earlier email, try running smtp-sink which
  does no disk I/O at all. If things are still slow, then the problem
  is network or MS-Exchange configuration.
 
 I found the source. How do I build it in Debian? I realize this is probably 
 a stupid question, but I don't have tons of experience with Linux.

cd $postfixsource
make

 Exchange hasn't been involved at all. I've been running everything from 
 scripts, hoping to zero in on the bottleneck.

You MUST run deliveries in parallel. SMTP is a stupid protocol;
one connection cannot saturate a server.

In fact, Postfix actually has safeguards for naive benchmark
measurements that send mail into Postfix over only a few connections.

Wietse

Re: Performance tuning

[Brandon Hilkert]

- Original Message - 
From: Ralf Hildebrandt ralf.hildebra...@charite.de

To: postfix-users@postfix.org
Sent: Friday, March 20, 2009 9:14 PM
Subject: Re: Performance tuning



* Brandon Hilkert bhilk...@vt.edu:


I was able to get it to mount to tmpfs and it showed no change in
performance, so that would theoretcially rule out any existing disk issue
right?


You said ext3 was faster, thus I think your ramfs test was flawed.



I was able to mount it to a tmpfs partition. There was no change in 
throughput with my script on a tmpfs vs ext3 drive.


So that would mean my disk is not a contribution factor right?


--
Ralf Hildebrandt
Postfix - Einrichtung, Betrieb und Wartung   Tel. +49 (0)30-450 
570-155

http://www.computerbeschimpfung.de
In our world, software has to be small, has to be debugged, has to
ship as part of a major initiative, has to avoid compatibility
problems, has to avoid end user calls.  Bill Gates 

Re: Performance tuning

[Ralf Hildebrandt]

* Brandon Hilkert bhilk...@vt.edu:

 You said ext3 was faster, thus I think your ramfs test was flawed.


 I was able to mount it to a tmpfs partition. There was no change in  
 throughput with my script on a tmpfs vs ext3 drive.

 So that would mean my disk is not a contribution factor right?

Probably.

-- 
Ralf Hildebrandt
Postfix - Einrichtung, Betrieb und Wartung   Tel. +49 (0)30-450 570-155
http://www.computerbeschimpfung.de

Re: Performance tuning

[Duane Hill]

On Fri, 20 Mar 2009, Brandon Hilkert wrote:

I was able to mount it to a tmpfs partition. There was no change in 
throughput with my script on a tmpfs vs ext3 drive.


So that would mean my disk is not a contribution factor right?


I'm just following this thread because of curiosity.

tmpfs? Or, do you mean ramfs (like Ralf spoke of). I believe there was a 
response already made by Wietse with regards to tmpfs that stated:


tmpfs is backed by the swap file, which is on disk.

Therefore, I would think you would not see a difference.

Re: Performance tuning

[Brandon Hilkert]

- Original Message - 
From: Duane Hill d.h...@yournetplus.com

To: postfix-users@postfix.org
Sent: Friday, March 20, 2009 10:08 PM
Subject: Re: Performance tuning



On Fri, 20 Mar 2009, Brandon Hilkert wrote:

I was able to mount it to a tmpfs partition. There was no change in 
throughput with my script on a tmpfs vs ext3 drive.


So that would mean my disk is not a contribution factor right?


I'm just following this thread because of curiosity.

tmpfs? Or, do you mean ramfs (like Ralf spoke of). I believe there was a 
response already made by Wietse with regards to tmpfs that stated:


tmpfs is backed by the swap file, which is on disk.

Therefore, I would think you would not see a difference.


Yeah I was kind of thinking the same thing. With tmpfs, you can easily set a 
fixed size, so postfix sees that and will allow you to place the queue in it 
because it appears larger than 1.5 * message_size_limit. However, when I 
mount a ramfs, there is no fixed size, so it looks like zero to postfix and 
will therefore not accept mail because it seems too small, even though the 
space would expand to it allow it.


I haven't figure out how to get around the messages. I tried setting the min 
queue free like I mentioned before, but anything over zero I would think 
would flag it as too much.


any thoughts on how to mount the ramfs to get a true test of running the 
queue in memory? 

Re: Performance tuning

[Wietse Venema]

Brandon Hilkert:
 We send out a pretty volume of emails right now using a combination
 of SQL and IIS SMTP. We get rates now of about 5,000/min. We're
 looking to not only improve the rates, but incorporate DKIM/Domainkey
 signing into the process. The choice has been made to go with
 postfix along with a queue directory on an XFS file system.
 
 I'm using postfix as a relay, and having it sign the outgoing
 emails with DKIM. That process was about twice as slow as without
 it. Without DKIM, I'm getting a rate of 700/min.
 
 I was expecting much better performance out of the box. I realize
 in my tests I'm going from a single host, to another single postfix
 box that is receiving all the emails to one account, so there may
 be bottlenecks that exist in my test setup, that may not be there
 in our production environment.
 
 I'm really hoping to move away from MS SMTP and don't have tons
 of experience with Linux, so I'm hoping people could give me some
 good ideas about improving outbound performance. The server will
 not be directly exposed to the internet, so security is less of
 a concern as it will be behind several Cisco firewalls. This server
 will not be receiving any email, so nothing needs to be considered
 for that side.

To improve performance, you need to identify the bottleneck.
Usually, the file system is the slowest component.

On Linux systems, fixing this can be as simple as editing the
syslog.conf file and making sure that the syslog daemon does not
sync the disk after every logfile record. Something like:

mail.*  -/var/log/maillog

Postfix was recently clocked at several thousand messages a second
while sending mail, but receiving mail is much more expensive since
each message needs to be made persistent on the local disk.

You also need to keep in mind that you can't deliver mail as fast
to ONE single machine as you can deliver to the whole Internet.

In the one machine case, all the mail is fighting for access to
the same file system, while with mail into the Internet, you are
effectively spreading the load over many remote file systems.

Wietse

Re: Performance tuning

[Brandon Hilkert]

Thanks for the response.

Our test system is a pretty standard SATA disk with 2GB memory. If disk is 
the necessary resource, would we see an immediate benefit by going to a SCSI 
disk or even a SCSI array, or does that hardware benefit flatten out at some 
point?


As I mentioned, we're using the XFS system for the queue, does that provide 
any additional benefit, or would ext3 perform the same? Keep in mind, we 
will be dealing with 1,000,000 piece mailouts during a session. My findings 
were that XFS might handle more of the small html email files better, 
especially if they get backed up in a deferred queue for some reason, is 
this true?


In your response, you mention that receiving is much more taxing than just 
simply sending. Would a relay machine (as ours is acting) be considered a 
receiving server, or are you just referring to a server where mail is the 
last stop for a locally delivered account? What do you think is a reasonable 
relay rate for a standard setup?


I understand what you mean about sending to one server. I'm going to try and 
setup a few more receiving servers so that I can more accurately simulate 
sending it out to the internet.


Thanks again for your help!

Brandon

- Original Message - 
From: Wietse Venema wie...@porcupine.org

To: Postfix users postfix-users@postfix.org
Sent: Thursday, March 19, 2009 4:30 PM
Subject: Re: Performance tuning



Brandon Hilkert:

We send out a pretty volume of emails right now using a combination
of SQL and IIS SMTP. We get rates now of about 5,000/min. We're
looking to not only improve the rates, but incorporate DKIM/Domainkey
signing into the process. The choice has been made to go with
postfix along with a queue directory on an XFS file system.

I'm using postfix as a relay, and having it sign the outgoing
emails with DKIM. That process was about twice as slow as without
it. Without DKIM, I'm getting a rate of 700/min.

I was expecting much better performance out of the box. I realize
in my tests I'm going from a single host, to another single postfix
box that is receiving all the emails to one account, so there may
be bottlenecks that exist in my test setup, that may not be there
in our production environment.

I'm really hoping to move away from MS SMTP and don't have tons
of experience with Linux, so I'm hoping people could give me some
good ideas about improving outbound performance. The server will
not be directly exposed to the internet, so security is less of
a concern as it will be behind several Cisco firewalls. This server
will not be receiving any email, so nothing needs to be considered
for that side.


To improve performance, you need to identify the bottleneck.
Usually, the file system is the slowest component.

On Linux systems, fixing this can be as simple as editing the
syslog.conf file and making sure that the syslog daemon does not
sync the disk after every logfile record. Something like:

   mail.* -/var/log/maillog

Postfix was recently clocked at several thousand messages a second
while sending mail, but receiving mail is much more expensive since
each message needs to be made persistent on the local disk.

You also need to keep in mind that you can't deliver mail as fast
to ONE single machine as you can deliver to the whole Internet.

In the one machine case, all the mail is fighting for access to
the same file system, while with mail into the Internet, you are
effectively spreading the load over many remote file systems.

Wietse 

Re: Performance tuning

[Victor Duchovni]

On Thu, Mar 19, 2009 at 09:52:42PM -0400, Brandon Hilkert wrote:

 I understand what you mean about sending to one server. I'm going to try 
 and setup a few more receiving servers so that I can more accurately 
 simulate sending it out to the internet.

Did you at least take time to rule out the syslog bottleneck? This is
a common problem with stock Linux configurations, where syslog hammers
the disk so hard that Postfix can't get any I/O done. Throughput as low
as 10 msgs/sec is strongly suggestive of something like that, or perhaps
just failure to send in parallel, or insufficient concurrency in output
processing because all the test messages are routed to the same local(8)
mailbox.

Linux servers that are 5 years can do 300-400 msgs/sec, when the disk is
managed by a RAID controller with an 8MB battery cache, and IIRC somewhere
between 50 and 100 msgs/sec with the cache off.

Start with syslog, then figure out where the messages are accumulating,
see QSHAPE_README.

-- 
Viktor.

Disclaimer: off-list followups get on-list replies or get ignored.
Please do not ignore the Reply-To header.

To unsubscribe from the postfix-users list, visit
http://www.postfix.org/lists.html or click the link below:
mailto:majord...@postfix.org?body=unsubscribe%20postfix-users

If my response solves your problem, the best way to thank me is to not
send an it worked, thanks follow-up. If you must respond, please put
It worked, thanks in the Subject so I can delete these quickly.