Re: Puting the Postfix's queue into RAM disk

[Istvan Prosinger]

The problem is tha there is that one VPS and I wanted 2nd opinions about 
my dirty plan.

Thanks

On 2015-11-15 19:03, Matthew McGehrin wrote:

Is it possible to configure a 2nd VPS instance just for
fallback_relay? That way your primary queue is only for deliveries,
and your 2nd instance can handle the bounces.

I was working for an Online Gaming company and we would deliver 1-2
million messages per day, we had 3 active queues, and 1 delayed queue.

On the active queues we would set the fallback_relay to the delayed
queue, thus ensuring only active mail was being sent first.



Istvan Prosinger wrote:



On 13.11.2015 22:53, Phil Stracchino wrote:

On 11/13/15 14:17, Istvan Prosinger wrote:
I got two options that I know of. Signifficantly shortening the 
queue

lifetime, or (not) losing the queue from the RAM disk.
Just trying to measure which is worse (or to hear something new for 
me)


If you lack disk space for the queue, the server instance isn't up to
the job in the first place.

If you really think that keeping the queue in RAM is a better option
than on disk, despite that you probably have far more disk than RAM,
consider using a tmpfs instead of a RAMdisk, with the size of the 
tmpfs
capped at the size RAMdisk you were plannind.  A RAMdisk will use all 
of

that RAM all of the time, whether needed for queued mail or not.  A
tmpfs will consume only as much RAM at any given time as it needs 
right

then.

Re: Puting the Postfix's queue into RAM disk

[Matthew McGehrin]

Is it possible to configure a 2nd VPS instance just for fallback_relay? 
That way your primary queue is only for deliveries, and your 2nd 
instance can handle the bounces.


I was working for an Online Gaming company and we would deliver 1-2 
million messages per day, we had 3 active queues, and 1 delayed queue.


On the active queues we would set the fallback_relay to the delayed 
queue, thus ensuring only active mail was being sent first.




Istvan Prosinger wrote:



On 13.11.2015 22:53, Phil Stracchino wrote:

On 11/13/15 14:17, Istvan Prosinger wrote:

I got two options that I know of. Signifficantly shortening the queue
lifetime, or (not) losing the queue from the RAM disk.
Just trying to measure which is worse (or to hear something new for me)


If you lack disk space for the queue, the server instance isn't up to
the job in the first place.

If you really think that keeping the queue in RAM is a better option
than on disk, despite that you probably have far more disk than RAM,
consider using a tmpfs instead of a RAMdisk, with the size of the tmpfs
capped at the size RAMdisk you were plannind.  A RAMdisk will use all of
that RAM all of the time, whether needed for queued mail or not.  A
tmpfs will consume only as much RAM at any given time as it needs right
then.

Re: Puting the Postfix's queue into RAM disk

[Istvan Prosinger]

On 13.11.2015 22:53, Phil Stracchino wrote:

On 11/13/15 14:17, Istvan Prosinger wrote:

I got two options that I know of. Signifficantly shortening the queue
lifetime, or (not) losing the queue from the RAM disk.
Just trying to measure which is worse (or to hear something new for me)


If you lack disk space for the queue, the server instance isn't up to
the job in the first place.

If you really think that keeping the queue in RAM is a better option
than on disk, despite that you probably have far more disk than RAM,
consider using a tmpfs instead of a RAMdisk, with the size of the tmpfs
capped at the size RAMdisk you were plannind.  A RAMdisk will use all of
that RAM all of the time, whether needed for queued mail or not.  A
tmpfs will consume only as much RAM at any given time as it needs right
then.





Thanks. It's not about a lack of space, puting it that word it's about a 
lack of an SSD drive.

Namely, it's about processing speed.

Re: Puting the Postfix's queue into RAM disk

[Patrick Ben Koetter]

* Istvan Prosinger :
> Hello,
> 
> I'll have a project to send 300-400k emails a day from a new IP
> address with one server. This can build up a signifficant mail queue
> on the server.
> We have several similar solutions already working but this time the
> idea is to have me do this on a VPS (no SSD drives involved),
> h

Let's take a look at the numbers and bring them down to something we can
handle in our brains:

400.000,00 day
 16.666,67 hour
277,78 minute
  4,63 second

Sending 4,63 msg/sec is something regular hardware can do. There's no
technical need to use SSDs or even a tmpfs.

If the machine goes down you risk loosing all messages in delivery. You can
resend them, but then you will have to figure out which ones had already been
delivered and which were still on the machine. I'd wouldn't risk that. It
doesn't pay and you do have the choice to go for something more stable.

p@rick

-- 
[*] sys4 AG
 
https://sys4.de, +49 (89) 30 90 46 64
Franziskanerstraße 15, 81669 München
 
Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Marc Schiffbauer
Aufsichtsratsvorsitzender: Florian Kirstein
 

Re: Puting the Postfix's queue into RAM disk

[Leonardo Rodrigues]

Em 13/11/15 08:09, Patrick Ben Koetter escreveu:

* Istvan Prosinger :

Hello,

I'll have a project to send 300-400k emails a day from a new IP
address with one server. This can build up a signifficant mail queue
on the server.
We have several similar solutions already working but this time the
idea is to have me do this on a VPS (no SSD drives involved),
h

Let's take a look at the numbers and bring them down to something we can
handle in our brains:

400.000,00 day
  16.666,67 hour
 277,78 minute
   4,63 second

Sending 4,63 msg/sec is something regular hardware can do. There's no
technical need to use SSDs or even a tmpfs.



That math fails because usually who send that much emails wants 
them to be delivered as fast as possible, so using the whole 24h account 
doesnt seem to be the ideal.



--


Atenciosamente / Sincerily,
Leonardo Rodrigues
Solutti Tecnologia
http://www.solutti.com.br

Minha armadilha de SPAM, NÃO mandem email
gertru...@solutti.com.br
My SPAMTRAP, do not email it

Puting the Postfix's queue into RAM disk

[Istvan Prosinger]

Hello,

I'll have a project to send 300-400k emails a day from a new IP address 
with one server. This can build up a signifficant mail queue on the 
server.
We have several similar solutions already working but this time the idea 
is to have me do this on a VPS (no SSD drives involved), h


What's your verdict about the idea from the subject?

All best,
Istvan

Re: Puting the Postfix's queue into RAM disk

[Phil Stracchino]

On 11/13/15 04:44, Istvan Prosinger wrote:
> Hello,
> 
> I'll have a project to send 300-400k emails a day from a new IP address 
> with one server. This can build up a signifficant mail queue on the 
> server.
> We have several similar solutions already working but this time the idea 
> is to have me do this on a VPS (no SSD drives involved), h
> 
> What's your verdict about the idea from the subject?

Let me pose a single question by way of answer:

"How much do you care about irretrievably losing undelivered mail?"


-- 
  Phil Stracchino
  Babylon Communications
  ph...@caerllewys.net
  p...@co.ordinate.org
  Landline: 603.293.8485

Re: Puting the Postfix's queue into RAM disk

[Ken Simpson]

We deliver tens of millions per server per day using SSD spools...
On Fri, Nov 13, 2015 at 11:18 AM Istvan Prosinger 
wrote:

> I got two options that I know of. Signifficantly shortening the queue
> lifetime, or (not) losing the queue from the RAM disk.
> Just trying to measure which is worse (or to hear something new for me)
>
> On 13.11.2015 16:17, Phil Stracchino wrote:
> > On 11/13/15 04:44, Istvan Prosinger wrote:
> >> Hello,
> >>
> >> I'll have a project to send 300-400k emails a day from a new IP address
> >> with one server. This can build up a signifficant mail queue on the
> >> server.
> >> We have several similar solutions already working but this time the idea
> >> is to have me do this on a VPS (no SSD drives involved), h
> >>
> >> What's your verdict about the idea from the subject?
> >
> > Let me pose a single question by way of answer:
> >
> > "How much do you care about irretrievably losing undelivered mail?"
> >
> >
>

Re: Puting the Postfix's queue into RAM disk

[Istvan Prosinger]

I got two options that I know of. Signifficantly shortening the queue 
lifetime, or (not) losing the queue from the RAM disk.

Just trying to measure which is worse (or to hear something new for me)

On 13.11.2015 16:17, Phil Stracchino wrote:

On 11/13/15 04:44, Istvan Prosinger wrote:

Hello,

I'll have a project to send 300-400k emails a day from a new IP address
with one server. This can build up a signifficant mail queue on the
server.
We have several similar solutions already working but this time the idea
is to have me do this on a VPS (no SSD drives involved), h

What's your verdict about the idea from the subject?


Let me pose a single question by way of answer:

"How much do you care about irretrievably losing undelivered mail?"

Re: Puting the Postfix's queue into RAM disk

[Phil Stracchino]

On 11/13/15 14:17, Istvan Prosinger wrote:
> I got two options that I know of. Signifficantly shortening the queue 
> lifetime, or (not) losing the queue from the RAM disk.
> Just trying to measure which is worse (or to hear something new for me)

If you lack disk space for the queue, the server instance isn't up to
the job in the first place.

If you really think that keeping the queue in RAM is a better option
than on disk, despite that you probably have far more disk than RAM,
consider using a tmpfs instead of a RAMdisk, with the size of the tmpfs
capped at the size RAMdisk you were plannind.  A RAMdisk will use all of
that RAM all of the time, whether needed for queued mail or not.  A
tmpfs will consume only as much RAM at any given time as it needs right
then.



-- 
  Phil Stracchino
  Babylon Communications
  ph...@caerllewys.net
  p...@co.ordinate.org
  Landline: 603.293.8485

Re: Puting the Postfix's queue into RAM disk

[Istvan Prosinger]

Ok. As I mentioned, SSD is not an option on this project. Only RAM or 
Raid 10 (shared with other VPSes)


On 13.11.2015 20:56, Ken Simpson wrote:

We deliver tens of millions per server per day using SSD spools...
On Fri, Nov 13, 2015 at 11:18 AM Istvan Prosinger > wrote:

I got two options that I know of. Signifficantly shortening the queue
lifetime, or (not) losing the queue from the RAM disk.
Just trying to measure which is worse (or to hear something new for me)

On 13.11.2015 16:17, Phil Stracchino wrote:
 > On 11/13/15 04:44, Istvan Prosinger wrote:
 >> Hello,
 >>
 >> I'll have a project to send 300-400k emails a day from a new IP
address
 >> with one server. This can build up a signifficant mail queue on the
 >> server.
 >> We have several similar solutions already working but this time
the idea
 >> is to have me do this on a VPS (no SSD drives involved), h
 >>
 >> What's your verdict about the idea from the subject?
 >
 > Let me pose a single question by way of answer:
 >
 > "How much do you care about irretrievably losing undelivered mail?"
 >
 >

Re: Puting the Postfix's queue into RAM disk

[Harald Koch]

On 13 November 2015 at 07:51, Istvan Prosinger  wrote:

>
> The point here is that at the start of this, a temporary deferred mail
> queue will build up signifficantly pushing most of the load on the file
> system, and the idea is to speed up the queue processing to prevent killing
> the server (extending the queue size it can process in a time frame)
> My client wouldn't make a problem to deliver the email after a week but
> I'm afraid that queue lifetime is out of the question (again, because of
> the queue size it would build up)
>

IME the only issue with queuing is the fsync() overhead when writing to the
queue. For the volumes you're talking about spinning disks might be slow,
but SSDs should be just fine. Everything else should already be cached in
memory by the operating system - in fact, using a tmpfs "steals" memory
that could be used for the OS disk buffer cache.

I think you will buy yourself a lot of (probably unnecessary) pain trying
to subvert Postfix's built-in reliability mechanisms.

(you mentioned using a VPS - all the major ones are 100% SSD these days, at
least here in North America. My Linode VPS is the fastest machine I have!
Also, I would expect modern VPS providers to be running battery-backed
caches down at the host layer - which means you'd already be running in
what is effectively an all-RAM scenario...)

-- 
Harald

Re: Puting the Postfix's queue into RAM disk

[Leonardo Rodrigues]

Em 13/11/15 08:09, Patrick Ben Koetter escreveu:

* Istvan Prosinger :

Hello,

I'll have a project to send 300-400k emails a day from a new IP
address with one server. This can build up a signifficant mail queue
on the server.
We have several similar solutions already working but this time the
idea is to have me do this on a VPS (no SSD drives involved),
h

If the machine goes down you risk loosing all messages in delivery. You can
resend them, but then you will have to figure out which ones had already been
delivered and which were still on the machine. I'd wouldn't risk that. It
doesn't pay and you do have the choice to go for something more stable.




I do run a similar system, which deliveres about 150-180k emails 
daily, and i do use tmpfs for the postfix spool. To avoid the risk of 
losing all data in the case of a system crash, which by the way never 
happened on the 3 years the system is running , i use an rsync routine 
to sync the tmpfs to the disk. Another script, which runs on the machine 
startup, check if the folders are there on the tmpfs and, if they are 
not there, creates them and sync from the disk to the tmpfs. The 'sync 
from tmpfs to disk' is also configured to run on an eventual system 
reboot and will trigger the 'sync from disk to tmpfs' when the system 
boots up.




--


Atenciosamente / Sincerily,
Leonardo Rodrigues
Solutti Tecnologia
http://www.solutti.com.br

Minha armadilha de SPAM, NÃO mandem email
gertru...@solutti.com.br
My SPAMTRAP, do not email it

Re: Puting the Postfix's queue into RAM disk

[Istvan Prosinger]

On 2015-11-13 12:53, Leonardo Rodrigues wrote:

Em 13/11/15 08:09, Patrick Ben Koetter escreveu:

* Istvan Prosinger :

Hello,

I'll have a project to send 300-400k emails a day from a new IP
address with one server. This can build up a signifficant mail queue
on the server.
We have several similar solutions already working but this time the
idea is to have me do this on a VPS (no SSD drives involved),
h
Let's take a look at the numbers and bring them down to something we 
can

handle in our brains:

400.000,00 day
  16.666,67 hour
 277,78 minute
   4,63 second

Sending 4,63 msg/sec is something regular hardware can do. There's no
technical need to use SSDs or even a tmpfs.



That math fails because usually who send that much emails wants
them to be delivered as fast as possible, so using the whole 24h
account doesnt seem to be the ideal.



The point here is that at the start of this, a temporary deferred mail 
queue will build up signifficantly pushing most of the load on the file 
system, and the idea is to speed up the queue processing to prevent 
killing the server (extending the queue size it can process in a time 
frame)
My client wouldn't make a problem to deliver the email after a week but 
I'm afraid that queue lifetime is out of the question (again, because of 
the queue size it would build up)