On 12.5.2013 18:46, Viktor Dukhovni wrote:
On Sun, May 12, 2013 at 11:22:22AM +0200, Patrik Rak wrote:
Yes, of course, I covered this in my earlier post, it would need to
be told an upper bound on the number of processes for deferred entries,
leaving the rest for new entries.
smtp_deferred_concurrency_limit = 0 | limit
OK, I am perfectly fine with this. In the context of the later mails,
I would perhaps call it <transport>_slow_transport_limit.
Even simpler: stop reading the deferred queue when more than N% of
the maximal number of recipient slots is from deferred mail.
We need to be a bit careful, starving the deferred queue can lead
to an ever growing deferred queue, with more messages coming and
getting deferred and never retried. If we are to impose a separate
deferred queue ceiling while continuing to take in new mail, we'll
need a much more stiff coupling between the output rate and the
input rate to avoid congested MTAs becoming bottomless pits for
ever more mail.
...
Interesting analysis. Few points, though:
- While I am not against revising the token based flow control, I would
just like to point out that adding explicit limit which Wietse suggest
doesn't change things much from how it works now (in a bad way, I mean).
Let me elaborate a bit:
Currently, when the the active queue becomes full, postfix favors new
mail (otherwise it round robins the queues). Now if the deferred mail
flows away about as fast as new mail, there's no big issue. However, if
the deferred mail delivery takes much longer than new mail delivery, the
deferred queue starts dominate the active queue up to the point when it
fills it entirely, at which point new mail delivery suffers badly,
incoming queue start to fill in and deferred mail may start to pile up
as well. I am just recapping things here, I am pretty sure you know this.
Now imagine you increase the active queue size by say 20% message slots,
and allow only new mail to use those. The situation wrt/ deferred queue
doesn't change at all. But you allow more new mail to flow by, reducing
the increase of the incoming queue or perhaps eliminating it entirely.
Overall, the situation improves.
Now if you instead reserve 20% of the active queue size for new mail
only, it's exactly the same as the example above, just scaled down by a
constant. And as the message default queue size is just an quite
arbitrarily chosen constant, I wouldn't be afraid of saying that for
most users it wouldn't make any difference. And those who would perhaps
need to hand tune this would likely want to hand tune it anyway.
- I also don't think we need any hysteresis in this case. When the
number of deferred mail in the active queue hits certain threshold,
simply scan incoming mail queue only, otherwise round robin the queues.
This quite naturally makes sure that whenever the number of deferred
mail drops below the threshold again, we use the deferred queue either
next time or the time after that, so we don't starve either queue.
Also note that this solution works fine regardless of if deferred mail
is actually slow or not, and likewise for the new mail. Each group
should automatically adjust the ratio of used slots over time according
to the ratio of the corresponding delivery speeds, with the deferred
queue being artificially capped slightly lower than the entire active queue.
- The whole point is to prevent deferred mail from blocking mail
deliveries. Therefore, we definitely do not want to stop accepting new
mail when the deferred mail in active queue reaches some level, as you
suggest. That would be just self-imposing yet another bottle neck. The
idea is that lot of the new mail can flow by regardless of how much of
the active queue is used by deferred mail.
- If you are afraid of the deferred queue piling up indefinitely, that's
not going to happen. As I just described above, it will get its share of
the active queue. The token flow mechanism shall take care of the rest.
And if it keeps piling anyway, the free space check will eventually kick
in and stop accepting incoming mail. :)
Patrik
