On 2020/06/16 11:32 AM, Viktor Dukhovni wrote:
On Jun 16, 2020, at 12:39 PM, Wietse Venema <wie...@porcupine.org> wrote:
- Adding layer of sender-based round-robin selection. Not sure if
that would explode at large scale.
When the input mix approximates steady-state, FIFO is essentially optimal,
with each type of message getting its average share of the output in a fair
and timely manner. If/when we stray from FIFO, we need to be carefuly not
be simply underserving some fraction of the expected steady-state load
distribution.
This would mean being able to somehow detect a "burst" of traffic and
characterise to distinguish the messages that are contributing to the
burst from other messages. That's a tricky problem. The sender address
for bulk traffic is liable to have VERP tags, the client IP of interest
may be a few hops back from the edge system that encounters delays in
delivering an input burst to remote organizations (ADMDs in the language
of RFC5598).
- Adding some artificial 'cost' value that is computed while delivery
requests are added to per-destination queues. Cost could depend on
the number of delivery requests per sender email address, the message
size, and so on. Then, the scheduler could choose what-to-deliver
based on artificial cost in addition to the things that it already
considers now.
Ideally we'd have an algorithm that could group related messages into
a set of logical "bulk" sources, and apply the current bulk message
preëmption algorithm not only to multi-recipient mail, but also to
multi-message bulk sources. The hard part is the classification,
especially in a single-threaded queue manager that needs to do this
in O(1 millisecond).
Perhaps the best proxy for related messages is origin domain (ignoring
localpart) + approximate message size. Related messages are likely to
carry similar content of approximately the same size to all recipients.
But completely ignoring the localpart may be too coarse. It is not
obvious to me how to extract common elements from a per-recipient-salted
localpart...
A traditional multi-recipient message automatically qualifies as a single
source (same message size for all recipients and same origin domain).
We could attempt to group "related" messages in a fuzzy manner as above,
and then apply the existing preëmption algorithm. Still not sure how to
do a good job of the aggregation.
I've got a lot to learn here. Didn't expect so much detail!
Ideally, one could flag the big list somehow and not worry about the little
ones.
I turned off concurrency because TWC.com was rejecting bundles of emails.
Serious slow down.
Postfix steadily delivers, but would like to de-prioritize the list emails
so regular traffic is not delayed.
I will look up how to create transports and assign mail to them.
But it appears that there would be no way to redirect what is already in the
queue.
Is that correct?
(Sheepish grin) Feature request? (-:
