One possible bottle neck is the receiving mail relay. If you are
running sendmail I think it spawns a single process to deliver a
message regardless of the number of recipients.
I have had the same sort of problem you are experiencing and while
my mail is not as time critical as yours I wanted to figure out a
workaround.
What I did is send out a few test e-mails and ran a tail -f on the
maillog file. By doing that I could tell what mail servers were
delaying things. Doing it a number of times over the course of a few
days eliminated - to some degree - temporary problems. What I
then did was edit the list file and moved the offending e-mail
addresses to the bottom of the file and I contacted the admins of
the mail server. I know this isn't very technical but it did work for
me. Probably not too helpful for a high volume, large subscriber list.
Brian