Re: [Mailman-Users] Ignore DMARC bounces?
Could we not send the message out as usual, then on a p=reject bounce, forward the original message (so it comes from the mailing list) along with an explanation of what is transpiring to the bounced user, plus to the message author? Maybe include a note suggesting the author change mail providers. This way if the message author's domain causes 20 bounces, they get 20 messages letting them know they need to change mail providers. -Original Message- From: Mailman-Users [mailto:mailman-users-bounces+bryan.wright=rigaku@python.org] On Behalf Of Stephen J. Turnbull Sent: Saturday, June 14, 2014 7:42 AM To: Sparr Cc: mailman-users@python.org Subject: Re: [Mailman-Users] Ignore DMARC bounces? Sparr writes: > Modifying the messages bothers me (and a lot of other people, as > > indicated by the last dozen times similar conversations have been had, > > about changing Reply-To and From and Subject and ...) and should be > the > last resort. Well, actually the point is that lists need to do fewer modifications than they already do. DMARC has two tests, one for the domain in From being equivalent to the IP of the SMTP client, which will fail unless the author is at the mailing list's domain, and a DKIM signature. The signature will survive and be valid at the recipient in the case that the message is completely unmodified. However, mailing lists typically make one or more of the following modifications: add a list tag to the Subject field, add a header or footer to the body, remove prohibited MIME bodies (.exes, text/html, etc), or transform text/html to text/plain. Any of those will cause the usual DKIM signature to be invalidated. DMARC-using domains typically sign From (required by the DKIM protocol), To, Cc, Subject, and the whole body (effectively including the end of the message, preventing appended material such as a footer). My personal opinion is that these traditional changes are expected and desired by mailing list subscribers, and that posting from "p=reject" domains is thereby a violation of the policy of the "p=reject" domain, and places other subscribers at risk. I think mailing lists should reject such posts (if the signature is valid), or silently discard them (if it is not). However, subscribers from those domains are unlikely to agree -- Mailman-Users mailing list Mailman-Users@python.org https://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Security Policy: http://wiki.list.org/x/QIA9 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-users/bryan.wright%40rigaku.com -- Mailman-Users mailing list Mailman-Users@python.org https://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Security Policy: http://wiki.list.org/x/QIA9 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-users/archive%40jab.org
Re: [Mailman-Users] Performance issues with 12K subscribers and 14K subscribers and personalized footers.
Mark, Thanks for the help. Followed the instructions in the FAQ to get sendmail listening on an alternate port in deferred mode. That made a huge difference, although not as speedy as your environment, 8122 messages made it from mailman to sendmail in about 30 minutes. Sep 30 10:07:18 2013 (22267) smtp to catherine.klein for 8122 recips, completed in 1782.344 seconds Now I just need to fine-tune sendmail so it speeds up as well, but that's a question for elsewhere. Thanks again for the help. Bryan Wright -Original Message- From: Mark Sapiro [mailto:m...@msapiro.net] Sent: Saturday, September 28, 2013 3:33 PM To: Bryan Wright; mailman-users@python.org Subject: Re: [Mailman-Users] Performance issues with 12K subscribers and 14K subscribers and personalized footers. On 09/27/2013 10:40 PM, Bryan Wright wrote: > > So back to the speed. If multiple qrunners won't fix the issue, is > there any way to speed this up? To get it to process multiple outgoing > messages at once? Before trying to send from Mailman in parallel, you need to look into speeding up the SMTP interaction between Mailman and sendmail, bevcause that's where the real delay exists. > Could I create multiple lists and then just have the lists subscribed > to the one list? Then each of these lists could use a qrunner? Yes and no. You could create an umbrella list or use regular_include_lists (see the FAQ at <http://wiki.list.org/x/TIA9>) to create multiple outgoing messages with fewer recipients, but there is no guarantee that these message's out/ queue entries would all wind up in different OutgoingRunner slices. Actually ensuring that each message is processed by a different OutgoingRunner would, I think, require some kludgey code mods. It would be easier and more productive to modify SMTPDirect to use multiple threads. (There is an 'experimental' MAX_DELIVERY_THREADS setting documented in Defaults.py, but it is not currently implemented.) > Could disk access time be an issue? Could I place the mailman queue > directory in a ram disk? Putting the queue on a ram disk won't help. The queue entry is only picked up at the start of processing and whether it takes microseconds or milliseconds, it is not the source of any significant delay. Putting the server's swap space on a ram disk could help, but there are other things to try first. As I said in my original reply, see the FAQs at <http://wiki.list.org/x/doA9> and <http://wiki.list.org/x/q4A9>. A local caching DNS may help and if sendmail is doing DNS verification at incoming SMTP time, stopping that will really help. Also ensure you haven't changed the default SMTP_MAX_SESSIONS_PER_CONNECTION = 0 setting. Note: as a point of reference, my production server is not very high powered, runs CentOS 5, and has no specific performance tuning. Messages from Mailman use Mailman's VERP feature so each recipient gets an individual message whether or not personalization is on. Mailman delivers over 50 messages per second to Postfix or 12,000 messages in 4 minutes. This is based on up to 600+ messages (my maximum list size is currently 629), and may slow down some with larger lists as Postfix gets busier sending, but I'm sure delivery to Postfix of 12,000 messages would take minutes as opposed to hours. -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, Californiabetter use your sense - B. Dylan -- Mailman-Users mailing list Mailman-Users@python.org https://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Security Policy: http://wiki.list.org/x/QIA9 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-users/archive%40jab.org
Re: [Mailman-Users] Performance issues with 12K subscribers and 14K subscribers and personalized footers.
That makes sense now. And explains why all my attempts to speed up processing have hear no effect. So the final message should be logged once delivery is complete. So back to the speed. If multiple qrunners won't fix the issue, is there any way to speed this up? To get it to process multiple outgoing messages at once? Could I create multiple lists and then just have the lists subscribed to the one list? Then each of these lists could use a qrunner? Could disk access time be an issue? Could I place the mailman queue directory in a ram disk? Sent from my Verizon Wireless 4G LTE Smartphone Original message From: Mark Sapiro Date: 09/28/2013 12:00 AM (GMT-06:00) To: mailman-users@python.org Subject: Re: [Mailman-Users] Performance issues with 12K subscribers and 14K subscribers and personalized footers. On 09/27/2013 09:32 PM, Mark Sapiro wrote: > Also, you only see them while delivery of the post is > proceeding because of multiple qrunners, but I'm not sure why locking > doesn't stop even that. Actually, it's because OutgoingRunner doesn't lock the list. I was confusing it with IncomingRunner which does lock the list. -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, Californiabetter use your sense - B. Dylan -- Mailman-Users mailing list Mailman-Users@python.org https://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Security Policy: http://wiki.list.org/x/QIA9 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-users/bryan.wright%40rigaku.com -- Mailman-Users mailing list Mailman-Users@python.org https://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Security Policy: http://wiki.list.org/x/QIA9 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-users/archive%40jab.org
[Mailman-Users] Performance issues with 12K subscribers and 14K subscribers and personalized footers.
Hello, I'm trying to figure out some performance issues with mailman. I have it configured for SMTP direct, and pointed at localhost, port 25. The MTA is sendmail. Sendmail is configured to use our exchange server as a smart host. We have two lists configured and send out 1 message a month to each list. It usually takes about 12-24 hours to finish sending all the messages. There are never more than about 5 to 10 messages in the sendmail queue at any given time, so this leads me to believe that the bottleneck is not on the sendmail side. I've configured mailman for 48 outgoing qrunners, and have tried as many as 72 qrunners, with no noticeable difference, it's like it's using only the 1st outgoing qrunner to process the message and the rest stay there idle. And then it uses that one qrunner process to create each separate message. So 12k individual messages processed by one qrunner thread each time. Is there some way to speed this up and/or get it to use multiple outgoing qrunners simultaneously for the same post, since with personalization it has to create individual copies of each message? The other odd thing is it doesn't log all 12K messages in the smtp log from mailman, but it logs more than 1. I would have thought it would either log only 1 message, or all 12K messages. Here is the SMTP log from mailman: Sep 26 14:13:17 2013 (12407) smtp to newsletter for 1 recips, completed in 10.368 seconds Sep 26 15:11:57 2013 (12439) smtp to newsletter for 1 recips, completed in 8.107 seconds Sep 26 15:42:19 2013 (12381) smtp to newsletter for 1 recips, completed in 0.086 seconds Sep 26 16:56:31 2013 (12458) smtp to newsletter for 1 recips, completed in 11.179 seconds Sep 26 17:21:12 2013 (12469) smtp to newsletter for 1 recips, completed in 1.012 seconds Sep 26 19:10:10 2013 (12429) smtp to newsletter for 1 recips, completed in 10.988 seconds Sep 26 20:23:43 2013 (12449) smtp to newsletter for 1 recips, completed in 6.958 seconds Sep 27 00:47:25 2013 (12380) smtp to newsletter for 1 recips, completed in 6.096 seconds Sep 27 01:17:13 2013 (12448) smtp to newsletter for 1 recips, completed in 6.166 seconds Sep 27 02:11:27 2013 (12442) smtp to newsletter for 1 recips, completed in 5.095 seconds Sep 27 02:30:36 2013 (12399) smtp to newsletter for 1 recips, completed in 6.013 seconds Sep 27 02:40:08 2013 (12472) smtp to newsletter for 1 recips, completed in 5.356 seconds Sep 27 05:07:31 2013 (12380) smtp to newsletter for 1 recips, completed in 0.083 seconds Sep 27 05:08:20 2013 (12422) smtp to newsletter for 1 recips, completed in 0.077 seconds Sep 27 05:08:23 2013 (12474) smtp to newsletter for 1 recips, completed in 0.073 seconds Sep 27 05:31:24 2013 (12438) smtp to newsletter for 1 recips, completed in 10.251 seconds Sep 27 06:56:34 2013 (12469) smtp to newsletter for 1 recips, completed in 5.967 seconds Sep 27 06:56:50 2013 (12396) smtp to newsletter for 1 recips, completed in 5.153 seconds Sep 27 07:02:33 2013 (12452) smtp to newsletter for 1 recips, completed in 10.248 seconds Sep 27 07:59:36 2013 (12406) smtp to newsletter for 1 recips, completed in 11.085 seconds Sep 27 08:05:46 2013 (12417) smtp to newsletter for 1 recips, completed in 10.141 seconds Sep 27 08:10:38 2013 (12449) smtp to newsletter for 1 recips, completed in 10.232 seconds Sep 27 08:11:40 2013 (12435) smtp to newsletter for 1 recips, completed in 10.093 seconds Sep 27 08:28:37 2013 (12473) smtp to newsletter for 1 recips, completed in 5.124 seconds Sep 27 08:35:08 2013 (12409) smtp to newsletter for 1 recips, completed in 10.346 seconds Sep 27 09:51:39 2013 (12428) smtp to newsletter for 1 recips, completed in 5.045 seconds Sep 27 10:34:31 2013 (12393) smtp to newsletter for 1 recips, completed in 0.049 seconds Sep 27 11:39:33 2013 (12396) smtp to newsletter for 1 recips, completed in 15.162 seconds Sep 27 11:40:15 2013 (12389) smtp to newsletter for 1 recips, completed in 10.073 seconds Sep 27 11:53:41 2013 (12438) smtp to newsletter for 1 recips, completed in 5.098 seconds Sep 27 13:00:33 2013 (12430) smtp to newsletter for 1 recips, completed in 10.125 seconds Thanks in advance for any advice/help you can give. Bryan -- Mailman-Users mailing list Mailman-Users@python.org https://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Security Policy: http://wiki.list.org/x/QIA9 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-users/archive%40jab.org
