RE: Mailing to email list using CF
I don't get it. If you check the addresses first, why check again? If an existing list, just check the format once. Mostly for the PHB's satisfaction.Back before I was around, the guys who originally built the system didn't bother to do ANY validation, at input or send time.They had lots of problems, as you might imagine.That's about 99.99% fixed, but there are still some records that we can't touch (for political reasons) that are from back then, that may or may not be valid email addresses.Hence the double validation. Cheers, barneyb [Todays Threads] [This Message] [Subscription] [Fast Unsubscribe] [User Settings] [Donations and Support]
Re: Mailing to email list using CF
So doing a DNS check will not help much in verifying an account I just checked my list via dns using worldcast and about 10% failed, but many with messages about not having an IP they can reverse look-up, so useless. To send a newsletter in a slower way with CF, use an email trickler (a page that reloads and cycles through a query). Matt Robertson has posted it before here and one on easy-cfm or whatever its called. Google for coldfusion and email trickler. Kym, very helpful. Hard to know what emails are getting rejected for exactly what, or just tossed, so what do estimate the hourly threshhold is for big guys like AOL, Hotmail, etc? I have a list of 3000 subscribers and have been trickling it out at only 300/hour out of paranoia. Also, do you think the presence of a localhost IP (from 127.0.0.1 for [ISP IP] etc.) and the coldfusion name as the sending program in the header is a big spam filter problem? One thing that annoys me is I send via my local IPS account with a reply-to for our domain, and a certain percentage of hosts rejects that as no relaying. That's not relaying. Or maybe it's the localhost IP in there, not sure. [Todays Threads] [This Message] [Subscription] [Fast Unsubscribe] [User Settings] [Donations and Support]
Re: Mailing to email list using CF
Hi Stylo, So doing a DNS check will not help much in verifying an account I just checked my list via dns using worldcast and about 10% failed, but many with messages about not having an IP they can reverse look-up, so useless. Yes, one thing that gets checked by most servers now is to make sure the mail coming in is from another mail server, which should have a reverse lookup of its IP address. If there is a reverse lookup response then it is often checked against lists of dial-up lines, etc., to again confirm that it another mail server, not a consumer machine. The re4verse name of the mail server should not have to match the email's sending domain name but it should match the name the mail server itself identified itself as. That should stop email coming in from home machines and other suspect sources. Kym, very helpful. Hard to know what emails are getting rejected for exactly what, or just tossed, so what do estimate the hourly threshhold is for big guys like AOL, Hotmail, etc? I have a list of 3000 subscribers and have been trickling it out at only 300/hour out of paranoia. Send it full-blast to everyone except the big three and pick up on who bounces what and then modify the feed rate to match. That means you need two streams to send but that should not be too hard. For the big three the rules are dirfferent and constanlty change as they try to keep ahead of thr SPAMmers but they do publish their restrictions (go to places like: http://postmaster.info.aol.com/ to find out) Also, do you think the presence of a localhost IP (from 127.0.0.1 for [ISP IP] etc.) and the coldfusion name as the sending program in the header is a big spam filter problem? At one time the word Allaire would cause an email to be stopped by some email servers as they were fussy about engines that could send bulk quantities of email but were not email servers themselves but that should no longer be the case. As to things like private IP addresses in the headers it should be OK in the first line or two as the mail gets out of a system into the larger world but records with private IP addresses after that it could be treated as suspicious. One thing that annoys me is I send via my local IPS account with a reply-to for our domain, and a certain percentage of hosts rejects that as no relaying. That's not relaying. Or maybe it's the localhost IP in there, not sure. What some servers look for is a match between the source domains in the headers and they should not do so as having a ReplyTo: that goes to a different place than the sender is not an improper thing. It is often the result of poor configuration but getting it fixed is not easy :-) -- Yours, Kym [Todays Threads] [This Message] [Subscription] [Fast Unsubscribe] [User Settings] [Donations and Support]
Re: Mailing to email list using CF
Unfortunately, good/best programming practices don't really have anything to do with this.If they did, as you say, a regex would solve the problem. For starters, what happens when users have the ability to change addresses, or import new ones in large blocks, via means out of your programmatic control?If you can't control the inputs... Besides, users -- especially non-technical middle and senior managers -- won't often allow themselves to be brought under control.Try, for example, to tell a senior executive vice-poobah who has had a membership list for X years -- and has been blissfully ignorant of all the bounces since they failed silently prior to CF 6 -- that they have to qualify it all over again with an opt-in.They'll be laughing all the way to the balcony, and more so as you sail off of it. The bottom line, imho, is that a robust, properly written cfmail application ignores whether the inputs have been tested or not.It tests for failures prior to the xmit, and gracefully catches those failures so the job can continue.I learned this one the hard way, personally. Further, if your circumstances require it, you may have to spell out in big letters exactly what is wrong with the address so the drooling slackjaws who let the list get into this shape in the first place can go fix it.Sadly, thats a business requirement that has bupkis to do with efficient programming. -- --Matt Robertson-- MSB Designs, Inc. mysecretbase.com [Todays Threads] [This Message] [Subscription] [Fast Unsubscribe] [User Settings] [Donations and Support]
Re: Mailing to email list using CF
One last thing.If you see annoyance in my last post, its not directed at anyone here.I just have had more of these political struggles with large clients than I would care to.Real frustrating stuff, but they pay the bills so, in the end, if they insist they get their way. --Matt-- [Todays Threads] [This Message] [Subscription] [Fast Unsubscribe] [User Settings] [Donations and Support]
Re: Mailing to email list using CF
that they have to qualify it all over again with an opt-in.They'll be laughing all the way to the balcony, and more so as you sail off of it. I think you've misread something when replying to me. I didn't say - nor did anyone I can see - to re-qualify a list with an opt-in. Personally I'd check format once, then use it, then delete emails after 3 bounces. If a website based list, though, I'd certainly start using a double opt-in for any new emails added. [Todays Threads] [This Message] [Subscription] [Fast Unsubscribe] [User Settings] [Donations and Support]
Re: Mailing to email list using CF
Paul Vernon wrote: Surely making sure your addresses are clean and correct is a good thing??? Its paramount, I think.I have a user who has a roughly 7000-address mailing list.They get probably 500-750 bounces per mailing (with bad addresses kept for political reasons) and the resulting backwash amounts to a denial of service attack on my mail server. While the validation thing I posted -- which I know is not efficient but is designed to be used by manager-types and not developers -- will catch bad formats it doesn't do squat for correctly-formatted but nonexistent addresses, which is what DDOS's me. Looking at your tag, it might be the component I need to solve this. I'll contact you off-list. --Matt Robertson-- MSB Designs, Inc. mysecretbase.com [Todays Threads] [This Message] [Subscription] [Fast Unsubscribe] [User Settings] [Donations and Support]
Re: Mailing to email list using CF
At send time, it's just to kick out the bad ones. I don't get it. If you check the addresses first, why check again? If an existing list, just check the format once. There is a udf on cflib.org for that. Done. Then use as normal. If an ongoing list, ALWAYS send an email to it that must be confirmed (link in it clicked) in order to be added to the list. Never just add an address. Even if correctly formatted it might be wrong. Show where you sent it so the customer can see if a spelling error: Thank you. We have sent an email to you at [EMAIL PROTECTED] Check your email. You must confirm it to be added to our list. Then you have zero bad addresses (and no real need to even check the format, though I do to point out errors first). The bad ones never arrive in their inbox and are thus never confirmed. [Todays Threads] [This Message] [Subscription] [Fast Unsubscribe] [User Settings] [Donations and Support]
Re: Mailing to email list using CF
I forgot to add, can't a dns check fail because rejected just like spam? Would AOL let you (and spammers) check via dns tons of AOL emails? Maybe technically unable to stop it as you disconnect as soon as connected, but maybe not. Anyone know? BTW, you could download worldcast for free and use their dns check on your list. It's a crappy little program but works. [Todays Threads] [This Message] [Subscription] [Fast Unsubscribe] [User Settings] [Donations and Support]
Re: Mailing to email list using CF
Hi Stylo, I forgot to add, can't a dns check fail because rejected just like spam? Would AOL let you (and spammers) check via dns tons of AOL emails? Maybe technically unable to stop it as you disconnect as soon as connected, but maybe not. Anyone know? I can give you an idea of what we do to try and keep the email-baddies at bay. DNS itself is harder to check and control than a mail server but the email address style that uses the domain name ([EMAIL PROTECTED]) actually is a wildcard DNS entry (*.domainname.com in the example above) pointing to the mail server and the mail server then works out what the account actually is. So doing a DNS check will not help much in verifying an account other than flagging that the domain name actually exists. On our mail servers we monitor the IP address of every incoming mail server or email client that talks to ours and check the frequency of connection. If we get a burst of many connections in a very short space of time we ban that IP address for an hour then turn it back on again. It doesn't affect legitimate connections and if someone is sending many dozen emails at once (very unusual) then they will eventually get sent in bursts of a few at a time depending on the thresholds used but we have found that illegitimate connections, such as a script kiddie scanning for valid POP accounts, will pick up the banned status and move on to the next mail server in their list and leave us alone. As a Hosting provider we have a very large number of domain names and related email accounts in our system which gives a potential attacked or SPAMmer a rich harvest so we need to keep things as tight as possible. Looking in the mail server's admin interface right now I see just over a hundred banned IP addresses and if we look up most of those they turn out to be at the other end of a DSL connection so are probably personal computers that have been attacked by a trojan horse and are sending SPAM/viruses without the owner knowledge. -- Yours, Kym [Todays Threads] [This Message] [Subscription] [Fast Unsubscribe] [User Settings] [Donations and Support]
RE: Mailing to email list using CF
What Kym does is very important to the users, and worthwhile, however a similar thing has bitten a client of mine.He runs seminars amongst university students and school students, who have a lot of hotmail accounts. He wants to send (at the users' request!) newsletters, but hotmail does something similar to what Kym does - too many emails in aburst and he gets banned. How can we send emails to several thousand of my client's users, a large number of whom are hotmail users? Cheers Mike Kear _ From: Kym Kovan [mailto:[EMAIL PROTECTED] Sent: Sunday, 27 June 2004 2:24 PM To: CF-Talk Subject: Re: Mailing to email list using CF [snip] On our mail servers we monitor the IP address of every incoming mail server or email client that talks to ours and check the frequency of connection. If we get a burst of many connections in a very short space of time we ban that IP address for an hour then turn it back on again. It doesn't affect legitimate connections and if someone is sending many dozen emails at once (very unusual) then they will eventually get sent in bursts of a few at a time depending on the thresholds used but we have found that illegitimate connections, such as a script kiddie scanning for valid POP accounts, will pick up the banned status and move on to the next mail server in their list and leave us alone. [snip] [Todays Threads] [This Message] [Subscription] [Fast Unsubscribe] [User Settings] [Donations and Support]
RE: Mailing to email list using CF
Hi Mike, What Kym does is very important to the users, and worthwhile, however a similar thing has bitten a client of mine.He runs seminars amongst university students and school students, who have a lot of hotmail accounts. He wants to send (at the users' request!) newsletters, but hotmail does something similar to what Kym does - too many emails in aburst and he gets banned. How can we send emails to several thousand of my client's users, a large number of whom are hotmail users? Two things: send selectively and get onto White Lists. For the first option you need to separate out the email addresses by domain name and for hotmail, AOL and MSN, plus others possibly, and send to them in batches rather than a mass in one go. This again can be split up, most email servers will reject too many recipients in one email, AOL is very hot on this, but only some will do the rate-measure technique that I described which is by the number of connections rather than the number of recipients in the one post. The second option is good if you are a legitimate sender of emails. Most of the big organisations have a list of trusted email servers that they will accept emails from without restriction. Getting on that list is not easy, but possible if you are legit. -- Yours, Kym [Todays Threads] [This Message] [Subscription] [Fast Unsubscribe] [User Settings] [Donations and Support]
Mailing to email list using CF
I'm storing a list of users who have signed up for an email list in a database.I have a simple web app for users to use when they want to send something to their email list.When they click send, I'm using cfmail query=blah to send the mail out to all of the users individually.The problem I've run into is that CF has choked on invalid email addresses (legacy ones that have been brought to me from other databases) and when that happens, CF produces an error and quits. I thought about doing cfoutput query=blahcfmail to=#email#.../cfoutput but I'm fairly sure that takes some serious performance hits or else there wouldn't be the query attribute to CFMAIL.Is there any way to tell CFMAIL to keep processing and just skip the invalid address?I'm guessing that a cftry/cfcatch block will only catch the error and move on but will not tell CF to finish the mailing. Any ideas? thoughts? Rants? etc? John [Todays Threads] [This Message] [Subscription] [Fast Unsubscribe] [User Settings] [Donations and Support]
RE: Mailing to email list using CF
Doing CFOUTPUT/CFMAIL is a huge performance hit, so you don't want to do it that way.We manage mailing lists of pretty significant size, and we use a combination of fairly aggressive validation at insertion time, and at send time, we error check the list with a simple RE that every address must validate against (the same RE is used as the final step of insertion validation).Here's some example code: cfquery ... Name=get SELECT name, email FROM recipients /cfquery cfset emails = arrayNew(1) / cfloop query=get cfif Refind(email_re, email) EQ 1 cfset arrayAppend(emails, email) / cfelse cfset arrayAppend(emails, ) / /cfif /cfloop cfset queryAddColumn(get, email2, emails) / cfquery dbtype=query name=get SELECT * FROM get WHERE email2 '' /cfquery cfmail query=get /cfmail -Original Message- From: Burns, John D [mailto:[EMAIL PROTECTED] Sent: Friday, June 25, 2004 11:29 AM To: CF-Talk Subject: Mailing to email list using CF I'm storing a list of users who have signed up for an email list in a database.I have a simple web app for users to use when they want to send something to their email list.When they click send, I'm using cfmail query=blah to send the mail out to all of the users individually.The problem I've run into is that CF has choked on invalid email addresses (legacy ones that have been brought to me from other databases) and when that happens, CF produces an error and quits. I thought about doing cfoutput query=blahcfmail to=#email#.../cfoutput but I'm fairly sure that takes some serious performance hits or else there wouldn't be the query attribute to CFMAIL.Is there any way to tell CFMAIL to keep processing and just skip the invalid address?I'm guessing that a cftry/cfcatch block will only catch the error and move on but will not tell CF to finish the mailing. Any ideas? thoughts? Rants? etc? John [Todays Threads] [This Message] [Subscription] [Fast Unsubscribe] [User Settings] [Donations and Support]
Re: Mailing to email list using CF
This one has come up before, you can bet. Short answer:If you have not validated your addresses you cannot use the query= attribute safely. If you wrap cfmail in a try/catch block and just let it continue on if it hits a bad address, then you'll never scrub your db.What I do looks like what you see below. includes/inc_verify_addresses.cfm runs a 40-odd-step address test. Far less efficient than a regex but it creates a list of messages that describe everything wrong with the address.This is an html-formatted unordered list and is dumped into variables.failurelist. Then the mail xmit is attempted.If an error was found in the format checker, the error is logged into a special db table dedicated to collecting system errors (one error per record, the errmsg and bad address goes into a long text field). If the format checker missed something, then the try/catch block bats cleanup and handles it.Here again the error is logged in the same fashion and the process is allowed to continue. When done, the list of errors is collected and can be emailed automatically to some responsible party.Since this could be a Hyoooge email its not a bad idea to create some other method of viewing the error log if you're dealing with any kind of decent-sized list. HtH, -- --Matt Robertson-- MSB Designs, Inc. mysecretbase.com cfloop query=MailRun cfset variables.InclEmailAddr=MailRun.EmailAddr cfinclude template=includes/inc_verify_addresses.cfm cfif not Len(variables.FailureList) cftry cfmail to=#MailRun.Email# from=#my.Email# subject=#my.Title# server=#my.lServer# type=HTML ...blah blah blah... /cfmail cfcatch type=Any ...log the error... /cfcatch /cftry cfelse ...log the error... /cfif /cfloop ...report the error to the email inbox of your choice. [Todays Threads] [This Message] [Subscription] [Fast Unsubscribe] [User Settings] [Donations and Support]
RE: Mailing to email list using CF
So looping over the thousands of email addresses and running a regex against it will not cause a large performance hit?I considered doing that on output to catch the ones that I didn't handle coming in. Everything that comes through any of my sites goes through a cfc method that scrubs the data and runs an re on it to ensure it's at least in the valid format. John -Original Message- From: Barney Boisvert [mailto:[EMAIL PROTECTED] Sent: Friday, June 25, 2004 2:43 PM To: CF-Talk Subject: RE: Mailing to email list using CF Doing CFOUTPUT/CFMAIL is a huge performance hit, so you don't want to do it that way.We manage mailing lists of pretty significant size, and we use a combination of fairly aggressive validation at insertion time, and at send time, we error check the list with a simple RE that every address must validate against (the same RE is used as the final step of insertion validation).Here's some example code: cfquery ... Name=get SELECT name, email FROM recipients /cfquery cfset emails = arrayNew(1) / cfloop query=get cfif Refind(email_re, email) EQ 1 cfset arrayAppend(emails, email) / cfelse cfset arrayAppend(emails, ) / /cfif /cfloop cfset queryAddColumn(get, email2, emails) / cfquery dbtype=query name=get SELECT * FROM get WHERE email2 '' /cfquery cfmail query=get /cfmail -Original Message- From: Burns, John D [mailto:[EMAIL PROTECTED] Sent: Friday, June 25, 2004 11:29 AM To: CF-Talk Subject: Mailing to email list using CF I'm storing a list of users who have signed up for an email list in a database.I have a simple web app for users to use when they want to send something to their email list.When they click send, I'm using cfmail query=blah to send the mail out to all of the users individually.The problem I've run into is that CF has choked on invalid email addresses (legacy ones that have been brought to me from other databases) and when that happens, CF produces an error and quits. I thought about doing cfoutput query=blahcfmail to=#email#.../cfoutput but I'm fairly sure that takes some serious performance hits or else there wouldn't be the query attribute to CFMAIL.Is there any way to tell CFMAIL to keep processing and just skip the invalid address?I'm guessing that a cftry/cfcatch block will only catch the error and move on but will not tell CF to finish the mailing. Any ideas? thoughts? Rants? etc? John [Todays Threads] [This Message] [Subscription] [Fast Unsubscribe] [User Settings] [Donations and Support]
RE: Mailing to email list using CF
John, I've recently released a CFX tag I wrote some months back to help me scrub my e-mail address data because CFMX is much better at spotting crappy addresses and stopping before you send so I had to really clean up the data going in but that still left me with thousands of addresses already in the DB that needed to be checked http://www.web-architect.co.uk/cfxtags/cfx_tag.cfm?ProductID=12 It should be able to validate about 1000 addresses a second :) Paul [Todays Threads] [This Message] [Subscription] [Fast Unsubscribe] [User Settings] [Donations and Support]
RE: Mailing to email list using CF
I didn't say that.But in my experience, it's a lot faster than looping the CFMAIL tag itself.We also validate very heavily before inserting any email addresses (including the same RE test we use at send time).There the focus is on identifying errors and giving meaningful messages.At send time, it's just to kick out the bad ones.It seems foolish from the outside, but the nature of the communication we do has zero room for errors, so we pay the price of revalidation, just to be safe.We've discovered a couple chinks in the input validation armor (amazingly subtle ones) that we were saved from by the send-time checking, so it's worth it. Cheers, barneyb -Original Message- From: Burns, John D [mailto:[EMAIL PROTECTED] Sent: Friday, June 25, 2004 12:42 PM To: CF-Talk Subject: RE: Mailing to email list using CF So looping over the thousands of email addresses and running a regex against it will not cause a large performance hit?I considered doing that on output to catch the ones that I didn't handle coming in. Everything that comes through any of my sites goes through a cfc method that scrubs the data and runs an re on it to ensure it's at least in the valid format. John -Original Message- From: Barney Boisvert [mailto:[EMAIL PROTECTED] Sent: Friday, June 25, 2004 2:43 PM To: CF-Talk Subject: RE: Mailing to email list using CF Doing CFOUTPUT/CFMAIL is a huge performance hit, so you don't want to do it that way.We manage mailing lists of pretty significant size, and we use a combination of fairly aggressive validation at insertion time, and at send time, we error check the list with a simple RE that every address must validate against (the same RE is used as the final step of insertion validation).Here's some example code: cfquery ... Name=get SELECT name, email FROM recipients /cfquery cfset emails = arrayNew(1) / cfloop query=get cfif Refind(email_re, email) EQ 1 cfset arrayAppend(emails, email) / cfelse cfset arrayAppend(emails, ) / /cfif /cfloop cfset queryAddColumn(get, email2, emails) / cfquery dbtype=query name=get SELECT * FROM get WHERE email2 '' /cfquery cfmail query=get /cfmail -Original Message- From: Burns, John D [mailto:[EMAIL PROTECTED] Sent: Friday, June 25, 2004 11:29 AM To: CF-Talk Subject: Mailing to email list using CF I'm storing a list of users who have signed up for an email list in a database.I have a simple web app for users to use when they want to send something to their email list.When they click send, I'm using cfmail query=blah to send the mail out to all of the users individually.The problem I've run into is that CF has choked on invalid email addresses (legacy ones that have been brought to me from other databases) and when that happens, CF produces an error and quits. I thought about doing cfoutput query=blahcfmail to=#email#.../cfoutput but I'm fairly sure that takes some serious performance hits or else there wouldn't be the query attribute to CFMAIL.Is there any way to tell CFMAIL to keep processing and just skip the invalid address?I'm guessing that a cftry/cfcatch block will only catch the error and move on but will not tell CF to finish the mailing. Any ideas? thoughts? Rants? etc? John [Todays Threads] [This Message] [Subscription] [Fast Unsubscribe] [User Settings] [Donations and Support]
RE: Mailing to email list using CF
It is my understanding it's better to do your own loop for just this reason.If you use the query option in cfmail you have to make sure all your address are clean and correct. cfloop query = blah cftry cfmail . message /cfmail cfcatch ... Do something when bad e-mails are generated. /cfcatch /cftry /cfloop -- Ian Skinner Web Programmer BloodSource file:///C:\Documents%20and%20Settings\iskinner\Application%20Data\Micro soft\Signatures\www.BloodSource.org www.BloodSource.org http://www.BloodSource.orgSacramento Sacramento, CA C code. C code run. Run code run. Please! - Cynthia Dunning Confidentiality Notice:This message including any attachments is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender and delete any copies of this message. [Todays Threads] [This Message] [Subscription] [Fast Unsubscribe] [User Settings] [Donations and Support]
RE: Mailing to email list using CF
It is my understanding it's better to do your own loop for just this reason.If you use the query option in cfmail you have to make sure all your address are clean and correct. Surely making sure your addresses are clean and correct is a good thing??? Paul [Todays Threads] [This Message] [Subscription] [Fast Unsubscribe] [User Settings] [Donations and Support]
