Re: mass emailing from CF
Why would you need to create all those cfschedules? Just have one schedule run and check for unsent mail. When your recordcount is down to zero, just delete the cfschedule. Or you can just let it run as there is minimal impact on checking a page that contains a query that doesn't return any records. >... > As for deciding how many cfscchedules to make, that would be some > pretty > basic math. > > Something like this should do the trick... > > > > > > > > > > Without testing... that SHOULD spit back 11 > . > or just add the stragglers to the last batch. > ~| Introducing the Fusion Authority Quarterly Update. 80 pages of hard-hitting, up-to-date ColdFusion information by your peers, delivered to your door four times a year. http://www.fusionauthority.com/quarterly Archive: http://www.houseoffusion.com/groups/CF-Talk/message.cfm/messageid:252054 Subscription: http://www.houseoffusion.com/groups/CF-Talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4
RE: mass emailing from CF
Well, again thanks for all the help and tips guys and gal (and Will) ;-P I managed to cut the processing time of the send tool over half with some spring cleaning... or basically... rewriting it from the ground up to get rid of some seriously mentally challenged code ;-) Stored procedures helped TREMENDOUSLY over normal cfquery tags. I dont normally cache queries in administrative tools but the ones that didnt get converted to stored procs got cached for 1 minute as well... that also cut off a small amount from the overall time. Caching the actual content of the page didnt seem to change anything really. But now that the entire tool is in AJAX... it gives the IMPRESSION of being basically instant :-) I can stop email bombing my co-workers now :-) ..:.:.:.:.:.:.:.:.:.:.:.:. Bobby Hartsfield http://acoderslife.com -Original Message- From: Bobby Hartsfield [mailto:[EMAIL PROTECTED] Sent: Monday, September 04, 2006 6:27 PM To: CF-Talk Subject: RE: mass emailing from CF Well that definitely SOUNDS good and even makes sense since the spool directory stays busy sending email after the page request has come and gone... but that is really all that is going on with that page when it times out hopefully the simple cache solves it. It definitely speeds it up to me. Thank everyone for their input and help. ...:.:.:.:.:.:.:.:.:.:.:.:. Bobby Hartsfield http://acoderslife.com -Original Message- From: Jochem van Dieten [mailto:[EMAIL PROTECTED] Sent: Monday, September 04, 2006 1:23 PM To: CF-Talk Subject: Re: mass emailing from CF Bobby Hartsfield wrote: > > Just another side as to why I thought what I did about that Jochem, but if I > trigger the page locally... it takes about half a second to process. If I > remote into a computer off site and hit the page remotely (or have someone > else do it) it takes MUCH longer to process. > > When I did it locally, the page request was over like I said... about half a > second but the spool directory still had emails in it that gradually kept > disappearing as they sent. > > So if the there is that much of a difference just spooling the emails why > wouldn?t the user's speed matter and what is the cause of that huge > difference. I have no idea what causes the difference, but the speed of cfmail is not related to the speed of the client connection. Uploading a large file with e-mail addresses might make a difference, sending a few MB of data back to the client might make the difference, but it does not matter for cfmail. Jochem ~| Introducing the Fusion Authority Quarterly Update. 80 pages of hard-hitting, up-to-date ColdFusion information by your peers, delivered to your door four times a year. http://www.fusionauthority.com/quarterly Archive: http://www.houseoffusion.com/groups/CF-Talk/message.cfm/messageid:252008 Subscription: http://www.houseoffusion.com/groups/CF-Talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
RE: mass emailing from CF
I meant an elegant way of sorting a list of email addresses so that all mail to AOL (and att, yahoo, gmail) is evenly distributed throughout the batch - the reason I do this is not because of page timeouts.. but because when I send too many messages to AOL, Yahoo and ATT, they go into a black hole. By slowing them down, they go through much better. A scheduled task that take about 15ms every 5 minutes puts very little stress on the server. It is about 4 seconds a day and is insignificant compared to the number of pages sent a day. > > I would love to see an elegant way of doing that :) > >Check out Matt's solution to that (or modified solution to it) > >http://mysecretbase.com/Slowing_Down_CFMAIL.cfm > > > >I don't think I'd like to have a cfshedule that may or may not have anything >to do running every 5 minutes but thanks just the same :-) ~| Introducing the Fusion Authority Quarterly Update. 80 pages of hard-hitting, up-to-date ColdFusion information by your peers, delivered to your door four times a year. http://www.fusionauthority.com/quarterly Archive: http://www.houseoffusion.com/groups/CF-Talk/message.cfm/messageid:252005 Subscription: http://www.houseoffusion.com/groups/CF-Talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
RE: mass emailing from CF
Well that definitely SOUNDS good and even makes sense since the spool directory stays busy sending email after the page request has come and gone... but that is really all that is going on with that page when it times out hopefully the simple cache solves it. It definitely speeds it up to me. Thank everyone for their input and help. ..:.:.:.:.:.:.:.:.:.:.:.:. Bobby Hartsfield http://acoderslife.com -Original Message- From: Jochem van Dieten [mailto:[EMAIL PROTECTED] Sent: Monday, September 04, 2006 1:23 PM To: CF-Talk Subject: Re: mass emailing from CF Bobby Hartsfield wrote: > > Just another side as to why I thought what I did about that Jochem, but if I > trigger the page locally... it takes about half a second to process. If I > remote into a computer off site and hit the page remotely (or have someone > else do it) it takes MUCH longer to process. > > When I did it locally, the page request was over like I said... about half a > second but the spool directory still had emails in it that gradually kept > disappearing as they sent. > > So if the there is that much of a difference just spooling the emails why > wouldn?t the user's speed matter and what is the cause of that huge > difference. I have no idea what causes the difference, but the speed of cfmail is not related to the speed of the client connection. Uploading a large file with e-mail addresses might make a difference, sending a few MB of data back to the client might make the difference, but it does not matter for cfmail. Jochem ~| Introducing the Fusion Authority Quarterly Update. 80 pages of hard-hitting, up-to-date ColdFusion information by your peers, delivered to your door four times a year. http://www.fusionauthority.com/quarterly Archive: http://www.houseoffusion.com/groups/CF-Talk/message.cfm/messageid:251994 Subscription: http://www.houseoffusion.com/groups/CF-Talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Re: mass emailing from CF
Bobby Hartsfield wrote: > > Just another side as to why I thought what I did about that Jochem, but if I > trigger the page locally... it takes about half a second to process. If I > remote into a computer off site and hit the page remotely (or have someone > else do it) it takes MUCH longer to process. > > When I did it locally, the page request was over like I said... about half a > second but the spool directory still had emails in it that gradually kept > disappearing as they sent. > > So if the there is that much of a difference just spooling the emails why > wouldn?t the user's speed matter and what is the cause of that huge > difference. I have no idea what causes the difference, but the speed of cfmail is not related to the speed of the client connection. Uploading a large file with e-mail addresses might make a difference, sending a few MB of data back to the client might make the difference, but it does not matter for cfmail. Jochem ~| Introducing the Fusion Authority Quarterly Update. 80 pages of hard-hitting, up-to-date ColdFusion information by your peers, delivered to your door four times a year. http://www.fusionauthority.com/quarterly Archive: http://www.houseoffusion.com/groups/CF-Talk/message.cfm/messageid:251984 Subscription: http://www.houseoffusion.com/groups/CF-Talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Re: mass emailing from CF
> You don't need to split up the emails in batches that way. I think it really depends what else is going on programmatically and what is being used as far as software / hardware configuration and bandwidth issues. Some systems may require the batch setup others may not. ~| Introducing the Fusion Authority Quarterly Update. 80 pages of hard-hitting, up-to-date ColdFusion information by your peers, delivered to your door four times a year. http://www.fusionauthority.com/quarterly Archive: http://www.houseoffusion.com/groups/CF-Talk/message.cfm/messageid:251955 Subscription: http://www.houseoffusion.com/groups/CF-Talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4
RE: mass emailing from CF
Yeah, I've never had to deal with much bulk emailing before though (apparently) :-) ..:.:.:.:.:.:.:.:.:.:.:.:. Bobby Hartsfield http://acoderslife.com -Original Message- From: Will Tomlinson [mailto:[EMAIL PROTECTED] Sent: Sunday, September 03, 2006 9:56 PM To: CF-Talk Subject: Re: mass emailing from CF Jen, I checked out some of your cart. Shoot me an email girl! Bobby, I think this is the FIRST time I've ever seen you ask a question! lol! I use Matt's trickler and it's worked sweet for me. Will ~| Introducing the Fusion Authority Quarterly Update. 80 pages of hard-hitting, up-to-date ColdFusion information by your peers, delivered to your door four times a year. http://www.fusionauthority.com/quarterly Archive: http://www.houseoffusion.com/groups/CF-Talk/message.cfm/messageid:251931 Subscription: http://www.houseoffusion.com/groups/CF-Talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
RE: mass emailing from CF
> I would love to see an elegant way of doing that :) Check out Matt's solution to that (or modified solution to it) http://mysecretbase.com/Slowing_Down_CFMAIL.cfm I dont think I'd like to have a cfshedule that may or may not have anything to do running every 5 minutes but thanks just the same :-) ..:.:.:.:.:.:.:.:.:.:.:.:. Bobby Hartsfield http://acoderslife.com -- No virus found in this incoming message. Checked by AVG Free Edition. Version: 7.1.405 / Virus Database: 268.11.7/436 - Release Date: 9/1/2006 -- No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.1.405 / Virus Database: 268.11.7/436 - Release Date: 9/1/2006 ~| Introducing the Fusion Authority Quarterly Update. 80 pages of hard-hitting, up-to-date ColdFusion information by your peers, delivered to your door four times a year. http://www.fusionauthority.com/quarterly Archive: http://www.houseoffusion.com/groups/CF-Talk/message.cfm/messageid:251930 Subscription: http://www.houseoffusion.com/groups/CF-Talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Re: mass emailing from CF
Jen, I checked out some of your cart. Shoot me an email girl! Bobby, I think this is the FIRST time I've ever seen you ask a question! lol! I use Matt's trickler and it's worked sweet for me. Will ~| Introducing the Fusion Authority Quarterly Update. 80 pages of hard-hitting, up-to-date ColdFusion information by your peers, delivered to your door four times a year. http://www.fusionauthority.com/quarterly Archive: http://www.houseoffusion.com/groups/CF-Talk/message.cfm/messageid:251929 Subscription: http://www.houseoffusion.com/groups/CF-Talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4
RE: mass emailing from CF
I do it with a scheduled task that runs every 5 minutes around the clock. It checks for newsletters that have to be sent and if there are some to be sent, it sends the next 500 and marks those as sent. works nicely so far... I have an status page that refreshes itself every few minutes and shows when a job started, how many were in the entire job, how many were sent and estimated completion time. This way, if it appears that it is going to take too long, we can change the number sent per batch. That status page can be restarted at any time to see check the progress. Here's the twist: I have embarrassingly ugly code to spread mail to AOL, ATT Yahoo, etc.. evenly over each batch and to interleave them if possible, to slow down delivery to these few big providers. I would love to see an elegant way of doing that:) Al ~| Introducing the Fusion Authority Quarterly Update. 80 pages of hard-hitting, up-to-date ColdFusion information by your peers, delivered to your door four times a year. http://www.fusionauthority.com/quarterly Archive: http://www.houseoffusion.com/groups/CF-Talk/message.cfm/messageid:251928 Subscription: http://www.houseoffusion.com/groups/CF-Talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
RE: mass emailing from CF
You don't need to split up the emails in batches that way. By spooling them all to disk then dealing with the spool is a lot easier. -Original Message- From: Bobby Hartsfield [mailto:[EMAIL PROTECTED] Sent: 03 September 2006 19:00 To: CF-Talk Subject: RE: mass emailing from CF You're right, it would create more possible points of failure but the only failure so far is timeouts from a remote user. It's way too fast locally to timeout so the chances of failure would really be less from what I can tell. As for deciding how many cfscchedules to make, that would be some pretty basic math. Something like this should do the trick... Without testing... that SHOULD spit back 11 .or just add the stragglers to the last batch. ...:.:.:.:.:.:.:.:.:.:.:.:. Bobby Hartsfield http://acoderslife.com -- No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.1.405 / Virus Database: 268.11.7/436 - Release Date: 01/09/2006 ~| Introducing the Fusion Authority Quarterly Update. 80 pages of hard-hitting, up-to-date ColdFusion information by your peers, delivered to your door four times a year. http://www.fusionauthority.com/quarterly Archive: http://www.houseoffusion.com/groups/CF-Talk/message.cfm/messageid:251923 Subscription: http://www.houseoffusion.com/groups/CF-Talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
RE: mass emailing from CF
Ahh ok, gotcha. That should be pretty easy replicate. > Mail is send from the server, not from the client, so the speed of the client connection is irrelevant. Just another side as to why I thought what I did about that Jochem, but if I trigger the page locally... it takes about half a second to process. If I remote into a computer off site and hit the page remotely (or have someone else do it) it takes MUCH longer to process. When I did it locally, the page request was over like I said... about half a second but the spool directory still had emails in it that gradually kept disappearing as they sent. So if the there is that much of a difference just spooling the emails why wouldnt the user's speed matter and what is the cause of that huge difference. Normal pages with no excessive processing are pretty quick locally and remotely. That might sound like an argument but it's not. Take it as a request for clarification and reasoning not a debate invitation, because quite frankly, I dont know... I'm just assuming :-) ..:.:.:.:.:.:.:.:.:.:.:.:. Bobby Hartsfield http://acoderslife.com -Original Message- From: D F [mailto:[EMAIL PROTECTED] Sent: Sunday, September 03, 2006 2:06 PM To: CF-Talk Subject: Re: mass emailing from CF >That would be a nifty trick to keep batches from overlapping... got any more >details on that? I created two templates to get things started. cfmailer_setup.cfm mailer.cfm cfmailer.cfm - basically sets the batches up. To keep the batches from overlapping...what I did was set a server variable which consisted of an array with blank strings. Something like ... then it would set the batches up as scheduled tasks... These tasks would then call the mailer.cfm page, passing in appropriate url variables. mailer.cfm - would check to see if any array spaces were open / blank and if so would write its scheduled task id into that free space and then continue on to process the batch. Obviously then if there are no free spaces then a call needs to be made to cfmailer_setup.cfm to reschedule the remaining batches. ~| Introducing the Fusion Authority Quarterly Update. 80 pages of hard-hitting, up-to-date ColdFusion information by your peers, delivered to your door four times a year. http://www.fusionauthority.com/quarterly Archive: http://www.houseoffusion.com/groups/CF-Talk/message.cfm/messageid:251918 Subscription: http://www.houseoffusion.com/groups/CF-Talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Re: mass emailing from CF
>That would be a nifty trick to keep batches from overlapping... got any more >details on that? I created two templates to get things started. cfmailer_setup.cfm mailer.cfm cfmailer.cfm - basically sets the batches up. To keep the batches from overlapping...what I did was set a server variable which consisted of an array with blank strings. Something like ... then it would set the batches up as scheduled tasks... These tasks would then call the mailer.cfm page, passing in appropriate url variables. mailer.cfm - would check to see if any array spaces were open / blank and if so would write its scheduled task id into that free space and then continue on to process the batch. Obviously then if there are no free spaces then a call needs to be made to cfmailer_setup.cfm to reschedule the remaining batches. ~| Introducing the Fusion Authority Quarterly Update. 80 pages of hard-hitting, up-to-date ColdFusion information by your peers, delivered to your door four times a year. http://www.fusionauthority.com/quarterly Archive: http://www.houseoffusion.com/groups/CF-Talk/message.cfm/messageid:251917 Subscription: http://www.houseoffusion.com/groups/CF-Talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
RE: mass emailing from CF
I was under the impression that what CF is doing when it times out is spooling the emails. Once that is done, the user is done and can have their browser back. CF Continues to work on the spooled emails and sending them. Is that wrong? ..:.:.:.:.:.:.:.:.:.:.:.:. Bobby Hartsfield http://acoderslife.com -Original Message- From: Jochem van Dieten [mailto:[EMAIL PROTECTED] Sent: Sunday, September 03, 2006 1:58 PM To: CF-Talk Subject: Re: mass emailing from CF Bobby Hartsfield wrote: > That?s the thing... it doesn?t really take that long as far as I can tell > but I'm also on a pretty fast connection. I know some slow pokes that can > help me test though. Mail is send from the server, not from the client, so the speed of the client connection is irrelevant. Jochem ~| Introducing the Fusion Authority Quarterly Update. 80 pages of hard-hitting, up-to-date ColdFusion information by your peers, delivered to your door four times a year. http://www.fusionauthority.com/quarterly Archive: http://www.houseoffusion.com/groups/CF-Talk/message.cfm/messageid:251916 Subscription: http://www.houseoffusion.com/groups/CF-Talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4
Re: mass emailing from CF
Bobby Hartsfield wrote: > That?s the thing... it doesn?t really take that long as far as I can tell > but I'm also on a pretty fast connection. I know some slow pokes that can > help me test though. Mail is send from the server, not from the client, so the speed of the client connection is irrelevant. Jochem ~| Introducing the Fusion Authority Quarterly Update. 80 pages of hard-hitting, up-to-date ColdFusion information by your peers, delivered to your door four times a year. http://www.fusionauthority.com/quarterly Archive: http://www.houseoffusion.com/groups/CF-Talk/message.cfm/messageid:251915 Subscription: http://www.houseoffusion.com/groups/CF-Talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4
RE: mass emailing from CF
You're right, it would create more possible points of failure but the only failure so far is timeouts from a remote user. It's way too fast locally to timeout so the chances of failure would really be less from what I can tell. As for deciding how many cfscchedules to make, that would be some pretty basic math. Something like this should do the trick... Without testing... that SHOULD spit back 11 or just add the stragglers to the last batch. ..:.:.:.:.:.:.:.:.:.:.:.:. Bobby Hartsfield http://acoderslife.com -Original Message- From: Matt Robertson [mailto:[EMAIL PROTECTED] Sent: Sunday, September 03, 2006 1:17 PM To: CF-Talk Subject: Re: mass emailing from CF On 9/3/06, D F <[EMAIL PROTECTED]> wrote: > The logic seems fine to me. If you are sending 10k emails then you are hoping that 10 processes run properly, rather than one. I see that as a tenfold increase in potential points of failure. Sure you can add a 'Stage X complete' email at the end of each send process but that doesn't address the problem, or help solve it... only create the ability to spot a problem with your solution. And what happens when you need to send 11,000? Or 10,001? You need to create a scheduled 11th process. And then a 12th. That means you either have to babysit your maillist count so you can manually add another task or write in code that divides the total list by 10 and run only group 1 of 10, 2 of 10 etc. Which is also not scaleable since you will, sooner or later, exceed whatever your max batch size is (and leading up to that you will watch performance decrease since you haven't addressed the root problem) and have to create Schedule Item 11, then 12 and so on. Just some stuff that came immediately to mind. If you have mail capacity o connection issues I recommend ou address them directly. CF shouldn't be the problem. -- [EMAIL PROTECTED] Janitor, MSB Web Systems mysecretbase.com ~| Introducing the Fusion Authority Quarterly Update. 80 pages of hard-hitting, up-to-date ColdFusion information by your peers, delivered to your door four times a year. http://www.fusionauthority.com/quarterly Archive: http://www.houseoffusion.com/groups/CF-Talk/message.cfm/messageid:251913 Subscription: http://www.houseoffusion.com/groups/CF-Talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
RE: mass emailing from CF
That would be a nifty trick to keep batches from overlapping... got any more details on that? ..:.:.:.:.:.:.:.:.:.:.:.:. Bobby Hartsfield http://acoderslife.com -Original Message- From: D F [mailto:[EMAIL PROTECTED] Sent: Sunday, September 03, 2006 1:00 PM To: CF-Talk Subject: Re: mass emailing from CF Hi Bobby, Another thing we've done is to set server variables to limit the amount of batches that are concurrently running. So if there are more than say 3, it prevents the next batch from executing and reschedules the remaining batches. As well after monitoring performance we observed that batches took longer during peak hours ( as to be expected ) so we coded our system to adapt to the time of day etc. ~| Introducing the Fusion Authority Quarterly Update. 80 pages of hard-hitting, up-to-date ColdFusion information by your peers, delivered to your door four times a year. http://www.fusionauthority.com/quarterly Archive: http://www.houseoffusion.com/groups/CF-Talk/message.cfm/messageid:251912 Subscription: http://www.houseoffusion.com/groups/CF-Talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
RE: mass emailing from CF
I will be caching it but it's not being cached now. (I didnt write it) > This shouldn't take long. Thats the thing... it doesnt really take that long as far as I can tell but I'm also on a pretty fast connection. I know some slow pokes that can help me test though. ..:.:.:.:.:.:.:.:.:.:.:.:. Bobby Hartsfield http://acoderslife.com -Original Message- From: Jochem van Dieten [mailto:[EMAIL PROTECTED] Sent: Sunday, September 03, 2006 12:54 PM To: CF-Talk Subject: Re: mass emailing from CF Bobby Hartsfield wrote: > > I'm just curious how most people are handling sending out massive amounts of > in short periods of time. Right now it is basically... > > 1) Set the timeout to a ridiculously high number > 2) Select all the > 3) Loop over all users and cfmail the newsletter > 4) when people complain about timeouts, increase ridiculously high timeout > number This shouldn't take long. All CF has to do is generate the content of the e-mail (it is the same for every email and you are caching it, right?) and write it out to a file in the spool directory. I think you would be much better of investigating why this takes long then trying to work around it. Jochem ~| Introducing the Fusion Authority Quarterly Update. 80 pages of hard-hitting, up-to-date ColdFusion information by your peers, delivered to your door four times a year. http://www.fusionauthority.com/quarterly Archive: http://www.houseoffusion.com/groups/CF-Talk/message.cfm/messageid:251911 Subscription: http://www.houseoffusion.com/groups/CF-Talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
RE: mass emailing from CF
I'm afraid I dont have control over the mail server but I could talk to the guy who does and see what he thinks. The person who coded this originally was also over the mail server (and web servers) so maybe the new guy has some ideas he can share in that department After reading your post however, I think you missed one part of my logic. I wasnt planning on grabbing the top x, sending, flagging, then starting over. After sending the TOP X number of emails... that process is basically over and done with. A minute or 2 later, the same process will happen again and grab the next TOP X emails. That will most definitely be lighter on everything and everyone (except me in the code department heh) unless I'm missing something else. O'Hare must have an 8 hour minimum layover rule lol I had to weight darn near 8 hours last time I was there. I'll try and keep you occupied for a bit longer... no, no it's the least I can do to come up with questions for you. Think nothing of it ;-p ..:.:.:.:.:.:.:.:.:.:.:.:. Bobby Hartsfield http://acoderslife.com -Original Message- From: Matt Robertson [mailto:[EMAIL PROTECTED] Sent: Sunday, September 03, 2006 12:56 PM To: CF-Talk Subject: Re: mass emailing from CF I think you are going to have trouble with your Top X Items idea. I try to shy away from batching mail in chunks like that. Its just not scaleable. Either that or it has a fatal flaw hiding inside that is just as bad as whatever your curent problem is. In this case I think you have both. Consider the fact that CF will happily Select Top X, shoot out X emails, mark X db records and loop around and do it again. Sounds great. Except CF will do it at speeds that no doubt FAR exceed the capacity of your presently-hobbled system. I'd bet you aren't going to accomplish a thing with it as-described. So to solve that poblem you say to youself "I will slow it down programmatically". The olde tyme way was with a CFLoop inside your mail loop set to run for X iterations where running those iterations where those iterations take up some real time. This has the side effect of eating up processor cycles like a Tyrannosaur crashing the Brontosaurus Family Reunion and also tying up a thread while the brontosaurs are mowed down. To which you can say "Aha... I have CFMX Version [whatever] so I can call a java sleep method. This is great news to the brontosaurs' social calendar but still sucks down a thread for ever and ever. .Yes I know I am blabbing but I am in the middle of an 8 hr layover at O'Hare Int'l Airport and am bored... Why not look at your mail server? Forget about CF for the moment. Seems like the mail server is the problem. Can you install, for example, iMS? There is a cheap version, and a free one too. Run it off a nonstandard port if you have port 25 already taken. -- [EMAIL PROTECTED] Janitor, MSB Web Systems mysecretbase.com ~| Introducing the Fusion Authority Quarterly Update. 80 pages of hard-hitting, up-to-date ColdFusion information by your peers, delivered to your door four times a year. http://www.fusionauthority.com/quarterly Archive: http://www.houseoffusion.com/groups/CF-Talk/message.cfm/messageid:251910 Subscription: http://www.houseoffusion.com/groups/CF-Talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
RE: mass emailing from CF
I send out about 6k at a time. My work around, as I'm on windows and the mail server isn't as fast as I'd like, is to spool all the messages to disk, then have a scheduled task run once an hour to make sure the spool is cleared. My bulk emails are all triggered by the scheduler. Perhaps you could give the user the option, or give no choice, and have the mail run set to run from the scheduler instead of expecting it to run there and then. Even if it was just 5 mins it would save them staring at a screen waiting for something to happen. jenny -Original Message- From: Bobby Hartsfield [mailto:[EMAIL PROTECTED] Sent: 03 September 2006 16:18 To: CF-Talk Subject: mass emailing from CF I've recently been handed a huge mess of a project to clean up. Apparently there have been major issues with a newsletter tool timing out due to the increased number of registrants. I'm just curious how most people are handling sending out massive amounts of in short periods of time. Right now it is basically... 1) Set the timeout to a ridiculously high number 2) Select all the 3) Loop over all users and cfmail the newsletter 4) when people complain about timeouts, increase ridiculously high timeout number I'm sure this worked fine when there were 100-200 people in the database but that number is now very near 10,000. I'm probably not thinking hard enough just yet but there has GOT to be a better/easier way. I'd appreciate any comments or suggestions. . This does need to be fully automated and I can't expect the user to open a meta refreshed page and let it run. (that would be too easy lol) Thanks ...:.:.:.:.:.:.:.:.:.:.:.:. Bobby Hartsfield http://acoderslife.com -- No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.1.405 / Virus Database: 268.11.7/436 - Release Date: 9/1/2006 ~| Introducing the Fusion Authority Quarterly Update. 80 pages of hard-hitting, up-to-date ColdFusion information by your peers, delivered to your door four times a year. http://www.fusionauthority.com/quarterly Archive: http://www.houseoffusion.com/groups/CF-Talk/message.cfm/messageid:251909 Subscription: http://www.houseoffusion.com/groups/CF-Talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4
Re: mass emailing from CF
On 9/3/06, D F <[EMAIL PROTECTED]> wrote: > The logic seems fine to me. If you are sending 10k emails then you are hoping that 10 processes run properly, rather than one. I see that as a tenfold increase in potential points of failure. Sure you can add a 'Stage X complete' email at the end of each send process but that doesn't address the problem, or help solve it... only create the ability to spot a problem with your solution. And what happens when you need to send 11,000? Or 10,001? You need to create a scheduled 11th process. And then a 12th. That means you either have to babysit your maillist count so you can manually add another task or write in code that divides the total list by 10 and run only group 1 of 10, 2 of 10 etc. Which is also not scaleable since you will, sooner or later, exceed whatever your max batch size is (and leading up to that you will watch performance decrease since you haven't addressed the root problem) and have to create Schedule Item 11, then 12 and so on. Just some stuff that came immediately to mind. If you have mail capacity o connection issues I recommend ou address them directly. CF shouldn't be the problem. -- [EMAIL PROTECTED] Janitor, MSB Web Systems mysecretbase.com ~| Introducing the Fusion Authority Quarterly Update. 80 pages of hard-hitting, up-to-date ColdFusion information by your peers, delivered to your door four times a year. http://www.fusionauthority.com/quarterly Archive: http://www.houseoffusion.com/groups/CF-Talk/message.cfm/messageid:251906 Subscription: http://www.houseoffusion.com/groups/CF-Talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4
Re: mass emailing from CF
Hi Bobby, Another thing we've done is to set server variables to limit the amount of batches that are concurrently running. So if there are more than say 3, it prevents the next batch from executing and reschedules the remaining batches. As well after monitoring performance we observed that batches took longer during peak hours ( as to be expected ) so we coded our system to adapt to the time of day etc. ~| Introducing the Fusion Authority Quarterly Update. 80 pages of hard-hitting, up-to-date ColdFusion information by your peers, delivered to your door four times a year. http://www.fusionauthority.com/quarterly Archive: http://www.houseoffusion.com/groups/CF-Talk/message.cfm/messageid:251904 Subscription: http://www.houseoffusion.com/groups/CF-Talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4
Re: mass emailing from CF
Bobby Hartsfield wrote: > > I'm just curious how most people are handling sending out massive amounts of > in short periods of time. Right now it is basically... > > 1) Set the timeout to a ridiculously high number > 2) Select all the > 3) Loop over all users and cfmail the newsletter > 4) when people complain about timeouts, increase ridiculously high timeout > number This shouldn't take long. All CF has to do is generate the content of the e-mail (it is the same for every email and you are caching it, right?) and write it out to a file in the spool directory. I think you would be much better of investigating why this takes long then trying to work around it. Jochem ~| Introducing the Fusion Authority Quarterly Update. 80 pages of hard-hitting, up-to-date ColdFusion information by your peers, delivered to your door four times a year. http://www.fusionauthority.com/quarterly Archive: http://www.houseoffusion.com/groups/CF-Talk/message.cfm/messageid:251903 Subscription: http://www.houseoffusion.com/groups/CF-Talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Re: mass emailing from CF
>Thanks Dan. >By batch do you mean just set up a cfschedules to send X number at a time? Yep. The logic seems fine to me. Depending on the size of your emails you might want to consider leaving more time in between batches. For example if your newsletter is only 20k then you less time is needed between batches if it was 100k. Only way to know is to test... Watch out for IIS's mail server though, we found it was unreliable for this kind of setup using CF. But if you are only sending 10k and the emails are small you may be able to get away with that for a while. ~| Introducing the Fusion Authority Quarterly Update. 80 pages of hard-hitting, up-to-date ColdFusion information by your peers, delivered to your door four times a year. http://www.fusionauthority.com/quarterly Archive: http://www.houseoffusion.com/groups/CF-Talk/message.cfm/messageid:251902 Subscription: http://www.houseoffusion.com/groups/CF-Talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Re: mass emailing from CF
I think you are going to have trouble with your Top X Items idea. I try to shy away from batching mail in chunks like that. Its just not scaleable. Either that or it has a fatal flaw hiding inside that is just as bad as whatever your curent problem is. In this case I think you have both. Consider the fact that CF will happily Select Top X, shoot out X emails, mark X db records and loop around and do it again. Sounds great. Except CF will do it at speeds that no doubt FAR exceed the capacity of your presently-hobbled system. I'd bet you aren't going to accomplish a thing with it as-described. So to solve that poblem you say to youself "I will slow it down programmatically". The olde tyme way was with a CFLoop inside your mail loop set to run for X iterations where running those iterations where those iterations take up some real time. This has the side effect of eating up processor cycles like a Tyrannosaur crashing the Brontosaurus Family Reunion and also tying up a thread while the brontosaurs are mowed down. To which you can say "Aha... I have CFMX Version [whatever] so I can call a java sleep method. This is great news to the brontosaurs' social calendar but still sucks down a thread for ever and ever. Yes I know I am blabbing but I am in the middle of an 8 hr layover at O'Hare Int'l Airport and am bored... Why not look at your mail server? Forget about CF for the moment. Seems like the mail server is the problem. Can you install, for example, iMS? There is a cheap version, and a free one too. Run it off a nonstandard port if you have port 25 already taken. -- [EMAIL PROTECTED] Janitor, MSB Web Systems mysecretbase.com ~| Introducing the Fusion Authority Quarterly Update. 80 pages of hard-hitting, up-to-date ColdFusion information by your peers, delivered to your door four times a year. http://www.fusionauthority.com/quarterly Archive: http://www.houseoffusion.com/groups/CF-Talk/message.cfm/messageid:251901 Subscription: http://www.houseoffusion.com/groups/CF-Talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4
RE: mass emailing from CF
Thanks Dan. By batch do you mean just set up a cfschedules to send X number at a time? Here's the thought I had for cfschedules. 1) Give everyone a 'sent flag' in the database and initialize it to 0 (not sent) 2) Write a page that sends the newsletter to top X number of users whose 'sent flag' is 0. As they are sent, their 'sent flag' is set to 1 3) When the form to send out the emails is submitted - reset everyone's 'sent flag' to 0 - do a little math to find out how many batches to create - create that many cfschedules to hit the page from step 2 and separate them by say a minute or 2 If my messed up gorilla way of thinking is correct, that would make the processing of the 'send newsletter' tool much less strenuous and therefore much less likely to timeout. Do you (or anyone else) see anything wrong with that logic? Is that more or less what you meant by 'set up batches'? Thanks ..:.:.:.:.:.:.:.:.:.:.:.:. Bobby Hartsfield http://acoderslife.com -Original Message- From: D F [mailto:[EMAIL PROTECTED] Sent: Sunday, September 03, 2006 11:59 AM To: CF-Talk Subject: Re: mass emailing from CF Hi Bobby: We current send out around over 40-50k using CF and it runs like a champ. However we are using CF on Linux ( which since we switched from Windows based CF mailer runs much much much better ). I would say you need to set up batches of 1000 or similar number and make sure not to run the batches on top of each other as far as timing goes. It you need any more advice be glad to help. Dan >I've recently been handed a huge mess of a project to clean up. Apparently >there have been major issues with a newsletter tool timing out due to the >increased number of registrants. > >I'm just curious how most people are handling sending out massive amounts of >in short periods of time. Right now it is basically... > >1) Set the timeout to a ridiculously high number >2) Select all the >3) Loop over all users and cfmail the newsletter >4) when people complain about timeouts, increase ridiculously high timeout >number > >I'm sure this worked fine when there were 100-200 people in the database but >that number is now very near 10,000. > >I'm probably not thinking hard enough just yet but there has GOT to be a >better/easier way. I'd appreciate any comments or suggestions. > >... >This does need to be fully automated and I can't expect the user to open a >meta refreshed page and let it run. (that would be too easy lol) > >Thanks > >.:.:.:.:.:.:.:.:.:.:.:.:. >Bobby Hartsfield >http://acoderslife.com > > > > > > > >-- >No virus found in this outgoing message. >Checked by AVG Free Edition. >Version: 7.1.405 / Virus Database: 268.11.7/436 - Release Date: 9/1/2006 ~| Introducing the Fusion Authority Quarterly Update. 80 pages of hard-hitting, up-to-date ColdFusion information by your peers, delivered to your door four times a year. http://www.fusionauthority.com/quarterly Archive: http://www.houseoffusion.com/groups/CF-Talk/message.cfm/messageid:251898 Subscription: http://www.houseoffusion.com/groups/CF-Talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4
Re: mass emailing from CF
Hi Bobby: We current send out around over 40-50k using CF and it runs like a champ. However we are using CF on Linux ( which since we switched from Windows based CF mailer runs much much much better ). I would say you need to set up batches of 1000 or similar number and make sure not to run the batches on top of each other as far as timing goes. It you need any more advice be glad to help. Dan >I've recently been handed a huge mess of a project to clean up. Apparently >there have been major issues with a newsletter tool timing out due to the >increased number of registrants. > >I'm just curious how most people are handling sending out massive amounts of >in short periods of time. Right now it is basically... > >1) Set the timeout to a ridiculously high number >2) Select all the >3) Loop over all users and cfmail the newsletter >4) when people complain about timeouts, increase ridiculously high timeout >number > >I'm sure this worked fine when there were 100-200 people in the database but >that number is now very near 10,000. > >I'm probably not thinking hard enough just yet but there has GOT to be a >better/easier way. I'd appreciate any comments or suggestions. > >... >This does need to be fully automated and I can't expect the user to open a >meta refreshed page and let it run. (that would be too easy lol) > >Thanks > >.:.:.:.:.:.:.:.:.:.:.:.:. >Bobby Hartsfield >http://acoderslife.com > > > > > > > >-- >No virus found in this outgoing message. >Checked by AVG Free Edition. >Version: 7.1.405 / Virus Database: 268.11.7/436 - Release Date: 9/1/2006 ~| Introducing the Fusion Authority Quarterly Update. 80 pages of hard-hitting, up-to-date ColdFusion information by your peers, delivered to your door four times a year. http://www.fusionauthority.com/quarterly Archive: http://www.houseoffusion.com/groups/CF-Talk/message.cfm/messageid:251895 Subscription: http://www.houseoffusion.com/groups/CF-Talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4
RE: mass emailing from CF
Wow, that IS kind of low in comparison to a couple other databases I just checked. The code behind those is basically the same flow too... weird. It must be a slow connection issue then. ..:.:.:.:.:.:.:.:.:.:.:.:. Bobby Hartsfield http://acoderslife.com -Original Message- From: Bobby Hartsfield [mailto:[EMAIL PROTECTED] Sent: Sunday, September 03, 2006 11:49 AM To: CF-Talk Subject: RE: mass emailing from CF Hi Matt, funny you should be the one to reply... and that fast :) I was just checking out http://mysecretbase.com/Slowing_Down_CFMAIL.cfm since it came up in a search for mass emailing with CF. > Are you sure CF is the problem? It very well may not be. It may be the user timing out due to a slow connection, but either way it really needs to be split up to avoid those timeouts wouldnt you think? The number of emails going out is just going to increase with time. ...:.:.:.:.:.:.:.:.:.:.:.:. Bobby Hartsfield http://acoderslife.com -- No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.1.405 / Virus Database: 268.11.7/436 - Release Date: 9/1/2006 ~| Introducing the Fusion Authority Quarterly Update. 80 pages of hard-hitting, up-to-date ColdFusion information by your peers, delivered to your door four times a year. http://www.fusionauthority.com/quarterly Archive: http://www.houseoffusion.com/groups/CF-Talk/message.cfm/messageid:251894 Subscription: http://www.houseoffusion.com/groups/CF-Talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4
RE: mass emailing from CF
Hi Matt, funny you should be the one to reply... and that fast :) I was just checking out http://mysecretbase.com/Slowing_Down_CFMAIL.cfm since it came up in a search for mass emailing with CF. > Are you sure CF is the problem? It very well may not be. It may be the user timing out due to a slow connection, but either way it really needs to be split up to avoid those timeouts wouldnt you think? The number of emails going out is just going to increase with time. ..:.:.:.:.:.:.:.:.:.:.:.:. Bobby Hartsfield http://acoderslife.com -- No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.1.405 / Virus Database: 268.11.7/436 - Release Date: 9/1/2006 ~| Introducing the Fusion Authority Quarterly Update. 80 pages of hard-hitting, up-to-date ColdFusion information by your peers, delivered to your door four times a year. http://www.fusionauthority.com/quarterly Archive: http://www.houseoffusion.com/groups/CF-Talk/message.cfm/messageid:251892 Subscription: http://www.houseoffusion.com/groups/CF-Talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4
Re: mass emailing from CF
10k emails is a pretty small number insofa as CF is concerned. Are you sure CF is the problem? Sure doesn't sound like it. -- [EMAIL PROTECTED] Janitor, MSB Web Systems mysecretbase.com ~| Introducing the Fusion Authority Quarterly Update. 80 pages of hard-hitting, up-to-date ColdFusion information by your peers, delivered to your door four times a year. http://www.fusionauthority.com/quarterly Archive: http://www.houseoffusion.com/groups/CF-Talk/message.cfm/messageid:251891 Subscription: http://www.houseoffusion.com/groups/CF-Talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4
mass emailing from CF
I've recently been handed a huge mess of a project to clean up. Apparently there have been major issues with a newsletter tool timing out due to the increased number of registrants. I'm just curious how most people are handling sending out massive amounts of in short periods of time. Right now it is basically... 1) Set the timeout to a ridiculously high number 2) Select all the 3) Loop over all users and cfmail the newsletter 4) when people complain about timeouts, increase ridiculously high timeout number I'm sure this worked fine when there were 100-200 people in the database but that number is now very near 10,000. I'm probably not thinking hard enough just yet but there has GOT to be a better/easier way. I'd appreciate any comments or suggestions. This does need to be fully automated and I can't expect the user to open a meta refreshed page and let it run. (that would be too easy lol) Thanks ..:.:.:.:.:.:.:.:.:.:.:.:. Bobby Hartsfield http://acoderslife.com -- No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.1.405 / Virus Database: 268.11.7/436 - Release Date: 9/1/2006 ~| Introducing the Fusion Authority Quarterly Update. 80 pages of hard-hitting, up-to-date ColdFusion information by your peers, delivered to your door four times a year. http://www.fusionauthority.com/quarterly Archive: http://www.houseoffusion.com/groups/CF-Talk/message.cfm/messageid:251890 Subscription: http://www.houseoffusion.com/groups/CF-Talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4
