Thanks very much for this great response. I'm going to look at implementing the idea you had for sending things out in batches. I've got another big send to do this week so we'll see how it goes!
> -----Original Message----- > From: Bobby Hartsfield [mailto:bo...@acoderslife.com] > Sent: 21 May 2011 14:25 > To: cf-talk > Subject: RE: CFThread Memory Issues sending emails > > > For a quick fix, I'd split those up into smaller batches. For instance, your > cfthread could process 1000 emails (or whatever number you decide is > manageable by your configuration) id make the count a parameter that is > easily passed in via URL so all you'd have to do is change the url var in a > scheduled task to increase/decrease the count. > > Then the thread's process would be something like... > > 1) Grab top X records that need to be sent. > 2) Send the emails for those top X records > 3) Update those records to indicate that they have been processed so they > arent grabbed again by step 1. > 4) Stop. > > Then just schedule it to run every minute or so. > > If you use timestamps as your step 3 indicator, you still have the start and > stop times once there are no emails left to send. > > > For a more elaborate solution, id queue individual emails up into a DB table > and have multiple threads process them one at a time until they were done > (makeshift CF message queue). > > I've found that letting a thread process 100s of thousands of small items > individually makes that thread much happier than if it had to process even a > fraction of that amount all at once. > > That said, I would have to agree and think your described setup should have > zero problem with sending that many emails in a single loop. > > Have you monitored CF to see how the thread was doing? Maybe watch > jconsole memory stats as the thread is running to see if it is indeed climbing > rapidly and causing the GC? > > Maybe the thread is being spawned more than once? Maybe you could try > loading all of the used variables into a single struct within that thread so you > can clear that struct at the end of the loop? > > I'm just throwing out ideas here... > > > .:.:.:.:.:.:.:.:.:.:.:.:.:. > Bobby Hartsfield > http://acoderslife.com > http://cf4em.com > > > > -----Original Message----- > From: Firewall [mailto:firew...@cc.uk.com] > Sent: Thursday, May 19, 2011 7:38 AM > To: cf-talk > Subject: CFThread Memory Issues sending emails > > > This one has me really stumped. I'm running a pretty simple cfmail script to > send out email messages to a user list. Up until now I was simply doing it > with a browser reload but the client recently doubled the number of emails > they want to send to about 57,000. So I decided to implement this with > cfthread wrapped around my loop containing the cfquery and cfmail code. > The > idea being that the server simply returns the page to the browser and kicks > of a massive send job in the background. Once the mail script completes an > email is send to the user saying " I sent your email to X people, I started > at xx:xx and finished at xx:xx" > > My testing worked perfectly and the client used it on smaller email jobs > without a problem, I tried it for the first time yesterday on the main > massive list and about a third of the messages were sent out before the > server seemed to hang. Looking at the logs I saw loads of JVM errors > referring to "GC" aka the garbage collector. > The specific log entry was: "java.lang.OutOfMemoryError: GC overhead limit > exceeded" > > The server seems to have no problems sending about 10-15k emails using > this > process but when I try it on this one big list it results in all kinds of > problems so I'm wondering if this is an issue with cfthread, JVM, CF 8.0.1 > or my code. > > I'm wondering if this anyone else has tried to do this and seen the same > problem; and if so how did you go about solving it? > > My setup is Coldfusion 8.0.1 (64bit) running on Red Hat Enterprise 5, the > server itself is a nearly new core i7 Dell PowerEdge with Raid SSD hard > drives and 12Gb RAM; in other words it should look at 57k emails and just > laugh at the smallness of the list :) > > I've included the outline of my send code just in case someone spots > something really stupid in my code: > > <cfthread name="sendEmail#Form.id#" action="run"> > > <cfset startingTime = '#timeformat(now(), "HH:MM:SS")#'> > > <cfquery datasource="#Application.ds#" name="gn"> > Grab info to send in email > </cfquery> > > <cfloop list="#Form.sendToList#" index="e"> > <cfoutput> > > <cfquery datasource="#Application.ds#" name="gu"> > Grab user details based on ID contained in "e" > </cfquery> > > <cftry> > > <cfmail to="#emailSendTo#" > from="#gn.addName# > <#gn.addEmail#>" > subject="#gn.msgSubj#" > type="html"> > <cfinclude > template="email-body-#gn.emailType#.cfm"> > </cfmail> > > <cfcatch type="any"> > <div style="color:red;"> > Failed send: #gu.email# - > #cfcatch.message#<br /> > </div> > </cfcatch> > > </cftry> > > > </cfoutput> > </cfloop> > > <cfset endingTime = '#timeformat(now(), "HH:MM:SS")#'> > > <cfmail to="#getauthuser()#" bcc="e...@cc.uk.com" from="Ed > <e...@cc.uk.com>" > subject="I finished sending your newsletter">#gn.msgSubj# sent to > #listlen(Form.sendToList)# people > > Started: #startingTime# > Finished: #endingTime# > </cfmail> > > </cfthread> > > > best wishes, > > -= Ed > > -------------------------------------------- > Edward Chanter > Technical Innovations Manager > Collaborative Connections Ltd. > > http://www.cc.uk.com/ > -------------------------------------------- > > > > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > ~~~~~~~~~~~| > Order the Adobe Coldfusion Anthology now! > http://www.amazon.com/Adobe-Coldfusion- > Anthology/dp/1430272155/?tag=houseoffusion > Archive: http://www.houseoffusion.com/groups/cf- > talk/message.cfm/messageid:344803 > Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm > Unsubscribe: http://www.houseoffusion.com/groups/cf- > talk/unsubscribe ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~| Order the Adobe Coldfusion Anthology now! http://www.amazon.com/Adobe-Coldfusion-Anthology/dp/1430272155/?tag=houseoffusion Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:344834 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/groups/cf-talk/unsubscribe.cfm
