Hi Nitai, why putting threads inside a loop is a bad idea? Thats esencially the same than doing a simple cfm that uses threads requested 1000000 times in a row, would be even worse if requested several at a time.
If you see Im spawning just only 5 threads and wait them to be disposed so at next loop everything should be fine. We are not even talking about creating vars inside the threads, or creating 5000000 threads. Just only 5 threads at a time with no process inside the thread!!! with that simple code the mem is eaten... Unfortunately I cant do it other way, I have done an api thats working for web inputs but also a Hadoop cluster is running the API to inject recolected data from other sources. I think there is a memory leak here. On Wednesday, October 3, 2012 6:04:41 PM UTC+2, Nitai @ Razuna wrote: > > David, > > Putting thread into a loop is a very bad idea as it will completely > eat up your memory (as you have experienced). If those jobs inside the > thread need even more memory you are in some serious trouble. > > Isn't there any other way you can fire off the jobs or segment this > further? It just seams a bit crazy to fire off 1 Mil. jobs (at any web > server that is). > > Kind Regards, > Nitai > > On Wed, Oct 3, 2012 at 5:43 PM, David G Ortega > <[email protected]<javascript:>> > wrote: > > Hi!! > > > > This week I realized that my app can no use threads, when I do that all > the > > memory allocated to jetty is completely eaten... > > I need to run near 1000000 jobs (1 million) that runs a multithreaded > > function... the whole process takes like 10 hours, with threads, without > > would be 50 hours more or less. Its crucial to make it within a day. I > > thought that maybe somehow I was creating references that could not be > > recollected so I tried a much more simple test and I got same results. > > In my machine takes 3 min to eat all the 4GB allocated. The cpu is fine, > > under 20%. > > Following several useful posts I tried to manually set ouput="false" in > the > > cfc or functions but it didn't solve the leak. > > > http://blog.maestropublishing.com/fixing-a-mysterious-memory-leak-on-coldfusion > > > > > Use the cfc below as a rest service at > > http://myopenbdserver/Test.cfc?method=test > > > > > > Server : Windows 2008 > > Memory allocated to openbd : 4GB > > CPU : Intel Core i3 2130 2x2(HT)x3.4+ > GHz > > openbd version : 2.0.2 Nightly Build > > > > Test.cfc > > > > <cfcomponent> > > <cffunction name = 'test' access='remote' returnformat='json'> > > <cfscript> > > var ii = 0; > > while(ii < 500000) > > { > > for(var i=1; i<=5; i++) > > { > > thread action="run" name="thread#ii##i#" myvar=i > > { > > console("running thread number #myvar#"); //if you use desktop version > ;) > > you will see it running > > } > > } > > > > for(var i=1; i<=5; i++) > > { > > thread action="join" name="thread#ii##i#"; > > thread action="terminate" name="thread#ii##i#"; > > } > > > > ii++; > > } > > > > return "done ;)"; > > </cfscript> > > </cffunction> > > </cfcomponent> > > > > -- > > online documentation: http://openbd.org/manual/ > > http://groups.google.com/group/openbd?hl=en > > > > -- > See for yourself how easy it is to manage files today. Join the > revolution! > > Razuna - Hosted Digital Asset Management Solution > http://www.razuna.com/ > > Razuna - Open Source Digital Asset Management > http://www.razuna.org/ > > Twitter - http://twitter.com/razunahq > Facebook - http://www.facebook.com/razunahq > Support Platform - http://getsatisfaction.com/razuna > -- online documentation: http://openbd.org/manual/ http://groups.google.com/group/openbd?hl=en
