Sorry, that cfscript code still messes me up from time to time :-) My experience with threads inside a loop is that they tend to pile up like crazy. Even thought you are only starting 5 threads and killing them again, I still believe there are many more building built up in the backend then 5. Have you tried it with a smaller number?
Also you might want to beef up the memory on your machine. We have 32GB RAM on our machines and allocated Tomcat around 3-5 GB of RAM. Kind Regards, Nitai On Wed, Oct 3, 2012 at 6:24 PM, David G Ortega <[email protected]> wrote: > Hi Nitai, > > why putting threads inside a loop is a bad idea? Thats esencially the same > than doing a simple cfm that uses threads requested 1000000 times in a row, > would be even worse if requested several at a time. > > If you see Im spawning just only 5 threads and wait them to be disposed so > at next loop everything should be fine. We are not even talking about > creating vars inside the threads, or creating 5000000 threads. Just only 5 > threads at a time with no process inside the thread!!! with that simple code > the mem is eaten... > > Unfortunately I cant do it other way, I have done an api thats working for > web inputs but also a Hadoop cluster is running the API to inject recolected > data from other sources. > > I think there is a memory leak here. > > > On Wednesday, October 3, 2012 6:04:41 PM UTC+2, Nitai @ Razuna wrote: >> >> David, >> >> Putting thread into a loop is a very bad idea as it will completely >> eat up your memory (as you have experienced). If those jobs inside the >> thread need even more memory you are in some serious trouble. >> >> Isn't there any other way you can fire off the jobs or segment this >> further? It just seams a bit crazy to fire off 1 Mil. jobs (at any web >> server that is). >> >> Kind Regards, >> Nitai >> >> On Wed, Oct 3, 2012 at 5:43 PM, David G Ortega <[email protected]> >> wrote: >> > Hi!! >> > >> > This week I realized that my app can no use threads, when I do that all >> > the >> > memory allocated to jetty is completely eaten... >> > I need to run near 1000000 jobs (1 million) that runs a multithreaded >> > function... the whole process takes like 10 hours, with threads, without >> > would be 50 hours more or less. Its crucial to make it within a day. I >> > thought that maybe somehow I was creating references that could not be >> > recollected so I tried a much more simple test and I got same results. >> > In my machine takes 3 min to eat all the 4GB allocated. The cpu is fine, >> > under 20%. >> > Following several useful posts I tried to manually set ouput="false" in >> > the >> > cfc or functions but it didn't solve the leak. >> > >> > http://blog.maestropublishing.com/fixing-a-mysterious-memory-leak-on-coldfusion >> > >> > Use the cfc below as a rest service at >> > http://myopenbdserver/Test.cfc?method=test >> > >> > >> > Server : Windows 2008 >> > Memory allocated to openbd : 4GB >> > CPU : Intel Core i3 2130 2x2(HT)x3.4+ >> > GHz >> > openbd version : 2.0.2 Nightly Build >> > >> > Test.cfc >> > >> > <cfcomponent> >> > <cffunction name = 'test' access='remote' returnformat='json'> >> > <cfscript> >> > var ii = 0; >> > while(ii < 500000) >> > { >> > for(var i=1; i<=5; i++) >> > { >> > thread action="run" name="thread#ii##i#" myvar=i >> > { >> > console("running thread number #myvar#"); //if you use desktop version >> > ;) >> > you will see it running >> > } >> > } >> > >> > for(var i=1; i<=5; i++) >> > { >> > thread action="join" name="thread#ii##i#"; >> > thread action="terminate" name="thread#ii##i#"; >> > } >> > >> > ii++; >> > } >> > >> > return "done ;)"; >> > </cfscript> >> > </cffunction> >> > </cfcomponent> >> > >> > -- >> > online documentation: http://openbd.org/manual/ >> > http://groups.google.com/group/openbd?hl=en >> >> >> >> -- >> See for yourself how easy it is to manage files today. Join the >> revolution! >> >> Razuna - Hosted Digital Asset Management Solution >> http://www.razuna.com/ >> >> Razuna - Open Source Digital Asset Management >> http://www.razuna.org/ >> >> Twitter - http://twitter.com/razunahq >> Facebook - http://www.facebook.com/razunahq >> Support Platform - http://getsatisfaction.com/razuna > > -- > online documentation: http://openbd.org/manual/ > http://groups.google.com/group/openbd?hl=en -- See for yourself how easy it is to manage files today. Join the revolution! Razuna - Hosted Digital Asset Management Solution http://www.razuna.com/ Razuna - Open Source Digital Asset Management http://www.razuna.org/ Twitter - http://twitter.com/razunahq Facebook - http://www.facebook.com/razunahq Support Platform - http://getsatisfaction.com/razuna -- online documentation: http://openbd.org/manual/ http://groups.google.com/group/openbd?hl=en
