Well, my server is 16GB, 4GB allocated to Jetty, on single thread the app is super efficient. Mostly all the hard things are done in java or through the JNI. Uses and averaged 156MB in TOTAL!!! So as i said there must be a bug here. Im creating 5 threads and disposing them does not matter how many times I do it. Maybe the garbage collector... I would definetely open a bug but I would like to see what Al, you, Matt and others think about this...
I forgot to mention that Im running java 6 and its pretty difficult to update to 7, if i dont have to i wouldnt do it. On Wednesday, October 3, 2012 6:34:56 PM UTC+2, Nitai @ Razuna wrote: > > Sorry, that cfscript code still messes me up from time to time :-) > > My experience with threads inside a loop is that they tend to pile up > like crazy. Even thought you are only starting 5 threads and killing > them again, I still believe there are many more building built up in > the backend then 5. Have you tried it with a smaller number? > > Also you might want to beef up the memory on your machine. We have > 32GB RAM on our machines and allocated Tomcat around 3-5 GB of RAM. > > Kind Regards, > Nitai > > On Wed, Oct 3, 2012 at 6:24 PM, David G Ortega > <[email protected]<javascript:>> > wrote: > > Hi Nitai, > > > > why putting threads inside a loop is a bad idea? Thats esencially the > same > > than doing a simple cfm that uses threads requested 1000000 times in a > row, > > would be even worse if requested several at a time. > > > > If you see Im spawning just only 5 threads and wait them to be disposed > so > > at next loop everything should be fine. We are not even talking about > > creating vars inside the threads, or creating 5000000 threads. Just only > 5 > > threads at a time with no process inside the thread!!! with that simple > code > > the mem is eaten... > > > > Unfortunately I cant do it other way, I have done an api thats working > for > > web inputs but also a Hadoop cluster is running the API to inject > recolected > > data from other sources. > > > > I think there is a memory leak here. > > > > > > On Wednesday, October 3, 2012 6:04:41 PM UTC+2, Nitai @ Razuna wrote: > >> > >> David, > >> > >> Putting thread into a loop is a very bad idea as it will completely > >> eat up your memory (as you have experienced). If those jobs inside the > >> thread need even more memory you are in some serious trouble. > >> > >> Isn't there any other way you can fire off the jobs or segment this > >> further? It just seams a bit crazy to fire off 1 Mil. jobs (at any web > >> server that is). > >> > >> Kind Regards, > >> Nitai > >> > >> On Wed, Oct 3, 2012 at 5:43 PM, David G Ortega <[email protected]> > >> wrote: > >> > Hi!! > >> > > >> > This week I realized that my app can no use threads, when I do that > all > >> > the > >> > memory allocated to jetty is completely eaten... > >> > I need to run near 1000000 jobs (1 million) that runs a multithreaded > >> > function... the whole process takes like 10 hours, with threads, > without > >> > would be 50 hours more or less. Its crucial to make it within a day. > I > >> > thought that maybe somehow I was creating references that could not > be > >> > recollected so I tried a much more simple test and I got same > results. > >> > In my machine takes 3 min to eat all the 4GB allocated. The cpu is > fine, > >> > under 20%. > >> > Following several useful posts I tried to manually set ouput="false" > in > >> > the > >> > cfc or functions but it didn't solve the leak. > >> > > >> > > http://blog.maestropublishing.com/fixing-a-mysterious-memory-leak-on-coldfusion > > >> > > >> > Use the cfc below as a rest service at > >> > http://myopenbdserver/Test.cfc?method=test > >> > > >> > > >> > Server : Windows 2008 > >> > Memory allocated to openbd : 4GB > >> > CPU : Intel Core i3 2130 > 2x2(HT)x3.4+ > >> > GHz > >> > openbd version : 2.0.2 Nightly Build > >> > > >> > Test.cfc > >> > > >> > <cfcomponent> > >> > <cffunction name = 'test' access='remote' returnformat='json'> > >> > <cfscript> > >> > var ii = 0; > >> > while(ii < 500000) > >> > { > >> > for(var i=1; i<=5; i++) > >> > { > >> > thread action="run" name="thread#ii##i#" myvar=i > >> > { > >> > console("running thread number #myvar#"); //if you use desktop > version > >> > ;) > >> > you will see it running > >> > } > >> > } > >> > > >> > for(var i=1; i<=5; i++) > >> > { > >> > thread action="join" name="thread#ii##i#"; > >> > thread action="terminate" name="thread#ii##i#"; > >> > } > >> > > >> > ii++; > >> > } > >> > > >> > return "done ;)"; > >> > </cfscript> > >> > </cffunction> > >> > </cfcomponent> > >> > > >> > -- > >> > online documentation: http://openbd.org/manual/ > >> > http://groups.google.com/group/openbd?hl=en > >> > >> > >> > >> -- > >> See for yourself how easy it is to manage files today. Join the > >> revolution! > >> > >> Razuna - Hosted Digital Asset Management Solution > >> http://www.razuna.com/ > >> > >> Razuna - Open Source Digital Asset Management > >> http://www.razuna.org/ > >> > >> Twitter - http://twitter.com/razunahq > >> Facebook - http://www.facebook.com/razunahq > >> Support Platform - http://getsatisfaction.com/razuna > > > > -- > > online documentation: http://openbd.org/manual/ > > http://groups.google.com/group/openbd?hl=en > > > > -- > See for yourself how easy it is to manage files today. Join the > revolution! > > Razuna - Hosted Digital Asset Management Solution > http://www.razuna.com/ > > Razuna - Open Source Digital Asset Management > http://www.razuna.org/ > > Twitter - http://twitter.com/razunahq > Facebook - http://www.facebook.com/razunahq > Support Platform - http://getsatisfaction.com/razuna > -- online documentation: http://openbd.org/manual/ http://groups.google.com/group/openbd?hl=en
