Re: Cfmx 7.1 on 2.3k locking up once a day
One thing I noticed in your thread dumps were some threads waiting on a UUID to be generated (the code in question is line 81 of D:\www\com\gf\ad\ad.cfc). Does this code get run once or more per request? Take a peek at jrpp-100 thru jrpp-107 in the thread dumps. -Dave >We just rolled out mx 7.1 on win 2.3k, Fusebox 4.1. It's a simple >application, just content management with 150-200k views a day. > >Without fail that mx server hangs in under 24 hours. Here is the evidence we >have found. > >I started app. Server at the command line: > >When I peep'd in while it was hung it was dumping out the following: > >java.lang.RuntimeException: Request timed out waiting for an available >thread to run. You may want to consider increasing the number of active >threads in the thread pool. > >This is repeated, again and again. > >NOTE: found some articles suggesting ideas on this, turn of debug (done), >make sure db is responding (appears to be). Adjust some of the jrun.xml >settings (appears to already be done in 7.1) > >Perf Mon. >went back to before the crash, the only thing odd is the data seems to miss >a beat nearing the crash and some data seems off the chart. But when we >examine logs we see no burst of traffic. I exported the perf. Mon data and >don't see any bursts. > >Did a stack dump while it was hung found a ton of these: > >"jrpp-108" prio=5 tid=0x04430d60 nid=0x1b08 in Object.wait() >[c4af000..c4afdb8] > at java.lang.Object.wait(Native Method) > - waiting on <0x172c0fa8> (a jrunx.scheduler.ThreadPool$Throttle) > at jrunx.scheduler.ThreadPool$Throttle.enter(ThreadPool.java:107) > - locked <0x172c0fa8> (a jrunx.scheduler.ThreadPool$Throttle) > at >jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:423 >) > at >jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:26 >4) > at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66) > > >Whole log: http://icc.getfused.com/out.txt > > >We have tried moving app. To a 2nd 7.1 and win 2.3k server.. Same issue. We >moved the db to another machine, same issue. No other sites using the db >seem to be having any issues. > >Anyone have any thoughts? Im about to toss in the towel and call adobe and >burn 500.00 clams. Any help would be apprecaited, if it saves me the 500.00 >then a shopping spree at Amazon could be in your future. > >Thanks > >gabe ~| Logware (www.logware.us): a new and convenient web-based time tracking application. Start tracking and documenting hours spent on a project or with a client with Logware today. Try it for free with a 15 day trial account. http://www.houseoffusion.com/banners/view.cfm?bannerid=67 Message: http://www.houseoffusion.com/lists.cfm/link=i:4:225527 Archives: http://www.houseoffusion.com/cf_lists/threads.cfm/4 Subscription: http://www.houseoffusion.com/lists.cfm/link=s:4 Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4 Donations & Support: http://www.houseoffusion.com/tiny.cfm/54
Re: Cfmx 7.1 on 2.3k locking up once a day
On 11/28/05, gabriel l smallman <[EMAIL PROTECTED]> wrote: > We just rolled out mx 7.1 on win 2.3k, Fusebox 4.1. It's a simple > application, just content management with 150-200k views a day. > > Without fail that mx server hangs in under 24 hours. Here is the evidence we > have found. > > I started app. Server at the command line: > > When I peep'd in while it was hung it was dumping out the following: > > java.lang.RuntimeException: Request timed out waiting for an available > thread to run. You may want to consider increasing the number of active > threads in the thread pool. > > This is repeated, again and again. > > NOTE: found some articles suggesting ideas on this, turn of debug (done), > make sure db is responding (appears to be). Adjust some of the jrun.xml > settings (appears to already be done in 7.1) > > Perf Mon. > went back to before the crash, the only thing odd is the data seems to miss > a beat nearing the crash and some data seems off the chart. But when we > examine logs we see no burst of traffic. I exported the perf. Mon data and > don't see any bursts. > > Did a stack dump while it was hung found a ton of these: > > "jrpp-108" prio=5 tid=0x04430d60 nid=0x1b08 in Object.wait() > [c4af000..c4afdb8] > at java.lang.Object.wait(Native Method) > - waiting on <0x172c0fa8> (a jrunx.scheduler.ThreadPool$Throttle) > at jrunx.scheduler.ThreadPool$Throttle.enter(ThreadPool.java:107) > - locked <0x172c0fa8> (a jrunx.scheduler.ThreadPool$Throttle) > at > jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:423 > ) > at > jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:26 > 4) > at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66) > > > Whole log: http://icc.getfused.com/out.txt > > > We have tried moving app. To a 2nd 7.1 and win 2.3k server.. Same issue. We > moved the db to another machine, same issue. No other sites using the db > seem to be having any issues. > > Anyone have any thoughts? Im about to toss in the towel and call adobe and > burn 500.00 clams. Any help would be apprecaited, if it saves me the 500.00 > then a shopping spree at Amazon could be in your future. > This blog posting by Steven Erat seems timely, as it covers your error message: http://www.talkingtree.com/blog/index.cfm/2005/11/28/Request-timed-out-waiting-for-an-available-thread-to-run Make sure you check the activeThread settings in your jrun.xml file. The defaults are woefully wrong, even for the most powerful boxes. Also check the threadWaitTimeout settings. Regards, Dave. ~| Logware (www.logware.us): a new and convenient web-based time tracking application. Start tracking and documenting hours spent on a project or with a client with Logware today. Try it for free with a 15 day trial account. http://www.houseoffusion.com/banners/view.cfm?bannerid=67 Message: http://www.houseoffusion.com/lists.cfm/link=i:4:225495 Archives: http://www.houseoffusion.com/cf_lists/threads.cfm/4 Subscription: http://www.houseoffusion.com/lists.cfm/link=s:4 Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4 Donations & Support: http://www.houseoffusion.com/tiny.cfm/54
RE: Cfmx 7.1 on 2.3k locking up once a day
Yes on min and maxheap sizes. I have never seen the heap over 500megs. The trace logs show very steady memory levels. gabe -Original Message- From: Michael Dinowitz [mailto:[EMAIL PROTECTED] Sent: Monday, November 28, 2005 9:10 PM To: CF-Talk Subject: Re: Cfmx 7.1 on 2.3k locking up once a day min as in minheap and max as in maxheap? Does the jrun.exe current or virtual memory size ever exceed 1000? > Min memory is set to 500, max to 1000, system seems to idle around 400 > megs, > up and down a little but perf. Mon shows no memory usages spikes. > > Client vars are disabled. Though sessions are turned on in the index.cfm > file, very little use for them except for the content administrator login. > Thought each page request does set three session vars by default. We have > no > checkout, or anything that is complex in session scope. > > gabe > > > -Original Message- > From: Michael Dinowitz [mailto:[EMAIL PROTECTED] > Sent: Monday, November 28, 2005 8:52 PM > To: CF-Talk > Subject: Re: Cfmx 7.1 on 2.3k locking up once a day > > What's your memory like on the machine? What's the JVM set as? Are you > using > lots of session or client vars? > > >> We just rolled out mx 7.1 on win 2.3k, Fusebox 4.1. It's a simple >> application, just content management with 150-200k views a day. >> >> Without fail that mx server hangs in under 24 hours. Here is the evidence >> we >> have found. >> >> I started app. Server at the command line: >> >> When I peep'd in while it was hung it was dumping out the following: >> >> java.lang.RuntimeException: Request timed out waiting for an available >> thread to run. You may want to consider increasing the number of active >> threads in the thread pool. >> >> This is repeated, again and again. >> >> NOTE: found some articles suggesting ideas on this, turn of debug (done), >> make sure db is responding (appears to be). Adjust some of the jrun.xml >> settings (appears to already be done in 7.1) >> >> Perf Mon. >> went back to before the crash, the only thing odd is the data seems to >> miss >> a beat nearing the crash and some data seems off the chart. But when we >> examine logs we see no burst of traffic. I exported the perf. Mon data >> and >> don't see any bursts. >> >> Did a stack dump while it was hung found a ton of these: >> >> "jrpp-108" prio=5 tid=0x04430d60 nid=0x1b08 in Object.wait() >> [c4af000..c4afdb8] >> at java.lang.Object.wait(Native Method) >> - waiting on <0x172c0fa8> (a jrunx.scheduler.ThreadPool$Throttle) >> at jrunx.scheduler.ThreadPool$Throttle.enter(ThreadPool.java:107) >> - locked <0x172c0fa8> (a jrunx.scheduler.ThreadPool$Throttle) >> at >> > jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:423 >> ) >> at >> > jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:26 >> 4) >> at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66) >> >> >> Whole log: http://icc.getfused.com/out.txt >> >> >> We have tried moving app. To a 2nd 7.1 and win 2.3k server.. Same issue. >> We >> moved the db to another machine, same issue. No other sites using the db >> seem to be having any issues. >> >> Anyone have any thoughts? Im about to toss in the towel and call adobe >> and >> burn 500.00 clams. Any help would be apprecaited, if it saves me the >> 500.00 >> then a shopping spree at Amazon could be in your future. >> >> Thanks >> >> gabe >> >> >> > > > > ~| Logware (www.logware.us): a new and convenient web-based time tracking application. Start tracking and documenting hours spent on a project or with a client with Logware today. Try it for free with a 15 day trial account. http://www.houseoffusion.com/banners/view.cfm?bannerid=67 Message: http://www.houseoffusion.com/lists.cfm/link=i:4:225492 Archives: http://www.houseoffusion.com/cf_lists/threads.cfm/4 Subscription: http://www.houseoffusion.com/lists.cfm/link=s:4 Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4 Donations & Support: http://www.houseoffusion.com/tiny.cfm/54
RE: Cfmx 7.1 on 2.3k locking up once a day
Gabe - consider some alternative settings for your JVM - things like garbage collection and "newsize" etc. This blog http://mkruger.cfwebtools.com/index.cfm?mode=entry&entry=395F3C8A-058B-CADD- 826F03A60674D5B3 has some good info - plus links to other blogs on the subject. -Mark -Original Message- From: gabriel l smallman [mailto:[EMAIL PROTECTED] Sent: Monday, November 28, 2005 8:04 PM To: CF-Talk Subject: RE: Cfmx 7.1 on 2.3k locking up once a day Min memory is set to 500, max to 1000, system seems to idle around 400 megs, up and down a little but perf. Mon shows no memory usages spikes. Client vars are disabled. Though sessions are turned on in the index.cfm file, very little use for them except for the content administrator login. Thought each page request does set three session vars by default. We have no checkout, or anything that is complex in session scope. gabe -Original Message- From: Michael Dinowitz [mailto:[EMAIL PROTECTED] Sent: Monday, November 28, 2005 8:52 PM To: CF-Talk Subject: Re: Cfmx 7.1 on 2.3k locking up once a day What's your memory like on the machine? What's the JVM set as? Are you using lots of session or client vars? > We just rolled out mx 7.1 on win 2.3k, Fusebox 4.1. It's a simple > application, just content management with 150-200k views a day. > > Without fail that mx server hangs in under 24 hours. Here is the evidence > we > have found. > > I started app. Server at the command line: > > When I peep'd in while it was hung it was dumping out the following: > > java.lang.RuntimeException: Request timed out waiting for an available > thread to run. You may want to consider increasing the number of active > threads in the thread pool. > > This is repeated, again and again. > > NOTE: found some articles suggesting ideas on this, turn of debug (done), > make sure db is responding (appears to be). Adjust some of the jrun.xml > settings (appears to already be done in 7.1) > > Perf Mon. > went back to before the crash, the only thing odd is the data seems to > miss > a beat nearing the crash and some data seems off the chart. But when we > examine logs we see no burst of traffic. I exported the perf. Mon data and > don't see any bursts. > > Did a stack dump while it was hung found a ton of these: > > "jrpp-108" prio=5 tid=0x04430d60 nid=0x1b08 in Object.wait() > [c4af000..c4afdb8] > at java.lang.Object.wait(Native Method) > - waiting on <0x172c0fa8> (a jrunx.scheduler.ThreadPool$Throttle) > at jrunx.scheduler.ThreadPool$Throttle.enter(ThreadPool.java:107) > - locked <0x172c0fa8> (a jrunx.scheduler.ThreadPool$Throttle) > at > jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:423 > ) > at > jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:26 > 4) > at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66) > > > Whole log: http://icc.getfused.com/out.txt > > > We have tried moving app. To a 2nd 7.1 and win 2.3k server.. Same issue. > We > moved the db to another machine, same issue. No other sites using the db > seem to be having any issues. > > Anyone have any thoughts? Im about to toss in the towel and call adobe and > burn 500.00 clams. Any help would be apprecaited, if it saves me the > 500.00 > then a shopping spree at Amazon could be in your future. > > Thanks > > gabe > > > ~| Logware (www.logware.us): a new and convenient web-based time tracking application. Start tracking and documenting hours spent on a project or with a client with Logware today. Try it for free with a 15 day trial account. http://www.houseoffusion.com/banners/view.cfm?bannerid=67 Message: http://www.houseoffusion.com/lists.cfm/link=i:4:225489 Archives: http://www.houseoffusion.com/cf_lists/threads.cfm/4 Subscription: http://www.houseoffusion.com/lists.cfm/link=s:4 Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4 Donations & Support: http://www.houseoffusion.com/tiny.cfm/54
Re: Cfmx 7.1 on 2.3k locking up once a day
min as in minheap and max as in maxheap? Does the jrun.exe current or virtual memory size ever exceed 1000? > Min memory is set to 500, max to 1000, system seems to idle around 400 > megs, > up and down a little but perf. Mon shows no memory usages spikes. > > Client vars are disabled. Though sessions are turned on in the index.cfm > file, very little use for them except for the content administrator login. > Thought each page request does set three session vars by default. We have > no > checkout, or anything that is complex in session scope. > > gabe > > > -Original Message- > From: Michael Dinowitz [mailto:[EMAIL PROTECTED] > Sent: Monday, November 28, 2005 8:52 PM > To: CF-Talk > Subject: Re: Cfmx 7.1 on 2.3k locking up once a day > > What's your memory like on the machine? What's the JVM set as? Are you > using > lots of session or client vars? > > >> We just rolled out mx 7.1 on win 2.3k, Fusebox 4.1. It's a simple >> application, just content management with 150-200k views a day. >> >> Without fail that mx server hangs in under 24 hours. Here is the evidence >> we >> have found. >> >> I started app. Server at the command line: >> >> When I peep'd in while it was hung it was dumping out the following: >> >> java.lang.RuntimeException: Request timed out waiting for an available >> thread to run. You may want to consider increasing the number of active >> threads in the thread pool. >> >> This is repeated, again and again. >> >> NOTE: found some articles suggesting ideas on this, turn of debug (done), >> make sure db is responding (appears to be). Adjust some of the jrun.xml >> settings (appears to already be done in 7.1) >> >> Perf Mon. >> went back to before the crash, the only thing odd is the data seems to >> miss >> a beat nearing the crash and some data seems off the chart. But when we >> examine logs we see no burst of traffic. I exported the perf. Mon data >> and >> don't see any bursts. >> >> Did a stack dump while it was hung found a ton of these: >> >> "jrpp-108" prio=5 tid=0x04430d60 nid=0x1b08 in Object.wait() >> [c4af000..c4afdb8] >> at java.lang.Object.wait(Native Method) >> - waiting on <0x172c0fa8> (a jrunx.scheduler.ThreadPool$Throttle) >> at jrunx.scheduler.ThreadPool$Throttle.enter(ThreadPool.java:107) >> - locked <0x172c0fa8> (a jrunx.scheduler.ThreadPool$Throttle) >> at >> > jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:423 >> ) >> at >> > jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:26 >> 4) >> at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66) >> >> >> Whole log: http://icc.getfused.com/out.txt >> >> >> We have tried moving app. To a 2nd 7.1 and win 2.3k server.. Same issue. >> We >> moved the db to another machine, same issue. No other sites using the db >> seem to be having any issues. >> >> Anyone have any thoughts? Im about to toss in the towel and call adobe >> and >> burn 500.00 clams. Any help would be apprecaited, if it saves me the >> 500.00 >> then a shopping spree at Amazon could be in your future. >> >> Thanks >> >> gabe >> >> >> > > > > ~| Logware (www.logware.us): a new and convenient web-based time tracking application. Start tracking and documenting hours spent on a project or with a client with Logware today. Try it for free with a 15 day trial account. http://www.houseoffusion.com/banners/view.cfm?bannerid=67 Message: http://www.houseoffusion.com/lists.cfm/link=i:4:225488 Archives: http://www.houseoffusion.com/cf_lists/threads.cfm/4 Subscription: http://www.houseoffusion.com/lists.cfm/link=s:4 Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4 Donations & Support: http://www.houseoffusion.com/tiny.cfm/54
RE: Cfmx 7.1 on 2.3k locking up once a day
Min memory is set to 500, max to 1000, system seems to idle around 400 megs, up and down a little but perf. Mon shows no memory usages spikes. Client vars are disabled. Though sessions are turned on in the index.cfm file, very little use for them except for the content administrator login. Thought each page request does set three session vars by default. We have no checkout, or anything that is complex in session scope. gabe -Original Message- From: Michael Dinowitz [mailto:[EMAIL PROTECTED] Sent: Monday, November 28, 2005 8:52 PM To: CF-Talk Subject: Re: Cfmx 7.1 on 2.3k locking up once a day What's your memory like on the machine? What's the JVM set as? Are you using lots of session or client vars? > We just rolled out mx 7.1 on win 2.3k, Fusebox 4.1. It's a simple > application, just content management with 150-200k views a day. > > Without fail that mx server hangs in under 24 hours. Here is the evidence > we > have found. > > I started app. Server at the command line: > > When I peep'd in while it was hung it was dumping out the following: > > java.lang.RuntimeException: Request timed out waiting for an available > thread to run. You may want to consider increasing the number of active > threads in the thread pool. > > This is repeated, again and again. > > NOTE: found some articles suggesting ideas on this, turn of debug (done), > make sure db is responding (appears to be). Adjust some of the jrun.xml > settings (appears to already be done in 7.1) > > Perf Mon. > went back to before the crash, the only thing odd is the data seems to > miss > a beat nearing the crash and some data seems off the chart. But when we > examine logs we see no burst of traffic. I exported the perf. Mon data and > don't see any bursts. > > Did a stack dump while it was hung found a ton of these: > > "jrpp-108" prio=5 tid=0x04430d60 nid=0x1b08 in Object.wait() > [c4af000..c4afdb8] > at java.lang.Object.wait(Native Method) > - waiting on <0x172c0fa8> (a jrunx.scheduler.ThreadPool$Throttle) > at jrunx.scheduler.ThreadPool$Throttle.enter(ThreadPool.java:107) > - locked <0x172c0fa8> (a jrunx.scheduler.ThreadPool$Throttle) > at > jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:423 > ) > at > jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:26 > 4) > at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66) > > > Whole log: http://icc.getfused.com/out.txt > > > We have tried moving app. To a 2nd 7.1 and win 2.3k server.. Same issue. > We > moved the db to another machine, same issue. No other sites using the db > seem to be having any issues. > > Anyone have any thoughts? Im about to toss in the towel and call adobe and > burn 500.00 clams. Any help would be apprecaited, if it saves me the > 500.00 > then a shopping spree at Amazon could be in your future. > > Thanks > > gabe > > > ~| Discover CFTicket - The leading ColdFusion Help Desk and Trouble Ticket application http://www.houseoffusion.com/banners/view.cfm?bannerid=48 Message: http://www.houseoffusion.com/lists.cfm/link=i:4:225486 Archives: http://www.houseoffusion.com/cf_lists/threads.cfm/4 Subscription: http://www.houseoffusion.com/lists.cfm/link=s:4 Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4 Donations & Support: http://www.houseoffusion.com/tiny.cfm/54
Re: Cfmx 7.1 on 2.3k locking up once a day
What's your memory like on the machine? What's the JVM set as? Are you using lots of session or client vars? > We just rolled out mx 7.1 on win 2.3k, Fusebox 4.1. It's a simple > application, just content management with 150-200k views a day. > > Without fail that mx server hangs in under 24 hours. Here is the evidence > we > have found. > > I started app. Server at the command line: > > When I peep'd in while it was hung it was dumping out the following: > > java.lang.RuntimeException: Request timed out waiting for an available > thread to run. You may want to consider increasing the number of active > threads in the thread pool. > > This is repeated, again and again. > > NOTE: found some articles suggesting ideas on this, turn of debug (done), > make sure db is responding (appears to be). Adjust some of the jrun.xml > settings (appears to already be done in 7.1) > > Perf Mon. > went back to before the crash, the only thing odd is the data seems to > miss > a beat nearing the crash and some data seems off the chart. But when we > examine logs we see no burst of traffic. I exported the perf. Mon data and > don't see any bursts. > > Did a stack dump while it was hung found a ton of these: > > "jrpp-108" prio=5 tid=0x04430d60 nid=0x1b08 in Object.wait() > [c4af000..c4afdb8] > at java.lang.Object.wait(Native Method) > - waiting on <0x172c0fa8> (a jrunx.scheduler.ThreadPool$Throttle) > at jrunx.scheduler.ThreadPool$Throttle.enter(ThreadPool.java:107) > - locked <0x172c0fa8> (a jrunx.scheduler.ThreadPool$Throttle) > at > jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:423 > ) > at > jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:26 > 4) > at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66) > > > Whole log: http://icc.getfused.com/out.txt > > > We have tried moving app. To a 2nd 7.1 and win 2.3k server.. Same issue. > We > moved the db to another machine, same issue. No other sites using the db > seem to be having any issues. > > Anyone have any thoughts? Im about to toss in the towel and call adobe and > burn 500.00 clams. Any help would be apprecaited, if it saves me the > 500.00 > then a shopping spree at Amazon could be in your future. > > Thanks > > gabe > > > ~| Logware (www.logware.us): a new and convenient web-based time tracking application. Start tracking and documenting hours spent on a project or with a client with Logware today. Try it for free with a 15 day trial account. http://www.houseoffusion.com/banners/view.cfm?bannerid=67 Message: http://www.houseoffusion.com/lists.cfm/link=i:4:225483 Archives: http://www.houseoffusion.com/cf_lists/threads.cfm/4 Subscription: http://www.houseoffusion.com/lists.cfm/link=s:4 Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4 Donations & Support: http://www.houseoffusion.com/tiny.cfm/54
RE: Cfmx 7.1 on 2.3k locking up once a day
Thanks for the ideas rob couple thoughts on them, 1. Haven't tried seefusion, I will install and see what I can see. 2. queries are all short, we have 3 other cf server running db sites off the same db server so I don't think that's it 3. Max requests are set to 6, which is about standard for twin cpu, I could bump it for a test. 4. Long running pages, yes, but with fusebox it just tells you hey "index.cfm" took X seconds.. But I have it on anyway and no sign of long running pages except when fusebox reloads. -Original Message- From: Robert Munn [mailto:[EMAIL PROTECTED] Sent: Monday, November 28, 2005 7:34 PM To: CF-Talk Subject: Re: Cfmx 7.1 on 2.3k locking up once a day could be lots of things. have you tried running seefusion to get more info on the application server? have you checked your database to see if there are long-running queries? i had a client that kept killing their site, and it turned out that the whole problem was a mis-behaving query that would take down the sql server and force timeouts on the cf server. have you tried increasing your max request count in cf admin? try that. start logging long-running requests if you are not yet, and see what comes back. you might have a runaway page somewhere. >We just rolled out mx 7.1 on win 2.3k, Fusebox 4.1. It's a simple >application, just content management with 150-200k views a day. > >Without fail that mx server hangs in under 24 hours. Here is the >evidence we have found. > >I started app. Server at the command line: > >When I peep'd in while it was hung it was dumping out the following: > >java.lang.RuntimeException: Request timed out waiting for an available >thread to run. You may want to consider increasing the number of active >threads in the thread pool. > >This is repeated, again and again. > >NOTE: found some articles suggesting ideas on this, turn of debug >(done), make sure db is responding (appears to be). Adjust some of the >jrun.xml settings (appears to already be done in 7.1) > >Perf Mon. >went back to before the crash, the only thing odd is the data seems to >miss a beat nearing the crash and some data seems off the chart. But >when we examine logs we see no burst of traffic. I exported the perf. >Mon data and don't see any bursts. > >Did a stack dump while it was hung found a ton of these: > >"jrpp-108" prio=5 tid=0x04430d60 nid=0x1b08 in Object.wait() >[c4af000..c4afdb8] > at java.lang.Object.wait(Native Method) > - waiting on <0x172c0fa8> (a jrunx.scheduler.ThreadPool$Throttle) > at jrunx.scheduler.ThreadPool$Throttle.enter(ThreadPool.java:107) > - locked <0x172c0fa8> (a jrunx.scheduler.ThreadPool$Throttle) > at >jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.jav >a:423 >) > at >jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.ja >va:26 >4) > at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66) > > >Whole log: http://icc.getfused.com/out.txt > > >We have tried moving app. To a 2nd 7.1 and win 2.3k server.. Same >issue. We moved the db to another machine, same issue. No other sites >using the db seem to be having any issues. > >Anyone have any thoughts? Im about to toss in the towel and call adobe >and burn 500.00 clams. Any help would be apprecaited, if it saves me >the 500.00 then a shopping spree at Amazon could be in your future. > >Thanks > >gabe ~| Logware (www.logware.us): a new and convenient web-based time tracking application. Start tracking and documenting hours spent on a project or with a client with Logware today. Try it for free with a 15 day trial account. http://www.houseoffusion.com/banners/view.cfm?bannerid=67 Message: http://www.houseoffusion.com/lists.cfm/link=i:4:225482 Archives: http://www.houseoffusion.com/cf_lists/threads.cfm/4 Subscription: http://www.houseoffusion.com/lists.cfm/link=s:4 Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4 Donations & Support: http://www.houseoffusion.com/tiny.cfm/54
Re: Cfmx 7.1 on 2.3k locking up once a day
could be lots of things. have you tried running seefusion to get more info on the application server? have you checked your database to see if there are long-running queries? i had a client that kept killing their site, and it turned out that the whole problem was a mis-behaving query that would take down the sql server and force timeouts on the cf server. have you tried increasing your max request count in cf admin? try that. start logging long-running requests if you are not yet, and see what comes back. you might have a runaway page somewhere. >We just rolled out mx 7.1 on win 2.3k, Fusebox 4.1. It's a simple >application, just content management with 150-200k views a day. > >Without fail that mx server hangs in under 24 hours. Here is the evidence we >have found. > >I started app. Server at the command line: > >When I peep'd in while it was hung it was dumping out the following: > >java.lang.RuntimeException: Request timed out waiting for an available >thread to run. You may want to consider increasing the number of active >threads in the thread pool. > >This is repeated, again and again. > >NOTE: found some articles suggesting ideas on this, turn of debug (done), >make sure db is responding (appears to be). Adjust some of the jrun.xml >settings (appears to already be done in 7.1) > >Perf Mon. >went back to before the crash, the only thing odd is the data seems to miss >a beat nearing the crash and some data seems off the chart. But when we >examine logs we see no burst of traffic. I exported the perf. Mon data and >don't see any bursts. > >Did a stack dump while it was hung found a ton of these: > >"jrpp-108" prio=5 tid=0x04430d60 nid=0x1b08 in Object.wait() >[c4af000..c4afdb8] > at java.lang.Object.wait(Native Method) > - waiting on <0x172c0fa8> (a jrunx.scheduler.ThreadPool$Throttle) > at jrunx.scheduler.ThreadPool$Throttle.enter(ThreadPool.java:107) > - locked <0x172c0fa8> (a jrunx.scheduler.ThreadPool$Throttle) > at >jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:423 >) > at >jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:26 >4) > at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66) > > >Whole log: http://icc.getfused.com/out.txt > > >We have tried moving app. To a 2nd 7.1 and win 2.3k server.. Same issue. We >moved the db to another machine, same issue. No other sites using the db >seem to be having any issues. > >Anyone have any thoughts? Im about to toss in the towel and call adobe and >burn 500.00 clams. Any help would be apprecaited, if it saves me the 500.00 >then a shopping spree at Amazon could be in your future. > >Thanks > >gabe ~| Discover CFTicket - The leading ColdFusion Help Desk and Trouble Ticket application http://www.houseoffusion.com/banners/view.cfm?bannerid=48 Message: http://www.houseoffusion.com/lists.cfm/link=i:4:225479 Archives: http://www.houseoffusion.com/cf_lists/threads.cfm/4 Subscription: http://www.houseoffusion.com/lists.cfm/link=s:4 Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4 Donations & Support: http://www.houseoffusion.com/tiny.cfm/54
