[rt-users] AssetTracker crashes loading asset...raw horsepower solution?
On Fri, 7 Nov 2008, Curtis Bruneau wrote: I think i may have found a bit of a solution to the AT problem. we're running on 2 quad-core servers as frontends with 8gb memory each. the fault im seeing in AT is due to a timeout in apache waiting for its backend processes to respond from huge queries...so: IfModule mod_fcgid.c AddHandler fcgid-script .fcgi OutputBufferSize 128 IdleTimeout 600 ProcessLifeTime 3600 MaxProcessCount 8 DefaultMinClassProcessCount 3 DefaultMaxClassProcessCount 3 /IfModule IPCConnectTimeout 20 IPCCommTimeout 600 has resolved the AT issue from what im seeing. queries still take a long time, but thats just because RT is pulling lots of other data from mysql that im not certain is even pertanent to the asset/ticket at hand. could the speed of the query be increased by changing maxprocesscount to a higher value? im running into issues now with the population of a user rights page that includes 300-400 users...namely: RT::Group::Privileged Unimplemented in HTML::Mason::Commands. (/opt/rt3/share/html/Elements/ShowUserConcise line 52) any ideas? Date: Fri, 07 Nov 2008 12:52:04 -0500 From: Curtis Bruneau [EMAIL PROTECTED] To: John [EMAIL PROTECTED], rt-users@lists.bestpractical.com Subject: Re: [rt-users] AssetTracker crashes loading asset The queries would execute fine, the problem at least in my case is it tries to load the whole result set into memory causing oom-killer to go on rampage. Apache will peak out, to sort of help this situation I added a mem limit to the fastcgi handler which will basically die before things get really ugly. I'm able to see this query in the log, it has no limit clause so it really is getting everything. While doing a form of debug/trace on the apache/handler I'm also able to see it trying to load the full row times search results (table.*) into memory. This is all on a ticket display from a large search result. So unless you are loading it in memory as opposed to say a shell client output you won't run into memory problems. Best of luck with your situation, I have not been able to get any confirmation until now with you. I've produced somewhat detailed bug report but it appears to be somewhat isolated.. people with low tickets will never run into this problem. Curtis John wrote: curtis: ive played around re-executing the mysql queries related to the RT crash, and to no avail. they execute just fine. could this perhaps be a syslog issue like in RT? is there a seperate control for syslogging in AT? On Fri, 7 Nov 2008, Curtis Bruneau wrote: Date: Fri, 07 Nov 2008 10:43:48 -0500 From: Curtis Bruneau [EMAIL PROTECTED] To: John [EMAIL PROTECTED], rt-users@lists.bestpractical.com Subject: Re: [rt-users] AssetTracker crashes loading asset Sounds exactly like the issue I have, I think something is trying to get all those records, I tried to trace it but with no luck, I think it may be related to the back/next links on the ticket display page. It's checking each record for something, This is ok with small results but crashes with large sets. I really wish I could figure this one out, I get the occasional 500 and users are made aware to keep search results smaller. In my testing this affected 3.4.x 3.6.x and 3.8.x John wrote: well, just as i thought the RT slowness issue had been resolved, assettracker is now dying when loading a ticket out of a large list. example: clicking all assets enumerates a 9000 row list fine, but clicking on any element in the list will crunch for a while then die with a 500 error. smaller lists with say 1000-3000 rows are sluggish when loading an asset from the list, but succeed. [EMAIL PROTECTED] SDF Public Access UNIX System - http://sdf.lonestar.org ___ http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users Community help: http://wiki.bestpractical.com Commercial support: [EMAIL PROTECTED] Discover RT's hidden secrets with RT Essentials from O'Reilly Media. Buy a copy at http://rtbook.bestpractical.com [EMAIL PROTECTED] SDF Public Access UNIX System - http://sdf.lonestar.org [EMAIL PROTECTED] SDF Public Access UNIX System - http://sdf.lonestar.org ___ http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users Community help: http://wiki.bestpractical.com Commercial support: [EMAIL PROTECTED] Discover RT's hidden secrets with RT Essentials from O'Reilly Media. Buy a copy at http://rtbook.bestpractical.com
Re: [rt-users] AssetTracker crashes loading asset...raw horsepower solution?
How many assets do you have? Nobody has ever seen this issue before On Tue, Nov 11, 2008 at 12:57 PM, John [EMAIL PROTECTED] wrote: On Fri, 7 Nov 2008, Curtis Bruneau wrote: I think i may have found a bit of a solution to the AT problem. we're running on 2 quad-core servers as frontends with 8gb memory each. the fault im seeing in AT is due to a timeout in apache waiting for its backend processes to respond from huge queries...so: IfModule mod_fcgid.c AddHandler fcgid-script .fcgi OutputBufferSize 128 IdleTimeout 600 ProcessLifeTime 3600 MaxProcessCount 8 DefaultMinClassProcessCount 3 DefaultMaxClassProcessCount 3 /IfModule IPCConnectTimeout 20 IPCCommTimeout 600 has resolved the AT issue from what im seeing. queries still take a long time, but thats just because RT is pulling lots of other data from mysql that im not certain is even pertanent to the asset/ticket at hand. could the speed of the query be increased by changing maxprocesscount to a higher value? im running into issues now with the population of a user rights page that includes 300-400 users...namely: RT::Group::Privileged Unimplemented in HTML::Mason::Commands. (/opt/rt3/share/html/Elements/ShowUserConcise line 52) any ideas? Date: Fri, 07 Nov 2008 12:52:04 -0500 From: Curtis Bruneau [EMAIL PROTECTED] To: John [EMAIL PROTECTED], rt-users@lists.bestpractical.com Subject: Re: [rt-users] AssetTracker crashes loading asset The queries would execute fine, the problem at least in my case is it tries to load the whole result set into memory causing oom-killer to go on rampage. Apache will peak out, to sort of help this situation I added a mem limit to the fastcgi handler which will basically die before things get really ugly. I'm able to see this query in the log, it has no limit clause so it really is getting everything. While doing a form of debug/trace on the apache/handler I'm also able to see it trying to load the full row times search results (table.*) into memory. This is all on a ticket display from a large search result. So unless you are loading it in memory as opposed to say a shell client output you won't run into memory problems. Best of luck with your situation, I have not been able to get any confirmation until now with you. I've produced somewhat detailed bug report but it appears to be somewhat isolated.. people with low tickets will never run into this problem. Curtis John wrote: curtis: ive played around re-executing the mysql queries related to the RT crash, and to no avail. they execute just fine. could this perhaps be a syslog issue like in RT? is there a seperate control for syslogging in AT? On Fri, 7 Nov 2008, Curtis Bruneau wrote: Date: Fri, 07 Nov 2008 10:43:48 -0500 From: Curtis Bruneau [EMAIL PROTECTED] To: John [EMAIL PROTECTED], rt-users@lists.bestpractical.com Subject: Re: [rt-users] AssetTracker crashes loading asset Sounds exactly like the issue I have, I think something is trying to get all those records, I tried to trace it but with no luck, I think it may be related to the back/next links on the ticket display page. It's checking each record for something, This is ok with small results but crashes with large sets. I really wish I could figure this one out, I get the occasional 500 and users are made aware to keep search results smaller. In my testing this affected 3.4.x 3.6.x and 3.8.x John wrote: well, just as i thought the RT slowness issue had been resolved, assettracker is now dying when loading a ticket out of a large list. example: clicking all assets enumerates a 9000 row list fine, but clicking on any element in the list will crunch for a while then die with a 500 error. smaller lists with say 1000-3000 rows are sluggish when loading an asset from the list, but succeed. [EMAIL PROTECTED] SDF Public Access UNIX System - http://sdf.lonestar.org ___ http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users Community help: http://wiki.bestpractical.com Commercial support: [EMAIL PROTECTED] Discover RT's hidden secrets with RT Essentials from O'Reilly Media. Buy a copy at http://rtbook.bestpractical.com [EMAIL PROTECTED] SDF Public Access UNIX System - http://sdf.lonestar.org [EMAIL PROTECTED] SDF Public Access UNIX System - http://sdf.lonestar.org ___ http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users Community help: http://wiki.bestpractical.com Commercial support: [EMAIL PROTECTED] Discover RT's hidden secrets with RT Essentials from O'Reilly Media. Buy a copy at http://rtbook.bestpractical.com ___ http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users Community help:
Re: [rt-users] AssetTracker crashes loading asset...raw horsepower solution?
9,359 On Tue, 11 Nov 2008, Todd Chapman wrote: Date: Tue, 11 Nov 2008 13:04:14 -0500 From: Todd Chapman [EMAIL PROTECTED] To: John [EMAIL PROTECTED] Cc: Curtis Bruneau [EMAIL PROTECTED], rt-users@lists.bestpractical.com Subject: Re: [rt-users] AssetTracker crashes loading asset...raw horsepower solution? How many assets do you have? Nobody has ever seen this issue before On Tue, Nov 11, 2008 at 12:57 PM, John [EMAIL PROTECTED] wrote: On Fri, 7 Nov 2008, Curtis Bruneau wrote: I think i may have found a bit of a solution to the AT problem. we're running on 2 quad-core servers as frontends with 8gb memory each. the fault im seeing in AT is due to a timeout in apache waiting for its backend processes to respond from huge queries...so: IfModule mod_fcgid.c AddHandler fcgid-script .fcgi OutputBufferSize 128 IdleTimeout 600 ProcessLifeTime 3600 MaxProcessCount 8 DefaultMinClassProcessCount 3 DefaultMaxClassProcessCount 3 /IfModule IPCConnectTimeout 20 IPCCommTimeout 600 has resolved the AT issue from what im seeing. queries still take a long time, but thats just because RT is pulling lots of other data from mysql that im not certain is even pertanent to the asset/ticket at hand. could the speed of the query be increased by changing maxprocesscount to a higher value? im running into issues now with the population of a user rights page that includes 300-400 users...namely: RT::Group::Privileged Unimplemented in HTML::Mason::Commands. (/opt/rt3/share/html/Elements/ShowUserConcise line 52) any ideas? Date: Fri, 07 Nov 2008 12:52:04 -0500 From: Curtis Bruneau [EMAIL PROTECTED] To: John [EMAIL PROTECTED], rt-users@lists.bestpractical.com Subject: Re: [rt-users] AssetTracker crashes loading asset The queries would execute fine, the problem at least in my case is it tries to load the whole result set into memory causing oom-killer to go on rampage. Apache will peak out, to sort of help this situation I added a mem limit to the fastcgi handler which will basically die before things get really ugly. I'm able to see this query in the log, it has no limit clause so it really is getting everything. While doing a form of debug/trace on the apache/handler I'm also able to see it trying to load the full row times search results (table.*) into memory. This is all on a ticket display from a large search result. So unless you are loading it in memory as opposed to say a shell client output you won't run into memory problems. Best of luck with your situation, I have not been able to get any confirmation until now with you. I've produced somewhat detailed bug report but it appears to be somewhat isolated.. people with low tickets will never run into this problem. Curtis John wrote: curtis: ive played around re-executing the mysql queries related to the RT crash, and to no avail. they execute just fine. could this perhaps be a syslog issue like in RT? is there a seperate control for syslogging in AT? On Fri, 7 Nov 2008, Curtis Bruneau wrote: Date: Fri, 07 Nov 2008 10:43:48 -0500 From: Curtis Bruneau [EMAIL PROTECTED] To: John [EMAIL PROTECTED], rt-users@lists.bestpractical.com Subject: Re: [rt-users] AssetTracker crashes loading asset Sounds exactly like the issue I have, I think something is trying to get all those records, I tried to trace it but with no luck, I think it may be related to the back/next links on the ticket display page. It's checking each record for something, This is ok with small results but crashes with large sets. I really wish I could figure this one out, I get the occasional 500 and users are made aware to keep search results smaller. In my testing this affected 3.4.x 3.6.x and 3.8.x John wrote: well, just as i thought the RT slowness issue had been resolved, assettracker is now dying when loading a ticket out of a large list. example: clicking all assets enumerates a 9000 row list fine, but clicking on any element in the list will crunch for a while then die with a 500 error. smaller lists with say 1000-3000 rows are sluggish when loading an asset from the list, but succeed. [EMAIL PROTECTED] SDF Public Access UNIX System - http://sdf.lonestar.org ___ http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users Community help: http://wiki.bestpractical.com Commercial support: [EMAIL PROTECTED] Discover RT's hidden secrets with RT Essentials from O'Reilly Media. Buy a copy at http://rtbook.bestpractical.com [EMAIL PROTECTED] SDF Public Access UNIX System - http://sdf.lonestar.org [EMAIL PROTECTED] SDF Public Access UNIX System - http://sdf.lonestar.org ___ http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users Community help: http://wiki.bestpractical.com Commercial support: [EMAIL PROTECTED] Discover RT's hidden secrets
