RE: Performance tuning in complex environment
Avnish, The book goes through this a bit on p61. There are a few tools out there, including three from Oracle. As I mention in the book, I use our own Hotsos Profiler, described at www.hotsos.com/products/profiler.html. Cary Millsap Hotsos Enterprises, Ltd. http://www.hotsos.com Upcoming events: - Performance Diagnosis 101: 12/16 Detroit, 1/27 Atlanta - SQL Optimization 101: 2/16 Dallas - Hotsos Symposium 2004: March 7-10 Dallas - Visit www.hotsos.com for schedule details... -Original Message- [EMAIL PROTECTED] Sent: Friday, December 12, 2003 3:44 PM To: Multiple recipients of list ORACLE-L I will try to get the output of v$system_event and will send it you guys. In the mean time I have more question.. I am reading Cary's 'Optimizing Oracle Performance Book'. I am half way thru and over looked rest of the chapters but didnt find an easy way to analyze thousands of lines trace file. I am not very good in analyzing big trace files and wondering how you guys analyze do that. Do you do it manually or use any tool to get summarized report. I didnt see anything in that book. I am also planning to take class from HotSos in Feb, 2004 in Seattle to see if that will help. I really appreciate all of your input. Thanks -Original Message- Jared Still Sent: Thursday, December 11, 2003 10:04 PM To: Multiple recipients of list ORACLE-L The wholesale system wide collection of timing data is not generally a good way to go about trouble shooting performance issues. You need to pick a process, collect the timing data for that process, and *only* that process, diagnose where the most time is being spent, and determine what can be done to speed it up. This in a nutshell is the basis of Cary's book, at least per my reading of it. Always try to fix the stuff with the biggest payoff. It could be a SQL statement, it could be a misconfigured or malfunctioning network card. It could be that a developer is filling a temporary table with lots of data during a transaction, then deleting the data and doing it over and over again, all the while doing full table scans. FTS is expensive when you want to retrieve 3 rows from a temp table with 500 meg of extents in it. Just for grins though, how about running this script and posting the output for us? Sometimes you get lucky, and something may appear really out of whack. No guarantees though. Troubleshooting system performance problems takes more than an email. HTH Jared = col event format a35 head 'EVENT NAME' col total_waits format 999,999,999 head TOTAL|WAITS col total_timeouts format 999,999,999 head TOTAL|TIMEOUTS col time_waited format 999,999,999 head TIME|WAITED|SECONDS col average_wait format 9 head AVG|WAIT|100ths set line 150 set trimspool on select event, total_waits, total_timeouts, time_waited/100 time_waited, average_wait from v$system_event order by time_waited / On Thu, 2003-12-11 at 10:34, [EMAIL PROTECTED] wrote: Hello Everyone, I am trying to get some help/suggestions reg. how to troubleshoot performance issues. Little back ground about our environment. Its third party application (Logician) from GE. There are total 11 databases, all on oracle 8174 H-UX 11i in cluster environment. All the databases are on EMC Symmetrix using 6 disks. All the clients are connecting to database thru Citrix terminal servers. In last one year we spend lots of time/money in tuning databases, replacing Citrix servers but end result is same. I was wondering if anybody out there has ran into same kind of situation. Our (DBAs) guess is the disk layout is not optimal but we also dont have any data to prove that disks are the bottleneck. Is there any way to collect these kinds of stats in Oracle. We aren't getting much help from our SAN administrator. DISCLAIMER: This message is intended for the sole use of the individual to whom it is addressed, and may contain information that is privileged, confidential and exempt from disclosure under applicable law. If you are not the addressee you are hereby notified that you may not use, copy, disclose, or distribute to anyone the message or any information contained in the message. If you have received this message in error, please immediately advise the sender by reply email and delete this message. -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: [EMAIL PROTECTED] INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538-5051 http://www.fatcity.com San Diego, California-- Mailing list and web hosting services - To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you
Re: Performance tuning in complex environment
All, This sounds wy too familiar to me. My (blind) guess is that sql*net round trips is killing performance. System-wide could indicate this, but, as Jared states, trace out a specific session, and grab the session-specific info from v$sesstat, before and after. We brute forced the issue of sqlnet round tripsin the past by having the site upgrade the Citrix Servers from fast ethernet to gigabit network cards. Most new servers these days in the intel space ship with 2 integrated gigabit NICs, is just a matter of having GigE switched ports available. If you're moving too much data between client and server, increasing the SDU could help, but your mileage will vary. PdJared Still [EMAIL PROTECTED] wrote: The wholesale system wide collection of timing data is not generallya good way to go about trouble shooting performance issues.You need to pick a process, collect the timing data for that process,and *only* that process, diagnose where the most time is being spent,and determine what can be done to speed it up.This in a nutshell is the basis of Cary's book, at least per my reading of it.Always try to fix the stuff with the biggest payoff.It could be a SQL statement, it could be a misconfigured ormalfunctioning network card. It could be that a developer is filling a temporary table with lots of data during a transaction,then deleting the data and doing it over and over again, all thewhile doing full table scans. FTS is expensive when you want to retrieve 3 rows from a temp table with 500 meg of extents in it.Just for grins though, how about running this script and postingthe output for us? Sometimes you get lucky, and something mayappear really out of whack. No guarantees though. Troubleshootingsystem performance problems takes more than an email.HTHJared=col event format a35 head 'EVENT NAME'col total_waits format 999,999,999 head "TOTAL|WAITS"col total_timeouts format 999,999,999 head "TOTAL|TIMEOUTS"col time_waited format 999,999,999 head "TIME|WAITED|SECONDS"col average_wait format 9 head "AVG|WAIT|100ths"set line 150set trimspool onselectevent,total_waits,total_timeouts,time_waited/100 time_waited,average_waitfrom v$system_eventorder by time_waited/On Thu, 2003-12-11 at 10:34, [EMAIL PROTECTED] wrote: Hello Everyon! e, I am trying to get some help/suggestions reg. how to troubleshoot performance issues. Little back ground about our environment. Its third party application (Logician) from GE. There are total 11 databases, all on oracle 8174 H-UX 11i in cluster environment. All the databases are on EMC Symmetrix using 6 disks. All the clients are connecting to database thru Citrix terminal servers. In last one year we spend lots of time/money in tuning databases, replacing Citrix servers but end result is same. I was wondering if anybody out there has ran into same kind of situation. Our (DBAs) guess is the disk layout is not optimal but we also dont have any data to prove that disks are the bottleneck. Is there any way to collect these kinds of stats in Oracle. We aren't getting much help from our SAN administrator.DISCLAIMER: This message is intended for the sole use of the individual to whom it is addressed, and may contain information that is privileged, confidential and exempt from disclosure under applicable law. If you are not the addressee you are hereby notified that you may not use, copy, disclose, or distribute to anyone the message or any information contained in the message. If you have received this message in error, please immediately advise the sender by reply email and delete this message. -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: <[EMAIL PROTECTED] INET: [EMAIL PROTECTED] Fat City Network Services -- 858-538-5051 http://www.fatcity.com San Diego, California -- Mailing list and web hosting services - To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a l! ine containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing). -- Please see the official ORACLE-L FAQ: http://www.orafaq.net-- Author: Jared StillINET: [EMAIL PROTECTED]Fat City Network Services -- 858-538-5051 http://www.fatcity.comSan Diego, California -- Mailing list and web hosting services-To REMOVE yourself from this mailing list, send an E-Mail messageto: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and inthe message BODY, include a line containing: UNSUB ORACLE-L(or the name of mailing list you want to be removed from). You mayalso send the HELP command for other information (like subscribing). Do you Yahoo!? Free Pop-Up Blocker - Get
RE: Performance tuning in complex environment
I will try to get the output of v$system_event and will send it you guys. In the mean time I have more question.. I am reading Cary's 'Optimizing Oracle Performance Book'. I am half way thru and over looked rest of the chapters but didnt find an easy way to analyze thousands of lines trace file. I am not very good in analyzing big trace files and wondering how you guys analyze do that. Do you do it manually or use any tool to get summarized report. I didnt see anything in that book. I am also planning to take class from HotSos in Feb, 2004 in Seattle to see if that will help. I really appreciate all of your input. Thanks -Original Message- Jared Still Sent: Thursday, December 11, 2003 10:04 PM To: Multiple recipients of list ORACLE-L The wholesale system wide collection of timing data is not generally a good way to go about trouble shooting performance issues. You need to pick a process, collect the timing data for that process, and *only* that process, diagnose where the most time is being spent, and determine what can be done to speed it up. This in a nutshell is the basis of Cary's book, at least per my reading of it. Always try to fix the stuff with the biggest payoff. It could be a SQL statement, it could be a misconfigured or malfunctioning network card. It could be that a developer is filling a temporary table with lots of data during a transaction, then deleting the data and doing it over and over again, all the while doing full table scans. FTS is expensive when you want to retrieve 3 rows from a temp table with 500 meg of extents in it. Just for grins though, how about running this script and posting the output for us? Sometimes you get lucky, and something may appear really out of whack. No guarantees though. Troubleshooting system performance problems takes more than an email. HTH Jared = col event format a35 head 'EVENT NAME' col total_waits format 999,999,999 head TOTAL|WAITS col total_timeouts format 999,999,999 head TOTAL|TIMEOUTS col time_waited format 999,999,999 head TIME|WAITED|SECONDS col average_wait format 9 head AVG|WAIT|100ths set line 150 set trimspool on select event, total_waits, total_timeouts, time_waited/100 time_waited, average_wait from v$system_event order by time_waited / On Thu, 2003-12-11 at 10:34, [EMAIL PROTECTED] wrote: Hello Everyone, I am trying to get some help/suggestions reg. how to troubleshoot performance issues. Little back ground about our environment. Its third party application (Logician) from GE. There are total 11 databases, all on oracle 8174 H-UX 11i in cluster environment. All the databases are on EMC Symmetrix using 6 disks. All the clients are connecting to database thru Citrix terminal servers. In last one year we spend lots of time/money in tuning databases, replacing Citrix servers but end result is same. I was wondering if anybody out there has ran into same kind of situation. Our (DBAs) guess is the disk layout is not optimal but we also dont have any data to prove that disks are the bottleneck. Is there any way to collect these kinds of stats in Oracle. We aren't getting much help from our SAN administrator. DISCLAIMER: This message is intended for the sole use of the individual to whom it is addressed, and may contain information that is privileged, confidential and exempt from disclosure under applicable law. If you are not the addressee you are hereby notified that you may not use, copy, disclose, or distribute to anyone the message or any information contained in the message. If you have received this message in error, please immediately advise the sender by reply email and delete this message. -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: [EMAIL PROTECTED] INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538-5051 http://www.fatcity.com San Diego, California-- Mailing list and web hosting services - To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing). -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: Jared Still INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538-5051 http://www.fatcity.com San Diego, California-- Mailing list and web hosting services - To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY,
RE: Performance tuning in complex environment
On Fri, 12 Dec 2003 [EMAIL PROTECTED] wrote: I will try to get the output of v$system_event and will send it you guys. In the mean time I have more question.. I am reading Cary's 'Optimizing Oracle Performance Book'. I am half way thru and over looked rest of the chapters but didnt find an easy way to analyze thousands of lines trace file. I am not very good in analyzing big trace files and wondering how you guys analyze do that. Do you do it manually or use any tool to get summarized report. I didnt see anything in that book. I am also planning to take class from HotSos in Feb, 2004 in Seattle to see if that will help. I really appreciate all of your input. I brewed up my own (still semi-baked) profiler in Perl. The tricky part is sorting out the recursive dependencies between db calls, but the rest is pretty straightforward, and Perl makes chewing up the trace file a snap. Currently it works 'well enough' for basic traces (w/o a lot of recursive calls) to give me a decent picture of what a session is doing. -- Dan Daniel Hanks - Systems/Database Administrator About Inc., Web Services Division -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: Daniel Hanks INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538-5051 http://www.fatcity.com San Diego, California-- Mailing list and web hosting services - To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing).
Performance tuning in complex environment
Hello Everyone, I am trying to get some help/suggestions reg. how to troubleshoot performance issues. Little back ground about our environment. Its third party application (Logician) from GE. There are total 11 databases, all on oracle 8174 H-UX 11i in cluster environment. All the databases are on EMC Symmetrix using 6 disks. All the clients are connecting to database thru Citrix terminal servers. In last one year we spend lots of time/money in tuning databases, replacing Citrix servers but end result is same. I was wondering if anybody out there has ran into same kind of situation. Our (DBAs) guess is the disk layout is not optimal but we also dont have any data to prove that disks are the bottleneck. Is there any way to collect these kinds of stats in Oracle. We aren't getting much help from our SAN administrator. DISCLAIMER: This message is intended for the sole use of the individual to whom it is addressed, and may contain information that is privileged, confidential and exempt from disclosure under applicable law. If you are not the addressee you are hereby notified that you may not use, copy, disclose, or distribute to anyone the message or any information contained in the message. If you have received this message in error, please immediately advise the sender by reply email and delete this message. -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: [EMAIL PROTECTED] INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538-5051 http://www.fatcity.com San Diego, California-- Mailing list and web hosting services - To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing).
RE: Performance tuning in complex environment
Ummm ... what was the problem that prompted you guys to replace citrix servers? Raj Rajendra dot Jamadagni at nospamespn dot com All Views expressed in this email are strictly personal. QOTD: Any clod can have facts, having an opinion is an art ! -Original Message- [mailto:[EMAIL PROTECTED] Sent: Thursday, December 11, 2003 1:35 PM To: Multiple recipients of list ORACLE-L Hello Everyone, I am trying to get some help/suggestions reg. how to troubleshoot performance issues. Little back ground about our environment. Its third party application (Logician) from GE. There are total 11 databases, all on oracle 8174 H-UX 11i in cluster environment. All the databases are on EMC Symmetrix using 6 disks. All the clients are connecting to database thru Citrix terminal servers. In last one year we spend lots of time/money in tuning databases, replacing Citrix servers but end result is same. I was wondering if anybody out there has ran into same kind of situation. Our (DBAs) guess is the disk layout is not optimal but we also dont have any data to prove that disks are the bottleneck. Is there any way to collect these kinds of stats in Oracle. We aren't getting much help from our SAN administrator. DISCLAIMER: This message is intended for the sole use of the individual to whom it is addressed, and may contain information that is privileged, confidential and exempt from disclosure under applicable law. If you are not the addressee you are hereby notified that you may not use, copy, disclose, or distribute to anyone the message or any information contained in the message. If you have received this message in error, please immediately advise the sender by reply email and delete this message. -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: [EMAIL PROTECTED] INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538-5051 http://www.fatcity.com San Diego, California-- Mailing list and web hosting services - To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing). ** This e-mail message is confidential, intended only for the named recipient(s) above and may contain information that is privileged, attorney work product or exempt from disclosure under applicable law. If you have received this message in error, or are not the named recipient(s), please immediately notify corporate MIS at (860) 766-2000 and delete this e-mail message from your computer, Thank you. **5 -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: Jamadagni, Rajendra INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538-5051 http://www.fatcity.com San Diego, California-- Mailing list and web hosting services - To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing).
Re: Performance tuning in complex environment
DBAs should never 'guess' about performance. If they are guessing you need new DBAs. They should be running statspacks, sql trace, and looking at timing data. Its too much to explain in an email. Fire your DBAs and find people who dont 'guess'. How much are you paying these guys? From: [EMAIL PROTECTED] Date: 2003/12/11 Thu PM 01:34:52 EST To: Multiple recipients of list ORACLE-L [EMAIL PROTECTED] Subject: Performance tuning in complex environment Hello Everyone, I am trying to get some help/suggestions reg. how to troubleshoot performance issues. Little back ground about our environment. Its third party application (Logician) from GE. There are total 11 databases, all on oracle 8174 H-UX 11i in cluster environment. All the databases are on EMC Symmetrix using 6 disks. All the clients are connecting to database thru Citrix terminal servers. In last one year we spend lots of time/money in tuning databases, replacing Citrix servers but end result is same. I was wondering if anybody out there has ran into same kind of situation. Our (DBAs) guess is the disk layout is not optimal but we also dont have any data to prove that disks are the bottleneck. Is there any way to collect these kinds of stats in Oracle. We aren't getting much help from our SAN administrator. DISCLAIMER: This message is intended for the sole use of the individual to whom it is addressed, and may contain information that is privileged, confidential and exempt from disclosure under applicable law. If you are not the addressee you are hereby notified that you may not use, copy, disclose, or distribute to anyone the message or any information contained in the message. If you have received this message in error, please immediately advise the sender by reply email and delete this message. -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: [EMAIL PROTECTED] INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538-5051 http://www.fatcity.com San Diego, California-- Mailing list and web hosting services - To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing). -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: [EMAIL PROTECTED] INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538-5051 http://www.fatcity.com San Diego, California-- Mailing list and web hosting services - To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing).
RE: Performance tuning in complex environment
Not really sure what happened and why we decided to that. I was involved in the beginning of project and remembered that PM was mentioning about talking to another Logician client who were facing same issues. -Original Message- Jamadagni, Rajendra Sent: Thursday, December 11, 2003 10:55 AM To: Multiple recipients of list ORACLE-L Ummm ... what was the problem that prompted you guys to replace citrix servers? Raj Rajendra dot Jamadagni at nospamespn dot com All Views expressed in this email are strictly personal. QOTD: Any clod can have facts, having an opinion is an art ! -Original Message- [mailto:[EMAIL PROTECTED] Sent: Thursday, December 11, 2003 1:35 PM To: Multiple recipients of list ORACLE-L Hello Everyone, I am trying to get some help/suggestions reg. how to troubleshoot performance issues. Little back ground about our environment. Its third party application (Logician) from GE. There are total 11 databases, all on oracle 8174 H-UX 11i in cluster environment. All the databases are on EMC Symmetrix using 6 disks. All the clients are connecting to database thru Citrix terminal servers. In last one year we spend lots of time/money in tuning databases, replacing Citrix servers but end result is same. I was wondering if anybody out there has ran into same kind of situation. Our (DBAs) guess is the disk layout is not optimal but we also dont have any data to prove that disks are the bottleneck. Is there any way to collect these kinds of stats in Oracle. We aren't getting much help from our SAN administrator. DISCLAIMER: This message is intended for the sole use of the individual to whom it is addressed, and may contain information that is privileged, confidential and exempt from disclosure under applicable law. If you are not the addressee you are hereby notified that you may not use, copy, disclose, or distribute to anyone the message or any information contained in the message. If you have received this message in error, please immediately advise the sender by reply email and delete this message. -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: [EMAIL PROTECTED] INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538-5051 http://www.fatcity.com San Diego, California-- Mailing list and web hosting services - To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing). ** This e-mail message is confidential, intended only for the named recipient(s) above and may contain information that is privileged, attorney work product or exempt from disclosure under applicable law. If you have received this message in error, or are not the named recipient(s), please immediately notify corporate MIS at (860) 766-2000 and delete this e-mail message from your computer, Thank you. **5 -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: Jamadagni, Rajendra INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538-5051 http://www.fatcity.com San Diego, California-- Mailing list and web hosting services - To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing). DISCLAIMER: This message is intended for the sole use of the individual to whom it is addressed, and may contain information that is privileged, confidential and exempt from disclosure under applicable law. If you are not the addressee you are hereby notified that you may not use, copy, disclose, or distribute to anyone the message or any information contained in the message. If you have received this message in error, please immediately advise the sender by reply email and delete this message. -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: [EMAIL PROTECTED] INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538-5051 http://www.fatcity.com San Diego, California-- Mailing list and web hosting services - To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT
RE: Performance tuning in complex environment
Thanks I asked because we also use Citrix and so far we never had a problem related to Citrix, only problems we had were inefficient coding and oracle bugs, nothing related to HW/disk/WTS etc. The only problem initially with Citrix was configuring client printers, but our guys figured it out as well. Raj Rajendra dot Jamadagni at nospamespn dot com All Views expressed in this email are strictly personal. QOTD: Any clod can have facts, having an opinion is an art ! -Original Message- [mailto:[EMAIL PROTECTED] Sent: Thursday, December 11, 2003 2:29 PM To: Multiple recipients of list ORACLE-L Not really sure what happened and why we decided to that. I was involved in the beginning of project and remembered that PM was mentioning about talking to another Logician client who were facing same issues. ** This e-mail message is confidential, intended only for the named recipient(s) above and may contain information that is privileged, attorney work product or exempt from disclosure under applicable law. If you have received this message in error, or are not the named recipient(s), please immediately notify corporate MIS at (860) 766-2000 and delete this e-mail message from your computer, Thank you. **5 -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: Jamadagni, Rajendra INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538-5051 http://www.fatcity.com San Diego, California-- Mailing list and web hosting services - To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing).
RE: Performance tuning in complex environment
Oh I've run into THIS beforeyou are in a sticky technical AND political situation I am sure. It is really not that complex. I'll bet they (your DBAs) have already been told you that the app is horribly designed and it was a mistake and that the hardware is under powered canned dataserver install was poorly laid out. The vendor says everything is fine and are basically silent but willing to send in expensive consultants. You don't trust you DBA's. They are pissed at you and the vendor. (imagine that) Now you are stuck with hearing exactly what you didn't want to hear. The DBA's (If they are Oracle DBA's) have done everything they can and now you are on a fishing expedition because you don't trust your DBAs...you probably have been glazing over when ever you hear them talk about issues with this system...because of that they have washed their hands of it. They can't help you if you don't all sit down and listen and respect each other. I'll put got $20 bucks that if I came in and sat down with them for ten minutes I'd find out they were sharp competent people who have been given the crap because of this lousy application. So why don't you send your DBAs in here and let them tell the folks on this board what they have done thus far and maybe there will be some good come out if it. Also expect that apps purchased two years ago are going to royally suck for the most part as the software industry was a giant scam at that time. Frankly if you have multiple users on citrix attaching to oracle expect poor performance. Think about it. A Citrix server with dozens of users using the same resources to display an entire desktop across the networkHave you determined if it is app performance or oracle? What this tells me is this is a client server app licensed to sacrifice cost versus performance. You bought the Citrix version to save money no doubt. Now you are stuck. Either way you have fell for the sell them the sizzle give them the bacon later software bait and switch. Do queries typically run faster on a stand alone install of the client than the Citrix? Do the queries run by the app run the same or faster when executed through say sqlplus... How far flung is this Citrix app do you have remote users? In remote offices? What kind of bandwidth do you have? How much data is pumped between the database and the client in a typical connection? Sorry if a came off a bit frank, but If I were you I would go back and listen to your DBA's again. This time more closely because I am sure you left a lot of other information out. Stop listening to the Citrix and Logician marketing hype and listen to your people. Software vendors could care less how well your app runs as long as they get their stinking money. They are in the business first to SELL SOFTWARE, always remember that. You have to get everybody working together, accept the situation, set some new expectations and hold the vendor partially accountable if you are going to move forward on this problem. If you indeed can say: When I push this button it take ten minutes to get the data and the same query takes just as long from sqlplus then you POSSIBLY have a database issue, a program design issue or both. Good luck. Brad Odland -Original Message- [mailto:[EMAIL PROTECTED] Sent: Thursday, December 11, 2003 12:35 PM To: Multiple recipients of list ORACLE-L Hello Everyone, I am trying to get some help/suggestions reg. how to troubleshoot performance issues. Little back ground about our environment. Its third party application (Logician) from GE. There are total 11 databases, all on oracle 8174 H-UX 11i in cluster environment. All the databases are on EMC Symmetrix using 6 disks. All the clients are connecting to database thru Citrix terminal servers. In last one year we spend lots of time/money in tuning databases, replacing Citrix servers but end result is same. I was wondering if anybody out there has ran into same kind of situation. Our (DBAs) guess is the disk layout is not optimal but we also dont have any data to prove that disks are the bottleneck. Is there any way to collect these kinds of stats in Oracle. We aren't getting much help from our SAN administrator. DISCLAIMER: This message is intended for the sole use of the individual to whom it is addressed, and may contain information that is privileged, confidential and exempt from disclosure under applicable law. If you are not the addressee you are hereby notified that you may not use, copy, disclose, or distribute to anyone the message or any information contained in the message. If you have received this message in error, please immediately advise the sender by reply email and delete this message. -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: [EMAIL PROTECTED] INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538-5051 http://www.fatcity.com San Diego, California-- Mailing list and web hosting services
RE: Performance tuning in complex environment
OK. Before we go blaming Oracle DB, you need to look at the entire picture. 1. Are other applications within the environment affected by slow performance? 2. What other apps are running on the network? 3. Have any network-related diagnostics been performed to ensure that Network bottlenecks are not causing the issue. 4. What is your disk configuration look like? Mirroring/Striping, etc. 5. How large/small are the transactions? Thank You Stephen P. Karniotis Technical Alliance Manager Compuware Corporation Direct: (313) 227-4350 Mobile: (248) 408-2918 Email: [EMAIL PROTECTED] Web:www.compuware.com -Original Message- [EMAIL PROTECTED] Sent: Thursday, December 11, 2003 2:29 PM To: Multiple recipients of list ORACLE-L Subject:RE: Performance tuning in complex environment Not really sure what happened and why we decided to that. I was involved in the beginning of project and remembered that PM was mentioning about talking to another Logician client who were facing same issues. -Original Message- Jamadagni, Rajendra Sent: Thursday, December 11, 2003 10:55 AM To: Multiple recipients of list ORACLE-L Ummm ... what was the problem that prompted you guys to replace citrix servers? Raj Rajendra dot Jamadagni at nospamespn dot com All Views expressed in this email are strictly personal. QOTD: Any clod can have facts, having an opinion is an art ! -Original Message- [mailto:[EMAIL PROTECTED] Sent: Thursday, December 11, 2003 1:35 PM To: Multiple recipients of list ORACLE-L Hello Everyone, I am trying to get some help/suggestions reg. how to troubleshoot performance issues. Little back ground about our environment. Its third party application (Logician) from GE. There are total 11 databases, all on oracle 8174 H-UX 11i in cluster environment. All the databases are on EMC Symmetrix using 6 disks. All the clients are connecting to database thru Citrix terminal servers. In last one year we spend lots of time/money in tuning databases, replacing Citrix servers but end result is same. I was wondering if anybody out there has ran into same kind of situation. Our (DBAs) guess is the disk layout is not optimal but we also dont have any data to prove that disks are the bottleneck. Is there any way to collect these kinds of stats in Oracle. We aren't getting much help from our SAN administrator. DISCLAIMER: This message is intended for the sole use of the individual to whom it is addressed, and may contain information that is privileged, confidential and exempt from disclosure under applicable law. If you are not the addressee you are hereby notified that you may not use, copy, disclose, or distribute to anyone the message or any information contained in the message. If you have received this message in error, please immediately advise the sender by reply email and delete this message. -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: [EMAIL PROTECTED] INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538-5051 http://www.fatcity.com San Diego, California-- Mailing list and web hosting services - To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing). ** This e-mail message is confidential, intended only for the named recipient(s) above and may contain information that is privileged, attorney work product or exempt from disclosure under applicable law. If you have received this message in error, or are not the named recipient(s), please immediately notify corporate MIS at (860) 766-2000 and delete this e-mail message from your computer, Thank you. **5 -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: Jamadagni, Rajendra INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538-5051 http://www.fatcity.com San Diego, California-- Mailing list and web hosting services - To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing). DISCLAIMER: This message is intended for the sole use of the individual to whom it is addressed, and may contain information
RE: Performance tuning in complex environment
Avnish - Since nobody has mentioned it yet (my posts arrive late, so probably will by the time this appears), get Cary Millsap's book Optimizing Oracle Performance http://search.barnesandnoble.com/textbooks/booksearch/isbnInquiry.asp?userid =6WIANMIL0Hisbn=059600527XTXT=Yitm=1 His methods sound exactly suited to your issues. Dennis Williams DBA Lifetouch, Inc. [EMAIL PROTECTED] -Original Message- [mailto:[EMAIL PROTECTED] Sent: Thursday, December 11, 2003 12:35 PM To: Multiple recipients of list ORACLE-L Hello Everyone, I am trying to get some help/suggestions reg. how to troubleshoot performance issues. Little back ground about our environment. Its third party application (Logician) from GE. There are total 11 databases, all on oracle 8174 H-UX 11i in cluster environment. All the databases are on EMC Symmetrix using 6 disks. All the clients are connecting to database thru Citrix terminal servers. In last one year we spend lots of time/money in tuning databases, replacing Citrix servers but end result is same. I was wondering if anybody out there has ran into same kind of situation. Our (DBAs) guess is the disk layout is not optimal but we also dont have any data to prove that disks are the bottleneck. Is there any way to collect these kinds of stats in Oracle. We aren't getting much help from our SAN administrator. DISCLAIMER: This message is intended for the sole use of the individual to whom it is addressed, and may contain information that is privileged, confidential and exempt from disclosure under applicable law. If you are not the addressee you are hereby notified that you may not use, copy, disclose, or distribute to anyone the message or any information contained in the message. If you have received this message in error, please immediately advise the sender by reply email and delete this message. -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: [EMAIL PROTECTED] INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538-5051 http://www.fatcity.com San Diego, California-- Mailing list and web hosting services - To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing). -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: DENNIS WILLIAMS INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538-5051 http://www.fatcity.com San Diego, California-- Mailing list and web hosting services - To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing).
RE: Performance tuning in complex environment
Reminds me of the day when a third-party developed app (main batch program) ran *very* slowly - the user department went out and bought this app and server on their own without IT's blessing or support (a different story). Dialogue below: Third-party Developer (TPD): This same batch program which runs 1hr 30 min on your box completes under 30 min at our Office with *your* data. We suggest obtaining an IBM S80 because it is 3 times faster than your current box (IBM On-site person: Yes! Yes!!!) User Department Manager (UDM): Ok - we have a $100,000 budget for this - lets go out and buy this h/w (We need to go through IT for this purchase) My Manager, when approached with this issue (MM): I know your TPD has this view, but can my Sr. DBA look at this problem? UDM: Ok, but I doubt anything can be done since my TPD says so... TPD: Hey, your DBA can't mess with our code! Sr.DBA (Me!): Ok - let's take a look at V$SYSTEM_EVENT, V$SESSION_EVENT and V$SESSION_WAIT when your program runs... Me: Hey - what's this session doing with 'SQL*Net Message from dblink'? This is the top wait (more than 99% of TIME_WAITED in V$SESSION_EVENT) TPD: Yeah - we have a view that makes a call to your employee table sitting on your prod box to fetch the Emp name, once for every row in the loop (1000s of rows, 3300 rows a pop) Me: Haven't you guys heard of Replicated Tables? TPD: What's that? Me: (after creating a local copy and replacing the view with an indexed table) Run your program now... TPD: Hey - it finished in 5 minutes!!! We don't need to buy any other box! UDM: I like that!!! Thanks!! MM: Well done - I knew my DBA could do it! (IBM On-site person: [EMAIL PROTECTED]@#*()+__@) Me: (Hitting myself on the head, and thinking to myself: I should have asked for just 1% of the $$ that would have otherwise been unnecessarily spent on that great big H/w box :( Moral of the story: (a) Never ass*u*me anything - ask for stats to prove any 'assumption' (b) Get the right tools to determine the problem area (and use it correctly) Afterthought (c) - Follow Gary Goodman's principle: Ask for 10% of the $$ allocated for the h/w that would have otherwise been spent on *trying* to solve the problem by throwing h/w at it! (Cary - correct me if I erred here!) John Kanagaraj DB Soft Inc Phone: 408-970-7002 (W) Grace - Getting something we do NOT deserve Mercy - NOT getting something we DO deserve Click on 'http://www.needhim.org' for Grace and Mercy that is freely available! ** The opinions and facts contained in this message are entirely mine and do not reflect those of my employer or customers ** -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Sent: Thursday, December 11, 2003 11:29 AM To: Multiple recipients of list ORACLE-L Subject: RE: Performance tuning in complex environment Not really sure what happened and why we decided to that. I was involved in the beginning of project and remembered that PM was mentioning about talking to another Logician client who were facing same issues. -Original Message- Jamadagni, Rajendra Sent: Thursday, December 11, 2003 10:55 AM To: Multiple recipients of list ORACLE-L Ummm ... what was the problem that prompted you guys to replace citrix servers? Raj --- - Rajendra dot Jamadagni at nospamespn dot com All Views expressed in this email are strictly personal. QOTD: Any clod can have facts, having an opinion is an art ! -Original Message- [mailto:[EMAIL PROTECTED] Sent: Thursday, December 11, 2003 1:35 PM To: Multiple recipients of list ORACLE-L Hello Everyone, I am trying to get some help/suggestions reg. how to troubleshoot performance issues. Little back ground about our environment. Its third party application (Logician) from GE. There are total 11 databases, all on oracle 8174 H-UX 11i in cluster environment. All the databases are on EMC Symmetrix using 6 disks. All the clients are connecting to database thru Citrix terminal servers. In last one year we spend lots of time/money in tuning databases, replacing Citrix servers but end result is same. I was wondering if anybody out there has ran into same kind of situation. Our (DBAs) guess is the disk layout is not optimal but we also dont have any data to prove that disks are the bottleneck. Is there any way to collect these kinds of stats in Oracle. We aren't getting much help from our SAN administrator. DISCLAIMER: This message is intended for the sole use of the individual to whom it is addressed, and may contain information that is privileged, confidential and exempt from disclosure under applicable law. If you are not the addressee you are hereby notified that you may not use, copy, disclose, or distribute to anyone the message or any information contained in the message. If you have received this message in error, please immediately advise the sender by reply email
Re: Performance tuning in complex environment
The wholesale system wide collection of timing data is not generally a good way to go about trouble shooting performance issues. You need to pick a process, collect the timing data for that process, and *only* that process, diagnose where the most time is being spent, and determine what can be done to speed it up. This in a nutshell is the basis of Cary's book, at least per my reading of it. Always try to fix the stuff with the biggest payoff. It could be a SQL statement, it could be a misconfigured or malfunctioning network card. It could be that a developer is filling a temporary table with lots of data during a transaction, then deleting the data and doing it over and over again, all the while doing full table scans. FTS is expensive when you want to retrieve 3 rows from a temp table with 500 meg of extents in it. Just for grins though, how about running this script and posting the output for us? Sometimes you get lucky, and something may appear really out of whack. No guarantees though. Troubleshooting system performance problems takes more than an email. HTH Jared = col event format a35 head 'EVENT NAME' col total_waits format 999,999,999 head TOTAL|WAITS col total_timeouts format 999,999,999 head TOTAL|TIMEOUTS col time_waited format 999,999,999 head TIME|WAITED|SECONDS col average_wait format 9 head AVG|WAIT|100ths set line 150 set trimspool on select event, total_waits, total_timeouts, time_waited/100 time_waited, average_wait from v$system_event order by time_waited / On Thu, 2003-12-11 at 10:34, [EMAIL PROTECTED] wrote: Hello Everyone, I am trying to get some help/suggestions reg. how to troubleshoot performance issues. Little back ground about our environment. Its third party application (Logician) from GE. There are total 11 databases, all on oracle 8174 H-UX 11i in cluster environment. All the databases are on EMC Symmetrix using 6 disks. All the clients are connecting to database thru Citrix terminal servers. In last one year we spend lots of time/money in tuning databases, replacing Citrix servers but end result is same. I was wondering if anybody out there has ran into same kind of situation. Our (DBAs) guess is the disk layout is not optimal but we also dont have any data to prove that disks are the bottleneck. Is there any way to collect these kinds of stats in Oracle. We aren't getting much help from our SAN administrator. DISCLAIMER: This message is intended for the sole use of the individual to whom it is addressed, and may contain information that is privileged, confidential and exempt from disclosure under applicable law. If you are not the addressee you are hereby notified that you may not use, copy, disclose, or distribute to anyone the message or any information contained in the message. If you have received this message in error, please immediately advise the sender by reply email and delete this message. -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: [EMAIL PROTECTED] INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538-5051 http://www.fatcity.com San Diego, California-- Mailing list and web hosting services - To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing). -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: Jared Still INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538-5051 http://www.fatcity.com San Diego, California-- Mailing list and web hosting services - To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing).
