My bad. I should also have posted the 10046 trace. I did a 10046 trace and also a STATSPACK report. Of the total time, more than 50% of the time is spent waiting on "DIRECT PATH WRITE" and around 40% of the time it waits on "DIRECT PATH READ". This is what prompted me to do a truss and see what it is doing..
>From one of the Veritas whitepapers - I found this Quick I/O allows databases to access regular files on a VxFS file system as raw character devices, improving throughput for Oracle databases. Unlike raw devices, Quick I/O files can still be managed as regular UNIX files. There are three requirements to use Quick I/O: 1. Quick I/O driver (VRTSqio) must be loaded before a regular file can be accessed through the Quick I/O interface. 2. Files must be preallocated on a VxFS file system. The file must be preallocated because the file cannot be extended through writes via its Quick I/O interface. This preallocation can be done using the qiomkfile command. This command ensures that the file is created as a single large extent, or as a chain of direct extents. Using direct extents is inherently faster than using indirect extents. 3. The file must be accessed via its Quick I/O name extension (::cdev:vxfs:). In a VxFS file system, a file can be accessed using two types of interface: regular file and device file. The device file interface allows a regular file to be accessed as a raw character device. This is achieved by using the Quick I/O naming extension of ::cdev:vxfs: while accessing a regular file. For example, a file named system.dbf can be accessed as a raw character device when the name system.dbf::cdev:vxfs: is used for database access. Converting Oracle Database Files on VERITAS File System to use Quick/IO The scripts getdbfiles.sh and mkqio.sh are provided to easily change Oracle database files to use Quick I/O. The database files must be on VxFS file systems before they can be converted. The getdbfiles.sh script is a shell script that can be run by the Oracle DBA (with appropriate user ID) while the database instance is up and running. This script extracts the filenames from the system tables of the database and stores the filenames in a file called mkqio.dat. Alternatively, you can manually create the mkqio.dat file containing the Oracle database filenames to convert for use with Quick I/O. The mkqio.sh script processes a list of filenames in the file mkqio.dat and converts them to use Quick I/O. This conversion process should be performed while the database is closed. The mkqio.sh script must also be run by the Oracle user of the database instance to avoid any permission problems. I think Quick IO is needed on VxFS to perform KAIO calls. Have anyone done this before? Any input is greatly appreciated. Babu Cary Millsap <[EMAIL PROTECTED] To: Multiple recipients of list ORACLE-L <[EMAIL PROTECTED]> sos.com> cc: Sent by: Subject: RE: performance issues on sun [EMAIL PROTECTED] 02/25/03 05:23 PM Please respond to ORACLE-L I wish now that I hadn't deleted what I composed this morning... It was this: People probably get sick of seeing me say the same thing over and over and over... You have some interesting information from the truss that you've done. But you can't tell how long something took by counting how many times it happened. The easiest way to determine what's consuming the most *time* is to use extended SQL trace (event 10046 level 8). The resulting trace file will tell you exactly where your time has gone, and it will enable you to determine whether your performance problem is a result of the kaio calls or not. You'll probably find that the system is doing what you suspect: issuing an async write call, failing, and then calling a synchronous write call. However, without knowing the impact of this behavior upon response time, it's hard to know whether the time you invest into "checking" stuff and "fixing" stuff is worth anything. The worst feeling is to invest your time into fixing something, succeeding, and then finding you've made no impact because the thing you fixed accounts for only a small amount of response time. ...Find out what activity is consuming the largest chunk of your response time, and then try to figure out how to do that thing less. The cheapest, fastest, most error-free way that I know to do that is to collect the 10046 level-8 data. Cary Millsap Hotsos Enterprises, Ltd. http://www.hotsos.com Upcoming events: - RMOUG Training Days 2003, Mar 5-6 Denver - Hotsos Clinic 101, Mar 25-27 London -----Original Message----- [EMAIL PROTECTED] Sent: Tuesday, February 25, 2003 2:45 PM To: Multiple recipients of list ORACLE-L I did this and its taking the same amount of time. The difference this time is that it does not do the KAIO call. But the time has not improved. Its still doing pwrite calls. TIA Babu John Kanagaraj <[EMAIL PROTECTED] To: Multiple recipients of list ORACLE-L <[EMAIL PROTECTED]> ds.com> cc: Sent by: Subject: RE: performance issues on sun [EMAIL PROTECTED] 02/25/03 01:04 PM Please respond to ORACLE-L Babu, > I think it is trying to do a KAIO call and failing. Then it attempts a > synchronous PWRITE call. > > But our SAs are not able to help us to confirm this. Have any > of you seen > this issue? I think you have hit the nail on the head. By default, the Oracle port on Solaris sets 'disk_async_io' to TRUE. Set this to FALSE by introducing such an entry in init.ora. Let us know if tihis solves your issue... John Kanagaraj Oracle Applications DBA DBSoft Inc (W): 408-970-7002 I don't know what the future holds for me, but I do know who holds my future! ** The opinions and statements above are entirely my own and not those of my employer or clients ** -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: John Kanagaraj INET: [EMAIL PROTECTED] Fat City Network Services -- 858-538-5051 http://www.fatcity.com San Diego, California -- Mailing list and web hosting services --------------------------------------------------------------------- To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing). _____________ This e-mail transmission and any attachments to it are intended solely for the use of the individual or entity to whom it is addressed and may contain confidential and privileged information. If you are not the intended recipient, your use, forwarding, printing, storing, disseminating, distribution, or copying of this communication is prohibited. If you received this communication in error, please notify the sender immediately by replying to this message and delete it from your computer. -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: INET: [EMAIL PROTECTED] Fat City Network Services -- 858-538-5051 http://www.fatcity.com San Diego, California -- Mailing list and web hosting services --------------------------------------------------------------------- To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing). -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: Cary Millsap INET: [EMAIL PROTECTED] Fat City Network Services -- 858-538-5051 http://www.fatcity.com San Diego, California -- Mailing list and web hosting services --------------------------------------------------------------------- To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing). _____________ This e-mail transmission and any attachments to it are intended solely for the use of the individual or entity to whom it is addressed and may contain confidential and privileged information. If you are not the intended recipient, your use, forwarding, printing, storing, disseminating, distribution, or copying of this communication is prohibited. If you received this communication in error, please notify the sender immediately by replying to this message and delete it from your computer. -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: INET: [EMAIL PROTECTED] Fat City Network Services -- 858-538-5051 http://www.fatcity.com San Diego, California -- Mailing list and web hosting services --------------------------------------------------------------------- To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing).