Re: ARS7 install with oracle 10G RAC ... the saga continues ...
I've been thinking about this a bit. The biggest difference I see between running ars with a local db versus a remote db is the method by which arserver connects to the db. With a local db, arserver can connect using IPC, which has a lot less overhead and variables than using TCP with some application protocol. I plan to perform some tests to see if I can isolate any differences in performance and what can be tuned with the oracle client/db to tune the performance. I am guessing that the type of network (gb/mb, jumbo frames, etc), latency, and a number of other factors will drive how things should be configured. Axton Grams On 7/10/07, Lammey, Peter A. [EMAIL PROTECTED] wrote: Does this lead anyone to believe or have people noticed other general activities in the applications are slow based on this? Thanks Peter Lammey ESPN MIT Technical Services Applications Management 860-766-4761 -Original Message- From: Action Request System discussion list(ARSList) [mailto:[EMAIL PROTECTED] On Behalf Of Axton Sent: Tuesday, July 10, 2007 11:47 AM To: arslist@ARSLIST.ORG Subject: Re: ARS7 install with oracle 10G RAC ... the saga continues ... The issue is widespread, esp. with a remote Oracle db. Our servers are switched with fiber/gb. Even if I have 2 vm's runnning on the same physical host (one db and one ars), I see the same issue. The packet logs do not show a problem with throughput and my switches show no congestion. Axton Grams On 7/10/07, Shellman, David [EMAIL PROTECTED] wrote: When I spoke to support about how long it took the system to start up they said that start up was chatty between the server app and the database. With a network link that running near capacity (our case), flaky router/switch, slow link, etc the length of time to respond between app server and fb server is increased.. All these situations are often not apparent with out getting network folks involved. Dave -- [EMAIL PROTECTED] (Wireless) - Original Message - From: Action Request System discussion list(ARSList) arslist@ARSLIST.ORG To: arslist@ARSLIST.ORG arslist@ARSLIST.ORG Sent: Tue Jul 10 10:19:29 2007 Subject: Re: ARS7 install with oracle 10G RAC ... the saga continues ... The problem does not seem to be in the way that the query to the db, but in the way the results are processed. I don't know about that. If the problem was the way that arserverd is processing the results of the query, why would startup be like 1 minute when the db is on the same box? My bet is that this has something to do with the way that arserverd is connecting to oracle ... perhaps incorrect use of the api? All that waiting between queries really makes me think there's a timeout somewhere ... like waiting for a local listener to respond, i.e. looking for the database to be on the same machine, and only querying remotely when the local connection times out. Man, this is why I sometimes hate dealing with proprietary software. If this was open, you could bet someone would have fixed it by now, and if they hadn't we'd at least have a fighting chance of fixing it ourselves. thanks again, everyone for all your help. looks like the only answer for now is the Run-DMC response: it's like that, and that's the way it is ... huh! -A ;-) On Jul 10, 2007, at 8:30 AM, Axton wrote: Good luck with support (and I sincerely mean that). The problem does not seem to be in the way that the query to the db, but in the way the results are processed. If the arserver could serialize the memory contents as part of the shutdown process, then reload them on startup, it could probably be designed to be much more efficient, but there would be trade offs with that route. A few that come to mind: - storage of the serialized contents (possibly gb's) - time required to serialize the contents (slower shutdown) The trade off is for a faster startup. Another method that this could be addressed is to expose some configuration parameters in how the current process works: - chunk size (would be contingent on memory available and architecture limits 32 vs 64 bit) Another method is to streamline the current process; where do the bottlenecks exist in the current design? Can they be eliminated? Just some thoughts... Axton Grams On 7/10/07, Tony Worthington [EMAIL PROTECTED] wrote: Yup. And the second query (each taking about four minutes on our boxes) is: SELECT schemaId,fieldId,groupId,permission FROM field_permissions ORDER BY 1 ASC,2 ASC,3 ASC I wonder if we could get a defect filed, and have this looked at by engineering... and get an answer more than as designed. -tony -- Tony Worthington [EMAIL PROTECTED] 262-703-5911 Axton [EMAIL PROTECTED] Sent by: Action Request System discussion list(ARSList) arslist@ARSLIST.ORG 07/09
Re: ARS7 install with oracle 10G RAC ... the saga continues ...
Do you really think you will get an answer other than as designed? :-) -- Jarl Yuppi yeee, got an ok to travel to Vancouver in october :-) On 7/10/07, Tony Worthington [EMAIL PROTECTED] wrote: Yup. And the second query (each taking about four minutes on our boxes) is: SELECT schemaId,fieldId,groupId,permission FROM field_permissions ORDER BY 1 ASC,2 ASC,3 ASC I wonder if we could get a defect filed, and have this looked at by engineering... and get an answer more than as designed. -tony -- Tony Worthington [EMAIL PROTECTED] 262-703-5911 Axton [EMAIL PROTECTED] Sent by: Action Request System discussion list(ARSList) arslist@ARSLIST.ORG 07/09/2007 05:55 PM Please respond to arslist@ARSLIST.ORG To arslist@ARSLIST.ORG cc Subject Re: ARS7 install with oracle 10G RAC ... the saga continues ... Remedy fetches information from it's data dictionary (e.g., field_disp_prop, field, etc.), then processes it in chunks of 100, 500, or 1000. So if you have a table field_dispprop that has 120k records and 500 records are processed at a time, 15 seconds per chunk, you end up with a start time that totals 3600 seconds (for that part of the startup). If you add a -t parameter to arserverd in the armonitor.conf file, a special startup log file is created. The log file will look something like this: Jul 09 2007 18:29:06.6577 Startup TID:3086968512 Set up thread control block with key = 0 Jul 09 2007 18:29:06.6582 Startup TID:3086968512 Initialize thread local storage block Jul 09 2007 18:29:06.6583 Startup TID:3086968512 Initialize mutiple-byte environment Jul 09 2007 18:29:06.6585 Startup TID:3086968512 InstallDir = /u01/arsystem/arsdev Jul 09 2007 18:29:06.6587 Startup TID:3086968512 Initialize Server utility Jul 09 2007 18:29:06.6599 Startup TID:3086968512 Initialize License Library Jul 09 2007 18:29:06.6602 Startup TID:3086968512 LicenseFilename = /etc/arsystem/arsdev/arsystem.lic Jul 09 2007 18:29:06.6603 Startup TID:3086968512 Initialize Language setting and locale Jul 09 2007 18:29:06.6610 Startup TID:3086968512 Initialize the Decimal Math libray Jul 09 2007 18:29:06.6610 Startup TID:3086968512 Open shared catalog Jul 09 2007 18:29:06.6648 Startup TID:3086968512 Load encryption shared library Jul 09 2007 18:29:06.6653 Startup TID:3086968512 Load encryption static functions Jul 09 2007 18:29:06.6654 Startup TID:3086968512 Initialize default configuration information Jul 09 2007 18:29:06.6654 Startup TID:3086968512 Create Mutexes Jul 09 2007 18:29:06.6655 Startup TID:3086968512 Initialize parse environment Jul 09 2007 18:29:06.6657 Startup TID:3086968512 Initialize date time information Jul 09 2007 18:29:06.6659 Startup TID:3086968512 Initialize notification strings Jul 09 2007 18:29:06.6659 Startup TID:3086968512 Initialize RPC queue type strings Jul 09 2007 18:29:06.6660 Startup TID:3086968512 Initialize filter strings Jul 09 2007 18:29:06.6662 Startup TID:3086968512 Initialize escalation strings Jul 09 2007 18:29:06.6662 Startup TID:3086968512 Load system configuration file Jul 09 2007 18:29:06.6669 Startup TID:3086968512 arsCodeSet=windows-1252[0] arDbCodeSet=windows-1252[0] Jul 09 2007 18:29:06.6670 Startup TID:3086968512 Initialize pending lists Jul 09 2007 18:29:06.6671 Startup TID:3086968512 Create cache read write lock Jul 09 2007 18:29:06.6671 Startup TID:3086968512 Create full text status read write lock Jul 09 2007 18:29:06.6672 Startup TID:3086968512 Initialize fork proxy Jul 09 2007 18:29:06.6672 Startup TID:3086968512 Check licensing Jul 09 2007 18:29:06.6693 Startup TID:3086968512 Initialize user cache Jul 09 2007 18:29:06.6707 Startup TID:3086968512 Open log file Jul 09 2007 18:29:06.6745 Startup TID:3086968512 Initialize XML parser Jul 09 2007 18:29:06.6782 Startup TID:3086968512 Initialize thread list Jul 09 2007 18:29:06.6784 Startup TID:3086968512 Check multiple servers Jul 09 2007 18:29:06.6787 Startup TID:3086968512 Initialize dead thread list Jul 09 2007 18:29:06.6790 Startup TID:3086968512 Initialize timed calls Jul 09 2007 18:29:06.6792 Startup TID:3086968512 CreateRPCQueue(min=1, max=1, rpc=390600) Jul 09 2007 18:29:06.6793 Startup TID:3086968512 Await cache ready... Jul 09 2007 18:29:06.8988 Startup TID:0030571424 InitializeServerCache: LoadInitialSchemaInfo Begin Jul 09 2007 18:29:11.7241 Startup TID:0030571424 Begin LoadDisplayInfoList Jul 09 2007 18:29:25.6076 Startup TID:0030571424 LoadDisplayInfoList: 500 rows Jul 09 2007 18:29:42.7008 Startup TID:0030571424 LoadDisplayInfoList: 1000 rows If you then review the sql that is sent to the db while the server is unavailable, you will see this statement, followed by silence: SELECT schemaId,fieldId,vuiId,propShort,propLong FROM field_dispprop ORDER BY 1 ASC, 2 ASC, 3 ASC What is happening here is that arserverd fetches the field display property info, then processes it 500 records at a time. Depending on the size of your server (number of forms, views, fields, etc), there may be a couple of sql statements
Re: ARS7 install with oracle 10G RAC ... the saga continues ...
Good luck with support (and I sincerely mean that). The problem does not seem to be in the way that the query to the db, but in the way the results are processed. If the arserver could serialize the memory contents as part of the shutdown process, then reload them on startup, it could probably be designed to be much more efficient, but there would be trade offs with that route. A few that come to mind: - storage of the serialized contents (possibly gb's) - time required to serialize the contents (slower shutdown) The trade off is for a faster startup. Another method that this could be addressed is to expose some configuration parameters in how the current process works: - chunk size (would be contingent on memory available and architecture limits 32 vs 64 bit) Another method is to streamline the current process; where do the bottlenecks exist in the current design? Can they be eliminated? Just some thoughts... Axton Grams On 7/10/07, Tony Worthington [EMAIL PROTECTED] wrote: Yup. And the second query (each taking about four minutes on our boxes) is: SELECT schemaId,fieldId,groupId,permission FROM field_permissions ORDER BY 1 ASC,2 ASC,3 ASC I wonder if we could get a defect filed, and have this looked at by engineering... and get an answer more than as designed. -tony -- Tony Worthington [EMAIL PROTECTED] 262-703-5911 Axton [EMAIL PROTECTED] Sent by: Action Request System discussion list(ARSList) arslist@ARSLIST.ORG 07/09/2007 05:55 PM Please respond to arslist@ARSLIST.ORG To arslist@ARSLIST.ORG cc Subject Re: ARS7 install with oracle 10G RAC ... the saga continues ... Remedy fetches information from it's data dictionary (e.g., field_disp_prop, field, etc.), then processes it in chunks of 100, 500, or 1000. So if you have a table field_dispprop that has 120k records and 500 records are processed at a time, 15 seconds per chunk, you end up with a start time that totals 3600 seconds (for that part of the startup). If you add a -t parameter to arserverd in the armonitor.conf file, a special startup log file is created. The log file will look something like this: Jul 09 2007 18:29:06.6577 Startup TID:3086968512 Set up thread control block with key = 0 Jul 09 2007 18:29:06.6582 Startup TID:3086968512 Initialize thread local storage block Jul 09 2007 18:29:06.6583 Startup TID:3086968512 Initialize mutiple-byte environment Jul 09 2007 18:29:06.6585 Startup TID:3086968512 InstallDir = /u01/arsystem/arsdev Jul 09 2007 18:29:06.6587 Startup TID:3086968512 Initialize Server utility Jul 09 2007 18:29:06.6599 Startup TID:3086968512 Initialize License Library Jul 09 2007 18:29:06.6602 Startup TID:3086968512 LicenseFilename = /etc/arsystem/arsdev/arsystem.lic Jul 09 2007 18:29:06.6603 Startup TID:3086968512 Initialize Language setting and locale Jul 09 2007 18:29:06.6610 Startup TID:3086968512 Initialize the Decimal Math libray Jul 09 2007 18:29:06.6610 Startup TID:3086968512 Open shared catalog Jul 09 2007 18:29:06.6648 Startup TID:3086968512 Load encryption shared library Jul 09 2007 18:29:06.6653 Startup TID:3086968512 Load encryption static functions Jul 09 2007 18:29:06.6654 Startup TID:3086968512 Initialize default configuration information Jul 09 2007 18:29:06.6654 Startup TID:3086968512 Create Mutexes Jul 09 2007 18:29:06.6655 Startup TID:3086968512 Initialize parse environment Jul 09 2007 18:29:06.6657 Startup TID:3086968512 Initialize date time information Jul 09 2007 18:29:06.6659 Startup TID:3086968512 Initialize notification strings Jul 09 2007 18:29:06.6659 Startup TID:3086968512 Initialize RPC queue type strings Jul 09 2007 18:29:06.6660 Startup TID:3086968512 Initialize filter strings Jul 09 2007 18:29:06.6662 Startup TID:3086968512 Initialize escalation strings Jul 09 2007 18:29:06.6662 Startup TID:3086968512 Load system configuration file Jul 09 2007 18:29:06.6669 Startup TID:3086968512 arsCodeSet=windows-1252[0] arDbCodeSet=windows-1252[0] Jul 09 2007 18:29:06.6670 Startup TID:3086968512 Initialize pending lists Jul 09 2007 18:29:06.6671 Startup TID:3086968512 Create cache read write lock Jul 09 2007 18:29:06.6671 Startup TID:3086968512 Create full text status read write lock Jul 09 2007 18:29:06.6672 Startup TID:3086968512 Initialize fork proxy Jul 09 2007 18:29:06.6672 Startup TID:3086968512 Check licensing Jul 09 2007 18:29:06.6693 Startup TID:3086968512 Initialize user cache Jul 09 2007 18:29:06.6707 Startup TID:3086968512 Open log file Jul 09 2007 18:29:06.6745 Startup TID:3086968512 Initialize XML parser Jul 09 2007 18:29:06.6782 Startup TID:3086968512 Initialize thread list Jul 09 2007 18:29:06.6784 Startup TID:3086968512 Check multiple servers Jul 09 2007 18:29:06.6787 Startup TID:3086968512 Initialize dead thread list Jul 09 2007 18:29:06.6790 Startup TID:3086968512 Initialize timed calls Jul 09 2007 18:29:06.6792 Startup TID:3086968512 CreateRPCQueue(min=1, max=1, rpc=390600) Jul 09 2007 18:29:06.6793 Startup TID:3086968512 Await cache ready... Jul 09 2007 18
Re: ARS7 install with oracle 10G RAC ... the saga continues ...
Another possible solution is to cache information like the mid-tier does it When its requested, then fetch it. And the possibilites to choose which elements to be fetched at startup. -- Jarl On 7/10/07, Axton [EMAIL PROTECTED] wrote: Good luck with support (and I sincerely mean that). The problem does not seem to be in the way that the query to the db, but in the way the results are processed. If the arserver could serialize the memory contents as part of the shutdown process, then reload them on startup, it could probably be designed to be much more efficient, but there would be trade offs with that route. A few that come to mind: - storage of the serialized contents (possibly gb's) - time required to serialize the contents (slower shutdown) The trade off is for a faster startup. Another method that this could be addressed is to expose some configuration parameters in how the current process works: - chunk size (would be contingent on memory available and architecture limits 32 vs 64 bit) Another method is to streamline the current process; where do the bottlenecks exist in the current design? Can they be eliminated? Just some thoughts... Axton Grams On 7/10/07, Tony Worthington [EMAIL PROTECTED] wrote: Yup. And the second query (each taking about four minutes on our boxes) is: SELECT schemaId,fieldId,groupId,permission FROM field_permissions ORDER BY 1 ASC,2 ASC,3 ASC I wonder if we could get a defect filed, and have this looked at by engineering... and get an answer more than as designed. -tony -- Tony Worthington [EMAIL PROTECTED] 262-703-5911 Axton [EMAIL PROTECTED] Sent by: Action Request System discussion list(ARSList) arslist@ARSLIST.ORG 07/09/2007 05:55 PM Please respond to arslist@ARSLIST.ORG To arslist@ARSLIST.ORG cc Subject Re: ARS7 install with oracle 10G RAC ... the saga continues ... Remedy fetches information from it's data dictionary (e.g., field_disp_prop, field, etc.), then processes it in chunks of 100, 500, or 1000. So if you have a table field_dispprop that has 120k records and 500 records are processed at a time, 15 seconds per chunk, you end up with a start time that totals 3600 seconds (for that part of the startup). If you add a -t parameter to arserverd in the armonitor.conf file, a special startup log file is created. The log file will look something like this: Jul 09 2007 18:29:06.6577 Startup TID:3086968512 Set up thread control block with key = 0 Jul 09 2007 18:29:06.6582 Startup TID:3086968512 Initialize thread local storage block Jul 09 2007 18:29:06.6583 Startup TID:3086968512 Initialize mutiple-byte environment Jul 09 2007 18:29:06.6585 Startup TID:3086968512 InstallDir = /u01/arsystem/arsdev Jul 09 2007 18:29:06.6587 Startup TID:3086968512 Initialize Server utility Jul 09 2007 18:29:06.6599 Startup TID:3086968512 Initialize License Library Jul 09 2007 18:29:06.6602 Startup TID:3086968512 LicenseFilename = /etc/arsystem/arsdev/arsystem.lic Jul 09 2007 18:29:06.6603 Startup TID:3086968512 Initialize Language setting and locale Jul 09 2007 18:29:06.6610 Startup TID:3086968512 Initialize the Decimal Math libray Jul 09 2007 18:29:06.6610 Startup TID:3086968512 Open shared catalog Jul 09 2007 18:29:06.6648 Startup TID:3086968512 Load encryption shared library Jul 09 2007 18:29:06.6653 Startup TID:3086968512 Load encryption static functions Jul 09 2007 18:29:06.6654 Startup TID:3086968512 Initialize default configuration information Jul 09 2007 18:29:06.6654 Startup TID:3086968512 Create Mutexes Jul 09 2007 18:29:06.6655 Startup TID:3086968512 Initialize parse environment Jul 09 2007 18:29:06.6657 Startup TID:3086968512 Initialize date time information Jul 09 2007 18:29:06.6659 Startup TID:3086968512 Initialize notification strings Jul 09 2007 18:29:06.6659 Startup TID:3086968512 Initialize RPC queue type strings Jul 09 2007 18:29:06.6660 Startup TID:3086968512 Initialize filter strings Jul 09 2007 18:29:06.6662 Startup TID:3086968512 Initialize escalation strings Jul 09 2007 18:29:06.6662 Startup TID:3086968512 Load system configuration file Jul 09 2007 18:29:06.6669 Startup TID:3086968512 arsCodeSet=windows-1252[0] arDbCodeSet=windows-1252[0] Jul 09 2007 18:29:06.6670 Startup TID:3086968512 Initialize pending lists Jul 09 2007 18:29:06.6671 Startup TID:3086968512 Create cache read write lock Jul 09 2007 18:29:06.6671 Startup TID:3086968512 Create full text status read write lock Jul 09 2007 18:29:06.6672 Startup TID:3086968512 Initialize fork proxy Jul 09 2007 18:29:06.6672 Startup TID:3086968512 Check licensing Jul 09 2007 18:29:06.6693 Startup TID:3086968512 Initialize user cache Jul 09 2007 18:29:06.6707 Startup TID:3086968512 Open log file Jul 09 2007 18:29:06.6745 Startup TID:3086968512 Initialize XML parser Jul 09 2007 18:29:06.6782 Startup TID:3086968512 Initialize thread list Jul 09 2007 18:29:06.6784 Startup TID:3086968512 Check multiple servers
Re: ARS7 install with oracle 10G RAC ... the saga continues ...
Working with a slight problem :-) -Original Message- From: Action Request System discussion list(ARSList) [mailto:[EMAIL PROTECTED] Behalf Of Jarl Grøneng Sent: Tuesday, July 10, 2007 9:03 AM To: arslist@ARSLIST.ORG Subject: Re: ARS7 install with oracle 10G RAC ... the saga continues ... ** Do you really think you will get an answer other than as designed? :-) -- Jarl Yuppi yeee, got an ok to travel to Vancouver in october :-) On 7/10/07, Tony Worthington [EMAIL PROTECTED] wrote: Yup. And the second query (each taking about four minutes on our boxes) is: SELECT schemaId,fieldId,groupId,permission FROM field_permissions ORDER BY 1 ASC,2 ASC,3 ASC I wonder if we could get a defect filed, and have this looked at by engineering... and get an answer more than as designed. -tony -- Tony Worthington [EMAIL PROTECTED] 262-703-5911 Axton [EMAIL PROTECTED] Sent by: Action Request System discussion list(ARSList) arslist@ARSLIST.ORG 07/09/2007 05:55 PM Please respond to arslist@ARSLIST.ORG To arslist@ARSLIST.ORG cc Subject Re: ARS7 install with oracle 10G RAC ... the saga continues ... Remedy fetches information from it's data dictionary (e.g., field_disp_prop, field, etc.), then processes it in chunks of 100, 500, or 1000. So if you have a table field_dispprop that has 120k records and 500 records are processed at a time, 15 seconds per chunk, you end up with a start time that totals 3600 seconds (for that part of the startup). If you add a -t parameter to arserverd in the armonitor.conf file, a special startup log file is created. The log file will look something like this: Jul 09 2007 18:29:06.6577 Startup TID:3086968512 Set up thread control block with key = 0 Jul 09 2007 18:29:06.6582 Startup TID:3086968512 Initialize thread local storage block Jul 09 2007 18:29:06.6583 Startup TID:3086968512 Initialize mutiple-byte environment Jul 09 2007 18:29:06.6585 Startup TID:3086968512 InstallDir = /u01/arsystem/arsdev Jul 09 2007 18:29:06.6587 Startup TID:3086968512 Initialize Server utility Jul 09 2007 18:29:06.6599 Startup TID:3086968512 Initialize License Library Jul 09 2007 18:29:06.6602 Startup TID:3086968512 LicenseFilename = /etc/arsystem/arsdev/arsystem.lic Jul 09 2007 18:29:06.6603 Startup TID:3086968512 Initialize Language setting and locale Jul 09 2007 18:29:06.6610 Startup TID:3086968512 Initialize the Decimal Math libray Jul 09 2007 18:29:06.6610 Startup TID:3086968512 Open shared catalog Jul 09 2007 18:29:06.6648 Startup TID:3086968512 Load encryption shared library Jul 09 2007 18:29: 06.6653 Startup TID:3086968512 Load encryption static functions Jul 09 2007 18:29:06.6654 Startup TID:3086968512 Initialize default configuration information Jul 09 2007 18:29: 06.6654 Startup TID:3086968512 Create Mutexes Jul 09 2007 18:29:06.6655 Startup TID:3086968512 Initialize parse environment Jul 09 2007 18:29:06.6657 Startup TID:3086968512 Initialize date time information Jul 09 2007 18:29:06.6659 Startup TID:3086968512 Initialize notification strings Jul 09 2007 18:29:06.6659 Startup TID:3086968512 Initialize RPC queue type strings Jul 09 2007 18:29:06.6660 Startup TID:3086968512 Initialize filter strings Jul 09 2007 18:29:06.6662 Startup TID:3086968512 Initialize escalation strings Jul 09 2007 18:29:06.6662 Startup TID:3086968512 Load system configuration file Jul 09 2007 18:29:06.6669 Startup TID:3086968512 arsCodeSet=windows-1252[0] arDbCodeSet=windows-1252[0] Jul 09 2007 18:29:06.6670 Startup TID:3086968512 Initialize pending lists Jul 09 2007 18:29:06.6671 Startup TID:3086968512 Create cache read write lock Jul 09 2007 18:29: 06.6671 Startup TID:3086968512 Create full text status read write lock Jul 09 2007 18:29:06.6672 Startup TID:3086968512 Initialize fork proxy Jul 09 2007 18:29:06.6672 Startup TID:3086968512 Check licensing Jul 09 2007 18:29:06.6693 Startup TID:3086968512 Initialize user cache Jul 09 2007 18:29:06.6707 Startup TID:3086968512 Open log file Jul 09 2007 18:29:06.6745 Startup TID:3086968512 Initialize XML parser Jul 09 2007 18:29:06.6782 Startup TID:3086968512 Initialize thread list Jul 09 2007 18:29: 06.6784 Startup TID:3086968512 Check multiple servers Jul 09 2007 18:29:06.6787 Startup TID:3086968512 Initialize dead thread list Jul 09 2007 18:29:06.6790 Startup TID:3086968512 Initialize timed calls Jul 09 2007 18:29:06.6792 Startup TID:3086968512 CreateRPCQueue(min=1, max=1, rpc=390600) Jul 09 2007 18:29:06.6793 Startup TID:3086968512 Await cache ready... Jul 09 2007 18:29:06.8988 Startup TID:0030571424 InitializeServerCache: LoadInitialSchemaInfo Begin Jul 09 2007 18:29:11.7241 Startup TID:0030571424 Begin
Re: ARS7 install with oracle 10G RAC ... the saga continues ...
The problem does not seem to be in the way that the query to the db, but in the way the results are processed. I don't know about that. If the problem was the way that arserverd is processing the results of the query, why would startup be like 1 minute when the db is on the same box? My bet is that this has something to do with the way that arserverd is connecting to oracle ... perhaps incorrect use of the api? All that waiting between queries really makes me think there's a timeout somewhere ... like waiting for a local listener to respond, i.e. looking for the database to be on the same machine, and only querying remotely when the local connection times out. Man, this is why I sometimes hate dealing with proprietary software. If this was open, you could bet someone would have fixed it by now, and if they hadn't we'd at least have a fighting chance of fixing it ourselves. thanks again, everyone for all your help. looks like the only answer for now is the Run-DMC response: it's like that, and that's the way it is ... huh! -A ;-) On Jul 10, 2007, at 8:30 AM, Axton wrote: Good luck with support (and I sincerely mean that). The problem does not seem to be in the way that the query to the db, but in the way the results are processed. If the arserver could serialize the memory contents as part of the shutdown process, then reload them on startup, it could probably be designed to be much more efficient, but there would be trade offs with that route. A few that come to mind: - storage of the serialized contents (possibly gb's) - time required to serialize the contents (slower shutdown) The trade off is for a faster startup. Another method that this could be addressed is to expose some configuration parameters in how the current process works: - chunk size (would be contingent on memory available and architecture limits 32 vs 64 bit) Another method is to streamline the current process; where do the bottlenecks exist in the current design? Can they be eliminated? Just some thoughts... Axton Grams On 7/10/07, Tony Worthington [EMAIL PROTECTED] wrote: Yup. And the second query (each taking about four minutes on our boxes) is: SELECT schemaId,fieldId,groupId,permission FROM field_permissions ORDER BY 1 ASC,2 ASC,3 ASC I wonder if we could get a defect filed, and have this looked at by engineering... and get an answer more than as designed. -tony -- Tony Worthington [EMAIL PROTECTED] 262-703-5911 Axton [EMAIL PROTECTED] Sent by: Action Request System discussion list(ARSList) arslist@ARSLIST.ORG 07/09/2007 05:55 PM Please respond to arslist@ARSLIST.ORG To arslist@ARSLIST.ORG cc Subject Re: ARS7 install with oracle 10G RAC ... the saga continues ... Remedy fetches information from it's data dictionary (e.g., field_disp_prop, field, etc.), then processes it in chunks of 100, 500, or 1000. So if you have a table field_dispprop that has 120k records and 500 records are processed at a time, 15 seconds per chunk, you end up with a start time that totals 3600 seconds (for that part of the startup). If you add a -t parameter to arserverd in the armonitor.conf file, a special startup log file is created. The log file will look something like this: Jul 09 2007 18:29:06.6577 Startup TID:3086968512 Set up thread control block with key = 0 Jul 09 2007 18:29:06.6582 Startup TID:3086968512 Initialize thread local storage block Jul 09 2007 18:29:06.6583 Startup TID:3086968512 Initialize mutiple-byte environment Jul 09 2007 18:29:06.6585 Startup TID:3086968512 InstallDir = /u01/arsystem/arsdev Jul 09 2007 18:29:06.6587 Startup TID:3086968512 Initialize Server utility Jul 09 2007 18:29:06.6599 Startup TID:3086968512 Initialize License Library Jul 09 2007 18:29:06.6602 Startup TID:3086968512 LicenseFilename = /etc/arsystem/arsdev/arsystem.lic Jul 09 2007 18:29:06.6603 Startup TID:3086968512 Initialize Language setting and locale Jul 09 2007 18:29:06.6610 Startup TID:3086968512 Initialize the Decimal Math libray Jul 09 2007 18:29:06.6610 Startup TID:3086968512 Open shared catalog Jul 09 2007 18:29:06.6648 Startup TID:3086968512 Load encryption shared library Jul 09 2007 18:29:06.6653 Startup TID:3086968512 Load encryption static functions Jul 09 2007 18:29:06.6654 Startup TID:3086968512 Initialize default configuration information Jul 09 2007 18:29:06.6654 Startup TID:3086968512 Create Mutexes Jul 09 2007 18:29:06.6655 Startup TID:3086968512 Initialize parse environment Jul 09 2007 18:29:06.6657 Startup TID:3086968512 Initialize date time information Jul 09 2007 18:29:06.6659 Startup TID:3086968512 Initialize notification strings Jul 09 2007 18:29:06.6659 Startup TID:3086968512 Initialize RPC queue type strings Jul 09 2007 18:29:06.6660 Startup TID:3086968512 Initialize filter strings Jul 09 2007 18:29:06.6662 Startup TID:3086968512 Initialize escalation strings Jul 09 2007 18:29:06.6662 Startup TID:3086968512 Load system configuration file Jul
Re: ARS7 install with oracle 10G RAC ... the saga continues ...
Definitely opaque. All I have to go on are packet dumps, sqlnet logs, low level system calls, and an arserver startup log generated by using an undocumented flag to arserverd. Axton Grams On 7/10/07, Andrew Hicox [EMAIL PROTECTED] wrote: The problem does not seem to be in the way that the query to the db, but in the way the results are processed. I don't know about that. If the problem was the way that arserverd is processing the results of the query, why would startup be like 1 minute when the db is on the same box? My bet is that this has something to do with the way that arserverd is connecting to oracle ... perhaps incorrect use of the api? All that waiting between queries really makes me think there's a timeout somewhere ... like waiting for a local listener to respond, i.e. looking for the database to be on the same machine, and only querying remotely when the local connection times out. Man, this is why I sometimes hate dealing with proprietary software. If this was open, you could bet someone would have fixed it by now, and if they hadn't we'd at least have a fighting chance of fixing it ourselves. thanks again, everyone for all your help. looks like the only answer for now is the Run-DMC response: it's like that, and that's the way it is ... huh! -A ;-) On Jul 10, 2007, at 8:30 AM, Axton wrote: Good luck with support (and I sincerely mean that). The problem does not seem to be in the way that the query to the db, but in the way the results are processed. If the arserver could serialize the memory contents as part of the shutdown process, then reload them on startup, it could probably be designed to be much more efficient, but there would be trade offs with that route. A few that come to mind: - storage of the serialized contents (possibly gb's) - time required to serialize the contents (slower shutdown) The trade off is for a faster startup. Another method that this could be addressed is to expose some configuration parameters in how the current process works: - chunk size (would be contingent on memory available and architecture limits 32 vs 64 bit) Another method is to streamline the current process; where do the bottlenecks exist in the current design? Can they be eliminated? Just some thoughts... Axton Grams On 7/10/07, Tony Worthington [EMAIL PROTECTED] wrote: Yup. And the second query (each taking about four minutes on our boxes) is: SELECT schemaId,fieldId,groupId,permission FROM field_permissions ORDER BY 1 ASC,2 ASC,3 ASC I wonder if we could get a defect filed, and have this looked at by engineering... and get an answer more than as designed. -tony -- Tony Worthington [EMAIL PROTECTED] 262-703-5911 Axton [EMAIL PROTECTED] Sent by: Action Request System discussion list(ARSList) arslist@ARSLIST.ORG 07/09/2007 05:55 PM Please respond to arslist@ARSLIST.ORG To arslist@ARSLIST.ORG cc Subject Re: ARS7 install with oracle 10G RAC ... the saga continues ... Remedy fetches information from it's data dictionary (e.g., field_disp_prop, field, etc.), then processes it in chunks of 100, 500, or 1000. So if you have a table field_dispprop that has 120k records and 500 records are processed at a time, 15 seconds per chunk, you end up with a start time that totals 3600 seconds (for that part of the startup). If you add a -t parameter to arserverd in the armonitor.conf file, a special startup log file is created. The log file will look something like this: Jul 09 2007 18:29:06.6577 Startup TID:3086968512 Set up thread control block with key = 0 Jul 09 2007 18:29:06.6582 Startup TID:3086968512 Initialize thread local storage block Jul 09 2007 18:29:06.6583 Startup TID:3086968512 Initialize mutiple-byte environment Jul 09 2007 18:29:06.6585 Startup TID:3086968512 InstallDir = /u01/arsystem/arsdev Jul 09 2007 18:29:06.6587 Startup TID:3086968512 Initialize Server utility Jul 09 2007 18:29:06.6599 Startup TID:3086968512 Initialize License Library Jul 09 2007 18:29:06.6602 Startup TID:3086968512 LicenseFilename = /etc/arsystem/arsdev/arsystem.lic Jul 09 2007 18:29:06.6603 Startup TID:3086968512 Initialize Language setting and locale Jul 09 2007 18:29:06.6610 Startup TID:3086968512 Initialize the Decimal Math libray Jul 09 2007 18:29:06.6610 Startup TID:3086968512 Open shared catalog Jul 09 2007 18:29:06.6648 Startup TID:3086968512 Load encryption shared library Jul 09 2007 18:29:06.6653 Startup TID:3086968512 Load encryption static functions Jul 09 2007 18:29:06.6654 Startup TID:3086968512 Initialize default configuration information Jul 09 2007 18:29:06.6654 Startup TID:3086968512 Create Mutexes Jul 09 2007 18:29:06.6655 Startup TID:3086968512 Initialize parse environment Jul 09 2007 18:29:06.6657 Startup TID:3086968512 Initialize date time information Jul 09 2007 18:29:06.6659 Startup TID:3086968512 Initialize notification strings Jul 09 2007 18:29:06.6659 Startup TID
Re: ARS7 install with oracle 10G RAC ... the saga continues ...
When I spoke to support about how long it took the system to start up they said that start up was chatty between the server app and the database. With a network link that running near capacity (our case), flaky router/switch, slow link, etc the length of time to respond between app server and fb server is increased.. All these situations are often not apparent with out getting network folks involved. Dave -- [EMAIL PROTECTED] (Wireless) - Original Message - From: Action Request System discussion list(ARSList) arslist@ARSLIST.ORG To: arslist@ARSLIST.ORG arslist@ARSLIST.ORG Sent: Tue Jul 10 10:19:29 2007 Subject: Re: ARS7 install with oracle 10G RAC ... the saga continues ... The problem does not seem to be in the way that the query to the db, but in the way the results are processed. I don't know about that. If the problem was the way that arserverd is processing the results of the query, why would startup be like 1 minute when the db is on the same box? My bet is that this has something to do with the way that arserverd is connecting to oracle ... perhaps incorrect use of the api? All that waiting between queries really makes me think there's a timeout somewhere ... like waiting for a local listener to respond, i.e. looking for the database to be on the same machine, and only querying remotely when the local connection times out. Man, this is why I sometimes hate dealing with proprietary software. If this was open, you could bet someone would have fixed it by now, and if they hadn't we'd at least have a fighting chance of fixing it ourselves. thanks again, everyone for all your help. looks like the only answer for now is the Run-DMC response: it's like that, and that's the way it is ... huh! -A ;-) On Jul 10, 2007, at 8:30 AM, Axton wrote: Good luck with support (and I sincerely mean that). The problem does not seem to be in the way that the query to the db, but in the way the results are processed. If the arserver could serialize the memory contents as part of the shutdown process, then reload them on startup, it could probably be designed to be much more efficient, but there would be trade offs with that route. A few that come to mind: - storage of the serialized contents (possibly gb's) - time required to serialize the contents (slower shutdown) The trade off is for a faster startup. Another method that this could be addressed is to expose some configuration parameters in how the current process works: - chunk size (would be contingent on memory available and architecture limits 32 vs 64 bit) Another method is to streamline the current process; where do the bottlenecks exist in the current design? Can they be eliminated? Just some thoughts... Axton Grams On 7/10/07, Tony Worthington [EMAIL PROTECTED] wrote: Yup. And the second query (each taking about four minutes on our boxes) is: SELECT schemaId,fieldId,groupId,permission FROM field_permissions ORDER BY 1 ASC,2 ASC,3 ASC I wonder if we could get a defect filed, and have this looked at by engineering... and get an answer more than as designed. -tony -- Tony Worthington [EMAIL PROTECTED] 262-703-5911 Axton [EMAIL PROTECTED] Sent by: Action Request System discussion list(ARSList) arslist@ARSLIST.ORG 07/09/2007 05:55 PM Please respond to arslist@ARSLIST.ORG To arslist@ARSLIST.ORG cc Subject Re: ARS7 install with oracle 10G RAC ... the saga continues ... Remedy fetches information from it's data dictionary (e.g., field_disp_prop, field, etc.), then processes it in chunks of 100, 500, or 1000. So if you have a table field_dispprop that has 120k records and 500 records are processed at a time, 15 seconds per chunk, you end up with a start time that totals 3600 seconds (for that part of the startup). If you add a -t parameter to arserverd in the armonitor.conf file, a special startup log file is created. The log file will look something like this: Jul 09 2007 18:29:06.6577 Startup TID:3086968512 Set up thread control block with key = 0 Jul 09 2007 18:29:06.6582 Startup TID:3086968512 Initialize thread local storage block Jul 09 2007 18:29:06.6583 Startup TID:3086968512 Initialize mutiple-byte environment Jul 09 2007 18:29:06.6585 Startup TID:3086968512 InstallDir = /u01/arsystem/arsdev Jul 09 2007 18:29:06.6587 Startup TID:3086968512 Initialize Server utility Jul 09 2007 18:29:06.6599 Startup TID:3086968512 Initialize License Library Jul 09 2007 18:29:06.6602 Startup TID:3086968512 LicenseFilename = /etc/arsystem/arsdev/arsystem.lic Jul 09 2007 18:29:06.6603 Startup TID:3086968512 Initialize Language setting and locale Jul 09 2007 18:29:06.6610 Startup TID:3086968512 Initialize the Decimal Math libray Jul 09 2007 18:29:06.6610 Startup TID:3086968512 Open shared catalog Jul 09 2007 18:29:06.6648 Startup TID:3086968512 Load encryption shared library Jul 09 2007
Re: ARS7 install with oracle 10G RAC ... the saga continues ...
The issue is widespread, esp. with a remote Oracle db. Our servers are switched with fiber/gb. Even if I have 2 vm's runnning on the same physical host (one db and one ars), I see the same issue. The packet logs do not show a problem with throughput and my switches show no congestion. Axton Grams On 7/10/07, Shellman, David [EMAIL PROTECTED] wrote: When I spoke to support about how long it took the system to start up they said that start up was chatty between the server app and the database. With a network link that running near capacity (our case), flaky router/switch, slow link, etc the length of time to respond between app server and fb server is increased.. All these situations are often not apparent with out getting network folks involved. Dave -- [EMAIL PROTECTED] (Wireless) - Original Message - From: Action Request System discussion list(ARSList) arslist@ARSLIST.ORG To: arslist@ARSLIST.ORG arslist@ARSLIST.ORG Sent: Tue Jul 10 10:19:29 2007 Subject: Re: ARS7 install with oracle 10G RAC ... the saga continues ... The problem does not seem to be in the way that the query to the db, but in the way the results are processed. I don't know about that. If the problem was the way that arserverd is processing the results of the query, why would startup be like 1 minute when the db is on the same box? My bet is that this has something to do with the way that arserverd is connecting to oracle ... perhaps incorrect use of the api? All that waiting between queries really makes me think there's a timeout somewhere ... like waiting for a local listener to respond, i.e. looking for the database to be on the same machine, and only querying remotely when the local connection times out. Man, this is why I sometimes hate dealing with proprietary software. If this was open, you could bet someone would have fixed it by now, and if they hadn't we'd at least have a fighting chance of fixing it ourselves. thanks again, everyone for all your help. looks like the only answer for now is the Run-DMC response: it's like that, and that's the way it is ... huh! -A ;-) On Jul 10, 2007, at 8:30 AM, Axton wrote: Good luck with support (and I sincerely mean that). The problem does not seem to be in the way that the query to the db, but in the way the results are processed. If the arserver could serialize the memory contents as part of the shutdown process, then reload them on startup, it could probably be designed to be much more efficient, but there would be trade offs with that route. A few that come to mind: - storage of the serialized contents (possibly gb's) - time required to serialize the contents (slower shutdown) The trade off is for a faster startup. Another method that this could be addressed is to expose some configuration parameters in how the current process works: - chunk size (would be contingent on memory available and architecture limits 32 vs 64 bit) Another method is to streamline the current process; where do the bottlenecks exist in the current design? Can they be eliminated? Just some thoughts... Axton Grams On 7/10/07, Tony Worthington [EMAIL PROTECTED] wrote: Yup. And the second query (each taking about four minutes on our boxes) is: SELECT schemaId,fieldId,groupId,permission FROM field_permissions ORDER BY 1 ASC,2 ASC,3 ASC I wonder if we could get a defect filed, and have this looked at by engineering... and get an answer more than as designed. -tony -- Tony Worthington [EMAIL PROTECTED] 262-703-5911 Axton [EMAIL PROTECTED] Sent by: Action Request System discussion list(ARSList) arslist@ARSLIST.ORG 07/09/2007 05:55 PM Please respond to arslist@ARSLIST.ORG To arslist@ARSLIST.ORG cc Subject Re: ARS7 install with oracle 10G RAC ... the saga continues ... Remedy fetches information from it's data dictionary (e.g., field_disp_prop, field, etc.), then processes it in chunks of 100, 500, or 1000. So if you have a table field_dispprop that has 120k records and 500 records are processed at a time, 15 seconds per chunk, you end up with a start time that totals 3600 seconds (for that part of the startup). If you add a -t parameter to arserverd in the armonitor.conf file, a special startup log file is created. The log file will look something like this: Jul 09 2007 18:29:06.6577 Startup TID:3086968512 Set up thread control block with key = 0 Jul 09 2007 18:29:06.6582 Startup TID:3086968512 Initialize thread local storage block Jul 09 2007 18:29:06.6583 Startup TID:3086968512 Initialize mutiple-byte environment Jul 09 2007 18:29:06.6585 Startup TID:3086968512 InstallDir = /u01/arsystem/arsdev Jul 09 2007 18:29:06.6587 Startup TID:3086968512 Initialize Server utility Jul 09 2007 18:29:06.6599 Startup TID:3086968512 Initialize License Library Jul 09
Re: ARS7 install with oracle 10G RAC ... the saga continues ...
Does this lead anyone to believe or have people noticed other general activities in the applications are slow based on this? Thanks Peter Lammey ESPN MIT Technical Services Applications Management 860-766-4761 -Original Message- From: Action Request System discussion list(ARSList) [mailto:[EMAIL PROTECTED] On Behalf Of Axton Sent: Tuesday, July 10, 2007 11:47 AM To: arslist@ARSLIST.ORG Subject: Re: ARS7 install with oracle 10G RAC ... the saga continues ... The issue is widespread, esp. with a remote Oracle db. Our servers are switched with fiber/gb. Even if I have 2 vm's runnning on the same physical host (one db and one ars), I see the same issue. The packet logs do not show a problem with throughput and my switches show no congestion. Axton Grams On 7/10/07, Shellman, David [EMAIL PROTECTED] wrote: When I spoke to support about how long it took the system to start up they said that start up was chatty between the server app and the database. With a network link that running near capacity (our case), flaky router/switch, slow link, etc the length of time to respond between app server and fb server is increased.. All these situations are often not apparent with out getting network folks involved. Dave -- [EMAIL PROTECTED] (Wireless) - Original Message - From: Action Request System discussion list(ARSList) arslist@ARSLIST.ORG To: arslist@ARSLIST.ORG arslist@ARSLIST.ORG Sent: Tue Jul 10 10:19:29 2007 Subject: Re: ARS7 install with oracle 10G RAC ... the saga continues ... The problem does not seem to be in the way that the query to the db, but in the way the results are processed. I don't know about that. If the problem was the way that arserverd is processing the results of the query, why would startup be like 1 minute when the db is on the same box? My bet is that this has something to do with the way that arserverd is connecting to oracle ... perhaps incorrect use of the api? All that waiting between queries really makes me think there's a timeout somewhere ... like waiting for a local listener to respond, i.e. looking for the database to be on the same machine, and only querying remotely when the local connection times out. Man, this is why I sometimes hate dealing with proprietary software. If this was open, you could bet someone would have fixed it by now, and if they hadn't we'd at least have a fighting chance of fixing it ourselves. thanks again, everyone for all your help. looks like the only answer for now is the Run-DMC response: it's like that, and that's the way it is ... huh! -A ;-) On Jul 10, 2007, at 8:30 AM, Axton wrote: Good luck with support (and I sincerely mean that). The problem does not seem to be in the way that the query to the db, but in the way the results are processed. If the arserver could serialize the memory contents as part of the shutdown process, then reload them on startup, it could probably be designed to be much more efficient, but there would be trade offs with that route. A few that come to mind: - storage of the serialized contents (possibly gb's) - time required to serialize the contents (slower shutdown) The trade off is for a faster startup. Another method that this could be addressed is to expose some configuration parameters in how the current process works: - chunk size (would be contingent on memory available and architecture limits 32 vs 64 bit) Another method is to streamline the current process; where do the bottlenecks exist in the current design? Can they be eliminated? Just some thoughts... Axton Grams On 7/10/07, Tony Worthington [EMAIL PROTECTED] wrote: Yup. And the second query (each taking about four minutes on our boxes) is: SELECT schemaId,fieldId,groupId,permission FROM field_permissions ORDER BY 1 ASC,2 ASC,3 ASC I wonder if we could get a defect filed, and have this looked at by engineering... and get an answer more than as designed. -tony -- Tony Worthington [EMAIL PROTECTED] 262-703-5911 Axton [EMAIL PROTECTED] Sent by: Action Request System discussion list(ARSList) arslist@ARSLIST.ORG 07/09/2007 05:55 PM Please respond to arslist@ARSLIST.ORG To arslist@ARSLIST.ORG cc Subject Re: ARS7 install with oracle 10G RAC ... the saga continues ... Remedy fetches information from it's data dictionary (e.g., field_disp_prop, field, etc.), then processes it in chunks of 100, 500, or 1000. So if you have a table field_dispprop that has 120k records and 500 records are processed at a time, 15 seconds per chunk, you end up with a start time that totals 3600 seconds (for that part of the startup). If you add a -t parameter to arserverd in the armonitor.conf file, a special
Re: ARS7 install with oracle 10G RAC ... the saga continues ... KM-000010001381 (UNCLASSIFIED)
Classification: UNCLASSIFIED Caveats: NONE We have been having performance issues with our Oracle installation and found this KM article with some ideas on how to improve performance and initial configuration: * The following is information for ITSM 7.0.x installs within an Oracle environment: 1. Have your DBA set the SGA Size to min 250 and max 850. 2. Also, Knowledge Base entry KM-10001381 should prove to be helpful to you: * PROBLEM: Below are some settings which have been helpful for Oracle customers to avoid ARERR 91, 92, 93, 94 timeout issues, especially during application installations SOLUTION: 1) AR Server version 7.01 or AR Server version 6.03 patch 19 The above version or later contain new method of Oracle client calls which enhance performance 2) For Oracle 9i or 10g customer: add CURSOR_SHARING=FORCE to the Oracle init file, 3) For Oracle 9i or 10g customer: add the following parameters to ar.conf or ar.cfg Oracle-Cursor-Sharing: FORCE 4) For all Windows (with 2 CPU) customers, Suggest to assign private RPC to RE (in ar.conf or ar.cfg) RE-RPC-Socket: 390621 Private-RPC-Socket: 390621 10 10 For UNIX platforms, RE-RPC-Socket: 390621 Private-RPC-Socket: 390621 6 6 As always, turning of your database and ARServer is a task for your dba, system administrator and Remedy administrator working in concert. The settings above have proved helpful in several cases, but they should be used as a starting point for investigation rather than standard configuration applicable in all environments. A defect SW00254374 has been logged for Engineering to consider whether this information or an official recommendation should be included in the shipped product docs. ** Scott. -Original Message- From: Action Request System discussion list(ARSList) [mailto:[EMAIL PROTECTED] On Behalf Of Lammey, Peter A. Sent: Tuesday, July 10, 2007 11:51 AM To: arslist@ARSLIST.ORG Subject: Re: ARS7 install with oracle 10G RAC ... the saga continues ... Does this lead anyone to believe or have people noticed other general activities in the applications are slow based on this? Thanks Peter Lammey ESPN MIT Technical Services Applications Management 860-766-4761 -Original Message- From: Action Request System discussion list(ARSList) [mailto:[EMAIL PROTECTED] On Behalf Of Axton Sent: Tuesday, July 10, 2007 11:47 AM To: arslist@ARSLIST.ORG Subject: Re: ARS7 install with oracle 10G RAC ... the saga continues ... The issue is widespread, esp. with a remote Oracle db. Our servers are switched with fiber/gb. Even if I have 2 vm's runnning on the same physical host (one db and one ars), I see the same issue. The packet logs do not show a problem with throughput and my switches show no congestion. Axton Grams On 7/10/07, Shellman, David [EMAIL PROTECTED] wrote: When I spoke to support about how long it took the system to start up they said that start up was chatty between the server app and the database. With a network link that running near capacity (our case), flaky router/switch, slow link, etc the length of time to respond between app server and fb server is increased.. All these situations are often not apparent with out getting network folks involved. Dave -- [EMAIL PROTECTED] (Wireless) - Original Message - From: Action Request System discussion list(ARSList) arslist@ARSLIST.ORG To: arslist@ARSLIST.ORG arslist@ARSLIST.ORG Sent: Tue Jul 10 10:19:29 2007 Subject: Re: ARS7 install with oracle 10G RAC ... the saga continues ... The problem does not seem to be in the way that the query to the db, but in the way the results are processed. I don't know about that. If the problem was the way that arserverd is processing the results of the query, why would startup be like 1 minute when the db is on the same box? My bet is that this has something to do with the way that arserverd is connecting to oracle ... perhaps incorrect use of the api? All that waiting between queries really makes me think there's a timeout somewhere ... like waiting for a local listener to respond, i.e. looking for the database to be on the same machine, and only querying remotely when the local connection times out. Man, this is why I sometimes hate dealing with proprietary software. If this was open, you could bet someone would have fixed it by now, and if they hadn't we'd at least have a fighting chance of fixing it ourselves. thanks again, everyone for all your help. looks like the only answer for now is the Run-DMC response: it's like that, and that's the way it is ... huh! -A ;-) On Jul 10, 2007, at 8:30 AM, Axton wrote: Good luck with support (and I sincerely mean that). The problem does
Re: ARS7 install with oracle 10G RAC ... the saga continues ... KM-000010001381 (UNCLASSIFIED)
Now I'm confused. There's a cursor sharing whitepaper Oracle's Cursor Sharing for BMC® Remedy® Products that specifies that it be set at SIMILAR. Now which is it BMC. SIMILAR or FORCE? Best practices mandate specifically that, with 6.0.1, users should use force, and with 6.3 or 7.0, users should use similar, but the difference between the two settings is small. The syntax of the cursor_sharing setting in the ar.conf file is as follows (set only one): Oracle-Cursor-Sharing: SIMILAR Oracle-Cursor-Sharing: FORCE -- Tony Worthington [EMAIL PROTECTED] 262-703-5911 HARTWICK, SCOTT G CTR DISA JSSC [EMAIL PROTECTED] Sent by: Action Request System discussion list(ARSList) arslist@ARSLIST.ORG 07/10/2007 11:02 AM Please respond to arslist@ARSLIST.ORG To arslist@ARSLIST.ORG cc Subject Re: ARS7 install with oracle 10G RAC ... the saga continues ... KM-10001381 (UNCLASSIFIED) Classification: UNCLASSIFIED Caveats: NONE We have been having performance issues with our Oracle installation and found this KM article with some ideas on how to improve performance and initial configuration: * The following is information for ITSM 7.0.x installs within an Oracle environment: 1. Have your DBA set the SGA Size to min 250 and max 850. 2. Also, Knowledge Base entry KM-10001381 should prove to be helpful to you: * PROBLEM: Below are some settings which have been helpful for Oracle customers to avoid ARERR 91, 92, 93, 94 timeout issues, especially during application installations SOLUTION: 1) AR Server version 7.01 or AR Server version 6.03 patch 19 The above version or later contain new method of Oracle client calls which enhance performance 2) For Oracle 9i or 10g customer: add CURSOR_SHARING=FORCE to the Oracle init file, 3) For Oracle 9i or 10g customer: add the following parameters to ar.conf or ar.cfg Oracle-Cursor-Sharing: FORCE 4) For all Windows (with 2 CPU) customers, Suggest to assign private RPC to RE (in ar.conf or ar.cfg) RE-RPC-Socket: 390621 Private-RPC-Socket: 390621 10 10 For UNIX platforms, RE-RPC-Socket: 390621 Private-RPC-Socket: 390621 6 6 As always, turning of your database and ARServer is a task for your dba, system administrator and Remedy administrator working in concert. The settings above have proved helpful in several cases, but they should be used as a starting point for investigation rather than standard configuration applicable in all environments. A defect SW00254374 has been logged for Engineering to consider whether this information or an official recommendation should be included in the shipped product docs. ** Scott. -Original Message- From: Action Request System discussion list(ARSList) [mailto:[EMAIL PROTECTED] On Behalf Of Lammey, Peter A. Sent: Tuesday, July 10, 2007 11:51 AM To: arslist@ARSLIST.ORG Subject: Re: ARS7 install with oracle 10G RAC ... the saga continues ... Does this lead anyone to believe or have people noticed other general activities in the applications are slow based on this? Thanks Peter Lammey ESPN MIT Technical Services Applications Management 860-766-4761 -Original Message- From: Action Request System discussion list(ARSList) [mailto:[EMAIL PROTECTED] On Behalf Of Axton Sent: Tuesday, July 10, 2007 11:47 AM To: arslist@ARSLIST.ORG Subject: Re: ARS7 install with oracle 10G RAC ... the saga continues ... The issue is widespread, esp. with a remote Oracle db. Our servers are switched with fiber/gb. Even if I have 2 vm's runnning on the same physical host (one db and one ars), I see the same issue. The packet logs do not show a problem with throughput and my switches show no congestion. Axton Grams On 7/10/07, Shellman, David [EMAIL PROTECTED] wrote: When I spoke to support about how long it took the system to start up they said that start up was chatty between the server app and the database. With a network link that running near capacity (our case), flaky router/switch, slow link, etc the length of time to respond between app server and fb server is increased.. All these situations are often not apparent with out getting network folks involved. Dave -- [EMAIL PROTECTED] (Wireless) - Original Message - From: Action Request System discussion list(ARSList) arslist@ARSLIST.ORG To: arslist@ARSLIST.ORG arslist@ARSLIST.ORG Sent: Tue Jul 10 10:19:29 2007 Subject: Re: ARS7 install with oracle 10G RAC ... the saga continues ... The problem does not seem to be in the way that the query to the db, but in the way the results are processed. I don't know about that. If the problem was the way that arserverd is processing the results of the query, why would startup be like 1 minute when the db is on the same box
Re: ARS7 install with oracle 10G RAC ... the saga continues ... KM-000010001381 (UNCLASSIFIED)
Something else to consider is that the CLOB-in-row setting (which you should also consider setting), will only have its full effect if set prior to installation. When running our initial Discovery, we noticed that our DB had increased 10x in size - far more than the data would have indicated. Changing the CLOB-in-row setting on an EXISTING DB did NOT recover the wasted disk space. There is an RFE in now on the fact that Remedy does not use Oracle bind parameters as often as it could, which really hurts performance. I bet that the Remedy engineers kind of wish Oracle would act like the rest of the DBs, but that's probably a common wish across many platforms. Rick On 7/10/07, HARTWICK, SCOTT G CTR DISA JSSC [EMAIL PROTECTED] wrote: Classification: UNCLASSIFIED Caveats: NONE We have been having performance issues with our Oracle installation and found this KM article with some ideas on how to improve performance and initial configuration: * The following is information for ITSM 7.0.x installs within an Oracle environment: 1. Have your DBA set the SGA Size to min 250 and max 850. 2. Also, Knowledge Base entry KM-10001381 should prove to be helpful to you: * PROBLEM: Below are some settings which have been helpful for Oracle customers to avoid ARERR 91, 92, 93, 94 timeout issues, especially during application installations SOLUTION: 1) AR Server version 7.01 or AR Server version 6.03 patch 19 The above version or later contain new method of Oracle client calls which enhance performance 2) For Oracle 9i or 10g customer: add CURSOR_SHARING=FORCE to the Oracle init file, 3) For Oracle 9i or 10g customer: add the following parameters to ar.conf or ar.cfg Oracle-Cursor-Sharing: FORCE 4) For all Windows (with 2 CPU) customers, Suggest to assign private RPC to RE (in ar.conf or ar.cfg) RE-RPC-Socket: 390621 Private-RPC-Socket: 390621 10 10 For UNIX platforms, RE-RPC-Socket: 390621 Private-RPC-Socket: 390621 6 6 As always, turning of your database and ARServer is a task for your dba, system administrator and Remedy administrator working in concert. The settings above have proved helpful in several cases, but they should be used as a starting point for investigation rather than standard configuration applicable in all environments. A defect SW00254374 has been logged for Engineering to consider whether this information or an official recommendation should be included in the shipped product docs. ** Scott. -Original Message- From: Action Request System discussion list(ARSList) [mailto:[EMAIL PROTECTED] On Behalf Of Lammey, Peter A. Sent: Tuesday, July 10, 2007 11:51 AM To: arslist@ARSLIST.ORG Subject: Re: ARS7 install with oracle 10G RAC ... the saga continues ... Does this lead anyone to believe or have people noticed other general activities in the applications are slow based on this? Thanks Peter Lammey ESPN MIT Technical Services Applications Management 860-766-4761 -Original Message- From: Action Request System discussion list(ARSList) [mailto:[EMAIL PROTECTED] On Behalf Of Axton Sent: Tuesday, July 10, 2007 11:47 AM To: arslist@ARSLIST.ORG Subject: Re: ARS7 install with oracle 10G RAC ... the saga continues ... The issue is widespread, esp. with a remote Oracle db. Our servers are switched with fiber/gb. Even if I have 2 vm's runnning on the same physical host (one db and one ars), I see the same issue. The packet logs do not show a problem with throughput and my switches show no congestion. Axton Grams On 7/10/07, Shellman, David [EMAIL PROTECTED] wrote: When I spoke to support about how long it took the system to start up they said that start up was chatty between the server app and the database. With a network link that running near capacity (our case), flaky router/switch, slow link, etc the length of time to respond between app server and fb server is increased.. All these situations are often not apparent with out getting network folks involved. Dave -- [EMAIL PROTECTED] (Wireless) - Original Message - From: Action Request System discussion list(ARSList) arslist@ARSLIST.ORG To: arslist@ARSLIST.ORG arslist@ARSLIST.ORG Sent: Tue Jul 10 10:19:29 2007 Subject: Re: ARS7 install with oracle 10G RAC ... the saga continues ... The problem does not seem to be in the way that the query to the db, but in the way the results are processed. I don't know about that. If the problem was the way that arserverd is processing the results of the query, why would startup be like 1 minute when the db is on the same box? My bet is that this has something to do with the way that arserverd is connecting to oracle ... perhaps incorrect use of the api? All that waiting between queries really makes me think
Re: ARS7 install with oracle 10G RAC ... the saga continues ... KM-000010001381 (UNCLASSIFIED)
For performance of applications on a single server it is ok. But I have noticed that if you run 2 AR servers with ITSM apps you get timeouts. Emad On 7/10/07, HARTWICK, SCOTT G CTR DISA JSSC [EMAIL PROTECTED] wrote: Classification: UNCLASSIFIED Caveats: NONE We have been having performance issues with our Oracle installation and found this KM article with some ideas on how to improve performance and initial configuration: * The following is information for ITSM 7.0.x installs within an Oracle environment: 1. Have your DBA set the SGA Size to min 250 and max 850. 2. Also, Knowledge Base entry KM-10001381 should prove to be helpful to you: * PROBLEM: Below are some settings which have been helpful for Oracle customers to avoid ARERR 91, 92, 93, 94 timeout issues, especially during application installations SOLUTION: 1) AR Server version 7.01 or AR Server version 6.03 patch 19 The above version or later contain new method of Oracle client calls which enhance performance 2) For Oracle 9i or 10g customer: add CURSOR_SHARING=FORCE to the Oracle init file, 3) For Oracle 9i or 10g customer: add the following parameters to ar.conf or ar.cfg Oracle-Cursor-Sharing: FORCE 4) For all Windows (with 2 CPU) customers, Suggest to assign private RPC to RE (in ar.conf or ar.cfg) RE-RPC-Socket: 390621 Private-RPC-Socket: 390621 10 10 For UNIX platforms, RE-RPC-Socket: 390621 Private-RPC-Socket: 390621 6 6 As always, turning of your database and ARServer is a task for your dba, system administrator and Remedy administrator working in concert. The settings above have proved helpful in several cases, but they should be used as a starting point for investigation rather than standard configuration applicable in all environments. A defect SW00254374 has been logged for Engineering to consider whether this information or an official recommendation should be included in the shipped product docs. ** Scott. -Original Message- From: Action Request System discussion list(ARSList) [mailto:[EMAIL PROTECTED] On Behalf Of Lammey, Peter A. Sent: Tuesday, July 10, 2007 11:51 AM To: arslist@ARSLIST.ORG Subject: Re: ARS7 install with oracle 10G RAC ... the saga continues ... Does this lead anyone to believe or have people noticed other general activities in the applications are slow based on this? Thanks Peter Lammey ESPN MIT Technical Services Applications Management 860-766-4761 -Original Message- From: Action Request System discussion list(ARSList) [mailto:[EMAIL PROTECTED] On Behalf Of Axton Sent: Tuesday, July 10, 2007 11:47 AM To: arslist@ARSLIST.ORG Subject: Re: ARS7 install with oracle 10G RAC ... the saga continues ... The issue is widespread, esp. with a remote Oracle db. Our servers are switched with fiber/gb. Even if I have 2 vm's runnning on the same physical host (one db and one ars), I see the same issue. The packet logs do not show a problem with throughput and my switches show no congestion. Axton Grams On 7/10/07, Shellman, David [EMAIL PROTECTED] wrote: When I spoke to support about how long it took the system to start up they said that start up was chatty between the server app and the database. With a network link that running near capacity (our case), flaky router/switch, slow link, etc the length of time to respond between app server and fb server is increased.. All these situations are often not apparent with out getting network folks involved. Dave -- [EMAIL PROTECTED] (Wireless) - Original Message - From: Action Request System discussion list(ARSList) arslist@ARSLIST.ORG To: arslist@ARSLIST.ORG arslist@ARSLIST.ORG Sent: Tue Jul 10 10:19:29 2007 Subject: Re: ARS7 install with oracle 10G RAC ... the saga continues ... The problem does not seem to be in the way that the query to the db, but in the way the results are processed. I don't know about that. If the problem was the way that arserverd is processing the results of the query, why would startup be like 1 minute when the db is on the same box? My bet is that this has something to do with the way that arserverd is connecting to oracle ... perhaps incorrect use of the api? All that waiting between queries really makes me think there's a timeout somewhere ... like waiting for a local listener to respond, i.e. looking for the database to be on the same machine, and only querying remotely when the local connection times out. Man, this is why I sometimes hate dealing with proprietary software. If this was open, you could bet someone would have fixed it by now, and if they hadn't we'd at least have a fighting chance of fixing it ourselves. thanks again, everyone for all your help. looks like the only answer for now is the Run-DMC
Re: ARS7 install with oracle 10G RAC ... the saga continues ... KM-000010001381 (UNCLASSIFIED)
Correction: 2 AR Servers on the same DB i.e. for load balancing. On 7/10/07, Emad Zaky [EMAIL PROTECTED] wrote: For performance of applications on a single server it is ok. But I have noticed that if you run 2 AR servers with ITSM apps you get timeouts. Emad On 7/10/07, HARTWICK, SCOTT G CTR DISA JSSC [EMAIL PROTECTED] wrote: Classification: UNCLASSIFIED Caveats: NONE We have been having performance issues with our Oracle installation and found this KM article with some ideas on how to improve performance and initial configuration: * The following is information for ITSM 7.0.x installs within an Oracle environment: 1. Have your DBA set the SGA Size to min 250 and max 850. 2. Also, Knowledge Base entry KM-10001381 should prove to be helpful to you: * PROBLEM: Below are some settings which have been helpful for Oracle customers to avoid ARERR 91, 92, 93, 94 timeout issues, especially during application installations SOLUTION: 1) AR Server version 7.01 or AR Server version 6.03 patch 19 The above version or later contain new method of Oracle client calls which enhance performance 2) For Oracle 9i or 10g customer: add CURSOR_SHARING=FORCE to the Oracle init file, 3) For Oracle 9i or 10g customer: add the following parameters to ar.conf or ar.cfg Oracle-Cursor-Sharing: FORCE 4) For all Windows (with 2 CPU) customers, Suggest to assign private RPC to RE (in ar.conf or ar.cfg) RE-RPC-Socket: 390621 Private-RPC-Socket: 390621 10 10 For UNIX platforms, RE-RPC-Socket: 390621 Private-RPC-Socket: 390621 6 6 As always, turning of your database and ARServer is a task for your dba, system administrator and Remedy administrator working in concert. The settings above have proved helpful in several cases, but they should be used as a starting point for investigation rather than standard configuration applicable in all environments. A defect SW00254374 has been logged for Engineering to consider whether this information or an official recommendation should be included in the shipped product docs. ** Scott. -Original Message- From: Action Request System discussion list(ARSList) [mailto:[EMAIL PROTECTED] On Behalf Of Lammey, Peter A. Sent: Tuesday, July 10, 2007 11:51 AM To: arslist@ARSLIST.ORG Subject: Re: ARS7 install with oracle 10G RAC ... the saga continues ... Does this lead anyone to believe or have people noticed other general activities in the applications are slow based on this? Thanks Peter Lammey ESPN MIT Technical Services Applications Management 860-766-4761 -Original Message- From: Action Request System discussion list(ARSList) [mailto: [EMAIL PROTECTED] On Behalf Of Axton Sent: Tuesday, July 10, 2007 11:47 AM To: arslist@ARSLIST.ORG Subject: Re: ARS7 install with oracle 10G RAC ... the saga continues ... The issue is widespread, esp. with a remote Oracle db. Our servers are switched with fiber/gb. Even if I have 2 vm's runnning on the same physical host (one db and one ars), I see the same issue. The packet logs do not show a problem with throughput and my switches show no congestion. Axton Grams On 7/10/07, Shellman, David [EMAIL PROTECTED] wrote: When I spoke to support about how long it took the system to start up they said that start up was chatty between the server app and the database. With a network link that running near capacity (our case), flaky router/switch, slow link, etc the length of time to respond between app server and fb server is increased.. All these situations are often not apparent with out getting network folks involved. Dave -- [EMAIL PROTECTED] (Wireless) - Original Message - From: Action Request System discussion list(ARSList) arslist@ARSLIST.ORG To: arslist@ARSLIST.ORG arslist@ARSLIST.ORG Sent: Tue Jul 10 10:19:29 2007 Subject: Re: ARS7 install with oracle 10G RAC ... the saga continues ... The problem does not seem to be in the way that the query to the db, but in the way the results are processed. I don't know about that. If the problem was the way that arserverd is processing the results of the query, why would startup be like 1 minute when the db is on the same box? My bet is that this has something to do with the way that arserverd is connecting to oracle ... perhaps incorrect use of the api? All that waiting between queries really makes me think there's a timeout somewhere ... like waiting for a local listener to respond, i.e. looking for the database to be on the same machine, and only querying remotely when the local connection times out. Man, this is why I sometimes hate dealing with proprietary software. If this was open, you could
Re: ARS7 install with oracle 10G RAC ... the saga continues ... KM-000010001381 (UNCLASSIFIED)
FORCE was recommended On 7/10/07, Tony Worthington [EMAIL PROTECTED] wrote: Now I'm confused. There's a cursor sharing whitepaper Oracle's Cursor Sharing for BMC(r) Remedy(r) Products that specifies that it be set at SIMILAR. Now which is it BMC. SIMILAR or FORCE? Best practices mandate specifically that, with 6.0.1, users should use force, and with 6.3 or 7.0, users should use similar, but the difference between the two settings is small. The syntax of the cursor_sharing setting in the ar.conf file is as follows (set only one): Oracle-Cursor-Sharing: SIMILAR Oracle-Cursor-Sharing: FORCE -- Tony Worthington [EMAIL PROTECTED] 262-703-5911 HARTWICK, SCOTT G CTR DISA JSSC [EMAIL PROTECTED] Sent by: Action Request System discussion list(ARSList) arslist@ARSLIST.ORG 07/10/2007 11:02 AM Please respond to arslist@ARSLIST.ORG To arslist@ARSLIST.ORG cc Subject Re: ARS7 install with oracle 10G RAC ... the saga continues ... KM-10001381 (UNCLASSIFIED) Classification: UNCLASSIFIED Caveats: NONE We have been having performance issues with our Oracle installation and found this KM article with some ideas on how to improve performance and initial configuration: * The following is information for ITSM 7.0.x installs within an Oracle environment: 1. Have your DBA set the SGA Size to min 250 and max 850. 2. Also, Knowledge Base entry KM-10001381 should prove to be helpful to you: * PROBLEM: Below are some settings which have been helpful for Oracle customers to avoid ARERR 91, 92, 93, 94 timeout issues, especially during application installations SOLUTION: 1) AR Server version 7.01 or AR Server version 6.03 patch 19 The above version or later contain new method of Oracle client calls which enhance performance 2) For Oracle 9i or 10g customer: add CURSOR_SHARING=FORCE to the Oracle init file, 3) For Oracle 9i or 10g customer: add the following parameters to ar.conf or ar.cfg Oracle-Cursor-Sharing: FORCE 4) For all Windows (with 2 CPU) customers, Suggest to assign private RPC to RE (in ar.conf or ar.cfg) RE-RPC-Socket: 390621 Private-RPC-Socket: 390621 10 10 For UNIX platforms, RE-RPC-Socket: 390621 Private-RPC-Socket: 390621 6 6 As always, turning of your database and ARServer is a task for your dba, system administrator and Remedy administrator working in concert. The settings above have proved helpful in several cases, but they should be used as a starting point for investigation rather than standard configuration applicable in all environments. A defect SW00254374 has been logged for Engineering to consider whether this information or an official recommendation should be included in the shipped product docs. ** Scott. -Original Message- From: Action Request System discussion list(ARSList) [mailto:[EMAIL PROTECTED] On Behalf Of Lammey, Peter A. Sent: Tuesday, July 10, 2007 11:51 AM To: arslist@ARSLIST.ORG Subject: Re: ARS7 install with oracle 10G RAC ... the saga continues ... Does this lead anyone to believe or have people noticed other general activities in the applications are slow based on this? Thanks Peter Lammey ESPN MIT Technical Services Applications Management 860-766-4761 -Original Message- From: Action Request System discussion list(ARSList) [mailto:[EMAIL PROTECTED] On Behalf Of Axton Sent: Tuesday, July 10, 2007 11:47 AM To: arslist@ARSLIST.ORG Subject: Re: ARS7 install with oracle 10G RAC ... the saga continues ... The issue is widespread, esp. with a remote Oracle db. Our servers are switched with fiber/gb. Even if I have 2 vm's runnning on the same physical host (one db and one ars), I see the same issue. The packet logs do not show a problem with throughput and my switches show no congestion. Axton Grams On 7/10/07, Shellman, David [EMAIL PROTECTED] wrote: When I spoke to support about how long it took the system to start up they said that start up was chatty between the server app and the database. With a network link that running near capacity (our case), flaky router/switch, slow link, etc the length of time to respond between app server and fb server is increased.. All these situations are often not apparent with out getting network folks involved. Dave -- [EMAIL PROTECTED] (Wireless) - Original Message - From: Action Request System discussion list(ARSList) arslist@ARSLIST.ORG To: arslist@ARSLIST.ORG arslist@ARSLIST.ORG Sent: Tue Jul 10 10:19:29 2007 Subject: Re: ARS7 install with oracle 10G RAC ... the saga continues ... The problem does not seem to be in the way that the query to the db, but in the way the results are processed. I don't know about that. If the problem was the way that arserverd is processing the results of the query, why would startup
Re: ARS7 install with oracle 10G RAC ... the saga continues ... KM-000010001381 (UNCLASSIFIED)
The only problem I see with storing the clob in row is that a table scan will now include the clob data. With the way remedy is structured, there are tons of full table scans. When you allow the user to query a table on any value with wildcards, that's what you get. The only advantage I see to storing the clob in row is that when a clob is accessed, you drop some of the reads from disk, namely the address space where the clob is stored, which is small in comparison. I would be interested to see a benchmark between the two configurations for an ITSM 7 server with a production load in terms of consistent gets, physical reads, disk io and the likes. Axton Grams On 7/10/07, Rick Cook [EMAIL PROTECTED] wrote: ** Something else to consider is that the CLOB-in-row setting (which you should also consider setting), will only have its full effect if set prior to installation. When running our initial Discovery, we noticed that our DB had increased 10x in size - far more than the data would have indicated. Changing the CLOB-in-row setting on an EXISTING DB did NOT recover the wasted disk space. There is an RFE in now on the fact that Remedy does not use Oracle bind parameters as often as it could, which really hurts performance. I bet that the Remedy engineers kind of wish Oracle would act like the rest of the DBs, but that's probably a common wish across many platforms. Rick On 7/10/07, HARTWICK, SCOTT G CTR DISA JSSC [EMAIL PROTECTED] wrote: Classification: UNCLASSIFIED Caveats: NONE We have been having performance issues with our Oracle installation and found this KM article with some ideas on how to improve performance and initial configuration: * The following is information for ITSM 7.0.x installs within an Oracle environment: 1. Have your DBA set the SGA Size to min 250 and max 850. 2. Also, Knowledge Base entry KM-10001381 should prove to be helpful to you: * PROBLEM: Below are some settings which have been helpful for Oracle customers to avoid ARERR 91, 92, 93, 94 timeout issues, especially during application installations SOLUTION: 1) AR Server version 7.01 or AR Server version 6.03 patch 19 The above version or later contain new method of Oracle client calls which enhance performance 2) For Oracle 9i or 10g customer: add CURSOR_SHARING=FORCE to the Oracle init file, 3) For Oracle 9i or 10g customer: add the following parameters to ar.conf or ar.cfg Oracle-Cursor-Sharing: FORCE 4) For all Windows (with 2 CPU) customers, Suggest to assign private RPC to RE (in ar.conf or ar.cfg) RE-RPC-Socket: 390621 Private-RPC-Socket: 390621 10 10 For UNIX platforms, RE-RPC-Socket: 390621 Private-RPC-Socket: 390621 6 6 As always, turning of your database and ARServer is a task for your dba, system administrator and Remedy administrator working in concert. The settings above have proved helpful in several cases, but they should be used as a starting point for investigation rather than standard configuration applicable in all environments. A defect SW00254374 has been logged for Engineering to consider whether this information or an official recommendation should be included in the shipped product docs. ** Scott. -Original Message- From: Action Request System discussion list(ARSList) [mailto:[EMAIL PROTECTED] On Behalf Of Lammey, Peter A. Sent: Tuesday, July 10, 2007 11:51 AM To: arslist@ARSLIST.ORG Subject: Re: ARS7 install with oracle 10G RAC ... the saga continues ... Does this lead anyone to believe or have people noticed other general activities in the applications are slow based on this? Thanks Peter Lammey ESPN MIT Technical Services Applications Management 860-766-4761 -Original Message- From: Action Request System discussion list(ARSList) [mailto: [EMAIL PROTECTED] On Behalf Of Axton Sent: Tuesday, July 10, 2007 11:47 AM To: arslist@ARSLIST.ORG Subject: Re: ARS7 install with oracle 10G RAC ... the saga continues ... The issue is widespread, esp. with a remote Oracle db. Our servers are switched with fiber/gb. Even if I have 2 vm's runnning on the same physical host (one db and one ars), I see the same issue. The packet logs do not show a problem with throughput and my switches show no congestion. Axton Grams On 7/10/07, Shellman, David [EMAIL PROTECTED] wrote: When I spoke to support about how long it took the system to start up they said that start up was chatty between the server app and the database. With a network link that running near capacity (our case), flaky router/switch, slow link, etc the length of time to respond between app server and fb server is increased.. All these situations are often not apparent with out getting network folks involved. Dave
Re: ARS7 install with oracle 10G RAC ... the saga continues ... KM-000010001381 (UNCLASSIFIED)
Setting the cursor_sharing parameter to similar or force will force the sql to be parsed using bind variables, though you introduce a parse for every sql statement that gets dispatched to the db, which introduces additional cpu overhead. Ideally, Remedy would generate the sql, when using an oracle backend, so that the bind variable were defined in the sql when it was sent to the db. Axton Grams On 7/10/07, Rick Cook [EMAIL PROTECTED] wrote: ** Something else to consider is that the CLOB-in-row setting (which you should also consider setting), will only have its full effect if set prior to installation. When running our initial Discovery, we noticed that our DB had increased 10x in size - far more than the data would have indicated. Changing the CLOB-in-row setting on an EXISTING DB did NOT recover the wasted disk space. There is an RFE in now on the fact that Remedy does not use Oracle bind parameters as often as it could, which really hurts performance. I bet that the Remedy engineers kind of wish Oracle would act like the rest of the DBs, but that's probably a common wish across many platforms. Rick On 7/10/07, HARTWICK, SCOTT G CTR DISA JSSC [EMAIL PROTECTED] wrote: Classification: UNCLASSIFIED Caveats: NONE We have been having performance issues with our Oracle installation and found this KM article with some ideas on how to improve performance and initial configuration: * The following is information for ITSM 7.0.x installs within an Oracle environment: 1. Have your DBA set the SGA Size to min 250 and max 850. 2. Also, Knowledge Base entry KM-10001381 should prove to be helpful to you: * PROBLEM: Below are some settings which have been helpful for Oracle customers to avoid ARERR 91, 92, 93, 94 timeout issues, especially during application installations SOLUTION: 1) AR Server version 7.01 or AR Server version 6.03 patch 19 The above version or later contain new method of Oracle client calls which enhance performance 2) For Oracle 9i or 10g customer: add CURSOR_SHARING=FORCE to the Oracle init file, 3) For Oracle 9i or 10g customer: add the following parameters to ar.conf or ar.cfg Oracle-Cursor-Sharing: FORCE 4) For all Windows (with 2 CPU) customers, Suggest to assign private RPC to RE (in ar.conf or ar.cfg) RE-RPC-Socket: 390621 Private-RPC-Socket: 390621 10 10 For UNIX platforms, RE-RPC-Socket: 390621 Private-RPC-Socket: 390621 6 6 As always, turning of your database and ARServer is a task for your dba, system administrator and Remedy administrator working in concert. The settings above have proved helpful in several cases, but they should be used as a starting point for investigation rather than standard configuration applicable in all environments. A defect SW00254374 has been logged for Engineering to consider whether this information or an official recommendation should be included in the shipped product docs. ** Scott. -Original Message- From: Action Request System discussion list(ARSList) [mailto:[EMAIL PROTECTED] On Behalf Of Lammey, Peter A. Sent: Tuesday, July 10, 2007 11:51 AM To: arslist@ARSLIST.ORG Subject: Re: ARS7 install with oracle 10G RAC ... the saga continues ... Does this lead anyone to believe or have people noticed other general activities in the applications are slow based on this? Thanks Peter Lammey ESPN MIT Technical Services Applications Management 860-766-4761 -Original Message- From: Action Request System discussion list(ARSList) [mailto: [EMAIL PROTECTED] On Behalf Of Axton Sent: Tuesday, July 10, 2007 11:47 AM To: arslist@ARSLIST.ORG Subject: Re: ARS7 install with oracle 10G RAC ... the saga continues ... The issue is widespread, esp. with a remote Oracle db. Our servers are switched with fiber/gb. Even if I have 2 vm's runnning on the same physical host (one db and one ars), I see the same issue. The packet logs do not show a problem with throughput and my switches show no congestion. Axton Grams On 7/10/07, Shellman, David [EMAIL PROTECTED] wrote: When I spoke to support about how long it took the system to start up they said that start up was chatty between the server app and the database. With a network link that running near capacity (our case), flaky router/switch, slow link, etc the length of time to respond between app server and fb server is increased.. All these situations are often not apparent with out getting network folks involved. Dave -- [EMAIL PROTECTED] (Wireless) - Original Message - From: Action Request System discussion list(ARSList) arslist@ARSLIST.ORG To: arslist@ARSLIST.ORG arslist@ARSLIST.ORG Sent: Tue Jul 10 10:19:29 2007 Subject: Re: ARS7 install
Re: ARS7 install with oracle 10G RAC ... the saga continues ...
Please search the list archives at www.arslist.org or google groups. This has been discussed at length, and the slow startup with a remote Oracle DB is not uncommon. Our systems take about 8 minutes. Even tickets submitted to support confirm this same delay. They sometimes suggest to ensure all NIC's are set to full duplex, but that's about it. Sorry... tony -- Tony Worthington [EMAIL PROTECTED] 262-703-5911 Andrew Hicox [EMAIL PROTECTED] Sent by: Action Request System discussion list(ARSList) arslist@ARSLIST.ORG 07/09/2007 03:00 PM Please respond to arslist@ARSLIST.ORG To arslist@ARSLIST.ORG cc Subject ARS7 install with oracle 10G RAC ... the saga continues ... ** Hello everyone: I've managed to get through the (incredibly flawed) solaris 10 arsystem 7 install. This is 7.0.01 patch 2. I'm using oracle 10G client library with a remote database on an oracle 10 Rapid Application Cluster. So ... now the server is INCREDIBLY slow to start. The server is configured to run on a TCP port (1). When I issue the 'arsystem start' command, I get the usual output on the console indicating that the server has started. However, the TCP port is not listening, and of course, no clients can connect to the server. If I wait (approx 7 minutes), I will see the following output on the console: (ARNOTE 0) Server indicates that it's up. (ARNOTE 0) ARMonitor child process (pid:29595) started. ./arplugin Action Request System(R) Fork Daemon Version 7.0.01 patch 002 200704021644 Copyright (c) 2000 - 2006 BMC Software, Inc. All rights reserved. Action Request System(R) Plug-In Server Version 7.0.01 patch 002 200704021644 Copyright (c) 2001 - 2006 BMC Software, Inc. All rights reserved. Loaded Web Services plugin properly After the above, the TCP port opens up and clients can connect, and it seems to work well enough. What I presume is happening here is that arserverd must issue some sort of indication that it's completed start up so that arforkd and armonitor can do their thing and start listeners and plugins, and that arserverd is for some reason taking on average 7 minutes to do that. I can't manage to get ANY sort of logging during the 7 minutes of mystery. is there some way to figgure out what the heck is going on durring this period of time (is it timing out DNS? is the db slow to respond? is it waiting on disk access? what is it doing?). We have an identical set up in our dev lab, with the exception that the database server is not a RAC and it's installed on the same machine. The startup is quite snappy on that setup. There don't appear to be any issues with network latency between the remedy server and the RAC. Anyone ever seen this before? Any suggestions? thanks, Andrew __20060125___This posting was submitted with HTML in it___ CONFIDENTIALITY NOTICE: This is a transmission from Kohl's Department Stores, Inc. and may contain information which is confidential and proprietary. If you are not the addressee, any disclosure, copying or distribution or use of the contents of this message is expressly prohibited. If you have received this transmission in error, please destroy it and notify us immediately at 262-703-7000. CAUTION: Internet and e-mail communications are Kohl's property and Kohl's reserves the right to retrieve and read any message created, sent and received. Kohl's reserves the right to monitor messages to or from authorized Kohl's Associates at any time without any further consent. ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org ARSlist:Where the Answers Are
Re: ARS7 install with oracle 10G RAC ... the saga continues ...
Yes..I have seen this before. Can you login to the Admin tool after the server starts up? On 7/9/07, Andrew Hicox [EMAIL PROTECTED] wrote: ** Hello everyone: I've managed to get through the (incredibly flawed) solaris 10 arsystem 7 install. This is 7.0.01 patch 2. I'm using oracle 10G client library with a remote database on an oracle 10 Rapid Application Cluster. So ... now the server is INCREDIBLY slow to start. The server is configured to run on a TCP port (1). When I issue the 'arsystem start' command, I get the usual output on the console indicating that the server has started. However, the TCP port is not listening, and of course, no clients can connect to the server. If I wait (approx 7 minutes), I will see the following output on the console: (ARNOTE 0) Server indicates that it's up. (ARNOTE 0) ARMonitor child process (pid:29595) started. ./arplugin Action Request System(R) Fork Daemon Version 7.0.01 patch 002 200704021644 Copyright (c) 2000 - 2006 BMC Software, Inc. All rights reserved. Action Request System(R) Plug-In Server Version 7.0.01 patch 002 200704021644 Copyright (c) 2001 - 2006 BMC Software, Inc. All rights reserved. Loaded Web Services plugin properly After the above, the TCP port opens up and clients can connect, and it seems to work well enough. What I presume is happening here is that arserverd must issue some sort of indication that it's completed start up so that arforkd and armonitor can do their thing and start listeners and plugins, and that arserverd is for some reason taking on average 7 minutes to do that. I can't manage to get ANY sort of logging during the 7 minutes of mystery. is there some way to figgure out what the heck is going on durring this period of time (is it timing out DNS? is the db slow to respond? is it waiting on disk access? what is it doing?). We have an identical set up in our dev lab, with the exception that the database server is not a RAC and it's installed on the same machine. The startup is quite snappy on that setup. There don't appear to be any issues with network latency between the remedy server and the RAC. Anyone ever seen this before? Any suggestions? thanks, Andrew __20060125___This posting was submitted with HTML in it___ ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org ARSlist:Where the Answers Are
Re: ARS7 install with oracle 10G RAC ... the saga continues ...
Andrew, As noted been there done that when or app server and the database server were some distance apart by network. It would take over 5 minutes to see the arfork line scroll by. Once the two servers were moved to the same room, start up time dropped to around 30 seconds. Dave -- [EMAIL PROTECTED] (Wireless) - Original Message - From: Action Request System discussion list(ARSList) arslist@ARSLIST.ORG To: arslist@ARSLIST.ORG arslist@ARSLIST.ORG Sent: Mon Jul 09 16:08:12 2007 Subject: Re: ARS7 install with oracle 10G RAC ... the saga continues ... Please search the list archives at www.arslist.org or google groups. This has been discussed at length, and the slow startup with a remote Oracle DB is not uncommon. Our systems take about 8 minutes. Even tickets submitted to support confirm this same delay. They sometimes suggest to ensure all NIC's are set to full duplex, but that's about it. Sorry... tony -- Tony Worthington [EMAIL PROTECTED] 262-703-5911 Andrew Hicox [EMAIL PROTECTED] Sent by: Action Request System discussion list(ARSList) arslist@ARSLIST.ORG 07/09/2007 03:00 PM Please respond to arslist@ARSLIST.ORG To arslist@ARSLIST.ORG cc Subject ARS7 install with oracle 10G RAC ... the saga continues ... ** Hello everyone: I've managed to get through the (incredibly flawed) solaris 10 arsystem 7 install. This is 7.0.01 patch 2. I'm using oracle 10G client library with a remote database on an oracle 10 Rapid Application Cluster. So ... now the server is INCREDIBLY slow to start. The server is configured to run on a TCP port (1). When I issue the 'arsystem start' command, I get the usual output on the console indicating that the server has started. However, the TCP port is not listening, and of course, no clients can connect to the server. If I wait (approx 7 minutes), I will see the following output on the console: (ARNOTE 0) Server indicates that it's up. (ARNOTE 0) ARMonitor child process (pid:29595) started. ./arplugin Action Request System(R) Fork Daemon Version 7.0.01 patch 002 200704021644 Copyright (c) 2000 - 2006 BMC Software, Inc. All rights reserved. Action Request System(R) Plug-In Server Version 7.0.01 patch 002 200704021644 Copyright (c) 2001 - 2006 BMC Software, Inc. All rights reserved. Loaded Web Services plugin properly After the above, the TCP port opens up and clients can connect, and it seems to work well enough. What I presume is happening here is that arserverd must issue some sort of indication that it's completed start up so that arforkd and armonitor can do their thing and start listeners and plugins, and that arserverd is for some reason taking on average 7 minutes to do that. I can't manage to get ANY sort of logging during the 7 minutes of mystery. is there some way to figgure out what the heck is going on durring this period of time (is it timing out DNS? is the db slow to respond? is it waiting on disk access? what is it doing?). We have an identical set up in our dev lab, with the exception that the database server is not a RAC and it's installed on the same machine. The startup is quite snappy on that setup. There don't appear to be any issues with network latency between the remedy server and the RAC. Anyone ever seen this before? Any suggestions? thanks, Andrew __20060125___This posting was submitted with HTML in it___ CONFIDENTIALITY NOTICE: This is a transmission from Kohl's Department Stores, Inc. and may contain information which is confidential and proprietary. If you are not the addressee, any disclosure, copying or distribution or use of the contents of this message is expressly prohibited. If you have received this transmission in error, please destroy it and notify us immediately at 262-703-7000. CAUTION: Internet and e-mail communications are Kohl's property and Kohl's reserves the right to retrieve and read any message created, sent and received. Kohl's reserves the right to monitor messages to or from authorized Kohl's Associates at any time without any further consent. ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org ARSlist:Where the Answers Are
Re: ARS7 install with oracle 10G RAC ... the saga continues ...
Remedy fetches information from it's data dictionary (e.g., field_disp_prop, field, etc.), then processes it in chunks of 100, 500, or 1000. So if you have a table field_dispprop that has 120k records and 500 records are processed at a time, 15 seconds per chunk, you end up with a start time that totals 3600 seconds (for that part of the startup). If you add a -t parameter to arserverd in the armonitor.conf file, a special startup log file is created. The log file will look something like this: Jul 09 2007 18:29:06.6577 Startup TID:3086968512 Set up thread control block with key = 0 Jul 09 2007 18:29:06.6582 Startup TID:3086968512 Initialize thread local storage block Jul 09 2007 18:29:06.6583 Startup TID:3086968512 Initialize mutiple-byte environment Jul 09 2007 18:29:06.6585 Startup TID:3086968512 InstallDir = /u01/arsystem/arsdev Jul 09 2007 18:29:06.6587 Startup TID:3086968512 Initialize Server utility Jul 09 2007 18:29:06.6599 Startup TID:3086968512 Initialize License Library Jul 09 2007 18:29:06.6602 Startup TID:3086968512 LicenseFilename = /etc/arsystem/arsdev/arsystem.lic Jul 09 2007 18:29:06.6603 Startup TID:3086968512 Initialize Language setting and locale Jul 09 2007 18:29:06.6610 Startup TID:3086968512 Initialize the Decimal Math libray Jul 09 2007 18:29:06.6610 Startup TID:3086968512 Open shared catalog Jul 09 2007 18:29:06.6648 Startup TID:3086968512 Load encryption shared library Jul 09 2007 18:29:06.6653 Startup TID:3086968512 Load encryption static functions Jul 09 2007 18:29:06.6654 Startup TID:3086968512 Initialize default configuration information Jul 09 2007 18:29:06.6654 Startup TID:3086968512 Create Mutexes Jul 09 2007 18:29:06.6655 Startup TID:3086968512 Initialize parse environment Jul 09 2007 18:29:06.6657 Startup TID:3086968512 Initialize date time information Jul 09 2007 18:29:06.6659 Startup TID:3086968512 Initialize notification strings Jul 09 2007 18:29:06.6659 Startup TID:3086968512 Initialize RPC queue type strings Jul 09 2007 18:29:06.6660 Startup TID:3086968512 Initialize filter strings Jul 09 2007 18:29:06.6662 Startup TID:3086968512 Initialize escalation strings Jul 09 2007 18:29:06.6662 Startup TID:3086968512 Load system configuration file Jul 09 2007 18:29:06.6669 Startup TID:3086968512 arsCodeSet=windows-1252[0] arDbCodeSet=windows-1252[0] Jul 09 2007 18:29:06.6670 Startup TID:3086968512 Initialize pending lists Jul 09 2007 18:29:06.6671 Startup TID:3086968512 Create cache read write lock Jul 09 2007 18:29:06.6671 Startup TID:3086968512 Create full text status read write lock Jul 09 2007 18:29:06.6672 Startup TID:3086968512 Initialize fork proxy Jul 09 2007 18:29:06.6672 Startup TID:3086968512 Check licensing Jul 09 2007 18:29:06.6693 Startup TID:3086968512 Initialize user cache Jul 09 2007 18:29:06.6707 Startup TID:3086968512 Open log file Jul 09 2007 18:29:06.6745 Startup TID:3086968512 Initialize XML parser Jul 09 2007 18:29:06.6782 Startup TID:3086968512 Initialize thread list Jul 09 2007 18:29:06.6784 Startup TID:3086968512 Check multiple servers Jul 09 2007 18:29:06.6787 Startup TID:3086968512 Initialize dead thread list Jul 09 2007 18:29:06.6790 Startup TID:3086968512 Initialize timed calls Jul 09 2007 18:29:06.6792 Startup TID:3086968512 CreateRPCQueue(min=1, max=1, rpc=390600) Jul 09 2007 18:29:06.6793 Startup TID:3086968512 Await cache ready... Jul 09 2007 18:29:06.8988 Startup TID:0030571424 InitializeServerCache: LoadInitialSchemaInfo Begin Jul 09 2007 18:29:11.7241 Startup TID:0030571424 Begin LoadDisplayInfoList Jul 09 2007 18:29:25.6076 Startup TID:0030571424 LoadDisplayInfoList: 500 rows Jul 09 2007 18:29:42.7008 Startup TID:0030571424 LoadDisplayInfoList: 1000 rows If you then review the sql that is sent to the db while the server is unavailable, you will see this statement, followed by silence: SELECT schemaId,fieldId,vuiId,propShort,propLong FROM field_dispprop ORDER BY 1 ASC, 2 ASC, 3 ASC What is happening here is that arserverd fetches the field display property info, then processes it 500 records at a time. Depending on the size of your server (number of forms, views, fields, etc), there may be a couple of sql statements that are followed by a pause. What is happening internally is anyone's guess; I would guess that they are processing X rows at a time to operate within certain memory constraints, and 'maybe' in a horribly inefficient way. Some things are for certain though; (1) a lot of data is transferred from the db to the arserver. My test server, which has 3293 rows in field_dispprop (pretty much a base arserver install) sends 411K of data over the wire. (2) the field_dispprop data is processed in chunks of 500 (at least on Linux). Axton Grams On 7/9/07, Andrew Hicox [EMAIL PROTECTED] wrote: ** Hello everyone: I've managed to get through the (incredibly flawed) solaris 10 arsystem 7 install. This is 7.0.01 patch 2. I'm using oracle 10G client library with a remote database on an oracle 10 Rapid Application Cluster. So
