Just catching up on this thread that I overlooked... Understandably since the installer may need to shuffle data into temporary tables and back. Production probably had ~100,000 more records in CMDB staging datasets than test. It is possible that a back end forms like FB:History had way more records in one system than the other however the db sizes were pretty close. I am not sure a difference in number of records still explains the times when the installer log and a db trace are not showing any activity.
Jason On Thu, Oct 20, 2011 at 7:15 PM, pritch <pri...@ptd.net> wrote: > You may want to compare the data in the system between your test and prod > envs. I'm not a DBA, but what I've seen seems to be that the more rows in > your db that need to have columns added / changed / etc, the longer it > takes. > > ----- Original Message ----- > From: "Jason Miller" <jason.mil...@gmail.com> > To: arslist@ARSLIST.ORG > Sent: Thursday, October 20, 2011 6:01:43 PM > Subject: Re: ARS 7.6.04 SP2 Solaris installer Oracle db. > > ** The hardware size appears to have little to do with the install times. > Of course there is some type of relation between hardware and time but I > can't figure it out. > > In our case the production db is a physical server with 24 cores and 48 gb > of ram and test is a VM with 4 CPUs assigned and 8 gb of ram. The > production app server is a VM with 2 cores and 6 gb of ram and test is a VM > with 2 cores and 4 gb of ram. There is a little more data in the > production db vs. the test db. I would think this would fairly negligible > since both were not really being used. > > I scheduled our production outage for 48 hours (with a little extra time) > based on the time it took to perform the same steps in test with the > additional step of running BPCU (I skipped this in production since we had > about 6 custom/modified objects that I converted manually). Test took a > little less than 48 hours and this included ironing out the process and had > the extra BPCU time. I was pretty certain upgrading production would be > much faster. It turns out that it took production longer than a lighter > weight test environment. > > Jason > > > On Thu, Oct 20, 2011 at 1:09 PM, strauss < stra...@unt.edu > wrote: > > > ** > > > > I restored my test/sample data system back to where ITSM was still SP1 > (ARS and Atrium at SP2) and performed two manual interventions before > kicking off ITSM 7.6.04 SP2 again: > > > > Delete join form VIS:Status_Stage_Flow_TransitionRulesLookup which is > carrying an outdated View ID for the Default Administrator View (the only > view present) > > I am told that this was _ supposed _ to have been changed in the > upgrade to 7.6.03, but none of my systems reflect that change. > > > > Delete all existing records in DLD:SYS:DataLoadStatus that were preventing > SP2 from creating a unique index on InstanceId > > There were 126 old records in there with GUIDs that are NOT UNIQUE on a > case-insensitive system (SQL Server). > > Looking at the production system which had no records in that form, it > actually looks like the unique index was created by the upgrade to > 7.6.04.0000 – which is when I have been told it was introduced. Somehow, > my test system got past that and SP1 both without the installation hanging > – until SP2. > > > > The installation of ITSM 7.6.04 SP2 has been running since 10:30 AM; we’ll > see how long it takes, but this is on a low powered VM and isn’t a good > measure of what it will take on production grade equipment. From what > you’re saying, Pat, it doesn’t do very well on MUCH higher-powered servers, > either. I guess the Thanksgiving holiday will be my first window of > opportunity for applying a service pack that runs for hours and hours and > hours to the production environment. > > > > > Christopher Strauss, Ph.D. > Call Tracking Administration Manager > University of North Texas Computing & IT Center > http://itsm.unt.edu/ > > > From: Action Request System discussion list(ARSList) [mailto: > arslist@ARSLIST.ORG ] On Behalf Of patrick zandi > Sent: Thursday, October 20, 2011 2:49 PM > > > > To: arslist@ARSLIST.ORG > Subject: Re: ARS 7.6.04 SP2 Solaris installer Oracle db. > > > > > > > > > ** Yeah::: Upgrades look like they are going to disappear for the create a > new server and migrate data option soon. > > Most change windows: especially Enterprise wise:: could they really go > down for 5-7 hours? NO! not happening.. > so My guess is there is some sort of single FIFO job process single Java > thread into the server needs to be change to something a little more > robust. don't know what the answer is.. but I can see this as an issue... > > I have 64 CPU's and it took forever.. cpu never went over 6%... had tons > of ram too... > not sure what the answer is ? BMC Engineering might have a better clue... > > > On Thu, Oct 20, 2011 at 3:25 PM, Jason Miller < jason.mil...@gmail.com > > wrote: > > ** Congratulations! > > Could somebody from BMC explain what is happening during this time where > nothing appears to be happening? I remember having the same experience > during the 7.5 to 7.6.04 SP1 upgrade. The logs do not indicate that > anything is happening and a db trace shows very little activity. You just > have to hold your breath, have faith and hope for the best. > > Below is a snippet of our production upgrade notes the times of each > install. Fortunately I was able to take the system down for two days > because we only had three people using this new system for production data > (contracts). We finished just a hair past our 9 AM time to return to > production (I got lucky). > ++++++++++++++++++++++++++++++++++++ > Run ARSystem_7604SP1_Windows (5/25/2011 14:02 - 16:00) > • A handful of errors: > § arerror.log > § Wed May 25 15:01:05 2011 390600 : Could not find or read > the tags file. (ARNOTE 452) > § Wed May 25 15:01:05 2011 C:\Program Files\Common > Files\AR System\Licenses\<servername>\arsystem.tag > § Wed May 25 15:02:03 2011 390600 : The value(s) for this > entry violate a unique index that has been defined for this form (ARERR 382) > § Wed May 25 15:02:02 2011 390600 : Creation of an > application form failed. (ARERR 4550) > § An error regarding the email engine. > Run atrium7604SP1.win (5/25/2011 23:17 - 5/26/2011 13:21) > • RIK hung at 00:23, acknowledged and process started again at 08:03 > • Finished without further error > • The issues in test with Active Link guides after Atrium Core did not > happen in production > • Create Windows symbolic link to account for AIE directory change > § mklink /J "D:\Program Files (x86)\BMC > Software\AtriumCore\aie\service" "D:\Program Files (x86)\BMC > Software\AtriumCore\aie\service64" > Run BMCRemedyITSMSuite7604SP1.windows (5/26/2011 14:00 - 5/27/2011 07:33) > Run ServiceLevelManagement7.6.04.SP1.windows (5/27/2011 08:07 - 09:02) > Run BmcRkm7604SP1.windows (5/29/2001 23:30 - 5/30/2011 02:04) > Run atriumintegrator7604SP1.client_server.win / > atriumintegrator7604.server.win (6/4/2011 20:56 - 21:30) > ++++++++++++++++++++++++++++++++++++ > > Jason > > > On Thu, Oct 20, 2011 at 5:38 AM, patrick zandi < remedy...@gmail.com > > wrote: > > ** ok, it finished: I ran it over night// had several warnings.. but no > failures.. > > > > off to the logs.. total time :: 7.5 hours > > > > > > > On Wed, Oct 19, 2011 at 3:09 PM, strauss < stra...@unt.edu > wrote: > > ** > > > > My installs are failing after only 70 or 80 minutes. I cleaned up the > first defect and it has now failed on another. Apparently SP2 depends on > things that failed to happen properly during my 7.6.03 upgrades from > earlier versions, so it is unable to complete. Nothing here to instill > confidence for planning a rapid move to SP2 in the production environment. > > > > > Christopher Strauss, Ph.D. > Call Tracking Administration Manager > University of North Texas Computing & IT Center > http://itsm.unt.edu/ > > > From: Action Request System discussion list(ARSList) [mailto: > arslist@ARSLIST.ORG ] On Behalf Of patrick zandi > Sent: Wednesday, October 19, 2011 1:56 PM > > > > To: arslist@ARSLIST.ORG > Subject: Re: ARS 7.6.04 SP2 Solaris installer Oracle db. > > > > ** OK back at it again::: According to Support::: > > > > > so now I am asking again::: > ENV: 64 cpu's 32 gig of ram > > How many hours is the task update suppose to take? 6 - 8 - 12 - 24 hours? > Cause were over 5 already.. > > > > > On Tue, Oct 18, 2011 at 6:11 PM, Patrick Zandi < remedy...@gmail.com > > wrote: > > > > So they broke it twice ! Lol > > > > > Sent from my iPhone so typo's or funky words can and do happen! > > > > > On Oct 18, 2011, at 6:02 PM, strauss < stra...@unt.edu > wrote: > > > > > > > ** > > > BMC Support indicates that there is a known defect with a view ID that > basically prevents the ITSM SP2 installer from completing successfully. It > looks like it will affect all of my available environments. > > > > Christopher Strauss, Ph.D. > Call Tracking Administration Manager > University of North Texas Computing & IT Center > http://itsm.unt.edu/ > > > From: Action Request System discussion list(ARSList) [mailto: > arslist@ARSLIST.ORG ] On Behalf Of strauss > Sent: Tuesday, October 18, 2011 1:43 PM > To: arslist@ARSLIST.ORG > Subject: Re: ARS 7.6.04 SP2 Solaris installer Oracle db. > > > > ** > > The rik.err error was: > > > > Definition File (D:\Program Files (x86)\BMC > Software\BMCRemedyITSMSuite\ITSMT01\Workflow\install\7.6.04_sp2_overlay_install\systems\fnd\workflow\en\.\fnd_deplapp.def) > import failed with error code (392) :- (Field/VUI name must be unique for > the form -- there is already a field or VUI using this name) > > > > Since there are almost 300 forms in that def file, I’m going to let BMC > figured out what they did wrong. The only form I overlaid on the server is > HPD:Help Desk (one field, Reported Source was given Other as a Default > value) and one of the 7.6.04 upgrades added an overlay for > HPD:HelpDesk_AuditLogSystem, and neither of those is in the problematic def > file. > > > > Christopher Strauss, Ph.D. > Call Tracking Administration Manager > University of North Texas Computing & IT Center > http://itsm.unt.edu/ > > > From: Action Request System discussion list(ARSList) [mailto: > arslist@ARSLIST.ORG ] On Behalf Of strauss > Sent: Tuesday, October 18, 2011 1:19 PM > To: arslist@ARSLIST.ORG > Subject: Re: ARS 7.6.04 SP2 Solaris installer Oracle db. > > > > ** > > Oh, I doubt that (that there is any hope for me)… my downloads are from > last week. The Atrium SP2 applied clean on the second try, but the ITSM > SP2 just completed with a failure. So far the SP2 installers are NOT > looking good at all. > > > > Christopher Strauss, Ph.D. > Call Tracking Administration Manager > University of North Texas Computing & IT Center > http://itsm.unt.edu/ > > > From: Action Request System discussion list(ARSList) [mailto: > arslist@ARSLIST.ORG ] On Behalf Of patrick zandi > Sent: Tuesday, October 18, 2011 1:11 PM > To: arslist@ARSLIST.ORG > Subject: Re: ARS 7.6.04 SP2 Solaris installer Oracle db. > > > > ** you will not believe it:: LIKE THIS HAS NEVER HAPPENED:::: > > Files uploaded to the download site are BAD::: they just replaced them > with the correct files yesterday?!? I guess.. > Go Figure.. > > So I have re reinstall everything Again.. Wonderful.. So there is hope for > ya Chris!!! > LOL........ > > > On Tue, Oct 18, 2011 at 1:29 PM, Grooms, Frederick W < > frederick.w.gro...@xo.com > wrote: > > ** > > > > Do you have an SQL log?? It looks like it is updating 1 record at a time > > > > Fred > > > > > > > From: Action Request System discussion list(ARSList) [mailto: > arslist@ARSLIST.ORG ] On Behalf Of patrick zandi > Sent: Tuesday, October 18, 2011 11:57 AM > > > > > To: arslist@ARSLIST.ORG > Subject: Re: ARS 7.6.04 SP2 Solaris installer Oracle db. > > > > > > ** > > OK, 3 hours of this::: NOW WHAT>>>???? > > > > (Oct 18 2011 03:54:07.112 PM > +0000),INFO,com.bmc.install.product.rik.statement.RIKLoadApplicationStatement, > > LOG EVENT {Description=[Code],Detail=[550]} > > (Oct 18 2011 03:54:07.113 PM > +0000),INFO,com.bmc.install.product.rik.statement.RIKLoadApplicationStatement, > > LOG EVENT {Description=[Code],Detail=[551]} > > (Oct 18 2011 03:54:07.113 PM > +0000),INFO,com.bmc.install.product.rik.statement.RIKLoadApplicationStatement, > > LOG EVENT {Description=[Code],Detail=[552]} > > (Oct 18 2011 03:54:07.114 PM > +0000),INFO,com.bmc.install.product.rik.statement.RIKLoadApplicationStatement, > > LOG EVENT {Description=[Code],Detail=[8303]} > > (Oct 18 2011 03:54:07.115 PM > +0000),INFO,com.bmc.install.product.rik.statement.RIKLoadApplicationStatement, > > LOG EVENT {Description=[RIK load application > command],Detail=[/var/tmp/Utilities/rik/rik loadapp -f > /u01/app/ITSM/Workflow/install/7.6.04_sp2_overlay_install/systems/fnd/workflow/en/adf_fnd.xml > -m nosamp -x HOSTNAME -t 2106 -u Demo -p ** -l /u01/app/ITSM -n > InstallRAF7604 -L]} > > > > > On Tue, Oct 18, 2011 at 12:12 PM, patrick zandi < remedy...@gmail.com > > wrote: > > OK, Atrium core sp2 was not a problem, but I was doing oob install too. > the bmc itsm overlay install is running now, for over 2 hours... I am > getting a little concerned.. it appears stuck on task progress, executing.. > I do not see anything in the /var/tmp files .. I must be missing > something.. > but we shall see.. > > > > > > > On Tue, Oct 18, 2011 at 10:14 AM, strauss < stra...@unt.edu > wrote: > > ** > > > > Sounds like you had better luck than I. I started running the SP2 > installs on my sample data environment which has been stable on some > lightweight 2008 R2 VMs since 7.5, and was upgraded to 7.6, 7.6.03, 7.6.04, > and SP1 without incident. The SP2 ARS installer ran just fine – updated > ALL components. The Atrium installer exploded in a shower of sparks and > left the system as toast; apparently it tried unsuccessfully to restart the > AR Service during the install, and stopped the service but not the process > – which remained connected to the SQL Server, and proceeded to spew some of > the coolest error messages I have seen in a while; here is a selection for > your entertainment: > > > > Mon Oct 17 15:15:26 2011 390600 : Cannot establish a network connection > to the AR System Plug-In server (ARERR 8760) > > Mon Oct 17 15:15:26 2011 ITSMT01 (9556) BMC.FILTERAPI.NOE : RPC: > Miscellaneous tli error - System error (Connection refused) > > > > Mon Oct 17 15:15:45 2011 : Action Request System(R) Server x64 Version > 7.6.04 SP2 201110080614 > > (c) Copyright 1991-2011 BMC Software, Inc. > > Mon Oct 17 15:15:45 2011 ProcessMain : Another copy of the server is > already running on the same RPC socket (ARERR 35) > > Mon Oct 17 15:15:45 2011 d:*program files (x86)*bmc > software*arsystem*confRemedyMutex390600 > > Mon Oct 17 15:15:45 2011: AR System server terminated — fatal error > occurred in ARSERVER (ARNOTE 21) > > Mon Oct 17 16:04:22 2011 390602 : Cannot establish a network connection > to the AR System Plug-In server (ARERR 8760) > > Mon Oct 17 16:04:22 2011 ITSMT01 (9999) RMDY.ITSM.RKM.FILESYSTEM : > RPC: Miscellaneous tli error - System error (Connection refused) > > Mon Oct 17 16:04:23 2011 390602 : Cannot establish a network connection > to the AR System Plug-In server (ARERR 8760) > > Mon Oct 17 16:04:26 2011 ITSMT01 (22701) > REMEDY.ARDBC.SERVER.ADMINISTRATION : RPC: Miscellaneous tli error - System > error (Connection refused) > > > > Mon Oct 17 16:06:19 2011 390600 : Cannot establish a network connection > to the AR System Plug-In server (ARERR 8760) > > Mon Oct 17 16:06:19 2011 ITSMT01 (22701) : RPC: Miscellaneous tli > error - System error (Connection refused) > > Mon Oct 17 16:06:20 2011 390600 : Cannot establish a network connection > to the AR System Plug-In server (ARERR 8760) > > Mon Oct 17 16:06:20 2011 itsmt01.unt.edu (9999) ARSYS.ARF.REGISTRY : > RPC: Miscellaneous tli error - System error (Connection refused) > > Mon Oct 17 16:06:21 2011 390600 : Cannot establish a network connection > to the AR System Plug-In server (ARERR 8760) > > Mon Oct 17 16:06:21 2011 ITSMT01 (9556) BMC.FILTERAPI.NOE : RPC: > Miscellaneous tli error - System error (Connection refused) > > Mon Oct 17 16:06:22 2011 390600 : Cannot establish a network connection > to the AR System Plug-In server (ARERR 8760) > > Mon Oct 17 16:06:22 2011 ITSMT01 (9999) RMDY.ITSM.RLE : RPC: > Miscellaneous tli error - System error (Connection refused) > > Mon Oct 17 16:06:23 2011 390600 : Cannot establish a network connection > to the AR System Plug-In server (ARERR 8760) > > Mon Oct 17 16:06:23 2011 itsmt01.unt.edu (9998) ARSYS.ARF.FTS : RPC: > Miscellaneous tli error - System error (Connection refused) > > Mon Oct 17 16:18:06 2011 : Action Request System(R) Server x64 Version > 7.6.04 SP2 201110080614 > > (c) Copyright 1991-2011 BMC Software, Inc. > > Mon Oct 17 16:18:07 2011 ProcessMain : Another copy of the server is > already running on the same RPC socket (ARERR 35) > > Mon Oct 17 16:18:07 2011 d:*program files (x86)*bmc > software*arsystem*confRemedyMutex390600 > > Mon Oct 17 16:18:07 2011: AR System server terminated — fatal error > occurred in ARSERVER (ARNOTE 21) > > > > So far I am not impressed. Also, the problem with logging in that I have > been fighting in Migrator 7.6.04 – all versions since 7.6.03 – was not > fixed in SP2, so that was no improvement. > > > > Christopher Strauss, Ph.D. > Call Tracking Administration Manager > University of North Texas Computing & IT Center > http://itsm.unt.edu/ > > > From: Action Request System discussion list(ARSList) [mailto: > arslist@ARSLIST.ORG ] On Behalf Of patrick zandi > Sent: Tuesday, October 18, 2011 8:14 AM > To: arslist@ARSLIST.ORG > Subject: Re: ARS 7.6.04 SP2 Solaris installer Oracle db. > > > > ** This was an upgrade from 7.6.04 SP1 -- forgot to mention. > > > > > On Tue, Oct 18, 2011 at 9:05 AM, patrick zandi < remedy...@gmail.com > > wrote: > > OK folks, > ARS 7.6.04 SP2 (released Oct 11) - Oracle 11.iR2 > Solaris 10 I installed, and the installer was pretty intelligent, and I > had 1 minor weird issue, and the fix was odd. Where the page is that asked > for the aradmin and password (put in the password). mine was default for > aradmin however it did not like it when I got to the page that asked for > the system password it says the aradmin password was wrong (you would think > that was already done).. so if you go back 4 - 5 steps and put in the > aradmin and password, then as you move forward : bang it knows the rest and > does it (fairly intelligent). Strange but it went through the install > without a hitch.. > I have not see anything like this in a while.. have not tested fully.. > but wanted you all to know.. we are making progress... > > Thanks BMC > > -- > Patrick Zandi > > > > > > _attend WWRUG11 www.wwrug.com ARSlist: "Where the Answers Are"_ > > > > > -- > Patrick Zandi > _attend WWRUG11 www.wwrug.com ARSlist: "Where the Answers Are"_ > > _attend WWRUG11 www.wwrug.com ARSlist: "Where the Answers Are"_ > > _attend WWRUG11 www.wwrug.com ARSlist: "Where the Answers Are"_ > > _attend WWRUG11 www.wwrug.com ARSlist: "Where the Answers Are"_ > > > > > -- > Patrick Zandi > _attend WWRUG11 www.wwrug.com ARSlist: "Where the Answers Are"_ > > > > _attend WWRUG11 www.wwrug.com ARSlist: "Where the Answers Are"_ > > > > > -- > Patrick Zandi > _attend WWRUG11 www.wwrug.com ARSlist: "Where the Answers Are"_ > > > > > _attend WWRUG11 www.wwrug.com ARSlist: "Where the Answers Are"_ > > > > > -- > Patrick Zandi > _attend WWRUG11 www.wwrug.com ARSlist: "Where the Answers Are"_ > > > _attend WWRUG11 www.wwrug.com ARSlist: "Where the Answers Are"_ > _attend WWRUG11 www.wwrug.com ARSlist: "Where the Answers Are"_ > > > _______________________________________________________________________________ > UNSUBSCRIBE or access ARSlist Archives at www.arslist.org > attend wwrug12 www.wwrug12.com ARSList: "Where the Answers Are" > _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug12 www.wwrug12.com ARSList: "Where the Answers Are"
