is this an AIX? box.. there is a white paper on tuning.. on that.. On Tue, Jun 28, 2011 at 12:01 PM, Reiser, John J <john.j.rei...@lmco.com>wrote:
> ** > > Unqualified searches are turned off.**** > > BMC has had me set the next block size, reset serverside table chunks, max > return from getlist.**** > > Nothing worked**** > > ** ** > > --- > John J. Reiser > Remedy Developer/Administrator **** > > Senior Software Development Analyst > Lockheed Martin - MS2 > The star that burns twice as bright burns half as long. > Pay close attention and be illuminated by its brilliance. - paraphrased by > me **** > > ** ** > > *From:* Action Request System discussion list(ARSList) [mailto: > arslist@ARSLIST.ORG] *On Behalf Of *Andrew C Goodall > *Sent:* Tuesday, June 28, 2011 11:50 AM > *To:* arslist@ARSLIST.ORG > > *Subject:* Re: EXTERNAL: Re: arserver.exe is consuming 100% cpu - possible > DB corruption? (Long Post)**** > > ** ** > > ** **** > > Are you sure there is not a user out there performing an unqualified search > or full text search? I assume you deny unqualified searching and limit the > returns for a get list?**** > > ** ** > > Regards,**** > > **** > > *Andrew Goodall***** > > Software Engineer 2 | Development Services | jcpenney . www.jcp.com > <http://www.jcp.com/> > **** > ------------------------------ > > *From:* Action Request System discussion list(ARSList) [mailto: > arslist@ARSLIST.ORG] *On Behalf Of *Reiser, John J > *Sent:* Tuesday, June 28, 2011 10:38 AM > *To:* arslist@ARSLIST.ORG > *Subject:* Re: EXTERNAL: Re: arserver.exe is consuming 100% cpu - possible > DB corruption? (Long Post)**** > > ** ** > > Rick,**** > > I’ve gone in a few times and set the affinity to CPU0. I know that’s not > exactly what you mean but it allows me to do other things like look at log > files because the arserver.exe is only hitting one of the cpus.**** > > I’ll check on the RunIf qualifications but not a whole lot of workflow was > touched recently. The problem just sort of appeared.**** > > Though I was in Dev Studio at the time someone reported the outage to me I > wasn’t actually working in Dev Studio.**** > > When I came back to the window it was hung. So was the Usertool on my PC.* > *** > > We don’t have any OOTB modules with the exception of maybe the Samples. No > ITSM, No CMDB. This is a basic ARServer system with all home built forms. > And nothing new has been added that didn’t get built on the development VM. > And that one works fine.**** > > We even stopped both ARservers, pointed the Dev server to the Production > database and the problem came with it. That’s what makes me think there is > something wrong in the part of the db that houses the definitions.**** > > Restoring the db to a state prior to the hiccup on Tuesday should eliminate > any object changes. Then I can re-import all of the data records from the > forms that I exported before the restore.**** > > If we restore and cannot reproduce the error then re-import and it comes > back then it could be the Diary Field issue but I hope that doesn’t happen. > **** > > ** ** > > During this past week I’ve disabled notification filters, stopped the email > engine, stopped Tomcat. The problem comes back and BMC doesn’t see anything > in the logs.**** > > ** ** > > ** ** > > --- > John J. Reiser > Remedy Developer/Administrator **** > > Senior Software Development Analyst > Lockheed Martin - MS2 > The star that burns twice as bright burns half as long. > Pay close attention and be illuminated by its brilliance. - paraphrased by > me **** > > ** ** > > *From:* Action Request System discussion list(ARSList) [mailto: > arslist@ARSLIST.ORG] *On Behalf Of *Rick Cook > *Sent:* Tuesday, June 28, 2011 10:50 AM > *To:* arslist@ARSLIST.ORG > *Subject:* EXTERNAL: Re: arserver.exe is consuming 100% cpu - possible DB > corruption? (Long Post)**** > > ** ** > > ** **** > > There is also a hotfix for 7.1 involving some embedded spaces in Filter Run > If quals that was causing similar problems. Wouldn't put it past the devs > to reintroduce a problem that was fixed in a previous version. In fact, I > just fixed one of those yesterday. **** > > Rick**** > > On Jun 28, 2011 7:46 AM, "dcharters" <dchart...@www.charterssoftware.com> > wrote: > > Do you have AIE and/or Recon jobs running at all? Are those engines even > > installed anywhere? > > I have also seen this with special characters being passed in Diary > fields. > > That was a few versions back but something to look at. > > > > > > On Tue, 28 Jun 2011 08:30:24 -0400, Reiser, John J wrote > >> Mark, > >> I've been working with BMC Support since last week. They have had > >> me send many log files trying to capture an event but nothing has > >> shown up yet. I'll check into Spotlight. > >> > >> Theo, > >> Already disabled escalations, alerts etc. I even disabled every > >> filter with a notify action because that was the first bit of > >> workflow that would send arserver.exe running away. The system still > >> overloads. > >> > >> Joe, > >> I'll ask the VM admins to check the NIC settings. Since the SQL > >> Server is remote and on a huge SAN that side should be ok. The DBA > >> said he saw no unusual traffic to the ARSystem db. > >> > >> Rick, > >> There ARE FOUR Lights. Sorry can you tell I'm losing it. > >> The VM has 2 CPUs configured. > >> VMware Tools says version 8.3.7 build 341836. > >> > >> LJ, > >> I've been capturing Filter, thread, API, SQL logs and sent them to > >> BMC Support. They see no long queries or transactions that could be > >> causing the high cpu usage. I did combine sql and api. I'll add the > >> filter logs and see what kind of timing I get between the three. > >> > >> Thanks for the feedback. > >> If BMC can't solve it today I think I'm going to use Misi's rrrChive > >> and export and reload the user data after we rollback the meta(?) > >> data from before the problem started. > >> > >> --- > >> John J. Reiser > >> Remedy Developer/Administrator > >> Senior Software Development Analyst > >> Lockheed Martin - MS2 > >> The star that burns twice as bright burns half as long. > >> Pay close attention and be illuminated by its brilliance. - > >> paraphrased by me > >> > >> -----Original Message----- > >> From: Action Request System discussion list(ARSList) > >> [mailto:arslist@ARSLIST.ORG] On Behalf Of Walters, Mark Sent: > >> Tuesday, June 28, 2011 4:59 AM To: arslist@ARSLIST.ORG Subject: > >> EXTERNAL: Re: arserver.exe is consuming 100% cpu - possible DB > >> corruption? (Long Post) > >> > >> Grab a copy of Spotlight on Windows from www.quest.com and you can > >> use it to view the various threads within the arserverd.exe and work > >> out which one is causing the high CPU load. Once you have this you > >> can reference the thread/sql/api/filter logs to see what activity it > >> is. > >> > >> Mark > >> > >> I work for BMC, I don't speak for them. > >> > >> -----Original Message----- > >> From: Action Request System discussion list(ARSList) > >> [mailto:arslist@ARSLIST.ORG] On Behalf Of Reiser, John J Sent: 27 > >> June 2011 22:27 To: arslist@ARSLIST.ORG Subject: arserver.exe is > >> consuming 100% cpu - possible DB corruption? (Long Post) > >> > >> Hello Listers, > >> ARS 7.6.03 > >> MS 2003 Enterprise > >> MS SQL 2005 (remote) > >> Total home grown system. No OOTB modules. > >> > >> I have a real stumper here. It even has BMC scratching their heads. > >> I have a production system that is experiencing cpu overload that > >> runs up to 99 in the processes and sits there. The ARSystem server > >> is virtual machine. We thought maybe it was a MS "Patch Tuesday" > >> issue and we removed the 10 recent MS patches one at a time and > >> restarted the machine each time. The problem still exists after the > >> arserver service starts. Sometime immediately and sometimes it will > >> sit for 1- 20 minutes before it starts to hog the CPUs. To eliminate > >> any other OS and file system issues we grabbed a two week old backup > >> image of the server and restored it. The system came back ok for a > >> short while and then started to lock up the CPU again. Working with > >> BMC I set the logs on and restarted. We saw the system jump to 100% > >> within a minute and captured a 10MB arsql.log file. It can force the > >> overload at anytime by firing filter workflow with a notification > >> action in it. I disabled this one filter but the system still loaded > >> up. I added a Filter that ran a 0 and the only action was Goto 1000 > >> to jump all Filter actions that fired on the change of the Status > >> field in question. Still no joy. I've disabled every piece of Notify > >> workflow. That worked the best and kept the system alive for longer > >> stretches but we can't run a system that way. > >> > >> I've come to the realization that there may be corrupted information > >> in the DB object tables and I wanted to get some feedback. Using > >> rrrChive I can pull a copy of every form's data since, say, two > >> weeks ago. Then have the DBA restore the entire system from that > >> date. After the restore I would use rrrChive to reload the two > >> weeks' data (Modified date' > "06/11/2011") and hope for the best. > >> > >> Any workflow that was changed in the last two weeks is negligible > >> and could be recreated/updated as needed. > >> > >> Do you think this is a viable solution? > >> When I asked the BMC tech if I could dump the T,H & B tables ; > >> restore the db and reload the T, H & B tables he reminded me that > >> the arschema and other meta tables would probably be out of synch. > >> That's when I thought of using rrrChive. > >> > >> Sorry to be so long winded but I need to get this back online, BMC > >> can't find anything in the logs and I don't want to lose the tickets > >> we've taken in the last week. > >> > >> --- > >> John J. Reiser > >> Remedy Developer/Administrator > >> Senior Software Development Analyst > >> Lockheed Martin - MS2 > >> The star that burns twice as bright burns half as long. > >> Pay close attention and be illuminated by its brilliance. - > >> paraphrased by me > >> > >> > > > _____________________________________________________________________________ > > __ > >> UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend > >> wwrug11 www.wwrug.com ARSList: "Where the Answers Are" > >> > >> > > > _____________________________________________________________________________ > > __ > >> UNSUBSCRIBE or access ARSlist Archives at www.arslist.org > >> attend wwrug11 www.wwrug.com ARSList: "Where the Answers Are" > >> > >> > > > _____________________________________________________________________________ > > __ > >> UNSUBSCRIBE or access ARSlist Archives at www.arslist.org > >> attend wwrug11 www.wwrug.com ARSList: "Where the Answers Are" > > > > > > -- > > Open WebMail Project (http://openwebmail.org) > > > > > _______________________________________________________________________________ > > UNSUBSCRIBE or access ARSlist Archives at www.arslist.org > > attend wwrug11 www.wwrug.com ARSList: "Where the Answers Are"**** > > _attend WWRUG11 www.wwrug.com ARSlist: "Where the Answers Are"_ **** > > > The information transmitted is intended only for the person or entity to > which it is addressed and > may contain confidential and/or privileged material. If the reader of this > message is not the intended > recipient, you are hereby notified that your access is unauthorized, and > any review, dissemination, > distribution or copying of this message including any attachments is > strictly prohibited. If you are not > the intended recipient, please contact the sender and delete the material > from any computer.**** > > ** ** > > _attend WWRUG11 www.wwrug.com ARSlist: "Where the Answers Are"_ _attend > WWRUG11 www.wwrug.com ARSlist: "Where the Answers Are"_**** > _attend WWRUG11 www.wwrug.com ARSlist: "Where the Answers Are"_ > -- Patrick Zandi _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug11 www.wwrug.com ARSList: "Where the Answers Are"
