Re: Reconciliation process time and power
Thanks Allan for jumping in, and formally endorsing this best practice architecture, and most importantly making it known. I didn't even know this document existed, I see it was released in March 2009. I got the recommendation last year, so I guess this means practical knowledge is being formalized as white papers. This is a very good step forward, please keep it up. -Guillaume -Original Message- From: Action Request System discussion list(ARSList) on behalf of O'Farrell, Allan Sent: Mon 09/21/09 7:07 PM To: arslist@ARSLIST.ORG Subject: Re: Reconciliation process time and power Hi List, I'm sorry if this has been covered before; but you can tune some of the impact that the recon jobs have on the AR server by restricting/reducing the number of threads it can use. By configuring an RPC for your recon engine and setting a moderate number of threads to this RPC; you could "throttle-down" how much processing grunt is available to recon jobs. This may slow down reconciliation, but will reduce impact to the rest of the system. Obviously, there is a limit to how much work one AR server can do, so in the end, you may still need to setup a Server group. It will take some testing to find the right balance of threads for your AR server and reconciliation load, but trying this will be easier/faster than implementing a new server. You might also want to double-check that the identification fields being used for the merge operation are properly indexed. They may be inefficient. On the subject of BMC's recommendation around Server Groups and dedicated "non-user" AR servers, page 12 of the 'Reference Architecture for BMC Service Support Solutions" whitepaper provides some brief guidance on this topic (very brief, but it's there): Using servers dedicated to specific functions provides an excellent way to scale the installation. The medium and large configurations described in this document use a dedicated integration server. This server is not part of the load balanced environment, and so does not see any end user activity. However, it is part of the AR System server group and so uses the same database. To relieve the load on other servers in the group, assign all batch and integration operations such as reconciliation, Atrium Integration Engine (AIE), and escalations to this server. Anyway, hope this helps. Note - this is my first ARSList post, so go easy please. :) Allan O'Farrell The opinions, statements, and/or suggested courses of action expressed in this E-mail do not necessarily reflect those of BMC Software, Inc. My voluntary participation in this forum is not intended to convey a role as a spokesperson, liaison or public relations representative for BMC Software, Inc. From: Action Request System discussion list(ARSList) [mailto:arsl...@arslist.org] On Behalf Of patrick zandi Sent: Tuesday, 22 September 2009 4:39 AM To: arslist@ARSLIST.ORG Subject: Re: Reconciliation process time and power ** makes sense, but I thought Price was more or different.. plus you need more servers physically.. Thanks On Mon, Sep 21, 2009 at 1:23 PM, Guillaume Rheault mailto:guilla...@dcshq.com>> wrote: ** That is right, the recommendation is to have a server group in which you have a dedicated server for "admin tasks"that will run the tasks that I listed below. Where is this written? Well it is not written anywhere publicly. However, I have personally talked to BMC Remedy consultants and architects from BMC Global Services (the consulting arm of BMC) and that's what they recommend. I guess eventually this knowledge will be formalized as a best practice architecture. We have that architecture set up in production, and works very well, so going forward, that's what is recommended for ITSM customers that have a CMDB implementation that involves reconciliation. The minimum for the server group is 2, one server dedicated to "admin/recon tasks", and the other one user facing. So if you have three servers, one for admin/recon tasks, the other two servicing users. Make sure you configure your load balancer so no user request gets forwarded to the "admin" server. -Guillaume -Original Message- From: Action Request System discussion list(ARSList) on behalf of patrick zandi Sent: Mon 09/21/09 12:46 PM To: arslist@ARSLIST.ORG<mailto:arslist@ARSLIST.ORG> Subject: Re: Reconciliation process time and power Are you saying the Recommendation is using a Server Group? I don't see any need for that, unless you might be talking 15K nodes... and a over million relationships.. I would think. how many in the minimum for a server group??? what are they saying about that... Where is this written? On Mon, Sep 21, 2009 at 12:35 PM, Guillaume Rheault mailto:guilla...@dcshq.com>>wrote: > ** > > It is now a common admission/recommendation that
Re: Reconciliation process time and power
Hi List, I'm sorry if this has been covered before; but you can tune some of the impact that the recon jobs have on the AR server by restricting/reducing the number of threads it can use. By configuring an RPC for your recon engine and setting a moderate number of threads to this RPC; you could "throttle-down" how much processing grunt is available to recon jobs. This may slow down reconciliation, but will reduce impact to the rest of the system. Obviously, there is a limit to how much work one AR server can do, so in the end, you may still need to setup a Server group. It will take some testing to find the right balance of threads for your AR server and reconciliation load, but trying this will be easier/faster than implementing a new server. You might also want to double-check that the identification fields being used for the merge operation are properly indexed. They may be inefficient. On the subject of BMC's recommendation around Server Groups and dedicated "non-user" AR servers, page 12 of the 'Reference Architecture for BMC Service Support Solutions" whitepaper provides some brief guidance on this topic (very brief, but it's there): Using servers dedicated to specific functions provides an excellent way to scale the installation. The medium and large configurations described in this document use a dedicated integration server. This server is not part of the load balanced environment, and so does not see any end user activity. However, it is part of the AR System server group and so uses the same database. To relieve the load on other servers in the group, assign all batch and integration operations such as reconciliation, Atrium Integration Engine (AIE), and escalations to this server. Anyway, hope this helps. Note - this is my first ARSList post, so go easy please. :) Allan O'Farrell The opinions, statements, and/or suggested courses of action expressed in this E-mail do not necessarily reflect those of BMC Software, Inc. My voluntary participation in this forum is not intended to convey a role as a spokesperson, liaison or public relations representative for BMC Software, Inc. From: Action Request System discussion list(ARSList) [mailto:arsl...@arslist.org] On Behalf Of patrick zandi Sent: Tuesday, 22 September 2009 4:39 AM To: arslist@ARSLIST.ORG Subject: Re: Reconciliation process time and power ** makes sense, but I thought Price was more or different.. plus you need more servers physically.. Thanks On Mon, Sep 21, 2009 at 1:23 PM, Guillaume Rheault mailto:guilla...@dcshq.com>> wrote: ** That is right, the recommendation is to have a server group in which you have a dedicated server for "admin tasks"that will run the tasks that I listed below. Where is this written? Well it is not written anywhere publicly. However, I have personally talked to BMC Remedy consultants and architects from BMC Global Services (the consulting arm of BMC) and that's what they recommend. I guess eventually this knowledge will be formalized as a best practice architecture. We have that architecture set up in production, and works very well, so going forward, that's what is recommended for ITSM customers that have a CMDB implementation that involves reconciliation. The minimum for the server group is 2, one server dedicated to "admin/recon tasks", and the other one user facing. So if you have three servers, one for admin/recon tasks, the other two servicing users. Make sure you configure your load balancer so no user request gets forwarded to the "admin" server. -Guillaume -Original Message- From: Action Request System discussion list(ARSList) on behalf of patrick zandi Sent: Mon 09/21/09 12:46 PM To: arslist@ARSLIST.ORG<mailto:arslist@ARSLIST.ORG> Subject: Re: Reconciliation process time and power Are you saying the Recommendation is using a Server Group? I don't see any need for that, unless you might be talking 15K nodes... and a over million relationships.. I would think. how many in the minimum for a server group??? what are they saying about that... Where is this written? On Mon, Sep 21, 2009 at 12:35 PM, Guillaume Rheault mailto:guilla...@dcshq.com>>wrote: > ** > > It is now a common admission/recommendation that a dedicated server should > be used in production for the following, to run the following: > > - escalations > - reconciliation jobs > - approval server > - assignment engine > - email > - SLM > > This means an extra server license unfortunately. > > -Guillaume > > > -Original Message- > From: Action Request System discussion list(ARSList) on behalf of patrick > zandi > Sent: Mon 08/17/09 2:37 PM > To: arslist@ARSLIST.ORG<mailto:arslist@ARSLIST.ORG> > Subject: qq: Reconciliation process time and power > > Quick Question: > ARS
Re: Reconciliation process time and power
makes sense, but I thought Price was more or different.. plus you need more servers physically.. Thanks On Mon, Sep 21, 2009 at 1:23 PM, Guillaume Rheault wrote: > ** > > That is right, the recommendation is to have a server group in which you > have a dedicated server for "admin tasks"that will run the tasks that I > listed below. > > Where is this written? Well it is not written anywhere publicly. However, I > have personally talked to BMC Remedy consultants and architects from BMC > Global Services (the consulting arm of BMC) and that's what they recommend. > I guess eventually this knowledge will be formalized as a best practice > architecture. > > We have that architecture set up in production, and works very well, so > going forward, that's what is recommended for ITSM customers that have a > CMDB implementation that involves reconciliation. > > The minimum for the server group is 2, one server dedicated to "admin/recon > tasks", and the other one user facing. So if you have three servers, one for > admin/recon tasks, the other two servicing users. Make sure you configure > your load balancer so no user request gets forwarded to the "admin" server. > > -Guillaume > > > -Original Message- > From: Action Request System discussion list(ARSList) on behalf of patrick > zandi > Sent: Mon 09/21/09 12:46 PM > To: arslist@ARSLIST.ORG > Subject: Re: Reconciliation process time and power > > Are you saying the Recommendation is using a Server Group? > I don't see any need for that, unless you might be talking 15K nodes... > and a over million relationships.. I would think. > > how many in the minimum for a server group??? what are they saying about > that... > > Where is this written? > > On Mon, Sep 21, 2009 at 12:35 PM, Guillaume Rheault >wrote: > > > ** > > > > It is now a common admission/recommendation that a dedicated server > should > > be used in production for the following, to run the following: > > > > - escalations > > - reconciliation jobs > > - approval server > > - assignment engine > > - email > > - SLM > > > > This means an extra server license unfortunately. > > > > -Guillaume > > > > > > -Original Message- > > From: Action Request System discussion list(ARSList) on behalf of patrick > > zandi > > Sent: Mon 08/17/09 2:37 PM > > To: arslist@ARSLIST.ORG > > Subject: qq: Reconciliation process time and power > > > > Quick Question: > > ARS 7.1 > > CMDB 2.1 Patch 6 > > Solaris 10 > > FD, and CM. > > > > --- > > When running oob jobs it uses 97% of 16 CPU's making the aruser tool and > > mid-tier Sluggish at best for over 1 hour. > > > > What are others seeing? > > > > -- > > Patrick Zandi > > > > > > > ___ > > UNSUBSCRIBE or access ARSlist Archives at www.arslist.org > > Platinum > > Sponsor:rmisoluti...@verizon.net > >ARSlist: > "Where the Answers Are" > > > > _Platinum Sponsor: rmisoluti...@verizon.net ARSlist: "Where the > Answers > > Are"_ > > > > > > -- > Patrick Zandi > > > ___ > UNSUBSCRIBE or access ARSlist Archives at www.arslist.org > Platinum > Sponsor:rmisoluti...@verizon.netARSlist: > "Where the Answers Are" > > _Platinum Sponsor: rmisoluti...@verizon.net ARSlist: "Where the Answers > Are"_ > -- Patrick Zandi ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor:rmisoluti...@verizon.net ARSlist: "Where the Answers Are"
Re: Reconciliation process time and power
That is right, the recommendation is to have a server group in which you have a dedicated server for "admin tasks"that will run the tasks that I listed below. Where is this written? Well it is not written anywhere publicly. However, I have personally talked to BMC Remedy consultants and architects from BMC Global Services (the consulting arm of BMC) and that's what they recommend. I guess eventually this knowledge will be formalized as a best practice architecture. We have that architecture set up in production, and works very well, so going forward, that's what is recommended for ITSM customers that have a CMDB implementation that involves reconciliation. The minimum for the server group is 2, one server dedicated to "admin/recon tasks", and the other one user facing. So if you have three servers, one for admin/recon tasks, the other two servicing users. Make sure you configure your load balancer so no user request gets forwarded to the "admin" server. -Guillaume -Original Message- From: Action Request System discussion list(ARSList) on behalf of patrick zandi Sent: Mon 09/21/09 12:46 PM To: arslist@ARSLIST.ORG Subject: Re: Reconciliation process time and power Are you saying the Recommendation is using a Server Group? I don't see any need for that, unless you might be talking 15K nodes... and a over million relationships.. I would think. how many in the minimum for a server group??? what are they saying about that... Where is this written? On Mon, Sep 21, 2009 at 12:35 PM, Guillaume Rheault wrote: > ** > > It is now a common admission/recommendation that a dedicated server should > be used in production for the following, to run the following: > > - escalations > - reconciliation jobs > - approval server > - assignment engine > - email > - SLM > > This means an extra server license unfortunately. > > -Guillaume > > > -Original Message- > From: Action Request System discussion list(ARSList) on behalf of patrick > zandi > Sent: Mon 08/17/09 2:37 PM > To: arslist@ARSLIST.ORG > Subject: qq: Reconciliation process time and power > > Quick Question: > ARS 7.1 > CMDB 2.1 Patch 6 > Solaris 10 > FD, and CM. > > --- > When running oob jobs it uses 97% of 16 CPU's making the aruser tool and > mid-tier Sluggish at best for over 1 hour. > > What are others seeing? > > -- > Patrick Zandi > > > ___ > UNSUBSCRIBE or access ARSlist Archives at www.arslist.org > Platinum > Sponsor:rmisoluti...@verizon.netARSlist: > "Where the Answers Are" > > _Platinum Sponsor: rmisoluti...@verizon.net ARSlist: "Where the Answers > Are"_ > -- Patrick Zandi ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor:rmisoluti...@verizon.net ARSlist: "Where the Answers Are" ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor:rmisoluti...@verizon.net ARSlist: "Where the Answers Are"
Re: Reconciliation process time and power
Are you saying the Recommendation is using a Server Group? I don't see any need for that, unless you might be talking 15K nodes... and a over million relationships.. I would think. how many in the minimum for a server group??? what are they saying about that... Where is this written? On Mon, Sep 21, 2009 at 12:35 PM, Guillaume Rheault wrote: > ** > > It is now a common admission/recommendation that a dedicated server should > be used in production for the following, to run the following: > > - escalations > - reconciliation jobs > - approval server > - assignment engine > - email > - SLM > > This means an extra server license unfortunately. > > -Guillaume > > > -Original Message- > From: Action Request System discussion list(ARSList) on behalf of patrick > zandi > Sent: Mon 08/17/09 2:37 PM > To: arslist@ARSLIST.ORG > Subject: qq: Reconciliation process time and power > > Quick Question: > ARS 7.1 > CMDB 2.1 Patch 6 > Solaris 10 > FD, and CM. > > --- > When running oob jobs it uses 97% of 16 CPU's making the aruser tool and > mid-tier Sluggish at best for over 1 hour. > > What are others seeing? > > -- > Patrick Zandi > > > ___ > UNSUBSCRIBE or access ARSlist Archives at www.arslist.org > Platinum > Sponsor:rmisoluti...@verizon.netARSlist: > "Where the Answers Are" > > _Platinum Sponsor: rmisoluti...@verizon.net ARSlist: "Where the Answers > Are"_ > -- Patrick Zandi ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor:rmisoluti...@verizon.net ARSlist: "Where the Answers Are"
Re: Reconciliation process time and power
It is now a common admission/recommendation that a dedicated server should be used in production for the following, to run the following: - escalations - reconciliation jobs - approval server - assignment engine - email - SLM This means an extra server license unfortunately. -Guillaume -Original Message- From: Action Request System discussion list(ARSList) on behalf of patrick zandi Sent: Mon 08/17/09 2:37 PM To: arslist@ARSLIST.ORG Subject: qq: Reconciliation process time and power Quick Question: ARS 7.1 CMDB 2.1 Patch 6 Solaris 10 FD, and CM. --- When running oob jobs it uses 97% of 16 CPU's making the aruser tool and mid-tier Sluggish at best for over 1 hour. What are others seeing? -- Patrick Zandi ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor:rmisoluti...@verizon.net ARSlist: "Where the Answers Are" ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor:rmisoluti...@verizon.net ARSlist: "Where the Answers Are"
Re: qq: Reconciliation process time and power
My experience is that the identification activities use the most CPU, especially the database CPU. During thus time the users cannot use the system. After identification the merge activities use less resources and the users haven't complained about any performance impacts. Which oob jobs are you referring to? > Quick Question: > ARS 7.1 > CMDB 2.1 Patch 6 > Solaris 10 > FD, and CM. > > --- > When running oob jobs it uses 97% of 16 CPU's making the aruser tool and > mid-tier Sluggish at best for over 1 hour. > > What are others seeing? > > -- > Patrick Zandi > > ___ > UNSUBSCRIBE or access ARSlist Archives at www.arslist.org > Platinum Sponsor:rmisoluti...@verizon.net ARSlist: "Where the Answers Are" > ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor:rmisoluti...@verizon.net ARSlist: "Where the Answers Are"
Re: Reconciliation process time and power
I have seen strange things on CMDB 7.5. When a OOTB Recon job is running (e.g. for Sandbox), the arrecond process takes a normal 2 to 10 % CPU usage. This does cause sluggishness, but is still manageable. Now as soon as the jobs complete, suddenly the CPU goes to 50% and stays there, making the system unusable for most parts. I have to kill the process. Let me know if you find something. Regards, Murtuza. From: Action Request System discussion list(ARSList) [mailto:arsl...@arslist.org] On Behalf Of patrick zandi Sent: Tuesday, August 18, 2009 12:07 AM To: arslist@ARSLIST.ORG Subject: qq: Reconciliation process time and power ** Quick Question: ARS 7.1 CMDB 2.1 Patch 6 Solaris 10 FD, and CM. --- When running oob jobs it uses 97% of 16 CPU's making the aruser tool and mid-tier Sluggish at best for over 1 hour. What are others seeing? -- Patrick Zandi _Platinum Sponsor: rmisoluti...@verizon.net ARSlist: "Where the Answers Are"_ ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor:rmisoluti...@verizon.net ARSlist: "Where the Answers Are" Visit our website at http://www.ubs.com This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mails are not encrypted and cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message which arise as a result of e-mail transmission. If verification is required please request a hard-copy version. This message is provided for informational purposes and should not be construed as a solicitation or offer to buy or sell any securities or related financial instruments. UBS reserves the right to retain all messages. Messages are protected and accessed only in legally justified cases. ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor:rmisoluti...@verizon.net ARSlist: "Where the Answers Are"
qq: Reconciliation process time and power
Quick Question: ARS 7.1 CMDB 2.1 Patch 6 Solaris 10 FD, and CM. --- When running oob jobs it uses 97% of 16 CPU's making the aruser tool and mid-tier Sluggish at best for over 1 hour. What are others seeing? -- Patrick Zandi ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor:rmisoluti...@verizon.net ARSlist: "Where the Answers Are"
