RESOLVED: Re: AIE Import Stops at 130000 Records
Thanks for the suggestion Henry. There is a chunk size for getting the CMDB keys but no such setting for getting the external data. The problem appears to have been a resource issue on the BMC Config Discovery database virtual server. This was moved to new hardware for other reasons and his seems to have solved the problem. The 600,000 CIs now import fine. As far as the memory required for the AIE engine is concerned I got no help from BMC. The manuals indicate that it should use almost all the 2GB available to it on Windows but it uses only a couple of hundred meg. The logs indicate that external data is dealt with in chunks of 10,000 records. This would give the memory usage I see. Thanks to all those that responded to this. Cheers Peter > There's a chunk size parameter in aie's reg file - perhaps making the > chunks smaller might be of benefit here? > > > > > > Peter Romain > Sent by: "Action Request System discussion list(ARSList)" > > 11/24/2008 05:01 PM > Please respond to > arslist@ARSLIST.ORG > > > To > arslist@ARSLIST.ORG > cc > > Subject > AIE Import Stops at 13 Records > > > > > > > Hi All, > > I am testing the BMC Config Management integration to CMDB via AIE and am > running into an issue with the application and patch exchanges. > > CM has ~60 applications but after importing ~13 the creation of > new CMDB CIs stops. The logs show that AIE is still attempting to create > CIs but that these fail because the CMDB Name field is being set to null. > > The application exchange is running in its own AIE instance and has 12 > threads allocated. > > Could I be hitting a fundamental AIE process memory limit on Windows? > > Has anyone else seen this issue? > > Cheers > > Peter > ARS 7.01 Patch 3 > CMDB 2.01 Patch 7 > AIE 7.1 Patch 4 > Windows 2003 Server > SQLServer 2005 > > ___ > UNSUBSCRIBE or access ARSlist Archives at www.arslist.org > Platinum Sponsor: www.rmsportal.com ARSlist: "Where the Answers Are" > > > > > > > > Jefferies archives and monitors outgoing and incoming e-mail. The contents > of this email, including any attachments, are confidential to the ordinary > user of the email address to which it was addressed. If you are not the > addressee of this email you may not copy, forward, disclose or otherwise > use it or any part of it in any form whatsoever. This email may be > produced at the request of regulators or in connection with civil > litigation. Jefferies accepts no liability for any errors or omissions > arising as a result of transmission. Use by other than intended recipients > is prohibited. In the United Kingdom, Jefferies operates as Jefferies > International Limited; registered in England: no. 1978621; registered > office: Vintners Place, 68 Upper Thames Street, London EC4V 3BJ. > Jefferies International Limited is authorised and regulated by the > Financial Services Authority. > > ___ > UNSUBSCRIBE or access ARSlist Archives at www.arslist.org > Platinum Sponsor: www.rmsportal.com ARSlist: "Where the Answers Are" ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor: www.rmsportal.com ARSlist: "Where the Answers Are"
Re: AIE Import Stops at 130000 Records
There's a chunk size parameter in aie's reg file - perhaps making the chunks smaller might be of benefit here? Peter Romain Sent by: "Action Request System discussion list(ARSList)" 11/24/2008 05:01 PM Please respond to arslist@ARSLIST.ORG To arslist@ARSLIST.ORG cc Subject AIE Import Stops at 13 Records Hi All, I am testing the BMC Config Management integration to CMDB via AIE and am running into an issue with the application and patch exchanges. CM has ~60 applications but after importing ~13 the creation of new CMDB CIs stops. The logs show that AIE is still attempting to create CIs but that these fail because the CMDB Name field is being set to null. The application exchange is running in its own AIE instance and has 12 threads allocated. Could I be hitting a fundamental AIE process memory limit on Windows? Has anyone else seen this issue? Cheers Peter ARS 7.01 Patch 3 CMDB 2.01 Patch 7 AIE 7.1 Patch 4 Windows 2003 Server SQLServer 2005 ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor: www.rmsportal.com ARSlist: "Where the Answers Are" Jefferies archives and monitors outgoing and incoming e-mail. The contents of this email, including any attachments, are confidential to the ordinary user of the email address to which it was addressed. If you are not the addressee of this email you may not copy, forward, disclose or otherwise use it or any part of it in any form whatsoever. This email may be produced at the request of regulators or in connection with civil litigation. Jefferies accepts no liability for any errors or omissions arising as a result of transmission. Use by other than intended recipients is prohibited. In the United Kingdom, Jefferies operates as Jefferies International Limited; registered in England: no. 1978621; registered office: Vintners Place, 68 Upper Thames Street, London EC4V 3BJ. Jefferies International Limited is authorised and regulated by the Financial Services Authority. ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor: www.rmsportal.com ARSlist: "Where the Answers Are"
Re: FW: AIE Import Stops at 130000 Records
Hi Janie, Thanks for sharing your experience. I have checked the source. In fact the query to the source data excludes records where the field mapped to the CMDB Name field is null so there shouldn't be any issues. I run the big exchange isolated on one instance so there is no sharing of data with other exchanges. The memory estimate for my system would be 2*1.2*60 = 1.4G so there shouldn't be a problem. AIE doesn't crash, it just continues to try and create records. Shallow logging shows the data is correct from the source even when the submit fails. I've put all the logging on now and will see if that reveals anything. Cheers Peter > > > -Original Message- > From: Action Request System discussion list(ARSList) > [mailto:[EMAIL PROTECTED] On Behalf Of Janie > Sent: 24 November 2008 23:33 > To: arslist@ARSLIST.ORG > Subject: Re: AIE Import Stops at 13 Records > > Peter, > > One AIE instance can use up to 2GB of memory -- if it exceeds that, it > will > crash. > > Each active Data mapping used on a single AIE instance splits the 2GB > evenly. So 2 data mappings on 1 instance will allow each data mapping 1GB > when it runs. > > In each data mapping, one key field mapping uses 1.2kb times the number of > records in your AR target form. Obviously, additional keys will increase > the memory used. > > > > Is there legitimate data in the source that contains the 'Name' field -- > any > possibility it's Null and thus returning Null? > > Generally, I've found that if there is a process memory limit, AIE will > crash and the process will no longer be running... At all. If AIE is > running and the exchange is pulling data, I've found that the source data > is > usually the culprit when there is a problem -- especially when the > majority > of items are coming through correctly. > > HTH, > Janie > > > > -Original Message- > From: Action Request System discussion list(ARSList) > [mailto:[EMAIL PROTECTED] On Behalf Of Peter Romain > Sent: Monday, November 24, 2008 2:01 PM > To: arslist@ARSLIST.ORG > Subject: AIE Import Stops at 13 Records > > Hi All, > > I am testing the BMC Config Management integration to CMDB via AIE and am > running into an issue with the application and patch exchanges. > > CM has ~60 applications but after importing ~13 the creation of > new > CMDB CIs stops. The logs show that AIE is still attempting to create CIs > but > that these fail because the CMDB Name field is being set to null. > > The application exchange is running in its own AIE instance and has 12 > threads allocated. > > Could I be hitting a fundamental AIE process memory limit on Windows? > > Has anyone else seen this issue? > > Cheers > > Peter > ARS 7.01 Patch 3 > CMDB 2.01 Patch 7 > AIE 7.1 Patch 4 > Windows 2003 Server > SQLServer 2005 > > > ___ > UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum > Sponsor: > www.rmsportal.com ARSlist: "Where the Answers Are" > > > ___ > UNSUBSCRIBE or access ARSlist Archives at www.arslist.org > Platinum Sponsor: www.rmsportal.com ARSlist: "Where the Answers Are" > > ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor: www.rmsportal.com ARSlist: "Where the Answers Are"
Re: AIE Import Stops at 130000 Records
Peter, I highly doubt that. With my experience with AIE, this often happens when your import options on AIE are not correctly set. Could you take a snapshot of your import options (I forget the exact name of that tab) and send them across? I'll review them later this evening when time permits and let you know if I spot something. Joe From: Peter Romain <[EMAIL PROTECTED]> To: arslist@ARSLIST.ORG Sent: Monday, November 24, 2008 5:01:11 PM Subject: AIE Import Stops at 130000 Records Hi All, I am testing the BMC Config Management integration to CMDB via AIE and am running into an issue with the application and patch exchanges. CM has ~60 applications but after importing ~13 the creation of new CMDB CIs stops. The logs show that AIE is still attempting to create CIs but that these fail because the CMDB Name field is being set to null. The application exchange is running in its own AIE instance and has 12 threads allocated. Could I be hitting a fundamental AIE process memory limit on Windows? Has anyone else seen this issue? Cheers Peter ARS 7.01 Patch 3 CMDB 2.01 Patch 7 AIE 7.1 Patch 4 Windows 2003 Server SQLServer 2005 ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor: www.rmsportal.com ARSlist: "Where the Answers Are"
Re: AIE Import Stops at 130000 Records
Peter, One AIE instance can use up to 2GB of memory -- if it exceeds that, it will crash. Each active Data mapping used on a single AIE instance splits the 2GB evenly. So 2 data mappings on 1 instance will allow each data mapping 1GB when it runs. In each data mapping, one key field mapping uses 1.2kb times the number of records in your AR target form. Obviously, additional keys will increase the memory used. Is there legitimate data in the source that contains the 'Name' field -- any possibility it's Null and thus returning Null? Generally, I've found that if there is a process memory limit, AIE will crash and the process will no longer be running... At all. If AIE is running and the exchange is pulling data, I've found that the source data is usually the culprit when there is a problem -- especially when the majority of items are coming through correctly. HTH, Janie -Original Message- From: Action Request System discussion list(ARSList) [mailto:[EMAIL PROTECTED] On Behalf Of Peter Romain Sent: Monday, November 24, 2008 2:01 PM To: arslist@ARSLIST.ORG Subject: AIE Import Stops at 13 Records Hi All, I am testing the BMC Config Management integration to CMDB via AIE and am running into an issue with the application and patch exchanges. CM has ~60 applications but after importing ~13 the creation of new CMDB CIs stops. The logs show that AIE is still attempting to create CIs but that these fail because the CMDB Name field is being set to null. The application exchange is running in its own AIE instance and has 12 threads allocated. Could I be hitting a fundamental AIE process memory limit on Windows? Has anyone else seen this issue? Cheers Peter ARS 7.01 Patch 3 CMDB 2.01 Patch 7 AIE 7.1 Patch 4 Windows 2003 Server SQLServer 2005 ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor: www.rmsportal.com ARSlist: "Where the Answers Are" ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor: www.rmsportal.com ARSlist: "Where the Answers Are"
AIE Import Stops at 130000 Records
Hi All, I am testing the BMC Config Management integration to CMDB via AIE and am running into an issue with the application and patch exchanges. CM has ~60 applications but after importing ~13 the creation of new CMDB CIs stops. The logs show that AIE is still attempting to create CIs but that these fail because the CMDB Name field is being set to null. The application exchange is running in its own AIE instance and has 12 threads allocated. Could I be hitting a fundamental AIE process memory limit on Windows? Has anyone else seen this issue? Cheers Peter ARS 7.01 Patch 3 CMDB 2.01 Patch 7 AIE 7.1 Patch 4 Windows 2003 Server SQLServer 2005 ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor: www.rmsportal.com ARSlist: "Where the Answers Are"
