Re: EIE vs. arimportcmd flat file import performance
Joe, If you look at the log files generated it: - parses the flat data flat file - gets the .tbl file containing the data mapping definition - does some analysis of each field in the data mapping definition - does some analysis of each field in the CMDB that is referred to - creates the records in the CMDB The analysis takes place for each record in the data file. It's a few months since I was last involved in this but the precessing seemed unnecessarily thorough at the time. Interestingly, we could have used CMDB web services which was much faster (similar to the Import Tool) Cheers Peter Peter, When you state 'see all the processing it's doing to get the data mapped', what exactly do you mean. What does it do? Searches against tables with large volumes of data? Run external scripts / commands / utilities and wait for their output? Joe D'Souza -Original Message- From: Action Request System discussion list(ARSList) [mailto:[EMAIL PROTECTED] Behalf Of Peter Romain Sent: Tuesday, October 02, 2007 1:16 PM To: arslist@ARSLIST.ORG Subject: Re: EIE vs. arimportcmd flat file import performance I did some initial testing of EIE into CMDB 2 and found it much slower (about 1/3 the speed). When you look at the logs and see all the processing it's doing to get the data mapped from the flat files then you can understand why! However, by careful splitting of the data it is possible to use multiple parallel EIE data imports to speed things up. It is also possible to run multiple EIE daemons and direct the data at these to further improve performance. There is a white paper on EIE performance tuning. I guess it comes down to how many records, where they will go (CMDB is particularly slow), how much time you can allow and how complex you want the import to be. I doubt you'll get away without testing. Cheers Peter We are strategizing a way to import a large number of records on a routine basis. I'll just skip all the details since they really are not relevant to the core question: Has anyone compared import speed between the EIE and using the old school arimportcmd command line process in order to import a flat file? I am just curious to see if there is a big advantage in speed either way. There certainly advantages to both tools for doing this particular data import but I have no idea how the speed compares and I don't have the EIE installed on this server (and I'd rather not if I don't have to). William Rentfrow, Principal Consultant [EMAIL PROTECTED] C 701-306-6157 O 952-432-0227 No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.5.488 / Virus Database: 269.13.39/1044 - Release Date: 10/2/2007 11:10 AM ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org ARSlist:Where the Answers Are ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org ARSlist:Where the Answers Are
Re: EIE vs. arimportcmd flat file import performance
If analyzing a flat file is what is taking time, then I'm afraid there is nothing you can do to speed up that.. Having said that.. what sort of analysis does it perform? Does it compare the information in the flat file against information that may already be within the database? Is that what usually takes time? Joe D'Souza -Original Message- From: Action Request System discussion list(ARSList) [mailto:[EMAIL PROTECTED] Behalf Of Peter Romain Sent: Wednesday, October 03, 2007 3:37 AM To: arslist@ARSLIST.ORG Subject: Re: EIE vs. arimportcmd flat file import performance Joe, If you look at the log files generated it: - parses the flat data flat file - gets the .tbl file containing the data mapping definition - does some analysis of each field in the data mapping definition - does some analysis of each field in the CMDB that is referred to - creates the records in the CMDB The analysis takes place for each record in the data file. It's a few months since I was last involved in this but the precessing seemed unnecessarily thorough at the time. Interestingly, we could have used CMDB web services which was much faster (similar to the Import Tool) Cheers Peter Peter, When you state 'see all the processing it's doing to get the data mapped', what exactly do you mean. What does it do? Searches against tables with large volumes of data? Run external scripts / commands / utilities and wait for their output? Joe D'Souza -Original Message- From: Action Request System discussion list(ARSList) [mailto:[EMAIL PROTECTED] Behalf Of Peter Romain Sent: Tuesday, October 02, 2007 1:16 PM To: arslist@ARSLIST.ORG Subject: Re: EIE vs. arimportcmd flat file import performance I did some initial testing of EIE into CMDB 2 and found it much slower (about 1/3 the speed). When you look at the logs and see all the processing it's doing to get the data mapped from the flat files then you can understand why! However, by careful splitting of the data it is possible to use multiple parallel EIE data imports to speed things up. It is also possible to run multiple EIE daemons and direct the data at these to further improve performance. There is a white paper on EIE performance tuning. I guess it comes down to how many records, where they will go (CMDB is particularly slow), how much time you can allow and how complex you want the import to be. I doubt you'll get away without testing. Cheers Peter We are strategizing a way to import a large number of records on a routine basis. I'll just skip all the details since they really are not relevant to the core question: Has anyone compared import speed between the EIE and using the old school arimportcmd command line process in order to import a flat file? I am just curious to see if there is a big advantage in speed either way. There certainly advantages to both tools for doing this particular data import but I have no idea how the speed compares and I don't have the EIE installed on this server (and I'd rather not if I don't have to). William Rentfrow, Principal Consultant [EMAIL PROTECTED] C 701-306-6157 O 952-432-0227 No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.5.488 / Virus Database: 269.14.0/1046 - Release Date: 10/3/2007 10:08 AM ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org ARSlist:Where the Answers Are
EIE vs. arimportcmd flat file import performance
We are strategizing a way to import a large number of records on a routine basis. I'll just skip all the details since they really are not relevant to the core question: Has anyone compared import speed between the EIE and using the old school arimportcmd command line process in order to import a flat file? I am just curious to see if there is a big advantage in speed either way. There certainly advantages to both tools for doing this particular data import but I have no idea how the speed compares and I don't have the EIE installed on this server (and I'd rather not if I don't have to). William Rentfrow, Principal Consultant [EMAIL PROTECTED] C 701-306-6157 O 952-432-0227 ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org ARSlist:Where the Answers Are
Re: EIE vs. arimportcmd flat file import performance
Peter, When you state 'see all the processing it's doing to get the data mapped', what exactly do you mean. What does it do? Searches against tables with large volumes of data? Run external scripts / commands / utilities and wait for their output? Joe D'Souza -Original Message- From: Action Request System discussion list(ARSList) [mailto:[EMAIL PROTECTED] Behalf Of Peter Romain Sent: Tuesday, October 02, 2007 1:16 PM To: arslist@ARSLIST.ORG Subject: Re: EIE vs. arimportcmd flat file import performance I did some initial testing of EIE into CMDB 2 and found it much slower (about 1/3 the speed). When you look at the logs and see all the processing it's doing to get the data mapped from the flat files then you can understand why! However, by careful splitting of the data it is possible to use multiple parallel EIE data imports to speed things up. It is also possible to run multiple EIE daemons and direct the data at these to further improve performance. There is a white paper on EIE performance tuning. I guess it comes down to how many records, where they will go (CMDB is particularly slow), how much time you can allow and how complex you want the import to be. I doubt you'll get away without testing. Cheers Peter We are strategizing a way to import a large number of records on a routine basis. I'll just skip all the details since they really are not relevant to the core question: Has anyone compared import speed between the EIE and using the old school arimportcmd command line process in order to import a flat file? I am just curious to see if there is a big advantage in speed either way. There certainly advantages to both tools for doing this particular data import but I have no idea how the speed compares and I don't have the EIE installed on this server (and I'd rather not if I don't have to). William Rentfrow, Principal Consultant [EMAIL PROTECTED] C 701-306-6157 O 952-432-0227 No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.5.488 / Virus Database: 269.13.39/1044 - Release Date: 10/2/2007 11:10 AM ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org ARSlist:Where the Answers Are
