On Mon, Feb 14, 2011 at 5:44 PM, Matt Robertson <websitema...@gmail.com>wrote:
> > My question is how do I reduce processing time... is there anything I > could be doing better? > > I am tasked with doing an import on a file whose size can essentially > be unlimited. We've been able to handle in the 10-15mb range but it > recently ballooned to 100 mb, and its going to get larger. Processing > time seems to be about 66 hours for a 45 mb file and thats a disaster. > For a 14mb file its about 90 minutes. > > Whats happening is this: CF is looping over a very large number of > records that are in themselves fairly complex. The more records there > are, the longer the loop over them takes. More records + more time > per record = a seemingly geometric increase in processing time. > > Are you using MS SQL 2005? Because throwing the XML file directly at SQL server works a 1000% better. I've basically given up on processing large XML files directly with coldfusion. IF you can save these xml files on the SQL server you can make use of BulkInsert but that needs to read the file from that server. OR just setup a stored procedure, send your xml over and let it do it's thing. http://msdn.microsoft.com/en-us/library/ms345117%28v=SQL.90%29.aspx ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~| Order the Adobe Coldfusion Anthology now! http://www.amazon.com/Adobe-Coldfusion-Anthology/dp/1430272155/?tag=houseoffusion Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:342215 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/groups/cf-talk/unsubscribe.cfm
