Either that, or start looking at something like StaX - http://stax.codehaus.org/Home
Pulling that large a file in memory is going to suck a lot, no matter what you do. Mark On Tue, Feb 15, 2011 at 10:24 AM, Casey Dougall <ca...@uberwebsitesolutions.com> wrote: > > On Mon, Feb 14, 2011 at 5:44 PM, Matt Robertson <websitema...@gmail.com>wrote: > >> >> My question is how do I reduce processing time... is there anything I >> could be doing better? >> >> I am tasked with doing an import on a file whose size can essentially >> be unlimited. We've been able to handle in the 10-15mb range but it >> recently ballooned to 100 mb, and its going to get larger. Processing >> time seems to be about 66 hours for a 45 mb file and thats a disaster. >> For a 14mb file its about 90 minutes. >> >> Whats happening is this: CF is looping over a very large number of >> records that are in themselves fairly complex. The more records there >> are, the longer the loop over them takes. More records + more time >> per record = a seemingly geometric increase in processing time. >> >> > > > > Are you using MS SQL 2005? Because throwing the XML file directly at SQL > server works a 1000% better. I've basically given up on processing large XML > files directly with coldfusion. IF you can save these xml files on the SQL > server you can make use of BulkInsert but that needs to read the file from > that server. OR just setup a stored procedure, send your xml over and let it > do it's thing. > > http://msdn.microsoft.com/en-us/library/ms345117%28v=SQL.90%29.aspx > > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~| Order the Adobe Coldfusion Anthology now! http://www.amazon.com/Adobe-Coldfusion-Anthology/dp/1430272155/?tag=houseoffusion Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:342216 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/groups/cf-talk/unsubscribe.cfm
