Either that, or start looking at something like StaX -
http://stax.codehaus.org/Home

Pulling that large a file in memory is going to suck a lot, no matter
what you do.

Mark

On Tue, Feb 15, 2011 at 10:24 AM, Casey Dougall
<ca...@uberwebsitesolutions.com> wrote:
>
> On Mon, Feb 14, 2011 at 5:44 PM, Matt Robertson <websitema...@gmail.com>wrote:
>
>>
>> My question is how do I reduce processing time... is there anything I
>> could be doing better?
>>
>> I am tasked with doing an import on a file whose size can essentially
>> be unlimited.  We've been able to handle in the 10-15mb range but it
>> recently ballooned to 100 mb, and its going to get larger.  Processing
>> time seems to be about 66 hours for a 45 mb file and thats a disaster.
>>  For a 14mb file its about 90 minutes.
>>
>> Whats happening is this:  CF is looping over a very large number of
>> records that are in themselves fairly complex.  The more records there
>> are, the longer the loop over them takes.  More records + more time
>> per record = a seemingly geometric increase in processing time.
>>
>>
>
>
>
> Are you using MS SQL 2005? Because throwing the XML file directly at SQL
> server works a 1000% better. I've basically given up on processing large XML
> files directly with coldfusion. IF you can save these xml files on the SQL
> server you can make use of BulkInsert but that needs to read the file from
> that server. OR just setup a stored procedure, send your xml over and let it
> do it's thing.
>
> http://msdn.microsoft.com/en-us/library/ms345117%28v=SQL.90%29.aspx
>
>
> 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Order the Adobe Coldfusion Anthology now!
http://www.amazon.com/Adobe-Coldfusion-Anthology/dp/1430272155/?tag=houseoffusion
Archive: 
http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:342216
Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/groups/cf-talk/unsubscribe.cfm

Reply via email to