Matt,

Very cool - an innovative approach I would not have thought of. Of course we
all know that Jochem's a certified genius :)

-Mark

Mark A. Kruger, MCSE, CFG
(402) 408-3733 ext 105
Skype: markakruger
www.cfwebtools.com
www.coldfusionmuse.com
www.necfug.com



-----Original Message-----
From: Matt Robertson [mailto:websitema...@gmail.com] 
Sent: Saturday, February 19, 2011 8:06 PM
To: cf-talk
Subject: Re: Big XML files processing Really s-l-o-w. Solution?


Here's the update:

Jochem's StAX-based solution worked so well -- and it allowed me to
re-use existing code since I was already processing xml -- that I
didn't bother to consider testing the xml2csv utility.

Original code processing a 45mb file yielded an insert time of 90
seconds per record with total file processing time being a projected
66 hours.

Using code that incorporated xmlSplitter.cfc, the routine created 2682
discrete xml files in approximately 45 seconds.  From there, the
insert loop did its work pulling out xml and pouring it into roughly
100 db table fields at ... 20 records per second.

Total time to read in the 45mb xml file, create the discrete files,
read them, insert them into the db and delete them (one at a time as I
went along in the loop) was 192250ms.

A bit more than three minutes.

Thats an acceptable level of improvement.

I owe you one, Jochem.

-- 
--m@Robertson--
Janitor, The Robertson Team
mysecretbase.com



~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Order the Adobe Coldfusion Anthology now!
http://www.amazon.com/Adobe-Coldfusion-Anthology/dp/1430272155/?tag=houseoffusion
Archive: 
http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:342453
Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/groups/cf-talk/unsubscribe.cfm

Reply via email to