Matt, as you want a non CF solution, you might be best posting elsewhere to find a database guru who could do it all directly on the DB. If no-one here pipes up then try www.odesk.com
On Wed, Feb 16, 2011 at 6:09 PM, Rick Faircloth <r...@whitestonemedia.com>wrote: > > Hi, Matt... > > I know you might have looked at this link at dev.mysql.com > for a solution to processing your data feed, but I thought I'd > bring it up, just in case you haven't seen it. Looks like it > might be right up your alley. > > > http://dev.mysql.com/tech-resources/articles/xml-in-mysql5.1-6.0.html#xml-5. > 1-in-and-out > > While I didn't have to process xml files, I do have to process > a daily data feed of real estate data in comma-delimited files > and insert that info into a MySQL 5 database. > > At first, I tried processing the files with CF, but it was a > *really* slow process using CF, taking almost a minute to process > some files. Finally, someone on this list > suggested I take a look at the MySQL commands for loading data > from files, such as, in your case, "Load_File()", which opens > an entire XML document, makes it available as a string, > and inserts this string into a table column. In my case, using > MySQL's load data infile, the time was reduced to less than a second. > > Check out the page above for details on usage and syntax. > > hth, > > Rick > > -----Original Message----- > From: Matt Robertson [mailto:websitema...@gmail.com] > Sent: Wednesday, February 16, 2011 12:22 PM > To: cf-talk > Subject: Re: Big XML files processing Really s-l-o-w. Solution? > > > Hey everybody, we're in a bind here and I pitched the idea to the > client of bringing in a ringer. Is there someone out there who is > interested in building a direct-to-SQL Server or direct-to-mySQL-based > solution? > > Solution would have to, on a scheduled basis, grab a file in a > location on the server on its own without CF and process it (we have > stuff that retrieves/places the file). Current file size is about 50 > mb but must be expected to go to around 100. At 42mb using just CF I > am at about 70 seconds per record loop. We need a *significant* > improvement in performance. Work would be on a dedicated Windows > server. Present box is a fairly powerful Crystaltech Win2003 Server > w/4GB RAM and a Xeon 2.8 processor w/4 cores. We'd consider a > Win2008, 64-bit installation to improve speed. Looking at a 24-core > server at Viviotech w/16gb as a hi-horsepower alternative. > > Our feed partner is probably going to be able to use a different > method of delivery that will reduce file size, but for now we need to > plan for the worst and move on it. > > Anyone interested? You can email me at my for-reals email at matt AT > mysecretbase DOT com. > > -- > --m@Robertson-- > Janitor, The Robertson Team > mysecretbase.com > > > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~| Order the Adobe Coldfusion Anthology now! http://www.amazon.com/Adobe-Coldfusion-Anthology/dp/1430272155/?tag=houseoffusion Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:342343 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/groups/cf-talk/unsubscribe.cfm
