On Sun, Oct 31, 2010 at 7:08 AM, Lutz Steinborn <l.steinb...@4c-ag.de>wrote:
> On Sat, 30 Oct 2010 23:49:29 +0200 > Viktor Bojović <viktor.bojo...@gmail.com> wrote: > > > > > many tries have failed because 8GB of ram and 10gb of swap were not > enough. > > also sometimes i get that more than 2^32 operations were performed, and > > functions stopped to work. > > > we have a similar problem and we use the Amara xml Toolkit for python. To > avoid > the big memory consumption use pushbind. A 30G bme catalog file takes a > maximum > up to 20min to import. It might be faster because we are preparing complex > objects with an orm. So the time consumption depends how complex the > catalog is. > If you use amara only to perform a conversion from xml to csv the final > import > can be done much faster. > > regards > > -- > Lutz > > http://www.4c-gmbh.de > > Thanx Lutz, I will try to use that Amara and also I will try to parse it with SAX. I have tried twig and some other parsers but they consumed too much RAM. -- --------------------------------------- Viktor Bojović --------------------------------------- Wherever I go, Murphy goes with me
