On Tue, 2014-03-11 at 08:51 -0700, Tim Gustafson wrote: > And one last follow-up about this: here are my import statistics. The > importer used up to about 800MB of RAM, so better than the serializer, > but still pretty high.
Both of those (6G for serializing, and 800M for importing) are definitely higher than I'd expect. If you're on 4.2.2, then you have the fixes to explicitly turn off the caching layer. If you're interested in poking further, doing some analysis of the in-memory objects to determine what's leaking would be helpful. Instrumenting the main loop of either of the serializer or importer with Devel::Gladiator::arena_ref_counts to look for what sorts of objects are leaking might be instructive. > Also, 108 hours seems like a long time to import records that took only > 4 hours to export, and this is on a pretty high-end MySQL server. Is > there any way to speed things up a bit? Yeesh, that's pretty bad. I'd try running under Devel::NYTProf to see where the hotspots are -- I don't have any off-the-cuff suggestions. - Alex -- RT Training London, March 19-20 and Dallas May 20-21 http://bestpractical.com/training