On Wed, 27 Jan 2010, Tim Donohue (JIRA) wrote: > Just wanted to double check a few things. If my math is right, >hypothetically speaking, if we were loading 4,000 items into your >repository of 120,000 you are saying you would see the following:
I shall respond to this, because I ran the tests; my colleague produced a patch. > * Before patch (4.7 secs/item): about 313 minutes of process time (5 hrs, > 13mins) > * After patch (4.9 items/sec or about 0.2secs/ item): about 13 minutes of > process time > > So, you've found the patch would decrease your processing time by 5 hours in > this hypothetical situation? Yes. > Also, did you see a much larger load on the server after the patch (in >terms of memory / cpu usage, etc) than before the patch? Trying to get a >sense of whether decreasing processing time this drastically causes a >large increase in server load. Quite the contrary. With the default configuration, the load on the database is enormous during the entire import. This is because it's constantly hammering the database to prune indexes. With our patch, the load on the server drops dramatically, except after the last item (the only point where the indexes are built), where it is the same as with the default code. I should point out that we run the DB and app on separate servers; the load on the app server is never very high during batch imports, it is almost entirely focused on the database. We have tested and demonstrated (timing using two instances of siege, from separate servers) that the higher the load on the database, the slower the load time for the web pages. Best regards, -- Tom De Mulder <[email protected]> - Cambridge University Computing Service +44 1223 3 31843 - New Museums Site, Pembroke Street, Cambridge CB2 3QH -> 28/01/2010 : The Moon is Waxing Gibbous (82% of Full) ------------------------------------------------------------------------------ The Planet: dedicated and managed hosting, cloud storage, colocation Stay online with enterprise data centers and the best network in the business Choose flexible plans and management services without long-term contracts Personal 24x7 support from experience hosting pros just a phone call away. http://p.sf.net/sfu/theplanet-com _______________________________________________ Dspace-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/dspace-devel
