[...] > 1) Free: break your rsync's into several executions rather than one huge > one. Do several sub-directory trees, each separately. If your data > files are not organized in such a way that they can easily be divided > into a reasonable number of sub-directory trees, consider re-organizing > them so that they can be: it will pay off in many other sys-admin > benefits as well.
It's customer data, so we have no control over how it's organized. All we can depend on is the data being in /home, which is what we rsync. > 2) Cheap: buy more swap space. These days random-access magnetic > storage is running close to 0.50 USD per gig (e.g. here: > http://www.buy.com/retail/product.asp?sku=10360313 is 200GB for $105 in > the US, including shipping). At the stated rate of 100 bytes per file, > this is enough storage to add 2 billion files to each rsync that you > run, for a price that is less than many programmers want for a week of > coding. If you have much more than 2 billion files in each sub- > directory tree, you are probably doing something very wrong. :-) The servers already have 2-4GB of ram, with another gig of swap. Yes, these servers have a LOT of small files. > 3) Free: If your problem is not that you are running *out* of memory but > rather that rsync is (temporarily) 'stealing' the core (solid-state) > memory from the other 'more important' (i.e. requiring quicker response > time) processes (causing their data to get swapped out, which might > reduce response-time when that data later needs to get swapped back in), > you might also consider using the operating system to either lock-down > the memory used by your important server programs so that it cannot be [snip] Yes, the problem is that memory is being stolen from the processes that the servers exist for to begin with. Your solution isn't really viable though =) > 4) Expensive: buy more solid-state memory. Possibly still cheaper than > paying for coding, but at any rate, in my experience, more core is > rarely the best solution for lack-of-core problems. I agree, which is why we're willing to pay someone with rsync coding fu to fix it =) -- Matthew S. Hallacy -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
