Re: High memory usage - any way around it other than splitting jobs?

2020-06-25 Thread Perry Hutchison via rsync
Andy Smith via rsync wrote: > I have a virtual machine with 2G of memory. On this VM there is a > directory tree with 33.3 million files in it. When attempting to > rsync (rsync -PSHav --delete /source /dest) this tree from one > directory to another on the same host, rsync uses all the memory

Re: High memory usage - any way around it other than splitting jobs?

2020-06-25 Thread Perry Hutchison via rsync
Kevin Korb via rsync wrote: > Unfortunately the hard links are the problem. In order to keep > them straight rsync has to remember the details of every file it > finds with a link count >1 making it grow and grow. I _hope_ it is only remembering the source and destination inode numbers, and

Re: High memory usage - any way around it other than splitting jobs?

2020-06-25 Thread Kevin Korb via rsync
Unfortunately the hard links are the problem. In order to keep them straight rsync has to remember the details of every file it finds with a link count >1 making it grow and grow. Of course without -H rsync will end up duplicating them. On 6/25/20 10:30 AM, Andy Smith via rsync wrote: > Hi, >

High memory usage - any way around it other than splitting jobs?

2020-06-25 Thread Andy Smith via rsync
Hi, I have a virtual machine with 2G of memory. On this VM there is a directory tree with 33.3 million files in it. When attempting to rsync (rsync -PSHav --delete /source /dest) this tree from one directory to another on the same host, rsync uses all the memory and is killed by oom-killer. This