On 21/5/2014 3:58 πμ, Chris Murphy wrote:
On May 20, 2014, at 4:56 PM, Konstantinos Skarlatos <k.skarla...@gmail.com>
wrote:
On 21/5/2014 1:37 πμ, Mark Fasheh wrote:
On Tue, May 20, 2014 at 01:07:50AM +0300, Konstantinos Skarlatos wrote:
Duperemove will be shipping as supported software in a major SUSE release so
it will be bug fixed, etc as you would expect. At the moment I'm very busy
trying to fix qgroup bugs so I haven't had much time to add features, or
handle external bug reports, etc. Also I'm not very good at advertising my
software which would be why it hasn't really been mentioned on list lately
:)
I would say that state that it's in is that I've gotten the feature set to a
point which feels reasonable, and I've fixed enough bugs that I'd appreciate
folks giving it a spin and providing reasonable feedback.
Well, after having good results with duperemove with a few gigs of data, i
tried it on a 500gb subvolume. After it scanned all files, it is stuck at
100% of one cpu core for about 5 hours, and still hasn't done any deduping.
My cpu is an Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz, so i guess thats
not the problem. So I guess the speed of duperemove drops dramatically as
data volume increases.
Yeah I doubt it's your CPU. Duperemove is right now targeted at smaller data
sets (a few VMS, iso images, etc) than you threw it at as you undoubtedly
have figured out. It will need a bit of work before it can handle entire
file systems. My guess is that it was spending an enormous amount of time
finding duplicates (it has a very thorough check that could probably be
optimized).
It finished after 9 or so hours, so I agree it was checking for duplicates. It
does a few GB in just seconds, so time probably scales exponentially with data
size.
I'm going to guess it ran out of memory. I wonder what happens if you take an
SSD and specify a humongous swap partition on it. Like, 4x, or more, the amount
of installed memory.
Just tried it again, with 32GiB swap added on an SSD. My test files are
633GiB.
duperemove -rv /storage/test 19537.67s user 183.86s system 89% cpu
6:06:56.96 total
Duperemove was using about 1GiB or RAM, had one core at 100%, and I
think swap was not touched at all.
This same trick has been mentioned on the XFS list for use with xfsrepair when
memory requirements exceed system memory, and is immensely faster.
Chris Murphy
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
