On Wed, 09 Nov 2016 12:24:51 +0100, Niccolò Belli <darkba...@linuxsystems.it> wrote : > > On martedì 8 novembre 2016 23:36:25 CET, Saint Germain wrote: > > Please be aware of these other similar softwares: > > - jdupes: https://github.com/jbruchon/jdupes > > - rmlint: https://github.com/sahib/rmlint > > And of course fdupes. > > > > Some intesting points I have seen in them: > > - use xxhash to identify potential duplicates (huge speedup) > > - ability to deduplicate read-only snapshots > > - identify potential reflinked files (see also my email here: > > https://www.spinics.net/lists/linux-btrfs/msg60081.html) > > - ability to filter out hardlinks > > - triangle problem: see jdupes readme > > - jdupes has started the process to be included in Debian > > > > I hope that will help and that you can share some codes with them ! > > > Hi, > What do you think about jdupes? I'm searching an alternative to > duperemove and rmlint doesn't seem to support btrfs deduplication, so > I would like to try jdupes. My main problem with duperemove is a > memory leak, also it seems to lead to greater disk usage: > https://github.com/markfasheh/duperemove/issues/163
rmlint is supporting btrfs deduplication: rmlint --algorithm=xxhash --types="duplicates" --hidden --config=sh:handler=clone --no-hardlinked I've used jdupes and rmlint to deduplicate 2TB with 4GB RAM and it took a few hours. So it is acceptable from a performance point of view. The problems I found have been corrected by both. Jdupes author is really kind and reactive ! -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
