Re: send/receive and bedup
On 21/5/2014 3:58 πμ, Chris Murphy wrote: On May 20, 2014, at 4:56 PM, Konstantinos Skarlatos k.skarla...@gmail.com wrote: On 21/5/2014 1:37 πμ, Mark Fasheh wrote: On Tue, May 20, 2014 at 01:07:50AM +0300, Konstantinos Skarlatos wrote: Duperemove will be shipping as supported software in a major SUSE release so it will be bug fixed, etc as you would expect. At the moment I'm very busy trying to fix qgroup bugs so I haven't had much time to add features, or handle external bug reports, etc. Also I'm not very good at advertising my software which would be why it hasn't really been mentioned on list lately :) I would say that state that it's in is that I've gotten the feature set to a point which feels reasonable, and I've fixed enough bugs that I'd appreciate folks giving it a spin and providing reasonable feedback. Well, after having good results with duperemove with a few gigs of data, i tried it on a 500gb subvolume. After it scanned all files, it is stuck at 100% of one cpu core for about 5 hours, and still hasn't done any deduping. My cpu is an Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz, so i guess thats not the problem. So I guess the speed of duperemove drops dramatically as data volume increases. Yeah I doubt it's your CPU. Duperemove is right now targeted at smaller data sets (a few VMS, iso images, etc) than you threw it at as you undoubtedly have figured out. It will need a bit of work before it can handle entire file systems. My guess is that it was spending an enormous amount of time finding duplicates (it has a very thorough check that could probably be optimized). It finished after 9 or so hours, so I agree it was checking for duplicates. It does a few GB in just seconds, so time probably scales exponentially with data size. I'm going to guess it ran out of memory. I wonder what happens if you take an SSD and specify a humongous swap partition on it. Like, 4x, or more, the amount of installed memory. Just tried it again, with 32GiB swap added on an SSD. My test files are 633GiB. duperemove -rv /storage/test 19537.67s user 183.86s system 89% cpu 6:06:56.96 total Duperemove was using about 1GiB or RAM, had one core at 100%, and I think swap was not touched at all. This same trick has been mentioned on the XFS list for use with xfsrepair when memory requirements exceed system memory, and is immensely faster. Chris Murphy -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: send/receive and bedup
On May 23, 2014, at 9:48 AM, Konstantinos Skarlatos k.skarla...@gmail.com wrote: On 21/5/2014 3:58 πμ, Chris Murphy wrote: On May 20, 2014, at 4:56 PM, Konstantinos Skarlatos k.skarla...@gmail.com wrote: On 21/5/2014 1:37 πμ, Mark Fasheh wrote: On Tue, May 20, 2014 at 01:07:50AM +0300, Konstantinos Skarlatos wrote: Duperemove will be shipping as supported software in a major SUSE release so it will be bug fixed, etc as you would expect. At the moment I'm very busy trying to fix qgroup bugs so I haven't had much time to add features, or handle external bug reports, etc. Also I'm not very good at advertising my software which would be why it hasn't really been mentioned on list lately :) I would say that state that it's in is that I've gotten the feature set to a point which feels reasonable, and I've fixed enough bugs that I'd appreciate folks giving it a spin and providing reasonable feedback. Well, after having good results with duperemove with a few gigs of data, i tried it on a 500gb subvolume. After it scanned all files, it is stuck at 100% of one cpu core for about 5 hours, and still hasn't done any deduping. My cpu is an Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz, so i guess thats not the problem. So I guess the speed of duperemove drops dramatically as data volume increases. Yeah I doubt it's your CPU. Duperemove is right now targeted at smaller data sets (a few VMS, iso images, etc) than you threw it at as you undoubtedly have figured out. It will need a bit of work before it can handle entire file systems. My guess is that it was spending an enormous amount of time finding duplicates (it has a very thorough check that could probably be optimized). It finished after 9 or so hours, so I agree it was checking for duplicates. It does a few GB in just seconds, so time probably scales exponentially with data size. I'm going to guess it ran out of memory. I wonder what happens if you take an SSD and specify a humongous swap partition on it. Like, 4x, or more, the amount of installed memory. Just tried it again, with 32GiB swap added on an SSD. My test files are 633GiB. duperemove -rv /storage/test 19537.67s user 183.86s system 89% cpu 6:06:56.96 total Duperemove was using about 1GiB or RAM, had one core at 100%, and I think swap was not touched at all. Guess currently it's not as memory intensive as it is cpu intensive while also not threading. Chris Murphy -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: send/receive and bedup
On 20 May 2014 06:07, Konstantinos Skarlatos k.skarla...@gmail.com wrote: On 19/5/2014 8:38 μμ, Mark Fasheh wrote: Well, after having good results with duperemove with a few gigs of data, i tried it on a 500gb subvolume. After it scanned all files, it is stuck at 100% of one cpu core for about 5 hours, and still hasn't done any deduping. My cpu is an Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz, so i guess thats not the problem. So I guess the speed of duperemove drops dramatically as data volume increases. There's a TODO list which gives a decent idea of what's on my mind for possible future improvements. I think what I'm most wanting to do right now is some sort of (optional) writeout to a file of what was done during a run. The idea is that you could feed that data back to duperemove to improve the speed of subsequent runs. My priorities may change depending on feedback from users of course. I also at some point want to rewrite some of the duplicate extent finding code as it got messy and could be a bit faster. --Mark I'm glad about this discussion. While I am no where near an expert on file systems, my knowledge has increased a lot through BtrFS. ZFS uses RAM to store its checksum tables. Opendedup recommends a separate HDD. Opendedup uses 4k block sizes. Both are always on. I'm not against using a separate HDD to store csums. Cheaper than RAM, albeit slower. The part of duperemove I like is the ability to CHOOSE when and how I want to dedupe. Scott -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: send/receive and bedup
On Tue, May 20, 2014 at 01:07:50AM +0300, Konstantinos Skarlatos wrote: Duperemove will be shipping as supported software in a major SUSE release so it will be bug fixed, etc as you would expect. At the moment I'm very busy trying to fix qgroup bugs so I haven't had much time to add features, or handle external bug reports, etc. Also I'm not very good at advertising my software which would be why it hasn't really been mentioned on list lately :) I would say that state that it's in is that I've gotten the feature set to a point which feels reasonable, and I've fixed enough bugs that I'd appreciate folks giving it a spin and providing reasonable feedback. Well, after having good results with duperemove with a few gigs of data, i tried it on a 500gb subvolume. After it scanned all files, it is stuck at 100% of one cpu core for about 5 hours, and still hasn't done any deduping. My cpu is an Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz, so i guess thats not the problem. So I guess the speed of duperemove drops dramatically as data volume increases. Yeah I doubt it's your CPU. Duperemove is right now targeted at smaller data sets (a few VMS, iso images, etc) than you threw it at as you undoubtedly have figured out. It will need a bit of work before it can handle entire file systems. My guess is that it was spending an enormous amount of time finding duplicates (it has a very thorough check that could probably be optimized). For what it's worth, handling larger data sets is the type of work I want to be doing on it in the future. --Mark -- Mark Fasheh -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: send/receive and bedup
On 21/5/2014 1:37 πμ, Mark Fasheh wrote: On Tue, May 20, 2014 at 01:07:50AM +0300, Konstantinos Skarlatos wrote: Duperemove will be shipping as supported software in a major SUSE release so it will be bug fixed, etc as you would expect. At the moment I'm very busy trying to fix qgroup bugs so I haven't had much time to add features, or handle external bug reports, etc. Also I'm not very good at advertising my software which would be why it hasn't really been mentioned on list lately :) I would say that state that it's in is that I've gotten the feature set to a point which feels reasonable, and I've fixed enough bugs that I'd appreciate folks giving it a spin and providing reasonable feedback. Well, after having good results with duperemove with a few gigs of data, i tried it on a 500gb subvolume. After it scanned all files, it is stuck at 100% of one cpu core for about 5 hours, and still hasn't done any deduping. My cpu is an Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz, so i guess thats not the problem. So I guess the speed of duperemove drops dramatically as data volume increases. Yeah I doubt it's your CPU. Duperemove is right now targeted at smaller data sets (a few VMS, iso images, etc) than you threw it at as you undoubtedly have figured out. It will need a bit of work before it can handle entire file systems. My guess is that it was spending an enormous amount of time finding duplicates (it has a very thorough check that could probably be optimized). It finished after 9 or so hours, so I agree it was checking for duplicates. It does a few GB in just seconds, so time probably scales exponentially with data size. For what it's worth, handling larger data sets is the type of work I want to be doing on it in the future. I can help with testing :) I would also suggest that you publish in this list any changes that you do, so that your program becomes better known among btrfs users. Or even a new announcement mail or a page in the btrfs wiki. Finally, i would like to request the ability to do file level dedup, with a reflink. That has the advantage of consuming very little metadata compared to block level dedup. It could be done with a two pass dedup, first comparing all the same-sized files and after that doing your normal block level dedup. Btw does anybody have a good program/script that can do file level dedup with reflinks and checksum comparison? Kind regards, Konstantinos Skarlatos --Mark -- Mark Fasheh -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: send/receive and bedup
On May 20, 2014, at 4:56 PM, Konstantinos Skarlatos k.skarla...@gmail.com wrote: On 21/5/2014 1:37 πμ, Mark Fasheh wrote: On Tue, May 20, 2014 at 01:07:50AM +0300, Konstantinos Skarlatos wrote: Duperemove will be shipping as supported software in a major SUSE release so it will be bug fixed, etc as you would expect. At the moment I'm very busy trying to fix qgroup bugs so I haven't had much time to add features, or handle external bug reports, etc. Also I'm not very good at advertising my software which would be why it hasn't really been mentioned on list lately :) I would say that state that it's in is that I've gotten the feature set to a point which feels reasonable, and I've fixed enough bugs that I'd appreciate folks giving it a spin and providing reasonable feedback. Well, after having good results with duperemove with a few gigs of data, i tried it on a 500gb subvolume. After it scanned all files, it is stuck at 100% of one cpu core for about 5 hours, and still hasn't done any deduping. My cpu is an Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz, so i guess thats not the problem. So I guess the speed of duperemove drops dramatically as data volume increases. Yeah I doubt it's your CPU. Duperemove is right now targeted at smaller data sets (a few VMS, iso images, etc) than you threw it at as you undoubtedly have figured out. It will need a bit of work before it can handle entire file systems. My guess is that it was spending an enormous amount of time finding duplicates (it has a very thorough check that could probably be optimized). It finished after 9 or so hours, so I agree it was checking for duplicates. It does a few GB in just seconds, so time probably scales exponentially with data size. I'm going to guess it ran out of memory. I wonder what happens if you take an SSD and specify a humongous swap partition on it. Like, 4x, or more, the amount of installed memory. This same trick has been mentioned on the XFS list for use with xfsrepair when memory requirements exceed system memory, and is immensely faster. Chris Murphy -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: send/receive and bedup
On 19 May 2014 09:07, Marc MERLIN m...@merlins.org wrote: On Wed, May 14, 2014 at 11:36:03PM +0800, Scott Middleton wrote: I read so much about BtrFS that I mistaked Bedup with Duperemove. Duperemove is actually what I am testing. I'm currently using programs that find files that are the same, and hardlink them together: http://marc.merlins.org/perso/linux/post_2012-05-01_Handy-tip-to-save-on-inodes-and-disk-space_-finddupes_-fdupes_-and-hardlink_py.html hardlink.py actually seems to be the faster (memory and CPU) one event though it's in python. I can get others to run out of RAM on my 8GB server easily :( Bedup should be better, but last I tried I couldn't get it to work. It's been updated since then, I just haven't had the chance to try it again since then. Please post what you find out, or if you have a hardlink maker that's better than the ones I found :) Thanks for that. I may be completely wrong in my approach. I am not looking for a file level comparison. Bedup worked fine for that. I have a lot of virtual images and shadow protect images where only a few megabytes may be the difference. So a file level hash and comparison doesn't really achieve my goals. I thought duperemove may be on a lower level. https://github.com/markfasheh/duperemove Duperemove is a simple tool for finding duplicated extents and submitting them for deduplication. When given a list of files it will hash their contents on a block by block basis and compare those hashes to each other, finding and categorizing extents that match each other. When given the -d option, duperemove will submit those extents for deduplication using the btrfs-extent-same ioctl. It defaults to 128k but you can make it smaller. I hit a hurdle though. The 3TB HDD I used seemed OK when I did a long SMART test but seems to die every few hours. Admittedly it was part of a failed mdadm RAID array that I pulled out of a clients machine. The only other copy I have of the data is the original mdadm array that was recently replaced with a new server, so I am loathe to use that HDD yet. At least for another couple of weeks! I am still hopeful duperemove will work. In another month I will put the 2 X 4TB HDDs online in BtrFS RAID 1 format on the production machine and have a crack on duperemove on that after hours. I will convert the onsite backup machine to BtrFS with its 2 x 4TB HDDs to BtrFS not long after. The ultimate goal is to be able to back up on a block level very large files offsite where maybe a GB is changed on a daily basis. I realise that I will have to make an original copy and manually take that to my datacentre but hopefully I can backup multiple clients data after hours, or possibly, a trickle, constantly. Kind Regards Scott -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: send/receive and bedup
On 19/05/14 15:00, Scott Middleton wrote: On 19 May 2014 09:07, Marc MERLIN m...@merlins.org wrote: On Wed, May 14, 2014 at 11:36:03PM +0800, Scott Middleton wrote: I read so much about BtrFS that I mistaked Bedup with Duperemove. Duperemove is actually what I am testing. I'm currently using programs that find files that are the same, and hardlink them together: http://marc.merlins.org/perso/linux/post_2012-05-01_Handy-tip-to-save-on-inodes-and-disk-space_-finddupes_-fdupes_-and-hardlink_py.html hardlink.py actually seems to be the faster (memory and CPU) one event though it's in python. I can get others to run out of RAM on my 8GB server easily :( Interesting app. An issue with hardlinking (with the backups use-case, this problem isn't likely to happen), is that if you modify a file, all the hardlinks get changed along with it - including the ones that you don't want changed. @Marc: Since you've been using btrfs for a while now I'm sure you've already considered whether or not a reflink copy is the better/worse option. Bedup should be better, but last I tried I couldn't get it to work. It's been updated since then, I just haven't had the chance to try it again since then. Please post what you find out, or if you have a hardlink maker that's better than the ones I found :) Thanks for that. I may be completely wrong in my approach. I am not looking for a file level comparison. Bedup worked fine for that. I have a lot of virtual images and shadow protect images where only a few megabytes may be the difference. So a file level hash and comparison doesn't really achieve my goals. I thought duperemove may be on a lower level. https://github.com/markfasheh/duperemove Duperemove is a simple tool for finding duplicated extents and submitting them for deduplication. When given a list of files it will hash their contents on a block by block basis and compare those hashes to each other, finding and categorizing extents that match each other. When given the -d option, duperemove will submit those extents for deduplication using the btrfs-extent-same ioctl. It defaults to 128k but you can make it smaller. I hit a hurdle though. The 3TB HDD I used seemed OK when I did a long SMART test but seems to die every few hours. Admittedly it was part of a failed mdadm RAID array that I pulled out of a clients machine. The only other copy I have of the data is the original mdadm array that was recently replaced with a new server, so I am loathe to use that HDD yet. At least for another couple of weeks! I am still hopeful duperemove will work. Duperemove does look exactly like what you are looking for. The last traffic on the mailing list regarding that was in August last year. It looks like it was pulled into the main kernel repository on September 1st. The last commit to the duperemove application was on April 20th this year. Maybe Mark (cc'd) can provide further insight on its current status. -- __ Brendan Hide http://swiftspirit.co.za/ http://www.webafrica.co.za/?AFF1E97 -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: send/receive and bedup
On 19/5/2014 7:01 μμ, Brendan Hide wrote: On 19/05/14 15:00, Scott Middleton wrote: On 19 May 2014 09:07, Marc MERLIN m...@merlins.org wrote: On Wed, May 14, 2014 at 11:36:03PM +0800, Scott Middleton wrote: I read so much about BtrFS that I mistaked Bedup with Duperemove. Duperemove is actually what I am testing. I'm currently using programs that find files that are the same, and hardlink them together: http://marc.merlins.org/perso/linux/post_2012-05-01_Handy-tip-to-save-on-inodes-and-disk-space_-finddupes_-fdupes_-and-hardlink_py.html hardlink.py actually seems to be the faster (memory and CPU) one event though it's in python. I can get others to run out of RAM on my 8GB server easily :( Interesting app. An issue with hardlinking (with the backups use-case, this problem isn't likely to happen), is that if you modify a file, all the hardlinks get changed along with it - including the ones that you don't want changed. @Marc: Since you've been using btrfs for a while now I'm sure you've already considered whether or not a reflink copy is the better/worse option. Bedup should be better, but last I tried I couldn't get it to work. It's been updated since then, I just haven't had the chance to try it again since then. Please post what you find out, or if you have a hardlink maker that's better than the ones I found :) Thanks for that. I may be completely wrong in my approach. I am not looking for a file level comparison. Bedup worked fine for that. I have a lot of virtual images and shadow protect images where only a few megabytes may be the difference. So a file level hash and comparison doesn't really achieve my goals. I thought duperemove may be on a lower level. https://github.com/markfasheh/duperemove Duperemove is a simple tool for finding duplicated extents and submitting them for deduplication. When given a list of files it will hash their contents on a block by block basis and compare those hashes to each other, finding and categorizing extents that match each other. When given the -d option, duperemove will submit those extents for deduplication using the btrfs-extent-same ioctl. It defaults to 128k but you can make it smaller. I hit a hurdle though. The 3TB HDD I used seemed OK when I did a long SMART test but seems to die every few hours. Admittedly it was part of a failed mdadm RAID array that I pulled out of a clients machine. The only other copy I have of the data is the original mdadm array that was recently replaced with a new server, so I am loathe to use that HDD yet. At least for another couple of weeks! I am still hopeful duperemove will work. Duperemove does look exactly like what you are looking for. The last traffic on the mailing list regarding that was in August last year. It looks like it was pulled into the main kernel repository on September 1st. The last commit to the duperemove application was on April 20th this year. Maybe Mark (cc'd) can provide further insight on its current status. I have been testing duperemove and it seems to work just fine, in contrast with bedup that i have been unable to install/compile/sort out the mess with python versions. I have 2 questions about duperemove: 1) can it use existing filesystem csums instead of calculating its own? 2) can it be included in btrfs-progs so that it becomes a standard feature of btrfs? Thanks -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: send/receive and bedup
On Mon, May 19, 2014 at 06:01:25PM +0200, Brendan Hide wrote:
On 19/05/14 15:00, Scott Middleton wrote:
On 19 May 2014 09:07, Marc MERLIN m...@merlins.org wrote:
Thanks for that.
I may be completely wrong in my approach.
I am not looking for a file level comparison. Bedup worked fine for
that. I have a lot of virtual images and shadow protect images where
only a few megabytes may be the difference. So a file level hash and
comparison doesn't really achieve my goals.
I thought duperemove may be on a lower level.
https://github.com/markfasheh/duperemove
Duperemove is a simple tool for finding duplicated extents and
submitting them for deduplication. When given a list of files it will
hash their contents on a block by block basis and compare those hashes
to each other, finding and categorizing extents that match each
other. When given the -d option, duperemove will submit those
extents for deduplication using the btrfs-extent-same ioctl.
It defaults to 128k but you can make it smaller.
I hit a hurdle though. The 3TB HDD I used seemed OK when I did a long
SMART test but seems to die every few hours. Admittedly it was part of
a failed mdadm RAID array that I pulled out of a clients machine.
The only other copy I have of the data is the original mdadm array
that was recently replaced with a new server, so I am loathe to use
that HDD yet. At least for another couple of weeks!
I am still hopeful duperemove will work.
Duperemove does look exactly like what you are looking for. The last
traffic on the mailing list regarding that was in August last year. It
looks like it was pulled into the main kernel repository on September 1st.
I'm confused - you need to avoid a file scan completely? Duperemove does do
that just to be clear.
In your mind, what would be the alternative to that sort of a scan?
By the way, if you know exactly where the changes are you
could just feed the duplicate extents directly to the ioctl via a script. I
have a small tool in the duperemove repositry that can do that for you
('make btrfs-extent-same').
The last commit to the duperemove application was on April 20th this year.
Maybe Mark (cc'd) can provide further insight on its current status.
Duperemove will be shipping as supported software in a major SUSE release so
it will be bug fixed, etc as you would expect. At the moment I'm very busy
trying to fix qgroup bugs so I haven't had much time to add features, or
handle external bug reports, etc. Also I'm not very good at advertising my
software which would be why it hasn't really been mentioned on list lately
:)
I would say that state that it's in is that I've gotten the feature set to a
point which feels reasonable, and I've fixed enough bugs that I'd appreciate
folks giving it a spin and providing reasonable feedback.
There's a TODO list which gives a decent idea of what's on my mind for
possible future improvements. I think what I'm most wanting to do right now
is some sort of (optional) writeout to a file of what was done during a run.
The idea is that you could feed that data back to duperemove to improve the
speed of subsequent runs. My priorities may change depending on feedback
from users of course.
I also at some point want to rewrite some of the duplicate extent finding
code as it got messy and could be a bit faster.
--Mark
--
Mark Fasheh
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: send/receive and bedup
On Mon, May 19, 2014 at 08:12:03PM +0300, Konstantinos Skarlatos wrote: On 19/5/2014 7:01 μμ, Brendan Hide wrote: On 19/05/14 15:00, Scott Middleton wrote: Duperemove does look exactly like what you are looking for. The last traffic on the mailing list regarding that was in August last year. It looks like it was pulled into the main kernel repository on September 1st. The last commit to the duperemove application was on April 20th this year. Maybe Mark (cc'd) can provide further insight on its current status. I have been testing duperemove and it seems to work just fine, in contrast with bedup that i have been unable to install/compile/sort out the mess with python versions. I have 2 questions about duperemove: 1) can it use existing filesystem csums instead of calculating its own? Not right now, though that may be something we can feed to it in the future. I haven't thought about this much and to be honest I don't recall *exactly* how btrfs stores it's checksums. That said, I think feasibility of doing this comes down to a few things: 1) how expensive is it to get at the on-disk checksums? This might not make sense if it's simply faster to scan a file than its checksums. 2) are they stored in a manner which makes sense for dedupe. By that I mean, do we have a checksum for every X bytes? If so, then theoretically life is easy - we just make our blocksize to X and load the checksums into duperemoves internal block checksum tree. If checksums can cover arbitrary sized extents than we might not be able to use them at all or maybe we would have to 'fill in the blanks' so to speak. 3) what is the tradeoff of false positives? Btrfs checksums are there for detecting bad blocks, as opposed to duplicate data. The difference is that btrfs doesn't have to use very strong hashing as a result. So we just want to make sure that we don't wind up passing *so* many false positives to the kernel that it was just faster to scan the file and checksum on our own. Not that any of those questions are super difficult to answer by the way, it's more about how much time I've had :) 2) can it be included in btrfs-progs so that it becomes a standard feature of btrfs? I have to think about this one personally as it implies some tradeoffs in my development on duperemove that I'm not sure I want to make yet. --Mark -- Mark Fasheh -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: send/receive and bedup
On 2014-05-19 13:12, Konstantinos Skarlatos wrote: On 19/5/2014 7:01 μμ, Brendan Hide wrote: On 19/05/14 15:00, Scott Middleton wrote: On 19 May 2014 09:07, Marc MERLIN m...@merlins.org wrote: On Wed, May 14, 2014 at 11:36:03PM +0800, Scott Middleton wrote: I read so much about BtrFS that I mistaked Bedup with Duperemove. Duperemove is actually what I am testing. I'm currently using programs that find files that are the same, and hardlink them together: http://marc.merlins.org/perso/linux/post_2012-05-01_Handy-tip-to-save-on-inodes-and-disk-space_-finddupes_-fdupes_-and-hardlink_py.html hardlink.py actually seems to be the faster (memory and CPU) one event though it's in python. I can get others to run out of RAM on my 8GB server easily :( Interesting app. An issue with hardlinking (with the backups use-case, this problem isn't likely to happen), is that if you modify a file, all the hardlinks get changed along with it - including the ones that you don't want changed. @Marc: Since you've been using btrfs for a while now I'm sure you've already considered whether or not a reflink copy is the better/worse option. Bedup should be better, but last I tried I couldn't get it to work. It's been updated since then, I just haven't had the chance to try it again since then. Please post what you find out, or if you have a hardlink maker that's better than the ones I found :) Thanks for that. I may be completely wrong in my approach. I am not looking for a file level comparison. Bedup worked fine for that. I have a lot of virtual images and shadow protect images where only a few megabytes may be the difference. So a file level hash and comparison doesn't really achieve my goals. I thought duperemove may be on a lower level. https://github.com/markfasheh/duperemove Duperemove is a simple tool for finding duplicated extents and submitting them for deduplication. When given a list of files it will hash their contents on a block by block basis and compare those hashes to each other, finding and categorizing extents that match each other. When given the -d option, duperemove will submit those extents for deduplication using the btrfs-extent-same ioctl. It defaults to 128k but you can make it smaller. I hit a hurdle though. The 3TB HDD I used seemed OK when I did a long SMART test but seems to die every few hours. Admittedly it was part of a failed mdadm RAID array that I pulled out of a clients machine. The only other copy I have of the data is the original mdadm array that was recently replaced with a new server, so I am loathe to use that HDD yet. At least for another couple of weeks! I am still hopeful duperemove will work. Duperemove does look exactly like what you are looking for. The last traffic on the mailing list regarding that was in August last year. It looks like it was pulled into the main kernel repository on September 1st. The last commit to the duperemove application was on April 20th this year. Maybe Mark (cc'd) can provide further insight on its current status. I have been testing duperemove and it seems to work just fine, in contrast with bedup that i have been unable to install/compile/sort out the mess with python versions. I have 2 questions about duperemove: 1) can it use existing filesystem csums instead of calculating its own? While this might seem like a great idea at first, it really isn't. BTRFS uses CRC32c at the moment as it's checksum algorithm, and while that is relatively good at detecting small differences (i.e. a single bit flipped out of every 64 or so bytes), it is known to have issues with hash collisions. Normally, the data on disk won't change enough even from a media error to cause a hash collision, but when you start using it to compare extents that aren't known to be the same to begin with, and then try to merge those extents, you run the risk of serious file corruption. Also, AFAIK, BTRFS doesn't expose the block checksum to userspace directly (although I may be wrong about this, in which case i retract the following statement) this would therefore require some kernelspace support. 2) can it be included in btrfs-progs so that it becomes a standard feature of btrfs? I would definitely like to second this suggestion, I hear a lot of people talking about how BTRFS has batch deduplication, but it's almost impossible to make use of without extra software or writing your own code. smime.p7s Description: S/MIME Cryptographic Signature
Re: send/receive and bedup
On Mon, May 19, 2014 at 01:59:01PM -0400, Austin S Hemmelgarn wrote: On 2014-05-19 13:12, Konstantinos Skarlatos wrote: I have been testing duperemove and it seems to work just fine, in contrast with bedup that i have been unable to install/compile/sort out the mess with python versions. I have 2 questions about duperemove: 1) can it use existing filesystem csums instead of calculating its own? While this might seem like a great idea at first, it really isn't. BTRFS uses CRC32c at the moment as it's checksum algorithm, and while that is relatively good at detecting small differences (i.e. a single bit flipped out of every 64 or so bytes), it is known to have issues with hash collisions. Normally, the data on disk won't change enough even from a media error to cause a hash collision, but when you start using it to compare extents that aren't known to be the same to begin with, and then try to merge those extents, you run the risk of serious file corruption. Also, AFAIK, BTRFS doesn't expose the block checksum to userspace directly (although I may be wrong about this, in which case i retract the following statement) this would therefore require some kernelspace support. I'm pretty sure you could get the checkums via ioctl. The thing about dedupe though is that kernel is always doing a byte-by-byte comparison of the file data before merging it so we should never corrupt just because userspace gave us a bad range to dedupe. That said I don't necessarily disagree that it might not be as good an idea as it sounds. --Mark -- Mark Fasheh -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: send/receive and bedup
On 19/5/2014 8:38 μμ, Mark Fasheh wrote:
On Mon, May 19, 2014 at 06:01:25PM +0200, Brendan Hide wrote:
On 19/05/14 15:00, Scott Middleton wrote:
On 19 May 2014 09:07, Marc MERLIN m...@merlins.org wrote:
Thanks for that.
I may be completely wrong in my approach.
I am not looking for a file level comparison. Bedup worked fine for
that. I have a lot of virtual images and shadow protect images where
only a few megabytes may be the difference. So a file level hash and
comparison doesn't really achieve my goals.
I thought duperemove may be on a lower level.
https://github.com/markfasheh/duperemove
Duperemove is a simple tool for finding duplicated extents and
submitting them for deduplication. When given a list of files it will
hash their contents on a block by block basis and compare those hashes
to each other, finding and categorizing extents that match each
other. When given the -d option, duperemove will submit those
extents for deduplication using the btrfs-extent-same ioctl.
It defaults to 128k but you can make it smaller.
I hit a hurdle though. The 3TB HDD I used seemed OK when I did a long
SMART test but seems to die every few hours. Admittedly it was part of
a failed mdadm RAID array that I pulled out of a clients machine.
The only other copy I have of the data is the original mdadm array
that was recently replaced with a new server, so I am loathe to use
that HDD yet. At least for another couple of weeks!
I am still hopeful duperemove will work.
Duperemove does look exactly like what you are looking for. The last
traffic on the mailing list regarding that was in August last year. It
looks like it was pulled into the main kernel repository on September 1st.
I'm confused - you need to avoid a file scan completely? Duperemove does do
that just to be clear.
In your mind, what would be the alternative to that sort of a scan?
By the way, if you know exactly where the changes are you
could just feed the duplicate extents directly to the ioctl via a script. I
have a small tool in the duperemove repositry that can do that for you
('make btrfs-extent-same').
The last commit to the duperemove application was on April 20th this year.
Maybe Mark (cc'd) can provide further insight on its current status.
Duperemove will be shipping as supported software in a major SUSE release so
it will be bug fixed, etc as you would expect. At the moment I'm very busy
trying to fix qgroup bugs so I haven't had much time to add features, or
handle external bug reports, etc. Also I'm not very good at advertising my
software which would be why it hasn't really been mentioned on list lately
:)
I would say that state that it's in is that I've gotten the feature set to a
point which feels reasonable, and I've fixed enough bugs that I'd appreciate
folks giving it a spin and providing reasonable feedback.
Well, after having good results with duperemove with a few gigs of data,
i tried it on a 500gb subvolume. After it scanned all files, it is stuck
at 100% of one cpu core for about 5 hours, and still hasn't done any
deduping. My cpu is an Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz, so i
guess thats not the problem. So I guess the speed of duperemove drops
dramatically as data volume increases.
There's a TODO list which gives a decent idea of what's on my mind for
possible future improvements. I think what I'm most wanting to do right now
is some sort of (optional) writeout to a file of what was done during a run.
The idea is that you could feed that data back to duperemove to improve the
speed of subsequent runs. My priorities may change depending on feedback
from users of course.
I also at some point want to rewrite some of the duplicate extent finding
code as it got messy and could be a bit faster.
--Mark
--
Mark Fasheh
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: send/receive and bedup
On Wed, May 14, 2014 at 11:36:03PM +0800, Scott Middleton wrote: I read so much about BtrFS that I mistaked Bedup with Duperemove. Duperemove is actually what I am testing. I'm currently using programs that find files that are the same, and hardlink them together: http://marc.merlins.org/perso/linux/post_2012-05-01_Handy-tip-to-save-on-inodes-and-disk-space_-finddupes_-fdupes_-and-hardlink_py.html hardlink.py actually seems to be the faster (memory and CPU) one event though it's in python. I can get others to run out of RAM on my 8GB server easily :( Bedup should be better, but last I tried I couldn't get it to work. It's been updated since then, I just haven't had the chance to try it again since then. Please post what you find out, or if you have a hardlink maker that's better than the ones I found :) Thanks, Marc -- A mouse is a device used to point at the xterm you want to type in - A.S.R. Microsoft is to operating systems what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ | PGP 1024R/763BE901 -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: send/receive and bedup
Scott Middleton posted on Mon, 12 May 2014 20:27:13 +0800 as excerpted: Hi Everyone History: I just recently discovered BtrFS. Well really only just started reading a lot about it. Starting with blogs by Jim Salters and Marc Merlin. So, thanks for those blogs guys. This also introduced me to ZoL (ZFS). It seemed a bit more stable but one of the features I really wanted was deduplication and needing 20GB RAM for 1TB of deduped data and the fact it is always on - pushed me away. Some of those guys really don't like BtrFS BTW! What I want to be able to do is backup Virtual images (VirtualBox and some older VMware) over ADSL. I hoped a mixture of Dedupe and the send/receive functions of BtrFS might accomplish this. I am in the process of building a test BtrFS system at home now but I hope to put it into production in the next few months. Current server is a Dual Xeon, Intel server board, 32GB RAM, 2 x 2TB Hardware RAID SAS (consisting of 4 x 2TB SATA drives) as / and /home in ext4 format. I also have 2 unused 4TB SATA drives that will be eventually be BtrFS RAID1 as /Backup. Ubuntu 14.04. Its is mainly a VM host, small file storage for word docs etc, some large archival .pst files and shadowprotect backups of the Terminal Server. I only just built this server over the last weekend to replace their other aging server and I purposely over engineered it. Onsite there is also a backup server that is pretty basic with 2 x 4TB HDDs and 8GB RAM. I plan on converting it to BtrFS as well. Currently is Ubuntu 12.04 but I will be upgrading it soon to 14.04. Offsite in a data centre I have an aging 1 RU server that I will be upgrading. It'll probably have 8GB RAM, 1 X 60GB SSD as boot/swap and 2 X 4TB HDD BtrFS in RAID 1. Currently running 32 bit Debian 7.5. It has had many partial hardware and OS upgrades over the years as it originally started as Slink or even Hamm. Time to start again since I need to move to 64bit. What I want to do is backup the / and /home directories on the main server to /Backup BtrFS directory, run bedup then send it to the onsite backup server. The onsite backup server will send it to the offsite server. I am assuming (correctly I hope) that the deduplication will also be replicated across the machines. I'll have NOCOW on the VM images, Archived PST files, Shadow protect images and some other stuff. I guess the first question is this even possible? I don't believe that much actual non duplicated data changes all that much mostly just word docs that I already send offsite. I'm really hoping to backup the VMs and the shadow protects offsite as well. I can upgrade the broadband to fibre but before I do that (spend a lot of money) I want to be able to see that it would be possible. I left this for a couple days hoping someone else with a more directly similar use-case would answer, but none so far, so I'll give it a go... First some general boilerplate. Btrfs is still under heavy development and keeping current with especially the kernel is *STRONGLY* recommended, as every new kernel still brings lots of fixes, meaning if you're running an old kernel, you're running known-buggy code with fixes available in a current kernel. Similarly, you probably don't want to let the btrfs-progs userspace tools get too outdated either, tho that's not as critical as it mostly means not being able to take advantage of the latest features and fixes for maintenance, not the risk of operational data loss if one of the known-fixed old-version kernel bugs hits that you have when running an older kernel. Second, as you've done some research already you're likely aware of this, but just in case, let me give you the link to the wiki. If you haven't read up there, please do, as it's likely to be quite helpful. =:^) Memory or bookmark: https://btrfs.wiki.kernel.org User documentation bookmark: https://btrfs.wiki.kernel.org/index.php/Main_Page#Guides_and_usage_information On to your proposed setup. In general, it looks reasonable. My first concern upon reading about the VM images was of course the fragmentation issues that come with the VM images territory on btrfs, but if you keep your working partitions as ext4 and use btrfs primarily for backup, that issue goes away to a large extent, since the operational rewriting will be happening on the ext4, which should handle it a bit better than btrfs does at this point, while btrfs will only be getting the more sequentially written backups and not have to deal with the live in-place updates on the VM images. The problem with btrfs hosting operational VM images is snapshots, particularly when using btrfs send for backup and/or when doing frequent, often scripted, snapshotting. The problem with send is that it takes a read-only snapshot and sends from that, so it's a snapshotting issue either way. The problem with snapshots is that for NOCOW files, the first write to a block after
Re: send/receive and bedup
Hi I left this for a couple days hoping someone else with a more directly similar use-case would answer, but none so far, so I'll give it a go... Thanks for getting back to me mate! First some general boilerplate. Btrfs is still under heavy development and keeping current with especially the kernel is *STRONGLY* recommended, as every new kernel still brings lots of fixes, meaning if you're running an old kernel, you're running known-buggy code with fixes available in a current kernel. Similarly, you probably don't want to let the btrfs-progs userspace tools get too outdated either, tho that's not as critical as it mostly means not being able to take advantage of the latest features and fixes for maintenance, not the risk of operational data loss if one of the known-fixed old-version kernel bugs hits that you have when running an older kernel. My test environment is: root@Ubuntu-14:~/btrfs/btrfs-progs# uname -a Linux Ubuntu-14 3.14.1-031401-generic #201404141220 SMP Mon Apr 14 16:21:48 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux root@Ubuntu-14:~/btrfs/btrfs-progs# which btrfs /usr/local/bin/btrfs root@Ubuntu-14:~/btrfs/btrfs-progs# btrfs --version Btrfs v3.14.1 I read so much about BtrFS that I mistaked Bedup with Duperemove. Duperemove is actually what I am testing. I also plan on testing SDFS - opendedup again! They have some development since the last time I tried, Not a fan of having to use Java though! Second, as you've done some research already you're likely aware of this, but just in case, let me give you the link to the wiki. If you haven't read up there, please do, as it's likely to be quite helpful. =:^) Memory or bookmark: https://btrfs.wiki.kernel.org Lots of bookmarks. One of my faves is: https://wiki.archlinux.org/index.php/Btrfs It will be interesting on what happens. I rsync'd the clients data from a 2TB mdadm RAID to a standard 3TB BtrFS drive, Currently running Duperemove on it. I reckon it will take days but it will be interesting to see what happens. -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
