Re: [survey] sysfs layout for btrfs

2015-08-18 Thread Gabriel de Perthuis
On Sat, 15 Aug 2015 07:40:40 +0800, Anand Jain wrote: Hello, as of now btrfs sysfs does not include the attributes for the volume manager part in its sysfs layout, so its being developed and there are two types of layout here below, so I have a quick survey to know which will be

Re: btrfs filesystem show _exact_ freaking size?

2014-11-18 Thread Gabriel de Perthuis
Le 18/11/2014 11:39, Robert White a écrit : Howdy, How does one get the exact size (in blocks preferably, but bytes okay) of the filesystem inside a partition? I know how to get the partition size, but that's not useful when shrinking a partition... dev_item.total_bytes in

Re: Manual deduplication would be useful

2013-07-23 Thread Gabriel de Perthuis
Hello, For over a year now, I've been experimenting with stacked filesystems as a way to save on resources. A basic OS layer is shared among Containers, each of which stacks a layer with modifications on top of it. This approach means that Containers share buffer cache and loaded

Re: Q: Why subvolumes?

2013-07-23 Thread Gabriel de Perthuis
Now... since the snapshot's FS tree is a direct duplicate of the original FS tree (actually, it's the same tree, but they look like different things to the outside world), they share everything -- including things like inode numbers. This is OK within a subvolume, because we have the

Re: Q: Why subvolumes?

2013-07-23 Thread Gabriel de Perthuis
Le mar. 23 juil. 2013 21:30:13 CEST, Hugo Mills a écrit : On Tue, Jul 23, 2013 at 07:47:41PM +0200, Gabriel de Perthuis wrote: Now... since the snapshot's FS tree is a direct duplicate of the original FS tree (actually, it's the same tree, but they look like different things to the outside

Re: Lots of harddrive chatter on after booting with btrfs on root (slow boot)

2013-07-20 Thread Gabriel de Perthuis
On Sat, 20 Jul 2013 17:15:50 +0200, Jason Russell wrote: Ive also noted that this excessive hdd chatter does not occur immediately after a fresh format with arch on btrfs root. Ive made some deductions/assumptions: This only seems to occur with btrfs roots. This only happens after some

Re: [PATCH 4/4] btrfs: offline dedupe

2013-07-16 Thread Gabriel de Perthuis
On Mon, 15 Jul 2013 13:55:51 -0700, Zach Brown wrote: I'd get rid of all this code by only copying each input argument on to the stack as it's needed and by getting rid of the writable output struct fields. (more on this later) As I said, I'd get rid of the output fields. Like the other

[XFSTESTS PATCH] btrfs: Test deduplication

2013-06-26 Thread Gabriel de Perthuis
--- The matching kernel patch is here: https://github.com/g2p/linux/tree/v3.10%2Bextent-same (rebased on 3.10, fixing a small conflict) Requires the btrfs-extent-same command: - http://permalink.gmane.org/gmane.comp.file-systems.btrfs/26579 - https://github.com/markfasheh/duperemove

Re: Two identical copies of an image mounted result in changes to both images if only one is modified

2013-06-20 Thread Gabriel de Perthuis
On Thu, 20 Jun 2013 10:16:22 +0100, Hugo Mills wrote: On Thu, Jun 20, 2013 at 10:47:53AM +0200, Clemens Eisserer wrote: Hi, I've observed a rather strange behaviour while trying to mount two identical copies of the same image to different mount points. Each modification to one image is also

Re: Two identical copies of an image mounted result in changes to both images if only one is modified

2013-06-20 Thread Gabriel de Perthuis
Instead of redirecting to a different block device, Btrfs could and should refuse to mount an already-mounted superblock when the block device doesn't match, somewhere in or below btrfs_mount. Registering extra, distinct superblocks for an already mounted raid is a different matter, but that

Re: [PATCH 0/4] btrfs: offline dedupe v2

2013-06-11 Thread Gabriel de Perthuis
Le 11/06/2013 22:31, Mark Fasheh a écrit : Perhaps this isn't a limiation per-se but extent-same requires read/write access to the files we want to dedupe. During my last series I had a conversation with Gabriel de Perthuis about access checking where we tried to maintain the ability

Re: [PATCH 0/4] btrfs: offline dedupe v2

2013-06-11 Thread Gabriel de Perthuis
Le 11/06/2013 23:04, Mark Fasheh a écrit : On Tue, Jun 11, 2013 at 10:56:59PM +0200, Gabriel de Perthuis wrote: What I found however is that neither of these is a great idea ;) - We want to require that the inode be open for writing so that an unprivileged user can't do things like run

Re: [PATCH 4/4] btrfs: offline dedupe

2013-05-24 Thread Gabriel de Perthuis
+#define BTRFS_MAX_DEDUPE_LEN (16 * 1024 * 1024) +#define BTRFS_ONE_DEDUPE_LEN (1 * 1024 * 1024) + +static long btrfs_ioctl_file_extent_same(struct file *file, +void __user *argp) +{ + struct btrfs_ioctl_same_args *args; + struct

Re: [PATCH 4/4] btrfs: offline dedupe

2013-05-24 Thread Gabriel de Perthuis
Le sam. 25 mai 2013 00:38:27 CEST, Mark Fasheh a écrit : On Fri, May 24, 2013 at 09:50:14PM +0200, Gabriel de Perthuis wrote: Sure. Actually, you got me thinking about some sanity checks... I need to add at least this check: if (btrfs_root_readonly(root)) return -EROFS

Re: [PATCH 0/4] a structure for the disks scan for btrfs

2013-05-17 Thread Gabriel de Perthuis
On Fri, 17 May 2013 18:54:38 +0800, Anand Jain wrote: The idea was to introduce /dev/mapper to find for btrfs disk, However I found first we need to congregate the disk scan procedure at a function so it would help to consistently tune it across the btrfs-progs. As of now both fi show and

Re: [PATCH 0/4] a structure for the disks scan for btrfs

2013-05-17 Thread Gabriel de Perthuis
Just scan /dev/block/*. That contains all block devices. Oh, this is about finding nicer names. Never mind. -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at

Re: subvol copying

2013-05-15 Thread Gabriel de Perthuis
A user of a workstation has a home directory /home/john as a subvolume. I wrote a cron job to make read-only snapshots of it under /home/john/backup which was fortunate as they just ran a script that did something like rm -rf ~. Apart from copying dozens of gigs of data back, is there a

Re: [PATCH 3/4] btrfs: Introduce extent_read_full_page_nolock()

2013-05-09 Thread Gabriel de Perthuis
We want this for btrfs_extent_same. Basically readpage and friends do their own extent locking but for the purposes of dedupe, we want to have both files locked down across a set of readpage operations (so that we can compare data). Introduce this variant and a flag which can be set for

Re: [PATCH 3/4] btrfs: Introduce extent_read_full_page_nolock()

2013-05-07 Thread Gabriel de Perthuis
We want this for btrfs_extent_same. Basically readpage and friends do their own extent locking but for the purposes of dedupe, we want to have both files locked down across a set of readpage operations (so that we can compare data). Introduce this variant and a flag which can be set for

Re: [RFC 0/5] BTRFS hot relocation support

2013-05-07 Thread Gabriel de Perthuis
How will it compare to bcache? I'm currently thinking about buying an SSD but bcache requires some efforts in migrating the storage to use. And after all those hassles I am even not sure if it would work easily with a dracut generated initramfs. On the side note: dm-cache, which is

Re: Possible to dedpulicate read-only snapshots for space-efficient backups

2013-05-07 Thread Gabriel de Perthuis
Do you plan to support deduplication on a finer grained basis than file level? As an example, in the end it could be interesting to deduplicate 1M blocks of huge files. Backups of VM images come to my mind as a good candidate. While my current backup script[1] takes care of this by using

Re: [RFC 0/5] BTRFS hot relocation support

2013-05-07 Thread Gabriel de Perthuis
On Tue, 07 May 2013 23:58:08 +0200, Kai Krakow wrote: Gabriel de Perthuis g2p.c...@gmail.com schrieb: On the side note: dm-cache, which is already in-kernel, do not need to reformat backing storage. On the other hand dm-cache is somewhat complex to assemble, and letting the system

Re: Possible to deduplicate read-only snapshots for space-efficient backups

2013-05-07 Thread Gabriel de Perthuis
On Wed, 08 May 2013 01:04:38 +0200, Kai Krakow wrote: Gabriel de Perthuis g2p.c...@gmail.com schrieb: It sounds simple, and was sort-of prompted by the new syscall taking short ranges, but it is tricky figuring out a sane heuristic (when to hash, when to bail, when to submit without comparing

[PATCH] btrfs: don't stop searching after encountering the wrong item

2013-05-06 Thread Gabriel de Perthuis
The search ioctl skips items that are too large for a result buffer, but inline items of a certain size occuring before any search result is found would trigger an overflow and stop the search entirely. Bug: https://bugzilla.kernel.org/show_bug.cgi?id=57641 Signed-off-by: Gabriel de Perthuis

[PATCH] btrfs: don't stop searching after encountering the wrong item

2013-05-06 Thread Gabriel de Perthuis
-by: Gabriel de Perthuis g2p.code+bt...@gmail.com --- (resent, with the correct header to have stable copied) fs/btrfs/ioctl.c | 10 +- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/fs/btrfs/ioctl.c b/fs/btrfs/ioctl.c index 2c02310..f49b62f 100644 --- a/fs/btrfs/ioctl.c

Re: Possible to dedpulicate read-only snapshots for space-efficient backups

2013-05-05 Thread Gabriel de Perthuis
On Sun, 05 May 2013 12:07:17 +0200, Kai Krakow wrote: Hey list, I wonder if it is possible to deduplicate read-only snapshots. Background: I'm using an bash/rsync script[1] to backup my whole system on a nightly basis to an attached USB3 drive into a scratch area, then take a snapshot

Re: Best Practice - Partition, or not?

2013-05-01 Thread Gabriel de Perthuis
Hello If I want to manage a complete disk with btrfs, what's the Best Practice? Would it be best to create the btrfs filesystem on /dev/sdb, or would it be better to create just one partition from start to end and then do mkfs.btrfs /dev/sdb1? Partitions (GPT) are always more flexible and

Re: [PATCH v3 2/2] Btrfs: online data deduplication

2013-05-01 Thread Gabriel de Perthuis
#define BTRFS_IOC_DEV_REPLACE _IOWR(BTRFS_IOCTL_MAGIC, 53, \ struct btrfs_ioctl_dev_replace_args) +#define BTRFS_IOC_DEDUP_REGISTER _IO(BTRFS_IOCTL_MAGIC, 54) This number has already been used by the offline dedup patches. -- To unsubscribe from this

Re: [PATCH 0/4] [RFC] btrfs: offline dedupe

2013-04-22 Thread Gabriel de Perthuis
On Sat, Apr 20, 2013 at 05:49:25PM +0200, Gabriel de Perthuis wrote: Hi, The following series of patches implements in btrfs an ioctl to do offline deduplication of file extents. I am a fan of this patch, the API is just right. I just have a few tweaks to suggest to the argument checking

Re: [PATCH 0/4] [RFC] btrfs: offline dedupe

2013-04-20 Thread Gabriel de Perthuis
Hi, The following series of patches implements in btrfs an ioctl to do offline deduplication of file extents. I am a fan of this patch, the API is just right. I just have a few tweaks to suggest to the argument checking. At first the 1M limitation on the length was a bit inconvenient, but

Re: Problem with building instructions for btrfs-tools in https://btrfs.wiki.kernel.org/index.php/Btrfs_source_repositories

2013-03-20 Thread Gabriel de Perthuis
There is a missing dependency: liblzo2-dev I suggest to make amendment to the wiki and add a liblzo2-dev to the apt-get line for Ubuntu/Debian. Added. Other distros may need some additions too. Anyone can edit the wiki, as the spambots will attest; a ConfirmEdit captcha at signup would be

Permanent uncancellable balance

2013-03-02 Thread Gabriel de Perthuis
Hello, I have a filesystem that has become unusable because of a balance I can't stop. It is very close to full, and the balance is preventing me from growing it. It was started like this: sudo btrfs filesystem balance start -v -musage=60 -dusage=60 /srv/backups It has been stuck at 0% across

Re: Permanent uncancellable balance

2013-03-02 Thread Gabriel de Perthuis
On Sat, 02 Mar 2013 17:12:37 +0600, Roman Mamedov wrote: Mount with the skip_balance option https://btrfs.wiki.kernel.org/index.php/Mount_options then you can issue btrfs fi balance cancel and it will succeed. Excellent, thank you. I had just thought of doing the same thing with ro and it

[PATCH] Fix a sign bug causing invalid memory access in the ino_paths ioctl.

2012-10-10 Thread Gabriel de Perthuis
To see the problem, create many hardlinks to the same file (120 should do it), then look up paths by inode with: ls -i btrfs inspect inode-resolve -v $ino /mnt/btrfs I noticed the memory layout of the fspath-val data had some irregularities (some unnecessary gaps that stop appearing about

[PATCH] warn when skipping snapshots created with older

2012-09-05 Thread Gabriel de Perthuis
. Also skip invalid high OIDs, to prevent spurious warnings. Signed-off-by: Gabriel de Perthuis g2p.code+bt...@gmail.com --- send-utils.c | 7 ++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/send-utils.c b/send-utils.c index a43d47e..03ca72a 100644 --- a/send-utils.c +++ b/send

[PATCH] Warn when skipping snapshots created with older kernels.

2012-09-04 Thread Gabriel de Perthuis
This message is more explicit than ERROR: could not resolve root_id, the message that will be shown immediately before `btrfs send` bails. Also skip invalid high OIDs. Signed-off-by: Gabriel de Perthuis g2p.code+bt...@gmail.com --- send-utils.c | 6 ++ 1 file changed, 6 insertions(+) diff