[PATCH] Btrfs: implement support for fallocate collapse range

2014-06-23 Thread Filipe David Borba Manana
This implements fallocate's FALLOC_FL_COLLAPSE_RANGE operation for BTRFS. This fallocate operation was introduced in the linux kernel version 3.15. Existing tests in xfstests already test this operation explicitly and implicitly via fsstress. Signed-off-by: Filipe David Borba Manana

[PATCH] generic/017: skip invalid block sizes for btrfs

2014-06-23 Thread Filipe David Borba Manana
In btrfs the block size (called sector size in btrfs) can not be smaller then the page size. Therefore skip block sizes smaller then page size if the fs is btrfs, so that the test can succeed on btrfs (testing only with block sizes of 4kb on systems with a page size of 4Kb). Signed-off-by: Filipe

Re: [PATCH] Btrfs: Refactor btrfs_lock_cluster() to kill compiler warning

2014-06-23 Thread Miao Xie
On Sun, 22 Jun 2014 14:30:09 +0200, Geert Uytterhoeven wrote: fs/btrfs/extent-tree.c: In function ‘btrfs_lock_cluster’: fs/btrfs/extent-tree.c:6399: warning: ‘used_bg’ may be used uninitialized in this function - Replace again: ... goto again; by standard C while (1) { ... }, - Move

Re: [PATCH] generic/017: skip invalid block sizes for btrfs

2014-06-23 Thread Satoru Takeuchi
Hi Filipe, (2014/06/23 19:28), Filipe David Borba Manana wrote: In btrfs the block size (called sector size in btrfs) can not be smaller then the page size. Therefore skip block sizes smaller then page size if the fs is btrfs, so that the test can succeed on btrfs (testing only with block

Re: [PATCH] generic/017: skip invalid block sizes for btrfs

2014-06-23 Thread Filipe David Manana
On Mon, Jun 23, 2014 at 11:48 AM, Satoru Takeuchi takeuchi_sat...@jp.fujitsu.com wrote: Hi Filipe, (2014/06/23 19:28), Filipe David Borba Manana wrote: In btrfs the block size (called sector size in btrfs) can not be smaller then the page size. Therefore skip block sizes smaller then page

[PATCH 2/6 v5] Btrfs: send, implement total data size command to allow for progress estimation

2014-06-23 Thread Filipe David Borba Manana
This new send flag makes send calculate first the amount of new file data (in bytes) the send root has relatively to the parent root, or for the case of a non-incremental send, the total amount of file data the stream will create (including holes and prealloc extents). In other words, it

[PATCH 3/6 v5] Btrfs: send, use fallocate command to punch holes

2014-06-23 Thread Filipe David Borba Manana
Instead of sending a write command with a data buffer filled with 0 value bytes, use the fallocate command, introduced in the send stream version 2, to tell the receiver to punch a file hole using the fallocate system call. Signed-off-by: Filipe David Borba Manana fdman...@gmail.com --- V2: A v2

[PATCH 4/6 v5] Btrfs: send, use fallocate command to allocate extents

2014-06-23 Thread Filipe David Borba Manana
The send stream version 2 adds the fallocate command, which can be used to allocate extents for a file or punch holes in a file. Previously we were ignoring file prealloc extents or treating them as extents filled with 0 bytes and sending a regular write command to the stream. After this change,

[PATCH 1/6 v5] Btrfs: send, bump stream version

2014-06-23 Thread Filipe David Borba Manana
This increases the send stream version from version 1 to version 2, adding new commands: 1) total data size - used to tell the receiver how much file data the stream will add or update; 2) fallocate - used to pre-allocate space for files and to punch holes in files; 3) inode set flags; 4)

[PATCH 5/6 v5] Btrfs: add missing cleanup on sysfs init failure

2014-06-23 Thread Filipe David Borba Manana
If we failed during initialization of sysfs, we weren't unregistering the top level btrfs sysfs entry nor the debugfs stuff. Not unregistering the top level sysfs entry makes future attempts to reload the btrfs module impossible and the following is reported in dmesg: [ 2246.451296] WARNING: CPU:

[PATCH 6/6 v5] Btrfs: add send_stream_version attribute to sysfs

2014-06-23 Thread Filipe David Borba Manana
So that applications can find out what's the highest send stream version supported/implemented by the running kernel: $ cat /sys/fs/btrfs/send/stream_version 2 Signed-off-by: Filipe David Borba Manana fdman...@gmail.com Reviewed-by: David Sterba dste...@suse.cz --- V1..V4: There's no

Re: [PATCH] generic/017: skip invalid block sizes for btrfs

2014-06-23 Thread Satoru Takeuchi
Hi Filipe, (2014/06/23 19:55), Filipe David Manana wrote: On Mon, Jun 23, 2014 at 11:48 AM, Satoru Takeuchi takeuchi_sat...@jp.fujitsu.com wrote: Hi Filipe, (2014/06/23 19:28), Filipe David Borba Manana wrote: In btrfs the block size (called sector size in btrfs) can not be smaller then the

[PATCH] Btrfs: fix crash when starting transaction

2014-06-23 Thread Filipe David Borba Manana
Often when starting a transaction we commit the currently running transaction, which can end up writing block group caches when the current process has its journal_info set to NULL (and not to a transaction). This makes our assertion at btrfs_check_data_free_space() (current_journal != NULL) fail,

Re: btrfs on whole disk (no partitions)

2014-06-23 Thread Martin K. Petersen
Chris == Chris Murphy li...@colorremedies.com writes: Chris Does anyone know if blktrace will intercept the actual SCSI Chris commands sent to the drive? Or is there a better utility to use Chris for this? When I use it unfiltered, I'm not seeing SCSI write Chris commands at all. # echo

Re: [PATCH] generic/017: skip invalid block sizes for btrfs

2014-06-23 Thread Brendan Hide
Not subscribed to fstests so not sure if this will reach that mailing list... I feel Takeuchi's instincts are right, even if the analysis *may* be wrong. As is it looks like there should be a btrfs) selector inside the case. On 23/06/14 12:48, Satoru Takeuchi wrote: Hi Filipe, (2014/06/23

Re: btrfs on whole disk (no partitions)

2014-06-23 Thread Martin K. Petersen
Duncan == Duncan 1i5t5.dun...@cox.net writes: Duncan Tho as you point out elsewhere, levels under the filesystem Duncan layer may split the btrfs 4096 byte block size into 512 byte Duncan logical sector sizes if appropriate, but that has nothing to do Duncan with btrfs except that it operates

Re: [PATCH] generic/017: skip invalid block sizes for btrfs

2014-06-23 Thread Lukáš Czerner
On Mon, 23 Jun 2014, Filipe David Borba Manana wrote: Date: Mon, 23 Jun 2014 11:28:00 +0100 From: Filipe David Borba Manana fdman...@gmail.com To: fste...@vger.kernel.org Cc: linux-btrfs@vger.kernel.org, Filipe David Borba Manana fdman...@gmail.com Subject: [PATCH] generic/017: skip

Re: [PATCH] Properly size the leafsize field in the mdrestore_struct struct.

2014-06-23 Thread David Sterba
On Wed, Jun 18, 2014 at 03:01:32PM +0900, Satoru Takeuchi wrote: (2014/06/13 7:57), Adam Buchbinder wrote: It's 32 bits as defined in ctree.h, but the struct had it as 64 bits. Found using MemorySanitizer. Signed-off-by: Adam Buchbinder abuchbin...@google.com It looks good to me.

Re: [PATCH] btrfs-progs: restore: check lzo compress length

2014-06-23 Thread David Sterba
On Wed, Jun 18, 2014 at 06:51:19PM +0200, Vincent Stehlé wrote: This patch actually allowed me to finish a btrfs restore of a damaged filesystem, which was repeateadly crashing otherwise. This was with v3.12, but I think it still makes sense. It does, thanks. -- To unsubscribe from this list:

Re: [PATCH] generic/017: skip invalid block sizes for btrfs

2014-06-23 Thread Lukáš Czerner
On Mon, 23 Jun 2014, Lukáš Czerner wrote: Date: Mon, 23 Jun 2014 14:35:50 +0200 (CEST) From: Lukáš Czerner lczer...@redhat.com To: Filipe David Borba Manana fdman...@gmail.com Cc: fste...@vger.kernel.org, linux-btrfs@vger.kernel.org Subject: Re: [PATCH] generic/017: skip invalid block sizes

Re: [PATCH 2/4] btrfs-progs: deal with invalid option combinations for btrfs-image

2014-06-23 Thread David Sterba
On Thu, Jun 19, 2014 at 09:46:01AM +0800, Gui Hecheng wrote: For btrfs-image, dumpmay not come with option '-o' -r may not come with option '-c', '-s', '-w', dev_cnt != 1 -m may not come with dev_cnt 2 All of the above should be regarded as invalid

Re: [PATCH 3/4] btrfs-progs: delete invalid output file when btrfs-image failed

2014-06-23 Thread David Sterba
On Thu, Jun 19, 2014 at 09:46:02AM +0800, Gui Hecheng wrote: When btrfs-image failed to create an image, the invalid output file had better be deleted to prevent being used mistakenly in the future. Signed-off-by: Gui Hecheng guihc.f...@cn.fujitsu.com --- btrfs-image.c | 10 +- 1

Re: [PATCH] btrfs-progs: Add minimum device size check.

2014-06-23 Thread David Sterba
On Thu, Jun 19, 2014 at 11:25:38AM +0800, Qu Wenruo wrote: Btrfs has global block reservation, so even mkfs.btrfs can execute without problem, there is still a possibility that the filesystem can't be mounted. For example when mkfs.btrfs on a 8M file on x86_64 platform, kernel will refuse to

Re: btrfs: qgroup: account shared subtrees during snapshot delete

2014-06-23 Thread David Sterba
On Fri, Jun 20, 2014 at 08:29:58AM -0700, Mark Fasheh wrote: +static int account_shared_subtree(struct btrfs_trans_handle *trans, + struct btrfs_root *root, + struct extent_buffer *root_eb, + u64

Re: [PATCH v2] btrfs-progs: Add minimum device size check.

2014-06-23 Thread David Sterba
On Fri, Jun 20, 2014 at 11:13:41AM +0800, Qu Wenruo wrote: Btrfs has global block reservation, so even mkfs.btrfs can execute without problem, there is still a possibility that the filesystem can't be mounted. For example when mkfs.btrfs on a 8M file on x86_64 platform, kernel will refuse to

Re: [PATCH v2] btrfs-progs: Add uninstall targets to Makefiles.

2014-06-23 Thread David Sterba
On Mon, Jun 23, 2014 at 04:23:48AM +0200, Nils Steinger wrote: + rmdir -p --ignore-fail-on-non-empty $(DESTDIR)$(man8dir) + rmdir -p --ignore-fail-on-non-empty $(DESTDIR)$(libdir) + rmdir -p --ignore-fail-on-non-empty $(DESTDIR)$(bindir) I don't think it's right to remove the

Re: [PATCH] generic/017: skip invalid block sizes for btrfs

2014-06-23 Thread Filipe David Manana
On Mon, Jun 23, 2014 at 3:09 PM, Lukáš Czerner lczer...@redhat.com wrote: On Mon, 23 Jun 2014, Lukáš Czerner wrote: Date: Mon, 23 Jun 2014 14:35:50 +0200 (CEST) From: Lukáš Czerner lczer...@redhat.com To: Filipe David Borba Manana fdman...@gmail.com Cc: fste...@vger.kernel.org,

Btrfs suddenly unmountable, open_ctree failed

2014-06-23 Thread Mike Hartman
Distro: Linux Mint 16 Kernel: 3.11.0-12-generic Btrfs (used to create): 0.20-rc1 (current version in Mint repo) Btrfs (used to troubleshoot): 3.14.2 Setup: sda6 is a luks container with btrfs inside. Btrfs has subvolumes @ (/) and @home (/home). I was doing nothing in particular on my laptop

Re: [PATCH] generic/017: skip invalid block sizes for btrfs

2014-06-23 Thread Zach Brown
On Mon, Jun 23, 2014 at 11:28:00AM +0100, Filipe David Borba Manana wrote: In btrfs the block size (called sector size in btrfs) can not be smaller then the page size. Nor larger. commit 8d082fb727ac11930ea20bf1612e334ea7c2b697 Author: Liu Bo liubo2...@cn.fujitsu.com Date: Tue Apr 3 09:56:53

Re: [PATCH] Properly size the leafsize field in the mdrestore_struct struct.

2014-06-23 Thread Satoru Takeuchi
(2014/06/23 22:44), David Sterba wrote: On Wed, Jun 18, 2014 at 03:01:32PM +0900, Satoru Takeuchi wrote: (2014/06/13 7:57), Adam Buchbinder wrote: It's 32 bits as defined in ctree.h, but the struct had it as 64 bits. Found using MemorySanitizer. Signed-off-by: Adam Buchbinder

[PATCH] handle start_unlink_transaction the same for an exceded quota , limit as an out of space error.

2014-06-23 Thread Kevin Brandstatter
--- fs/btrfs/inode.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c index 0ec8766..41209e8 100644 --- a/fs/btrfs/inode.c +++ b/fs/btrfs/inode.c @@ -3751,10 +3751,10 @@ static struct btrfs_trans_handle *__unlink_start_trans(struct inode

Re: Removing file = quota exceded

2014-06-23 Thread Kevin Brandstatter
On 06/22/2014 11:38 AM, Josef Bacik wrote: On 06/21/2014 06:16 PM, Kevin Brandstatter wrote: so ive come accross the issue of being unable to remove a file when a subvolume quota is reached. This can be resolved by truncating the file first, or removing the quota temporarily. However, it

Re: [PATCH] btrfs-progs: Add minimum device size check.

2014-06-23 Thread Qu Wenruo
Original Message Subject: Re: [PATCH] btrfs-progs: Add minimum device size check. From: David Sterba dste...@suse.cz To: Qu Wenruo quwen...@cn.fujitsu.com Date: 2014年06月23日 22:36 On Thu, Jun 19, 2014 at 11:25:38AM +0800, Qu Wenruo wrote: Btrfs has global block reservation, so

[PATCH v3] btrfs-progs: Add minimum device size check.

2014-06-23 Thread Qu Wenruo
Btrfs has global block reservation, so even mkfs.btrfs can execute without problem, there is still a possibility that the filesystem can't be mounted. For example when mkfs.btrfs on a 8M file on x86_64 platform, kernel will refuse to mount due to ENOSPC, since system block group takes 4M and mixed

Re: Btrfs suddenly unmountable, open_ctree failed

2014-06-23 Thread Chris Murphy
On Jun 23, 2014, at 4:18 PM, Mike Hartman m...@hartmanipulation.com wrote: Can anyone offer any suggestions? Is this data really unrecoverable? I have no idea what could have gone so severely wrong. btrfs check --repair /media/mint/usb_data/sda6_check.img btrfs check --repair --init-csum-tree

Re: [PATCH 3/4] btrfs-progs: delete invalid output file when btrfs-image failed

2014-06-23 Thread Gui Hecheng
On Mon, 2014-06-23 at 16:15 +0200, David Sterba wrote: On Thu, Jun 19, 2014 at 09:46:02AM +0800, Gui Hecheng wrote: When btrfs-image failed to create an image, the invalid output file had better be deleted to prevent being used mistakenly in the future. Signed-off-by: Gui Hecheng

Re: [PATCH 2/4] btrfs-progs: deal with invalid option combinations for btrfs-image

2014-06-23 Thread Gui Hecheng
On Mon, 2014-06-23 at 16:12 +0200, David Sterba wrote: On Thu, Jun 19, 2014 at 09:46:01AM +0800, Gui Hecheng wrote: For btrfs-image, dumpmay not come with option '-o' -r may not come with option '-c', '-s', '-w', dev_cnt != 1 -m may not come with dev_cnt 2 All

Re: Btrfs suddenly unmountable, open_ctree failed

2014-06-23 Thread Wang Shilong
On 06/24/2014 09:17 AM, Chris Murphy wrote: On Jun 23, 2014, at 4:18 PM, Mike Hartman m...@hartmanipulation.com wrote: Can anyone offer any suggestions? Is this data really unrecoverable? I have no idea what could have gone so severely wrong. btrfs check --repair

Re: [PATCH] Btrfs: fix crash when starting transaction

2014-06-23 Thread Satoru Takeuchi
Hi Filipe, (2014/06/23 20:58), Filipe David Borba Manana wrote: Often when starting a transaction we commit the currently running transaction, which can end up writing block group caches when the current process has its journal_info set to NULL (and not to a transaction). This makes our

[PATCH v2 2/4] btrfs-progs: deal with invalid option combinations for btrfs-image

2014-06-23 Thread Gui Hecheng
For btrfs-image, dumpmay not come with option '-o' -r may not come with option '-c', '-s', '-w', dev_cnt != 1 -m may not come with dev_cnt 2 All of the above should be regarded as invalid combinations, and the usage will show up. Signed-off-by: Gui Hecheng

[PATCH v2 3/4] btrfs-progs: delete invalid output file when btrfs-image failed

2014-06-23 Thread Gui Hecheng
When btrfs-image failed to create an image, the invalid output file had better be deleted to prevent being used mistakenly in the future. Signed-off-by: Gui Hecheng guihc.f...@cn.fujitsu.com --- changelog v1-v2: use a new local variable to avoid return value overwritten --- btrfs-image.c

Re: [PATCH v2 2/4] btrfs-progs: deal with invalid option combinations for btrfs-image

2014-06-23 Thread Gui Hecheng
On Tue, 2014-06-24 at 10:36 +0800, Gui Hecheng wrote: For btrfs-image, dumpmay not come with option '-o' -r may not come with option '-c', '-s', '-w', dev_cnt != 1 -m may not come with dev_cnt 2 All of the above should be regarded as invalid combinations, and

Re: Btrfs suddenly unmountable, open_ctree failed

2014-06-23 Thread Mike Hartman
I have a dd image, but not a btrfs-image. I ran the btrfs-image command, but it threw the same errors as everything else and generated a 0 byte file. I agree that it SOUNDS like some kind of media failure, but if so it seems odd to me that I was able to dd the entire partition with no read

Re: Btrfs suddenly unmountable, open_ctree failed

2014-06-23 Thread Mike Hartman
I have no particular desire to use it. I just tried everything else first and thought it was worth a shot. If you think that version would help, can you point me to the git repo? The one I grabbed was git://git.kernel.org/pub/scm/linux/kernel/git/mason/btrfs-progs.git. On Mon, Jun 23, 2014 at

Re: [PATCH v2 3/4] btrfs-progs: delete invalid output file when btrfs-image failed

2014-06-23 Thread Gui Hecheng
On Tue, 2014-06-24 at 10:38 +0800, Gui Hecheng wrote: When btrfs-image failed to create an image, the invalid output file had better be deleted to prevent being used mistakenly in the future. Signed-off-by: Gui Hecheng guihc.f...@cn.fujitsu.com --- changelog v1-v2: use a new local

Re: Btrfs suddenly unmountable, open_ctree failed

2014-06-23 Thread Wang Shilong
Hi Mike, On 06/24/2014 11:04 AM, Mike Hartman wrote: I have no particular desire to use it. I just tried everything else first and thought it was worth a shot. If you think that version would help, can you point me to the git repo? The one I grabbed was

[PATCH v3 2/4] btrfs-progs: deal with invalid option combinations for btrfs-image

2014-06-23 Thread Gui Hecheng
For btrfs-image, dumpmay not come with option '-o' -r may not come with option '-c', '-s', '-w', dev_cnt != 1 -m may not come with dev_cnt 2 All of the above should be regarded as invalid combinations, and the usage will show up. Signed-off-by: Gui Hecheng

[PATCH v3 3/4] btrfs-progs: delete invalid output file when btrfs-image failed

2014-06-23 Thread Gui Hecheng
When btrfs-image failed to create an image, the invalid output file had better be deleted to prevent being used mistakenly in the future. Signed-off-by: Gui Hecheng guihc.f...@cn.fujitsu.com --- changelog v1-v2: use a new local variable to avoid return value overwritten v2-v3: fix

Re: Btrfs suddenly unmountable, open_ctree failed

2014-06-23 Thread Chris Murphy
On Jun 23, 2014, at 8:58 PM, Mike Hartman m...@hartmanipulation.com wrote: I have a dd image, but not a btrfs-image. I ran the btrfs-image command, but it threw the same errors as everything else and generated a 0 byte file. I agree that it SOUNDS like some kind of media failure, but if so

Re: [PATCH] generic/017: skip invalid block sizes for btrfs

2014-06-23 Thread Dave Chinner
On Mon, Jun 23, 2014 at 04:09:18PM +0200, Lukáš Czerner wrote: On Mon, 23 Jun 2014, Lukáš Czerner wrote: Date: Mon, 23 Jun 2014 14:35:50 +0200 (CEST) From: Lukáš Czerner lczer...@redhat.com To: Filipe David Borba Manana fdman...@gmail.com Cc: fste...@vger.kernel.org,

Re: Btrfs suddenly unmountable, open_ctree failed

2014-06-23 Thread Mike Hartman
Of course it could just be a bug so it's worth trying David's integration branch. I'll try that shortly. * Firmware Version: 0006 Firmware 0007 is current for this SSD. I assume that's probably not something I should mess with right now though, right? 6 0x008 4

Re: [PATCH] generic/017: skip invalid block sizes for btrfs

2014-06-23 Thread Roman Mamedov
On Mon, 23 Jun 2014 11:28:00 +0100 Filipe David Borba Manana fdman...@gmail.com wrote: In btrfs the block size (called sector size in btrfs) can not be smaller then the page size. Just in case anyone misses this, there is some work to address this limitation:

Re: Btrfs suddenly unmountable, open_ctree failed

2014-06-23 Thread Chris Murphy
On Jun 23, 2014, at 10:34 PM, Mike Hartman m...@hartmanipulation.com wrote: Firmware 0007 is current for this SSD. I assume that's probably not something I should mess with right now though, right? I would deal with that later. A firmware change now might make things worse if you care

Re: [PATCH] generic/017: skip invalid block sizes for btrfs

2014-06-23 Thread Lukáš Czerner
On Tue, 24 Jun 2014, Dave Chinner wrote: Date: Tue, 24 Jun 2014 14:26:46 +1000 From: Dave Chinner da...@fromorbit.com To: Lukáš Czerner lczer...@redhat.com Cc: Filipe David Borba Manana fdman...@gmail.com, fste...@vger.kernel.org, linux-btrfs@vger.kernel.org Subject: Re: [PATCH]

Re: Removing file = quota exceded

2014-06-23 Thread Duncan
Duncan posted on Mon, 23 Jun 2014 01:53:45 + as excerpted: However, because btrfs stores very small files (generally something under 16 MiB, the precise size depends on filesystem parameters) entirely within metadata Hmm. Should be under 16 KiB I believe, not 16 MiB. -- Duncan - List

Re: btrfs on whole disk (no partitions)

2014-06-23 Thread Duncan
Martin K. Petersen posted on Mon, 23 Jun 2014 08:24:04 -0400 as excerpted: Anyway. The short answer is that Linux will pretty much always do I/O in multiples of the system page size regardless of the logical block size of the underlying device. There are a few exceptions to this such as

Re: Btrfs suddenly unmountable, open_ctree failed

2014-06-23 Thread Mike Hartman
My two cents:-) If you really want to use btrfs check --init-csum-tree --init-extent-tree, i'd suggest you use David Latest btrfs-progs branch which includes some latest bug fixes. I have no particular desire to use it. I just tried everything else first and thought it was worth a shot.

Re: Btrfs suddenly unmountable, open_ctree failed

2014-06-23 Thread Mike Hartman
I zeroed out the drive and ran every smartctl test on it I could find and it never threw any more errors. Zeroing SSDs isn't a good way to do it. Use ATA Secure Erase instead. The drive is overprovisioned, so there are pages without LBAs assigned, which means they can't be written to by