[PATCH] btrfs: Fix a deadlock in btrfs_dev_replace_finishing()
btrfs-transacion:5657 [stack snip] btrfs_bio_map() btrfs_bio_counter_inc_blocked() percpu_counter_inc(&fs_info->bio_counter) ###bio_counter > 0(A) __btrfs_bio_map() btrfs_dev_replace_lock() mutex_lock(dev_replace->lock) ###wait mutex(B) btrfs:32612 [stack snip] btrfs_dev_replace_start() btrfs_dev_replace_lock() mutex_lock(dev_replace->lock) ###hold mutex(B) btrfs_dev_replace_finishing() btrfs_rm_dev_replace_blocked() wait until percpu_counter_sum == 0 ###wait on bio_counter(A) This bug can be triggered quite easily by the following test script: http://pastebin.com/MQmb37Cy This patch will fix the ABBA problem by calling btrfs_dev_replace_unlock() before btrfs_rm_dev_replace_blocked(). The consistency of btrfs devices list and their superblocks is protected by device_list_mutex, not btrfs_dev_replace_lock/unlock(). So it is safe the move btrfs_dev_replace_unlock() before btrfs_rm_dev_replace_blocked(). Reported-by: Zhao Lei Signed-off-by: Qu Wenruo Cc: Stefan Behrens --- fs/btrfs/dev-replace.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/fs/btrfs/dev-replace.c b/fs/btrfs/dev-replace.c index eea26e1..d738ff8 100644 --- a/fs/btrfs/dev-replace.c +++ b/fs/btrfs/dev-replace.c @@ -567,6 +567,8 @@ static int btrfs_dev_replace_finishing(struct btrfs_fs_info *fs_info, btrfs_kobj_rm_device(fs_info, src_device); btrfs_kobj_add_device(fs_info, tgt_device); + btrfs_dev_replace_unlock(dev_replace); + btrfs_rm_dev_replace_blocked(fs_info); btrfs_rm_dev_replace_srcdev(fs_info, src_device); @@ -580,7 +582,6 @@ static int btrfs_dev_replace_finishing(struct btrfs_fs_info *fs_info, * superblock is scratched out so that it is no longer marked to * belong to this filesystem. */ - btrfs_dev_replace_unlock(dev_replace); mutex_unlock(&root->fs_info->fs_devices->device_list_mutex); mutex_unlock(&root->fs_info->chunk_mutex); -- 2.0.4 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] Btrfs: fix crash on endio of reading corrupted block
On Tue, Aug 19, 2014 at 04:42:42PM -0500, Eric Sandeen wrote: > On 8/19/14, 10:33 AM, Liu Bo wrote: > > The crash is > > > > [ cut here ] > > kernel BUG at fs/btrfs/extent_io.c:2124! > > [...] > > Workqueue: btrfs-endio normal_work_helper [btrfs] > > RIP: 0010:[] [] > > end_bio_extent_readpage+0xb45/0xcd0 [btrfs] > > > > This is in fact a regression. > > It'd be helpful to identify the commit, or at least kernel release, which > caused > the regression. Okay, got it. > > > It is because we forgot to increase @offset properly in reading corrupted > > block, > > so that the @offset remains, and this leads to checksum errors while reading > > left blocks queued up in the same bio, and then ends up with hiting the > > above > > BUG_ON. > > So does that mean that any checksum error on this path will crash the kernel? > > That sounds like this bug has exposed a more fundamental problem, no? Eric, you're right, I was hiding some details, now writing a new commit log... thanks, -liubo > > Thanks, > -Eric > > > Reported-by: Chris Murphy > > Signed-off-by: Liu Bo > > --- > > fs/btrfs/extent_io.c | 1 + > > 1 file changed, 1 insertion(+) > > > > diff --git a/fs/btrfs/extent_io.c b/fs/btrfs/extent_io.c > > index 3af4966..be41e4d 100644 > > --- a/fs/btrfs/extent_io.c > > +++ b/fs/btrfs/extent_io.c > > @@ -2602,6 +2602,7 @@ static void end_bio_extent_readpage(struct bio *bio, > > int err) > > test_bit(BIO_UPTODATE, &bio->bi_flags); > > if (err) > > uptodate = 0; > > + offset += len; > > continue; > > } > > } > > > -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] Btrfs: cleanup the same name in end_bio_extent_readpage
On Tue, Aug 19, 2014 at 03:31:00PM -0400, Chris Mason wrote: > On 08/19/2014 11:32 AM, Liu Bo wrote: > > We've defined a 'offset' out of bio_for_each_segment_all. > > This isn't causing problems though? It should just be shadowing the > bio_for_each_segment_all variable for the duration of the curlies. > > No objection as a cleanup, just making sure I'm not missing something. It did't cause any problems yet in my testers at least, so just a cleanup. thanks, -liubo -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH v5] Btrfs: send, lower mem requirements for processing xattrs
Maximum xattr size can be up to nearly the leaf size. For an fs with a leaf size larger than the page size, using kmalloc requires allocating multiple pages that are contiguous, which might not be possible if there's heavy memory fragmentation. Therefore fallback to vmalloc if we fail to allocate with kmalloc. Also start with a smaller buffer size, since xattr values typically are smaller than a page. Reported-by: Chris Murphy Signed-off-by: Filipe Manana --- V2: Use is_vmalloc_addr() instead of keeping a boolean variable around. V3: Use krealloc instead of kfree + kmalloc. V4: Fixed a checkpatch warning about missing blank line after var declaration. V5: Use kvfree() and pass __GFP_NOWARN to krealloc(). fs/btrfs/send.c | 40 1 file changed, 32 insertions(+), 8 deletions(-) diff --git a/fs/btrfs/send.c b/fs/btrfs/send.c index 3c63b29..3290da9 100644 --- a/fs/btrfs/send.c +++ b/fs/btrfs/send.c @@ -1006,11 +1006,13 @@ static int iterate_dir_item(struct btrfs_root *root, struct btrfs_path *path, int num; u8 type; - if (found_key->type == BTRFS_XATTR_ITEM_KEY) - buf_len = BTRFS_MAX_XATTR_SIZE(root); - else - buf_len = PATH_MAX; - + /* +* Start with a small buffer (1 page). If later we end up needing more +* space, which can happen for xattrs on a fs with a leaf size greater +* then the page size, attempt to increase the buffer. Typically xattr +* values are small. +*/ + buf_len = PATH_MAX; buf = kmalloc(buf_len, GFP_NOFS); if (!buf) { ret = -ENOMEM; @@ -1037,7 +1039,7 @@ static int iterate_dir_item(struct btrfs_root *root, struct btrfs_path *path, ret = -ENAMETOOLONG; goto out; } - if (name_len + data_len > buf_len) { + if (name_len + data_len > BTRFS_MAX_XATTR_SIZE(root)) { ret = -E2BIG; goto out; } @@ -1045,12 +1047,34 @@ static int iterate_dir_item(struct btrfs_root *root, struct btrfs_path *path, /* * Path too long */ - if (name_len + data_len > buf_len) { + if (name_len + data_len > PATH_MAX) { ret = -ENAMETOOLONG; goto out; } } + if (name_len + data_len > buf_len) { + buf_len = name_len + data_len; + if (is_vmalloc_addr(buf)) { + vfree(buf); + buf = NULL; + } else { + char *tmp = krealloc(buf, buf_len, +GFP_NOFS | __GFP_NOWARN); + + if (!tmp) + kfree(buf); + buf = tmp; + } + if (!buf) { + buf = vmalloc(buf_len); + if (!buf) { + ret = -ENOMEM; + goto out; + } + } + } + read_extent_buffer(eb, buf, (unsigned long)(di + 1), name_len + data_len); @@ -1071,7 +1095,7 @@ static int iterate_dir_item(struct btrfs_root *root, struct btrfs_path *path, } out: - kfree(buf); + kvfree(buf); return ret; } -- 1.9.1 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH v2] Btrfs: fix crash on endio of reading corrupted block
The crash is [ cut here ] kernel BUG at fs/btrfs/extent_io.c:2124! invalid opcode: [#1] SMP ... CPU: 3 PID: 88 Comm: kworker/u8:7 Not tainted 3.17.0-0.rc1.git0.1.fc22.x86_64 #1 Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006 Workqueue: btrfs-endio normal_work_helper [btrfs] task: 8800d7152700 ti: 8800d729c000 task.ti: 8800d729c000 RIP: 0010:[] [] end_bio_extent_readpage+0xb45/0xcd0 [btrfs] Call Trace: [] ? __enqueue_entity+0x78/0x80 [] ? enqueue_entity+0x2e9/0x990 [] bio_endio+0x6b/0xa0 [] bio_endio_nodec+0x12/0x20 [] end_workqueue_fn+0x37/0x40 [btrfs] [] normal_work_helper+0xbd/0x280 [btrfs] [] process_one_work+0x17e/0x430 [] worker_thread+0x6b/0x4a0 [] ? rescuer_thread+0x2a0/0x2a0 [] kthread+0xea/0x100 [] ? kthread_create_on_node+0x1a0/0x1a0 [] ret_from_fork+0x7c/0xb0 [] ? kthread_create_on_node+0x1a0/0x1a0 This is in fact a regression. It is because we forgot to increase @offset properly in reading corrupted block, so that the @offset remains unchanged, and it leads to checksum errors while reading left blocks queued up in the same bio, and then btrfs tries to iterate copies for those blocks in order to get good data, and hits the BUG_ON() which we set to avoid finding good copies for blocks without problems. Reported-by: Chris Murphy Signed-off-by: Liu Bo --- v2: - Improve the commit log to be clear, suggested by Eric. fs/btrfs/extent_io.c | 1 + 1 file changed, 1 insertion(+) diff --git a/fs/btrfs/extent_io.c b/fs/btrfs/extent_io.c index 3af4966..be41e4d 100644 --- a/fs/btrfs/extent_io.c +++ b/fs/btrfs/extent_io.c @@ -2602,6 +2602,7 @@ static void end_bio_extent_readpage(struct bio *bio, int err) test_bit(BIO_UPTODATE, &bio->bi_flags); if (err) uptodate = 0; + offset += len; continue; } } -- 1.8.1.4 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Questions on using BtrFS for fileserver
we are thinking about using BtrFS on standard hardware for a fileserver with about 50T (100T raw) of storage (25×4TByte). I would recommend carefully reading this thread titled: "1 week to rebuid 4x 3TB raid10 is a long time!" So I have a 2 x 2.6 TB devices in btrfs RAID-1, 716G used. Linux 3.16. One of the disks failed. "btrfs device delete missing /home" is taking 9 days so far, on an idle system: root 4828 0.3 0.0 17844 260 pts/1D+ Aug11 38:18 btrfs device delete missing /home There is some kind of btrfs debug info printed in dmesg which seems to tell me that the operation is working, like: [744657.598810] BTRFS info (device sda4): relocating block group 908951814144 flags 17 [744672.021612] BTRFS info (device sda4): found 4784 extents [744688.604997] BTRFS info (device sda4): found 4784 extents [744689.133397] BTRFS info (device sda4): relocating block group 91002968 flags 17 [744701.162678] BTRFS info (device sda4): found 4196 extents [744725.000459] BTRFS info (device sda4): found 4196 extents but other than that, the recovery time doesn't look optimistic to me, there is no ability to check the progress etc. -- Tomasz Chmielewski http://www.sslrack.com -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [btrfs] 8d875f95: xfstests.generic.226.fail
On Tue, 19 Aug 2014 10:58:09 -0400, Chris Mason wrote: > On 08/19/2014 10:23 AM, David Sterba wrote: >> On Tue, Aug 19, 2014 at 07:58:20PM +0800, Fengguang Wu wrote: >>> We noticed an xfstests failure on commit >>> >>> 8d875f95da43c6a8f18f77869f2ef26e9594fecc ("btrfs: disable strict file >>> flushes for renames and truncates") >>> >>> It's 100% reproducible in the 5 test runs. >> >> Same here, different mkfs configurations. >> >> generic/226 28s ...[16:11:52] [16:12:55] - output mismatch (see >> /root/xfstests/results//generic/226.out.bad) >> --- tests/generic/226.out 2013-05-29 17:16:03.0 +0200 >> +++ /root/xfstests/results//generic/226.out.bad 2014-08-19 >> 16:12:55.0 +0200 >> @@ -1,6 +1,8 @@ >> QA output created by 226 >> --> mkfs 256m filesystem >> --> 16 buffered 64m writes in a loop >> -1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 >> +1 2 3 4 pwrite64: No space left on device >> +5 6 7 8 9 10 11 12 pwrite64: No space left on device >> +13 14 15 16 >> >> enospc on a small filesystem (256M) > > I'm calling filemap flush more often, but otherwise everything else is > the same. I'll take a look. The above patch also introduced a performance regression(~70%DOWN). We can reproduce this regression by fio, here is the config: [global] ioengine=falloc iodepth=1 direct=0 buffered=0 directory= nrfiles=1 filesize=100m group_reporting [sequential aio-dio write] stonewall ioengine=posixaio numjobs=1 iodepth=128 buffered=0 direct=0 rw=write bs=64k filename=fragmented_file I found the problem is caused by the following function: int btrfs_release_file(struct inode *inode, struct file *filp) { ... filemap_flush(inode->i_mapping); return 0; } I don't think we need flush file at most situation. Ext4 flushes the file only after someone truncate the file to be zero-length, I don't know the real reason why ext4 flush the file only after the file is truncated, someone said it is to reduce the risk that the users find a zero-length file after a crash, which happens after truncate-write-close process. If we change btrfs_release_file by ext4's implementation, both the failure of xfstests's generic/226 and performance regression can be fixed. Thanks Miao > > -chris > -- > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ > -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 1/3] btrfs-progs: random fixes of btrfs-filesystem documentation
On Tue, Aug 19, 2014 at 10:33:44AM -0500, Eric Sandeen wrote: > Seems like using /proc/partitions would make more sense in that case > than a recursive scan of every file under /dev, wouldn't it? > Any details on those reports? It does make sense. > I'm just wondering when you might possibly have success looking deep > into the /dev tree if you didn't have success in /proc/partitions. I haven't figured out any advantage of /dev. > It just seems a bit bizarre to have so many ways to get the same info. I think keeping blkid as default and /proc/filesystems as fallback should cover the usecases. This meanas that the option -d stays, and the scanning method can be changed to BTRFS_SCAN_PROC (fi show and dev scan). -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Questions on using BtrFS for fileserver
As a counter-argument, I use BTRFS on a filesystem with about 280TB raw right now as a fileserver. Note that this is mostly transient data (raid 0), and I have stuck with 3.14, hearing the horror stories of 3.15/3.16 locking up. Even at that size (29TB LUNs), I have been able to add and remove devices and rebalance with no issues other than it causing increased IO and taking several weeks to move that much data around. Definitely avoid 3.15/3.16 and test out your workload first if possible to make sure it performs properly. Also build the filesystem and put data on it and test device removals/rebuilds to make sure it works with your OS, btrfs tools, and kernel version. All that being said it works great for us where we value COW and expansion/rebalancing over performance and redundancy. The filesystem is exported via NFS and rsync to over 200 clients over 10gb/sec ethernet, and hits around 5-7gb/sec balanced between reads and writes. One of our alternative file storage needs that I'd also hoped to move to BTRFS consisted of subtrees of 255 directories, each with 255 directories under them, and 255 directories under them with 1 file in each (don't ask). That *did not* work well under BTRFS -- probably due to the metadata juggling required in creating or removing any one file that far down in such a bizarre tree. We kept that particular area under XFS. -ben Tomasz Chmielewski wrote: we are thinking about using BtrFS on standard hardware for a fileserver with about 50T (100T raw) of storage (25×4TByte). I would recommend carefully reading this thread titled: "1 week to rebuid 4x 3TB raid10 is a long time!" So I have a 2 x 2.6 TB devices in btrfs RAID-1, 716G used. Linux 3.16. One of the disks failed. "btrfs device delete missing /home" is taking 9 days so far, on an idle system: root 4828 0.3 0.0 17844 260 pts/1 D+ Aug11 38:18 btrfs device delete missing /home There is some kind of btrfs debug info printed in dmesg which seems to tell me that the operation is working, like: [744657.598810] BTRFS info (device sda4): relocating block group 908951814144 flags 17 [744672.021612] BTRFS info (device sda4): found 4784 extents [744688.604997] BTRFS info (device sda4): found 4784 extents [744689.133397] BTRFS info (device sda4): relocating block group 91002968 flags 17 [744701.162678] BTRFS info (device sda4): found 4196 extents [744725.000459] BTRFS info (device sda4): found 4196 extents but other than that, the recovery time doesn't look optimistic to me, there is no ability to check the progress etc. -- - Benjamin O'Connor TechOps Systems Administrator TripAdvisor Media Group bocon...@tripadvisor.com c. 617-312-9072 - -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [btrfs] 8d875f95: xfstests.generic.226.fail
On 08/20/2014 06:52 AM, Miao Xie wrote: > On Tue, 19 Aug 2014 10:58:09 -0400, Chris Mason wrote: >> On 08/19/2014 10:23 AM, David Sterba wrote: >>> On Tue, Aug 19, 2014 at 07:58:20PM +0800, Fengguang Wu wrote: We noticed an xfstests failure on commit 8d875f95da43c6a8f18f77869f2ef26e9594fecc ("btrfs: disable strict file flushes for renames and truncates") It's 100% reproducible in the 5 test runs. >>> >>> Same here, different mkfs configurations. >>> >>> generic/226 28s ...[16:11:52] [16:12:55] - output mismatch (see >>> /root/xfstests/results//generic/226.out.bad) >>> --- tests/generic/226.out 2013-05-29 17:16:03.0 +0200 >>> +++ /root/xfstests/results//generic/226.out.bad 2014-08-19 >>> 16:12:55.0 +0200 >>> @@ -1,6 +1,8 @@ >>> QA output created by 226 >>> --> mkfs 256m filesystem >>> --> 16 buffered 64m writes in a loop >>> -1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 >>> +1 2 3 4 pwrite64: No space left on device >>> +5 6 7 8 9 10 11 12 pwrite64: No space left on device >>> +13 14 15 16 >>> >>> enospc on a small filesystem (256M) >> >> I'm calling filemap flush more often, but otherwise everything else is >> the same. I'll take a look. > > I found the problem is caused by the following function: > > int btrfs_release_file(struct inode *inode, struct file *filp) > { > ... > filemap_flush(inode->i_mapping); > return 0; > } > > I don't think we need flush file at most situation. Ext4 flushes the file only > after someone truncate the file to be zero-length, I don't know the real > reason > why ext4 flush the file only after the file is truncated, someone said it is > to > reduce the risk that the users find a zero-length file after a crash, which > happens > after truncate-write-close process. > > If we change btrfs_release_file by ext4's implementation, both the failure of > xfstests's generic/226 and performance regression can be fixed. > You're completely right, my original had more checks here and I stripped them out by accident. Fixing, thanks! -chris -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] Btrfs: show real function name in btrfs workqueue tracepoint
On Fri, Aug 15, 2014 at 11:38:06PM +0800, Liu Bo wrote: > Use %pf instead of %p, just same as kernel workqueue tracepoints. > > Signed-off-by: Liu Bo Reviewed-by: David Sterba -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] Btrfs: send, lower mem requirements for processing xattrs
On Sun, Aug 10, 2014 at 11:52:09PM +, Josef Bacik wrote: > Sigh I can only top post from my phone. Can you at least snip the original text? -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 8/8 v2] btrfs: rename total_bytes to avoid confusion
On Wed, Aug 20, 2014 at 10:54:17AM +0800, Anand Jain wrote: > we are assigning number_devices to the total_bytes, > that's very confusing for a moment > > Signed-off-by: Anand Jain Thanks. Reviewed-by: David Sterba -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH] Btrfs: fix filemap_flush call in btrfs_file_release
We should only be flushing on close if the file was flagged as needing it during truncate. I broke this with my ordered data vs transaction commit deadlock fix. Thanks to Miao Xie for catching this. Signed-off-by: Chris Mason Reported-by: Miao Xie Reported-by: Fengguang Wu diff --git a/fs/btrfs/file.c b/fs/btrfs/file.c index f15c13f..36861b7 100644 --- a/fs/btrfs/file.c +++ b/fs/btrfs/file.c @@ -1840,7 +1840,15 @@ int btrfs_release_file(struct inode *inode, struct file *filp) { if (filp->private_data) btrfs_ioctl_trans_end(filp); - filemap_flush(inode->i_mapping); + /* +* ordered_data_close is set by settattr when we are about to truncate +* a file from a non-zero size to a zero size. This tries to +* flush down new bytes that may have been written if the +* application were using truncate to replace a file in place. +*/ + if (test_and_clear_bit(BTRFS_INODE_ORDERED_DATA_CLOSE, + &BTRFS_I(inode)->runtime_flags)) + filemap_flush(inode->i_mapping); return 0; } -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH -rc2] btrfs: remove stale code after removing ordered operations
The commit "btrfs: disable strict file flushes for renames and truncates" (8d875f95da43c6a8f18f77869f2ef26e9594fecc) left some unused code and defines. Signed-off-by: David Sterba --- This is a cleanup after a 3.17-rc1 patch. fs/btrfs/btrfs_inode.h | 8 fs/btrfs/ctree.h | 7 --- fs/btrfs/inode.c | 10 -- 3 files changed, 25 deletions(-) diff --git a/fs/btrfs/btrfs_inode.h b/fs/btrfs/btrfs_inode.h index 43527fd78825..ee9d37d4f883 100644 --- a/fs/btrfs/btrfs_inode.h +++ b/fs/btrfs/btrfs_inode.h @@ -25,14 +25,6 @@ #include "ordered-data.h" #include "delayed-inode.h" -/* - * ordered_data_close is set by truncate when a file that used - * to have good data has been truncated to zero. When it is set - * the btrfs file release call will add this inode to the - * ordered operations list so that we make sure to flush out any - * new data the application may have written before commit. - */ -#define BTRFS_INODE_ORDERED_DATA_CLOSE 0 #define BTRFS_INODE_ORPHAN_META_RESERVED 1 #define BTRFS_INODE_DUMMY 2 #define BTRFS_INODE_IN_DEFRAG 3 diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h index 8e29b614fe93..69cd3262193e 100644 --- a/fs/btrfs/ctree.h +++ b/fs/btrfs/ctree.h @@ -62,13 +62,6 @@ struct btrfs_ordered_sum; #define BTRFS_COMPAT_EXTENT_TREE_V0 -/* - * files bigger than this get some pre-flushing when they are added - * to the ordered operations list. That way we limit the total - * work done by the commit - */ -#define BTRFS_ORDERED_OPERATIONS_FLUSH_LIMIT (8 * 1024 * 1024) - /* holds pointers to all of the tree roots */ #define BTRFS_ROOT_TREE_OBJECTID 1ULL diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c index 03708ef3deef..ded36a4e47a8 100644 --- a/fs/btrfs/inode.c +++ b/fs/btrfs/inode.c @@ -4545,16 +4545,6 @@ static int btrfs_setsize(struct inode *inode, struct iattr *attr) ret = btrfs_update_inode(trans, root, inode); btrfs_end_transaction(trans, root); } else { - - /* -* We're truncating a file that used to have good data down to -* zero. Make sure it gets into the ordered flush list so that -* any new writes get down to disk quickly. -*/ - if (newsize == 0) - set_bit(BTRFS_INODE_ORDERED_DATA_CLOSE, - &BTRFS_I(inode)->runtime_flags); - /* * 1 for the orphan item we're going to add * 1 for the orphan item deletion. -- 1.9.0 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH -rc2] btrfs: remove stale code after removing ordered operations
On 08/20/2014 11:03 AM, David Sterba wrote: > The commit "btrfs: disable strict file flushes for renames and > truncates" (8d875f95da43c6a8f18f77869f2ef26e9594fecc) left some unused > code and defines. See my followup for the perf regression, we still need these. I was never able to trigger the enospc problem with xfstests here, but I did double check that we're flushing or not properly now. -chris -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Questions on using BtrFS for fileserver
On 08/19/2014 05:38 PM, Andrej Manduch wrote: > Hi, > > On 08/19/2014 06:21 PM, M G Berberich wrote:> · Are there any > reports/papers/web-pages about BtrFS-systems this size >> in use? Praises, complains, performance-reviews, whatever… > > I don't know about papers or benchmarks but few weeks ago there was a > guy who has problem with really long mounting with btrfs with similiar size. > https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg36226.html > > And I would not recommend 3TB disks. *I'm not btrfs dev* but as far as I > know there is a quite different between rebuilding disk on real RAID and > btrfs RAID. The problem is btrfs has RAID on filesystem level not on hw > level so there is bigger mechanical overheat on drives and thus it take > significantli longer than regular RAID. It really suprises me that so many people come to this conclusion, but maybe they don't provide as much slack space as I do on my systems. In general you will only have a longer rebuild on BTRFS than on hardware RAID if the filesystem is more than about 50% full. On my desktop array (4x 1TB disks using BTRFS RAID10), I've replaced disks before and it took less than an hour for the operation. Of course that array is usually not more than 10% full. Interestingly, it took less time to rebuild this array the last time I lost a disk than it did back when it was 3x 1TB disks in a BTRFS RAID1, so things might improve overall with a larger number of disks in the array. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH] btrfs: remove stale define after removing ordered operations
Last user removed in commit "btrfs: disable strict file flushes for renames and truncates" (8d875f95da43c6a8f18f77869f2ef26e9594fecc). Signed-off-by: David Sterba --- fs/btrfs/ctree.h | 7 --- 1 file changed, 7 deletions(-) diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h index 8e29b614fe93..69cd3262193e 100644 --- a/fs/btrfs/ctree.h +++ b/fs/btrfs/ctree.h @@ -62,13 +62,6 @@ struct btrfs_ordered_sum; #define BTRFS_COMPAT_EXTENT_TREE_V0 -/* - * files bigger than this get some pre-flushing when they are added - * to the ordered operations list. That way we limit the total - * work done by the commit - */ -#define BTRFS_ORDERED_OPERATIONS_FLUSH_LIMIT (8 * 1024 * 1024) - /* holds pointers to all of the tree roots */ #define BTRFS_ROOT_TREE_OBJECTID 1ULL -- 1.9.0 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 00/15] xfstests: new btrfs stress test cases
This patchset add new stress test cases for btrfs by running two different btrfs operations simultaneously under fsstress to ensure btrfs doesn't hang or oops in such situations. btrfs scrub and btrfs check will be run after each test. The test matrix is the combination of 6 btrfs operations: balance create/mount/umount/delete subvolume replace device scrub defrag remount with different compress algorithms Short descriptions: 057: balance-subvolume 058: balance-scrub 059: balance-defrag 060: balance-remount 061: balance-replace 062: subvolume-replace 063: subvolume-scrub 064: subvolume-defrag 065: subvolume-remount 066: replace-scrub 067: replace-defrag 068: replace-remount 069: scrub-defrag 070: scrub-remount 071: defrag-remount Some issues I've seen: 1. subvolume cannot be mounted with selinux context, so you'll see such logs in dmesg SELinux: mount invalid. Same superblock, different security settings for (dev dm-8, type btrfs) I've reported the bug to btrfs list, see [BUG] cannot mount subvolume with selinux context 2. btrfs replace operation always returns ENOENT if balance is running So in 061.full you'll see ERROR: ioctl(DEV_REPLACE_START) failed on "/mnt/testarea/scratch": No such file or directory, no error Not sure if it's btrfs bug, at least I think the error code is misleading 3. replace operation hangs the kernel(3.16-rc4+) with fsstress running So case 062/066/067/068 will hang This is my first attemp to add these tests, any comments are welcomed! Thanks, Eryu Guan Eryu Guan (15): btrfs: new test to run btrfs balance and subvolume test simultaneously btrfs: new test to run btrfs balance and scrub simltaneously btrfs: new test to run btrfs balance and defrag operations simultaneously btrfs: new case to run btrfs balance and remount with different compress algorithms btrfs: new case to run btrfs balance and device replace simultaneously btrfs: new case to run btrfs subvolume create/delete operations and device replace simultaneously btrfs: new case to run btrfs subvolume create/delete operations and scrub simultaneously btrfs: new case to run btrfs subvolume create/delete and defrag operations simultaneously btrfs: new case to run subvolume create/delete and remount with different compress algorithms btrfs: new case to run device replace and scrub operations simultaneously btrfs: new case to run device replace and defrag operations simultaneously btrfs: new case to run device replace and remount with different compress algorithms simultaneously btrfs: new case to run btrfs scrub and defrag operations simultaneously btrfs: new case to run btrfs scrub and remount with different compress algorithms simultaneously btrfs: new case to run defrag and remount with different compress algorithms simultaneously common/rc | 24 tests/btrfs/057 | 147 tests/btrfs/057.out | 2 + tests/btrfs/058 | 141 +++ tests/btrfs/058.out | 2 + tests/btrfs/059 | 150 + tests/btrfs/059.out | 2 + tests/btrfs/060 | 142 +++ tests/btrfs/060.out | 2 + tests/btrfs/061 | 161 + tests/btrfs/061.out | 2 + tests/btrfs/062 | 167 ++ tests/btrfs/062.out | 2 + tests/btrfs/063 | 146 tests/btrfs/063.out | 2 + tests/btrfs/064 | 155 +++ tests/btrfs/064.out | 2 + tests/btrfs/065 | 148 + tests/btrfs/065.out | 2 + tests/btrfs/066 | 161 + tests/btrfs/066.out | 2 + tests/btrfs/067 | 171 tests/btrfs/067.out | 2 + tests/btrfs/068 | 163 + tests/btrfs/068.out | 2 + tests/btrfs/069 | 150 + tests/btrfs/069.out | 2 + tests/btrfs/070 | 142 +++ tests/btrfs/070.out | 2 + tests/btrfs/071 | 152 ++ tests/btrfs/071.out | 2 + tests/btrfs/group | 15 + 32 files changed, 2365 insertions(+) create mode 100755 tests/btrfs/057 create mode 100644 tests/btrfs/057.out create mode 100755 tests/btrfs/058 create mode 100644 tests/btrfs/058.out create mode 100755 tests/btrfs/059 create mode 100644 tests/btrfs/059.out create mode 100755 tests/btrfs/060 create mode 100644 tests/btrfs/060.out create mode 100755
[PATCH 09/15] btrfs: new case to run subvolume create/delete and remount with different compress algorithms
Run btrfs subvolume create/mount/umount/delete and remount with different compress algorithms simultaneously, with fsstress running in background. Signed-off-by: Eryu Guan --- tests/btrfs/065 | 148 tests/btrfs/065.out | 2 + tests/btrfs/group | 1 + 3 files changed, 151 insertions(+) create mode 100755 tests/btrfs/065 create mode 100644 tests/btrfs/065.out diff --git a/tests/btrfs/065 b/tests/btrfs/065 new file mode 100755 index 000..6737cb1 --- /dev/null +++ b/tests/btrfs/065 @@ -0,0 +1,148 @@ +#! /bin/bash +# FSQA Test No. btrfs/065 +# +# Run btrfs subvolume create/mount/umount/delete and remount with +# different compress algorithms simultaneously, with fsstress running +# in background. +# +#--- +# Copyright (C) 2014 Red Hat Inc. All rights reserved. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it would be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write the Free Software Foundation, +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA +# +#--- +# + +seq=`basename $0` +seqres=$RESULT_DIR/$seq +echo "QA output created by $seq" + +here=`pwd` +tmp=/tmp/$$ +status=1 +trap "_cleanup; exit \$status" 0 1 2 3 15 + +_cleanup() +{ + cd / + rm -fr $tmp.* +} + +# get standard environment, filters and checks +. ./common/rc +. ./common/filter + +# real QA test starts here +_supported_fs btrfs +_supported_os Linux +_require_scratch +_require_scratch_dev_pool 4 + +rm -f $seqres.full + +# test case array +tcs=( + "-m single -d single" + "-m dup -d single" + "-m raid0 -d raid0" + "-m raid1 -d raid0" + "-m raid1 -d raid1" + "-m raid10 -d raid10" + "-m raid5 -d raid5" + "-m raid6 -d raid6" +) + +run_test() +{ + local mkfs_opts=$1 + local saved_mkfs_opts=$MKFS_OPTIONS + local subvol_mnt=$tmp.mnt + + echo "Test $mkfs_opts with $with_compress" >>$seqres.full + + MKFS_OPTIONS="$MKFS_OPTIONS $mkfs_opts" + # dup only works on single device + if [[ "$mkfs_opts" =~ dup ]]; then + _scratch_mkfs >>$seqres.full 2>&1 + else + _scratch_pool_mkfs >>$seqres.full 2>&1 + fi + local ret=$? + MKFS_OPTIONS=$saved_mkfs_opts + # make sure we created btrfs with desired options + if [ $ret -ne 0 ]; then + echo "mkfs $mkfs_opts failed" + return + fi + _scratch_mount >>$seqres.full 2>&1 + + args=`_scale_fsstress_args -p 20 -n 100 $FSSTRESS_AVOID -d $SCRATCH_MNT/stressdir` + echo "Run fsstress $args" >>$seqres.full + $FSSTRESS_PROG $args >/dev/null 2>&1 & + fsstress_pid=$! + + echo -n "Start subvolume worker: " >>$seqres.full + mkdir -p $subvol_mnt + ( + while true; do + $BTRFS_UTIL_PROG subvolume create $SCRATCH_MNT/subvol_$$ + $MOUNT_PROG -o subvol=subvol_$$ $SCRATCH_DEV $subvol_mnt + $UMOUNT_PROG $subvol_mnt + $BTRFS_UTIL_PROG subvolume delete $SCRATCH_MNT/subvol_$$ + done + ) >/dev/null 2>&1 & + subvol_pid=$! + echo "$subvol_pid" >>$seqres.full + + echo -n "Start remount worker: " >>$seqres.full + ( + while true; do + for algo in no zlib lzo; do + $MOUNT_PROG -o remount,compress=$algo $SCRATCH_MNT >>$seqres.full 2>&1 + done + done + ) >/dev/null 2>&1 & + remount_pid=$! + echo "$remount_pid" >>$seqres.full + + echo "Wait for fsstress to exit and kill all background workers" >>$seqres.full + wait $fsstress_pid + + kill $subvol_pid $remount_pid + wait + # wait for the remount loop process to finish + while ps aux | grep "mount.*$SCRATCH_MNT" | grep -qv grep; do + sleep 1 + done + + echo "Scrub the filesystem" >>$seqres.full + $BTRFS_UTIL_PROG scrub start -B $SCRATCH_MNT >>$seqres.full 2>&1 + if [ $? -ne 0 ]; then + echo "Scrub find errors in \"$mkfs_opts\" test" | tee -a $seqres.full + fi + + $BTRFS_UTIL_PROG filesystem sync $SCRATCH_MNT >/dev/null 2>&1 + # in case the subvolume is still mounted + $UMOUNT_PROG $subvol_mnt >/dev/null 2>&1 + _scratch_unmount + _check_scr
[PATCH 08/15] btrfs: new case to run btrfs subvolume create/delete and defrag operations simultaneously
Run btrfs subvolume create/mount/umount/delete and btrfs defrag operation simultaneously, with fsstress running in background. Signed-off-by: Eryu Guan --- tests/btrfs/064 | 155 tests/btrfs/064.out | 2 + tests/btrfs/group | 1 + 3 files changed, 158 insertions(+) create mode 100755 tests/btrfs/064 create mode 100644 tests/btrfs/064.out diff --git a/tests/btrfs/064 b/tests/btrfs/064 new file mode 100755 index 000..b8009aa --- /dev/null +++ b/tests/btrfs/064 @@ -0,0 +1,155 @@ +#! /bin/bash +# FSQA Test No. btrfs/064 +# +# Run btrfs subvolume create/mount/umount/delete and btrfs defrag +# operation simultaneously, with fsstress running in background. +# +#--- +# Copyright (C) 2014 Red Hat Inc. All rights reserved. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it would be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write the Free Software Foundation, +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA +# +#--- +# + +seq=`basename $0` +seqres=$RESULT_DIR/$seq +echo "QA output created by $seq" + +here=`pwd` +tmp=/tmp/$$ +status=1 +trap "_cleanup; exit \$status" 0 1 2 3 15 + +_cleanup() +{ + cd / + rm -fr $tmp.* +} + +# get standard environment, filters and checks +. ./common/rc +. ./common/filter + +# real QA test starts here +_supported_fs btrfs +_supported_os Linux +_require_scratch +_require_scratch_dev_pool 4 + +rm -f $seqres.full + +# test case array +tcs=( + "-m single -d single" + "-m dup -d single" + "-m raid0 -d raid0" + "-m raid1 -d raid0" + "-m raid1 -d raid1" + "-m raid10 -d raid10" + "-m raid5 -d raid5" + "-m raid6 -d raid6" +) + +run_test() +{ + local mkfs_opts=$1 + local with_compress=$2 + local saved_mkfs_opts=$MKFS_OPTIONS + local subvol_mnt=$tmp.mnt + + echo "Test $mkfs_opts with $with_compress" >>$seqres.full + + MKFS_OPTIONS="$MKFS_OPTIONS $mkfs_opts" + # dup only works on single device + if [[ "$mkfs_opts" =~ dup ]]; then + _scratch_mkfs >>$seqres.full 2>&1 + else + _scratch_pool_mkfs >>$seqres.full 2>&1 + fi + local ret=$? + MKFS_OPTIONS=$saved_mkfs_opts + # make sure we created btrfs with desired options + if [ $ret -ne 0 ]; then + echo "mkfs $mkfs_opts failed" + return + fi + _scratch_mount >>$seqres.full 2>&1 + + args=`_scale_fsstress_args -p 20 -n 100 $FSSTRESS_AVOID -d $SCRATCH_MNT/stressdir` + echo "Run fsstress $args" >>$seqres.full + $FSSTRESS_PROG $args >/dev/null 2>&1 & + fsstress_pid=$! + + echo -n "Start subvolume worker: " >>$seqres.full + mkdir -p $subvol_mnt + ( + while true; do + $BTRFS_UTIL_PROG subvolume create $SCRATCH_MNT/subvol_$$ + $MOUNT_PROG -o subvol=subvol_$$ $SCRATCH_DEV $subvol_mnt + $UMOUNT_PROG $subvol_mnt + $BTRFS_UTIL_PROG subvolume delete $SCRATCH_MNT/subvol_$$ + done + ) >/dev/null 2>&1 & + subvol_pid=$! + echo "$subvol_pid" >>$seqres.full + + echo -n "Start defrag worker: " >>$seqres.full + ( + while true; do + if [ "$with_compress" == "nocompress" ]; then + find $SCRATCH_MNT \( -type f -o -type d \) -exec \ + $BTRFS_UTIL_PROG filesystem defrag {} \; >>$seqres.full 2>&1 + else + find $SCRATCH_MNT \( -type f -o -type d \) -exec \ + $BTRFS_UTIL_PROG filesystem defrag -clzo {} \; >>$seqres.full 2>&1 + find $SCRATCH_MNT \( -type f -o -type d \) -exec \ + $BTRFS_UTIL_PROG filesystem defrag -czlib {} \; >>$seqres.full 2>&1 + fi + done + ) & + defrag_pid=$! + echo "$defrag_pid" >>$seqres.full + + echo "Wait for fsstress to exit and kill all background workers" >>$seqres.full + wait $fsstress_pid + + kill $subvol_pid $defrag_pid + wait + # wait for btrfs defrag process to exit, otherwise it will block umount + while ps aux | grep "btrfs filesystem defrag" | grep -qv gre
[PATCH 01/15] btrfs: new test to run btrfs balance and subvolume test simultaneously
Run btrfs balance and subvolume create/mount/umount/delete simultaneously, with fsstress running in background. Signed-off-by: Eryu Guan --- tests/btrfs/057 | 147 tests/btrfs/057.out | 2 + tests/btrfs/group | 1 + 3 files changed, 150 insertions(+) create mode 100755 tests/btrfs/057 create mode 100644 tests/btrfs/057.out diff --git a/tests/btrfs/057 b/tests/btrfs/057 new file mode 100755 index 000..2f507a7 --- /dev/null +++ b/tests/btrfs/057 @@ -0,0 +1,147 @@ +#! /bin/bash +# FSQA Test No. btrfs/057 +# +# Run btrfs balance and subvolume create/mount/umount/delete simultaneously, +# with fsstress running in background. +# +#--- +# Copyright (C) 2014 Red Hat Inc. All rights reserved. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it would be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write the Free Software Foundation, +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA +# +#--- +# + +seq=`basename $0` +seqres=$RESULT_DIR/$seq +echo "QA output created by $seq" + +here=`pwd` +tmp=/tmp/$$ +status=1 +trap "_cleanup; exit \$status" 0 1 2 3 15 + +_cleanup() +{ + cd / + rm -fr $tmp.* +} + +# get standard environment, filters and checks +. ./common/rc +. ./common/filter + +# real QA test starts here +_supported_fs btrfs +_supported_os Linux +_require_scratch +_require_scratch_dev_pool 4 + +rm -f $seqres.full + +# test case array +tcs=( + "-m single -d single" + "-m dup -d single" + "-m raid0 -d raid0" + "-m raid1 -d raid0" + "-m raid1 -d raid1" + "-m raid10 -d raid10" + "-m raid5 -d raid5" + "-m raid6 -d raid6" +) + +run_test() +{ + local mkfs_opts=$1 + local saved_mkfs_opts=$MKFS_OPTIONS + local subvol_mnt=$tmp.mnt + + echo "Test $mkfs_opts" >>$seqres.full + + MKFS_OPTIONS="$MKFS_OPTIONS $mkfs_opts" + # dup only works on single device + if [[ "$mkfs_opts" =~ dup ]]; then + _scratch_mkfs >>$seqres.full 2>&1 + else + _scratch_pool_mkfs >>$seqres.full 2>&1 + fi + ret=$? + MKFS_OPTIONS=$saved_mkfs_opts + # make sure we created btrfs with desired options + if [ $ret -ne 0 ]; then + echo "mkfs $mkfs_opts failed" + return + fi + + _scratch_mount >>$seqres.full 2>&1 + + args=`_scale_fsstress_args -p 20 -n 100 $FSSTRESS_AVOID -d $SCRATCH_MNT/stressdir` + echo "Run fsstress $args" >>$seqres.full + $FSSTRESS_PROG $args >/dev/null 2>&1 & + fsstress_pid=$! + + echo -n "Start balance worker: " >>$seqres.full + ( + while true; do + $BTRFS_UTIL_PROG balance start $SCRATCH_MNT + done + ) >/dev/null 2>&1 & + balance_pid=$! + echo "$balance_pid" >>$seqres.full + + echo -n "Start subvolume worker: " >>$seqres.full + mkdir -p $subvol_mnt + ( + while true; do + $BTRFS_UTIL_PROG subvolume create $SCRATCH_MNT/subvol_$$ + $MOUNT_PROG -o subvol=subvol_$$ $SCRATCH_DEV $subvol_mnt + $UMOUNT_PROG $subvol_mnt + $BTRFS_UTIL_PROG subvolume delete $SCRATCH_MNT/subvol_$$ + done + ) >/dev/null 2>&1 & + subvol_pid=$! + echo "$subvol_pid" >>$seqres.full + + echo "Wait for fsstress to exit and kill all background workers" >>$seqres.full + wait $fsstress_pid + + kill $balance_pid $subvol_pid + wait + # the balance process might be still in D state and cannot be killed + # which could block umount, wait for it to finish + while ps aux | grep "balance start" | grep -qv grep; do + sleep 1 + done + + echo "Scrub the filesystem" >>$seqres.full + $BTRFS_UTIL_PROG scrub start -B $SCRATCH_MNT >>$seqres.full 2>&1 + if [ $? -ne 0 ]; then + echo "Scrub find errors in \"$mkfs_opts\" test" | tee -a $seqres.full + fi + + $BTRFS_UTIL_PROG filesystem sync $SCRATCH_MNT >/dev/null 2>&1 + # in case the subvolume is still mounted + $UMOUNT_PROG $subvol_mnt >/dev/null 2>&1 + _scratch_unmount + _check_scratch_fs +} + +echo "Silence is golden" +for t in "${tcs[@]}"; do + run_test "$t" +done + +status=0 +exit diff --git a/tests/btrfs/057
[PATCH 02/15] btrfs: new test to run btrfs balance and scrub simltaneously
Run btrfs balance and scrub operations simultaneously with fsstress running in background. Signed-off-by: Eryu Guan --- tests/btrfs/058 | 141 tests/btrfs/058.out | 2 + tests/btrfs/group | 1 + 3 files changed, 144 insertions(+) create mode 100755 tests/btrfs/058 create mode 100644 tests/btrfs/058.out diff --git a/tests/btrfs/058 b/tests/btrfs/058 new file mode 100755 index 000..999727d --- /dev/null +++ b/tests/btrfs/058 @@ -0,0 +1,141 @@ +#! /bin/bash +# FSQA Test No. btrfs/058 +# +# Run btrfs balance and scrub operations simultaneously with fsstress +# running in background. +# +#--- +# Copyright (C) 2014 Red Hat Inc. All rights reserved. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it would be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write the Free Software Foundation, +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA +# +#--- +# + +seq=`basename $0` +seqres=$RESULT_DIR/$seq +echo "QA output created by $seq" + +here=`pwd` +tmp=/tmp/$$ +status=1 +trap "_cleanup; exit \$status" 0 1 2 3 15 + +_cleanup() +{ + cd / + rm -fr $tmp.* +} + +# get standard environment, filters and checks +. ./common/rc +. ./common/filter + +# real QA test starts here +_supported_fs btrfs +_supported_os Linux +_require_scratch +_require_scratch_dev_pool 4 + +rm -f $seqres.full + +# test case array +tcs=( + "-m single -d single" + "-m dup -d single" + "-m raid0 -d raid0" + "-m raid1 -d raid0" + "-m raid1 -d raid1" + "-m raid10 -d raid10" + "-m raid5 -d raid5" + "-m raid6 -d raid6" +) + +run_test() +{ + local mkfs_opts=$1 + local saved_mkfs_opts=$MKFS_OPTIONS + + echo "Test $mkfs_opts" >>$seqres.full + + MKFS_OPTIONS="$MKFS_OPTIONS $mkfs_opts" + # dup only works on single device + if [[ "$mkfs_opts" =~ dup ]]; then + _scratch_mkfs >>$seqres.full 2>&1 + else + _scratch_pool_mkfs >>$seqres.full 2>&1 + fi + local ret=$? + MKFS_OPTIONS=$saved_mkfs_opts + # make sure we created btrfs with desired options + if [ $ret -ne 0 ]; then + echo "mkfs $mkfs_opts failed" + return + fi + _scratch_mount >>$seqres.full 2>&1 + + args=`_scale_fsstress_args -p 20 -n 100 $FSSTRESS_AVOID -d $SCRATCH_MNT/stressdir` + echo "Run fsstress $args" >>$seqres.full + $FSSTRESS_PROG $args >/dev/null 2>&1 & + fsstress_pid=$! + + echo -n "Start balance worker: " >>$seqres.full + ( + while true; do + $BTRFS_UTIL_PROG balance start $SCRATCH_MNT + done + ) >/dev/null 2>&1 & + balance_pid=$! + echo "$balance_pid" >>$seqres.full + + echo -n "Start scrub worker: " >>$seqres.full + ( + while true; do + $BTRFS_UTIL_PROG scrub start -B $SCRATCH_MNT >>$seqres.full 2>&1 + done + ) & + scrub_pid=$! + echo "$scrub_pid" >>$seqres.full + + echo "Wait for fsstress to exit and kill all background workers" >>$seqres.full + wait $fsstress_pid + kill $balance_pid $scrub_pid + wait + # the balance/scrub process might be still in D state and cannot be killed + # which could block umount, wait for it to finish + while ps aux | grep "balance start" | grep -qv grep; do + sleep 1 + done + while ps aux | grep "scrub start" | grep -qv grep; do + sleep 1 + done + + echo "Scrub the filesystem" >>$seqres.full + $BTRFS_UTIL_PROG scrub start -B $SCRATCH_MNT >>$seqres.full 2>&1 + if [ $? -ne 0 ]; then + echo "Scrub find errors in \"$mkfs_opts\" test" | tee -a $seqres.full + fi + + $BTRFS_UTIL_PROG filesystem sync $SCRATCH_MNT >/dev/null 2>&1 + _scratch_unmount + _check_scratch_fs +} + +echo "Silence is golden" +for t in "${tcs[@]}"; do + run_test "$t" +done + +status=0 +exit diff --git a/tests/btrfs/058.out b/tests/btrfs/058.out new file mode 100644 index 000..fb5ca60 --- /dev/null +++ b/tests/btrfs/058.out @@ -0,0 +1,2 @@ +QA output created by 058 +Silence is golden diff --git a/tests/btrfs/group b/tests/btrfs/group index 08fd54a..c6b1baa 100644 --- a/tests/btrfs/group +++ b/tests/btrfs/group @@ -60,3 +60,4 @@
[PATCH 06/15] btrfs: new case to run btrfs subvolume create/delete operations and device replace simultaneously
Run btrfs subvolume create/mount/umount/delete and device replace operation simultaneously, with fsstress running in background. Signed-off-by: Eryu Guan --- tests/btrfs/062 | 167 tests/btrfs/062.out | 2 + tests/btrfs/group | 1 + 3 files changed, 170 insertions(+) create mode 100755 tests/btrfs/062 create mode 100644 tests/btrfs/062.out diff --git a/tests/btrfs/062 b/tests/btrfs/062 new file mode 100755 index 000..d110af8 --- /dev/null +++ b/tests/btrfs/062 @@ -0,0 +1,167 @@ +#! /bin/bash +# FSQA Test No. btrfs/062 +# +# Run btrfs subvolume create/mount/umount/delete and device replace +# operation simultaneously, with fsstress running in background. +# +#--- +# Copyright (C) 2014 Red Hat Inc. All rights reserved. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it would be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write the Free Software Foundation, +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA +# +#--- +# + +seq=`basename $0` +seqres=$RESULT_DIR/$seq +echo "QA output created by $seq" + +here=`pwd` +tmp=/tmp/$$ +status=1 +trap "_cleanup; exit \$status" 0 1 2 3 15 + +_cleanup() +{ + cd / + rm -fr $tmp.* +} + +# get standard environment, filters and checks +. ./common/rc +. ./common/filter + +# real QA test starts here +_supported_fs btrfs +_supported_os Linux +_require_scratch +_require_scratch_dev_pool 5 +_require_scratch_dev_pool_equal_size + +rm -f $seqres.full + +# test case array +tcs=( + "-m single -d single" + "-m raid0 -d raid0" + "-m raid1 -d raid0" + "-m raid1 -d raid1" + "-m raid10 -d raid10" +# "-m raid5 -d raid5" # raid5 does not support replace operation yet +# "-m raid6 -d raid6" # raid6 does not support replace operation yet +) + +run_test() +{ + local mkfs_opts=$1 + local saved_mkfs_opts=$MKFS_OPTIONS + local saved_scratch_dev_pool=$SCRATCH_DEV_POOL + local subvol_mnt=$tmp.mnt + + echo "Test $mkfs_opts" >>$seqres.full + + # remove the last device from the SCRATCH_DEV_POOL list so + # _scratch_pool_mkfs won't use all devices in pool + local last_dev="`echo $SCRATCH_DEV_POOL | $AWK_PROG '{print $NF}'`" + SCRATCH_DEV_POOL=`echo $SCRATCH_DEV_POOL | sed -e "s# *$last_dev *##"` + MKFS_OPTIONS="$MKFS_OPTIONS $mkfs_opts" + _scratch_pool_mkfs >>$seqres.full 2>&1 + # make sure we created btrfs with desired options + if [ $? -ne 0 ]; then + echo "mkfs $mkfs_opts failed" + MKFS_OPTIONS=$saved_mkfs_opts + SCRATCH_DEV_POOL=$saved_scratch_dev_pool + return + fi + + _scratch_mount >>$seqres.full 2>&1 + + args=`_scale_fsstress_args -p 20 -n 100 $FSSTRESS_AVOID -d $SCRATCH_MNT/stressdir` + echo "Run fsstress $args" >>$seqres.full + $FSSTRESS_PROG $args >/dev/null 2>&1 & + fsstress_pid=$! + + echo -n "Start subvolume worker: " >>$seqres.full + mkdir -p $subvol_mnt + ( + while true; do + $BTRFS_UTIL_PROG subvolume create $SCRATCH_MNT/subvol_$$ + $MOUNT_PROG -o subvol=subvol_$$ $SCRATCH_DEV $subvol_mnt + $UMOUNT_PROG $subvol_mnt + $BTRFS_UTIL_PROG subvolume delete $SCRATCH_MNT/subvol_$$ + done + ) >/dev/null 2>&1 & + subvol_pid=$! + echo "$subvol_pid" >>$seqres.full + + echo -n "Start replace worker: " >>$seqres.full + ( + # take the last device as free device + free_dev=$last_dev + # replace all devices in btrfs_devices in turn + # do not replace SCRATCH_DEV which will be used in _scratch_mount + # and _check_scratch_fs etc. + btrfs_devices=`echo $SCRATCH_DEV_POOL | sed -e "s# *$SCRATCH_DEV *##"` + # set first device in btrfs_devices as first src_dev + src_dev=`echo $btrfs_devices | $AWK_PROG '{print $1}'` + echo "btrfs_devices=$btrfs_devices" >>$seqres.full + echo "free_dev=$free_dev, src_dev=$src_dev" >>$seqres.full + while true; do + echo "Replacing $src_dev with $free_dev" >>$seqres.full + $BTRFS_UTIL_PROG replace start -fB $src_dev $free_dev $SCRATCH_MNT >>$seqres.
[PATCH 07/15] btrfs: new case to run btrfs subvolume create/delete operations and scrub simultaneously
Run btrfs subvolume create/mount/umount/delete and btrfs scrub operation simultaneously, with fsstress running in background. Signed-off-by: Eryu Guan --- tests/btrfs/063 | 146 tests/btrfs/063.out | 2 + tests/btrfs/group | 1 + 3 files changed, 149 insertions(+) create mode 100755 tests/btrfs/063 create mode 100644 tests/btrfs/063.out diff --git a/tests/btrfs/063 b/tests/btrfs/063 new file mode 100755 index 000..68e6a25 --- /dev/null +++ b/tests/btrfs/063 @@ -0,0 +1,146 @@ +#! /bin/bash +# FSQA Test No. btrfs/063 +# +# Run btrfs subvolume create/mount/umount/delete and btrfs scrub +# operation simultaneously, with fsstress running in background. +# +#--- +# Copyright (C) 2014 Red Hat Inc. All rights reserved. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it would be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write the Free Software Foundation, +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA +# +#--- +# + +seq=`basename $0` +seqres=$RESULT_DIR/$seq +echo "QA output created by $seq" + +here=`pwd` +tmp=/tmp/$$ +status=1 +trap "_cleanup; exit \$status" 0 1 2 3 15 + +_cleanup() +{ + cd / + rm -fr $tmp.* +} + +# get standard environment, filters and checks +. ./common/rc +. ./common/filter + +# real QA test starts here +_supported_fs btrfs +_supported_os Linux +_require_scratch +_require_scratch_dev_pool 4 + +rm -f $seqres.full + +# test case array +tcs=( + "-m single -d single" + "-m dup -d single" + "-m raid0 -d raid0" + "-m raid1 -d raid0" + "-m raid1 -d raid1" + "-m raid10 -d raid10" + "-m raid5 -d raid5" + "-m raid6 -d raid6" +) + +run_test() +{ + local mkfs_opts=$1 + local saved_mkfs_opts=$MKFS_OPTIONS + local subvol_mnt=$tmp.mnt + + echo "Test $mkfs_opts" >>$seqres.full + + MKFS_OPTIONS="$MKFS_OPTIONS $mkfs_opts" + # dup only works on single device + if [[ "$mkfs_opts" =~ dup ]]; then + _scratch_mkfs >>$seqres.full 2>&1 + else + _scratch_pool_mkfs >>$seqres.full 2>&1 + fi + local ret=$? + MKFS_OPTIONS=$saved_mkfs_opts + # make sure we created btrfs with desired options + if [ $? -ne 0 ]; then + echo "mkfs $mkfs_opts failed" + return + fi + _scratch_mount >>$seqres.full 2>&1 + + args=`_scale_fsstress_args -p 20 -n 100 $FSSTRESS_AVOID -d $SCRATCH_MNT/stressdir` + echo "Run fsstress $args" >>$seqres.full + $FSSTRESS_PROG $args >/dev/null 2>&1 & + fsstress_pid=$! + + echo -n "Start subvolume worker: " >>$seqres.full + mkdir -p $subvol_mnt + ( + while true; do + $BTRFS_UTIL_PROG subvolume create $SCRATCH_MNT/subvol_$$ + $MOUNT_PROG -o subvol=subvol_$$ $SCRATCH_DEV $subvol_mnt + $UMOUNT_PROG $subvol_mnt + $BTRFS_UTIL_PROG subvolume delete $SCRATCH_MNT/subvol_$$ + done + ) >/dev/null 2>&1 & + subvol_pid=$! + echo "$subvol_pid" >>$seqres.full + + echo -n "Start scrub worker: " >>$seqres.full + ( + while true; do + $BTRFS_UTIL_PROG scrub start -B $SCRATCH_MNT >>$seqres.full 2>&1 + done + ) & + scrub_pid=$! + echo "$scrub_pid" >>$seqres.full + + echo "Wait for fsstress to exit and kill all background workers" >>$seqres.full + wait $fsstress_pid + + kill $subvol_pid $scrub_pid + wait + # the scrub process might be still in D state and cannot be killed + # which could block umount, wait for it to finish + while ps aux | grep "scrub start" | grep -qv grep; do + sleep 1 + done + + echo "Scrub the filesystem" >>$seqres.full + $BTRFS_UTIL_PROG scrub start -B $SCRATCH_MNT >>$seqres.full 2>&1 + if [ $? -ne 0 ]; then + echo "Scrub find errors in \"$mkfs_opts\" test" | tee -a $seqres.full + fi + + $BTRFS_UTIL_PROG filesystem sync $SCRATCH_MNT >/dev/null 2>&1 + # in case the subvolume is still mounted + $UMOUNT_PROG $subvol_mnt >/dev/null 2>&1 + _scratch_unmount + _check_scratch_fs +} + +echo "Silence is golden" +for t in "${tcs[@]}"; do + run_test "$t" +done + +status=0 +exit diff
[PATCH 04/15] btrfs: new case to run btrfs balance and remount with different compress algorithms
Run btrfs balance and remount with different compress algorithms simultaneously, with fsstress running in background. Signed-off-by: Eryu Guan --- tests/btrfs/060 | 142 tests/btrfs/060.out | 2 + tests/btrfs/group | 1 + 3 files changed, 145 insertions(+) create mode 100755 tests/btrfs/060 create mode 100644 tests/btrfs/060.out diff --git a/tests/btrfs/060 b/tests/btrfs/060 new file mode 100755 index 000..8410a99 --- /dev/null +++ b/tests/btrfs/060 @@ -0,0 +1,142 @@ +#! /bin/bash +# FSQA Test No. btrfs/060 +# +# Run btrfs balance and remount with different compress algorithms +# simultaneously, with fsstress running in background. +# +#--- +# Copyright (C) 2014 Red Hat Inc. All rights reserved. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it would be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write the Free Software Foundation, +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA +# +#--- +# + +seq=`basename $0` +seqres=$RESULT_DIR/$seq +echo "QA output created by $seq" + +here=`pwd` +tmp=/tmp/$$ +status=1 +trap "_cleanup; exit \$status" 0 1 2 3 15 + +_cleanup() +{ + cd / + rm -fr $tmp.* +} + +# get standard environment, filters and checks +. ./common/rc +. ./common/filter + +# real QA test starts here +_supported_fs btrfs +_supported_os Linux +_require_scratch +_require_scratch_dev_pool 4 + +rm -f $seqres.full + +# test case array +tcs=( + "-m single -d single" + "-m dup -d single" + "-m raid0 -d raid0" + "-m raid1 -d raid0" + "-m raid1 -d raid1" + "-m raid10 -d raid10" + "-m raid5 -d raid5" + "-m raid6 -d raid6" +) + +run_test() +{ + local mkfs_opts=$1 + local saved_mkfs_opts=$MKFS_OPTIONS + + echo "Test $mkfs_opts" >>$seqres.full + + MKFS_OPTIONS="$MKFS_OPTIONS $mkfs_opts" + # dup only works on single device + if [[ "$mkfs_opts" =~ dup ]]; then + _scratch_mkfs >>$seqres.full 2>&1 + else + _scratch_pool_mkfs >>$seqres.full 2>&1 + fi + local ret=$? + MKFS_OPTIONS=$saved_mkfs_opts + # make sure we created btrfs with desired options + if [ $ret -ne 0 ]; then + echo "mkfs $mkfs_opts failed" + return + fi + _scratch_mount >>$seqres.full 2>&1 + + args=`_scale_fsstress_args -p 20 -n 100 $FSSTRESS_AVOID -d $SCRATCH_MNT/stressdir` + echo "Run fsstress $args" >>$seqres.full + $FSSTRESS_PROG $args >/dev/null 2>&1 & + fsstress_pid=$! + + echo -n "Start balance worker: " >>$seqres.full + ( + while true; do + $BTRFS_UTIL_PROG balance start $SCRATCH_MNT + done + ) >/dev/null 2>&1 & + balance_pid=$! + echo "$balance_pid" >>$seqres.full + + echo -n "Start remount worker: " >>$seqres.full + ( + while true; do + for algo in no zlib lzo; do + $MOUNT_PROG -o remount,compress=$algo $SCRATCH_MNT >>$seqres.full 2>&1 + done + done + ) >/dev/null 2>&1 & + remount_pid=$! + echo "$remount_pid" >>$seqres.full + + echo "Wait for fsstress to exit and kill all background workers" >>$seqres.full + wait $fsstress_pid + kill $balance_pid $remount_pid + wait + # wait for the balance and remount loop to finish + while ps aux | grep "balance start" | grep -qv grep; do + sleep 1 + done + while ps aux | grep "mount.*$SCRATCH_MNT" | grep -qv grep; do + sleep 1 + done + + echo "Scrub the filesystem" >>$seqres.full + $BTRFS_UTIL_PROG scrub start -B $SCRATCH_MNT >>$seqres.full 2>&1 + if [ $? -ne 0 ]; then + echo "Scrub find errors in \"$mkfs_opts\" test" | tee -a $seqres.full + fi + + $BTRFS_UTIL_PROG filesystem sync $SCRATCH_MNT >/dev/null 2>&1 + _scratch_unmount + _check_scratch_fs +} + +echo "Silence is golden" +for t in "${tcs[@]}"; do + run_test "$t" +done + +status=0 +exit diff --git a/tests/btrfs/060.out b/tests/btrfs/060.out new file mode 100644 index 000..8ffce4d --- /dev/null +++ b/tests/btrfs/060.out @@ -0,0 +1,2 @@ +QA output created by 060 +Silence is golden diff --git a/tests/btrfs/group b/tests/btrfs/group
[PATCH 03/15] btrfs: new test to run btrfs balance and defrag operations simultaneously
Run btrfs balance and defrag operations simultaneously with fsstress running in background. Signed-off-by: Eryu Guan --- tests/btrfs/059 | 150 tests/btrfs/059.out | 2 + tests/btrfs/group | 1 + 3 files changed, 153 insertions(+) create mode 100755 tests/btrfs/059 create mode 100644 tests/btrfs/059.out diff --git a/tests/btrfs/059 b/tests/btrfs/059 new file mode 100755 index 000..897a300 --- /dev/null +++ b/tests/btrfs/059 @@ -0,0 +1,150 @@ +#! /bin/bash +# FSQA Test No. btrfs/059 +# +# Run btrfs balance and defrag operations simultaneously with fsstress +# running in background. +# +#--- +# Copyright (C) 2014 Red Hat Inc. All rights reserved. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it would be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write the Free Software Foundation, +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA +# +#--- +# + +seq=`basename $0` +seqres=$RESULT_DIR/$seq +echo "QA output created by $seq" + +here=`pwd` +tmp=/tmp/$$ +status=1 +trap "_cleanup; exit \$status" 0 1 2 3 15 + +_cleanup() +{ + cd / + rm -fr $tmp.* +} + +# get standard environment, filters and checks +. ./common/rc +. ./common/filter + +# real QA test starts here +_supported_fs btrfs +_supported_os Linux +_require_scratch +_require_scratch_dev_pool 4 + +rm -f $seqres.full + +# test case array +tcs=( + "-m single -d single" + "-m dup -d single" + "-m raid0 -d raid0" + "-m raid1 -d raid0" + "-m raid1 -d raid1" + "-m raid10 -d raid10" + "-m raid5 -d raid5" + "-m raid6 -d raid6" +) + +run_test() +{ + local mkfs_opts=$1 + local with_compress=$2 + local saved_mkfs_opts=$MKFS_OPTIONS + + echo "Test $mkfs_opts with $with_compress" >>$seqres.full + + MKFS_OPTIONS="$MKFS_OPTIONS $mkfs_opts" + # dup only works on single device + if [[ "$mkfs_opts" =~ dup ]]; then + _scratch_mkfs >>$seqres.full 2>&1 + else + _scratch_pool_mkfs >>$seqres.full 2>&1 + fi + local ret=$? + MKFS_OPTIONS=$saved_mkfs_opts + # make sure we created btrfs with desired options + if [ $ret -ne 0 ]; then + echo "mkfs $mkfs_opts failed" + return + fi + _scratch_mount >>$seqres.full 2>&1 + + args=`_scale_fsstress_args -p 20 -n 100 $FSSTRESS_AVOID -d $SCRATCH_MNT/stressdir` + echo "Run fsstress $args" >>$seqres.full + $FSSTRESS_PROG $args >/dev/null 2>&1 & + fsstress_pid=$! + + echo -n "Start balance worker: " >>$seqres.full + ( + while true; do + $BTRFS_UTIL_PROG balance start $SCRATCH_MNT + done + ) >/dev/null 2>&1 & + balance_pid=$! + echo "$balance_pid" >>$seqres.full + + echo -n "Start defrag worker: " >>$seqres.full + ( + while true; do + if [ "$with_compress" == "nocompress" ]; then + find $SCRATCH_MNT \( -type f -o -type d \) -exec \ + $BTRFS_UTIL_PROG filesystem defrag {} \; >>$seqres.full 2>&1 + else + find $SCRATCH_MNT \( -type f -o -type d \) -exec \ + $BTRFS_UTIL_PROG filesystem defrag -clzo {} \; >>$seqres.full 2>&1 + find $SCRATCH_MNT \( -type f -o -type d \) -exec \ + $BTRFS_UTIL_PROG filesystem defrag -czlib {} \; >>$seqres.full 2>&1 + fi + done + ) & + defrag_pid=$! + echo "$defrag_pid" >>$seqres.full + + echo "Wait for fsstress to exit and kill all background workers" >>$seqres.full + wait $fsstress_pid + kill $balance_pid $defrag_pid + wait + # wait for the balance and defrag operations to finish + while ps aux | grep "balance start" | grep -qv grep; do + sleep 1 + done + while ps aux | grep "btrfs filesystem defrag" | grep -qv grep; do + sleep 1 + done + + echo "Scrub the filesystem" >>$seqres.full + $BTRFS_UTIL_PROG scrub start -B $SCRATCH_MNT >>$seqres.full 2>&1 + if [ $? -ne 0 ]; then + echo "Scrub find errors in \"$mkfs_opts\" test" | tee -a $
[PATCH 05/15] btrfs: new case to run btrfs balance and device replace simultaneously
Run btrfs balance and replace operations simultaneously with fsstress running in background. Signed-off-by: Eryu Guan --- common/rc | 24 tests/btrfs/061 | 161 tests/btrfs/061.out | 2 + tests/btrfs/group | 1 + 4 files changed, 188 insertions(+) create mode 100755 tests/btrfs/061 create mode 100644 tests/btrfs/061.out diff --git a/common/rc b/common/rc index 7ea1614..9212547 100644 --- a/common/rc +++ b/common/rc @@ -1985,6 +1985,24 @@ _require_scratch_dev_pool() done } +# ensure devices in SCRATCH_DEV_POOL are of the same size +# must be called after _require_scratch_dev_pool +_require_scratch_dev_pool_equal_size() +{ + local _size + local _newsize + local _dev + + # SCRATCH_DEV has been set to the first device in SCRATCH_DEV_POOL + _size=`_get_device_size $SCRATCH_DEV` + for _dev in $SCRATCH_DEV_POOL; do + _newsize=`_get_device_size $_dev` + if [ $_size -ne $_newsize ]; then + _notrun "This test requires devices in SCRATCH_DEV_POOL have the same size" + fi + done +} + # We will check if the device is deletable _require_deletable_scratch_dev_pool() { @@ -2231,6 +2249,12 @@ _require_btrfs_fs_feature() _notrun "Feature $feat not supported by the available btrfs version" } +# return device size in kb +_get_device_size() +{ + grep `_short_dev $1` /proc/partitions | awk '{print $3}' +} + init_rc() { if [ "$iam" == new ] diff --git a/tests/btrfs/061 b/tests/btrfs/061 new file mode 100755 index 000..28640cc --- /dev/null +++ b/tests/btrfs/061 @@ -0,0 +1,161 @@ +#! /bin/bash +# FSQA Test No. btrfs/061 +# +# Run btrfs balance and replace operations simultaneously with fsstress +# running in background. +# +#--- +# Copyright (C) 2014 Red Hat Inc. All rights reserved. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it would be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write the Free Software Foundation, +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA +# +#--- +# + +seq=`basename $0` +seqres=$RESULT_DIR/$seq +echo "QA output created by $seq" + +here=`pwd` +tmp=/tmp/$$ +status=1 +trap "_cleanup; exit \$status" 0 1 2 3 15 + +_cleanup() +{ + cd / + rm -fr $tmp.* +} + +# get standard environment, filters and checks +. ./common/rc +. ./common/filter + +# real QA test starts here +_supported_fs btrfs +_supported_os Linux +_require_scratch +_require_scratch_dev_pool 5 +_require_scratch_dev_pool_equal_size + +rm -f $seqres.full + +# test case array +tcs=( + "-m single -d single" + "-m raid0 -d raid0" + "-m raid1 -d raid0" + "-m raid1 -d raid1" + "-m raid10 -d raid10" +# "-m raid5 -d raid5" # raid5 does not support replace operation yet +# "-m raid6 -d raid6" # raid6 does not support replace operation yet +) + +run_test() +{ + local mkfs_opts=$1 + local saved_mkfs_opts=$MKFS_OPTIONS + local saved_scratch_dev_pool=$SCRATCH_DEV_POOL + + echo "Test $mkfs_opts" >>$seqres.full + + # remove the last device from the SCRATCH_DEV_POOL list so + # _scratch_pool_mkfs won't use all devices in pool + local last_dev="`echo $SCRATCH_DEV_POOL | $AWK_PROG '{print $NF}'`" + SCRATCH_DEV_POOL=`echo $SCRATCH_DEV_POOL | sed -e "s# *$last_dev *##"` + MKFS_OPTIONS="$MKFS_OPTIONS $mkfs_opts" + _scratch_pool_mkfs >>$seqres.full 2>&1 + # make sure we created btrfs with desired options + if [ $? -ne 0 ]; then + echo "mkfs $mkfs_opts failed" + MKFS_OPTIONS=$saved_mkfs_opts + SCRATCH_DEV_POOL=$saved_scratch_dev_pool + return + fi + _scratch_mount >>$seqres.full 2>&1 + + args=`_scale_fsstress_args -p 20 -n 100 $FSSTRESS_AVOID -d $SCRATCH_MNT/stressdir` + echo "Run fsstress $args" >>$seqres.full + $FSSTRESS_PROG $args >/dev/null 2>&1 & + fsstress_pid=$! + + echo -n "Start balance worker: " >>$seqres.full + ( + while true; do + $BTRFS_UTIL_PROG balance start $SCRATCH_MNT + done + ) >/dev/null 2>&1 & + balance_pid=$! + echo "$balance_pid" >>$seqres.full + + echo -n "Start replace worker: " >>$seqres.full + ( +
[PATCH 12/15] btrfs: new case to run device replace and remount with different compress algorithms simultaneously
Run btrfs replace operations and remount with different compress algorithms simultaneously with fsstress running in background. Signed-off-by: Eryu Guan --- tests/btrfs/068 | 163 tests/btrfs/068.out | 2 + tests/btrfs/group | 1 + 3 files changed, 166 insertions(+) create mode 100755 tests/btrfs/068 create mode 100644 tests/btrfs/068.out diff --git a/tests/btrfs/068 b/tests/btrfs/068 new file mode 100755 index 000..c66dd80 --- /dev/null +++ b/tests/btrfs/068 @@ -0,0 +1,163 @@ +#! /bin/bash +# FSQA Test No. btrfs/068 +# +# Run btrfs replace operations and remount with different compress +# algorithms simultaneously with fsstress running in background. +# +#--- +# Copyright (C) 2014 Red Hat Inc. All rights reserved. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it would be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write the Free Software Foundation, +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA +# +#--- +# + +seq=`basename $0` +seqres=$RESULT_DIR/$seq +echo "QA output created by $seq" + +here=`pwd` +tmp=/tmp/$$ +status=1 +trap "_cleanup; exit \$status" 0 1 2 3 15 + +_cleanup() +{ + cd / + rm -fr $tmp.* +} + +# get standard environment, filters and checks +. ./common/rc +. ./common/filter + +# real QA test starts here +_supported_fs btrfs +_supported_os Linux +_require_scratch +_require_scratch_dev_pool 5 +_require_scratch_dev_pool_equal_size + +rm -f $seqres.full + +# test case array +tcs=( + "-m single -d single" + "-m raid0 -d raid0" + "-m raid1 -d raid0" + "-m raid1 -d raid1" + "-m raid10 -d raid10" +# "-m raid5 -d raid5" # raid5 does not support replace operation yet +# "-m raid6 -d raid6" # raid6 does not support replace operation yet +) + +run_test() +{ + local mkfs_opts=$1 + local saved_mkfs_opts=$MKFS_OPTIONS + local saved_scratch_dev_pool=$SCRATCH_DEV_POOL + + echo "Test $mkfs_opts" >>$seqres.full + + # remove the last device from the SCRATCH_DEV_POOL list so + # _scratch_pool_mkfs won't use all devices in pool + local last_dev="`echo $SCRATCH_DEV_POOL | $AWK_PROG '{print $NF}'`" + SCRATCH_DEV_POOL=`echo $SCRATCH_DEV_POOL | sed -e "s# *$last_dev *##"` + MKFS_OPTIONS="$MKFS_OPTIONS $mkfs_opts" + _scratch_pool_mkfs >>$seqres.full 2>&1 + # make sure we created btrfs with desired options + if [ $? -ne 0 ]; then + echo "mkfs $mkfs_opts failed" + MKFS_OPTIONS=$saved_mkfs_opts + SCRATCH_DEV_POOL=$saved_scratch_dev_pool + return + fi + _scratch_mount >>$seqres.full 2>&1 + + args=`_scale_fsstress_args -p 20 -n 100 $FSSTRESS_AVOID -d $SCRATCH_MNT/stressdir` + echo "Run fsstress $args" >>$seqres.full + $FSSTRESS_PROG $args >/dev/null 2>&1 & + fsstress_pid=$! + + echo -n "Start replace worker: " >>$seqres.full + ( + # take the last device as free device + free_dev=$last_dev + # replace all devices in btrfs_devices in turn + # do not replace SCRATCH_DEV which will be used in _scratch_mount + # and _check_scratch_fs etc. + btrfs_devices=`echo $SCRATCH_DEV_POOL | sed -e "s# *$SCRATCH_DEV *##"` + # set first device in btrfs_devices as first src_dev + src_dev=`echo $btrfs_devices | $AWK_PROG '{print $1}'` + echo "btrfs_devices=$btrfs_devices" >>$seqres.full + echo "free_dev=$free_dev, src_dev=$src_dev" >>$seqres.full + while true; do + echo "Replacing $src_dev with $free_dev" >>$seqres.full + $BTRFS_UTIL_PROG replace start -fB $src_dev $free_dev $SCRATCH_MNT >>$seqres.full 2>&1 + if [ $? -ne 0 ]; then + continue + fi + btrfs_devices="$btrfs_devices $free_dev" + btrfs_devices=`echo $btrfs_devices | sed -e "s# *$src_dev *##"` + free_dev=$src_dev + src_dev=`echo $btrfs_devices | $AWK_PROG '{print $1}'` + done + ) >/dev/null 2>&1 & + replace_pid=$! + echo "$replace_pid" >>$seqres.full + + echo -n "Start remount worker: " >>$seqres.full +
[PATCH 10/15] btrfs: new case to run device replace and scrub operations simultaneously
Run btrfs replace operations and scrub simultaneously with fsstress running in background. Signed-off-by: Eryu Guan --- tests/btrfs/066 | 161 tests/btrfs/066.out | 2 + tests/btrfs/group | 1 + 3 files changed, 164 insertions(+) create mode 100755 tests/btrfs/066 create mode 100644 tests/btrfs/066.out diff --git a/tests/btrfs/066 b/tests/btrfs/066 new file mode 100755 index 000..5d9a42d --- /dev/null +++ b/tests/btrfs/066 @@ -0,0 +1,161 @@ +#! /bin/bash +# FSQA Test No. btrfs/066 +# +# Run btrfs replace operations and scrub simultaneously with fsstress +# running in background. +# +#--- +# Copyright (C) 2014 Red Hat Inc. All rights reserved. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it would be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write the Free Software Foundation, +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA +# +#--- +# + +seq=`basename $0` +seqres=$RESULT_DIR/$seq +echo "QA output created by $seq" + +here=`pwd` +tmp=/tmp/$$ +status=1 +trap "_cleanup; exit \$status" 0 1 2 3 15 + +_cleanup() +{ + cd / + rm -fr $tmp.* +} + +# get standard environment, filters and checks +. ./common/rc +. ./common/filter + +# real QA test starts here +_supported_fs btrfs +_supported_os Linux +_require_scratch +_require_scratch_dev_pool 5 +_require_scratch_dev_pool_equal_size + +rm -f $seqres.full + +# test case array +tcs=( + "-m single -d single" + "-m raid0 -d raid0" + "-m raid1 -d raid0" + "-m raid1 -d raid1" + "-m raid10 -d raid10" +# "-m raid5 -d raid5" # raid5 does not support replace operation yet +# "-m raid6 -d raid6" # raid6 does not support replace operation yet +) + +run_test() +{ + local mkfs_opts=$1 + local saved_mkfs_opts=$MKFS_OPTIONS + local saved_scratch_dev_pool=$SCRATCH_DEV_POOL + + echo "Test $mkfs_opts" >>$seqres.full + + # remove the last device from the SCRATCH_DEV_POOL list so + # _scratch_pool_mkfs won't use all devices in pool + local last_dev="`echo $SCRATCH_DEV_POOL | $AWK_PROG '{print $NF}'`" + SCRATCH_DEV_POOL=`echo $SCRATCH_DEV_POOL | sed -e "s# *$last_dev *##"` + MKFS_OPTIONS="$MKFS_OPTIONS $mkfs_opts" + _scratch_pool_mkfs >>$seqres.full 2>&1 + # make sure we created btrfs with desired options + if [ $? -ne 0 ]; then + echo "mkfs $mkfs_opts failed" + MKFS_OPTIONS=$saved_mkfs_opts + SCRATCH_DEV_POOL=$saved_scratch_dev_pool + return + fi + _scratch_mount >>$seqres.full 2>&1 + + args=`_scale_fsstress_args -p 20 -n 100 $FSSTRESS_AVOID -d $SCRATCH_MNT/stressdir` + echo "Run fsstress $args" >>$seqres.full + $FSSTRESS_PROG $args >/dev/null 2>&1 & + fsstress_pid=$! + + echo -n "Start replace worker: " >>$seqres.full + ( + # take the last device as free device + free_dev=$last_dev + # replace all devices in btrfs_devices in turn + # do not replace SCRATCH_DEV which will be used in _scratch_mount + # and _check_scratch_fs etc. + btrfs_devices=`echo $SCRATCH_DEV_POOL | sed -e "s# *$SCRATCH_DEV *##"` + # set first device in btrfs_devices as first src_dev + src_dev=`echo $btrfs_devices | $AWK_PROG '{print $1}'` + echo "btrfs_devices=$btrfs_devices" >>$seqres.full + echo "free_dev=$free_dev, src_dev=$src_dev" >>$seqres.full + while true; do + echo "Replacing $src_dev with $free_dev" >>$seqres.full + $BTRFS_UTIL_PROG replace start -fB $src_dev $free_dev $SCRATCH_MNT >>$seqres.full 2>&1 + if [ $? -ne 0 ]; then + continue + fi + btrfs_devices="$btrfs_devices $free_dev" + btrfs_devices=`echo $btrfs_devices | sed -e "s# *$src_dev *##"` + free_dev=$src_dev + src_dev=`echo $btrfs_devices | $AWK_PROG '{print $1}'` + done + ) >/dev/null 2>&1 & + replace_pid=$! + echo "$replace_pid" >>$seqres.full + + echo -n "Start scrub worker: " >>$seqres.full + ( + while true; do + $BTRFS_UTIL_
[PATCH 13/15] btrfs: new case to run btrfs scrub and defrag operations simultaneously
Run btrfs scrub and defrag operations simultaneously with fsstress running in background. Signed-off-by: Eryu Guan --- tests/btrfs/069 | 150 tests/btrfs/069.out | 2 + tests/btrfs/group | 1 + 3 files changed, 153 insertions(+) create mode 100755 tests/btrfs/069 create mode 100644 tests/btrfs/069.out diff --git a/tests/btrfs/069 b/tests/btrfs/069 new file mode 100755 index 000..9d773b1 --- /dev/null +++ b/tests/btrfs/069 @@ -0,0 +1,150 @@ +#! /bin/bash +# FSQA Test No. btrfs/069 +# +# Run btrfs scrub and defrag operations simultaneously with fsstress +# running in background. +# +#--- +# Copyright (C) 2014 Red Hat Inc. All rights reserved. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it would be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write the Free Software Foundation, +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA +# +#--- +# + +seq=`basename $0` +seqres=$RESULT_DIR/$seq +echo "QA output created by $seq" + +here=`pwd` +tmp=/tmp/$$ +status=1 +trap "_cleanup; exit \$status" 0 1 2 3 15 + +_cleanup() +{ + cd / + rm -fr $tmp.* +} + +# get standard environment, filters and checks +. ./common/rc +. ./common/filter + +# real QA test starts here +_supported_fs btrfs +_supported_os Linux +_require_scratch +_require_scratch_dev_pool 4 + +rm -f $seqres.full + +# test case array +tcs=( + "-m single -d single" + "-m dup -d single" + "-m raid0 -d raid0" + "-m raid1 -d raid0" + "-m raid1 -d raid1" + "-m raid10 -d raid10" + "-m raid5 -d raid5" + "-m raid6 -d raid6" +) + +run_test() +{ + local mkfs_opts=$1 + local with_compress=$2 + local saved_mkfs_opts=$MKFS_OPTIONS + + echo "Test $mkfs_opts with $with_compress" >>$seqres.full + + MKFS_OPTIONS="$MKFS_OPTIONS $mkfs_opts" + # dup only works on single device + if [[ "$mkfs_opts" =~ dup ]]; then + _scratch_mkfs >>$seqres.full 2>&1 + else + _scratch_pool_mkfs >>$seqres.full 2>&1 + fi + local ret=$? + MKFS_OPTIONS=$saved_mkfs_opts + # make sure we created btrfs with desired options + if [ $ret -ne 0 ]; then + echo "mkfs $mkfs_opts failed" + return + fi + _scratch_mount >>$seqres.full 2>&1 + + args=`_scale_fsstress_args -p 20 -n 100 $FSSTRESS_AVOID -d $SCRATCH_MNT/stressdir` + echo "Run fsstress $args" >>$seqres.full + $FSSTRESS_PROG $args >/dev/null 2>&1 & + fsstress_pid=$! + + echo -n "Start scrub worker: " >>$seqres.full + ( + while true; do + $BTRFS_UTIL_PROG scrub start -B $SCRATCH_MNT >>$seqres.full 2>&1 + done + ) & + scrub_pid=$! + echo "$scrub_pid" >>$seqres.full + + echo -n "Start defrag worker: " >>$seqres.full + ( + while true; do + if [ "$with_compress" == "nocompress" ]; then + find $SCRATCH_MNT \( -type f -o -type d \) -exec \ + $BTRFS_UTIL_PROG filesystem defrag {} \; >>$seqres.full 2>&1 + else + find $SCRATCH_MNT \( -type f -o -type d \) -exec \ + $BTRFS_UTIL_PROG filesystem defrag -clzo {} \; >>$seqres.full 2>&1 + find $SCRATCH_MNT \( -type f -o -type d \) -exec \ + $BTRFS_UTIL_PROG filesystem defrag -czlib {} \; >>$seqres.full 2>&1 + fi + done + ) & + defrag_pid=$! + echo "$defrag_pid" >>$seqres.full + + echo "Wait for fsstress to exit and kill all background workers" >>$seqres.full + wait $fsstress_pid + kill $scrub_pid $defrag_pid + wait + # wait for the scrub and defrag operations to finish + while ps aux | grep "scrub start" | grep -qv grep; do + sleep 1 + done + while ps aux | grep "btrfs filesystem defrag" | grep -qv grep; do + sleep 1 + done + + echo "Scrub the filesystem" >>$seqres.full + $BTRFS_UTIL_PROG scrub start -B $SCRATCH_MNT >>$seqres.full 2>&1 + if [ $? -ne 0 ]; then + echo "Scrub find errors in \"$mkfs_opts\" test" | tee -a $seqres.ful
[PATCH 14/15] btrfs: new case to run btrfs scrub and remount with different compress algorithms simultaneously
Run btrfs scrub and remount with different compress algorithms simultaneously with fsstress running in background. Signed-off-by: Eryu Guan --- tests/btrfs/070 | 142 tests/btrfs/070.out | 2 + tests/btrfs/group | 1 + 3 files changed, 145 insertions(+) create mode 100755 tests/btrfs/070 create mode 100644 tests/btrfs/070.out diff --git a/tests/btrfs/070 b/tests/btrfs/070 new file mode 100755 index 000..b53198c --- /dev/null +++ b/tests/btrfs/070 @@ -0,0 +1,142 @@ +#! /bin/bash +# FSQA Test No. btrfs/070 +# +# Run btrfs scrub and remount with different compress algorithms +# simultaneously with fsstress running in background. +# +#--- +# Copyright (C) 2014 Red Hat Inc. All rights reserved. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it would be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write the Free Software Foundation, +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA +# +#--- +# + +seq=`basename $0` +seqres=$RESULT_DIR/$seq +echo "QA output created by $seq" + +here=`pwd` +tmp=/tmp/$$ +status=1 +trap "_cleanup; exit \$status" 0 1 2 3 15 + +_cleanup() +{ + cd / + rm -fr $tmp.* +} + +# get standard environment, filters and checks +. ./common/rc +. ./common/filter + +# real QA test starts here +_supported_fs btrfs +_supported_os Linux +_require_scratch +_require_scratch_dev_pool 4 + +rm -f $seqres.full + +# test case array +tcs=( + "-m single -d single" + "-m dup -d single" + "-m raid0 -d raid0" + "-m raid1 -d raid0" + "-m raid1 -d raid1" + "-m raid10 -d raid10" + "-m raid5 -d raid5" + "-m raid6 -d raid6" +) + +run_test() +{ + local mkfs_opts=$1 + local saved_mkfs_opts=$MKFS_OPTIONS + + echo "Test $mkfs_opts with $with_compress" >>$seqres.full + + MKFS_OPTIONS="$MKFS_OPTIONS $mkfs_opts" + # dup only works on single device + if [[ "$mkfs_opts" =~ dup ]]; then + _scratch_mkfs >>$seqres.full 2>&1 + else + _scratch_pool_mkfs >>$seqres.full 2>&1 + fi + local ret=$? + MKFS_OPTIONS=$saved_mkfs_opts + # make sure we created btrfs with desired options + if [ $ret -ne 0 ]; then + echo "mkfs $mkfs_opts failed" + return + fi + _scratch_mount >>$seqres.full 2>&1 + + args=`_scale_fsstress_args -p 20 -n 100 $FSSTRESS_AVOID -d $SCRATCH_MNT/stressdir` + echo "Run fsstress $args" >>$seqres.full + $FSSTRESS_PROG $args >/dev/null 2>&1 & + fsstress_pid=$! + + echo -n "Start scrub worker: " >>$seqres.full + ( + while true; do + $BTRFS_UTIL_PROG scrub start -B $SCRATCH_MNT >>$seqres.full 2>&1 + done + ) & + scrub_pid=$! + echo "$scrub_pid" >>$seqres.full + + echo -n "Start remount worker: " >>$seqres.full + ( + while true; do + for algo in no zlib lzo; do + $MOUNT_PROG -o remount,compress=$algo $SCRATCH_MNT >>$seqres.full 2>&1 + done + done + ) >/dev/null 2>&1 & + remount_pid=$! + echo "$remount_pid" >>$seqres.full + + echo "Wait for fsstress to exit and kill all background workers" >>$seqres.full + wait $fsstress_pid + kill $scrub_pid $remount_pid + wait + # wait for the scrub and remount operations to finish + while ps aux | grep "scrub start" | grep -qv grep; do + sleep 1 + done + while ps aux | grep "mount.*$SCRATCH_MNT" | grep -qv grep; do + sleep 1 + done + + echo "Scrub the filesystem" >>$seqres.full + $BTRFS_UTIL_PROG scrub start -B $SCRATCH_MNT >>$seqres.full 2>&1 + if [ $? -ne 0 ]; then + echo "Scrub find errors in \"$mkfs_opts\" test" | tee -a $seqres.full + fi + + $BTRFS_UTIL_PROG filesystem sync $SCRATCH_MNT >/dev/null 2>&1 + _scratch_unmount + _check_scratch_fs +} + +echo "Silence is golden" +for t in "${tcs[@]}"; do + run_test "$t" +done + +status=0 +exit diff --git a/tests/btrfs/070.out b/tests/btrfs/070.out new file mode 100644 index 000..8940c5d --- /dev/null +++ b/tests/btrfs/070.out @@ -0,0 +1,2 @@ +QA output created by 070 +Silence is golden diff --git a/tests/btrfs/group b/test
[PATCH 15/15] btrfs: new case to run defrag and remount with different compress algorithms simultaneously
Run btrfs defrag operations and remount with different compress algorithms simultaneously with fsstress running in background. Signed-off-by: Eryu Guan --- tests/btrfs/071 | 152 tests/btrfs/071.out | 2 + tests/btrfs/group | 1 + 3 files changed, 155 insertions(+) create mode 100755 tests/btrfs/071 create mode 100644 tests/btrfs/071.out diff --git a/tests/btrfs/071 b/tests/btrfs/071 new file mode 100755 index 000..4434cd6 --- /dev/null +++ b/tests/btrfs/071 @@ -0,0 +1,152 @@ +#! /bin/bash +# FSQA Test No. btrfs/069 +# +# Run btrfs defrag operations and remount with different compress algorithms +# simultaneously with fsstress running in background. +# +#--- +# Copyright (C) 2014 Red Hat Inc. All rights reserved. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it would be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write the Free Software Foundation, +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA +# +#--- +# + +seq=`basename $0` +seqres=$RESULT_DIR/$seq +echo "QA output created by $seq" + +here=`pwd` +tmp=/tmp/$$ +status=1 +trap "_cleanup; exit \$status" 0 1 2 3 15 + +_cleanup() +{ + cd / + rm -fr $tmp.* +} + +# get standard environment, filters and checks +. ./common/rc +. ./common/filter + +# real QA test starts here +_supported_fs btrfs +_supported_os Linux +_require_scratch +_require_scratch_dev_pool 4 + +rm -f $seqres.full + +# test case array +tcs=( + "-m single -d single" + "-m dup -d single" + "-m raid0 -d raid0" + "-m raid1 -d raid0" + "-m raid1 -d raid1" + "-m raid10 -d raid10" + "-m raid5 -d raid5" + "-m raid6 -d raid6" +) + +run_test() +{ + local mkfs_opts=$1 + local with_compress=$2 + local saved_mkfs_opts=$MKFS_OPTIONS + + echo "Test $mkfs_opts with $with_compress" >>$seqres.full + + MKFS_OPTIONS="$MKFS_OPTIONS $mkfs_opts" + # dup only works on single device + if [[ "$mkfs_opts" =~ dup ]]; then + _scratch_mkfs >>$seqres.full 2>&1 + else + _scratch_pool_mkfs >>$seqres.full 2>&1 + fi + local ret=$? + MKFS_OPTIONS=$saved_mkfs_opts + # make sure we created btrfs with desired options + if [ $ret -ne 0 ]; then + echo "mkfs $mkfs_opts failed" + return + fi + _scratch_mount >>$seqres.full 2>&1 + + args=`_scale_fsstress_args -p 20 -n 100 $FSSTRESS_AVOID -d $SCRATCH_MNT/stressdir` + echo "Run fsstress $args" >>$seqres.full + $FSSTRESS_PROG $args >/dev/null 2>&1 & + fsstress_pid=$! + + echo -n "Start defrag worker: " >>$seqres.full + ( + while true; do + if [ "$with_compress" == "nocompress" ]; then + find $SCRATCH_MNT \( -type f -o -type d \) -exec \ + $BTRFS_UTIL_PROG filesystem defrag {} \; >>$seqres.full 2>&1 + else + find $SCRATCH_MNT \( -type f -o -type d \) -exec \ + $BTRFS_UTIL_PROG filesystem defrag -clzo {} \; >>$seqres.full 2>&1 + find $SCRATCH_MNT \( -type f -o -type d \) -exec \ + $BTRFS_UTIL_PROG filesystem defrag -czlib {} \; >>$seqres.full 2>&1 + fi + done + ) & + defrag_pid=$! + echo "$defrag_pid" >>$seqres.full + + echo -n "Start remount worker: " >>$seqres.full + ( + while true; do + for algo in no zlib lzo; do + $MOUNT_PROG -o remount,compress=$algo $SCRATCH_MNT >>$seqres.full 2>&1 + done + done + ) >/dev/null 2>&1 & + remount_pid=$! + echo "$remount_pid" >>$seqres.full + + echo "Wait for fsstress to exit and kill all background workers" >>$seqres.full + wait $fsstress_pid + kill $defrag_pid $remount_pid + wait + # wait for the defrag and remount operations to finish + while ps aux | grep "btrfs filesystem defrag" | grep -qv grep; do + sleep 1 + done + while ps aux | grep "mount.*$SCRATCH_MNT" | grep -qv grep; do + sleep 1 + done + + echo "Scrub the filesystem"
[PATCH 11/15] btrfs: new case to run device replace and defrag operations simultaneously
Run btrfs replace operations and defrag simultaneously with fsstress running in background. Signed-off-by: Eryu Guan --- tests/btrfs/067 | 171 tests/btrfs/067.out | 2 + tests/btrfs/group | 1 + 3 files changed, 174 insertions(+) create mode 100755 tests/btrfs/067 create mode 100644 tests/btrfs/067.out diff --git a/tests/btrfs/067 b/tests/btrfs/067 new file mode 100755 index 000..477b85f --- /dev/null +++ b/tests/btrfs/067 @@ -0,0 +1,171 @@ +#! /bin/bash +# FSQA Test No. btrfs/067 +# +# Run btrfs replace operations and defrag simultaneously with fsstress +# running in background. +# +#--- +# Copyright (C) 2014 Red Hat Inc. All rights reserved. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it would be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write the Free Software Foundation, +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA +# +#--- +# + +seq=`basename $0` +seqres=$RESULT_DIR/$seq +echo "QA output created by $seq" + +here=`pwd` +tmp=/tmp/$$ +status=1 +trap "_cleanup; exit \$status" 0 1 2 3 15 + +_cleanup() +{ + cd / + rm -fr $tmp.* +} + +# get standard environment, filters and checks +. ./common/rc +. ./common/filter + +# real QA test starts here +_supported_fs btrfs +_supported_os Linux +_require_scratch +_require_scratch_dev_pool 5 +_require_scratch_dev_pool_equal_size + +rm -f $seqres.full + +# test case array +tcs=( + "-m single -d single" + "-m raid0 -d raid0" + "-m raid1 -d raid0" + "-m raid1 -d raid1" + "-m raid10 -d raid10" +# "-m raid5 -d raid5" # raid5 does not support replace operation yet +# "-m raid6 -d raid6" # raid6 does not support replace operation yet +) + +run_test() +{ + local mkfs_opts=$1 + local with_compress=$2 + local saved_mkfs_opts=$MKFS_OPTIONS + local saved_scratch_dev_pool=$SCRATCH_DEV_POOL + + echo "Test $mkfs_opts with $with_compress" >>$seqres.full + + # remove the last device from the SCRATCH_DEV_POOL list so + # _scratch_pool_mkfs won't use all devices in pool + local last_dev="`echo $SCRATCH_DEV_POOL | $AWK_PROG '{print $NF}'`" + SCRATCH_DEV_POOL=`echo $SCRATCH_DEV_POOL | sed -e "s# *$last_dev *##"` + MKFS_OPTIONS="$MKFS_OPTIONS $mkfs_opts" + _scratch_pool_mkfs >>$seqres.full 2>&1 + # make sure we created btrfs with desired options + if [ $? -ne 0 ]; then + echo "mkfs $mkfs_opts failed" + MKFS_OPTIONS=$saved_mkfs_opts + SCRATCH_DEV_POOL=$saved_scratch_dev_pool + return + fi + _scratch_mount >>$seqres.full 2>&1 + + args=`_scale_fsstress_args -p 20 -n 100 $FSSTRESS_AVOID -d $SCRATCH_MNT/stressdir` + echo "Run fsstress $args" >>$seqres.full + $FSSTRESS_PROG $args >/dev/null 2>&1 & + fsstress_pid=$! + + echo -n "Start replace worker: " >>$seqres.full + ( + # take the last device as free device + free_dev=$last_dev + # replace all devices in btrfs_devices in turn + # do not replace SCRATCH_DEV which will be used in _scratch_mount + # and _check_scratch_fs etc. + btrfs_devices=`echo $SCRATCH_DEV_POOL | sed -e "s# *$SCRATCH_DEV *##"` + # set first device in btrfs_devices as first src_dev + src_dev=`echo $btrfs_devices | $AWK_PROG '{print $1}'` + echo "btrfs_devices=$btrfs_devices" >>$seqres.full + echo "free_dev=$free_dev, src_dev=$src_dev" >>$seqres.full + while true; do + echo "Replacing $src_dev with $free_dev" >>$seqres.full + $BTRFS_UTIL_PROG replace start -fB $src_dev $free_dev $SCRATCH_MNT >>$seqres.full 2>&1 + if [ $? -ne 0 ]; then + continue + fi + btrfs_devices="$btrfs_devices $free_dev" + btrfs_devices=`echo $btrfs_devices | sed -e "s# *$src_dev *##"` + free_dev=$src_dev + src_dev=`echo $btrfs_devices | $AWK_PROG '{print $1}'` + done + ) >/dev/null 2>&1 & + replace_pid=$! + echo "$replace_pid" >>$seqres.full + + echo -n "Start defrag worker: " >>$seqres.full + ( +
[PATCH 00/15] xfstests: new btrfs stress test cases
This patchset add new stress test cases for btrfs by running two different btrfs operations simultaneously under fsstress to ensure btrfs doesn't hang or oops in such situations. btrfs scrub and btrfs check will be run after each test. The test matrix is the combination of 6 btrfs operations: balance create/mount/umount/delete subvolume replace device scrub defrag remount with different compress algorithms Short descriptions: 057: balance-subvolume 058: balance-scrub 059: balance-defrag 060: balance-remount 061: balance-replace 062: subvolume-replace 063: subvolume-scrub 064: subvolume-defrag 065: subvolume-remount 066: replace-scrub 067: replace-defrag 068: replace-remount 069: scrub-defrag 070: scrub-remount 071: defrag-remount Some issues I've seen: 1. subvolume cannot be mounted with selinux context, so you may see such logs in dmesg SELinux: mount invalid. Same superblock, different security settings for (dev dm-8, type btrfs) I've reported the bug to btrfs list, see [BUG] cannot mount subvolume with selinux context 2. btrfs replace operation always returns ENOENT if balance is running So in 061.full you'll see ERROR: ioctl(DEV_REPLACE_START) failed on "/mnt/testarea/scratch": No such file or directory, no error Not sure if it's btrfs bug, at least I think the error code is misleading 3. replace operation hangs the kernel(3.16-rc4+) with fsstress running So case 062/066/067/068 will hang This is my first attemp to add these tests, any comments are welcomed! Thanks, Eryu Guan Eryu Guan (15): btrfs: new test to run btrfs balance and subvolume test simultaneously btrfs: new test to run btrfs balance and scrub simltaneously btrfs: new test to run btrfs balance and defrag operations simultaneously btrfs: new case to run btrfs balance and remount with different compress algorithms btrfs: new case to run btrfs balance and device replace simultaneously btrfs: new case to run btrfs subvolume create/delete operations and device replace simultaneously btrfs: new case to run btrfs subvolume create/delete operations and scrub simultaneously btrfs: new case to run btrfs subvolume create/delete and defrag operations simultaneously btrfs: new case to run subvolume create/delete and remount with different compress algorithms btrfs: new case to run device replace and scrub operations simultaneously btrfs: new case to run device replace and defrag operations simultaneously btrfs: new case to run device replace and remount with different compress algorithms simultaneously btrfs: new case to run btrfs scrub and defrag operations simultaneously btrfs: new case to run btrfs scrub and remount with different compress algorithms simultaneously btrfs: new case to run defrag and remount with different compress algorithms simultaneously common/rc | 24 tests/btrfs/057 | 147 tests/btrfs/057.out | 2 + tests/btrfs/058 | 141 +++ tests/btrfs/058.out | 2 + tests/btrfs/059 | 150 + tests/btrfs/059.out | 2 + tests/btrfs/060 | 142 +++ tests/btrfs/060.out | 2 + tests/btrfs/061 | 161 + tests/btrfs/061.out | 2 + tests/btrfs/062 | 167 ++ tests/btrfs/062.out | 2 + tests/btrfs/063 | 146 tests/btrfs/063.out | 2 + tests/btrfs/064 | 155 +++ tests/btrfs/064.out | 2 + tests/btrfs/065 | 148 + tests/btrfs/065.out | 2 + tests/btrfs/066 | 161 + tests/btrfs/066.out | 2 + tests/btrfs/067 | 171 tests/btrfs/067.out | 2 + tests/btrfs/068 | 163 + tests/btrfs/068.out | 2 + tests/btrfs/069 | 150 + tests/btrfs/069.out | 2 + tests/btrfs/070 | 142 +++ tests/btrfs/070.out | 2 + tests/btrfs/071 | 152 ++ tests/btrfs/071.out | 2 + tests/btrfs/group | 15 + 32 files changed, 2365 insertions(+) create mode 100755 tests/btrfs/057 create mode 100644 tests/btrfs/057.out create mode 100755 tests/btrfs/058 create mode 100644 tests/btrfs/058.out create mode 100755 tests/btrfs/059 create mode 100644 tests/btrfs/059.out create mode 100755 tests/btrfs/060 create mode 100644 tests/btrfs/060.out create mode 100755
Re: [PATCH 00/15] xfstests: new btrfs stress test cases
On Thu, Aug 21, 2014 at 01:33:48AM +0800, Eryu Guan wrote: > This patchset add new stress test cases for btrfs by running two > different btrfs operations simultaneously under fsstress to ensure > btrfs doesn't hang or oops in such situations. btrfs scrub and > btrfs check will be run after each test. Cool. > The test matrix is the combination of 6 btrfs operations: > > balance > create/mount/umount/delete subvolume > replace device > scrub > defrag > remount with different compress algorithms > > Short descriptions: > > 057: balance-subvolume > 058: balance-scrub > 059: balance-defrag > 060: balance-remount > 061: balance-replace > 062: subvolume-replace > 063: subvolume-scrub > 064: subvolume-defrag > 065: subvolume-remount > 066: replace-scrub > 067: replace-defrag > 068: replace-remount > 069: scrub-defrag > 070: scrub-remount > 071: defrag-remount But I'm not sure it should be built this way. At the very least each operation's implementation should be in a shared function somewhere instead of being duplicated in each test. But I don't think there should be a seperate test for each combination. With a bit of fiddly bash you can automate generating unique combinations of operations that are defined as functions in one test. btrfs_op_balance() { echo hi } btrfs_op_scrub() { echo hi } btrfs_op_defrag() { echo hi } ops=($(declare -F | awk '/-f btrfs_op_/ {print $3}')) nr=${#ops[@]} for i in $(seq 0 $((nr - 2))); do for j in $(seq $((i + 1)) $((nr - 1))); do echo ${ops[i]} ${ops[j]} done done Something like that. - z -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs on bcache
Hi, has this issue been resolved? I would like to use the bcache + btrfs combo. Thanks -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 0/3] btrfs-progs: remove full /dev scanning
btrfs fileystem show and btrfs device scan today both have the "-d" option to scan everything under /dev. But we also have a mechanism to scan everything in /proc/partitions, which should always be sufficient. If anyone knows why we'd find something deep under /dev but not in /proc/partitions, speak now or forever hold your peace... Tested this by running through a matrix of -d, -m, or "" args for show/scan, for a 2-device fs, with and without a symlinked device, with and without a symlinked mountpoint. All output was identical. Thanks, -Eric -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 1/3] btrfs-progs: scan /proc/partitions not all of /dev with "-d"
We can scan for btrfs devices in a few ways. By default libblkid is used for "device scan" and "filesystem show"; with the -m option only mounted filesystems are scanned, and with -d we physically read every system device. But there's no reason for the complexity of a descent through /dev; /proc/partitions has every device known to the kernel, so just use that when -d is specified. Signed-off-by: Eric Sandeen --- diff --git a/cmds-device.c b/cmds-device.c index b6772b9..519681a 100644 --- a/cmds-device.c +++ b/cmds-device.c @@ -224,7 +224,7 @@ static int cmd_scan_dev(int argc, char **argv) break; switch (c) { case 'd': - where = BTRFS_SCAN_DEV; + where = BTRFS_SCAN_PROC; all = 1; break; default: diff --git a/cmds-filesystem.c b/cmds-filesystem.c index bf87bbe..0ad7e8f 100644 --- a/cmds-filesystem.c +++ b/cmds-filesystem.c @@ -614,7 +614,7 @@ static int cmd_show(int argc, char **argv) break; switch (c) { case 'd': - where = BTRFS_SCAN_DEV; + where = BTRFS_SCAN_PROC; break; case 'm': where = BTRFS_SCAN_MOUNTED; @@ -638,7 +638,7 @@ static int cmd_show(int argc, char **argv) * right away */ if (type == BTRFS_ARG_BLKDEV) { - if (where == BTRFS_SCAN_DEV) { + if (where == BTRFS_SCAN_PROC) { /* we need to do this because * legacy BTRFS_SCAN_DEV * provides /dev/dm-x paths @@ -681,7 +681,7 @@ static int cmd_show(int argc, char **argv) } } - if (where == BTRFS_SCAN_DEV) + if (where == BTRFS_SCAN_PROC) goto devs_only; /* show mounted btrfs */ -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 2/3] btrfs-progs: don't fall back to recursive /dev scan
If we didn't find what we are looking for in /proc/partitions, we're not going to find it by scanning every node under /dev, either. But that's just what btrfs_scan_for_fsid() does. Remove that fallback; at that point btrfs_scan_for_fsid() just calls scan_for_btrfs(), so remove the wrapper & call it directly. Side note: so, these paths always use /proc/partitions, not libblkid. Userspace-intiated scans default to libblkid. I presume this is part of the design, and intentional? Anyway, not changing it now! Signed-off-by: Eric Sandeen --- diff --git a/disk-io.c b/disk-io.c index d10d647..077eb7e 100644 --- a/disk-io.c +++ b/disk-io.c @@ -1018,7 +1018,7 @@ int btrfs_scan_fs_devices(int fd, const char *path, } if (total_devs != 1) { - ret = btrfs_scan_for_fsid(run_ioctl); + ret = scan_for_btrfs(BTRFS_SCAN_PROC, run_ioctl); if (ret) return ret; } diff --git a/utils.c b/utils.c index d61cbec..12ed7a2 100644 --- a/utils.c +++ b/utils.c @@ -1151,7 +1151,7 @@ int check_mounted_where(int fd, const char *file, char *where, int size, /* scan other devices */ if (is_btrfs && total_devs > 1) { - if ((ret = btrfs_scan_for_fsid(!BTRFS_UPDATE_KERNEL))) + if ((ret = scan_for_btrfs(BTRFS_SCAN_PROC, !BTRFS_UPDATE_KERNEL))) return ret; } @@ -1339,16 +1339,6 @@ fail: return ret; } -int btrfs_scan_for_fsid(int run_ioctls) -{ - int ret; - - ret = scan_for_btrfs(BTRFS_SCAN_PROC, run_ioctls); - if (ret) - ret = scan_for_btrfs(BTRFS_SCAN_DEV, run_ioctls); - return ret; -} - int btrfs_device_already_in_root(struct btrfs_root *root, int fd, int super_offset) { -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 3/3] btrfs-progs: remove BTRFS_SCAN_DEV and btrfs_scan_one_dir
After the previous 2 patches, nothing uses whole-dev-tree scanning, so remove the code which implemented that functionality. Signed-off-by: Eric Sandeen --- diff --git a/utils.c b/utils.c index 12ed7a2..765a214 100644 --- a/utils.c +++ b/utils.c @@ -1228,117 +1228,6 @@ void btrfs_register_one_device(char *fname) close(fd); } -int btrfs_scan_one_dir(char *dirname, int run_ioctl) -{ - DIR *dirp = NULL; - struct dirent *dirent; - struct pending_dir *pending; - struct stat st; - int ret; - int fd; - int dirname_len; - char *fullpath; - struct list_head pending_list; - struct btrfs_fs_devices *tmp_devices; - u64 num_devices; - - INIT_LIST_HEAD(&pending_list); - - pending = malloc(sizeof(*pending)); - if (!pending) - return -ENOMEM; - strcpy(pending->name, dirname); - -again: - dirname_len = strlen(pending->name); - fullpath = malloc(PATH_MAX); - dirname = pending->name; - - if (!fullpath) { - ret = -ENOMEM; - goto fail; - } - dirp = opendir(dirname); - if (!dirp) { - fprintf(stderr, "Unable to open %s for scanning\n", dirname); - ret = -errno; - goto fail; - } - while(1) { - dirent = readdir(dirp); - if (!dirent) - break; - if (dirent->d_name[0] == '.') - continue; - if (dirname_len + strlen(dirent->d_name) + 2 > PATH_MAX) { - ret = -EFAULT; - goto fail; - } - snprintf(fullpath, PATH_MAX, "%s/%s", dirname, dirent->d_name); - ret = lstat(fullpath, &st); - if (ret < 0) { - fprintf(stderr, "failed to stat %s\n", fullpath); - continue; - } - if (S_ISLNK(st.st_mode)) - continue; - if (S_ISDIR(st.st_mode)) { - struct pending_dir *next = malloc(sizeof(*next)); - if (!next) { - ret = -ENOMEM; - goto fail; - } - strcpy(next->name, fullpath); - list_add_tail(&next->list, &pending_list); - } - if (!S_ISBLK(st.st_mode)) { - continue; - } - fd = open(fullpath, O_RDONLY); - if (fd < 0) { - /* ignore the following errors: - ENXIO (device don't exists) - ENOMEDIUM (No medium found -> - like a cd tray empty) - */ - if(errno != ENXIO && errno != ENOMEDIUM) - fprintf(stderr, "failed to read %s: %s\n", - fullpath, strerror(errno)); - continue; - } - ret = btrfs_scan_one_device(fd, fullpath, &tmp_devices, - &num_devices, - BTRFS_SUPER_INFO_OFFSET, 0); - if (ret == 0 && run_ioctl > 0) { - btrfs_register_one_device(fullpath); - } - close(fd); - } - if (!list_empty(&pending_list)) { - free(pending); - pending = list_entry(pending_list.next, struct pending_dir, -list); - free(fullpath); - list_del(&pending->list); - closedir(dirp); - dirp = NULL; - goto again; - } - ret = 0; -fail: - free(pending); - free(fullpath); - while (!list_empty(&pending_list)) { - pending = list_entry(pending_list.next, struct pending_dir, -list); - list_del(&pending->list); - free(pending); - } - if (dirp) - closedir(dirp); - return ret; -} - int btrfs_device_already_in_root(struct btrfs_root *root, int fd, int super_offset) { @@ -2291,9 +2180,6 @@ int scan_for_btrfs(int where, int update_kernel) case BTRFS_SCAN_PROC: ret = btrfs_scan_block_devices(update_kernel); break; - case BTRFS_SCAN_DEV: - ret = btrfs_scan_one_dir("/dev", update_kernel); - break; case BTRFS_SCAN_LBLKID: ret = btrfs_scan_lblkid(update_kernel); break; diff --git a/utils.h b/utils.h index 0c9b65f..3a05131 100644 --- a/utils.h +++ b/utils.h @@ -27,9 +27,8 @@ #define BTRFS_MKFS_SMALL_VOLUME_SIZE (1024 * 1024 * 1024) #define BTRFS_SCAN_PROC
Re: [PATCH] xfstest: add tests/btrfs/059 to test seed operations
On Fri, Aug 15, 2014 at 11:16:21PM +0800, Anand Jain wrote: > This test contains a set of 5 sub test cases which will tests > the device add and replacement on a seed FS. The kernel messages > are checked at the end of the each of the 5 test cases to see > if there are any kernel warnings or bugs reported. As in general > btrfs do report warning when device counts such as num_devices > and rw_devices appears to be wrong. > > Since the tests has to deal with the replace, it would need > up to five scratch device with same size. Please separate this out into 5 tests using shared infrastructure. Cheers, Dave. -- Dave Chinner da...@fromorbit.com -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 01/15] btrfs: new test to run btrfs balance and subvolume test simultaneously
Original Message Subject: [PATCH 01/15] btrfs: new test to run btrfs balance and subvolume test simultaneously From: Eryu Guan To: Date: 2014年08月21日 01:33 Run btrfs balance and subvolume create/mount/umount/delete simultaneously, with fsstress running in background. Signed-off-by: Eryu Guan --- tests/btrfs/057 | 147 tests/btrfs/057.out | 2 + tests/btrfs/group | 1 + 3 files changed, 150 insertions(+) create mode 100755 tests/btrfs/057 create mode 100644 tests/btrfs/057.out diff --git a/tests/btrfs/057 b/tests/btrfs/057 new file mode 100755 index 000..2f507a7 --- /dev/null +++ b/tests/btrfs/057 @@ -0,0 +1,147 @@ +#! /bin/bash +# FSQA Test No. btrfs/057 +# +# Run btrfs balance and subvolume create/mount/umount/delete simultaneously, +# with fsstress running in background. +# +#--- +# Copyright (C) 2014 Red Hat Inc. All rights reserved. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it would be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write the Free Software Foundation, +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA +# +#--- +# + +seq=`basename $0` +seqres=$RESULT_DIR/$seq +echo "QA output created by $seq" + +here=`pwd` +tmp=/tmp/$$ +status=1 +trap "_cleanup; exit \$status" 0 1 2 3 15 + +_cleanup() +{ + cd / + rm -fr $tmp.* +} + +# get standard environment, filters and checks +. ./common/rc +. ./common/filter + +# real QA test starts here +_supported_fs btrfs +_supported_os Linux +_require_scratch +_require_scratch_dev_pool 4 + +rm -f $seqres.full + +# test case array +tcs=( + "-m single -d single" + "-m dup -d single" + "-m raid0 -d raid0" + "-m raid1 -d raid0" + "-m raid1 -d raid1" + "-m raid10 -d raid10" + "-m raid5 -d raid5" + "-m raid6 -d raid6" +) I wonder should we add the mkfs options there. Since xfstests already use environment MKFS_OPTIONS to do mkfs, if really need to test all mkfs options, IMO it is better to change MKFS_OPTIONS on each test round. + +run_test() +{ + local mkfs_opts=$1 + local saved_mkfs_opts=$MKFS_OPTIONS + local subvol_mnt=$tmp.mnt + + echo "Test $mkfs_opts" >>$seqres.full + + MKFS_OPTIONS="$MKFS_OPTIONS $mkfs_opts" + # dup only works on single device + if [[ "$mkfs_opts" =~ dup ]]; then + _scratch_mkfs >>$seqres.full 2>&1 + else + _scratch_pool_mkfs >>$seqres.full 2>&1 + fi + ret=$? + MKFS_OPTIONS=$saved_mkfs_opts + # make sure we created btrfs with desired options + if [ $ret -ne 0 ]; then + echo "mkfs $mkfs_opts failed" + return + fi + + _scratch_mount >>$seqres.full 2>&1 + + args=`_scale_fsstress_args -p 20 -n 100 $FSSTRESS_AVOID -d $SCRATCH_MNT/stressdir` + echo "Run fsstress $args" >>$seqres.full + $FSSTRESS_PROG $args >/dev/null 2>&1 & + fsstress_pid=$! + + echo -n "Start balance worker: " >>$seqres.full + ( + while true; do + $BTRFS_UTIL_PROG balance start $SCRATCH_MNT + done + ) >/dev/null 2>&1 & + balance_pid=$! + echo "$balance_pid" >>$seqres.full + + echo -n "Start subvolume worker: " >>$seqres.full + mkdir -p $subvol_mnt + ( + while true; do + $BTRFS_UTIL_PROG subvolume create $SCRATCH_MNT/subvol_$$ + $MOUNT_PROG -o subvol=subvol_$$ $SCRATCH_DEV $subvol_mnt + $UMOUNT_PROG $subvol_mnt + $BTRFS_UTIL_PROG subvolume delete $SCRATCH_MNT/subvol_$$ + done + ) >/dev/null 2>&1 & + subvol_pid=$! + echo "$subvol_pid" >>$seqres.full + + echo "Wait for fsstress to exit and kill all background workers" >>$seqres.full + wait $fsstress_pid What about integrate this 'run in background; record PID; wait; kill' thing into one function or two? and then thing would be like this: add_test_background TEST_FUNC1 PID_RET1 add_test_background TEST_FUNC2 PID_RET2 ... stop_test_backgroupd PID_RET1 stop_test_backgroupd PID_RET2 Which will be much cleaner. + + kill $balance_pid $subvol_pid + wait + # the balance process might be still in D state and cannot be killed + # which could block umount, wait for it t
[PATCH 1/2] btrfs-progs: cleanup duplicate assignment of variable leaf for btrfs-restore
The value of variable leaf in while loop don't have to be set for every round. Just move it outside. Signed-off-by: Gui Hecheng --- cmds-restore.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/cmds-restore.c b/cmds-restore.c index a6f535c..f417e0b 100644 --- a/cmds-restore.c +++ b/cmds-restore.c @@ -962,8 +962,9 @@ static int do_list_roots(struct btrfs_root *root) return -1; } + leaf = path->nodes[0]; + while (1) { - leaf = path->nodes[0]; slot = path->slots[0]; if (slot >= btrfs_header_nritems(leaf)) { ret = btrfs_next_leaf(root, path); -- 1.8.1.4 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 2/2] btrfs-progs: remove unused flags for btrfs_path
The three flags of @btrfs_path: btrfs_path { unsigned int keep_locks:1; unsigned int skip_locking:1; unsigned int leave_spinning:1; } have little meaning, because the userspace @btrfs_search_slot() is free of locking and no other routines will decide their behavior on these. So just remove them. Signed-off-by: Gui Hecheng --- cmds-restore.c | 2 -- ctree.h| 3 --- extent-tree.c | 6 -- 3 files changed, 11 deletions(-) diff --git a/cmds-restore.c b/cmds-restore.c index f417e0b..bfd883d 100644 --- a/cmds-restore.c +++ b/cmds-restore.c @@ -566,7 +566,6 @@ static int copy_file(struct btrfs_root *root, int fd, struct btrfs_key *key, fprintf(stderr, "Ran out of memory\n"); return -ENOMEM; } - path->skip_locking = 1; ret = btrfs_lookup_inode(NULL, root, path, key, 0); if (ret == 0) { @@ -704,7 +703,6 @@ static int search_dir(struct btrfs_root *root, struct btrfs_key *key, fprintf(stderr, "Ran out of memory\n"); return -ENOMEM; } - path->skip_locking = 1; key->offset = 0; key->type = BTRFS_DIR_INDEX_KEY; diff --git a/ctree.h b/ctree.h index 35d3633..fe1cd53 100644 --- a/ctree.h +++ b/ctree.h @@ -552,9 +552,6 @@ struct btrfs_path { * and to force calls to keep space in the nodes */ unsigned int search_for_split:1; - unsigned int keep_locks:1; - unsigned int skip_locking:1; - unsigned int leave_spinning:1; unsigned int skip_check_block:1; }; diff --git a/extent-tree.c b/extent-tree.c index c46c92b..5443ec8 100644 --- a/extent-tree.c +++ b/extent-tree.c @@ -1418,7 +1418,6 @@ int btrfs_inc_extent_ref(struct btrfs_trans_handle *trans, return -ENOMEM; path->reada = 1; - path->leave_spinning = 1; ret = insert_inline_extent_backref(trans, root->fs_info->extent_root, path, bytenr, num_bytes, parent, @@ -1440,7 +1439,6 @@ int btrfs_inc_extent_ref(struct btrfs_trans_handle *trans, btrfs_release_path(path); path->reada = 1; - path->leave_spinning = 1; /* now insert the actual backref */ ret = insert_extent_backref(trans, root->fs_info->extent_root, @@ -2195,7 +2193,6 @@ static int __free_extent(struct btrfs_trans_handle *trans, return -ENOMEM; path->reada = 1; - path->leave_spinning = 1; is_data = owner_objectid >= BTRFS_FIRST_FREE_OBJECTID; if (is_data) @@ -2239,7 +2236,6 @@ static int __free_extent(struct btrfs_trans_handle *trans, is_data); BUG_ON(ret); btrfs_release_path(path); - path->leave_spinning = 1; key.objectid = bytenr; @@ -2304,7 +2300,6 @@ static int __free_extent(struct btrfs_trans_handle *trans, BUG_ON(ret < 0); btrfs_release_path(path); - path->leave_spinning = 1; key.objectid = bytenr; key.type = BTRFS_EXTENT_ITEM_KEY; @@ -2711,7 +2706,6 @@ static int alloc_reserved_tree_block(struct btrfs_trans_handle *trans, path = btrfs_alloc_path(); BUG_ON(!path); - path->leave_spinning = 1; ret = btrfs_insert_empty_item(trans, fs_info->extent_root, path, ins, size); BUG_ON(ret); -- 1.8.1.4 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 00/15] xfstests: new btrfs stress test cases
On Wed, Aug 20, 2014 at 11:24:37AM -0700, Zach Brown wrote: > On Thu, Aug 21, 2014 at 01:33:48AM +0800, Eryu Guan wrote: > > This patchset add new stress test cases for btrfs by running two > > different btrfs operations simultaneously under fsstress to ensure > > btrfs doesn't hang or oops in such situations. btrfs scrub and > > btrfs check will be run after each test. > > Cool. > > > The test matrix is the combination of 6 btrfs operations: > > > > balance > > create/mount/umount/delete subvolume > > replace device > > scrub > > defrag > > remount with different compress algorithms > > > > Short descriptions: > > > > 057: balance-subvolume > > 058: balance-scrub > > 059: balance-defrag > > 060: balance-remount > > 061: balance-replace > > 062: subvolume-replace > > 063: subvolume-scrub > > 064: subvolume-defrag > > 065: subvolume-remount > > 066: replace-scrub > > 067: replace-defrag > > 068: replace-remount > > 069: scrub-defrag > > 070: scrub-remount > > 071: defrag-remount > > But I'm not sure it should be built this way. > > At the very least each operation's implementation should be in a shared > function somewhere instead of being duplicated in each test. I was thinking about it too, my concern is that my test cases might be the only user of these shared functions, so I'm just not sure if it's good idea to share these functions just for my test cases. I'll share them in v2. > > But I don't think there should be a seperate test for each combination. > With a bit of fiddly bash you can automate generating unique > combinations of operations that are defined as functions in one test. Yes, that's just how it works in Red Hat internal test case, one test to generate all the combinations using shared functions. But I think that test time will be too long for a xfstests test case, and seperated and targeted test cases might be good idea for xfstests. Thanks for the review! Eryu > > btrfs_op_balance() > { > echo hi > } > > btrfs_op_scrub() > { > echo hi > } > > btrfs_op_defrag() > { > echo hi > } > > ops=($(declare -F | awk '/-f btrfs_op_/ {print $3}')) > nr=${#ops[@]} > > for i in $(seq 0 $((nr - 2))); do > for j in $(seq $((i + 1)) $((nr - 1))); do > echo ${ops[i]} ${ops[j]} > done > done > > Something like that. > > - z -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Significance of high number of mails on this list?
Hello. People on this list have been kind enough to reply to my technical questions. However, seeing the high number of mails on this list, esp with the title PATCH, I have a question about the development itself: Is this just an indication of a vibrant user/devel community [*] and healthy development of many new nice features to eventually come out in stable form later, or are we still at the fixing rough edges stage? IOW what is the proportion of commits adding new features to those stabilising/fixing features? [* Since there is no separate btrfs-users vs brtfs-dev I'm not able to gauge this difference either. i.e. if there were a dedicated -dev list I might not be alarmed by a high number of mails indicating fast development.] Mostly I have read like "BTRFS is mostly stable but there might be a few corner cases as yet unknown since this is a totally new generation of FSs". But still given the volume of mails here I wanted to ask... I'm sorry I realize I'm being a bit vague but I'm not sure how to exactly express what I'm feeling about BTRFS right now... -- Shriramana Sharma ஶ்ரீரமணஶர்மா श्रीरमणशर्मा -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs restore memory corruption (bug: 82701)
On Mon, 2014-08-18 at 11:25 +0200, Marc Dietrich wrote: > Hi, > > I did a checkout of the latest btrfs progs to repair my damaged filesystem. > Running btrfs restore gives me several failed to inflate: -6 and crashes with > some memory corruption. I ran it again with valgrind and got: > > valgrind --log-file=x2 -v --leak-check=yes btrfs restore /dev/sda9 /mnt/backup > > ==8528== Memcheck, a memory error detector > ==8528== Copyright (C) 2002-2012, and GNU GPL'd, by Julian Seward et al. > ==8528== Using Valgrind-3.8.1 and LibVEX; rerun with -h for copyright info > ==8528== Command: btrfs restore /dev/sda9 /mnt/backup > ==8528== Parent PID: 8453 > ==8528== > ==8528== Syscall param pwrite64(buf) points to uninitialised byte(s) Hi, Marc For this one, It is because we use malloc to alloc space for buf which contains the stuff we are going to write out. But the actual length of the output stuff is shorter than sizeof(buf). To deal with this piece, I think use calloc instead of malloc will clear this WARNING away. I will send a patch for this. -Gui > ==8528==at 0x59BE3C3: __pwrite_nocancel (in /lib64/libpthread-2.18.so) > ==8528==by 0x41F22F: search_dir (cmds-restore.c:392) > ==8528==by 0x41F8D0: search_dir (cmds-restore.c:895) > ==8528==by 0x41F8D0: search_dir (cmds-restore.c:895) > ==8528==by 0x41F8D0: search_dir (cmds-restore.c:895) > ==8528==by 0x41F8D0: search_dir (cmds-restore.c:895) > ==8528==by 0x41F8D0: search_dir (cmds-restore.c:895) > ==8528==by 0x4204B8: cmd_restore (cmds-restore.c:1284) > ==8528==by 0x4043FE: main (btrfs.c:286) > ==8528== Address 0x66956a0 is 7,056 bytes inside a block of size 8,192 > alloc'd > ==8528==at 0x4C277AB: malloc (in /usr/lib64/valgrind/vgpreload_memcheck- > amd64-linux.so) > ==8528==by 0x41EEAD: search_dir (cmds-restore.c:316) > ==8528==by 0x41F8D0: search_dir (cmds-restore.c:895) > ==8528==by 0x41F8D0: search_dir (cmds-restore.c:895) > ==8528==by 0x41F8D0: search_dir (cmds-restore.c:895) > ==8528==by 0x41F8D0: search_dir (cmds-restore.c:895) > ==8528==by 0x41F8D0: search_dir (cmds-restore.c:895) > ==8528==by 0x4204B8: cmd_restore (cmds-restore.c:1284) > ==8528==by 0x4043FE: main (btrfs.c:286) > ==8528== > ==8528== Invalid read of size 1 > ==8528==at 0x4C2BF15: memcpy@@GLIBC_2.14 (in > /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) > ==8528==by 0x43818F: read_extent_buffer (string3.h:51) > ==8528==by 0x41EC66: search_dir (cmds-restore.c:233) > ==8528==by 0x41F8D0: search_dir (cmds-restore.c:895) > ==8528==by 0x41F8D0: search_dir (cmds-restore.c:895) > ==8528==by 0x41F8D0: search_dir (cmds-restore.c:895) > ==8528==by 0x41F8D0: search_dir (cmds-restore.c:895) > ==8528==by 0x41F8D0: search_dir (cmds-restore.c:895) > ==8528==by 0x4204B8: cmd_restore (cmds-restore.c:1284) > ==8528==by 0x4043FE: main (btrfs.c:286) > ==8528== Address 0x684c186 is 1,110 bytes inside a block of size 4,224 free'd > ==8528==at 0x4C28ADC: free (in /usr/lib64/valgrind/vgpreload_memcheck- > amd64-linux.so) > ==8528==by 0x437895: free_extent_buffer (extent_io.c:618) > ==8528==by 0x41E053: next_leaf (cmds-restore.c:202) > ==8528==by 0x41E50F: search_dir (cmds-restore.c:731) > ==8528==by 0x41F8D0: search_dir (cmds-restore.c:895) > ==8528==by 0x41F8D0: search_dir (cmds-restore.c:895) > ==8528==by 0x41F8D0: search_dir (cmds-restore.c:895) > ==8528==by 0x41F8D0: search_dir (cmds-restore.c:895) > ==8528==by 0x41F8D0: search_dir (cmds-restore.c:895) > ==8528==by 0x4204B8: cmd_restore (cmds-restore.c:1284) > ==8528==by 0x4043FE: main (btrfs.c:286) > ==8528== > ==8528== Invalid read of size 8 > ==8528==at 0x4C2BF40: memcpy@@GLIBC_2.14 (in > /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) > ==8528==by 0x43818F: read_extent_buffer (string3.h:51) > ==8528==by 0x41EC66: search_dir (cmds-restore.c:233) > ==8528==by 0x41F8D0: search_dir (cmds-restore.c:895) > ==8528==by 0x41F8D0: search_dir (cmds-restore.c:895) > ==8528==by 0x41F8D0: search_dir (cmds-restore.c:895) > ==8528==by 0x41F8D0: search_dir (cmds-restore.c:895) > ==8528==by 0x41F8D0: search_dir (cmds-restore.c:895) > ==8528==by 0x4204B8: cmd_restore (cmds-restore.c:1284) > ==8528==by 0x4043FE: main (btrfs.c:286) > ==8528== Address 0x684c178 is 1,096 bytes inside a block of size 4,224 free'd > ==8528==at 0x4C28ADC: free (in /usr/lib64/valgrind/vgpreload_memcheck- > amd64-linux.so) > ==8528==by 0x437895: free_extent_buffer (extent_io.c:618) > ==8528==by 0x41E053: next_leaf (cmds-restore.c:202) > ==8528==by 0x41E50F: search_dir (cmds-restore.c:731) > ==8528==by 0x41F8D0: search_dir (cmds-restore.c:895) > ==8528==by 0x41F8D0: search_dir (cmds-restore.c:895) > ==8528==by 0x41F8D0: search_dir (cmds-restore.c:895) > ==8528==by 0x41F8D0: search_dir (cmds-restore.c:895) > ==8528==by 0x41F8D0: search_dir (cmds-r
[PATCH] btrfs-progs: init uninitialized output buf for btrfs-restore
A memory problem reported by valgrind as follows: === Syscall param pwrite64(buf) points to uninitialised byte(s) When running: # valgrind --leak-check=yes btrfs restore /dev/sda9 /mnt/backup Because the output buf size is alloced with malloc, but the length of output data is shorter than the sizeof(buf), so valgrind report uninitialised byte(s). We could use calloc to repalce malloc and clear this WARNING away. Reported-by: Marc Dietrich Signed-off-by: Gui Hecheng --- cmds-restore.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/cmds-restore.c b/cmds-restore.c index cbda6bb..bb72311 100644 --- a/cmds-restore.c +++ b/cmds-restore.c @@ -251,7 +251,7 @@ static int copy_one_inline(int fd, struct btrfs_path *path, u64 pos) } ram_size = btrfs_file_extent_ram_bytes(leaf, fi); - outbuf = malloc(ram_size); + outbuf = calloc(1, ram_size); if (!outbuf) { fprintf(stderr, "No memory\n"); return -ENOMEM; @@ -320,7 +320,7 @@ static int copy_one_extent(struct btrfs_root *root, int fd, } if (compress != BTRFS_COMPRESS_NONE) { - outbuf = malloc(ram_size); + outbuf = calloc(1, ram_size); if (!outbuf) { fprintf(stderr, "No memory\n"); free(inbuf); -- 1.8.1.4 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 01/15] btrfs: new test to run btrfs balance and subvolume test simultaneously
On Thu, Aug 21, 2014 at 10:04:30AM +0800, Qu Wenruo wrote: > > Original Message > Subject: [PATCH 01/15] btrfs: new test to run btrfs balance and subvolume > test simultaneously > From: Eryu Guan > To: > Date: 2014年08月21日 01:33 > >Run btrfs balance and subvolume create/mount/umount/delete simultaneously, > >with fsstress running in background. > > > >Signed-off-by: Eryu Guan > >--- > > tests/btrfs/057 | 147 > > > > tests/btrfs/057.out | 2 + > > tests/btrfs/group | 1 + > > 3 files changed, 150 insertions(+) > > create mode 100755 tests/btrfs/057 > > create mode 100644 tests/btrfs/057.out > > > >diff --git a/tests/btrfs/057 b/tests/btrfs/057 > >new file mode 100755 > >index 000..2f507a7 > >--- /dev/null > >+++ b/tests/btrfs/057 > >@@ -0,0 +1,147 @@ > >+#! /bin/bash > >+# FSQA Test No. btrfs/057 > >+# > >+# Run btrfs balance and subvolume create/mount/umount/delete simultaneously, > >+# with fsstress running in background. > >+# > >+#--- > >+# Copyright (C) 2014 Red Hat Inc. All rights reserved. > >+# > >+# This program is free software; you can redistribute it and/or > >+# modify it under the terms of the GNU General Public License as > >+# published by the Free Software Foundation. > >+# > >+# This program is distributed in the hope that it would be useful, > >+# but WITHOUT ANY WARRANTY; without even the implied warranty of > >+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > >+# GNU General Public License for more details. > >+# > >+# You should have received a copy of the GNU General Public License > >+# along with this program; if not, write the Free Software Foundation, > >+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA > >+# > >+#--- > >+# > >+ > >+seq=`basename $0` > >+seqres=$RESULT_DIR/$seq > >+echo "QA output created by $seq" > >+ > >+here=`pwd` > >+tmp=/tmp/$$ > >+status=1 > >+trap "_cleanup; exit \$status" 0 1 2 3 15 > >+ > >+_cleanup() > >+{ > >+cd / > >+rm -fr $tmp.* > >+} > >+ > >+# get standard environment, filters and checks > >+. ./common/rc > >+. ./common/filter > >+ > >+# real QA test starts here > >+_supported_fs btrfs > >+_supported_os Linux > >+_require_scratch > >+_require_scratch_dev_pool 4 > >+ > >+rm -f $seqres.full > >+ > >+# test case array > >+tcs=( > >+"-m single -d single" > >+"-m dup -d single" > >+"-m raid0 -d raid0" > >+"-m raid1 -d raid0" > >+"-m raid1 -d raid1" > >+"-m raid10 -d raid10" > >+"-m raid5 -d raid5" > >+"-m raid6 -d raid6" > >+) > I wonder should we add the mkfs options there. > Since xfstests already use environment MKFS_OPTIONS to do mkfs, > if really need to test all mkfs options, IMO it is better to change > MKFS_OPTIONS on each test round. Most of the data/metadata profiles tested here require multiple devices, so if you set MKFS_OPTIONS to, say "-m raid10 -d raid10", other tests that only require a single scratch device will fail at mkfs time. Unlike other mkfs options, these options are not working for every test. And btrfs/011 and btrfs/023 do the test in a similar way, so I just followed this way :) > > >+ > >+run_test() > >+{ > >+local mkfs_opts=$1 > >+local saved_mkfs_opts=$MKFS_OPTIONS > >+local subvol_mnt=$tmp.mnt > >+ > >+echo "Test $mkfs_opts" >>$seqres.full > >+ > >+MKFS_OPTIONS="$MKFS_OPTIONS $mkfs_opts" > >+# dup only works on single device > >+if [[ "$mkfs_opts" =~ dup ]]; then > >+_scratch_mkfs >>$seqres.full 2>&1 > >+else > >+_scratch_pool_mkfs >>$seqres.full 2>&1 > >+fi > >+ret=$? > >+MKFS_OPTIONS=$saved_mkfs_opts > >+# make sure we created btrfs with desired options > >+if [ $ret -ne 0 ]; then > >+echo "mkfs $mkfs_opts failed" > >+return > >+fi > >+ > >+_scratch_mount >>$seqres.full 2>&1 > >+ > >+args=`_scale_fsstress_args -p 20 -n 100 $FSSTRESS_AVOID -d > >$SCRATCH_MNT/stressdir` > >+echo "Run fsstress $args" >>$seqres.full > >+$FSSTRESS_PROG $args >/dev/null 2>&1 & > >+fsstress_pid=$! > >+ > >+echo -n "Start balance worker: " >>$seqres.full > >+( > >+while true; do > >+$BTRFS_UTIL_PROG balance start $SCRATCH_MNT > >+done > >+) >/dev/null 2>&1 & > >+balance_pid=$! > >+echo "$balance_pid" >>$seqres.full > >+ > >+echo -n "Start subvolume worker: " >>$seqres.full > >+mkdir -p $subvol_mnt > >+( > >+while true; do > >+$BTRFS_UTIL_PROG subvolume create $SCRATCH_MNT/subvol_$$ > >+$MOUNT_PROG -o subvol=subvol_$$ $SCRATCH_DEV $subvol_mnt > >+$UMOUNT_PROG $subvol_mnt > >+$BTRFS_UTIL_PROG subvolume delete $SCRATCH_MNT/subvol_$$ > >+done > >+
btrfs restore
Hello, I'm create btrfs multi-device volume: (2TB * 5disks) 1. one device was converted from ext3, then delete sub-volume 'ext2'; 2. 'btrfs device add /dev/sd* /mnt/sdc1'; and some times again, Every disk adds by one, then move data from other disks. 2 times was running 'btrfs fi defrag -r -v -t 700K /mnt/sdc1; btrfs balance start /mnt/sdc1; btrfs scrub start /mnt/sdc1'. Finally I have FS at 5 devices. After, go to reboot pc, and I seen that 1 drive have no any partition. I'm full of sadness. I think that device was not correctly unmounted and one disk fail. mount -t btrfs -o degreded,ro,recovery,nospace_cache ... mount -t btrfs -o recovery,nospace_cache ... btrfs-find-root, btrfs-zero-log .. Finaly: btrfs restore -t 10404875644928 -v -i /dev/sda1 /mnt/uh1/eric/ - Success! but some directory is absent; 3.4 TB restored, but 4 (available disks)*1.82 Tb = 7.28 TB; and 7.28-3.4 = about 3.8 TB which is missing . Where is it ? How can i get it back? How can i try to restore more data? Please advice. _ mount -t btrfs -o degraded,ro,recovery,nospace_cache /dev/sda1 /mnt/tmp3; dmesg | tail [11774.958996] BTRFS info (device sde1): allowing degraded mounts [11774.959001] BTRFS info (device sde1): enabling auto recovery [11774.959003] BTRFS info (device sde1): disabling disk space caching [11774.959955] BTRFS warning (device sde1): devid 5 missing [11774.986391] BTRFS: failed to read tree root on sde1 [11774.986408] BTRFS: failed to read tree root on sde1 [11774.986456] BTRFS: failed to read tree root on sde1 [11774.986473] BTRFS: failed to read tree root on sde1 [11774.986481] BTRFS: failed to read tree root on sde1 [11775.014471] BTRFS: open_ctree failed # btrfs-zero-log /dev/sda1 warning devid 5 not found already Check tree block failed, want=16845270495232, have=0 read block failed check_tree_block Couldn't read tree root __ # uname -a Linux oit-875u 3.16.0-031600rc7-generic #201407271635 SMP Sun Jul 27 20:36:48 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux # btrfs fi show Label: none uuid: 841f1b5c-adb2-485f-a4cf-83bbf880dced Total devices 5 FS bytes used 9.09TiB devid1 size 1.82TiB used 1.82TiB path /dev/sda1 devid2 size 1.82TiB used 1.82TiB path /dev/sdc1 devid3 size 1.82TiB used 1.82TiB path /dev/sdd1 devid4 size 1.82TiB used 1.82TiB path /dev/sde1 *** Some devices missing Btrfs v3.14.1 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [BUG] cannot mount subvolume with selinux context
Original Message Subject: Re: [BUG] cannot mount subvolume with selinux context From: Eryu Guan To: Zach Brown Date: 2014年08月20日 11:57 On Tue, Aug 19, 2014 at 10:28:54AM -0700, Zach Brown wrote: On Tue, Aug 19, 2014 at 11:32:16AM +0800, Eryu Guan wrote: Hi, Description of the problem: mount btrfs with selinux context, then create a subvolume, the new subvolume cannot be mounted, even with the same context. mkfs -t btrfs /dev/sda5 mount -o context=system_u:object_r:nfs_t:s0 /dev/sda5 /mnt/btrfs btrfs subvolume create /mnt/btrfs/subvol mount -o subvol=subvol,context=system_u:object_r:nfs_t:s0 /dev/sda5 /mnt/test Submit a xfstest? Sure, will do. Thanks, Eryu The security_sb_copy_data() takes out selinux context data to "secdata", then mount_subvol() calls mount_fs() (via vfs_kern_mount()) again without selinux context, so mount_subvol() fails, which fails the whole mount. Not sure what's the proper fix. Zach suggestted that the fix will probably be to rework the vfs functions a bit as he said in rh bugzilla[1]. Yeah, I have no idea what'd be preferred here: - rework the vfs _kern_ mount api to offer one that doesn't mess with selinux mount options - add a flag to have the second _kern_ mount ignore selinux (but not MS_KERNMOUNT?) - binary data and fs selinux handling? (like nfs) In fact, we can just make btrfs deal with "subvol=" mount option in a new method. Current, btrfs handle "subvol=" by call vfs_kern_mount again and use vfs level mount_subtree() to do the path search thing. But on the other hand, btrfs does not call vfs_kern_mount() when handling default subvolume or "subvolid=" mount, so, I think we can do all the path search inside btrfs instead of reuse vfs level functions, and convert "subvol=" mount option to "subvolid=", which should be selinux friendly now. (And in this method mount_subvol() should be called just before get_default_root()). If I am wrong, please tell me. BTW, it seems that if mainline kernel accept the patchset which convert "subvolid=" to "subvol=", it will make the bug more seriously. :-( Thank goddness, the successor patch uses get_path() Thanks, Qu - z -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html