date:20131028

Re: No space left on device, problem

2013-10-28 Thread Igor M

On Mon, Oct 28, 2013 at 12:27 AM, Chris Murphy li...@colorremedies.com wrote:

 On Oct 27, 2013, at 4:46 PM, Chris Murphy li...@colorremedies.com wrote:


 On Oct 27, 2013, at 3:53 PM, Igor M igor...@gmail.com wrote:

 I made some more tests. Disk is 3TB, first cca 225GB is copied without 
 errors.
 Then errors 'No space left on device' begins.

 Post the full entire dmesg somewhere please. pastebin.com is one option.


 And on list or pasted, the output for the disk from:

 smartctl -x /dev/sdX


dmesg: http://pastebin.com/t2H1QYye
source disk: http://pastebin.com/JqKxkxKr
dest disk: http://pastebin.com/ez9jALS2
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

kernel BUG at fs/btrfs/inode.c:1595!

2013-10-28 Thread Tomasz Chmielewski

I seem to be hitting this only when creating snapshots (in addition to
other fs activity, like adding, removing, modifying multiple files at
different speeds at once, running sync etc.).

Before kernel BUG at fs/btrfs/inode.c:1595! was reported, I could also see 
the following:

BTRFS critical (device sdb5): unable to find logical 8091413542285438976 len 
4096

btrfs bad fsid on block 0


The filesystem shows no more activity after that (hangs).

reboot won't reboot the server (would be waiting for hanged processes 
accessing the fs), I have to echo b to sysrq-trigger to recover.

This is with 3.12.0-rc7.

The fs passes scrub without showing any errors; memtest passes memory test.


Here is a full trace, the fs lasted just 2 h after the last similar hang:

[ 7921.918722] [ cut here ]
[ 7921.918781] WARNING: CPU: 3 PID: 11764 at fs/btrfs/ctree.c:1322 
btrfs_search_old_slot+0x338/0x81d [btrfs]()
[ 7921.918872] Modules linked in: veth ipt_MASQUERADE iptable_nat 
nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables 
x_tables cpufreq_ondemand cpufreq_conservative cpufreq_powersave cpufreq_stats 
bridge stp llc ipv6 btrfs xor raid6_pq zlib_deflate loop button i2c_i801 video 
pcspkr acpi_cpufreq freq_table i2c_core ehci_pci ehci_hcd lpc_ich mfd_core ext4 
crc16 jbd2 mbcache raid1 sg sd_mod ahci libahci libata scsi_mod r8169 mii
[ 7921.920316] CPU: 3 PID: 11764 Comm: btrfs-endio-wri Not tainted 3.12.0-rc7 #1
[ 7921.920365] Hardware name: System manufacturer System Product Name/P8H77-M 
PRO, BIOS 1101 02/04/2013
[ 7921.920455]  0009 8805018877e8 8138176b 
0006
[ 7921.920545]   880501887828 810366a9 
020d25bf8000
[ 7921.920635]  a023ae23 8807f0010090 880036e08800 
8802c9f09e00
[ 7921.920725] Call Trace:
[ 7921.920773]  [8138176b] dump_stack+0x46/0x58
[ 7921.920821]  [810366a9] warn_slowpath_common+0x77/0x91
[ 7921.920873]  [a023ae23] ? btrfs_search_old_slot+0x338/0x81d [btrfs]
[ 7921.920923]  [810366d8] warn_slowpath_null+0x15/0x17
[ 7921.920973]  [a023ae23] btrfs_search_old_slot+0x338/0x81d [btrfs]
[ 7921.921029]  [a029e6e8] __resolve_indirect_refs+0x10f/0x481 [btrfs]
[ 7921.921085]  [a026fdf3] ? free_extent_buffer+0x77/0x7c [btrfs]
[ 7921.921139]  [a029f0dd] find_parent_nodes+0x33b/0x5d6 [btrfs]
[ 7921.921193]  [a029f40b] btrfs_find_all_roots+0x7c/0xd7 [btrfs]
[ 7921.921246]  [a02a0300] ? ulist_add_merge+0x7e/0x14e [btrfs]
[ 7921.921300]  [a02a2562] btrfs_qgroup_account_ref+0x109/0x40d 
[btrfs]
[ 7921.921391]  [a0242805] 
btrfs_delayed_refs_qgroup_accounting+0xac/0xe0 [btrfs]
[ 7921.921485]  [a0256475] __btrfs_end_transaction+0x51/0x2bd [btrfs]
[ 7921.921539]  [a025670c] btrfs_end_transaction+0xb/0xd [btrfs]
[ 7921.921593]  [a025f95d] relink_extent_backref+0x68a/0x6cf [btrfs]
[ 7921.921646]  [a0260185] btrfs_finish_ordered_io+0x7e3/0x8e4 [btrfs]
[ 7921.921700]  [a0260296] finish_ordered_fn+0x10/0x12 [btrfs]
[ 7921.921754]  [a027b322] worker_loop+0x15e/0x494 [btrfs]
[ 7921.921806]  [a027b1c4] ? btrfs_queue_worker+0x26d/0x26d [btrfs]
[ 7921.921856]  [8104e6be] kthread+0xb5/0xbd
[ 7921.921903]  [8104e609] ? kthread_freezable_should_stop+0x43/0x43
[ 7921.921952]  [813864fc] ret_from_fork+0x7c/0xb0
[ 7921.922000]  [8104e609] ? kthread_freezable_should_stop+0x43/0x43
[ 7921.922057] ---[ end trace c54471a3030c30db ]---
[ 7922.511022] btrfs bad fsid on block 0
[ 7922.57] [ cut here ]
[ 7922.511173] WARNING: CPU: 1 PID: 11764 at fs/btrfs/ctree.c:1322 
btrfs_search_old_slot+0x338/0x81d [btrfs]()
[ 7922.511264] Modules linked in: veth ipt_MASQUERADE iptable_nat 
nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables 
x_tables cpufreq_ondemand cpufreq_conservative cpufreq_powersave cpufreq_stats 
bridge stp llc ipv6 btrfs xor raid6_pq zlib_deflate loop button i2c_i801 video 
pcspkr acpi_cpufreq freq_table i2c_core ehci_pci ehci_hcd lpc_ich mfd_core ext4 
crc16 jbd2 mbcache raid1 sg sd_mod ahci libahci libata scsi_mod r8169 mii
[ 7922.511576] CPU: 1 PID: 11764 Comm: btrfs-endio-wri Tainted: GW
3.12.0-rc7 #1
[ 7922.511665] Hardware name: System manufacturer System Product Name/P8H77-M 
PRO, BIOS 1101 02/04/2013
[ 7922.511788]  0009 8805018877e8 8138176b 
0006
[ 7922.511895]   880501887828 810366a9 
020d25bf8000
[ 7922.511984]  a023ae23 8807f0010090 880036e08800 
8802c9f09e00
[ 7922.512073] Call Trace:
[ 7922.512119]  [8138176b] dump_stack+0x46/0x58
[ 7922.512168]  [810366a9] warn_slowpath_common+0x77/0x91
[ 7922.512218]  [a023ae23] ? btrfs_search_old_slot+0x338/0x81d [btrfs]
[ 7922.512268]  [810366d8] warn_slowpath_null+0x15/0x17
[

Re: Why cannot I move a read-only snapshot around?

2013-10-28 Thread Karl Kiniger

On Sat 131026, Christian Robert wrote:
 you can change a ro snapshot into a rw snapshot

 you just snapshot it without the -r option

Yes, I know but the new rw snapshots are not really identical
in terms of ID, generation count and UUID (the file contents will
remain the same of course).

The question is how this will affect btrfs send and receive,
especially considering the -p and -c option.

Karl

--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH] Btrfs: take ordered root lock when removing ordered operations inode

2013-10-28 Thread Josef Bacik

A user reported a list corruption warning from btrfs_remove_ordered_extent, it
is because we aren't taking the ordered_root_lock when we remove the inode from
the ordered operations list.  Thanks,

Signed-off-by: Josef Bacik jba...@fusionio.com
---
 fs/btrfs/ordered-data.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/fs/btrfs/ordered-data.c b/fs/btrfs/ordered-data.c
index bbb1a38..8a5eff3 100644
--- a/fs/btrfs/ordered-data.c
+++ b/fs/btrfs/ordered-data.c
@@ -537,7 +537,9 @@ void btrfs_remove_ordered_extent(struct inode *inode,
 */
if (RB_EMPTY_ROOT(tree-tree) 
!mapping_tagged(inode-i_mapping, PAGECACHE_TAG_DIRTY)) {
+   spin_lock(root-fs_info-ordered_root_lock);
list_del_init(BTRFS_I(inode)-ordered_operations);
+   spin_unlock(root-fs_info-ordered_root_lock);
}
 
if (!root-nr_ordered_extents) {
-- 
1.8.3.1

--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] Btrfs-progs: allow --init-extent-tree to work when extent tree is borked

2013-10-28 Thread Josef Bacik

On Sun, Oct 27, 2013 at 12:16:12AM +0100, Martin wrote:
 On 25/10/13 19:31, Josef Bacik wrote:
  On Fri, Oct 25, 2013 at 07:27:24PM +0100, Martin wrote:
  On 25/10/13 19:01, Josef Bacik wrote:
  Unfortunately you can't run --init-extent-tree if you can't actually read 
  the
  extent root.  Fix this by allowing partial starts with no extent root and 
  then
  have fsck only check to see if the extent root is uptodate _after_ the 
  check to
  see if we are init'ing the extent tree.  Thanks,
 
  Signed-off-by: Josef Bacik jba...@fusionio.com
  ---
   cmds-check.c |  9 ++---
   disk-io.c| 16 ++--
   2 files changed, 20 insertions(+), 5 deletions(-)
 
  diff --git a/cmds-check.c b/cmds-check.c
  index 69b0327..8ed7baa 100644
  --- a/cmds-check.c
  +++ b/cmds-check.c
 
  Hey! Quick work!...
 
  Is that worth patching locally and trying against my example?
 
  
  Yes, I'm a little worried about your particular case so I'd like to see if 
  it
  works.  If you don't see a lot of output after say 5 minutes let's assume I
  didn't fix your problem and let me know so I can make the other change I
  considered.  Thanks,
 
 Nope... No-go.
 

Ok I've sent

[PATCH] Btrfs-progs: rework open_ctree to take flags, add a new one

which should address your situation.  Thanks,

Josef
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH] Btrfs-progs: rework open_ctree to take flags, add a new one

2013-10-28 Thread Josef Bacik

So I needed to add a flag to not try to read block groups when doing
--init-extent-tree since we could hang there, but that meant adding a whole
other 0/1 type flag to open_ctree_fs_info.  So instead I've converted it all
over to using a flags setting and added the flag that I needed.  This has been
tested with xfstests and make test.  Thanks,

Signed-off-by: Josef Bacik jba...@fusionio.com
---
 btrfs-convert.c   |   6 +--
 btrfs-corrupt-block.c |   2 +-
 btrfs-debug-tree.c|   2 +-
 btrfs-image.c |   9 -
 btrfs-zero-log.c  |   2 +-
 btrfstune.c   |   2 +-
 cmds-check.c  |  14 +++
 disk-io.c | 107 ++
 disk-io.h |  26 ++--
 mkfs.c|   2 +-
 quick-test.c  |  10 ++---
 super-recover.c   |   4 +-
 utils.c   |   2 +-
 13 files changed, 75 insertions(+), 113 deletions(-)

diff --git a/btrfs-convert.c b/btrfs-convert.c
index 26c7b5f..ae10eed 100644
--- a/btrfs-convert.c
+++ b/btrfs-convert.c
@@ -2256,7 +2256,7 @@ static int do_convert(const char *devname, int datacsum, 
int packing,
fprintf(stderr, unable to update system chunk\n);
goto fail;
}
-   root = open_ctree_fd(fd, devname, super_bytenr, O_RDWR);
+   root = open_ctree_fd(fd, devname, super_bytenr, OPEN_CTREE_WRITES);
if (!root) {
fprintf(stderr, unable to open ctree\n);
goto fail;
@@ -2317,7 +2317,7 @@ static int do_convert(const char *devname, int datacsum, 
int packing,
goto fail;
}
 
-   root = open_ctree_fd(fd, devname, 0, O_RDWR);
+   root = open_ctree_fd(fd, devname, 0, OPEN_CTREE_WRITES);
if (!root) {
fprintf(stderr, unable to open ctree\n);
goto fail;
@@ -2418,7 +2418,7 @@ static int do_rollback(const char *devname, int force)
fprintf(stderr, unable to open %s\n, devname);
goto fail;
}
-   root = open_ctree_fd(fd, devname, 0, O_RDWR);
+   root = open_ctree_fd(fd, devname, 0, OPEN_CTREE_WRITES);
if (!root) {
fprintf(stderr, unable to open ctree\n);
goto fail;
diff --git a/btrfs-corrupt-block.c b/btrfs-corrupt-block.c
index b40a529..f0c14a9 100644
--- a/btrfs-corrupt-block.c
+++ b/btrfs-corrupt-block.c
@@ -799,7 +799,7 @@ int main(int ac, char **av)
radix_tree_init();
cache_tree_init(root_cache);
 
-   root = open_ctree(dev, 0, 1);
+   root = open_ctree(dev, 0, OPEN_CTREE_WRITES);
if (!root) {
fprintf(stderr, Open ctree failed\n);
exit(1);
diff --git a/btrfs-debug-tree.c b/btrfs-debug-tree.c
index 4a9d89d..4a83770 100644
--- a/btrfs-debug-tree.c
+++ b/btrfs-debug-tree.c
@@ -171,7 +171,7 @@ int main(int ac, char **av)
if (ac != 1)
print_usage();
 
-   info = open_ctree_fs_info(av[optind], 0, 0, 0, 1, 0);
+   info = open_ctree_fs_info(av[optind], 0, 0, OPEN_CTREE_PARTIAL);
if (!info) {
fprintf(stderr, unable to open %s\n, av[optind]);
exit(1);
diff --git a/btrfs-image.c b/btrfs-image.c
index 40ed483..7bcfc06 100644
--- a/btrfs-image.c
+++ b/btrfs-image.c
@@ -2267,7 +2267,10 @@ static int __restore_metadump(const char *input, FILE 
*out, int old_restore,
/* NOTE: open with write mode */
if (fixup_offset) {
BUG_ON(!target);
-   info = open_ctree_fs_info_restore(target, 0, 0, 1, 1);
+   info = open_ctree_fs_info(target, 0, 0,
+ OPEN_CTREE_WRITES |
+ OPEN_CTREE_RESTORE |
+ OPEN_CTREE_PARTIAL);
if (!info) {
fprintf(stderr, %s: open ctree failed\n, __func__);
ret = -EIO;
@@ -2555,7 +2558,9 @@ int main(int argc, char *argv[])
u64 total_devs;
int i;
 
-   info = open_ctree_fs_info_restore(target, 0, 0, 0, 1);
+   info = open_ctree_fs_info(target, 0, 0,
+ OPEN_CTREE_PARTIAL |
+ OPEN_CTREE_RESTORE);
if (!info) {
int e = errno;
fprintf(stderr, unable to open %s error = %s\n,
diff --git a/btrfs-zero-log.c b/btrfs-zero-log.c
index 432adff..ab7f418 100644
--- a/btrfs-zero-log.c
+++ b/btrfs-zero-log.c
@@ -60,7 +60,7 @@ int main(int ac, char **av)
goto out;
}
 
-   root = open_ctree(av[1], 0, 1);
+   root = open_ctree(av[1], 0, OPEN_CTREE_WRITES);
 
if (root == NULL)
return 1;
diff --git a/btrfstune.c b/btrfstune.c
index 1cf6a68..50724ba 100644
--- a/btrfstune.c
+++ b/btrfstune.c
@@ -148,7 +148,7 @@ int main(int argc, char *argv[])

[PATCH] xfstests: add generic/320 to test fsync() on directories V2

2013-10-28 Thread Josef Bacik

Btrfs had some issues with fsync()'ing directories and fsync()'ing after
renames.  These three new tests cover the 3 different issues we were seeing.
This breaks out the dmflakey stuff into a common helper to be shared between
generic/311 and generic/320. Thanks,

Signed-off-by: Josef Bacik jba...@fusionio.com
---
V1-V2: moved this out into its own test instead of adding it to 311

 common/dmflakey   |  74 ++
 tests/generic/311 |  56 +++-
 tests/generic/320 | 140 ++
 tests/generic/320.out |   9 
 tests/generic/group   |   1 +
 5 files changed, 231 insertions(+), 49 deletions(-)
 create mode 100644 common/dmflakey
 create mode 100644 tests/generic/320
 create mode 100644 tests/generic/320.out

diff --git a/common/dmflakey b/common/dmflakey
new file mode 100644
index 000..bc984bd
--- /dev/null
+++ b/common/dmflakey
@@ -0,0 +1,74 @@
+##/bin/bash
+#
+# Copyright (c) 2013 Fusion IO, Inc.  All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#
+#
+# common functions for setting up and tearing down a dmflakey device
+
+FLAKEY_ALLOW_WRITES=0
+FLAKEY_DROP_WRITES=1
+
+_init_flakey()
+{
+   local BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
+   FLAKEY_DEV=/dev/mapper/flakey-test
+   FLAKEY_TABLE=0 $BLK_DEV_SIZE flakey $SCRATCH_DEV 0 180 0
+   FLAKEY_TABLE_DROP=0 $BLK_DEV_SIZE flakey $SCRATCH_DEV 0 0 180 1 
drop_writes
+   $DMSETUP_PROG create flakey-test --table $FLAKEY_TABLE || \
+   _fatal failed to create flakey device
+}
+
+_mount_flakey()
+{
+   mount -t $FSTYP $MOUNT_OPTIONS $FLAKEY_DEV $SCRATCH_MNT
+}
+
+_unmount_flakey()
+{
+   $UMOUNT_PROG $SCRATCH_MNT
+}
+
+_cleanup_flakey()
+{
+   # If dmsetup load fails then we need to make sure to do resume here
+   # otherwise the umount will hang
+   $DMSETUP_PROG resume flakey-test  /dev/null 21
+   $UMOUNT_PROG $SCRATCH_MNT  /dev/null 21
+   $DMSETUP_PROG remove flakey-test  /dev/null 21
+}
+
+# _load_flakey_table table [lockfs]
+#
+# This defaults to --nolockfs, which doesn't freeze_fs() before loading the new
+# table, so it simulates power failure.
+_load_flakey_table()
+{
+
+   table=$FLAKEY_TABLE
+   [ $1 -eq $FLAKEY_DROP_WRITES ]  table=$FLAKEY_TABLE_DROP
+
+   suspend_opt=--nolockfs
+   [ $# -gt 1 ]  [ $2 -eq 1 ]  suspend_opt=
+
+   $DMSETUP_PROG suspend $suspend_opt flakey-test
+   [ $? -ne 0 ]  _fatal failed to suspend flakey-test
+
+   $DMSETUP_PROG load flakey-test --table $table
+   [ $? -ne 0 ]  _fatal failed to load table into flakey-test
+
+   $DMSETUP_PROG resume flakey-test
+   [ $? -ne 0 ]  _fatal  failed to resumeflakey-test
+}
diff --git a/tests/generic/311 b/tests/generic/311
index 675d927..6802a96 100644
--- a/tests/generic/311
+++ b/tests/generic/311
@@ -41,17 +41,14 @@ status=1# failure is the default!
 
 _cleanup()
 {
-   # If dmsetup load fails then we need to make sure to do resume here
-   # otherwise the umount will hang
-   $DMSETUP_PROG resume flakey-test  /dev/null 21
-   $UMOUNT_PROG $SCRATCH_MNT  /dev/null 21
-   $DMSETUP_PROG remove flakey-test  /dev/null 21
+   _cleanup_flakey
 }
 trap _cleanup; exit \$status 0 1 2 3 15
 
 # get standard environment, filters and checks
 . ./common/rc
 . ./common/filter
+. ./common/dmflakey
 
 # real QA test starts here
 _supported_fs generic
@@ -63,49 +60,12 @@ _require_dm_flakey
 [ -x $here/src/fsync-tester ] || _notrun fsync-tester not build
 
 rm -f $seqres.full
-BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
-FLAKEY_DEV=/dev/mapper/flakey-test
 SEED=1
 testfile=$SCRATCH_MNT/$seq.fsync
-FLAKEY_TABLE=0 $BLK_DEV_SIZE flakey $SCRATCH_DEV 0 180 0
-FLAKEY_TABLE_DROP=0 $BLK_DEV_SIZE flakey $SCRATCH_DEV 0 0 180 1 drop_writes
-_TEST_OPTIONS=
-
-_mount_flakey()
-{
-   mount -t $FSTYP $MOUNT_OPTIONS $FLAKEY_DEV $SCRATCH_MNT
-}
-
-_unmount_flakey()
-{
-   $UMOUNT_PROG $SCRATCH_MNT
-}
-
-_load_flakey_table()
-{
-   # _load_flakey_table table
-
-   table=$FLAKEY_TABLE
-   [ $1 -eq 1 ]  table=$FLAKEY_TABLE_DROP
-
-   suspend_opt=
-   [ $nolockfs -eq 1 ]  suspend_opt=--nolockfs
-
-   $DMSETUP_PROG suspend $suspend_opt flakey-test
-   [ $? -ne 0 ]  _fatal failed to suspend flakey-test
-
-   $DMSETUP_PROG load flakey-test --table $table
-

Re: [patch 2/3] [PATCH 2/3] xfstests: add test for sysfs publication of btrfs info

2013-10-28 Thread Josef Bacik

On Fri, Oct 25, 2013 at 04:52:48PM -0400, Jeff Mahoney wrote:
 This tests the sysfs publishing for btrfs allocation and device
 membership info under a number of different layouts, similar to the
 btrfs replace test. We test the allocation files only for existence and
 that they contain numerical values. We test the device membership
 by mapping the devices used to create the file system to sysfs paths
 and matching them against the paths used for the device membership
 symlinks.
 
 Since this is a new feature, it passes on kernels without
 a /sys/fs/btrfs/fsid directory.
 
 Signed-off-by: Jeff Mahoney je...@suse.com
 ---
  common/config   |   1 +
  tests/btrfs/101 | 188 
 
  tests/btrfs/101.out |   2 +
  3 files changed, 191 insertions(+)
  create mode 100755 tests/btrfs/101
  create mode 100644 tests/btrfs/101.out
 
 diff --git a/common/config b/common/config
 index 20c5b09..5f9e2e8 100644
 --- a/common/config
 +++ b/common/config
 @@ -208,6 +208,7 @@ case $HOSTOS in
  export MKFS_UDF_PROG=`set_prog_path mkudffs`
  export MKFS_BTRFS_PROG=`set_btrfs_mkfs_prog_path_with_opts`
  export BTRFS_UTIL_PROG=`set_prog_path btrfs`
 +export UDEVADM_PROG=`set_prog_path udevadm`
  export BTRFS_SHOW_SUPER_PROG=`set_prog_path btrfs-show-super`
  export XFS_FSR_PROG=`set_prog_path xfs_fsr`
  export MKFS_NFS_PROG=false
 diff --git a/tests/btrfs/101 b/tests/btrfs/101
 new file mode 100755
 index 000..9a2834b
 --- /dev/null
 +++ b/tests/btrfs/101
 @@ -0,0 +1,188 @@
 +#! /bin/bash
 +# FS QA Test No. btrfs/101
 +#
 +# Test of the btrfs sysfs publishing
 +#
 +#---
 +# Copyright (C) 2013 SUSE.  All rights reserved.
 +#
 +# This program is free software; you can redistribute it and/or
 +# modify it under the terms of the GNU General Public License as
 +# published by the Free Software Foundation.
 +#
 +# This program is distributed in the hope that it would be useful,
 +# but WITHOUT ANY WARRANTY; without even the implied warranty of
 +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 +# GNU General Public License for more details.
 +#
 +# You should have received a copy of the GNU General Public License
 +# along with this program; if not, write the Free Software Foundation,
 +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
 +#
 +#---
 +#
 +
 +seq=`basename $0`
 +seqres=$RESULT_DIR/$seq
 +echo QA output created by $seq
 +
 +here=`pwd`
 +tmp=/tmp/$$
 +status=1
 +
 +# get standard environment, filters and checks
 +. ./common/rc
 +. ./common/filter
 +
 +# real QA test starts here
 +_need_to_be_root
 +_supported_fs btrfs
 +_require_scratch
 +_require_scratch_dev_pool
 +_require_command $UDEVADM_PROG udevadm

I don't think you need the extra udevadm here.  Thanks,

Josef
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [patch 1/3] [PATCH 1/3] xfstests: add test for global metadata reservation publishing

2013-10-28 Thread Josef Bacik

On Fri, Oct 25, 2013 at 04:52:47PM -0400, Jeff Mahoney wrote:
 My publishing patchset added the ability for the kernel to report
 the size of the global metadata reservation via ioctl and sysfs.
 
 This test confirms that we get sane results on an empty file system.
 
 ENOTTY and missing /sys/fs/btrfs/fsid/allocation are not considered
 failures.
 
 Signed-off-by: Jeff Mahoney je...@suse.com
 ---
  src/Makefile |  3 +-
  src/btrfs_ioctl_helper.c | 90 ++
  tests/btrfs/100  | 92 
 
  tests/btrfs/100.out  |  2 ++
  4 files changed, 186 insertions(+), 1 deletion(-)
  create mode 100644 src/btrfs_ioctl_helper.c
  create mode 100755 tests/btrfs/100
  create mode 100644 tests/btrfs/100.out
 
 diff --git a/src/Makefile b/src/Makefile
 index 84c8297..299f675 100644
 --- a/src/Makefile
 +++ b/src/Makefile
 @@ -18,7 +18,8 @@ LINUX_TARGETS = xfsctl bstat t_mtab getdevicesize 
 preallo_rw_pattern_reader \
   locktest unwritten_mmap bulkstat_unlink_test t_stripealign \
   bulkstat_unlink_test_modified t_dir_offset t_futimens t_immutable \
   stale_handle pwrite_mmap_blocked t_dir_offset2 seek_sanity_test \
 - seek_copy_test t_readdir_1 t_readdir_2 fsync-tester nsexec
 + seek_copy_test t_readdir_1 t_readdir_2 fsync-tester nsexec \
 + btrfs_ioctl_helper
  
  SUBDIRS =
  
 diff --git a/src/btrfs_ioctl_helper.c b/src/btrfs_ioctl_helper.c
 new file mode 100644
 index 000..a88b7b5
 --- /dev/null
 +++ b/src/btrfs_ioctl_helper.c
 @@ -0,0 +1,90 @@
 +#include sys/ioctl.h
 +#include stdio.h
 +#include sys/fcntl.h
 +#include errno.h
 +#include string.h
 +#include stdint.h
 +#include unistd.h
 +
 +#ifndef BTRFS_IOCTL_MAGIC
 +#define BTRFS_IOCTL_MAGIC 0x94
 +#endif
 +
 +#ifndef BTRFS_IOC_GLOBAL_RSV
 +#define BTRFS_IOC_GLOBAL_RSV _IOR(BTRFS_IOCTL_MAGIC, 20, uint64_t)
 +#endif
 +
 +static int global_rsv_ioctl(int fd, int argc, char *argv[])
 +{
 + uint64_t reserved;
 + int ret = ioctl(fd, BTRFS_IOC_GLOBAL_RSV, reserved);
 + if (ret)
 + return -errno;
 +
 + printf(%llu\n, (unsigned long long)reserved);
 + return 0;
 +}
 +
 +#define IOCTL_TABLE_ENTRY(_ioctl_name, _handler) \
 + { .name = #_ioctl_name, .ioctl_cmd = BTRFS_IOC_##_ioctl_name, \
 +   .handler = _handler, }
 +
 +struct ioctl_table_entry {
 + const char *name;
 + unsigned ioctl_cmd;
 + int (*handler)(int fd, int argc, char *argv[]);
 +};
 +
 +static struct ioctl_table_entry ioctls[] = {
 + IOCTL_TABLE_ENTRY(GLOBAL_RSV, global_rsv_ioctl),
 +};
 +
 +int
 +main(int argc, char *argv[])
 +{
 + int fd;
 + int ret;
 + struct ioctl_table_entry *entry = NULL;
 + int i;
 +
 + if (argc  3) {
 + fprintf(stderr,
 + usage: %s fs mount point ioctl name [args..]\n,
 + argv[0]);
 + return 1;
 + }
 +
 + fd = open(argv[1], O_RDONLY|O_DIRECTORY);
 + if (fd  0) {
 + perror(argv[1]);
 + return 1;
 + }
 +
 + for (i = 0; i  (sizeof(ioctls)/sizeof(ioctls[0])); i++) {
 + if (strcmp(argv[2], ioctls[i].name) == 0) {
 + entry = ioctls[i];
 + break;
 + }
 + }
 +
 + if (!entry) {
 + fprintf(stderr, ERROR: unknown ioctl %s\n, argv[2]);
 + close(fd);
 + return 1;
 + }
 +
 + ret = entry-handler(fd, argc - 3, argv + 3);
 + if (ret == -ENOTTY) {
 + printf(Not implemented.\n);
 + close(fd);
 + return 0;
 + } else if (ret) {
 + fprintf(stderr, ERROR: %s failed: %s\n,
 + entry-name, strerror(-ret));
 + close(fd);
 + return 1;
 + }
 +
 + close(fd);
 + return 0;
 +}
 diff --git a/tests/btrfs/100 b/tests/btrfs/100
 new file mode 100755
 index 000..d2a40b4
 --- /dev/null
 +++ b/tests/btrfs/100
 @@ -0,0 +1,92 @@
 +#!/bin/bash
 +# FA QA Test No. 100
 +#
 +# Test global metadata reservation reporting
 +#
 +# 1) Create empty file system
 +# 2) Call the BTRFS_IOC_GLOBAL_RSV ioctl and confirm it is 0  x  10MB
 +# 3) Read the /sys/fs/btrfs/fsid/allocation/global_rsv_reserved file
 +#and confirm the value is 0  x  10 MB
 +#
 +#---
 +# Copyright (c) 2013 SUSE, All Rights Reserved.
 +#
 +# This program is free software; you can redistribute it and/or
 +# modify it under the terms of the GNU General Public License as
 +# published by the Free Software Foundation.
 +#
 +# This program is distributed in the hope that it would be useful,
 +# but WITHOUT ANY WARRANTY; without even the implied warranty of
 +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 +# GNU General Public License for more details.
 +#
 +# You should have received a copy of the GNU General Public License
 +# along with this

Re: [patch 3/3] [PATCH 3/3] xfstests: add tests for new feature ioctl and sysfs interfaces

2013-10-28 Thread Josef Bacik

On Fri, Oct 25, 2013 at 04:52:49PM -0400, Jeff Mahoney wrote:
 This tests the exporting of feature information from the kernel via
 sysfs and ioctl. The first test works whether the sysfs permissions
 are correct, if the information exported via sysfs matches
 what the ioctls are reporting, and if they both match the on-disk
 superblock's version of the feature sets. The second and third tests
 test online setting and clearing of feature bits via the sysfs and
 ioctl interfaces, checking whether they match the on-disk super on
 each cycle.
 
 In every case, if the features are not present, it is not considered
 a failure and a message indicating that will be dumped to the $num.full
 file.
 
 Signed-off-by: Jeff Mahoney je...@suse.com
 ---
  src/btrfs_ioctl_helper.c |  93 +
  tests/btrfs/102  | 263 
 +++
  tests/btrfs/102.out  |   2 +
  tests/btrfs/103  | 188 +
  tests/btrfs/103.out  |   2 +
  tests/btrfs/104  | 186 +
  tests/btrfs/104.out  |   2 +
  7 files changed, 736 insertions(+)
  create mode 100755 tests/btrfs/102
  create mode 100644 tests/btrfs/102.out
  create mode 100755 tests/btrfs/103
  create mode 100644 tests/btrfs/103.out
  create mode 100755 tests/btrfs/104
  create mode 100644 tests/btrfs/104.out
 
 diff --git a/src/btrfs_ioctl_helper.c b/src/btrfs_ioctl_helper.c
 index a88b7b5..73cfa60 100644
 --- a/src/btrfs_ioctl_helper.c
 +++ b/src/btrfs_ioctl_helper.c
 @@ -5,6 +5,7 @@
  #include string.h
  #include stdint.h
  #include unistd.h
 +#include stdlib.h
  
  #ifndef BTRFS_IOCTL_MAGIC
  #define BTRFS_IOCTL_MAGIC 0x94
 @@ -14,6 +15,21 @@
  #define BTRFS_IOC_GLOBAL_RSV _IOR(BTRFS_IOCTL_MAGIC, 20, uint64_t)
  #endif
  
 +#ifndef BTRFS_IOC_GET_FEATURES
 +struct btrfs_ioctl_feature_flags {
 + uint64_t compat_flags;
 + uint64_t compat_ro_flags;
 + uint64_t incompat_flags;
 +};
 +
 +#define BTRFS_IOC_GET_FEATURES _IOR(BTRFS_IOCTL_MAGIC, 57, \
 +   struct btrfs_ioctl_feature_flags)
 +#define BTRFS_IOC_SET_FEATURES _IOW(BTRFS_IOCTL_MAGIC, 57, \
 +   struct btrfs_ioctl_feature_flags[2])
 +#define BTRFS_IOC_GET_SUPPORTED_FEATURES _IOR(BTRFS_IOCTL_MAGIC, 57, \
 +   struct btrfs_ioctl_feature_flags[3])
 +#endif
 +
  static int global_rsv_ioctl(int fd, int argc, char *argv[])
  {
   uint64_t reserved;
 @@ -25,6 +41,80 @@ static int global_rsv_ioctl(int fd, int argc, char *argv[])
   return 0;
  }
  
 +static int get_features_ioctl(int fd, int argc, char *argv[])
 +{
 + struct btrfs_ioctl_feature_flags flags;
 + int ret = ioctl(fd, BTRFS_IOC_GET_FEATURES, flags);
 + if (ret)
 + return -errno;
 +
 + printf(0x%llx 0x%llx 0x%llx\n,
 +(unsigned long long)flags.compat_flags,
 +(unsigned long long)flags.compat_ro_flags,
 +(unsigned long long)flags.incompat_flags);
 + return 0;
 +}
 +
 +static int set_features_ioctl(int fd, int argc, char *argv[])
 +{
 + struct btrfs_ioctl_feature_flags flags[2];
 + uint64_t bit, *bits, *mask;
 + if (argc != 3)
 + goto usage;
 +
 + memset(flags, 0, sizeof(flags));
 +
 + errno = 0;
 + bit = strtoull(argv[2], NULL, 10);
 + if (errno)
 + goto usage;
 +
 + if (strcmp(argv[1], compat) == 0) {
 + mask = flags[0].compat_flags;
 + bits = flags[1].compat_flags;
 + } else if (strcmp(argv[1], compat_ro) == 0) {
 + mask = flags[0].compat_ro_flags;
 + bits = flags[1].compat_ro_flags;
 + } else if (strcmp(argv[1], incompat) == 0) {
 + mask = flags[0].incompat_flags;
 + bits = flags[1].incompat_flags;
 + } else
 + goto usage;
 +
 + *mask |= bit;
 +
 + if (strcmp(argv[0], set) == 0)
 + *bits |= bit;
 +
 + return ioctl(fd, BTRFS_IOC_SET_FEATURES, flags);
 +usage:
 + fprintf(stderr, usage: SET_FEATURES set|clear 
 compat|compat_ro|incompat base-10 bitmask\n);
 + return -EINVAL;
 +}
 +
 +static int get_supported_features_ioctl(int fd, int argc, char *argv[])
 +{
 + struct btrfs_ioctl_feature_flags flags[3];
 + int ret;
 + int i;
 +
 + ret = ioctl(fd, BTRFS_IOC_GET_SUPPORTED_FEATURES, flags);
 + if (ret)
 + return -errno;
 +
 + for (i = 0; i  3; i++)
 + printf(0x%llx 0x%llx 0x%llx ,
 +(unsigned long long)flags[i].compat_flags,
 +(unsigned long long)flags[i].compat_ro_flags,
 +(unsigned long long)flags[i].incompat_flags);
 +
 + printf(\n);
 + return 0;
 +usage:
 + fprintf(stderr,
 + usage: GET_SUPPORTED_FEATURES 
 supported|settable|clearable\n);
 + return -EINVAL;
 +}
  #define IOCTL_TABLE_ENTRY(_ioctl_name, _handler) \
   { .name =

Null pointer dereference bug in btrfs_find_all_root

2013-10-28 Thread Pedro Fonseca


Hi,

I've encountered a few NULL pointer dereference bugs while running a 
custom test case on btrfs (kernel 3.11.1), inside a QEMU based VM.


Btrfs was statically compiled and the btrfs filesystem was mounted on 
one or two loop devices. All cases occurred during execution of a 
filesystem balance operation and while another file system operation 
was concurrently executed. In one case the other operation was a mknod 
and in another it was a filesystem show operation.


In all cases the NULL pointer dereference occurred in the 
btrfs_find_all_roots function and all cases produced a bug report 
similar to the one bellow:


[  138.573390] BUG: unable to handle kernel NULL pointer dereference 
at   (null)

[  138.573390] IP: [c12feeb3] ulist_next+0x4/0x26
[  138.573390] *pde = 
[  138.573390] Oops:  [#1] SMP
[  138.573390] Modules linked in: loop rtc_cmos pcspkr tpm_tis 
freq_table mperf i2c_piix4
[  138.573390] CPU: 0 PID: 2794 Comm: btrfs-endio-wri Not tainted 
3.11.1 #2

[  138.573390] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2007
[  138.573390] task: c5dea6b0 ti: c5e12000 task.ti: c5e12000
[  138.573390] EIP: 0060:[c12feeb3] EFLAGS: 0246 CPU: 0
[  138.573390] EIP is at ulist_next+0x4/0x26
[  138.573390] EAX:  EBX:  ECX: d9c29e38 EDX: c5e13da4
[  138.573390] ESI:  EDI: c5e13e00 EBP: c5e13d94 ESP: c5e13d90
[  138.573390]  DS: 007b ES: 007b FS: 00d8 GS:  SS: 0068
[  138.573390] CR0: 8005003b CR2:  CR3: 059d8000 CR4: 0690
[  138.573390] DR0:  DR1:  DR2:  DR3: 
[  138.573390] DR6:  DR7: 
[  138.573390] Stack:
[  138.573390]   c5e13db4 c12fe7b8 de4d1000 d9cead48 0001 
010dd000 deeab810
[  138.573390]  0001 c5e13e10 c12fe8cf 00af2000   
0003 c5e13e00
[  138.573390]  deeabe00  0003 deeab800 de4d1000 d9cead48 
de4d14b8 de4d14b8

[  138.573390] Call Trace:
[  138.573390]  [c12fe7b8] btrfs_find_all_roots+0x88/0xa6
[  138.573390]  [c12fe8cf] iterate_extent_inodes+0xf9/0x1b2
[  138.573390]  [c12feba9] iterate_inodes_from_logical+0x88/0x97
[  138.573390]  [c12b8880] ? record_extent_backrefs+0x8b/0x8b
[  138.573390]  [c12b8845] record_extent_backrefs+0x50/0x8b
[  138.573390]  [c12b8880] ? record_extent_backrefs+0x8b/0x8b
[  138.573390]  [c12c1f8e] btrfs_finish_ordered_io+0x77e/0x876
[  138.573390]  [c12c2091] finish_ordered_fn+0xb/0xd
[  138.573390]  [c12db161] worker_loop+0x111/0x3ec
[  138.573390]  [c104a2c4] ? default_wake_function+0xb/0xd
[  138.573390]  [c1040c43] kthread+0x90/0x95
[  138.573390]  [c12db050] ? btrfs_queue_worker+0x1c8/0x1c8
[  138.573390]  [c176c277] ret_from_kernel_thread+0x1b/0x28
[  138.573390]  [c1040bb3] ? kthread_freezable_should_stop+0x4f/0x4f
[  138.573390] Code: 5b 5e 5f 5d c3 90 90 90 55 8d 50 10 89 e5 c7 00 
00 00 00 00 89 50 08 c7 40 04 10 00 00 00 c7 40 0c 00 00 00 00 5d c3 
55 89 e5 53 8b 08 85 c9 74 15 8b 1a 85 db 78 0f 39 cb 73 0b 6b cb 1c 
43 03

[  138.573390] EIP: [c12feeb3] ulist_next+0x4/0x26 SS:ESP 0068:c5e13d90
[  138.573390] CR2: 
[  138.573390] ---[ end trace 626899e1abbe ]---
[  138.573390] BUG: unable to handle kernel NULL pointer dereference 
at   (null)

[  138.573390] IP: [c12feeb3] ulist_next+0x4/0x26
[  138.573390] *pde = 
[  138.573390] Oops:  [#2] SMP
[  138.573390] Modules linked in: loop rtc_cmos pcspkr tpm_tis 
freq_table mperf i2c_piix4
[  138.573390] CPU: 1 PID: 2793 Comm: btrfs-endio-wri Tainted: G  
D  3.11.1 #2

[  138.573390] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2007
[  138.573390] task: c59cc190 ti: dedc8000 task.ti: dedc8000
[  138.573390] EIP: 0060:[c12feeb3] EFLAGS: 0246 CPU: 1
[  138.573390] EIP is at ulist_next+0x4/0x26
[  138.573390] EAX:  EBX:  ECX: d9c296c8 EDX: dedc9da4
[  138.573390] ESI:  EDI: dedc9e00 EBP: dedc9d94 ESP: dedc9d90
[  138.573390]  DS: 007b ES: 007b FS: 00d8 GS:  SS: 0068
[  138.573390] CR0: 8005003b CR2:  CR3: 03ebe000 CR4: 0690
[  138.573390] DR0:  DR1:  DR2:  DR3: 
[  138.573390] DR6:  DR7: 
[  138.573390] Stack:
[  138.573390]   dedc9db4 c12fe7b8 de4d1000 d9ceaf50 0001 
00dea000 de4c6410
[  138.573390]  0001 dedc9e10 c12fe8cf 00af2000   
0005 dedc9e00
[  138.573390]  de4c6c00  0005 de4c6400 de4d1000 d9ceaf50 
de4d14b8 c5e13de8

[  138.573390] Call Trace:
[  138.573390]  [c12fe7b8] btrfs_find_all_roots+0x88/0xa6
[  138.573390]  [c12fe8cf] iterate_extent_inodes+0xf9/0x1b2
[  138.573390]  [c12feba9] iterate_inodes_from_logical+0x88/0x97
[  138.573390]  [c12b8880] ? record_extent_backrefs+0x8b/0x8b
[  138.573390]  [c12b8845] record_extent_backrefs+0x50/0x8b
[  138.573390]  [c12b8880] ? record_extent_backrefs+0x8b/0x8b
[  138.573390]  [c12c1f8e] btrfs_finish_ordered_io+0x77e/0x876
[  138.573390]  [c12c2091] finish_ordered_fn+0xb/0xd
[  138.573390]  [c12db161] worker_loop+0x111/0x3ec
[

[PATCH] Btrfs: fix verification of dir_item

2013-10-28 Thread Filipe David Borba Manana

We were ignoring the name component of the dir_item. Both the
name and data must fit within BTRFS_MAX_XATTR_SIZE(root).

Signed-off-by: Filipe David Borba Manana fdman...@gmail.com
---
 fs/btrfs/dir-item.c |6 --
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/fs/btrfs/dir-item.c b/fs/btrfs/dir-item.c
index 1c529db..c031ea3 100644
--- a/fs/btrfs/dir-item.c
+++ b/fs/btrfs/dir-item.c
@@ -474,8 +474,10 @@ int verify_dir_item(struct btrfs_root *root,
}
 
/* BTRFS_MAX_XATTR_SIZE is the same for all dir items */
-   if (btrfs_dir_data_len(leaf, dir_item)  BTRFS_MAX_XATTR_SIZE(root)) {
-   printk(KERN_CRIT btrfs: invalid dir item data len: %u\n,
+   if ((btrfs_dir_data_len(leaf, dir_item) +
+btrfs_dir_name_len(leaf, dir_item))  BTRFS_MAX_XATTR_SIZE(root)) {
+   printk(KERN_CRIT btrfs: invalid dir item name + data len: %u + 
%u\n,
+  (unsigned)btrfs_dir_name_len(leaf, dir_item),
   (unsigned)btrfs_dir_data_len(leaf, dir_item));
return 1;
}
-- 
1.7.9.5

--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH] Btrfs: fix csum search offset/length calculation in log tree

2013-10-28 Thread Filipe David Borba Manana

We were setting the csums search offset and length to the right values if
the extent is compressed, but later on right before doing the csums lookup
we were overriding these two parameters regardless of compression being
set or not for the extent.

Signed-off-by: Filipe David Borba Manana fdman...@gmail.com
---
 fs/btrfs/tree-log.c |   14 +++---
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/fs/btrfs/tree-log.c b/fs/btrfs/tree-log.c
index 2ff031a..d75303e 100644
--- a/fs/btrfs/tree-log.c
+++ b/fs/btrfs/tree-log.c
@@ -3458,11 +3458,6 @@ static int log_one_extent(struct btrfs_trans_handle 
*trans,
if (skip_csum)
return 0;
 
-   if (em-compress_type) {
-   csum_offset = 0;
-   csum_len = block_len;
-   }
-
/*
 * First check and see if our csums are on our outstanding ordered
 * extents.
@@ -3546,8 +3541,13 @@ unlocked:
if (!mod_len || ret)
return ret;
 
-   csum_offset = mod_start - em-start;
-   csum_len = mod_len;
+   if (em-compress_type) {
+   csum_offset = 0;
+   csum_len = block_len;
+   } else {
+   csum_offset = mod_start - em-start;
+   csum_len = mod_len;
+   }
 
/* block start is already adjusted for the file extent offset. */
ret = btrfs_lookup_csums_range(log-fs_info-csum_root,
-- 
1.7.9.5

--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH] Btrfs: log recovery, don't unlink inode always on error

2013-10-28 Thread Filipe David Borba Manana

If we get any error while doing a dir index/item lookup in the
log tree, we were always unlinking the corresponding inode in
the subvolume. It makes sense to unlink only if the lookup failed
to find the dir index/item, which corresponds to NULL or -ENOENT,
and not when other errors happen (like a transient -ENOMEM or -EIO).

Signed-off-by: Filipe David Borba Manana fdman...@gmail.com
---
 fs/btrfs/tree-log.c |5 -
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/fs/btrfs/tree-log.c b/fs/btrfs/tree-log.c
index d75303e..2990fa4 100644
--- a/fs/btrfs/tree-log.c
+++ b/fs/btrfs/tree-log.c
@@ -1832,7 +1832,7 @@ again:
 dir_key-offset,
 name, name_len, 0);
}
-   if (IS_ERR_OR_NULL(log_di)) {
+   if (!log_di || (IS_ERR(log_di)  PTR_ERR(log_di) == -ENOENT)) {
btrfs_dir_item_key_to_cpu(eb, di, location);
btrfs_release_path(path);
btrfs_release_path(log_path);
@@ -1869,6 +1869,9 @@ again:
goto again;
ret = 0;
goto out;
+   } else if (IS_ERR(log_di)) {
+   kfree(name);
+   return PTR_ERR(log_di);
}
btrfs_release_path(log_path);
kfree(name);
-- 
1.7.9.5

--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: No space left on device, problem

2013-10-28 Thread Chris Murphy


On Oct 28, 2013, at 1:40 AM, Igor M igor...@gmail.com wrote:
 
 dmesg: http://pastebin.com/t2H1QYye


You've got a warning related to pcie bridge on boot, with a trace that follows. 
I don't know if this could be related to some problems.

[0.325976] [ cut here ]
[0.326086] WARNING: CPU: 5 PID: 1 at drivers/pci/search.c:46 
pci_find_upstream_pcie_bridge+0x50/0x70()
[0.326263] Modules linked in:
[0.326412] CPU: 5 PID: 1 Comm: swapper/0 Not tainted 3.11.6 #1
[0.326520] Hardware name: System manufacturer System Product Name/P8H77-V 
LE, BIOS 0601 06/06/2012
[0.326695]   0009 81449f86 

[0.327032]  810514b1 88040b031898 88040b031800 
88040b031898
[0.327367]  8000 77ff8000 812191a0 
0004
[0.327704] Call Trace:
[0.327819]  [81449f86] ? dump_stack+0x41/0x51
[0.327931]  [810514b1] ? warn_slowpath_common+0x81/0xb0
[0.328040]  [812191a0] ? pci_find_upstream_pcie_bridge+0x50/0x70
[0.328151]  [81d3] ? intel_iommu_add_device+0x43/0x210
[0.328261]  [81330510] ? bus_set_iommu+0x60/0x60
[0.328370]  [8133053c] ? add_iommu_group+0x2c/0x60
[0.328481]  [81292a3d] ? bus_for_each_dev+0x4d/0x80
[0.328591]  [813304fa] ? bus_set_iommu+0x4a/0x60
[0.328701]  [8163bc0d] ? intel_iommu_init+0xb20/0xc45
[0.328812]  [81612919] ? unpack_to_rootfs+0x24b/0x25b
[0.328922]  [816163c6] ? pci_iommu_init+0xe/0x37
[0.329031]  [816163b8] ? memblock_find_dma_reserve+0x148/0x148
[0.329142]  [810002f2] ? do_one_initcall+0x102/0x150
[0.329252]  [81611e43] ? kernel_init_freeable+0xfd/0x18e
[0.329362]  [816117cf] ? do_early_param+0x83/0x83
[0.329471]  [8180] ? rest_init+0x70/0x70
[0.329579]  [8189] ? kernel_init+0x9/0xe0
[0.329688]  [8144f36c] ? ret_from_fork+0x7c/0xb0
[0.329797]  [8180] ? rest_init+0x70/0x70
[0.329907] ---[ end trace 0946f959337cff8b ]---


There are also numerous ACPI errors.

ACPI Error: [DSSP] Namespace lookup failure

check this:
http://forums.gentoo.org/viewtopic-t-960476-start-0.html


Anyway, I don't see any read or write failures for any of the drives which is 
what I was kinda expecting.


 
 dest disk: http://pastebin.com/ez9jALS2

This is a new drive with only 71 power on hours yet I'm seeing this:

• 0x0009  2   27  Transition from drive PhyRdy to drive PhyNRdy
• 0x000a  2   27  Device-to-host register FISes sent due to a 
COMRESET



That's unexpected but I don't know that it's releated. The dmesg doesn't report 
any phy issues with the drive. Maybe check syslog or journalctl with a case 
insensitive search for phy and see if you find anything.




Chris Murphy

--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH] Btrfs-progs: rework open_ctree to take flags, add a new one V2

2013-10-28 Thread Josef Bacik

So I needed to add a flag to not try to read block groups when doing
--init-extent-tree since we could hang there, but that meant adding a whole
other 0/1 type flag to open_ctree_fs_info.  So instead I've converted it all
over to using a flags setting and added the flag that I needed.  This has been
tested with xfstests and make test.  Thanks,

Signed-off-by: Josef Bacik jba...@fusionio.com
---
V1-V2: forgot to make clean first, I missed converting one of the users

 btrfs-convert.c   |   6 +--
 btrfs-corrupt-block.c |   2 +-
 btrfs-debug-tree.c|   2 +-
 btrfs-image.c |   9 -
 btrfs-zero-log.c  |   2 +-
 btrfstune.c   |   2 +-
 cmds-check.c  |  14 +++
 cmds-restore.c|   3 +-
 disk-io.c | 107 ++
 disk-io.h |  26 ++--
 mkfs.c|   2 +-
 quick-test.c  |  10 ++---
 super-recover.c   |   4 +-
 utils.c   |   2 +-
 14 files changed, 77 insertions(+), 114 deletions(-)

diff --git a/btrfs-convert.c b/btrfs-convert.c
index 26c7b5f..ae10eed 100644
--- a/btrfs-convert.c
+++ b/btrfs-convert.c
@@ -2256,7 +2256,7 @@ static int do_convert(const char *devname, int datacsum, 
int packing,
fprintf(stderr, unable to update system chunk\n);
goto fail;
}
-   root = open_ctree_fd(fd, devname, super_bytenr, O_RDWR);
+   root = open_ctree_fd(fd, devname, super_bytenr, OPEN_CTREE_WRITES);
if (!root) {
fprintf(stderr, unable to open ctree\n);
goto fail;
@@ -2317,7 +2317,7 @@ static int do_convert(const char *devname, int datacsum, 
int packing,
goto fail;
}
 
-   root = open_ctree_fd(fd, devname, 0, O_RDWR);
+   root = open_ctree_fd(fd, devname, 0, OPEN_CTREE_WRITES);
if (!root) {
fprintf(stderr, unable to open ctree\n);
goto fail;
@@ -2418,7 +2418,7 @@ static int do_rollback(const char *devname, int force)
fprintf(stderr, unable to open %s\n, devname);
goto fail;
}
-   root = open_ctree_fd(fd, devname, 0, O_RDWR);
+   root = open_ctree_fd(fd, devname, 0, OPEN_CTREE_WRITES);
if (!root) {
fprintf(stderr, unable to open ctree\n);
goto fail;
diff --git a/btrfs-corrupt-block.c b/btrfs-corrupt-block.c
index b40a529..f0c14a9 100644
--- a/btrfs-corrupt-block.c
+++ b/btrfs-corrupt-block.c
@@ -799,7 +799,7 @@ int main(int ac, char **av)
radix_tree_init();
cache_tree_init(root_cache);
 
-   root = open_ctree(dev, 0, 1);
+   root = open_ctree(dev, 0, OPEN_CTREE_WRITES);
if (!root) {
fprintf(stderr, Open ctree failed\n);
exit(1);
diff --git a/btrfs-debug-tree.c b/btrfs-debug-tree.c
index 4a9d89d..4a83770 100644
--- a/btrfs-debug-tree.c
+++ b/btrfs-debug-tree.c
@@ -171,7 +171,7 @@ int main(int ac, char **av)
if (ac != 1)
print_usage();
 
-   info = open_ctree_fs_info(av[optind], 0, 0, 0, 1, 0);
+   info = open_ctree_fs_info(av[optind], 0, 0, OPEN_CTREE_PARTIAL);
if (!info) {
fprintf(stderr, unable to open %s\n, av[optind]);
exit(1);
diff --git a/btrfs-image.c b/btrfs-image.c
index 40ed483..7bcfc06 100644
--- a/btrfs-image.c
+++ b/btrfs-image.c
@@ -2267,7 +2267,10 @@ static int __restore_metadump(const char *input, FILE 
*out, int old_restore,
/* NOTE: open with write mode */
if (fixup_offset) {
BUG_ON(!target);
-   info = open_ctree_fs_info_restore(target, 0, 0, 1, 1);
+   info = open_ctree_fs_info(target, 0, 0,
+ OPEN_CTREE_WRITES |
+ OPEN_CTREE_RESTORE |
+ OPEN_CTREE_PARTIAL);
if (!info) {
fprintf(stderr, %s: open ctree failed\n, __func__);
ret = -EIO;
@@ -2555,7 +2558,9 @@ int main(int argc, char *argv[])
u64 total_devs;
int i;
 
-   info = open_ctree_fs_info_restore(target, 0, 0, 0, 1);
+   info = open_ctree_fs_info(target, 0, 0,
+ OPEN_CTREE_PARTIAL |
+ OPEN_CTREE_RESTORE);
if (!info) {
int e = errno;
fprintf(stderr, unable to open %s error = %s\n,
diff --git a/btrfs-zero-log.c b/btrfs-zero-log.c
index 432adff..ab7f418 100644
--- a/btrfs-zero-log.c
+++ b/btrfs-zero-log.c
@@ -60,7 +60,7 @@ int main(int ac, char **av)
goto out;
}
 
-   root = open_ctree(av[1], 0, 1);
+   root = open_ctree(av[1], 0, OPEN_CTREE_WRITES);
 
if (root == NULL)
return 1;
diff --git a/btrfstune.c b/btrfstune.c
index 1cf6a68..50724ba

[PATCH] Btrfs: make sure the delalloc workers actually flush compressed writes

2013-10-28 Thread Josef Bacik

When using delalloc workers in a non-waiting way (like for enospc handling) we
can end up not actually waiting for the dirty pages to be started if we have
compression.  We need to add an extra filemap flush to make sure any async
extents that have started are actually moved along before returning.  Thanks,

Signed-off-by: Josef Bacik jba...@fusionio.com
---
 fs/btrfs/inode.c | 18 --
 1 file changed, 12 insertions(+), 6 deletions(-)

diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c
index a239de0..0c7212a 100644
--- a/fs/btrfs/inode.c
+++ b/fs/btrfs/inode.c
@@ -8181,18 +8181,24 @@ out_notrans:
 static void btrfs_run_delalloc_work(struct btrfs_work *work)
 {
struct btrfs_delalloc_work *delalloc_work;
+   struct inode *inode;
 
delalloc_work = container_of(work, struct btrfs_delalloc_work,
 work);
-   if (delalloc_work-wait)
-   btrfs_wait_ordered_range(delalloc_work-inode, 0, (u64)-1);
-   else
-   filemap_flush(delalloc_work-inode-i_mapping);
+   inode = delalloc_work-inode;
+   if (delalloc_work-wait) {
+   btrfs_wait_ordered_range(inode, 0, (u64)-1);
+   } else {
+   filemap_flush(inode-i_mapping);
+   if (test_bit(BTRFS_INODE_HAS_ASYNC_EXTENT,
+BTRFS_I(inode)-runtime_flags))
+   filemap_flush(inode-i_mapping);
+   }
 
if (delalloc_work-delay_iput)
-   btrfs_add_delayed_iput(delalloc_work-inode);
+   btrfs_add_delayed_iput(inode);
else
-   iput(delalloc_work-inode);
+   iput(inode);
complete(delalloc_work-completion);
 }
 
-- 
1.8.3.1

--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: No space left on device, problem

2013-10-28 Thread Josef Bacik

On Sun, Oct 27, 2013 at 09:50:37AM +0100, Igor M wrote:
 On Sun, Oct 27, 2013 at 2:00 AM, Tomasz Chmielewski t...@virtall.com wrote:
  Still no messages. Parameter seems to be active as
  /sys/module/printk/parameters/ignore_loglevel is Y, but there are no
  messages in log files or dmesg. Maybe I need to turn on some kernel
  debugging option and recompile kernel ?
  Also I should mention that cca 230G+ data was copied before this error
  started to occur.
 
  I think I saw a similar issue before.
 
  Can you try using rsync with --bwlimit XY option to copy the files?
 
  The option will limit the speed, in kB, at which the file is being
  copied; it will work even when source and destination files are on a
  local machine.
 
 
 Also I run strace cp -a ..
 ...
 read(3, 350348f07$0$24520$c3e8da3$fb4835..., 65536) = 65536
 write(4, 350348f07$0$24520$c3e8da3$fb4835..., 65536) = 65536
 read(3, 62.76C52BF412E849CB86D4FF3898B94..., 65536) = 65536
 write(4, 62.76C52BF412E849CB86D4FF3898B94..., 65536) = -1 ENOSPC (No
 space left on device)
 
 Last two write calls take a lot more time, and then last one returns
 ENOSPC. But if this write is retryed, then it succeeds.
 I tried with midnight commander and when error occurs, if I Retry
 operation then it finishes copying this file until error occurs again
 at next file.
 
 With --bwlimit it seems to be better, lower the speed later the error
 occurs, and if it's slow enough copy is successfull.
 But now I'm not sure anymore. I copied a few files with bwlimit, and
 now sudenly error doesn't occur anymore, even with no bwlimit.
 I'll do some more tests.

I just sent a patch to the list

[PATCH] Btrfs: make sure the delalloc workers actually flush compressed writes

Can you run this patch and see if it makes a difference for your test?  Thanks,

Josef
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: No space left on device, problem

2013-10-28 Thread Igor M

On Mon, Oct 28, 2013 at 6:57 PM, Chris Murphy li...@colorremedies.com wrote:

 On Oct 28, 2013, at 1:40 AM, Igor M igor...@gmail.com wrote:

 dmesg: http://pastebin.com/t2H1QYye


 You've got a warning related to pcie bridge on boot, with a trace that 
 follows. I don't know if this could be related to some problems.

 [0.325976] [ cut here ]
 [0.326086] WARNING: CPU: 5 PID: 1 at drivers/pci/search.c:46 
 pci_find_upstream_pcie_bridge+0x50/0x70()
 [0.326263] Modules linked in:
 [0.326412] CPU: 5 PID: 1 Comm: swapper/0 Not tainted 3.11.6 #1
 [0.326520] Hardware name: System manufacturer System Product Name/P8H77-V 
 LE, BIOS 0601 06/06/2012
 [0.326695]   0009 81449f86 
 
 [0.327032]  810514b1 88040b031898 88040b031800 
 88040b031898
 [0.327367]  8000 77ff8000 812191a0 
 0004
 [0.327704] Call Trace:
 [0.327819]  [81449f86] ? dump_stack+0x41/0x51
 [0.327931]  [810514b1] ? warn_slowpath_common+0x81/0xb0
 [0.328040]  [812191a0] ? pci_find_upstream_pcie_bridge+0x50/0x70
 [0.328151]  [81d3] ? intel_iommu_add_device+0x43/0x210
 [0.328261]  [81330510] ? bus_set_iommu+0x60/0x60
 [0.328370]  [8133053c] ? add_iommu_group+0x2c/0x60
 [0.328481]  [81292a3d] ? bus_for_each_dev+0x4d/0x80
 [0.328591]  [813304fa] ? bus_set_iommu+0x4a/0x60
 [0.328701]  [8163bc0d] ? intel_iommu_init+0xb20/0xc45
 [0.328812]  [81612919] ? unpack_to_rootfs+0x24b/0x25b
 [0.328922]  [816163c6] ? pci_iommu_init+0xe/0x37
 [0.329031]  [816163b8] ? memblock_find_dma_reserve+0x148/0x148
 [0.329142]  [810002f2] ? do_one_initcall+0x102/0x150
 [0.329252]  [81611e43] ? kernel_init_freeable+0xfd/0x18e
 [0.329362]  [816117cf] ? do_early_param+0x83/0x83
 [0.329471]  [8180] ? rest_init+0x70/0x70
 [0.329579]  [8189] ? kernel_init+0x9/0xe0
 [0.329688]  [8144f36c] ? ret_from_fork+0x7c/0xb0
 [0.329797]  [8180] ? rest_init+0x70/0x70
 [0.329907] ---[ end trace 0946f959337cff8b ]---


 There are also numerous ACPI errors.

 ACPI Error: [DSSP] Namespace lookup failure

 check this:
 http://forums.gentoo.org/viewtopic-t-960476-start-0.html


 Anyway, I don't see any read or write failures for any of the drives which is 
 what I was kinda expecting.



 dest disk: http://pastebin.com/ez9jALS2

 This is a new drive with only 71 power on hours yet I'm seeing this:

 • 0x0009  2   27  Transition from drive PhyRdy to drive 
 PhyNRdy
 • 0x000a  2   27  Device-to-host register FISes sent due to a 
 COMRESET



 That's unexpected but I don't know that it's releated. The dmesg doesn't 
 report any phy issues with the drive. Maybe check syslog or journalctl with a 
 case insensitive search for phy and see if you find anything.




 Chris Murphy


Drive should be ok. About pcie_bridge warning, I'm not sure how to
solve, otherwise this computer is stable.
There is no error if compression is turned off or non-compressible
file is copied.
I'll try on different computer and see if the same will happen.
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: No space left on device, problem

2013-10-28 Thread Igor M

On Mon, Oct 28, 2013 at 8:05 PM, Josef Bacik jba...@fusionio.com wrote:
 On Sun, Oct 27, 2013 at 09:50:37AM +0100, Igor M wrote:
 On Sun, Oct 27, 2013 at 2:00 AM, Tomasz Chmielewski t...@virtall.com wrote:
  Still no messages. Parameter seems to be active as
  /sys/module/printk/parameters/ignore_loglevel is Y, but there are no
  messages in log files or dmesg. Maybe I need to turn on some kernel
  debugging option and recompile kernel ?
  Also I should mention that cca 230G+ data was copied before this error
  started to occur.
 
  I think I saw a similar issue before.
 
  Can you try using rsync with --bwlimit XY option to copy the files?
 
  The option will limit the speed, in kB, at which the file is being
  copied; it will work even when source and destination files are on a
  local machine.
 

 Also I run strace cp -a ..
 ...
 read(3, 350348f07$0$24520$c3e8da3$fb4835..., 65536) = 65536
 write(4, 350348f07$0$24520$c3e8da3$fb4835..., 65536) = 65536
 read(3, 62.76C52BF412E849CB86D4FF3898B94..., 65536) = 65536
 write(4, 62.76C52BF412E849CB86D4FF3898B94..., 65536) = -1 ENOSPC (No
 space left on device)

 Last two write calls take a lot more time, and then last one returns
 ENOSPC. But if this write is retryed, then it succeeds.
 I tried with midnight commander and when error occurs, if I Retry
 operation then it finishes copying this file until error occurs again
 at next file.

 With --bwlimit it seems to be better, lower the speed later the error
 occurs, and if it's slow enough copy is successfull.
 But now I'm not sure anymore. I copied a few files with bwlimit, and
 now sudenly error doesn't occur anymore, even with no bwlimit.
 I'll do some more tests.

 I just sent a patch to the list

 [PATCH] Btrfs: make sure the delalloc workers actually flush compressed writes

 Can you run this patch and see if it makes a difference for your test?  
 Thanks,

 Josef

I'll try with this patch.
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 1/3] xfstests: btrfs/021: test global metadata reservation reporting

2013-10-28 Thread Jeff Mahoney

Btrfs can now report the size of the global metadata reservation
via icotl and sysfs.

This test confirms that we get sane results on an empty file system.

ENOTTY and missing /sys/fs/btrfs/fsid/allocation are not considered
failures.

Signed-off-by: Jeff Mahoney je...@suse.com
---
 common/rc|6 +++
 src/Makefile |3 +-
 src/btrfs_ioctl_helper.c |   90 ++
 tests/btrfs/021  |   88 
 tests/btrfs/021.out  |2 +
 5 files changed, 188 insertions(+), 1 deletions(-)
 create mode 100644 src/btrfs_ioctl_helper.c
 create mode 100755 tests/btrfs/021
 create mode 100644 tests/btrfs/021.out

Index: xfstests/common/rc
===
--- xfstests.orig/common/rc
+++ xfstests/common/rc
@@ -65,6 +65,12 @@ _btrfs_get_subvolid()
$BTRFS_UTIL_PROG sub list $mnt | grep $name | awk '{ print $2 }'
 }
 
+_btrfs_get_fsid()
+{
+   local dev=$1
+   $BTRFS_UTIL_PROG filesystem show $dev|awk '/uuid:/ {print $NF}'
+}
+
 # Prints the md5 checksum of a given file
 _md5_checksum()
 {
Index: xfstests/src/Makefile
===
--- xfstests.orig/src/Makefile
+++ xfstests/src/Makefile
@@ -18,7 +18,8 @@ LINUX_TARGETS = xfsctl bstat t_mtab getd
locktest unwritten_mmap bulkstat_unlink_test t_stripealign \
bulkstat_unlink_test_modified t_dir_offset t_futimens t_immutable \
stale_handle pwrite_mmap_blocked t_dir_offset2 seek_sanity_test \
-   seek_copy_test t_readdir_1 t_readdir_2 fsync-tester nsexec
+   seek_copy_test t_readdir_1 t_readdir_2 fsync-tester nsexec \
+   btrfs_ioctl_helper
 
 SUBDIRS =
 
Index: xfstests/src/btrfs_ioctl_helper.c
===
--- /dev/null
+++ xfstests/src/btrfs_ioctl_helper.c
@@ -0,0 +1,90 @@
+#include sys/ioctl.h
+#include stdio.h
+#include sys/fcntl.h
+#include errno.h
+#include string.h
+#include stdint.h
+#include unistd.h
+
+#ifndef BTRFS_IOCTL_MAGIC
+#define BTRFS_IOCTL_MAGIC 0x94
+#endif
+
+#ifndef BTRFS_IOC_GLOBAL_RSV
+#define BTRFS_IOC_GLOBAL_RSV _IOR(BTRFS_IOCTL_MAGIC, 20, uint64_t)
+#endif
+
+static int global_rsv_ioctl(int fd, int argc, char *argv[])
+{
+   uint64_t reserved;
+   int ret = ioctl(fd, BTRFS_IOC_GLOBAL_RSV, reserved);
+   if (ret)
+   return -errno;
+
+   printf(%llu\n, (unsigned long long)reserved);
+   return 0;
+}
+
+#define IOCTL_TABLE_ENTRY(_ioctl_name, _handler) \
+   { .name = #_ioctl_name, .ioctl_cmd = BTRFS_IOC_##_ioctl_name, \
+ .handler = _handler, }
+
+struct ioctl_table_entry {
+   const char *name;
+   unsigned ioctl_cmd;
+   int (*handler)(int fd, int argc, char *argv[]);
+};
+
+static struct ioctl_table_entry ioctls[] = {
+   IOCTL_TABLE_ENTRY(GLOBAL_RSV, global_rsv_ioctl),
+};
+
+int
+main(int argc, char *argv[])
+{
+   int fd;
+   int ret;
+   struct ioctl_table_entry *entry = NULL;
+   int i;
+
+   if (argc  3) {
+   fprintf(stderr,
+   usage: %s fs mount point ioctl name [args..]\n,
+   argv[0]);
+   return 1;
+   }
+
+   fd = open(argv[1], O_RDONLY|O_DIRECTORY);
+   if (fd  0) {
+   perror(argv[1]);
+   return 1;
+   }
+
+   for (i = 0; i  (sizeof(ioctls)/sizeof(ioctls[0])); i++) {
+   if (strcmp(argv[2], ioctls[i].name) == 0) {
+   entry = ioctls[i];
+   break;
+   }
+   }
+
+   if (!entry) {
+   fprintf(stderr, ERROR: unknown ioctl %s\n, argv[2]);
+   close(fd);
+   return 1;
+   }
+
+   ret = entry-handler(fd, argc - 3, argv + 3);
+   if (ret == -ENOTTY) {
+   printf(Not implemented.\n);
+   close(fd);
+   return 0;
+   } else if (ret) {
+   fprintf(stderr, ERROR: %s failed: %s\n,
+   entry-name, strerror(-ret));
+   close(fd);
+   return 1;
+   }
+
+   close(fd);
+   return 0;
+}
Index: xfstests/tests/btrfs/021
===
--- /dev/null
+++ xfstests/tests/btrfs/021
@@ -0,0 +1,88 @@
+#!/bin/bash
+# FA QA Test No. 021
+#
+# Test global metadata reservation reporting
+#
+# 1) Create empty file system
+# 2) Call the BTRFS_IOC_GLOBAL_RSV ioctl and confirm it is 0  x  10MB
+# 3) Read the /sys/fs/btrfs/fsid/allocation/global_rsv_reserved file
+#and confirm the value is 0  x  10 MB
+#
+#---
+# Copyright (c) 2013 SUSE, All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free

[PATCH 0/3] xfstests: for btrfs publishing/export patchset (v2)

2013-10-28 Thread Jeff Mahoney

These are the tests that Josef requested accompany the submission of
the btrfs publishing/export patchset[1].

021: Test new global metadata reservation ioctl
022: Test new sysfs exporting of allocation and device membership
023: Test new sysfs exporting of feature bits
024: Test setting/clearing feature bits via ioctl
025: Test setting/clearing feature bits via sysfs

As part of the ioctl testing, I added a helper that can be extended
easily to export simple information via ioctl. This means that the
tests can be independent of btrfsprogs version or its output formatting.

V2:
- Used next numbers in sequence
- Better commit messages
- Fixed handling of the missing feature cases
- Fixed issues raised in Josef's review

-Jeff

[1] http://thread.gmane.org/gmane.comp.file-systems.btrfs/29307

--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 3/3] xfstests: btrfs/023,024,025: test feature ioctl and sysfs interfaces

2013-10-28 Thread Jeff Mahoney

This tests the exporting of feature information from the kernel via
sysfs and ioctl. The first test works whether the sysfs permissions
are correct, if the information exported via sysfs matches
what the ioctls are reporting, and if they both match the on-disk
superblock's version of the feature sets. The second and third tests
test online setting and clearing of feature bits via the sysfs and
ioctl interfaces, checking whether they match the on-disk super on
each cycle.

In every case, if the features are not present, it is not considered
a failure and a message indicating that will be dumped to the $num.full
file.

Signed-off-by: Jeff Mahoney je...@suse.com
---
 common/rc|7 ++
 src/btrfs_ioctl_helper.c |   93 
 tests/btrfs/023  |  269 ++
 tests/btrfs/023.out  |2 +
 tests/btrfs/024  |  185 +++
 tests/btrfs/024.out  |2 +
 tests/btrfs/025  |  178 ++
 tests/btrfs/025.out  |2 +
 8 files changed, 738 insertions(+), 0 deletions(-)
 create mode 100755 tests/btrfs/023
 create mode 100644 tests/btrfs/023.out
 create mode 100755 tests/btrfs/024
 create mode 100644 tests/btrfs/024.out
 create mode 100755 tests/btrfs/025
 create mode 100644 tests/btrfs/025.out

Index: xfstests/common/rc
===
--- xfstests.orig/common/rc
+++ xfstests/common/rc
@@ -71,6 +71,13 @@ _btrfs_get_fsid()
$BTRFS_UTIL_PROG filesystem show $dev|awk '/uuid:/ {print $NF}'
 }
 
+_btrfs_get_feature_flags()
+{
+   local dev=$1
+   local class=$2
+   $BTRFS_SHOW_SUPER_PROG $dev | grep ^${class}_flags | awk '{print $NF}'
+}
+
 # Prints the md5 checksum of a given file
 _md5_checksum()
 {
Index: xfstests/src/btrfs_ioctl_helper.c
===
--- xfstests.orig/src/btrfs_ioctl_helper.c
+++ xfstests/src/btrfs_ioctl_helper.c
@@ -5,6 +5,7 @@
 #include string.h
 #include stdint.h
 #include unistd.h
+#include stdlib.h
 
 #ifndef BTRFS_IOCTL_MAGIC
 #define BTRFS_IOCTL_MAGIC 0x94
@@ -14,6 +15,21 @@
 #define BTRFS_IOC_GLOBAL_RSV _IOR(BTRFS_IOCTL_MAGIC, 20, uint64_t)
 #endif
 
+#ifndef BTRFS_IOC_GET_FEATURES
+struct btrfs_ioctl_feature_flags {
+   uint64_t compat_flags;
+   uint64_t compat_ro_flags;
+   uint64_t incompat_flags;
+};
+
+#define BTRFS_IOC_GET_FEATURES _IOR(BTRFS_IOCTL_MAGIC, 57, \
+   struct btrfs_ioctl_feature_flags)
+#define BTRFS_IOC_SET_FEATURES _IOW(BTRFS_IOCTL_MAGIC, 57, \
+   struct btrfs_ioctl_feature_flags[2])
+#define BTRFS_IOC_GET_SUPPORTED_FEATURES _IOR(BTRFS_IOCTL_MAGIC, 57, \
+   struct btrfs_ioctl_feature_flags[3])
+#endif
+
 static int global_rsv_ioctl(int fd, int argc, char *argv[])
 {
uint64_t reserved;
@@ -25,6 +41,76 @@ static int global_rsv_ioctl(int fd, int
return 0;
 }
 
+static int get_features_ioctl(int fd, int argc, char *argv[])
+{
+   struct btrfs_ioctl_feature_flags flags;
+   int ret = ioctl(fd, BTRFS_IOC_GET_FEATURES, flags);
+   if (ret)
+   return -errno;
+
+   printf(0x%llx 0x%llx 0x%llx\n,
+  (unsigned long long)flags.compat_flags,
+  (unsigned long long)flags.compat_ro_flags,
+  (unsigned long long)flags.incompat_flags);
+   return 0;
+}
+
+static int set_features_ioctl(int fd, int argc, char *argv[])
+{
+   struct btrfs_ioctl_feature_flags flags[2];
+   uint64_t bit, *bits, *mask;
+   if (argc != 3)
+   goto usage;
+
+   memset(flags, 0, sizeof(flags));
+
+   errno = 0;
+   bit = strtoull(argv[2], NULL, 10);
+   if (errno)
+   goto usage;
+
+   if (strcmp(argv[1], compat) == 0) {
+   mask = flags[0].compat_flags;
+   bits = flags[1].compat_flags;
+   } else if (strcmp(argv[1], compat_ro) == 0) {
+   mask = flags[0].compat_ro_flags;
+   bits = flags[1].compat_ro_flags;
+   } else if (strcmp(argv[1], incompat) == 0) {
+   mask = flags[0].incompat_flags;
+   bits = flags[1].incompat_flags;
+   } else
+   goto usage;
+
+   *mask |= bit;
+
+   if (strcmp(argv[0], set) == 0)
+   *bits |= bit;
+
+   return ioctl(fd, BTRFS_IOC_SET_FEATURES, flags);
+usage:
+   fprintf(stderr, usage: SET_FEATURES set|clear 
compat|compat_ro|incompat base-10 bitmask\n);
+   return -EINVAL;
+}
+
+static int get_supported_features_ioctl(int fd, int argc, char *argv[])
+{
+   struct btrfs_ioctl_feature_flags flags[3];
+   int ret;
+   int i;
+
+   ret = ioctl(fd, BTRFS_IOC_GET_SUPPORTED_FEATURES, flags);
+   if (ret)
+   return -errno;
+
+   for (i = 0; i  3; i++)
+   printf(0x%llx 0x%llx 0x%llx ,
+

Re: [patch 1/3] [PATCH 1/3] xfstests: add test for global metadata reservation publishing

2013-10-28 Thread Jeff Mahoney

On 10/28/13, 11:59 AM, Josef Bacik wrote:
 On Fri, Oct 25, 2013 at 04:52:47PM -0400, Jeff Mahoney wrote:
 My publishing patchset added the ability for the kernel to report
 the size of the global metadata reservation via ioctl and sysfs.

 This test confirms that we get sane results on an empty file system.

 ENOTTY and missing /sys/fs/btrfs/fsid/allocation are not considered
 failures.

 Signed-off-by: Jeff Mahoney je...@suse.com
 ---
  src/Makefile |  3 +-
  src/btrfs_ioctl_helper.c | 90 ++
  tests/btrfs/100  | 92 
 
  tests/btrfs/100.out  |  2 ++
  4 files changed, 186 insertions(+), 1 deletion(-)
  create mode 100644 src/btrfs_ioctl_helper.c
  create mode 100755 tests/btrfs/100
  create mode 100644 tests/btrfs/100.out

 diff --git a/src/Makefile b/src/Makefile
 index 84c8297..299f675 100644
 --- a/src/Makefile
 +++ b/src/Makefile
 @@ -18,7 +18,8 @@ LINUX_TARGETS = xfsctl bstat t_mtab getdevicesize 
 preallo_rw_pattern_reader \
  locktest unwritten_mmap bulkstat_unlink_test t_stripealign \
  bulkstat_unlink_test_modified t_dir_offset t_futimens t_immutable \
  stale_handle pwrite_mmap_blocked t_dir_offset2 seek_sanity_test \
 -seek_copy_test t_readdir_1 t_readdir_2 fsync-tester nsexec
 +seek_copy_test t_readdir_1 t_readdir_2 fsync-tester nsexec \
 +btrfs_ioctl_helper
  
  SUBDIRS =
  
 diff --git a/src/btrfs_ioctl_helper.c b/src/btrfs_ioctl_helper.c
 new file mode 100644
 index 000..a88b7b5
 --- /dev/null
 +++ b/src/btrfs_ioctl_helper.c
 @@ -0,0 +1,90 @@
 +#include sys/ioctl.h
 +#include stdio.h
 +#include sys/fcntl.h
 +#include errno.h
 +#include string.h
 +#include stdint.h
 +#include unistd.h
 +
 +#ifndef BTRFS_IOCTL_MAGIC
 +#define BTRFS_IOCTL_MAGIC 0x94
 +#endif
 +
 +#ifndef BTRFS_IOC_GLOBAL_RSV
 +#define BTRFS_IOC_GLOBAL_RSV _IOR(BTRFS_IOCTL_MAGIC, 20, uint64_t)
 +#endif
 +
 +static int global_rsv_ioctl(int fd, int argc, char *argv[])
 +{
 +uint64_t reserved;
 +int ret = ioctl(fd, BTRFS_IOC_GLOBAL_RSV, reserved);
 +if (ret)
 +return -errno;
 +
 +printf(%llu\n, (unsigned long long)reserved);
 +return 0;
 +}
 +
 +#define IOCTL_TABLE_ENTRY(_ioctl_name, _handler) \
 +{ .name = #_ioctl_name, .ioctl_cmd = BTRFS_IOC_##_ioctl_name, \
 +  .handler = _handler, }
 +
 +struct ioctl_table_entry {
 +const char *name;
 +unsigned ioctl_cmd;
 +int (*handler)(int fd, int argc, char *argv[]);
 +};
 +
 +static struct ioctl_table_entry ioctls[] = {
 +IOCTL_TABLE_ENTRY(GLOBAL_RSV, global_rsv_ioctl),
 +};
 +
 +int
 +main(int argc, char *argv[])
 +{
 +int fd;
 +int ret;
 +struct ioctl_table_entry *entry = NULL;
 +int i;
 +
 +if (argc  3) {
 +fprintf(stderr,
 +usage: %s fs mount point ioctl name [args..]\n,
 +argv[0]);
 +return 1;
 +}
 +
 +fd = open(argv[1], O_RDONLY|O_DIRECTORY);
 +if (fd  0) {
 +perror(argv[1]);
 +return 1;
 +}
 +
 +for (i = 0; i  (sizeof(ioctls)/sizeof(ioctls[0])); i++) {
 +if (strcmp(argv[2], ioctls[i].name) == 0) {
 +entry = ioctls[i];
 +break;
 +}
 +}
 +
 +if (!entry) {
 +fprintf(stderr, ERROR: unknown ioctl %s\n, argv[2]);
 +close(fd);
 +return 1;
 +}
 +
 +ret = entry-handler(fd, argc - 3, argv + 3);
 +if (ret == -ENOTTY) {
 +printf(Not implemented.\n);
 +close(fd);
 +return 0;
 +} else if (ret) {
 +fprintf(stderr, ERROR: %s failed: %s\n,
 +entry-name, strerror(-ret));
 +close(fd);
 +return 1;
 +}
 +
 +close(fd);
 +return 0;
 +}
 diff --git a/tests/btrfs/100 b/tests/btrfs/100
 new file mode 100755
 index 000..d2a40b4
 --- /dev/null
 +++ b/tests/btrfs/100
 @@ -0,0 +1,92 @@
 +#!/bin/bash
 +# FA QA Test No. 100
 +#
 +# Test global metadata reservation reporting
 +#
 +# 1) Create empty file system
 +# 2) Call the BTRFS_IOC_GLOBAL_RSV ioctl and confirm it is 0  x  10MB
 +# 3) Read the /sys/fs/btrfs/fsid/allocation/global_rsv_reserved file
 +#and confirm the value is 0  x  10 MB
 +#
 +#---
 +# Copyright (c) 2013 SUSE, All Rights Reserved.
 +#
 +# This program is free software; you can redistribute it and/or
 +# modify it under the terms of the GNU General Public License as
 +# published by the Free Software Foundation.
 +#
 +# This program is distributed in the hope that it would be useful,
 +# but WITHOUT ANY WARRANTY; without even the implied warranty of
 +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 +# GNU General Public License for more details.
 +#
 +# You should have received a copy of the GNU General Public License
 +# along with this program; if not, write

[PATCH 2/3] xfstests: btrfs/022: test sysfs exports of allocation and device membership info

2013-10-28 Thread Jeff Mahoney

This tests the sysfs publishing for btrfs allocation and device
membership info under a number of different layouts, similar to the
btrfs replace test. We test the allocation files only for existence and
that they contain numerical values. We test the device membership
by mapping the devices used to create the file system to sysfs paths
and matching them against the paths used for the device membership
symlinks.

Since this is a new feature, it passes on kernels without
a /sys/fs/btrfs/fsid directory.

Signed-off-by: Jeff Mahoney je...@suse.com
---
 common/config   |1 +
 tests/btrfs/022 |  190 +++
 tests/btrfs/022.out |2 +
 3 files changed, 193 insertions(+), 0 deletions(-)
 create mode 100755 tests/btrfs/022
 create mode 100644 tests/btrfs/022.out

Index: xfstests/common/config
===
--- xfstests.orig/common/config
+++ xfstests/common/config
@@ -209,6 +209,7 @@ case $HOSTOS in
 export MKFS_UDF_PROG=`set_prog_path mkudffs`
 export MKFS_BTRFS_PROG=`set_btrfs_mkfs_prog_path_with_opts`
 export BTRFS_UTIL_PROG=`set_prog_path btrfs`
+export UDEVADM_PROG=`set_prog_path udevadm`
 export BTRFS_SHOW_SUPER_PROG=`set_prog_path btrfs-show-super`
 export XFS_FSR_PROG=`set_prog_path xfs_fsr`
 export MKFS_NFS_PROG=false
Index: xfstests/tests/btrfs/022
===
--- /dev/null
+++ xfstests/tests/btrfs/022
@@ -0,0 +1,190 @@
+#! /bin/bash
+# FS QA Test No. 022
+#
+# Test of the btrfs sysfs publishing
+#
+#---
+# Copyright (C) 2013 SUSE.  All rights reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#
+#---
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo QA output created by $seq
+
+here=`pwd`
+tmp=/tmp/$$
+status=1
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# real QA test starts here
+_need_to_be_root
+_supported_fs btrfs
+_require_scratch
+_require_scratch_dev_pool
+_require_command $UDEVADM_PROG
+
+rm -f $seqres.full
+rm -f $tmp.tmp
+
+check_file() {
+   local file=$1
+   base=$(echo $file | sed -e 's#/sys/fs/btrfs/[0-9a-f-][0-9a-f-]*/##')
+   if [ ! -f $file ]; then
+   echo $base missing.
+   return 0
+   else
+   value=$(cat $file)
+   if [ -n $(echo $value | tr -d 0-9) ]; then
+   echo ERROR: $base: numerical value expected \
+(got $value)
+   return 0
+   fi
+   fi
+   return 1
+}
+
+check_chunk() {
+   path=$1
+   mkfs_options=$2
+   error=false
+
+   chunktype=$(basename $path)
+   if [ ! -d $path ]; then
+   echo No $chunktype directory.
+   exit 1
+   fi
+
+   for file in bytes_may_use bytes_pinned bytes_reserved bytes_used \
+   disk_total flags total_bytes total_bytes_pinned; do
+   if check_file $path/$file; then
+   error=true
+   fi
+   done
+
+   if [ $chunktype = data -o $chunktype = mixed ]; then
+   opt=-d
+   elif [ $chunktype = metadata -o $chunktype = system ]; then
+   opt=-m
+   fi
+
+   profile=$(echo $mkfs_options | sed -e s/.*$opt \([[:alnum:]]*\).*/\1/)
+   if [ ! -d $path/$profile ]; then
+   echo No $profile dir for $chunktype
+   exit 1
+   fi
+
+   for file in total_bytes used_bytes; do
+   if check_file $path/$profile/$file; then
+   error=true
+   fi
+   done
+
+   $error  exit 1
+}
+
+check_dev_link() {
+   local dev=$1
+   DEV=/sys/$($UDEVADM_PROG info --query=path $dev)
+   DEV=$(readlink -f $DEV)
+   found=false
+   for link in $sysfs_base/devices/*; do
+   LINK=$(readlink -f $link)
+   if [ $LINK = $DEV ]; then
+   found=true
+   break
+   fi
+   done
+   if ! $found; then
+   echo Symlink for $dev missing in $sysfs_base/devices
+   return 1
+   fi
+

Re: [PATCH 2/3] xfstests: btrfs/022: test sysfs exports of allocation and device membership info

2013-10-28 Thread Jeff Mahoney

On 10/28/13, 3:52 PM, Jeff Mahoney wrote:
 This tests the sysfs publishing for btrfs allocation and device
 membership info under a number of different layouts, similar to the
 btrfs replace test. We test the allocation files only for existence and
 that they contain numerical values. We test the device membership
 by mapping the devices used to create the file system to sysfs paths
 and matching them against the paths used for the device membership
 symlinks.
 
 Since this is a new feature, it passes on kernels without
 a /sys/fs/btrfs/fsid directory.
 
 Signed-off-by: Jeff Mahoney je...@suse.com
 ---
  common/config   |1 +
  tests/btrfs/022 |  190 
 +++
  tests/btrfs/022.out |2 +
  3 files changed, 193 insertions(+), 0 deletions(-)
  create mode 100755 tests/btrfs/022
  create mode 100644 tests/btrfs/022.out
 
 Index: xfstests/common/config
 ===
 --- xfstests.orig/common/config
 +++ xfstests/common/config
 @@ -209,6 +209,7 @@ case $HOSTOS in
  export MKFS_UDF_PROG=`set_prog_path mkudffs`
  export MKFS_BTRFS_PROG=`set_btrfs_mkfs_prog_path_with_opts`
  export BTRFS_UTIL_PROG=`set_prog_path btrfs`
 +export UDEVADM_PROG=`set_prog_path udevadm`
  export BTRFS_SHOW_SUPER_PROG=`set_prog_path btrfs-show-super`
  export XFS_FSR_PROG=`set_prog_path xfs_fsr`
  export MKFS_NFS_PROG=false
 Index: xfstests/tests/btrfs/022
 ===
 --- /dev/null
 +++ xfstests/tests/btrfs/022
 @@ -0,0 +1,190 @@
 +#! /bin/bash
 +# FS QA Test No. 022
 +#
 +# Test of the btrfs sysfs publishing
 +#
 +#---
 +# Copyright (C) 2013 SUSE.  All rights reserved.
 +#
 +# This program is free software; you can redistribute it and/or
 +# modify it under the terms of the GNU General Public License as
 +# published by the Free Software Foundation.
 +#
 +# This program is distributed in the hope that it would be useful,
 +# but WITHOUT ANY WARRANTY; without even the implied warranty of
 +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 +# GNU General Public License for more details.
 +#
 +# You should have received a copy of the GNU General Public License
 +# along with this program; if not, write the Free Software Foundation,
 +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
 +#
 +#---
 +#
 +
 +seq=`basename $0`
 +seqres=$RESULT_DIR/$seq
 +echo QA output created by $seq
 +
 +here=`pwd`
 +tmp=/tmp/$$
 +status=1
 +
 +# get standard environment, filters and checks
 +. ./common/rc
 +. ./common/filter
 +
 +# real QA test starts here
 +_need_to_be_root
 +_supported_fs btrfs
 +_require_scratch
 +_require_scratch_dev_pool
 +_require_command $UDEVADM_PROG
 +
 +rm -f $seqres.full
 +rm -f $tmp.tmp
 +
 +check_file() {
 + local file=$1
 + base=$(echo $file | sed -e 's#/sys/fs/btrfs/[0-9a-f-][0-9a-f-]*/##')
 + if [ ! -f $file ]; then
 + echo $base missing.
 + return 0
 + else
 + value=$(cat $file)
 + if [ -n $(echo $value | tr -d 0-9) ]; then
 + echo ERROR: $base: numerical value expected \
 +  (got $value)
 + return 0
 + fi
 + fi
 + return 1
 +}
 +
 +check_chunk() {
 + path=$1
 + mkfs_options=$2
 + error=false
 +
 + chunktype=$(basename $path)
 + if [ ! -d $path ]; then
 + echo No $chunktype directory.
 + exit 1
 + fi
 +
 + for file in bytes_may_use bytes_pinned bytes_reserved bytes_used \
 + disk_total flags total_bytes total_bytes_pinned; do
 + if check_file $path/$file; then
 + error=true
 + fi
 + done
 +
 + if [ $chunktype = data -o $chunktype = mixed ]; then
 + opt=-d
 + elif [ $chunktype = metadata -o $chunktype = system ]; then
 + opt=-m
 + fi
 +
 + profile=$(echo $mkfs_options | sed -e s/.*$opt \([[:alnum:]]*\).*/\1/)
 + if [ ! -d $path/$profile ]; then
 + echo No $profile dir for $chunktype
 + exit 1
 + fi
 +
 + for file in total_bytes used_bytes; do
 + if check_file $path/$profile/$file; then
 + error=true
 + fi
 + done
 +
 + $error  exit 1
 +}
 +
 +check_dev_link() {
 + local dev=$1
 + DEV=/sys/$($UDEVADM_PROG info --query=path $dev)
 + DEV=$(readlink -f $DEV)
 + found=false
 + for link in $sysfs_base/devices/*; do
 + LINK=$(readlink -f $link)
 + if [ $LINK = $DEV ]; then
 + found=true
 + break
 + fi
 + done
 + if ! $found; then
 + echo Symlink for

[patch 3/3] xfstests: btrfs/023,024,025: test feature ioctl and sysfs interfaces

2013-10-28 Thread Jeff Mahoney

This tests the exporting of feature information from the kernel via
sysfs and ioctl. The first test works whether the sysfs permissions
are correct, if the information exported via sysfs matches
what the ioctls are reporting, and if they both match the on-disk
superblock's version of the feature sets. The second and third tests
test online setting and clearing of feature bits via the sysfs and
ioctl interfaces, checking whether they match the on-disk super on
each cycle.

In every case, if the features are not present, it is not considered
a failure and a message indicating that will be dumped to the $num.full
file.

Signed-off-by: Jeff Mahoney je...@suse.com
---
 common/rc|7 ++
 src/btrfs_ioctl_helper.c |   93 
 tests/btrfs/023  |  269 ++
 tests/btrfs/023.out  |2 +
 tests/btrfs/024  |  185 +++
 tests/btrfs/024.out  |2 +
 tests/btrfs/025  |  178 ++
 tests/btrfs/025.out  |2 +
 8 files changed, 738 insertions(+), 0 deletions(-)
 create mode 100755 tests/btrfs/023
 create mode 100644 tests/btrfs/023.out
 create mode 100755 tests/btrfs/024
 create mode 100644 tests/btrfs/024.out
 create mode 100755 tests/btrfs/025
 create mode 100644 tests/btrfs/025.out

Index: xfstests/common/rc
===
--- xfstests.orig/common/rc
+++ xfstests/common/rc
@@ -71,6 +71,13 @@ _btrfs_get_fsid()
$BTRFS_UTIL_PROG filesystem show $dev|awk '/uuid:/ {print $NF}'
 }
 
+_btrfs_get_feature_flags()
+{
+   local dev=$1
+   local class=$2
+   $BTRFS_SHOW_SUPER_PROG $dev | grep ^${class}_flags | awk '{print $NF}'
+}
+
 # Prints the md5 checksum of a given file
 _md5_checksum()
 {
Index: xfstests/src/btrfs_ioctl_helper.c
===
--- xfstests.orig/src/btrfs_ioctl_helper.c
+++ xfstests/src/btrfs_ioctl_helper.c
@@ -5,6 +5,7 @@
 #include string.h
 #include stdint.h
 #include unistd.h
+#include stdlib.h
 
 #ifndef BTRFS_IOCTL_MAGIC
 #define BTRFS_IOCTL_MAGIC 0x94
@@ -14,6 +15,21 @@
 #define BTRFS_IOC_GLOBAL_RSV _IOR(BTRFS_IOCTL_MAGIC, 20, uint64_t)
 #endif
 
+#ifndef BTRFS_IOC_GET_FEATURES
+struct btrfs_ioctl_feature_flags {
+   uint64_t compat_flags;
+   uint64_t compat_ro_flags;
+   uint64_t incompat_flags;
+};
+
+#define BTRFS_IOC_GET_FEATURES _IOR(BTRFS_IOCTL_MAGIC, 57, \
+   struct btrfs_ioctl_feature_flags)
+#define BTRFS_IOC_SET_FEATURES _IOW(BTRFS_IOCTL_MAGIC, 57, \
+   struct btrfs_ioctl_feature_flags[2])
+#define BTRFS_IOC_GET_SUPPORTED_FEATURES _IOR(BTRFS_IOCTL_MAGIC, 57, \
+   struct btrfs_ioctl_feature_flags[3])
+#endif
+
 static int global_rsv_ioctl(int fd, int argc, char *argv[])
 {
uint64_t reserved;
@@ -25,6 +41,76 @@ static int global_rsv_ioctl(int fd, int
return 0;
 }
 
+static int get_features_ioctl(int fd, int argc, char *argv[])
+{
+   struct btrfs_ioctl_feature_flags flags;
+   int ret = ioctl(fd, BTRFS_IOC_GET_FEATURES, flags);
+   if (ret)
+   return -errno;
+
+   printf(0x%llx 0x%llx 0x%llx\n,
+  (unsigned long long)flags.compat_flags,
+  (unsigned long long)flags.compat_ro_flags,
+  (unsigned long long)flags.incompat_flags);
+   return 0;
+}
+
+static int set_features_ioctl(int fd, int argc, char *argv[])
+{
+   struct btrfs_ioctl_feature_flags flags[2];
+   uint64_t bit, *bits, *mask;
+   if (argc != 3)
+   goto usage;
+
+   memset(flags, 0, sizeof(flags));
+
+   errno = 0;
+   bit = strtoull(argv[2], NULL, 10);
+   if (errno)
+   goto usage;
+
+   if (strcmp(argv[1], compat) == 0) {
+   mask = flags[0].compat_flags;
+   bits = flags[1].compat_flags;
+   } else if (strcmp(argv[1], compat_ro) == 0) {
+   mask = flags[0].compat_ro_flags;
+   bits = flags[1].compat_ro_flags;
+   } else if (strcmp(argv[1], incompat) == 0) {
+   mask = flags[0].incompat_flags;
+   bits = flags[1].incompat_flags;
+   } else
+   goto usage;
+
+   *mask |= bit;
+
+   if (strcmp(argv[0], set) == 0)
+   *bits |= bit;
+
+   return ioctl(fd, BTRFS_IOC_SET_FEATURES, flags);
+usage:
+   fprintf(stderr, usage: SET_FEATURES set|clear 
compat|compat_ro|incompat base-10 bitmask\n);
+   return -EINVAL;
+}
+
+static int get_supported_features_ioctl(int fd, int argc, char *argv[])
+{
+   struct btrfs_ioctl_feature_flags flags[3];
+   int ret;
+   int i;
+
+   ret = ioctl(fd, BTRFS_IOC_GET_SUPPORTED_FEATURES, flags);
+   if (ret)
+   return -errno;
+
+   for (i = 0; i  3; i++)
+   printf(0x%llx 0x%llx 0x%llx ,
+

[patch 1/3] xfstests: btrfs/021: test global metadata reservation reporting

2013-10-28 Thread Jeff Mahoney

Btrfs can now report the size of the global metadata reservation
via icotl and sysfs.

This test confirms that we get sane results on an empty file system.

ENOTTY and missing /sys/fs/btrfs/fsid/allocation are not considered
failures.

Signed-off-by: Jeff Mahoney je...@suse.com
---
 common/rc|6 +++
 src/Makefile |3 +-
 src/btrfs_ioctl_helper.c |   90 ++
 tests/btrfs/021  |   88 
 tests/btrfs/021.out  |2 +
 5 files changed, 188 insertions(+), 1 deletions(-)
 create mode 100644 src/btrfs_ioctl_helper.c
 create mode 100755 tests/btrfs/021
 create mode 100644 tests/btrfs/021.out

Index: xfstests/common/rc
===
--- xfstests.orig/common/rc
+++ xfstests/common/rc
@@ -65,6 +65,12 @@ _btrfs_get_subvolid()
$BTRFS_UTIL_PROG sub list $mnt | grep $name | awk '{ print $2 }'
 }
 
+_btrfs_get_fsid()
+{
+   local dev=$1
+   $BTRFS_UTIL_PROG filesystem show $dev|awk '/uuid:/ {print $NF}'
+}
+
 # Prints the md5 checksum of a given file
 _md5_checksum()
 {
Index: xfstests/src/Makefile
===
--- xfstests.orig/src/Makefile
+++ xfstests/src/Makefile
@@ -18,7 +18,8 @@ LINUX_TARGETS = xfsctl bstat t_mtab getd
locktest unwritten_mmap bulkstat_unlink_test t_stripealign \
bulkstat_unlink_test_modified t_dir_offset t_futimens t_immutable \
stale_handle pwrite_mmap_blocked t_dir_offset2 seek_sanity_test \
-   seek_copy_test t_readdir_1 t_readdir_2 fsync-tester nsexec
+   seek_copy_test t_readdir_1 t_readdir_2 fsync-tester nsexec \
+   btrfs_ioctl_helper
 
 SUBDIRS =
 
Index: xfstests/src/btrfs_ioctl_helper.c
===
--- /dev/null
+++ xfstests/src/btrfs_ioctl_helper.c
@@ -0,0 +1,90 @@
+#include sys/ioctl.h
+#include stdio.h
+#include sys/fcntl.h
+#include errno.h
+#include string.h
+#include stdint.h
+#include unistd.h
+
+#ifndef BTRFS_IOCTL_MAGIC
+#define BTRFS_IOCTL_MAGIC 0x94
+#endif
+
+#ifndef BTRFS_IOC_GLOBAL_RSV
+#define BTRFS_IOC_GLOBAL_RSV _IOR(BTRFS_IOCTL_MAGIC, 20, uint64_t)
+#endif
+
+static int global_rsv_ioctl(int fd, int argc, char *argv[])
+{
+   uint64_t reserved;
+   int ret = ioctl(fd, BTRFS_IOC_GLOBAL_RSV, reserved);
+   if (ret)
+   return -errno;
+
+   printf(%llu\n, (unsigned long long)reserved);
+   return 0;
+}
+
+#define IOCTL_TABLE_ENTRY(_ioctl_name, _handler) \
+   { .name = #_ioctl_name, .ioctl_cmd = BTRFS_IOC_##_ioctl_name, \
+ .handler = _handler, }
+
+struct ioctl_table_entry {
+   const char *name;
+   unsigned ioctl_cmd;
+   int (*handler)(int fd, int argc, char *argv[]);
+};
+
+static struct ioctl_table_entry ioctls[] = {
+   IOCTL_TABLE_ENTRY(GLOBAL_RSV, global_rsv_ioctl),
+};
+
+int
+main(int argc, char *argv[])
+{
+   int fd;
+   int ret;
+   struct ioctl_table_entry *entry = NULL;
+   int i;
+
+   if (argc  3) {
+   fprintf(stderr,
+   usage: %s fs mount point ioctl name [args..]\n,
+   argv[0]);
+   return 1;
+   }
+
+   fd = open(argv[1], O_RDONLY|O_DIRECTORY);
+   if (fd  0) {
+   perror(argv[1]);
+   return 1;
+   }
+
+   for (i = 0; i  (sizeof(ioctls)/sizeof(ioctls[0])); i++) {
+   if (strcmp(argv[2], ioctls[i].name) == 0) {
+   entry = ioctls[i];
+   break;
+   }
+   }
+
+   if (!entry) {
+   fprintf(stderr, ERROR: unknown ioctl %s\n, argv[2]);
+   close(fd);
+   return 1;
+   }
+
+   ret = entry-handler(fd, argc - 3, argv + 3);
+   if (ret == -ENOTTY) {
+   printf(Not implemented.\n);
+   close(fd);
+   return 0;
+   } else if (ret) {
+   fprintf(stderr, ERROR: %s failed: %s\n,
+   entry-name, strerror(-ret));
+   close(fd);
+   return 1;
+   }
+
+   close(fd);
+   return 0;
+}
Index: xfstests/tests/btrfs/021
===
--- /dev/null
+++ xfstests/tests/btrfs/021
@@ -0,0 +1,88 @@
+#!/bin/bash
+# FA QA Test No. 021
+#
+# Test global metadata reservation reporting
+#
+# 1) Create empty file system
+# 2) Call the BTRFS_IOC_GLOBAL_RSV ioctl and confirm it is 0  x  10MB
+# 3) Read the /sys/fs/btrfs/fsid/allocation/global_rsv_reserved file
+#and confirm the value is 0  x  10 MB
+#
+#---
+# Copyright (c) 2013 SUSE, All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free

[patch 0/3] xfstests: for btrfs publishing/export patchset (v3)

2013-10-28 Thread Jeff Mahoney

These are the tests that Josef requested accompany the submission of
the btrfs publishing/export patchset[1].

021: Test new global metadata reservation ioctl
022: Test new sysfs exporting of allocation and device membership
023: Test new sysfs exporting of feature bits
024: Test setting/clearing feature bits via ioctl
025: Test setting/clearing feature bits via sysfs

As part of the ioctl testing, I added a helper that can be extended
easily to export simple information via ioctl. This means that the
tests can be independent of btrfsprogs version or its output formatting.

V2:
- Used next numbers in sequence
- Better commit messages
- Fixed handling of the missing feature cases
- Fixed issues raised in Josef's review

V3:
- Fixed missing output in btrfs/023 when the ioctls arent't supported
- Fixed missing umount introduced in V2

-Jeff

[1] http://thread.gmane.org/gmane.comp.file-systems.btrfs/29307

--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[patch 2/3] xfstests: btrfs/022: test sysfs exports of allocation and device membership info

2013-10-28 Thread Jeff Mahoney

This tests the sysfs publishing for btrfs allocation and device
membership info under a number of different layouts, similar to the
btrfs replace test. We test the allocation files only for existence and
that they contain numerical values. We test the device membership
by mapping the devices used to create the file system to sysfs paths
and matching them against the paths used for the device membership
symlinks.

Since this is a new feature, it passes on kernels without
a /sys/fs/btrfs/fsid directory.

Signed-off-by: Jeff Mahoney je...@suse.com
---
 common/config   |1 +
 tests/btrfs/022 |  190 +++
 tests/btrfs/022.out |2 +
 3 files changed, 193 insertions(+), 0 deletions(-)
 create mode 100755 tests/btrfs/022
 create mode 100644 tests/btrfs/022.out

Index: xfstests/common/config
===
--- xfstests.orig/common/config
+++ xfstests/common/config
@@ -209,6 +209,7 @@ case $HOSTOS in
 export MKFS_UDF_PROG=`set_prog_path mkudffs`
 export MKFS_BTRFS_PROG=`set_btrfs_mkfs_prog_path_with_opts`
 export BTRFS_UTIL_PROG=`set_prog_path btrfs`
+export UDEVADM_PROG=`set_prog_path udevadm`
 export BTRFS_SHOW_SUPER_PROG=`set_prog_path btrfs-show-super`
 export XFS_FSR_PROG=`set_prog_path xfs_fsr`
 export MKFS_NFS_PROG=false
Index: xfstests/tests/btrfs/022
===
--- /dev/null
+++ xfstests/tests/btrfs/022
@@ -0,0 +1,190 @@
+#! /bin/bash
+# FS QA Test No. 022
+#
+# Test of the btrfs sysfs publishing
+#
+#---
+# Copyright (C) 2013 SUSE.  All rights reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#
+#---
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo QA output created by $seq
+
+here=`pwd`
+tmp=/tmp/$$
+status=1
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# real QA test starts here
+_need_to_be_root
+_supported_fs btrfs
+_require_scratch
+_require_scratch_dev_pool
+_require_command $UDEVADM_PROG
+
+rm -f $seqres.full
+rm -f $tmp.tmp
+
+check_file() {
+   local file=$1
+   base=$(echo $file | sed -e 's#/sys/fs/btrfs/[0-9a-f-][0-9a-f-]*/##')
+   if [ ! -f $file ]; then
+   echo $base missing.
+   return 0
+   else
+   value=$(cat $file)
+   if [ -n $(echo $value | tr -d 0-9) ]; then
+   echo ERROR: $base: numerical value expected \
+(got $value)
+   return 0
+   fi
+   fi
+   return 1
+}
+
+check_chunk() {
+   path=$1
+   mkfs_options=$2
+   error=false
+
+   chunktype=$(basename $path)
+   if [ ! -d $path ]; then
+   echo No $chunktype directory.
+   exit 1
+   fi
+
+   for file in bytes_may_use bytes_pinned bytes_reserved bytes_used \
+   disk_total flags total_bytes total_bytes_pinned; do
+   if check_file $path/$file; then
+   error=true
+   fi
+   done
+
+   if [ $chunktype = data -o $chunktype = mixed ]; then
+   opt=-d
+   elif [ $chunktype = metadata -o $chunktype = system ]; then
+   opt=-m
+   fi
+
+   profile=$(echo $mkfs_options | sed -e s/.*$opt \([[:alnum:]]*\).*/\1/)
+   if [ ! -d $path/$profile ]; then
+   echo No $profile dir for $chunktype
+   exit 1
+   fi
+
+   for file in total_bytes used_bytes; do
+   if check_file $path/$profile/$file; then
+   error=true
+   fi
+   done
+
+   $error  exit 1
+}
+
+check_dev_link() {
+   local dev=$1
+   DEV=/sys/$($UDEVADM_PROG info --query=path $dev)
+   DEV=$(readlink -f $DEV)
+   found=false
+   for link in $sysfs_base/devices/*; do
+   LINK=$(readlink -f $link)
+   if [ $LINK = $DEV ]; then
+   found=true
+   break
+   fi
+   done
+   if ! $found; then
+   echo Symlink for $dev missing in $sysfs_base/devices
+   return 1
+   fi
+

memory leak in =3.11.6

2013-10-28 Thread Kai Krakow

Hello!

Is this output in slabtop normal after a few days uptime with a daily rsync 
(btrfs to btrfs) and some compiling (gentoo emerge):

 Active / Total Objects (% used): 1120836 / 1810907 (61,9%)
 Active / Total Slabs (% used)  : 42492 / 42492 (100,0%)
 Active / Total Caches (% used) : 72 / 101 (71,3%)
 Active / Total Size (% used)   : 253898,56K / 324674,92K (78,2%)
 Minimum / Average / Maximum Object : 0,01K / 0,18K / 15,69K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME   
109220  99107  90%0,98K   6830   16109280K btrfs_inode

If I close all my desktop programs, my system stays at +12 GB RAM usage 
while after a fresh boot it has 12+ GB free (of 16 GB). Cache stays low at 2 
GB while after a few minutes uptime cache is about 5 GB.

This looks like memory leaking somewhere. Top does not show where all this 
memory went (biggest process is 500 MB RES, next is 200 MB and lower).

Thanks,
Kai

--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Null pointer dereference bug in btrfs_find_all_root

2013-10-28 Thread Liu Bo

Hi, 

On Mon, Oct 28, 2013 at 05:02:11PM +0100, Pedro Fonseca wrote:
 Hi,
 
 I've encountered a few NULL pointer dereference bugs while running a
 custom test case on btrfs (kernel 3.11.1), inside a QEMU based VM.
 
 Btrfs was statically compiled and the btrfs filesystem was mounted
 on one or two loop devices. All cases occurred during execution of a
 filesystem balance operation and while another file system
 operation was concurrently executed. In one case the other operation
 was a mknod and in another it was a filesystem show operation.

I'm wonderring that if there is anything like 'aborted transaction' in
the dmesg log besides this NULL pointer dereference?

-liubo

 
 In all cases the NULL pointer dereference occurred in the
 btrfs_find_all_roots function and all cases produced a bug report
 similar to the one bellow:
 
 [  138.573390] BUG: unable to handle kernel NULL pointer
 dereference at   (null)
 [  138.573390] IP: [c12feeb3] ulist_next+0x4/0x26
 [  138.573390] *pde = 
 [  138.573390] Oops:  [#1] SMP
 [  138.573390] Modules linked in: loop rtc_cmos pcspkr tpm_tis
 freq_table mperf i2c_piix4
 [  138.573390] CPU: 0 PID: 2794 Comm: btrfs-endio-wri Not tainted
 3.11.1 #2
 [  138.573390] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2007
 [  138.573390] task: c5dea6b0 ti: c5e12000 task.ti: c5e12000
 [  138.573390] EIP: 0060:[c12feeb3] EFLAGS: 0246 CPU: 0
 [  138.573390] EIP is at ulist_next+0x4/0x26
 [  138.573390] EAX:  EBX:  ECX: d9c29e38 EDX: c5e13da4
 [  138.573390] ESI:  EDI: c5e13e00 EBP: c5e13d94 ESP: c5e13d90
 [  138.573390]  DS: 007b ES: 007b FS: 00d8 GS:  SS: 0068
 [  138.573390] CR0: 8005003b CR2:  CR3: 059d8000 CR4: 0690
 [  138.573390] DR0:  DR1:  DR2:  DR3: 
 [  138.573390] DR6:  DR7: 
 [  138.573390] Stack:
 [  138.573390]   c5e13db4 c12fe7b8 de4d1000 d9cead48
 0001 010dd000 deeab810
 [  138.573390]  0001 c5e13e10 c12fe8cf 00af2000 
  0003 c5e13e00
 [  138.573390]  deeabe00  0003 deeab800 de4d1000
 d9cead48 de4d14b8 de4d14b8
 [  138.573390] Call Trace:
 [  138.573390]  [c12fe7b8] btrfs_find_all_roots+0x88/0xa6
 [  138.573390]  [c12fe8cf] iterate_extent_inodes+0xf9/0x1b2
 [  138.573390]  [c12feba9] iterate_inodes_from_logical+0x88/0x97
 [  138.573390]  [c12b8880] ? record_extent_backrefs+0x8b/0x8b
 [  138.573390]  [c12b8845] record_extent_backrefs+0x50/0x8b
 [  138.573390]  [c12b8880] ? record_extent_backrefs+0x8b/0x8b
 [  138.573390]  [c12c1f8e] btrfs_finish_ordered_io+0x77e/0x876
 [  138.573390]  [c12c2091] finish_ordered_fn+0xb/0xd
 [  138.573390]  [c12db161] worker_loop+0x111/0x3ec
 [  138.573390]  [c104a2c4] ? default_wake_function+0xb/0xd
 [  138.573390]  [c1040c43] kthread+0x90/0x95
 [  138.573390]  [c12db050] ? btrfs_queue_worker+0x1c8/0x1c8
 [  138.573390]  [c176c277] ret_from_kernel_thread+0x1b/0x28
 [  138.573390]  [c1040bb3] ? kthread_freezable_should_stop+0x4f/0x4f
 [  138.573390] Code: 5b 5e 5f 5d c3 90 90 90 55 8d 50 10 89 e5 c7
 00 00 00 00 00 89 50 08 c7 40 04 10 00 00 00 c7 40 0c 00 00 00 00
 5d c3 55 89 e5 53 8b 08 85 c9 74 15 8b 1a 85 db 78 0f 39 cb 73
 0b 6b cb 1c 43 03
 [  138.573390] EIP: [c12feeb3] ulist_next+0x4/0x26 SS:ESP 0068:c5e13d90
 [  138.573390] CR2: 
 [  138.573390] ---[ end trace 626899e1abbe ]---
 [  138.573390] BUG: unable to handle kernel NULL pointer
 dereference at   (null)
 [  138.573390] IP: [c12feeb3] ulist_next+0x4/0x26
 [  138.573390] *pde = 
 [  138.573390] Oops:  [#2] SMP
 [  138.573390] Modules linked in: loop rtc_cmos pcspkr tpm_tis
 freq_table mperf i2c_piix4
 [  138.573390] CPU: 1 PID: 2793 Comm: btrfs-endio-wri Tainted: G
 D  3.11.1 #2
 [  138.573390] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2007
 [  138.573390] task: c59cc190 ti: dedc8000 task.ti: dedc8000
 [  138.573390] EIP: 0060:[c12feeb3] EFLAGS: 0246 CPU: 1
 [  138.573390] EIP is at ulist_next+0x4/0x26
 [  138.573390] EAX:  EBX:  ECX: d9c296c8 EDX: dedc9da4
 [  138.573390] ESI:  EDI: dedc9e00 EBP: dedc9d94 ESP: dedc9d90
 [  138.573390]  DS: 007b ES: 007b FS: 00d8 GS:  SS: 0068
 [  138.573390] CR0: 8005003b CR2:  CR3: 03ebe000 CR4: 0690
 [  138.573390] DR0:  DR1:  DR2:  DR3: 
 [  138.573390] DR6:  DR7: 
 [  138.573390] Stack:
 [  138.573390]   dedc9db4 c12fe7b8 de4d1000 d9ceaf50
 0001 00dea000 de4c6410
 [  138.573390]  0001 dedc9e10 c12fe8cf 00af2000 
  0005 dedc9e00
 [  138.573390]  de4c6c00  0005 de4c6400 de4d1000
 d9ceaf50 de4d14b8 c5e13de8
 [  138.573390] Call Trace:
 [  138.573390]  [c12fe7b8] btrfs_find_all_roots+0x88/0xa6
 [  138.573390]  [c12fe8cf] iterate_extent_inodes+0xf9/0x1b2
 [  138.573390]  [c12feba9] iterate_inodes_from_logical+0x88/0x97
 [  138.573390]  [c12b8880] ? record_extent_backrefs+0x8b/0x8b
 [  138.573390]  [c12b8845]

[PATCH] Btrfs: do not run snapshot-aware defragment on error

2013-10-28 Thread Liu Bo

If something wrong happens in write endio, running snapshot-aware defragment
can end up with undefined results, maybe a crash, so we should avoid it.

In order to share similar code, this also adds a helper to free the struct for
snapshot-aware defrag.

Signed-off-by: Liu Bo bo.li@oracle.com
---
 fs/btrfs/inode.c | 47 ---
 1 file changed, 28 insertions(+), 19 deletions(-)

diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c
index 51e3afa..8b8eff0 100644
--- a/fs/btrfs/inode.c
+++ b/fs/btrfs/inode.c
@@ -2367,10 +2367,23 @@ out_unlock:
return ret;
 }
 
+static void free_sa_defrag_extent(struct new_sa_defrag_extent *new)
+{
+   struct old_sa_defrag_extent *old, *tmp;
+
+   if (!new)
+   return;
+
+   list_for_each_entry_safe(old, tmp, new-head, list) {
+   list_del(old-list);
+   kfree(old);
+   }
+   kfree(new);
+}
+
 static void relink_file_extents(struct new_sa_defrag_extent *new)
 {
struct btrfs_path *path;
-   struct old_sa_defrag_extent *old, *tmp;
struct sa_defrag_extent_backref *backref;
struct sa_defrag_extent_backref *prev = NULL;
struct inode *inode;
@@ -2413,16 +2426,11 @@ static void relink_file_extents(struct 
new_sa_defrag_extent *new)
kfree(prev);
 
btrfs_free_path(path);
-
-   list_for_each_entry_safe(old, tmp, new-head, list) {
-   list_del(old-list);
-   kfree(old);
-   }
 out:
+   free_sa_defrag_extent(new);
+
atomic_dec(root-fs_info-defrag_running);
wake_up(root-fs_info-transaction_wait);
-
-   kfree(new);
 }
 
 static struct new_sa_defrag_extent *
@@ -2432,7 +2440,7 @@ record_old_file_extents(struct inode *inode,
struct btrfs_root *root = BTRFS_I(inode)-root;
struct btrfs_path *path;
struct btrfs_key key;
-   struct old_sa_defrag_extent *old, *tmp;
+   struct old_sa_defrag_extent *old;
struct new_sa_defrag_extent *new;
int ret;
 
@@ -2480,7 +2488,7 @@ record_old_file_extents(struct inode *inode,
if (slot = btrfs_header_nritems(l)) {
ret = btrfs_next_leaf(root, path);
if (ret  0)
-   goto out_free_list;
+   goto out_free_path;
else if (ret  0)
break;
continue;
@@ -2509,7 +2517,7 @@ record_old_file_extents(struct inode *inode,
 
old = kmalloc(sizeof(*old), GFP_NOFS);
if (!old)
-   goto out_free_list;
+   goto out_free_path;
 
offset = max(new-file_pos, key.offset);
end = min(new-file_pos + new-len, key.offset + num_bytes);
@@ -2531,15 +2539,10 @@ next:
 
return new;
 
-out_free_list:
-   list_for_each_entry_safe(old, tmp, new-head, list) {
-   list_del(old-list);
-   kfree(old);
-   }
 out_free_path:
btrfs_free_path(path);
 out_kfree:
-   kfree(new);
+   free_sa_defrag_extent(new);
return NULL;
 }
 
@@ -2710,8 +2713,14 @@ out:
btrfs_remove_ordered_extent(inode, ordered_extent);
 
/* for snapshot-aware defrag */
-   if (new)
-   relink_file_extents(new);
+   if (new) {
+   if (ret) {
+   free_sa_defrag_extent(new);
+   atomic_dec(root-fs_info-defrag_running);
+   } else {
+   relink_file_extents(new);
+   }
+   }
 
/* once for us */
btrfs_put_ordered_extent(ordered_extent);
-- 
1.8.2.1

--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: memory leak in =3.11.6

2013-10-28 Thread Jérôme Poulin

On Mon, Oct 28, 2013 at 6:34 PM, Kai Krakow hurikhan77+bt...@gmail.com wrote:

 If I close all my desktop programs, my system stays at +12 GB RAM usage
 while after a fresh boot it has 12+ GB free (of 16 GB). Cache stays low at 2
 GB while after a few minutes uptime cache is about 5 GB.

I probably have the same problem over here, after about 2 weeks of
random read/write it seems my memory and swap get almost full and even
after killing all process and getting in single user mode, memory
won't free up.  Would you happen to have quota enabled too? Kernel is
3.11.4
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Null pointer dereference bug in btrfs_find_all_root

2013-10-28 Thread Pedro Fonseca



On 10/29/13 3:05 AM, Liu Bo wrote:

Hi,

On Mon, Oct 28, 2013 at 05:02:11PM +0100, Pedro Fonseca wrote:

Hi,

I've encountered a few NULL pointer dereference bugs while running a
custom test case on btrfs (kernel 3.11.1), inside a QEMU based VM.

Btrfs was statically compiled and the btrfs filesystem was mounted
on one or two loop devices. All cases occurred during execution of a
filesystem balance operation and while another file system
operation was concurrently executed. In one case the other operation
was a mknod and in another it was a filesystem show operation.

I'm wonderring that if there is anything like 'aborted transaction' in
the dmesg log besides this NULL pointer dereference?

-liubo
No, I didn't find any references to aborted transactions. But I did get 
a message about relocation of block groups and a few other messages 
caused by resizing operations.


These are all the filesystem messages I got before the actual bug report:

[  111.836409] loop: module loaded
[  112.939586] device fsid acb9f83e-75d7-4177-9f7b-4b7c8e0f53d9 devid 1 
transid 7 /dev/loop0

[  122.328339] btrfs: new size for /dev/loop0 is 311427072
[  122.396188] btrfs: new size for /dev/loop0 is 301989888
[  122.646333] btrfs: new size for /dev/loop0 is 299892736
[  127.788383] btrfs: new size for /dev/loop0 is 305135616
[  127.830656] btrfs: relocating block group 20971520 flags 1


In all cases the NULL pointer dereference occurred in the
btrfs_find_all_roots function and all cases produced a bug report
similar to the one bellow:


[  138.573390] BUG: unable to handle kernel NULL pointer
dereference at   (null)
[  138.573390] IP: [c12feeb3] ulist_next+0x4/0x26
[  138.573390] *pde = 
[  138.573390] Oops:  [#1] SMP
[  138.573390] Modules linked in: loop rtc_cmos pcspkr tpm_tis
freq_table mperf i2c_piix4
[  138.573390] CPU: 0 PID: 2794 Comm: btrfs-endio-wri Not tainted
3.11.1 #2
[  138.573390] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2007
[  138.573390] task: c5dea6b0 ti: c5e12000 task.ti: c5e12000
[  138.573390] EIP: 0060:[c12feeb3] EFLAGS: 0246 CPU: 0
[  138.573390] EIP is at ulist_next+0x4/0x26
[  138.573390] EAX:  EBX:  ECX: d9c29e38 EDX: c5e13da4
[  138.573390] ESI:  EDI: c5e13e00 EBP: c5e13d94 ESP: c5e13d90
[  138.573390]  DS: 007b ES: 007b FS: 00d8 GS:  SS: 0068
[  138.573390] CR0: 8005003b CR2:  CR3: 059d8000 CR4: 0690
[  138.573390] DR0:  DR1:  DR2:  DR3: 
[  138.573390] DR6:  DR7: 
[  138.573390] Stack:
[  138.573390]   c5e13db4 c12fe7b8 de4d1000 d9cead48
0001 010dd000 deeab810
[  138.573390]  0001 c5e13e10 c12fe8cf 00af2000 
 0003 c5e13e00
[  138.573390]  deeabe00  0003 deeab800 de4d1000
d9cead48 de4d14b8 de4d14b8
[  138.573390] Call Trace:
[  138.573390]  [c12fe7b8] btrfs_find_all_roots+0x88/0xa6
[  138.573390]  [c12fe8cf] iterate_extent_inodes+0xf9/0x1b2
[  138.573390]  [c12feba9] iterate_inodes_from_logical+0x88/0x97
[  138.573390]  [c12b8880] ? record_extent_backrefs+0x8b/0x8b
[  138.573390]  [c12b8845] record_extent_backrefs+0x50/0x8b
[  138.573390]  [c12b8880] ? record_extent_backrefs+0x8b/0x8b
[  138.573390]  [c12c1f8e] btrfs_finish_ordered_io+0x77e/0x876
[  138.573390]  [c12c2091] finish_ordered_fn+0xb/0xd
[  138.573390]  [c12db161] worker_loop+0x111/0x3ec
[  138.573390]  [c104a2c4] ? default_wake_function+0xb/0xd
[  138.573390]  [c1040c43] kthread+0x90/0x95
[  138.573390]  [c12db050] ? btrfs_queue_worker+0x1c8/0x1c8
[  138.573390]  [c176c277] ret_from_kernel_thread+0x1b/0x28
[  138.573390]  [c1040bb3] ? kthread_freezable_should_stop+0x4f/0x4f
[  138.573390] Code: 5b 5e 5f 5d c3 90 90 90 55 8d 50 10 89 e5 c7
00 00 00 00 00 89 50 08 c7 40 04 10 00 00 00 c7 40 0c 00 00 00 00
5d c3 55 89 e5 53 8b 08 85 c9 74 15 8b 1a 85 db 78 0f 39 cb 73
0b 6b cb 1c 43 03
[  138.573390] EIP: [c12feeb3] ulist_next+0x4/0x26 SS:ESP 0068:c5e13d90
[  138.573390] CR2: 
[  138.573390] ---[ end trace 626899e1abbe ]---
[  138.573390] BUG: unable to handle kernel NULL pointer
dereference at   (null)
[  138.573390] IP: [c12feeb3] ulist_next+0x4/0x26
[  138.573390] *pde = 
[  138.573390] Oops:  [#2] SMP
[  138.573390] Modules linked in: loop rtc_cmos pcspkr tpm_tis
freq_table mperf i2c_piix4
[  138.573390] CPU: 1 PID: 2793 Comm: btrfs-endio-wri Tainted: G
D  3.11.1 #2
[  138.573390] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2007
[  138.573390] task: c59cc190 ti: dedc8000 task.ti: dedc8000
[  138.573390] EIP: 0060:[c12feeb3] EFLAGS: 0246 CPU: 1
[  138.573390] EIP is at ulist_next+0x4/0x26
[  138.573390] EAX:  EBX:  ECX: d9c296c8 EDX: dedc9da4
[  138.573390] ESI:  EDI: dedc9e00 EBP: dedc9d94 ESP: dedc9d90
[  138.573390]  DS: 007b ES: 007b FS: 00d8 GS:  SS: 0068
[  138.573390] CR0: 8005003b CR2:  CR3: 03ebe000 CR4: 0690
[  138.573390] DR0:  DR1:  DR2:  DR3: 
[  138.573390]

Re: memory leak in =3.11.6

2013-10-28 Thread Tomasz Chmielewski

 I probably have the same problem over here, after about 2 weeks of
 random read/write it seems my memory and swap get almost full and even
 after killing all process and getting in single user mode, memory
 won't free up.  Would you happen to have quota enabled too? Kernel is
 3.11.4

I experience the same daily (out of memory, server dying); sometimes
several times daily.

It's a *big* downer.

I use 3.12-rc7, but I had this issue with earlier kernels, too.

Yes, I have qgroups enabled.


A bit desperate, I've started dumping various system parameters (free,
uptime, ps, top, /proc/meminfo etc.) every minute - memory usage can
jump from some 8 GB to 32 GB within one minute; a minute later the
system is no longer responsive and has to be hard resetted.


Example - free-2013-10-29-03:29:01

 total   used   free sharedbuffers cached
Mem:  32638388   32484484 153904  0   4960   24122452
-/+ buffers/cache:8357072   24281316
Swap: 16776116132   16775984


free-2013-10-29-03:30:01

 total   used   free sharedbuffers cached
Mem:  32638388   32473636 164752  05441191896
-/+ buffers/cache:   312811961357192
Swap: 16776116408   16775708



What would cause such a huge memory usage jump?

top for the same two minutes; no userspace process is using the memory.
First one - we're using ~8 GB RAM, the second one - all 32 GB is used.

top-2013-10-29-03:29:01:

top - 03:29:01 up  2:00,  1 user,  load average: 9.00, 8.48, 8.82
Tasks: 268 total,   1 running, 267 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.4 us,  0.9 sy,  0.2 ni, 77.2 id, 21.2 wa,  0.0 hi,  0.1 si,  0.0 st
KiB Mem:  32638388 total, 32485196 used,   153192 free, 4964 buffers
KiB Swap: 16776116 total,  132 used, 16775984 free, 24121388 cached

  PID USER  PR  NI  VIRT  RES  SHR S  %CPU %MEMTIME+  COMMAND
15568 root  20   0 99220  71m 1144 D   6.3  0.2   0:27.59 rsync --exclude=/
16384 root  20   0 000 S   6.3  0.0   0:11.12 [btrfs-endio-met]
17016 root  20   0 000 S   6.3  0.0   0:02.89 [btrfs-endio-met]
19956 root  20   0  147m  46m  984 D   6.3  0.1   0:19.60 rsync --delete --
21798 root  20   0 48268  13m 1132 D   6.3  0.0   0:07.61 rsync --exclude=/
21824 root  20   0 24696 1568 1096 R   6.3  0.0   0:00.01 top -b -c -n 1
1 root  20   0 10644  912  764 S   0.0  0.0   0:00.46 init [2]
2 root  20   0 000 S   0.0  0.0   0:00.03 [kthreadd]
3 root  20   0 000 S   0.0  0.0   0:00.22 [ksoftirqd/0]
4 root  20   0 000 S   0.0  0.0   0:00.00 [kworker/0:0]
5 root   0 -20 000 S   0.0  0.0   0:00.00 [kworker/0:0H]
6 root  20   0 000 S   0.0  0.0   0:04.01 [kworker/u16:0]
7 root  rt   0 000 S   0.0  0.0   0:00.43 [migration/0]
8 root  20   0 000 S   0.0  0.0   0:00.00 [rcu_bh]
9 root  20   0 000 S   0.0  0.0   0:09.07 [rcu_sched]
   10 root  rt   0 000 S   0.0  0.0   0:00.02 [watchdog/0]
   11 root  rt   0 000 S   0.0  0.0   0:00.03 [watchdog/1]
   12 root  rt   0 000 S   0.0  0.0   0:00.00 [migration/1]
   13 root  20   0 000 S   0.0  0.0   0:00.10 [ksoftirqd/1]
   14 root  20   0 000 S   0.0  0.0   0:00.00 [kworker/1:0]
   15 root   0 -20 000 S   0.0  0.0   0:00.00 [kworker/1:0H]
   16 root  rt   0 000 S   0.0  0.0   0:00.03 [watchdog/2]
   17 root  rt   0 000 S   0.0  0.0   0:00.00 [migration/2]
   18 root  20   0 000 S   0.0  0.0   0:00.09 [ksoftirqd/2]
   20 root   0 -20 000 S   0.0  0.0   0:00.00 [kworker/2:0H]
   21 root  rt   0 000 S   0.0  0.0   0:00.02 [watchdog/3]
   22 root  rt   0 000 S   0.0  0.0   0:00.01 [migration/3]
   23 root  20   0 000 S   0.0  0.0   0:00.10 [ksoftirqd/3]
   24 root  20   0 000 S   0.0  0.0   0:00.00 [kworker/3:0]
   25 root   0 -20 000 S   0.0  0.0   0:00.00 [kworker/3:0H]
   26 root  rt   0 000 S   0.0  0.0   0:00.03 [watchdog/4]
   27 root  rt   0 000 S   0.0  0.0   0:00.02 [migration/4]
   28 root  20   0 000 S   0.0  0.0   0:00.01 [ksoftirqd/4]
   29 root  20   0 000 S   0.0  0.0   0:00.00 [kworker/4:0]
   30 root   0 -20 000 S   0.0  0.0   0:00.00 [kworker/4:0H]
   31 root  rt   0 000 S   0.0  0.0   0:00.02 [watchdog/5]
   32 root  rt   0 000 S   0.0  0.0   0:00.02 [migration/5]
   33 root  20   0 000 S   0.0  0.0   0:00.01 [ksoftirqd/5]
   34 root  20   0 000 S   0.0  0.0   0:00.00 [kworker/5:0]
   35 root   0 -20 000 S   0.0  0.0   0:00.00 [kworker/5:0H]
   36

Re: No space left on device, problem

kernel BUG at fs/btrfs/inode.c:1595!

Re: Why cannot I move a read-only snapshot around?

[PATCH] Btrfs: take ordered root lock when removing ordered operations inode

Re: [PATCH] Btrfs-progs: allow --init-extent-tree to work when extent tree is borked

[PATCH] Btrfs-progs: rework open_ctree to take flags, add a new one

[PATCH] xfstests: add generic/320 to test fsync() on directories V2

Re: [patch 2/3] [PATCH 2/3] xfstests: add test for sysfs publication of btrfs info

Re: [patch 1/3] [PATCH 1/3] xfstests: add test for global metadata reservation publishing

Re: [patch 3/3] [PATCH 3/3] xfstests: add tests for new feature ioctl and sysfs interfaces

Null pointer dereference bug in btrfs_find_all_root

[PATCH] Btrfs: fix verification of dir_item

[PATCH] Btrfs: fix csum search offset/length calculation in log tree

[PATCH] Btrfs: log recovery, don't unlink inode always on error

Re: No space left on device, problem

[PATCH] Btrfs-progs: rework open_ctree to take flags, add a new one V2

[PATCH] Btrfs: make sure the delalloc workers actually flush compressed writes

Re: No space left on device, problem

Re: No space left on device, problem

Re: No space left on device, problem

[PATCH 1/3] xfstests: btrfs/021: test global metadata reservation reporting

[PATCH 0/3] xfstests: for btrfs publishing/export patchset (v2)

[PATCH 3/3] xfstests: btrfs/023,024,025: test feature ioctl and sysfs interfaces

Re: [patch 1/3] [PATCH 1/3] xfstests: add test for global metadata reservation publishing

[PATCH 2/3] xfstests: btrfs/022: test sysfs exports of allocation and device membership info

Re: [PATCH 2/3] xfstests: btrfs/022: test sysfs exports of allocation and device membership info

[patch 3/3] xfstests: btrfs/023,024,025: test feature ioctl and sysfs interfaces

[patch 1/3] xfstests: btrfs/021: test global metadata reservation reporting

[patch 0/3] xfstests: for btrfs publishing/export patchset (v3)

[patch 2/3] xfstests: btrfs/022: test sysfs exports of allocation and device membership info

memory leak in =3.11.6

Re: Null pointer dereference bug in btrfs_find_all_root

[PATCH] Btrfs: do not run snapshot-aware defragment on error

Re: memory leak in =3.11.6

Re: Null pointer dereference bug in btrfs_find_all_root

Re: memory leak in =3.11.6

36 matches

Site Navigation

Mail list logo

Footer information