Re: Is stability a joke? (wiki updated)
On Mon, Sep 12, 2016 at 02:44:35PM -0600, Chris Murphy wrote: > Just to cut yourself some slack, you could skip 3.14 because it's EOL > now, and just go from 4.4. Don't the btrfs-tools used to create the filesystem also play a huge role in this game? Greetings Marc -- - Marc Haber | "I don't trust Computers. They | Mailadresse im Header Leimen, Germany| lose things."Winona Ryder | Fon: *49 6224 1600402 Nordisch by Nature | How to make an American Quilt | Fax: *49 6224 1600421 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH] Btrfs: remove BUG_ON in start_transaction
Since we could get errors from the concurrent aborted transaction,
the check of this BUG_ON in start_transaction is not true any more.
Say, while flushing free space cache inode's dirty pages,
btrfs_finish_ordered_io
-> btrfs_join_transaction_nolock
(the transaction has been aborted.)
-> BUG_ON(type == TRANS_JOIN_NOLOCK);
Signed-off-by: Liu Bo
---
fs/btrfs/transaction.c | 5 +
1 file changed, 1 insertion(+), 4 deletions(-)
diff --git a/fs/btrfs/transaction.c b/fs/btrfs/transaction.c
index 95d4191..6d06365 100644
--- a/fs/btrfs/transaction.c
+++ b/fs/btrfs/transaction.c
@@ -549,11 +549,8 @@ again:
}
} while (ret == -EBUSY);
- if (ret < 0) {
- /* We must get the transaction if we are JOIN_NOLOCK. */
- BUG_ON(type == TRANS_JOIN_NOLOCK);
+ if (ret < 0)
goto join_fail;
- }
cur_trans = root->fs_info->running_transaction;
--
2.5.5
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH] Btrfs: fix memory leak in do_walk_down
The extent buffer 'next' needs to be free'd conditionally.
Signed-off-by: Liu Bo
---
fs/btrfs/extent-tree.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index 5a940ab..779fd72 100644
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -8882,6 +8882,7 @@ static noinline int do_walk_down(struct
btrfs_trans_handle *trans,
&wc->flags[level - 1]);
if (ret < 0) {
btrfs_tree_unlock(next);
+ free_extent_buffer(next);
return ret;
}
--
2.5.5
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH v6 1/6] fstests: common: Introduce _post_mount_hook for btrfs
Introduce _post_mount_hook(), which will be executed after mounting
scratch/test.
It's quite useful for fs(OK, only btrfs yet, again) which needs to
use ioctl other than mount option to enable some of its feature.
Now only btrfs quota needs this hook to allow enabling quota to be
enabled for *ALL* existing test cases.
This should dramatically improve the test coverage to expose quota
related bugs.
Signed-off-by: Qu Wenruo
---
common/rc | 23 +++
1 file changed, 23 insertions(+)
diff --git a/common/rc b/common/rc
index 23c007a..631397f 100644
--- a/common/rc
+++ b/common/rc
@@ -321,6 +321,27 @@ _overlay_scratch_unmount()
$UMOUNT_PROG $SCRATCH_MNT
}
+_run_btrfs_post_mount_hook()
+{
+ mnt_point=$1
+ for n in $ALWAYS_ENABLE_BTRFS_FEATURE; do
+ if [ $n == "quota" -o $n == "qgroup" ]; then
+ # Quota can be enabled for several times
+ # and won't cause bug
+ _run_btrfs_util_prog quota enable $mnt_point
+ fi
+ done
+}
+
+_post_mount_hook()
+{
+ mnt_point=$1
+
+ if [ $FSTYP == "btrfs" -a -v ALWAYS_ENABLE_BTRFS_FEATURE ]; then
+ _run_btrfs_post_mount_hook $mnt_point
+ fi
+}
+
_scratch_mount()
{
if [ "$FSTYP" == "overlay" ]; then
@@ -328,6 +349,7 @@ _scratch_mount()
return $?
fi
_mount -t $FSTYP `_scratch_mount_options $*`
+_post_mount_hook $SCRATCH_MNT
}
_scratch_unmount()
@@ -377,6 +399,7 @@ _test_mount()
fi
_test_options mount
_mount -t $FSTYP $TEST_OPTIONS $TEST_FS_MOUNT_OPTS $SELINUX_MOUNT_OPTIONS
$* $TEST_DEV $TEST_DIR
+_post_mount_hook $TEST_DIR
}
_test_unmount()
--
2.7.4
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH v6 2/6] fstests: common: rename _require_btrfs to _require_btrfs_subcommand
Rename _require_btrfs() to _require_btrfs_subcommand() to avoid
confusion, as all other _require_btrfs_* has a quite clear suffix, like
_require_btrfs_mkfs_feature() or _require_btrfs_fs_feature().
Signed-off-by: Qu Wenruo
---
common/rc | 2 +-
tests/btrfs/004 | 2 +-
tests/btrfs/048 | 2 +-
tests/btrfs/059 | 2 +-
4 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/common/rc b/common/rc
index 631397f..636cba6 100644
--- a/common/rc
+++ b/common/rc
@@ -3001,7 +3001,7 @@ _require_deletable_scratch_dev_pool()
}
# We check for btrfs and (optionally) features of the btrfs command
-_require_btrfs()
+_require_btrfs_subcommand()
{
cmd=$1
_require_command "$BTRFS_UTIL_PROG" btrfs
diff --git a/tests/btrfs/004 b/tests/btrfs/004
index 905770a..2ce628e 100755
--- a/tests/btrfs/004
+++ b/tests/btrfs/004
@@ -51,7 +51,7 @@ _supported_fs btrfs
_supported_os Linux
_require_scratch
_require_no_large_scratch_dev
-_require_btrfs inspect-internal
+_require_btrfs_subcommand inspect-internal
_require_command "/usr/sbin/filefrag" filefrag
rm -f $seqres.full
diff --git a/tests/btrfs/048 b/tests/btrfs/048
index 0b907b0..ac731d1 100755
--- a/tests/btrfs/048
+++ b/tests/btrfs/048
@@ -48,7 +48,7 @@ _supported_fs btrfs
_supported_os Linux
_require_test
_require_scratch
-_require_btrfs "property"
+_require_btrfs_subcommand "property"
send_files_dir=$TEST_DIR/btrfs-test-$seq
diff --git a/tests/btrfs/059 b/tests/btrfs/059
index 8f106d2..fd67ebb 100755
--- a/tests/btrfs/059
+++ b/tests/btrfs/059
@@ -51,7 +51,7 @@ _supported_fs btrfs
_supported_os Linux
_require_test
_require_scratch
-_require_btrfs "property"
+_require_btrfs_subcommand "property"
rm -f $seqres.full
--
2.7.4
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH v6 3/6] fstests: Add btrfs dedupe post mount hook
Now fstests can run any test cases with btrfs inband-dedupe enabled. Signed-off-by: Qu Wenruo --- common/rc | 20 ++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/common/rc b/common/rc index 636cba6..e0da69b 100644 --- a/common/rc +++ b/common/rc @@ -326,10 +326,26 @@ _run_btrfs_post_mount_hook() mnt_point=$1 for n in $ALWAYS_ENABLE_BTRFS_FEATURE; do if [ $n == "quota" -o $n == "qgroup" ]; then - # Quota can be enabled for several times - # and won't cause bug + # Re-enable quota won't cause anything wrong _run_btrfs_util_prog quota enable $mnt_point fi + if [ $n == "dedupe" -o $n == "inband-dedupe" -o $n == "dedupe-inband" ]; then + # Need to check if kernel/progs support dedupe + _require_btrfs_fs_feature dedupe + _require_btrfs_subcommand dedupe + + # Check if dedupe is already enabled, or we will + # overwrite previous on-disk backend config + $BTRFS_UTIL_PROG dedupe status $mnt_point | grep -q "Enabled" + if [ $1 -eq 0 ]; then + continue + fi + + # This will enable dedupe with default options, so + # btrfs dedupe test cases needs to use '-f' option to + # override default config. + _run_btrfs_util_prog dedupe enable $mnt_point + fi done } -- 2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH v6 4/6] fstests: btrfs: Add basic test for btrfs in-band de-duplication
Add basic test for btrfs in-band de-duplication(inmemory backend), including:
1) Enable
3) Dedup rate
4) File correctness
5) Disable
Signed-off-by: Qu Wenruo
---
common/defrag | 13 ++
tests/btrfs/200 | 116
tests/btrfs/200.out | 22 ++
tests/btrfs/group | 1 +
4 files changed, 152 insertions(+)
create mode 100755 tests/btrfs/200
create mode 100644 tests/btrfs/200.out
diff --git a/common/defrag b/common/defrag
index 986b4bf..73c7a99 100644
--- a/common/defrag
+++ b/common/defrag
@@ -59,6 +59,19 @@ _extent_count()
$XFS_IO_PROG -c "fiemap" $1 | tail -n +2 | grep -v hole | wc -l|
$AWK_PROG '{print $1}'
}
+# Get the number of unique file extents
+# Unique file extents means they have different ondisk bytenr
+# Some filesystem supports reflinkat() or in-band de-dup can create
+# a file whose all file extents points to the same ondisk bytenr
+# this can be used to test if such reflinkat() or in-band de-dup works
+_extent_count_uniq()
+{
+ file=$1
+ $XFS_IO_PROG -c "fiemap" $file >> $seqres.full 2>&1
+ $XFS_IO_PROG -c "fiemap" $file | tail -n +2 | grep -v hole |\
+ $AWK_PROG '{print $3}' | sort | uniq | wc -l
+}
+
_check_extent_count()
{
min=$1
diff --git a/tests/btrfs/200 b/tests/btrfs/200
new file mode 100755
index 000..5a190dd
--- /dev/null
+++ b/tests/btrfs/200
@@ -0,0 +1,116 @@
+#! /bin/bash
+# FS QA Test 200
+#
+# Basic btrfs inband dedupe test for inmemory backend
+#
+#---
+# Copyright (c) 2016 Fujitsu. All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+#---
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1 # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+ cd /
+ rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/defrag
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+
+_supported_fs btrfs
+_supported_os Linux
+_require_scratch
+_require_btrfs_subcommand dedupe
+_require_btrfs_fs_feature dedupe
+
+# File size is twice the maximum file extent of btrfs
+# So even fallbacked to non-dedupe, it will have at least 2 extents
+file_size=256m
+
+_scratch_mkfs >> $seqres.full 2>&1
+_scratch_mount
+
+do_dedupe_test()
+{
+ dedupe_bs=$1
+
+ echo "Testing inmemory dedupe backend with block size $dedupe_bs"
+ _run_btrfs_util_prog dedupe enable -f -s inmemory -b $dedupe_bs \
+ $SCRATCH_MNT
+ # do sync write to ensure dedupe hash is added into dedupe pool
+ $XFS_IO_PROG -f -c "pwrite -b $dedupe_bs 0 $dedupe_bs" -c "fsync"\
+ $SCRATCH_MNT/initial_block | _filter_xfs_io
+
+ # do sync write to ensure we can get stable fiemap later
+ $XFS_IO_PROG -f -c "pwrite -b $dedupe_bs 0 $file_size" -c "fsync"\
+ $SCRATCH_MNT/real_file | _filter_xfs_io
+
+ # Test if real_file is de-duplicated
+ nr_uniq_extents=$(_extent_count_uniq $SCRATCH_MNT/real_file)
+ nr_total_extents=$(_extent_count $SCRATCH_MNT/real_file)
+ nr_deduped_extents=$(($nr_total_extents - $nr_uniq_extents))
+
+ echo "deduped/total: $nr_deduped_extents/$nr_total_extents" \
+ >> $seqres.full
+ # Allow a small amount of dedupe miss, as commit interval or
+ # memory pressure may break a dedupe_bs block and cause
+ # small extent which won't go through dedupe routine
+ _within_tolerance "number of deduped extents" $nr_deduped_extents \
+ $nr_total_extents 5% -v
+
+ # Also check the md5sum to ensure data is not corrupted
+ md5=$(_md5_checksum $SCRATCH_MNT/real_file)
+ echo "md5sum: $md5"
+}
+
+# Test inmemory dedupe first, use 64K dedupe bs to keep compatibility
+# with 64K page size
+do_dedupe_test 64K
+
+# Test 128K(default) dedupe bs
+do_dedupe_test 128K
+
+# Test 1M dedupe bs
+do_dedupe_test 1M
+
+# Check dedupe disable
+_run_btrfs_util_prog dedupe disable $SCRATCH_MNT
+
+# success, all done
+status=0
+exit
+# Check dedupe disable
+_run_btrfs_util_prog dedupe disable $SCRATCH_MNT
+
+# s
[PATCH v6 6/6] fstests: btrfs: Test inband dedupe with data balance.
Btrfs balance will reloate date extent, but its hash is removed too late
at run_delayed_ref() time, which will cause extent ref increased
during balance, cause either find_data_references() gives WARN_ON()
or even run_delayed_refs() fails and cause transaction abort.
Add such concurrency test for inband dedupe and data balance.
Signed-off-by: Qu Wenruo
---
tests/btrfs/203 | 110
tests/btrfs/203.out | 3 ++
tests/btrfs/group | 1 +
3 files changed, 114 insertions(+)
create mode 100755 tests/btrfs/203
create mode 100644 tests/btrfs/203.out
diff --git a/tests/btrfs/203 b/tests/btrfs/203
new file mode 100755
index 000..96af734
--- /dev/null
+++ b/tests/btrfs/203
@@ -0,0 +1,110 @@
+#! /bin/bash
+# FS QA Test 203
+#
+# Btrfs inband dedupe with balance concurrency test
+#
+# This can spot inband dedupe error which will increase delayed ref on
+# an data extent inside RO block group
+#
+#---
+# Copyright (c) 2016 Fujitsu. All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+#---
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1 # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+ cd /
+ kill $populate_pid &> /dev/null
+ kill $balance_pid &> /dev/null
+ wait
+ # Check later comment for reason
+ $BTRFS_UTIL_PROG balance cancel $SCRATCH_MNT &> /dev/null
+ rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/reflink
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+
+_supported_fs btrfs
+_supported_os Linux
+_require_scratch
+_require_cp_reflink
+_require_btrfs_subcommand dedupe
+_require_btrfs_fs_feature dedupe
+
+dedupe_bs=128k
+file_size_in_kilo=4096
+init_file=$SCRATCH_MNT/foo
+run_time=$((60 * $TIME_FACTOR))
+
+_scratch_mkfs >> $seqres.full 2>&1
+_scratch_mount
+
+do_dedupe_balance_test()
+{
+ _run_btrfs_util_prog dedupe enable -f -b $dedupe_bs -s inmemory \
+ $SCRATCH_MNT
+
+ # create the initial file and fill hash pool
+ $XFS_IO_PROG -f -c "pwrite -S 0x0 -b $dedupe_bs 0 $dedupe_bs" -c
"fsync" \
+ $init_file | _filter_xfs_io
+
+ _btrfs_stress_balance $SCRATCH_MNT >/dev/null 2>&1 &
+ balance_pid=$!
+
+ # Populate fs with all 0 data, to trigger enough in-band dedupe work
+ # to race with balance
+ _populate_fs -n 5 -f 1000 -d 1 -r $SCRATCH_MNT \
+ -s $file_size_in_kilo &> /dev/null &
+ populate_pid=$!
+
+ sleep $run_time
+
+ kill $populate_pid
+ kill $balance_pid
+ wait
+
+ # Sometimes even we killed $balance_pid and wait returned,
+ # balance may still be running, use balance cancel to wait it.
+ # As this is just a workaround, we don't want it pollute seqres
+ # so call $BTRFS_UTIL_PROG directly
+ $BTRFS_UTIL_PROG balance cancel $SCRATCH_MNT &> /dev/null
+
+ rm $SCRATCH_MNT/* -rf &> /dev/null
+ _run_btrfs_util_prog dedupe disable $SCRATCH_MNT
+}
+
+do_dedupe_balance_test
+
+# success, all done
+status=0
+exit
diff --git a/tests/btrfs/203.out b/tests/btrfs/203.out
new file mode 100644
index 000..404394c
--- /dev/null
+++ b/tests/btrfs/203.out
@@ -0,0 +1,3 @@
+QA output created by 203
+wrote 131072/131072 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
diff --git a/tests/btrfs/group b/tests/btrfs/group
index 2e539d0..c35b919 100644
--- a/tests/btrfs/group
+++ b/tests/btrfs/group
@@ -135,3 +135,4 @@
130 auto clone send
200 auto ib-dedupe
201 auto ib-dedupe
+203 auto ib-dedupe balance
--
2.7.4
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH v6 0/6] Btrfs in-band de-duplication test cases
Btrfs in-band de-duplication test case for in-memory backend. With extra option ALWAYS_ENABLE_BTRFS_FEATURE macro to enable dedupe/quota for all test cases. This is quite handy to hugely increase the coverage without introducing a lot new test cases. v6: Introduce ALWAYS_ENABLE_BTRFS_FEATURE macro to enable dedupe/quota for all existing test cases. v5: Due to kernel ioctl change, add FORCE flag for "dedupe enable" ioctl call. v4: Due to kernel patchset re-organization, remove on-disk backend test cases v3: Add new test cases for on-disk backend with metadata balance v2: Add new test cases for on-disk backend with full balance Qu Wenruo (6): fstests: common: Introduce _post_mount_hook for btrfs fstests: common: rename _require_btrfs to _require_btrfs_subcommand fstests: Add btrfs dedupe post mount hook fstests: btrfs: Add basic test for btrfs in-band de-duplication fstests: btrfs: Add testcase for btrfs dedupe and metadata balance race test fstests: btrfs: Test inband dedupe with data balance. common/defrag | 13 ++ common/rc | 41 ++- tests/btrfs/004 | 2 +- tests/btrfs/048 | 2 +- tests/btrfs/059 | 2 +- tests/btrfs/200 | 116 tests/btrfs/200.out | 22 ++ tests/btrfs/201 | 112 ++ tests/btrfs/201.out | 2 + tests/btrfs/203 | 110 + tests/btrfs/203.out | 3 ++ tests/btrfs/group | 3 ++ 12 files changed, 424 insertions(+), 4 deletions(-) create mode 100755 tests/btrfs/200 create mode 100644 tests/btrfs/200.out create mode 100755 tests/btrfs/201 create mode 100644 tests/btrfs/201.out create mode 100755 tests/btrfs/203 create mode 100644 tests/btrfs/203.out -- 2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH v6 5/6] fstests: btrfs: Add testcase for btrfs dedupe and metadata balance race test
Btrfs balance with inband dedupe enable/disable will expose a lot of
hidden dedupe bug:
1) Enable/disable race bug
2) Btrfs dedupe tree balance corrupted delayed_ref
3) Btrfs disable and balance will cause balance BUG_ON
Reported-by: Satoru Takeuchi
Signed-off-by: Qu Wenruo
---
tests/btrfs/201 | 112
tests/btrfs/201.out | 2 +
tests/btrfs/group | 1 +
3 files changed, 115 insertions(+)
create mode 100755 tests/btrfs/201
create mode 100644 tests/btrfs/201.out
diff --git a/tests/btrfs/201 b/tests/btrfs/201
new file mode 100755
index 000..152e9f7
--- /dev/null
+++ b/tests/btrfs/201
@@ -0,0 +1,112 @@
+#! /bin/bash
+# FS QA Test 201
+#
+# Btrfs inband dedup enable/disable race with metadata balance
+#
+# This tests will test the following bugs exposed in development:
+# 1) enable/disable race
+# 2) tree balance cause delayed ref corruption
+# 3) disable and balance cause BUG_ON
+#
+#---
+# Copyright (c) 2016 Fujitsu. All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+#---
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1 # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+ cd /
+ rm -f $tmp.*
+ killall $FSSTRESS_PROG &> /dev/null
+ kill $trigger_pid &> /dev/null
+ kill $balance_pid &> /dev/null
+ wait
+
+ # See comment later
+ $BTRFS_UTIL_PROG balance cancel $SCRATCH_MNT &> /dev/null
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+
+_supported_fs btrfs
+_supported_os Linux
+_require_scratch
+_require_btrfs_subcommand dedupe
+_require_btrfs_fs_feature dedupe
+
+# Use 64K dedupe size to keep compatibility for 64K page size
+dedupe_bs=64K
+
+_scratch_mkfs >> $seqres.full 2>&1
+_scratch_mount
+
+mkdir -p $SCRATCH_MNT/stressdir
+
+runtime=$((60 * $TIME_FACTOR))
+
+trigger_work()
+{
+ while true; do
+ _run_btrfs_util_prog dedupe enable -f -s inmemory \
+ -b $dedupe_bs $SCRATCH_MNT
+ sleep 1
+ _run_btrfs_util_prog dedupe disable $SCRATCH_MNT
+ sleep 1
+ done
+}
+
+# redirect all output, as error output like 'balance cancelled by user'
+# will populuate the golden output.
+_btrfs_stress_balance -m $SCRATCH_MNT &> /dev/null &
+balance_pid=$!
+
+$FSSTRESS_PROG $(_scale_fsstress_args -p 1 -n 1000) $FSSTRESS_AVOID \
+ -d $SCRATCH_MNT/stressdir > /dev/null 2>&1 &
+
+trigger_work &
+trigger_pid=$!
+
+sleep $runtime
+killall $FSSTRESS_PROG &> /dev/null
+kill $trigger_pid &> /dev/null
+kill $balance_pid &> /dev/null
+wait
+
+# Manually stop balance as it's possible balance is still running for a short
+# time. And we don't want to populate $seqres.full, so call $BTRFS_UTIL_PROG
+# directly
+$BTRFS_UTIL_PROG balance cancel $SCRATCH_MNT &> /dev/null
+
+echo "Silence is golden"
+# success, all done
+status=0
+exit
diff --git a/tests/btrfs/201.out b/tests/btrfs/201.out
new file mode 100644
index 000..5ac973f
--- /dev/null
+++ b/tests/btrfs/201.out
@@ -0,0 +1,2 @@
+QA output created by 201
+Silence is golden
diff --git a/tests/btrfs/group b/tests/btrfs/group
index ea9c36b..2e539d0 100644
--- a/tests/btrfs/group
+++ b/tests/btrfs/group
@@ -134,3 +134,4 @@
129 auto quick send
130 auto clone send
200 auto ib-dedupe
+201 auto ib-dedupe
--
2.7.4
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
mkfs+mount failure of small fs on ppc64
on ppc64, 4.7-rc kernel, git btrfs-progs, v4.7.2: # truncate --size=500m testfile # ./mkfs.btrfs testfile # mkdir -p mnt # mount -o loop testfile mnt btrfs-progs v4.7.2 See http://btrfs.wiki.kernel.org for more information. Label: (null) UUID: c531b759-a491-4c9f-a954-4787cea9106d Node size: 65536 Sector size:65536 Filesystem size:500.00MiB Block group profiles: Data: single8.00MiB Metadata: DUP 32.00MiB System: DUP 8.00MiB SSD detected: no Incompat features: extref, skinny-metadata Number of devices: 1 Devices: IDSIZE PATH 1 500.00MiB testfile # dmesg -c [ 61.210287] loop: module loaded [ 61.247105] BTRFS: device fsid a8d79cd0-977f-4b93-8410-246dc08b3683 devid 1 transid 5 /dev/loop0 [ 61.247391] BTRFS info (device loop0): disk space caching is enabled [ 61.247397] BTRFS info (device loop0): has skinny extents [ 61.270492] BTRFS info (device loop0): creating UUID tree [ 61.312149] BTRFS warning (device loop0): failed to create the UUID tree: -28 [ 61.483028] BTRFS: open_ctree failed 2nd mount works: # mount -o loop testfile mnt # dmesg -c [ 87.504564] BTRFS info (device loop0): disk space caching is enabled [ 87.504579] BTRFS info (device loop0): has skinny extents [ 87.506979] BTRFS info (device loop0): creating UUID tree Any ideas? This seems to have regressed since 3.9.1, but there are a couple other mkfs breakages in between, and my bisect was not fruitful. Thanks, -Eric -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Filesystem forced to readonly after use
>From the fsck... bad block 160420741120 I can't tell though if that's a bad Btrfs leaf/node where both dup copies are bad; or if it's a bad sector. I'd mount it ro, and take a backup of anything you care about before proceeding further. smartctl -x might reveal if there are problems the drive itself is aware of. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: BTRFS constantly reports "No space left on device" even with a huge unallocated space
On 09/13/2016 04:49 PM, Ronan Arraes Jardim Chagas wrote: Hi guys, One more time I saw the problem. It begins to happen on a daily basis now. Unfortunately the `enospc_debug` flag did not help. I did not see any new information in the logs. This time, only a hard reset worked. I could not even reboot using gnome panel. I just started paying attention to this, the last kernel I saw you were running was 4.7. Have you tried a recent kernel, like chris's tree? git://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs.git for-linus-4.8 is what I would like you to try if not. Thanks, Josef -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Filesystem forced to readonly after use
On 13-09-2016 16:39, Chris Murphy wrote: I just wouldn't use btrfs repair with this version of progs, go back to v4.6.1 or upgrade to 4.7.2. Thanks for the tip. I upgraded to 4.7.2. Cesar -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: BTRFS constantly reports "No space left on device" even with a huge unallocated space
Hi guys, One more time I saw the problem. It begins to happen on a daily basis now. Unfortunately the `enospc_debug` flag did not help. I did not see any new information in the logs. This time, only a hard reset worked. I could not even reboot using gnome panel. Best regards, Ronan Arraes -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Filesystem forced to readonly after use
On 13-09-2016 16:49, Austin S. Hemmelgarn wrote: I'd be kind of curious to see the results from btrfs check run without repair, but I doubt that will help narrow things down any further. Attached. As of right now, the absolute first thing I'd do is check your logs to see if you can find any indication of errors from the disk itself. I don't think it's likely, but it's worth checking. Will do. The couple of lines just before the crash in the attached kernel log would indicate to me that some of the metadata is corrupted. There are two likely possibilities for how that happened: 1. Running with no extra space for new chunks to be allocated is not a common use case, so it's not well tested, and it wouldn't surprise me if some accounting falls apart in that situation. Indeed. I periodically remove old snapshots and check for disk space, bit I guess I ran a bit too near the limit this time. 2. You might have bad RAM or a bad PSU. This is the second thing you should check after checking to see if the disk is OK, as either will likely cause any repair attempts to make things worse. RAM is pretty easy to check, but for a PSU you need a proper testing device. You can get such a device on Amazon or similar sites for about 25USD, and it's generally worth having around for troubleshooting. Understood. This notebook has occasional failures when resuming from hibernation. I suppose, from the point of view of the filesystem, this corresponds to an unclean reboot. Assuming your disk and RAM are good, the next thing to do would be try and get the filesystem into a more usable state. The best option for this is to expand the filesystem if possible. Given that you're running right near capacity, I'd suggest at least 16G of extra space if possible. If that isn't a viable solution for you, the other option is to delete some of the oldest snapshots (Ideally enough that you have at least a few GB of extra space in the data chunks and a few hundred MB in the metadata chunks), then add a 4-8GB device to the FS temporarily (a ramdisk or flash drive works well for this), and run a full balance. If you're lucky, this will fix any metadata that's messed up, and the system should be usable. If not, it shouldn't make things any worse, and you probably want to look at btrfs restore to copy out the data to a new filesystem (ideally a bigger one). I will try this next. Thanks for the help! Cesar checking extents parent transid verify failed on 160420773888 wanted 181826 found 181573 parent transid verify failed on 160420773888 wanted 181826 found 181573 parent transid verify failed on 160420773888 wanted 181826 found 181573 parent transid verify failed on 160420773888 wanted 181826 found 181573 Ignoring transid failure leaf parent key incorrect 160420773888 parent transid verify failed on 160420773888 wanted 181826 found 181573 Ignoring transid failure leaf parent key incorrect 160420773888 parent transid verify failed on 160420773888 wanted 181826 found 181573 Ignoring transid failure leaf parent key incorrect 160420773888 parent transid verify failed on 160420773888 wanted 181826 found 181573 Ignoring transid failure leaf parent key incorrect 160420773888 parent transid verify failed on 160420773888 wanted 181826 found 181573 Ignoring transid failure leaf parent key incorrect 160420773888 parent transid verify failed on 160420773888 wanted 181826 found 181573 Ignoring transid failure leaf parent key incorrect 160420773888 parent transid verify failed on 160420773888 wanted 181826 found 181573 Ignoring transid failure leaf parent key incorrect 160420773888 parent transid verify failed on 160420773888 wanted 181826 found 181573 Ignoring transid failure leaf parent key incorrect 160420773888 parent transid verify failed on 160420773888 wanted 181826 found 181573 Ignoring transid failure leaf parent key incorrect 160420773888 parent transid verify failed on 160418889728 wanted 181826 found 181572 parent transid verify failed on 160418889728 wanted 181826 found 181572 parent transid verify failed on 160418889728 wanted 181826 found 181572 parent transid verify failed on 160418889728 wanted 181826 found 181572 parent transid verify failed on 160420741120 wanted 181826 found 181573 parent transid verify failed on 160420741120 wanted 181826 found 181573 parent transid verify failed on 160420741120 wanted 181826 found 181573 parent transid verify failed on 160420741120 wanted 181826 found 181573 Ignoring transid failure leaf parent key incorrect 160420741120 bad block 160420741120 Errors found in extent allocation tree or chunk allocation parent transid verify failed on 160420773888 wanted 181826 found 181573 Ignoring transid failure parent transid verify failed on 160418889728 wanted 181826 found 181572 parent transid verify failed on 160418889728 wanted 181826 found 181572 parent transid verify failed on 160420741120 wanted 181826 found 181573 Ignoring transid failure Error: could not find btree root ext
Re: Filesystem forced to readonly after use
On Tue, Sep 13, 2016 at 1:49 PM, Austin S. Hemmelgarn wrote: > On 2016-09-13 15:20, Cesar Strauss wrote: >> >> btrfs-progs v4.7 > > It's always good to see people who are staying up-to-date on the kernel and > userspace :) Yes, although it and 4.7.1 are marked as do not use. https://btrfs.wiki.kernel.org/index.php/Changelog#btrfs-progs-4.7.2_.28Sep_2016.29 -- Chris Murphy -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: State of the fuzzer
I've booted another instance with btrfs-progs checked out to 2b7c507 and collected some bugs which remained from the run before the current one. The current run discovered what qgroups are just three days ago and will spend some time on that. I've also added UBSAN- and MSAN-logging to my setup and there were three bugs found so far (one is already fixed). I will boot a third instance to run lowmem-mode exclusively in the next few days. There are 11 bugs open at the moment, all have a reproducing image attached to them. The whole list is at https://bugzilla.kernel.org/buglist.cgi?bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&component=btrfs&email1=lukas.lueg%40gmail.com&emailreporter1=1&emailtype1=exact&list_id=858441&query_format=advanced 2016-09-09 16:00 GMT+02:00 David Sterba : > On Tue, Sep 06, 2016 at 10:32:28PM +0200, Lukas Lueg wrote: >> I'm currently fuzzing rev 2076992 and things start to slowly, slowly >> quiet down. We will probably run out of steam at the end of the week >> when a total of (roughly) half a billion BTRFS-images have passed by. >> I will switch revisions to current HEAD and restart the whole process >> then. A few things: >> >> * There are a couple of crashes (mostly segfaults) I have not reported >> yet. I'll report them if they show up again with the latest revision. > > Ok. > >> * The coverage-analysis shows assertion failures which are currently >> silenced. An assertion failure is technically a worse disaster >> successfully prevented, it still constitutes unexpected/unusable >> behaviour, though. Do you want assertions to be enabled and images >> triggering those assertions reported? This is basically the same >> conundrum as with BUG_ON and abort(). > > Yes please. I'd like to turn most bugons/assertions into a normal > failure report if it would make sense. > >> * A few endless loops entered into by btrfsck are currently >> unmitigated (see bugs 155621, 155571, 11 and 155151). It would be >> nice if those had been taken care of by next week if possible. > > Two of them are fixed, the other two need more work, updating all > callers of read_node_slot and the callchain. So you may still see that > kind of looping in more images. I don't have an ETA for the fix, I won't > be available during the next week. > > At the moment, the initial sanity checks should catch most of the > corrupted values, so I'm expecting that you'll see different classes of > problems in the next rounds. > > The testsuite now contains all images that you reported and we have a > fix in git. There are more utilities run on the images, there may be > more problems for us to fix. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: BTRFS constantly reports "No space left on device" even with a huge unallocated space
On 09/08/2016 07:02 PM, Jeff Mahoney wrote: On 9/8/16 2:49 PM, Jeff Mahoney wrote: On 9/8/16 2:24 PM, Ronan Arraes Jardim Chagas wrote: Hi all! Em Seg, 2016-09-05 às 16:49 +0800, Qu Wenruo escreveu: Just like what Wang has mentioned, would you please paste all the output of the contents of /sys/fs/btrfs//allocation? It's recommended to use "grep . -IR " to get all the data as it will show the file name. So, one more time, I see the problem. This time I was just using Firefox and I cannot recover using `btrfs balance`. I think that, one more time, I will need to reboot this machine. This problem is really causing me a lot of troubles :( I have a hunch the list is about to be flooded with similar reports if we don't find this one before 4.8. commit d555b6c380c644af63dbdaa7cc14bba041a4e4dd Author: Josef Bacik Date: Fri Mar 25 13:25:51 2016 -0400 Btrfs: warn_on for unaccounted spaces This commit isn't the source of the bug, but it's making it a lot more noisy. I spent a few hours last night trying to track down why xfstests was throwing these warnings and I was able to reproduce them at least as far back as 4.4-vanilla with -oenospc_debug enabled. Speaking of which, can you turn on mounting with -oenospc_debug if you haven't already? In my case, space_info->bytes_may_use was getting accounted incorrectly. I am able to reproduce that even with the following commit: commit 18513091af9483ba84328d42092bd4d42a3c958f Author: Wang Xiaoguang Date: Mon Jul 25 15:51:40 2016 +0800 btrfs: update btrfs_space_info's bytes_may_use timely And the btrfs_free_reserved_data_space_noquota WARN_ON I was seeing is fixed by: commit ed7a6948394305b810d0c6203268648715e5006f Author: Wang Xiaoguang Date: Fri Aug 26 11:33:14 2016 +0800 btrfs: do not decrease bytes_may_use when replaying extents ... which shouldn't change anything for your issue, unfortunately. I still see these: WARNING: CPU: 2 PID: 8166 at ../fs/btrfs/extent-tree.c:9582 btrfs_free_block_groups+0x2a8/0x400 [btrfs]() Modules linked in: loop dm_flakey af_packet iscsi_ibft iscsi_boot_sysfs msr ext4 crc16 mbcache jbd2 ipmi_ssif dm_mod igb ptp pps_core acpi_cpufreq tpm_infineon kvm_amd ipmi_si kvm dca pcspkr ipmi_msghandler 8250_fintek sp5100_tco fjes irqbypass i2c_piix4 shpchp processor button amd64_edac_mod edac_mce_amd edac_core k10temp btrfs xor raid6_pq sd_mod ata_generic mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect ohci_pci sysimgblt ehci_pci serio_raw ohci_hcd fb_sys_fops pata_atiixp ehci_hcd ttm ahci libahci drm usbcore libata usb_common sg scsi_mod autofs4 CPU: 2 PID: 8166 Comm: umount Tainted: GW 4.4.19-11.g81405db-vanilla #1 Hardware name: HP ProLiant DL165 G7, BIOS O37 10/17/2012 880230317d10 813170ec a0472528 880230317d48 8107d816 88009ab03600 8800ba106288 8800ab75a000 8800ba106200 Call Trace: [] dump_stack+0x63/0x87 [] warn_slowpath_common+0x86/0xc0 [] warn_slowpath_null+0x1a/0x20 [] btrfs_free_block_groups+0x2a8/0x400 [btrfs] [] close_ctree+0x15b/0x330 [btrfs] [] btrfs_put_super+0x19/0x20 [btrfs] [] generic_shutdown_super+0x6f/0x100 [] kill_anon_super+0x12/0x20 [] btrfs_kill_super+0x18/0x120 [btrfs] [] deactivate_locked_super+0x43/0x70 [] deactivate_super+0x46/0x60 [] cleanup_mnt+0x3f/0x80 [] __cleanup_mnt+0x12/0x20 [] task_work_run+0x86/0xb0 [] exit_to_usermode_loop+0x73/0xa2 [] syscall_return_slowpath+0x8d/0xa0 [] int_ret_from_sys_call+0x25/0x8f ---[ end trace 09a0cc2892b6305c ]--- BTRFS: space_info 1 has 7946240 free, is not full BTRFS: space_info total=8388608, used=442368, pinned=0, reserved=0, may_use=4096, readonly=0 ... where the value of may_use varies. What test are you seeing this with? Thanks, Josef -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Filesystem forced to readonly after use
On 2016-09-13 15:20, Cesar Strauss wrote: Hello, I have a BTRFS filesystem that is reverting to read-only after a few moments of use. There is a stack trace visible in the kernel log, which is attached. Here is my system information: # uname -a Linux rescue 4.7.2-1-ARCH #1 SMP PREEMPT Sat Aug 20 23:02:56 CEST 2016 x86_64 GNU/Linux # btrfs --version btrfs-progs v4.7 It's always good to see people who are staying up-to-date on the kernel and userspace :) # btrfs fi show Label: 'linux' uuid: 79862c20-d0b0-4ffa-a9af-e3a40868a243 Total devices 1 FS bytes used 284.60GiB devid1 size 300.03GiB used 300.03GiB path /dev/sdb5 Given this, you're running with the whole device fully allocated by the chunk allocator, this is not a good state to be in for any extended period of time on a filesystem which is being written to and modified. # btrfs fi df /mnt Data, single: total=278.00GiB, used=274.68GiB System, DUP: total=8.00MiB, used=64.00KiB System, single: total=4.00MiB, used=0.00B Metadata, DUP: total=11.00GiB, used=9.92GiB Metadata, single: total=8.00MiB, used=0.00B GlobalReserve, single: total=512.00MiB, used=0.00B But you appear to have a reasonable amount of slack space within the chunks themselves. As soon as the problem started, I saw that the Metadata, DUP was completely used. It become a little better (like above) after a scrub. I can easily recover disk space by removing old snapshots, if needed. The dmesg output is attached. Before making further recovery attempts, or even restoring from backup, I would like to ask for the best option to proceed. I'd be kind of curious to see the results from btrfs check run without repair, but I doubt that will help narrow things down any further. As of right now, the absolute first thing I'd do is check your logs to see if you can find any indication of errors from the disk itself. I don't think it's likely, but it's worth checking. The couple of lines just before the crash in the attached kernel log would indicate to me that some of the metadata is corrupted. There are two likely possibilities for how that happened: 1. Running with no extra space for new chunks to be allocated is not a common use case, so it's not well tested, and it wouldn't surprise me if some accounting falls apart in that situation. 2. You might have bad RAM or a bad PSU. This is the second thing you should check after checking to see if the disk is OK, as either will likely cause any repair attempts to make things worse. RAM is pretty easy to check, but for a PSU you need a proper testing device. You can get such a device on Amazon or similar sites for about 25USD, and it's generally worth having around for troubleshooting. Assuming your disk and RAM are good, the next thing to do would be try and get the filesystem into a more usable state. The best option for this is to expand the filesystem if possible. Given that you're running right near capacity, I'd suggest at least 16G of extra space if possible. If that isn't a viable solution for you, the other option is to delete some of the oldest snapshots (Ideally enough that you have at least a few GB of extra space in the data chunks and a few hundred MB in the metadata chunks), then add a 4-8GB device to the FS temporarily (a ramdisk or flash drive works well for this), and run a full balance. If you're lucky, this will fix any metadata that's messed up, and the system should be usable. If not, it shouldn't make things any worse, and you probably want to look at btrfs restore to copy out the data to a new filesystem (ideally a bigger one). -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Filesystem forced to readonly after use
I just wouldn't use btrfs repair with this version of progs, go back to v4.6.1 or upgrade to 4.7.2. You could do an offline check (no repair) and see if that reveals anything useful for developers. But I can't tell what's going on from the call trace. -- Chris Murphy -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Filesystem forced to readonly after use
Hello, I have a BTRFS filesystem that is reverting to read-only after a few moments of use. There is a stack trace visible in the kernel log, which is attached. Here is my system information: # uname -a Linux rescue 4.7.2-1-ARCH #1 SMP PREEMPT Sat Aug 20 23:02:56 CEST 2016 x86_64 GNU/Linux # btrfs --version btrfs-progs v4.7 # btrfs fi show Label: 'linux' uuid: 79862c20-d0b0-4ffa-a9af-e3a40868a243 Total devices 1 FS bytes used 284.60GiB devid1 size 300.03GiB used 300.03GiB path /dev/sdb5 # btrfs fi df /mnt Data, single: total=278.00GiB, used=274.68GiB System, DUP: total=8.00MiB, used=64.00KiB System, single: total=4.00MiB, used=0.00B Metadata, DUP: total=11.00GiB, used=9.92GiB Metadata, single: total=8.00MiB, used=0.00B GlobalReserve, single: total=512.00MiB, used=0.00B As soon as the problem started, I saw that the Metadata, DUP was completely used. It become a little better (like above) after a scrub. I can easily recover disk space by removing old snapshots, if needed. The dmesg output is attached. Before making further recovery attempts, or even restoring from backup, I would like to ask for the best option to proceed. Thanks, Cesar [20048.035688] BTRFS info (device sdb5): disk space caching is enabled [20190.871802] BTRFS error (device sdb5): parent transid verify failed on 160420773888 wanted 181826 found 181573 [20190.882573] BTRFS error (device sdb5): parent transid verify failed on 160420773888 wanted 181826 found 181573 [20190.882607] [ cut here ] [20190.882642] WARNING: CPU: 3 PID: 5026 at fs/btrfs/extent-tree.c:2963 btrfs_run_delayed_refs+0x28c/0x2c0 [btrfs] [20190.882645] BTRFS: Transaction aborted (error -5) [20190.882648] Modules linked in: hid_generic usbhid hid btrfs xor raid6_pq sr_mod cdrom intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass joydev mousedev crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul dell_wmi dell_laptop amdkfd amd_iommu_v2 glue_helper sparse_keymap dell_smbios uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_core videodev radeon media ums_realtek ablk_helper ttm cryptd snd_hda_codec_hdmi arc4 snd_hda_codec_realtek snd_hda_codec_generic dcdbas dell_smm_hwmon iwldvm iTCO_wdt mac80211 iTCO_vendor_support xhci_pci xhci_hcd r8169 snd_hda_intel snd_hda_codec mii iwlwifi evdev i915 input_leds led_class btusb btrtl btbcm btintel bluetooth intel_cstate intel_rapl_perf cfg80211 psmouse pcspkr [20190.882725] snd_hda_core mac_hid rfkill snd_hwdep thermal wmi snd_pcm drm_kms_helper snd_timer drm snd soundcore intel_gtt shpchp syscopyarea ahci sysfillrect sysimgblt fb_sys_fops i2c_algo_bit libahci fjes libata button ac battery mei_me video mei i2c_i801 lpc_ich dell_smo8800 tpm_tis tpm sch_fq_codel ip_tables x_tables ext4 crc16 jbd2 mbcache sd_mod uas usb_storage scsi_mod serio_raw atkbd libps2 ehci_pci ehci_hcd usbcore usb_common i8042 serio [20190.882782] CPU: 3 PID: 5026 Comm: kworker/u16:2 Tainted: GW 4.7.2-1-ARCH #1 [20190.882785] Hardware name: Dell Inc. Dell System Vostro 3450/0GG0VM, BIOS A05 05/24/2011 [20190.882814] Workqueue: btrfs-extent-refs btrfs_extent_refs_helper [btrfs] [20190.882818] 0286 860a9f71 88015420fc90 812eb132 [20190.882824] 88015420fce0 88015420fcd0 8107a3ab [20190.882828] 0b938efd7800 a0c0be15 880085917688 034c [20190.882834] Call Trace: [20190.882842] [] dump_stack+0x63/0x81 [20190.882847] [] __warn+0xcb/0xf0 [20190.882852] [] warn_slowpath_fmt+0x5f/0x80 [20190.882875] [] btrfs_run_delayed_refs+0x28c/0x2c0 [btrfs] [20190.882895] [] delayed_ref_async_start+0x94/0xb0 [btrfs] [20190.882920] [] btrfs_scrubparity_helper+0x77/0x350 [btrfs] [20190.882943] [] btrfs_extent_refs_helper+0xe/0x10 [btrfs] [20190.882948] [] process_one_work+0x1e5/0x480 [20190.882953] [] worker_thread+0x48/0x4e0 [20190.882958] [] ? process_one_work+0x480/0x480 [20190.882962] [] ? process_one_work+0x480/0x480 [20190.882968] [] kthread+0xd8/0xf0 [20190.882975] [] ret_from_fork+0x1f/0x40 [20190.882981] [] ? kthread_worker_fn+0x170/0x170 [20190.882985] ---[ end trace 99d6d7ec847d19d4 ]--- [20190.882990] BTRFS: error (device sdb5) in btrfs_run_delayed_refs:2963: errno=-5 IO failure [20190.882994] BTRFS info (device sdb5): forced readonly [20295.373706] BTRFS error (device sdb5): cleaner transaction attach returned -30
Re: [RFC] Preliminary BTRFS Encryption
Hi Anand, these are great news! Thanks for yor work. I'm looking forward to use the encryption. I would like to ask a few question regarding the feature set. 1. is encryption of an existing, filled and unencrypted subvolume without manually moving the data possible? 2. What about encrypting the root and boot subvolume? Will it work with grub2? 3. How does btrfs rescue handle the encrypted subvolume to recover data in case of an emergency? 4. Is it possible to unlock a subvolume using a keyfile? Thanks in advance, Wilson > Am 13.09.2016 um 15:39 schrieb Anand Jain : > > > This patchset adds btrfs encryption support. > > The main objective of this series is to have bugs fixed and stability. > I have verified with fstests to confirm that there is no regression. > > A design write-up is coming next, however here below is the quick example > on the cli usage. Please try out, let me know if I have missed something. > > Also would like to mention that a review from the security experts is due, > which is important and I believe those review comments can be accommodated > without major changes from here. > > Also yes, thanks for the emails, I hear, per file encryption and inline > with vfs layer is also important, which is wip among other things in the > list. > > As of now these patch set supports encryption on per subvolume, as > managing properties on per subvolume is a kind of core to btrfs, which is > easier for data center solution-ing, seamlessly persistent and easy to > manage. > > > Steps: > - > > Make sure following kernel TFMs are compiled in. > # cat /proc/crypto | egrep 'cbc\(aes\)|ctr\(aes\)' > name : ctr(aes) > name : cbc(aes) > > Create encrypted subvolume. > # btrfs su create -e 'ctr(aes)' /btrfs/e1 > Create subvolume '/btrfs/e1' > Passphrase: > Again passphrase: > > A key is created and its hash is updated into the subvolume item, > and then added to the system keyctl. > # btrfs su show /btrfs/e1 | egrep -i encrypt >Encryption:ctr(aes)@btrfs:75197c8e (594790215) > > # keyctl show 594790215 > Keyring > 594790215 --alsw-v 0 0 logon: btrfs:75197c8e > > > Now any file data extents under the subvol /btrfs/e1 will be > encrypted. > > You may revoke key using keyctl or btrfs(8) as below. > # btrfs su encrypt -k out /btrfs/e1 > > # btrfs su show /btrfs/e1 | egrep -i encrypt >Encryption:ctr(aes)@btrfs:75197c8e (Required key not available) > > # keyctl show 594790215 > Keyring > Unable to dump key: Key has been revoked > > As the key hash is updated, If you provide wrong passphrase in the next > key in, it won't add key to the system. So we have key verification > from the day1. > > # btrfs su encrypt -k in /btrfs/e1 > Passphrase: > Again passphrase: > ERROR: failed to set attribute 'btrfs.encrypt' to 'ctr(aes)@btrfs:75197c8e' : > Key was rejected by service > > ERROR: key set failed: Key was rejected by service > > # btrfs su encrypt -k in /btrfs/e1 > Passphrase: > Again passphrase: > key for '/btrfs/e1' has logged in with keytag 'btrfs:75197c8e' > > Now if you revoke the key the read / write fails with key error. > > # md5sum /btrfs/e1/2k-test-file > 8c9fbc69125ebe84569a5c1ca088cb14 /btrfs/e1/2k-test-file > > # btrfs su encrypt -k out /btrfs/e1 > > # md5sum /btrfs/e1/2k-test-file > md5sum: /btrfs/e1/2k-test-file: Key has been revoked > > # cp /tfs/1k-test-file /btrfs/e1/ > cp: cannot create regular file ‘/btrfs/e1/1k-test-file’: Key has been revoked > > Plain text memory scratches for security reason is pending. As there are some > key revoke notification challenges to coincide with encryption context switch, > which I do believe should be fixed in the due course, but is not a roadblock > at this stage. > > Thanks, Anand > > > Anand Jain (1): > btrfs: Encryption: Add btrfs encryption support > > fs/btrfs/Makefile | 4 +- > fs/btrfs/btrfs_inode.h | 6 + > fs/btrfs/compression.c | 30 +- > fs/btrfs/compression.h | 10 +- > fs/btrfs/ctree.h| 4 + > fs/btrfs/disk-io.c | 3 + > fs/btrfs/encrypt.c | 807 > fs/btrfs/encrypt.h | 94 + > fs/btrfs/inode.c| 255 - > fs/btrfs/ioctl.c| 67 > fs/btrfs/lzo.c | 2 +- > fs/btrfs/props.c| 331 +++- > fs/btrfs/super.c| 27 +- > fs/btrfs/tests/crypto-tests.c | 376 +++ > fs/btrfs/tests/crypto-tests.h | 38 ++ > fs/btrfs/zlib.c | 2 +- > include/uapi/linux/btrfs_tree.h | 6 +- > 17 files changed, 2027 insertions(+), 35 deletions(-) > create mode 100644 fs/btrfs/encrypt.c > create mode 100644 fs/btrfs/encrypt.h > create mode 100755 fs/btrfs/tests/crypto-tests.c > create mode 100755 fs/btrfs/tests/crypto-tests.h > > Anand Jain (2): > btrfs-progs: make wait_for_commit non static >
Re: possible recursive locking detected, 4.8.0-0.rc6.git0.1.fc25.x86_64+debug
Still happens in rc6.
[ 588.463987] =
[ 588.463988] [ INFO: possible recursive locking detected ]
[ 588.463998] 4.8.0-0.rc6.git0.1.fc25.x86_64+debug #1 Tainted: GW
[ 588.463998] -
[ 588.464000] URL Classifier/2203 is trying to acquire lock:
[ 588.464001] (&ei->log_mutex){+.+...}, at: []
btrfs_log_inode+0x162/0x1190 [btrfs]
[ 588.464024]
but task is already holding lock:
[ 588.464025] (&ei->log_mutex){+.+...}, at: []
btrfs_log_inode+0x162/0x1190 [btrfs]
[ 588.464040]
other info that might help us debug this:
[ 588.464041] Possible unsafe locking scenario:
[ 588.464042]CPU0
[ 588.464042]
[ 588.464043] lock(&ei->log_mutex);
[ 588.464045] lock(&ei->log_mutex);
[ 588.464046]
*** DEADLOCK ***
[ 588.464048] May be due to missing lock nesting notation
[ 588.464049] 3 locks held by URL Classifier/2203:
[ 588.464050] #0: (&sb->s_type->i_mutex_key#14){+.+.+.}, at:
[] btrfs_sync_file+0x163/0x4c0 [btrfs]
[ 588.464064] #1: (sb_internal){.+.+.+}, at: []
__sb_start_write+0xb4/0xf0
[ 588.464069] #2: (&ei->log_mutex){+.+...}, at:
[] btrfs_log_inode+0x162/0x1190 [btrfs]
[ 588.464082]
stack backtrace:
[ 588.464085] CPU: 2 PID: 2203 Comm: URL Classifier Tainted: G
W 4.8.0-0.rc6.git0.1.fc25.x86_64+debug #1
[ 588.464086] Hardware name: Apple Inc.
MacBookPro8,2/Mac-94245A3940C91C80, BIOS
MBP81.88Z.0047.B2C.1510261540 10/26/15
[ 588.464087] 0086 b11b9806 8bbba123f840
89465753
[ 588.464090] 8ab07530 8bbba1238000 8bbba123f908
8910eeae
[ 588.464093] 0282 0003 a123f870
8a428200
[ 588.464095] Call Trace:
[ 588.464099] [] dump_stack+0x86/0xc3
[ 588.464102] [] __lock_acquire+0x78e/0x1290
[ 588.464105] [] ? debug_check_no_obj_freed+0x154/0x260
[ 588.464107] [] ? mark_held_locks+0x76/0xa0
[ 588.464116] [] ? btrfs_free_path+0x26/0x30 [btrfs]
[ 588.464118] [] lock_acquire+0xf6/0x1f0
[ 588.464129] [] ? btrfs_log_inode+0x162/0x1190 [btrfs]
[ 588.464132] [] mutex_lock_nested+0x86/0x3f0
[ 588.464143] [] ? btrfs_log_inode+0x162/0x1190 [btrfs]
[ 588.464153] [] ? btrfs_log_inode+0x162/0x1190 [btrfs]
[ 588.464155] [] ? kmem_cache_alloc+0x27a/0x2d0
[ 588.464163] [] ? btrfs_alloc_path+0x1a/0x20 [btrfs]
[ 588.464174] [] btrfs_log_inode+0x162/0x1190 [btrfs]
[ 588.464176] [] ? __might_sleep+0x49/0x80
[ 588.464187] [] btrfs_log_inode+0xd1b/0x1190 [btrfs]
[ 588.464189] [] ? sched_clock+0x9/0x10
[ 588.464199] [] btrfs_log_inode_parent+0x40f/0x940 [btrfs]
[ 588.464202] [] ? dget_parent+0x99/0x2a0
[ 588.464212] [] btrfs_log_dentry_safe+0x62/0x80 [btrfs]
[ 588.464223] [] btrfs_sync_file+0x311/0x4c0 [btrfs]
[ 588.464226] [] vfs_fsync_range+0x4b/0xb0
[ 588.464228] [] do_fsync+0x3d/0x70
[ 588.464229] [] SyS_fsync+0x10/0x20
[ 588.464231] [] entry_SYSCALL_64_fastpath+0x1f/0xbd
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] btrfs: Encryption: Add btrfs encryption support
Hi Anand,
[auto build test WARNING on btrfs/next]
[cannot apply to v4.8-rc6 next-20160913]
[if your patch is applied to the wrong git tree, please drop us a note to help
improve the system]
[Suggest to use git(>=2.9.0) format-patch --base= (or --base=auto for
convenience) to record what (public, well-known) commit your patch series was
built on]
[Check https://git-scm.com/docs/git-format-patch for more information]
url:
https://github.com/0day-ci/linux/commits/Anand-Jain/btrfs-Encryption-Add-btrfs-encryption-support/20160913-214237
base: https://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs.git
next
config: i386-allmodconfig (attached as .config)
compiler: gcc-6 (Debian 6.1.1-9) 6.1.1 20160705
reproduce:
# save the attached .config to linux build tree
make ARCH=i386
All warnings (new ones prefixed by >>):
fs/btrfs/tests/crypto-tests.c: In function 'test_print_data':
>> fs/btrfs/tests/crypto-tests.c:110:28: warning: format '%lu' expects argument
>> of type 'long unsigned int', but argument 3 has type 'size_t {aka unsigned
>> int}' [-Wformat=]
printk("_BTRFS_: %s: sz %lu: ", prefix, sz);
^
fs/btrfs/tests/crypto-tests.c: In function 'test_ablkciphear2':
fs/btrfs/tests/crypto-tests.c:314:44: warning: format '%lu' expects argument
of type 'long unsigned int', but argument 3 has type 'size_t {aka unsigned
int}' [-Wformat=]
printk("BTRFS_TEST: Encrypt '%s' size '%lu' Failed\n",
^
fs/btrfs/tests/crypto-tests.c:324:44: warning: format '%lu' expects argument
of type 'long unsigned int', but argument 3 has type 'size_t {aka unsigned
int}' [-Wformat=]
printk("BTRFS_TEST: Decrypt '%s' size '%lu' Failed\n",
^
vim +110 fs/btrfs/tests/crypto-tests.c
94 for (offset = 0; offset < TEST_DATA_SIZE; offset = offset +
dlen)
95 memcpy(kaddr + offset, str, dlen);
96
97 flush_kernel_dcache_page(known_data_page);
98 }
99
100 void test_fini(void)
101 {
102 if (known_data_page)
103 __free_page(known_data_page);
104 }
105
106
107 void test_print_data(const char *str, char *prefix, size_t sz, int
print_as_str)
108 {
109 int i;
> 110 printk("_BTRFS_: %s: sz %lu: ", prefix, sz);
111
112 if (print_as_str)
113 for (i = 0; i < sz; i++) printk("%c", str[i]);
114 else
115 for (i = 0; i < sz; i++) printk("%02x ", 0xF & str[i]);
116
117 printk("\n");
118 }
---
0-DAY kernel test infrastructureOpen Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
.config.gz
Description: Binary data
Re: Mixing partitioned and non-partitioned discs in a RAID?
On Mon, Sep 12, 2016 at 4:21 PM, Kai Krakow wrote: > Am Sun, 21 Aug 2016 02:19:33 + (UTC) > schrieb Duncan <1i5t5.dun...@cox.net>: > >> Chris Murphy posted on Sat, 20 Aug 2016 18:36:21 -0600 as excerpted: >> >> > FAT leaves a lot to be desired but it's pretty universally >> > supported and almost trivial to repair *if* the volume is >> > repairable in the first place. The much bigger issue with ESP on >> > Linux is this neurotic tendency of distros to persistently mount >> > shit that does not need to be mounted. Like the ESP, and even the >> > dedicated boot volume. They only need to be mounted when being >> > updated and then should be umounted. And worse the convention is to >> > do nested mount with /boot and then /boot/efi for the ESP so it's >> > twice as bad a practice. By virtue of mounting the ESP the dirty >> > bit is set, so any crash means it must be fsck'd and if that >> > doesn't work, it's game over for that volume. Fragile setup. >> >> Depends on the distro. On gentoo, you set it up the way you want of >> course, but the recommendation has always been /boot, and now the >> ESP, not mounted by default. >> >> But that would be /expected/ on gentoo, since being able to configure >> it the way you want is the whole /point/ of running gentoo in the >> first place. Sort of like arch, only much more so. > > systemd systems (I'm booting Gentoo with systemd) should auto-mount ESP > to /boot on access, and auto-unmount after a short timeout. So the > solution to this problem is already wired into systemd if you use (a) > proper GPT setup (with correct GUIDs) and (b) do not mention /boot in > fstab. What distro uses this layout by default? It's certainly incompatible with multiboot where either Windows or macOS exist first. It also necessitates an excessively large ESP to store all possible kernels and initrds for every OS, making it all the harder to keep sync'd in any case where even a simple raid1 setup is desired. The ESP is a file system that doesn't support permissions, acls, or security labels either. So I'm not really a fan of this, or even nested /boot/efi either for that matter. They should be mounted in /run only on demand and only by the process that has permission to modify each. -- Chris Murphy -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] btrfs: Encryption: Add btrfs encryption support
Hi Anand,
[auto build test WARNING on btrfs/next]
[cannot apply to v4.8-rc6 next-20160913]
[if your patch is applied to the wrong git tree, please drop us a note to help
improve the system]
[Suggest to use git(>=2.9.0) format-patch --base= (or --base=auto for
convenience) to record what (public, well-known) commit your patch series was
built on]
[Check https://git-scm.com/docs/git-format-patch for more information]
url:
https://github.com/0day-ci/linux/commits/Anand-Jain/btrfs-Encryption-Add-btrfs-encryption-support/20160913-214237
base: https://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs.git
next
config: i386-randconfig-s0-201637 (attached as .config)
compiler: gcc-6 (Debian 6.1.1-9) 6.1.1 20160705
reproduce:
# save the attached .config to linux build tree
make ARCH=i386
All warnings (new ones prefixed by >>):
In file included from include/linux/printk.h:6:0,
from include/linux/kernel.h:13,
from include/linux/crypto.h:21,
from fs/btrfs/encrypt.c:20:
fs/btrfs/encrypt.c: In function 'btrfs_blkcipher':
include/linux/kern_levels.h:4:18: warning: format '%lu' expects argument of
type 'long unsigned int', but argument 2 has type 'size_t {aka unsigned int}'
[-Wformat=]
#define KERN_SOH "\001" /* ASCII Start Of Header */
^
include/linux/kern_levels.h:10:18: note: in expansion of macro 'KERN_SOH'
#define KERN_ERR KERN_SOH "3" /* error conditions */
^~~~
include/linux/printk.h:264:9: note: in expansion of macro 'KERN_ERR'
printk(KERN_ERR pr_fmt(fmt), ##__VA_ARGS__)
^~~~
>> fs/btrfs/encrypt.c:327:3: note: in expansion of macro 'pr_err'
pr_err("BTRFS: crypto, blk can't work with len %lu\n", len);
^~
fs/btrfs/encrypt.c: In function 'btrfs_decrypt_pages_bio':
include/linux/kern_levels.h:4:18: warning: format '%lu' expects argument of
type 'long unsigned int', but argument 4 has type 'size_t {aka unsigned int}'
[-Wformat=]
#define KERN_SOH "\001" /* ASCII Start Of Header */
^
include/linux/kern_levels.h:10:18: note: in expansion of macro 'KERN_SOH'
#define KERN_ERR KERN_SOH "3" /* error conditions */
^~~~
include/linux/printk.h:264:9: note: in expansion of macro 'KERN_ERR'
printk(KERN_ERR pr_fmt(fmt), ##__VA_ARGS__)
^~~~
fs/btrfs/encrypt.c:757:3: note: in expansion of macro 'pr_err'
pr_err("BTRFS: crypto, untested: pages to be decrypted is less than
expected, "\
^~
vim +/pr_err +327 fs/btrfs/encrypt.c
311 int ret = -EFAULT;
312 struct scatterlist sg;
313 unsigned int ivsize = 0;
314 unsigned int blksize = 0;
315 char *cipher = "cbc(aes)";
316 struct blkcipher_desc desc;
317 struct crypto_blkcipher *blkcipher = NULL;
318
319 blkcipher = crypto_alloc_blkcipher(cipher, 0, 0);
320 if (IS_ERR(blkcipher)) {
321 pr_err("BTRFS: crypto, allocate blkcipher handle for
%s\n", cipher);
322 return -PTR_ERR(blkcipher);
323 }
324
325 blksize = crypto_blkcipher_blocksize(blkcipher);
326 if (len < blksize) {
> 327 pr_err("BTRFS: crypto, blk can't work with len %lu\n",
> len);
328 ret = -EINVAL;
329 goto out;
330 }
331
332 if (crypto_blkcipher_setkey(blkcipher, btrfs_req->key,
333 btrfs_req->key_len)) {
334 pr_err("BTRFS: crypto, key could not be set\n");
335 ret = -EAGAIN;
---
0-DAY kernel test infrastructureOpen Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
.config.gz
Description: Binary data
Re: [PATCH] btrfs: Encryption: Add btrfs encryption support
Hi Anand,
[auto build test WARNING on btrfs/next]
[cannot apply to v4.8-rc6 next-20160913]
[if your patch is applied to the wrong git tree, please drop us a note to help
improve the system]
[Suggest to use git(>=2.9.0) format-patch --base= (or --base=auto for
convenience) to record what (public, well-known) commit your patch series was
built on]
[Check https://git-scm.com/docs/git-format-patch for more information]
url:
https://github.com/0day-ci/linux/commits/Anand-Jain/btrfs-Encryption-Add-btrfs-encryption-support/20160913-214237
base: https://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs.git
next
config: i386-randconfig-x009-201637 (attached as .config)
compiler: gcc-6 (Debian 6.1.1-9) 6.1.1 20160705
reproduce:
# save the attached .config to linux build tree
make ARCH=i386
All warnings (new ones prefixed by >>):
In file included from include/linux/printk.h:6:0,
from include/linux/kernel.h:13,
from include/linux/list.h:8,
from include/linux/hashtable.h:9,
from fs/btrfs/props.c:19:
fs/btrfs/props.c: In function '__btrfs_set_prop':
include/linux/kern_levels.h:4:18: warning: format '%lu' expects argument of
type 'long unsigned int', but argument 5 has type 'size_t {aka unsigned int}'
[-Wformat=]
#define KERN_SOH "\001" /* ASCII Start Of Header */
^
include/linux/kern_levels.h:10:18: note: in expansion of macro 'KERN_SOH'
#define KERN_ERR KERN_SOH "3" /* error conditions */
^~~~
include/linux/printk.h:264:9: note: in expansion of macro 'KERN_ERR'
printk(KERN_ERR pr_fmt(fmt), ##__VA_ARGS__)
^~~~
>> fs/btrfs/props.c:177:3: note: in expansion of macro 'pr_err'
pr_err("BTRFS: property apply failed %s %d %s %lu\n",
^~
fs/btrfs/props.c: In function 'prop_encrypt_validate':
include/linux/kern_levels.h:4:18: warning: format '%lu' expects argument of
type 'long unsigned int', but argument 4 has type 'size_t {aka unsigned int}'
[-Wformat=]
#define KERN_SOH "\001" /* ASCII Start Of Header */
^
include/linux/kern_levels.h:10:18: note: in expansion of macro 'KERN_SOH'
#define KERN_ERR KERN_SOH "3" /* error conditions */
^~~~
include/linux/printk.h:264:9: note: in expansion of macro 'KERN_ERR'
printk(KERN_ERR pr_fmt(fmt), ##__VA_ARGS__)
^~~~
fs/btrfs/props.c:567:3: note: in expansion of macro 'pr_err'
pr_err("BTRFS: %lu mal formed value '%s' %lu\n",
^~
fs/btrfs/props.c: In function 'prop_cryptoiv_apply':
include/linux/kern_levels.h:4:18: warning: format '%lu' expects argument of
type 'long unsigned int', but argument 4 has type 'size_t {aka unsigned int}'
[-Wformat=]
#define KERN_SOH "\001" /* ASCII Start Of Header */
^
include/linux/kern_levels.h:10:18: note: in expansion of macro 'KERN_SOH'
#define KERN_ERR KERN_SOH "3" /* error conditions */
^~~~
include/linux/printk.h:264:9: note: in expansion of macro 'KERN_ERR'
printk(KERN_ERR pr_fmt(fmt), ##__VA_ARGS__)
^~~~
fs/btrfs/props.c:696:3: note: in expansion of macro 'pr_err'
pr_err("BTRFS: %lu prop_cryptoiv_apply failed ret %d len %lu\n",
^~
vim +/pr_err +177 fs/btrfs/props.c
161 ASSERT(ret == 0);
162
163 return ret;
164 }
165
166 ret = handler->validate(inode, value, value_len);
167 if (ret) {
168 return ret;
169 }
170 ret = __btrfs_setxattr(trans, inode, handler->xattr_name,
171 value, value_len, flags);
172 if (ret) {
173 return ret;
174 }
175 ret = handler->apply(inode, value, value_len);
176 if (ret && ret != -EKEYREJECTED) {
> 177 pr_err("BTRFS: property apply failed %s %d %s %lu\n",
178 name, ret, value, value_len);
179 __btrfs_setxattr(trans, inode, handler->xattr_name,
180 NULL, 0, flags);
181 return ret;
182 }
183
184 set_bit(BTRFS_INODE_HAS_PROPS, &BTRFS_I(inode)->runtime_flags);
185
---
0-DAY kernel test infrastructureOpen Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
.config.gz
Description: Binary data
[PATCH 2/2] btrfs-progs: add encryption support
Based on v4.7.2
Depends on keyctl-utils and libscrypt packages.
Signed-off-by: Anand Jain
---
Makefile.in | 5 +-
btrfs-list.c | 23 +++
cmds-filesystem.c | 4 +-
cmds-restore.c| 16 ++
cmds-subvolume.c | 101 +++-
commands.h| 1 +
ctree.h | 5 +-
encrypt.c | 455 ++
encrypt.h | 46 ++
props.c | 4 +
utils.h | 1 +
11 files changed, 654 insertions(+), 7 deletions(-)
create mode 100644 encrypt.c
create mode 100644 encrypt.h
diff --git a/Makefile.in b/Makefile.in
index fd68b3eeeba7..6e857b763213 100644
--- a/Makefile.in
+++ b/Makefile.in
@@ -44,7 +44,8 @@ DISABLE_BTRFSCONVERT = @DISABLE_BTRFSCONVERT@
BTRFSCONVERT_EXT2 = @BTRFSCONVERT_EXT2@
EXTRA_CFLAGS :=
-EXTRA_LDFLAGS :=
+# asj fixme, remove path hardcode
+EXTRA_LDFLAGS := /usr/lib/libscrypt.so.0 /usr/lib/libkeyutils.so
DEBUG_CFLAGS_DEFAULT = -O0 -U_FORTIFY_SOURCE -ggdb3
DEBUG_CFLAGS_INTERNAL =
@@ -87,7 +88,7 @@ objects = ctree.o disk-io.o radix-tree.o extent-tree.o
print-tree.o \
extent-cache.o extent_io.o volumes.o utils.o repair.o \
qgroup.o raid6.o free-space-cache.o list_sort.o props.o \
ulist.o qgroup-verify.o backref.o string-table.o task-utils.o \
- inode.o file.o find-root.o free-space-tree.o help.o
+ inode.o file.o find-root.o free-space-tree.o help.o encrypt.o
cmds_objects = cmds-subvolume.o cmds-filesystem.o cmds-device.o cmds-scrub.o \
cmds-inspect.o cmds-balance.o cmds-send.o cmds-receive.o \
cmds-quota.o cmds-qgroup.o cmds-replace.o cmds-check.o \
diff --git a/btrfs-list.c b/btrfs-list.c
index 4e67fe28b9b5..34722731b652 100644
--- a/btrfs-list.c
+++ b/btrfs-list.c
@@ -1923,3 +1923,26 @@ int wait_for_commit(int fd)
return ret;
return ioctl(fd, BTRFS_IOC_WAIT_SYNC, NULL);
}
+
+/*
+ * Fixme: A kind of workaround as of now, actual fix needs
+ * per subvol sync instead of entire FS.
+ */
+int wait_for_commit_subvol(char *subvol)
+{
+ int fd;
+ int ret;
+ DIR *ds;
+
+ fd = open_file_or_dir3(subvol, &ds, O_RDWR);
+ if (fd == -1) {
+ ret = -errno;
+ fprintf(stderr, "ERROR: open '%s' failed: %s\n",
+ subvol, strerror(-ret));
+ return ret;
+ }
+
+ ret = wait_for_commit(fd);
+ close_file_or_dir(fd, ds);
+ return ret;
+}
diff --git a/cmds-filesystem.c b/cmds-filesystem.c
index 76ea82edf23c..4a866c28420a 100644
--- a/cmds-filesystem.c
+++ b/cmds-filesystem.c
@@ -952,6 +952,8 @@ static int parse_compress_type(char *s)
return BTRFS_COMPRESS_ZLIB;
else if (strcmp(optarg, "lzo") == 0)
return BTRFS_COMPRESS_LZO;
+ else if (strcmp(optarg, "ctr(aes)") == 0)
+ return BTRFS_ENCRYPT_AES;
else {
error("unknown compression type %s", s);
exit(1);
@@ -964,7 +966,7 @@ static const char * const cmd_filesystem_defrag_usage[] = {
"",
"-v be verbose",
"-r defragment files recursively",
- "-c[zlib,lzo] compress the file while defragmenting",
+ "-c[zlib,lzo,ctr(aes)] compress/encrypt the file while defragmenting
if it wasn't",
"-f flush data to disk immediately after defragmenting",
"-s start defragment only from byte onward",
"-l len defragment only up to len bytes",
diff --git a/cmds-restore.c b/cmds-restore.c
index b491f083b72b..bed0aba77740 100644
--- a/cmds-restore.c
+++ b/cmds-restore.c
@@ -155,6 +155,19 @@ static int decompress_lzo(struct btrfs_root *root,
unsigned char *inbuf,
return 0;
}
+static int decrypt_ctr_aes(struct btrfs_root *root, unsigned char *inbuf,
+ char *outbuf, u64 compress_len, u64 *decompress_len)
+{
+ /*
+* fixme: This is only for testing, which works only with
+* kernel option BTRFS_CRYPTO_TEST_BYDUMMYENC, where
+* ciphertext == plaintext
+*/
+ memcpy(outbuf, inbuf, compress_len);
+ *decompress_len = compress_len;
+ return 0;
+}
+
static int decompress(struct btrfs_root *root, char *inbuf, char *outbuf,
u64 compress_len, u64 *decompress_len, int compress)
{
@@ -165,6 +178,9 @@ static int decompress(struct btrfs_root *root, char *inbuf,
char *outbuf,
case BTRFS_COMPRESS_LZO:
return decompress_lzo(root, (unsigned char *)inbuf, outbuf,
compress_len, decompress_len);
+ case BTRFS_ENCRYPT_AES:
+ return decrypt_ctr_aes(root, (unsigned char *)inbuf, outbuf,
+ compress_len, decompress_len);
default:
break;
}
diff --git a/cmds-subvolume.c b/cmds-subvolume.c
index 5df7af56c7f8..452e71755
[PATCH] fstests: btrfs: support encryption
This will help to test kernel encryption patch, and
when compiled with the below defines. So to use the
existing fstests test-cases on top of encryption.
diff --git a/fs/btrfs/encrypt.h b/fs/btrfs/encrypt.h
index 8e794da9d8f5..1ae6840d0742 100644
--- a/fs/btrfs/encrypt.h
+++ b/fs/btrfs/encrypt.h
@@ -25,9 +25,9 @@
#ifndef BTRFS_CRYPT_SUB_FEATURES
//testing
//enable method
- #define BTRFS_CRYPTO_TEST_ENABLE_BYMNTOPT 0
+ #define BTRFS_CRYPTO_TEST_ENABLE_BYMNTOPT 1
//key choice
- #define BTRFS_CRYPTO_TEST_BYDUMMYKEY0 //off rest
+ #define BTRFS_CRYPTO_TEST_BYDUMMYKEY1 //off rest
#define BTRFS_CRYPTO_TEST_BYDUMMYENC0 //off rest
Now use the following mount option during fstests to
exercise the extents with encryption.
MOUNT_OPTIONS="-o compress=ctr(aes)"
As of now this mount option isn't for the end users but
for the testing only, but inspired by ecryptfs, we could
provide such an interface if useful.
(Not sending this patch to fstests community as of now, but
it would be in the long run).
Signed-off-by: Anand Jain
---
common/filter.btrfs | 2 +-
common/rc | 2 +-
tests/btrfs/041 | 2 +
tests/btrfs/041.out | 13
tests/btrfs/052 | 12 +++
tests/btrfs/052.out | 214
tests/btrfs/079 | 2 +
tests/btrfs/125 | 2 +-
tests/generic/297 | 6 +-
tests/generic/298 | 2 +-
10 files changed, 251 insertions(+), 6 deletions(-)
diff --git a/common/filter.btrfs b/common/filter.btrfs
index 9970f4d42fce..cf93f6156247 100644
--- a/common/filter.btrfs
+++ b/common/filter.btrfs
@@ -69,7 +69,7 @@ _filter_btrfs_subvol_delete()
_filter_btrfs_compress_property()
{
- sed -e "s/compression=\(lzo\|zlib\)/COMPRESSION=XXX/g"
+ sed -e "s/compression=\(lzo\|zlib\|ctr(aes)\)/COMPRESSION=XXX/g"
}
# filter name of the property from the output, optionally verify against $1
diff --git a/common/rc b/common/rc
index 67762a7fc834..a0e486bf55d2 100644
--- a/common/rc
+++ b/common/rc
@@ -3481,7 +3481,7 @@ _btrfs_stress_remount_compress()
{
local btrfs_mnt=$1
while true; do
- for algo in no zlib lzo; do
+ for algo in no zlib lzo 'ctr(aes)'; do
$MOUNT_PROG -o remount,compress=$algo $btrfs_mnt
done
done
diff --git a/tests/btrfs/041 b/tests/btrfs/041
index 8bb74cd2a241..be4a10fb3746 100755
--- a/tests/btrfs/041
+++ b/tests/btrfs/041
@@ -106,6 +106,8 @@ echo "Testing restore of file compressed with lzo"
test_btrfs_restore "lzo"
echo "Testing restore of file compressed with zlib"
test_btrfs_restore "zlib"
+echo "Testing restore of file encrypted with ctr(aes)"
+test_btrfs_restore "ctr(aes)"
echo "Testing restore of file without any compression"
test_btrfs_restore
diff --git a/tests/btrfs/041.out b/tests/btrfs/041.out
index 9f4e53dec979..b8d5234649ef 100644
--- a/tests/btrfs/041.out
+++ b/tests/btrfs/041.out
@@ -25,6 +25,19 @@ wrote 100/100 bytes at offset 99000
XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
67edd038aaa42adb5a1aa78f2eb1d2b6 SCRATCH_MNT/foo
67edd038aaa42adb5a1aa78f2eb1d2b6
+Testing restore of file encrypted with ctr(aes)
+wrote 10/10 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 10/10 bytes at offset 10
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 2/2 bytes at offset 1
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 11/11 bytes at offset 33000
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 100/100 bytes at offset 99000
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+67edd038aaa42adb5a1aa78f2eb1d2b6 SCRATCH_MNT/foo
+67edd038aaa42adb5a1aa78f2eb1d2b6
Testing restore of file without any compression
wrote 10/10 bytes at offset 0
XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
diff --git a/tests/btrfs/052 b/tests/btrfs/052
index 599d2616f92f..94b555c4f422 100755
--- a/tests/btrfs/052
+++ b/tests/btrfs/052
@@ -186,5 +186,17 @@ _scratch_unmount
echo "Testing with a nocow file and zlib compression"
test_btrfs_clone_same_file "nodatacow,compress-force=zlib"
+_scratch_unmount
+
+echo "Testing with a cow file and ctr(aes) encryption"
+test_btrfs_clone_same_file "compress-force=ctr(aes)"
+
+_scratch_unmount
+
+echo "Testing with a nocow file and ctr(aes) encryption"
+test_btrfs_clone_same_file "nodatacow,compress-force=ctr(aes)"
+
+_scratch_unmount
+
status=0
exit
diff --git a/tests/btrfs/052.out b/tests/btrfs/052.out
index ac5924ecfa04..034d54fa7248 100644
--- a/tests/btrfs/052.out
+++ b/tests/btrfs/052.out
@@ -641,3 +641,217 @@ Blocks modified: [0 - 1]
23 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05
*
30
+Testing with a cow file and ctr(aes) encryption
+Blocks modified: [0 - 1]
+Blocks modified: [2 - 3]
+Blocks modified: [4 - 5]
+Blocks modified:
[RFC] Preliminary BTRFS Encryption
This patchset adds btrfs encryption support. The main objective of this series is to have bugs fixed and stability. I have verified with fstests to confirm that there is no regression. A design write-up is coming next, however here below is the quick example on the cli usage. Please try out, let me know if I have missed something. Also would like to mention that a review from the security experts is due, which is important and I believe those review comments can be accommodated without major changes from here. Also yes, thanks for the emails, I hear, per file encryption and inline with vfs layer is also important, which is wip among other things in the list. As of now these patch set supports encryption on per subvolume, as managing properties on per subvolume is a kind of core to btrfs, which is easier for data center solution-ing, seamlessly persistent and easy to manage. Steps: - Make sure following kernel TFMs are compiled in. # cat /proc/crypto | egrep 'cbc\(aes\)|ctr\(aes\)' name : ctr(aes) name : cbc(aes) Create encrypted subvolume. # btrfs su create -e 'ctr(aes)' /btrfs/e1 Create subvolume '/btrfs/e1' Passphrase: Again passphrase: A key is created and its hash is updated into the subvolume item, and then added to the system keyctl. # btrfs su show /btrfs/e1 | egrep -i encrypt Encryption: ctr(aes)@btrfs:75197c8e (594790215) # keyctl show 594790215 Keyring 594790215 --alsw-v 0 0 logon: btrfs:75197c8e Now any file data extents under the subvol /btrfs/e1 will be encrypted. You may revoke key using keyctl or btrfs(8) as below. # btrfs su encrypt -k out /btrfs/e1 # btrfs su show /btrfs/e1 | egrep -i encrypt Encryption: ctr(aes)@btrfs:75197c8e (Required key not available) # keyctl show 594790215 Keyring Unable to dump key: Key has been revoked As the key hash is updated, If you provide wrong passphrase in the next key in, it won't add key to the system. So we have key verification from the day1. # btrfs su encrypt -k in /btrfs/e1 Passphrase: Again passphrase: ERROR: failed to set attribute 'btrfs.encrypt' to 'ctr(aes)@btrfs:75197c8e' : Key was rejected by service ERROR: key set failed: Key was rejected by service # btrfs su encrypt -k in /btrfs/e1 Passphrase: Again passphrase: key for '/btrfs/e1' has logged in with keytag 'btrfs:75197c8e' Now if you revoke the key the read / write fails with key error. # md5sum /btrfs/e1/2k-test-file 8c9fbc69125ebe84569a5c1ca088cb14 /btrfs/e1/2k-test-file # btrfs su encrypt -k out /btrfs/e1 # md5sum /btrfs/e1/2k-test-file md5sum: /btrfs/e1/2k-test-file: Key has been revoked # cp /tfs/1k-test-file /btrfs/e1/ cp: cannot create regular file ‘/btrfs/e1/1k-test-file’: Key has been revoked Plain text memory scratches for security reason is pending. As there are some key revoke notification challenges to coincide with encryption context switch, which I do believe should be fixed in the due course, but is not a roadblock at this stage. Thanks, Anand Anand Jain (1): btrfs: Encryption: Add btrfs encryption support fs/btrfs/Makefile | 4 +- fs/btrfs/btrfs_inode.h | 6 + fs/btrfs/compression.c | 30 +- fs/btrfs/compression.h | 10 +- fs/btrfs/ctree.h| 4 + fs/btrfs/disk-io.c | 3 + fs/btrfs/encrypt.c | 807 fs/btrfs/encrypt.h | 94 + fs/btrfs/inode.c| 255 - fs/btrfs/ioctl.c| 67 fs/btrfs/lzo.c | 2 +- fs/btrfs/props.c| 331 +++- fs/btrfs/super.c| 27 +- fs/btrfs/tests/crypto-tests.c | 376 +++ fs/btrfs/tests/crypto-tests.h | 38 ++ fs/btrfs/zlib.c | 2 +- include/uapi/linux/btrfs_tree.h | 6 +- 17 files changed, 2027 insertions(+), 35 deletions(-) create mode 100644 fs/btrfs/encrypt.c create mode 100644 fs/btrfs/encrypt.h create mode 100755 fs/btrfs/tests/crypto-tests.c create mode 100755 fs/btrfs/tests/crypto-tests.h Anand Jain (2): btrfs-progs: make wait_for_commit non static btrfs-progs: add encryption support Makefile.in | 5 +- btrfs-list.c | 33 cmds-filesystem.c | 4 +- cmds-restore.c| 16 ++ cmds-subvolume.c | 112 -- commands.h| 1 + ctree.h | 5 +- encrypt.c | 455 ++ encrypt.h | 46 ++ props.c | 4 + utils.h | 2 + 11 files changed, 665 insertions(+), 18 deletions(-) create mode 100644 encrypt.c create mode 100644 encrypt.h Anand Jain (1): fstests: btrfs: support encryption common/filter.btrfs | 2 +- common/rc | 2 +- tests/btrfs/041 | 2 + tests/btrfs/041.out | 13 tests/btrfs/052 | 12 +++ tests/btrfs/052.out | 214 +++
[PATCH] btrfs: Encryption: Add btrfs encryption support
Adds encryption support. Based on v4.7-rc3.
Signed-off-by: Anand Jain
---
fs/btrfs/Makefile | 4 +-
fs/btrfs/btrfs_inode.h | 6 +
fs/btrfs/compression.c | 30 +-
fs/btrfs/compression.h | 10 +-
fs/btrfs/ctree.h| 4 +
fs/btrfs/disk-io.c | 3 +
fs/btrfs/encrypt.c | 807
fs/btrfs/encrypt.h | 94 +
fs/btrfs/inode.c| 255 -
fs/btrfs/ioctl.c| 67
fs/btrfs/lzo.c | 2 +-
fs/btrfs/props.c| 331 +++-
fs/btrfs/super.c| 27 +-
fs/btrfs/tests/crypto-tests.c | 376 +++
fs/btrfs/tests/crypto-tests.h | 38 ++
fs/btrfs/zlib.c | 2 +-
include/uapi/linux/btrfs_tree.h | 6 +-
17 files changed, 2027 insertions(+), 35 deletions(-)
create mode 100644 fs/btrfs/encrypt.c
create mode 100644 fs/btrfs/encrypt.h
create mode 100755 fs/btrfs/tests/crypto-tests.c
create mode 100755 fs/btrfs/tests/crypto-tests.h
diff --git a/fs/btrfs/Makefile b/fs/btrfs/Makefile
index 128ce17a80b0..c185b2f18953 100644
--- a/fs/btrfs/Makefile
+++ b/fs/btrfs/Makefile
@@ -9,7 +9,7 @@ btrfs-y += super.o ctree.o extent-tree.o print-tree.o
root-tree.o dir-item.o \
export.o tree-log.o free-space-cache.o zlib.o lzo.o \
compression.o delayed-ref.o relocation.o delayed-inode.o scrub.o \
reada.o backref.o ulist.o qgroup.o send.o dev-replace.o raid56.o \
- uuid-tree.o props.o hash.o free-space-tree.o
+ uuid-tree.o props.o hash.o free-space-tree.o encrypt.o
btrfs-$(CONFIG_BTRFS_FS_POSIX_ACL) += acl.o
btrfs-$(CONFIG_BTRFS_FS_CHECK_INTEGRITY) += check-integrity.o
@@ -17,4 +17,4 @@ btrfs-$(CONFIG_BTRFS_FS_CHECK_INTEGRITY) += check-integrity.o
btrfs-$(CONFIG_BTRFS_FS_RUN_SANITY_TESTS) += tests/free-space-tests.o \
tests/extent-buffer-tests.o tests/btrfs-tests.o \
tests/extent-io-tests.o tests/inode-tests.o tests/qgroup-tests.o \
- tests/free-space-tree-tests.o
+ tests/free-space-tree-tests.o tests/crypto-tests.o
diff --git a/fs/btrfs/btrfs_inode.h b/fs/btrfs/btrfs_inode.h
index 4919aedb5fc1..8d2ce6c0e384 100644
--- a/fs/btrfs/btrfs_inode.h
+++ b/fs/btrfs/btrfs_inode.h
@@ -24,6 +24,7 @@
#include "extent_io.h"
#include "ordered-data.h"
#include "delayed-inode.h"
+#include "encrypt.h"
/*
* ordered_data_close is set by truncate when a file that used
@@ -207,6 +208,11 @@ struct btrfs_inode {
struct rw_semaphore dio_sem;
struct inode vfs_inode;
+
+ unsigned char key_payload[BTRFS_CRYPTO_KEY_SIZE];
+ u32 key_len;
+ unsigned char cryptoiv[BTRFS_CRYPTO_IV_SIZE];
+ u32 iv_len;
};
extern unsigned char btrfs_filetype_table[];
diff --git a/fs/btrfs/compression.c b/fs/btrfs/compression.c
index 658c39b70fba..fc65f8831a1d 100644
--- a/fs/btrfs/compression.c
+++ b/fs/btrfs/compression.c
@@ -41,6 +41,7 @@
#include "compression.h"
#include "extent_io.h"
#include "extent_map.h"
+#include "encrypt.h"
struct compressed_bio {
/* number of bios pending for this compressed extent */
@@ -83,7 +84,7 @@ struct compressed_bio {
static int btrfs_decompress_biovec(int type, struct page **pages_in,
u64 disk_start, struct bio_vec *bvec,
- int vcnt, size_t srclen);
+ int vcnt, size_t srclen, struct bio *bio);
static inline int compressed_bio_size(struct btrfs_root *root,
unsigned long disk_size)
@@ -180,9 +181,9 @@ static void end_compressed_bio_read(struct bio *bio)
cb->start,
cb->orig_bio->bi_io_vec,
cb->orig_bio->bi_vcnt,
- cb->compressed_len);
+ cb->compressed_len, cb->orig_bio);
csum_failed:
- if (ret)
+ if (ret && ret != -ENOKEY)
cb->errors = 1;
/* release the compressed pages */
@@ -754,15 +755,21 @@ static struct {
static const struct btrfs_compress_op * const btrfs_compress_op[] = {
&btrfs_zlib_compress,
&btrfs_lzo_compress,
+ &btrfs_encrypt_ops,
};
void __init btrfs_init_compress(void)
{
int i;
+ int type;
for (i = 0; i < BTRFS_COMPRESS_TYPES; i++) {
struct list_head *workspace;
+ type = i + 1;
+ if (type == BTRFS_ENCRYPT_AES)
+ continue;
+
INIT_LIST_HEAD(&btrfs_comp_ws[i].idle_ws);
spin_lock_init(&btrfs_comp_ws[i].ws_lock);
atomic_set(&btrfs_comp_ws[i].total_ws, 0);
@@ -801,6 +808,10 @@ static struct list_head *find_workspace(int type)
atomic_t *total_ws = &btrfs_comp_ws[idx].total_ws;
[PATCH 1/2] btrfs-progs: make wait_for_commit non static
wait_for_commit() is needed by encrypt patch set so this patch makes
it non static.
Also as utils.h is included twice deletes one of it.
Signed-off-by: Anand Jain
---
btrfs-list.c | 10 ++
cmds-subvolume.c | 11 ---
utils.h | 1 +
3 files changed, 11 insertions(+), 11 deletions(-)
diff --git a/btrfs-list.c b/btrfs-list.c
index 4cc2ed498536..4e67fe28b9b5 100644
--- a/btrfs-list.c
+++ b/btrfs-list.c
@@ -1913,3 +1913,13 @@ int btrfs_list_get_path_rootid(int fd, u64 *treeid)
*treeid = args.treeid;
return 0;
}
+
+int wait_for_commit(int fd)
+{
+ int ret;
+
+ ret = ioctl(fd, BTRFS_IOC_START_SYNC, NULL);
+ if (ret < 0)
+ return ret;
+ return ioctl(fd, BTRFS_IOC_WAIT_SYNC, NULL);
+}
diff --git a/cmds-subvolume.c b/cmds-subvolume.c
index e7ef67d3449b..5df7af56c7f8 100644
--- a/cmds-subvolume.c
+++ b/cmds-subvolume.c
@@ -36,7 +36,6 @@
#include "commands.h"
#include "utils.h"
#include "btrfs-list.h"
-#include "utils.h"
static int is_subvolume_cleaned(int fd, u64 subvolid)
{
@@ -223,16 +222,6 @@ out:
return retval;
}
-static int wait_for_commit(int fd)
-{
- int ret;
-
- ret = ioctl(fd, BTRFS_IOC_START_SYNC, NULL);
- if (ret < 0)
- return ret;
- return ioctl(fd, BTRFS_IOC_WAIT_SYNC, NULL);
-}
-
static const char * const cmd_subvol_delete_usage[] = {
"btrfs subvolume delete [options] [...]",
"Delete subvolume(s)",
diff --git a/utils.h b/utils.h
index da23bfcc9166..729e50a113a2 100644
--- a/utils.h
+++ b/utils.h
@@ -225,6 +225,7 @@ int test_isdir(const char *path);
const char *subvol_strip_mountpoint(const char *mnt, const char *full_path);
int get_subvol_info(const char *fullpath, struct root_info *get_ri);
+int wait_for_commit(int fd);
/*
* Btrfs minimum size calculation is complicated, it should include at least:
--
2.7.0
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: BTRFS constantly reports "No space left on device" even with a huge unallocated space
Hi! Em Ter, 2016-09-13 às 11:17 +0800, Wang Xiaoguang escreveu: > It maybe a irrelevant question, but do you have compression enabled? > > Regards, > Xiaoguang Wang No, I do not have compression enabled. I'm using openSUSE's default configuration. By the way, I was wrongly mounting the filesystem with `enospc_debug`. It turns out that I modified the fstab in a backup directory, sorry :) Now, I did it correctly so, hopefully, we will have much more information about the problem the next time I see it! Best regards, Ronan Arraes -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Security implications of btrfs receive?
On 2016-09-12 16:25, Chris Murphy wrote: On Mon, Sep 12, 2016 at 5:24 AM, Austin S. Hemmelgarn wrote: After device discovery, specify UUID= instead of a device node. Oh yeah good point, -U --uuid is also doable. I'm not sure what the benefit is of using sysfs to delete a device's node after umounting. If the umount isn't successful or the accounting for all mount points isn't right, there is no warning for device delete, it's gone and it's like physically yanking the device out of the port. Seems dangerous for udisksd to use that? Unbinding the device node from the driver forces the driver to flush all commands and then stop talking to the device. In essence, this is insurance that everything that got written to the device is actually on the device. They're also trying to emulate the Windows behavior of making the disk disappear when you tell the system to prep it for physical disconnection. I personally feel it's more important that we fix the whole UUID issue first. If we fix that, it is likely to at least make things better in this particular case as well. Fair enough. The problem with trying to get this fixed upstream in userspace is that udisks is essentially deprecated because of the work on storaged (which will almost certainly depend on systemd), so you're almost certainly get nothing fixed until someone decides to fork it and maintain it themselves like happened for ConsoleKit. As far as what udisks is doing wrong, based on what you've said and minimal experimentation on my systems, the issue is that it's not identifying the array as one filesystem. They mount by device node to try and avoid the UUID issues (because they affect every filesystem to some degree), but because of that they see a bunch of filesystems. By trying to avoid the UUID issues, they end up with another problem. In this case though, it's really only a problem for BTRFS, and it technically is a result of our design, not theirs, as what we've done with integrating the volume management into the filesystem isn't something that's ever really been used on removable media before. I guess right now the only sure fire options they have: a. Don't automount any Btrfs. (This is what I've recommended.) b. pass --uuid as well as device= for each device in the array as discovered by BTRFS_IOC_FS_INFO and BTRFS_IOC_DEV_INFO: while also not mounting by device node alone for Btrfs. It's important to differentiate between auto-mounting (udisks default), on-demand mounting (what autofs and BSD's amd actually do), and manual mounting as a regular user (the other operational mode for udisks). In this case, both operational modes for udisks are broken with BTRFS, but the autofs/amd method works as long as you configure them right. On my system for example, I have an autofs directory set up to do on-demand mounting by device node, and it works perfectly fine with multi-disk BTRFS filesystems, because the kernel handles the bind mounting of the already mounted FS correctly. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Is stability a joke? (wiki updated)
Am Dienstag, 13. September 2016, 07:28:38 CEST schrieb Austin S. Hemmelgarn: > On 2016-09-12 16:44, Chris Murphy wrote: > > On Mon, Sep 12, 2016 at 2:35 PM, Martin Steigerwald wrote: > >> Am Montag, 12. September 2016, 23:21:09 CEST schrieb Pasi Kärkkäinen: > >>> On Mon, Sep 12, 2016 at 09:57:17PM +0200, Martin Steigerwald wrote: > Am Montag, 12. September 2016, 18:27:47 CEST schrieb David Sterba: > > On Mon, Sep 12, 2016 at 04:27:14PM +0200, David Sterba wrote: […] > > https://btrfs.wiki.kernel.org/index.php/Status > > Great. > > I made to minor adaption. I added a link to the Status page to my > warning > in before the kernel log by feature page. And I also mentioned that at > the time the page was last updated the latest kernel version was 4.7. > Yes, thats some extra work to update the kernel version, but I think > its > beneficial to explicitely mention the kernel version the page talks > about. Everyone who updates the page can update the version within a > second. > >>> > >>> Hmm.. that will still leave people wondering "but I'm running Linux 4.4, > >>> not 4.7, I wonder what the status of feature X is.." > >>> > >>> Should we also add a column for kernel version, so we can add "feature X > >>> is > >>> known to be OK on Linux 3.18 and later".. ? Or add those to "notes" > >>> field, > >>> where applicable? > >> > >> That was my initial idea, and it may be better than a generic kernel > >> version for all features. Even if we fill in 4.7 for any of the features > >> that are known to work okay for the table. > >> > >> For RAID 1 I am willing to say it works stable since kernel 3.14, as this > >> was the kernel I used when I switched /home and / to Dual SSD RAID 1 on > >> this ThinkPad T520. > > > > Just to cut yourself some slack, you could skip 3.14 because it's EOL > > now, and just go from 4.4. > > That reminds me, we should probably make a point to make it clear that > this is for the _upstream_ mainline kernel versions, not for versions > from some arbitrary distro, and that people should check the distro's > documentation for that info. I´d do the following: Really state the first known to work stable kernel version for a feature. But before the table state this: 1) Instead of the first known to work stable kernel for a feature recommend to use the latest upstream kernel or alternatively the latest upstream LTS kernel for those users who want to play it a bit safer. 2) For stable distros such as SLES, RHEL, Ubuntu LTS, Debian Stable recommend to check distro documentation. Note that some distro kernels track upstream kernels quite closely like Debian backport kernel or Ubuntu kernel backports PPA. Thanks, -- Martin -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Is stability a joke? (wiki updated)
On 2016-09-12 16:08, Chris Murphy wrote: On Mon, Sep 12, 2016 at 10:56 AM, Austin S. Hemmelgarn wrote: Things listed as TBD status: 1. Seeding: Seems to work fine the couple of times I've tested it, however I've only done very light testing, and the whole feature is pretty much undocumented. Mostly OK. Odd behaviors: - mount seed (ro), add device, remount mountpoint: this just changed the mounted fs volume UUID - if two sprouts for a seed exist, ambiguous which is remounted rw, you'd have to check - remount should probably be disallowed in this case somehow; require explicit mount of the sprout btrfs fi usage crash when multiple device volume contains seed device https://bugzilla.kernel.org/show_bug.cgi?id=115851 Yeah, like I said, I've only done very light testing. I kind of lost interest in seeding when overlayfs went mainline, as it offers pretty much everything I care about that seeding does, and it's filesystem agnostic. 2. Device Replace: Works perfectly as long as the filesystem itself is not corrupted, all the component devices are working, and the FS isn't using any raid56 profiles. Works fine if only the device being replaced is failing. I've not done much testing WRT replacement when multiple devices are suspect, but what I've done seems to suggest that it might be possible to make it work, but it doesn't currently. On raid56 it sometimes works fine, sometimes corrupts data, and sometimes takes an insanely long time to complete (putting data at risk from subsequent failures while the replace is running). 3. Balance: Works perfectly as long as the filesystem is not corrupted and nothing throws any read or write errors. IOW, only run this on a generally healthy filesystem. Similar caveats to those for replace with raid56 apply here too. 4. File Range Cloning and Out-of-band Dedupe: Similarly, work fine if the FS is healthy. Concur. Missing from the matrix: - default file system for distros recommendation e.g. between enospc and btrfsck status, I'd say in general this is not currently recommended by upstream (short of having a Btrfs kernel developer on staff) I'd add the whole UUID issue to that too. - enospc status e.g. there's new stuff in 4.8 that probably still needs to shake out, and Jeff's found some metadata accounting problem resulting in enospc where there's tons of unallocated space available. e.g. I have empty block groups, and they are not being deallocated, they just stick around, and this is with 4.7 and 4.8 kernels; so whatever was at one time automatically removing totally empty bg's isn't happening anymore. FWIW, that's still working on my systems. - btrfsck status e.g. btrfs-progs 4.7.2 still warns against using --repair, and lists it under dangerous options also; while that's true, Btrfs can't be considered stable or recommended by default e.g. There's still way too many separate repair tools for Btrfs. Depending on how you count there's at least 4, and more realistically 8 ways, scattered across multiple commands. This excludes btrfs check's -E, -r, and -s flags. And it ignores sequence in the success rate. The permutations are just excessive. It's definitely not easy to know how to fix a Btrfs volume should things go wrong. I assume you're counting balance and scrub in that, plus check gives 3, what are you considering the 4th? In the case of just balance, scrub, and check, the differentiation there makes more sense IMHO than combining them, check only runs on offline filesystems (and as much as we want online fsck, I doubt that that will happen any time soon), while scrub and balance operate on online filesystems and do two semantically different things. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Commit 'synchronize incompat feature bits with sysfs files' still missing in for-next?
I've noticed that the 4.5-rc commit 14e46e04: 'btrfs: synchronize incompat feature bits with sysfs files' [1] was reverted later in [2], but despite fixes to protect sysfs with locks & exorcise GFP_NOFS in favor of GFP_KERNEL it was never reinstated - neither for 4.5-final, nor later..and it's been quite a while since then. Is this still valid? I ask because I've just noticed I've had this in my tree since forever, but have never encountered a problem during balance - probably because of all the other fixes. thanks, Holger [1] goo.gl/2DBMSe [2] goo.gl/cIKgv5 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Is stability a joke? (wiki updated)
On 2016-09-12 16:44, Chris Murphy wrote: On Mon, Sep 12, 2016 at 2:35 PM, Martin Steigerwald wrote: Am Montag, 12. September 2016, 23:21:09 CEST schrieb Pasi Kärkkäinen: On Mon, Sep 12, 2016 at 09:57:17PM +0200, Martin Steigerwald wrote: Am Montag, 12. September 2016, 18:27:47 CEST schrieb David Sterba: On Mon, Sep 12, 2016 at 04:27:14PM +0200, David Sterba wrote: I therefore would like to propose that some sort of feature / stability matrix for the latest kernel is added to the wiki preferably somewhere where it is easy to find. It would be nice to archive old matrix'es as well in case someone runs on a bit older kernel (we who use Debian tend to like older kernels). In my opinion it would make things bit easier and perhaps a bit less scary too. Remember if you get bitten badly once you tend to stay away from from it all just in case, if you on the other hand know what bites you can safely pet the fluffy end instead :) Somebody has put that table on the wiki, so it's a good starting point. I'm not sure we can fit everything into one table, some combinations do not bring new information and we'd need n-dimensional matrix to get the whole picture. https://btrfs.wiki.kernel.org/index.php/Status Great. I made to minor adaption. I added a link to the Status page to my warning in before the kernel log by feature page. And I also mentioned that at the time the page was last updated the latest kernel version was 4.7. Yes, thats some extra work to update the kernel version, but I think its beneficial to explicitely mention the kernel version the page talks about. Everyone who updates the page can update the version within a second. Hmm.. that will still leave people wondering "but I'm running Linux 4.4, not 4.7, I wonder what the status of feature X is.." Should we also add a column for kernel version, so we can add "feature X is known to be OK on Linux 3.18 and later".. ? Or add those to "notes" field, where applicable? That was my initial idea, and it may be better than a generic kernel version for all features. Even if we fill in 4.7 for any of the features that are known to work okay for the table. For RAID 1 I am willing to say it works stable since kernel 3.14, as this was the kernel I used when I switched /home and / to Dual SSD RAID 1 on this ThinkPad T520. Just to cut yourself some slack, you could skip 3.14 because it's EOL now, and just go from 4.4. That reminds me, we should probably make a point to make it clear that this is for the _upstream_ mainline kernel versions, not for versions from some arbitrary distro, and that people should check the distro's documentation for that info. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Is stability a joke? (wiki updated)
On 2016-09-13 04:38, Timofey Titovets wrote: https://btrfs.wiki.kernel.org/index.php/Status I suggest to mark RAID1/10 as 'mostly ok' as on btrfs RAID1/10 is safe to data, but not for application that uses it. i.e. it not hide I/O error even if it's can be masked. https://www.spinics.net/lists/linux-btrfs/msg56739.html /* Retest it with upstream 4.7.2 - not fixed */ This doesn't match with what my own testing indicates at least for raid1 mode. I run similar tests myself every time a new stable kernel version comes out (but only on the most recent stable version) once I get my own patches re-based onto it, and I haven't seen issues like this in any of the 4.7 kernels, and don't recall any issues like this in any of the 4.6 kernels. In fact, I've actually dealt with systems with failing disks using BTRFS raid1 mode, including one at work just yesterday where the SATA cable had worked loose from vibrations and was causing significant data corruption. It survived just fine, as have all the other systems I've dealt with which had hardware issues while running BTRFS in raid1 mode. The indicated behavior would be consistent with issues seen sometimes when using compression however, but the OP in the linked message made no indication of there being any in-line compression involved. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Is stability a joke? (wiki updated)
https://btrfs.wiki.kernel.org/index.php/Status I suggest to mark RAID1/10 as 'mostly ok' as on btrfs RAID1/10 is safe to data, but not for application that uses it. i.e. it not hide I/O error even if it's can be masked. https://www.spinics.net/lists/linux-btrfs/msg56739.html /* Retest it with upstream 4.7.2 - not fixed */ -- Have a nice day, Timofey. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
ENOSPACE linux 4.8-rc6 BTRFS: space_info 4 has 18446743524878843904 free, is not full
Hi, this is vanilla linux 4.8-rc6 and i still have ENOSPC issues with btrfs - caused by wrong space_tree entries. [ 9736.921995] [ cut here ] [ 9736.923342] WARNING: CPU: 1 PID: 23942 at fs/btrfs/extent-tree.c:5734 btrfs_free_block_groups+0x35e/0x440 [btrfs] [ 9736.926229] Modules linked in: netconsole xt_multiport iptable_filter ip_tables x_tables 8021q garp bonding sb_edac edac_core x86_pkg_temp_thermal coretemp kvm_intel kvm ipmi_si irqbypass i2c_i801 crc32_pclmul i2c_smbus shpchp ghash_clmulni_intel ipmi_msghandler button loop btrfs dm_mod raid10 raid0 multipath linear raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 md_mod sg sd_mod usbhid xhci_pci igb ehci_pci i2c_algo_bit xhci_hcd ehci_hcd i40e i2c_core ahci usbcore ptp usb_common libahci aacraid pps_core [ 9736.941228] CPU: 1 PID: 23942 Comm: umount Not tainted 4.8.0-rc6 #6 [ 9736.943497] Hardware name: Supermicro Super Server/X10SRi-F, BIOS 2.0 12/17/2015 [ 9736.945561] 8a571d3d3cf8 a33de0b3 [ 9736.947720] 8a571d3d3d38 a3084e01 1666a30e2499 [ 9736.949880] 8a577b216088 8a577b216000 8a577aa17200 [ 9736.952043] Call Trace: [ 9736.952711] [] dump_stack+0x63/0x90 [ 9736.954139] [] __warn+0xd1/0xf0 [ 9736.955466] [] warn_slowpath_null+0x1d/0x20 [ 9737.022831] [] btrfs_free_block_groups+0x35e/0x440 [btrfs] [ 9737.091125] [] close_ctree+0x15d/0x340 [btrfs] [ 9737.159547] [] btrfs_put_super+0x19/0x20 [btrfs] [ 9737.227648] [] generic_shutdown_super+0x6f/0x100 [ 9737.295227] [] kill_anon_super+0x12/0x20 [ 9737.362199] [] btrfs_kill_super+0x16/0xa0 [btrfs] [ 9737.428716] [] deactivate_locked_super+0x43/0x70 [ 9737.494608] [] deactivate_super+0x5c/0x60 [ 9737.559338] [] cleanup_mnt+0x3f/0x90 [ 9737.623414] [] __cleanup_mnt+0x12/0x20 [ 9737.687439] [] task_work_run+0x7e/0xa0 [ 9737.750376] [] exit_to_usermode_loop+0xb0/0xc0 [ 9737.813436] [] do_syscall_64+0x189/0x1f0 [ 9737.875948] [] entry_SYSCALL64_slow_path+0x25/0x25 [ 9737.938449] ---[ end trace 767418320c59f391 ]--- [ 9738.000649] [ cut here ] [ 9738.062721] WARNING: CPU: 1 PID: 23942 at fs/btrfs/extent-tree.c:5735 btrfs_free_block_groups+0x37d/0x440 [btrfs] [ 9738.128037] Modules linked in: netconsole xt_multiport iptable_filter ip_tables x_tables 8021q garp bonding sb_edac edac_core x86_pkg_temp_thermal coretemp kvm_intel kvm ipmi_si irqbypass i2c_i801 crc32_pclmul i2c_smbus shpchp ghash_clmulni_intel ipmi_msghandler button loop btrfs dm_mod raid10 raid0 multipath linear raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 md_mod sg sd_mod usbhid xhci_pci igb ehci_pci i2c_algo_bit xhci_hcd ehci_hcd i40e i2c_core ahci usbcore ptp usb_common libahci aacraid pps_core [ 9738.487487] CPU: 1 PID: 23942 Comm: umount Tainted: GW 4.8.0-rc6 #6 [ 9738.564325] Hardware name: Supermicro Super Server/X10SRi-F, BIOS 2.0 12/17/2015 [ 9738.641240] 8a571d3d3cf8 a33de0b3 [ 9738.719075] 8a571d3d3d38 a3084e01 1667a30e2499 [ 9738.796597] 8a577b216088 8a577b216000 8a577aa17200 [ 9738.874120] Call Trace: [ 9738.950936] [] dump_stack+0x63/0x90 [ 9739.028733] [] __warn+0xd1/0xf0 [ 9739.106261] [] warn_slowpath_null+0x1d/0x20 [ 9739.184672] [] btrfs_free_block_groups+0x37d/0x440 [btrfs] [ 9739.263666] [] close_ctree+0x15d/0x340 [btrfs] [ 9739.341059] [] btrfs_put_super+0x19/0x20 [btrfs] [ 9739.416799] [] generic_shutdown_super+0x6f/0x100 [ 9739.490619] [] kill_anon_super+0x12/0x20 [ 9739.562240] [] btrfs_kill_super+0x16/0xa0 [btrfs] [ 9739.632301] [] deactivate_locked_super+0x43/0x70 [ 9739.700692] [] deactivate_super+0x5c/0x60 [ 9739.767536] [] cleanup_mnt+0x3f/0x90 [ 9739.833250] [] __cleanup_mnt+0x12/0x20 [ 9739.898076] [] task_work_run+0x7e/0xa0 [ 9739.962977] [] exit_to_usermode_loop+0xb0/0xc0 [ 9740.028080] [] do_syscall_64+0x189/0x1f0 [ 9740.092352] [] entry_SYSCALL64_slow_path+0x25/0x25 [ 9740.156898] ---[ end trace 767418320c59f392 ]--- [ 9740.221712] [ cut here ] [ 9740.286589] WARNING: CPU: 1 PID: 23942 at fs/btrfs/extent-tree.c:10062 btrfs_free_block_groups+0x2a9/0x440 [btrfs] [ 9740.354299] Modules linked in: netconsole xt_multiport iptable_filter ip_tables x_tables 8021q garp bonding sb_edac edac_core x86_pkg_temp_thermal coretemp kvm_intel kvm ipmi_si irqbypass i2c_i801 crc32_pclmul i2c_smbus shpchp ghash_clmulni_intel ipmi_msghandler button loop btrfs dm_mod raid10 raid0 multipath linear raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 md_mod sg sd_mod usbhid xhci_pci igb ehci_pci i2c_algo_bit xhci_hcd ehci_hcd i40e i2c_core ahci usbcore ptp usb_common libahci aacraid pps_core [ 9740.720097] CPU: 1 PID: 23942 Comm: umount Tainted: GW 4.8.0-rc6 #6 [ 9740.797412] Hardware
