btrfs rare silent data corruption with kernel data leak

2016-09-20 Thread Zygo Blaxell
Summary: There seem to be two btrfs bugs here: one loses data on writes, and the other leaks data from the kernel to replace it on reads. It all happens after checksums are verified, so the corruption is entirely silent--no EIO errors, kernel messages, or device event statistics. Compressed ext

[PATCH v2 12/14] btrfs-progs: check: fix the return value bug of cmd_check()

2016-09-20 Thread Qu Wenruo
From: Lu Fengqi The function cmd_check() is called by the main function of btrfs.c, its return value will be returned by exit(). Resulting in the loss of significant bits in some cases, for example this value is greater than 0377. If use a bool value "err" to store all of the return value, this w

[PATCH v2 04/14] btrfs-progs: check: introduce function to check inode_extref

2016-09-20 Thread Qu Wenruo
From: Lu Fengqi Introduce a new function check_inode_extref() to check INODE_EXTREF, and call find_dir_item() to find the related DIR_ITEM/DIR_INDEX. Signed-off-by: Lu Fengqi Signed-off-by: Qu Wenruo --- cmds-check.c | 78 1 file ch

[PATCH v2 14/14] btrfs-progs: check: Enhance leaf traversal function to handle missing inode item

2016-09-20 Thread Qu Wenruo
The leaf traversal function in lowmem mode will skip to the first inode item of leaf and begin to check the inode. That's designed to avoid checking overlapping part of a leaf. But that will cause problem in fsck/010 test case, as in that case inode item of the first inode(256) is missing. So abo

[PATCH v2 06/14] btrfs-progs: check: introduce a function to check dir_item

2016-09-20 Thread Qu Wenruo
From: Lu Fengqi Introduce a new function check_dir_item() to check DIR_ITEM/DIR_INDEX, and call find_inode_ref() to find the related INODE_REF/INODE_EXTREF. Signed-off-by: Lu Fengqi Signed-off-by: Qu Wenruo --- cmds-check.c | 125 +++ 1

[PATCH v2 03/14] btrfs-progs: check: introduce function to check inode_ref

2016-09-20 Thread Qu Wenruo
From: Lu Fengqi Introduce a new function check_inode_ref() to check INODE_REF, and call find_dir_item() to find the related DIR_ITEM/DIR_INDEX. Signed-off-by: Lu Fengqi Signed-off-by: Qu Wenruo --- cmds-check.c | 76 1 file changed,

[PATCH v2 05/14] btrfs-progs: check: introduce function to find inode_ref

2016-09-20 Thread Qu Wenruo
From: Lu Fengqi Introduce a new function find_inode_ref() to find INODE_REF/INODE_EXTREF for the given key, and check it with the specified DIR_ITEM/DIR_INDEX match. Signed-off-by: Lu Fengqi Signed-off-by: Qu Wenruo --- cmds-check.c | 149 ++

[PATCH v2 02/14] btrfs-progs: check: introduce function to find dir_item

2016-09-20 Thread Qu Wenruo
From: Lu Fengqi Introduce a new function find_dir_item() to find DIR_ITEM for the given key, and check it with the specified INODE_REF/INODE_EXTREF match. Signed-off-by: Lu Fengqi Signed-off-by: Qu Wenruo --- cmds-check.c | 140 +++ 1 fi

[PATCH v2 00/14] Btrfsck low memory mode with fs/subvol tree check

2016-09-20 Thread Qu Wenruo
The branch can be fetched from my github: https://github.com/adam900710/btrfs-progs/tree/lowmem_fs_tree Already merged lowmem mode fsck only works for extent/chunk tree check. And for fs tree, it's still using original mode codes. This makes btrfs check still eat tons of memory for large fs. Now

[PATCH v2 01/14] btrfs-progs: move btrfs_extref_hash() to hash.h

2016-09-20 Thread Qu Wenruo
From: Lu Fengqi Move btrfs_extref_hash() from inode-item.c to hash.h, so that the function can be called elsewhere. Signed-off-by: Lu Fengqi Signed-off-by: Qu Wenruo --- hash.h | 10 ++ inode-item.c | 8 +--- 2 files changed, 11 insertions(+), 7 deletions(-) diff --git a/h

[PATCH v2 13/14] btrfs-progs: check: skip shared node or leaf check for low_memory mode

2016-09-20 Thread Qu Wenruo
From: Wang Xiaoguang The basic idea is simple. Assume a middle tree node A is shared and its referenceing fs/file tree root ids are 5, 258 and 260, then we only check node A in the tree who has the smallest root id. That means in this case, when checking root tree(5), we check inode A, for root t

[PATCH v2 11/14] btrfs-progs: check: introduce low_memory mode fs_tree check

2016-09-20 Thread Qu Wenruo
From: Lu Fengqi Introduce a new function check_fs_roots_v2() for check fs_tree in low_memory mode. It call check_fs_root_v2() to check fs_root, and call check_root_ref() to check root_ref. Signed-off-by: Lu Fengqi Signed-off-by: Qu Wenruo --- cmds-check.c | 106 +++

[PATCH v2 10/14] btrfs-progs: check: introduce function to check root ref

2016-09-20 Thread Qu Wenruo
From: Lu Fengqi Introduce a new function check_root_ref() to check root_ref/root_backref. Signed-off-by: Lu Fengqi Signed-off-by: Qu Wenruo --- cmds-check.c | 93 1 file changed, 93 insertions(+) diff --git a/cmds-check.c b/cmds-ch

[PATCH v2 09/14] btrfs-progs: check: introduce function to check fs root

2016-09-20 Thread Qu Wenruo
From: Lu Fengqi Introduce a new function check_fs_root_v2() to check fs root, and call check_inode_item to check the items in the tree. Signed-off-by: Lu Fengqi Signed-off-by: Qu Wenruo --- cmds-check.c | 76 1 file changed, 76 inse

[PATCH v2 08/14] btrfs-progs: check: introduce function to check inode item

2016-09-20 Thread Qu Wenruo
From: Lu Fengqi Introduce a new function check_inode_item() to check INODE_ITEM and related ITEMs that have the same inode id. Signed-off-by: Lu Fengqi Signed-off-by: Qu Wenruo --- cmds-check.c | 168 +++ 1 file changed, 168 insertions(+

[PATCH v2 07/14] btrfs-progs: check: introduce function to check file extent

2016-09-20 Thread Qu Wenruo
From: Lu Fengqi Introduce a new function check_file_extent() to check file extent, such as datasum, hole, size. Signed-off-by: Lu Fengqi Signed-off-by: Qu Wenruo --- cmds-check.c | 97 1 file changed, 97 insertions(+) diff --git a/

Re: how to understand "btrfs fi show" output? "No space left" issues

2016-09-20 Thread Chris Murphy
On Tue, Sep 20, 2016 at 12:47 AM, Tomasz Chmielewski wrote: > How to understand the following "btrfs fi show" output? > > # btrfs fi show /var/lib/lxd > Label: 'btrfs' uuid: f5f30428-ec5b-4497-82de-6e20065e6f61 > Total devices 2 FS bytes used 136.18GiB > devid1 size 423.13GiB

Re: Can't mount btrfs raid1

2016-09-20 Thread Chris Murphy
On Tue, Sep 20, 2016 at 5:16 PM, Mirak M wrote: > Hello, > > I have a failure when mounting btrfs. > >> mount -oro,recovery /dev/sda2 sda2_btrfs >> mount: /dev/sda2: can't read superblock What do you get for 'btrfs super-recover -v ' and 'btrfs check ' For this purpose any 4.4+ version is probab

Can't mount btrfs raid1

2016-09-20 Thread Mirak M
Hello, I have a failure when mounting btrfs. > mount -oro,recovery /dev/sda2 sda2_btrfs > mount: /dev/sda2: can't read superblock The kernel log is here http://pastebin.com/tHihHT92 and at the bottom of the email I must admit I did the error of running btrfs check --repair at some point, not kn

Re: ChaCha20 vs. AES performance

2016-09-20 Thread Mathieu Chouquet-Stringer
kent.overstr...@gmail.com (Kent Overstreet) writes: > On Tue, Sep 20, 2016 at 10:23:20AM -0400, Theodore Ts'o wrote: >> On Tue, Sep 20, 2016 at 03:15:19AM -0800, Kent Overstreet wrote: >> > Not on the list or I would've replied directly, but on Haswell, ChaCha20 >> > (in >> > software) is over 2x

Re: splat in split_leaf with integrity checking

2016-09-20 Thread Adam Borowski
On Mon, Sep 19, 2016 at 07:21:47PM -0700, Liu Bo wrote: > On Tue, Sep 20, 2016 at 03:39:27AM +0200, Adam Borowski wrote: > > Hi! > > I just had the following splat in 4.8-rc6 for the third time in a week: > > Sorry for the trouble, this is caused by my patch and here are two fixes[1] > to get it r

Re: multi-device btrfs with single data mode and disk failure

2016-09-20 Thread Chris Murphy
On Tue, Sep 20, 2016 at 2:18 PM, Alexandre Poux wrote: > > > Le 20/09/2016 à 21:46, Chris Murphy a écrit : >> On Tue, Sep 20, 2016 at 1:31 PM, Alexandre Poux wrote: >>> >>> Le 20/09/2016 à 21:11, Chris Murphy a écrit : And no backup? Umm, I'd resolve that sooner than anything else. >>> Yeah

Re: multi-device btrfs with single data mode and disk failure

2016-09-20 Thread Alexandre Poux
Le 20/09/2016 à 22:18, Alexandre Poux a écrit : > > Le 20/09/2016 à 21:46, Chris Murphy a écrit : >> On Tue, Sep 20, 2016 at 1:31 PM, Alexandre Poux wrote: >>> Le 20/09/2016 à 21:11, Chris Murphy a écrit : And no backup? Umm, I'd resolve that sooner than anything else. >>> Yeah you are abso

Re: multi-device btrfs with single data mode and disk failure

2016-09-20 Thread Graham Cobb
On 20/09/16 19:53, Alexandre Poux wrote: > As for moving data to an another volume, since it's only data and > nothing fancy (no subvolume or anything), a simple rsync would do the trick. > My problem in this case is that I don't have enough available space > elsewhere to move my data. > That's why

[PATCH 0/4][V3] metadata throttling in writeback patches

2016-09-20 Thread Josef Bacik
This is the latest set of patches based on my conversations with Jan and Johannes. The biggest change has been changing the metadata accounting counters to be in bytes intead of pages in order to better support varying blocksizes. I've also stopped messing with the other pagecache related counters

[PATCH 3/4] writeback: convert WB_WRITTEN/WB_DIRITED counters to bytes

2016-09-20 Thread Josef Bacik
These are counters that constantly go up in order to do bandwidth calculations. It isn't important what the units are in, as long as they are consistent between the two of them, so convert them to count bytes written/dirtied, and allow the metadata accounting stuff to change the counters as well.

[PATCH 4/4] writeback: introduce super_operations->write_metadata

2016-09-20 Thread Josef Bacik
Now that we have metadata counters in the VM, we need to provide a way to kick writeback on dirty metadata. Introduce super_operations->write_metadata. This allows file systems to deal with writing back any dirty metadata we need based on the writeback needs of the system. Since there is no inod

[PATCH 2/4] writeback: allow for dirty metadata accounting

2016-09-20 Thread Josef Bacik
Btrfs has no bounds except memory on the amount of dirty memory that we have in use for metadata. Historically we have used a special inode so we could take advantage of the balance_dirty_pages throttling that comes with using pagecache. However as we'd like to support different blocksizes it woul

[PATCH 1/4] remove mapping from balance_dirty_pages*()

2016-09-20 Thread Josef Bacik
The only reason we pass in the mapping is to get the inode in order to see if writeback cgroups is enabled, and even then it only checks the bdi and a super block flag. balance_dirty_pages() doesn't even use the mapping. Since balance_dirty_pages*() works on a bdi level, just pass in the bdi and

Re: Post ext3 conversion problems

2016-09-20 Thread Sean Greenslade
On Tue, Sep 20, 2016 at 01:02:38PM +0800, Qu Wenruo wrote: > > Glad to hear you've found the core of the issue. > > > > At this point, I can trigger it immediately. As soon as I log in and run > > dmenu, it will attempt to rebuild its cache file (small text file that's > > just a list of all execu

Re: ChaCha20 vs. AES performance

2016-09-20 Thread Alex Elsayed
On Tue, 20 Sep 2016 07:51:52 -0800, Kent Overstreet wrote: > On Tue, Sep 20, 2016 at 10:23:20AM -0400, Theodore Ts'o wrote: >> On Tue, Sep 20, 2016 at 03:15:19AM -0800, Kent Overstreet wrote: >> > Not on the list or I would've replied directly, but on Haswell, >> > ChaCha20 (in software) is over 2

Re: multi-device btrfs with single data mode and disk failure

2016-09-20 Thread Alexandre Poux
Le 20/09/2016 à 21:46, Chris Murphy a écrit : > On Tue, Sep 20, 2016 at 1:31 PM, Alexandre Poux wrote: >> >> Le 20/09/2016 à 21:11, Chris Murphy a écrit : >>> And no backup? Umm, I'd resolve that sooner than anything else. >> Yeah you are absolutely right, this was a temporary solution which cam

Re: multi-device btrfs with single data mode and disk failure

2016-09-20 Thread Chris Murphy
On Tue, Sep 20, 2016 at 1:54 PM, Alexandre Poux wrote: > > OK, good idea, but to be able to do that, I have to use the patch that > allow me to mount the partition in rw, otherwise I won't be able to > shrink it I suppose.. > And even with the patch I'm not sure that I won't get an IO error the >

Re: multi-device btrfs with single data mode and disk failure

2016-09-20 Thread Chris Murphy
On Tue, Sep 20, 2016 at 1:43 PM, Austin S. Hemmelgarn wrote: >> First off, as Chris said, if you can read the data and don't already have a > backup, that should be your first priority. This really is an edge case > that's not well tested, and the kernel technically doesn't officially > support

Re: multi-device btrfs with single data mode and disk failure

2016-09-20 Thread Alexandre Poux
Le 20/09/2016 à 21:43, Austin S. Hemmelgarn a écrit : > On 2016-09-20 14:53, Alexandre Poux wrote: >> >> >> Le 20/09/2016 à 20:38, Chris Murphy a écrit : >>> On Tue, Sep 20, 2016 at 12:19 PM, Alexandre Poux >>> wrote: Le 20/09/2016 à 19:54, Chris Murphy a écrit : > On Tue, Sep 20,

Re: multi-device btrfs with single data mode and disk failure

2016-09-20 Thread Chris Murphy
On Tue, Sep 20, 2016 at 1:31 PM, Alexandre Poux wrote: > > > Le 20/09/2016 à 21:11, Chris Murphy a écrit : >> And no backup? Umm, I'd resolve that sooner than anything else. > Yeah you are absolutely right, this was a temporary solution which came > to be not that temporary. > And I regret it alr

Re: multi-device btrfs with single data mode and disk failure

2016-09-20 Thread Austin S. Hemmelgarn
On 2016-09-20 14:53, Alexandre Poux wrote: Le 20/09/2016 à 20:38, Chris Murphy a écrit : On Tue, Sep 20, 2016 at 12:19 PM, Alexandre Poux wrote: Le 20/09/2016 à 19:54, Chris Murphy a écrit : On Tue, Sep 20, 2016 at 11:03 AM, Alexandre Poux wrote: If I wanted to try to edit my partitions

Re: multi-device btrfs with single data mode and disk failure

2016-09-20 Thread Chris Murphy
On Tue, Sep 20, 2016 at 12:53 PM, Alexandre Poux wrote: > > > Le 20/09/2016 à 20:38, Chris Murphy a écrit : >> On Tue, Sep 20, 2016 at 12:19 PM, Alexandre Poux wrote: >>> >>> Le 20/09/2016 à 19:54, Chris Murphy a écrit : On Tue, Sep 20, 2016 at 11:03 AM, Alexandre Poux wrote: > If

Re: multi-device btrfs with single data mode and disk failure

2016-09-20 Thread Alexandre Poux
Le 20/09/2016 à 20:56, Austin S. Hemmelgarn a écrit : > On 2016-09-20 13:54, Chris Murphy wrote: >> On Tue, Sep 20, 2016 at 11:03 AM, Alexandre Poux >> wrote: >> >>> If I wanted to try to edit my partitions with an hex editor, where >>> would >>> I find infos on how to do that ? >>> I really don't

Re: multi-device btrfs with single data mode and disk failure

2016-09-20 Thread Austin S. Hemmelgarn
On 2016-09-20 13:54, Chris Murphy wrote: On Tue, Sep 20, 2016 at 11:03 AM, Alexandre Poux wrote: If I wanted to try to edit my partitions with an hex editor, where would I find infos on how to do that ? I really don't want to go this way, but if this is relatively simple, it may be worth to tr

Re: multi-device btrfs with single data mode and disk failure

2016-09-20 Thread Alexandre Poux
Le 20/09/2016 à 20:38, Chris Murphy a écrit : > On Tue, Sep 20, 2016 at 12:19 PM, Alexandre Poux wrote: >> >> Le 20/09/2016 à 19:54, Chris Murphy a écrit : >>> On Tue, Sep 20, 2016 at 11:03 AM, Alexandre Poux wrote: >>> If I wanted to try to edit my partitions with an hex editor, where wou

Re: multi-device btrfs with single data mode and disk failure

2016-09-20 Thread Chris Murphy
On Tue, Sep 20, 2016 at 12:19 PM, Alexandre Poux wrote: > > > Le 20/09/2016 à 19:54, Chris Murphy a écrit : >> On Tue, Sep 20, 2016 at 11:03 AM, Alexandre Poux wrote: >> >>> If I wanted to try to edit my partitions with an hex editor, where would >>> I find infos on how to do that ? >>> I really

Re: multi-device btrfs with single data mode and disk failure

2016-09-20 Thread Alexandre Poux
Le 20/09/2016 à 19:54, Chris Murphy a écrit : > On Tue, Sep 20, 2016 at 11:03 AM, Alexandre Poux wrote: > >> If I wanted to try to edit my partitions with an hex editor, where would >> I find infos on how to do that ? >> I really don't want to go this way, but if this is relatively simple, it >>

Re: [PATCH] Btrfs: kill BUG_ON in do_relocation

2016-09-20 Thread Liu Bo
On Tue, Sep 20, 2016 at 10:03:43AM +0200, David Sterba wrote: > On Mon, Sep 19, 2016 at 04:11:44PM -0700, Liu Bo wrote: > > > > That's EIO. Sometimes the EIO is big enough we have to abort, but > > > > really the abort is just adding bonus. > > > > > > I think we misuse the EIO where we should r

Re: [PATCH] Btrfs: memset to avoid stale content in btree node block

2016-09-20 Thread Liu Bo
On Tue, Sep 20, 2016 at 03:16:36PM +0200, David Sterba wrote: > On Wed, Sep 14, 2016 at 05:22:57PM -0700, Liu Bo wrote: > > During updating btree, we could push items between sibling > > nodes/leaves, for leaves data sections starts reversely from > > the end of the block while for nodes we only ha

Re: multi-device btrfs with single data mode and disk failure

2016-09-20 Thread Chris Murphy
On Tue, Sep 20, 2016 at 11:03 AM, Alexandre Poux wrote: > If I wanted to try to edit my partitions with an hex editor, where would > I find infos on how to do that ? > I really don't want to go this way, but if this is relatively simple, it > may be worth to try. Simple is relative. First you'd

Re: multi-device btrfs with single data mode and disk failure

2016-09-20 Thread Alexandre Poux
Le 20/09/2016 à 00:05, Alexandre Poux a écrit : > > Le 15/09/2016 à 23:54, Chris Murphy a écrit : >> On Thu, Sep 15, 2016 at 3:48 PM, Alexandre Poux wrote: >>> Le 15/09/2016 à 18:54, Chris Murphy a écrit : On Thu, Sep 15, 2016 at 10:30 AM, Alexandre Poux wrote: > Thank you very much fo

Re: Is stability a joke?

2016-09-20 Thread Chris Murphy
btrfs-convert has been rewritten as of btrfs-progs 4.6, and therefore the conversion page could use an update: https://btrfs.wiki.kernel.org/index.php/Conversion_from_Ext3 Anyone wanting to update the page should advise the code is new, check the changelog, the latest btrfs-progs version should be

Re: [PATCH]btrfs-progs: Post btrfs-convert verify permissions and acls

2016-09-20 Thread lakshmipathi . g
On 2016-09-19 20:17, Qu Wenruo wrote: Hi Laksmipathi, At 09/06/2016 03:27 AM, Lakshmipathi.G wrote: Signed-off-by: Lakshmipathi.G --- tests/common.convert | 95 +++- 1 file changed, 94 insertions(+), 1 deletion(-) diff --git a/tests/common.co

Re: [PATCH]btrfs-progs: Add fast,slow symlinks and fifo types to convert test

2016-09-20 Thread lakshmipathi . g
On 2016-09-19 18:21, Qu Wenruo wrote: Just curious, did the new fifo/slow_symlink exposed any convert bug? Thanks, Qu Unfortunately no. I was hoping something will fail, but sadly convert-tests.sh passed! -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of

Re: [PATCH]btrfs-progs: Add fast,slow symlinks and fifo types to convert test

2016-09-20 Thread lakshmipathi . g
On 2016-09-19 11:05, David Sterba wrote: On Thu, Sep 15, 2016 at 11:34:07AM +0200, Lakshmipathi.G wrote: + slow_symlink) + for num in $(seq 1 $DATASET_SIZE); do + fname64=`date +%s | sha256sum | cut -f1 -d'-'` Do you need to gen

Re: ChaCha20 vs. AES performance

2016-09-20 Thread Kent Overstreet
On Tue, Sep 20, 2016 at 10:23:20AM -0400, Theodore Ts'o wrote: > On Tue, Sep 20, 2016 at 03:15:19AM -0800, Kent Overstreet wrote: > > Not on the list or I would've replied directly, but on Haswell, ChaCha20 (in > > software) is over 2x as fast as AES (in hardware), at realistic (for a > > filesyste

Re: Experimental btrfs encryption

2016-09-20 Thread Chris Mason
On 09/19/2016 10:50 PM, Theodore Ts'o wrote: On Mon, Sep 19, 2016 at 08:32:34PM -0400, Chris Mason wrote: That key is used to protect the contents of the data file, and to encrypt filenames and symlink targets --- since filenames can leak significant information about what the user is doing.

Re: [RFC] Preliminary BTRFS Encryption

2016-09-20 Thread Anand Jain
Hi David, On 09/18/2016 02:45 AM, David Sterba wrote: On Sat, Sep 17, 2016 at 12:38:30AM -0400, Zygo Blaxell wrote: There's also a nasty problem with the extent tree--there's only one per filesystem, it's shared between all subvols and block groups, and every extent in that tree has back ref

Re: ChaCha20 vs. AES performance

2016-09-20 Thread Theodore Ts'o
On Tue, Sep 20, 2016 at 03:15:19AM -0800, Kent Overstreet wrote: > Not on the list or I would've replied directly, but on Haswell, ChaCha20 (in > software) is over 2x as fast as AES (in hardware), at realistic (for a > filesystem) block sizes: On Skylake and Broadwell processors, AES is faster (th

[PATCH 4/5] btrfs: convert pr_* to btrfs_* where possible

2016-09-20 Thread jeffm
From: Jeff Mahoney For many printks, we want to know which file system issued the message. This patch converts most pr_* calls to use the btrfs_* versions instead. In some cases, this means adding plumbing to allow call sites access to an fs_info pointer. fs/btrfs/check-integrity.c is left alon

[PATCH 5/5] btrfs: convert send's verbose_printk to btrfs_debug

2016-09-20 Thread jeffm
From: Jeff Mahoney This was basically an open-coded, less flexible dynamic printk. We can just use btrfs_debug instead. Signed-off-by: Jeff Mahoney diff --git a/fs/btrfs/send.c b/fs/btrfs/send.c index ee10345..96bc99d 100644 --- a/fs/btrfs/send.c +++ b/fs/btrfs/send.c @@ -36,10 +36,6 @@ #inc

[PATCH 3/5] btrfs: convert printk(KERN_* to use pr_* calls

2016-09-20 Thread jeffm
From: Jeff Mahoney This patch converts printk(KERN_* style messages to use the pr_* versions. One side effect is that anything that was KERN_DEBUG is now automatically a dynamic debug message. Signed-off-by: Jeff Mahoney diff --git a/fs/btrfs/check-integrity.c b/fs/btrfs/check-integrity.c ind

[PATCH 2/5] btrfs: unsplit printed strings

2016-09-20 Thread jeffm
From: Jeff Mahoney CodingStyle chapter 2: "[...] never break user-visible strings such as printk messages, because that breaks the ability to grep for them." This patch unsplits user-visible strings. Signed-off-by: Jeff Mahoney diff --git a/fs/btrfs/backref.c b/fs/btrfs/backref.c index 455a6b

[PATCH 1/5] btrfs: add dynamic debug support

2016-09-20 Thread jeffm
From: Jeff Mahoney We can re-use the dynamic debugging descriptor to make use of the dynamic debugging mechanism but still use our own printk interface. Defining the DEBUG macro works as it did before. When it's defined, all of the messages default to print. We can also enable all debug messag

[PATCH 0/5] btrfs: printing cleanup patchset

2016-09-20 Thread jeffm
From: Jeff Mahoney This is a patchset I've been working on to clean up message printing, make it adhere to kernel style, and be more consistent. The end result is that we: * use dynamic debugging for debugging messages * merge strings that exceed 80 characters into a single greppable string * co

Re: [PATCH] Btrfs: add error handling for extent buffer in print tree

2016-09-20 Thread David Sterba
On Wed, Sep 14, 2016 at 05:23:39PM -0700, Liu Bo wrote: > Somehow we missed btrfs_print_tree when last time we > updated error handling for read_extent_block(). > > This keeps us from getting a NULL pointer panic when > btrfs_print_tree's read_extent_block() fails. > > Signed-off-by: Liu Bo Rev

Re: [PATCH] Btrfs: memset to avoid stale content in btree node block

2016-09-20 Thread David Sterba
On Wed, Sep 14, 2016 at 05:22:57PM -0700, Liu Bo wrote: > During updating btree, we could push items between sibling > nodes/leaves, for leaves data sections starts reversely from > the end of the block while for nodes we only have key pairs > which are stored one by one from the start of the block

Re: stat(2) returning device ID not existing in mountinfo

2016-09-20 Thread Jeff Mahoney
On 9/16/16 4:28 PM, Tomasz Sterna wrote: > Hi. > > I have spotted an issue with stat(2) call on files on btrfs. > It is giving me dev_t st_dev number that does not correspond to any > mounted filesystem in proc's mountinfo. That's by design. Your particular file system may only use one device bu

Re: Fs: Btrfs - Fix possible ERR_PTR() dereferencing.

2016-09-20 Thread Jeff Mahoney
On 9/20/16 2:48 AM, Shailendra Verma wrote: > This is of course wrong to call kfree() if memdup_user() fails, > no memory was allocated and the error in the error-valued pointer > should be returned. > > Reviewed-by: Ravikant Sharma > Signed-off-by: Shailendra Verma Hi Shailendra - In all thre

[PATCH] btrfs: clean the old superblocks before freeing the device

2016-09-20 Thread jeffm
From: Jeff Mahoney btrfs_rm_device frees the block device but then re-opens it using the saved device name. A race exists between the close and the re-open that allows the block size to be changed. The result is getting stuck forever in the reclaim loop in __getblk_slow. This patch moves the s

Re: Is stability a joke? (wiki updated)

2016-09-20 Thread Austin S. Hemmelgarn
On 2016-09-19 16:15, Zygo Blaxell wrote: On Mon, Sep 19, 2016 at 01:38:36PM -0400, Austin S. Hemmelgarn wrote: I'm not sure if the brfsck is really all that helpful to user as much as it is for developers to better learn about the failure vectors of the file system. ReiserFS had no working fsc

Re: State of the fuzzer

2016-09-20 Thread Lukas Lueg
There are now 21 bugs open on bko, most of them crashes and some undefined behavior. The nodes are now mostly running idle as no new paths are discovered (after around one billion images tested in the current run). My thoughts are to wait until the current bugs have been fixed, then restart the wh

ChaCha20 vs. AES performance

2016-09-20 Thread Kent Overstreet
Not on the list or I would've replied directly, but on Haswell, ChaCha20 (in software) is over 2x as fast as AES (in hardware), at realistic (for a filesystem) block sizes: testing speed of ctr(aes) (ctr(aes-aesni)) decryption test 0 (128 bit key, 16 byte blocks): 1 operation in 378 cycles (16 by

Re: how to understand "btrfs fi show" output? "No space left" issues

2016-09-20 Thread Peter Becker
Output from my nightly balance script for my 15 TB Raid 1 btrfs pool (3x 3TB + 1x 6TB) with ~100 snapshots: Before balance of /media/RAID Data, RAID1: total=5.57TiB, used=5.45TiB System, RAID1: total=32.00MiB, used=832.00KiB Metadata, RAID1: total=7.00GiB, used=6.03GiB GlobalReserve, single: total

Re: how to understand "btrfs fi show" output? "No space left" issues

2016-09-20 Thread Peter Becker
2016-09-20 10:48 GMT+02:00 Hugo Mills : > On Tue, Sep 20, 2016 at 10:34:49AM +0200, Peter Becker wrote: >> More details on the issue and a complete explantion you can find here: >> >> http://marc.merlins.org/perso/btrfs/post_2014-05-04_Fixing-Btrfs-Filesystem-Full-Problems.html >> and >> (Help! I r

Re: how to understand "btrfs fi show" output? "No space left" issues

2016-09-20 Thread Peter Becker
2016-09-20 10:30 GMT+02:00 Andrei Borzenkov : > On Tue, Sep 20, 2016 at 11:20 AM, Peter Becker wrote: > I still do do understand where ENOSPC comes from in the first place. > Filesystem is half empty. Do you suggest that it is normal to get > ENOSPC in this case? Its how the block allocator and t

Re: how to understand "btrfs fi show" output? "No space left" issues

2016-09-20 Thread Hugo Mills
On Tue, Sep 20, 2016 at 10:34:49AM +0200, Peter Becker wrote: > More details on the issue and a complete explantion you can find here: > > http://marc.merlins.org/perso/btrfs/post_2014-05-04_Fixing-Btrfs-Filesystem-Full-Problems.html > and > (Help! I ran out of disk space! ) > https://btrfs.wiki.k

Re: stability matrix (was: Is stability a joke?)

2016-09-20 Thread David Sterba
On Mon, Sep 19, 2016 at 09:45:46PM +0200, Christoph Anton Mitterer wrote: > +1 for all your changes with the following comments in addition... > > > On Mon, 2016-09-19 at 17:27 +0200, David Sterba wrote: > > That's more like a usecase, thats out of the scope of the tabular > > overview. But we ha

Re: how to understand "btrfs fi show" output? "No space left" issues

2016-09-20 Thread Peter Becker
More details on the issue and a complete explantion you can find here: http://marc.merlins.org/perso/btrfs/post_2014-05-04_Fixing-Btrfs-Filesystem-Full-Problems.html and (Help! I ran out of disk space! ) https://btrfs.wiki.kernel.org/index.php/FAQ#Help.21_I_ran_out_of_disk_space.21 And an explant

Re: how to understand "btrfs fi show" output? "No space left" issues

2016-09-20 Thread Andrei Borzenkov
On Tue, Sep 20, 2016 at 11:20 AM, Peter Becker wrote: > The last option completly avoid the ENOSPC issue but produce aditional > workload for your harddrives. > I still do do understand where ENOSPC comes from in the first place. Filesystem is half empty. Do you suggest that it is normal to get E

Re: how to understand "btrfs fi show" output? "No space left" issues

2016-09-20 Thread Peter Becker
Normaly total and used should deviate us a few gb. depend on your write workload you should run btrfs balance start -dusage=60 /mnt every week to avoid "ENOSPC" if you use newer btrfs-progs who supper balance limit filters you should run btrfs balance start -dusage=99 -dlimit=10 /mnt every 3 h

Re: stability matrix (was: Is stability a joke?)

2016-09-20 Thread Hugo Mills
On Tue, Sep 20, 2016 at 07:59:44AM +, Duncan wrote: > Christoph Anton Mitterer posted on Mon, 19 Sep 2016 21:45:46 +0200 as > excerpted: > > > On Mon, 2016-09-19 at 17:27 +0200, David Sterba wrote: > > > >> > For specific features: > >> > - Autodefrag > >> >   - didn't that also cause reflink

Re: [PATCH] Btrfs: kill BUG_ON in do_relocation

2016-09-20 Thread David Sterba
On Mon, Sep 19, 2016 at 04:11:44PM -0700, Liu Bo wrote: > > > That's EIO. Sometimes the EIO is big enough we have to abort, but > > > really the abort is just adding bonus. > > > > I think we misuse the EIO where we should really return EFSCORRUPTED > > that's an alias for EUCLEAN, looking at xf

Re: stability matrix (was: Is stability a joke?)

2016-09-20 Thread Duncan
Christoph Anton Mitterer posted on Mon, 19 Sep 2016 21:45:46 +0200 as excerpted: > On Mon, 2016-09-19 at 17:27 +0200, David Sterba wrote: > >> > For specific features: >> > - Autodefrag >> >   - didn't that also cause reflinks to be broken up? >> >> No and never had. > > Absolutely sure? One ye

Re: how to understand "btrfs fi show" output? "No space left" issues

2016-09-20 Thread Tomasz Chmielewski
On 2016-09-20 16:27, Peter Becker wrote: You have 417(total)-131(used) blocks wo are only partial filled. You should balance your file-system. (...) #or a full balance btrfs balance start /mnt OK, does it mean that btrfs needs some userspace daemon which does the following from time to ti

Re: how to understand "btrfs fi show" output? "No space left" issues

2016-09-20 Thread Tomasz Chmielewski
Yes, have it disabled already (for their datadirs). Tomasz Chmielewski https://lxadm.com On 2016-09-20 16:30, Peter Becker wrote: for the future. disable COW for all database containers 2016-09-20 9:28 GMT+02:00 Peter Becker : * If this NOT solve the "No space left" issues you must remove ol

Re: [PATCH] btrfs: Fix handling of -ENOENT from btrfs_uuid_iter_rem

2016-09-20 Thread Nikolay Borisov
[Resend due to gmail mobile interface defaulting to html layout] >> >> We know its returning -ENOENT, so it should in theory be enough to just >> goto again_search_slot, assuming that we just raced with the deletion. > > > I will apply this on the machine which are exhibitting problems and will > r

Re: how to understand "btrfs fi show" output? "No space left" issues

2016-09-20 Thread Peter Becker
for the future. disable COW for all database containers 2016-09-20 9:28 GMT+02:00 Peter Becker : > * If this NOT solve the "No space left" issues you must remove old snapshots. > > 2016-09-20 9:27 GMT+02:00 Peter Becker : >> Data, RAID1: total=417.12GiB, used=131.33GiB >> >> You have 417(total)-13

Re: how to understand "btrfs fi show" output? "No space left" issues

2016-09-20 Thread Peter Becker
* If this NOT solve the "No space left" issues you must remove old snapshots. 2016-09-20 9:27 GMT+02:00 Peter Becker : > Data, RAID1: total=417.12GiB, used=131.33GiB > > You have 417(total)-131(used) blocks wo are only partial filled. > You should balance your file-system. > > At first you need so

Re: how to understand "btrfs fi show" output? "No space left" issues

2016-09-20 Thread Peter Becker
Data, RAID1: total=417.12GiB, used=131.33GiB You have 417(total)-131(used) blocks wo are only partial filled. You should balance your file-system. At first you need some free space. You could remove some files / old snapshots etc. or you add a empty USB-Stick with min. 4 GB to your BTRFS-Pool (af

Re: how to understand "btrfs fi show" output? "No space left" issues

2016-09-20 Thread Tomasz Chmielewski
OK, according to that - it means 423.13GiB out of total available space, 423.13GiB, has been allocated. Is it good? Is it bad? Is it why I'm getting "No space left" issues? Why has it allocated all available space, if only around 1/3 of space is in use, according to other tools (less than 140