Re: btrfs convert problem

2018-09-13 Thread Nikolay Borisov
On 14.09.2018 02:17, Qu Wenruo wrote: > > > On 2018/9/14 上午12:37, Nikolay Borisov wrote: >> >> >> On 13.09.2018 19:15, Serhat Sevki Dincer wrote: -1 seems to be EPERM, is your device write-protected, readonly or something like that ? >>> >>> I am able to use ext4 partition

Re: [PATCH v2 5/7] btrfs-progs: lowmem: continue to check item in last slot while checking inodes

2018-09-13 Thread Su Yue
On 09/14/2018 07:43 AM, Qu Wenruo wrote: On 2018/9/13 上午4:49, damenly...@gmail.com wrote: From: Su Yue After call of check_inode_item(), path may point to the last unchecked slot of the leaf. The outer walk_up_tree() always treats the position as checked item then skips to next item.

Re: [PATCH v2 4/7] btrfs-progs: lowmem: search key of root again after check_fs_root() under repair

2018-09-13 Thread Su Yue
On 09/14/2018 07:37 AM, Qu Wenruo wrote: On 2018/9/13 上午4:49, damenly...@gmail.com wrote: From: Su Yue In check_fs_roots_lowmem(), we do search and follow the resulted path to call check_fs_root(), then call btrfs_next_item() to check next root. However, if repair is enabled, the root

Re: [PATCH v2 3/7] btrfs-progs: lowmem check: find dir_item by di_key in check_dir_item()

2018-09-13 Thread Su Yue
On 09/14/2018 07:33 AM, Qu Wenruo wrote: On 2018/9/13 上午4:49, damenly...@gmail.com wrote: From: Su Yue In check_dir_item, we are going to search corresponding dir_item/index. Commit 564901eac7a4 ("btrfs-progs: check: introduce print_dir_item_err()") Changed argument name from key to

Re: [PATCH v2 7/7] btrfs-progs: fsck-tests: add test case inode_extref without dir_item and dir_index

2018-09-13 Thread Qu Wenruo
On 2018/9/13 上午4:49, damenly...@gmail.com wrote: > From: Su Yue > > This case contains an inode_extref: > == > ... >item 1 key (257 INODE_EXTREF 3460996356) itemoff 3947 itemsize 24 > index 257 parent 256 namelen 6 name: foo255 > ...

Re: [PATCH v2 6/7] btrfs-progs: lowmem: improve check_inode_extref()

2018-09-13 Thread Qu Wenruo
On 2018/9/13 上午4:49, damenly...@gmail.com wrote: > From: Su Yue > > inode_extref is much similar with inode_ref, most codes are reused in > check_inode_extref(). > Exception: > There is no need to check root directory, so remove it. > Make check_inode_extref() verify hash value with key

Re: [PATCH v2 5/7] btrfs-progs: lowmem: continue to check item in last slot while checking inodes

2018-09-13 Thread Qu Wenruo
On 2018/9/13 上午4:49, damenly...@gmail.com wrote: > From: Su Yue > > After call of check_inode_item(), path may point to the last unchecked > slot of the leaf. The outer walk_up_tree() always treats the position > as checked item then skips to next item. So check_inode_item() always set its

Re: [PATCH v2 4/7] btrfs-progs: lowmem: search key of root again after check_fs_root() under repair

2018-09-13 Thread Qu Wenruo
On 2018/9/13 上午4:49, damenly...@gmail.com wrote: > From: Su Yue > > In check_fs_roots_lowmem(), we do search and follow the resulted path > to call check_fs_root(), then call btrfs_next_item() to check next > root. > However, if repair is enabled, the root tree can be cowed, the > existed

Re: [PATCH v2 3/7] btrfs-progs: lowmem check: find dir_item by di_key in check_dir_item()

2018-09-13 Thread Qu Wenruo
On 2018/9/13 上午4:49, damenly...@gmail.com wrote: > From: Su Yue > > In check_dir_item, we are going to search corresponding > dir_item/index. > > Commit 564901eac7a4 ("btrfs-progs: check: introduce > print_dir_item_err()") Changed argument name from key to di_key but > forgot to change the

Re: [PATCH v2 2/7] btrfs-progs: make btrfs_unlink() lookup inode_extref

2018-09-13 Thread Qu Wenruo
On 2018/9/13 上午4:49, damenly...@gmail.com wrote: > From: Su Yue > > btrfs_unlink() uses btrfs_lookup_inode_ref() to look up inode_ref > but forget inode_extref case. > > Let btrfs_unlink() call btrfs_lookup_inode_extref() if inode_ref is > found and EXTENDED_IREF feature is enabled. > >

Re: [PATCH v2 1/7] btrfs-progs: adjust arguments of btrfs_lookup_inode_extref()

2018-09-13 Thread Qu Wenruo
On 2018/9/13 上午4:49, damenly...@gmail.com wrote: > From: Su Yue > > The argument index is not used in btrfs_lookup_inode_extref(), > so remove it. > And adjust positions its arguments to make it consistent with > kernel part. > > No functional change. > > Fixes: 260675657767 ("btrfs-progs:

Re: btrfs convert problem

2018-09-13 Thread Qu Wenruo
On 2018/9/14 上午12:37, Nikolay Borisov wrote: > > > On 13.09.2018 19:15, Serhat Sevki Dincer wrote: >>> -1 seems to be EPERM, is your device write-protected, readonly or >>> something like that ? >> >> I am able to use ext4 partition read/write, created a file and wrote >> in it, un/re mounted

Re: [PATCH 3/3] btrfs-progs: check: lowmem: Refactor extent type checks in check_file_extent

2018-09-13 Thread Qu Wenruo
On 2018/9/13 下午8:05, Nikolay Borisov wrote: > Make the checks in check_file_extent a bit more explicit. First we check > for unknown type and fail accordingly. Then we check for inline extent > and handle it in the newly introduced check_file_extent_inline. Finally > if none of the above checks

Re: [PATCH 2/3] btrfs-progs: check: lowmem: Refactor extent len test in check_file_extent_inline

2018-09-13 Thread Qu Wenruo
On 2018/9/13 下午8:05, Nikolay Borisov wrote: > Instead of having another top-level if which checks for > 'extent_num_bytes != item_inline_len' only if we are !compressed, just > move the 'if' inside the 'else' branch of the first top-level if, since > it has already checked for !compressed or

Re: [PATCH 1/3] btrfs-progs: check: lowmem: Factor out inline extent checking code in its own function

2018-09-13 Thread Qu Wenruo
On 2018/9/13 下午8:05, Nikolay Borisov wrote: > Since the inline extent code can be largely self-sufficient, factor > it out from check_file_extent. No functional changes. > > Signed-off-by: Nikolay Borisov Reviewed-by: Qu Wenruo Indeed makes more sense to refactor inline check out. Thanks,

[PATCH v2] Btrfs: remove level==0 check in balance_level

2018-09-13 Thread Liu Bo
btrfs_search_slot() if (level != 0) setup_nodes_for_search() balance_level() It is just impossible to have level=0 in balance_level. Reviewed-by: Nikolay Borisov Signed-off-by: Liu Bo --- v2: add assertion for level just in case someone breaks the rule in the future.

[PATCH v2] Btrfs: assert page dirty bit

2018-09-13 Thread Liu Bo
Just in case that someone breaks the rule that pages are dirty as long as eb is dirty. Signed-off-by: Liu Bo --- v2: fix typo of CONFIG_BTRFS_DEBUG fs/btrfs/extent_io.c | 5 + 1 file changed, 5 insertions(+) diff --git a/fs/btrfs/extent_io.c b/fs/btrfs/extent_io.c index

[PATCH v2] Btrfs: skip set_page_dirty if eb is dirty

2018-09-13 Thread Liu Bo
As long as @eb is marked with EXTENT_BUFFER_DIRTY, all of its pages are dirty, so no need to set pages dirty again. Ftrace showed that the loop took 10us on my dev box, so removing this can save us at least 10us if eb is already dirty. Signed-off-by: Liu Bo --- v2: Update the patch's

Re: [PATCH] Btrfs: skip set_page_dirty if eb is dirty

2018-09-13 Thread Liu Bo
On Thu, Sep 13, 2018 at 01:29:59PM +0200, David Sterba wrote: > On Wed, Sep 12, 2018 at 12:27:46PM -0700, Liu Bo wrote: > > On Wed, Sep 12, 2018 at 09:37:20AM +0300, Nikolay Borisov wrote: > > > > > > > > > On 12.09.2018 01:06, Liu Bo wrote: > > > > As long as @eb is marked with

Re: btrfs convert problem

2018-09-13 Thread Nikolay Borisov
On 13.09.2018 19:15, Serhat Sevki Dincer wrote: >> -1 seems to be EPERM, is your device write-protected, readonly or >> something like that ? > > I am able to use ext4 partition read/write, created a file and wrote > in it, un/re mounted it, all is ok. > The drive only has a microusb port and

Re: btrfs send hangs after partial transfer and blocks all IO

2018-09-13 Thread Chris Murphy
(resend to all) On Thu, Sep 13, 2018 at 9:44 AM, Nikolay Borisov wrote: > > > On 13.09.2018 18:30, Chris Murphy wrote: >> This is the 2nd or 3rd thread containing hanging btrfs send, with >> kernel 4.18.x. The subject of one is "btrfs send hung in pipe_wait" >> and the other I can't find at the

Re: btrfs convert problem

2018-09-13 Thread Serhat Sevki Dincer
> -1 seems to be EPERM, is your device write-protected, readonly or > something like that ? I am able to use ext4 partition read/write, created a file and wrote in it, un/re mounted it, all is ok. The drive only has a microusb port and a little led light; no rw switch or anything like that..

Re: btrfs convert problem

2018-09-13 Thread Nikolay Borisov
On 13.09.2018 15:22, Serhat Sevki Dincer wrote: > Hi, > > I have an external usb HDD (WD my passport, just usb cable, no > external power) with a single ext4 partition occupying the whole disk > with 698 GiB capacity and 188 GiB empty space. The data on disk is not > very important. > > I

Re: btrfs send hangs after partial transfer and blocks all IO

2018-09-13 Thread Nikolay Borisov
On 13.09.2018 18:30, Chris Murphy wrote: > This is the 2nd or 3rd thread containing hanging btrfs send, with > kernel 4.18.x. The subject of one is "btrfs send hung in pipe_wait" > and the other I can't find at the moment. In that case though the hang > is reproducible in 4.14.x and weirdly it

Re: btrfs send hangs after partial transfer and blocks all IO

2018-09-13 Thread Chris Murphy
This is the 2nd or 3rd thread containing hanging btrfs send, with kernel 4.18.x. The subject of one is "btrfs send hung in pipe_wait" and the other I can't find at the moment. In that case though the hang is reproducible in 4.14.x and weirdly it only happens when a snapshot contains (perhaps many)

Re: [PATCH V5] Btrfs: enchanse raid1/10 balance heuristic

2018-09-13 Thread Timofey Titovets
сб, 7 июл. 2018 г. в 18:24, Timofey Titovets : > > From: Timofey Titovets > > Currently btrfs raid1/10 balancer bаlance requests to mirrors, > based on pid % num of mirrors. > > Make logic understood: > - if one of underline devices are non rotational > - Queue leght to underline devices > > By

Re: btrfs send hangs after partial transfer and blocks all IO

2018-09-13 Thread Nikolay Borisov
On 13.09.2018 15:30, Jürgen Herrmann wrote: > OK, I will install kdump later and perform a dump after the hang. > > One more noob question beforehand: does this dump contain sensitive > information, for example the luks encryption key for the disk etc? A > Google search only brings up one

Re: btrfs send hangs after partial transfer and blocks all IO

2018-09-13 Thread Jürgen Herrmann
OK, I will install kdump later and perform a dump after the hang. One more noob question beforehand: does this dump contain sensitive information, for example the luks encryption key for the disk etc? A Google search only brings up one relevant search result which can only be viewed with a

Re: [PATCH 0/3] Misc refactoring of check_file_extent

2018-09-13 Thread Lu Fengqi
On Thu, Sep 13, 2018 at 03:05:04PM +0300, Nikolay Borisov wrote: >While looking at check_file_extent I thought that the code might be a bit >cleaner than it actually is and cleaner as well. The first patch factors out >the code dealing with inline extents into a separate function aptly named

btrfs convert problem

2018-09-13 Thread Serhat Sevki Dincer
Hi, I have an external usb HDD (WD my passport, just usb cable, no external power) with a single ext4 partition occupying the whole disk with 698 GiB capacity and 188 GiB empty space. The data on disk is not very important. I also have a laptop with Manjaro 64-bit XFCE, kernel 4.14.68,

[PATCH 2/3] btrfs-progs: check: lowmem: Refactor extent len test in check_file_extent_inline

2018-09-13 Thread Nikolay Borisov
Instead of having another top-level if which checks for 'extent_num_bytes != item_inline_len' only if we are !compressed, just move the 'if' inside the 'else' branch of the first top-level if, since it has already checked for !compressed or not. No functional changes. Signed-off-by: Nikolay

[PATCH 3/3] btrfs-progs: check: lowmem: Refactor extent type checks in check_file_extent

2018-09-13 Thread Nikolay Borisov
Make the checks in check_file_extent a bit more explicit. First we check for unknown type and fail accordingly. Then we check for inline extent and handle it in the newly introduced check_file_extent_inline. Finally if none of the above checks triggered then we must have a regular or prealloc

[PATCH 1/3] btrfs-progs: check: lowmem: Factor out inline extent checking code in its own function

2018-09-13 Thread Nikolay Borisov
Since the inline extent code can be largely self-sufficient, factor it out from check_file_extent. No functional changes. Signed-off-by: Nikolay Borisov --- check/mode-lowmem.c | 142 ++-- 1 file changed, 83 insertions(+), 59 deletions(-) diff --git

[PATCH 0/3] Misc refactoring of check_file_extent

2018-09-13 Thread Nikolay Borisov
While looking at check_file_extent I thought that the code might be a bit cleaner than it actually is and cleaner as well. The first patch factors out the code dealing with inline extents into a separate function aptly named check_file_extent_inline. This allows to remove some inline-specific

Re: btrfs send hangs after partial transfer and blocks all IO

2018-09-13 Thread Nikolay Borisov
On 13.09.2018 14:50, Jürgen Herrmann wrote: > I was echoing "w" to /proc/sysrq_trigger every 0.5s which did work also > after the hang because I started the loop before the hang. The dmesg > output should show the hanging tasks from second 346 on or so. Still not > useful? > So from 346 it's

Re: [PATCH] btrfs: wait on caching when putting the bg cache

2018-09-13 Thread David Sterba
On Wed, Sep 12, 2018 at 08:40:44PM +0200, David Sterba wrote: > On Wed, Sep 12, 2018 at 10:45:45AM -0400, Josef Bacik wrote: > > While testing my backport I noticed there was a panic if I ran > > generic/416 generic/417 generic/418 all in a row. This just happened to > > uncover a race where we

Re: btrfs send hangs after partial transfer and blocks all IO

2018-09-13 Thread Jürgen Herrmann
I was echoing "w" to /proc/sysrq_trigger every 0.5s which did work also after the hang because I started the loop before the hang. The dmesg output should show the hanging tasks from second 346 on or so. Still not useful? Best regards, Jürgen Am 13. September 2018 13:04:39 schrieb Nikolay

Re: [PATCH] Btrfs: skip set_page_dirty if eb is dirty

2018-09-13 Thread David Sterba
On Wed, Sep 12, 2018 at 12:27:46PM -0700, Liu Bo wrote: > On Wed, Sep 12, 2018 at 09:37:20AM +0300, Nikolay Borisov wrote: > > > > > > On 12.09.2018 01:06, Liu Bo wrote: > > > As long as @eb is marked with EXTENT_BUFFER_DIRTY, all of its pages > > > are dirty, so no need to set pages dirty

Re: btrfs send hangs after partial transfer and blocks all IO

2018-09-13 Thread Nikolay Borisov
On 13.09.2018 13:56, Jürgen Herrmann wrote: > Both loops were started before the hang because after the hang I cannot > do that anymore. That's why there is progress in the logs at first. The > hang continues for at least 1.5 hours. No data is transferred anymore > during this time. I never

Re: btrfs send hangs after partial transfer and blocks all IO

2018-09-13 Thread Jürgen Herrmann
Both loops were started before the hang because after the hang I cannot do that anymore. That's why there is progress in the logs at first. The hang continues for at least 1.5 hours. No data is transferred anymore during this time. I never waited longer than 1.5 hours. Best regards, Jürgen

Re: btrfs send hangs after partial transfer and blocks all IO

2018-09-13 Thread Nikolay Borisov
On 13.09.2018 13:29, Jürgen Herrmann wrote: > Am 13.9.2018 10:40, schrieb Nikolay Borisov: >> On 13.09.2018 11:34, Jürgen Herrmann wrote: >>> Hello! >>> >>> I have a newly installed laptop running a freshly installed (abt. two >>> months ago) laptop running latest linux mint 19. Root filesystem

Re: btrfs send hangs after partial transfer and blocks all IO

2018-09-13 Thread Jürgen Herrmann
Am 13.9.2018 10:40, schrieb Nikolay Borisov: On 13.09.2018 11:34, Jürgen Herrmann wrote: Hello! I have a newly installed laptop running a freshly installed (abt. two months ago) laptop running latest linux mint 19. Root filesystem is on a 1TB Samsung 860 M.2 SSD with btrfs on top of a LUKS

Re: [PATCH] btrfs: Handle error of get_old_root

2018-09-13 Thread David Sterba
On Thu, Sep 13, 2018 at 11:35:10AM +0300, Nikolay Borisov wrote: > In btrfs_search_old_slot get_old_root is always used with the > assumption it cannot fail. However, this is not true in rare > circumstance it can fail and return null. Currently this will not happen, as the code returning NULL

Re: [PATCH] btrfs-progs: lowmem: fix false alert about the existence of gaps in the check_file_extent

2018-09-13 Thread Lu Fengqi
On Thu, Sep 13, 2018 at 12:12:27PM +0300, Nikolay Borisov wrote: > > >On 13.09.2018 11:20, Lu Fengqi wrote: >> In the check_inode_item function, the extent_end variable used to store the >> end of the last file extent that has checked. When it passes to >> check_file_extent, if the offset of the

Re: [PATCH] btrfs-progs: Eliminate remaining uses of strerror(errno)

2018-09-13 Thread David Sterba
On Thu, Sep 13, 2018 at 09:17:17AM +0300, Nikolay Borisov wrote: > > > On 13.09.2018 03:52, Rosen Penev wrote: > > %m allows a smaller filesize. Useful on embedded systems. > > > > Signed-off-by: Rosen Penev > > --- > > build/Documentation/Makefile | 142 + > > build/Makefile.inc

Re: [PATCH] btrfs-progs: lowmem: fix false alert about the existence of gaps in the check_file_extent

2018-09-13 Thread Nikolay Borisov
On 13.09.2018 11:20, Lu Fengqi wrote: > In the check_inode_item function, the extent_end variable used to store the > end of the last file extent that has checked. When it passes to > check_file_extent, if the offset of the next file extent is not equal to > it, there is a gap between the two

Re: [PATCH] btrfs: Handle error of get_old_root

2018-09-13 Thread Lu Fengqi
On Thu, Sep 13, 2018 at 11:35:10AM +0300, Nikolay Borisov wrote: >In btrfs_search_old_slot get_old_root is always used with the >assumption it cannot fail. However, this is not true in rare >circumstance it can fail and return null. This will lead to null >point dereference when the header is

Re: [PATCH] btrfs: Remove logically dead code from btrfs_orphan_cleanup

2018-09-13 Thread Lu Fengqi
On Thu, Sep 13, 2018 at 11:35:00AM +0300, Nikolay Borisov wrote: >In btrfs_orphan_cleanup the final 'if (ret) goto out' cannot ever be >executed. This is due to the last assignment to 'ret' depending on >the return value of btrfs_iget. If an error other than -ENOENT is >returned then the loop is

Re: btrfs send hangs after partial transfer and blocks all IO

2018-09-13 Thread Nikolay Borisov
On 13.09.2018 11:34, Jürgen Herrmann wrote: > Hello! > > I have a newly installed laptop running a freshly installed (abt. two > months ago) laptop running latest linux mint 19. Root filesystem is on a > 1TB Samsung 860 M.2 SSD with btrfs on top of a LUKS encrypted 900G > partition.

[PATCH] btrfs: Handle error of get_old_root

2018-09-13 Thread Nikolay Borisov
In btrfs_search_old_slot get_old_root is always used with the assumption it cannot fail. However, this is not true in rare circumstance it can fail and return null. This will lead to null point dereference when the header is read. Fix this by checking the return value and properly handling NULL by

[PATCH] btrfs: Remove logically dead code from btrfs_orphan_cleanup

2018-09-13 Thread Nikolay Borisov
In btrfs_orphan_cleanup the final 'if (ret) goto out' cannot ever be executed. This is due to the last assignment to 'ret' depending on the return value of btrfs_iget. If an error other than -ENOENT is returned then the loop is prematurely terminated by 'goto out'. On the other hand, if the error

btrfs send hangs after partial transfer and blocks all IO

2018-09-13 Thread Jürgen Herrmann
Hello! I have a newly installed laptop running a freshly installed (abt. two months ago) laptop running latest linux mint 19. Root filesystem is on a 1TB Samsung 860 M.2 SSD with btrfs on top of a LUKS encrypted 900G partition. Timeshift-btrfs is enabled for root (@) and home (@home)

Re: [PATCH] btrfs-progs: calibrate extent_end when found a gap

2018-09-13 Thread Lu Fengqi
On Thu, Sep 13, 2018 at 04:30:28PM +0800, Lu Fengqi wrote: >On Tue, Sep 11, 2018 at 04:41:21PM +0200, David Sterba wrote: >>On Tue, Sep 04, 2018 at 08:42:01PM +0800, Lu Fengqi wrote: >>> The extent_end will be used to check whether there is gap between this >>> extent and next extent. If it is not

Re: [PATCH] btrfs-progs: calibrate extent_end when found a gap

2018-09-13 Thread Lu Fengqi
On Tue, Sep 11, 2018 at 04:41:21PM +0200, David Sterba wrote: >On Tue, Sep 04, 2018 at 08:42:01PM +0800, Lu Fengqi wrote: >> The extent_end will be used to check whether there is gap between this >> extent and next extent. If it is not calibrated, check_file_extent will > >Do you mean

[PATCH] btrfs-progs: lowmem: fix false alert about the existence of gaps in the check_file_extent

2018-09-13 Thread Lu Fengqi
In the check_inode_item function, the extent_end variable used to store the end of the last file extent that has checked. When it passes to check_file_extent, if the offset of the next file extent is not equal to it, there is a gap between the two file extents. In the case of a gap existing, it

Re: [PATCH] btrfs-progs: Eliminate remaining uses of strerror(errno)

2018-09-13 Thread Nikolay Borisov
On 13.09.2018 03:52, Rosen Penev wrote: > %m allows a smaller filesize. Useful on embedded systems. > > Signed-off-by: Rosen Penev > --- > build/Documentation/Makefile | 142 + > build/Makefile.inc | 42 ++ > build/config.h | 139 + > build/config.log