Re: [f2fs-dev] [PATCH v10 00/62] Convert page cache to XArray
On 04/14/2018 02:58 PM, Matthew Wilcox wrote: > On Sat, Apr 14, 2018 at 12:50:30PM -0700, Matthew Wilcox wrote: >> On Mon, Apr 09, 2018 at 04:18:07PM -0500, Goldwyn Rodrigues wrote: >> >> I'm sorry I missed this email. My inbox is a disaster :( >> >>> I tried these patches against next-20180329 and added the patch for the >>> bug reported by Mike Kravetz. I am getting the following BUG on ext4 and >>> xfs, running generic/048 tests of fstests. Each trace is from a >>> different instance/run. >> >> Yikes. I haven't been able to reproduce this. Maybe it's a matter of >> filesystem or some other quirk. >> >> It seems easy for you to reproduce it, so would you mind bisecting it? >> Should be fairly straightforward; I'd start at commit "xarray: Add >> MAINTAINERS entry", since the page cache shouldn't be affected by anything >> up to that point, then bisect forwards from there. >> >>> BTW, for my convenience, do you have these patches in a public git tree? >> >> I didn't publish it; it's hard to push out a tree based on linux-next. >> I'll try to make that happen. > > Figured it out: > > http://git.infradead.org/users/willy/linux-dax.git/shortlog/refs/heads/xarray-20180413 > > aka > git://git.infradead.org/users/willy/linux-dax.git xarray-20180413 Thanks. I found the erroneous commit is e14a33134244 mm: Convert workingset to XArray mapping->nrexceptional is becoming negative. An easy way to reproduce is to perform a large enough I/O to force it to swap out and inodes are evicted. -- Goldwyn -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Linux-f2fs-devel mailing list Linux-f2fs-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
Re: [f2fs-dev] [PATCH v10 00/62] Convert page cache to XArray
On Sat, Apr 14, 2018 at 12:50:30PM -0700, Matthew Wilcox wrote: > On Mon, Apr 09, 2018 at 04:18:07PM -0500, Goldwyn Rodrigues wrote: > > I'm sorry I missed this email. My inbox is a disaster :( > > > I tried these patches against next-20180329 and added the patch for the > > bug reported by Mike Kravetz. I am getting the following BUG on ext4 and > > xfs, running generic/048 tests of fstests. Each trace is from a > > different instance/run. > > Yikes. I haven't been able to reproduce this. Maybe it's a matter of > filesystem or some other quirk. > > It seems easy for you to reproduce it, so would you mind bisecting it? > Should be fairly straightforward; I'd start at commit "xarray: Add > MAINTAINERS entry", since the page cache shouldn't be affected by anything > up to that point, then bisect forwards from there. > > > BTW, for my convenience, do you have these patches in a public git tree? > > I didn't publish it; it's hard to push out a tree based on linux-next. > I'll try to make that happen. Figured it out: http://git.infradead.org/users/willy/linux-dax.git/shortlog/refs/heads/xarray-20180413 aka git://git.infradead.org/users/willy/linux-dax.git xarray-20180413 -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Linux-f2fs-devel mailing list Linux-f2fs-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
Re: [f2fs-dev] [PATCH v10 00/62] Convert page cache to XArray
On Mon, Apr 09, 2018 at 04:18:07PM -0500, Goldwyn Rodrigues wrote: I'm sorry I missed this email. My inbox is a disaster :( > I tried these patches against next-20180329 and added the patch for the > bug reported by Mike Kravetz. I am getting the following BUG on ext4 and > xfs, running generic/048 tests of fstests. Each trace is from a > different instance/run. Yikes. I haven't been able to reproduce this. Maybe it's a matter of filesystem or some other quirk. It seems easy for you to reproduce it, so would you mind bisecting it? Should be fairly straightforward; I'd start at commit "xarray: Add MAINTAINERS entry", since the page cache shouldn't be affected by anything up to that point, then bisect forwards from there. > BTW, for my convenience, do you have these patches in a public git tree? I didn't publish it; it's hard to push out a tree based on linux-next. I'll try to make that happen. -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Linux-f2fs-devel mailing list Linux-f2fs-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
[f2fs-dev] [PATCH v10 00/62] Convert page cache to XArray
Hi Matthew, On 03/29/2018 10:41 PM, Matthew Wilcox wrote: > From: Matthew Wilcox > > I'd like to thank Andrew for taking the first eight XArray patches > into -next. He's understandably nervous about taking the rest of the > patches into -next given how few of the remaining patches have review > tags on them. So ... if you're on the cc, I'd really appreciate a review > on something that you feel somewhat responsible for, eg the particular > filesystem (nilfs, f2fs, lustre) that I've touched, or something in the > mm/ or fs/ directories that you've worked on recently. > > This is against next-20180329. I tried these patches against next-20180329 and added the patch for the bug reported by Mike Kravetz. I am getting the following BUG on ext4 and xfs, running generic/048 tests of fstests. Each trace is from a different instance/run. BTW, for my convenience, do you have these patches in a public git tree? [ 222.007071] BUG: Bad page state in process kswapd0 pfn:132f25 [ 222.007108] page:d6f144cbc940 count:0 mapcount:0 mapping:94b2735e3918 index:0x1 [ 222.007140] flags: 0x4000() [ 222.007157] raw: 4000 94b2735e3918 0001 [ 222.007186] raw: dead0100 dead0200 [ 222.007216] page dumped because: non-NULL mapping [ 222.007288] CPU: 0 PID: 55 Comm: kswapd0 Tainted: GE 4.16.0-rc7-next-20180329-xarray #2 [ 222.007289] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.0.0-prebuilt.qemu-project.org 04/01/2014 [ 222.007290] Call Trace: [ 222.007297] dump_stack+0x63/0x85 [ 222.007300] bad_page+0xd5/0x140 [ 222.007302] free_pages_check_bad+0x5f/0x70 [ 222.007304] free_pcppages_bulk+0x423/0x5c0 [ 222.007308] ? xas_load+0x3d/0xc0 [ 222.007310] free_unref_page_commit+0xad/0xd0 [ 222.007312] free_unref_page_list+0x101/0x190 [ 222.007315] release_pages+0x17c/0x3f0 [ 222.007317] __pagevec_release+0x2f/0x40 [ 222.007319] invalidate_mapping_pages+0x2d8/0x310 [ 222.007323] ? memcg_drain_all_list_lrus+0x120/0x120 [ 222.007326] inode_lru_isolate+0x131/0x180 [ 222.007328] __list_lru_walk_one.isra.7+0x92/0x150 [ 222.007329] ? iput+0x220/0x220 [ 222.007331] list_lru_walk_one+0x23/0x30 [ 222.007332] prune_icache_sb+0x40/0x60 [ 222.007334] super_cache_scan+0x137/0x1b0 [ 222.007336] shrink_slab.part.53+0x1ae/0x3a0 [ 222.007338] shrink_slab+0x35/0x40 [ 222.007340] shrink_node+0x158/0x490 [ 222.007342] balance_pgdat+0x149/0x320 [ 222.007344] kswapd+0x15f/0x400 [ 222.007347] ? wait_woken+0x80/0x80 [ 222.007350] kthread+0x121/0x140 [ 222.007352] ? balance_pgdat+0x320/0x320 [ 222.007353] ? kthread_create_worker_on_cpu+0x50/0x50 [ 222.007356] ret_from_fork+0x35/0x40 [ 222.007357] Disabling lock debugging due to kernel taint 17252.906122] [ cut here ] [17252.906124] kernel BUG at fs/inode.c:512! [17252.906150] invalid opcode: [#1] SMP PTI [17252.906467] CPU: 2 PID: 31588 Comm: umount Tainted: GE 4.16.0-rc7-next-20180329-xarray #2 [17252.906492] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.0.0-prebuilt.qemu-project.org 04/01/2014 [17252.906523] RIP: 0010:clear_inode+0x8a/0xa0 [17252.906536] RSP: 0018:ba2302213d28 EFLAGS: 00010086 [17252.906552] RAX: RBX: 8f1efb3976d8 RCX: [17252.906571] RDX: 0001 RSI: RDI: 8f1efb397858 [17252.906590] RBP: ba2302213d38 R08: R09: [17252.906609] R10: ba2302213ae8 R11: ba2302213ae8 R12: 8f1efb397858 [17252.906628] R13: c067e580 R14: 8f1dcc4281e8 R15: 8f1ef9fddc68 [17252.906648] FS: 7f6b9eae0fc0() GS:8f1effd0() knlGS: [17252.906670] CS: 0010 DS: ES: CR0: 80050033 [17252.906686] CR2: 55927e5f2118 CR3: ab29a000 CR4: 06e0 [17252.906708] DR0: DR1: DR2: [17252.906728] DR3: DR6: fffe0ff0 DR7: 0400 [17252.906747] Call Trace: [17252.906786] ext4_clear_inode+0x1a/0x80 [ext4] [17252.906808] ext4_evict_inode+0x54/0x590 [ext4] [17252.906823] evict+0xca/0x1a0 [17252.906833] dispose_list+0x39/0x50 [17252.906844] evict_inodes+0x158/0x170 [17252.906857] generic_shutdown_super+0x44/0x120 [17252.906871] kill_block_super+0x27/0x50 [17252.906883] deactivate_locked_super+0x48/0x80 [17252.906897] deactivate_super+0x40/0x60 [17252.906910] cleanup_mnt+0x3f/0x80 [17252.906921] __cleanup_mnt+0x12/0x20 [17252.906933] task_work_run+0x9d/0xc0 [17252.907593] exit_to_usermode_loop+0xa5/0xb0 [17252.908237] do_syscall_64+0x14a/0x1e0 [17252.908884] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [17252.909522] RIP: 0033:0x7f6b9e3b2a57 [17252.910154] RSP: 002b:71d2e6f8 EFLAGS: 0246 ORIG_RAX: 00a6 [17252.910810] RAX: RBX: 55927e5e
Re: [f2fs-dev] [PATCH v10 00/62] Convert page cache to XArray
On 04/04/2018 08:52 PM, Matthew Wilcox wrote:
> On Wed, Apr 04, 2018 at 09:35:46AM -0700, Mike Kravetz wrote:
>> Running with this XArray series on top of next-20180329 consistently 'hangs'
>> on shutdown looping (?forever?) in tag_pages_for_writeback/xas_for_each_tag.
>> All I have to do is make sure there is some activity on the ext4 fs before
>> shutdown. Not sure if this is a 'next-20180329' issue or XArray issue.
>> But the fact that we are looping in xas_for_each_tag looks suspicious.
>
> Thanks for your help debugging this! Particularly collecting the xa_dump.
> I got bit by the undefined behaviour of shifting by BITS_PER_LONG,
> but of course it was subtle.
>
> The userspace testing framework wasn't catching this for a couple of
> reasons; I'll work on making sure it catches this kind of thing in
> the future.
>
> I'll fold this in and post a v11 later this week or early next week.
Nice! Added patch and it solves the hangs I observed.
--
Mike Kravetz
>
> diff --git a/include/linux/xarray.h b/include/linux/xarray.h
> index eac04922eba2..f5b7e507a86f 100644
> --- a/include/linux/xarray.h
> +++ b/include/linux/xarray.h
> @@ -904,9 +929,12 @@ static inline unsigned int xas_find_chunk(struct
> xa_state *xas, bool advance,
> if (advance)
> offset++;
> if (XA_CHUNK_SIZE == BITS_PER_LONG) {
> - unsigned long data = *addr & (~0UL << offset);
> - if (data)
> - return __ffs(data);
> + if (offset < XA_CHUNK_SIZE) {
> + unsigned long data = *addr & (~0UL << offset);
> +
> + if (data)
> + return __ffs(data);
> + }
> return XA_CHUNK_SIZE;
> }
>
>
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
Re: [f2fs-dev] [PATCH v10 00/62] Convert page cache to XArray
On Wed, Apr 04, 2018 at 09:35:46AM -0700, Mike Kravetz wrote:
> Running with this XArray series on top of next-20180329 consistently 'hangs'
> on shutdown looping (?forever?) in tag_pages_for_writeback/xas_for_each_tag.
> All I have to do is make sure there is some activity on the ext4 fs before
> shutdown. Not sure if this is a 'next-20180329' issue or XArray issue.
> But the fact that we are looping in xas_for_each_tag looks suspicious.
Thanks for your help debugging this! Particularly collecting the xa_dump.
I got bit by the undefined behaviour of shifting by BITS_PER_LONG,
but of course it was subtle.
The userspace testing framework wasn't catching this for a couple of
reasons; I'll work on making sure it catches this kind of thing in
the future.
I'll fold this in and post a v11 later this week or early next week.
diff --git a/include/linux/xarray.h b/include/linux/xarray.h
index eac04922eba2..f5b7e507a86f 100644
--- a/include/linux/xarray.h
+++ b/include/linux/xarray.h
@@ -904,9 +929,12 @@ static inline unsigned int xas_find_chunk(struct xa_state
*xas, bool advance,
if (advance)
offset++;
if (XA_CHUNK_SIZE == BITS_PER_LONG) {
- unsigned long data = *addr & (~0UL << offset);
- if (data)
- return __ffs(data);
+ if (offset < XA_CHUNK_SIZE) {
+ unsigned long data = *addr & (~0UL << offset);
+
+ if (data)
+ return __ffs(data);
+ }
return XA_CHUNK_SIZE;
}
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
Re: [f2fs-dev] [PATCH v10 00/62] Convert page cache to XArray
On 03/29/2018 08:41 PM, Matthew Wilcox wrote: > From: Matthew Wilcox > > I'd like to thank Andrew for taking the first eight XArray patches > into -next. He's understandably nervous about taking the rest of the > patches into -next given how few of the remaining patches have review > tags on them. So ... if you're on the cc, I'd really appreciate a review > on something that you feel somewhat responsible for, eg the particular > filesystem (nilfs, f2fs, lustre) that I've touched, or something in the > mm/ or fs/ directories that you've worked on recently. > > This is against next-20180329. > I applied this series to next-20180329 and booted in a debug environment. My root fs is ext4, and next-20180329 had the first (bad) fix in bug https://bugzilla.kernel.org/show_bug.cgi?id=199185 so, I had to apply the revised fix. Running with this XArray series on top of next-20180329 consistently 'hangs' on shutdown looping (?forever?) in tag_pages_for_writeback/xas_for_each_tag. All I have to do is make sure there is some activity on the ext4 fs before shutdown. Not sure if this is a 'next-20180329' issue or XArray issue. But the fact that we are looping in xas_for_each_tag looks suspicious. #0 xas_find_chunk (tag=, advance=, xas=) at ./include/linux/xarray.h:886 #1 xas_next_tag (tag=, max=, xas=) at ./include/linux/xarray.h:915 #2 tag_pages_for_writeback (mapping=, start=, end=2251799813685247) at mm/page-writeback.c:2109 #3 0x812eccf0 in ext4_writepages (mapping=0x88012bf9b918, wbc=) at fs/ext4/inode.c:2793 #4 0x811bbe4b in do_writepages (mapping=0xc90001727a28, wbc=0x) at mm/page-writeback.c:2332 #5 0x812743bd in __writeback_single_inode (inode=0xc90001727a28, wbc=0xc90001727cc0) at fs/fs-writeback.c:1315 #6 0x81274aaf in writeback_sb_inodes (sb=0x88012e2e2e98, wb=0x88012c02e000, work=0x88012c4aae18) at fs/fs-writeback.c:1579 #7 0x81274ff7 in wb_writeback (wb=0x88012c02e000, work=0x88012c4aae18) at fs/fs-writeback.c:1755 #8 0x812757df in wb_do_writeback (wb=) at fs/fs-writeback.c:1900 #9 wb_workfn (work=0xc90001727a28) at fs/fs-writeback.c:1941 #10 0x810b7415 in process_one_work (worker=0x88012eff6d68, work=0x88012c02e190) at kernel/workqueue.c:2145 #11 0x810b762e in worker_thread (__worker=0x88012eff6d68) at kernel/workqueue.c:2279 #12 0x810bd7c3 in kthread (_create=0x88012e88fc28) at kernel/kthread.c:238 #13 0x81a00205 in ret_from_fork () at arch/x86/entry/entry_64.S:411 #14 0x in ?? () -- Mike Kravetz -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Linux-f2fs-devel mailing list Linux-f2fs-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
[f2fs-dev] [PATCH v10 00/62] Convert page cache to XArray
From: Matthew Wilcox I'd like to thank Andrew for taking the first eight XArray patches into -next. He's understandably nervous about taking the rest of the patches into -next given how few of the remaining patches have review tags on them. So ... if you're on the cc, I'd really appreciate a review on something that you feel somewhat responsible for, eg the particular filesystem (nilfs, f2fs, lustre) that I've touched, or something in the mm/ or fs/ directories that you've worked on recently. This is against next-20180329. Patch 1 is just cleanup of a couple of merge glitches. You can ignore this one, unless you're Andrew Morton, in which case I'd appreciate it being added to -next. Patch 2 is huge and mostly mechanical and boring. Probably best to ignore it. Patches 3-16 are the XArray implementation. You don't have to read through them unless you really want to. If you're going to look at anything, spotting where I have gaps in the test-suite would probably be a good use of time. Some of the tests aren't added until much later in the series (eg some of the xa_load tests aren't added until xa_store is added because it's a pain to use the radix tree API to do the stores and then use the XArray API to do the loads). Patches 17-45 are generic code. This is where I could really use some review. Each one is pretty small, usually only touching a single function. It should bisect perfectly to any point in this series because the radix tree & XArray are interoperable (for now ...). It should be easy to review. If not, let me know. Patch 46 is btrfs and has been reviewed by David Sterba. Yay! Patches 47-48 are more generic code. Hmm, probably should have reordered them to be with 17-37. This is what I get for working alphabetically. Patches 49-51 are individual filesystems. If you're an expert in that filesystem, please test and be sure I didn't break you. Patches 52-60 are DAX patches. I just redid the DAX patches and I'm finally happy with how this part of the series came out; I think it should be much easier to review, and I even fixed a few minor bugs. Patches 61 & 62 are the icing on the cake, just some small cleanups. I'm sure everybody wants to review patch 62 since it's just deleting unused code. Changes from v9: - Rebased on next-20180329 - Added XA_STATE_ORDER - Decided that xas_load should return non-NULL if *any* entry overlapping the specified range is non-NULL (it won't necessarily be any of the entries in the range) - Added test suite for both above items - Redid DAX XArray conversion; should be easier to review now. Matthew Wilcox (62): page cache: Use xa_lock xarray: Replace exceptional entries xarray: Change definition of sibling entries xarray: Add definition of struct xarray xarray: Define struct xa_node xarray: Add documentation xarray: Add xa_load xarray: Add xa_get_tag, xa_set_tag and xa_clear_tag xarray: Add xa_store xarray: Add xa_cmpxchg and xa_insert xarray: Add xa_for_each xarray: Add xa_extract xarray: Add xa_destroy xarray: Add xas_next and xas_prev xarray: Add xas_create_range xarray: Add MAINTAINERS entry page cache: Rearrange address_space page cache: Convert hole search to XArray page cache: Add and replace pages using the XArray page cache: Convert page deletion to XArray page cache: Convert page cache lookups to XArray page cache: Convert delete_batch to XArray page cache: Remove stray radix comment page cache: Convert filemap_range_has_page to XArray mm: Convert page-writeback to XArray mm: Convert workingset to XArray mm: Convert truncate to XArray mm: Convert add_to_swap_cache to XArray mm: Convert delete_from_swap_cache to XArray mm: Convert __do_page_cache_readahead to XArray mm: Convert page migration to XArray mm: Convert huge_memory to XArray mm: Convert collapse_shmem to XArray mm: Convert khugepaged_scan_shmem to XArray pagevec: Use xa_tag_t shmem: Convert replace to XArray shmem: Convert shmem_confirm_swap to XArray shmem: Convert find_swap_entry to XArray shmem: Convert shmem_add_to_page_cache to XArray shmem: Convert shmem_alloc_hugepage to XArray shmem: Convert shmem_free_swap to XArray shmem: Convert shmem_partial_swap_usage to XArray memfd: Convert shmem_tag_pins to XArray memfd: Convert shmem_wait_for_pins to XArray shmem: Comment fixups btrfs: Convert page cache to XArray fs: Convert buffer to XArray fs: Convert writeback to XArray nilfs2: Convert to XArray f2fs: Convert to XArray lustre: Convert to XArray dax: Fix use of zero page dax: dax_insert_mapping_entry always succeeds dax: Rename some functions dax: Hash on XArray instead of mapping dax: Convert dax_insert_pfn_mkwrite to XArray dax: Convert dax_layout_busy_page to XArray dax: Convert __dax_invalidate_entry to XArray dax: Convert dax writeback to XArray dax: Convert page fault handlers to XArray page cache: Finish XArray conversion radix t
