A cmd stops pio submission and waits for the inflight pios before it rewrites the BAT levels or deltas[], and expects nothing to touch them until it resumes. Konstantin found three ways this does not hold while reviewing the VSTOR-151979 series (VSTOR-152985, VSTOR-152984, VSTOR-152981):
- a submitter racing the stop either runs without the cmd waiting for it, or is parked after resume and never completes (patch 1); - the ploop_allocator() thread is not stopped at all. It truncates the top delta while merge, notify_merged or flip moves it in deltas[], cutting the live image (patch 3). The thread can also miss a wakeup, which stalls the device by itself and would hang the new wait, so patch 2 fixes that first; - a grow failing to copy a cluster resumes submission with that cluster's BAT page still busy, and writes to it hang (patch 4). Patches 1 and 3 both lose data: a pio slipping past the stop can read and write through a delta that merge has already released, and the allocator cuts the live image. Patch 3 is the easiest to hit, a snapshot delete under write load is enough. Tested on dev-cl04, stock 6.12.0-211.52.1.22.11.vz10, dm-ploop built out of tree: - each bug reproduced on the stock module, the races with a test-only kprobe module widening the window: a pio running past the suspend of a cmd; a pio never completing after resume; the top image truncated from 512M to 128M by a merge and to 256M by a flip; after a grow failing with ENOSPC on a full host fs, a writer stuck in D state, an unsuspendable device and a later cmd failing with ETIMEDOUT. None of them reproduce with the series; - set_noresume, suspend, a grow to 2T, which relocates 8 clusters and restarts the allocator, and flip_upper_deltas with the allocator held: v1 cut the old top image from 640M to 256M, v2 waits for the allocator and the image keeps its data. On a stock kernel the flip oopses first: the check in ploop_check_delta_before_flip() walks past the md pages of the lower delta once the BAT has grown. The test ran with a local fix for that, which will be sent separately; - fio 4k random reads and writes, O_DIRECT, 1 to 48 jobs, on a fully allocated image over brd: IOPS and system CPU stay within the run-to-run spread of the stock module. This setup tops out at about 420K IOPS elsewhere in dm-ploop and would not show a lock taken on submission either, so patch 1 keeps that path lockless; - the ploop pytest suite, ploop and qcow2 formats, passes. A test for patch 4 goes to the ploop tests separately. Changes in v2, after Konstantin's review: - 1/4: notify_merged named among the cmds hit by the race; - 2/4: the stall does not end by itself, and the reorder fixes a lost kthread_stop() wakeup as well; - 3/4: also wait for the allocator in ploop_flip_upper_deltas(), as a resize of a dm suspended device restarts it after postsuspend; the message names notify_merged, which moves the top delta in both directions; - 4/4: unchanged; - rebased on rh10-6.12.0-211.52.1.22.17.vz10, Reviewed-by tags added to 1/4, 2/4 and 4/4. v1: http://lore.virtuozzo.com/kernel/[email protected]/ Denis V. Lunev (4): dm-ploop: fix pio submission racing with the suspend and resume of a cmd dm-ploop: fix a lost wakeup of the ploop_allocator() thread dm-ploop: wait for the ploop_allocator() thread before changing deltas[] dm-ploop: fix a failed cluster copy on grow leaving the BAT page busy drivers/md/dm-ploop-cmd.c | 24 ++++++++++++++++++++++-- drivers/md/dm-ploop-map.c | 22 +++++++++++++++------- drivers/md/dm-ploop-target.c | 4 ++++ drivers/md/dm-ploop.h | 1 + 4 files changed, 42 insertions(+), 9 deletions(-) base-commit: 95ab11e2c57e8468412ca586d4be8cd75b6bfbf0 -- 2.53.0 _______________________________________________ Devel mailing list [email protected] https://lists.openvz.org/mailman/listinfo/devel
