Public bug reported: [Impact]
Ubuntu Noble's AWS 7.0 kernel bundles OpenZFS 2.4.1-1ubuntu5. With dedup=on under concurrent duplicate-block writes, file blocks are silently replaced by holes that read as all zeros while file sizes remain unchanged. `zpool status` remains healthy and reports no read, write, or checksum errors. This was observed on an arm64 EC2 CI host running PostgreSQL on a ZFS pool. Captures included: * A 32 KiB B-tree file whose blocks 0, 1, and 3 read as all zeros while block 2 remained valid. * A 16 KiB B-tree file whose two blocks both read as all zeros. * A PostgreSQL WAL segment containing an all-zero region. * PostgreSQL primary errors: `index ... contains unexpected zero page at block 0`. * PostgreSQL standby failures: `PANIC: WAL contains references to invalid pages`. Immediately before containment, the workload produced 240 zero-page errors and five standby PANICs in 15 minutes. After `zfs set dedup=off`, there were no new client zero-page errors and no standby PANICs during the observation window, while the same workload continued and both standbys remained streaming. [Versions and configuration] * Ubuntu 24.04 Noble, arm64 * Linux 7.0.0-1012-aws (7.0.0-1012.12~24.04.1) * Loaded ZFS module: 2.4.1-1ubuntu5 * Module owner: linux-modules-7.0.0-1012-aws * zfsutils-linux: 2.2.2-0ubuntu9.5 * dedup=on before containment * compression=lz4 * recordsize=128K * sync=disabled * Single-device ephemeral pool on /dev/ram0 The previous 7.0.0-1011-aws kernel also bundles ZFS 2.4.1-1ubuntu5. A currently available Noble 6.8.0-1063-aws package was inspected and bundles ZFS 2.2.2-0ubuntu9.4, which predates the regression. [Upstream] Upstream issue: https://github.com/openzfs/zfs/issues/18366 Upstream fix: https://github.com/openzfs/zfs/commit/6fb72fda0f60d9efb591e320f83f78b19ec451cc The upstream diagnosis matches the observed failure exactly: zio_ddt_write computes have_dvas before taking dde_io_lock; a concurrent error path can zero DVA[0], after which stale have_dvas causes the zero DVA to be copied into the block pointer. ZFS interprets that pointer as a hole, returning zeros with no checksum error. The regression was introduced by a41ef36858 and the fix is included in OpenZFS 2.4.4. [Request] Please backport 6fb72fda0f60d9efb591e320f83f78b19ec451cc to Noble's zfs- linux 2.4.1 package and ensure the corrected module is vendored into the Noble linux-aws 7.0 kernel packages. [Test Case] Use the concurrent dedup copy/verification reproducer attached to upstream issue #18366: https://github.com/user-attachments/files/26250092/file_copy_check_repeat.sh Run repeated concurrent writes with dedup=on and verify that copied files retain their original checksums and contain no unexpected zero blocks. Repeat with dedup=off as a control. Run the OpenZFS dedup and zio test groups. Verify the resulting zfs.ko through the linux-aws package, not only zfs-dkms, because Ubuntu kernels vendor the module. [Where problems could occur] The patch changes lock ordering and when DDT DVA state is read in zio_ddt_write. A regression could affect deduplicated writes through incorrect DDT references, write failures, or increased lock contention. Testing should include high-concurrency duplicate writes, injected write errors, DDT pruning, pool import/export, scrub, and the standard OpenZFS regression suite. ** Affects: linux-aws (Ubuntu) Importance: Undecided Status: New ** Affects: zfs-linux (Ubuntu) Importance: Undecided Status: New ** Affects: linux-aws (Ubuntu Noble) Importance: Undecided Status: New ** Affects: zfs-linux (Ubuntu Noble) Importance: Undecided Status: New ** Tags: data-corruption linux-aws noble sru zfs -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2167150 Title: [SRU][noble] OpenZFS 2.4.1 dedup race causes silent data corruption in linux-aws 7.0 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-aws/+bug/2167150/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
