On Thu, 13 Mar 2025, Marco Felsch wrote:
> Hi,
>
> sorry for the late reply but we had to run several tests and analyze
> multiple test outputs offline via a small self-written tool.
>
> On 25-02-14, Mikulas Patocka wrote:
> > Hi
> >
> > On Tue, 11 Feb 2025, Marco Felsch wrote:
> >
> > > Hi all,
> > >
> > > as written in the subject we do see an odd dm-integrity behaviour during
> > > the journal replay step.
> > >
> > > First things first, a short introduction to our setup:
> > > - Linux v6.8
> > > - We do have a dm-integrity+dm-crypt setup to provide an authenticated
> > > encrypted ext4-based rw data partition.
> > > - The setup is done with a custom script [1] since we are making use of
> > > the kernel trusted-keys infrastructure which isn't supported by LUKS2
> > > at the moment.
> > > - The device has no power failsafe e.g. hard power-cuts can appear.
> > > Therefore we use the dm-integrity J(ournal) mode.
> > > - The storage backend is an eMMC with 512Byte block size.
> >
> > Could you retest it with an eMMC or SDCARD from a different vendor, just
> > to test if it is hardware issue?
>
> We saw the issue on different eMMC devices from different manufacturers.
>
> > > - We use the dm-integrity 4K block size option to reduce the
> > > tag/metadata overhead.
> > >
> > > From time to time we do see "AEAD ERROR"s [2] while fsck tries to repair
> > > the filesystem which of course abort the fsck run.
> > > After a while within the rescue shell
> >
> > So, when you run fsck again from the rescue shell (without deactivating
> > and activating the device), the bug goes away? What is the time interval
> > after which the bug goes away?
>
> This happened from time to time, yes but only in rare cases and I'm not
> sure if our tester did something wrong.
>
> > Could you upload somewhere the image of the eMMC storage when the bug
> > happens and send me a link to it, so that I can look what kind of
> > corruption is there?
>
> Of course I need to align it with our customer but since the data is
> encrypted you could get the dm-integrity dump.
>
> > > and a following reboot the fsck
> > > run on the same file system doesn't trigger any "AEAD ERROR".
> > >
> > > The dm-integrity table is added twice [1] since we gather the
> > > provided_data_sectors information from the first call. I know that this
> > > isn't optimal. The provided_data_sectors should be stored and not
> > > gathered again but this shouldn't bother a system with a valid
> > > dm-integrity superblock already written.
> >
> > That should work fine - there is no problem with activating the device
> > twice.
>
> Also with activating it with differnet sizes as we do? I think there is
> no problem with it too. Our script doesn't different between the initial
> setup (no superblock available) and the "normal" setup.
>
> > > To debug the issue we uncommented the "#define DEBUG_PRINT" and noticed
> > > that the replay is happening twice [2] albeit this should be a
> > > synchronous operation and once the dm resume returned successfully the
> > > replay should be done.
> >
> > The replay happens twice because you activate it twice - this should be
> > harmless because each replay replays the same data - there is "replaying
> > 364 sections, starting at 143, commit seq 2" twice in the log.
>
> Yes, but after replaying it the first time we thought the code marks the
> entries as unused and don't have to replay these entries a 2nd time. But
> as I said, there should be no issue with it.
>
> > > We also noticed that once we uncommented "#define DEBUG_PRINT" it was
> > > harder to trigger the issue.
> > >
> > > We also checked the eMMC bus to see if the CMD23 is sent correctly (with
> > > the reliable write bit set) in case of a FUA request which is the case.
> > > So now with the above knowledge we suspect the replay path to be not
> > > synchronous, the dm resume is returning too early, while not all of the
> > > writes kicked off by copy_from_journal having reached the storage.
> >
> > There is "struct journal_completion comp" in do_journal_write and
> > "wait_for_completion_io(&comp.comp);" at the end of do_journal_write -
> > that should wait until all I/O submitted by copy_from_journal finishes.
> >
> > > Maybe you guys have some pointers we could follow or have an idea of
> > > what may go wrong here.
> > >
> > > Regards,
> > > Marco
> >
> > I read through the dm-integrity source code and I didn't find anything.
> >
> > There are two dm-crypt bugs that could cause your problems - see the
> > upstream commits 8b8f8037765757861f899ed3a2bfb34525b5c065 and
> > 9fdbbdbbc92b1474a87b89f8b964892a63734492. Please, backport these commits
> > to your kernel and re-run the tests.
>
> This helped, there are also other dm-crypt fixes which looked closely
> related. Therefore we updated to 6.12.16 and ran the tests. Tests which
> could reproduce the issue quite sufficient are now passing and it really
> looks like dm-crypt was the issue :/
>
> The issue could be reproduced by the following script which was started
> right after the boot:
>
> | #!/bin/bash
> |
> | set -x
> |
> | suffix=$(hexdump /dev/urandom -n4 -e '"%u"')
> | suffix=$((suffix%220))
> | testdir=/mnt/data/test
> | testfile=$testdir/test.$suffix
> |
> | # Make testdir
> | mkdir -p $testdir
> |
> | [ -f $testfile ] && rm $testfile
> |
> | dd if=/dev/mapper/data of=/dev/null && \
> | dd if=/dev/urandom of=$testfile bs=1M oflag=sync count=1 && \
> | dd if=/dev/mapper/data of=/dev/null && \
> | reboot -f
> |
> | echo "Something failed, please check!"
>
> Our data partition was quite small (300MB) which lead the journal-size
> to hold up to ~2MB. Since we use the ext4 on top of it we used a size of
> 1MB. If reading the whole device before or after the script stops and we
> dumped the raw eMMC partition (mmcblkXpY).
>
> Afterwards we analyzed the dump by our small tool to see if we can found
> the AEAD sector within the journal, which was the case.
>
> Now we checked if this particular sector (8 sectors since we provide 4K
> IO-size to the upper layers) can be found on the data area. Which was
> the case.
>
> Lastly we checked if we can find the entry metadata journal content
> within the metadata area. Which was the case too.
>
> Both the data and metadata belong to the same "area" so we do assume
> that the journal was correctly replayed but got the wrong
> <data><metadata> pair from the dm-crypt layer.
>
> Please correct me if our conclusion is wrong, but after the kernel
> update the issues are gone for now (we do need more test samples).
It's nice to hear that the issue is fixed. Thanks for testing it.
> Thanks,
> Marco
Mikulas
