[Bug 1967814] Comment bridged from LTC Bugzilla
--- Comment From vanessa.ba...@ibm.com 2022-04-28 18:36 EDT--- (In reply to comment #98) > > @Ubuntu: I think we can and should move forward with this. The problem with > the > previous test results is, that the qemu from the PPA wasn't installed at > all, and thus we ended up just verifying that the old one is still broken. > > I read Vanessas comment like, she the issue is not observed any more with > the PPA installed: i.e. the fix works at least for the test-case that used > to trigger the problem reliably. @Vanessa: Please correct me if I'm wrong. That is correct, the fix works, no miscompares while doing the test-case I've been using to reproduce this issue (tested for a few hours, no more than 30 seconds in between error injects). root@ilzlnx3:~# /usr/bin/qemu-system-s390x --version QEMU emulator version 4.2.1 (Debian 1:4.2-3ubuntu6.22~focalppa1) Copyright (c) 2003-2019 Fabrice Bellard and the QEMU Project developers -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1967814 Title: Ubuntu 20.04.3 - ilzlnx3g1 - virtio-scsi devs on KVM guest having miscompares on disktests when there is a failed path. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1967814/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1967814] Comment bridged from LTC Bugzilla
--- Comment From pa...@de.ibm.com 2022-04-28 05:22 EDT--- (In reply to comment #97) > (In reply to comment #94) > > Thanks Vanessa for the testing on the PPA! > > > > @Halil - I'd leave the debugging of the remaining issue to you as while you > > can't reproduce it yet it still is much closer to you than it is to me :-/ > > Thanks in advance, let us know what you find. > > > > In the meantime I have prepared the SRU content and got a PR review, so once > > we are convinced it is good - or found what we need to change - we should be > > ready to continue. > > I was not testing the PPA you provided properly, I pulled it again and made > sure it was installed and running: > > root@ilzlnx3:~# /usr/bin/qemu-system-s390x --version > QEMU emulator version 4.2.1 (Debian 1:4.2-3ubuntu6.22~focalppa1) > Copyright (c) 2003-2019 Fabrice Bellard and the QEMU Project developers > > I ran the error injects for a few hours and no miscompares were encountered. > Thanks @Halil, for bringing the issue to my attention! @Ubuntu: I think we can and should move forward with this. The problem with the previous test results is, that the qemu from the PPA wasn't installed at all, and thus we ended up just verifying that the old one is still broken. I read Vanessas comment like, she the issue is not observed any more with the PPA installed: i.e. the fix works at least for the test-case that used to trigger the problem reliably. @Vanessa: Please correct me if I'm wrong. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1967814 Title: Ubuntu 20.04.3 - ilzlnx3g1 - virtio-scsi devs on KVM guest having miscompares on disktests when there is a failed path. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1967814/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1967814] Comment bridged from LTC Bugzilla
--- Comment From vanessa.ba...@ibm.com 2022-04-25 19:20 EDT--- (In reply to comment #94) > Thanks Vanessa for the testing on the PPA! > > @Halil - I'd leave the debugging of the remaining issue to you as while you > can't reproduce it yet it still is much closer to you than it is to me :-/ > Thanks in advance, let us know what you find. > > In the meantime I have prepared the SRU content and got a PR review, so once > we are convinced it is good - or found what we need to change - we should be > ready to continue. I was not testing the PPA you provided properly, I pulled it again and made sure it was installed and running: root@ilzlnx3:~# /usr/bin/qemu-system-s390x --version QEMU emulator version 4.2.1 (Debian 1:4.2-3ubuntu6.22~focalppa1) Copyright (c) 2003-2019 Fabrice Bellard and the QEMU Project developers I ran the error injects for a few hours and no miscompares were encountered. Thanks @Halil, for bringing the issue to my attention! -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1967814 Title: Ubuntu 20.04.3 - ilzlnx3g1 - virtio-scsi devs on KVM guest having miscompares on disktests when there is a failed path. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1967814/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1967814] Comment bridged from LTC Bugzilla
--- Comment From pa...@de.ibm.com 2022-04-08 05:20 EDT--- (In reply to comment #81) > (In reply to comment #80) > > Let me know if you had time to push the PPA through your testing. > > Pulled the PPA > > root@ilzlnx3:~# apt info qemu > Package: qemu > Version: 1:4.2-3ubuntu6.22~focalppa1 > > I triggered a path failure from the host and saw the faulty paths on the > guest but no data miscompares (first time I've ever seen it not having > miscompares at the first try). > Recovered and tried again, this time I did got the miscompares, I captured > the DBGINFO and sosreport, let me know if something else is needed. Maybe we are hitting a different case. Maybe not. In the past we used to observe multiple types of miscompares one of which is the all zero, and another one is wrong but still disktest data. I'm curious if we see the wrong disktest data type of miscompare with the patched qemu. Also I would like to know if the miscompare is still observable after a reboot (i.e. if you destroy and re-start the guest, and then do a manual compare with disktest on the given block (without the write phase). Also can I help you install a self-built upstream qemu so we can test with that as well? Maybe we are just missing some more patches. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1967814 Title: Ubuntu 20.04.3 - ilzlnx3g1 - virtio-scsi devs on KVM guest having miscompares on disktests when there is a failed path. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1967814/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1967814] Comment bridged from LTC Bugzilla
--- Comment From pa...@de.ibm.com 2022-04-08 05:13 EDT--- (In reply to comment #81) > (In reply to comment #80) > > Let me know if you had time to push the PPA through your testing. > > Pulled the PPA > > root@ilzlnx3:~# apt info qemu > Package: qemu > Version: 1:4.2-3ubuntu6.22~focalppa1 > > I triggered a path failure from the host and saw the faulty paths on the > guest but no data miscompares (first time I've ever seen it not having > miscompares at the first try). > Recovered and tried again, this time I did got the miscompares, I captured > the DBGINFO and sosreport, let me know if something else is needed. > > ** EXPECTED (Target: /fc/mapper/scsi_32G_d1_ilsd9840l/test, LBA: > 1243520, Offset: 5) ** > 00 00 00 00 00 12 F9 80 00 00 00 00 00 00 00 01 > > 0010 00 00 00 00 62 4F 60 1C 00 00 00 00 00 00 06 EB > bO`. > 0020 69 6C 7A 6C 6E 78 33 67 31 00 00 00 00 00 00 00 > ilzlnx3g1... > 0030 2F 66 63 2F 6D 61 70 70 65 72 2F 73 63 73 69 5F > /fc/mapper/scsi_ > 0040 33 32 47 5F 64 31 5F 69 6C 73 64 39 38 34 30 6C > 32G_d1_ilsd9840l > > ** ACTUAL (Target: /fc/mapper/scsi_32G_d1_ilsd9840l/test, LBA: > 1243520, Offset: 5) ** > 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > > 0010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > > 0020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > > 0030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > That is very bad news! :( I was not able to trigger the problem with the patched qemu. I will try harder, but if I can't trigger the problem it becomes very difficult for me to work on it. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1967814 Title: Ubuntu 20.04.3 - ilzlnx3g1 - virtio-scsi devs on KVM guest having miscompares on disktests when there is a failed path. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1967814/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1967814] Comment bridged from LTC Bugzilla
--- Comment From vanessa.ba...@ibm.com 2022-04-07 18:42 EDT--- (In reply to comment #80) > Let me know if you had time to push the PPA through your testing. Pulled the PPA root@ilzlnx3:~# apt info qemu Package: qemu Version: 1:4.2-3ubuntu6.22~focalppa1 I triggered a path failure from the host and saw the faulty paths on the guest but no data miscompares (first time I've ever seen it not having miscompares at the first try). Recovered and tried again, this time I did got the miscompares, I captured the DBGINFO and sosreport, let me know if something else is needed. ** EXPECTED (Target: /fc/mapper/scsi_32G_d1_ilsd9840l/test, LBA: 1243520, Offset: 5) ** 00 00 00 00 00 12 F9 80 00 00 00 00 00 00 00 01 0010 00 00 00 00 62 4F 60 1C 00 00 00 00 00 00 06 EB bO`. 0020 69 6C 7A 6C 6E 78 33 67 31 00 00 00 00 00 00 00 ilzlnx3g1... 0030 2F 66 63 2F 6D 61 70 70 65 72 2F 73 63 73 69 5F /fc/mapper/scsi_ 0040 33 32 47 5F 64 31 5F 69 6C 73 64 39 38 34 30 6C 32G_d1_ilsd9840l ** ACTUAL (Target: /fc/mapper/scsi_32G_d1_ilsd9840l/test, LBA: 1243520, Offset: 5) ** 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1967814 Title: Ubuntu 20.04.3 - ilzlnx3g1 - virtio-scsi devs on KVM guest having miscompares on disktests when there is a failed path. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1967814/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1967814] Comment bridged from LTC Bugzilla
--- Comment From pa...@de.ibm.com 2022-04-06 09:24 EDT--- (In reply to comment #76) > I can confirm that just on the patch-level only two need backporting, the > rest applies as is and I have regenerated them to match the packaging > requirements. The backport-adaptations themselves are minimal. > > From the content I guess it is complex enough that nobody can be fully sure. > I'm still reading it ... > > Until then a few questions: > - I wonder if we'd also want/need dc293f6 "scsi: fix sense code for > EREMOTEIO" (to ensure this kind of ioerror gets to the guest as well) - what > do you think? > - I also wondered about 424740d "scsi-disk: do not complete requests early > for rerror/werror=ignore" but we do not have 40dce4ee applied so that should > be ok I have nothing against including those. I was under the impression that a minimal fix is desired, and my tests indicate that dose patches are not strictly necessary. Generally I think that those are good patches, and having them is better than not having them. But then I hope most of the patches are good patches, and obviously backporting all the good patches is not very practicable -- hence the principle of minimality. It is just my two cents. My understanding of SCSI is quite poor. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1967814 Title: Ubuntu 20.04.3 - ilzlnx3g1 - virtio-scsi devs on KVM guest having miscompares on disktests when there is a failed path. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1967814/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
