On 4/15/23, FreeBSD User <free...@walstatt-de.de> wrote: > Am Sat, 15 Apr 2023 07:36:25 -0700 > Cy Schubert <cy.schub...@cschubert.com> schrieb: > >> In message <20230415115452.08911...@thor.intern.walstatt.dynvpn.de>, >> FreeBSD Us >> er writes: >> > Am Thu, 13 Apr 2023 22:18:04 -0700 >> > Mark Millard <mark...@yahoo.com> schrieb: >> > >> > > On Apr 13, 2023, at 21:44, Charlie Li <vish...@freebsd.org> wrote: >> > > >> > > > Mark Millard wrote: >> > > >> FYI: in my original report for a context that has never had >> > > >> block_cloning enabled, I reported BOTH missing files and >> > > >> file content corruption in the poudriere-devel bulk build >> > > >> testing. This predates: >> > > >> https://people.freebsd.org/~pjd/patches/brt_revert.patch >> > > >> but had the changes from: >> > > >> https://github.com/openzfs/zfs/pull/14739/files >> > > >> The files were missing from packages installed to be used >> > > >> during a port's build. No other types of examples of missing >> > > >> files happened. (But only 11 ports failed.) >> > > > I also don't have block_cloning enabled. "Missing files" prior to >> > > > brt_rev >> > ert may actually >> > > > be present, but as the corruption also messes with the file(1) >> > > > signature, >> > some tools like >> > > > ldconfig report them as missing. >> > > >> > > For reference, the specific messages that were not explicit >> > > null-byte complaints were (some shown with a little context): >> > > >> > > >> > > ===> py39-lxml-4.9.2 depends on shared library: libxml2.so - not >> > > found >> > > ===> Installing existing package /packages/All/libxml2-2.10.3_1.pkg >> > > >> > > [CA72_ZFS] Installing libxml2-2.10.3_1... >> > > [CA72_ZFS] Extracting libxml2-2.10.3_1: .......... done >> > > ===> py39-lxml-4.9.2 depends on shared library: libxml2.so - found >> > > >> > > (/usr/local/lib/libxml2.so) . . . >> > > [CA72_ZFS] Extracting libxslt-1.1.37: .......... done >> > > ===> py39-lxml-4.9.2 depends on shared library: libxslt.so - found >> > > >> > > (/usr/local/lib/libxslt.so) ===> Returning to build of >> > > py39-lxml-4.9.2 >> > > . . . >> > > ===> Configuring for py39-lxml-4.9.2 >> > > Building lxml version 4.9.2. >> > > Building with Cython 0.29.33. >> > > Error: Please make sure the libxml2 and libxslt development packages >> > > are in >> > stalled. >> > > >> > > >> > > [CA72_ZFS] Extracting libunistring-1.1: .......... done >> > > ===> libidn2-2.3.4 depends on shared library: libunistring.so - not >> > > found >> > >> > > >> > > >> > > [CA72_ZFS] Extracting gmp-6.2.1: .......... done >> > > ===> mpfr-4.2.0,1 depends on shared library: libgmp.so - not found >> > > >> > > >> > > >> > > ===> nettle-3.8.1 depends on shared library: libgmp.so - not found >> > > ===> Installing existing package /packages/All/gmp-6.2.1.pkg >> > > [CA72_ZFS] Installing gmp-6.2.1... >> > > the most recent version of gmp-6.2.1 is already installed >> > > ===> nettle-3.8.1 depends on shared library: libgmp.so - not found >> > > >> > > *** Error code 1 >> > > >> > > >> > > autom4te: error: need GNU m4 1.4 or later: /usr/local/bin/gm4 >> > > >> > > >> > > checking for GNU >> > > M4 that supports accurate traces... configure: error: no acceptable m4 >> > > coul >> > d be found in >> > > $PATH. GNU M4 1.4.6 or later is required; 1.4.16 or newer is >> > > recommended. >> > > GNU M4 1.4.15 uses a buggy replacement strstr on some systems. >> > > Glibc 2.9 - 2.12 and GNU M4 1.4.11 - 1.4.15 have another strstr bug. >> > > >> > > >> > > ld: error: /usr/local/lib/libblkid.a: unknown file type >> > > >> > > >> > > === >> > > Mark Millard >> > > marklmi at yahoo.com >> > > >> > > >> > >> > Hello >> > >> > whar is the recent status of fixing/mitigate this desatrous bug? >> > Especially f >> > or those with the >> > new option enabled on ZFS pools. Any advice? >> > >> > In an act of precausion (or call it panic) I shutdown several servers to >> > prev >> > ent irreversible >> > damages to databases and data storages. We face on one host with >> > /usr/ports r >> > esiding on ZFS >> > always errors on the same files created while staging (using portmaster, >> > leav >> > es the system >> > with noninstalled software, i.e. www/apache24 in our case). Deleting the >> > work >> > folder doesn't >> > seem to change anything, even when starting a scrubbing of the entire >> > pool (R >> > AIDZ1 pool) - >> > cause unknown, why it affects always the same files to be corrupted. >> > Same wit >> > h deve/ruby-gems. >> > >> > Poudriere has been shutdown for the time being to avoid further issues. >> > >> > >> > Are there any advies to proceed apart from conserving the boxes via >> > shutdown? >> > >> > Thank you ;-) >> > oh >> > >> > >> > >> > -- >> > O. Hartmann >> >> With an up-to-date tree + pjd@'s "Fix data corruption when cloning >> embedded >> blocks. #14739" patch I didn't have any issues, except for email messages >> >> with corruption in my sent directory, nowhere else. I'm still >> investigating >> the email messages issue. IMO one is generally safe to run poudriere on >> the >> latest ZFS with the additional patch. >> >> My tests of the additional patch concluded that it resolved my last >> problems, except for the sent email problem I'm still investigating. I'm >> sure there's a simple explanation for it, i.e. the email thread was >> corrupted by the EXDEV regression which cannot be fixed by anything, even >> >> reverting to the previous ZFS -- the data in those files will remain >> damaged regardless. >> >> I cannot speak to the others who have had poudriere and other issues. I >> never had any problems with poudriere on top of the new ZFS. >> >> WRT reverting block_cloning pools to without, your only option is to >> backup >> your pool and recreate it without block_cloning. Then restore your data. >> >> > > All right, I interpret the answer that way, that I need a most recent source > tree (and > accordingly built and installed OS) AND a patch that isn't officially > commited? > > On a box I'm with: > > FreeBSD 14.0-CURRENT #8 main-n262175-5ee1c90e50ce: Sat Apr 15 07:57:16 CEST > 2023 amd64 > > The box is crashing while trying to update ports with the well known issue: > > Panic String: VERIFY(!zil_replaying(zilog, tx)) failed > > At the moment all alarm bells are ringing and I lost track about what has > been patched and > already commited and what is still as patch available but in the phase of > evaluation or > inofficially emmited here. > > According to the EXDEV issue: in cases of poudriere or ports trees on ZFS, > what do I have to > do to ensure that those datasets are clean? The OS should detect file > corruption but in my > case the box is crashing :-( > > I did several times scrubbing, but this seems to be the action of a helpless > and desperate man > ... ;-/ > > Greetings >
Using block cloning is still not safe, but somewhere in this thread pjd had a patch to keep it operatinal for already cloned files without adding new ones. Anyhow, as was indicated by vishwin@ there was data corruption *unrelated* to block cloning which also came with the import, I narrowed it down: https://github.com/openzfs/zfs/issues/14753 That said now I'm testing a kernel which does not do block cloning and does not have the other problematic commit, we will see if things work. -- Mateusz Guzik <mjguzik gmail.com>