Re: continuous ffs_blkfree_common panic
On Sat, May 04, 2024 at 12:12:35AM +0300, Andrius V wrote:
> On Fri, May 3, 2024 at 3:59 PM Andrius V wrote:
> >
> > Hi,
> >
> > Today I reinstalled one of my systems (NetBSD 10) and while setting up
> > newly created user's password I received a panic. Since then system
> > always panics in the same way on boot:
> >
> > [ 6.252795] panic: ffs_blkfree_common: freeing free frag: dev =
> > 0xa801, block = 186040, fs = /
> > [ 6.252795] cpu0: Begin traceback...
> > [ 6.252795] vpanic() at netbsd:vpanic+0x183
> > [ 6.252795] panic() at netbsd:panic+0x3c
> > [ 6.252795] ffs_blkfree_common.isra.0() at
> > netbsd:ffs_blkfree_common.isra.0+0x951
> > [ 6.252795] ffs_blkfree_cg() at netbsd:ffs_blkfree_cg+0x106
> > [ 6.252795] ffs_realloccg() at netbsd:ffs_realloccg+0x899
> > [ 6.262794] ffs_balloc() at netbsd:ffs_balloc+0x1290
> > [ 6.262794] ufs_gop_alloc() at netbsd:ufs_gop_alloc+0xa7
> > [ 6.262794] ufs_balloc_range() at netbsd:ufs_balloc_range+0x154
> > [ 6.262794] ffs_write() at netbsd:ffs_write+0x34e
> > [ 6.262794] VOP_WRITE() at netbsd:VOP_WRITE+0xa6
> > [ 6.262794] vn_write() at netbsd:vn_write+0x10e
> > [ 6.262794] dofilewrite() at netbsd:dofilewrite+0x80
> > [ 6.262794] sys_write() at netbsd:sys_write+0x49
> > [ 6.272794] syscall() at netbsd:syscall+0x1fc
> > [ 6.272794] --- syscall (number 4) ---
> > [ 6.272794] netbsd:syscall+0x1fc:
> > [ 6.272794] cpu0: End traceback...
> >
> > The disk is Kingston KC400 SSD, there's more than enough space on it.
> > There are three gpt partitions (root/home/swap). root and home uses
> > ffs2ea and wapbl enabled (log). The actual boot happens from USB drive
> > but later altroot is used to switch to SSD.
once a UFS fs that is used with logging becomes corrupted, it will often
stay corrupted until you manually run a full fsck on it ("fsck -fy").
the "fsck -p" that is run automatically only does log replay, and if
the metadata changes in the log do not fix the problem then the problem
will escape detection and remain unfixed even when the fs is mounted r/w.
we should add a way (such as setting a flag in the superblock) to mark
a file system as corrupted so that the automatic fsck will not mark the fs
as clean after only replaying the log, but instead require a full fsck
before the fs can be marked clean again.
> > [ 3.402785] wd0 at atabus0 drive 0
> > [ 3.402785] wd0:
> > [ 3.402785] wd0: drive supports 16-sector PIO transfers, LBA48
> > addressing
> > [ 3.402785] wd0: 476 GB, 992277 cyl, 16 head, 63 sec, 512
> > bytes/sect x 1000215216 sectors
> > [ 3.402785] wd0: GPT GUID: d220d34c-d8d6-44a9-8a2a-09818d238115
> > [ 3.402785] dk8 at wd0: "8baeac2c-5905-4005-a876-f61c560390f8",
> > 262144 blocks at 2048, type: msdos
> > [ 3.402785] dk9 at wd0: "nasroot", 724992000 blocks at 264192, type: ffs
> > [ 3.402785] dk10 at wd0: "nashome", 242522112 blocks at 725258240,
> > type: ffs
> >
> > Mounting partitions from another NetBSD system doesn't cause panic and
> > I can see files.
> >
> > Did I hit some hardware issue or is it software bug? What could be an
> > option to go forward to restore the system (reinstall, relocate files,
> > etc?)?
corruption could be either hardware or software, there's often no way
to tell from a single occurance.
> > Regards,
> > Andrius V
>
> Seems like some data corruption occurred for some reason causing the
> issue (/etc/passwd file got filled with \xff values, master.passwd
> file had jiberrish values at the end of the file, password database
> corrupted as well). After some actions crash is not reproducible
> anymore though. I guess issue can be considered as some kind of fluke.
this can happen with our journal implementation because regular file data
is not journaled and there is no other mechanism to make sure that
uninitialized blocks are either initialized or removed from the file
after a crash. no one has had the time to deal with this yet.
-Chuck
Re: ATF tests panic assertion "uvmexp.swpgonly > 0" failed
On Fri, Dec 08, 2023 at 06:13:42PM +0100, Manuel Bouyer wrote: > Hello again > I see a second rare panic running ATF tests on Xen: > lib/libc/regex/t_exhaust (236/949): 1 test cases > regcomp_too_big: [ 1254.5816543] panic: kernel diagnostic assertion > "uvmexp.swpgonly > 0" failed: file "/usr/src/sys/uvm/uvm_anon.c", line 175 > [ 1254.6116351] cpu1: Begin traceback... > [ 1254.6216378] > vpanic(c12d3bf8,d855adcc,d855ade8,c0d03f72,c12d3bf8,c12d3b5f,c13da23e,c13da189,af,c3b00ac0) > at netbsd:vpanic+0x184 > [ 1254.6516393] > kern_assert(c12d3bf8,c12d3b5f,c13da23e,c13da189,af,c3b00ac0,c2d9f8d0,0,d855ae0c,c0d041c4) > at netbsd:kern_assert+0x23 > [ 1254.6716402] > uvm_anfree(c2d9f8d0,c2342000,3,c543b0c0,0,c3b00ac0,1,d855ae58,c0d20d1d,c2d9f8d0) > at netbsd:uvm_anfree+0x2b8 > [ 1254.7016358] > uvm_anon_release(c2d9f8d0,1,d72f2000,c543b0c0,d72f2000,d72f1000,1,0,0,d855ae84) > at netbsd:uvm_anon_release+0x85 > [ 1254.7216389] > uvm_aio_aiodone_pages(d855ae84,1,1,0,c243c400,1ebf140,0,0,d855ae84,c1e6cc24) > at netbsd:uvm_aio_aiodone_pages+0x2fc > [ 1254.7716201] > uvm_aio_aiodone(c5560180,8e016,3,0,72bec,c26ff204,c26ff204,c5560180,d855af20,c0e54a31) > at netbsd:uvm_aio_aiodone+0x97 > [ 1254.7916387] > biodone2(c5560180,1000,0,c25e4cec,c0db8d85,c5561024,c0e54996,d82f8000,d855af48,c0e0a32d) > at netbsd:biodone2+0x95 > [ 1254.8216356] > dkiodone(c5561024,10,10,d85502ac,c010293f,d855af70,c5561024,3,d855af70,c0e0a49e) > at netbsd:dkiodone+0x9b > [ 1254.8416368] > biodone2(3,0,c010293f,8a260008,2,d8550004,c243c980,d85502ac,d855afe0,c0d7f5c5) > at netbsd:biodone2+0x95 > [ 1254.8815946] biointr(0,0,0,0,0,0,0,0,0,0) at netbsd:biointr+0x4c > [ 1254.8916211] > softint_dispatch(c243c400,3,c2c2c2c2,c2c2c2c2,c2c2c2c2,c2c2c2c2,d855dff0,d855df14,c268b000,80050033) > at netbsd:softint_dispatch+0xe0 > [ 1254.9216366] Bad frame pointer: 0xd82fcf20 > [ 1254.9316199] cpu1: End traceback... > > The first time seems to be > https://largo.lip6.fr/~bouyer/NetBSD-tests/xen/HEAD/i386-hvm/202310061820Z_anita.txt > > and the second time was > https://largo.lip6.fr/~bouyer/NetBSD-tests/xen/HEAD/i386-hvm/202311250810Z_anita.txt > > Has anyone else seen this ? yes, various people have been seeing this assertion (or some other related ones) occasionally for years now. I've looked into it a few times but I have been unable to spot the problem. -Chuck
Re: dtracing unlink
On Tue, Oct 31, 2023 at 10:17:25PM -0700, bch wrote: > > I think we can just use SMAP_DISABLE/SMAP_ENABLE like in the standard > > copyinstr() in sys/arch/amd64/amd64/copy.S. > > > Yeah - I saw that, saw the registered function, the release notes… > > I followed the hint in the notes, and the aesthetic on the page. The call > does the same thing as the macros, but via a call/return. Literally the > same thing - so the proper hotpatching is still applied. It saves seeing > the SHOUTING_MACRO and the #include to get the macro in scope, but > otherwise, ~same thing as I see it, and I’m not otherwise invested. Notable > - this case and a single Xen spot are the only use of > smap_disable()/smap_enable() - if we’re just going the macro route, should > the function just be deleted? it would be better if dtrace could use the SMAP_DISABLE/SMAP_ENABLE hotpatch macros like the base kernel copy.S does, but hotpatching does not currently work for kernel modules, only for the base kernel. so at the moment it's expedient to just call a hotpatched function in the base kernel like you proposed. further, any function that is called in dtrace probe context should have a name starting with "dtrace_" so that the fbt provider will avoid creating fbt probes for those functions. I committed a patch like the above and verified that the dtrace script from wiz works now. -Chuck
Re: HEADS UP: UFS2 extended attribute changes will be committed tomorrow
On Mon, Dec 05, 2022 at 11:32:38AM +0100, Matthias Petermann wrote: > Hi Chuck, > > Am 22.11.22 um 07:10 schrieb Chuck Silvers: > [] > > as you noted in your later mail, this is documented only in the fsck_ffs > > manpage > > since it only applies to fsck_ffs and not the fs-independent fsck wrapper > > program. > > > > thanks for your feedback! > > > > -Chuck > > thank you very much for the explanations. The UFS2ea is now running stable > for a few weeks at my place (migrated with fsck_ffs -p -c ea) as SYSVOL of a > Samba server, as well as for some filesystems used as large file shares. > > In my records, I noticed another note that I had made more than a year ago. > That was around the time the Posix ACLs were imported into NetBSD. I'm not > sure if this is on anyone's radar, or if it's generally considered a > requirement. I would be interested to know how dump / restore are affected > by the current changes, or if there are plans to add support for Extended > Attributes / ACLs to them. Back in FreeBSD 6.0 times there was a patch[1] > that did exactly that for the FreeBSD variants of dump / restore. Probably > the source code deviates however far from it. christos applied the dump changes for extattrs from freebsd to our code last year: commit 52fea75266aee4480e62dd763d4d3d74b002ea5c Author: christos Date: Sat Jun 19 13:56:34 2021 + Add external attribute dumping and restoring support from FreeBSD. Does not fully work yet, attributes are being saved and restored correctly, but don't appear in the restored files somehow. there is a bug in that code that effectively prevents restoring NFSv4 ACLs. I have a fix for this bug that I need to apply to freebsd and then I'll apply it to our code as well. -Chuck > Kind regards and thanks again for all the work with this much appreciated > feature > Matthias > > [1] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=93085
Re: HEADS UP: UFS2 extended attribute changes will be committed tomorrow
On Fri, Nov 18, 2022 at 09:09:22AM +0100, Matthias Petermann wrote: > Hello Chuck, hello all > > I have done some tests with a current build. I did a complete > reinstallation. I noticed a few small things that I would like to point out. > Some of the things are probably not caused by the current commit, I just > want to mention them in the context from a user's point of view: > > - UFS2 seems to be the default in Sysinst when FFSv2 is selected. I did not > find a selection to explicitly choose UFS2ea. A new installation with UFS2ea > enabled FFSv2 as root filesystem is currently only possible via a detour > (conversion via fsck in single user mode). Would an addition of sysinst be > desirable here? yes, there should be a way to tell sysinst to create UFS2ea rather than UFS2 (or vice-versa if we make UFS2ea the default in sysinst). this was on my list of outstanding issues but I think I was thinking that Martin would take care of it (since he's been the one doing most of the sysinst changes for a long time now). at any rate, either Martin or I will add this. > - A FFSv2 with UFS2 (without ea) can be mounted without error message with > the option "posix1eacls". Only when trying to manipulate an ACL with setfacl > you get an error message "Operation not supported". Is this due to the fact > that the mount command generally does not check in advance whether the > addressed file system supports the options or does not know about them, or > is this a place where the magic bytes still have to be updated? that was an oversight. I think that it would be best to fail the mount if the fs being mounted cannot support a requested mount option like this. I'll add a check for that (though it may be a while before I can get to this). > - The mount option "posix1eacls" (I think there was a second ACL related > one?) is missing in the man page mount(8). I feel like I've seen this before > in a man page...was I looking for it in the wrong place? the other ACL option is "nfs4acls". I would have thought that mount options specific to certain file systems would go in the manpage for that file system's mount command (mount_ffs in this case), but it looks like some options that apply only to one file system type are documented in mount.8. however, most file-system-type-specific mount options and arguments are documented only in their respective mount_foo.8 manpages, and I think this is the better practice to follow. I'll add the acls options to mount_ffs.8 and see about moving some of the text in mount.8 to the appropriate mount_foo.8. > - The new fsck option "-c" to convert from UFS2 to UFS2es and vice versa is > missing in the man page fsck(8) as you noted in your later mail, this is documented only in the fsck_ffs manpage since it only applies to fsck_ffs and not the fs-independent fsck wrapper program. thanks for your feedback! -Chuck
HEADS UP: UFS2 extended attribute changes will be committed tomorrow
Hi folks, On Wednesday I'll be committing the changes that I proposed a while back that restore UFS2 backward-compatibility with previous NetBSD releases and create a new "UFS2ea" variant of UFS2 that supports extended attributes. The previous discussion of this issue started with this post: https://mail-index.netbsd.org/current-users/2022/05/24/msg042387.html The diff that I'll be committing (if no additional changes arise) is at: https://ftp.netbsd.org/pub/NetBSD/misc/chs/diff.ufs2ea.20221114.1 There is a wiki page with instructions for updating a -current installation from before this change to after this change at: https://wiki.netbsd.org/features/UFS2ea/ Please let me know if there are any questions or concerns. -Chuck
Re: "zfs send" freezes system (was: Re: pgdaemon high CPU consumption)
On Tue, Jul 19, 2022 at 08:46:07AM +0200, Matthias Petermann wrote: > Hello, > > On 13.07.22 12:30, Matthias Petermann wrote: > > > I can now confirm that reverting the patch also solved my problem. Of > > course I first fell into the trap, because I had not considered that the > > ZFS code is loaded as a module and had only changed the kernel. As a > > result, it looked at first as if this would not help. Finally it did...I > > am now glad that I can use a zfs send again in this way. This previously > > led reproducibly to a crash, whereby I could not make backups. This is > > critical for me and I would like to support tests regarding this. > > > > In contrast to the PR, there are hardly any xcalls in my use case - > > however, my system only has 4 CPU cores, 2 of which are physical. > > > > > > Many greetings > > Matthias > > > > Roundabout one week after removing the patch, my system with ZFS is behaving > "normally" for the most part and the freezes have disappeared. What is the > recommended way given the 10 branch? If it is not foreseeable that the basic > problem can be solved shortly, would it also be an option to withdraw the > patch in the sources to get at least a stable behavior? (Not only) on the > sidelines, I would still be interested in whether this "zfs send" problem > occurs in general, or whether certain hardware requirements have a favorable > effect on it. > > Kind regards > Matthias hi, sorry for the delay in getting to this. what is happening here is that the pagedaemon is hitting the check for "uvm_km_va_starved_p()", which tries to keep the usage of kernel memory below 90% of the virtual space available for kernel memory. the checks that I changed (effectively removed for 64-bit kernels) in that previous patch tried to keep the ARC kernel memory usage below 75% of the kernel virtual space. on other OSs that support ZFS, the kernel allocates enough virtual space for the kernel be able to allocate almost all of RAM for itself if it wants, but on netbsd we have this calculation in kmeminit_nkmempages(): #if defined(KMSAN) npages = (physmem / 8); #elif defined(PMAP_MAP_POOLPAGE) npages = (physmem / 4); #else npages = (physmem / 3) * 2; #endif /* defined(PMAP_MAP_POOLPAGE) */ #ifndef NKMEMPAGES_MAX_UNLIMITED if (npages > NKMEMPAGES_MAX) npages = NKMEMPAGES_MAX; #endif this limits the amount of kernel memory to 1/4 of RAM on 64-bit platforms. PMAP_MAP_POOLPAGE is for accessing pool objects that are smaller than a page using a direct-mapped region of virtual addresses. all 64-bit kernels can do this... though it looks like sparc64 doesn't do this for such pool allocations even though it could? weird. most 64-bit kernels also define NKMEMPAGES_MAX_UNLIMITED to indicate that no arbitrary fixed limit should be imposed on kernel memory usage. though again not all platforms that could define this actually do. this time it's the mips kernels that don't enable this one. for ZFS, the memory used for the ARC cache is allocated through pools but the allocation sizes are almost all larger than a page, so basically none of these allocations will be able to use the direct map, and instead they will all have to allocate kernel virtual space. I don't think it makes sense for the kernel to arbitrarily limit the ZFS ARC cache to 1/4 of RAM just because that's how much virtual space is made available for kernel memory mappings, so instead I think we should increase the size of the kernel virtual space on 64-bit kernels to support mapping all of RAM, something like the attached patch. however even with this change, reading an bunch of data into the ZFS ARC still results in the system hanging, this time due to running out of physical memory. there are other mechanisms that ZFS also uses to try to control its memory usage, and some part of that is apparently not working either. I'm continuing to look into this. -Chuck Index: src/sys/uvm/uvm_km.c === RCS file: /home/chs/netbsd/cvs/src/sys/uvm/uvm_km.c,v retrieving revision 1.160 diff -u -p -r1.160 uvm_km.c --- src/sys/uvm/uvm_km.c13 Mar 2021 15:29:55 - 1.160 +++ src/sys/uvm/uvm_km.c26 Jul 2022 20:24:14 - @@ -237,6 +237,8 @@ kmeminit_nkmempages(void) #ifndef NKMEMPAGES_MAX_UNLIMITED if (npages > NKMEMPAGES_MAX) npages = NKMEMPAGES_MAX; +#else + npages = physmem; #endif if (npages < NKMEMPAGES_MIN)
Re: File system corruption due to UFS2 extended attributes
On Tue, May 24, 2022 at 07:51:08AM -0400, Greg Troxel wrote: > > Chuck Silvers writes: > > > The introduction in NetBSD's implementation of UFS2 of the extended > > attribute code from FreeBSD has introduced a compatibility problem > > with previous releases of NetBSD. The explanation of this problem is > > a bit involved and requires knowing some history, so please bear with me > > as I explain. > > Your analysis and approach make sense to me, even though it's > regrettable that it is necessary. I guess UFS needs zfs-style feature > flags > > What about compatibility with FreeBSD? > > - What happens if someone takes a FreeBSD UFS2 filesystem and mounts > it under NetBSD 9? FreeBSD UFS2 and NetBSD 9 UFS2 are "somewhat" compatible, the main exceptions being extended attributes and the interpretation of some of the fs_flags bits in the superblock. These fs_flags bits that are used different between the two control enablement of various optional features, such as "check hashes" in FreeBSD, and wapbl and "quota2" in NetBSD. Note that FreeBSD's bit for "check hashes" and NetBSD's bit for "quota2" are the same bit, so if this bit is set by one OS then the other OS will do the wrong thing. FreeBSD would decide that everything in the NetBSD fs is corrupt because none of the check hashes matches. NetBSD will refuse to mount a FreeBSD fs read/write because other quota2 information is missing or wrong (this one I know from recent experience). Similarly, the bits for FreeBSD "NFS4 ACLs" and NetBSD "wapbl" are the same. FreeBSD only clears some unknown fs_flags bits, whereas NetBSD clears all unknown fs_flags bits. Looking again now, I see that various of the newer superblock fields are also different. These fields were added by reusing some of the various "spare" bytes that were available, but often the same "spare" bytes were reused for different purposes by each OS. I'm sure the different interpretations of some of these newer fields can cause trouble, however sometimes nothing obviously bad happens when a file system created on one OS is used on the other OS. It all depends on exactly what you do. > - What happens if someone tries to mount a NetBSD <=9 UFS2 filesystem > on FreeBSD? A 10 UFS2 filesystem w/o ea? with? NetBSD <=9 UFS2 vs FreeBSD UFS2 is described above. NetBSD 10 UFS2 (non-ea) will be the same as NetBSD <=9 UFS2 after the changes that I am proposing now. NetBSD 10 UFS2ea will not be recognized at all by FreeBSD (or by NetBSD <=9). > Or is it already the case that FreeBSD and NetBSD do not interoperate > with UFS2? They will each try to operate on the other's UFS2 file systems (because they can't tell the difference), but there is a good chance that data loss will result if you mount read/write from the other OS. > And same questions for the other active BSD variants, which I think is > mostly OpenBSD and Dragonfly these days but I have lost track. OpenBSD UFS2 appears to be the same as NetBSD <=9 with respect to extended attributes (extattrs are not supported). OpenBSD's treatment of fs_flags is different as well, only two fs_flags bits are recognized and unknown flags are not cleared. At least one superblock field is different too. Dragonfly does not support UFS2 at all. -Chuck
Re: File system corruption due to UFS2 extended attributes
On Tue, May 24, 2022 at 06:25:34AM -, Michael van Elst wrote: > c...@chuq.com (Chuck Silvers) writes: > > > - fsck will take a new option "-c ea" to specify that an existing UFS2 > > file system should be converted to support extended attributes > > (ie. converted to UFS2ea). This conversion first clears all of the > > on-disk > > pointers to extended attribute blocks (the inode "di_extb" field), > > since in NetBSD releases prior to NetBSD 10, those pointers could only > > have been set to non-zero values by corruption in the file system. > > There should be a way back so that the filesystem becomes usuable > by netbsd-9 again (basically: clear di_extb and set magic to UFS2). > Would also be nice to pull up that feature to netbsd-9. (please don't remove current-users from the cc, this discussion is as much for that audience as it is for tech-kern) having an option to fsck to convert back to non-ea UFS2 is reasonable, with the warning that this results in throwing away all extattrs in the fs. I'll add that. note that this will also free any blocks which were being used to store extattr data. back-porting that option to netbsd-9 can be done as well, though of course it wouldn't help if the fs in question is the root fs. -Chuck
File system corruption due to UFS2 extended attributes
The introduction in NetBSD's implementation of UFS2 of the extended attribute code from FreeBSD has introduced a compatibility problem with previous releases of NetBSD. The explanation of this problem is a bit involved and requires knowing some history, so please bear with me as I explain. On 2002-06-20, initial support for UFS2 was added to FreeBSD in svn r98542 (now git 1c85e6a35d93195e896b030d9a55f7ac4ccee2c3). In this version of the code, the on-disk field for extended attributes (di_extb[] in struct dinode) was present but unused. This field in the dinode was initialized to zero and never accessed after that. On 2002-07-19, the FreeBSD UFS2 kernel code was changed in svn r100344 (now git 7aca6291e3fb803b6563ce6e39680da3a2ba0feb) to actually use the di_extb[] field to store pointers to blocks containing extended attributes. On 2002-09-23, support was added in FreeBSD fsck_ffs in svn r103885 (now git c18ef4c018881d7e39b27df5bf7b65b4d58cac6b) to consider the blocks referenced by di_extb[] as allocated, and thus to refrain from freeing any blocks which were in use for extended attribute data. All of the above changes were made during the FreeBSD 5 development cycle, so FreeBSD 4.x did not have UFS2 at all and FreeBSD 5.0 had full support in both the kernel and fsck_ffs for UFS2 with extended attributes. Over in NetBSD-land, on 2003-04-02 a version of UFS2 was imported which was based on the original 2002-06-20 version from FreeBSD, and did not include the later changes for extended attributes, with a note in the commit message that the extended attribute support would be added "later". This version of UFS2 was first released in NetBSD 2, and it did not include any support for extended attributes. Fast-forward to 2020, when the UFS2 code in NetBSD was finally enhanced to support extended attributes. This support will be first released in NetBSD 10. Unlike in FreeBSD where the first release of UFS2 included all of the support for extended attributes, in NetBSD there have been many releases which support UFS2 but without any knowledge of UFS2 extended attributes. This is where we have a problem. If a UFS2 file system is mounted under NetBSD 10 and an extended attribute (such as an ACL) is stored on a file, and then that file system is transported to a NetBSD 9 system, fsck_ffs on the Netbsd 9 system will report that the file system is corrupted because the extended attribute block is marked as allocated, but this block is not referenced in any way that NetBSD 9 knows about, and thus fsck_ffs will repair this corruption by marking that extended attribute block as free, but it will also not clear the file's di_extb[] field, because that field is not used in NetBSD 9. NetBSD 9 now thinks that the file system is ready to use, and the file system can be mounted and used normally. NetBSD 10 on the other hand would think that this file system was fine before the NetBSD 9 fsck_ffs was run, and after the NetBSD 9 fsck_ffs has been run, the NetBSD 10 fsck_ffs would say that the file system is now corrupted, because the blocks that the di_extb[] field points to are marked as free. Because the extended attribute block is now free, it can be allocated to a different inode for a different purpose, eg. to a regular file as a data block. If this file system is then transported back to on a NetBSD 10 system, the file system will be mountable without running fsck because the NetBSD 9 kernel will mark the file system clean during unmount, and now the NetBSD 10 kernel will try to process the new contents of the dup block as both application data and extended attribute metadata. The NetBSD 10 fsck_ffs will report that the file system is corrupted again because extended attribute block which was freed and then allocated again is now referenced by multiple inodes (di_extb[] from the original inode and eg. di_db[] from a different inode in this example). NetBSD 10 fsck_ffs will then repair this dup-blocks corruption by zeroing both inodes that refer to this block. So what can we do about this? There aren't any really great options. But the only change which will guarantee that all old NetBSD releases (which do not know about extend attributes) will not corrupt file system images where extended attributes have been stored is to create a new variant of UFS2 with a different magic number (the "fs_magic" field in the superblock). This is what I propose to do. I spoke with Kirk McKusick about this problem and he agreed that creating a new UFS2 variant with a different magic number is the best way to deal with this situation. This new UFS2 variant (which I'm calling "UFS2ea") will be different from the existing NetBSD UFS2 as shipped in previous NetBSD release only in that it will add support for extended attributes, and that it will only be supported by NetBSD 10 and later. In all other respects, UFS2ea will be the same as NetBSD's existing UFS2. The user-interface changes for UFS2ea will include: - newfs
Re: cmake hang solution?
On Mon, May 02, 2022 at 08:16:42PM +0200, Manuel Bouyer wrote: > On Mon, May 02, 2022 at 11:13:45AM -0700, Chuck Silvers wrote: > > it looks like the diff won't apply as-is, but I think the concept still > > applies. > > > > note that there have been a LOT of changes in libpthread since netbsd-9, > > and some of those changes also claim to fix problems where threads hang > > waiting on locks and/or condvars. it would be more useful to test > > with a HEAD libpthread (which I'll guess requires a HEAD libc too). wiz@ tells me that he has reproduced the cmake hang with HEAD kernel and userland plus my libpthread patch, so no one else needs to try that now. I'll work with him to figure out what else is still causing the cmake variation of this. -Chuck
Re: cmake hang solution?
On Mon, May 02, 2022 at 10:12:02PM +0200, Michael van Elst wrote: > On Sun, May 01, 2022 at 01:24:01PM -0700, Chuck Silvers wrote: > > On Tue, Apr 05, 2022 at 02:10:36PM -, Michael van Elst wrote: > > > I see both in almost every pbulk run. > > > > please try this patch for the cmake variation of this hang: > > > > http://www.netbsd.org/~chs/diff.pthread-park-stuck.1 > > The bulk builds use the latest release, i.e netbsd-9, but that > patch is for -current. Do you think that netbsd-9 has the same > issue and the patch could be reworked for the older code ? netbsd-9 almost certainly has the issue that this latest patch is trying to fix. the patch above is just a one-liner and can easily be adapted for netbsd-9. but this patch alone is probably not enough... there have been many changes to libpthread in HEAD since netbsd-9, claiming to fix problems with the same kind of hang symptoms, and some of those would almost certainly be needed in netbsd-9 as well. I don't know for sure which of the changes from HEAD are needed, so that's why I'm suggesting we just test with all of those changes, ie. test the HEAD libpthread. -Chuck
Re: cmake hang solution?
On Mon, May 02, 2022 at 08:16:42PM +0200, Manuel Bouyer wrote: > On Mon, May 02, 2022 at 11:13:45AM -0700, Chuck Silvers wrote: > > it looks like the diff won't apply as-is, but I think the concept still > > applies. > > > > note that there have been a LOT of changes in libpthread since netbsd-9, > > and some of those changes also claim to fix problems where threads hang > > waiting on locks and/or condvars. it would be more useful to test > > with a HEAD libpthread (which I'll guess requires a HEAD libc too). > > the goal is to build the official netbsd-9 packages, so that's not an option it's possible to have individual binaries use shared libraries from a non-standard location, such that you could have cmake itself use the HEAD shared libraries to run, but that everything that the build process builds would use link against the netbsd-9 shared libraries in the normal location. I've done this kind of thing before by just editing an executable binary to change the RPATH in the ELF headers (or if the target executable has no RPATH then editing the binary to use an alterate ld.elf_so, and having the alternate ld.elf_so use an alternate default RPATH.) if you don't like that then you could also change the cmake build to link the cmake executable with an alternate rpath. I'm only suggesting all of this as a way to test if all of these hang bugs are fixed in HEAD, not that you should run your netbsd-9 userland system with this hacky setup indefinitely. -Chuck
Re: cmake hang solution?
On Mon, May 02, 2022 at 12:55:39PM +0200, Manuel Bouyer wrote: > On Sun, May 01, 2022 at 01:24:01PM -0700, Chuck Silvers wrote: > > On Tue, Apr 05, 2022 at 02:10:36PM -, Michael van Elst wrote: > > > w...@netbsd.org (Thomas Klausner) writes: > > > >I never saw the cmake hang myself. I still see hangs in guile. > > > > > > > > > I see both in almost every pbulk run. > > > > > > please try this patch for the cmake variation of this hang: > > > > http://www.netbsd.org/~chs/diff.pthread-park-stuck.1 > > would this apply to netbsd-9 too ? The hang I'm seeing is on a system > with a HEAD kernel and a netbsd-9 userland it looks like the diff won't apply as-is, but I think the concept still applies. note that there have been a LOT of changes in libpthread since netbsd-9, and some of those changes also claim to fix problems where threads hang waiting on locks and/or condvars. it would be more useful to test with a HEAD libpthread (which I'll guess requires a HEAD libc too). -Chuck
Re: cmake hang solution?
On Tue, Apr 05, 2022 at 02:10:36PM -, Michael van Elst wrote: > w...@netbsd.org (Thomas Klausner) writes: > >I never saw the cmake hang myself. I still see hangs in guile. > > > I see both in almost every pbulk run. please try this patch for the cmake variation of this hang: http://www.netbsd.org/~chs/diff.pthread-park-stuck.1 this fixes the problem as seen with taylor's strdup / jemalloc reproducer, and paulg reports it fixes the hang in building guile too. what's going on here is that the first time that libpthread calls _lwp_park when it wants to sleep to wait for a mutex, instead of calling the libc function directly it first has to call into rtld to resolve the symbol. the rtld code will call _rtld_shared_enter(), which might also need to sleep using _lwp_park to wait for the rtld internal lock. the rtld internal usage of _lwp_park can accidentally consume an unpark from another thread that was intended for the libpthread code, and if that happens, then when rtld is done resolving the symbol and libpthread actually calls the real _lwp_park function, the unpark has been lost and the libpthread call to _lwp_park will sleep forever. the above patch simply resolves the symbol for the libpthread call to _lwp_park while the process is still single-threaded, by calling the _lwp_park to both unpark and park itself, which just returns immediately. after that, the libpthread calls to _lwp_park will no longer call into rtld, so attempts to unpark libpthread can no longer be lost. when I wrote that patch I thought it would be a complete fix, but upon reading the previous email threads about this problem I saw a mention of signals, and signal handlers can call a wide variety of functions, so this patch turns out to only fix what is probably the most common way that this problem manifests. the nature of lwp_park/unpark (with just a single "already unparked" flag per thread in the kernel) is such that they cannot safely be used in a nested fashion like they are in libpthread and rtld, so we need to change one or both of these callers to use some other primitive to implement sleeping to wait for a lock, such as futexes, which do not have this kind of per-thread flag that prevents safe nested usage. -Chuck
Re: Filesystem corruption in current 9.99.92 (posix1eacl & log enabled FFSv2)
On Wed, Dec 29, 2021 at 08:01:53PM +0100, Matthias Petermann wrote: > Hello, > > On 27.12.21 06:20, Matthias Petermann wrote: > > I did not try to move the file around as you recommended because I would > > like to ask if there is anything I can do at this point to gather more > > diagnostic data to help understand the root cause? > > in the meantime I migrated all files to a freshly created filesystem using > the patched kernel and so "solved" the problem for now. > > The broken filesystem still exists, but I am now running out of space on the > host (the filesystems are in sparse allocated VNDs). I would have to delete > the broken filesystem in a timely manner, but would still like to run > diagnostic steps on the root cause first, if any. Unfortunately, the > filesystem is very large and remote, and I don't know how to reasonably > isolate the affected portion to save space for further analysis. Are there > any other reasonable steps I could do asap? let me write something to allow you to extract the contents of the corrupted extattr block, so that I can reproduce your exact situation on a test machine. once we have a copy of the corrupted block then you can reclaim the existing fs image. hopefully I'll have that later today. > One more question I would have about the patch. It helped very well to avoid > the freeze when working in such a corrupted filesystem. In this case, the > filesystem behaves as you described - no ACL is applied or issued on the > affected directory. When I try to set a new ACL on the affected directory, > it seems to have no effect, but no error message appears. Would it make > sense to include the patch with appropriate error logging in the official > sources, so that when the problem occurs for which we do not know the cause > at the moment, we will at least get some output (instead of the current > behavior - the infinite loop)? yea, the final patch will include printing a message on the console, and also include some way to restore the ability to set ACLs on the file without needing to recreate the file (let alone recreating the whole fs). that will take a little longer though. -Chuck
Re: Filesystem corruption in current 9.99.92 (posix1eacl & log enabled FFSv2)
On Thu, Dec 23, 2021 at 12:30:14PM +0100, Matthias Petermann wrote: > Hello, > > for tracking down an FFS issue in current I would appreciate some advice. > There is a NetBSD 9.99.92 Xen/PV VM (storage provided by file backed VND). > The kernel is built from ~2012-11-27 CVS source. The root partition is a > normal FFSv2 with WAPBL. In addition there is a data partition for which I > have posix1eacls enabled (for samba network shares and sysvol). > > The data partition causes problems. Without the host being crashed or rudely > shut down in the past, the filesystem seems to have become inconsistent. I > first noticed this because the "find" of the daily cron job was still > running late in the morning with 100% CPU load but no disk I/O ongoing. > > Then I took the filesystem offline for safety and forced a fsck. Errors were > detected and solved: > > ``` > $ doas fsck -f NAME=export > ** /dev/rdk3 > ** File system is already clean > ** Last Mounted on /export > ** Phase 1 - Check Blocks and Sizes > ** Phase 2 - Check Pathnames > ** Phase 3 - Check Connectivity > ** Phase 4 - Check Reference Counts > ** Phase 5 - Check Cyl groups > CG 31: PASS5: BAD MAGIC NUMBER > ALTERNATE SUPERBLK(S) ARE INCORRECT > SALVAGE? [yn] > > CG 31: PASS5: BAD MAGIC NUMBER > ALTERNATE SUPERBLK(S) ARE INCORRECT > SALVAGE? [yn] y > > SUMMARY INFORMATION BAD > SALVAGE? [yn] y > > BLK(S) MISSING IN BIT MAPS > SALVAGE? [yn] y > > CG 799: PASS5: BAD MAGIC NUMBER > CG 801: PASS5: BAD MAGIC NUMBER > CG 806: PASS5: BAD MAGIC NUMBER > CG 823: PASS5: BAD MAGIC NUMBER > CG 962: PASS5: BAD MAGIC NUMBER > CG 966: PASS5: BAD MAGIC NUMBER > 482470 files, 113827090 used, 67860178 free (3818 frags, 8482045 blocks, > 0.0% fragmentation) > > * FILE SYSTEM WAS MODIFIED * > ``` > > I did not find too much information what this magic numbers of a cylinder > group means and what could have caused them to be "bad" :-/ a "cylinder group" is a metadata structure in FFS that describes the allocation state of a portion of the blocks and inodes of the file system and contains the inode records themselves. the header for this structure also contains a "magic number" field that is supposed to contain a certain constant value as a way to sanity-check that this metadata on disk was not overwritten with some completely unrelated contents. in your case, since the magic number field does not actually contain the value that it's supposed to contain, we know that the storage underneath the file system has gotten corrupted somehow. you'll want to track down how that happened, but that is separate from your immediate problem. > Anyway, a repeated fsck does not show further errors so I thought it should > be fine. However, after mounting the FS to /export with > > ``` > $ find /export > ``` > > i can still trigger the above mentioned 100% CPU problem in a reproduce-able > manner. Thereby find always hangs at the same directory entry. > > Does anyone have an idea how I can investigate this further? I have already > done a ktrace on find, but in the state in question there seems to be no > activity going on in find itself. > > Kind regards > Matthias this sounds like a bug I have seen before, where the extended attribute block for a file has been corrupted. please try the attached patch and see if this prevents the infinite loop. if that does prevent the infinite loop, then the file will probably appear not to have an ACL anymore, and I'm not sure what will happen if you try to set a new ACL on the file when it is in this state. for right now, the safest thing you can do will be to make a copy of the file without trying to preserve extended attributes (ie. do not use cp's "-p" option), then delete the original file, then move the copy of the file to have the original file's name, then you can change the new file's owner/group/mode/ACL to be what the original file had. -Chuck Index: sys/ufs/ffs/ffs_extattr.c === RCS file: /home/chs/netbsd/cvs/src/sys/ufs/ffs/ffs_extattr.c,v retrieving revision 1.8 diff -u -p -r1.8 ffs_extattr.c --- sys/ufs/ffs/ffs_extattr.c 14 Dec 2021 11:06:50 - 1.8 +++ sys/ufs/ffs/ffs_extattr.c 23 Dec 2021 16:52:18 - @@ -393,6 +393,9 @@ ffs_findextattr(u_char *ptr, u_int lengt /* make sure this entry is complete */ if (EXTATTR_NEXT(eap) > eaend) break; + /* handle corrupted ea_length */ + if (EXTATTR_NEXT(eap) < eap + 1) + break; if (eap->ea_namespace != nspace || eap->ea_namelength != nlen || memcmp(eap->ea_name, name, nlen) != 0) continue; @@ -857,6 +860,9 @@ ffs_listextattr(void *v) /* make sure this entry is complete */ if (EXTATTR_NEXT(eap) > eaend) break; + /* handle corrupted ea_length */ +
Re: lang/mono or futex problem
On Sun, Aug 01, 2021 at 02:42:59PM +0100, Robert Swindells wrote: > > Thomas Klausner wrote: > >I've just tried updating the .net program 'torrentzip' > >(archivers/torrentzip) to the latest version. The current pkgsrc > >version can be run by calling 'mono' on it (the package installs a > >wrapper), but the updated version is a Linux binary. > > [snip] > > >But i'm not sure if that means there is a problem with futex() or if > >that is already in the error handling. > > The futex() implementation in -current doesn't work correctly, so I > would suspect that first. > > There is a newer version in the thorpej-futex branch, it works better > with Linux OpenJDK but still fails to wake up some threads when > expected. "dotnet -h" fails the same way under a netbsd-9 kernel, so this problem is not related to the futex rewrite that was committed to HEAD last april. it could be something else wrong or missing in the linux emulation code that this program happens to notice. I see that this program tries and fails to find a few shared libraries: liblttng-ust.so.0 liblttng-ust-tracepoint.so.0 and also a procfs file that our procfs does not provide even in linux mode: /proc/self/mountinfo either of those could also be what this program is unhappy about. -Chuck
Re: Automated report: NetBSD-current/i386 test failure
On Sun, Jan 17, 2021 at 08:03:20PM +0200, Andreas Gustafsson wrote: > The cause of the 1000+ new test failures has now been narrowed down to > the following commit: > > 2021.01.16.23.50.49 chs src/sys/rump/librump/rumpkern/rump.c,v 1.352 > 2021.01.16.23.51.50 chs src/sys/arch/arm/arm/psci.c,v 1.5 > 2021.01.16.23.51.50 chs src/sys/conf/files,v 1.1278 > 2021.01.16.23.51.51 chs src/sys/lib/libkern/arch/hppa/bcopy.S,v 1.16 > 2021.01.16.23.51.51 chs src/sys/lib/libkern/libkern.h,v 1.141 > 2021.01.16.23.51.51 chs src/sys/sys/cdefs.h,v 1.156 > 2021.01.16.23.51.51 chs src/sys/sys/queue.h,v 1.76 > > Logs: > > > http://releng.netbsd.org/b5reports/i386/commits-2021.01.html#2021.01.16.23.51.51 > > -- > Andreas Gustafsson, g...@gson.org thanks for pointing that out, it should be fixed now. -Chuck
Re: uvm_map_enter entry merging (was Re: vrelel...)
hi Yorick,
On Sat, Nov 28, 2020 at 12:39:56AM +0200, Yorick Hardy wrote:
> May I ask if you have an opinion on this patch? I have
> not noticed any bad behaviour if it is omitted but, if I read
> the code correctly, I don't think it is correct to fall through
> for this case.
this function is very hard to follow, it's very tangled.
I stared at it for a while and I didn't see anything wrong,
but it's hard to be sure just from reading the code.
could you explain the specific case that you think is wrong now and
that your patch fixes?
even better would be if you could write a set of atf tests to exercise
all of the possible merge cases and verify that the contents of memory
after the new mapping is created is what it should be.
any previous and next mapping should have the same contents as before,
and the new mapping should have either zeroes (for a new amap mapping)
or the uobj contents at that offset (for a new uobj mapping).
note that a vm_map_entry can reference both a uobj and an amap at the
same time, so there are 4 possible cases for the each of previous and next
entries (none, uobj, amap, uobj+amap), and two possible cases for the
new entry (uobj, amap). then I guess there are two more factors of 2
for whether the forward and/or backward merges succeed, so that gives
at least 128 cases to test. I think there are some more cases hidden
in there because there are multiple reasons why the merges might fail
and those checks are in different places, so it would really be best
to test all of the different possible paths through this function.
I would be reluctant to change anything here without such a set of
comprehensive tests, because even if we are sure that a change fixes
one case, it would be very hard to be sure that it doesn't break
some other case.
-Chuck
> --
> Kind regards,
>
> Yorick Hardy
>
> Index: uvm_map.c
> ===
> RCS file: /cvsroot/src/sys/uvm/uvm_map.c,v
> retrieving revision 1.385
> diff -u -r1.385 uvm_map.c
> --- uvm_map.c 9 Jul 2020 05:57:15 - 1.385
> +++ uvm_map.c 19 Nov 2020 16:04:07 -
> @@ -1477,6 +1477,13 @@
> amapwaitflag | AMAP_EXTEND_BACKWARDS))
> goto nomerge;
> }
> +
> + /*
> + * We could not extend either amap, just skip on.
> + */
> + else {
> + goto nomerge;
> + }
> } else {
> /*
>* Pull the next entry's amap backwards to cover this
Re: Panic: vrelel: bad ref count (9.99.54)
Hi Yorick, On Fri, Nov 27, 2020 at 06:29:07PM +0200, Yorick Hardy wrote: > > I think that uvm_mremap did not keep pace with changes in uvm. > This patch seems to fix it for me, although I have only tested > for two days so far (I am usually able to trigger the panic by > now ... but lets see). Your patch looks good, please go ahead and commit it. -Chuck > Leonardo, would you be willing to try the patch? > > -- > Kind regards, > > Yorick Hardy > > Index: sys/uvm/uvm_mremap.c > === > RCS file: /cvsroot/src/sys/uvm/uvm_mremap.c,v > retrieving revision 1.20 > diff -u -r1.20 uvm_mremap.c > --- sys/uvm/uvm_mremap.c 23 Feb 2020 15:46:43 - 1.20 > +++ sys/uvm/uvm_mremap.c 26 Nov 2020 19:14:06 - > @@ -80,10 +80,8 @@ > error = E2BIG; /* XXX */ > goto done; > } > - rw_enter(uobj->vmobjlock, RW_WRITER); > - KASSERT(uobj->uo_refs > 0); > - atomic_inc_uint(>uo_refs); > - rw_exit(uobj->vmobjlock); > + if (uobj->pgops->pgo_reference) > + uobj->pgops->pgo_reference(uobj); > reserved_entry->object.uvm_obj = uobj; > reserved_entry->offset = newoffset; > }
Re: diskless sparc locks up under memory pressure
On Mon, Nov 09, 2020 at 11:21:56AM +1100, Paul Ripke wrote: > On Sat, Nov 07, 2020 at 06:33:26PM +0300, Valery Ushakov wrote: > > I've upgraded my Krups (64MB RAM, diskless) to 9.99.75 as of Nov 6 and > > the machine is locking up at boot time in rc.d/fccache. If I disable > > fccache in rc.conf it boots but then eventually locks up when > > makemandb is run. The machine doesn't respond to pings, etc, but can > > enter ddb. In all cases the ddb stacktrace goes through: > > > > ... > > uvm_fault_internal > > mem_access_fault4m > > memfault_sun4m > > copyout > > > > with the top part of the stack trace varying. > > I reported something similar with amd64 back in May: > > https://mail-index.netbsd.org/current-users/2020/05/02/msg038498.html > > If I get a chance, I'll re-test that and see if it still locks up. I have the same questions for both of you: could you give some examples of what the rest of the stack has? what's the stack trace for the pagedaemon thread? try printing the UVM activity counters in ddb with "show uvmexp", then continue from ddb and wait 10 seconds, then print the counters again. by looking at which counters change we can see what (if anything) the pagedaemon is doing to try to free up some memory. what's the state of the relevant pools for network activity, such as mc_cache and mcl_cache, and do those change over time? you can print these from ddb with "show pool *mb_cache" and "show pool *mcl_cache". -Chuck
Re: wired memory
On Tue, Nov 10, 2020 at 02:11:32PM +, Patrick Welche wrote: > On Tue, Nov 10, 2020 at 05:26:26AM -0800, Chuck Silvers wrote: > > On Tue, Nov 10, 2020 at 11:40:08AM +, Patrick Welche wrote: > > > My 4 Nov -current/amd64 build box seems to be building slowly, with a lot > > > of "biowait", e.g. watching the RES of a cc1plus slowly crawl up to SIZE > > > at around 3M per 5s. > > > > > > Is 10G of "Wired" normal? (32G RAM + 64G swap) > > > (lots of malloc(9) and no free(9)?) > > > > kernel memory usage is not reported as "wired" memory. > > I got that notion from: > > https://www.netbsd.org/docs/internals/en/chap-memory.html#wired_memory kernel memory is "wired" in the sense that it is not reclaimable directly by the pagedaemon, but it is not reported as "wired" by top. perhaps it used to be reported that way (I don't remember), but it's mostly now. I'm not sure why the zfs pool memory was apparently being reported as "wired", it looks like we are being inconsistent. > > > Memory: 3235M Act, 115M Inact, 10G Wired, 75M Exec, 2860M File, 2800M Free > > > Swap: 64G Total, 11G Used, 53G Free > > > > no, 10GB wired memory out of 32GB total RAM is not a typical situation. > > could you send me the output of "ps aux" and "vmstat -m" ? > > Thanks - sent off list I just committed a fix (in the "solaris" module) for the garbage pool names. > > with 11GB of swap in use, it's not surprising that the system would be > > very slow. > > Part of the question is why it needs to swap... (where did the 10G go?) looks like you figured that out. however, if the 10GB of zfs/solaris kernel memory wasn't being freed up even though it was no longer needed, then it looks like our memory-pressure feedback mechanism is not working as well as it should. > > > it is "just" building... > > > > > > Could kern.maxfiles = 108928 be an issue? (3404 default below) > > > > do you mean that on the system that is slow, kern.maxfiles is 108928? > > that doesn't seem overly large for 32GB RAM. > > > > I'm not sure what you mean by "3404 default below". > > just that the box below had the default kern.maxfiles=3404 setting, but > as you say that shouldn't be too large, and without it, one sees complaints > of insufficient files. oh, is the default maxfiles 3404 even on the system with a ton of memory? if so, we ought to change that default to scale with system RAM, like many other defaults do. -Chuck
Re: wired memory
On Tue, Nov 10, 2020 at 11:40:08AM +, Patrick Welche wrote: > My 4 Nov -current/amd64 build box seems to be building slowly, with a lot > of "biowait", e.g. watching the RES of a cc1plus slowly crawl up to SIZE > at around 3M per 5s. > > Is 10G of "Wired" normal? (32G RAM + 64G swap) > (lots of malloc(9) and no free(9)?) kernel memory usage is not reported as "wired" memory. > Memory: 3235M Act, 115M Inact, 10G Wired, 75M Exec, 2860M File, 2800M Free > Swap: 64G Total, 11G Used, 53G Free no, 10GB wired memory out of 32GB total RAM is not a typical situation. could you send me the output of "ps aux" and "vmstat -m" ? with 11GB of swap in use, it's not surprising that the system would be very slow. > it is "just" building... > > Could kern.maxfiles = 108928 be an issue? (3404 default below) do you mean that on the system that is slow, kern.maxfiles is 108928? that doesn't seem overly large for 32GB RAM. I'm not sure what you mean by "3404 default below". > Just looked at another amd64 box for comparison (also building - no swap): > > Memory: 15G Act, 3820K Inact, 16M Wired, 44M Exec, 14G File, 135G Free > Swap: this system has a lot more RAM and most of it isn't even used, so I would expect this system to perform much better than the first one. -Chuck
Re: diskless sparc locks up under memory pressure
On Sat, Nov 07, 2020 at 06:33:26PM +0300, Valery Ushakov wrote: > I've upgraded my Krups (64MB RAM, diskless) to 9.99.75 as of Nov 6 and > the machine is locking up at boot time in rc.d/fccache. If I disable > fccache in rc.conf it boots but then eventually locks up when > makemandb is run. The machine doesn't respond to pings, etc, but can > enter ddb. In all cases the ddb stacktrace goes through: > > ... > uvm_fault_internal > mem_access_fault4m > memfault_sun4m > copyout > > with the top part of the stack trace varying. > > -uwe do any threads (eg. the pagedaemon) still run when the machine locks up? that's the stack trace of the pagedaemon thread? when the machine is in this state, could you run "show uvmexp" in ddb, then continue and wait for 10 seconds, then "show uvmexp" again? then we can compare the counters from the two samples to see what the pagedaemon is doing during this time. -Chuck
Re: 9.99.73 NFS file corruption
On Tue, Oct 27, 2020 at 01:18:26PM +0100, Thomas Klausner wrote: > I'm still running a September 17 kernel, and I see rare NFS corruption. > > Basically, I'm rezipping a lot of zip archives (~2TB) that are on NFS > and four of them had blocks of 0 bytes afterwards, not much. For the > ones I kept, the corruption is about 3k each (3413 bytes in one case, > 3160 in the other). > > Could there be another NetBSD NFS bug here, or is that just random noise? > (The NFS server is a Linux RAID). > Thomas all of the known bugs in this area are fixed in HEAD now. could you try again with HEAD from today or later and let me know if you still have any problem? -Chuck
Re: panic rebooting yesterday's kernel
On Wed, Oct 21, 2020 at 09:26:49AM +0100, Patrick Welche wrote: > Booted a yesterday's source amd64 kernel, and on reboot > > [ 17037.8583948] unmounting 0xfef63b813000 / (/dev/dk14)... > [ 17037.8583948] forcefully unmounting / (/dev/dk14)... > [ 17037.8783949] dk14 at wd4 (root) deleted > [ 17037.8783949] wd4: detached > [ 17037.8883950] atabus8: detached > [ 17037.8883950] ahcisata1: detached > [ 17038.6083908] Kernel lock error: _kernel_lock,244: spinout > > [ 17038.6083908] lock address : 0x81082f00 type : > spin > [ 17038.6083908] initialized : 0x80bc8340 > [ 17038.6083908] shared holds : 0 exclusive: > 1 > [ 17038.6083908] shares wanted: 0 exclusive: > 3 > [ 17038.6083908] relevant cpu : 0 last held: > 1 > [ 17038.6083908] relevant lwp : 0xfefd19744080 last held: > 0xfef642bf4280 > [ 17038.6083908] last locked* : 0x80a42ab1 unlocked : > 0x80323d4c > [ 17038.6083908] curcpu holds : 0 wanted by: > 0xfefd19744080 > > [ 17038.6763377] Skipping crash dump on recursive panic > [ 17038.6816213] panic: LOCKDEBUG: Kernel lock error: _kernel_lock,244: > spinout > [ 17038.6886673] cpu0: Begin traceback... > [ 17038.6886673] vpanic() at netbsd:vpanic+0x156 > [ 17038.6983892] snprintf() at netbsd:snprintf > [ 17038.6983892] lockdebug_more() at netbsd:lockdebug_more > [ 17038.7083890] _kernel_lock() at netbsd:_kernel_lock+0x22a > [ 17038.7183890] intr_biglock_wrapper() at netbsd:intr_biglock_wrapper+0x14 > [ 17038.7183890] Xhandle_ioapic_edge17() at netbsd:Xhandle_ioapic_edge17+0x6e > [ 17038.7296195] --- interrupt --- > [ 17038.7296195] _kernel_lock() at netbsd:_kernel_lock+0x1f8 > [ 17038.7403251] callout_softclock() at netbsd:callout_softclock+0x425 > [ 17038.7483888] softint_dispatch() at netbsd:softint_dispatch+0xf5 > address 0xc8025cf990b8 is invalid > address 0xc8025cf990b0 is invalid > address 0xc8025cf990c0 is invalid > address 0xc8025cf990b8 is invalid > address 0xc8025cf990c8 is invalid > address 0xc8025cf990c0 is invalid > address 0xc8025cf990d0 is invalid > address 0xc8025cf990c8 is invalid > [ 17038.7784674] DDB lost frame for netbsd:Xsoftintr+0x4f, trying > 0xc8025cf98ff0 > [ 17038.7900680] Xsoftintr() at netbsd:Xsoftintr+0x4f > [ 17038.7993371] --- interrupt --- > address 0xc8025cf990c8 is invalid > address 0xc8025cf99080 is invalid > [ 17038.8084475] Bad frame pointer: 0xc8025cf98450 > [ 17038.8084475] c8025cf98450: > [ 17038.8185590] cpu0: End traceback... > > > so no ddb either... > > > Cheers, > > Patrick unfortunately there's no information here indicating what the underlying problem is. I'm looking into changing something to avoid the second panic, so that if this happens again then you might get a dump next time. -Chuck
Re: Automated report: NetBSD-current/i386 test failure
On Wed, Oct 21, 2020 at 08:30:17PM +0900, Rin Okuyama wrote: > On 2020/10/21 20:10, Andreas Gustafsson wrote: > > Two days ago, the NetBSD Test Fixture wrote: > > > This is an automatically generated notice of new failures of the > > > NetBSD test suite. > > > > > > The newly failing test cases are: > > > > > > sbin/resize_ffs/t_grow:grow_16M_v0_8192 > > > sbin/resize_ffs/t_grow:grow_16M_v1_16384 > > > sbin/resize_ffs/t_grow:grow_16M_v2_32768 > > > sbin/resize_ffs/t_grow_swapped:grow_16M_v0_65536 > > > sbin/resize_ffs/t_grow_swapped:grow_16M_v1_4096 > > > sbin/resize_ffs/t_grow_swapped:grow_16M_v2_8192 > > > sbin/resize_ffs/t_shrink:shrink_24M_16M_v0_32768 > > > sbin/resize_ffs/t_shrink:shrink_24M_16M_v1_65536 > > > sbin/resize_ffs/t_shrink_swapped:shrink_24M_16M_v0_4096 > > > sbin/resize_ffs/t_shrink_swapped:shrink_24M_16M_v1_8192 > > > > These are still failing as of source date 2020.10.21.06.36.10, and the > > commit that triggered the failures has now been identified: > > > >2020.10.18.18.22.29 chs src/sys/rump/librump/rumpvfs/vm_vfs.c 1.39 > >2020.10.18.18.22.29 chs src/sys/uvm/uvm_page.c 1.248 > >2020.10.18.18.22.29 chs src/sys/uvm/uvm_pager.c 1.130 > > > > Logs from real amd64 hardware are at: > > > > > > http://www.gson.org/netbsd/bugs/build/amd64-baremetal/commits-2020.10.html#2020.10.18.18.22.29 > > This seems due to src/sys/rump/librump/rumpvfs/vm_vfs.c 1.39; > tests pass if only this commit is reverted in librumpvfs.so. I committed a fix for this just now. thanks for pointing it out. -Chuck
Re: hard hang in 9.99.74
On Mon, Oct 19, 2020 at 08:49:01AM +0200, Thomas Klausner wrote: > Hi! > > I updated to 9.99.74/amd64 from last night to test the bugfix for the NFS > problem in 9.99.73, started a bulk build and went to bed. > > In the morning the machine had hung hard, I couldn't even get into > DDB. From the logs it looks like the machine had been up for about 2 > hours. > > There was a bulk build running and heavy network traffic, mostly NFS. > I see nothing in the logs. > > Thomas ugh, it's so annoying when ddb doesn't work. can you narrow down the problem to a specific commit or smaller range of commits where it was introduced? -Chuck
Re: file system corruption
On Sun, Oct 11, 2020 at 11:19:16PM +0200, Thomas Klausner wrote: > Hi! > > I've recently updated from 9.99.73 from Sep 17 to one of Oct 5. > > I've had serious file system corruption. Mostly in mercurial and > sqlite3 databases, but also in normal files. what platform is this on? > Some of the file systems where this happened are NFS-served from Linux, > but I also saw it on a local FFSv2. I would ask what the fs block size is, but with NFS there isn't any fs block size involved, so I doubt it matters for the local file systems either. > 2c2 > < $NetBSD: uvm_amap.c,v 1.123 2020/08/18 10:40:20 chs Exp $ > --- > > $NetBSD: uvm_amap.c,v 1.125 2020/09/21 18:41:59 chs Exp $ > > 11c11 > < $NetBSD: uvm_io.c,v 1.28 2016/05/25 17:43:58 christos Exp $ > --- > > $NetBSD: uvm_io.c,v 1.29 2020/09/21 18:41:59 chs Exp $ the above changes go togther, and they were long enough ago that if they were the cause of the corruption then it seems likely that someone else would have reported it before you. also, these changes are about process address space manipulation and not file systems, so if this were the problem then you would be getting non-file-system symptoms too. > 5c5 > < $NetBSD: uvm_bio.c,v 1.121 2020/07/09 09:24:32 rin Exp $ > --- > > $NetBSD: uvm_bio.c,v 1.122 2020/10/05 04:48:23 rin Exp $ this change is somewhat more recent and specifically about file systems, so this seems more likely. could you try testing with each of the above sets of changes separately backed out, to see if you can narrow it down to one change? if the problem is not due to either of those sets of changes then your best bet is to bisect to find the change that introduced the problem. I tried to check the automated test results to see if those are showing any problems that look related, but that web server is down right now. > Anyone else having problems? > > Any ideas? > Thomas -Chuck
Re: System panicing on boot since recent uvm changes
this should be fixed now. sorry about that, the problem did not happen for me and it took me forever to find a way that I could reproduce it. -Chuck On Sat, Aug 15, 2020 at 09:01:53PM +0300, Andreas Gustafsson wrote: > Hi chs, > > At least i386, amd64, and sparc are all panicing on boot since this commit: > > 2020.08.14.09.06.14 chs src/sys/miscfs/genfs/genfs_io.c 1.100 > 2020.08.14.09.06.15 chs src/sys/uvm/uvm_extern.h 1.231 > 2020.08.14.09.06.15 chs src/sys/uvm/uvm_object.c 1.24 > 2020.08.14.09.06.15 chs src/sys/uvm/uvm_object.h 1.39 > 2020.08.14.09.06.15 chs src/sys/uvm/uvm_page.c 1.245 > 2020.08.14.09.06.15 chs src/sys/uvm/uvm_page_status.c 1.6 > 2020.08.14.09.06.15 chs src/sys/uvm/uvm_pager.c 1.129 > 2020.08.14.09.06.15 chs src/sys/uvm/uvm_vnode.c 1.116 > > Logs: > > > http://releng.netbsd.org/b5reports/i386/commits-2020.08.html#2020.08.14.09.06.15 > > Please revert the commit. > -- > Andreas Gustafsson, g...@gson.org
Re: [PATCH] net/samba4: relocate Sysvol to persist between reboots & move variable data out of /usr/pkg/etc/...
On Thu, Jul 30, 2020 at 08:52:43AM +0100, Chavdar Ivanov wrote: > > if you have done that and are still crashing due to corruption in your > > root file system, then we still have another bug in the kernel somewhere. > > So it seems to me; the peculiarities here are that in both cases / is > a GPT slice and that I have 'log' as a mount option; it was suggested > 'posix1eacls' should be used on its own. please try this patch: http://ftp.netbsd.org/pub/NetBSD/misc/chs/diff.ffs-extattr-wapbl.1 with this patch, the samba-tool provisioning command works for me either with or without the "log" mount option. -Chuck
Re: [PATCH] net/samba4: relocate Sysvol to persist between reboots & move variable data out of /usr/pkg/etc/...
On Thu, Jul 30, 2020 at 08:52:43AM +0100, Chavdar Ivanov wrote: > So it seems to me; the peculiarities here are that in both cases / is > a GPT slice and that I have 'log' as a mount option; it was suggested > 'posix1eacls' should be used on its own. I tried with both "posix1eacls" and "log", and that triggers the corruption and crash for me too. the UFS2 extattr code hasn't been updated to have the necessary hooks to integrate with wapbl, I'll take a look at what is needed for that. until that is done, please only use acls without "log". -Chuck
Re: [PATCH] net/samba4: relocate Sysvol to persist between reboots & move variable data out of /usr/pkg/etc/...
On Wed, Jul 29, 2020 at 06:13:03PM +0100, Chavdar Ivanov wrote: > On Wed, 29 Jul 2020 at 08:33, Matthias Petermann wrote: > > > > Hello Chavdar, > > > > Am 28.07.2020 um 18:48 schrieb Chavdar Ivanov: > > > This being a place people are trying samba4 as a DC, I got a > > > repeatable panic on one of the systems I am trying it on, as follows: > > > > > > crash: _kvm_kvatop(0) > > > Crash version 9.99.69, image version 9.99.69. > > > Kernel compiled without options LOCKDEBUG. > > > System panicked: /: bad dir ino 657889 at offset 0: Bad dir (not > > > rounded), reclen=0x2e33, namlen=51, dirsiz=60 <= reclen=11827 <= > > > maxsize=512, flags=0x2005900, entryoffsetinblock=0, dirblksiz=512 > > > > > > Backtrace from time of crash is available. > > > _KERNEL_OPT_NARCNET() at 0 > > > _KERNEL_OPT_DDB_HISTORY_SIZE() at _KERNEL_OPT_DDB_HISTORY_SIZE > > > sys_reboot() at sys_reboot > > > vpanic() at vpanic+0x15b > > > snprintf() at snprintf > > > ufs_lookup() at ufs_lookup+0x518 > > > VOP_LOOKUP() at VOP_LOOKUP+0x42 > > > lookup_once() at lookup_once+0x1a1 > > > namei_tryemulroot() at namei_tryemulroot+0xacf > > > namei() at namei+0x29 > > > vn_open() at vn_open+0x9a > > > do_open() at do_open+0x112 > > > do_sys_openat() at do_sys_openat+0x72 > > > sys_open() at sys_open+0x24 > > > syscall() at syscall+0x26e > > > --- syscall (number 5) --- > > > syscall+0x26e: > > > > > > > > > that still looks like a file system inconsistency. Before the patch from > > Chuck I also had the case several times that a filesystem that was > > apparently repaired with fsck could no longer be trusted. After > > importing the patched kernel, to be on the safe side, I recreated all > > the file systems previously mounted with posix1eacls with newfs. > > Hard that one, as it was the root file system... Anyway, a couple of > fsck's seem to have sorted out this one. how exactly did you run fsck to fix this? the most reliable way is to boot the machine single-user, then run "fsck -fy ..." from the console shell, then run the same fsck command again to make sure that it says that everything is ok, then reboot. if you have done that and are still crashing due to corruption in your root file system, then we still have another bug in the kernel somewhere. > > Presumably fsck is not prepared for the kind of inconsistency, and only > > a newfs can restore a trustworthy initial state. What is the starting > > point for you? Has the file system been created after the patch, or has > > it only been treated with fsck so far? > > I think it may have been created before the patch to the filesystem > code, but before the second version of the samba4 package. > > > > > In any case, I would advise you - if you have not already done so - to > > use a separate partition or LVM volume for the sysvol with its own file > > system, and to mount only this with the posix1eacls option. It seems the > > ACL code still needs a lot of testingh, so at least you can be sure that > > your root filesystem will not be affected. > > As this was running on a XCP-NG guest, I added a small 1GB disk to the > vm, created the filesystem (-O 2) and mounted it on /var/db/samba4. > > I removed the 'posix1eacls' options from the other existing > filesystems and left it only for the one mounted on /var/db/samba4 . > In this case, the provisioning fails with a message that the > filesystem does not support acls - so it perhaps checks the root > filesystem after all. I then re-added this option to /, newfs'd > /var/db/samba4, rebooted and retried the provisioning. This resulted > in a similar to the above panic, this time after perhaps 10 minutes > work of python8 doing database conversion from v1 to v2 - the third > database in the list. As this was seen on the console of the XCP-NG > guest, I took screenshots of the panic, in case someone is interested. yes, I'd like to see the screenshots please. -Chuck
Re: uvm/busy page deadlock in current (related to loading Raspberry Pi 3B+ Wi-Fi firmware, but more of a timing bug with the VM system)
eventually I realized that the aiodone_queue workqueue thread was made redundant long ago... we no longer need this mechanism to hand off any specific iodone processing to a worker thread because these days all iodone processing is done in a (softint) worker thread. I just commited a patch to remove this workqueue thread and associated code. -Chuck On Wed, Feb 12, 2020 at 05:03:39PM -0800, Rob Newberry wrote: > Thanks very much, Chuck -- this patch fixed my problem. > > I noticed you removed a couple of KASSERTs -- shouldn't those be cases be > EVEN MORE true now than they were before? Given what I debugged, I'm > wondering if the asserts would help make sure future code doesn't end up > trying to do something similar in the future... > > Rob
Re: uvm/busy page deadlock in current (related to loading Raspberry Pi 3B+ Wi-Fi firmware, but more of a timing bug with the VM system)
On Thu, Feb 06, 2020 at 04:31:47PM -0800, Rob Newberry wrote:
> Hi.
>
> I spent last weekend -- and a few days this week -- tracking down a problem
> that exists in current.
> I found a workaround, but I don't know what the "proper" fix is.
> Digging through the VM layer and debugging with printfs was slow --
> and it's a boot-time issue, so I had to swap a lot of SD cards back and forth
> :-).
> Hopefully someone here is better at this than me.
>
>
> [analysis...]
good job working your way through all that, this code is pretty complicated.
> 3) Start "aiodone_queue" earlier in the sequence. I don't have a rich enough
> understanding of
> this part of the kernel and user land startup process to know how hard this
> is, or how hacky it is.
this is the right way to fix it. please try the attached patch.
> BTW, I'm ASSUMING that if uvm.aiodone_queue were present, the asynchronous
> completion would somehow
> handle marking the pages as "not busy". But I actually never debugged that
> code path,
> so I can't be sure that's helpful.
right, the "aiodone_queue" workqueue will call uvm_aiodone_worker() on the
buffer,
and bp->b_iodone will have been set to uvm_aio_aiodone, which unbusies the pages
among other things.
-Chuck
Index: src/sys/kern/init_main.c
===
RCS file: /home/chs/netbsd/cvs/src/sys/kern/init_main.c,v
retrieving revision 1.519
diff -u -p -r1.519 init_main.c
--- src/sys/kern/init_main.c28 Jan 2020 16:35:39 - 1.519
+++ src/sys/kern/init_main.c7 Feb 2020 18:14:06 -
@@ -655,6 +655,22 @@ main(void)
sysctl_finalize();
+ /* Create the aiodone daemon kernel thread. */
+ if (workqueue_create(_queue, "aiodoned",
+ uvm_aiodone_worker, NULL, PRI_VM, IPL_NONE, WQ_MPSAFE))
+ panic("fork aiodoned");
+
+ /* Create the pageout daemon kernel thread. */
+ uvm_swap_init();
+ if (kthread_create(PRI_PGDAEMON, KTHREAD_MPSAFE, NULL, uvm_pageout,
+ NULL, NULL, "pgdaemon"))
+ panic("fork pagedaemon");
+
+ /* Create the filesystem syncer kernel thread. */
+ if (kthread_create(PRI_IOFLUSH, KTHREAD_MPSAFE, NULL, sched_sync,
+ NULL, NULL, "ioflush"))
+ panic("fork syncer");
+
/*
* Now that autoconfiguration has completed, we can determine
* the root and dump devices.
@@ -709,22 +725,6 @@ main(void)
ci->ci_schedstate.spc_lastmod = time_second;
}
- /* Create the pageout daemon kernel thread. */
- uvm_swap_init();
- if (kthread_create(PRI_PGDAEMON, KTHREAD_MPSAFE, NULL, uvm_pageout,
- NULL, NULL, "pgdaemon"))
- panic("fork pagedaemon");
-
- /* Create the filesystem syncer kernel thread. */
- if (kthread_create(PRI_IOFLUSH, KTHREAD_MPSAFE, NULL, sched_sync,
- NULL, NULL, "ioflush"))
- panic("fork syncer");
-
- /* Create the aiodone daemon kernel thread. */
- if (workqueue_create(_queue, "aiodoned",
- uvm_aiodone_worker, NULL, PRI_VM, IPL_NONE, WQ_MPSAFE))
- panic("fork aiodoned");
-
/* Wait for final configure threads to complete. */
config_finalize_mountroot();
Index: src/sys/miscfs/genfs/genfs_io.c
===
RCS file: /home/chs/netbsd/cvs/src/sys/miscfs/genfs/genfs_io.c,v
retrieving revision 1.84
diff -u -p -r1.84 genfs_io.c
--- src/sys/miscfs/genfs/genfs_io.c 15 Jan 2020 17:55:44 - 1.84
+++ src/sys/miscfs/genfs/genfs_io.c 7 Feb 2020 18:22:31 -
@@ -606,9 +606,6 @@ genfs_getpages_read(struct vnode *vp, st
if (kva == 0)
return EBUSY;
- if (uvm.aiodone_queue == NULL)
- async = 0;
-
mbp = getiobuf(vp, true);
mbp->b_bufsize = totalbytes;
mbp->b_data = (void *)kva;
@@ -1396,7 +1393,6 @@ genfs_gop_write(struct vnode *vp, struct
UVMPAGER_MAPIN_WRITE | UVMPAGER_MAPIN_WAITOK);
len = npages << PAGE_SHIFT;
- KASSERT(uvm.aiodone_queue != NULL);
error = genfs_do_io(vp, off, kva, len, flags, UIO_WRITE,
uvm_aio_biodone);
@@ -1429,7 +1425,6 @@ genfs_gop_write_rwmap(struct vnode *vp,
UVMPAGER_MAPIN_READ | UVMPAGER_MAPIN_WAITOK);
len = npages << PAGE_SHIFT;
- KASSERT(uvm.aiodone_queue != NULL);
error = genfs_do_io(vp, off, kva, len, flags, UIO_WRITE,
uvm_aio_biodone);
Index: src/sys/uvm/uvm_swap.c
===
RCS file: /home/chs/netbsd/cvs/src/sys/uvm/uvm_swap.c,v
retrieving revision 1.185
diff -u -p -r1.185 uvm_swap.c
--- src/sys/uvm/uvm_swap.c 27 Dec 2019 09:41:51 - 1.185
+++ src/sys/uvm/uvm_swap.c 7 Feb 2020 18:26:49 -
@@ -1778,10 +1778,6 @@ uvm_swap_io(struct vm_page **pps, int st
Re: More build.sh ctf fallout on a linux host
On Sun, Jun 03, 2018 at 10:27:06PM +0300, Valery Ushakov wrote: > # compile libctf/ctf_error.lo > cc -pipe -O2 -DCTF_OLD_VERSIONS > -I/home/uwe/work/netbsd/ro/src/tools/libctf/../compat > -I/home/uwe/work/netbsd/ro/src/tools/libctf/../../external/cddl/osnet/sys > -I/home/uwe/work/netbsd/ro/src/tools/libctf/../../external/cddl/osnet/include > > -I/home/uwe/work/netbsd/ro/src/tools/libctf/../../external/cddl/osnet/dist/head > > -I/home/uwe/work/netbsd/ro/src/tools/libctf/../../external/cddl/osnet/dist/common/ctf > > -I/home/uwe/work/netbsd/ro/src/tools/libctf/../../external/cddl/osnet/dist/lib/libctf/common > > -I/home/uwe/work/netbsd/ro/src/tools/libctf/../../external/cddl/osnet/dist/uts/common > > -I/home/uwe/work/netbsd/ro/src/tools/libctf/../../external/bsd/elftoolchain/dist/libelf > -DHAVE_NBTOOL_CONFIG_H=1 -D_FILE_OFFSET_BITS=64 > -I/home/uwe/work/netbsd/build/tools/include/compat > -I/home/uwe/work/netbsd/ro/src/tools/compat -DHAVE_NBTOOL_CONFIG_H=1 > -D_FILE_OFFSET_BITS=64 -I/home/uwe/work/netbsd/build/tools/include > -I/home/uwe/work/netbsd/build/tools/include/nbinclude -c -o ctf_error.lo.o > /home/uwe/work/netbsd/ro/src/tools/libctf/../../external/cddl/osnet/dist/common/ctf/ctf_error.c > In file included from > /home/uwe/work/netbsd/ro/src/tools/libctf/../compat/compat_defs.h:75:0, > from > /home/uwe/work/netbsd/build/tools/include/compat/nbtool_config.h:882, > from > /home/uwe/work/netbsd/ro/src/tools/libctf/../../external/cddl/osnet/dist/common/ctf/ctf_error.c:23: > /home/uwe/work/netbsd/ro/src/tools/libctf/../../external/cddl/osnet/dist/uts/common/rpc/types.h:57:9: > error: unknown type name 'u_longlong_t' > typedef u_longlong_t ulonglong_t; > ^ I was afraid that might happen... which distribution and version of linux was that on? the one I did my fix for linux on was fedora 26. -Chuck
Re: Cannot vfork (Resource temporarily unavailable)
On Sat, Dec 02, 2017 at 04:35:17PM +, Patrick Welche wrote: > When trying to build pkgsrc/net/wget on -current/amd64, I see > > --- wget.info --- > /tmp/pkgsrc/net/wget/work.x86_64/.tools/bin/makeinfo: Cannot vfork (Resource > temporarily unavailable) > --- ./wget.info --- > /tmp/pkgsrc/net/wget/work.x86_64/.tools/bin/makeinfo: Cannot vfork (Resource > temporarily unavailable) > > (or /usr/pkgsrc/mk/gnu-config/missing) > > There isn't much going on, > > CPU states: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle > > and that is when running "make replace" in the directory for the nth time, > so hard to see the lack of resource. > > Could this be an easier to repeat version of the "out of swap" kill "sh" > on boot problem? I tried this and I get the same failure. I also get this on the console: proc: table is full - increase kern.maxproc or NPROC do you get that message too? if so, this is different from the out-of-swap thing. it looks like a fork-bomb bug in one of the pkgsrc wrapper scripts. -Chuck
Re: X11 on Lenovo T430
On Tue, Dec 23, 2014 at 04:59:07PM +0100, Havard Eidnes wrote: Hi, I'm running netbsd-7 code on my new Lenovo T430 laptop. I'm using code from November 27 at the moment, with the DRM/KMS kernel, and there are a few glitches: 1) Sometimes the rendering of images e.g. in a web browser (firefox) is mangled / interlaced (not sure how to best describe it). Sometimes causing a re-paint fixes the glitch, sometimes it doesn't (by the looks of it). I get this as well: http://ftp.netbsd.org/pub/NetBSD/misc/chs/drm2/firefox-drm2-intel965.png that partcular instance was immediately after starting firefox right after a reboot, but usually this problem doesn't happen. there's not much that could be different from one time to the next, the thing that seem most likely to be relevant is alignment of the data in memory. 2) Sometimes X11 appears to hang, and I get these messages in my kernel message buffer: drm: stuck on blitter ring drm: GPU HANG: ecode 2:0x87d0fff5, reason: Ring hung, action: reset drm: GPU hangs can indicate a bug anywhere in the entire gfx stack, including userspace. drm: Please file a _new_ bug report on bugs.freedesktop.org against DRI - DRM/Intel drm: drm/i915 developers can then reassign to the right component if it's not a kernel issue. drm: The gpu crash dump is required to analyze gpu hangs, so please always attach it. drm: GPU crash dump saved to /sys/class/drm/card0/error i915drmkms0: interrupting at ioapic0 pin 16 (pci0@pci::00:02.0) drm: Enabling RC6 states: RC6 on, RC6p on, RC6pp off yea, there are quite a few problems still at this point. the only one I can consistently trivially trigger is by running x11perf -movetree. this causes the screen to go blank and the X server to print: (EE) intel(0): Failed to submit batch buffer, expect rendering corruption: Invalid argument. and this is logged in /var/log/messages: Dec 27 18:26:10 tp2 /netbsd: drm: GPU HANG: ecode -1:0x, reason: Kicking stuck wait on render ring, action: continue Dec 27 18:26:10 tp2 /netbsd: drm: GPU hangs can indicate a bug anywhere in the entire gfx stack, including userspace. Dec 27 18:26:10 tp2 /netbsd: drm: Please file a _new_ bug report on bugs.freedesktop.org against DRI - DRM/Intel Dec 27 18:26:10 tp2 /netbsd: drm: drm/i915 developers can then reassign to the right component if it's not a kernel issue. Dec 27 18:26:10 tp2 /netbsd: drm: The gpu crash dump is required to analyze gpu hangs, so please always attach it. Dec 27 18:26:10 tp2 /netbsd: drm: GPU crash dump saved to /sys/class/drm/card0/error Dec 27 18:26:12 tp2 /netbsd: drm: GPU HANG: ecode -1:0x, reason: Kicking stuck wait on render ring, action: continue Dec 27 18:26:22 tp2 syslogd[541]: last message repeated 5 times Dec 27 18:26:22 tp2 /netbsd: drm: no progress on render ring Dec 27 18:26:22 tp2 /netbsd: drm: GPU HANG: ecode -1:0x, reason: Ring hung, action: reset Dec 27 18:26:23 tp2 /netbsd: DRM error in i915_reset: Failed to reset chip: -60 the only way I know to recover after that is rebooting. I haven't been gotten anywhere investigating this yet. this is all on a thinkpad t61 with intel 965 graphics. -Chuck
Re: Re: compat linux exec arguments weirdness
On Sun, Feb 09, 2014 at 10:52:24AM +0100, Onno van der Linden wrote: I wrote: $ /emul/linux/usr/bin/uname -bagger : invalid option -- 'b' Try ' --help' for more information. That error output should have been something like -- uname: invalid option -- 'b' -- Try 'uname --help' for more information So, it has nothing to do with the shell as I first thought but argv[0] getting overwritten somewhere in the linux emul exec code path. Reverting sys/compat/linux/common/linux_exec_elf32.c back to the previous version fixes things for me: -- $ /emul/linux/usr/bin/uname -bagger -- /emul/linux/usr/bin/uname: invalid option -- 'b' -- Try '/emul/linux/usr/bin/uname --help' for more information. Looks like the implementation of AT_RANDOM messes up the argument stack (at least for the elf32 case, can't test the amd64 case myself). this should be fixed now, please update and give it a try. -Chuck
Re: Marvell 88E1149 PHY working with makphy?
On Mon, Jul 15, 2013 at 05:01:27PM -0400, Thor Lancelot Simon wrote: On Mon, Jul 15, 2013 at 01:01:51PM -0500, John D. Baker wrote: I discovered all this today and enabled ukphy to get more information and perhaps make it work better. That confirmed which PHY I had. Then I flipped the #if test in makphy.c to restore the 88E1149 to the device list and my machine works fine with it. I know the comment in the commit disabling this specific device says ukphy works fine with it. Why can't you just use ukphy with this PHY? Does it not work for you? I don't think it's a good idea to re-enable it in the makphy driver unless a user who could reproduce the original problem says it's fixed. I'm the one who disabled it, and it still doesn't work for me on the box where it didn't work before. if someone would like to add a check to allow enabling this via a config option, that would be ok, but please keep the default as disabled. -Chuck
