On 2020/12/01 10:32, Sebastien Marie wrote:
> On Tue, Dec 01, 2020 at 06:59:22AM +0100, Sebastien Marie wrote:
> > On Mon, Nov 30, 2020 at 11:14:46PM +0000, Stuart Henderson wrote:
> > > Thought I'd try a WITNESS kernel to see if that gives any clues about
> > > what's going on with my APU crashing all over the place (long shot but
> > > I got bored with trying different older kernels..)
> > >
> > > I see these from time to time (one during netstart, and another 4 in
> > > 15 mins uptime), anyone know if they're important?
> >
> > this check ("lock_object uninitialized") was recently added in witness.
> >
> > it means that the rwlock was uninitialized (the witness flag
> > LO_INITIALIZED isn't present whereas it should) when witness check the
> > lock.
> >
> > it could be:
> > - someone omits to call rw_init() or RWLOCK_INITIALIZER() (possible bug if
> > memory isn't zero)
> > - the struct has been zeroed (possible bug too)
> >
> > > witness: lock_object uninitialized: 0xffff800000bcf0d8
> > > Starting stack trace...
> > > witness_checkorder(ffff800000bcf0d8,9,0) at witness_checkorder+0xab
> > > rw_enter_write(ffff800000bcf0c8) at rw_enter_write+0x43
> > > noise_remote_decrypt(ffff800000bcea48,978dc3d,0,fffffd805e22cb7c,10) at
> > > noise_remote_decrypt+0x135
> > > wg_decap(ffff80000054a000,fffffd8061835200) at wg_decap+0xda
> > > wg_decap_worker(ffff80000054a000) at wg_decap_worker+0x7a
> > > taskq_thread(ffff80000012d900) at taskq_thread+0x9f
> > > end trace frame: 0x0, count: 251
> > > End of stack trace.
> >
> > from the trace, the sole rw_enter_write() usage in noise_remote_decrypt() is
> >
> > rw_enter_write(&r->r_keypair_lock)
> >
> > but I am seeing few rw_init() on r_keypair_lock. I will see if I could
> > see the source of the problem.
> >
>
> Jason, Matt,
>
> sthen@ told me that the same lock is reported several times (exactly,
> two locks are reported several times: lock1, lock2, lock1, lock2...)
>
> witness(4) reports when a lock doesn't have LO_INITIALIZED flag set in
> internal part of `struct rwlock'. Next it sets it, and skip futher
> analysis for this particular lock.
>
> if the same lock is reported several times, it means the memory is
> zeroed (and so the flag removed). it could be intented or not. but in
> all cases, a rwlock should be properly initialized with rw_init() or
> RWLOCK_INITIALIZER() before use.
>
> I don't have enough experiency with wg(4) stuff to well understand the
> issue. in wg_decap(), the peer is get from a `struct wg_tag` (from
> wg_tag_get()). If i didn't mess myself, the p_remote could come from
> wg_send_initiation() or wg_send_response(). but for both, it comes
> from wg_peer_create(), and p_remote is initialized with
> noise_remote_init() (and so lock have proper rw_init()).
>
> do you have idea on the cause of the lost of the rwlock flag ?
>
> if you want to test it with witness(4) enabled, you will need to build
> a kernel with "option WITNESS" in config file. you could uncomment it
> from sys/arch/amd64/conf/GENERIC.MP, and run make config, make clean,
> make
>
> Thanks.
> --
> Sebastien Marie
>
Config of the machine is simple; one wg peer, the peer is on a fixed
internet address, this machine is natted but the address won't change.
It isn't routing, wg is only to reach the machine itself.
Here's a more complete console log but I don't think it adds anything
to the above.
pf enabled
starting network
em0: no link....
em0: 192.168.42.220 lease accepted from 192.168.42.21 (00:1b:21:9b:68:10)
add net 192.168.40.0/21: gateway 192.168.43.33
add net 195.95.187.0/24: gateway 192.168.43.33
reordering libraries:witness: lock_object uninitialized: 0xffff800000bcf580
Starting stack trace...
witness_checkorder(ffff800000bcf580,9,0) at witness_checkorder+0xab
rw_enter_write(ffff800000bcf570) at rw_enter_write+0x43
noise_remote_decrypt(ffff800000bcea48,3f71de8d,0,fffffd80073918fc,10) at
noise_remote_decrypt+0x135
wg_decap(ffff80000054a000,fffffd8067faf900) at wg_decap+0xda
wg_decap_worker(ffff80000054a000) at wg_decap_worker+0x7a
taskq_thread(ffff80000012d900) at taskq_thread+0x9f
end trace frame: 0x0, count: 251
End of stack trace.
done.
starting early daemons: syslogd pflogd ntpd.
starting RPC daemons:.
savecore: no core dump
checking quotas: done.
clearing /tmp
kern.securelevel: 0 -> 1
creating runtime link editor directory cache.
preserving editor files.
starting network daemons: sshd smtpd sndiod.
starting package daemons: conserver lldpd.
starting local daemons: cron.
Mon Nov 30 22:59:18 GMT 2020
reorder_kernel: failed -- see /usr/share/relink/kernel/WITNESS.MP/relink.log
OpenBSD/amd64 (cons1) (tty00)
login: witness: lock_object uninitialized: 0xffff800000bcf0d8
Starting stack trace...
witness_checkorder(ffff800000bcf0d8,9,0) at witness_checkorder+0xab
rw_enter_write(ffff800000bcf0c8) at rw_enter_write+0x43
noise_remote_decrypt(ffff800000bcea48,c4992785,0,fffffd80073c89bc,10) at
noise_remote_decrypt+0x135
wg_decap(ffff80000054a000,fffffd805f53ac00) at wg_decap+0xda
wg_decap_worker(ffff80000054a000) at wg_decap_worker+0x7a
taskq_thread(ffff80000012d900) at taskq_thread+0x9f
end trace frame: 0x0, count: 251
End of stack trace.
witness: lock_object uninitialized: 0xffff800000bcf580
Starting stack trace...
witness_checkorder(ffff800000bcf580,9,0) at witness_checkorder+0xab
rw_enter_write(ffff800000bcf570) at rw_enter_write+0x43
noise_remote_decrypt(ffff800000bcea48,45419020,0,fffffd80073c89bc,10) at
noise_remote_decrypt+0x135
wg_decap(ffff80000054a000,fffffd80751c3c00) at wg_decap+0xda
wg_decap_worker(ffff80000054a000) at wg_decap_worker+0x7a
taskq_thread(ffff80000012d900) at taskq_thread+0x9f
end trace frame: 0x0, count: 251
End of stack trace.
witness: lock_object uninitialized: 0xffff800000bcf0d8
Starting stack trace...
witness_checkorder(ffff800000bcf0d8,9,0) at witness_checkorder+0xab
rw_enter_write(ffff800000bcf0c8) at rw_enter_write+0x43
noise_remote_decrypt(ffff800000bcea48,978dc3d,0,fffffd805e22cb7c,10) at
noise_remote_decrypt+0x135
wg_decap(ffff80000054a000,fffffd8061835200) at wg_decap+0xda
wg_decap_worker(ffff80000054a000) at wg_decap_worker+0x7a
taskq_thread(ffff80000012d900) at taskq_thread+0x9f
end trace frame: 0x0, count: 251
End of stack trace.
OpenBSD/amd64 (cons1) (tty00)
login: witness: lock_object uninitialized: 0xffff800000bcf580
Starting stack trace...
witness_checkorder(ffff800000bcf580,9,0) at witness_checkorder+0xab
rw_enter_write(ffff800000bcf570) at rw_enter_write+0x43
noise_remote_decrypt(ffff800000bcea48,a6aee19a,0,fffffd805e2299fc,10) at
noise_remote_decrypt+0x135
wg_decap(ffff80000054a000,fffffd8075fb5300) at wg_decap+0xda
wg_decap_worker(ffff80000054a000) at wg_decap_worker+0x7a
taskq_thread(ffff80000012d900) at taskq_thread+0x9f
end trace frame: 0x0, count: 251
End of stack trace.
