Re: installer: support encryption with key disks
On Mon, Oct 16, 2023 at 07:46:10PM +, Klemens Nanni wrote:
> On Mon, Sep 04, 2023 at 09:57:40PM +, Klemens Nanni wrote:
> > Extend the yes/no question to no/passphrase/keydisk and have users pick an
> > existing, preformated RAID partition; no support (yet) for creating one.
> >
> > Thanks to how ask_which() works, users can always say 'done' to land back
> > at question to either skip crypto or use a passphrase instead.
> >
> > All code remains contained behind interactive non-default installations.
> > Code is straight forward, I've not been able to break it; rest unchanged.
> >
> > Example install with root disk sd0 and ready-to-use key disk sd1:
> >
> > Available disks are: sd0 sd1.
> > Which disk is the root disk? ('?' for details) [sd0]
> > Encrypt the root disk with a (p)assphrase or (k)eydisk? [no] k
> > Available disks are: sd1.
> > Which disk contains the key disk? (or 'done') [sd1]
> > Available sd1 partitions are: a.
> > Which sd1 partition is the key disk? (or 'done') [a]
> >
> > Configuring the crypto chunk sd0...
> >
> > No valid MBR or GPT.
> > Use (W)hole disk MBR, whole disk (G)PT or (E)dit? [whole]
> > Setting OpenBSD MBR partition to whole sd0...done.
> > sd2 at scsibus2 targ 1 lun 0:
> > sd2: 1023MB, 512 bytes/sector, 2096560 sectors
> >
> > Configuring the root disk sd2...
> >
> > No valid MBR or GPT.
> > Use (W)hole disk MBR, whole disk (G)PT or (E)dit? [whole]
> >
> >
> > Feedback? OK?
This seems to work well for me and the implementation looks reasonable.
I think we want to update INSTALL with some extra docs as well, likely
pointing folks to how to create a keydisk.
OK afresh1@
>
> Ping.
>
> Index: install.sub
> ===
> RCS file: /cvs/src/distrib/miniroot/install.sub,v
> retrieving revision 1.1255
> diff -u -p -r1.1255 install.sub
> --- install.sub 21 Aug 2023 14:33:55 - 1.1255
> +++ install.sub 16 Oct 2023 19:36:55 -
> @@ -3074,8 +3074,32 @@ do_autoinstall() {
> exec reboot
> }
>
> +# Chose an existing partition as key disk and set global $KEYDISK on success,
> +# otherwise return non-zero.
> +pick_keydisk() {
> + KEYDISK=
> + local _disk _label
> +
> + ask_which disk 'contains the key disk' '$(rmel $ROOTDISK $(get_dkdevs))'
> + [[ $resp == done ]] && return 1
> + _disk=$resp
> +
> + make_dev $_disk
> + if disklabel $_disk 2>/dev/null | ! grep -qw RAID; then
> + echo "$_disk must contain a RAID partition."
> + return 1
> + fi
> +
> + ask_which "$_disk partition" 'is the key disk' \
> + "\$(disklabel $_disk 2>/dev/null |
> + sed -En 's/^ ([a-p]):.*RAID.*$/\1/p')"
> + [[ $resp == done ]] && return 1
> + _label=$resp
> + KEYDISK=$_disk$_label
> +}
> +
> encrypt_root() {
> - local _chunk=$ROOTDISK
> + local _args _chunk=$ROOTDISK
>
> [[ $MDBOOTSR == y ]] || return
>
> @@ -3088,13 +3112,30 @@ encrypt_root() {
> # e.g. auto-assembled at boot or done in (S)hell.
> [[ -z $(get_softraid_volumes) ]] || return
>
> - ask_yn 'Encrypt the root disk with a passphrase?' || return
> + while :; do
> + ask 'Encrypt the root disk with a (p)assphrase or (k)eydisk?' no
> + case $resp in
> + # Retry on failure to allow passphrase or skip.
> + [kK]*)
> + pick_keydisk || continue
> + _args=-k$KEYDISK
> + break
> + ;;
> + # Do nothing, bioctl(8) will handle the passphrase.
> + [pP]*) break
> + ;;
> + [nN]*) return
> + ;;
> + *) echo "'$resp' is not a valid choice."
> + ;;
> + esac
> + done
>
> echo "\nConfiguring the crypto chunk $_chunk...\n"
> md_prep_fdisk $_chunk
> echo 'RAID *' | disklabel -w -A -T- $_chunk
>
> - bioctl -Cforce -cC -l${_chunk}a softraid0 >/dev/null
> + bioctl -Cforce -cC -l${_chunk}a $_args softraid0 >/dev/null
>
> # No volumes existed before asking, but we just created one.
> ROOTDISK=$(get_softraid_volumes)
>
--
andrew
($do || !$do) && undef($try) ; # Master of Perl, Yoda is. H?
Re: better fix for Term::Cap
On Wed, Oct 18, 2023 at 10:25:59AM +0200, Marc Espie wrote:
> Instead of an archaic limit, just keep track of tc's.
> This is actually slightly faster, because the termcap links is a tree,
> not a list.
I like it, but I don't want to carry this patch against upstream. I
will happily pull in this change from there as soon as they merge it
though.
Changing the 32 to 64 is an easy patch to keep up on and to solve merge
conflicts with when upgrading. This larger change is much less so.
That said, there is so much in this file that could be improved, why
stop here?
>
> Please test.
>
> Index: Cap.pm
> ===
> RCS file: /cvs/src/gnu/usr.bin/perl/cpan/Term-Cap/Cap.pm,v
> retrieving revision 1.3
> diff -u -p -r1.3 Cap.pm
> --- Cap.pm18 Oct 2023 01:49:26 - 1.3
> +++ Cap.pm18 Oct 2023 08:24:54 -
> @@ -280,7 +280,7 @@ sub Tgetent
>
> $first = 0;# first entry (keeps term name)
>
> -$max = 64; # max :tc=...:'s
> +my $seen = {}; # keep track of :tc=...:s
>
> if ($entry)
> {
> @@ -291,6 +291,7 @@ sub Tgetent
> if ( $entry =~ s/:tc=([^:]+):/:/ )
> {
> $tmp_term = $1;
> + $seen->{$tmp_term} = 1;
>
> # protect any pattern metacharacters in $tmp_term
> $termpat = $tmp_term;
> @@ -332,10 +333,7 @@ sub Tgetent
> }
> else
> {
> -
> # do the same file again
> -# prevent endless recursion
> -$max-- || croak "failed termcap loop at $tmp_term";
> $state = 1;# ok, maybe do a new file next time
> }
>
> @@ -345,11 +343,20 @@ sub Tgetent
> close TERMCAP;
>
> # If :tc=...: found then search this file again
> -$entry =~ s/:tc=([^:]+):/:/ && ( $tmp_term = $1, $state = 2 );
> +while ($entry =~ s/:tc=([^:]+):/:/) {
> + $tmp_term = $1;
> + if ($seen->{$tmp_term}) {
> + # XXX first version of this croaked, but we can actually
> + # get several intermediate entries with the same tc !
> + next;
> + }
> + $seen->{$tmp_term} = 1;
> + $state = 2;
>
> -# protect any pattern metacharacters in $tmp_term
> -$termpat = $tmp_term;
> -$termpat =~ s/(\W)/\\$1/g;
> + # protect any pattern metacharacters in $tmp_term
> + $termpat = $tmp_term;
> + $termpat =~ s/(\W)/\\$1/g;
> + }
> }
>
> croak "Can't find $term" if $entry eq '';
>
--
andrew
"Programming today is a race between software engineers striving to
build bigger and better idiot-proof programs, and the Universe
trying to produce bigger and better idiots. So far, the Universe is
winning." -- Rich Cook
Re: pkg_add: No progress meter: failed termcap loop
On Tue, Oct 17, 2023 at 06:34:58PM -0700, Andrew Hewus Fresh wrote:
> On Tue, Oct 17, 2023 at 06:13:30PM -0700, Andrew Hewus Fresh wrote:
> > On Tue, Oct 17, 2023 at 05:10:44PM -0700, Greg Steuck wrote:
> > > I just got myself a fresh snapshot with libncurses.so.15.0:
> > > OpenBSD 7.4-current (GENERIC.MP) #1409: Tue Oct 17 17:08:49 MDT 2023
> > >
> > > I get some unhappiness now:
> > > # pkg_add -ui
> > > No progress meter: failed termcap loop
> > > quirks-6.161 signed on 2023-10-16T11:16:03Z
> > >
> > > Fallout from the ncurses upgrade?
And this is committed with an upstream PR created.
https://github.com/jonathanstowe/Term-Cap/pull/17
> Index: gnu/usr.bin/perl/cpan/Term-Cap/Cap.pm
> ===
> RCS file: /cvs/src/gnu/usr.bin/perl/cpan/Term-Cap/Cap.pm,v
> retrieving revision 1.2
> diff -u -p -r1.2 Cap.pm
> --- gnu/usr.bin/perl/cpan/Term-Cap/Cap.pm 5 Feb 2017 00:32:03 -
> 1.2
> +++ gnu/usr.bin/perl/cpan/Term-Cap/Cap.pm 18 Oct 2023 01:25:11 -
> @@ -280,7 +280,7 @@ sub Tgetent
>
> $first = 0;# first entry (keeps term name)
>
> -$max = 32; # max :tc=...:'s
> +$max = 64; # max :tc=...:'s
>
> if ($entry)
> {
>
Re: pkg_add: No progress meter: failed termcap loop
On Tue, Oct 17, 2023 at 06:34:58PM -0700, Andrew Hewus Fresh wrote:
> On Tue, Oct 17, 2023 at 06:13:30PM -0700, Andrew Hewus Fresh wrote:
> > On Tue, Oct 17, 2023 at 05:10:44PM -0700, Greg Steuck wrote:
> > > I just got myself a fresh snapshot with libncurses.so.15.0:
> > > OpenBSD 7.4-current (GENERIC.MP) #1409: Tue Oct 17 17:08:49 MDT 2023
> > >
> > > I get some unhappiness now:
> > > # pkg_add -ui
> > > No progress meter: failed termcap loop
> > > quirks-6.161 signed on 2023-10-16T11:16:03Z
> > >
> > > Fallout from the ncurses upgrade?
> >
> > Seems likely. I can reproduce with this code that's similar to
> > what OpenBSD::ProgressMeter::Term does:
> >
> > $ perl -MPOSIX -MTerm::Cap -e 'Term::Cap->Tgetent({ OSPEED =>
> > POSIX::Termios->new->getospeed })'
> > failed termcap loop at att610+cvis at -e line 1.
> >
> > or even just:
> >
> > $ perl -MTerm::Cap -e 'Term::Cap->Tgetent'
> >
> > failed termcap loop at att610+cvis at -e line 1.
> >
> > Wow though, the only hits google has for that "failed termcap loop" is
> > the file it is in so incredibly impressive.
>
> It appears 20 years ago nobody expected this complicated an
> /etc/termcap. This patch seems to fix the glitch.
>
> It has been 32 since at least it was moved back from SVN to CVS in 2004
> which is when git history stops.
> https://github.com/jonathanstowe/Term-Cap/blob/3a1193a8c5457d0ca77ea43d1be73712303a359c/Cap.pm#L254
Looking a bit harder, before that upstream for Term::Cap was perl core.
So that 32 limit dates back to 1995 and perl 5.001, I guess thinking 28
years into the future was enough.
https://github.com/Perl/perl5/commit/748a93069b3d16374a9859d1456065dd3ae11394#diff-0961d7f68bf68cb479f1fbd33f77f275bab83bfab124c64ab82b20ec5e9d6d87R77
> (I only needed to change this to 36, but how future proof is that?)
>
> Comments, OK?
>
>
> Index: gnu/usr.bin/perl/cpan/Term-Cap/Cap.pm
> ===
> RCS file: /cvs/src/gnu/usr.bin/perl/cpan/Term-Cap/Cap.pm,v
> retrieving revision 1.2
> diff -u -p -r1.2 Cap.pm
> --- gnu/usr.bin/perl/cpan/Term-Cap/Cap.pm 5 Feb 2017 00:32:03 -
> 1.2
> +++ gnu/usr.bin/perl/cpan/Term-Cap/Cap.pm 18 Oct 2023 01:25:11 -
> @@ -280,7 +280,7 @@ sub Tgetent
>
> $first = 0;# first entry (keeps term name)
>
> -$max = 32; # max :tc=...:'s
> +$max = 64; # max :tc=...:'s
>
> if ($entry)
> {
>
--
andrew
Hey, I think I see a barn up ahead.
-- The American Astronaut
Re: pkg_add: No progress meter: failed termcap loop
On Tue, Oct 17, 2023 at 06:13:30PM -0700, Andrew Hewus Fresh wrote:
> On Tue, Oct 17, 2023 at 05:10:44PM -0700, Greg Steuck wrote:
> > I just got myself a fresh snapshot with libncurses.so.15.0:
> > OpenBSD 7.4-current (GENERIC.MP) #1409: Tue Oct 17 17:08:49 MDT 2023
> >
> > I get some unhappiness now:
> > # pkg_add -ui
> > No progress meter: failed termcap loop
> > quirks-6.161 signed on 2023-10-16T11:16:03Z
> >
> > Fallout from the ncurses upgrade?
>
> Seems likely. I can reproduce with this code that's similar to
> what OpenBSD::ProgressMeter::Term does:
>
> $ perl -MPOSIX -MTerm::Cap -e 'Term::Cap->Tgetent({ OSPEED =>
> POSIX::Termios->new->getospeed })'
> failed termcap loop at att610+cvis at -e line 1.
>
> or even just:
>
> $ perl -MTerm::Cap -e 'Term::Cap->Tgetent'
>
> failed termcap loop at att610+cvis at -e line 1.
>
> Wow though, the only hits google has for that "failed termcap loop" is
> the file it is in so incredibly impressive.
It appears 20 years ago nobody expected this complicated an
/etc/termcap. This patch seems to fix the glitch.
It has been 32 since at least it was moved back from SVN to CVS in 2004
which is when git history stops.
https://github.com/jonathanstowe/Term-Cap/blob/3a1193a8c5457d0ca77ea43d1be73712303a359c/Cap.pm#L254
(I only needed to change this to 36, but how future proof is that?)
Comments, OK?
Index: gnu/usr.bin/perl/cpan/Term-Cap/Cap.pm
===
RCS file: /cvs/src/gnu/usr.bin/perl/cpan/Term-Cap/Cap.pm,v
retrieving revision 1.2
diff -u -p -r1.2 Cap.pm
--- gnu/usr.bin/perl/cpan/Term-Cap/Cap.pm 5 Feb 2017 00:32:03 -
1.2
+++ gnu/usr.bin/perl/cpan/Term-Cap/Cap.pm 18 Oct 2023 01:25:11 -
@@ -280,7 +280,7 @@ sub Tgetent
$first = 0;# first entry (keeps term name)
-$max = 32; # max :tc=...:'s
+$max = 64; # max :tc=...:'s
if ($entry)
{
Re: pkg_add: No progress meter: failed termcap loop
On Tue, Oct 17, 2023 at 05:10:44PM -0700, Greg Steuck wrote:
> I just got myself a fresh snapshot with libncurses.so.15.0:
> OpenBSD 7.4-current (GENERIC.MP) #1409: Tue Oct 17 17:08:49 MDT 2023
>
> I get some unhappiness now:
> # pkg_add -ui
> No progress meter: failed termcap loop
> quirks-6.161 signed on 2023-10-16T11:16:03Z
>
> Fallout from the ncurses upgrade?
Seems likely. I can reproduce with this code that's similar to
what OpenBSD::ProgressMeter::Term does:
$ perl -MPOSIX -MTerm::Cap -e 'Term::Cap->Tgetent({ OSPEED =>
POSIX::Termios->new->getospeed })'
failed termcap loop at att610+cvis at -e line 1.
or even just:
$ perl -MTerm::Cap -e 'Term::Cap->Tgetent'
failed termcap loop at att610+cvis at -e line 1.
Wow though, the only hits google has for that "failed termcap loop" is
the file it is in so incredibly impressive.
l8rZ,
--
andrew
The programmer's national anthem is 'GH!!'.
Re: Don't lock package db if fw_update(8) is not installing firmware
On Sat, Oct 14, 2023 at 09:02:41PM +0200, Marc Espie wrote:
> On Sat, Oct 14, 2023 at 11:24:08AM -0700, Andrew Hewus Fresh wrote:
> > While testing another patch, I noticed that fw_update will lock the
> > package database even if it's just downloading and not touching the
> > installed packages.
> >
> > Currently we do _read_ the existing firmware as part of detecting what
> > we might need to download or upgrade without locking. I'm unsure if
> > that is an issue.
> >
> > Comments, OK?
> >
> >
> >
> > Index: fw_update.sh
> > ===
> > RCS file: /cvs/src/usr.sbin/fw_update/fw_update.sh,v
> > retrieving revision 1.51
> > diff -u -p -r1.51 fw_update.sh
> > --- fw_update.sh14 Oct 2023 18:10:47 - 1.51
> > +++ fw_update.sh14 Oct 2023 18:12:08 -
> > @@ -593,7 +593,7 @@ kept=''
> > unregister=''
> >
> > if [ "${devices[*]:-}" ]; then
> > - lock_db
> > + "$INSTALL" && lock_db
> > for f in "${devices[@]}"; do
> > d="$( firmware_devicename "$f" )"
> >
> >
> >
> the package tools have two lock modes: read-only and read-write.
> If you want to match them, you want lock read-only for this.
Everywhere but during detection "installed_firmware" is called with the
read-write lock.
I think using that read-only lock during detection would look like this.
Index: fw_update.sh
===
RCS file: /cvs/src/usr.sbin/fw_update/fw_update.sh,v
retrieving revision 1.51
diff -u -p -r1.51 fw_update.sh
--- fw_update.sh14 Oct 2023 18:10:47 - 1.51
+++ fw_update.sh14 Oct 2023 19:39:25 -
@@ -69,7 +69,7 @@ cleanup() {
fi
[ "${FTPPID:-}" ] && kill -TERM -"$FTPPID" 2>/dev/null
- [ "${LOCKPID:-}" ] && kill -TERM -"$LOCKPID" 2>/dev/null
+ unlock_db
[ "${FWPKGTMP:-}" ] && rm -rf "$FWPKGTMP"
"$REMOVE_LOCALSRC" && rm -rf "$LOCALSRC"
[ -e "$CFILE" ] && [ ! -s "$CFILE" ] && rm -f "$CFILE"
@@ -259,7 +259,7 @@ lock_db() {
[ -e /usr/bin/perl ] || return 0
set -o monitor
- perl <<-'EOL' |&
+ _shared=${1:-0} perl <<-'EOL' |&
no lib ('/usr/local/libdata/perl5/site_perl');
use v5.36;
use OpenBSD::PackageInfo qw< lock_db >;
@@ -267,7 +267,7 @@ lock_db() {
$|=1;
$0 = "fw_update: lock_db";
- lock_db(0);
+ lock_db($ENV{_shared});
say $$;
@@ -284,6 +284,20 @@ EOL
return 0
}
+unlock_db() {
+ [ "${LOCKPID:-}" ] || return 0
+ local _seconds=0
+
+ kill -TERM -"$LOCKPID"
+
+ while kill -0 -"$LOCKPID" 2>/dev/null; do
+ sleep 1
+ ((_seconds++ > 5)) && kill -KILL -"$LOCKPID"
+ done
+
+ unset LOCKPID
+}
+
installed_firmware() {
local _pre="$1" _match="$2" _post="$3" _firmware _fw
set -sA _firmware -- $(
@@ -581,7 +595,9 @@ if [ "${devices[*]:-}" ]; then
"$ALL" && warn "Cannot use -a and devices/files" && usage
else
((VERBOSE > 1)) && echo -n "Detect firmware ..."
+ lock_db 1
set -sA devices -- $( detect_firmware )
+ unlock_db
((VERBOSE > 1)) &&
{ [ "${devices[*]:-}" ] && echo " found." || echo " done." ; }
fi
@@ -593,7 +609,7 @@ kept=''
unregister=''
if [ "${devices[*]:-}" ]; then
- lock_db
+ "$INSTALL" && lock_db
for f in "${devices[@]}"; do
d="$( firmware_devicename "$f" )"
Re: Improve fw_update(8) output on errors somewhat
Here's the latest iteration of the fw_update output cleanup.
The big change here is improved output if running fw_update while the
package database is locked, pkg_add -u in one shell and fw_update in
another.
I'll be testing this a bit more and hoping to commit it after
snapshots start rolling out again.
Comments, test results, OK?
Index: fw_update.sh
===
RCS file: /cvs/src/usr.sbin/fw_update/fw_update.sh,v
retrieving revision 1.51
diff -u -p -r1.51 fw_update.sh
--- fw_update.sh14 Oct 2023 18:10:47 - 1.51
+++ fw_update.sh14 Oct 2023 19:09:56 -
@@ -60,7 +60,8 @@ cleanup() {
if [ -d "$FD_DIR" ]; then
echo "" >&"$STATUS_FD"
- exec 4>&-
+ ((STATUS_FD == 3)) && exec 3>&-
+ ((WARN_FD == 4)) && exec 4>&-
[ -s "$FD_DIR/status" ] && cat "$FD_DIR/status"
[ -s "$FD_DIR/warn" ] && cat "$FD_DIR/warn" >&2
@@ -107,21 +108,24 @@ spin() {
fetch() {
local _src="${FWURL}/${1##*/}" _dst=$1 _user=_file _exit _error=''
+ local _ftp_errors="$FD_DIR/ftp_errors"
+ rm -f "$_ftp_errors"
# The installer uses a limited doas(1) as a tiny su(1)
set -o monitor # make sure ftp gets its own process group
(
_flags=-vm
case "$VERBOSE" in
- 0|1) _flags=-VM ;;
+ 0|1) _flags=-VM ; exec 2>"$_ftp_errors" ;;
2) _flags=-Vm ;;
esac
+
if [ -x /usr/bin/su ]; then
exec /usr/bin/su -s /bin/ksh "$_user" -c \
- "/usr/bin/ftp -N '${0##/}' -D 'Get/Verify' $_flags -o-
'$_src'" > "$_dst"
+ "/usr/bin/ftp -N error -D 'Get/Verify' $_flags -o- '$_src'"
> "$_dst"
else
exec /usr/bin/doas -u "$_user" \
- /usr/bin/ftp -N "${0##/}" -D 'Get/Verify' $_flags -o-
"$_src" > "$_dst"
+ /usr/bin/ftp -N error -D 'Get/Verify' $_flags -o- "$_src" >
"$_dst"
fi
) & FTPPID=$!
set +o monitor
@@ -151,13 +155,32 @@ fetch() {
unset FTPPID
- if [ "$_exit" -ne 0 ]; then
+ if ((_exit != 0)); then
rm -f "$_dst"
+
+ # ftp doesn't provide useful exit codes
+ # so we have to grep its STDERR.
+ # _exit=2 means don't keep trying
+ _exit=2
+
+ # If it was 404, we might succeed at another file
+ if [ -s "$_ftp_errors" ] && grep -q "404 Not Found"
"$_ftp_errors"; then
+ _exit=1
+ _error=" (404 Not Found)"
+ rm -f "$_ftp_errors"
+ fi
+
warn "Cannot fetch $_src$_error"
- return 1
fi
- return 0
+ # If we have ftp errors, print them out,
+ # removing any cntrl characters (like 0x0d),
+ # and any leading blank lines.
+ [ -s "$_ftp_errors" ] &&
+ sed -e 's/[[:cntrl:]]//g' \
+ -e '/./,$!d' "$_ftp_errors" >&"$WARN_FD"
+
+ return "$_exit"
}
# If we fail to fetch the CFILE, we don't want to try again
@@ -165,12 +188,12 @@ fetch() {
# a blank file indicating failure.
check_cfile() {
if [ -e "$CFILE" ]; then
- [ -s "$CFILE" ] || return 1
+ [ -s "$CFILE" ] || return 2
return 0
fi
if ! fetch_cfile; then
echo -n > "$CFILE"
- return 1
+ return 2
fi
return 0
}
@@ -192,7 +215,7 @@ fetch_cfile() {
}
verify() {
- check_cfile || return 1
+ check_cfile || return $?
# The installer sha256 lacks -C, do it by hand
if ! grep -Fqx "SHA256 (${1##*/}) = $( /bin/sha256 -qb "$1" )" "$CFILE"
then
@@ -207,7 +230,7 @@ verify() {
# if VERBOSE is 0, don't show the checksum failure of an existing file.
verify_existing() {
local _v=$VERBOSE
- check_cfile || return 1
+ check_cfile || return $?
((_v == 0)) && "$DOWNLOAD" && _v=1
( VERBOSE=$_v verify "$@" )
@@ -242,7 +265,7 @@ firmware_in_dmesg() {
}
firmware_filename() {
- check_cfile || return 1
+ check_cfile || return $?
sed -n "s/.*(\($1-firmware-.*\.tgz\)).*/\1/p" "$CFILE" | sed '$!d'
}
@@ -253,6 +276,7 @@ firmware_devicename() {
}
lock_db() {
+ local _waited
[ "${LOCKPID:-}" ] && return 0
# The installer doesn't have perl, so we can't lock there
@@ -267,9 +291,22 @@ lock_db() {
$|=1;
$0 = "fw_update: lock_db";
- lock_db(0);
+ my $waited = 0;
+ package OpenBSD::FwUpdateState {
+ use parent 'OpenBSD::BaseState';
+ sub errprint ($self, @p) {
+ if ($p[0] && $p[0] =~ /already locked/) {
+
Don't lock package db if fw_update(8) is not installing firmware
While testing another patch, I noticed that fw_update will lock the
package database even if it's just downloading and not touching the
installed packages.
Currently we do _read_ the existing firmware as part of detecting what
we might need to download or upgrade without locking. I'm unsure if
that is an issue.
Comments, OK?
Index: fw_update.sh
===
RCS file: /cvs/src/usr.sbin/fw_update/fw_update.sh,v
retrieving revision 1.51
diff -u -p -r1.51 fw_update.sh
--- fw_update.sh14 Oct 2023 18:10:47 - 1.51
+++ fw_update.sh14 Oct 2023 18:12:08 -
@@ -593,7 +593,7 @@ kept=''
unregister=''
if [ "${devices[*]:-}" ]; then
- lock_db
+ "$INSTALL" && lock_db
for f in "${devices[@]}"; do
d="$( firmware_devicename "$f" )"
Re: Improve fw_update(9) output on errors somewhat
I have not gotten nearly enough positive feedback for the previous
iterations of this to be really comfortable this close to release.
This version doesn't solve any of the less common failure cases, but it
makes the normal "fw_update" when there is no network look nicer.
I can look at bringing back the previous patch after the release.
Before:
$ doas fw_update
fw_update:fw_update: firmware.openbsd.org: no address associated with name
add none; update none
Cannot fetch http://firmware.openbsd.org/firmware/snapshots/SHA256.sig
After:
$ doas ksh ./fw_update.sh
fw_update.sh: add none; update none
fw_update.sh: firmware.openbsd.org: no address associated with name
Cannot fetch http://firmware.openbsd.org/firmware/snapshots/SHA256.sig
Index: fw_update.sh
===
RCS file: /cvs/src/usr.sbin/fw_update/fw_update.sh,v
retrieving revision 1.50
diff -u -p -r1.50 fw_update.sh
--- fw_update.sh28 Sep 2023 01:18:52 - 1.50
+++ fw_update.sh1 Oct 2023 14:24:20 -
@@ -113,7 +113,7 @@ fetch() {
(
_flags=-vm
case "$VERBOSE" in
- 0|1) _flags=-VM ;;
+ 0|1) _flags=-VM ; exec 2>&"$WARN_FD" ;;
2) _flags=-Vm ;;
esac
if [ -x /usr/bin/su ]; then
Re: Improve fw_update(9) output on errors somewhat
Here's the next iteration. I'll probably commit it later this weekend
unless someone finds issues with it.
Here we trap errors from `ftp` unless we're using the verbose modes with
progress bars. We then parse those to guess whether we might be able to
find a different firmware (currently only 404) and save the errors for
later printing. We then return a different error code indicating
whether we should exit immediately or not. We can then output those
errors after we've finished outputting the current line.
Comments, testing results, OK?
Index: fw_update.sh
===
RCS file: /cvs/src/usr.sbin/fw_update/fw_update.sh,v
retrieving revision 1.50
diff -u -p -r1.50 fw_update.sh
--- fw_update.sh28 Sep 2023 01:18:52 - 1.50
+++ fw_update.sh30 Sep 2023 20:18:44 -
@@ -61,6 +61,7 @@ cleanup() {
if [ -d "$FD_DIR" ]; then
echo "" >&"$STATUS_FD"
exec 4>&-
+ exec 5>&-
[ -s "$FD_DIR/status" ] && cat "$FD_DIR/status"
[ -s "$FD_DIR/warn" ] && cat "$FD_DIR/warn" >&2
@@ -107,21 +108,24 @@ spin() {
fetch() {
local _src="${FWURL}/${1##*/}" _dst=$1 _user=_file _exit _error=''
+ local _ftp_errors="$FD_DIR/ftp_errors"
+ rm -f "$_ftp_errors"
# The installer uses a limited doas(1) as a tiny su(1)
set -o monitor # make sure ftp gets its own process group
(
_flags=-vm
case "$VERBOSE" in
- 0|1) _flags=-VM ;;
+ 0|1) _flags=-VM ; exec 2>"$_ftp_errors" ;;
2) _flags=-Vm ;;
esac
+
if [ -x /usr/bin/su ]; then
exec /usr/bin/su -s /bin/ksh "$_user" -c \
- "/usr/bin/ftp -N '${0##/}' -D 'Get/Verify' $_flags -o-
'$_src'" > "$_dst"
+ "/usr/bin/ftp -N error -D 'Get/Verify' $_flags -o- '$_src'"
> "$_dst"
else
exec /usr/bin/doas -u "$_user" \
- /usr/bin/ftp -N "${0##/}" -D 'Get/Verify' $_flags -o-
"$_src" > "$_dst"
+ /usr/bin/ftp -N error -D 'Get/Verify' $_flags -o- "$_src" >
"$_dst"
fi
) & FTPPID=$!
set +o monitor
@@ -151,13 +155,32 @@ fetch() {
unset FTPPID
- if [ "$_exit" -ne 0 ]; then
+ if ((_exit != 0)); then
rm -f "$_dst"
+
+ # ftp doesn't provide useful exit codes
+ # so we have to grep its STDERR.
+ # _exit=2 means don't keep trying
+ _exit=2
+
+ # If it was 404, we might succeed at another file
+ if [ -s "$_ftp_errors" ] && grep -q "404 Not Found"
"$_ftp_errors"; then
+ _exit=1
+ _error=" (404 Not Found)"
+ rm -f "$_ftp_errors"
+ fi
+
warn "Cannot fetch $_src$_error"
- return 1
fi
- return 0
+ # If we have ftp errors, print them out,
+ # removing any cntrl characters (like 0x0d),
+ # and any leading blank lines.
+ [ -s "$_ftp_errors" ] &&
+ sed -e 's/[[:cntrl:]]//g' \
+ -e '/./,$!d' "$_ftp_errors" >&"$WARN_FD"
+
+ return "$_exit"
}
# If we fail to fetch the CFILE, we don't want to try again
@@ -165,12 +188,12 @@ fetch() {
# a blank file indicating failure.
check_cfile() {
if [ -e "$CFILE" ]; then
- [ -s "$CFILE" ] || return 1
+ [ -s "$CFILE" ] || return 2
return 0
fi
if ! fetch_cfile; then
echo -n > "$CFILE"
- return 1
+ return 2
fi
return 0
}
@@ -192,7 +215,7 @@ fetch_cfile() {
}
verify() {
- check_cfile || return 1
+ check_cfile || return $?
# The installer sha256 lacks -C, do it by hand
if ! grep -Fqx "SHA256 (${1##*/}) = $( /bin/sha256 -qb "$1" )" "$CFILE"
then
@@ -207,7 +230,7 @@ verify() {
# if VERBOSE is 0, don't show the checksum failure of an existing file.
verify_existing() {
local _v=$VERBOSE
- check_cfile || return 1
+ check_cfile || return $?
((_v == 0)) && "$DOWNLOAD" && _v=1
( VERBOSE=$_v verify "$@" )
@@ -242,7 +265,7 @@ firmware_in_dmesg() {
}
firmware_filename() {
- check_cfile || return 1
+ check_cfile || return $?
sed -n "s/.*(\($1-firmware-.*\.tgz\)).*/\1/p" "$CFILE" | sed '$!d'
}
@@ -334,8 +357,11 @@ add_firmware () {
-s ",^firmware,${DESTDIR}/etc/firmware," \
-C / -zxphf - "+*" "firmware/*"
- _pkg="$( sed -n '/^@name /{s///p;q;}' "${FWPKGTMP}/+CONTENTS" )"
- if [ ! "$_pkg" ]; then
+
+ [ -s "${FWPKGTMP}/+CONTENTS" ] &&
+ _pkg="$( sed -n '/^@name /{s///p;q;}' "${FWPKGTMP}/+CONTENTS" )"
+
+ if [ ! "${_pkg:-}" ]; then
warn "Failed to extract
Re: Improve fw_update(9) output on errors somewhat
Here's a slightly simpler try suggested by deraadt@. I capture ftp's
STDERR and move it to the special "WARN_FD" that groups errors without
confusing the status line.
I also strip the "name" off the error as it looks a bit nicer than
repeating it everywhere.
If the fetch fails, we return 1 if we find "404 Not Found" and 2
otherwise. We then can continue on to another file in the 1 case and
exit immediately in the 2.
We always fail immediately on failures to fetch SHA256.sig as that
indicates that we're not going to be able to do much.
diff --git a/fw_update.sh b/fw_update.sh
index a379d74..8062da0 100644
--- a/fw_update.sh
+++ b/fw_update.sh
@@ -107,6 +107,9 @@ spin() {
fetch() {
local _src="${FWURL}/${1##*/}" _dst=$1 _user=_file _exit _error=''
+ local _name=${0##*/}
+ local _ftp_errors="$FD_DIR/ftp_errors"
+ rm -f "$_ftp_errors"
# The installer uses a limited doas(1) as a tiny su(1)
set -o monitor # make sure ftp gets its own process group
@@ -118,12 +121,12 @@ fetch() {
esac
if [ -x /usr/bin/su ]; then
exec /usr/bin/su -s /bin/ksh "$_user" -c \
- "/usr/bin/ftp -N '${0##/}' -D 'Get/Verify' $_flags -o-
'$_src'" > "$_dst"
+ "/usr/bin/ftp -N '$_name' -D 'Get/Verify' $_flags -o-
'$_src'" > "$_dst"
else
exec /usr/bin/doas -u "$_user" \
- /usr/bin/ftp -N "${0##/}" -D 'Get/Verify' $_flags -o-
"$_src" > "$_dst"
+ /usr/bin/ftp -N "$_name" -D 'Get/Verify' $_flags -o-
"$_src" > "$_dst"
fi
- ) & FTPPID=$!
+ ) 2>"$_ftp_errors" & FTPPID=$!
set +o monitor
SECONDS=0
@@ -151,10 +154,27 @@ fetch() {
unset FTPPID
- if [ "$_exit" -ne 0 ]; then
+ if ((_exit != 0)); then
+ # ftp doesn't provide useful exit codes
+ # so we have to grep its STDERR.
+ # _exit=2 means don't keep trying
+ _exit=2
+
+ # If it was 404, it's we might succeed at another file.
+ if grep -q "404 Not Found" "$_ftp_errors"; then
+ _exit=1
+ _error=" (404 Not Found)"
+ rm -f "$_ftp_errors"
+ fi
+ fi
+
+ [ -s "$_ftp_errors" ] &&
+ sed "s/^$_name: //" "$_ftp_errors" >&"$WARN_FD"
+
+ if ((_exit != 0)); then
rm -f "$_dst"
warn "Cannot fetch $_src$_error"
- return 1
+ return "$_exit"
fi
return 0
@@ -165,12 +185,12 @@ fetch() {
# a blank file indicating failure.
check_cfile() {
if [ -e "$CFILE" ]; then
- [ -s "$CFILE" ] || return 1
+ [ -s "$CFILE" ] || return 2
return 0
fi
if ! fetch_cfile; then
echo -n > "$CFILE"
- return 1
+ return 2
fi
return 0
}
@@ -192,7 +212,7 @@ fetch_cfile() {
}
verify() {
- check_cfile || return 1
+ check_cfile || return $?
# The installer sha256 lacks -C, do it by hand
if ! grep -Fqx "SHA256 (${1##*/}) = $( /bin/sha256 -qb "$1" )" "$CFILE"
then
@@ -207,7 +227,7 @@ verify() {
# if VERBOSE is 0, don't show the checksum failure of an existing file.
verify_existing() {
local _v=$VERBOSE
- check_cfile || return 1
+ check_cfile || return $?
((_v == 0)) && "$DOWNLOAD" && _v=1
( VERBOSE=$_v verify "$@" )
@@ -242,7 +262,7 @@ firmware_in_dmesg() {
}
firmware_filename() {
- check_cfile || return 1
+ check_cfile || return $?
sed -n "s/.*(\($1-firmware-.*\.tgz\)).*/\1/p" "$CFILE" | sed '$!d'
}
@@ -599,7 +619,19 @@ if [ "${devices[*]:-}" ]; then
verify_existing=true
if [ "$f" = "$d" ]; then
- f=$( firmware_filename "$d" ) || continue
+ f=$( firmware_filename "$d" ) || {
+ # Fetching the CFILE here is often the
+ # first attempt to talk to FWURL
+ # Fetching the CFILE here is often the
+ # If it does, no point in continuing.
+ if (($? > 1)); then
+ status " failed"
+ exit 1
+ fi
+
+ # If it does, no point in continuing.
+ continue
+ }
if [ ! "$f" ]; then
if "$INSTALL" && unregister_firmware "$d"; then
unregister="$unregister,$d"
@@ -717,11 +749,18 @@ for f in "${add[@]}" _update_ "${update[@]}"; do
fi
fetch "$f" &&
verify "$f" || {
+
Improve fw_update(9) output on errors somewhat
I did commit a few other fw_update fixes, so if you see any problems
with it, let me know soon, although I probably won't have time to look
until this weekend.
Theo rightly complained that the output from fw_update when the network
isn't available isn't very good.
$ doas fw_update
fw_update:fw_update: firmware.openbsd.org: no address associated with name
add none; update none
Cannot fetch http://firmware.openbsd.org/firmware/snapshots/SHA256.sig
We can improve that somewhat in the common case when the network is down
with the patch at the end of this email:
$ doas ksh ./fw_update.sh
fw_update.sh: firmware.openbsd.org: no address associated with name
Cannot fetch http://firmware.openbsd.org/firmware/snapshots/SHA256.sig
It's not quite as nice when DNS works, and the connection times out:
(I could special case this further though)
$ doas ksh ./fw_update.sh
fw_update.sh|
Cannot fetch http://firmware.openbsd.org/firmware/snapshots/SHA256.sig (timed
out)
And when specifying firmware files that doesn't exist it's still really
terrible.
$ doas ksh ./fw_update.sh atus-firmware-1.0p1.tgz intel-firmware-20210608v0.tgz
fw_update.sh: add atus: Error retrieving
http://firmware.openbsd.org/firmware/snapshots/atus-firmware-1.0p1.tgz: 404 Not
Found
failed (atus-firmware-1.0p1.tgz); update intel: Error retrieving
http://firmware.openbsd.org/firmware/snapshots/intel-firmware-20210608v0.tgz:
404 Not Found
failed (intel-firmware-20210608v0.tgz)
Cannot fetch
http://firmware.openbsd.org/firmware/snapshots/atus-firmware-1.0p1.tgz
Cannot fetch
http://firmware.openbsd.org/firmware/snapshots/intel-firmware-20210608v0.tgz
I can tidy that up a bit if I exit early on errors downloading firmware,
but that would mean it never attempted installing a firmware that did
exist. (that's not part of this patch)
$ doas ksh ./fw_update.sh atus-firmware-1.0p1.tgz intel-firmware-20210608v0.tgz
fw_update.sh: add atus: Error retrieving
http://firmware.openbsd.org/firmware/snapshots/atus-firmware-1.0p1.tgz: 404 Not
Found
Cannot fetch
http://firmware.openbsd.org/firmware/snapshots/atus-firmware-1.0p1.tgz
I'm not sure how much better it's possible to make this without trapping
all output from external commands and filtering them ourselves. That
though could mean output that just disappers if something goes wrong
which seems like a bad failure mode.
(I think there are some whitespace fixes I need)
Comments, OK?
Testing with different setups and different levels of verbosity are
welcome. I have been testing some, but it's easy to miss things.
(what other failure modes am I not considering?)
diff --git a/fw_update.sh b/fw_update.sh
index a379d74..151b16c 100644
--- a/fw_update.sh
+++ b/fw_update.sh
@@ -59,7 +59,7 @@ cleanup() {
set +o errexit # ignore errors from killing ftp
if [ -d "$FD_DIR" ]; then
- echo "" >&"$STATUS_FD"
+ [ -e "$FD_DIR/status" ] && echo "" >&"$STATUS_FD"
exec 4>&-
[ -s "$FD_DIR/status" ] && cat "$FD_DIR/status"
@@ -112,16 +112,18 @@ fetch() {
set -o monitor # make sure ftp gets its own process group
(
_flags=-vm
+ _name=${0##/}
case "$VERBOSE" in
- 0|1) _flags=-VM ;;
- 2) _flags=-Vm ;;
+ 0) _flags=-VM; _name="" ;;
+ 1) _flags=-VM ;;
+ 2) _flags=-Vm ;;
esac
if [ -x /usr/bin/su ]; then
exec /usr/bin/su -s /bin/ksh "$_user" -c \
- "/usr/bin/ftp -N '${0##/}' -D 'Get/Verify' $_flags -o-
'$_src'" > "$_dst"
+ "/usr/bin/ftp -N '$_name' -D 'Get/Verify' $_flags -o-
'$_src'" > "$_dst"
else
exec /usr/bin/doas -u "$_user" \
- /usr/bin/ftp -N "${0##/}" -D 'Get/Verify' $_flags -o-
"$_src" > "$_dst"
+ /usr/bin/ftp -N "$_name" -D 'Get/Verify' $_flags -o-
"$_src" > "$_dst"
fi
) & FTPPID=$!
set +o monitor
@@ -165,12 +167,12 @@ fetch() {
# a blank file indicating failure.
check_cfile() {
if [ -e "$CFILE" ]; then
- [ -s "$CFILE" ] || return 1
+ [ -s "$CFILE" ] || return 2
return 0
fi
if ! fetch_cfile; then
echo -n > "$CFILE"
- return 1
+ return 2
fi
return 0
}
@@ -192,7 +194,7 @@ fetch_cfile() {
}
verify() {
- check_cfile || return 1
+ check_cfile || return $?
# The installer sha256 lacks -C, do it by hand
if ! grep -Fqx "SHA256 (${1##*/}) = $( /bin/sha256 -qb "$1" )" "$CFILE"
then
@@ -207,7 +209,7 @@ verify() {
# if VERBOSE is 0, don't show the checksum failure of an existing file.
verify_existing() {
local _v=$VERBOSE
- check_cfile || return 1
+ check_cfile || return $?
((_v == 0)) && "$DOWNLOAD" && _v=1
( VERBOSE=$_v verify "$@" )
@@ -242,7 +244,7 @@
Re: fw_update throwing exit 1 on successful install when passed a driver by name
On Fri, Sep 22, 2023 at 01:05:17PM -0500, Brian Conway wrote:
> Greetings. I noticed the above behavior with the latest 7.4-beta when
> scripting some fw_update runs. Perhaps a corner case was missed in the
> recent work? Steps below, let me know if more information is required
> to reproduce. Thanks in advance.
Yup, I bet I missed several exit code issues. This one is obviously
wrong, but I have to decide what to do in the:
fw_update -d intel nonexist
fw_update -d intel # not installed, success?
fw_updateintel nonexist
fw_updateintel # already installed, success?
I'll take a look at it. Not entirely sure if I'll have time before a
week from tomorrow, but we shall see.
> root:current-amd64.124:0:~# fw_update -d intel; echo ${?}
> fw_update: delete intel
> 0
> root:current-amd64.124:0:~# fw_update; echo ${?}
> fw_update: add intel; update none; keep inteldrm,vmm
> 0
> root:current-amd64.124:0:~# fw_update -d intel; echo ${?}
> fw_update: delete intel
> 0
> root:current-amd64.124:0:~# fw_update intel; echo ${?}
> fw_update: add intel; update none
> 1
That is a bit surprising though, I kind of assumed if it got to printing
the end of the status it would always exit 0.
>
> Brian Conway
> Owner
> RCE Software, LLC
>
> OpenBSD 7.4-beta (GENERIC.MP) #1373: Thu Sep 21 09:26:32 MDT 2023
> dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> real mem = 4160446464 (3967MB)
> avail mem = 4014624768 (3828MB)
> random: good seed from bootblocks
> mpath0 at root
> scsibus0 at mpath0: 256 targets
> mainbus0 at root
> bios0 at mainbus0: SMBIOS rev. 3.2 @ 0x9b81 (97 entries)
> bios0: vendor American Megatrends Inc. version "5.17" date 03/25/2021
> bios0: Default string Default string
> efi0 at bios0: UEFI 2.7
> efi0: American Megatrends rev 0x50011
> acpi0 at bios0: ACPI 6.2
> acpi0: sleep states S0 S5
> acpi0: tables DSDT FACP MCFG SSDT FIDT SSDT HPET SSDT SSDT SSDT NHLT LPIT
> SSDT SSDT DBGP DBG2 SSDT DMAR WSMT APIC FPDT
> acpi0: wakeup devices PS2K(S0) PS2M(S0) RP01(S0) PXSX(S0) RP02(S0) PXSX(S0)
> RP03(S0) PXSX(S0) RP04(S0) PXSX(S0) RP05(S0) PXSX(S0) RP06(S0) PXSX(S0)
> RP07(S0) PXSX(S0) [...]
> acpitimer0 at acpi0: 3579545 Hz, 24 bits
> acpimcfg0 at acpi0
> acpimcfg0: addr 0xe000, bus 0-255
> acpihpet0 at acpi0: 2399 Hz
> acpimadt0 at acpi0 addr 0xfee0: PC-AT compat
> cpu0 at mainbus0: apid 0 (boot processor)
> cpu0: Intel(R) Celeron(R) CPU 5305U @ 2.30GHz, 2194.89 MHz, 06-8e-0c, patch
> 00f8
> cpu0:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,SMEP,ERMS,INVPCID,MPX,RDSEED,SMAP,CLFLUSHOPT,PT,SRBDS_CTRL,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,IBRS_ALL,SKIP_L1DFL,MDS_NO,MISC_PKG_CT,ENERGY_FILT,FB_CLEAR,RRSBA,GDS_CTRL,XSAVEOPT,XSAVEC,XGETBV1,XSAVES
> cpu0: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 256KB
> 64b/line 4-way L2 cache, 2MB 64b/line 8-way L3 cache
> cpu0: smt 0, core 0, package 0
> mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
> cpu0: apic clock running at 24MHz
> cpu0: mwait min=64, max=64, C-substates=0.2.1.2.4.1.1.1, IBE
> cpu1 at mainbus0: apid 2 (application processor)
> cpu1: Intel(R) Celeron(R) CPU 5305U @ 2.30GHz, 2194.90 MHz, 06-8e-0c, patch
> 00f8
> cpu1:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,SMEP,ERMS,INVPCID,MPX,RDSEED,SMAP,CLFLUSHOPT,PT,SRBDS_CTRL,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,IBRS_ALL,SKIP_L1DFL,MDS_NO,MISC_PKG_CT,ENERGY_FILT,FB_CLEAR,RRSBA,GDS_CTRL,XSAVEOPT,XSAVEC,XGETBV1,XSAVES
> cpu1: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 256KB
> 64b/line 4-way L2 cache, 2MB 64b/line 8-way L3 cache
> cpu1: smt 0, core 1, package 0
> ioapic0 at mainbus0: apid 2 pa 0xfec0, version 20, 120 pins
> acpiprt0 at acpi0: bus 0 (PCI0)
> acpiprt1 at acpi0: bus -1 (PEG0)
> acpiprt2 at acpi0: bus -1 (PEG1)
> acpiprt3 at acpi0: bus -1 (PEG2)
> acpiprt4 at acpi0: bus -1 (RP01)
> acpiprt5 at acpi0: bus -1 (RP02)
> acpiprt6 at acpi0: bus -1 (RP03)
> acpiprt7 at acpi0: bus -1 (RP04)
> acpiprt8 at acpi0: bus -1 (RP05)
> acpiprt9 at acpi0: bus -1 (RP06)
> acpiprt10 at acpi0: bus 1 (RP07)
> acpiprt11 at acpi0: bus 2 (RP08)
> acpiprt12 at acpi0: bus -1 (RP09)
> acpiprt13 at acpi0: bus -1 (RP10)
> acpiprt14 at acpi0: bus -1 (RP11)
> acpiprt15 at acpi0: bus -1 (RP12)
> acpiprt16 at acpi0: bus 3 (RP13)
> acpiprt17 at acpi0: bus 4 (RP14)
> acpiprt18 at acpi0: bus 5 (RP15)
> acpiprt19 at
make install in firmware ports doesn't mark @option firmware
While testing some other stuff, job@ noticed that fw_update(8) kept
reinstalling the bwfm firmware. After some research it turns out that
fw_update ignores installed packages that aren't marked "@option
firmware" and it does that too hard. I'll likely commit the below patch
to at least warn folks that something is wrong, but I don't think I want
to start ignoring that @option. Installing firmware from the ports tree
with `make install` is one way for this to happen.
One side effect of this is fw_update will move a temporary directory
full of package registration into the existing directory, which the
below patch will also fix. Unfortunately if that has happened, neither
fw_update or pkg_delete will happily clean up that package and cleanup
requires rm -r. Should I try to work up a patch to clean that up
automatically?
Should packaging firmware add that option to the +CONTENTS, instead of
relying on fw_update at install time?
$ cd /usr/ports/sysutils/firmware/bwfm
$ make install
===> Installing bwfm-firmware-20200316.1.3p3 from
/usr/ports/packages/sparc64/all/
bwfm-firmware-20200316.1.3p3: ok
$ grep @option /var/db/pkg/bwfm-firmware-20200316.1.3p3/+CONTENTS
@option manual-installation
$ doas pkg_delete bwfm-firmware-20200316.1.3p3
bwfm-firmware-20200316.1.3p3: ok
$ doas fw_update
/usr/ports/packages/sparc64/all/bwfm-firmware-20200316.1.3p3.tgz
fw_update: added bwfm; updated none; kept none
$ grep @option /var/db/pkg/bwfm-firmware-20200316.1.3p3/+CONTENTS
@option manual-installation
@option firmware
$
Comments or suggestions before I commit this?
diff --git a/fw_update.sh b/fw_update.sh
index 02261dd..11eae49 100644
--- a/fw_update.sh
+++ b/fw_update.sh
@@ -320,7 +320,8 @@ detect_firmware() {
}
add_firmware () {
- local _f="${1##*/}" _m="${2:-Install}" _pkgname
+ local _f="${1##*/}" _m="${2:-Install}"
+ local _pkgdir="${DESTDIR}/var/db/pkg" _pkg
FWPKGTMP="$( tmpdir "${DESTDIR}/var/db/pkg/.firmware" )"
local _flags=-vm
case "$VERBOSE" in
@@ -333,9 +334,16 @@ add_firmware () {
-s ",^firmware,${DESTDIR}/etc/firmware," \
-C / -zxphf - "+*" "firmware/*"
- _pkgname="$( sed -n '/^@name /{s///p;q;}' "${FWPKGTMP}/+CONTENTS" )"
- if [ ! "$_pkgname" ]; then
- echo "Failed to extract name from $1, partial install" 2>&1
+ _pkg="$( sed -n '/^@name /{s///p;q;}' "${FWPKGTMP}/+CONTENTS" )"
+ if [ ! "$_pkg" ]; then
+ warn "Failed to extract name from $1, partial install"
+ rm -rf "$FWPKGTMP"
+ unset FWPKGTMP
+ return 1
+ fi
+
+ if [ -e "$_pkgdir/$_pkg" ]; then
+ warn "Failed to register: $_pkgdir/$_pkg is not firmware"
rm -rf "$FWPKGTMP"
unset FWPKGTMP
return 1
@@ -351,7 +359,7 @@ w
EOL
chmod 755 "$FWPKGTMP"
- mv "$FWPKGTMP" "${DESTDIR}/var/db/pkg/${_pkgname}"
+ mv "$FWPKGTMP" "$_pkgdir/$_pkg"
unset FWPKGTMP
}
Re: fw_update lock_db should exit when parent exits
On Thu, Aug 24, 2023 at 06:53:27AM -0600, Todd C. Miller wrote: > On Wed, 23 Aug 2023 18:23:40 -0700, Andrew Hewus Fresh wrote: > > > I would have to see an example of doing that between ksh and perl. > > Standard output should already be a pipe in the perl process by > virtue of running as a co-process. In theory you should be able > to poll on it checking for POLLHUP. Since our pipes are actually > bidiretional we can cheat and use select. Something like this: I hadn't considered waiting for STDOUT to be readable, that makes total sense. Thanks. I'll probably wait to commit these fw_update fixes until I'm back on Wednesday for a long weekend away from computers.
Re: fw_update lock_db should exit when parent exits
On Wed, Aug 23, 2023 at 08:42:03AM -0600, Todd C. Miller wrote: > On Tue, 22 Aug 2023 19:55:56 -0700, Andrew Hewus Fresh wrote: > > > I noticed this when testing how signal handling worked in fw_update, it > > turns out that if you `pkill -KILL -f fw_update` it may leave behind a perl > > process that is locking the package database. Instead of just waiting > > to be killed, we can have that process check to see if its parent is > > still around and exit if not. > > > > Is there a more appropriate solution to this? > > What's the right way to notice your parent exited? > > One way is to have a pipe between the parent and child and use > select() instead of the sleep() to tell when it goes away. I would have to see an example of doing that between ksh and perl.
fw_update lock_db should exit when parent exits
I noticed this when testing how signal handling worked in fw_update, it
turns out that if you `pkill -KILL -f fw_update` it may leave behind a perl
process that is locking the package database. Instead of just waiting
to be killed, we can have that process check to see if its parent is
still around and exit if not.
Is there a more appropriate solution to this?
What's the right way to notice your parent exited?
Comments, OK?
Index: fw_update.sh
===
RCS file: /cvs/src/usr.sbin/fw_update/fw_update.sh,v
retrieving revision 1.44
diff -u -p -r1.44 fw_update.sh
--- fw_update.sh12 Dec 2022 02:30:51 - 1.44
+++ fw_update.sh23 Aug 2023 02:47:54 -
@@ -221,10 +221,15 @@ lock_db() {
$|=1;
+ $0 = "fw_update: lock_db";
lock_db(0);
say $$;
- sleep;
+
+ # If our parent exits unexpectedly,
+ # ppid will become init and we should exit.
+ my $parent = getppid;
+ sleep 1 while getppid == $parent;
EOL
set +o monitor
Re: Have sysupgrade run fw_update -vv
On Sat, Aug 19, 2023 at 03:11:46PM -0700, Andrew Hewus Fresh wrote: > On Sun, Aug 13, 2023 at 04:42:49PM -0600, Theo de Raadt wrote: > > Stuart Henderson wrote: > > > > > On 2023/08/13 11:44, Andrew Hewus Fresh wrote: > > > > My laptop doesn't have the fastest wifi and sysupgrade already uses a > > > > progress bar to show what it's doing, so I'd really like to provide more > > > > feedback on what it is doing: > > > > > > Does a single -v give enough feedback? > > > > As shown below, -vv is ridiculously verbose. > > > > I think -v is also too verbose. > > That makes sense to me. Here's a fairly intrusive patch that in the > normal case prints the status line "as we go" so you can see that it is > doing something. Errors from ftp are not super friendly, although I'm > not entirely sure what it should do yet, so right now I do nothing. Some additional detail on this. One example is if the ftp process fails to download a file, for example it is killed or times out or something which results in (in the case of a SIGTERM): $ doas ksh ./fw_update.sh fw_update.sh: add intel /Terminated Terminated ,inteldrm,iwm,uvideo,vmm; update none Cannot fetch http://firmware.openbsd.org/firmware/snapshots/intel-firmware-20230808v0.tgz Would it be an improvement to provide more detail? $ doas ksh ./fw_update.sh fw_update.sh: add intel /Terminated Terminated (intel-firmware-20230808v0.tgz download failed),inteldrm,iwm,uvideo,vmm; update none Cannot fetch http://firmware.openbsd.org/firmware/snapshots/intel-firmware-20230808v0.tgz Since I don't know what, if anything, was printed by the failure, it's hard to know how much would be helpful. > In the normal and single -v case, it also adds a spinner while it is > downloading if there STDOUT is a tty. > > We now go in two phases for install/update where we first detect what > needs to be done and then do it so we can separate install from > download. > > In any case, I think this is ready for wider testing. > > Comments, suggestions, OK? > -- andrew Full-time system administration is a delicate balance between proactiveness and laziness. -- jhorwitz from use.perl.org
Re: Have sysupgrade run fw_update -vv
On Sun, Aug 13, 2023 at 04:42:49PM -0600, Theo de Raadt wrote:
> Stuart Henderson wrote:
>
> > On 2023/08/13 11:44, Andrew Hewus Fresh wrote:
> > > My laptop doesn't have the fastest wifi and sysupgrade already uses a
> > > progress bar to show what it's doing, so I'd really like to provide more
> > > feedback on what it is doing:
> >
> > Does a single -v give enough feedback?
>
> As shown below, -vv is ridiculously verbose.
>
> I think -v is also too verbose.
That makes sense to me. Here's a fairly intrusive patch that in the
normal case prints the status line "as we go" so you can see that it is
doing something. Errors from ftp are not super friendly, although I'm
not entirely sure what it should do yet, so right now I do nothing.
In the normal and single -v case, it also adds a spinner while it is
downloading if there STDOUT is a tty.
We now go in two phases for install/update where we first detect what
needs to be done and then do it so we can separate install from
download.
In any case, I think this is ready for wider testing.
Comments, suggestions, OK?
Index: fw_update.sh
===
RCS file: /cvs/src/usr.sbin/fw_update/fw_update.sh,v
retrieving revision 1.44
diff -u -p -r1.44 fw_update.sh
--- fw_update.sh12 Dec 2022 02:30:51 - 1.44
+++ fw_update.sh19 Aug 2023 21:49:39 -
@@ -40,18 +40,39 @@ DELETE=false
DOWNLOAD=true
INSTALL=true
LOCALSRC=
+ENABLE_SPINNER=false
+[ -t 1 ] && ENABLE_SPINNER=true
+
+integer STATUS_FD=1
+integer WARN_FD=2
+FD_DIR=
unset FTPPID
unset LOCKPID
unset FWPKGTMP
REMOVE_LOCALSRC=false
+
+status() { echo -n "$*" >&"$STATUS_FD"; }
+warn() { echo"$*" >&"$WARN_FD"; }
+
cleanup() {
set +o errexit # ignore errors from killing ftp
+
+ if [ -d "$FD_DIR" ]; then
+ echo "" >&"$STATUS_FD"
+ exec 4>&-
+
+ [ -s "$FD_DIR/status" ] && cat "$FD_DIR/status"
+ [ -s "$FD_DIR/warn" ] && cat "$FD_DIR/warn" >&2
+
+ rm -rf "$FD_DIR"
+ fi
+
[ "${FTPPID:-}" ] && kill -TERM -"$FTPPID" 2>/dev/null
[ "${LOCKPID:-}" ] && kill -TERM -"$LOCKPID" 2>/dev/null
[ "${FWPKGTMP:-}" ] && rm -rf "$FWPKGTMP"
"$REMOVE_LOCALSRC" && rm -rf "$LOCALSRC"
- [ -e "${CFILE}" ] && [ ! -s "$CFILE" ] && rm -f "$CFILE"
+ [ -e "$CFILE" ] && [ ! -s "$CFILE" ] && rm -f "$CFILE"
}
trap cleanup EXIT
@@ -70,6 +91,22 @@ tmpdir() {
echo "$_dir"
}
+spin() {
+ if ! "$ENABLE_SPINNER"; then
+ sleep 1
+ return 0
+ fi
+
+ {
+ echo -n ' '
+ for p in '/' '-' '\\' '|'; do
+ echo -n '\010'"$p"
+ sleep 0.25
+ done
+ echo -n "\010 \010\010"
+ }>/dev/tty
+}
+
fetch() {
local _src="${FWURL}/${1##*/}" _dst=$1 _user=_file _exit _error=''
@@ -99,13 +136,13 @@ fetch() {
if [[ $_last -ne $5 ]]; then
_last=$5
SECONDS=0
- sleep 1
+ spin
else
kill -INT -"$FTPPID" 2>/dev/null
_error=" (timed out)"
fi
else
- sleep 1
+ spin
fi
done
@@ -118,7 +155,7 @@ fetch() {
if [ "$_exit" -ne 0 ]; then
rm -f "$_dst"
- echo "Cannot fetch $_src$_error" >&2
+ warn "Cannot fetch $_src$_error"
return 1
fi
@@ -133,7 +170,7 @@ check_cfile() {
[ -s "$CFILE" ] || return 1
return 0
fi
- if ! fetch_cfile "$@"; then
+ if ! fetch_cfile; then
echo -n > "$CFILE"
return 1
fi
@@ -146,10 +183,10 @@ fetch_cfile() {
fetch "$CFILE" || return 1
set -o noclobber
! signify -qVep "$FWPUB_KEY" -x "$CFILE" -m "$CFILE" &&
- echo "Signature check of SHA256.sig failed" >&2 &&
+ w
Re: installer: proper disk crypto passphrase prompt loop
On Tue, Aug 15, 2023 at 09:08:44AM +, Klemens Nanni wrote:
> On Wed, Aug 02, 2023 at 11:51:09AM +, Klemens Nanni wrote:
> > An alternative approach could be a new bioctl(8)) flag
> > -K Keep prompting until new and re-typed passphrases match.
> > to repeat the prompt (during interactive creation only) until match or ^C:
> >
> > # ./obj/bioctl -K -cC -Cforce -lvnd0a softraid0
> > New passphrase:
> > Re-type passphrase:
> > bioctl: Passphrases did not match, try again
> > New passphrase:
> > Re-type passphrase:
> > ...
> > softraid0: CRYPTO volume attached as sd3
> >
> > That means:
> > * straight forward installer code
> > * -K could be extended to unlock (not creation) prompts if deemed useful
> >
> >
> > I see value in both approaches, but do tend towards the first -s one
> > leaving it to the installer, mainly because it touches bioctl.c less and
> > makes the prompt text in line with other questions.
>
> On the other hand, sticking to bioctl's prompt in the installer question
> means consistency across installer, bootloader and userland (change
> passphrase, create/unlock extra disks).
>
> -K makes bioctl(8) similar to passwd(1) and other prompting tools we have,
> although -K is for one very specific and thus very simple, i.e. it retries
> endlessly rather than aborting after N tries like other tools do.
>
> Given it is an extra optional flag and the complexity it avoids, this seems
> like an acceptable solution.
>
> Feedback? Objection? OK?
I prefer this approach, if there is one use for retrying, there are
probably more.
> Index: distrib/miniroot/install.sub
> ===
> RCS file: /cvs/src/distrib/miniroot/install.sub,v
> retrieving revision 1.1253
> diff -u -p -r1.1253 install.sub
> --- distrib/miniroot/install.sub 10 Aug 2023 17:09:34 - 1.1253
> +++ distrib/miniroot/install.sub 15 Aug 2023 09:07:47 -
> @@ -3075,7 +3075,7 @@ do_autoinstall() {
> }
>
> encrypt_root() {
> - local _chunk _tries=0
> + local _chunk
>
> [[ $MDBOOTSR == y ]] || return
>
> @@ -3097,10 +3097,7 @@ encrypt_root() {
> md_prep_fdisk $_chunk
> echo 'RAID *' | disklabel -w -A -T- $_chunk
>
> - until bioctl -Cforce -cC -l${_chunk}a softraid0 >/dev/null; do
> - # Most likely botched passphrases, silently retry twice.
> - ((++_tries < 3)) || exit
> - done
> + bioctl -K -Cforce -cC -l${_chunk}a softraid0 >/dev/null
>
> # No volumes existed before asking, but we just created one.
> ROOTDISK=$(get_softraid_volumes)
> Index: sbin/bioctl/bioctl.8
> ===
> RCS file: /cvs/src/sbin/bioctl/bioctl.8,v
> retrieving revision 1.111
> diff -u -p -r1.111 bioctl.8
> --- sbin/bioctl/bioctl.8 6 Jul 2023 21:08:50 - 1.111
> +++ sbin/bioctl/bioctl.8 15 Aug 2023 09:06:20 -
> @@ -41,7 +41,7 @@
> .Ar device
> .Pp
> .Nm bioctl
> -.Op Fl dhiPqsv
> +.Op Fl dhiKPqsv
> .Op Fl C Ar flag Ns Op Pf , Ar ...
> .Op Fl c Ar raidlevel
> .Op Fl k Ar keydisk
> @@ -245,6 +245,8 @@ they become part of the array again.
> .It Fl d
> Detach volume specified by
> .Ar device .
> +.It Fl K
> +Keep prompting until new and re-typed passphrases match.
> .It Fl k Ar keydisk
> Use special device
> .Ar keydisk
> Index: sbin/bioctl/bioctl.c
> ===
> RCS file: /cvs/src/sbin/bioctl/bioctl.c,v
> retrieving revision 1.151
> diff -u -p -r1.151 bioctl.c
> --- sbin/bioctl/bioctl.c 18 Oct 2022 07:04:20 - 1.151
> +++ sbin/bioctl/bioctl.c 15 Aug 2023 09:06:21 -
> @@ -90,6 +90,7 @@ int human;
> int verbose;
> u_int32_tcflags = 0;
> int rflag = 0;
> +int keeptrying = 0;
> char *password;
>
> void *bio_cookie;
> @@ -114,8 +115,8 @@ main(int argc, char *argv[])
> if (argc < 2)
> usage();
>
> - while ((ch = getopt(argc, argv, "a:b:C:c:dH:hik:l:O:Pp:qr:R:st:u:v")) !=
> - -1) {
> + while ((ch = getopt(argc, argv, "a:b:C:c:dH:hiKk:l:O:Pp:qr:R:st:u:v"))
> + != -1) {
> switch (ch) {
> case 'a': /* alarm */
> func |= BIOC_ALARM;
> @@ -163,6 +164,9 @@ main(int argc, char *argv[])
> case 'i': /* inquiry */
> func |= BIOC_INQ;
> break;
> + case 'K': /* Keep retrying new passphrase. */
> + keeptrying = 1;
> + break;
> case 'k': /* Key disk. */
> key_disk = optarg;
> break;
> @@ -289,7 +293,7 @@ usage(void)
> "\t[-t patrol-function] "
> "[-u channel:target[.lun]] "
> "device\n"
> -
Have sysupgrade run fw_update -vv
My laptop doesn't have the fastest wifi and sysupgrade already uses a
progress bar to show what it's doing, so I'd really like to provide more
feedback on what it is doing:
$ doas fw_update -d intel
fw_update: deleted intel
$ time doas fw_update
fw_update: added intel; updated none; kept inteldrm,iwm,uvideo,vmm
0m58.45s real 0m00.51s user 0m00.35s system
$ doas fw_update -d intel
fw_update: deleted intel
$ time doas fw_update -vv
Detect firmware ... found.
Get/Verify SHA256.sig 100% |**| 2371 00:00
Get/Verify intel-firmware-2023080... 100% |*| 12155 KB01:04
Install intel-firmware-2023080... 100% || 12155 KB00:00
fw_update: added intel; updated none; kept inteldrm,iwm,uvideo,vmm
1m17.46s real 0m00.56s user 0m00.34s system
Comments, OK?
Index: usr.sbin/sysupgrade/sysupgrade.sh
===
RCS file: /cvs/src/usr.sbin/sysupgrade/sysupgrade.sh,v
retrieving revision 1.48
diff -u -p -r1.48 sysupgrade.sh
--- usr.sbin/sysupgrade/sysupgrade.sh 8 Jun 2022 09:03:11 - 1.48
+++ usr.sbin/sysupgrade/sysupgrade.sh 13 Aug 2023 18:22:02 -
@@ -205,7 +205,7 @@ if [[ ${_NEXTKERNV[1]} == '-current' ]];
else
FW_URL=http://firmware.openbsd.org/firmware/${_NEXTKERNV[0]}/
fi
-VNAME="${_NEXTKERNV[0]}" fw_update -p ${FW_URL} || true
+VNAME="${_NEXTKERNV[0]}" fw_update -vv -p ${FW_URL} || true
install -F -m 700 bsd.rd /bsd.upgrade
logger -t sysupgrade -p kern.info "installed new /bsd.upgrade. Old kernel
version: $(sysctl -n kern.version)"
Re: Stop using direct syscall(2) from perl(1)
On Sun, Jul 09, 2023 at 01:29:58PM -0700, Andrew Hewus Fresh wrote:
> Here is a patch to replace perl(1)'s use of syscall(2) with a dispatcher
> that will call the libc function instead.
>
> I have to do some work on style before this is ready to commit, but it
> should be ready for some testing.
>
> I don't currently plan on committing syscall_emulator.c because we need
> to regenerate it each time as I'm not sure how to tie it into sys/kern's
> `make syscalls` to only do it when things change.
>
> Looking for tests from folks who use syscall from perl as well as style
> suggestions (like how do I correctly sort headers?) and maybe even
> ways to enable additional syscalls.
Here's a new version, I think I addressed all of the feedback I got,
although may have missed something.
Hopefully it is easier to test with sthen@'s fix.
Thanks miod@, espie@, sthen@, and especially gkoehler@ for a lot of
excellent feedback.
I'm sure there's still stuff missing and obvious problems that don't
stand out to someone who writes a lot of perl.
Starting Saturday I'm traveling for work for the week, so will probably
be even less responsive than usual until I'm back.
Index: gnu/usr.bin/perl/MANIFEST
===
RCS file: /home/afresh1/OpenBSD-perl/OP/cvs/src/gnu/usr.bin/perl/MANIFEST,v
retrieving revision 1.67
diff -u -p -a -u -p -r1.67 MANIFEST
--- gnu/usr.bin/perl/MANIFEST 8 Jul 2023 14:18:35 - 1.67
+++ gnu/usr.bin/perl/MANIFEST 3 Aug 2023 04:34:38 -
@@ -6605,6 +6605,7 @@ t/op/svleak.plTest file for svleak.t
t/op/svleak.t See if stuff leaks SVs
t/op/switch.t See if switches (given/when) work
t/op/symbolcache.t See if undef/delete works on stashes with
functions
+t/op/syscall_emulator.tTests that syscall works via the
emulator
t/op/sysio.t See if sysread and syswrite work
t/op/taint.t See if tainting works
t/op/threads.t Misc. tests for perl features with threads
Index: gnu/usr.bin/perl/Makefile.SH
===
RCS file: /home/afresh1/OpenBSD-perl/OP/cvs/src/gnu/usr.bin/perl/Makefile.SH,v
retrieving revision 1.60
diff -u -p -a -u -p -r1.60 Makefile.SH
--- gnu/usr.bin/perl/Makefile.SH8 Jul 2023 14:18:35 - 1.60
+++ gnu/usr.bin/perl/Makefile.SH3 Aug 2023 04:34:38 -
@@ -541,7 +541,7 @@ c1 = av.c scope.c op.c doop.c doio.c dum
c2 = perly.c pp.c pp_hot.c pp_ctl.c pp_sys.c regcomp.c regexec.c utf8.c sv.c
c3 = taint.c toke.c util.c deb.c run.c builtin.c universal.c pad.c globals.c
keywords.c
c4 = perlio.c numeric.c mathoms.c locale.c pp_pack.c pp_sort.c caretx.c
dquote.c time64.c
-c5 = $(mallocsrc)
+c5 = $(mallocsrc) syscall_emulator.c
!NO!SUBS!
@@ -557,7 +557,7 @@ c = $(c1) $(c2) $(c3) $(c4) $(c5) minipe
obj1 = $(mallocobj) gv$(OBJ_EXT) toke$(OBJ_EXT) perly$(OBJ_EXT) pad$(OBJ_EXT)
regcomp$(OBJ_EXT) dump$(OBJ_EXT) util$(OBJ_EXT) mg$(OBJ_EXT) reentr$(OBJ_EXT)
mro_core$(OBJ_EXT) keywords$(OBJ_EXT) builtin$(OBJ_EXT)
obj2 = hv$(OBJ_EXT) av$(OBJ_EXT) run$(OBJ_EXT) pp_hot$(OBJ_EXT) sv$(OBJ_EXT)
pp$(OBJ_EXT) scope$(OBJ_EXT) pp_ctl$(OBJ_EXT) pp_sys$(OBJ_EXT)
-obj3 = doop$(OBJ_EXT) doio$(OBJ_EXT) regexec$(OBJ_EXT) utf8$(OBJ_EXT)
taint$(OBJ_EXT) deb$(OBJ_EXT) globals$(OBJ_EXT) perlio$(OBJ_EXT)
numeric$(OBJ_EXT) mathoms$(OBJ_EXT) locale$(OBJ_EXT) pp_pack$(OBJ_EXT)
pp_sort$(OBJ_EXT) caretx$(OBJ_EXT) dquote$(OBJ_EXT) time64$(OBJ_EXT)
+obj3 = doop$(OBJ_EXT) doio$(OBJ_EXT) regexec$(OBJ_EXT) utf8$(OBJ_EXT)
taint$(OBJ_EXT) deb$(OBJ_EXT) globals$(OBJ_EXT) perlio$(OBJ_EXT)
numeric$(OBJ_EXT) mathoms$(OBJ_EXT) locale$(OBJ_EXT) pp_pack$(OBJ_EXT)
pp_sort$(OBJ_EXT) caretx$(OBJ_EXT) dquote$(OBJ_EXT) time64$(OBJ_EXT)
syscall_emulator$(OBJ_EXT)
# split the objects into 3 exclusive sets: those used by both miniperl and
# perl, and those used by just one or the other. Doesn't include the
Index: gnu/usr.bin/perl/Makefile.bsd-wrapper
===
RCS file:
/home/afresh1/OpenBSD-perl/OP/cvs/src/gnu/usr.bin/perl/Makefile.bsd-wrapper,v
retrieving revision 1.113
diff -u -p -a -u -p -r1.113 Makefile.bsd-wrapper
--- gnu/usr.bin/perl/Makefile.bsd-wrapper 15 Feb 2023 01:38:20 -
1.113
+++ gnu/usr.bin/perl/Makefile.bsd-wrapper 3 Aug 2023 04:34:38 -
@@ -39,11 +39,18 @@ cleandir:
fi
cd ${.CURDIR} && ${MAKE} -f Makefile.bsd-wrapper1 cleandir
-all: config.sh
+all: syscall_emulator.c config.sh
cd ${.CURDIR} && exec ${MAKE} -f Makefile.bsd-wrapper1 perl.build
cd ${.CURDIR} && exec ${MAKE} -f Makefile.bsd-wrapper1 mansrc.build
install:
cd ${.CURDIR} && exec ${MAKE} -f Makefile.bsd-wrapper1 install
+
+
+syscall_emulator.c: gen_syscall_emul
Re: Stop using direct syscall(2) from perl(1)
Here's a new patch, mostly for sthen@'s obvious fix, although I have
added some of gkoehler@'s suggestions, although I have some questions
and comments inline.
I am looking at some of the other suggestions, but they showed up while
I was testing this set.
One thing to note, on the advice of miod@, I adjusted the mapping of
size_t from u_int to u_long, but that means we no longer recognize
getlogin_r. When I asked, miod@ suggested that syscalls.master was
confused.
$ grep getlogin_r /usr/src/sys/kern/syscalls.master /usr/include/unistd.h
/usr/src/sys/kern/syscalls.master:141 STD { int
sys_getlogin_r(char *namebuf, u_int namelen); }
/usr/include/unistd.h:intgetlogin_r(char *, size_t)
On Thu, Jul 13, 2023 at 07:38:22AM +0100, Stuart Henderson wrote:
> On 2023/07/13 00:57, George Koehler wrote:
> > On Sun, 9 Jul 2023 13:29:58 -0700
> > Andrew Hewus Fresh wrote:
> >
> > > Here is a patch to replace perl(1)'s use of syscall(2) with a dispatcher
> > > that will call the libc function instead.
> >
> > patch(1) didn't "chmod +x gen_syscall_emulator.pl", but I needed to
> > do so to get around this this error,
>
> There shouldn't be 'x' bits in files committed to the tree, so the
> Makefile should explicitly run this with perl.
Oops, fixed!
On Mon, Jul 17, 2023 at 11:54:18PM -0400, George Koehler wrote:
> I prefer braces like this,
>
> case SYS_truncate: {
> ...
> break;
> }
Easy enough, I didn't see that called out in style(9).
> In my opinion, all va_arg()s should look like
>
> off_t length = (off_t)va_arg(args, long);
>
> because perl passes every argument as long (from pp_syscall in
> pp_sys.c). I worry that va_arg(args, off_t) would act strangely
> on platforms with 32-bit long and 64-bit off_t. Only a few syscalls
> have off_t arguments, and these few calls are almost useless in Perl,
> so this might not affect real programs.
>
> How to pread($fd, $buf, 4, 16)?
>
> 1. syscall(169, $fd, $buf, 4, 0, 16)
> 2. syscall(169, $fd, $buf, 4, 0, 0, 16)
> 3. syscall(169, $fd, $buf, 4, 16, 0)
> 4. syscall(169, $fd, $buf, 4, 0, 16, 0)
> 5. syscall(169, $fd, $buf, 4, 16)
>
> If va_args(args, off_t) takes 2 longs, then pick line 1 if big-endian,
> or line 3 if little-endian. If it skips a 3rd long for alignment,
> then pick line 2 or 4. If it takes 1 long, then pick line 5.
>
> If we (off_t)va_args(args, long), then it always takes 1 long, so
> every platform picks line 5, but the offset must fit in a long.
Which is the way it works without the this middleman?
On Wed, Jul 19, 2023 at 03:58:25PM -0400, George Koehler wrote:
> On Sun, 9 Jul 2023 13:29:58 -0700
> Andrew Hewus Fresh wrote:
>
> > +syscall_emulator.h Emulator to dispatch syscalls to libc
>
> This got installed as
> /usr/libdata/perl5/powerpc64-openbsd/CORE/syscall_emulator.h
I was wondering about why there was a list of headers in the Makefile,
I went cross eyed while trying to track that down, I'm testing what I
expect will remove that now.
> ExtUtils::MakeMaker is using cc -DHAS_SYSCALL_EMULATOR to build my XS
> module; the define causes to #include .
Oh right, I forgot that I hacked that in as I tried to figure out
_where_ I should do that. I will tidy that up too.
> Can we hide this feature from XS? I wish that only pp_sys.c and
> syscall_emulator.c would #include , that
> syscall_emulator.h would not be installed, and that
> -DHAS_SYSCALL_EMULATOR would not appear in non-core modules.
I can just patch it in without the define. I may have to do a bit more
when syscall is actually removed, since Configure probes for it, but I
_think_ I found the right define to do that, but it always takes me too
long staring at this before it makes sense and I've been kept too busy
away from computers to do that these last couple weeks. Preliminary
testing tells me it probably works.
> Also, the new test has a heap overflow, because $sb is too small for
> a struct stat,
>
> $ cd /usr/src/gnu/usr.bin/perl/obj
> $ MALLOC_OPTIONS=S perl t/op/syscall_emulator.t
> /usr/include/sys/syscall.h -> /usr/include/sys/syscall.ph
> 1..13
> ok 1 - Opened test.txt for write/create
> ok 2 - Wrote out to test.txt
> ok 3 - closed test.txt
> perl(15511) in malloc(): write after free 0xc36d1efcddc0
> Abort trap (core dumped)
Hmm, I guess I need to get a powerpc64 machine.
> I prevented the overflow by making $sb too big,
>
> --- t/op/syscall_emulator.t.beforeThu Jul 13 00:39:10 2023
> +++ t/op/syscall_emulator.t Wed Jul 19 15:32:40 2023
> @@ -47,7 +47,7 @@
> my $out = "Hello World\n";
> my $in = "\0" x 32;
> my ($in_p, $in_v);
> -my $sb = &qu
Stop using direct syscall(2) from perl(1)
.sh
+all: syscall_emulator.c config.sh
cd ${.CURDIR} && exec ${MAKE} -f Makefile.bsd-wrapper1 perl.build
cd ${.CURDIR} && exec ${MAKE} -f Makefile.bsd-wrapper1 mansrc.build
install:
cd ${.CURDIR} && exec ${MAKE} -f Makefile.bsd-wrapper1 install
+
+
+syscall_emulator.c: gen_syscall_emulator.pl /usr/include/sys/syscall.h
/usr/include/sys/syscallargs.h
+ $(.CURDIR)/gen_syscall_emulator.pl > $@
.include
Index: gnu/usr.bin/perl/config.over
===
RCS file: /home/afresh1/OpenBSD-perl/OP/cvs/src/gnu/usr.bin/perl/config.over,v
retrieving revision 1.22
diff -u -p -a -u -p -r1.22 config.over
--- gnu/usr.bin/perl/config.over5 Feb 2017 00:33:38 - 1.22
+++ gnu/usr.bin/perl/config.over9 Jul 2023 19:42:26 -
@@ -64,3 +64,10 @@ myuname='openbsd'
# force to use ranlib
ranlib='ranlib'
+
+# Enable the syscall emulator,
+# enabling syscall even if we don't have it
+ccflags="-DHAS_SYSCALL_EMULATOR $ccflags"
+d_syscall=define
+d_syscallproto=define
+
Index: gnu/usr.bin/perl/gen_syscall_emulator.pl
===
RCS file: gnu/usr.bin/perl/gen_syscall_emulator.pl
diff -N gnu/usr.bin/perl/gen_syscall_emulator.pl
--- /dev/null 1 Jan 1970 00:00:00 -
+++ gnu/usr.bin/perl/gen_syscall_emulator.pl9 Jul 2023 19:42:50 -
@@ -0,0 +1,354 @@
+#!/usr/bin/perl
+# $OpenBSD$ #
+use v5.36;
+
+# Copyright (c) 2023 Andrew Hewus Fresh
+#
+# Permission to use, copy, modify, and distribute this software for any
+# purpose with or without fee is hereby granted, provided that the above
+# copyright notice and this permission notice appear in all copies.
+#
+# THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+# WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+# MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+# ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+# WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+# ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+# OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+
+my $includes = '/usr/include';
+
+# See also /usr/src/sys/kern/syscalls.master
+my %syscalls = parse_syscalls(
+"$includes/sys/syscall.h",
+"$includes/sys/syscallargs.h",
+);
+delete $syscalls{MAXSYSCALL}; # not an actual function
+
+# The ordered list of all the headers we need
+my @headers = qw<
+ sys/syscall.h
+ stdarg.h
+ errno.h
+
+ sys/socket.h
+ sys/event.h
+ sys/futex.h
+ sys/ioctl.h
+ sys/ktrace.h
+ sys/mman.h
+ sys/mount.h
+ sys/msg.h
+ sys/poll.h
+ sys/ptrace.h
+ sys/resource.h
+ sys/select.h
+ sys/sem.h
+ sys/shm.h
+ sys/stat.h
+ sys/sysctl.h
+ sys/time.h
+ sys/uio.h
+ sys/wait.h
+
+ dirent.h
+ fcntl.h
+ sched.h
+ signal.h
+ stdlib.h
+ stdio.h
+ syslog.h
+ tib.h
+ time.h
+ unistd.h
+>;
+
+foreach my $header (@headers) {
+ my $file = "$includes/$header";
+ open my $fh, '<', $file or die "Unable to open $file: $!";
+ my $content = do { local $/; readline $fh };
+ close $fh;
+
+ # Look for matching syscalls in this header
+ foreach my $name (sort keys %syscalls) {
+ my $s = $syscalls{$name};
+ my $func_sig = find_func_sig($content, $name, $s);
+
+ if (ref $func_sig) {
+ die "Multiple defs for $name <$header> <$s->{header}>"
+ if $s->{header};
+ $s->{func} = $func_sig;
+ $s->{header} = $header;
+ } elsif ($func_sig) {
+ $s->{mismatched_sig} = "$func_sig <$header>";
+ }
+ }
+}
+
+say "/*\n * Generated from gen_syscall_emulator.pl\n */";
+say "#include <$_>" for @headers;
+print <<"EOL";
+
+long
+syscall_emulator(int syscall, ...)
+{
+ long ret = 0;
+ va_list args;
+ va_start(args, syscall);
+
+ switch(syscall) {
+EOL
+
+foreach my $name (
+ sort { $syscalls{$a}{id} <=> $syscalls{$b}{id} } keys %syscalls
+) {
+ my %s = %{ $syscalls{$name} };
+
+ # Some syscalls we can't emulate, so we comment those out.
+ $s{skip} //= "Indirect syscalls not supported"
+ if !$s{argtypes} && ($s{args}[-1] || '') eq '...';
+ $s{skip} //= "Mismatched func: $s{mismatched_sig}"
+ if $s{mismatched_sig} and not $s{func};
+ $s{skip} //= "No signature found in headers"
+
Re: OpenBSD::MkTemp vs Devel::Cover
On Sat, Jul 08, 2023 at 12:09:01PM -0700, Andrew Hewus Fresh wrote:
> On Sat, Jul 08, 2023 at 11:18:00AM +0200, Marc Espie wrote:
> > Hey, Philip, you wrote this a long time ago.
> >
> > Now, I'm trying to get some coverage out of Devel::Cover on pkg_add,
> > and somehow, it gets in the way.
> >
> > # perl -MDevel::Cover=+select,OpenBSD/.* /usr/sbin/pkg_add random_run
>
> > the problem was:
> >
> > Found type 9 GLOB(0x64a7292fc10), but it is not representable by the Sereal
> > encoding format at
> > /usr/local/libdata/perl5/site_perl/amd64-openbsd/Devel/Cover/DB/IO/Sereal.pm
> > line 46.
> >
> >
> > I have zero idea if Devel::Cover is to blame or if OpenBSD::MkTemp is
> > missing
> > some magic annotation to cover for _GEN_1, _GEN_2, but the end result is
> > that NO coverage data gets written, none at all (I suspect some missing
> > annotations, see below)
>
> Same, especially when I can't reproduce on my laptop that needs to be
> updated or on my sparc64 that just got updated today:
I take that back, running the same command as root I see the same
problem.
However, trying to reproduce with a simple case doesn't show me the same
issue, so not quite sure what Devel::Cover is trying to store.
$ cat mktemp.pl
#!/usr/bin/perl
use v5.36;
use OpenBSD::MkTemp qw< mkstemp mkdtemp >;
sub output($fh, $content) {
print $fh $content;
}
my $d = mkdtemp("fooX");
my ($fh, $file) = mkstemp("fooX");
output($fh, "Hello World\n");
close $fh;
$ doas perl -MDevel::Cover=+select,OpenBSD.* ./mktemp.pl
This version of Devel::Cover was built with Perl version 5.036000.
It is now being run with Perl version 5.036001.
Attempting to make adjustments, but you may find that some of your modules do
not have coverage data collected. You may need to alter the +-inc, +-ignore
and +-select options.
Devel::Cover 1.40: Collecting coverage data for branch, condition, pod,
statement, subroutine and time.
Selecting packages matching:
OpenBSD.*
Ignoring packages matching:
/Devel/Cover[./]
Ignoring packages in:
/usr/local/libdata/perl5/site_perl/sparc64-openbsd
/usr/local/libdata/perl5/site_perl
/usr/libdata/perl5/sparc64-openbsd
/usr/libdata/perl5
Devel::Cover: Deleting old coverage for changed file mktemp.pl
Devel::Cover: getting BEGIN block coverage
Devel::Cover: 100% - 0s taken
Devel::Cover: getting CHECK block coverage
Devel::Cover: 100% - 0s taken
Devel::Cover: getting END/INIT block coverage
Devel::Cover: 100% - 0s taken
Devel::Cover: getting CV coverage
Devel::Cover: 100% - 0s taken
Devel::Cover: Writing coverage database to
/tmp/x/cover_db/runs/1688930470.15647.20075
-- -- -- -- -- -- --
File stmt bran condsubpod time total
-- -- -- -- -- -- --
...openbsd/OpenBSD/MkTemp.pm 77.7 25.0 25.0 83.30.00.7 61.7
mktemp.pl 100.0n/an/a 100.0n/a 99.3 100.0
Total 87.5 25.0 25.0 88.80.0 100.0 74.5
-- -- -- -- -- -- --
Re: OpenBSD::MkTemp vs Devel::Cover
On Sat, Jul 08, 2023 at 11:18:00AM +0200, Marc Espie wrote: > Hey, Philip, you wrote this a long time ago. > > Now, I'm trying to get some coverage out of Devel::Cover on pkg_add, > and somehow, it gets in the way. > > # perl -MDevel::Cover=+select,OpenBSD/.* /usr/sbin/pkg_add random_run > the problem was: > > Found type 9 GLOB(0x64a7292fc10), but it is not representable by the Sereal > encoding format at > /usr/local/libdata/perl5/site_perl/amd64-openbsd/Devel/Cover/DB/IO/Sereal.pm > line 46. > > > I have zero idea if Devel::Cover is to blame or if OpenBSD::MkTemp is missing > some magic annotation to cover for _GEN_1, _GEN_2, but the end result is > that NO coverage data gets written, none at all (I suspect some missing > annotations, see below) Same, especially when I can't reproduce on my laptop that needs to be updated or on my sparc64 that just got updated today: $ sysctl kern.version kern.version=OpenBSD 7.3-current (GENERIC.MP) #1193: Fri May 19 11:04:34 MDT 2023 dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP $ perl -MDevel::Cover=+select,OpenBSD/.* /usr/sbin/pkg_add random_run Devel::Cover 1.40: Collecting coverage data for branch, condition, pod, statement, subroutine and time. Selecting packages matching: OpenBSD/.* Ignoring packages matching: /Devel/Cover[./] Ignoring packages in: /usr/local/libdata/perl5/site_perl/amd64-openbsd /usr/local/libdata/perl5/site_perl /usr/libdata/perl5/amd64-openbsd /usr/libdata/perl5 pkg_add: pkg_add must be run as root Devel::Cover: getting BEGIN block coverage Devel::Cover: 100% - 1s taken Devel::Cover: getting CHECK block coverage Devel::Cover: 100% - 0s taken Devel::Cover: getting END/INIT block coverage Devel::Cover: 100% - 0s taken Devel::Cover: getting CV coverage Devel::Cover: 100% - 5s taken Devel::Cover: Writing coverage database to /tmp/x/cover_db/runs/1688842881.32512.14344 -- -- -- -- -- -- -- File stmt bran condsubpod time total -- -- -- -- -- -- -- ...data/perl5/OpenBSD/Add.pm 13.30.00.0 26.10.04.7 10.0 ...penBSD/AddCreateDelete.pm 51.5 35.0 20.0 39.30.00.7 45.2 ...erl5/OpenBSD/AddDelete.pm 31.2 12.0 17.1 33.30.05.6 25.9 ...perl5/OpenBSD/ArcCheck.pm 12.60.00.0 41.60.03.69.6 ...erl5/OpenBSD/BaseState.pm 24.37.4 11.1 31.00.00.7 19.2 ...ta/perl5/OpenBSD/Error.pm 54.0 37.50.0 52.10.00.6 48.2 ...a/perl5/OpenBSD/Getopt.pm 47.9 15.00.0 83.30.00.2 39.2 ...a/perl5/OpenBSD/Handle.pm9.60.00.0 18.50.00.67.0 .../perl5/OpenBSD/IdCache.pm 29.60.0n/a 33.30.00.1 23.4 ...l5/OpenBSD/Interactive.pm 11.90.00.0 50.00.00.29.0 .../perl5/OpenBSD/LibSpec.pm6.80.00.08.10.00.55.1 ...data/perl5/OpenBSD/Log.pm 13.10.00.0 20.00.00.2 10.2 ...l5/OpenBSD/PackageInfo.pm 24.48.80.0 33.30.00.7 19.8 ...l5/OpenBSD/PackageName.pm3.50.00.05.50.00.82.6 ...OpenBSD/PackingElement.pm6.51.30.08.60.05.55.4 ...l5/OpenBSD/PackingList.pm5.90.00.09.20.01.44.1 ...ta/perl5/OpenBSD/Paths.pm 10.50.00.08.60.00.36.4 ...a/perl5/OpenBSD/PkgAdd.pm7.60.55.0 19.40.03.46.0 ...a/perl5/OpenBSD/PkgCfl.pm 19.70.00.0 41.60.00.4 16.1 .../OpenBSD/ProgressMeter.pm 28.2 10.0 22.2 16.00.00.3 19.8 ...BSD/ProgressMeter/Term.pm 19.86.05.2 21.40.0 41.6 14.9 ...a/perl5/OpenBSD/Search.pm4.50.0n/a6.60.00.64.1 ...rl5/OpenBSD/SharedLibs.pm 26.00.00.0 35.20.00.4 19.5 ...ta/perl5/OpenBSD/State.pm 44.0 26.9 25.0 43.50.01.0 38.1 ...ta/perl5/OpenBSD/Subst.pm 15.50.00.0 30.70.00.3 12.6 ...ata/perl5/OpenBSD/Temp.pm 19.70.00.0 33.30.00.4 16.9 ...erl5/OpenBSD/UpdateSet.pm3.60.00.06.10.00.92.9 ...ta/perl5/OpenBSD/Ustar.pm5.40.00.0 14.00.02.04.2 ...ta/perl5/OpenBSD/Vstat.pm7.10.00.09.60.00.95.5 ...openbsd/OpenBSD/MkTemp.pm 61.10.00.0 66.60.00.1 44.1 /usr/sbin/pkg_add 90.4 75.0n/a 100.0n/a 19.9 89.6 Total 14.12.63.4 20.10.0 100.0 11.1 -- -- -- -- -- -- -- $ sysctl kern.version kern.version=OpenBSD 7.3-current (GENERIC.MP) #1815: Thu Jul 6 14:47:12 MDT 2023 dera...@sparc64.openbsd.org:/usr/src/sys/arch/sparc64/compile/GENERIC.MP $ perl -MDevel::Cover=+select,OpenBSD/.* /usr/sbin/pkg_add
Re: OpenBSD perl 5.36.1 - Call for Testing
On Thu, Jul 06, 2023 at 11:47:02AM +0300, Alexander Bluhm wrote: > I think you should commit 5.36.1, we will see fallout in regress > test. Sigh. I realized I haven't built a release with this patch recently. I don't expect failure but don't want to commit without doing that. I'll see if it's done before I go to bed, but otherwise it'll be in the morning. > After that we have only the syscall Perl diff floating around and > can concentrate on that. I think I fixed it up to generate the sendsyslog case correctly, will regen the patch and mail it out after I've gotten the other committed. https://github.com/afresh1/OpenBSD-perl/commit/7493f2536bf6d0755be097673fb433b381b29842 l8rZ, -- andrew A hacker does for love what others would not do for money.
Re: pkg-config tweaks
On Mon, May 22, 2023 at 09:33:48AM +0200, Marc Espie wrote:
> - move to 5.36: signatures + prototypes mostly everywhere
> Not used for Getopt::Long, because the calling conventions are somewhat
> too verbose.
> - use constant for the mode{libs} and mode{cflags} values
> - remove two completely unneeded [] in regexps
> - fix indentation and parentheses in a few locations
>
>
> There should be no behavior change, please test.
Mostly this seems OK. There is one that isn't a behavior change, but
using $a and $b as names can sometimes cause confusing behavior, so I
generally try to avoid those names.
https://perldoc.perl.org/perlvar#$a
Changing the flag values from what they were makes it really hard for me
to tell if they are right. I think they all seem to have translated
properly, but wow, the new stuff is so much easier to read, I'm glad it
changed, but it was hard to review!
The rest _looked_ right, but I may have missed a typo. If so I expect
that would be obvious on running it.
> Deeper question: as it stands \w and such do handle unicode, more or less.
> Is this something we actually want/need in pkg-config ? should we look at
> restricting the regexps through one of the locale modifiers ?
I assume we'll only want to support UTF-8, which is probably just adding
the "standard preamble" to switch undeclared file streams to use the
UTF-8 encoding.
https://perldoc.perl.org/perlunicook#%E2%84%9E-0:-Standard-preamble
https://perldoc.perl.org/perlunicook#%E2%84%9E-17:-Make-file-I/O-default-to-utf8
We can also switch the regex character classes to be limited to ascii.
https://perldoc.perl.org/perlunicook#%E2%84%9E-24:-Disabling-Unicode-awareness-in-builtin-charclasses
I don't know enough about when pkg-config might encounter anything that
isn't ascii, so it's hard for me to say what the right answer is.
>
> Index: pkg-config
> ===
> RCS file: /cvs/src/usr.bin/pkg-config/pkg-config,v
> retrieving revision 1.95
> diff -u -p -r1.95 pkg-config
> --- pkg-config15 Sep 2020 07:18:45 - 1.95
> +++ pkg-config22 May 2023 07:22:10 -
> @@ -16,14 +16,20 @@
> # ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
> # OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
>
> -use strict;
> -use warnings;
> +use v5.36;
> use Config;
> use Getopt::Long;
> use File::Basename;
> use File::stat;
> use OpenBSD::PkgConfig;
>
> +use constant {
> + ONLY_I => 1,
> + ONLY_l => 2,
> + ONLY_L => 4,
> + ONLY_OTHER => 8
> +};
> +
> my @PKGPATH = qw(/usr/lib/pkgconfig
>/usr/local/lib/pkgconfig
>/usr/local/share/pkgconfig
> @@ -70,7 +76,7 @@ defined $ENV{PKG_CONFIG_DEBUG_SPEW} ? $m
>
> if ($logfile) {
> open my $L, ">>" , $logfile or die;
> - print $L beautify_list($0, @ARGV), "\n";
> + say $L beautify_list($0, @ARGV);
> close $L;
> }
>
> @@ -87,7 +93,7 @@ GetOptions( 'debug' => \$mode{debug},
> 'help' => \, #does not return
> 'usage' => \, #does not return
> 'list-all' => \$mode{list},
> - 'version' => sub { print "$version\n" ; exit(0);}
> ,
> + 'version' => sub { say $version ; exit(0);} ,
> 'errors-to-stdout' => sub { $mode{estdout} = 1},
> 'print-errors' => sub { $mode{printerr} = 1},
> 'silence-errors'=> sub { $mode{printerr} = 0},
> @@ -97,13 +103,13 @@ GetOptions( 'debug' =>
> \$mode{debug},
> 'print-requires'=> \$mode{printrequires},
> 'print-requires-private' => \$mode{printrequiresprivate},
>
> - 'cflags'=> sub { $mode{cflags} = 3},
> - 'cflags-only-I' => sub { $mode{cflags} |= 1},
> - 'cflags-only-other' => sub { $mode{cflags} |= 2},
> - 'libs' => sub { $mode{libs} = 7},
> - 'libs-only-l' => sub { $mode{libs} |= 1},
> - 'libs-only-L' => sub { $mode{libs} |= 2},
> - 'libs-only-other' => sub { $mode{libs} |= 4},
> + 'cflags'=> sub { $mode{cflags} =
> ONLY_I|ONLY_OTHER},
> + 'cflags-only-I' => sub { $mode{cflags} |= ONLY_I},
> + 'cflags-only-other' => sub { $mode{cflags} |= ONLY_OTHER},
> + 'libs' => sub { $mode{libs} =
> ONLY_L|ONLY_l|ONLY_OTHER},
> + 'libs-only-l' => sub { $mode{libs} |= ONLY_l},
> + 'libs-only-L' => sub { $mode{libs} |= ONLY_L},
> + 'libs-only-other' => sub { $mode{libs} |= ONLY_OTHER},
> 'exists'=> sub { $mode{exists} = 1} ,
> 'validate' => sub {
Re: installer: arm64: better disk variable handling
On Sat, Mar 25, 2023 at 07:13:13PM +, Klemens Nanni wrote:
> On Sat, Mar 25, 2023 at 07:02:24PM +, Klemens Nanni wrote:
> > Call the function argument `_disk' and construct full device paths where
> > needed, so that other code can reuse the mnemonic variable.
> >
> > This matches install.sub's convention around ROOTDISK=sd0, ROOTDEV=sd0a and
> > other disk/device code.
> >
> > Since it touches the installboot block, hoist it before the sysctl block
> > which merely assigns `_plat'.
> >
> > A future diff then turns out much better.
>
> > - dd if=$_mdec/u-boot-sunxi-with-spl.bin of=${_disk}c \
> > + dd if=$_mdec/u-boot-sunxi-with-spl.bin of=/dev/${_disk}c \
>
> Just noticed that this should use the raw device as done elsewhere across
> distrib/ and explained in INSTALL.* docs.
>
> Easily missed before the diff, obvious with it.
>
> OK?
This looks right to me, but I can't test it at the moment.
> Index: ramdisk/install.md
> ===
> RCS file: /cvs/src/distrib/arm64/ramdisk/install.md,v
> retrieving revision 1.37
> diff -u -p -r1.37 install.md
> --- ramdisk/install.md25 Mar 2023 18:29:37 - 1.37
> +++ ramdisk/install.md25 Mar 2023 19:06:43 -
> @@ -36,7 +36,13 @@ NCPU=$(sysctl -n hw.ncpufound)
> MOUNT_ARGS_msdos="-o-l"
>
> md_installboot() {
> - local _disk=/dev/$1 _mdec _plat
> + local _disk=$1 _mdec _plat
> +
> + if ! installboot -r /mnt $_disk; then
> + echo "\nFailed to install bootblocks."
> + echo "You will not be able to boot OpenBSD from ${_disk}."
> + exit
> + fi
>
> case $(sysctl -n machdep.compatible) in
> apple,*)_plat=apple;;
> @@ -44,12 +50,6 @@ md_installboot() {
> raspberrypi,*) _plat=rpi;
> esac
>
> - if ! installboot -r /mnt ${1}; then
> - echo "\nFailed to install bootblocks."
> - echo "You will not be able to boot OpenBSD from ${1}."
> - exit
> - fi
> -
> # Apply some final tweaks on selected platforms
> _mdec=/usr/mdec/$_plat
>
> @@ -65,11 +65,11 @@ md_installboot() {
> fi
> ;;
> pine64)
> - dd if=$_mdec/u-boot-sunxi-with-spl.bin of=${_disk}c \
> + dd if=$_mdec/u-boot-sunxi-with-spl.bin of=/dev/r${_disk}c \
> bs=1024 seek=8 status=none
> ;;
> rpi)
> - mount ${MOUNT_ARGS_msdos} ${_disk}i /mnt/mnt
> + mount ${MOUNT_ARGS_msdos} /dev/${_disk}i /mnt/mnt
> cp $_mdec/{bootcode.bin,start*.elf,fixup*.dat,*.dtb} /mnt/mnt/
> cp $_mdec/u-boot.bin /mnt/mnt/
> mkdir -p /mnt/mnt/overlays
>
--
andrew
The programmer's national anthem is 'GH!!'.
Re: [installer] default answer to "Is the disk partition already mounted?"
On Wed, Apr 05, 2023 at 11:11:12AM +, Klemens Nanni wrote:
> 04.04.2023 19:58, Mikhail пишет:
> > In conversation with Klemens (kn@) new iteration has been born, it
> > follows to what Omar (op@) has suggested - we're not trying to change
> > the default answer for all cases, since no one complained for all these
> > years, but just flip default for installation mode, which should save
> > some keystrokes and back-and-forth movement for novice users.
>
> Feeback?
>
> This is OK kn if anyone wants to commit, else I'll take OKs.
OK afresh1@
I can't recall the last time I installed off a disk, but the idea makes
sense to me.
> >
> > Next attempt (with input from kn@):
> >
> > diff /usr/src
> > commit - 3f266bdacd577948d2ae789cfe31cc2939a83cf9
> > path + /usr/src
> > blob - 5919eeece326bed27a65b43e96be50e7c72471e1
> > file + distrib/miniroot/install.sub
> > --- distrib/miniroot/install.sub
> > +++ distrib/miniroot/install.sub
> > @@ -2043,7 +2043,12 @@ install_disk() {
> > # Install sets from disk.
> > # Ask for the disk device containing the set files.
> > install_disk() {
> > - if ! ask_yn "Is the disk partition already mounted?" yes; then
> > + local _ismounted=yes
> > +
> > + # No partitions are mounted prior to regular installation.
> > + [[ $MODE == install ]] && _ismounted=no
> > +
> > + if ! ask_yn "Is the disk partition already mounted?" $_ismounted; then
> > ask_which "disk" "contains the $MODE media" \
> > '$(bsort $(get_dkdevs))' \
> > '$(get_dkdevs_uninitialized)'
> >
>
--
andrew
Hey! It compiles! Ship it!
Re: installer: apple arm64: simplify firmware copy
On Sat, Mar 25, 2023 at 06:38:56PM +, Klemens Nanni wrote:
> Less duplicate code, simpler to cope with newer firmware.
> Same idiom is used in install.sub already.
>
> OK?
While unlikely to fail, I do prefer `cd ... && ...` as a general rule.
However, this definitely makes it easier to tell that we're doing the
same thing.
OK afresh1@
> Index: ramdisk/install.md
> ===
> RCS file: /cvs/src/distrib/arm64/ramdisk/install.md,v
> retrieving revision 1.37
> diff -u -p -r1.37 install.md
> --- ramdisk/install.md25 Mar 2023 18:29:37 - 1.37
> +++ ramdisk/install.md25 Mar 2023 18:35:52 -
> @@ -55,14 +55,9 @@ md_installboot() {
>
> case $_plat in
> apple)
> - if [[ -d /etc/firmware/apple ]]; then
> - (cd /etc/firmware
> - pax -rw apple /mnt/etc/firmware)
> - fi
> - if [[ -d /etc/firmware/apple-bwfm ]]; then
> - (cd /etc/firmware
> - pax -rw apple-bwfm /mnt/etc/firmware)
> - fi
> + (cd /etc/firmware; for _dir in apple{,-bwfm}; do
> + [[ -d $_dir ]] && pax -rw $_dir /mnt/etc/firmware
> + done)
> ;;
> pine64)
> dd if=$_mdec/u-boot-sunxi-with-spl.bin of=${_disk}c \
>
Re: installer: simplify MAKEDEV invocation
On Sat, Mar 25, 2023 at 04:17:06PM +, Klemens Nanni wrote:
> The script takes multiple arguments and loops over them on its own,
> so no reason to handroll the loop.
>
> OK?
This makes sense to me. I wonder if MAKEDEV changed to allow multiple
arguments or if the original author didn't realize it.
> In case anyone wonders why explicit devices are passed while "all" is used:
> it only creates that many devices, so if during install you created,
> e.g. sd14, that won't be covered by "all", hence passing explicit devices.
>
> Index: install.sub
> ===
> RCS file: /cvs/src/distrib/miniroot/install.sub,v
> retrieving revision 1.1234
> diff -u -p -r1.1234 install.sub
> --- install.sub 10 Mar 2023 10:33:40 - 1.1234
> +++ install.sub 14 Mar 2023 16:47:32 -
> @@ -2931,13 +2931,9 @@ finish_up() {
> fi
>
> echo -n "Making all device nodes..."
> - (cd /mnt/dev; sh MAKEDEV all
> - # Make sure any devices we found during probe are created in the
> - # installed system.
> - for _dev in $(get_dkdevs) $(get_cddevs); do
> - sh MAKEDEV $_dev
> - done
> - )
> + # Make sure any devices we found during probe are created in the
> + # installed system.
> + (cd /mnt/dev; sh MAKEDEV $(get_dkdevs) $(get_cddevs) all)
> echo " done."
>
> # We may run some programs in chroot, and some of them might be
>
Improve fw_update(8) in nonexistent dir
In a private mail it was noticed that tar(1) doesn't like if you remove
the directory out from under it.
$ mkdir -p /tmp/x && cd $_ && rm -rf $_ && tar -C / -f /nonexistent
(null): Can't open current working directory.: No such file or directory
While that's not really a supported configuration, we can make
fw_update(8) behave a little nicer.
First we `cd /` before running tar, which avoids the ability to remove
the cwd from under it. This makes a normal fw_update and the "-d" flag
work ask expected.
Through some more testing, we obviously can't support -F in the case
when we don't have a cwd, but the output in that case is a bit ugly due
to the way we use subshells and a background ftp process:
$ mkdir -p /tmp/x && cd $_ && rm -rf $_ && doas fw_update -F ogx
/usr/sbin/fw_update: cannot create ./SHA256.sig: No such file or directory
Cannot fetch http://firmware.openbsd.org/firmware/snapshots/SHA256.sig
/usr/sbin/fw_update: cannot create ./SHA256.sig: No such file or directory
fw_update: downloaded none; kept none
With this change it stops after the first two lines. This part is a bit
racy as the directory could be removed at any point during the download
and other errors will continue to be ugly, so maybe best to ignore that
problem.
Comments, obvious bugs, OK?
Index: usr.sbin/fw_update/fw_update.sh
===
RCS file: /cvs/src/usr.sbin/fw_update/fw_update.sh,v
retrieving revision 1.44
diff -u -p -r1.44 fw_update.sh
--- usr.sbin/fw_update/fw_update.sh 12 Dec 2022 02:30:51 - 1.44
+++ usr.sbin/fw_update/fw_update.sh 17 Mar 2023 00:08:45 -
@@ -143,7 +143,7 @@ check_cfile() {
fetch_cfile() {
if "$DOWNLOAD"; then
set +o noclobber # we want to get the latest CFILE
- fetch "$CFILE" || return 1
+ fetch "$CFILE" || exit 2
set -o noclobber
! signify -qVep "$FWPUB_KEY" -x "$CFILE" -m "$CFILE" &&
echo "Signature check of SHA256.sig failed" >&2 &&
@@ -277,10 +277,12 @@ add_firmware () {
2|3) _flags=-Vm ;;
esac
- ftp -N "${0##/}" -D "$_m" "$_flags" -o- "file:${1}" |
+ ftp -N "${0##/}" -D "$_m" "$_flags" -o- "file:${1}" | (
+ cd / # tar does not like if cwd is nonexistent
tar -s ",^\+,${FWPKGTMP}/+," \
-s ",^firmware,${DESTDIR}/etc/firmware," \
-C / -zxphf - "+*" "firmware/*"
+ )
_pkgname="$( sed -n '/^@name /{s///p;q;}' "${FWPKGTMP}/+CONTENTS" )"
if [ ! "$_pkgname" ]; then
@@ -514,7 +516,10 @@ if [ "${devices[*]:-}" ]; then
verify_existing=true
if [ "$f" = "$d" ]; then
- f=$( firmware_filename "$d" ) || continue
+ f=$( firmware_filename "$d" ) || {
+ [ $? = 1 ] || exit 1
+ continue
+ }
if [ ! "$f" ]; then
if "$INSTALL" && unregister_firmware "$d"; then
unregister="$unregister,$d"
Small tweak to simplify vlan suggestion in installer
Instead of looking at vlans and adding one to the last, which if someone
did manual configuration could be inaccurate as ifconfig returns vlans
in the order they were created.
(oops, just noticed the extra newline, will fix that)
Comments, OK?
Index: install.sub
===
RCS file: /cvs/src/distrib/miniroot/install.sub,v
retrieving revision 1.1228
diff -u -p -r1.1228 install.sub
--- install.sub 16 Feb 2023 18:10:28 - 1.1228
+++ install.sub 19 Feb 2023 20:09:01 -
@@ -1308,7 +1308,7 @@ ieee80211_config() {
# Set up IPv4 and IPv6 interface configuration.
configure_ifs() {
- local _first _hn _if _ifs _lladdr _name _p _q _vi _vn
+ local _first _hn _if _ifs _lladdr _name _p _q _vn
resp=
# Always need lo0 configured.
@@ -1324,14 +1324,12 @@ configure_ifs() {
# physical interface to begin with.
((${#_ifs[*]} == 0)) && break
- # Discover last configured vlan interface and increment its
- # minor for the next offered vlan interface.
- _vi=
- for _if in "${_ifs[@]}"; do
- [[ $_if = vlan+([[:digit:]]) ]] && _vi=${_if#vlan}
+ # Discover the next unconfigured vlan interface
+ _vn=vlan0
+ while isin $_vn "${_ifs[@]}"; do
+ _vn=vlan$(( ${_vn#vlan} + 1 ))
done
- [[ -n $_vi ]] && ((_vi++))
- [[ -n ${_ifs[*]} ]] && _vn="vlan${_vi:-0}"
+
echo "Available network interfaces are: ${_ifs[*]} $_vn."
if [[ $resp == '?' ]]; then
Update share/locale/ctype/en_US.UTF-8.src to Unicode 14.0.0
With the perl update, we get a new version of unicode available to update this file as well. This was just running the script with the new perl version. OK? Index: share/locale/ctype/en_US.UTF-8.src === RCS file: /cvs/src/share/locale/ctype/en_US.UTF-8.src,v retrieving revision 1.12 diff -u -p -r1.12 en_US.UTF-8.src --- share/locale/ctype/en_US.UTF-8.src 16 May 2021 22:48:05 - 1.12 +++ share/locale/ctype/en_US.UTF-8.src 15 Feb 2023 00:47:03 - @@ -1,4 +1,4 @@ -/* $OpenBSD: en_US.UTF-8.src,v 1.12 2021/05/16 22:48:05 afresh1 Exp $ */ +/* $OpenBSD$ */ /* * COPYRIGHT AND PERMISSION NOTICE @@ -40,7 +40,7 @@ ENCODING"UTF8" VARIABLECODESET=UTF-8 -/* Unicode Version 13.0.0 */ +/* Unicode Version 14.0.0 */ /* * U+ - U+007F : Basic Latin @@ -906,14 +906,14 @@ ALPHA 0x06d5 - 0x06dc 0x06e1 - 0x06 ALPHA 0x06ff CONTROL 0x061c DIGIT 0x0660 - 0x0669 0x06f0 - 0x06f9 -GRAPH 0x0600 - 0x061c 0x061e - 0x06ff -PUNCT 0x0606 - 0x060f 0x061b 0x061e - 0x061f 0x066a - 0x066d 0x06d4 +GRAPH 0x0600 - 0x06ff +PUNCT 0x0606 - 0x060f 0x061b 0x061d - 0x061f 0x066a - 0x066d 0x06d4 PUNCT 0x06de 0x06e9 0x06fd - 0x06fe -PRINT 0x0600 - 0x061c 0x061e - 0x06ff +PRINT 0x0600 - 0x06ff SPECIAL 0x0600 - 0x0605 0x0658 0x06dd 0x06df - 0x06e0 0x06ea - 0x06ec SWIDTH0 0x0600 - 0x0605 0x0610 - 0x061a 0x061c 0x064b - 0x065f 0x0670 SWIDTH0 0x06d6 - 0x06dd 0x06df - 0x06e4 0x06e7 - 0x06e8 0x06ea - 0x06ed -SWIDTH1 0x0606 - 0x060f 0x061b 0x061e - 0x064a 0x0660 - 0x066f +SWIDTH1 0x0606 - 0x060f 0x061b 0x061d - 0x064a 0x0660 - 0x066f SWIDTH1 0x0671 - 0x06d5 0x06de 0x06e5 - 0x06e6 0x06e9 0x06ee - 0x06ff TODIGIT < 0x0660 - 0x0669 : 0x > @@ -1005,21 +1005,28 @@ SWIDTH1 0x0860 - 0x086a /* - * U+0870 - U+089F : No_Block + * U+0870 - U+089F : Arabic Extended-B */ +ALPHA 0x0870 - 0x0887 0x0889 - 0x088e +GRAPH 0x0870 - 0x088e 0x0890 - 0x0891 0x0898 - 0x089f +PUNCT 0x0888 +PRINT 0x0870 - 0x088e 0x0890 - 0x0891 0x0898 - 0x089f +SPECIAL 0x0890 - 0x0891 0x0898 - 0x089f +SWIDTH0 0x0890 - 0x0891 0x0898 - 0x089f +SWIDTH1 0x0870 - 0x088e + /* * U+08A0 - U+08FF : Arabic Extended-A */ -ALPHA 0x08a0 - 0x08b4 0x08b6 - 0x08c7 0x08d4 - 0x08df 0x08e3 - 0x08e9 -ALPHA 0x08f0 - 0x08ff -GRAPH 0x08a0 - 0x08b4 0x08b6 - 0x08c7 0x08d3 - 0x08ff -PRINT 0x08a0 - 0x08b4 0x08b6 - 0x08c7 0x08d3 - 0x08ff -SPECIAL 0x08d3 0x08e0 - 0x08e2 0x08ea - 0x08ef -SWIDTH0 0x08d3 - 0x08ff -SWIDTH1 0x08a0 - 0x08b4 0x08b6 - 0x08c7 +ALPHA 0x08a0 - 0x08c9 0x08d4 - 0x08df 0x08e3 - 0x08e9 0x08f0 - 0x08ff +GRAPH 0x08a0 - 0x08ff +PRINT 0x08a0 - 0x08ff +SPECIAL 0x08ca - 0x08d3 0x08e0 - 0x08e2 0x08ea - 0x08ef +SWIDTH0 0x08ca - 0x08ff +SWIDTH1 0x08a0 - 0x08c9 /* @@ -1187,20 +1194,22 @@ TODIGIT < 0x0bf2 1000 > ALPHA 0x0c00 - 0x0c03 0x0c05 - 0x0c0c 0x0c0e - 0x0c10 0x0c12 - 0x0c28 ALPHA 0x0c2a - 0x0c39 0x0c3d - 0x0c44 0x0c46 - 0x0c48 0x0c4a - 0x0c4c -ALPHA 0x0c55 - 0x0c56 0x0c58 - 0x0c5a 0x0c60 - 0x0c63 +ALPHA 0x0c55 - 0x0c56 0x0c58 - 0x0c5a 0x0c5d 0x0c60 - 0x0c63 DIGIT 0x0c66 - 0x0c6f GRAPH 0x0c00 - 0x0c0c 0x0c0e - 0x0c10 0x0c12 - 0x0c28 0x0c2a - 0x0c39 -GRAPH 0x0c3d - 0x0c44 0x0c46 - 0x0c48 0x0c4a - 0x0c4d 0x0c55 - 0x0c56 -GRAPH 0x0c58 - 0x0c5a 0x0c60 - 0x0c63 0x0c66 - 0x0c6f 0x0c77 - 0x0c7f +GRAPH 0x0c3c - 0x0c44 0x0c46 - 0x0c48 0x0c4a - 0x0c4d 0x0c55 - 0x0c56 +GRAPH 0x0c58 - 0x0c5a 0x0c5d 0x0c60 - 0x0c63 0x0c66 - 0x0c6f +GRAPH 0x0c77 - 0x0c7f PUNCT 0x0c77 0x0c7f PRINT 0x0c00 - 0x0c0c 0x0c0e - 0x0c10 0x0c12 - 0x0c28 0x0c2a - 0x0c39 -PRINT 0x0c3d - 0x0c44 0x0c46 - 0x0c48 0x0c4a - 0x0c4d 0x0c55 - 0x0c56 -PRINT 0x0c58 - 0x0c5a 0x0c60 - 0x0c63 0x0c66 - 0x0c6f 0x0c77 - 0x0c7f -SPECIAL 0x0c04 0x0c4d 0x0c78 - 0x0c7e -SWIDTH0 0x0c00 0x0c04 0x0c3e - 0x0c40 0x0c46 - 0x0c48 0x0c4a - 0x0c4d -SWIDTH0 0x0c55 - 0x0c56 0x0c62 - 0x0c63 +PRINT 0x0c3c - 0x0c44 0x0c46 - 0x0c48 0x0c4a - 0x0c4d 0x0c55 - 0x0c56 +PRINT 0x0c58 - 0x0c5a 0x0c5d 0x0c60 - 0x0c63 0x0c66 - 0x0c6f +PRINT 0x0c77 - 0x0c7f +SPECIAL 0x0c04 0x0c3c 0x0c4d 0x0c78 - 0x0c7e +SWIDTH0 0x0c00 0x0c04 0x0c3c 0x0c3e - 0x0c40 0x0c46 - 0x0c48 +SWIDTH0 0x0c4a - 0x0c4d 0x0c55 - 0x0c56 0x0c62 - 0x0c63 SWIDTH1 0x0c01 - 0x0c03 0x0c05 - 0x0c0c 0x0c0e - 0x0c10 0x0c12 - 0x0c28 -SWIDTH1 0x0c2a - 0x0c39 0x0c3d 0x0c41 - 0x0c44 0x0c58 - 0x0c5a +SWIDTH1 0x0c2a - 0x0c39 0x0c3d 0x0c41 - 0x0c44 0x0c58 - 0x0c5a 0x0c5d SWIDTH1 0x0c60 - 0x0c61 0x0c66 - 0x0c6f 0x0c77 - 0x0c7f TODIGIT < 0x0c66 - 0x0c6f : 0x > @@ -1213,23 +1222,23 @@ TODIGIT < 0x0c7c - 0x0c7e : 1 > ALPHA 0x0c80 - 0x0c83 0x0c85 - 0x0c8c 0x0c8e - 0x0c90 0x0c92 - 0x0ca8 ALPHA 0x0caa - 0x0cb3 0x0cb5 - 0x0cb9
OpenBSD perl 5.36.0 - Call for Testing
Good news! I got all our patches updated for perl 5.36.0. To skip the
summary of new features, jump down to the bottom for details on how you
can help test it.
I missed 5.34 due to non-computer projects that got in the way, but that
means this update has twice the cool newness. Skip to the end for
details about getting the patched version and testing it out.
There wasn't a whole lot in perl 5.34, mostly p5p worked on changes to
the governance model. They also came up with an RFC process to try to
overcome some of the bikeshedding that the perl community has turned
into an art form.
The one big new feature in 5.34 was experimental try/catch support, but
you can read all about it in the 5.34 perldelta.
https://metacpan.org/release/XSAWYERX/perl-5.34.0/view/pod/perldelta.pod
On the other hand, that RFC process and the new governance model
unblocked a bunch of changes, the most anticipated being that the long
experimental signatures are now considered stable, with the one caveat
that using @_ inside a signatured sub is still experimental.
In addition, we're up to Unicode 14, which means I'll be sending in an
update to en_US.UTF-8.src before long.
Another really nice new feature is that it's now possible to tell that a
value is a "boolean". That is, `!!1` is a magic "true" value and `!!0`
is magic false (other boolean things like `1 == 1` also return these
booleans). There are new functions that can use these values to find
out if a value is not only "truthy" or "falsy", but whether it is one of
these booleans. The primary use for this is in serializers like JSON
that can now output these values as true and false.
The feature that I'm most excited for, and probably what will make me
`use v5.36` in my scripts is multi-value iteration. Examples from the
docs:
for my ($key, $value) (%hash) { ... }
for my ($left, $right, $gripping) (@moties) { ... }
Another solution to the bikeshedding led to a lot of new built-ins,
which live in a new "builtin" namespace for functions. While you can
import these like any other, they are always available as
`builtin::function()`. This allowed implementing a lot of new
functions, like builtin::trim to strip leading and trailing whitespace
from a string "correctly". The discussion of that was amusing with
several examples with "I just do $this", with the reply of "the bug in
that way is $explanation". A few of the other builtins, like indexed,
true, false, and is_bool go along with the earlier features like
multi-for and stable booleans. Several things just led to more
optimized and easier to access implementations of things that shipped
with perl in other modules already like blessed, weaken, refaddr, and
reftype. This isn't the full list, so go read the details in the delta.
The final big new feature, which presents and is documented as two
features are the experimental defer and finally (for the new try/catch)
blocks. They both run code at the end of a scope, although for
different reasons.
Read the whole delta for a better explanation of these new things as
well as other improvements and new modules.
https://metacpan.org/release/RJBS/perl-5.36.0/view/pod/perldelta.pod
I did additionally pull in the latest version of Compress::Raw::Zlib
that uses the upstream fix for the recent zlib CVE, although we still
link perl against system zlib. The upgrade also allows the tests for
that module to pass with the system zlib on my macppc.
You can see full test results for the architectures I have on Github
https://github.com/afresh1/OpenBSD-perl/tree/blead/build_logs/5.36.0
If you have an architecture that I don't, or just want to try out the
new perl, there are instructions in the README in that repository.
https://github.com/afresh1/OpenBSD-perl
download the patches and scripts
https://github.com/afresh1/OpenBSD-perl/archive/blead.tar.gz
and extract someplace
or git clone https://github.com/afresh1/OpenBSD-perl.git
download perl-5.36.0.tar.gz into the same directory
https://cpan.metacpan.org/authors/id/R/RJ/RJBS/perl-5.36.0.tar.gz
cd to someplace you have room
run /path/to/OpenBSD-perl/build_perl
wait
send me the log file(s) it generates
If it was successful, you can also follow the instructions at the end to
use `make -f Makefile.bsd-wrwapper install` to install it as your system
perl, although that may cause issues with any XS code that you don't
compile yourself, such as those in packages.
There are also some files that have been removed, if you've installed
the new perl:
rm -rf /usr/lib/libperl.so.22.0 \
/usr/libdata/perl5/*/CORE/stadtx_hash.h \
/usr/libdata/perl5/pod/perldos.pod \
/usr/libdata/perl5/pod/perlmacos.pod \
/usr/libdata/perl5/pod/perlnetware.pod \
/usr/libdata/perl5/pod/perlsymbian.pod \
/usr/libdata/perl5/unicore/To/Digit.pl \
/usr/libdata/perl5/unicore/To/Fold.pl \
Re: perl: fix build with clang 15
On Tue, Dec 27, 2022 at 02:22:43AM +0100, Jeremie Courreges-Anglas wrote:
>
> MD5.xs:375:21: error: mixing declarations and code is incompatible with
> standards before C99 [-Werror,-Wdeclaration-after-statement]
> unsigned char *buf = (unsigned char *)(SvPV(ST(2), len));
>
> The build system asks for -Werror *and* -Wdeclaration-after-statement so
> let's fix the code. ok?
I just happened to be in this local patch today updating for 5.36. This
seems OK to me and I'll make sure I update it.
Upstream declares variable there that we don't use.
https://github.com/afresh1/OpenBSD-perl/blob/blead/patches/GOOD/use_our_MD5.patch#L373-L374
> Index: gnu/usr.bin/perl/cpan/Digest-MD5/MD5.xs
> ===
> RCS file: /home/cvs/src/gnu/usr.bin/perl/cpan/Digest-MD5/MD5.xs,v
> retrieving revision 1.20
> diff -u -p -r1.20 MD5.xs
> --- gnu/usr.bin/perl/cpan/Digest-MD5/MD5.xs 1 Mar 2021 23:21:24 -
> 1.20
> +++ gnu/usr.bin/perl/cpan/Digest-MD5/MD5.xs 27 Dec 2022 01:20:54 -
> @@ -371,8 +371,8 @@ context(ctx, ...)
> PPCODE:
> if (items > 2) {
> STRLEN len;
> - ctx->count = SvUV(ST(1)) << 3;
> unsigned char *buf = (unsigned char *)(SvPV(ST(2), len));
> + ctx->count = SvUV(ST(1)) << 3;
> ctx->state[0] = buf[ 0] | (buf[ 1]<<8) | (buf[ 2]<<16) | (buf[
> 3]<<24);
> ctx->state[1] = buf[ 4] | (buf[ 5]<<8) | (buf[ 6]<<16) | (buf[
> 7]<<24);
> ctx->state[2] = buf[ 8] | (buf[ 9]<<8) | (buf[10]<<16) |
> (buf[11]<<24);
>
>
> --
> jca | PGP : 0x1524E7EE / 5135 92C1 AD36 5293 2BDF DDCC 0DFA 74AE 1524 E7EE
Re: Configure interface by lladdr during install
On Fri, Dec 16, 2022 at 04:00:40PM -0800, Andrew Hewus Fresh wrote:
> On Wed, Dec 07, 2022 at 07:35:53PM -0800, Andrew Hewus Fresh wrote:
> > On Wed, Dec 07, 2022 at 09:51:51AM -0800, Andrew Hewus Fresh wrote:
> > > On Wed, Dec 07, 2022 at 10:28:05AM +, Stuart Henderson wrote:
> > > > On 2022/12/06 19:57, Andrew Hewus Fresh wrote:
> > > > > Which interface do you wish to configure? (name, lladdr, '?', or
> > > > > 'done') [vio0] ?
> > > > > Available network interfaces are: vio0 vlan0.
> > > > > vio0: lladdr fe:e1:bb:d1:dd:97
> > > > > Which interface do you wish to configure? (name, lladdr, '?', or
> > > > > 'done') [vio0] fe:e1:bb:d1:dd:97
> > > > > IPv4 address for vio0? (or 'autoconf' or 'none') [autoconf]
> > > > > IPv6 address for vio0? (or 'autoconf' or 'none') [none]
> > > > > Available network interfaces are: vio0 vlan0.
> > > >
> > > > It doesn't seem right to offer vio0 in the list if it was already
> > > > configured via lladdr?
> > >
> > > The installer currently allows re-configuring interfaces and deletes the
> > > hostname.if before doing it. I think the most useful solution is to
> > > check for "the other" configuration for that interface and remove it as
> > > well when reconfiguring.
> >
> > Here's the next version that cleans up hostname.lladdr when configuring
> > by name and hostname.name when configuring by lladdr.
>
> This version is mostly just rebasing onto the recent changes to swap
> priority. There is a little tidying of variables and such mixed in.
Just a couple of minor tidies and such, avoiding some weird constructs
and extra execs. Some tweaks from kn@.
Index: distrib/miniroot/install.sub
===
RCS file: /cvs/src/distrib/miniroot/install.sub,v
retrieving revision 1.1217
diff -u -p -r1.1217 install.sub
--- distrib/miniroot/install.sub16 Dec 2022 17:47:34 - 1.1217
+++ distrib/miniroot/install.sub21 Dec 2022 04:05:42 -
@@ -356,7 +356,7 @@ get_ifs() {
done
}
-# Map an interface to its MAC address if it is unique
+# Map an interface to its MAC address if it is unique.
if_name_to_lladdr() {
local _lladdr
@@ -365,6 +365,16 @@ if_name_to_lladdr() {
[[ -n $_lladdr && -n $(ifconfig -M "$_lladdr") ]] && echo $_lladdr
}
+# Print a list of interface names and unique lladdr.
+get_ifs_and_lladdrs() {
+ local _if
+
+ for _if in $(get_ifs); do
+ echo $_if
+ if_name_to_lladdr $_if
+ done
+}
+
# Return the device name of the disk device $1, which may be a disklabel UID.
get_dkdev_name() {
local _dev=${1#/dev/} _d
@@ -1300,7 +1310,8 @@ ieee80211_config() {
# Set up IPv4 and IPv6 interface configuration.
configure_ifs() {
- local _first _hn _if _name _p _vi _vn
+ local _first _hn _if _ifs _lladdr _name _p _q _vi _vn
+ resp=
# Always need lo0 configured.
ifconfig lo0 inet 127.0.0.1/8
@@ -1309,20 +1320,55 @@ configure_ifs() {
rm -f /tmp/i/mygate
while :; do
+ set -sA _ifs -- $(get_ifs)
+
# Discover last configured vlan interface and increment its
# minor for the next offered vlan interface.
- _vi=$(get_ifs vlan | sed '$!d;s/^vlan//')
+ _vi=
+ for _if in "${_ifs[@]}"; do
+ [[ $_if = vlan+([[:digit:]]) ]] && _vi=${_if#vlan}
+ done
[[ -n $_vi ]] && ((_vi++))
- [[ -n $(get_ifs) ]] && _vn="vlan${_vi:-0}"
+ [[ -n ${_ifs[*]} ]] && _vn="vlan${_vi:-0}"
+
+ echo "Available network interfaces are: ${_ifs[*]} $_vn."
+ if [[ $resp == '?' ]]; then
+ for _if in "${_ifs[@]}"; do
+ _lladdr=$(if_name_to_lladdr $_if)
+ [[ -n $_lladdr ]] && echo " $_if: lladdr
$_lladdr"
+ done
+ fi
+
+ _q="Network interface to configure?"
+ ask_until "$_q (name, lladdr, '?', or 'done')" \
+ ${_p:-$( (get_ifs netboot; get_ifs) | sed q )}
- ask_which "network interface" "do you wish to configure" \
- "\$(get_ifs) $_vn" \
- ${_p:-'$( (get_ifs netboot; get_ifs) | sed q )'}
[[ $resp == done ]] && break
+ [[ $resp == '?' ]] && continue
+
+ # Quote $resp to p
Re: Configure interface by lladdr during install
On Wed, Dec 07, 2022 at 07:35:53PM -0800, Andrew Hewus Fresh wrote:
> On Wed, Dec 07, 2022 at 09:51:51AM -0800, Andrew Hewus Fresh wrote:
> > On Wed, Dec 07, 2022 at 10:28:05AM +, Stuart Henderson wrote:
> > > On 2022/12/06 19:57, Andrew Hewus Fresh wrote:
> > > > Which interface do you wish to configure? (name, lladdr, '?', or
> > > > 'done') [vio0] ?
> > > > Available network interfaces are: vio0 vlan0.
> > > > vio0: lladdr fe:e1:bb:d1:dd:97
> > > > Which interface do you wish to configure? (name, lladdr, '?', or
> > > > 'done') [vio0] fe:e1:bb:d1:dd:97
> > > > IPv4 address for vio0? (or 'autoconf' or 'none') [autoconf]
> > > > IPv6 address for vio0? (or 'autoconf' or 'none') [none]
> > > > Available network interfaces are: vio0 vlan0.
> > >
> > > It doesn't seem right to offer vio0 in the list if it was already
> > > configured via lladdr?
> >
> > The installer currently allows re-configuring interfaces and deletes the
> > hostname.if before doing it. I think the most useful solution is to
> > check for "the other" configuration for that interface and remove it as
> > well when reconfiguring.
>
> Here's the next version that cleans up hostname.lladdr when configuring
> by name and hostname.name when configuring by lladdr.
This version is mostly just rebasing onto the recent changes to swap
priority. There is a little tidying of variables and such mixed in.
Index: distrib/miniroot/install.sub
===
RCS file: /cvs/src/distrib/miniroot/install.sub,v
retrieving revision 1.1216
diff -u -p -r1.1216 install.sub
--- distrib/miniroot/install.sub16 Dec 2022 04:04:25 - 1.1216
+++ distrib/miniroot/install.sub16 Dec 2022 23:59:39 -
@@ -365,6 +365,16 @@ if_name_to_lladdr() {
[[ -n $_lladdr && -n $(ifconfig -M "$_lladdr") ]] && echo $_lladdr
}
+# Print a list of interface names and lladdr for validating allowed interfaces.
+get_ifs_and_lladdrs() {
+ local _if
+
+ for _if in $(get_ifs); do
+ echo $_if
+ if_name_to_lladdr $_if
+ done
+}
+
# Return the device name of the disk device $1, which may be a disklabel UID.
get_dkdev_name() {
local _dev=${1#/dev/} _d
@@ -1300,7 +1310,8 @@ ieee80211_config() {
# Set up IPv4 and IPv6 interface configuration.
configure_ifs() {
- local _first _hn _if _name _p _vi _vn
+ local _first _hn _if _lladdr _name _p _q _vi _vn
+ resp=
# Always need lo0 configured.
ifconfig lo0 inet 127.0.0.1/8
@@ -1315,14 +1326,46 @@ configure_ifs() {
[[ -n $_vi ]] && ((_vi++))
[[ -n $(get_ifs) ]] && _vn="vlan${_vi:-0}"
- ask_which "network interface" "do you wish to configure" \
- "\$(get_ifs) $_vn" \
- ${_p:-'$( (get_ifs netboot; get_ifs) | sed q )'}
+ set -- $(get_ifs) $_vn; _q="$*"
+ echo "Available network interfaces are: $_q."
+ if [[ $resp == \? ]]; then
+ for _if in $(get_ifs); do
+ _lladdr=$(if_name_to_lladdr $_if)
+ [[ -n $_lladdr ]] && echo " $_if: lladdr
$_lladdr"
+ done
+ fi
+
+ _q="Network interface to configure?"
+ ask_until "$_q (name, lladdr, '?', or 'done')" \
+ ${_p:-$( (get_ifs netboot; get_ifs) | sed q )}
+
[[ $resp == done ]] && break
+ [[ $resp == \? ]] && continue
+
+ # Quote $resp to prevent user from confusing isin() by
+ # entering something like 'a a'.
+ if ! isin "$resp" $(get_ifs_and_lladdrs) $_vn done; then
+ echo "'$resp' is not a valid choice."
+ $AI && [[ -n $AI_RESPFILE ]] && exit 1
+ continue
+ fi
_if=$resp
_hn=/tmp/i/hostname.$_if
rm -f $_hn
+
+ # Test if we're configuring by lladdr
+ # If we are, map it to the interface name
+ # in either case remove the other config for this interface
+ if [[ $_if == ??:??:??:??:??:?? ]]; then
+ _lladdr=$_if
+ _if=$(ifconfig -M $_lladdr)
+ [[ -z $_if ]] && continue # should not be possible
+ rm -f /tmp/i/hostname.$_if
+ else
+ _lladdr=$(if_name_to_lladdr $_if)
+ [[ -n $_lladdr ]] && rm -f /tmp/i/hostname.$_lladdr
+ fi
# If the offered vlan is chosen, ask the relevant
# questions and bring it up.
Re: Switch priority of hostname.if lladdr over name
On Thu, Dec 15, 2022 at 06:31:40AM +, Klemens Nanni wrote:
> On Sun, Dec 11, 2022 at 01:05:13PM -0800, Andrew Hewus Fresh wrote:
> > This would be the diff to swap priority of name/lladdr as some folks
> > wanted to see. I still don't recommend having both as you can still
> > have surprising outcomes with more complex configurations.
> >
> > The install.sub includes an "if_name_to_lladdr" function from the
> > diff to support creating a hostname.lladdr during install. If we don't
> > decide to support that, I'd probably in-line that as is done in
> > netstart.
> >
> > I did find that my iwm doesn't like being configured by lladdr at this
> > point of the upgrade because it doesn't have firmware loaded when it's
> > looking to see if it's a valid hostname.lladdr, but the iwm's lladdr is
> > all zeros still.
> >
> >
> > Index: etc/netstart
> > ===
> > RCS file: /cvs/src/etc/netstart,v
> > retrieving revision 1.230
> > diff -u -p -r1.230 netstart
> > --- etc/netstart5 Dec 2022 20:12:00 - 1.230
> > +++ etc/netstart11 Dec 2022 20:44:34 -
>
> More context for this diff:
> 138 if [[ $_if == ??:??:??:??:??:?? ]]; then
> 139 if ! _line=$( ifconfig -M $_if ); then
> 140 print -u2 "${0##*/}: $_if is not unique."
> 141 return
> 142 fi
>
> > @@ -144,11 +144,14 @@ ifstart() {
> > [[ -z $_line ]] && return
>
> _line is the interface name "em0", please call it _ifname or _if_name.
> Just anything obviously descriptive.
Will do. I ended up just adding _lladdr and moving things around
appropriately. I think it worked out very well.
> > _if=$_line
> > _line=
>
> So the MAC in _if is overwritten with the interface name in _line which
> is being zeroed, i.e. we looked up the interface via lladdr and proceed
> as if the user had placed an .em0 file instead.
>
> With s/_inline/_ifname/ that would be much easier to follow.
>
> > -
> > - if [[ -e /etc/hostname.$_if ]]; then
> > - print -u2 "${0##*/}: $_hn: /etc/hostname.$_if overrides"
> > + else
> > + _line=$(ifconfig $_if | sed -n
> > 's/^[[:space:]]*lladdr[[:space:]]//p')
>
> This will print "_if: no such interface" for nonexistent interfaces and
> must be silenced.
Ahh, yes. fixed.
> > + if [[ -n $_line && -n $(ifconfig -M $_line) \
>
> Pretty sure the -M call needs "$_line" since you can hit existent
> interfaces without lladdr, so _line is empty and `ifconfig -M' would
> print usage while `ifconfig -M ""' is silent as expected.
Wouldn't $_line be blank then and this would short-circuit at "-n $line"
instead of trying to run ifconfig -M with no arguments?
$ x=; [[ -n $x && -n $( print -u2 $x!; ifconfig -M $x ) ]] && echo oops
$
I will wrap it in quotes to be less surprising.
>
> > + && -e /etc/hostname.$_line ]]; then
>
> The same sanity check further below uses -f. Not sure if existence is
> enough here or whether we should only switch files if there is an actual
> file...
We should be consistent in ignoring things that aren't files.
>
> > + print -u2 "${0##*/}: $_hn: /etc/hostname.$_line:
> > overrides"
> > return
> > fi
> > + _line=
> > fi
>
> Otherwise the else block logic makes sense, but now you're using _if as
> an interface name before the input validation, which follows immediately
> afterwards.
>
> Just merge the two like this:
> if [[ $_if == ??:??:??:??:??:?? ]]; then
> ...
> else if [[ $_if == +([[:alpha:]])+([[:digit:]]) ]]; then
> ...
> else
> return
>
> Diff below to make things clear.
> Maybe it'd be better to clean up things a bit and then do the prio swap
> separately?
I tried just adding a variable and doing a bit of tidying first, but it
didn't seem worthwhile.
After seeing your suggestion to combine the logic into an elif, a
lightbulb went on and I reversed the order of the checks so the more
common (name/unit) would be handled first. Overall I think it feels
much nicer. Thanks!
I used many of your suggestions, but I kept sed instead of switching to
awk to keep that the same with install.sub. It is still in a helper
function in install.sub, since I believe we are going to reuse that
logic later when we allow that answer to the q
Re: netstart: do not try to create physical interfaces
OK afresh1@, I would have to dig for a device to actually test it with,
but my reading of the code makes me think it's correct.
I do wonder if moving the ifcreate logic into vifscreate would allow for
a friendlier message from `-n`.
(untested diff follows, also missing the deletion of ifcreate)
Index: etc/netstart
===
RCS file: /cvs/src/etc/netstart,v
retrieving revision 1.230
diff -u -p -r1.230 netstart
--- etc/netstart5 Dec 2022 20:12:00 - 1.230
+++ etc/netstart15 Dec 2022 01:13:13 -
@@ -122,8 +122,12 @@ vifscreate() {
continue
fi
- if ! ifcreate $_if; then
- print -u2 "${0##*/}: create for '$_if' failed."
+ if ! ifconfig $_if >/dev/null 2>&1; then
+ if $PRINT_ONLY; then
+ print -r -- "ifconfig $_if create"
+ elif ! ifconfig $_if create >/dev/null 2>&1;
then
+ print -u2 "${0##*/}: create for '$_if'
failed."
+ fi
fi
done
done
On Wed, Nov 09, 2022 at 01:15:08PM +, Klemens Nanni wrote:
> On Tue, Nov 01, 2022 at 01:57:21PM +, Klemens Nanni wrote:
> > vifscreate() is used to create all virtual interfaces up-front and is
> > always called at the beginning of netstart, whether an explicit list of
> > interfaces is passed or none, i.e. all are to be configured.
> >
> > Yet, to check the given interface exists, ifstart() uses ifcreate()
> > which obviously tries to create interfaces.
> >
> > When ifstart() is run for every hostname.if(5) file, every virtual
> > interface is guaranteed to exist thanks to vifscreate().
> >
> > Nonexistent physical interfaces with an existent config, e.g. urndis(4)
> > and cdce(4), will be skipped by ifstart() due to the failed ifcreate()
> > call, but not without ifcreate() trying the impossible:
> >
> > $ ifconfig urndis0
> > urndis0: no such interface
> > # sh /etc/netstart -n urndis0
> > { ifconfig urndis0 || ifconfig urndis0 create; }
> > ifconfig urndis0 inet6 autoconf
> > ifconfig urndis0 inet autoconf
> >
> > This dry-run output does NOT match what netstart would really do:
> >
> > # sh -x /etc/netstart urndis0 2>&1 | tail -n4
> > + vifscreate urndis0
> > + ifstart urndis0
> > + defaultroute
> > + return
> >
> > Here, ifstart() runs but bails out on the failing ifcreate() call and
> > thus skips configuring urndis0 entirely.
> >
> >
> > So clarify the comment and replace the ifcreate() call with a simpler,
> > more obvious `ifconfig' check, which is exactly what ifcreate() boils
> > down to for existing interfaces:
> >
> > # sh ./netstart -n urndis0 ; echo $?
> > 0
> >
> > Actual steps taken remain the same, i.e. none, as the new dry-run output
> > truthfully tells:
> >
> > # sh -x ./netstart urndis0 2>&1 | tail -n4
> > + vifscreate urndis0
> > + ifstart urndis0
> > + defaultroute
> > + return
> >
> >
> > Virtual interfaces are now also created only once:
> >
> > # sh /etc/netstart -n veb0
> > { ifconfig veb0 || ifconfig veb0 create; }
> > { ifconfig veb0 || ifconfig veb0 create; }
> > ifconfig veb0 description 'vmd(4) uplink'
> > ifconfig veb0 up
> > # sh ./netstart -n veb0
> > { ifconfig veb0 || ifconfig veb0 create; }
> > ifconfig veb0 description 'vmd(4) uplink'
> > ifconfig veb0 up
> >
> > Feedback? OK?
>
> Ping.
>
> netstart and the installer's copy still remain in sync as best as
> possible: the installer keeps creating interfaces as it doesn't have
> any of this create-virtual-interfaces-up-front logic at all.
>
> Index: netstart
> ===
> RCS file: /cvs/src/etc/netstart,v
> retrieving revision 1.229
> diff -u -p -r1.229 netstart
> --- netstart 5 Nov 2022 12:06:05 - 1.229
> +++ netstart 9 Nov 2022 13:14:46 -
> @@ -152,8 +152,8 @@ ifstart() {
> chown -LR root:wheel $_hn
> fi
>
> - # Check for ifconfig'able interface, except if -n option is specified.
> - ifcreate $_if || return
> + # Skip missing physical interface, virtual ones were created up front.
> + ifconfig $_if >/dev/null 2>&1 || return
>
> # Parse the hostname.if(5) file and fill _cmds array with interface
> # configuration commands.
>
--
andrew
Real programmers don't document.
If it was hard to write, it should be hard to understand.
Re: hostname.if(5): lladdr tweaks
On Mon, Dec 12, 2022 at 06:48:36AM +, Jason McIntyre wrote: > On Fri, Dec 09, 2022 at 09:32:13PM +, Jason McIntyre wrote: > > hi. > > > > two points about the recent ability to use lladdr: > > > > - the example of "bridge0" made sense when bridge was regarded as a > > separate entity and not integrated with ifconfig. plus a list of one > > example looks rubbish. now that we have a second example (lladdr) and > > bridge is not flagged as a special case, i think we can simply the > > text and reduce it to two examples I thought it was an example showing that it works for "dynamic" interfaces, but that's probably obvious enough without. > > - i'm not sure about using "lladdr". although we use this term in > > ifconfig(8), we explain it. and people may miss it if they are thinking > > of mac address. i've attempted to both write the term fully as "link > > layer local address" and add a "(MAC)". i suppose you could argue that > > people who think of the term as "lladdr" might miss that (!) but i > > don;t think that is a real worry. I think that since `ifconfig` prints out "lladdr" that even folks who don't know what it means should be able to figure out how to use it. I am a fan of defining terms though. > > so here's my cut at tweaking... > > jmc > > > > on the back of afresh's prioroty flip diff, here's a revision of my diff > after some input from deraadt. it will need adjusting again if we flip > priority... This looks OK to me. I'm waiting to see if kn@ or others have feedback on the swap and other installer bits before committing, but it seems the idea of preferring lladdr has been given the go-ahead, so it's just a matter of time. I tend think of the interface name as its driver + number, so I did find the distinction that the name is just the part without the number to be surprising. ifconfig(8) does say it is "name unit" with the "name" not including the number, so this is correct. > jmc > > Index: hostname.if.5 > === > RCS file: /cvs/src/share/man/man5/hostname.if.5,v > retrieving revision 1.80 > diff -u -p -r1.80 hostname.if.5 > --- hostname.if.5 5 Dec 2022 20:12:00 - 1.80 > +++ hostname.if.5 12 Dec 2022 06:48:15 - > @@ -40,13 +40,16 @@ > The > .Nm hostname.*\& > files contain information regarding the configuration of each network > interface. > -The interface can be referenced by name or lladdr, such as > -.Pa hostname.fxp0 , > -.Pa hostname.00:00:5e:00:53:af , > -or > -.Pa hostname.bridge0 . > -One file should exist for each interface that is to be configured, > -with priority given to configuration by interface name over lladdr. > +Interfaces are referenced by name and number, > +such as > +.Dq hostname.fxp0 . > +For some machines, > +autoconfiguration makes this system inconsistent, > +so interfaces can alternatively be referenced by > +their link layer address (lladdr), > +such as > +.Dq hostname.00:00:5e:00:53:af . > +Priority is given to configuration by interface name/number over lladdr. > A configuration file is not needed for lo0. > .Pp > The configuration information is expressed in a line-by-line packed format >
Switch priority of hostname.if lladdr over name
This would be the diff to swap priority of name/lladdr as some folks
wanted to see. I still don't recommend having both as you can still
have surprising outcomes with more complex configurations.
The install.sub includes an "if_name_to_lladdr" function from the
diff to support creating a hostname.lladdr during install. If we don't
decide to support that, I'd probably in-line that as is done in
netstart.
I did find that my iwm doesn't like being configured by lladdr at this
point of the upgrade because it doesn't have firmware loaded when it's
looking to see if it's a valid hostname.lladdr, but the iwm's lladdr is
all zeros still.
Index: etc/netstart
===
RCS file: /cvs/src/etc/netstart,v
retrieving revision 1.230
diff -u -p -r1.230 netstart
--- etc/netstart5 Dec 2022 20:12:00 - 1.230
+++ etc/netstart11 Dec 2022 20:44:34 -
@@ -144,11 +144,14 @@ ifstart() {
[[ -z $_line ]] && return
_if=$_line
_line=
-
- if [[ -e /etc/hostname.$_if ]]; then
- print -u2 "${0##*/}: $_hn: /etc/hostname.$_if overrides"
+ else
+ _line=$(ifconfig $_if | sed -n
's/^[[:space:]]*lladdr[[:space:]]//p')
+ if [[ -n $_line && -n $(ifconfig -M $_line) \
+ && -e /etc/hostname.$_line ]]; then
+ print -u2 "${0##*/}: $_hn: /etc/hostname.$_line:
overrides"
return
fi
+ _line=
fi
# Interface names must be alphanumeric only. We check to avoid
Index: distrib/miniroot/install.sub
===
RCS file: /cvs/src/distrib/miniroot/install.sub,v
retrieving revision 1.1215
diff -u -p -r1.1215 install.sub
--- distrib/miniroot/install.sub5 Dec 2022 20:12:00 - 1.1215
+++ distrib/miniroot/install.sub11 Dec 2022 20:44:34 -
@@ -356,6 +356,15 @@ get_ifs() {
done
}
+# Maps an interface name to lladdr,
+# filtered by whether it's valid for use by ifconfig -M
+if_name_to_lladdr() {
+ local _lladdr
+ _lladdr=$(ifconfig $1 2>/dev/null |
+ sed -n 's/^[[:space:]]*lladdr[[:space:]]//p')
+ [[ -n $_lladdr && -n $(ifconfig -M $_lladdr) ]] && echo $_lladdr
+}
+
# Return the device name of the disk device $1, which may be a disklabel UID.
get_dkdev_name() {
local _dev=${1#/dev/} _d
@@ -2434,7 +2443,10 @@ ifstart() {
if [[ $_if == ??:??:??:??:??:?? ]]; then
_if=$(ifconfig -M $_if)
[[ -z $_if ]] && return # invalid interface
- [[ -e /mnt/etc/hostname.$_if ]] && return # duplicate config
+ else
+ _line=$(if_name_to_lladdr $_if)
+ [[ -n $_line && -e /mnt/etc/hostname.$_line ]] && continue
+ _line=
fi
# Create interface if it does not yet exist.
Index: share/man/man5/hostname.if.5
===
RCS file: /cvs/src/share/man/man5/hostname.if.5,v
retrieving revision 1.80
diff -u -p -r1.80 hostname.if.5
--- share/man/man5/hostname.if.55 Dec 2022 20:12:00 - 1.80
+++ share/man/man5/hostname.if.511 Dec 2022 20:44:34 -
@@ -46,7 +46,7 @@ The interface can be referenced by name
or
.Pa hostname.bridge0 .
One file should exist for each interface that is to be configured,
-with priority given to configuration by interface name over lladdr.
+with priority given to configuration by interface lladdr over name.
A configuration file is not needed for lo0.
.Pp
The configuration information is expressed in a line-by-line packed format
Re: Configure interface by lladdr during install
On Wed, Dec 07, 2022 at 09:51:51AM -0800, Andrew Hewus Fresh wrote:
> On Wed, Dec 07, 2022 at 10:28:05AM +, Stuart Henderson wrote:
> > On 2022/12/06 19:57, Andrew Hewus Fresh wrote:
> > > Which interface do you wish to configure? (name, lladdr, '?', or 'done')
> > > [vio0] ?
> > > Available network interfaces are: vio0 vlan0.
> > > vio0: lladdr fe:e1:bb:d1:dd:97
> > > Which interface do you wish to configure? (name, lladdr, '?', or 'done')
> > > [vio0] fe:e1:bb:d1:dd:97
> > > IPv4 address for vio0? (or 'autoconf' or 'none') [autoconf]
> > > IPv6 address for vio0? (or 'autoconf' or 'none') [none]
> > > Available network interfaces are: vio0 vlan0.
> >
> > It doesn't seem right to offer vio0 in the list if it was already
> > configured via lladdr?
>
> The installer currently allows re-configuring interfaces and deletes the
> hostname.if before doing it. I think the most useful solution is to
> check for "the other" configuration for that interface and remove it as
> well when reconfiguring.
Here's the next version that cleans up hostname.lladdr when configuring
by name and hostname.name when configuring by lladdr.
Available network interfaces are: vio0 vio1 vio2 vio3 vlan0.
Network interface to configure? (name, lladdr, '?', or 'done') [vio0]
IPv4 address for vio0? (or 'autoconf' or 'none') [autoconf] none
IPv6 address for vio0? (or 'autoconf' or 'none') [none] autoconf
Available network interfaces are: vio0 vio1 vio2 vio3 vlan0.
Network interface to configure? (name, lladdr, '?', or 'done') [done] ?
Available network interfaces are: vio0 vio1 vio2 vio3 vlan0.
vio0: lladdr fe:e1:bb:d1:dd:62
vio1: lladdr fe:e1:bb:d2:d2:99
vio2: lladdr fe:e1:bb:d3:2c:eb
vio3: lladdr fe:e1:bb:d4:08:54
Network interface to configure? (name, lladdr, '?', or 'done') [done]
fe:e1:bb:d1:dd:a2
'fe:e1:bb:d1:dd:a2' is not a valid choice.
Available network interfaces are: vio0 vio1 vio2 vio3 vlan0.
Network interface to configure? (name, lladdr, '?', or 'done') [done]
fe:e1:bb:d1:dd:62
IPv4 address for vio0? (or 'autoconf' or 'none') [autoconf] none
IPv6 address for vio0? (or 'autoconf' or 'none') [autoconf]
Available network interfaces are: vio0 vio1 vio2 vio3 vlan0.
Network interface to configure? (name, lladdr, '?', or 'done') [done] vio1
Symbolic (host) name for vio1? [foo]
IPv4 address for vio1? (or 'autoconf' or 'none') [autoconf] none
IPv6 address for vio1? (or 'autoconf' or 'none') [none] autoconf
Available network interfaces are: vio0 vio1 vio2 vio3 vlan0.
Network interface to configure? (name, lladdr, '?', or 'done') [done] !
Type 'exit' to return to install.
foo# grep . /tmp/i/hostname.*
/tmp/i/hostname.fe:e1:bb:d1:dd:62:inet6 autoconf
/tmp/i/hostname.vio1:inet6 autoconf
foo# exit
Network interface to configure? (name, lladdr, '?', or 'done') [done] vio0
IPv4 address for vio0? (or 'autoconf' or 'none') [autoconf] none
IPv6 address for vio0? (or 'autoconf' or 'none') [autoconf]
Available network interfaces are: vio0 vio1 vio2 vio3 vlan0.
Network interface to configure? (name, lladdr, '?', or 'done') [done] !
Type 'exit' to return to install.
foo# grep . /tmp/i/hostname.*
/tmp/i/hostname.vio0:inet6 autoconf
/tmp/i/hostname.vio1:inet6 autoconf
Index: distrib/miniroot/install.sub
===
RCS file: /cvs/src/distrib/miniroot/install.sub,v
retrieving revision 1.1215
diff -u -p -r1.1215 install.sub
--- distrib/miniroot/install.sub5 Dec 2022 20:12:00 - 1.1215
+++ distrib/miniroot/install.sub8 Dec 2022 03:22:07 -
@@ -356,6 +356,24 @@ get_ifs() {
done
}
+# Maps an interface name to lladdr,
+# filtered by whether it's valid for use by ifconfig -M
+if_name_to_lladdr() {
+ local _lladdr
+ _lladdr=$(ifconfig $1 | sed -n 's/^[[:space:]]*lladdr[[:space:]]//p')
+ [[ -n $_lladdr && -n $(ifconfig -M $_lladdr) ]] && echo $_lladdr
+}
+
+# Prints a list of interface names and lladdr from if_name_to_lladdr
+# to allow validating how we're allowed to configure interfaces.
+get_ifs_and_lladdrs() {
+ local _if
+ for _if in $(get_ifs); do
+ echo $_if
+ if_name_to_lladdr $_if
+ done
+}
+
# Return the device name of the disk device $1, which may be a disklabel UID.
get_dkdev_name() {
local _dev=${1#/dev/} _d
@@ -1291,7 +1309,8 @@ ieee80211_config() {
# Set up IPv4 and IPv6 interface configuration.
configure_ifs() {
- local _first _hn _if _name _p _vi _vn
+ local _first _hn _if _name _p _q _vi _vn
+ resp=
# Always need lo0 configured.
ifconfig lo0 inet 127.0.0.1/8
@@ -1306,14 +1325,48 @@ configure_ifs() {
[[ -n $_vi ]] && ((_vi++))
[[ -n $(get_ifs) ]] && _vn="vlan${_vi:-0}"
-
Re: Configure interface by lladdr during install
On Wed, Dec 07, 2022 at 10:28:05AM +, Stuart Henderson wrote: > On 2022/12/06 19:57, Andrew Hewus Fresh wrote: > > Which interface do you wish to configure? (name, lladdr, '?', or 'done') > > [vio0] ? > > Available network interfaces are: vio0 vlan0. > > vio0: lladdr fe:e1:bb:d1:dd:97 > > Which interface do you wish to configure? (name, lladdr, '?', or 'done') > > [vio0] fe:e1:bb:d1:dd:97 > > IPv4 address for vio0? (or 'autoconf' or 'none') [autoconf] > > IPv6 address for vio0? (or 'autoconf' or 'none') [none] > > Available network interfaces are: vio0 vlan0. > > It doesn't seem right to offer vio0 in the list if it was already > configured via lladdr? The installer currently allows re-configuring interfaces and deletes the hostname.if before doing it. I think the most useful solution is to check for "the other" configuration for that interface and remove it as well when reconfiguring. > (Brushing aside the issue that fe:e1:b[ab]:dX:XX:XX addresses > are usually random so the hostname.lladdr scheme won't work for > them anyway..) This is true, but testing stuff in vmm is so handy!
Re: Configure interface by lladdr during install
On Tue, Dec 06, 2022 at 06:11:33PM -0800, Andrew Hewus Fresh wrote:
> Here's an initial draft of a diff to allow configuring by lladdr in the
> installer. I'm not a super fan of how it works, but could use some
> feedback from folks trying it and suggesting improvements.
>
> It's a larger diff than I first expected since I didn't want to include
> the lladdrs in the "available interfaces" list, but ask_which() doesn't
> allow two lists.
Here's a second try, with some nits picked and some bugs squashed.
The output from "?" is quite a bit less verbose now, and although I'd
like to be able to see which interface has the cable plugged in right
there, it's hard to know where to stop.
The main thing that still needs addressing is that the question worded
as it was wraps and really shouldn't. I didn't want to reword the question and
break folks answer files, but we may need to. I'll keep thinking about how
best to handle this. Especially as when answering with an lladdr it makes
things really long.
Now though, dinner time.
System hostname? (short form, e.g. 'foo') foo
Available network interfaces are: vio0 vlan0.
Which interface do you wish to configure? (name, lladdr, '?', or 'done') [vio0]
?
Available network interfaces are: vio0 vlan0.
vio0: lladdr fe:e1:bb:d1:dd:97
Which interface do you wish to configure? (name, lladdr, '?', or 'done') [vio0]
fe:e1:bb:d1:dd:97
IPv4 address for vio0? (or 'autoconf' or 'none') [autoconf]
IPv6 address for vio0? (or 'autoconf' or 'none') [none]
Available network interfaces are: vio0 vlan0.
Which interface do you wish to configure? (name, lladdr, '?', or 'done') [done]
?
Available network interfaces are: vio0 vlan0.
vio0: lladdr fe:e1:bb:d1:dd:97
Which interface do you wish to configure? (name, lladdr, '?', or 'done') [done]
Using DNS domainname my.domain
Using DNS nameservers at 100.64.1.2
Index: distrib/miniroot/install.sub
===
RCS file: /cvs/src/distrib/miniroot/install.sub,v
retrieving revision 1.1215
diff -u -p -r1.1215 install.sub
--- distrib/miniroot/install.sub5 Dec 2022 20:12:00 - 1.1215
+++ distrib/miniroot/install.sub7 Dec 2022 03:51:38 -
@@ -356,6 +356,17 @@ get_ifs() {
done
}
+get_ifs_and_lladdrs() {
+ local _if _lladdr
+ for _if in $(get_ifs); do
+ echo $_if
+ _lladdr=$(ifconfig $_if |
+ sed -n 's/^[[:space:]]*lladdr[[:space:]]//p' )
+ [[ -n $_lladdr && -n $(ifconfig -M $_lladdr) ]] &&
+ echo $_lladdr
+ done
+}
+
# Return the device name of the disk device $1, which may be a disklabel UID.
get_dkdev_name() {
local _dev=${1#/dev/} _d
@@ -1291,7 +1302,7 @@ ieee80211_config() {
# Set up IPv4 and IPv6 interface configuration.
configure_ifs() {
- local _first _hn _if _name _p _vi _vn
+ local _first _hn _if _name _p _q _vi _vn
# Always need lo0 configured.
ifconfig lo0 inet 127.0.0.1/8
@@ -1306,12 +1317,41 @@ configure_ifs() {
[[ -n $_vi ]] && ((_vi++))
[[ -n $(get_ifs) ]] && _vn="vlan${_vi:-0}"
- ask_which "network interface" "do you wish to configure" \
- "\$(get_ifs) $_vn" \
- ${_p:-'$( (get_ifs netboot; get_ifs) | sed q )'}
- [[ $resp == done ]] && break
+ set -- $(get_ifs) $_vn; _q="$*"
+ echo "Available network interfaces are: $_q."
+ if [[ $_if = DETAILS ]]; then
+ for _if in $(get_ifs); do
+ _name=$(ifconfig $_if |
+ sed -n
's/^[[:space:]]*lladdr[[:space:]]//p' )
+ [[ -n $_name && -n $(ifconfig -M $_name) ]] &&
+ echo " $_if: lladdr $_name"
+ done
+ _if=
+ _name=
+ fi
+ _q="Which interface do you wish to configure?"
+ ask_until "$_q (name, lladdr, '?', or 'done')" \
+ ${_p:-$( (get_ifs netboot; get_ifs) | sed q )}
+ case $resp in
+ done) break
+ ;;
+ \?) _if=DETAILS
+ continue
+ ;;
+ *) # Quote $resp to prevent user from confusing isin() by
+ # entering something like 'a a'.
+ if isin "$resp" $( get_ifs_and_lladdrs ) $_vn done; then
+ log_answers "$_q" "$resp"
+ _if=$resp
+ else
+ ec
Configure interface by lladdr during install
Here's an initial draft of a diff to allow configuring by lladdr in the
installer. I'm not a super fan of how it works, but could use some
feedback from folks trying it and suggesting improvements.
It's a larger diff than I first expected since I didn't want to include
the lladdrs in the "available interfaces" list, but ask_which() doesn't
allow two lists.
Index: distrib/miniroot/install.sub
===
RCS file: /cvs/src/distrib/miniroot/install.sub,v
retrieving revision 1.1215
diff -u -p -r1.1215 install.sub
--- distrib/miniroot/install.sub5 Dec 2022 20:12:00 - 1.1215
+++ distrib/miniroot/install.sub7 Dec 2022 02:08:03 -
@@ -356,6 +356,16 @@ get_ifs() {
done
}
+get_ifs_and_lladdrs() {
+ local _if _lladdr
+ for _if in $(get_ifs); do
+ _lladdr=$(ifconfig $_if |
+ sed -n 's/^[[:space:]]*lladdr[[:space:]]//p' )
+ [[ -n $_lladdr && -n $(ifconfig -M $_lladdr) ]] &&
+ echo $_lladdr
+ done
+}
+
# Return the device name of the disk device $1, which may be a disklabel UID.
get_dkdev_name() {
local _dev=${1#/dev/} _d
@@ -1291,7 +1301,7 @@ ieee80211_config() {
# Set up IPv4 and IPv6 interface configuration.
configure_ifs() {
- local _first _hn _if _name _p _vi _vn
+ local _first _hn _if _name _p _q _vi _vn
# Always need lo0 configured.
ifconfig lo0 inet 127.0.0.1/8
@@ -1306,12 +1316,33 @@ configure_ifs() {
[[ -n $_vi ]] && ((_vi++))
[[ -n $(get_ifs) ]] && _vn="vlan${_vi:-0}"
- ask_which "network interface" "do you wish to configure" \
- "\$(get_ifs) $_vn" \
- ${_p:-'$( (get_ifs netboot; get_ifs) | sed q )'}
- [[ $resp == done ]] && break
+ echo "Available network interfaces are: $(get_ifs) $_vn."
+ _q="Which network interface do you wish to configure?"
+ ask_until "$_q (name, lladdr, '?', or 'done')" \
+ ${_p:-$( (get_ifs netboot; get_ifs) | sed q )}
+ case $resp in
+ done) break
+ ;;
+ \?) for _name in $(get_ifs); do
+ ifconfig $_name | grep -e ^[^[:space:]] \
+ -e ^[[:space:]]*lladdr[[:space:]] \
+ -e ^[[:space:]]*status[[:space:]]
+ done
+ ;;
+ *) # Quote $resp to prevent user from confusing isin() by
+ # entering something like 'a a'.
+ if isin "$resp" $( get_ifs_and_lladdrs ) $_vn done; then
+ log_answers "$_q" "$resp"
+ _if=$resp
+ else
+ echo "'$resp' is not a valid choice."
+ $AI && [[ -n $AI_RESPFILE ]] && exit 1
+ fi
+ ;;
+ esac
+
+ [[ -z $_if ]] && continue
- _if=$resp
_hn=/tmp/i/hostname.$_if
rm -f $_hn
@@ -1319,6 +1350,15 @@ configure_ifs() {
# questions and bring it up.
if [[ $_if == vlan+([0-9]) ]]; then
vlan_config $_if || continue
+ fi
+
+ # Test if we're configuring by lladdr
+ if [[ $_if == ??:??:??:??:??:?? ]]; then
+ _if=$( ifconfig -M $_if )
+ if [[ -z $_if ]]; then
+ echo "'${#/tmp/i/hostname.}' does not refer to
a valid interface."
+ continue
+ fi
fi
# Test if it is an 802.11 interface.
Re: lladdr support for netstart/hostname.if
How about this adjustment to the man page? Index: share/man/man5/hostname.if.5 === RCS file: /cvs/src/share/man/man5/hostname.if.5,v retrieving revision 1.79 diff -u -p -r1.79 hostname.if.5 --- share/man/man5/hostname.if.526 Jul 2022 00:36:54 - 1.79 +++ share/man/man5/hostname.if.52 Dec 2022 02:51:27 - @@ -40,8 +40,10 @@ The .Nm hostname.*\& files contain information regarding the configuration of each network interface. -One file should exist for each interface that is to be configured, such as -.Pa hostname.fxp0 +One file should exist for each interface that is to be configured. +The interface can be referenced by name or lladdr, such as +.Pa hostname.fxp0 , +.Pa hostname.00:00:5e:00:53:af , or .Pa hostname.bridge0 . A configuration file is not needed for lo0.
Re: lladdr support for netstart/hostname.if
On Sat, Nov 26, 2022 at 07:12:12PM +, Klemens Nanni wrote:
> On Sat, Nov 26, 2022 at 10:44:05AM -0800, Andrew Hewus Fresh wrote:
> > I think this is the complete diff, with the man page bits mostly from
> > Martijn. Now that the ifconfig -M bits are in, this should be easier
> > to test. I was able to test the installer bits, and they worked for me.
>
> This reads straight forward, but see inline.
>
> > + # If the _if looks like an lladdr and look it up.
>
> That sentence is not proper english.
> "Look up the interface if _if seems like an lladdr."
> or
> "Obtain the interface if _if is a MAC"
> ?
I realized the comment wasn't particularly useful, so I deleted it.
> > + if [[ $_if = ??:??:??:??:??:?? ]]; then
>
> Please use == for comparison; I know = works but it reads off.
> Base should use == consistently, imho.
Somehow I thought the right side being a glob match only worked with "="
not "==", but turns out I was wrong. Fixed.
>
> > + if ! _line="$( ifconfig -M "$_if" )"; then
>
> Here you quote the subshell and the variable...
That is a heck of a habit I have. Apparently I can't avoid typing
those. Fixed.
>
> > + print -u2 "${0##*/}: unique if for lladdr $_if not
> > found."
>
> Maybe just "$_if is not unique"?
I like it! Thanks!
> >
> > + if [[ $_if = ??:??:??:??:??:?? ]]; then
> > + _if=$(ifconfig -M $_if)
>
> ... here you quote neither subshell nor variable.
I was trying to match the style like this, but not quoting things is
hard for me to do.
> I think that's fine, both install.sub and netstart already use unquoted
> $_if and the subshell does not have to be quoted, anyway, since word
> spliting does not occur:
>
> $ v=$(echo 1 2 3)
> $ echo $v
> 1 2 3
It does split though, echo just re-assembles it back the way it started:
$ v=$(printf "%s\n" 1 2 3 )
$ # without the quotes , it gets tokenized again and the newlines disappear
$ echo "$v"
1
2
3
> > + elif [[ $_if = ??:??:??:??:??:?? ]]; then
>
> ==, please.
Can do. Too used to writing portable things for /bin/sh I guess.
> > +Additionally the interface name can be replaced with the interface's
> > lladdr,
> > +given that the lladdr is unique across the system and no
> > +.Nm
> > +exists for the interface name.
>
> Isn't that too vague?
>
> .Nm is hostname.if but you meant to say that .fxp0 should not exist when .MAC
> already exists, no?
I'm not sure, it makes sense to me. Hopefully jmc provides an excellent
option, but I'll think on wording.
Here's another revision, not really any changes to the man page, but
your suggestions got applied, I think.
Index: etc/netstart
===
RCS file: /cvs/src/etc/netstart,v
retrieving revision 1.229
diff -u -p -r1.229 netstart
--- etc/netstart5 Nov 2022 12:06:05 - 1.229
+++ etc/netstart26 Nov 2022 21:02:08 -
@@ -135,6 +135,22 @@ ifstart() {
local _if=$1 _hn=/etc/hostname.$1 _cmds _i=0 _line _stat
set -A _cmds
+ if [[ $_if == ??:??:??:??:??:?? ]]; then
+ if ! _line=$( ifconfig -M $_if ); then
+ print -u2 "${0##*/}: $_if is not unique."
+ return
+ fi
+
+ [[ -z "$_line" ]] && return
+ _if=$_line
+ _line=
+
+ if [[ -e /etc/hostname.$_if ]]; then
+ print -u2 "${0##*/}: $_hn: /etc/hostname.$_if overrides"
+ return
+ fi
+ fi
+
# Interface names must be alphanumeric only. We check to avoid
# configuring backup or temp files, and to catch the "*" case.
[[ $_if != +([[:alpha:]])+([[:digit:]]) ]] && return
@@ -183,14 +199,16 @@ ifmstart() {
local _sifs=$1 _xifs=$2 _hn _if _sif _xif
for _sif in ${_sifs:-ALL}; do
- for _hn in /etc/hostname.+([[:alpha:]])+([[:digit:]]); do
+ for _hn in
/etc/hostname.@(+([[:alpha:]])+([[:digit:]])|??:??:??:??:??:??); do
[[ -f $_hn ]] || continue
_if=${_hn#/etc/hostname.}
- # Skip unwanted ifs.
- for _xif in $_xifs; do
- [[ $_xif == ${_if%%[0-9]*} ]] && continue 2
- done
+ if [[ $_if == +([[:alpha:]])+([[:digit:]]) ]]; then
+ # Skip unwanted ifs.
+ for _xif in $_xifs; do
+
Re: lladdr support for netstart/hostname.if
I think this is the complete diff, with the man page bits mostly from
Martijn. Now that the ifconfig -M bits are in, this should be easier
to test. I was able to test the installer bits, and they worked for me.
Index: etc/netstart
===
RCS file: /cvs/src/etc/netstart,v
retrieving revision 1.229
diff -u -p -r1.229 netstart
--- etc/netstart5 Nov 2022 12:06:05 - 1.229
+++ etc/netstart26 Nov 2022 17:21:38 -
@@ -135,6 +135,23 @@ ifstart() {
local _if=$1 _hn=/etc/hostname.$1 _cmds _i=0 _line _stat
set -A _cmds
+ # If the _if looks like an lladdr and look it up.
+ if [[ $_if = ??:??:??:??:??:?? ]]; then
+ if ! _line="$( ifconfig -M "$_if" )"; then
+ print -u2 "${0##*/}: unique if for lladdr $_if not
found."
+ return
+ fi
+
+ [[ -z "$_line" ]] && return
+ _if=$_line
+ _line=
+
+ if [[ -e /etc/hostname.$_if ]]; then
+ print -u2 "${0##*/}: $_hn: /etc/hostname.$_if overrides"
+ return
+ fi
+ fi
+
# Interface names must be alphanumeric only. We check to avoid
# configuring backup or temp files, and to catch the "*" case.
[[ $_if != +([[:alpha:]])+([[:digit:]]) ]] && return
@@ -183,14 +200,16 @@ ifmstart() {
local _sifs=$1 _xifs=$2 _hn _if _sif _xif
for _sif in ${_sifs:-ALL}; do
- for _hn in /etc/hostname.+([[:alpha:]])+([[:digit:]]); do
+ for _hn in
/etc/hostname.@(+([[:alpha:]])+([[:digit:]])|??:??:??:??:??:??); do
[[ -f $_hn ]] || continue
_if=${_hn#/etc/hostname.}
- # Skip unwanted ifs.
- for _xif in $_xifs; do
- [[ $_xif == ${_if%%[0-9]*} ]] && continue 2
- done
+ if [[ $_if = +([[:alpha:]])+([[:digit:]]) ]]; then
+ # Skip unwanted ifs.
+ for _xif in $_xifs; do
+ [[ $_xif == ${_if%%[0-9]*} ]] &&
continue 2
+ done
+ fi
# Start wanted ifs.
[[ $_sif == @(ALL|${_if%%[0-9]*}) ]] && ifstart $_if
Index: distrib/miniroot/install.sub
===
RCS file: /cvs/src/distrib/miniroot/install.sub,v
retrieving revision 1.1214
diff -u -p -r1.1214 install.sub
--- distrib/miniroot/install.sub6 Nov 2022 21:32:54 - 1.1214
+++ distrib/miniroot/install.sub26 Nov 2022 17:21:38 -
@@ -2431,6 +2431,12 @@ ifstart() {
local _if=$1 _hn=/mnt/etc/hostname.$1 _cmds _i=0 _line
set -A _cmds
+ if [[ $_if = ??:??:??:??:??:?? ]]; then
+ _if=$(ifconfig -M $_if)
+ [[ -z $_if ]] && return # invalid interface
+ [[ -e /mnt/etc/hostname.$_if ]] && return # duplicate config
+ fi
+
# Create interface if it does not yet exist.
{ ifconfig $_if || ifconfig $_if create; } >/dev/null 2>&1 || return
@@ -2472,6 +2478,9 @@ enable_ifs() {
svlan) _svlans="$_svlans $_if";;
vlan) _vlans="$_vlans $_if";;
esac
+ elif [[ $_if = ??:??:??:??:??:?? ]]; then
+ # start by lladdr
+ ifstart $_if
else
# 'Real' interfaces (if available) are done now.
ifconfig $_if >/dev/null 2>&1 && ifstart $_if
Index: share/man/man5/hostname.if.5
===
RCS file: /cvs/src/share/man/man5/hostname.if.5,v
retrieving revision 1.79
diff -u -p -r1.79 hostname.if.5
--- share/man/man5/hostname.if.526 Jul 2022 00:36:54 - 1.79
+++ share/man/man5/hostname.if.526 Nov 2022 17:21:38 -
@@ -41,9 +41,14 @@ The
.Nm hostname.*\&
files contain information regarding the configuration of each network
interface.
One file should exist for each interface that is to be configured, such as
-.Pa hostname.fxp0
+.Pa hostname.fxp0 ,
+.Pa hostname.12:34:56:78:ab ,
or
.Pa hostname.bridge0 .
+Additionally the interface name can be replaced with the interface's lladdr,
+given that the lladdr is unique across the system and no
+.Nm
+exists for the interface name.
A configuration file is not needed for lo0.
.Pp
The configuration information is expressed in a line-by-line packed format
Re: lladdr support for netstart/hostname.if
I think this is the bits the installer needs. I might adjust
/etc/netstart to use this glob or this to use *:*, still thinking on it.
It's a bit simpler because I figured the installer could be quieter
so I could just return instead of printing errors first.
I haven't tested this yet. We'll see how my time is today with the US
holiday, but too much further work will probably be pushed to the
weekend.
With the discussion otherwhere in the thread, I assume that `ifconfig
-M` is going to start filtering dynamic interfaces so we're safe
starting by lladdr early instead of mapping it to a name and putting it
into the appropriate dynamic interface list.
Index: distrib/miniroot/install.sub
===
RCS file: /cvs/src/distrib/miniroot/install.sub,v
retrieving revision 1.1214
diff -u -p -r1.1214 install.sub
--- distrib/miniroot/install.sub6 Nov 2022 21:32:54 - 1.1214
+++ distrib/miniroot/install.sub24 Nov 2022 18:49:30 -
@@ -2431,6 +2431,12 @@ ifstart() {
local _if=$1 _hn=/mnt/etc/hostname.$1 _cmds _i=0 _line
set -A _cmds
+ if [[ $_if = ??:??:??:??:??:?? ]]; then
+ _if=$(ifconfig -M $_if)
+ [[ -z $_if ]] && return # invalid interface
+ [[ -e /mnt/etc/hostname.$_if ]] && return # duplicate config
+ fi
+
# Create interface if it does not yet exist.
{ ifconfig $_if || ifconfig $_if create; } >/dev/null 2>&1 || return
@@ -2472,6 +2478,9 @@ enable_ifs() {
svlan) _svlans="$_svlans $_if";;
vlan) _vlans="$_vlans $_if";;
esac
+ elif [[ $_if = ??:??:??:??:??:?? ]]; then
+ # start by lladdr
+ ifstart $_if
else
# 'Real' interfaces (if available) are done now.
ifconfig $_if >/dev/null 2>&1 && ifstart $_if
Re: lladdr support for netstart/hostname.if
On Wed, Nov 23, 2022 at 05:28:35PM -0800, Andrew Hewus Fresh wrote:
> On Tue, Nov 22, 2022 at 08:09:11AM -0700, Theo de Raadt wrote:
> > Florian Obser wrote:
> > > ifconfig(8) already knows about these (see -C option). Which made me
> > > think, it might be easier to just ask ifconfig(8).
>
> >
> > I've done it as -M
>
> With `ifconfig -M $lladdr` support, and ignoring the check for both
> /etc/hostname.$if and /etc/hostname.$lladdr pointing to the same
> interface, this gets pretty simple.
>
> First, in the case of `netstart $lladdr` we ask `ifconfig -M` to
> translate to a $if and if that works, we continue on, using
> hostname.$lladdr.
>
> In the case of netstart without any arguments, it gets a little less
> cozy. We have to iterate all hostname.* files, and if they look like
> an interface name (as before) we use that directly, and otherwise we
> pass the extension as above, hoping it is an lladdr and trusting the
> validation there.
>
> That does mean calling `ifconfig -M` with any random extension we find
> on /etc/hostname.*, we found, although we could put back the `LLGLOB`
> and do a bit more validation here before calling ifconfig -M.
Theo had some suggestions, one obvious one is that we can at least
filter for only /etc/hostname.*:* and ignore anything without a :, as it
can't be a mac address. The other, that if both exist,
/etc/hostname.$if will override /etc/hostname.$lladdr.
Index: etc/netstart
===
RCS file: /cvs/src/etc/netstart,v
retrieving revision 1.229
diff -u -p -r1.229 netstart
--- etc/netstart5 Nov 2022 12:06:05 - 1.229
+++ etc/netstart24 Nov 2022 02:10:51 -
@@ -135,6 +135,23 @@ ifstart() {
local _if=$1 _hn=/etc/hostname.$1 _cmds _i=0 _line _stat
set -A _cmds
+ # If the _if has a :, assume it is an lladdr and look it up.
+ if [[ $_if = *:* ]]; then
+ if ! _line="$( ifconfig -M "$_if" )"; then
+ print -u2 "${0##*/}: unique if for lladdr $_if not
found."
+ return
+ fi
+
+ [[ -z "$_line" ]] && return
+ _if=$_line
+ _line=
+
+ if [[ -e /etc/hostname.$_if ]]; then
+ print -u2 "${0##*/}: $_hn: /etc/hostname.$_if overrides"
+ return
+ fi
+ fi
+
# Interface names must be alphanumeric only. We check to avoid
# configuring backup or temp files, and to catch the "*" case.
[[ $_if != +([[:alpha:]])+([[:digit:]]) ]] && return
@@ -183,14 +200,16 @@ ifmstart() {
local _sifs=$1 _xifs=$2 _hn _if _sif _xif
for _sif in ${_sifs:-ALL}; do
- for _hn in /etc/hostname.+([[:alpha:]])+([[:digit:]]); do
+ for _hn in /etc/hostname.@(+([[:alpha:]])+([[:digit:]])|*:*); do
[[ -f $_hn ]] || continue
_if=${_hn#/etc/hostname.}
- # Skip unwanted ifs.
- for _xif in $_xifs; do
- [[ $_xif == ${_if%%[0-9]*} ]] && continue 2
- done
+ if [[ $_if = +([[:alpha:]])+([[:digit:]]) ]]; then
+ # Skip unwanted ifs.
+ for _xif in $_xifs; do
+ [[ $_xif == ${_if%%[0-9]*} ]] &&
continue 2
+ done
+ fi
# Start wanted ifs.
[[ $_sif == @(ALL|${_if%%[0-9]*}) ]] && ifstart $_if
Re: lladdr support for netstart/hostname.if
On Tue, Nov 22, 2022 at 08:09:11AM -0700, Theo de Raadt wrote:
> Florian Obser wrote:
> > ifconfig(8) already knows about these (see -C option). Which made me
> > think, it might be easier to just ask ifconfig(8).
>
> I've done it as -M
With `ifconfig -M $lladdr` support, and ignoring the check for both
/etc/hostname.$if and /etc/hostname.$lladdr pointing to the same
interface, this gets pretty simple.
First, in the case of `netstart $lladdr` we ask `ifconfig -M` to
translate to a $if and if that works, we continue on, using
hostname.$lladdr.
In the case of netstart without any arguments, it gets a little less
cozy. We have to iterate all hostname.* files, and if they look like
an interface name (as before) we use that directly, and otherwise we
pass the extension as above, hoping it is an lladdr and trusting the
validation there.
That does mean calling `ifconfig -M` with any random extension we find
on /etc/hostname.*, we found, although we could put back the `LLGLOB`
and do a bit more validation here before calling ifconfig -M.
Index: etc/netstart
===
RCS file: /cvs/src/etc/netstart,v
retrieving revision 1.229
diff -u -p -r1.229 netstart
--- etc/netstart5 Nov 2022 12:06:05 - 1.229
+++ etc/netstart24 Nov 2022 01:20:28 -
@@ -135,6 +135,17 @@ ifstart() {
local _if=$1 _hn=/etc/hostname.$1 _cmds _i=0 _line _stat
set -A _cmds
+ # If the _if doesn't look like an interface name,
+ # assume it is an lladdr and look it up.
+ if [[ $_if != +([[:alpha:]])+([[:digit:]]) ]]; then
+ if ! _line="$( ifconfig -M "$_if" )"; then
+ print -u2 "${0##*/}: unique if for lladdr $_if not
found."
+ return
+ fi
+ [[ -n "$_line" ]] && _if=$_line
+ _line=
+ fi
+
# Interface names must be alphanumeric only. We check to avoid
# configuring backup or temp files, and to catch the "*" case.
[[ $_if != +([[:alpha:]])+([[:digit:]]) ]] && return
@@ -183,14 +194,16 @@ ifmstart() {
local _sifs=$1 _xifs=$2 _hn _if _sif _xif
for _sif in ${_sifs:-ALL}; do
- for _hn in /etc/hostname.+([[:alpha:]])+([[:digit:]]); do
+ for _hn in /etc/hostname.*; do
[[ -f $_hn ]] || continue
_if=${_hn#/etc/hostname.}
- # Skip unwanted ifs.
- for _xif in $_xifs; do
- [[ $_xif == ${_if%%[0-9]*} ]] && continue 2
- done
+ if [[ $_if = +([[:alpha:]])+([[:digit:]]) ]]; do
+ # Skip unwanted ifs.
+ for _xif in $_xifs; do
+ [[ $_xif == ${_if%%[0-9]*} ]] &&
continue 2
+ done
+ fi
# Start wanted ifs.
[[ $_sif == @(ALL|${_if%%[0-9]*}) ]] && ifstart $_if
Re: lladdr support for netstart/hostname.if
On Tue, Nov 22, 2022 at 03:37:20PM +, Klemens Nanni wrote: > On Tue, Nov 22, 2022 at 08:09:11AM -0700, Theo de Raadt wrote: > > Florian Obser wrote: > > > ifconfig(8) already knows about these (see -C option). Which made me > > > think, it might be easier to just ask ifconfig(8). > > > > > > $ ifconfig -Q 00:80:41:7b:f3:c3 > > > vio0 > > > > > > Would that be helpful? > > > > I'm unsure about the rest of your proposal, where MAC works in place if > > the IF argument. Let's say we do this in ifcconfig. Do we do it in route? > > Or ten other commands? I think that is the wrong way to go. > > > > But this first idea is valid. We've now seen 3 monster functions trying to > > do this task of convering MAC to IF in shell. Here's code to do it in > > ifconfig. > > > > I've done it as -M > > Better than [Q]uery. > > > > > It fails if the same MAC is on multiple interfaces. Go back to using > > hostname.if# files, you heathen. > > This reads like a viable approach, much cleaner than the netstart > globbing attempts. > > Using ifconfig -M, I can give the shell bits a try later this week. I agree that offloading mapping of the unique identifier to ifconfig makes the shell bits easier. We currently need to be able to go both directions, mostly to check existence of /etc/hostname.ure0 and /etc/hostname.$lladdr when when trying to netstart ure0 so we can complain that there are two configurations for the same interface. We can, of course, use the existing ifconfig parsing to look up the "lladdr ...". Maybe this is a case of letting folks shoot themselves in the foot. I do think that referring to it as -M for MAC lookup makes me expect it to return all interface names with that mac, not error if there is more than one. Unlike it it referred to a "unique identifier" that might be a mac or maybe IMEI. I could be wrong, but it might make swapping umb devices from different providers easer.
Re: lladdr support for netstart/hostname.if (was: Re: Locking network card configuration)
On Mon, Nov 21, 2022 at 04:56:07PM +0100, Martijn van Duren wrote: > On Sun, 2022-11-20 at 19:35 -0700, Theo de Raadt wrote: > > Steve Litt wrote: > > > > > Vitaliy Makkoveev said on Mon, 21 Nov 2022 03:48:21 +0300 > > > > > > > > On 20 Nov 2022, at 18:06, Odd Martin Baanrud > > > > > wrote: > > > > > > > > > > Hello, > > > > > > > > > > I have a Raspberry Pi 4 with 2 USB NIC’s attached. > > > > > One via USB3 (ure0), and the other via USB2 (ure1). > > > > > Since they are connected to different USB interfaces, I thaught they > > > > > would get configured the same way on reboot. But that’s not the case. > > > > > They became swapped on reboot. > > > > > Is there a way to “lock” the configuration I want? > > > > > So the USB3 NIC always become ure0, and the USB2 ure1. > > > > > > > > > > Regards, Martin > > > > > > > > > > > > > You could parse ifconfig(8) output to determine which names network > > > > interfaces received. But unfortunately, you can’t rename interfaces. > > > > > > During your parsing you could assign each one to an environment > > > variable such that, for instance, $lan contains the network card name > > > of the LAN one, and $wan contains the network name of the one going to > > > the Internet. Unfortunately, this would probably mean changing a lot of > > > existing shellscripts, but it's doable. > > > > But that is not the problem. > > > > hostname.* installs addresses on an interface, based upon the name of that > > interface. > > > > So it is too late for what you suggest. > > > > Unless the suggestion is have each hostname.* do a !command to a script > > which > > does the assigning. That is pretty crazy. > > > > pf.conf is not the problem either, because that can be entirely written > > using > > egress and groups. > > > > > > > > There is a problem with device attachment -> naming a device at that > > moment -> using that name in netstart.. but I am not sure how we could > > solve this without creating bigger problems for everyone else in the > > other non-hot-plug configurations, which is the majority of users with > > > 1 network device. > > > > We also hit this problem with disks, and we worked around it with the > > DUID subsystem. > > > > > > I suppose there is some argument that we should support hostname.MAC > > files > > > I don't have a usecase for this myself, but it seemed like a nice > exercise and might get the ball rolling. I also don't have much > experience with our rc/netstart shellcode, so I'm expecting this diff > should be taken as a starting-point iff we want something like this. > > I've chosen to error out on missing lladdr, duplicate lladdr and when > there's a hostname.if for both the lladdr and the if variant. This means > that there's smaller chance for order confusion or doubly applied > configs. Downside is that if someone decided to backup their hostname.if > to hostname.lladdr that will break their setup. However, I don't expect > people to backup their config files in this manner, but you never know. > > Errors: > On duplicate lladdr (in this case em0 and iwx0 in trunk0): > $ doas sh /usr/src/etc/netstart 88:a4:c2:fb:84:77 > netstart: /etc/hostname.88:a4:c2:fb:84:77: unique if for lladdr not found. > > On missing lladdr: > $ doas sh /usr/src/etc/netstart 88:a4:c2:fb:84:76 > netstart: /etc/hostname.88:a4:c2:fb:84:76: unique if for lladdr not found. > > And having both hostname.if and hostname.lladdr installed: > $ doas sh ./netstart 00:11:22:33:44:55 > netstart: /etc/hostname.00:11:22:33:44:55: duplicate config found in > /etc/hostname.vio0. > $ doas sh ./netstart vio0 > netstart: /etc/hostname.vio0: duplicate config found in > /etc/hostname.00:11:22:33:44:55. > > Two omissions I considered but didn't implement: > 1) I didn't test if the lladdr is owned by one of `ifconfig -C` >interfaces. Not sure if this is an upside or downside. > 2) Allowing /etc/netstart if1 and parsing the hostname.lladdr1 and vice >versa. I got interested in this, and looked at it a bit. My diff is also a bit preliminary, but a couple of things. First, I only parse ifconfig output once and save the LLADDR_MAP to look up later. This makes the lookup functions a bit simpler. Also, the glob now uses xdigit, which seems more correct, unless there's something I am missing about mac addresses. I also thought the error message for `netstart $lladdr` when /etc/hostname.$lladdr doesn't exist, but /etc/hostname.$if does was poor (it claimed duplicate configs which wasn't true) so I thought the easiest solution was to implement your #2 there and allow it to start the $if when you specify the $lladdr. Unfortunately, I then looked at the clock and realized it's time for bed, but I figured you might be interested in another take, even if it's probably incomplete. In any case, tomorrow is dinner with friends, so it will be Wednesday before I again have a chance to think on this. Index: etc/netstart
Re: match recent Intel CPUs in fw_update(8)
On Tue, Jun 21, 2022 at 02:58:46PM +1000, Jonathan Gray wrote:
> Intel CPUs used to have strings like
> cpu0: Intel(R) Pentium(R) M processor 1200MHz ("GenuineIntel" 686-class) 1.20
> GHz
> cpu0: Intel(R) Core(TM) i7-5600U CPU @ 2.60GHz, 2494.61 MHz, 06-3d-04
> recent CPUs use
> cpu0: 11th Gen Intel(R) Core(TM) i5-1130G7 @ 1.10GHz, 30009.37 MHz, 06-8c-01
> cpu0: 12th Gen Intel(R) Core(TM) i5-12400, 4390.71 MHz, 06-97-02
> cpu0: 12th Gen Intel(R) Core(TM) i7-1260P, 1995.55 MHz, 06-9a-03
>
> Index: patterns.c
> ===
> RCS file: /cvs/src/usr.sbin/fw_update/patterns.c,v
> retrieving revision 1.3
> diff -u -p -r1.3 patterns.c
> --- patterns.c10 Mar 2022 07:12:13 - 1.3
> +++ patterns.c21 Jun 2022 04:31:24 -
> @@ -94,7 +94,7 @@ main(void)
> printf("%s\n", "bwfm");
> printf("%s\n", "bwi");
> printf("%s\n", "intel");
> - printf("%s\n", "intel ^cpu0: Intel");
> + printf("%s\n", "intel ^cpu0:*Intel(R)");
With the recent commit to fw_update, this pattern should work now.
> printf("%s\n", "inteldrm");
> print_devices("inteldrm", i915_devices, nitems(i915_devices));
> printf("%s\n", "ipw");
>
Re: include cpuid 0 string in dmesg for fw_update
On Wed, Jul 27, 2022 at 06:47:56AM -0700, Andrew Hewus Fresh wrote:
> On Wed, Jul 27, 2022 at 11:06:39AM +0200, Alexander Hall wrote:
> > I think replacing '*' with '*([![:cntrl:]])' can be the droid your looking
> > for.
>
> As I was falling asleep last night I was trying to figure out how to get
> '*([!$_nl])' into the match and couldn't think of a good one. Your
> solution is much better.
Except for that issue with ksh patterns inside a variable.
I had some time to think about this while doing some construction.
Finally had time to try to implement it and it seems like although it's
a bit ugly, it is performant and seems to allow me to use "^cpu0:*Intel"
in the patterns file and (I believe) just match a line that starts with
cpu0: and contains Intel.
My alpha agrees with the speed assessment, the current implementation
that doesn't support the "*" in the patterns runs at about 5.5 seconds
(as before), while this new version takes 6.5 seconds. That's a lot
better than the 17.5 seconds it took to match against each line (and
still didn't actually expand the *).
Index: fw_update.sh
===
RCS file: /cvs/src/usr.sbin/fw_update/fw_update.sh,v
retrieving revision 1.42
diff -u -p -r1.42 fw_update.sh
--- fw_update.sh20 Feb 2022 21:53:04 - 1.42
+++ fw_update.sh3 Aug 2022 01:11:53 -
@@ -168,21 +168,32 @@ verify() {
}
firmware_in_dmesg() {
- local _d _m _line _dmesgtail _last='' _nl=$( echo )
+ local IFS
+ local _d _m _dmesgtail _last='' _nl='
+'
# The dmesg can contain multiple boots, only look in the last one
_dmesgtail="$( echo ; sed -n 'H;/^OpenBSD/h;${g;p;}'
/var/run/dmesg.boot )"
grep -v '^[[:space:]]*#' "$FWPATTERNS" |
while read -r _d _m; do
- [ "$_d" = "$_last" ] && continue
+ [ "$_d" = "$_last" ] && continue
[ "$_m" ] || _m="${_nl}${_d}[0-9] at "
[ "$_m" = "${_m#^}" ] || _m="${_nl}${_m#^}"
- if [[ $_dmesgtail = *$_m* ]]; then
- echo "$_d"
- _last="$_d"
- fi
+ IFS='*'
+ set -- $_m
+ unset IFS
+
+ case $# in
+ 1) [[ $_dmesgtail = *$** ]] ||
continue;;
+ 2) [[ $_dmesgtail = *$1*([!$_nl])$2* ]] ||
continue;;
+ 3) [[ $_dmesgtail = *$1*([!$_nl])$2*([!$_nl])$3* ]] ||
continue;;
+ *) echo "bad pattern $_m"; exit 1 ;;
+ esac
+
+ echo "$_d"
+ _last="$_d"
done
}
Re: include cpuid 0 string in dmesg for fw_update
On Wed, Jul 27, 2022 at 11:06:39AM +0200, Alexander Hall wrote: > I think replacing '*' with '*([![:cntrl:]])' can be the droid your looking > for. As I was falling asleep last night I was trying to figure out how to get '*([!$_nl])' into the match and couldn't think of a good one. Your solution is much better. I think that would end up looking like: ^cpu0:*([![:cntrl:]])Intel(R) although I recall some weirdness with metachars in a variable so I'll have to test it out. On Wed, Jul 27, 2022 at 12:01:51PM +, Klemens Nanni wrote: > On Wed, Jul 27, 2022 at 01:55:58PM +0200, Alexander Hall wrote: > > On a sidenote, the > > > > _nl=$(echo) > > > > in fw_update.sh is AFAICT deceitful and moot, since ksh strips away > > trailing linefeeds. Use the uglier > > > > _nl=' > > ' > > > > instead. > > Correct: > > $ _nl=$(echo) > $ printf '%s' "$_nl" > $ _nl=' > > ' > $ printf '%s' "$_nl" > > $ printf '\n' > > $ This is correct, I swear I had tested it before and that it worked, but I can't reproduce it now. The switch to per-line was using the ugly version. I'll work up a patch to get that fixed anyway.
Re: include cpuid 0 string in dmesg for fw_update
On Sun, Jul 24, 2022 at 10:34:04AM -0700, Andrew Hewus Fresh wrote:
> On Sun, Jul 24, 2022 at 09:14:30AM -0700, Andrew Hewus Fresh wrote:
> > On Sun, Jul 24, 2022 at 10:01:26AM -0600, Theo de Raadt wrote:
> > > Jonathan Gray wrote:
> > >
> > > > On Sun, Jul 24, 2022 at 08:05:26AM -0600, Theo de Raadt wrote:
> > > > > Why not match on cpu0: .*Intel
> > > >
> > > > I sent a diff a month ago with ^cpu0:*Intel(R)
> > > >
> > > > semarie mentioned false positives as it could match another
> > > > Intel device on another line on a non-Intel CPU
> > > >
> > > > https://marc.info/?l=openbsd-tech=165579653107768=2
> > >
> > > Well, then fw_update should be fixed to not perform the match over
> > > multiple lines.
> >
> > I'm looking at fixing fw_update to match each line instead of the whole
> > dmesg. I'll try to get a patch out for that today.
>
> This patch matches patterns against each line in the dmesg instead of
> anchoring on newlines and matching against the entire string at once
> anchored by newlines. This would mean that ^cpu0:*Intel(R) would Just
> Work.
>
> The problem is that it's a little over 3 times slower on my laptop to do
> the matching this way.
As an example, on my alpha (that I admit to only keeping around because
a 166MHz single processor machine that runs OpenBSD is useful for
testing how things work on a slow machine), the current code runs and
doesn't find any firmware in 5.5 seconds and with this patch it now
takes 17.5 seconds.
> If any ksh folks have tricks to speed that up,
> I'd appreciate it. I'll try to think about whether I can build a sed
> program that will spit out matches. Sadly at the moment, distracted by
> a non-computer project that has been taking up all my free time.
> Hopefully that will be finished in the next couple weeks though.
>
> I think I could do some magic to replace the "^cpu:*Intel(R)" above with
> "^cpu:*([!\n])Intel(R)" (although not with directly as ksh wouldn't
> recognize it). That would be annoying to implement though, but would
> then still be able to do the faster single match.
>
>
> --- fw_update.sh.orig Sun Jul 24 10:07:40 2022
> +++ fw_update.sh Sun Jul 24 10:14:21 2022
> @@ -168,21 +168,31 @@
> }
>
> firmware_in_dmesg() {
> - local _d _m _line _dmesgtail _last='' _nl=$( echo )
> + local _d _m _line _dmesgtail _last='' _oldifs="$IFS"
>
> + IFS='
> +'
> # The dmesg can contain multiple boots, only look in the last one
> - _dmesgtail="$( echo ; sed -n 'H;/^OpenBSD/h;${g;p;}'
> /var/run/dmesg.boot )"
> + set -A _dmesgtail $( sed -n 'H;/^OpenBSD/h;${g;p;}' /var/run/dmesg.boot
> )
> + IFS="$_oldifs"
>
> grep -v '^[[:space:]]*#' "$FWPATTERNS" |
> while read -r _d _m; do
> [ "$_d" = "$_last" ] && continue
> - [ "$_m" ] || _m="${_nl}${_d}[0-9] at "
> - [ "$_m" = "${_m#^}" ] || _m="${_nl}${_m#^}"
> -
> - if [[ $_dmesgtail = *$_m* ]]; then
> - echo "$_d"
> - _last="$_d"
> + [ "$_m" ] || _m="^${_d}[0-9] at "
> + if [ "$_m" = "${_m#^}" ]; then
> + _m="*$_m"
> + else
> + _m="${_m#^}"
> fi
> +
> + for _line in "${_dmesgtail[@]}"; do
> + if [[ $_line = $_m* ]]; then
> + echo "$_d"
> + _last="$_d"
> + break
> + fi
> + done
> done
> }
>
>
--
andrew
A printer consists of three main parts:
the case, the jammed paper tray and the blinking red light.
Re: include cpuid 0 string in dmesg for fw_update
On Sun, Jul 24, 2022 at 10:34:04AM -0700, Andrew Hewus Fresh wrote:
> I'll try to think about whether I can build a sed program that will
> spit out matches.
Unfortunately the sed version seems to be even slower than matching one
line at a time. Just building the patterns for sed is slower than the
original single match.
--- fw_update.sh.orig Sun Jul 24 10:07:40 2022
+++ fw_update.shSun Jul 24 20:50:02 2022
@@ -168,22 +168,34 @@
}
firmware_in_dmesg() {
- local _d _m _line _dmesgtail _last='' _nl=$( echo )
+ local _cmd
- # The dmesg can contain multiple boots, only look in the last one
- _dmesgtail="$( echo ; sed -n 'H;/^OpenBSD/h;${g;p;}'
/var/run/dmesg.boot )"
-
- grep -v '^[[:space:]]*#' "$FWPATTERNS" |
+ _cmd="$( grep -v '^[[:space:]]*#' "$FWPATTERNS" | (
+ _s='/^%s/i\\\n%s\n'
+ _last=
+ _patterns=
while read -r _d _m; do
- [ "$_d" = "$_last" ] && continue
- [ "$_m" ] || _m="${_nl}${_d}[0-9] at "
- [ "$_m" = "${_m#^}" ] || _m="${_nl}${_m#^}"
+ [ "$_m" ] || _m="^${_d}[0-9] at "
+ [ "$_m" = "${_m#*/}" ] || _m="$( printf "$_m" | sed 's,/,\\/,g'
)"
- if [[ $_dmesgtail = *$_m* ]]; then
- echo "$_d"
- _last="$_d"
+ if [ "$_patterns" ] && [ "$_d" != "$_last" ]; then
+ printf "$_s" "$_patterns" "$_last"
+ _last=
+ _patterns=
fi
+
+ _last="$_d"
+ if [ "$_patterns" ]; then
+ _patterns="$_patterns|$_m"
+ else
+ _patterns="$_m"
+ fi
done
+ [ "$_patterns" ] && printf "$_s" "$_patterns" "$_last"
+ ) )"
+
+ # The dmesg can contain multiple boots, only look in the last one
+ sed -n 'H;/^OpenBSD/h;${g;p;}' /var/run/dmesg.boot | sed -En "$_cmd"
}
firmware_filename() {
Re: include cpuid 0 string in dmesg for fw_update
On Sun, Jul 24, 2022 at 09:14:30AM -0700, Andrew Hewus Fresh wrote:
> On Sun, Jul 24, 2022 at 10:01:26AM -0600, Theo de Raadt wrote:
> > Jonathan Gray wrote:
> >
> > > On Sun, Jul 24, 2022 at 08:05:26AM -0600, Theo de Raadt wrote:
> > > > Why not match on cpu0: .*Intel
> > >
> > > I sent a diff a month ago with ^cpu0:*Intel(R)
> > >
> > > semarie mentioned false positives as it could match another
> > > Intel device on another line on a non-Intel CPU
> > >
> > > https://marc.info/?l=openbsd-tech=165579653107768=2
> >
> > Well, then fw_update should be fixed to not perform the match over
> > multiple lines.
>
> I'm looking at fixing fw_update to match each line instead of the whole
> dmesg. I'll try to get a patch out for that today.
This patch matches patterns against each line in the dmesg instead of
anchoring on newlines and matching against the entire string at once
anchored by newlines. This would mean that ^cpu0:*Intel(R) would Just
Work.
The problem is that it's a little over 3 times slower on my laptop to do
the matching this way. If any ksh folks have tricks to speed that up,
I'd appreciate it. I'll try to think about whether I can build a sed
program that will spit out matches. Sadly at the moment, distracted by
a non-computer project that has been taking up all my free time.
Hopefully that will be finished in the next couple weeks though.
I think I could do some magic to replace the "^cpu:*Intel(R)" above with
"^cpu:*([!\n])Intel(R)" (although not with directly as ksh wouldn't
recognize it). That would be annoying to implement though, but would
then still be able to do the faster single match.
--- fw_update.sh.orig Sun Jul 24 10:07:40 2022
+++ fw_update.shSun Jul 24 10:14:21 2022
@@ -168,21 +168,31 @@
}
firmware_in_dmesg() {
- local _d _m _line _dmesgtail _last='' _nl=$( echo )
+ local _d _m _line _dmesgtail _last='' _oldifs="$IFS"
+ IFS='
+'
# The dmesg can contain multiple boots, only look in the last one
- _dmesgtail="$( echo ; sed -n 'H;/^OpenBSD/h;${g;p;}'
/var/run/dmesg.boot )"
+ set -A _dmesgtail $( sed -n 'H;/^OpenBSD/h;${g;p;}' /var/run/dmesg.boot
)
+ IFS="$_oldifs"
grep -v '^[[:space:]]*#' "$FWPATTERNS" |
while read -r _d _m; do
[ "$_d" = "$_last" ] && continue
- [ "$_m" ] || _m="${_nl}${_d}[0-9] at "
- [ "$_m" = "${_m#^}" ] || _m="${_nl}${_m#^}"
-
- if [[ $_dmesgtail = *$_m* ]]; then
- echo "$_d"
- _last="$_d"
+ [ "$_m" ] || _m="^${_d}[0-9] at "
+ if [ "$_m" = "${_m#^}" ]; then
+ _m="*$_m"
+ else
+ _m="${_m#^}"
fi
+
+ for _line in "${_dmesgtail[@]}"; do
+ if [[ $_line = $_m* ]]; then
+ echo "$_d"
+ _last="$_d"
+ break
+ fi
+ done
done
}
Re: include cpuid 0 string in dmesg for fw_update
On Sun, Jul 24, 2022 at 10:01:26AM -0600, Theo de Raadt wrote: > Jonathan Gray wrote: > > > On Sun, Jul 24, 2022 at 08:05:26AM -0600, Theo de Raadt wrote: > > > Why not match on cpu0: .*Intel > > > > I sent a diff a month ago with ^cpu0:*Intel(R) > > > > semarie mentioned false positives as it could match another > > Intel device on another line on a non-Intel CPU > > > > https://marc.info/?l=openbsd-tech=165579653107768=2 > > Well, then fw_update should be fixed to not perform the match over > multiple lines. I'm looking at fixing fw_update to match each line instead of the whole dmesg. I'll try to get a patch out for that today. > Absolutely noone expects a pattern containing '^' to match over > multiple lines. It is only a matter of time before another pattern > acts wrong, so I think fw_update should get a proper line-matcher > so that the pattern ^cpu0: *Intel' can be used. Agreed.
Re: sysupgrade(8): Pick correct firmware directory
On Tue, Mar 01, 2022 at 10:06:10AM +0100, Stefan Hagen wrote:
> Stefan Hagen wrote (2022-02-28 22:16 CET):
> > Theo de Raadt wrote (2022-02-28 18:02 CET):
> > > When you run sysupgrade, you want the firmwares matching what you are
> > > installing.
> > >
> > > The system you are running right now is irrelevant. It's drivers
> > > have already loaded the firmwares they need, and a reboot into new
> > > bsd.rd is about to happen.
> > >
> > > So why does there continue to be conversation about the current
> > > system release? It is irrelevant.
> >
> > Good point. This diff should be better then. It's not trying to reuse
> > NEXT_VERSION or _KERN based on beta/current tag guesswork, but gets
> > all information from the target kernel.
> >
> > If config -ao should not be used, here a version with what(1):
>
> Again with correct amount of [ and ] and the use of what(1) with the sed
> tips from Leon.
>
> I tested:
> Upgrade 69 -> 70
> Upgrade 70 -> 71-beta
> Upgrade 70-current -> 71-beta
> Upgrade 71-beta -> 71-beta
>
> It did the right thing in all those cases.
>
> OK?
This looks pretty much like what I came up with, although I used
[[:blank]] similarly to what sthen@ suggested, but either way.
OK afresh1@
>
> Best Regards,
> Stefan
>
> Index: sysupgrade.sh
> ===
> RCS file: /home/cvs/src/usr.sbin/sysupgrade/sysupgrade.sh,v
> retrieving revision 1.45
> diff -u -p -u -p -r1.45 sysupgrade.sh
> --- sysupgrade.sh 11 Feb 2022 12:58:18 - 1.45
> +++ sysupgrade.sh 1 Mar 2022 08:50:40 -
> @@ -123,10 +123,8 @@ fi
>
> if $SNAP; then
> URL=${MIRROR}/snapshots/${ARCH}/
> - FW_URL=http://firmware.openbsd.org/firmware/snapshots/
> else
> URL=${MIRROR}/${NEXT_VERSION}/${ARCH}/
> - FW_URL=http://firmware.openbsd.org/firmware/${NEXT_VERSION}/
> fi
>
> install -d -o 0 -g 0 -m 0755 ${SETSDIR}
> @@ -196,7 +194,15 @@ __EOT
> fi
>
> echo Fetching updated firmware.
> -fw_update -p ${FW_URL} || true
> +set -A _NEXTKERNV -- $(what bsd |
> + sed -n '2s/^OpenBSD \([1-9][0-9]*\.[0-9]\)\([^ ]*\).*/\1 \2/p')
> +
> +if [[ ${_NEXTKERNV[1]} == '-current' ]]; then
> + FW_URL=http://firmware.openbsd.org/firmware/snapshots/
> +else
> + FW_URL=http://firmware.openbsd.org/firmware/${_NEXTKERNV[0]}/
> +fi
> +VNAME="${_NEXTKERNV[0]}" fw_update -p ${FW_URL} || true
>
> install -F -m 700 bsd.rd /bsd.upgrade
> logger -t sysupgrade -p kern.info "installed new /bsd.upgrade. Old kernel
> version: $(sysctl -n kern.version)"
>
--
andrew
If your computer says, "Printer out of Paper," this problem cannot
be resolved by continuously clicking the "OK" button.
Re: sysupgrade(8): Pick correct firmware directory
On Mon, Feb 28, 2022 at 05:28:19PM +0100, Stefan Hagen wrote:
> Andrew Hewus Fresh wrote (2022-02-28 15:40 CET):
> > On Mon, Feb 28, 2022 at 02:47:24PM +0100, Stefan Hagen wrote:
> > > But now I see that NEXT_VERSION is set earlier and this should fix it:
> > >
> > > -if $RELEASE && [[ ${_KERNV[1]} == '-beta' ]]; then
> > > +if $RELEASE || [[ ${_KERNV[1]} == '-beta' ]]; then
> >
> > It would fix this particular case, but this code is here for the
> > situation where:
> >
> > A user is on 7.1-beta, 7.1 is release and the user does `sysupgrade -r`
> >
> > This patch would break that case. I will keep thinking on a proper
> > fix.
>
> Are you sure?
I'm sorry, PagerDuty woke me up at 4am and I replied above without
coffee. This is not the case that would break.
> I made myself a small table with the possible upgrade paths how I
> see them:
>
> Source rel. | Target rel. | Firmware dir.
> +-+
> 7.0-release | 7.1-release | ${NEXT_VERSION}
This case, with `sysupgrade -r`, would break. In this case, RELEASE is
true and _KERNV[0] is 7.0, which makes this true:
+if $RELEASE || [[ ${_KERNV[1]} == '-beta' ]]; then
NEXT_VERSION=${_KERNV[0]}
and sets NEXT_VERSION to 7.0, which is not what I would expect.
> 7.0-release | 7.1-beta| ${NEXT_VERSION}
> 7.0-release | 7.0-current | snapshots
> -
> 7.0-current | 7.1-release | ${NEXT_VERSION}
> 7.0-current | 7.1-beta| ${NEXT_VERSION}
> 7.0-current | 7.0-current | snapshots
> -
> 7.1-beta| 7.1-release | ${_KERNV[0]}
> 7.1-beta| 7.1-beta| ${_KERNV[0]}
> 7.1-beta| 7.1-current | snapshots
>
> If you don't look at the release/beta/current tags and compare the
> release version number instead, it gets simpler:
>
> source_VER == target_VER:${_KERNV[0]}
> source_VER != target_VER:${NEXT_VERSION}
> both overruled by -current: snapshots
>
> Regarding the regex: The output of
> sysctl -n kern.version
> is the same as
> echo exit | config -eo /dev/null bsd
>
> So the regex used to parse the information for _KERN can be reused.
I'll make sure I'm doing that.
Re: sysupgrade(8): Pick correct firmware directory
On Mon, Feb 28, 2022 at 02:47:24PM +0100, Stefan Hagen wrote:
> But now I see that NEXT_VERSION is set earlier and this should fix it:
>
> -if $RELEASE && [[ ${_KERNV[1]} == '-beta' ]]; then
> +if $RELEASE || [[ ${_KERNV[1]} == '-beta' ]]; then
It would fix this particular case, but this code is here for the
situation where:
A user is on 7.1-beta, 7.1 is release and the user does `sysupgrade -r`
This patch would break that case. I will keep thinking on a proper
fix.
Re: sysupgrade(8): Pick correct firmware directory
On Mon, Feb 28, 2022 at 08:27:13AM +0100, Florian Obser wrote:
> On 2022-02-27 21:33 -08, Andrew Hewus Fresh
> wrote:
> > So, sdk@ noticed that sysupgrade didn't get updated for the new logic in
> > the firmware directories. Specifically that the only time we use
> > "snapshots" directory is with -current. The rest of the time, including
> > during -beta we use the version directory. This diff should handle that
> > case, installing the correct firmware for the system we are about to
> > install.
> >
> > This also uses a secret feature of fw_update(8) where if you set "VNAME"
> > in the environment, it uses that instead of asking sysctl to calculate
> > the name of the signify key to use. Probably that could use an
> > improvement, maybe trusting the untrusted comment at the top of the
> > file..
> >
> > I'm not sure if there's a better way to find the version string from a
> > bsd kernel, so I used the one I knew about.
>
> I think you can just look at _KERNV which is populated thusly:
>
>
> 98set -A _KERNV -- $(sysctl -n kern.version |
> 99sed 's/^OpenBSD \([1-9][0-9]*\.[0-9]\)\([^ ]*\).*/\1
> \2/;q')
>
> $ sysctl kern.version
> kern.version=OpenBSD 7.0-current (GENERIC.MP) #370: Sat Feb 19 10:36:59 MST
> 2022
> dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
>
> $ set -A _KERNV -- $(sysctl -n kern.version |
> > sed 's/^OpenBSD \([1-9][0-9]*\.[0-9]\)\([^ ]*\).*/\1 \2/;q')
>
> $ echo ${_KERNV[1]}
> -current
I don't think so, that is the running kernel, I need the version of the
_new_ kernel. Imagine this situation:
Someone installs -current shortly after release, a new version of DRM is
imported (or really, any firmware is updated along with the driver) and
then we move to -beta. This someone sees that and realizes they should
upgrade. They are still on 7.0-current, but the new kernel is 7.1-beta
and they should be looking for firmware in /7.1, not /snapshots (and
using the 7.1 signify key).
I think I see a way to figure this out from the logic above
(specifically whether we're using the new signify keys with a
snapshot) but I'll have to stare at it a bit more.
fw_update: improve 'kept' messaging
One of the things on my TODO list has been to improve the "kept"
messaging from fw_update.
For example, deleting a firmware doesn't list what was kept and if you
specified a firmware to install on the command-line the "kept" list only
included the specified firmware.
Instead, we only keep track of the firmware that was modified and use
that to remove from the full list of installed firmware. That works not
only in the above case, but in the "autodetect" case that included the
installed firmware in what we attempt to update.
The one difference is when we are downloading firmware to a local
directory, in which case the "kept" list is the list of firmware that
would have been downloaded but was already there, as it was previously.
Comments? Suggestions on something better? OK?
I only just learned the "${foo[@]:+${foo[@]}}" trick to avoid the
"parameter not set" error from "set -o nounset" when I need to iterate
an empty list. I can get rid of so many unnecessary levels of
indentation now!
Index: fw_update.sh
===
RCS file: /cvs/src/usr.sbin/fw_update/fw_update.sh,v
retrieving revision 1.42
diff -u -p -r1.42 fw_update.sh
--- fw_update.sh20 Feb 2022 21:53:04 - 1.42
+++ fw_update.sh20 Feb 2022 23:45:13 -
@@ -241,6 +241,24 @@ installed_firmware() {
done
}
+_tokenize() {
+ local IFS=,
+ echo $*
+}
+
+kept_firmware() {
+ local _modified _m _i _d
+ set -A _modified -- $( _tokenize "$@" )
+
+ for _i in $( installed_firmware '*' '-firmware-' '*' ); do
+ _d="$( firmware_devicename "$_i" )"
+ for _m in "${_modified[@]:+${_modified[@]}}"; do
+ [ "$_d" = "$_m" ] && continue 2
+ done
+ echo "$_d"
+ done
+}
+
detect_firmware() {
local _devices _last='' _d
@@ -471,8 +489,14 @@ if "$DELETE"; then
done
fi
+ kept=''
+ for k in $( kept_firmware ); do
+ kept="$kept,$k"
+ done
+
deleted="${deleted#,}"
- echo "${0:##*/}: deleted ${deleted:-none}";
+ kept="${kept#,}"
+ echo "${0:##*/}: deleted ${deleted:-none}; kept ${kept:-none}";
exit
fi
@@ -529,7 +553,6 @@ if [ "${devices[*]:-}" ]; then
for i in "${installed[@]}"; do
if [ "${f##*/}" = "$i.tgz" ]; then
((VERBOSE > 2)) && echo "Keep $i"
- kept="$kept,$d"
continue 2
fi
done
@@ -616,6 +639,11 @@ updated="${updated:#,}"
kept="${kept:#,}"
[ "${unregister:-}" ] && unregister="; unregistered ${unregister:#,}"
if "$INSTALL"; then
+ kept=''
+ for k in $( kept_firmware "$added" "$updated" "$unregister" ); do
+ kept="$kept,$k"
+ done
+ kept="${kept:#,}"
echo "${0##*/}: added ${added:-none}; updated ${updated:-none}; kept
${kept:-none}${unregister}"
else
echo "${0##*/}: downloaded ${added:-none}; kept
${kept:-none}${unregister}"
Show none in fw_update when none are detected
In the case where no firmware are detected/installed fw_update is too
quiet.
This should fix things to show "none" in that case:
# fw_update
fw_update: added none; updated none; kept none
Is that what folks would expect?
This patch does not include the extra level of indentation that will
result, but that would obfuscate the change.
Comments, OK?
Index: usr.sbin/fw_update/fw_update.sh
===
RCS file: /cvs/src/usr.sbin/fw_update/fw_update.sh,v
retrieving revision 1.39
diff -u -p -r1.39 fw_update.sh
--- usr.sbin/fw_update/fw_update.sh 19 Feb 2022 02:19:42 - 1.39
+++ usr.sbin/fw_update/fw_update.sh 19 Feb 2022 19:54:24 -
@@ -492,14 +492,13 @@ else
{ [ "${devices[*]:-}" ] && echo " found." || echo " done." ; }
fi
-[ "${devices[*]:-}" ] || exit
-
-lock_db
added=''
updated=''
kept=''
unregister=''
+if [ "${devices[*]:-}" ]; then
+lock_db
for f in "${devices[@]}"; do
d="$( firmware_devicename "$f" )"
@@ -609,6 +608,7 @@ for f in "${devices[@]}"; do
updated="$updated,$d"
fi
done
+fi
added="${added:#,}"
updated="${updated:#,}"
Re: fw_update perl glue tweaks
On Mon, Feb 14, 2022 at 11:37:55AM +0100, Marc Espie wrote:
> - you don't need to call unlock_db. When the process dies, the fd dies with
> it, and the lock.
Makes sense.
it probably doesn't need the $SIG{TERM} handler if we don't need to
trigger the END block anymore either.
> - I've committed glue to PackageInfo.pm to make lock_db self-contained
> (it will pull BaseState in on an "as needed" basis, which also comes in
> handy for other locks prior to having a state)
Very nice! Thanks!
Also, this change is OK afresh1@
> Index: fw_update.sh
> ===
> RCS file: /cvs/src/usr.sbin/fw_update/fw_update.sh,v
> retrieving revision 1.37
> diff -u -p -r1.37 fw_update.sh
> --- fw_update.sh 11 Feb 2022 00:46:58 - 1.37
> +++ fw_update.sh 14 Feb 2022 10:35:57 -
> @@ -206,13 +206,11 @@ lock_db() {
> perl <<'EOL' |&
> use v5.16;
> use warnings;
> - use OpenBSD::PackageInfo qw< lock_db unlock_db >;
> - use OpenBSD::BaseState;
> + use OpenBSD::PackageInfo qw< lock_db >;
>
> $|=1;
>
> - lock_db(0, 'OpenBSD::BaseState');
> - END { unlock_db }
> + lock_db(0);
> $SIG{TERM} = sub { exit };
>
> say $$;
fw_update(8): lock pkg database while running
I was reminded that fw_update(8) updates the package database without
locking currently. That can cause issues when running it concurrently
with pkg_add, for example starting `pkg_add -u` in one terminal and
`sysupgrade` in another.
This diff checks to see if perl is available and if so starts a perl
co-process that locks the database and then kills it when exiting.
(Figuring out how to wait for the lock to be acquired and find out the
pid of the co-process was a bit of a challenge, so if someone has a
suggestion on improving that, I'm open to suggestions)
Comments, OK?
Index: usr.sbin/fw_update/fw_update.sh
===
RCS file: /cvs/src/usr.sbin/fw_update/fw_update.sh,v
retrieving revision 1.36
diff -u -p -r1.36 fw_update.sh
--- usr.sbin/fw_update/fw_update.sh 10 Feb 2022 00:29:32 - 1.36
+++ usr.sbin/fw_update/fw_update.sh 10 Feb 2022 03:22:20 -
@@ -42,11 +42,13 @@ INSTALL=true
LOCALSRC=
unset FTPPID
+unset LOCKPID
unset FWPKGTMP
REMOVE_LOCALSRC=false
cleanup() {
set +o errexit # ignore errors from killing ftp
[ "${FTPPID:-}" ] && kill -TERM -"$FTPPID" 2>/dev/null
+ [ "${LOCKPID:-}" ] && kill -TERM -"$LOCKPID" 2>/dev/null
[ "${FWPKGTMP:-}" ] && rm -rf "$FWPKGTMP"
"$REMOVE_LOCALSRC" && rm -rf "$LOCALSRC"
[ -e "${CFILE}" ] && [ ! -s "$CFILE" ] && rm -f "$CFILE"
@@ -194,6 +196,36 @@ firmware_devicename() {
echo "$_d"
}
+lock_db() {
+ [ "${LOCKPID:-}" ] && return 0
+
+ # The installer doesn't have perl, so we can't lock there
+ [ -e /usr/bin/perl ] || return 0
+
+ set -o monitor
+ perl <<'EOL' |&
+ use v5.16;
+ use warnings;
+ use OpenBSD::PackageInfo qw< lock_db unlock_db >;
+ use OpenBSD::State;
+
+ $|=1;
+
+ lock_db(0, OpenBSD::State->new);
+ END { unlock_db }
+ $SIG{TERM} = sub { exit };
+
+ say $$;
+ sleep;
+EOL
+ set +o monitor
+
+ read -rp LOCKPID
+
+ return 0
+}
+
+
installed_firmware() {
local _pre="$1" _match="$2" _post="$3" _firmware _fw
set -sA _firmware -- $(
@@ -401,6 +433,7 @@ set -sA devices -- "$@"
if "$DELETE"; then
[ "$OPT_F" ] && echo "Cannot use -F and -d" >&2 && usage
+ lock_db
# Show the "Uninstall" message when just deleting not upgrading
((VERBOSE)) && VERBOSE=3
@@ -463,6 +496,8 @@ else
fi
[ "${devices[*]:-}" ] || exit
+
+lock_db
added=''
updated=''
Re: perl clang -Wcompound-token-split-by-macro
On Mon, Feb 07, 2022 at 08:11:55PM +0100, Alexander Bluhm wrote: > On Tue, Jan 25, 2022 at 12:26:05PM -0800, Andrew Hewus Fresh wrote: > > On Tue, Jan 25, 2022 at 09:20:55PM +0100, Alexander Bluhm wrote: > > > On Tue, Jan 25, 2022 at 12:05:48PM -0800, Andrew Hewus Fresh wrote: > > > > On Tue, Jan 25, 2022 at 06:45:12PM +0100, Alexander Bluhm wrote: > > > > > On Tue, Jan 25, 2022 at 05:13:01PM +0100, Alexander Bluhm wrote: > > > > > > On Sat, Jan 22, 2022 at 02:24:51AM +0100, Marc Espie wrote: > > > > > > > Or we can automate this with something like this: > > > > > > > > > > > > > > > > Our Devel::PPPort is too old. We ship with 3.57, p5-CDB_File and > > > > > p5-Moose > > > > > ship with ppport.h generated by 3.62. > > > > > > > > We could update Devel::PPPort in base. > > > > > > This will fix the current problem. But we can always have an old > > > Devel::PPPort in base and a have module in ports that comes with > > > and needs a new ppport.h. > > > > > > Somehow the porter should have a mechanism to handle this. Only 2 > > > of 300 ports that I test have issues, so it is a rare action. Per > > > default espie@'s idea works well. If we manually fix 2 ports that > > > is fine for me. > > > > > > Or we replace ppport.h only if it is outdated. > > > > We could add espie's target to those two ports. > > How do we proceed? > > I have 62 warnings in base build. This ppport was not updated yet: > /usr/src/gnu/usr.bin/perl/cpan/Term-ReadKey/ppport.h > > I have 17558 warnings in my ports test logfile. This is annoying. > Still I don't understand what this warning is all about. I looks > like llvm wants to be smarter than 30 years of preprocessor hacks. > > On CPAN is a new version 3.64 of Devel::PPPort that has fixed the issue. > https://metacpan.org/dist/Devel-PPPort/changes > https://github.com/Dual-Life/Devel-PPPort/commit/e124e16191f3c971c6efad3f9def300e2ee57ef8 > > I think we should start updating Devel::PPPort in base. Then we > can regenerate other ppport.h in base and ports. Note that some ports > need newer Devel::PPPort than we have now. So updating seems > unavoidable. > > databases/p5-CDB_File and devel/p5-Moose build with this version. > > Do we want to go this way? This seems reasonable to me. Perl 5.34 ships with 3.62, so if that works and we do want to get in 5.34 at some point we might want to stick with that. We can also keep the newer version until we get to 5.36 which I assume comes with at least this version. I guess if we are going to auto-update the ppport.h we'll just have to keep following newer versions as they get released ahead of whatever perl we have when our updates require them? Kind of annoying, but probably the best solution. > I have untared Devel-PPPort-3.64.tar.gz into > /usr/src/gnu/usr.bin/perl/dist/Devel-PPPort and cvs added new files. > Is this the correct way to update base modules? Pretty much, yes. If it added any new files they may need to be added to the MANIFEST and new dirs to Makefile.SH (I think just for make clean). https://github.com/afresh1/OpenBSD-perl/blob/blead/patches/GOOD/put_OpenBSD-MkTemp_in_MANIFEST.patch I maintain them separately, although just Term-ReadKey from the CPAN. https://github.com/afresh1/OpenBSD-perl/tree/blead/files > Some test fail, but that is not worse than before. > -t/ppphtest.t . Failed 179/238 subtests > +t/ppphtest.t . Failed 178/235 subtests I know perl tests don't like to run in /usr/src for reasons I don't recall, but I do know that normally I can run tests with all the patches applied. > Diff has 22948 lines, so I do not send it per mail. > cvs:~bluhm/Devel-PPPort.diff > > ok? ok afresh1@ on the concept, although I didn't read it. If you can build a release with it, that's good enough for me.
Re: perl clang -Wcompound-token-split-by-macro
On Tue, Jan 25, 2022 at 09:20:55PM +0100, Alexander Bluhm wrote: > Plans for new Perl are good to hear. Thanks for updating it regualry. One thing that's blocking me on that is the "vendorlib" patch over on tech@[1] and the related patch on ports@[2]. I think it's a good idea, but haven't gotten much feedback. It's a pretty big change and so I'd really appreciate another set of eyes and maybe some verification that it won't break your stuff. Let me know if you need any more details. [1] https://marc.info/?l=openbsd-tech=162769530004204=2 [2] https://marc.info/?l=openbsd-ports=163417552406211=2 Thanks!
Re: perl clang -Wcompound-token-split-by-macro
On Tue, Jan 25, 2022 at 09:20:55PM +0100, Alexander Bluhm wrote: > On Tue, Jan 25, 2022 at 12:05:48PM -0800, Andrew Hewus Fresh wrote: > > On Tue, Jan 25, 2022 at 06:45:12PM +0100, Alexander Bluhm wrote: > > > On Tue, Jan 25, 2022 at 05:13:01PM +0100, Alexander Bluhm wrote: > > > > On Sat, Jan 22, 2022 at 02:24:51AM +0100, Marc Espie wrote: > > > > > Or we can automate this with something like this: > > > > > > > > > > Our Devel::PPPort is too old. We ship with 3.57, p5-CDB_File and p5-Moose > > > ship with ppport.h generated by 3.62. > > > > We could update Devel::PPPort in base. > > This will fix the current problem. But we can always have an old > Devel::PPPort in base and a have module in ports that comes with > and needs a new ppport.h. > > Somehow the porter should have a mechanism to handle this. Only 2 > of 300 ports that I test have issues, so it is a rare action. Per > default espie@'s idea works well. If we manually fix 2 ports that > is fine for me. > > Or we replace ppport.h only if it is outdated. We could add espie's target to those two ports. > > I do plan to work on getting 5.34 in after I finish my fw_update(8) TODO > > list and get the "vendor lib" patch committed. I'm not sure that will > > make 7.1 though. > > Plans for new Perl are good to hear. Thanks for updating it regualry. Sure thing, some exciting stuff coming in the future. I think signatures may actually make it out of experimental status. l8rZ, -- andrew Adding manpower to a late software project makes it later.
Re: perl clang -Wcompound-token-split-by-macro
On Tue, Jan 25, 2022 at 06:45:12PM +0100, Alexander Bluhm wrote: > On Tue, Jan 25, 2022 at 05:13:01PM +0100, Alexander Bluhm wrote: > > On Sat, Jan 22, 2022 at 02:24:51AM +0100, Marc Espie wrote: > > > Or we can automate this with something like this: > > > > Our Devel::PPPort is too old. We ship with 3.57, p5-CDB_File and p5-Moose > ship with ppport.h generated by 3.62. We could update Devel::PPPort in base. I do plan to work on getting 5.34 in after I finish my fw_update(8) TODO list and get the "vendor lib" patch committed. I'm not sure that will make 7.1 though. > > ppport.h -- Perl/Pollution/Portability Version 3.57 > Automatically created by Devel::PPPort running under perl 5.032001. > > ppport.h -- Perl/Pollution/Portability Version 3.62 > Automatically created by Devel::PPPort running under perl 5.032000. > > ppport.h -- Perl/Pollution/Portability Version 3.62 > Automatically created by Devel::PPPort running under perl 5.033008. > > bluhm >
Re: fw_update: unregister firmware that has moved to base
On Sun, Jan 23, 2022 at 06:01:44PM -0800, Andrew Hewus Fresh wrote:
> This patch will "unregister" a firmware package if we detect that it has
> moved into the base system by removing the package files in /var/db/pkg
> and leaving the files in /etc/firmware alone.
>
> This situation is detected with a special entry in the firmware_patterns
> file where the "pattern" is "INBASE" instead of an actual pattern.
> Normally pkg_add stores that information in the quirks package, but we
> don't have easy access to that, so overloading the firmware_patterns
> file seemed the best course of action. If folks have a better idea
> where we could learn that a firmware now comes with the base system, I'd
> be interested in suggestions.
>
> Comments, OK?
It was suggested that instead of having quirks in the firmware_patterns
file, if we can't find a filename for an installed firmware, that is
enough indication that it has moved to base.
Here is a patch that implements that suggestion.
Index: usr.sbin/fw_update/fw_update.sh
===
RCS file: /cvs/src/usr.sbin/fw_update/fw_update.sh,v
retrieving revision 1.31
diff -u -p -r1.31 fw_update.sh
--- usr.sbin/fw_update/fw_update.sh 22 Jan 2022 05:03:47 - 1.31
+++ usr.sbin/fw_update/fw_update.sh 24 Jan 2022 02:26:13 -
@@ -170,9 +170,7 @@ firmware_in_dmesg() {
firmware_filename() {
local _f
[ -e "$CFILE" ] || fetch_cfile || return 1
- _f="$( sed -n "s/.*(\($1-firmware-.*\.tgz\)).*/\1/p" "$CFILE" | sed
'$!d' )"
- ! [ "$_f" ] && echo "Unable to find firmware for $1" >&2 && return 1
- echo "$_f"
+ sed -n "s/.*(\($1-firmware-.*\.tgz\)).*/\1/p" "$CFILE" | sed '$!d'
}
firmware_devicename() {
@@ -252,6 +250,21 @@ EOL
unset FWPKGTMP
}
+remove_files() {
+ # Use rm -f, not removing files/dirs is probably not worth failing over
+ for _r in "$@" ; do
+ if [ -d "$_r" ]; then
+ # The installer lacks rmdir,
+ # but we only want to remove empty directories.
+ set +o noglob
+ [ "$_r/*" = "$( echo "$_r"/* )" ] && rm -rf "$_r"
+ set -o noglob
+ else
+ rm -f "$_r"
+ fi
+ done
+}
+
delete_firmware() {
local _cwd _pkg="$1" _pkgdir="${DESTDIR}/var/db/pkg"
@@ -278,24 +291,32 @@ delete_firmware() {
esac
done < "${_pkgdir}/${_pkg}/+CONTENTS"
- # Use rm -f, not removing files/dirs is probably not worth failing over
- for _r in "${_remove[@]}" ; do
- if [ -d "$_r" ]; then
- # The installer lacks rmdir,
- # but we only want to remove empty directories.
- set +o noglob
- [ "$_r/*" = "$( echo "$_r"/* )" ] && rm -rf "$_r"
- set -o noglob
- else
- rm -f "$_r"
- fi
- done
+ remove_files "${_remove[@]}"
((VERBOSE > 2)) && echo " done."
return 0
}
+unregister_firmware() {
+ local _d="$1" _pkgdir="${DESTDIR}/var/db/pkg"
+
+ set -A installed -- $( installed_firmware '' "$d-firmware-" '*' )
+ if [ "${installed:-}" ]; then
+ for fw in "${installed[@]}"; do
+ ((VERBOSE)) && echo "Unregister $fw"
+ "$DRYRUN" && continue
+ remove_files \
+ "$_pkgdir/$fw/+CONTENTS" \
+ "$_pkgdir/$fw/+DESC" \
+ "$_pkgdir/$fw/"
+ done
+ return 0
+ fi
+
+ return 1
+}
+
usage() {
echo "usage: ${0##*/} [-adFnv] [-p path] [driver | file ...]"
exit 2
@@ -430,13 +451,21 @@ fi
added=''
updated=''
kept=''
+unregister=''
for f in "${devices[@]}"; do
d="$( firmware_devicename "$f" )"
verify_existing="$DOWNLOAD"
if [ "$f" = "$d" ]; then
f=$( firmware_filename "$d" || true )
- [ "$f" ] || continue
+ if [ ! "$f" ]; then
+ if "$INSTALL" && unregister_firmware "$d"; then
+ unregister=
fw_update: unregister firmware that has moved to base
This patch will "unregister" a firmware package if we detect that it has
moved into the base system by removing the package files in /var/db/pkg
and leaving the files in /etc/firmware alone.
This situation is detected with a special entry in the firmware_patterns
file where the "pattern" is "INBASE" instead of an actual pattern.
Normally pkg_add stores that information in the quirks package, but we
don't have easy access to that, so overloading the firmware_patterns
file seemed the best course of action. If folks have a better idea
where we could learn that a firmware now comes with the base system, I'd
be interested in suggestions.
Comments, OK?
Index: usr.sbin/fw_update/fw_update.sh
===
RCS file: /cvs/src/usr.sbin/fw_update/fw_update.sh,v
retrieving revision 1.31
diff -u -p -r1.31 fw_update.sh
--- usr.sbin/fw_update/fw_update.sh 22 Jan 2022 05:03:47 - 1.31
+++ usr.sbin/fw_update/fw_update.sh 24 Jan 2022 01:58:07 -
@@ -157,6 +157,7 @@ firmware_in_dmesg() {
grep -v '^[[:space:]]*#' "$FWPATTERNS" |
while read -r _d _m; do
[ "$_d" = "$_last" ] && continue
+ [ "$_m" = INBASE ] && continue
[ "$_m" ] || _m="${_nl}${_d}[0-9] at "
[ "$_m" = "${_m#^}" ] || _m="${_nl}${_m#^}"
@@ -170,9 +171,7 @@ firmware_in_dmesg() {
firmware_filename() {
local _f
[ -e "$CFILE" ] || fetch_cfile || return 1
- _f="$( sed -n "s/.*(\($1-firmware-.*\.tgz\)).*/\1/p" "$CFILE" | sed
'$!d' )"
- ! [ "$_f" ] && echo "Unable to find firmware for $1" >&2 && return 1
- echo "$_f"
+ sed -n "s/.*(\($1-firmware-.*\.tgz\)).*/\1/p" "$CFILE" | sed '$!d'
}
firmware_devicename() {
@@ -252,6 +251,21 @@ EOL
unset FWPKGTMP
}
+remove_files() {
+ # Use rm -f, not removing files/dirs is probably not worth failing over
+ for _r in "$@" ; do
+ if [ -d "$_r" ]; then
+ # The installer lacks rmdir,
+ # but we only want to remove empty directories.
+ set +o noglob
+ [ "$_r/*" = "$( echo "$_r"/* )" ] && rm -rf "$_r"
+ set -o noglob
+ else
+ rm -f "$_r"
+ fi
+ done
+}
+
delete_firmware() {
local _cwd _pkg="$1" _pkgdir="${DESTDIR}/var/db/pkg"
@@ -278,24 +292,32 @@ delete_firmware() {
esac
done < "${_pkgdir}/${_pkg}/+CONTENTS"
- # Use rm -f, not removing files/dirs is probably not worth failing over
- for _r in "${_remove[@]}" ; do
- if [ -d "$_r" ]; then
- # The installer lacks rmdir,
- # but we only want to remove empty directories.
- set +o noglob
- [ "$_r/*" = "$( echo "$_r"/* )" ] && rm -rf "$_r"
- set -o noglob
- else
- rm -f "$_r"
- fi
- done
+ remove_files "${_remove[@]}"
((VERBOSE > 2)) && echo " done."
return 0
}
+unregister_if_moved_to_base() {
+ local _d="$1" _pkgdir="${DESTDIR}/var/db/pkg"
+ grep -q "^$_d[[:space:]]*INBASE" "$FWPATTERNS" || return 1
+
+ set -A installed -- $( installed_firmware '' "$d-firmware-" '*' )
+ if [ "${installed:-}" ]; then
+ for fw in "${installed[@]}"; do
+ ((VERBOSE)) && echo "Unregister $fw"
+ "$DRYRUN" && continue
+ remove_files \
+ "$_pkgdir/$fw/+CONTENTS" \
+ "$_pkgdir/$fw/+DESC" \
+ "$_pkgdir/$fw/"
+ done
+ fi
+
+ return 0
+}
+
usage() {
echo "usage: ${0##*/} [-adFnv] [-p path] [driver | file ...]"
exit 2
@@ -430,13 +452,21 @@ fi
added=''
updated=''
kept=''
+unregister=''
for f in "${devices[@]}"; do
d="$( firmware_devicename "$f" )"
verify_existing="$DOWNLOAD"
if [ "$f" = "$d" ]; then
f=$( firmware_filename "$d" || true )
- [ "$f" ] || continue
+ if [ ! "$f" ]; then
+ if "$INSTALL" && unregister_if_moved_to_base "$d"; then
+ unregister="$unregister,$d"
+ else
+ echo "Unable to find firmware for $d" >&2
+ fi
+ continue
+ fi
f="$LOCALSRC/$f"
elif ! "$INSTALL" && ! grep -Fq "($f)" "$CFILE" ; then
echo "Cannot download local file $f" >&2
@@ -532,8 +562,9 @@ done
added="${added:#,}"
updated="${updated:#,}"
kept="${kept:#,}"
+[ "${unregister:-}" ] && unregister="; unregistered ${unregister:#,}"
if "$INSTALL"; then
- echo "${0##*/}: added
Re: fw_update(8) Redownload files that fail validation
Here's an updated patch that applies after recent commits. I'll commit
this sometime this weekend unless someone has other ideas.
On Sat, Jan 08, 2022 at 08:30:09PM -0800, Andrew Hewus Fresh wrote:
> That last patch I committed made it so that if you specify a local path
> to install from with -p, while it will look in the SHA256.sig file for
> the filename it didn't actually verify the checksum.
>
> In any case, this patch means we verify the checksum unless someone
> specified a local filename on the command line. If we translate a
> driver name into a filename we verify the checksum.
>
> This also adjusts so that if the existing file fails the checksum and
> we're downloading (with -F for example) it will rm the file and
> re-download it.
>
> It _doesn't_ try again for a file that is downloaded and fails
> validation, either right after having rm'd a file that failed or if it
> didn't previously exist and it does leave the successfully downloaded
> (but failed validation) file on disk. I haven't decided if that's the
> right choice yet.
>
> Comments, OK?
Index: fw_update.sh
===
RCS file: /cvs/src/usr.sbin/fw_update/fw_update.sh,v
retrieving revision 1.31
diff -u -p -r1.31 fw_update.sh
--- fw_update.sh22 Jan 2022 05:03:47 - 1.31
+++ fw_update.sh22 Jan 2022 05:27:58 -
@@ -433,7 +433,7 @@ kept=''
for f in "${devices[@]}"; do
d="$( firmware_devicename "$f" )"
- verify_existing="$DOWNLOAD"
+ verify_existing=true
if [ "$f" = "$d" ]; then
f=$( firmware_filename "$d" || true )
[ "$f" ] || continue
@@ -459,24 +459,27 @@ for f in "${devices[@]}"; do
fi
pending_status=false
- if [ -e "$f" ]; then
- if "$verify_existing" && ! "$DRYRUN"; then
- if ((VERBOSE == 1)); then
- echo -n "Verify ${f##*/} ..."
- pending_status=true
- fi
- ((VERBOSE > 1)) && ! "$INSTALL" &&
- echo "Keep/Verify ${f##*/}"
- verify "$f" || {
- "$pending_status" && echo " failed."
- continue
- }
- "$pending_status" && ! "$INSTALL" && echo " done."
- else
- ((VERBOSE > 1)) && ! "$INSTALL" &&
- echo "Keep ${f##*/}"
+ if "$verify_existing" && [ -e "$f" ]; then
+ if ((VERBOSE == 1)); then
+ echo -n "Verify ${f##*/} ..."
+ pending_status=true
+ elif ((VERBOSE > 1)) && ! "$INSTALL"; then
+ echo "Keep/Verify ${f##*/}"
fi
- "$INSTALL" || kept="$kept,$d"
+
+ if "$DRYRUN" || verify "$f"; then
+ "$INSTALL" || kept="$kept,$d"
+ elif "$DOWNLOAD"; then
+ ((VERBOSE > 1)) && echo "Refetching $f"
+ rm -f $f
+ else
+ "$pending_status" && echo " failed."
+ continue
+ fi
+ fi
+
+ if [ -e "$f" ]; then
+ "$pending_status" && ! "$INSTALL" && echo " done."
elif "$DOWNLOAD"; then
if "$DRYRUN"; then
((VERBOSE)) && echo "Get/Verify ${f##*/}"
Re: perl clang -Wcompound-token-split-by-macro
On Sat, Jan 22, 2022 at 02:24:51AM +0100, Marc Espie wrote:
> Or we can automate this with something like this:
I didn't try this, but it seems OK to me. I'll leave the final OK to
sthen though as I would think it most likely to break things for him.
(AFAIU new versions are not supposed to cause breakage though)
> Index: perl.port.mk
> ===
> RCS file: /cvs/ports/infrastructure/mk/perl.port.mk,v
> retrieving revision 1.32
> diff -u -p -r1.32 perl.port.mk
> --- perl.port.mk 12 Dec 2021 19:25:39 - 1.32
> +++ perl.port.mk 21 Jan 2022 17:39:18 -
> @@ -56,6 +56,11 @@ MODPERL_pre-configure = for f in ${MODPE
> ${MODPERL_BIN_ADJ} ${WRKSRC}/$${f}; done
> .endif
>
> +MODPERL_gen = cd ${WRKDIST} && \
> + if test -f ppport.h; then \
> + perl -MDevel::PPPort -e'Devel::PPPort::WriteFile'; \
> + fi
> +
> .if ${CONFIGURE_STYLE:L:Mmodbuild}
> MODPERL_configure = \
> cd ${WRKSRC}; ${SETENV} ${CONFIGURE_ENV} \
Re: perl clang -Wcompound-token-split-by-macro
On Fri, Jan 21, 2022 at 04:34:13PM +0100, Marc Espie wrote:
> On Fri, Jan 21, 2022 at 02:12:25PM +0100, Alexander Bluhm wrote:
> > Hi,
> >
> > Since clang 13 each Perl or Perl XS module compile spits out a lot
> > of -Wcompound-token-split-by-macro warnings. E.g. p5-Net-SSLeay
> > produces 3882 warnings generated. You cannot spot anything useful.
> > The problem is burried deeply in the Perl macros and copied to
> > everywhere.
> >
> > If we compile Perl with -Wno-compound-token-split-by-macro it gets
> > stored in Config.pm and is used for most modules.
> >
> > $ perl -MConfig -e 'print $Config{ccflags},"\n"'
> > -Wno-compound-token-split-by-macro -fno-strict-aliasing
> > -fno-delete-null-pointer-checks -pipe -fstack-protector-strong
> > -I/usr/local/include
> >
> > ok?
>
> I'm wondering whether this warning makes any sense for the ({ gnu C statement
> (which isn't really a C token per se)
>
> As for "deep within perl", it comes from a combination of two things:
>
> - perl.h uses STMT_START and STMT_END (the normal ones just
> creating do {} while (0) grouping.
Which jsg changed recently for llvm 13 clang to get rid of warnings
building perl itself.
https://github.com/openbsd/src/commit/97a5f6039458a57e6c05bb910c104ac2ecee7622
> - there's a module called Devel::PPPort that generates a ppport.h (normally
> the most recent).
>
> using
> $ perl -MDevel::PPPort -e'Devel::PPPort::WriteFile'
>
> as documented in the manpage to regen a more recent ppport.h results in
> p5-Net-SSLeay producing exactly 0 warning while building.
>
> So I don't really think perl requires any change.
>
> Possibly hacking a bit on ports that use an outdated version of ppport.h
Updating ppport.h seems reasonable to me.
Re: Add versioned lib to system perl's @INC for non-packaged modules
The patch over in ports that was blocking this has been in for a while:
https://github.com/openbsd/ports/commit/bd756bd601bfcfa9ee7fc265d6c90c3536e51727
Although I haven't committed any of the port fixes to tell the ones that
don't pick it up automatically to install into the vendor lib, the
patches are over in that thread. There will likely be other ports that
need some sort of fix:
https://marc.info/?l=openbsd-ports=162769529204194=2
In any case, I'd like to get this in to hopefully lower the number of
broken systems when folks install modules with CPAN clients in the
future.
I am considering removing the installperl patch and creating the
directory in Makefile.bsd-wraapper for fewer local patches to maintain.
Comments, test results, OK?
On Fri, Jul 30, 2021 at 05:34:40PM -0700, Andrew Hewus Fresh wrote:
> Below is the corrected patch for this, but it's not quite ready yet.
> See the email over on ports@ with some adjustments to perl.port.mk and
> other patches to fix some ports fallout that sthen@ was super helpful in
> tracking down.
>
> The main issue was that somehow I lost a quote in the patch and so
> it didn't work right. But, this should apply and let you play with it
> if you like.
>
> Once we have a path forward with the ports fallout, I'll be looking for
> more comments and maybe OKs.
>
> On Sun, May 16, 2021 at 03:30:39PM -0700, Andrew Hewus Fresh wrote:
> > This patch is should make it easier to recover from problems that happen
> > when installing perl modules outside of the package system. You might
> > recognize the "loadable library and perl binaries are mismatched" error.
> >
> > Adding a versioned subdirectory for cpan modules means that when the
> > perl version changes it will no longer find those old modules and it
> > will be easy to remove them without disturbing things from packages.
> >
> > Only slightly related, but we can avoid breaking base tools like pkg_add
> > and friend by using "no lib '/usr/local/libdata/perl5/site_perl';" to
> > avoid looking in those directories at all.
> >
> >
> > The new @INC looks like this. The first "5.32.1" sitelib is where
> > external things, like cpan(1), will install. The current "site_perl"
> > directory (now vendorlib) will continue to be where ports/packages go.
> > Finally, "/usr/libdata" privlib is for things that ship with the base
> > system.
> >
> > $ perl -V | perl -ne 'print if /INC/..eof'
> > @INC:
> > /usr/local/libdata/perl5/site_perl/5.32.1/sparc64-openbsd
> > /usr/local/libdata/perl5/site_perl/5.32.1
> > /usr/local/libdata/perl5/site_perl/sparc64-openbsd
> > /usr/local/libdata/perl5/site_perl
> > /usr/libdata/perl5/sparc64-openbsd
> > /usr/libdata/perl5
> >
> >
> > One place this might cause temporary issues in the ports tree is
> > anything that doesn't use the overrides in perl.port.mk and looks up
> > sitelib itself. It should be fairly obvious as it should fail to
> > package because the files in the PLIST will not exist (they'll be in the
> > versioned subdir). I didn't run into that in my testing, but I didn't
> > complete a full bulk build.
> >
> > There do appear to be some annoyances with still shared directories for
> > man pages, in that if you install a CPAN module and then attempt to
> > pkg_add it, it complains because "files already exist" in the man
> > directory. We could separate the cpan man pages, or I think un-setting
> > the site_lib man*dir will mean cpan won't instal them. I haven't yet
> > tested that, but I'm not sure what would be best there.
> >
> >
> > Unfortunately I did do a bunch of testing of different ways to add
> > things to @INC which meant this patch also does some reorganization of
> > the contents, I can turn it into a smaller patch if requested and do the
> > cleanup separately.
> >
> >
> > Comments, OK? Should I see if sthen@ will do a full bulk build first?
Index: gnu/usr.bin/perl/config.over
===
RCS file: /cvs/src/gnu/usr.bin/perl/config.over,v
retrieving revision 1.22
diff -u -p -r1.22 config.over
--- gnu/usr.bin/perl/config.over5 Feb 2017 00:33:38 - 1.22
+++ gnu/usr.bin/perl/config.over31 Jul 2021 00:02:14 -
@@ -9,49 +9,85 @@ archname="`arch -s`-${osname}"
myarchname="$archname"
# Use correct paths for a distribution
-prefix='/usr'
+# site is where cpan clients install files
+# vendor is where the ports tree puts packages
+# priv is where the system installs files
+
+# We set up vendor paths for por
fw_update(8) minor fixes
I pulled some fixes out of the verbose patch so hopefully I can get them
in while folks are trying out the more complex patch.
Plus one change to usage from jmc@. His other option was to move the
[-F | -d] into the man page, but we can just complain more specifically
if they try to use incompatible arguments and keep the documentation
simpler.
Comments, OK?
--- usr.sbin/fw_update/fw_update.sh.origMon Jan 10 19:25:52 2022
+++ usr.sbin/fw_update/fw_update.sh Mon Jan 10 19:51:05 2022
@@ -95,7 +95,7 @@
SECONDS=0
sleep 1
else
- kill -INT -"$FTPPID"
+ kill -INT -"$FTPPID" 2>/dev/null
_error=" (timed out)"
fi
else
@@ -201,14 +201,14 @@
set -sA _devices -- $(
firmware_in_dmesg
for _d in $( installed_firmware '*' '-firmware-' '*' ); do
- echo "$( firmware_devicename "$_d" )"
+ firmware_devicename "$_d"
done
)
[ "${_devices[*]:-}" ] || return 0
for _d in "${_devices[@]}"; do
- [[ $_last = $_d ]] && continue
- echo $_d
+ [ "$_last" = "$_d" ] && continue
+ echo "$_d"
_last="$_d"
done
}
@@ -286,7 +286,7 @@
}
usage() {
- echo "usage: ${0##*/} [-d | -F] [-av] [-p path] [driver | file ...]"
+ echo "usage: ${0##*/} [-adFnv] [-p path] [driver | file ...]"
exit 2
}
@@ -352,11 +352,11 @@
set -sA devices -- "$@"
if "$DELETE"; then
- [ "$OPT_F" ] && usage 22
+ [ "$OPT_F" ] && echo "Cannot use -F and -d" >&2 && usage 22
set -A installed
if [ "${devices[*]:-}" ]; then
- "$ALL" && usage 22
+ "$ALL" && echo "Cannot use -a and devices/files" >&2 && usage 22
set -A installed -- $(
for d in "${devices[@]}"; do
@@ -389,7 +389,7 @@
done
fi
- deleted="${deleted:+${deleted#,}}"
+ deleted="${deleted#,}"
echo "${0:##*/}: deleted ${deleted:-none}";
exit
@@ -403,7 +403,7 @@
CFILE="$LOCALSRC/$CFILE"
if [ "${devices[*]:-}" ]; then
- "$ALL" && usage 22
+ "$ALL" && echo "Cannot use -a and devices/files" >&2 && usage 22
else
"$VERBOSE" && echo -n "Detecting firmware ..."
set -sA devices -- $( detect_firmware )
fw_update(8) Redownload files that fail validation
That last patch I committed made it so that if you specify a local path
to install from with -p, while it will look in the SHA256.sig file for
the filename it didn't actually verify the checksum.
In any case, this patch means we verify the checksum unless someone
specified a local filename on the command line. If we translate a
driver name into a filename we verify the checksum.
This also adjusts so that if the existing file fails the checksum and
we're downloading (with -F for example) it will rm the file and
re-download it.
It _doesn't_ try again for a file that is downloaded and fails
validation, either right after having rm'd a file that failed or if it
didn't previously exist and it does leave the successfully downloaded
(but failed validation) file on disk. I haven't decided if that's the
right choice yet.
Comments, OK?
Index: fw_update.sh
===
RCS file: /cvs/src/usr.sbin/fw_update/fw_update.sh,v
retrieving revision 1.28
diff -u -p -r1.28 fw_update.sh
--- fw_update.sh8 Jan 2022 22:32:00 - 1.28
+++ fw_update.sh9 Jan 2022 04:21:09 -
@@ -424,7 +424,7 @@ kept=''
for f in "${devices[@]}"; do
d="$( firmware_devicename "$f" )"
- verify_existing="$DOWNLOAD"
+ verify_existing=true
if [ "$f" = "$d" ]; then
f=$( firmware_filename "$d" || true )
[ "$f" ] || continue
@@ -450,19 +450,24 @@ for f in "${devices[@]}"; do
done
fi
+ if "$verify_existing" && [ -e "$f" ]; then
+ msg="Keep/Verify"
+ "$INSTALL" && msg="Verify"
+ "$VERBOSE" && ! "$INSTALL" &&
+ echo "$msg ${f##*/}"
+
+ if "$DRYRUN" || verify "$f"; then
+ "$INSTALL" || kept="$kept,$d"
+ elif "$DOWNLOAD"; then
+ "$VERBOSE" && echo "Refetching $f"
+ rm -f $f
+ else
+ continue
+ fi
+ fi
+
if [ -e "$f" ]; then
- if "$DOWNLOAD"; then
- if "$verify_existing" && ! "$DRYRUN"; then
- "$VERBOSE" && ! "$INSTALL" &&
- echo "Keep/Verify ${f##*/}"
- verify "$f" || continue
- else
- "$VERBOSE" && ! "$INSTALL" &&
- echo "Keep ${f##*/}"
- fi
- "$INSTALL" || kept="$kept,$d"
- # else assume it was verified when downloaded
- fi
+ true # verified above
elif "$DOWNLOAD"; then
if "$DRYRUN"; then
"$VERBOSE" && echo "Get/Verify ${f##*/}"
Re: fw_update(8) Avoid downloading SHA256.sig unless we need it
On Thu, Jan 06, 2022 at 06:13:33PM -0800, Andrew Hewus Fresh wrote:
> deraadt@ noticed that if you install a list of local files fw_update
> still tries to download the SHA256.sig and will fail if it can't. That
> doesn't make sense, so instead only download it if we're going to use
> it and it doesn't exist. It does continue to attempt to update the
> SHA256.sig if we're downloading firmware in case there is a new one and
> possibly new firmware.
>
> Comments, OK?
Well, that attempt didn't quite work and apparently I failed to break my
network enough to notice. This one seems better though.
protip, return 0 at the end of a successful function if you care about
the return value.
Index: usr.sbin/fw_update/fw_update.sh
===
RCS file: /cvs/src/usr.sbin/fw_update/fw_update.sh,v
retrieving revision 1.26
diff -u -p -r1.26 fw_update.sh
--- usr.sbin/fw_update/fw_update.sh 6 Jan 2022 20:15:54 - 1.26
+++ usr.sbin/fw_update/fw_update.sh 7 Jan 2022 04:07:36 -
@@ -117,14 +117,34 @@ fetch() {
echo "Cannot fetch $_src$_error" >&2
return 1
fi
+
+ return 0
+}
+
+fetch_cfile() {
+ if "$DOWNLOAD"; then
+ set +o noclobber # we want to get the latest CFILE
+ fetch "$CFILE" || return 1
+ set -o noclobber
+ ! signify -qVep "$FWPUB_KEY" -x "$CFILE" -m "$CFILE" &&
+ echo "Signature check of SHA256.sig failed" >&2 && return 1
+ elif [ ! -e "$CFILE" ]; then
+ echo "${0##*/}: $CFILE: No such file or directory" >&2
+ return 2
+ fi
+
+ return 0
}
verify() {
+ [ -e "$CFILE" ] || fetch_cfile || return 1
# On the installer we don't get sha256 -C, so fake it.
if ! fgrep -qx "SHA256 (${1##*/}) = $( /bin/sha256 -qb "$1" )"
"$CFILE"; then
echo "Checksum test for ${1##*/} failed." >&2
return 1
fi
+
+ return 0
}
firmware_in_dmesg() {
@@ -149,6 +169,7 @@ firmware_in_dmesg() {
firmware_filename() {
local _f
+ [ -e "$CFILE" ] || fetch_cfile || return 1
_f="$( sed -n "s/.*(\($1-firmware-.*\.tgz\)).*/\1/p" "$CFILE" | sed
'$!d' )"
! [ "$_f" ] && echo "Unable to find firmware for $1" >&2 && return 1
echo "$_f"
@@ -313,6 +334,17 @@ fi
if [ "$OPT_F" ]; then
INSTALL=false
LOCALSRC="${LOCALSRC:-.}"
+
+ # Always check for latest CFILE and so latest firmware
+ if [ -e "$LOCALSRC/$CFILE" ]; then
+ mv "$LOCALSRC/$CFILE" "$LOCALSRC/$CFILE-OLD"
+ if fetch_cfile; then
+ rm -f "$LOCALSRC/$CFILE-OLD"
+ else
+ mv "$LOCALSRC/$CFILE-OLD" "$LOCALSRC/$CFILE"
+ echo "Using existing $CFILE" >&2
+ fi
+ fi
elif [ "$LOCALSRC" ]; then
DOWNLOAD=false
fi
@@ -386,24 +418,13 @@ fi
[ "${devices[*]:-}" ] || exit
-if "$DOWNLOAD"; then
- set +o noclobber # we want to get the latest CFILE
- fetch "$CFILE"
- set -o noclobber
- ! signify -qVep "$FWPUB_KEY" -x "$CFILE" -m "$CFILE" &&
- echo "Signature check of SHA256.sig failed" >&2 && exit 1
-elif [ ! -e "$CFILE" ]; then
- # TODO: We shouldn't need a CFILE if all arguments are files.
- echo "${0##*/}: $CFILE: No such file or directory" >&2
- exit 2
-fi
-
added=''
updated=''
kept=''
for f in "${devices[@]}"; do
d="$( firmware_devicename "$f" )"
+ verify_existing="$DOWNLOAD"
if [ "$f" = "$d" ]; then
f=$( firmware_filename "$d" || true )
[ "$f" ] || continue
@@ -411,6 +432,10 @@ for f in "${devices[@]}"; do
elif ! "$INSTALL" && ! grep -Fq "($f)" "$CFILE" ; then
echo "Cannot download local file $f" >&2
exit 2
+ else
+ # If someone specified a filename on the command-line
+ # we don't want to verify it.
+ verify_existing=false
fi
set -A installed -- $( installed_firmware '' "$d-firmware-" '*' )
@@ -429,7 +454,9 @@ for f in "${devices[@]}"; do
if "$DOWNLOAD"; then
"$VERBOSE" && ! "$INSTALL" &&
echo "Keep/Verify ${f##*/}"
- "$DRYRUN" || verify "$f" || continue
+ if "$verify_existing" && ! "$DRYRUN"; then
+ verify "$f" || continue
+ fi
"$INSTALL" || kept="$kept,$d"
# else assume it was verified when downloaded
fi
fw_update(8) Avoid downloading SHA256.sig unless we need it
deraadt@ noticed that if you install a list of local files fw_update
still tries to download the SHA256.sig and will fail if it can't. That
doesn't make sense, so instead only download it if we're going to use
it and it doesn't exist. It does continue to attempt to update the
SHA256.sig if we're downloading firmware in case there is a new one and
possibly new firmware.
Comments, OK?
Index: fw_update.sh
===
RCS file: /cvs/src/usr.sbin/fw_update/fw_update.sh,v
retrieving revision 1.25
diff -u -p -r1.25 fw_update.sh
--- fw_update.sh6 Jan 2022 19:27:01 - 1.25
+++ fw_update.sh7 Jan 2022 02:08:34 -
@@ -119,7 +119,21 @@ fetch() {
fi
}
+fetch_cfile() {
+ if "$DOWNLOAD"; then
+ set +o noclobber # we want to get the latest CFILE
+ fetch "$CFILE" || return 1
+ set -o noclobber
+ ! signify -qVep "$FWPUB_KEY" -x "$CFILE" -m "$CFILE" &&
+ echo "Signature check of SHA256.sig failed" >&2 && return 1
+ elif [ ! -e "$CFILE" ]; then
+ echo "${0##*/}: $CFILE: No such file or directory" >&2
+ return 2
+ fi
+}
+
verify() {
+ [ -e "$CFILE" ] || fetch_cfile
# On the installer we don't get sha256 -C, so fake it.
if ! fgrep -qx "SHA256 (${1##*/}) = $( /bin/sha256 -qb "$1" )"
"$CFILE"; then
echo "Checksum test for ${1##*/} failed." >&2
@@ -149,6 +163,7 @@ firmware_in_dmesg() {
firmware_filename() {
local _f
+ [ -e "$CFILE" ] || fetch_cfile
_f="$( sed -n "s/.*(\($1-firmware-.*\.tgz\)).*/\1/p" "$CFILE" | sed
'$!d' )"
! [ "$_f" ] && echo "Unable to find firmware for $1" >&2 && return 1
echo "$_f"
@@ -313,6 +328,17 @@ fi
if [ "$OPT_F" ]; then
INSTALL=false
LOCALSRC="${LOCALSRC:-.}"
+
+ # Always check for latest CFILE and so latest firmware
+ if [ -e "$LOCALSRC/$CFILE" ]; then
+ mv "$LOCALSRC/$CFILE" "$LOCALSRC/$CFILE-OLD"
+ if fetch_cfile; then
+ rm -f "$LOCALSRC/$CFILE-OLD"
+ else
+ mv "$LOCALSRC/$CFILE-OLD" "$LOCALSRC/$CFILE"
+ echo "Using existing $CFILE" >&2
+ fi
+ fi
elif [ "$LOCALSRC" ]; then
DOWNLOAD=false
fi
@@ -385,18 +411,6 @@ else
fi
[ "${devices[*]:-}" ] || exit
-
-if "$DOWNLOAD"; then
- set +o noclobber # we want to get the latest CFILE
- fetch "$CFILE"
- set -o noclobber
- ! signify -qVep "$FWPUB_KEY" -x "$CFILE" -m "$CFILE" &&
- echo "Signature check of SHA256.sig failed" >&2 && exit 1
-elif [ ! -e "$CFILE" ]; then
- # TODO: We shouldn't need a CFILE if all arguments are files.
- echo "${0##*/}: $CFILE: No such file or directory" >&2
- exit 2
-fi
added=''
updated=''
fw_update(8) improve verbose output
After getting fw_update(8) into a state where it could get some testing,
I found that the man page indicated that -v should indicate different
levels of verbosity and I currently only had one.
This was useful as I didn't really like the output anyway.
Now one -v prints out an additional line when it's doing something to
the firmware. Two add a progress bar and mentions "detecting". Three
provide a bit more debugging mostly from ftp(1).
Also a couple extra small improvements, hiding errors from killing the
ftp subprocess which could happen if it exits before we do, just
using `firmware_devicename "$_d"` instead of `echo "${...}"`, and using
a normal [=] comparison instead of [[=]] because we don't want pattern
matching there.
Comments, OK?
Index: usr.sbin/fw_update/fw_update.sh
===
RCS file: /cvs/src/usr.sbin/fw_update/fw_update.sh,v
retrieving revision 1.24
diff -u -p -r1.24 fw_update.sh
--- usr.sbin/fw_update/fw_update.sh 5 Jan 2022 16:32:46 - 1.24
+++ usr.sbin/fw_update/fw_update.sh 6 Jan 2022 01:28:03 -
@@ -35,7 +35,7 @@ FWURL=http://firmware.openbsd.org/firmwa
FWPUB_KEY=${DESTDIR}/etc/signify/openbsd-${VERSION}-fw.pub
DRYRUN=false
-VERBOSE=false
+VERBOSE=0
DELETE=false
DOWNLOAD=true
INSTALL=true
@@ -75,14 +75,17 @@ fetch() {
# we have su(1) and doas(1) is unlikely to be configured.
set -o monitor # make sure ftp gets its own process group
(
- flags=-VM
- "$VERBOSE" && flags=-vm
+ _flags=-vm
+ case "$VERBOSE" in
+ 0|1) _flags=-VM ;;
+ 2) _flags=-Vm ;;
+ esac
if [ -x /usr/bin/su ]; then
exec /usr/bin/su -s /bin/ksh "$_user" -c \
- "/usr/bin/ftp -N '${0##/}' -D 'Get/Verify' $flags -o-
'$_src'" > "$_dst"
+ "/usr/bin/ftp -N '${0##/}' -D 'Get/Verify' $_flags -o-
'$_src'" > "$_dst"
else
exec /usr/bin/doas -u "$_user" \
- /usr/bin/ftp -N "${0##/}" -D 'Get/Verify' $flags -o-
"$_src" > "$_dst"
+ /usr/bin/ftp -N "${0##/}" -D 'Get/Verify' $_flags -o-
"$_src" > "$_dst"
fi
) & FTPPID=$!
set +o monitor
@@ -97,7 +100,7 @@ fetch() {
SECONDS=0
sleep 1
else
- kill -INT -"$FTPPID"
+ kill -INT -"$FTPPID" 2>/dev/null
_error=" (timed out)"
fi
else
@@ -183,24 +186,28 @@ detect_firmware() {
set -sA _devices -- $(
firmware_in_dmesg
for _d in $( installed_firmware '*' '-firmware-' '*' ); do
- echo "$( firmware_devicename "$_d" )"
+ firmware_devicename "$_d"
done
)
[ "${_devices[*]:-}" ] || return 0
for _d in "${_devices[@]}"; do
- [[ $_last = $_d ]] && continue
- echo $_d
+ [ "$_last" = "$_d" ] && continue
+ echo "$_d"
_last="$_d"
done
}
add_firmware () {
- local _f="${1##*/}" _pkgname
+ local _f="${1##*/}" _m="${2:-Install}" _pkgname
FWPKGTMP="$( tmpdir "${DESTDIR}/var/db/pkg/.firmware" )"
- local flags=-VM
- "$VERBOSE" && flags=-vm
- ftp -N "${0##/}" -D "Install" "$flags" -o- "file:${1}" |
+ local _flags=-vm
+ case "$VERBOSE" in
+ 0|1) _flags=-VM ;;
+ 2|3) _flags=-Vm ;;
+ esac
+
+ ftp -N "${0##/}" -D "$_m" "$_flags" -o- "file:${1}" |
tar -s ",^\+,${FWPKGTMP}/+," \
-s ",^firmware,${DESTDIR}/etc/firmware," \
-C / -zxphf - "+*" "firmware/*"
@@ -232,7 +239,7 @@ delete_firmware() {
local _cwd _pkg="$1" _pkgdir="${DESTDIR}/var/db/pkg"
# TODO: Check hash for files before deleting
- "$VERBOSE" && echo "Uninstalling $_pkg"
+ [ "$VERBOSE" -gt 2 ] && echo -n "Uninstall $_pkg ..."
_cwd="${_pkgdir}/$_pkg"
if [ ! -e "$_cwd/+CONTENTS" ] ||
@@ -267,6 +274,10 @@ delete_firmware() {
rm -f "$_r"
fi
done
+
+ [ "$VERBOSE" -gt 2 ] && echo " done."
+
+ return 0
}
usage() {
@@ -284,7 +295,7 @@ do
D) OPT_D=true ;;
n) DRYRUN=true ;;
p) LOCALSRC="$OPTARG" ;;
- v) VERBOSE=true ;;
+ v) VERBOSE=$(( VERBOSE + 1 )) ;;
:)
echo "${0##*/}: option requires an argument -- -$OPTARG" >&2
usage 2
@@ -327,6 +338,9 @@ set -sA devices -- "$@"
if "$DELETE"; then
[ "$OPT_D" ] && usage 22
+ # Show the "Uninstalling" message when just deleting not upgrading
+ [ "$VERBOSE" -gt 1 ] && VEROBOSE=3
+
set -A installed
if [ "${devices[*]:-}" ]; then
"$ALL" && usage 22
@@ -354,7 +368,7
Re: new site.8: document site*.tgz and /{upgrade,install}.site
I like it, some comments in-line but overall I think this would have
helped me get started with siteXX stuff, so OK afresh1@
On Fri, Nov 05, 2021 at 03:19:03PM +, Klemens Nanni wrote:
> On Wed, Oct 27, 2021 at 07:35:28PM -0500, Aaron Poffenberger wrote:
> > Looks good. Nice to see this moving forward. Thanks.
> >
> > On 2021-10-27 21:13 +, Klemens Nanni wrote:
> > > On Mon, Sep 06, 2021 at 02:29:50PM -0500, Aaron Poffenberger wrote:
> > > > Ping.
> > > >
> > > > Will someone commit this? Seems like no one knows about /upgrade.site
> > > > and it
> > > > fits well with sysupgrade(8).
> > >
> > > sysupgrade(8) is one potential /upgrade.site user but I disagree about
> > > documenting the latter through the former.
> > >
> > > Here is a new manual roughly merging the FAQ bits with what you, Aaron,
> > > provided.
> > >
> > > site(8) is a lame but fitting name; autoinstall(8) and sysypgrade(8)
> > > reference it and both sets and scripts are documented.
> > >
> > > Examples are intentionally brief to be shorter and more concise than the
> > > FAQ while showing enough to get going.
> > >
> > >
> > > I am quite certain that wording, examples and the references from other
> > > manuals can be tweaked, but this diff looks like a good enough start
> > > and --if this is the way to go-- I'd prefer to commit and keep polishing
> > > in-tree.
> > >
> > > Feedback? Objections? OK?
>
> I have not received any feedback on this except from Aaron.
>
> Anyone? Anything?
>
> Index: distrib/sets/lists/man/mi
> ===
> RCS file: /cvs/src/distrib/sets/lists/man/mi,v
> retrieving revision 1.1645
> diff -u -p -r1.1645 mi
> --- distrib/sets/lists/man/mi 2 Nov 2021 22:07:33 - 1.1645
> +++ distrib/sets/lists/man/mi 3 Nov 2021 02:26:16 -
> @@ -2561,6 +2561,7 @@
> ./usr/share/man/man8/security.8
> ./usr/share/man/man8/sendmail.8
> ./usr/share/man/man8/sensorsd.8
> +./usr/share/man/man8/site.8
> ./usr/share/man/man8/sftp-server.8
> ./usr/share/man/man8/showmount.8
> ./usr/share/man/man8/shutdown.8
> Index: usr.sbin/sysupgrade/sysupgrade.8
> ===
> RCS file: /cvs/src/usr.sbin/sysupgrade/sysupgrade.8,v
> retrieving revision 1.10
> diff -u -p -r1.10 sysupgrade.8
> --- usr.sbin/sysupgrade/sysupgrade.8 3 Oct 2019 12:43:58 - 1.10
> +++ usr.sbin/sysupgrade/sysupgrade.8 27 Oct 2021 20:42:26 -
> @@ -67,6 +67,10 @@ This is the default if the system is cur
> Upgrade to a snapshot.
> This is the default if the system is currently running a snapshot.
> .El
> +.Pp
> +See
> +.Xr site 8
> +for how to customize the upgrade process.
> .Sh FILES
> .Bl -tag -width "/auto_upgrade.conf" -compact
> .It Pa /auto_upgrade.conf
> @@ -83,7 +87,8 @@ Directory the upgrade is downloaded to.
> .Xr signify 1 ,
> .Xr installurl 5 ,
> .Xr autoinstall 8 ,
> -.Xr release 8
> +.Xr release 8 ,
> +.Xr site 8
> .Sh HISTORY
> .Nm
> first appeared in
> Index: share/man/man8/Makefile
> ===
> RCS file: /cvs/src/share/man/man8/Makefile,v
> retrieving revision 1.102
> diff -u -p -r1.102 Makefile
> --- share/man/man8/Makefile 1 May 2021 16:11:10 - 1.102
> +++ share/man/man8/Makefile 27 Oct 2021 20:38:11 -
> @@ -6,7 +6,7 @@ MAN= afterboot.8 autoinstall.8 boot_conf
> crash.8 daily.8 \
> diskless.8 genassym.sh.8 intro.8 netstart.8 rc.8 \
> rc.conf.8 rc.d.8 rc.shutdown.8 rc.subr.8 release.8 \
> - security.8 ssl.8 starttls.8 sticky.8 yp.8
> + security.8 site.8 ssl.8 starttls.8 sticky.8 yp.8
>
> SUBDIR= man8.alpha man8.amd64 man8.arm64 man8.armv7 \
> man8.hppa man8.i386 man8.landisk \
> Index: share/man/man8/autoinstall.8
> ===
> RCS file: /cvs/src/share/man/man8/autoinstall.8,v
> retrieving revision 1.23
> diff -u -p -r1.23 autoinstall.8
> --- share/man/man8/autoinstall.8 18 Jul 2021 11:08:34 - 1.23
> +++ share/man/man8/autoinstall.8 27 Oct 2021 20:42:24 -
> @@ -32,6 +32,10 @@ file and HTTP to fetch the file.
> If that fails, the installer asks for the location which can either be
> a URL or a local path.
> .Pp
> +See
> +.Xr site 8
> +for how to provide custom configuration.
> +.Pp
> To start unattended installation or upgrade choose '(A)utoinstall' at the
> install prompt.
> If there is only one network interface, the installer fetches the response
> @@ -235,7 +239,8 @@ host foo {
> .Sh SEE ALSO
> .Xr dhcp-options 5 ,
> .Xr dhcpd.conf 5 ,
> -.Xr diskless 8
> +.Xr diskless 8 ,
> +.Xr site 8
> .Sh HISTORY
> The
> .Nm
> Index: share/man/man8/site.8
> ===
> RCS file: share/man/man8/site.8
> diff -N share/man/man8/site.8
> --- /dev/null 1 Jan 1970 00:00:00 -
> +++ share/man/man8/site.8 27 Oct 2021 21:11:48
Re: Fix vi(1) recovery - new method
In my quick test, this works a lot better than what we have now. At
least I get back more of the file I was working on. I also haven't been
able to reproduce the annoying segfault and half the original file
disappearing, although that was infrequent before so could just be luck.
User experience with this patch is improved in my opinion, I'd like to
see it go in, so OK afresh1@
On Sat, Oct 09, 2021 at 08:26:13PM -0400, trondd wrote:
> This is a new attempt at fixing vi(1) recovery by actually writing
> to the recovery file. Previously I restored the SIGALRM method that
> was deleted in the 90's but wondered if that was still the best way
> to handle this. Checking and syncing to the recovery every 2 minutes
> seems arbitrary and overly cautious.
>
> This attempt takes it to the other direction. I'm writing each
> change to the recovery file immediately after the in-memory database
> is modified. Though, I can see that this might have a noticeable
> impact on slower file systems.
>
> VIM takes a sort of hybrid approach and writes to the backup every
> 200 characters or after 4 seconds of idle time. Perhaps this is the
> best method to not get too far behind while also not hammering the
> filesystem with quick edits.
>
> For now, I'm sticking to the naive approach for review. The diff is
> smaller than using SIGALRM and more straight forward and I'd like to
> hear what method might make sense to improve the process. This code
> would probably be the basis for other improvements.
>
> Below is my original explanation of the problem with vi(1)'s
> recovery.
>
> This is a reference to the older SIGALRM diff (I have a version
> updated to use the atomic signal flags if we want to look at the
> SIGALRM process instead).
> https://marc.info/?l=openbsd-tech=162940011614049
>
> Tim.
>
> -
>
> While investigating an occasional crash when recovering a file with 'vi -r'
> after a power failure, I noticed that the recovery files are actually never
> updated during an editing session. The recovery files are created upon
> initial modification of the file which saves the state of the file at the
> time of the edit. You can work on a file for as long as you want and even
> write it to disk but the recovery file is never updated. If the session is
> then lost due to power failure or a SIGKILL and you attempt to recover with
> -r, you'll be presented with the contents of the file from that first edit.
> It won't contain unsaved changes nor even any changes manually written to
> disk to the original file. Accepting the recovered version would lose all
> of your work.
>
> Reading the vi docs, man page, and source comments in the OpenBSD tree, they
> all mention the use of SIGALRM to periodically save changes to the recovery
> file. However, the code never sets up a handler or captures SIGALRM. It
> only ever updates the recovery file on a SIGTERM but then also exits, I
> guess to cover the case of an inadvertent clean system shutdown.
>
> I dug through an nvi source repository[0] that seemed to cover it's entire
> history and it seems this functionality was lost somewhere around 1994 and I
> don't see it having been replaced by anything else. Our version seems to be
> from 1996 and editors/nvi in ports still lacks code to update the recovery
> file, as well.
>
> What I've done is re-implement periodic updates to the recovery file using
> SIGALRM and a timer like the original implementation but rewritten a bit to
> fit the newer source file layout and event handling. That keeps the recovery
> updated every 2 minutes. Then it seemed silly to be able to write changes to
> the original file and if a crash happens before the next SIGALRM, recovery
> would try to roll you back to before those saved changes. So I've also added
> a call to sync the recovery file if you explicitly write changes to disk. I
> don't think the recovery system should try to punish you for actively saving
> your work even if it is only at most 2 minutes worth.
>
> Comments or feedback? I'm unsure I've covered all caveats with this code or
> if there are vi/ex usecases where it won't work correctly. For testing, I've
> covered my usage and several scenarios I could contrive but I don't regularly
> use ex, for example, or change many options from the default. I've been
> running with this code for a week. And I suppose there must be a reason no
> one has noticed or cared about this for over 20 years. Everyone else uses
> vim, I guess?
>
> Tim.
>
> [0] https://repo.or.cz/nvi.git
>
> -
>
>
> Index: common/exf.h
> ===
> RCS file: /cvs/src/usr.bin/vi/common/exf.h,v
> retrieving revision 1.5
> diff -u -p -r1.5 exf.h
> --- common/exf.h 24 Apr 2015 21:48:31 - 1.5
> +++ common/exf.h 9 Oct 2021 22:40:17 -
> @@ -58,7 +58,8 @@ struct _exf {
> #define F_RCV_NORM 0x020 /* Don't delete recovery files.
> */
> #define
Re: Add versioned lib to system perl's @INC for non-packaged modules
Below is the corrected patch for this, but it's not quite ready yet.
See the email over on ports@ with some adjustments to perl.port.mk and
other patches to fix some ports fallout that sthen@ was super helpful in
tracking down.
The main issue was that somehow I lost a quote in the patch and so
it didn't work right. But, this should apply and let you play with it
if you like.
Once we have a path forward with the ports fallout, I'll be looking for
more comments and maybe OKs.
On Sun, May 16, 2021 at 03:30:39PM -0700, Andrew Hewus Fresh wrote:
> This patch is should make it easier to recover from problems that happen
> when installing perl modules outside of the package system. You might
> recognize the "loadable library and perl binaries are mismatched" error.
>
> Adding a versioned subdirectory for cpan modules means that when the
> perl version changes it will no longer find those old modules and it
> will be easy to remove them without disturbing things from packages.
>
> Only slightly related, but we can avoid breaking base tools like pkg_add
> and friend by using "no lib '/usr/local/libdata/perl5/site_perl';" to
> avoid looking in those directories at all.
>
>
> The new @INC looks like this. The first "5.32.1" sitelib is where
> external things, like cpan(1), will install. The current "site_perl"
> directory (now vendorlib) will continue to be where ports/packages go.
> Finally, "/usr/libdata" privlib is for things that ship with the base
> system.
>
> $ perl -V | perl -ne 'print if /INC/..eof'
> @INC:
> /usr/local/libdata/perl5/site_perl/5.32.1/sparc64-openbsd
> /usr/local/libdata/perl5/site_perl/5.32.1
> /usr/local/libdata/perl5/site_perl/sparc64-openbsd
> /usr/local/libdata/perl5/site_perl
> /usr/libdata/perl5/sparc64-openbsd
> /usr/libdata/perl5
>
>
> One place this might cause temporary issues in the ports tree is
> anything that doesn't use the overrides in perl.port.mk and looks up
> sitelib itself. It should be fairly obvious as it should fail to
> package because the files in the PLIST will not exist (they'll be in the
> versioned subdir). I didn't run into that in my testing, but I didn't
> complete a full bulk build.
>
> There do appear to be some annoyances with still shared directories for
> man pages, in that if you install a CPAN module and then attempt to
> pkg_add it, it complains because "files already exist" in the man
> directory. We could separate the cpan man pages, or I think un-setting
> the site_lib man*dir will mean cpan won't instal them. I haven't yet
> tested that, but I'm not sure what would be best there.
>
>
> Unfortunately I did do a bunch of testing of different ways to add
> things to @INC which meant this patch also does some reorganization of
> the contents, I can turn it into a smaller patch if requested and do the
> cleanup separately.
>
>
> Comments, OK? Should I see if sthen@ will do a full bulk build first?
Index: gnu/usr.bin/perl/config.over
===
RCS file: /cvs/src/gnu/usr.bin/perl/config.over,v
retrieving revision 1.22
diff -u -p -r1.22 config.over
--- gnu/usr.bin/perl/config.over5 Feb 2017 00:33:38 - 1.22
+++ gnu/usr.bin/perl/config.over31 Jul 2021 00:02:14 -
@@ -9,49 +9,85 @@ archname="`arch -s`-${osname}"
myarchname="$archname"
# Use correct paths for a distribution
-prefix='/usr'
+# site is where cpan clients install files
+# vendor is where the ports tree puts packages
+# priv is where the system installs files
+
+# We set up vendor paths for ports/packages
+usevendorprefix="${define}"
+d_vendorbin="${define}"
+d_vendorlib="${define}"
+d_vendorarch="${define}"
+#d_vendorscript="${define}"
+
+# Things from base go into /usr
prefixexp='/usr'
+prefix="${prefixexp}"
-# But site binaries go in /usr/local/bin for ports
-siteprefix='/usr/local'
+# But vendor and site files go in /usr/local for ports and cpan
siteprefixexp='/usr/local'
-installsitebin='/usr/local/bin'
+siteprefix="${siteprefixexp}"
+installsitebin='${siteprefix}/bin'
+
+vendorprefixexp='/usr/local'
+vendorprefix="${vendorprefixexp}"
+installvendorbin='${vendorprefix}/bin'
-installarchlib="/usr/libdata/perl5/${archname}"
+installprivlib="${prefix}/libdata/perl5"
+privlib="${installprivlib}"
+privlibexp="${privlib}"
+installarchlib="${installprivlib}/${archname}"
archlib="${installarchlib}"
archlibexp="${archlib}"
test $useshrplib = "true" && ccdlflags="-Wl,-R${installarchlib}/CORE"
-installprivlib="/usr/
Add versioned lib to system perl's @INC for non-packaged modules
This patch is should make it easier to recover from problems that happen
when installing perl modules outside of the package system. You might
recognize the "loadable library and perl binaries are mismatched" error.
Adding a versioned subdirectory for cpan modules means that when the
perl version changes it will no longer find those old modules and it
will be easy to remove them without disturbing things from packages.
Only slightly related, but we can avoid breaking base tools like pkg_add
and friend by using "no lib '/usr/local/libdata/perl5/site_perl';" to
avoid looking in those directories at all.
The new @INC looks like this. The first "5.32.1" sitelib is where
external things, like cpan(1), will install. The current "site_perl"
directory (now vendorlib) will continue to be where ports/packages go.
Finally, "/usr/libdata" privlib is for things that ship with the base
system.
$ perl -V | perl -ne 'print if /INC/..eof'
@INC:
/usr/local/libdata/perl5/site_perl/5.32.1/sparc64-openbsd
/usr/local/libdata/perl5/site_perl/5.32.1
/usr/local/libdata/perl5/site_perl/sparc64-openbsd
/usr/local/libdata/perl5/site_perl
/usr/libdata/perl5/sparc64-openbsd
/usr/libdata/perl5
One place this might cause temporary issues in the ports tree is
anything that doesn't use the overrides in perl.port.mk and looks up
sitelib itself. It should be fairly obvious as it should fail to
package because the files in the PLIST will not exist (they'll be in the
versioned subdir). I didn't run into that in my testing, but I didn't
complete a full bulk build.
There do appear to be some annoyances with still shared directories for
man pages, in that if you install a CPAN module and then attempt to
pkg_add it, it complains because "files already exist" in the man
directory. We could separate the cpan man pages, or I think un-setting
the site_lib man*dir will mean cpan won't instal them. I haven't yet
tested that, but I'm not sure what would be best there.
Unfortunately I did do a bunch of testing of different ways to add
things to @INC which meant this patch also does some reorganization of
the contents, I can turn it into a smaller patch if requested and do the
cleanup separately.
Comments, OK? Should I see if sthen@ will do a full bulk build first?
Index: gnu/usr.bin/perl/config.over
===
RCS file: /cvs/src/gnu/usr.bin/perl/config.over,v
retrieving revision 1.22
diff -u -p -r1.22 config.over
--- gnu/usr.bin/perl/config.over5 Feb 2017 00:33:38 - 1.22
+++ gnu/usr.bin/perl/config.over16 May 2021 20:43:16 -
@@ -9,49 +9,85 @@ archname="`arch -s`-${osname}"
myarchname="$archname"
# Use correct paths for a distribution
-prefix='/usr'
+# site is where cpan clients install files
+# vendor is where the ports tree puts packages
+# priv is where the system installs files
+
+# We set up vendor paths for ports/packages
+usevendorprefix="${define}"
+d_vendorbin="${define}"
+d_vendorlib="${define}"
+d_vendorarch="${define}"
+#d_vendorscript="${define}"
+
+# Things from base go into /usr
prefixexp='/usr'
+prefix="${prefixexp}"
-# But site binaries go in /usr/local/bin for ports
-siteprefix='/usr/local'
+# But vendor and site files go in /usr/local for ports and cpan
siteprefixexp='/usr/local'
-installsitebin='/usr/local/bin'
+siteprefix="${siteprefixexp}"
+installsitebin='${siteprefix}/bin'
+
+vendorprefixexp='/usr/local'
+vendorprefix="${vendorprefixexp}"
+installvendorbin='${vendorprefix}/bin'
-installarchlib="/usr/libdata/perl5/${archname}"
+installprivlib="${prefix}/libdata/perl5"
+privlib="${installprivlib}"
+privlibexp="${privlib}"
+installarchlib="${installprivlib}/${archname}"
archlib="${installarchlib}"
archlibexp="${archlib}"
test $useshrplib = "true" && ccdlflags="-Wl,-R${installarchlib}/CORE"
-installprivlib="/usr/libdata/perl5"
-privlib="${installprivlib}"
-privlibexp="${privlib}"
-
-installsitearch="/usr/local/libdata/perl5/site_perl/${archname}"
-sitearch="${installsitearch}"
-sitearchexp="${sitearch}"
-installsitelib="/usr/local/libdata/perl5/site_perl"
+# We keep vendor in site_perl so we don't have to bump every perl port
+# otherwise this would be pkg_perl
+# Not versioned here because pkg tools handle that for us.
+installvendorlib="${vendorprefix}/libdata/perl5/site_perl"
+vendorlib="${installvendorlib}"
+vendorlibexp="${vendorlib}"
+installvendorarch="${installvendorlib}/${archname}"
+vendorarch="${installvendorarch}"
+vendorarchexp="${vendorarch}"
+
+# cpan(1) and similar tools install in a $version dir that's easy
+# to remove after upgrade and won't cause issues for packages when
+# perl versions change.
+installsitelib="${siteprefix}/libdata/perl5/site_perl/${version}"
sitelib="${installsitelib}"
sitelibexp="${sitelib}"
+installsitearch="${installsitelib}/${archname}"
+sitearch="${installsitearch}"
+sitearchexp="${sitearch}"
installstyle="${privlib}"
-# We
Update en_US.UTF-8.src to Unicode Version 13.0.0
With the update to perl v5.32, we get Unicode 13 in base, which means we can update the table. This is the mechanical change from running the gen_ctype_utf8.l script. https://www.unicode.org/versions/Unicode13.0.0 Comments, OK? Index: share/locale/ctype/gen_ctype_utf8.pl === RCS file: /cvs/src/share/locale/ctype/gen_ctype_utf8.pl,v retrieving revision 1.5 diff -u -p -r1.5 gen_ctype_utf8.pl --- share/locale/ctype/gen_ctype_utf8.pl20 Feb 2020 02:51:56 - 1.5 +++ share/locale/ctype/gen_ctype_utf8.pl3 Mar 2021 00:34:36 - @@ -53,7 +53,7 @@ print <<'EOL'; /* * COPYRIGHT AND PERMISSION NOTICE * - * Copyright (c) 1991-2019 Unicode, Inc. All rights reserved. + * Copyright (c) 1991-2020 Unicode, Inc. All rights reserved. * Distributed under the Terms of Use in * https://www.unicode.org/copyright.html. * Index: share/locale/ctype/en_US.UTF-8.src === RCS file: /cvs/src/share/locale/ctype/en_US.UTF-8.src,v retrieving revision 1.10 diff -u -p -r1.10 en_US.UTF-8.src --- share/locale/ctype/en_US.UTF-8.src 20 Feb 2020 02:53:53 - 1.10 +++ share/locale/ctype/en_US.UTF-8.src 3 Mar 2021 00:34:36 - @@ -3,7 +3,7 @@ /* * COPYRIGHT AND PERMISSION NOTICE * - * Copyright (c) 1991-2019 Unicode, Inc. All rights reserved. + * Copyright (c) 1991-2020 Unicode, Inc. All rights reserved. * Distributed under the Terms of Use in * https://www.unicode.org/copyright.html. * @@ -40,7 +40,7 @@ ENCODING"UTF8" VARIABLECODESET=UTF-8 -/* Unicode Version 12.1.0 */ +/* Unicode Version 13.0.0 */ /* * U+ - U+007F : Basic Latin @@ -1013,13 +1013,13 @@ SWIDTH1 0x0860 - 0x086a * U+08A0 - U+08FF : Arabic Extended-A */ -ALPHA 0x08a0 - 0x08b4 0x08b6 - 0x08bd 0x08d4 - 0x08df 0x08e3 - 0x08e9 +ALPHA 0x08a0 - 0x08b4 0x08b6 - 0x08c7 0x08d4 - 0x08df 0x08e3 - 0x08e9 ALPHA 0x08f0 - 0x08ff -GRAPH 0x08a0 - 0x08b4 0x08b6 - 0x08bd 0x08d3 - 0x08ff -PRINT 0x08a0 - 0x08b4 0x08b6 - 0x08bd 0x08d3 - 0x08ff +GRAPH 0x08a0 - 0x08b4 0x08b6 - 0x08c7 0x08d3 - 0x08ff +PRINT 0x08a0 - 0x08b4 0x08b6 - 0x08c7 0x08d3 - 0x08ff SPECIAL 0x08d3 0x08e0 - 0x08e2 0x08ea - 0x08ef SWIDTH0 0x08d3 - 0x08ff -SWIDTH1 0x08a0 - 0x08b4 0x08b6 - 0x08bd +SWIDTH1 0x08a0 - 0x08b4 0x08b6 - 0x08c7 /* @@ -1135,15 +1135,15 @@ ALPHA 0x0b5f - 0x0b63 0x0b71 DIGIT 0x0b66 - 0x0b6f GRAPH 0x0b01 - 0x0b03 0x0b05 - 0x0b0c 0x0b0f - 0x0b10 0x0b13 - 0x0b28 GRAPH 0x0b2a - 0x0b30 0x0b32 - 0x0b33 0x0b35 - 0x0b39 0x0b3c - 0x0b44 -GRAPH 0x0b47 - 0x0b48 0x0b4b - 0x0b4d 0x0b56 - 0x0b57 0x0b5c - 0x0b5d +GRAPH 0x0b47 - 0x0b48 0x0b4b - 0x0b4d 0x0b55 - 0x0b57 0x0b5c - 0x0b5d GRAPH 0x0b5f - 0x0b63 0x0b66 - 0x0b77 PUNCT 0x0b70 PRINT 0x0b01 - 0x0b03 0x0b05 - 0x0b0c 0x0b0f - 0x0b10 0x0b13 - 0x0b28 PRINT 0x0b2a - 0x0b30 0x0b32 - 0x0b33 0x0b35 - 0x0b39 0x0b3c - 0x0b44 -PRINT 0x0b47 - 0x0b48 0x0b4b - 0x0b4d 0x0b56 - 0x0b57 0x0b5c - 0x0b5d +PRINT 0x0b47 - 0x0b48 0x0b4b - 0x0b4d 0x0b55 - 0x0b57 0x0b5c - 0x0b5d PRINT 0x0b5f - 0x0b63 0x0b66 - 0x0b77 -SPECIAL 0x0b3c 0x0b4d 0x0b72 - 0x0b77 -SWIDTH0 0x0b01 0x0b3c 0x0b3f 0x0b41 - 0x0b44 0x0b4d 0x0b56 +SPECIAL 0x0b3c 0x0b4d 0x0b55 0x0b72 - 0x0b77 +SWIDTH0 0x0b01 0x0b3c 0x0b3f 0x0b41 - 0x0b44 0x0b4d 0x0b55 - 0x0b56 SWIDTH0 0x0b62 - 0x0b63 SWIDTH1 0x0b02 - 0x0b03 0x0b05 - 0x0b0c 0x0b0f - 0x0b10 0x0b13 - 0x0b28 SWIDTH1 0x0b2a - 0x0b30 0x0b32 - 0x0b33 0x0b35 - 0x0b39 0x0b3d - 0x0b3e @@ -1238,21 +1238,21 @@ TODIGIT < 0x0ce6 - 0x0cef : 0x > * U+0D00 - U+0D7F : Malayalam */ -ALPHA 0x0d00 - 0x0d03 0x0d05 - 0x0d0c 0x0d0e - 0x0d10 0x0d12 - 0x0d3a -ALPHA 0x0d3d - 0x0d44 0x0d46 - 0x0d48 0x0d4a - 0x0d4c 0x0d4e -ALPHA 0x0d54 - 0x0d57 0x0d5f - 0x0d63 0x0d7a - 0x0d7f +ALPHA 0x0d00 - 0x0d0c 0x0d0e - 0x0d10 0x0d12 - 0x0d3a 0x0d3d - 0x0d44 +ALPHA 0x0d46 - 0x0d48 0x0d4a - 0x0d4c 0x0d4e 0x0d54 - 0x0d57 +ALPHA 0x0d5f - 0x0d63 0x0d7a - 0x0d7f DIGIT 0x0d66 - 0x0d6f -GRAPH 0x0d00 - 0x0d03 0x0d05 - 0x0d0c 0x0d0e - 0x0d10 0x0d12 - 0x0d44 -GRAPH 0x0d46 - 0x0d48 0x0d4a - 0x0d4f 0x0d54 - 0x0d63 0x0d66 - 0x0d7f +GRAPH 0x0d00 - 0x0d0c 0x0d0e - 0x0d10 0x0d12 - 0x0d44 0x0d46 - 0x0d48 +GRAPH 0x0d4a - 0x0d4f 0x0d54 - 0x0d63 0x0d66 - 0x0d7f PUNCT 0x0d4f 0x0d79 -PRINT 0x0d00 - 0x0d03 0x0d05 - 0x0d0c 0x0d0e - 0x0d10 0x0d12 - 0x0d44 -PRINT 0x0d46 - 0x0d48 0x0d4a - 0x0d4f 0x0d54 - 0x0d63 0x0d66 - 0x0d7f +PRINT 0x0d00 - 0x0d0c 0x0d0e - 0x0d10 0x0d12 - 0x0d44 0x0d46 - 0x0d48 +PRINT 0x0d4a - 0x0d4f 0x0d54 - 0x0d63 0x0d66 - 0x0d7f SPECIAL 0x0d3b - 0x0d3c 0x0d4d 0x0d58 - 0x0d5e 0x0d70 - 0x0d78 SWIDTH0 0x0d00 - 0x0d01 0x0d3b - 0x0d3c 0x0d41 - 0x0d44 0x0d4d SWIDTH0 0x0d62 - 0x0d63 -SWIDTH1 0x0d02 -
install smtpd-filters(7)?
It would be nice to have this reference installed, would this be the right place to do that? (ok, mostly I want to link https://man.openbsd.org/smtpd-filter.7 from https://github.com/afresh1/OpenSMTPd-Filter/) Comments, OK? Index: usr.sbin/smtpd/smtpd/Makefile === RCS file: /cvs/src/usr.sbin/smtpd/smtpd/Makefile,v retrieving revision 1.107 diff -u -p -r1.107 Makefile --- usr.sbin/smtpd/smtpd/Makefile 31 Jan 2020 22:01:20 - 1.107 +++ usr.sbin/smtpd/smtpd/Makefile 28 Feb 2021 02:18:32 - @@ -79,7 +79,7 @@ SRCS+=scheduler_proc.c SRCS+= stat_ramstat.c -MAN= sendmail.8 smtpd.8 smtpd.conf.5 table.5 +MAN= sendmail.8 smtpd.8 smtpd.conf.5 smtpd-filters.7 table.5 BINDIR=/usr/sbin LDADD+=-levent -lutil -lssl -lcrypto -lm -lz
Re: OpenBSD perl 5.32.1 - Call for Testing
On Sat, Feb 20, 2021 at 02:11:45PM -0800, Andrew Hewus Fresh wrote: > I've probably missed making it in for 6.9, but it is again time for > testing a perl update so it can become /usr/bin/perl It was pointed out that there is still time to get this in for 6.9, and sthen@ says it looks OK in a bulk ports build, so I'll be working on preparing the import this weekend. Please let me know of any reasons to hold off. > > Several good changes this time: > https://metacpan.org/pod/distribution/perl/pod/perl5320delta.pod > https://metacpan.org/pod/release/SHAY/perl-5.32.1/pod/perldelta.pod > > * A new "isa" operator > * A new feature "no indirect" that disables indirect object notation > * Chained comparisons ( $x < $y <= $z ) > * Unicode 13.0 > * IO::Compress now comes with the streamzip utility. > > Plus a bunch of other unicode and regex things as well as new versions > of many modules. > > It also includes the recent PERL_HASH fix for 64bit strict alignment > architectures and removes several local patches that were merged > upstream. > > It would be nice to get feedback on some of the architectures I don't > have, and a bulk ports build to see what sort of fallout happens there. > > The logs for what I do have are here: > https://github.com/afresh1/OpenBSD-perl/tree/blead/build_logs/perl-5.32.1 > > Which are alpha, amd64, arm64, armv7, macppc, octeon, and sparc64. > > The issues I had in 5.30 with capturing output from forked processes on my > alpha are still there, could just be that it's slow. > > For running the perl test suite, you can follow the instructions on > github, repeated here: > > https://github.com/afresh1/OpenBSD-perl > > download the patches and scripts > https://github.com/afresh1/OpenBSD-perl/archive/blead.tar.gz > and extract someplace > or git clone https://github.com/afresh1/OpenBSD-perl.git > download perl-5.30.0.tar.gz into the same directory > https://cpan.metacpan.org/authors/id/S/SH/SHAY/perl-5.32.1.tar.gz > cd to someplace you have room > run /path/to/OpenBSD-perl/build_perl > wait > send me the log file(s) it generates > > > You can download a pre-patched version of perl that can be extracted to > replace > src/gnu/usr.bin/perl for building a system with the new perl: > https://cvs.afresh1.com/~andrew/perl-update/OpenBSD-perl-5.32.1.tar.gz > > I also have what should be a mostly correct sets/lists patch for building a > release. > https://cvs.afresh1.com/~andrew/perl-update/OpenBSD-perl-5.32.1-sets.patch > > There are also several files that are no longer part of the perl distribution: > > rm -r /usr/bin/podselect \ > /usr/lib/libperl.so.20.0 \ > /usr/lib/libperl.so.21.0 \ > /usr/libdata/perl5/*/CORE/dquote_inline.h \ > /usr/libdata/perl5/*/Tie \ > /usr/libdata/perl5/*/auto/Tie \ > /usr/libdata/perl5/Pod/Find.pm \ > /usr/libdata/perl5/Pod/InputObjects.pm \ > /usr/libdata/perl5/Pod/ParseUtils.pm \ > /usr/libdata/perl5/Pod/Parser.pm \ > /usr/libdata/perl5/Pod/PlainText.pm \ > /usr/libdata/perl5/Pod/Select.pm \ > /usr/libdata/perl5/pod/perlce.pod \ > /usr/libdata/perl5/unicore/Heavy.pl \ > /usr/libdata/perl5/unicore/lib/Lb/EB.pl \ > /usr/libdata/perl5/unicore/lib/Perl/_PerlNon.pl \ > /usr/libdata/perl5/unicore/lib/Sc/Armn.pl \ > /usr/libdata/perl5/utf8_heavy.pl \ > /usr/share/man/man1/podselect.1 \ > /usr/share/man/man3p/Pod::Find.3p \ > /usr/share/man/man3p/Pod::InputObjects.3p \ > /usr/share/man/man3p/Pod::ParseUtils.3p \ > /usr/share/man/man3p/Pod::Parser.3p \ > /usr/share/man/man3p/Pod::PlainText.3p \ > /usr/share/man/man3p/Pod::Select.3p > > > There's also patch file that should apply to a -current tree updating to > perl 5.32.1. You'll still need the sets lists patch if you're planning > to build a release. > > https://cvs.afresh1.com/~andrew/perl-update/OpenBSD-perl-5.32.1.patch > > Unfortunately the patch doesn't actually apply due to non-ascii files in > the diff, not quite sure the magic incantation to make that work, so I > recommend replacing src/gnu/usr.bin/perl with the extracted tar.gz > linked above. > > If it did apply, you would copy the patch to /usr/src/perl-5.32.1.patch > (or adjust the paths below) > > cd /usr/src/ > patch -p0 -uNE < perl-5.32.1.patch > > # Remove patch cruft > find gnu/usr.bin/perl -name '*.orig' -delete > > # Add and remove binary and zero sized files that patch doesn't understand > grep -B1 -e '^Index:' -e 'Binary files /tmp/.* and /dev
OpenBSD perl 5.32.1 - Call for Testing
I've probably missed making it in for 6.9, but it is again time for
testing a perl update so it can become /usr/bin/perl
Several good changes this time:
https://metacpan.org/pod/distribution/perl/pod/perl5320delta.pod
https://metacpan.org/pod/release/SHAY/perl-5.32.1/pod/perldelta.pod
* A new "isa" operator
* A new feature "no indirect" that disables indirect object notation
* Chained comparisons ( $x < $y <= $z )
* Unicode 13.0
* IO::Compress now comes with the streamzip utility.
Plus a bunch of other unicode and regex things as well as new versions
of many modules.
It also includes the recent PERL_HASH fix for 64bit strict alignment
architectures and removes several local patches that were merged
upstream.
It would be nice to get feedback on some of the architectures I don't
have, and a bulk ports build to see what sort of fallout happens there.
The logs for what I do have are here:
https://github.com/afresh1/OpenBSD-perl/tree/blead/build_logs/perl-5.32.1
Which are alpha, amd64, arm64, armv7, macppc, octeon, and sparc64.
The issues I had in 5.30 with capturing output from forked processes on my
alpha are still there, could just be that it's slow.
For running the perl test suite, you can follow the instructions on
github, repeated here:
https://github.com/afresh1/OpenBSD-perl
download the patches and scripts
https://github.com/afresh1/OpenBSD-perl/archive/blead.tar.gz
and extract someplace
or git clone https://github.com/afresh1/OpenBSD-perl.git
download perl-5.30.0.tar.gz into the same directory
https://cpan.metacpan.org/authors/id/S/SH/SHAY/perl-5.32.1.tar.gz
cd to someplace you have room
run /path/to/OpenBSD-perl/build_perl
wait
send me the log file(s) it generates
You can download a pre-patched version of perl that can be extracted to replace
src/gnu/usr.bin/perl for building a system with the new perl:
https://cvs.afresh1.com/~andrew/perl-update/OpenBSD-perl-5.32.1.tar.gz
I also have what should be a mostly correct sets/lists patch for building a
release.
https://cvs.afresh1.com/~andrew/perl-update/OpenBSD-perl-5.32.1-sets.patch
There are also several files that are no longer part of the perl distribution:
rm -r /usr/bin/podselect \
/usr/lib/libperl.so.20.0 \
/usr/lib/libperl.so.21.0 \
/usr/libdata/perl5/*/CORE/dquote_inline.h \
/usr/libdata/perl5/*/Tie \
/usr/libdata/perl5/*/auto/Tie \
/usr/libdata/perl5/Pod/Find.pm \
/usr/libdata/perl5/Pod/InputObjects.pm \
/usr/libdata/perl5/Pod/ParseUtils.pm \
/usr/libdata/perl5/Pod/Parser.pm \
/usr/libdata/perl5/Pod/PlainText.pm \
/usr/libdata/perl5/Pod/Select.pm \
/usr/libdata/perl5/pod/perlce.pod \
/usr/libdata/perl5/unicore/Heavy.pl \
/usr/libdata/perl5/unicore/lib/Lb/EB.pl \
/usr/libdata/perl5/unicore/lib/Perl/_PerlNon.pl \
/usr/libdata/perl5/unicore/lib/Sc/Armn.pl \
/usr/libdata/perl5/utf8_heavy.pl \
/usr/share/man/man1/podselect.1 \
/usr/share/man/man3p/Pod::Find.3p \
/usr/share/man/man3p/Pod::InputObjects.3p \
/usr/share/man/man3p/Pod::ParseUtils.3p \
/usr/share/man/man3p/Pod::Parser.3p \
/usr/share/man/man3p/Pod::PlainText.3p \
/usr/share/man/man3p/Pod::Select.3p
There's also patch file that should apply to a -current tree updating to
perl 5.32.1. You'll still need the sets lists patch if you're planning
to build a release.
https://cvs.afresh1.com/~andrew/perl-update/OpenBSD-perl-5.32.1.patch
Unfortunately the patch doesn't actually apply due to non-ascii files in
the diff, not quite sure the magic incantation to make that work, so I
recommend replacing src/gnu/usr.bin/perl with the extracted tar.gz
linked above.
If it did apply, you would copy the patch to /usr/src/perl-5.32.1.patch
(or adjust the paths below)
cd /usr/src/
patch -p0 -uNE < perl-5.32.1.patch
# Remove patch cruft
find gnu/usr.bin/perl -name '*.orig' -delete
# Add and remove binary and zero sized files that patch doesn't understand
grep -B1 -e '^Index:' -e 'Binary files /tmp/.* and /dev/null differ' \
perl-5.32.1.patch | sed -ne 's/^diff -N //p' |
while read f; do if [ -e $f ]; then rm $f; else touch $f; fi; done
cd gnu/usr.bin/perl && find -d . \
\( -type d -o -path '*/CVS' -prune \) \
! -name CVS \
-exec test -e {}/CVS \; \
-execdir sh -c 'test $( ls -1 {} | grep -v '^CVS/$' | wc -l ) -eq 0' \; \
-exec rm -r {} \;
Document vi range
It took me far too long to figure out how to do this, and I seem to have found this patch when trying to figure it out again but a year later I've forgotten how I figured it out. I'm sure I messed up the formatting terribly, but being able to remember `:.,'cs/foo/bar/` by looking in the manual would make me happy. I always expect the backtick to do it. Comments, OK? Index: usr.bin/vi/docs/USD.doc/vi.man/vi.1 === RCS file: /cvs/src/usr.bin/vi/docs/USD.doc/vi.man/vi.1,v retrieving revision 1.75 diff -u -p -r1.75 vi.1 --- usr.bin/vi/docs/USD.doc/vi.man/vi.1 12 Feb 2018 01:10:46 - 1.75 +++ usr.bin/vi/docs/USD.doc/vi.man/vi.1 14 May 2019 20:38:22 - @@ -537,6 +537,21 @@ A cursor movement command which indicate of text, the first being the current cursor position. Repeating the command character makes it affect the whole current line. +.It Ar range +A +.Op Ar start,end +range of lines from 'start' to an optional 'end'. +The 'start' and 'end' are either line numbers or +.Ar \&. +for the current line, +.Ar $ +for the end of the file, +.Ar 'c +the line marked by 'c', +.Ar % +all the lines in the file, +.Ar g/pattern/ +all lines that match 'pattern'. .It Ar word A sequence of letters, digits or underscores. .El
Re: Remove some customization from our perl build
I think this patch is now cleaned up enough to look for OKs.
The patch to numeric.c works around an issue with clang and
-Wdeclaration-after-statement that was fixed more correctly upstream,
but pulling in the full fix in seemed like overkill for now.
https://github.com/Perl/perl5/issues/17725
https://github.com/Perl/perl5/pull/17750
This patch does miss `rm gnu/usr.bin/perl/DynaLoader.sed` that we no
longer need, not sure which cvs diff flag I missed on that.
Other than that, as described in my original email:
On Fri, Apr 10, 2020 at 06:17:33PM -0700, Andrew Hewus Fresh wrote:
> Recently it was pointed out that we don't link /usr/lib/libperl.so.* to
> libm the way is expected for code that also links to libperl. That led
> me to go digging again into the customization we have around the perl
> build and getting terribly confused. That did somewhat clear up after
> reading more about bsd.*.mk, but still feel like some of this mess was
> to make the vax work, but I couldn't actually figure it out from the cvs
> logs why it exists.
>
> In any case, this patch does a few things, some of which I can split up
> and put in separately if it comes to that.
>
> * Puts back some of the upstream Makefile.SH that we removed
> * and a little Dynaloader too
>
> * Changes Configure flags to -de instead of -dsE
> * So Configure does the work previously handled by depend.done
>
> * Adjusts the installperl script to put libperl where we want it
>
> * Moves some build flag discovery into hint/openbsd.sh
> (Which I can then push upstream)
> * Figuring out the correct PICFLAG, which means perl will now use the
> same one, not -fpic for things built by Makefile.bsdwapper on archs
> that want it and -fPIC for everything else.
> * Using no-tree-ter on alpha, due to a compiler bug.
>
> * Lets the perl infrastructure build libperl again
> * Notably, this stops creating libperl.a, but I have a patch
> around that puts it back, just not sure if we need it.
> * Which means "we" don't actually build anything anymore, we leave all
> that to the perl upstream Makefile so all the "stuff" to do with
> that can go away.
> * Which means ldd now mentions libm as it should
>
> * Some tidying of the rest of Makefile.bsd-wrapper*
>
> It seems to build fine on my
> alpha, amd64, arm64, armv7, i386, macppc, octeon, and sparc64.
>
>
> The individual changes are committed, in a clean-up-build branch, to the
> GitHub repo where I keep track of them, along with build logs from my
> test machines both with and without this patch:
> https://github.com/afresh1/OpenBSD-perl/tree/clean-up-build
Index: gnu/usr.bin/perl/numeric.c
===
RCS file: /cvs/src/gnu/usr.bin/perl/numeric.c,v
retrieving revision 1.6
diff -u -p -r1.6 numeric.c
--- gnu/usr.bin/perl/numeric.c 9 Apr 2020 01:32:11 - 1.6
+++ gnu/usr.bin/perl/numeric.c 17 May 2020 16:44:29 -
@@ -34,8 +34,8 @@ values, including such things as replace
PERL_STATIC_INLINE NV
S_strtod(pTHX_ const char * const s, char ** e)
{
-DECLARATION_FOR_LC_NUMERIC_MANIPULATION;
NV result;
+DECLARATION_FOR_LC_NUMERIC_MANIPULATION;
STORE_LC_NUMERIC_SET_TO_NEEDED();
Index: distrib/sets/lists/comp/mi
===
RCS file: /cvs/src/distrib/sets/lists/comp/mi,v
retrieving revision 1.1498
diff -u -p -r1.1498 mi
--- distrib/sets/lists/comp/mi 22 Apr 2020 02:04:00 - 1.1498
+++ distrib/sets/lists/comp/mi 17 May 2020 16:44:21 -
@@ -1536,7 +1536,6 @@
./usr/lib/libpanelw_p.a
./usr/lib/libpcap.a
./usr/lib/libpcap_p.a
-./usr/lib/libperl.a
./usr/lib/libpthread.a
./usr/lib/libpthread_p.a
./usr/lib/libradius.a
Index: gnu/usr.bin/perl/DynaLoader.sed
===
RCS file: gnu/usr.bin/perl/DynaLoader.sed
diff -N gnu/usr.bin/perl/DynaLoader.sed
--- gnu/usr.bin/perl/DynaLoader.sed 17 Nov 2014 21:00:55 - 1.4
+++ /dev/null 1 Jan 1970 00:00:00 -
@@ -1,6 +0,0 @@
-s@"dlutils.c"@"ext/DynaLoader/dlutils.c"@
-/^#ifndef RTLD_LAZY/{N;N;c\
-#define VERSION "1.20"\
-#define XS_VERSION "1.20"\
-#define DLOPEN_WONT_DO_RELATIVE_PATHS
-}
Index: gnu/usr.bin/perl/Makefile.SH
===
RCS file: /cvs/src/gnu/usr.bin/perl/Makefile.SH,v
retrieving revision 1.51
diff -u -p -r1.51 Makefile.SH
--- gnu/usr.bin/perl/Makefile.SH25 Apr 2020 21:15:23 - 1.51
+++ gnu/usr.bin/perl/Makefile.SH17 May 2020 16:44:23 -
@@ -822,11 +822,11 @@ CCDLFLAGS = `echo $ccdlflags|sed -e 's@
LIBPERL_NONSHR = libperl_nonshr$(LIB_EXT)
MINIPERL_NONSHR= miniperl_nonshr$(EXE_EXT
Reenable json_pp in perl build
With the advent of more things speaking JSON, I thought perhaps we might
want to re-enable the json_pp script that normally ships with perl, but
OpenBSD doesn't install. It's not terribly useful as it is mostly just
a "pretty printer", although it does also allow converting to
Data::Dumper format if you find a use for that. The name does actually
mean "json pure perl" not "json pretty print", so it's just a happy
coincidence. It can also "unpretty" with "--json_opt pretty=0".
You can read the documentation on the MetaCPAN
https://metacpan.org/pod/json_pp
Comments, OK?
$ echo '{"foo":{"bar":"baz","qux":[1,2,3]}}' | json_pp
{
"foo" : {
"bar" : "baz",
"qux" : [
1,
2,
3
]
}
}
Index: distrib/sets/lists/base/mi
===
RCS file: /cvs/src/distrib/sets/lists/base/mi,v
retrieving revision 1.987
diff -u -p -r1.987 mi
--- distrib/sets/lists/base/mi 25 Apr 2020 14:17:04 - 1.987
+++ distrib/sets/lists/base/mi 10 May 2020 00:19:01 -
@@ -427,6 +427,7 @@
./usr/bin/ipcs
./usr/bin/join
./usr/bin/jot
+./usr/bin/json_pp
./usr/bin/kdump
./usr/bin/keynote
./usr/bin/ktrace
Index: distrib/sets/lists/man/mi
===
RCS file: /cvs/src/distrib/sets/lists/man/mi,v
retrieving revision 1.1581
diff -u -p -r1.1581 mi
--- distrib/sets/lists/man/mi 28 Apr 2020 10:49:37 - 1.1581
+++ distrib/sets/lists/man/mi 10 May 2020 00:19:02 -
@@ -126,6 +126,7 @@
./usr/share/man/man1/ipcs.1
./usr/share/man/man1/join.1
./usr/share/man/man1/jot.1
+./usr/share/man/man1/json_pp.1
./usr/share/man/man1/kdump.1
./usr/share/man/man1/keynote.1
./usr/share/man/man1/kill.1
Index: gnu/usr.bin/perl/Makefile.bsd-wrapper1
===
RCS file: /cvs/src/gnu/usr.bin/perl/Makefile.bsd-wrapper1,v
retrieving revision 1.39
diff -u -p -r1.39 Makefile.bsd-wrapper1
--- gnu/usr.bin/perl/Makefile.bsd-wrapper1 9 Apr 2020 02:54:54 -
1.39
+++ gnu/usr.bin/perl/Makefile.bsd-wrapper1 10 May 2020 00:19:02 -
@@ -64,6 +64,7 @@ _quick3p=
h2xs 1 utils/h2xs \
instmodsh 1utils/instmodsh \
Internals 1lib/Internals.pod \
+json_pp 1 utils/json_pp \
libnetcfg 1utils/libnetcfg \
PACKAGING 1PACKAGING \
perl 1 pod/perl.pod \
Index: gnu/usr.bin/perl/utils.lst
===
RCS file: /cvs/src/gnu/usr.bin/perl/utils.lst,v
retrieving revision 1.24
diff -u -p -r1.24 utils.lst
--- gnu/usr.bin/perl/utils.lst 30 Dec 2019 02:15:16 - 1.24
+++ gnu/usr.bin/perl/utils.lst 10 May 2020 00:19:02 -
@@ -10,6 +10,7 @@ utils/encguess
utils/h2ph
utils/h2xs
utils/instmodsh
+utils/json_pp
utils/libnetcfg
utils/perlbug
utils/perldoc
Index: gnu/usr.bin/perl/t/porting/dual-life.t
===
RCS file: /cvs/src/gnu/usr.bin/perl/t/porting/dual-life.t,v
retrieving revision 1.17
diff -u -p -r1.17 dual-life.t
--- gnu/usr.bin/perl/t/porting/dual-life.t 30 Dec 2019 02:15:18 -
1.17
+++ gnu/usr.bin/perl/t/porting/dual-life.t 10 May 2020 00:19:02 -
@@ -26,8 +26,6 @@ use File::Spec::Functions;
my $not_installed = qr{^(?:
\.\./cpan/Archive-Tar/bin/ptar.*
|
- \.\./cpan/JSON-PP/bin/json_pp
- |
\.\./cpan/IO-Compress/bin/zipdetails
|
\.\./cpan/Encode/bin/u(?:cm(?:2table|lint|sort)|nidump)
Index: gnu/usr.bin/perl/utils/Makefile.PL
===
RCS file: /cvs/src/gnu/usr.bin/perl/utils/Makefile.PL,v
retrieving revision 1.14
diff -u -p -r1.14 Makefile.PL
--- gnu/usr.bin/perl/utils/Makefile.PL 30 Dec 2019 02:15:18 - 1.14
+++ gnu/usr.bin/perl/utils/Makefile.PL 10 May 2020 00:19:02 -
@@ -35,9 +35,9 @@ print $fh <<'EOT';
# Files to be built with variable substitution after miniperl is
# available. Dependencies handled manually below (for now).
-pl = corelist.PL cpan.PL h2ph.PL h2xs.PL instmodsh.PL perlbug.PL perldoc.PL
perlivp.PL pl2pm.PL prove.PL splain.PL libnetcfg.PL piconv.PL enc2xs.PL
encguess.PL xsubpp.PL pod2html.PL
-plextract = corelist cpan h2ph h2xs instmodsh perlbug perldoc perlivp pl2pm
prove splain libnetcfg piconv enc2xs encguess xsubpp pod2html
-plextractexe = ./corelist ./cpan ./h2ph ./h2xs ./instmodsh ./perlbug ./perldoc
./perlivp ./pl2pm ./prove ./splain ./libnetcfg ./piconv ./enc2xs ./encguess
./xsubpp ./pod2html
+pl = corelist.PL cpan.PL h2ph.PL h2xs.PL instmodsh.PL json_pp.PL perlbug.PL
perldoc.PL perlivp.PL pl2pm.PL prove.PL splain.PL libnetcfg.PL piconv.PL
enc2xs.PL encguess.PL xsubpp.PL pod2html.PL
+plextract = corelist cpan h2ph
Re: Remove some customization from our perl build
On Fri, Apr 10, 2020 at 07:34:21PM -0600, Todd C. Miller wrote: > On Fri, 10 Apr 2020 18:17:33 -0700, Andrew Hewus Fresh wrote: > > > Recently it was pointed out that we don't link /usr/lib/libperl.so.* to > > libm the way is expected for code that also links to libperl. That led > > me to go digging again into the customization we have around the perl > > build and getting terribly confused. That did somewhat clear up after > > reading more about bsd.*.mk, but still feel like some of this mess was > > to make the vax work, but I couldn't actually figure it out from the cvs > > logs why it exists. > > The way OpenBSD builds is that the libraries are built and installed > first, then the binaries that link against those binaries get built > and linked against the new libs. This guarantees that we don't > link new binaries against the old libs. I was so confused, had to rebuild with and without and compare logs, and it didn't seem like this was happening for libperl. Then I figured out it stopped happening for libperl a while back. So, it does seem like we shouldn't need all this code anymore. http://cvsweb.openbsd.org/src/Makefile?rev=1.92=text/x-cvsweb-markup http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/Makefile.diff?r1=1.91=1.92=h https://github.com/openbsd/src/commit/7c8333e2365b13cb84a047094edcf2469959cffa l8rZ, -- andrew - http://afresh1.com Instructions are just another man's opinion of how to do something. -- Weldboy #DPWisdom
Remove some customization from our perl build
Recently it was pointed out that we don't link /usr/lib/libperl.so.* to
libm the way is expected for code that also links to libperl. That led
me to go digging again into the customization we have around the perl
build and getting terribly confused. That did somewhat clear up after
reading more about bsd.*.mk, but still feel like some of this mess was
to make the vax work, but I couldn't actually figure it out from the cvs
logs why it exists.
In any case, this patch does a few things, some of which I can split up
and put in separately if it comes to that.
* Puts back some of the upstream Makefile.SH that we removed
* and a little Dynaloader too
* Changes Configure flags to -de instead of -dsE
* So Configure does the work previously handled by depend.done
* Adjusts the installperl script to put libperl where we want it
* Moves some build flag discovery into hint/openbsd.sh
(Which I can then push upstream)
* Figuring out the correct PICFLAG, which means perl will now use the
same one, not -fpic for things built by Makefile.bsdwapper on archs
that want it and -fPIC for everything else.
* Using no-tree-ter on alpha, due to a compiler bug.
* Lets the perl infrastructure build libperl again
* Notably, this stops creating libperl.a, but I have a patch
around that puts it back, just not sure if we need it.
* Which means "we" don't actually build anything anymore, we leave all
that to the perl upstream Makefile so all the "stuff" to do with
that can go away.
* Which means ldd now mentions libm as it should
* Some tidying of the rest of Makefile.bsd-wrapper*
It seems to build fine on my
alpha, amd64, arm64, armv7, i386, macppc, octeon, and sparc64.
The individual changes are committed, in a clean-up-build branch, to the
GitHub repo where I keep track of them, along with build logs from my
test machines both with and without this patch:
https://github.com/afresh1/OpenBSD-perl/tree/clean-up-build
Does anyone know if we need any this and if so, why?
(especially libperl.a)
Is what I did in the hints file a reasonable way to find the PICFLAG?
This script should only ever run on OpenBSD, so should I just assume
bsd.own.mk exists?
There is probably further cleanup that can be done.
Thanks for your feedback.
Index: distrib/sets/lists/comp/mi
===
RCS file: /cvs/src/distrib/sets/lists/comp/mi,v
retrieving revision 1.1495
diff -u -p -r1.1495 mi
--- distrib/sets/lists/comp/mi 2 Mar 2020 20:59:38 - 1.1495
+++ distrib/sets/lists/comp/mi 9 Apr 2020 02:29:59 -
@@ -1531,7 +1531,6 @@
./usr/lib/libpanelw_p.a
./usr/lib/libpcap.a
./usr/lib/libpcap_p.a
-./usr/lib/libperl.a
./usr/lib/libpthread.a
./usr/lib/libpthread_p.a
./usr/lib/libradius.a
Index: gnu/usr.bin/perl/installperl
===
RCS file: /cvs/src/gnu/usr.bin/perl/installperl,v
retrieving revision 1.49
diff -u -p -r1.49 installperl
--- gnu/usr.bin/perl/installperl30 Dec 2019 02:15:16 - 1.49
+++ gnu/usr.bin/perl/installperl9 Apr 2020 02:30:06 -
@@ -383,7 +383,7 @@ elsif ($Is_Cygwin) { # On Cygwin symlink
# [als] hard-coded 'libperl' name... not good!
#@corefiles = <*.h libperl*.* perl*$Config{lib_ext}>;
@corefiles = <*.h *.inc perl*$Config{lib_ext}>;
-push(@corefiles,) unless defined($ENV{"NOLIBINSTALL"});
+install($libperl, "$opts{destdir}$Config{glibpth}/$libperl", "0444");
# AIX needs perl.exp installed as well.
push(@corefiles,'perl.exp') if $^O eq 'aix';
Index: gnu/usr.bin/perl/hints/openbsd.sh
===
RCS file: /cvs/src/gnu/usr.bin/perl/hints/openbsd.sh,v
retrieving revision 1.71
diff -u -p -r1.71 openbsd.sh
--- gnu/usr.bin/perl/hints/openbsd.sh 30 Dec 2019 02:15:18 - 1.71
+++ gnu/usr.bin/perl/hints/openbsd.sh 9 Apr 2020 02:30:37 -
@@ -47,7 +47,11 @@ alpha-2.[0-8]|mips-2.[0-8]|powerpc-2.[0-
test -z "$usedl" && usedl=$define
# We use -fPIC here because -fpic is *NOT* enough for some of the
# extensions like Tk on some OpenBSD platforms (ie: sparc)
- cccdlflags="-DPIC -fPIC $cccdlflags"
+ PICFLAG=-fPIC
+ if [ -e /usr/share/mk/bsd.own.mk ]; then
+ PICFLAG=`make -f /usr/share/mk/bsd.own.mk -V PICFLAG`
+ fi
+ cccdlflags="-DPIC ${PICFLAG} $cccdlflags"
case "$osvers" in
[01].*|2.[0-7]|2.[0-7].*)
lddlflags="-Bshareable $lddlflags"
@@ -58,7 +62,7 @@ alpha-2.[0-8]|mips-2.[0-8]|powerpc-2.[0-
;;
*) # from 3.1 onwards
ld=${cc:-cc}
- lddlflags="-shared -fPIC $lddlflags"
+ lddlflags="-shared ${PICFLAG} $lddlflags"
libswanted=`echo $libswanted | sed 's/ dl / /'`
;;
esac
@@ -84,6 +88,7 @@ esac
# around for old NetBSD binaries.
libswanted=`echo
OpenBSD perl 5.30.2 - Call for Testing
There's a minor update for perl 5.30.2 out, mostly just some bugfixes,
but also documentation updates because the main repository and bug
tracker for perl has moved to GitHub.
The full perldeltas are here the main one for for 5.30.0, and 5.30.1
that we have in-tree now and the smaller bugfixes in 5.30.2.
https://metacpan.org/pod/release/SHAY/perl-5.30.2/pod/perl5300delta.pod
https://metacpan.org/pod/release/SHAY/perl-5.30.2/pod/perl5301delta.pod
https://metacpan.org/pod/release/SHAY/perl-5.30.2/pod/perldelta.pod
The logs for the testing I have done is here.
https://github.com/afresh1/OpenBSD-perl/tree/master/build_logs/5.30.2
Which includes alpha, amd64, arm64, armv7, i386, macppc, sparc64, and octeon.
On my alpha there's an odd failure in 01_IPC_Cmd.t, but that's not new
and seems to be something with buffers not getting flushed as expected.
For running the perl test suite, you can follow the instructions on
GitHub, repeated here:
https://github.com/afresh1/OpenBSD-perl
download the patches and scripts
https://github.com/afresh1/OpenBSD-perl/archive/master.tar.gz
and extract someplace
or git clone https://github.com/afresh1/OpenBSD-perl.git
download perl-5.30.2.tar.gz into the same directory
https://cpan.metacpan.org/authors/id/S/SH/SHAY/perl-5.30.2.tar.gz
cd to someplace you have room
run /path/to/OpenBSD-perl/build_perl
wait
send me the log file(s) it generates
You can download a pre-patched version of perl that can be extracted to
replace src/gnu/usr.bin/perl for building a system with the new perl:
https://cvs.afresh1.com/~andrew/perl-update/OpenBSD-perl-5.30.2.tar.gz
I also have what should be a mostly correct sets/lists patch for building a
release.
https://cvs.afresh1.com/~andrew/perl-update/OpenBSD-perl-5.30.2-sets_lists.patch
Unfortunately, there are some characters in test files that are causing
`patch` to fail, so although you can get the patch here, I haven't been
able to figure out how to apply it successfully. If you want to try,
this is supposed to apply to -current to update it to perl 5.30.2, to do
that you would copy it into /usr/src (or adjust the paths below) and
follow these ugly instructions. You'll still need the above sets lists
patch if you're planning to build a release.
https://cvs.afresh1.com/~andrew/perl-update/OpenBSD-perl-5.30.2.patch.gz
or
https://cvs.afresh1.com/~andrew/perl-update/OpenBSD-perl-5.30.2.patch
cd /usr/src
patch -p0 -uNE < perl-5.30.2.patch
# Remove patch cruft
find gnu/usr.bin/perl -name '*.orig' -delete
# Add and remove binary and zero sized files that patch doesn't understand
grep -B1 -e '^Index:' -e 'Binary files /tmp/.* and /dev/null differ' \
perl-5.30.2.patch | sed -ne 's/^diff -N //p' |
while read f; do if [ -e $f ]; then rm $f; else touch $f; fi; done
cd gnu/usr.bin/perl && find -d . \
\( -type d -o -path '*/CVS' -prune \) \
! -name CVS \
-exec test -e {}/CVS \; \
-execdir sh -c 'test $( ls -1 {} | grep -v '^CVS/$' | wc -l ) -eq 0' \; \
-exec rm -r {} \;
