Re: UEFI + SOL / COM ports = boot hang

[Matt Simerson]

The default state of this server is that FreeBSD hangs when booting off UEFI 
and the COM ports are enabled. That seems to be because console includes or 
adds "efi" by default. Under every circumstance I've tested, when console 
contains "efi", the server hangs at the same point shown in the first message.

The workaround I've found is setting:

console="comconsole"

FreeBSD assumes that COM 1 is serial, which likely fails to initialize and so 
with only that setting, the video console remains primary. When that happens, I 
get no Beastie boot menu on either console. Since the server is now bootable 
with COM ports, I can tell the kernel where to find the serial port:

# without the next 2 lines, the serial console isn't found and the video 
console remains primary
comconsole_port="0x2F8"
comconsole_speed="115200"

And now it boots, I get the Beastie menu on the serial console, and later the 
FreeBSD kernel reports:

Dual Console: Serial Primary, Video Secondary

Changing the settings in BIOS related to console redirection seem not to matter 
much at all. Pretty much any combination of settings works.

This isn't perfect. The UEFI video console is certainly nicer, and I'd prefer 
to have it be primary and serial be secondary. But I cannot include 'efi' in 
loader.conf console setting while the COM ports are enabled else FreeBSD hangs.

The contents of the EFI variable ConOut with console ports disabled:

# efivar -p -d 8be4df61-93ca-11d2-aa0d-00e098032b8c-ConOut
8be4df61-93ca-11d2-aa0d-00e098032b8c-ConOut
: PciRoot(0x0)/Pci(0x1c,0x0)/Pci(0x0,0x0)/Pci(0x0,0x0)/AcpiAdr(0x80010100)

With console ports enabled and console="comconsole":

# efivar -p -d 8be4df61-93ca-11d2-aa0d-00e098032b8c-ConOut
8be4df61-93ca-11d2-aa0d-00e098032b8c-ConOut
: 
PciRoot(0x0)/Pci(0x1c,0x0)/Pci(0x0,0x0)/Pci(0x0,0x0)/AcpiAdr(0x80010100),/PciRoot(0x0)/Pci(0x1f,0x0)/Serial(0x0)/Uart(115200,8,N,1)/UartFlowCtrl(Hardware)/VenUtf8(),/PciRoot(0x0)/Pci(0x1f,0x0)/Serial(0x1)/Uart(115200,8,N,1)/UartFlowCtrl(Hardware)/VenUtf8()

From dmesg:

EFI framebuffer information:
addr, size 0x9c00, 0x30
dimensions 1024 x 768
stride 1024
masks  0x00ff, 0xff00, 0x00ff, 0xff00

VT(efifb): resolution 1024x768
...
vgapci0:  port 0x2000-0x207f mem 
0x9c00-0x9cff,0x9d00-0x9d01 irq 16 at device 0.0 numa-domain 0 
on pci2
vgapci0: Boot video device

I was hoping that setting hw.uart.console="io:0x2f8,br:115200" in loader.conf 
would help, but the server still hangs if 'efi' is included in the console list.

Messing with hint.uart.N as suggested in https://wiki.freebsd.org/SerialConsole 
didn't help, I'm guessing that's because those settings aren't used by 
loader.efi.

Any ideas or pointers for other settings which may enable the EFI console to be 
primary while also having a serial console?

Thanks,
Matt


> On Sep 23, 2024, at 3:22 PM, Matt Simerson  wrote:
> 
> I have a Quanta D52B-1U server. When I boot it via legacy BIOS, there is no 
> issue. When I boot with UEFI and COM ports & console redirection disabled, 
> there is no issue. However, when boot UEFI and enable the COM ports and/or 
> console redirection, the server hangs at boot time right here:
> 
> Version 2.20.1276. Copyright (C) 2023 American Megatrends, Inc.BIOS Date: 
> 12/04/2023 17:24:26 Ver: 3B28.Q102 
> Consoles: EFI console
> Reading loader env vars from /efi/freebsd/loader.env
> Setting currdev to disk0p1:
> FreeBSD/amd64 EFI loader, Revision 1.1
> 
>Command line arguments: loader.efi
>Image base: 0x5e993000
>EFI version: 2.70
>EFI Firmware: American Megatrends (rev 5.14)
>Console: efi (0x2000)
>Load Path: \EFI\FREEBSD\LOADER.EFI
>Load Device: 
> PciRoot(0x9)/Pci(0x0,0x0)/Pci(0x0,0x0)/NVMe(0x1,00-01-30-54-9D-E4-D2-5C)/HD(1,GPT,F669D424-7170-11EF-83D1-A0369F6BA978,0x28,0x82000)
>BootCurrent: 
>BootOrder: [*] 0003 0006 0007 0002 0008
>BootInfo Path: 
> HD(1,GPT,F669D424-7170-11EF-83D1-A0369F6BA978,0x28,0x82000)/\EFI\FREEBSD\LOADER.EFI
> Ignoring Boot: Only one DP found
> Trying ESP: 
> PciRoot(0x9)/Pci(0x0,0x0)/Pci(0x0,0x0)/NVMe(0x1,00-01-30-54-9D-E4-D2-5C)/HD(1,GPT,F669D424-7170-11EF-83D1-A0369F6BA978,0x28,0x82000)
> Setting currdev to disk0p1:
> Trying: 
> PciRoot(0x9)/Pci(0x0,0x0)/Pci(0x0,0x0)/NVMe(0x1,00-01-30-54-9D-E4-D2-5C)/HD(2,GPT,4F537B49-7176-11EF-83D1-A0369F6BA978,0x82028,0x100)
> Setting currdev to disk0p2:
> Trying: 
> PciRoot(0x9)/Pci(0x0,0x0)/Pci(0x0,0x0)/NVMe(0x1,00-01-30-54-9D-E4-D2-5C)/HD(3,GPT,B6D8B384-7176-11EFLoading
>  /boot/defaults/loader.conf86860)
> Loading /boot/defaults/loader.confdefault:
> Loading /boot/device.hints
> Loading /boot/loader.conf
> Loading /boot/loader.conf.local

UEFI + SOL / COM ports = boot hang

[Matt Simerson]

I have a Quanta D52B-1U server. When I boot it via legacy BIOS, there is no 
issue. When I boot with UEFI and COM ports & console redirection disabled, 
there is no issue. However, when boot UEFI and enable the COM ports and/or 
console redirection, the server hangs at boot time right here:

Version 2.20.1276. Copyright (C) 2023 American Megatrends, Inc.
BIOS Date: 12/04/2023 17:24:26 Ver: 3B28.Q102 
Consoles: EFI console
Reading loader env vars from /efi/freebsd/loader.env
Setting currdev to disk0p1:
FreeBSD/amd64 EFI loader, Revision 1.1

   Command line arguments: loader.efi
   Image base: 0x5e993000
   EFI version: 2.70
   EFI Firmware: American Megatrends (rev 5.14)
   Console: efi (0x2000)
   Load Path: \EFI\FREEBSD\LOADER.EFI
   Load Device: 
PciRoot(0x9)/Pci(0x0,0x0)/Pci(0x0,0x0)/NVMe(0x1,00-01-30-54-9D-E4-D2-5C)/HD(1,GPT,F669D424-7170-11EF-83D1-A0369F6BA978,0x28,0x82000)
   BootCurrent: 
   BootOrder: [*] 0003 0006 0007 0002 0008
   BootInfo Path: 
HD(1,GPT,F669D424-7170-11EF-83D1-A0369F6BA978,0x28,0x82000)/\EFI\FREEBSD\LOADER.EFI
Ignoring Boot: Only one DP found
Trying ESP: 
PciRoot(0x9)/Pci(0x0,0x0)/Pci(0x0,0x0)/NVMe(0x1,00-01-30-54-9D-E4-D2-5C)/HD(1,GPT,F669D424-7170-11EF-83D1-A0369F6BA978,0x28,0x82000)
Setting currdev to disk0p1:
Trying: 
PciRoot(0x9)/Pci(0x0,0x0)/Pci(0x0,0x0)/NVMe(0x1,00-01-30-54-9D-E4-D2-5C)/HD(2,GPT,4F537B49-7176-11EF-83D1-A0369F6BA978,0x82028,0x100)
Setting currdev to disk0p2:
Trying: 
PciRoot(0x9)/Pci(0x0,0x0)/Pci(0x0,0x0)/NVMe(0x1,00-01-30-54-9D-E4-D2-5C)/HD(3,GPT,B6D8B384-7176-11EFLoading
 /boot/defaults/loader.conf86860)
Loading /boot/defaults/loader.confdefault:
Loading /boot/device.hints
Loading /boot/loader.conf
Loading /boot/loader.conf.local
|

On one occasion, I left it setting there for minutes.

I currently have FreeBSD 14.1 installed and I also replicated this when booting 
off a FreeBSD 15.0-CURRENT ISO. 

The chipset is Intel C621 & C624. Output from dmesg is here 
<https://gist.github.com/msimerson/1b6a3f3e34f9036b30e0972477993cfc>.

BIOS settings which enable console redirection and SOL are:

Super IO
Port 0, Disabled or Enabled (IO=3F8h, IRQ=4)
Port 1, Enabled (IO=2F8h, IRQ=3)   <-- SOL port
Serial Port Console Redirection
COM0
Console Redirection, Port is Disabled
SOL
Console Redirection, [Enabled]
Console Redirection Settings, (VT100, 115200, None)
Legacy Console Redirection Settings
Redirection COM Port: SOL (other option: COM0)
Serial Port for OOB Management/Windows EMS
Console Redirection: Enabled
Console Redirection Settings, Out-of-Band Mgmt Port: SOL (other 
option: COM0)
Console Redirection Settings, (VT100, 115200, None)

There are variations on those settings which work for getting BIOS POST 
messages and the FreeBSD loader messages redirected to the IPMI SOL port.  
However, any combination that works also causes FreeBSD to hang at exactly the 
same spot.

I'm accustomed to adding a few lines to loader.conf to tell FreeBSD how to 
direct console output, but nothing added there makes any difference. I've 
twiddled the hint.uart.* knobs in /boot/device.hints to no avail.

This is what works on my Dell R630:

# console port via IPMI & DRAC
boot_multicons="YES"
boot_serial="YES"
console="efi,comconsole"
# for legacy BIOS
#console="comconsole,vidconsole"
comconsole_speed="115200"

How does one go about troubleshooting this?

Matt

Re: A few good ports on release iso images ?

[Tomoaki AOKI]

On Sat, 10 Aug 2024 09:02:13 -0700 (PDT)
"Rodney W. Grimes"  wrote:

> > Shawn Webb writes:
> > 
> > > While probably less efficient than just running the tools outright, I
> > > usually just set up a tmpfs that I chroot into and install those kinds
> > > of packages.
> > 
> > Yeah, I do something similar, with the footnote that I more often than
> > not have no internet connection, so I have to remember to bring the
> > packages.
> 
> Build yourself a full release on USB and just  be done with it.
> The project keeps deleting stuff from base I find very useful,
> I have given up on releases as a tool, I build a stick install
> and tweak all the things I need, image it "just in case" and
> keep it around for emergencies, and daily special use.
> 
> A new stick is created for each release.
> 
> 
> -- 
> Rod Grimes rgri...@freebsd.org

Currently, IIRC, DVD image contains at least some of pkgs, while other
images contains none.

What would be useful for ALL images would be:
  *ports-mgmt/pkg,
  *All (at least, network interface ones) kmod ports which are allowed
   by their license and built with exactly the same src tree as the
   release itself.

In some cases, any of kmod ports are needed to get internet access
working, especially relatively new notebooks.

What's important is, as already noted above, the bundled kmod ports
SHALL be built with the kernel to be installed, exactly.

For *.0 releases, there should be no problem with official pkgs.
But for *.1 and later, official pkgs are built against oldest supported
release of the corresponding stable branch, thus cause failing to
kldload.

Basically, my opinion is to force kmod ports to be always built from
ports and stop providing official pkgs for them, but the above is the
only exception. Internet access is needed to obtain/sync ports tree and
distfiles to build. So kmods for "bootstrapping" would be important.

For example, I've recently found this thread on forums.freebsd.org. [1]

[1]
https://forums.freebsd.org/threads/new-system-hardware-supported.93574/

-- 
Tomoaki AOKI

Re: A few good ports on release iso images ?

[Rodney W. Grimes]

> Shawn Webb writes:
> 
> > While probably less efficient than just running the tools outright, I
> > usually just set up a tmpfs that I chroot into and install those kinds
> > of packages.
> 
> Yeah, I do something similar, with the footnote that I more often than
> not have no internet connection, so I have to remember to bring the
> packages.

Build yourself a full release on USB and just  be done with it.
The project keeps deleting stuff from base I find very useful,
I have given up on releases as a tool, I build a stick install
and tweak all the things I need, image it "just in case" and
keep it around for emergencies, and daily special use.

A new stick is created for each release.


-- 
Rod Grimes rgri...@freebsd.org

Re: A few good ports on release iso images ?

[Poul-Henning Kamp]

Shawn Webb writes:

> While probably less efficient than just running the tools outright, I
> usually just set up a tmpfs that I chroot into and install those kinds
> of packages.

Yeah, I do something similar, with the footnote that I more often than
not have no internet connection, so I have to remember to bring the
packages.


-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.

Re: A few good ports on release iso images ?

[Shawn Webb]

On Tue, Jul 30, 2024 at 07:07:51PM +, Poul-Henning Kamp wrote:
> I do not want want this to turn into a everything-and-Emacs bloat-party,
> but I would find it really helpful if our install-ISO images had two
> HW-spelunking ports installed:
> 
>   sysutils/smartmontools
> 
> and
> 
>   sysutils/dmidecode
> 
> Is that even possible ?
> 
> Am I the only one who thinks so ?

While probably less efficient than just running the tools outright, I
usually just set up a tmpfs that I chroot into and install those kinds
of packages.

Just boot memstick.img, mount a tmpfs and a /dev under that, chroot
and bring up networking, then pkg install.

Simple process, but certainly more steps than having the tools
pre-installed on the install media.

Thanks,

-- 
Shawn Webb
Cofounder / Security Engineer
HardenedBSD

Tor-ified Signal: +1 303-901-1600 / shawn_webb_opsec.50
https://git.hardenedbsd.org/hardenedbsd/pubkeys/-/raw/master/Shawn_Webb/03A4CBEBB82EA5A67D9F3853FF2E67A277F8E1FA.pub.asc


signature.asc
Description: PGP signature

Re: A few good ports on release iso images ?

[Amar Takhar]

On 2024-07-30 19:07 +, Poul-Henning Kamp wrote:
 
>   sysutils/smartmontools
> 
> and
> 
>   sysutils/dmidecode
> 
> Is that even possible ?
> 
> Am I the only one who thinks so ?

I have missed having these tools numerous times over the years.  Whether it's 
installing on a new machine or trying to figure out a failure on a current one 
both of these are indispensable.  Would love to see them included.


Amar.

A few good ports on release iso images ?

[Poul-Henning Kamp]

I do not want want this to turn into a everything-and-Emacs bloat-party,
but I would find it really helpful if our install-ISO images had two
HW-spelunking ports installed:

sysutils/smartmontools

and

sysutils/dmidecode

Is that even possible ?

Am I the only one who thinks so ?

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.

Re: pkg-1.21.0: after upgrade 1.20.9_1 -> 1.21.0: pkg core dumps on specific ports

[Baptiste Daroussin]

Le 9 avril 2024 18:06:12 GMT+02:00, FreeBSD User  a 
écrit :
>Am Tue, 9 Apr 2024 17:10:52 +0200
>Rainer Hurling  schrieb:
>
>> Am 09.04.24 um 09:20 schrieb Baptiste Daroussin:
>> > On Sat 06 Apr 09:23, Rainer Hurling wrote:  
>> >> Am 06.04.24 um 09:05 schrieb FreeBSD User:  
>> >>> Hello,
>> >>>
>> >>> after updating (portmaster and make) ports-mgmt/ports from 1.20.9_1 -> 
>> >>> 1.21.0 on CURRENT
>> >>> and 14-STABLE, I can't update several ports:
>> >>>
>> >>> www/apache24
>> >>> databases/redis
>> >>>
>> >>> pkg core dumps while performing installation. apache24 and redis are 
>> >>> ports I realized
>> >>> this misbehaviour on ALL 14-STABLE and CURRENT boxes (both OS variants 
>> >>> latest builds,
>> >>> i.e. FreeBSD 15.0-CURRENT #32 main-n269135-da2b732288c7: Fri Apr  5 
>> >>> 20:30:39 CEST 2024
>> >>> amd64).
>> >>>
>> >>> After some updates on a poudriere builder (CURRENT base host, 
>> >>> 14.0-RELENG jail with
>> >>> poudriere) building packages for 14.0-RELENG, I observed the same 
>> >>> behaviour when
>> >>> updating packages on target hosts where pkg is first updated, on those 
>> >>> hosts,
>> >>> nextcloud-server and icinga2 host utilizing also databases/redis and 
>> >>> www/apache24, pkg
>> >>> fails the same way.
>> >>>
>> >>> I do not dare to update our poudriere hosts since the problem seems to 
>> >>> pop up when pkg
>> >>> 1.21.0 is installed, no matter whether I use poudriere built ports (from 
>> >>> our own builder
>> >>> hosts) or recent source tree with portmaster/make build process.
>> >>>
>> >>> Looks like a serious bug to me and not a site/user specific problem. 
>> >>> Hopefully others do
>> >>> realize the same ...
>> >>>
>> >>> Thanks in advance,
>> >>>
>> >>> oh  
>> >>
>> >>
>> >> Hmm, I just tried to reproduce that. Both ports mentioned, databases/redis
>> >> and www/apache24, can be built and installed with Portmaster. The box is a
>> >> 15.0-CURRENT with pkg-1.21.0.
>> >>
>> >> Maybe 'pkg check -Bn' or 'portmaster --check-depends --check-port-dbdir'
>> >> show some inconsistencies?
>> >>
>> >> Best wishes,
>> >> Rainer
>> >>
>> >>  
>> > using portmaster or not are strictly unlikely to be helpful here.
>> > 
>> > The right way to test if to report running with pkg - and also to 
>> > recommand
>> > testing with default options in pkg.conf.
>> > 
>> > Best regards,
>> > Bapt  
>> 
>> This is correct and certainly better. I was not aware of this.
>> 
>> Fortunately, my less optimal suggestions helped O. Hartmann in this case 
>> to find the missing and outdated dependencies.
>> 
>> In any case, many thanks for this helpfull advice.
>> 
>> Regards,
>> Rainer
>> 
>> 
>
>Hello,
>
>@Babptist : it should be pkg -d, shouldn't it? Or do I miss again something 
>here?

Each d will provide a more verbose level of debug

Bapt

Re: pkg-1.21.0: after upgrade 1.20.9_1 -> 1.21.0: pkg core dumps on specific ports

[FreeBSD User]

Am Tue, 9 Apr 2024 17:10:52 +0200
Rainer Hurling  schrieb:

> Am 09.04.24 um 09:20 schrieb Baptiste Daroussin:
> > On Sat 06 Apr 09:23, Rainer Hurling wrote:  
> >> Am 06.04.24 um 09:05 schrieb FreeBSD User:  
> >>> Hello,
> >>>
> >>> after updating (portmaster and make) ports-mgmt/ports from 1.20.9_1 -> 
> >>> 1.21.0 on CURRENT
> >>> and 14-STABLE, I can't update several ports:
> >>>
> >>> www/apache24
> >>> databases/redis
> >>>
> >>> pkg core dumps while performing installation. apache24 and redis are 
> >>> ports I realized
> >>> this misbehaviour on ALL 14-STABLE and CURRENT boxes (both OS variants 
> >>> latest builds,
> >>> i.e. FreeBSD 15.0-CURRENT #32 main-n269135-da2b732288c7: Fri Apr  5 
> >>> 20:30:39 CEST 2024
> >>> amd64).
> >>>
> >>> After some updates on a poudriere builder (CURRENT base host, 14.0-RELENG 
> >>> jail with
> >>> poudriere) building packages for 14.0-RELENG, I observed the same 
> >>> behaviour when
> >>> updating packages on target hosts where pkg is first updated, on those 
> >>> hosts,
> >>> nextcloud-server and icinga2 host utilizing also databases/redis and 
> >>> www/apache24, pkg
> >>> fails the same way.
> >>>
> >>> I do not dare to update our poudriere hosts since the problem seems to 
> >>> pop up when pkg
> >>> 1.21.0 is installed, no matter whether I use poudriere built ports (from 
> >>> our own builder
> >>> hosts) or recent source tree with portmaster/make build process.
> >>>
> >>> Looks like a serious bug to me and not a site/user specific problem. 
> >>> Hopefully others do
> >>> realize the same ...
> >>>
> >>> Thanks in advance,
> >>>
> >>> oh  
> >>
> >>
> >> Hmm, I just tried to reproduce that. Both ports mentioned, databases/redis
> >> and www/apache24, can be built and installed with Portmaster. The box is a
> >> 15.0-CURRENT with pkg-1.21.0.
> >>
> >> Maybe 'pkg check -Bn' or 'portmaster --check-depends --check-port-dbdir'
> >> show some inconsistencies?
> >>
> >> Best wishes,
> >> Rainer
> >>
> >>  
> > using portmaster or not are strictly unlikely to be helpful here.
> > 
> > The right way to test if to report running with pkg - and also to 
> > recommand
> > testing with default options in pkg.conf.
> > 
> > Best regards,
> > Bapt  
> 
> This is correct and certainly better. I was not aware of this.
> 
> Fortunately, my less optimal suggestions helped O. Hartmann in this case 
> to find the missing and outdated dependencies.
> 
> In any case, many thanks for this helpfull advice.
> 
> Regards,
> Rainer
> 
> 

Hello,

@Babptist : it should be pkg -d, shouldn't it? Or do I miss again something 
here?

With today's update to pkg 1.21.1 the problem has vanished. 

@R. Hurling: Thanks for the tip using the checks. I missed that and somehow it 
revealed some
problems here I hopefully have fixed so far.

Kind regads and thanks,

oh
-- 
O. Hartmann

Re: pkg-1.21.0: after upgrade 1.20.9_1 -> 1.21.0: pkg core dumps on specific ports

[Rainer Hurling]

Am 09.04.24 um 09:20 schrieb Baptiste Daroussin:

On Sat 06 Apr 09:23, Rainer Hurling wrote:

Am 06.04.24 um 09:05 schrieb FreeBSD User:

Hello,

after updating (portmaster and make) ports-mgmt/ports from 1.20.9_1 -> 1.21.0 
on CURRENT and
14-STABLE, I can't update several ports:

www/apache24
databases/redis

pkg core dumps while performing installation. apache24 and redis are ports I 
realized this
misbehaviour on ALL 14-STABLE and CURRENT boxes (both OS variants latest 
builds, i.e. FreeBSD
15.0-CURRENT #32 main-n269135-da2b732288c7: Fri Apr  5 20:30:39 CEST 2024 
amd64).

After some updates on a poudriere builder (CURRENT base host, 14.0-RELENG jail 
with poudriere)
building packages for 14.0-RELENG, I observed the same behaviour when updating 
packages on
target hosts where pkg is first updated, on those hosts, nextcloud-server and 
icinga2 host
utilizing also databases/redis and www/apache24, pkg fails the same way.

I do not dare to update our poudriere hosts since the problem seems to pop up 
when pkg 1.21.0
is installed, no matter whether I use poudriere built ports (from our own 
builder hosts) or
recent source tree with portmaster/make build process.

Looks like a serious bug to me and not a site/user specific problem. Hopefully 
others do
realize the same ...

Thanks in advance,

oh



Hmm, I just tried to reproduce that. Both ports mentioned, databases/redis
and www/apache24, can be built and installed with Portmaster. The box is a
15.0-CURRENT with pkg-1.21.0.

Maybe 'pkg check -Bn' or 'portmaster --check-depends --check-port-dbdir'
show some inconsistencies?

Best wishes,
Rainer



using portmaster or not are strictly unlikely to be helpful here.

The right way to test if to report running with pkg - and also to recommand
testing with default options in pkg.conf.

Best regards,
Bapt


This is correct and certainly better. I was not aware of this.

Fortunately, my less optimal suggestions helped O. Hartmann in this case 
to find the missing and outdated dependencies.


In any case, many thanks for this helpfull advice.

Regards,
Rainer

Re: pkg-1.21.0: after upgrade 1.20.9_1 -> 1.21.0: pkg core dumps on specific ports

[Rainer Hurling]

Am 06.04.24 um 09:56 schrieb FreeBSD User:

Am Sat, 6 Apr 2024 09:23:30 +0200
Rainer Hurling  schrieb:


Am 06.04.24 um 09:05 schrieb FreeBSD User:

Hello,

after updating (portmaster and make) ports-mgmt/ports from 1.20.9_1 -> 1.21.0 
on CURRENT
and 14-STABLE, I can't update several ports:

www/apache24
databases/redis

pkg core dumps while performing installation. apache24 and redis are ports I 
realized this
misbehaviour on ALL 14-STABLE and CURRENT boxes (both OS variants latest 
builds, i.e.
FreeBSD 15.0-CURRENT #32 main-n269135-da2b732288c7: Fri Apr  5 20:30:39 CEST 
2024 amd64).

After some updates on a poudriere builder (CURRENT base host, 14.0-RELENG jail 
with
poudriere) building packages for 14.0-RELENG, I observed the same behaviour 
when updating
packages on target hosts where pkg is first updated, on those hosts, 
nextcloud-server and
icinga2 host utilizing also databases/redis and www/apache24, pkg fails the 
same way.

I do not dare to update our poudriere hosts since the problem seems to pop up 
when pkg
1.21.0 is installed, no matter whether I use poudriere built ports (from our 
own builder
hosts) or recent source tree with portmaster/make build process.

Looks like a serious bug to me and not a site/user specific problem. Hopefully 
others do
realize the same ...

Thanks in advance,

oh



Hmm, I just tried to reproduce that. Both ports mentioned,
databases/redis and www/apache24, can be built and installed with
Portmaster. The box is a 15.0-CURRENT with pkg-1.21.0.

Maybe 'pkg check -Bn' or 'portmaster --check-depends --check-port-dbdir'
show some inconsistencies?

Best wishes,
Rainer



Hello,

thanks for the quick response.

I checked on the CURRENT systems here at hand and must confess - it is a mess! 
pkg check -Bn
dropped a lot of missing shared objects missing from autotools and missing 
guile2 :-(

Thank you very much,
oh



You're really welcome. I myself have failed several times precisely 
because some dependencies were not in order. And that's not always 
obvious :)


Best wishes,
Rainer

Re: pkg-1.21.0: after upgrade 1.20.9_1 -> 1.21.0: pkg core dumps on specific ports

[Baptiste Daroussin]

On Sat 06 Apr 09:23, Rainer Hurling wrote:
> Am 06.04.24 um 09:05 schrieb FreeBSD User:
> > Hello,
> > 
> > after updating (portmaster and make) ports-mgmt/ports from 1.20.9_1 -> 
> > 1.21.0 on CURRENT and
> > 14-STABLE, I can't update several ports:
> > 
> > www/apache24
> > databases/redis
> > 
> > pkg core dumps while performing installation. apache24 and redis are ports 
> > I realized this
> > misbehaviour on ALL 14-STABLE and CURRENT boxes (both OS variants latest 
> > builds, i.e. FreeBSD
> > 15.0-CURRENT #32 main-n269135-da2b732288c7: Fri Apr  5 20:30:39 CEST 2024 
> > amd64).
> > 
> > After some updates on a poudriere builder (CURRENT base host, 14.0-RELENG 
> > jail with poudriere)
> > building packages for 14.0-RELENG, I observed the same behaviour when 
> > updating packages on
> > target hosts where pkg is first updated, on those hosts, nextcloud-server 
> > and icinga2 host
> > utilizing also databases/redis and www/apache24, pkg fails the same way.
> > 
> > I do not dare to update our poudriere hosts since the problem seems to pop 
> > up when pkg 1.21.0
> > is installed, no matter whether I use poudriere built ports (from our own 
> > builder hosts) or
> > recent source tree with portmaster/make build process.
> > 
> > Looks like a serious bug to me and not a site/user specific problem. 
> > Hopefully others do
> > realize the same ...
> > 
> > Thanks in advance,
> > 
> > oh
> 
> 
> Hmm, I just tried to reproduce that. Both ports mentioned, databases/redis
> and www/apache24, can be built and installed with Portmaster. The box is a
> 15.0-CURRENT with pkg-1.21.0.
> 
> Maybe 'pkg check -Bn' or 'portmaster --check-depends --check-port-dbdir'
> show some inconsistencies?
> 
> Best wishes,
> Rainer
> 
> 
using portmaster or not are strictly unlikely to be helpful here.

The right way to test if to report running with pkg - and also to recommand
testing with default options in pkg.conf.

Best regards,
Bapt

Re: pkg-1.21.0: after upgrade 1.20.9_1 -> 1.21.0: pkg core dumps on specific ports

[Baptiste Daroussin]

On Sat 06 Apr 09:05, FreeBSD User wrote:
> Hello,
> 
> after updating (portmaster and make) ports-mgmt/ports from 1.20.9_1 -> 1.21.0 
> on CURRENT and
> 14-STABLE, I can't update several ports:
> 
> www/apache24
> databases/redis
> 
> pkg core dumps while performing installation. apache24 and redis are ports I 
> realized this
> misbehaviour on ALL 14-STABLE and CURRENT boxes (both OS variants latest 
> builds, i.e. FreeBSD
> 15.0-CURRENT #32 main-n269135-da2b732288c7: Fri Apr  5 20:30:39 CEST 2024 
> amd64).
> 
> After some updates on a poudriere builder (CURRENT base host, 14.0-RELENG 
> jail with poudriere)
> building packages for 14.0-RELENG, I observed the same behaviour when 
> updating packages on
> target hosts where pkg is first updated, on those hosts, nextcloud-server and 
> icinga2 host
> utilizing also databases/redis and www/apache24, pkg fails the same way.
> 
> I do not dare to update our poudriere hosts since the problem seems to pop up 
> when pkg 1.21.0
> is installed, no matter whether I use poudriere built ports (from our own 
> builder hosts) or
> recent source tree with portmaster/make build process.
> 
> Looks like a serious bug to me and not a site/user specific problem. 
> Hopefully others do
> realize the same ...
> 
> Thanks in advance,
> 
> oh 
> 
https://github.com/freebsd/pkg/issues/2270

set HANDLE_RC_SCRIPTS=false in your pkg.conf

a Fix was made last friday, given this is a non default option I waited for the
Week end to pass to see if there were other regressions, but no more reports so
I will issue a pkg 1.21.1 now.

Best regards,
Bapt

Re: pkg-1.21.0: after upgrade 1.20.9_1 -> 1.21.0: pkg-static core dumps on some ports

[FreeBSD User]

Am Sat, 6 Apr 2024 09:05:00 +0200
FreeBSD User  schrieb:

> Hello,
> 
> after updating (portmaster and make) ports-mgmt/ports from 1.20.9_1 -> 1.21.0 
> on CURRENT and
> 14-STABLE, I can't update several ports:
> 
> www/apache24
> databases/redis
> 
> pkg core dumps while performing installation. apache24 and redis are ports I 
> realized this
> misbehaviour on ALL 14-STABLE and CURRENT boxes (both OS variants latest 
> builds, i.e. FreeBSD
> 15.0-CURRENT #32 main-n269135-da2b732288c7: Fri Apr  5 20:30:39 CEST 2024 
> amd64).
> 
> After some updates on a poudriere builder (CURRENT base host, 14.0-RELENG 
> jail with
> poudriere) building packages for 14.0-RELENG, I observed the same behaviour 
> when updating
> packages on target hosts where pkg is first updated, on those hosts, 
> nextcloud-server and
> icinga2 host utilizing also databases/redis and www/apache24, pkg fails the 
> same way.
> 
> I do not dare to update our poudriere hosts since the problem seems to pop up 
> when pkg 1.21.0
> is installed, no matter whether I use poudriere built ports (from our own 
> builder hosts) or
> recent source tree with portmaster/make build process.
> 
> Looks like a serious bug to me and not a site/user specific problem. 
> Hopefully others do
> realize the same ...
> 
> Thanks in advance,
> 
> oh 
> 
> 

Hello,

after following a recommnedation checking dependencies on ports via pkg check 
-Bn, recompiling
pkg via "portmaster -df ports-mgmt/pkg" along with all ports found by the check 
command as
well a sqlite (precaustion), still the pkg-static binary drops core dumps on 
some ports.

Phenomenon: When updating existing ports, like

www/apache24
databases/redis
net/openldap26-server
misc/e2fsprogs-libuuid

building the port runs smootly, but pkg-static dies on deleting attempt of the
old/to-be-reinstalled port. The problem arises by using portmaster as well as 
performing "make
install/make deinstall" in the specific target port.

Last port I hit is  misc/e2fsprogs-libuuid.

My skills debugging core dumps are rather limited, our boxes do have debugging 
disabled. Since
the problem spreads across several hosts running CURRENT (same IcyBridge CPU 
generation, but
one host most recent CURRENT with LLVM18, the other one running a CURRENT 
compiled 4 days ago
and as of last week the problem arose also on 14-STABLE on a box in the lab 
when performing
the tansition from pkg 1.20.9_1 -> 1.21.0), I'd exclude a hardware/memory issue.

Using (a freshly recompiled) gdb 14 from ports gives not much:

[...]
root@thor:/usr/ports # gdb  /usr/local/sbin/pkg-static
/packages/portmaster-backup/pkg-static.core
GNU gdb (GDB) 14.1 [GDB v14.1 for FreeBSD]
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd15.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/local/sbin/pkg-static...
(No debugging symbols found in /usr/local/sbin/pkg-static)
[New LWP 101269]
Core was generated by `/usr/local/sbin/pkg-static delete -yf 
e2fsprogs-libuuid-1.47.0'.
Program terminated with signal SIGSEGV, Segmentation fault.
Address not mapped to object.
#0  0x00b3de2b in strlen_baseline ()



Kind regards,
oh



-- 
O. Hartmann

Re: pkg-1.21.0: after upgrade 1.20.9_1 -> 1.21.0: pkg core dumps on specific ports

[Rainer Hurling]

Am 06.04.24 um 09:05 schrieb FreeBSD User:

Hello,

after updating (portmaster and make) ports-mgmt/ports from 1.20.9_1 -> 1.21.0 
on CURRENT and
14-STABLE, I can't update several ports:

www/apache24
databases/redis

pkg core dumps while performing installation. apache24 and redis are ports I 
realized this
misbehaviour on ALL 14-STABLE and CURRENT boxes (both OS variants latest 
builds, i.e. FreeBSD
15.0-CURRENT #32 main-n269135-da2b732288c7: Fri Apr  5 20:30:39 CEST 2024 
amd64).

After some updates on a poudriere builder (CURRENT base host, 14.0-RELENG jail 
with poudriere)
building packages for 14.0-RELENG, I observed the same behaviour when updating 
packages on
target hosts where pkg is first updated, on those hosts, nextcloud-server and 
icinga2 host
utilizing also databases/redis and www/apache24, pkg fails the same way.

I do not dare to update our poudriere hosts since the problem seems to pop up 
when pkg 1.21.0
is installed, no matter whether I use poudriere built ports (from our own 
builder hosts) or
recent source tree with portmaster/make build process.

Looks like a serious bug to me and not a site/user specific problem. Hopefully 
others do
realize the same ...

Thanks in advance,

oh



Hmm, I just tried to reproduce that. Both ports mentioned, 
databases/redis and www/apache24, can be built and installed with 
Portmaster. The box is a 15.0-CURRENT with pkg-1.21.0.


Maybe 'pkg check -Bn' or 'portmaster --check-depends --check-port-dbdir' 
show some inconsistencies?


Best wishes,
Rainer

pkg-1.21.0: after upgrade 1.20.9_1 -> 1.21.0: pkg core dumps on specific ports

[FreeBSD User]

Hello,

after updating (portmaster and make) ports-mgmt/ports from 1.20.9_1 -> 1.21.0 
on CURRENT and
14-STABLE, I can't update several ports:

www/apache24
databases/redis

pkg core dumps while performing installation. apache24 and redis are ports I 
realized this
misbehaviour on ALL 14-STABLE and CURRENT boxes (both OS variants latest 
builds, i.e. FreeBSD
15.0-CURRENT #32 main-n269135-da2b732288c7: Fri Apr  5 20:30:39 CEST 2024 
amd64).

After some updates on a poudriere builder (CURRENT base host, 14.0-RELENG jail 
with poudriere)
building packages for 14.0-RELENG, I observed the same behaviour when updating 
packages on
target hosts where pkg is first updated, on those hosts, nextcloud-server and 
icinga2 host
utilizing also databases/redis and www/apache24, pkg fails the same way.

I do not dare to update our poudriere hosts since the problem seems to pop up 
when pkg 1.21.0
is installed, no matter whether I use poudriere built ports (from our own 
builder hosts) or
recent source tree with portmaster/make build process.

Looks like a serious bug to me and not a site/user specific problem. Hopefully 
others do
realize the same ...

Thanks in advance,

oh 


-- 
O. Hartmann

Re: git repo ports issues?

[Jamie Landeg-Jones]

Yuri  wrote:

> Hi Jamie,
>
>
> This was a faulty script where 2023 wasn't changed to 2024.
>
> Thanks for the notice.

Ahh, ok, I did think it was probably a bit more than coincidence that it
was exactly 1 year to the day!

Cheers for the quick reply, Jamie

Re: The ports quarterly tree 2023Q4 is borked for www/firefox

[Dennis Clarke]

On 12/18/23 08:35, Dimitry Andric wrote:

On 18 Dec 2023, at 13:59, Dennis Clarke  wrote:



I do not know where else to post this. Seems that a bug report about the
problem does not mean much :


Bug 275814 - www/firefox has the wrong version in the Makefile
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=275814


Should be fixed by:
https://cgit.freebsd.org/ports/commit/?h=2023Q4&id=ec15ee20bdd912ca31982b1be0062da5fb783ac7

-Dimitry




awesome !  we have liftoff once again :

hydra#
hydra# poudriere ports -l
2023Q4git+ssh 2023-12-18 12:29:26 /usr/local/poudriere/ports/2023Q4
hydra#
hydra# poudriere ports -p 2023Q4 -u
[00:00:00] Updating portstree "2023Q4" with git+ssh... done
hydra#
hydra# poudriere bulk -J 6:8 -j 132_amd64 -p 2023Q4 -f pkgs_to_build.txt
[00:00:00] Creating the reference jail... done
[00:00:01] Mounting system devices for 132_amd64-2023Q4
[00:00:01] Mounting ports/packages/distfiles
[00:00:01] Stashing existing package repository
[00:00:01] Mounting packages from: 
/usr/local/poudriere/data/packages/132_amd64-2023Q4
/etc/resolv.conf -> 
/usr/local/poudriere/data/.m/132_amd64-2023Q4/ref/etc/resolv.conf

[00:00:01] Starting jail 132_amd64-2023Q4
[00:00:01] Will build as nobody: (65534:65534)
[00:00:01] Logs: 
/usr/local/poudriere/data/logs/bulk/132_amd64-2023Q4/2023-12-18_13h40m31s
[00:00:01] Loading MOVED for 
/usr/local/poudriere/data/.m/132_amd64-2023Q4/ref/usr/ports

[00:00:02] Ports supports: FLAVORS SELECTED_OPTIONS
[00:00:02] Gathering ports metadata
[00:00:08] Calculating ports order and dependencies
[00:00:09] Sanity checking the repository
[00:00:09] Checking packages for incremental rebuild needs
[00:00:11] Deleting stale symlinks... done
[00:00:11] Deleting empty directories... done
[00:00:12] Cleaning the build queue
[00:00:12] Sanity checking build queue
[00:00:12] Processing PRIORITY_BOOST
[00:00:12] Balancing pool
[00:00:12] Recording filesystem state for prepkg... done
[00:00:12] Building 1 packages using 1 builders
[00:00:12] Starting/Cloning builders
[00:00:14] Hit CTRL+t at any time to see build progress and stats
[00:00:14] [01] [00:00:00] Building www/firefox | firefox-121.0,2



Thank you very much !



Dennis Clarke
RISC-V/SPARC/PPC/ARM/CISC
UNIX and Linux spoken

Re: The ports quarterly tree 2023Q4 is borked for www/firefox

[Dimitry Andric]

On 18 Dec 2023, at 13:59, Dennis Clarke  wrote:
> 
> 
> I do not know where else to post this. Seems that a bug report about the
> problem does not mean much :
> 
> 
> Bug 275814 - www/firefox has the wrong version in the Makefile
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=275814

Should be fixed by:
https://cgit.freebsd.org/ports/commit/?h=2023Q4&id=ec15ee20bdd912ca31982b1be0062da5fb783ac7

-Dimitry

The ports quarterly tree 2023Q4 is borked for www/firefox

[Dennis Clarke]

I do not know where else to post this. Seems that a bug report about the
problem does not mean much :


Bug 275814 - www/firefox has the wrong version in the Makefile
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=275814

This issue is pretty clear.

This morning I flushed out my ports tree and started over fresh :

hydra# poudriere ports -c -U ssh://anon...@git.freebsd.org/ports.git \
? -B 2023Q4 \
? -f zroot/poudriere/ports/2023Q4 \
? -m 'git+ssh' -p 2023Q4 -v
[00:00:00] Creating 2023Q4 fs at /usr/local/poudriere/ports/2023Q4... done
[00:00:01] Cloning the ports tree...Cloning into 
'/usr/local/poudriere/ports/2023Q4'...

remote: Enumerating objects: 192710, done.
remote: Counting objects: 100% (192710/192710), done.
remote: Compressing objects: 100% (181019/181019), done.
remote: Total 192710 (delta 11046), reused 115681 (delta 5175), 
pack-reused 0

Receiving objects: 100% (192710/192710), 83.07 MiB | 4.91 MiB/s, done.
Resolving deltas: 100% (11046/11046), done.
Updating files: 100% (155563/155563), done.
 done
hydra#
hydra# poudriere ports -l
2023Q4git+ssh 2023-12-18 12:29:26 /usr/local/poudriere/ports/2023Q4
hydra#

hydra#
hydra# cat /usr/local/poudriere/ports/2023Q4/www/firefox/distinfo
TIMESTAMP = 1702328424
SHA256 (firefox-121.0.source.tar.xz) = 
edc7a5159d23ff2a23e22bf5abe22231658cee2902b93b5889ee73958aa06aa4

SIZE (firefox-121.0.source.tar.xz) = 530302784
hydra#
hydra# head /usr/local/poudriere/ports/2023Q4/www/firefox/Makefile
PORTNAME=   firefox
DISTVERSION=120.0.1
PORTEPOCH=  2
CATEGORIES= www wayland
MASTER_SITES= 
MOZILLA/${PORTNAME}/releases/${DISTVERSION}${DISTVERSIONSUFFIX}/source \


MOZILLA/${PORTNAME}/candidates/${DISTVERSION}${DISTVERSIONSUFFIX}-candidates/build1/source
DISTFILES=  ${DISTNAME}.source${EXTRACT_SUFX}

MAINTAINER= ge...@freebsd.org
COMMENT=Web browser based on the browser portion of Mozilla
hydra#

So the Makefile does not match the distfile and this borks up poudriere
over and over and over for about a week now :

hydra#
hydra# pwd
/usr/local/poudriere/ports/2023Q4
hydra# git blame www/firefox/distinfo
^17b2a0299 (Christoph Moench-Tegeder 2023-12-17 23:45:07 +0100 1) 
TIMESTAMP = 1702328424
^17b2a0299 (Christoph Moench-Tegeder 2023-12-17 23:45:07 +0100 2) SHA256 
(firefox-121.0.source.tar.xz) = 
edc7a5159d23ff2a23e22bf5abe22231658cee2902b93b5889ee73958aa06aa4
^17b2a0299 (Christoph Moench-Tegeder 2023-12-17 23:45:07 +0100 3) SIZE 
(firefox-121.0.source.tar.xz) = 530302784

hydra#

There is no way I can be the only person on the planet that likes to
have Firefox on their machine ... built from the sources.


Dennis Clarke
RISC-V/SPARC/PPC/ARM/CISC
UNIX and Linux spoken

Re: something magic about the size of a ports tree

[Allan Jude]

On 2023-10-03 12:24, Dag-Erling Smørgrav wrote:

Matthias Apitz  writes:

I have on my poudriere build host a ports tree and wanted to move it to
the host where the resulting packages are installed:

root@jet:/usr/local/poudriere/ports # du -sh ports20230806
397Mports20230806
root@jet:/usr/local/poudriere/ports # tar cf p.tar ports20230806
root@jet:/usr/local/poudriere/ports # ls -lh p.tar
-rw-r--r--  1 root wheel  672M Oct  3 18:00 p.tar

already the size of the tar file is somewhat magic; but if you un-tar it
on the other host I will get:

[guru@c720-1400094 ~]$ ls -lh p.tar
-rw-r--r--  1 guru wheel  672M  3 oct.  18:00 p.tar
[guru@c720-1400094 ~]$ tar xf p.tar
[guru@c720-1400094 ~]$ du -sh ports20230806
1,2Gports20230806

How this is possible?


Most files in the ports tree are very small.  On disk, each file gets
rounded up to the nearest multiple of the filesystem block size, which
could be as small as 512 bytes or as large as 8 kB (or even more in
pathological cases).  In a tarball, they get rounded up to the nearest
multiple of 512 bytes plus an additional 512 bytes per file for
metadata.

For instance, your average distinfo file (of which there are 30k in the
ports tree) is only 200-250 bytes long, but it occupies 512 bytes on an
FFS filesystem, 1 kB in a tarball, and 4 kB on a typical ZFS filesystem.



As an interesting side note to this, if ZFS is able to compress the file 
to under 112 bytes, ZFS will not allocate a sector, but instead store 
the file in an "embedded blockpointer", basically using the space it 
would normally store the LBAs and checksum of the file, to store the 
actual file data, resulting in a file that appears to use 0 bytes of 
space, because it entirely fits in the indirect block that would have 
pointed to the block itself.



Note that if the target system is FreeBSD 14 or newer, you can simply
mount the tarball (`sudo mount -rt tarfs p.tar /usr/ports`).

DES




--
Allan Jude

Re: something magic about the size of a ports tree

[Dag-Erling Smørgrav]

Warner Losh  writes:
> Do we support any compression on top of that? Has support for
> poudriere been added for it?

Yes (zstd) and no.

DES
-- 
Dag-Erling Smørgrav - d...@freebsd.org

Re: something magic about the size of a ports tree

[Warner Losh]

On Tue, Oct 3, 2023, 10:24 AM Dag-Erling Smørgrav  wrote:

> Matthias Apitz  writes:
> > I have on my poudriere build host a ports tree and wanted to move it to
> > the host where the resulting packages are installed:
> >
> > root@jet:/usr/local/poudriere/ports # du -sh ports20230806
> > 397Mports20230806
> > root@jet:/usr/local/poudriere/ports # tar cf p.tar ports20230806
> > root@jet:/usr/local/poudriere/ports # ls -lh p.tar
> > -rw-r--r--  1 root wheel  672M Oct  3 18:00 p.tar
> >
> > already the size of the tar file is somewhat magic; but if you un-tar it
> > on the other host I will get:
> >
> > [guru@c720-1400094 ~]$ ls -lh p.tar
> > -rw-r--r--  1 guru wheel  672M  3 oct.  18:00 p.tar
> > [guru@c720-1400094 ~]$ tar xf p.tar
> > [guru@c720-1400094 ~]$ du -sh ports20230806
> > 1,2G  ports20230806
> >
> > How this is possible?
>
> Most files in the ports tree are very small.  On disk, each file gets
> rounded up to the nearest multiple of the filesystem block size, which
> could be as small as 512 bytes or as large as 8 kB (or even more in
> pathological cases).  In a tarball, they get rounded up to the nearest
> multiple of 512 bytes plus an additional 512 bytes per file for
> metadata.
>
> For instance, your average distinfo file (of which there are 30k in the
> ports tree) is only 200-250 bytes long, but it occupies 512 bytes on an
> FFS filesystem, 1 kB in a tarball, and 4 kB on a typical ZFS filesystem.
>
> Note that if the target system is FreeBSD 14 or newer, you can simply
> mount the tarball (`sudo mount -rt tarfs p.tar /usr/ports`).
>

Do we support any compression on top of that? Has support for poudriere
been added for it?

Aldo I want a pony I'm mostly curious... I have no immediate plans here
(though aligning with the boot loader and supporting this on a block device
to support rootfs would be cool). Maybe some or all of these wishes would
make good GSOC projects?

Warner

DES
> --
> Dag-Erling Smørgrav - d...@freebsd.org
>
>

Re: something magic about the size of a ports tree

[Michael Gmelin]

> On 3. Oct 2023, at 18:27, Matthias Apitz  wrote:
> 
> El día martes, octubre 03, 2023 a las 06:14:23p. m. +0200, Olivier Certner 
> escribió:
> 
>> Hi Matthias,
>> 
>> Some ZFS dataset with zstd compression on jet, and no compression on 
>> c720-1400094?
>> 
> 
> Yes, on jet it is ZFS:
> 
> root@jet:/usr/local/poudriere/ports # mount | grep ports2023
> poudriere/poudriere/ports/ports20230806 on 
> /usr/local/poudriere/ports/ports20230806 (zfs, local, noatime, nfsv4acls)
> 
> on c720-1400094 it is only plain UFS.
> 

Try

du -hA file

Also, to experience the difference, try:

dd if=/dev/zero of=tempfile bs=1m count=10

and compare the results of ls, du -h, du -hA on the different filesystems.

   zfs get all | grep compr

can also be quite enlightening.

Cheers


>matthias
> 
> -- 
> Matthias Apitz, ✉ g...@unixarea.de, http://www.unixarea.de/ +49-176-38902045
> Public GnuPG key: http://www.unixarea.de/key.pub
> 

Re: something magic about the size of a ports tree

[Alan Somers]

With ZFS, you might be using transparent compression.  "du -sh" will
show you a file's compressed size.  But "ls -lh" will show you the
logical size.  That's probably why the tarball looked so much bigger
than the ports tree on the first system.  If you do "du -sh" on the
tarball, I bet you'll see a much smaller number.

On Tue, Oct 3, 2023 at 9:27 AM Matthias Apitz  wrote:
>
> El día martes, octubre 03, 2023 a las 06:14:23p. m. +0200, Olivier Certner 
> escribió:
>
> > Hi Matthias,
> >
> > Some ZFS dataset with zstd compression on jet, and no compression on 
> > c720-1400094?
> >
>
> Yes, on jet it is ZFS:
>
> root@jet:/usr/local/poudriere/ports # mount | grep ports2023
> poudriere/poudriere/ports/ports20230806 on 
> /usr/local/poudriere/ports/ports20230806 (zfs, local, noatime, nfsv4acls)
>
> on c720-1400094 it is only plain UFS.
>
> matthias
>
> --
> Matthias Apitz, ✉ g...@unixarea.de, http://www.unixarea.de/ +49-176-38902045
> Public GnuPG key: http://www.unixarea.de/key.pub
>

Re: something magic about the size of a ports tree

[Matthias Apitz]

El día martes, octubre 03, 2023 a las 06:14:23p. m. +0200, Olivier Certner 
escribió:

> Hi Matthias,
> 
> Some ZFS dataset with zstd compression on jet, and no compression on 
> c720-1400094?
> 

Yes, on jet it is ZFS:

root@jet:/usr/local/poudriere/ports # mount | grep ports2023
poudriere/poudriere/ports/ports20230806 on 
/usr/local/poudriere/ports/ports20230806 (zfs, local, noatime, nfsv4acls)

on c720-1400094 it is only plain UFS.

matthias

-- 
Matthias Apitz, ✉ g...@unixarea.de, http://www.unixarea.de/ +49-176-38902045
Public GnuPG key: http://www.unixarea.de/key.pub

Re: something magic about the size of a ports tree

[Dag-Erling Smørgrav]

Matthias Apitz  writes:
> I have on my poudriere build host a ports tree and wanted to move it to
> the host where the resulting packages are installed:
>
> root@jet:/usr/local/poudriere/ports # du -sh ports20230806
> 397Mports20230806
> root@jet:/usr/local/poudriere/ports # tar cf p.tar ports20230806
> root@jet:/usr/local/poudriere/ports # ls -lh p.tar
> -rw-r--r--  1 root wheel  672M Oct  3 18:00 p.tar
>
> already the size of the tar file is somewhat magic; but if you un-tar it
> on the other host I will get:
>
> [guru@c720-1400094 ~]$ ls -lh p.tar
> -rw-r--r--  1 guru wheel  672M  3 oct.  18:00 p.tar
> [guru@c720-1400094 ~]$ tar xf p.tar
> [guru@c720-1400094 ~]$ du -sh ports20230806
> 1,2G  ports20230806
>
> How this is possible?

Most files in the ports tree are very small.  On disk, each file gets
rounded up to the nearest multiple of the filesystem block size, which
could be as small as 512 bytes or as large as 8 kB (or even more in
pathological cases).  In a tarball, they get rounded up to the nearest
multiple of 512 bytes plus an additional 512 bytes per file for
metadata.

For instance, your average distinfo file (of which there are 30k in the
ports tree) is only 200-250 bytes long, but it occupies 512 bytes on an
FFS filesystem, 1 kB in a tarball, and 4 kB on a typical ZFS filesystem.

Note that if the target system is FreeBSD 14 or newer, you can simply
mount the tarball (`sudo mount -rt tarfs p.tar /usr/ports`).

DES
-- 
Dag-Erling Smørgrav - d...@freebsd.org

Re: something magic about the size of a ports tree

[Olivier Certner]

Hi Matthias,

Some ZFS dataset with zstd compression on jet, and no compression on 
c720-1400094?

-- 
Olivier Certner

something magic about the size of a ports tree

[Matthias Apitz]

I have on my poudriere build host a ports tree and wanted to move it to
the host where the resulting packages are installed:

root@jet:/usr/local/poudriere/ports # du -sh ports20230806
397Mports20230806
root@jet:/usr/local/poudriere/ports # tar cf p.tar ports20230806
root@jet:/usr/local/poudriere/ports # ls -lh p.tar
-rw-r--r--  1 root wheel  672M Oct  3 18:00 p.tar

already the size of the tar file is somewhat magic; but if you un-tar it
on the other host I will get:

[guru@c720-1400094 ~]$ ls -lh p.tar
-rw-r--r--  1 guru wheel  672M  3 oct.  18:00 p.tar
[guru@c720-1400094 ~]$ tar xf p.tar
[guru@c720-1400094 ~]$ du -sh ports20230806
1,2Gports20230806

How this is possible?

matthias

-- 
Matthias Apitz, ✉ g...@unixarea.de, http://www.unixarea.de/ +49-176-38902045
Public GnuPG key: http://www.unixarea.de/key.pub

Re: kernel 100% CPU, and ports-mgmt/poudriere-devel 'Inspecting ports tree for modifications to git checkout...' for an extraordinarily long time

[Graham Perrin]

On 03/09/2023 10:26, Michael Gmelin wrote:


On Sat, 2 Sep 2023 09:53:38 +0100
Graham Perrin  wrote:


Some inspections are extraordinarily time-consuming. Others complete
very quickly, as they should.

One recent inspection took more than half an hour.

Anyone else?


Does `git clone https://git.freebsd.org/ports.git` work for you?
(currently it's not working from where I am). Maybe related.

Best
Michael


Today, yes.

Sorry for not replying sooner, Gmail sent your 2nd September email to spam.

% pwd
/tmp
% time git clone https://git.freebsd.org/ports.git && rm -r ports
Cloning into 'ports'...
remote: Enumerating objects: 5943170, done.
remote: Counting objects: 100% (943/943), done.
remote: Compressing objects: 100% (127/127), done.
remote: Total 5943170 (delta 923), reused 816 (delta 816), pack-reused 
5942227

Receiving objects: 100% (5943170/5943170), 1.11 GiB | 6.29 MiB/s, done.
Resolving deltas: 100% (3586216/3586216), done.
Updating files: 100% (156931/156931), done.
941.630u 59.980s 10:11.66 163.7%    +442k 14+0io 58pf+16w
override r--r--r-- grahamperrin/wheel for 
ports/.git/objects/pack/pack-d72c55d78249720bb87ae380c69ecb3c6dc5fe94.idx? 
^C

% sudo rm -r /tmp/ports
grahamperrin's password:
%

Re: kernel 100% CPU, and ports-mgmt/poudriere-devel 'Inspecting ports tree for modifications to git checkout...' for an extraordinarily long time

[Michael Gmelin]

On Sat, 2 Sep 2023 09:53:38 +0100
Graham Perrin  wrote:

> Some inspections are extraordinarily time-consuming. Others complete 
> very quickly, as they should.
> 
> One recent inspection took more than half an hour.
> 
> Anyone else?
> 

Does `git clone https://git.freebsd.org/ports.git` work for you?
(currently it's not working from where I am). Maybe related.

Best
Michael


-- 
Michael Gmelin

Re: kernel 100% CPU, and ports-mgmt/poudriere-devel 'Inspecting ports tree for modifications to git checkout...' for an extraordinarily long time

[Mateusz Guzik]

On 9/2/23, Graham Perrin  wrote:
> On 02/09/2023 10:17, Mateusz Guzik wrote:
>> On 9/2/23, Graham Perrin  wrote:
>>> Some inspections are extraordinarily time-consuming. Others complete
>>> very quickly, as they should.
>>>
>>> One recent inspection took more than half an hour.
>>>
>>> Anyone else?
>>>
>>> A screenshot: 
>>>
>>> % pkg iinfo poudriere-devel
>>> poudriere-devel-3.3.99.20220831
>>> % uname -aKU
>>> FreeBSD mowa219-gjp4-8570p-freebsd 15.0-CURRENT FreeBSD 15.0-CURRENT
>>> amd64 150 #10 main-n265053-315ee00fa961-dirty: Mon Aug 28 06:22:31
>>> BST 2023
>>> grahamperrin@mowa219-gjp4-8570p-freebsd:/usr/obj/usr/src/amd64.amd64/sys/GENERIC-NODEBUG
>>>
>>> amd64 150 150
>>> %
>> get a flamegraph with dtrace
>>
>> https://github.com/brendangregg/FlameGraph
>
> Thanks! TIL, it's ported.
>
> Graph attached, as a PNG, although I don't know whether it'll be useful,
> because I began the trace /after/ the issue became observable.
>
> Will it be more meaningful to begin a trace and then reproduce the issue
> (before the trace ends)?
>
> 
>
> root@mowa219-gjp4-8570p-freebsd:/tmp # dtrace -x stackframes=100 -n
> 'profile-997 /arg0/ { @[stack()] = count(); } tick-60s { exit(0); }' -o
> out.kern_stacks
> dtrace: description 'profile-997 ' matched 2 probes
> root@mowa219-gjp4-8570p-freebsd:/tmp # stackcollapse.pl out.kern_stacks
>  > out.kern_folded
> root@mowa219-gjp4-8570p-freebsd:/tmp # flamegraph.pl out.kern_folded >
> kernel.svg
> root@mowa219-gjp4-8570p-freebsd:/tmp #
>

Looks like you have a lot of unrelated traffic in there.

Run this script:
#pragma D option dynvarsize=32m

profile:::profile-997
/execname == "find"/
{
@oncpu[stack(), "oncpu"] = count();
}

/*
 * The p_flag & 0x4 test filters out kernel threads.
 */

sched:::off-cpu
/execname == "find"/
{
self->ts = timestamp;
}

sched:::on-cpu
/self->ts/
{
@offcpu[stack(30), "offcpu"] = sum(timestamp - self->ts);
self->ts = 0;
}

dtrace:::END
{
normalize(@offcpu, 100);
printa("%k\n%s\n%@d\n\n", @offcpu);
printa("%k\n%s\n%@d\n\n", @oncpu);
}

dtrace -s script.d -o out

this can be fed to generate a flamegraph the same way; upload it to freefall

-- 
Mateusz Guzik 

Re: kernel 100% CPU, and ports-mgmt/poudriere-devel 'Inspecting ports tree for modifications to git checkout...' for an extraordinarily long time

[Alexander Motin]

On 02.09.2023 09:32, Graham Perrin wrote:

On 02/09/2023 10:17, Mateusz Guzik wrote:

get a flamegraph with dtrace

https://github.com/brendangregg/FlameGraph

See  for a 
PDF of a reply that probably did not reach the list.


Graham, the original SVG was scalable and searchable in browser.  Your 
PNG inside PDF is not.


--
Alexander Motin

Re: kernel 100% CPU, and ports-mgmt/poudriere-devel 'Inspecting ports tree for modifications to git checkout...' for an extraordinarily long time

[Graham Perrin]

On 02/09/2023 10:17, Mateusz Guzik wrote:

On 9/2/23, Graham Perrin  wrote:

Some inspections are extraordinarily time-consuming. Others complete
very quickly, as they should.

One recent inspection took more than half an hour.

Anyone else?

A screenshot: 

% pkg iinfo poudriere-devel
poudriere-devel-3.3.99.20220831
% uname -aKU
FreeBSD mowa219-gjp4-8570p-freebsd 15.0-CURRENT FreeBSD 15.0-CURRENT
amd64 150 #10 main-n265053-315ee00fa961-dirty: Mon Aug 28 06:22:31
BST 2023
grahamperrin@mowa219-gjp4-8570p-freebsd:/usr/obj/usr/src/amd64.amd64/sys/GENERIC-NODEBUG

amd64 150 150
%




get a flamegraph with dtrace

https://github.com/brendangregg/FlameGraph


Thanks!

See  for a 
PDF of a reply that probably did not reach the list.

Re: kernel 100% CPU, and ports-mgmt/poudriere-devel 'Inspecting ports tree for modifications to git checkout...' for an extraordinarily long time

[Mateusz Guzik]

On 9/2/23, Graham Perrin  wrote:
> Some inspections are extraordinarily time-consuming. Others complete
> very quickly, as they should.
>
> One recent inspection took more than half an hour.
>
> Anyone else?
>
> A screenshot: 
>
> % pkg iinfo poudriere-devel
> poudriere-devel-3.3.99.20220831
> % uname -aKU
> FreeBSD mowa219-gjp4-8570p-freebsd 15.0-CURRENT FreeBSD 15.0-CURRENT
> amd64 150 #10 main-n265053-315ee00fa961-dirty: Mon Aug 28 06:22:31
> BST 2023
> grahamperrin@mowa219-gjp4-8570p-freebsd:/usr/obj/usr/src/amd64.amd64/sys/GENERIC-NODEBUG
>
> amd64 150 150
> %
>
>
>

get a flamegraph with dtrace

https://github.com/brendangregg/FlameGraph

-- 
Mateusz Guzik 

kernel 100% CPU, and ports-mgmt/poudriere-devel 'Inspecting ports tree for modifications to git checkout...' for an extraordinarily long time

[Graham Perrin]

Some inspections are extraordinarily time-consuming. Others complete 
very quickly, as they should.


One recent inspection took more than half an hour.

Anyone else?

A screenshot: 

% pkg iinfo poudriere-devel
poudriere-devel-3.3.99.20220831
% uname -aKU
FreeBSD mowa219-gjp4-8570p-freebsd 15.0-CURRENT FreeBSD 15.0-CURRENT 
amd64 150 #10 main-n265053-315ee00fa961-dirty: Mon Aug 28 06:22:31 
BST 2023 
grahamperrin@mowa219-gjp4-8570p-freebsd:/usr/obj/usr/src/amd64.amd64/sys/GENERIC-NODEBUG 
amd64 150 150

%

Re: ports-mgmt/portconf – not src.conf(5) – to specify multiple flavours of a port for buildkernel purposes

[Graham Perrin]

On 22/08/2023 06:42, Graham Perrin wrote:


…
% cat /usr/local/etc/ports.conf
graphics/gpu-firmware-radeon-kmod: FLAVORS=btc sumo turks
%


Still, there's guesswork. I have /no/ idea whether the FLAVORS part of 
that last line is valid :-)


Time will tell.

/me tails /var/log/buildkernel.log, rocks with laughter at innumerable 
warnings, cancels,


…
make[4028]: "/usr/ports/Mk/bsd.port.mk" line 5393: warning: duplicate 
script for target "sumo" ignored
make[4028]: "/usr/ports/Mk/Uses/kmod.mk" line 74: warning: using 
previous script for "sumo" defined here
make[4028]: "/usr/ports/Mk/bsd.port.mk" line 5402: warning: duplicate 
script for target 
"/usr/obj/usr/src/amd64.amd64/sys/GENERIC/usr/ports/graphics/gpu-firmware-radeon-kmod/work-"btc" 
ignored
make[4028]: "/usr/ports/Mk/Uses/kmod.mk" line 74: warning: using 
previous script for 
"/usr/obj/usr/src/amd64.amd64/sys/GENERIC/usr/ports/graphics/gpu-firmware-radeon-kmod/work-"btc" 
defined here
make[4028]: "/usr/ports/Mk/bsd.port.mk" line 5402: warning: duplicate 
script for target "sumo" ignored
make[4028]: "/usr/ports/Mk/Uses/kmod.mk" line 74: warning: using 
previous script for "sumo" defined here

env: sumo: No such file or directory
^C
%


, begins a new build that excludes GPU firmware whilst ignoring the 
relevance of 
<https://github.com/freebsd/freebsd-src/commit/cedc82c0466a>, intends to 
build the modules with poudriere-devel after the OS is updated, 
rebelliously  begins a paragraph with a comma and makes a whitespace 
error that should be barely visible in the HTML version of this email 
that will not be seen in an archive :-)

ports-mgmt/portconf – not src.conf(5) – to specify multiple flavours of a port for buildkernel purposes

[Graham Perrin]

On 22/08/2023 05:34, Warner Losh wrote:


How might I use /etc/src.conf to achieve much the same, with a
different port?



I thought stuff like this went in ports.conf...

…


Warner solves another mystery. Thanks!

Honestly, I was oblivious to the possibility:

% man -P cat 5 ports.conf
No manual entry for ports.conf
% apropos ports.conf
apropos: nothing appropriate
% rg -i -e 'ports\.conf' /usr/doc/website/content/en
% rg -i -e 'ports\.conf' /usr/doc/documentation/content/en
%

– and so on. As far as I can tell, it's not documented in the usual places.

Eventually, Google helped to remind me of a 2021 comment 
<https://github.com/freebsd/poudriere/issues/835#issuecomment-962728743>, 
where part of the previous person's comment had never sunk in. I wrote:



(I never used ports-mgmt/portconf, and so on.)



So:


% gh repo sync grahamperrin/freebsd-ports && git -C /usr/ports pull 
--ff-only --quiet && git -C /usr/ports pull --ff-only freebsd main

✓Synced the "grahamperrin:main" branch from "freebsd:main"
Updating files: 100% (35/35), done.
From https://git.freebsd.org/ports
* branch  main   -> FETCH_HEAD
  15a5c70847f1..355374a1f6be  main   -> freebsd/main
Already up to date.
% sudo pkg install ports-mgmt/portconf
grahamperrin's password:
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
Updating poudriere repository catalogue...
Fetching meta.conf: 100%    163 B   0.2kB/s    00:01
Fetching packagesite.pkg: 100%    2 KiB   2.2kB/s    00:01
The provides database is up-to-date.
Processing entries: 100%
poudriere repository update completed. 7 packages processed.
All repositories are up to date.
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
   portconf: 1.6_1 [FreeBSD]

Number of packages to be installed: 1

2 KiB to be downloaded.

Proceed with this action? [y/N]: y
[1/1] Fetching portconf-1.6_1.pkg: 100%    2 KiB   2.3kB/s    00:01
Checking integrity... done (0 conflicting)
[1/1] Installing portconf-1.6_1...
[1/1] Extracting portconf-1.6_1: 100%
Spamming /etc/make.conf... Done.
=
Message from portconf-1.6_1:

--
To set port-specific make variables, create the
/usr/local/etc/ports.conf configuration file
with the following syntax:


# this is a comment
*: NOPORTDOCS
editors/openoffice-3: WITH_CCACHE|LOCALIZED_LANG=it
print/ghostscript-* print/lpr-wrapper: A4
sysutils/fusefs-kmod*: !KERNCONF | !NOPORTDOCS
www/firefox-i18n: WITHOUT_SWITCHER | FIREFOX_I18N=fr it
x11/fakeport: CONFIGURE_ARGS=--with-modules="aaa bbb ccc"


Global port directory patterns and blanks around the
pipe "|" symbol are allowed.
Values shouldn't be quoted even if they contain spaces.
Lines beginning with a '#' are comments.
% apropos ports.conf
apropos: nothing appropriate
% rg --count -e 'ports\.conf' /usr/ports
/usr/ports/net-mgmt/ocsinventory-ocsreports/files/pkg-message.in:1
/usr/ports/ports-mgmt/portconf/files/pkg-message.in:1
/usr/ports/ports-mgmt/portconf/files/portconf.sh.in:1
^C
% sudo nano /usr/local/etc/ports.conf
grahamperrin's password:
% cat /usr/local/etc/ports.conf
graphics/gpu-firmware-radeon-kmod: FLAVORS=btc sumo turks
%


Still, there's guesswork. I have /no/ idea whether the FLAVORS part of 
that last line is valid :-)


Time will tell.

Re: OpenSSL 3 ports fallout

[Mark Millard]

Gleb Popov  wrote on
Date: Sun, 13 Aug 2023 20:30:48 UTC :

> Some of the ports I'm using are failing to build after OpenSSL 3
> import due to the following problem. OpenSSL headers that are shipped
> in base contain declarations of various deprecated functions for which
> libcrypto.so doesn't contain definitions. Some of them are
> RSA_generate_key and ERR_* family. These declarations aren't guarded
> by any #ifdef and are visible for ports software. VirtualBox and
> net-p2p/cardano-node detect these functions, try to use them and then
> fail to link due to undefined references. I believe this should be
> fixed in the base rather than patching each port?


FreeBSD ports is using a 2021-08-24 version of cryptography/hazmat/ for
which there have long been more recent versions that no longer have
the kind of issue below:

  File 
"/usr/local/lib/python3.9/site-packages/cryptography/hazmat/bindings/openssl/binding.py",
 line 14, in 
from cryptography.hazmat.bindings._openssl import ffi, lib
ImportError: 
/usr/local/lib/python3.9/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so:
 Undefined symbol "ERR_GET_FUNC"

From what I wrote on the lists back on 2023-Jul-10:

QUOTE
In use: security/py-cryptography is at 3.4.8 (2021-08-24)
Vintage fixed: 35.0.0 of cryptography dates back to 2021-09-29.
Current for cryptography is 41.0.1 (2023-06-01).
END QUOTE

The full fix is inside cryptography, not in the environments that
use it (such as FreeBSD).

To me it looks like the tradeoffs now in place suggest adjusting
the constraints that are stopping progressing past 3.4.8 in ports,
even if there are some other consequences for some people.

Note: The above message is from a broken kyua test's backtrace. A
bunch of kyua testing involves use of python and ends up with
cryptography/hazmat/ involved (and, so, the python is broken in
some way).

===
Mark Millard
marklmi at yahoo.com

OpenSSL 3 ports fallout

[Gleb Popov]

Some of the ports I'm using are failing to build after OpenSSL 3
import due to the following problem. OpenSSL headers that are shipped
in base contain declarations of various deprecated functions for which
libcrypto.so doesn't contain definitions. Some of them are
RSA_generate_key and ERR_* family. These declarations aren't guarded
by any #ifdef and are visible for ports software. VirtualBox and
net-p2p/cardano-node detect these functions, try to use them and then
fail to link due to undefined references. I believe this should be
fixed in the base rather than patching each port?

CURRENT with ZFS not bootable when built with LLVM from ports

[Pero Orsolic]

Building -CURRENT (2023-07-17 be4c7f273508) without system compiler (with 
LLVM16 from ports) results in unbootable system:Preloaded elf kernel 
"/boot/kernel/kernel" at 0x82328000.
Preloaded elf obj module "/boot/kernel/zfs.ko" at 0x82329090.
Preloaded boot_entropy_cache "/boot/entropy" at 0x823298f8.
Preloaded elf obj module "/boot/kernel/cryptodev.ko" at 0x82329950.
Preloaded hostuuid "/etc/hostid" at 0x8232a140.
kldload: unexpected relocation type 42, symbol index 8662
link_elf_obj: symbol __stack_chk_guard undefined
KLD file zfs.ko - could not finalize loading

Patching and rebuilding LLVM16 with this patch results in bootable system:

Patch was taken from:
https://cgit.freebsd.org/src/commit/?h=stable/13&id=e8e5d75e6a9676e76c3bfd6d1d52561ffbb40846

Few months ago it was possible to use LLVM from ports to build kernel and 
world, but I don't remember the details.
This patch was tested in bhyve VM and on real hardware.

On real hardware it was possible to boot system without patched compiler when 
zfs.ko (options ZFS and GEOM_ELI with devices crypto and cryptodev) is built 
into the kernel.
But then other modules (acpi_ibm, iic, drm, i915kms, acpi_video, ... pf, 
various ng_* and so on) will fail to load with same reason.
Didn't try including zfs.ko into the kernel in the VM.

Bug report:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=272740

Re: Problem compiling py-* ports

[Filippo Moretti]

 After completing installworld and rebuilding python39 I do get the same error 
as reported previouslyFilippo

On Tuesday, April 18, 2023 at 12:54:33 PM UTC, Mateusz Guzik 
 wrote:  
 
 On 4/18/23, Filippo Moretti  wrote:
> Good morning,                      I run this versione of Frrebsd and al
> py-* ports fail with the following message.sincerelyFilippo
>
> FreeBSD STING 14.0-CURRENT FreeBSD 14.0-CURRENT #6
> main-n261981-63b113af5706: Tue Apr  4 16:57:47 CEST 2023
> filippo@STING:/usr/obj/usr/src/amd64.amd64/sys/STING amd64
>
>

you are on a zfs commit with known data corruption (in fact, 2)

bare minimum you need to update to fresh main and reinstall all python stuff

>
>
>
>    return _bootstrap._gcd_import(name[level:], package,
> level)
>    File "", line 1030, in
> _gcd_import
>
>  File "", line 1007, in
> _find_and_load
>
>  File "", line 972, in
> _find_and_load_unlocked
>
>  File "", line 228, in
> _call_with_frames_removed
>  File "", line 1030, in _gcd_import
>  File "", line 1007, in _find_and_load
>  File "", line 986, in
> _find_and_load_unlocked
>  File "", line 680, in _load_unlocked
>  File "", line 850, in exec_module
>  File "", line 228, in
> _call_with_frames_removed
>  File "/usr/local/lib/python3.9/site-packages/setuptools/__init__.py", line
> 18, in 
>    from setuptools.dist import Distribution
>  File "/usr/local/lib/python3.9/site-packages/setuptools/dist.py", line 34,
> in 
>    from ._importlib import metadata
>  File "/usr/local/lib/python3.9/site-packages/setuptools/_importlib.py",
> line 39, in 
>    disable_importlib_metadata_finder(metadata)
>  File "/usr/local/lib/python3.9/site-packages/setuptools/_importlib.py",
> line 28, in disable_importlib_metadata_finder
>    to_remove = [
>  File "/usr/local/lib/python3.9/site-packages/setuptools/_importlib.py",
> line 31, in 
>    if isinstance(ob, importlib_metadata.MetadataPathFinder)
> AttributeError: module 'importlib_metadata' has no attribute
> 'MetadataPathFinder'
>
> ERROR Backend subprocess exited when trying to invoke
> get_requires_for_build_wheel
> *** Error code 1
>
> Stop.
> make: stopped in /usr/ports/textproc/py-pygments
>
> ===>>> make build failed for textproc/py-pygments@py39
> ===>>> Aborting update
>
> ===>>> Update for textproc/py-pygments@py39 failed
> ===>>> Aborting update
>
>
> ===>>> You can restart from the point of failure with this command line:
>
>
>
>


-- 
Mateusz Guzik 

  

Re: Problem compiling py-* ports

[Mateusz Guzik]

On 4/18/23, Filippo Moretti  wrote:
> Good morning,   I run this versione of Frrebsd and al
> py-* ports fail with the following message.sincerelyFilippo
>
> FreeBSD STING 14.0-CURRENT FreeBSD 14.0-CURRENT #6
> main-n261981-63b113af5706: Tue Apr  4 16:57:47 CEST 2023
> filippo@STING:/usr/obj/usr/src/amd64.amd64/sys/STING amd64
>
>

you are on a zfs commit with known data corruption (in fact, 2)

bare minimum you need to update to fresh main and reinstall all python stuff

>
>
>
>return _bootstrap._gcd_import(name[level:], package,
> level)
>File "", line 1030, in
> _gcd_import
>
>   File "", line 1007, in
> _find_and_load
>
>   File "", line 972, in
> _find_and_load_unlocked
>
>   File "", line 228, in
> _call_with_frames_removed
>   File "", line 1030, in _gcd_import
>   File "", line 1007, in _find_and_load
>   File "", line 986, in
> _find_and_load_unlocked
>   File "", line 680, in _load_unlocked
>   File "", line 850, in exec_module
>   File "", line 228, in
> _call_with_frames_removed
>   File "/usr/local/lib/python3.9/site-packages/setuptools/__init__.py", line
> 18, in 
> from setuptools.dist import Distribution
>   File "/usr/local/lib/python3.9/site-packages/setuptools/dist.py", line 34,
> in 
> from ._importlib import metadata
>   File "/usr/local/lib/python3.9/site-packages/setuptools/_importlib.py",
> line 39, in 
> disable_importlib_metadata_finder(metadata)
>   File "/usr/local/lib/python3.9/site-packages/setuptools/_importlib.py",
> line 28, in disable_importlib_metadata_finder
> to_remove = [
>   File "/usr/local/lib/python3.9/site-packages/setuptools/_importlib.py",
> line 31, in 
> if isinstance(ob, importlib_metadata.MetadataPathFinder)
> AttributeError: module 'importlib_metadata' has no attribute
> 'MetadataPathFinder'
>
> ERROR Backend subprocess exited when trying to invoke
> get_requires_for_build_wheel
> *** Error code 1
>
> Stop.
> make: stopped in /usr/ports/textproc/py-pygments
>
> ===>>> make build failed for textproc/py-pygments@py39
> ===>>> Aborting update
>
> ===>>> Update for textproc/py-pygments@py39 failed
> ===>>> Aborting update
>
>
> ===>>> You can restart from the point of failure with this command line:
>
>
>
>


-- 
Mateusz Guzik 

Re: Problem compiling py-* ports

[Filippo Moretti]

 
After following the instructions provided I get the following:==>>> All >> 
py39-pygments-2.14.0 (1/9)

===>  Building for py39-pygments-2.15.0
/usr/local/bin/python3.9: No module named build
*** Error code 1

Stop.
make: stopped in /usr/ports/textproc/py-pygments

===>>> make build failed for textproc/py-pygments@py39
===>>> Aborting update

===>>> Update for textproc/py-pygments@py39 failed
===>>> Aborting update



On Tuesday, April 18, 2023 at 09:57:31 AM UTC, Felix Palmen 
 wrote:  
 
 * Marek Zarychta  [20230418 11:41]:
> What is the culprit?

Most likely some leftovers from older package versions (so it's only a
problem when building in the live system. A clean jail won't have the
offending files).

>                      Removing blindly all py- packages for all affected
> hosts with dependent software is not the best solution.

Depending on your situation, it might or might not be faster than trying
to identify the exact offending files and just removing them ;)

-- 
 Felix Palmen     {private}  fe...@palmen-it.de
 -- ports committer (mentee) --            {web}  http://palmen-it.de
 {pgp public key}  http://palmen-it.de/pub.txt
 {pgp fingerprint} 6936 13D5 5BBF 4837 B212  3ACC 54AD E006 9879 F231
  

Re: Problem compiling py-* ports

[Felix Palmen]

* Marek Zarychta  [20230418 11:41]:
> What is the culprit?

Most likely some leftovers from older package versions (so it's only a
problem when building in the live system. A clean jail won't have the
offending files).

>  Removing blindly all py- packages for all affected
> hosts with dependent software is not the best solution.

Depending on your situation, it might or might not be faster than trying
to identify the exact offending files and just removing them ;)

-- 
 Felix Palmen  {private}   fe...@palmen-it.de
 -- ports committer (mentee) --{web}  http://palmen-it.de
 {pgp public key}  http://palmen-it.de/pub.txt
 {pgp fingerprint} 6936 13D5 5BBF 4837 B212  3ACC 54AD E006 9879 F231


signature.asc
Description: PGP signature

Re: Problem compiling py-* ports

[Christos Chatzaras]

> On 18 Apr 2023, at 11:12, Filippo Moretti  wrote:
> 
> Good morning,
>I run this versione of Frrebsd and al py-* ports fail 
> with the following message.
> sincerely
> Filippo

Maybe this helps:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=269545#c6

Re: Problem compiling py-* ports

[Marek Zarychta]

W dniu 18.04.2023 o 10:46, Felix Palmen pisze:

* Filippo Moretti  [20230418 08:12]:

   File "/usr/local/lib/python3.9/site-packages/setuptools/_importlib.py", line 31, 
in 
     if isinstance(ob, importlib_metadata.MetadataPathFinder)
AttributeError: module 'importlib_metadata' has no attribute 
'MetadataPathFinder'
I am also hitting this bug with Python 3.8, so I recently considered an 
upgrade to 3.10, but it comes out that flaw is not version dependent.

Looks like you're building directly on your host (not in clean jails)
and the build picks up "something" that's already installed.
What is the culprit? Removing blindly all py- packages for all affected 
hosts with dependent software is not the best solution.


I'd probably try to uninstall all python packages, check whether there
are leftovers in LOCALBASE/lib/python3.9 (and remove them if necessary)
and then do a rebuild.

You can avoid this class of issues using poudriere btw...



--
Marek Zarychta

Re: Problem compiling py-* ports

[Felix Palmen]

* Filippo Moretti  [20230418 08:12]:
>   File "/usr/local/lib/python3.9/site-packages/setuptools/_importlib.py", 
> line 31, in 
>     if isinstance(ob, importlib_metadata.MetadataPathFinder)
> AttributeError: module 'importlib_metadata' has no attribute 
> 'MetadataPathFinder'

Looks like you're building directly on your host (not in clean jails)
and the build picks up "something" that's already installed.

I'd probably try to uninstall all python packages, check whether there
are leftovers in LOCALBASE/lib/python3.9 (and remove them if necessary)
and then do a rebuild.

You can avoid this class of issues using poudriere btw...

-- 
 Felix Palmen  {private}   fe...@palmen-it.de
 -- ports committer (mentee) --{web}  http://palmen-it.de
 {pgp public key}  http://palmen-it.de/pub.txt
 {pgp fingerprint} 6936 13D5 5BBF 4837 B212  3ACC 54AD E006 9879 F231


signature.asc
Description: PGP signature

Problem compiling py-* ports

[Filippo Moretti]

Good morning,   I run this versione of Frrebsd and al py-* 
ports fail with the following message.sincerelyFilippo

FreeBSD STING 14.0-CURRENT FreeBSD 14.0-CURRENT #6 main-n261981-63b113af5706: 
Tue Apr  4 16:57:47 CEST 2023 
filippo@STING:/usr/obj/usr/src/amd64.amd64/sys/STING amd64





   return _bootstrap._gcd_import(name[level:], package, level)  
      File "", line 1030, in _gcd_import   
     
  File "", line 1007, in _find_and_load
     
  File "", line 972, in _find_and_load_unlocked
     
  File "", line 228, in _call_with_frames_removed
  File "", line 1030, in _gcd_import
  File "", line 1007, in _find_and_load
  File "", line 986, in _find_and_load_unlocked
  File "", line 680, in _load_unlocked
  File "", line 850, in exec_module
  File "", line 228, in _call_with_frames_removed
  File "/usr/local/lib/python3.9/site-packages/setuptools/__init__.py", line 
18, in 
    from setuptools.dist import Distribution
  File "/usr/local/lib/python3.9/site-packages/setuptools/dist.py", line 34, in 

    from ._importlib import metadata
  File "/usr/local/lib/python3.9/site-packages/setuptools/_importlib.py", line 
39, in 
    disable_importlib_metadata_finder(metadata)
  File "/usr/local/lib/python3.9/site-packages/setuptools/_importlib.py", line 
28, in disable_importlib_metadata_finder
    to_remove = [
  File "/usr/local/lib/python3.9/site-packages/setuptools/_importlib.py", line 
31, in 
    if isinstance(ob, importlib_metadata.MetadataPathFinder)
AttributeError: module 'importlib_metadata' has no attribute 
'MetadataPathFinder'

ERROR Backend subprocess exited when trying to invoke 
get_requires_for_build_wheel
*** Error code 1

Stop.
make: stopped in /usr/ports/textproc/py-pygments

===>>> make build failed for textproc/py-pygments@py39
===>>> Aborting update

===>>> Update for textproc/py-pygments@py39 failed
===>>> Aborting update


===>>> You can restart from the point of failure with this command line:

Re: Unusual errors on recent stable/13 22-Dec-2022 (a related problem report on freebsd-ports?)

[Konstantin Belousov]

On Fri, Dec 23, 2022 at 03:47:33PM -0800, Mark Millard wrote:
> Jonathan Chen  wrote on
> Date: Fri, 23 Dec 2022 18:40:27 UTC :
> 
> > On 24/12/22 07:14, Mark Millard wrote:
> > > Jonathan Chen  wrote on
> > > Date: Thu, 22 Dec 2022 19:21:37 UTC :
> > > 
> > >> I recently updated my package builder machine to the
> > >> stable/13-n253297-fc15d5bf1109of (22-Dec-2022); and appear to be having
> > >> some unusual issues when building with a high number of jobs. My package
> > >> builder uses synth (which uses nullfs on ZFS), and I have had failures
> > >> with missing files, as well as what appears to be sequencing issues with
> > >> Makefiles. If I re-run the build, these errors usually do not reoccur.
> > >>
> > >> I'm puzzled as to what is happening. Is this just happening to me? It
> > >> appears that the ZFS code has been updated recently, and I'm wondering
> > >> whether a regression has crept in. [Or it could just be my hardware?]
> > > 
> > > 
> > > https://lists.freebsd.org/archives/freebsd-ports/2022-December/003153.html
> > > 
> > > indicates a problem with tmpfs use such that using USE_TMPFS=no
> > > avoids a problem for poudriere bulk builds on 13.1 amd64.
> > > (Unclear if the note is for stable/13 vs. releng/13.1 vs. both.)
> > 
> > I'll try disabling tmpfs on synth.
> > 
> 
> 
> FYI . . .
> 
> The following is about the tmpfs issue referenced in
> freebsd-ports/2022-December/003153.html .
> 
> Here is what is going on (manually entered commands, not a script). First 
> under a tmpfs:
> 
> # df -m
> Filesystem 1M-blocks Used Avail Capacity Mounted on
> /dev/ufs/rootfs 221683 97879 106068 48% /
> devfs 0 0 0 100% /dev
> /dev/msdosfs/MSDOSBOOT 49 31 18 62% /boot/msdos
> tmpfs 7716 0 7716 0% /tmp
> # cd /tmp
> # : > mmjnk.test
> # ls -Tld mmjnk.test
> -rw-r--r-- 1 root wheel 0 Mar 9 08:56:53 2022 mmjnk.test
> # : > mmjnk.test
> # ls -Tld mmjnk.test
> -rw-r--r-- 1 root wheel 0 Mar 9 08:56:53 2022 mmjnk.test
> 
> (no time change). The makefile involved is using ": > NAME" notation
> to try to update timestamps on deliberately empty files.
> 
> Vs. under (for example) UFS:
> 
> # cd ~/
> # : > mmjnk.test
> # ls -Tld mmjnk.test
> -rw-r--r-- 1 root wheel 0 Mar 9 09:00:45 2022 mmjnk.test
> # : > mmjnk.test
> # ls -Tld mmjnk.test
> -rw-r--r-- 1 root wheel 0 Mar 9 09:00:54 2022 mmjnk.test
> 
> (time changed).
> 
> Back in tmpfs land . . .
> 
> Part of this is that the file is already of size zero and continues
> to be so. By contrast, starting with a file with 15 bytes in it:
> 
> # ls -Tld mmjnk.test
> -rw-r--r-- 1 root wheel 15 Mar 9 09:07:38 2022 mmjnk.test
> # : > mmjnk.test
> # ls -Tld mmjnk.test
> -rw-r--r-- 1 root wheel 0 Mar 9 09:07:49 2022 mmjnk.test
> # ls -Tld mmjnk.test
> -rw-r--r-- 1 root wheel 0 Mar 9 09:07:49 2022 mmjnk.test
> 
> The lack of a timestamp change when the file already has size zero
> looks like an example of a bug to me.
> 
> truncate for tmpfs files behaves similarly (showing just the
> lack of timestamp change context):
> 
> # truncate -s 0 mmjnk.test
> # ls -Tld mmjnk.test
> -rw-r--r-- 1 root wheel 0 Mar 9 09:11:31 2022 mmjnk.test
> # truncate -s 0 mmjnk.test
> # ls -Tld mmjnk.test
> -rw-r--r-- 1 root wheel 0 Mar 9 09:11:31 2022 mmjnk.test
> 
> (UFS got a timestamp update from such a sequence.)
> 
> 
> I'll note that touch does not get this tmpfs behavior:
> 
> # touch mmjnk.test
> # ls -Tld mmjnk.test
> -rw-r--r-- 1 root wheel 0 Mar 9 09:11:26 2022 mmjnk.test
> # touch mmjnk.test
> # ls -Tld mmjnk.test
> -rw-r--r-- 1 root wheel 0 Mar 9 09:11:31 2022 mmjnk.test
> 
> (But it would not force size zero on its down.)
> 
> I did these tests on:
> 
> # uname -apKU
> FreeBSD generic 13.1-STABLE FreeBSD 13.1-STABLE #0 
> stable/13-n253133-b51ee7ac252c: Wed Nov 23 03:36:16 UTC 2022 
> r...@releng3.nyi.freebsd.org:/usr/obj/usr/src/arm64.aarch64/sys/GENERIC arm64 
> aarch64 1301509 1301509
> 
> However, I previously did a devel/nasm bulk test with with USE_TMPFS=all on:
> 
> # uname -apKU
> FreeBSD amd64_ZFS 14.0-CURRENT FreeBSD 14.0-CURRENT #55 
> main-n259064-f83db6441a2f-dirty: Sun Nov 6 16:31:55 PST 2022 
> root@amd64_ZFS:/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG
>  amd64 amd64 1400073 1400073
> 
> and it got the problem. (I normally use USE_TMPFS=data , which does not
> get the problem because the files in question end up not on a tmpfs.)
> 
> So: not specific to amd64 , not specific to stable/13 , existed in early
> November in main. This may have been around for some time for tmpfs.

Should be fixed by https://reviews.freebsd.org/D37866

Re: Unusual errors on recent stable/13 22-Dec-2022 (a related problem report on freebsd-ports?)

[Mark Millard]

Jonathan Chen  wrote on
Date: Fri, 23 Dec 2022 18:40:27 UTC :

> On 24/12/22 07:14, Mark Millard wrote:
> > Jonathan Chen  wrote on
> > Date: Thu, 22 Dec 2022 19:21:37 UTC :
> > 
> >> I recently updated my package builder machine to the
> >> stable/13-n253297-fc15d5bf1109of (22-Dec-2022); and appear to be having
> >> some unusual issues when building with a high number of jobs. My package
> >> builder uses synth (which uses nullfs on ZFS), and I have had failures
> >> with missing files, as well as what appears to be sequencing issues with
> >> Makefiles. If I re-run the build, these errors usually do not reoccur.
> >>
> >> I'm puzzled as to what is happening. Is this just happening to me? It
> >> appears that the ZFS code has been updated recently, and I'm wondering
> >> whether a regression has crept in. [Or it could just be my hardware?]
> > 
> > 
> > https://lists.freebsd.org/archives/freebsd-ports/2022-December/003153.html
> > 
> > indicates a problem with tmpfs use such that using USE_TMPFS=no
> > avoids a problem for poudriere bulk builds on 13.1 amd64.
> > (Unclear if the note is for stable/13 vs. releng/13.1 vs. both.)
> 
> I'll try disabling tmpfs on synth.
> 


FYI . . .

The following is about the tmpfs issue referenced in
freebsd-ports/2022-December/003153.html .

Here is what is going on (manually entered commands, not a script). First under 
a tmpfs:

# df -m
Filesystem 1M-blocks Used Avail Capacity Mounted on
/dev/ufs/rootfs 221683 97879 106068 48% /
devfs 0 0 0 100% /dev
/dev/msdosfs/MSDOSBOOT 49 31 18 62% /boot/msdos
tmpfs 7716 0 7716 0% /tmp
# cd /tmp
# : > mmjnk.test
# ls -Tld mmjnk.test
-rw-r--r-- 1 root wheel 0 Mar 9 08:56:53 2022 mmjnk.test
# : > mmjnk.test
# ls -Tld mmjnk.test
-rw-r--r-- 1 root wheel 0 Mar 9 08:56:53 2022 mmjnk.test

(no time change). The makefile involved is using ": > NAME" notation
to try to update timestamps on deliberately empty files.

Vs. under (for example) UFS:

# cd ~/
# : > mmjnk.test
# ls -Tld mmjnk.test
-rw-r--r-- 1 root wheel 0 Mar 9 09:00:45 2022 mmjnk.test
# : > mmjnk.test
# ls -Tld mmjnk.test
-rw-r--r-- 1 root wheel 0 Mar 9 09:00:54 2022 mmjnk.test

(time changed).

Back in tmpfs land . . .

Part of this is that the file is already of size zero and continues
to be so. By contrast, starting with a file with 15 bytes in it:

# ls -Tld mmjnk.test
-rw-r--r-- 1 root wheel 15 Mar 9 09:07:38 2022 mmjnk.test
# : > mmjnk.test
# ls -Tld mmjnk.test
-rw-r--r-- 1 root wheel 0 Mar 9 09:07:49 2022 mmjnk.test
# ls -Tld mmjnk.test
-rw-r--r-- 1 root wheel 0 Mar 9 09:07:49 2022 mmjnk.test

The lack of a timestamp change when the file already has size zero
looks like an example of a bug to me.

truncate for tmpfs files behaves similarly (showing just the
lack of timestamp change context):

# truncate -s 0 mmjnk.test
# ls -Tld mmjnk.test
-rw-r--r-- 1 root wheel 0 Mar 9 09:11:31 2022 mmjnk.test
# truncate -s 0 mmjnk.test
# ls -Tld mmjnk.test
-rw-r--r-- 1 root wheel 0 Mar 9 09:11:31 2022 mmjnk.test

(UFS got a timestamp update from such a sequence.)


I'll note that touch does not get this tmpfs behavior:

# touch mmjnk.test
# ls -Tld mmjnk.test
-rw-r--r-- 1 root wheel 0 Mar 9 09:11:26 2022 mmjnk.test
# touch mmjnk.test
# ls -Tld mmjnk.test
-rw-r--r-- 1 root wheel 0 Mar 9 09:11:31 2022 mmjnk.test

(But it would not force size zero on its down.)

I did these tests on:

# uname -apKU
FreeBSD generic 13.1-STABLE FreeBSD 13.1-STABLE #0 
stable/13-n253133-b51ee7ac252c: Wed Nov 23 03:36:16 UTC 2022 
r...@releng3.nyi.freebsd.org:/usr/obj/usr/src/arm64.aarch64/sys/GENERIC arm64 
aarch64 1301509 1301509

However, I previously did a devel/nasm bulk test with with USE_TMPFS=all on:

# uname -apKU
FreeBSD amd64_ZFS 14.0-CURRENT FreeBSD 14.0-CURRENT #55 
main-n259064-f83db6441a2f-dirty: Sun Nov 6 16:31:55 PST 2022 
root@amd64_ZFS:/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG
 amd64 amd64 1400073 1400073

and it got the problem. (I normally use USE_TMPFS=data , which does not
get the problem because the files in question end up not on a tmpfs.)

So: not specific to amd64 , not specific to stable/13 , existed in early
November in main. This may have been around for some time for tmpfs.

===
Mark Millard
marklmi at yahoo.com

Re: Has anyone tried to build QEMU from ports lately?

[Dennis Clarke]

On 10/11/22 04:20, Dennis Clarke wrote:

On 10/10/22 17:53, Warner Losh wrote:

On Mon, Oct 10, 2022 at 10:56 AM Warner Losh  wrote:


I know what's causing this problem. I'll resolve.

tl/dr: _pv_entry.h depends on sys/param.h being included before its use.



https://reviews.freebsd.org/D36927 fixes it by making sys/_pv_entry.h 
more

self-contained and less reliant on param.h pollution. The kernel includes
that
everywhere it's used, but userland is more hit or miss because
machine/pmap.h
isn't a well defined interface, but is needed for some things sometimes.



I am not sure where this is related, however, there is a change in QEMU:


https://github.com/qemu/qemu/commit/a4a9a4432e2bf280a989ca344466d7375db7993f 




Which seems to provide a way around some really long ISA cpu name data 
that gets caught in sys/riscv/riscv/identcpu.c at around line 113 or

so :




Forgot to mention :

https://github.com/qemu/qemu/blob/master/hw/riscv/virt.c#L248

I wonder what the string is that tossed the KASSERT.



--
Dennis Clarke
RISC-V/SPARC/PPC/ARM/CISC
UNIX and Linux spoken
GreyBeard and suspenders optional

Re: Has anyone tried to build QEMU from ports lately?

[Dennis Clarke]

 | yo`:.:o 
   `+-
 |  1. Boot Multi user [Enter] |  y/   -/` 
  -o/
 |  2. Boot Single user| .- 
::/sy+:.
 |  3. Escape to loader prompt | / 
`--  /
 |  4. Reboot  |`: 
 :`
 |  5. Cons: Video |`: 
 :`
 | | / 
 /
 |  Options:   | .- 
-.
 |  6. Kernel: default/kernel (1 of 1) |  -- 
   -.

 |  7. Boot Options|   `:`  `:`
 | | .-- `--.
 | |.---..
 +-+
   Autoboot in 0 seconds. [Space] to pause
Loading kernel...
/boot/kernel/kernel text=0x5b3a88 text=0x175c2c data=0xf4208 
data=0x1f74+0x2738dc 0x8+0x1f46fb8+0x8+0xf3f1e

Loading configured modules...
can't find '/etc/hostid'
can't find '/boot/entropy'
Using DTB provided by EFI at 0x87efb000.
Kernel entry at 0xf680002e...
Kernel args: (null)
---<>---
GDB: no debug ports present
KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2022 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 14.0-CURRENT #0 main-n258483-b05b1ecbef0: Fri Oct  7 05:52:47 
UTC 2022


r...@releng1.nyi.freebsd.org:/usr/obj/usr/src/riscv.riscv64/sys/GENERIC 
riscv
FreeBSD clang version 14.0.5 (https://github.com/llvm/llvm-project.git 
llvmorg-14.0.5-0-gc12386ae247c)

WARNING: WITNESS option enabled, expect reduced performance.
VT: init without driver.
SBI: OpenSBI v1.1
SBI Specification Version: 1.0
CPU(0): Unknown Implementer Unknown Processor
real memory  = 34359738368 (32768 MB)
avail memory = 33401937920 (31854 MB)
Starting CPU 1 (hart 0)
Starting CPU 2 (hart 1)
Starting CPU 3 (hart 2)
Starting CPU 4 (hart 3)
Starting CPU 5 (hart 4)
Starting CPU 6 (hart 5)
Starting CPU 7 (hart 6)
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
panic: ISA string truncated
cpuid = 0
time = 1
KDB: stack backtrace:
db_trace_self() at db_trace_self
db_trace_self_wrapper() at db_trace_self_wrapper+0x38
kdb_backtrace() at kdb_backtrace+0x2c
vpanic() at vpanic+0x126
panic() at panic+0x2a
fill_elf_hwcap() at fill_elf_hwcap+0x220
mi_startup() at mi_startup+0x1d2
va() at va+0x7e
KDB: enter: panic
[ thread pid 0 tid 0 ]
Stopped at  kdb_enter+0x4a: sd  zero,0(s1)
db> halt
qemu-system-riscv64: terminating on signal 1 from pid 2765 (csh)
sedna$

due to :

/*
 * Iterate through the CPUs and examine their ISA string. While we
 * could assign elf_hwcap to be whatever the boot CPU supports, to
 * handle the (unusual) case of running a system with hetergeneous
 * ISAs, keep only the extension bits that are common to all harts.
 */
for (node = OF_child(node); node > 0; node = OF_peer(node)) {
/* Skip any non-CPU nodes, such as cpu-map. */
if (!ofw_bus_node_is_compatible(node, "riscv"))
continue;

len = OF_getprop(node, "riscv,isa", isa, sizeof(isa));
KASSERT(len <= ISA_NAME_MAXLEN, ("ISA string truncated"));
if (len == -1) {
if (bootverbose)
printf("fill_elf_hwcap: "
"Can't find riscv,isa property\n");
return;
} else if (strncmp(isa, ISA_PREFIX, ISA_PREFIX_LEN) != 0) {
if (bootverbose)
printf("fill_elf_hwcap: "
"Unsupported ISA string: %s\n", isa);
return;
}


again in sys/riscv/riscv/identcpu.c

However this seems to work :

sedna$ /usr/local/bin/qemu-system-riscv64 \
> -machine virt -m 32768M -smp 8 -cpu rv64,short-isa-string=on \
> -nographic \
> -bios /usr/local/share/opensbi/lp64/generic/firmware/fw_jump.elf \
> -kernel /usr/local/share/u-boot/u-boot-qemu-riscv64/u-boot.bin \
> -drive 
file=/opt/qemu/riscv/fbsd14/fbsd14_rv64imafdc.qcow2,format=qcow2,id=hd0 \

> -device virtio-blk-device,drive=hd0 \
> -object rng-random,filename=/dev/urandom,id=rng0 \
> -device virtio-rng-device,rng=rng0 \
> -device virtio-net-device,netdev=usernet \
> -netdev user,id=usernet,hostfwd=tcp::1-:22


OpenSBI v1.1
   _  _
  / __ \  / |  _ \_   _|
 | |  | |_ __   ___ _ __ | (___ | |_) || |
 | |  | | '_ \ / _ \ '_ \ \___ \|  _ < | |
 | |__| | |_) 

Re: Has anyone tried to build QEMU from ports lately?

[Warner Losh]

On Mon, Oct 10, 2022 at 10:56 AM Warner Losh  wrote:

> I know what's causing this problem. I'll resolve.
>
> tl/dr: _pv_entry.h depends on sys/param.h being included before its use.
>

https://reviews.freebsd.org/D36927 fixes it by making sys/_pv_entry.h more
self-contained and less reliant on param.h pollution. The kernel includes
that
everywhere it's used, but userland is more hit or miss because
machine/pmap.h
isn't a well defined interface, but is needed for some things sometimes.

Warner


> Warner
>
> On Mon, Oct 10, 2022 at 4:38 AM Dennis Clarke 
> wrote:
>
>>
>> On :
>>
>> phobos#
>> phobos# uname -apKU
>> FreeBSD phobos 14.0-CURRENT FreeBSD 14.0-CURRENT #14
>> main-n258340-497cdf9673e: Sun Oct  2 09:51:14 GMT 2022
>> root@phobos:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64 amd64 1400072
>> 1400072
>> phobos#
>>
>>
>> Seems to fail in a pretty consistent fashion :
>>
>> .
>> .
>> .
>> [5633/6772] Compiling C object libqemu-arm-bsd-user.fa.p/bsd-user_mmap.c.o
>> [5634/6772] Compiling C object
>> libqemu-arm-bsd-user.fa.p/bsd-user_signal.c.o
>> FAILED: libqemu-arm-bsd-user.fa.p/bsd-user_signal.c.o
>> cc -m64 -mcx16 -Ilibqemu-arm-bsd-user.fa.p -I. -I.. -Itarget/arm
>> -I../target/arm -I../common-user/host/x86_64 -I../bsd-user/include
>> -Ibsd-user/freebsd -I../bsd-user/freebsd -I../bsd-user/host/x86_64
>> -Ibsd-user -I../bsd-user -I../bsd-user/arm -Iqapi -Itrace -Iui
>> -Iui/shader -I/usr/local/include/capstone
>> -I/usr/ports/emulators/qemu-devel/work/qemu-09ed077d7fae5f825e18ff9a2004dcdd1b165edb
>>
>> -I/usr/local/include -I/usr/local/include/glib-2.0
>> -I/usr/local/lib/glib-2.0/include -fcolor-diagnostics -Wall
>> -Winvalid-pch -std=gnu11 -O0 -g -iquote . -iquote
>> /usr/ports/emulators/qemu-devel/work/qemu-09ed077d7fae5f825e18ff9a2004dcdd1b165edb
>>
>> -iquote
>> /usr/ports/emulators/qemu-devel/work/qemu-09ed077d7fae5f825e18ff9a2004dcdd1b165edb/include
>>
>> -iquote
>> /usr/ports/emulators/qemu-devel/work/qemu-09ed077d7fae5f825e18ff9a2004dcdd1b165edb/tcg/i386
>>
>> -pthread -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE
>> -Wstrict-prototypes -Wredundant-decls -Wundef -Wwrite-strings
>> -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv
>> -Wold-style-definition -Wtype-limits -Wformat-security -Wformat-y2k
>> -Winit-self -Wignored-qualifiers -Wempty-body -Wnested-externs
>> -Wendif-labels -Wexpansion-to-defined -Wno-initializer-overrides
>> -Wno-missing-include-dirs -Wno-shift-negative-value -Wno-string-plus-int
>> -Wno-typedef-redefinition -Wno-tautological-type-limit-compare
>> -Wno-psabi -fstack-protector-strong -O2 -pipe -fstack-protector-strong
>> -fno-strict-aliasing '-DPREFIX=\""/usr/local\""' -fPIE -DNEED_CPU_H
>> '-DCONFIG_TARGET="arm-bsd-user-config-target.h"'
>> '-DCONFIG_DEVICES="arm-bsd-user-config-devices.h"' -MD -MQ
>> libqemu-arm-bsd-user.fa.p/bsd-user_signal.c.o -MF
>> libqemu-arm-bsd-user.fa.p/bsd-user_signal.c.o.d -o
>> libqemu-arm-bsd-user.fa.p/bsd-user_signal.c.o -c ../bsd-user/signal.c
>> In file included from ../bsd-user/signal.c:27:
>> In file included from ../bsd-user/host/x86_64/host-signal.h:14:
>> In file included from /usr/include/vm/pmap.h:92:
>> /usr/include/machine/pmap.h:452:2: error: fields must have a constant
>> size: 'variable length array in structure' extension will never be
>> supported
>>  PV_CHUNK_HEADER
>>  ^
>> /usr/include/machine/pmap.h:448:12: note: expanded from macro
>> 'PV_CHUNK_HEADER'
>>  uint64_tpc_map[_NPCM];  /* bitmap; 1 = free */  \
>>  ^
>> /usr/include/machine/pmap.h:456:2: error: fields must have a constant
>> size: 'variable length array in structure' extension will never be
>> supported
>>  PV_CHUNK_HEADER
>>  ^
>> /usr/include/machine/pmap.h:448:12: note: expanded from macro
>> 'PV_CHUNK_HEADER'
>>  uint64_tpc_map[_NPCM];  /* bitmap; 1 = free */  \
>>  ^
>> 2 errors generated.
>> ninja: build stopped: subcommand failed.
>> gmake[3]: *** [Makefile:162: run-ninja] Error 1
>> gmake[3]: Leaving directory
>>
>> '/usr/ports/emulators/qemu-devel/work/qemu-09ed077d7fae5f825e18ff9a2004dcdd1b165edb/build'
>> gmake[2]: *** [GNUmakefile:11: all] Error 2
>> gmake[2]: Leaving directory
>>
>> '/usr/ports/emulators/qemu-devel/work/qemu-09ed077d7fae5f825e18ff9a2004dcdd1b165edb'
>> *** Error code 2
>>
>> Stop.
>> make[1]: stopped in /usr/ports/emulators/qemu-devel
>> *** Error code 1
>>
>> Stop.
>> make: stopped in /usr/ports/emulators/qemu-devel
>> phobos#
>> phobos# pwd
>> /usr/ports/emulators/qemu-devel
>> phobos#
>>
>> Possibly a problem with -linotify or who knows what ?
>>
>> Also see :
>>
>>https://lists.nongnu.org/archive/html/qemu-devel/2022-10/msg01280.html
>>
>> In any case, something feels wrong here.
>>
>>
>> --
>> Dennis Clarke
>> RISC-V/SPARC/PPC/ARM/CISC
>> UNIX and Linux spoken
>> GreyBeard and suspenders optional
>>
>>

Re: Has anyone tried to build QEMU from ports lately?

[Warner Losh]

I know what's causing this problem. I'll resolve.

tl/dr: _pv_entry.h depends on sys/param.h being included before its use.

Warner

On Mon, Oct 10, 2022 at 4:38 AM Dennis Clarke  wrote:

>
> On :
>
> phobos#
> phobos# uname -apKU
> FreeBSD phobos 14.0-CURRENT FreeBSD 14.0-CURRENT #14
> main-n258340-497cdf9673e: Sun Oct  2 09:51:14 GMT 2022
> root@phobos:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64 amd64 1400072
> 1400072
> phobos#
>
>
> Seems to fail in a pretty consistent fashion :
>
> .
> .
> .
> [5633/6772] Compiling C object libqemu-arm-bsd-user.fa.p/bsd-user_mmap.c.o
> [5634/6772] Compiling C object
> libqemu-arm-bsd-user.fa.p/bsd-user_signal.c.o
> FAILED: libqemu-arm-bsd-user.fa.p/bsd-user_signal.c.o
> cc -m64 -mcx16 -Ilibqemu-arm-bsd-user.fa.p -I. -I.. -Itarget/arm
> -I../target/arm -I../common-user/host/x86_64 -I../bsd-user/include
> -Ibsd-user/freebsd -I../bsd-user/freebsd -I../bsd-user/host/x86_64
> -Ibsd-user -I../bsd-user -I../bsd-user/arm -Iqapi -Itrace -Iui
> -Iui/shader -I/usr/local/include/capstone
> -I/usr/ports/emulators/qemu-devel/work/qemu-09ed077d7fae5f825e18ff9a2004dcdd1b165edb
>
> -I/usr/local/include -I/usr/local/include/glib-2.0
> -I/usr/local/lib/glib-2.0/include -fcolor-diagnostics -Wall
> -Winvalid-pch -std=gnu11 -O0 -g -iquote . -iquote
> /usr/ports/emulators/qemu-devel/work/qemu-09ed077d7fae5f825e18ff9a2004dcdd1b165edb
>
> -iquote
> /usr/ports/emulators/qemu-devel/work/qemu-09ed077d7fae5f825e18ff9a2004dcdd1b165edb/include
>
> -iquote
> /usr/ports/emulators/qemu-devel/work/qemu-09ed077d7fae5f825e18ff9a2004dcdd1b165edb/tcg/i386
>
> -pthread -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE
> -Wstrict-prototypes -Wredundant-decls -Wundef -Wwrite-strings
> -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv
> -Wold-style-definition -Wtype-limits -Wformat-security -Wformat-y2k
> -Winit-self -Wignored-qualifiers -Wempty-body -Wnested-externs
> -Wendif-labels -Wexpansion-to-defined -Wno-initializer-overrides
> -Wno-missing-include-dirs -Wno-shift-negative-value -Wno-string-plus-int
> -Wno-typedef-redefinition -Wno-tautological-type-limit-compare
> -Wno-psabi -fstack-protector-strong -O2 -pipe -fstack-protector-strong
> -fno-strict-aliasing '-DPREFIX=\""/usr/local\""' -fPIE -DNEED_CPU_H
> '-DCONFIG_TARGET="arm-bsd-user-config-target.h"'
> '-DCONFIG_DEVICES="arm-bsd-user-config-devices.h"' -MD -MQ
> libqemu-arm-bsd-user.fa.p/bsd-user_signal.c.o -MF
> libqemu-arm-bsd-user.fa.p/bsd-user_signal.c.o.d -o
> libqemu-arm-bsd-user.fa.p/bsd-user_signal.c.o -c ../bsd-user/signal.c
> In file included from ../bsd-user/signal.c:27:
> In file included from ../bsd-user/host/x86_64/host-signal.h:14:
> In file included from /usr/include/vm/pmap.h:92:
> /usr/include/machine/pmap.h:452:2: error: fields must have a constant
> size: 'variable length array in structure' extension will never be
> supported
>  PV_CHUNK_HEADER
>  ^
> /usr/include/machine/pmap.h:448:12: note: expanded from macro
> 'PV_CHUNK_HEADER'
>  uint64_tpc_map[_NPCM];  /* bitmap; 1 = free */  \
>  ^
> /usr/include/machine/pmap.h:456:2: error: fields must have a constant
> size: 'variable length array in structure' extension will never be
> supported
>  PV_CHUNK_HEADER
>  ^
> /usr/include/machine/pmap.h:448:12: note: expanded from macro
> 'PV_CHUNK_HEADER'
>  uint64_tpc_map[_NPCM];  /* bitmap; 1 = free */  \
>  ^
> 2 errors generated.
> ninja: build stopped: subcommand failed.
> gmake[3]: *** [Makefile:162: run-ninja] Error 1
> gmake[3]: Leaving directory
>
> '/usr/ports/emulators/qemu-devel/work/qemu-09ed077d7fae5f825e18ff9a2004dcdd1b165edb/build'
> gmake[2]: *** [GNUmakefile:11: all] Error 2
> gmake[2]: Leaving directory
>
> '/usr/ports/emulators/qemu-devel/work/qemu-09ed077d7fae5f825e18ff9a2004dcdd1b165edb'
> *** Error code 2
>
> Stop.
> make[1]: stopped in /usr/ports/emulators/qemu-devel
> *** Error code 1
>
> Stop.
> make: stopped in /usr/ports/emulators/qemu-devel
> phobos#
> phobos# pwd
> /usr/ports/emulators/qemu-devel
> phobos#
>
> Possibly a problem with -linotify or who knows what ?
>
> Also see :
>
>https://lists.nongnu.org/archive/html/qemu-devel/2022-10/msg01280.html
>
> In any case, something feels wrong here.
>
>
> --
> Dennis Clarke
> RISC-V/SPARC/PPC/ARM/CISC
> UNIX and Linux spoken
> GreyBeard and suspenders optional
>
>

Has anyone tried to build QEMU from ports lately?

[Dennis Clarke]

On :

phobos#
phobos# uname -apKU
FreeBSD phobos 14.0-CURRENT FreeBSD 14.0-CURRENT #14 
main-n258340-497cdf9673e: Sun Oct  2 09:51:14 GMT 2022 
root@phobos:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64 amd64 1400072 
1400072

phobos#


Seems to fail in a pretty consistent fashion :

.
.
.
[5633/6772] Compiling C object libqemu-arm-bsd-user.fa.p/bsd-user_mmap.c.o
[5634/6772] Compiling C object libqemu-arm-bsd-user.fa.p/bsd-user_signal.c.o
FAILED: libqemu-arm-bsd-user.fa.p/bsd-user_signal.c.o
cc -m64 -mcx16 -Ilibqemu-arm-bsd-user.fa.p -I. -I.. -Itarget/arm 
-I../target/arm -I../common-user/host/x86_64 -I../bsd-user/include 
-Ibsd-user/freebsd -I../bsd-user/freebsd -I../bsd-user/host/x86_64 
-Ibsd-user -I../bsd-user -I../bsd-user/arm -Iqapi -Itrace -Iui 
-Iui/shader -I/usr/local/include/capstone 
-I/usr/ports/emulators/qemu-devel/work/qemu-09ed077d7fae5f825e18ff9a2004dcdd1b165edb 
-I/usr/local/include -I/usr/local/include/glib-2.0 
-I/usr/local/lib/glib-2.0/include -fcolor-diagnostics -Wall 
-Winvalid-pch -std=gnu11 -O0 -g -iquote . -iquote 
/usr/ports/emulators/qemu-devel/work/qemu-09ed077d7fae5f825e18ff9a2004dcdd1b165edb 
-iquote 
/usr/ports/emulators/qemu-devel/work/qemu-09ed077d7fae5f825e18ff9a2004dcdd1b165edb/include 
-iquote 
/usr/ports/emulators/qemu-devel/work/qemu-09ed077d7fae5f825e18ff9a2004dcdd1b165edb/tcg/i386 
-pthread -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE 
-Wstrict-prototypes -Wredundant-decls -Wundef -Wwrite-strings 
-Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv 
-Wold-style-definition -Wtype-limits -Wformat-security -Wformat-y2k 
-Winit-self -Wignored-qualifiers -Wempty-body -Wnested-externs 
-Wendif-labels -Wexpansion-to-defined -Wno-initializer-overrides 
-Wno-missing-include-dirs -Wno-shift-negative-value -Wno-string-plus-int 
-Wno-typedef-redefinition -Wno-tautological-type-limit-compare 
-Wno-psabi -fstack-protector-strong -O2 -pipe -fstack-protector-strong 
-fno-strict-aliasing '-DPREFIX=\""/usr/local\""' -fPIE -DNEED_CPU_H 
'-DCONFIG_TARGET="arm-bsd-user-config-target.h"' 
'-DCONFIG_DEVICES="arm-bsd-user-config-devices.h"' -MD -MQ 
libqemu-arm-bsd-user.fa.p/bsd-user_signal.c.o -MF 
libqemu-arm-bsd-user.fa.p/bsd-user_signal.c.o.d -o 
libqemu-arm-bsd-user.fa.p/bsd-user_signal.c.o -c ../bsd-user/signal.c

In file included from ../bsd-user/signal.c:27:
In file included from ../bsd-user/host/x86_64/host-signal.h:14:
In file included from /usr/include/vm/pmap.h:92:
/usr/include/machine/pmap.h:452:2: error: fields must have a constant 
size: 'variable length array in structure' extension will never be supported

PV_CHUNK_HEADER
^
/usr/include/machine/pmap.h:448:12: note: expanded from macro 
'PV_CHUNK_HEADER'

uint64_tpc_map[_NPCM];  /* bitmap; 1 = free */  \
^
/usr/include/machine/pmap.h:456:2: error: fields must have a constant 
size: 'variable length array in structure' extension will never be supported

PV_CHUNK_HEADER
^
/usr/include/machine/pmap.h:448:12: note: expanded from macro 
'PV_CHUNK_HEADER'

uint64_tpc_map[_NPCM];  /* bitmap; 1 = free */  \
^
2 errors generated.
ninja: build stopped: subcommand failed.
gmake[3]: *** [Makefile:162: run-ninja] Error 1
gmake[3]: Leaving directory 
'/usr/ports/emulators/qemu-devel/work/qemu-09ed077d7fae5f825e18ff9a2004dcdd1b165edb/build'

gmake[2]: *** [GNUmakefile:11: all] Error 2
gmake[2]: Leaving directory 
'/usr/ports/emulators/qemu-devel/work/qemu-09ed077d7fae5f825e18ff9a2004dcdd1b165edb'

*** Error code 2

Stop.
make[1]: stopped in /usr/ports/emulators/qemu-devel
*** Error code 1

Stop.
make: stopped in /usr/ports/emulators/qemu-devel
phobos#
phobos# pwd
/usr/ports/emulators/qemu-devel
phobos#

Possibly a problem with -linotify or who knows what ?

Also see :

  https://lists.nongnu.org/archive/html/qemu-devel/2022-10/msg01280.html

In any case, something feels wrong here.


--
Dennis Clarke
RISC-V/SPARC/PPC/ARM/CISC
UNIX and Linux spoken
GreyBeard and suspenders optional

Re: Header symbols that shouldn't be visible to ports?

[David Chisnall]

On 7 Sep 2022, at 15:55, Cy Schubert  wrote:
> 
> This is exactly what happened with DMD D. When 64-bit statfs was introduced 
> all DMD D compiled programs failed to run and recompiling didn't help. The 
> DMD upstream failed to understand the problem. Eventually the port had to 
> be removed.

I’m not sure that I understand the problem.  This should matter only for 
libraries that pass a statbuf across their ABI boundary.  Anyone using libc 
will see the old version of the symbol and just use the old statbuf.  Anyone 
using the old syscall number and doing system calls directly will see the 
compat version of the structure.  Anyone taking the statbuf and passing it to a 
C library compiled with the newer headers will see compat problems (but the 
same is true for a C library asking a C program to pass it a statbuf and having 
the two compiled against different kernel versions).

There’s a lot that we could do in system headers to make them more 
FFI-friendly.  For example:

 - Use `enum`s rather than `#define`s for constants.

 - Add the flags-enum attribute for flags, so that FFI layers that can parse 
attributes get more semantic information.

 - Add non-null attributes on all arguments and returns that 

 - Use `static inline` functions instead of macros where possible and expose 
them with a macro for `static inline` so that an FFI layer can compile the 
headers in a mode that makes these functions that they can link against.  For 
Rust, this can be compiled to LLVM IR and linked against and inlined into the 
Rust code, so things like the Capsicum permissions bitmap setting code wouldn’t 
need duplicating in Rust.

 - Mark functions with availability attributes so that FFI knows when it’s 
using deprecated / unstable values and can make strong ABI guarantees.

 - Add tests for the headers to the tree.

In 12.0, someone decided to rewrite a load of kernel headers to use macros 
instead of inline functions, which then broke C++ code in the kernel by 
changing properly namespaced things into symbols that would replace every 
identifier.  I’d love to see a concerted effort to use a post-1999 style for 
our headers.

David

Re: Header symbols that shouldn't be visible to ports?

[Alan Somers]

On Wed, Sep 7, 2022 at 8:55 AM Cy Schubert  wrote:
>
> In message  om>
> , Alan Somers writes:
> > On Sat, Sep 3, 2022 at 11:10 PM Konstantin Belousov  
> > wro
> > te:
> > >
> > > On Sat, Sep 03, 2022 at 10:19:12AM -0600, Alan Somers wrote:
> > > > Our /usr/include headers define a lot of symbols that are used by
> > > > critical utilities in the base system like ps and ifconfig, but aren't
> > > > stable across major releases.  Since they aren't stable, utilities
> > > > built for older releases won't run correctly on newer ones.  Would it
> > > > make sense to guard these symbols so they can't be used by programs in
> > > > the ports tree?  There is some precedent for that, for example
> > > > _WANT_SOCKET and _WANT_MNTOPTNAMES.
> > > _WANT_SOCKET is clearly about exposing parts of the kernel definitions
> > > for userspace code that wants to dig into kernel structures.  Similarly
> > > for _WANT_MNTOPTNAMES, but in fact this thing is quite stable.  The
> > > definitions are guarded by additional defines not due to their 
> > > instability,
> > > but because using them in userspace requires (much) more preparation from
> > > userspace environment, which is either not trivial (_WANT_SOCKET) or
> > > contradicts to standartized use of the header (_WANT_MNTOPTNAMES +
> > > sys/mount.h).
> > >
> > > >
> > > > I'm particular, I'm thinking about symbols like the following:
> > > > MINCORE_SUPER
> > > Why this symbol should be hidden?  It is implementation-defined and
> > > intended to be exposed to userspace.  All MINCORE_* not only MINCORE_SUPER
> > > are under BSD_VISIBLE braces, because POSIX does not define the symbols.
> >
> > Because it isn't stable.  It changed for example in rev 847ab36bf22
> > for 13.0.  Programs using the older value (including virtually every
> > Rust program) won't work on 13.0 and later.
> >
> > >
> > > > TDF_*
> > > These symbols coming from non-standard header sys/proc.h.  If userspace
> > > includes the header, it is already outside any formal standard, and I
> > > do not see a reason to make the implementation more convoluted there.
> > >
> > > > PRI_MAX*
> > > > PRI_MIN*
> > > > PI_*, PRIBIO, PVFS, etc
> > > > IFCAP_*
> > > These are all implementation-specific and come from non-standard headers,
> > > unless I am mistaken, then please correct me.
> > >
> > > > RLIM_NLIMITS
> > > > IFF_*
> > > Same.
> > >
> > > > *_MAXID
> > > This is too broad.
> >
> > I'm talking about symbols like IPV6CTL_MAXID, which record the size of
> > sysctl lists.  Obviously, these symbols can't be stable, and probably
> > aren't useful outside of the base system.
> >
> > >
> > > >
> > > > Clearly delineating private symbols like this would ease the
> > > > maintenance burden on languages that rely on FFI, like Ruby and Rust.
> > > > FFI basically assumes that symbols once defined will never change.
> > >
> > > Why e.g. sys/proc.h is ever consumed by FFI wrappers?
> >
> > I should add a little detail.  Rust uses FFI to access C functions,
> > and #define'd constants are redefined in the Rust bindings.  For most
> > Rust programs, the build process doesn't check the contents of
> > /usr/include in any way.  Instead, all of that stuff is hard-coded in
> > the Rust bindings.  That makes cross-compiling a breeze!  But it does
> > cause problems when the C library changes.  Adding a new symbol, like
> > copy_file_range, isn't so bad.  If your Rust program doesn't use it,
> > then the Rust binding will become an unused symbol and get eliminated
> > by the linker.  If your Rust program does use it OTOH, then it will be
> > resolved by the dynamic linker at runtime - if you're running on
> > FreeBSD 13 or newer.  Otherwise, your program will fail to run.  A
> > bigger problem is with symbols that change.  For example, the 64-bit
> > inode stuff.  Rust programs still use a FreeBSD 11 ABI (we're working
> > on that).  But other symbols change more frequently.  Things like
> > PRI_MAX_REALTIME can change between any two releases.  That creates a
> > big maintenance burden to keep track of them in the FFI bindings.  And
> > they also aren't very useful in cross-compiled programs targeting a
> > FreeBSD 1

Re: Header symbols that shouldn't be visible to ports?

[Cy Schubert]

In message 
, Alan Somers writes:
> On Sat, Sep 3, 2022 at 11:10 PM Konstantin Belousov  wro
> te:
> >
> > On Sat, Sep 03, 2022 at 10:19:12AM -0600, Alan Somers wrote:
> > > Our /usr/include headers define a lot of symbols that are used by
> > > critical utilities in the base system like ps and ifconfig, but aren't
> > > stable across major releases.  Since they aren't stable, utilities
> > > built for older releases won't run correctly on newer ones.  Would it
> > > make sense to guard these symbols so they can't be used by programs in
> > > the ports tree?  There is some precedent for that, for example
> > > _WANT_SOCKET and _WANT_MNTOPTNAMES.
> > _WANT_SOCKET is clearly about exposing parts of the kernel definitions
> > for userspace code that wants to dig into kernel structures.  Similarly
> > for _WANT_MNTOPTNAMES, but in fact this thing is quite stable.  The
> > definitions are guarded by additional defines not due to their instability,
> > but because using them in userspace requires (much) more preparation from
> > userspace environment, which is either not trivial (_WANT_SOCKET) or
> > contradicts to standartized use of the header (_WANT_MNTOPTNAMES +
> > sys/mount.h).
> >
> > >
> > > I'm particular, I'm thinking about symbols like the following:
> > > MINCORE_SUPER
> > Why this symbol should be hidden?  It is implementation-defined and
> > intended to be exposed to userspace.  All MINCORE_* not only MINCORE_SUPER
> > are under BSD_VISIBLE braces, because POSIX does not define the symbols.
>
> Because it isn't stable.  It changed for example in rev 847ab36bf22
> for 13.0.  Programs using the older value (including virtually every
> Rust program) won't work on 13.0 and later.
>
> >
> > > TDF_*
> > These symbols coming from non-standard header sys/proc.h.  If userspace
> > includes the header, it is already outside any formal standard, and I
> > do not see a reason to make the implementation more convoluted there.
> >
> > > PRI_MAX*
> > > PRI_MIN*
> > > PI_*, PRIBIO, PVFS, etc
> > > IFCAP_*
> > These are all implementation-specific and come from non-standard headers,
> > unless I am mistaken, then please correct me.
> >
> > > RLIM_NLIMITS
> > > IFF_*
> > Same.
> >
> > > *_MAXID
> > This is too broad.
>
> I'm talking about symbols like IPV6CTL_MAXID, which record the size of
> sysctl lists.  Obviously, these symbols can't be stable, and probably
> aren't useful outside of the base system.
>
> >
> > >
> > > Clearly delineating private symbols like this would ease the
> > > maintenance burden on languages that rely on FFI, like Ruby and Rust.
> > > FFI basically assumes that symbols once defined will never change.
> >
> > Why e.g. sys/proc.h is ever consumed by FFI wrappers?
>
> I should add a little detail.  Rust uses FFI to access C functions,
> and #define'd constants are redefined in the Rust bindings.  For most
> Rust programs, the build process doesn't check the contents of
> /usr/include in any way.  Instead, all of that stuff is hard-coded in
> the Rust bindings.  That makes cross-compiling a breeze!  But it does
> cause problems when the C library changes.  Adding a new symbol, like
> copy_file_range, isn't so bad.  If your Rust program doesn't use it,
> then the Rust binding will become an unused symbol and get eliminated
> by the linker.  If your Rust program does use it OTOH, then it will be
> resolved by the dynamic linker at runtime - if you're running on
> FreeBSD 13 or newer.  Otherwise, your program will fail to run.  A
> bigger problem is with symbols that change.  For example, the 64-bit
> inode stuff.  Rust programs still use a FreeBSD 11 ABI (we're working
> on that).  But other symbols change more frequently.  Things like
> PRI_MAX_REALTIME can change between any two releases.  That creates a
> big maintenance burden to keep track of them in the FFI bindings.  And
> they also aren't very useful in cross-compiled programs targeting a
> FreeBSD 11 ABI.  Instead, they really need to have bindings
> automatically generated at build time.  That's possible, but it's not
> the default.

This is exactly what happened with DMD D. When 64-bit statfs was introduced 
all DMD D compiled programs failed to run and recompiling didn't help. The 
DMD upstream failed to understand the problem. Eventually the port had to 
be removed.

>
> So what the Rust community really needs is a way to know which symbols
> will be stable across releases, and which might vary.  Are you
> suggesting that anything from a non-POSIX header file should be
> considered variable?
>

Rust and every other community.


-- 
Cheers,
Cy Schubert 
FreeBSD UNIX: Web:  http://www.FreeBSD.org
NTP:   Web:  https://nwtime.org

e^(i*pi)+1=0

Re: Header symbols that shouldn't be visible to ports?

[Konstantin Belousov]

On Tue, Sep 06, 2022 at 10:36:52AM -0600, Alan Somers wrote:
> On Tue, Sep 6, 2022 at 9:07 AM Warner Losh  wrote:
> >
> >
> >
> > On Tue, Sep 6, 2022 at 7:34 AM Konstantin Belousov  
> > wrote:
> >>
> >> On Mon, Sep 05, 2022 at 08:41:58AM -0600, Alan Somers wrote:
> >> > On Sat, Sep 3, 2022 at 11:10 PM Konstantin Belousov 
> >> >  wrote:
> >> > >
> >> > > On Sat, Sep 03, 2022 at 10:19:12AM -0600, Alan Somers wrote:
> >> > > > Our /usr/include headers define a lot of symbols that are used by
> >> > > > critical utilities in the base system like ps and ifconfig, but 
> >> > > > aren't
> >> > > > stable across major releases.  Since they aren't stable, utilities
> >> > > > built for older releases won't run correctly on newer ones.  Would it
> >> > > > make sense to guard these symbols so they can't be used by programs 
> >> > > > in
> >> > > > the ports tree?  There is some precedent for that, for example
> >> > > > _WANT_SOCKET and _WANT_MNTOPTNAMES.
> >> > > _WANT_SOCKET is clearly about exposing parts of the kernel definitions
> >> > > for userspace code that wants to dig into kernel structures.  Similarly
> >> > > for _WANT_MNTOPTNAMES, but in fact this thing is quite stable.  The
> >> > > definitions are guarded by additional defines not due to their 
> >> > > instability,
> >> > > but because using them in userspace requires (much) more preparation 
> >> > > from
> >> > > userspace environment, which is either not trivial (_WANT_SOCKET) or
> >> > > contradicts to standartized use of the header (_WANT_MNTOPTNAMES +
> >> > > sys/mount.h).
> >> > >
> >> > > >
> >> > > > I'm particular, I'm thinking about symbols like the following:
> >> > > > MINCORE_SUPER
> >> > > Why this symbol should be hidden?  It is implementation-defined and
> >> > > intended to be exposed to userspace.  All MINCORE_* not only 
> >> > > MINCORE_SUPER
> >> > > are under BSD_VISIBLE braces, because POSIX does not define the 
> >> > > symbols.
> >> >
> >> > Because it isn't stable.  It changed for example in rev 847ab36bf22
> >> > for 13.0.  Programs using the older value (including virtually every
> >> > Rust program) won't work on 13.0 and later.
> >> As Mark replied, older values still mostly work.  It was considered to
> >> not make unacceptable ABI change.
> >>
> >> >
> >> > >
> >> > > > TDF_*
> >> > > These symbols coming from non-standard header sys/proc.h.  If userspace
> >> > > includes the header, it is already outside any formal standard, and I
> >> > > do not see a reason to make the implementation more convoluted there.
> >> > >
> >> > > > PRI_MAX*
> >> > > > PRI_MIN*
> >> > > > PI_*, PRIBIO, PVFS, etc
> >> > > > IFCAP_*
> >> > > These are all implementation-specific and come from non-standard 
> >> > > headers,
> >> > > unless I am mistaken, then please correct me.
> >> > >
> >> > > > RLIM_NLIMITS
> >> > > > IFF_*
> >> > > Same.
> >> > >
> >> > > > *_MAXID
> >> > > This is too broad.
> >> >
> >> > I'm talking about symbols like IPV6CTL_MAXID, which record the size of
> >> > sysctl lists.  Obviously, these symbols can't be stable, and probably
> >> > aren't useful outside of the base system.
> >> The programs are not forced to use the symbols.  FFI bindings should not
> >> provide them, why do we need to specifically hide such defines?
> 
> Because if anybody ever adds it to the libc crate, then it's basically
> stuck there forever.  There's precedent for hiding defines like this:
> https://reviews.freebsd.org/D25816
> 
> >>
> >> >
> >> > >
> >> > > >
> >> > > > Clearly delineating private symbols like this would ease the
> >> > > > maintenance burden on languages that rely on FFI, like Ruby and Rust.
> >> > > > FFI basically assumes that symbols once defined will never change.
> >> > >
> &

Re: Header symbols that shouldn't be visible to ports?

[Alan Somers]

On Tue, Sep 6, 2022 at 9:07 AM Warner Losh  wrote:
>
>
>
> On Tue, Sep 6, 2022 at 7:34 AM Konstantin Belousov  
> wrote:
>>
>> On Mon, Sep 05, 2022 at 08:41:58AM -0600, Alan Somers wrote:
>> > On Sat, Sep 3, 2022 at 11:10 PM Konstantin Belousov  
>> > wrote:
>> > >
>> > > On Sat, Sep 03, 2022 at 10:19:12AM -0600, Alan Somers wrote:
>> > > > Our /usr/include headers define a lot of symbols that are used by
>> > > > critical utilities in the base system like ps and ifconfig, but aren't
>> > > > stable across major releases.  Since they aren't stable, utilities
>> > > > built for older releases won't run correctly on newer ones.  Would it
>> > > > make sense to guard these symbols so they can't be used by programs in
>> > > > the ports tree?  There is some precedent for that, for example
>> > > > _WANT_SOCKET and _WANT_MNTOPTNAMES.
>> > > _WANT_SOCKET is clearly about exposing parts of the kernel definitions
>> > > for userspace code that wants to dig into kernel structures.  Similarly
>> > > for _WANT_MNTOPTNAMES, but in fact this thing is quite stable.  The
>> > > definitions are guarded by additional defines not due to their 
>> > > instability,
>> > > but because using them in userspace requires (much) more preparation from
>> > > userspace environment, which is either not trivial (_WANT_SOCKET) or
>> > > contradicts to standartized use of the header (_WANT_MNTOPTNAMES +
>> > > sys/mount.h).
>> > >
>> > > >
>> > > > I'm particular, I'm thinking about symbols like the following:
>> > > > MINCORE_SUPER
>> > > Why this symbol should be hidden?  It is implementation-defined and
>> > > intended to be exposed to userspace.  All MINCORE_* not only 
>> > > MINCORE_SUPER
>> > > are under BSD_VISIBLE braces, because POSIX does not define the symbols.
>> >
>> > Because it isn't stable.  It changed for example in rev 847ab36bf22
>> > for 13.0.  Programs using the older value (including virtually every
>> > Rust program) won't work on 13.0 and later.
>> As Mark replied, older values still mostly work.  It was considered to
>> not make unacceptable ABI change.
>>
>> >
>> > >
>> > > > TDF_*
>> > > These symbols coming from non-standard header sys/proc.h.  If userspace
>> > > includes the header, it is already outside any formal standard, and I
>> > > do not see a reason to make the implementation more convoluted there.
>> > >
>> > > > PRI_MAX*
>> > > > PRI_MIN*
>> > > > PI_*, PRIBIO, PVFS, etc
>> > > > IFCAP_*
>> > > These are all implementation-specific and come from non-standard headers,
>> > > unless I am mistaken, then please correct me.
>> > >
>> > > > RLIM_NLIMITS
>> > > > IFF_*
>> > > Same.
>> > >
>> > > > *_MAXID
>> > > This is too broad.
>> >
>> > I'm talking about symbols like IPV6CTL_MAXID, which record the size of
>> > sysctl lists.  Obviously, these symbols can't be stable, and probably
>> > aren't useful outside of the base system.
>> The programs are not forced to use the symbols.  FFI bindings should not
>> provide them, why do we need to specifically hide such defines?

Because if anybody ever adds it to the libc crate, then it's basically
stuck there forever.  There's precedent for hiding defines like this:
https://reviews.freebsd.org/D25816

>>
>> >
>> > >
>> > > >
>> > > > Clearly delineating private symbols like this would ease the
>> > > > maintenance burden on languages that rely on FFI, like Ruby and Rust.
>> > > > FFI basically assumes that symbols once defined will never change.
>> > >
>> > > Why e.g. sys/proc.h is ever consumed by FFI wrappers?
>> >
>> > I should add a little detail.  Rust uses FFI to access C functions,
>> > and #define'd constants are redefined in the Rust bindings.  For most
>> > Rust programs, the build process doesn't check the contents of
>> > /usr/include in any way.  Instead, all of that stuff is hard-coded in
>> > the Rust bindings.  That makes cross-compiling a breeze!
>> Well, at the cost of the maintaining Rust libc crate.
>> [Sorry, cannot refrain https://kib.kiev.ua/kib/rust_c

Re: Header symbols that shouldn't be visible to ports?

[Warner Losh]

On Tue, Sep 6, 2022 at 7:34 AM Konstantin Belousov 
wrote:

> On Mon, Sep 05, 2022 at 08:41:58AM -0600, Alan Somers wrote:
> > On Sat, Sep 3, 2022 at 11:10 PM Konstantin Belousov 
> wrote:
> > >
> > > On Sat, Sep 03, 2022 at 10:19:12AM -0600, Alan Somers wrote:
> > > > Our /usr/include headers define a lot of symbols that are used by
> > > > critical utilities in the base system like ps and ifconfig, but
> aren't
> > > > stable across major releases.  Since they aren't stable, utilities
> > > > built for older releases won't run correctly on newer ones.  Would it
> > > > make sense to guard these symbols so they can't be used by programs
> in
> > > > the ports tree?  There is some precedent for that, for example
> > > > _WANT_SOCKET and _WANT_MNTOPTNAMES.
> > > _WANT_SOCKET is clearly about exposing parts of the kernel definitions
> > > for userspace code that wants to dig into kernel structures.  Similarly
> > > for _WANT_MNTOPTNAMES, but in fact this thing is quite stable.  The
> > > definitions are guarded by additional defines not due to their
> instability,
> > > but because using them in userspace requires (much) more preparation
> from
> > > userspace environment, which is either not trivial (_WANT_SOCKET) or
> > > contradicts to standartized use of the header (_WANT_MNTOPTNAMES +
> > > sys/mount.h).
> > >
> > > >
> > > > I'm particular, I'm thinking about symbols like the following:
> > > > MINCORE_SUPER
> > > Why this symbol should be hidden?  It is implementation-defined and
> > > intended to be exposed to userspace.  All MINCORE_* not only
> MINCORE_SUPER
> > > are under BSD_VISIBLE braces, because POSIX does not define the
> symbols.
> >
> > Because it isn't stable.  It changed for example in rev 847ab36bf22
> > for 13.0.  Programs using the older value (including virtually every
> > Rust program) won't work on 13.0 and later.
> As Mark replied, older values still mostly work.  It was considered to
> not make unacceptable ABI change.
>
> >
> > >
> > > > TDF_*
> > > These symbols coming from non-standard header sys/proc.h.  If userspace
> > > includes the header, it is already outside any formal standard, and I
> > > do not see a reason to make the implementation more convoluted there.
> > >
> > > > PRI_MAX*
> > > > PRI_MIN*
> > > > PI_*, PRIBIO, PVFS, etc
> > > > IFCAP_*
> > > These are all implementation-specific and come from non-standard
> headers,
> > > unless I am mistaken, then please correct me.
> > >
> > > > RLIM_NLIMITS
> > > > IFF_*
> > > Same.
> > >
> > > > *_MAXID
> > > This is too broad.
> >
> > I'm talking about symbols like IPV6CTL_MAXID, which record the size of
> > sysctl lists.  Obviously, these symbols can't be stable, and probably
> > aren't useful outside of the base system.
> The programs are not forced to use the symbols.  FFI bindings should not
> provide them, why do we need to specifically hide such defines?
>
> >
> > >
> > > >
> > > > Clearly delineating private symbols like this would ease the
> > > > maintenance burden on languages that rely on FFI, like Ruby and Rust.
> > > > FFI basically assumes that symbols once defined will never change.
> > >
> > > Why e.g. sys/proc.h is ever consumed by FFI wrappers?
> >
> > I should add a little detail.  Rust uses FFI to access C functions,
> > and #define'd constants are redefined in the Rust bindings.  For most
> > Rust programs, the build process doesn't check the contents of
> > /usr/include in any way.  Instead, all of that stuff is hard-coded in
> > the Rust bindings.  That makes cross-compiling a breeze!
> Well, at the cost of the maintaining Rust libc crate.
> [Sorry, cannot refrain https://kib.kiev.ua/kib/rust_c_ffi.png ]
>
> > But it does
> > cause problems when the C library changes.  Adding a new symbol, like
> > copy_file_range, isn't so bad.  If your Rust program doesn't use it,
> > then the Rust binding will become an unused symbol and get eliminated
> > by the linker.  If your Rust program does use it OTOH, then it will be
> > resolved by the dynamic linker at runtime - if you're running on
> > FreeBSD 13 or newer.  Otherwise, your program will fail to run.
> The program would either fail at start if it does not reference the

Re: Header symbols that shouldn't be visible to ports?

[Konstantin Belousov]

On Mon, Sep 05, 2022 at 08:41:58AM -0600, Alan Somers wrote:
> On Sat, Sep 3, 2022 at 11:10 PM Konstantin Belousov  
> wrote:
> >
> > On Sat, Sep 03, 2022 at 10:19:12AM -0600, Alan Somers wrote:
> > > Our /usr/include headers define a lot of symbols that are used by
> > > critical utilities in the base system like ps and ifconfig, but aren't
> > > stable across major releases.  Since they aren't stable, utilities
> > > built for older releases won't run correctly on newer ones.  Would it
> > > make sense to guard these symbols so they can't be used by programs in
> > > the ports tree?  There is some precedent for that, for example
> > > _WANT_SOCKET and _WANT_MNTOPTNAMES.
> > _WANT_SOCKET is clearly about exposing parts of the kernel definitions
> > for userspace code that wants to dig into kernel structures.  Similarly
> > for _WANT_MNTOPTNAMES, but in fact this thing is quite stable.  The
> > definitions are guarded by additional defines not due to their instability,
> > but because using them in userspace requires (much) more preparation from
> > userspace environment, which is either not trivial (_WANT_SOCKET) or
> > contradicts to standartized use of the header (_WANT_MNTOPTNAMES +
> > sys/mount.h).
> >
> > >
> > > I'm particular, I'm thinking about symbols like the following:
> > > MINCORE_SUPER
> > Why this symbol should be hidden?  It is implementation-defined and
> > intended to be exposed to userspace.  All MINCORE_* not only MINCORE_SUPER
> > are under BSD_VISIBLE braces, because POSIX does not define the symbols.
> 
> Because it isn't stable.  It changed for example in rev 847ab36bf22
> for 13.0.  Programs using the older value (including virtually every
> Rust program) won't work on 13.0 and later.
As Mark replied, older values still mostly work.  It was considered to
not make unacceptable ABI change.

> 
> >
> > > TDF_*
> > These symbols coming from non-standard header sys/proc.h.  If userspace
> > includes the header, it is already outside any formal standard, and I
> > do not see a reason to make the implementation more convoluted there.
> >
> > > PRI_MAX*
> > > PRI_MIN*
> > > PI_*, PRIBIO, PVFS, etc
> > > IFCAP_*
> > These are all implementation-specific and come from non-standard headers,
> > unless I am mistaken, then please correct me.
> >
> > > RLIM_NLIMITS
> > > IFF_*
> > Same.
> >
> > > *_MAXID
> > This is too broad.
> 
> I'm talking about symbols like IPV6CTL_MAXID, which record the size of
> sysctl lists.  Obviously, these symbols can't be stable, and probably
> aren't useful outside of the base system.
The programs are not forced to use the symbols.  FFI bindings should not
provide them, why do we need to specifically hide such defines?

> 
> >
> > >
> > > Clearly delineating private symbols like this would ease the
> > > maintenance burden on languages that rely on FFI, like Ruby and Rust.
> > > FFI basically assumes that symbols once defined will never change.
> >
> > Why e.g. sys/proc.h is ever consumed by FFI wrappers?
> 
> I should add a little detail.  Rust uses FFI to access C functions,
> and #define'd constants are redefined in the Rust bindings.  For most
> Rust programs, the build process doesn't check the contents of
> /usr/include in any way.  Instead, all of that stuff is hard-coded in
> the Rust bindings.  That makes cross-compiling a breeze!
Well, at the cost of the maintaining Rust libc crate.
[Sorry, cannot refrain https://kib.kiev.ua/kib/rust_c_ffi.png ]

> But it does
> cause problems when the C library changes.  Adding a new symbol, like
> copy_file_range, isn't so bad.  If your Rust program doesn't use it,
> then the Rust binding will become an unused symbol and get eliminated
> by the linker.  If your Rust program does use it OTOH, then it will be
> resolved by the dynamic linker at runtime - if you're running on
> FreeBSD 13 or newer.  Otherwise, your program will fail to run.
The program would either fail at start if it does not reference the
symbol version in some other way (due to other symbol), or at runtime
when trying to do dynamic binding to that symbol otherwise.

> A
> bigger problem is with symbols that change.  For example, the 64-bit
> inode stuff.  Rust programs still use a FreeBSD 11 ABI (we're working
> on that).
We did not changed symbols for ino64.  Old symbols were retained, the new
symbols were added under the new version.

> But other symbols change more frequently.  Things like
> PRI_MAX_REALTIME c

Re: Header symbols that shouldn't be visible to ports?

[Alan Somers]

On Mon, Sep 5, 2022 at 8:53 AM Mark Johnston  wrote:
>
> On Mon, Sep 05, 2022 at 08:41:58AM -0600, Alan Somers wrote:
> > On Sat, Sep 3, 2022 at 11:10 PM Konstantin Belousov  
> > wrote:
> > >
> > > On Sat, Sep 03, 2022 at 10:19:12AM -0600, Alan Somers wrote:
> > > > Our /usr/include headers define a lot of symbols that are used by
> > > > critical utilities in the base system like ps and ifconfig, but aren't
> > > > stable across major releases.  Since they aren't stable, utilities
> > > > built for older releases won't run correctly on newer ones.  Would it
> > > > make sense to guard these symbols so they can't be used by programs in
> > > > the ports tree?  There is some precedent for that, for example
> > > > _WANT_SOCKET and _WANT_MNTOPTNAMES.
> > > _WANT_SOCKET is clearly about exposing parts of the kernel definitions
> > > for userspace code that wants to dig into kernel structures.  Similarly
> > > for _WANT_MNTOPTNAMES, but in fact this thing is quite stable.  The
> > > definitions are guarded by additional defines not due to their 
> > > instability,
> > > but because using them in userspace requires (much) more preparation from
> > > userspace environment, which is either not trivial (_WANT_SOCKET) or
> > > contradicts to standartized use of the header (_WANT_MNTOPTNAMES +
> > > sys/mount.h).
> > >
> > > >
> > > > I'm particular, I'm thinking about symbols like the following:
> > > > MINCORE_SUPER
> > > Why this symbol should be hidden?  It is implementation-defined and
> > > intended to be exposed to userspace.  All MINCORE_* not only MINCORE_SUPER
> > > are under BSD_VISIBLE braces, because POSIX does not define the symbols.
> >
> > Because it isn't stable.  It changed for example in rev 847ab36bf22
> > for 13.0.  Programs using the older value (including virtually every
> > Rust program) won't work on 13.0 and later.
>
> Why won't they work?  Code that tests (vec[i] & MINCORE_SUPER) using the
> old value will still give the same result when running on a newer
> kernel, since MINCORE_PSIND(1) is 0x20, the old MINCORE_SUPER value.
> This isn't to say that the change was perfectly backwards compatible,
> but I haven't seen an example of code which was broken by the change.

Well, from mincore(2):

In particular, applications compiled using the old value of
MINCORE_SUPER will not identify large pages with size index 2 as being
large pages.

Re: Header symbols that shouldn't be visible to ports?

[Mark Johnston]

On Mon, Sep 05, 2022 at 08:41:58AM -0600, Alan Somers wrote:
> On Sat, Sep 3, 2022 at 11:10 PM Konstantin Belousov  
> wrote:
> >
> > On Sat, Sep 03, 2022 at 10:19:12AM -0600, Alan Somers wrote:
> > > Our /usr/include headers define a lot of symbols that are used by
> > > critical utilities in the base system like ps and ifconfig, but aren't
> > > stable across major releases.  Since they aren't stable, utilities
> > > built for older releases won't run correctly on newer ones.  Would it
> > > make sense to guard these symbols so they can't be used by programs in
> > > the ports tree?  There is some precedent for that, for example
> > > _WANT_SOCKET and _WANT_MNTOPTNAMES.
> > _WANT_SOCKET is clearly about exposing parts of the kernel definitions
> > for userspace code that wants to dig into kernel structures.  Similarly
> > for _WANT_MNTOPTNAMES, but in fact this thing is quite stable.  The
> > definitions are guarded by additional defines not due to their instability,
> > but because using them in userspace requires (much) more preparation from
> > userspace environment, which is either not trivial (_WANT_SOCKET) or
> > contradicts to standartized use of the header (_WANT_MNTOPTNAMES +
> > sys/mount.h).
> >
> > >
> > > I'm particular, I'm thinking about symbols like the following:
> > > MINCORE_SUPER
> > Why this symbol should be hidden?  It is implementation-defined and
> > intended to be exposed to userspace.  All MINCORE_* not only MINCORE_SUPER
> > are under BSD_VISIBLE braces, because POSIX does not define the symbols.
> 
> Because it isn't stable.  It changed for example in rev 847ab36bf22
> for 13.0.  Programs using the older value (including virtually every
> Rust program) won't work on 13.0 and later.

Why won't they work?  Code that tests (vec[i] & MINCORE_SUPER) using the
old value will still give the same result when running on a newer
kernel, since MINCORE_PSIND(1) is 0x20, the old MINCORE_SUPER value.
This isn't to say that the change was perfectly backwards compatible,
but I haven't seen an example of code which was broken by the change.

Re: Header symbols that shouldn't be visible to ports?

[Alan Somers]

On Sat, Sep 3, 2022 at 11:10 PM Konstantin Belousov  wrote:
>
> On Sat, Sep 03, 2022 at 10:19:12AM -0600, Alan Somers wrote:
> > Our /usr/include headers define a lot of symbols that are used by
> > critical utilities in the base system like ps and ifconfig, but aren't
> > stable across major releases.  Since they aren't stable, utilities
> > built for older releases won't run correctly on newer ones.  Would it
> > make sense to guard these symbols so they can't be used by programs in
> > the ports tree?  There is some precedent for that, for example
> > _WANT_SOCKET and _WANT_MNTOPTNAMES.
> _WANT_SOCKET is clearly about exposing parts of the kernel definitions
> for userspace code that wants to dig into kernel structures.  Similarly
> for _WANT_MNTOPTNAMES, but in fact this thing is quite stable.  The
> definitions are guarded by additional defines not due to their instability,
> but because using them in userspace requires (much) more preparation from
> userspace environment, which is either not trivial (_WANT_SOCKET) or
> contradicts to standartized use of the header (_WANT_MNTOPTNAMES +
> sys/mount.h).
>
> >
> > I'm particular, I'm thinking about symbols like the following:
> > MINCORE_SUPER
> Why this symbol should be hidden?  It is implementation-defined and
> intended to be exposed to userspace.  All MINCORE_* not only MINCORE_SUPER
> are under BSD_VISIBLE braces, because POSIX does not define the symbols.

Because it isn't stable.  It changed for example in rev 847ab36bf22
for 13.0.  Programs using the older value (including virtually every
Rust program) won't work on 13.0 and later.

>
> > TDF_*
> These symbols coming from non-standard header sys/proc.h.  If userspace
> includes the header, it is already outside any formal standard, and I
> do not see a reason to make the implementation more convoluted there.
>
> > PRI_MAX*
> > PRI_MIN*
> > PI_*, PRIBIO, PVFS, etc
> > IFCAP_*
> These are all implementation-specific and come from non-standard headers,
> unless I am mistaken, then please correct me.
>
> > RLIM_NLIMITS
> > IFF_*
> Same.
>
> > *_MAXID
> This is too broad.

I'm talking about symbols like IPV6CTL_MAXID, which record the size of
sysctl lists.  Obviously, these symbols can't be stable, and probably
aren't useful outside of the base system.

>
> >
> > Clearly delineating private symbols like this would ease the
> > maintenance burden on languages that rely on FFI, like Ruby and Rust.
> > FFI basically assumes that symbols once defined will never change.
>
> Why e.g. sys/proc.h is ever consumed by FFI wrappers?

I should add a little detail.  Rust uses FFI to access C functions,
and #define'd constants are redefined in the Rust bindings.  For most
Rust programs, the build process doesn't check the contents of
/usr/include in any way.  Instead, all of that stuff is hard-coded in
the Rust bindings.  That makes cross-compiling a breeze!  But it does
cause problems when the C library changes.  Adding a new symbol, like
copy_file_range, isn't so bad.  If your Rust program doesn't use it,
then the Rust binding will become an unused symbol and get eliminated
by the linker.  If your Rust program does use it OTOH, then it will be
resolved by the dynamic linker at runtime - if you're running on
FreeBSD 13 or newer.  Otherwise, your program will fail to run.  A
bigger problem is with symbols that change.  For example, the 64-bit
inode stuff.  Rust programs still use a FreeBSD 11 ABI (we're working
on that).  But other symbols change more frequently.  Things like
PRI_MAX_REALTIME can change between any two releases.  That creates a
big maintenance burden to keep track of them in the FFI bindings.  And
they also aren't very useful in cross-compiled programs targeting a
FreeBSD 11 ABI.  Instead, they really need to have bindings
automatically generated at build time.  That's possible, but it's not
the default.

So what the Rust community really needs is a way to know which symbols
will be stable across releases, and which might vary.  Are you
suggesting that anything from a non-POSIX header file should be
considered variable?

Re: Header symbols that shouldn't be visible to ports?

[Konstantin Belousov]

On Sat, Sep 03, 2022 at 10:19:12AM -0600, Alan Somers wrote:
> Our /usr/include headers define a lot of symbols that are used by
> critical utilities in the base system like ps and ifconfig, but aren't
> stable across major releases.  Since they aren't stable, utilities
> built for older releases won't run correctly on newer ones.  Would it
> make sense to guard these symbols so they can't be used by programs in
> the ports tree?  There is some precedent for that, for example
> _WANT_SOCKET and _WANT_MNTOPTNAMES.
_WANT_SOCKET is clearly about exposing parts of the kernel definitions
for userspace code that wants to dig into kernel structures.  Similarly
for _WANT_MNTOPTNAMES, but in fact this thing is quite stable.  The
definitions are guarded by additional defines not due to their instability,
but because using them in userspace requires (much) more preparation from
userspace environment, which is either not trivial (_WANT_SOCKET) or
contradicts to standartized use of the header (_WANT_MNTOPTNAMES + 
sys/mount.h).

> 
> I'm particular, I'm thinking about symbols like the following:
> MINCORE_SUPER
Why this symbol should be hidden?  It is implementation-defined and
intended to be exposed to userspace.  All MINCORE_* not only MINCORE_SUPER
are under BSD_VISIBLE braces, because POSIX does not define the symbols.

> TDF_*
These symbols coming from non-standard header sys/proc.h.  If userspace
includes the header, it is already outside any formal standard, and I
do not see a reason to make the implementation more convoluted there.

> PRI_MAX*
> PRI_MIN*
> PI_*, PRIBIO, PVFS, etc
> IFCAP_*
These are all implementation-specific and come from non-standard headers,
unless I am mistaken, then please correct me.

> RLIM_NLIMITS
> IFF_*
Same.

> *_MAXID
This is too broad.

> 
> Clearly delineating private symbols like this would ease the
> maintenance burden on languages that rely on FFI, like Ruby and Rust.
> FFI basically assumes that symbols once defined will never change.

Why e.g. sys/proc.h is ever consumed by FFI wrappers?

Header symbols that shouldn't be visible to ports?

[Alan Somers]

Our /usr/include headers define a lot of symbols that are used by
critical utilities in the base system like ps and ifconfig, but aren't
stable across major releases.  Since they aren't stable, utilities
built for older releases won't run correctly on newer ones.  Would it
make sense to guard these symbols so they can't be used by programs in
the ports tree?  There is some precedent for that, for example
_WANT_SOCKET and _WANT_MNTOPTNAMES.

I'm particular, I'm thinking about symbols like the following:
MINCORE_SUPER
TDF_*
PRI_MAX*
PRI_MIN*
PI_*, PRIBIO, PVFS, etc
IFCAP_*
RLIM_NLIMITS
IFF_*
*_MAXID

Clearly delineating private symbols like this would ease the
maintenance burden on languages that rely on FFI, like Ruby and Rust.
FFI basically assumes that symbols once defined will never change.

-Alan

Re: MegaCLI port is ports-only -- how would you deploy it?

[Doug Ambrisko]

On Tue, Aug 16, 2022 at 03:33:57PM -0400, Dan Mahoney wrote:
| > On Aug 16, 2022, at 14:04, Doug Ambrisko  wrote:
| > 
| > On Tue, Aug 16, 2022 at 11:58:40AM -0600, Warner Losh wrote:
| > |On Tue, Aug 16, 2022 at 11:40 AM Ed Maste <[1]ema...@freebsd.org>
| > |wrote:
| > | 
| > |  On Mon, 15 Aug 2022 at 20:03, Doug Ambrisko
| > |  <[2]ambri...@ambrisko.com> wrote:
| > |  > | > | > I'd have to put in -current first then look at MFC later
| > |  on.  If looks
| > |  > | > | > good for you then I'll put it up for review.  I just don't
| > |  use this
| > |  > | > | > stuff day to day anymore.
| > |  I think it would be good to put this into review, perhaps separate
| > |  ones for the kernel and userland parts. Feel free to put me on as a
| > |  reviewer or subscriber.
| > | 
| > |I can review as well. I like this plan.
| > 
| > I'll add John as well since I think he wrote mfiutil originally.
| 
| I haven't attached the makefile patch to the bug I opened yet.  (And the
| existing patch is a link, not an attach).

That isn't critical since I'm the one that made it and plan to be
the committer to deal with it.  I grabbed ownership of the bug.
I updated my personal branch with the fix so I shouldn't drop it
again.  I had factored out some other local changes I had in the
driver.  It had issue with various things that the team I used be
part of would hit since that SW sent lots of concurrent I/O to the
card.  It was made worse on the low end system without cache.
Broadcom didn't seem interested in addressing it when I used to work
with them more closely.  I had patches in that teams driver.
 
| If further diagnostics are needed (I'm not going to like, try to use it
| to blow away and re-create an array, sorry), I can provide SOME of that.
| Just tell me how I can help.

That shouldn't really be needed since the MFI commands are consistent
being MegaRAID SAS controllers.  The main thing that has changed is
the driver.  I've heard they are changing for their next controller that
would require new tools and driver.  In a few months I might have
a machine with one to play with.
 
| In the mean time, since the system that I'm testing this on is one of
| the dayjob's console servers, and we still want to be able to run puppet
| on it periodically, y'all have motivated me to get puppet to fix their
| "ports" package provider, which still depends on pkg-classic binaries
| being present.  Not relevant to this problem, other than noting the cool
| ripple effect.

Thanks,

Doug A.

Re: MegaCLI port is ports-only -- how would you deploy it?

[Doug Ambrisko]

On Tue, Aug 16, 2022 at 11:58:40AM -0600, Warner Losh wrote:
|On Tue, Aug 16, 2022 at 11:40 AM Ed Maste <[1]ema...@freebsd.org>
|wrote:
| 
|  On Mon, 15 Aug 2022 at 20:03, Doug Ambrisko
|  <[2]ambri...@ambrisko.com> wrote:
|  > | > | > I'd have to put in -current first then look at MFC later
|  on.  If looks
|  > | > | > good for you then I'll put it up for review.  I just don't
|  use this
|  > | > | > stuff day to day anymore.
|  I think it would be good to put this into review, perhaps separate
|  ones for the kernel and userland parts. Feel free to put me on as a
|  reviewer or subscriber.
| 
|I can review as well. I like this plan.

I'll add John as well since I think he wrote mfiutil originally.

Thanks,

Doug A.

Re: MegaCLI port is ports-only -- how would you deploy it?

[Warner Losh]

On Tue, Aug 16, 2022 at 11:40 AM Ed Maste  wrote:

> On Mon, 15 Aug 2022 at 20:03, Doug Ambrisko  wrote:
> > | > | > I'd have to put in -current first then look at MFC later on.  If
> looks
> > | > | > good for you then I'll put it up for review.  I just don't use
> this
> > | > | > stuff day to day anymore.
>
> I think it would be good to put this into review, perhaps separate
> ones for the kernel and userland parts. Feel free to put me on as a
> reviewer or subscriber.
>

I can review as well. I like this plan.

Warner

Re: MegaCLI port is ports-only -- how would you deploy it?

[Ed Maste]

On Mon, 15 Aug 2022 at 20:03, Doug Ambrisko  wrote:
> | > | > I'd have to put in -current first then look at MFC later on.  If looks
> | > | > good for you then I'll put it up for review.  I just don't use this
> | > | > stuff day to day anymore.

I think it would be good to put this into review, perhaps separate
ones for the kernel and userland parts. Feel free to put me on as a
reviewer or subscriber.

Re: MegaCLI port is ports-only -- how would you deploy it?

[Doug Ambrisko]

On Sat, Aug 13, 2022 at 10:41:33PM -0400, Dan Mahoney wrote:
| 
| 
| > On Aug 12, 2022, at 12:35, Doug Ambrisko  wrote:
| > 
| > On Fri, Aug 12, 2022 at 12:32:56PM -0400, Dan Mahoney wrote:
| > | 
| > | 
| > | > On Aug 12, 2022, at 12:31, Doug Ambrisko  wrote:
| > | > 
| > | > On Fri, Aug 12, 2022 at 12:21:36PM -0400, Dan Mahoney wrote:
| > | > | 
| > | > | > On Aug 8, 2022, at 16:45, Doug Ambrisko  
wrote:
| > | > | > 
| > | > | > On Mon, Aug 08, 2022 at 04:10:10PM -0400, Dan Mahoney wrote:
| > | > | > | 
| > | > | > | 
| > | > | > | > On Aug 8, 2022, at 15:57, Doug Ambrisko  
wrote:
| > | > | > | > 
| > | > | > | > On Thu, Aug 04, 2022 at 05:22:29PM +0300, Ruslan Makhmatkhanov 
wrote:
| > | > | > | > |03.08.2022, 02:07, "Dan Mahoney" :
| > | > | > | > |  Hey there all,
| > | > | > | > |  At the dayjob we have a fleet of Dell Poweredge servers 
that can use
| > | > | > | > |  either mptsas or mrsas -- if you use mptsas, you use 
mptutil (in
| > | > | > | > |  base) to check the state of the card.
| > | > | > | > |  If you use mrsas, you need megacli, which is only in 
ports, and the
| > | > | > | > |  port hasn't translated to pkg probably because of license
| > | > | > | > |  restrictions. ( _LICENSE_RESTRICTED = delete-package
| > | > | > | > |  delete-distfiles), but the license listed is just 
"megacli".
| > | > | > | > |  * We want to deploy a cron job to periodically check the 
raid status
| > | > | > | > |  (we're writing a wrapper, also having it check mfiutil, 
zpool, etc).
| > | > | > | > |  * We do not want to install and manage a whole ports 
tree on every
| > | > | > | > |  machine in our fleet, just to install a raid utlity.
| > | > | > | > |  Option A:
| > | > | > | > |  Make a local package somehow.
| > | > | > | > |  The port just downloads a static binary, there's nothing 
to build
| > | > | > | > |  here, but we want to do this the "right" way. Is there 
some way to
| > | > | > | > |  have pkg deploy a single local package for this that 
will, for
| > | > | > | > |  example, report the right package ownership, without 
moving every
| > | > | > | > |  other package to our poudriere install (we're just using 
base
| > | > | > | > |  packages, we keep poudriere around for testing in case 
we need to
| > | > | > | > |  hot-patch something).
| > | > | > | > |  For what it's worth, we use puppet for config 
management, so pushing
| > | > | > | > |  out the static binary is not the worst answer, but it 
also feels
| > | > | > | > |  "dirty".
| > | > | > | > |  Option B:
| > | > | > | > |  Figure out how to fix the license. I have no idea what 
this would
| > | > | > | > |  involve.
| > | > | > | > |  Option C:
| > | > | > | > |  Also, apparently MegaCLI is no longer maintained 
(replaced by
| > | > | > | > |  StorCLI), but there's no port for StorCLI, and...there 
are multiple
| > | > | > | > |  raid-card specific versions? Jeez.
| > | > | > | > |  Feels even more dirty.
| > | > | > | > |  
[1]https://support.siliconmechanics.com/portal/en/kb/articles/storcl
| > | > | > | > |  i-for-freebsd-and-other-operating-systems
| > | > | > | > |  Ideas welcome?
| > | > | > | > |  -Dan Mahoney
| > | > | > | > 
| > | > | > | > Although the path to get to StorCli goes through various cards 
the
| > | > | > | > latest greatest seem to work on all earlier cards.  It works on
| > | > | > | > HBAs and not just RAID cards.  At work I did a Linux/FreeBSD
| > | > | > | > POC for FW management and found the FreeBSD version could flash 
the HBA
| > | > | > | > and drive FW.  I've moved to StorCli from MegaCli.  I would 
suggest
| > | > | > | > we drop the MegaCli port and move to StorCli.
| > | > | > | > 
| > | > | > | > I have code to make mfiutil into mrsasutil and added the MFI 
ioctl
| > | > | > | > handler to mrsas.  I'm not sure how much value that has.  I 
don't
| > | > | > | > deal with supporting FreeBSD and RAID much anymore.  If 
interested
| > | > | > | > I could send patches.
| > | > | > | 
| > | > | > | This feels like it should be in base, regardless.  J

ASAN&UBSAN world in chroot tree, ports built previously, variable results finding /usr/local/lib/*.so* files

[Mark Millard]

Note: like the /usr/local/lib/*.so* files, the
wget and git were built previously to the existence
of the ASAN&UBSAN world.

chrooted into the world I built with ASAN and UBSAN:

# ldd `which wget`
/usr/local/bin/wget:
libintl.so.8 => /usr/local/lib/libintl.so.8 (0x800edc000)
libunistring.so.2 => /usr/local/lib/libunistring.so.2 (0x80182b000)
libidn2.so.0 => /usr/local/lib/libidn2.so.0 (0x801b76000)
libssl.so.111 => /usr/lib/libssl.so.111 (0x80229d000)
libcrypto.so.111 => /lib/libcrypto.so.111 (0x803a0)
libdl.so.1 => /usr/lib/libdl.so.1 (0x8034ed000)
libz.so.6 => /lib/libz.so.6 (0x80585d000)
libc.so.7 => /lib/libc.so.7 (0x806e0)
libthr.so.3 => /lib/libthr.so.3 (0x8063fc000)

# ldd `which git`
/usr/local/bin/git:
libpcre2-8.so.0 => not found (0)
libz.so.6 => /lib/libz.so.6 (0x801bc5000)
libintl.so.8 => not found (0)
libthr.so.3 => /lib/libthr.so.3 (0x8009ab000)
libc.so.7 => /lib/libc.so.7 (0x80400)

Note the differing results for finding libintl.so.8 .

The ldd and which runs get no ASAN reports and no UBSAN reports.

SIDE NOTE
I will note that attempting to use the wget results in:

# wget
ld-elf.so.1: /usr/lib/libssl.so.111: Undefined symbol 
"__asan_option_detect_stack_use_after_return"
END SIDE NOTE


Instead chrooted into my normal world built for chrooting into
main (no ASAN/UBSAN involvement):

% ldd `which wget`
/usr/local/bin/wget:
libintl.so.8 => /usr/local/lib/libintl.so.8 (0x8022a3000)
libunistring.so.2 => /usr/local/lib/libunistring.so.2 (0x801c69000)
libidn2.so.0 => /usr/local/lib/libidn2.so.0 (0x803227000)
libssl.so.111 => /usr/lib/libssl.so.111 (0x80372d000)
libcrypto.so.111 => /lib/libcrypto.so.111 (0x803a37000)
libdl.so.1 => /usr/lib/libdl.so.1 (0x805668000)
libz.so.6 => /lib/libz.so.6 (0x804618000)
libc.so.7 => /lib/libc.so.7 (0x806aca000)
libthr.so.3 => /lib/libthr.so.3 (0x80468a000)

% ldd `which git`
/usr/local/bin/git:
libpcre2-8.so.0 => /usr/local/lib/libpcre2-8.so.0 (0x801b2d000)
libz.so.6 => /lib/libz.so.6 (0x801324000)
libintl.so.8 => /usr/local/lib/libintl.so.8 (0x801fe8000)
libthr.so.3 => /lib/libthr.so.3 (0x802ed9000)
libc.so.7 => /lib/libc.so.7 (0x803d98000)

Note that both libintl.so.8 and libintl.so.8 were found,
as expected. (wget and git both run just fine as well.)


It is the same /usr/local/ file system, used from the two
contexts (mounted into the chroot trees).

It appears that lots of ports from that /usr/local/ tree
get "Shared object not found" messages for one or more
/usr/local/lib/*.so* files (matching up with ldd reports
of not found).

===
Mark Millard
marklmi at yahoo.com

Re: Problem compiling ports

[Tomoaki AOKI]

Confirmed.
Thanks for your quick reaction!

On Tue, 30 Nov 2021 22:45:37 +
Brooks Davis  wrote:

> Yeah, I've got this in progress.
> 
> -- Brooks
> 
> On Wed, Dec 01, 2021 at 06:57:56AM +0900, Tomoaki AOKI wrote:
> > It would be better making new special handling like BE_AMDGPU for it.
> > Building with BE_STANDARD just for this would be a pain for some users.
> > 
> > (CC'ing freebsd-ports ML.)
> > 
> > 
> > On Tue, 30 Nov 2021 17:45:30 +
> > Brooks Davis  wrote:
> > 
> > > In the config for devel/llvm11, is BE_STANDARD enabled?  If not, you
> > > won't have the web assembly backend.
> > > 
> > > -- Brooks
> > > 
> > > On Tue, Nov 30, 2021 at 05:24:45PM +, Filippo Moretti via current 
> > > wrote:
> > > > error: unable to create target: 'No available targets are compatible 
> > > > with triple "wasm32-unknown-wasi"'
> > > > 1 error generated.
> > > > gmake[3]: *** [Makefile:380: 
> > > > /usr/ports/devel/wasi-libc/work/wasi-libc-ad5133410f66b93a2381db5b542aad5e0964db96/build/dlmalloc/src/dlmalloc.o]
> > > >  Error 1
> > > > error: unable to create target: 'No available targets are compatible 
> > > > with triple "wasm32-unknown-wasi"'
> > > > 1 error generated.
> > > > gmake[3]: *** [Makefile:440: startup_files] Error 1
> > > > gmake[3]: Leaving directory 
> > > > '/usr/ports/devel/wasi-libc/work/wasi-libc-ad5133410f66b93a2381db5b542aad5e0964db96'
> > > > ===> Compilation failed unexpectedly.
> > > > Try to set MAKE_JOBS_UNSAFE=yes and rebuild before reporting the 
> > > > failure to
> > > > the maintainer.
> > > > *** Error code 1
> > > > 
> > > > Stop.
> > > > make[2]: stopped in /usr/ports/devel/wasi-libc
> > > > *** Error code 1
> > > > 
> > > > Stop.
> > > > make[1]: stopped in /usr/ports/devel/wasi-libcxx*** Error code 1
> > > > 
> > > > Stop.
> > > > make: stopped in /usr/ports/www/firefox
> > > > 
> > > > ===>>> make build failed for www/firefox
> > > > ===>>> Aborting update
> > > > 
> > > > 
> > > > ??File "/usr/local/lib/python3.8/site-packages/setuptools/dist.py", 
> > > > line 799, in 
> > > > ?? eps = map(lambda e: e.load(), 
> > > > pkg_resources.iter_entry_points(group))
> > > > ?? File 
> > > > "/usr/local/lib/python3.8/site-packages/pkg_resources/__init__.py", 
> > > > line 2449, in load
> > > > ?? self.require(*args, **kwargs)
> > > > ?? File 
> > > > "/usr/local/lib/python3.8/site-packages/pkg_resources/__init__.py", 
> > > > line 2472, in require
> > > > ?? items = working_set.resolve(reqs, env, installer, 
> > > > extras=self.extras)
> > > > ?? File 
> > > > "/usr/local/lib/python3.8/site-packages/pkg_resources/__init__.py", 
> > > > line 772, in resolve
> > > > ?? raise DistributionNotFound(req, requirers)
> > > > pkg_resources.DistributionNotFound: The 'packaging>=20.0' distribution 
> > > > was not found and is required by the application
> > > > *** Error code 1
> > > > 
> > > > Stop.
> > > > make: stopped in /usr/ports/devel/py-pycparser
> > > > 
> > > > ===>>> make build failed for devel/py-pycparser@py38
> > > > ===>>> Aborting update
> > > > 
> > > > FreeBSD sting 14.0-CURRENT FreeBSD 14.0-CURRENT #62 
> > > > heads/main-n251146-d109559ddbf: Mon Nov 29 12:18:48 CET 2021 
> > > > root@sting:/usr/obj/usr/src/amd64.amd64/sys/STING?? amd64
> > > > 
> > > > 
> > 
> > 
> > -- 
> > Tomoaki AOKI
> > 


-- 
青木 知明  [Tomoaki AOKI]

Re: Problem compiling ports

[Brooks Davis]

Yeah, I've got this in progress.

-- Brooks

On Wed, Dec 01, 2021 at 06:57:56AM +0900, Tomoaki AOKI wrote:
> It would be better making new special handling like BE_AMDGPU for it.
> Building with BE_STANDARD just for this would be a pain for some users.
> 
> (CC'ing freebsd-ports ML.)
> 
> 
> On Tue, 30 Nov 2021 17:45:30 +
> Brooks Davis  wrote:
> 
> > In the config for devel/llvm11, is BE_STANDARD enabled?  If not, you
> > won't have the web assembly backend.
> > 
> > -- Brooks
> > 
> > On Tue, Nov 30, 2021 at 05:24:45PM +, Filippo Moretti via current wrote:
> > > error: unable to create target: 'No available targets are compatible with 
> > > triple "wasm32-unknown-wasi"'
> > > 1 error generated.
> > > gmake[3]: *** [Makefile:380: 
> > > /usr/ports/devel/wasi-libc/work/wasi-libc-ad5133410f66b93a2381db5b542aad5e0964db96/build/dlmalloc/src/dlmalloc.o]
> > >  Error 1
> > > error: unable to create target: 'No available targets are compatible with 
> > > triple "wasm32-unknown-wasi"'
> > > 1 error generated.
> > > gmake[3]: *** [Makefile:440: startup_files] Error 1
> > > gmake[3]: Leaving directory 
> > > '/usr/ports/devel/wasi-libc/work/wasi-libc-ad5133410f66b93a2381db5b542aad5e0964db96'
> > > ===> Compilation failed unexpectedly.
> > > Try to set MAKE_JOBS_UNSAFE=yes and rebuild before reporting the failure 
> > > to
> > > the maintainer.
> > > *** Error code 1
> > > 
> > > Stop.
> > > make[2]: stopped in /usr/ports/devel/wasi-libc
> > > *** Error code 1
> > > 
> > > Stop.
> > > make[1]: stopped in /usr/ports/devel/wasi-libcxx*** Error code 1
> > > 
> > > Stop.
> > > make: stopped in /usr/ports/www/firefox
> > > 
> > > ===>>> make build failed for www/firefox
> > > ===>>> Aborting update
> > > 
> > > 
> > > ??File "/usr/local/lib/python3.8/site-packages/setuptools/dist.py", line 
> > > 799, in 
> > > ?? eps = map(lambda e: e.load(), 
> > > pkg_resources.iter_entry_points(group))
> > > ?? File 
> > > "/usr/local/lib/python3.8/site-packages/pkg_resources/__init__.py", line 
> > > 2449, in load
> > > ?? self.require(*args, **kwargs)
> > > ?? File 
> > > "/usr/local/lib/python3.8/site-packages/pkg_resources/__init__.py", line 
> > > 2472, in require
> > > ?? items = working_set.resolve(reqs, env, installer, 
> > > extras=self.extras)
> > > ?? File 
> > > "/usr/local/lib/python3.8/site-packages/pkg_resources/__init__.py", line 
> > > 772, in resolve
> > > ?? raise DistributionNotFound(req, requirers)
> > > pkg_resources.DistributionNotFound: The 'packaging>=20.0' distribution 
> > > was not found and is required by the application
> > > *** Error code 1
> > > 
> > > Stop.
> > > make: stopped in /usr/ports/devel/py-pycparser
> > > 
> > > ===>>> make build failed for devel/py-pycparser@py38
> > > ===>>> Aborting update
> > > 
> > > FreeBSD sting 14.0-CURRENT FreeBSD 14.0-CURRENT #62 
> > > heads/main-n251146-d109559ddbf: Mon Nov 29 12:18:48 CET 2021 
> > > root@sting:/usr/obj/usr/src/amd64.amd64/sys/STING?? amd64
> > > 
> > > 
> 
> 
> -- 
> Tomoaki AOKI
> 


signature.asc
Description: PGP signature

Re: Problem compiling ports

[Tomoaki AOKI]

It would be better making new special handling like BE_AMDGPU for it.
Building with BE_STANDARD just for this would be a pain for some users.

(CC'ing freebsd-ports ML.)


On Tue, 30 Nov 2021 17:45:30 +
Brooks Davis  wrote:

> In the config for devel/llvm11, is BE_STANDARD enabled?  If not, you
> won't have the web assembly backend.
> 
> -- Brooks
> 
> On Tue, Nov 30, 2021 at 05:24:45PM +, Filippo Moretti via current wrote:
> > error: unable to create target: 'No available targets are compatible with 
> > triple "wasm32-unknown-wasi"'
> > 1 error generated.
> > gmake[3]: *** [Makefile:380: 
> > /usr/ports/devel/wasi-libc/work/wasi-libc-ad5133410f66b93a2381db5b542aad5e0964db96/build/dlmalloc/src/dlmalloc.o]
> >  Error 1
> > error: unable to create target: 'No available targets are compatible with 
> > triple "wasm32-unknown-wasi"'
> > 1 error generated.
> > gmake[3]: *** [Makefile:440: startup_files] Error 1
> > gmake[3]: Leaving directory 
> > '/usr/ports/devel/wasi-libc/work/wasi-libc-ad5133410f66b93a2381db5b542aad5e0964db96'
> > ===> Compilation failed unexpectedly.
> > Try to set MAKE_JOBS_UNSAFE=yes and rebuild before reporting the failure to
> > the maintainer.
> > *** Error code 1
> > 
> > Stop.
> > make[2]: stopped in /usr/ports/devel/wasi-libc
> > *** Error code 1
> > 
> > Stop.
> > make[1]: stopped in /usr/ports/devel/wasi-libcxx*** Error code 1
> > 
> > Stop.
> > make: stopped in /usr/ports/www/firefox
> > 
> > ===>>> make build failed for www/firefox
> > ===>>> Aborting update
> > 
> > 
> > ??File "/usr/local/lib/python3.8/site-packages/setuptools/dist.py", line 
> > 799, in 
> > ?? eps = map(lambda e: e.load(), pkg_resources.iter_entry_points(group))
> > ?? File "/usr/local/lib/python3.8/site-packages/pkg_resources/__init__.py", 
> > line 2449, in load
> > ?? self.require(*args, **kwargs)
> > ?? File "/usr/local/lib/python3.8/site-packages/pkg_resources/__init__.py", 
> > line 2472, in require
> > ?? items = working_set.resolve(reqs, env, installer, extras=self.extras)
> > ?? File "/usr/local/lib/python3.8/site-packages/pkg_resources/__init__.py", 
> > line 772, in resolve
> > ?? raise DistributionNotFound(req, requirers)
> > pkg_resources.DistributionNotFound: The 'packaging>=20.0' distribution was 
> > not found and is required by the application
> > *** Error code 1
> > 
> > Stop.
> > make: stopped in /usr/ports/devel/py-pycparser
> > 
> > ===>>> make build failed for devel/py-pycparser@py38
> > ===>>> Aborting update
> > 
> > FreeBSD sting 14.0-CURRENT FreeBSD 14.0-CURRENT #62 
> > heads/main-n251146-d109559ddbf: Mon Nov 29 12:18:48 CET 2021 
> > root@sting:/usr/obj/usr/src/amd64.amd64/sys/STING?? amd64
> > 
> > 


-- 
Tomoaki AOKI

Re: Problem compiling ports

[Brooks Davis]

In the config for devel/llvm11, is BE_STANDARD enabled?  If not, you
won't have the web assembly backend.

-- Brooks

On Tue, Nov 30, 2021 at 05:24:45PM +, Filippo Moretti via current wrote:
> error: unable to create target: 'No available targets are compatible with 
> triple "wasm32-unknown-wasi"'
> 1 error generated.
> gmake[3]: *** [Makefile:380: 
> /usr/ports/devel/wasi-libc/work/wasi-libc-ad5133410f66b93a2381db5b542aad5e0964db96/build/dlmalloc/src/dlmalloc.o]
>  Error 1
> error: unable to create target: 'No available targets are compatible with 
> triple "wasm32-unknown-wasi"'
> 1 error generated.
> gmake[3]: *** [Makefile:440: startup_files] Error 1
> gmake[3]: Leaving directory 
> '/usr/ports/devel/wasi-libc/work/wasi-libc-ad5133410f66b93a2381db5b542aad5e0964db96'
> ===> Compilation failed unexpectedly.
> Try to set MAKE_JOBS_UNSAFE=yes and rebuild before reporting the failure to
> the maintainer.
> *** Error code 1
> 
> Stop.
> make[2]: stopped in /usr/ports/devel/wasi-libc
> *** Error code 1
> 
> Stop.
> make[1]: stopped in /usr/ports/devel/wasi-libcxx*** Error code 1
> 
> Stop.
> make: stopped in /usr/ports/www/firefox
> 
> ===>>> make build failed for www/firefox
> ===>>> Aborting update
> 
> 
> ??File "/usr/local/lib/python3.8/site-packages/setuptools/dist.py", line 799, 
> in 
> ?? eps = map(lambda e: e.load(), pkg_resources.iter_entry_points(group))
> ?? File "/usr/local/lib/python3.8/site-packages/pkg_resources/__init__.py", 
> line 2449, in load
> ?? self.require(*args, **kwargs)
> ?? File "/usr/local/lib/python3.8/site-packages/pkg_resources/__init__.py", 
> line 2472, in require
> ?? items = working_set.resolve(reqs, env, installer, extras=self.extras)
> ?? File "/usr/local/lib/python3.8/site-packages/pkg_resources/__init__.py", 
> line 772, in resolve
> ?? raise DistributionNotFound(req, requirers)
> pkg_resources.DistributionNotFound: The 'packaging>=20.0' distribution was 
> not found and is required by the application
> *** Error code 1
> 
> Stop.
> make: stopped in /usr/ports/devel/py-pycparser
> 
> ===>>> make build failed for devel/py-pycparser@py38
> ===>>> Aborting update
> 
> FreeBSD sting 14.0-CURRENT FreeBSD 14.0-CURRENT #62 
> heads/main-n251146-d109559ddbf: Mon Nov 29 12:18:48 CET 2021 
> root@sting:/usr/obj/usr/src/amd64.amd64/sys/STING?? amd64
> 
> 


signature.asc
Description: PGP signature

Problem compiling ports

[Filippo Moretti via current]

error: unable to create target: 'No available targets are compatible with 
triple "wasm32-unknown-wasi"'
1 error generated.
gmake[3]: *** [Makefile:380: 
/usr/ports/devel/wasi-libc/work/wasi-libc-ad5133410f66b93a2381db5b542aad5e0964db96/build/dlmalloc/src/dlmalloc.o]
 Error 1
error: unable to create target: 'No available targets are compatible with 
triple "wasm32-unknown-wasi"'
1 error generated.
gmake[3]: *** [Makefile:440: startup_files] Error 1
gmake[3]: Leaving directory 
'/usr/ports/devel/wasi-libc/work/wasi-libc-ad5133410f66b93a2381db5b542aad5e0964db96'
===> Compilation failed unexpectedly.
Try to set MAKE_JOBS_UNSAFE=yes and rebuild before reporting the failure to
the maintainer.
*** Error code 1

Stop.
make[2]: stopped in /usr/ports/devel/wasi-libc
*** Error code 1

Stop.
make[1]: stopped in /usr/ports/devel/wasi-libcxx*** Error code 1

Stop.
make: stopped in /usr/ports/www/firefox

===>>> make build failed for www/firefox
===>>> Aborting update


 File "/usr/local/lib/python3.8/site-packages/setuptools/dist.py", line 799, in 

    eps = map(lambda e: e.load(), pkg_resources.iter_entry_points(group))
  File "/usr/local/lib/python3.8/site-packages/pkg_resources/__init__.py", line 
2449, in load
    self.require(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/pkg_resources/__init__.py", line 
2472, in require
    items = working_set.resolve(reqs, env, installer, extras=self.extras)
  File "/usr/local/lib/python3.8/site-packages/pkg_resources/__init__.py", line 
772, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'packaging>=20.0' distribution was not 
found and is required by the application
*** Error code 1

Stop.
make: stopped in /usr/ports/devel/py-pycparser

===>>> make build failed for devel/py-pycparser@py38
===>>> Aborting update

FreeBSD sting 14.0-CURRENT FreeBSD 14.0-CURRENT #62 
heads/main-n251146-d109559ddbf: Mon Nov 29 12:18:48 CET 2021 
root@sting:/usr/obj/usr/src/amd64.amd64/sys/STING  amd64

Re: git: "overlay" of own remote-branch on official freebsd-ports repo

[FreeBSD User]

On Tue, 12 Oct 2021 10:01:00 -0600
Warner Losh  wrote:

> On Tue, Oct 12, 2021 at 9:32 AM FreeBSD User  wrote:
> 
> > I do not know whether I'm right in this list, but since the subject is
> > mutual so common
> > in development and GIT, I have the strong feeling I'm right here.
> >
> > Im quite new to git, so apologizes for any inconvenience reading my
> > question.
> >
> > Using poudriere on 14-CURRENT to create a selection of packages also
> > includes updating
> > the ports tree on a regular basis. I maintain some "special" ports not
> > official part of
> > the FreeBSD ports tree and some other ports are part of those I'm supposed
> > to maintain. I
> > keep personally track of the changes in a git repo of my own.
> >
> > Now I'd like to "overlay" the official portas repo by that of mine to
> > include changes.
> > With SVN, there was no problem to have local changes not overwritten by
> > regular updates
> > of the ports tree as long as the specific port in question wasn't updated
> > by the official
> > site. In git, this behaviour seems to have changed, any changes I made so
> > far are gone
> > after poudriere is adviced to update the tree.
> >
> > I'd like to ask how FreeBSD developers and maintainers do the trick. If
> > there is an
> > official cookbook fpr maintainers (I haven't found it yet ...), please be
> > so kind and
> > refer to it. Any advice is welcome.
> >  
> 
> tl;dr: branches are cheap and well supported in git. You just make a branch
> for your
> local changes, and update that however you see fit.
> 
> For ports I have like that, I've just created a branch in git. I rebase the
> branch forward
> each time I update. For me, though, the branch is mostly uncommitted in
> upstream
> changes that may not be ready for some reason... There's two ways to do
> this.
> You can just merge, which is OK if you aren't upstreaming the changes, or
> you can
> rebase if the changes or a subset of the changes likely will end up in
> FreeBSD.
> 
> Others might recommend stash, but I find it too unwieldy for more than a
> couple
> of things.
> 
> So the workflow would be:
> 
> git clone -o freebsd  freebsd-ports
> cd freebsd-ports
> git checkout -b hartmann-specials
> 
> 
> Now, you can use poudriere-ports with the -B hartmann-specials and your
> local repo as the source of truth.
> 
> to update
> 
> cd freebsd-ports
> git checkout main
> git pull --rebase  # or --ff-only, I use
> --rebase because I push and it's habit
> git rebase -i main hartmann-specials
> 
> Note: if you need to have multiple trees with this branch you are modifying,
> then a git pull --rebase will let you cope with the forced-pushes this
> method
> would require. You can also do it with git merge on hartmann-specials if you
> don't need to keep the changes separate or you have a lot of downstreams
> that would get cranky, which doesn't sound like the case here.
> 
> Warner

As I wrote, I'm quite new to git and always surprised. Thanks for the help and
the neat tip. I'll try.

Kind regards and thanks,

Oliver

Re: git: "overlay" of own remote-branch on official freebsd-ports repo

[FreeBSD User]

On Tue, 12 Oct 2021 18:01:56 +
Brooks Davis  wrote:

> On Tue, Oct 12, 2021 at 05:31:48PM +0200, FreeBSD User wrote:
> > I'd like to ask how FreeBSD developers and maintainers do the trick. If
> > there is an official cookbook fpr maintainers (I haven't found it yet ...),
> > please be so kind and refer to it. Any advice is welcome.  
> 
> If you only want to add extra ports, I'd recommend maintaining a
> separate repo for use with the ports collection's under-documented
> overlay feature.  This avoids the need to rebase or merge your trees.
> 
> You create the overlay in poudriere with something like:
> 
> poudriere ports -c -p cheri-ports-overlay -U
> https://github.com/CTSRD-CHERI/cheri-ports-overlay.git -m git -B main
> 
> You then use it by adding -O cheri-ports-overlay to your other poudriere
> commands like poudriere bulk.
> 
> Note that you may need to install poudriere-devel or install it by hand
> to get this feature.
> 
> -- Brooks

Hello,

that sounds very good and usefull.

Thank you very much

Kind regards,

Oliver

Re: git: "overlay" of own remote-branch on official freebsd-ports repo

[Brooks Davis]

On Tue, Oct 12, 2021 at 05:31:48PM +0200, FreeBSD User wrote:
> I'd like to ask how FreeBSD developers and maintainers do the trick. If there 
> is an
> official cookbook fpr maintainers (I haven't found it yet ...), please be so 
> kind and
> refer to it. Any advice is welcome.

If you only want to add extra ports, I'd recommend maintaining a
separate repo for use with the ports collection's under-documented
overlay feature.  This avoids the need to rebase or merge your trees.

You create the overlay in poudriere with something like:

poudriere ports -c -p cheri-ports-overlay -U 
https://github.com/CTSRD-CHERI/cheri-ports-overlay.git -m git -B main

You then use it by adding -O cheri-ports-overlay to your other poudriere
commands like poudriere bulk.

Note that you may need to install poudriere-devel or install it by hand
to get this feature.

-- Brooks


signature.asc
Description: PGP signature

Re: git: "overlay" of own remote-branch on official freebsd-ports repo

[Felix Palmen]

* Warner Losh  [20211012 10:01]:
> tl;dr: branches are cheap and well supported in git. You just make a branch
> for your
> local changes, and update that however you see fit.
> 
> For ports I have like that, I've just created a branch in git. I rebase the
> branch forward
> each time I update.

+1, I do basically the same. I'd say having a local branch in your local
repo is one ot the killer features of git over svn.

I just do a few details slightly different, maybe it helps as one
possible example. I only fetch 'main' from the official repo and rebase
my 'local' branch onto that. During the rebase, it's easy to solve any
conflicts with changes in 'main'. You still keep a list of your local
commits, well organized.

I recently decided to also publish my local branch. So now I just have
two remotes on my local ports repo, the official FreeBSD repo and
another one on github, looking like this:

  # git remote -v
  origin  https://github.com/Zirias/zfbsd-ports.git (fetch)
  origin  https://github.com/Zirias/zfbsd-ports.git (push)
  upstreamhttps://git.freebsd.org/ports.git (fetch)
  upstreamhttps://git.freebsd.org/ports.git (push)

With that and remote branch tracking set up correctly, I just need these
commands (I have them in a script) to update from the official repo,
rebase and force push to my github repo:

  git checkout local
  git fetch upstream main:main && git rebase main
  git push -f --all

BTW, git leaves working copy changes alone if possible, so maybe it's
poudriere's way of using git that erases them – just use git directly
instead of poudriere. And then, I always hated having random local
changes not organized in any way, this quickly grows into a maintenance
nightmare, so I'm very thankful I can now easily use git branches
instead.

-- 
 Dipl.-Inform. Felix Palmen ,.//..
 {web}  http://palmen-it.de  {jabber} [see email]   ,//palmen-it.de
 {pgp public key} http://palmen-it.de/pub.txt   //   """""""""""
 {pgp fingerprint} A891 3D55 5F2E 3A74 3965 B997 3EF2 8B0A BC02 DA2A


signature.asc
Description: PGP signature

Re: git: "overlay" of own remote-branch on official freebsd-ports repo

[Warner Losh]

On Tue, Oct 12, 2021 at 9:32 AM FreeBSD User  wrote:

> I do not know whether I'm right in this list, but since the subject is
> mutual so common
> in development and GIT, I have the strong feeling I'm right here.
>
> Im quite new to git, so apologizes for any inconvenience reading my
> question.
>
> Using poudriere on 14-CURRENT to create a selection of packages also
> includes updating
> the ports tree on a regular basis. I maintain some "special" ports not
> official part of
> the FreeBSD ports tree and some other ports are part of those I'm supposed
> to maintain. I
> keep personally track of the changes in a git repo of my own.
>
> Now I'd like to "overlay" the official portas repo by that of mine to
> include changes.
> With SVN, there was no problem to have local changes not overwritten by
> regular updates
> of the ports tree as long as the specific port in question wasn't updated
> by the official
> site. In git, this behaviour seems to have changed, any changes I made so
> far are gone
> after poudriere is adviced to update the tree.
>
> I'd like to ask how FreeBSD developers and maintainers do the trick. If
> there is an
> official cookbook fpr maintainers (I haven't found it yet ...), please be
> so kind and
> refer to it. Any advice is welcome.
>

tl;dr: branches are cheap and well supported in git. You just make a branch
for your
local changes, and update that however you see fit.

For ports I have like that, I've just created a branch in git. I rebase the
branch forward
each time I update. For me, though, the branch is mostly uncommitted in
upstream
changes that may not be ready for some reason... There's two ways to do
this.
You can just merge, which is OK if you aren't upstreaming the changes, or
you can
rebase if the changes or a subset of the changes likely will end up in
FreeBSD.

Others might recommend stash, but I find it too unwieldy for more than a
couple
of things.

So the workflow would be:

git clone -o freebsd  freebsd-ports
cd freebsd-ports
git checkout -b hartmann-specials


Now, you can use poudriere-ports with the -B hartmann-specials and your
local repo as the source of truth.

to update

cd freebsd-ports
git checkout main
git pull --rebase  # or --ff-only, I use
--rebase because I push and it's habit
git rebase -i main hartmann-specials

Note: if you need to have multiple trees with this branch you are modifying,
then a git pull --rebase will let you cope with the forced-pushes this
method
would require. You can also do it with git merge on hartmann-specials if you
don't need to keep the changes separate or you have a lot of downstreams
that would get cranky, which doesn't sound like the case here.

Warner

git: "overlay" of own remote-branch on official freebsd-ports repo

[FreeBSD User]

I do not know whether I'm right in this list, but since the subject is mutual 
so common
in development and GIT, I have the strong feeling I'm right here.

Im quite new to git, so apologizes for any inconvenience reading my question.

Using poudriere on 14-CURRENT to create a selection of packages also includes 
updating
the ports tree on a regular basis. I maintain some "special" ports not official 
part of
the FreeBSD ports tree and some other ports are part of those I'm supposed to 
maintain. I
keep personally track of the changes in a git repo of my own.

Now I'd like to "overlay" the official portas repo by that of mine to include 
changes.
With SVN, there was no problem to have local changes not overwritten by regular 
updates
of the ports tree as long as the specific port in question wasn't updated by 
the official
site. In git, this behaviour seems to have changed, any changes I made so far 
are gone
after poudriere is adviced to update the tree.

I'd like to ask how FreeBSD developers and maintainers do the trick. If there 
is an
official cookbook fpr maintainers (I haven't found it yet ...), please be so 
kind and
refer to it. Any advice is welcome.

Kind regards and thanks in advance,

O. Hartmann

Re: FreeBSD base pkg (packaging) and critical ports build alongside

[Chris]

On 2021-09-29 01:28, FreeBSD User wrote:

Hello,

I use FreeBSD-base packages built on self hosted systems to update 13-STABLE
and CURRENT hosts.  I run into the problem, that the packages of the FreeBSD
base, built via the FreeBSD framework and from most recent 13-STABLE 
sources,
are often oit of synchronisation with our poudriere packaging builders, that 
is

especially true for critical ports with kernel modules, like i915 drm,
virtualbox and so on. The problem is, obviously, barehanded: 13-STABLE 
sources
and probably the API changes more rapidly than those of the appropriate 
builder

hosts for poudriere and since it takes a bunch of days to build a whole
poudriere packages repository, there is often a gap between the revision of 
the

kernel and the port containing kernel modules.

So, the question is: how can I add ports to the building process of the 
FreeBSD

sources tree in the way they get build every time I build the FreeBSD-base
packages alongside the OS?
The simple answer is; by keeping/getting both trees where you want them 
before

you initiate a build.

This is what I do;

Pick some point in time, or in git(1) parlance; hash/revision. I then
git co/clone git hash/revision for both trees.
I then fire off a build for both. Creating $BASE install(s)/images &&
packages. Since I'm subscribed to the freebsd-security-notifications
ML. I get announcements whenever FreeBSD pushes security patches. I then
check the git log for when the patch(s) was/were pushed/committed. Then 
update

the affected tree to that hash/revision, and update the ports tree to
the same place in time. Then build both trees and update the affected
boxes (servers/hosts). You probably will also want to monitor the commit
list (WARNING it's a high volume list) for CVE notices. So as to keep
your ports tree safe. Simply do the same -- update the ports tree that
contains the CVE commit && build up/deploy your packages from it.
Customization (adding ports drivers to your $BASE (src) build:
Simply add
PORTS_MODULES=
to your make.conf(5). For instance; your i915 driver.

That pretty much covers it I think. :-)

See also; man make(1) man make.conf(5) && man ports(7)

HTH

--Chris


Thanks in advance,

oh


0xBDE49540.asc
Description: application/pgp-keys

Re: FreeBSD base pkg (packaging) and critical ports build alongside

[Emmanuel Vadot]

On Wed, 29 Sep 2021 09:42:46 +0100
David Chisnall  wrote:

> Hi,
> 
> I think your best option would be to do the opposite of what you 
> suggest.  Poudriere can build pkgbase sets from a source tree and 
> populate a jail from them.  

 No it doesn't populate the jail from the pkgbase sets, the jail will
still be populated using make installworld and friends.
 I still need to add the possibility to create a jail (and update it)
just from a pkgbase but haven't got time yet for this.

> The flow that I'd suggest is:
> 
>   - Poudriere jail to build a jail from an existing source tree.
>   - If there are kernel changes, install the packages on the package 
> builder and reboot.
>   - Poudriere bulk in the new jail to build the new package set.
> 
> Note: You can *normally* skip the second step (drm ports, for example, 
> will be built against the new kernel sources in the jail, though they 
> might not be loadable on the host) but there's no guarantee that you can 
> run a newer userland on an older kernel so things may break.
> 
> If you enable reproduceable builds in the src.conf that you use for 
> building the jail then you should be able to just diff the kernel binary 
> to see if anything has changed.
> 
> If you have bhyve or are running on a cloud platform then you can 
> replace the second step with a poudriere image invocation to build a VM 
> image containing poudriere and your newly-built base system and deploy 
> this to build the packages.  I'm planning on working on some tooling to 
> do this in Azure with GitHub Actions.
> 
> Note that poudriere uses packages installed on the host system to build 
> a jail.  If you have, for example, installed llvm12 then you can put a 
> line in your src-env.conf for the jail to tell it to use that as an 
> external toolchain and skip the toolchain-bootstrap phase of the build. 
>   This means that the base-build is fairly fast even on quite modest 
> hardware (it still builds clang, but at least it does it only once).
> 
> David
> 
> 
> On 29/09/2021 09:28, FreeBSD User wrote:
> > Hello,
> > 
> > I use FreeBSD-base packages built on self hosted systems to update 13-STABLE
> > and CURRENT hosts.  I run into the problem, that the packages of the FreeBSD
> > base, built via the FreeBSD framework and from most recent 13-STABLE 
> > sources,
> > are often oit of synchronisation with our poudriere packaging builders, 
> > that is
> > especially true for critical ports with kernel modules, like i915 drm,
> > virtualbox and so on. The problem is, obviously, barehanded: 13-STABLE 
> > sources
> > and probably the API changes more rapidly than those of the appropriate 
> > builder
> > hosts for poudriere and since it takes a bunch of days to build a whole
> > poudriere packages repository, there is often a gap between the revision of 
> > the
> > kernel and the port containing kernel modules.
> > 
> > So, the question is: how can I add ports to the building process of the 
> > FreeBSD
> > sources tree in the way they get build every time I build the FreeBSD-base
> > packages alongside the OS?
> > 
> > Thanks in advance,
> > 
> > oh
> > 
> 


-- 
Emmanuel Vadot  

Re: FreeBSD base pkg (packaging) and critical ports build alongside

[David Chisnall]

Hi,

I think your best option would be to do the opposite of what you 
suggest.  Poudriere can build pkgbase sets from a source tree and 
populate a jail from them.  The flow that I'd suggest is:


 - Poudriere jail to build a jail from an existing source tree.
 - If there are kernel changes, install the packages on the package 
builder and reboot.

 - Poudriere bulk in the new jail to build the new package set.

Note: You can *normally* skip the second step (drm ports, for example, 
will be built against the new kernel sources in the jail, though they 
might not be loadable on the host) but there's no guarantee that you can 
run a newer userland on an older kernel so things may break.


If you enable reproduceable builds in the src.conf that you use for 
building the jail then you should be able to just diff the kernel binary 
to see if anything has changed.


If you have bhyve or are running on a cloud platform then you can 
replace the second step with a poudriere image invocation to build a VM 
image containing poudriere and your newly-built base system and deploy 
this to build the packages.  I'm planning on working on some tooling to 
do this in Azure with GitHub Actions.


Note that poudriere uses packages installed on the host system to build 
a jail.  If you have, for example, installed llvm12 then you can put a 
line in your src-env.conf for the jail to tell it to use that as an 
external toolchain and skip the toolchain-bootstrap phase of the build. 
 This means that the base-build is fairly fast even on quite modest 
hardware (it still builds clang, but at least it does it only once).


David


On 29/09/2021 09:28, FreeBSD User wrote:

Hello,

I use FreeBSD-base packages built on self hosted systems to update 13-STABLE
and CURRENT hosts.  I run into the problem, that the packages of the FreeBSD
base, built via the FreeBSD framework and from most recent 13-STABLE sources,
are often oit of synchronisation with our poudriere packaging builders, that is
especially true for critical ports with kernel modules, like i915 drm,
virtualbox and so on. The problem is, obviously, barehanded: 13-STABLE sources
and probably the API changes more rapidly than those of the appropriate builder
hosts for poudriere and since it takes a bunch of days to build a whole
poudriere packages repository, there is often a gap between the revision of the
kernel and the port containing kernel modules.

So, the question is: how can I add ports to the building process of the FreeBSD
sources tree in the way they get build every time I build the FreeBSD-base
packages alongside the OS?

Thanks in advance,

oh

Re: FreeBSD base pkg (packaging) and critical ports build alongside

[Emmanuel Vadot]

 Hello,

On Wed, 29 Sep 2021 10:28:05 +0200
FreeBSD User  wrote:

> Hello,
> 
> I use FreeBSD-base packages built on self hosted systems to update 13-STABLE
> and CURRENT hosts.  I run into the problem, that the packages of the FreeBSD
> base, built via the FreeBSD framework and from most recent 13-STABLE sources,
> are often oit of synchronisation with our poudriere packaging builders, that 
> is
> especially true for critical ports with kernel modules, like i915 drm,
> virtualbox and so on. The problem is, obviously, barehanded: 13-STABLE sources
> and probably the API changes more rapidly than those of the appropriate 
> builder
> hosts for poudriere and since it takes a bunch of days to build a whole
> poudriere packages repository, there is often a gap between the revision of 
> the
> kernel and the port containing kernel modules.
> 
> So, the question is: how can I add ports to the building process of the 
> FreeBSD
> sources tree in the way they get build every time I build the FreeBSD-base
> packages alongside the OS?
> 
> Thanks in advance,
> 
> oh
> 

 What I do to have packages (from ports) and pkgbase in sync is that I
use poudriere to also build pkgbase.
 It's available in poudriere-devel using -B when creating the jail.
 Then I simply cpdup the packages and pkgbase at the same time at the
end of the package build.
 poudriere knows how to do make update-package for pkgbase so your
machine will just update the modified packages (provided that you are
on a RELEASE/STABLE branch or have WITH_REPRODUCIBLE_BUILD=yes in your
jail src.conf for CURRENT).
 There is still one problem with that approch for kernel modules, they
will be recompiled but as the version isn't bumped it will not be
upgraded, so I just pkg install -f drm-devel-kmod from time to time.

 Cheers,

-- 
Emmanuel Vadot  

FreeBSD base pkg (packaging) and critical ports build alongside

[FreeBSD User]

Hello,

I use FreeBSD-base packages built on self hosted systems to update 13-STABLE
and CURRENT hosts.  I run into the problem, that the packages of the FreeBSD
base, built via the FreeBSD framework and from most recent 13-STABLE sources,
are often oit of synchronisation with our poudriere packaging builders, that is
especially true for critical ports with kernel modules, like i915 drm,
virtualbox and so on. The problem is, obviously, barehanded: 13-STABLE sources
and probably the API changes more rapidly than those of the appropriate builder
hosts for poudriere and since it takes a bunch of days to build a whole
poudriere packages repository, there is often a gap between the revision of the
kernel and the port containing kernel modules.

So, the question is: how can I add ports to the building process of the FreeBSD
sources tree in the way they get build every time I build the FreeBSD-base
packages alongside the OS?

Thanks in advance,

oh

"ahci0: AHCI v1.30 with 6 6Gbps ports" while only 2 of those are SATA3

[Yuri Pankov]

That is on somewhat older Supermicro X9DRI-LN4F+ board:

ahci0:  port
0x9050-0x9057,0x9040-0x9043,0x9030-0x9037,0x9020-0x9023,0x9000-0x901f
mem 0xdfa21000-0xdfa217ff irq 18 at device 31.2 numa-domain 0 on pci0
ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported

and then I was confused by messages about SATA3 drives connected to
ports 2-5 (see the "transfers" line):

ada1 at ahcich2 bus 0 scbus2 target 0 lun 0
ada1:  ACS-3 ATA SATA 3.x device
ada1: Serial Number WD-WX32D7088CCV
ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 3815447MB (7814037168 512 byte sectors)

Checking the board manual made it clear than only ports 0-1 are SATA3,
and 2-5 are indeed SATA2.  While the issue is purely cosmetic, I wonder
if it's possible to print real port speeds for the controller, i.e. if
this information is available to driver?
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Build of some ports hang up on recent 14-CURRENT amd64

[Yasuhiro Kimura]

From: Yasuhiro Kimura 
Subject: Re: Build of some ports hang up on recent 14-CURRENT amd64
Date: Sat, 10 Apr 2021 06:30:22 +0900 (JST)

> From: Michael Butler via freebsd-current 
> Subject: Re: Build of some ports hang up on recent 14-CURRENT amd64
> Date: Fri, 9 Apr 2021 17:20:44 -0400
> 
>> This is likely fixed as of ~30 mins ago on commit
>> e8b9c508b7ae5be618ada089103468c400e465cd
>> 
>> The cause appears to have been commit d36d6816151705907393889
>> 
>>  imb
> 
> Thanks for information. I'll try another update.

After the update to 1a7fe55ab8c I confirmed build of the ports
completes successfuly.

---
Yasuhiro Kimura
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Build of some ports hang up on recent 14-CURRENT amd64

[Yasuhiro Kimura]

From: Michael Butler via freebsd-current 
Subject: Re: Build of some ports hang up on recent 14-CURRENT amd64
Date: Fri, 9 Apr 2021 17:20:44 -0400

> This is likely fixed as of ~30 mins ago on commit
> e8b9c508b7ae5be618ada089103468c400e465cd
> 
> The cause appears to have been commit d36d6816151705907393889
> 
>   imb

Thanks for information. I'll try another update.

---
Yasuhiro Kimura
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Build of some ports hang up on recent 14-CURRENT amd64

[Michael Butler via freebsd-current]

This is likely fixed as of ~30 mins ago on commit 
e8b9c508b7ae5be618ada089103468c400e465cd


The cause appears to have been commit d36d6816151705907393889

imb

On 4/9/21 4:55 PM, Yasuhiro Kimura wrote:

yasu@rolling-vm-freebsd1[1044]% uname -a
FreeBSD rolling-vm-freebsd1.home.utahime.org 14.0-CURRENT FreeBSD 14.0-CURRENT 
#0 main-n245912-f2400e6e832: Sat Apr 10 01:42:33 JST 2021 
ro...@rolling-vm-freebsd1.home.utahime.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC
  amd64

After the update from 36d6e65722e to f2400e6e832 I noticed build of
some ports hang up in the middle of it on my 14-CURRENT amd64 host.

One of such examples is security/py-cryptography.

--
root@rolling-vm-freebsd1[1008]# make
===>  License APACHE20 BSD3CLAUSE accepted by the user
===>   py39-cryptography-3.3.2 depends on file: /usr/local/sbin/pkg - found
===> Fetching all distfiles required by py39-cryptography-3.3.2 for building
===>  Extracting for py39-cryptography-3.3.2
=> SHA256 Checksum OK for cryptography-3.3.2.tar.gz.
===>  Patching for py39-cryptography-3.3.2
===>   py39-cryptography-3.3.2 depends on package: py39-cffi>=1.8 - found
===>   py39-cryptography-3.3.2 depends on package: py39-setuptools>0 - found
===>   py39-cryptography-3.3.2 depends on file: /usr/local/bin/python3.9 - found
===>  Configuring for py39-cryptography-3.3.2
running config
--

And another one is devel/arcanist-lib.

--
root@rolling-vm-freebsd1[1023]# make
===>  License APACHE20 accepted by the user
===>   arcanist-lib-php74-20210113 depends on file: /usr/local/sbin/pkg - found
===> Fetching all distfiles required by arcanist-lib-php74-20210113 for building
===>  Extracting for arcanist-lib-php74-20210113
=> SHA256 Checksum OK for phacility-arcanist-20210113-b2e715f_GH0.tar.gz.
===>  Patching for arcanist-lib-php74-20210113
===>  Applying FreeBSD patches for arcanist-lib-php74-20210113 from 
/usr/ports/devel/arcanist-lib/files
===>  Configuring for arcanist-lib-php74-20210113
===>  Staging for arcanist-lib-php74-20210113
===>   arcanist-lib-php74-20210113 depends on file: 
/usr/local/include/php/main/php.h - found
===>   arcanist-lib-php74-20210113 depends on file: 
/usr/local/lib/php/20190902/curl.so - found
===>   arcanist-lib-php74-20210113 depends on file: 
/usr/local/lib/php/20190902/dom.so - found
===>   arcanist-lib-php74-20210113 depends on file: 
/usr/local/lib/php/20190902/json.so - found
===>   arcanist-lib-php74-20210113 depends on file: 
/usr/local/lib/php/20190902/simplexml.so - found
===>   arcanist-lib-php74-20210113 depends on file: 
/usr/local/lib/php/20190902/zlib.so - found
===>   arcanist-lib-php74-20210113 depends on file: 
/usr/local/lib/php/20190902/mbstring.so - found
===>   Generating temporary packing list
cd /usr/ports/devel/arcanist-lib/work-php74/arcanist-b2e715f ; /bin/pax -rw * 
/usr/ports/devel/arcanist-lib/work-php74/stage/usr/local/lib/php/arcanist
install -l rs 
/usr/ports/devel/arcanist-lib/work-php74/stage/usr/local/lib/php/arcanist/support/shell/hooks/bash-completion.sh
  
/usr/ports/devel/arcanist-lib/work-php74/stage/usr/local/share/bash-completion/completions/arc
/usr/ports/devel/arcanist-lib/work-php74/stage/usr/local/lib/php/arcanist/bin/arc
 shell-complete --generate
  GENERATE  Generating shell completion rules...
  RULES  Wrote updated completion rules for "bash" to: 
work-php74/stage/usr/local/lib/php/arcanist/support/shell/rules/bash-rules.sh.
  NOTE  You may need to open a new terminal window or launch a new shell before 
the changes take effect.
--

The problem also happens with poudriere.

I tried boot with old 36d6e65722e kernel but the problem still
happens. So the cause may be older than it.

Does anyone else have the same problem?

Best Regards.

---
Yasuhiro Kimura
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"



___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Build of some ports hang up on recent 14-CURRENT amd64

[Yasuhiro Kimura]

yasu@rolling-vm-freebsd1[1044]% uname -a
FreeBSD rolling-vm-freebsd1.home.utahime.org 14.0-CURRENT FreeBSD 14.0-CURRENT 
#0 main-n245912-f2400e6e832: Sat Apr 10 01:42:33 JST 2021 
ro...@rolling-vm-freebsd1.home.utahime.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC
  amd64

After the update from 36d6e65722e to f2400e6e832 I noticed build of
some ports hang up in the middle of it on my 14-CURRENT amd64 host.

One of such examples is security/py-cryptography.

--
root@rolling-vm-freebsd1[1008]# make
===>  License APACHE20 BSD3CLAUSE accepted by the user
===>   py39-cryptography-3.3.2 depends on file: /usr/local/sbin/pkg - found
===> Fetching all distfiles required by py39-cryptography-3.3.2 for building
===>  Extracting for py39-cryptography-3.3.2
=> SHA256 Checksum OK for cryptography-3.3.2.tar.gz.
===>  Patching for py39-cryptography-3.3.2
===>   py39-cryptography-3.3.2 depends on package: py39-cffi>=1.8 - found
===>   py39-cryptography-3.3.2 depends on package: py39-setuptools>0 - found
===>   py39-cryptography-3.3.2 depends on file: /usr/local/bin/python3.9 - found
===>  Configuring for py39-cryptography-3.3.2
running config
--

And another one is devel/arcanist-lib.

--
root@rolling-vm-freebsd1[1023]# make
===>  License APACHE20 accepted by the user
===>   arcanist-lib-php74-20210113 depends on file: /usr/local/sbin/pkg - found
===> Fetching all distfiles required by arcanist-lib-php74-20210113 for building
===>  Extracting for arcanist-lib-php74-20210113
=> SHA256 Checksum OK for phacility-arcanist-20210113-b2e715f_GH0.tar.gz.
===>  Patching for arcanist-lib-php74-20210113
===>  Applying FreeBSD patches for arcanist-lib-php74-20210113 from 
/usr/ports/devel/arcanist-lib/files
===>  Configuring for arcanist-lib-php74-20210113
===>  Staging for arcanist-lib-php74-20210113
===>   arcanist-lib-php74-20210113 depends on file: 
/usr/local/include/php/main/php.h - found
===>   arcanist-lib-php74-20210113 depends on file: 
/usr/local/lib/php/20190902/curl.so - found
===>   arcanist-lib-php74-20210113 depends on file: 
/usr/local/lib/php/20190902/dom.so - found
===>   arcanist-lib-php74-20210113 depends on file: 
/usr/local/lib/php/20190902/json.so - found
===>   arcanist-lib-php74-20210113 depends on file: 
/usr/local/lib/php/20190902/simplexml.so - found
===>   arcanist-lib-php74-20210113 depends on file: 
/usr/local/lib/php/20190902/zlib.so - found
===>   arcanist-lib-php74-20210113 depends on file: 
/usr/local/lib/php/20190902/mbstring.so - found
===>   Generating temporary packing list
cd /usr/ports/devel/arcanist-lib/work-php74/arcanist-b2e715f ; /bin/pax -rw * 
/usr/ports/devel/arcanist-lib/work-php74/stage/usr/local/lib/php/arcanist
install -l rs 
/usr/ports/devel/arcanist-lib/work-php74/stage/usr/local/lib/php/arcanist/support/shell/hooks/bash-completion.sh
  
/usr/ports/devel/arcanist-lib/work-php74/stage/usr/local/share/bash-completion/completions/arc
/usr/ports/devel/arcanist-lib/work-php74/stage/usr/local/lib/php/arcanist/bin/arc
 shell-complete --generate
 GENERATE  Generating shell completion rules...
 RULES  Wrote updated completion rules for "bash" to: 
work-php74/stage/usr/local/lib/php/arcanist/support/shell/rules/bash-rules.sh.
 NOTE  You may need to open a new terminal window or launch a new shell before 
the changes take effect.
--

The problem also happens with poudriere.

I tried boot with old 36d6e65722e kernel but the problem still
happens. So the cause may be older than it.

Does anyone else have the same problem?

Best Regards.

---
Yasuhiro Kimura
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: On 14-CURRENT: no ports options anymore?

[John Baldwin]

On 3/13/21 12:58 PM, Guido Falsi via freebsd-current wrote:

On 13/03/21 20:17, Hartmann, O. wrote:

Since I moved on to 14-CURRENT, I face a very strange behaviour when trying to 
set
options via "make config" or via poudriere accordingly. I always get "===> 
Options
unchanged" (when options has been already set and I'd expect a dialog menu).
This misbehaviour is throughout ALL 14-CURRENT systems (the oldest is at FreeBSD
14.0-CURRENT #49 main-n245422-cecfaf9bede9: Fri Mar 12 16:08:09 CET 2021 amd64).

I do not see such a behaviour with 13-STABLE, 12-STABLE, 12.2-RELENG.

How to fix this? What happened?


I encountered something similar, some base shared library has changed,
guess this is related with the ncurses changes in base.

If I remember correctly force reinstalling dialog4ports package fixed
it. Make sure you reinstall a freshly rebuilt one.

Most probably anything using ncurses will require rebuild/reinstall.

The cause is dialog4ports failing to start and the system sees no option
changed.

If that's not enough try

# ldd -v /usr/local/bin/dialog4ports

And see if it reports some useful information.


There was an ABI breakage for ncurses that broke 12.x dialog4ports binaries.
The shared library versions for everything that depended on ncurses were bumped
for 13 and 14 after the branch of stable/13.0 (commit 6e1fe6d26ea2).  After
that commit, if you upgraded from 12 to 13 you should have been fine, but if
you had updated before that, the 12.x dialog4ports was still going to fail
as the 12.x version of those libraries were already broken.  I haven't checked
to see if the affected libraries have been added to misc/compat12x.

--
John Baldwin
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: On 14-CURRENT: no ports options anymore?

[Alastair Hogge]

On 2021-03-18 18:07, Jamie Landeg-Jones wrote:
> "Hartmann, O."  wrote:
> 
>> On Sat, 13 Mar 2021 15:13:15 -0500
>> Michael Butler via freebsd-current  wrote:
>>
>> > On 3/13/21 3:00 PM, Hartmann, O. wrote:
>> > > On Sat, 13 Mar 2021 19:52:47 + (UTC)
>> > > Filippo Moretti via freebsd-current  wrote:
>> > >
>> > >>
>> > >> I had the same problem in console modeFiippo
>> > >>  On Saturday, March 13, 2021, 8:33:57 PM GMT+1, Stefan Esser 
>> > >> 
>> > >> wrote:
>> > >>
>> > >>   Am 13.03.21 um 20:17 schrieb Hartmann, O.:
>> > >>> Since I moved on to 14-CURRENT, I face a very strange behaviour when 
>> > >>> trying to set
>> > >>> options via "make config" or via poudriere accordingly. I always get 
>> > >>> "===> Options
>> > >>> unchanged" (when options has been already set and I'd expect a dialog 
>> > >>> menu).
>> > >>> This misbehaviour is throughout ALL 14-CURRENT systems (the oldest is 
>> > >>> at FreeBSD
>> > >>> 14.0-CURRENT #49 main-n245422-cecfaf9bede9: Fri Mar 12 16:08:09 CET 
>> > >>> 2021 amd64).
>> >
>> > I ran into something similar where dialog4ports would dump core after an
>> > upgrade. Rebuilding the port (ports-mgmt/dialog4ports) solved it for me,
>> >
>> >imb
>>
>> Thanks, recompilig ports-mgmt/dialog4ports solved the problem! I was sure 
>> that after some
>> ncurses changes in 14-CURRENT I've rebuilt every single port on all 
>> 14-CURRENT systems
>> after that incident, bad luck.
> 
> This bit me a few years ago - in my case, the terminal type wasn't
> recognised. If dialog4ports
> doesn't run successfully, the error is ignored by the ports scripts.
> This was on a new
> install (hence the reason I hadn't yet installed my termcap), and it
> meant that before I
> noticed, quite a number of ports got installed without the options I wanted.
> 
> Since then, I've patched bsd.port.mk with:
> 
> --- bsd.port.mk.orig2017-06-06 17:38:00.0 +0100
> +++ bsd.port.mk 2017-06-08 01:31:36.320294000 +0100
> @@ -4729,8 +4729,8 @@
> trap "${RM} $${TMPOPTIONSFILE}; exit 1" 1 2 3 5 10 13 15; \
> ${SETENV} ${D4P_ENV} ${SH} ${SCRIPTSDIR}/dialog4ports.sh
> $${TMPOPTIONSFILE} || { \
> ${RM} $${TMPOPTIONSFILE}; \
> -   ${ECHO_MSG} "===> Options unchanged"; \
> -   exit 0; \
> +   ${ECHO_MSG} "===> ERROR: Options unchanged"; \
> +   exit 1; \
> }; \
> ${ECHO_CMD}; \
> if [ ! -e $${TMPOPTIONSFILE} ]; then \
> 
> I never submitted this patch, because the way it was written seemed
> intentional. Maybe this should be reviewed?

Something like that would be great, please can we get a more informative
message? I was bitten by this a few times recently.
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: On 14-CURRENT: no ports options anymore?

[Jamie Landeg-Jones]

"Hartmann, O."  wrote:

> On Sat, 13 Mar 2021 15:13:15 -0500
> Michael Butler via freebsd-current  wrote:
>
> > On 3/13/21 3:00 PM, Hartmann, O. wrote:
> > > On Sat, 13 Mar 2021 19:52:47 + (UTC)
> > > Filippo Moretti via freebsd-current  wrote:
> > >   
> > >>   
> > >> I had the same problem in console modeFiippo
> > >>  On Saturday, March 13, 2021, 8:33:57 PM GMT+1, Stefan Esser 
> > >> 
> > >> wrote:
> > >>
> > >>   Am 13.03.21 um 20:17 schrieb Hartmann, O.:  
> > >>> Since I moved on to 14-CURRENT, I face a very strange behaviour when 
> > >>> trying to set
> > >>> options via "make config" or via poudriere accordingly. I always get 
> > >>> "===> Options
> > >>> unchanged" (when options has been already set and I'd expect a dialog 
> > >>> menu).
> > >>> This misbehaviour is throughout ALL 14-CURRENT systems (the oldest is 
> > >>> at FreeBSD
> > >>> 14.0-CURRENT #49 main-n245422-cecfaf9bede9: Fri Mar 12 16:08:09 CET 
> > >>> 2021 amd64).
> > 
> > I ran into something similar where dialog4ports would dump core after an 
> > upgrade. Rebuilding the port (ports-mgmt/dialog4ports) solved it for me,
> > 
> > imb
>
> Thanks, recompilig ports-mgmt/dialog4ports solved the problem! I was sure 
> that after some
> ncurses changes in 14-CURRENT I've rebuilt every single port on all 
> 14-CURRENT systems
> after that incident, bad luck.

This bit me a few years ago - in my case, the terminal type wasn't recognised. 
If dialog4ports
doesn't run successfully, the error is ignored by the ports scripts. This was 
on a new
install (hence the reason I hadn't yet installed my termcap), and it meant that 
before I
noticed, quite a number of ports got installed without the options I wanted.

Since then, I've patched bsd.port.mk with:

--- bsd.port.mk.orig2017-06-06 17:38:00.0 +0100
+++ bsd.port.mk 2017-06-08 01:31:36.320294000 +0100
@@ -4729,8 +4729,8 @@
trap "${RM} $${TMPOPTIONSFILE}; exit 1" 1 2 3 5 10 13 15; \
${SETENV} ${D4P_ENV} ${SH} ${SCRIPTSDIR}/dialog4ports.sh 
$${TMPOPTIONSFILE} || { \
${RM} $${TMPOPTIONSFILE}; \
-   ${ECHO_MSG} "===> Options unchanged"; \
-   exit 0; \
+   ${ECHO_MSG} "===> ERROR: Options unchanged"; \
+   exit 1; \
}; \
${ECHO_CMD}; \
if [ ! -e $${TMPOPTIONSFILE} ]; then \

I never submitted this patch, because the way it was written seemed 
intentional. Maybe this should be reviewed?

Cheers,
  Jamie

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: On 14-CURRENT: no ports options anymore?

[Dan Mahoney (Ports)]

If this isn’t at least in /usr/ports/UPDATING it sure should be.

-Dan

> On Mar 13, 2021, at 12:58 PM, Guido Falsi via freebsd-ports 
>  wrote:
> 
> On 13/03/21 20:17, Hartmann, O. wrote:
>> Since I moved on to 14-CURRENT, I face a very strange behaviour when trying 
>> to set
>> options via "make config" or via poudriere accordingly. I always get "===> 
>> Options
>> unchanged" (when options has been already set and I'd expect a dialog menu).
>> This misbehaviour is throughout ALL 14-CURRENT systems (the oldest is at 
>> FreeBSD
>> 14.0-CURRENT #49 main-n245422-cecfaf9bede9: Fri Mar 12 16:08:09 CET 2021 
>> amd64).
>> I do not see such a behaviour with 13-STABLE, 12-STABLE, 12.2-RELENG.
>> How to fix this? What happened?
> 
> I encountered something similar, some base shared library has changed, guess 
> this is related with the ncurses changes in base.
> 
> If I remember correctly force reinstalling dialog4ports package fixed it. 
> Make sure you reinstall a freshly rebuilt one.
> 
> Most probably anything using ncurses will require rebuild/reinstall.
> 
> The cause is dialog4ports failing to start and the system sees no option 
> changed.
> 
> If that's not enough try
> 
> # ldd -v /usr/local/bin/dialog4ports
> 
> And see if it reports some useful information.
> 
> -- 
> Guido Falsi 
> ___
> freebsd-po...@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-ports
> To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"