Re: Meson 0.56.2 and Python39

[Ken Cunningham]

I will try it.

If I might say — I would expect that anyone submitting a PR has at least built 
the software on their local system, and used it enough to be sure it at least 
basically works, if not ran the whole test suite.

SO — meson must have worked properly on at least the system that was used to 
submit the PR.

I will (and do) thoroughly test the PRs and commits I personally do — which is 
why the cctools and libmacho and other updates take as long as they do.

I have been reviewing, looking over the changes, checking to build logs and 
checking the installed files on the PRs before I commit them.

Sometimes, I miss things — out of 150 PRs committed, Ryan and Josh have found 
corrections (revbumps, etc) in three or four. So I accept that rate.

But I do not expect to find massive broken changes submitted as PRs (we have 
been seeing some of that at times, hopefully I caught most of it), or otherwise 
wholly untested PRs that are thrown up without even a passing local build.

That is asking too much of PR reviewers to have absolutely no trust in the PR 
submitters.

Ken






> On Feb 2, 2021, at 2:32 PM, Craig Treleaven  wrote:
> 
> Hi:
> 
> It might be that the meson update and/or the switch to python39 has broken 
> builds on older Mac OS versions.  Specifically, I updated dav1d to 0.8.1 and 
> it no longer configures successfully on 10.9 and older versions.  Upstream 
> says that the now-failing configure test (a simple test to see if the 
> compiler can create an executable) has not changed.  
> 
> Am I the only one seeing this?
> 
> The meson project doesn’t seem to have a change log and I haven’t waded 
> through all of the commits between versions.  Is there a test suite with 
> meson?  Have we checked it?  
> 
> Tomorrow, I think I’m going to try to bring up some VM’s and see if 
> meson-log.txt has anything interesting on a failing OS version.
> 
> Craig

Re: invisible universal variant and merger_must_run_binaries

[Ken Cunningham]

> But if "his system will never see that port could be universal" then how 
> could he "download and install that port +universal" in the first place?

that is the crux of it, indeed. 

Should our poor soul, who cannot build the port +universal, still be able to 
download it +universal from the buildbot and use it, if it exists (built by the 
arm64 machine).

K

Re: invisible universal variant and merger_must_run_binaries

[Ken Cunningham]

> On Feb 2, 2021, at 11:22 PM, Ryan Schmidt  wrote:
> 
> On Feb 3, 2021, at 01:12, Ken Cunningham wrote:
> 
>>> But if "his system will never see that port could be universal" then how 
>>> could he "download and install that port +universal" in the first place?
>> 
>> that is the crux of it, indeed. 
>> 
>> Should our poor soul, who cannot build the port +universal, still be able to 
>> download it +universal from the buildbot and use it, if it exists (built by 
>> the arm64 machine).
> 
> At present, no, that would not be possible.
> 
> I consider the buildbot a "nice to have". It makes life easier by 
> precompiling things but it is not essential. When something is not available 
> precompiled, MacPorts builds from source. If something doesn't build 
> correctly from source, let's fix that.
> 

OK — for users out there, then, we should begin to make it clear that building 
+universal on a BigSur Intel machine is not going to work out for a number of 
ports, and although they might find some ports can build universal on BigSur 
Intel, for comprehensive +universal support, they should expect to use BigSur 
arm64.

That is fine with me — IMHO nobody should really be putting anything out that 
hasn’t been actually tried on an actual BigSur arm64 machine anyway.

But should be clear to people what to expect.

Ken

Re: Meson 0.56.2 and Python39

[Ken Cunningham]

It's a rare port that actually passes all the tests.

Best to compare with a current system and the previous meson before you 
conclude anything is wrong for certain.

I've been running around enabling test suites on everything I can, including 
meson 

 , in the hopes of getting people to be more rigorous about such things, but 
I'm often surprised at the poor performance of things on test suites.

K

On Feb 2, 2021, at 17:32, Craig Treleaven  wrote:

>> On Feb 2, 2021, at 7:26 PM, Craig Treleaven  wrote:
>> 
>> Anyway, I have 'port test meson' running on my 10.10 system even though I 
>> don’t expect anything.  I don’t have a working virtualization system at the 
>> moment.  Perhaps tomorrow I can get that back up and try the tests on 10.9 
>> and earlier.  And try to build dav1d to get better log info.
> 
> Actually, the test phase failed on my 10.10 Mac.  I think this is specific 
> test that made it fall over:
> 
> g-ir-scanner: link: /usr/bin/clang -o 
> "/opt/local/var/macports/build/_Users_craigtreleaven_mp_macports-ports_devel_meson/meson/work/meson-0.56.2/b
>  b6aebd73f7/tmp-introspectof76ehud/Meson-1.0" -Os 
> "/opt/local/var/macports/build/_Users_craigtreleaven_mp_macports-ports_devel_meson/meson/work/meson-0.56.2/b
>  b6aebd73f7/tmp-introspectof76ehud/Meson-1.0.o" -L. -Wl,-rpath,. 
> "-L/opt/local/var/macports/build/_Users_craigtreleaven_mp_macports-ports_devel_meson/meson/work/meson-0.56.2/b
>  b6aebd73f7/samelibname" 
> "-Wl,-rpath,/opt/local/var/macports/build/_Users_craigtreleaven_mp_macports-ports_devel_meson/meson/work/meson-0.56.2/b
>  b6aebd73f7/samelibname" 
> "-L/opt/local/var/macports/build/_Users_craigtreleaven_mp_macports-ports_devel_meson/meson/work/meson-0.56.2/b
>  b6aebd73f7/" 
> "-Wl,-rpath,/opt/local/var/macports/build/_Users_craigtreleaven_mp_macports-ports_devel_meson/meson/work/meson-0.56.2/b
>  b6aebd73f7/" -lsample -lgobject-2.0 -lglib-2.0 -lintl -lgirepository-1.0 
> -lgio-2.0 -lgobject-2.0 -lgmodule-2.0 -lglib-2.0 -lintl -Wl,-framework 
> -Wl,CoreFoundation -L/opt/local/lib -Wl,-headerpad_max_install_names
> 
> dyld: Symbol not found: _meson_sample_get_type
> 
>  Referenced from: 
> /opt/local/var/macports/build/_Users_craigtreleaven_mp_macports-ports_devel_meson/meson/work/meson-0.56.2/b
>  b6aebd73f7/tmp-introspectof76ehud/Meson-1.0
>  Expected in: 
> /opt/local/var/macports/build/_Users_craigtreleaven_mp_macports-ports_devel_meson/meson/work/meson-0.56.2/b
>  b6aebd73f7/samelibname/libsample.dylib
> in 
> /opt/local/var/macports/build/_Users_craigtreleaven_mp_macports-ports_devel_meson/meson/work/meson-0.56.2/b
>  b6aebd73f7/tmp-introspectof76ehud/Meson-1.0
> Command 
> '['/opt/local/var/macports/build/_Users_craigtreleaven_mp_macports-ports_devel_meson/meson/work/meson-0.56.2/b
>  b6aebd73f7/tmp-introspectof76ehud/Meson-1.0', 
> '--introspect-dump=/opt/local/var/macports/build/_Users_craigtreleaven_mp_macports-ports_devel_meson/meson/work/meson-0.56.2/b
>  
> b6aebd73f7/tmp-introspectof76ehud/functions.txt,/opt/local/var/macports/build/_Users_craigtreleaven_mp_macports-ports_devel_meson/meson/work/meson-0.56.2/b
>  b6aebd73f7/tmp-introspectof76ehud/dump.xml']' died with .
> ninja: build stopped: subcommand failed.
> 
> 
> I also note that there are a lot of messages saying “dirty”, “missing”, 
> “skipping”, and “warning”.  I would hope that if the test phase ran to 
> completion, it would produce a summary of anything important.
> 
> Craig
> 

Re: Meson 0.56.2 and Python39 --> dav1d failing

[Ken Cunningham]

dav1d is failing on 10.7 and 10.8 on the configure test for atomic.

it's building on 10.6 with clang 9.0, so probably needs to blacklist clangs 
older than 10.9's clang.

nothing to do with meson, looks to me...

K

Re: Meson 0.56.2 and Python39 --> dav1d failing

[Ken Cunningham]

sorry, another look, 10.9 fails so blacklist clangs older that 10.10's clang 
should do it.

Re: invisible universal variant and merger_must_run_binaries

[Ken Cunningham]

Well, as per the title, it will be every port that lists 
merger_must_run_binaries that will not build universal on BigSur Intel, however 
many that is.

And all the ports that cascade down as dependents of those ports...

K

Re: invisible universal variant and merger_must_run_binaries

[Ken Cunningham]

The good news is I currently only see 7 ports that set 
"merger_must_run_binaries" grepping the repo.

If we don't have to add too many more to the list as we explore more universal 
builds, then perhaps we can fix these to not need it, and solve our problem 
cleanly that way.

Several are core ports, though.

icu, readline, libgpg-error are probably the three biggest to fix, if possible.

Ken

Re: Meson 0.56.2 and Python39 --> dav1d failing

[Ken Cunningham]

That was a very nice fix...good to see how to do it properly now and then!

K

> On Feb 3, 2021, at 6:24 AM, Joshua Root  wrote:
> 
> On 2021-2-3 23:49 , Ken Cunningham wrote:
>> dav1d is failing on 10.7 and 10.8 on the configure test for atomic.
>> it's building on 10.6 with clang 9.0, so probably needs to blacklist clangs 
>> older than 10.9's clang.
>> nothing to do with meson, looks to me...
> 
> It's claimed to be written in C99, but requires either C11's stdatomic.h or 
> GCC extensions. Hmmm.
> 
> - Josh

mesa OpenGL header problem with mesa > 17 on SnowLeopard (and older) -- help wanted

[Ken Cunningham]

I’m stuck, and that doesn’t happen so often to me.

Something is going on that makes mesa versions > 17 not build on 10.6 (but does 
on 10.7+) — some kind of redefinition hiccup. 

It is proving quite vexing.

Almost ready to throw in the towel and peg mesa at 17.x for older systems 
unless someone has some inspiration on what the heck is going on …

thx ken

Re: Meson 0.56.2 and Python39 --> dav1d failing

[Ken Cunningham]

> On Feb 3, 2021, at 11:49 AM, Craig Treleaven  wrote:
> 
> But configure still failed on 10.7 and 10.8:

Oh no! It looked so great! I was really learning some things there. Perhaps it 
can be tweaked still.

If not, I guess we can still use the compiler_blacklist_versions approach and 
blacklist all the clangs < whatever 10.10 comes with.

Ken

Re: Meson 0.56.2 and Python39 --> dav1d failing

[Ken Cunningham]

well don't forget it builds just fine with the current meson on 10.6 using 
clang-9.0.

so...nothing to do with meson, I would say.

Really, let's just blacklist older clangs and move alonglife is short.

Ken

On Feb 4, 2021, at 08:48, Craig Treleaven  wrote:

>> On Feb 3, 2021, at 10:01 PM, Joshua Root  wrote:
>> 
>> On 2021-2-4 12:19 , Ken Cunningham wrote:
>>>> On Feb 3, 2021, at 11:49 AM, Craig Treleaven  
>>>> wrote:
>>>> 
>>>> But configure still failed on 10.7 and 10.8:
>>> Oh no! It looked so great! I was really learning some things there. Perhaps 
>>> it can be tweaked still.
>>> If not, I guess we can still use the compiler_blacklist_versions approach 
>>> and blacklist all the clangs < whatever 10.10 comes with.
>> 
>> Well now I'm just confused, because the test program builds fine on 10.7 
>> outside of meson. Need to see the meson-log.txt.
>> 
>> - Josh
> 
> meson-log.txt from a failed configure on a 10.7 VM is at:
> 
> https://paste.macports.org/52ea26dc0724
> 
> In this case, clang clang-600.0.57 on 10.7 seems to be choking on 
> "-Werror=ignored-optimization-argument”.  I don’t have 10.8 or 10.9 VM’s 
> running at the moment but would suspect that they have the same problem.  
> Please recall that dav1d used to build OK on these systems before meson was 
> updated to 0.56.2.
> 
> One of the devs on the dav1d project spotted that there was a recent commit 
> to meson that appears to be the cause of this dav1d build problem:
> 
> https://github.com/mesonbuild/meson/blob/cd94cf8995bcddc40e627e94037e549b7a18b20e/mesonbuild/compilers/mixins/clang.py#L87
> 
> This resulted from a commit on Oct. 4, 2020:
> 
> https://github.com/mesonbuild/meson/commit/cd59ce98dc88318c6784cfddfe3fadda2495041b
> 
> This stuff is all well above my pay grade but it looks to me that it would 
> likely break our builds on older OS X versions for other ports using meson.  
> Should we reverse this commit in our meson and file a bug upstream?
> 
> Craig

Re: Meson 0.56.2 and Python39 --> dav1d failing

[Ken Cunningham]

successful build with current meson on 10.6

https://build.macports.org/builders/ports-10.6_x86_64-builder/builds/47455/steps/install-port/logs/stdio

On Feb 4, 2021, at 08:48, Craig Treleaven  wrote:

>> On Feb 3, 2021, at 10:01 PM, Joshua Root  wrote:
>> 
>> On 2021-2-4 12:19 , Ken Cunningham wrote:
>>>> On Feb 3, 2021, at 11:49 AM, Craig Treleaven  
>>>> wrote:
>>>> 
>>>> But configure still failed on 10.7 and 10.8:
>>> Oh no! It looked so great! I was really learning some things there. Perhaps 
>>> it can be tweaked still.
>>> If not, I guess we can still use the compiler_blacklist_versions approach 
>>> and blacklist all the clangs < whatever 10.10 comes with.
>> 
>> Well now I'm just confused, because the test program builds fine on 10.7 
>> outside of meson. Need to see the meson-log.txt.
>> 
>> - Josh
> 
> meson-log.txt from a failed configure on a 10.7 VM is at:
> 
> https://paste.macports.org/52ea26dc0724
> 
> In this case, clang clang-600.0.57 on 10.7 seems to be choking on 
> "-Werror=ignored-optimization-argument”.  I don’t have 10.8 or 10.9 VM’s 
> running at the moment but would suspect that they have the same problem.  
> Please recall that dav1d used to build OK on these systems before meson was 
> updated to 0.56.2.
> 
> One of the devs on the dav1d project spotted that there was a recent commit 
> to meson that appears to be the cause of this dav1d build problem:
> 
> https://github.com/mesonbuild/meson/blob/cd94cf8995bcddc40e627e94037e549b7a18b20e/mesonbuild/compilers/mixins/clang.py#L87
> 
> This resulted from a commit on Oct. 4, 2020:
> 
> https://github.com/mesonbuild/meson/commit/cd59ce98dc88318c6784cfddfe3fadda2495041b
> 
> This stuff is all well above my pay grade but it looks to me that it would 
> likely break our builds on older OS X versions for other ports using meson.  
> Should we reverse this commit in our meson and file a bug upstream?
> 
> Craig

Re: Meson 0.56.2 and Python39 --> dav1d failing

[Ken Cunningham]

wait, I could have missed the issue...if meson is now adding a default argument 
to all checks that means it can never build anything with the default compilers 
on <10.10, that would be something we should consider fixing in meson, even if 
upstream doesn't want to.

So if testing shows that all builds on 10.7 -10.9 of meson ports are broken, 
then yeah, that's different, sure.

Ken

Re: mesa OpenGL header problem with mesa > 17 on SnowLeopard (and older) -- help wanted

[Ken Cunningham]

yes, it's a longstanding issue.

but somwhow it all held together until this last mesa update, and I just can't 
spot why it's broken now.

I have a PR ready to hold back mesa to 17.x ... on < 10.7.

I'll push that later today then...and after thst, wait for divine inspiration.

Ken

"cask" ports just keep on rolling in...

[Ken Cunningham]

With no plan, we’ll just keep getting more and more of these.

>

This kind of port just repackages the DMG into many tgz archives.

It’s wasteful. People want them.

What we should have instead of this is some kinda tech that

1. downloads the DMG
2. installs the app
3. records some way of uninstalling everything


What we have instead is a repackaging of the DMG into many, identical, 
system-specific archive bundles.

Yuk.

Ken

Re: "cask" ports just keep on rolling in...

[Ken Cunningham]

What would we really need to do this properly?

The current fetch, checksum phases are OK.

The extract phase can be used as is for some software (simple copies), but for 
other software we don’t want to extract it, we want to run the installer.

The configure and build phases need to be overridden and disappear.

The destroot phase needs to — what — record the files that will be installed in 
some kind of an index file? And also know about the stuff that gets installed 
into ~/Library, etc.

Then run the “cask” destroot without installing any files into the actual 
destroot:

copy the apps and stuff
or 
run the installer
or
extract from the pkg
or 
… 

and then NOT have the entire software repackaged into a MP archive file to be 
stored 12 different times…

Or some such plan

Best,

Ken





> On Feb 5, 2021, at 11:00 AM, Ken Cunningham  
> wrote:
> 
> With no plan, we’ll just keep getting more and more of these.
> 
> <https://github.com/macports/macports-ports/pull/9936 
> <https://github.com/macports/macports-ports/pull/9936>>
> 
> This kind of port just repackages the DMG into many tgz archives.
> 
> It’s wasteful. People want them.
> 
> What we should have instead of this is some kinda tech that
> 
> 1. downloads the DMG
> 2. installs the app
> 3. records some way of uninstalling everything
> 
> 
> What we have instead is a repackaging of the DMG into many, identical, 
> system-specific archive bundles.
> 
> Yuk.
> 
> Ken
> 

Re: "cask" ports just keep on rolling in...

[Ken Cunningham]

The idea of MacPorts is to manage installing inter-related libraries and 
executables.

But people want to use it to install prebuilt binaries as well, like homebrew 
does for cask installs.

MP has not decided if it will or will not accept these, but they just keep 
rolling in more and more anyway.

Installing them as "ports" is kinda silly, and wasteful, but simple, practical, 
and slides them in under the radar.

Having an actual plan for these would be better.

If software _can_ be built, we want to do it that way, so (IMHO) your work is 
not wasted. 

We get many benefits, I believe, from building things ourselves.

Whether we do or don't accept binary "cask" installs ins not up to me -- but 
I'm just point out that they are coming in anyway so a plan would be good, esp 
for the multi-megabyte behemoths that are out there.


Ken




On 2021-02-05, at 2:58 PM, Jason Liu wrote:

> The destroot phase needs to — what — record the files that will be installed 
> in some kind of an index file? And also know about the stuff that gets 
> installed into ~/Library, etc.
> 
> You would probably need to get the list of files installed by the installer 
> by running either pkgutil --files or lsbom, if we're talking about a PKG 
> installer. (Or maybe it would be easier to just grab the bom file in its 
> entirety.) Typically running an ls -R on the .app bundle inside a DMG 
> installer would be sufficient for those kinds of installers.
> 
> But then this begs the question: what happens to all of the work that went 
> into the build-from-source packages? Wouldn't this end up rendering the 
> hundreds of hours I spent getting the Blender package to work a complete 
> waste?
> 
> -- 
> Jason Liu
> 
> 
> On Fri, Feb 5, 2021 at 5:31 PM Ken Cunningham 
> mailto:ken.cunningham.web...@gmail.com>> 
> wrote:
> What would we really need to do this properly?
> 
> The current fetch, checksum phases are OK.
> 
> The extract phase can be used as is for some software (simple copies), but 
> for other software we don’t want to extract it, we want to run the installer.
> 
> The configure and build phases need to be overridden and disappear.
> 
> The destroot phase needs to — what — record the files that will be installed 
> in some kind of an index file? And also know about the stuff that gets 
> installed into ~/Library, etc.
> 
> Then run the “cask” destroot without installing any files into the actual 
> destroot:
> 
> copy the apps and stuff
> or 
> run the installer
> or
> extract from the pkg
> or 
> … 
> 
> and then NOT have the entire software repackaged into a MP archive file to be 
> stored 12 different times…
> 
> Or some such plan
> 
> Best,
> 
> Ken
> 
> 
> 
> 
> 
>> On Feb 5, 2021, at 11:00 AM, Ken Cunningham > <mailto:ken.cunningham.web...@gmail.com>> wrote:
>> 
>> With no plan, we’ll just keep getting more and more of these.
>> 
>> <https://github.com/macports/macports-ports/pull/9936 
>> <https://github.com/macports/macports-ports/pull/9936>>
>> 
>> This kind of port just repackages the DMG into many tgz archives.
>> 
>> It’s wasteful. People want them.
>> 
>> What we should have instead of this is some kinda tech that
>> 
>> 1. downloads the DMG
>> 2. installs the app
>> 3. records some way of uninstalling everything
>> 
>> 
>> What we have instead is a repackaging of the DMG into many, identical, 
>> system-specific archive bundles.
>> 
>> Yuk.
>> 
>> Ken
>> 
> 

Re: Meson 0.56.2 and Python39 --> dav1d failing

[Ken Cunningham]

> On Feb 5, 2021, at 10:04 PM, Ryan Schmidt  wrote:
> 
> That sounds like what Craig is saying: that meson is now adding a compiler 
> flag that old compilers don't understand. Meson should check whether a 
> compiler supports a flag before it adds it. We should not be responsible for 
> fixing this; the developers of meson should.

I haven’t tested it all to see for myself, but if someone knows for sure it so, 
then yeah.

Personally I would be happy to hotpatch it rather than have Every Single Meson 
Build on 10.7 through 10.9 fail until the patch makes it out the door to us in 
a released version.

K

Re: "cask" ports just keep on rolling in...

[Ken Cunningham]

> On Feb 5, 2021, at 10:00 PM, Ryan Schmidt  wrote:
> 
> 
> 
> On Feb 5, 2021, at 13:00, Ken Cunningham wrote:
> 
>> With no plan, we’ll just keep getting more and more of these.
> 
> I'm not aware of a huge influx of these, but I'm also not really paying 
> attention to the PR queue. And I'm not intending to get drawn into this 
> discussion of binary ports again.

The last discussion didn’t get anywhere past the appropriate naming scheme :> 
We never even got into implementation details.

>> This kind of port just repackages the DMG into many tgz archives.
>> 
>> It’s wasteful. People want them.
>> 
>> What we should have instead of this is some kinda tech that
>> 
>> 1. downloads the DMG
>> 2. installs the app
>> 3. records some way of uninstalling everything
> 
> MacPorts already does all these things... The submitted Portfile works fine, 
> presumably. There's no need to reinvent the existing MacPorts functionality 
> that does all these things.
> 

Well, IMHO there is, but I’ve looked at it quite a bit.

Look back a month or so and see my post about a “cask” port for SageMath for an 
idea of why this won’t work (well) in general.

For trivially simple ports, a few MB or less for a *.app that copies into 
/Applications/MacPorts — yeah, who cares?

> 
>> What we have instead is a repackaging of the DMG into many, identical, 
>> system-specific archive bundles.
>> 
>> Yuk.
> 
> If your objection is that we waste server space storing several copies of the 
> same thing, then we could invent a new way for a portfile to indicate that it 
> does not want binary archives stored on the server. It could be a new 
> separate keyword or a new pseudo license type, like we already have for 
> "nomirror" which tells the buildbot not to mirror the distfiles.
> A port like the one we're talking about in the above PR could set e.g. 
> "license GPL-2 nopackage", and buildbot could be modified to understand that 
> this means that it should not upload the packages.

Yes, that would help, if these binary ports start getting to any size.

> 
> MacPorts base would still try to download the nonexistent package. MacPorts 
> base currently does not use license values except to display to the user and 
> has no idea if a port is distributable. Distributability is handled by a 
> separate script. It should be integrated into base so that base can know if 
> something could have been distributed, and if it could not have, then it 
> shouldn't even try to download it. https://trac.macports.org/ticket/60315
> 

Yes. That leverages our existing tools in a nice way. Or a “cask” PortGroup 
could set “archive_sites” to “” I guess, perhaps more easily.

> We might also want to modify the MacPorts base command line "-b" binary only 
> mode to allow installing these "nopackage" ports rather than error out as it 
> would currently do.
> 
> If the few steps like disabling the configure and build phases and adding the 
> hypothetical "nopackage" to the license line are too much work for a portfile 
> submitter to do, a portgroup could be created that does those things. But you 
> have often complained about "magic" portgroups doing things you didn't know 
> about or didn't expect, so there is something to be said for ports doing what 
> they need to do explicitly, when it's not so many steps, and when it's not 
> always the same steps needing to be done. For example, surely each port will 
> still need to specify what the archs of the binary files are, what license 
> they're under, what type of distfile it is and where it is, and which files 
> inside needs to be copied where.
> 

It’s not that it’s too much work. It’s that these things are very simple, and 
people submit them but don’t know all these details you mention.

And —

this won’t work for larger ports too well (see SageMath message for an extreme 
example)
we don’t want to run rev-upgrade on them (surprises, surprises, surprises — eg 
SageMath again)
we want to be able to run a pkg-installer into some destination…. and nobody 
except three people know how to do that right
we want to be able to uninstall all the cruft the software installs in 
~/Library, ~/Preferences, etc easily

So — it’s doable. What we do now is — meh. 

IF we are going to do it, we should do it right.

And — we STILL have no naming scheme, so a user will have NO IDEA if he’s 
downloading an app from some website on the internet rather than something 
build by MacPorts.

And I think we should have a good way of identifying them, whatever it is. And 
yes, I still think identifying them by using a “+prebuilt” variant name is not 
the way to do it, if for no other r

Re: "cask" ports just keep on rolling in...

[Ken Cunningham]

try the sagemath one if you like. It's all set, for a newish system at present:

https://github.com/kencu/myports/blob/master/binary/sagemath-binary/Portfile

It is a stress test, for sure -- and I fully admit you may know simple tweaks 
to make it much more efficient.

Ken

Re: "cask" ports just keep on rolling in...

[Ken Cunningham]

those are the small things.

I stopped when the broad strokes, as it is now, were a unworkable disaster.

K

Re: "cask" ports just keep on rolling in...

[Ken Cunningham]

true, without ever trying it, you couldn't know

Re: "cask" ports just keep on rolling in...

[Ken Cunningham]

although I was concerned about getting this pattern right before we had too 
many of these to fix, it does seem the admins feel there's really no issue to 
worry about here.

So I guess we just open the gate and let them in. There is no recommendation 
for a requirement for a naming convention or other identifier.

I have gone ahead and committed the ones in the PR queue I found there. Feel 
free to use those as a template.

K

> On Feb 6, 2021, at 18:16, Mark Anderson  wrote:
> 
> Yeah, it seems like a lot of the stuff is in place, but we just need some 
> tweaks. I like the idea of a portgroup like BinaryOnly or something, but what 
> else needs to happen?
> 
> I'd be more than happy to help, what needs to be done? Should we maybe take 
> to Trac to get a to-do and proposal going?
>  
> —Mark
> ___
> Mark E. Anderson 
> MacPorts Trac WikiPage
> GitHub Profile
> 
> 
> 
>> On Sat, Feb 6, 2021 at 3:28 AM Ken Cunningham 
>>  wrote:
>> 
>> 
>> > On Feb 5, 2021, at 10:00 PM, Ryan Schmidt  wrote:
>> > 
>> > 
>> > 
>> > On Feb 5, 2021, at 13:00, Ken Cunningham wrote:
>> > 
>> >> With no plan, we’ll just keep getting more and more of these.
>> > 
>> > I'm not aware of a huge influx of these, but I'm also not really paying 
>> > attention to the PR queue. And I'm not intending to get drawn into this 
>> > discussion of binary ports again.
>> 
>> The last discussion didn’t get anywhere past the appropriate naming scheme 
>> :> We never even got into implementation details.
>> 
>> >> This kind of port just repackages the DMG into many tgz archives.
>> >> 
>> >> It’s wasteful. People want them.
>> >> 
>> >> What we should have instead of this is some kinda tech that
>> >> 
>> >> 1. downloads the DMG
>> >> 2. installs the app
>> >> 3. records some way of uninstalling everything
>> > 
>> > MacPorts already does all these things... The submitted Portfile works 
>> > fine, presumably. There's no need to reinvent the existing MacPorts 
>> > functionality that does all these things.
>> > 
>> 
>> Well, IMHO there is, but I’ve looked at it quite a bit.
>> 
>> Look back a month or so and see my post about a “cask” port for SageMath for 
>> an idea of why this won’t work (well) in general.
>> 
>> For trivially simple ports, a few MB or less for a *.app that copies into 
>> /Applications/MacPorts — yeah, who cares?
>> 
>> > 
>> >> What we have instead is a repackaging of the DMG into many, identical, 
>> >> system-specific archive bundles.
>> >> 
>> >> Yuk.
>> > 
>> > If your objection is that we waste server space storing several copies of 
>> > the same thing, then we could invent a new way for a portfile to indicate 
>> > that it does not want binary archives stored on the server. It could be a 
>> > new separate keyword or a new pseudo license type, like we already have 
>> > for "nomirror" which tells the buildbot not to mirror the distfiles.
>> > A port like the one we're talking about in the above PR could set e.g. 
>> > "license GPL-2 nopackage", and buildbot could be modified to understand 
>> > that this means that it should not upload the packages.
>> 
>> Yes, that would help, if these binary ports start getting to any size.
>> 
>> > 
>> > MacPorts base would still try to download the nonexistent package. 
>> > MacPorts base currently does not use license values except to display to 
>> > the user and has no idea if a port is distributable. Distributability is 
>> > handled by a separate script. It should be integrated into base so that 
>> > base can know if something could have been distributed, and if it could 
>> > not have, then it shouldn't even try to download it. 
>> > https://trac.macports.org/ticket/60315
>> > 
>> 
>> Yes. That leverages our existing tools in a nice way. Or a “cask” PortGroup 
>> could set “archive_sites” to “” I guess, perhaps more easily.
>> 
>> > We might also want to modify the MacPorts base command line "-b" binary 
>> > only mode to allow installing these "nopackage" ports rather than error 
>> > out as it would currently do.
>> > 
>> > If the few steps like disabling the configure and build phases and adding 
>> > the hypothetical "nopackage" to the lice

Re: "cask" ports just keep on rolling in...

[Ken Cunningham]

> On Feb 7, 2021, at 03:17, Clemens Lang  wrote:
> 
> Hi Ken,
> 
>> On Sat, Feb 06, 2021 at 11:35:58PM -0800, Ken Cunningham wrote:
>> although I was concerned about getting this pattern right before we
>> had too many of these to fix, it does seem the admins feel there's
>> really no issue to worry about here.
> 
> I don't like this tone, Ken. "The admins" have as much obligation to
> provide infrastructure as anybody else in this project, which is none.

It wasn't meant to be a tone. Just stating the exact facts. 

> 
> If you feel repackaging binary archives is a thing MacPorts should
> support better, please invest the time to come up with patches that do
> this, or find somebody that will.
> 

I did wonder about that, started some thought about how that might proceed, but 
the clear message was the current system is fine and there is nothing of 
significance that needs fixing.

> 
>> So I guess we just open the gate and let them in. There is no
>> recommendation for a requirement for a naming convention or other
>> identifier.
> 
> Personally, I don't like this trend at all. It always used to be
> MacPorts' policy to compile from source except in cases where Apple's
> limitations made this impossible (e.g. because valid signatures with a
> developer certificate were required and an ad-hoc signature would not
> work).
> 
> Now, we're apparently shipping binaries compiled by other people with
> other people's toolchains. When I previously installed things from
> MacPorts, I knew that I'd either compile those with my own toolchain
> locally, or that they had been compiled on MacPorts' buildbots.
> Repackaging binaries breaks that assumption and adds additional trusted
> third parties. If such parties were infiltrated by supply chain attacks
> such as Xcode Ghost, we'd now ship malware via 'port install'.

Now that the reality of what this really means comes to the fore, people are 
starting the be more vocal about their thoughts on it. This is good. These 
ports have been coming in, with no plan so far.


> 
> I do know that we have recently started making more and more exceptions
> to this rule, e.g. for Java and Haskell, and I'm guilty of preferring
> these approaches to a broken build of a very outdated version, but I'd
> like to argue that we should keep asking ourselves the question "should
> we really trust this person's toolchain" before merging such ports and
> keep pushing for builds from source where those are feasible.
> 
> Also, keep in mind that some licenses (GPL, LGPL) require us to ship the
> source code that was used to compile a certain binary. Our distfiles
> server does that automatically for everything that's compiled from
> source, but repackaging a binary that contains compiled GPL or LGPL code
> puts us in legal muddy water very quickly.
> 
> -- 
> Clemens



All true.

So new policy coming then?

As I stated at the very beginning months ago, with no plan or guidance, these 
just keep coming in. I am not championing this, but raising the flag here that 
there is a potential problem.

If it took a slightly more obvious message about what this really meant to get 
everyone to notice, I guess I accept that.

K

Re: "cask" ports just keep on rolling in...

[Ken Cunningham]

Just for the record here, I'm signing off the "cask" port issue, and will leave 
it you to sort out what you want to do, if anything.

I won't review or commit any cask port submissions, and I have no opinion about 
them other to say that I don't use them myself.

Best,

Ken

Re: Meson 0.56.2 and Python39 --> dav1d failing

[Ken Cunningham]

as meson was currently broken on older systems I pushed through a fix for this 
just now.

We can revise it to whatever upstream wants to do when they get around to it, 
if they want to do anything about it.

I took the opportunity to also add a cross file similar to the ones I added 
previously for i386 and x86_64, and this should allow cross building of meson 
ports on BigSur.

I deeply apologize — I stupidly made a tiny typo by leaving in a testing tweak, 
and only saw it when I looked at the commit afterwards to make sure it was all 
well. So I had to fix that quickly.

Normally I would have PR’d this, and waited for review but as in one case the 
systems were broken, and in the other it involved the cross-building I added, I 
went ahead with it.

My apologies if this is felt to break protocol; just trying to do the right 
thing here.

Bes;t

Ken


PS — any feedback on the success of the cross building, or any failures 
encountered, would be much appreciated.  I believe this is the only setup that 
currently allows meson cross building on BigSur that is available for macOS.

Re: Meson 0.56.2 and Python39 --> dav1d failing

[Ken Cunningham]

> On Feb 9, 2021, at 12:04 AM, Ken Cunningham  
> wrote:
> 
> 
> PS — any feedback on the success of the cross building, or any failures 
> encountered, would be much appreciated.  I believe this is the only setup 
> that currently allows meson cross building on BigSur that is available for 
> macOS.


Yep, success! meson cross-building to arm64 from BigSur Intel finally works now:

% port -v installed dav1d
The following ports are currently installed:
  dav1d @0.8.1_0+universal (active) platform='darwin 20' archs='arm64 x86_64' 
date='2021-02-09T01:15:26-0800'

% file /opt/local/lib/libdav1d.5.dylib
/opt/local/lib/libdav1d.5.dylib: Mach-O universal binary with 2 architectures: 
[x86_64:Mach-O 64-bit dynamically linked shared library x86_64] [arm64]
/opt/local/lib/libdav1d.5.dylib (for architecture x86_64):  Mach-O 64-bit 
dynamically linked shared library x86_64
/opt/local/lib/libdav1d.5.dylib (for architecture arm64):   Mach-O 64-bit 
dynamically linked shared library arm64


and presumably the other way around as well. So that's something new.


dav1d needs a couple of minor tweaks to get it to build universal, but they are 
minor.

K

glib2-devel --> interested parties please test https://github.com/macports/macports-ports/pull/5808

[Ken Cunningham]

Our glib2 is very outdated, and a number of software projects cannot be updated 
until it is modernized.

There is a 4000-PR-old PR to update it here 
https://github.com/macports/macports-ports/pull/5808 
 from Lord-Kamina who has 
not been around much lately.

The PR was outdated and needed rebasing, amongst other things.

The major holdup has been the move to meson, which until yesterday did not 
allow universal builds on BigSur, and other issues.

I’ve been working on glib2 today, and it builds -universal at present.

The universal builds, I think, are almost there — hopefully fixable with simple 
tweak to the portfile to accommodate multiple build folders needing a header 
copied in (right now the Portfile only expects one build folder). I was 
thinking to work on that tomorrow unless someone beats me to it before then.

BUT — glib2 is a core port. Many other ports rely on it. Bootstrapping from 
older systems relies on it, etc, etc.

I thought I would ask if any interested parties might kick the tires and see 
what we might need to do to get this beast committed.

Thanks,

Ken

Re: glib2-devel --> interested parties please test https://github.com/macports/macports-ports/pull/5808

[Ken Cunningham]

So I fixed the universal build, I think properly for now, and pushed the update 
to glib2-devel.

Anyone interested please try it out and see what needs fixin’.

Bes;t

Ken

> On Feb 10, 2021, at 11:57 PM, Ken Cunningham 
>  wrote:
> 
> Our glib2 is very outdated, and a number of software projects cannot be 
> updated until it is modernized.
> 
> There is a 4000-PR-old PR to update it here 
> https://github.com/macports/macports-ports/pull/5808 
> <https://github.com/macports/macports-ports/pull/5808> from Lord-Kamina who 
> has not been around much lately.
> 
> The PR was outdated and needed rebasing, amongst other things.
> 
> The major holdup has been the move to meson, which until yesterday did not 
> allow universal builds on BigSur, and other issues.
> 
> I’ve been working on glib2 today, and it builds -universal at present.
> 
> The universal builds, I think, are almost there — hopefully fixable with 
> simple tweak to the portfile to accommodate multiple build folders needing a 
> header copied in (right now the Portfile only expects one build folder). I 
> was thinking to work on that tomorrow unless someone beats me to it before 
> then.
> 
> BUT — glib2 is a core port. Many other ports rely on it. Bootstrapping from 
> older systems relies on it, etc, etc.
> 
> I thought I would ask if any interested parties might kick the tires and see 
> what we might need to do to get this beast committed.
> 
> Thanks,
> 
> Ken
> 
> 

fun fact about ICU +universal

[Ken Cunningham]

If you just strip all the muniversal shenanigans out of the Portfile, it
builds just fine universal on BigSur Intel without any fuss.

K

% uname -a

Darwin MacBookPro-2012.local 20.3.0 Darwin Kernel Version 20.3.0: Thu Jan
21 00:07:06 PST 2021; root:xnu-7195.81.3~1/RELEASE_X86_64 x86_64



% port -v installed icu

The following ports are currently installed:

  icu @67.1_3+universal (active) platform='darwin 20' archs='arm64 x86_64'
date='2021-02-14T13:33:16-0800'


% file /opt/local/lib/libicudata.dylib

/opt/local/lib/libicudata.dylib: Mach-O universal binary with 2
architectures: [x86_64:Mach-O 64-bit dynamically linked shared library
x86_64] [arm64]

/opt/local/lib/libicudata.dylib (for architecture x86_64): Mach-O 64-bit
dynamically linked shared library x86_64

/opt/local/lib/libicudata.dylib (for architecture arm64): Mach-O 64-bit
dynamically linked shared library arm64

Re: fun fact about ICU +universal

[Ken Cunningham]

The i386/x86_64 universal build looks fine too, so far, with all the
muniveral stuff out.

Now -- have to see why it was ever put in there in the first place, I guess.

K

On Sun, Feb 14, 2021 at 1:35 PM Ken Cunningham <
ken.cunningham.web...@gmail.com> wrote:

> If you just strip all the muniversal shenanigans out of the Portfile, it
> builds just fine universal on BigSur Intel without any fuss.
>
> K
>
> % uname -a
>
> Darwin MacBookPro-2012.local 20.3.0 Darwin Kernel Version 20.3.0: Thu Jan
> 21 00:07:06 PST 2021; root:xnu-7195.81.3~1/RELEASE_X86_64 x86_64
>
>
>
> % port -v installed icu
>
> The following ports are currently installed:
>
>   icu @67.1_3+universal (active) platform='darwin 20' archs='arm64 x86_64'
> date='2021-02-14T13:33:16-0800'
>
>
> % file /opt/local/lib/libicudata.dylib
>
> /opt/local/lib/libicudata.dylib: Mach-O universal binary with 2
> architectures: [x86_64:Mach-O 64-bit dynamically linked shared library
> x86_64] [arm64]
>
> /opt/local/lib/libicudata.dylib (for architecture x86_64): Mach-O 64-bit
> dynamically linked shared library x86_64
>
> /opt/local/lib/libicudata.dylib (for architecture arm64): Mach-O 64-bit
> dynamically linked shared library arm64
>
>

Re: fun fact about ICU +universal

[Ken Cunningham]

>  If i386/x86_64 also works, that suggests that bitness isn't an issue,
but endianness would be if ppc were included. Alignment is another potential
issue, but less likely to be in these cases.

Things have become a little simpler there -- as we have no compiler in
MacPorts at present that can cross-compile c++11 code between Intel and PPC
architectures, and as so many things, including ICU, need c++11 now, that
is not happening.

Ken

On Sun, Feb 14, 2021 at 4:48 PM Ken Cunningham <
ken.cunningham.web...@gmail.com> wrote:

> The i386/x86_64 universal build looks fine too, so far, with all the
> muniveral stuff out.
>
> Now -- have to see why it was ever put in there in the first place, I
> guess.
>
> K
>
> On Sun, Feb 14, 2021 at 1:35 PM Ken Cunningham <
> ken.cunningham.web...@gmail.com> wrote:
>
>> If you just strip all the muniversal shenanigans out of the Portfile, it
>> builds just fine universal on BigSur Intel without any fuss.
>>
>> K
>>
>> % uname -a
>>
>> Darwin MacBookPro-2012.local 20.3.0 Darwin Kernel Version 20.3.0: Thu Jan
>> 21 00:07:06 PST 2021; root:xnu-7195.81.3~1/RELEASE_X86_64 x86_64
>>
>>
>>
>> % port -v installed icu
>>
>> The following ports are currently installed:
>>
>>   icu @67.1_3+universal (active) platform='darwin 20' archs='arm64
>> x86_64' date='2021-02-14T13:33:16-0800'
>>
>>
>> % file /opt/local/lib/libicudata.dylib
>>
>> /opt/local/lib/libicudata.dylib: Mach-O universal binary with 2
>> architectures: [x86_64:Mach-O 64-bit dynamically linked shared library
>> x86_64] [arm64]
>>
>> /opt/local/lib/libicudata.dylib (for architecture x86_64): Mach-O 64-bit
>> dynamically linked shared library x86_64
>>
>> /opt/local/lib/libicudata.dylib (for architecture arm64): Mach-O 64-bit
>> dynamically linked shared library arm64
>>
>>

Re: fun fact about ICU +universal

[Ken Cunningham]

Yeah, the real downside of the muniversal  PG is that once it has been
chucked in to fix a build, for whatever reason it may have been once upon a
time, convincing everyone you DON'T need it after that is much harder than
if you never used it in the first place.

K

Re: fun fact about ICU +universal

[Ken Cunningham]

It's not complaining, it's irony.

I spent hours trying to fix a many-years-broken problem with ICU building
universal  only to find out that it actually builds universal trivially
easily.

I'll get around to fixing the small bits of that soonishly, once I have a
bit of spare time.

And yes, c++11 cross-arch-cross-compiling is broken with respect to PPC
(and with gcc arm/Intel) ... although I have been working on that too.

Thanks for the encouragement!

Ken

[no subject]

[Ken Cunningham]

> I’ve learned that irony doesn’t work very well via e-mail, because irony
usually requires knowing the author well when communicating via a text-only
medium, and it is often interpreted as complaining otherwise. I’m sorry to
say that I also interpreted some of your recent messages as complaining. I
think your work on MacPorts is greatly appreciated though.

Cheers, Nils.


Well -- I think you're still sore about me questioning you about adding 162
variants to a port.

https://github.com/macports/macports-ports/pull/9768

But -- thanks for taking the time to respond.

Ken

Apple notarisation process - LC_VERSION_MIN_MACOSX

[Ken Cunningham]

We really should have MacPorts libs be utilizable by projects that want to
put them on the App Store or use them any way they want to use them, if at
all possible.

The LC_VERSION_MIN_MACOSX issue is interesting.

There is lots of code in ld64 that appears designed to add this, and to
fall back on the DEPLOYMENT_TARGET to set it if it's not on the command
line, and to handle mismatches between the DEP target and the command line.

And yet I too can't see that chunk being added to dylibs either using xcode
ld64 or ld64-latest, whether building from source or downloaded from the
buildbot.

Something is amiss -- not sure it is a MacPorts issue, exactly, at this
moment. I await inspiration or insight...

K

Re: Octave port: website search results, 6.2.0, and Tablicious

[Ken Cunningham]

the update to Octave 6.2.0 is trivial, just a version change and checksums, 
other than my qt4 fixes for 6.1.0 — the patch for qt4 doesn’t apply now, so I 
need to take another look at that and see if it is still workable.

As to how to go about hacking on software that is being built by MacPorts, that 
is probably something that needs a proper writeup in the Wiki someday, as it 
takes a bit of insight into how MP works to know how to approach that.

K

the whole world is renaming the git default branch to main...

[Ken Cunningham]

I suggest we should do that too.

Ken

Re: the whole world is renaming the git default branch to main...

[Ken Cunningham]

You’re joking, right?

K

> On Mar 14, 2021, at 10:46 AM, Lothar Haeger  wrote:
> 
> Why?
> 
>> Am 14.03.2021 um 18:40 schrieb Ken Cunningham 
>> :
>> 
>> I suggest we should do that too.
>> 
>> Ken
> 

macports-libcxx: new libc++.dylib for older systems, installed in parallel

[Ken Cunningham]

New software uses new libc++ features that older systems don’t support. 

I put together a proof-of-concept technology preview here for a new 
libc++.dylib for older systems:

https://github.com/macports/macports-ports/pull/10238 


it allows std::filesystem, etc to be used on systems < 10.15.

I have a little demonstrator there too that requires these features, 
mkvtoolnix, updated as an example of how to use it.

No doubt there are warts / a better way forward, but I think this is a 
reasonable plan to start with, to see how this works out. 

Nothing comes free, however — discussion re: ODR violation possibilities, etc, 
is here:

https://trac.macports.org/ticket/62426 

Ken

gcc will now support -stdlib=libc++, etc

[Ken Cunningham]

Just an interesting FYI that gcc mainline now has support for the 
-stdlib=libc++ flag, and will build software against libc++ using the 
libc++ headers if desired.


https://github.com/gcc-mirror/gcc/commit/662b9c55cf06d3df6682ef865fb2b685820317a9

I'm not certain if this is rolling out in the soon-to-be-released gcc 
10.3, or will wait for gcc-11, but it will be a change in the prevailing 
logic to a degree, and opens up some interesting possibilities.


K

icu +universal on Darwin20+

[Ken Cunningham]

I believe I have come up with a robust fix for this issue.

However, before I update ICU to the newly-released version  69.1 from March 18 
and force everyone on MacPorts to install the new version with the 
“same-but-different” build Portfile, I would greatly appreciate anyone and 
everyone on this list to install the “new” existing version 67 from source, 
universal on as many systems as possible, and severely kick the tires for us to 
ensure that no warts remain (or have been introduced) that I have not noticed.

Best Regards,

Ken

Re: icu +universal on Darwin20+

[Ken Cunningham]

Is anyone aware of any port that uses icu’s pkgdata utility to generate a 
library version of data (other than ICU itself)?

thanks, 

Ken

Re: icu +universal on Darwin20+

[Ken Cunningham]

OK, everything is going smoothly, and I’m going to leave the icu port in this 
state for a while.

The only remaining issue I see is an old one — pkgdata has the archs stripped 
out of it’s inc file, so it will only generate one arch, even if icu is 
installed universal, when generating libraries of data.

If I leave the archs in the inc file, then it will generate the proper fat 
libraries, but (probably) anything that uses pkgdata and the muniversal PG will 
fail due to trying to lipo together two fat files.

I presume that either nothing uses pkgdata to generate libraries (likely), or 
if it does, it already uses the muniversal PG to do it.

That’s all then — update to icu coming out soon. That means dozens if not 
hundreds of ports to revbump against the new icu. 


Ken

Re: Publicizing MacPorts

[Ken Cunningham]

various people have come along with some good ideas over the five years I’ve 
been around.

A set of pages with some screenshots of some of the cool GUI software and 
games, like SNAP has for Ubuntu.

A rewrite of the very old and dated and cobwebby website.

To be honest —  these people in the end have not been encouraged to proceed. 
“I’ll get to that” or “Who is going to do that?” or “Why should we do that?” 
are far more common responses.

So — start there, I would volunteer.

Ken

Re: MacPorts 2.7.0-beta1 now available for testing

[Ken Cunningham]

Started testing on 10.4 Intel:

after upgrading /opt/local to the release-2.7 branch tip building from git 
source I am seeing this, which I have not previously noticed:

$ port
sqlite error: attempt to write a readonly database (1544) while executing 
query: ATTACH DATABASE '/opt/local/var/macports/registry/registry.db' AS 
registry
while executing
"registry::open $db_path"
(procedure "mportinit" line 712)
invoked from within
"mportinit ui_options global_options global_variations"
Error: /opt/local/bin/port: Failed to initialize MacPorts, sqlite error: 
attempt to write a readonly database (1544) while executing query: ATTACH 
DATABASE '/opt/local/var/macports/registry/registry.db' AS registry

$ port -v
sqlite error: attempt to write a readonly database (1544) while executing 
query: ATTACH DATABASE '/opt/local/var/macports/registry/registry.db' AS 
registry
while executing
"registry::open $db_path"
(procedure "mportinit" line 712)
invoked from within
"mportinit ui_options global_options global_variations"
Error: /opt/local/bin/port: Failed to initialize MacPorts, sqlite error: 
attempt to write a readonly database (1544) while executing query: ATTACH 
DATABASE '/opt/local/var/macports/registry/registry.db' AS registry

$ ls -la /opt/local/var/macports/registry/registry.db
-rw-r--r--   1 root  admin  212586496 Apr 24 10:51 
/opt/local/var/macports/registry/registry.db


using “sudo” for routine port commands, like “sudo port -v outdated” still 
works.



(how I build MacPorts:

I build base in /opt/bootstrap, then install curl in that /opt/boostrap 
installation.

then I build and install base in /opt/local like this:

./configure —with-curlprefix=/opt/bootstrap —with-sqlite3prefix=/opt/bootstrap

—Ken)

Re: MacPorts 2.7.0-beta1 now available for testing

[Ken Cunningham]

> On Apr 24, 2021, at 7:36 PM, Ryan Schmidt  wrote:
> 
> 
> 
> On Apr 24, 2021, at 13:19, Ken Cunningham wrote:
> 
>> Started testing on 10.4 Intel:
>> 
>> after upgrading /opt/local to the release-2.7 branch tip building from git 
>> source I am seeing this, which I have not previously noticed:
>> 
>> $ port
>> sqlite error: attempt to write a readonly database (1544) while executing 
>> query: ATTACH DATABASE '/opt/local/var/macports/registry/registry.db' AS 
>> registry
>>   while executing
>> "registry::open $db_path"
>>   (procedure "mportinit" line 712)
>>   invoked from within
>> "mportinit ui_options global_options global_variations"
>> Error: /opt/local/bin/port: Failed to initialize MacPorts, sqlite error: 
>> attempt to write a readonly database (1544) while executing query: ATTACH 
>> DATABASE '/opt/local/var/macports/registry/registry.db' AS registry
>> 
>> $ port -v
>> sqlite error: attempt to write a readonly database (1544) while executing 
>> query: ATTACH DATABASE '/opt/local/var/macports/registry/registry.db' AS 
>> registry
>>   while executing
>> "registry::open $db_path"
>>   (procedure "mportinit" line 712)
>>   invoked from within
>> "mportinit ui_options global_options global_variations"
>> Error: /opt/local/bin/port: Failed to initialize MacPorts, sqlite error: 
>> attempt to write a readonly database (1544) while executing query: ATTACH 
>> DATABASE '/opt/local/var/macports/registry/registry.db' AS registry
>> 
>> $ ls -la /opt/local/var/macports/registry/registry.db
>> -rw-r--r--   1 root  admin  212586496 Apr 24 10:51 
>> /opt/local/var/macports/registry/registry.db
>> 
>> 
>> using “sudo” for routine port commands, like “sudo port -v outdated” still 
>> works.
>> 
>> 
>> 
>> (how I build MacPorts:
>> 
>> I build base in /opt/bootstrap, then install curl in that /opt/boostrap 
>> installation.
>> 
>> then I build and install base in /opt/local like this:
>> 
>> ./configure —with-curlprefix=/opt/bootstrap 
>> —with-sqlite3prefix=/opt/bootstrap
> 
> Sounds like it might have been caused by the changes that fixed this bug, 
> which were made recently and I haven't had a chance to try them yet:
> 
> https://trac.macports.org/ticket/61154 
> <https://trac.macports.org/ticket/61154>
> 
> Does only the first non-sudo `port` command fail? Does running the command 
> with `sudo` once allow it to work without `sudo` a second time?
> 
> (There could be a one-time need to modify the registry sqlite database for 
> the new method.)


Yes, indeed that seems to be the case. I just tried it again on that same 
system, and indeed I no longer need to use sudo for basic commands. Good catch.

Ken

Re: MacPorts 2.7.0-beta1 now available for testing

[Ken Cunningham]

> On Apr 24, 2021, at 11:59 PM, Ryan Schmidt  wrote:
> 
> 
> 
> On Apr 24, 2021, at 23:23, Ken Cunningham wrote:
> 
>> On Apr 24, 2021, at 7:36 PM, Ryan Schmidt wrote:
>> 
>>> Does only the first non-sudo `port` command fail? Does running the command 
>>> with `sudo` once allow it to work without `sudo` a second time?
>>> 
>>> (There could be a one-time need to modify the registry sqlite database for 
>>> the new method.)
>> 
>> Yes, indeed that seems to be the case. I just tried it again on that same 
>> system, and indeed I no longer need to use sudo for basic commands. Good 
>> catch.
> 
> Ok, then this is probably just the way it is for now. I seem to recall a 
> similar situation happening with some previous MacPorts base update. Whenever 
> we change the structure of the registry in some way, we increase the registry 
> version and there is code (our update_db function) that knows what SQL 
> statements need to run to convert an old registry version into the new one. 
> This requires write access to the database.
> 
> Maybe there is a way that we could postpone the sqlite update process until 
> you run a sudo port command. But depending on what the modifications are that 
> the update performs, a new MacPorts might not be able to understand the 
> structure of an old MacPorts registry until the sqlite update is performed.
> 


Well, now I am a bit confused. I restarted the Tiger machine, and the problem 
is back:

$ uname -a
Darwin MacMini.local 8.11.1 Darwin Kernel Version 8.11.1: Wed Oct 10 18:23:28 
PDT 2007; root:xnu-792.25.20~1/RELEASE_I386 i386 i386

$ which port
/opt/local/bin/port

$ port -v
sqlite error: attempt to write a readonly database (1544) while executing 
query: ATTACH DATABASE '/opt/local/var/macports/registry/registry.db' AS 
registry
while executing
"registry::open $db_path"
(procedure "mportinit" line 712)
invoked from within
"mportinit ui_options global_options global_variations"
Error: /opt/local/bin/port: Failed to initialize MacPorts, sqlite error: 
attempt to write a readonly database (1544) while executing query: ATTACH 
DATABASE '/opt/local/var/macports/registry/registry.db' AS registry

$ sudo port -v
MacPorts 2.6.99
Entering shell mode... ("help" for help, "quit" to quit)
[Users/cunningh] > quit
Goodbye



My apologies for not immediately understanding why it worked last night and not 
today.

As Josh says, perhaps this is my own cross to bear due to how I build MacPorts. 
I will try it on another Tiger system I have, PPC.

Ken

Re: MacPorts 2.7.0-beta1 now available for testing

[Ken Cunningham]

I went through the exact same process on a different Tiger machine, using the 
/opt/bootstrap process with curl and sqlite3, and for whatever reason, this 
other system does not show the same “sudo” issue with “port”. 

So — it must be something sporadic on that machine I guess… otherwise there are 
no noted issues so I don’t know what, but it’s not consistent and reproducible, 
so … I’ll just see what happens there.

Ken







> On Apr 25, 2021, at 9:57 AM, Ken Cunningham  
> wrote:
> 
> 
> 
>> On Apr 24, 2021, at 11:59 PM, Ryan Schmidt > <mailto:ryandes...@macports.org>> wrote:
>> 
>> 
>> 
>> On Apr 24, 2021, at 23:23, Ken Cunningham wrote:
>> 
>>> On Apr 24, 2021, at 7:36 PM, Ryan Schmidt wrote:
>>> 
>>>> Does only the first non-sudo `port` command fail? Does running the command 
>>>> with `sudo` once allow it to work without `sudo` a second time?
>>>> 
>>>> (There could be a one-time need to modify the registry sqlite database for 
>>>> the new method.)
>>> 
>>> Yes, indeed that seems to be the case. I just tried it again on that same 
>>> system, and indeed I no longer need to use sudo for basic commands. Good 
>>> catch.
>> 
>> Ok, then this is probably just the way it is for now. I seem to recall a 
>> similar situation happening with some previous MacPorts base update. 
>> Whenever we change the structure of the registry in some way, we increase 
>> the registry version and there is code (our update_db function) that knows 
>> what SQL statements need to run to convert an old registry version into the 
>> new one. This requires write access to the database.
>> 
>> Maybe there is a way that we could postpone the sqlite update process until 
>> you run a sudo port command. But depending on what the modifications are 
>> that the update performs, a new MacPorts might not be able to understand the 
>> structure of an old MacPorts registry until the sqlite update is performed.
>> 
> 
> 
> Well, now I am a bit confused. I restarted the Tiger machine, and the problem 
> is back:
> 
> $ uname -a
> Darwin MacMini.local 8.11.1 Darwin Kernel Version 8.11.1: Wed Oct 10 18:23:28 
> PDT 2007; root:xnu-792.25.20~1/RELEASE_I386 i386 i386
> 
> $ which port
> /opt/local/bin/port
> 
> $ port -v
> sqlite error: attempt to write a readonly database (1544) while executing 
> query: ATTACH DATABASE '/opt/local/var/macports/registry/registry.db' AS 
> registry
> while executing
> "registry::open $db_path"
> (procedure "mportinit" line 712)
> invoked from within
> "mportinit ui_options global_options global_variations"
> Error: /opt/local/bin/port: Failed to initialize MacPorts, sqlite error: 
> attempt to write a readonly database (1544) while executing query: ATTACH 
> DATABASE '/opt/local/var/macports/registry/registry.db' AS registry
> 
> $ sudo port -v
> MacPorts 2.6.99
> Entering shell mode... ("help" for help, "quit" to quit)
> [Users/cunningh] > quit
> Goodbye
> 
> 
> 
> My apologies for not immediately understanding why it worked last night and 
> not today.
> 
> As Josh says, perhaps this is my own cross to bear due to how I build 
> MacPorts. I will try it on another Tiger system I have, PPC.
> 
> Ken
> 

Re: MacPorts 2.7.0-beta1 now available for testing

[Ken Cunningham]

WTH — five minutes later, and now the new install is also doing the same issue, 
whereas it was not a bit ago.

This is very strange.

K

> On Apr 25, 2021, at 3:31 PM, Ken Cunningham  
> wrote:
> 
> I went through the exact same process on a different Tiger machine, using the 
> /opt/bootstrap process with curl and sqlite3, and for whatever reason, this 
> other system does not show the same “sudo” issue with “port”. 
> 
> So — it must be something sporadic on that machine I guess… otherwise there 
> are no noted issues so I don’t know what, but it’s not consistent and 
> reproducible, so … I’ll just see what happens there.
> 
> Ken
> 
> 
> 
> 
> 
> 
> 
>> On Apr 25, 2021, at 9:57 AM, Ken Cunningham > <mailto:ken.cunningham.web...@gmail.com>> wrote:
>> 
>> 
>> 
>>> On Apr 24, 2021, at 11:59 PM, Ryan Schmidt >> <mailto:ryandes...@macports.org>> wrote:
>>> 
>>> 
>>> 
>>> On Apr 24, 2021, at 23:23, Ken Cunningham wrote:
>>> 
>>>> On Apr 24, 2021, at 7:36 PM, Ryan Schmidt wrote:
>>>> 
>>>>> Does only the first non-sudo `port` command fail? Does running the 
>>>>> command with `sudo` once allow it to work without `sudo` a second time?
>>>>> 
>>>>> (There could be a one-time need to modify the registry sqlite database 
>>>>> for the new method.)
>>>> 
>>>> Yes, indeed that seems to be the case. I just tried it again on that same 
>>>> system, and indeed I no longer need to use sudo for basic commands. Good 
>>>> catch.
>>> 
>>> Ok, then this is probably just the way it is for now. I seem to recall a 
>>> similar situation happening with some previous MacPorts base update. 
>>> Whenever we change the structure of the registry in some way, we increase 
>>> the registry version and there is code (our update_db function) that knows 
>>> what SQL statements need to run to convert an old registry version into the 
>>> new one. This requires write access to the database.
>>> 
>>> Maybe there is a way that we could postpone the sqlite update process until 
>>> you run a sudo port command. But depending on what the modifications are 
>>> that the update performs, a new MacPorts might not be able to understand 
>>> the structure of an old MacPorts registry until the sqlite update is 
>>> performed.
>>> 
>> 
>> 
>> Well, now I am a bit confused. I restarted the Tiger machine, and the 
>> problem is back:
>> 
>> $ uname -a
>> Darwin MacMini.local 8.11.1 Darwin Kernel Version 8.11.1: Wed Oct 10 
>> 18:23:28 PDT 2007; root:xnu-792.25.20~1/RELEASE_I386 i386 i386
>> 
>> $ which port
>> /opt/local/bin/port
>> 
>> $ port -v
>> sqlite error: attempt to write a readonly database (1544) while executing 
>> query: ATTACH DATABASE '/opt/local/var/macports/registry/registry.db' AS 
>> registry
>> while executing
>> "registry::open $db_path"
>> (procedure "mportinit" line 712)
>> invoked from within
>> "mportinit ui_options global_options global_variations"
>> Error: /opt/local/bin/port: Failed to initialize MacPorts, sqlite error: 
>> attempt to write a readonly database (1544) while executing query: ATTACH 
>> DATABASE '/opt/local/var/macports/registry/registry.db' AS registry
>> 
>> $ sudo port -v
>> MacPorts 2.6.99
>> Entering shell mode... ("help" for help, "quit" to quit)
>> [Users/cunningh] > quit
>> Goodbye
>> 
>> 
>> 
>> My apologies for not immediately understanding why it worked last night and 
>> not today.
>> 
>> As Josh says, perhaps this is my own cross to bear due to how I build 
>> MacPorts. I will try it on another Tiger system I have, PPC.
>> 
>> Ken
>> 
> 

Re: MacPorts 2.7.0-beta1 now available for testing

[Ken Cunningham]

Perhaps useful, I see this exact same database attach permissions error when I 
installed tip-of-tree macports-base on Ubuntu Linux last night.

Ken





> On Apr 25, 2021, at 3:38 PM, Ken Cunningham  
> wrote:
> 
> WTH — five minutes later, and now the new install is also doing the same 
> issue, whereas it was not a bit ago.
> 
> This is very strange.
> 
> K
> 
>> On Apr 25, 2021, at 3:31 PM, Ken Cunningham > <mailto:ken.cunningham.web...@gmail.com>> wrote:
>> 
>> I went through the exact same process on a different Tiger machine, using 
>> the /opt/bootstrap process with curl and sqlite3, and for whatever reason, 
>> this other system does not show the same “sudo” issue with “port”. 
>> 
>> So — it must be something sporadic on that machine I guess… otherwise there 
>> are no noted issues so I don’t know what, but it’s not consistent and 
>> reproducible, so … I’ll just see what happens there.
>> 
>> Ken
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>>> On Apr 25, 2021, at 9:57 AM, Ken Cunningham 
>>> mailto:ken.cunningham.web...@gmail.com>> 
>>> wrote:
>>> 
>>> 
>>> 
>>>> On Apr 24, 2021, at 11:59 PM, Ryan Schmidt >>> <mailto:ryandes...@macports.org>> wrote:
>>>> 
>>>> 
>>>> 
>>>> On Apr 24, 2021, at 23:23, Ken Cunningham wrote:
>>>> 
>>>>> On Apr 24, 2021, at 7:36 PM, Ryan Schmidt wrote:
>>>>> 
>>>>>> Does only the first non-sudo `port` command fail? Does running the 
>>>>>> command with `sudo` once allow it to work without `sudo` a second time?
>>>>>> 
>>>>>> (There could be a one-time need to modify the registry sqlite database 
>>>>>> for the new method.)
>>>>> 
>>>>> Yes, indeed that seems to be the case. I just tried it again on that same 
>>>>> system, and indeed I no longer need to use sudo for basic commands. Good 
>>>>> catch.
>>>> 
>>>> Ok, then this is probably just the way it is for now. I seem to recall a 
>>>> similar situation happening with some previous MacPorts base update. 
>>>> Whenever we change the structure of the registry in some way, we increase 
>>>> the registry version and there is code (our update_db function) that knows 
>>>> what SQL statements need to run to convert an old registry version into 
>>>> the new one. This requires write access to the database.
>>>> 
>>>> Maybe there is a way that we could postpone the sqlite update process 
>>>> until you run a sudo port command. But depending on what the modifications 
>>>> are that the update performs, a new MacPorts might not be able to 
>>>> understand the structure of an old MacPorts registry until the sqlite 
>>>> update is performed.
>>>> 
>>> 
>>> 
>>> Well, now I am a bit confused. I restarted the Tiger machine, and the 
>>> problem is back:
>>> 
>>> $ uname -a
>>> Darwin MacMini.local 8.11.1 Darwin Kernel Version 8.11.1: Wed Oct 10 
>>> 18:23:28 PDT 2007; root:xnu-792.25.20~1/RELEASE_I386 i386 i386
>>> 
>>> $ which port
>>> /opt/local/bin/port
>>> 
>>> $ port -v
>>> sqlite error: attempt to write a readonly database (1544) while executing 
>>> query: ATTACH DATABASE '/opt/local/var/macports/registry/registry.db' AS 
>>> registry
>>> while executing
>>> "registry::open $db_path"
>>> (procedure "mportinit" line 712)
>>> invoked from within
>>> "mportinit ui_options global_options global_variations"
>>> Error: /opt/local/bin/port: Failed to initialize MacPorts, sqlite error: 
>>> attempt to write a readonly database (1544) while executing query: ATTACH 
>>> DATABASE '/opt/local/var/macports/registry/registry.db' AS registry
>>> 
>>> $ sudo port -v
>>> MacPorts 2.6.99
>>> Entering shell mode... ("help" for help, "quit" to quit)
>>> [Users/cunningh] > quit
>>> Goodbye
>>> 
>>> 
>>> 
>>> My apologies for not immediately understanding why it worked last night and 
>>> not today.
>>> 
>>> As Josh says, perhaps this is my own cross to bear due to how I build 
>>> MacPorts. I will try it on another Tiger system I have, PPC.
>>> 
>>> Ken
>>> 
>> 
> 

Re: MacPorts 2.7.0-beta1 now available for testing

[Ken Cunningham]

An observation:

When I first install MP, it appears I can execute “port -v” and simple non-sudo 
port commands without using sudo.

However, after the first time I use “sudo port ..”, something changes, and I 
must forever use “sudo port … “ after that.

Then, if I reinstall base again right over top of the old installation, once 
again I don’t have to use “sudo” — until I use “sudo port .. “ once, and then 
it’s back to only being able to use “sudo port .. “ again.

Ken

Re: MacPorts 2.7.0-beta1 sqlite error: attempt to write a readonly database

[Ken Cunningham]

Here’s a quick reproducer on Mojave (or probably any system) for anyone 
interested:

/opt/local/bin/port install sqlite3

cd /tmp
git clone https://github.com/macports/macports-base.git 

cd macports-base
./configure --with-sqlite3prefix=/opt/local 
--with-applications-dir=/opt/tester/Applications --prefix=/opt/tester
make
sudo make install

export 
PATH=/opt/tester/bin:/opt/tester/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
/opt/tester/bin/port -v
sudo /opt/tester/bin/port -v
/opt/tester/bin/port -v

Re: MacPorts 2.7.0-beta1 sqlite error: attempt to write a readonly database

[Ken Cunningham]

Indeed, that appears to have fixed this issue, with 20 minutes testing so far.

repeated connections, with and without sudo, function normally. Ports install, 
deactivate, etc

Ken


On 2021-04-29, at 10:35 PM, Joshua Root wrote:

> Could you please try the tip of the release-2.7 branch, Ken?

Re: Help with legacy-support

[Ken Cunningham]

thread_local storage on MacOS is handled by dyld and uses and Apple-specific 
implementation that came into being in 10.7.

The thread_local infrastructure uses symbols that start with “_tlv_*”.

See here for details:

https://github.com/apple-opensource/dyld/blob/b6b86eb2db14440d373f6f7fd21be4a2bc0da897/src/threadLocalVariables.c
 



On < 10.6 I have implemented the “emulated_tls” infrastructure that systems 
without OS implemented TLS use. 

This is what gcc uses on every macOS system from 10.4 to BigSur.

The variables have different names, cxa_thread_*.

They normally live in libc on Linux, but I added them the libcxxabi using their 
mechanism for doing that on systems other than Apple.

To make that change, I added a small test and switch the implementation here:

https://github.com/macports/macports-ports/blob/19157729c2b2081d3434eca941440846ef8b796d/lang/llvm-9.0/files/9000-patch-clang-7.0-support-emulated-tls.diff#L23
 



I have pondered making some “work-alike” functions with the Apple names 
“tlv_init” etc, so the links would work, and then making perhaps the tlv_* 
functions call the cxa_thread_* functions, but I just never got around to it.

It might be do-able. Or we might do better to figure out how to make rust use 
the cxa_thread_* functions instead; probably, it already can. rust is backended 
by llvm, and we’ve already sorted out how to make llvm do this.

Stay in touch!

Ken

Re: Help with legacy-support

[Ken Cunningham]

should say:

On <= 10.6 I have implemented the “emulated_tls” infrastructure that systems 
without OS implemented TLS use. 

Re: Help with legacy-support

[Ken Cunningham]

I feel like I should have explained this better.

Apple’s thread local implementation is in here, and uses their tlv names for 
these variables.

https://github.com/apple-opensource/dyld/blob/b6b86eb2db14440d373f6f7fd21be4a2bc0da897/src/threadLocalVariables.c
 
<https://github.com/apple-opensource/dyld/blob/b6b86eb2db14440d373f6f7fd21be4a2bc0da897/src/threadLocalVariables.c>

gcc’s thread local implementation is here, and uses their names:

https://github.com/gcc-mirror/gcc/blob/master/libgcc/emutls.c 
<https://github.com/gcc-mirror/gcc/blob/master/libgcc/emutls.c>

llvm / clang can support both system tlv and emulated TlS. On Apple systems >= 
10.7, they use the dyld implementation. On Apple systems < 10.7, by default, 
they don’t support any. So all standard macOS software that uses thread_local 
uses the dyld implementation and the names they have there.

llvm/clang does have an emulated_tls implementation of their own here:

https://github.com/llvm-mirror/compiler-rt/blob/master/lib/builtins/emutls.c 
<https://github.com/llvm-mirror/compiler-rt/blob/master/lib/builtins/emutls.c>

They just don’t enable it by default on Apple. So I forced it to be available 
and turned that on by default on < 10.7.

For c++ threads, the exit function is named differently, so you have to call 
the right one, which is what this does:

https://github.com/macports/macports-ports/blob/19157729c2b2081d3434eca941440846ef8b796d/lang/llvm-9.0/files/9000-patch-clang-7.0-support-emulated-tls.diff#L23
 
<https://github.com/macports/macports-ports/blob/19157729c2b2081d3434eca941440846ef8b796d/lang/llvm-9.0/files/9000-patch-clang-7.0-support-emulated-tls.diff#L23>



If we wanted to bring the dyld thread_local symbols into < 10.7, we would have 
to implement the functions in the dyld threadLocalVariables.c file, either 
somehow making that compile and work, or calling into the emutls.c functions 
with the threadLocalVariables.c names, if that is possible.

The exit function "_tlv_atexit” would than presumably be made to call 
“__cxa_thread_atexit”.

I hope I did a better job this time.

Ken



> On May 1, 2021, at 10:18 PM, Ken Cunningham  
> wrote:
> 
> thread_local storage on MacOS is handled by dyld and uses and Apple-specific 
> implementation that came into being in 10.7.
> 
> The thread_local infrastructure uses symbols that start with “_tlv_*”.
> 
> See here for details:
> 
> https://github.com/apple-opensource/dyld/blob/b6b86eb2db14440d373f6f7fd21be4a2bc0da897/src/threadLocalVariables.c
>  
> <https://github.com/apple-opensource/dyld/blob/b6b86eb2db14440d373f6f7fd21be4a2bc0da897/src/threadLocalVariables.c>
> 
> 
> On < 10.6 I have implemented the “emulated_tls” infrastructure that systems 
> without OS implemented TLS use. 
> 
> This is what gcc uses on every macOS system from 10.4 to BigSur.
> 
> The variables have different names, cxa_thread_*.
> 
> They normally live in libc on Linux, but I added them the libcxxabi using 
> their mechanism for doing that on systems other than Apple.
> 
> To make that change, I added a small test and switch the implementation here:
> 
> https://github.com/macports/macports-ports/blob/19157729c2b2081d3434eca941440846ef8b796d/lang/llvm-9.0/files/9000-patch-clang-7.0-support-emulated-tls.diff#L23
>  
> <https://github.com/macports/macports-ports/blob/19157729c2b2081d3434eca941440846ef8b796d/lang/llvm-9.0/files/9000-patch-clang-7.0-support-emulated-tls.diff#L23>
> 
> 
> I have pondered making some “work-alike” functions with the Apple names 
> “tlv_init” etc, so the links would work, and then making perhaps the tlv_* 
> functions call the cxa_thread_* functions, but I just never got around to it.
> 
> It might be do-able. Or we might do better to figure out how to make rust use 
> the cxa_thread_* functions instead; probably, it already can. rust is 
> backended by llvm, and we’ve already sorted out how to make llvm do this.
> 
> Stay in touch!
> 
> Ken

python port notes suggest to "select" one -- but do we want that

[Ken Cunningham]

Sat am ponder:

I just updated python and saw the note displayed to select a python if you want 
it the default.

However we/I spend lots of time undoing this kind of thing, asking people to 
unselect everything, when their builds inevitably fail.

Perhaps this note might be considered not so helpful then, as we actually don’t 
want people (for the most part) to select anything.

And now back to your regularly scheduled programming…

Ken

llvm-9.0 on 10.6.8 shows red -- but it builds

[Ken Cunningham]

People are quoting 
>
  the lack of llvm-9.0 for SnowLeopard on macports as a sign of the demise of 
older systems, and it does for some reason show red here:

https://ports.macports.org/port/llvm-9.0/builds 


but it builds fine, along with most later llvms and all of the earlier ones, 
and it exists on the packages server in all archs:

http://packages.macports.org/llvm-9.0/llvm-9.0-9.0.1_1+emulated_tls.darwin_10.x86_64.tbz2
 


http://packages.macports.org/llvm-9.0/llvm-9.0-9.0.1_1+emulated_tls.darwin_10.i386.tbz2
 


http://packages.macports.org/llvm-9.0/llvm-9.0-9.0.1_1+emulated_tls+universal.darwin_10.i386-x86_64.tbz2
 


If anyone knows how to make the ports.macports.org  
website show it in green, that would be appreciated.

Thanks,

Ken

Re: Publicizing MacPorts [installation]

[Ken Cunningham]

> Yes, thanks for the tips! I am pretty sure that it is possible to automate it 
> one way or another. But my point is that it would be helpful to have a 
> one-liner to install MacPorts and maintain it as a part of the main 
> repository. 

This was of course suggested years ago as well, when homebrew first did it, but 
at that time was that it was both not needed and not a useful addition to 
MacPorts, if I recall the full email exchange correctly, so we let the idea die.

I wrote up a MacPorts install script for Jeremy’s Xquartz project here 
 
that adds a bit more trickery he needed, but the basic guts was extremely 
simple and what I recommend to people who complain that MacPorts is extremely 
difficult to get installed and they claim to have spent hours and hours and 
hours trying to make it work:

==

cd /tmp
git clone -b release-2.7 https://github.com/macports/macports-base.git
cd macports-base
./configure && make && sudo make install


and then add to the $PATH as usual

==

BigSur buildbot building software that will only run on 11.3+ ?

[Ken Cunningham]

In this ticket

https://trac.macports.org/ticket/63003 

it has been suggested by a knowledgable contributor that this issue:

xcrun: error: unable execute utility "/opt/local/libexec/llvm-11/bin/clang" 
because it requires a newer version of macOS.

Is being caused by the buildbot now being on 11.3, and that any binaries from 
that buildbot cannot run on any BigSur < 11.3.

If that is so, that is both a change from what has happened in the past for our 
buildbots, and I have to disable buildbot binaries on my BigSur machines (at 
least until OpenCore gets sorted out for my MacPro, to allow BigSur beyond 
11.2).

K

Re: perl and python version update

[Ken Cunningham]

> is there any overall strategy regarding the update of perl and python version 
> as dependencies?
The basic idea was to be rational about things, so that end-users don’t need 
many different perls and pythons installed just because, for example, someone 
noticed a new perl came out last Tuesday and so changed their ports to that.

The admins would set the “recommended” perl and python based on updates and 
software conformance, and all ports would try to use that (unless some given 
version would be the only version that would work).

And then, en-masse, at the right moment the “recommended” version would change, 
all the ports would more-or-less move to the new default at once, if we could.

How well this is working, whether it is working at all, and how well it is or 
is not generally supported by the group I could not say.

But it seemed like a good idea, when for example one needed to build and 
install two or three perls and two or three pythons just to install git.

Ken

Re: perl and python version update

[Ken Cunningham]

Seems like a fine idea to me. Thing is, you actually don't want to be that
current anyway.

The priority is on everything working, not newest/coolest -- so if the
designated perl or python is several years old, that is most likely
perfect. Nothing we need it for needs this week's versions, or this year's.

Now make new versions available if some soul wants to play; just don't make
it the designated version until enough time has gone by that everything
actually has been updated to work with it.

K

On Tuesday, June 1, 2021, Daniel J. Luke  wrote:

> On Jun 1, 2021, at 4:25 PM, Ken Cunningham  com> wrote:
> >> is there any overall strategy regarding the update of perl and python
> version as dependencies?
> >>
> > The basic idea was to be rational about things, so that end-users don’t
> need many different perls and pythons installed just because, for example,
> someone noticed a new perl came out last Tuesday and so changed their ports
> to that.
> >
> > The admins would set the “recommended” perl and python based on updates
> and software conformance, and all ports would try to use that (unless some
> given version would be the only version that would work).
> >
> > And then, en-masse, at the right moment the “recommended” version would
> change, all the ports would more-or-less move to the new default at once,
> if we could.
> >
> > How well this is working, whether it is working at all, and how well it
> is or is not generally supported by the group I could not say.
> >
> > But it seemed like a good idea, when for example one needed to build and
> install two or three perls and two or three pythons just to install git.
>
> For perl, we should just ship one perl as 'perl5' and have everything
> depend on it (and revbump the world of perl when we upgrade it). It takes
> us too long to migrate everything 'nicely'
>
> I suspect we could do this for python as well, but I've not looked
> recently at how disruptive newer python versions are.
>
> ... but I've said it before and people don't really like that idea, I
> guess :)
>
> --
> Daniel J. Luke
>
>

Re: perl and python version update

[Ken Cunningham]

On 2021-06-02, at 4:12 AM, Giuseppe 'ferdy' Miceli wrote:

> ciao,
> 
>> On 2 Jun 2021, at 09:29, Ken Cunningham  
>> wrote:
>> 
>> Seems like a fine idea to me. Thing is, you actually don't want to be that 
>> current anyway.
> 
> 1. as far i understood, for perl the recommended version should be perl5.30 
> which is stable (even tho’ not maintained), one year old (latest updated 
> 20200601):
> 
>   version latest update   status
>   5.302020-06-01  old version - not maintained
>   5.322021-01-23  old version - still maintained
>   5.342021-05-20  current stable - not yet in macports
> 

I would personally upgrade the recommended perl the day it stops getting 
security updates, to one version newer. Minimum fuss, maximum compatibility. 
Let the well-funded Linux distros fix all the tedious headaches updating 
modules to newer perl versions. But as you have heard, others feel a reason to 
be more cutting edge.

We find modules get updated to new perl or python versions without anyone 
testing the build or function, or running the test suite. Just because it shows 
up as an option is no indication that it actually works (which is the whole 
point of hanging back a bit).



> 2. what about python? as far as i understood should be 3.9 (also one year old 
> and with 3.10 still in beta, expected release oct 2021):
> 
>   version maintainancerelease end of support
>   3.9 bugfix  2020-10-05  2025-10
>   3.8 bugfix  2019-10-14  2024-10
>   3.7 security2018-06-27  2023-06-27
>   3.6 security2016-12-23  2021-12-23
>   2.7 end-of-life 2010-07-03  2020-01-01
> 

I believe macports' recommended python version is already 3.9 at present, we 
have that, and most ports default to that. Ones that don't should be updated to 
do so, if they actaully work with python39 (we have found a number of them 
don't).

> 
> does it make sense? also, what are we supposed to do with python2.7? 


We need python27 for various bootstrapping things, and for all the software 
(like llvm !) that still needs it to work properly.

So I think we'll have python27 in some form or other forever.

K







> —
> ferdy
> 
>> On Tuesday, June 1, 2021, Daniel J. Luke  wrote:
>> On Jun 1, 2021, at 4:25 PM, Ken Cunningham  
>> wrote:
>> >> is there any overall strategy regarding the update of perl and python 
>> >> version as dependencies?
>> >> 
>> > The basic idea was to be rational about things, so that end-users don’t 
>> > need many different perls and pythons installed just because, for example, 
>> > someone noticed a new perl came out last Tuesday and so changed their 
>> > ports to that.
>> > 
>> > The admins would set the “recommended” perl and python based on updates 
>> > and software conformance, and all ports would try to use that (unless some 
>> > given version would be the only version that would work).
>> > 
>> > And then, en-masse, at the right moment the “recommended” version would 
>> > change, all the ports would more-or-less move to the new default at once, 
>> > if we could.
>> > 
>> > How well this is working, whether it is working at all, and how well it is 
>> > or is not generally supported by the group I could not say.
>> > 
>> > But it seemed like a good idea, when for example one needed to build and 
>> > install two or three perls and two or three pythons just to install git.
>> 
>> For perl, we should just ship one perl as 'perl5' and have everything depend 
>> on it (and revbump the world of perl when we upgrade it). It takes us too 
>> long to migrate everything 'nicely'
>> 
>> I suspect we could do this for python as well, but I've not looked recently 
>> at how disruptive newer python versions are.
>> 
>> ... but I've said it before and people don't really like that idea, I guess 
>> :)
>> 
>> -- 
>> Daniel J. Luke
>> 
> 

Re: perl and python version update

[Ken Cunningham]

On 2021-06-02, at 8:56 AM, Marius Schamschula wrote:


> 
> Somehow FreeBSD can do w/o Python 2.7 (as of Python 2.7 EOL, FreeBSD 12.2 
> IIRC). They have the same llvm/clang infrastructure as macOS/MacPorts.
> 

for example:

https://trac.macports.org/ticket/62308


In addition to bootstrapping (for which we'll need python27 or 
python27-bootstrap forever), at present we support things like llvm/clang 5.0 
to 9.0 that need python27 to build.

And who knows how many other older ports we don't want to ditch yet.

K

Re: perl and python version update

[Ken Cunningham]

> For perl, we should just ship one perl as 'perl5' and have everything depend 
> on it (and revbump the world of perl when we upgrade it). It takes us too 
> long to migrate everything ‘nicely'

Is that not what the current “perl5” port does?

As I understand it, if you’re not fussy about which perl you want, you put in a 
dep on “port:perl5” then spec your perl as ${prefix}/bin/perl5 and you’re done.

the perl5 port symlinks to the currently blessed perl.

It’s quite possible I don’t fully grasp the nuances, though.

K

live check fetch failures

[Ken Cunningham]

Hey, maybe we will finally start bundling curl (there is a years-old ticket
about this, and someone already laid out the autotools changes to do it).

I have been building MacPorts against a custom libcurl on most systems
10.10 and older for years and years.

It's trivially simple to do, but to make it easy and resilient I keep a
"default" MacPorts installed in /opt/bootstrap and install curl in that,
then use that for the primary MP in /opt/local.

Re: live check fetch failures

[Ken Cunningham]

I don't subscribe to any of MacPorts' email lists, or any others.

So any comments come in new. Didn't realize that was troubling anyone, but
thank for letting me know it is.

K

On Saturday, June 19, 2021, Chris Jones  wrote:

> Hi Ken,
>
> OT, but whenever you reply to a thread on these lists your emailer always
> seems to screw up the headers such that your reply starts a new thread. We
> get the messages but it makes following the discussion harder. I don’t know
> what emailer you use, but maybe you could look into it ?
>
> Chris
>
> > On 19 Jun 2021, at 3:43 pm, Ken Cunningham  com> wrote:
> >
> > Hey, maybe we will finally start bundling curl (there is a years-old
> ticket about this, and someone already laid out the autotools changes to do
> it).
> >
> > I have been building MacPorts against a custom libcurl on most systems
> 10.10 and older for years and years.
> >
> > It's trivially simple to do, but to make it easy and resilient I keep a
> "default" MacPorts installed in /opt/bootstrap and install curl in that,
> then use that for the primary MP in /opt/local.
> >
> >
>
>

We will need an isolated gcc c++11 bootstrap compiler

[Ken Cunningham]

gcc11+ requires a c++11 compiler to build, and won't build with older
clang versions for other reasons besides c++11 capability.

So to bring the older systems up to parity with current and supported gcc
versions, and hopefully have them standardize on libgcc11 to make it all
easier on everyone, I'm going to need to create a stand-alone gcc compiler
that is self-contained with it's own libraries to use as a bootstrap.

I'm thinking this will either be gcc5 or gcc7 at present as they are both
capable and very easy to build on every system using system roots.

The bootstrap gcc5 or gcc7 will be tucked away somewhere off the beaten
path, and used only to build gcc11/libgcc11 on older systems, and other
similar things that might be needed for bootstrapping.

Ken

Re: We will need an isolated gcc c++11 bootstrap compiler

[Ken Cunningham]

Confirmed that Leopard i386, at least, can make the steps:

system compiler -> gcc-6 self-contained -> gcc11/libgcc11 without any
trouble.

Tiger i386 testing is underway and then PPC. There is reasonable hope that
in due course the entire platform list from Tiger to the current will
standardize on gcc11/libgcc11.

Along a similar path, we could also use the self-contained gcc6 on 10.4
through 10.6 to directly build a bootstrap version of clang-3.8 or 4.0 or
7.0 for example (linking against libstdc++),  skipping clang-3.4,
clang-3.7, and the partly-compromised libcxx without thread_local support
we build now, and then use that bootstrap clang to directly build libcxx
with thread_local support from the start, and go right to clang-9.0.

system roots -> gcc6 self-contained -> clang-bootstrap ->
libcxx +emulated_tls -> clang-9.0.

Apparently clang-4 when linked against libstdc++ is set to look in it's own
tree for libstdc++ first, so it could be self-contained if we copy in
libstdc++ from gcc6-bootstrap after it is built.

That potentially gets us out of a whole lot of "clang_dependency" hell.

Ken


On Tue, Jul 6, 2021 at 12:33 AM Ken Cunningham <
ken.cunningham.web...@gmail.com> wrote:

> gcc11+ requires a c++11 compiler to build, and won't build with older
> clang versions for other reasons besides c++11 capability.
>
> So to bring the older systems up to parity with current and supported gcc
> versions, and hopefully have them standardize on libgcc11 to make it all
> easier on everyone, I'm going to need to create a stand-alone gcc compiler
> that is self-contained with it's own libraries to use as a bootstrap.
>
> I'm thinking this will either be gcc5 or gcc7 at present as they are both
> capable and very easy to build on every system using system roots.
>
> The bootstrap gcc5 or gcc7 will be tucked away somewhere off the beaten
> path, and used only to build gcc11/libgcc11 on older systems, and other
> similar things that might be needed for bootstrapping.
>
> Ken
>

re: Meson & non-Xcode compilers -- esp older OSX versions

[Ken Cunningham]

nothing to do with meson or older compilers or @rpaths in general.

the otool in cctools 949 does not call llvm-objdump-11+ (including -devel)
with full args it likes. (llvm-11+ changed their supported args. )

That is why cctools defaults to llvm-10.

don't use cctools +llvmdev unless you're prepared to help out with devel
issues, I would (and did) tell the user. Just use the defaults and you'll
be fine.

I will update cctools at an opportune moment, and if they have fixed all
the llvm-11+ related issues, we'll default to that.

Best,

Ken

glib2 update to 2.62.5

[Ken Cunningham]

Our glib2 port is severely out of date, and has been preventing port
updates of some ports for a while now. The latest is gimp2, and so the
pressure is on for that now.

The glib2-devel port is considerably newer (although not current),  has
been out since February, and other than a couple of minor issues, seems OK.
I've been using it since February.

Evan and I are working on a few minor SDK related issues building on <
SnowLeopard, but other than that everything else seems to be working.

There was a question about +universal building one port, but that does not
appear to be (widely) reproducible, and whatever that might have been would
seem to be fixable if it shows up again.

glib2 is a build dep for many dozens of ports. If you would like to test
your favourite port against the new version please do so. Would be much
happier to find any breakage now, rather than later.

https://github.com/macports/macports-ports/pull/11805

Thanks,

Ken

Re: Haskell Stack Ports on Apple Silicon

[Ken Cunningham]

> I’d like to start hacking a Portfile update that will either compile pandoc 
> on the M1 with stack, or install the x86_64 binary and hope it runs, like the 
> x86_64 stack binary does.

You won't be surprised to find out I did that exact thing about a year ago:


https://github.com/macports/macports-ports/commit/39f5b1344934fd8777143d548c2fb04ef4affec6#diff-e9cf6cd1c3ba2b6e7eaa364835a16941f172cd2a16598b24cde0fce39ab55ee5

It was reverted / replaced though. I was told that MacPorts did this
automatically now (MacPorts would fall back and install the Intel binary on
arm64) so it was not needed to do this any longer.

Perhaps something has changed.

Ken

Re: Haskell Stack Ports on Apple Silicon

[Ken Cunningham]

I don't think it's overly simple to guess what might actually happen, or
work.

You have to try it various ways, and as Ryan says, I guess also try it when
pandoc is pulled in as a dependency rather than directly installed, and see.

The automatic fallback to other supported arches (eg arm64 -> x86_64) part
of macports base remains opaque to me  and I would suspect many. I guess I
should go find and read the relevant TCL to see what it is trying to do and
how.

K

On Fri, Aug 13, 2021 at 8:58 AM Steven Smith 
wrote:

> If the Macports-compiled stack runs on arm64, then the prebuilt download
> will too.
>
> The issue as far as I can tell from the internet is that stack will
> generate x86_64 binaries, even if running on an M1.
> https://www.haskell.org/ghc/blog/20200515-ghc-on-arm.html
>
> These x86_64 binaries should run on an M1.
>
> If so, is the best approach to remove the supported_archs line from the
> stack Portfile, or add arm64?
>
>
> On Aug 13, 2021, at 09:21, Ryan Schmidt  wrote:
>
> The stack port's +prebuilt variant installs a prebuilt binary of a
> particular architecture or architectures. In that variant, the port must
> declare using supported_archs what the architectures of that prebuilt
> binary are.
>
>
> On Aug 13, 2021, at 07:59, Christopher Jones 
> wrote:
>
>
>
> That line is indeed limiting support to intel machines. If it works on arm
> add that to the list, or probably better just remove it and rely on the
> defaults.
>
>
> Chris
>
>
> On 13 Aug 2021, at 1:55 pm, Steven Smith  wrote:
>
>
> Is this line in the stack Portfile the issue? Ports (like pandoc) that are
> built using stack depend on the stack port, and port stack says that x86_64
> is supported, but not arm64. However, stack installs and runs just fine on
> an M1 box.
>
>
> supported_archs x86_64
>
>
>
> https://github.com/macports/macports-ports/blob/4e94528cf34ba0ac86ee26d8f33b43351214/lang/stack/Portfile#L31
>
>
>
> On Aug 13, 2021, at 5:27 AM, Ryan Schmidt  wrote:
>
>
> As far as I could tell, this applies to individual ports, but not to a
> port's dependencies. See:
>
>
> https://trac.macports.org/ticket/63092
>
>
>
>
>

Re: Haskell Stack Ports on Apple Silicon

[Ken Cunningham]

You could override it, in pandoc or in the PortGroup:

depends_skip_archcheck-append stack

Re: Haskell Stack Ports on Apple Silicon

[Ken Cunningham]

> On Aug 14, 2021, at 7:13 PM, Joshua Root  wrote:
> 
> On 2021-8-15 11:52 , Steven Smith wrote:
>> stack (and ghc, cabal) can only build x86_64, so it’s an x86_64 binary that 
>> runs on the M1 (whether built on x86_64 or arm64).
>> The problem is that with the current supported_archs setting in stack, you 
>> hit this architecture mismatch error when trying to install pandoc on the M1:
>>> Cannot install pandoc for the arch 'arm64' because
>>> It’s dependency stack is only installed for the arch 'x86_64'
>>> and does not have a universal variant.
>>> Unable to execute port: architecture mismatch
> 
> Then the error is correct; you can't build an arm64 pandoc with an x86_64 
> stack. The pandoc port is just following the global build_arch setting since 
> it does not set supported_archs and the default is to assume that all archs 
> are supported. Since it can only actually build for x86_64, pandoc should set 
> supported_archs accordingly.
> 
> - Josh


IF pandoc set it’s supported_archs to x86_64 (which then matches stack) — would 
that now install properly on an M1 Mac, using the fallback archs?

that would be just right, if that’s how it works, and a proper fix for it all…

K

Re: Haskell Stack Ports on Apple Silicon

[Ken Cunningham]

> On Aug 15, 2021, at 5:16 AM, Steven Smith  wrote:
> 
>> Cannot install shellcheck for the arch 'arm64’ because


IF shellcheck had supported_archs x86_64 in the Portfile, it should never try 
to install the arm64 arch in the first place.

So I’m guessing probably that was missing.



It seems to work like this:

1. you try to install a haskell port on arm64
2. there is no supported_archs in the Portfile you are trying to install
3. base will try to build for the native arch (arm64)
3. base then examines all the deps.
4. If the deps don’t support arm64, you get an error.

Done.


So:

1. set supported_archs x86_64 in all the haskell ports
2. try to install a haskell port on arm64
3. base sees arm64 is not supported, but x86_64 is a supported_arch
4. base uses x86_64 as a fallback arch
5. base examines all the deps. All build as x86_64.
6. Good to go.

Re: Help offered (macOS VMs)

[Ken Cunningham]

I find Parallels is also very good for macOS VMs.

I have a 24-core MacPro 5,1 with 32GB RAM that I mostly use for systems I don’t 
have as hardware, and have VMs set up in Parallels for MacOS 10.5 through 
BigSur. I usually give a dozen cores and 16GB of RAM to the VM when I’m working 
with it, but it runs with much less. They can be run headless in the background 
as well.

For emulating Tiger i386 I use VirtualBox. I give it 3GB of RAM and one core. 
In the past, with VirtualBox 5.x, I could use two cores, but with VB 6.x there 
is some regression and now Tiger will only run with one core otherwise kernel 
panic. Still, quick for Tiger, with a current fast processor. I also have Tiger 
i386 on hardware.

I have qemu installations of 10.4 PPC and 10.5 PPC, but they are just awfully 
slow; they can only use 1 emulated processor. If they could multicore up to 4 
processors I might consider them usable, but that doesn’t work last time I 
tried it a year or so ago. For PPC work I just use one of my G5s; they are 
pretty quick and usably fast.


> On Sep 22, 2021, at 12:18 PM, Blake Garner  wrote:
> 
> Is there a reason macports would not use VMware Fusion for x86 versions? It 
> has the best macOS support and I'm sure that macports could get some licenses 
> as needed for free. For PPC I have had good luck with qemu in the past. 
> 
> On Tue, Sep 7, 2021 at 1:41 AM Ruben Di Battista  > wrote:
> I'm also very slowly working on this (the `slowly` is PhD related) and I was 
> targeting KVM. I'll probably give a try also to Hyper-V since I started using 
> Windows 11 with WSL for development. 
> 
> You can probably search in the list posts from me where we discussed this a 
> bit in details. 
> 
> On Tue, 7 Sep 2021, 10:09 Bjarne D Mathiesen,  > wrote:
> Mojca Miklavec wrote:
> > 5.) We're looking for someone willing to play around with macOS VMs,
> > in particular the older ones: 10.6 (or even 10.4/10.5 for utter geeks)
> > and newer. We currently run CI jobs just on what GitHub provides (2 or
> > 3 latest OSes), but it would be nice to set up disposable VMs where we
> > could fire up a VM, build the ports from CI, and destroy the VM.
> 
> What hypervisor it the target ?
> 
> I have tried to get a 10.6.8 to work in VirtualBox; but
> 1) it kept crashing
> 2) I couldn't get it to up the RAM above 2GB
>even though some mac systems ran 10.6 up to at least 8GB
> 
> -- 
> Bjarne D Mathiesen
> Korsør ; Danmark ; Europa
> ---
> denne besked er skrevet i et totalt M$-frit miljø
> MacPro 2010 ; OpenCore + macOS 10.15.7 Catalina
> 2 x 3,46 GHz 6-Core Intel Xeon ; 256 GB 1333 MHz DDR3 ECC RDIMM
> ATI Radeon RX 590 8 GB

Re: changing 'configure' options for testing

[Ken Cunningham]

On 2021-09-28 7:44 a.m., Daniel J. Luke wrote:

On Sep 20, 2021, at 10:20 AM, Daniel J. Luke  wrote:

On Sep 20, 2021, at 8:15 AM, Frank Dean  wrote:

Daniel J. Luke  writes:

The newest version of clamav uses cmake for builds. In the 'configure' stage, I 
have it disabling tests because otherwise it won't build without the test 
dependencies installed (check and pytest).

Do we have a template or example of a canonical way to handle this? I don't see 
an obvious hook for when someone is running `port test` to change 
configure.args (I could, of course, add a post-extract/pre-configure and do 
some non-declaritive test to see if `port test` is being run and use that to 
branch - but that feels like a bad design choice).

The rapidjson port implements a 'tests' variant to handle a similar
situation.  I used the same pattern for the libosmium port.  The tests
can then be run with `sudo port -d test current +tests`.

That works, I guess.

Is there interest in having base auto-add +tests if `port test` is called? (I 
haven't looked at base/ code in a while, but it seems possible). I like to 
imagine a future where we have enough infrastructure that we would run `port 
test` for any ports that have test.run set.

On this same train of thought - clamav tests run against the installed 
libraries (like some other ports tend to do) - in long-ago times I'd solve this 
by setting DYLD_LIBRARY_PATH - but since SIP started removing these that no 
longer works normally. I think trace mode has a(n elaborate) workaround where 
it copies binaries and executes the copies, but that doesn't seem like 
something I should implement in a portfile ;-) [This is another instance where 
if trace mode were the default, things would 'just work']

Has anyone else already solved this?



The way our cmake PortGroup sets things up, running tests is harder. You 
can configure cmake to allow testing to find the libraries in the build 
tree. See for example what I did in libcbor:



variant tests description {enable tests} {
    depends_test-append port:cmocka
    configure.args-append   -DWITH_TESTS=ON
    configure.pre_args-replace -DCMAKE_BUILD_WITH_INSTALL_RPATH:BOOL=ON \
-DCMAKE_BUILD_WITH_INSTALL_RPATH:BOOL=OFF
    test.run    yes
    test.target test
}

Ken

Re: changing 'configure' options for testing

[Ken Cunningham]

Ryan asked (when he had this same issue about six months ago) why our cmake
PortGroup sets it ON when it breaks all the in-tree tests, and whether we
should change the PG to turn it OFF all the time. (The default is OFF I
believe.)

In fact, we probably should. I'm not sure why we set it like it was.

Ken

On Tue, Sep 28, 2021 at 1:49 PM Daniel J. Luke  wrote:

> Thanks Ken and Jason!
>
> This is helpful (now I just need to figure out why a couple of tests fail
> when run this way vs. when run against the installed libraries).
>
> I was worried that the installed rpath would be different with
> CMAKE_BUILD_WITH_INSTALL_RPATH:BOOL=OFF, but it looks like it gets set to
> $prefix in destroot, so the installed binaries are fine.
>
> > On Sep 28, 2021, at 11:38 AM, Jason Liu  wrote:
> >
> > In addition to the code Ken just provided, he and others have helped me
> with some test-related "protection" code, so that in addition to a +tests
> variant, there is the following section of code (I usually place it after
> any build-related sections, and before any destroot-related sections, in
> order to keep my portfiles laid out in the same order as the port phases):
> >
> > pre-test {
> > if {![variant_isset tests]} {
> > ui_error "'tests' variant must be activated to enable test
> support"
> > error "Please enable the 'tests' variant and try again"
> > }
> > }
> >
> > --
> > Jason Liu
> >
> >
> > On Tue, Sep 28, 2021 at 11:13 AM Ken Cunningham <
> ken.cunningham.web...@gmail.com> wrote:
> >
> > On 2021-09-28 7:44 a.m., Daniel J. Luke wrote:
> > > On Sep 20, 2021, at 10:20 AM, Daniel J. Luke 
> wrote:
> > >> On Sep 20, 2021, at 8:15 AM, Frank Dean  wrote:
> > >>> Daniel J. Luke  writes:
> > >>>> The newest version of clamav uses cmake for builds. In the
> 'configure' stage, I have it disabling tests because otherwise it won't
> build without the test dependencies installed (check and pytest).
> > >>>>
> > >>>> Do we have a template or example of a canonical way to handle this?
> I don't see an obvious hook for when someone is running `port test` to
> change configure.args (I could, of course, add a post-extract/pre-configure
> and do some non-declaritive test to see if `port test` is being run and use
> that to branch - but that feels like a bad design choice).
> > >>> The rapidjson port implements a 'tests' variant to handle a similar
> > >>> situation.  I used the same pattern for the libosmium port.  The
> tests
> > >>> can then be run with `sudo port -d test current +tests`.
> > >> That works, I guess.
> > >>
> > >> Is there interest in having base auto-add +tests if `port test` is
> called? (I haven't looked at base/ code in a while, but it seems possible).
> I like to imagine a future where we have enough infrastructure that we
> would run `port test` for any ports that have test.run set.
> > > On this same train of thought - clamav tests run against the installed
> libraries (like some other ports tend to do) - in long-ago times I'd solve
> this by setting DYLD_LIBRARY_PATH - but since SIP started removing these
> that no longer works normally. I think trace mode has a(n elaborate)
> workaround where it copies binaries and executes the copies, but that
> doesn't seem like something I should implement in a portfile ;-) [This is
> another instance where if trace mode were the default, things would 'just
> work']
> > >
> > > Has anyone else already solved this?
> >
> >
> > The way our cmake PortGroup sets things up, running tests is harder. You
> > can configure cmake to allow testing to find the libraries in the build
> > tree. See for example what I did in libcbor:
> >
> >
> > variant tests description {enable tests} {
> >  depends_test-append port:cmocka
> >  configure.args-append   -DWITH_TESTS=ON
> >  configure.pre_args-replace -DCMAKE_BUILD_WITH_INSTALL_RPATH:BOOL=ON
> \
> > -DCMAKE_BUILD_WITH_INSTALL_RPATH:BOOL=OFF
> >  test.runyes
> >  test.target test
> > }
> >
> > Ken
>
> --
> Daniel J. Luke
>
>

upgrade to openssl 3.0.0

[Ken Cunningham]

openssl 3.0.0 is out, with a new and very favourable Apache-2 license that will 
let many previously non-distributable ports become distributable.  However, 
there are 758 ports that indicate they link against openssl. That is a daunting 
number of ports indeed.

One option might be to move all of our openssl ports (1.0, 1.1, and 3.x) into 
subdirectories out of ${prefix}, have no default openssl installed in $[prefix} 
directly, create a new PortGroup, openssl_select or some such, add that PG to 
all 758 ports, default it to openssl 1.1.1, and then allow people to gradually 
move the 758 ports to openssl 3.0.0 as they get to it.

Although that is possible with suitable effort, I don’t think that is workable 
for many reasons, the most obvious being that we would then have to modify 
every single port in some way to find an openssl installed into a non-default 
prefix. Creating the PG and adding it to 758 ports might be work enough, but 
then finding the right way to force all 758 ports to build properly against an 
openssl that is not in the default prefix is the real horror, and seems like a 
nightmare of wasted labours.

So it would appear that the same option that was chosen last time, ie upgrade 
the default openssl in ${prefix} to the newer openssl, and then fix the subset 
of ports that will not build against it to use an older openssl appears both 
the better option and lot less work (assuming most ports do build against 
openssl 3.0.0, which seems to be the case so far). Some will disagree, but I 
put it to you that it is going to be far less work in the end to force a few % 
of the ports to a specific alternate openssl than force all of them, all the 
time, forever.

Most things I have attempted to rebuild over the past few days have rebuilt 
without any issues, but a few things don’t build with openssl 3.0.0 yet and 
will need to stay with openssl 1.1.1 for a while until patched or updated (or 
forever). That will require both forcing those ports to find an alternate 
openssl installation, and also (the tricky part) forcing them to ignore the 
openssl in the default prefix.

To support this, there is a new openssl11 port that provides openssl 1.1.1 
tucked away in a subdir, much like we have openssl10, and a few new options 
were added to the old_openssl PortGroup to allow most ports to be forced to the 
alternate openssl with minimal fuss. Add the PortGroup, spec the branch, and 
choose the method, for the most part.

If this plan holds, I would anticipate that we move ports that we find need to 
stay on openssl 1.1.1 to openssl11 using the old_openssl PortGroup soon or now, 
before we upgrade to openssl 3.0.0 to minimize fuss. Then once we have done 
that, upgrading the default openssl to 3.0.0 and revbumping the deps would 
occur.

So I am asking that anyone with an interest in a port that uses openssl please 
try building it against openssl 3.0.0 in the PR below, and if there is a 
problem that can’t be solved by an update or a patch, consider trying to use 
the old_openssl PortGroup to fix the build and move it over. As there are so 
many ports, it would help if people pitched in with the ones that are important 
to them.

The openssl 3.0.0 upgrade PR is here:

https://github.com/macports/macports-ports/pull/12410 


and my own WIP repo of a few ports that I have found won’t build with openssl 
3.0.0 at present and fixed to use the old_openssl PG is here:

https://github.com/kencu/mybigsuroverlay/ 


the idea is that the ports in that repo would be moved into the main repo as we 
work through the main ones that use openssl at least.


K

PS. One thing I have not yet sorted out how to do is how to support libressl in 
the setting of needing to force a port to openssl 1.1.1 using the old_openssl 
PortGroup. Open to any thoughts on this. This case wsa basically just ignored 
last time with openssl10, and we may have to do the same again unless someone 
has a bright idea.

Re: upgrade to openssl 3.0.0

[Ken Cunningham]

All the pythons build against openssl 3.0.0, so that python issue with all it's 
trail-down conflicts will disappear with the upgrade and python revbump.

A very very large % of ports do as well (and those that don't now soon will, as 
everyone moves to openssl 3.0.0 as the default, which homebrew did pretty much 
the second openssl 3 came out).

So I am hoping that this upgrade is the beginning of the end of this particular 
PITA for all of us.

Ken


> On Oct 2, 2021, at 11:37 AM, Jason Liu  wrote:
> 
> That was also the Blender devs' claim, which I assume is why they decided it 
> wasn't necessary to include the GPL-OpenSSL exception text, since any 
> licensing conflicts would self-resolve once Blender starts using OpenSSL 3.0. 
> But currently, their pre-built release binary downloads and compiles OpenSSL 
> 1.1.1g, which is still under the OpenSSL license, not Apache 2.0.
> 
> -- 
> Jason Liu
> 
> 
> On Sat, Oct 2, 2021 at 2:25 PM Joshua Root  > wrote:
> Blender is GPL-2+, which means it can be distributed when linked with 
> OpenSSL 3.0, since GPL-3 is compatible with Apache-2.
> 
> - Josh
> 
> On 2021-10-3 05:09 , Jason Liu wrote:
> > I hope the question that I'm about to ask doesn't induce 
> > "Inception"-style migraines, but since it directly relates to one of the 
> > ports I maintain, here goes. What if one of my ports indirectly uses 
> > openssl through one of its dependencies, and the licensing terms in the 
> > current version of my port only covers openssl 1.1.1, but not 3.0?
> > 
> > To use a specific example, Blender uses openssl 1.1.1 indirectly, by way 
> > of using networking through python. I contacted the Blender devs about 
> > this, and even though they acknowledged that they were unknowingly using 
> > OpenSSL without including the OpenSSL license, the only thing they ended 
> > up doing was to add the OpenSSL license to their licenses directory. 
> > They refuse, even now, to add the chunk of text specifying the 
> > GPL-OpenSSL exception to their licenses. Somehow their legal team seems 
> > to think that's enough to allow them to distribute pre-compiled 
> > binaries, but the MacPorts automatic license checking is flagging my 
> > Blender port as being non-distributable. Since my port is a couple of 
> > versions behind the latest release of Blender (there are some new 
> > dependent libraries that I need to submit first), how should we specify 
> > in our Portfiles that one of the dependencies should continue using the 
> > old openssl11, without adding the old_openssl PortGroup, and thus a 
> > direct dependency on openssl? Does this mean that the dependencies which 
> > are directly dependent on openssl will need new variants, e.g. +openssl 
> > and +openssl11?
> > 
> > -- 
> > Jason Liu

Re: upgrade to openssl 3.0.0

[Ken Cunningham]

That is exactly the approach that I don't find workable, Chris.

Specifically:

1. every single port (think: rust, ghc, blah blah) will have to be needlessly 
managed to build against an alternate openssl location.
2. there will be no default openssl ever in prefix.

So I am not on board for trying to implement that myself, but should that be 
the selected course I will watch from the sidelines as others proceed down that 
path.

K

> On Oct 2, 2021, at 12:02 PM, Chris Jones  wrote:
> 
> Hi,
> 
> Personally I would strongly recommend following a migration path that does 
> not require an enmass change but allows ports to migrate  at their own pace. 
> I would personally very much recommend an approach similar to that we used 
> for boost recently
> 
> 1. Introduced a set of new isolated opensslXYZ ports that install specific 
> versions into isolated install areas under libexec.
> 2. Create a new portgroup that handles the confirmation of ports to pick the 
> openssl version they wish to use, with some default, and performs various 
> standard configurations for standard build systems such that for most ports 
> they work out the box with the new portgroup. For those that don’t provide 
> various proc methods that return the configured install paths such that ports 
> can used them to configure themselves as required.
> 3. Make the current openssl port a basic shim port that just installs sym 
> links into the primary install area to the default openssl version (for now 
> 1.1.1), so ports building against the current openssl port carry on working 
> just as they are now, but are redirected to use the same versioned 1.1.1 port 
> as those using the PG.
> 
> Once this is in place, ports can be migrated to use the new PG individually 
> as and when we want.
> 
> Chris
> 
>> On 2 Oct 2021, at 6:17 pm, Ken Cunningham  
>> wrote:
>> 
>> 
>> openssl 3.0.0 is out, with a new and very favourable Apache-2 license that 
>> will let many previously non-distributable ports become distributable.  
>> However, there are 758 ports that indicate they link against openssl. That 
>> is a daunting number of ports indeed.
>> 
>> One option might be to move all of our openssl ports (1.0, 1.1, and 3.x) 
>> into subdirectories out of ${prefix}, have no default openssl installed in 
>> $[prefix} directly, create a new PortGroup, openssl_select or some such, add 
>> that PG to all 758 ports, default it to openssl 1.1.1, and then allow people 
>> to gradually move the 758 ports to openssl 3.0.0 as they get to it.
>> 
>> Although that is possible with suitable effort, I don’t think that is 
>> workable for many reasons, the most obvious being that we would then have to 
>> modify every single port in some way to find an openssl installed into a 
>> non-default prefix. Creating the PG and adding it to 758 ports might be work 
>> enough, but then finding the right way to force all 758 ports to build 
>> properly against an openssl that is not in the default prefix is the real 
>> horror, and seems like a nightmare of wasted labours.
>> 
>> So it would appear that the same option that was chosen last time, ie 
>> upgrade the default openssl in ${prefix} to the newer openssl, and then fix 
>> the subset of ports that will not build against it to use an older openssl 
>> appears both the better option and lot less work (assuming most ports do 
>> build against openssl 3.0.0, which seems to be the case so far). Some will 
>> disagree, but I put it to you that it is going to be far less work in the 
>> end to force a few % of the ports to a specific alternate openssl than force 
>> all of them, all the time, forever.
>> 
>> Most things I have attempted to rebuild over the past few days have rebuilt 
>> without any issues, but a few things don’t build with openssl 3.0.0 yet and 
>> will need to stay with openssl 1.1.1 for a while until patched or updated 
>> (or forever). That will require both forcing those ports to find an 
>> alternate openssl installation, and also (the tricky part) forcing them to 
>> ignore the openssl in the default prefix.
>> 
>> To support this, there is a new openssl11 port that provides openssl 1.1.1 
>> tucked away in a subdir, much like we have openssl10, and a few new options 
>> were added to the old_openssl PortGroup to allow most ports to be forced to 
>> the alternate openssl with minimal fuss. Add the PortGroup, spec the branch, 
>> and choose the method, for the most part.
>> 
>> If this plan holds, I would anticipate that we move ports that we find need 
>> to stay on openssl 1.1.1 to openssl11 using the old_openssl PortGroup soon 
>

Re: upgrade to openssl 3.0.0

[Ken Cunningham]

as you wish, Chris.


I was hoping to move this along for the overwhelming benefit of the
license, but TBH the push-back so far is 99.99% negative about moving to
openssl 3.0.0 this year, so too controversial for me to get involved with.
I'll sit back for six to twelve months and see what you guys work out over
the coming year.

K

On Mon, Oct 4, 2021 at 8:18 AM Chris Jones  wrote:

>
> Reading this thread again, I think there is some confusion over what I
> am suggesting, compared to Renee/Ken, as really the differences are
> minor and cosmetic packaging differences.
>
> The main difference is I am suggesting aiming for consistency across
> where all the openssl ports install to. Instead of having some install
> to libexec, and then one 'blessed' default port install directly into
> ${prefix} have opensslXYZ ports for *all* versions we need, that all
> install to their own libexec area. Then, have a trivial shim port for
> the general 'openssl' port that just installs a few sym links from
> whatever the 'default' openssl version is deemed to be (currently 1.1.1)
> into the primary ${prefix} install.
>
> We could do this, right now, i.e. move the current openssl port to an
> isolated libexec openssl11 port, and update openssl to just be a shim to
> that install. This would be completely transparent to all ports using
> openssl.
>
> The advantage of this to me would be
>
> 1. Consistency across all the openssl ports, as they would all,
> regardless of their version, be treated (installed) the same way.
>
> 2. Once there, we could then introduce a new port group that extends
> what the current old_openssl PG does but in a more general way, as it
> would work for not just 'old' openssl versions, but also the current
> default and also new ones. e.g. we could then add a openssl30 port and
> allow a few ports to experiment with it.
>
> 3. Once we are OK with a few ports using openssl30 we could 'throw the
> switch' and update the shim port to point to the 3.0 version.
>
> The above, if you think about it, is really no different to what
> Ken/Renee are suggesting in terms of what ports have to do. If ports are
> OK with just relying on the default they can carry on using the
> 'openssl' (shim) port, just as they are now. Nothing is forcing us to
> change these ports. *If* ports wanted to migrate to use the new PG that
> is fine, but its an optional choice (unless they are not compatible with
> 3.0 of course, but then they will need updating anyway once 3.0 is the
> default).
>
> So why do the above instead of what Ken/Renee suggest ? for me the
> primary reason is we end up with a system where in future it will be
> much easier to deal with future new major versions, without having to
> have these discussions again... Say 3.1 or 4.0 comes out, a new isolated
> opensslXY port can be added to provide that, that will live alongside
> the current ones. A few ports could be tested against it, and as and
> when we are ready the default can be flipped again to point to it.
>
> Chris
>
>
> On 04/10/2021 10:24 am, Christopher Jones wrote:
> >
> >
> >> On 2 Oct 2021, at 8:06 pm, Ken Cunningham
> >>  >> <mailto:ken.cunningham.web...@gmail.com>> wrote:
> >>
> >> That is exactly the approach that I don't find workable, Chris.
> >>
> >> Specifically:
> >>
> >> 1. every single port (think: rust, ghc, blah blah) will have to be
> >> needlessly managed to build against an alternate openssl location.
> >
> > why ? They can start of using the shim port and only use the isolated
> > builds as and when they are ready
> >
> >> 2. there will be no default openssl ever in prefix.
> >
> > why not ? The boost set has one and I see no reason why openssl cannot
> > as well.
> >
> > Chris
> >
> >>
> >> So I am not on board for trying to implement that myself, but should
> >> that be the selected course I will watch from the sidelines as others
> >> proceed down that path.
> >>
> >> K
> >>
> >>> On Oct 2, 2021, at 12:02 PM, Chris Jones  >>> <mailto:jon...@hep.phy.cam.ac.uk>> wrote:
> >>>
> >>> Hi,
> >>>
> >>> Personally I would strongly recommend following a migration path that
> >>> does not require an enmass change but allows ports to migrate  at
> >>> their own pace. I would personally very much recommend an approach
> >>> similar to that we used for boost recently
> >>>
> >>> 1. Introduced a set of new isolated opensslXYZ ports that inst

Re: upgrade to openssl 3.0.0

[Ken Cunningham]

For whoever gets up the enthusiasm to take on the storm of nay-sayers:

Although I found about 90% of the 100 or so ports I tried built without any
changes against openssl 3.0.0 (rust, cargo, qt5, qt4-mac, etc, etc), and
the rest were easy < 5 min fixes to use our openssl11 port, I noted in the
openssl 3 migration guide that the FIPS mode is disabled by default on the
openssl 3 build, and has to be expressly enabled.

I recall that most of the (very few) build failures I saw were in fact FIPS
failures, so enabling that module might fix a bunch of them.

Best,

Ken


On Tue, Oct 5, 2021 at 12:54 PM Fred Wright  wrote:

>
> On Mon, 4 Oct 2021, Christopher Jones wrote:
> >> On 4 Oct 2021, at 5:54 pm, Ken Cunningham <
> ken.cunningham.web...@gmail.com> wrote:
> >>
> >> I was hoping to move this along for the overwhelming benefit of the
> >> license, but TBH the push-back so far is 99.99% negative about moving
> >> to openssl 3.0.0 this year, so too controversial for me to get involved
> >> with. I'll sit back for six to twelve months and see what you guys work
> >> out over the coming year.
> >
> > All the more reason to follow my suggested migration path then I would
> > say, as it allows an openssl30 port to be made available, and those
> > ports that wish to can use it via the new PG, but it doesn’t have to
> > become the default until some later date.
>
> The PR thread contained (approximately) the following two statements:
>
> 1) Unless v3 is the default, nobody will bother to use it.
>
> 2) Everybody is really, *really* anxious to move to v3 for the more
> permissive license.
>
> Clearly those two statements are in conflict.
>
> At Google, we had a process called "canarying".  Although technically a
> misnomer, it referred to the "canary in the coal mine" concept, with the
> idea that rolling out new stuff with possible issues should start small,
> so that problems could be found (and hopefully fixed) before they caused
> large-scale breakage.
>
> If the OpenSSL folks were committed to maintaining backward compatibility,
> then none of this nonsense would be necessary, but it's clear that they're
> not.  And there's no reason to assume that they won't pull the same crap
> again in the future (having done so at least twice already), so having a
> mechanism for multiple coexisting OpenSSL "major" versions could have
> long-term value beyond the v3 transition.
>
> > TBH I also was quite dubious of making 3.0.0 the default any time ’soon’
>
> I agree, especially if the only end benefit is the license.  Remember,
> OpenSSL is the poster child for why *not* to assume that that newer is
> more secure. :-)
>
> Fred Wright

Re: upgrade to openssl 3.0.0

[Ken Cunningham]

Well thanks, Rene!  

I’m so glad to see this is actually happening now, after a momentary delay.

I think my comment about enabling the openssl3 FIPS mode was somehow missed; it 
has to be specifically turned on in openssl3, but it does allow more things to 
work with openssl3 I believe.

Ken




> On Nov 6, 2021, at 6:34 AM, Renee Otten  wrote:
> 
> Dear all, 
> 
> 
> Chris has done the work to add the openssl3 port and openssl-1.0 PortGroup to 
> ease the transition towards openssl v3. There is now an open PR 
> (https://github.com/macports/macports-ports/pull/12807 
> <https://github.com/macports/macports-ports/pull/12807>) to switch en masse  
> the default openssl version to 3. The Apache-2 license of openssl3 will allow 
> many ports that are currently not, to become distributable as there will be 
> no license conflict anymore.
> 
> If maintainers want to pro-actively test whether “their” ports build properly 
> with the latest openssl version please do so now. If that doesn’t work 
> out-of-the-box then the status quo (i.e., building with openssl v1.1) can 
> easily be restored by the adding the following code to the Portfile:
> 
> PortGroup openssl 1.0
> openssl.branch1.1
> 
> and depending on the build-system one can also specify “openssl.configure” 
> which takes the options: 'pkgconfig' (default), 'build_flags', or 'cmake’. 
> 
> For other options, please refer to the PortGroup code: 
> https://github.com/macports/macports-ports/blob/master/_resources/port1.0/group/openssl-1.0.tcl
>  
> <https://github.com/macports/macports-ports/blob/master/_resources/port1.0/group/openssl-1.0.tcl>.
>  
> 
> Given the number of maintainers and ports involved we should probably leave 
> the above mentioned PR open for a bit more than the usual 72 hours, but I’d 
> imagine that in about a week or so from now someone will re-run the script to 
> revbump affected ports and push the merge button. Ideally most of the ports 
> will just work with openssl3, whereas others might have been set to build 
> with openssl1 by their maintainers by then. Yes, there will for sure be some 
> breakage of unmaintained ports that will need fixing, but that will need to 
> happen at some point anyway so we can as well just flip the switch soon and 
> deal with it.
> 
> 
> 
> 
>> On Oct 7, 2021, at 12:16 PM, Christopher Jones > <mailto:jon...@hep.phy.cam.ac.uk>> wrote:
>> 
>> 
>> https://github.com/macports/macports-ports/pull/12514 
>> <https://github.com/macports/macports-ports/pull/12514>
>> 
>> 
>>> On 6 Oct 2021, at 5:46 pm, Christopher Jones >> <mailto:jon...@hep.phy.cam.ac.uk>> wrote:
>>> 
>>> I’m working on the basic changes to implement my suggestion at the moment. 
>>> Once that is there testing specific ports against version 3 ’the canaries’ 
>>> will be trivial. more in a bit.
>>> 
>>>> On 6 Oct 2021, at 5:40 pm, Ken Cunningham >>> <mailto:ken.cunningham.web...@gmail.com>> wrote:
>>>> 
>>>> For whoever gets up the enthusiasm to take on the storm of nay-sayers:
>>>> 
>>>> Although I found about 90% of the 100 or so ports I tried built without 
>>>> any changes against openssl 3.0.0 (rust, cargo, qt5, qt4-mac, etc, etc), 
>>>> and the rest were easy < 5 min fixes to use our openssl11 port, I noted in 
>>>> the openssl 3 migration guide that the FIPS mode is disabled by default on 
>>>> the openssl 3 build, and has to be expressly enabled.
>>>> 
>>>> I recall that most of the (very few) build failures I saw were in fact 
>>>> FIPS failures, so enabling that module might fix a bunch of them.
>>>> 
>>>> Best,
>>>> 
>>>> Ken
>>>> 
>>>> 
>>>> On Tue, Oct 5, 2021 at 12:54 PM Fred Wright >>> <mailto:f...@fwright.net>> wrote:
>>>> 
>>>> On Mon, 4 Oct 2021, Christopher Jones wrote:
>>>> >> On 4 Oct 2021, at 5:54 pm, Ken Cunningham 
>>>> >> >>> >> <mailto:ken.cunningham.web...@gmail.com>> wrote:
>>>> >>
>>>> >> I was hoping to move this along for the overwhelming benefit of the 
>>>> >> license, but TBH the push-back so far is 99.99% negative about moving 
>>>> >> to openssl 3.0.0 this year, so too controversial for me to get involved 
>>>> >> with. I'll sit back for six to twelve months and see what you guys work 
>>>> >> out over the coming year.
>>>> >
>>&

Re: port:libcxx - why so old

[Ken Cunningham]

> Hi,
> 
> Judging from the version number, port:libcxx ships a version that's long 
> outdated. Is there a reason for that, like it's the latest version that 
> builds on all OS X versions that require the libc++ conversion?
> 
> I have a local version that is currently at v8.0.0 on my 10.9.5 system and 
> have had that install the runtime shared libraries (= not the ones used by 
> the linker) in $prefix/lib via an additional +runtime variant which also 
> installs a launch wrapper that adds $prefix/lib to DYLD_INSERT_LIBRARIES 
> (DYLD_LIBRARY_PATH typically isn't enough on Mac). It's a lot newer than my 
> system versions even (than v5.0.1) and yet I can't recall having had any 
> issues with it. On the contrary, it fixed a number of strange C++ runtime 
> warnings in certain KDE software and must give me that bit more performance 
> because built with a more recent clang, full CPU-specific optimisation and 
> LTO.
> 
> R.
> 

libcxx 5.0 is quite new in fact — it supports libcxx features that only show up 
in 10.13+ (which give us the strange fact of ports that build on 10.6, and 
10.13+, but nothing in between. :>

I had libcxx 7.0 ready to push, which I was going to do if I ever sorted out 
how to bootstrap it to the full Thread_Local_Storage supporting version without 
tricks.

But I never noticed a single port that built with libcxx 7.0 that would not 
build with libcxx 5.0, so there was no big rush.

I did replace the libc++.dylib on my 10.7.5 system with the one from llvm-5.0 
and promptly found certain applications (ical, for example) crashing regularly.

So I gave up on the plan of a blanket replacement, and came up with 
macports-libcxx instead, which had more general use across the entire lineup of 
systems.


There are definite ABI changes between libcxx versions. It is 100% not 
guaranteed that things that are built against libcxx-11.0 will work with things 
built against libcxx-5.0, or the system libcxx, and there are known examples I 
have worked through with upstream where such backward incompatibility caused 
errors, and was specifically never going to be fixed by upstream.

The exact comment was something like: “We don’t want to encourage people to 
think this should work, so we won’t support any fixes for it”.

And then a bit of a slap for still being on 10.6.8, don’t you know that is 
insecure, etc.

So I dropped all of that line of discussion a year or two ago.

Ken

Re: port:libcxx - why so old

[Ken Cunningham]

> On Nov 28, 2021, at 12:30 PM, Ken Cunningham 
>  wrote:
> 
>> Hi,
>> 
>> Judging from the version number, port:libcxx ships a version that's long 
>> outdated. Is there a reason for that, like it's the latest version that 
>> builds on all OS X versions that require the libc++ conversion?
>> 
>> I have a local version that is currently at v8.0.0 on my 10.9.5 system and 
>> have had that install the runtime shared libraries (= not the ones used by 
>> the linker) in $prefix/lib via an additional +runtime variant which also 
>> installs a launch wrapper that adds $prefix/lib to DYLD_INSERT_LIBRARIES 
>> (DYLD_LIBRARY_PATH typically isn't enough on Mac). It's a lot newer than my 
>> system versions even (than v5.0.1) and yet I can't recall having had any 
>> issues with it. On the contrary, it fixed a number of strange C++ runtime 
>> warnings in certain KDE software and must give me that bit more performance 
>> because built with a more recent clang, full CPU-specific optimisation and 
>> LTO.
>> 
>> R.
>> 
> 
> libcxx 5.0 is quite new in fact — it supports libcxx features that only show 
> up in 10.13+ (which give us the strange fact of ports that build on 10.6, and 
> 10.13+, but nothing in between. :>
> 
> I had libcxx 7.0 ready to push, which I was going to do if I ever sorted out 
> how to bootstrap it to the full Thread_Local_Storage supporting version 
> without tricks.
> 
> But I never noticed a single port that built with libcxx 7.0 that would not 
> build with libcxx 5.0, so there was no big rush.
> 
> I did replace the libc++.dylib on my 10.7.5 system with the one from llvm-5.0 
> and promptly found certain applications (ical, for example) crashing 
> regularly.
> 
> So I gave up on the plan of a blanket replacement, and came up with 
> macports-libcxx instead, which had more general use across the entire lineup 
> of systems.
> 
> 
> There are definite ABI changes between libcxx versions. It is 100% not 
> guaranteed that things that are built against libcxx-11.0 will work with 
> things built against libcxx-5.0, or the system libcxx, and there are known 
> examples I have worked through with upstream where such backward 
> incompatibility caused errors, and was specifically never going to be fixed 
> by upstream.
> 
> The exact comment was something like: “We don’t want to encourage people to 
> think this should work, so we won’t support any fixes for it”.
> 
> And then a bit of a slap for still being on 10.6.8, don’t you know that is 
> insecure, etc.
> 
> So I dropped all of that line of discussion a year or two ago.
> 
> Ken
> 
> 

Oh, BTW you can build our libcxx-5.0 with any compiler you want to — we do 
already get LTO, but if you want to build it with clang-9.0, or clang-13 to see 
if you get even newer super-duper LTO on steroids or something, go ahead. I did 
build it with clang-9.0 I think; I haven’t tried with clang 13, but it would 
probably build.

The real change in libcxx (I think it was you who pointed it out in fact) that 
was likely to break libcxx against the system libc++.dylib software was around 
llvm-8.0.

Somewhere there is a ticket where you pointed that out.

So that’s why I never personally tried a full replacement with libcxx-8.0, and 
left it libcxx-7.0 at the most (for my personal systems).

Ken

Re: OWL - wayland for mac

[Ken Cunningham]

In fact I have OWL running on my Mojave system now. It built in about 10 
minutes last week, with only minor tweaking.



However, so far I can only get wayland clients to connect to it, and 
indicate a successful connection, but not display anything. I think 
perhaps the supported wayland protocol spec has drifted a bit?


If you find a wayland client example that works with OWL, please let me 
know.


Sergey has not yet responded to my email about which clients he has been 
using to test.



Ken

Re: [10.6.8] certbot & pyOpenSSL problems

[Ken Cunningham]

> System Version: Mac OS X 10.6.8 (10K549) Kernel Version: Darwin 10.8.0 

It appears that you are running a 10.6.8 system (on a macmini, I think I saw?) 
and I think you appear to be using it as a server for various things.

In another spot, I see that it is a 32bit EFI Mac:

> platform: macOS-10.6.8-i386-64bit 

If you are not using for much else beyond being a server, I thought I would let 
you know that this tiny like c program:

https://github.com/demonicsweaters/make_single_eltorito.c

that compiles in 10 seconds or so, will slightly modify an Ubuntu 64 bit 
installer image to properly boot a 32bit EFI Mac, and you can then install 
Ubuntu 64 bit on there.

Once it is installed, you’re good to go, and you can use 64bit software 
everywhere, including any 64 bit software you download rather than get from the 
main repo, like Spotify, Google Chrome, etc.

I have this now running on 3 older Macs with 32bit EFI, and it works just 
beautifully. And if you are just using that as a server, you would be most 
likely many miles ahead to be running Ubuntu 20.04 with webmin than trying to 
keep teasing SnowLeopard along as a server.

Just my $0.02 for whatever.

Ken


(PS - dual booting a MacOSX install of 10.6 or 10.7 works just fine too, using 
ReFIND: https://www.rodsbooks.com/refind/ )

Re: [10.6.8] certbot & pyOpenSSL problems

[Ken Cunningham]

I used this set of instructions and the tool referenced in the first 
comment myself:


https://www.youtube.com/watch?v=EJA4OUIDa7Q

to upgrade the firmware in a macmini 1,1 to 2,1, to allow it to see the 
rest of the installed RAM. It was, for me, very quick and very painless.


As you noted, many macOS systems of this vintage (and other makes using 
the same Intel chipsets) can only use a little over 3GB of the 4GB of 
RAM due to hardware addressing constraints, even when running other 
operating systems like linux.



Best,


Ken

Re: latest git version no longer detected by base ==> and it is a pain to use

[Ken Cunningham]

the new git is causing headaches for me too.

It delivers errors based on the ownership of directories above it, to address 
this:

https://github.blog/2022-04-12-git-security-vulnerability-announced/

which means that any kind of git work in the macports trees (owned by macports) 
even as root now causes errors unless folder by folder exceptions are made. Yuk.

I rolled back to the last git before this fix for the time being.

Ken

Re: Reason for rpath usage on arm64

[Ken Cunningham]

> Does anyone know why things seem to be using rpath on arm64 builds

Basically, it comes down to the fact that Apple has disallowed 
DYLD_LIBRARY_PATH to work on newer systems.

gcc for Darwin has one developer working on it, practically. Iain. He has been 
maintaining it for 10+ years, wrote the objective C implementation it uses, and 
is pretty much the only reason gcc works on Darwin at all. Perhaps someone 
would fill his shoes if he stopped doing it, hard to say.

He maintains hardware machines from 10.5 to the current, and he tries to keep 
as many versions of gcc as possible running on as many systems as he can.

Once Apple disallowed DYLD_LIBRARY_PATH, he has had a lot of trouble running 
the test suites on the gcc libraries, as he can’t force the libraries under 
test to be the ones used. Instead the libraries installed are the ones used, 
which means he has to actually install the gcc before testing it.

Multiply that by perhaps 6 active gcc versions on each system, and 50+ test 
runs of gcc weekly, and you can see the problem.

Using @rpath gets around that problem very nicely.

Now, LLVM and many other software projects have the same issue. They also use 
@rpath to do it. But on INSTALLATION these @rpaths can be rewritten 
automatically by cmake (or meson) to be full pathnames. So that is what 
MacPorts does.

gcc’s build system does not currently offer that installation option. I don’t 
know how hard it would be to add it (any volunteers?) but for now at least, 
Iain is not keen on adding that @rpath rewrite-on-install business to gcc.

So gcc is installing it’s libraries with the @rpath linkages — it’s not just 
arm, by the way. It’s just arm right now because that is the tip of the spear. 
Very soon ALL gcc installs will use @rpaths, as that is (for now, barring a 
better option) the only way Iain can maintain gcc at all on Darwin.

Ken

Re: Reason for rpath usage on arm64

[Ken Cunningham]

That comes right from Iain - here's the relevant gcc ticket I believe:

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=88590

I'm sure Iain would be delighted to find a way around this.



On Tue, May 24, 2022 at 9:50 AM Joshua Root  wrote:

> On 2022-5-25 01:22 , Ken Cunningham wrote:
> >> Does anyone know why things seem to be using rpath on arm64 builds
> >
> > Basically, it comes down to the fact that Apple has disallowed
> > DYLD_LIBRARY_PATH to work on newer systems.
>
> This is the first I've heard of it. Is this supposed to be the case only
> on arm64? I know DYLD_* variables are ignored when running SIP-flagged
> executables, but that wouldn't apply to something that has just been built.
>
> We use DYLD_LIBRARY_PATH in our own build system:
> <https://github.com/macports/macports-base/blob/master/vendor/tclsh.in>
>
> - Josh
>

Re: Reason for rpath usage on arm64

[Ken Cunningham]

Oh, nothing whatsoever to do with arm64 specifically. This is going to
affect all gcc builds on all darwin-supported architectures once Iain
pushes it through to all the branches he plans to keep supporting.

On Tue, May 24, 2022 at 9:50 AM Joshua Root  wrote:

> On 2022-5-25 01:22 , Ken Cunningham wrote:
> >> Does anyone know why things seem to be using rpath on arm64 builds
> >
> > Basically, it comes down to the fact that Apple has disallowed
> > DYLD_LIBRARY_PATH to work on newer systems.
>
> This is the first I've heard of it. Is this supposed to be the case only
> on arm64? I know DYLD_* variables are ignored when running SIP-flagged
> executables, but that wouldn't apply to something that has just been built.
>
> We use DYLD_LIBRARY_PATH in our own build system:
> <https://github.com/macports/macports-base/blob/master/vendor/tclsh.in>
>
> - Josh
>

Re: Reason for rpath usage on arm64

[Ken Cunningham]

ah yes, your question was arm64-specific so that led to the gcc assumption.

the way the cmake PG configures cmake breaks testing on all archs and
systems.

this comes up regularly, eg

https://lists.macports.org/pipermail/macports-dev/2021-September/043754.html

Perhaps it might beneficially be changed.

K

On Sun, May 29, 2022 at 22:17 Ryan Schmidt  wrote:

> On May 24, 2022, at 10:22, Ken Cunningham wrote:
> >
> >> Does anyone know why things seem to be using rpath on arm64 builds
> >
> > Basically, it comes down to the fact that Apple has disallowed
> DYLD_LIBRARY_PATH to work on newer systems.
>
> > So gcc is installing it’s libraries with the @rpath linkage
>
> So you're talking about how SIP doesn't pass DYLD_LIBRARY_PATH to child
> processes on El Capitan and later, which breaks the gcc test suite,
> therefore gcc now uses @rpath at build time to fix the problem, and does
> not rewrite the install_names at install time, therefore things with @rpath
> get installed.
>
> That's not the issue that prompted me to start this thread.
>
> When I enabled the test suite for the portmidi port (which does not use
> gcc)...
>
>
> https://github.com/macports/macports-ports/commit/fd447d210e1814b1303c5013d088efa673470d77
>
> ...I set "DYLD_LIBRARY_PATH=." in test.env so that the executable it runs
> can find the library it just built. This worked fine on Catalina x86_64 but
> on Big Sur arm64 it failed with the error message:
>
> dyld: Library not loaded: @rpath/libportmidi.2.dylib
>   Referenced from: /path/to/portmidi/work/build/./qtest
>   Reason: image not found
>
> Seeing "@rpath" in this error message, I added
> "-DCMAKE_BUILD_WITH_INSTALL_RPATH=OFF" to configure.args (to counteract
> "-DCMAKE_BUILD_WITH_INSTALL_RPATH=ON" set by the cmake portgroup) and the
> problem went away.
>
> Perhaps placing the blame on @rpath was the wrong conclusion, since I now
> realize that the install_names also use @rpath on Catalina x86_64 yet the
> test worked there. I just don't understand then why disabling @rpath on Big
> Sur arm64 made the tests work.
>
> It seemed to me like I was seeing a lot of @rpath-related issues
> especially on arm64 lately, and this latest instance with portmidi prompted
> me to ask about it, but it could be that most or all of the other instances
> were related to the gcc situation you already discussed. I know we are
> using a different branch of gcc for arm64 so maybe that explains why the
> @rpath-related changes you mentioned only appear on arm64 so far.
>
>

Re: Reason for rpath usage on arm64

[Ken Cunningham]

Iain has added what MacPorts needed to the gcc branches he maintains, and
plans to push them through. This option will let us add the needed rpath to
where MacPorts puts the gcc libraries.

--with-darwin-add-path=


Also, re cmake PG, changing the cmake option for in-build library paths
should have no affect on the final installed files.

Best,

Ken


On Sun, May 29, 2022 at 23:56 Ryan Schmidt  wrote:

> On May 30, 2022, at 01:01, Ken Cunningham wrote:
>
> > ah yes, your question was arm64-specific so that led to the gcc
> assumption.
> >
> > the way the cmake PG configures cmake breaks testing on all archs and
> systems.
>
> It worked for me on Catalina x86_64 but not on Big Sur arm64, and I was
> trying to understand why this difference exists.
>
>
> > this comes up regularly, eg
> >
> >
> https://lists.macports.org/pipermail/macports-dev/2021-September/043754.html
> >
> > Perhaps it might beneficially be changed.
>
> I'm assuming that just changing the default in the cmake 1.1 portgroup
> could cause problems (or at least nonreproducibility) for ports already
> using the portgroup, so perhaps this change, plus any others we want to
> make to the cmake portgroup, should be in a new cmake 1.2 portgroup.
>
>

Re: MacPorts Status

[Ken Cunningham]

> If you are looking for some serious effort, it's the wrong
> place; there is none here, they are all volunteers sadly...
> 

> That's what I said, no coherence or coordination.
> So what are your reasons to not use HomeBrew?
> The right questions is how it needs to be,
> probably more on the organisational side
> of things than the technical I would say...

Just ignore this flamebait.
Ken

help with a regex please

[Ken Cunningham]

to fix the universal build of some ports, eg jemalloc, I would like to strip 
out the host bits from these text strings, eg convert this:

echo "--prefix=/opt/local --with-jemalloc-prefix= 
--host=aarch64-apple-darwin21.5.0 host_alias=aarch64-apple-darwin21.5.0 
CC=/usr/bin/clang”

to this:

echo "--prefix=/opt/local --with-jemalloc-prefix= CC=/usr/bin/clang”

however, my regex-fu is not strong.

Anyone know the secret reinplace regex that might do that?

thx