Why does macports require a migration across major version bumps?

[Eitan Adler]

What is the underlying reason to require a reinstall across major
versions? I did something silly/stupid/smart  and removed the check
from libexec/macports/lib/macports1.0/macports.tcl and just did 'sudo
port upgrade outdated' to rebuild the world. It got about 90% of the
way through and only errored on a few expected
broken-for-other-reasons ports.

Obvious I'm down the deep hole of unsupported actions but I'm curious
about what gets missed when someone does this?


-- 
Eitan Adler

Re: Why does macports require a migration across major version bumps?

[Mojca Miklavec]

On 30 August 2017 at 03:01, Eitan Adler wrote:
> What is the underlying reason to require a reinstall across major
> versions? I did something silly/stupid/smart  and removed the check
> from libexec/macports/lib/macports1.0/macports.tcl and just did 'sudo
> port upgrade outdated' to rebuild the world. It got about 90% of the
> way through and only errored on a few expected
> broken-for-other-reasons ports.
>
> Obvious I'm down the deep hole of unsupported actions but I'm curious
> about what gets missed when someone does this?

It's hard or nearly impossible to tell. Most of the time everything
will work until you try to upgrade or install something new. Problems
might include:

- a port A that builds differently depending on OS version and another
port B that has certain expectations of A
- change of compiler, potentially incompatible cincompatible compiler

Re: Why does macports require a migration across major version bumps?

[Mojca Miklavec]

Hi,

Sorry, I accidentally clicked on the wrong place (Send) while the
browser was doing its smart layout changes.

On 30 August 2017 at 12:16, Mojca Miklavec wrote:
> On 30 August 2017 at 03:01, Eitan Adler wrote:
>> What is the underlying reason to require a reinstall across major
>> versions? I did something silly/stupid/smart  and removed the check
>> from libexec/macports/lib/macports1.0/macports.tcl and just did 'sudo
>> port upgrade outdated' to rebuild the world. It got about 90% of the
>> way through and only errored on a few expected
>> broken-for-other-reasons ports.
>>
>> Obvious I'm down the deep hole of unsupported actions but I'm curious
>> about what gets missed when someone does this?

It's hard or nearly impossible to tell. Most of the time everything
will work until you try to upgrade or install something new. Problems
might include:

- a port A that builds differently depending on OS version and another
port B that has certain expectations of A; or maybe a port doesn't
even build and an upgrade will cause you troubles while B could easily
skip dependency on A
- change of compiler, potentially totally incompatible compiler that
will lead to further crashes
- some software hardcodes the compiler which will no longer exist on
the new machine and thus that software will be broken
- some software or libraries remember existing features of the
compiler used; when you compile another software against that library,
wrong features will be used and you might run into unexpected troubles
(anything from build failures to crashes)

The worst idea would be keeping the installation for transition from
10.8 to 10.9. I guess your system would pretty much "explode" there
due to the change of default stdlib, not to mention inability to
install many ports.

The best part is that most likely 99% of stuff will work, but those
remaining 1% might easily cost you 99% of time you'll spend fighting
with MacPorts. You might run into random problems that will likely
cost you a lot more time the time spent doing a proper migration in
the first place. Other people won't be able to help you and your time
spent debugging will be totally wasted as it won't help anyone else in
the long term. If you spend time on something that everyone could
benefit from, that would be your time much better spent than fighting
with your machine for its own sake.

Of course there are also lots of problems when using macports
"properly" :), like ports no longer compiling on newer or older OSes
or using some mix-and-match compilers, compiler bugs, software bugs
...

Random thoughts. But I have absolutely no experience with this, so
you'll probably know it better if you try it out.

Mojca

Re: Why does macports require a migration across major version bumps?

[Clemens Lang]

Hi,

On Tue, Aug 29, 2017 at 06:01:36PM -0700, Eitan Adler wrote:
> What is the underlying reason to require a reinstall across major
> versions? I did something silly/stupid/smart  and removed the check
> from libexec/macports/lib/macports1.0/macports.tcl and just did 'sudo
> port upgrade outdated' to rebuild the world.

Use 'sudo port -f selfupdate' instead, which should also recompile your
version of MacPorts. I'm not sure whether this will work in the
'disabled due to OS upgrade' mode, but if it doesn't, maybe you could
submit a pull request to allow it?

> It got about 90% of the way through and only errored on a few expected
> broken-for-other-reasons ports.
> 
> Obvious I'm down the deep hole of unsupported actions but I'm curious
> about what gets missed when someone does this?

The reason why we don't recommend doing this is that we don't know
whether things will keep working over an OS upgrade. A couple of
examples where this could lead to problems:

 - Architecture changes. Builds of ports might try to run old binaries
   built for old architectures and fail to build, or even worse, try to
   link against old libraries built for old architectures and fail to
   link.
 - Changes in the tooling Apple ships with macOS. The most famous
   example of this is probably the removal of gnutar in favor of BSD
   tar, which caused extractions in MacPorts to fail.
 - Missing libraries. Apple might remove (or upgrade to an incompatible
   version) a library that was required by one of the build tools used
   by the builds of our ports, causing those tools to fail to start and
   thus likely fail the build. For example, you might have GNU awk
   installed with MacPorts, but (for some reasons) it might link against
   an older libintl.dylib in /usr/lib, which no longer exists.
   Attempting to start this copy of gawk would immediately abort, and
   builds would fail (even though you have a perfectly working other
   version of awk around).

Most of these things should not cause problems if there are no
'automagic' dependencies, i.e. dependencies that get picked up and used
when they are present. In the absence of such problems, MacPorts should
already sort all ports topologically and re-installed them one-by-one
without issue.

In theory, reinstalling MacPorts and using 'sudo port -t upgrade
outdated' after an OS update should thus have the same effect as the
migration instructions. In fact, I've been exclusively doing this
approach for my migration for the last three OS upgrades. However, I
also happen to know what I'm doing and how to debug things if they
break, so I cannot generall recommend this to others.

-- 
Clemens

Re: Why does macports require a migration across major version bumps?

[Joshua Root]

On 2017-8-30 11:01 , Eitan Adler wrote:

What is the underlying reason to require a reinstall across major
versions? I did something silly/stupid/smart  and removed the check
from libexec/macports/lib/macports1.0/macports.tcl and just did 'sudo
port upgrade outdated' to rebuild the world. It got about 90% of the
way through and only errored on a few expected
broken-for-other-reasons ports.

Obvious I'm down the deep hole of unsupported actions but I'm curious
about what gets missed when someone does this?


It's the same reason there are separate installers per OS version in the 
first place: Many features are detected at configure time. If the same 
features are not available at runtime, stuff breaks.


It's possible that nothing relevant changes between certain major OS 
versions. It definitely does between others.


If we were able to detect all features at runtime, we could have a 
single (base) build that would work on all supported OS versions. All 
the many ports that detect features at configure time would still need 
to be reinstalled of course.


- Josh

Re: Why does macports require a migration across major version bumps?

[Eitan Adler]

On Friday, 1 September 2017, Joshua Root  wrote:

> On 2017-8-30 11:01 , Eitan Adler wrote:
>
>> What is the underlying reason to require a reinstall across major
>> versions? I did something silly/stupid/smart  and removed the check
>> from libexec/macports/lib/macports1.0/macports.tcl and just did 'sudo
>> port upgrade outdated' to rebuild the world. It got about 90% of the
>> way through and only errored on a few expected
>> broken-for-other-reasons ports.
>>
>> Obvious I'm down the deep hole of unsupported actions but I'm curious
>> about what gets missed when someone does this?
>>
>
> It's the same reason there are separate installers per OS version in the
> first place: Many features are detected at configure time. If the same
> features are not available at runtime, stuff breaks.
>
> It's possible that nothing relevant changes between certain major OS
> versions. It definitely does between others.
>
> If we were able to detect all features at runtime, we could have a single
> (base) build that would work on all supported OS versions. All the many
> ports that detect features at configure time would still need to be
> reinstalled of course.


Note that I am doing a complete reinstall by using port upgrade outdated.
Is this not sufficient?

>
> - Josh
>


-- 
Sent from my Turing Machine

Re: Why does macports require a migration across major version bumps?

[Joshua Root]

On 2017-9-2 16:29 , Eitan Adler wrote:



On Friday, 1 September 2017, Joshua Root > wrote:


On 2017-8-30 11:01 , Eitan Adler wrote:

What is the underlying reason to require a reinstall across major
versions? I did something silly/stupid/smart  and removed the check
from libexec/macports/lib/macports1.0/macports.tcl and just did
'sudo
port upgrade outdated' to rebuild the world. It got about 90% of the
way through and only errored on a few expected
broken-for-other-reasons ports.

Obvious I'm down the deep hole of unsupported actions but I'm
curious
about what gets missed when someone does this?


It's the same reason there are separate installers per OS version in
the first place: Many features are detected at configure time. If
the same features are not available at runtime, stuff breaks.

It's possible that nothing relevant changes between certain major OS
versions. It definitely does between others.

If we were able to detect all features at runtime, we could have a
single (base) build that would work on all supported OS versions.
All the many ports that detect features at configure time would
still need to be reinstalled of course.


Note that I am doing a complete reinstall by using port upgrade 
outdated. Is this not sufficient?


*If* you have installed a version of macports base configured for your 
current OS (which is what you were initially not doing), and *if* you 
use trace mode (the -t flag), this should be sufficient.


Simply upgrading the ports without trace mode can fail due to things 
like opportunistic use of tools by configure scripts (e.g. grep, gsed, 
gawk). These aren't actually needed because the system versions work 
fine, but autotools prefers the GNU versions when it finds them. So 
there's no dependency declared, so we can't ensure that things are 
upgraded in the right order, so executables may fail to run due to 
missing or incompatible libraries at certain points, or more subtle 
incorrect behaviour may result from using something that was configured 
for the old OS version.


There are many examples of such failures, which were resolved by 
following the Migration instructions, in Trac and the mailing lists over 
the years.


- Josh

Re: Why does macports require a migration across major version bumps?

[db]

On 30 Aug 2017, at 20:29, Clemens Lang  wrote:
> Use 'sudo port -f selfupdate' instead, which should also recompile your
> version of MacPorts.

> In theory, reinstalling MacPorts and using 'sudo port -t upgrade
> outdated' after an OS update should thus have the same effect as the
> migration instructions.

Running these two commands would then leave you with a working system, except 
for those arch-dependent ports?