Re: llvm / clang / cctools / ld64 version consistency question
On 28.07.2016 04:26 AM, Mihai Moldovan wrote:
> On 27.07.2016 03:22 PM, Ken Cunningham wrote:
>> Also, cctools and ld64 also have a specific variant for each version of
>> llvm, it
>> appears from the following command.
>
> The MacPorts-provided compilers (gcc-mp-*, clang-mp-*) (to my knowledge) all
> use
> the linker as installed by ld64 or ld64-${version} ports. (Actually, ld64 only
> creates wrapper-symlinks more or less to a specific ld64 version.)
> The ld64-${version} subports compile the linker (ld) with the compiler
> selected
> via one of the provided variants - on 10.9+ the default variant is llvm38, so
> libraries from that LLVM version will be linked and *should* be used at run
> time. Due to the bug you already noticed, they aren't correctly used
> currently.
> We'll need to get this fixed.
I have to amend a few things in this paragraph. By default, the libLTO as
provided by the selected LLVM version should be used.
When compiling and linking code with a specific version, though, we indeed want
to use its libLTO too, in order to always create the same binaries with the same
compiler - i.e., use the LLVM-version-specific LTO code.
So that isn't a bug.
Other than that, I think Jeremy fixed the issue (with a minor new problem
allowing clang-3.3 installs on OS X 10.12, which he will probably fix up
afterwards?) It'll take some time for it to be deployed to mirrors, but users
will probably be able to get fixed versions later today if they update.
Mihai
signature.asc
Description: OpenPGP digital signature
___
macports-dev mailing list
macports-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: llvm / clang / cctools / ld64 version consistency question
Thanks for taking the time to give such a detailed answer. I can see what I did, after reading your note and ticket -- I ran into the libLTO bug as well, but instead of seeing it as a bug, I thought I had messed up something in the tool configuration by mixing clang/ld64 versions (and I guess sorta did). I did manage to work around it and get a functional clang-3.8 system, and a functional ld64 (and deactivated clang-3.7). Maybe I'll leave all that alone for a while now :> I see Jeremy is sorting all this out so others don't fall into this trap. Ken On 2016-07-27, at 7:26 PM, Mihai Moldovan wrote: > On 27.07.2016 03:22 PM, Ken Cunningham wrote: >> Having run into an unexpected problem with clang-3.7 segfaulting not long ago >> after a 3.7 minor version update, and then an interesting libLTO incorrect >> "version error" when I tried to upgrade the compiler chain from clang-3.7 to >> clang-3.8, I'd like to have a slightly better understanding of how these >> tools >> fit together. > > This is a bug and not intended behavior. I have created a new ticket #51929 > for > this, including a description of what is wrong.[0] > ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: llvm / clang / cctools / ld64 version consistency question
On 27.07.2016 03:22 PM, Ken Cunningham wrote:
> Having run into an unexpected problem with clang-3.7 segfaulting not long ago
> after a 3.7 minor version update, and then an interesting libLTO incorrect
> "version error" when I tried to upgrade the compiler chain from clang-3.7 to
> clang-3.8, I'd like to have a slightly better understanding of how these tools
> fit together.
This is a bug and not intended behavior. I have created a new ticket #51929 for
this, including a description of what is wrong.[0]
> There is a port select mechanism for clang, and a port select mechanism for
> llvm. But would it be fair to say you would always do both of them together,
> ie
> if you're switching to clang-3.8, you'll also need to switch to llvm-3.8? This
> seems pretty obvious, and I always do switch both when I do it. Should the
> port
> select mechanism just do that automatically for both if you ask it do it for
> one?
In theory, yes, but: MacPorts by default uses the compiler as installed by Xcode
by default and if possible (GCC on older systems, clang on newer systems) with
the built-in tools (assembler, linker, ...)
Some ports though override this, because they need specific (newer) versions or
fixed compiler versions in general. Hence, ports can blacklist compilers (which
means that the first available compiler from a fallback list will be used - with
prior installation if necessary) or even select specific compilers through
whitelisting explicitly.
Using port select to create symlinks only changed ${prefix}/bin/clang (and other
stuff) to the selected variant. Nothing within MacPorts itself is relying on it,
so whether a particular version is selected or not is not relevant to MacPorts -
it's merely a convenience-provider for users, so that executing "clang" will
automatically use a specific selected version iff ${prefix}/bin is within the
user's PATH environment variable.
> Also, cctools and ld64 also have a specific variant for each version of llvm,
> it
> appears from the following command.
The MacPorts-provided compilers (gcc-mp-*, clang-mp-*) (to my knowledge) all use
the linker as installed by ld64 or ld64-${version} ports. (Actually, ld64 only
creates wrapper-symlinks more or less to a specific ld64 version.)
The ld64-${version} subports compile the linker (ld) with the compiler selected
via one of the provided variants - on 10.9+ the default variant is llvm38, so
libraries from that LLVM version will be linked and *should* be used at run
time. Due to the bug you already noticed, they aren't correctly used currently.
We'll need to get this fixed.
cctools is similar, although cctools generally only ships (important) support
binaries - not really interesting in regard to our linking problem at hand.
> sudo port -v -n upgrade --enforce-variants cctools -llvm33 -llvm34 +llvm37
> configure.compiler=macports-clang-3.7
> sudo port -v upgrade --enforce-variants ld64 -llvm33 -llvm34 +llvm37
> configure.compiler=macports-clang-3.7
You shouldn't need to override configure.compiler. (Unless you're following
"LibcxxOnOlderSystems", of course, in that case please skip this part of my
answer or take it with a grain of salt.)
`sudo port -svt install --unrequested ld64 cctools -llvm38+llvm37`
should work around the mentioned problem for now (+ leave the other build
around, which can easily be re-activated later.)
> So is it also the case that if you're switching clang versions (and then llvm
> versions) you need to run commands like the following to bring your cctools
> and
> ld64 variants into sync?
According to what I tried to explain above, no. :)
> And finally, for those of us who have installed libc++ on older systems, just
> to
> clarify that libc++ is a fixed entity, and it does not change or need to
> change
> with any of the above - build it once, install it, and leave it alone,
> basically?
Define older systems. There's a wiki page for that[1]. In general, if you
switched to libc++ once, you won't be able to go back without scrapping your
complete prefix first. And naturally, you should never delete libc++. It might
chance, but backwards compatibility should be a thing, hopefully. Generally, I'd
say you don't have to cater to it manually anymore after the initial prefix
"conversion".
Mihai
[0] https://trac.macports.org/ticket/51929
[1] https://trac.macports.org/wiki/LibcxxOnOlderSystems
signature.asc
Description: OpenPGP digital signature
___
macports-dev mailing list
macports-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: Determine before installation whether a port can be installed
> On Jul 27, 2016, at 12:09 AM, Ryan Schmidt wrote: > > I would like MacPorts to be extended so that it can be determined in advance, > from the command line, before running "port install", whether a port can be > installed (barring unexpected build failures and bugs). Currently, MacPorts > assumes any port can be installed on any system, which is not the case. Many > ports cannot be built on certain versions of macOS. Some require libc++. Some > require dependencies to be installed with a nondefault variant. Some have > other requirements. Some ports, like p5-graveyard or other ports that serve > as placeholders designed to inform the user of the discontinuation of a port, > are designed to fail. Being able to determine in advance whether a port is > supposed to be installable will let us skip those ports when triggering > builds on the buildbot. This will cut down on unnecessary work performed by > the buildbot, and will avoid unnecessary emails sent to maintainers who > already know the port will fail in those circumstances. > > How can we accomplish this? We currently use "return -code error" to trigger > these kinds of error messages from ports, but we do so within a phase, such > as pre-configure or pre-fetch, but that means that code doesn't run until > those phases are running, and I want to know before those phases run, indeed > even before dependencies are calculated and installed. > > One solution that occurs to me is to define a new "preflight" phase, to be > run before dependencies are computed. Ports can override that phase and do > whatever checks they need and exit with "return -code error" if needed. This > seems like the most straightforward and flexible solution. > > Another possible solution could be to define a new Boolean variable > "installable" to indicate if the port is installable, which would default to > "yes". If a port sets this to "no", MacPorts could print a generic failure > message. There could be a second variable which the port developer could set > to a custom failure message. > > A third possibility could be to codify each of the reasons why a build might > fail, and introduce new variables for each reason. For example, a variable to > indicate the supported C++ libraries, or a variable to indicate the supported > macOS versions. There might be some advantage to this, in that it could be > used to programmatically answer questions like what C++ libraries or macOS > versions a port supports, but it is the least flexible and most complicated > solution. Could I submit that there may be two issues being conflated? 1) Some ports are known to crap out on our buildbots. 2) It is known that some ports cannot be installed on particular system configurations. For 1), I think we should simply add a keyword to the portfile that tells the buildbot not to attempt to build this port. “mp_buildbot_skip” or some such. Add this to the p5-graveyard and all other ports that we know the buildbot cannot or should not attempt to build. For 2), we already have a series of tools that provide information about the environment and let the port do the right thing. I like the idea of a preflight phase—a number of ports ‘abuse’ the pre-fetch phase to essentially do this. However I’m not sure how prevalent or serious the problem is in 2) that we’re trying to solve. Craig (And there is the -y flag to ‘dry-run’ the installation.) ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: Determine before installation whether a port can be installed
> On Jul 27, 2016, at 8:26 AM, m...@macports.org wrote: > > Hi Ryan, > > >> On Jul 26, 2016, at 10:09 PM, Ryan Schmidt wrote: >> >> I would like MacPorts to be extended so that it can be determined in >> advance, from the command line, before running "port install", whether a >> port can be installed (barring unexpected build failures and bugs). >> Currently, MacPorts assumes any port can be installed on any system, which >> is not the case. Many ports cannot be built on certain versions of macOS. >> Some require libc++. Some require dependencies to be installed with a >> nondefault variant. Some have other requirements. Some ports, like >> p5-graveyard or other ports that serve as placeholders designed to inform >> the user of the discontinuation of a port, are designed to fail. Being able >> to determine in advance whether a port is supposed to be installable will >> let us skip those ports when triggering builds on the buildbot. This will >> cut down on unnecessary work performed by the buildbot, and will avoid >> unnecessary emails sent to maintainers who already know the port will fail >> in those circumstances. >> >> How can we accomplish this? We currently use "return -code error" to trigger >> these kinds of error messages from ports, but we do so within a phase, such >> as pre-configure or pre-fetch, but that means that code doesn't run until >> those phases are running, and I want to know before those phases run, indeed >> even before dependencies are calculated and installed. >> >> One solution that occurs to me is to define a new "preflight" phase, to be >> run before dependencies are computed. Ports can override that phase and do >> whatever checks they need and exit with "return -code error" if needed. This >> seems like the most straightforward and flexible solution. >> >> Another possible solution could be to define a new Boolean variable >> "installable" to indicate if the port is installable, which would default to >> "yes". If a port sets this to "no", MacPorts could print a generic failure >> message. There could be a second variable which the port developer could set >> to a custom failure message. >> >> A third possibility could be to codify each of the reasons why a build might >> fail, and introduce new variables for each reason. For example, a variable >> to indicate the supported C++ libraries, or a variable to indicate the >> supported macOS versions. There might be some advantage to this, in that it >> could be used to programmatically answer questions like what C++ libraries >> or macOS versions a port supports, but it is the least flexible and most >> complicated solution. > > In my mind, the existing and largely useless “platform" variable should be > re-purposed and used for this purpose. It currently usually says “darwin” but > that could be expanded for each version of MacOS. > > To implement this, I think the default value in base should be changed to the > following for the next version of Macports for Sierra: > > platforms tiger leopard snowleopard lion mountainlion mavericks yosemite > elcapitan sierra > > Then we remove the “platform” variable from all Portfiles when we release the > new version of base. For ports that cannot build on, for example, lion the > Portfile’s platforms variable can either be overridden as such: > > platforms tiger leopard snowleopard mountainlion mavericks yosemite elcapitan > sierra > > or we expand base to allow a construct like: > > platforms-remove lion > > Or maybe this functionality already exists? > > For each new version of MacOS, we need to release a new version of base > anyway. That version of base will get the new MacOS version added to the > default “platforms” variable… > > platforms tiger leopard snowleopard mountainlion mavericks yosemite elcapitan > sierra sequoia > > For the libc++ issue, we simply add a keyword: > > requires_libc++ yes > > The p5-graveyard port(s) can set the platforms variable to blank. Your suggestion seems like a variation of my "third possibility", which has the disadvantages that every new reason why a port might not be installable would need a new variable introduced in MacPorts base and a new release of MacPorts base. What about ports that need the user to install Java first? What about ports that need the user to download a file manually first (for example oracle-instantclient which requires users to agree to a license agreement before downloading distfiles)? Your idea for the p5-graveyard and similar ports to clear the platforms variable to indicate the port is not installable is clever, but wouldn't it be clearer to just call the variable "installable", which the port could set using any logic it requires, or to put such logic in a preflight phase? Your idea to use string names for each version of macOS has the problem that it means a ton of updates have to happen each time a new version of macOS is released, which is
Re: Determine before installation whether a port can be installed
Hi Ryan, > On Jul 26, 2016, at 10:09 PM, Ryan Schmidt wrote: > > I would like MacPorts to be extended so that it can be determined in advance, > from the command line, before running "port install", whether a port can be > installed (barring unexpected build failures and bugs). Currently, MacPorts > assumes any port can be installed on any system, which is not the case. Many > ports cannot be built on certain versions of macOS. Some require libc++. Some > require dependencies to be installed with a nondefault variant. Some have > other requirements. Some ports, like p5-graveyard or other ports that serve > as placeholders designed to inform the user of the discontinuation of a port, > are designed to fail. Being able to determine in advance whether a port is > supposed to be installable will let us skip those ports when triggering > builds on the buildbot. This will cut down on unnecessary work performed by > the buildbot, and will avoid unnecessary emails sent to maintainers who > already know the port will fail in those circumstances. > > How can we accomplish this? We currently use "return -code error" to trigger > these kinds of error messages from ports, but we do so within a phase, such > as pre-configure or pre-fetch, but that means that code doesn't run until > those phases are running, and I want to know before those phases run, indeed > even before dependencies are calculated and installed. > > One solution that occurs to me is to define a new "preflight" phase, to be > run before dependencies are computed. Ports can override that phase and do > whatever checks they need and exit with "return -code error" if needed. This > seems like the most straightforward and flexible solution. > > Another possible solution could be to define a new Boolean variable > "installable" to indicate if the port is installable, which would default to > "yes". If a port sets this to "no", MacPorts could print a generic failure > message. There could be a second variable which the port developer could set > to a custom failure message. > > A third possibility could be to codify each of the reasons why a build might > fail, and introduce new variables for each reason. For example, a variable to > indicate the supported C++ libraries, or a variable to indicate the supported > macOS versions. There might be some advantage to this, in that it could be > used to programmatically answer questions like what C++ libraries or macOS > versions a port supports, but it is the least flexible and most complicated > solution. In my mind, the existing and largely useless “platform" variable should be re-purposed and used for this purpose. It currently usually says “darwin” but that could be expanded for each version of MacOS. To implement this, I think the default value in base should be changed to the following for the next version of Macports for Sierra: platforms tiger leopard snowleopard lion mountainlion mavericks yosemite elcapitan sierra Then we remove the “platform” variable from all Portfiles when we release the new version of base. For ports that cannot build on, for example, lion the Portfile’s platforms variable can either be overridden as such: platforms tiger leopard snowleopard mountainlion mavericks yosemite elcapitan sierra or we expand base to allow a construct like: platforms-remove lion Or maybe this functionality already exists? For each new version of MacOS, we need to release a new version of base anyway. That version of base will get the new MacOS version added to the default “platforms” variable… platforms tiger leopard snowleopard mountainlion mavericks yosemite elcapitan sierra sequoia For the libc++ issue, we simply add a keyword: requires_libc++ yes The p5-graveyard port(s) can set the platforms variable to blank. Cheers! Frank ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
llvm / clang / cctools / ld64 version consistency question
Dear smart people, Having run into an unexpected problem with clang-3.7 segfaulting not long ago after a 3.7 minor version update, and then an interesting libLTO incorrect "version error" when I tried to upgrade the compiler chain from clang-3.7 to clang-3.8, I'd like to have a slightly better understanding of how these tools fit together. There is a port select mechanism for clang, and a port select mechanism for llvm. But would it be fair to say you would always do both of them together, ie if you're switching to clang-3.8, you'll also need to switch to llvm-3.8? This seems pretty obvious, and I always do switch both when I do it. Should the port select mechanism just do that automatically for both if you ask it do it for one? Also, cctools and ld64 also have a specific variant for each version of llvm, it appears from the following command. sudo port -v -n upgrade --enforce-variants cctools -llvm33 -llvm34 +llvm37 configure.compiler=macports-clang-3.7 sudo port -v upgrade --enforce-variants ld64 -llvm33 -llvm34 +llvm37 configure.compiler=macports-clang-3.7 So is it also the case that if you're switching clang versions (and then llvm versions) you need to run commands like the following to bring your cctools and ld64 variants into sync? And finally, for those of us who have installed libc++ on older systems, just to clarify that libc++ is a fixed entity, and it does not change or need to change with any of the above - build it once, install it, and leave it alone, basically? Hope this question is clear. Thanks, Ken___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
