Re: OpenBLAS binary packages

[Mojca Miklavec]

On Wed, 15 May 2019 at 12:23, Artur Szostak  wrote:
>
> After our 4 years of experience preparing RPM and MacPorts package 
> repositories of our software we have seen that MacPorts packages take an 
> order of magnitude more time to build and cause significantly more problems 
> than RPMs. Mainly because there is a non-trivial probability that some port 
> in the dependency tree will start breaking, and there is no way to pin or 
> encode version ranges for the Portfile dependencies to prevent this.

On top of what Ryan replied ...

1.) Most linux distributions simply freeze all the software. If you
want something newer, you need a newer distribution. That's a lot
easier to test.

>From what I remember CentOS 6(?) doesn't even have proper support for
C++11 out of the box.

If you take a rolling release like Arch linux or Debian experimental
you are also highly likely to run into various issues; you would
usually run at least Debian testing or even stable, and simply live
with ancient software that never gets updated except for security
patches. If you freeze the software, you can do a lot more testing. We
could also freeze all software with every major OS release, but that
would make the distribution much less attractive (not to say mostly
useless). While it would be useful to have a model similar to Debian
with some experimental and stable branches of MacPorts, we simply
don't have sufficient workforce to do that.

If we now update libpng, we should theoretically test (build from
source and run the test suite) every single package that depends on
libpng, and we don't even have CPU capacities to do such kind of
testing.

2.) We have too many abandoned packages and lots of broken software,
so it's nearly impossible to differentiate between something that's
not used by anyone and should have been axed years ago, and software
that's used by many and has only recently been broken. Part of this
will hopefully be easier to monitor after this summer with one of the
GSOC projects.

Mojca

Re: OpenBLAS binary packages

[Ryan Schmidt]

On May 15, 2019, at 04:01, Artur Szostak wrote:

> This raises the question: how is the MacPorts team mitigating the cost of 
> compiling OpenBLAS over and over again when trying to prepare production 
> binary packages for other ports where OpenBLAS happens to be in the 
> dependency tree?

We keep all* ports installed but deactivated on our buildbot machines. When a 
new build request comes in, it reactivates the dependencies, which is much 
quicker than having to rebuild the dependencies from source and also faster 
than having to download their binaries from a server.

*Since we are running out of space on some of our build machines, we will 
modify this so that not all ports remain installed there. For example we might 
uninstall ports that are not dependencies of any other port. See 
https://trac.macports.org/ticket/57464




On May 15, 2019, at 05:23, Artur Szostak wrote:

> I fear there is also a procedural and policy enforcement issue here. Clearly 
> different people have implemented things in different ways, giving a 
> non-standard and non-uniform look and feel to MacPorts ports. It makes for a 
> poor user experience and the MacPorts community look bad.

As far as I know we don't have a set procedure or policy for this situation of 
how to handle software that needs to be built differently on different CPUs for 
performance reasons. The creation of such a policy could certainly be discussed.


> After our 4 years of experience preparing RPM and MacPorts package 
> repositories of our software we have seen that MacPorts packages take an 
> order of magnitude more time to build and cause significantly more problems 
> than RPMs. Mainly because there is a non-trivial probability that some port 
> in the dependency tree will start breaking, and there is no way to pin or 
> encode version ranges for the Portfile dependencies to prevent this.

I'm sorry to hear that your experience with MacPorts hasn't been as good as 
with other package managers. I don't have experience with package management 
systems other than MacPorts; I would guess that applies to many of our 
contributors. So we don't know how others are handling these issues. If you or 
anyone has suggestions for specific things we should be doing differently, 
please let us know.

You are correct that MacPorts does not contain a feature that would allow 
dependencies to be pinned to a version range, and I can't think of a way that 
such a feature could work. Only one version of a port can be active at a time, 
so if you support the idea of one port requiring a specific older version of a 
dependency, you cause problem for all the other ports that wanted the current 
version of that dependency.

What we do allow is for separate ports to be created for separate versions of 
the same software, in situations where it is thought that that will be useful; 
perhaps this is what other package managers are doing as well. For example, we 
have separate ports for libsdl version 1 and 2, so that ports for software that 
has been upgraded to support libsdl 2 can use that and ports for older software 
that requires libsdl 1 can use that. If there's a specific port you're thinking 
of where we need to similarly offer multiple versions, please let us know.

With regard to a port upgrade causing another port to fail to build, as with 
any other bug, I don't know what we can do other than to fix the issues as we 
discover them, and if you've discovered something that fails to build, please 
let us know by filing a ticket in the issue tracker as usual or by submitting a 
PR to fix it. We are not mind readers and we often cannot anticipate when a 
port upgrade would cause problems for another port. Sometimes it is clear from 
the beginning: it was clear that libsdl 2 was going to be backward-incompatible 
with libsdl 1, so we made a separate port. Sometimes it is not clear: libpng 
1.6 made private certain structure fields that had previously been public, 
causing some software to fail to build. We could have at that point decided to 
create a separate libpng 1.4 port for old software; instead, we decided to fix 
the old ports that were broken by the upgrade, either by upgrading those ports 
to new libpng-1.6-compatible versions or by patching the software ourselves.

I have a feeling that package management systems on other platforms have more 
users, because on other platforms the package management system is how 
libraries and programs used by the OS itself are distributed and updated, so 
100% of users of those other platforms will use the package manager, so any 
problems will be discovered, reported, and fixed quickly. MacPorts, on the 
other hand, is only used by the fraction of macOS users who have decided to 
install it, so there is an increased likelihood that you will be the first user 
to encounter a particular problem vs. other systems with more users. The sooner 
you report a problem to us, the sooner someone can work on fixing it. Or if you 

Re: help please

[Craig Treleaven]

> On May 15, 2019, at 12:58 AM, James Linder  wrote:
> 
> Error: Failed to install qt511-qtxmlpatterns: no destroot found at: 

Appears you’ve run into an intermittent problem with MacPorts that we’ve 
unfortunately never been able to track down and fix.

See:

https://trac.macports.org/wiki/ProblemHotlist#nodestrootfound

Just clean and try again.

Craig

Re: OpenBLAS binary packages

[Nicolas Pavillon]

> On May 15, 2019, at 22:04, Chris Jones  wrote:
> 
> 
>> Now that OpenBLAS became more widely used and is now a dependency for 
>> several other ports, I could look into making a ‘generic’ version as 
>> suggested, if it is considered preferable to reduce the building time at the 
>> cost of performance.
>> I would have to confirm if a generic version could also be made, as there 
>> have also been issues with build failures with older architectures.
> 
> I suspect it would be fairly easy to do. The portfile now has in it settings 
> to enable AVX/AVX2 etc., via the NO_AVX=1 flags. You would just need to 
> always pass this in the default build, and only (conditionally) remove them 
> when a non-default 'native' variant is enabled…

Sure. I am just being a little bit paranoid, as on one hand, we are forcing 
certain compilation flags, while still relying on the automatic detection of 
the CPU architecture, and we got several tickets recently where this detection 
was not working as it was supposed to be, mostly for older machines. But it 
would indeed probably be all right. 

Also, it comes down to what should be considered a ‘generic’ version. Cutting 
down all AVX instructions would imply a significant loss in performance, even 
though probably that a large majority of computers can handle these, but I 
guess it would be pretty hard to avoid that, so that we would have to settle 
for the least common denominator. 

Cheers,

Nicolas

Re: OpenBLAS binary packages

[Nicolas Pavillon]

Hi, 

> The Portfile does disable the use of binary archives, by clearing the
> archive_sites option.
> 
> 
> The assumption seems to be that if you are using OpenBLAS, then you must
> require maximum performance and hence a build that is tuned for your
> specific hardware. A pre-built binary of course cannot provide this.

It was indeed the point when I made the Portfile. It was also consistent with 
comparable Portfiles such as atlas, which uses the same technique. 
Also, at the point of developing the Portfile, it was mostly standalone, so 
that the burden of building the port would be limited to users specifically 
installing it. 

Now that OpenBLAS became more widely used and is now a dependency for several 
other ports, I could look into making a ‘generic’ version as suggested, if it 
is considered preferable to reduce the building time at the cost of 
performance. 
I would have to confirm if a generic version could also be made, as there have 
also been issues with build failures with older architectures.

Cheers, 

Nicolas

> On May 15, 2019, at 17:38, Joshua Root  wrote:
> 
>> I am trying to understand why I am not able to get the OpenBLAS port to 
>> install exclusively from binary packages, i.e. using something like:
>>  sudo port -b install OpenBLAS
>> Even if I prepare a local repository with the prebuilt binary packages it 
>> indicates that it is not able to find any binary package and the above 
>> command fails. I have no such problem with other ports.
>> I had a quick look to see if there is something in the Portfile itself that 
>> forces compilation from source but did not find anything obvious. Is there 
>> such a mechanism? what is it? and can this be disabled / overridden?
> 
> The Portfile does disable the use of binary archives, by clearing the
> archive_sites option.
> 
> 
> The assumption seems to be that if you are using OpenBLAS, then you must
> require maximum performance and hence a build that is tuned for your
> specific hardware. A pre-built binary of course cannot provide this.
> 
> There is no way to change this short of editing the Portfile or setting
> your own archive_sites value on the command line.
> 
> - JOsh

Re: OpenBLAS binary packages

[Artur Szostak]

I fear there is also a procedural and policy enforcement issue here. Clearly 
different people have implemented things in different ways, giving a 
non-standard and non-uniform look and feel to MacPorts ports. It makes for a 
poor user experience and the MacPorts community look bad.

After our 4 years of experience preparing RPM and MacPorts package repositories 
of our software we have seen that MacPorts packages take an order of magnitude 
more time to build and cause significantly more problems than RPMs. Mainly 
because there is a non-trivial probability that some port in the dependency 
tree will start breaking, and there is no way to pin or encode version ranges 
for the Portfile dependencies to prevent this.


From: macports-users  on behalf of 
Chris Jones 
Sent: 15 May 2019 11:57
To: macports-users@lists.macports.org
Subject: Re: OpenBLAS binary packages

Hi,

On 15/05/2019 10:47 am, Joshua Root wrote:
> On 2019-5-15 19:01 , Artur Szostak wrote:
>> OK, I must be blind. Thanks for indicating archive_sites. I completely 
>> missed that when looking at the Portfile yesterday. This explains everything.
>>
>> This raises the question: how is the MacPorts team mitigating the cost of 
>> compiling OpenBLAS over and over again when trying to prepare production 
>> binary packages for other ports where OpenBLAS happens to be in the 
>> dependency tree?
>> Here at ESO we have gone to great effort to build our software as MacPorts 
>> packages, where each package is built in a pristine clean environment (using 
>> VMware virtualisation on a MacPro). Now we see that a large fraction of the 
>> time is being spent building OpenBLAS, rather than our software, which is a 
>> 3rd party dependency in our case.
>
> I would guess we're not mitigating that cost, which as you point out is
> not a good thing.
>
> My preferred solution for ports of this kind would be to build a generic
> binary for minimum spec hardware when no variants are requested, and
> have a consistently-named variant that clears archive_sites and turns on
> tuning, instruction set usage etc. specific to the user's machine. The
> gmp port is another that could use this treatment, and no doubt there
> are a few more.

I was going to suggest exactly the same, as its what I have done in a
few other places where builds can potentially benefit from 'native'
builds, i.e. use ` -march=native` or similar in the build.

Because of this I have generally called these variants 'native'. See for
instance py-tensorflow. Because this is a non-default variant, the user
has to actively use it, I never bothered with clearing archive_sites as
the build bots never build non-default variants. But perhaps it is still
a good idea to add it as well ;)

> Of course this plan "just" needs someone to do the work to implement it.

Indeed. and the best ways to achieve this are, in order of preference

1. submit a github PR implementing it, for review.
2. submit a trac ticket requesting the change (with or without a patch,
although with will likely speed up the process).

cheers Chris

>
> - Josh
>

Re: OpenBLAS binary packages

[Chris Jones]

Hi,

On 15/05/2019 10:47 am, Joshua Root wrote:

On 2019-5-15 19:01 , Artur Szostak wrote:

OK, I must be blind. Thanks for indicating archive_sites. I completely missed 
that when looking at the Portfile yesterday. This explains everything.

This raises the question: how is the MacPorts team mitigating the cost of 
compiling OpenBLAS over and over again when trying to prepare production binary 
packages for other ports where OpenBLAS happens to be in the dependency tree?
Here at ESO we have gone to great effort to build our software as MacPorts 
packages, where each package is built in a pristine clean environment (using 
VMware virtualisation on a MacPro). Now we see that a large fraction of the 
time is being spent building OpenBLAS, rather than our software, which is a 3rd 
party dependency in our case.


I would guess we're not mitigating that cost, which as you point out is
not a good thing.

My preferred solution for ports of this kind would be to build a generic
binary for minimum spec hardware when no variants are requested, and
have a consistently-named variant that clears archive_sites and turns on
tuning, instruction set usage etc. specific to the user's machine. The
gmp port is another that could use this treatment, and no doubt there
are a few more.


I was going to suggest exactly the same, as its what I have done in a 
few other places where builds can potentially benefit from 'native' 
builds, i.e. use ` -march=native` or similar in the build.


Because of this I have generally called these variants 'native'. See for 
instance py-tensorflow. Because this is a non-default variant, the user 
has to actively use it, I never bothered with clearing archive_sites as 
the build bots never build non-default variants. But perhaps it is still 
a good idea to add it as well ;)



Of course this plan "just" needs someone to do the work to implement it.


Indeed. and the best ways to achieve this are, in order of preference

1. submit a github PR implementing it, for review.
2. submit a trac ticket requesting the change (with or without a patch, 
although with will likely speed up the process).


cheers Chris



- Josh

Re: OpenBLAS binary packages

[Joshua Root]

On 2019-5-15 19:01 , Artur Szostak wrote:
> OK, I must be blind. Thanks for indicating archive_sites. I completely missed 
> that when looking at the Portfile yesterday. This explains everything.
> 
> This raises the question: how is the MacPorts team mitigating the cost of 
> compiling OpenBLAS over and over again when trying to prepare production 
> binary packages for other ports where OpenBLAS happens to be in the 
> dependency tree?
> Here at ESO we have gone to great effort to build our software as MacPorts 
> packages, where each package is built in a pristine clean environment (using 
> VMware virtualisation on a MacPro). Now we see that a large fraction of the 
> time is being spent building OpenBLAS, rather than our software, which is a 
> 3rd party dependency in our case.

I would guess we're not mitigating that cost, which as you point out is
not a good thing.

My preferred solution for ports of this kind would be to build a generic
binary for minimum spec hardware when no variants are requested, and
have a consistently-named variant that clears archive_sites and turns on
tuning, instruction set usage etc. specific to the user's machine. The
gmp port is another that could use this treatment, and no doubt there
are a few more.

Of course this plan "just" needs someone to do the work to implement it.

- Josh

Re: OpenBLAS binary packages

[Chris Jones]

On 15/05/2019 10:37 am, Joshua Root wrote:

Chris Jones wrote:

Its not possible to get binary installs for OpenBLAS due to license
conflicts between it and one of its deps, openssl IIRC. Because of this
the buildbots flag the builds as non-distributable.


That's often the reason why binary packages are not available, but not
in this case.

% ./macports-infrastructure/jobs/port_binary_distributable.tcl -v OpenBLAS
"OpenBLAS" is distributable

And indeed the archives are present in
. The Portfile just prevents
them from being used.


my mistake, my memory was failing me here, I was sure I had seen the 
builds flagged as not distributable in the build bots due to license 
conflicts...




- Josh

Re: OpenBLAS binary packages

[Joshua Root]

Chris Jones wrote:
> Its not possible to get binary installs for OpenBLAS due to license 
> conflicts between it and one of its deps, openssl IIRC. Because of this 
> the buildbots flag the builds as non-distributable.

That's often the reason why binary packages are not available, but not
in this case.

% ./macports-infrastructure/jobs/port_binary_distributable.tcl -v OpenBLAS
"OpenBLAS" is distributable

And indeed the archives are present in
. The Portfile just prevents
them from being used.

- Josh

Re: OpenBLAS binary packages

[Artur Szostak]

OK, I must be blind. Thanks for indicating archive_sites. I completely missed 
that when looking at the Portfile yesterday. This explains everything.

This raises the question: how is the MacPorts team mitigating the cost of 
compiling OpenBLAS over and over again when trying to prepare production binary 
packages for other ports where OpenBLAS happens to be in the dependency tree?
Here at ESO we have gone to great effort to build our software as MacPorts 
packages, where each package is built in a pristine clean environment (using 
VMware virtualisation on a MacPro). Now we see that a large fraction of the 
time is being spent building OpenBLAS, rather than our software, which is a 3rd 
party dependency in our case.


From: Joshua Root 
Sent: 15 May 2019 10:38:13
To: Artur Szostak
Cc: MacPorts Users
Subject: Re: OpenBLAS binary packages

> I am trying to understand why I am not able to get the OpenBLAS port to 
> install exclusively from binary packages, i.e. using something like:
>   sudo port -b install OpenBLAS
> Even if I prepare a local repository with the prebuilt binary packages it 
> indicates that it is not able to find any binary package and the above 
> command fails. I have no such problem with other ports.
> I had a quick look to see if there is something in the Portfile itself that 
> forces compilation from source but did not find anything obvious. Is there 
> such a mechanism? what is it? and can this be disabled / overridden?

The Portfile does disable the use of binary archives, by clearing the
archive_sites option.


The assumption seems to be that if you are using OpenBLAS, then you must
require maximum performance and hence a build that is tuned for your
specific hardware. A pre-built binary of course cannot provide this.

There is no way to change this short of editing the Portfile or setting
your own archive_sites value on the command line.

- JOsh

Re: no destroot found (was: help please)

[James Linder]

mac

> On 15 May 2019, at 4:22 pm, Joshua Root  wrote:
> 
>> what does this mean …
>> 
>> [haycorn] /Users/jam/DEVEL [580]% sudo port install qt511-qtwebkit
>> --->  Computing dependencies for qt511-qtwebkit
>> --->  Installing qt511-qtxmlpatterns @5.11.3_0
>> Error: Failed to install qt511-qtxmlpatterns: no destroot found at: 
>> /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_aqua_qt511/qt511-qtxmlpatterns/work/destroot
> 
> It's a bug. Quite a few people have encountered it, but we've never been
> able to reproduce it. If you know how to reproduce it, please add that
> information to .
> 
> The workaround is simple however: clean the affected port.

Thanks that did indeed work.
In tems of doing something funny - I think I did not, but here goes

port install qt5 qt5-qtwebkit

Hmmm 12 did not work (I’ll go back to see why - mysql plugins)

port uninstall `port installed |grep qt5`

port install qt511 qt511-qtwebkit

James

Re: OpenBLAS binary packages

[Chris Jones]

On 15/05/2019 9:35 am, Artur Szostak wrote:

Its not possible to get binary installs for OpenBLAS due to license
conflicts between it and one of its deps, openssl IIRC. Because of this
the buildbots flag the builds as non-distributable.


OK. But again. If I am trying to get the thing installed from a local 
repository of my own prebuilt binaries, why does that not work? How is the 
buildbot involved in that case?


Without more details its hard to comment... Run with 'port -d' to see 
debug messages to see what is actually happening internally.


However, as Joshua has reminded me OpenBLAS internally uses various 
instructions sets, AVX, AVX2 etc. These are not universally available, 
so in order to get the best build for *your* machine it forces the build 
to be run from source each time, so it can determine what your machine 
supports. This is perhaps what is affecting you with your local repo


Chris




I am trying to understand why I am not able to get the OpenBLAS port to install 
exclusively from binary packages, i.e. using something like:
sudo port -b install OpenBLAS
Even if I prepare a local repository with the prebuilt binary packages it 
indicates that it is not able to find any binary package and the above
command fails. I have no such problem with other ports.
I had a quick look to see if there is something in the Portfile itself that 
forces compilation from source but did not find anything obvious.
Is there such a mechanism? what is it? and can this be disabled / overridden?

Re: OpenBLAS binary packages

[Joshua Root]

> I am trying to understand why I am not able to get the OpenBLAS port to 
> install exclusively from binary packages, i.e. using something like:
>   sudo port -b install OpenBLAS
> Even if I prepare a local repository with the prebuilt binary packages it 
> indicates that it is not able to find any binary package and the above 
> command fails. I have no such problem with other ports.
> I had a quick look to see if there is something in the Portfile itself that 
> forces compilation from source but did not find anything obvious. Is there 
> such a mechanism? what is it? and can this be disabled / overridden?

The Portfile does disable the use of binary archives, by clearing the
archive_sites option.


The assumption seems to be that if you are using OpenBLAS, then you must
require maximum performance and hence a build that is tuned for your
specific hardware. A pre-built binary of course cannot provide this.

There is no way to change this short of editing the Portfile or setting
your own archive_sites value on the command line.

- JOsh

Re: OpenBLAS binary packages

[Artur Szostak]

> Its not possible to get binary installs for OpenBLAS due to license
> conflicts between it and one of its deps, openssl IIRC. Because of this
> the buildbots flag the builds as non-distributable.

OK. But again. If I am trying to get the thing installed from a local 
repository of my own prebuilt binaries, why does that not work? How is the 
buildbot involved in that case?

>> I am trying to understand why I am not able to get the OpenBLAS port to 
>> install exclusively from binary packages, i.e. using something like:
>>sudo port -b install OpenBLAS
>> Even if I prepare a local repository with the prebuilt binary packages it 
>> indicates that it is not able to find any binary package and the above
>> command fails. I have no such problem with other ports.
>> I had a quick look to see if there is something in the Portfile itself that 
>> forces compilation from source but did not find anything obvious.
>> Is there such a mechanism? what is it? and can this be disabled / overridden?

Re: OpenBLAS binary packages

[Chris Jones]

Hi,

Its not possible to get binary installs for OpenBLAS due to license 
conflicts between it and one of its deps, openssl IIRC. Because of this 
the buildbots flag the builds as non-distributable.


Chris

On 15/05/2019 9:18 am, Artur Szostak wrote:

Dear MacPorts users and developers,

I am trying to understand why I am not able to get the OpenBLAS port to install 
exclusively from binary packages, i.e. using something like:
   sudo port -b install OpenBLAS
Even if I prepare a local repository with the prebuilt binary packages it 
indicates that it is not able to find any binary package and the above command 
fails. I have no such problem with other ports.
I had a quick look to see if there is something in the Portfile itself that 
forces compilation from source but did not find anything obvious. Is there such 
a mechanism? what is it? and can this be disabled / overridden?

Kind regards.

Artur

no destroot found (was: help please)

[Joshua Root]

> what does this mean …
> 
> [haycorn] /Users/jam/DEVEL [580]% sudo port install qt511-qtwebkit
> --->  Computing dependencies for qt511-qtwebkit
> --->  Installing qt511-qtxmlpatterns @5.11.3_0
> Error: Failed to install qt511-qtxmlpatterns: no destroot found at: 
> /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_aqua_qt511/qt511-qtxmlpatterns/work/destroot

It's a bug. Quite a few people have encountered it, but we've never been
able to reproduce it. If you know how to reproduce it, please add that
information to .

The workaround is simple however: clean the affected port.

- Josh

OpenBLAS binary packages

[Artur Szostak]

Dear MacPorts users and developers,

I am trying to understand why I am not able to get the OpenBLAS port to install 
exclusively from binary packages, i.e. using something like:
  sudo port -b install OpenBLAS
Even if I prepare a local repository with the prebuilt binary packages it 
indicates that it is not able to find any binary package and the above command 
fails. I have no such problem with other ports.
I had a quick look to see if there is something in the Portfile itself that 
forces compilation from source but did not find anything obvious. Is there such 
a mechanism? what is it? and can this be disabled / overridden?

Kind regards.

Artur