distfile downloads failing on https

[Jan Stary]

Recently, I have tweaked textproc/lowdown
https://github.com/macports/macports-ports/pull/1245
to download the distfile from a http master site, not a https one,
because the https server rejects the weak ssl of some older MacOS systems,
which could not download the distfile then.

That worked for a while, but now the server (kristaps.bsd.lv)
redirects any http request to https anyway, so the downloads fail again.
I understand this is a general problem, not specific to the lowdown port.

One workaround would be to have a copies of the distfiles in
http://distfiles.macports.org/, but it seems the distfile mirroring
has been broken for more than a year:
https://trac.macports.org/ticket/53347

A more general proposed solution was to bunlde a newer curl with MP,
in partcular one built against a newer SSL/TLS library:
https://trac.macports.org/ticket/51516

Is there any progress in any of these?

To be sure: when downloading a distfile, does MP always use the system's
native curl(1)? Or does it try the port's curl(1), if installed?
Or does it try the port's curl(1) _first_?

On this old system where I'm experiencing the problem, these are:

$ /usr/bin/curl --version
curl 7.19.7 (universal-apple-darwin10.0) libcurl/7.19.7 OpenSSL/0.9.8y
zlib/1.2.3
Protocols: tftp ftp telnet dict ldap http file https ftps 
Features: GSS-Negotiate IPv6 Largefile NTLM SSL libz 

$ /opt/local/bin/curl --version
curl 7.54.1 (x86_64-apple-darwin10.8.0) libcurl/7.54.1 OpenSSL/1.0.2n
zlib/1.2.11
Release-Date: 2017-06-14
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s
rtsp smb smbs smtp smtps telnet tftp 
Features: IPv6 Largefile NTLM NTLM_WB SSL libz TLS-SRP UnixSockets HTTPS-proxy 

Jan

Re: distfile downloads failing on https

[Jan Stary]

On Feb 21 12:43:28, h...@stare.cz wrote:
> To be sure: when downloading a distfile, does MP always use the system's
> native curl(1)? Or does it try the port's curl(1), if installed?
> Or does it try the port's curl(1) _first_?

Apparently not: the base /usr/bin/curl reports an "Unknown SSL error",
while /opt/local/bin/curl downloads the distfile just fine.

Re: distfile downloads failing on https

[Jan Stary]

On Feb 21 12:43:28, h...@stare.cz wrote:
> A more general proposed solution was to bunlde a newer curl with MP,
> in partcular one built against a newer SSL/TLS library:
> https://trac.macports.org/ticket/51516

It was also suggested there to recompile MP
using its own already installed curl port.

Indeed,
./configure --with-curlprefix=/opt/local/
results in MacPorts that uses its own /opt/local/bin/curl
to download distfiles, and my immediate problem disappears.

But the circular dependency on itself doesn't seem right.

Should the curl port be a fetch dependency (if there is such a thing)
for the affected ports, and should MP use the port's curl(1)
to download their distfiles?

Jan

Re: distfile downloads failing on https

[Jan Stary]

On Feb 21 13:18:47, h...@stare.cz wrote:
> On Feb 21 12:43:28, h...@stare.cz wrote:
> > A more general proposed solution was to bunlde a newer curl with MP,
> > in partcular one built against a newer SSL/TLS library:
> > https://trac.macports.org/ticket/51516
> 
> It was also suggested there to recompile MP
> using its own already installed curl port.
> 
> Indeed,
> ./configure --with-curlprefix=/opt/local/
> results in MacPorts that uses its own /opt/local/bin/curl
> to download distfiles, and my immediate problem disappears.

The problem is not curl itself of course,
but the SSL library it is linked against,
as already stated in the ticket. On my system:

$ otool -L /usr/bin/curl
/usr/bin/curl:
/usr/lib/libcurl.4.dylib (compatibility version 6.0.0, current version 6.1.0)
/usr/lib/libssl.0.9.8.dylib (compatibility version 0.9.8, current version 0.9.8)
/usr/lib/libcrypto.0.9.8.dylib (compatibility version 0.9.8, current version 
0.9.8)
/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.3)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 
125.2.11)

$ otool -L /opt/local/bin/curl
/opt/local/bin/curl:
/opt/local/lib/libcurl.4.dylib (compatibility version 9.0.0, current version 
9.0.0)
/opt/local/lib/libssl.1.0.0.dylib (compatibility version 1.0.0, current version 
1.0.0)
/opt/local/lib/libcrypto.1.0.0.dylib (compatibility version 1.0.0, current 
version 1.0.0)
/opt/local/lib/libz.1.dylib (compatibility version 1.0.0, current version 
1.2.11)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 
125.2.11)

$ port provides /opt/local/lib/libssl.1.0.0.dylib
/opt/local/lib/libssl.1.0.0.dylib is provided by: openssl

Jan

Re: distfile downloads failing on https

[Ken Cunningham]

see https://trac.macports.org/ticket/51516#comment:19

for a workaround that works quite well on Tiger up until this gets fixed

K

> On Feb 21, 2018, at 04:26, Jan Stary  wrote:
> 
>> On Feb 21 13:18:47, h...@stare.cz wrote:
>>> On Feb 21 12:43:28, h...@stare.cz wrote:
>>> A more general proposed solution was to bunlde a newer curl with MP,
>>> in partcular one built against a newer SSL/TLS library:
>>> https://trac.macports.org/ticket/51516
>> 
>> It was also suggested there to recompile MP
>> using its own already installed curl port.
>> 
>> Indeed,
>> ./configure --with-curlprefix=/opt/local/
>> results in MacPorts that uses its own /opt/local/bin/curl
>> to download distfiles, and my immediate problem disappears.
> 
> The problem is not curl itself of course,
> but the SSL library it is linked against,
> as already stated in the ticket. On my system:
> 
> $ otool -L /usr/bin/curl
> /usr/bin/curl:
> /usr/lib/libcurl.4.dylib (compatibility version 6.0.0, current version 6.1.0)
> /usr/lib/libssl.0.9.8.dylib (compatibility version 0.9.8, current version 
> 0.9.8)
> /usr/lib/libcrypto.0.9.8.dylib (compatibility version 0.9.8, current version 
> 0.9.8)
> /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.3)
> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 
> 125.2.11)
> 
> $ otool -L /opt/local/bin/curl
> /opt/local/bin/curl:
> /opt/local/lib/libcurl.4.dylib (compatibility version 9.0.0, current version 
> 9.0.0)
> /opt/local/lib/libssl.1.0.0.dylib (compatibility version 1.0.0, current 
> version 1.0.0)
> /opt/local/lib/libcrypto.1.0.0.dylib (compatibility version 1.0.0, current 
> version 1.0.0)
> /opt/local/lib/libz.1.dylib (compatibility version 1.0.0, current version 
> 1.2.11)
> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 
> 125.2.11)
> 
> $ port provides /opt/local/lib/libssl.1.0.0.dylib
> /opt/local/lib/libssl.1.0.0.dylib is provided by: openssl
> 
>Jan
> 

Re: distfile downloads failing on https

[Ken Cunningham]

I should have been more descriptive about the /opt/bootstrap part of the post 
below. 

Like  you, I didn't like the circular dependency. If you "sudo port uninstall 
active", you're hooped. So here's what I actually do on all systems 10.4 to 10.7

using Macports from this page  and more 
specifically from here 

 and with reference to the instructions for 


1. install a copy of macports from source code into /opt/bootstrap

cd MacPorts-2.4.2
./configure --prefix=/opt/bootstrap 
--with-applications-dir=/opt/bootstrap/applications
make
sudo make install
cd ..
rm -rf MacPorts-2.4.2

then as per 

bbedit /opt/bootstrap/etc/macports/macports.conf
and add 
startupitem_install no

2. once that is finished, I do this

sudo /opt/bootstrap/bin/port sync
sudo /opt/bootstrap/bin/port -v -N install curl

You should now have an independent free-standing macports installation with curl

3. now using a fresh copy of the Macports source

cd MacPorts-2.4.2
./configure --with-curlprefix=/opt/bootstrap
make
sudo make install

and then adjust your PATH to point to /opt/local/bin as usual for MacPorts.

You are done. No more circular dependency. You can use MacPorts as you wish to.

You can pretty much forget about /opt/bootstrap after that if you want to. The 
curl is 10 years newer, and so will likely last as long as you need. 


You can update /opt/bootstrap if you want to. You just have to keep the PATHs 
straight, which requires a clear head.

I always disable the PATH pointing to /opt/local/bin before I do this, and then 
either add the path to /opt/bootstrap/bin, or just:

sudo /opt/bootstrap/bin/port -v sync
sudo /opt/bootstrap/bin/port -v -N upgrade outdated



NOTE:

when Macports updates to a new version, it does not remember the curlprefix, 
and so it installs against /usr/lib/libcurl again.

For this reason, I always 

sudo port -v sync

to update macpports ports.

I manually install updates to macports using the souce code and the 
instructions in #3 above.

Sorry if this is long and seems complicated -- I was trying to be as complete 
as possible.

Ken






On 2018-02-21, at 4:18 AM, Jan Stary wrote:

> On Feb 21 12:43:28, h...@stare.cz wrote:
>> A more general proposed solution was to bunlde a newer curl with MP,
>> in partcular one built against a newer SSL/TLS library:
>> https://trac.macports.org/ticket/51516
> 
> It was also suggested there to recompile MP
> using its own already installed curl port.
> 
> Indeed,
> ./configure --with-curlprefix=/opt/local/
> results in MacPorts that uses its own /opt/local/bin/curl
> to download distfiles, and my immediate problem disappears.
> 
> But the circular dependency on itself doesn't seem right.
> 
> Should the curl port be a fetch dependency (if there is such a thing)
> for the affected ports, and should MP use the port's curl(1)
> to download their distfiles?
> 
>   Jan
> 

Re: distfile downloads failing on https

[Jan Stary]

On Feb 21 08:11:26, ken.cunningham.web...@gmail.com wrote:
> I should have been more descriptive about the /opt/bootstrap part of the post 
> below. 
> 
> Like  you, I didn't like the circular dependency. If you "sudo port uninstall 
> active", you're hooped. So here's what I actually do on all systems 10.4 to 
> 10.7
> 
> using Macports from this page  and more 
> specifically from here 
> 
>  and with reference to the instructions for 
> 
> 
> 1. install a copy of macports from source code into /opt/bootstrap
> 
> cd MacPorts-2.4.2
> ./configure --prefix=/opt/bootstrap 
> --with-applications-dir=/opt/bootstrap/applications
> make
> sudo make install
> cd ..
> rm -rf MacPorts-2.4.2
> 
> then as per 
> 
> bbedit /opt/bootstrap/etc/macports/macports.conf
> and add 
> startupitem_install no
> 
> 2. once that is finished, I do this
> 
> sudo /opt/bootstrap/bin/port sync
> sudo /opt/bootstrap/bin/port -v -N install curl
> 
> You should now have an independent free-standing macports installation with 
> curl

I like the multiple installation and the extra setup even less.

> 3. now using a fresh copy of the Macports source
> 
> cd MacPorts-2.4.2
> ./configure --with-curlprefix=/opt/bootstrap
> make
> sudo make install
> 
> and then adjust your PATH to point to /opt/local/bin as usual for MacPorts.
> You are done. No more circular dependency.

Yes. One installation of MP depends on another installation.



If I am reading https://guide.macports.org/chunked/reference.phases.html
right, there is are no "fetch dependencies". Would it make sense
to introduce fetch dependencies just like we have build dependencies
and run dependencies, so that the affected ports could specify e.g. curl,
and MP vuld use _that_ for those ports?

> NOTE:
> when Macports updates to a new version,
> it does not remember the curlprefix,
> and so it installs against /usr/lib/libcurl again.

I install and upgrade manually from git.

> Sorry if this is long and seems complicated --
> I was trying to be as complete as possible.

No problem - thank you for the insight.

Jan

Re: distfile downloads failing on https

[Clemens Lang]

Hi Jan,

On Wed, Feb 21, 2018 at 08:14:13PM +0100, Jan Stary wrote:
> If I am reading
> https://guide.macports.org/chunked/reference.phases.html right, there
> is are no "fetch dependencies". Would it make sense to introduce fetch
> dependencies just like we have build dependencies and run
> dependencies, so that the affected ports could specify e.g. curl, and
> MP vuld use _that_ for those ports?

That's basically https://trac.macports.org/ticket/51516#comment:21. See
https://trac.macports.org/ticket/51516#comment:22 for a partial response
to that.

MacPorts does not use the curl command to download files, it uses the
libcurl library. We cannot simply switch to a different libcurl library,
and reimplementing the code in pextlib1.0/curl.c to use the curl command
line is a huge effort for little gain.

-- 
Clemens

Re: distfile downloads failing on https

[Rainer Müller]

On 2018-02-21 20:14, Jan Stary wrote:
> If I am reading https://guide.macports.org/chunked/reference.phases.html
> right, there is are no "fetch dependencies". Would it make sense
> to introduce fetch dependencies just like we have build dependencies
> and run dependencies, so that the affected ports could specify e.g. curl,
> and MP vuld use _that_ for those ports?

depends_fetch exists, but apparently is not documented in the guide.

However, there should not be individual workarounds in ports, as it is a
general problem. Sorry, but your system has become too old to
communicate with servers on the internet. You will experience the same
problem with more and more servers over time as they begin to require
TLS 1.2.

The best workaround would be distfile mirroring, as everyone would
benefit from that. Most mirrors are still accessible with no TLS or less
strict cipher requirements.

Rainer

Re: distfile downloads failing on https

[Jan Stary]

On Feb 22 17:09:22, rai...@macports.org wrote:
> On 2018-02-21 20:14, Jan Stary wrote:
> > If I am reading https://guide.macports.org/chunked/reference.phases.html
> > right, there is are no "fetch dependencies". Would it make sense
> > to introduce fetch dependencies just like we have build dependencies
> > and run dependencies, so that the affected ports could specify e.g. curl,
> > and MP vuld use _that_ for those ports?
> 
> depends_fetch exists, but apparently is not documented in the guide.

Thanks for the insight. Currently, nothing is using it,
except: these explicitly clear it (why?)

  ./editors/HexFiend/Portfile:depends_fetch
  ./editors/textmate2/Portfile:depends_fetch
  ./python/py-pyobjc-cocoa/Portfile:depends_fetch
  ./python/py-pyobjc-fsevents/Portfile:depends_fetch

and this one declares 'git' as its fetch dependency:

  ./multimedia/mplayer-devel/Portfile:depends_fetch-append bin:git:git

(why don't other git-downloading ports declare it too?)

If it is this much unused, wuld it be better to remove it,
as opposed to document it and keep it for this one port?

> However, there should not be individual workarounds in ports,
> as it is a general problem.

Understood. But isn't the whole premise of depends_fetch
to allow for individual workarounds for ports that need
to fetch in a certain way / with a certain tool?

> Sorry, but your system has become too old to communicate
> with servers on the internet. You will experience the same
> problem with more and more servers over time as they begin
> to require TLS 1.2.

Yes. For now, I am using MP that is rebuilt against
a standalone curl that can download from anywhere,
as a workaround on these systems.

> The best workaround would be distfile mirroring, as everyone would
> benefit from that. Most mirrors are still accessible with no TLS or less
> strict cipher requirements.

I can't wait :-)

Thanks again,

Jan

Re: distfile downloads failing on https

[Jan Stary]

On Feb 22 19:04:41, h...@stare.cz wrote:
> > depends_fetch exists, but apparently is not documented in the guide.
> 
> Thanks for the insight. Currently, nothing is using it,
> except: these explicitly clear it (why?)

Is there a default nonempty value for depends_fetch
that makes these ports want to explicitly clear it?

>   ./editors/HexFiend/Portfile:depends_fetch
>   ./editors/textmate2/Portfile:depends_fetch
>   ./python/py-pyobjc-cocoa/Portfile:depends_fetch
>   ./python/py-pyobjc-fsevents/Portfile:depends_fetch

Re: distfile downloads failing on https

[Ryan Schmidt]

On Feb 21, 2018, at 13:14, Jan Stary wrote:
> 
> Would it make sense
> to introduce fetch dependencies just like we have build dependencies
> and run dependencies, so that the affected ports could specify e.g. curl,
> and MP vuld use _that_ for those ports?

As was mentioned, depends_fetch exists. There is usually no need for ports to 
specify it since MacPorys base does so automatically based on fetch.type. 

MacPorts fetches from ftp/http/https servers using the libcurl it was 
configured and compiled with. No amount of changing the fetch dependencies will 
change that. MacPorts base contains no code for using the curl *program*, only 
for using the curl *library*, and it will always use the library it was 
compiled with. 

Re: distfile downloads failing on https

[Ken Cunningham]

On 2018-02-21, at 11:14 AM, Jan Stary wrote:

>> You are done. No more circular dependency.
> 
> Yes. One installation of MP depends on another installation.
> 

That's not circular. CIrcular is when an installation of MacPorts depends on 
itself, and is therefore fragile. The installation of libcurl in /opt/bootstrap 
is fully independent of the one in /opt/local, and is not fragile.


I say this so as not to dissuade all those out there with broken MacPorts 
installations from using this method. 

I believe it is presently the only reliable method of getting a generally 
working MacPorts installation that can download all distfiles on older systems 
until either distfile mirroring is fixed, or until we bundle in libcurl.


Ken

Re: distfile downloads failing on https

[Ryan Schmidt]

On Feb 22, 2018, at 20:04, Ken Cunningham wrote:

> or until we bundle in libcurl.

Note that bundling libcurl is insufficient to fix the issue; we would have to 
bundle an ssl library too, and there was an objection that we don't want to be 
in the business of needing to quickly release a new version of MacPorts when a 
new vulnerability in that ssl library is found.

https://trac.macports.org/ticket/51516#comment:23

Re: distfile downloads failing on https

[Jan Stary]

On Feb 22 18:04:23, ken.cunningham.web...@gmail.com wrote:
> >> You are done. No more circular dependency.
> > Yes. One installation of MP depends on another installation.
> > 
> 
> That's not circular. Circular is when an installation of MacPorts depends on 
> itself,
> and is therefore fragile. The installation of libcurl in /opt/bootstrap is 
> fully independent of the one in /opt/local, and is not fragile.

I should have been clearer, sorry:
Yes means "Yes, you are right, it's no longer circular" :-)

> I say this so as not to dissuade all those out there with
> broken MacPorts installations from using this method. 

In the meantime, I installed me a standalone (lib)curl in $HOME
and build the real MP against that - having a separate bootstrap MP installation
just to have curl seemed like overkill.

> I believe it is presently the only reliable method of getting
> a generally working MacPorts installation that can download all distfiles
> on older systems until either distfile mirroring is fixed,
> or until we bundle in libcurl.

I root for the dist mirorrs

Jan

Re: distfile downloads failing on https

[Ken Cunningham]

> On Feb 23, 2018, at 01:49, Jan Stary  wrote:
> 
> On Feb 22 18:04:23, ken.cunningham.web...@gmail.com wrote:
 You are done. No more circular dependency.
>>> Yes. One installation of MP depends on another installation.
>>> 
>> 
>> That's not circular. Circular is when an installation of MacPorts depends on 
>> itself,
>> and is therefore fragile. The installation of libcurl in /opt/bootstrap is 
>> fully independent of the one in /opt/local, and is not fragile.
> 
> I should have been clearer, sorry:
> Yes means "Yes, you are right, it's no longer circular" :-)
> 
>> I say this so as not to dissuade all those out there with
>> broken MacPorts installations from using this method. 
> 
> In the meantime, I installed me a standalone (lib)curl in $HOME
> and build the real MP against that - having a separate bootstrap MP 
> installation
> just to have curl seemed like overkill.


I tried to see a way around it, too  but libcurl links against libssl, so 
you'd need one of those freestanding as well as you want to stay away from the 
one in /opt/local, , and a libz probably, and there was yet another that you'd 
need ... can't think of it just now. And then the new certs as well... 

In the end, a freestanding /opt/bootstrap just came forward as the best way out 
it -- let Macports take care of all the details!



> 
>> I believe it is presently the only reliable method of getting
>> a generally working MacPorts installation that can download all distfiles
>> on older systems until either distfile mirroring is fixed,
>> or until we bundle in libcurl.
> 
> I root for the dist mirorrs


Absolutely. I looked into the git webhooks a while back  but the fix did not 
appear to be a 1 day thing, for me at least. 

Finding someone with those skills to spend enough time on that has proven  to 
be tricky, as such people are on 10.13 usually and don't need this.




> 
>Jan
> 

Re: distfile downloads failing on https

[Jan Stary]

On Feb 22 19:04:41, h...@stare.cz wrote:
> On Feb 22 17:09:22, rai...@macports.org wrote:
> > On 2018-02-21 20:14, Jan Stary wrote:
> > > If I am reading https://guide.macports.org/chunked/reference.phases.html
> > > right, there is are no "fetch dependencies". Would it make sense
> > > to introduce fetch dependencies just like we have build dependencies
> > > and run dependencies, so that the affected ports could specify e.g. curl,
> > > and MP vuld use _that_ for those ports?
> > 
> > depends_fetch exists, but apparently is not documented in the guide.
> 
> Thanks for the insight. Currently, nothing is using it,
> except: these explicitly clear it (why?)
> 
>   ./editors/HexFiend/Portfile:depends_fetch
>   ./editors/textmate2/Portfile:depends_fetch
>   ./python/py-pyobjc-cocoa/Portfile:depends_fetch
>   ./python/py-pyobjc-fsevents/Portfile:depends_fetch
> 
> and this one declares 'git' as its fetch dependency:
> 
>   ./multimedia/mplayer-devel/Portfile:depends_fetch-append bin:git:git
> 
> (why don't other git-downloading ports declare it too?)
> 
> If it is this much unused, wuld it be better to remove it,
> as opposed to document it and keep it for this one port?

Seriously, this is a dep_* feature used by one port.
Would it be better to simplify base by removing it?
Or should other git-fetching ports do what mplayer-devel does?

Jan

Re: distfile downloads failing on https

[Ryan Schmidt]

On Feb 26, 2018, at 00:57, Jan Stary wrote:

> On Feb 22 19:04:41, Jan Stary wrote:
>> On Feb 22 17:09:22, Rainer Mueller wrote:
>>> On 2018-02-21 20:14, Jan Stary wrote:
 If I am reading https://guide.macports.org/chunked/reference.phases.html
 right, there is are no "fetch dependencies". Would it make sense
 to introduce fetch dependencies just like we have build dependencies
 and run dependencies, so that the affected ports could specify e.g. curl,
 and MP vuld use _that_ for those ports?
>>> 
>>> depends_fetch exists, but apparently is not documented in the guide.
>> 
>> Thanks for the insight. Currently, nothing is using it,
>> except: these explicitly clear it (why?)
>> 
>>  ./editors/HexFiend/Portfile:depends_fetch
>>  ./editors/textmate2/Portfile:depends_fetch
>>  ./python/py-pyobjc-cocoa/Portfile:depends_fetch
>>  ./python/py-pyobjc-fsevents/Portfile:depends_fetch

I will answer this below.


>> and this one declares 'git' as its fetch dependency:
>> 
>>  ./multimedia/mplayer-devel/Portfile:depends_fetch-append bin:git:git
>> 
>> (why don't other git-downloading ports declare it too?)
>> 
>> If it is this much unused, wuld it be better to remove it,
>> as opposed to document it and keep it for this one port?
> 
> Seriously, this is a dep_* feature used by one port.
> Would it be better to simplify base by removing it?

depends_fetch is used by MacPorts base for ports that use a nonstandard 
fetch.type. For example, if a port uses "fetch.type git", this causes MacPorts 
base to automatically add something like "depends_fetch-append bin:git:git". So 
of course this feature should not be removed from MacPorts base.


> Or should other git-fetching ports do what mplayer-devel does?

mplayer-devel is unique. If you look at the portfile, it overrides the fetch 
phase and fetches both from an svn repository and from a git repository. It 
sets "fetch.type svn" which causes MacPorts base to add something like 
"depends_fetch-append bin:svn:subversion". Then the port manually adds the git 
dependency.



As to why some ports clear depends_fetch or other dependencies, it's because 
some ports do not work on all macOS versions. Some of those ports include a 
block that errors out with a message to that effect. And some of those ports 
clear their dependencies in that case. It can be annoying to a user when they 
request to install a port, and lots of dependencies get installed, and then the 
actual port they wanted to use errors out saying it can never work on their OS 
version anyway. It wastes user time, bandwidth and disk space.

This is what textmate2 is doing. textmate2 uses "fetch.type git", and therefore 
has a git fetch dependency automatically added. If the user is on an old OS 
version textmate2 doesn't support, it clears its dependencies so the user 
doesn't have to wait before seeing the message that what they wanted to install 
can't be installed.

py-pyobjc-cocoa used to conditionally use "fetch.type svn" for its Python 2.4 
subport, which is why it cleared depends_fetch on Tiger, since it requires 
Leopard or later. py-pyobjc-fsevents was originally copied from py-pyobjc-cocoa 
and this code was left in, although it started off with only Python 2.6 and 
later subports, so it never had a Python 2.4 subport and never used "fetch.type 
svn". I've removed the clearing of depends_fetch from these ports now.

In the case of HexFiend, I seem to have erroneously copied the 
dependency-clearing code from another port. HexFiend has no dependencies, so 
clearing the dependencies does nothing. I've removed the dependency-clearing 
code from HexFiend now.

Re: distfile downloads failing on https

[Jan Stary]

>  If I am reading https://guide.macports.org/chunked/reference.phases.html
>  right, there is are no "fetch dependencies". Would it make sense
>  to introduce fetch dependencies just like we have build dependencies
>  and run dependencies, so that the affected ports could specify e.g. curl,
>  and MP vuld use _that_ for those ports?
> >>> 
> >>> depends_fetch exists, but apparently is not documented in the guide.
> >> 
> >> Thanks for the insight. Currently, nothing is using it,
> >> except: these explicitly clear it (why?)
> >> 
> >>  ./editors/HexFiend/Portfile:depends_fetch
> >>  ./editors/textmate2/Portfile:depends_fetch
> >>  ./python/py-pyobjc-cocoa/Portfile:depends_fetch
> >>  ./python/py-pyobjc-fsevents/Portfile:depends_fetch
> 
> I will answer this below.
> 
> >> and this one declares 'git' as its fetch dependency:
> >> 
> >>  ./multimedia/mplayer-devel/Portfile:depends_fetch-append bin:git:git
> >> 
> >> (why don't other git-downloading ports declare it too?)
> >> 
> >> If it is this much unused, wuld it be better to remove it,
> >> as opposed to document it and keep it for this one port?
> > 
> > Seriously, this is a dep_* feature used by one port.
> > Would it be better to simplify base by removing it?
> 
> depends_fetch is used by MacPorts base for ports that use a nonstandard 
> fetch.type. For example, if a port uses "fetch.type git", this causes 
> MacPorts base to automatically add something like "depends_fetch-append 
> bin:git:git". So of course this feature should not be removed from MacPorts 
> base.

Understood, thanks.

Back to the original problem: using depends_fetch is not a solution
to downloads of tar.gz failing on some https sites, right? Because
whatever the port adds to depends_fetch, the https fetching will
still be done with the (lib)curl macports was built with, right?

Jan

Re: distfile downloads failing on https

[Ryan Schmidt]

On Feb 28, 2018, at 01:12, Jan Stary wrote:

> Back to the original problem: using depends_fetch is not a solution
> to downloads of tar.gz failing on some https sites, right? Because
> whatever the port adds to depends_fetch, the https fetching will
> still be done with the (lib)curl macports was built with, right?

Right.

The solution is for me to implement distfile mirroring on our server. Our CDN 
is accessible via http or https.