Re: Disable ftp in pkg_add syspatch sysupgrade

[Andy Lemin]

For completeness, I discovered I was having issues with downloading the sources 
for the sysupgrade command on my edge firewall also! So it was not limited to 
internet servers as first thought.

Since upgrading the 6.6 (had to run sysupgrade 4 times to get it to complete 
the downloads), the issues seems to have been resolved and now all packages are 
installing first time every time..

So I am not sure if this is just me or a 6.5 issue. Most likely the former ;)

Kind regards, Andy.



Sent from a teeny tiny keyboard, so please excuse typos

> On 31 Oct 2019, at 01:47, Stuart Henderson  wrote:
> 
>> On 2019-10-30, Andrew Lemin  wrote:
>> - But throws errors when I try and use flavours which is critical for
>> installing python for example (NB; This is a different error to before,
>> where I was getting 'timeout' instead of 'Invalid argument');
>> [HOME]root@testbsd1:/local#pkg_add python%2 py-pip python%3 py3-pip
>> py3-setuptools
>> quirks-3.124 signed on 2019-10-16T20:27:45Z
>> http://mirror.bytemark.co.uk/pub/OpenBSD/6.5/packages/amd64/py3-setuptools-40.0.0v0.tgz:
>> ftp: Receiving HTTP reply: Invalid argument
>> signify: gzheader truncated
>> Couldn't install py3-setuptools-40.0.0v0
> 
> Odd. Can you try replicating on 6.6?
> 
> Does "pkg_add py-pip py3-pip py3-setuptools" (i.e. allow pkg_add to
> find the dependencies by itself) work?
> 
> 

Re: Disable ftp in pkg_add syspatch sysupgrade

[Andy Lemin]

Hahaha
Thanks Theo, that made me smile.

But you have answered my question perfectly, albeit in a round about way.

Indeed it doesn’t matter what it is called, and would be clearer with a generic 
name, as we got caught out by a program calling another program with colliding 
name.

For example, Having ‘pkg_add’ call a program named ‘ftp’ to perform http and 
https downloads. But where errors in the ftp subprocess are printed by the 
pkg_add process, making it seem like pkg_add was failing on an ftp protocol 
request, rather than the ‘ftp’ client process failing (while doing an http 
call)..

So I think it was pretty fair for us to end up scratching our heads ;)

Thanks, Andy.


Sent from a teeny tiny keyboard, so please excuse typos

> On 30 Oct 2019, at 15:54, Theo de Raadt  wrote:
> 
> Andrew Lemin  wrote:
> 
>> To me this seems unusual (was expecting 'curl' or 'wget' etc to avoid code
>> duplication) and confusing? What do you think?
> 
> curl is not in openbsd
> 
> wget is not in openbsd
> 
> Maybe we should rename our downloading software to lemin, which is
> obviously a randomly chosen name with some obscure acronym we'll invent
> to back the name, being a name noone recognizes we can probably avoid
> assumptions as to what it does, whether it does ftp, or http, or https,
> or who knows what.  Of course such a strange name would also lead people
> to not discovering it, and make them install some monster software
> package off the internet with another strange name.
> 
> In summary I think it's turning into a shitty world with selection by
> meme.
> 
> 

Re: Disable ftp in pkg_add syspatch sysupgrade

[Stuart Henderson]

On 2019-10-30, Andrew Lemin  wrote:
> - But throws errors when I try and use flavours which is critical for
> installing python for example (NB; This is a different error to before,
> where I was getting 'timeout' instead of 'Invalid argument');
> [HOME]root@testbsd1:/local#pkg_add python%2 py-pip python%3 py3-pip
> py3-setuptools
> quirks-3.124 signed on 2019-10-16T20:27:45Z
> http://mirror.bytemark.co.uk/pub/OpenBSD/6.5/packages/amd64/py3-setuptools-40.0.0v0.tgz:
> ftp: Receiving HTTP reply: Invalid argument
> signify: gzheader truncated
> Couldn't install py3-setuptools-40.0.0v0

Odd. Can you try replicating on 6.6?

Does "pkg_add py-pip py3-pip py3-setuptools" (i.e. allow pkg_add to
find the dependencies by itself) work?

Re: Disable ftp in pkg_add syspatch sysupgrade

[Theo de Raadt]

Andrew Lemin  wrote:

> To me this seems unusual (was expecting 'curl' or 'wget' etc to avoid code
> duplication) and confusing? What do you think?

curl is not in openbsd

wget is not in openbsd

Maybe we should rename our downloading software to lemin, which is
obviously a randomly chosen name with some obscure acronym we'll invent
to back the name, being a name noone recognizes we can probably avoid
assumptions as to what it does, whether it does ftp, or http, or https,
or who knows what.  Of course such a strange name would also lead people
to not discovering it, and make them install some monster software
package off the internet with another strange name.

In summary I think it's turning into a shitty world with selection by
meme.

Re: Disable ftp in pkg_add syspatch sysupgrade

[Andrew Lemin]

Hi gents,

Sorry for the slow reply, and thank you for all your responses! :D

Raf, you are correct. It seems that the ftp client is performing an http(s)
downloads.
To me this seems unusual (was expecting 'curl' or 'wget' etc to avoid code
duplication) and confusing? What do you think?

Stuart, thanks for your suggestion. This confirmed the ftp client is using
http(s);
[HOME]root@testbsd1:/local#pgrep -lf ftp
40379 /usr/bin/ftp -o -
http://mirror.bytemark.co.uk/pub/OpenBSD/6.5/packages-stable/amd64/quirks-3.124.tgz

Tom/PJ, Understood. Was just very confused why ftp was getting involved..


Anyway, I have tested this some more, and it looks like the issue is
related to when using "flavors", and looks like maybe their is some sort of
timeout occuring maybe.

- We can see that pkg_add is working fine when specifying packages
explicitly;
[HOME]root@testbsd1:/local#pkg_add sudo--gettext bash htop
quirks-3.124 signed on 2019-10-16T20:27:45Z
[HOME]root@testbsd1:/local#pkg_add vim--no_x11-perl-python3-ruby
unzip--iconv bzip2 git fzf
quirks-3.124 signed on 2019-10-16T20:27:45Z

- But throws errors when I try and use flavours which is critical for
installing python for example (NB; This is a different error to before,
where I was getting 'timeout' instead of 'Invalid argument');
[HOME]root@testbsd1:/local#pkg_add python%2 py-pip python%3 py3-pip
py3-setuptools
quirks-3.124 signed on 2019-10-16T20:27:45Z
http://mirror.bytemark.co.uk/pub/OpenBSD/6.5/packages/amd64/py3-setuptools-40.0.0v0.tgz:
ftp: Receiving HTTP reply: Invalid argument
signify: gzheader truncated
Couldn't install py3-setuptools-40.0.0v0

- This package is accessible as seen here;
[HOME]root@testbsd1:/local#wget
http://mirror.bytemark.co.uk/pub/OpenBSD/6.5/packages/amd64/py3-setuptools-40.0.0v0.tgz
/tmp/
--2019-10-30 14:29:28--
http://mirror.bytemark.co.uk/pub/OpenBSD/6.5/packages/amd64/py3-setuptools-40.0.0v0.tgz
Resolving mirror.bytemark.co.uk (mirror.bytemark.co.uk)... 80.68.83.150,
212.110.163.12, 2001:41c8:20:5e6::150, ...
Connecting to mirror.bytemark.co.uk (mirror.bytemark.co.uk)|80.68.83.150|:80...
connected.
HTTP request sent, awaiting response... 200 OK
Length: 731604 (714K) [application/x-gzip]
Saving to: ‘py3-setuptools-40.0.0v0.tgz’

py3-setuptools-40.0.0v0.tgz
100%[===>]
714.46K   270KB/sin 2.6s

- And works if specified on its own;
[HOME]root@testbsd1:/local#pkg_add py3-setuptools
quirks-3.124 signed on 2019-10-16T20:27:45Z

If I try the line with flavors again "pkg_add python%2 py-pip python%3
py3-pip py3-setuptools" it works..

As others would be crying about this too if it were a wide issue, I thought
this was maybe a bad mirror...
So I have now tried every mirror in the UK, and they all do the same thing
- intermittent issues accessing packages when using flavours..

I am not running squid or any kind of web proxy, http and https are being
passed out with nothing more than standard NAT and a pass rule.
I will try and figure out what is going on. Leave this with me. If I find
anything meanful and useful I will let you know. For now, consider this an
issue with my setup..

PS; has anyone managed to get ftpproxy working in an rdomain?

Thanks for your time and responses.. :)
Andy.

On Wed, Oct 30, 2019 at 9:17 AM PJ  wrote:

> Am 30.10.19 um 07:32 schrieb tom ryan:
> > On 2019-10-29 20:19, PJ wrote:
> >> Am 28.10.19 um 23:52 schrieb Stuart Henderson:
> >>> On 2019-10-28, Andy Lemin  wrote:
>  Hi guys,
> 
>  Does anyone know if it is possible to completely disable ftp in the
> package management utilities; pkg_add, syspatch, sysupgrade etc?
> 
>  My PKG_PATH references http:// urls, as does /etc/install. But I
> cannot stop these tools trying to use ftp which does not work! :(
> >>> Can you show some example URLs, for example from "pgrep -lf ftp" while
> >>> trying to use one of these utilities?
> >>>
> >>> The only place I would expect to see ftp:// URLs used
> >> grep ftp /usr/sbin/sysupgrade
> > $ grep -ne ftp -e URL -e MIRROR /usr/sbin/sysupgrade
> > 102:0)  MIRROR=$(sed 's/#.*//;/^$/d' /etc/installurl) 2>/dev/null ||
> > 103:MIRROR=https://cdn.openbsd.org/pub/OpenBSD
> > 105:1)  MIRROR=$1
> > 117:URL=${MIRROR}/snapshots/${ARCH}/
> > 119:URL=${MIRROR}/${NEXT_VERSION}/${ARCH}/
> > 136:unpriv -f SHA256.sig ftp -Vmo SHA256.sig ${URL}SHA256.sig
> > 176:unpriv -f $f ftp -Vmo ${f} ${URL}${f}
> >
> > Your point?
>
> I understand that I misread the question, sorry.
>
>
> >>> is when fetching
> >>> certain distfiles while building some things from ports (and they would
> >>> usually fallback to http://ftp.openbsd.org/pub/OpenBSD/distfiles if
> >>> the ftp fetch failed)..
>
>

Re: Disable ftp in pkg_add syspatch sysupgrade

[PJ]

Am 30.10.19 um 07:32 schrieb tom ryan:
> On 2019-10-29 20:19, PJ wrote:
>> Am 28.10.19 um 23:52 schrieb Stuart Henderson:
>>> On 2019-10-28, Andy Lemin  wrote:
 Hi guys,

 Does anyone know if it is possible to completely disable ftp in the 
 package management utilities; pkg_add, syspatch, sysupgrade etc?

 My PKG_PATH references http:// urls, as does /etc/install. But I cannot 
 stop these tools trying to use ftp which does not work! :(
>>> Can you show some example URLs, for example from "pgrep -lf ftp" while
>>> trying to use one of these utilities?
>>>
>>> The only place I would expect to see ftp:// URLs used
>> grep ftp /usr/sbin/sysupgrade
> $ grep -ne ftp -e URL -e MIRROR /usr/sbin/sysupgrade
> 102:0)  MIRROR=$(sed 's/#.*//;/^$/d' /etc/installurl) 2>/dev/null ||
> 103:MIRROR=https://cdn.openbsd.org/pub/OpenBSD
> 105:1)  MIRROR=$1
> 117:URL=${MIRROR}/snapshots/${ARCH}/
> 119:URL=${MIRROR}/${NEXT_VERSION}/${ARCH}/
> 136:unpriv -f SHA256.sig ftp -Vmo SHA256.sig ${URL}SHA256.sig
> 176:unpriv -f $f ftp -Vmo ${f} ${URL}${f}
>
> Your point?

I understand that I misread the question, sorry.


>>> is when fetching
>>> certain distfiles while building some things from ports (and they would
>>> usually fallback to http://ftp.openbsd.org/pub/OpenBSD/distfiles if
>>> the ftp fetch failed)..

Re: Disable ftp in pkg_add syspatch sysupgrade

[tom ryan]

On 2019-10-29 20:19, PJ wrote:
> Am 28.10.19 um 23:52 schrieb Stuart Henderson:
>> On 2019-10-28, Andy Lemin  wrote:
>>> Hi guys,
>>>
>>> Does anyone know if it is possible to completely disable ftp in the package 
>>> management utilities; pkg_add, syspatch, sysupgrade etc?
>>>
>>> My PKG_PATH references http:// urls, as does /etc/install. But I cannot 
>>> stop these tools trying to use ftp which does not work! :(
>> Can you show some example URLs, for example from "pgrep -lf ftp" while
>> trying to use one of these utilities?
>>
>> The only place I would expect to see ftp:// URLs used
> 
> 
> grep ftp /usr/sbin/sysupgrade

$ grep -ne ftp -e URL -e MIRROR /usr/sbin/sysupgrade
102:0)  MIRROR=$(sed 's/#.*//;/^$/d' /etc/installurl) 2>/dev/null ||
103:MIRROR=https://cdn.openbsd.org/pub/OpenBSD
105:1)  MIRROR=$1
117:URL=${MIRROR}/snapshots/${ARCH}/
119:URL=${MIRROR}/${NEXT_VERSION}/${ARCH}/
136:unpriv -f SHA256.sig ftp -Vmo SHA256.sig ${URL}SHA256.sig
176:unpriv -f $f ftp -Vmo ${f} ${URL}${f}

Your point?


> 
>> is when fetching
>> certain distfiles while building some things from ports (and they would
>> usually fallback to http://ftp.openbsd.org/pub/OpenBSD/distfiles if
>> the ftp fetch failed)..
>>
>>
>>
> 

Re: Disable ftp in pkg_add syspatch sysupgrade

[Stuart Henderson]

On 2019-10-29, PJ  wrote:
> Am 28.10.19 um 23:52 schrieb Stuart Henderson:
>> On 2019-10-28, Andy Lemin  wrote:
>>> Hi guys,
>>>
>>> Does anyone know if it is possible to completely disable ftp in the package 
>>> management utilities; pkg_add, syspatch, sysupgrade etc?
>>>
>>> My PKG_PATH references http:// urls, as does /etc/install. But I cannot 
>>> stop these tools trying to use ftp which does not work! :(
>> Can you show some example URLs, for example from "pgrep -lf ftp" while
>> trying to use one of these utilities?
>>
>> The only place I would expect to see ftp:// URLs used
>
>
> grep ftp /usr/sbin/sysupgrade

Yes I know how sysupgrade works. There is no reason why it would try
to connect via the FTP protocol unless either an ftp:// url is set in
/etc/installurl or an ftp:// url is specified on the command line.

ftp(1) is not restricted to just the FTP protocol, it is most often
used as an http/https fetcher these days.

Re: Disable ftp in pkg_add syspatch sysupgrade

[PJ]

Am 28.10.19 um 23:52 schrieb Stuart Henderson:
> On 2019-10-28, Andy Lemin  wrote:
>> Hi guys,
>>
>> Does anyone know if it is possible to completely disable ftp in the package 
>> management utilities; pkg_add, syspatch, sysupgrade etc?
>>
>> My PKG_PATH references http:// urls, as does /etc/install. But I cannot stop 
>> these tools trying to use ftp which does not work! :(
> Can you show some example URLs, for example from "pgrep -lf ftp" while
> trying to use one of these utilities?
>
> The only place I would expect to see ftp:// URLs used


grep ftp /usr/sbin/sysupgrade


> is when fetching
> certain distfiles while building some things from ports (and they would
> usually fallback to http://ftp.openbsd.org/pub/OpenBSD/distfiles if
> the ftp fetch failed)..
>
>
>

Re: Disable ftp in pkg_add syspatch sysupgrade

[Stuart Henderson]

On 2019-10-28, Andy Lemin  wrote:
> Hi guys,
>
> Does anyone know if it is possible to completely disable ftp in the package 
> management utilities; pkg_add, syspatch, sysupgrade etc?
>
> My PKG_PATH references http:// urls, as does /etc/install. But I cannot stop 
> these tools trying to use ftp which does not work! :(

Can you show some example URLs, for example from "pgrep -lf ftp" while
trying to use one of these utilities?

The only place I would expect to see ftp:// URLs used is when fetching
certain distfiles while building some things from ports (and they would
usually fallback to http://ftp.openbsd.org/pub/OpenBSD/distfiles if
the ftp fetch failed)..

Re: Disable ftp in pkg_add syspatch sysupgrade

[Raf Czlonka]

On Mon, Oct 28, 2019 at 06:43:03PM GMT, Andy Lemin wrote:
> Hi guys,
> 
> Does anyone know if it is possible to completely disable ftp in the package 
> management utilities; pkg_add, syspatch, sysupgrade etc?
> 
> My PKG_PATH references http:// urls, as does /etc/install. But I cannot stop 
> these tools trying to use ftp which does not work! :(
> 
> Every time I try and install a bunch of packages, some work fine via http, 
> and then some will fall back to ftp, which of course fails. 
> 
> However when I test the url for the package it reports it could not download, 
> due to the ftp connect timeout, the download works fine via http etc. So it 
> is very frustrating.
> 
> We cannot use ftpproxy as we use rdomains, and running the ftpproxy daemon 
> within the rdomain that has the default route to the internet doesn’t seem to 
> work. And generally speaking no one in my last two companies has ever asked 
> me to make ftp work.
> 
> As ftp is very much dead in most cases, could we please stop using it for 
> package management tools? :( I cannot update any of my OpenBSD servers which 
> reside inside the firewall. Only the firewall itself get et updates because 
> of the ftp dependence :(
> 
> Thanks for your time.
> Andy.
> 
> 
> Sent from a teeny tiny keyboard, so please excuse typos
> 

Hi Andy,

If, like you say, PKG_PATH and installurl(5) both reference HTTP(S),
then you're getting confused - warning/error messages come from ftp(1).
It has nothing to do with FTP (the protocol).

Regards,

Raf

Disable ftp in pkg_add syspatch sysupgrade

[Andy Lemin]

Hi guys,

Does anyone know if it is possible to completely disable ftp in the package 
management utilities; pkg_add, syspatch, sysupgrade etc?

My PKG_PATH references http:// urls, as does /etc/install. But I cannot stop 
these tools trying to use ftp which does not work! :(

Every time I try and install a bunch of packages, some work fine via http, and 
then some will fall back to ftp, which of course fails. 

However when I test the url for the package it reports it could not download, 
due to the ftp connect timeout, the download works fine via http etc. So it is 
very frustrating.

We cannot use ftpproxy as we use rdomains, and running the ftpproxy daemon 
within the rdomain that has the default route to the internet doesn’t seem to 
work. And generally speaking no one in my last two companies has ever asked me 
to make ftp work.

As ftp is very much dead in most cases, could we please stop using it for 
package management tools? :( I cannot update any of my OpenBSD servers which 
reside inside the firewall. Only the firewall itself get et updates because of 
the ftp dependence :(

Thanks for your time.
Andy.


Sent from a teeny tiny keyboard, so please excuse typos