Re: Why does the ports system delete distfiles?
On Mon, May 07, 2012 at 12:02:55AM -0400, Alan Corey wrote: Yeah, at one point when I was getting 5.0 set up I actually took my laptop somewhere with a WiFi connection. I must have already installed databases/sqlports because I extracted a list of distfiles from that, matched it against what I already had from 4.7, then put the site url on the beginning of each line and .tar.gz on the end and fed it to wget. I got about 100 megs of distfiles in about 15 minutes and saved about 10 hours of downloading by modem. Of course they aren't all tar.gz and the site didn't work for all of them, but it helped. I'm still working on parts of LibreOffice. Next time just use dpb -F and save you a lot of pain and manual fucking around.
Re: Why does the ports system delete distfiles?
I just saw another good reason to hit ctrl-C. I'm on a modem, and I just hit boost_1_42_0.tar.gz in an install. That's 40 megs, more than I can download in a day. I need to use a different process, like put the url in a text file and feed it to wget with --continue -i file So I hit ctrl-C, and that got me out of downloading from the first site, only to start downloading from another, then another. What I really wanted was to be able to copy a working url and paste it into a text file. Alan On Wed, 25 Apr 2012, Alan Corey wrote: I've seen this before, I wonder if there's some environment variable I can set to stop it? I try make fetch on a port, it fails due to a bad site. I hit Ctrl-C to stop it, it goes to the next site and downloads the file. Then it deletes the file when it finishes. I type make install and it tries the bad site again... Alan
Re: Why does the ports system delete distfiles?
On Sunday 06 May 2012 18:24:21 Alan Corey wrote: I just saw another good reason to hit ctrl-C. I'm on a modem, and I just hit boost_1_42_0.tar.gz in an install. That's 40 megs, more than I can download in a day. I need to use a different process, like put the url in a text file and feed it to wget with --continue -i file So I hit ctrl-C, and that got me out of downloading from the first site, only to start downloading from another, then another. What I really wanted was to be able to copy a working url and paste it into a text file. Alan On Wed, 25 Apr 2012, Alan Corey wrote: I've seen this before, I wonder if there's some environment variable I can set to stop it? I try make fetch on a port, it fails due to a bad site. I hit Ctrl-C to stop it, it goes to the next site and downloads the file. Then it deletes the file when it finishes. I type make install and it tries the bad site again... Alan You are able to get a working url. All info for that url is in the Makefile of the port. You could cut and paste an url or do some shell scripting to extract the url's. gr Renzo
Re: Why does the ports system delete distfiles?
On 2012-05-06, Renzo Fabriek rfabr...@nerdshack.com wrote: On Sunday 06 May 2012 18:24:21 Alan Corey wrote: I just saw another good reason to hit ctrl-C. I'm on a modem, and I just hit boost_1_42_0.tar.gz in an install. That's 40 megs, more than I can download in a day. I need to use a different process, like put the url in a text file and feed it to wget with --continue -i file So I hit ctrl-C, and that got me out of downloading from the first site, only to start downloading from another, then another. What I really wanted was to be able to copy a working url and paste it into a text file. Alan On Wed, 25 Apr 2012, Alan Corey wrote: I've seen this before, I wonder if there's some environment variable I can set to stop it? I try make fetch on a port, it fails due to a bad site. I hit Ctrl-C to stop it, it goes to the next site and downloads the file. Then it deletes the file when it finishes. I type make install and it tries the bad site again... Alan You are able to get a working url. All info for that url is in the Makefile of the port. You could cut and paste an url or do some shell scripting to extract the url's. gr Renzo It's a bit awkward actually but can be done. You have to take output of 'make show=DISTFILES' and prepend contents of 'make show=MASTER_SITES' (which can give you multiple URLs), except where the distfile ends in :0, :1, :2, ... :9 in which case you need MASTER_SITES0, ...1, etc. Similar for PATCHFILES and maybe SUPDISTFILES.. You can't do it reliably without 'make show=...' because the URL can come from a *.port.nk modules, Makefile.inc or network.conf. And sometimes you need to fallback a backup at an openbsd.org server.
Re: Why does the ports system delete distfiles?
On Sun, 6 May 2012, Stuart Henderson wrote: On 2012-05-06, Renzo Fabriek rfabr...@nerdshack.com wrote: On Sunday 06 May 2012 18:24:21 Alan Corey wrote: I just saw another good reason to hit ctrl-C. I'm on a modem, and I just hit boost_1_42_0.tar.gz in an install. That's 40 megs, more than I can download in a day. I need to use a different process, like put the url in a text file and feed it to wget with --continue -i file So I hit ctrl-C, and that got me out of downloading from the first site, only to start downloading from another, then another. What I really wanted was to be able to copy a working url and paste it into a text file. Alan On Wed, 25 Apr 2012, Alan Corey wrote: I've seen this before, I wonder if there's some environment variable I can set to stop it? I try make fetch on a port, it fails due to a bad site. I hit Ctrl-C to stop it, it goes to the next site and downloads the file. Then it deletes the file when it finishes. I type make install and it tries the bad site again... Alan You are able to get a working url. All info for that url is in the Makefile of the port. You could cut and paste an url or do some shell scripting to extract the url's. gr Renzo It's a bit awkward actually but can be done. You have to take output of 'make show=DISTFILES' and prepend contents of 'make show=MASTER_SITES' (which can give you multiple URLs), except where the distfile ends in :0, :1, :2, ... :9 in which case you need MASTER_SITES0, ...1, etc. Similar for PATCHFILES and maybe SUPDISTFILES.. You can't do it reliably without 'make show=...' because the URL can come from a *.port.nk modules, Makefile.inc or network.conf. And sometimes you need to fallback a backup at an openbsd.org server. Yeah, at one point when I was getting 5.0 set up I actually took my laptop somewhere with a WiFi connection. I must have already installed databases/sqlports because I extracted a list of distfiles from that, matched it against what I already had from 4.7, then put the site url on the beginning of each line and .tar.gz on the end and fed it to wget. I got about 100 megs of distfiles in about 15 minutes and saved about 10 hours of downloading by modem. Of course they aren't all tar.gz and the site didn't work for all of them, but it helped. I'm still working on parts of LibreOffice. Alan
Re: Why does the ports system delete distfiles?
On 2012-04-26, Alan Corey ab...@devio.us wrote: This site in question is for net/trafshow, the nsk.su site: The server refuses login. I've mirrored this, port in -current has the new url, you can fetch it from http://spacehopper.org/mirrors/trafshow-3.1.tgz Back to manually fetching I guess. post 5.0, but in -current or 5.1, you can do PATH=$PATH:/usr/ports/infrastructure/bin dpb -F1 editors/libreoffice Or if you're having a problem with some sites you could try this in mk.conf MASTER_SITE_OVERRIDE=Yes and maybe also set MASTER_SITE_BACKUP to one or more of http://ftp.openbsd.org/pub/OpenBSD/distfiles/ http://ftp.usa.openbsd.org/pub/OpenBSD/distfiles/ http://ftp.fr.openbsd.org/pub/OpenBSD/distfiles/
Re: Why does the ports system delete distfiles?
On Wed, Apr 25, 2012 at 10:48 PM, Jan Stary h...@stare.cz wrote: On Apr 25 23:34:24, Alan Corey wrote: I've seen this before, I wonder if there's some environment variable I can set to stop it? I try make fetch on a port, it fails due to a bad site. I hit Ctrl-C to stop it, it goes to the next site and downloads the file. Then it deletes the file when it finishes. I can confirm this happens; for example, audio/sox: as pointed out by Alexander Hall it is make doing this because before the target is build, the process is aborted: $ cat touch_tmp_testdottxt.sh #!/bin/sh set -x touch /tmp/test.txt # sleep 5 second allow for ^C echo sleeping for 5 second. go ahead and control-C out of make sleep 5 echo done! $ cat Makefile test.txt: /bin/sh touch_tmp_testdottxt.sh $ make -n /bin/sh touch_tmp_testdottxt.sh $ make /bin/sh touch_tmp_testdottxt.sh + touch /tmp/test.txt + echo sleeping for 5 second. go ahead and control-C out of make sleeping for 5 second. go ahead and control-C out of make + sleep 5 ^C*** test.txt removed HTH, --patrick # make fetch === Checking files for sox-14.4.0p1 Fetch http://downloads.sourceforge.net/sourceforge/sox/sox-14.4.0.tar.gz ^C http fetch aborted. Fetch http://easynews.dl.sourceforge.net/sourceforge/sox/sox-14.4.0.tar.gz ^C Fetch http://puzzle.dl.sourceforge.net/sourceforge/sox/sox-14.4.0.tar.gz ^C Fetch http://optusnet.dl.sourceforge.net/sourceforge/sox/sox-14.4.0.tar.gz ftp: Error retrieving file: 404 Not Found Fetch http://heanet.dl.sourceforge.net/sourceforge/sox/sox-14.4.0.tar.gz ^C Fetch http://jaist.dl.sourceforge.net/sourceforge/sox/sox-14.4.0.tar.gz ^C http fetch aborted. Fetch http://nchc.dl.sourceforge.net/sourceforge/sox/sox-14.4.0.tar.gz ^C Fetch http://switch.dl.sourceforge.net/sourceforge/sox/sox-14.4.0.tar.gz ^C Fetch http://kent.dl.sourceforge.net/sourceforge/sox/sox-14.4.0.tar.gz ^C Fetch http://internap.dl.sourceforge.net/sourceforge/sox/sox-14.4.0.tar.gz ftp: no address associated with name: internap.dl.sourceforge.net Fetch http://mesh.dl.sourceforge.net/sourceforge/sox/sox-14.4.0.tar.gz ^C Fetch http://ovh.dl.sourceforge.net/sourceforge/sox/sox-14.4.0.tar.gz ^C Fetch http://surfnet.dl.sourceforge.net/sourceforge/sox/sox-14.4.0.tar.gz ^C Fetch http://ufpr.dl.sourceforge.net/sourceforge/sox/sox-14.4.0.tar.gz ^C Fetch http://ftp.openbsd.org/pub/OpenBSD/distfiles//sox-14.4.0.tar.gz sox-14.4.0.tar.gz 100% || 1079 KB 00:03 *** /usr/ports/distfiles/sox-14.4.0.tar.gz removed
Re: Why does the ports system delete distfiles?
On Wed, Apr 25, 2012 at 11:34:24PM -0400, Alan Corey wrote: I've seen this before, I wonder if there's some environment variable I can set to stop it? Nope. I try make fetch on a port, it fails due to a bad site. I hit Ctrl-C to stop it, it goes to the next site and downloads the file. Then it deletes the file when it finishes. I type make install and it tries the bad site again... That's the way make works. Don't hit ^C. Changing this is impossible, since make sees the ^C, being the controlling process and all. Oh, and if the site is really bad, report the site. If it's not, fix your network config. I hardly notice anymore since dpb fetches things for me.
Re: Why does the ports system delete distfiles?
On Thu, Apr 26, 2012 at 07:33, Alexander Hall wrote: Alan Corey ab...@devio.us wrote: I've seen this before, I wonder if there's some environment variable I can set to stop it? I try make fetch on a port, it fails due to a bad site. I hit Ctrl-C to stop it, it goes to the next site and downloads the file. Then it deletes the file when it finishes. I type make install and it tries the bad site again... Alan I'd guess one or more subprocesses ignore SIGINT while others (=make?) don't, and thus the fetching proceeds but when it's done, make exits, after the appropriate cleanup. Don't know if it's trivially fixed. ^Z, pkill ftp, fg...
Re: Why does the ports system delete distfiles?
On Thu, 26 Apr 2012, Marc Espie wrote: On Wed, Apr 25, 2012 at 11:34:24PM -0400, Alan Corey wrote: I've seen this before, I wonder if there's some environment variable I can set to stop it? Nope. I try make fetch on a port, it fails due to a bad site. I hit Ctrl-C to stop it, it goes to the next site and downloads the file. Then it deletes the file when it finishes. I type make install and it tries the bad site again... That's the way make works. Don't hit ^C. Changing this is impossible, since make sees the ^C, being the controlling process and all. Oh, and if the site is really bad, report the site. If it's not, fix your network config. I hardly notice anymore since dpb fetches things for me. It doesn't have to be just ^C that stops it, if my ISP disconnects me or the connection times out, anything that can go wrong causes it to happen. I'd like to be able to resume with wget --continue but there's nothing to continue. In fact I have FETCH_CMD defined as 'wget --continue' but it never has a chance to work. I can spend an hour downloading then lose it all. This site in question is for net/trafshow, the nsk.su site: Resolving ftp.nsk.su (ftp.nsk.su)... 212.20.0.102 Connecting to ftp.nsk.su (ftp.nsk.su)|212.20.0.102|:21... connected. Logging in as anonymous ... The server refuses login. Retrying. It could be temporary (too many users at the moment, etc.) but I got it 2 days in a row. Back to manually fetching I guess. FreeBSD has something like make fetchlist recursive that gives paths to the distfile and dependancies which is a big help. I'm still working on fetching all the parts to libreoffice, for at least 2 weeks now. The default site in Hungary times out on modem slowness so I search with Google for each file and find them buried at fedoraproject in directory names like 5eb8e1950f91198ae1e7ddc6c1dd19fd. Alan
Re: Why does the ports system delete distfiles?
On Thu, Apr 26, 2012 at 11:09:38AM -0400, Ted Unangst wrote: Alan I'd guess one or more subprocesses ignore SIGINT while others (=make?) don't, and thus the fetching proceeds but when it's done, make exits, after the appropriate cleanup. Don't know if it's trivially fixed. ^Z, pkill ftp, fg... Or seriously, use dpb instead. The fetch code was specifically written to alleviate all of normal make fetch problems, including resuming partial downloads and doing somewhat smart things wrt checksums. Like I said, I don't even notice the ftp issues any more.
Re: Why does the ports system delete distfiles?
On 04/26/12 08:31, Marc Espie wrote:
On Wed, Apr 25, 2012 at 11:34:24PM -0400, Alan Corey wrote:
I've seen this before, I wonder if there's some environment variable
I can set to stop it?
Nope.
I try make fetch on a port, it fails due to a bad site. I hit
Ctrl-C to stop it, it goes to the next site and downloads the file.
Then it deletes the file when it finishes. I type make install and
it tries the bad site again...
That's the way make works. Don't hit ^C.
Changing this is impossible, since make sees the ^C, being the controlling
process and all.
Hmmm, but still, if the process is interrupted, it should not continue
processing the rest of them, if the result is to be discarded anyway.
I believe this comes from a few places where we trap a few signals,
including SIGINT, to perform various cleanup operations. Those should,
apart from the cleanup, also exit the process, like the default SIGINT
handler would.
In the diff below, I put the cleanup in the EXIT signal handler,
while the rest of the trapped signals merely do 'exit 1', triggering
the EXIT handler, but exiting the process with a non-zero exit code.
Fixes (as in immediately exists, and performs the appropriate cleanup)
the OP's example. Also contains the same change to all the traps in the
ports makefiles, but I only tested the OP's case, as I'm not sure where
the other cases are used.
OK?
/Alexander
Index: bsd.port.mk
===
RCS file: /cvs/ports/infrastructure/mk/bsd.port.mk,v
retrieving revision 1.1164
diff -u -p -r1.1164 bsd.port.mk
--- bsd.port.mk 22 Apr 2012 10:39:48 - 1.1164
+++ bsd.port.mk 26 Apr 2012 22:28:00 -
@@ -1528,7 +1528,9 @@ _DO_LOCK = \
. endfor
_SIMPLE_LOCK = \
- ${_LOCK}; locked=true; trap 'if $$locked; then ${_UNLOCK};
locked=false; fi' 0 1 2 3 13 15
+ ${_LOCK}; locked=true; \
+ trap 'if $$locked; then ${_UNLOCK}; locked=false; fi' 0; \
+ trap 'exit 1' 1 2 3 13 15
.endif
_SIMPLE_LOCK ?= :
@@ -2809,13 +2811,15 @@ describe:
readme:
@tmpdir=`mktemp -d ${TMPDIR}/readme.XX`; \
- trap rm -r $$tmpdir 0 1 2 3 13 15; \
+ trap rm -r $$tmpdir 0; \
+ trap 'exit 1' 1 2 3 13 15; \
cd ${.CURDIR} PKGPATH=${PKGPATH} ${MAKE} TMPDIR=$$tmpdir
README_NAME=${README_NAME} \
${READMES_TOP}/${PKGPATH}/${FULLPKGNAME${SUBPACKAGE}}.html
readmes:
@tmpdir=`mktemp -d ${TMPDIR}/readme.XX`; \
- trap rm -r $$tmpdir 0 1 2 3 13 15; \
+ trap rm -r $$tmpdir 0; \
+ trap 'exit 1' 1 2 3 13 15; \
cd ${.CURDIR} PKGPATH=${PKGPATH} ${MAKE} TMPDIR=$$tmpdir
README_NAME=${README_NAME} \
${_READMES}
Index: bsd.port.subdir.mk
===
RCS file: /cvs/ports/infrastructure/mk/bsd.port.subdir.mk,v
retrieving revision 1.106
diff -u -p -r1.106 bsd.port.subdir.mk
--- bsd.port.subdir.mk 17 Feb 2012 07:41:22 - 1.106
+++ bsd.port.subdir.mk 26 Apr 2012 22:28:00 -
@@ -193,7 +193,8 @@ clean:
readmes:
@${_subdir_fragment}
@tmpdir=`mktemp -d ${TMPDIR}/readme.XX`; \
- trap 'rm -r $$tmpdir' 0 1 2 3 13 15; \
+ trap 'rm -r $$tmpdir' 0; \
+ trap 'exit 1' 1 2 3 13 15; \
cd ${.CURDIR} ${MAKE} TMPDIR=$$tmpdir \
${READMES_TOP}/${PKGPATH}/README.html
Index: pkgpath.mk
===
RCS file: /cvs/ports/infrastructure/mk/pkgpath.mk,v
retrieving revision 1.44
diff -u -p -r1.44 pkgpath.mk
--- pkgpath.mk 17 Feb 2012 07:40:35 - 1.44
+++ pkgpath.mk 26 Apr 2012 22:28:00 -
@@ -95,7 +95,8 @@ _depfile_fragment = \
case X$${_DEPENDS_FILE} in \
X) _DEPENDS_FILE=`mktemp ${TMPDIR}/depends.X|| exit 1`;
\
export _DEPENDS_FILE; \
- trap rm -f $${_DEPENDS_FILE} 0 1 2 3 13 15;; \
+ trap rm -f $${_DEPENDS_FILE} 0; \
+ trap 'exit 1' 1 2 3 13 15;; \
esac
# the cache may be filled in as root, so try to remove as normal user, THEN
@@ -104,7 +105,8 @@ _cache_fragment = \
case X$${_DEPENDS_CACHE} in \
X) _DEPENDS_CACHE=`mktemp -d ${TMPDIR}/dep_cache.X||
exit 1`; \
export _DEPENDS_CACHE; \
- trap rm -rf 2/dev/null $${_DEPENDS_CACHE} || ${SUDO} rm -rf
$${_DEPENDS_CACHE} 0 1 2 3 13 15;; \
+ trap rm -rf 2/dev/null $${_DEPENDS_CACHE} || ${SUDO} rm -rf
$${_DEPENDS_CACHE} 0; \
+ trap 'exit 1' 1 2 3 13 15;; \
esac; PKGPATH=${PKGPATH}; export PKGPATH
HTMLIFY = sed -e 's//\amp;/g' -e 's//\gt;/g' -e 's//\lt;/g'
Re: Why does the ports system delete distfiles?
Alan Corey ab...@devio.us wrote: I've seen this before, I wonder if there's some environment variable I can set to stop it? I try make fetch on a port, it fails due to a bad site. I hit Ctrl-C to stop it, it goes to the next site and downloads the file. Then it deletes the file when it finishes. I type make install and it tries the bad site again... Alan I'd guess one or more subprocesses ignore SIGINT while others (=make?) don't, and thus the fetching proceeds but when it's done, make exits, after the appropriate cleanup. Don't know if it's trivially fixed. /Alexander
Re: Why does the ports system delete distfiles?
On Apr 25 23:34:24, Alan Corey wrote: I've seen this before, I wonder if there's some environment variable I can set to stop it? I try make fetch on a port, it fails due to a bad site. I hit Ctrl-C to stop it, it goes to the next site and downloads the file. Then it deletes the file when it finishes. I can confirm this happens; for example, audio/sox: # make fetch === Checking files for sox-14.4.0p1 Fetch http://downloads.sourceforge.net/sourceforge/sox/sox-14.4.0.tar.gz ^C http fetch aborted. Fetch http://easynews.dl.sourceforge.net/sourceforge/sox/sox-14.4.0.tar.gz ^C Fetch http://puzzle.dl.sourceforge.net/sourceforge/sox/sox-14.4.0.tar.gz ^C Fetch http://optusnet.dl.sourceforge.net/sourceforge/sox/sox-14.4.0.tar.gz ftp: Error retrieving file: 404 Not Found Fetch http://heanet.dl.sourceforge.net/sourceforge/sox/sox-14.4.0.tar.gz ^C Fetch http://jaist.dl.sourceforge.net/sourceforge/sox/sox-14.4.0.tar.gz ^C http fetch aborted. Fetch http://nchc.dl.sourceforge.net/sourceforge/sox/sox-14.4.0.tar.gz ^C Fetch http://switch.dl.sourceforge.net/sourceforge/sox/sox-14.4.0.tar.gz ^C Fetch http://kent.dl.sourceforge.net/sourceforge/sox/sox-14.4.0.tar.gz ^C Fetch http://internap.dl.sourceforge.net/sourceforge/sox/sox-14.4.0.tar.gz ftp: no address associated with name: internap.dl.sourceforge.net Fetch http://mesh.dl.sourceforge.net/sourceforge/sox/sox-14.4.0.tar.gz ^C Fetch http://ovh.dl.sourceforge.net/sourceforge/sox/sox-14.4.0.tar.gz ^C Fetch http://surfnet.dl.sourceforge.net/sourceforge/sox/sox-14.4.0.tar.gz ^C Fetch http://ufpr.dl.sourceforge.net/sourceforge/sox/sox-14.4.0.tar.gz ^C Fetch http://ftp.openbsd.org/pub/OpenBSD/distfiles//sox-14.4.0.tar.gz sox-14.4.0.tar.gz 100% || 1079 KB 00:03 *** /usr/ports/distfiles/sox-14.4.0.tar.gz removed
