Re: Package Building in the Large
Doug Barton wrote: Jason C. Wells wrote: Doug Barton wrote: On Mon, 19 Nov 2007, Jason C. Wells wrote: What I am trying to do is to build 30 or so packages including the big ones like X, kde, gnome, plus all of their dependencies on a build host and then use pkg_add on various machines. I have had a variety of difficulties with all of the methods I have used thus far (portmaster, portupgrade, homegrown). What problems did you have with portmaster? Did the backup package creation fail in some way? Not all dependencies had a package built for them. For my list of 31 ports that I actually desired to build there was a dependency list (make all-depends-list) of 758 ports. Of those 758 ports there were 427 packages built. That's disturbing, but I think I know why it happened, see below. I'm more disturbed that this piece of news isn't common knowledge. Those numbers actually understate the problem. Just one commonly required port, one of the browsers like Firefox, alone brings in over 300 dependencies. At least in my own opinion, the largest part of that dependency list is VERY weakly required, mainly a matter of a porter saying to himself I have that port, I like it a lot, everybody should have it and not this port won't run without that port That's my own main motivation behind all that work I'm doing aboout making a ports keyword list, so as to better control the growth of dependency lists. It's no problem at all to show ludicrous examples of overly agressive dependency lists taking the choices of what ports to add out of the hands of the users. As soon as I get the keyword list written (asnd who knows, maybe accepted?), then I intend to push what I see as the second part of this, a tool that looks at what ports are installed, the state of your keyword lists, and a user's personal interests, and make suggestions of what ports a user might find interesting. Sort of a ports-advertiser. This would take the place of overly agressive dependency lists, but not by removing the user from the process, but instead by making that user's selection job easier to make. Such a tool could have a link to a ports installer, even, so as to further ease things, but not to remove the choice from the user, as it's moving towards today. ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Package Building in the Large
Chuck Robey wrote: Doug Barton wrote: Jason C. Wells wrote: Doug Barton wrote: On Mon, 19 Nov 2007, Jason C. Wells wrote: What I am trying to do is to build 30 or so packages including the big ones like X, kde, gnome, plus all of their dependencies on a build host and then use pkg_add on various machines. I have had a variety of difficulties with all of the methods I have used thus far (portmaster, portupgrade, homegrown). What problems did you have with portmaster? Did the backup package creation fail in some way? Not all dependencies had a package built for them. For my list of 31 ports that I actually desired to build there was a dependency list (make all-depends-list) of 758 ports. Of those 758 ports there were 427 packages built. That's disturbing, but I think I know why it happened, see below. I'm more disturbed that this piece of news isn't common knowledge. Those numbers actually understate the problem. Just one commonly required port, one of the browsers like Firefox, alone brings in over 300 dependencies. xorg itself brings in 262 dependencies. It is because metaports like xorg and gnome2 have been split into smaller and smaller ports. Thus far, I have never installed a port that I felt brought in unwanted dependencies. ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Package Building in the Large
Doug Barton wrote:
On Mon, 19 Nov 2007, Jason C. Wells wrote:
What I am trying to do is to build 30 or so packages including the big
ones like X, kde, gnome, plus all of their dependencies on a build
host and then use pkg_add on various machines. I have had a variety
of difficulties with all of the methods I have used thus far
(portmaster, portupgrade, homegrown).
What I do is this. After building the ports I want (and I admit that I
use vanilla cd the-port; make install clean, I run a little script I
call make-packages. And after I have updated the ports, I run a
program called make-packages-purge to get rid of the old stuff.
I enclose them as attachments.
#!/usr/bin/make -f
PACKAGES?=/usr/home/stephen/packages-7
PKG_DBDIR?=/var/db/pkg
PORTSDIR?=/usr/ports
PKG_LIST!=ls ${PKG_DBDIR}
all:${PKG_LIST:C+(.*)+${PACKAGES}/All/\1.tbz+}
.for target in ${PKG_LIST}
${PACKAGES}/All/${target}.tbz: ${PKG_DBDIR}/${target}/+CONTENTS
@origin=${PORTSDIR}/`sed -n 's/@comment ORIGIN://p' \
${PKG_DBDIR}/${target}/+CONTENTS`; \
if [ $$origin != ${PORTSDIR}/ ] [ -e $$origin ] (! [ -e
$$origin/work ] || [ -e $$origin/work/.install_done* ]); then \
echo creating package ${target}; \
make -C $$origin PACKAGES=${PACKAGES} package-links; \
pkg_create -GjY -b ${target} $@ || (rm -f $@ false); \
fi
.endfor
.BEGIN:
@mkdir -p ${PACKAGES}/All
#!/bin/sh
PACKAGES=/usr/home/stephen/packages-7
DB=/var/db/pkg
for p in `ls $PACKAGES/All | sed s/\.tbz//`; do
if ! [ -e ${DB}/$p ]; then
rm -i $PACKAGES/All/$p.tbz
fi
done
for l in `find $PACKAGES -type l`; do
if ! stat -q -L $l /dev/null; then
rm -f $l
fi
done
___
freebsd-ports@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Package Building in the Large
On Mon, 19 Nov 2007, Jason C. Wells wrote: What I am trying to do is to build 30 or so packages including the big ones like X, kde, gnome, plus all of their dependencies on a build host and then use pkg_add on various machines. I have had a variety of difficulties with all of the methods I have used thus far (portmaster, portupgrade, homegrown). What problems did you have with portmaster? Did the backup package creation fail in some way? Doug -- This .signature sanitized for your protection ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Package Building in the Large
Doug Barton wrote: On Mon, 19 Nov 2007, Jason C. Wells wrote: What I am trying to do is to build 30 or so packages including the big ones like X, kde, gnome, plus all of their dependencies on a build host and then use pkg_add on various machines. I have had a variety of difficulties with all of the methods I have used thus far (portmaster, portupgrade, homegrown). What problems did you have with portmaster? Did the backup package creation fail in some way? Not all dependencies had a package built for them. For my list of 31 ports that I actually desired to build there was a dependency list (make all-depends-list) of 758 ports. Of those 758 ports there were 427 packages built. I also ended up with shared a library version problem in at least one port (grip) in spite of having started my build with a completely vacant /usr/local. It seems that portmaster is intended to be run on the host where the existing ports are currently installed and where the new ports will be eventually installed. It looks to have good port upgrading abilities, but that is not what I am after. What I am trying to do is operate a build host and distribute packages from there. Not all hosts run the same set of packages. Add the fact that I am finicky about customizing the kerberos dependencies. I am trying to find a good method to build my ports at every minor release (7.0 upcoming) with little user intervention. Perhaps I misunderstand the -g and -b switches. I don't want backup packages of old ports. I already have those. I do want packages for all new ports that are built during a run. I used 'portmaster -GgDt -p /usr/ports/ports $pmarg' for my run where $pmarg was the list of 31 ports. I think portmaster worked as expected. It just didn't do what I desired. Hence my original question to the list. The tinderbox port looks like the right functionality. It also looks heavyweight requiring apache+mysql. I am trying to avoid dealing with extra databases. I spent a lot of time messing around with the database under portupgrade. I have come to the opinion (like the idea behind portmaster) that there are already databases built into the port system. I'd rather just use them in place. I would also like the portmanagers to know that I think they do a bang-up job. I just have my ultra narrow fussy roll your own way of doing things. I am looking for the right method _for me_. All of the above is no criticism. Thanks, Jason C. Wells ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Package Building in the Large
Jason C. Wells wrote: Doug Barton wrote: On Mon, 19 Nov 2007, Jason C. Wells wrote: What I am trying to do is to build 30 or so packages including the big ones like X, kde, gnome, plus all of their dependencies on a build host and then use pkg_add on various machines. I have had a variety of difficulties with all of the methods I have used thus far (portmaster, portupgrade, homegrown). What problems did you have with portmaster? Did the backup package creation fail in some way? Not all dependencies had a package built for them. For my list of 31 ports that I actually desired to build there was a dependency list (make all-depends-list) of 758 ports. Of those 758 ports there were 427 packages built. That's disturbing, but I think I know why it happened, see below. I also ended up with shared a library version problem in at least one port (grip) in spite of having started my build with a completely vacant /usr/local. That sounds like a problem with the ports. I can't think of any way that portmaster would cause that. It seems that portmaster is intended to be run on the host where the existing ports are currently installed and where the new ports will be eventually installed. That is its primary purpose, yes. It looks to have good port upgrading abilities, but that is not what I am after. What I am trying to do is operate a build host and distribute packages from there. That is not its primary purpose, but assuming that your systems are similarly configured, and running an OS version that's fairly close in age, there is no reason that portmaster shouldn't be able to at least create the packages that you can then distribute manually. Not all hosts run the same set of packages. That shouldn't matter as long as different hosts aren't running different versions of the same package. Perhaps I misunderstand the -g and -b switches. I don't want backup packages of old ports. That's what -b does. I do want packages for all new ports that are built during a run. That is what -g does. I used 'portmaster -GgDt -p /usr/ports/ports $pmarg' for my run where $pmarg was the list of 31 ports. That's definitely not going to work, for several reasons. The -G switch should only be used _after_ you've already run portmaster without it for a given port build. The OPTIONS are there for a reason, and when they are changed it can (and often does) change the dependency tree. Also, since the OPTIONS framework has been fixed, if the options in the Makefile haven't changed, portmaster won't prompt you to reset options you've already set, so you get all the benefits with very little of the cost. This flag should be used primarily when you have to restart a build of a given port because the first build failed. If you're starting with a clean /usr/local then the -t option is not needed, but it probably won't hurt anything. You do misunderstand what the -p switch is for, and specifying the list of ports in the manner you did is probably not the best way to go about it. I'm also unsure what /usr/ports/ports is supposed to be, unless that's a local path issue. Assuming that your list of ports is relative to the /usr/ports directory (e.g., 'archivers/unzip astro/planets', etc.), what I would do is: for dir in $pmarg; do portmaster -BgD -p /usr/ports/$dir done Replace the literal /usr/ports/ in that string with the actual location of your ports tree if its different. You probably also want to add the -v switch in there for the first few runs to familiarize yourself with what's going on, and review the man page. I think portmaster worked as expected. If you invoked it the way you described, I'm pretty sure it didn't do what you expected it to do. :) It just didn't do what I desired. Hence my original question to the list. Please note, I am not saying that portmaster will definitely be the right solution for you. However, I do think it's worth pointing out that if used properly it should at least be able to do what it says it can do. The tinderbox port looks like the right functionality. It also looks heavyweight requiring apache+mysql. I am trying to avoid dealing with extra databases. I spent a lot of time messing around with the database under portupgrade. I have come to the opinion (like the idea behind portmaster) that there are already databases built into the port system. I'd rather just use them in place. I would also like the portmanagers to know that I think they do a bang-up job. I just have my ultra narrow fussy roll your own way of doing things. I am looking for the right method _for me_. All of the above is no criticism. I certainly don't take it that way, and I appreciate both your kind words and the amount of thought that you put into this. Best of luck to you. Doug -- This .signature sanitized for your protection ___ freebsd-ports@freebsd.org mailing list
Re: Package Building in the Large
Jason C. Wells wrote: Most of the message below is just rounding out the discussion. There is one significant question on recursion though. Sounds good, I'll snip the bits that don't require comment. Doug Barton wrote: I also ended up with shared a library version problem in at least one port (grip) in spite of having started my build with a completely vacant /usr/local. That sounds like a problem with the ports. I can't think of any way that portmaster would cause that. It might be due to a non-clean /usr/local on the target install host. It was a foo.so.3 is required but foo.so.2 is installed sort of error. I haven't dealt with the installation step in a meaningful way yet. Don't worry about responding to this bit. I think you're on the right track there. I used 'portmaster -GgDt -p /usr/ports/ports $pmarg' for my run where $pmarg was the list of 31 ports. That's definitely not going to work, for several reasons. The -G switch should only be used _after_ you've already run portmaster without it for a given port build. I had manually run 'make config-recursive' previously. I'll do as advised at any rate. The config-recursive target does most of what portmaster does, but not all. Most significantly, it's not adaptive to changing an OPTION that results in a new dependency graph. What probably happened (and this is just a WAG) is that because portmaster didn't see some of the dependencies further down the tree they were installed by the ports infrastructure, so portmaster made no packages for them. In version 1.24 I added the -DNO_DEPENDS flag to 'make install' so that this problem can't happen. I'm also unsure what /usr/ports/ports is supposed to be, unless that's a local path issue. It's a local hierarchy issue. I pushed the ports tree up one level and therein lie 6.2, -current, and the soon to be 7.0 trees. Ok, you might want to take a look at the PORTSDIR variable in ports/Mk/bsd.port.mk, and set that in your environment accordingly. Assuming that your list of ports is relative to the /usr/ports directory (e.g., 'archivers/unzip astro/planets', etc.), what I would do is: for dir in $pmarg; do portmaster -BgD -p /usr/ports/$dir done Aha! I had an error (don't recall exactly) without using -p where the complaint had something to do with not finding something in /usr/ports. Yeah, that makes sense. If you set PORTSDIR in your environment you should be able to just specify the list of ports (astro/planets, etc.) without the -p option. But what about recursion? I held the notion the portmaster did magically delicious recursion Heh, I like that phrase. and so I thought that providing the list of ports to a single invocation of portmaster would allow portmaster to sort the build order correctly. Well, yes and no. If you give the whole long list of ports to build it will save the information about which ports are already up to date between builds, but that feature has only been tested for a few ports at a time. In theory what you're proposing should work, but I haven't tested it. In fact, if you do that I'd be interested in your results. :) There is one other issue that I think would help you understand the process; the concept of root, trunk, branch, and leaf ports. If you do 'portmaster -l' on a system that has the stuff you want installed on it already, you'll get a good idea what I'm talking about. To start from scratch on a clean system you want to extract the list of root (no dependencies, not depended on) and leaf (have dependencies, not depended on) ports from a working system (add any additional ports you want of course), and then use that list to build on your clean system. This will allow portmaster to handle the recursion through the dependencies (trunk and branch ports). The code snippet you show would cause portmaster to _forget_ recursion information during each iteration of the loop. My concern was that some dependencies would be built more than once. Well good news there, if the installed version of the port is up to date it won't be rebuilt. The version check is pretty light weight, and only occurs once during the config phase. After that it's cached. Like I said above, specifying the whole list of ports on the command line _should_ work, and would avoid even having to do the check more than once, but doing the roots and leaves one by one shouldn't add very much of a performance penalty either. hope this helps, Doug -- This .signature sanitized for your protection ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Package Building in the Large
Most of the message below is just rounding out the discussion. There is one significant question on recursion though. Doug Barton wrote: I also ended up with shared a library version problem in at least one port (grip) in spite of having started my build with a completely vacant /usr/local. That sounds like a problem with the ports. I can't think of any way that portmaster would cause that. It might be due to a non-clean /usr/local on the target install host. It was a foo.so.3 is required but foo.so.2 is installed sort of error. I haven't dealt with the installation step in a meaningful way yet. Don't worry about responding to this bit. It looks to have good port upgrading abilities, but that is not what I am after. What I am trying to do is operate a build host and distribute packages from there. That is not its primary purpose, but assuming that your systems are similarly configured, and running an OS version that's fairly close in age, there is no reason that portmaster shouldn't be able to at least create the packages that you can then distribute manually. All my systems run a -release and all ports are tagged with the same -release. (With a few odd cases when security issues demand.) A few years ago I decided to stop tracking -stable and just run pure releases. I now spend less time administering my five or so systems. By what you claim, portmaster should suffice. I used 'portmaster -GgDt -p /usr/ports/ports $pmarg' for my run where $pmarg was the list of 31 ports. That's definitely not going to work, for several reasons. The -G switch should only be used _after_ you've already run portmaster without it for a given port build. I had manually run 'make config-recursive' previously. I'll do as advised at any rate. I'm also unsure what /usr/ports/ports is supposed to be, unless that's a local path issue. It's a local hierarchy issue. I pushed the ports tree up one level and therein lie 6.2, -current, and the soon to be 7.0 trees. /usr/ports/ports is a link to the tree I was working on in which is /usr/ports/ports-current. Once the FreeBSD portmanager pulls the trigger I will be dealing with 7.0 exclusively. Assuming that your list of ports is relative to the /usr/ports directory (e.g., 'archivers/unzip astro/planets', etc.), what I would do is: for dir in $pmarg; do portmaster -BgD -p /usr/ports/$dir done Aha! I had an error (don't recall exactly) without using -p where the complaint had something to do with not finding something in /usr/ports. So I fired up the manual and then found the -p switch and assumed that was how I told portmaster where the ports tree was stored. I think this is the ticket. I don't have my manpages handy right now but I'll take a closer look. But what about recursion? I held the notion the portmaster did magically delicious recursion and so I thought that providing the list of ports to a single invocation of portmaster would allow portmaster to sort the build order correctly. The code snippet you show would cause portmaster to _forget_ recursion information during each iteration of the loop. My concern was that some dependencies would be built more than once. That's dog slow. Replace the literal /usr/ports/ in that string with the actual location of your ports tree if its different. You probably also want to add the -v switch in there for the first few runs to familiarize yourself with what's going on, and review the man page. Please note, I am not saying that portmaster will definitely be the right solution for you. However, I do think it's worth pointing out that if used properly it should at least be able to do what it says it can do. I'll give it another go. Later, Jason ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Package Building in the Large
On Thu, Nov 22, 2007 at 07:13:04AM -0800, Jason C. Wells wrote: ... Not all dependencies had a package built for them. For my list of 31 ports that I actually desired to build there was a dependency list (make all-depends-list) of 758 ports. Of those 758 ports there were 427 packages built. I also ended up with shared a library version problem in at least one port (grip) in spite of having started my build with a completely vacant /usr/local. ... I would also like the portmanagers to know that I think they do a bang-up job. I just have my ultra narrow fussy roll your own way of doing things. I am looking for the right method _for me_. All of the above is no criticism. I usually just use pkg_create -b within simple shell scripts. It's as simple as it gets and works every time. ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Package Building in the Large
Jason C. Wells wrote: I have been toying with a variety of package building methods lately. My latest effort involves looking into the Third Party Release Engineering documented here: http://www.freebsd.org/doc/en_US.ISO8859-1/articles/releng-packages/index.html. Where do I start if I am looking for package building tools that the FreeBSD project uses for burning onto ROMs? Is ports/Tools/scripts/release the right place? The dates on the files there seem stale. ports/Tools/portbuild/scripts/ What I am trying to do is to build 30 or so packages including the big ones like X, kde, gnome, plus all of their dependencies on a build host and then use pkg_add on various machines. I have had a variety of difficulties with all of the methods I have used thus far (portmaster, portupgrade, homegrown). Probably you want the tinderbox port though, unless you need to distribute across many build hosts. Kris ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to [EMAIL PROTECTED]
Package Building in the Large
I have been toying with a variety of package building methods lately. My latest effort involves looking into the Third Party Release Engineering documented here: http://www.freebsd.org/doc/en_US.ISO8859-1/articles/releng-packages/index.html. Where do I start if I am looking for package building tools that the FreeBSD project uses for burning onto ROMs? Is ports/Tools/scripts/release the right place? The dates on the files there seem stale. What I am trying to do is to build 30 or so packages including the big ones like X, kde, gnome, plus all of their dependencies on a build host and then use pkg_add on various machines. I have had a variety of difficulties with all of the methods I have used thus far (portmaster, portupgrade, homegrown). Thanks, Jason C. Wells ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to [EMAIL PROTECTED]
