ports INDEX file layout?
Where can I find the description of the /usr/ports/INDEX-8 file? ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: ports INDEX file layout?
Aiza aiz...@comclark.com writes: Where can I find the description of the /usr/ports/INDEX-8 file? Try bsd.ports.mk. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: ports INDEX file
Benjamin Lee wrote: On 07/22/2010 06:20 PM, Fbsd8 wrote: I have a pristine install of 8.0. There is no /usr/ports directory yet. I am trying to use the portcheckout port and the porteasy port to just populate the ports tree with only the ports I use. Problem is in both cases the above ports require an existing INDEX file to process and since I have none they don't work. How can I just download the ports INDEX file? Portsnap is not a solution. I see in the source of porteasy that its fetching http://www.freebsd.org/ports/INDEX-8.bz2 How can I verify this? Usually the index file is placed at $INDEXDIR/$INDEXFILE, as defined in $PORTSDIR/Mk/bsd.port.mk. In your case, by default, that would be /usr/ports/INDEX-8. As Matthew asked, do you really want to do this? By modern standards, the space required for the ports tree is modest (~550MB uncompressed), and you can learn a lot about what's available and how things work by looking through it. Plus you save the time required to implement this partial ports tree approach. If you really need to save the disk space, and don't have other special requirements, then considering using binary packages instead of compiling from source. If you do have special requirements -- e.g., you need to build ports with non-default options or special flags, or you don't trust foreign binary packages (in that case, though, you should probably be prepared to do a lot of work auditing the source code as well), and you don't have at least one machine with the required disk space, then maybe this approach is worthwhile. However, that seems unlikely. If you pursue the partial ports tree approach, you don't need to make or fetch an INDEX(which, although it may be a useful summary, may be inappropriate for parsing dependencies for ports built with non-default options), and you don't need to use either of the ports that you mentioned: as someone else said, you could just write a shell script to fetch the necessary infrastructure Makefiles (those in /usr/ports/Mk and the needed category subdirectories), and the desired port and it's dependencies, using cvs(1) (but you have to choose a server that permits anonymous cvs access, and learn cvs), csup(1) (configured to use a suitable cvsup server using the ports-all collection and the -i flag, which would permit you to grab only parts of that collection), or even an http client like fetch(1) (exploiting the fact that single ports can be downloaded in tarball form from cvsweb.freebsd.org in links of the form: http://www.FreeBSD.org/cgi/cvsweb.cgi/ports/$CATEGORY/$PORT/$PORT.tar.gz?tarball=1 and single Makefiles via other links). b. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: ports INDEX file
b. f. wrote: Benjamin Lee wrote: On 07/22/2010 06:20 PM, Fbsd8 wrote: I have a pristine install of 8.0. There is no /usr/ports directory yet. I am trying to use the portcheckout port and the porteasy port to just populate the ports tree with only the ports I use. Problem is in both cases the above ports require an existing INDEX file to process and since I have none they don't work. How can I just download the ports INDEX file? Portsnap is not a solution. I see in the source of porteasy that its fetching http://www.freebsd.org/ports/INDEX-8.bz2 How can I verify this? Usually the index file is placed at $INDEXDIR/$INDEXFILE, as defined in $PORTSDIR/Mk/bsd.port.mk. In your case, by default, that would be /usr/ports/INDEX-8. As Matthew asked, do you really want to do this? By modern standards, the space required for the ports tree is modest (~550MB uncompressed), and you can learn a lot about what's available and how things work by looking through it. Plus you save the time required to implement this partial ports tree approach. If you really need to save the disk space, and don't have other special requirements, then considering using binary packages instead of compiling from source. If you do have special requirements -- e.g., you need to build ports with non-default options or special flags, or you don't trust foreign binary packages (in that case, though, you should probably be prepared to do a lot of work auditing the source code as well), and you don't have at least one machine with the required disk space, then maybe this approach is worthwhile. However, that seems unlikely. If you pursue the partial ports tree approach, you don't need to make or fetch an INDEX(which, although it may be a useful summary, may be inappropriate for parsing dependencies for ports built with non-default options), and you don't need to use either of the ports that you mentioned: as someone else said, you could just write a shell script to fetch the necessary infrastructure Makefiles (those in /usr/ports/Mk and the needed category subdirectories), and the desired port and it's dependencies, using cvs(1) (but you have to choose a server that permits anonymous cvs access, and learn cvs), csup(1) (configured to use a suitable cvsup server using the ports-all collection and the -i flag, which would permit you to grab only parts of that collection), or even an http client like fetch(1) (exploiting the fact that single ports can be downloaded in tarball form from cvsweb.freebsd.org in links of the form: http://www.FreeBSD.org/cgi/cvsweb.cgi/ports/$CATEGORY/$PORT/$PORT.tar.gz?tarball=1 and single Makefiles via other links). Well first thanks for the info you provided though it was all negative. I will explain what my goal is. First though, I have verified that the /usr/ports/INDEX-8 file can be gotten without the using cvs or cvsup. fetch -m http://www.freebsd.org/ports/INDEX-8.bz2; does if fact work and the data on the file is as of 3 hours ago. So that indicates its being kept current. The -m means that if the date of the remote file is NOT newer then the local one, the download is bypassed. Now about my project. Since about 4.0 I stopped using the ports tree method. I now all most totally use the package system. I do not upgrade a RELEASE but instead use the install from scratch method about a few weeks after a new RELEASE is published. So since the package system is also re-build a new for each new RELEASE, I am all ways in sync. Now there are exceptions to using packages. In my case php5 was changed 3 RELEASES ago to no longer contain the apache module, so I now have to compile php5 from the port. But to short cut the compile process, I pre-install all of php5's dependents as packages. And of course I had to figure out who they all were by hand the first time and built a script that automates the whole procedure. I use cvsup at NEW RELEASE time to populate the empty ports tree with ports-base. Then I use cvsup to checkout the php5 make files and them make install and everything comes together just fine. Now the Freebsd method of the 22,000 individual ports each with 3 to 5 files is a method which has out lived its usefulness. TAKE NOTE: NO FLAME WAR INTENDED. I just think a option should exist for us who don't follow the bleeding edge. Sure to some people that big ports tree is no big deal, but I bet they don't do backups. That ports tree directory is a large resource hog if you lift the blinders and look at the big picture. I don't need a reason to convince the budget handlers for money to buy bigger and faster cpu machines or larger disk farms. I come from a world where one has to make do with what one has at hand. So in that light. Anything that can be done to reduce the size of the ports tree is money saved and resources conserved. And I bet I am not alone in the Freebsd world who believes in this. So since I have a method all ready working as I explained
Re: ports INDEX file
On 7/23/10, Fbsd8 fb...@a1poweruser.com wrote: b. f. wrote: Benjamin Lee wrote: On 07/22/2010 06:20 PM, Fbsd8 wrote: ... Well first thanks for the info you provided though it was all negative. I think that you were misinterpreting what I wrote if you think that it was all negative. I will explain what my goal is. First though, I have verified that the /usr/ports/INDEX-8 file can be gotten without the using cvs or cvsup. fetch -m http://www.freebsd.org/ports/INDEX-8.bz2; does if fact work and the data on the file is as of 3 hours ago. So that indicates its being kept current. The -m means that if the date of the remote file is NOT newer then the local one, the download is bypassed Yes, as long as the server supports this, and doesn't unnecessarily change the mtime of the file. Minus some variable expansions, this is basically what the fetchindex target does. If you don't care about hard-coding version numbers, etc., then you might as well not invoke make at all, and just do what you're doing, because it's faster. ... compile php5 from the port. But to short cut the compile process, I pre-install all of php5's dependents as packages. And of course I had to figure out who they all were by hand the first time and built a script that automates the whole procedure. I use cvsup at NEW RELEASE time to populate the empty ports tree with ports-base. Then I use cvsup to checkout the php5 make files and them make install and everything comes together just fine. You may be interested in using ports-mgmt/portmaster ( a shell script with minimal dependencies), which can do something similar to what you are trying to do with the --index-only, -P/-PP, and --packages-build flags. Now the Freebsd method of the 22,000 individual ports each with 3 to 5 files is a method which has out lived its usefulness. TAKE NOTE: NO There is no doubt that the increasing size of the tree, and the fact that some parts of the build infrastructure don't scale well, have created some challenges. But I hardly think that it has outlived its usefulness. FLAME WAR INTENDED. I just think a option should exist for us who don't follow the bleeding edge. Sure to some people that big ports tree is no Well, the bleeding edge versus snapshot issue is a bit different from the debate about the size and modularity of the ports tree. big deal, but I bet they don't do backups. That ports tree directory is a large resource hog if you lift the blinders and look at the big picture. I guess it depends upon the constraints that you are operating under. But fetching a new index is going to take about as much network traffic as an update of the ports tree with csup. So since I have a method all ready working as I explained above, I am collecting information on the elements needed to write a shell script port application based on the method already described. Figure I will use cvsup to populate the port-base and checkout just the parent port make files. Read the INDEX file to automate finding the parent port dependents and reading the /var/db/pkg to skip an dependents all ready installed and then launch pkg_add to install the dependents and on any package failures cycle back and use cvsup to also checkout its make files, before issuing the make install on the parent. Just bear in mind that the default INDEX contains the dependencies for ports built with default options. Changing the options may result in different dependencies. Consider using portmaster. Along this same line of thought, Another area I have problems with is why don't the port make system go and checkout any dependent ports missing make files instead of halting like it does now. The ports system wasn't designed to meet your objectives. Delegating authority to perform bursts of unsupervised network activity at unpredictable intervals would probably be considered a problem by many users. And some tasks require the entirety of the tree to be present. When installing a package it will auto install all of it dependents. There is interest in work with fat packages to do something like you describe: Complete (a.k.a. Fat) packages Suggested Summer of Code 2010 project idea Technical contact: Brooks Davis When bootstrapping systems it would be useful to be able to create a single package file that contains one or more packages and all the required dependent packages. This is conceptually similar to, but different from PC-BSD's PBI package format. PBI's contain a private copy of all dependencies, fat packages would contain each individual package and once installed it would be as though each package was individually installed in the usual manner. This project would consist of additions to the pkg_tools to support creation and installation of a new package file format and to ports to build these packages. Requirements: Strong knowledge of C code. A basic understanding of the inner workings of the ports tree. Also, there is some ongoing work
Re: ports INDEX file
On 07/23/2010 03:56 AM, Fbsd8 wrote: Now the Freebsd method of the 22,000 individual ports each with 3 to 5 files is a method which has out lived its usefulness. TAKE NOTE: NO FLAME WAR INTENDED. I just think a option should exist for us who don't follow the bleeding edge. Sure to some people that big ports tree is no big deal, but I bet they don't do backups. That ports tree directory is a large resource hog if you lift the blinders and look at the big picture. Not really: # mkisofs -D -R -no-pad -iso-level 4 -V ports-$(date +%Y%m%d%H%M%S) -o ports.iso /usr/ports # mkuzip -s 65536 -o ports.iso.uzip ports.iso # mdconfig -a -t vnode -u 7 -f ports.iso # kldload geom_uzip # mount -t cd9660 /dev/md7.uzip /usr/ports # du -sh ports ports.iso ports.iso.uzip # As of last update July 4th 834Mports 565Mports.iso 69Mports.iso.uzip Needs mkisofs and FreeBSD = 7, but it reduces the impact of the tree drastically, and can speed up metadata operations, if your disk happens to be slower than your CPU, as the whole tree (or at least all the filesystem metadata) can be feasibly cached compressed in memory. It also ensures congruent package versions, if you process the tree on one machine and distribute it to all others. Plus, you can exclude the tree from backup entirely and just cache the compressed file someplace safe. I use this same trick with Gentoo's Portage tree, with squashfs, and observe similar benefits. -- Fuzzy love, -CyberLeo Technical Administrator CyberLeo.Net Webhosting http://www.CyberLeo.Net cyber...@cyberleo.net Furry Peace! - http://.fur.com/peace/ ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: ports INDEX file
On 07/23/2010 01:56 AM, Fbsd8 wrote: Now about my project. Since about 4.0 I stopped using the ports tree method. I now all most totally use the package system. I do not upgrade a RELEASE but instead use the install from scratch method about a few weeks after a new RELEASE is published. So since the package system is also re-build a new for each new RELEASE, I am all ways in sync. Now there are exceptions to using packages. In my case php5 was changed 3 RELEASES ago to no longer contain the apache module, so I now have to compile php5 from the port. But to short cut the compile process, I pre-install all of php5's dependents as packages. And of course I had to figure out who they all were by hand the first time and built a script that automates the whole procedure. I use cvsup at NEW RELEASE time to populate the empty ports tree with ports-base. Then I use cvsup to checkout the php5 make files and them make install and everything comes together just fine. Why not build packages in-house then? You've already assumed the bootstrapping cost of a full ports tree checkout to do the dependency scan for php5 -- why not build the binary package (with your relevant make options) there as well? Then the rest of your machines can install *everything* from packages, and therefore won't require *any* of the ports tree, not even some subset of exceptions that need to be compiled. This would save even more resources, since you only compile php5 once, rather than once per machine. -- Benjamin Lee http://www.b1c1l1.com/ signature.asc Description: OpenPGP digital signature
Re: ports INDEX file
On Fri, 23 Jul 2010 16:56:56 +0800 Fbsd8 fb...@a1poweruser.com wrote: tree is no big deal, but I bet they don't do backups. If that's an issue, don't back it up. That ports tree directory is a large resource hog if you lift the blinders and look at the big picture. Just my 2 cents. Funnily enough that's not far off how much it cost to store it. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
ports INDEX file
I have a pristine install of 8.0. There is no /usr/ports directory yet. I am trying to use the portcheckout port and the porteasy port to just populate the ports tree with only the ports I use. Problem is in both cases the above ports require an existing INDEX file to process and since I have none they don't work. How can I just download the ports INDEX file? Portsnap is not a solution. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: ports INDEX file
On 7/22/2010 8:20 PM, Fbsd8 wrote: I have a pristine install of 8.0. There is no /usr/ports directory yet. I am trying to use the portcheckout port and the porteasy port to just populate the ports tree with only the ports I use. Problem is in both cases the above ports require an existing INDEX file to process and since I have none they don't work. How can I just download the ports INDEX file? Portsnap is not a solution. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org You can use 'csup' to get the ports tree down. You'll find the relevant config file (assuming you installed the source tree) at: /usr/src/share/examples/cvsup -- Tim Daneliuk tun...@tundraware.com PGP Key: http://www.tundraware.com/PGP/ ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: ports INDEX file
Tim Daneliuk wrote: On 7/22/2010 8:20 PM, Fbsd8 wrote: I have a pristine install of 8.0. There is no /usr/ports directory yet. I am trying to use the portcheckout port and the porteasy port to just populate the ports tree with only the ports I use. Problem is in both cases the above ports require an existing INDEX file to process and since I have none they don't work. How can I just download the ports INDEX file? Portsnap is not a solution. ___ You can use 'csup' to get the ports tree down. You'll find the relevant config file (assuming you installed the source tree) at: /usr/src/share/examples/cvsup Not interested in the ports tree. Just the INDEX file ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: ports INDEX file
On 07/22/2010 06:20 PM, Fbsd8 wrote: I have a pristine install of 8.0. There is no /usr/ports directory yet. I am trying to use the portcheckout port and the porteasy port to just populate the ports tree with only the ports I use. Problem is in both cases the above ports require an existing INDEX file to process and since I have none they don't work. How can I just download the ports INDEX file? Portsnap is not a solution. Well, The INDEX file is a component of the ports tree distribution. If you choose not to use the supported method of installing it (i.e. installing the ports tree), you'll have to create your own. Hint: Per ports(7), take a look at the definition of the 'fetchindex' target. -- Benjamin Lee http://www.b1c1l1.com/ signature.asc Description: OpenPGP digital signature
Re: ports INDEX file
Benjamin Lee wrote: On 07/22/2010 06:20 PM, Fbsd8 wrote: I have a pristine install of 8.0. There is no /usr/ports directory yet. I am trying to use the portcheckout port and the porteasy port to just populate the ports tree with only the ports I use. Problem is in both cases the above ports require an existing INDEX file to process and since I have none they don't work. How can I just download the ports INDEX file? Portsnap is not a solution. Well, The INDEX file is a component of the ports tree distribution. If you choose not to use the supported method of installing it (i.e. installing the ports tree), you'll have to create your own. Hint: Per ports(7), take a look at the definition of the 'fetchindex' target. I see in the source of porteasy that its fetching http://www.freebsd.org/ports/INDEX-8.bz2 How can I verify this? ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: ports INDEX file
On 23/07/2010 02:20:02, Fbsd8 wrote: I have a pristine install of 8.0. There is no /usr/ports directory yet. I am trying to use the portcheckout port and the porteasy port to just populate the ports tree with only the ports I use. I've heard of a few people trying to do things like this, and mostly the consensus is that's it's more trouble than it's worth. Good luck. In order to make your cut-down tree work properly, you'ld have to maintain custom versions of /usr/ports/Makefile and which ever of the category Makefiles you use (ie. the Makefiles one level down the tree). Problem is in both cases the above ports require an existing INDEX file to process and since I have none they don't work. How can I just download the ports INDEX file? Portsnap is not a solution. You can use my ports-mgmt/p5-FreeBSD-Portindex port to build an INDEX file -- ideally you should get it to run without complaints about missing dependencies and such, but if you don't it will do the best it can to produce something resembling an INDEX. Cheers, Matthew -- Dr Matthew J Seaman MA, D.Phil. 7 Priory Courtyard Flat 3 PGP: http://www.infracaninophile.co.uk/pgpkey Ramsgate JID: matt...@infracaninophile.co.uk Kent, CT11 9PW signature.asc Description: OpenPGP digital signature
Re: ports INDEX file
On Fri, Jul 23, 2010 at 06:14:12AM +0100, Matthew Seaman thus spake: On 23/07/2010 02:20:02, Fbsd8 wrote: I have a pristine install of 8.0. There is no /usr/ports directory yet. I am trying to use the portcheckout port and the porteasy port to just populate the ports tree with only the ports I use. Portcheckout really won't grab all the dependencies that are needed. I've filed a PR for this. I wrote up a quick script that has the same output that grabs all the dependencies. Portcheckout doesn't grab dependencies of dependencies (ie. make all-depends-list) I've heard of a few people trying to do things like this, and mostly the consensus is that's it's more trouble than it's worth. Good luck. In order to make your cut-down tree work properly, you'ld have to maintain custom versions of /usr/ports/Makefile and which ever of the category Makefiles you use (ie. the Makefiles one level down the tree). Problem is in both cases the above ports require an existing INDEX file to process and since I have none they don't work. How can I just download the ports INDEX file? Portsnap is not a solution. You can use my ports-mgmt/p5-FreeBSD-Portindex port to build an INDEX file -- ideally you should get it to run without complaints about missing dependencies and such, but if you don't it will do the best it can to produce something resembling an INDEX. Cheers, Matthew -- Dr Matthew J Seaman MA, D.Phil. 7 Priory Courtyard Flat 3 PGP: http://www.infracaninophile.co.uk/pgpkey Ramsgate JID: matt...@infracaninophile.co.uk Kent, CT11 9PW ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
