Re: Utility for safe updating of ports in base system
On Fri, 21 Mar 2008 14:35:00 +0100 Nikola Lečić [EMAIL PROTECTED] wrote: -BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 On Fri, 21 Mar 2008 00:49:05 +0100 Kris Kennaway [EMAIL PROTECTED] wrote: In the past we've even talked about how to move all of pkg_* out of src and into ports, i.e. the opposite case. A propos (non-)removing pkg_* into the ports, what do you people think of introducing bin-install ports make target (similar to what pkgsrc has)? Besides, bin[-install]-fetch[-recursive] come to mind as well. IMHO, this would have nice pedagogical and heuristic effect since it could 1) resolve eternal users' confusion on what is the difference between ports and packages and between 'make install' and 'pkg_add': it's different if a package is actually installed by e,g. cd /usr/ports/lang/python ; make [BIN_PKGSITES=...] bin-install 2) parallelise binary-install/compiling/local-package-building logics; 3) simplify things since it would cut proliferation of similar command-line options and utilities/methods that do the same thing. Best regards. - -- Nikola Lečić = Никола Лечић fingerprint : FEF3 66AF C90E EDC3 D878 7CDC 956D F4AB A377 1C9B I like the 'make update' target as implemented in OpenBSD. Its then very convenient to update a port when binary update is not an option. my 2 cents, - Etienne ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Utility for safe updating of ports in base system
On Sat, Apr 12, 2008 at 07:01:15PM +0200, clemens fischer wrote: On Fri, 21 Mar 2008 17:21:10 +1100 Peter Jeremy wrote: Note that UFS is a database: If I've understood you correctly, the main problem is that there is no appropriate index to map a port directory to an installed package name. This could be fixed... sorry to be late. how about creating a special directory holding symlinks from package names to port directories? The other way around. ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Utility for safe updating of ports in base system
On Fri, 21 Mar 2008 17:21:10 +1100 Peter Jeremy wrote: Note that UFS is a database: If I've understood you correctly, the main problem is that there is no appropriate index to map a port directory to an installed package name. This could be fixed... sorry to be late. how about creating a special directory holding symlinks from package names to port directories? clemens ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Utility for safe updating of ports in base system
Ulrich Spoerlein [EMAIL PROTECTED] writes: Some people mentioned license issues with certain ports that would disallow the package building: These issues are non-existant if you are talking about in-house distribution only. All our jdks are pkg_add(1)ed and would love to be upgraded just the same. I'm not sure I understand the point correctly, so let me just ask: isn't this the difference between RESTRICTED and NO_PACKAGE? ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Utility for safe updating of ports in base system
Sorry for the late reply, catching up on emails ... On Thu, 20.03.2008 at 23:32:49 +0100, Pav Lucistnik wrote: Doug Barton píše v čt 20. 03. 2008 v 13:12 -0700: Pav Lucistnik wrote: Doug Barton píše v c(t 20. 03. 2008 v 01:05 -0700: On Thu, 20 Mar 2008, Michel Talon wrote: i would venture to say that such an utility should be able to upgrade things based of *binary* packages, and consequently that portmaster is not a suitable candidate. That ability is not included in the current requirements document, and was not specifically mentioned the last time we had the discussion on the list. If the portmgr folks intend that to be a requirement, the current ideas list entry should be amended. Yes, I think ability to work with packages on a remote FTP site with no local /usr/ports, solely relying on an INDEX file, is a solid must have requirement. I have added that to the entry in the Ideas page. Amen. To give people in this thread an idea for a very important use case for businesses running on FreeBSD: We have a build machine set up, that will build all our required ports on certain releases with our custom make.conf settings. All these releases are revisioned themselves, so we can always tell which exact version of what binaries where present on those systems. Our FreeBSD machines themselves do not have any /usr/ports and don't (can't) mount it via NFS. They solely have to rely on pkg_add(1) using FTP to update their packages. This is where an automatic tool would come in handy :) Some people mentioned license issues with certain ports that would disallow the package building: These issues are non-existant if you are talking about in-house distribution only. All our jdks are pkg_add(1)ed and would love to be upgraded just the same. I would be more sympathetic to this idea if we could somehow push security-sensitive package builds up to the top of the list (so they would be available ASAP), but the last time I inquired about this I was told it isn't possible. It's not possible at the moment. There are dreams of having an on-commit action that would trigger a rebuild of the changed port and only the changed port on all platforms. But I feel this would be fairly hard to implement inside the current pointyhat framework. I hacked something together once, that would more or less make it trivial to implement this. I used make(1) (of course) to get the package dependancy in the form foo.tbz: bar.tbz baz.tbz # black magic to build foo.tbz inside clean tree This immediately allows one to use 'make -j64 openoffice.org.tbz' and one could add the following dependancy, to have the port rebuild after every commit (to Makefile, or some suitable file of the port) foo.tbz: category/foo/Makefile Though this will trigger quite a hefty rebuild, if you ever touch the Makefile of the gettext port for example. Cheers, Ulrich Spoerlein -- It is better to remain silent and be thought a fool, than to speak, and remove all doubt. ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Utility for safe updating of ports in base system
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 On Fri, 21 Mar 2008 00:49:05 +0100 Kris Kennaway [EMAIL PROTECTED] wrote: In the past we've even talked about how to move all of pkg_* out of src and into ports, i.e. the opposite case. A propos (non-)removing pkg_* into the ports, what do you people think of introducing bin-install ports make target (similar to what pkgsrc has)? Besides, bin[-install]-fetch[-recursive] come to mind as well. IMHO, this would have nice pedagogical and heuristic effect since it could 1) resolve eternal users' confusion on what is the difference between ports and packages and between 'make install' and 'pkg_add': it's different if a package is actually installed by e,g. cd /usr/ports/lang/python ; make [BIN_PKGSITES=...] bin-install 2) parallelise binary-install/compiling/local-package-building logics; 3) simplify things since it would cut proliferation of similar command-line options and utilities/methods that do the same thing. Best regards. - -- Nikola Lečić = Никола Лечић fingerprint : FEF3 66AF C90E EDC3 D878 7CDC 956D F4AB A377 1C9B -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.8 (FreeBSD) iJwEAQEDAAYFAkfjuY4ACgkQ/MM/0rYIoZjhIQQAjHJ9qFtjoWpXkXDVR4VmoJVX SIBqPPS3us4+kXidhGeNAFXdg3WSVBA+HSjZvGqlmBtcYnHb6+1aQf7T10EKosVK crXmNMlotSy9RcBjy/sVCiRSPVhlFbMg82W+0jPLMtOCirIaezrMaY0NL8UvlJzI 6xA/WG6O7EXitZ8hU9c= =Jn0E -END PGP SIGNATURE- ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Utility for safe updating of ports in base system
On Thu, 20 Mar 2008, Michel Talon wrote: Doug Barton wrote: So, I renew my inquiry. :) Is portmaster a suitable candidate to fulfill the role of the utility described, and if not, why not? At the risk of being flamed, I certainly hope not. :) i would venture to say that such an utility should be able to upgrade things based of *binary* packages, and consequently that portmaster is not a suitable candidate. That ability is not included in the current requirements document, and was not specifically mentioned the last time we had the discussion on the list. If the portmgr folks intend that to be a requirement, the current ideas list entry should be amended. For example pkg_add installs a binary package, if you want to compile and install you run make all install clean in the ports tree. Um, you lost me there. One of the requirements of an upgrade system is predictability, this can only be achieved by using binary packages. You gain a certain amount of flexibility with packages, at the expense of being able to customize things. As long as the user understands that, then it's fine. Another requirement, in my opinion, is speed, and the lack of speed, which is completely hidden when you compile your packages will be immediately apparent if you try to use packages. Indeed portupgrade has options -P and -PP to work with packages which could serve as a prototype for a pkg_upgrade written in C, except that they work poorly, and in particular run slowly. Where do you think the slowness is? In my opinion, an example of a correct pkg_upgrade type programm written in C++ is the Debian apt-get. It works predictably, fast, etc. One of its features, that i consider very important for correct operation, is that it computes the list of all packages to be deleted and all packages to be installed and asks the user if he agrees before doing anything. Why do you consider this an important feature? (I'm not disagreeing, just curious about your thought process here.) It fetches all necessary packages before installing or deleting anything. That seems sensible, thanks for mentioning that bit. Hence you can be sure that the upgrade process will not end in a mess if something crashes in the middle, like it is the case with all present standard FreeBSD upgraders. Not sure if this helps the situation you're referring to or not, but portmaster will by default make a backup package of each port that it updates, so if something dies in the middle you could back out of it by hand if you need to. Now all that said, I'd love to see us move to a much more robust package management system, or even just a better interface to the one we have. The problem is that I don't have the time to do that as a volunteer project, and I don't think anyone else does either. :) Doug -- This .signature sanitized for your protection ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Utility for safe updating of ports in base system
On Thu, Mar 20, 2008 at 01:05:27AM -0700, Doug Barton wrote: On Thu, 20 Mar 2008, Michel Talon wrote: Doug Barton wrote: i would venture to say that such an utility should be able to upgrade things based of *binary* packages, and consequently that portmaster is not a suitable candidate. That ability is not included in the current requirements document, and was not specifically mentioned the last time we had the discussion on the list. If the portmgr folks intend that to be a requirement, the current ideas list entry should be amended. Indeed you are right, but i think this should be a requirement for the reasons discussed below, and is implicit in the fact that the projects refers to a portupgrade written in C, and portupgrade has such a feature. For example pkg_add installs a binary package, if you want to compile and install you run make all install clean in the ports tree. Um, you lost me there. This is simple, all pkg_* tools operate on binary packages at present, it would be strange that a newcomer, pkg_upgrade has no way to operate on binary packages. One of the requirements of an upgrade system is predictability, this can only be achieved by using binary packages. You gain a certain amount of flexibility with packages, at the expense of being able to customize things. As long as the user understands that, then it's fine. I agree that the possibility of compiling from source brings ability to costumize things. However this very ability ruins all hope of having a smooth upgrade system. Due to customization, as soon as you have many ports installed, there is a combinatorial explosion of possibilities and nobody can test that the upgrade process works. OpenBSD people have been converted to this idea, and now push for an upgrade from binary packages exclusively. So i think that the user should have two possibilities: - either he wants to customize, and he is on his own. He will need to compile his software, and in this case portmaster is the perfect tool for managing his upgrades. Since the compilation will take most of the time, it is not relevant to consider performance questions on the portmaster side. - or he wants to use a tried and true upgrade path, he doesn't want to spend time compiling, he wants speed. Basically he wants the Debian or Ubuntu experience. In this case, using prebuilt binary packages is the only reasonable way of achieving the result. I agree that due to licence peculiarities (think e.g. Java) there are a small number of ports which will need compilation, so the experience cannot be perfect. Another requirement, in my opinion, is speed, and the lack of speed, which is completely hidden when you compile your packages will be immediately apparent if you try to use packages. Indeed portupgrade has options -P and -PP to work with packages which could serve as a prototype for a pkg_upgrade written in C, except that they work poorly, and in particular run slowly. Where do you think the slowness is? Several causes of slowness have already been identified. Parsing /var/db/pkg is slow, this is why the idea of using a database has been advocated. Much worse, running make in a port directory, even only to get the value of a variable is slow. Programs like portupgrade do such things repeatedly without caching the results in memory and incur large time penalties. Similarly for each package he wants to download, portupgrade opens an ftp connection and retreives it independently, etc. Obviously no effort at all has been spent so that things are fast. One of its features, that i consider very important for correct operation, is that it computes the list of all packages to be deleted and all packages to be installed and asks the user if he agrees before doing anything. Why do you consider this an important feature? (I'm not disagreeing, just curious about your thought process here.) Because you can see that something is going to break in advance and renounce to the upgrade. This occurred several times for me on Debian. Usually this is because some package has a security problem, and this is solved waiting one or two days. It fetches all necessary packages before installing or deleting anything. That seems sensible, thanks for mentioning that bit. Hence you can be sure that the upgrade process will not end in a mess if something crashes in the middle, like it is the case with all present standard FreeBSD upgraders. Not sure if this helps the situation you're referring to or not, but portmaster will by default make a backup package of each port that it updates, so if something dies in the middle you could back out of it by hand if you need to. Yes, so does portupgrade, but it deletes the backup as soon as the installation of the new port succeeds. This means that in the event of a crash in the middle you remain with a half upgraded system, and no way to backup completely to the previous working state.
Re: Utility for safe updating of ports in base system
Michel Talon [EMAIL PROTECTED] writes: Doug Barton wrote: So, I renew my inquiry. :) Is portmaster a suitable candidate to fulfill the role of the utility described, and if not, why not? At the risk of being flamed, i would venture to say that such an utility should be able to upgrade things based of *binary* packages, and I think this should be the main usage of a possible `pkg_upgrade', just like `pkg_add -u' in OpenBSD. If one is going to install/upgrade packages without a ports tree in the system, `pkg_upgrade' should be able do all the dependency checkings, downloadings and installings/upgradings. consequently that portmaster is not a suitable candidate. For example pkg_add installs a binary package, if you want to compile and install you run make all install clean in the ports tree. One of the requirements of an upgrade system is predictability, this can only be achieved by using binary packages. Another requirement, in my opinion, is speed, and the lack of speed, which is completely hidden when you compile your packages will be immediately apparent if you try to use packages. Indeed portupgrade has options -P and -PP to work with packages which could serve as a prototype for a pkg_upgrade written in C, except that they work poorly, and in particular run slowly. The most inconvenient is that `portupgrade' depends on ruby to run, which makes it a little bit, maybe, annoying. In my opinion, an example of a correct pkg_upgrade type programm written in C++ is the Debian apt-get. It works predictably, fast, etc. One of its features, that i consider very important for correct operation, is that it computes the list of all packages to be deleted and all packages to be installed and asks the user if he agrees before Yes, I think you are right. The new `pkg_upgrade' should be able to report the unavailability of a package and do a graceful exit. doing anything. It fetches all necessary packages before installing or deleting anything. Hence you can be sure that the upgrade process will not end in a mess if something crashes in the middle, like it is the case with all present standard FreeBSD upgraders. Actually I don't think a batch download and install process would help much, especially for a freshly installed system because it might be a huge download job and much waiting time if one is going to install GNOME/KDE etc. from scratch. Perhaps the new `pkg_upgrade' could provide versatile options to complete such tasks. -- Denise H. G. darcsis AT gmail DOT com ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Utility for safe updating of ports in base system
On Thu, Mar 20, 2008 at 01:05:27AM -0700, Doug Barton wrote: ... One of the requirements of an upgrade system is predictability, this can only be achieved by using binary packages. You gain a certain amount of flexibility with packages, at the expense of being able to customize things. As long as the user understands that, then it's fine. ... With respect, that (the notion that use of packages inherently reduces flexibility) doesn't quite follow, from my perspective: it depends on who makes the packages. (What follows is unlikely new to Doug, as I touched on it a while back in a private exchange; I thought it might be of use or interest to the list, though.) My preferred MO is to build from sources, but within a given set of related machines (of sufficiently similar architectures), avoid doing the builds themselves on production machines: I have a machine set up to do FreeBSD builds, for example, and install FreeBSD, built on that system, onto other system(s) here at home. That is why the output of uname -a on my firewall machine (janus) shows that the kernel was actually built on a machine named freebeast: janus(6.3-S)[1] uname -a FreeBSD janus.catwhisker.org 6.3-STABLE FreeBSD 6.3-STABLE #24: Sun Mar 2 07:13:33 PST 2008 [EMAIL PROTECTED]:/common/S1/obj/usr/src/sys/JANUS i386 janus(6.3-S)[2] I would prefer to do something similar for ports: build my own packages on that machine, then be able to use my preferred port management tool to run through the list of installed ports on (say) my firewall box, and have it fetch the packages from my local machine (or effectively *on* the machine being updated, as my build machine exports certain file systems to facilitate installation on other local machines). (And this is, in fact, how I had things set up when I used a different port management program.) This also allows me to have built a package on the non-production build machine, test it, and after I'm satisfied that the package is likely to work in my environment, upgrade the port on a somewhat more sensitive machine (e.g., the one my spouse uses for email Web browsing). And there are some ports -- OpenOffice and Firefox come to mind -- where building a given version more than once veers into the masochistic classification (IMO). (And in the case of OpenOffice, in particular, the port might feasibly be installed and used on a system that doesn't really have the resources to build it.) Thus, for me, being able to customize by building my own packages, then using those custom-built packages for upgrading other systems is a useful approach. While in theory, I could do this manually (roughly: run the port management tool with a -n flag so it won't actually change anything, but would report what warrants updating, then, for each port that has an available update, force a pkg_delete, then do a pkg_add using the updated port's package), that's a lot of clerical work to do by hand -- and to me, that's just about synonymous with error-prone. So having my preferred port management tool be able to perfom upgrades using packages would be useful for me. Peace, david -- David H. Wolfskill [EMAIL PROTECTED] I submit that conspiracy would be an appropriate collective noun for cats. See http://www.catwhisker.org/~david/publickey.gpg for my public key. pgp6J0oy7TCxz.pgp Description: PGP signature
Re: Utility for safe updating of ports in base system
The real question is: ports or packages? Please do not mix them together! If we are talking about ports and ports updating then portmaster is (now) an excellent candidate for this job and I vote for portmaster. If we are talking about packages and package management then portmaster and all other ports updating tools are of course out of context. They are ports management tools - not - package management tools. So please change http://www.freebsd.org/projects/ideas/index.html#p-ports-upgrade The project idea is called 'Utility for safe updating of ports in base system'. The jobs description is false because of using 'pkg_*' but meaning 'port*'. And it really looks just like a request for rewrite portupgrade in C and to put this rewritten portupgrade in base. I strongly vote against it! The entry 'Package tools improvements' http://www.freebsd.org/projects/ideas/index.html#p-ports-pkgtools should be extended / rewritten for a new package management tool (/framework) idea. Michel Talon had described such a tool. So we are actually talking about two different things ... -- Regards, Sticky Bit [EMAIL PROTECTED] ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Utility for safe updating of ports in base system
On Thu, Mar 20, 2008 at 05:59:28PM +0800, Denise H. G. wrote: Michel Talon [EMAIL PROTECTED] writes: Actually I don't think a batch download and install process would help much, especially for a freshly installed system because it might be a huge download job and much waiting time if one is going to install GNOME/KDE etc. from scratch. Perhaps the new `pkg_upgrade' could provide versatile options to complete such tasks. In fact a batch download, followed by batch install is much faster than constant interspersing of backup, download, install, etc. like portupgrade does. In particular there is only one ftp connection for downloading everything which cuts on time, and you can do backups at the same time. If you don't beleive me you can try the prototype (in python) that i have written a year ago, and which does precisely that: http://www.lpthe.jussieu.fr/~talon/pkgupgrade (which needs http://www.lpthe.jussieu.fr/~talon/pkg_save.py) Most of the time in the script is spent recomputing the INDEX for all installed files, because i assumed the INDEX is not necessarily up to date. -- Denise H. G. darcsis AT gmail DOT com -- Michel TALON ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Utility for safe updating of ports in base system
Michel Talon [EMAIL PROTECTED] writes: On Thu, Mar 20, 2008 at 05:59:28PM +0800, Denise H. G. wrote: Michel Talon [EMAIL PROTECTED] writes: Actually I don't think a batch download and install process would help much, especially for a freshly installed system because it might be a huge download job and much waiting time if one is going to install GNOME/KDE etc. from scratch. Perhaps the new `pkg_upgrade' could provide versatile options to complete such tasks. In fact a batch download, followed by batch install is much faster than constant interspersing of backup, download, install, etc. like portupgrade does. In particular there is only one ftp connection for downloading everything which cuts on time, and you can do backups at the Yes, you are right. I just think that people could have options while doing things. This would make the world more satisfying. same time. If you don't beleive me you can try the prototype (in python) that i have written a year ago, and which does precisely that: http://www.lpthe.jussieu.fr/~talon/pkgupgrade (which needs http://www.lpthe.jussieu.fr/~talon/pkg_save.py) Most of the time in the script is spent recomputing the INDEX for all installed files, because i assumed the INDEX is not necessarily up to date. Yes, I've had great impressions by the debian's apt- tools. But it seems that the debian package servers maintain an index or something for all the packages. And if you want to upgrade or install a certain package, you just fetch the meta info for that package from the package server and do a comparison with your local index. This makes versioned dependencies rather easy to play around. -- Denise H. G. darcsis AT gmail DOT com -- Denise H. G. darcsis AT gmail DOT com ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Utility for safe updating of ports in base system
On Thu, 20 Mar 2008, Michel Talon wrote: On Thu, Mar 20, 2008 at 05:59:28PM +0800, Denise H. G. wrote: Michel Talon [EMAIL PROTECTED] writes: Actually I don't think a batch download and install process would help much, especially for a freshly installed system because it might be a huge download job and much waiting time if one is going to install GNOME/KDE etc. from scratch. Perhaps the new `pkg_upgrade' could provide versatile options to complete such tasks. In fact a batch download, followed by batch install is much faster than constant interspersing of backup, download, install, etc. like portupgrade does. In particular there is only one ftp connection for downloading everything which cuts on time, and you can do backups at the same time. If you don't beleive me you can try the prototype (in python) that i have written a year ago, and which does precisely that: http://www.lpthe.jussieu.fr/~talon/pkgupgrade (which needs http://www.lpthe.jussieu.fr/~talon/pkg_save.py) Most of the time in the script is spent recomputing the INDEX for all installed files, because i assumed the INDEX is not necessarily up to date. Except for the package portion, I also wrote a script to update ports[1] when Perl was still in the base. The features I really like about it: 1. No dependency on INDEX. This does make it more expensive to start, but the program does cache as much as possible as it runs. No need to perform a duplicate make all-depends-list in a directory if it has already been performed. 2. Precalculation of updates before starting. This allows a lot of operations to be performed upfront: ignore checking, conflict checking, prefetching and ports no longer needed by child ports. 3. Crash recovery. It saves what it has completed along with the build tree to a database, so recovery/restarts are quick. 4. Uses portconf for settings and its own rc file to determine what not to build. BTW, I think the +IGNOREME files for portmaster should be in /var/db/ports, so they may traverse a manual pkg_delete make install. Cons: 1. I have not put much time into it lately. 2. Perl is not part of base. 3. Does not respect options set in /var/db/ports. It solely relies on portconf. 4. The tree building code is really convoluted and needs replacement. Sean 1. http://www.farley.org/freebsd/#pc -- [EMAIL PROTECTED] ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Utility for safe updating of ports in base system
On Thu, 20 Mar 2008, Doug Barton wrote: On Thu, 20 Mar 2008, Michel Talon wrote: In my opinion, an example of a correct pkg_upgrade type programm written in C++ is the Debian apt-get. It works predictably, fast, etc. One of its features, that i consider very important for correct operation, is that it computes the list of all packages to be deleted and all packages to be installed and asks the user if he agrees before doing anything. Why do you consider this an important feature? (I'm not disagreeing, just curious about your thought process here.) Personally, I like to know everything that will happen before it happens. When options change, I am not always sure what it will bring. For example, I do not have HAL (WITHOUT_HAL via portconf) installed on my system. Some ports may try to bring it in regardless of the setting; I would like to see that first. In this case, I think portmaster -na gives me an idea of what will happen. Sean -- [EMAIL PROTECTED] ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Utility for safe updating of ports in base system
On Thu, 20 Mar 2008 05:23:23 -0700 David Wolfskill wrote: I would prefer to do something similar for ports: build my own packages on that machine, then be able to use my preferred port management tool to run through the list of installed ports on (say) my firewall box, and have it fetch the packages from my local machine The port ports-mgmt/tinderbox does just what you need. WBR -- bsam ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Utility for safe updating of ports in base system
On Thu, Mar 20, 2008 at 09:34:38PM +0800, Denise H. G. wrote: Yes, I've had great impressions by the debian's apt- tools. But it seems that the debian package servers maintain an index or something for all the packages. And if you want to upgrade or install a certain package, you just fetch the meta info for that package from the package server and do a comparison with your local index. This makes versioned dependencies rather easy to play around. Indeed there is a compressed file on the Debian repository, containing all information on each available package. It is stored locally by apt-get when you run apt-get update in a custom database. Similarly apt-get maintains information in the database of the installed packages. Hence comparison is easy and fast. In the FreeBSD case there is always an INDEX file in the FreeBSD repositories which lists similar information (perhaps not enough information) for each package. So in principle one could do similar things as Debian does. -- Michel TALON ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Utility for safe updating of ports in base system
Doug Barton píše v čt 20. 03. 2008 v 01:05 -0700: On Thu, 20 Mar 2008, Michel Talon wrote: i would venture to say that such an utility should be able to upgrade things based of *binary* packages, and consequently that portmaster is not a suitable candidate. That ability is not included in the current requirements document, and was not specifically mentioned the last time we had the discussion on the list. If the portmgr folks intend that to be a requirement, the current ideas list entry should be amended. Yes, I think ability to work with packages on a remote FTP site with no local /usr/ports, solely relying on an INDEX file, is a solid must have requirement. I have added that to the entry in the Ideas page. At the same time, ability to work on a local /usr/ports _without_ INDEX file is also a requirement. I think we should be pushing our packages and package-only modes of operation to the mainstream users. Especially now when we can afford to build a complete package set for all existing platforms/architectures on a 48-hour cycle basis. There should also be an overhaul of current ftp mirrors infrastructure, which might not be able to sustain the constant flow of new packages. Moving over from ftp to http would also eliminate a lot of the latency issues as mentioned elsewhere in this thread. -- Pav Lucistnik [EMAIL PROTECTED] [EMAIL PROTECTED] Your sig line (k) was stolen! -more- There is a puff of smoke! signature.asc Description: Toto je digitálně podepsaná část zprávy
Re: Utility for safe updating of ports in base system
Hi, On Thu, Mar 20, 2008 at 01:05:27AM -0700, Doug Barton wrote: Now all that said, I'd love to see us move to a much more robust package management system, or even just a better interface to the one we have. The problem is that I don't have the time to do that as a volunteer project, and I don't think anyone else does either. :) I did a lot of this work, when I did have the time: http://sourceforge.net/projects/fpkg/ Supports upgrading (doesn't move the files to compat, but that would be a five line change), backups, db files, versioned depends, pkg_which. Just about all of the ideas on that page, but not command line compatible with portupgrade. I haven't played with it in years. It would need all of the pkg_install features and fixes from about 2004 (or maybe earlier) merged in (or reimplemented). Regards, -Jeremy -- FreeBSD - Because the best things in life are free... http://www.freebsd.org/ ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Utility for safe updating of ports in base system
Michel Talon wrote: On Thu, Mar 20, 2008 at 01:05:27AM -0700, Doug Barton wrote: On Thu, 20 Mar 2008, Michel Talon wrote: Doug Barton wrote: i would venture to say that such an utility should be able to upgrade things based of *binary* packages, and consequently that portmaster is not a suitable candidate. That ability is not included in the current requirements document, and was not specifically mentioned the last time we had the discussion on the list. If the portmgr folks intend that to be a requirement, the current ideas list entry should be amended. Indeed you are right, but i think this should be a requirement for the reasons discussed below, and is implicit in the fact that the projects refers to a portupgrade written in C, and portupgrade has such a feature. Implicit requirements suck. :) However pav obviously agrees with you. For example pkg_add installs a binary package, if you want to compile and install you run make all install clean in the ports tree. Um, you lost me there. This is simple, all pkg_* tools operate on binary packages at present, Ok, but that's not compile and install, which is why I wanted to clarify. it would be strange that a newcomer, pkg_upgrade has no way to operate on binary packages. I think reasonable minds could differ on that, but now we know what pav at least thinks. One of the requirements of an upgrade system is predictability, this can only be achieved by using binary packages. You gain a certain amount of flexibility with packages, at the expense of being able to customize things. As long as the user understands that, then it's fine. I agree that the possibility of compiling from source brings ability to costumize things. However this very ability ruins all hope of having a smooth upgrade system. Due to customization, as soon as you have many ports installed, there is a combinatorial explosion of possibilities and nobody can test that the upgrade process works. I agree with what you're saying to a point, however in my experience it's very rare that at least my particular combination of options leads to an explosion, so I'm not really sold on the idea of mandatory binary-only upgrades. OpenBSD people have been converted to this idea, and now push for an upgrade from binary packages exclusively. I think we need to keep the tools not policy mantra in mind here. So i think that the user should have two possibilities: - either he wants to customize, and he is on his own. He will need to compile his software, and in this case portmaster is the perfect tool for managing his upgrades. Thanks. :) Since the compilation will take most of the time, it is not relevant to consider performance questions on the portmaster side. Having spent a substantial amount of time doing performance tuning on portmaster, I beg to differ. You're right of course that for the average port the majority of time is going to be spent in compiling, however if you're going to be doing something like 'portmaster -a' simply caching the results of the up-to-date tests of the lower level dependencies can cut the portmaster time for an upgrade of several ports by more than 1/3rd. And portmaster has a lot of other optimizations as well, such as downloading distfiles in the background, etc. - or he wants to use a tried and true upgrade path, he doesn't want to spend time compiling, he wants speed. Basically he wants the Debian or Ubuntu experience. In this case, using prebuilt binary packages is the only reasonable way of achieving the result. Well, given the overwhelming number of ports, and the lack of resources to do QA on them, I think we should substitute more likely to succeed for tried and true there. That said, I have used Ubuntu for a project I did for a client, and I really liked the package management system. However you're talking orders of magnitude more work (and several redesigns of core elements) before we could get close to the Ubuntu experience. I'm concerned that in the interest of waiting for the perfect thing we do our users a disservice by not taking incremental steps toward the goal. Another requirement, in my opinion, is speed, and the lack of speed, which is completely hidden when you compile your packages will be immediately apparent if you try to use packages. Indeed portupgrade has options -P and -PP to work with packages which could serve as a prototype for a pkg_upgrade written in C, except that they work poorly, and in particular run slowly. Where do you think the slowness is? Several causes of slowness have already been identified. Parsing /var/db/pkg is slow, s/is/can be/. There are some things I need to dig out of /var/db/pkg that are relatively slow, yes. By far the most expensive is determining the name of an installed port by grep'ing for ORIGIN in all the +CONTENTS files. However even that is not overwhelmingly slow. If I do a cold search for the very last installed port on my
Re: Utility for safe updating of ports in base system
Sean C. Farley wrote: BTW, I think the +IGNOREME files for portmaster should be in /var/db/ports, so they may traverse a manual pkg_delete make install. I'm ambivalent about that, since the way I personally tend to use +IGNOREME is to avoid dealing with something till I'm ready to update it, and then I generally want the +IGNOREME file to go away. I would think that at minimum it should look both places. Also, I'm a little more sympathetic to this idea now that portmaster does a better job of letting the user know that there is an +IGNOREME file present at various decision making points. Doug -- This .signature sanitized for your protection ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Utility for safe updating of ports in base system
Pav Lucistnik wrote: Doug Barton píe v c(t 20. 03. 2008 v 01:05 -0700: On Thu, 20 Mar 2008, Michel Talon wrote: i would venture to say that such an utility should be able to upgrade things based of *binary* packages, and consequently that portmaster is not a suitable candidate. That ability is not included in the current requirements document, and was not specifically mentioned the last time we had the discussion on the list. If the portmgr folks intend that to be a requirement, the current ideas list entry should be amended. Yes, I think ability to work with packages on a remote FTP site with no local /usr/ports, solely relying on an INDEX file, is a solid must have requirement. I have added that to the entry in the Ideas page. Fair enough, but can we please come quickly to a consensus on what _all_ of the requirements should be? Two things I'd like to avoid. One is the feeling that no matter how many hoops I jump through, there is always going to be one more placed in my path because we really don't want portmaster in the base. The other is frustration on the part of any student brave enough to tackle this task. At the same time, ability to work on a local /usr/ports _without_ INDEX file is also a requirement. That should be mentioned then, but the good news is that portmaster already fulfills that one. It never uses the INDEX file for anything. I think we should be pushing our packages and package-only modes of operation to the mainstream users. Especially now when we can afford to build a complete package set for all existing platforms/architectures on a 48-hour cycle basis. I would be more sympathetic to this idea if we could somehow push security-sensitive package builds up to the top of the list (so they would be available ASAP), but the last time I inquired about this I was told it isn't possible. There should also be an overhaul of current ftp mirrors infrastructure, which might not be able to sustain the constant flow of new packages. You also need to look at the other side of that, which is an exponential increase in the number of package downloads, and the incumbent costs in terms of bandwidth, processor time, etc. Doug -- This .signature sanitized for your protection ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Utility for safe updating of ports in base system
On Thu, Mar 20, 2008 at 01:12:06PM -0700, Doug Barton wrote: Fair enough, but can we please come quickly to a consensus on what _all_ of the requirements should be? Two things I'd like to avoid. One is the feeling that no matter how many hoops I jump through, there is always going to be one more placed in my path because we really don't want portmaster in the base. The other is frustration on the part of any student brave enough to tackle this task. Now that it is clear that 2 different things are desirable, a utility to upgrade by ports, and a utility to upgrade by packages, may i say that i agree completely that portmaster does the first perfectly, and should be in the base system, notably since it is a shell script which doesn't cost anything. But i remain convinced that the second one is also necessary and in fact much more important. Since portmaster is already here, a SOC project should concentrate on the second aim, which is perfectly summarized by the name pkg_upgrade (by opposition to port_upgrade). The ideas necessary to develop such a project are apparent in the ruby program portupgrade and/or my python pkgupgrade, but a C program is desirable so that there is no external dependency. More precisely portupgrade and pkgupgrade both use extensively data structures such as arrays and dictionaries (perl hashes) so it may be that using C++ and the STL containers is more convenient than C. You also need to look at the other side of that, which is an exponential increase in the number of package downloads, and the incumbent costs in terms of bandwidth, processor time, etc. All big Linux distributions, which have many times more users, incur this cost without apparent problem. For example my provider, which is a quite large one, has a mirror of many distributions, including Free and NetBSD, with incredible bandwith. See ftp://ftp.free.fr/mirrors I can download a DVD from my office at 100Mb/s from here at any time. I would venture to say that the problem you mention is really a non problem. -- Michel TALON ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Utility for safe updating of ports in base system
Doug Barton píše v čt 20. 03. 2008 v 13:12 -0700: Pav Lucistnik wrote: Doug Barton píše v c(t 20. 03. 2008 v 01:05 -0700: On Thu, 20 Mar 2008, Michel Talon wrote: i would venture to say that such an utility should be able to upgrade things based of *binary* packages, and consequently that portmaster is not a suitable candidate. That ability is not included in the current requirements document, and was not specifically mentioned the last time we had the discussion on the list. If the portmgr folks intend that to be a requirement, the current ideas list entry should be amended. Yes, I think ability to work with packages on a remote FTP site with no local /usr/ports, solely relying on an INDEX file, is a solid must have requirement. I have added that to the entry in the Ideas page. Fair enough, but can we please come quickly to a consensus on what _all_ of the requirements should be? Two things I'd like to avoid. One is the feeling that no matter how many hoops I jump through, there is always going to be one more placed in my path because we really don't want portmaster in the base. I will install portmaster on my machine now and give it a tryout. Tell you what I like and what I dislike about it. I'm afraid the moment I start pushing for making something that is not portupgrade a new de-facto standard, it will turn into a muddy politics really fast. Well I guess I have to bite a bullet. And I don't think anyone started talking to the src guys about including it in base system yet. The other is frustration on the part of any student brave enough to tackle this task. It is not marked as a Summer of Code suggested project. Only the parallelization task is. I think we should be pushing our packages and package-only modes of operation to the mainstream users. Especially now when we can afford to build a complete package set for all existing platforms/architectures on a 48-hour cycle basis. I would be more sympathetic to this idea if we could somehow push security-sensitive package builds up to the top of the list (so they would be available ASAP), but the last time I inquired about this I was told it isn't possible. It's not possible at the moment. There are dreams of having an on-commit action that would trigger a rebuild of the changed port and only the changed port on all platforms. But I feel this would be fairly hard to implement inside the current pointyhat framework. There should also be an overhaul of current ftp mirrors infrastructure, which might not be able to sustain the constant flow of new packages. You also need to look at the other side of that, which is an exponential increase in the number of package downloads, and the incumbent costs in terms of bandwidth, processor time, etc. My impression is that current mirrors are vastly underutilized, and that it's not a big problem to get more mirrors if we ask. But the current cvsup/rsync method of distributing changes from ftp-master onto mirrors is not good enough. It introduces a huge latency, it's unpredictable and inconsistent. We would need to build a CDN for packages, ideally. Also, if we want to get serious about packages, we need to ensure atomic updates of package sets on the ftp master and on the mirrors. We currently don't do that. -- Pav Lucistnik [EMAIL PROTECTED] [EMAIL PROTECTED] Quantum physics was developed in the 1930's, as a result of a bet between Albert Einstein and Niels Bohr, to see who could come up with the most ridiculous theory and still have it published. signature.asc Description: Toto je digitálně podepsaná část zprávy
Re: Utility for safe updating of ports in base system
Doug Barton wrote: So, I renew my inquiry. :) Is portmaster a suitable candidate to fulfill the role of the utility described, and if not, why not? At the risk of being flamed, i would venture to say that such an utility should be able to upgrade things based of *binary* packages, and consequently that portmaster is not a suitable candidate. For example pkg_add installs a binary package, if you want to compile and install you run make all install clean in the ports tree. One of the requirements of an upgrade system is predictability, this can only be achieved by using binary packages. Another requirement, in my opinion, is speed, and the lack of speed, which is completely hidden when you compile your packages will be immediately apparent if you try to use packages. Indeed portupgrade has options -P and -PP to work with packages which could serve as a prototype for a pkg_upgrade written in C, except that they work poorly, and in particular run slowly. In my opinion, an example of a correct pkg_upgrade type programm written in C++ is the Debian apt-get. It works predictably, fast, etc. One of its features, that i consider very important for correct operation, is that it computes the list of all packages to be deleted and all packages to be installed and asks the user if he agrees before doing anything. It fetches all necessary packages before installing or deleting anything. Hence you can be sure that the upgrade process will not end in a mess if something crashes in the middle, like it is the case with all present standard FreeBSD upgraders. -- Michel TALON ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Utility for safe updating of ports in base system
On Tue, 17 Apr 2007 20:43:51 -0700 Garrett Cooper [EMAIL PROTECTED] wrote: RW wrote: On Tue, 17 Apr 2007 14:31:39 -0400 Adam Stroud [EMAIL PROTECTED] wrote: I was just on the FreeBSD list of projects and ideas fot volunteers page and I was wondering if anyone was working on the portupgrade in C utility. I would be willing to help (code/document/test) if it's needed. What would be the point of putting any port-upgrading tool in the base-system? The ports tree isn't branched, so why branch such tools in the base-system. IMO they logically belong in ports where they are better able to follow any developments in the ports-system. I'm working on combining the pkg_* tools along with the existing makefile system with a bourne shell file for my SoC project(*) As for writing a utility in C, why? Almost everything's there right now and just needs to be strung together with some clever scripting to make it all work. C in this case is just overkill IMO. I'm sorry RW, because while I do agree to some extent, there should be something out of the box that works with the Makefiles and does everything necessary to install ports, apart from someone manually going to each directory and typing in make install, or using some advanced functionality in the Makefiles. You make install (or pkg_add) once to build your preferred tool. If that's too much trouble, sysinstall might add the package as part of a standard profile - it does this with xorg. Or it could add the package when it asks you if you want the ports tree. This is not a serious argument for putting something in the base-system. If you put a build tool in the base-system then sooner or later someone will need a newer version, which means having a port too and flags to say which is the preferred version. This will lead to needless confusion. And since few people will want to upgrade their up-to-date ports with an out-of-date tool, I suspect that most people will go for the ports version anyway. I've yet to hear a single cogent argument for this. ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Utility for safe updating of ports in base system
Garrett Cooper youshi10 at u.washington.edu writes: I'm working on combining the pkg_* tools along with the existing makefile system with a bourne shell file for my SoC project(*) As for writing a utility in C, why? Almost everything's there right now and just needs to be strung together with some clever scripting to make it all work. C in this case is just overkill IMO. I'm sorry RW, because while I do agree to some extent, there should be something out of the box that works with the Makefiles and does everything necessary to install ports, apart from someone manually going to each directory and typing in make install, or using some advanced functionality in the Makefiles. (*) This is the 2nd goal on my list out of 3 items, so it's going to be a while before I get to it (I estimate 2 months). -Garrett I was not suggesting writing the utilities in C, I was just indicating the the topic on the FreeBSD ideas page. Frankly, I am a best tool for the job type of guy. My intent was just to probe the community and see if anything was being developed and if anyone needed any help. Adam ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Utility for safe updating of ports in base system
On Tue, 17 Apr 2007 14:31:39 -0400 Adam Stroud [EMAIL PROTECTED] wrote: I was just on the FreeBSD list of projects and ideas fot volunteers page and I was wondering if anyone was working on the portupgrade in C utility. I would be willing to help (code/document/test) if it's needed. What would be the point of putting any port-upgrading tool in the base-system? The ports tree isn't branched, so why branch such tools in the base-system. IMO they logically belong in ports where they are better able to follow any developments in the ports-system. ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Utility for safe updating of ports in base system
On 4/18/07, Adam Stroud [EMAIL PROTECTED] wrote: All: I was just on the FreeBSD list of projects and ideas fot volunteers page and I was wondering if anyone was working on the portupgrade in C utility. I would be willing to help (code/document/test) if it's needed. I think that some projects were started, but did not go very far. The current consensus is that portmaster, which is a collection of sh scripts, should be further developed and eventually committed to the base system. The developer of portmaster is one of those who does not really like that idea. ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Utility for safe updating of ports in base system
RW wrote: On Tue, 17 Apr 2007 14:31:39 -0400 Adam Stroud [EMAIL PROTECTED] wrote: I was just on the FreeBSD list of projects and ideas fot volunteers page and I was wondering if anyone was working on the portupgrade in C utility. I would be willing to help (code/document/test) if it's needed. What would be the point of putting any port-upgrading tool in the base-system? The ports tree isn't branched, so why branch such tools in the base-system. IMO they logically belong in ports where they are better able to follow any developments in the ports-system. I'm working on combining the pkg_* tools along with the existing makefile system with a bourne shell file for my SoC project(*) As for writing a utility in C, why? Almost everything's there right now and just needs to be strung together with some clever scripting to make it all work. C in this case is just overkill IMO. I'm sorry RW, because while I do agree to some extent, there should be something out of the box that works with the Makefiles and does everything necessary to install ports, apart from someone manually going to each directory and typing in make install, or using some advanced functionality in the Makefiles. (*) This is the 2nd goal on my list out of 3 items, so it's going to be a while before I get to it (I estimate 2 months). -Garrett ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to [EMAIL PROTECTED]
