Re: poudriere: reduce the number of rebuilt packages?
On Fri, Jan 02, 2015 at 12:03:54PM +0100, Stefan Ehmann wrote: I've recently switched from portmaster to poudriere/'pkg upgrade' to manage my port updates. Basically it works fine, but incremental builds don't quite work as I expected. poudriere rebuilds all packages if any dependency has changed. If there are only some ports with new versions, possibly hundreds of packages are rebuilt. So far it looks like I'll end up rebuilding packages like libreoffice/KDE/chromium several times a week. The rebuilt packages won't even be installed by 'pkg upgrade' because their version number has not changed. That's a huge waste of resources. With portmaster only ports with increased version numbers are rebuilt. Can I use poudriere to rebuild only packages where the version number changed? The problem here in consistency while in theory we can cherry-pick what we really want to rebuild based on libraries provided/required in binary packages poudriere has to deal with the ports tree and compatibility. The ports tree was a heavy user of pkg_add which became pkg add, this tool was relying on the version of dependencies as registered in the package creation: - A-v1 was depending on B-v2 if B-v2 is bumped to B-v3 then A-v1 dependency chain is broken in regard pkg_add. Just for that we have no choices but rebuilding everything that depends on B This can now be easily fixed because pkg_install is gone and we do not have to rely on compatiblity with it anymore, the problem is people willing to work on that (flexible dependencies and smart dependencies) have been mostly killed by the nightmare this compatibility has introduced into pkg. Lots of scripts still rely on the pkg_add behaviour and until all of them are killed I'm afraid we won't be able to prevent those massive rebuilds. That is if you are doing the things correctly. Now there is an alternative. Introduce a new repository format for file:// kind of url (like Zypper) which will not need all the metadata and be blazing fast to produce. Use pkg install instead of pkg add in ports and then we can reduce the massive rebuilds to only rebuild things that really requires a library and only a library being removed/upgraded. Any volunteers? regards, Bapt pgpLkP2eHxgtm.pgp Description: PGP signature
Re: poudriere: reduce the number of rebuilt packages?
Matthew Seaman matt...@freebsd.org writes: poudriere only knows that the dependency changed. In effect, to find out if the package of interest would be changed because of that, it has no other recourse than to build the package. Now, if you can come up with some heuristics whereby you can examine the changes to a port and determine that they will not cause significant downstream changes, and do that reliably and faster than just rebuilding the package, then I'm sure the poudriere developers would be eager to incorporate them. Failing that, poudriere re-building everything that might be affected is the sensible choice. If I know that only the actually changed ports need to be rebuilt, I go into my jail but instead of running poudriere, I use portmaster -g. Unfortunately, it's really easy to be wrong about knowing that. ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: poudriere: reduce the number of rebuilt packages?
On 12.01.2015 18:55, Mathieu Arnold wrote: Well, no, there is not, and unless you figure out an algorithm to do it, and I'm saying algorithm in the mathematical sense, not heuristic, that is, one that is always right, feel free to submit a patch for it :-) Now, there's a good chance that it will be slower than rebuilding all the dependencies. What about the logic in pkg, aren't both programs solving the same problem? Regards, Karel ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: poudriere: reduce the number of rebuilt packages?
On 12/01/2015 21:05, Karel Miklav wrote: On 12.01.2015 18:55, Mathieu Arnold wrote: Well, no, there is not, and unless you figure out an algorithm to do it, and I'm saying algorithm in the mathematical sense, not heuristic, that is, one that is always right, feel free to submit a patch for it :-) Now, there's a good chance that it will be slower than rebuilding all the dependencies. What about the logic in pkg, aren't both programs solving the same problem? No, the problems solved by pkg and poudriere are very different. pkg(8) has the massive advantage that it already has access to the built packages, including analysis of the dynamic linkage of all the binaries included there. That makes it relatively simple for pkg to work out if a change in some dependency has had a material effect on the package being considered. poudriere only knows that the dependency changed. In effect, to find out if the package of interest would be changed because of that, it has no other recourse than to build the package. Now, if you can come up with some heuristics whereby you can examine the changes to a port and determine that they will not cause significant downstream changes, and do that reliably and faster than just rebuilding the package, then I'm sure the poudriere developers would be eager to incorporate them. Failing that, poudriere re-building everything that might be affected is the sensible choice. Cheers, Matthew -- Dr Matthew J Seaman MA, D.Phil. PGP: http://www.infracaninophile.co.uk/pgpkey signature.asc Description: OpenPGP digital signature
Re: poudriere: reduce the number of rebuilt packages?
On 12.01.2015 18:45, Kurt Jaeger wrote: Hi! The option -S (Don't recrusively rebuild packages affected by other packages requiring incremental rebuild) looked promising. But automatic packages are not always rebuilt and I've also encountered build problems. I'll try this option now and would appreciate if you could describe the problems you had with it. Details help. Not all ports with new versions were rebuilt. That is kind of expected if there's no recursive rebuild. As a workaround I simply passed the output of pkg query %o to poudriere bulk -f. Now poudriere tried to build only the packages with newer versions - just as desired. But then there were build errors for some packages. Unfortunately, I no longer have the log files. But I think the problem was something like this: A --depends on-- B --depends on-- C * B was built depending on C version x * C is upgraded from version x to version y * rebuild of A is requested * poudriere tries to install B version x during build * installation of B fails because version x of C is not longer available ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: poudriere: reduce the number of rebuilt packages?
On 12.01.2015 18:08, Mathieu Arnold wrote: +--On 12 janvier 2015 17:59:36 +0100 Stefan Ehmann shoes...@gmx.net wrote: | | But it would be nice to have a poudriere option to avoid rebuilds of | ports without version bumps. If something should go horribly wrong every | now and then, you can still fall back to the default rebuild behavior. | | I think it should be possibly provided that port versions are bumped | correctly. But maybe I'm wrong. Like someone else said, you can use bulk -S, but don't complain if you end up with something that's horribly broken :-) bulk -S didn't really work well for me (see original post). But I think I my original question is answered by now: There's no supported way of avoiding excessive rebuilds with poudriere. What I'm now doing is: * Dry run of poudriere bulk with normal list of ports * make a list of ports that will be rebuilt because of new version * Run poudriere bulk with new list There are still some ports without version bump rebuilt, but it's a much smaller number. Especially libreoffice/KDE, etc. rebuilds are avoided. This procedure probably has some issues. But in case of problems I'll just do a normal incremental build. ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: poudriere: reduce the number of rebuilt packages?
Hi! The option -S (Don't recrusively rebuild packages affected by other packages requiring incremental rebuild) looked promising. But automatic packages are not always rebuilt and I've also encountered build problems. I'll try this option now and would appreciate if you could describe the problems you had with it. Details help. -- p...@opsec.eu+49 171 3101372 5 years to go ! ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: poudriere: reduce the number of rebuilt packages?
+--On 12 janvier 2015 18:45:06 +0100 Stefan Ehmann shoes...@gmx.net wrote: | On 12.01.2015 18:08, Mathieu Arnold wrote: | | | +--On 12 janvier 2015 17:59:36 +0100 Stefan Ehmann shoes...@gmx.net | wrote: | | | | But it would be nice to have a poudriere option to avoid rebuilds of | | ports without version bumps. If something should go horribly wrong | | every now and then, you can still fall back to the default rebuild | | behavior. | | | | I think it should be possibly provided that port versions are bumped | | correctly. But maybe I'm wrong. | | Like someone else said, you can use bulk -S, but don't complain if you | end up with something that's horribly broken :-) | | bulk -S didn't really work well for me (see original post). | | But I think I my original question is answered by now: | There's no supported way of avoiding excessive rebuilds with poudriere. Well, no, there is not, and unless you figure out an algorithm to do it, and I'm saying algorithm in the mathematical sense, not heuristic, that is, one that is always right, feel free to submit a patch for it :-) Now, there's a good chance that it will be slower than rebuilding all the dependencies. | What I'm now doing is: | * Dry run of poudriere bulk with normal list of ports | * make a list of ports that will be rebuilt because of new version | * Run poudriere bulk with new list | | There are still some ports without version bump rebuilt, but it's a much | smaller number. Especially libreoffice/KDE, etc. rebuilds are avoided. | | This procedure probably has some issues. But in case of problems I'll | just do a normal incremental build. -- Mathieu Arnold ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: poudriere: reduce the number of rebuilt packages?
On 12.01.2015 01:27, Mathieu Arnold wrote: +--On 4 janvier 2015 18:24:24 +0100 Stefan Ehmann shoes...@gmx.net wrote: | On 02.01.2015 12:03, Stefan Ehmann wrote: | I've recently switched from portmaster to poudriere/'pkg upgrade' to | manage my port updates. Basically it works fine, but incremental builds | don't quite work as I expected. | | poudriere rebuilds all packages if any dependency has changed. If there | are only some ports with new versions, possibly hundreds of packages are | rebuilt. So far it looks like I'll end up rebuilding packages like | libreoffice/KDE/chromium several times a week. The rebuilt packages | won't even be installed by 'pkg upgrade' because their version number | has not changed. | | Here's an actual example from today. | | There are new versions for three ports. poudriere will rebuild 70 ports, | 67 of them will never be installed on the host. You can't know that. Say there is a shlib change in one of the updated packages, its version is bumped, or there is a new dependency, you need to rebuild the 67 ports, and pkg will detect and reinstall them. I remember one instance where a library change was detected (Can't remember which package was affected). The package was re-installed although the version number did not change. But like 99% of the time packages without version bump will not be installed. One could argue that the version number should have been bumped in the other case. I understand that conservative rebuilding is important for the official pkg repo. But it would be nice to have a poudriere option to avoid rebuilds of ports without version bumps. If something should go horribly wrong every now and then, you can still fall back to the default rebuild behavior. I think it should be possibly provided that port versions are bumped correctly. But maybe I'm wrong. ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: poudriere: reduce the number of rebuilt packages?
+--On 12 janvier 2015 17:59:36 +0100 Stefan Ehmann shoes...@gmx.net wrote: | | But it would be nice to have a poudriere option to avoid rebuilds of | ports without version bumps. If something should go horribly wrong every | now and then, you can still fall back to the default rebuild behavior. | | I think it should be possibly provided that port versions are bumped | correctly. But maybe I'm wrong. Like someone else said, you can use bulk -S, but don't complain if you end up with something that's horribly broken :-) -- Mathieu Arnold ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: poudriere: reduce the number of rebuilt packages?
On 1/12/2015 11:45 AM, Kurt Jaeger wrote: Hi! The option -S (Don't recrusively rebuild packages affected by other packages requiring incremental rebuild) looked promising. But automatic packages are not always rebuilt and I've also encountered build problems. I'll try this option now and would appreciate if you could describe the problems you had with it. Details help. This option was added for an intent that never panned out. I should remove it. It's dangerous and will create a broken repository. -- Regards, Bryan Drewery signature.asc Description: OpenPGP digital signature
Re: poudriere: reduce the number of rebuilt packages?
+--On 4 janvier 2015 18:24:24 +0100 Stefan Ehmann shoes...@gmx.net wrote: | On 02.01.2015 12:03, Stefan Ehmann wrote: | I've recently switched from portmaster to poudriere/'pkg upgrade' to | manage my port updates. Basically it works fine, but incremental builds | don't quite work as I expected. | | poudriere rebuilds all packages if any dependency has changed. If there | are only some ports with new versions, possibly hundreds of packages are | rebuilt. So far it looks like I'll end up rebuilding packages like | libreoffice/KDE/chromium several times a week. The rebuilt packages | won't even be installed by 'pkg upgrade' because their version number | has not changed. | | Here's an actual example from today. | | There are new versions for three ports. poudriere will rebuild 70 ports, | 67 of them will never be installed on the host. You can't know that. Say there is a shlib change in one of the updated packages, its version is bumped, or there is a new dependency, you need to rebuild the 67 ports, and pkg will detect and reinstall them. -- Mathieu Arnold ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: poudriere: reduce the number of rebuilt packages?
On 04.01.2015 18:24, Stefan Ehmann wrote: On 02.01.2015 12:03, Stefan Ehmann wrote: I've recently switched from portmaster to poudriere/'pkg upgrade' to manage my port updates. Basically it works fine, but incremental builds don't quite work as I expected. poudriere rebuilds all packages if any dependency has changed. If there are only some ports with new versions, possibly hundreds of packages are rebuilt. So far it looks like I'll end up rebuilding packages like libreoffice/KDE/chromium several times a week. The rebuilt packages won't even be installed by 'pkg upgrade' because their version number has not changed. Here's an actual example from today. There are new versions for three ports. poudriere will rebuild 70 ports, 67 of them will never be installed on the host. Checking packages for incremental rebuild needed Deleting cups-client-1.7.3_3.txz: new version: 1.7.3_4 Same story here. On each ports update Poudriere churns couple of days, mostly wasting time on some version of the damn webkit. Then on pkg update a couple of insignificant ports are updated. It's easy to fix version check in Poudriere, but I haven't decided to dig out the corresponding lines in pkg to match them. Will there be some option in Poudriere for this? Regards, Karel ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: poudriere: reduce the number of rebuilt packages?
02 янв. 2015 г. 14:04 пользователь Stefan Ehmann shoes...@gmx.net написал: I've recently switched from portmaster to poudriere/'pkg upgrade' to manage my port updates. Basically it works fine, but incremental builds don't quite work as I expected. poudriere rebuilds all packages if any dependency has changed. If there are only some ports with new versions, possibly hundreds of packages are rebuilt. So far it looks like I'll end up rebuilding packages like libreoffice/KDE/chromium several times a week. The rebuilt packages won't even be installed by 'pkg upgrade' because their version number has not changed. Absolutely agree. A tired to rebuild kde and libreoffice each time when some port 's (Perl, python )revision bumped. That's a huge waste of resources. With portmaster only ports with increased version numbers are rebuilt. Can I use poudriere to rebuild only packages where the version number changed? The option -S (Don't recrusively rebuild packages affected by other packages requiring incremental rebuild) looked promising. But automatic packages are not always rebuilt and I've also encountered build problems. CHECK_CHANGED_DEPS=no doesn't seem to reduce the number of packages rebuilt. Is there a way to do what I want with poudriere? Or should I switch back to portmaster? ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: poudriere: reduce the number of rebuilt packages?
On 02.01.2015 12:03, Stefan Ehmann wrote: I've recently switched from portmaster to poudriere/'pkg upgrade' to manage my port updates. Basically it works fine, but incremental builds don't quite work as I expected. poudriere rebuilds all packages if any dependency has changed. If there are only some ports with new versions, possibly hundreds of packages are rebuilt. So far it looks like I'll end up rebuilding packages like libreoffice/KDE/chromium several times a week. The rebuilt packages won't even be installed by 'pkg upgrade' because their version number has not changed. Here's an actual example from today. There are new versions for three ports. poudriere will rebuild 70 ports, 67 of them will never be installed on the host. Checking packages for incremental rebuild needed Deleting cups-client-1.7.3_3.txz: new version: 1.7.3_4 Deleting gcc-4.8.3_2.txz: new version: 4.8.4 Deleting gsoap-2.8.18_1.txz: new version: 2.8.21 Deleting chromium-39.0.2171.95_2.txz: missing dependency: cups-client-1.7.3_3 Deleting cups-image-1.7.3_3.txz: missing dependency: cups-client-1.7.3_3 Deleting ghostscript9-9.06_10.txz: missing dependency: cups-image-1.7.3_3 Deleting gtk2-2.24.25_1.txz: missing dependency: cups-client-1.7.3_3 Deleting gtk3-3.14.6.txz: missing dependency: cups-client-1.7.3_3 Deleting gtkspell-2.0.16_5.txz: missing dependency: gtk2-2.24.25_1 Deleting gutenprint-base-5.2.10.txz: missing dependency: gtk2-2.24.25_1 Deleting gutenprint-ijs-5.2.10.txz: missing dependency: gtk2-2.24.25_1 Deleting kBuild-0.1.9998_3.txz: missing dependency: gcc-4.8.3_2 Deleting libglade2-2.6.4_7.txz: missing dependency: gtk2-2.24.25_1 Deleting libpurple-2.10.11.txz: missing dependency: gtk2-2.24.25_1 Deleting libreoffice-4.3.5_2.txz: missing dependency: cups-client-1.7.3_3 Deleting librsvg2-2.40.6.txz: missing dependency: gtk2-2.24.25_1 Deleting libspectre-0.2.7.txz: missing dependency: ghostscript9-9.06_10 Deleting mlt-0.9.2_1.txz: missing dependency: gtk2-2.24.25_1 Deleting musicpd-0.18.11_5.txz: missing dependency: gcc-4.8.3_2 Deleting nspluginwrapper-1.4.4_4.txz: missing dependency: gtk2-2.24.25_1 Deleting nvidia-settings-340.24_1.txz: missing dependency: gtk2-2.24.25_1 Deleting okular-4.14.2_1.txz: missing dependency: libspectre-0.2.7 Deleting openbox-3.5.2_7.txz: missing dependency: gtk2-2.24.25_1 Deleting pidgin-2.10.11.txz: missing dependency: gtk2-2.24.25_1 Deleting pidgin-otr-4.0.0_5.txz: missing dependency: gtk2-2.24.25_1 Deleting policykit-gnome-0.9.2_7.txz: missing dependency: gtk2-2.24.25_1 Deleting py27-gimp-2.8.14.txz: missing dependency: gtk2-2.24.25_1 Deleting py27-gtk2-2.24.0_3.txz: missing dependency: gtk2-2.24.25_1 Deleting py27-mcomix-1.00_2.txz: missing dependency: gtk2-2.24.25_1 Deleting swfdec-0.8.4_5.txz: missing dependency: gtk2-2.24.25_1 Deleting thunderbird-31.3.0_1.txz: missing dependency: gtk2-2.24.25_1 Deleting webkit-gtk2-2.4.7.txz: missing dependency: gtk2-2.24.25_1 Deleting wx30-gtk2-3.0.2_1.txz: missing dependency: gtk2-2.24.25_1 Deleting xsane-0.999_4.txz: missing dependency: gtk2-2.24.25_1 Deleting ImageMagick-6.9.0.2,1.txz: missing dependency: ghostscript9-9.06_10 Deleting calibre-1.48.0_2.txz: missing dependency: ImageMagick-6.9.0.2,1 Deleting firefox-34.0.5_1,1.txz: missing dependency: gtk2-2.24.25_1 Deleting gconf2-3.2.6_3.txz: missing dependency: gtk2-2.24.25_1 Deleting gegl-0.2.0_14.txz: missing dependency: librsvg2-2.40.6 Deleting gimp-2.8.14,2.txz: missing dependency: py27-gimp-2.8.14 Deleting gimp-app-2.8.14_1,1.txz: missing dependency: gegl-0.2.0_14 Deleting gimp-gutenprint-5.2.10_2.txz: missing dependency: gimp-app-2.8.14_1,1 Deleting gnome-mount-0.8_12.txz: missing dependency: gconf2-3.2.6_3 Deleting goffice010-0.10.11.txz: missing dependency: gtk2-2.24.25_1 Deleting gqview-2.0.4_15.txz: missing dependency: gtk2-2.24.25_1 Deleting gtk-engines2-2.20.2_2.txz: missing dependency: gtk2-2.24.25_1 Deleting gutenprint-5.2.10.txz: missing dependency: gutenprint-base-5.2.10 Deleting gvfs-1.20.3_1.txz: missing dependency: gnome-mount-0.8_12 Deleting kdenlive-0.9.10.txz: missing dependency: mlt-0.9.2_1 Deleting libdmtx-0.7.4_6.txz: missing dependency: ImageMagick-6.9.0.2,1 Deleting mkvtoolnix-7.3.0_1.txz: missing dependency: wx30-gtk2-3.0.2_1 Deleting prison-1.0_1.txz: missing dependency: libdmtx-0.7.4_6 Deleting gnumeric-1.12.11_3.txz: missing dependency: goffice010-0.10.11 Deleting kde-workspace-4.11.13_2.txz: missing dependency: prison-1.0_1 Deleting kdepimlibs-4.14.2.txz: missing dependency: prison-1.0_1 Deleting libkfbapi-1.0_3.txz: missing dependency: kdepimlibs-4.14.2 Deleting libkgapi-2.2.0.txz: missing dependency: kdepimlibs-4.14.2 Deleting libkolab-0.5.3.txz: missing dependency: kdepimlibs-4.14.2 Deleting baloo-4.14.2.txz: missing dependency: kdepimlibs-4.14.2 Deleting baloo-widgets-4.14.2.txz: missing dependency: baloo-4.14.2 Deleting gwenview-4.14.2.txz: missing dependency: baloo-4.14.2 Deleting kde-4.14.2.txz: missing dependency: kde-workspace-4.11.13_2 Deleting
