Re: [oe] Eliminating dependency race-conditions (was Re: [PATCH] net-snmp: disable libnl use)
On Fri, 2011-03-18 at 06:29 +0100, Esben Haabendal wrote: Yes, I get your point. You could of-course still specify build dependencies using recipe names/provides, and then just unpack all target packages build by that recipe. This would give you the major part of the KISS win I see, except for in the dependency handling code of BitBake (and in the learning curve of OE developers). Keep in mind that by using packages for build-time dependencies, the mechanism is exactly identical for run-time and build-time dependencies. Exactly the same code in BitBake handles both, and the same mechanism needs to be understood. This is also part of the KISS win of doing it this way. Right. So, it seems to me that the appropriate way to attack the problem in OE is to start by doing exactly that: leave the syntax of DEPENDS alone for now, treat each entry in DEPENDS as meaning all packages built by the named recipe, and then implement the per-recipe sysroot construction that I think we are agreed is highly desirable. Then, as a later refinement, we could introduce a mechanism for specifying dependencies more precisely, at the package level rather than the recipe level. We could do that either by extending the syntax of DEPENDS, something like: DEPENDS = boost:boost-iostreams (to say that you wanted just the boost-iostreams package from the boost recipe) or, alternately, by introducing a new variable which would either supplement or replace DEPENDS. Either of those would allow us to migrate to fine-grained dependencies without breaking existing recipes. (How) do you deal with library package renaming in OE-lite? What exactly do you mean? (We are doing several things with library package naming...) I was thinking of situations like the Debian package autonamer, ie the thing that causes glibc-dev to come out named libc6-dev.ipk or similar depending on the soname of the library that was built. It seems like this would be a bit awkward to deal with if your dependencies were specified purely in terms of output packages. p. ___ Openembedded-devel mailing list Openembedded-devel@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
Re: [oe] Eliminating dependency race-conditions (was Re: [PATCH] net-snmp: disable libnl use)
On Thu, 2011-03-17 at 20:58 +0100, Esben Haabendal wrote: On Thu, 2011-03-17 at 18:05 +, Phil Blundell wrote: On Thu, 2011-03-17 at 18:52 +0100, Esben Haabendal wrote: Well, it might be possible to minimize the disruption for a transitional period by carefully specifying some catch-all build-time package dependencies pulling in all packages for recipes not ported yet. Yes, that's the sort of thing I was thinking of. It isn't even totally obvious to me that specifying individual packages is any better than all packages from recipe X, since with the former you then have a potential maintenance headache if files get moved from one package to another. There is a number of ways that I believe package based build dependencies are better than recipe based. a) It is possible to depend on parts of a recipe, which fx. is useful when a recipe provides more than 1 library, where you might not want all of them. b) To build a recipe, you depend on some stuff which you don't need to, or perhaps even really don't want to pass on to recipes depending on this recipe. c) KISS. Using the actual target packages for satisfying build-time dependencies are a very simple approach, which I strongly believe will make OE a better tool in the long run, by reducing complexity, and thus lowering the bar for contributing to OE archictural work. I'd like to step in here and point out that your approach is not as simple as you claim. It certainly is a really nice solution for one very specific use case. There are however many other problems which appear when you step away from your isolated usecase. Take the case of different package managers, how is that handled? Can you only have one enabled at a given time? What happens if you switch package manager half way through a build. Currently package managers and package format are abstracted, I suspect we lose that with your implementation. What if the package managers have slightly different behaviours? Secondly, how about task output that isn't packages? The output of do_deploy for example(kernel/firmware/bootloader)? The pkgdata generated by do_package? A generic extra task added by the user which outputs some data? What you're proposing is to totally actually totally collapse our current generic task structure into something entirely governed by target packages and their specific requirements. Of course you could say the existing mechanisms are still available but then where is the simplification? sstate is very simple in concept and very simple in operation compared to anything we've previously had. It is generic being task based and can apply to any task as required. It doesn't require radical changes to the existing model, there is a clear migration path and it also embraces things like the checksum/signature generation which I believe are going to play a significant role in the future. This isn't to say what you're doing is wrong or that we shouldn't be looking at it for ideas but I believe it needs to be something incremental on top of what we already have and functionality we have now cannot simply be lost as people use it. Cheers, Richard ___ Openembedded-devel mailing list Openembedded-devel@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
Re: [oe] Eliminating dependency race-conditions (was Re: [PATCH] net-snmp: disable libnl use)
On Tue, 2011-03-15 at 10:08 +0100, Esben Haabendal wrote: The current dependency/staging model of OE basically has this feature that a build can be influenced not only by it's own dependencies, but also what has been build before it (or not). I strongly believe that this has to be fixed on the architectural level, and not just on a case-by-case level as is currently needed. Yes, agreed. This clearly sucks, and indeed I think it is probably the largest single area of architectural suckage in OE today. I am still very much interested in discussing how to move this technology from OE-lite to OE, but as it impacts all recipe metadata (build dependencies has to be redefined), OE community at a large really needs to value the benefits of solving this problem. The benefits of solving the problem are clearly very great, but I don't think OE itself is really in a position to embrace a big bang kind of change which would require redefining all the build dependencies. I think we need to find a technological solution which will work with our current DEPENDS scheme. I did do some work in that direction about a year ago, but I got sidetracked into dealing with some toolchain suckage (which was making it difficult to construct the per-recipe sysroots in the way that I wanted) and then ran out of time. I would like to get back to working on this stuff but right now it is difficult for me to set aside the time for it. p. ___ Openembedded-devel mailing list Openembedded-devel@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
Re: [oe] Eliminating dependency race-conditions (was Re: [PATCH] net-snmp: disable libnl use)
On Thu, 2011-03-17 at 11:18 +, Phil Blundell wrote: I am still very much interested in discussing how to move this technology from OE-lite to OE, but as it impacts all recipe metadata (build dependencies has to be redefined), OE community at a large really needs to value the benefits of solving this problem. The benefits of solving the problem are clearly very great, but I don't think OE itself is really in a position to embrace a big bang kind of change which would require redefining all the build dependencies. I think we need to find a technological solution which will work with our current DEPENDS scheme. I know that this seems to be the general consensus in OE, but I do not agree with it. Sometimes architectural work requires working out the details on a branch, and then when it is mostly done, do the big bang which will then not really be that big anymore, as most of the issues have been resolved. Is OE really in a position to permantly settle for something suboptimal in such a central area? /Esben ___ Openembedded-devel mailing list Openembedded-devel@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
Re: [oe] Eliminating dependency race-conditions (was Re: [PATCH] net-snmp: disable libnl use)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 17-03-11 15:52, Graeme Gregory wrote: On 17/03/2011 14:43, Esben Haabendal wrote: On Thu, 2011-03-17 at 11:18 +, Phil Blundell wrote: I am still very much interested in discussing how to move this technology from OE-lite to OE, but as it impacts all recipe metadata (build dependencies has to be redefined), OE community at a large really needs to value the benefits of solving this problem. The benefits of solving the problem are clearly very great, but I don't think OE itself is really in a position to embrace a big bang kind of change which would require redefining all the build dependencies. I think we need to find a technological solution which will work with our current DEPENDS scheme. I know that this seems to be the general consensus in OE, but I do not agree with it. Sometimes architectural work requires working out the details on a branch, and then when it is mostly done, do the big bang which will then not really be that big anymore, as most of the issues have been resolved. Is OE really in a position to permantly settle for something suboptimal in such a central area? I would suggest a branch of oe-core is the ideal place to do this work. It can be proved on oe-core then rolled out to meta-oe at a later date. OE-core already features per-machine sysroots, so the main problem we had, relocatability, has already been solved. regards, Koen -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.5 (Darwin) iD8DBQFNgielMkyGM64RGpERAqcIAKCyT44a9MzDF+esO6DgRbtBXi4L8ACdGzpN 4vN5d5EtUwbo3cNHfMl9XQ4= =eFCq -END PGP SIGNATURE- ___ Openembedded-devel mailing list Openembedded-devel@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
Re: [oe] Eliminating dependency race-conditions (was Re: [PATCH] net-snmp: disable libnl use)
On Thu, 2011-03-17 at 15:07 +, Phil Blundell wrote: On Thu, 2011-03-17 at 15:43 +0100, Esben Haabendal wrote: Is OE really in a position to permantly settle for something suboptimal in such a central area? No, but rejecting the big bang doesn't mean that we can't make the change; it just means that we need to find a way to make the old and new arrangements coexist for a transition period. This is the way every other architectural change in OE has been done and I don't see any reason that this one needs to be different. Exactly If we really did want to go ahead and have a metadata flag day then no doubt there are plenty of other things we would like to change at the same time. But, thus far, this has never seemed to justify the disruption of breaking every recipe in both the main OE repo and third party overlays. Well, it might be possible to minimize the disruption for a transitional period by carefully specifying some catch-all build-time package dependencies pulling in all packages for recipes not ported yet. I think it would be much more worthwhile to put effort into this than to push for per-recipe sysroot with the current sstate solution. Having package-based build-time dependencies, using the same packages as when building run-time images, is much simpler than what is currently done in OE, while at the same time giving increased flexibility in specifying the build-time dependencies needed for a recipe and it's packages. But you might have to try it out before really understanding why you cannot live without it ;-) I must admit that I'm also slightly unclear about why the change in build dependency specifications is a prerequisite for doing per-recipe sysroots. It might not be, but the result will of-course not be the same. Is that just an artifact of the way you implemented it in OE-lite or is there some fundamental constraint that means it needs to be this way? I don't think there are any fundamental constraint. Sometimes doing things right are simply so much easier. As a result, I already do have a solution with both per-recipe staging/sysroot and buld-time dependencies using target packages. I feel pretty confident that this was the fast path, which is sort of backed by the fact that it is done, whereas the path (detour?) currently being worked on in OE is still just on it's way. /Esben ___ Openembedded-devel mailing list Openembedded-devel@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
Re: [oe] Eliminating dependency race-conditions (was Re: [PATCH] net-snmp: disable libnl use)
On Thu, 2011-03-17 at 18:52 +0100, Esben Haabendal wrote: Well, it might be possible to minimize the disruption for a transitional period by carefully specifying some catch-all build-time package dependencies pulling in all packages for recipes not ported yet. Yes, that's the sort of thing I was thinking of. It isn't even totally obvious to me that specifying individual packages is any better than all packages from recipe X, since with the former you then have a potential maintenance headache if files get moved from one package to another. p. ___ Openembedded-devel mailing list Openembedded-devel@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
Re: [oe] Eliminating dependency race-conditions (was Re: [PATCH] net-snmp: disable libnl use)
On Thu, 2011-03-17 at 18:05 +, Phil Blundell wrote: On Thu, 2011-03-17 at 18:52 +0100, Esben Haabendal wrote: Well, it might be possible to minimize the disruption for a transitional period by carefully specifying some catch-all build-time package dependencies pulling in all packages for recipes not ported yet. Yes, that's the sort of thing I was thinking of. It isn't even totally obvious to me that specifying individual packages is any better than all packages from recipe X, since with the former you then have a potential maintenance headache if files get moved from one package to another. There is a number of ways that I believe package based build dependencies are better than recipe based. a) It is possible to depend on parts of a recipe, which fx. is useful when a recipe provides more than 1 library, where you might not want all of them. b) To build a recipe, you depend on some stuff which you don't need to, or perhaps even really don't want to pass on to recipes depending on this recipe. c) KISS. Using the actual target packages for satisfying build-time dependencies are a very simple approach, which I strongly believe will make OE a better tool in the long run, by reducing complexity, and thus lowering the bar for contributing to OE archictural work. The maintenance headache you talk about is already there. In OE-lite build-time dependency 95% of the time just follow the run-time dependencies, perhaps making it easier to maintain than OE, as we don't have to think about another type of item names to depend on. /Esben ___ Openembedded-devel mailing list Openembedded-devel@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
Re: [oe] Eliminating dependency race-conditions (was Re: [PATCH] net-snmp: disable libnl use)
On Thu, 2011-03-17 at 20:58 +0100, Esben Haabendal wrote: On Thu, 2011-03-17 at 18:05 +, Phil Blundell wrote: On Thu, 2011-03-17 at 18:52 +0100, Esben Haabendal wrote: Well, it might be possible to minimize the disruption for a transitional period by carefully specifying some catch-all build-time package dependencies pulling in all packages for recipes not ported yet. Yes, that's the sort of thing I was thinking of. It isn't even totally obvious to me that specifying individual packages is any better than all packages from recipe X, since with the former you then have a potential maintenance headache if files get moved from one package to another. There is a number of ways that I believe package based build dependencies are better than recipe based. a) It is possible to depend on parts of a recipe, which fx. is useful when a recipe provides more than 1 library, where you might not want all of them. You'd still need to have built all of them (at least as far as the end of do_compile) though, right? In that case it doesn't seem as though installing the unnecessary ones in the sysroot is all that big a hardship. b) To build a recipe, you depend on some stuff which you don't need to, or perhaps even really don't want to pass on to recipes depending on this recipe. Sorry, I don't understand what you're saying here. c) KISS. Using the actual target packages for satisfying build-time dependencies are a very simple approach, which I strongly believe will make OE a better tool in the long run, by reducing complexity, and thus lowering the bar for contributing to OE archictural work. I agree with this, but it seems unrelated to the question of whether build dependencies should be specified in terms of recipes or output packages. The maintenance headache you talk about is already there. In OE-lite build-time dependency 95% of the time just follow the run-time dependencies, perhaps making it easier to maintain than OE, as we don't have to think about another type of item names to depend on. (How) do you deal with library package renaming in OE-lite? p. ___ Openembedded-devel mailing list Openembedded-devel@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
Re: [oe] Eliminating dependency race-conditions (was Re: [PATCH] net-snmp: disable libnl use)
On Thu, 2011-03-17 at 21:00 +, Phil Blundell wrote: On Thu, 2011-03-17 at 20:58 +0100, Esben Haabendal wrote: There is a number of ways that I believe package based build dependencies are better than recipe based. a) It is possible to depend on parts of a recipe, which fx. is useful when a recipe provides more than 1 library, where you might not want all of them. You'd still need to have built all of them (at least as far as the end of do_compile) though, right? Of-course. All recipes providing something for a do_stage must be build up till the do_package stage, so I can unpack the actual target packages needed. In that case it doesn't seem as though installing the unnecessary ones in the sysroot is all that big a hardship. It is not to save time, it is about controlling what goes into a build. As many software packages scans for presence of various components through the availability of headers and libraries, I believe it is important to decide not just what goes into the stage directory, but also what does not. So in OE-lite, if I say I depend on libfoo, I do not get foobar, just because it was built by the same recipe. b) To build a recipe, you depend on some stuff which you don't need to, or perhaps even really don't want to pass on to recipes depending on this recipe. Sorry, I don't understand what you're saying here. Example: I build recipe A. Recipe A depends on package B-b from recipe B. Recipe B depends on package C-c from recipe C, and D-d-native from recipe D. Recipe A only get's package B-b staged, unless the B-b package has specified some build-time dependencies. This is relevant both for target library and native tools dependencies. c) KISS. Using the actual target packages for satisfying build-time dependencies are a very simple approach, which I strongly believe will make OE a better tool in the long run, by reducing complexity, and thus lowering the bar for contributing to OE archictural work. I agree with this, but it seems unrelated to the question of whether build dependencies should be specified in terms of recipes or output packages. Yes, I get your point. You could of-course still specify build dependencies using recipe names/provides, and then just unpack all target packages build by that recipe. This would give you the major part of the KISS win I see, except for in the dependency handling code of BitBake (and in the learning curve of OE developers). Keep in mind that by using packages for build-time dependencies, the mechanism is exactly identical for run-time and build-time dependencies. Exactly the same code in BitBake handles both, and the same mechanism needs to be understood. This is also part of the KISS win of doing it this way. The maintenance headache you talk about is already there. In OE-lite build-time dependency 95% of the time just follow the run-time dependencies, perhaps making it easier to maintain than OE, as we don't have to think about another type of item names to depend on. (How) do you deal with library package renaming in OE-lite? What exactly do you mean? (We are doing several things with library package naming...) /Esben ___ Openembedded-devel mailing list Openembedded-devel@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
Re: [oe] Eliminating dependency race-conditions (was Re: [PATCH] net-snmp: disable libnl use)
On Tue, Mar 15, 2011 at 10:08:37AM +0100, Esben Haabendal wrote: On Fri, 2011-03-11 at 12:04 +0100, Steffen Sledz wrote: which occurred sometimes depending on build order (not in clean package only builds). I would like to raise awareness of the underlying problem here. The current dependency/staging model of OE basically has this feature that a build can be influenced not only by it's own dependencies, but also what has been build before it (or not). I strongly believe that this has to be fixed on the architectural level, and not just on a case-by-case level as is currently needed. I haven't received much feedback on the preivous posting about the per-recipe staging principle implemented in OE-lite, but I decided to take this opportunity to re-iterate the fact that the OE-lite implementation of staging and build dependencies eliminates this problem. I am still very much interested in discussing how to move this technology from OE-lite to OE, but as it impacts all recipe metadata (build dependencies has to be redefined), OE community at a large really needs to value the benefits of solving this problem. Esben, Thanks for raising this issue and making people aware of it! This has previously been discussed several times, including last time during the Yocto Summit in December, where I brought up the question of per-package staging/dependency to Richard's attention, when he introduced the new Shared State (sstate). While per-package dependency wasn't considered a critical must-have feature right away, it was acknowledged as something worthwhile looking at and, according to Richard, should be easy to accomplish with the new sstate functionality... -- Denys ___ Openembedded-devel mailing list Openembedded-devel@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
Re: [oe] Eliminating dependency race-conditions (was Re: [PATCH] net-snmp: disable libnl use)
On 03/15/2011 07:38 PM, Esben Haabendal wrote: On Fri, 2011-03-11 at 12:04 +0100, Steffen Sledz wrote: which occurred sometimes depending on build order (not in clean package only builds). I would like to raise awareness of the underlying problem here. The current dependency/staging model of OE basically has this feature that a build can be influenced not only by it's own dependencies, but also what has been build before it (or not). I strongly believe that this has to be fixed on the architectural level, and not just on a case-by-case level as is currently needed. I haven't received much feedback on the preivous posting about the per-recipe staging principle implemented in OE-lite, but I decided to take this opportunity to re-iterate the fact that the OE-lite implementation of staging and build dependencies eliminates this problem. I am still very much interested in discussing how to move this technology from OE-lite to OE, but as it impacts all recipe metadata (build dependencies has to be redefined), OE community at a large really needs to value the benefits of solving this problem. Best regards, Esben ___ Openembedded-devel mailing list Openembedded-devel@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel I did some work last year to fix some of the low hanging fruit (on a case by case basis). My methodology was to do two runs of bitbake -k world, then compare the output. In case anyone is interested, I used the script below to identify the problematic recipes. Last time I ran it was in November, by which time the output had reduced to a list of 5-10 items most of the time. #!/bin/sh # # Quick and dirty script to determine which tasks fail in one world.log, # where they succeeded in another. world.log should be generated with, e.g. # $ bitbake -k world | tee world.log # # If there are too many failures, this script will take too long as grep -f # is slooow. if [ $# != 2 ]; then echo usage: $0 world.log.1 world.log.2 exit 1 fi tasks=do_setscene do_fetch do_distribute_sources do_unpack do_prepsources do_patch do_configure do_qa_configure do_compile do_install do_populate_sysroot do_package do_package_write_ipk do_package_write do_package_stage do_package_stage_all do_build do_qa_staging tmp1=`mktemp /tmp/world_regress.XX` tmp2=`mktemp /tmp/world_regress.XX` for task in $tasks; do grep task $task: Failed $2 \ | sed -e 's/^.*NOTE: package //' \ -e 's/task \(.*\): Failed/\1/' \ $tmp1 grep task $task: Succeeded $1 \ | sed -e 's/^.*NOTE: package //' \ -e 's/task \(.*\): Succeeded/\1/' \ $tmp2 grep -f $tmp1 $tmp2 \ | sort -n done rm -f $tmp1 $tmp2 ___ Openembedded-devel mailing list Openembedded-devel@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
