Hi all, During UDS conversations with various people started to condense some thoughts that had been floating in my head for a while around recipes and looms: the difference, similarities, overlap and clashes that they have. This is another long mail. I apologise for that now, but this is a deep topic, and I think it warrants the length, as it is fundamental to the things that we plan to do.
Looms ===== Looms are a different branch format that allow for multiple "threads" to exist in a single branch, with a total ordering on them. I believe they were conceived with packaging in mind, but either way they are a good fit for the way we work. For packaging the bottom thread is the upstream code. Above this are zero or more threads that modify the code, the "distribution patches" as they are now, where the patches become the diff between each thread and the one below. Above that is the packaging thread that adds the "./debian" directory, or at least modifies it with the latest changelog and the like. You can imagine changes to this scheme, but that's the basic idea that we can work on for now. Usually the threads are kept fully merged, so each tip is merged in to the thread above. Whenever a change is made to a thread it is merged in to the thread above, and so on to the top. This means that changes bubble up to the top thread, so that you can build the top thread and get the complete package. Looms are currently available in the bzr-loom plugin, but it is generally agreed that this needs some polish to be really usable. There are discussions about whether the "stack" model presented by loom is the best, as it entagles threads that don't necessarily have a dependency. There are alternative models proposed, including a full DAG as implemented by TopGit. My fear is that they are too complicated for people to easily grasp, and that they are not the best way to solve the problems they aim to. Looms also allow for versioning at the loom level, as well as at the revision level. This means that it is possible to merge a loom as a whole, including operations that add and delete threads. This is crucial to building a packaging system on top of them. Looms also have a nice two-way mapping to the new v3 source format, so we could both present Debian's packages as looms, and present Debian with v3 source packages. Recipes ======= Recipes are implemented by the bzr-builder plugin. They are a textual representation of a group of operations to perform on a set of branches to build a particular tree. They allow for merge operations, but also "nest" operations. The current focus is on building a tree, but it could be extended to build a branch, and then to be run later to update that branch. The recipe could also be versioned in a branch of its own, allowing you to merge two people's recipes. The overlap =========== There are some parallels between a recipe and a loom, where the branch contains the code, the recipe describes the threads, and the recipes can be merged as they are versioned. This suggests that we have two systems that solve similar, but different, problems, and that have nothing to do with each other currently. The last point alone makes me nervous. Two completely differing systems leads to a fractured development environment, extra learning required, and plenty of opportunity for missed advantadges. Therefore I am keen to at least discuss how to make them work together, if not to see if we can just use one of them. The differences =============== Recipes don't currently have native versioning, and they require grabbing multiple things, the recipe and the branch that was created, in order to work on it. There are also issues around making changes. In order to, say, add a patch, I have to add a line to the recipe, and then run it in order to update the branch, but without causing it to e.g. pull in the new upstream. This suggests to me we may end up with some loom-like commands to make changes. Recipes also currently don't implement their own branch format, so you don't know that you have to edit the recipe to do certain things with the branch. We would also need some code to help with recovering from this mistake. Recipes also require a proliferation of branches. To add a patch you would have to create a new branch, push it to LP, add it to the recipe, and then pull it in. While we can script this, it is also fragile, and perhaps not very elegant. Looms on the other hand currently have no parent pointers per-thread. This makes it impossible to do a full update without extra information. At the least we need to be able to pull in the upstream changes to the bottom thread for daily builds, but recipes allow you to pull in changes to all the other threads at the same time. Looms are also currently missing a way to do the "nest". While this may be just a requirement to have nested trees in core, there may need to be changes I am not considering. Conclusion ========== As you can see, there are a fair number of differences, but none of them seem intractable. There is clearly a disconnect between the two systems though, and I fear it will be a mistake to continue down this path. Given that recipes don't appear to work very well at all for normal Ubuntu development (add a patch type work), and looms could be modified to support the daily builds type work (update all the threads from their parents and build the result), it seems to me as though we should perhaps be looking to modify looms, rather than develop a whole new system. However I also feel that I am missing something with this conclusion, and recipes have more features that perhaps don't fit in to the loom model. Can anyone see anything I have missed? Does anyone come to a different conclusion? Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel