Re: App installer design: click packages
On Wed, May 08, 2013 at 03:25:42PM +0200, Matthias Klumpp wrote: > Disclaimer: As developer of Listaller, AppStream and PackageKit, I am > of course biased on this topic ;-) Thanks for your mail. I'm looking over Listaller at the moment, and expect to have some more comments once I've done that. However, I'm going to be mostly offline from Friday to Sunday, so a partial reply in the meantime: > I find it very sad that so many parties currently start reinventing > the basic ideas of Listaller, instead of helping the Listaller project > to improve, and kill the remaining flaws. I would prefer to keep the discussion technical rather than emotional. > 2013/5/8 Colin Watson : > > * not limited to installing as root, although there may be similar > >constraints elsewhere to ensure that apps can't edit their own code > >at run-time > > Non-root installs into user's $HOME are implemented & working in > Listaller, but due to the tight integration into PackageKit, this > feature should not be used. In practice, I don't actually intend to install packages as the ordinary calling user directly; as I indicated above, this would be a bad idea because it would permit apps to overwrite their own code. However, it's convenient for testing, and it may make sense to unpack apps as a special "software" user or similar to reduce the system's exposure to things like vulnerabilities in underlying unarchiving tools, which are rare but not unknown. (Or we could just hit it with AppArmor. I don't much care either way as long as we avoid the obvious flaw.) > > * building packages requires only the Python standard library, with the > >intent that it should be possible to build these packages quite > >easily on non-Ubuntu or even non-Linux systems > > Hmm, why? If a non-Python application should be packaged, this feature > would be lost, as people will have to crossbuild it anyway. And in > order to build stuff for the upcoming Ubuntu products, people will > need to install it (at least in a VM) anyway, for testing purposes. Remember that the main target of our project, at least for the time being, is Ubuntu Touch - phone/tablet apps. We expect a significant percentage of these to be written in pure QML or HTML5/CSS, and to make it possible to carry these over to desktop apps in time as well. I think it's important to minimise the barrier to entry to creating such apps; if somebody wants to create them on other systems they should be able to. Running a separate test device may well be an easier proposition than building on it. For example, you might have a physical device (Nexus 7 or whatever) that you can easily install things on; that doesn't mean you want to run your development environment there. I agree that building native apps will typically involve building in a clean Ubuntu environment. > > * binary packaging format sufficiently similar to existing one that we > >could add support to higher-level tools with minimal effort > > * strawman design for hooks into system packages, which will be > >entirely declarative from the app's point of view > > Is this going to be Ubuntu-specific? If not, you will have some > trouble with that... I wasn't planning to put "ubuntu" in the hook paths or anything, but it is indeed up to the system to declare how it wants apps to integrate with it. I don't see a sensible way around that; it is entirely wrong to put system/app integration code in the installer itself, and any such approach would result in problems with the system and the installer getting out of sync further down the line. dpkg is right to defer all integration code to packages; the hook scheme I'm considering is mainly inspired by dpkg triggers. (We may or may not be able to call dpkg triggers directly; for performance it is vital to avoid having to actually open the dpkg database.) > > Footnote: Why not use one of the existing systems out there? > > > > We still might, but at present it's not clear that it would make a whole > > lot of difference. > > If it doesn't, why don't you use them? :P It's a trade-off between the benefit of using an existing system and the cost of twisting it around to fit what we actually need it to do. I'm not fundamentally opposed to using an existing system - I certainly understand why people want us to, and that there's a perception that Canonical unnecessarily does its own thing (you can look up my own commit record if you like) - but if there's a mismatch between our goals then it may end up being rather more trouble than it's worth. One thing I'm particularly concerned about is that a system that has tried to be very general may end up reinventing similar scaling and reliability problems as the packaging system we already have. (To be clear - hi Slashdot - I'm very fond of the dpkg/apt system and have put a great deal of effort into it over the last decade or so, both on my own time and for pay; I think it's excellent and have no desire to
Re: App installer design: click packages
Hello Colin, I read you ubuntu-devel mail about research into packaging. You asked for pointers, so I'm mailing you one. Many of the points you are looking for in a package manager are covered by Nix. Nix is a distro independent package manager that is very different from the standard package managers. Instead of putting all applications in /usr, each version of each package is stored in a dedicated folder. It is possible to have many versions of packages and different users can use different versions. Nix works on ubuntu and many other operating systems today. (Also, there is an OS that uses Nix as the main package manager, NixOS.) Packages are built from Nix expressions, which is a simple functional language. A Nix expression describes everything that goes into a package build action (a “derivation”): other packages, sources, the build script, environment variables for the build script, etc. Nix tries very hard to ensure that Nix expressions are deterministic: building a Nix expression twice should yield the same result. The Nix community on #nix is very active and helpful. To avoid dependencies between apps, you can group nix expressions into one package. Installing a package means unpacking and setting some symlinks. Switching between versions is just a matter of setting symlinks once both versions are unpacked. Any user can install any package for themselves. A huge amount of research including PhD thesis went into creating Nix. https://nixos.org/nix/ Cheers, Jos -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: App installer design: click packages
On 2013-05-08 11:14, Colin Watson wrote: There've been many discussions of late about the challenges of scaling app installation to a very large number of apps, including making app packages much easier to automatically audit and sandbox, and making them safer and quicker to install without the possibility of system-level breakage. This is needed to make it quicker and easier for app developers to get their software into Ubuntu. There are several pieces to this (including a good deal of prior discussion in https://wiki.ubuntu.com/AppDevUploadProcess), but it seems clear that one of those is some kind of simplified packaging format which we can offer to third-party application developers. Hi, 0install developer here. I'd be interested to know more about what extra features 0install would need to be useful to you. As Matthias said, creating yet another packaging system would be disappointing, especially as your goals seem to match ours exactly. This is not aimed at changing packages that are already part of the Ubuntu archive; for the most part our existing system works well for those, and they tend to have non-trivial dependency structures. We'll continue to use dpkg and apt for building the Ubuntu operating system, syncing with Debian, and so on. There's no point developing a packaging system for apps and making it have the full panoply of features needed for the Ubuntu archive: it'd just be second-system-effect on top of our current packaging system. So the scope of what I've been considering is purely leaf apps built on a fixed "base system", which in the case of the initial target of the Ubuntu phone/tablet work would be the run-time part of the Ubuntu SDK. The radically-reduced dependency structure means that most apps will be Ubuntu-SDK-specific to start with, although I'd like to make sure that the package format design includes enough support up-front to allow this to be useful for other platforms that define suitable base system profiles in future. So, at Steve Langasek's request, I've been putting together a proof of concept of a low-level app package installer and packaging format. Highlights of what it can do so far are: * no dependencies between apps; single implicit dependency on the base system by way of a Click-Base-System field 0install packages can depend on debs if they want to, so if you create a virtual deb package representing the Ubuntu base system then you could just depend on that (e.g. "requires Ubuntu >= 13.4"). If you want to be cross platform, it would make more sense to specify individual dependencies, though. * installs each app to an entirely separate directory Yes. * entirely declarative: maintainer scripts are forbidden Yes. * base package manager overhead, i.e. the time required to install a trivial package containing a single small file, is about 0.15 seconds on a newish x86 laptop and about 0.6 seconds on a Nexus 7 (and that's with the current prototype implementation in Python; a later implementation could be in C and would then be faster still) 0.109 seconds here on my laptop (including the time to download the archive from Apache running on localhost; normally network delays will dominate, of course). * not limited to installing as root, although there may be similar constraints elsewhere to ensure that apps can't edit their own code at run-time Same. If system-wide sharing is enabled then downloads go under /var/cache and so apps cannot edit their own code. Without system-wide sharing, the files are stored in the user's home directory. They're read-only, but Linux doesn't currently provide us with a way to stop an app running as a user from changing that user's files. We're looking into using the new user namespaces support for sandboxing. * packages built by feeding the intended output directory tree to a simple Python tool, plus a manifest.json file Same, but the metadata is XML, not JSON. * building packages requires only the Python standard library, with the intent that it should be possible to build these packages quite easily on non-Ubuntu or even non-Linux systems Same. 0install packages can run across Linux, OS X, Windows, Unix, etc (assuming the packaged software can too). * binary packaging format sufficiently similar to existing one that we could add support to higher-level tools with minimal effort Package format is .tar.bz2 (other similar compressed archive formats, such as .zip are also supported). * strawman design for hooks into system packages, which will be entirely declarative from the app's point of view What does this mean? * unit-tested from the start Of course. Obvious items I still need to work on: * produce a strawman hooks implementation with some real worked examples * integrate (or demonstrate how to integrate) the container isolation properties worked on elsewhere Would be intereste
