Re: App installer design: click packages
On 2013-05-14 14:37, Colin Watson wrote:> On Thu, May 09, 2013 at 09:41:26AM +0100, Thomas Leonard wrote: >> On 2013-05-08 11:14, Colin Watson wrote: >>> So, at Steve Langasek's request, I've been putting together a proof of >>> concept of a low-level app package installer and packaging format. >>> Highlights of what it can do so far are: >>> >>> * no dependencies between apps; single implicit dependency on the base >>> system by way of a Click-Base-System field >> >> 0install packages can depend on debs if they want to, so if you >> create a virtual deb package representing the Ubuntu base system >> then you could just depend on that (e.g. "requires Ubuntu >= 13.4"). >> If you want to be cross platform, it would make more sense to >> specify individual dependencies, though. > > This is a deliberate constraint to avoid interaction with the (large) > system package database, rather than a feature that was too hard to > implement. To avoid speed problems, when 0install queries the dpkg database, it caches the results here: ~/.cache/0install.net/injector/dpkg-status.cache This is a text file with lines of the form "name=versionarch". It invalidates the cache if the mtime of /var/lib/dpkg/status changes. In general, 0install should be fast as long as the dpkg database isn't changing frequently (and if it is, those updates will be causing far more slowdown than 0install's querying of it on the next app update). > Is it easy to disable this feature in 0install and do > something more like the system that Listaller and click-package use of > depending on system frameworks declared in text files? The code for distribution-specific packages is in zeroinstall/injector/distro.py. Change get_host_distribution() to return Distribution() rather than DebianDistribution() if you want to completely ignore distribution packages. Of course, that means that existing packages won't work unless all their (declared) dependencies as available as 0install packages too. If a 0install package doesn't depend on anything which could be provided by the distribution, then dpkg will never be queried. However, many existing programs do depend on such things. For example, Python and Java are usually provided by distribution packages, so all existing Python and Java programs will cause a check of whether their VM is available natively. You could provide an alternative to DebianDistribution that gets its information from a text file, but you'd have to update the text file somehow based on the dpkg database, which is basically what 0install is doing anyway. > (I suppose it's not critical; the primary requirement is bulletproof > reliability, and performance is merely an important secondary > requirement.) > >>> * base package manager overhead, i.e. the time required to install a >>> trivial package containing a single small file, is about 0.15 seconds >>> on a newish x86 laptop and about 0.6 seconds on a Nexus 7 (and that's >>> with the current prototype implementation in Python; a later >>> implementation could be in C and would then be faster still) >> >> 0.109 seconds here on my laptop (including the time to download the >> archive from Apache running on localhost; normally network delays >> will dominate, of course). > > debs install pretty quickly on my laptop too, but querying the package > database is going to be slow on typical phone hardware, so I'd still > like to make sure we avoid that. > >>> * not limited to installing as root, although there may be similar >>> constraints elsewhere to ensure that apps can't edit their own code >>> at run-time >> >> Same. If system-wide sharing is enabled then downloads go under >> /var/cache and so apps cannot edit their own code. Without >> system-wide sharing, the files are stored in the user's home >> directory. They're read-only, but Linux doesn't currently provide us >> with a way to stop an app running as a user from changing that >> user's files. We're looking into using the new user namespaces >> support for sandboxing. > > I think we will have some degree of sharing but not to the extent where > one user can force another to upgrade, so we'll be running the installer > as a separate user. We share the package data, but not the metadata saying which version is current. So each user updates on their own schedule, but when it comes to download time, they may find the version they want is already in the system cache because some other user downloaded it already. Files in the system cache are owned by the &q
Re: App installer design: click packages
On 2013-05-15 03:47, Barry Warsaw wrote: On May 14, 2013, at 02:37 PM, Colin Watson wrote: Barry spent some time looking at 0install, and it wasn't too bad a fit but we would still need to solve many of the same system integration problems. Is there a list of these issues somewhere? Barry Warsaw would have to speak to that, I think. But mainly, there's no way for the system to declare any kind of integration facilities at all, as far as I can see, so we'd have to solve that problem anyway. What I think Colin is getting at here is that any system is going to have to deal with declaring the files that allow an app to interoperate with system services. Think along the lines of dbus .service files, .desktop files, etc. These will be things we might know about right now (such as the above), but they maybe be additional integration points in the future. So the idea here is that packages can't specify their own .desktop or .service files directly, since that would allow them to escape any sandboxing or otherwise make suprising changes to the system behaviour. Instead, the installer creates these files in response to user actions. For example, when a user adds a program using the 0desktop GUI, 0desktop creates a .desktop file to launch the application. Hints in the metadata help it do this (for example, the element indicates that the .desktop file should launch it via the user's preferred terminal emulator). Since the format is XML, custom elements and attributes can be added for new cases without changing 0install itself. Ideally, there would also be a 0service command for adding services, e.g. 0service add myapache http://.../apache would create a systemd or Upstart file which would run Apache (under a new "myapache" user, with data in /srv/myapache, etc). I was looking at Juju a while back as a possible way to do this, but I feel it's too heavy for managing simple local services (requires at least one whole VM per service). As an example of what I'm trying to avoid: a while ago I did "apt-get install timidity" because I wanted to be able to do "timidity file.midi" to play some music. Later, I discovered it had also installed some kind of music server process, running as root :-( 0install is designed to make sure that can't happen without the user's explicit request. But this is just my personal preference. You are always free to scan the installed apps' metadata looking for hooks, if you feel that won't create security issues. Let's say for example, some future system service comes with a new version of the SDK or OS image, that allows apps to define push notifications, by including a .push file that has to get installed somewhere to be registered. The app itself will only declare that it has a myapp.push, and you won't have anything like a maintainer script to install myapp.push. Instead, the system's push notifier will have to be responsible for registering a plugin with the installer system to say "okay, now when you see .push files, put them here and do this something to register it." You don't want the logic for that in the installer, and you don't want it in the app. You want it as part of the system push notifier. This latter will register a trigger that gets run whenever the installer sees that an app has declared a .push file *and* you want it to run over all app .push files when the system push notifier is installed. In the latter case, imagine that you have a dozen apps that support push notifications installed on your phone, but the system push notifier hasn't been installed yet. Once it is, all your already installed push-enabled apps will Just Work. This all means that the installer must have plugins for handling new functionality and the declarative manifest must be extensible, i.e. the installer will ignore extensions that it doesn't (yet) have a registered plugin for. If you wanted to avoid plugins, you could define a fixed naming scheme, so that if APP provides HOOK then the installer creates an entry in e.g. /hooks/HOOK/APP.hook Then the push service could look in /hooks/push/*.hook to find all apps that support it. You might as well add all hooks, whether the service using them is installed or not. I didn't get far enough on 0install (or don't remember ;) whether it has that capability. I will certainly concede that the rest of 0install is very flexible and has a usefully pluggable design; you seem to have done a good job on it and Barry spoke well of it. It just seems that there's a bit of a mismatch in that it's kind of over the top in terms of flexibility for what we need and seems to be entirely missing some other things we need, as above; and it would be a lot of code to rewrite in some other language if (as is possible, though not imminent) we decide that Python is just too heavyweight to run routinely on the Ubuntu phone. On the installing side, you're always going to be limited by the network rather than the CP
Re: App installer design: click packages: 0install
ou might save a little power with a more efficient language, but I doubt it's very much. For running apps, you'd probably want to replace the Python with something faster. When you do 0install run foo it loads .config/0install.net/apps/foo/selections.xml, sets the environment variables and launches the app. That doesn't need to be Python and ideally it wouldn't be. We use Python because it makes 0install easy to use on a wide range of platforms, even where distribution support is missing. One final thought here: you don't necessarily need to run the solver on the phone anyway. You could run a central store which selects the default versions of each program and library and just serves those selections to the phone. e.g. A developer provides a 0install app called http://.../someapp. You want this in your store. On your server, you do: 0install download http://.../someapp --xml > someapp-selections.xml When a user wants this app on their phone, you just transfer someapp-selections.xml to the phone, which then requests any missing packages (from your mirror server probably). From time to time, you update the selections on your server. Here, 0install provides an automated way to get software from developers to your store rather than all the way to the phone. You'd probably still want 0install on the phone in case the user wanted to run 3rd-party apps or use different versions, but it wouldn't be used in the default case. It would nice if there were an easy way to swich between Ubuntu-recommended versions and upstream/custom versions. On just few other points: I'm not really a big fan of XML formats, since they're not human consumable (readable or writable). Others may disagree, which is fine, but I don't have any interest in arguing about it. Probably not a deal breaker, since the XML files can always be generated, either by the 0install tools or something that takes a JSON or YAML as input, etc. One of the nice things about XML is that it can be extended in an obvious way by third-party tools, e.g. to define new attributes without messing up existing tools. I liked that it was fairly easy to wrap executables in "security" containers, but I didn't dig into this in a lot of detail. It seemed to have good security considerations. IIRC, there's no built-in support for uninstallation. 0install destroy APP will remove the launchers for APP[*], but doesn't automatically evict the software from the cache. "0store manage" lets you remove things individually, but for a phone you'd want the system to GC packages after an app has been destroyed. I think the Sugar desktop environment does this, and the Windows version has something similar too. It's pretty easy (~/.config/0install.net/apps/*/selections.xml lists all the packages and libraries you need, so delete anything that's in the cache but not referenced there), but so far I've never run out of disk space so I haven't implemented it. [*] 0desktop predates this and currently has a separate destroy feature. At some point, "0desktop" and "0install add" will be unified. We probably want something like an online-only search REST API for finding apps, and then downloading from the URL such a search gives us. Thus, there's no cleverness in the repository layout, nor on the device, for finding apps. "0install search QUERY" does that (since 0install 2.1), but the search engine at the server end isn't very clever at the moment. You'd probably want to run your own back-end. Anyway, that's about all the additional stuff I've gleaned from my previously private experiments with 0install that hasn't already been discussed in this thread. -Barry P.S. If we *were* to accept that the installer could be written in Python, it would *have* to be Python 3. No more Python 2 on the phone, please! 0install runs with Python 2 or Python 3. The Debian package defaults to using Python 2 because it's faster and the API is more stable (i.e. less chance of breakage when a new Python release comes out). BTW, if using Python 3, it also (optionally) supports the new Tulip event loop, in case you want to drop the GObject dependency. -- Dr Thomas Leonardhttp://0install.net/ GPG: 9242 9807 C985 3C07 44A6 8B9A AE07 8280 59A5 3CC1 GPG: DA98 25AE CAD0 8975 7CDA BD8E 0713 3F96 CA74 D8BA -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: App installer design: click packages
rate (or demonstrate how to integrate) the container isolation properties worked on elsewhere Would be interested to hear more about this. * Click-Base-System field is skeletally simple right now, and may need to be expanded to at least leave open the possibility of multiple flavours of base system (see also GNOME's profiles idea) * adjust unpack handling to avoid problems with project renames and name clashes, and to unpack each version into its own directory and flip symlinks to allow for multi-user independence * integrate into the Ubuntu SDK, as well as providing examples of how it can be integrated into other build systems too To do, but not in this project: * package acquisition * indexing support and other frontend things (i.e. the app store) * upload handling So far I've demonstrated to my own satisfaction that we can get acceptable performance and isolation characteristics with very little work; the installer part on its own is under 300 lines of Python right now, which I consider useful since bugginess is often proportional to code size. I should have the prototype ready for people to look at in time for UDS next week, and I'll ensure that there's a session scheduled for this. Is there anything else people can think of that a system like this needs to consider? Package signing, key distribution and trust, mirroring, dependencies, roll-back, parallel installation of stable and testing versions, checking for updates, source packages, ...? ;-) Footnote: Why not use one of the existing systems out there? We still might, but at present it's not clear that it would make a whole lot of difference. Many of the existing app packaging systems are more of a reflection of the system they were built for than anything else. If you look at, say, Android's APK format, it's essentially a zip file with a manifest and some conventions about Dalvik class installation and the like. Most of the other mobile app formats are similar. Things like Listaller might be a reasonable fit, but I'm worried about importing things like a full dependency solver into this: much though I love dependency-based packaging systems, they necessarily involve scanning the system packaging database at some point and I would much prefer app packages to be as independent of that as possible, mainly for performance but also to be as bulletproof as possible. GNOME app bundles appear to have a rather different sandboxing model than we need (e.g. enforced restrictions on dependencies), we probably want better system integration, and perhaps other differences; but that said I like the coarse-grained profile model of dependencies (and unknowingly imitated it), and it might make sense to share some profile names. > Barry spent some time looking at 0install, and it wasn't too bad a fit but we would still need to solve many of the same system integration problems. Is there a list of these issues somewhere? The proof of concept I wrote also isn't entirely new code. It's tiny due to using .deb as a container format (minus maintainer scripts, full dependencies, etc.), so I get to save effort by using dpkg to unpack things, which leaves us room to selectively use more of its features in future if we want to. This is quite a compelling approach to me for various reasons. I could be persuaded to look at something else, but I don't think there's actually much code to share at the packaging format level anyway, as it's so simple; quite possibly more at the sandboxing level, but that's really a separate layer. Furthermore, the priority for this system at present is for Ubuntu phone/tablet app packages, which haven't actually been built yet. As a result, I'm less worried about fragmentation than I might otherwise be. I would like to make sure that it's usable elsewhere too, although somebody would need to work out what that means in terms of base system profiles. Although I'm very familiar with dpkg and traditional package management, I only started looking at this side of things a couple of weeks ago, so I'm sure I'm behind on research. Pointers welcome. Regards, -- Dr Thomas Leonardhttp://0install.net/ GPG: 9242 9807 C985 3C07 44A6 8B9A AE07 8280 59A5 3CC1 GPG: DA98 25AE CAD0 8975 7CDA BD8E 0713 3F96 CA74 D8BA -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
