Re: App installer design: click packages
On 2013-05-14 14:37, Colin Watson wrote: On Thu, May 09, 2013 at 09:41:26AM +0100, Thomas Leonard wrote: On 2013-05-08 11:14, Colin Watson wrote: So, at Steve Langasek's request, I've been putting together a proof of concept of a low-level app package installer and packaging format. Highlights of what it can do so far are: * no dependencies between apps; single implicit dependency on the base system by way of a Click-Base-System field 0install packages can depend on debs if they want to, so if you create a virtual deb package representing the Ubuntu base system then you could just depend on that (e.g. requires Ubuntu = 13.4). If you want to be cross platform, it would make more sense to specify individual dependencies, though. This is a deliberate constraint to avoid interaction with the (large) system package database, rather than a feature that was too hard to implement. To avoid speed problems, when 0install queries the dpkg database, it caches the results here: ~/.cache/0install.net/injector/dpkg-status.cache This is a text file with lines of the form name=versionTabarch. It invalidates the cache if the mtime of /var/lib/dpkg/status changes. In general, 0install should be fast as long as the dpkg database isn't changing frequently (and if it is, those updates will be causing far more slowdown than 0install's querying of it on the next app update). Is it easy to disable this feature in 0install and do something more like the system that Listaller and click-package use of depending on system frameworks declared in text files? The code for distribution-specific packages is in zeroinstall/injector/distro.py. Change get_host_distribution() to return Distribution() rather than DebianDistribution() if you want to completely ignore distribution packages. Of course, that means that existing packages won't work unless all their (declared) dependencies as available as 0install packages too. If a 0install package doesn't depend on anything which could be provided by the distribution, then dpkg will never be queried. However, many existing programs do depend on such things. For example, Python and Java are usually provided by distribution packages, so all existing Python and Java programs will cause a check of whether their VM is available natively. You could provide an alternative to DebianDistribution that gets its information from a text file, but you'd have to update the text file somehow based on the dpkg database, which is basically what 0install is doing anyway. (I suppose it's not critical; the primary requirement is bulletproof reliability, and performance is merely an important secondary requirement.) * base package manager overhead, i.e. the time required to install a trivial package containing a single small file, is about 0.15 seconds on a newish x86 laptop and about 0.6 seconds on a Nexus 7 (and that's with the current prototype implementation in Python; a later implementation could be in C and would then be faster still) 0.109 seconds here on my laptop (including the time to download the archive from Apache running on localhost; normally network delays will dominate, of course). debs install pretty quickly on my laptop too, but querying the package database is going to be slow on typical phone hardware, so I'd still like to make sure we avoid that. * not limited to installing as root, although there may be similar constraints elsewhere to ensure that apps can't edit their own code at run-time Same. If system-wide sharing is enabled then downloads go under /var/cache and so apps cannot edit their own code. Without system-wide sharing, the files are stored in the user's home directory. They're read-only, but Linux doesn't currently provide us with a way to stop an app running as a user from changing that user's files. We're looking into using the new user namespaces support for sandboxing. I think we will have some degree of sharing but not to the extent where one user can force another to upgrade, so we'll be running the installer as a separate user. We share the package data, but not the metadata saying which version is current. So each user updates on their own schedule, but when it comes to download time, they may find the version they want is already in the system cache because some other user downloaded it already. Files in the system cache are owned by the zeroinst user. We don't do any downloading of files as the zeroinst user (we used to, but it increases the attack surface a lot and operations such as using mirrors and cancelling shared downloads are tricky). Instead, the user's 0install process downloads the package itself and passes it to zeroinst, which verifies that its contents match its name (a secure hash) and stores it under that hash (e.g. as /var/cache/0install.net/implementations/sha256=e4f835.../). * strawman design for hooks into system
Re: App installer design: click packages
Hello,
You asked for pointers, so here is my idea for the Click Package
definition file. I think since originally Click Packages will be
targeting the Ubuntu SDK, and thus apps will be using a lot of QML, the
Click Package definition file should use a format similar to QML, but
not necessarily actually QML and the Qt classes to use the file in
C++/whatever, since it would not be used for graphics. I certainly
would find this much simpler then XML and more powerful than a
key/value table, like the .desktop file format. To define an
application, one might use something like this:
import ClickPackaging 1.0
Application {
id: app
name: Hello World!
pkg: org.example.helloworld
frameworks: [
Ubuntu = 1.0
]
}
To make things simpler and have a uniform file format, you could even
add in support for generating desktop files, DBus services, push
notifications, etc:
// Import the DesktopLauncher service definition, which will handle
generating the launcher
import DesktopLauncher 1.0
// Inside the Application object
DesktopLauncher {
name: app.name
exec: app.exec // app.exec could be generated by the
Application def object to point to the app's executable, where ever it
is
icon: icon
...
}
If the application definition got too big, or one had a big service
definition, here are two ideas for how to load it from a seperate file:
Application {
services: [ desktoplauncher.qml, ... ]
// OR
Service { file: desktoplauncher.qml }
}
When the packaging system builds the package, it would have each object
then build its part of the package, so then the DesktopLauncher would
generate the *.desktop information out of the object's properties.
In a more recent message, somebody mentioned being able to install
things like push notification definitions even when the push
notifications service is itself installed. Using my file format
suggestion, the only requirement would be that the push notifications
service definition (or any required service's definition) would need to
be installed to build the push definition for the package.
Other than this, Click Packages sounds awesome for developers and I'm
looking forward to packaging apps using it.
Blessings,
Michael Spencer
--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: App installer design: click packages
Hello. New to the thread and the forum. Already sent a message directly to cjwatson, I won't repost here. Just wanted to raise the issue of LGPL compliance, which is not possible for commercial applications in the proposed system. The dependency system on the debian packaging system is what allows linux desktop apps to be LGPL compliant, but the new model does not allow shared library packages or apps that depend on other packages. Bundling an LGPL library with an app is only possible (and debatably so at that), if the app is itself LGPL/GPL. This is my understanding based on personal experience and legal counsel while delivering commercial enterprise applications. Sometimes, even the LGPL library is dynamically linked, a commercial license is needed when the library must be bundled with the installer (for example, to allow offline installs). In order to allow LGPL compliant applications, your packaging system must at least do the following (as always, I am not a lawyer, consult a professional, etc.. ): 1) Applications and their LGPL library dependencies must be packaged and distributed separately. Bundling is the same as static linking for the purposes of LGPL compliance, even if the user can theoretically replace the dynamic library after your app is installed. You may often see this particular point made explicit by the LGPL author. Having separate package downloads works. Bundling the .so file in your application package does not work. 2) Applications must link to the LGPL library dynamically. 3) Users must be able to either modify the existing shared library binaries, or install a new version in such a way that your application will use it instead of the original. I suggested an alternative to the way the desktop system does it, in that we could allow shared library packages, but have them self contained like apps, and let the user dynamically switch between shared libraries on a per app basis, including their own custom builds. I have no idea how technically feasible this would be, however, or whether it would actually qualify as compliance with the LGPL, but it's the only thing I could come up with short of allowing apps to download additional shared libraries after install and load then dynamically, which of course would be a security risk. Regards, Jeremy On Tue, May 14, 2013 at 10:47 PM, Barry Warsaw ba...@ubuntu.com wrote: On May 14, 2013, at 02:37 PM, Colin Watson wrote: Barry spent some time looking at 0install, and it wasn't too bad a fit but we would still need to solve many of the same system integration problems. Is there a list of these issues somewhere? Barry Warsaw would have to speak to that, I think. But mainly, there's no way for the system to declare any kind of integration facilities at all, as far as I can see, so we'd have to solve that problem anyway. What I think Colin is getting at here is that any system is going to have to deal with declaring the files that allow an app to interoperate with system services. Think along the lines of dbus .service files, .desktop files, etc. These will be things we might know about right now (such as the above), but they maybe be additional integration points in the future. Let's say for example, some future system service comes with a new version of the SDK or OS image, that allows apps to define push notifications, by including a .push file that has to get installed somewhere to be registered. The app itself will only declare that it has a myapp.push, and you won't have anything like a maintainer script to install myapp.push. Instead, the system's push notifier will have to be responsible for registering a plugin with the installer system to say okay, now when you see .push files, put them here and do this something to register it. You don't want the logic for that in the installer, and you don't want it in the app. You want it as part of the system push notifier. This latter will register a trigger that gets run whenever the installer sees that an app has declared a .push file *and* you want it to run over all app .push files when the system push notifier is installed. In the latter case, imagine that you have a dozen apps that support push notifications installed on your phone, but the system push notifier hasn't been installed yet. Once it is, all your already installed push-enabled apps will Just Work. This all means that the installer must have plugins for handling new functionality and the declarative manifest must be extensible, i.e. the installer will ignore extensions that it doesn't (yet) have a registered plugin for. I didn't get far enough on 0install (or don't remember ;) whether it has that capability. I will certainly concede that the rest of 0install is very flexible and has a usefully pluggable design; you seem to have done a good job on it and Barry spoke well of it. It just seems that there's a bit of a mismatch in that it's kind of over
Re: App installer design: click packages
What if we made nearly every LGPL library of consequence a part of the Ubuntu SDK? On 05/14/2013 08:22 PM, Jeremy Bell wrote: Hello. New to the thread and the forum. Already sent a message directly to cjwatson, I won't repost here. Just wanted to raise the issue of LGPL compliance, which is not possible for commercial applications in the proposed system. The dependency system on the debian packaging system is what allows linux desktop apps to be LGPL compliant, but the new model does not allow shared library packages or apps that depend on other packages. Bundling an LGPL library with an app is only possible (and debatably so at that), if the app is itself LGPL/GPL. This is my understanding based on personal experience and legal counsel while delivering commercial enterprise applications. Sometimes, even the LGPL library is dynamically linked, a commercial license is needed when the library must be bundled with the installer (for example, to allow offline installs). In order to allow LGPL compliant applications, your packaging system must at least do the following (as always, I am not a lawyer, consult a professional, etc.. ): 1) Applications and their LGPL library dependencies must be packaged and distributed separately. Bundling is the same as static linking for the purposes of LGPL compliance, even if the user can theoretically replace the dynamic library after your app is installed. You may often see this particular point made explicit by the LGPL author. Having separate package downloads works. Bundling the .so file in your application package does not work. 2) Applications must link to the LGPL library dynamically. 3) Users must be able to either modify the existing shared library binaries, or install a new version in such a way that your application will use it instead of the original. I suggested an alternative to the way the desktop system does it, in that we could allow shared library packages, but have them self contained like apps, and let the user dynamically switch between shared libraries on a per app basis, including their own custom builds. I have no idea how technically feasible this would be, however, or whether it would actually qualify as compliance with the LGPL, but it's the only thing I could come up with short of allowing apps to download additional shared libraries after install and load then dynamically, which of course would be a security risk. Regards, Jeremy On Tue, May 14, 2013 at 10:47 PM, Barry Warsaw ba...@ubuntu.com mailto:ba...@ubuntu.com wrote: On May 14, 2013, at 02:37 PM, Colin Watson wrote: Barry spent some time looking at 0install, and it wasn't too bad a fit but we would still need to solve many of the same system integration problems. Is there a list of these issues somewhere? Barry Warsaw would have to speak to that, I think. But mainly, there's no way for the system to declare any kind of integration facilities at all, as far as I can see, so we'd have to solve that problem anyway. What I think Colin is getting at here is that any system is going to have to deal with declaring the files that allow an app to interoperate with system services. Think along the lines of dbus .service files, .desktop files, etc. These will be things we might know about right now (such as the above), but they maybe be additional integration points in the future. Let's say for example, some future system service comes with a new version of the SDK or OS image, that allows apps to define push notifications, by including a .push file that has to get installed somewhere to be registered. The app itself will only declare that it has a myapp.push, and you won't have anything like a maintainer script to install myapp.push. Instead, the system's push notifier will have to be responsible for registering a plugin with the installer system to say okay, now when you see .push files, put them here and do this something to register it. You don't want the logic for that in the installer, and you don't want it in the app. You want it as part of the system push notifier. This latter will register a trigger that gets run whenever the installer sees that an app has declared a .push file *and* you want it to run over all app .push files when the system push notifier is installed. In the latter case, imagine that you have a dozen apps that support push notifications installed on your phone, but the system push notifier hasn't been installed yet. Once it is, all your already installed push-enabled apps will Just Work. This all means that the installer must have plugins for handling new functionality and the declarative manifest must be extensible, i.e. the installer will ignore
Re: App installer design: click packages: 0install
[ I thought I sent this already, but I didn't see it appear, so resending a slightly updated version ] On 2013-05-15 03:47, Barry Warsaw wrote: On May 14, 2013, at 02:37 PM, Colin Watson wrote: Barry spent some time looking at 0install, and it wasn't too bad a fit but we would still need to solve many of the same system integration problems. Is there a list of these issues somewhere? Barry Warsaw would have to speak to that, I think. But mainly, there's no way for the system to declare any kind of integration facilities at all, as far as I can see, so we'd have to solve that problem anyway. What I think Colin is getting at here is that any system is going to have to deal with declaring the files that allow an app to interoperate with system services. Think along the lines of dbus .service files, .desktop files, etc. These will be things we might know about right now (such as the above), but they maybe be additional integration points in the future. So the idea here is that packages can't specify their own .desktop or .service files directly, since that would allow them to escape any sandboxing or otherwise make suprising changes to the system behaviour. Instead, the installer creates these files in response to user actions. For example, when a user adds a program using the 0desktop GUI, 0desktop creates a .desktop file to launch the application. Hints in the metadata help it do this (for example, the needs-terminal/ element indicates that the .desktop file should launch it via the user's preferred terminal emulator). Since the format is XML, custom elements and attributes can be added for new cases without changing 0install itself. But this is just my personal preference. You are always free to scan the installed apps' metadata looking for hooks, if you feel that won't create security issues. For example, if the user tries to open a file, you could scan the installed apps to see which can handle that MIME type. But you'd have to confirm the choice with the user, at least the first time, since otherwise any random application could register itself to open any file. Let's say for example, some future system service comes with a new version of the SDK or OS image, that allows apps to define push notifications, by including a .push file that has to get installed somewhere to be registered. The app itself will only declare that it has a myapp.push, and you won't have anything like a maintainer script to install myapp.push. Instead, the system's push notifier will have to be responsible for registering a plugin with the installer system to say okay, now when you see .push files, put them here and do this something to register it. You don't want the logic for that in the installer, and you don't want it in the app. You want it as part of the system push notifier. This latter will register a trigger that gets run whenever the installer sees that an app has declared a .push file *and* you want it to run over all app .push files when the system push notifier is installed. In the latter case, imagine that you have a dozen apps that support push notifications installed on your phone, but the system push notifier hasn't been installed yet. Once it is, all your already installed push-enabled apps will Just Work. This all means that the installer must have plugins for handling new functionality and the declarative manifest must be extensible, i.e. the installer will ignore extensions that it doesn't (yet) have a registered plugin for. If you wanted to avoid plugins, you could define a fixed naming scheme, so that if APP provides HOOK then the installer creates an entry in e.g. config/hooks/HOOK/APP.hook Then the push service could look in config/hooks/push/*.hook to find all apps that support it quickly. You could then just add all hooks, whether the service using them is installed or not. For hooks with security implications, it might be better to wait until the user requests the feature. An annoyance of Android is that you can only grant permissions at install time, so every application asks for every possible permission up front. I didn't get far enough on 0install (or don't remember ;) whether it has that capability. I will certainly concede that the rest of 0install is very flexible and has a usefully pluggable design; you seem to have done a good job on it and Barry spoke well of it. It just seems that there's a bit of a mismatch in that it's kind of over the top in terms of flexibility for what we need and seems to be entirely missing some other things we need, as above; and it would be a lot of code to rewrite in some other language if (as is possible, though not imminent) we decide that Python is just too heavyweight to run routinely on the Ubuntu phone. On the installing side, you're always going to be limited by the network rather than the CPU. You might save a little power with a more efficient language, but I doubt it's very much. For
RE: App installer design: click packages
A quick update and status report from the PackageKit and Listaller project for those interested: http://blog.tenstral.net/2013/05/packagekit-appstream-and-listaller-a-status-report.html -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: App installer design: click packages
Hello everyone, I'm new here and this may have been mentioned before, but have we thought about doing what android does where each app is a different 'user'? That way apps could be part of different levels of security (i.e. this app isn't part of the camera group, but this one is). -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: App installer design: click packages
On 2013-05-15 03:47, Barry Warsaw wrote: On May 14, 2013, at 02:37 PM, Colin Watson wrote: Barry spent some time looking at 0install, and it wasn't too bad a fit but we would still need to solve many of the same system integration problems. Is there a list of these issues somewhere? Barry Warsaw would have to speak to that, I think. But mainly, there's no way for the system to declare any kind of integration facilities at all, as far as I can see, so we'd have to solve that problem anyway. What I think Colin is getting at here is that any system is going to have to deal with declaring the files that allow an app to interoperate with system services. Think along the lines of dbus .service files, .desktop files, etc. These will be things we might know about right now (such as the above), but they maybe be additional integration points in the future. So the idea here is that packages can't specify their own .desktop or .service files directly, since that would allow them to escape any sandboxing or otherwise make suprising changes to the system behaviour. Instead, the installer creates these files in response to user actions. For example, when a user adds a program using the 0desktop GUI, 0desktop creates a .desktop file to launch the application. Hints in the metadata help it do this (for example, the needs-terminal/ element indicates that the .desktop file should launch it via the user's preferred terminal emulator). Since the format is XML, custom elements and attributes can be added for new cases without changing 0install itself. Ideally, there would also be a 0service command for adding services, e.g. 0service add myapache http://.../apache would create a systemd or Upstart file which would run Apache (under a new myapache user, with data in /srv/myapache, etc). I was looking at Juju a while back as a possible way to do this, but I feel it's too heavy for managing simple local services (requires at least one whole VM per service). As an example of what I'm trying to avoid: a while ago I did apt-get install timidity because I wanted to be able to do timidity file.midi to play some music. Later, I discovered it had also installed some kind of music server process, running as root :-( 0install is designed to make sure that can't happen without the user's explicit request. But this is just my personal preference. You are always free to scan the installed apps' metadata looking for hooks, if you feel that won't create security issues. Let's say for example, some future system service comes with a new version of the SDK or OS image, that allows apps to define push notifications, by including a .push file that has to get installed somewhere to be registered. The app itself will only declare that it has a myapp.push, and you won't have anything like a maintainer script to install myapp.push. Instead, the system's push notifier will have to be responsible for registering a plugin with the installer system to say okay, now when you see .push files, put them here and do this something to register it. You don't want the logic for that in the installer, and you don't want it in the app. You want it as part of the system push notifier. This latter will register a trigger that gets run whenever the installer sees that an app has declared a .push file *and* you want it to run over all app .push files when the system push notifier is installed. In the latter case, imagine that you have a dozen apps that support push notifications installed on your phone, but the system push notifier hasn't been installed yet. Once it is, all your already installed push-enabled apps will Just Work. This all means that the installer must have plugins for handling new functionality and the declarative manifest must be extensible, i.e. the installer will ignore extensions that it doesn't (yet) have a registered plugin for. If you wanted to avoid plugins, you could define a fixed naming scheme, so that if APP provides HOOK then the installer creates an entry in e.g. config/hooks/HOOK/APP.hook Then the push service could look in config/hooks/push/*.hook to find all apps that support it. You might as well add all hooks, whether the service using them is installed or not. I didn't get far enough on 0install (or don't remember ;) whether it has that capability. I will certainly concede that the rest of 0install is very flexible and has a usefully pluggable design; you seem to have done a good job on it and Barry spoke well of it. It just seems that there's a bit of a mismatch in that it's kind of over the top in terms of flexibility for what we need and seems to be entirely missing some other things we need, as above; and it would be a lot of code to rewrite in some other language if (as is possible, though not imminent) we decide that Python is just too heavyweight to run routinely on the Ubuntu phone. On the installing side, you're always going to be limited by the network
Re: App installer design: click packages
I would like to introduce some additional thoughts about some possible useful features of new package manager. Idea: autorun application after install with information about path where it was executed. Explanation: Let's assume we want to install some application with large additional downloads, for example we want to install *some digital games distribution system* SDGDS ;). Suppose, new manager packages with extension *.upk (Ubuntu package). We have DVD with game on it (mounted to /media/NewGame2013) and Setup_NewGame2013.upk. User run this *.upk with single click/touch and then this package gets installed, it contains our SDGS, not a game. If we already have SDGDS - it's updated or not, depends on version. Just right after user clicked *.upk, package manager attempts (if there are such option in declarative meta-info inside upk) to run this software with path /media/NewGame2013 as an argument. Thus our SDGDS gets information that we have media with some useful content (game) and begin to install it into special path, which is granted by declarative meta-info of upk. User always gets simple always-ready installation. let's imagine, we can deploy ZeroInstall and many other thing instead supposed SDGDS (ok, let's call it Steam), and it will install it's additional contents into special limited (by meta-info) path or somehow else specified paths... Such approach we can see in some proprietary OS, where is has some installer and every installation begin with checking it's version, then update, then software get's selected (components) and finally installed. Colin, what do you think about this? Just some meta-option in package and software runs with path of place where it's attempted to be installed. Sorry for my english, I can describe this in other words if you think this idea reasonable. P.S. Excuse me for my previous post, it was too emotional... -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: App installer design: click packages
On Thu, May 09, 2013 at 02:47:47PM +0400, Sergey Shambir wrote: 1. Declarative manifest Main benefit is ability to handle this file automatically: index it, provide UI and either generate automatically from build system information or parse to get information back. In my mind, the main benefit is that we eliminate root-privileged maintainer scripts. The current situation with debian is that all package maintainers have the ability to write a postinst script that will run as root on your system, and this script can be in any language, without any form of controls on it. That kind of situation made sense in 1993 when there were only a few dozen package maintainers who all knew and trusted each other, but it really hasn't scaled. Just think: every package you've ever installed, every PPA you've enabled, you've implicitly given root access on your machine to the author of that package. Who are these people? Do you trust them? It's insane! The goal of a declarative manifest is to make it possible to install a package on the system which does not require unlimited arbitrary root code execution by random untrusted packages downloaded from the internet. -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: App installer design: click packages
On Mon, May 13, 2013 at 6:53 PM, Colin Watson cjwat...@ubuntu.com wrote: On Mon, May 13, 2013 at 01:56:43PM -0400, Marc Deslauriers wrote: On 13-05-13 11:05 AM, Alejandro J. Cura wrote: This thread assumes that packages need to be uncompressed and installed before usage, so I'd like to ask if there was any discussion re: using something like squashfs images as the distributed packages instead of a zip or tar-like file. This would mean that such downloaded images can be mounted read-only by whatever launches applications, using nosuid, nodev, and with the required uid, and then run immediately, instead of having to go thru a copy of files from the package to the storage, which slows down installation and usually requires double the storage space. That would mean we'd need to have a privileged helper to be able to mount application packages at application execution time. There are a lot of security implications of doing something like this, and I fear this would be a substantial attack surface. And even if we mounted them all just once at boot, (a) we'd still need to use root privilege to mount at application installation/upgrade time, (b) any bug in squashfs would now become an easy escalate-to-kernel vulnerability exploitable through the app store. Now, (b) is still true for dpkg/tar/etc., but tar is already assumed to take hostile input, and the relevant parts of dpkg are mainly shared with unprivileged operations such as 'dpkg -c' which are also assumed to take hostile input and have had a good deal of attention over the years; and even if there is a problem we can at least contain it to a less-privileged specialised 'software' user. Yes, any squashfs overflow sounds scary for this use case, so I'm convinced now that it's not such a good idea. Thanks for your replies! -- alecu -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: App installer design: click packages
This just seems to be a complex enough case that it's simply not a good target for this packaging system. I'm OK with that. As I said, this is not trying to be a complete replacement for debs, because if it tried to do that it would rapidly turn into a second system with a different set of flaws. Sad to hear that... Debs (and Rpms) is a reason why bug #1 will not be solved in foreseeable future. Thus, developers still needs to create their ugly or use 3rd-party installers which act on system or homepage without limitations because root/sudo password, users need do manual changing execute bit with entering sudo password which is not convenient and hard for regular users. Solution which I suggest is dependency-free and it only seems as complex, but actually just encapsulates your packages (and it's OK that they are read-only) into single one. It would be very convenient to have them in single place instead downloading and installing separate 5-10 packages of some software which have components, especially when some of them logically required. The only simple things required is to provide additional file extension for such pure-declarative installer and GUI itself (with possibility to install from CLI). Just imagine, developer will want to deploy Mono-based application. Such package can contain Mono runtime package + application package itself. What if Mono-runtime of _exact_ version already installed with your packaging system? Then just mark in installer GUI that this required component already installed. And what if other application will want same runtime of same version? No problems. What if version differs? Then it installs another one alongside with current. And Java? And Python, let's assume it's not preinstalled because we need 2.7 and Ubuntu provides 3.x, or it's old package which needs to be installed on Ubuntu 22.04 where only Python 4.x available... Let's don't repeat Android's errors, look at how Qt-based applications deployed for this platform. When user attempts to run application it prompts user to install huge additional component through market. And what if user out of network coverage or in expensive roaming network? Noting bad in dependencies itself, it's just bad Linux's implementation contains flaws: 1) Can't install under user, because files writing in system paths. 2) Can't install many versions of same library, because library space is flat. 3) Packages for APT/RPM does not contain necessary dependencies. Look at ZeroInstall - it's nice except dumb deploying experience of stand-alone network-free packages (absence of which - main drawback of Ubuntu and other Linux-system in part of software management), because we need to create package which contains non-declarative parts which should be run with password, because ZI not installed in Linux system by-default, and we need to set executable bit... Insane. Ideal solution: do users and developers at least same level of possibilities at least as in Mac OS X. It's not late to create such package management system, or at least PLEASE, don't say it will never be in Ubuntu. -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: App installer design: click packages
On May 14, 2013, at 02:37 PM, Colin Watson wrote: Barry spent some time looking at 0install, and it wasn't too bad a fit but we would still need to solve many of the same system integration problems. Is there a list of these issues somewhere? Barry Warsaw would have to speak to that, I think. But mainly, there's no way for the system to declare any kind of integration facilities at all, as far as I can see, so we'd have to solve that problem anyway. What I think Colin is getting at here is that any system is going to have to deal with declaring the files that allow an app to interoperate with system services. Think along the lines of dbus .service files, .desktop files, etc. These will be things we might know about right now (such as the above), but they maybe be additional integration points in the future. Let's say for example, some future system service comes with a new version of the SDK or OS image, that allows apps to define push notifications, by including a .push file that has to get installed somewhere to be registered. The app itself will only declare that it has a myapp.push, and you won't have anything like a maintainer script to install myapp.push. Instead, the system's push notifier will have to be responsible for registering a plugin with the installer system to say okay, now when you see .push files, put them here and do this something to register it. You don't want the logic for that in the installer, and you don't want it in the app. You want it as part of the system push notifier. This latter will register a trigger that gets run whenever the installer sees that an app has declared a .push file *and* you want it to run over all app .push files when the system push notifier is installed. In the latter case, imagine that you have a dozen apps that support push notifications installed on your phone, but the system push notifier hasn't been installed yet. Once it is, all your already installed push-enabled apps will Just Work. This all means that the installer must have plugins for handling new functionality and the declarative manifest must be extensible, i.e. the installer will ignore extensions that it doesn't (yet) have a registered plugin for. I didn't get far enough on 0install (or don't remember ;) whether it has that capability. I will certainly concede that the rest of 0install is very flexible and has a usefully pluggable design; you seem to have done a good job on it and Barry spoke well of it. It just seems that there's a bit of a mismatch in that it's kind of over the top in terms of flexibility for what we need and seems to be entirely missing some other things we need, as above; and it would be a lot of code to rewrite in some other language if (as is possible, though not imminent) we decide that Python is just too heavyweight to run routinely on the Ubuntu phone. On just few other points: I'm not really a big fan of XML formats, since they're not human consumable (readable or writable). Others may disagree, which is fine, but I don't have any interest in arguing about it. Probably not a deal breaker, since the XML files can always be generated, either by the 0install tools or something that takes a JSON or YAML as input, etc. I liked that it was fairly easy to wrap executables in security containers, but I didn't dig into this in a lot of detail. It seemed to have good security considerations. IIRC, there's no built-in support for uninstallation. We probably want something like an online-only search REST API for finding apps, and then downloading from the URL such a search gives us. Thus, there's no cleverness in the repository layout, nor on the device, for finding apps. Anyway, that's about all the additional stuff I've gleaned from my previously private experiments with 0install that hasn't already been discussed in this thread. -Barry P.S. If we *were* to accept that the installer could be written in Python, it would *have* to be Python 3. No more Python 2 on the phone, please! -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: App installer design: click packages
Hi, Thanks for efforts on simplifying package management! I'll wait for further information from UDS session, but first I want to provide some additional info that can help you in research. I'm QtCreator contibutor, worked on source code models and project management subsystems, including build systems support. Click packages have many decisions that can scare experienced Linux users, but are reasonable. So, for marketing purposes and in order to defend idea, please make more detailed overview of each design decision, which includes: 1. Comparison of various approaches in this concrete problem, with pros and cons. 2. Destroying of common myths associated with selected approach (like if dependencies will be eliminated, size of packages will grow significantly). Possible research agenda: 1. Which benefits brings declarative property-based app manifest? 2. Which benefits brings reversing build system - package manager interaction order? 3. Can qmake automatically create app manifest in near future? 4. Does size of real applications packages really grow with no-dependencies system with SDK (which includes Qt)? 5. Which issues are in Listaller and existing build systems (mostly in qmake)? 6. Which pros and cons have installing a) from root b) from user to $HOME c) from partially privilegied, but non-root user (similar to wwwrun or vboxuser)? 7. Which ways exist to support development model build on newer system, deploy to older? Let me throw first stone: 1. Declarative manifest Main benefit is ability to handle this file automatically: index it, provide UI and either generate automatically from build system information or parse to get information back. Build systems is an excellent example: XCode, Visual Studio and CodeBlocks IDEs have property-based build settings, and they can automatically edit such settings for years. Meanwhile, no one IDE can automatically edit Autotools or Premake files, CMake editing is implemented only in KDevelop and is buggy; editing qmake in QtCreator also works wrong in complex cases. 2. Reversing build system - package manager Typical project selects one build system, but targets to many platforms. At first, when package manager handles buildsystem, user should manually debug package manager scripts on each platform. At second, build system developers do not want to implement creating packages (because package manager designed to handle build system): https://bugreports.qt-project.org/browse/QBS-16. At third, this situation previously led to ugly design of make-based build systems: they are script-based and expose hooks to package manager with environment variables or configure options, but have no ability to easily use different build configurations (Debug, Release) in same time, and portability between platforms achieved with hacks or annoying debugging. Declarative build systems have no such problems. 4. Packages size on disk and in memory. QtCreator in Ubuntu depends on Qt and Botan libraries. It doesn't depend on Botan in upstream version, since it keeps Botan copy in sources. Most crossplatform apps keep dependencies set as small as possible, because Windows and Mac OS X have no dependency-based package system. So, size growth will be insignificant. 5a. Listaller issues Currently Listaller handles build system: http://listaller.tenstral.net/docs/doc/packaging.html. Meanwhile, buildsystem should handle listaller, and do it easily. No one developer wants to manually write and debug list of files to install. 5b. Qmake issues QMake have INSTALL variable that tracks installable files, but doesn't keep target bundle version, description and name. Since QtCreator have qmake evaluator, it can be patched to create or update manifest file, but some information should be typed by developer. 7. Build on newer system, deploy to older Listaller and LSB tools from Linux Foundation both use wrappers around gcc and special linker wrapper / special version of glibc respectively. Another way used in Mac OS X and iOS: XCode contains several SDKs, and application builds with only one in sysroot. -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: App installer design: click packages
Hello! I want to ask some questions about new packaging manager (which I awaited for many years...) and add some requests while it's not too late. Questions: 1) Which libraries you will consider as BASE system? I hope not just Ubuntu default installation will treated as base system in purposes of distributive- and OS- independence... This set of libraries and it's versions will change in time, new libraries will require some system calls which is not available in older kernel versions. Also if this library set is quite small, could they be placed in every package? 2) Do you plan to use components in packages? Some details required about this question. Let's assume we want to pack Qt SDK with your brand new packaging manager. It will consist of Qt Creator, Qt library itself, documentation and sources. And of course, it would be convenient to include GCC or some different compiler it same package. But we want set of packages without unconvenient dependencies... I suggest to add support of package of packages as it done in Mac OS X (also, Qt SDK installer also could do this). Back to example, single Qt SDK package with several components packages inside, and we can choose which to install. If we already have compiler, we click to not install this component. Of course, this package of packages on top of plain packages, which installing by single click... I hope to see restore functionality (if we want to reset package to defaults and restore missing files). 3) Will directory with contents of package include it's version? P.S. Excuse me for my worse english... -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: App installer design: click packages
mailto:cjwat...@ubuntu.com%20 Hi, mailto:cjwat...@ubuntu.com%20 I'm independent programmer not very connected over few years with Open Source software (like earlier). Your information about Click packages was cited in various IT portals and I have few comments: First - some time ago I was thinking about something similar, but it wasn't accepted by admins: http://brainstorm.ubuntu.com/idea/30533/ Second - I think, it would be nice to consider such scenario: there are left possibilities of installing current software and new format is built on top of it (like you propose), but it's a little more mature and during installation user is informed about few things: 1. Firewall network rules required by application (if application will try to get more after installation, system won't allow for it) 2. Task scheduler tasks (what will be run in cyclic manner and what will be run for example during system startup) 3. Permissions required by application (similar like in Android - for example access to concrete devices like GPS or Bluetooth) Later after installation it's possible to check it and revoke some of them (make them disabled). Format should allow for such things like providing kernel modules or access to concrete USB devices (it will allow for making for example players for concrete TV tuners) I was thinking also about limiting access for files by format (like in Windows) and user directories (in easy way - for details please refer to http://brainstorm.ubuntu.com/idea/30533/), but in Linux distribution it won't be probably possible. For now I think too, that there should be resolved problem of temp files (but rather on higher OS level) - OS should have ability of easy making RAM Disk for temp directory and temp files (separated of course) from Click apps should go there too. Regards, Marcin -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: App installer design: click packages
On Wed, May 08, 2013 at 11:14:08AM +0100, Colin Watson wrote: I should have the prototype ready for people to look at in time for UDS next week, lp:click-package - of course, no interfaces there should be relied on, and many are skeletal. and I'll ensure that there's a session scheduled for this. http://summit.ubuntu.com/uds-1305/meeting/21760/foundations-1305-click-package/ -- Colin Watson [cjwat...@ubuntu.com] -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: App installer design: click packages
On Mon, May 13, 2013 at 12:05:57PM -0300, Alejandro J. Cura wrote: Colin Watson wrote on 08/05/13 11:14: Is there anything else people can think of that a system like this needs to consider? This thread assumes that packages need to be uncompressed and installed before usage, so I'd like to ask if there was any discussion re: using something like squashfs images as the distributed packages instead of a zip or tar-like file. I know some of the other relevant systems do this, and I did think about it. The main downside I see is that I want apps to be always fully present on the filesystem; it doesn't make so much difference on a phablet, I suppose, but later on a desktop system people are going to want to look around manually, symlink to things, wrap things in scripts, etc. If we use a mounted-image scheme that then implies that we need to take care to mount all apps at boot time. I'm concerned about the possible effect on boot speed and reliability that that could have. I'm also not clear that there's been much research on whether this type of scheme has significant negative effects on things like elevator algorithm performance, caching, etc. The bottom line for me is that I'd like to keep things as simple as possible, and unpacking seems to me to be essentially simpler than a persistent-loop-mount scheme. Thanks, -- Colin Watson [cjwat...@ubuntu.com] -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: App installer design: click packages
On Mon, May 13, 2013 at 08:30:16PM +0400, Alex M wrote: 1) Which libraries you will consider as BASE system? I hope not just Ubuntu default installation will treated as base system in purposes of distributive- and OS- independence... This set of libraries and it's versions will change in time, new libraries will require some system calls which is not available in older kernel versions. At present, the target is the Ubuntu SDK, although this will be explicit in the packages which rely on it (Click-Framework: ubuntu-sdk-13.10 etc.). Also if this library set is quite small, could they be placed in every package? They will not be small. :-) 2) Do you plan to use components in packages? Some details required about this question. Let's assume we want to pack Qt SDK with your brand new packaging manager. It will consist of Qt Creator, Qt library itself, documentation and sources. And of course, it would be convenient to include GCC or some different compiler it same package. But we want set of packages without unconvenient dependencies... This just seems to be a complex enough case that it's simply not a good target for this packaging system. I'm OK with that. As I said, this is not trying to be a complete replacement for debs, because if it tried to do that it would rapidly turn into a second system with a different set of flaws. I hope to see restore functionality (if we want to reset package to defaults and restore missing files). The unpacked package directory will not be writable by the ordinary user or the application directly, and should therefore not change. As such, I don't think a restore feature should be necessary. 3) Will directory with contents of package include it's version? Yes. There'll be a symlink to the current version for convenience. -- Colin Watson [cjwat...@ubuntu.com] -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: App installer design: click packages
On 13-05-13 11:05 AM, Alejandro J. Cura wrote: Colin Watson wrote on 08/05/13 11:14: Is there anything else people can think of that a system like this needs to consider? This thread assumes that packages need to be uncompressed and installed before usage, so I'd like to ask if there was any discussion re: using something like squashfs images as the distributed packages instead of a zip or tar-like file. This would mean that such downloaded images can be mounted read-only by whatever launches applications, using nosuid, nodev, and with the required uid, and then run immediately, instead of having to go thru a copy of files from the package to the storage, which slows down installation and usually requires double the storage space. I'm surely missing some bits of the picture, so please flame me if that's the case. That would mean we'd need to have a privileged helper to be able to mount application packages at application execution time. There are a lot of security implications of doing something like this, and I fear this would be a substantial attack surface. Marc. -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: App installer design: click packages
Colin Watson wrote: On Mon, May 13, 2013 at 08:30:16PM +0400, Alex M wrote: I hope to see restore functionality (if we want to reset package to defaults and restore missing files). The unpacked package directory will not be writable by the ordinary user or the application directly, and should therefore not change. As such, I don't think a restore feature should be necessary. For those users who might need such a thing (e.g. partial loss of data from recoverably damaged secondary storage), wouldn't a reinstall of the relevant package work? dpkg is usually happy to overwrite files for the same package, and I don't see anything in the current code on launchpad that ought prevent this method working. I've certainly used the equivalent for normal packages in such situations in the past (although in practice, it is typically less painful to reinstall for significant recoverable damage). -- Emmet HIKORY -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: App installer design: click packages
On Tue, May 14, 2013 at 03:23:26AM +0900, Emmet Hikory wrote: Colin Watson wrote: On Mon, May 13, 2013 at 08:30:16PM +0400, Alex M wrote: I hope to see restore functionality (if we want to reset package to defaults and restore missing files). The unpacked package directory will not be writable by the ordinary user or the application directly, and should therefore not change. As such, I don't think a restore feature should be necessary. For those users who might need such a thing (e.g. partial loss of data from recoverably damaged secondary storage), wouldn't a reinstall of the relevant package work? Yes. Or we could just remove the unpacked package directory and install afresh, since it contains nothing irreplaceable. -- Colin Watson [cjwat...@ubuntu.com] -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: App installer design: click packages
On Mon, May 13, 2013 at 01:56:43PM -0400, Marc Deslauriers wrote: On 13-05-13 11:05 AM, Alejandro J. Cura wrote: This thread assumes that packages need to be uncompressed and installed before usage, so I'd like to ask if there was any discussion re: using something like squashfs images as the distributed packages instead of a zip or tar-like file. This would mean that such downloaded images can be mounted read-only by whatever launches applications, using nosuid, nodev, and with the required uid, and then run immediately, instead of having to go thru a copy of files from the package to the storage, which slows down installation and usually requires double the storage space. That would mean we'd need to have a privileged helper to be able to mount application packages at application execution time. There are a lot of security implications of doing something like this, and I fear this would be a substantial attack surface. And even if we mounted them all just once at boot, (a) we'd still need to use root privilege to mount at application installation/upgrade time, (b) any bug in squashfs would now become an easy escalate-to-kernel vulnerability exploitable through the app store. Now, (b) is still true for dpkg/tar/etc., but tar is already assumed to take hostile input, and the relevant parts of dpkg are mainly shared with unprivileged operations such as 'dpkg -c' which are also assumed to take hostile input and have had a good deal of attention over the years; and even if there is a problem we can at least contain it to a less-privileged specialised 'software' user. -- Colin Watson [cjwat...@ubuntu.com] -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: App installer design: click packages
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Colin Watson wrote on 08/05/13 11:14: ... So, at Steve Langasek's request, I've been putting together a proof of concept of a low-level app package installer and packaging format. Highlights of what it can do so far are: ... * installs each app to an entirely separate directory * entirely declarative: maintainer scripts are forbidden Do these two entail that an installation/update/removal can be interrupted at any point without corrupting the system? This is, as I understand it, a long-standing problem with deb/dpkg. A PC has a power outage or a kernel panic while a package is installing or updating, and the packaged software becomes unusable, and possibly stymies future package operations too. An interruption like this is much more likely to happen on a phone or tablet, because it's much more often on battery while you're installing something. ... To do, but not in this project: * package acquisition While designing session-installer error messages, I was told that apt returns localized error messages but not error codes, making it impractical for a UI wrapper to respond appropriately to different kinds of error (e.g. providing a Try Again button only when trying again might actually work). Avoiding this in any new system would be desirable. * indexing support and other frontend things (i.e. the app store) That will affect the package format. For example, each package providing metadata in multiple languages. ... Is there anything else people can think of that a system like this needs to consider? ... A system that aims for an order of magnitude more applications, than Debian or Ubuntu has packages, should not rely on every package having a unique name. For example, a package should not determine the name of the installation directory itself. Finally (and most bikesheddingly), a system initially aimed at phones and tablets should not be named after something you do with a mouse. Cheers - -- mpt -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlGMz9EACgkQ6PUxNfU6ecpWFgCfV9GhyHH95qe/1rZtWCJaNJ/6 3vUAnAzDJiALnaK1bRg525P0yyoomoQS =vMfc -END PGP SIGNATURE- -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: App installer design: click packages
hi, On Fr, 2013-05-10 at 11:45 +0100, Matthew Paul Thomas wrote: Finally (and most bikesheddingly), a system initially aimed at phones and tablets should not be named after something you do with a mouse. convergence ;) ciao oli signature.asc Description: This is a digitally signed message part -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: App installer design: click packages
Hi! 2013/5/9 Colin Watson cjwat...@ubuntu.com: On Wed, May 08, 2013 at 03:25:42PM +0200, Matthias Klumpp wrote: Disclaimer: As developer of Listaller, AppStream and PackageKit, I am of course biased on this topic ;-) Thanks for your mail. I'm looking over Listaller at the moment, and expect to have some more comments once I've done that. However, I'm going to be mostly offline from Friday to Sunday, so a partial reply in the meantime: Great! Please keep in mind that the project will undergo a transition to greatly simplyify dependency solving soon, with an option to restore and improve the 0install compatibility too. (there has been discussion about this with 0install people some time ago, but we did not find a working solution back then). The key values for Listaller are distribution-independence and integration (= everything which supports PackageKit can handle Listaller stuff too). 2013/5/8 Colin Watson cjwat...@ubuntu.com: * not limited to installing as root, although there may be similar constraints elsewhere to ensure that apps can't edit their own code at run-time Non-root installs into user's $HOME are implemented working in Listaller, but due to the tight integration into PackageKit, this feature should not be used. In practice, I don't actually intend to install packages as the ordinary calling user directly; as I indicated above, this would be a bad idea because it would permit apps to overwrite their own code. However, it's convenient for testing, and it may make sense to unpack apps as a special software user or similar to reduce the system's exposure to things like vulnerabilities in underlying unarchiving tools, which are rare but not unknown. (Or we could just hit it with AppArmor. I don't much care either way as long as we avoid the obvious flaw.) Yes, that is the rationale for allowing things like this :-) If there is a good way to lock 3rd-party applications with AppArmor, having a feature like this would definitely help. But maybe the sandbox will already provide enough security. * building packages requires only the Python standard library, with the intent that it should be possible to build these packages quite easily on non-Ubuntu or even non-Linux systems Hmm, why? If a non-Python application should be packaged, this feature would be lost, as people will have to crossbuild it anyway. And in order to build stuff for the upcoming Ubuntu products, people will need to install it (at least in a VM) anyway, for testing purposes. Remember that the main target of our project, at least for the time being, is Ubuntu Touch - phone/tablet apps. We expect a significant percentage of these to be written in pure QML or HTML5/CSS, and to make it possible to carry these over to desktop apps in time as well. I think it's important to minimise the barrier to entry to creating such apps; if somebody wants to create them on other systems they should be able to. For all QML/HTML5 stuff, this indeed makes sense. But since packages are not magic at all (often just compressed tarballs with some meta information), implementing that is easily possible. 0install proves that it even works for deploying stuff on MacOS, Windows and Linux. * binary packaging format sufficiently similar to existing one that we could add support to higher-level tools with minimal effort * strawman design for hooks into system packages, which will be entirely declarative from the app's point of view Is this going to be Ubuntu-specific? If not, you will have some trouble with that... I wasn't planning to put ubuntu in the hook paths or anything, but it is indeed up to the system to declare how it wants apps to integrate with it. I don't see a sensible way around that; it is entirely wrong to put system/app integration code in the installer itself, and any such approach would result in problems with the system and the installer getting out of sync further down the line. You mean the package-information with installer? Or do you mean the application which installs things? The package itself should not have any assumptions on how it will be installed on the system, and how the system would look like. The application which installs the package, however, is responsible for integrating the new application into the system, and also for negotiating with the native package manager about installed requirements. Both Listaller and 0install force applications to be developed in a way that they don't have any fixed assumptions on the system they are installed on. Listaller even contains many tools due to it's partial Autopackage origin which help to minimize and lower system dependencies. All of these features don't matter much if you want to use a 3rd-party package manager just on the Ubuntu Phone, because I assume that the phone software will have some kind of API/Platform level applications will be built for. But if you want to target the
Re: App installer design: click packages
Hello Colin, I read you ubuntu-devel mail about research into packaging. You asked for pointers, so I'm mailing you one. Many of the points you are looking for in a package manager are covered by Nix. Nix is a distro independent package manager that is very different from the standard package managers. Instead of putting all applications in /usr, each version of each package is stored in a dedicated folder. It is possible to have many versions of packages and different users can use different versions. Nix works on ubuntu and many other operating systems today. (Also, there is an OS that uses Nix as the main package manager, NixOS.) Packages are built from Nix expressions, which is a simple functional language. A Nix expression describes everything that goes into a package build action (a “derivation”): other packages, sources, the build script, environment variables for the build script, etc. Nix tries very hard to ensure that Nix expressions are deterministic: building a Nix expression twice should yield the same result. The Nix community on #nix is very active and helpful. To avoid dependencies between apps, you can group nix expressions into one package. Installing a package means unpacking and setting some symlinks. Switching between versions is just a matter of setting symlinks once both versions are unpacked. Any user can install any package for themselves. A huge amount of research including PhD thesis went into creating Nix. https://nixos.org/nix/ Cheers, Jos -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: App installer design: click packages
On 2013-05-08 11:14, Colin Watson wrote: There've been many discussions of late about the challenges of scaling app installation to a very large number of apps, including making app packages much easier to automatically audit and sandbox, and making them safer and quicker to install without the possibility of system-level breakage. This is needed to make it quicker and easier for app developers to get their software into Ubuntu. There are several pieces to this (including a good deal of prior discussion in https://wiki.ubuntu.com/AppDevUploadProcess), but it seems clear that one of those is some kind of simplified packaging format which we can offer to third-party application developers. Hi, 0install developer here. I'd be interested to know more about what extra features 0install would need to be useful to you. As Matthias said, creating yet another packaging system would be disappointing, especially as your goals seem to match ours exactly. This is not aimed at changing packages that are already part of the Ubuntu archive; for the most part our existing system works well for those, and they tend to have non-trivial dependency structures. We'll continue to use dpkg and apt for building the Ubuntu operating system, syncing with Debian, and so on. There's no point developing a packaging system for apps and making it have the full panoply of features needed for the Ubuntu archive: it'd just be second-system-effect on top of our current packaging system. So the scope of what I've been considering is purely leaf apps built on a fixed base system, which in the case of the initial target of the Ubuntu phone/tablet work would be the run-time part of the Ubuntu SDK. The radically-reduced dependency structure means that most apps will be Ubuntu-SDK-specific to start with, although I'd like to make sure that the package format design includes enough support up-front to allow this to be useful for other platforms that define suitable base system profiles in future. So, at Steve Langasek's request, I've been putting together a proof of concept of a low-level app package installer and packaging format. Highlights of what it can do so far are: * no dependencies between apps; single implicit dependency on the base system by way of a Click-Base-System field 0install packages can depend on debs if they want to, so if you create a virtual deb package representing the Ubuntu base system then you could just depend on that (e.g. requires Ubuntu = 13.4). If you want to be cross platform, it would make more sense to specify individual dependencies, though. * installs each app to an entirely separate directory Yes. * entirely declarative: maintainer scripts are forbidden Yes. * base package manager overhead, i.e. the time required to install a trivial package containing a single small file, is about 0.15 seconds on a newish x86 laptop and about 0.6 seconds on a Nexus 7 (and that's with the current prototype implementation in Python; a later implementation could be in C and would then be faster still) 0.109 seconds here on my laptop (including the time to download the archive from Apache running on localhost; normally network delays will dominate, of course). * not limited to installing as root, although there may be similar constraints elsewhere to ensure that apps can't edit their own code at run-time Same. If system-wide sharing is enabled then downloads go under /var/cache and so apps cannot edit their own code. Without system-wide sharing, the files are stored in the user's home directory. They're read-only, but Linux doesn't currently provide us with a way to stop an app running as a user from changing that user's files. We're looking into using the new user namespaces support for sandboxing. * packages built by feeding the intended output directory tree to a simple Python tool, plus a manifest.json file Same, but the metadata is XML, not JSON. * building packages requires only the Python standard library, with the intent that it should be possible to build these packages quite easily on non-Ubuntu or even non-Linux systems Same. 0install packages can run across Linux, OS X, Windows, Unix, etc (assuming the packaged software can too). * binary packaging format sufficiently similar to existing one that we could add support to higher-level tools with minimal effort Package format is .tar.bz2 (other similar compressed archive formats, such as .zip are also supported). * strawman design for hooks into system packages, which will be entirely declarative from the app's point of view What does this mean? * unit-tested from the start Of course. Obvious items I still need to work on: * produce a strawman hooks implementation with some real worked examples * integrate (or demonstrate how to integrate) the container isolation properties worked on elsewhere Would be interested to
Re: App installer design: click packages
On Wed, May 08, 2013 at 03:25:42PM +0200, Matthias Klumpp wrote: Disclaimer: As developer of Listaller, AppStream and PackageKit, I am of course biased on this topic ;-) Thanks for your mail. I'm looking over Listaller at the moment, and expect to have some more comments once I've done that. However, I'm going to be mostly offline from Friday to Sunday, so a partial reply in the meantime: I find it very sad that so many parties currently start reinventing the basic ideas of Listaller, instead of helping the Listaller project to improve, and kill the remaining flaws. I would prefer to keep the discussion technical rather than emotional. 2013/5/8 Colin Watson cjwat...@ubuntu.com: * not limited to installing as root, although there may be similar constraints elsewhere to ensure that apps can't edit their own code at run-time Non-root installs into user's $HOME are implemented working in Listaller, but due to the tight integration into PackageKit, this feature should not be used. In practice, I don't actually intend to install packages as the ordinary calling user directly; as I indicated above, this would be a bad idea because it would permit apps to overwrite their own code. However, it's convenient for testing, and it may make sense to unpack apps as a special software user or similar to reduce the system's exposure to things like vulnerabilities in underlying unarchiving tools, which are rare but not unknown. (Or we could just hit it with AppArmor. I don't much care either way as long as we avoid the obvious flaw.) * building packages requires only the Python standard library, with the intent that it should be possible to build these packages quite easily on non-Ubuntu or even non-Linux systems Hmm, why? If a non-Python application should be packaged, this feature would be lost, as people will have to crossbuild it anyway. And in order to build stuff for the upcoming Ubuntu products, people will need to install it (at least in a VM) anyway, for testing purposes. Remember that the main target of our project, at least for the time being, is Ubuntu Touch - phone/tablet apps. We expect a significant percentage of these to be written in pure QML or HTML5/CSS, and to make it possible to carry these over to desktop apps in time as well. I think it's important to minimise the barrier to entry to creating such apps; if somebody wants to create them on other systems they should be able to. Running a separate test device may well be an easier proposition than building on it. For example, you might have a physical device (Nexus 7 or whatever) that you can easily install things on; that doesn't mean you want to run your development environment there. I agree that building native apps will typically involve building in a clean Ubuntu environment. * binary packaging format sufficiently similar to existing one that we could add support to higher-level tools with minimal effort * strawman design for hooks into system packages, which will be entirely declarative from the app's point of view Is this going to be Ubuntu-specific? If not, you will have some trouble with that... I wasn't planning to put ubuntu in the hook paths or anything, but it is indeed up to the system to declare how it wants apps to integrate with it. I don't see a sensible way around that; it is entirely wrong to put system/app integration code in the installer itself, and any such approach would result in problems with the system and the installer getting out of sync further down the line. dpkg is right to defer all integration code to packages; the hook scheme I'm considering is mainly inspired by dpkg triggers. (We may or may not be able to call dpkg triggers directly; for performance it is vital to avoid having to actually open the dpkg database.) Footnote: Why not use one of the existing systems out there? We still might, but at present it's not clear that it would make a whole lot of difference. If it doesn't, why don't you use them? :P It's a trade-off between the benefit of using an existing system and the cost of twisting it around to fit what we actually need it to do. I'm not fundamentally opposed to using an existing system - I certainly understand why people want us to, and that there's a perception that Canonical unnecessarily does its own thing (you can look up my own commit record if you like) - but if there's a mismatch between our goals then it may end up being rather more trouble than it's worth. One thing I'm particularly concerned about is that a system that has tried to be very general may end up reinventing similar scaling and reliability problems as the packaging system we already have. (To be clear - hi Slashdot - I'm very fond of the dpkg/apt system and have put a great deal of effort into it over the last decade or so, both on my own time and for pay; I think it's excellent and have no desire to kill it. But I also recognise
App installer design: click packages
There've been many discussions of late about the challenges of scaling app installation to a very large number of apps, including making app packages much easier to automatically audit and sandbox, and making them safer and quicker to install without the possibility of system-level breakage. This is needed to make it quicker and easier for app developers to get their software into Ubuntu. There are several pieces to this (including a good deal of prior discussion in https://wiki.ubuntu.com/AppDevUploadProcess), but it seems clear that one of those is some kind of simplified packaging format which we can offer to third-party application developers. This is not aimed at changing packages that are already part of the Ubuntu archive; for the most part our existing system works well for those, and they tend to have non-trivial dependency structures. We'll continue to use dpkg and apt for building the Ubuntu operating system, syncing with Debian, and so on. There's no point developing a packaging system for apps and making it have the full panoply of features needed for the Ubuntu archive: it'd just be second-system-effect on top of our current packaging system. So the scope of what I've been considering is purely leaf apps built on a fixed base system, which in the case of the initial target of the Ubuntu phone/tablet work would be the run-time part of the Ubuntu SDK. The radically-reduced dependency structure means that most apps will be Ubuntu-SDK-specific to start with, although I'd like to make sure that the package format design includes enough support up-front to allow this to be useful for other platforms that define suitable base system profiles in future. So, at Steve Langasek's request, I've been putting together a proof of concept of a low-level app package installer and packaging format. Highlights of what it can do so far are: * no dependencies between apps; single implicit dependency on the base system by way of a Click-Base-System field * installs each app to an entirely separate directory * entirely declarative: maintainer scripts are forbidden * base package manager overhead, i.e. the time required to install a trivial package containing a single small file, is about 0.15 seconds on a newish x86 laptop and about 0.6 seconds on a Nexus 7 (and that's with the current prototype implementation in Python; a later implementation could be in C and would then be faster still) * not limited to installing as root, although there may be similar constraints elsewhere to ensure that apps can't edit their own code at run-time * packages built by feeding the intended output directory tree to a simple Python tool, plus a manifest.json file * building packages requires only the Python standard library, with the intent that it should be possible to build these packages quite easily on non-Ubuntu or even non-Linux systems * binary packaging format sufficiently similar to existing one that we could add support to higher-level tools with minimal effort * strawman design for hooks into system packages, which will be entirely declarative from the app's point of view * unit-tested from the start Obvious items I still need to work on: * produce a strawman hooks implementation with some real worked examples * integrate (or demonstrate how to integrate) the container isolation properties worked on elsewhere * Click-Base-System field is skeletally simple right now, and may need to be expanded to at least leave open the possibility of multiple flavours of base system (see also GNOME's profiles idea) * adjust unpack handling to avoid problems with project renames and name clashes, and to unpack each version into its own directory and flip symlinks to allow for multi-user independence * integrate into the Ubuntu SDK, as well as providing examples of how it can be integrated into other build systems too To do, but not in this project: * package acquisition * indexing support and other frontend things (i.e. the app store) * upload handling So far I've demonstrated to my own satisfaction that we can get acceptable performance and isolation characteristics with very little work; the installer part on its own is under 300 lines of Python right now, which I consider useful since bugginess is often proportional to code size. I should have the prototype ready for people to look at in time for UDS next week, and I'll ensure that there's a session scheduled for this. Is there anything else people can think of that a system like this needs to consider? Footnote: Why not use one of the existing systems out there? We still might, but at present it's not clear that it would make a whole lot of difference. Many of the existing app packaging systems are more of a reflection of the system they were built for than anything else. If you look at, say, Android's APK format, it's essentially a zip file with a manifest and some conventions about Dalvik
Re: App installer design: click packages
On May 08, 2013, at 11:14 AM, Colin Watson wrote: * Click-Base-System field is skeletally simple right now, and may need to be expanded to at least leave open the possibility of multiple flavours of base system (see also GNOME's profiles idea) We'll have a session at vUDS about image based updates: https://blueprints.launchpad.net/ubuntu/+spec/foundations-1305-image-based-updates There, we've defined a build number which defines the image version your device is currently at, so we can calculate where and how we can upgrade it. I suspect that the build number will be different than the CBS number, with the latter indicating the version of the SDK that the app depends on, and this will probably be different than device image version number. Still, I wanted to mention it in case there's overlap that I might be missing. -Barry -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: App installer design: click packages
Hi! Disclaimer: As developer of Listaller, AppStream and PackageKit, I am of course biased on this topic ;-) I find it very sad that so many parties currently start reinventing the basic ideas of Listaller, instead of helping the Listaller project to improve, and kill the remaining flaws. 2013/5/8 Colin Watson cjwat...@ubuntu.com: There've been many discussions of late about the challenges of scaling app installation to a very large number of apps, including making app packages much easier to automatically audit and sandbox, and making them safer and quicker to install without the possibility of system-level breakage. This is needed to make it quicker and easier for app developers to get their software into Ubuntu. There are several pieces to this (including a good deal of prior discussion in https://wiki.ubuntu.com/AppDevUploadProcess), but it seems clear that one of those is some kind of simplified packaging format which we can offer to third-party application developers. This is the rationale of Listaller, see http://listaller.tenstral.net/ So, at Steve Langasek's request, I've been putting together a proof of concept of a low-level app package installer and packaging format. Highlights of what it can do so far are: * no dependencies between apps; single implicit dependency on the base system by way of a Click-Base-System field Implemented in Listaller, but poorly working. The dependency-solver is currently being rewritten, with the upcoming KDE-Frameworks 5 and GNOME-Platform (aka GNOME-OS) in mind. * installs each app to an entirely separate directory Done in Listaller, but several shared resources can - well - easily be shared. * entirely declarative: maintainer scripts are forbidden Design decision for Listaller (resulted in controversial discussions, but finally everyone seems to be fine with it). * base package manager overhead, i.e. the time required to install a trivial package containing a single small file, is about 0.15 seconds on a newish x86 laptop and about 0.6 seconds on a Nexus 7 (and that's with the current prototype implementation in Python; a later implementation could be in C and would then be faster still) I did not (yet) do measurements on this, but as Listaller was rewritten in Vala some time ago, I expect it to be reasonably fast. * not limited to installing as root, although there may be similar constraints elsewhere to ensure that apps can't edit their own code at run-time Non-root installs into user's $HOME are implemented working in Listaller, but due to the tight integration into PackageKit, this feature should not be used. * packages built by feeding the intended output directory tree to a simple Python tool, plus a manifest.json file Listaller uses a depscan tool to auto-detect dependencies. Currently only works for binary shared libraries, due to lack of time and Python/Ruby/Perl knowledge to make a sane implementation for these languages. But the tool is designed with that in mind. Output is a textfile in Debian-control-file style. * building packages requires only the Python standard library, with the intent that it should be possible to build these packages quite easily on non-Ubuntu or even non-Linux systems Hmm, why? If a non-Python application should be packaged, this feature would be lost, as people will have to crossbuild it anyway. And in order to build stuff for the upcoming Ubuntu products, people will need to install it (at least in a VM) anyway, for testing purposes. * binary packaging format sufficiently similar to existing one that we could add support to higher-level tools with minimal effort * strawman design for hooks into system packages, which will be entirely declarative from the app's point of view Is this going to be Ubuntu-specific? If not, you will have some trouble with that... Listaller also offers update-sources for installed 3rd-party apps, a way to build binary apps using older library symbols and some cross-distro compatibility stuff, a tight integration with PackageKit (that might actually be a problem for Ubuntu...), which allows updates for these packages to be displayed in standard package management tools, alongside other software which was deployed using the system package manager. Footnote: Why not use one of the existing systems out there? We still might, but at present it's not clear that it would make a whole lot of difference. If it doesn't, why don't you use them? :P Many of the existing app packaging systems are more of a reflection of the system they were built for than anything else. If you look at, say, Android's APK format, it's essentially a zip file with a manifest and some conventions about Dalvik class installation and the like. Most of the other mobile app formats are similar. Things like Listaller might be a reasonable fit, but I'm worried about importing things like a full dependency solver into this: much
