Re: App installer design: click packages

[Scott Ritchie]

What if we made nearly every LGPL library of consequence a part of the
Ubuntu SDK?

On 05/14/2013 08:22 PM, Jeremy Bell wrote:
> Hello. New to the thread and the forum. Already sent a message directly
> to cjwatson, I won't repost here. Just wanted to raise the issue of LGPL
> compliance, which is not possible for commercial applications in the
> proposed system. The dependency system on the debian packaging system is
> what allows linux desktop apps to be LGPL compliant, but the new model
> does not allow shared library packages or apps that depend on other
> packages. Bundling an LGPL library with an app is only possible (and
> debatably so at that), if the app is itself LGPL/GPL. This is my
> understanding based on personal experience and legal counsel while
> delivering commercial enterprise applications. Sometimes, even the LGPL
> library is dynamically linked, a commercial license is needed when the
> library must be bundled with the installer (for example, to allow
> offline installs).
> 
> In order to allow LGPL compliant applications, your packaging system
> must at least do the following (as always, I am not a lawyer, consult a
> professional, etc.. ):
> 1) Applications and their LGPL library dependencies must be packaged and
> distributed separately. Bundling is the same as static linking for the
> purposes of LGPL compliance, even if the user can theoretically replace
> the dynamic library after your app is installed. You may often see this
> particular point made explicit by the LGPL author. Having separate
> package downloads works. Bundling the .so file in your application
> package does not work.
> 2) Applications must link to the LGPL library dynamically.
> 3) Users must be able to either modify the existing shared library
> binaries, or install a new version in such a way that your application
> will use it instead of the original.
> 
> I suggested an alternative to the way the desktop system does it, in
> that we could allow shared library packages, but have them self
> contained like apps, and let the user dynamically "switch" between
> shared libraries on a per app basis, including their own custom builds.
> I have no idea how technically feasible this would be, however, or
> whether it would actually qualify as compliance with the LGPL, but it's
> the only thing I could come up with short of allowing apps to download
> additional shared libraries after install and load then dynamically,
> which of course would be a security risk.
> 
> Regards,
> Jeremy
> 
> 
> 
> On Tue, May 14, 2013 at 10:47 PM, Barry Warsaw  > wrote:
> 
> On May 14, 2013, at 02:37 PM, Colin Watson wrote:
> 
> >> > Barry spent
> >> >some time looking at 0install, and it wasn't too bad a fit but
> we would
> >> >still need to solve many of the same system integration problems.
> >>
> >> Is there a list of these issues somewhere?
> >
> >Barry Warsaw would have to speak to that, I think.  But mainly, there's
> >no way for the system to declare any kind of integration facilities at
> >all, as far as I can see, so we'd have to solve that problem anyway.
> 
> What I think Colin is getting at here is that any system is going to
> have to
> deal with declaring the files that allow an app to interoperate with
> system
> services.  Think along the lines of dbus .service files, .desktop
> files, etc.
> These will be things we might know about right now (such as the
> above), but
> they maybe be additional integration points in the future.
> 
> Let's say for example, some future system service comes with a new
> version of
> the SDK or OS image, that allows apps to define push notifications, by
> including a .push file that has to get installed somewhere to be
> registered.
> The app itself will only declare that it has a myapp.push, and you
> won't have
> anything like a maintainer script to install myapp.push.  Instead, the
> system's push notifier will have to be responsible for registering a
> plugin
> with the installer system to say "okay, now when you see .push
> files, put them
> here and do this something to register it."
> 
> You don't want the logic for that in the installer, and you don't
> want it in
> the app.  You want it as part of the system push notifier.  This
> latter will
> register a trigger that gets run whenever the installer sees that an
> app has
> declared a .push file *and* you want it to run over all app .push
> files when
> the system push notifier is installed.  In the latter case, imagine
> that you
> have a dozen apps that support push notifications installed on your
> phone, but
> the system push notifier hasn't been installed yet.  Once it is, all
> your
> already installed push-enabled apps will Just Work.
> 
> This all means that the installer must have plugins for handling n

Re: App installer design: click packages

[Jeremy Bell]

Hello. New to the thread and the forum. Already sent a message directly to
cjwatson, I won't repost here. Just wanted to raise the issue of LGPL
compliance, which is not possible for commercial applications in the
proposed system. The dependency system on the debian packaging system is
what allows linux desktop apps to be LGPL compliant, but the new model does
not allow shared library packages or apps that depend on other packages.
Bundling an LGPL library with an app is only possible (and debatably so at
that), if the app is itself LGPL/GPL. This is my understanding based on
personal experience and legal counsel while delivering commercial
enterprise applications. Sometimes, even the LGPL library is dynamically
linked, a commercial license is needed when the library must be bundled
with the installer (for example, to allow offline installs).

In order to allow LGPL compliant applications, your packaging system must
at least do the following (as always, I am not a lawyer, consult a
professional, etc.. ):
1) Applications and their LGPL library dependencies must be packaged and
distributed separately. Bundling is the same as static linking for the
purposes of LGPL compliance, even if the user can theoretically replace the
dynamic library after your app is installed. You may often see this
particular point made explicit by the LGPL author. Having separate package
downloads works. Bundling the .so file in your application package does not
work.
2) Applications must link to the LGPL library dynamically.
3) Users must be able to either modify the existing shared library
binaries, or install a new version in such a way that your application will
use it instead of the original.

I suggested an alternative to the way the desktop system does it, in that
we could allow shared library packages, but have them self contained like
apps, and let the user dynamically "switch" between shared libraries on a
per app basis, including their own custom builds. I have no idea how
technically feasible this would be, however, or whether it would actually
qualify as compliance with the LGPL, but it's the only thing I could come
up with short of allowing apps to download additional shared libraries
after install and load then dynamically, which of course would be a
security risk.

Regards,
Jeremy



On Tue, May 14, 2013 at 10:47 PM, Barry Warsaw  wrote:

> On May 14, 2013, at 02:37 PM, Colin Watson wrote:
>
> >> > Barry spent
> >> >some time looking at 0install, and it wasn't too bad a fit but we would
> >> >still need to solve many of the same system integration problems.
> >>
> >> Is there a list of these issues somewhere?
> >
> >Barry Warsaw would have to speak to that, I think.  But mainly, there's
> >no way for the system to declare any kind of integration facilities at
> >all, as far as I can see, so we'd have to solve that problem anyway.
>
> What I think Colin is getting at here is that any system is going to have
> to
> deal with declaring the files that allow an app to interoperate with system
> services.  Think along the lines of dbus .service files, .desktop files,
> etc.
> These will be things we might know about right now (such as the above), but
> they maybe be additional integration points in the future.
>
> Let's say for example, some future system service comes with a new version
> of
> the SDK or OS image, that allows apps to define push notifications, by
> including a .push file that has to get installed somewhere to be
> registered.
> The app itself will only declare that it has a myapp.push, and you won't
> have
> anything like a maintainer script to install myapp.push.  Instead, the
> system's push notifier will have to be responsible for registering a plugin
> with the installer system to say "okay, now when you see .push files, put
> them
> here and do this something to register it."
>
> You don't want the logic for that in the installer, and you don't want it
> in
> the app.  You want it as part of the system push notifier.  This latter
> will
> register a trigger that gets run whenever the installer sees that an app
> has
> declared a .push file *and* you want it to run over all app .push files
> when
> the system push notifier is installed.  In the latter case, imagine that
> you
> have a dozen apps that support push notifications installed on your phone,
> but
> the system push notifier hasn't been installed yet.  Once it is, all your
> already installed push-enabled apps will Just Work.
>
> This all means that the installer must have plugins for handling new
> functionality and the declarative manifest must be extensible, i.e. the
> installer will ignore extensions that it doesn't (yet) have a registered
> plugin for.
>
> I didn't get far enough on 0install (or don't remember ;) whether it has
> that
> capability.
>
> >I will certainly concede that the rest of 0install is very flexible and
> >has a usefully pluggable design; you seem to have done a good job on it
> >and Barry spoke well of it.  It just s

Re: App installer design: click packages

[The Spencers]

Hello,

You asked for pointers, so here is my idea for the Click Package 
definition file. I think since originally Click Packages will be 
targeting the Ubuntu SDK, and thus apps will be using a lot of QML, the 
Click Package definition file should use a format similar to QML, but 
not necessarily actually QML and the Qt classes to use the file in 
C++/whatever, since it would not be used for graphics. I certainly 
would find this much simpler then XML and more powerful than a 
key/value table, like the .desktop file format. To define an 
application, one might use something like this:


import ClickPackaging 1.0

Application {
        id: app
        name: "Hello World!"
        pkg: "org.example.helloworld"

        frameworks: [
                "Ubuntu >= 1.0"
        ]
}

To make things simpler and have a uniform file format, you could even 
add in support for generating desktop files, DBus services, push 
notifications, etc:


// Import the DesktopLauncher service definition, which will handle 
generating the launcher

import DesktopLauncher 1.0

// Inside the Application object
DesktopLauncher {
        name: app.name
        exec: app.exec // app.exec could be generated by the 
Application def object to point to the app's executable, where ever it 
is

        icon: ""
        ...
}

If the application definition got too big, or one had a big service 
definition, here are two ideas for how to load it from a seperate file:


Application {
        services: [ "desktoplauncher.qml", ... ] 
        // OR
        Service { file: "desktoplauncher.qml" }
}

When the packaging system builds the package, it would have each object 
then build its part of the package, so then the DesktopLauncher would 
generate the *.desktop information out of the object's properties.


In a more recent message, somebody mentioned being able to install 
things like push notification definitions even when the push 
notifications service is itself installed. Using my file format 
suggestion, the only requirement would be that the push notifications 
service definition (or any required service's definition) would need to 
be installed to build the push definition for the package.


Other than this, Click Packages sounds awesome for developers and I'm 
looking forward to packaging apps using it.


Blessings,
Michael Spencer
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: App installer design: click packages

[Thomas Leonard]

On 2013-05-14 14:37, Colin Watson wrote:> On Thu, May 09, 2013 at 
09:41:26AM +0100, Thomas Leonard wrote:

>> On 2013-05-08 11:14, Colin Watson wrote:
>>> So, at Steve Langasek's request, I've been putting together a proof of
>>> concept of a low-level app package installer and packaging format.
>>> Highlights of what it can do so far are:
>>>
>>>   * no dependencies between apps; single implicit dependency on the 
base

>>> system by way of a Click-Base-System field
>>
>> 0install packages can depend on debs if they want to, so if you
>> create a virtual deb package representing the Ubuntu base system
>> then you could just depend on that (e.g. "requires Ubuntu >= 13.4").
>> If you want to be cross platform, it would make more sense to
>> specify individual dependencies, though.
>
> This is a deliberate constraint to avoid interaction with the (large)
> system package database, rather than a feature that was too hard to
> implement.

To avoid speed problems, when 0install queries the dpkg database, it 
caches the results here:


  ~/.cache/0install.net/injector/dpkg-status.cache

This is a text file with lines of the form "name=versionarch".
It invalidates the cache if the mtime of /var/lib/dpkg/status changes.

In general, 0install should be fast as long as the dpkg database isn't 
changing frequently (and if it is, those updates will be causing far 
more slowdown than 0install's querying of it on the next app update).


> Is it easy to disable this feature in 0install and do
> something more like the system that Listaller and click-package use of
> depending on system frameworks declared in text files?

The code for distribution-specific packages is in 
zeroinstall/injector/distro.py. Change get_host_distribution() to return 
Distribution() rather than DebianDistribution() if you want to 
completely ignore distribution packages. Of course, that means that 
existing packages won't work unless all their (declared) dependencies as 
available as 0install packages too.


If a 0install package doesn't depend on anything which could be provided 
by the distribution, then dpkg will never be queried. However, many 
existing programs do depend on such things. For example, Python and Java 
are usually provided by distribution packages, so all existing Python 
and Java programs will cause a check of whether their VM is available 
natively.


You could provide an alternative to DebianDistribution that gets its 
information from a text file, but you'd have to update the text file 
somehow based on the dpkg database, which is basically what 0install is 
doing anyway.


> (I suppose it's not critical; the primary requirement is bulletproof
> reliability, and performance is merely an important secondary
> requirement.)
>
>>>   * base package manager overhead, i.e. the time required to install a
>>> trivial package containing a single small file, is about 0.15 
seconds
>>> on a newish x86 laptop and about 0.6 seconds on a Nexus 7 (and 
that's

>>> with the current prototype implementation in Python; a later
>>> implementation could be in C and would then be faster still)
>>
>> 0.109 seconds here on my laptop (including the time to download the
>> archive from Apache running on localhost; normally network delays
>> will dominate, of course).
>
> debs install pretty quickly on my laptop too, but querying the package
> database is going to be slow on typical phone hardware, so I'd still
> like to make sure we avoid that.
>
>>>   * not limited to installing as root, although there may be similar
>>> constraints elsewhere to ensure that apps can't edit their own code
>>> at run-time
>>
>> Same. If system-wide sharing is enabled then downloads go under
>> /var/cache and so apps cannot edit their own code. Without
>> system-wide sharing, the files are stored in the user's home
>> directory. They're read-only, but Linux doesn't currently provide us
>> with a way to stop an app running as a user from changing that
>> user's files. We're looking into using the new user namespaces
>> support for sandboxing.
>
> I think we will have some degree of sharing but not to the extent where
> one user can force another to upgrade, so we'll be running the installer
> as a separate user.

We share the package data, but not the metadata saying which version is 
current. So each user updates on their own schedule, but when it comes 
to download time, they may find the version they want is already in the 
system cache because some other user downloaded it already.


Files in the system cache are owned by the "zeroinst" user. We don't do 
any downloading of files as the "zeroinst" user (we used to, but it 
increases the attack surface a lot and operations such as using mirrors 
and cancelling shared downloads are tricky). Instead, the user's 
0install process downloads the package itself and passes it to 
"zeroinst", which verifies that its contents match its name (a secure 
hash) and stores it under that hash (e.g. a

Re: App installer design: click packages

[Thomas Leonard]

On 2013-05-15 03:47, Barry Warsaw wrote:

On May 14, 2013, at 02:37 PM, Colin Watson wrote:


Barry spent
some time looking at 0install, and it wasn't too bad a fit but we would
still need to solve many of the same system integration problems.


Is there a list of these issues somewhere?


Barry Warsaw would have to speak to that, I think.  But mainly, there's
no way for the system to declare any kind of integration facilities at
all, as far as I can see, so we'd have to solve that problem anyway.


What I think Colin is getting at here is that any system is going to have to
deal with declaring the files that allow an app to interoperate with system
services.  Think along the lines of dbus .service files, .desktop files, etc.
These will be things we might know about right now (such as the above), but
they maybe be additional integration points in the future.


So the idea here is that packages can't specify their own .desktop or 
.service files directly, since that would allow them to escape any 
sandboxing or otherwise make suprising changes to the system behaviour. 
Instead, the installer creates these files in response to user actions.


For example, when a user adds a program using the 0desktop GUI, 0desktop 
creates a .desktop file to launch the application. Hints in the metadata 
help it do this (for example, the  element indicates 
that the .desktop file should launch it via the user's preferred 
terminal emulator). Since the format is XML, custom elements and 
attributes can be added for new cases without changing 0install itself.


Ideally, there would also be a 0service command for adding services, e.g.

  0service add myapache http://.../apache

would create a systemd or Upstart file which would run Apache (under a 
new "myapache" user, with data in /srv/myapache, etc). I was looking at 
Juju a while back as a possible way to do this, but I feel it's too 
heavy for managing simple local services (requires at least one whole VM 
per service).


As an example of what I'm trying to avoid: a while ago I did "apt-get 
install timidity" because I wanted to be able to do "timidity file.midi" 
to play some music. Later, I discovered it had also installed some kind 
of music server process, running as root :-( 0install is designed to 
make sure that can't happen without the user's explicit request.


But this is just my personal preference. You are always free to scan the 
installed apps' metadata looking for hooks, if you feel that won't 
create security issues.



Let's say for example, some future system service comes with a new version of
the SDK or OS image, that allows apps to define push notifications, by
including a .push file that has to get installed somewhere to be registered.
The app itself will only declare that it has a myapp.push, and you won't have
anything like a maintainer script to install myapp.push.  Instead, the
system's push notifier will have to be responsible for registering a plugin
with the installer system to say "okay, now when you see .push files, put them
here and do this something to register it."

You don't want the logic for that in the installer, and you don't want it in
the app.  You want it as part of the system push notifier.  This latter will
register a trigger that gets run whenever the installer sees that an app has
declared a .push file *and* you want it to run over all app .push files when
the system push notifier is installed.  In the latter case, imagine that you
have a dozen apps that support push notifications installed on your phone, but
the system push notifier hasn't been installed yet.  Once it is, all your
already installed push-enabled apps will Just Work.

This all means that the installer must have plugins for handling new
functionality and the declarative manifest must be extensible, i.e. the
installer will ignore extensions that it doesn't (yet) have a registered
plugin for.


If you wanted to avoid plugins, you could define a fixed naming scheme, 
so that if APP provides HOOK then the installer creates an entry in e.g.


  /hooks/HOOK/APP.hook

Then the push service could look in /hooks/push/*.hook to find 
all apps that support it. You might as well add all hooks, whether the 
service using them is installed or not.



I didn't get far enough on 0install (or don't remember ;) whether it has that
capability.


I will certainly concede that the rest of 0install is very flexible and
has a usefully pluggable design; you seem to have done a good job on it
and Barry spoke well of it.  It just seems that there's a bit of a
mismatch in that it's kind of over the top in terms of flexibility for
what we need and seems to be entirely missing some other things we need,
as above; and it would be a lot of code to rewrite in some other
language if (as is possible, though not imminent) we decide that Python
is just too heavyweight to run routinely on the Ubuntu phone.


On the installing side, you're always going to be limited by the network 
rather than the CP

Re: App installer design: click packages: 0install

[Thomas Leonard]

[ I thought I sent this already, but I didn't see it appear, so 
resending a slightly updated version ]


On 2013-05-15 03:47, Barry Warsaw wrote:

On May 14, 2013, at 02:37 PM, Colin Watson wrote:


Barry spent
some time looking at 0install, and it wasn't too bad a fit but we would
still need to solve many of the same system integration problems.


Is there a list of these issues somewhere?


Barry Warsaw would have to speak to that, I think.  But mainly, there's
no way for the system to declare any kind of integration facilities at
all, as far as I can see, so we'd have to solve that problem anyway.


What I think Colin is getting at here is that any system is going to have to
deal with declaring the files that allow an app to interoperate with system
services.  Think along the lines of dbus .service files, .desktop files, etc.
These will be things we might know about right now (such as the above), but
they maybe be additional integration points in the future.


So the idea here is that packages can't specify their own .desktop or 
.service files directly, since that would allow them to escape any 
sandboxing or otherwise make suprising changes to the system behaviour. 
Instead, the installer creates these files in response to user actions.


For example, when a user adds a program using the 0desktop GUI, 0desktop 
creates a .desktop file to launch the application. Hints in the metadata 
help it do this (for example, the  element indicates 
that the .desktop file should launch it via the user's preferred 
terminal emulator). Since the format is XML, custom elements and 
attributes can be added for new cases without changing 0install itself.


But this is just my personal preference. You are always free to scan the 
installed apps' metadata looking for hooks, if you feel that won't 
create security issues.


For example, if the user tries to open a file, you could scan the 
installed apps to see which can handle that MIME type. But you'd have to 
confirm the choice with the user, at least the first time, since 
otherwise any random application could register itself to open any file.



Let's say for example, some future system service comes with a new version of
the SDK or OS image, that allows apps to define push notifications, by
including a .push file that has to get installed somewhere to be registered.
The app itself will only declare that it has a myapp.push, and you won't have
anything like a maintainer script to install myapp.push.  Instead, the
system's push notifier will have to be responsible for registering a plugin
with the installer system to say "okay, now when you see .push files, put them
here and do this something to register it."

You don't want the logic for that in the installer, and you don't want it in
the app.  You want it as part of the system push notifier.  This latter will
register a trigger that gets run whenever the installer sees that an app has
declared a .push file *and* you want it to run over all app .push files when
the system push notifier is installed.  In the latter case, imagine that you
have a dozen apps that support push notifications installed on your phone, but
the system push notifier hasn't been installed yet.  Once it is, all your
already installed push-enabled apps will Just Work.

This all means that the installer must have plugins for handling new
functionality and the declarative manifest must be extensible, i.e. the
installer will ignore extensions that it doesn't (yet) have a registered
plugin for.


If you wanted to avoid plugins, you could define a fixed naming scheme, 
so that if APP provides HOOK then the installer creates an entry in e.g.


  /hooks/HOOK/APP.hook

Then the push service could look in /hooks/push/*.hook to find 
all apps that support it quickly. You could then just add all hooks, 
whether the service using them is installed or not.


For hooks with security implications, it might be better to wait until 
the user requests the feature. An annoyance of Android is that you can 
only grant permissions at install time, so every application asks for 
every possible permission up front.



I didn't get far enough on 0install (or don't remember ;) whether it has that
capability.


I will certainly concede that the rest of 0install is very flexible and
has a usefully pluggable design; you seem to have done a good job on it
and Barry spoke well of it.  It just seems that there's a bit of a
mismatch in that it's kind of over the top in terms of flexibility for
what we need and seems to be entirely missing some other things we need,
as above; and it would be a lot of code to rewrite in some other
language if (as is possible, though not imminent) we decide that Python
is just too heavyweight to run routinely on the Ubuntu phone.


On the installing side, you're always going to be limited by the network 
rather than the CPU. You might save a little power with a more efficient 
language, but I doubt it's very much.


For running apps, you'd probabl

RE: App installer design: click packages

[Omar B .]

A quick update and status report from the PackageKit and Listaller project for 
those interested:
http://blog.tenstral.net/2013/05/packagekit-appstream-and-listaller-a-status-report.html
  -- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: App installer design: click packages

[Britt Mathis]

Hello everyone, I'm new here and this may have been mentioned before,
but have we thought about doing what android does where each app is a
different 'user'? That way apps could be part of different levels of
security (i.e. this app isn't part of the camera group, but this one
is).

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: App installer design: click packages

[Alex M]

I would like to introduce some additional thoughts about some possible
useful features of new package manager.

Idea: autorun application after install with information about path
where it was executed.

Explanation:
Let's assume we want to install some application with large additional
downloads, for example we want to install *some digital games
distribution system* SDGDS ;). Suppose, new manager packages with
extension *.upk (Ubuntu package). We have DVD with game on it (mounted
to /media/NewGame2013) and Setup_NewGame2013.upk. User run this *.upk
with single click/touch and then this package gets installed, it
contains our SDGS, not a game. If we already have SDGDS - it's updated
or not, depends on version.

Just right after user clicked *.upk, package manager attempts (if
there are such option in declarative meta-info inside upk) to run this
software with path /media/NewGame2013 as an argument. Thus our SDGDS
gets information that we have media with some useful content (game)
and begin to install it into special path, which is granted by
declarative meta-info of upk. User always gets simple always-ready
installation.

let's imagine, we can deploy ZeroInstall and many other thing instead
supposed SDGDS (ok, let's call it Steam), and it will install it's
additional contents into special limited (by meta-info) path or
somehow else specified paths...

Such approach we can see in some proprietary OS, where is has some
installer and every installation begin with checking it's version,
then update, then software get's selected (components) and finally
installed.

Colin, what do you think about this? Just some meta-option in package
and software runs with path of place where it's attempted to be
installed.

Sorry for my english, I can describe this in other words if you think
this idea reasonable.

P.S.
Excuse me for my previous post, it was too emotional...

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: App installer design: click packages

[Barry Warsaw]

On May 14, 2013, at 02:37 PM, Colin Watson wrote:

>> > Barry spent
>> >some time looking at 0install, and it wasn't too bad a fit but we would
>> >still need to solve many of the same system integration problems.
>> 
>> Is there a list of these issues somewhere?
>
>Barry Warsaw would have to speak to that, I think.  But mainly, there's
>no way for the system to declare any kind of integration facilities at
>all, as far as I can see, so we'd have to solve that problem anyway.

What I think Colin is getting at here is that any system is going to have to
deal with declaring the files that allow an app to interoperate with system
services.  Think along the lines of dbus .service files, .desktop files, etc.
These will be things we might know about right now (such as the above), but
they maybe be additional integration points in the future.

Let's say for example, some future system service comes with a new version of
the SDK or OS image, that allows apps to define push notifications, by
including a .push file that has to get installed somewhere to be registered.
The app itself will only declare that it has a myapp.push, and you won't have
anything like a maintainer script to install myapp.push.  Instead, the
system's push notifier will have to be responsible for registering a plugin
with the installer system to say "okay, now when you see .push files, put them
here and do this something to register it."

You don't want the logic for that in the installer, and you don't want it in
the app.  You want it as part of the system push notifier.  This latter will
register a trigger that gets run whenever the installer sees that an app has
declared a .push file *and* you want it to run over all app .push files when
the system push notifier is installed.  In the latter case, imagine that you
have a dozen apps that support push notifications installed on your phone, but
the system push notifier hasn't been installed yet.  Once it is, all your
already installed push-enabled apps will Just Work.

This all means that the installer must have plugins for handling new
functionality and the declarative manifest must be extensible, i.e. the
installer will ignore extensions that it doesn't (yet) have a registered
plugin for.

I didn't get far enough on 0install (or don't remember ;) whether it has that
capability.

>I will certainly concede that the rest of 0install is very flexible and
>has a usefully pluggable design; you seem to have done a good job on it
>and Barry spoke well of it.  It just seems that there's a bit of a
>mismatch in that it's kind of over the top in terms of flexibility for
>what we need and seems to be entirely missing some other things we need,
>as above; and it would be a lot of code to rewrite in some other
>language if (as is possible, though not imminent) we decide that Python
>is just too heavyweight to run routinely on the Ubuntu phone.

On just few other points:

I'm not really a big fan of XML formats, since they're not human consumable
(readable or writable).  Others may disagree, which is fine, but I don't have
any interest in arguing about it.  Probably not a deal breaker, since the XML
files can always be generated, either by the 0install tools or something that
takes a JSON or YAML as input, etc.

I liked that it was fairly easy to wrap executables in "security" containers,
but I didn't dig into this in a lot of detail.

It seemed to have good security considerations.

IIRC, there's no built-in support for uninstallation.

We probably want something like an online-only search REST API for finding
apps, and then downloading from the URL such a search gives us.  Thus, there's
no cleverness in the repository layout, nor on the device, for finding apps.

Anyway, that's about all the additional stuff I've gleaned from my previously
private experiments with 0install that hasn't already been discussed in this
thread.

-Barry

P.S. If we *were* to accept that the installer could be written in Python, it
would *have* to be Python 3.  No more Python 2 on the phone, please!

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: App installer design: click packages

[Colin Watson]

On Thu, May 09, 2013 at 09:41:26AM +0100, Thomas Leonard wrote:
> On 2013-05-08 11:14, Colin Watson wrote:
> >So, at Steve Langasek's request, I've been putting together a proof of
> >concept of a low-level app package installer and packaging format.
> >Highlights of what it can do so far are:
> >
> >  * no dependencies between apps; single implicit dependency on the base
> >system by way of a Click-Base-System field
> 
> 0install packages can depend on debs if they want to, so if you
> create a virtual deb package representing the Ubuntu base system
> then you could just depend on that (e.g. "requires Ubuntu >= 13.4").
> If you want to be cross platform, it would make more sense to
> specify individual dependencies, though.

This is a deliberate constraint to avoid interaction with the (large)
system package database, rather than a feature that was too hard to
implement.  Is it easy to disable this feature in 0install and do
something more like the system that Listaller and click-package use of
depending on system frameworks declared in text files?

(I suppose it's not critical; the primary requirement is bulletproof
reliability, and performance is merely an important secondary
requirement.)

> >  * base package manager overhead, i.e. the time required to install a
> >trivial package containing a single small file, is about 0.15 seconds
> >on a newish x86 laptop and about 0.6 seconds on a Nexus 7 (and that's
> >with the current prototype implementation in Python; a later
> >implementation could be in C and would then be faster still)
> 
> 0.109 seconds here on my laptop (including the time to download the
> archive from Apache running on localhost; normally network delays
> will dominate, of course).

debs install pretty quickly on my laptop too, but querying the package
database is going to be slow on typical phone hardware, so I'd still
like to make sure we avoid that.

> >  * not limited to installing as root, although there may be similar
> >constraints elsewhere to ensure that apps can't edit their own code
> >at run-time
> 
> Same. If system-wide sharing is enabled then downloads go under
> /var/cache and so apps cannot edit their own code. Without
> system-wide sharing, the files are stored in the user's home
> directory. They're read-only, but Linux doesn't currently provide us
> with a way to stop an app running as a user from changing that
> user's files. We're looking into using the new user namespaces
> support for sandboxing.

I think we will have some degree of sharing but not to the extent where
one user can force another to upgrade, so we'll be running the installer
as a separate user.

> >  * strawman design for hooks into system packages, which will be
> >entirely declarative from the app's point of view
> 
> What does this mean?

You mean "entirely declarative from the app's point of view"?  I mean
that the app contains no code executed with the permissions of the
package installer, although system packages may contain code that is
executed with those permissions when apps install certain kinds of files
or declare certain kinds of metadata.

> >Obvious items I still need to work on:
> >
> >  * produce a strawman hooks implementation with some real worked
> >examples
> >  * integrate (or demonstrate how to integrate) the container isolation
> >properties worked on elsewhere
> 
> Would be interested to hear more about this.

https://click-package.readthedocs.org/en/latest/hooks.html is the short
answer for now.

> >Is there anything else people can think of that a system like this needs
> >to consider?
> 
> Package signing, key distribution and trust, mirroring,

All of these belong in a higher-level tool.

> dependencies,

Intentionally excluded.

> roll-back, parallel installation of stable and testing
> versions,

Different versions are unpacked into different directories, so these are
easy.  (In fact the main thing to do is the converse: decide when to
garbage-collect old versions.)

> checking for updates,

Higher-level tool.

> source packages, ...? ;-)

I made a conscious choice not to do anything like Debian source
packages.  I have no problem with them personally, but they're a
consistent source of confusion among app developers, and of course (ugh)
there is the inevitable confusion for distributors of proprietary apps.
However, I'm conscious that it's important to make it easy for people to
distribute free software the right way.  This is an open question for
now and I've put it on the to-do list.

> > Barry spent
> >some time looking at 0install, and it wasn't too bad a fit but we would
> >still need to solve many of the same system integration problems.
> 
> Is there a list of these issues somewhere?

Barry Warsaw would have to speak to that, I think.  But mainly, there's
no way for the system to declare any kind of integration facilities at
all, as far as I can see, so we'd have to solve that problem anyway.

I will certainly concede th

Re: App installer design: click packages

[Alejandro J. Cura]

On Mon, May 13, 2013 at 6:53 PM, Colin Watson  wrote:
> On Mon, May 13, 2013 at 01:56:43PM -0400, Marc Deslauriers wrote:
>> On 13-05-13 11:05 AM, Alejandro J. Cura wrote:
>> > This thread assumes that packages need to be uncompressed and
>> > installed before usage, so I'd like to ask if there was any discussion
>> > re: using something like squashfs images as the distributed packages
>> > instead of a zip or tar-like file.
>> >
>> > This would mean that such downloaded images can be mounted read-only
>> > by whatever launches applications, using nosuid, nodev, and with the
>> > required uid, and then run immediately, instead of having to go thru a
>> > copy of files from the package to the storage, which slows down
>> > installation and usually requires double the storage space.
>>
>> That would mean we'd need to have a privileged helper to be able to
>> mount application packages at application execution time. There are a
>> lot of security implications of doing something like this, and I fear
>> this would be a substantial attack surface.
>
> And even if we mounted them all just once at boot, (a) we'd still need
> to use root privilege to mount at application installation/upgrade time,
> (b) any bug in squashfs would now become an easy escalate-to-kernel
> vulnerability exploitable through the app store.  Now, (b) is still true
> for dpkg/tar/etc., but tar is already assumed to take hostile input, and
> the relevant parts of dpkg are mainly shared with unprivileged
> operations such as 'dpkg -c' which are also assumed to take hostile
> input and have had a good deal of attention over the years; and even if
> there is a problem we can at least contain it to a less-privileged
> specialised 'software' user.

Yes, any squashfs overflow sounds scary for this use case, so I'm
convinced now that it's not such a good idea.

Thanks for your replies!
--
alecu

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: App installer design: click packages

[Alex M]

> This just seems to be a complex enough case that it's simply not a good
> target for this packaging system.  I'm OK with that.  As I said, this is
> not trying to be a complete replacement for debs, because if it tried to
> do that it would rapidly turn into a second system with a different set
> of flaws.

Sad to hear that... Debs (and Rpms) is a reason why bug #1 will not be
solved in foreseeable future. Thus, developers still needs to create
their ugly or use 3rd-party installers which act on system or homepage
without limitations because root/sudo password, users need do manual
changing execute bit with entering sudo password which is not
convenient and hard for regular users.

Solution which I suggest is dependency-free and it only seems as
complex, but actually just encapsulates your packages (and it's OK
that they are read-only) into single one. It would be very convenient
to have them in single place instead downloading and installing
separate 5-10 packages of some software which have components,
especially when some of them logically required.

The only simple things required is to provide additional file
extension for such pure-declarative installer and GUI itself (with
possibility to install from CLI).

Just imagine, developer will want to deploy Mono-based application.
Such package can contain Mono runtime package + application package
itself. What if Mono-runtime of _exact_ version already installed with
your packaging system? Then just mark in installer GUI that this
required component already installed. And what if other application
will want same runtime of same version? No problems. What if version
differs? Then it installs another one alongside with current. And
Java? And Python, let's assume it's not preinstalled because we need
2.7 and Ubuntu provides 3.x, or it's old package which needs to be
installed on Ubuntu 22.04 where only Python 4.x available...

Let's don't repeat Android's errors, look at how Qt-based applications
deployed for this platform. When user attempts to run application it
prompts user to install huge additional component through market. And
what if user out of network coverage or in expensive roaming network?

Noting bad in dependencies itself, it's just bad Linux's
implementation contains flaws:
1) Can't install under user, because files writing in system paths.
2) Can't install many versions of same library, because library space is flat.
3) Packages for APT/RPM does not contain necessary dependencies.

Look at ZeroInstall - it's nice except dumb deploying experience of
stand-alone network-free packages (absence of which - main drawback of
Ubuntu and other Linux-system in part of software management), because
we need to create package which contains non-declarative parts which
should be run with password, because ZI not installed in Linux system
by-default, and we need to set executable bit... Insane.

Ideal solution: do users and developers at least same level of
possibilities at least as in Mac OS X.

It's not late to create such package management system, or at least
PLEASE, don't say "it will never be in Ubuntu".

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

RE: App installer design: click packages

[Omar B .]

UDS is tomorrow so there will be some good discussions about this.
http://summit.ubuntu.com/uds-1305/track/foundations/

However, I fear things will be rather inconclusive as there's no clear picture 
yet.
So this should have an entry in the Wiki with a list of present and future 
features that are not only necessary, but could give Ubuntu more advantages 
compared to other OSs (and not just catch up, but lead).
Also a comparison of systems (both the proprietary and the ones available for 
linux) will also be needed to easier visualize and make things a bit more 
transparent.
There should be a list of advantages and disadvantages of each one as well as 
possible issues.
for example here's one from 0install:  http://0install.net/comparison.html
However is fairly simple and doesn't have all options listed like:
 Linstaller, Nix or even glick2.
If one offers enough pros or some killer features for the future that gives the 
OS an advantage than would be worth having it available.


  -- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: App installer design: click packages

[Robert Bruce Park]

On Thu, May 09, 2013 at 02:47:47PM +0400, Sergey Shambir wrote:
> 1. Declarative manifest
> Main benefit is ability to handle this file automatically: index it,
> provide UI and either generate automatically from build system
> information or parse to get information back.

In my mind, the main benefit is that we eliminate root-privileged
maintainer scripts. The current situation with debian is that all
package maintainers have the ability to write a postinst script that
will run as root on your system, and this script can be in any
language, without any form of controls on it.

That kind of situation made sense in 1993 when there were only a few
dozen package maintainers who all knew and trusted each other, but it
really hasn't scaled. Just think: every package you've ever installed,
every PPA you've enabled, you've implicitly given root access on your
machine to the author of that package. Who are these people? Do you
trust them? It's insane!

The goal of a declarative manifest is to make it possible to install a
package on the system which does not require unlimited arbitrary root
code execution by random untrusted packages downloaded from the
internet.

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: App installer design: click packages

[Colin Watson]

On Mon, May 13, 2013 at 01:56:43PM -0400, Marc Deslauriers wrote:
> On 13-05-13 11:05 AM, Alejandro J. Cura wrote:
> > This thread assumes that packages need to be uncompressed and
> > installed before usage, so I'd like to ask if there was any discussion
> > re: using something like squashfs images as the distributed packages
> > instead of a zip or tar-like file.
> > 
> > This would mean that such downloaded images can be mounted read-only
> > by whatever launches applications, using nosuid, nodev, and with the
> > required uid, and then run immediately, instead of having to go thru a
> > copy of files from the package to the storage, which slows down
> > installation and usually requires double the storage space.
> 
> That would mean we'd need to have a privileged helper to be able to
> mount application packages at application execution time. There are a
> lot of security implications of doing something like this, and I fear
> this would be a substantial attack surface.

And even if we mounted them all just once at boot, (a) we'd still need
to use root privilege to mount at application installation/upgrade time,
(b) any bug in squashfs would now become an easy escalate-to-kernel
vulnerability exploitable through the app store.  Now, (b) is still true
for dpkg/tar/etc., but tar is already assumed to take hostile input, and
the relevant parts of dpkg are mainly shared with unprivileged
operations such as 'dpkg -c' which are also assumed to take hostile
input and have had a good deal of attention over the years; and even if
there is a problem we can at least contain it to a less-privileged
specialised 'software' user.

-- 
Colin Watson   [cjwat...@ubuntu.com]

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: App installer design: click packages

[Colin Watson]

On Tue, May 14, 2013 at 03:23:26AM +0900, Emmet Hikory wrote:
> Colin Watson wrote:
> > On Mon, May 13, 2013 at 08:30:16PM +0400, Alex M wrote:
> > > I hope to see restore functionality (if we want to reset package to
> > > defaults and restore missing files).
> > 
> > The unpacked package directory will not be writable by the ordinary user
> > or the application directly, and should therefore not change.  As such,
> > I don't think a restore feature should be necessary.
> 
> For those users who might need such a thing (e.g. partial loss of data
> from recoverably damaged secondary storage), wouldn't a reinstall of the
> relevant package work?

Yes.  Or we could just remove the unpacked package directory and install
afresh, since it contains nothing irreplaceable.

-- 
Colin Watson   [cjwat...@ubuntu.com]

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: App installer design: click packages

[Emmet Hikory]

Colin Watson wrote:
> On Mon, May 13, 2013 at 08:30:16PM +0400, Alex M wrote:
> > I hope to see restore functionality (if we want to reset package to
> > defaults and restore missing files).
> 
> The unpacked package directory will not be writable by the ordinary user
> or the application directly, and should therefore not change.  As such,
> I don't think a restore feature should be necessary.

For those users who might need such a thing (e.g. partial loss of data
from recoverably damaged secondary storage), wouldn't a reinstall of the
relevant package work?  dpkg is usually happy to overwrite files for the same
package, and I don't see anything in the current code on launchpad that ought
prevent this method working.  I've certainly used the equivalent for normal
packages in such situations in the past (although in practice, it is typically
less painful to reinstall for significant recoverable damage).

-- 
Emmet HIKORY

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: App installer design: click packages

[Marc Deslauriers]

On 13-05-13 11:05 AM, Alejandro J. Cura wrote:
> Colin Watson wrote on 08/05/13 11:14:
>> Is there anything else people can think of that a system like this
>> needs to consider?
> 
> This thread assumes that packages need to be uncompressed and
> installed before usage, so I'd like to ask if there was any discussion
> re: using something like squashfs images as the distributed packages
> instead of a zip or tar-like file.
> 
> This would mean that such downloaded images can be mounted read-only
> by whatever launches applications, using nosuid, nodev, and with the
> required uid, and then run immediately, instead of having to go thru a
> copy of files from the package to the storage, which slows down
> installation and usually requires double the storage space.
> 
> I'm surely missing some bits of the picture, so please flame me if
> that's the case.

That would mean we'd need to have a privileged helper to be able to
mount application packages at application execution time. There are a
lot of security implications of doing something like this, and I fear
this would be a substantial attack surface.

Marc.




-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: App installer design: click packages

[Colin Watson]

On Mon, May 13, 2013 at 08:30:16PM +0400, Alex M wrote:
> 1) Which libraries you will consider as BASE system?
> 
> I hope not just "Ubuntu default installation" will treated as base system
> in purposes of distributive- and OS- independence... This set of libraries
> and it's versions will change in time, new libraries will require some
> system calls which is not available in older kernel versions.

At present, the target is the Ubuntu SDK, although this will be explicit
in the packages which rely on it ("Click-Framework: ubuntu-sdk-13.10"
etc.).

> Also if this library set is quite small, could they be placed in every
> package?

They will not be small. :-)

> 2) Do you plan to use components in packages?
> 
> Some details required about this question. Let's assume we want to pack Qt
> SDK with your brand new packaging manager. It will consist of Qt Creator,
> Qt library itself, documentation and sources. And of course, it would be
> convenient to include GCC or some different compiler it same package.
> 
> But we want set of packages without unconvenient dependencies...

This just seems to be a complex enough case that it's simply not a good
target for this packaging system.  I'm OK with that.  As I said, this is
not trying to be a complete replacement for debs, because if it tried to
do that it would rapidly turn into a second system with a different set
of flaws.

> I hope to see restore functionality (if we want to reset package to
> defaults and restore missing files).

The unpacked package directory will not be writable by the ordinary user
or the application directly, and should therefore not change.  As such,
I don't think a restore feature should be necessary.

> 3) Will directory with contents of package include it's version?

Yes.  There'll be a symlink to the current version for convenience.

-- 
Colin Watson   [cjwat...@ubuntu.com]

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: App installer design: click packages

[Colin Watson]

On Mon, May 13, 2013 at 12:05:57PM -0300, Alejandro J. Cura wrote:
> Colin Watson wrote on 08/05/13 11:14:
> > Is there anything else people can think of that a system like this
> > needs to consider?
> 
> This thread assumes that packages need to be uncompressed and
> installed before usage, so I'd like to ask if there was any discussion
> re: using something like squashfs images as the distributed packages
> instead of a zip or tar-like file.

I know some of the other relevant systems do this, and I did think about
it.  The main downside I see is that I want apps to be always fully
present on the filesystem; it doesn't make so much difference on a
phablet, I suppose, but later on a desktop system people are going to
want to look around manually, symlink to things, wrap things in scripts,
etc.  If we use a mounted-image scheme that then implies that we need to
take care to mount all apps at boot time.  I'm concerned about the
possible effect on boot speed and reliability that that could have.  I'm
also not clear that there's been much research on whether this type of
scheme has significant negative effects on things like elevator
algorithm performance, caching, etc.

The bottom line for me is that I'd like to keep things as simple as
possible, and unpacking seems to me to be essentially simpler than a
persistent-loop-mount scheme.

Thanks,

-- 
Colin Watson   [cjwat...@ubuntu.com]

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: App installer design: click packages

[Colin Watson]

On Wed, May 08, 2013 at 11:14:08AM +0100, Colin Watson wrote:
> I should have the prototype ready for people to look at in time for
> UDS next week,

lp:click-package - of course, no interfaces there should be relied on,
and many are skeletal.

> and I'll ensure that there's a session scheduled for this.

  
http://summit.ubuntu.com/uds-1305/meeting/21760/foundations-1305-click-package/

-- 
Colin Watson   [cjwat...@ubuntu.com]

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: App installer design: click packages

[Marcin Wiacek]

  Hi,

  I'm independent programmer not very
connected over few years with Open Source software (like earlier). Your
information about Click packages was cited in various IT portals and I have
few comments:

 

First - some time ago I was thinking about something similar, but it wasn't
accepted by admins: http://brainstorm.ubuntu.com/idea/30533/

 

Second - I think, it would be nice to consider such scenario: there are left
possibilities of installing current software and "new" format is built on
top of it (like you propose), but it's a little more mature and during
installation user is informed about few things:

 

1.  Firewall network rules required by application (if application will
try to get more after installation, system won't allow for it)

2.  Task scheduler tasks (what will be run in cyclic manner and what
will be run for example during system startup)

3.  Permissions required by application (similar like in Android - for
example access to concrete devices like GPS or Bluetooth)

 

Later after installation it's possible to check it and revoke some of them
(make them disabled). Format should allow for such things like providing
kernel modules or access to concrete USB devices (it will allow for making
for example players for concrete TV tuners)

 

I was thinking also about limiting access for files by format (like in
Windows) and user directories (in easy way - for details please refer to
http://brainstorm.ubuntu.com/idea/30533/), but in Linux distribution it
won't be probably possible. For now I think too, that there should be
resolved problem of temp files (but rather on higher OS level) - OS should
have ability of easy making RAM Disk for temp directory and temp files
(separated of course) from Click apps should go there too.

 

Regards,

 

Marcin

 

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: App installer design: click packages

[Alejandro J. Cura]

Colin Watson wrote on 08/05/13 11:14:
> Is there anything else people can think of that a system like this
> needs to consider?

This thread assumes that packages need to be uncompressed and
installed before usage, so I'd like to ask if there was any discussion
re: using something like squashfs images as the distributed packages
instead of a zip or tar-like file.

This would mean that such downloaded images can be mounted read-only
by whatever launches applications, using nosuid, nodev, and with the
required uid, and then run immediately, instead of having to go thru a
copy of files from the package to the storage, which slows down
installation and usually requires double the storage space.

I'm surely missing some bits of the picture, so please flame me if
that's the case.

cheers,
--
alecu

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: App installer design: click packages

[Sergey Shambir]

Hi,

Thanks for efforts on simplifying package management! I'll wait for 
further information from UDS session, but first I want to provide some 
additional info that can help you in research. I'm QtCreator contibutor, 
worked on source code models and project management subsystems, 
including build systems support.


Click packages have many decisions that can scare experienced Linux 
users, but are reasonable. So, for marketing purposes and in order to 
defend idea, please make more detailed overview of each design decision, 
which includes: 1. Comparison of various approaches in this concrete 
problem, with pros and cons. 2. Destroying of common myths associated 
with selected approach (like "if dependencies will be eliminated, size 
of packages will grow significantly").


Possible research agenda:
1. Which benefits brings declarative property-based app manifest?
2. Which benefits brings reversing build system <-> package manager 
interaction order?

3. Can qmake automatically create app manifest in near future?
4. Does size of real applications packages really grow with 
no-dependencies system with SDK (which includes Qt)?
5. Which issues are in Listaller and existing build systems (mostly in 
qmake)?
6. Which pros and cons have installing a) from root b) from user to 
$HOME c) from partially privilegied, but non-root user (similar to 
"wwwrun" or "vboxuser")?
7. Which ways exist to support development model "build on newer system, 
deploy to older"?


Let me throw first stone:

1. Declarative manifest
Main benefit is ability to handle this file automatically: index it, 
provide UI and either generate automatically from build system 
information or parse to get information back. Build systems is an 
excellent example: XCode, Visual Studio and CodeBlocks IDEs have 
property-based build settings, and they can automatically edit such 
settings for years. Meanwhile, no one IDE can automatically edit 
Autotools or Premake files, CMake editing is implemented only in 
KDevelop and is buggy; editing qmake in QtCreator also works wrong in 
complex cases.


2. Reversing build system <-> package manager
Typical project selects one build system, but targets to many platforms. 
At first, when package manager handles buildsystem, user should manually 
debug package manager scripts on each platform. At second, build system 
developers do not want to implement creating packages (because package 
manager designed to handle build system): 
https://bugreports.qt-project.org/browse/QBS-16. At third, this 
situation previously led to ugly design of make-based build systems: 
they are script-based and expose hooks to package manager with 
environment variables or configure options, but have no ability to 
easily use different build configurations (Debug, Release) in same time, 
and portability between platforms achieved with hacks or annoying 
debugging. Declarative build systems have no such problems.


4. Packages size on disk and in memory.
QtCreator in Ubuntu depends on Qt and Botan libraries. It doesn't depend 
on Botan in upstream version, since it keeps Botan copy in sources. Most 
crossplatform apps keep dependencies set as small as possible, because 
Windows and Mac OS X have no dependency-based package system. So, size 
growth will be insignificant.


5a. Listaller issues
Currently Listaller handles build system: 
http://listaller.tenstral.net/docs/doc/packaging.html. Meanwhile, 
buildsystem should handle listaller, and do it easily. No one developer 
wants to manually write and debug list of files to install.


5b. Qmake issues
QMake have INSTALL variable that tracks installable files, but doesn't 
keep target bundle version, description and name. Since QtCreator have 
qmake evaluator, it can be patched to create or update manifest file, 
but some information should be typed by developer.


7. Build on newer system, deploy to older
Listaller and LSB tools from Linux Foundation both use wrappers around 
gcc and special linker wrapper / special version of glibc respectively. 
Another way used in Mac OS X and iOS: XCode contains several SDKs, and 
application builds with only one in sysroot.


--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: App installer design: click packages

[Alex M]

Hello!
I want to ask some questions about new packaging manager (which I awaited
for many years...) and add some requests while it's not too late.

Questions:

1) Which libraries you will consider as BASE system?

I hope not just "Ubuntu default installation" will treated as base system
in purposes of distributive- and OS- independence... This set of libraries
and it's versions will change in time, new libraries will require some
system calls which is not available in older kernel versions.

Also if this library set is quite small, could they be placed in every
package?

2) Do you plan to use components in packages?

Some details required about this question. Let's assume we want to pack Qt
SDK with your brand new packaging manager. It will consist of Qt Creator,
Qt library itself, documentation and sources. And of course, it would be
convenient to include GCC or some different compiler it same package.

But we want set of packages without unconvenient dependencies...

I suggest to add support of "package of packages" as it done in Mac OS X
(also, Qt SDK installer also could do this). Back to example, single Qt SDK
package with several components packages inside, and we can choose which to
install. If we already have compiler, we click to not install this
component.

Of course, this "package of packages" on top of plain packages, which
installing by single click...

I hope to see restore functionality (if we want to reset package to
defaults and restore missing files).

3) Will directory with contents of package include it's version?

P.S.
Excuse me for my worse english...
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: App installer design: click packages

[Colin Watson]

On Fri, May 10, 2013 at 11:45:37AM +0100, Matthew Paul Thomas wrote:
> Colin Watson wrote on 08/05/13 11:14:
> > So, at Steve Langasek's request, I've been putting together a
> > proof of concept of a low-level app package installer and
> > packaging format. Highlights of what it can do so far are:
> > 
> > ...
> > 
> > * installs each app to an entirely separate directory
> > 
> > * entirely declarative: maintainer scripts are forbidden
> 
> Do these two entail that an installation/update/removal can be
> interrupted at any point without corrupting the system?

Yes.

> This is, as I understand it, a long-standing problem with deb/dpkg. A
> PC has a power outage or a kernel panic while a package is installing
> or updating, and the packaged software becomes unusable, and possibly
> stymies future package operations too.

Well, people have derived this kind of idea from observing the behaviour
of the whole system rather than from knowledge of the specific software
components involved, and so the notion of where problems lie tends to
get misattributed.  dpkg itself always leaves things in a known and
valid state (in its terms; that is, there's a defined next step you can
take to repair things where necessary).  The problem is that layers
above dpkg are not very good at coping with some of the valid states
dpkg leaves things in, and can end up requiring manual intervention as a
result.

This won't be a problem here.  At least to start with, we'll be
unpacking each version of each package into its own directory, and
atomically flipping a symlink at the end.  An incomplete unpack won't
affect the previously-installed version of the package, and there's no
possibility of affecting future package installations since there is no
shared package database.

> > * indexing support and other frontend things (i.e. the app store)
> 
> That will affect the package format. For example, each package
> providing metadata in multiple languages.

Noted.

> > Is there anything else people can think of that a system like this 
> > needs to consider?
> 
> A system that aims for an order of magnitude more applications, than
> Debian or Ubuntu has packages, should not rely on every package having
> a unique name. For example, a package should not determine the name of
> the installation directory itself.

In an entirely flat namespace this is a problem.  The current plan is to
use reverse-network-domain names, e.g. com.ubuntu.apps.calculator or
org.wikipedia.  This is the same scheme used in the Android Market, and
I think is adequate protection against this class of problem.

I considered using UUIDs or similar, but those have the annoying
property that it's difficult to look at a bit of the filesystem
hierarchy and know what it means without looking inside it and guessing
(a property I've always hated when trying to work out what's wrong with
some bits of Windows systems); this is less relevant on phones/tablets
but will matter later for convergence.

> Finally (and most bikesheddingly), a system initially aimed at phones
> and tablets should not be named after something you do with a mouse.

I fear the name predates my involvement.  Perhaps Stuart can comment.

-- 
Colin Watson   [cjwat...@ubuntu.com]

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: App installer design: click packages

[Matthias Klumpp]

Hi!

2013/5/9 Colin Watson :
> On Wed, May 08, 2013 at 03:25:42PM +0200, Matthias Klumpp wrote:
>> Disclaimer: As developer of Listaller, AppStream and PackageKit, I am
>> of course biased on this topic ;-)
>
> Thanks for your mail.  I'm looking over Listaller at the moment, and
> expect to have some more comments once I've done that.  However, I'm
> going to be mostly offline from Friday to Sunday, so a partial reply in
> the meantime:
Great! Please keep in mind that the project will undergo a transition
to greatly simplyify dependency solving soon, with an option to
restore and improve the 0install compatibility too. (there has been
discussion about this with 0install people some time ago, but we did
not find a working solution back then).
The key values for Listaller are distribution-independence and
integration (= everything which supports PackageKit can handle
Listaller stuff too).

>> 2013/5/8 Colin Watson :
>> >  * not limited to installing as root, although there may be similar
>> >constraints elsewhere to ensure that apps can't edit their own code
>> >at run-time
>>
>> Non-root installs into user's $HOME are implemented & working in
>> Listaller, but due to the tight integration into PackageKit, this
>> feature should not be used.
>
> In practice, I don't actually intend to install packages as the ordinary
> calling user directly; as I indicated above, this would be a bad idea
> because it would permit apps to overwrite their own code.  However, it's
> convenient for testing, and it may make sense to unpack apps as a
> special "software" user or similar to reduce the system's exposure to
> things like vulnerabilities in underlying unarchiving tools, which are
> rare but not unknown.  (Or we could just hit it with AppArmor.  I don't
> much care either way as long as we avoid the obvious flaw.)
Yes, that is the rationale for allowing things like this :-) If there
is a good way to lock 3rd-party applications with AppArmor, having a
feature like this would definitely help. But maybe the sandbox will
already provide enough security.

>> >  * building packages requires only the Python standard library, with the
>> >intent that it should be possible to build these packages quite
>> >easily on non-Ubuntu or even non-Linux systems
>>
>> Hmm, why? If a non-Python application should be packaged, this feature
>> would be lost, as people will have to crossbuild it anyway. And in
>> order to build stuff for the upcoming Ubuntu products, people will
>> need to install it (at least in a VM) anyway, for testing purposes.
>
> Remember that the main target of our project, at least for the time
> being, is Ubuntu Touch - phone/tablet apps.  We expect a significant
> percentage of these to be written in pure QML or HTML5/CSS, and to make
> it possible to carry these over to desktop apps in time as well.  I
> think it's important to minimise the barrier to entry to creating such
> apps; if somebody wants to create them on other systems they should be
> able to.
For all QML/HTML5 stuff, this indeed makes sense. But since packages
are not magic at all (often just compressed tarballs with some meta
information), implementing that is easily possible. 0install proves
that it even works for deploying stuff on MacOS, Windows and Linux.

>> >  * binary packaging format sufficiently similar to existing one that we
>> >could add support to higher-level tools with minimal effort
>> >  * strawman design for hooks into system packages, which will be
>> >entirely declarative from the app's point of view
>>
>> Is this going to be Ubuntu-specific? If not, you will have some
>> trouble with that...
>
> I wasn't planning to put "ubuntu" in the hook paths or anything, but it
> is indeed up to the system to declare how it wants apps to integrate
> with it.  I don't see a sensible way around that; it is entirely wrong
> to put system/app integration code in the installer itself, and any such
> approach would result in problems with the system and the installer
> getting out of sync further down the line.
You mean the package-information with "installer"? Or do you mean the
application which installs things? The package itself should not have
any assumptions on how it will be installed on the system, and how the
system would look like. The application which installs the package,
however, is responsible for integrating the new application into the
system, and also for negotiating with the native package manager about
installed requirements.
Both Listaller and 0install force applications to be developed in a
way that they don't have any fixed assumptions on the system they are
installed on. Listaller even contains many tools due to it's partial
Autopackage origin which help to minimize and lower system
dependencies.
All of these features don't matter much if you want to use a 3rd-party
package manager just on the Ubuntu Phone, because I assume that the
phone software will have some kind of API/Platform level applicatio

Re: App installer design: click packages

[Oliver Grawert]

hi,
On Fr, 2013-05-10 at 11:45 +0100, Matthew Paul Thomas wrote:

> 
> Finally (and most bikesheddingly), a system initially aimed at phones
> and tablets should not be named after something you do with a mouse.

convergence 
;)

ciao
oli


signature.asc
Description: This is a digitally signed message part
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: App installer design: click packages

[Matthew Paul Thomas]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Colin Watson wrote on 08/05/13 11:14:
> ...
> 
> So, at Steve Langasek's request, I've been putting together a
> proof of concept of a low-level app package installer and
> packaging format. Highlights of what it can do so far are:
> 
> ...
> 
> * installs each app to an entirely separate directory
> 
> * entirely declarative: maintainer scripts are forbidden

Do these two entail that an installation/update/removal can be
interrupted at any point without corrupting the system?

This is, as I understand it, a long-standing problem with deb/dpkg. A
PC has a power outage or a kernel panic while a package is installing
or updating, and the packaged software becomes unusable, and possibly
stymies future package operations too.

An interruption like this is much more likely to happen on a phone or
tablet, because it's much more often on battery while you're
installing something.

> ...
> 
> To do, but not in this project:
> 
> * package acquisition

While designing session-installer error messages, I was told that apt
returns localized error messages but not error codes, making it
impractical for a UI wrapper to respond appropriately to different
kinds of error (e.g. providing a "Try Again" button only when trying
again might actually work). Avoiding this in any new system would be
desirable.

> * indexing support and other frontend things (i.e. the app store)

That will affect the package format. For example, each package
providing metadata in multiple languages.

> ...
> 
> Is there anything else people can think of that a system like this 
> needs to consider?
> 
> ...

A system that aims for an order of magnitude more applications, than
Debian or Ubuntu has packages, should not rely on every package having
a unique name. For example, a package should not determine the name of
the installation directory itself.

Finally (and most bikesheddingly), a system initially aimed at phones
and tablets should not be named after something you do with a mouse.

Cheers
- -- 
mpt
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iEYEARECAAYFAlGMz9EACgkQ6PUxNfU6ecpWFgCfV9GhyHH95qe/1rZtWCJaNJ/6
3vUAnAzDJiALnaK1bRg525P0yyoomoQS
=vMfc
-END PGP SIGNATURE-

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: App installer design: click packages

[Colin Watson]

On Wed, May 08, 2013 at 03:25:42PM +0200, Matthias Klumpp wrote:
> Disclaimer: As developer of Listaller, AppStream and PackageKit, I am
> of course biased on this topic ;-)

Thanks for your mail.  I'm looking over Listaller at the moment, and
expect to have some more comments once I've done that.  However, I'm
going to be mostly offline from Friday to Sunday, so a partial reply in
the meantime:

> I find it very sad that so many parties currently start reinventing
> the basic ideas of Listaller, instead of helping the Listaller project
> to improve, and kill the remaining flaws.

I would prefer to keep the discussion technical rather than emotional.

> 2013/5/8 Colin Watson :
> >  * not limited to installing as root, although there may be similar
> >constraints elsewhere to ensure that apps can't edit their own code
> >at run-time
> 
> Non-root installs into user's $HOME are implemented & working in
> Listaller, but due to the tight integration into PackageKit, this
> feature should not be used.

In practice, I don't actually intend to install packages as the ordinary
calling user directly; as I indicated above, this would be a bad idea
because it would permit apps to overwrite their own code.  However, it's
convenient for testing, and it may make sense to unpack apps as a
special "software" user or similar to reduce the system's exposure to
things like vulnerabilities in underlying unarchiving tools, which are
rare but not unknown.  (Or we could just hit it with AppArmor.  I don't
much care either way as long as we avoid the obvious flaw.)

> >  * building packages requires only the Python standard library, with the
> >intent that it should be possible to build these packages quite
> >easily on non-Ubuntu or even non-Linux systems
> 
> Hmm, why? If a non-Python application should be packaged, this feature
> would be lost, as people will have to crossbuild it anyway. And in
> order to build stuff for the upcoming Ubuntu products, people will
> need to install it (at least in a VM) anyway, for testing purposes.

Remember that the main target of our project, at least for the time
being, is Ubuntu Touch - phone/tablet apps.  We expect a significant
percentage of these to be written in pure QML or HTML5/CSS, and to make
it possible to carry these over to desktop apps in time as well.  I
think it's important to minimise the barrier to entry to creating such
apps; if somebody wants to create them on other systems they should be
able to.

Running a separate test device may well be an easier proposition than
building on it.  For example, you might have a physical device (Nexus 7
or whatever) that you can easily install things on; that doesn't mean
you want to run your development environment there.

I agree that building native apps will typically involve building in a
clean Ubuntu environment.

> >  * binary packaging format sufficiently similar to existing one that we
> >could add support to higher-level tools with minimal effort
> >  * strawman design for hooks into system packages, which will be
> >entirely declarative from the app's point of view
> 
> Is this going to be Ubuntu-specific? If not, you will have some
> trouble with that...

I wasn't planning to put "ubuntu" in the hook paths or anything, but it
is indeed up to the system to declare how it wants apps to integrate
with it.  I don't see a sensible way around that; it is entirely wrong
to put system/app integration code in the installer itself, and any such
approach would result in problems with the system and the installer
getting out of sync further down the line.

dpkg is right to defer all integration code to packages; the hook scheme
I'm considering is mainly inspired by dpkg triggers.  (We may or may not
be able to call dpkg triggers directly; for performance it is vital to
avoid having to actually open the dpkg database.)

> > Footnote: Why not use one of the existing systems out there?
> >
> > We still might, but at present it's not clear that it would make a whole
> > lot of difference.
> 
> If it doesn't, why don't you use them? :P

It's a trade-off between the benefit of using an existing system and the
cost of twisting it around to fit what we actually need it to do.  I'm
not fundamentally opposed to using an existing system - I certainly
understand why people want us to, and that there's a perception that
Canonical unnecessarily does its own thing (you can look up my own
commit record if you like) - but if there's a mismatch between our goals
then it may end up being rather more trouble than it's worth.  One thing
I'm particularly concerned about is that a system that has tried to be
very general may end up reinventing similar scaling and reliability
problems as the packaging system we already have.

(To be clear - hi Slashdot - I'm very fond of the dpkg/apt system and
have put a great deal of effort into it over the last decade or so, both
on my own time and for pay; I think it's excellent and have no desire to

Re: App installer design: click packages

[Jos van den Oever]

Hello Colin,

I read you ubuntu-devel mail about research into packaging. You asked for 
pointers, so I'm mailing you one.

Many of the points you are looking for in a package manager are covered by Nix.
Nix is a distro independent package manager that is very different from the 
standard package managers.

Instead of putting all applications in /usr, each version of each package is 
stored in a dedicated folder. It is possible to have many versions of packages 
and different users can use different versions.

Nix works on ubuntu and many other operating systems today. (Also, there is an 
OS that uses Nix as the main package manager, NixOS.)

Packages are built from Nix expressions, which is a simple functional language. 
A Nix expression describes everything that goes into a package build action (a 
“derivation”): other packages, sources, the build script, environment variables 
for the build script, etc. Nix tries very hard to ensure that Nix expressions 
are deterministic: building a Nix expression twice should yield the same result.

The Nix community on #nix is very active and helpful.

To avoid dependencies between apps, you can group nix expressions into one 
package.

Installing a package means unpacking and setting some symlinks. Switching 
between versions is just a matter of setting symlinks once both versions are 
unpacked.

Any user can install any package for themselves.

A huge amount of research including PhD thesis went into creating Nix.

https://nixos.org/nix/

Cheers,
Jos

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: App installer design: click packages

[Thomas Leonard]

On 2013-05-08 11:14, Colin Watson wrote:

There've been many discussions of late about the challenges of scaling
app installation to a very large number of apps, including making app
packages much easier to automatically audit and sandbox, and making them
safer and quicker to install without the possibility of system-level
breakage.  This is needed to make it quicker and easier for app
developers to get their software into Ubuntu.  There are several pieces
to this (including a good deal of prior discussion in
https://wiki.ubuntu.com/AppDevUploadProcess), but it seems clear that
one of those is some kind of simplified packaging format which we can
offer to third-party application developers.


Hi, 0install developer here. I'd be interested to know more about what 
extra features 0install would need to be useful to you. As Matthias 
said, creating yet another packaging system would be disappointing, 
especially as your goals seem to match ours exactly.



This is not aimed at changing packages that are already part of the
Ubuntu archive; for the most part our existing system works well for
those, and they tend to have non-trivial dependency structures.  We'll
continue to use dpkg and apt for building the Ubuntu operating system,
syncing with Debian, and so on.  There's no point developing a packaging
system for apps and making it have the full panoply of features needed
for the Ubuntu archive: it'd just be second-system-effect on top of our
current packaging system.  So the scope of what I've been considering is
purely leaf apps built on a fixed "base system", which in the case of
the initial target of the Ubuntu phone/tablet work would be the run-time
part of the Ubuntu SDK.  The radically-reduced dependency structure
means that most apps will be Ubuntu-SDK-specific to start with, although
I'd like to make sure that the package format design includes enough
support up-front to allow this to be useful for other platforms that
define suitable base system profiles in future.

So, at Steve Langasek's request, I've been putting together a proof of
concept of a low-level app package installer and packaging format.
Highlights of what it can do so far are:

  * no dependencies between apps; single implicit dependency on the base
system by way of a Click-Base-System field


0install packages can depend on debs if they want to, so if you create a 
virtual deb package representing the Ubuntu base system then you could 
just depend on that (e.g. "requires Ubuntu >= 13.4"). If you want to be 
cross platform, it would make more sense to specify individual 
dependencies, though.



  * installs each app to an entirely separate directory


Yes.


  * entirely declarative: maintainer scripts are forbidden


Yes.


  * base package manager overhead, i.e. the time required to install a
trivial package containing a single small file, is about 0.15 seconds
on a newish x86 laptop and about 0.6 seconds on a Nexus 7 (and that's
with the current prototype implementation in Python; a later
implementation could be in C and would then be faster still)


0.109 seconds here on my laptop (including the time to download the 
archive from Apache running on localhost; normally network delays will 
dominate, of course).



  * not limited to installing as root, although there may be similar
constraints elsewhere to ensure that apps can't edit their own code
at run-time


Same. If system-wide sharing is enabled then downloads go under 
/var/cache and so apps cannot edit their own code. Without system-wide 
sharing, the files are stored in the user's home directory. They're 
read-only, but Linux doesn't currently provide us with a way to stop an 
app running as a user from changing that user's files. We're looking 
into using the new user namespaces support for sandboxing.



  * packages built by feeding the intended output directory tree to a
simple Python tool, plus a manifest.json file


Same, but the metadata is XML, not JSON.


  * building packages requires only the Python standard library, with the
intent that it should be possible to build these packages quite
easily on non-Ubuntu or even non-Linux systems


Same. 0install packages can run across Linux, OS X, Windows, Unix, etc 
(assuming the packaged software can too).



  * binary packaging format sufficiently similar to existing one that we
could add support to higher-level tools with minimal effort


Package format is .tar.bz2 (other similar compressed archive formats, 
such as .zip are also supported).



  * strawman design for hooks into system packages, which will be
entirely declarative from the app's point of view


What does this mean?


  * unit-tested from the start


Of course.


Obvious items I still need to work on:

  * produce a strawman hooks implementation with some real worked
examples
  * integrate (or demonstrate how to integrate) the container isolation
properties worked on elsewhere


Would be intereste

Re: App installer design: click packages

[Matthias Klumpp]

Hi!
Disclaimer: As developer of Listaller, AppStream and PackageKit, I am
of course biased on this topic ;-)
I find it very sad that so many parties currently start reinventing
the basic ideas of Listaller, instead of helping the Listaller project
to improve, and kill the remaining flaws.

2013/5/8 Colin Watson :
> There've been many discussions of late about the challenges of scaling
> app installation to a very large number of apps, including making app
> packages much easier to automatically audit and sandbox, and making them
> safer and quicker to install without the possibility of system-level
> breakage.  This is needed to make it quicker and easier for app
> developers to get their software into Ubuntu.  There are several pieces
> to this (including a good deal of prior discussion in
> https://wiki.ubuntu.com/AppDevUploadProcess), but it seems clear that
> one of those is some kind of simplified packaging format which we can
> offer to third-party application developers.
This is the rationale of Listaller, see http://listaller.tenstral.net/

> So, at Steve Langasek's request, I've been putting together a proof of
> concept of a low-level app package installer and packaging format.
> Highlights of what it can do so far are:
>
>  * no dependencies between apps; single implicit dependency on the base
>system by way of a Click-Base-System field
Implemented in Listaller, but poorly working. The dependency-solver is
currently being rewritten, with the upcoming KDE-Frameworks 5 and
GNOME-Platform (aka "GNOME-OS") in mind.

>  * installs each app to an entirely separate directory
Done in Listaller, but several shared resources can - well - easily be shared.

>  * entirely declarative: maintainer scripts are forbidden
Design decision for Listaller (resulted in controversial discussions,
but finally everyone seems to be fine with it).

>  * base package manager overhead, i.e. the time required to install a
>trivial package containing a single small file, is about 0.15 seconds
>on a newish x86 laptop and about 0.6 seconds on a Nexus 7 (and that's
>with the current prototype implementation in Python; a later
>implementation could be in C and would then be faster still)
I did not (yet) do measurements on this, but as Listaller was
rewritten in Vala some time ago, I expect it to be reasonably fast.

>  * not limited to installing as root, although there may be similar
>constraints elsewhere to ensure that apps can't edit their own code
>at run-time
Non-root installs into user's $HOME are implemented & working in
Listaller, but due to the tight integration into PackageKit, this
feature should not be used.

>  * packages built by feeding the intended output directory tree to a
>simple Python tool, plus a manifest.json file
Listaller uses a "depscan" tool to auto-detect dependencies. Currently
only works for binary shared libraries, due to lack of time and
Python/Ruby/Perl knowledge to make a sane implementation for these
languages. But the tool is designed with that in mind. Output is a
textfile in Debian-control-file style.

>  * building packages requires only the Python standard library, with the
>intent that it should be possible to build these packages quite
>easily on non-Ubuntu or even non-Linux systems
Hmm, why? If a non-Python application should be packaged, this feature
would be lost, as people will have to crossbuild it anyway. And in
order to build stuff for the upcoming Ubuntu products, people will
need to install it (at least in a VM) anyway, for testing purposes.

>  * binary packaging format sufficiently similar to existing one that we
>could add support to higher-level tools with minimal effort
>  * strawman design for hooks into system packages, which will be
>entirely declarative from the app's point of view
Is this going to be Ubuntu-specific? If not, you will have some
trouble with that...

Listaller also offers update-sources for installed 3rd-party apps, a
way to build binary apps using older library symbols and some
cross-distro compatibility stuff, a tight integration with PackageKit
(that might actually be a problem for Ubuntu...), which allows updates
for these packages to be displayed in standard package management
tools, alongside other software which was deployed using the system
package manager.

> Footnote: Why not use one of the existing systems out there?
>
> We still might, but at present it's not clear that it would make a whole
> lot of difference.
If it doesn't, why don't you use them? :P

> Many of the existing app packaging systems are more of a reflection of
> the system they were built for than anything else.  If you look at, say,
> Android's APK format, it's essentially a zip file with a manifest and
> some conventions about Dalvik class installation and the like.  Most of
> the other mobile app formats are similar.  Things like Listaller might
> be a reasonable fit, but I'm worried about importing things like a full
> depend

Re: App installer design: click packages

[Barry Warsaw]

On May 08, 2013, at 11:14 AM, Colin Watson wrote:

> * Click-Base-System field is skeletally simple right now, and may need
>   to be expanded to at least leave open the possibility of multiple
>   flavours of base system (see also GNOME's profiles idea)

We'll have a session at vUDS about image based updates:

https://blueprints.launchpad.net/ubuntu/+spec/foundations-1305-image-based-updates

There, we've defined a "build number" which defines the image version your
device is currently at, so we can calculate where and how we can upgrade it.

I suspect that the build number will be different than the CBS number, with
the latter indicating the version of the SDK that the app depends on, and this
will probably be different than device image version number.  Still, I wanted
to mention it in case there's overlap that I might be missing.

-Barry

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel