Re: Experimental Python interpreter snap

[James Henstridge]

On 20 February 2017 at 10:41, Spencer  wrote:
> I thought a main feature of snaps was to include all dependencies so that 
> they couldn't be changed out from underneath a package.  For example, my 
> script was written for 3.6, but would be incompatible with a future release, 
> say 4.0.

I've included two content interface slots on my snap: one with the
content ID "python3" and the other with the content ID "python3.6".
The idea being that a program that doesn't care about getting new
versions (e.g. my trivial hello world snap) could use the first, while
ones that really want python 3.6.x (e.g. if they contain compiled
extensions) could request the second.

I don't think there is any point in distinguishing micro releases
though, since the Python core developers have a good track record when
it comes to releasing security/bug fix updates to their previous
releases.


> Still, the ability to share a dependency like the Python interpreter among 
> python snaps may be a good idea if there are zillions of Python snaps.
>
> At my work, we got tired of everyone maintaining their own local python 
> installation, because we all ended up with slightly different versions of 
> various python modules installed.  Worse, some modules installed for some 
> while not for others.  So we started tracking a shared Python installation in 
> git.  One problem we found is that Python is not relocatable.  This showed up 
> when people cloned our repository in a different place.  Are you sure that 
> your snapped Python is relocatable?  If so, I'd like to know how that works.  
> Is the $ORIGIN variable standard?

Note that the the only things being shared here is the Python
interpreter and the standard library.  If one snap includes a bunch of
third party Python packages in their $SNAP/lib/python3.6/site-packages
directory, they won't be visible to a second snap that has also used
the interface.  The effect is quite similar to the isolation you get
from virtualenv.

As far as relocatability goes, part of it is provided by Python
proper.  Here's sys.path from the interpreter provided by the
python36-jamesh snap:

$ python36-jamesh.python3
Python 3.6.0 (default, Feb 20 2017, 01:27:20)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/snap/python36-jamesh/7/lib/python36.zip',
'/snap/python36-jamesh/7/lib/python3.6',
'/snap/python36-jamesh/7/lib/python3.6/lib-dynload']

And here it is when running in the context of my hello-world snap:

$ snap run --shell hello-world
To run a command as administrator (user "root"), use "sudo ".
See "man sudo_root" for details.

$ $SNAP/python/bin/python3
Python 3.6.0 (default, Feb 20 2017, 01:27:20)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/snap/hello-world/x1/python/lib/python36.zip',
'/snap/hello-world/x1/python/lib/python3.6',
'/snap/hello-world/x1/python/lib/python3.6/lib-dynload']

What I added was configuring DT_RUNPATH for the executable and
extensions.  This augments the set of directories the dynamic linker
searches for shared library dependencies.  If a directory in this list
contains the token "$ORIGIN", it will be expanded to the the directory
containing the program or library.  So by including "$ORIGIN/../lib"
in the runpath for bin/python3.6, I can make sure it will find the
libpython in the directory next to it, no matter where it happens to
be bind mounted.  You can find more information about this in the
ld.so(8) man page.

James.

-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Re: Experimental Python interpreter snap

[XiaoGuo Liu]

Hi James,

Nice. This is a nice example showing how to reduce a python snap package.
A few days ago, I also made a small example to make use of the python3
coming with the core at:

https://github.com/liu-xiao-guo/httpstat

In the above example, I in fact do not package the python. It works.

Thanks & best regards,
XiaoGuo

On Mon, Feb 20, 2017 at 10:16 AM, James Henstridge <
james.henstri...@canonical.com> wrote:

> To learn a bit more about I put together a snap for Python 3.6.0,
> which can be installed with:
>
> snap install --edge python36-jamesh
>
> You can then run "python36-jamesh.python3", which will give you the a
> Python shell running with strict confinement, with the full standard
> library available.
>
> Now I know Snapcraft already has support for packaging Python
> applications, so what benefits does this package add?  There were a
> few extra points in how I built the package:
>
> 1. the interpreter binary and extension modules all have appropriate
> $ORIGIN relative rpath set.
>
> 2. a sitecustomize.py is provided that will add
> $SNAP/lib/python3.6/site-packages to sys.path. (more on why this is
> useful later)
>
> This makes the interpreter fully relocatable in the file system while
> still being able to find the bundled libraries.  In turn, this means
> the interpreter is functional when exported to another snap via the
> content interface.
>
> To demonstrate this, I put together a trivial "hello world" snapcraft
> project here:
>
> https://github.com/jhenstridge/python-snap-pkg/tree/master/examples/hello-
> world
>
> After building this package, it can be run after installing and
> connecting the interface:
>
> $ snap install --dangerous hello-world_0.1_amd64.snap
> hello-world 0.1 installed
> $ snap connect hello-world:python3 python36-jamesh:python3
> $ hello-world
> Hello world!
>
> Since the hello-world snap doesn't actually include Python, it is
> quite light weight (4 kB, which I think is as small as a squashfs
> gets).  The space savings may not be that great with a single snap
> (the interpreter snap is almost 20MB), but the space savings increase
> as you install more snaps using the interface.  It also means that we
> could upgrade to Python 3.6.1 (when it comes out) without rebuilding
> this snap.
>
> And since the interpreter is being run under the hello-world snap's
> confinement policy, it can do potentially do things the main
> "python36-jamesh.python3" binary can't.  For example, if you add the
> "network" plug, you'll be able to access the network.
>
> And the sitecustomize script will also mean the interpreter can locate
> packages shipped in the plug snap.
>
> I'd be interested in any suggestions or feedback about the snap.
>
> Thanks,
>
> James.
>
> --
> Snapcraft mailing list
> Snapcraft@lists.snapcraft.io
> Modify settings or unsubscribe at: https://lists.ubuntu.com/
> mailman/listinfo/snapcraft
>



-- 
XiaoGuo, Liu
-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Re: Experimental Python interpreter snap

[Spencer]

I thought a main feature of snaps was to include all dependencies so that they 
couldn't be changed out from underneath a package.  For example, my script was 
written for 3.6, but would be incompatible with a future release, say 4.0.

Still, the ability to share a dependency like the Python interpreter among 
python snaps may be a good idea if there are zillions of Python snaps.

At my work, we got tired of everyone maintaining their own local python 
installation, because we all ended up with slightly different versions of 
various python modules installed.  Worse, some modules installed for some while 
not for others.  So we started tracking a shared Python installation in git.  
One problem we found is that Python is not relocatable.  This showed up when 
people cloned our repository in a different place.  Are you sure that your 
snapped Python is relocatable?  If so, I'd like to know how that works.  Is the 
$ORIGIN variable standard?


> On Feb 19, 2017, at 7:16 PM, James Henstridge 
>  wrote:
> 
> To learn a bit more about I put together a snap for Python 3.6.0,
> which can be installed with:
> 
>snap install --edge python36-jamesh
> 
> You can then run "python36-jamesh.python3", which will give you the a
> Python shell running with strict confinement, with the full standard
> library available.
> 
> Now I know Snapcraft already has support for packaging Python
> applications, so what benefits does this package add?  There were a
> few extra points in how I built the package:
> 
> 1. the interpreter binary and extension modules all have appropriate
> $ORIGIN relative rpath set.
> 
> 2. a sitecustomize.py is provided that will add
> $SNAP/lib/python3.6/site-packages to sys.path. (more on why this is
> useful later)
> 
> This makes the interpreter fully relocatable in the file system while
> still being able to find the bundled libraries.  In turn, this means
> the interpreter is functional when exported to another snap via the
> content interface.
> 
> To demonstrate this, I put together a trivial "hello world" snapcraft
> project here:
> 
> https://github.com/jhenstridge/python-snap-pkg/tree/master/examples/hello-world
> 
> After building this package, it can be run after installing and
> connecting the interface:
> 
>$ snap install --dangerous hello-world_0.1_amd64.snap
>hello-world 0.1 installed
>$ snap connect hello-world:python3 python36-jamesh:python3
>$ hello-world
>Hello world!
> 
> Since the hello-world snap doesn't actually include Python, it is
> quite light weight (4 kB, which I think is as small as a squashfs
> gets).  The space savings may not be that great with a single snap
> (the interpreter snap is almost 20MB), but the space savings increase
> as you install more snaps using the interface.  It also means that we
> could upgrade to Python 3.6.1 (when it comes out) without rebuilding
> this snap.
> 
> And since the interpreter is being run under the hello-world snap's
> confinement policy, it can do potentially do things the main
> "python36-jamesh.python3" binary can't.  For example, if you add the
> "network" plug, you'll be able to access the network.
> 
> And the sitecustomize script will also mean the interpreter can locate
> packages shipped in the plug snap.
> 
> I'd be interested in any suggestions or feedback about the snap.
> 
> Thanks,
> 
> James.
> 
> -- 
> Snapcraft mailing list
> Snapcraft@lists.snapcraft.io
> Modify settings or unsubscribe at: 
> https://lists.ubuntu.com/mailman/listinfo/snapcraft

-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Experimental Python interpreter snap

[James Henstridge]

To learn a bit more about I put together a snap for Python 3.6.0,
which can be installed with:

snap install --edge python36-jamesh

You can then run "python36-jamesh.python3", which will give you the a
Python shell running with strict confinement, with the full standard
library available.

Now I know Snapcraft already has support for packaging Python
applications, so what benefits does this package add?  There were a
few extra points in how I built the package:

1. the interpreter binary and extension modules all have appropriate
$ORIGIN relative rpath set.

2. a sitecustomize.py is provided that will add
$SNAP/lib/python3.6/site-packages to sys.path. (more on why this is
useful later)

This makes the interpreter fully relocatable in the file system while
still being able to find the bundled libraries.  In turn, this means
the interpreter is functional when exported to another snap via the
content interface.

To demonstrate this, I put together a trivial "hello world" snapcraft
project here:

https://github.com/jhenstridge/python-snap-pkg/tree/master/examples/hello-world

After building this package, it can be run after installing and
connecting the interface:

$ snap install --dangerous hello-world_0.1_amd64.snap
hello-world 0.1 installed
$ snap connect hello-world:python3 python36-jamesh:python3
$ hello-world
Hello world!

Since the hello-world snap doesn't actually include Python, it is
quite light weight (4 kB, which I think is as small as a squashfs
gets).  The space savings may not be that great with a single snap
(the interpreter snap is almost 20MB), but the space savings increase
as you install more snaps using the interface.  It also means that we
could upgrade to Python 3.6.1 (when it comes out) without rebuilding
this snap.

And since the interpreter is being run under the hello-world snap's
confinement policy, it can do potentially do things the main
"python36-jamesh.python3" binary can't.  For example, if you add the
"network" plug, you'll be able to access the network.

And the sitecustomize script will also mean the interpreter can locate
packages shipped in the plug snap.

I'd be interested in any suggestions or feedback about the snap.

Thanks,

James.

-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Re: Determining the set of snapd capabilities

[Mark Shuttleworth]

On 19/02/17 12:07, Joseph Rushton Wakeling wrote:
> This would be super-useful.  Conversely, to what extent might it be
> possible to guide the package developer to a clear understanding of
> what capabilities a particular snap package assumes?

Hmm... that's a Turing-complete problem :)

However, a useful approach would be to run the snap through an automated
test process using a range of supported platforms. For example, spinning
up VMs of snapd-supporting distros, and testing the snap there.

As a precursor to that we would want to have health mechanism which
provide a standard way to know if a snap considers itself healthy. Then
we could run a service for developers which screens a snap for health
across a wide range of devices and VMs, including the commonly used
versions of snapd in those environments, to pick up potential problems
and make such recommendations.

Mark


-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Re: Determining the set of snapd capabilities

[Joseph Rushton Wakeling]

On 19/02/17 15:26, Sergio Schvezov wrote:

It was my intention to fully document this. I just did half of the job as I 
only added information about it under the python plugin update. Sorry about 
this. I have updated the release notes[1] so the next person doesn't fall into 
this time sync trap!


Oh, no apology needed!  FWIW I would suggest maybe giving an example (in the 
'classic confinement' section) of how to change the `command:` declaration in 
order to deal with this change.


You give a working example later in 'Setting up environment' but it's not 
obvious that this is related to classic confinement or a change in snapcraft 
behaviour.


--
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Re: Determining the set of snapd capabilities

[Sergio Schvezov]

On Sun, 19 Feb 2017 13:07:48 +0100, Joseph Rushton Wakeling wrote:
> On 19/02/17 12:21, Mark Shuttleworth wrote:
>> Step 4, announce new capabilities on the list
>>
>> In many cases, the existence of a new capability is meaningful for a
>> large number of snapcrafters, lets share the beta documentation on the
>> list when the capability lands in a --proposed or --candidate release of
>> snapd+snapcraft.
>
> Not just capabilities, but any breaking changes in behaviour.  For example, 
> while Sergio's support has been great regarding my recent experiences with 
> snapcraft 2.27, I think I could have avoided taking his time if the release 
> notes had mentioned that classic snaps would no longer see 
> environment variables 
> set in app wrapper scripts (and ideally, advised on the changes required to 
> `snapcraft.yaml` files).

It was my intention to fully document this. I just did half of the job as I 
only added information about it under the python plugin update. Sorry about 
this. I have updated the release notes[1] so the next person doesn't fall into 
this time sync trap!

Cheers
Sergio

[1] https://github.com/snapcore/snapcraft/releases/tag/2.27

-- 


-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Re: Determining the set of snapd capabilities

[Joseph Rushton Wakeling]

On 19/02/17 12:21, Mark Shuttleworth wrote:

We have a nice mechanism to ensure that snaps which use newer
capabilities don't end up on systems with older snapd's that don't
support those capabilities, which is the 'assumes' keyword. This email
is a proposal to make that usable.


This would be super-useful.  Conversely, to what extent might it be possible to 
guide the package developer to a clear understanding of what capabilities a 
particular snap package assumes?



Step 3, introduce capabilities as 'beta-capability'


Sounds good.  Working with classic snaps (for example) is obviously chasing a 
somewhat moving target right now.  That was clear up front, but it would be nice 
to have it explicitly documented.



Step 4, announce new capabilities on the list

In many cases, the existence of a new capability is meaningful for a
large number of snapcrafters, lets share the beta documentation on the
list when the capability lands in a --proposed or --candidate release of
snapd+snapcraft.


Not just capabilities, but any breaking changes in behaviour.  For example, 
while Sergio's support has been great regarding my recent experiences with 
snapcraft 2.27, I think I could have avoided taking his time if the release 
notes had mentioned that classic snaps would no longer see environment variables 
set in app wrapper scripts (and ideally, advised on the changes required to 
`snapcraft.yaml` files).



Step 5, nobody expects the Spanish Inquisition.


But I hope all snapcraft developers have comfy chairs? :-)

Thanks & best wishes,

-- Joe

--
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Determining the set of snapd capabilities

[Mark Shuttleworth]

We have a nice mechanism to ensure that snaps which use newer
capabilities don't end up on systems with older snapd's that don't
support those capabilities, which is the 'assumes' keyword. This email
is a proposal to make that usable.

The mechanism is mentioned in the snapcraft docs at
https://github.com/snapcore/snapcraft/blob/master/docs/snapcraft-syntax.md
which is published at https://snapcraft.io/docs/build-snaps/syntax but
not clarified anywhere else that I can see. I have been unable to find a
list of the available capabilities that can be assumed. Whatever else is
true, the fact that I couldn't find it despite the help of the AIs at
Google suggests that we can improve the discoverability and usefulness
of this potentially very useful capability :)


Step 1, show what capabilities a snapd offers.
https://bugs.launchpad.net/snapd/+bug/1665995

'snap --version' is a nice way to know what versions are relevant for a
particular system. That would be a good place to show the capabilities
that can be assume'd for snaps being installed on that system. It
doesn't require a new command and it is in many ways more relevant to
developers than the particular version. At least, the intent of
capabilities is to have fewer version-specific dependencies, and more
capability-specific dependencies!


Step 2, reference 'snap version' in the docs

The existing docs for snapcraft.yaml and snap.yaml should both make
clear reference to 'snap version' as a way to discover these
capabilities. Also, there should be express documentation on each of
them, organized as such. So the docs should say "use 'snap version' to
see what a particular system supports, and see  for documentation
about each of those capabilities". That documentation should explain any
constraints or usage of the capabilities.


Step 3, introduce capabilities as 'beta-capability'

When we land new features, there are often rough edges. For example,
there is a new capability to support direct setting of environment
variables in snap.yaml which is super useful (many snaps have wrappers
just to do this, so that's one less layer needed in the common case :)).
However, there is a bug in the implementation which means you sometimes
get a newline on the environment variable, which is unhelpful. I think
this should be a beta-capability till we know it is fully fleshed out.


Step 4, announce new capabilities on the list

In many cases, the existence of a new capability is meaningful for a
large number of snapcrafters, lets share the beta documentation on the
list when the capability lands in a --proposed or --candidate release of
snapd+snapcraft.


Step 5, nobody expects the Spanish Inquisition.


Mark



-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Re: LDC compiler snap issues on 14.04 (ABI compatibility?)

[Joseph Rushton Wakeling]

On 16/02/17 05:10, Sergio Schvezov wrote:

I'll check your snap first thing once I am at a computer again. From experience
with classic confinement  though, stage-packages to provide runnables are most
of the time the root cause. There was discusssion on having ld set
--library-path in be core snap to have called binaries not reach out to the
system and get confused.


Just to follow up on this: I was able to rebuild the package using snapcraft 
2.27 and `cleanbuild`, and as far as I can tell the result works on 14.04 
without a hitch.  So thank you for all the great work here! :-)


I have an outstanding issue with a linker plugin included in the snap, but that 
supports an optional feature and I didn't expect it to work with 14.04.  In any 
case I'll follow up on that in a separate thread once I have a clearer picture 
of exactly what might be the problem.


Thanks again & best wishes,

-- Joe

--
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft