Re: your mail
Hi Leo, great response, On Tue, Mar 28, 2017 at 08:56:25PM -0600, Leo Arias wrote: > It's stored in $SNAP_USER_COMMON, ~/snap/bitcoin/common. > This was the only ugly part. There are some bugs in bitcoin reported > [...] > problem is not yet affecting anyone else. So, for now we are patching > bitcoin's default data dir [2]. bitcoin may be the most extreme example I can think of quickly but surely it's not alone in this. > > - Have you run into problems moving BDB forward or backwards when > > upgrading or downgrading? Is BDB used in the storage of the blockchain > > or is it used for 'simpler' snap storage? > > I'm not sure what do you mean with simpler snap storage. There's no > storage specific to the snap. Where the bitcoin code uses the default > data dir, that's unversioned. Where it uses HOME or XDG dirs directly, > that data is versioned. I don't have high hopes for better data > handling because the bugs I found were from a couple of years ago, and > mentioned the split of the wallet code that was supposed to happen > soon at that time, but never happened. I was mostly curious if BDB was being used for the blockchain or user preferences or other metadata; I had some impression many years ago that BDB was touchy, so I wondered if you had run into trouble. > > It feels like bitcoin's 100-ish gigabyte blob makes an interesting > > constraint that many simpler tools may not need to address. > > That was precisely my thinking. Would you like to help us maintaining > the snap until we make the transfer upstream? It would be great to > increase our bus factor, and more hands will make it easier to improve > the packaging. Thanks for the invitation but I'm already feeling vastly oversubscribed. :( Thanks signature.asc Description: PGP signature -- Snapcraft mailing list Snapcraft@lists.snapcraft.io Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/snapcraft
Re: your mail
Hello Seth, In many things, this works just like the tar.gz that you download from the bitcoin website. On Tue, Mar 28, 2017 at 4:49 PM, Seth Arnold wrote: > - At what point do you download the blockchain? It will download the blockchain when you open the qt client. > - Where does the blockchain get stored? It's stored in $SNAP_USER_COMMON, ~/snap/bitcoin/common. This was the only ugly part. There are some bugs in bitcoin reported to allow more control over the data files. But currently, it stores everything in $HOME, which inside the snap is redirected to $SNAP_USER_DATA. This path is versioned, but we don't want to copy the entire blockchain every time the snap is updated. A few days ago I proposed an option to have an unversioned home [1], but it seems this problem is not yet affecting anyone else. So, for now we are patching bitcoin's default data dir [2]. > - How are interrupted downloads handled? Nothing specific to the snap here, bitcoin handles it. You close the app or get disconnected and the download stops. You open the app or reconnect and it resumes. > - How are package upgrades handled? Is a copy of the blockchain made? No, it's kept in the common dir, unversioned. > - How are package downgrades handled? Is a copy of the blockchain made? Nothing specific to the snap here yet. Currently, if the upgrade is not backwards compatible, the same will happen as if you open an older bitcoin version with a directory holding newer data. We could do fancier things on upgrades and downgrades with the epoch work, but that's not yet documented, I'm not even sure if it's complete. > - Have you run into problems moving BDB forward or backwards when > upgrading or downgrading? Is BDB used in the storage of the blockchain > or is it used for 'simpler' snap storage? I'm not sure what do you mean with simpler snap storage. There's no storage specific to the snap. Where the bitcoin code uses the default data dir, that's unversioned. Where it uses HOME or XDG dirs directly, that data is versioned. I don't have high hopes for better data handling because the bugs I found were from a couple of years ago, and mentioned the split of the wallet code that was supposed to happen soon at that time, but never happened. We haven't had problems with updates and downgrades, but well, this is the first version pushed to the store, no much room for failure there. What's cool with the snap is that we get a new version pushed to edge each day, so we can catch any potential breakages right after they are introduced. And as soon as upstream starts tagging candidates for the 0.14.1 release we can move them to the candidate channel and easily test updates from 0.14 with snap refresh bitcoin --candidate. However, these checks are not yet automated, so the success of the snap depends on the community testing before we push to stable. That's the reason for this call for testing. During the experiments with IPFS [3] we already had 3 or 4 successful stable updates, and the process is just lovely and it has worked without any incidents. That's why I'm interested in this more complex scenario with bitcoin. > It feels like bitcoin's 100-ish gigabyte blob makes an interesting > constraint that many simpler tools may not need to address. That was precisely my thinking. Would you like to help us maintaining the snap until we make the transfer upstream? It would be great to increase our bus factor, and more hands will make it easier to improve the packaging. pura vida. [1] https://lists.ubuntu.com/archives/snapcraft/2017-March/003494.html [2] https://github.com/elopio/blockchain-snaps/blob/master/bitcoin/snap/patches/default_data_dir.patch [3] http://elopio.net/blog/ipfs-crowdtesting/ -- Snapcraft mailing list Snapcraft@lists.snapcraft.io Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/snapcraft
Re: your mail
On Tue, Mar 28, 2017 at 11:20:41AM -0600, Leo Arias wrote: > Here are the sources for the snap and CI scripts, in case you want to > verify them or build it yourself: > https://github.com/elopio/blockchain-snaps/blob/master/bitcoin/snap/snapcraft.yaml Hi Leo, it looks like you've documented this fairly well but I'm not familiar enough with the details of snap packaging to know how it actually works in practice. This might serve as a good example for the snap community: - At what point do you download the blockchain? - Where does the blockchain get stored? - How are interrupted downloads handled? - How are package upgrades handled? Is a copy of the blockchain made? - How are package downgrades handled? Is a copy of the blockchain made? - Have you run into problems moving BDB forward or backwards when upgrading or downgrading? Is BDB used in the storage of the blockchain or is it used for 'simpler' snap storage? It feels like bitcoin's 100-ish gigabyte blob makes an interesting constraint that many simpler tools may not need to address. Thanks signature.asc Description: PGP signature -- Snapcraft mailing list Snapcraft@lists.snapcraft.io Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/snapcraft
Re:
this is really cool leo. thanks for sharing. i'll definitely give it a try. /manik On Tue, Mar 28, 2017 at 10:20 AM, Leo Arias wrote: > Hello, > > It has never been easier, snapcraft and profit! > > For a couple of weeks we have been figuring out how to confine > bitcoin. This is one of the most important use cases of the snap > security because you don't want any program messing with your digital > wallet. And when there's a vulnerability in the code, all the clients > should be updated as fast as possible to keep the network sane. With > snaps, just push a fixed version and see everybody update without any > work from their side. > > We now need help testing the snap before making it public: > > $ sudo snap install bitcoin --candidate > > Then launch it from the dash or running the bitcoin.qt command. > > This was built from the tag of the latest released bitcoin and pushed > to the store, all automatically by travis. But, be warned, you will be > the first users so we might still find unforeseen problems. Use your > bitcoins with caution during the candidate phase, or just test without > any of your money. > > Here are the sources for the snap and CI scripts, in case you want to > verify them or build it yourself: > https://github.com/elopio/blockchain-snaps/blob/master/ > bitcoin/snap/snapcraft.yaml > > Another interesting detail is that most of the cryptocurrencies out > there are forks of the bitcoin source code, so they will all need a > similar snapcraft.yaml. The other one we pushed is bitcoin-unlimited, > which is promoting a hard fork of the bitcoin network to remove the > block size limit. You can have both snaps installed to give them a try > and choose the network you want to support, because both will be fully > confined and independent: > > $ sudo snap install bitcoin-unlimited --candidate > > Of course, this would require twice the time and space to download > both blockchains. > > Thanks to Gal Buki (torusJKL) for his help. > > Expect more options in the store soon. > > pura vida > -- > ¡paz y baile! > http://www.ubuntu.com > > -- > Snapcraft mailing list > Snapcraft@lists.snapcraft.io > Modify settings or unsubscribe at: https://lists.ubuntu.com/ > mailman/listinfo/snapcraft > -- Snapcraft mailing list Snapcraft@lists.snapcraft.io Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/snapcraft
[no subject]
Hello, It has never been easier, snapcraft and profit! For a couple of weeks we have been figuring out how to confine bitcoin. This is one of the most important use cases of the snap security because you don't want any program messing with your digital wallet. And when there's a vulnerability in the code, all the clients should be updated as fast as possible to keep the network sane. With snaps, just push a fixed version and see everybody update without any work from their side. We now need help testing the snap before making it public: $ sudo snap install bitcoin --candidate Then launch it from the dash or running the bitcoin.qt command. This was built from the tag of the latest released bitcoin and pushed to the store, all automatically by travis. But, be warned, you will be the first users so we might still find unforeseen problems. Use your bitcoins with caution during the candidate phase, or just test without any of your money. Here are the sources for the snap and CI scripts, in case you want to verify them or build it yourself: https://github.com/elopio/blockchain-snaps/blob/master/bitcoin/snap/snapcraft.yaml Another interesting detail is that most of the cryptocurrencies out there are forks of the bitcoin source code, so they will all need a similar snapcraft.yaml. The other one we pushed is bitcoin-unlimited, which is promoting a hard fork of the bitcoin network to remove the block size limit. You can have both snaps installed to give them a try and choose the network you want to support, because both will be fully confined and independent: $ sudo snap install bitcoin-unlimited --candidate Of course, this would require twice the time and space to download both blockchains. Thanks to Gal Buki (torusJKL) for his help. Expect more options in the store soon. pura vida -- ¡paz y baile! http://www.ubuntu.com -- Snapcraft mailing list Snapcraft@lists.snapcraft.io Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/snapcraft
Re: Strategy for Snaps with Debugging Symbols
Just wanted to add that if we went for the approach where we replace an application with its debug version, another issue is that complex applications may behave differently depending on added instrumentation in a debug build. For example, if you added debugging code that slows down an application in general it may stop triggering race conditions or certain code paths. In other words, there is a difference between running a 'debugging build' and a 'release build with some debugging symbols loaded'. Best Regards, Dmitrii Shcherbakov Dmitrii Shcherbakov | Canonical Field Software Engineer dmitrii.shcherba...@canonical.com IRC (freenode): Dmitrii-Sh On Tue, Mar 28, 2017 at 10:44 AM, Christian Ehrhardt wrote: > > On Tue, Mar 28, 2017 at 8:13 AM, Alfonso Sanchez-Beato > wrote: >> >> > One idea for snaps is to use -dbg tracks which are going to contain >> > snaps >> > with embedded debugging symbols. Of course, some snaps may contain a >> > number >> > of services so you'd have to customize your build process and create >> > additional automation around snapcraft to make it more manageable. >> > >> >> I would love to see this, it would ease a lot the debugging pain for >> snaps. >> IMHO using a debugging track makes lots of sense, provided gdbserver/gdb >> in >> Ubuntu Core can be configured to find the symbols automatically. > > > It might come down to teach gdb and co certain path. > On the classic packaging is is about /usr/lib/debug prefixing the canonical > path of the binary. > > So lib and lib+debug symbols are: > $ file > /usr/lib/debug/usr/lib/libvirt/connection-driver/libvirt_driver_interface.so > /usr/lib/debug/usr/lib/libvirt/connection-driver/libvirt_driver_interface.so: > ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, > BuildID[sha1]=1d163a878a64d3f180806e713ad0cb9363492316, not stripped > $ file /usr/lib/libvirt/connection-driver/libvirt_driver_interface.so > /usr/lib/libvirt/connection-driver/libvirt_driver_interface.so: ELF 64-bit > LSB shared object, x86-64, version 1 (SYSV), dynamically linked, > BuildID[sha1]=1d163a878a64d3f180806e713ad0cb9363492316, stripped > > The matching flag build-id in there make sure programs can check they match > and pick it up automatically [1]. > > Now a binary of a snap appears like in a versioned path, but there is the > current link which should be stable. > So the binary of e.g. the latest vlc snap is always in: > /snap/vlc/current/bin/vlc > > That would mean if there would be symbols at > "/usr/lib/debug/snap/vlc/current/bin/vlc.debug" would likely work as of > today in case you want to link yourself something up. > Actually I think it would be fair that snaps could "own" the snap subdir of > the global debug dir being "/usr/lib/debug/snap" and mirror the snap layout > there but populated with the debug symbols. > > Also I agree that this should not be a manual task of some sort, just like > the .deb packaging an optional -dbg snap or such could fall out of any > build that does contain the non-stripped binaries. > > All that is true for a classic gdb, but a snapped gdb will itself run > confined and in its local path to some extend - that has to be considered as > well going forward. > > Also the whole command semantic has to be sound and tasteful for those who > want to use it, while not distracting non debugging snap users too much. > As the debug snaps clearly have a reduced subset of actions (e.g. no > interfaces) maybe one option would be to hide that reduced subset all behind > one subcommand like: "snap dbg [find|list|install|...]" ? > > [1]: https://sourceware.org/gdb/onlinedocs/gdb/Separate-Debug-Files.html > > -- > Christian Ehrhardt > Software Engineer, Ubuntu Server > Canonical Ltd -- Snapcraft mailing list Snapcraft@lists.snapcraft.io Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/snapcraft
Re: Moving from strict to classic confinement
Hi Renato, 2017-03-27 15:31 GMT+02:00 Renato Filho : > 2017-03-27 4:22 GMT-03:00 Eloy García (PC Actual) < > eloy.garcia@gmail.com>: > > Hello everybody. > > > > I currently have a snap package published on the store. It is called > > wallpaperdownloader and it is a Java-based application. So far, it has > been > > packaged using the strict confinement. The application basically > downloads > > wallpapers from the Internet and sets the wallpaper when the user > requests > > it. Because there are a lot of desktop environments, I have ended up > using > > a lot of Linux commands from the java application in order to achieve > these > > goals. For some DE there is no problem at all: for example, using > gsettings > > interface, GNOME Shell and Unity change the background flawlessly. MATE > is > > working properly too (I had to include mate-desktop-common as stage > > package) but XFCE and KDE... no way. In addition, some of the most simple > > features (open a file explorer for example) don't work. > > I am not sure what exactly the problem that you have to change KDE > wallpapers but it can be related with the fact that you are using > "desktop-gtk3", because this only prepare the environment for gtk > apps. Some environment vars will be missing for Qt apps. > > My main problem is that, first, I need to know Plasma version. in the native installation, I execute, form Java, "plasmashell --version" command, which gives me this version. This command in the snap confinement doesn't exit. > The options that you have are: > > 1 - create a wrapper file and export the vars by yourself. > 2 - Use "ubuntu-app-platform" that contains libraries and a launcher > compatible with most gtk and qt apps. > > Do you think that using ubuntu-app-platform part could have, for example, the command I described above?. In order to change the wallpaper I have to invoke qdbus. Same problem here. Using this part will the command be available? > I hope this help > > Thank you very much!!! :) > -- > Snapcraft mailing list > Snapcraft@lists.snapcraft.io > Modify settings or unsubscribe at: https://lists.ubuntu.com/ > mailman/listinfo/snapcraft > -- Eloy García Almadén -- Snapcraft mailing list Snapcraft@lists.snapcraft.io Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/snapcraft
Re: Moving from strict to classic confinement
2017-03-27 15:14 GMT+02:00 Sergio Schvezov : > On Mon, 27 Mar 2017 09:22:03 +0200, Eloy García (PC Actual) wrote: > > Hello everybody. > > > > I currently have a snap package published on the store. It is called > > wallpaperdownloader and it is a Java-based application. So far, it has > been > > packaged using the strict confinement. The application basically > downloads > > wallpapers from the Internet and sets the wallpaper when the user > requests > > it. Because there are a lot of desktop environments, I have ended up > using > > a lot of Linux commands from the java application in order to achieve > these > > goals. For some DE there is no problem at all: for example, using > gsettings > > interface, GNOME Shell and Unity change the background flawlessly. MATE > is > > working properly too (I had to include mate-desktop-common as stage > > package) but XFCE and KDE... no way. In addition, some of the most simple > > features (open a file explorer for example) don't work. > > > > I have been testing the classic confinement and all these features work! > In > > addition, I don't need to include some dependencies like desktop-gtk3 or > > use a shell script wrapper to launch the application. Then, I'm > considering > > to move my snap package from stric to classic confinement but i don't > know > > the possible implications: > > > > 1.- Are the interfaces still needed when using classic confinement? > > No, they are not. > > > 2.- From the user point of view: if wallpaperdownloader is already > > installed and I change the confinement, when snapd upgrades it, will it > > work flawlessly? I mean, the upgrade process will be automatic or it will > > require manual intervention from the user? > > They will need updating. HOME in strict confinement points to the > segregated data stores, in classic it is the traditional home. > > > 3.- As I see classic confinement, it is a feature to make snap packages > > more easily but they only will work on a "classic" Ubuntu system. If > Ubuntu > > is finally migrated to a Snappy system, I guess classic confinement won't > > work and all these packages should be migrated to strict confinement > again, > > am I right? > > This is correct, a pure snappy based system is not a classic system so > classic confinement does not apply. > > > Thank you all for your time :) > > A few more considerations: > > - I would stick to strict having gotten this far and just work on > unblocking yourself with interfaces or other means. > Ok Sergio, you convinced me :) It is true. i have had to deal with a lot of problems until getting an almost totally functional snap for my application, and if the user should have to upgrade manually the snap... I'll stick on strict mode! :D > - strict confinment means you have a stable base (core), while classic > means you will have an unstable one (well, not unstable, just different > depending on the system where the snap is installed), which means you will > have a different set of bugs to deal with depending on the base classic > system where the snap is installed. > - you shouldn't rely on libraries on classic systems, not all of them with > have gtk3 nor will all of them be ABI compatible with the glibc stuff your > snap uses (this might be a different story with java). > The "problem" with wallpaperdownloader is that I try to be fully compatible with the main Desktop Environments in Linux (at the moment, the application works fine with Unity 7, Gnome Shell. Mate, XFCE and KDE (not yet for vesion 2.6 but I'm implementing it for version 2.7)). In order to tell the DE to change the wallpaper I have to use some tools and commands which are available only for this DE and snap packaging is a challenge for me in order to take into account all this environments if I cannot "rely on" the host DE itself (I don't know if I'm expressing myself clearly, sorry for that) > - There is a missing feature in snapd/snap-confine to restrict the library > paths the classic confined snap can see, so you need to make sure this is > the case for you. This can be mitigated in snapcraft if you build all > runnables from parts (in your case the jre which I bet wouldn't be easy). > - I am going to be talking again a bout classic confined snaps in one of > the upcoming Ubuntu Testing days, so if interested, keep an eye on that ;-) > > I'll keep an eye on, for sure! Thank you very much!! Best, Eloy -- Eloy García Almadén -- Snapcraft mailing list Snapcraft@lists.snapcraft.io Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/snapcraft
Re: Install snaps in armhf chroot
Hi John, thanks for the quick reply! So building the image in a native armhf environment should do the trick. Regards, Simon On 2017-03-28 11:58, John Lenton wrote: > On 28 March 2017 at 10:35, Simon Stürz wrote: >> I'm currently building an armhf system using live-build and running >> snapd on it. I wanted to pre-install snaps during the image build >> process and run into several problems. >> Using qemu-arm-static in the chroot I get following error: > the go developers decided to not support being run in qemu: > > https://github.com/golang/go/issues/13024 > > “As beefy arm boards are widely available, I don't think we should > invest into qemu usermode emulation any more. It's slower than the > more powerful arm boards, and it doesn't support multithreaded > programs well.” > -- Snapcraft mailing list Snapcraft@lists.snapcraft.io Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/snapcraft
Re: Install snaps in armhf chroot
On 28 March 2017 at 10:35, Simon Stürz wrote: > > I'm currently building an armhf system using live-build and running > snapd on it. I wanted to pre-install snaps during the image build > process and run into several problems. > Using qemu-arm-static in the chroot I get following error: the go developers decided to not support being run in qemu: https://github.com/golang/go/issues/13024 “As beefy arm boards are widely available, I don't think we should invest into qemu usermode emulation any more. It's slower than the more powerful arm boards, and it doesn't support multithreaded programs well.” -- Snapcraft mailing list Snapcraft@lists.snapcraft.io Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/snapcraft
Install snaps in armhf chroot
Hi everyone! I'm currently building an armhf system using live-build and running snapd on it. I wanted to pre-install snaps during the image build process and run into several problems. Using qemu-arm-static in the chroot I get following error: $ snap install core fatal error: rt_sigaction failure runtime stack: runtime.throw(0xf6d56af9, 0x14) /usr/lib/go-1.7/src/runtime/panic.go:566 +0x80 runtime.setsigstack(0x21) /usr/lib/go-1.7/src/runtime/os_linux.go:426 +0xf4 runtime.initsig(0xf6febc00) /usr/lib/go-1.7/src/runtime/signal1_unix.go:78 +0x1b8 runtime.mstart1() /usr/lib/go-1.7/src/runtime/proc.go:1122 +0xf0 runtime.mstart() /usr/lib/go-1.7/src/runtime/proc.go:1096 +0x84 goroutine 1 [runnable]: runtime.main() /usr/lib/go-1.7/src/runtime/proc.go:106 runtime.goexit() /usr/lib/go-1.7/src/runtime/asm_arm.s:998 +0x4 goroutine 17 [syscall, locked to thread]: runtime.goexit() /usr/lib/go-1.7/src/runtime/asm_arm.s:998 +0x4 Looks like this is go / qemu-arm bug and I have found several bug reports on this issue without any helping information: https://bugs.launchpad.net/ubuntu/+source/golang/+bug/1501651 https://bugs.launchpad.net/ubuntu/+source/golang-1.6/+bug/1603846 Then I checked how the ubuntu-image tool works and pre-install snaps (but I didn't get the trick): https://git.launchpad.net/ubuntu-image/tree/ubuntu_image/builder.py?h=xenial#n117 https://git.launchpad.net/ubuntu-image/tree/ubuntu_image/helpers.py?h=xenial#n102 Is there any other possibility to install snaps in a armhf chroot? Thanks in advance, Simon -- Snapcraft mailing list Snapcraft@lists.snapcraft.io Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/snapcraft
a mechanism to make the snap kernel updrade failed
Hi all,
we are testing the our kernel snap (Linux4.9) for a orangepi plus 2e board.
We use the example showed at
https://github.com/snapcore/pi3-gadget/blob/master/prebuilt/uboot.env.in for
configuring u-boot to support the kernel update.
This is the code that we use for configuring uboot to support the kernel update:
snappy_boot=if test "${snap_mode}" = "try"; then setenv snap_mode "trying";
saveenv; if test "${snap_try_core}" != ""; then setenv snap_core
"${snap_try_core}"; fi; if test "${snap_try_kernel}" != ""; then setenv
snap_kernel "${snap_try_kernel}"; fi; elif test "${snap_mode}" = "trying"; then
setenv snap_mode ""; saveenv; fi; run loadfiles; setenv mmcroot
"/dev/disk/by-label/writable ${snappy_cmdline} snap_core=${snap_core}
snap_kernel=${snap_kernel}"; run mmcargs; bootz ${kernel_addr}
${initrd_addr}:${initrd_size} ${fdt_addr}
snappy_cmdline=net.ifnames=0 init=/lib/systemd/systemd ro panic=-1 fixrtc
snappy_mode=regular
snappy_trial_boot=0
This is the starting point.
1) is it the correct way to get to declare u-boot boot.cmd and setting for
first gadget deploy? what are the defconfig to use? U-boot version? patch to
apply to u-boot code?
We already tested successful upgrade of the kernel snap.
Now our goal is the possibility to simulate or provocate a failed upgrade for
kernel snap.
2) Any suggestions on how to trigger this situation?
PS: some other question about u-boot configuration:
3) we don't find documentation about snap_ variable usage: it seem that the
glue between this type of variable and update process is some action made by
snapd;
Where could we find this mechanism in the code or other place - e.g test ?
Thanks in advance for any help.
Cheers,
Nicolino
--
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/snapcraft
Re: Modular Application Updates: Libvirt and QEMU
On Mon, Mar 27, 2017 at 9:13 PM, Dmitrii Shcherbakov < dmitrii.shcherba...@canonical.com> wrote: > > > TL;DR: Putting libvirt and QEMU into the same snap removes an ability > to update them independently and only use new QEMU binaries after VM > shutdown. > [...] 7 If a QEMU process is terminated via SIGTERM or SIGKILL, the guest > kernel page cache and buffer cache will not be dropped which will > highly likely cause a file system corruption. > There are hooks you can link in yourself for upgrades IIRC. That could be used to at least gracefully shut them down - but I agree that there should be no reason to do so at all. The qemu's should continue to run through and after the update. [...] > The idea with any 'classic' package management system (for debs, rpms > etc.) is as follows: > > 1 Updates move new files over the old ones. That is, shared objects > and binaries are unlinked but not overwritten - if there is still a > process that has a file open (or mmaped which requires a file to be > open) an old inode and the related data on a file system is kept until > the reference count is zero; > > 2 Running programs can use old binaries and shared objects which they > have open until restart > [...] > 1 A new squashfs and an old squash fs are obviously different file > systems - hence inodes refer to different file systems; > > 2 All processes are killed during an update unconditionally and the > new file system is used to run new processes; > Yeah for server side things with a longer lifecycle that doesn't seem right. > 3 Some libraries are taken from the core snap's file system which > remains the same (but may change as the core snap may have been > updated before while a particular snap used an old version of it). > In some sense the squashfs entry points can be considered your new inode. All new application starts should be from the new version, but any old program could continue to run from the old content. That would be true for core-snap and application snap - only once all old refs are gone the old version can "really" go away. So think of an upgrade: PRE: content in /snap/app/oldver/foo UPGRAD adds: /snap/app/newver/foo UPGRADE changes: /snap/app/current is set to newver But /snap/app/oldver/foo would stay around and running applications kept alive. Only once the last one is gone /snap/app/oldver would completely vanish. IIRC we keep it around anyway to be able to roll back right [1]? We already make sure nothing new is started from old. Maybe it is just a way more advanced garbage collection and a change to the update behavior to leave things alive. Also on the "killing on update" as a major change in behavior - and to take it a bit out of the qemu/libvirt example. If I watch a movie on one screen in my snap vlc and in another console would refresh all snaps on my system I'd in no way expect it to kill my running vid. I haven't tried, but according to the report here that is what would happen right? I'd expect it to continue to run and whenever I start a new vlc next time it will be the new upgraded one. BTW - I realized that "snap remove vlc" leaves it alive, so refresh being harder than remove to a running app seems even more wrong. Thanks for sharing your thoughts Dimitri, I think you uncovered an important and interesting part of the snap implications which is worth to be thought through in detail. I hope the snap'xperts can jump in with a deeper view onto all this - especially on all the snap vs snap_data considerations that have to go into this when running from old+new at the same time would become allowed. [1]: https://developer.ubuntu.com/en/snappy/guides/garbage/ -- Snapcraft mailing list Snapcraft@lists.snapcraft.io Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/snapcraft
Re: Strategy for Snaps with Debugging Symbols
On Tue, Mar 28, 2017 at 8:13 AM, Alfonso Sanchez-Beato < alfonso.sanchez-be...@canonical.com> wrote: > > One idea for snaps is to use -dbg tracks which are going to contain snaps > > with embedded debugging symbols. Of course, some snaps may contain a > number > > of services so you'd have to customize your build process and create > > additional automation around snapcraft to make it more manageable. > > > > I would love to see this, it would ease a lot the debugging pain for snaps. > IMHO using a debugging track makes lots of sense, provided gdbserver/gdb in > Ubuntu Core can be configured to find the symbols automatically. It might come down to teach gdb and co certain path. On the classic packaging is is about /usr/lib/debug prefixing the canonical path of the binary. So lib and lib+debug symbols are: $ file /usr/lib/debug/usr/lib/libvirt/connection-driver/libvirt_driver_interface.so /usr/lib/debug/usr/lib/libvirt/connection-driver/libvirt_driver_interface.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=1d163a878a64d3f180806e713ad0cb9363492316, not stripped $ file /usr/lib/libvirt/connection-driver/libvirt_driver_interface.so /usr/lib/libvirt/connection-driver/libvirt_driver_interface.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=1d163a878a64d3f180806e713ad0cb9363492316, stripped The matching flag build-id in there make sure programs can check they match and pick it up automatically [1]. Now a binary of a snap appears like in a versioned path, but there is the current link which should be stable. So the binary of e.g. the latest vlc snap is always in: /snap/vlc/current/bin/vlc That would mean if there would be symbols at "/usr/lib/debug/snap/vlc/current/bin/vlc.debug" would likely work as of today in case you want to link yourself something up. Actually I think it would be fair that snaps could "own" the snap subdir of the global debug dir being "/usr/lib/debug/snap" and mirror the snap layout there but populated with the debug symbols. Also I agree that this should not be a manual task of some sort, just like the .deb packaging an optional -dbg snap or such could fall out of any build that does contain the non-stripped binaries. All that is true for a classic gdb, but a snapped gdb will itself run confined and in its local path to some extend - that has to be considered as well going forward. Also the whole command semantic has to be sound and tasteful for those who want to use it, while not distracting non debugging snap users too much. As the debug snaps clearly have a reduced subset of actions (e.g. no interfaces) maybe one option would be to hide that reduced subset all behind one subcommand like: "snap dbg [find|list|install|...]" ? [1]: https://sourceware.org/gdb/onlinedocs/gdb/Separate-Debug-Files.html -- Christian Ehrhardt Software Engineer, Ubuntu Server Canonical Ltd -- Snapcraft mailing list Snapcraft@lists.snapcraft.io Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/snapcraft
