You're making progress, now the two LXD daemons are definitely talking to each other. You're running into CRIU issues :)
If you did not intend to do live migration, then stop the source container and try moving it again. If you did intend to use live migration, then there are quite a few issues still being worked on: https://launchpad.net/ubuntu/+source/criu/+bugs Yours appears to be number 3 on that list. You can workaround it by unmounting /sys/kernel/debug/tracing/ and /sys/kernel/debug/ in your source container. On Wed, Aug 17, 2016 at 03:11:38PM -0700, jjs - mainphrame wrote: > Aw, shucks, it was looking like it might work - but no joy. > > root@olympia:~# lxc config set core.https_address 192.168.111.193:8443 > root@olympia:~# lxc move kangal lxd1: > error: Error transferring container data: migration restore failed > (00.192913) 1: Error (mount.c:2406): mnt: Can't mount at > ./sys/kernel/debug: Invalid argument > (00.206829) Error (cr-restore.c:1352): 23251 killed by signal 9 > (00.255466) Error (cr-restore.c:2182): Restoring FAILED. > > Any ideas from the error message? The only thing I see in the logs is this, > only sending side: > Aug 17 14:49:15 olympia kernel: [317636.764104] cgroup: new mount options > do not match the existing superblock, will be ignored > Aug 17 14:49:16 olympia kernel: [317636.897001] cgroup: new mount options > do not match the existing superblock, will be ignored > Aug 17 14:49:16 olympia kernel: [317636.965993] cgroup: new mount options > do not match the existing superblock, will be ignored > Aug 17 14:49:16 olympia kernel: [317637.025985] cgroup: new mount options > do not match the existing superblock, will be ignored > Aug 17 14:49:16 olympia kernel: [317637.099008] cgroup: new mount options > do not match the existing superblock, will be ignored > Aug 17 14:49:16 olympia kernel: [317637.223028] cgroup: new mount options > do not match the existing superblock, will be ignored > Aug 17 14:49:16 olympia kernel: [317637.276026] cgroup: new mount options > do not match the existing superblock, will be ignored > Aug 17 14:49:16 olympia kernel: [317637.407065] cgroup: new mount options > do not match the existing superblock, will be ignored > Aug 17 14:49:16 olympia kernel: [317637.466059] cgroup: new mount options > do not match the existing superblock, will be ignored > Aug 17 14:49:16 olympia kernel: [317637.559652] cgroup: new mount options > do not match the existing superblock, will be ignored > Aug 17 14:49:16 olympia kernel: [317637.671028] cgroup: new mount options > do not match the existing superblock, will be ignored > Aug 17 14:49:16 olympia kernel: [317637.760296] cgroup: new mount options > do not match the existing superblock, will be ignored > Aug 17 14:49:17 olympia kernel: [317637.840036] cgroup: new mount options > do not match the existing superblock, will be ignored > Aug 17 14:49:17 olympia kernel: [317637.944061] cgroup: new mount options > do not match the existing superblock, will be ignored > Aug 17 14:49:17 olympia kernel: [317638.017056] cgroup: new mount options > do not match the existing superblock, will be ignored > Aug 17 14:49:17 olympia kernel: [317638.074090] cgroup: new mount options > do not match the existing superblock, will be ignored > Aug 17 14:49:20 olympia kernel: [317640.800234] cgroup: new mount options > do not match the existing superblock, will be ignored > Aug 17 14:49:20 olympia kernel: [317641.335276] cgroup: new mount options > do not match the existing superblock, will be ignored > Aug 17 14:49:25 olympia kernel: [317646.079503] cgroup: new mount options > do not match the existing superblock, will be ignored > Aug 17 14:49:25 olympia kernel: [317646.273487] cgroup: new mount options > do not match the existing superblock, will be ignored > Aug 17 14:49:25 olympia kernel: [317646.444487] cgroup: new mount options > do not match the existing superblock, will be ignored > Aug 17 14:49:25 olympia kernel: [317646.694511] cgroup: new mount options > do not match the existing superblock, will be ignored > Aug 17 14:49:26 olympia kernel: [317647.107553] cgroup: new mount options > do not match the existing superblock, will be ignored > Aug 17 14:49:27 olympia kernel: [317648.322648] cgroup: new mount options > do not match the existing superblock, will be ignored > Aug 17 14:49:27 olympia kernel: [317648.549680] cgroup: new mount options > do not match the existing superblock, will be ignored > Aug 17 14:49:27 olympia kernel: [317648.606610] cgroup: new mount options > do not match the existing superblock, will be ignored > Aug 17 14:49:28 olympia kernel: [317649.377678] cgroup: new mount options > do not match the existing superblock, will be ignored > Aug 17 14:49:28 olympia kernel: [317649.445660] cgroup: new mount options > do not match the existing superblock, will be ignored > Aug 17 14:49:28 olympia kernel: [317649.509650] cgroup: new mount options > do not match the existing superblock, will be ignored > Aug 17 14:49:29 olympia kernel: [317649.921865] Netfilter messages via > NETLINK v0.30. > Aug 17 14:49:29 olympia kernel: [317649.946436] ctnetlink v0.93: > registering with nfnetlink. > Aug 17 14:49:29 olympia kernel: [317650.033557] br0: port 3(vethVHQ0XE) > entered disabled state > Aug 17 14:49:29 olympia kernel: [317650.039297] device vethVHQ0XE left > promiscuous mode > Aug 17 14:49:29 olympia kernel: [317650.039302] br0: port 3(vethVHQ0XE) > entered disabled state > Aug 17 14:49:30 olympia kernel: [317651.765330] audit_printk_skb: 69 > callbacks suppressed > Aug 17 14:49:30 olympia kernel: [317651.765333] audit: type=1400 > audit(1471470570.966:77): apparmor="STATUS" operation="profile_remove" > info="profile does not exist" error=-2 profile="unconfined" > name="lxd-kangal_</var/lib/lxd>" pid=25080 comm="apparmor_parser" > > > And this, on the receiving side: > Aug 17 14:49:56 ronnie kernel: [107460.037655] audit: type=1400 > audit(1471470596.676:89): apparmor="STATUS" operation="profile_load" > profile="unconfined" name="lxd-kangal_</var/lib/lxd>" pid=23239 > comm="apparmor_parser" > Aug 17 14:49:56 ronnie NetworkManager[2696]: <warn> [1471470596.8817] > device (vethIA06PF): failed to find device 11 'vethIA06PF' with udev > Aug 17 14:49:56 ronnie NetworkManager[2696]: <info> [1471470596.8828] > manager: (vethIA06PF): new Veth device > (/org/freedesktop/NetworkManager/Devices/10) > Aug 17 14:49:56 ronnie NetworkManager[2696]: <info> [1471470596.8991] > devices added (path: /sys/devices/virtual/net/vethIA06PF, iface: vethIA06PF) > Aug 17 14:49:56 ronnie NetworkManager[2696]: <info> [1471470596.8991] > device added (path: /sys/devices/virtual/net/vethIA06PF, iface: > vethIA06PF): no ifupdown configuration found. > Aug 17 14:49:57 ronnie NetworkManager[2696]: <info> [1471470597.0303] > device (vethIA06PF): driver 'veth' does not support carrier detection. > Aug 17 14:49:57 ronnie NetworkManager[2696]: <info> [1471470597.0351] > devices removed (path: /sys/devices/virtual/net/vethIA06PF, iface: > vethIA06PF) > > > > Jake > > > On Wed, Aug 17, 2016 at 2:50 PM, jjs - mainphrame <[email protected]> > wrote: > > > Thanks Stephane, that goes to the root of the problem. I must have been > > thinking in the back of my mind that I'd already set that, but come to > > think of it, that was back when this box was running 14.04. before a clean > > install of 16.04 > > > > Jake > > > > On Wed, Aug 17, 2016 at 1:45 PM, Stéphane Graber <[email protected]> > > wrote: > > > >> So one way around the problem would be with: > >> > >> lxc config set core.https_address 192.168.111.193:8443 > >> > >> on your olympia host. This will force LXD to use that IP during > >> container transfer and should fix your problem. > >> > >> On Wed, Aug 17, 2016 at 01:28:52PM -0700, jjs - mainphrame wrote: > >> > Hi Stephane - > >> > > >> > lxd1, lxd2, and kangal are all on the same lan, and connectivity is > >> good: > >> > > >> > root@olympia:~# ssh kangal w > >> > root@kangal's password: > >> > 13:26:53 up 3 days, 14:50, 0 users, load average: 0.29, 0.37, 0.37 > >> > USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT > >> > > >> > root@ronnie:~# ssh kangal w > >> > root@kangal's password: > >> > 13:27:09 up 3 days, 14:51, 0 users, load average: 0.29, 0.36, 0.37 > >> > USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT > >> > > >> > Regards, > >> > > >> > Jake > >> > > >> > > >> > On Wed, Aug 17, 2016 at 1:24 PM, Stéphane Graber <[email protected]> > >> > wrote: > >> > > >> > > On Wed, Aug 17, 2016 at 01:14:43PM -0700, jjs - mainphrame wrote: > >> > > > Greetings, > >> > > > > >> > > > I'm running lxd version 2.0.3-0ubuntu1~ubuntu16.04.2 > >> > > > > >> > > > I'm trying to get lxd to correctly execute a move of a container > >> from one > >> > > > lxd host to another. I have two ubuntu 16.04 hosts, ronnie > >> (designated as > >> > > > lxd1) and olympia (designated as lxd2): > >> > > > > >> > > > root@olympia:~# lxc remote list > >> > > > +-----------------+----------------------------------------- > >> > > -+---------------+--------+--------+ > >> > > > | NAME | URL | > >> PROTOCOL > >> > > > | PUBLIC | STATIC | > >> > > > +-----------------+----------------------------------------- > >> > > -+---------------+--------+--------+ > >> > > > | images | https://images.linuxcontainers.org | lxd > >> > > > | YES | NO | > >> > > > +-----------------+----------------------------------------- > >> > > -+---------------+--------+--------+ > >> > > > | local (default) | unix:// | lxd > >> > > > | NO | YES | > >> > > > +-----------------+----------------------------------------- > >> > > -+---------------+--------+--------+ > >> > > > | lxd1 | https://192.168.111.20:8443 | lxd > >> > > > | NO | NO | > >> > > > +-----------------+----------------------------------------- > >> > > -+---------------+--------+--------+ > >> > > > | ubuntu | https://cloud-images.ubuntu.com/releases | > >> > > > simplestreams | YES | YES | > >> > > > +-----------------+----------------------------------------- > >> > > -+---------------+--------+--------+ > >> > > > | ubuntu-daily | https://cloud-images.ubuntu.com/daily | > >> > > > simplestreams | YES | YES | > >> > > > +-----------------+----------------------------------------- > >> > > -+---------------+--------+--------+ > >> > > > > >> > > > root@ronnie:~# lxc remote list > >> > > > +-----------------+----------------------------------------- > >> > > -+---------------+--------+--------+ > >> > > > | NAME | URL | > >> PROTOCOL > >> > > > | PUBLIC | STATIC | > >> > > > +-----------------+----------------------------------------- > >> > > -+---------------+--------+--------+ > >> > > > | images | https://images.linuxcontainers.org | lxd > >> > > > | YES | NO | > >> > > > +-----------------+----------------------------------------- > >> > > -+---------------+--------+--------+ > >> > > > | local (default) | unix:// | lxd > >> > > > | NO | YES | > >> > > > +-----------------+----------------------------------------- > >> > > -+---------------+--------+--------+ > >> > > > | lxd2 | https://192.168.111.193:8443 | lxd > >> > > > | NO | NO | > >> > > > +-----------------+----------------------------------------- > >> > > -+---------------+--------+--------+ > >> > > > | ubuntu | https://cloud-images.ubuntu.com/releases | > >> > > > simplestreams | YES | YES | > >> > > > +-----------------+----------------------------------------- > >> > > -+---------------+--------+--------+ > >> > > > | ubuntu-daily | https://cloud-images.ubuntu.com/daily | > >> > > > simplestreams | YES | YES | > >> > > > +-----------------+----------------------------------------- > >> > > -+---------------+--------+--------+ > >> > > > > >> > > > You can see that the remotes are configured with their local lan > >> > > addresses. > >> > > > So far so good? > >> > > > > >> > > > > >> > > > > >> > > > Here are the 2 containers currently on lxd2: > >> > > > root@olympia:~# lxc list > >> > > > +--------+---------+-----------------------+------+--------- > >> > > ---+-----------+ > >> > > > | NAME | STATE | IPV4 | IPV6 | TYPE | > >> > > SNAPSHOTS | > >> > > > +--------+---------+-----------------------+------+--------- > >> > > ---+-----------+ > >> > > > | akita | RUNNING | 192.168.111.22 (eth0) | | PERSISTENT | 0 > >> > > | > >> > > > +--------+---------+-----------------------+------+--------- > >> > > ---+-----------+ > >> > > > | kangal | RUNNING | 192.168.111.44 (eth0) | | PERSISTENT | 0 > >> > > | > >> > > > +--------+---------+-----------------------+------+--------- > >> > > ---+-----------+ > >> > > > > >> > > > > >> > > > > >> > > > Now, I try to move a container from lxd2 to lxd1: > >> > > > root@olympia:~# lxc move kangal lxd1: > >> > > > error: Error transferring container data: Unable to connect to: > >> > > > 192.168.1.8:8443 > >> > > > >> > > Is there any way for lxd1 to connect to kangal? > >> > > > >> > > The way LXD currently deals with cross-host communication is that the > >> > > client has the source host issue a token which the client sends to the > >> > > target along with instructions on how to connect to the source. > >> > > > >> > > The target then directly connects to the source to fetch the data > >> > > (in this case, the container). > >> > > > >> > > This means that there must be a way for the target to connect to the > >> > > source on the LXD port without being blocked by firewalls or going > >> > > through NAT. > >> > > > >> > > > >> > > Some more details can be found at > >> > > https://www.stgraber.org/2016/04/12/lxd-2-0-remote-hosts- > >> > > and-container-migration-612/ > >> > > in the "network requirements" and "how this all works" sections. > >> > > > >> > > > >> > > The client is currently supposed to iterate through all the IPs that > >> the > >> > > source server advertises (see addresses field in "lxc info kangal"), > >> the > >> > > one that's in the error message is usually the last one of those. > >> > > > >> > > > >> > > If core.https_address is set on the source host, then only that > >> address > >> > > will be attempted since it's the only one LXD will be listening on. > >> > > > >> > > > >> > > As mentioned in the blog post, we do have a plan to improve the > >> > > situation by having the client relay the data in cases where the two > >> > > servers can't talk, but we haven't made much progress on implementing > >> > > that so far. > >> > > > >> > > > > >> > > > Why is it trying to connect to 192.168.1.8? That is a local wireless > >> > > > address on lxd2, but it was never mentioned in any lxd > >> configuration: > >> > > > > >> > > > root@olympia:~# ifconfig wlp3s0 > >> > > > wlp3s0 Link encap:Ethernet HWaddr 80:56:f2:05:ce:6c > >> > > > inet addr:192.168.1.8 Bcast:192.168.1.255 > >> Mask:255.255.255.0 > >> > > > inet6 addr: fe80::8256:f2ff:fe05:ce6c/64 Scope:Link > >> > > > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > >> > > > RX packets:401712 errors:0 dropped:0 overruns:0 frame:0 > >> > > > TX packets:219214 errors:0 dropped:0 overruns:0 carrier:0 > >> > > > collisions:0 txqueuelen:1000 > >> > > > RX bytes:95097222 (95.0 MB) TX bytes:34201360 (34.2 MB) > >> > > > > >> > > > > >> > > > So my question is, how do we get lxd to ignore the local wireless > >> IP, and > >> > > > execute the lxc move command using the configured IPs? > >> > > > > >> > > > Regards, > >> > > > > >> > > > Jake > >> > > > >> > > > _______________________________________________ > >> > > > lxc-users mailing list > >> > > > [email protected] > >> > > > http://lists.linuxcontainers.org/listinfo/lxc-users > >> > > > >> > > > >> > > -- > >> > > Stéphane Graber > >> > > Ubuntu developer > >> > > http://www.ubuntu.com > >> > > > >> > > _______________________________________________ > >> > > lxc-users mailing list > >> > > [email protected] > >> > > http://lists.linuxcontainers.org/listinfo/lxc-users > >> > > > >> > >> > _______________________________________________ > >> > lxc-users mailing list > >> > [email protected] > >> > http://lists.linuxcontainers.org/listinfo/lxc-users > >> > >> > >> -- > >> Stéphane Graber > >> Ubuntu developer > >> http://www.ubuntu.com > >> > >> _______________________________________________ > >> lxc-users mailing list > >> [email protected] > >> http://lists.linuxcontainers.org/listinfo/lxc-users > >> > > > > > _______________________________________________ > lxc-users mailing list > [email protected] > http://lists.linuxcontainers.org/listinfo/lxc-users -- Stéphane Graber Ubuntu developer http://www.ubuntu.com
signature.asc
Description: PGP signature
_______________________________________________ lxc-users mailing list [email protected] http://lists.linuxcontainers.org/listinfo/lxc-users
