Hi Sergey,
prlsrvctl report id: 99722287
Environment details (identical software stack on the 2 hosts):
virtuozzo-release-7.0.0-39.vz7.x86_64
CentOS Linux release 7.2.1511 (Core)
Just to reiterate - as of the previous update, live migration began
working properly; it then broke with the next day's update
I've attached logs from both OVZ hosts (hachi = source, annie = target)
Preparing to submit bug report now -
Jake
On Fri, May 13, 2016 at 2:13 AM, Sergey Bronnikov <serg...@openvz.org> wrote:
> Hi
>
> Thank you for report.
>
> Could submit a new bug to the bugs.openvz.org?
> and provide more details about your environment:
> - Vz7 version (rpm -qa virtuozzo-release)
> - problem report id (prlsrvctl problem-report --send)
>
> Sergey
>
> On 19:42 Thu 12 May , jjs - mainphrame wrote:
>> Aw shucks, I've spoken too soon. After today's updates, live migration
>> is back to not working:
>>
>> [root@hachi ~]# time vzmigrate annie 888 --online
>> Connection to destination node (annie) is successfully established
>> Moving/copying CT 888 -> CT 888, [], [] ...
>> locking 888
>> Checking bindmounts
>> Check cluster ID
>> Checking keep dir for private area copy
>> Check of requires kernel modules
>> Checking technologies
>> Checking templates for CT
>> Checking IP addresses on destination node
>> Checking RATE parameters in config
>> Checking ploop format 2
>> copy CT private /vz/private/888
>> Live migration stage started
>> Phaul service failed to live migrate CT
>> Can't move/copy CT 888 -> CT 888, [], [] : Phaul service failed to
>> live migrate CT
>>
>> real 2m51.840s
>> user 0m0.385s
>> sys 0m3.457s
>> [root@hachi ~]#
>>
>> Jake
>>
>> On Wed, May 11, 2016 at 9:28 PM, jjs - mainphrame <j...@mainphrame.com>
>> wrote:
>> > Thank you Sergey -
>> >
>> > As of today's updates, I find that live container migration is working
>> > reliably and well here.
>> >
>> > Great work!
>> >
>> > Jake
>> >
>> > On Thu, Apr 28, 2016 at 8:29 AM, Sergey Bronnikov <serg...@openvz.org>
>> > wrote:
>> >> Hi everyone,
>> >>
>> >> As usual, I'd like to share with you the latest Virtuozzo 7 development
>> >> news.
>> >>
>> >> First and foremost, more features have become available in Virtuozzo 7:
>> >>
>> >> - Containers migration via vzmigrate in Vz7 [1]
>> >> - IP and MAC filtering for VMs [2]
>> >> - VM data stats supported switch to native KVM ones [3]
>> >> - AutoFS migration [4]
>> >>
>> >> All these features are available in the latests Virtuozzo 7 builds
>> >> https://download.openvz.org/virtuozzo/factory/x86_64/iso/
>> >>
>> >> Also we have published new portions of Virtuozzo source code:
>> >>
>> >> - EZ templates:
>> >> * Ubuntu 15.10 x86_64
>> >> * Fedora 23 x86_64
>> >> * Debian 7.0 x86_64
>> >> * VzLinux 6 x86_64
>> >> * VzLinux 7 x86_64
>> >> - DRS module for rmond [5]
>> >>
>> >> Events
>> >> ======
>> >>
>> >> Earlier we participated in Linux Storage, Filesystem and Memory Management
>> >> Conference. See notes from the conference [6] if you interested.
>> >>
>> >> Upcoming talks and conferences:
>> >>
>> >> - Ross Boucher will talk about running services cloning with Docker and
>> >> CRIU at
>> >> DockerCon 2016 [7]
>> >> - Sanidhya Kashyap will present an instant OS updater using CRIU at USENIX
>> >> ATC'16 [8]
>> >> - Checkpoint-Restore micro-conference will be held 2-4 Nov at Santa-Fe,
>> >> NM at
>> >> Linux Plumbers 2016. The proposal for the conferense was accepted. [9]
>> >>
>> >> Links
>> >> =====
>> >>
>> >> [1] https://lists.openvz.org/pipermail/users/2016-March/006833.html
>> >> [2] https://lists.openvz.org/pipermail/users/2016-April/006845.html
>> >> [3] https://lists.openvz.org/pipermail/users/2016-April/006850.html
>> >> [4] https://lists.openvz.org/pipermail/users/2016-April/006861.html
>> >> [5] https://github.com/OpenVZ/rmond-drs
>> >> [6] http://lwn.net/Articles/lsfmm2016/
>> >> [7] https://blog.docker.com/2016/04/black-belt-talks-dockercon-2016/
>> >> [8]
>> >> https://www.usenix.org/conference/atc16/technical-sessions/presentation/kashyap
>> >> [9]
>> >> https://www.linuxplumbersconf.org/2016/checkpoint-restore-microconference-accepted-into-2016-linux-plumbers-conference/
>> >>
>> >> Sincerely,
>> >> Sergey
>> >> _______________________________________________
>> >> Users mailing list
>> >> Users@openvz.org
>> >> https://lists.openvz.org/mailman/listinfo/users
>> _______________________________________________
>> Users mailing list
>> Users@openvz.org
>> https://lists.openvz.org/mailman/listinfo/users
>
> --
> sergeyb@
[root@hachi ~]# time vzmigrate annie 888 --online
Connection to destination node (annie) is successfully established
Moving/copying CT 888 -> CT 888, [], [] ...
locking 888
Checking bindmounts
Check cluster ID
Checking keep dir for private area copy
Check of requires kernel modules
Checking technologies
Checking templates for CT
Checking IP addresses on destination node
Checking RATE parameters in config
Checking ploop format 2
copy CT private /vz/private/888
Live migration stage started
Phaul service failed to live migrate CT
Can't move/copy CT 888 -> CT 888, [], [] : Phaul service failed to live migrate
CT
real 2m54.298s
user 0m0.438s
sys 0m3.456s
[root@hachi ~]#
20:26:06.144: 52839:
20:26:06.144: 52839:
20:26:06.144: 52839:
20:26:06.144: 52839: Starting p.haul
20:26:06.144: 52839: Use existing connections, fdrpc=9 fdmem=11 fdfs=root.hdd/root.hds:13
20:26:06.335: 52839: Setting up local
20:26:06.335: 52839: Loading config file from /etc/vz/conf/
20:26:06.337: 52839: Initialize ploop hauler
20:26:06.337: 52839: `- /vz/private/888/root.hdd/root.hds
20:26:06.338: 52839: Passing (ctl:10, data:8) pair to CRIU
20:26:06.340: 52839: Setting up remote
20:26:06.371: 52839: Start migration in live mode
20:26:06.371: 52839: Checking CPU compatibility
20:26:06.371: 52839: `- Dumping CPU info
20:26:06.372: 52839: `- Sending CPU info
20:26:06.382: 52839: `- Checking CPU info
20:26:06.387: 52839: Checking criu version
20:26:06.405: 52839: Checking for Dirty Tracking
20:26:06.405: 52839: `- Explicitly enabled
20:26:06.405: 52839: Preliminary FS migration
20:28:40.114: 52839: Fs driver transfer 1503657984 bytes (~1434Mb)
20:28:40.114: 52839: * Iteration 0
20:28:40.124: 52839: Making directory /vz/dump/dmp-kFPTka-16.05.13-20.26/img/1
20:28:40.124: 52839: Issuing pre-dump command to service
20:28:50.770: 52839: Dumped 28797 pages, 0 skipped
20:28:50.770: 52839: Fs driver transfer 0 bytes
20:28:50.770: 52839: Checking iteration progress:
20:28:50.770: 52839: > Proceed to next iteration
20:28:50.770: 52839: * Iteration 1
20:28:50.780: 52839: Making directory /vz/dump/dmp-kFPTka-16.05.13-20.26/img/2
20:28:50.780: 52839: Issuing pre-dump command to service
20:28:51.278: 52839: Dumped 134 pages, 28663 skipped
20:28:51.278: 52839: Fs driver transfer 0 bytes
20:28:51.279: 52839: Checking iteration progress:
20:28:51.279: 52839: > Proceed to next iteration
20:28:51.279: 52839: * Iteration 2
20:28:51.288: 52839: Making directory /vz/dump/dmp-kFPTka-16.05.13-20.26/img/3
20:28:51.288: 52839: Issuing pre-dump command to service
20:28:51.778: 52839: Dumped 121 pages, 28676 skipped
20:28:51.778: 52839: Fs driver transfer 0 bytes
20:28:51.779: 52839: Checking iteration progress:
20:28:51.779: 52839: > Proceed to next iteration
20:28:51.779: 52839: * Iteration 3
20:28:51.788: 52839: Making directory /vz/dump/dmp-kFPTka-16.05.13-20.26/img/4
20:28:51.788: 52839: Issuing pre-dump command to service
20:28:52.241: 52839: Dumped 105 pages, 28692 skipped
20:28:52.241: 52839: Fs driver transfer 0 bytes
20:28:52.241: 52839: Checking iteration progress:
20:28:52.242: 52839: > Proceed to next iteration
20:28:52.242: 52839: * Iteration 4
20:28:52.251: 52839: Making directory /vz/dump/dmp-kFPTka-16.05.13-20.26/img/5
20:28:52.251: 52839: Issuing pre-dump command to service
20:28:52.748: 52839: Dumped 123 pages, 28674 skipped
20:28:52.748: 52839: Fs driver transfer 0 bytes
20:28:52.749: 52839: Checking iteration progress:
20:28:52.749: 52839: > Iteration grows
20:28:52.749: 52839: Final dump and restore
20:28:52.758: 52839: Making directory /vz/dump/dmp-kFPTka-16.05.13-20.26/img/6
20:28:52.758: 52839: Issuing dump command to service
20:28:52.760: 52839: Notify (pre-dump)
20:28:52.766: 52839: Notify (network-lock)
20:28:53.438: 52839: Final FS and images sync
20:28:54.345: 52839: Sending images to target
20:28:54.348: 52839: Pack
20:28:54.386: 52839: Add htype images
20:28:54.641: 52839: Asking target host to restore
20:28:57.588: 52839: Remote exception
20:28:57.588: 52839: Restore failed (17)
Traceback (most recent call last):
File "/usr/libexec/phaul/p.haul", line 47, in <module>
worker.start_migration()
File "/usr/lib/python2.7/site-packages/phaul/iters.py", line 159, in start_migration
self.__start_live_migration()
File "/usr/lib/python2.7/site-packages/phaul/iters.py", line 229, in __start_live_migration
self.target_host.restore_from_images()
File "/usr/lib/python2.7/site-packages/phaul/xem_rpc_client.py", line 26, in __call__
raise Exception(resp[1])
Exception: Restore failed (17)
17:26:06.333: 68411:
17:26:06.333: 68411:
17:26:06.334: 68411:
17:26:06.334: 68411: Starting p.haul service
17:26:06.334: 68411: Use existing connections, fdrpc=8 fdmem=10 fdfs=root.hdd/root.hds:12
17:26:06.335: 68411: Connected
17:26:06.342: 68411: Setting up service side ('vz', '888')
17:26:06.347: 68411: Loading config file from /etc/vz/conf/
17:26:06.356: 68411: Initialize ploop receiver
17:26:06.356: 68411: `- /vz/private/888/root.hdd/root.hds
17:26:06.359: 68411: Passing (ctl:15, data:7) pair to CRIU
17:26:06.375: 68411: Started images server
17:26:06.379: 68411: Waiting for images to unpack
17:26:06.383: 68411: Checking cpuinfo
17:26:06.386: 68411: `- True
17:26:06.391: 68411: Checking criu version
17:26:06.404: 68411: `- 2.1.0.23 -> 2.1.0.23
17:26:06.404: 68411: `- True
17:28:40.116: 68411: Making directory /vz/dump/rst-ee6Sr6-16.05.13-17.26/img/1
17:28:40.116: 68411: Starting page server for iter 1
17:28:40.117: 68411: Sending criu rpc req
17:28:40.123: 68411: Page server started at 68999
17:28:50.772: 68411: Making directory /vz/dump/rst-ee6Sr6-16.05.13-17.26/img/2
17:28:50.773: 68411: Starting page server for iter 2
17:28:50.773: 68411: Sending criu rpc req
17:28:50.779: 68411: Page server started at 69001
17:28:51.280: 68411: Making directory /vz/dump/rst-ee6Sr6-16.05.13-17.26/img/3
17:28:51.281: 68411: Starting page server for iter 3
17:28:51.281: 68411: Sending criu rpc req
17:28:51.287: 68411: Page server started at 69003
17:28:51.780: 68411: Making directory /vz/dump/rst-ee6Sr6-16.05.13-17.26/img/4
17:28:51.781: 68411: Starting page server for iter 4
17:28:51.781: 68411: Sending criu rpc req
17:28:51.787: 68411: Page server started at 69005
17:28:52.243: 68411: Making directory /vz/dump/rst-ee6Sr6-16.05.13-17.26/img/5
17:28:52.244: 68411: Starting page server for iter 5
17:28:52.244: 68411: Sending criu rpc req
17:28:52.251: 68411: Page server started at 69008
17:28:52.750: 68411: Making directory /vz/dump/rst-ee6Sr6-16.05.13-17.26/img/6
17:28:52.751: 68411: Starting page server for iter 6
17:28:52.751: 68411: Sending criu rpc req
17:28:52.758: 68411: Page server started at 69010
17:28:54.348: 68411: Started images server
17:28:54.392: 68411: Waiting for images to unpack
17:28:54.642: 68411: Restoring from images
17:28:54.643: 68411: Starting vzctl restore
17:28:57.581: 68411: > Restoring the Container ...
17:28:57.582: 68411: > Mount image: /vz/private/888/root.hdd
17:28:57.582: 68411: > Opening delta /vz/private/888/root.hdd/root.hds
17:28:57.582: 68411: > Adding delta dev=/dev/ploop35223 img=/vz/private/888/root.hdd/root.hds (rw)
17:28:57.583: 68411: > Mounted /dev/ploop35223p1 at /vz/root/888 fstype=ext4 data=',balloon_ino=12'
17:28:57.583: 68411: > Container is mounted
17:28:57.583: 68411: > cat: /vz/dump/rst-ee6Sr6-16.05.13-17.26/img/6/vz_clock_bootbased.img: No such file or directory
17:28:57.584: 68411: > cat: /vz/dump/rst-ee6Sr6-16.05.13-17.26/img/6/vz_clock_monotonic.img: No such file or directory
17:28:57.584: 68411: > cat: /vz/dump/rst-ee6Sr6-16.05.13-17.26/img/6/vz_aio_max_nr.img: No such file or directory
17:28:57.585: 68411: > The restore log was saved in /vz/dump/rst-ee6Sr6-16.05.13-17.26/criu_restore.7.log
17:28:57.585: 68411: > Unmount image: /vz/private/888/root.hdd
17:28:57.585: 68411: > Unmounting file system at /vz/root/888
17:28:57.586: 68411: > Unmounting device /dev/ploop35223
17:28:57.586: 68411: > Container is unmounted
17:28:57.586: 68411: > Failed to restore the Container
17:28:57.636: 68411: Disconnected
17:28:57.638: 68411: Closing images
17:28:57.638: 68411: Keeping images
17:28:57.638: 68411: Images are kept in /vz/dump/rst-ee6Sr6-16.05.13-17.26
17:28:57.775: 68411: Stop by 15
17:28:57.775: 68411: RPC Service stops
17:28:57.776: 68411: Bye!
_______________________________________________
Users mailing list
Users@openvz.org
https://lists.openvz.org/mailman/listinfo/users
