[ovirt-users] hosted engine migration
Hi all, I have an ovirt4.3.10.4 environment of 2 hosts. Normal vms in this environment could be migrated, but the hosted engine vm could not be migrated. Anyone can help? Thanks a lot! hosts status: normal vm migration: hosted engine vm migration:___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/ZXHE2AJX42HNHOMYHTDCUUIU3VQTQTLF/
[ovirt-users] Hosted engine Migration
Am I reading these right in that manual migration is not possible? ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/7NP3HP3Q6DCDDOEZZK7LS4P2C6TUMZEB/
Re: [ovirt-users] Hosted Engine migration problems
It looks like the engine does not recognize engine1 and engine3 hosts as good hosts for the HE VM migration, can you please check the HE score of this hosts, before you run the migration? On Sat, Feb 11, 2017 at 8:18 AM, Jim Kusznirwrote: > Hi again: > > I thought I had fixed the hosted engine migration that was preventing me > from updating the host the engine was running on. Today it let me migrate > it from ovirt1 to ovirt2, and perform needed updates on ovirt1. When I > tried to migrate it back to ovirt1 after the updates, I got errors that it > failed migration. I tried an auto-migrate, and it claimed that the other > two nodes (including the node it was running on) do not meet minimum > requrements, specifically that they are not HA nodesBut I did > explicitly set them up as HA nodes. > > Here's the engine.log output from the command: > > 2017-02-11 06:12:03,078 INFO > [org.ovirt.engine.core.bll.scheduling.SchedulingManager] > (default task-41) [252e1f97] Candidate host 'engine1' > ('1e182fb9-8057-42ed-abd6-bc5bc343ccc6') was filtered out by > 'VAR__FILTERTYPE__INTERNAL' filter 'HA' (correlation id: null) > 2017-02-11 06:12:03,078 INFO > [org.ovirt.engine.core.bll.scheduling.SchedulingManager] > (default task-41) [252e1f97] Candidate host 'engine3' > ('bac8ace2-cf7e-48ea-9113-b82343cd87f7') was filtered out by > 'VAR__FILTERTYPE__INTERNAL' filter 'HA' (correlation id: null) > 2017-02-11 06:12:03,081 INFO > [org.ovirt.engine.core.bll.scheduling.SchedulingManager] > (default task-41) [252e1f97] Candidate host 'engine2' > ('76c075fc-1dfb-479d-98ef-57575ec11787') was filtered out by > 'VAR__FILTERTYPE__INTERNAL' filter 'Migration' (correlation id: null) > 2017-02-11 06:12:03,081 WARN [org.ovirt.engine.core.bll.MigrateVmCommand] > (default task-41) [252e1f97] Validation of action 'MigrateVm' failed for > user admin@internal-authz. Reasons: VAR__ACTION__MIGRATE,VAR__ > TYPE__VM,SCHEDULING_ALL_HOSTS_FILTERED_OUT,VAR__FILTERTYPE__INTERNAL,$hostName > engine1,$filterName HA,VAR__DETAIL__NOT_HE_HOST,SCHEDULING_HOST_FILTERED_ > REASON_WITH_DETAIL,VAR__FILTERTYPE__INTERNAL,$hostName > engine3,$filterName HA,VAR__DETAIL__NOT_HE_HOST,SCHEDULING_HOST_FILTERED_ > REASON_WITH_DETAIL,VAR__FILTERTYPE__INTERNAL,$hostName > engine2,$filterName Migration,VAR__DETAIL__SAME_ > HOST,SCHEDULING_HOST_FILTERED_REASON_WITH_DETAIL > > I'm a bit confused by thisI followed the ovirt+gluster howto > referenced from the contributed documentation page. > > --Jim > > ___ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > > ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
[ovirt-users] Hosted Engine migration problems
Hi again: I thought I had fixed the hosted engine migration that was preventing me from updating the host the engine was running on. Today it let me migrate it from ovirt1 to ovirt2, and perform needed updates on ovirt1. When I tried to migrate it back to ovirt1 after the updates, I got errors that it failed migration. I tried an auto-migrate, and it claimed that the other two nodes (including the node it was running on) do not meet minimum requrements, specifically that they are not HA nodesBut I did explicitly set them up as HA nodes. Here's the engine.log output from the command: 2017-02-11 06:12:03,078 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (default task-41) [252e1f97] Candidate host 'engine1' ('1e182fb9-8057-42ed-abd6-bc5bc343ccc6') was filtered out by 'VAR__FILTERTYPE__INTERNAL' filter 'HA' (correlation id: null) 2017-02-11 06:12:03,078 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (default task-41) [252e1f97] Candidate host 'engine3' ('bac8ace2-cf7e-48ea-9113-b82343cd87f7') was filtered out by 'VAR__FILTERTYPE__INTERNAL' filter 'HA' (correlation id: null) 2017-02-11 06:12:03,081 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (default task-41) [252e1f97] Candidate host 'engine2' ('76c075fc-1dfb-479d-98ef-57575ec11787') was filtered out by 'VAR__FILTERTYPE__INTERNAL' filter 'Migration' (correlation id: null) 2017-02-11 06:12:03,081 WARN [org.ovirt.engine.core.bll.MigrateVmCommand] (default task-41) [252e1f97] Validation of action 'MigrateVm' failed for user admin@internal-authz. Reasons: VAR__ACTION__MIGRATE,VAR__TYPE__VM,SCHEDULING_ALL_HOSTS_FILTERED_OUT,VAR__FILTERTYPE__INTERNAL,$hostName engine1,$filterName HA,VAR__DETAIL__NOT_HE_HOST,SCHEDULING_HOST_FILTERED_REASON_WITH_DETAIL,VAR__FILTERTYPE__INTERNAL,$hostName engine3,$filterName HA,VAR__DETAIL__NOT_HE_HOST,SCHEDULING_HOST_FILTERED_REASON_WITH_DETAIL,VAR__FILTERTYPE__INTERNAL,$hostName engine2,$filterName Migration,VAR__DETAIL__SAME_HOST,SCHEDULING_HOST_FILTERED_REASON_WITH_DETAIL I'm a bit confused by thisI followed the ovirt+gluster howto referenced from the contributed documentation page. --Jim ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] hosted-engine migration problems on 3.6, 3.6.1, 3.6.2
I cannot get the HE to run on node1. I did 'hosted-engine -vm-shutdown' on node2. After it was down I did 'hosted-engine --vm-start' on node1 but did not even get a qemu process. I did 'hosted-engine --vm-poweroff' on node1 and 'hosted-engine --vm-start' on node2 and got it up and running again. How do I get the HE storage domain imported to the engine to fix all of this? Is it a database thing? Thanks, Peter On 01/28/2016 11:02 AM, Peter wrote: > Martin, > > The current state is HE on node2 and all other VMs on node1. Node2 is > in local maintenance as of last night and HE should have migrated but > can't. The requested logs for this situation are at > ftp://aftp.fsl.noaa.gov/divisions/its/peter/ovirt-logs-201601028a/. > > I will try shutting down HE on node2 and see if it will run on node1. > Last night it would not. > > Thanks. > Peter > > On 01/28/2016 03:14 AM, Martin Sivak wrote: >> Hi, >> >> we really need more logs. Preferably the full agent, vdsm and engine >> log around the time the migration is attempted. >> >> The warn/error messages are all related to the fact that hosted engine >> runs in the "3.5 mode" with the storage domain not being imported to >> the engine yet. >> >> -- >> Martin Sivak >> oVirt / SLA >> >> >> On Thu, Jan 28, 2016 at 9:28 AM, Nir Sofferwrote: >>> Adding Martin >>> >>> On Thu, Jan 28, 2016 at 5:54 AM, Peter wrote: I am running oVirt 3.6.2 (original install was 3.6) hosted-engine on a couple of Centos 7.2 servers with SAS attached storage using the new FC support to connect to the LUNs. Neither the hosted-engine storage nor the hosted-engine VM show up in the GUI. I know there have been a lot of bugs that have been fixed for this but I still can't import anything. This may be the basis of my problem. The problem is that I can't reliably get the hosted-engine to migrate to the other head. If all VMs and HE are running on node1 and I try to put it in maintenance the VMs will migrate but the HE will not and node1 will show 1 VM remaining on the node (even though the HE VM doesn't show up in the VM list) and will not go into maintenance. I try setting 'hosted-engine --set-maintenance --mode=local' but that doesn't always help. The output of 'hosted-engine --vm-status' eventually changes to show the local node in maintenance with score 0 but often it doesn't move. In /var/log/ovirt-hosted-engine-ha/agent.log I see MainThread::INFO::2016-01-28 03:15:42,951::config::205::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::WARNING::2016-01-28 03:15:43,296::ovf_store::105::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Unable to find OVF_STORE MainThread::ERROR::2016-01-28 03:15:43,297::config::234::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) Unable to get vm.conf from OVF_STORE, falling back to initial vm.conf MainThread::WARNING::2016-01-28 03:15:43,316::hosted_engine::767::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_domain_monitor_if_possible) The VM is running locally or we have no data, keeping the domain monitor. I've restarted servers and/or processes but I'm not getting anywhere. I appreciate any help. ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] hosted-engine migration problems on 3.6, 3.6.1, 3.6.2
Adding Martin On Thu, Jan 28, 2016 at 5:54 AM, Peterwrote: > > I am running oVirt 3.6.2 (original install was 3.6) hosted-engine on a couple > of Centos 7.2 servers with SAS attached storage using the new FC support to > connect to the LUNs. > > Neither the hosted-engine storage nor the hosted-engine VM show up in the > GUI. I know there have been a lot of bugs that have been fixed for this but > I still can't import anything. This may be the basis of my problem. > > The problem is that I can't reliably get the hosted-engine to migrate to the > other head. If all VMs and HE are running on node1 and I try to put it in > maintenance the VMs will migrate but the HE will not and node1 will show 1 VM > remaining on the node (even though the HE VM doesn't show up in the VM list) > and will not go into maintenance. > > I try setting 'hosted-engine --set-maintenance --mode=local' but that doesn't > always help. The output of 'hosted-engine --vm-status' eventually changes to > show the local node in maintenance with score 0 but often it doesn't move. > In /var/log/ovirt-hosted-engine-ha/agent.log I see > > MainThread::INFO::2016-01-28 > 03:15:42,951::config::205::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) > Trying to get a fresher copy of vm configuration from the OVF_STORE > MainThread::WARNING::2016-01-28 > 03:15:43,296::ovf_store::105::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) > Unable to find OVF_STORE > MainThread::ERROR::2016-01-28 > 03:15:43,297::config::234::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) > Unable to get vm.conf from OVF_STORE, falling back to initial vm.conf > MainThread::WARNING::2016-01-28 > 03:15:43,316::hosted_engine::767::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_domain_monitor_if_possible) > The VM is running locally or we have no data, keeping the domain monitor. > > I've restarted servers and/or processes but I'm not getting anywhere. I > appreciate any help. > > ___ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] hosted-engine migration problems on 3.6, 3.6.1, 3.6.2
Hi, we really need more logs. Preferably the full agent, vdsm and engine log around the time the migration is attempted. The warn/error messages are all related to the fact that hosted engine runs in the "3.5 mode" with the storage domain not being imported to the engine yet. -- Martin Sivak oVirt / SLA On Thu, Jan 28, 2016 at 9:28 AM, Nir Sofferwrote: > Adding Martin > > On Thu, Jan 28, 2016 at 5:54 AM, Peter wrote: >> >> I am running oVirt 3.6.2 (original install was 3.6) hosted-engine on a >> couple of Centos 7.2 servers with SAS attached storage using the new FC >> support to connect to the LUNs. >> >> Neither the hosted-engine storage nor the hosted-engine VM show up in the >> GUI. I know there have been a lot of bugs that have been fixed for this but >> I still can't import anything. This may be the basis of my problem. >> >> The problem is that I can't reliably get the hosted-engine to migrate to the >> other head. If all VMs and HE are running on node1 and I try to put it in >> maintenance the VMs will migrate but the HE will not and node1 will show 1 >> VM remaining on the node (even though the HE VM doesn't show up in the VM >> list) and will not go into maintenance. >> >> I try setting 'hosted-engine --set-maintenance --mode=local' but that >> doesn't always help. The output of 'hosted-engine --vm-status' eventually >> changes to show the local node in maintenance with score 0 but often it >> doesn't move. In /var/log/ovirt-hosted-engine-ha/agent.log I see >> >> MainThread::INFO::2016-01-28 >> 03:15:42,951::config::205::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) >> Trying to get a fresher copy of vm configuration from the OVF_STORE >> MainThread::WARNING::2016-01-28 >> 03:15:43,296::ovf_store::105::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) >> Unable to find OVF_STORE >> MainThread::ERROR::2016-01-28 >> 03:15:43,297::config::234::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) >> Unable to get vm.conf from OVF_STORE, falling back to initial vm.conf >> MainThread::WARNING::2016-01-28 >> 03:15:43,316::hosted_engine::767::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_domain_monitor_if_possible) >> The VM is running locally or we have no data, keeping the domain monitor. >> >> I've restarted servers and/or processes but I'm not getting anywhere. I >> appreciate any help. >> >> ___ >> Users mailing list >> Users@ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] hosted-engine migration problems on 3.6, 3.6.1, 3.6.2
Martin, The current state is HE on node2 and all other VMs on node1. Node2 is in local maintenance as of last night and HE should have migrated but can't. The requested logs for this situation are at ftp://aftp.fsl.noaa.gov/divisions/its/peter/ovirt-logs-201601028a/. I will try shutting down HE on node2 and see if it will run on node1. Last night it would not. Thanks. Peter On 01/28/2016 03:14 AM, Martin Sivak wrote: > Hi, > > we really need more logs. Preferably the full agent, vdsm and engine > log around the time the migration is attempted. > > The warn/error messages are all related to the fact that hosted engine > runs in the "3.5 mode" with the storage domain not being imported to > the engine yet. > > -- > Martin Sivak > oVirt / SLA > > > On Thu, Jan 28, 2016 at 9:28 AM, Nir Sofferwrote: >> Adding Martin >> >> On Thu, Jan 28, 2016 at 5:54 AM, Peter wrote: >>> I am running oVirt 3.6.2 (original install was 3.6) hosted-engine on a >>> couple of Centos 7.2 servers with SAS attached storage using the new FC >>> support to connect to the LUNs. >>> >>> Neither the hosted-engine storage nor the hosted-engine VM show up in the >>> GUI. I know there have been a lot of bugs that have been fixed for this >>> but I still can't import anything. This may be the basis of my problem. >>> >>> The problem is that I can't reliably get the hosted-engine to migrate to >>> the other head. If all VMs and HE are running on node1 and I try to put it >>> in maintenance the VMs will migrate but the HE will not and node1 will show >>> 1 VM remaining on the node (even though the HE VM doesn't show up in the VM >>> list) and will not go into maintenance. >>> >>> I try setting 'hosted-engine --set-maintenance --mode=local' but that >>> doesn't always help. The output of 'hosted-engine --vm-status' eventually >>> changes to show the local node in maintenance with score 0 but often it >>> doesn't move. In /var/log/ovirt-hosted-engine-ha/agent.log I see >>> >>> MainThread::INFO::2016-01-28 >>> 03:15:42,951::config::205::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) >>> Trying to get a fresher copy of vm configuration from the OVF_STORE >>> MainThread::WARNING::2016-01-28 >>> 03:15:43,296::ovf_store::105::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) >>> Unable to find OVF_STORE >>> MainThread::ERROR::2016-01-28 >>> 03:15:43,297::config::234::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) >>> Unable to get vm.conf from OVF_STORE, falling back to initial vm.conf >>> MainThread::WARNING::2016-01-28 >>> 03:15:43,316::hosted_engine::767::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_domain_monitor_if_possible) >>> The VM is running locally or we have no data, keeping the domain monitor. >>> >>> I've restarted servers and/or processes but I'm not getting anywhere. I >>> appreciate any help. >>> >>> ___ >>> Users mailing list >>> Users@ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
[ovirt-users] hosted-engine migration problems on 3.6, 3.6.1, 3.6.2
I am running oVirt 3.6.2 (original install was 3.6) hosted-engine on a couple of Centos 7.2 servers with SAS attached storage using the new FC support to connect to the LUNs. Neither the hosted-engine storage nor the hosted-engine VM show up in the GUI. I know there have been a lot of bugs that have been fixed for this but I still can't import anything. This may be the basis of my problem. The problem is that I can't reliably get the hosted-engine to migrate to the other head. If all VMs and HE are running on node1 and I try to put it in maintenance the VMs will migrate but the HE will not and node1 will show 1 VM remaining on the node (even though the HE VM doesn't show up in the VM list) and will not go into maintenance. I try setting 'hosted-engine --set-maintenance --mode=local' but that doesn't always help. The output of 'hosted-engine --vm-status' eventually changes to show the local node in maintenance with score 0 but often it doesn't move. In /var/log/ovirt-hosted-engine-ha/agent.log I see MainThread::INFO::2016-01-28 03:15:42,951::config::205::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::WARNING::2016-01-28 03:15:43,296::ovf_store::105::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Unable to find OVF_STORE MainThread::ERROR::2016-01-28 03:15:43,297::config::234::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) Unable to get vm.conf from OVF_STORE, falling back to initial vm.conf MainThread::WARNING::2016-01-28 03:15:43,316::hosted_engine::767::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_domain_monitor_if_possible) The VM is running locally or we have no data, keeping the domain monitor. I've restarted servers and/or processes but I'm not getting anywhere. I appreciate any help. ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] hosted-engine migration problems on 3.6, 3.6.1, 3.6.2
Can you please attach logs? Yaniv Dary Technical Product Manager Red Hat Israel Ltd. 34 Jerusalem Road Building A, 4th floor Ra'anana, Israel 4350109 Tel : +972 (9) 7692306 8272306 Email: yd...@redhat.com IRC : ydary On Jan 28, 2016 06:09, "Peter"wrote: > > I am running oVirt 3.6.2 (original install was 3.6) hosted-engine on a > couple of Centos 7.2 servers with SAS attached storage using the new FC > support to connect to the LUNs. > > Neither the hosted-engine storage nor the hosted-engine VM show up in the > GUI. I know there have been a lot of bugs that have been fixed for this > but I still can't import anything. This may be the basis of my problem. > > The problem is that I can't reliably get the hosted-engine to migrate to > the other head. If all VMs and HE are running on node1 and I try to put it > in maintenance the VMs will migrate but the HE will not and node1 will show > 1 VM remaining on the node (even though the HE VM doesn't show up in the VM > list) and will not go into maintenance. > > I try setting 'hosted-engine --set-maintenance --mode=local' but that > doesn't always help. The output of 'hosted-engine --vm-status' eventually > changes to show the local node in maintenance with score 0 but often it > doesn't move. In /var/log/ovirt-hosted-engine-ha/agent.log I see > > MainThread::INFO::2016-01-28 > 03:15:42,951::config::205::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) > Trying to get a fresher copy of vm configuration from the OVF_STORE > MainThread::WARNING::2016-01-28 > 03:15:43,296::ovf_store::105::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) > Unable to find OVF_STORE > MainThread::ERROR::2016-01-28 > 03:15:43,297::config::234::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) > Unable to get vm.conf from OVF_STORE, falling back to initial vm.conf > MainThread::WARNING::2016-01-28 > 03:15:43,316::hosted_engine::767::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_domain_monitor_if_possible) > The VM is running locally or we have no data, keeping the domain monitor. > > I've restarted servers and/or processes but I'm not getting anywhere. I > appreciate any help. > > ___ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Hosted Engine Migration fails
=on ßsnip à Here is the content of the .meta file in the directories of the 2 images above --snip-- DOMAIN=276e9ba7-e19a-49c5-8ad7-26711934d5e4 VOLTYPE=LEAF CTIME=1424445378 FORMAT=RAW IMAGE=c37bfa94-718c-4125-9202-bc299535eca5 DISKTYPE=2 PUUID=---- LEGALITY=LEGAL MTIME=0 POOL_UUID= DESCRIPTION={DiskAlias:al-exchange-01_Disk1,DiskDescription:System} TYPE=SPARSE SIZE=146800640 EOF --snip-- --snip-- DOMAIN=276e9ba7-e19a-49c5-8ad7-26711934d5e4 CTIME=1424785898 FORMAT=RAW DISKTYPE=2 LEGALITY=LEGAL SIZE=419430400 VOLTYPE=LEAF DESCRIPTION={DiskAlias:al-exchange-01_Disk2,DiskDescription:Data} IMAGE=75134ccc-b74e-4955-90a5-95d4ceff403b PUUID=---- MTIME=0 POOL_UUID= TYPE=SPARSE EOF --snip-- *From:*Roy Golan [mailto:rgo...@redhat.com] *Sent:* Wednesday, February 18, 2015 12:12 PM *To:* Soeren Malchow; users@ovirt.org *Subject:* Re: [ovirt-users] Hosted Engine Migration fails On 02/16/2015 04:55 AM, Soeren Malchow wrote: Dear all, we ahve a setup with several hosts running fedora 10 with the virt-preview packages installed (for snapshot live merge) and a hosted engine running centos 6.6- We are experiencing a problem with the Live Migration of the Hosted Engine, in the case of setting the host for the Engine into maintenance as well as a manual migration. I tried this on the “ovirtmgmt” network and when that failed I did some research and tried to use another network interface (separate from ovirtmgmt on layer 2), this also fails. It looks as if the migration is still going through the ovirtmgmt interface, at least judging from the network traffic, and I think the error that I found (RHBZ#919201) is actually the right one. -- vdsm.log:Thread-6745::WARNING::2015-02-16 03:22:25,743::migration::458::vm.Vm::(monitor_migration) vmId=`XXX`::Migration stalling: remaining (35MiB) lowmark (15MiB). Refer to RHBZ#919201. vdsm.log:Thread-6745::WARNING::2015-02-16 03:22:35,745::migration::458::vm.Vm::(monitor_migration) vmId=`XXX`::Migration stalling: remaining (129MiB) lowmark (15MiB). Refer to RHBZ#919201. vdsm.log:Thread-6745::WARNING::2015-02-16 03:22:45,747::migration::458::vm.Vm::(monitor_migration) vmId=`XXX`::Migration stalling: remaining (42MiB) lowmark (15MiB). Refer to RHBZ#919201. vdsm.log:Thread-6745::WARNING::2015-02-16 03:22:55,749::migration::458::vm.Vm::(monitor_migration) vmId=`XXX`::Migration stalling: remaining (88MiB) lowmark (15MiB). Refer to RHBZ#919201. -- The ovirtmgmt interface is 2 x 1Gbit (LACP connected to Dell Switches with MLAG) and by far not fully utilized. Can anyone help me where to go form here ? still need the whole log. probably the guest (engine) is doing lots of memory i/o if you got a fair amount of running VMs and Hosts. that will stalls the migration because the guest pages are getting dirty faster than qemu can copy. you have 2 options: 1. try several more times. 2. shutdown the engine vm, it should start on another host Regards Soeren Hi Soeren, the key issue seems to be this one- vdsm.log:Thread-49551::WARNING::2015-02-26 15:50:27,359::migration::445::vm.Vm::(monitor_migration) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Migration is stuck: Hasn't progressed in 150.069458961 seconds. Aborting. Basically it means that the migration process is not converging since moving the data from one host to the other is too slow. The reason for this could be- 1. Extremely slow or busy network. 2. Something inside your VM is changing memory very fast (faster than the copying rate). In order to rule out (2), you can start an empty VM with nothing inside but a minimal OS. If you're not sure you can use tiny Linux[1]. Such a minimal VM should have no problem with migrating from one machine to the other. If it has an issue, it means that you have a problem in your network that causes the migration process to take longer than it should. Give it a try and let us know how it goes. Doron [1] http://distro.ibiblio.org/tinycorelinux/ ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Hosted Engine Migration fails
=---- MTIME=0 POOL_UUID= TYPE=SPARSE EOF --snip-- From: Roy Golan [mailto:rgo...@redhat.com] Sent: Wednesday, February 18, 2015 12:12 PM To: Soeren Malchow; users@ovirt.org Subject: Re: [ovirt-users] Hosted Engine Migration fails On 02/16/2015 04:55 AM, Soeren Malchow wrote: Dear all, we ahve a setup with several hosts running fedora 10 with the virt-preview packages installed (for snapshot live merge) and a hosted engine running centos 6.6- We are experiencing a problem with the Live Migration of the Hosted Engine, in the case of setting the host for the Engine into maintenance as well as a manual migration. I tried this on the ovirtmgmt network and when that failed I did some research and tried to use another network interface (separate from ovirtmgmt on layer 2), this also fails. It looks as if the migration is still going through the ovirtmgmt interface, at least judging from the network traffic, and I think the error that I found (RHBZ#919201) is actually the right one. -- vdsm.log:Thread-6745::WARNING::2015-02-16 03:22:25,743::migration::458::vm.Vm::(monitor_migration) vmId=`XXX`::Migration stalling: remaining (35MiB) lowmark (15MiB). Refer to RHBZ#919201. vdsm.log:Thread-6745::WARNING::2015-02-16 03:22:35,745::migration::458::vm.Vm::(monitor_migration) vmId=`XXX`::Migration stalling: remaining (129MiB) lowmark (15MiB). Refer to RHBZ#919201. vdsm.log:Thread-6745::WARNING::2015-02-16 03:22:45,747::migration::458::vm.Vm::(monitor_migration) vmId=`XXX`::Migration stalling: remaining (42MiB) lowmark (15MiB). Refer to RHBZ#919201. vdsm.log:Thread-6745::WARNING::2015-02-16 03:22:55,749::migration::458::vm.Vm::(monitor_migration) vmId=`XXX`::Migration stalling: remaining (88MiB) lowmark (15MiB). Refer to RHBZ#919201. -- The ovirtmgmt interface is 2 x 1Gbit (LACP connected to Dell Switches with MLAG) and by far not fully utilized. Can anyone help me where to go form here ? still need the whole log. probably the guest (engine) is doing lots of memory i/o if you got a fair amount of running VMs and Hosts. that will stalls the migration because the guest pages are getting dirty faster than qemu can copy. you have 2 options: 1. try several more times. 2. shutdown the engine vm, it should start on another host Regards Soeren ___ Users mailing list Users@ovirt.orgmailto:Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Hosted Engine Migration fails
On 02/16/2015 04:55 AM, Soeren Malchow wrote: Dear all, we ahve a setup with several hosts running fedora 10 with the virt-preview packages installed (for snapshot live merge) and a hosted engine running centos 6.6- We are experiencing a problem with the Live Migration of the Hosted Engine, in the case of setting the host for the Engine into maintenance as well as a manual migration. I tried this on the “ovirtmgmt” network and when that failed I did some research and tried to use another network interface (separate from ovirtmgmt on layer 2), this also fails. It looks as if the migration is still going through the ovirtmgmt interface, at least judging from the network traffic, and I think the error that I found (RHBZ#919201) is actually the right one. -- vdsm.log:Thread-6745::WARNING::2015-02-16 03:22:25,743::migration::458::vm.Vm::(monitor_migration) vmId=`XXX`::Migration stalling: remaining (35MiB) lowmark (15MiB). Refer to RHBZ#919201. vdsm.log:Thread-6745::WARNING::2015-02-16 03:22:35,745::migration::458::vm.Vm::(monitor_migration) vmId=`XXX`::Migration stalling: remaining (129MiB) lowmark (15MiB). Refer to RHBZ#919201. vdsm.log:Thread-6745::WARNING::2015-02-16 03:22:45,747::migration::458::vm.Vm::(monitor_migration) vmId=`XXX`::Migration stalling: remaining (42MiB) lowmark (15MiB). Refer to RHBZ#919201. vdsm.log:Thread-6745::WARNING::2015-02-16 03:22:55,749::migration::458::vm.Vm::(monitor_migration) vmId=`XXX`::Migration stalling: remaining (88MiB) lowmark (15MiB). Refer to RHBZ#919201. -- The ovirtmgmt interface is 2 x 1Gbit (LACP connected to Dell Switches with MLAG) and by far not fully utilized. Can anyone help me where to go form here ? still need the whole log. probably the guest (engine) is doing lots of memory i/o if you got a fair amount of running VMs and Hosts. that will stalls the migration because the guest pages are getting dirty faster than qemu can copy. you have 2 options: 1. try several more times. 2. shutdown the engine vm, it should start on another host Regards Soeren ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
[ovirt-users] Hosted Engine Migration fails
Dear all, we ahve a setup with several hosts running fedora 10 with the virt-preview packages installed (for snapshot live merge) and a hosted engine running centos 6.6- We are experiencing a problem with the Live Migration of the Hosted Engine, in the case of setting the host for the Engine into maintenance as well as a manual migration. I tried this on the ovirtmgmt network and when that failed I did some research and tried to use another network interface (separate from ovirtmgmt on layer 2), this also fails. It looks as if the migration is still going through the ovirtmgmt interface, at least judging from the network traffic, and I think the error that I found (RHBZ#919201) is actually the right one. -- vdsm.log:Thread-6745::WARNING::2015-02-16 03:22:25,743::migration::458::vm.Vm::(monitor_migration) vmId=`XXX`::Migration stalling: remaining (35MiB) lowmark (15MiB). Refer to RHBZ#919201. vdsm.log:Thread-6745::WARNING::2015-02-16 03:22:35,745::migration::458::vm.Vm::(monitor_migration) vmId=`XXX`::Migration stalling: remaining (129MiB) lowmark (15MiB). Refer to RHBZ#919201. vdsm.log:Thread-6745::WARNING::2015-02-16 03:22:45,747::migration::458::vm.Vm::(monitor_migration) vmId=`XXX`::Migration stalling: remaining (42MiB) lowmark (15MiB). Refer to RHBZ#919201. vdsm.log:Thread-6745::WARNING::2015-02-16 03:22:55,749::migration::458::vm.Vm::(monitor_migration) vmId=`XXX`::Migration stalling: remaining (88MiB) lowmark (15MiB). Refer to RHBZ#919201. -- The ovirtmgmt interface is 2 x 1Gbit (LACP connected to Dell Switches with MLAG) and by far not fully utilized. Can anyone help me where to go form here ? Regards Soeren ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users