[ovirt-users] OVN Provider setup issues
Hi all, I've been messing around with the ovirt-ovn-provider [1] and I've run into some issues during the initial setup. I have a 5-node cluster (running the hosted-engine VA), LEGACY virtual switch; this test was done on a single host. Following the instructions from the aforementioned blog post, I have downloaded the ovirt-provider-ovn and ovirt-provider-ovn-driver rpms, and built the rpm packages for: - openvswitch (2.6.90) - openvswitch-ovn-common - openvswitch-ovn-host - openvswitch-ovn-central - python-openvswitch I set up a dedicated VM for the OVN controller, installed ovs and ovn-central, started the ovn-northd and ovirt-provider-ovn services. So far so good. I then moved on to the oVirt host and installed the above packages (minus ovn-central) as well as the ovirt-provider-ovn-driver provided, started the ovn-controller service and ran # vdsm-tool ovn-config Executing the suggested checks I noticed that something didn't quite go as planned. Below is the /var/log/openvswitch/ovn-controller.log from the host machine. There are no firewalls involved (not even on the servers) and I also tried disabling SELinux but to no avail. Any ideas? Thanks, Andrea [1] http://www.ovirt.org/blog/2016/11/ovirt-provider-ovn/ 2016-11-07T14:22:09.552Z|1|vlog|INFO|opened log file /var/log/openvswitch/ovn-controller.log 2016-11-07T14:22:09.553Z|2|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting... 2016-11-07T14:22:09.553Z|3|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected 2016-11-07T14:22:09.555Z|4|reconnect|INFO|tcp:10.100.248.11:6642: connecting... 2016-11-07T14:22:09.555Z|5|reconnect|INFO|tcp:10.100.248.11:6642: connected 2016-11-07T14:22:09.556Z|6|ofctrl|INFO|unix:/var/run/openvswitch/br-int.mgmt: connecting to switch 2016-11-07T14:22:09.556Z|7|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: connecting... 2016-11-07T14:22:09.557Z|8|pinctrl|INFO|unix:/var/run/openvswitch/br-int.mgmt: connecting to switch 2016-11-07T14:22:09.557Z|9|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: connecting... 2016-11-07T14:22:09.557Z|00010|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: connected 2016-11-07T14:22:09.557Z|00011|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: connected 2016-11-07T14:22:09.558Z|00012|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x9): OFPBMC_BAD_FIELD OFPT_FLOW_MOD (OF1.3) (xid=0x9): (***truncated to 64 bytes from 240***) 04 0e 00 f0 00 00 00 09-00 00 00 00 00 00 00 00 || 0010 00 00 00 00 00 00 00 00-22 00 00 00 00 00 00 00 |"...| 0020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 || 0030 00 01 00 04 00 00 00 00-00 04 00 b8 00 00 00 00 || ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] OVN Provider setup issues
On 16/11/2016 17:23, Dan Kenigsberg wrote: On Tue, Nov 08, 2016 at 09:21:13AM +0100, Andrea Fagiani wrote: Hi all, I've been messing around with the ovirt-ovn-provider [1] and I've run into some issues during the initial setup. I have a 5-node cluster (running the hosted-engine VA), LEGACY virtual switch; this test was done on a single host. Following the instructions from the aforementioned blog post, I have downloaded the ovirt-provider-ovn and ovirt-provider-ovn-driver rpms, and built the rpm packages for: - openvswitch (2.6.90) - openvswitch-ovn-common - openvswitch-ovn-host - openvswitch-ovn-central - python-openvswitch I set up a dedicated VM for the OVN controller, installed ovs and ovn-central, started the ovn-northd and ovirt-provider-ovn services. So far so good. I then moved on to the oVirt host and installed the above packages (minus ovn-central) as well as the ovirt-provider-ovn-driver provided, started the ovn-controller service and ran # vdsm-tool ovn-config Executing the suggested checks I noticed that something didn't quite go as planned. Below is the /var/log/openvswitch/ovn-controller.log from the host machine. There are no firewalls involved (not even on the servers) and I also tried disabling SELinux but to no avail. Any ideas? Thanks, Andrea [1] http://www.ovirt.org/blog/2016/11/ovirt-provider-ovn/ 2016-11-07T14:22:09.552Z|1|vlog|INFO|opened log file /var/log/openvswitch/ovn-controller.log 2016-11-07T14:22:09.553Z|2|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting... 2016-11-07T14:22:09.553Z|3|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected 2016-11-07T14:22:09.555Z|4|reconnect|INFO|tcp:10.100.248.11:6642: connecting... 2016-11-07T14:22:09.555Z|5|reconnect|INFO|tcp:10.100.248.11:6642: connected 2016-11-07T14:22:09.556Z|6|ofctrl|INFO|unix:/var/run/openvswitch/br-int.mgmt: connecting to switch 2016-11-07T14:22:09.556Z|7|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: connecting... 2016-11-07T14:22:09.557Z|8|pinctrl|INFO|unix:/var/run/openvswitch/br-int.mgmt: connecting to switch 2016-11-07T14:22:09.557Z|9|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: connecting... 2016-11-07T14:22:09.557Z|00010|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: connected 2016-11-07T14:22:09.557Z|00011|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: connected 2016-11-07T14:22:09.558Z|00012|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x9): OFPBMC_BAD_FIELD OFPT_FLOW_MOD (OF1.3) (xid=0x9): (***truncated to 64 bytes from 240***) 04 0e 00 f0 00 00 00 09-00 00 00 00 00 00 00 00 || 0010 00 00 00 00 00 00 00 00-22 00 00 00 00 00 00 00 |"...| 0020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 || 0030 00 01 00 04 00 00 00 00-00 04 00 b8 00 00 00 00 || Andrea, could you tell (mostly Lance) which kernel version are you using? Here is the yum info output for the kernel package on the offending host: Installed Packages Name : kernel Arch: x86_64 Version : 3.10.0 Release : 327.36.3.el7 Andrea ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] OVN Provider setup issues
Hi Lance, thanks, I have currently deployed oVirt using the oVirt Node images, so indeed I would like to avoid updating; out of curiosity, is there actually a beta/pre-release version of the node avaiable? I have since reinstalled the host to perform further testing but I'll give it a shot as soon as soon as I find the time. Regards, Andrea On 17/11/2016 21:57, Lance Richardson wrote: From: "Lance Richardson" To: "Andrea Fagiani" Cc: "Dan Kenigsberg" , users@ovirt.org, mmire...@redhat.com Sent: Wednesday, November 16, 2016 12:37:57 PM Subject: Re: [ovirt-users] OVN Provider setup issues From: "Andrea Fagiani" To: "Dan Kenigsberg" Cc: users@ovirt.org, "Lance Richardson" , mmire...@redhat.com Sent: Wednesday, November 16, 2016 12:24:15 PM Subject: Re: [ovirt-users] OVN Provider setup issues On 16/11/2016 17:23, Dan Kenigsberg wrote: On Tue, Nov 08, 2016 at 09:21:13AM +0100, Andrea Fagiani wrote: Hi all, I've been messing around with the ovirt-ovn-provider [1] and I've run into some issues during the initial setup. I have a 5-node cluster (running the hosted-engine VA), LEGACY virtual switch; this test was done on a single host. Following the instructions from the aforementioned blog post, I have downloaded the ovirt-provider-ovn and ovirt-provider-ovn-driver rpms, and built the rpm packages for: - openvswitch (2.6.90) - openvswitch-ovn-common - openvswitch-ovn-host - openvswitch-ovn-central - python-openvswitch I set up a dedicated VM for the OVN controller, installed ovs and ovn-central, started the ovn-northd and ovirt-provider-ovn services. So far so good. I then moved on to the oVirt host and installed the above packages (minus ovn-central) as well as the ovirt-provider-ovn-driver provided, started the ovn-controller service and ran # vdsm-tool ovn-config Executing the suggested checks I noticed that something didn't quite go as planned. Below is the /var/log/openvswitch/ovn-controller.log from the host machine. There are no firewalls involved (not even on the servers) and I also tried disabling SELinux but to no avail. Any ideas? Thanks, Andrea [1] http://www.ovirt.org/blog/2016/11/ovirt-provider-ovn/ 2016-11-07T14:22:09.552Z|1|vlog|INFO|opened log file /var/log/openvswitch/ovn-controller.log 2016-11-07T14:22:09.553Z|2|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting... 2016-11-07T14:22:09.553Z|3|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected 2016-11-07T14:22:09.555Z|4|reconnect|INFO|tcp:10.100.248.11:6642: connecting... 2016-11-07T14:22:09.555Z|5|reconnect|INFO|tcp:10.100.248.11:6642: connected 2016-11-07T14:22:09.556Z|6|ofctrl|INFO|unix:/var/run/openvswitch/br-int.mgmt: connecting to switch 2016-11-07T14:22:09.556Z|7|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: connecting... 2016-11-07T14:22:09.557Z|8|pinctrl|INFO|unix:/var/run/openvswitch/br-int.mgmt: connecting to switch 2016-11-07T14:22:09.557Z|9|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: connecting... 2016-11-07T14:22:09.557Z|00010|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: connected 2016-11-07T14:22:09.557Z|00011|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: connected 2016-11-07T14:22:09.558Z|00012|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x9): OFPBMC_BAD_FIELD OFPT_FLOW_MOD (OF1.3) (xid=0x9): (***truncated to 64 bytes from 240***) 04 0e 00 f0 00 00 00 09-00 00 00 00 00 00 00 00 || 0010 00 00 00 00 00 00 00 00-22 00 00 00 00 00 00 00 |"...| 0020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 || 0030 00 01 00 04 00 00 00 00-00 04 00 b8 00 00 00 00 || Andrea, could you tell (mostly Lance) which kernel version are you using? Here is the yum info output for the kernel package on the offending host: Installed Packages Name : kernel Arch: x86_64 Version : 3.10.0 Release : 327.36.3.el7 Andrea OK, the 327 kernel did not support conntrack and cannot be used for OVN. RHEL 7.3 or newer is the minimum requirement for OVN. Lance Hi Andrea, If you'd like to avoid upgrading to 7.3, one workaround would be to build and install the OVS kernel module RPM. The easiest way to build the OVS kernel RPM would be, from the ovs directory where you already executed "./boot.sh" and "./configure", to execute: make rpm-fedora-kmod (Don't be misled by the name, this also works for CentOS and RHEL). The resulting RPM will be under the ./rpm/RPMS/ directory, something like: ./rpm/rpmbuild/RPMS/x86_64/openvswitch-kmod-2.6.90-1.el7.centos.x86_64.rpm Regards, Lance ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] OVN Provider setup issues
Hi Dan, I was able to setup the OVN external provider building and loading the updated OVS kernel module; I am currently running it on all 5 hosts, ovs-vsctl shows all the tunnels correctly instantiated. However, after importing the provider into the oVirt engine and setting up a vNic profile, I cannot assign it to any VM; it doesn't show up in the vNic profiles list. Any suggestions? Thanks, Andrea On 18/11/2016 12:33, Dan Kenigsberg wrote: On Fri, Nov 18, 2016 at 09:13:53AM +0100, Andrea Fagiani wrote: Hi Lance, thanks, I have currently deployed oVirt using the oVirt Node images, so indeed I would like to avoid updating; out of curiosity, is there actually a beta/pre-release version of the node avaiable? I'm afraid that such version would be available only after the release of centos7.3 and ovirt-4.1-beta. Now we're still speaking about master-branch experiments. I have since reinstalled the host to perform further testing but I'll give it a shot as soon as soon as I find the time. We'd love to hear how that works for you. Regards, Dan. ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] OVN Provider setup issues
Marcin, thanks! I keep forgetting that part of the configuration even exists. After attaching the network to the cluster I was able to configure the vNics as intended. I then had to disable iptables (as mentioned in the blog post) on the hosts running the VMs and I was able to establish connectivity between two VMs running on different hosts using the OVN-provided network! Thanks everyone involved for the help, I'll be sure to report back after further testing. Regards, Andrea On 23/11/2016 11:27, Marcin Mirecki wrote: Andrea, Please check if the network is attached to the cluster. Thanks, Marcin - Original Message - From: "Andrea Fagiani" To: users@ovirt.org Cc: "Dan Kenigsberg" , "Lance Richardson" , mmire...@redhat.com Sent: Wednesday, November 23, 2016 11:02:10 AM Subject: Re: [ovirt-users] OVN Provider setup issues Hi Dan, I was able to setup the OVN external provider building and loading the updated OVS kernel module; I am currently running it on all 5 hosts, ovs-vsctl shows all the tunnels correctly instantiated. However, after importing the provider into the oVirt engine and setting up a vNic profile, I cannot assign it to any VM; it doesn't show up in the vNic profiles list. Any suggestions? Thanks, Andrea On 18/11/2016 12:33, Dan Kenigsberg wrote: On Fri, Nov 18, 2016 at 09:13:53AM +0100, Andrea Fagiani wrote: Hi Lance, thanks, I have currently deployed oVirt using the oVirt Node images, so indeed I would like to avoid updating; out of curiosity, is there actually a beta/pre-release version of the node avaiable? I'm afraid that such version would be available only after the release of centos7.3 and ovirt-4.1-beta. Now we're still speaking about master-branch experiments. I have since reinstalled the host to perform further testing but I'll give it a shot as soon as soon as I find the time. We'd love to hear how that works for you. Regards, Dan. ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
[ovirt-users] high number of IOPS from ovirt hosts
Hi, We've got a 8-host cluster running oVirt 4.1; each host is connected via iscsi to a EQL 4100 storage (2x). Most of the VMs use barely any IOPS, however on our storage controller we've noticed a constant steam of reads, about 2000 IOPS; any idea what might cause the ovirt hosts to hammer on the storage so much? Looking at iotop, it seems that systemd-udevd is being activated every few seconds (and its IO% is usually fairly high) so I assume this might be somehow related. Thanks, Andrea ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] high number of IOPS from ovirt hosts
Hi Andrea, thanks for the insight; no I wasn't able to pinpoint the issue to a single host, the load seems to be evenly spread. We have another cluster running 4.0.6 on a compellent as well and we haven't experienced any similar issues. Thanks, Andrea On 10/02/2017 17:17, Andrea Ghelardi wrote: Hello Andrea, I saw that happening on my HE hosts connected to a COMPELLENT SC040 when I was running Ovirt 3.5 All HE hosts were hammering the SAN with 2~4k IOPS each Upgrading to 4.0.5 solved my case. Are you able to check if IOPS are generated from a particular host or rather spread among all nodes? Cheers AG -Original Message- From: users-boun...@ovirt.org [mailto:users-boun...@ovirt.org] On Behalf Of Andrea Fagiani Sent: Friday, February 10, 2017 12:47 PM To: users@ovirt.org Subject: [ovirt-users] high number of IOPS from ovirt hosts Hi, We've got a 8-host cluster running oVirt 4.1; each host is connected via iscsi to a EQL 4100 storage (2x). Most of the VMs use barely any IOPS, however on our storage controller we've noticed a constant steam of reads, about 2000 IOPS; any idea what might cause the ovirt hosts to hammer on the storage so much? Looking at iotop, it seems that systemd-udevd is being activated every few seconds (and its IO% is usually fairly high) so I assume this might be somehow related. Thanks, Andrea ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] high number of IOPS from ovirt hosts
Thanks for the heads up, I'll take a look at Mark's thread. We are presenting 2 storage LUNs to a single ovirt cluster and yes, all hosts are hosted engine hosts. Thanks, Andrea On 12/02/2017 18:27, Andrea Ghelardi wrote: Andrea, I don't want to point any fingers but Mark Greenall faced some issues with Ovirt 4.06/4.1 linked to equallogic as well. You may want to search posts in this ML with subject " Ovirt 4.0.6 guests 'Not Responding' " to see if you find any similarities. How many storages are you presenting to how many clusters? Also, are your nodes all Hosted Engine Hosts? In my old configuration, normal hosts read IOPS ranged from 50 to 500 while HE ones peaked even ~3000 Cheers AG -Original Message- From: Andrea Fagiani [mailto:andrea.fagi...@immobiliare.it] Sent: Friday, February 10, 2017 5:56 PM To: Andrea Ghelardi ; users@ovirt.org Subject: Re: [ovirt-users] high number of IOPS from ovirt hosts Hi Andrea, thanks for the insight; no I wasn't able to pinpoint the issue to a single host, the load seems to be evenly spread. We have another cluster running 4.0.6 on a compellent as well and we haven't experienced any similar issues. Thanks, Andrea On 10/02/2017 17:17, Andrea Ghelardi wrote: Hello Andrea, I saw that happening on my HE hosts connected to a COMPELLENT SC040 when I was running Ovirt 3.5 All HE hosts were hammering the SAN with 2~4k IOPS each Upgrading to 4.0.5 solved my case. Are you able to check if IOPS are generated from a particular host or rather spread among all nodes? Cheers AG -Original Message- From: users-boun...@ovirt.org [mailto:users-boun...@ovirt.org] On Behalf Of Andrea Fagiani Sent: Friday, February 10, 2017 12:47 PM To: users@ovirt.org Subject: [ovirt-users] high number of IOPS from ovirt hosts Hi, We've got a 8-host cluster running oVirt 4.1; each host is connected via iscsi to a EQL 4100 storage (2x). Most of the VMs use barely any IOPS, however on our storage controller we've noticed a constant steam of reads, about 2000 IOPS; any idea what might cause the ovirt hosts to hammer on the storage so much? Looking at iotop, it seems that systemd-udevd is being activated every few seconds (and its IO% is usually fairly high) so I assume this might be somehow related. Thanks, Andrea ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] high number of IOPS from ovirt hosts
Hi Simone, we are exposing 2x 5TB LUNs, used solely by the ovirt cluster. Thanks Andrea On 12/02/2017 20:21, Simone Tiraboschi wrote: On Fri, Feb 10, 2017 at 12:46 PM, Andrea Fagiani mailto:andrea.fagi...@immobiliare.it>> wrote: Hi, We've got a 8-host cluster running oVirt 4.1; each host is connected via iscsi to a EQL 4100 storage (2x). Most of the VMs use barely any IOPS, however on our storage controller we've noticed a constant steam of reads, about 2000 IOPS; any idea what might cause the ovirt hosts to hammer on the storage so much? Looking at iotop, it seems that systemd-udevd is being activated every few seconds (and its IO% is usually fairly high) so I assume this might be somehow related. mmmh this could be an hint: how many LUNs are you exposing on the iSCSI portal used for the hosted-engine LUN? Thanks, Andrea ___ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users <http://lists.ovirt.org/mailman/listinfo/users> ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] high number of IOPS from ovirt hosts
Indeed, deploying a few hosts without the hosted engine immediately resulted in a noticeable decrease in IOPS; thanks a lot for the help, now we can figure out a way to work around the issue. Thanks, Andrea On 13/02/2017 10:43, Andrea Ghelardi wrote: Andrea, Among other advices, I would also suggest you to re-deploy an host without make it an host-engine node. It may be worth checking if HE service is responsible of so many IOPS as it was in my case. Ciao AG -Original Message- From: Andrea Fagiani [mailto:andrea.fagi...@immobiliare.it] Sent: Monday, February 13, 2017 10:05 AM To: Andrea Ghelardi ; users@ovirt.org Subject: Re: [ovirt-users] high number of IOPS from ovirt hosts Thanks for the heads up, I'll take a look at Mark's thread. We are presenting 2 storage LUNs to a single ovirt cluster and yes, all hosts are hosted engine hosts. Thanks, Andrea On 12/02/2017 18:27, Andrea Ghelardi wrote: Andrea, I don't want to point any fingers but Mark Greenall faced some issues with Ovirt 4.06/4.1 linked to equallogic as well. You may want to search posts in this ML with subject " Ovirt 4.0.6 guests 'Not Responding' " to see if you find any similarities. How many storages are you presenting to how many clusters? Also, are your nodes all Hosted Engine Hosts? In my old configuration, normal hosts read IOPS ranged from 50 to 500 while HE ones peaked even ~3000 Cheers AG -Original Message- From: Andrea Fagiani [mailto:andrea.fagi...@immobiliare.it] Sent: Friday, February 10, 2017 5:56 PM To: Andrea Ghelardi ; users@ovirt.org Subject: Re: [ovirt-users] high number of IOPS from ovirt hosts Hi Andrea, thanks for the insight; no I wasn't able to pinpoint the issue to a single host, the load seems to be evenly spread. We have another cluster running 4.0.6 on a compellent as well and we haven't experienced any similar issues. Thanks, Andrea On 10/02/2017 17:17, Andrea Ghelardi wrote: Hello Andrea, I saw that happening on my HE hosts connected to a COMPELLENT SC040 when I was running Ovirt 3.5 All HE hosts were hammering the SAN with 2~4k IOPS each Upgrading to 4.0.5 solved my case. Are you able to check if IOPS are generated from a particular host or rather spread among all nodes? Cheers AG -Original Message- From: users-boun...@ovirt.org [mailto:users-boun...@ovirt.org] On Behalf Of Andrea Fagiani Sent: Friday, February 10, 2017 12:47 PM To: users@ovirt.org Subject: [ovirt-users] high number of IOPS from ovirt hosts Hi, We've got a 8-host cluster running oVirt 4.1; each host is connected via iscsi to a EQL 4100 storage (2x). Most of the VMs use barely any IOPS, however on our storage controller we've noticed a constant steam of reads, about 2000 IOPS; any idea what might cause the ovirt hosts to hammer on the storage so much? Looking at iotop, it seems that systemd-udevd is being activated every few seconds (and its IO% is usually fairly high) so I assume this might be somehow related. Thanks, Andrea ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] high number of IOPS from ovirt hosts
Hi, just a quick update on the issue. The high amount of IOPS was indeed due to having the hosted-engine on the EQL storage; we worked around the issue by moving the hosted-engine to a gluster storage instead. Thanks to everyone involved, Andrea On 13/02/2017 13:24, Nir Soffer wrote: On Mon, Feb 13, 2017 at 12:32 PM, Andrea Fagiani mailto:andrea.fagi...@immobiliare.it>> wrote: Hi Nir, the amount of IOPS is pretty much constant, attaching the logs from the last hour or so. Andrea, can you send also logs from ovirt-hosted-engine-ha? (I'm not sure about the name of the service or the location of the logs) Andrea On 13/02/2017 10:13, Nir Soffer wrote: On Mon, Feb 13, 2017 at 11:05 AM, Andrea Fagiani mailto:andrea.fagi...@immobiliare.it>> wrote: Thanks for the heads up, I'll take a look at Mark's thread. We are presenting 2 storage LUNs to a single ovirt cluster and yes, all hosts are hosted engine hosts. Thanks, Andrea I think we already have hosted engine bug about doing to many calls to vdsm that cause reloads of lvm data; this can cause increased io. Can you attach vdsm and hosted engine agent logs showing the timeframe when you have lot of iops on the server? Nir On 12/02/2017 18:27, Andrea Ghelardi wrote: Andrea, I don't want to point any fingers but Mark Greenall faced some issues with Ovirt 4.06/4.1 linked to equallogic as well. You may want to search posts in this ML with subject " Ovirt 4.0.6 guests 'Not Responding' " to see if you find any similarities. How many storages are you presenting to how many clusters? Also, are your nodes all Hosted Engine Hosts? In my old configuration, normal hosts read IOPS ranged from 50 to 500 while HE ones peaked even ~3000 Cheers AG -Original Message- From: Andrea Fagiani [mailto:andrea.fagi...@immobiliare.it <mailto:andrea.fagi...@immobiliare.it>] Sent: Friday, February 10, 2017 5:56 PM To: Andrea Ghelardi mailto:a.ghela...@iontrading.com>>; users@ovirt.org <mailto:users@ovirt.org> Subject: Re: [ovirt-users] high number of IOPS from ovirt hosts Hi Andrea, thanks for the insight; no I wasn't able to pinpoint the issue to a single host, the load seems to be evenly spread. We have another cluster running 4.0.6 on a compellent as well and we haven't experienced any similar issues. Thanks, Andrea On 10/02/2017 17:17, Andrea Ghelardi wrote: Hello Andrea, I saw that happening on my HE hosts connected to a COMPELLENT SC040 when I was running Ovirt 3.5 All HE hosts were hammering the SAN with 2~4k IOPS each Upgrading to 4.0.5 solved my case. Are you able to check if IOPS are generated from a particular host or rather spread among all nodes? Cheers AG -Original Message- From: users-boun...@ovirt.org <mailto:users-boun...@ovirt.org> [mailto:users-boun...@ovirt.org <mailto:users-boun...@ovirt.org>] On Behalf Of Andrea Fagiani Sent: Friday, February 10, 2017 12:47 PM To: users@ovirt.org <mailto:users@ovirt.org> Subject: [ovirt-users] high number of IOPS from ovirt hosts Hi, We've got a 8-host cluster running oVirt 4.1; each host is connected via iscsi to a EQL 4100 storage (2x). Most of the VMs use barely any IOPS, however on our storage controller we've noticed a constant steam of reads, about 2000 IOPS; any idea what might cause the ovirt hosts to hammer on the storage so much? Looking at iotop, it seems that systemd-udevd is being activated every few seconds (and its IO% is usually fairly high) so I assume this might be somehow related. Thanks, Andrea ___ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users <http://lists.ovirt.org/mailman/listinfo/users> ___ Users mailing list Users@ovirt.or
Re: [ovirt-users] high number of IOPS from ovirt hosts
Correct, coincidentally, there no instances of systemd-udevd going haywire either. Andrea On 14/02/2017 15:20, Yaniv Kaul wrote: On Tue, Feb 14, 2017 at 2:15 PM, Andrea Fagiani mailto:andrea.fagi...@immobiliare.it>> wrote: Hi, just a quick update on the issue. The high amount of IOPS was indeed due to having the hosted-engine on the EQL storage; we worked around the issue by moving the hosted-engine to a gluster storage instead. And now there are no high amount of IOPS on the Gluster storage? Y. Thanks to everyone involved, Andrea On 13/02/2017 13:24, Nir Soffer wrote: On Mon, Feb 13, 2017 at 12:32 PM, Andrea Fagiani mailto:andrea.fagi...@immobiliare.it>> wrote: Hi Nir, the amount of IOPS is pretty much constant, attaching the logs from the last hour or so. Andrea, can you send also logs from ovirt-hosted-engine-ha? (I'm not sure about the name of the service or the location of the logs) Andrea On 13/02/2017 10:13, Nir Soffer wrote: On Mon, Feb 13, 2017 at 11:05 AM, Andrea Fagiani mailto:andrea.fagi...@immobiliare.it>> wrote: Thanks for the heads up, I'll take a look at Mark's thread. We are presenting 2 storage LUNs to a single ovirt cluster and yes, all hosts are hosted engine hosts. Thanks, Andrea I think we already have hosted engine bug about doing to many calls to vdsm that cause reloads of lvm data; this can cause increased io. Can you attach vdsm and hosted engine agent logs showing the timeframe when you have lot of iops on the server? Nir On 12/02/2017 18:27, Andrea Ghelardi wrote: Andrea, I don't want to point any fingers but Mark Greenall faced some issues with Ovirt 4.06/4.1 linked to equallogic as well. You may want to search posts in this ML with subject " Ovirt 4.0.6 guests 'Not Responding' " to see if you find any similarities. How many storages are you presenting to how many clusters? Also, are your nodes all Hosted Engine Hosts? In my old configuration, normal hosts read IOPS ranged from 50 to 500 while HE ones peaked even ~3000 Cheers AG -Original Message- From: Andrea Fagiani [mailto:andrea.fagi...@immobiliare.it <mailto:andrea.fagi...@immobiliare.it>] Sent: Friday, February 10, 2017 5:56 PM To: Andrea Ghelardi mailto:a.ghela...@iontrading.com>>; users@ovirt.org <mailto:users@ovirt.org> Subject: Re: [ovirt-users] high number of IOPS from ovirt hosts Hi Andrea, thanks for the insight; no I wasn't able to pinpoint the issue to a single host, the load seems to be evenly spread. We have another cluster running 4.0.6 on a compellent as well and we haven't experienced any similar issues. Thanks, Andrea On 10/02/2017 17:17, Andrea Ghelardi wrote: Hello Andrea, I saw that happening on my HE hosts connected to a COMPELLENT SC040 when I was running Ovirt 3.5 All HE hosts were hammering the SAN with 2~4k IOPS each Upgrading to 4.0.5 solved my case. Are you able to check if IOPS are generated from a particular host or rather spread among all nodes? Cheers AG -Original Message- From: users-boun...@ovirt.org <mailto:users-boun...@ovirt.org> [mailto:users-boun...@ovirt.org <mailto:users-boun...@ovirt.org>] On Behalf Of Andrea Fagiani Sent: Friday, February 10, 2017 12:47 PM To: users@ovirt.org <mailto:users@ovirt.org> Subject: [ovirt-users] high number of IOPS from ovirt hosts Hi, We've got a 8-host cluster running oVirt 4.1; each host is connected via iscsi to a EQL 4100 storage (2x). Most of the VMs use barely any IOPS, however on our storage controller we've noticed a constant steam of reads, about 2000 IOPS;