Re: [ovirt-users] high number of IOPS from ovirt hosts

[Andrea Fagiani]

Correct,
coincidentally, there no instances of systemd-udevd going haywire either.

Andrea


On 14/02/2017 15:20, Yaniv Kaul wrote:



On Tue, Feb 14, 2017 at 2:15 PM, Andrea Fagiani 
mailto:andrea.fagi...@immobiliare.it>> 
wrote:


Hi,

just a quick update on the issue. The high amount of IOPS was
indeed due to having the hosted-engine on the EQL storage; we
worked around the issue by moving the hosted-engine to a gluster
storage instead.


And now there are no high amount of IOPS on the Gluster storage?
Y.

Thanks to everyone involved,
Andrea


On 13/02/2017 13:24, Nir Soffer wrote:

On Mon, Feb 13, 2017 at 12:32 PM, Andrea Fagiani
mailto:andrea.fagi...@immobiliare.it>> wrote:

Hi Nir,

the amount of IOPS is pretty much constant, attaching the
logs from the last hour or so.


Andrea, can you send also logs from ovirt-hosted-engine-ha?
(I'm not sure about the name of the service or the location of
the logs)

Andrea


On 13/02/2017 10:13, Nir Soffer wrote:

On Mon, Feb 13, 2017 at 11:05 AM, Andrea Fagiani
mailto:andrea.fagi...@immobiliare.it>> wrote:

Thanks for the heads up, I'll take a look at Mark's thread.
We are presenting 2 storage LUNs to a single ovirt
cluster and yes, all hosts are hosted engine hosts.

Thanks,
Andrea


I think we already have hosted engine bug about doing to
many calls to vdsm
that cause reloads of lvm data; this can cause increased io.

Can you attach vdsm and hosted engine agent logs showing the
timeframe when
you have lot of iops on the server?

Nir




On 12/02/2017 18:27, Andrea Ghelardi wrote:

Andrea,
I don't want to point any fingers but Mark Greenall
faced some issues with Ovirt 4.06/4.1 linked to
equallogic as well.
You may want to search posts in this ML with subject
" Ovirt 4.0.6 guests 'Not Responding' " to see if
you find any similarities.
How many storages are you presenting to how many
clusters?
Also, are your nodes all Hosted Engine Hosts? In my
old configuration, normal hosts read IOPS ranged
from 50 to 500 while HE ones peaked even ~3000
Cheers
AG


-Original Message-
From: Andrea Fagiani
[mailto:andrea.fagi...@immobiliare.it
<mailto:andrea.fagi...@immobiliare.it>]
Sent: Friday, February 10, 2017 5:56 PM
To: Andrea Ghelardi mailto:a.ghela...@iontrading.com>>; users@ovirt.org
        <mailto:users@ovirt.org>
    Subject: Re: [ovirt-users] high number of IOPS from
ovirt hosts

Hi Andrea,

thanks for the insight; no I wasn't able to pinpoint
the issue to a single host, the load seems to be
evenly spread.
We have another cluster running 4.0.6 on a
compellent as well and we haven't experienced any
similar issues.

Thanks,
Andrea


On 10/02/2017 17:17, Andrea Ghelardi wrote:

Hello Andrea,
I saw that happening on my HE hosts connected to
a COMPELLENT SC040
when I was running Ovirt 3.5

All HE hosts were hammering the SAN with 2~4k
IOPS each

Upgrading to 4.0.5 solved my case.
Are you able to check if IOPS are generated from
a particular host or rather spread among all nodes?

Cheers
AG


-Original Message-
From: users-boun...@ovirt.org
<mailto:users-boun...@ovirt.org>
[mailto:users-boun...@ovirt.org
<mailto:users-boun...@ovirt.org>] On
Behalf Of Andrea Fagiani
Sent: Friday, February 10, 2017 12:47 PM
To: users@ovirt.org <mailto:users@ovirt.org>
Subject: [ovirt-users] high number of IOPS from
ovirt hosts

Hi,

We've got a 8-host cluster running oVirt 4.1;
each host is connected via iscsi to a EQL 4100
storage (2x).
Most of the VMs use barely any IOPS, however on
our storage controller we've noticed a constant
steam of reads, about 2000 IOPS; 

Re: [ovirt-users] high number of IOPS from ovirt hosts

[Yaniv Kaul]

On Tue, Feb 14, 2017 at 2:15 PM, Andrea Fagiani <
andrea.fagi...@immobiliare.it> wrote:

> Hi,
>
> just a quick update on the issue. The high amount of IOPS was indeed due
> to having the hosted-engine on the EQL storage; we worked around the issue
> by moving the hosted-engine to a gluster storage instead.
>

And now there are no high amount of IOPS on the Gluster storage?
Y.

Thanks to everyone involved,
> Andrea
>
> On 13/02/2017 13:24, Nir Soffer wrote:
>
> On Mon, Feb 13, 2017 at 12:32 PM, Andrea Fagiani <
> andrea.fagi...@immobiliare.it> wrote:
>
>> Hi Nir,
>>
>> the amount of IOPS is pretty much constant, attaching the logs from the
>> last hour or so.
>>
>
> Andrea, can you send also logs from ovirt-hosted-engine-ha?
> (I'm not sure about the name of the service or the location of the logs)
>
>
>> Andrea
>>
>> On 13/02/2017 10:13, Nir Soffer wrote:
>>
>> On Mon, Feb 13, 2017 at 11:05 AM, Andrea Fagiani <
>> andrea.fagi...@immobiliare.it> wrote:
>>
>>> Thanks for the heads up, I'll take a look at Mark's thread.
>>> We are presenting 2 storage LUNs to a single ovirt cluster and yes, all
>>> hosts are hosted engine hosts.
>>>
>>> Thanks,
>>> Andrea
>>
>>
>> I think we already have hosted engine bug about doing to many calls to
>> vdsm
>> that cause reloads of lvm data; this can cause increased io.
>>
>> Can you attach vdsm and hosted engine agent logs showing the timeframe
>> when
>> you have lot of iops on the server?
>>
>> Nir
>>
>>
>>>
>>>
>>>
>>> On 12/02/2017 18:27, Andrea Ghelardi wrote:
>>>
>>>> Andrea,
>>>> I don't want to point any fingers but Mark Greenall faced some issues
>>>> with Ovirt 4.06/4.1 linked to equallogic as well.
>>>> You may want to search posts in this ML with subject " Ovirt 4.0.6
>>>> guests 'Not Responding' " to see if you find any similarities.
>>>> How many storages are you presenting to how many clusters?
>>>> Also, are your nodes all Hosted Engine Hosts? In my old configuration,
>>>> normal hosts read IOPS ranged from 50 to 500 while HE ones peaked even 
>>>> ~3000
>>>> Cheers
>>>> AG
>>>>
>>>>
>>>> -Original Message-
>>>> From: Andrea Fagiani [mailto:andrea.fagi...@immobiliare.it]
>>>> Sent: Friday, February 10, 2017 5:56 PM
>>>> To: Andrea Ghelardi ; users@ovirt.org
>>>> Subject: Re: [ovirt-users] high number of IOPS from ovirt hosts
>>>>
>>>> Hi Andrea,
>>>>
>>>> thanks for the insight; no I wasn't able to pinpoint the issue to a
>>>> single host, the load seems to be evenly spread.
>>>> We have another cluster running 4.0.6 on a compellent as well and we
>>>> haven't experienced any similar issues.
>>>>
>>>> Thanks,
>>>> Andrea
>>>>
>>>>
>>>> On 10/02/2017 17:17, Andrea Ghelardi wrote:
>>>>
>>>>> Hello Andrea,
>>>>> I saw that happening on my HE hosts connected to a COMPELLENT SC040
>>>>> when I was running Ovirt 3.5
>>>>>
>>>>> All HE hosts were hammering the SAN with 2~4k IOPS each
>>>>>
>>>>> Upgrading to 4.0.5 solved my case.
>>>>> Are you able to check if IOPS are generated from a particular host or
>>>>> rather spread among all nodes?
>>>>>
>>>>> Cheers
>>>>> AG
>>>>>
>>>>>
>>>>> -Original Message-
>>>>> From: users-boun...@ovirt.org [mailto:users-boun...@ovirt.org] On
>>>>> Behalf Of Andrea Fagiani
>>>>> Sent: Friday, February 10, 2017 12:47 PM
>>>>> To: users@ovirt.org
>>>>> Subject: [ovirt-users] high number of IOPS from ovirt hosts
>>>>>
>>>>> Hi,
>>>>>
>>>>> We've got a 8-host cluster running oVirt 4.1; each host is connected
>>>>> via iscsi to a EQL 4100 storage (2x).
>>>>> Most of the VMs use barely any IOPS, however on our storage controller
>>>>> we've noticed a constant steam of reads, about 2000 IOPS; any idea what
>>>>> might cause the ovirt hosts to hammer on the storage so much?
>>>>>
>>>>> Looking at iotop, it seems that systemd-udevd is being activated every
>>>>> few seconds (and its IO% is usually fairly high) so I assume this might be
>>>>> somehow related.
>>>>>
>>>>> Thanks,
>>>>> Andrea
>>>>> ___
>>>>> Users mailing list
>>>>> Users@ovirt.org
>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>>
>>>>
>>> ___
>>> Users mailing list
>>> Users@ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/users
>>>
>>
>>
>>
>
>
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] high number of IOPS from ovirt hosts

[Andrea Fagiani]

Hi,

just a quick update on the issue. The high amount of IOPS was indeed due 
to having the hosted-engine on the EQL storage; we worked around the 
issue by moving the hosted-engine to a gluster storage instead.


Thanks to everyone involved,
Andrea


On 13/02/2017 13:24, Nir Soffer wrote:
On Mon, Feb 13, 2017 at 12:32 PM, Andrea Fagiani 
mailto:andrea.fagi...@immobiliare.it>> 
wrote:


Hi Nir,

the amount of IOPS is pretty much constant, attaching the logs
from the last hour or so.


Andrea, can you send also logs from ovirt-hosted-engine-ha?
(I'm not sure about the name of the service or the location of the logs)

Andrea


On 13/02/2017 10:13, Nir Soffer wrote:

On Mon, Feb 13, 2017 at 11:05 AM, Andrea Fagiani
mailto:andrea.fagi...@immobiliare.it>> wrote:

Thanks for the heads up, I'll take a look at Mark's thread.
We are presenting 2 storage LUNs to a single ovirt cluster
and yes, all hosts are hosted engine hosts.

Thanks,
Andrea


I think we already have hosted engine bug about doing to many
calls to vdsm
that cause reloads of lvm data; this can cause increased io.

Can you attach vdsm and hosted engine agent logs showing the
timeframe when
you have lot of iops on the server?

Nir




On 12/02/2017 18:27, Andrea Ghelardi wrote:

Andrea,
I don't want to point any fingers but Mark Greenall faced
some issues with Ovirt 4.06/4.1 linked to equallogic as well.
You may want to search posts in this ML with subject "
Ovirt 4.0.6 guests 'Not Responding' " to see if you find
any similarities.
How many storages are you presenting to how many clusters?
Also, are your nodes all Hosted Engine Hosts? In my old
configuration, normal hosts read IOPS ranged from 50 to
500 while HE ones peaked even ~3000
Cheers
AG


-Original Message-
From: Andrea Fagiani
[mailto:andrea.fagi...@immobiliare.it
<mailto:andrea.fagi...@immobiliare.it>]
Sent: Friday, February 10, 2017 5:56 PM
To: Andrea Ghelardi mailto:a.ghela...@iontrading.com>>; users@ovirt.org
    <mailto:users@ovirt.org>
Subject: Re: [ovirt-users] high number of IOPS from ovirt
hosts

Hi Andrea,

thanks for the insight; no I wasn't able to pinpoint the
issue to a single host, the load seems to be evenly spread.
We have another cluster running 4.0.6 on a compellent as
well and we haven't experienced any similar issues.

Thanks,
Andrea


On 10/02/2017 17:17, Andrea Ghelardi wrote:

Hello Andrea,
I saw that happening on my HE hosts connected to a
COMPELLENT SC040
when I was running Ovirt 3.5

All HE hosts were hammering the SAN with 2~4k IOPS each

Upgrading to 4.0.5 solved my case.
Are you able to check if IOPS are generated from a
particular host or rather spread among all nodes?

Cheers
AG


-Original Message-
From: users-boun...@ovirt.org
<mailto:users-boun...@ovirt.org>
[mailto:users-boun...@ovirt.org
<mailto:users-boun...@ovirt.org>] On
Behalf Of Andrea Fagiani
Sent: Friday, February 10, 2017 12:47 PM
To: users@ovirt.org <mailto:users@ovirt.org>
Subject: [ovirt-users] high number of IOPS from ovirt
hosts

Hi,

We've got a 8-host cluster running oVirt 4.1; each
host is connected via iscsi to a EQL 4100 storage (2x).
Most of the VMs use barely any IOPS, however on our
storage controller we've noticed a constant steam of
reads, about 2000 IOPS; any idea what might cause the
ovirt hosts to hammer on the storage so much?

Looking at iotop, it seems that systemd-udevd is
being activated every few seconds (and its IO% is
usually fairly high) so I assume this might be
somehow related.

Thanks,
Andrea
___
Users mailing list
Users@ovirt.org <mailto:Users@ovirt.org>
http://lists.ovirt.org/mailman/listinfo/users
<http://lists.ovirt.org/mailman/listinfo/users>


___
Users mailing list
Users@ovirt.or

Re: [ovirt-users] high number of IOPS from ovirt hosts

[Martin Sivak]

> Andrea, can you send also logs from ovirt-hosted-engine-ha?
> (I'm not sure about the name of the service or the location of the logs)

/var/log/ovirt-hosted-engine-ha/agent.log :)

But you will see all the prepareImage and repostats calls there, we
still haven't tackled that issue.

Martin

On Mon, Feb 13, 2017 at 1:24 PM, Nir Soffer  wrote:
> On Mon, Feb 13, 2017 at 12:32 PM, Andrea Fagiani
>  wrote:
>>
>> Hi Nir,
>>
>> the amount of IOPS is pretty much constant, attaching the logs from the
>> last hour or so.
>
>
> Andrea, can you send also logs from ovirt-hosted-engine-ha?
> (I'm not sure about the name of the service or the location of the logs)
>
>>
>> Andrea
>>
>>
>> On 13/02/2017 10:13, Nir Soffer wrote:
>>
>> On Mon, Feb 13, 2017 at 11:05 AM, Andrea Fagiani
>>  wrote:
>>>
>>> Thanks for the heads up, I'll take a look at Mark's thread.
>>> We are presenting 2 storage LUNs to a single ovirt cluster and yes, all
>>> hosts are hosted engine hosts.
>>>
>>> Thanks,
>>> Andrea
>>
>>
>> I think we already have hosted engine bug about doing to many calls to
>> vdsm
>> that cause reloads of lvm data; this can cause increased io.
>>
>> Can you attach vdsm and hosted engine agent logs showing the timeframe
>> when
>> you have lot of iops on the server?
>>
>> Nir
>>
>>>
>>>
>>>
>>>
>>> On 12/02/2017 18:27, Andrea Ghelardi wrote:
>>>>
>>>> Andrea,
>>>> I don't want to point any fingers but Mark Greenall faced some issues
>>>> with Ovirt 4.06/4.1 linked to equallogic as well.
>>>> You may want to search posts in this ML with subject " Ovirt 4.0.6
>>>> guests 'Not Responding' " to see if you find any similarities.
>>>> How many storages are you presenting to how many clusters?
>>>> Also, are your nodes all Hosted Engine Hosts? In my old configuration,
>>>> normal hosts read IOPS ranged from 50 to 500 while HE ones peaked even 
>>>> ~3000
>>>> Cheers
>>>> AG
>>>>
>>>>
>>>> -Original Message-
>>>> From: Andrea Fagiani [mailto:andrea.fagi...@immobiliare.it]
>>>> Sent: Friday, February 10, 2017 5:56 PM
>>>> To: Andrea Ghelardi ; users@ovirt.org
>>>> Subject: Re: [ovirt-users] high number of IOPS from ovirt hosts
>>>>
>>>> Hi Andrea,
>>>>
>>>> thanks for the insight; no I wasn't able to pinpoint the issue to a
>>>> single host, the load seems to be evenly spread.
>>>> We have another cluster running 4.0.6 on a compellent as well and we
>>>> haven't experienced any similar issues.
>>>>
>>>> Thanks,
>>>> Andrea
>>>>
>>>>
>>>> On 10/02/2017 17:17, Andrea Ghelardi wrote:
>>>>>
>>>>> Hello Andrea,
>>>>> I saw that happening on my HE hosts connected to a COMPELLENT SC040
>>>>> when I was running Ovirt 3.5
>>>>>
>>>>> All HE hosts were hammering the SAN with 2~4k IOPS each
>>>>>
>>>>> Upgrading to 4.0.5 solved my case.
>>>>> Are you able to check if IOPS are generated from a particular host or
>>>>> rather spread among all nodes?
>>>>>
>>>>> Cheers
>>>>> AG
>>>>>
>>>>>
>>>>> -Original Message-
>>>>> From: users-boun...@ovirt.org [mailto:users-boun...@ovirt.org] On
>>>>> Behalf Of Andrea Fagiani
>>>>> Sent: Friday, February 10, 2017 12:47 PM
>>>>> To: users@ovirt.org
>>>>> Subject: [ovirt-users] high number of IOPS from ovirt hosts
>>>>>
>>>>> Hi,
>>>>>
>>>>> We've got a 8-host cluster running oVirt 4.1; each host is connected
>>>>> via iscsi to a EQL 4100 storage (2x).
>>>>> Most of the VMs use barely any IOPS, however on our storage controller
>>>>> we've noticed a constant steam of reads, about 2000 IOPS; any idea what
>>>>> might cause the ovirt hosts to hammer on the storage so much?
>>>>>
>>>>> Looking at iotop, it seems that systemd-udevd is being activated every
>>>>> few seconds (and its IO% is usually fairly high) so I assume this might be
>>>>> somehow related.
>>>>>
>>>>> Thanks,
>>>>> Andrea
>>>>> ___
>>>>> Users mailing list
>>>>> Users@ovirt.org
>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>
>>>
>>> ___
>>> Users mailing list
>>> Users@ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/users
>>
>>
>>
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] high number of IOPS from ovirt hosts

[Nir Soffer]

On Mon, Feb 13, 2017 at 12:32 PM, Andrea Fagiani <
andrea.fagi...@immobiliare.it> wrote:

> Hi Nir,
>
> the amount of IOPS is pretty much constant, attaching the logs from the
> last hour or so.
>

Andrea, can you send also logs from ovirt-hosted-engine-ha?
(I'm not sure about the name of the service or the location of the logs)


> Andrea
>
> On 13/02/2017 10:13, Nir Soffer wrote:
>
> On Mon, Feb 13, 2017 at 11:05 AM, Andrea Fagiani <
> andrea.fagi...@immobiliare.it> wrote:
>
>> Thanks for the heads up, I'll take a look at Mark's thread.
>> We are presenting 2 storage LUNs to a single ovirt cluster and yes, all
>> hosts are hosted engine hosts.
>>
>> Thanks,
>> Andrea
>
>
> I think we already have hosted engine bug about doing to many calls to vdsm
> that cause reloads of lvm data; this can cause increased io.
>
> Can you attach vdsm and hosted engine agent logs showing the timeframe
> when
> you have lot of iops on the server?
>
> Nir
>
>
>>
>>
>>
>> On 12/02/2017 18:27, Andrea Ghelardi wrote:
>>
>>> Andrea,
>>> I don't want to point any fingers but Mark Greenall faced some issues
>>> with Ovirt 4.06/4.1 linked to equallogic as well.
>>> You may want to search posts in this ML with subject " Ovirt 4.0.6
>>> guests 'Not Responding' " to see if you find any similarities.
>>> How many storages are you presenting to how many clusters?
>>> Also, are your nodes all Hosted Engine Hosts? In my old configuration,
>>> normal hosts read IOPS ranged from 50 to 500 while HE ones peaked even ~3000
>>> Cheers
>>> AG
>>>
>>>
>>> -Original Message-
>>> From: Andrea Fagiani [mailto:andrea.fagi...@immobiliare.it]
>>> Sent: Friday, February 10, 2017 5:56 PM
>>> To: Andrea Ghelardi ; users@ovirt.org
>>> Subject: Re: [ovirt-users] high number of IOPS from ovirt hosts
>>>
>>> Hi Andrea,
>>>
>>> thanks for the insight; no I wasn't able to pinpoint the issue to a
>>> single host, the load seems to be evenly spread.
>>> We have another cluster running 4.0.6 on a compellent as well and we
>>> haven't experienced any similar issues.
>>>
>>> Thanks,
>>> Andrea
>>>
>>>
>>> On 10/02/2017 17:17, Andrea Ghelardi wrote:
>>>
>>>> Hello Andrea,
>>>> I saw that happening on my HE hosts connected to a COMPELLENT SC040
>>>> when I was running Ovirt 3.5
>>>>
>>>> All HE hosts were hammering the SAN with 2~4k IOPS each
>>>>
>>>> Upgrading to 4.0.5 solved my case.
>>>> Are you able to check if IOPS are generated from a particular host or
>>>> rather spread among all nodes?
>>>>
>>>> Cheers
>>>> AG
>>>>
>>>>
>>>> -Original Message-
>>>> From: users-boun...@ovirt.org [mailto:users-boun...@ovirt.org] On
>>>> Behalf Of Andrea Fagiani
>>>> Sent: Friday, February 10, 2017 12:47 PM
>>>> To: users@ovirt.org
>>>> Subject: [ovirt-users] high number of IOPS from ovirt hosts
>>>>
>>>> Hi,
>>>>
>>>> We've got a 8-host cluster running oVirt 4.1; each host is connected
>>>> via iscsi to a EQL 4100 storage (2x).
>>>> Most of the VMs use barely any IOPS, however on our storage controller
>>>> we've noticed a constant steam of reads, about 2000 IOPS; any idea what
>>>> might cause the ovirt hosts to hammer on the storage so much?
>>>>
>>>> Looking at iotop, it seems that systemd-udevd is being activated every
>>>> few seconds (and its IO% is usually fairly high) so I assume this might be
>>>> somehow related.
>>>>
>>>> Thanks,
>>>> Andrea
>>>> ___
>>>> Users mailing list
>>>> Users@ovirt.org
>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>
>>>
>> ___
>> Users mailing list
>> Users@ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>>
>
>
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] high number of IOPS from ovirt hosts

[Andrea Fagiani]

Indeed, deploying a few hosts without the hosted engine immediately 
resulted in a noticeable decrease in IOPS; thanks a lot for the help, 
now we can figure out a way to work around the issue.


Thanks,
Andrea


On 13/02/2017 10:43, Andrea Ghelardi wrote:

Andrea,
Among other advices, I would also suggest you to re-deploy an host without make 
it an host-engine node.
It may be worth checking if HE service is responsible of so many IOPS as it was 
in my case.
Ciao
AG

-Original Message-
From: Andrea Fagiani [mailto:andrea.fagi...@immobiliare.it]
Sent: Monday, February 13, 2017 10:05 AM
To: Andrea Ghelardi ; users@ovirt.org
Subject: Re: [ovirt-users] high number of IOPS from ovirt hosts

Thanks for the heads up, I'll take a look at Mark's thread.
We are presenting 2 storage LUNs to a single ovirt cluster and yes, all hosts 
are hosted engine hosts.

Thanks,
Andrea


On 12/02/2017 18:27, Andrea Ghelardi wrote:

Andrea,
I don't want to point any fingers but Mark Greenall faced some issues with 
Ovirt 4.06/4.1 linked to equallogic as well.
You may want to search posts in this ML with subject " Ovirt 4.0.6 guests 'Not 
Responding' " to see if you find any similarities.
How many storages are you presenting to how many clusters?
Also, are your nodes all Hosted Engine Hosts? In my old configuration,
normal hosts read IOPS ranged from 50 to 500 while HE ones peaked even
~3000 Cheers AG


-Original Message-
From: Andrea Fagiani [mailto:andrea.fagi...@immobiliare.it]
Sent: Friday, February 10, 2017 5:56 PM
To: Andrea Ghelardi ; users@ovirt.org
Subject: Re: [ovirt-users] high number of IOPS from ovirt hosts

Hi Andrea,

thanks for the insight; no I wasn't able to pinpoint the issue to a single 
host, the load seems to be evenly spread.
We have another cluster running 4.0.6 on a compellent as well and we haven't 
experienced any similar issues.

Thanks,
Andrea


On 10/02/2017 17:17, Andrea Ghelardi wrote:

Hello Andrea,
I saw that happening on my HE hosts connected to a COMPELLENT SC040
when I was running Ovirt 3.5

All HE hosts were hammering the SAN with 2~4k IOPS each

Upgrading to 4.0.5 solved my case.
Are you able to check if IOPS are generated from a particular host or rather 
spread among all nodes?

Cheers
AG


-Original Message-
From: users-boun...@ovirt.org [mailto:users-boun...@ovirt.org] On
Behalf Of Andrea Fagiani
Sent: Friday, February 10, 2017 12:47 PM
To: users@ovirt.org
Subject: [ovirt-users] high number of IOPS from ovirt hosts

Hi,

We've got a 8-host cluster running oVirt 4.1; each host is connected via iscsi 
to a EQL 4100 storage (2x).
Most of the VMs use barely any IOPS, however on our storage controller we've 
noticed a constant steam of reads, about 2000 IOPS; any idea what might cause 
the ovirt hosts to hammer on the storage so much?

Looking at iotop, it seems that systemd-udevd is being activated every few 
seconds (and its IO% is usually fairly high) so I assume this might be somehow 
related.

Thanks,
Andrea
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users


___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] high number of IOPS from ovirt hosts

[Andrea Ghelardi]

Andrea,
Among other advices, I would also suggest you to re-deploy an host without make 
it an host-engine node.
It may be worth checking if HE service is responsible of so many IOPS as it was 
in my case.
Ciao
AG

-Original Message-
From: Andrea Fagiani [mailto:andrea.fagi...@immobiliare.it] 
Sent: Monday, February 13, 2017 10:05 AM
To: Andrea Ghelardi ; users@ovirt.org
Subject: Re: [ovirt-users] high number of IOPS from ovirt hosts

Thanks for the heads up, I'll take a look at Mark's thread.
We are presenting 2 storage LUNs to a single ovirt cluster and yes, all hosts 
are hosted engine hosts.

Thanks,
Andrea


On 12/02/2017 18:27, Andrea Ghelardi wrote:
> Andrea,
> I don't want to point any fingers but Mark Greenall faced some issues with 
> Ovirt 4.06/4.1 linked to equallogic as well.
> You may want to search posts in this ML with subject " Ovirt 4.0.6 guests 
> 'Not Responding' " to see if you find any similarities.
> How many storages are you presenting to how many clusters?
> Also, are your nodes all Hosted Engine Hosts? In my old configuration, 
> normal hosts read IOPS ranged from 50 to 500 while HE ones peaked even 
> ~3000 Cheers AG
>
>
> -Original Message-
> From: Andrea Fagiani [mailto:andrea.fagi...@immobiliare.it]
> Sent: Friday, February 10, 2017 5:56 PM
> To: Andrea Ghelardi ; users@ovirt.org
> Subject: Re: [ovirt-users] high number of IOPS from ovirt hosts
>
> Hi Andrea,
>
> thanks for the insight; no I wasn't able to pinpoint the issue to a single 
> host, the load seems to be evenly spread.
> We have another cluster running 4.0.6 on a compellent as well and we haven't 
> experienced any similar issues.
>
> Thanks,
> Andrea
>
>
> On 10/02/2017 17:17, Andrea Ghelardi wrote:
>> Hello Andrea,
>> I saw that happening on my HE hosts connected to a COMPELLENT SC040 
>> when I was running Ovirt 3.5
>>
>> All HE hosts were hammering the SAN with 2~4k IOPS each
>>
>> Upgrading to 4.0.5 solved my case.
>> Are you able to check if IOPS are generated from a particular host or rather 
>> spread among all nodes?
>>
>> Cheers
>> AG
>>
>>
>> -Original Message-
>> From: users-boun...@ovirt.org [mailto:users-boun...@ovirt.org] On 
>> Behalf Of Andrea Fagiani
>> Sent: Friday, February 10, 2017 12:47 PM
>> To: users@ovirt.org
>> Subject: [ovirt-users] high number of IOPS from ovirt hosts
>>
>> Hi,
>>
>> We've got a 8-host cluster running oVirt 4.1; each host is connected via 
>> iscsi to a EQL 4100 storage (2x).
>> Most of the VMs use barely any IOPS, however on our storage controller we've 
>> noticed a constant steam of reads, about 2000 IOPS; any idea what might 
>> cause the ovirt hosts to hammer on the storage so much?
>>
>> Looking at iotop, it seems that systemd-udevd is being activated every few 
>> seconds (and its IO% is usually fairly high) so I assume this might be 
>> somehow related.
>>
>> Thanks,
>> Andrea
>> ___
>> Users mailing list
>> Users@ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users

___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] high number of IOPS from ovirt hosts

[Martin Sivak]

> I think we already have hosted engine bug about doing to many calls to vdsm
> that cause reloads of lvm data; this can cause increased io.

https://bugzilla.redhat.com/show_bug.cgi?id=1337914

Just for the reference.

Martin

On Mon, Feb 13, 2017 at 10:13 AM, Nir Soffer  wrote:
> On Mon, Feb 13, 2017 at 11:05 AM, Andrea Fagiani
>  wrote:
>>
>> Thanks for the heads up, I'll take a look at Mark's thread.
>> We are presenting 2 storage LUNs to a single ovirt cluster and yes, all
>> hosts are hosted engine hosts.
>>
>> Thanks,
>> Andrea
>
>


> Can you attach vdsm and hosted engine agent logs showing the timeframe when
> you have lot of iops on the server?
>
> Nir
>
>>
>>
>>
>>
>> On 12/02/2017 18:27, Andrea Ghelardi wrote:
>>>
>>> Andrea,
>>> I don't want to point any fingers but Mark Greenall faced some issues
>>> with Ovirt 4.06/4.1 linked to equallogic as well.
>>> You may want to search posts in this ML with subject " Ovirt 4.0.6 guests
>>> 'Not Responding' " to see if you find any similarities.
>>> How many storages are you presenting to how many clusters?
>>> Also, are your nodes all Hosted Engine Hosts? In my old configuration,
>>> normal hosts read IOPS ranged from 50 to 500 while HE ones peaked even ~3000
>>> Cheers
>>> AG
>>>
>>>
>>> -Original Message-
>>> From: Andrea Fagiani [mailto:andrea.fagi...@immobiliare.it]
>>> Sent: Friday, February 10, 2017 5:56 PM
>>> To: Andrea Ghelardi ; users@ovirt.org
>>> Subject: Re: [ovirt-users] high number of IOPS from ovirt hosts
>>>
>>> Hi Andrea,
>>>
>>> thanks for the insight; no I wasn't able to pinpoint the issue to a
>>> single host, the load seems to be evenly spread.
>>> We have another cluster running 4.0.6 on a compellent as well and we
>>> haven't experienced any similar issues.
>>>
>>> Thanks,
>>> Andrea
>>>
>>>
>>> On 10/02/2017 17:17, Andrea Ghelardi wrote:
>>>>
>>>> Hello Andrea,
>>>> I saw that happening on my HE hosts connected to a COMPELLENT SC040
>>>> when I was running Ovirt 3.5
>>>>
>>>> All HE hosts were hammering the SAN with 2~4k IOPS each
>>>>
>>>> Upgrading to 4.0.5 solved my case.
>>>> Are you able to check if IOPS are generated from a particular host or
>>>> rather spread among all nodes?
>>>>
>>>> Cheers
>>>> AG
>>>>
>>>>
>>>> -Original Message-
>>>> From: users-boun...@ovirt.org [mailto:users-boun...@ovirt.org] On
>>>> Behalf Of Andrea Fagiani
>>>> Sent: Friday, February 10, 2017 12:47 PM
>>>> To: users@ovirt.org
>>>> Subject: [ovirt-users] high number of IOPS from ovirt hosts
>>>>
>>>> Hi,
>>>>
>>>> We've got a 8-host cluster running oVirt 4.1; each host is connected via
>>>> iscsi to a EQL 4100 storage (2x).
>>>> Most of the VMs use barely any IOPS, however on our storage controller
>>>> we've noticed a constant steam of reads, about 2000 IOPS; any idea what
>>>> might cause the ovirt hosts to hammer on the storage so much?
>>>>
>>>> Looking at iotop, it seems that systemd-udevd is being activated every
>>>> few seconds (and its IO% is usually fairly high) so I assume this might be
>>>> somehow related.
>>>>
>>>> Thanks,
>>>> Andrea
>>>> ___
>>>> Users mailing list
>>>> Users@ovirt.org
>>>> http://lists.ovirt.org/mailman/listinfo/users
>>
>>
>> ___
>> Users mailing list
>> Users@ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>
>
>
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] high number of IOPS from ovirt hosts

[Nir Soffer]

On Mon, Feb 13, 2017 at 11:05 AM, Andrea Fagiani <
andrea.fagi...@immobiliare.it> wrote:

> Thanks for the heads up, I'll take a look at Mark's thread.
> We are presenting 2 storage LUNs to a single ovirt cluster and yes, all
> hosts are hosted engine hosts.
>
> Thanks,
> Andrea


I think we already have hosted engine bug about doing to many calls to vdsm
that cause reloads of lvm data; this can cause increased io.

Can you attach vdsm and hosted engine agent logs showing the timeframe when
you have lot of iops on the server?

Nir


>
>
>
> On 12/02/2017 18:27, Andrea Ghelardi wrote:
>
>> Andrea,
>> I don't want to point any fingers but Mark Greenall faced some issues
>> with Ovirt 4.06/4.1 linked to equallogic as well.
>> You may want to search posts in this ML with subject " Ovirt 4.0.6 guests
>> 'Not Responding' " to see if you find any similarities.
>> How many storages are you presenting to how many clusters?
>> Also, are your nodes all Hosted Engine Hosts? In my old configuration,
>> normal hosts read IOPS ranged from 50 to 500 while HE ones peaked even ~3000
>> Cheers
>> AG
>>
>>
>> -Original Message-
>> From: Andrea Fagiani [mailto:andrea.fagi...@immobiliare.it]
>> Sent: Friday, February 10, 2017 5:56 PM
>> To: Andrea Ghelardi ; users@ovirt.org
>> Subject: Re: [ovirt-users] high number of IOPS from ovirt hosts
>>
>> Hi Andrea,
>>
>> thanks for the insight; no I wasn't able to pinpoint the issue to a
>> single host, the load seems to be evenly spread.
>> We have another cluster running 4.0.6 on a compellent as well and we
>> haven't experienced any similar issues.
>>
>> Thanks,
>> Andrea
>>
>>
>> On 10/02/2017 17:17, Andrea Ghelardi wrote:
>>
>>> Hello Andrea,
>>> I saw that happening on my HE hosts connected to a COMPELLENT SC040
>>> when I was running Ovirt 3.5
>>>
>>> All HE hosts were hammering the SAN with 2~4k IOPS each
>>>
>>> Upgrading to 4.0.5 solved my case.
>>> Are you able to check if IOPS are generated from a particular host or
>>> rather spread among all nodes?
>>>
>>> Cheers
>>> AG
>>>
>>>
>>> -Original Message-
>>> From: users-boun...@ovirt.org [mailto:users-boun...@ovirt.org] On
>>> Behalf Of Andrea Fagiani
>>> Sent: Friday, February 10, 2017 12:47 PM
>>> To: users@ovirt.org
>>> Subject: [ovirt-users] high number of IOPS from ovirt hosts
>>>
>>> Hi,
>>>
>>> We've got a 8-host cluster running oVirt 4.1; each host is connected via
>>> iscsi to a EQL 4100 storage (2x).
>>> Most of the VMs use barely any IOPS, however on our storage controller
>>> we've noticed a constant steam of reads, about 2000 IOPS; any idea what
>>> might cause the ovirt hosts to hammer on the storage so much?
>>>
>>> Looking at iotop, it seems that systemd-udevd is being activated every
>>> few seconds (and its IO% is usually fairly high) so I assume this might be
>>> somehow related.
>>>
>>> Thanks,
>>> Andrea
>>> ___
>>> Users mailing list
>>> Users@ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/users
>>>
>>
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] high number of IOPS from ovirt hosts

[Andrea Fagiani]

Hi Simone,
we are exposing 2x 5TB LUNs, used solely by the ovirt cluster.

Thanks
Andrea

On 12/02/2017 20:21, Simone Tiraboschi wrote:



On Fri, Feb 10, 2017 at 12:46 PM, Andrea Fagiani 
mailto:andrea.fagi...@immobiliare.it>> 
wrote:


Hi,

We've got a 8-host cluster running oVirt 4.1; each host is
connected via iscsi to a EQL 4100 storage (2x).
Most of the VMs use barely any IOPS, however on our storage
controller we've noticed a constant steam of reads, about 2000
IOPS; any idea what might cause the ovirt hosts to hammer on the
storage so much?

Looking at iotop, it seems that systemd-udevd is being activated
every few seconds (and its IO% is usually fairly high) so I assume
this might be somehow related.


mmmh this could be an hint: how many LUNs are you exposing on the 
iSCSI portal used for the hosted-engine LUN?



Thanks,
Andrea
___
Users mailing list
Users@ovirt.org 
http://lists.ovirt.org/mailman/listinfo/users





___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] high number of IOPS from ovirt hosts

[Andrea Fagiani]

Thanks for the heads up, I'll take a look at Mark's thread.
We are presenting 2 storage LUNs to a single ovirt cluster and yes, all 
hosts are hosted engine hosts.


Thanks,
Andrea


On 12/02/2017 18:27, Andrea Ghelardi wrote:

Andrea,
I don't want to point any fingers but Mark Greenall faced some issues with 
Ovirt 4.06/4.1 linked to equallogic as well.
You may want to search posts in this ML with subject " Ovirt 4.0.6 guests 'Not 
Responding' " to see if you find any similarities.
How many storages are you presenting to how many clusters?
Also, are your nodes all Hosted Engine Hosts? In my old configuration, normal 
hosts read IOPS ranged from 50 to 500 while HE ones peaked even ~3000
Cheers
AG


-Original Message-
From: Andrea Fagiani [mailto:andrea.fagi...@immobiliare.it]
Sent: Friday, February 10, 2017 5:56 PM
To: Andrea Ghelardi ; users@ovirt.org
Subject: Re: [ovirt-users] high number of IOPS from ovirt hosts

Hi Andrea,

thanks for the insight; no I wasn't able to pinpoint the issue to a single 
host, the load seems to be evenly spread.
We have another cluster running 4.0.6 on a compellent as well and we haven't 
experienced any similar issues.

Thanks,
Andrea


On 10/02/2017 17:17, Andrea Ghelardi wrote:

Hello Andrea,
I saw that happening on my HE hosts connected to a COMPELLENT SC040
when I was running Ovirt 3.5

All HE hosts were hammering the SAN with 2~4k IOPS each

Upgrading to 4.0.5 solved my case.
Are you able to check if IOPS are generated from a particular host or rather 
spread among all nodes?

Cheers
AG


-Original Message-
From: users-boun...@ovirt.org [mailto:users-boun...@ovirt.org] On
Behalf Of Andrea Fagiani
Sent: Friday, February 10, 2017 12:47 PM
To: users@ovirt.org
Subject: [ovirt-users] high number of IOPS from ovirt hosts

Hi,

We've got a 8-host cluster running oVirt 4.1; each host is connected via iscsi 
to a EQL 4100 storage (2x).
Most of the VMs use barely any IOPS, however on our storage controller we've 
noticed a constant steam of reads, about 2000 IOPS; any idea what might cause 
the ovirt hosts to hammer on the storage so much?

Looking at iotop, it seems that systemd-udevd is being activated every few 
seconds (and its IO% is usually fairly high) so I assume this might be somehow 
related.

Thanks,
Andrea
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users


___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] high number of IOPS from ovirt hosts

[Simone Tiraboschi]

On Fri, Feb 10, 2017 at 12:46 PM, Andrea Fagiani <
andrea.fagi...@immobiliare.it> wrote:

> Hi,
>
> We've got a 8-host cluster running oVirt 4.1; each host is connected via
> iscsi to a EQL 4100 storage (2x).
> Most of the VMs use barely any IOPS, however on our storage controller
> we've noticed a constant steam of reads, about 2000 IOPS; any idea what
> might cause the ovirt hosts to hammer on the storage so much?
>
> Looking at iotop, it seems that systemd-udevd is being activated every few
> seconds (and its IO% is usually fairly high) so I assume this might be
> somehow related.
>

mmmh this could be an hint: how many LUNs are you exposing on the iSCSI
portal used for the hosted-engine LUN?


>
> Thanks,
> Andrea
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] high number of IOPS from ovirt hosts

[Andrea Ghelardi]

Andrea,
I don't want to point any fingers but Mark Greenall faced some issues with 
Ovirt 4.06/4.1 linked to equallogic as well.
You may want to search posts in this ML with subject " Ovirt 4.0.6 guests 'Not 
Responding' " to see if you find any similarities.
How many storages are you presenting to how many clusters?
Also, are your nodes all Hosted Engine Hosts? In my old configuration, normal 
hosts read IOPS ranged from 50 to 500 while HE ones peaked even ~3000
Cheers
AG


-Original Message-
From: Andrea Fagiani [mailto:andrea.fagi...@immobiliare.it] 
Sent: Friday, February 10, 2017 5:56 PM
To: Andrea Ghelardi ; users@ovirt.org
Subject: Re: [ovirt-users] high number of IOPS from ovirt hosts

Hi Andrea,

thanks for the insight; no I wasn't able to pinpoint the issue to a single 
host, the load seems to be evenly spread.
We have another cluster running 4.0.6 on a compellent as well and we haven't 
experienced any similar issues.

Thanks,
Andrea


On 10/02/2017 17:17, Andrea Ghelardi wrote:
> Hello Andrea,
> I saw that happening on my HE hosts connected to a COMPELLENT SC040 
> when I was running Ovirt 3.5
>
> All HE hosts were hammering the SAN with 2~4k IOPS each
>
> Upgrading to 4.0.5 solved my case.
> Are you able to check if IOPS are generated from a particular host or rather 
> spread among all nodes?
>
> Cheers
> AG
>
>
> -Original Message-
> From: users-boun...@ovirt.org [mailto:users-boun...@ovirt.org] On 
> Behalf Of Andrea Fagiani
> Sent: Friday, February 10, 2017 12:47 PM
> To: users@ovirt.org
> Subject: [ovirt-users] high number of IOPS from ovirt hosts
>
> Hi,
>
> We've got a 8-host cluster running oVirt 4.1; each host is connected via 
> iscsi to a EQL 4100 storage (2x).
> Most of the VMs use barely any IOPS, however on our storage controller we've 
> noticed a constant steam of reads, about 2000 IOPS; any idea what might cause 
> the ovirt hosts to hammer on the storage so much?
>
> Looking at iotop, it seems that systemd-udevd is being activated every few 
> seconds (and its IO% is usually fairly high) so I assume this might be 
> somehow related.
>
> Thanks,
> Andrea
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users

___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] high number of IOPS from ovirt hosts

[Andrea Fagiani]

Hi Andrea,

thanks for the insight; no I wasn't able to pinpoint the issue to a 
single host, the load seems to be evenly spread.
We have another cluster running 4.0.6 on a compellent as well and we 
haven't experienced any similar issues.


Thanks,
Andrea


On 10/02/2017 17:17, Andrea Ghelardi wrote:

Hello Andrea,
I saw that happening on my HE hosts connected to a COMPELLENT SC040 when I was 
running Ovirt 3.5

All HE hosts were hammering the SAN with 2~4k IOPS each

Upgrading to 4.0.5 solved my case.
Are you able to check if IOPS are generated from a particular host or rather 
spread among all nodes?

Cheers
AG


-Original Message-
From: users-boun...@ovirt.org [mailto:users-boun...@ovirt.org] On Behalf Of 
Andrea Fagiani
Sent: Friday, February 10, 2017 12:47 PM
To: users@ovirt.org
Subject: [ovirt-users] high number of IOPS from ovirt hosts

Hi,

We've got a 8-host cluster running oVirt 4.1; each host is connected via iscsi 
to a EQL 4100 storage (2x).
Most of the VMs use barely any IOPS, however on our storage controller we've 
noticed a constant steam of reads, about 2000 IOPS; any idea what might cause 
the ovirt hosts to hammer on the storage so much?

Looking at iotop, it seems that systemd-udevd is being activated every few 
seconds (and its IO% is usually fairly high) so I assume this might be somehow 
related.

Thanks,
Andrea
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users


___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] high number of IOPS from ovirt hosts

[Andrea Ghelardi]

Hello Andrea,
I saw that happening on my HE hosts connected to a COMPELLENT SC040 when I was 
running Ovirt 3.5

All HE hosts were hammering the SAN with 2~4k IOPS each

Upgrading to 4.0.5 solved my case.
Are you able to check if IOPS are generated from a particular host or rather 
spread among all nodes?

Cheers
AG


-Original Message-
From: users-boun...@ovirt.org [mailto:users-boun...@ovirt.org] On Behalf Of 
Andrea Fagiani
Sent: Friday, February 10, 2017 12:47 PM
To: users@ovirt.org
Subject: [ovirt-users] high number of IOPS from ovirt hosts

Hi,

We've got a 8-host cluster running oVirt 4.1; each host is connected via iscsi 
to a EQL 4100 storage (2x).
Most of the VMs use barely any IOPS, however on our storage controller we've 
noticed a constant steam of reads, about 2000 IOPS; any idea what might cause 
the ovirt hosts to hammer on the storage so much?

Looking at iotop, it seems that systemd-udevd is being activated every few 
seconds (and its IO% is usually fairly high) so I assume this might be somehow 
related.

Thanks,
Andrea
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users