[openstack-dev] [nova] nova-compute blocking main thread under heavy disk IO

2016-02-22 Thread Chris Friesen
Hi all, We've recently run into some interesting behaviour that I thought I should bring up to see if we want to do anything about it. Basically the problem seems to be that nova-compute is doing disk I/O from the main thread, and if it blocks then it can block all of nova-compute (since all

Re: [openstack-dev] [nova] nova-compute blocking main thread under heavy disk IO

2016-02-22 Thread Jay Pipes
On 02/22/2016 10:43 AM, Chris Friesen wrote: Hi all, We've recently run into some interesting behaviour that I thought I should bring up to see if we want to do anything about it. Basically the problem seems to be that nova-compute is doing disk I/O from the main thread, and if it blocks then i

Re: [openstack-dev] [nova] nova-compute blocking main thread under heavy disk IO

2016-02-22 Thread Chris Friesen
On 02/22/2016 11:17 AM, Jay Pipes wrote: On 02/22/2016 10:43 AM, Chris Friesen wrote: Hi all, We've recently run into some interesting behaviour that I thought I should bring up to see if we want to do anything about it. Basically the problem seems to be that nova-compute is doing disk I/O fro

Re: [openstack-dev] [nova] nova-compute blocking main thread under heavy disk IO

2016-02-22 Thread Sean Dague
On 02/22/2016 10:43 AM, Chris Friesen wrote: > Hi all, > > We've recently run into some interesting behaviour that I thought I > should bring up to see if we want to do anything about it. > > Basically the problem seems to be that nova-compute is doing disk I/O > from the main thread, and if it b

Re: [openstack-dev] [nova] nova-compute blocking main thread under heavy disk IO

2016-02-22 Thread Mike Bayer
On 02/22/2016 11:30 AM, Chris Friesen wrote: On 02/22/2016 11:17 AM, Jay Pipes wrote: On 02/22/2016 10:43 AM, Chris Friesen wrote: Hi all, We've recently run into some interesting behaviour that I thought I should bring up to see if we want to do anything about it. Basically the problem see

Re: [openstack-dev] [nova] nova-compute blocking main thread under heavy disk IO

2016-02-22 Thread Daniel P. Berrange
On Mon, Feb 22, 2016 at 12:07:37PM -0500, Sean Dague wrote: > On 02/22/2016 10:43 AM, Chris Friesen wrote: > > Hi all, > > > > We've recently run into some interesting behaviour that I thought I > > should bring up to see if we want to do anything about it. > > > > Basically the problem seems to

Re: [openstack-dev] [nova] nova-compute blocking main thread under heavy disk IO

2016-02-22 Thread Andrew Laski
On Mon, Feb 22, 2016, at 12:15 PM, Mike Bayer wrote: > > > On 02/22/2016 11:30 AM, Chris Friesen wrote: > > On 02/22/2016 11:17 AM, Jay Pipes wrote: > >> On 02/22/2016 10:43 AM, Chris Friesen wrote: > >>> Hi all, > >>> > >>> We've recently run into some interesting behaviour that I thought I >

Re: [openstack-dev] [nova] nova-compute blocking main thread under heavy disk IO

2016-02-22 Thread Sean Dague
On 02/22/2016 12:20 PM, Daniel P. Berrange wrote: > On Mon, Feb 22, 2016 at 12:07:37PM -0500, Sean Dague wrote: >> On 02/22/2016 10:43 AM, Chris Friesen wrote: >>> Hi all, >>> >>> We've recently run into some interesting behaviour that I thought I >>> should bring up to see if we want to do anythin

Re: [openstack-dev] [nova] nova-compute blocking main thread under heavy disk IO

2016-02-22 Thread John Garbutt
On 22 February 2016 at 17:38, Sean Dague wrote: > On 02/22/2016 12:20 PM, Daniel P. Berrange wrote: >> On Mon, Feb 22, 2016 at 12:07:37PM -0500, Sean Dague wrote: >>> On 02/22/2016 10:43 AM, Chris Friesen wrote: Hi all, We've recently run into some interesting behaviour that I thoug

Re: [openstack-dev] [nova] nova-compute blocking main thread under heavy disk IO

2016-02-22 Thread Tim Bell
On 22/02/16 19:07, "John Garbutt" wrote: >On 22 February 2016 at 17:38, Sean Dague wrote: >> On 02/22/2016 12:20 PM, Daniel P. Berrange wrote: >>> On Mon, Feb 22, 2016 at 12:07:37PM -0500, Sean Dague wrote: On 02/22/2016 10:43 AM, Chris Friesen wrote: > Hi all, > > We've recent

Re: [openstack-dev] [nova] nova-compute blocking main thread under heavy disk IO

2016-02-22 Thread Tim Bell
On 22/02/16 19:07, "John Garbutt" wrote: >On 22 February 2016 at 17:38, Sean Dague wrote: >> On 02/22/2016 12:20 PM, Daniel P. Berrange wrote: >>> On Mon, Feb 22, 2016 at 12:07:37PM -0500, Sean Dague wrote: On 02/22/2016 10:43 AM, Chris Friesen wrote: > Hi all, > > We've rec

Re: [openstack-dev] [nova] nova-compute blocking main thread under heavy disk IO

2016-02-22 Thread Chris Friesen
On 02/22/2016 11:20 AM, Daniel P. Berrange wrote: On Mon, Feb 22, 2016 at 12:07:37PM -0500, Sean Dague wrote: On 02/22/2016 10:43 AM, Chris Friesen wrote: But the fact remains that nova-compute is doing disk I/O from the main thread, and if the guests push that disk hard enough then nova-comp

Re: [openstack-dev] [nova] nova-compute blocking main thread under heavy disk IO

2016-02-25 Thread Sam Matzek
For what it's worth Glance API also has to deal with file I/O blocking all greenthreads and has CooperativeReaders/Writers that yield around the file I/O to mitigate starvation. A while ago I hit an issue with 5 concurrent VM snapshots starving Nova compute eventlets due to the excessive file IO o