Re: [openstack-dev] [mistral][oslo][messaging] Removing “blocking” executor from oslo.messaging

[Renat Akhmerov]

Hi Ken,

Awesome! IMO it works for us.

Thanks

Renat Akhmerov
@Nokia
On 20 Oct 2018, 01:19 +0700, Ken Giusti , wrote:
> Hi Renat,
> After discussing this a bit with Ben on IRC we're going to push the removal 
> off to T milestone 1.
>
> I really like Ben's idea re: adding a blocking entry to your project's 
> setup.cfg file.  We can remove the explicit check for blocking in 
> oslo.messaging so you won't get an annoying warning if you want to load 
> blocking on your own.
>
> Let me know what you think, thanks.
>
> > On Fri, Oct 19, 2018 at 12:02 AM Renat Akhmerov  
> > wrote:
> > > Hi,
> > >
> > >
> > > @Ken, I understand your considerations. I get that. I’m only asking not 
> > > to remove it *for now*. And yes, if you think it should be discouraged 
> > > from using it’s totally fine. But practically, it’s been the only 
> > > reliable option for Mistral so far that may be our fault, I have to 
> > > admit, because we weren’t able to make it work well with other executor 
> > > types but we’ll try to fix that.
> > >
> > > By the way, I was playing with different options yesterday and it seems 
> > > like that setting the executor to “threading” and the 
> > > “executor_thread_pool_size” property to 1 behaves the same way as 
> > > “blocking”. So may be that’s an option for us too, even if “blocking” is 
> > > completely removed. But I would still be in favour of having some extra 
> > > time to prove that with thorough testing.
> > >
> > > @Ben, including the executor via setup.cfg also looks OK to me. I see no 
> > > issues with this approach.
> > >
> > >
> > > Thanks
> > >
> > > Renat Akhmerov
> > > @Nokia
> > > On 18 Oct 2018, 23:35 +0700, Ben Nemec , wrote:
> > > >
> > > >
> > > > On 10/18/18 9:59 AM, Ken Giusti wrote:
> > > > > Hi Renat,
> > > > >
> > > > > The biggest issue with the blocking executor (IMHO) is that it blocks
> > > > > the protocol I/O while  RPC processing is in progress.  This increases
> > > > > the likelihood that protocol processing will not get done in a timely
> > > > > manner and things start to fail in weird ways.  These failures are
> > > > > timing related and are typically hard to reproduce or root-cause.   
> > > > > This
> > > > > isn't something we can fix as blocking is the nature of the executor.
> > > > >
> > > > > If we are to leave it in we'd really want to discourage its use.
> > > >
> > > > Since it appears the actual executor code lives in futurist, would it be
> > > > possible to remove the entrypoint for blocking from oslo.messaging and
> > > > have mistral just pull it in with their setup.cfg? Seems like they
> > > > should be able to add something like:
> > > >
> > > > oslo.messaging.executors =
> > > > blocking = futurist:SynchronousExecutor
> > > >
> > > > to their setup.cfg to keep it available to them even if we drop it from
> > > > oslo.messaging itself. That seems like a good way to strongly discourage
> > > > use of it while still making it available to projects that are really
> > > > sure they want it.
> > > >
> > > > >
> > > > > However I'm ok with leaving it available if the policy for using
> > > > > blocking is 'use at your own risk', meaning that bug reports may have 
> > > > > to
> > > > > be marked 'won't fix' if we have reason to believe that blocking is at
> > > > > fault.  That implies removing 'blocking' as the default executor value
> > > > > in the API and having applications explicitly choose it.  And we keep
> > > > > the deprecation warning.
> > > > >
> > > > > We could perhaps implement time duration checks around the executor
> > > > > callout and log a warning if the executor blocked for an extended 
> > > > > amount
> > > > > of time (extended=TBD).
> > > > >
> > > > > Other opinions so we can come to a consensus?
> > > > >
> > > > >
> > > > > On Thu, Oct 18, 2018 at 3:24 AM Renat Akhmerov 
> > > > >  > > > > > wrote:
> > > > >
> > > > > Hi Oslo Team,
> > > > >
> > > > > Can we retain “blocking” executor for now in Oslo Messaging?
> > > > >
> > > > >
> > > > > Some background..
> > > > >
> > > > > For a while we had to use Oslo Messaging with “blocking” executor in
> > > > > Mistral because of incompatibility of MySQL driver with green
> > > > > threads when choosing “eventlet” executor. Under certain conditions
> > > > > we would get deadlocks between green threads. Some time ago we
> > > > > switched to using PyMysql driver which is eventlet friendly and did
> > > > > a number of tests that showed that we could safely switch to
> > > > > “eventlet” executor (with that driver) so we introduced a new option
> > > > > in Mistral where we could choose an executor in Oslo Messaging. The
> > > > > corresponding bug is [1].
> > > > >
> > > > > The issue is that we recently found that not everything actually
> > > > > works as expected when using combination PyMysql + “eventlet”
> > > > > executor. We also tried “threading” executor and the system *seems*
> > > > > to work with it but surprisingly p

Re: [openstack-dev] [mistral][oslo][messaging] Removing “blocking” executor from oslo.messaging

[Ken Giusti]

Hi Renat,
After discussing this a bit with Ben on IRC we're going to push the removal
off to T milestone 1.

I really like Ben's idea re: adding a blocking entry to your project's
setup.cfg file.  We can remove the explicit check for blocking in
oslo.messaging so you won't get an annoying warning if you want to load
blocking on your own.

Let me know what you think, thanks.

On Fri, Oct 19, 2018 at 12:02 AM Renat Akhmerov 
wrote:

> Hi,
>
>
> @Ken, I understand your considerations. I get that. I’m only asking not to
> remove it *for now*. And yes, if you think it should be discouraged from
> using it’s totally fine. But practically, it’s been the only reliable
> option for Mistral so far that may be our fault, I have to admit, because
> we weren’t able to make it work well with other executor types but we’ll
> try to fix that.
>
> By the way, I was playing with different options yesterday and it seems
> like that setting the executor to “threading” and the
> “executor_thread_pool_size” property to 1 behaves the same way as
> “blocking”. So may be that’s an option for us too, even if “blocking” is
> completely removed. But I would still be in favour of having some extra
> time to prove that with thorough testing.
>
> @Ben, including the executor via setup.cfg also looks OK to me. I see no
> issues with this approach.
>
>
> Thanks
>
> Renat Akhmerov
> @Nokia
> On 18 Oct 2018, 23:35 +0700, Ben Nemec , wrote:
>
>
>
> On 10/18/18 9:59 AM, Ken Giusti wrote:
>
> Hi Renat,
>
> The biggest issue with the blocking executor (IMHO) is that it blocks
> the protocol I/O while  RPC processing is in progress.  This increases
> the likelihood that protocol processing will not get done in a timely
> manner and things start to fail in weird ways.  These failures are
> timing related and are typically hard to reproduce or root-cause.   This
> isn't something we can fix as blocking is the nature of the executor.
>
> If we are to leave it in we'd really want to discourage its use.
>
>
> Since it appears the actual executor code lives in futurist, would it be
> possible to remove the entrypoint for blocking from oslo.messaging and
> have mistral just pull it in with their setup.cfg? Seems like they
> should be able to add something like:
>
> oslo.messaging.executors =
> blocking = futurist:SynchronousExecutor
>
> to their setup.cfg to keep it available to them even if we drop it from
> oslo.messaging itself. That seems like a good way to strongly discourage
> use of it while still making it available to projects that are really
> sure they want it.
>
>
> However I'm ok with leaving it available if the policy for using
> blocking is 'use at your own risk', meaning that bug reports may have to
> be marked 'won't fix' if we have reason to believe that blocking is at
> fault.  That implies removing 'blocking' as the default executor value
> in the API and having applications explicitly choose it.  And we keep
> the deprecation warning.
>
> We could perhaps implement time duration checks around the executor
> callout and log a warning if the executor blocked for an extended amount
> of time (extended=TBD).
>
> Other opinions so we can come to a consensus?
>
>
> On Thu, Oct 18, 2018 at 3:24 AM Renat Akhmerov  > wrote:
>
> Hi Oslo Team,
>
> Can we retain “blocking” executor for now in Oslo Messaging?
>
>
> Some background..
>
> For a while we had to use Oslo Messaging with “blocking” executor in
> Mistral because of incompatibility of MySQL driver with green
> threads when choosing “eventlet” executor. Under certain conditions
> we would get deadlocks between green threads. Some time ago we
> switched to using PyMysql driver which is eventlet friendly and did
> a number of tests that showed that we could safely switch to
> “eventlet” executor (with that driver) so we introduced a new option
> in Mistral where we could choose an executor in Oslo Messaging. The
> corresponding bug is [1].
>
> The issue is that we recently found that not everything actually
> works as expected when using combination PyMysql + “eventlet”
> executor. We also tried “threading” executor and the system *seems*
> to work with it but surprisingly performance is much worse.
>
> Given all of that we’d like to ask Oslo Team not to remove
> “blocking” executor for now completely, if that’s possible. We have
> a strong motivation to switch to “eventlet” for other reasons
> (parallelism => better performance etc.) but seems like we need some
> time to make it smoothly.
>
>
> [1] https://bugs.launchpad.net/mistral/+bug/1696469
>
>
> Thanks
>
> Renat Akhmerov
> @Nokia
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> 
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> --
> Ken

Re: [openstack-dev] [mistral][oslo][messaging] Removing “blocking” executor from oslo.messaging

[Renat Akhmerov]

Hi,


@Ken, I understand your considerations. I get that. I’m only asking not to 
remove it *for now*. And yes, if you think it should be discouraged from using 
it’s totally fine. But practically, it’s been the only reliable option for 
Mistral so far that may be our fault, I have to admit, because we weren’t able 
to make it work well with other executor types but we’ll try to fix that.

By the way, I was playing with different options yesterday and it seems like 
that setting the executor to “threading” and the “executor_thread_pool_size” 
property to 1 behaves the same way as “blocking”. So may be that’s an option 
for us too, even if “blocking” is completely removed. But I would still be in 
favour of having some extra time to prove that with thorough testing.

@Ben, including the executor via setup.cfg also looks OK to me. I see no issues 
with this approach.


Thanks

Renat Akhmerov
@Nokia
On 18 Oct 2018, 23:35 +0700, Ben Nemec , wrote:
>
>
> On 10/18/18 9:59 AM, Ken Giusti wrote:
> > Hi Renat,
> >
> > The biggest issue with the blocking executor (IMHO) is that it blocks
> > the protocol I/O while  RPC processing is in progress.  This increases
> > the likelihood that protocol processing will not get done in a timely
> > manner and things start to fail in weird ways.  These failures are
> > timing related and are typically hard to reproduce or root-cause.   This
> > isn't something we can fix as blocking is the nature of the executor.
> >
> > If we are to leave it in we'd really want to discourage its use.
>
> Since it appears the actual executor code lives in futurist, would it be
> possible to remove the entrypoint for blocking from oslo.messaging and
> have mistral just pull it in with their setup.cfg? Seems like they
> should be able to add something like:
>
> oslo.messaging.executors =
> blocking = futurist:SynchronousExecutor
>
> to their setup.cfg to keep it available to them even if we drop it from
> oslo.messaging itself. That seems like a good way to strongly discourage
> use of it while still making it available to projects that are really
> sure they want it.
>
> >
> > However I'm ok with leaving it available if the policy for using
> > blocking is 'use at your own risk', meaning that bug reports may have to
> > be marked 'won't fix' if we have reason to believe that blocking is at
> > fault.  That implies removing 'blocking' as the default executor value
> > in the API and having applications explicitly choose it.  And we keep
> > the deprecation warning.
> >
> > We could perhaps implement time duration checks around the executor
> > callout and log a warning if the executor blocked for an extended amount
> > of time (extended=TBD).
> >
> > Other opinions so we can come to a consensus?
> >
> >
> > On Thu, Oct 18, 2018 at 3:24 AM Renat Akhmerov  > > wrote:
> >
> > Hi Oslo Team,
> >
> > Can we retain “blocking” executor for now in Oslo Messaging?
> >
> >
> > Some background..
> >
> > For a while we had to use Oslo Messaging with “blocking” executor in
> > Mistral because of incompatibility of MySQL driver with green
> > threads when choosing “eventlet” executor. Under certain conditions
> > we would get deadlocks between green threads. Some time ago we
> > switched to using PyMysql driver which is eventlet friendly and did
> > a number of tests that showed that we could safely switch to
> > “eventlet” executor (with that driver) so we introduced a new option
> > in Mistral where we could choose an executor in Oslo Messaging. The
> > corresponding bug is [1].
> >
> > The issue is that we recently found that not everything actually
> > works as expected when using combination PyMysql + “eventlet”
> > executor. We also tried “threading” executor and the system *seems*
> > to work with it but surprisingly performance is much worse.
> >
> > Given all of that we’d like to ask Oslo Team not to remove
> > “blocking” executor for now completely, if that’s possible. We have
> > a strong motivation to switch to “eventlet” for other reasons
> > (parallelism => better performance etc.) but seems like we need some
> > time to make it smoothly.
> >
> >
> > [1] https://bugs.launchpad.net/mistral/+bug/1696469
> >
> >
> > Thanks
> >
> > Renat Akhmerov
> > @Nokia
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > 
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> >
> >
> > --
> > Ken Giusti  (kgiu...@gmail.com )
> >
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailm

Re: [openstack-dev] [mistral][oslo][messaging] Removing “blocking” executor from oslo.messaging

[Ben Nemec]

On 10/18/18 9:59 AM, Ken Giusti wrote:

Hi Renat,

The biggest issue with the blocking executor (IMHO) is that it blocks 
the protocol I/O while  RPC processing is in progress.  This increases 
the likelihood that protocol processing will not get done in a timely 
manner and things start to fail in weird ways.  These failures are 
timing related and are typically hard to reproduce or root-cause.   This 
isn't something we can fix as blocking is the nature of the executor.


If we are to leave it in we'd really want to discourage its use.


Since it appears the actual executor code lives in futurist, would it be 
possible to remove the entrypoint for blocking from oslo.messaging and 
have mistral just pull it in with their setup.cfg? Seems like they 
should be able to add something like:


oslo.messaging.executors =
  blocking = futurist:SynchronousExecutor

to their setup.cfg to keep it available to them even if we drop it from 
oslo.messaging itself. That seems like a good way to strongly discourage 
use of it while still making it available to projects that are really 
sure they want it.




However I'm ok with leaving it available if the policy for using 
blocking is 'use at your own risk', meaning that bug reports may have to 
be marked 'won't fix' if we have reason to believe that blocking is at 
fault.  That implies removing 'blocking' as the default executor value 
in the API and having applications explicitly choose it.  And we keep 
the deprecation warning.


We could perhaps implement time duration checks around the executor 
callout and log a warning if the executor blocked for an extended amount 
of time (extended=TBD).


Other opinions so we can come to a consensus?


On Thu, Oct 18, 2018 at 3:24 AM Renat Akhmerov > wrote:


Hi Oslo Team,

Can we retain “blocking” executor for now in Oslo Messaging?


Some background..

For a while we had to use Oslo Messaging with “blocking” executor in
Mistral because of incompatibility of MySQL driver with green
threads when choosing “eventlet” executor. Under certain conditions
we would get deadlocks between green threads. Some time ago we
switched to using PyMysql driver which is eventlet friendly and did
a number of tests that showed that we could safely switch to
“eventlet” executor (with that driver) so we introduced a new option
in Mistral where we could choose an executor in Oslo Messaging. The
corresponding bug is [1].

The issue is that we recently found that not everything actually
works as expected when using combination PyMysql + “eventlet”
executor. We also tried “threading” executor and the system *seems*
to work with it but surprisingly performance is much worse.

Given all of that we’d like to ask Oslo Team not to remove
“blocking” executor for now completely, if that’s possible. We have
a strong motivation to switch to “eventlet” for other reasons
(parallelism => better performance etc.) but seems like we need some
time to make it smoothly.


[1] https://bugs.launchpad.net/mistral/+bug/1696469


Thanks

Renat Akhmerov
@Nokia
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



--
Ken Giusti  (kgiu...@gmail.com )

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [mistral][oslo][messaging] Removing “blocking” executor from oslo.messaging

[Ken Giusti]

Hi Renat,

The biggest issue with the blocking executor (IMHO) is that it blocks the
protocol I/O while  RPC processing is in progress.  This increases the
likelihood that protocol processing will not get done in a timely manner
and things start to fail in weird ways.  These failures are timing related
and are typically hard to reproduce or root-cause.   This isn't something
we can fix as blocking is the nature of the executor.

If we are to leave it in we'd really want to discourage its use.

However I'm ok with leaving it available if the policy for using blocking
is 'use at your own risk', meaning that bug reports may have to be marked
'won't fix' if we have reason to believe that blocking is at fault.  That
implies removing 'blocking' as the default executor value in the API and
having applications explicitly choose it.  And we keep the deprecation
warning.

We could perhaps implement time duration checks around the executor callout
and log a warning if the executor blocked for an extended amount of time
(extended=TBD).

Other opinions so we can come to a consensus?


On Thu, Oct 18, 2018 at 3:24 AM Renat Akhmerov 
wrote:

> Hi Oslo Team,
>
> Can we retain “blocking” executor for now in Oslo Messaging?
>
>
> Some background..
>
> For a while we had to use Oslo Messaging with “blocking” executor in
> Mistral because of incompatibility of MySQL driver with green threads when
> choosing “eventlet” executor. Under certain conditions we would get
> deadlocks between green threads. Some time ago we switched to using PyMysql
> driver which is eventlet friendly and did a number of tests that showed
> that we could safely switch to “eventlet” executor (with that driver) so we
> introduced a new option in Mistral where we could choose an executor in
> Oslo Messaging. The corresponding bug is [1].
>
> The issue is that we recently found that not everything actually works as
> expected when using combination PyMysql + “eventlet” executor. We also
> tried “threading” executor and the system *seems* to work with it but
> surprisingly performance is much worse.
>
> Given all of that we’d like to ask Oslo Team not to remove “blocking”
> executor for now completely, if that’s possible. We have a strong
> motivation to switch to “eventlet” for other reasons (parallelism => better
> performance etc.) but seems like we need some time to make it smoothly.
>
>
> [1] https://bugs.launchpad.net/mistral/+bug/1696469
>
>
> Thanks
>
> Renat Akhmerov
> @Nokia
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>


-- 
Ken Giusti  (kgiu...@gmail.com)
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [mistral][oslo][messaging] Removing “blocking” executor from oslo.messaging

[Renat Akhmerov]

Hi Oslo Team,

Can we retain “blocking” executor for now in Oslo Messaging?


Some background..

For a while we had to use Oslo Messaging with “blocking” executor in Mistral 
because of incompatibility of MySQL driver with green threads when choosing 
“eventlet” executor. Under certain conditions we would get deadlocks between 
green threads. Some time ago we switched to using PyMysql driver which is 
eventlet friendly and did a number of tests that showed that we could safely 
switch to “eventlet” executor (with that driver) so we introduced a new option 
in Mistral where we could choose an executor in Oslo Messaging. The 
corresponding bug is [1].

The issue is that we recently found that not everything actually works as 
expected when using combination PyMysql + “eventlet” executor. We also tried 
“threading” executor and the system seems to work with it but surprisingly 
performance is much worse.

Given all of that we’d like to ask Oslo Team not to remove “blocking” executor 
for now completely, if that’s possible. We have a strong motivation to switch 
to “eventlet” for other reasons (parallelism => better performance etc.) but 
seems like we need some time to make it smoothly.


[1] https://bugs.launchpad.net/mistral/+bug/1696469


Thanks

Renat Akhmerov
@Nokia
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev