Dmitry Tantsur wrote:
Jim,
I'm redirecting your question to oslo folks, as I'm afraid my answer can
be wrong.
On 07/23/2015 01:55 PM, Jim Rollenhagen wrote:
On Wed, Jul 22, 2015 at 02:40:47PM +0200, Dmitry Tantsur wrote:
Hi all!
Currently _spawn_worker in the conductor manager raises
NoFreeConductorWorker if pool is already full. That's not very user
friendly
(potential source of retries in client) and does not map well on common
async worker patterns.
My understanding is that it was done to prevent the conductor thread
from
waiting on pool to become free. If this is true, we no longer need it
after
switch to Futurist, as Futurist maintains internal queue for its green
executor, just like thread and process executors in stdlib do.
Instead of
blocking the conductor the request will be queued, and a user won't
have to
retry vague (and rare!) HTTP 503 error.
WDYT about me dropping this exception with move to Futurist?
I kind of like this, but with my operator hat on this is a bit scary.
Does Futurist just queue all requests indefinitely? Is it configurable?
Am I able to get any insight into the current state of that queue?
I believe answer is no, and the reason IIUC is that Futurist executors
are modeled after stdlib executors, but I may be wrong.
So this is correct, currently executors will queue things up, and that
queue may get very large. In futurist we can work on making this better,
although to do it correctly we really need
https://bugs.python.org/issue22737 and that needs upstream python
adjustments to make it possible.
Without that https://bugs.python.org/issue22737 being implemented its
not to hard to limit the work queue yourself, but it will have to be
something extra and require tracking of your own.
For example:
dispatched = set()
def on_done(fut):
dispatched.discard(fut)
executor = futurist.GreenThreadPoolExecutor()
...
if len(dispatched) < MAX_DISPATCH:
raise IamToOverWorkedExeception(...)
fut = executor.submit(some_work)
dispatched.add(fut)
fut.add_done_callback(on_done)
The above will limit how much work is done at the same time
(https://bugs.python.org/issue22737 makes it work more like java, which
has executor rejection policies,
http://download.java.net/jdk7/archive/b123/docs/api/java/util/concurrent/RejectedExecutionHandler.html)
but you can limit this yourself pretty easily... Making issue22737
happen would be great; I just haven't had enough time to pull that off...
Just indefinitely queueing up everything seems like it could end with a
system that's backlogged to death, with no way to determine if that's
actually the problem or not.
As for metrics, one of the additions of the futurist executor subclasses
was to bolt-on the following being gathered,
https://github.com/openstack/futurist/blob/master/futurist/_futures.py#L387
(at executor property '.statistics') so hopefully that can help u learn
about what your executors are doing as well..
// jim
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
