The thing that worries me the most is stuck threads. I hate them! Could we move to multiple libvirt connections scheme? Where if a call takes too long we just close the connection. I know that the call is still running in libvirt but then it's their problem and not my problem. That way the thread pool doesn't need to handle this use case making it much simpler.
Because apart from the problem of libvirt calls getting stuck we just need a run of the mill threadpool solution. ----- Original Message ----- > From: "Francesco Romani" <from...@redhat.com> > To: "vdsm-devel" <vdsm-devel@lists.fedorahosted.org> > Cc: "Saggi Mizrahi" <smizr...@redhat.com>, "Yaniv Bronheim" > <ybron...@redhat.com> > Sent: Tuesday, March 25, 2014 1:55:36 PM > Subject: thread pool implementation > > Hello, > > in order to reduce the number of sampling threads, we'd like to move from a > one thread per VM > to a thread pool. > > The strongest requirement we have is to be able to detect if a worker pool is > not responding, > and if so to detach it from the pool and to kill it as soon as possible; then > a new worker should > be made available. > > This is because in sampling we are going to call libvirt and libvirt calls > can block or, even worse, > get stuck (I'm looking at you virDomainGetBlockInfo - > http://libvirt.org/html/libvirt-libvirt.html#virDomainGetBlockInfo ) > > So, we need a thread pool implementation :) > What is the best way forward? I see a few options: > > * we have a thread pool already in storage. Should we move it outside storage > to lib/ and extend it? > * there is a thread pool hidden inside the multiprocessing module! > (see > > http://docs.python.org/2/library/multiprocessing.html#module-multiprocessing.dummy) > should we switch to this, at least for sampling? > * Python 3.2+ has concurrent.futures which has a nice API and can use a > thread pool executor. > See > > http://docs.python.org/3.3/library/concurrent.futures.html#module-concurrent.futures > There is a backport for python 2.6/2.7 also: > https://pypi.python.org/pypi/futures > Maybe this is the most forward compatible way? > * Add an(other) thread pool? > > I don't really have any preference granted the requirement above is > satisfied. > > Thoughts? Especially Infra people's feedback would be appreciated. > > -- > Francesco Romani > RedHat Engineering Virtualization R & D > Phone: 8261328 > IRC: fromani > _______________________________________________ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
