Re: [vdsm] thread pool implementation
- Original Message - > From: "ybronhei" > To: "Francesco Romani" , "Saggi Mizrahi" > > Cc: "vdsm-devel" > Sent: Wednesday, March 26, 2014 9:53:52 AM > Subject: Re: [vdsm] thread pool implementation > > On 03/25/2014 03:15 PM, Francesco Romani wrote: > > > > - Original Message - > >> From: "Saggi Mizrahi" > >> To: "Francesco Romani" > >> Cc: "vdsm-devel" , "Yaniv Bronheim" > >> > >> Sent: Tuesday, March 25, 2014 1:45:22 PM > >> Subject: Re: thread pool implementation > >> > >> The thing that worries me the most is stuck threads. > >> I hate them! > >> > >> Could we move to multiple libvirt connections scheme? > >> Where if a call takes too long we just close the connection. > >> I know that the call is still running in libvirt but then it's > >> their problem and not my problem. That way the thread pool > >> doesn't need to handle this use case making it much simpler. > >> > >> Because apart from the problem of libvirt calls getting stuck > >> we just need a run of the mill threadpool solution. > > > > It is an interesting point. > > I'll investigate the multiple libvirt connection Idea. > > > we support multiple libvirt connections with libvirtconnection.py. > however, vdsm uses only one instance during its run. might be that this > was the initial thought of the libvirtconnection get function. but > still, if the request gets stuck, how multiply connections help us? we > still need threads to avoid freezing other flows. afaiu you plan to > cover each libvirt call in a thread from the pool? we can still use one > connection for that. if each thread will establish new session to > libvirt, it might cause very long delay (depends on the load) Maybe you can use something similar to my netlink socket pool. The sockets are created lazily and if some op fails the socket is not returned to the pool and it will leave space for a new socket creation. > > > Bests, > > > > > -- > Yaniv Bronhaim. > ___ > vdsm-devel mailing list > vdsm-devel@lists.fedorahosted.org > https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel > ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] thread pool implementation
On Mar 26, 2014, at 09:53 , ybronhei wrote: > On 03/25/2014 03:15 PM, Francesco Romani wrote: >> >> - Original Message - >>> From: "Saggi Mizrahi" >>> To: "Francesco Romani" >>> Cc: "vdsm-devel" , "Yaniv Bronheim" >>> >>> Sent: Tuesday, March 25, 2014 1:45:22 PM >>> Subject: Re: thread pool implementation >>> >>> The thing that worries me the most is stuck threads. >>> I hate them! >>> >>> Could we move to multiple libvirt connections scheme? >>> Where if a call takes too long we just close the connection. >>> I know that the call is still running in libvirt but then it's >>> their problem and not my problem. That way the thread pool >>> doesn't need to handle this use case making it much simpler. >>> >>> Because apart from the problem of libvirt calls getting stuck >>> we just need a run of the mill threadpool solution. >> >> It is an interesting point. >> I'll investigate the multiple libvirt connection Idea. >> > we support multiple libvirt connections with libvirtconnection.py. however, > vdsm uses only one instance during its run. might be that this was the > initial thought of the libvirtconnection get function. but still, if the > request gets stuck, how multiply connections help us? we still need threads > to avoid freezing other flows. afaiu you plan to cover each libvirt call in a > thread from the pool? we can still use one connection for that. if each > thread will establish new session to libvirt, it might cause very long delay > (depends on the load) another thing to consider is the number of worker threads on the libvirt side (set to 20 by default now IIRC, some are used internally) > >> Bests, >> > > > -- > Yaniv Bronhaim. > ___ > vdsm-devel mailing list > vdsm-devel@lists.fedorahosted.org > https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] thread pool implementation
On 03/25/2014 03:15 PM, Francesco Romani wrote: - Original Message - From: "Saggi Mizrahi" To: "Francesco Romani" Cc: "vdsm-devel" , "Yaniv Bronheim" Sent: Tuesday, March 25, 2014 1:45:22 PM Subject: Re: thread pool implementation The thing that worries me the most is stuck threads. I hate them! Could we move to multiple libvirt connections scheme? Where if a call takes too long we just close the connection. I know that the call is still running in libvirt but then it's their problem and not my problem. That way the thread pool doesn't need to handle this use case making it much simpler. Because apart from the problem of libvirt calls getting stuck we just need a run of the mill threadpool solution. It is an interesting point. I'll investigate the multiple libvirt connection Idea. we support multiple libvirt connections with libvirtconnection.py. however, vdsm uses only one instance during its run. might be that this was the initial thought of the libvirtconnection get function. but still, if the request gets stuck, how multiply connections help us? we still need threads to avoid freezing other flows. afaiu you plan to cover each libvirt call in a thread from the pool? we can still use one connection for that. if each thread will establish new session to libvirt, it might cause very long delay (depends on the load) Bests, -- Yaniv Bronhaim. ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] thread pool implementation
- Original Message - > From: "Saggi Mizrahi" > To: "Francesco Romani" > Cc: "vdsm-devel" , "Yaniv Bronheim" > > Sent: Tuesday, March 25, 2014 1:45:22 PM > Subject: Re: thread pool implementation > > The thing that worries me the most is stuck threads. > I hate them! > > Could we move to multiple libvirt connections scheme? > Where if a call takes too long we just close the connection. > I know that the call is still running in libvirt but then it's > their problem and not my problem. That way the thread pool > doesn't need to handle this use case making it much simpler. > > Because apart from the problem of libvirt calls getting stuck > we just need a run of the mill threadpool solution. It is an interesting point. I'll investigate the multiple libvirt connection Idea. Bests, -- Francesco Romani RedHat Engineering Virtualization R & D Phone: 8261328 IRC: fromani ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] thread pool implementation
- Original Message - > From: "Francesco Romani" > To: "vdsm-devel" > Sent: Tuesday, March 25, 2014 12:55:36 PM > Subject: [vdsm] thread pool implementation > > Hello, > > in order to reduce the number of sampling threads, we'd like to move from a > one thread per VM > to a thread pool. > > The strongest requirement we have is to be able to detect if a worker pool is > not responding, > and if so to detach it from the pool and to kill it as soon as possible; then > a new worker should > be made available. > > This is because in sampling we are going to call libvirt and libvirt calls > can block or, even worse, > get stuck (I'm looking at you virDomainGetBlockInfo - > http://libvirt.org/html/libvirt-libvirt.html#virDomainGetBlockInfo ) > > So, we need a thread pool implementation :) > What is the best way forward? I see a few options: > > * we have a thread pool already in storage. Should we move it outside storage > to lib/ and extend it? > * there is a thread pool hidden inside the multiprocessing module! > (see > > http://docs.python.org/2/library/multiprocessing.html#module-multiprocessing.dummy) > should we switch to this, at least for sampling? > * Python 3.2+ has concurrent.futures which has a nice API and can use a > thread pool executor. > See > > http://docs.python.org/3.3/library/concurrent.futures.html#module-concurrent.futures > There is a backport for python 2.6/2.7 also: > https://pypi.python.org/pypi/futures > Maybe this is the most forward compatible way? When I saw the subject of the email I immediately thought to suggest this. My vote is for this ;-) > * Add an(other) thread pool? > > I don't really have any preference granted the requirement above is > satisfied. > > Thoughts? Especially Infra people's feedback would be appreciated. > > -- > Francesco Romani > RedHat Engineering Virtualization R & D > Phone: 8261328 > IRC: fromani > ___ > vdsm-devel mailing list > vdsm-devel@lists.fedorahosted.org > https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel > ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] thread pool implementation
The thing that worries me the most is stuck threads. I hate them! Could we move to multiple libvirt connections scheme? Where if a call takes too long we just close the connection. I know that the call is still running in libvirt but then it's their problem and not my problem. That way the thread pool doesn't need to handle this use case making it much simpler. Because apart from the problem of libvirt calls getting stuck we just need a run of the mill threadpool solution. - Original Message - > From: "Francesco Romani" > To: "vdsm-devel" > Cc: "Saggi Mizrahi" , "Yaniv Bronheim" > > Sent: Tuesday, March 25, 2014 1:55:36 PM > Subject: thread pool implementation > > Hello, > > in order to reduce the number of sampling threads, we'd like to move from a > one thread per VM > to a thread pool. > > The strongest requirement we have is to be able to detect if a worker pool is > not responding, > and if so to detach it from the pool and to kill it as soon as possible; then > a new worker should > be made available. > > This is because in sampling we are going to call libvirt and libvirt calls > can block or, even worse, > get stuck (I'm looking at you virDomainGetBlockInfo - > http://libvirt.org/html/libvirt-libvirt.html#virDomainGetBlockInfo ) > > So, we need a thread pool implementation :) > What is the best way forward? I see a few options: > > * we have a thread pool already in storage. Should we move it outside storage > to lib/ and extend it? > * there is a thread pool hidden inside the multiprocessing module! > (see > > http://docs.python.org/2/library/multiprocessing.html#module-multiprocessing.dummy) > should we switch to this, at least for sampling? > * Python 3.2+ has concurrent.futures which has a nice API and can use a > thread pool executor. > See > > http://docs.python.org/3.3/library/concurrent.futures.html#module-concurrent.futures > There is a backport for python 2.6/2.7 also: > https://pypi.python.org/pypi/futures > Maybe this is the most forward compatible way? > * Add an(other) thread pool? > > I don't really have any preference granted the requirement above is > satisfied. > > Thoughts? Especially Infra people's feedback would be appreciated. > > -- > Francesco Romani > RedHat Engineering Virtualization R & D > Phone: 8261328 > IRC: fromani > ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
[vdsm] thread pool implementation
Hello, in order to reduce the number of sampling threads, we'd like to move from a one thread per VM to a thread pool. The strongest requirement we have is to be able to detect if a worker pool is not responding, and if so to detach it from the pool and to kill it as soon as possible; then a new worker should be made available. This is because in sampling we are going to call libvirt and libvirt calls can block or, even worse, get stuck (I'm looking at you virDomainGetBlockInfo - http://libvirt.org/html/libvirt-libvirt.html#virDomainGetBlockInfo ) So, we need a thread pool implementation :) What is the best way forward? I see a few options: * we have a thread pool already in storage. Should we move it outside storage to lib/ and extend it? * there is a thread pool hidden inside the multiprocessing module! (see http://docs.python.org/2/library/multiprocessing.html#module-multiprocessing.dummy) should we switch to this, at least for sampling? * Python 3.2+ has concurrent.futures which has a nice API and can use a thread pool executor. See http://docs.python.org/3.3/library/concurrent.futures.html#module-concurrent.futures There is a backport for python 2.6/2.7 also: https://pypi.python.org/pypi/futures Maybe this is the most forward compatible way? * Add an(other) thread pool? I don't really have any preference granted the requirement above is satisfied. Thoughts? Especially Infra people's feedback would be appreciated. -- Francesco Romani RedHat Engineering Virtualization R & D Phone: 8261328 IRC: fromani ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
