Re: Best thread pool implementation for the base system?
On Thu, Oct 24, 2019 at 12:57 PM Ian Lepore wrote: > On Thu, 2019-10-24 at 12:46 -0600, Alan Somers wrote: > > I count 5 thread pool implementations in contrib: > > * cddl/compat/opensolaris/misc/thread_pool.c > > * contrib/apr-util/misc/apr_thread_pool.c > > * contrib/llvm/lib/Support/ThreadPool.cpp > > * contrib/openmp/runtime/src/kmp_tasking.cpp > > * contrib/ofed/opensm/complib/cl_threadpool.c > > > > However, I can't find any examples outside of contrib. I'd like to > > use one > > in /sbin/geli. Shall I roll my own (as everybody else apparently > > does), or > > is there something I'm not aware of? > > > > > > Whenever the subject of thread pools comes up at $work I pose this > question: What task is it you need to accomplish where the cost of > pthread_create/delete is significant in relation to the actual work > that will be done during the lifetime of the thread? > > Over the years, the answers have been such that we never created a > thread pool class or helpers. What we did eventually come up with was > basically an async taskqueue that had a single thread, because almost > always the answer to the question was something like "the work to be > done is really tiny and not-time-critical, it just needs to be done on > a different thread to avoid [recursion|deadlock|whatever]." > > So I'd say the first thing to do is be sure that the best solution > isn't just to pthread_create() as needed. If it turns out the cost of > pthread_create() is like 1-2% of the total work to be done, you may not > need a pool of pre-created threads. > > -- Ian > Well, the time needed by pthread_create/delete is probably not significant. But the memory consumption is. I could have up to 400 tasks, and I don't want to run all of them in parallel because all of those stacks add up. But I definitely need some amount of parallelism. FYI my interest is in speeding up "geli attach" by parallelizing the I/O to multiple disks. Currently it attaches to the disks one at a time. -Alan ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Best thread pool implementation for the base system?
On Thu, 2019-10-24 at 12:46 -0600, Alan Somers wrote: > I count 5 thread pool implementations in contrib: > * cddl/compat/opensolaris/misc/thread_pool.c > * contrib/apr-util/misc/apr_thread_pool.c > * contrib/llvm/lib/Support/ThreadPool.cpp > * contrib/openmp/runtime/src/kmp_tasking.cpp > * contrib/ofed/opensm/complib/cl_threadpool.c > > However, I can't find any examples outside of contrib. I'd like to > use one > in /sbin/geli. Shall I roll my own (as everybody else apparently > does), or > is there something I'm not aware of? > > Whenever the subject of thread pools comes up at $work I pose this question: What task is it you need to accomplish where the cost of pthread_create/delete is significant in relation to the actual work that will be done during the lifetime of the thread? Over the years, the answers have been such that we never created a thread pool class or helpers. What we did eventually come up with was basically an async taskqueue that had a single thread, because almost always the answer to the question was something like "the work to be done is really tiny and not-time-critical, it just needs to be done on a different thread to avoid [recursion|deadlock|whatever]." So I'd say the first thing to do is be sure that the best solution isn't just to pthread_create() as needed. If it turns out the cost of pthread_create() is like 1-2% of the total work to be done, you may not need a pool of pre-created threads. -- Ian ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Best thread pool implementation for the base system?
I count 5 thread pool implementations in contrib: * cddl/compat/opensolaris/misc/thread_pool.c * contrib/apr-util/misc/apr_thread_pool.c * contrib/llvm/lib/Support/ThreadPool.cpp * contrib/openmp/runtime/src/kmp_tasking.cpp * contrib/ofed/opensm/complib/cl_threadpool.c However, I can't find any examples outside of contrib. I'd like to use one in /sbin/geli. Shall I roll my own (as everybody else apparently does), or is there something I'm not aware of? -Alan ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"