Re: Driving a thread pool from libev

2015-02-17 Thread Marc Lehmann
On Sun, Feb 15, 2015 at 03:55:14PM +0100, Rick van Rein wrote: > I overlooked that pthread_yield() is a technique from single-core CPU days. Whats worse, it's only really defined for realtime threads. > I fear I cannot continue to use one thread per TLS session; this works fine > for > the sho

Re: Driving a thread pool from libev

2015-02-15 Thread Rick van Rein
Hello Marc, To summarise, I tried to save a memcpy() but it inevitably backfires with more costly user/kernel interactions. Thanks for clarifying that. I overlooked that pthread_yield() is a technique from single-core CPU days. My mentioning of spinlocks was based the same oversight. Again, tha

Re: Fwd: Driving a thread pool from libev

2015-02-14 Thread Marc Lehmann
On Sat, Feb 14, 2015 at 03:47:01PM +0100, Rick van Rein wrote: > The ideal is to run this pthread_yield() once per thread, just before > returning to poll() — calling it from every callback function is And just before anybody reads this mail out of context - pthread_yield will *not* give other t

Re: Driving a thread pool from libev

2015-02-14 Thread Marc Lehmann
On Sun, Feb 15, 2015 at 01:26:09AM +0100, Marc Lehmann wrote: > register an ev_check watcher that checks your async flag (you likely want ev_prepare, even. -- The choice of a Deliantra, the free code+content MORPG -==- _GNU_ http://www.deliantra

Re: Driving a thread pool from libev

2015-02-14 Thread Marc Lehmann
On Sat, Feb 14, 2015 at 01:51:32PM +0100, Rick van Rein wrote: > * Then the event loop could yield() before it relaunches poll(), and give > the worker threads a chance to perform their read(). Most systems today have multiple cpus (or cores). What you describe only really makes sense on a uni

Re: Driving a thread pool from libev

2015-02-14 Thread Marc Lehmann
On Sat, Feb 14, 2015 at 10:10:40AM +0100, Rick van Rein wrote: > I am trying to drive a thread pool from libev, but the design appears to be > impossible; or has anyone no this list done this before? Many people have, apparently. Event loops and thread pools are mostly orthogonal to each other

Fwd: Driving a thread pool from libev

2015-02-14 Thread Rick van Rein
Hi, > Excellent solution, it indeed avoid memcpy(). Thanks :-D Conceptually the most logical model would be if libev would run multiple callbacks, each running in workers from the thread pool; and if it would use this mechanism to avoid starting multiple such callbacks on a single watcher. >

Re: Driving a thread pool from libev

2015-02-14 Thread echo
Excellent solution, it indeed avoid memcpy(). When you say "yeild()", it's some notification like ev_async_send(), right? 在 2015年02月14日 20:51, Rick van Rein 写道: Hi Echo, But in my opinion, the work in thread usually cost much more cpu time than read/write/malloc. Thanks for sharing that. I

Re: Driving a thread pool from libev

2015-02-14 Thread Rick van Rein
Hi Echo, > But in my opinion, the work in thread usually cost much more cpu time than > read/write/malloc. Thanks for sharing that. I am indeed wondering if my concerns aren’t a bit academic. (I am in the desing phase, haven’t coded or measured, and since the app is to be portable it would b

Re: Driving a thread pool from libev

2015-02-14 Thread echo
read/write a pipe is slow, but read/write circle-buffer is fast because its pure user space code and lockless. memcpy() can be avoid by write the pointer into circle-buffer instead of write the data itself. if you care about malloc() cost, you can use pre-allocated memory which are allocated dur

Re: Driving a thread pool from libev

2015-02-14 Thread Rick van Rein
Hi Echo, > 3) When main thread got a new request, it find an idle worker and write the > request data to the worker's circle-buffer and notify the worker's ev_async. This is what I’m hoping to avoid — slowing down the master with the read() of the available data, and causing a need for an extra

Re: Driving a thread pool from libev

2015-02-14 Thread echo
I think that my application is doing a similar work like yours. In my application, user requests are received from main thread, then the requests are sent to data thread workers which will do some database queries, and then results are sent back to main thread in which reponses are made to use

Driving a thread pool from libev

2015-02-14 Thread Rick van Rein
Hello, I am trying to drive a thread pool from libev, but the design appears to be impossible; or has anyone no this list done this before? The application: a TLS Pool [1], with worker threads that read(), then encrypt or decrypt, then write(). The workers should be part of a thread pool. And