Re: SO_REUSEPORT: strange kernel balancer behaviour

2013-07-23 Thread Adrian Chadd
On 23 July 2013 10:54, trafdev wrote: > Yes, and if you kill this first thread - second thread will start to receive > connections and so on. > That's why I've used one processes-shared acceptor socket which behaves > better (load is balancing > between processes but equality of distribution is fa

Re: SO_REUSEPORT: strange kernel balancer behaviour

2013-07-23 Thread trafdev
Yes, and if you kill this first thread - second thread will start to receive connections and so on. That's why I've used one processes-shared acceptor socket which behaves better (load is balancing between processes but equality of distribution is far from ideal). Btw as per https://lwn.net/Art

Re: SO_REUSEPORT: strange kernel balancer behaviour

2013-07-23 Thread Adrian Chadd
Answering my own email: SO_REUSEPORT on FreeBSD doesn't load balance incoming connections. Test case: * 8 threads * each creates a TCP socket, listening on port 1667, with SO_REUSEPORT * only the first thread ever sees incoming requests. I think this load distribution feature is useful to imple

Re: SO_REUSEPORT: strange kernel balancer behaviour

2013-07-23 Thread Adrian Chadd
On 23 July 2013 00:09, trafdev wrote: > It's like shared acceptor FD and N processes: [snip] looks like mine, but I use threads. > Accept conn callback is called in N processes on each connection, only one > wins, > others exit by errno == EAGAIN case. Overhead is almost zero. > Problem is that

Re: SO_REUSEPORT: strange kernel balancer behaviour

2013-07-23 Thread trafdev
It's like shared acceptor FD and N processes: Listening proc: bool HttpServer::Listen(unsigned short port, uint listen_backlog) { LOG4CXX_TRACE(kLogger, "Listen"); if ((sockd_acceptor_ = socket(AF_INET, SOCK_STREAM, 0)) == -1) { LOG4CXX_ERROR_ERRNO(kLogger, "socket"); return

Re: SO_REUSEPORT: strange kernel balancer behaviour

2013-07-22 Thread Adrian Chadd
eg: * one process, one listen thread, multiple dispatch threads? * n processes, one listen FD per process, all listening on the same IP:port? * one process, each thread listening on the same IP:port? * something else? Thanks, -adrian ___ freebsd-net@

Re: SO_REUSEPORT: strange kernel balancer behaviour

2013-07-22 Thread Adrian Chadd
Sweet, have any code you can share that can elicit this? I'm writing a network / disk application layer right now so i can model/test this stuff out. I'd love to see some pointers/example code. Thanks! -adrian ___ freebsd-net@freebsd.org mailing list

Re: SO_REUSEPORT: strange kernel balancer behaviour

2013-07-22 Thread trafdev
yep, FreeBSD 9.1-RELEASE-p3 On Mon Jul 22 21:05:55 2013, Adrian Chadd wrote: On 22 July 2013 14:26, trafdev wrote: Actually overhead is almost zero, the real problem is in non-equivalent load distribution between processes. As https://lwn.net/Articles/542629/ mentions - "At Google, they have s

Re: SO_REUSEPORT: strange kernel balancer behaviour

2013-07-22 Thread Adrian Chadd
On 22 July 2013 14:26, trafdev wrote: > Actually overhead is almost zero, the real problem is in non-equivalent load > distribution between processes. > As https://lwn.net/Articles/542629/ mentions - > "At Google, they have seen a factor-of-three difference between the thread > accepting the most

Re: SO_REUSEPORT: strange kernel balancer behaviour

2013-07-22 Thread trafdev
Actually overhead is almost zero, the real problem is in non-equivalent load distribution between processes. As https://lwn.net/Articles/542629/ mentions - "At Google, they have seen a factor-of-three difference between the thread accepting the most connections and the thread accepting the fewe

Re: SO_REUSEPORT: strange kernel balancer behaviour

2013-07-22 Thread John-Mark Gurney
trafdev wrote this message on Mon, Jul 15, 2013 at 13:04 -0700: > Yep I think it's wasting of resources, poll manager should somehow be > configured to update only one process/thread. > Anyone know how to do that? This isn't currently possible w/o a shared kqueue, since the event is level trigger

Re: SO_REUSEPORT: strange kernel balancer behaviour

2013-07-15 Thread Adrian Chadd
On 15 July 2013 13:04, trafdev wrote: > Yep I think it's wasting of resources, poll manager should somehow be > configured to update only one process/thread. > Anyone know how to do that? > Thanks. Well, the problem here is deciding which thread to throw the request at. If the threads are equall

Re: SO_REUSEPORT: strange kernel balancer behaviour

2013-07-15 Thread trafdev
Yep I think it's wasting of resources, poll manager should somehow be configured to update only one process/thread. Anyone know how to do that? Thanks. On Mon Jul 15 12:53:55 2013, Adrian Chadd wrote: i've noticed this when doing this stuff in a threaded program with each thread listening on th

Re: SO_REUSEPORT: strange kernel balancer behaviour

2013-07-15 Thread Adrian Chadd
i've noticed this when doing this stuff in a threaded program with each thread listening on the same port. All threads wake up on each accepted connection, one thread wins and the other threads get EAGAIN. -adrian On 15 July 2013 12:31, trafdev wrote: > Thanks for reply. > > This approach pro

Re: SO_REUSEPORT: strange kernel balancer behaviour

2013-07-15 Thread trafdev
Thanks for reply. This approach produces lot of "resource temporary unavailable" (eagain) on accept-ing connections in N-1 processes. Is this possible to avoid this by e.g. tweaking kqueue? On Sun Jul 14 19:37:59 2013, Sepherosa Ziehau wrote: On Sat, Jul 13, 2013 at 1:16 PM, trafdev wrote:

Re: SO_REUSEPORT: strange kernel balancer behaviour

2013-07-14 Thread Sepherosa Ziehau
On Sat, Jul 13, 2013 at 1:16 PM, trafdev wrote: > Hello. > > Could someone help with following problem of SO_REUSEPORT. The most portable "load balance" between processes listening on the same TCP addr/port probably is: s=socket(); bind(s); listen(s); /* various socketopt and fcntl as you needed

SO_REUSEPORT: strange kernel balancer behaviour

2013-07-12 Thread trafdev
Hello. Could someone help with following problem of SO_REUSEPORT. Created server: int sockd_acceptor_; ... if ((sockd_acceptor_ = socket(PF_INET, SOCK_STREAM, 0)) == -1) { LOG4CXX_ERROR_ERRNO(kLogger, "socket"); return false; } struct sockaddr_in sa_in; memset(&sa_i