On Mon, 6 Mar 2017 12:01:43 +0300
Konstantin Khomoutov <flatw...@users.sourceforge.net> wrote:
> > The application is working right now. Current work for me is to
> > found a way to reduce it's memory footprint as it will take at
> > least 1GB memory to hold only C10K idle connections.
> >
> > I know it's mainly me causing such memory usage, so I just want to
> > know if there are a better way to get around my problem without
> > changing too many of those already existing design. That's is when
> > the epoll-style idea came out.
> >
> > + epoll-styled design is seems suits better for my application by
> > nature.
>
> May be you could get away using an approach sported by certain
> networking servers (namely, the Dovecot IMAP server): they have a
> dedicated process managing idling connections, and pass socket
> descriptors back and forth between such "hibernation" process and the
> "master" process. The idea is that when a master process decides a
> particular client is idling, it discards or persists the client's
> state and passes its socket descriptor to the hibernation process;
> when the hibernation process detects a sensible incoming data on the
> socket, it sends it back to the master process which "un-hibernates"
> the client's state and resumes normal operation.
>
> This approach uses Unix-domain sockets which allow ownership transfers
> of file (and socket) descriptors between processes.
>
> Supposedly you could implement such a hibernation process yourself
> in C using libevent of libav with the master process still being
> written in Go, and see whether that would work out.
>
> Note though that the hibernation process may need to be made somewhat
> smart about the data it detects on idling connections: if your
> application-level protocol supports some sort of "keepalives" (in the
> form of "pinging" or otherwise), their support should supposedly be
> implemented directly in the hibernation process itself.
As of Go 1.7, the code which implements FD passing can be found in
{go}/src/syscall/syscall_unix_test.go file.
You might also take a look at [1] which directly uses epoll via
syscalls and basically allows you to wait on any number of sockets in
a single goroutine. This looks like you could implement such a
"hibernation process" using a dedicated goroutine: when a worker
goroutine thinks the connection it's serving starts idling, it passes
it to the "hibernation" goroutine and exits; when a hibernation
goroutine thinks there's an activity on a socket it manages, it simply
sends the socket to a goroutine which manages a pool of workers.
Unfortunately, the package only supports epoll, but at least you could
try to put up a PoC to see if it worth the effort of adding more
OS-specific select/poll backends.
1. https://godoc.org/github.com/hkwi/fdset
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
