Rob Lanphier wrote: > Since it's single threaded, it's handling each of the > configured logging destinations before reading the next packet. We're > not CPU-bound at this point. The existing solution seems to start > flaking out at 40% CPU with a complicated configuration, and is > humming along at 20% with the current simplified config. The problem > is that we're blocking while we fire up awk or whatever on the logging > side, and overflowing the socket buffer.
It is only launched once, then reused. So the problem is that they don't eat the pipe data fast enough, so the pipe accumulates 64Kb and finally blocks. > A solution that Tim and others are kicking around is reworking the > listener in one or more of the following ways: > 1. Move to some non-blocking networking library (e.g. Boost asio, libevent) > 2. Go multi-threaded > > The solutions we're talking about seem to solve the socket buffer > problem, but it sounds like we may also need to get some clearer > requirements on any new functionality that's needed. It sounds like > we'll be able to get some more mileage out of the existing solution > with some of the reworking described above. It's not entirely clear > yet if this buys us enough capacity+capability for the increased > requirements. I'll check in with Tomasz and others working on > fundraiser stuff to find out more. > > Rob Going multithread is really easy for a socket listener. However, not so much in the LogProcessors. If they are shared accross threads, you may end up with all threads blocked in the fwrite and if they aren't shared, the files may easily corrupt (depends on what you are exactly doing with them). Since the problem is that the socket buffer fills, it surprised me that the server didn't increase SO_RCVBUF. That's not a solution but should help (already set in /proc/sys/net/core/rmem_default ?). The real issue is: what are you placing on your pipes that are so slow to read from them? Optimizing those scripts could be a simpler solution. Wouldn't be hard to make the pipe writes non-blocking, properly blaming the slow pipes that couldn't be written _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
