Hi! > Going multithread is really easy for a socket listener.
Really? :) > However, not so > much in the LogProcessors. If they are shared accross threads, you may > end up with all threads blocked in the fwrite and if they aren't shared, > the files may easily corrupt (depends on what you are exactly doing with > them). I don't really understand what you say ;-) Do you mean lost data as 'corrupt'? > Since the problem is that the socket buffer fills, it surprised me that > the server didn't increase SO_RCVBUF. That's not a solution but should > help (already set in /proc/sys/net/core/rmem_default ?). It is long term CPU saturation issue - mux process isn't fast enough to handle 16 output streams. Do note, there're quite a few events a second :) > The real issue is: what are you placing on your pipes that are so slow > to read from them? > Optimizing those scripts could be a simpler solution. No, those scripts are not the bottleneck, there's plenty of CPU available, and they are not blocking (for too long, everything is blocking for a certain amount of time ;-) > Wouldn't be hard to make the pipe writes non-blocking, properly blaming > the slow pipes that couldn't be written There are no slow pipes. Bottleneck is udp2log step. Domas _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
