Hi, I've got an simple processing system using the 0.22 C++ broker, all on one box, where an external system posts messages to the default headers exchange, and an ingest process receives them using a ring queue, transforms them and outputs to a different headers exchange. Various other processes pick messages of interest off that exchange using ring queues. Recently however the system has been stalling - I'm still receiving lots of data from the other system, but the ingest process suddenly goes to <5% CPU usage and its queue fills up and messages start getting discarded from the ring, the follow on processes go to practically 0% CPU and qpidd hovers around 95-120% CPU (normally its ~75%) and the rest of the system pretty much goes idle (no swapping, there is free memory)
I attached to the ingest process with gdb and it was stuck in send (waitForCapacity/waitForCompletionImpl) - I notice this can block. However given the rest of the system is idle when this problem occurs I can't understand why this would happen. I added a SIGALARM handler around send with a timeout of 30s and the process did sometimes get killed. Looking at qpid-tool it does seem to still be processing messages, just extremely slowly. My other observation is from netstat, the Send-Q of qpidd to the ingest process is 16363, and the Recv-Q and Send-Q of the ingest process are both 0. Any ideas on what might be happening are very welcome! Cheers, Jimmy --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
