G'day Darren et al, We had several panics under load testing a week or two ago, using Solaris 9 SPARC with IP Filter 4.1.23 and pfil 2.1.13, with an enlarged state table (7-10x normal, prime size & max).
Initially thought table too large or not correctly enlarged, but now doubt that was the problem. Didn't see the problem with the standard size, but then the app would get heaps errors, wouldn't run for long enough/high enough load to demonstrate it. Having seen a few other similar reports I had a further look. Potential patch at the end (no chance to test as yet). For those who may have a similar problem, our console panic and reboot log (long and wide, sorry) is attached as the console_ap2a.log file. In fact, it looks like fr_tcpstate() *does* call MUTEX_ENTER on is->is_state twice with no intervening MUTEX_EXIT, i.e. recursively. This occurs in the case where a SYN is received to start a new connection when the previous connection has started to exit but has not completed (i.e. still to time out?) such as might happen if connections were being reused rapidly under our load testing. Looks like a simple cut+paste typo. I think the attached file ip_state.c.diff should fix it, but we haven't had a chance to test this yet. Heaps of context lines included, to easily see how the change relates to a similar section of code nearby. There are also a few comments in the archives concerning disabling PFILDEBUG when building the pfil module, to avoid the overhead of formatting the pretty-printing string on every packet, even if they aren't being logged. I'm intending to try the attached pfilstream.c patch also, which simply adds a check for the debug log level before doing the formatting, similar to what's done in a few other places within pfil. Others may wish to try a similar approach, see the pfilstream.c.diff file. As always, comments welcome. Rgds, Stuart. Stuart Remphrey RMIT ITS Infrastructure Services - Unix Systems Phone (03) 992 55 070 (or extension 55070)
ip_state.c.diff
Description: Binary data
console_ap2a.log
Description: Binary data
pfilstream.c.diff
Description: Binary data
