The setup: Slackware 11 server running MySQL version 5.0.24a. Data tables are InnoDB. Mostly used by a Tomcat server on the same 10.10.10.x network. Firewall and port forwarding are managed by GuardDog and GuideDog. The outside world has no direct access to the server and can not present queries to, everything goes through Tomcat and another simple Java POJO server.
Ran great for over a year. Then, two weeks ago, it suddenly had a query (unidentified) that caused it to lock any query that tried to insert anything and the server was running at 100% CPU utilization. Probably 50 users were on the system. We use Navicat. The Server monitor showed some queries from an IP that is internal (192.168.0.106.) Yet, that computer did not have anything that would hit MySQL. I eventually got the whole process back up by rebooting the Tomcat server and the MySQL server. I suspected a long running query so I started monitoring slow queries. Identified one that I thought my be the culprit (it was a query from a report that I was trying to fix.) Today, the scenario repeated. This time the queries were coming from 192.168.0.107. CPU was 100% and all inserts were locked. Navicat showed the query as one of our stored procedures which would normally run in a fraction of a second but there was just a flood of them. I shut down 192.168.0.107 but, surprisingly, the queries continued. I stopped Tomcat (very difficult) and stopped and started MySQL through mysqladmin. Ran OK for about five minutes and the flood of queries started again from 192.168.0.107 (which, of course, was completely powered down.) Thinking there was a stack of queries someplace, I rebooted the MySQL server and once again started Tomcat, etc. Within five minutes more queries appeared from 192.168.0.107 and the CPU went to 100%. After about 10 minutes, they all disappeared and everything returned to normal. The data seems intact, no damage that I can discern. All the transactions seemed to have processed or aborted. I am befuddled: 1. Queries coming from a powered down computer would seem to be either a) stacked up someplace or b) from a hacker/intruder. Rebooting the server should have cleared any stack so it looks like I am left with an intruder (which doesn't sound very good.) 2. If it is an intruder, why wouldn't he/she keep going until the server just died. If anybody has any ideas, I sure would like to hear them. TIA, Carl
