I have seen this when I was over-loading our zookeeper instance. When total data or total number of files gets large, the system can wind up in GC almost permanently. Zookeeper being Zookeeper, it does an amazing job of keeping on, but eventually things go bad.
To test if this is your problem, you can do a couple of things. - first, figure out how many files you have and how much space they take - second, review how much memory you gave ZK to start. It looks from what you show here that you didn't give it much memory. - third, turn on verbose GC and see what that says. - fourth, write back and tell me why I was wrong with this guess so that we all get smarter! On Tue, Jun 2, 2009 at 12:07 PM, Satish Bhatti <cthd2...@gmail.com> wrote: > I am running a 5 node ZooKeeper cluster and I noticed that one of them has > very high CPU usage: > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 6883 infact 22 0 725m 41m 4188 S 95 0.5 > 5671:54 java > > It is not "doing anything" application-wise at this point, so I was > wondering why the heck it's using up so much CPU! I have attached a jstack > logfile to this email. > > Satish > > -- Ted Dunning, CTO DeepDyve 111 West Evelyn Ave. Ste. 202 Sunnyvale, CA 94086 http://www.deepdyve.com 858-414-0013 (m) 408-773-0220 (fax)
