well, StatusLogger.java started shown in cassandra system.log, MessagingService.java also shown some stage (e.g. read, mutation) dropped.
It's strange it only happen in this node but this type of message does not shown in other node log file at the same time... Jason On Thu, Mar 5, 2015 at 4:26 AM, Jan <cne...@yahoo.com> wrote: > HI Jason; > > Whats in the log files at the moment jstat shows 100%. > What is the activity on the cluster & the node at the specific point in > time (reads/ writes/ joins etc) > > Jan/ > > > On Wednesday, March 4, 2015 5:59 AM, Jason Wee <peich...@gmail.com> > wrote: > > > Hi, our cassandra node using java 7 update 72 and we ran jstat on one of > the node, and notice some strange behaviour as indicated by output below. > any idea why when eden space stay the same for few seconds like 100% and > 18.02% for few seconds? we suspect such "stalling" cause timeout to our > cluster. > > any idea what happened, what went wrong and what could cause this? > > > $ jstat -gcutil 32276 1s > > 0.00 5.78 91.21 70.94 60.07 2657 73.437 4 0.056 73.493 > 0.00 5.78 100.00 70.94 60.07 2657 73.437 4 0.056 73.493 > 0.00 5.78 100.00 70.94 60.07 2657 73.437 4 0.056 73.493 > 0.00 5.78 100.00 70.94 60.07 2657 73.437 4 0.056 73.493 > 0.00 5.78 100.00 70.94 60.07 2657 73.437 4 0.056 73.493 > 0.00 5.78 100.00 70.94 60.07 2657 73.437 4 0.056 73.493 > 0.00 5.78 100.00 70.94 60.07 2657 73.437 4 0.056 73.493 > 0.00 5.78 100.00 70.94 60.07 2657 73.437 4 0.056 73.493 > 0.00 5.78 100.00 70.94 60.07 2657 73.437 4 0.056 73.493 > 0.00 5.78 100.00 70.94 60.07 2657 73.437 4 0.056 73.493 > 0.00 5.78 100.00 70.94 60.07 2657 73.437 4 0.056 73.493 > 0.00 5.78 100.00 70.94 60.07 2657 73.437 4 0.056 73.493 > 0.00 5.78 100.00 70.94 60.07 2657 73.437 4 0.056 73.493 > 0.00 5.78 100.00 70.94 60.07 2657 73.437 4 0.056 73.493 > 0.00 5.78 100.00 70.94 60.07 2657 73.437 4 0.056 73.493 > 0.00 5.78 100.00 70.94 60.07 2657 73.437 4 0.056 73.493 > 0.00 5.78 100.00 70.94 60.07 2657 73.437 4 0.056 73.493 > 0.00 4.65 29.66 71.00 60.07 2659 73.488 4 0.056 73.544 > 0.00 4.65 70.88 71.00 60.07 2659 73.488 4 0.056 73.544 > 0.00 4.65 71.58 71.00 60.07 2659 73.488 4 0.056 73.544 > 0.00 4.65 72.15 71.00 60.07 2659 73.488 4 0.056 73.544 > 0.00 4.65 72.33 71.00 60.07 2659 73.488 4 0.056 73.544 > 0.00 4.65 72.73 71.00 60.07 2659 73.488 4 0.056 73.544 > 0.00 4.65 73.20 71.00 60.07 2659 73.488 4 0.056 73.544 > 0.00 4.65 73.71 71.00 60.07 2659 73.488 4 0.056 73.544 > 0.00 4.65 73.84 71.00 60.07 2659 73.488 4 0.056 73.544 > 0.00 4.65 73.91 71.00 60.07 2659 73.488 4 0.056 73.544 > 0.00 4.65 74.18 71.00 60.07 2659 73.488 4 0.056 73.544 > 0.00 4.65 74.29 71.00 60.07 2659 73.488 4 0.056 73.544 > 0.00 4.65 74.29 71.00 60.07 2659 73.488 4 0.056 73.544 > 0.00 4.65 74.29 71.00 60.07 2659 73.488 4 0.056 73.544 > 0.00 4.65 74.29 71.00 60.07 2659 73.488 4 0.056 73.544 > 0.00 4.65 74.29 71.00 60.07 2659 73.488 4 0.056 73.544 > 0.00 4.65 74.29 71.00 60.07 2659 73.488 4 0.056 73.544 > 0.00 4.65 74.29 71.00 60.07 2659 73.488 4 0.056 73.544 > 0.00 5.43 12.64 71.09 60.07 2661 73.534 4 0.056 73.590 > 0.00 5.43 18.02 71.09 60.07 2661 73.534 4 0.056 73.590 > 0.00 5.43 18.02 71.09 60.07 2661 73.534 4 0.056 73.590 > 0.00 5.43 18.02 71.09 60.07 2661 73.534 4 0.056 73.590 > 0.00 5.43 18.02 71.09 60.07 2661 73.534 4 0.056 73.590 > 0.00 5.43 18.02 71.09 60.07 2661 73.534 4 0.056 73.590 > 0.00 5.43 18.02 71.09 60.07 2661 73.534 4 0.056 73.590 > 0.00 5.43 18.02 71.09 60.07 2661 73.534 4 0.056 73.590 > 0.00 5.43 18.02 71.09 60.07 2661 73.534 4 0.056 73.590 > 0.00 5.43 18.02 71.09 60.07 2661 73.534 4 0.056 73.590 > 0.00 5.43 18.02 71.09 60.07 2661 73.534 4 0.056 73.590 > 0.00 5.43 18.02 71.09 60.07 2661 73.534 4 0.056 73.590 > 0.00 5.43 18.02 71.09 60.07 2661 73.534 4 0.056 73.590 > 0.00 5.43 18.02 71.09 60.07 2661 73.534 4 0.056 73.590 > 0.00 5.43 18.02 71.09 60.07 2661 73.534 4 0.056 73.590 > 0.00 5.43 18.02 71.09 60.07 2661 73.534 4 0.056 73.590 > 0.00 5.43 18.02 71.09 60.07 2661 73.534 4 0.056 73.590 > 0.00 5.43 18.02 71.09 60.07 2661 73.534 4 0.056 73.590 > 0.00 5.43 69.24 71.09 60.07 2661 73.534 4 0.056 73.590 > 0.00 5.43 78.05 71.09 60.07 2661 73.534 4 0.056 73.590 > 0.00 5.43 78.97 71.09 60.07 2661 73.534 4 0.056 73.590 > 0.00 5.43 79.07 71.09 60.07 2661 73.534 4 0.056 73.590 > 0.00 5.43 79.18 71.09 60.07 2661 73.534 4 0.056 73.590 > 0.00 5.43 80.09 71.09 60.07 2661 73.534 4 0.056 73.590 > 0.00 5.43 80.36 71.09 60.07 2661 73.534 4 0.056 73.590 > 0.00 5.43 80.51 71.09 60.07 2661 73.534 4 0.056 73.590 > 0.00 5.43 80.70 71.09 60.07 2661 73.534 4 0.056 73.590 > 0.00 5.43 81.31 71.09 60.07 2661 73.534 4 0.056 73.590 > 0.00 5.43 81.63 71.09 60.07 2661 73.534 4 0.056 73.590 > 0.00 5.43 82.94 71.09 60.07 2661 73.534 4 0.056 73.590 > 0.00 5.43 82.94 71.09 60.07 2661 73.534 4 0.056 73.590 > 0.00 5.43 82.94 71.09 60.07 2661 73.534 4 0.056 73.590 > 0.00 5.43 82.94 71.09 60.07 2661 73.534 4 0.056 73.590 > 0.00 5.43 82.94 71.09 60.07 2661 73.534 4 0.056 73.590 > 0.00 5.43 82.94 71.09 60.07 2661 73.534 4 0.056 73.590 > 0.00 5.43 82.94 71.09 60.07 2661 73.534 4 0.056 73.590 > 0.00 9.37 65.64 71.21 60.07 2663 73.583 4 0.056 73.639 > 0.00 3.49 11.89 71.25 60.10 2665 73.623 4 0.056 73.679 > 0.00 3.49 66.11 71.25 60.10 2665 73.623 4 0.056 73.679 > 14.26 0.00 9.91 71.25 60.10 2666 73.649 4 0.056 73.705 > 14.26 0.00 73.25 71.25 60.10 2666 73.649 4 0.056 73.705 > 0.00 10.49 14.43 71.29 60.10 2667 73.675 4 0.056 73.731 > 0.00 10.49 66.07 71.29 60.10 2667 73.675 4 0.056 73.731 > 0.00 10.49 100.00 71.29 60.10 2667 73.675 4 0.056 73.731 > 0.00 10.49 100.00 71.29 60.10 2667 73.675 4 0.056 73.731 > 0.00 10.49 100.00 71.29 60.10 2667 73.675 4 0.056 73.731 > 0.00 10.49 100.00 71.29 60.10 2667 73.675 4 0.056 73.731 > 0.00 10.49 100.00 71.29 60.10 2667 73.675 4 0.056 73.731 > 0.00 10.49 100.00 71.29 60.10 2667 73.675 4 0.056 73.731 > 0.00 10.49 100.00 71.29 60.10 2667 73.675 4 0.056 73.731 > 0.00 10.49 100.00 71.29 60.10 2667 73.675 4 0.056 73.731 > 0.00 10.49 100.00 71.29 60.10 2667 73.675 4 0.056 73.731 > 0.00 10.49 100.00 71.29 60.10 2667 73.675 4 0.056 73.731 > 0.00 10.49 100.00 71.29 60.10 2667 73.675 4 0.056 73.731 > 0.00 10.49 100.00 71.29 60.10 2667 73.675 4 0.056 73.731 > 0.00 10.49 100.00 71.29 60.10 2667 73.675 4 0.056 73.731 > 0.00 10.49 100.00 71.29 60.10 2667 73.675 4 0.056 73.731 > 0.00 10.49 100.00 71.29 60.10 2667 73.675 4 0.056 73.731 > > > thanks. > > jason > > > >
