Hi,
I have an implementation of SSH client using Mina SSHD.
I've noticed that working with it on several hosts will take the CPU to 100%
for every SSH
client session I have. For me this is a severe state, so I ran some debug /
profiling stuff.
First I used jstack to see what's running.
I saw-
"ClientInputStreamPump" daemon prio=6 tid=0x34db8400 nid=0xec8 runnable
[0x478df000]
java.lang.Thread.State: RUNNABLE
at
org.apache.sshd.client.channel.ChannelSession.securedRead(ChannelSession.java:107)
at
org.apache.sshd.client.channel.ChannelSession.pumpInputStream(ChannelSession.java:81)
at
org.apache.sshd.client.channel.ChannelSession$1.run(ChannelSession.java:53)
Locked ownable synchronizers:
- None
The, I wrote an application that connects to the other VM and list the thread
utilization...
Here's what I got-
T ID: 2637 Name: JMX server connection timeout 2637
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 2636 Name: RMI TCP Connection(310)-10.35.17.171
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 2611 Name: RMI TCP Accept-0
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 2528 Name: WorkerThread#0[127.0.0.1:2566]
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 2519 Name: ClientInputStreamPump
endCpuTime : 17624 sec's
endUserTime : 17619 sec's
T ID: 1101 Name: http-0.0.0.0-8080-27
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 1100 Name: http-0.0.0.0-8080-26
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 1099 Name: http-0.0.0.0-8080-25
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 1095 Name: http-0.0.0.0-8080-24
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 1093 Name: http-0.0.0.0-8080-23
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 1092 Name: http-0.0.0.0-8080-22
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 1091 Name: http-0.0.0.0-8080-21
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 1090 Name: http-0.0.0.0-8080-20
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 1089 Name: http-0.0.0.0-8080-19
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 1088 Name: http-0.0.0.0-8080-18
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 1087 Name: http-0.0.0.0-8080-17
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 1086 Name: http-0.0.0.0-8080-16
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 1085 Name: http-0.0.0.0-8080-15
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 1084 Name: http-0.0.0.0-8080-14
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 1083 Name: http-0.0.0.0-8080-13
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 1082 Name: http-0.0.0.0-8080-12
endCpuTime : 114 sec's
endUserTime : 103 sec's
T ID: 1081 Name: http-0.0.0.0-8080-11
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 1080 Name: http-0.0.0.0-8080-10
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 1079 Name: http-0.0.0.0-8080-9
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 1078 Name: http-0.0.0.0-8080-8
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 1077 Name: http-0.0.0.0-8080-7
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 1076 Name: http-0.0.0.0-8080-6
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 1075 Name: http-0.0.0.0-8080-5
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 1074 Name: http-0.0.0.0-8080-4
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 1073 Name: pool-22-thread-6
endCpuTime : 544 sec's
endUserTime : 534 sec's
T ID: 274 Name: http-0.0.0.0-8080-3
endCpuTime : 13 sec's
endUserTime : 13 sec's
T ID: 201 Name: MultiThreadedHttpConnectionManager cleanup
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 181 Name: http-0.0.0.0-8080-2
endCpuTime : 497 sec's
endUserTime : 444 sec's
T ID: 173 Name: http-0.0.0.0-8080-1
endCpuTime : 67 sec's
endUserTime : 60 sec's
T ID: 172 Name: ajp-0.0.0.0-8009-Acceptor-0
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 171 Name: http-0.0.0.0-8080-Acceptor-0
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 170 Name: JBossLifeThread
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 161 Name: IdleRemover
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 154 Name: Thread-17
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 153 Name: JCA PoolFiller
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 152 Name: HSQLDB Timer @1dc8b22
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 151 Name: QuartzScheduler_QuartzSchedulerThread
endCpuTime : 23 sec's
endUserTime : 22 sec's
T ID: 150 Name: QuartzScheduler_Worker-99
endCpuTime : 13 sec's
endUserTime : 11 sec's
T ID: 149 Name: QuartzScheduler_Worker-98
endCpuTime : 12 sec's
endUserTime : 10 sec's
T ID: 148 Name: QuartzScheduler_Worker-97
endCpuTime : 12 sec's
endUserTime : 10 sec's
T ID: 147 Name: QuartzScheduler_Worker-96
endCpuTime : 12 sec's
endUserTime : 10 sec's
T ID: 146 Name: QuartzScheduler_Worker-95
endCpuTime : 13 sec's
endUserTime : 11 sec's
T ID: 145 Name: QuartzScheduler_Worker-94
endCpuTime : 11 sec's
endUserTime : 10 sec's
T ID: 144 Name: QuartzScheduler_Worker-93
endCpuTime : 13 sec's
endUserTime : 11 sec's
T ID: 143 Name: QuartzScheduler_Worker-92
endCpuTime : 12 sec's
endUserTime : 10 sec's
T ID: 142 Name: QuartzScheduler_Worker-91
endCpuTime : 12 sec's
endUserTime : 11 sec's
T ID: 141 Name: QuartzScheduler_Worker-90
endCpuTime : 13 sec's
endUserTime : 10 sec's
T ID: 140 Name: QuartzScheduler_Worker-89
endCpuTime : 12 sec's
endUserTime : 10 sec's
T ID: 139 Name: QuartzScheduler_Worker-88
endCpuTime : 13 sec's
endUserTime : 11 sec's
T ID: 138 Name: QuartzScheduler_Worker-87
endCpuTime : 12 sec's
endUserTime : 10 sec's
T ID: 137 Name: QuartzScheduler_Worker-86
endCpuTime : 12 sec's
endUserTime : 10 sec's
T ID: 136 Name: QuartzScheduler_Worker-85
endCpuTime : 12 sec's
endUserTime : 10 sec's
T ID: 135 Name: QuartzScheduler_Worker-84
endCpuTime : 13 sec's
endUserTime : 11 sec's
T ID: 134 Name: QuartzScheduler_Worker-83
endCpuTime : 12 sec's
endUserTime : 10 sec's
T ID: 133 Name: QuartzScheduler_Worker-82
endCpuTime : 13 sec's
endUserTime : 11 sec's
T ID: 132 Name: QuartzScheduler_Worker-81
endCpuTime : 13 sec's
endUserTime : 11 sec's
T ID: 131 Name: QuartzScheduler_Worker-80
endCpuTime : 12 sec's
endUserTime : 10 sec's
T ID: 130 Name: QuartzScheduler_Worker-79
endCpuTime : 12 sec's
endUserTime : 10 sec's
T ID: 129 Name: QuartzScheduler_Worker-78
endCpuTime : 12 sec's
endUserTime : 11 sec's
T ID: 128 Name: QuartzScheduler_Worker-77
endCpuTime : 12 sec's
endUserTime : 10 sec's
T ID: 127 Name: QuartzScheduler_Worker-76
endCpuTime : 13 sec's
endUserTime : 11 sec's
T ID: 126 Name: QuartzScheduler_Worker-75
endCpuTime : 11 sec's
endUserTime : 10 sec's
T ID: 125 Name: QuartzScheduler_Worker-74
endCpuTime : 12 sec's
endUserTime : 10 sec's
T ID: 124 Name: QuartzScheduler_Worker-73
endCpuTime : 12 sec's
endUserTime : 10 sec's
T ID: 123 Name: QuartzScheduler_Worker-72
endCpuTime : 12 sec's
endUserTime : 10 sec's
T ID: 122 Name: QuartzScheduler_Worker-71
endCpuTime : 13 sec's
endUserTime : 11 sec's
T ID: 121 Name: QuartzScheduler_Worker-70
endCpuTime : 11 sec's
endUserTime : 9 sec's
T ID: 120 Name: QuartzScheduler_Worker-69
endCpuTime : 12 sec's
endUserTime : 10 sec's
T ID: 119 Name: QuartzScheduler_Worker-68
endCpuTime : 13 sec's
endUserTime : 11 sec's
T ID: 118 Name: QuartzScheduler_Worker-67
endCpuTime : 11 sec's
endUserTime : 9 sec's
T ID: 117 Name: QuartzScheduler_Worker-66
endCpuTime : 12 sec's
endUserTime : 10 sec's
T ID: 116 Name: QuartzScheduler_Worker-65
endCpuTime : 12 sec's
endUserTime : 11 sec's
T ID: 115 Name: QuartzScheduler_Worker-64
endCpuTime : 13 sec's
endUserTime : 11 sec's
T ID: 114 Name: QuartzScheduler_Worker-63
endCpuTime : 12 sec's
endUserTime : 10 sec's
T ID: 113 Name: QuartzScheduler_Worker-62
endCpuTime : 12 sec's
endUserTime : 10 sec's
T ID: 112 Name: QuartzScheduler_Worker-61
endCpuTime : 12 sec's
endUserTime : 10 sec's
T ID: 111 Name: QuartzScheduler_Worker-60
endCpuTime : 12 sec's
endUserTime : 11 sec's
T ID: 110 Name: QuartzScheduler_Worker-59
endCpuTime : 13 sec's
endUserTime : 11 sec's
T ID: 109 Name: QuartzScheduler_Worker-58
endCpuTime : 12 sec's
endUserTime : 10 sec's
T ID: 108 Name: QuartzScheduler_Worker-57
endCpuTime : 12 sec's
endUserTime : 10 sec's
T ID: 107 Name: QuartzScheduler_Worker-56
endCpuTime : 12 sec's
endUserTime : 10 sec's
T ID: 106 Name: QuartzScheduler_Worker-55
endCpuTime : 13 sec's
endUserTime : 11 sec's
T ID: 105 Name: QuartzScheduler_Worker-54
endCpuTime : 12 sec's
endUserTime : 10 sec's
T ID: 104 Name: QuartzScheduler_Worker-53
endCpuTime : 12 sec's
endUserTime : 10 sec's
T ID: 103 Name: QuartzScheduler_Worker-52
endCpuTime : 11 sec's
endUserTime : 10 sec's
T ID: 102 Name: QuartzScheduler_Worker-51
endCpuTime : 11 sec's
endUserTime : 10 sec's
T ID: 101 Name: QuartzScheduler_Worker-50
endCpuTime : 12 sec's
endUserTime : 10 sec's
T ID: 100 Name: QuartzScheduler_Worker-49
endCpuTime : 13 sec's
endUserTime : 11 sec's
T ID: 99 Name: QuartzScheduler_Worker-48
endCpuTime : 12 sec's
endUserTime : 10 sec's
T ID: 98 Name: QuartzScheduler_Worker-47
endCpuTime : 12 sec's
endUserTime : 10 sec's
T ID: 97 Name: QuartzScheduler_Worker-46
endCpuTime : 12 sec's
endUserTime : 10 sec's
T ID: 96 Name: QuartzScheduler_Worker-45
endCpuTime : 13 sec's
endUserTime : 10 sec's
T ID: 95 Name: QuartzScheduler_Worker-44
endCpuTime : 12 sec's
endUserTime : 10 sec's
T ID: 94 Name: QuartzScheduler_Worker-43
endCpuTime : 12 sec's
endUserTime : 10 sec's
T ID: 93 Name: QuartzScheduler_Worker-42
endCpuTime : 12 sec's
endUserTime : 11 sec's
T ID: 92 Name: QuartzScheduler_Worker-41
endCpuTime : 13 sec's
endUserTime : 11 sec's
T ID: 91 Name: QuartzScheduler_Worker-40
endCpuTime : 13 sec's
endUserTime : 11 sec's
T ID: 90 Name: QuartzScheduler_Worker-39
endCpuTime : 12 sec's
endUserTime : 11 sec's
T ID: 89 Name: QuartzScheduler_Worker-38
endCpuTime : 12 sec's
endUserTime : 10 sec's
T ID: 88 Name: QuartzScheduler_Worker-37
endCpuTime : 12 sec's
endUserTime : 10 sec's
T ID: 87 Name: QuartzScheduler_Worker-36
endCpuTime : 12 sec's
endUserTime : 10 sec's
T ID: 86 Name: QuartzScheduler_Worker-35
endCpuTime : 12 sec's
endUserTime : 10 sec's
T ID: 85 Name: QuartzScheduler_Worker-34
endCpuTime : 12 sec's
endUserTime : 10 sec's
T ID: 84 Name: QuartzScheduler_Worker-33
endCpuTime : 12 sec's
endUserTime : 10 sec's
T ID: 83 Name: QuartzScheduler_Worker-32
endCpuTime : 12 sec's
endUserTime : 11 sec's
T ID: 82 Name: QuartzScheduler_Worker-31
endCpuTime : 12 sec's
endUserTime : 10 sec's
T ID: 81 Name: QuartzScheduler_Worker-30
endCpuTime : 12 sec's
endUserTime : 10 sec's
T ID: 80 Name: QuartzScheduler_Worker-29
endCpuTime : 12 sec's
endUserTime : 10 sec's
T ID: 79 Name: QuartzScheduler_Worker-28
endCpuTime : 12 sec's
endUserTime : 10 sec's
T ID: 78 Name: QuartzScheduler_Worker-27
endCpuTime : 12 sec's
endUserTime : 11 sec's
T ID: 77 Name: QuartzScheduler_Worker-26
endCpuTime : 12 sec's
endUserTime : 10 sec's
T ID: 76 Name: QuartzScheduler_Worker-25
endCpuTime : 11 sec's
endUserTime : 9 sec's
T ID: 75 Name: QuartzScheduler_Worker-24
endCpuTime : 12 sec's
endUserTime : 10 sec's
T ID: 74 Name: QuartzScheduler_Worker-23
endCpuTime : 12 sec's
endUserTime : 11 sec's
T ID: 73 Name: QuartzScheduler_Worker-22
endCpuTime : 13 sec's
endUserTime : 11 sec's
T ID: 72 Name: QuartzScheduler_Worker-21
endCpuTime : 12 sec's
endUserTime : 10 sec's
T ID: 71 Name: QuartzScheduler_Worker-20
endCpuTime : 11 sec's
endUserTime : 9 sec's
T ID: 70 Name: QuartzScheduler_Worker-19
endCpuTime : 13 sec's
endUserTime : 11 sec's
T ID: 69 Name: QuartzScheduler_Worker-18
endCpuTime : 12 sec's
endUserTime : 10 sec's
T ID: 68 Name: QuartzScheduler_Worker-17
endCpuTime : 13 sec's
endUserTime : 11 sec's
T ID: 67 Name: QuartzScheduler_Worker-16
endCpuTime : 12 sec's
endUserTime : 10 sec's
T ID: 66 Name: QuartzScheduler_Worker-15
endCpuTime : 12 sec's
endUserTime : 10 sec's
T ID: 65 Name: QuartzScheduler_Worker-14
endCpuTime : 13 sec's
endUserTime : 11 sec's
T ID: 64 Name: QuartzScheduler_Worker-13
endCpuTime : 13 sec's
endUserTime : 11 sec's
T ID: 63 Name: QuartzScheduler_Worker-12
endCpuTime : 12 sec's
endUserTime : 10 sec's
T ID: 62 Name: QuartzScheduler_Worker-11
endCpuTime : 12 sec's
endUserTime : 10 sec's
T ID: 61 Name: QuartzScheduler_Worker-10
endCpuTime : 12 sec's
endUserTime : 10 sec's
T ID: 60 Name: QuartzScheduler_Worker-9
endCpuTime : 12 sec's
endUserTime : 10 sec's
T ID: 59 Name: QuartzScheduler_Worker-8
endCpuTime : 11 sec's
endUserTime : 10 sec's
T ID: 58 Name: QuartzScheduler_Worker-7
endCpuTime : 12 sec's
endUserTime : 10 sec's
T ID: 57 Name: QuartzScheduler_Worker-6
endCpuTime : 12 sec's
endUserTime : 10 sec's
T ID: 56 Name: QuartzScheduler_Worker-5
endCpuTime : 12 sec's
endUserTime : 10 sec's
T ID: 55 Name: QuartzScheduler_Worker-4
endCpuTime : 12 sec's
endUserTime : 10 sec's
T ID: 54 Name: QuartzScheduler_Worker-3
endCpuTime : 13 sec's
endUserTime : 11 sec's
T ID: 53 Name: QuartzScheduler_Worker-2
endCpuTime : 13 sec's
endUserTime : 11 sec's
T ID: 52 Name: QuartzScheduler_Worker-1
endCpuTime : 11 sec's
endUserTime : 9 sec's
T ID: 51 Name: QuartzScheduler_Worker-0
endCpuTime : 14 sec's
endUserTime : 11 sec's
T ID: 50 Name: WorkManager(2)-1
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 49 Name: AOPListner
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 48 Name: ContainerBackgroundProcessor[StandardEngine[jboss.web]]
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 47 Name: RMI TCP Accept-4444
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 46 Name: PooledInvokerAcceptor#0-4445
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 43 Name: Thread-14
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 44 Name: Listener:4712
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 42 Name: Thread-13
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 41 Name: Thread-12
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 40 Name: Thread-11
endCpuTime : 5 sec's
endUserTime : 5 sec's
T ID: 38 Name: Listener:4713
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 37 Name: AcceptorThread[ServerSocket[addr=/0.0.0.0,port=0,localport=4446]]
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 36 Name: ServerSocketRefresh
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 35 Name: HDScanner
endCpuTime : 319 sec's
endUserTime : 136 sec's
T ID: 34 Name: AcceptorThread[ServerSocket[addr=/0.0.0.0,port=0,localport=3873]]
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 33 Name: ServerSocketRefresh
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 32 Name: secondaryServerSocketThread[0]
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 31 Name: AcceptorThread[ServerSocket[addr=/0.0.0.0,port=0,localport=4457]]
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 30 Name: ServerSocketRefresh
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 28 Name: RMI RenewClean-[qumranetyk:1175]
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 27 Name: RMI Scheduler(0)
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 25 Name: RMI TCP Accept-0
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 24 Name: RMI TCP Accept-1090
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 23 Name: AcceptorThread[ServerSocket[addr=/0.0.0.0,port=0,localport=5400]]
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 22 Name: ServerSocketRefresh
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 21 Name: JBoss System Threads(1)-2
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 20 Name: GC Daemon
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 19 Name: RMI Reaper
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 18 Name: RMI TCP Accept-1098
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 17 Name: JBoss System Threads(1)-1
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 16 Name: Timer-Log4jService
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 15 Name: ZipFile Lock Reaper
endCpuTime : 1 sec's
endUserTime : 1 sec's
T ID: 14 Name: Timer-0
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 11 Name: DestroyJavaVM
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 5 Name: Attach Listener
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 4 Name: Signal Dispatcher
endCpuTime : 0 sec's
endUserTime : 0 sec's
T ID: 3 Name: Finalizer
endCpuTime : 8 sec's
endUserTime : 7 sec's
T ID: 2 Name: Reference Handler
endCpuTime : 2 sec's
endUserTime : 1 sec's
Just in case you missed it-
T ID: 2519 Name: ClientInputStreamPump
endCpuTime : 17624 sec's
endUserTime : 17619 sec's
It looks like ClientInputStreamPump is pumping my CPU :(
Any idea how to resolve it ?
--
/d
