On Hudson, we've been seeing tests sporadically hang on an ipc Client
flush of params. I'm writing the list for suggestions or opinions on
what folks think might be happening or ideas on what to try next. See
below for the latest example for a thread dump from a recent patch build.
The usual scenario is that we are trying to simulate failed servers in a
mini-cluster. All servers -- hbase + dfs servers -- are up and running
inside the same JVM. The remote ipc Server will of-a-sudden have its
stop method run to simulate a server crash. The Client, unawares, tries
to go about its usual business.
[junit] "HMaster.metaScanner" daemon prio=10 tid=0x091ecde0 nid=0x4a
runnable [0xe2af9000..0xe2af9b38]
[junit] at java.net.SocketOutputStream.socketWrite0(Native Method)
[junit] at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
[junit] at
java.net.SocketOutputStream.write(SocketOutputStream.java:136)
[junit] at
org.apache.hadoop.ipc.Client$Connection$2.write(Client.java:190)
[junit] at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
[junit] at
java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
[junit] - locked <0xf7bb40e0> (a java.io.BufferedOutputStream)
[junit] at java.io.DataOutputStream.flush(DataOutputStream.java:106)
[junit] at
org.apache.hadoop.ipc.Client$Connection.sendParam(Client.java:325)
[junit] - locked <0xf7bb3f68> (a java.io.DataOutputStream)
[junit] at org.apache.hadoop.ipc.Client.call(Client.java:462)
[junit] - locked <0xf7bb3fa8> (a org.apache.hadoop.ipc.Client$Call)
[junit] at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:165)
[junit] at $Proxy8.openScanner(Unknown Source)
[junit] at
org.apache.hadoop.hbase.HMaster$BaseScanner.scanRegion(HMaster.java:207)
[junit] at
org.apache.hadoop.hbase.HMaster$MetaScanner.scanOneMetaRegion(HMaster.java:643)
[junit] - locked <0xf7b6b460> (a java.lang.Integer)
[junit] at
org.apache.hadoop.hbase.HMaster$MetaScanner.maintenanceScan(HMaster.java:694)
[junit] at
org.apache.hadoop.hbase.HMaster$BaseScanner.chore(HMaster.java:188)
[junit] at org.apache.hadoop.hbase.Chore.run(Chore.java:59)
Other threads in the thread dump will be parked at the DataOutputStream
synchronize block.
Please correct me if I am wrong, but it is my understanding that writes
do not timeout nor is this type of I/O interruptable. The connection is
probably already established else it would have timed out trying to
connect to the non-existent server and besides, the ipc Client pattern
seems to be keeps up the connection multiplexing 'commands' to the
remote server...
I'm wondering why don't we get an exception on client side when the
remote side of the socket goes away?
Am unable to reproduce locally.
Thanks for any input,
St.Ack
