hadoop Exception: java.io.IOException: Couldn't set up IO streams

[leiwang...@gmail.com]

Hi all, 

I write a pig script and run it on hadoop cluster.  Sometimes there will be 
this exception and sometimes not.

java.io.IOException: Failed on local exception: java.io.IOException: Couldn't 
set up IO streams; Host Details : local host is: 
"l-hadoop13.prod.cn2.corp.agrant.cn/10.2.1.45"; destination host is: 
"l-hadoop2.prod.cn2.corp.agrant.cn":8020; 
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:763)
        at org.apache.hadoop.ipc.Client.call(Client.java:1235)
        at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
        at com.sun.proxy.$Proxy9.delete(Unknown Source)
        at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:408)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
        at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
        at com.sun.proxy.$Proxy10.delete(Unknown Source)
        at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:1487)
        at 
org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:354)
        at 
org.apache.pig.backend.hadoop.datastorage.HPath.delete(HPath.java:118)
        at 
org.apache.pig.impl.io.FileLocalizer.deleteTempFiles(FileLocalizer.java:491)
        at org.apache.pig.Main.run(Main.java:642)
        at org.apache.pig.Main.main(Main.java:157)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
Caused by: java.io.IOException: Couldn't set up IO streams
        at 
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:663)
        at org.apache.hadoop.ipc.Client$Connection.access$2100(Client.java:243)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1284)
        at org.apache.hadoop.ipc.Client.call(Client.java:1202)
        ... 21 more
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:640)
        at 
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:656)
 
       ... 24 mor
 Hadoop verion: Hadoop 2.0.0-cdh4.3.1.

Any insight on this? How could i fix it? Why this error occurs occasionally?

Thanks,
Lei



leiwang...@gmail.com