Hi all,
I am loading data into hive tables by connecting to hiveserver through
thrift api using "load data local inpath ... " query .
Hive server is running as a background process for days . After some 4-5
days hiveserver throws this error :
11/02/11 11:20:02 WARN server.TThreadPoolServer: Transport error occurred
during acceptance of message.
org.apache.thrift.transport.TTransportException: java.net.SocketException:
Too many open files
at
org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:124)
at
org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:35)
at
org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
at
org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:184)
at
org.apache.hadoop.hive.service.HiveServer.main(HiveServer.java:388)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: java.net.SocketException: Too many open files
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
at java.net.ServerSocket.implAccept(ServerSocket.java:453)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at
org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:119)
... 9 more
I checked network connections using netstat and it show about 1000 active
network connections, most of them to hadoop datanode . Output is attached as
netstat.out
after i restarted hiveserver network connections falls to about 20 . output
is attached as netstat_after_HS_restart.out
Is this a problem with hive or hadoop configuration ?
Vaibhav Negi
