Raja, Also, this exception seems to be happening during the setup call while redeploying the operator. Do you have the exception from the first instance of the operator that made the operator go down?
Regards, Ashwin. On Thu, Mar 23, 2017 at 9:38 AM, Pramod Immaneni <[email protected]> wrote: > Hi Raja, > > What version of malhar are you using? Are you extending the > AbstractFileOutputOperator or are you use a stock implementation from > Malhar? > > Thanks > > On Thu, Mar 23, 2017 at 4:57 AM, Raja.Aravapalli < > [email protected]> wrote: > >> >> >> Hi Team, >> >> >> >> I have a file output operator, that is writing data into hdfs files, >> which is working fine for a day, after that started to fail with exception >> below: >> >> >> >> java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteEx >> ception(java.lang.ArrayIndexOutOfBoundsException): >> java.lang.ArrayIndexOutOfBoundsException >> >> at com.datatorrent.lib.io.fs.AbstractFileOutputOperator.setup(A >> bstractFileOutputOperator.java:428) >> >> at com.datatorrent.lib.io.fs.AbstractFileOutputOperator.setup(A >> bstractFileOutputOperator.java:112) >> >> at com.datatorrent.stram.engine.Node.setup(Node.java:187) >> >> at com.datatorrent.stram.engine.StreamingContainer.setupNode(St >> reamingContainer.java:1309) >> >> at com.datatorrent.stram.engine.StreamingContainer.access$100(S >> treamingContainer.java:130) >> >> at com.datatorrent.stram.engine.StreamingContainer$2.run(Stream >> ingContainer.java:1388) >> >> Caused by: >> org.apache.hadoop.ipc.RemoteException(java.lang.ArrayIndexOutOfBoundsException): >> java.lang.ArrayIndexOutOfBoundsException >> >> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1552) >> >> at org.apache.hadoop.ipc.Client.call(Client.java:1496) >> >> at org.apache.hadoop.ipc.Client.call(Client.java:1396) >> >> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke( >> ProtobufRpcEngine.java:233) >> >> at com.sun.proxy.$Proxy10.getBlockLocations(Unknown Source) >> >> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTran >> slatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:270) >> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> >> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce >> ssorImpl.java:62) >> >> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe >> thodAccessorImpl.java:43) >> >> at java.lang.reflect.Method.invoke(Method.java:497) >> >> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMeth >> od(RetryInvocationHandler.java:278) >> >> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(Ret >> ryInvocationHandler.java:194) >> >> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(Ret >> ryInvocationHandler.java:176) >> >> at com.sun.proxy.$Proxy11.getBlockLocations(Unknown Source) >> >> at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSCl >> ient.java:1236) >> >> at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1223) >> >> at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1211) >> >> at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndG >> etLastBlockLength(DFSInputStream.java:309) >> >> at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStrea >> m.java:274) >> >> at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:266) >> >> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1536) >> >> at org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall( >> DistributedFileSystem.java:330) >> >> at org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall( >> DistributedFileSystem.java:326) >> >> at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSyst >> emLinkResolver.java:81) >> >> at org.apache.hadoop.hdfs.DistributedFileSystem.open(Distribute >> dFileSystem.java:326) >> >> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:782) >> >> at com.datatorrent.lib.io.fs.AbstractFileOutputOperator.recover >> File(AbstractFileOutputOperator.java:455) >> >> at com.datatorrent.lib.io.fs.AbstractFileOutputOperator.setup(A >> bstractFileOutputOperator.java:373) >> >> ... 5 more >> >> 2017-03-23 06:53:47,942 INFO engine.StreamingContainer >> (StreamingContainer.java:processHeartbeatResponse(799)) - Undeploy >> request: [6] >> >> 2017-03-23 06:53:47,944 INFO engine.StreamingContainer >> (StreamingContainer.java:undeploy(561)) - Undeploy complete. >> >> >> >> >> >> >> >> Can someone pls help me fix this. It is breaking our production job! >> >> >> >> Thanks a lot. >> >> >> >> >> >> - Raja. >> > > -- Regards, Ashwin.
