I guess this is a bug in Hadoop 2.6.5 and has been fixed in Hadoop 2.8.0 [1]. You can work around it by explicitly setting the configration "hadoop.security.crypto.codec.classes.aes.ctr.nopadding" as "org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec, org.apache.hadoop.crypto.JceAesCtrCryptoCodec".
[1] https://issues.apache.org/jira/browse/HADOOP-11711 <https://issues.apache.org/jira/browse/HADOOP-11711> > 在 2019年10月28日，下午8:59，V N, Suchithra (Nokia - IN/Bangalore) > <suchithra....@nokia.com> 写道： > > Hi, > From debug logs I could see below logs in taskmanager. Please have a look. > > org.apache.hadoop.ipc.ProtobufRpcEngine Call: addBlock took 372ms"} > org.apache.hadoop.ipc.ProtobufRpcEngine Call: addBlock took 372ms"} > org.apache.hadoop.hdfs.DFSClient pipeline = 10.76.113.216:1044"} > org.apache.hadoop.hdfs.DFSClient pipeline = 10.76.113.216:1044"} > org.apache.hadoop.hdfs.DFSClient Connecting to datanode 10.76.113.216:1044"} > org.apache.hadoop.hdfs.DFSClient Connecting to datanode 10.76.113.216:1044"} > org.apache.hadoop.hdfs.DFSClient Send buf size 131072"} > org.apache.hadoop.hdfs.DFSClient Send buf size 131072"} > o.a.h.h.p.d.sasl.SaslDataTransferClient SASL client doing encrypted handshake > for addr = /10.76.113.216, datanodeId = 10.76.113.216:1044"} > o.a.h.h.p.d.sasl.SaslDataTransferClient SASL client doing encrypted handshake > for addr = /10.76.113.216, datanodeId = 10.76.113.216:1044"} > o.a.h.h.p.d.sasl.SaslDataTransferClient Client using encryption algorithm > 3des"} > o.a.h.h.p.d.sasl.SaslDataTransferClient Client using encryption algorithm > 3des"} > o.a.h.h.p.d.sasl.DataTransferSaslUtil Verifying QOP, requested QOP = > [auth-conf], negotiated QOP = auth-conf"} > o.a.h.h.p.d.sasl.DataTransferSaslUtil Verifying QOP, requested QOP = > [auth-conf], negotiated QOP = auth-conf"} > o.a.h.h.p.d.sasl.DataTransferSaslUtil Creating IOStreamPair of > CryptoInputStream and CryptoOutputStream."} > o.a.h.h.p.d.sasl.DataTransferSaslUtil Creating IOStreamPair of > CryptoInputStream and CryptoOutputStream."} > o.apache.hadoop.util.PerformanceAdvisory No crypto codec classes with cipher > suite configured."} > o.apache.hadoop.util.PerformanceAdvisory No crypto codec classes with cipher > suite configured."} > org.apache.hadoop.hdfs.DFSClient DataStreamer Exception"} > java.lang.NullPointerException: null > at > org.apache.hadoop.crypto.CryptoInputStream.<init>(CryptoInputStream.java:132) > at > org.apache.hadoop.hdfs.protocol.datatransfer.sasl.DataTransferSaslUtil.createStreamPair(DataTransferSaslUtil.java:345) > at > org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.doSaslHandshake(SaslDataTransferClient.java:489) > at > org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.getEncryptedStreams(SaslDataTransferClient.java:298) > at > org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.send(SaslDataTransferClient.java:241) > at > org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.checkTrustAndSend(SaslDataTransferClient.java:210) > at > org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.socketSend(SaslDataTransferClient.java:182) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1409) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1357) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:587) > > Regards, > Suchithra > > From: Dian Fu <dian0511...@gmail.com> > Sent: Monday, October 28, 2019 5:40 PM > To: V N, Suchithra (Nokia - IN/Bangalore) <suchithra....@nokia.com> > Cc: user@flink.apache.org > Subject: Re: Flink 1.8.1 HDFS 2.6.5 issue > > It seems that the CryptoCodec is null from the exception stack trace. This > may occur when "hadoop.security.crypto.codec.classes.aes.ctr.nopadding" is > misconfigured. You could change the log level to "DEBUG" and it will show > more detailed information about why CryptoCodec is null. > > 在 2019年10月28日，下午7:14，V N, Suchithra (Nokia - IN/Bangalore) > <suchithra....@nokia.com <mailto:suchithra....@nokia.com>> 写道： > > Hi, > > I am trying to execute Wordcount.jar in Flink 1.8.1 with Hadoop version > 2.6.5. HDFS is enabled with Kerberos+SSL. While writing output to HDFS, > facing the below exception and job will be failed. Please let me know if any > suggestions to debug this issue. > > Caused by: org.apache.flink.runtime.client.JobExecutionException: Job > execution failed. > at > org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:146) > at > org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:265) > ... 21 more > Caused by: java.io.IOException: DataStreamer Exception: > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:695) > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.crypto.CryptoInputStream.<init>(CryptoInputStream.java:132) > at > org.apache.hadoop.hdfs.protocol.datatransfer.sasl.DataTransferSaslUtil.createStreamPair(DataTransferSaslUtil.java:345) > at > org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.doSaslHandshake(SaslDataTransferClient.java:489) > at > org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.getEncryptedStreams(SaslDataTransferClient.java:298) > at > org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.send(SaslDataTransferClient.java:241) > at > org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.checkTrustAndSend(SaslDataTransferClient.java:210) > at > org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.socketSend(SaslDataTransferClient.java:182) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1409) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1357) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:587) > > Regards, > Suchithra
