Mostafa Mokhtar created IMPALA-7021: ---------------------------------------
Summary: Impala query fails with BlockMissingException while HDFS throws SSLHandshakeException Key: IMPALA-7021 URL: https://issues.apache.org/jira/browse/IMPALA-7021 Project: IMPALA Issue Type: Bug Components: Backend, Security Affects Versions: Impala 2.12.0 Reporter: Mostafa Mokhtar Assignee: Sailesh Mukil Impala query failed with status {code} Status: Disk I/O error: Error reading from HDFS file: hdfs://vc1512.halxg.cloudera.com:8020/user/hive/warehouse/tpcds_30000_parquet_6_rack.db/customer/1b4ebbf0879f988a-9733ba750000005b_76476061_data.0.parq Error(255): Unknown error 255 Root cause: BlockMissingException: Could not obtain block: BP-1933019065-10.17.221.22-1485905285703:blk_1084803316_11063749 file=/user/hive/warehouse/tpcds_30000_parquet_6_rack.db/customer/1b4ebbf0879f988a-9733ba750000005b_76476061_data.0.parq {code} Impala log {code} I0513 19:50:07.248292 150900 coordinator.cc:1092] Release admission control resources for query_id=f14b2814eea7b58b:c2eeddb400000000 W0513 19:50:11.313998 109914 ShortCircuitCache.java:826] ShortCircuitCache(0x188f4ffc): could not load 1084803339_BP-1933019065-10.17.221.22-1485905285703 due to InvalidToken exception. Java exception follows: org.apache.hadoop.security.token.SecretManager$InvalidToken: access control error while attempting to set up short-circuit access to /user/hive/warehouse/tpcds_30000_parquet_6_rack.db/customer/1b4ebbf0879f988a-9733ba7500000004_131907241_data.0.parq at org.apache.hadoop.hdfs.BlockReaderFactory.requestFileDescriptors(BlockReaderFactory.java:653) at org.apache.hadoop.hdfs.BlockReaderFactory.createShortCircuitReplicaInfo(BlockReaderFactory.java:552) at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.create(ShortCircuitCache.java:804) at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.fetchOrCreate(ShortCircuitCache.java:738) at org.apache.hadoop.hdfs.BlockReaderFactory.getBlockReaderLocal(BlockReaderFactory.java:485) at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:355) at org.apache.hadoop.hdfs.DFSInputStream.actualGetFromOneDataNode(DFSInputStream.java:1219) at org.apache.hadoop.hdfs.DFSInputStream.fetchBlockByteRange(DFSInputStream.java:1159) at org.apache.hadoop.hdfs.DFSInputStream.pread(DFSInputStream.java:1533) at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:1492) at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:92) W0513 19:50:11.317173 109914 ShortCircuitCache.java:826] ShortCircuitCache(0x188f4ffc): could not load 1084803339_BP-1933019065-10.17.221.22-1485905285703 due to InvalidToken exception. Java exception follows: org.apache.hadoop.security.token.SecretManager$InvalidToken: access control error while attempting to set up short-circuit access to /user/hive/warehouse/tpcds_30000_parquet_6_rack.db/customer/1b4ebbf0879f988a-9733ba7500000004_131907241_data.0.parq at org.apache.hadoop.hdfs.BlockReaderFactory.requestFileDescriptors(BlockReaderFactory.java:653) at org.apache.hadoop.hdfs.BlockReaderFactory.createShortCircuitReplicaInfo(BlockReaderFactory.java:552) at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.create(ShortCircuitCache.java:804) at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.fetchOrCreate(ShortCircuitCache.java:738) at org.apache.hadoop.hdfs.BlockReaderFactory.getBlockReaderLocal(BlockReaderFactory.java:485) at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:355) at org.apache.hadoop.hdfs.DFSInputStream.actualGetFromOneDataNode(DFSInputStream.java:1219) at org.apache.hadoop.hdfs.DFSInputStream.fetchBlockByteRange(DFSInputStream.java:1159) at org.apache.hadoop.hdfs.DFSInputStream.pread(DFSInputStream.java:1533) at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:1492) at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:92) W0513 19:50:11.318079 109914 DFSInputStream.java:1275] Connection failure: Failed to connect to /10.17.187.36:20002 for file /user/hive/warehouse/tpcds_30000_parquet_6_rack.db/customer/1b4ebbf0879f988a-9733ba7500000004_131907241_data.0.parq for block BP-1933019065-10.17.221.22-1485905285703:blk_1084803339_11063772:org.apache.hadoop.security.token.SecretManager$InvalidToken: access control error while attempting to set up short-circuit access to /user/hive/warehouse/tpcds_30000_parquet_6_rack.db/customer/1b4ebbf0879f988a-9733ba7500000004_131907241_data.0.parq {code} Impala error log {code} hdfsPread: FSDataInputStream#read error: BlockMissingException: Could not obtain block: BP-1933019065-10.17.221.22-1485905285703:blk_1084803339_11063772 file=/user/hive/warehouse/tpcds_30000_parquet_6_rack.db/customer/1b4ebbf0879f988a-9733ba7500000004_131907241_data.0.parqorg.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-1933019065-10.17.221.22-1485905285703:blk_1084803339_11063772 file=/user/hive/warehouse/tpcds_30000_parquet_6_rack.db/customer/1b4ebbf0879f988a-9733ba7500000004_131907241_data.0.parq at org.apache.hadoop.hdfs.DFSInputStream.refetchLocations(DFSInputStream.java:1048) at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:1031) at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:1010) at org.apache.hadoop.hdfs.DFSInputStream.fetchBlockByteRange(DFSInputStream.java:1156) at org.apache.hadoop.hdfs.DFSInputStream.pread(DFSInputStream.java:1533) at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:1492) at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:92) {code} NN log {code} 2018-05-13 19:51:54,857 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 11 millisecond(s). 2018-05-13 19:51:57,409 WARN org.mortbay.log: EXCEPTION javax.net.ssl.SSLHandshakeException: no cipher suites in common at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1949) at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302) at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:292) at sun.security.ssl.ServerHandshaker.chooseCipherSuite(ServerHandshaker.java:1045) at sun.security.ssl.ServerHandshaker.clientHello(ServerHandshaker.java:741) at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:224) at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026) at sun.security.ssl.Handshaker.process_record(Handshaker.java:961) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1062) at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387) at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:723) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) 2018-05-13 19:52:00,591 INFO BlockStateChange: BLOCK* processReport 0x422a616a86ad5ba4: discarded non-initial block report from DatanodeRegistration(10.17.187.39, datanodeUuid=42d8151a-7aaf-4f78-90d8-7d1bda177d5e, infoPort=0, infoSecurePort=20004, ipcPort=20001, storageInfo=lv=-56;cid=cluster2;nsid=606873798;c=0) because namenode still in startup phase {code} Datanode log {code} 2018-05-13 19:50:08,946 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: va1030.halxg.cloudera.com:20002:DataXceiver error processing un known operation src: /10.17.246.46:44103 dst: /10.17.187.40:20002 javax.security.sasl.SaslException: DIGEST-MD5: IO error acquiring password [Caused by org.apache.hadoop.hdfs.protocol.datatransfer.InvalidEncr yptionKeyException: Can't re-compute encryption key for nonce, since the required block key (keyID=1802676210) doesn't exist. Current key: 180 2676198] at com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:598) at com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslParticipant.evaluateChallengeOrResponse(SaslParticipant.java:115) at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.doSaslHandshake(SaslDataTransferServer.java:372) at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.getEncryptedStreams(SaslDataTransferServer.java:178) at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.receive(SaslDataTransferServer.java:110) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:193) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hdfs.protocol.datatransfer.InvalidEncryptionKeyException: Can't re-compute encryption key for nonce, since the re quired block key (keyID=1802676210) doesn't exist. Current key: 1802676198 at org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.retrieveDataEncryptionKey(BlockTokenSecretManager.java:430) at org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.retrieveDataEncryptionKey(BlockPoolTokenSecretManager.java:130) at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.getEncryptionKeyFromUserName(SaslDataTransferServer.java:267) at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.access$000(SaslDataTransferServer.java:71) at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer$1.apply(SaslDataTransferServer.java:175) at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer$SaslServerCallbackHandler.handle(SaslDataTransferServer.java:239) at com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) ... 7 more 2018-05-13 19:50:08,949 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: va1030.halxg.cloudera.com:20002:DataXceiver error processing unknown operation src: /10.17.246.46:44104 dst: /10.17.187.40:20002 {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org