[ https://issues.apache.org/jira/browse/IMPALA-7021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Ho reassigned IMPALA-7021: ---------------------------------- Assignee: (was: Sailesh Mukil) > Impala query fails with BlockMissingException while HDFS throws > SSLHandshakeException > ------------------------------------------------------------------------------------- > > Key: IMPALA-7021 > URL: https://issues.apache.org/jira/browse/IMPALA-7021 > Project: IMPALA > Issue Type: Bug > Components: Backend, Security > Affects Versions: Impala 2.12.0 > Reporter: Mostafa Mokhtar > Priority: Major > > Impala query failed with status > {code} > Status: Disk I/O error: Error reading from HDFS file: > hdfs://vc1512.halxg.cloudera.com:8020/user/hive/warehouse/tpcds_30000_parquet_6_rack.db/customer/1b4ebbf0879f988a-9733ba750000005b_76476061_data.0.parq > Error(255): Unknown error 255 Root cause: BlockMissingException: Could not > obtain block: > BP-1933019065-10.17.221.22-1485905285703:blk_1084803316_11063749 > file=/user/hive/warehouse/tpcds_30000_parquet_6_rack.db/customer/1b4ebbf0879f988a-9733ba750000005b_76476061_data.0.parq > {code} > Impala log > {code} > I0513 19:50:07.248292 150900 coordinator.cc:1092] Release admission control > resources for query_id=f14b2814eea7b58b:c2eeddb400000000 > W0513 19:50:11.313998 109914 ShortCircuitCache.java:826] > ShortCircuitCache(0x188f4ffc): could not load > 1084803339_BP-1933019065-10.17.221.22-1485905285703 due to InvalidToken > exception. > Java exception follows: > org.apache.hadoop.security.token.SecretManager$InvalidToken: access control > error while attempting to set up short-circuit access to > /user/hive/warehouse/tpcds_30000_parquet_6_rack.db/customer/1b4ebbf0879f988a-9733ba7500000004_131907241_data.0.parq > at > org.apache.hadoop.hdfs.BlockReaderFactory.requestFileDescriptors(BlockReaderFactory.java:653) > at > org.apache.hadoop.hdfs.BlockReaderFactory.createShortCircuitReplicaInfo(BlockReaderFactory.java:552) > at > org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.create(ShortCircuitCache.java:804) > at > org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.fetchOrCreate(ShortCircuitCache.java:738) > at > org.apache.hadoop.hdfs.BlockReaderFactory.getBlockReaderLocal(BlockReaderFactory.java:485) > at > org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:355) > at > org.apache.hadoop.hdfs.DFSInputStream.actualGetFromOneDataNode(DFSInputStream.java:1219) > at > org.apache.hadoop.hdfs.DFSInputStream.fetchBlockByteRange(DFSInputStream.java:1159) > at > org.apache.hadoop.hdfs.DFSInputStream.pread(DFSInputStream.java:1533) > at > org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:1492) > at > org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:92) > W0513 19:50:11.317173 109914 ShortCircuitCache.java:826] > ShortCircuitCache(0x188f4ffc): could not load > 1084803339_BP-1933019065-10.17.221.22-1485905285703 due to InvalidToken > exception. > Java exception follows: > org.apache.hadoop.security.token.SecretManager$InvalidToken: access control > error while attempting to set up short-circuit access to > /user/hive/warehouse/tpcds_30000_parquet_6_rack.db/customer/1b4ebbf0879f988a-9733ba7500000004_131907241_data.0.parq > at > org.apache.hadoop.hdfs.BlockReaderFactory.requestFileDescriptors(BlockReaderFactory.java:653) > at > org.apache.hadoop.hdfs.BlockReaderFactory.createShortCircuitReplicaInfo(BlockReaderFactory.java:552) > at > org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.create(ShortCircuitCache.java:804) > at > org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.fetchOrCreate(ShortCircuitCache.java:738) > at > org.apache.hadoop.hdfs.BlockReaderFactory.getBlockReaderLocal(BlockReaderFactory.java:485) > at > org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:355) > at > org.apache.hadoop.hdfs.DFSInputStream.actualGetFromOneDataNode(DFSInputStream.java:1219) > at > org.apache.hadoop.hdfs.DFSInputStream.fetchBlockByteRange(DFSInputStream.java:1159) > at > org.apache.hadoop.hdfs.DFSInputStream.pread(DFSInputStream.java:1533) > at > org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:1492) > at > org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:92) > W0513 19:50:11.318079 109914 DFSInputStream.java:1275] Connection failure: > Failed to connect to /10.17.187.36:20002 for file > /user/hive/warehouse/tpcds_30000_parquet_6_rack.db/customer/1b4ebbf0879f988a-9733ba7500000004_131907241_data.0.parq > for block > BP-1933019065-10.17.221.22-1485905285703:blk_1084803339_11063772:org.apache.hadoop.security.token.SecretManager$InvalidToken: > access control error while attempting to set up short-circuit access to > /user/hive/warehouse/tpcds_30000_parquet_6_rack.db/customer/1b4ebbf0879f988a-9733ba7500000004_131907241_data.0.parq > {code} > Impala error log > {code} > hdfsPread: FSDataInputStream#read error: > BlockMissingException: Could not obtain block: > BP-1933019065-10.17.221.22-1485905285703:blk_1084803339_11063772 > file=/user/hive/warehouse/tpcds_30000_parquet_6_rack.db/customer/1b4ebbf0879f988a-9733ba7500000004_131907241_data.0.parqorg.apache.hadoop.hdfs.BlockMissingException: > Could not obtain block: > BP-1933019065-10.17.221.22-1485905285703:blk_1084803339_11063772 > file=/user/hive/warehouse/tpcds_30000_parquet_6_rack.db/customer/1b4ebbf0879f988a-9733ba7500000004_131907241_data.0.parq > at > org.apache.hadoop.hdfs.DFSInputStream.refetchLocations(DFSInputStream.java:1048) > at > org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:1031) > at > org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:1010) > at > org.apache.hadoop.hdfs.DFSInputStream.fetchBlockByteRange(DFSInputStream.java:1156) > at > org.apache.hadoop.hdfs.DFSInputStream.pread(DFSInputStream.java:1533) > at > org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:1492) > at > org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:92) > {code} > NN log > {code} > 2018-05-13 19:51:54,857 INFO > org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: > Scanned 0 directive(s) and 0 block(s) in 11 millisecond(s). > 2018-05-13 19:51:57,409 WARN org.mortbay.log: EXCEPTION > javax.net.ssl.SSLHandshakeException: no cipher suites in common > at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) > at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1949) > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302) > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:292) > at > sun.security.ssl.ServerHandshaker.chooseCipherSuite(ServerHandshaker.java:1045) > at > sun.security.ssl.ServerHandshaker.clientHello(ServerHandshaker.java:741) > at > sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:224) > at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026) > at sun.security.ssl.Handshaker.process_record(Handshaker.java:961) > at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1062) > at > sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375) > at > sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403) > at > sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387) > at > org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:723) > at > org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) > 2018-05-13 19:52:00,591 INFO BlockStateChange: BLOCK* processReport > 0x422a616a86ad5ba4: discarded non-initial block report from > DatanodeRegistration(10.17.187.39, > datanodeUuid=42d8151a-7aaf-4f78-90d8-7d1bda177d5e, infoPort=0, > infoSecurePort=20004, ipcPort=20001, > storageInfo=lv=-56;cid=cluster2;nsid=606873798;c=0) because namenode still in > startup phase > {code} > Datanode log > {code} > 2018-05-13 19:50:08,946 ERROR > org.apache.hadoop.hdfs.server.datanode.DataNode: > va1030.halxg.cloudera.com:20002:DataXceiver error processing un > known operation src: /10.17.246.46:44103 dst: /10.17.187.40:20002 > javax.security.sasl.SaslException: DIGEST-MD5: IO error acquiring password > [Caused by org.apache.hadoop.hdfs.protocol.datatransfer.InvalidEncr > yptionKeyException: Can't re-compute encryption key for nonce, since the > required block key (keyID=1802676210) doesn't exist. Current key: 180 > 2676198] > at > com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:598) > at > com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) > at > org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslParticipant.evaluateChallengeOrResponse(SaslParticipant.java:115) > at > org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.doSaslHandshake(SaslDataTransferServer.java:372) > at > org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.getEncryptedStreams(SaslDataTransferServer.java:178) > at > org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.receive(SaslDataTransferServer.java:110) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:193) > at java.lang.Thread.run(Thread.java:745) > Caused by: > org.apache.hadoop.hdfs.protocol.datatransfer.InvalidEncryptionKeyException: > Can't re-compute encryption key for nonce, since the re > quired block key (keyID=1802676210) doesn't exist. Current key: 1802676198 > at > org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.retrieveDataEncryptionKey(BlockTokenSecretManager.java:430) > at > org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.retrieveDataEncryptionKey(BlockPoolTokenSecretManager.java:130) > at > org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.getEncryptionKeyFromUserName(SaslDataTransferServer.java:267) > at > org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.access$000(SaslDataTransferServer.java:71) > at > org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer$1.apply(SaslDataTransferServer.java:175) > at > org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer$SaslServerCallbackHandler.handle(SaslDataTransferServer.java:239) > at > com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) > ... 7 more > 2018-05-13 19:50:08,949 ERROR > org.apache.hadoop.hdfs.server.datanode.DataNode: > va1030.halxg.cloudera.com:20002:DataXceiver error processing unknown > operation src: /10.17.246.46:44104 dst: /10.17.187.40:20002 > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org