[ https://issues.apache.org/jira/browse/HDFS-12821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16254712#comment-16254712 ]
Gang Xie commented on HDFS-12821: --------------------------------- yes, should be the same issue. > Block invalid IOException causes the DFSClient domain socket being disabled > --------------------------------------------------------------------------- > > Key: HDFS-12821 > URL: https://issues.apache.org/jira/browse/HDFS-12821 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client > Affects Versions: 2.4.0, 2.6.0 > Reporter: Gang Xie > > We use HDFS2.4 & 2.6, and recently hit a issue that DFSClient domain socket > is disabled when datanode throw block invalid exception. > The block is invalidated for some reason on datanote and it's OK. Then > DFSClient tries to access this block on this datanode via domain socket. This > triggers a IOExcetion. On DFSClient side, when get a IOExcetion and error > code 'ERROR', it disables the domain socket and fails back to TCP. and the > worst is that it seems never recover the socket. > I think this is a defect and with such "block invalid" exception, we should > not disable the domain socket because the is nothing wrong about the domain > socket service. > And thoughts? > The code: > private ShortCircuitReplicaInfo requestFileDescriptors(DomainPeer peer, > Slot slot) throws IOException { > ShortCircuitCache cache = clientContext.getShortCircuitCache(); > final DataOutputStream out = > new DataOutputStream(new BufferedOutputStream(peer.getOutputStream())); > SlotId slotId = slot == null ? null : slot.getSlotId(); > new Sender(out).requestShortCircuitFds(block, token, slotId, 1); > DataInputStream in = new DataInputStream(peer.getInputStream()); > BlockOpResponseProto resp = BlockOpResponseProto.parseFrom( > PBHelper.vintPrefixed(in)); > DomainSocket sock = peer.getDomainSocket(); > switch (resp.getStatus()) { > case SUCCESS: > byte buf[] = new byte[1]; > FileInputStream fis[] = new FileInputStream[2]; > sock.recvFileInputStreams(fis, buf, 0, buf.length); > ShortCircuitReplica replica = null; > try { > ExtendedBlockId key = > new ExtendedBlockId(block.getBlockId(), block.getBlockPoolId()); > replica = new ShortCircuitReplica(key, fis[0], fis[1], cache, > Time.monotonicNow(), slot); > } catch (IOException e) { > // This indicates an error reading from disk, or a format error. Since > // it's not a socket communication problem, we return null rather than > // throwing an exception. > LOG.warn(this + ": error creating ShortCircuitReplica.", e); > return null; > } finally { > if (replica == null) { > IOUtils.cleanup(DFSClient.LOG, fis[0], fis[1]); > } > } > return new ShortCircuitReplicaInfo(replica); > case ERROR_UNSUPPORTED: > if (!resp.hasShortCircuitAccessVersion()) { > LOG.warn("short-circuit read access is disabled for " + > "DataNode " + datanode + ". reason: " + resp.getMessage()); > clientContext.getDomainSocketFactory() > .disableShortCircuitForPath(pathInfo.getPath()); > } else { > LOG.warn("short-circuit read access for the file " + > fileName + " is disabled for DataNode " + datanode + > ". reason: " + resp.getMessage()); > } > return null; > case ERROR_ACCESS_TOKEN: > String msg = "access control error while " + > "attempting to set up short-circuit access to " + > fileName + resp.getMessage(); > if (LOG.isDebugEnabled()) { > LOG.debug(this + ":" + msg); > } > return new ShortCircuitReplicaInfo(new InvalidToken(msg)); > default: > LOG.warn(this + ": unknown response code " + resp.getStatus() + > " while attempting to set up short-circuit access. " + > resp.getMessage()); > {color:#d04437} clientContext.getDomainSocketFactory() > .disableShortCircuitForPath(pathInfo.getPath());{color} > return null; > } -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org