[ https://issues.apache.org/jira/browse/HBASE-24021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Guanghao Zhang resolved HBASE-24021. ------------------------------------ Fix Version/s: 2.2.5 2.4.0 2.3.0 3.0.0 Resolution: Fixed Pushed to branch-2.2+. > Fail fast when bulkLoadHFiles method catch some IOException > ----------------------------------------------------------- > > Key: HBASE-24021 > URL: https://issues.apache.org/jira/browse/HBASE-24021 > Project: HBase > Issue Type: Improvement > Components: HFile, regionserver > Reporter: niuyulin > Assignee: niuyulin > Priority: Major > Fix For: 3.0.0, 2.3.0, 2.4.0, 2.2.5 > > > In production environment, we usually do bulkload huge amount hfile . It > reasonable fail fast when any IOException occur > > hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java > {code:java} > public Map<byte[], List<Path>> bulkLoadHFiles(Collection<Pair<byte[], > String>> familyPaths, > boolean assignSeqId, BulkLoadListener bulkLoadListener, > boolean copyFile, List<String> clusterIds, boolean replicate) throws > IOException { > ...... > try { > this.writeRequestsCount.increment(); > // There possibly was a split that happened between when the split keys > // were gathered and before the HRegion's write lock was taken. We need > // to validate the HFile region before attempting to bulk load all of them > List<IOException> ioes = new ArrayList<>(); > List<Pair<byte[], String>> failures = new ArrayList<>(); > for (Pair<byte[], String> p : familyPaths) { > byte[] familyName = p.getFirst(); > String path = p.getSecond(); > HStore store = getStore(familyName); > if (store == null) { > IOException ioe = new org.apache.hadoop.hbase.DoNotRetryIOException( > "No such column family " + Bytes.toStringBinary(familyName)); > ioes.add(ioe); > } else { > try { > store.assertBulkLoadHFileOk(new Path(path)); > } catch (WrongRegionException wre) { > // recoverable (file doesn't fit in region) > failures.add(p); > } catch (IOException ioe) { > // unrecoverable (hdfs problem) > ioes.add(ioe); > } > } > } > // validation failed because of some sort of IO problem. > if (ioes.size() != 0) { > IOException e = MultipleIOException.createIOException(ioes); > LOG.error("There were one or more IO errors when checking if the bulk > load is ok.", e); > throw e; > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)