shahrs87 commented on pull request #3504:
URL: https://github.com/apache/hbase/pull/3504#issuecomment-884494797
```
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
132.699 s <<< FAILURE! - in
org.apache.hadoop.hbase.replication.TestReplicationKillMasterRSWithSeparateOldWALs
[ERROR]
org.apache.hadoop.hbase.replication.TestReplicationKillMasterRSWithSeparateOldWALs.killOneMasterRS
Time elapsed: 68.343 s <<< ERROR!
org.apache.hadoop.hbase.client.RetriesExhaustedException:
Failed after attempts=2, exceptions:
2021-07-21T19:02:25.047Z, java.net.ConnectException: Call to
address=rushabh-ltmflld.internal.salesforce.com:58846 failed on connection
exception:
org.apache.hbase.thirdparty.io.netty.channel.ConnectTimeoutException:
connection timed out:
rushabh-ltmflld.internal.salesforce.com/10.33.225.220:58846
2021-07-21T19:03:15.052Z,
org.apache.hadoop.hbase.exceptions.TimeoutIOException: Timeout(49891ms) waiting
for region location for test, row='ddd', replicaId=0
Caused by: org.apache.hadoop.hbase.exceptions.TimeoutIOException:
Timeout(49891ms) waiting for region location for test, row='ddd', replicaId=0
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
214.664 s - in org.apache.hadoop.hbase.replication.TestReplicationDroppedTables
[INFO] Tests run: 23, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
355.69 s - in
org.apache.hadoop.hbase.security.access.TestSnapshotScannerHDFSAclController
[INFO]
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR]
TestReplicationKillMasterRSWithSeparateOldWALs>TestReplicationKillMasterRS.killOneMasterRS:47->TestReplicationKillRS.loadTableAndKillRS:61
ยป RetriesExhausted
[INFO]
[ERROR] Tests run: 28, Failures: 0, Errors: 1, Skipped: 0
[INFO]
```
TestReplicationKillMasterRSWithSeparateOldWALs is a valid failure.
It is failing here with the following stack trace
```
RSS path:
hdfs://localhost:50931/user/rushabh.shah/test-data/764d7a00-1268-227d-46e0-bbc4d930588d/oldWALs/rushabh-ltmflld.internal.salesforce.com%2C50948%2C1626900149374.1626900153398
RSS serverName: null
2021-07-21T16:42:56,624 ERROR
[RS_CLAIM_REPLICATION_QUEUE-regionserver/rushabh-ltmflld:0-0.replicationSource,2-rushabh-ltmflld.internal.salesforce.com,50948,1626900149374.replicationSource.wal-reader.rushabh-ltmflld.internal.salesforce.com%2C50948%2C1626900149374,2-rushabh-ltmflld.internal.salesforce.com,50948,1626900149374]
regionserver.ReplicationSource(428): Unexpected exception in
RS_CLAIM_REPLICATION_QUEUE-regionserver/rushabh-ltmflld:0-0.replicationSource,2-rushabh-ltmflld.internal.salesforce.com,50948,1626900149374.replicationSource.wal-reader.rushabh-ltmflld.internal.salesforce.com%2C50948%2C1626900149374,2-rushabh-ltmflld.internal.salesforce.com,50948,1626900149374
currentPath=hdfs://localhost:50931/user/rushabh.shah/test-data/764d7a00-1268-227d-46e0-bbc4d930588d/oldWALs/rushabh-ltm%2C50948%2C1626900149374.1626900153398
java.lang.NullPointerException: null
at
org.apache.hadoop.hbase.wal.AbstractFSWALProvider.findArchivedLog(AbstractFSWALProvider.java:526)
~[classes/:?]
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.handleFileNotFound(WALEntryStream.java:321)
~[classes/:?]
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.openReader(WALEntryStream.java:342)
~[classes/:?]
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.openNextLog(WALEntryStream.java:307)
~[classes/:?]
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.checkReader(WALEntryStream.java:297)
~[classes/:?]
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.tryAdvanceEntry(WALEntryStream.java:177)
~[classes/:?]
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.hasNext(WALEntryStream.java:103)
~[classes/:?]
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.tryAdvanceStreamAndCreateWALBatch(ReplicationSourceWALReader.java:243)
~[classes/:?]
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:141)
~[classes/:?]
```
I printed the wal file name and it failed with NPE above. Interestingly the
path contains oldWALs directory and it is trying to find archive log for
oldWALs directory.
Below is the method of findArchivedLog method
```
public static Path findArchivedLog(Path path, Configuration conf) throws
IOException {
Path walRootDir = CommonFSUtils.getWALRootDir(conf);
FileSystem fs = path.getFileSystem(conf);
// Try finding the log in old dir
Path oldLogDir = new Path(walRootDir, HConstants.HREGION_OLDLOGDIR_NAME);
Path archivedLogLocation = new Path(oldLogDir, path.getName());
if (fs.exists(archivedLogLocation)) {
LOG.info("Log " + path + " was moved to " + archivedLogLocation);
return archivedLogLocation;
}
ServerName serverName = getServerNameFromWALDirectoryName(path); ----->
This serverName is null
// Try finding the log in separate old log dir
oldLogDir =
new Path(walRootDir, new
StringBuilder(HConstants.HREGION_OLDLOGDIR_NAME)
.append(Path.SEPARATOR).append(serverName.getServerName()).toString());
archivedLogLocation = new Path(oldLogDir, path.getName());
if (fs.exists(archivedLogLocation)) {
LOG.info("Log " + path + " was moved to " + archivedLogLocation);
return archivedLogLocation;
}
LOG.error("Couldn't locate log: " + path);
return null;
}
```
@Apache9 could we exit `findArchivedLog` early if the path contains oldWALs
?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
