[
https://issues.apache.org/jira/browse/HDFS-17844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18024385#comment-18024385
]
ASF GitHub Bot commented on HDFS-17844:
---------------------------------------
Wang-Benjamin opened a new pull request, #8009:
URL: https://github.com/apache/hadoop/pull/8009
## Problem
There could be a race condition in generating namenode ID. What is
problematic is that different URIs resolve to the same namenode ID, which leads
to data integrity issues and security problems. The logic of the underlying
function used to detect such problem is incorrect as it throws an error
whenever the namenode exists even if the URI is the same. As a result,
`testViewFsMultipleExportPoint()` has a nondeterministic behavior when the test
execution order and timing change.
Command line used to identify the nondeterminism of
`testViewFsMultipleExportPoint()`:
```
mvn -pl hadoop-hdfs-project/hadoop-hdfs-nfs \
-Dcheckstyle.skip=true -Drat.skip=true \
edu.illinois:nondex-maven-plugin:2.1.7:nondex \
-Dtest=org.apache.hadoop.hdfs.nfs.nfs3.TestExportsTable#testViewFsMultipleExportPoint
\
-DnondexRuns=20 |& tee nondex-$(date +%s).log
```
Error example:
```
[ERROR] TestExportsTable.testViewFsMultipleExportPoint:111 » FileSystem
FS:viewfs, Namenode ID collision for path:/hdfs2 nnid:2130740544 uri being
added:hdfs://localhost:34111/ existing uri:hdfs://localhost:34111/
```
## Patch
The proposed fix has changed the collision detection logic in
`prepareAddressMap()` so that it will throw errors only when different URIs
resolves to the same namenode ID.
In addition, the namenode ID generation logic in `getNamenodeId()` is
modified where a robust hash function combines URI and address strings, so it
now reduces the chance of hash collision.
> Flaky test: testViewFsMultipleExportPoint may encounter race condition which
> leads to namenode ID collisions
> ------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-17844
> URL: https://issues.apache.org/jira/browse/HDFS-17844
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode, test
> Reporter: Yakun Wang
> Priority: Major
> Labels: patch
>
> There could be a race condition in namenode ID generation, causing ID
> collisions. However, the underlying collision detection logic is incorrect
> because it throws an error as long as the namenode ID exists, even if the URI
> is the same. Same namenode ID with same URI is safe. What needs to be
> detected is that different URIs resolve to the same namenode ID, which is a
> security problem.
> Error output:
> `[ERROR] Errors:
> [ERROR] TestExportsTable.testViewFsMultipleExportPoint:111 » FileSystem
> FS:viewfs, Namenode ID collision for path:/hdfs2 nnid:2130740544 uri being
> added:hdfs://localhost:34111/ existing uri:hdfs://localhost:34111/`
> Functions that are related to collision detection logic:
> # prepareAddressMap in
> hadoop/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/DFSClientCache.java
> # getNamenodeId in
> hadoop/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/Nfs3Utils.java
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
