[
https://issues.apache.org/jira/browse/HDFS-10592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15361632#comment-15361632
]
Rakesh R commented on HDFS-10592:
---------------------------------
As per the analysis the test case failure is due to the
{{ConcurrentModificationException}} while iterating over the mockito stubs.
Following are the two possible concurrent operations which results in the above
exception.
operation-1) NameNodeResourceMonitor is running and invoked
{{FSNamesystem.checkAvailableResources}}
operation-2) At the same time, adding the answer
{{Mockito.when(mockResourceChecker.hasAvailableDiskSpace()).thenReturn(false);}}
to the {{InvocationContainerImpl#stubbed}} linked list data structure.
Reference log:
{code}
2016-07-04 12:00:05,657 [DataNode:
[[[DISK]file:/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/3/dfs/data/data1/,
[DISK]file:/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/3/dfs/data/data2/]]
heartbeating to localhost/127.0.0.1:46553] INFO datanode.DataNode
(BPOfferService.java:processCommandFromActive(696)) - Got finalize command for
block pool BP-1174120436-172.17.0.2-1467633603414
2016-07-04 12:00:05,757 [main] INFO hdfs.MiniDFSCluster
(MiniDFSCluster.java:waitActive(2543)) - Cluster is active
2016-07-04 12:00:05,760
[org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor@640f11a1]
ERROR namenode.FSNamesystem (FSNamesystem.java:run(3808)) - Exception in
NameNodeResourceMonitor:
java.util.ConcurrentModificationException
at
java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966)
at java.util.LinkedList$ListItr.next(LinkedList.java:888)
at
org.mockito.internal.stubbing.InvocationContainerImpl.findAnswerFor(InvocationContainerImpl.java:71)
at org.mockito.internal.MockHandler.handle(MockHandler.java:93)
at
org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:47)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker$$EnhancerByMockitoWithCGLIB$$e4714180.hasAvailableDiskSpace(<generated>)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAvailableResources(FSNamesystem.java:3763)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor.run(FSNamesystem.java:3791)
at java.lang.Thread.run(Thread.java:745)
2016-07-04 12:01:05,772 [main] INFO hdfs.MiniDFSCluster
(MiniDFSCluster.java:shutdown(1907)) - Shutting down the Mini HDFS Cluster
2016-07-04 12:01:05,773 [main] INFO hdfs.MiniDFSCluster
(MiniDFSCluster.java:shutdownDataNodes(1947)) - Shutting down DataNode 0
{code}
org.mockito.internal.stubbing.InvocationContainerImpl#findAnswerFor is not
thread safe.
{code}
org.mockito.internal.stubbing.InvocationContainerImpl.java
public StubbedInvocationMatcher findAnswerFor(Invocation invocation) {
for (StubbedInvocationMatcher s : stubbed) {
if (s.matches(invocation)) {
s.markStubUsed(invocation);
invocation.markStubbed(new StubInfo(s));
return s;
}
}
return null;
}
{code}
*Proposed Fix:*
Instead of using {{Mockito}}, it is pretty simple to use
{{NameNodeResourceCheckerStub}} stubbing and sets the resource availability
explicitly. I will attach a patch soon.
> Fix intermittent test failure of
> TestNameNodeResourceChecker#testCheckThatNameNodeResourceMonitorIsRunning
> ----------------------------------------------------------------------------------------------------------
>
> Key: HDFS-10592
> URL: https://issues.apache.org/jira/browse/HDFS-10592
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Rakesh R
> Assignee: Rakesh R
>
> This jira is to fix the
> {{TestNameNodeResourceChecker#testCheckThatNameNodeResourceMonitorIsRunning}}
> test case failure.
> Reference
> [Build_15973|https://builds.apache.org/job/PreCommit-HDFS-Build/15973/testReport/junit/org.apache.hadoop.hdfs.server.namenode/TestNameNodeResourceChecker/testCheckThatNameNodeResourceMonitorIsRunning/]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
