[ https://issues.apache.org/jira/browse/HDFS-10830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15478611#comment-15478611 ]
Hadoop QA commented on HDFS-10830: ---------------------------------- | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 4 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 17s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 30s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 52s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 12s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 45s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 57s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 48s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 44s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 26s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 49s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 10s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 52s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 58m 9s{color} | {color:green} hadoop-hdfs in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 20s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 78m 0s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:9560f25 | | JIRA Issue | HDFS-10830 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12827843/HDFS-10830.03.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 8a2e1a257af6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / cba973f | | Default Java | 1.8.0_101 | | findbugs | v3.0.0 | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/16695/testReport/ | | modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/16695/console | | Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > FsDatasetImpl#removeVolumes() crashes with IllegalMonitorStateException when > vol being removed is in use > -------------------------------------------------------------------------------------------------------- > > Key: HDFS-10830 > URL: https://issues.apache.org/jira/browse/HDFS-10830 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs > Affects Versions: 3.0.0-alpha1 > Reporter: Manoj Govindassamy > Assignee: Arpit Agarwal > Attachments: HDFS-10830.01.patch, HDFS-10830.02.patch, > HDFS-10830.05.patch > > > {{FsDatasetImpl#removeVolumes()}} operation crashes abruptly with > IllegalMonitorStateException whenever the volume being removed is in use > concurrently. > Looks like {{removeVolumes()}} is waiting on a monitor object "this" (that is > FsDatasetImpl) which it has never locked, leading to > IllegalMonitorStateException. This monitor wait happens only the volume being > removed is in use (referencecount > 0). The thread performing this remove > volume operation thus crashes abruptly and block invalidations for the remove > volumes are totally skipped. > {code:title=FsDatasetImpl.java|borderStyle=solid} > @Override > public void removeVolumes(Set<File> volumesToRemove, boolean clearFailure) { > .. > .. > try (AutoCloseableLock lock = datasetLock.acquire()) { <== LOCK acquire > datasetLock > for (int idx = 0; idx < dataStorage.getNumStorageDirs(); idx++) { > .. .. .. > asyncDiskService.removeVolume(sd.getCurrentDir()); <== volume SD1 remove > volumes.removeVolume(absRoot, clearFailure); > volumes.waitVolumeRemoved(5000, this); <== WAIT on "this" > ?? But, we haven't locked it yet. > This will cause > IllegalMonitorStateException > and crash > getBlockReports()/FBR thread! > for (String bpid : volumeMap.getBlockPoolList()) { > List<ReplicaInfo> blocks = new ArrayList<>(); > for (Iterator<ReplicaInfo> it = volumeMap.replicas(bpid).iterator(); > it.hasNext(); ) { > .. .. .. > it.remove(); <== volumeMap removal > } > blkToInvalidate.put(bpid, blocks); > } > .. .. > } <== LOCK release > datasetLock > // Call this outside the lock. > for (Map.Entry<String, List<ReplicaInfo>> entry : > blkToInvalidate.entrySet()) { > .. > for (ReplicaInfo block : blocks) { > invalidate(bpid, block); <== Notify NN of > Block removal > } > } > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org