Thanks a lot for looking into HDFS-7722, Chris.
In HDFS-7722:
TestDataNodeVolumeFailureXXX tests reset data dir permissions in TearDown().
TestDataNodeHotSwapVolumes reset permissions in a finally clause.
Also I ran mvn test several times on my machine and all tests passed.
However, since in DiskChecker#checkDirAccess():
private static void checkDirAccess(File dir) throws DiskErrorException {
if (!dir.isDirectory()) {
throw new DiskErrorException("Not a directory: "
+ dir.toString());
}
checkAccessByFileMethods(dir);
}
One potentially safer alternative is replacing data dir with a regular
file to stimulate disk failures.
On Tue, Mar 10, 2015 at 2:19 PM, Chris Nauroth <cnaur...@hortonworks.com> wrote:
> TestDataNodeHotSwapVolumes, TestDataNodeVolumeFailure,
> TestDataNodeVolumeFailureReporting, and
> TestDataNodeVolumeFailureToleration all remove executable permissions from
> directories like the one Colin mentioned to simulate disk failures at data
> nodes. I reviewed the code for all of those, and they all appear to be
> doing the necessary work to restore executable permissions at the end of
> the test. The only recent uncommitted patch I¹ve seen that makes changes
> in these test suites is HDFS-7722. That patch still looks fine though. I
> don¹t know if there are other uncommitted patches that changed these test
> suites.
>
> I suppose it¹s also possible that the JUnit process unexpectedly died
> after removing executable permissions but before restoring them. That
> always would have been a weakness of these test suites, regardless of any
> recent changes.
>
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
>
>
>
>
>
>
> On 3/10/15, 1:47 PM, "Aaron T. Myers" <a...@cloudera.com> wrote:
>
>>Hey Colin,
>>
>>I asked Andrew Bayer, who works with Apache Infra, what's going on with
>>these boxes. He took a look and concluded that some perms are being set in
>>those directories by our unit tests which are precluding those files from
>>getting deleted. He's going to clean up the boxes for us, but we should
>>expect this to keep happening until we can fix the test in question to
>>properly clean up after itself.
>>
>>To help narrow down which commit it was that started this, Andrew sent me
>>this info:
>>
>>"/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-
>>Build/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data3/ has
>>500 perms, so I'm guessing that's the problem. Been that way since 9:32
>>UTC
>>on March 5th."
>>
>>--
>>Aaron T. Myers
>>Software Engineer, Cloudera
>>
>>On Tue, Mar 10, 2015 at 1:24 PM, Colin P. McCabe <cmcc...@apache.org>
>>wrote:
>>
>>> Hi all,
>>>
>>> A very quick (and not thorough) survey shows that I can't find any
>>> jenkins jobs that succeeded from the last 24 hours. Most of them seem
>>> to be failing with some variant of this message:
>>>
>>> [ERROR] Failed to execute goal
>>> org.apache.maven.plugins:maven-clean-plugin:2.5:clean (default-clean)
>>> on project hadoop-hdfs: Failed to clean project: Failed to delete
>>>
>>>
>>>/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/hadoop-hdfs-pr
>>>oject/hadoop-hdfs/target/test/data/dfs/data/data3
>>> -> [Help 1]
>>>
>>> Any ideas how this happened? Bad disk, unit test setting wrong
>>> permissions?
>>>
>>> Colin
>>>
>
--
Lei (Eddy) Xu
Software Engineer, Cloudera
