ConfX created MAPREDUCE-7443:
--------------------------------
Summary: state polluter for system file permissions
Key: MAPREDUCE-7443
URL: https://issues.apache.org/jira/browse/MAPREDUCE-7443
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: ConfX
Attachments: reproduce.sh
h2. What happened:
After setting {{fs.permissions.umask-mode}} to disable write permission of a
file, the file's write permission is also disabled in host permission.
h2. Buggy code:
When creating {{target/test-dir/output}} the RawLocalFileSystem directly
manipulate the system permission (line 978 of {{{}RawLocalFileSystem.java{}}}):
{noformat}
String perm = String.format("%04o", permission.toShort());
Shell.execCommand(Shell.getSetPermissionCommand(perm, false,
FileUtil.makeShellPath(pathToFile(p), true)));{noformat}
If the permission turns off the write permission to the folder, the test would
fail due to permission denied. However, the test does not clean the folder
properly (by chmod and clean in an [@after|https://github.com/after] method),
causing all the subsequent runs to be polluted.
h2. StackTrace:
{noformat}
java.io.IOException: Mkdirs failed to create
file:/home/ctestfuzz/fuzz-hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/target/test-dir/output/_temporary/1/_temporary/attempt_200707121733_0001_m_000000_0
(exists=false,
cwd=file:/home/ctestfuzz/fuzz-hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core),
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:515),
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:500),
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1195),
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1081),
at
org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:125),
at
org.apache.hadoop.mapred.TestFileOutputCommitter.testRecoveryInternal(TestFileOutputCommitter.java:109),
at
org.apache.hadoop.mapred.TestFileOutputCommitter.testRecoveryUpgradeV1V2(TestFileOutputCommitter.java:171){noformat}
h2. How to reproduce:
There are two ways to reproduce:
# (1) Set {{fs.permissions.umask-mode}} to {{243}}
(2) Run test:
{{org.apache.hadoop.mapred.TestFileOutputCommitter#testRecoveryUpgradeV1V2}}
and observe an IOException
(3) Check to see that current user has lost writing access to
{{target/test-dir/output}}
# (1) Add an {{assertTrue(False);}} to line 112 of
{{TestFileOutputCommitter.java}} to simulate the test failing in the middle
(2) Run test:
{{org.apache.hadoop.mapred.TestFileOutputCommitter#testRecoveryUpgradeV1V2}}
and observe an AssertionError
(3) Set {{fs.permissions.umask-mode}} to {{243}}
(4) Run test:
{{org.apache.hadoop.mapred.TestFileOutputCommitter#testRecoveryUpgradeV1V2}}
and observe an IOException.
(5) Check to see that the current user has lost writing access to
{{test-dir/output/_temporary/1/_temporary/attempt_200707121733_0001_m_000000_0}}
For an easy reproduction, run the reproduce.sh in the attachment.
We are happy to provide a patch if this issue is confirmed.
{{{}{}}}{{{}{}}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
