[jira] Commented: (MAPREDUCE-1415) With streaming jobs and LinuxTaskController, the localized streaming binary has 571 permissions instead of 570
[ https://issues.apache.org/jira/browse/MAPREDUCE-1415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12843906#action_12843906 ] Amareshwari Sriramadasu commented on MAPREDUCE-1415: bq. -1 contrib tests. Many sqoop tests fail with NoClassDefFoundError. All the tests passed on my machine. With streaming jobs and LinuxTaskController, the localized streaming binary has 571 permissions instead of 570 -- Key: MAPREDUCE-1415 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1415 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/streaming, security Reporter: Vinod K V Assignee: Amareshwari Sriramadasu Fix For: 0.22.0 Attachments: patch-1415-1.txt, patch-1415-2.txt, patch-1415-3.txt, patch-1415.txt After MAPREDUCE-856, all localized files are expected to have **0 permissions for the sake of security. This was found by Karam while testing LinuxTaskController functionality after MAPREDUCE-856. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1415) With streaming jobs and LinuxTaskController, the localized streaming binary has 571 permissions instead of 570
[ https://issues.apache.org/jira/browse/MAPREDUCE-1415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12842996#action_12842996 ] Hemanth Yamijala commented on MAPREDUCE-1415: - Looks good overall. Very minor suggestions: - In PipeMapRed.configure, can we use File.setExecutable(true) instead of FileUtil.chmod ? With FileUtil.chmod, I suppose an exception will be thrown if there's an invalid path passed. To maintain that semantics, we probably need to check the return value of File.setExecutable and throw a wrapped IOException when that happens. - In testStreaming, we can use TEST_DIR.toAbsolutePath() as the value of stream.tmpdir. - Very minor - in testStreamingWithDistCache, I suggest we use a variable to store the file name of the input script, test.sh. With streaming jobs and LinuxTaskController, the localized streaming binary has 571 permissions instead of 570 -- Key: MAPREDUCE-1415 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1415 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/streaming, security Reporter: Vinod K V Assignee: Amareshwari Sriramadasu Fix For: 0.22.0 Attachments: patch-1415-1.txt, patch-1415.txt After MAPREDUCE-856, all localized files are expected to have **0 permissions for the sake of security. This was found by Karam while testing LinuxTaskController functionality after MAPREDUCE-856. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1415) With streaming jobs and LinuxTaskController, the localized streaming binary has 571 permissions instead of 570
[ https://issues.apache.org/jira/browse/MAPREDUCE-1415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12843235#action_12843235 ] Hemanth Yamijala commented on MAPREDUCE-1415: - Looking at one of the failed tests, it appears that we have test cases that use default commands like 'xargs' as the mapper, but don't specify the full path. Hence the code tries to set permissions for it in the current working directory and fails (because the file does not exist). I thought FileUtil.fullyDelete actually throws an exception if there's a command failure, but I see in code that we actually ignore exceptions explicitly. I assume the failure should have been seen before as well - at some debug log level. So, one option could be to not check the return code of setPermissions. Another option is to change the tests to work with full paths. However, this is inherently system dependent (as xargs could potentially be under multiple directories depending on the system). With streaming jobs and LinuxTaskController, the localized streaming binary has 571 permissions instead of 570 -- Key: MAPREDUCE-1415 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1415 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/streaming, security Reporter: Vinod K V Assignee: Amareshwari Sriramadasu Fix For: 0.22.0 Attachments: patch-1415-1.txt, patch-1415-2.txt, patch-1415.txt After MAPREDUCE-856, all localized files are expected to have **0 permissions for the sake of security. This was found by Karam while testing LinuxTaskController functionality after MAPREDUCE-856. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1415) With streaming jobs and LinuxTaskController, the localized streaming binary has 571 permissions instead of 570
[
https://issues.apache.org/jira/browse/MAPREDUCE-1415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12842529#action_12842529
]
Amareshwari Sriramadasu commented on MAPREDUCE-1415:
Test failure TestBadRecords is not related to the patch.
The test failed with NoClassDefFoundError (MAPREDUCE-1275).
Stack trace for the failure :
{noformat}
java.lang.NoClassDefFoundError: org/apache/hadoop/io/SequenceFile$Reader
at
org.apache.hadoop.mapred.TestBadRecords.validateOutput(TestBadRecords.java:147)
at
org.apache.hadoop.mapred.TestBadRecords.runMapReduce(TestBadRecords.java:96)
at
org.apache.hadoop.mapred.TestBadRecords.testBadMapRed(TestBadRecords.java:213)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.io.SequenceFile$Reader
at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
Caused by: java.util.zip.ZipException: error reading zip file
at java.util.zip.ZipFile.read(Native Method)
at java.util.zip.ZipFile.access$1200(ZipFile.java:29)
at java.util.zip.ZipFile$ZipFileInputStream.read(ZipFile.java:447)
at java.util.zip.ZipFile$1.fill(ZipFile.java:230)
at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:141)
at sun.misc.Resource.getBytes(Resource.java:97)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:256)
at java.net.URLClassLoader.access$000(URLClassLoader.java:56)
at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
{noformat}
With streaming jobs and LinuxTaskController, the localized streaming binary
has 571 permissions instead of 570
--
Key: MAPREDUCE-1415
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1415
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/streaming, security
Reporter: Vinod K V
Assignee: Amareshwari Sriramadasu
Fix For: 0.22.0
Attachments: patch-1415-1.txt, patch-1415.txt
After MAPREDUCE-856, all localized files are expected to have **0 permissions
for the sake of security.
This was found by Karam while testing LinuxTaskController functionality after
MAPREDUCE-856.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1415) With streaming jobs and LinuxTaskController, the localized streaming binary has 571 permissions instead of 570
[ https://issues.apache.org/jira/browse/MAPREDUCE-1415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12841879#action_12841879 ] Hadoop QA commented on MAPREDUCE-1415: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12437990/patch-1415-1.txt against trunk revision 919335. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/503/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/503/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/503/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/503/console This message is automatically generated. With streaming jobs and LinuxTaskController, the localized streaming binary has 571 permissions instead of 570 -- Key: MAPREDUCE-1415 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1415 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/streaming, security Reporter: Vinod K V Assignee: Amareshwari Sriramadasu Fix For: 0.22.0 Attachments: patch-1415-1.txt, patch-1415.txt After MAPREDUCE-856, all localized files are expected to have **0 permissions for the sake of security. This was found by Karam while testing LinuxTaskController functionality after MAPREDUCE-856. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1415) With streaming jobs and LinuxTaskController, the localized streaming binary has 571 permissions instead of 570
[ https://issues.apache.org/jira/browse/MAPREDUCE-1415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12829616#action_12829616 ] Ravi Gummadi commented on MAPREDUCE-1415: - Recursive call of checkPermissionsOnDir() need to send file instead of dir as first param. As MAPREDUCE-1435 is also adding the same method and few similar methods for checking permissions of private ditributed cache files and public distributed cache files, let us regenerate this patch after that(after commit of MAPREDUCE-1435). With streaming jobs and LinuxTaskController, the localized streaming binary has 571 permissions instead of 570 -- Key: MAPREDUCE-1415 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1415 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/streaming, security Reporter: Vinod K V Assignee: Amareshwari Sriramadasu Fix For: 0.22.0 Attachments: patch-1415.txt After MAPREDUCE-856, all localized files are expected to have **0 permissions for the sake of security. This was found by Karam while testing LinuxTaskController functionality after MAPREDUCE-856. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1415) With streaming jobs and LinuxTaskController, the localized streaming binary has 571 permissions instead of 570
[ https://issues.apache.org/jira/browse/MAPREDUCE-1415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12828111#action_12828111 ] Hadoop QA commented on MAPREDUCE-1415: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12434373/patch-1415.txt against trunk revision 905008. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/421/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/421/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/421/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/421/console This message is automatically generated. With streaming jobs and LinuxTaskController, the localized streaming binary has 571 permissions instead of 570 -- Key: MAPREDUCE-1415 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1415 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/streaming, security Reporter: Vinod K V Assignee: Amareshwari Sriramadasu Fix For: 0.22.0 Attachments: patch-1415.txt After MAPREDUCE-856, all localized files are expected to have **0 permissions for the sake of security. This was found by Karam while testing LinuxTaskController functionality after MAPREDUCE-856. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1415) With streaming jobs and LinuxTaskController, the localized streaming binary has 571 permissions instead of 570
[ https://issues.apache.org/jira/browse/MAPREDUCE-1415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12828435#action_12828435 ] Amareshwari Sriramadasu commented on MAPREDUCE-1415: bq.-1 core tests. TestJobName failed because of NoClassDefFoundError (MAPREDUCE-1275). Same test passed on my machine With streaming jobs and LinuxTaskController, the localized streaming binary has 571 permissions instead of 570 -- Key: MAPREDUCE-1415 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1415 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/streaming, security Reporter: Vinod K V Assignee: Amareshwari Sriramadasu Fix For: 0.22.0 Attachments: patch-1415.txt After MAPREDUCE-856, all localized files are expected to have **0 permissions for the sake of security. This was found by Karam while testing LinuxTaskController functionality after MAPREDUCE-856. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1415) With streaming jobs and LinuxTaskController, the localized streaming binary has 571 permissions instead of 570
[ https://issues.apache.org/jira/browse/MAPREDUCE-1415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12805437#action_12805437 ] Vinod K V commented on MAPREDUCE-1415: -- Note that this still isn't a major security concern as all the parent directories are secure already. This fix is needed so we are consistent with permissions overall. With streaming jobs and LinuxTaskController, the localized streaming binary has 571 permissions instead of 570 -- Key: MAPREDUCE-1415 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1415 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/streaming, security Reporter: Vinod K V After MAPREDUCE-856, all localized files are expected to have **0 permissions for the sake of security. This was found by Karam while testing LinuxTaskController functionality after MAPREDUCE-856. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
