[jira] Updated: (MAPREDUCE-1432) Add the hooks in JobTracker and TaskTracker to load tokens from the token cache into the user's UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-1432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated MAPREDUCE-1432: --- Attachment: 1432.patch Attaching a patch that puts the hooks. I will refactor the tokencache out of mapreduce to common in a follow up patch. > Add the hooks in JobTracker and TaskTracker to load tokens from the token > cache into the user's UGI > --- > > Key: MAPREDUCE-1432 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1432 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Affects Versions: 0.22.0 >Reporter: Devaraj Das >Assignee: Devaraj Das > Fix For: 0.22.0 > > Attachments: 1432.patch > > > Related to HADOOP-6520. Here it is about putting hooks in the > JobTracker/TaskTracker for loading tokens in the user's UGI. This is required > when job files are copied from the HDFS on behalf of the user. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1429) New ant target to run all and only the linux task-controller related tests
[ https://issues.apache.org/jira/browse/MAPREDUCE-1429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806633#action_12806633 ] Hemanth Yamijala commented on MAPREDUCE-1429: - Kos, these tests are a little different from usual tests in that they rely on a binary called task-controller which must be built and configured as a 'setuid' binary and permissions set to be owned by root. They also run tests as a different user from the one firing the ant test target. If you notice these tests take some special parameters - a path to a binary file and a user identity. HADOOP-5771 introduced these tests for the first time into the system. In the ant build scripts, we only build the task-controller binary, but the step of making it a setuid is manually done in environments which use this feature. We did not know of a good way to automate this process on Hudson. And the tests do not make sense without this binary and/or user identity. They essentially exist to make sure the task-controller function works correctly in being able to run tasks as another user. Rather than completely ignoring automated testing because of these various limitations, we thought of still providing a facility by which atleast developers working on these features can have a way of testing them in an automated fashion, even if they mean, they have to run it 'specially'. Essentially, having something was better than having nothing. All this said, we absolutely *hate* not being able to run these tests in the right way on Hudson. I would appreciate any help / suggestions you can offer to fix this issue. > New ant target to run all and only the linux task-controller related tests > -- > > Key: MAPREDUCE-1429 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1429 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build, task-controller, test >Reporter: Vinod K V >Assignee: Vinod K V > Attachments: MAPREDUCE-1429-20100120.txt > > > The LinuxTaskController tests cannot be run automatically by Hudson and so > we've missed several bugs in the past because of not running some of these > tests explicitly ourselves. It's a real pain to run them manually one by one, > we should have an ant target to run them all in one swoop. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1412) TestTaskTrackerBlacklisting fails sometimes
[
https://issues.apache.org/jira/browse/MAPREDUCE-1412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas updated MAPREDUCE-1412:
-
Resolution: Fixed
Fix Version/s: 0.21.0
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I committed this.
> TestTaskTrackerBlacklisting fails sometimes
> ---
>
> Key: MAPREDUCE-1412
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1412
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Affects Versions: 0.21.0
>Reporter: Chris Douglas
>Assignee: Chris Douglas
>Priority: Minor
> Fix For: 0.21.0
>
> Attachments: M1412-0.patch
>
>
> {{TestTaskTrackerBlacklisting}} fails occasionally. The granularity of the
> timer is responsible; the unit test adds a day to the expiration interval to
> verify that the tracker is removed from the blacklist, but the tracker is not
> removed if the interval exactly matches 1 day.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1335) Add SASL DIGEST-MD5 authentication to TaskUmbilicalProtocol
[ https://issues.apache.org/jira/browse/MAPREDUCE-1335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kan Zhang updated MAPREDUCE-1335: - Attachment: m6419-15.patch A new patch that matches the new patch in Common. It also includes a new test on the TaskUmbilicalProtocol itself. > Add SASL DIGEST-MD5 authentication to TaskUmbilicalProtocol > --- > > Key: MAPREDUCE-1335 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1335 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: Kan Zhang >Assignee: Kan Zhang > Attachments: m6419-11.patch, m6419-15.patch > > > Use job token as the credential for Task to local TaskTracker authentication > over RPC. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1367) LocalJobRunner should support parallel mapper execution
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated MAPREDUCE-1367: - Resolution: Fixed Fix Version/s: 0.22.0 Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I've just committed this. Thanks Aaron! > LocalJobRunner should support parallel mapper execution > --- > > Key: MAPREDUCE-1367 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1367.2.patch, MAPREDUCE-1367.3.patch, > MAPREDUCE-1367.4.patch, MAPREDUCE-1367.5.patch, MAPREDUCE-1367.6.patch, > MAPREDUCE-1367.7.patch, MAPREDUCE-1367.patch > > > The LocalJobRunner currently supports only a single execution thread. Given > the prevalence of multi-core CPUs, it makes sense to allow users to run > multiple tasks in parallel for improved performance on small (local-only) > jobs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1367) LocalJobRunner should support parallel mapper execution
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806590#action_12806590 ] Aaron Kimball commented on MAPREDUCE-1367: -- Test failures seem unrelated - classpath issues on hudson again? > LocalJobRunner should support parallel mapper execution > --- > > Key: MAPREDUCE-1367 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: MAPREDUCE-1367.2.patch, MAPREDUCE-1367.3.patch, > MAPREDUCE-1367.4.patch, MAPREDUCE-1367.5.patch, MAPREDUCE-1367.6.patch, > MAPREDUCE-1367.7.patch, MAPREDUCE-1367.patch > > > The LocalJobRunner currently supports only a single execution thread. Given > the prevalence of multi-core CPUs, it makes sense to allow users to run > multiple tasks in parallel for improved performance on small (local-only) > jobs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1367) LocalJobRunner should support parallel mapper execution
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806585#action_12806585 ] Hadoop QA commented on MAPREDUCE-1367: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12431823/MAPREDUCE-1367.7.patch against trunk revision 904609. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/418/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/418/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/418/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/418/console This message is automatically generated. > LocalJobRunner should support parallel mapper execution > --- > > Key: MAPREDUCE-1367 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: MAPREDUCE-1367.2.patch, MAPREDUCE-1367.3.patch, > MAPREDUCE-1367.4.patch, MAPREDUCE-1367.5.patch, MAPREDUCE-1367.6.patch, > MAPREDUCE-1367.7.patch, MAPREDUCE-1367.patch > > > The LocalJobRunner currently supports only a single execution thread. Given > the prevalence of multi-core CPUs, it makes sense to allow users to run > multiple tasks in parallel for improved performance on small (local-only) > jobs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1412) TestTaskTrackerBlacklisting fails sometimes
[
https://issues.apache.org/jira/browse/MAPREDUCE-1412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806572#action_12806572
]
Arun C Murthy commented on MAPREDUCE-1412:
--
+1
> TestTaskTrackerBlacklisting fails sometimes
> ---
>
> Key: MAPREDUCE-1412
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1412
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Affects Versions: 0.21.0
>Reporter: Chris Douglas
>Assignee: Chris Douglas
>Priority: Minor
> Attachments: M1412-0.patch
>
>
> {{TestTaskTrackerBlacklisting}} fails occasionally. The granularity of the
> timer is responsible; the unit test adds a day to the expiration interval to
> verify that the tracker is removed from the blacklist, but the tracker is not
> removed if the interval exactly matches 1 day.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1425) archive throws OutOfMemoryError
[
https://issues.apache.org/jira/browse/MAPREDUCE-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo (Nicholas), SZE updated MAPREDUCE-1425:
--
Attachment: har.sh
m1425_20100129TextFileGenerator.patch
m1425_20100129TextFileGenerator.patch: a program to generate text files.
har.sh: useful commands for testing archive.
> archive throws OutOfMemoryError
> ---
>
> Key: MAPREDUCE-1425
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1425
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: harchive
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Mahadev konar
> Fix For: 0.22.0
>
> Attachments: har.sh, m1425_20100129TextFileGenerator.patch
>
>
> {noformat}
> -bash-3.1$ hadoop archive -archiveName t4.har -p . t4 .
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at java.util.regex.Pattern.compile(Pattern.java:1432)
> at java.util.regex.Pattern.(Pattern.java:1133)
> at java.util.regex.Pattern.compile(Pattern.java:847)
> at java.lang.String.replace(String.java:2208)
> at org.apache.hadoop.fs.Path.normalizePath(Path.java:146)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> at org.apache.hadoop.fs.Path.(Path.java:126)
> at org.apache.hadoop.fs.Path.makeQualified(Path.java:296)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.makeQualified(DistributedFileSystem.java:244)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:256)
> at
> org.apache.hadoop.tools.HadoopArchives.archive(HadoopArchives.java:393)
> at org.apache.hadoop.tools.HadoopArchives.run(HadoopArchives.java:736)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at
> org.apache.hadoop.tools.HadoopArchives.main(HadoopArchives.java:751)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1432) Add the hooks in JobTracker and TaskTracker to load tokens from the token cache into the user's UGI
Add the hooks in JobTracker and TaskTracker to load tokens from the token cache into the user's UGI --- Key: MAPREDUCE-1432 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1432 Project: Hadoop Map/Reduce Issue Type: New Feature Affects Versions: 0.22.0 Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.22.0 Related to HADOOP-6520. Here it is about putting hooks in the JobTracker/TaskTracker for loading tokens in the user's UGI. This is required when job files are copied from the HDFS on behalf of the user. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1367) LocalJobRunner should support parallel mapper execution
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Kimball updated MAPREDUCE-1367: - Status: Patch Available (was: Open) > LocalJobRunner should support parallel mapper execution > --- > > Key: MAPREDUCE-1367 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: MAPREDUCE-1367.2.patch, MAPREDUCE-1367.3.patch, > MAPREDUCE-1367.4.patch, MAPREDUCE-1367.5.patch, MAPREDUCE-1367.6.patch, > MAPREDUCE-1367.7.patch, MAPREDUCE-1367.patch > > > The LocalJobRunner currently supports only a single execution thread. Given > the prevalence of multi-core CPUs, it makes sense to allow users to run > multiple tasks in parallel for improved performance on small (local-only) > jobs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1367) LocalJobRunner should support parallel mapper execution
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Kimball updated MAPREDUCE-1367: - Attachment: MAPREDUCE-1367.7.patch new patch, sync'd with trunk. > LocalJobRunner should support parallel mapper execution > --- > > Key: MAPREDUCE-1367 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: MAPREDUCE-1367.2.patch, MAPREDUCE-1367.3.patch, > MAPREDUCE-1367.4.patch, MAPREDUCE-1367.5.patch, MAPREDUCE-1367.6.patch, > MAPREDUCE-1367.7.patch, MAPREDUCE-1367.patch > > > The LocalJobRunner currently supports only a single execution thread. Given > the prevalence of multi-core CPUs, it makes sense to allow users to run > multiple tasks in parallel for improved performance on small (local-only) > jobs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1367) LocalJobRunner should support parallel mapper execution
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Kimball updated MAPREDUCE-1367: - Status: Open (was: Patch Available) > LocalJobRunner should support parallel mapper execution > --- > > Key: MAPREDUCE-1367 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: MAPREDUCE-1367.2.patch, MAPREDUCE-1367.3.patch, > MAPREDUCE-1367.4.patch, MAPREDUCE-1367.5.patch, MAPREDUCE-1367.6.patch, > MAPREDUCE-1367.7.patch, MAPREDUCE-1367.patch > > > The LocalJobRunner currently supports only a single execution thread. Given > the prevalence of multi-core CPUs, it makes sense to allow users to run > multiple tasks in parallel for improved performance on small (local-only) > jobs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1431) archive does not work with distcp -update
[
https://issues.apache.org/jira/browse/MAPREDUCE-1431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806509#action_12806509
]
Mahadev konar commented on MAPREDUCE-1431:
--
good catch. ill fix that...
> archive does not work with distcp -update
> -
>
> Key: MAPREDUCE-1431
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1431
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: harchive
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Mahadev konar
> Fix For: 0.22.0
>
>
> The following distcp command works.
> {noformat}
> hadoop distcp -Dmapred.job.queue.name=q
> har://hdfs-nn_hostname:8020/user/tsz/t101.har/t101 t101_distcp
> {noformat}
> However, it does not work for -update.
> {noformat}
> -bash-3.1$ hadoop distcp -Dmapred.job.queue.name=q -update
> har://hdfs-nn_hostname:8020/user/tsz/t101.har/t101 t101_distcp
> 10/01/29 20:06:53 INFO tools.DistCp:
> srcPaths=[har://hdfs-nn_hostname:8020/user/tsz/t101.har/t101]
> 10/01/29 20:06:53 INFO tools.DistCp: destPath=t101
> java.lang.IllegalArgumentException: Wrong FS:
> har://hdfs-nn_hostname:8020/user/tsz/t101.har/t101/text-, expected:
> hdfs://nn_hostname
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:310)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.checkPath(DistributedFileSystem.java:99)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:155)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileChecksum(DistributedFileSystem.java:463)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileChecksum(DistributedFileSystem.java:46)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileChecksum(FilterFileSystem.java:250)
> at org.apache.hadoop.tools.DistCp.sameFile(DistCp.java:1204)
> at org.apache.hadoop.tools.DistCp.setup(DistCp.java:1084)
> ...
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1431) archive does not work with distcp -update
[
https://issues.apache.org/jira/browse/MAPREDUCE-1431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806502#action_12806502
]
Tsz Wo (Nicholas), SZE commented on MAPREDUCE-1431:
---
No. Distcp works if src and dest are in different filesystem schemes.
I briefly checked the codes. It seems that
harfs.listPath("har://hdfs-nn_hostname:8020/user/tsz/t101.har/t101") returns
file statuses with hdfs://, e.g.
hdfs://hostname:8020/user/tsz/t101.har/t101/file1.
> archive does not work with distcp -update
> -
>
> Key: MAPREDUCE-1431
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1431
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: harchive
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Mahadev konar
> Fix For: 0.22.0
>
>
> The following distcp command works.
> {noformat}
> hadoop distcp -Dmapred.job.queue.name=q
> har://hdfs-nn_hostname:8020/user/tsz/t101.har/t101 t101_distcp
> {noformat}
> However, it does not work for -update.
> {noformat}
> -bash-3.1$ hadoop distcp -Dmapred.job.queue.name=q -update
> har://hdfs-nn_hostname:8020/user/tsz/t101.har/t101 t101_distcp
> 10/01/29 20:06:53 INFO tools.DistCp:
> srcPaths=[har://hdfs-nn_hostname:8020/user/tsz/t101.har/t101]
> 10/01/29 20:06:53 INFO tools.DistCp: destPath=t101
> java.lang.IllegalArgumentException: Wrong FS:
> har://hdfs-nn_hostname:8020/user/tsz/t101.har/t101/text-, expected:
> hdfs://nn_hostname
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:310)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.checkPath(DistributedFileSystem.java:99)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:155)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileChecksum(DistributedFileSystem.java:463)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileChecksum(DistributedFileSystem.java:46)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileChecksum(FilterFileSystem.java:250)
> at org.apache.hadoop.tools.DistCp.sameFile(DistCp.java:1204)
> at org.apache.hadoop.tools.DistCp.setup(DistCp.java:1084)
> ...
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1431) archive does not work with distcp -update
[
https://issues.apache.org/jira/browse/MAPREDUCE-1431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mahadev konar updated MAPREDUCE-1431:
-
Fix Version/s: 0.22.0
Looks like this might be a problem with distcp whcih requires the src and dest
to be the same filesystem?
> archive does not work with distcp -update
> -
>
> Key: MAPREDUCE-1431
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1431
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: harchive
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Mahadev konar
> Fix For: 0.22.0
>
>
> The following distcp command works.
> {noformat}
> hadoop distcp -Dmapred.job.queue.name=q
> har://hdfs-nn_hostname:8020/user/tsz/t101.har/t101 t101_distcp
> {noformat}
> However, it does not work for -update.
> {noformat}
> -bash-3.1$ hadoop distcp -Dmapred.job.queue.name=q -update
> har://hdfs-nn_hostname:8020/user/tsz/t101.har/t101 t101_distcp
> 10/01/29 20:06:53 INFO tools.DistCp:
> srcPaths=[har://hdfs-nn_hostname:8020/user/tsz/t101.har/t101]
> 10/01/29 20:06:53 INFO tools.DistCp: destPath=t101
> java.lang.IllegalArgumentException: Wrong FS:
> har://hdfs-nn_hostname:8020/user/tsz/t101.har/t101/text-, expected:
> hdfs://nn_hostname
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:310)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.checkPath(DistributedFileSystem.java:99)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:155)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileChecksum(DistributedFileSystem.java:463)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileChecksum(DistributedFileSystem.java:46)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileChecksum(FilterFileSystem.java:250)
> at org.apache.hadoop.tools.DistCp.sameFile(DistCp.java:1204)
> at org.apache.hadoop.tools.DistCp.setup(DistCp.java:1084)
> ...
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1431) archive does not work with distcp -update
archive does not work with distcp -update
-
Key: MAPREDUCE-1431
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1431
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: harchive
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Mahadev konar
The following distcp command works.
{noformat}
hadoop distcp -Dmapred.job.queue.name=q
har://hdfs-nn_hostname:8020/user/tsz/t101.har/t101 t101_distcp
{noformat}
However, it does not work for -update.
{noformat}
-bash-3.1$ hadoop distcp -Dmapred.job.queue.name=q -update
har://hdfs-nn_hostname:8020/user/tsz/t101.har/t101 t101_distcp
10/01/29 20:06:53 INFO tools.DistCp:
srcPaths=[har://hdfs-nn_hostname:8020/user/tsz/t101.har/t101]
10/01/29 20:06:53 INFO tools.DistCp: destPath=t101
java.lang.IllegalArgumentException: Wrong FS:
har://hdfs-nn_hostname:8020/user/tsz/t101.har/t101/text-, expected:
hdfs://nn_hostname
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:310)
at
org.apache.hadoop.hdfs.DistributedFileSystem.checkPath(DistributedFileSystem.java:99)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:155)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileChecksum(DistributedFileSystem.java:463)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileChecksum(DistributedFileSystem.java:46)
at
org.apache.hadoop.fs.FilterFileSystem.getFileChecksum(FilterFileSystem.java:250)
at org.apache.hadoop.tools.DistCp.sameFile(DistCp.java:1204)
at org.apache.hadoop.tools.DistCp.setup(DistCp.java:1084)
...
{noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1430) JobTracker should be able to renew delegation tokens for the jobs
[ https://issues.apache.org/jira/browse/MAPREDUCE-1430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806477#action_12806477 ] Boris Shkolnik commented on MAPREDUCE-1430: --- We may need to add an expiration date to the o.a.h.security.Token class Created HADOOP-6523 for this. > JobTracker should be able to renew delegation tokens for the jobs > - > > Key: MAPREDUCE-1430 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1430 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: jobtracker >Affects Versions: 0.22.0 >Reporter: Devaraj Das >Assignee: Boris Shkolnik > Fix For: 0.22.0 > > > JobTracker should automatically renew delegation tokens for the jobs it is > currently running. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1126) shuffle should use serialization to get comparator
[ https://issues.apache.org/jira/browse/MAPREDUCE-1126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806472#action_12806472 ] Scott Carey commented on MAPREDUCE-1126: b...@scott: the annotations for Input/OutputFormat seem to be misplaced. It seems desirable to be able to write a single Map function that does wordcount on Strings, regardless of whether those strings are stored in newline-delimited text, sequence files, avro data files, or whatever. Philip, yes, they are not in the right place. I just wanted to bring into the conversation that 'SomeObject.setSomeBinding()' is not the only way to do these sort of things. Annotations, unlike setter methods, can be moved around and adapted to work in various ways without breaking APIs. For example, the Input/OutputFormat annotation could go on either a Map class, OR some other more specific annotation site, and with defaults and priority (set on configuration > annotated on configuration > annotated on map > default) determining which applies. After thinking about it a bit more, and doing some research into how other APIs do some tricky things with Annotations, there are a few things to consider. * It is possible in some situations to infer the generic types of a class at runtime by constructing an instance of an object with the same type arguments. Example: http://wiki.fasterxml.com/JacksonInFiveMinutes#Data_Binding_with_Generics. * Annotations on class A can be applied to class B "Mix-In Annotations"; http://wiki.fasterxml.com/JacksonMixInAnnotations * Post-compile time checks via an annotation processor can validate code before run time in cases where the current M/R framework only breaks at run time. What I think is most important to this discussion is that some layers of configuration complexity can be hidden from users, and some of it deferred to the future. The 'site' of the configuration can be moved around with Annotations, opening up ways to simplify the steps required to do declarative configuration. With this in mind, some additional complexity to the procedural configuration methods is more acceptable if there are good defaults and a later (backwards compatible) API addition simplifies things. Likewise, some elements of complexity can be skipped for now if it can be seen that those could be available through a configuration extension later. Perhaps the procedural API would never allow configuring a key and value to use different serializers to avoid API complexity, but an annotation extension in the future allows that. > shuffle should use serialization to get comparator > -- > > Key: MAPREDUCE-1126 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1126 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: task >Reporter: Doug Cutting >Assignee: Aaron Kimball > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1126.2.patch, MAPREDUCE-1126.3.patch, > MAPREDUCE-1126.4.patch, MAPREDUCE-1126.5.patch, MAPREDUCE-1126.6.patch, > MAPREDUCE-1126.patch, MAPREDUCE-1126.patch > > > Currently the key comparator is defined as a Java class. Instead we should > use the Serialization API to create key comparators. This would permit, > e.g., Avro-based comparators to be used, permitting efficient sorting of > complex data types without having to write a RawComparator in Java. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1383) Allow storage and caching of delegation token.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated MAPREDUCE-1383: --- Resolution: Fixed Fix Version/s: 0.22.0 Assignee: Boris Shkolnik Status: Resolved (was: Patch Available) I just committed this. Thanks, Boris! > Allow storage and caching of delegation token. > -- > > Key: MAPREDUCE-1383 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1383 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: Boris Shkolnik >Assignee: Boris Shkolnik > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1383-1.patch, MAPREDUCE-1383-10.patch, > MAPREDUCE-1383-11.patch, MAPREDUCE-1383-2.patch, MAPREDUCE-1383-5.patch, > MAPREDUCE-1383-6.patch, MAPREDUCE-1383-9.patch, MAPREDUCE-1383-BP20-3.patch, > MAPREDUCE-1383-BP20-4.patch > > > Client needs to obtain delegation tokens from all the NameNodes it is going > to work with and pass it to the application. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1383) Allow storage and caching of delegation token.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik updated MAPREDUCE-1383: -- Attachment: MAPREDUCE-1383-BP20-4.patch same change (ip:port) for backport. > Allow storage and caching of delegation token. > -- > > Key: MAPREDUCE-1383 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1383 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: Boris Shkolnik > Attachments: MAPREDUCE-1383-1.patch, MAPREDUCE-1383-10.patch, > MAPREDUCE-1383-11.patch, MAPREDUCE-1383-2.patch, MAPREDUCE-1383-5.patch, > MAPREDUCE-1383-6.patch, MAPREDUCE-1383-9.patch, MAPREDUCE-1383-BP20-3.patch, > MAPREDUCE-1383-BP20-4.patch > > > Client needs to obtain delegation tokens from all the NameNodes it is going > to work with and pass it to the application. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1430) JobTracker should be able to renew delegation tokens for the jobs
JobTracker should be able to renew delegation tokens for the jobs - Key: MAPREDUCE-1430 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1430 Project: Hadoop Map/Reduce Issue Type: New Feature Components: jobtracker Affects Versions: 0.22.0 Reporter: Devaraj Das Assignee: Boris Shkolnik Fix For: 0.22.0 JobTracker should automatically renew delegation tokens for the jobs it is currently running. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1369) JUnit tests should never depend on anything in conf
[ https://issues.apache.org/jira/browse/MAPREDUCE-1369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated MAPREDUCE-1369: -- Affects Version/s: (was: 0.20.2) 0.21.0 Fix Version/s: 0.21.0 > JUnit tests should never depend on anything in conf > --- > > Key: MAPREDUCE-1369 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1369 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 0.21.0, 0.22.0 >Reporter: Anatoli Fomenko >Assignee: Anatoli Fomenko >Priority: Blocker > Fix For: 0.21.0, 0.22.0 > > Attachments: MAPREDUCE-1369.patch > > > The recent change to mapred-queues.xml that causes many mapreduce tests to > break unless you delete conf/mapred-queues.xml out of your build tree is bad. > We need to make sure that nothing in conf is used in the unit tests. One > potential solution is to copy the templates into build/test/conf and use that > instead. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1383) Allow storage and caching of delegation token.
[
https://issues.apache.org/jira/browse/MAPREDUCE-1383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Boris Shkolnik updated MAPREDUCE-1383:
--
Attachment: MAPREDUCE-1383-11.patch
bq.The "/IP:PORT" looks odd. Please change the service to be
:PORT.
After discussion with Kan, we aggreed on "IP:PORT". Kan will match in his
implementation.
here is my diff:
< -
sb.append(NetUtils.normalizeHostName(uri.getHost())).append(":").append(port);
---
> +
> sb.append("/").append(NetUtils.normalizeHostName(uri.getHost())).append(":").append(port);
> Allow storage and caching of delegation token.
> --
>
> Key: MAPREDUCE-1383
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1383
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
>Reporter: Boris Shkolnik
> Attachments: MAPREDUCE-1383-1.patch, MAPREDUCE-1383-10.patch,
> MAPREDUCE-1383-11.patch, MAPREDUCE-1383-2.patch, MAPREDUCE-1383-5.patch,
> MAPREDUCE-1383-6.patch, MAPREDUCE-1383-9.patch, MAPREDUCE-1383-BP20-3.patch
>
>
> Client needs to obtain delegation tokens from all the NameNodes it is going
> to work with and pass it to the application.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1369) JUnit tests should never depend on anything in conf
[ https://issues.apache.org/jira/browse/MAPREDUCE-1369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated MAPREDUCE-1369: -- Affects Version/s: 0.22.0 0.20.2 > JUnit tests should never depend on anything in conf > --- > > Key: MAPREDUCE-1369 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1369 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 0.20.2, 0.22.0 >Reporter: Anatoli Fomenko >Assignee: Anatoli Fomenko >Priority: Blocker > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1369.patch > > > The recent change to mapred-queues.xml that causes many mapreduce tests to > break unless you delete conf/mapred-queues.xml out of your build tree is bad. > We need to make sure that nothing in conf is used in the unit tests. One > potential solution is to copy the templates into build/test/conf and use that > instead. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1429) New ant target to run all and only the linux task-controller related tests
[ https://issues.apache.org/jira/browse/MAPREDUCE-1429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806423#action_12806423 ] Konstantin Boudnik commented on MAPREDUCE-1429: --- I wonder why these tests haven't been executed by Hudson before? They seem to be a normal kind of tests located within the larger 'mapred' test suite. Also, I kinda don't like the idea of have a top level target for a very narrow subset of tests. What current approach does, basically, is a creation of an exclusive subset from a larger set of tests and giving them a privilege of a test suite. Doesn't look right to me. If they are so truly special then they need to be moved into a top-level test suite like 'unit' or 'mapred', aren't they? > New ant target to run all and only the linux task-controller related tests > -- > > Key: MAPREDUCE-1429 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1429 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build, task-controller, test >Reporter: Vinod K V >Assignee: Vinod K V > Attachments: MAPREDUCE-1429-20100120.txt > > > The LinuxTaskController tests cannot be run automatically by Hudson and so > we've missed several bugs in the past because of not running some of these > tests explicitly ourselves. It's a real pain to run them manually one by one, > we should have an ant target to run them all in one swoop. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-899) When using LinuxTaskController, localized files may become accessible to unintended users if permissions are misconfigured.
[ https://issues.apache.org/jira/browse/MAPREDUCE-899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806382#action_12806382 ] Hadoop QA commented on MAPREDUCE-899: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12431772/testplan.txt against trunk revision 904491. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/295/console This message is automatically generated. > When using LinuxTaskController, localized files may become accessible to > unintended users if permissions are misconfigured. > --- > > Key: MAPREDUCE-899 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-899 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Vinod K V >Assignee: Amareshwari Sriramadasu > Fix For: 0.22.0 > > Attachments: MAPREDUCE-899-20090828.txt, patch-899-1.txt, > patch-899-2.txt, patch-899-3.txt, patch-899-4.txt, patch-899-5.txt, > patch-899-6.txt, patch-899.txt, testplan.txt > > > To enforce the accessibility of job files to only the job-owner and the > TaskTracker, as per MAPREDUCE-842, it is _trusted_ that the setuid/setgid > linux TaskController binary is group owned by a _special group_ to which only > TaskTracker belongs and not just any group to which TT belongs. If the trust > is broken, possibly due to misconfiguration by admins, the local files become > accessible to unintended users, yet giving false sense of security to the > admins. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1429) New ant target to run all and only the linux task-controller related tests
[ https://issues.apache.org/jira/browse/MAPREDUCE-1429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806351#action_12806351 ] Hadoop QA commented on MAPREDUCE-1429: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12431766/MAPREDUCE-1429-20100120.txt against trunk revision 904397. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 19 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/417/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/417/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/417/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/417/console This message is automatically generated. > New ant target to run all and only the linux task-controller related tests > -- > > Key: MAPREDUCE-1429 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1429 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build, task-controller, test >Reporter: Vinod K V >Assignee: Vinod K V > Attachments: MAPREDUCE-1429-20100120.txt > > > The LinuxTaskController tests cannot be run automatically by Hudson and so > we've missed several bugs in the past because of not running some of these > tests explicitly ourselves. It's a real pain to run them manually one by one, > we should have an ant target to run them all in one swoop. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-899) When using LinuxTaskController, localized files may become accessible to unintended users if permissions are misconfigured.
[ https://issues.apache.org/jira/browse/MAPREDUCE-899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-899: -- Attachment: testplan.txt bq. I'd request you to document as a comment on this JIRA the various manual tests you have run with different binary permissions / ownership. Attaching the testplan. Ran all LinuxTaskController tests. Test failures are all known. > When using LinuxTaskController, localized files may become accessible to > unintended users if permissions are misconfigured. > --- > > Key: MAPREDUCE-899 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-899 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Vinod K V >Assignee: Amareshwari Sriramadasu > Fix For: 0.22.0 > > Attachments: MAPREDUCE-899-20090828.txt, patch-899-1.txt, > patch-899-2.txt, patch-899-3.txt, patch-899-4.txt, patch-899-5.txt, > patch-899-6.txt, patch-899.txt, testplan.txt > > > To enforce the accessibility of job files to only the job-owner and the > TaskTracker, as per MAPREDUCE-842, it is _trusted_ that the setuid/setgid > linux TaskController binary is group owned by a _special group_ to which only > TaskTracker belongs and not just any group to which TT belongs. If the trust > is broken, possibly due to misconfiguration by admins, the local files become > accessible to unintended users, yet giving false sense of security to the > admins. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1322) TestStreamingAsDifferentUser fails on trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-1322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hemanth Yamijala updated MAPREDUCE-1322:
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I just committed this to trunk. Thanks, Devaraj and Amareshwari !
> TestStreamingAsDifferentUser fails on trunk
> ---
>
> Key: MAPREDUCE-1322
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1322
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/streaming, test
>Affects Versions: 0.22.0
>Reporter: Amareshwari Sriramadasu
>Assignee: Devaraj Das
> Fix For: 0.22.0
>
> Attachments: 1322-1.patch, 1322.patch
>
>
> TestStreamingAsDifferentUser fails on trunk with following exception :
> Can not create a Path from a null string
> java.lang.IllegalArgumentException: Can not create a Path from a null string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:78)
> at org.apache.hadoop.fs.Path.(Path.java:90)
> at
> org.apache.hadoop.mapred.ClusterWithLinuxTaskController.createHomeAndStagingDirectory(ClusterWithLinuxTaskController.java:158)
> at
> org.apache.hadoop.mapred.ClusterWithLinuxTaskController.startCluster(ClusterWithLinuxTaskController.java:147)
> at
> org.apache.hadoop.streaming.TestStreamingAsDifferentUser.testStreaming(TestStreamingAsDifferentUser.java:49)
> The corresponding line for the exception :
> {code}
> Path stagingArea = new Path(conf.get(JTConfig.JT_STAGING_AREA_ROOT));
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1322) TestStreamingAsDifferentUser fails on trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-1322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806333#action_12806333
]
Hemanth Yamijala commented on MAPREDUCE-1322:
-
+1. Looks fine to me. I will commit this patch.
> TestStreamingAsDifferentUser fails on trunk
> ---
>
> Key: MAPREDUCE-1322
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1322
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/streaming, test
>Affects Versions: 0.22.0
>Reporter: Amareshwari Sriramadasu
>Assignee: Devaraj Das
> Fix For: 0.22.0
>
> Attachments: 1322-1.patch, 1322.patch
>
>
> TestStreamingAsDifferentUser fails on trunk with following exception :
> Can not create a Path from a null string
> java.lang.IllegalArgumentException: Can not create a Path from a null string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:78)
> at org.apache.hadoop.fs.Path.(Path.java:90)
> at
> org.apache.hadoop.mapred.ClusterWithLinuxTaskController.createHomeAndStagingDirectory(ClusterWithLinuxTaskController.java:158)
> at
> org.apache.hadoop.mapred.ClusterWithLinuxTaskController.startCluster(ClusterWithLinuxTaskController.java:147)
> at
> org.apache.hadoop.streaming.TestStreamingAsDifferentUser.testStreaming(TestStreamingAsDifferentUser.java:49)
> The corresponding line for the exception :
> {code}
> Path stagingArea = new Path(conf.get(JTConfig.JT_STAGING_AREA_ROOT));
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-899) When using LinuxTaskController, localized files may become accessible to unintended users if permissions are misconfigured.
[ https://issues.apache.org/jira/browse/MAPREDUCE-899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-899: -- Status: Patch Available (was: Open) > When using LinuxTaskController, localized files may become accessible to > unintended users if permissions are misconfigured. > --- > > Key: MAPREDUCE-899 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-899 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Vinod K V >Assignee: Amareshwari Sriramadasu > Fix For: 0.22.0 > > Attachments: MAPREDUCE-899-20090828.txt, patch-899-1.txt, > patch-899-2.txt, patch-899-3.txt, patch-899-4.txt, patch-899-5.txt, > patch-899-6.txt, patch-899.txt > > > To enforce the accessibility of job files to only the job-owner and the > TaskTracker, as per MAPREDUCE-842, it is _trusted_ that the setuid/setgid > linux TaskController binary is group owned by a _special group_ to which only > TaskTracker belongs and not just any group to which TT belongs. If the trust > is broken, possibly due to misconfiguration by admins, the local files become > accessible to unintended users, yet giving false sense of security to the > admins. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-899) When using LinuxTaskController, localized files may become accessible to unintended users if permissions are misconfigured.
[ https://issues.apache.org/jira/browse/MAPREDUCE-899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-899: -- Attachment: patch-899-6.txt Patch with comments addressed. bq. The wait introduced for in the ClusterWithLinuxTaskController does not seem required, because doesn't MiniMRCluster itself wait for clusters to join ? Verified. The new code is not required. If there are any timeouts because trackers could not come up, MAPREDUCE-1366 should address that. > When using LinuxTaskController, localized files may become accessible to > unintended users if permissions are misconfigured. > --- > > Key: MAPREDUCE-899 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-899 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Vinod K V >Assignee: Amareshwari Sriramadasu > Fix For: 0.22.0 > > Attachments: MAPREDUCE-899-20090828.txt, patch-899-1.txt, > patch-899-2.txt, patch-899-3.txt, patch-899-4.txt, patch-899-5.txt, > patch-899-6.txt, patch-899.txt > > > To enforce the accessibility of job files to only the job-owner and the > TaskTracker, as per MAPREDUCE-842, it is _trusted_ that the setuid/setgid > linux TaskController binary is group owned by a _special group_ to which only > TaskTracker belongs and not just any group to which TT belongs. If the trust > is broken, possibly due to misconfiguration by admins, the local files become > accessible to unintended users, yet giving false sense of security to the > admins. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1421) TestDebugScriptWithLinuxTaskController and TestJobExecutionAsDifferentUser fail on trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-1421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806331#action_12806331 ] Amareshwari Sriramadasu commented on MAPREDUCE-1421: bq. -1 core tests. Is due to MAPREDUCE-1412 > TestDebugScriptWithLinuxTaskController and TestJobExecutionAsDifferentUser > fail on trunk > > > Key: MAPREDUCE-1421 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1421 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: task-controller, tasktracker, test >Affects Versions: 0.22.0 >Reporter: Amareshwari Sriramadasu > Fix For: 0.22.0 > > Attachments: patch-1421-1.txt, patch-1421.txt, > TestJobExecutionAsDifferentUser.patch > > > TestDebugScriptWithLinuxTaskController and TestJobExecutionAsDifferentUser > fail on trunk after the commit of MAPREDUCE-1385 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1322) TestStreamingAsDifferentUser fails on trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-1322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806330#action_12806330
]
Amareshwari Sriramadasu commented on MAPREDUCE-1322:
bq. -1 core tests.
Is due to MAPREDUCE-1412
> TestStreamingAsDifferentUser fails on trunk
> ---
>
> Key: MAPREDUCE-1322
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1322
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/streaming, test
>Affects Versions: 0.22.0
>Reporter: Amareshwari Sriramadasu
>Assignee: Devaraj Das
> Fix For: 0.22.0
>
> Attachments: 1322-1.patch, 1322.patch
>
>
> TestStreamingAsDifferentUser fails on trunk with following exception :
> Can not create a Path from a null string
> java.lang.IllegalArgumentException: Can not create a Path from a null string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:78)
> at org.apache.hadoop.fs.Path.(Path.java:90)
> at
> org.apache.hadoop.mapred.ClusterWithLinuxTaskController.createHomeAndStagingDirectory(ClusterWithLinuxTaskController.java:158)
> at
> org.apache.hadoop.mapred.ClusterWithLinuxTaskController.startCluster(ClusterWithLinuxTaskController.java:147)
> at
> org.apache.hadoop.streaming.TestStreamingAsDifferentUser.testStreaming(TestStreamingAsDifferentUser.java:49)
> The corresponding line for the exception :
> {code}
> Path stagingArea = new Path(conf.get(JTConfig.JT_STAGING_AREA_ROOT));
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1421) TestDebugScriptWithLinuxTaskController and TestJobExecutionAsDifferentUser fail on trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-1421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806324#action_12806324 ] Hadoop QA commented on MAPREDUCE-1421: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12431760/patch-1421-1.txt against trunk revision 904397. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 14 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/294/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/294/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/294/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/294/console This message is automatically generated. > TestDebugScriptWithLinuxTaskController and TestJobExecutionAsDifferentUser > fail on trunk > > > Key: MAPREDUCE-1421 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1421 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: task-controller, tasktracker, test >Affects Versions: 0.22.0 >Reporter: Amareshwari Sriramadasu > Fix For: 0.22.0 > > Attachments: patch-1421-1.txt, patch-1421.txt, > TestJobExecutionAsDifferentUser.patch > > > TestDebugScriptWithLinuxTaskController and TestJobExecutionAsDifferentUser > fail on trunk after the commit of MAPREDUCE-1385 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1429) New ant target to run all and only the linux task-controller related tests
[ https://issues.apache.org/jira/browse/MAPREDUCE-1429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod K V updated MAPREDUCE-1429: - Attachment: MAPREDUCE-1429-20100120.txt Here we go. > New ant target to run all and only the linux task-controller related tests > -- > > Key: MAPREDUCE-1429 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1429 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build, task-controller, test >Reporter: Vinod K V > Attachments: MAPREDUCE-1429-20100120.txt > > > The LinuxTaskController tests cannot be run automatically by Hudson and so > we've missed several bugs in the past because of not running some of these > tests explicitly ourselves. It's a real pain to run them manually one by one, > we should have an ant target to run them all in one swoop. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1429) New ant target to run all and only the linux task-controller related tests
[ https://issues.apache.org/jira/browse/MAPREDUCE-1429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod K V updated MAPREDUCE-1429: - Assignee: Vinod K V Status: Patch Available (was: Open) > New ant target to run all and only the linux task-controller related tests > -- > > Key: MAPREDUCE-1429 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1429 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build, task-controller, test >Reporter: Vinod K V >Assignee: Vinod K V > Attachments: MAPREDUCE-1429-20100120.txt > > > The LinuxTaskController tests cannot be run automatically by Hudson and so > we've missed several bugs in the past because of not running some of these > tests explicitly ourselves. It's a real pain to run them manually one by one, > we should have an ant target to run them all in one swoop. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-899) When using LinuxTaskController, localized files may become accessible to unintended users if permissions are misconfigured.
[ https://issues.apache.org/jira/browse/MAPREDUCE-899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-899: -- Status: Open (was: Patch Available) > When using LinuxTaskController, localized files may become accessible to > unintended users if permissions are misconfigured. > --- > > Key: MAPREDUCE-899 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-899 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Vinod K V >Assignee: Amareshwari Sriramadasu > Fix For: 0.22.0 > > Attachments: MAPREDUCE-899-20090828.txt, patch-899-1.txt, > patch-899-2.txt, patch-899-3.txt, patch-899-4.txt, patch-899-5.txt, > patch-899.txt > > > To enforce the accessibility of job files to only the job-owner and the > TaskTracker, as per MAPREDUCE-842, it is _trusted_ that the setuid/setgid > linux TaskController binary is group owned by a _special group_ to which only > TaskTracker belongs and not just any group to which TT belongs. If the trust > is broken, possibly due to misconfiguration by admins, the local files become > accessible to unintended users, yet giving false sense of security to the > admins. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1322) TestStreamingAsDifferentUser fails on trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-1322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806315#action_12806315
]
Hadoop QA commented on MAPREDUCE-1322:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12431757/1322-1.patch
against trunk revision 904397.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/416/testReport/
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/416/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/416/artifact/trunk/build/test/checkstyle-errors.html
Console output:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/416/console
This message is automatically generated.
> TestStreamingAsDifferentUser fails on trunk
> ---
>
> Key: MAPREDUCE-1322
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1322
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/streaming, test
>Affects Versions: 0.22.0
>Reporter: Amareshwari Sriramadasu
>Assignee: Devaraj Das
> Fix For: 0.22.0
>
> Attachments: 1322-1.patch, 1322.patch
>
>
> TestStreamingAsDifferentUser fails on trunk with following exception :
> Can not create a Path from a null string
> java.lang.IllegalArgumentException: Can not create a Path from a null string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:78)
> at org.apache.hadoop.fs.Path.(Path.java:90)
> at
> org.apache.hadoop.mapred.ClusterWithLinuxTaskController.createHomeAndStagingDirectory(ClusterWithLinuxTaskController.java:158)
> at
> org.apache.hadoop.mapred.ClusterWithLinuxTaskController.startCluster(ClusterWithLinuxTaskController.java:147)
> at
> org.apache.hadoop.streaming.TestStreamingAsDifferentUser.testStreaming(TestStreamingAsDifferentUser.java:49)
> The corresponding line for the exception :
> {code}
> Path stagingArea = new Path(conf.get(JTConfig.JT_STAGING_AREA_ROOT));
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-899) When using LinuxTaskController, localized files may become accessible to unintended users if permissions are misconfigured.
[ https://issues.apache.org/jira/browse/MAPREDUCE-899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806307#action_12806307 ] Hemanth Yamijala commented on MAPREDUCE-899: A few minor comments: - Remove unused variables - tt_uid and tt_gid and associated code. - It will be good to check for user, group first and then permissions. It's a more natural code ordering than what exists currently, which is user, permissions, then group. - Rather than check errno as return of getgrgid, it seems more conventional to check for return value being null. This is what man states anyway for return values of getgrgid. It was not clear to me that errno will be set to non-zero if the GID does not exist. However note that errno can still be captured and used for strerror. - Should we rename mapreduce.tasktracker.taskcontroller.group to a more generic name and purpose like mapreduce.tasktracker.group. We could document that this is the group to which the tasktracker belongs and qualify that if LinuxTaskController is used, it takes more meaning - in that it should be the group owner of the binary etc. - The wait introduced for in the ClusterWithLinuxTaskController does not seem required, because doesn't MiniMRCluster itself wait for clusters to join ? Can you please check if the timeout code you added actually helps ? Thinking about tests, I think it is difficult to actually test all the paths verified by the C code without mocking the system calls or building different binary executables and setting them up with different permissions. While we work out details to improve the way we run LinuxTaskController tests - MAPREDUCE-1429, MAPREDUCE-1300 - I'd request you to document as a comment on this JIRA the various manual tests you have run with different binary permissions / ownership. > When using LinuxTaskController, localized files may become accessible to > unintended users if permissions are misconfigured. > --- > > Key: MAPREDUCE-899 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-899 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Vinod K V >Assignee: Amareshwari Sriramadasu > Fix For: 0.22.0 > > Attachments: MAPREDUCE-899-20090828.txt, patch-899-1.txt, > patch-899-2.txt, patch-899-3.txt, patch-899-4.txt, patch-899-5.txt, > patch-899.txt > > > To enforce the accessibility of job files to only the job-owner and the > TaskTracker, as per MAPREDUCE-842, it is _trusted_ that the setuid/setgid > linux TaskController binary is group owned by a _special group_ to which only > TaskTracker belongs and not just any group to which TT belongs. If the trust > is broken, possibly due to misconfiguration by admins, the local files become > accessible to unintended users, yet giving false sense of security to the > admins. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1370) TestCombineFileInputFormat.testSplitPlacement fails in trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-1370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated MAPREDUCE-1370: - Summary: TestCombineFileInputFormat.testSplitPlacement fails in trunk (was: org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat.testSplitPlacement fails in trunk) > TestCombineFileInputFormat.testSplitPlacement fails in trunk > > > Key: MAPREDUCE-1370 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1370 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 0.22.0 >Reporter: Iyappan Srinivasan > > http://hudson.zones.apache.org/hudson/view/Hadoop/job/Hadoop-Mapreduce-trunk/202/ > org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat.testSplitPlacement > (from TestCombineFileInputFormat) > Failing for the past 1 build (Since #202 ) > Took 18 sec. > add description > Error Message > port out of range:-1 > Stacktrace > java.lang.IllegalArgumentException: port out of range:-1 > at java.net.InetSocketAddress.(InetSocketAddress.java:118) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:377) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.activate(NameNode.java:317) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:308) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:416) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:410) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1230) > at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:290) > at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:153) > at > org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat.testSplitPlacement(TestCombineFileInputFormat.java:95) > raw output format: > [junit] Running > org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat > [junit] 2010-01-11 15:05:29,813 INFO namenode.FSNamesystem > (BlockManager.java:setConfigurationParameters(166)) - defaultReplication = 1 > [junit] 2010-01-11 15:05:29,816 INFO namenode.FSNamesystem > (BlockManager.java:setConfigurationParameters(167)) - maxReplication = 512 > [junit] 2010-01-11 15:05:29,817 INFO namenode.FSNamesystem > (BlockManager.java:setConfigurationParameters(168)) - minReplication = 1 > [junit] 2010-01-11 15:05:29,817 INFO namenode.FSNamesystem > (BlockManager.java:setConfigurationParameters(169)) - maxReplicationStreams = > 2 > [junit] 2010-01-11 15:05:29,818 INFO namenode.FSNamesystem > (BlockManager.java:setConfigurationParameters(170)) - > shouldCheckForEnoughRacks = false > [junit] 2010-01-11 15:05:29,826 INFO namenode.FSNamesystem > (FSNamesystem.java:setConfigurationParameters(389)) - fsOwner=hudson,hudson > [junit] 2010-01-11 15:05:29,827 INFO namenode.FSNamesystem > (FSNamesystem.java:setConfigurationParameters(395)) - supergroup=supergroup > [junit] 2010-01-11 15:05:29,827 INFO namenode.FSNamesystem > (FSNamesystem.java:setConfigurationParameters(396)) - isPermissionEnabled=true > [junit] 2010-01-11 15:05:29,831 INFO namenode.FSNamesystem > (FSNamesystem.java:setConfigurationParameters(434)) - > isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), > accessTokenLifetime=0 min(s) > [junit] 2010-01-11 15:05:29,864 INFO common.Storage > (FSImage.java:saveFSImage(1218)) - Image file of size 96 saved in 0 seconds. > [junit] 2010-01-11 15:05:29,890 INFO common.Storage > (FSImage.java:format(1289)) - Storage directory > /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build/test/data/dfs/name1 > has been successfully formatted. > [junit] 2010-01-11 15:05:29,891 INFO common.Storage > (FSImage.java:saveFSImage(1218)) - Image file of size 96 saved in 0 seconds. > [junit] 2010-01-11 15:05:29,905 INFO common.Storage > (FSImage.java:format(1289)) - Storage directory > /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build/test/data/dfs/name2 > has been successfully formatted. > [junit] 2010-01-11 15:05:29,933 INFO metrics.RpcMetrics > (RpcMetrics.java:(58)) - Initializing RPC Metrics with > hostName=NameNode, port=50403 > [junit] 2010-01-11 15:05:29,983 INFO jvm.JvmMetrics > (JvmMetrics.java:init(71)) - Initializing JVM Metrics with > processName=NameNode, sessionId=null > [junit] 2010-01-11 15:05:29,984 INFO metrics.NameNodeMetrics > (NameNodeMetrics.java:(103)) - Initializing NameNodeMeterics using > context object:org.apache.hadoop.metrics.spi.NullContext > [junit] 2010-01-11 15:05:29,985 INFO namenode.FSNamesystem > (BlockManager.java:setConfigurationParameters(166))
[jira] Commented: (MAPREDUCE-1429) New ant target to run all and only the linux task-controller related tests
[ https://issues.apache.org/jira/browse/MAPREDUCE-1429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806286#action_12806286 ] Arun C Murthy commented on MAPREDUCE-1429: -- +1 > New ant target to run all and only the linux task-controller related tests > -- > > Key: MAPREDUCE-1429 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1429 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build, task-controller, test >Reporter: Vinod K V > > The LinuxTaskController tests cannot be run automatically by Hudson and so > we've missed several bugs in the past because of not running some of these > tests explicitly ourselves. It's a real pain to run them manually one by one, > we should have an ant target to run them all in one swoop. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1421) TestDebugScriptWithLinuxTaskController and TestJobExecutionAsDifferentUser fail on trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-1421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1421: --- Status: Patch Available (was: Open) > TestDebugScriptWithLinuxTaskController and TestJobExecutionAsDifferentUser > fail on trunk > > > Key: MAPREDUCE-1421 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1421 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: task-controller, tasktracker, test >Affects Versions: 0.22.0 >Reporter: Amareshwari Sriramadasu > Fix For: 0.22.0 > > Attachments: patch-1421-1.txt, patch-1421.txt, > TestJobExecutionAsDifferentUser.patch > > > TestDebugScriptWithLinuxTaskController and TestJobExecutionAsDifferentUser > fail on trunk after the commit of MAPREDUCE-1385 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1421) TestDebugScriptWithLinuxTaskController and TestJobExecutionAsDifferentUser fail on trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-1421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1421: --- Attachment: patch-1421-1.txt Patch fixing TestPipes also > TestDebugScriptWithLinuxTaskController and TestJobExecutionAsDifferentUser > fail on trunk > > > Key: MAPREDUCE-1421 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1421 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: task-controller, tasktracker, test >Affects Versions: 0.22.0 >Reporter: Amareshwari Sriramadasu > Fix For: 0.22.0 > > Attachments: patch-1421-1.txt, patch-1421.txt, > TestJobExecutionAsDifferentUser.patch > > > TestDebugScriptWithLinuxTaskController and TestJobExecutionAsDifferentUser > fail on trunk after the commit of MAPREDUCE-1385 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1421) TestDebugScriptWithLinuxTaskController and TestJobExecutionAsDifferentUser fail on trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-1421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1421: --- Status: Open (was: Patch Available) > TestDebugScriptWithLinuxTaskController and TestJobExecutionAsDifferentUser > fail on trunk > > > Key: MAPREDUCE-1421 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1421 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: task-controller, tasktracker, test >Affects Versions: 0.22.0 >Reporter: Amareshwari Sriramadasu > Fix For: 0.22.0 > > Attachments: patch-1421.txt, TestJobExecutionAsDifferentUser.patch > > > TestDebugScriptWithLinuxTaskController and TestJobExecutionAsDifferentUser > fail on trunk after the commit of MAPREDUCE-1385 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1429) New ant target to run all and only the linux task-controller related tests
New ant target to run all and only the linux task-controller related tests -- Key: MAPREDUCE-1429 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1429 Project: Hadoop Map/Reduce Issue Type: Bug Components: build, task-controller, test Reporter: Vinod K V The LinuxTaskController tests cannot be run automatically by Hudson and so we've missed several bugs in the past because of not running some of these tests explicitly ourselves. It's a real pain to run them manually one by one, we should have an ant target to run them all in one swoop. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1322) TestStreamingAsDifferentUser fails on trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-1322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amareshwari Sriramadasu updated MAPREDUCE-1322:
---
Status: Patch Available (was: Open)
> TestStreamingAsDifferentUser fails on trunk
> ---
>
> Key: MAPREDUCE-1322
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1322
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/streaming, test
>Affects Versions: 0.22.0
>Reporter: Amareshwari Sriramadasu
>Assignee: Devaraj Das
> Fix For: 0.22.0
>
> Attachments: 1322-1.patch, 1322.patch
>
>
> TestStreamingAsDifferentUser fails on trunk with following exception :
> Can not create a Path from a null string
> java.lang.IllegalArgumentException: Can not create a Path from a null string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:78)
> at org.apache.hadoop.fs.Path.(Path.java:90)
> at
> org.apache.hadoop.mapred.ClusterWithLinuxTaskController.createHomeAndStagingDirectory(ClusterWithLinuxTaskController.java:158)
> at
> org.apache.hadoop.mapred.ClusterWithLinuxTaskController.startCluster(ClusterWithLinuxTaskController.java:147)
> at
> org.apache.hadoop.streaming.TestStreamingAsDifferentUser.testStreaming(TestStreamingAsDifferentUser.java:49)
> The corresponding line for the exception :
> {code}
> Path stagingArea = new Path(conf.get(JTConfig.JT_STAGING_AREA_ROOT));
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1322) TestStreamingAsDifferentUser fails on trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-1322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amareshwari Sriramadasu updated MAPREDUCE-1322:
---
Attachment: 1322-1.patch
Updated to trunk
> TestStreamingAsDifferentUser fails on trunk
> ---
>
> Key: MAPREDUCE-1322
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1322
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/streaming, test
>Affects Versions: 0.22.0
>Reporter: Amareshwari Sriramadasu
>Assignee: Devaraj Das
> Fix For: 0.22.0
>
> Attachments: 1322-1.patch, 1322.patch
>
>
> TestStreamingAsDifferentUser fails on trunk with following exception :
> Can not create a Path from a null string
> java.lang.IllegalArgumentException: Can not create a Path from a null string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:78)
> at org.apache.hadoop.fs.Path.(Path.java:90)
> at
> org.apache.hadoop.mapred.ClusterWithLinuxTaskController.createHomeAndStagingDirectory(ClusterWithLinuxTaskController.java:158)
> at
> org.apache.hadoop.mapred.ClusterWithLinuxTaskController.startCluster(ClusterWithLinuxTaskController.java:147)
> at
> org.apache.hadoop.streaming.TestStreamingAsDifferentUser.testStreaming(TestStreamingAsDifferentUser.java:49)
> The corresponding line for the exception :
> {code}
> Path stagingArea = new Path(conf.get(JTConfig.JT_STAGING_AREA_ROOT));
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1322) TestStreamingAsDifferentUser fails on trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-1322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amareshwari Sriramadasu updated MAPREDUCE-1322:
---
Status: Open (was: Patch Available)
Patch does not apply anymore
> TestStreamingAsDifferentUser fails on trunk
> ---
>
> Key: MAPREDUCE-1322
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1322
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/streaming, test
>Affects Versions: 0.22.0
>Reporter: Amareshwari Sriramadasu
>Assignee: Devaraj Das
> Fix For: 0.22.0
>
> Attachments: 1322-1.patch, 1322.patch
>
>
> TestStreamingAsDifferentUser fails on trunk with following exception :
> Can not create a Path from a null string
> java.lang.IllegalArgumentException: Can not create a Path from a null string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:78)
> at org.apache.hadoop.fs.Path.(Path.java:90)
> at
> org.apache.hadoop.mapred.ClusterWithLinuxTaskController.createHomeAndStagingDirectory(ClusterWithLinuxTaskController.java:158)
> at
> org.apache.hadoop.mapred.ClusterWithLinuxTaskController.startCluster(ClusterWithLinuxTaskController.java:147)
> at
> org.apache.hadoop.streaming.TestStreamingAsDifferentUser.testStreaming(TestStreamingAsDifferentUser.java:49)
> The corresponding line for the exception :
> {code}
> Path stagingArea = new Path(conf.get(JTConfig.JT_STAGING_AREA_ROOT));
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1421) TestDebugScriptWithLinuxTaskController and TestJobExecutionAsDifferentUser fail on trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-1421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod K V updated MAPREDUCE-1421: - Component/s: task-controller > TestDebugScriptWithLinuxTaskController and TestJobExecutionAsDifferentUser > fail on trunk > > > Key: MAPREDUCE-1421 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1421 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: task-controller, tasktracker, test >Affects Versions: 0.22.0 >Reporter: Amareshwari Sriramadasu > Fix For: 0.22.0 > > Attachments: patch-1421.txt, TestJobExecutionAsDifferentUser.patch > > > TestDebugScriptWithLinuxTaskController and TestJobExecutionAsDifferentUser > fail on trunk after the commit of MAPREDUCE-1385 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-899) When using LinuxTaskController, localized files may become accessible to unintended users if permissions are misconfigured.
[ https://issues.apache.org/jira/browse/MAPREDUCE-899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806273#action_12806273 ] Amareshwari Sriramadasu commented on MAPREDUCE-899: --- Ran all tests with LinuxTaskController. Test failures were all known ( MAPREDUCE-1421 and MAPREDUCE-1322) > When using LinuxTaskController, localized files may become accessible to > unintended users if permissions are misconfigured. > --- > > Key: MAPREDUCE-899 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-899 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Vinod K V >Assignee: Amareshwari Sriramadasu > Fix For: 0.22.0 > > Attachments: MAPREDUCE-899-20090828.txt, patch-899-1.txt, > patch-899-2.txt, patch-899-3.txt, patch-899-4.txt, patch-899-5.txt, > patch-899.txt > > > To enforce the accessibility of job files to only the job-owner and the > TaskTracker, as per MAPREDUCE-842, it is _trusted_ that the setuid/setgid > linux TaskController binary is group owned by a _special group_ to which only > TaskTracker belongs and not just any group to which TT belongs. If the trust > is broken, possibly due to misconfiguration by admins, the local files become > accessible to unintended users, yet giving false sense of security to the > admins. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
