[jira] [Updated] (MAPREDUCE-6040) distcp should automatically use /.reserved/raw when run by the superuser
[ https://issues.apache.org/jira/browse/MAPREDUCE-6040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sanjay Radia updated MAPREDUCE-6040: Attachment: HDFS-6134-Distcp-cp-UseCasesTable2.pdf Attached the distcp use cases table here for convenience. Note distcp needs to be fixed for both superuser and normal user (see the use case table). Row 1 and 4 in the use cases table are same for root and normal user. If for row 3 we believe, distcp should fail then again same for both root and normal user. > distcp should automatically use /.reserved/raw when run by the superuser > > > Key: MAPREDUCE-6040 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6040 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: distcp >Affects Versions: 3.0.0 >Reporter: Andrew Wang >Assignee: Charles Lamb > Attachments: HDFS-6134-Distcp-cp-UseCasesTable2.pdf, > MAPREDUCE-6040.001.patch > > > On HDFS-6134, [~sanjay.radia] asked for distcp to automatically prepend > /.reserved/raw if the distcp is being performed by the superuser and > /.reserved/raw is supported by both the source and destination filesystems. > This behavior only occurs if none of the src and target pathnames are > /.reserved/raw. > The -disablereservedraw flag can be used to disable this option. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5081) Backport DistCpV2 and the related JIRAs to branch-1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13614509#comment-13614509 ] Sanjay Radia commented on MAPREDUCE-5081: - bq. Why not leave the old distcp as is and add a new command for distcp2? Agree this is safer. Only challenge is whether we will carry distcp1 onto Hadoop 2 line for compatibility? We could take it forward and mark it deprecated and remove in a later release. > Backport DistCpV2 and the related JIRAs to branch-1 > --- > > Key: MAPREDUCE-5081 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5081 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: distcp >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Attachments: m5981_20130321b.patch, m5981_20130321.patch, > m5981_20130323.patch > > > Here is a list of DistCpV2 JIRAs: > - MAPREDUCE-2765: DistCpV2 main jira > - HADOOP-8703: turn CRC checking off for 0 byte size > - HDFS-3054: distcp -skipcrccheck has no effect. > - HADOOP-8431: Running distcp without args throws IllegalArgumentException > - HADOOP-8775: non-positive value to -bandwidth > - MAPREDUCE-4654: TestDistCp is ignored > - HADOOP-9022: distcp fails to copy file if -m 0 specified > - HADOOP-9025: TestCopyListing failing > - MAPREDUCE-5075: DistCp leaks input file handles > - distcp part of HADOOP-8341: Fix findbugs issues in hadoop-tools > - MAPREDUCE-5014: custom CopyListing -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3825) MR should not be getting duplicate tokens for a MR Job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13398726#comment-13398726 ] Sanjay Radia commented on MAPREDUCE-3825: - Summary: # Solution 4 with following changes ** FileSystem#addDelegationTokens returns the newly added tokens ** non-static method and also not-final ** add similar method to AbstractFileSystem # in trunk, and 2.0 remove addDelegationTokens - it was added in 0.23. Some customers are testing 0.23 - and hence we could remove this later from 0.23 # Add convenience method - FileUtil:AddTokens(renewer, path[] ps, credentials) - this can use filesystem or filecontext in its impl. > MR should not be getting duplicate tokens for a MR Job. > --- > > Key: MAPREDUCE-3825 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3825 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: security >Affects Versions: 0.23.1, 0.24.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: MAPREDUCE-3825.patch, TokenCache.pdf, solution4.patch > > > This is the counterpart to HADOOP-7967. > MR gets tokens for all input, output and the default filesystem when a MR job > is submitted. > The APIs in FileSystem make it challenging to avoid duplicate tokens when > there are file systems that have embedded > filesystems. > Here is the original description that Daryn wrote: > The token cache currently tries to assume a filesystem's token service key. > The assumption generally worked while there was a one to one mapping of > filesystem to token. With the advent of multi-token filesystems like viewfs, > the token cache will try to use a service key (ie. for viewfs) that will > never exist (because it really gets the mounted fs tokens). > The descriop -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4260) Use JobObject to spawn tasks on Windows
[ https://issues.apache.org/jira/browse/MAPREDUCE-4260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sanjay Radia resolved MAPREDUCE-4260. - Resolution: Fixed Commit to the windows branch. Thanks Bikas. > Use JobObject to spawn tasks on Windows > --- > > Key: MAPREDUCE-4260 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4260 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 1.0.0 >Reporter: Bikas Saha >Assignee: Bikas Saha > Attachments: MAPREDUCE-4260.branch-1-win.1.patch, > MAPREDUCE-4260.branch-1-win.2.patch, MAPREDUCE-4260.branch-1-win.patch, > MAPREDUCE-4260.patch, test.cpp > > > Currently, the Windows version spawns the task as a normal cmd shell from > which other downstream exe's are spawned. However, this is not bullet proof > because if an intermediate process exits before its child exits, then the > parent child process tree relationship cannot be constructed. Windows has a > concept of JobObject that is similar to the setsid behavior used in Linux. > The initial spawned task could be launched within its JobObject. Thereafter, > process termination, memory management etc could be operated on the JobObject. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4260) Use JobObject to spawn tasks on Windows
[ https://issues.apache.org/jira/browse/MAPREDUCE-4260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sanjay Radia updated MAPREDUCE-4260: Summary: Use JobObject to spawn tasks on Windows (was: Investigate use of JobObject to spawn tasks on Windows) > Use JobObject to spawn tasks on Windows > --- > > Key: MAPREDUCE-4260 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4260 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 1.0.0 >Reporter: Bikas Saha >Assignee: Bikas Saha > Attachments: MAPREDUCE-4260.branch-1-win.1.patch, > MAPREDUCE-4260.branch-1-win.2.patch, MAPREDUCE-4260.branch-1-win.patch, > MAPREDUCE-4260.patch, test.cpp > > > Currently, the Windows version spawns the task as a normal cmd shell from > which other downstream exe's are spawned. However, this is not bullet proof > because if an intermediate process exits before its child exits, then the > parent child process tree relationship cannot be constructed. Windows has a > concept of JobObject that is similar to the setsid behavior used in Linux. > The initial spawned task could be launched within its JobObject. Thereafter, > process termination, memory management etc could be operated on the JobObject. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3898) Hadoop for Windows - Interfacing with Windows to manage MR tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-3898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sanjay Radia updated MAPREDUCE-3898: Resolution: Duplicate Target Version/s: (was: 0.24.0, 1.1.0) Status: Resolved (was: Patch Available) Other MAPREDUCE jiras under HADOOP-8079 have been created to cover this: > Hadoop for Windows - Interfacing with Windows to manage MR tasks > > > Key: MAPREDUCE-3898 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3898 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 0.24.0, 1.1.0 >Reporter: Sanjay Radia > Labels: patch > Attachments: mapred-tasks.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4204) Refactor ProcfsBasedProcessTree to make the resource collection object pluggable
[ https://issues.apache.org/jira/browse/MAPREDUCE-4204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sanjay Radia resolved MAPREDUCE-4204. - Resolution: Fixed Thanks Bikas - Committed to branch-1-win > Refactor ProcfsBasedProcessTree to make the resource collection object > pluggable > > > Key: MAPREDUCE-4204 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4204 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Bikas Saha >Assignee: Bikas Saha > Attachments: MAPREDUCE-4204-1.patch, MAPREDUCE-4204.patch > > > Making it a pluggable interface will allow replacing the procfs based > implementation with ones for other platforms. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4201) Getting PID not working on Windows. Termination of Task/TaskJVM's not working
[ https://issues.apache.org/jira/browse/MAPREDUCE-4201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sanjay Radia resolved MAPREDUCE-4201. - Resolution: Fixed Thanks Bikas. Committed to branch-1-win > Getting PID not working on Windows. Termination of Task/TaskJVM's not working > - > > Key: MAPREDUCE-4201 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4201 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Bikas Saha >Assignee: Bikas Saha > Attachments: MAPREDUCE-4201.patch > > > Child Task not reporting PID because of Linux specific shell script > implementation. > Signaling task termination currently disabled by the initial Windows patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-2887) MR changes to match HADOOP-7524 (multiple RPC protocols)
[ https://issues.apache.org/jira/browse/MAPREDUCE-2887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sanjay Radia resolved MAPREDUCE-2887. - Resolution: Fixed Committed as part of HADOOP-7524 > MR changes to match HADOOP-7524 (multiple RPC protocols) > > > Key: MAPREDUCE-2887 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2887 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Fix For: 0.23.0, 0.24.0 > > Attachments: rpc6ForMR.patch, rpc7ForMR.patch > > -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2887) MR changes to match HADOOP-7524 (multiple RPC protocols)
[ https://issues.apache.org/jira/browse/MAPREDUCE-2887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sanjay Radia updated MAPREDUCE-2887: Status: Open (was: Patch Available) > MR changes to match HADOOP-7524 (multiple RPC protocols) > > > Key: MAPREDUCE-2887 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2887 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Fix For: 0.23.0, 0.24.0 > > Attachments: rpc6ForMR.patch, rpc7ForMR.patch > > -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2887) MR changes to match HADOOP-7524 (multiple RPC protocols)
[ https://issues.apache.org/jira/browse/MAPREDUCE-2887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sanjay Radia updated MAPREDUCE-2887: Attachment: rpc7ForMR.patch Minor fix to Job tracker - it passed only client protocol class when registering the rpc service. > MR changes to match HADOOP-7524 (multiple RPC protocols) > > > Key: MAPREDUCE-2887 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2887 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Fix For: 0.23.0, 0.24.0 > > Attachments: rpc6ForMR.patch, rpc7ForMR.patch > > -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2887) MR changes to match HADOOP-7524 (multiple RPC protocols)
[ https://issues.apache.org/jira/browse/MAPREDUCE-2887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sanjay Radia updated MAPREDUCE-2887: Attachment: rpc6ForMR.patch > MR changes to match HADOOP-7524 (multiple RPC protocols) > > > Key: MAPREDUCE-2887 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2887 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: rpc6ForMR.patch > > -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2887) MR changes to match HADOOP-7524 (multiple RPC protocols)
MR changes to match HADOOP-7524 (multiple RPC protocols) Key: MAPREDUCE-2887 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2887 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Sanjay Radia Assignee: Sanjay Radia -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-279) Map-Reduce 2.0
[ https://issues.apache.org/jira/browse/MAPREDUCE-279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12996852#comment-12996852 ] Sanjay Radia commented on MAPREDUCE-279: Am out of office and will return on March 2 2011. > Map-Reduce 2.0 > -- > > Key: MAPREDUCE-279 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-279 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker, tasktracker >Reporter: Arun C Murthy >Assignee: Arun C Murthy > Fix For: 0.23.0 > > > Re-factor MapReduce into a generic resource scheduler and a per-job, > user-defined component that manages the application execution. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-1548) Hadoop archives should be able to preserve times and other properties from original files
[ https://issues.apache.org/jira/browse/MAPREDUCE-1548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12843707#action_12843707 ] Sanjay Radia commented on MAPREDUCE-1548: - >I think liststatus should show the information stored in index file and not >the part file permissions. I disagree here. har -t should should show what is in the index. However, the har *filesystem* should show the properties it is implementing: - the replication factor of the har - this is the replication factor of the part file. - the file permissions of the har - this can be the permissions of the directory or the index or the part file (we need to pick one). The har file system should pass the following unit test: - listStatus - check that the permissions returned are enforced. > Hadoop archives should be able to preserve times and other properties from > original files > - > > Key: MAPREDUCE-1548 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1548 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: harchive >Reporter: Rodrigo Schmidt >Assignee: Rodrigo Schmidt > > Files inside hadoop archives don't keep their original: > - modification time > - access time > - permission > - owner > - group > all such properties are currently taken from the file storing the archive > index, and not the stored files. This doesn't look very correct. > There should be possible to preserve the original properties of the stored > files. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
