[jira] [Resolved] (HADOOP-7905) Port FileContext symlinks to FileSystem
[ https://issues.apache.org/jira/browse/HADOOP-7905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HADOOP-7905. - Resolution: Duplicate Resolving as dupe of HADOOP-8040 subtasks HADOOP-9414 and HADOOP-9416. > Port FileContext symlinks to FileSystem > --- > > Key: HADOOP-7905 > URL: https://issues.apache.org/jira/browse/HADOOP-7905 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Reporter: Eli Collins > > FileSystem isn't going away anytime soon (HADOOP-6446). It would be useful to > implement HADOOP-6421 for FileSystem, this would allow interoperability > between FileContext and FileSystem (eg currently a symlink created via > FileContext is not readable via FileSystem), which will help people migrate > to FileContext. The work is mostly moving the client-side link resolution > code to a shared place and porting the tests or modifying them to be FC/FS > agnostic. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-9419) CodecPool should avoid OOMs with buggy codecs
[ https://issues.apache.org/jira/browse/HADOOP-9419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans resolved HADOOP-9419. - Resolution: Won't Fix Never mind. I created a patch, and it is completely useless in fixing this problem. The tasks still OOM because the codec itself is so small and the MergeManager creates new codecs so quickly that on a job with lots of reduces it literally uses up all of the address space with direct byte buffers. Some of the processes get killed by the NM for going over the virtual address space before they OOM. We could try and have the CodecPool detect that the codec is doing the wrong thing and "correct" it for the codec, but that is too heavy handed in my opinion. > CodecPool should avoid OOMs with buggy codecs > - > > Key: HADOOP-9419 > URL: https://issues.apache.org/jira/browse/HADOOP-9419 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Robert Joseph Evans > > I recently found a bug in the gpl compression libraries that was causing map > tasks for a particular job to OOM. > https://github.com/omalley/hadoop-gpl-compression/issues/3 > Now granted it does not make a lot of sense for a job to use the LzopCodec > for map output compression over the LzoCodec, but arguably other codecs could > be doing similar things and causing the same sort of memory leaks. I propose > that we do a sanity check when creating a new decompressor/compressor. If > the codec newly created object does not match the value from getType... it > should turn off caching for that Codec. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-8817) Backport Network Topology Extension for Virtualization (HADOOP-8468) to branch-1
[ https://issues.apache.org/jira/browse/HADOOP-8817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Lu resolved HADOOP-8817. - Resolution: Fixed This jira is covered by JIRAs it "contains". > Backport Network Topology Extension for Virtualization (HADOOP-8468) to > branch-1 > > > Key: HADOOP-8817 > URL: https://issues.apache.org/jira/browse/HADOOP-8817 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 1.0.0 >Reporter: Junping Du >Assignee: Junping Du > Labels: features > Fix For: 1.2.0 > > Attachments: HADOOP-8817.patch, HADOOP-8817-v2.patch, > HADOOP-8817-v3.patch, HADOOP-8817-v4.patch > > > HADOOP-8468 propose network topology changes for running on virtualized > infrastructure, which includes: > 1. Add "NodeGroup" layer in new NetworkTopology (also known as > NetworkTopologyWithNodeGroup): HADOOP-8469, HADOOP-8470 > 2. Update Replica Placement/Removal Policy to reflect new topology layer: > HDFS-3498, HDFS-3601 > 3. Update balancer policy:HDFS-3495 > 4. Update Task Scheduling Policy to reflect new topology layer and support > the case that compute nodes (NodeManager or TaskTracker) and data nodes are > separated into different VMs, but still benefit from physical host locality: > YARN-18, YARN-19. > This JIRA will address the backport work on branch-1 which will be divided > into 4 issues/patches in related jira issues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-6942) Ability for having user's classes take precedence over the system classes for tasks' classpath
[ https://issues.apache.org/jira/browse/HADOOP-6942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved HADOOP-6942. - Resolution: Duplicate Fixed via MAPREDUCE-1938. Closing as dupe. > Ability for having user's classes take precedence over the system classes for > tasks' classpath > -- > > Key: HADOOP-6942 > URL: https://issues.apache.org/jira/browse/HADOOP-6942 > Project: Hadoop Common > Issue Type: Improvement > Components: scripts >Affects Versions: 0.22.0 >Reporter: Krishna Ramachandran > Attachments: HADOOP-6942.y20.patch, hadoop-common-6942.patch > > > Fix bin/hadoop script to facilitate mapred-1938 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
