http://git-wip-us.apache.org/repos/asf/hadoop/blob/19041008/hadoop-common-project/hadoop-common/src/site/markdown/release/0.23.1/RELEASENOTES.0.23.1.md ---------------------------------------------------------------------- diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/release/0.23.1/RELEASENOTES.0.23.1.md b/hadoop-common-project/hadoop-common/src/site/markdown/release/0.23.1/RELEASENOTES.0.23.1.md index 569e1ed..19364d3 100644 --- a/hadoop-common-project/hadoop-common/src/site/markdown/release/0.23.1/RELEASENOTES.0.23.1.md +++ b/hadoop-common-project/hadoop-common/src/site/markdown/release/0.23.1/RELEASENOTES.0.23.1.md @@ -23,285 +23,280 @@ These release notes cover new developer and user-facing incompatibilities, impor --- -* [HADOOP-8013](https://issues.apache.org/jira/browse/HADOOP-8013) | *Major* | **ViewFileSystem does not honor setVerifyChecksum** +* [MAPREDUCE-2784](https://issues.apache.org/jira/browse/MAPREDUCE-2784) | *Major* | **[Gridmix] TestGridmixSummary fails with NPE when run in DEBUG mode.** -**WARNING: No release note provided for this incompatible change.** +Fixed bugs in ExecutionSummarizer and ResourceUsageMatcher. --- -* [HADOOP-8009](https://issues.apache.org/jira/browse/HADOOP-8009) | *Critical* | **Create hadoop-client and hadoop-minicluster artifacts for downstream projects** +* [MAPREDUCE-2950](https://issues.apache.org/jira/browse/MAPREDUCE-2950) | *Major* | **[Gridmix] TestUserResolve fails in trunk** -Generate integration artifacts "org.apache.hadoop:hadoop-client" and "org.apache.hadoop:hadoop-minicluster" containing all the jars needed to use Hadoop client APIs, and to run Hadoop MiniClusters, respectively. Push these artifacts to the maven repository when mvn-deploy, along with existing artifacts. +Fixes bug in TestUserResolve. --- -* [HADOOP-7986](https://issues.apache.org/jira/browse/HADOOP-7986) | *Major* | **Add config for History Server protocol in hadoop-policy for service level authorization.** +* [HDFS-2130](https://issues.apache.org/jira/browse/HDFS-2130) | *Major* | **Switch default checksum to CRC32C** -Adding config for MapReduce History Server protocol in hadoop-policy.xml for service level authorization. +The default checksum algorithm used on HDFS is now CRC32C. Data from previous versions of Hadoop can still be read backwards-compatibly. --- -* [HADOOP-7963](https://issues.apache.org/jira/browse/HADOOP-7963) | *Blocker* | **test failures: TestViewFileSystemWithAuthorityLocalFileSystem and TestViewFileSystemLocalFileSystem** +* [HDFS-2129](https://issues.apache.org/jira/browse/HDFS-2129) | *Major* | **Simplify BlockReader to not inherit from FSInputChecker** -Fix ViewFS to catch a null canonical service-name and pass tests TestViewFileSystem\* +BlockReader has been reimplemented to use direct byte buffers. If you use a custom socket factory, it must generate sockets that have associated Channels. --- -* [HADOOP-7851](https://issues.apache.org/jira/browse/HADOOP-7851) | *Major* | **Configuration.getClasses() never returns the default value.** +* [MAPREDUCE-3297](https://issues.apache.org/jira/browse/MAPREDUCE-3297) | *Major* | **Move Log Related components from yarn-server-nodemanager to yarn-common** -Fixed Configuration.getClasses() API to return the default value if the key is not set. +Moved log related components into yarn-common so that HistoryServer and clients can use them without depending on the yarn-server-nodemanager module. --- -* [HADOOP-7802](https://issues.apache.org/jira/browse/HADOOP-7802) | *Major* | **Hadoop scripts unconditionally source "$bin"/../libexec/hadoop-config.sh.** +* [MAPREDUCE-3221](https://issues.apache.org/jira/browse/MAPREDUCE-3221) | *Minor* | **ant test TestSubmitJob failing on trunk** -Here is a patch to enable this behavior +Fixed a bug in TestSubmitJob. --- -* [HADOOP-7470](https://issues.apache.org/jira/browse/HADOOP-7470) | *Minor* | **move up to Jackson 1.8.8** +* [MAPREDUCE-3215](https://issues.apache.org/jira/browse/MAPREDUCE-3215) | *Minor* | **org.apache.hadoop.mapreduce.TestNoJobSetupCleanup failing on trunk** -**WARNING: No release note provided for this incompatible change.** +Reneabled and fixed bugs in the failing test TestNoJobSetupCleanup. --- -* [HADOOP-7348](https://issues.apache.org/jira/browse/HADOOP-7348) | *Major* | **Modify the option of FsShell getmerge from [addnl] to [-nl] for consistency** +* [MAPREDUCE-3219](https://issues.apache.org/jira/browse/MAPREDUCE-3219) | *Minor* | **ant test TestDelegationToken failing on trunk** -The 'fs -getmerge' tool now uses a -nl flag to determine if adding a newline at end of each file is required, in favor of the 'addnl' boolean flag that was used earlier. +Reenabled and fixed bugs in the failing test TestDelegationToken. --- -* [HDFS-2316](https://issues.apache.org/jira/browse/HDFS-2316) | *Major* | **[umbrella] WebHDFS: a complete FileSystem implementation for accessing HDFS over HTTP** +* [MAPREDUCE-3217](https://issues.apache.org/jira/browse/MAPREDUCE-3217) | *Minor* | **ant test TestAuditLogger fails on trunk** -Provide WebHDFS as a complete FileSystem implementation for accessing HDFS over HTTP. -Previous hftp feature was a read-only FileSystem and does not provide "write" accesses. +Reenabled and fixed bugs in the failing ant test TestAuditLogger. --- -* [HDFS-2246](https://issues.apache.org/jira/browse/HDFS-2246) | *Major* | **Shortcut a local client reads to a Datanodes files directly** +* [MAPREDUCE-3342](https://issues.apache.org/jira/browse/MAPREDUCE-3342) | *Critical* | **JobHistoryServer doesn't show job queue** -1. New configurations -a. dfs.block.local-path-access.user is the key in datanode configuration to specify the user allowed to do short circuit read. -b. dfs.client.read.shortcircuit is the key to enable short circuit read at the client side configuration. -c. dfs.client.read.shortcircuit.skip.checksum is the key to bypass checksum check at the client side. -2. By default none of the above are enabled and short circuit read will not kick in. -3. If security is on, the feature can be used only for user that has kerberos credentials at the client, therefore map reduce tasks cannot benefit from it in general. +Fixed JobHistoryServer to also show the job's queue name. --- -* [HDFS-2130](https://issues.apache.org/jira/browse/HDFS-2130) | *Major* | **Switch default checksum to CRC32C** +* [MAPREDUCE-3345](https://issues.apache.org/jira/browse/MAPREDUCE-3345) | *Major* | **Race condition in ResourceManager causing TestContainerManagerSecurity to fail sometimes** -The default checksum algorithm used on HDFS is now CRC32C. Data from previous versions of Hadoop can still be read backwards-compatibly. +Fixed a race condition in ResourceManager that was causing TestContainerManagerSecurity to fail sometimes. --- -* [HDFS-2129](https://issues.apache.org/jira/browse/HDFS-2129) | *Major* | **Simplify BlockReader to not inherit from FSInputChecker** +* [MAPREDUCE-3368](https://issues.apache.org/jira/browse/MAPREDUCE-3368) | *Critical* | **compile-mapred-test fails** -BlockReader has been reimplemented to use direct byte buffers. If you use a custom socket factory, it must generate sockets that have associated Channels. +Fixed ant test compilation. --- -* [HDFS-1314](https://issues.apache.org/jira/browse/HDFS-1314) | *Minor* | **dfs.blocksize accepts only absolute value** +* [MAPREDUCE-2733](https://issues.apache.org/jira/browse/MAPREDUCE-2733) | *Major* | **Gridmix v3 cpu emulation system tests.** -The default blocksize property 'dfs.blocksize' now accepts unit symbols to be used instead of byte length. Values such as "10k", "128m", "1g" are now OK to provide instead of just no. of bytes as was before. +Adds system tests for the CPU emulation feature in Gridmix3. --- -* [MAPREDUCE-3854](https://issues.apache.org/jira/browse/MAPREDUCE-3854) | *Major* | **Reinstate environment variable tests in TestMiniMRChildTask** +* [MAPREDUCE-3333](https://issues.apache.org/jira/browse/MAPREDUCE-3333) | *Blocker* | **MR AM for sort-job going out of memory** -Fixed and reenabled tests related to MR child JVM's environmental variables in TestMiniMRChildTask. +Fixed bugs in ContainerLauncher of MR AppMaster due to which per-container connections to NodeManager were lingering long enough to hit the ulimits on number of processes. --- -* [MAPREDUCE-3846](https://issues.apache.org/jira/browse/MAPREDUCE-3846) | *Critical* | **Restarted+Recovered AM hangs in some corner cases** +* [MAPREDUCE-3280](https://issues.apache.org/jira/browse/MAPREDUCE-3280) | *Major* | **MR AM should not read the username from configuration** -Addressed MR AM hanging issues during AM restart and then the recovery. +Removed the unnecessary job user-name configuration in mapred-site.xml. --- -* [MAPREDUCE-3834](https://issues.apache.org/jira/browse/MAPREDUCE-3834) | *Critical* | **If multiple hosts for a split belong to the same rack, the rack is added multiple times in the AM request table** +* [MAPREDUCE-3392](https://issues.apache.org/jira/browse/MAPREDUCE-3392) | *Blocker* | **Cluster.getDelegationToken() throws NPE if client.getDelegationToken() returns null.** -Changed MR AM to not add the same rack entry multiple times into the container request table when multiple hosts for a split happen to be on the same rack +Fixed Cluster's getDelegationToken's API to return null when there isn't a supported token. --- -* [MAPREDUCE-3815](https://issues.apache.org/jira/browse/MAPREDUCE-3815) | *Critical* | **Data Locality suffers if the AM asks for containers using IPs instead of hostnames** +* [MAPREDUCE-3379](https://issues.apache.org/jira/browse/MAPREDUCE-3379) | *Major* | **LocalResourceTracker should not tracking deleted cache entries** -Fixed MR AM to always use hostnames and never IPs when requesting containers so that scheduler can give off data local containers correctly. +Fixed LocalResourceTracker in NodeManager to remove deleted cache entries correctly. --- -* [MAPREDUCE-3808](https://issues.apache.org/jira/browse/MAPREDUCE-3808) | *Blocker* | **NPE in FileOutputCommitter when running a 0 reduce job** +* [MAPREDUCE-3325](https://issues.apache.org/jira/browse/MAPREDUCE-3325) | *Major* | **Improvements to CapacityScheduler doc** -Fixed an NPE in FileOutputCommitter for jobs with maps but no reduces. +document changes only. --- -* [MAPREDUCE-3804](https://issues.apache.org/jira/browse/MAPREDUCE-3804) | *Major* | **yarn webapp interface vulnerable to cross scripting attacks** +* [MAPREDUCE-3375](https://issues.apache.org/jira/browse/MAPREDUCE-3375) | *Major* | **Memory Emulation system tests.** -fix cross scripting attacks vulnerability through webapp interface. +Added system tests to test the memory emulation feature in Gridmix. --- -* [MAPREDUCE-3802](https://issues.apache.org/jira/browse/MAPREDUCE-3802) | *Critical* | **If an MR AM dies twice it looks like the process freezes** +* [MAPREDUCE-3102](https://issues.apache.org/jira/browse/MAPREDUCE-3102) | *Major* | **NodeManager should fail fast with wrong configuration or permissions for LinuxContainerExecutor** -Added test to validate that AM can crash multiple times and still can recover successfully after MAPREDUCE-3846. +Changed NodeManager to fail fast when LinuxContainerExecutor has wrong configuration or permissions. --- -* [MAPREDUCE-3784](https://issues.apache.org/jira/browse/MAPREDUCE-3784) | *Major* | **maxActiveApplications(\|PerUser) per queue is too low for small clusters** +* [MAPREDUCE-3355](https://issues.apache.org/jira/browse/MAPREDUCE-3355) | *Blocker* | **AM scheduling hangs frequently with sort job on 350 nodes** -Fixed CapacityScheduler so that maxActiveApplication and maxActiveApplicationsPerUser per queue are not too low for small clusters. +Fixed MR AM's ContainerLauncher to handle node-command timeouts correctly. --- -* [MAPREDUCE-3774](https://issues.apache.org/jira/browse/MAPREDUCE-3774) | *Major* | **yarn-default.xml should be moved to hadoop-yarn-common.** +* [MAPREDUCE-3407](https://issues.apache.org/jira/browse/MAPREDUCE-3407) | *Minor* | **Wrong jar getting used in TestMR\*Jobs\* for MiniMRYarnCluster** -MAPREDUCE-3774. Moved yarn-default.xml to hadoop-yarn-common from hadoop-server-common. +Fixed pom files to refer to the correct MR app-jar needed by the integration tests. --- -* [MAPREDUCE-3760](https://issues.apache.org/jira/browse/MAPREDUCE-3760) | *Major* | **Blacklisted NMs should not appear in Active nodes list** +* [HADOOP-7802](https://issues.apache.org/jira/browse/HADOOP-7802) | *Major* | **Hadoop scripts unconditionally source "$bin"/../libexec/hadoop-config.sh.** -Changed active nodes list to not contain unhealthy nodes on the webUI and metrics. +Here is a patch to enable this behavior --- -* [MAPREDUCE-3754](https://issues.apache.org/jira/browse/MAPREDUCE-3754) | *Major* | **RM webapp should have pages filtered based on App-state** +* [MAPREDUCE-3412](https://issues.apache.org/jira/browse/MAPREDUCE-3412) | *Major* | **'ant docs' is broken** -Modified RM UI to filter applications based on state of the applications. +Fixes 'ant docs' by removing stale references to capacity-scheduler docs. --- -* [MAPREDUCE-3752](https://issues.apache.org/jira/browse/MAPREDUCE-3752) | *Blocker* | **Headroom should be capped by queue max-cap** +* [HDFS-2246](https://issues.apache.org/jira/browse/HDFS-2246) | *Major* | **Shortcut a local client reads to a Datanodes files directly** -Modified application limits to include queue max-capacities besides the usual user limits. +1. New configurations +a. dfs.block.local-path-access.user is the key in datanode configuration to specify the user allowed to do short circuit read. +b. dfs.client.read.shortcircuit is the key to enable short circuit read at the client side configuration. +c. dfs.client.read.shortcircuit.skip.checksum is the key to bypass checksum check at the client side. +2. By default none of the above are enabled and short circuit read will not kick in. +3. If security is on, the feature can be used only for user that has kerberos credentials at the client, therefore map reduce tasks cannot benefit from it in general. --- -* [MAPREDUCE-3732](https://issues.apache.org/jira/browse/MAPREDUCE-3732) | *Blocker* | **CS should only use 'activeUsers with pending requests' for computing user-limits** +* [HADOOP-7851](https://issues.apache.org/jira/browse/HADOOP-7851) | *Major* | **Configuration.getClasses() never returns the default value.** -Modified CapacityScheduler to use only users with pending requests for computing user-limits. +Fixed Configuration.getClasses() API to return the default value if the key is not set. --- -* [MAPREDUCE-3720](https://issues.apache.org/jira/browse/MAPREDUCE-3720) | *Major* | **Command line listJobs should not visit each AM** - -Changed bin/mapred job -list to not print job-specific information not available at RM. +* [MAPREDUCE-3519](https://issues.apache.org/jira/browse/MAPREDUCE-3519) | *Blocker* | **Deadlock in LocalDirsHandlerService and ShuffleHandler** -Very minor incompatibility in cmd-line output, inevitable due to MRv2 architecture. +Fixed a deadlock in NodeManager LocalDirectories's handling service. --- -* [MAPREDUCE-3716](https://issues.apache.org/jira/browse/MAPREDUCE-3716) | *Blocker* | **java.io.File.createTempFile fails in map/reduce tasks** +* [MAPREDUCE-2863](https://issues.apache.org/jira/browse/MAPREDUCE-2863) | *Blocker* | **Support web-services for RM & NM** -Fixing YARN+MR to allow MR jobs to be able to use java.io.File.createTempFile to create temporary files as part of their tasks. +Support for web-services in YARN and MR components. --- -* [MAPREDUCE-3714](https://issues.apache.org/jira/browse/MAPREDUCE-3714) | *Blocker* | **Reduce hangs in a corner case** +* [MAPREDUCE-3426](https://issues.apache.org/jira/browse/MAPREDUCE-3426) | *Blocker* | **uber-jobs tried to write outputs into wrong dir** -Fixed EventFetcher and Fetcher threads to shut-down properly so that reducers don't hang in corner cases. +Fixed MR AM in uber mode to write map intermediate outputs in the correct directory to work properly in secure mode. --- -* [MAPREDUCE-3713](https://issues.apache.org/jira/browse/MAPREDUCE-3713) | *Blocker* | **Incorrect headroom reported to jobs** +* [MAPREDUCE-3398](https://issues.apache.org/jira/browse/MAPREDUCE-3398) | *Blocker* | **Log Aggregation broken in Secure Mode** -Fixed the way head-room is allocated to applications by CapacityScheduler so that it deducts current-usage per user and not per-application. +Fixed log aggregation to work correctly in secure mode. Contributed by Siddharth Seth. --- -* [MAPREDUCE-3711](https://issues.apache.org/jira/browse/MAPREDUCE-3711) | *Blocker* | **AppMaster recovery for Medium to large jobs take long time** +* [MAPREDUCE-3530](https://issues.apache.org/jira/browse/MAPREDUCE-3530) | *Blocker* | **Sometimes NODE\_UPDATE to the scheduler throws an NPE causing the scheduling to stop** -Fixed MR AM recovery so that only single selected task output is recovered and thus reduce the unnecessarily bloated recovery time. +Fixed an NPE occuring during scheduling in the ResourceManager. --- -* [MAPREDUCE-3710](https://issues.apache.org/jira/browse/MAPREDUCE-3710) | *Major* | **last split generated by FileInputFormat.getSplits may not have the best locality** +* [MAPREDUCE-3484](https://issues.apache.org/jira/browse/MAPREDUCE-3484) | *Major* | **JobEndNotifier is getting interrupted before completing all its retries.** -Improved FileInputFormat to return better locality for the last split. +Fixed JobEndNotifier to not get interrupted before completing all its retries. --- -* [MAPREDUCE-3703](https://issues.apache.org/jira/browse/MAPREDUCE-3703) | *Critical* | **ResourceManager should provide node lists in JMX output** - -New JMX Bean in ResourceManager to provide list of live node managers: +* [MAPREDUCE-3487](https://issues.apache.org/jira/browse/MAPREDUCE-3487) | *Critical* | **jobhistory web ui task counters no longer links to singletakecounter page** -Hadoop:service=ResourceManager,name=RMNMInfo LiveNodeManagers +Fixed JobHistory web-UI to display links to single task's counters' page. --- -* [MAPREDUCE-3699](https://issues.apache.org/jira/browse/MAPREDUCE-3699) | *Major* | **Default RPC handlers are very low for YARN servers** +* [MAPREDUCE-3564](https://issues.apache.org/jira/browse/MAPREDUCE-3564) | *Blocker* | **TestStagingCleanup and TestJobEndNotifier are failing on trunk.** -Increased RPC handlers for all YARN servers to reasonable values for working at scale. +Fixed failures in TestStagingCleanup and TestJobEndNotifier tests. --- -* [MAPREDUCE-3656](https://issues.apache.org/jira/browse/MAPREDUCE-3656) | *Blocker* | **Sort job on 350 scale is consistently failing with latest MRV2 code** +* [MAPREDUCE-778](https://issues.apache.org/jira/browse/MAPREDUCE-778) | *Major* | **[Rumen] Need a standalone JobHistory log anonymizer** -Fixed a race condition in MR AM which is failing the sort benchmark consistently. +Added an anonymizer tool to Rumen. Anonymizer takes a Rumen trace file and/or topology as input. It supports persistence and plugins to override the default behavior. --- -* [MAPREDUCE-3641](https://issues.apache.org/jira/browse/MAPREDUCE-3641) | *Blocker* | **CapacityScheduler should be more conservative assigning off-switch requests** +* [MAPREDUCE-3387](https://issues.apache.org/jira/browse/MAPREDUCE-3387) | *Critical* | **A tracking URL of N/A before the app master is launched breaks oozie** -Making CapacityScheduler more conservative so as to assign only one off-switch container in a single scheduling iteration. +Fixed AM's tracking URL to always go through the proxy, even before the job started, so that it works properly with oozie throughout the job execution. --- -* [MAPREDUCE-3639](https://issues.apache.org/jira/browse/MAPREDUCE-3639) | *Blocker* | **TokenCache likely broken for FileSystems which don't issue delegation tokens** +* [MAPREDUCE-3339](https://issues.apache.org/jira/browse/MAPREDUCE-3339) | *Blocker* | **Job is getting hanged indefinitely,if the child processes are killed on the NM. KILL\_CONTAINER eventtype is continuosly sent to the containers that are not existing** -Fixed TokenCache to work with absent FileSystem canonical service-names. +Fixed MR AM to stop considering node blacklisting after the number of nodes blacklisted crosses a threshold. --- -* [MAPREDUCE-3630](https://issues.apache.org/jira/browse/MAPREDUCE-3630) | *Critical* | **NullPointerException running teragen** +* [MAPREDUCE-3349](https://issues.apache.org/jira/browse/MAPREDUCE-3349) | *Blocker* | **No rack-name logged in JobHistory for unsuccessful tasks** -Committed to trunk and branch-0.23. Thanks Mahadev. +Unsuccessful tasks now log hostname and rackname to job history. --- -* [MAPREDUCE-3618](https://issues.apache.org/jira/browse/MAPREDUCE-3618) | *Major* | **TaskHeartbeatHandler holds a global lock for all task-updates** +* [MAPREDUCE-3586](https://issues.apache.org/jira/browse/MAPREDUCE-3586) | *Blocker* | **Lots of AMs hanging around in PIG testing** -Fixed TaskHeartbeatHandler to not hold a global lock for all task-updates. +Modified CompositeService to avoid duplicate stop operations thereby solving race conditions in MR AM shutdown. --- @@ -313,9 +308,9 @@ Rumen now provides {{Parsed\*}} objects. These objects provide extra information --- -* [MAPREDUCE-3586](https://issues.apache.org/jira/browse/MAPREDUCE-3586) | *Blocker* | **Lots of AMs hanging around in PIG testing** +* [MAPREDUCE-3399](https://issues.apache.org/jira/browse/MAPREDUCE-3399) | *Blocker* | **ContainerLocalizer should request new resources after completing the current one** -Modified CompositeService to avoid duplicate stop operations thereby solving race conditions in MR AM shutdown. +Modified ContainerLocalizer to send a heartbeat to NM immediately after downloading a resource instead of always waiting for a second. --- @@ -327,351 +322,356 @@ Optimized Job's progress calculations in MR AM. --- -* [MAPREDUCE-3564](https://issues.apache.org/jira/browse/MAPREDUCE-3564) | *Blocker* | **TestStagingCleanup and TestJobEndNotifier are failing on trunk.** +* [HADOOP-7348](https://issues.apache.org/jira/browse/HADOOP-7348) | *Major* | **Modify the option of FsShell getmerge from [addnl] to [-nl] for consistency** -Fixed failures in TestStagingCleanup and TestJobEndNotifier tests. +The 'fs -getmerge' tool now uses a -nl flag to determine if adding a newline at end of each file is required, in favor of the 'addnl' boolean flag that was used earlier. --- -* [MAPREDUCE-3549](https://issues.apache.org/jira/browse/MAPREDUCE-3549) | *Blocker* | **write api documentation for web service apis for RM, NM, mapreduce app master, and job history server** - -new files added: A hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/WebServicesIntro.apt.vm -A hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/NodeManagerRest.apt.vm -A hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ResourceManagerRest.apt.vm -A hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/MapredAppMasterRest.apt.vm -A hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/HistoryServerRest.apt.vm +* [MAPREDUCE-3462](https://issues.apache.org/jira/browse/MAPREDUCE-3462) | *Blocker* | **Job submission failing in JUnit tests** -The hadoop-project/src/site/site.xml is split into separate patch. +Fixed failing JUnit tests in Gridmix. --- -* [MAPREDUCE-3532](https://issues.apache.org/jira/browse/MAPREDUCE-3532) | *Critical* | **When 0 is provided as port number in yarn.nodemanager.webapp.address, NMs webserver component picks up random port, NM keeps on Reporting 0 port to RM** +* [HDFS-1314](https://issues.apache.org/jira/browse/HDFS-1314) | *Minor* | **dfs.blocksize accepts only absolute value** -Modified NM to report correct http address when an ephemeral web port is configured. +The default blocksize property 'dfs.blocksize' now accepts unit symbols to be used instead of byte length. Values such as "10k", "128m", "1g" are now OK to provide instead of just no. of bytes as was before. --- -* [MAPREDUCE-3530](https://issues.apache.org/jira/browse/MAPREDUCE-3530) | *Blocker* | **Sometimes NODE\_UPDATE to the scheduler throws an NPE causing the scheduling to stop** +* [MAPREDUCE-3490](https://issues.apache.org/jira/browse/MAPREDUCE-3490) | *Blocker* | **RMContainerAllocator counts failed maps towards Reduce ramp up** -Fixed an NPE occuring during scheduling in the ResourceManager. +Fixed MapReduce AM to count failed maps also towards Reduce ramp up. --- -* [MAPREDUCE-3528](https://issues.apache.org/jira/browse/MAPREDUCE-3528) | *Major* | **The task timeout check interval should be configurable independent of mapreduce.task.timeout** +* [MAPREDUCE-3511](https://issues.apache.org/jira/browse/MAPREDUCE-3511) | *Blocker* | **Counters occupy a good part of AM heap** -Fixed TaskHeartBeatHandler to use a new configuration for the thread loop interval separate from task-timeout configuration property. +Removed a multitude of cloned/duplicate counters in the AM thereby reducing the AM heap size and preventing full GCs. --- -* [MAPREDUCE-3519](https://issues.apache.org/jira/browse/MAPREDUCE-3519) | *Blocker* | **Deadlock in LocalDirsHandlerService and ShuffleHandler** +* [HADOOP-7963](https://issues.apache.org/jira/browse/HADOOP-7963) | *Blocker* | **test failures: TestViewFileSystemWithAuthorityLocalFileSystem and TestViewFileSystemLocalFileSystem** -Fixed a deadlock in NodeManager LocalDirectories's handling service. +Fix ViewFS to catch a null canonical service-name and pass tests TestViewFileSystem\* --- -* [MAPREDUCE-3512](https://issues.apache.org/jira/browse/MAPREDUCE-3512) | *Blocker* | **Batch jobHistory disk flushes** +* [MAPREDUCE-3528](https://issues.apache.org/jira/browse/MAPREDUCE-3528) | *Major* | **The task timeout check interval should be configurable independent of mapreduce.task.timeout** -Batching JobHistory flushing to DFS so that we don't flush for every event slowing down AM. +Fixed TaskHeartBeatHandler to use a new configuration for the thread loop interval separate from task-timeout configuration property. --- -* [MAPREDUCE-3511](https://issues.apache.org/jira/browse/MAPREDUCE-3511) | *Blocker* | **Counters occupy a good part of AM heap** +* [MAPREDUCE-3639](https://issues.apache.org/jira/browse/MAPREDUCE-3639) | *Blocker* | **TokenCache likely broken for FileSystems which don't issue delegation tokens** -Removed a multitude of cloned/duplicate counters in the AM thereby reducing the AM heap size and preventing full GCs. +Fixed TokenCache to work with absent FileSystem canonical service-names. --- -* [MAPREDUCE-3490](https://issues.apache.org/jira/browse/MAPREDUCE-3490) | *Blocker* | **RMContainerAllocator counts failed maps towards Reduce ramp up** +* [MAPREDUCE-3312](https://issues.apache.org/jira/browse/MAPREDUCE-3312) | *Major* | **Make MR AM not send a stopContainer w/o corresponding start container** -Fixed MapReduce AM to count failed maps also towards Reduce ramp up. +Modified MR AM to not send a stop-container request for a container that isn't launched at all. --- -* [MAPREDUCE-3487](https://issues.apache.org/jira/browse/MAPREDUCE-3487) | *Critical* | **jobhistory web ui task counters no longer links to singletakecounter page** +* [MAPREDUCE-3382](https://issues.apache.org/jira/browse/MAPREDUCE-3382) | *Critical* | **Network ACLs can prevent AMs to ping the Job-end notification URL** -Fixed JobHistory web-UI to display links to single task's counters' page. +Enhanced MR AM to use a proxy to ping the job-end notification URL. --- -* [MAPREDUCE-3484](https://issues.apache.org/jira/browse/MAPREDUCE-3484) | *Major* | **JobEndNotifier is getting interrupted before completing all its retries.** +* [MAPREDUCE-3299](https://issues.apache.org/jira/browse/MAPREDUCE-3299) | *Minor* | **Add AMInfo table to the AM job page** -Fixed JobEndNotifier to not get interrupted before completing all its retries. +Added AMInfo table to the MR AM job pages to list all the job-attempts when AM restarts and recovers. --- -* [MAPREDUCE-3481](https://issues.apache.org/jira/browse/MAPREDUCE-3481) | *Major* | **[Gridmix] Improve STRESS mode locking** +* [MAPREDUCE-3618](https://issues.apache.org/jira/browse/MAPREDUCE-3618) | *Major* | **TaskHeartbeatHandler holds a global lock for all task-updates** -Modified Gridmix STRESS mode locking structure. The submitted thread and the polling thread now run simultaneously without blocking each other. +Fixed TaskHeartbeatHandler to not hold a global lock for all task-updates. --- -* [MAPREDUCE-3462](https://issues.apache.org/jira/browse/MAPREDUCE-3462) | *Blocker* | **Job submission failing in JUnit tests** +* [MAPREDUCE-3512](https://issues.apache.org/jira/browse/MAPREDUCE-3512) | *Blocker* | **Batch jobHistory disk flushes** -Fixed failing JUnit tests in Gridmix. +Batching JobHistory flushing to DFS so that we don't flush for every event slowing down AM. --- -* [MAPREDUCE-3426](https://issues.apache.org/jira/browse/MAPREDUCE-3426) | *Blocker* | **uber-jobs tried to write outputs into wrong dir** +* [MAPREDUCE-3656](https://issues.apache.org/jira/browse/MAPREDUCE-3656) | *Blocker* | **Sort job on 350 scale is consistently failing with latest MRV2 code** -Fixed MR AM in uber mode to write map intermediate outputs in the correct directory to work properly in secure mode. +Fixed a race condition in MR AM which is failing the sort benchmark consistently. --- -* [MAPREDUCE-3417](https://issues.apache.org/jira/browse/MAPREDUCE-3417) | *Blocker* | **job access controls not working app master and job history UI's** +* [MAPREDUCE-3532](https://issues.apache.org/jira/browse/MAPREDUCE-3532) | *Critical* | **When 0 is provided as port number in yarn.nodemanager.webapp.address, NMs webserver component picks up random port, NM keeps on Reporting 0 port to RM** -Fixed job-access-controls to work with MR AM and JobHistoryServer web-apps. +Modified NM to report correct http address when an ephemeral web port is configured. --- -* [MAPREDUCE-3412](https://issues.apache.org/jira/browse/MAPREDUCE-3412) | *Major* | **'ant docs' is broken** +* [MAPREDUCE-3404](https://issues.apache.org/jira/browse/MAPREDUCE-3404) | *Critical* | **Speculative Execution: speculative map tasks launched even if -Dmapreduce.map.speculative=false** -Fixes 'ant docs' by removing stale references to capacity-scheduler docs. +Corrected MR AM to honor speculative configuration and enable speculating either maps or reduces. --- -* [MAPREDUCE-3407](https://issues.apache.org/jira/browse/MAPREDUCE-3407) | *Minor* | **Wrong jar getting used in TestMR\*Jobs\* for MiniMRYarnCluster** +* [MAPREDUCE-3641](https://issues.apache.org/jira/browse/MAPREDUCE-3641) | *Blocker* | **CapacityScheduler should be more conservative assigning off-switch requests** -Fixed pom files to refer to the correct MR app-jar needed by the integration tests. +Making CapacityScheduler more conservative so as to assign only one off-switch container in a single scheduling iteration. --- -* [MAPREDUCE-3404](https://issues.apache.org/jira/browse/MAPREDUCE-3404) | *Critical* | **Speculative Execution: speculative map tasks launched even if -Dmapreduce.map.speculative=false** +* [HADOOP-7986](https://issues.apache.org/jira/browse/HADOOP-7986) | *Major* | **Add config for History Server protocol in hadoop-policy for service level authorization.** -Corrected MR AM to honor speculative configuration and enable speculating either maps or reduces. +Adding config for MapReduce History Server protocol in hadoop-policy.xml for service level authorization. --- -* [MAPREDUCE-3399](https://issues.apache.org/jira/browse/MAPREDUCE-3399) | *Blocker* | **ContainerLocalizer should request new resources after completing the current one** +* [MAPREDUCE-3549](https://issues.apache.org/jira/browse/MAPREDUCE-3549) | *Blocker* | **write api documentation for web service apis for RM, NM, mapreduce app master, and job history server** -Modified ContainerLocalizer to send a heartbeat to NM immediately after downloading a resource instead of always waiting for a second. +new files added: A hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/WebServicesIntro.apt.vm +A hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/NodeManagerRest.apt.vm +A hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ResourceManagerRest.apt.vm +A hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/MapredAppMasterRest.apt.vm +A hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/HistoryServerRest.apt.vm + +The hadoop-project/src/site/site.xml is split into separate patch. --- -* [MAPREDUCE-3398](https://issues.apache.org/jira/browse/MAPREDUCE-3398) | *Blocker* | **Log Aggregation broken in Secure Mode** +* [MAPREDUCE-3710](https://issues.apache.org/jira/browse/MAPREDUCE-3710) | *Major* | **last split generated by FileInputFormat.getSplits may not have the best locality** -Fixed log aggregation to work correctly in secure mode. Contributed by Siddharth Seth. +Improved FileInputFormat to return better locality for the last split. --- -* [MAPREDUCE-3392](https://issues.apache.org/jira/browse/MAPREDUCE-3392) | *Blocker* | **Cluster.getDelegationToken() throws NPE if client.getDelegationToken() returns null.** +* [MAPREDUCE-3714](https://issues.apache.org/jira/browse/MAPREDUCE-3714) | *Blocker* | **Reduce hangs in a corner case** -Fixed Cluster's getDelegationToken's API to return null when there isn't a supported token. +Fixed EventFetcher and Fetcher threads to shut-down properly so that reducers don't hang in corner cases. --- -* [MAPREDUCE-3387](https://issues.apache.org/jira/browse/MAPREDUCE-3387) | *Critical* | **A tracking URL of N/A before the app master is launched breaks oozie** +* [MAPREDUCE-3630](https://issues.apache.org/jira/browse/MAPREDUCE-3630) | *Critical* | **NullPointerException running teragen** -Fixed AM's tracking URL to always go through the proxy, even before the job started, so that it works properly with oozie throughout the job execution. +Committed to trunk and branch-0.23. Thanks Mahadev. --- -* [MAPREDUCE-3382](https://issues.apache.org/jira/browse/MAPREDUCE-3382) | *Critical* | **Network ACLs can prevent AMs to ping the Job-end notification URL** +* [MAPREDUCE-3713](https://issues.apache.org/jira/browse/MAPREDUCE-3713) | *Blocker* | **Incorrect headroom reported to jobs** -Enhanced MR AM to use a proxy to ping the job-end notification URL. +Fixed the way head-room is allocated to applications by CapacityScheduler so that it deducts current-usage per user and not per-application. --- -* [MAPREDUCE-3379](https://issues.apache.org/jira/browse/MAPREDUCE-3379) | *Major* | **LocalResourceTracker should not tracking deleted cache entries** +* [MAPREDUCE-2765](https://issues.apache.org/jira/browse/MAPREDUCE-2765) | *Major* | **DistCp Rewrite** -Fixed LocalResourceTracker in NodeManager to remove deleted cache entries correctly. +DistCpV2 added to hadoop-tools. --- -* [MAPREDUCE-3375](https://issues.apache.org/jira/browse/MAPREDUCE-3375) | *Major* | **Memory Emulation system tests.** +* [MAPREDUCE-3699](https://issues.apache.org/jira/browse/MAPREDUCE-3699) | *Major* | **Default RPC handlers are very low for YARN servers** -Added system tests to test the memory emulation feature in Gridmix. +Increased RPC handlers for all YARN servers to reasonable values for working at scale. --- -* [MAPREDUCE-3368](https://issues.apache.org/jira/browse/MAPREDUCE-3368) | *Critical* | **compile-mapred-test fails** +* [MAPREDUCE-3360](https://issues.apache.org/jira/browse/MAPREDUCE-3360) | *Critical* | **Provide information about lost nodes in the UI.** -Fixed ant test compilation. +Added information about lost/rebooted/decommissioned nodes on the webapps. --- -* [MAPREDUCE-3360](https://issues.apache.org/jira/browse/MAPREDUCE-3360) | *Critical* | **Provide information about lost nodes in the UI.** +* [MAPREDUCE-3720](https://issues.apache.org/jira/browse/MAPREDUCE-3720) | *Major* | **Command line listJobs should not visit each AM** -Added information about lost/rebooted/decommissioned nodes on the webapps. +Changed bin/mapred job -list to not print job-specific information not available at RM. + +Very minor incompatibility in cmd-line output, inevitable due to MRv2 architecture. --- -* [MAPREDUCE-3355](https://issues.apache.org/jira/browse/MAPREDUCE-3355) | *Blocker* | **AM scheduling hangs frequently with sort job on 350 nodes** +* [MAPREDUCE-3732](https://issues.apache.org/jira/browse/MAPREDUCE-3732) | *Blocker* | **CS should only use 'activeUsers with pending requests' for computing user-limits** -Fixed MR AM's ContainerLauncher to handle node-command timeouts correctly. +Modified CapacityScheduler to use only users with pending requests for computing user-limits. --- -* [MAPREDUCE-3349](https://issues.apache.org/jira/browse/MAPREDUCE-3349) | *Blocker* | **No rack-name logged in JobHistory for unsuccessful tasks** +* [MAPREDUCE-3481](https://issues.apache.org/jira/browse/MAPREDUCE-3481) | *Major* | **[Gridmix] Improve STRESS mode locking** -Unsuccessful tasks now log hostname and rackname to job history. +Modified Gridmix STRESS mode locking structure. The submitted thread and the polling thread now run simultaneously without blocking each other. --- -* [MAPREDUCE-3345](https://issues.apache.org/jira/browse/MAPREDUCE-3345) | *Major* | **Race condition in ResourceManager causing TestContainerManagerSecurity to fail sometimes** +* [MAPREDUCE-3703](https://issues.apache.org/jira/browse/MAPREDUCE-3703) | *Critical* | **ResourceManager should provide node lists in JMX output** -Fixed a race condition in ResourceManager that was causing TestContainerManagerSecurity to fail sometimes. +New JMX Bean in ResourceManager to provide list of live node managers: + +Hadoop:service=ResourceManager,name=RMNMInfo LiveNodeManagers --- -* [MAPREDUCE-3342](https://issues.apache.org/jira/browse/MAPREDUCE-3342) | *Critical* | **JobHistoryServer doesn't show job queue** +* [MAPREDUCE-3716](https://issues.apache.org/jira/browse/MAPREDUCE-3716) | *Blocker* | **java.io.File.createTempFile fails in map/reduce tasks** -Fixed JobHistoryServer to also show the job's queue name. +Fixing YARN+MR to allow MR jobs to be able to use java.io.File.createTempFile to create temporary files as part of their tasks. --- -* [MAPREDUCE-3339](https://issues.apache.org/jira/browse/MAPREDUCE-3339) | *Blocker* | **Job is getting hanged indefinitely,if the child processes are killed on the NM. KILL\_CONTAINER eventtype is continuosly sent to the containers that are not existing** +* [MAPREDUCE-3754](https://issues.apache.org/jira/browse/MAPREDUCE-3754) | *Major* | **RM webapp should have pages filtered based on App-state** -Fixed MR AM to stop considering node blacklisting after the number of nodes blacklisted crosses a threshold. +Modified RM UI to filter applications based on state of the applications. --- -* [MAPREDUCE-3333](https://issues.apache.org/jira/browse/MAPREDUCE-3333) | *Blocker* | **MR AM for sort-job going out of memory** +* [MAPREDUCE-3774](https://issues.apache.org/jira/browse/MAPREDUCE-3774) | *Major* | **yarn-default.xml should be moved to hadoop-yarn-common.** -Fixed bugs in ContainerLauncher of MR AppMaster due to which per-container connections to NodeManager were lingering long enough to hit the ulimits on number of processes. +MAPREDUCE-3774. Moved yarn-default.xml to hadoop-yarn-common from hadoop-server-common. --- -* [MAPREDUCE-3325](https://issues.apache.org/jira/browse/MAPREDUCE-3325) | *Major* | **Improvements to CapacityScheduler doc** +* [HADOOP-7470](https://issues.apache.org/jira/browse/HADOOP-7470) | *Minor* | **move up to Jackson 1.8.8** -document changes only. +**WARNING: No release note provided for this change.** --- -* [MAPREDUCE-3312](https://issues.apache.org/jira/browse/MAPREDUCE-3312) | *Major* | **Make MR AM not send a stopContainer w/o corresponding start container** +* [MAPREDUCE-3752](https://issues.apache.org/jira/browse/MAPREDUCE-3752) | *Blocker* | **Headroom should be capped by queue max-cap** -Modified MR AM to not send a stop-container request for a container that isn't launched at all. +Modified application limits to include queue max-capacities besides the usual user limits. --- -* [MAPREDUCE-3299](https://issues.apache.org/jira/browse/MAPREDUCE-3299) | *Minor* | **Add AMInfo table to the AM job page** +* [MAPREDUCE-3784](https://issues.apache.org/jira/browse/MAPREDUCE-3784) | *Major* | **maxActiveApplications(\|PerUser) per queue is too low for small clusters** -Added AMInfo table to the MR AM job pages to list all the job-attempts when AM restarts and recovers. +Fixed CapacityScheduler so that maxActiveApplication and maxActiveApplicationsPerUser per queue are not too low for small clusters. --- -* [MAPREDUCE-3297](https://issues.apache.org/jira/browse/MAPREDUCE-3297) | *Major* | **Move Log Related components from yarn-server-nodemanager to yarn-common** +* [HADOOP-8013](https://issues.apache.org/jira/browse/HADOOP-8013) | *Major* | **ViewFileSystem does not honor setVerifyChecksum** -Moved log related components into yarn-common so that HistoryServer and clients can use them without depending on the yarn-server-nodemanager module. +**WARNING: No release note provided for this change.** --- -* [MAPREDUCE-3280](https://issues.apache.org/jira/browse/MAPREDUCE-3280) | *Major* | **MR AM should not read the username from configuration** +* [MAPREDUCE-3711](https://issues.apache.org/jira/browse/MAPREDUCE-3711) | *Blocker* | **AppMaster recovery for Medium to large jobs take long time** -Removed the unnecessary job user-name configuration in mapred-site.xml. +Fixed MR AM recovery so that only single selected task output is recovered and thus reduce the unnecessarily bloated recovery time. --- -* [MAPREDUCE-3221](https://issues.apache.org/jira/browse/MAPREDUCE-3221) | *Minor* | **ant test TestSubmitJob failing on trunk** +* [MAPREDUCE-3760](https://issues.apache.org/jira/browse/MAPREDUCE-3760) | *Major* | **Blacklisted NMs should not appear in Active nodes list** -Fixed a bug in TestSubmitJob. +Changed active nodes list to not contain unhealthy nodes on the webUI and metrics. --- -* [MAPREDUCE-3219](https://issues.apache.org/jira/browse/MAPREDUCE-3219) | *Minor* | **ant test TestDelegationToken failing on trunk** +* [MAPREDUCE-3417](https://issues.apache.org/jira/browse/MAPREDUCE-3417) | *Blocker* | **job access controls not working app master and job history UI's** -Reenabled and fixed bugs in the failing test TestDelegationToken. +Fixed job-access-controls to work with MR AM and JobHistoryServer web-apps. --- -* [MAPREDUCE-3217](https://issues.apache.org/jira/browse/MAPREDUCE-3217) | *Minor* | **ant test TestAuditLogger fails on trunk** +* [MAPREDUCE-3808](https://issues.apache.org/jira/browse/MAPREDUCE-3808) | *Blocker* | **NPE in FileOutputCommitter when running a 0 reduce job** -Reenabled and fixed bugs in the failing ant test TestAuditLogger. +Fixed an NPE in FileOutputCommitter for jobs with maps but no reduces. --- -* [MAPREDUCE-3215](https://issues.apache.org/jira/browse/MAPREDUCE-3215) | *Minor* | **org.apache.hadoop.mapreduce.TestNoJobSetupCleanup failing on trunk** +* [MAPREDUCE-3804](https://issues.apache.org/jira/browse/MAPREDUCE-3804) | *Major* | **yarn webapp interface vulnerable to cross scripting attacks** -Reneabled and fixed bugs in the failing test TestNoJobSetupCleanup. +fix cross scripting attacks vulnerability through webapp interface. --- -* [MAPREDUCE-3102](https://issues.apache.org/jira/browse/MAPREDUCE-3102) | *Major* | **NodeManager should fail fast with wrong configuration or permissions for LinuxContainerExecutor** +* [MAPREDUCE-3815](https://issues.apache.org/jira/browse/MAPREDUCE-3815) | *Critical* | **Data Locality suffers if the AM asks for containers using IPs instead of hostnames** -Changed NodeManager to fail fast when LinuxContainerExecutor has wrong configuration or permissions. +Fixed MR AM to always use hostnames and never IPs when requesting containers so that scheduler can give off data local containers correctly. --- -* [MAPREDUCE-2950](https://issues.apache.org/jira/browse/MAPREDUCE-2950) | *Major* | **[Gridmix] TestUserResolve fails in trunk** +* [MAPREDUCE-3834](https://issues.apache.org/jira/browse/MAPREDUCE-3834) | *Critical* | **If multiple hosts for a split belong to the same rack, the rack is added multiple times in the AM request table** -Fixes bug in TestUserResolve. +Changed MR AM to not add the same rack entry multiple times into the container request table when multiple hosts for a split happen to be on the same rack --- -* [MAPREDUCE-2863](https://issues.apache.org/jira/browse/MAPREDUCE-2863) | *Blocker* | **Support web-services for RM & NM** +* [MAPREDUCE-3846](https://issues.apache.org/jira/browse/MAPREDUCE-3846) | *Critical* | **Restarted+Recovered AM hangs in some corner cases** -Support for web-services in YARN and MR components. +Addressed MR AM hanging issues during AM restart and then the recovery. --- -* [MAPREDUCE-2784](https://issues.apache.org/jira/browse/MAPREDUCE-2784) | *Major* | **[Gridmix] TestGridmixSummary fails with NPE when run in DEBUG mode.** +* [HADOOP-8009](https://issues.apache.org/jira/browse/HADOOP-8009) | *Critical* | **Create hadoop-client and hadoop-minicluster artifacts for downstream projects** -Fixed bugs in ExecutionSummarizer and ResourceUsageMatcher. +Generate integration artifacts "org.apache.hadoop:hadoop-client" and "org.apache.hadoop:hadoop-minicluster" containing all the jars needed to use Hadoop client APIs, and to run Hadoop MiniClusters, respectively. Push these artifacts to the maven repository when mvn-deploy, along with existing artifacts. --- -* [MAPREDUCE-2765](https://issues.apache.org/jira/browse/MAPREDUCE-2765) | *Major* | **DistCp Rewrite** +* [MAPREDUCE-3802](https://issues.apache.org/jira/browse/MAPREDUCE-3802) | *Critical* | **If an MR AM dies twice it looks like the process freezes** -DistCpV2 added to hadoop-tools. +Added test to validate that AM can crash multiple times and still can recover successfully after MAPREDUCE-3846. --- -* [MAPREDUCE-2733](https://issues.apache.org/jira/browse/MAPREDUCE-2733) | *Major* | **Gridmix v3 cpu emulation system tests.** +* [MAPREDUCE-3854](https://issues.apache.org/jira/browse/MAPREDUCE-3854) | *Major* | **Reinstate environment variable tests in TestMiniMRChildTask** -Adds system tests for the CPU emulation feature in Gridmix3. +Fixed and reenabled tests related to MR child JVM's environmental variables in TestMiniMRChildTask. --- -* [MAPREDUCE-778](https://issues.apache.org/jira/browse/MAPREDUCE-778) | *Major* | **[Rumen] Need a standalone JobHistory log anonymizer** +* [HDFS-2316](https://issues.apache.org/jira/browse/HDFS-2316) | *Major* | **[umbrella] WebHDFS: a complete FileSystem implementation for accessing HDFS over HTTP** -Added an anonymizer tool to Rumen. Anonymizer takes a Rumen trace file and/or topology as input. It supports persistence and plugins to override the default behavior. +Provide WebHDFS as a complete FileSystem implementation for accessing HDFS over HTTP. +Previous hftp feature was a read-only FileSystem and does not provide "write" accesses.
http://git-wip-us.apache.org/repos/asf/hadoop/blob/19041008/hadoop-common-project/hadoop-common/src/site/markdown/release/0.23.10/CHANGES.0.23.10.md ---------------------------------------------------------------------- diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/release/0.23.10/CHANGES.0.23.10.md b/hadoop-common-project/hadoop-common/src/site/markdown/release/0.23.10/CHANGES.0.23.10.md index 75b818f..c94968f 100644 --- a/hadoop-common-project/hadoop-common/src/site/markdown/release/0.23.10/CHANGES.0.23.10.md +++ b/hadoop-common-project/hadoop-common/src/site/markdown/release/0.23.10/CHANGES.0.23.10.md @@ -27,112 +27,94 @@ | [YARN-707](https://issues.apache.org/jira/browse/YARN-707) | Add user info in the YARN ClientToken | Blocker | . | Bikas Saha | Jason Lowe | -### IMPORTANT ISSUES: - -| JIRA | Summary | Priority | Component | Reporter | Contributor | -|:---- |:---- | :--- |:---- |:---- |:---- | - - -### NEW FEATURES: - -| JIRA | Summary | Priority | Component | Reporter | Contributor | -|:---- |:---- | :--- |:---- |:---- |:---- | - - ### IMPROVEMENTS: | JIRA | Summary | Priority | Component | Reporter | Contributor | |:---- |:---- | :--- |:---- |:---- |:---- | -| [HADOOP-9686](https://issues.apache.org/jira/browse/HADOOP-9686) | Easy access to final parameters in Configuration | Major | conf | Jason Lowe | Jason Lowe | -| [HADOOP-8704](https://issues.apache.org/jira/browse/HADOOP-8704) | add request logging to jetty/httpserver | Major | . | Thomas Graves | Jonathan Eagles | | [HDFS-5010](https://issues.apache.org/jira/browse/HDFS-5010) | Reduce the frequency of getCurrentUser() calls from namenode | Major | namenode, performance | Kihwal Lee | Kihwal Lee | -| [MAPREDUCE-5640](https://issues.apache.org/jira/browse/MAPREDUCE-5640) | Rename TestLineRecordReader in jobclient module | Trivial | test | Jason Lowe | Jason Lowe | | [MAPREDUCE-1981](https://issues.apache.org/jira/browse/MAPREDUCE-1981) | Improve getSplits performance by using listLocatedStatus | Major | job submission | Hairong Kuang | Hairong Kuang | | [YARN-985](https://issues.apache.org/jira/browse/YARN-985) | Nodemanager should log where a resource was localized | Major | nodemanager | Ravi Prakash | Ravi Prakash | +| [HADOOP-9686](https://issues.apache.org/jira/browse/HADOOP-9686) | Easy access to final parameters in Configuration | Major | conf | Jason Lowe | Jason Lowe | +| [HADOOP-8704](https://issues.apache.org/jira/browse/HADOOP-8704) | add request logging to jetty/httpserver | Major | . | Thomas Graves | Jonathan Eagles | +| [MAPREDUCE-5640](https://issues.apache.org/jira/browse/MAPREDUCE-5640) | Rename TestLineRecordReader in jobclient module | Trivial | test | Jason Lowe | Jason Lowe | ### BUG FIXES: | JIRA | Summary | Priority | Component | Reporter | Contributor | |:---- |:---- | :--- |:---- |:---- |:---- | -| [HADOOP-10091](https://issues.apache.org/jira/browse/HADOOP-10091) | Job with a har archive as input fails on 0.23 | Blocker | fs | Jason Lowe | Jason Lowe | -| [HADOOP-9757](https://issues.apache.org/jira/browse/HADOOP-9757) | Har metadata cache can grow without limit | Major | fs | Jason Lowe | Cristina L. Abad | -| [HADOOP-9476](https://issues.apache.org/jira/browse/HADOOP-9476) | Some test cases in TestUserGroupInformation fail if ran after testSetLoginUser. | Major | security, test | Kihwal Lee | Robert Parker | -| [HDFS-5558](https://issues.apache.org/jira/browse/HDFS-5558) | LeaseManager monitor thread can crash if the last block is complete but another block is not. | Major | . | Kihwal Lee | Kihwal Lee | -| [HDFS-5557](https://issues.apache.org/jira/browse/HDFS-5557) | Write pipeline recovery for the last packet in the block may cause rejection of valid replicas | Critical | . | Kihwal Lee | Kihwal Lee | -| [HDFS-5526](https://issues.apache.org/jira/browse/HDFS-5526) | Datanode cannot roll back to previous layout version | Blocker | datanode | Tsz Wo Nicholas Sze | Kihwal Lee | -| [HDFS-5438](https://issues.apache.org/jira/browse/HDFS-5438) | Flaws in block report processing can cause data loss | Critical | namenode | Kihwal Lee | Kihwal Lee | -| [HDFS-5346](https://issues.apache.org/jira/browse/HDFS-5346) | Avoid unnecessary call to getNumLiveDataNodes() for each block during IBR processing | Major | namenode, performance | Kihwal Lee | Ravi Prakash | -| [HDFS-5047](https://issues.apache.org/jira/browse/HDFS-5047) | Supress logging of full stack trace of quota and lease exceptions | Major | namenode | Kihwal Lee | Robert Parker | -| [HDFS-5028](https://issues.apache.org/jira/browse/HDFS-5028) | LeaseRenewer throw java.util.ConcurrentModificationException when timeout | Major | . | zhaoyunjiong | zhaoyunjiong | -| [HDFS-4998](https://issues.apache.org/jira/browse/HDFS-4998) | TestUnderReplicatedBlocks fails intermittently | Major | test | Kihwal Lee | Kihwal Lee | -| [HDFS-4995](https://issues.apache.org/jira/browse/HDFS-4995) | Make getContentSummary() less expensive | Major | namenode | Kihwal Lee | Kihwal Lee | -| [HDFS-4994](https://issues.apache.org/jira/browse/HDFS-4994) | Audit log getContentSummary() calls | Minor | namenode | Kihwal Lee | Robert Parker | -| [HDFS-4993](https://issues.apache.org/jira/browse/HDFS-4993) | fsck can fail if a file is renamed or deleted | Major | . | Kihwal Lee | Robert Parker | +| [HDFS-3020](https://issues.apache.org/jira/browse/HDFS-3020) | Auto-logSync based on edit log buffer size broken | Critical | namenode | Todd Lipcon | Todd Lipcon | +| [MAPREDUCE-4579](https://issues.apache.org/jira/browse/MAPREDUCE-4579) | TestTaskAttempt fails jdk7 | Major | . | Thomas Graves | Thomas Graves | +| [HDFS-3970](https://issues.apache.org/jira/browse/HDFS-3970) | BlockPoolSliceStorage#doRollback(..) should use BlockPoolSliceStorage instead of DataStorage to read prev version file. | Major | datanode | Vinayakumar B | Vinayakumar B | +| [HDFS-4482](https://issues.apache.org/jira/browse/HDFS-4482) | ReplicationMonitor thread can exit with NPE due to the race between delete and replication of same file. | Blocker | namenode | Uma Maheswara Rao G | Uma Maheswara Rao G | +| [MAPREDUCE-4571](https://issues.apache.org/jira/browse/MAPREDUCE-4571) | TestHsWebServicesJobs fails on jdk7 | Major | webapps | Thomas Graves | Thomas Graves | +| [MAPREDUCE-4716](https://issues.apache.org/jira/browse/MAPREDUCE-4716) | TestHsWebServicesJobsQuery.testJobsQueryStateInvalid fails with jdk7 | Major | jobhistoryserver | Thomas Graves | Thomas Graves | +| [HDFS-4943](https://issues.apache.org/jira/browse/HDFS-4943) | WebHdfsFileSystem does not work when original file path has encoded chars | Minor | webhdfs | Jerry He | Jerry He | +| [MAPREDUCE-3193](https://issues.apache.org/jira/browse/MAPREDUCE-3193) | FileInputFormat doesn't read files recursively in the input path dir | Major | mrv1, mrv2 | Ramgopal N | Devaraj K | | [HDFS-4984](https://issues.apache.org/jira/browse/HDFS-4984) | Incorrect Quota counting in INodeFile | Major | namenode | Kihwal Lee | Jing Zhao | | [HDFS-4972](https://issues.apache.org/jira/browse/HDFS-4972) | [branch-0.23] permission check and operation are done in a separate lock for getBlockLocations() | Major | namenode | Kihwal Lee | Kihwal Lee | -| [HDFS-4943](https://issues.apache.org/jira/browse/HDFS-4943) | WebHdfsFileSystem does not work when original file path has encoded chars | Minor | webhdfs | Jerry He | Jerry He | -| [HDFS-4482](https://issues.apache.org/jira/browse/HDFS-4482) | ReplicationMonitor thread can exit with NPE due to the race between delete and replication of same file. | Blocker | namenode | Uma Maheswara Rao G | Uma Maheswara Rao G | -| [HDFS-4329](https://issues.apache.org/jira/browse/HDFS-4329) | DFSShell issues with directories with spaces in name | Major | hdfs-client | Andy Isaacson | Cristina L. Abad | -| [HDFS-3970](https://issues.apache.org/jira/browse/HDFS-3970) | BlockPoolSliceStorage#doRollback(..) should use BlockPoolSliceStorage instead of DataStorage to read prev version file. | Major | datanode | Vinayakumar B | Vinayakumar B | -| [HDFS-3020](https://issues.apache.org/jira/browse/HDFS-3020) | Auto-logSync based on edit log buffer size broken | Critical | namenode | Todd Lipcon | Todd Lipcon | -| [MAPREDUCE-5587](https://issues.apache.org/jira/browse/MAPREDUCE-5587) | TestTextOutputFormat fails on JDK7 | Major | . | Jonathan Eagles | Jonathan Eagles | -| [MAPREDUCE-5586](https://issues.apache.org/jira/browse/MAPREDUCE-5586) | TestCopyMapper#testCopyFailOnBlockSizeDifference fails when run from hadoop-tools/hadoop-distcp directory | Major | . | Jonathan Eagles | Jonathan Eagles | -| [MAPREDUCE-5585](https://issues.apache.org/jira/browse/MAPREDUCE-5585) | TestCopyCommitter#testNoCommitAction Fails on JDK7 | Major | . | Jonathan Eagles | Jonathan Eagles | -| [MAPREDUCE-5569](https://issues.apache.org/jira/browse/MAPREDUCE-5569) | FloatSplitter is not generating correct splits | Major | . | Nathan Roberts | Nathan Roberts | -| [MAPREDUCE-5561](https://issues.apache.org/jira/browse/MAPREDUCE-5561) | org.apache.hadoop.mapreduce.v2.app.job.impl.TestJobImpl testcase failing on trunk | Critical | . | Cindy Li | Karthik Kambatla | -| [MAPREDUCE-5543](https://issues.apache.org/jira/browse/MAPREDUCE-5543) | In-memory map outputs can be leaked after shuffle completes in 0.23 | Blocker | mrv2 | Jason Lowe | Jason Lowe | -| [MAPREDUCE-5513](https://issues.apache.org/jira/browse/MAPREDUCE-5513) | ConcurrentModificationException in JobControl | Major | . | Jason Lowe | Robert Parker | -| [MAPREDUCE-5504](https://issues.apache.org/jira/browse/MAPREDUCE-5504) | mapred queue -info inconsistent with types | Major | client | Thomas Graves | Kousuke Saruta | -| [MAPREDUCE-5475](https://issues.apache.org/jira/browse/MAPREDUCE-5475) | MRClientService does not verify ACLs properly | Blocker | mr-am, mrv2 | Jason Lowe | Jason Lowe | -| [MAPREDUCE-5468](https://issues.apache.org/jira/browse/MAPREDUCE-5468) | AM recovery does not work for map only jobs | Blocker | . | Yesha Vora | Vinod Kumar Vavilapalli | -| [MAPREDUCE-5440](https://issues.apache.org/jira/browse/MAPREDUCE-5440) | TestCopyCommitter Fails on JDK7 | Major | mrv2 | Robert Parker | Robert Parker | -| [MAPREDUCE-5425](https://issues.apache.org/jira/browse/MAPREDUCE-5425) | Junit in TestJobHistoryServer failing in jdk 7 | Major | jobhistoryserver | Ashwin Shankar | Robert Parker | -| [MAPREDUCE-5419](https://issues.apache.org/jira/browse/MAPREDUCE-5419) | TestSlive is getting FileNotFound Exception | Major | mrv2 | Robert Parker | Robert Parker | -| [MAPREDUCE-5414](https://issues.apache.org/jira/browse/MAPREDUCE-5414) | TestTaskAttempt fails jdk7 with NullPointerException | Major | test | Nemon Lou | Nemon Lou | | [MAPREDUCE-5380](https://issues.apache.org/jira/browse/MAPREDUCE-5380) | Invalid mapred command should return non-zero exit code | Major | . | Stephen Chu | Stephen Chu | -| [MAPREDUCE-5373](https://issues.apache.org/jira/browse/MAPREDUCE-5373) | TestFetchFailure.testFetchFailureMultipleReduces could fail intermittently | Major | . | Chuan Liu | Jonathan Eagles | +| [HDFS-4998](https://issues.apache.org/jira/browse/HDFS-4998) | TestUnderReplicatedBlocks fails intermittently | Major | test | Kihwal Lee | Kihwal Lee | | [MAPREDUCE-5317](https://issues.apache.org/jira/browse/MAPREDUCE-5317) | Stale files left behind for failed jobs | Major | mrv2 | Ravi Prakash | Ravi Prakash | +| [YARN-949](https://issues.apache.org/jira/browse/YARN-949) | Failed log aggregation can leave a file open. | Major | nodemanager | Kihwal Lee | Kihwal Lee | | [MAPREDUCE-5251](https://issues.apache.org/jira/browse/MAPREDUCE-5251) | Reducer should not implicate map attempt if it has insufficient space to fetch map output | Major | mrv2 | Jason Lowe | Ashwin Shankar | -| [MAPREDUCE-5001](https://issues.apache.org/jira/browse/MAPREDUCE-5001) | LocalJobRunner has race condition resulting in job failures | Major | . | Brock Noland | Sandy Ryza | -| [MAPREDUCE-4716](https://issues.apache.org/jira/browse/MAPREDUCE-4716) | TestHsWebServicesJobsQuery.testJobsQueryStateInvalid fails with jdk7 | Major | jobhistoryserver | Thomas Graves | Thomas Graves | -| [MAPREDUCE-4579](https://issues.apache.org/jira/browse/MAPREDUCE-4579) | TestTaskAttempt fails jdk7 | Major | . | Thomas Graves | Thomas Graves | -| [MAPREDUCE-4571](https://issues.apache.org/jira/browse/MAPREDUCE-4571) | TestHsWebServicesJobs fails on jdk7 | Major | webapps | Thomas Graves | Thomas Graves | -| [MAPREDUCE-3193](https://issues.apache.org/jira/browse/MAPREDUCE-3193) | FileInputFormat doesn't read files recursively in the input path dir | Major | mrv1, mrv2 | Ramgopal N | Devaraj K | -| [YARN-1419](https://issues.apache.org/jira/browse/YARN-1419) | TestFifoScheduler.testAppAttemptMetrics fails intermittently under jdk7 | Minor | scheduler | Jonathan Eagles | Jonathan Eagles | -| [YARN-1386](https://issues.apache.org/jira/browse/YARN-1386) | NodeManager mistakenly loses resources and relocalizes them | Critical | nodemanager | Jason Lowe | Jason Lowe | -| [YARN-1243](https://issues.apache.org/jira/browse/YARN-1243) | ResourceManager: Error in handling event type NODE\_UPDATE to the scheduler - NPE at SchedulerApp.java:411 | Major | capacityscheduler | Sanjay Upadhyay | Jason Lowe | -| [YARN-1176](https://issues.apache.org/jira/browse/YARN-1176) | RM web services ClusterMetricsInfo total nodes doesn't include unhealthy nodes | Critical | resourcemanager | Thomas Graves | Jonathan Eagles | -| [YARN-1101](https://issues.apache.org/jira/browse/YARN-1101) | Active nodes can be decremented below 0 | Major | resourcemanager | Robert Parker | Robert Parker | +| [MAPREDUCE-5419](https://issues.apache.org/jira/browse/MAPREDUCE-5419) | TestSlive is getting FileNotFound Exception | Major | mrv2 | Robert Parker | Robert Parker | +| [HDFS-5028](https://issues.apache.org/jira/browse/HDFS-5028) | LeaseRenewer throw java.util.ConcurrentModificationException when timeout | Major | . | yunjiong zhao | yunjiong zhao | +| [MAPREDUCE-5440](https://issues.apache.org/jira/browse/MAPREDUCE-5440) | TestCopyCommitter Fails on JDK7 | Major | mrv2 | Robert Parker | Robert Parker | +| [HADOOP-9476](https://issues.apache.org/jira/browse/HADOOP-9476) | Some test cases in TestUserGroupInformation fail if ran after testSetLoginUser. | Major | security, test | Kihwal Lee | Robert Parker | +| [MAPREDUCE-5425](https://issues.apache.org/jira/browse/MAPREDUCE-5425) | Junit in TestJobHistoryServer failing in jdk 7 | Major | jobhistoryserver | Ashwin Shankar | Robert Parker | +| [HDFS-5047](https://issues.apache.org/jira/browse/HDFS-5047) | Supress logging of full stack trace of quota and lease exceptions | Major | namenode | Kihwal Lee | Robert Parker | +| [HDFS-4993](https://issues.apache.org/jira/browse/HDFS-4993) | fsck can fail if a file is renamed or deleted | Major | . | Kihwal Lee | Robert Parker | +| [HADOOP-9757](https://issues.apache.org/jira/browse/HADOOP-9757) | Har metadata cache can grow without limit | Major | fs | Jason Lowe | Cristina L. Abad | | [YARN-1036](https://issues.apache.org/jira/browse/YARN-1036) | Distributed Cache gives inconsistent result if cache files get deleted from task tracker | Major | nodemanager | Ravi Prakash | Ravi Prakash | -| [YARN-1031](https://issues.apache.org/jira/browse/YARN-1031) | JQuery UI components reference external css in branch-23 | Major | . | Jonathan Eagles | Jonathan Eagles | -| [YARN-949](https://issues.apache.org/jira/browse/YARN-949) | Failed log aggregation can leave a file open. | Major | nodemanager | Kihwal Lee | Kihwal Lee | | [YARN-337](https://issues.apache.org/jira/browse/YARN-337) | RM handles killed application tracking URL poorly | Major | resourcemanager | Jason Lowe | Jason Lowe | +| [MAPREDUCE-5001](https://issues.apache.org/jira/browse/MAPREDUCE-5001) | LocalJobRunner has race condition resulting in job failures | Major | . | Brock Noland | Sandy Ryza | +| [HDFS-4994](https://issues.apache.org/jira/browse/HDFS-4994) | Audit log getContentSummary() calls | Minor | namenode | Kihwal Lee | Robert Parker | +| [MAPREDUCE-5468](https://issues.apache.org/jira/browse/MAPREDUCE-5468) | AM recovery does not work for map only jobs | Blocker | . | Yesha Vora | Vinod Kumar Vavilapalli | +| [HDFS-4329](https://issues.apache.org/jira/browse/HDFS-4329) | DFSShell issues with directories with spaces in name | Major | hdfs-client | Andy Isaacson | Cristina L. Abad | +| [YARN-1101](https://issues.apache.org/jira/browse/YARN-1101) | Active nodes can be decremented below 0 | Major | resourcemanager | Robert Parker | Robert Parker | +| [MAPREDUCE-5475](https://issues.apache.org/jira/browse/MAPREDUCE-5475) | MRClientService does not verify ACLs properly | Blocker | mr-am, mrv2 | Jason Lowe | Jason Lowe | +| [MAPREDUCE-5414](https://issues.apache.org/jira/browse/MAPREDUCE-5414) | TestTaskAttempt fails jdk7 with NullPointerException | Major | test | Nemon Lou | Nemon Lou | +| [YARN-1176](https://issues.apache.org/jira/browse/YARN-1176) | RM web services ClusterMetricsInfo total nodes doesn't include unhealthy nodes | Critical | resourcemanager | Thomas Graves | Jonathan Eagles | +| [MAPREDUCE-5504](https://issues.apache.org/jira/browse/MAPREDUCE-5504) | mapred queue -info inconsistent with types | Major | client | Thomas Graves | Kousuke Saruta | +| [YARN-1243](https://issues.apache.org/jira/browse/YARN-1243) | ResourceManager: Error in handling event type NODE\_UPDATE to the scheduler - NPE at SchedulerApp.java:411 | Major | capacityscheduler | Sanjay Upadhyay | Jason Lowe | +| [MAPREDUCE-5543](https://issues.apache.org/jira/browse/MAPREDUCE-5543) | In-memory map outputs can be leaked after shuffle completes in 0.23 | Blocker | mrv2 | Jason Lowe | Jason Lowe | +| [MAPREDUCE-5513](https://issues.apache.org/jira/browse/MAPREDUCE-5513) | ConcurrentModificationException in JobControl | Major | . | Jason Lowe | Robert Parker | +| [MAPREDUCE-5569](https://issues.apache.org/jira/browse/MAPREDUCE-5569) | FloatSplitter is not generating correct splits | Major | . | Nathan Roberts | Nathan Roberts | | [YARN-155](https://issues.apache.org/jira/browse/YARN-155) | TestAppManager intermittently fails with jdk7 | Major | . | Thomas Graves | Thomas Graves | +| [MAPREDUCE-5585](https://issues.apache.org/jira/browse/MAPREDUCE-5585) | TestCopyCommitter#testNoCommitAction Fails on JDK7 | Major | . | Jonathan Eagles | Jonathan Eagles | +| [MAPREDUCE-5586](https://issues.apache.org/jira/browse/MAPREDUCE-5586) | TestCopyMapper#testCopyFailOnBlockSizeDifference fails when run from hadoop-tools/hadoop-distcp directory | Major | . | Jonathan Eagles | Jonathan Eagles | +| [HDFS-5346](https://issues.apache.org/jira/browse/HDFS-5346) | Avoid unnecessary call to getNumLiveDataNodes() for each block during IBR processing | Major | namenode, performance | Kihwal Lee | Ravi Prakash | +| [MAPREDUCE-5587](https://issues.apache.org/jira/browse/MAPREDUCE-5587) | TestTextOutputFormat fails on JDK7 | Major | . | Jonathan Eagles | Jonathan Eagles | +| [MAPREDUCE-5561](https://issues.apache.org/jira/browse/MAPREDUCE-5561) | org.apache.hadoop.mapreduce.v2.app.job.impl.TestJobImpl testcase failing on trunk | Critical | . | Cindy Li | Karthik Kambatla | +| [YARN-1031](https://issues.apache.org/jira/browse/YARN-1031) | JQuery UI components reference external css in branch-23 | Major | . | Jonathan Eagles | Jonathan Eagles | +| [YARN-1386](https://issues.apache.org/jira/browse/YARN-1386) | NodeManager mistakenly loses resources and relocalizes them | Critical | nodemanager | Jason Lowe | Jason Lowe | +| [HADOOP-10091](https://issues.apache.org/jira/browse/HADOOP-10091) | Job with a har archive as input fails on 0.23 | Blocker | fs | Jason Lowe | Jason Lowe | +| [HDFS-4995](https://issues.apache.org/jira/browse/HDFS-4995) | Make getContentSummary() less expensive | Major | namenode | Kihwal Lee | Kihwal Lee | +| [HDFS-5438](https://issues.apache.org/jira/browse/HDFS-5438) | Flaws in block report processing can cause data loss | Critical | namenode | Kihwal Lee | Kihwal Lee | +| [MAPREDUCE-5373](https://issues.apache.org/jira/browse/MAPREDUCE-5373) | TestFetchFailure.testFetchFailureMultipleReduces could fail intermittently | Major | . | Chuan Liu | Jonathan Eagles | +| [YARN-1419](https://issues.apache.org/jira/browse/YARN-1419) | TestFifoScheduler.testAppAttemptMetrics fails intermittently under jdk7 | Minor | scheduler | Jonathan Eagles | Jonathan Eagles | +| [HDFS-5557](https://issues.apache.org/jira/browse/HDFS-5557) | Write pipeline recovery for the last packet in the block may cause rejection of valid replicas | Critical | . | Kihwal Lee | Kihwal Lee | +| [HDFS-5526](https://issues.apache.org/jira/browse/HDFS-5526) | Datanode cannot roll back to previous layout version | Blocker | datanode | Tsz Wo Nicholas Sze | Kihwal Lee | +| [HDFS-5558](https://issues.apache.org/jira/browse/HDFS-5558) | LeaseManager monitor thread can crash if the last block is complete but another block is not. | Major | . | Kihwal Lee | Kihwal Lee | ### TESTS: | JIRA | Summary | Priority | Component | Reporter | Contributor | |:---- |:---- | :--- |:---- |:---- |:---- | -| [MAPREDUCE-5610](https://issues.apache.org/jira/browse/MAPREDUCE-5610) | TestSleepJob fails in jdk7 | Major | . | Jonathan Eagles | Jonathan Eagles | | [YARN-1119](https://issues.apache.org/jira/browse/YARN-1119) | Add ClusterMetrics checks to tho TestRMNodeTransitions tests | Major | resourcemanager | Robert Parker | Mit Desai | +| [MAPREDUCE-5610](https://issues.apache.org/jira/browse/MAPREDUCE-5610) | TestSleepJob fails in jdk7 | Major | . | Jonathan Eagles | Jonathan Eagles | ### SUB-TASKS: | JIRA | Summary | Priority | Component | Reporter | Contributor | |:---- |:---- | :--- |:---- |:---- |:---- | -| [HADOOP-9956](https://issues.apache.org/jira/browse/HADOOP-9956) | RPC listener inefficiently assigns connections to readers | Major | ipc | Daryn Sharp | Daryn Sharp | -| [HADOOP-9955](https://issues.apache.org/jira/browse/HADOOP-9955) | RPC idle connection closing is extremely inefficient | Major | ipc | Daryn Sharp | Daryn Sharp | | [HADOOP-9748](https://issues.apache.org/jira/browse/HADOOP-9748) | Reduce blocking on UGI.ensureInitialized | Critical | security | Daryn Sharp | Daryn Sharp | -| [HDFS-5239](https://issues.apache.org/jira/browse/HDFS-5239) | Allow FSNamesystem lock fairness to be configurable | Major | namenode | Daryn Sharp | Daryn Sharp | | [YARN-573](https://issues.apache.org/jira/browse/YARN-573) | Shared data structures in Public Localizer and Private Localizer are not Thread safe. | Critical | . | Omkar Vinit Joshi | Omkar Vinit Joshi | - - -### OTHER: - -| JIRA | Summary | Priority | Component | Reporter | Contributor | -|:---- |:---- | :--- |:---- |:---- |:---- | +| [HDFS-5239](https://issues.apache.org/jira/browse/HDFS-5239) | Allow FSNamesystem lock fairness to be configurable | Major | namenode | Daryn Sharp | Daryn Sharp | +| [HADOOP-9956](https://issues.apache.org/jira/browse/HADOOP-9956) | RPC listener inefficiently assigns connections to readers | Major | ipc | Daryn Sharp | Daryn Sharp | +| [HADOOP-9955](https://issues.apache.org/jira/browse/HADOOP-9955) | RPC idle connection closing is extremely inefficient | Major | ipc | Daryn Sharp | Daryn Sharp | http://git-wip-us.apache.org/repos/asf/hadoop/blob/19041008/hadoop-common-project/hadoop-common/src/site/markdown/release/0.23.10/RELEASENOTES.0.23.10.md ---------------------------------------------------------------------- diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/release/0.23.10/RELEASENOTES.0.23.10.md b/hadoop-common-project/hadoop-common/src/site/markdown/release/0.23.10/RELEASENOTES.0.23.10.md index be5dc41..6a28435 100644 --- a/hadoop-common-project/hadoop-common/src/site/markdown/release/0.23.10/RELEASENOTES.0.23.10.md +++ b/hadoop-common-project/hadoop-common/src/site/markdown/release/0.23.10/RELEASENOTES.0.23.10.md @@ -25,7 +25,7 @@ These release notes cover new developer and user-facing incompatibilities, impor * [YARN-707](https://issues.apache.org/jira/browse/YARN-707) | *Blocker* | **Add user info in the YARN ClientToken** -**WARNING: No release note provided for this incompatible change.** +**WARNING: No release note provided for this change.** http://git-wip-us.apache.org/repos/asf/hadoop/blob/19041008/hadoop-common-project/hadoop-common/src/site/markdown/release/0.23.11/CHANGES.0.23.11.md ---------------------------------------------------------------------- diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/release/0.23.11/CHANGES.0.23.11.md b/hadoop-common-project/hadoop-common/src/site/markdown/release/0.23.11/CHANGES.0.23.11.md index 8313a69..436bb1b 100644 --- a/hadoop-common-project/hadoop-common/src/site/markdown/release/0.23.11/CHANGES.0.23.11.md +++ b/hadoop-common-project/hadoop-common/src/site/markdown/release/0.23.11/CHANGES.0.23.11.md @@ -20,79 +20,57 @@ ## Release 0.23.11 - 2014-06-25 -### INCOMPATIBLE CHANGES: - -| JIRA | Summary | Priority | Component | Reporter | Contributor | -|:---- |:---- | :--- |:---- |:---- |:---- | - - -### IMPORTANT ISSUES: - -| JIRA | Summary | Priority | Component | Reporter | Contributor | -|:---- |:---- | :--- |:---- |:---- |:---- | - - -### NEW FEATURES: - -| JIRA | Summary | Priority | Component | Reporter | Contributor | -|:---- |:---- | :--- |:---- |:---- |:---- | ### IMPROVEMENTS: | JIRA | Summary | Priority | Component | Reporter | Contributor | |:---- |:---- | :--- |:---- |:---- |:---- | -| [HADOOP-10454](https://issues.apache.org/jira/browse/HADOOP-10454) | Provide FileContext version of har file system | Major | . | Kihwal Lee | Kihwal Lee | -| [HADOOP-10164](https://issues.apache.org/jira/browse/HADOOP-10164) | Allow UGI to login with a known Subject | Major | . | Robert Joseph Evans | Robert Joseph Evans | | [HADOOP-7688](https://issues.apache.org/jira/browse/HADOOP-7688) | When a servlet filter throws an exception in init(..), the Jetty server failed silently. | Major | . | Tsz Wo Nicholas Sze | Uma Maheswara Rao G | -| [HDFS-6191](https://issues.apache.org/jira/browse/HDFS-6191) | Disable quota checks when replaying edit log. | Major | namenode | Kihwal Lee | Kihwal Lee | +| [HDFS-4461](https://issues.apache.org/jira/browse/HDFS-4461) | DirectoryScanner: volume path prefix takes up memory for every block that is scanned | Minor | . | Colin P. McCabe | Colin P. McCabe | | [HDFS-5637](https://issues.apache.org/jira/browse/HDFS-5637) | try to refeatchToken while local read InvalidToken occurred | Major | hdfs-client, security | Liang Xie | Liang Xie | -| [HDFS-4461](https://issues.apache.org/jira/browse/HDFS-4461) | DirectoryScanner: volume path prefix takes up memory for every block that is scanned | Minor | . | Colin Patrick McCabe | Colin Patrick McCabe | +| [HADOOP-10164](https://issues.apache.org/jira/browse/HADOOP-10164) | Allow UGI to login with a known Subject | Major | . | Robert Joseph Evans | Robert Joseph Evans | +| [HADOOP-10454](https://issues.apache.org/jira/browse/HADOOP-10454) | Provide FileContext version of har file system | Major | . | Kihwal Lee | Kihwal Lee | +| [HDFS-6191](https://issues.apache.org/jira/browse/HDFS-6191) | Disable quota checks when replaying edit log. | Major | namenode | Kihwal Lee | Kihwal Lee | ### BUG FIXES: | JIRA | Summary | Priority | Component | Reporter | Contributor | |:---- |:---- | :--- |:---- |:---- |:---- | -| [HADOOP-10588](https://issues.apache.org/jira/browse/HADOOP-10588) | Workaround for jetty6 acceptor startup issue | Major | . | Kihwal Lee | Kihwal Lee | -| [HADOOP-10332](https://issues.apache.org/jira/browse/HADOOP-10332) | HttpServer's jetty audit log always logs 200 OK | Major | . | Daryn Sharp | Jonathan Eagles | -| [HADOOP-10146](https://issues.apache.org/jira/browse/HADOOP-10146) | Workaround JDK7 Process fd close bug | Critical | util | Daryn Sharp | Daryn Sharp | +| [HADOOP-9230](https://issues.apache.org/jira/browse/HADOOP-9230) | TestUniformSizeInputFormat fails intermittently | Major | test | Karthik Kambatla | Karthik Kambatla | +| [YARN-500](https://issues.apache.org/jira/browse/YARN-500) | ResourceManager webapp is using next port if configured port is already in use | Major | resourcemanager | Nishan Shetty | Kenji Kikushima | +| [YARN-853](https://issues.apache.org/jira/browse/YARN-853) | maximum-am-resource-percent doesn't work after refreshQueues command | Major | capacityscheduler | Devaraj K | Devaraj K | +| [MAPREDUCE-5454](https://issues.apache.org/jira/browse/MAPREDUCE-5454) | TestDFSIO fails intermittently on JDK7 | Major | test | Karthik Kambatla | Karthik Kambatla | +| [YARN-1053](https://issues.apache.org/jira/browse/YARN-1053) | Diagnostic message from ContainerExitEvent is ignored in ContainerImpl | Blocker | . | Omkar Vinit Joshi | Omkar Vinit Joshi | | [HADOOP-10129](https://issues.apache.org/jira/browse/HADOOP-10129) | Distcp may succeed when it fails | Critical | tools/distcp | Daryn Sharp | Daryn Sharp | -| [HADOOP-10112](https://issues.apache.org/jira/browse/HADOOP-10112) | har file listing doesn't work with wild card | Major | tools | Brandon Li | Brandon Li | -| [HADOOP-10110](https://issues.apache.org/jira/browse/HADOOP-10110) | hadoop-auth has a build break due to missing dependency | Blocker | build | Chuan Liu | Chuan Liu | | [HADOOP-10081](https://issues.apache.org/jira/browse/HADOOP-10081) | Client.setupIOStreams can leak socket resources on exception or error | Critical | ipc | Jason Lowe | Tsuyoshi Ozawa | -| [HADOOP-9230](https://issues.apache.org/jira/browse/HADOOP-9230) | TestUniformSizeInputFormat fails intermittently | Major | test | Karthik Kambatla | Karthik Kambatla | -| [HADOOP-8826](https://issues.apache.org/jira/browse/HADOOP-8826) | Docs still refer to 0.20.205 as stable line | Minor | . | Robert Joseph Evans | Mit Desai | -| [HDFS-6449](https://issues.apache.org/jira/browse/HDFS-6449) | Incorrect counting in ContentSummaryComputationContext in 0.23. | Critical | . | Kihwal Lee | Kihwal Lee | -| [HDFS-6166](https://issues.apache.org/jira/browse/HDFS-6166) | revisit balancer so\_timeout | Blocker | balancer & mover | Nathan Roberts | Nathan Roberts | -| [HDFS-5881](https://issues.apache.org/jira/browse/HDFS-5881) | Fix skip() of the short-circuit local reader (legacy). | Critical | . | Kihwal Lee | Kihwal Lee | +| [MAPREDUCE-5623](https://issues.apache.org/jira/browse/MAPREDUCE-5623) | TestJobCleanup fails because of RejectedExecutionException and NPE. | Major | . | Tsuyoshi Ozawa | Jason Lowe | +| [YARN-1145](https://issues.apache.org/jira/browse/YARN-1145) | Potential file handle leak in aggregated logs web ui | Major | . | Rohith Sharma K S | Rohith Sharma K S | +| [YARN-1180](https://issues.apache.org/jira/browse/YARN-1180) | Update capacity scheduler docs to include types on the configs | Trivial | capacityscheduler | Thomas Graves | Chen He | +| [MAPREDUCE-5689](https://issues.apache.org/jira/browse/MAPREDUCE-5689) | MRAppMaster does not preempt reducers when scheduled maps cannot be fulfilled | Critical | . | Lohit Vijayarenu | Lohit Vijayarenu | +| [HADOOP-10146](https://issues.apache.org/jira/browse/HADOOP-10146) | Workaround JDK7 Process fd close bug | Critical | util | Daryn Sharp | Daryn Sharp | +| [YARN-1592](https://issues.apache.org/jira/browse/YARN-1592) | CapacityScheduler tries to reserve more than a node's total memory on branch-0.23 | Major | capacityscheduler | Thomas Graves | Thomas Graves | +| [HADOOP-10110](https://issues.apache.org/jira/browse/HADOOP-10110) | hadoop-auth has a build break due to missing dependency | Blocker | build | Chuan Liu | Chuan Liu | | [HDFS-5806](https://issues.apache.org/jira/browse/HDFS-5806) | balancer should set SoTimeout to avoid indefinite hangs | Major | balancer & mover | Nathan Roberts | Nathan Roberts | | [HDFS-5728](https://issues.apache.org/jira/browse/HDFS-5728) | [Diskfull] Block recovery will fail if the metafile does not have crc for all chunks of the block | Critical | datanode | Vinayakumar B | Vinayakumar B | -| [HDFS-4576](https://issues.apache.org/jira/browse/HDFS-4576) | Webhdfs authentication issues | Major | webhdfs | Daryn Sharp | Daryn Sharp | -| [MAPREDUCE-5789](https://issues.apache.org/jira/browse/MAPREDUCE-5789) | Average Reduce time is incorrect on Job Overview page | Major | jobhistoryserver, webapps | Rushabh S Shah | Rushabh S Shah | -| [MAPREDUCE-5778](https://issues.apache.org/jira/browse/MAPREDUCE-5778) | JobSummary does not escape newlines in the job name | Major | jobhistoryserver | Jason Lowe | Akira AJISAKA | -| [MAPREDUCE-5757](https://issues.apache.org/jira/browse/MAPREDUCE-5757) | ConcurrentModificationException in JobControl.toList | Major | client | Jason Lowe | Jason Lowe | -| [MAPREDUCE-5746](https://issues.apache.org/jira/browse/MAPREDUCE-5746) | Job diagnostics can implicate wrong task for a failed job | Major | jobhistoryserver | Jason Lowe | Jason Lowe | +| [HDFS-5881](https://issues.apache.org/jira/browse/HDFS-5881) | Fix skip() of the short-circuit local reader (legacy). | Critical | . | Kihwal Lee | Kihwal Lee | +| [HADOOP-10112](https://issues.apache.org/jira/browse/HADOOP-10112) | har file listing doesn't work with wild card | Major | tools | Brandon Li | Brandon Li | | [MAPREDUCE-5744](https://issues.apache.org/jira/browse/MAPREDUCE-5744) | Job hangs because RMContainerAllocator$AssignedRequests.preemptReduce() violates the comparator contract | Blocker | . | Sangjin Lee | Gera Shegalov | -| [MAPREDUCE-5689](https://issues.apache.org/jira/browse/MAPREDUCE-5689) | MRAppMaster does not preempt reducers when scheduled maps cannot be fulfilled | Critical | . | Lohit Vijayarenu | Lohit Vijayarenu | -| [MAPREDUCE-5623](https://issues.apache.org/jira/browse/MAPREDUCE-5623) | TestJobCleanup fails because of RejectedExecutionException and NPE. | Major | . | Tsuyoshi Ozawa | Jason Lowe | -| [MAPREDUCE-5454](https://issues.apache.org/jira/browse/MAPREDUCE-5454) | TestDFSIO fails intermittently on JDK7 | Major | test | Karthik Kambatla | Karthik Kambatla | +| [MAPREDUCE-5746](https://issues.apache.org/jira/browse/MAPREDUCE-5746) | Job diagnostics can implicate wrong task for a failed job | Major | jobhistoryserver | Jason Lowe | Jason Lowe | +| [MAPREDUCE-5757](https://issues.apache.org/jira/browse/MAPREDUCE-5757) | ConcurrentModificationException in JobControl.toList | Major | client | Jason Lowe | Jason Lowe | +| [MAPREDUCE-5778](https://issues.apache.org/jira/browse/MAPREDUCE-5778) | JobSummary does not escape newlines in the job name | Major | jobhistoryserver | Jason Lowe | Akira Ajisaka | +| [MAPREDUCE-5789](https://issues.apache.org/jira/browse/MAPREDUCE-5789) | Average Reduce time is incorrect on Job Overview page | Major | jobhistoryserver, webapps | Rushabh S Shah | Rushabh S Shah | +| [HADOOP-10332](https://issues.apache.org/jira/browse/HADOOP-10332) | HttpServer's jetty audit log always logs 200 OK | Major | . | Daryn Sharp | Jonathan Eagles | +| [YARN-1670](https://issues.apache.org/jira/browse/YARN-1670) | aggregated log writer can write more log data then it says is the log length | Critical | . | Thomas Graves | Mit Desai | +| [HDFS-6166](https://issues.apache.org/jira/browse/HDFS-6166) | revisit balancer so\_timeout | Blocker | balancer & mover | Nathan Roberts | Nathan Roberts | +| [HADOOP-8826](https://issues.apache.org/jira/browse/HADOOP-8826) | Docs still refer to 0.20.205 as stable line | Minor | . | Robert Joseph Evans | Mit Desai | +| [HDFS-4576](https://issues.apache.org/jira/browse/HDFS-4576) | Webhdfs authentication issues | Major | webhdfs | Daryn Sharp | Daryn Sharp | | [MAPREDUCE-3191](https://issues.apache.org/jira/browse/MAPREDUCE-3191) | docs for map output compression incorrectly reference SequenceFile | Trivial | . | Todd Lipcon | Chen He | -| [YARN-3829](https://issues.apache.org/jira/browse/YARN-3829) | The History Tracking UI is broken for Tez application on ResourceManager WebUI | Critical | applications | Irina Easterling | | | [YARN-1932](https://issues.apache.org/jira/browse/YARN-1932) | Javascript injection on the job status page | Blocker | . | Mit Desai | Mit Desai | -| [YARN-1670](https://issues.apache.org/jira/browse/YARN-1670) | aggregated log writer can write more log data then it says is the log length | Critical | . | Thomas Graves | Mit Desai | -| [YARN-1592](https://issues.apache.org/jira/browse/YARN-1592) | CapacityScheduler tries to reserve more than a node's total memory on branch-0.23 | Major | capacityscheduler | Thomas Graves | Thomas Graves | -| [YARN-1180](https://issues.apache.org/jira/browse/YARN-1180) | Update capacity scheduler docs to include types on the configs | Trivial | capacityscheduler | Thomas Graves | Chen He | -| [YARN-1145](https://issues.apache.org/jira/browse/YARN-1145) | Potential file handle leak in aggregated logs web ui | Major | . | Rohith Sharma K S | Rohith Sharma K S | -| [YARN-1053](https://issues.apache.org/jira/browse/YARN-1053) | Diagnostic message from ContainerExitEvent is ignored in ContainerImpl | Blocker | . | Omkar Vinit Joshi | Omkar Vinit Joshi | -| [YARN-853](https://issues.apache.org/jira/browse/YARN-853) | maximum-am-resource-percent doesn't work after refreshQueues command | Major | capacityscheduler | Devaraj K | Devaraj K | -| [YARN-500](https://issues.apache.org/jira/browse/YARN-500) | ResourceManager webapp is using next port if configured port is already in use | Major | resourcemanager | Nishan Shetty | Kenji Kikushima | - - -### TESTS: - -| JIRA | Summary | Priority | Component | Reporter | Contributor | -|:---- |:---- | :--- |:---- |:---- |:---- | +| [HADOOP-10588](https://issues.apache.org/jira/browse/HADOOP-10588) | Workaround for jetty6 acceptor startup issue | Major | . | Kihwal Lee | Kihwal Lee | +| [HDFS-6449](https://issues.apache.org/jira/browse/HDFS-6449) | Incorrect counting in ContentSummaryComputationContext in 0.23. | Critical | . | Kihwal Lee | Kihwal Lee | +| [YARN-3829](https://issues.apache.org/jira/browse/YARN-3829) | The History Tracking UI is broken for Tez application on ResourceManager WebUI | Critical | applications | Irina Easterling | | ### SUB-TASKS: @@ -103,9 +81,3 @@ | [YARN-1575](https://issues.apache.org/jira/browse/YARN-1575) | Public localizer crashes with "Localized unkown resource" | Critical | nodemanager | Jason Lowe | Jason Lowe | -### OTHER: - -| JIRA | Summary | Priority | Component | Reporter | Contributor | -|:---- |:---- | :--- |:---- |:---- |:---- | - - --------------------------------------------------------------------- To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
