http://git-wip-us.apache.org/repos/asf/hadoop/blob/19041008/hadoop-common-project/hadoop-common/src/site/markdown/release/1.2.0/RELEASENOTES.1.2.0.md ---------------------------------------------------------------------- diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/release/1.2.0/RELEASENOTES.1.2.0.md b/hadoop-common-project/hadoop-common/src/site/markdown/release/1.2.0/RELEASENOTES.1.2.0.md index 3fa573c..e7aad10 100644 --- a/hadoop-common-project/hadoop-common/src/site/markdown/release/1.2.0/RELEASENOTES.1.2.0.md +++ b/hadoop-common-project/hadoop-common/src/site/markdown/release/1.2.0/RELEASENOTES.1.2.0.md @@ -23,154 +23,154 @@ These release notes cover new developer and user-facing incompatibilities, impor --- -* [HADOOP-8971](https://issues.apache.org/jira/browse/HADOOP-8971) | *Major* | **Backport: hadoop.util.PureJavaCrc32 cache hit-ratio is low for static data (HADOOP-8926)** +* [HDFS-385](https://issues.apache.org/jira/browse/HDFS-385) | *Major* | **Design a pluggable interface to place replicas of blocks in HDFS** -Backport cache-aware improvements for PureJavaCrc32 from trunk (HADOOP-8926) +New experimental API BlockPlacementPolicy allows investigating alternate rules for locating block replicas. --- -* [HADOOP-8817](https://issues.apache.org/jira/browse/HADOOP-8817) | *Major* | **Backport Network Topology Extension for Virtualization (HADOOP-8468) to branch-1** +* [HADOOP-8164](https://issues.apache.org/jira/browse/HADOOP-8164) | *Major* | **Handle paths using back slash as path separator for windows only** -A new 4-layer network topology NetworkToplogyWithNodeGroup is available to make Hadoop more robust and efficient in virtualized environment. +This jira only allows providing paths using back slash as separator on Windows. The back slash on \*nix system will be used as escape character. The support for paths using back slash as path separator will be removed in HADOOP-8139 in release 23.3. --- -* [HADOOP-8470](https://issues.apache.org/jira/browse/HADOOP-8470) | *Major* | **Implementation of 4-layer subclass of NetworkTopology (NetworkTopologyWithNodeGroup)** +* [MAPREDUCE-4415](https://issues.apache.org/jira/browse/MAPREDUCE-4415) | *Major* | **Backport the Job.getInstance methods from MAPREDUCE-1505 to branch-1** -This patch should be checked in together (or after) with JIRA Hadoop-8469: https://issues.apache.org/jira/browse/HADOOP-8469 +Backported new APIs to get a Job object to 1.2.0 from 2.0.0. Job API static methods Job.getInstance(), Job.getInstance(Configuration) and Job.getInstance(Configuration, jobName) are now available across both releases to avoid porting pain. --- -* [HADOOP-8164](https://issues.apache.org/jira/browse/HADOOP-8164) | *Major* | **Handle paths using back slash as path separator for windows only** +* [HDFS-3697](https://issues.apache.org/jira/browse/HDFS-3697) | *Minor* | **Enable fadvise readahead by default** -This jira only allows providing paths using back slash as separator on Windows. The back slash on \*nix system will be used as escape character. The support for paths using back slash as path separator will be removed in HADOOP-8139 in release 23.3. +The datanode now performs 4MB readahead by default when reading data from its disks, if the native libraries are present. This has been shown to improve performance in many workloads. The feature may be disabled by setting dfs.datanode.readahead.bytes to "0". --- -* [HADOOP-7698](https://issues.apache.org/jira/browse/HADOOP-7698) | *Critical* | **jsvc target fails on x86\_64** +* [MAPREDUCE-4565](https://issues.apache.org/jira/browse/MAPREDUCE-4565) | *Major* | **Backport MR-2855 to branch-1: ResourceBundle lookup during counter name resolution takes a lot of time** -The jsvc build target is now supported for Mac OSX and other platforms as well. +Passing a cached class-loader to ResourceBundle creator to minimize counter names lookup time. --- -* [HDFS-4519](https://issues.apache.org/jira/browse/HDFS-4519) | *Major* | **Support override of jsvc binary and log file locations when launching secure datanode.** - -With this improvement the following options are available in release 1.2.0 and later on 1.x release stream: -1. jsvc location can be overridden by setting environment variable JSVC\_HOME. Defaults to jsvc binary packaged within the Hadoop distro. -2. jsvc log output is directed to the file defined by JSVC\_OUTFILE. Defaults to $HADOOP\_LOG\_DIR/jsvc.out. -3. jsvc error output is directed to the file defined by JSVC\_ERRFILE file. Defaults to $HADOOP\_LOG\_DIR/jsvc.err. - -With this improvement the following options are available in release 2.0.4 and later on 2.x release stream: -1. jsvc log output is directed to the file defined by JSVC\_OUTFILE. Defaults to $HADOOP\_LOG\_DIR/jsvc.out. -2. jsvc error output is directed to the file defined by JSVC\_ERRFILE file. Defaults to $HADOOP\_LOG\_DIR/jsvc.err. +* [MAPREDUCE-4629](https://issues.apache.org/jira/browse/MAPREDUCE-4629) | *Major* | **Remove JobHistory.DEBUG\_MODE** -For overriding jsvc location on 2.x releases, here is the release notes from HDFS-2303: -To run secure Datanodes users must install jsvc for their platform and set JSVC\_HOME to point to the location of jsvc in their environment. +**WARNING: No release note provided for this change.** --- -* [HDFS-4350](https://issues.apache.org/jira/browse/HDFS-4350) | *Major* | **Make enabling of stale marking on read and write paths independent** - -This patch makes an incompatible configuration change, as described below: -In releases 1.1.0 and other point releases 1.1.x, the configuration parameter "dfs.namenode.check.stale.datanode" could be used to turn on checking for the stale nodes. This configuration is no longer supported in release 1.2.0 onwards and is renamed as "dfs.namenode.avoid.read.stale.datanode". +* [HADOOP-7698](https://issues.apache.org/jira/browse/HADOOP-7698) | *Critical* | **jsvc target fails on x86\_64** -How feature works and configuring this feature: -As described in HDFS-3703 release notes, datanode stale period can be configured using parameter "dfs.namenode.stale.datanode.interval" in seconds (default value is 30 seconds). NameNode can be configured to use this staleness information for reads using configuration "dfs.namenode.avoid.read.stale.datanode". When this parameter is set to true, namenode picks a stale datanode as the last target to read from when returning block locations for reads. Using staleness information for writes is as described in the releases notes of HDFS-3912. +The jsvc build target is now supported for Mac OSX and other platforms as well. --- -* [HDFS-4337](https://issues.apache.org/jira/browse/HDFS-4337) | *Major* | **Backport HDFS-4240 to branch-1: Make sure nodes are avoided to place replica if some replica are already under the same nodegroup.** +* [MAPREDUCE-3678](https://issues.apache.org/jira/browse/MAPREDUCE-3678) | *Major* | **The Map tasks logs should have the value of input split it processed** -Backport HDFS-4240 to branch-1 +A map-task's syslogs now carries basic info on the InputSplit it processed. --- -* [HDFS-4320](https://issues.apache.org/jira/browse/HDFS-4320) | *Major* | **Add a separate configuration for namenode rpc address instead of only using fs.default.name** - -The namenode RPC address is currently identified from configuration "fs.default.name". In some setups where default FS is other than HDFS, the "fs.default.name" cannot be used to get the namenode address. When such a setup co-exists with HDFS, with this change namenode can be identified using a separate configuration parameter "dfs.namenode.rpc-address". +* [MAPREDUCE-4451](https://issues.apache.org/jira/browse/MAPREDUCE-4451) | *Major* | **fairscheduler fail to init job with kerberos authentication configured** -"dfs.namenode.rpc-address", when configured, overrides fs.default.name for identifying namenode RPC address. +Using FairScheduler with security configured, job initialization fails. The problem is that threads in JobInitializer runs as RPC user instead of jobtracker, pre-start all the threads fix this bug --- -* [HDFS-4122](https://issues.apache.org/jira/browse/HDFS-4122) | *Major* | **Cleanup HDFS logs and reduce the size of logged messages** +* [HDFS-4071](https://issues.apache.org/jira/browse/HDFS-4071) | *Minor* | **Add number of stale DataNodes to metrics for Branch-1** -The change from this jira changes the content of some of the log messages. No log message are removed. Only the content of the log messages is changed to reduce the size. If you have a tool that depends on the exact content of the log, please look at the patch and make appropriate updates to the tool. +This jira adds a new metric with name "StaleDataNodes" under metrics context "dfs" of type Gauge. This tracks the number of DataNodes marked as stale. A DataNode is marked stale when the heartbeat message from the DataNode is not received within the configured time ""dfs.namenode.stale.datanode.interval". ---- +Please see hdfs-default.xml documentation corresponding to "dfs.namenode.stale.datanode.interval" for more details on how to configure this feature. When this feature is not configured, this metrics would return zero. -* [HDFS-4071](https://issues.apache.org/jira/browse/HDFS-4071) | *Minor* | **Add number of stale DataNodes to metrics for Branch-1** -This jira adds a new metric with name "StaleDataNodes" under metrics context "dfs" of type Gauge. This tracks the number of DataNodes marked as stale. A DataNode is marked stale when the heartbeat message from the DataNode is not received within the configured time ""dfs.namenode.stale.datanode.interval". +--- +* [HADOOP-8971](https://issues.apache.org/jira/browse/HADOOP-8971) | *Major* | **Backport: hadoop.util.PureJavaCrc32 cache hit-ratio is low for static data (HADOOP-8926)** -Please see hdfs-default.xml documentation corresponding to "dfs.namenode.stale.datanode.interval" for more details on how to configure this feature. When this feature is not configured, this metrics would return zero. +Backport cache-aware improvements for PureJavaCrc32 from trunk (HADOOP-8926) --- -* [HDFS-3697](https://issues.apache.org/jira/browse/HDFS-3697) | *Minor* | **Enable fadvise readahead by default** +* [HDFS-4122](https://issues.apache.org/jira/browse/HDFS-4122) | *Major* | **Cleanup HDFS logs and reduce the size of logged messages** -The datanode now performs 4MB readahead by default when reading data from its disks, if the native libraries are present. This has been shown to improve performance in many workloads. The feature may be disabled by setting dfs.datanode.readahead.bytes to "0". +The change from this jira changes the content of some of the log messages. No log message are removed. Only the content of the log messages is changed to reduce the size. If you have a tool that depends on the exact content of the log, please look at the patch and make appropriate updates to the tool. --- -* [HDFS-385](https://issues.apache.org/jira/browse/HDFS-385) | *Major* | **Design a pluggable interface to place replicas of blocks in HDFS** +* [HDFS-4320](https://issues.apache.org/jira/browse/HDFS-4320) | *Major* | **Add a separate configuration for namenode rpc address instead of only using fs.default.name** -New experimental API BlockPlacementPolicy allows investigating alternate rules for locating block replicas. +The namenode RPC address is currently identified from configuration "fs.default.name". In some setups where default FS is other than HDFS, the "fs.default.name" cannot be used to get the namenode address. When such a setup co-exists with HDFS, with this change namenode can be identified using a separate configuration parameter "dfs.namenode.rpc-address". + +"dfs.namenode.rpc-address", when configured, overrides fs.default.name for identifying namenode RPC address. --- -* [MAPREDUCE-4737](https://issues.apache.org/jira/browse/MAPREDUCE-4737) | *Major* | ** Hadoop does not close output file / does not call Mapper.cleanup if exception in map** +* [HDFS-4337](https://issues.apache.org/jira/browse/HDFS-4337) | *Major* | **Backport HDFS-4240 to branch-1: Make sure nodes are avoided to place replica if some replica are already under the same nodegroup.** -Ensure that mapreduce APIs are semantically consistent with mapred API w.r.t Mapper.cleanup and Reducer.cleanup; in the sense that cleanup is now called even if there is an error. The old mapred API already ensures that Mapper.close and Reducer.close are invoked during error handling. Note that it is an incompatible change, however end-users can override Mapper.run and Reducer.run to get the old (inconsistent) behaviour. +Backport HDFS-4240 to branch-1 --- -* [MAPREDUCE-4629](https://issues.apache.org/jira/browse/MAPREDUCE-4629) | *Major* | **Remove JobHistory.DEBUG\_MODE** +* [HDFS-4350](https://issues.apache.org/jira/browse/HDFS-4350) | *Major* | **Make enabling of stale marking on read and write paths independent** + +This patch makes an incompatible configuration change, as described below: +In releases 1.1.0 and other point releases 1.1.x, the configuration parameter "dfs.namenode.check.stale.datanode" could be used to turn on checking for the stale nodes. This configuration is no longer supported in release 1.2.0 onwards and is renamed as "dfs.namenode.avoid.read.stale.datanode". -**WARNING: No release note provided for this incompatible change.** +How feature works and configuring this feature: +As described in HDFS-3703 release notes, datanode stale period can be configured using parameter "dfs.namenode.stale.datanode.interval" in seconds (default value is 30 seconds). NameNode can be configured to use this staleness information for reads using configuration "dfs.namenode.avoid.read.stale.datanode". When this parameter is set to true, namenode picks a stale datanode as the last target to read from when returning block locations for reads. Using staleness information for writes is as described in the releases notes of HDFS-3912. --- -* [MAPREDUCE-4565](https://issues.apache.org/jira/browse/MAPREDUCE-4565) | *Major* | **Backport MR-2855 to branch-1: ResourceBundle lookup during counter name resolution takes a lot of time** +* [HDFS-4519](https://issues.apache.org/jira/browse/HDFS-4519) | *Major* | **Support override of jsvc binary and log file locations when launching secure datanode.** -Passing a cached class-loader to ResourceBundle creator to minimize counter names lookup time. +With this improvement the following options are available in release 1.2.0 and later on 1.x release stream: +1. jsvc location can be overridden by setting environment variable JSVC\_HOME. Defaults to jsvc binary packaged within the Hadoop distro. +2. jsvc log output is directed to the file defined by JSVC\_OUTFILE. Defaults to $HADOOP\_LOG\_DIR/jsvc.out. +3. jsvc error output is directed to the file defined by JSVC\_ERRFILE file. Defaults to $HADOOP\_LOG\_DIR/jsvc.err. + +With this improvement the following options are available in release 2.0.4 and later on 2.x release stream: +1. jsvc log output is directed to the file defined by JSVC\_OUTFILE. Defaults to $HADOOP\_LOG\_DIR/jsvc.out. +2. jsvc error output is directed to the file defined by JSVC\_ERRFILE file. Defaults to $HADOOP\_LOG\_DIR/jsvc.err. + +For overriding jsvc location on 2.x releases, here is the release notes from HDFS-2303: +To run secure Datanodes users must install jsvc for their platform and set JSVC\_HOME to point to the location of jsvc in their environment. --- -* [MAPREDUCE-4451](https://issues.apache.org/jira/browse/MAPREDUCE-4451) | *Major* | **fairscheduler fail to init job with kerberos authentication configured** +* [HADOOP-8817](https://issues.apache.org/jira/browse/HADOOP-8817) | *Major* | **Backport Network Topology Extension for Virtualization (HADOOP-8468) to branch-1** -Using FairScheduler with security configured, job initialization fails. The problem is that threads in JobInitializer runs as RPC user instead of jobtracker, pre-start all the threads fix this bug +A new 4-layer network topology NetworkToplogyWithNodeGroup is available to make Hadoop more robust and efficient in virtualized environment. --- -* [MAPREDUCE-4415](https://issues.apache.org/jira/browse/MAPREDUCE-4415) | *Major* | **Backport the Job.getInstance methods from MAPREDUCE-1505 to branch-1** +* [MAPREDUCE-4737](https://issues.apache.org/jira/browse/MAPREDUCE-4737) | *Major* | ** Hadoop does not close output file / does not call Mapper.cleanup if exception in map** -Backported new APIs to get a Job object to 1.2.0 from 2.0.0. Job API static methods Job.getInstance(), Job.getInstance(Configuration) and Job.getInstance(Configuration, jobName) are now available across both releases to avoid porting pain. +Ensure that mapreduce APIs are semantically consistent with mapred API w.r.t Mapper.cleanup and Reducer.cleanup; in the sense that cleanup is now called even if there is an error. The old mapred API already ensures that Mapper.close and Reducer.close are invoked during error handling. Note that it is an incompatible change, however end-users can override Mapper.run and Reducer.run to get the old (inconsistent) behaviour. --- -* [MAPREDUCE-3678](https://issues.apache.org/jira/browse/MAPREDUCE-3678) | *Major* | **The Map tasks logs should have the value of input split it processed** +* [HADOOP-8470](https://issues.apache.org/jira/browse/HADOOP-8470) | *Major* | **Implementation of 4-layer subclass of NetworkTopology (NetworkTopologyWithNodeGroup)** -A map-task's syslogs now carries basic info on the InputSplit it processed. +This patch should be checked in together (or after) with JIRA Hadoop-8469: https://issues.apache.org/jira/browse/HADOOP-8469
http://git-wip-us.apache.org/repos/asf/hadoop/blob/19041008/hadoop-common-project/hadoop-common/src/site/markdown/release/1.2.1/CHANGES.1.2.1.md ---------------------------------------------------------------------- diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/release/1.2.1/CHANGES.1.2.1.md b/hadoop-common-project/hadoop-common/src/site/markdown/release/1.2.1/CHANGES.1.2.1.md index 603e235..7a58c16 100644 --- a/hadoop-common-project/hadoop-common/src/site/markdown/release/1.2.1/CHANGES.1.2.1.md +++ b/hadoop-common-project/hadoop-common/src/site/markdown/release/1.2.1/CHANGES.1.2.1.md @@ -20,65 +20,37 @@ ## Release 1.2.1 - 2013-08-01 -### INCOMPATIBLE CHANGES: - -| JIRA | Summary | Priority | Component | Reporter | Contributor | -|:---- |:---- | :--- |:---- |:---- |:---- | - - -### IMPORTANT ISSUES: - -| JIRA | Summary | Priority | Component | Reporter | Contributor | -|:---- |:---- | :--- |:---- |:---- |:---- | - - -### NEW FEATURES: - -| JIRA | Summary | Priority | Component | Reporter | Contributor | -|:---- |:---- | :--- |:---- |:---- |:---- | ### IMPROVEMENTS: | JIRA | Summary | Priority | Component | Reporter | Contributor | |:---- |:---- | :--- |:---- |:---- |:---- | -| [MAPREDUCE-5368](https://issues.apache.org/jira/browse/MAPREDUCE-5368) | Save memory by set capacity, load factor and concurrency level for ConcurrentHashMap in TaskInProgress | Major | mrv1 | zhaoyunjiong | zhaoyunjiong | | [MAPREDUCE-4838](https://issues.apache.org/jira/browse/MAPREDUCE-4838) | Add extra info to JH files | Major | . | Arun C Murthy | Zhijie Shen | +| [MAPREDUCE-5368](https://issues.apache.org/jira/browse/MAPREDUCE-5368) | Save memory by set capacity, load factor and concurrency level for ConcurrentHashMap in TaskInProgress | Major | mrv1 | yunjiong zhao | yunjiong zhao | ### BUG FIXES: | JIRA | Summary | Priority | Component | Reporter | Contributor | |:---- |:---- | :--- |:---- |:---- |:---- | -| [HADOOP-9730](https://issues.apache.org/jira/browse/HADOOP-9730) | fix hadoop.spec to add task-log4j.properties | Major | build | Giridharan Kesavan | Giridharan Kesavan | -| [HADOOP-9665](https://issues.apache.org/jira/browse/HADOOP-9665) | BlockDecompressorStream#decompress will throw EOFException instead of return -1 when EOF | Critical | . | Zhijie Shen | Zhijie Shen | -| [HADOOP-9504](https://issues.apache.org/jira/browse/HADOOP-9504) | MetricsDynamicMBeanBase has concurrency issues in createMBeanInfo | Critical | metrics | Liang Xie | Liang Xie | -| [HDFS-4880](https://issues.apache.org/jira/browse/HDFS-4880) | Diagnostic logging while loading name/edits files | Major | namenode | Arpit Agarwal | Suresh Srinivas | -| [HDFS-4699](https://issues.apache.org/jira/browse/HDFS-4699) | TestPipelinesFailover#testPipelineRecoveryStress fails sporadically | Major | test | Chris Nauroth | Chris Nauroth | | [HDFS-4581](https://issues.apache.org/jira/browse/HDFS-4581) | DataNode#checkDiskError should not be called on network errors | Major | datanode | Rohit Kochar | Rohit Kochar | +| [HDFS-4699](https://issues.apache.org/jira/browse/HDFS-4699) | TestPipelinesFailover#testPipelineRecoveryStress fails sporadically | Major | test | Chris Nauroth | Chris Nauroth | +| [MAPREDUCE-5206](https://issues.apache.org/jira/browse/MAPREDUCE-5206) | JT can show the same job multiple times in Retired Jobs section | Minor | . | Arun C Murthy | Arun C Murthy | +| [MAPREDUCE-5148](https://issues.apache.org/jira/browse/MAPREDUCE-5148) | Syslog missing from Map/Reduce tasks | Major | tasktracker | Yesha Vora | Arun C Murthy | +| [MAPREDUCE-3859](https://issues.apache.org/jira/browse/MAPREDUCE-3859) | CapacityScheduler incorrectly utilizes extra-resources of queue for high-memory jobs | Major | capacity-sched | Sergey Tryuber | Sergey Tryuber | | [HDFS-4261](https://issues.apache.org/jira/browse/HDFS-4261) | TestBalancerWithNodeGroup times out | Major | balancer & mover | Tsz Wo Nicholas Sze | Junping Du | -| [MAPREDUCE-5375](https://issues.apache.org/jira/browse/MAPREDUCE-5375) | Delegation Token renewal exception in jobtracker logs | Critical | . | Venkat Ranganathan | Venkat Ranganathan | -| [MAPREDUCE-5364](https://issues.apache.org/jira/browse/MAPREDUCE-5364) | Deadlock between RenewalTimerTask methods cancel() and run() | Major | . | Karthik Kambatla | Karthik Kambatla | -| [MAPREDUCE-5351](https://issues.apache.org/jira/browse/MAPREDUCE-5351) | JobTracker memory leak caused by CleanupQueue reopening FileSystem | Critical | jobtracker | Sandy Ryza | Sandy Ryza | +| [HDFS-4880](https://issues.apache.org/jira/browse/HDFS-4880) | Diagnostic logging while loading name/edits files | Major | namenode | Arpit Agarwal | Suresh Srinivas | +| [MAPREDUCE-5260](https://issues.apache.org/jira/browse/MAPREDUCE-5260) | Job failed because of JvmManager running into inconsistent state | Major | tasktracker | yunjiong zhao | yunjiong zhao | | [MAPREDUCE-5318](https://issues.apache.org/jira/browse/MAPREDUCE-5318) | Ampersand in JSPUtil.java is not escaped | Minor | jobtracker | Bohou Li | Bohou Li | -| [MAPREDUCE-5260](https://issues.apache.org/jira/browse/MAPREDUCE-5260) | Job failed because of JvmManager running into inconsistent state | Major | tasktracker | zhaoyunjiong | zhaoyunjiong | +| [HADOOP-9665](https://issues.apache.org/jira/browse/HADOOP-9665) | BlockDecompressorStream#decompress will throw EOFException instead of return -1 when EOF | Critical | . | Zhijie Shen | Zhijie Shen | +| [MAPREDUCE-5375](https://issues.apache.org/jira/browse/MAPREDUCE-5375) | Delegation Token renewal exception in jobtracker logs | Critical | . | Venkat Ranganathan | Venkat Ranganathan | +| [HADOOP-9504](https://issues.apache.org/jira/browse/HADOOP-9504) | MetricsDynamicMBeanBase has concurrency issues in createMBeanInfo | Critical | metrics | Liang Xie | Liang Xie | | [MAPREDUCE-5256](https://issues.apache.org/jira/browse/MAPREDUCE-5256) | CombineInputFormat isn't thread safe affecting HiveServer | Major | . | Vinod Kumar Vavilapalli | Vinod Kumar Vavilapalli | -| [MAPREDUCE-5206](https://issues.apache.org/jira/browse/MAPREDUCE-5206) | JT can show the same job multiple times in Retired Jobs section | Minor | . | Arun C Murthy | Arun C Murthy | -| [MAPREDUCE-5148](https://issues.apache.org/jira/browse/MAPREDUCE-5148) | Syslog missing from Map/Reduce tasks | Major | tasktracker | Yesha Vora | Arun C Murthy | +| [MAPREDUCE-5351](https://issues.apache.org/jira/browse/MAPREDUCE-5351) | JobTracker memory leak caused by CleanupQueue reopening FileSystem | Critical | jobtracker | Sandy Ryza | Sandy Ryza | +| [MAPREDUCE-5364](https://issues.apache.org/jira/browse/MAPREDUCE-5364) | Deadlock between RenewalTimerTask methods cancel() and run() | Major | . | Karthik Kambatla | Karthik Kambatla | +| [HADOOP-9730](https://issues.apache.org/jira/browse/HADOOP-9730) | fix hadoop.spec to add task-log4j.properties | Major | build | Giridharan Kesavan | Giridharan Kesavan | | [MAPREDUCE-4490](https://issues.apache.org/jira/browse/MAPREDUCE-4490) | JVM reuse is incompatible with LinuxTaskController (and therefore incompatible with Security) | Critical | task-controller, tasktracker | George Datskos | sam liu | -| [MAPREDUCE-3859](https://issues.apache.org/jira/browse/MAPREDUCE-3859) | CapacityScheduler incorrectly utilizes extra-resources of queue for high-memory jobs | Major | capacity-sched | Sergey Tryuber | Sergey Tryuber | - - -### TESTS: - -| JIRA | Summary | Priority | Component | Reporter | Contributor | -|:---- |:---- | :--- |:---- |:---- |:---- | - - -### SUB-TASKS: - -| JIRA | Summary | Priority | Component | Reporter | Contributor | -|:---- |:---- | :--- |:---- |:---- |:---- | ### OTHER: http://git-wip-us.apache.org/repos/asf/hadoop/blob/19041008/hadoop-common-project/hadoop-common/src/site/markdown/release/1.2.1/RELEASENOTES.1.2.1.md ---------------------------------------------------------------------- diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/release/1.2.1/RELEASENOTES.1.2.1.md b/hadoop-common-project/hadoop-common/src/site/markdown/release/1.2.1/RELEASENOTES.1.2.1.md index 162d153..4fc7ddb 100644 --- a/hadoop-common-project/hadoop-common/src/site/markdown/release/1.2.1/RELEASENOTES.1.2.1.md +++ b/hadoop-common-project/hadoop-common/src/site/markdown/release/1.2.1/RELEASENOTES.1.2.1.md @@ -23,16 +23,16 @@ These release notes cover new developer and user-facing incompatibilities, impor --- -* [HADOOP-10371](https://issues.apache.org/jira/browse/HADOOP-10371) | *Minor* | **The eclipse-plugin cannot work in my environment** +* [MAPREDUCE-3859](https://issues.apache.org/jira/browse/MAPREDUCE-3859) | *Major* | **CapacityScheduler incorrectly utilizes extra-resources of queue for high-memory jobs** -fixed +Fixed wrong CapacityScheduler resource allocation for high memory consumption jobs --- -* [MAPREDUCE-3859](https://issues.apache.org/jira/browse/MAPREDUCE-3859) | *Major* | **CapacityScheduler incorrectly utilizes extra-resources of queue for high-memory jobs** +* [HADOOP-10371](https://issues.apache.org/jira/browse/HADOOP-10371) | *Minor* | **The eclipse-plugin cannot work in my environment** -Fixed wrong CapacityScheduler resource allocation for high memory consumption jobs +fixed http://git-wip-us.apache.org/repos/asf/hadoop/blob/19041008/hadoop-common-project/hadoop-common/src/site/markdown/release/1.2.2/CHANGES.1.2.2.md ---------------------------------------------------------------------- diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/release/1.2.2/CHANGES.1.2.2.md b/hadoop-common-project/hadoop-common/src/site/markdown/release/1.2.2/CHANGES.1.2.2.md index 08cf273..cd67c45 100644 --- a/hadoop-common-project/hadoop-common/src/site/markdown/release/1.2.2/CHANGES.1.2.2.md +++ b/hadoop-common-project/hadoop-common/src/site/markdown/release/1.2.2/CHANGES.1.2.2.md @@ -18,55 +18,15 @@ --> # Apache Hadoop Changelog -## Release 1.2.2 - Unreleased (as of 2016-03-04) +## Release 1.2.2 - Unreleased (as of 2017-08-28) -### INCOMPATIBLE CHANGES: - -| JIRA | Summary | Priority | Component | Reporter | Contributor | -|:---- |:---- | :--- |:---- |:---- |:---- | - - -### IMPORTANT ISSUES: - -| JIRA | Summary | Priority | Component | Reporter | Contributor | -|:---- |:---- | :--- |:---- |:---- |:---- | - - -### NEW FEATURES: - -| JIRA | Summary | Priority | Component | Reporter | Contributor | -|:---- |:---- | :--- |:---- |:---- |:---- | - - -### IMPROVEMENTS: - -| JIRA | Summary | Priority | Component | Reporter | Contributor | -|:---- |:---- | :--- |:---- |:---- |:---- | ### BUG FIXES: | JIRA | Summary | Priority | Component | Reporter | Contributor | |:---- |:---- | :--- |:---- |:---- |:---- | -| [MAPREDUCE-6009](https://issues.apache.org/jira/browse/MAPREDUCE-6009) | Map-only job with new-api runs wrong OutputCommitter when cleanup scheduled in a reduce slot | Blocker | client, job submission | Gera Shegalov | Gera Shegalov | | [MAPREDUCE-5450](https://issues.apache.org/jira/browse/MAPREDUCE-5450) | Unnecessary Configuration instantiation in IFileInputStream slows down merge - Port to branch-1 | Blocker | mrv1 | Stanislav Barton | Stanislav Barton | - - -### TESTS: - -| JIRA | Summary | Priority | Component | Reporter | Contributor | -|:---- |:---- | :--- |:---- |:---- |:---- | - - -### SUB-TASKS: - -| JIRA | Summary | Priority | Component | Reporter | Contributor | -|:---- |:---- | :--- |:---- |:---- |:---- | - - -### OTHER: - -| JIRA | Summary | Priority | Component | Reporter | Contributor | -|:---- |:---- | :--- |:---- |:---- |:---- | +| [MAPREDUCE-6009](https://issues.apache.org/jira/browse/MAPREDUCE-6009) | Map-only job with new-api runs wrong OutputCommitter when cleanup scheduled in a reduce slot | Blocker | client, job submission | Gera Shegalov | Gera Shegalov | http://git-wip-us.apache.org/repos/asf/hadoop/blob/19041008/hadoop-common-project/hadoop-common/src/site/markdown/release/1.3.0/CHANGES.1.3.0.md ---------------------------------------------------------------------- diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/release/1.3.0/CHANGES.1.3.0.md b/hadoop-common-project/hadoop-common/src/site/markdown/release/1.3.0/CHANGES.1.3.0.md index 1a12646..f63021f 100644 --- a/hadoop-common-project/hadoop-common/src/site/markdown/release/1.3.0/CHANGES.1.3.0.md +++ b/hadoop-common-project/hadoop-common/src/site/markdown/release/1.3.0/CHANGES.1.3.0.md @@ -18,7 +18,7 @@ --> # Apache Hadoop Changelog -## Release 1.3.0 - Unreleased (as of 2016-03-04) +## Release 1.3.0 - Unreleased (as of 2017-08-28) ### INCOMPATIBLE CHANGES: @@ -27,111 +27,99 @@ | [MAPREDUCE-5777](https://issues.apache.org/jira/browse/MAPREDUCE-5777) | Support utf-8 text with BOM (byte order marker) | Major | . | bc Wong | zhihai xu | -### IMPORTANT ISSUES: - -| JIRA | Summary | Priority | Component | Reporter | Contributor | -|:---- |:---- | :--- |:---- |:---- |:---- | - - -### NEW FEATURES: - -| JIRA | Summary | Priority | Component | Reporter | Contributor | -|:---- |:---- | :--- |:---- |:---- |:---- | - - ### IMPROVEMENTS: | JIRA | Summary | Priority | Component | Reporter | Contributor | |:---- |:---- | :--- |:---- |:---- |:---- | -| [HADOOP-10614](https://issues.apache.org/jira/browse/HADOOP-10614) | CBZip2InputStream is not threadsafe | Major | . | Xiangrui Meng | Xiangrui Meng | -| [HADOOP-9855](https://issues.apache.org/jira/browse/HADOOP-9855) | Backport HADOOP-6578 to branch-1 | Major | . | James Kinley | James Kinley | +| [MAPREDUCE-2351](https://issues.apache.org/jira/browse/MAPREDUCE-2351) | mapred.job.tracker.history.completed.location should support an arbitrary filesystem URI | Major | . | Tom White | Tom White | | [HADOOP-9450](https://issues.apache.org/jira/browse/HADOOP-9450) | HADOOP\_USER\_CLASSPATH\_FIRST is not honored; CLASSPATH is PREpended instead of APpended | Major | scripts | Mitch Wyle | Harsh J | -| [HADOOP-8873](https://issues.apache.org/jira/browse/HADOOP-8873) | Port HADOOP-8175 (Add mkdir -p flag) to branch-1 | Major | . | Eli Collins | Akira AJISAKA | -| [HDFS-7312](https://issues.apache.org/jira/browse/HDFS-7312) | Update DistCp v1 to optionally not use tmp location (branch-1 only) | Minor | tools | Joseph Prosser | Joseph Prosser | -| [HDFS-5367](https://issues.apache.org/jira/browse/HDFS-5367) | Restoring namenode storage locks namenode due to unnecessary fsimage write | Major | . | zhaoyunjiong | zhaoyunjiong | -| [HDFS-5038](https://issues.apache.org/jira/browse/HDFS-5038) | Backport several branch-2 APIs to branch-1 | Minor | . | Jing Zhao | Jing Zhao | -| [HDFS-4963](https://issues.apache.org/jira/browse/HDFS-4963) | Improve multihoming support in namenode | Major | namenode | Arpit Agarwal | Arpit Agarwal | +| [HDFS-4521](https://issues.apache.org/jira/browse/HDFS-4521) | invalid network topologies should not be cached | Minor | . | Colin P. McCabe | Colin P. McCabe | +| [HADOOP-8873](https://issues.apache.org/jira/browse/HADOOP-8873) | Port HADOOP-8175 (Add mkdir -p flag) to branch-1 | Major | . | Eli Collins | Akira Ajisaka | | [HDFS-4903](https://issues.apache.org/jira/browse/HDFS-4903) | Print trash configuration and trash emptier state in namenode log | Minor | namenode | Suresh Srinivas | Arpit Agarwal | -| [HDFS-4521](https://issues.apache.org/jira/browse/HDFS-4521) | invalid network topologies should not be cached | Minor | . | Colin Patrick McCabe | Colin Patrick McCabe | -| [MAPREDUCE-6088](https://issues.apache.org/jira/browse/MAPREDUCE-6088) | TestTokenCache tests should use their own JobConf instances | Major | mrv1, test | zhihai xu | zhihai xu | -| [MAPREDUCE-5712](https://issues.apache.org/jira/browse/MAPREDUCE-5712) | Backport Fair Scheduler pool placement by secondary group | Major | scheduler | Ted Malaska | Ted Malaska | -| [MAPREDUCE-5651](https://issues.apache.org/jira/browse/MAPREDUCE-5651) | Backport Fair Scheduler queue placement policies to branch-1 | Major | scheduler | Sandy Ryza | Ted Malaska | -| [MAPREDUCE-5609](https://issues.apache.org/jira/browse/MAPREDUCE-5609) | Add debug log message when sending job end notification | Major | . | Robert Kanter | Robert Kanter | -| [MAPREDUCE-5457](https://issues.apache.org/jira/browse/MAPREDUCE-5457) | Add a KeyOnlyTextOutputReader to enable streaming to write out text files without separators | Major | . | Sandy Ryza | Sandy Ryza | -| [MAPREDUCE-5408](https://issues.apache.org/jira/browse/MAPREDUCE-5408) | CLONE - The logging level of the tasks should be configurable by the job | Major | . | Owen O'Malley | Arun C Murthy | | [MAPREDUCE-5406](https://issues.apache.org/jira/browse/MAPREDUCE-5406) | Improve logging around Task Tracker exiting with JVM manager inconsistent state | Major | tasktracker | Chelsey Chang | Chelsey Chang | +| [MAPREDUCE-5408](https://issues.apache.org/jira/browse/MAPREDUCE-5408) | CLONE - The logging level of the tasks should be configurable by the job | Major | . | Owen O'Malley | Arun C Murthy | | [MAPREDUCE-5367](https://issues.apache.org/jira/browse/MAPREDUCE-5367) | Local jobs all use same local working directory | Major | . | Sandy Ryza | Sandy Ryza | +| [HDFS-4963](https://issues.apache.org/jira/browse/HDFS-4963) | Improve multihoming support in namenode | Major | namenode | Arpit Agarwal | Arpit Agarwal | +| [HADOOP-9855](https://issues.apache.org/jira/browse/HADOOP-9855) | Backport HADOOP-6578 to branch-1 | Major | . | James Kinley | James Kinley | +| [HDFS-5038](https://issues.apache.org/jira/browse/HDFS-5038) | Backport several branch-2 APIs to branch-1 | Minor | . | Jing Zhao | Jing Zhao | +| [HDFS-5367](https://issues.apache.org/jira/browse/HDFS-5367) | Restoring namenode storage locks namenode due to unnecessary fsimage write | Major | . | yunjiong zhao | yunjiong zhao | +| [MAPREDUCE-5457](https://issues.apache.org/jira/browse/MAPREDUCE-5457) | Add a KeyOnlyTextOutputReader to enable streaming to write out text files without separators | Major | . | Sandy Ryza | Sandy Ryza | +| [MAPREDUCE-5609](https://issues.apache.org/jira/browse/MAPREDUCE-5609) | Add debug log message when sending job end notification | Major | . | Robert Kanter | Robert Kanter | | [MAPREDUCE-3310](https://issues.apache.org/jira/browse/MAPREDUCE-3310) | Custom grouping comparator cannot be set for Combiners | Major | client | Mathias Herberts | Alejandro Abdelnur | -| [MAPREDUCE-2351](https://issues.apache.org/jira/browse/MAPREDUCE-2351) | mapred.job.tracker.history.completed.location should support an arbitrary filesystem URI | Major | . | Tom White | Tom White | +| [MAPREDUCE-5651](https://issues.apache.org/jira/browse/MAPREDUCE-5651) | Backport Fair Scheduler queue placement policies to branch-1 | Major | scheduler | Sandy Ryza | Ted Malaska | +| [MAPREDUCE-5712](https://issues.apache.org/jira/browse/MAPREDUCE-5712) | Backport Fair Scheduler pool placement by secondary group | Major | scheduler | Theodore michael Malaska | Theodore michael Malaska | +| [HADOOP-10614](https://issues.apache.org/jira/browse/HADOOP-10614) | CBZip2InputStream is not threadsafe | Major | . | Xiangrui Meng | Xiangrui Meng | +| [MAPREDUCE-6088](https://issues.apache.org/jira/browse/MAPREDUCE-6088) | TestTokenCache tests should use their own JobConf instances | Major | mrv1, test | zhihai xu | zhihai xu | +| [HDFS-7312](https://issues.apache.org/jira/browse/HDFS-7312) | Update DistCp v1 to optionally not use tmp location (branch-1 only) | Minor | tools | Joseph Prosser | Joseph Prosser | ### BUG FIXES: | JIRA | Summary | Priority | Component | Reporter | Contributor | |:---- |:---- | :--- |:---- |:---- |:---- | -| [HADOOP-11035](https://issues.apache.org/jira/browse/HADOOP-11035) | distcp on mr1(branch-1) fails with NPE using a short relative source path. | Major | tools | zhihai xu | zhihai xu | -| [HADOOP-10562](https://issues.apache.org/jira/browse/HADOOP-10562) | Namenode exits on exception without printing stack trace in AbstractDelegationTokenSecretManager | Critical | . | Suresh Srinivas | Suresh Srinivas | -| [HADOOP-10142](https://issues.apache.org/jira/browse/HADOOP-10142) | Avoid groups lookup for unprivileged users such as "dr.who" | Major | . | Vinayakumar B | Vinayakumar B | -| [HADOOP-10090](https://issues.apache.org/jira/browse/HADOOP-10090) | Jobtracker metrics not updated properly after execution of a mapreduce job | Major | metrics | Ivan Mitic | Ivan Mitic | -| [HADOOP-10009](https://issues.apache.org/jira/browse/HADOOP-10009) | Backport HADOOP-7808 to branch-1 | Major | . | Haohui Mai | Haohui Mai | -| [HADOOP-9801](https://issues.apache.org/jira/browse/HADOOP-9801) | Configuration#writeXml uses platform defaulting encoding, which may mishandle multi-byte characters. | Major | conf | Chris Nauroth | Chris Nauroth | -| [HADOOP-9768](https://issues.apache.org/jira/browse/HADOOP-9768) | chown and chgrp reject users and groups with spaces on platforms where spaces are otherwise acceptable | Major | fs | Chris Nauroth | Chris Nauroth | +| [HADOOP-7140](https://issues.apache.org/jira/browse/HADOOP-7140) | IPC Reader threads do not stop when server stops | Critical | . | Todd Lipcon | Todd Lipcon | +| [HDFS-2264](https://issues.apache.org/jira/browse/HDFS-2264) | NamenodeProtocol has the wrong value for clientPrincipal in KerberosInfo annotation | Major | namenode | Aaron T. Myers | Aaron T. Myers | +| [MAPREDUCE-5046](https://issues.apache.org/jira/browse/MAPREDUCE-5046) | backport MAPREDUCE-1423 to mapred.lib.CombineFileInputFormat | Major | client | Sangjin Lee | | +| [MAPREDUCE-5047](https://issues.apache.org/jira/browse/MAPREDUCE-5047) | keep.failed.task.files=true causes job failure on secure clusters | Major | task, tasktracker | Sandy Ryza | Sandy Ryza | +| [MAPREDUCE-5070](https://issues.apache.org/jira/browse/MAPREDUCE-5070) | TestClusterStatus.testClusterMetrics fails on JDK7 | Major | test | Sandy Ryza | Sandy Ryza | +| [MAPREDUCE-5072](https://issues.apache.org/jira/browse/MAPREDUCE-5072) | TestDelegationTokenRenewal.testDTRenewal fails in MR1 on jdk7 | Major | test | Sandy Ryza | Sandy Ryza | +| [MAPREDUCE-5073](https://issues.apache.org/jira/browse/MAPREDUCE-5073) | TestJobStatusPersistency.testPersistency fails on JDK7 | Major | test | Sandy Ryza | Sandy Ryza | +| [HDFS-4622](https://issues.apache.org/jira/browse/HDFS-4622) | Remove redundant synchronized from FSNamesystem#rollEditLog in branch-1 | Trivial | . | Jing Zhao | Jing Zhao | +| [MAPREDUCE-2817](https://issues.apache.org/jira/browse/MAPREDUCE-2817) | MiniRMCluster hardcodes 'mapred.local.dir' configuration to 'build/test/mapred/local' | Minor | test | Alejandro Abdelnur | Robert Kanter | +| [MAPREDUCE-5133](https://issues.apache.org/jira/browse/MAPREDUCE-5133) | TestSubmitJob.testSecureJobExecution is flaky due to job dir deletion race | Major | test | Sandy Ryza | Sandy Ryza | +| [MAPREDUCE-5218](https://issues.apache.org/jira/browse/MAPREDUCE-5218) | Annotate (comment) internal classes as Private | Minor | . | Karthik Kambatla | Karthik Kambatla | +| [MAPREDUCE-5217](https://issues.apache.org/jira/browse/MAPREDUCE-5217) | DistCp fails when launched by Oozie in a secure cluster | Major | distcp, security | Venkat Ranganathan | Venkat Ranganathan | +| [MAPREDUCE-5095](https://issues.apache.org/jira/browse/MAPREDUCE-5095) | TestShuffleExceptionCount#testCheckException fails occasionally with JDK7 | Major | . | Arpit Agarwal | Arpit Agarwal | +| [MAPREDUCE-5250](https://issues.apache.org/jira/browse/MAPREDUCE-5250) | Searching for ';' in JobTracker History throws ArrayOutOfBoundException | Minor | . | Karthik Kambatla | Karthik Kambatla | +| [HDFS-4944](https://issues.apache.org/jira/browse/HDFS-4944) | WebHDFS cannot create a file path containing characters that must be URI-encoded, such as space. | Major | webhdfs | Chris Nauroth | Chris Nauroth | | [HADOOP-9678](https://issues.apache.org/jira/browse/HADOOP-9678) | TestRPC#testStopsAllThreads intermittently fails on Windows | Major | . | Ivan Mitic | Ivan Mitic | -| [HADOOP-9507](https://issues.apache.org/jira/browse/HADOOP-9507) | LocalFileSystem rename() is broken in some cases when destination exists | Minor | fs | Mostafa Elhemali | Chris Nauroth | | [HADOOP-9307](https://issues.apache.org/jira/browse/HADOOP-9307) | BufferedFSInputStream.read returns wrong results after certain seeks | Major | fs | Todd Lipcon | Todd Lipcon | -| [HADOOP-7140](https://issues.apache.org/jira/browse/HADOOP-7140) | IPC Reader threads do not stop when server stops | Critical | . | Todd Lipcon | Todd Lipcon | -| [HDFS-7503](https://issues.apache.org/jira/browse/HDFS-7503) | Namenode restart after large deletions can cause slow processReport (due to logging) | Major | namenode | Arpit Agarwal | Arpit Agarwal | -| [HDFS-6822](https://issues.apache.org/jira/browse/HDFS-6822) | In branch-1, Namenode and datanode fails to replace "\_HOST" to hostname for hadoop.http.authentication.kerberos.principal | Major | security | Jing Zhao | Jing Zhao | -| [HDFS-6649](https://issues.apache.org/jira/browse/HDFS-6649) | Documentation for setrep is wrong | Trivial | documentation | Alexander Fahlke | Akira AJISAKA | -| [HDFS-6141](https://issues.apache.org/jira/browse/HDFS-6141) | WebHdfsFileSystem#toUrl does not perform character escaping. | Major | webhdfs | Chris Nauroth | Chris Nauroth | -| [HDFS-5944](https://issues.apache.org/jira/browse/HDFS-5944) | LeaseManager:findLeaseWithPrefixPath can't handle path like /a/b/ right and cause SecondaryNameNode failed do checkpoint | Major | namenode | zhaoyunjiong | zhaoyunjiong | -| [HDFS-5685](https://issues.apache.org/jira/browse/HDFS-5685) | DistCp will fail to copy with -delete switch | Major | hdfs-client | Yongjun Zhang | Yongjun Zhang | -| [HDFS-5516](https://issues.apache.org/jira/browse/HDFS-5516) | WebHDFS does not require user name when anonymous http requests are disallowed. | Major | webhdfs | Chris Nauroth | Miodrag Radulovic | -| [HDFS-5245](https://issues.apache.org/jira/browse/HDFS-5245) | shouldRetry() in WebHDFSFileSystem generates excessive warnings | Minor | webhdfs | Haohui Mai | Haohui Mai | -| [HDFS-5211](https://issues.apache.org/jira/browse/HDFS-5211) | Race condition between DistributedFileSystem#close and FileSystem#close can cause return of a closed DistributedFileSystem instance from the FileSystem cache. | Major | hdfs-client | Chris Nauroth | Chris Nauroth | | [HDFS-5003](https://issues.apache.org/jira/browse/HDFS-5003) | TestNNThroughputBenchmark failed caused by existing directories | Minor | test | Xi Fang | Xi Fang | -| [HDFS-4944](https://issues.apache.org/jira/browse/HDFS-4944) | WebHDFS cannot create a file path containing characters that must be URI-encoded, such as space. | Major | webhdfs | Chris Nauroth | Chris Nauroth | +| [MAPREDUCE-5405](https://issues.apache.org/jira/browse/MAPREDUCE-5405) | Job recovery can fail if task log directory symlink from prior run still exists | Major | mrv1 | Chris Nauroth | Chris Nauroth | +| [MAPREDUCE-5288](https://issues.apache.org/jira/browse/MAPREDUCE-5288) | ResourceEstimator#getEstimatedTotalMapOutputSize suffers from divide by zero issues | Major | mrv1 | Harsh J | Karthik Kambatla | +| [HADOOP-9507](https://issues.apache.org/jira/browse/HADOOP-9507) | LocalFileSystem rename() is broken in some cases when destination exists | Minor | fs | Mostafa Elhemali | Chris Nauroth | +| [MAPREDUCE-4366](https://issues.apache.org/jira/browse/MAPREDUCE-4366) | mapred metrics shows negative count of waiting maps and reduces | Major | jobtracker | Thomas Graves | Sandy Ryza | +| [HADOOP-9768](https://issues.apache.org/jira/browse/HADOOP-9768) | chown and chgrp reject users and groups with spaces on platforms where spaces are otherwise acceptable | Major | fs | Chris Nauroth | Chris Nauroth | +| [HADOOP-9801](https://issues.apache.org/jira/browse/HADOOP-9801) | Configuration#writeXml uses platform defaulting encoding, which may mishandle multi-byte characters. | Major | conf | Chris Nauroth | Chris Nauroth | | [HDFS-4898](https://issues.apache.org/jira/browse/HDFS-4898) | BlockPlacementPolicyWithNodeGroup.chooseRemoteRack() fails to properly fallback to local rack | Minor | namenode | Eric Sirianni | Tsz Wo Nicholas Sze | +| [HDFS-5211](https://issues.apache.org/jira/browse/HDFS-5211) | Race condition between DistributedFileSystem#close and FileSystem#close can cause return of a closed DistributedFileSystem instance from the FileSystem cache. | Major | hdfs-client | Chris Nauroth | Chris Nauroth | +| [MAPREDUCE-5508](https://issues.apache.org/jira/browse/MAPREDUCE-5508) | JobTracker memory leak caused by unreleased FileSystem objects in JobInProgress#cleanupJob | Critical | jobtracker | Xi Fang | Xi Fang | +| [HDFS-5245](https://issues.apache.org/jira/browse/HDFS-5245) | shouldRetry() in WebHDFSFileSystem generates excessive warnings | Minor | webhdfs | Haohui Mai | Haohui Mai | +| [HADOOP-10009](https://issues.apache.org/jira/browse/HADOOP-10009) | Backport HADOOP-7808 to branch-1 | Major | . | Haohui Mai | Haohui Mai | +| [MAPREDUCE-5569](https://issues.apache.org/jira/browse/MAPREDUCE-5569) | FloatSplitter is not generating correct splits | Major | . | Nathan Roberts | Nathan Roberts | +| [MAPREDUCE-5512](https://issues.apache.org/jira/browse/MAPREDUCE-5512) | TaskTracker hung after failed reconnect to the JobTracker | Major | tasktracker | Ivan Mitic | Ivan Mitic | | [HDFS-4794](https://issues.apache.org/jira/browse/HDFS-4794) | Browsing filesystem via webui throws kerberos exception when NN service RPC is enabled in a secure cluster | Major | security | Benoy Antony | Benoy Antony | -| [HDFS-4622](https://issues.apache.org/jira/browse/HDFS-4622) | Remove redundant synchronized from FSNamesystem#rollEditLog in branch-1 | Trivial | . | Jing Zhao | Jing Zhao | -| [HDFS-2264](https://issues.apache.org/jira/browse/HDFS-2264) | NamenodeProtocol has the wrong value for clientPrincipal in KerberosInfo annotation | Major | namenode | Aaron T. Myers | Aaron T. Myers | -| [MAPREDUCE-6198](https://issues.apache.org/jira/browse/MAPREDUCE-6198) | NPE from JobTracker#resolveAndAddToTopology in MR1 cause initJob and heartbeat failure. | Major | mrv1 | zhihai xu | zhihai xu | -| [MAPREDUCE-6196](https://issues.apache.org/jira/browse/MAPREDUCE-6196) | Fix BigDecimal ArithmeticException in PiEstimator | Minor | . | Ray Chiang | Ray Chiang | -| [MAPREDUCE-6170](https://issues.apache.org/jira/browse/MAPREDUCE-6170) | TestUlimit failure on JDK8 | Major | contrib/streaming | bc Wong | bc Wong | -| [MAPREDUCE-6147](https://issues.apache.org/jira/browse/MAPREDUCE-6147) | Support mapreduce.input.fileinputformat.split.maxsize | Minor | mrv1 | zhihai xu | zhihai xu | -| [MAPREDUCE-6076](https://issues.apache.org/jira/browse/MAPREDUCE-6076) | Zero map split input length combine with none zero map split input length may cause MR1 job hung sometimes. | Major | mrv1 | zhihai xu | zhihai xu | -| [MAPREDUCE-6012](https://issues.apache.org/jira/browse/MAPREDUCE-6012) | DBInputSplit creates invalid ranges on Oracle | Major | . | Julien Serdaru | Wei Yan | -| [MAPREDUCE-6009](https://issues.apache.org/jira/browse/MAPREDUCE-6009) | Map-only job with new-api runs wrong OutputCommitter when cleanup scheduled in a reduce slot | Blocker | client, job submission | Gera Shegalov | Gera Shegalov | -| [MAPREDUCE-5979](https://issues.apache.org/jira/browse/MAPREDUCE-5979) | FairScheduler: zero weight can cause sort failures | Major | scheduler | Anubhav Dhoot | Anubhav Dhoot | -| [MAPREDUCE-5968](https://issues.apache.org/jira/browse/MAPREDUCE-5968) | Work directory is not deleted when downloadCacheObject throws IOException | Major | mrv1 | zhihai xu | zhihai xu | -| [MAPREDUCE-5966](https://issues.apache.org/jira/browse/MAPREDUCE-5966) | MR1 FairScheduler use of custom weight adjuster is not thread safe for comparisons | Major | scheduler | Anubhav Dhoot | Anubhav Dhoot | -| [MAPREDUCE-5877](https://issues.apache.org/jira/browse/MAPREDUCE-5877) | Inconsistency between JT/TT for tasks taking a long time to launch | Critical | jobtracker, tasktracker | Karthik Kambatla | Karthik Kambatla | -| [MAPREDUCE-5822](https://issues.apache.org/jira/browse/MAPREDUCE-5822) | FairScheduler does not preempt due to fairshare-starvation when fairshare is 1 | Major | scheduler | Anubhav Dhoot | Anubhav Dhoot | -| [MAPREDUCE-5808](https://issues.apache.org/jira/browse/MAPREDUCE-5808) | Port output replication factor configurable for terasort to Hadoop 1.x | Minor | examples | Chuan Liu | Chuan Liu | -| [MAPREDUCE-5710](https://issues.apache.org/jira/browse/MAPREDUCE-5710) | Backport MAPREDUCE-1305 to branch-1 | Major | . | Yongjun Zhang | Yongjun Zhang | +| [HADOOP-10142](https://issues.apache.org/jira/browse/HADOOP-10142) | Avoid groups lookup for unprivileged users such as "dr.who" | Major | . | Vinayakumar B | Vinayakumar B | +| [HADOOP-10090](https://issues.apache.org/jira/browse/HADOOP-10090) | Jobtracker metrics not updated properly after execution of a mapreduce job | Major | metrics | Ivan Mitic | Ivan Mitic | | [MAPREDUCE-5702](https://issues.apache.org/jira/browse/MAPREDUCE-5702) | TaskLogServlet#printTaskLog has spurious HTML closing tags | Trivial | task | Karthik Kambatla | Robert Kanter | | [MAPREDUCE-5698](https://issues.apache.org/jira/browse/MAPREDUCE-5698) | Backport MAPREDUCE-1285 to branch-1 | Major | . | Yongjun Zhang | Yongjun Zhang | -| [MAPREDUCE-5660](https://issues.apache.org/jira/browse/MAPREDUCE-5660) | Log info about possible thrashing (when using memory-based scheduling in Capacity Scheduler) is not printed | Trivial | capacity-sched, mrv1, tasktracker | Adam Kawa | Adam Kawa | -| [MAPREDUCE-5569](https://issues.apache.org/jira/browse/MAPREDUCE-5569) | FloatSplitter is not generating correct splits | Major | . | Nathan Roberts | Nathan Roberts | -| [MAPREDUCE-5556](https://issues.apache.org/jira/browse/MAPREDUCE-5556) | mapred docs have incorrect classpath | Trivial | . | Allen Wittenauer | Harsh J | -| [MAPREDUCE-5512](https://issues.apache.org/jira/browse/MAPREDUCE-5512) | TaskTracker hung after failed reconnect to the JobTracker | Major | tasktracker | Ivan Mitic | Ivan Mitic | -| [MAPREDUCE-5508](https://issues.apache.org/jira/browse/MAPREDUCE-5508) | JobTracker memory leak caused by unreleased FileSystem objects in JobInProgress#cleanupJob | Critical | jobtracker | Xi Fang | Xi Fang | -| [MAPREDUCE-5405](https://issues.apache.org/jira/browse/MAPREDUCE-5405) | Job recovery can fail if task log directory symlink from prior run still exists | Major | mrv1 | Chris Nauroth | Chris Nauroth | -| [MAPREDUCE-5288](https://issues.apache.org/jira/browse/MAPREDUCE-5288) | ResourceEstimator#getEstimatedTotalMapOutputSize suffers from divide by zero issues | Major | mrv1 | Harsh J | Karthik Kambatla | +| [HDFS-5685](https://issues.apache.org/jira/browse/HDFS-5685) | DistCp will fail to copy with -delete switch | Major | hdfs-client | Yongjun Zhang | Yongjun Zhang | +| [MAPREDUCE-5710](https://issues.apache.org/jira/browse/MAPREDUCE-5710) | Backport MAPREDUCE-1305 to branch-1 | Major | . | Yongjun Zhang | Yongjun Zhang | +| [HDFS-5944](https://issues.apache.org/jira/browse/HDFS-5944) | LeaseManager:findLeaseWithPrefixPath can't handle path like /a/b/ right and cause SecondaryNameNode failed do checkpoint | Major | namenode | yunjiong zhao | yunjiong zhao | +| [HDFS-5516](https://issues.apache.org/jira/browse/HDFS-5516) | WebHDFS does not require user name when anonymous http requests are disallowed. | Major | webhdfs | Chris Nauroth | Miodrag Radulovic | +| [MAPREDUCE-5808](https://issues.apache.org/jira/browse/MAPREDUCE-5808) | Port output replication factor configurable for terasort to Hadoop 1.x | Minor | examples | Chuan Liu | Chuan Liu | +| [MAPREDUCE-5822](https://issues.apache.org/jira/browse/MAPREDUCE-5822) | FairScheduler does not preempt due to fairshare-starvation when fairshare is 1 | Major | scheduler | Anubhav Dhoot | Anubhav Dhoot | +| [HDFS-6141](https://issues.apache.org/jira/browse/HDFS-6141) | WebHdfsFileSystem#toUrl does not perform character escaping. | Major | webhdfs | Chris Nauroth | Chris Nauroth | +| [HADOOP-10562](https://issues.apache.org/jira/browse/HADOOP-10562) | Namenode exits on exception without printing stack trace in AbstractDelegationTokenSecretManager | Critical | . | Suresh Srinivas | Suresh Srinivas | +| [MAPREDUCE-5877](https://issues.apache.org/jira/browse/MAPREDUCE-5877) | Inconsistency between JT/TT for tasks taking a long time to launch | Critical | jobtracker, tasktracker | Karthik Kambatla | Karthik Kambatla | +| [MAPREDUCE-5979](https://issues.apache.org/jira/browse/MAPREDUCE-5979) | FairScheduler: zero weight can cause sort failures | Major | scheduler | Anubhav Dhoot | Anubhav Dhoot | +| [MAPREDUCE-5966](https://issues.apache.org/jira/browse/MAPREDUCE-5966) | MR1 FairScheduler use of custom weight adjuster is not thread safe for comparisons | Major | scheduler | Anubhav Dhoot | Anubhav Dhoot | +| [MAPREDUCE-5968](https://issues.apache.org/jira/browse/MAPREDUCE-5968) | Work directory is not deleted when downloadCacheObject throws IOException | Major | mrv1 | zhihai xu | zhihai xu | +| [HDFS-6822](https://issues.apache.org/jira/browse/HDFS-6822) | In branch-1, Namenode and datanode fails to replace "\_HOST" to hostname for hadoop.http.authentication.kerberos.principal | Major | security | Jing Zhao | Jing Zhao | +| [MAPREDUCE-6012](https://issues.apache.org/jira/browse/MAPREDUCE-6012) | DBInputSplit creates invalid ranges on Oracle | Major | . | Julien Serdaru | Wei Yan | +| [HADOOP-11035](https://issues.apache.org/jira/browse/HADOOP-11035) | distcp on mr1(branch-1) fails with NPE using a short relative source path. | Major | tools | zhihai xu | zhihai xu | +| [MAPREDUCE-6009](https://issues.apache.org/jira/browse/MAPREDUCE-6009) | Map-only job with new-api runs wrong OutputCommitter when cleanup scheduled in a reduce slot | Blocker | client, job submission | Gera Shegalov | Gera Shegalov | +| [MAPREDUCE-6147](https://issues.apache.org/jira/browse/MAPREDUCE-6147) | Support mapreduce.input.fileinputformat.split.maxsize | Minor | mrv1 | zhihai xu | zhihai xu | +| [MAPREDUCE-6170](https://issues.apache.org/jira/browse/MAPREDUCE-6170) | TestUlimit failure on JDK8 | Major | contrib/streaming | bc Wong | bc Wong | +| [HDFS-7503](https://issues.apache.org/jira/browse/HDFS-7503) | Namenode restart after large deletions can cause slow processReport (due to logging) | Major | namenode | Arpit Agarwal | Arpit Agarwal | +| [MAPREDUCE-6196](https://issues.apache.org/jira/browse/MAPREDUCE-6196) | Fix BigDecimal ArithmeticException in PiEstimator | Minor | . | Ray Chiang | Ray Chiang | +| [MAPREDUCE-6198](https://issues.apache.org/jira/browse/MAPREDUCE-6198) | NPE from JobTracker#resolveAndAddToTopology in MR1 cause initJob and heartbeat failure. | Major | mrv1 | zhihai xu | zhihai xu | | [MAPREDUCE-5272](https://issues.apache.org/jira/browse/MAPREDUCE-5272) | A Minor Error in Javadoc of TestMRWithDistributedCache in Branch-1 | Trivial | test | Zhijie Shen | Zhijie Shen | -| [MAPREDUCE-5250](https://issues.apache.org/jira/browse/MAPREDUCE-5250) | Searching for ';' in JobTracker History throws ArrayOutOfBoundException | Minor | . | Karthik Kambatla | Karthik Kambatla | -| [MAPREDUCE-5218](https://issues.apache.org/jira/browse/MAPREDUCE-5218) | Annotate (comment) internal classes as Private | Minor | . | Karthik Kambatla | Karthik Kambatla | -| [MAPREDUCE-5217](https://issues.apache.org/jira/browse/MAPREDUCE-5217) | DistCp fails when launched by Oozie in a secure cluster | Major | distcp, security | Venkat Ranganathan | Venkat Ranganathan | +| [MAPREDUCE-5556](https://issues.apache.org/jira/browse/MAPREDUCE-5556) | mapred docs have incorrect classpath | Trivial | . | Allen Wittenauer | Harsh J | +| [MAPREDUCE-5660](https://issues.apache.org/jira/browse/MAPREDUCE-5660) | Log info about possible thrashing (when using memory-based scheduling in Capacity Scheduler) is not printed | Trivial | capacity-sched, mrv1, tasktracker | Adam Kawa | Adam Kawa | | [MAPREDUCE-5183](https://issues.apache.org/jira/browse/MAPREDUCE-5183) | In, TaskTracker#reportProgress logging of 0.0-1.0 progress is followed by percent sign | Minor | mrv1, tasktracker | Sandy Ryza | Niranjan Singh | -| [MAPREDUCE-5133](https://issues.apache.org/jira/browse/MAPREDUCE-5133) | TestSubmitJob.testSecureJobExecution is flaky due to job dir deletion race | Major | test | Sandy Ryza | Sandy Ryza | -| [MAPREDUCE-5095](https://issues.apache.org/jira/browse/MAPREDUCE-5095) | TestShuffleExceptionCount#testCheckException fails occasionally with JDK7 | Major | . | Arpit Agarwal | Arpit Agarwal | -| [MAPREDUCE-5073](https://issues.apache.org/jira/browse/MAPREDUCE-5073) | TestJobStatusPersistency.testPersistency fails on JDK7 | Major | test | Sandy Ryza | Sandy Ryza | -| [MAPREDUCE-5072](https://issues.apache.org/jira/browse/MAPREDUCE-5072) | TestDelegationTokenRenewal.testDTRenewal fails in MR1 on jdk7 | Major | test | Sandy Ryza | Sandy Ryza | -| [MAPREDUCE-5070](https://issues.apache.org/jira/browse/MAPREDUCE-5070) | TestClusterStatus.testClusterMetrics fails on JDK7 | Major | test | Sandy Ryza | Sandy Ryza | -| [MAPREDUCE-5047](https://issues.apache.org/jira/browse/MAPREDUCE-5047) | keep.failed.task.files=true causes job failure on secure clusters | Major | task, tasktracker | Sandy Ryza | Sandy Ryza | -| [MAPREDUCE-5046](https://issues.apache.org/jira/browse/MAPREDUCE-5046) | backport MAPREDUCE-1423 to mapred.lib.CombineFileInputFormat | Major | client | Sangjin Lee | | -| [MAPREDUCE-4366](https://issues.apache.org/jira/browse/MAPREDUCE-4366) | mapred metrics shows negative count of waiting maps and reduces | Major | jobtracker | Thomas Graves | Sandy Ryza | -| [MAPREDUCE-2817](https://issues.apache.org/jira/browse/MAPREDUCE-2817) | MiniRMCluster hardcodes 'mapred.local.dir' configuration to 'build/test/mapred/local' | Minor | test | Alejandro Abdelnur | Robert Kanter | +| [HDFS-6649](https://issues.apache.org/jira/browse/HDFS-6649) | Documentation for setrep is wrong | Trivial | documentation | Alexander Fahlke | Akira Ajisaka | +| [MAPREDUCE-6076](https://issues.apache.org/jira/browse/MAPREDUCE-6076) | Zero map split input length combine with none zero map split input length may cause MR1 job hung sometimes. | Major | mrv1 | zhihai xu | zhihai xu | ### TESTS: @@ -145,13 +133,7 @@ | JIRA | Summary | Priority | Component | Reporter | Contributor | |:---- |:---- | :--- |:---- |:---- |:---- | -| [HADOOP-9573](https://issues.apache.org/jira/browse/HADOOP-9573) | Fix test-patch script to work with the enhanced PreCommit-Admin script. | Major | build | Giridharan Kesavan | Giridharan Kesavan | | [MAPREDUCE-4049](https://issues.apache.org/jira/browse/MAPREDUCE-4049) | plugin for generic shuffle service | Major | performance, task, tasktracker | Avner BenHanoch | Avner BenHanoch | - - -### OTHER: - -| JIRA | Summary | Priority | Component | Reporter | Contributor | -|:---- |:---- | :--- |:---- |:---- |:---- | +| [HADOOP-9573](https://issues.apache.org/jira/browse/HADOOP-9573) | Fix test-patch script to work with the enhanced PreCommit-Admin script. | Major | build | Giridharan Kesavan | Giridharan Kesavan | http://git-wip-us.apache.org/repos/asf/hadoop/blob/19041008/hadoop-common-project/hadoop-common/src/site/markdown/release/1.3.0/RELEASENOTES.1.3.0.md ---------------------------------------------------------------------- diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/release/1.3.0/RELEASENOTES.1.3.0.md b/hadoop-common-project/hadoop-common/src/site/markdown/release/1.3.0/RELEASENOTES.1.3.0.md index a6cf8f8..f9d52c7 100644 --- a/hadoop-common-project/hadoop-common/src/site/markdown/release/1.3.0/RELEASENOTES.1.3.0.md +++ b/hadoop-common-project/hadoop-common/src/site/markdown/release/1.3.0/RELEASENOTES.1.3.0.md @@ -23,47 +23,47 @@ These release notes cover new developer and user-facing incompatibilities, impor --- -* [HADOOP-8873](https://issues.apache.org/jira/browse/HADOOP-8873) | *Major* | **Port HADOOP-8175 (Add mkdir -p flag) to branch-1** +* [MAPREDUCE-4049](https://issues.apache.org/jira/browse/MAPREDUCE-4049) | *Major* | **plugin for generic shuffle service** -FsShell mkdir now accepts a -p flag. Like unix, mkdir -p will not fail if the directory already exists. Unlike unix, intermediate directories are always created, regardless of the flag, to avoid incompatibilities. +Allow ReduceTask loading a third party plugin for shuffle (and merge) instead of the default shuffle. --- -* [HDFS-7312](https://issues.apache.org/jira/browse/HDFS-7312) | *Minor* | **Update DistCp v1 to optionally not use tmp location (branch-1 only)** +* [HADOOP-8873](https://issues.apache.org/jira/browse/HADOOP-8873) | *Major* | **Port HADOOP-8175 (Add mkdir -p flag) to branch-1** -DistCp v1 currently copies files to a temporary location and then renames that to the specified destination. This can cause performance issues on file systems such as S3. A -skiptmp flag is added to bypass this step and copy directly to the destination. +FsShell mkdir now accepts a -p flag. Like unix, mkdir -p will not fail if the directory already exists. Unlike unix, intermediate directories are always created, regardless of the flag, to avoid incompatibilities. --- -* [HDFS-5685](https://issues.apache.org/jira/browse/HDFS-5685) | *Major* | **DistCp will fail to copy with -delete switch** +* [MAPREDUCE-5408](https://issues.apache.org/jira/browse/MAPREDUCE-5408) | *Major* | **CLONE - The logging level of the tasks should be configurable by the job** -Has dependency on MAPREDUCE-1285/MAPREDUCE-5698 +Allow logging level of map/reduce tasks to be configurable. +Configuration changes: + add mapred.map.child.log.level + add mapred.reduce.child.log.level --- -* [MAPREDUCE-5777](https://issues.apache.org/jira/browse/MAPREDUCE-5777) | *Major* | **Support utf-8 text with BOM (byte order marker)** +* [HDFS-5685](https://issues.apache.org/jira/browse/HDFS-5685) | *Major* | **DistCp will fail to copy with -delete switch** -**WARNING: No release note provided for this incompatible change.** +Has dependency on MAPREDUCE-1285/MAPREDUCE-5698 --- -* [MAPREDUCE-5408](https://issues.apache.org/jira/browse/MAPREDUCE-5408) | *Major* | **CLONE - The logging level of the tasks should be configurable by the job** +* [MAPREDUCE-5777](https://issues.apache.org/jira/browse/MAPREDUCE-5777) | *Major* | **Support utf-8 text with BOM (byte order marker)** -Allow logging level of map/reduce tasks to be configurable. -Configuration changes: - add mapred.map.child.log.level - add mapred.reduce.child.log.level +**WARNING: No release note provided for this change.** --- -* [MAPREDUCE-4049](https://issues.apache.org/jira/browse/MAPREDUCE-4049) | *Major* | **plugin for generic shuffle service** +* [HDFS-7312](https://issues.apache.org/jira/browse/HDFS-7312) | *Minor* | **Update DistCp v1 to optionally not use tmp location (branch-1 only)** -Allow ReduceTask loading a third party plugin for shuffle (and merge) instead of the default shuffle. +DistCp v1 currently copies files to a temporary location and then renames that to the specified destination. This can cause performance issues on file systems such as S3. A -skiptmp flag is added to bypass this step and copy directly to the destination. --------------------------------------------------------------------- To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
