http://git-wip-us.apache.org/repos/asf/hadoop/blob/19041008/hadoop-common-project/hadoop-common/src/site/markdown/release/0.18.0/RELEASENOTES.0.18.0.md ---------------------------------------------------------------------- diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/release/0.18.0/RELEASENOTES.0.18.0.md b/hadoop-common-project/hadoop-common/src/site/markdown/release/0.18.0/RELEASENOTES.0.18.0.md index f57c602..32e3c01 100644 --- a/hadoop-common-project/hadoop-common/src/site/markdown/release/0.18.0/RELEASENOTES.0.18.0.md +++ b/hadoop-common-project/hadoop-common/src/site/markdown/release/0.18.0/RELEASENOTES.0.18.0.md @@ -23,523 +23,523 @@ These release notes cover new developer and user-facing incompatibilities, impor --- -* [HADOOP-3837](https://issues.apache.org/jira/browse/HADOOP-3837) | *Major* | **hadop streaming does not use progress reporting to detect hung tasks** +* [HADOOP-2585](https://issues.apache.org/jira/browse/HADOOP-2585) | *Major* | **Automatic namespace recovery from the secondary image.** -Changed streaming tasks to adhere to task timeout value specified in the job configuration. +Improved management of replicas of the name space image. If all replicas on the Name Node are lost, the latest check point can be loaded from the secondary Name Node. Use parameter "-importCheckpoint" and specify the location with "fs.checkpoint.dir." The directory structure on the secondary Name Node has changed to match the primary Name Node. --- -* [HADOOP-3808](https://issues.apache.org/jira/browse/HADOOP-3808) | *Blocker* | **[HOD] Include job tracker RPC in notes attribute after job submission** +* [HADOOP-2703](https://issues.apache.org/jira/browse/HADOOP-2703) | *Minor* | **New files under lease (before close) still shows up as MISSING files/blocks in fsck** -Modified HOD to include the RPC port of the JobTracker in the 'notes' attribute of the resource manager. The RPC port is included as the string 'Mapred RPC Port:\<port number\>'. Tools that depend on the value of the notes attribute must change to parse this new value. +Changed fsck to ignore files opened for writing. Introduced new option "-openforwrite" to explicitly show open files. --- -* [HADOOP-3703](https://issues.apache.org/jira/browse/HADOOP-3703) | *Blocker* | **[HOD] logcondense needs to use the new pattern of output in hadoop dfs -lsr** +* [HADOOP-2865](https://issues.apache.org/jira/browse/HADOOP-2865) | *Major* | **FsShell.ls() should print file attributes first then the path name.** -Modified logcondense.py to use the new format of hadoop dfs -lsr output. This version of logcondense would not work with previous versions of Hadoop and hence is incompatible. +Changed the output of the "fs -ls" command to more closely match familiar Linux format. Additional changes were made by HADOOP-3459. Applications that parse the command output should be reviewed. --- -* [HADOOP-3683](https://issues.apache.org/jira/browse/HADOOP-3683) | *Major* | **Hadoop dfs metric FilesListed shows number of files listed instead of operations** +* [HADOOP-3061](https://issues.apache.org/jira/browse/HADOOP-3061) | *Major* | **Writable for single byte and double** -Change FileListed to getNumGetListingOps and add CreateFileOps, DeleteFileOps and AddBlockOps metrics. +Introduced ByteWritable and DoubleWritable (implementing WritableComparable) implementations for Byte and Double. --- -* [HADOOP-3677](https://issues.apache.org/jira/browse/HADOOP-3677) | *Blocker* | **Problems with generation stamp upgrade** +* [HADOOP-3164](https://issues.apache.org/jira/browse/HADOOP-3164) | *Major* | **Use FileChannel.transferTo() when data is read from DataNode.** -Simplify generation stamp upgrade by making is a local upgrade on datandodes. Deleted distributed upgrade. +Changed data node to use FileChannel.tranferTo() to transfer block data. --- -* [HADOOP-3665](https://issues.apache.org/jira/browse/HADOOP-3665) | *Minor* | **WritableComparator newKey() fails for NullWritable** +* [HADOOP-3283](https://issues.apache.org/jira/browse/HADOOP-3283) | *Major* | **Need a mechanism for data nodes to update generation stamps.** -**WARNING: No release note provided for this incompatible change.** +Added an IPC server in DataNode and a new IPC protocol InterDatanodeProtocol. Added conf properties dfs.datanode.ipc.address and dfs.datanode.handler.count with defaults "0.0.0.0:50020" and 3, respectively. +Changed the serialization in DatanodeRegistration and DatanodeInfo, and therefore, updated the versionID in ClientProtocol, DatanodeProtocol, NamenodeProtocol. --- -* [HADOOP-3610](https://issues.apache.org/jira/browse/HADOOP-3610) | *Blocker* | **[HOD] HOD does not automatically create a cluster directory for the script option** +* [HADOOP-2797](https://issues.apache.org/jira/browse/HADOOP-2797) | *Critical* | **Withdraw CRC upgrade from HDFS** -Modified HOD to automatically create a cluster directory if the one specified with the script command does not exist. +Withdrew the upgrade-to-CRC facility. HDFS will no longer support upgrades from versions without CRCs for block data. Users upgrading from version 0.13 or earlier must first upgrade to an intermediate (0.14, 0.15, 0.16, 0.17) version before doing upgrade to version 0.18 or later. --- -* [HADOOP-3598](https://issues.apache.org/jira/browse/HADOOP-3598) | *Blocker* | **Map-Reduce framework needlessly creates temporary \_${taskid} directories for Maps** +* [HADOOP-544](https://issues.apache.org/jira/browse/HADOOP-544) | *Major* | **Replace the job, tip and task ids with objects.** -Changed Map-Reduce framework to no longer create temporary task output directories for staging outputs if staging outputs isn't necessary. ${mapred.out.dir}/\_temporary/\_${taskid} +Introduced new classes JobID, TaskID and TaskAttemptID, which should be used instead of their string counterparts. Deprecated functions in JobClient, TaskReport, RunningJob, jobcontrol.Job and TaskCompletionEvent that use string arguments. Applications can use xxxID.toString() and xxxID.forName() methods to convert/restore objects to/from strings. --- -* [HADOOP-3569](https://issues.apache.org/jira/browse/HADOOP-3569) | *Minor* | **KFS input stream read() returns 4 bytes instead of 1** +* [HADOOP-2188](https://issues.apache.org/jira/browse/HADOOP-2188) | *Major* | **RPC should send a ping rather than use client timeouts** -Fixed KFS to have read() read and return 1 byte instead of 4. +Replaced timeouts with pings to check that client connection is alive. Removed the property ipc.client.timeout from the default Hadoop configuration. Removed the metric RpcOpsDiscardedOPsNum. --- -* [HADOOP-3564](https://issues.apache.org/jira/browse/HADOOP-3564) | *Blocker* | **Sometime after successful hod allocation datanode fails to come up with java.net.BindException for dfs.datanode.ipc.address** +* [HADOOP-2181](https://issues.apache.org/jira/browse/HADOOP-2181) | *Minor* | **Input Split details for maps should be logged** -Modifed HOD to generate the dfs.datanode.ipc.address parameter in the hadoop-site.xml of datanodes that it launches. +Added logging for input splits in job tracker log and job history log. Added web UI for viewing input splits in the job UI and history UI. --- -* [HADOOP-3512](https://issues.apache.org/jira/browse/HADOOP-3512) | *Major* | **Split map/reduce tools into separate jars** +* [HADOOP-3317](https://issues.apache.org/jira/browse/HADOOP-3317) | *Minor* | **add default port for hdfs namenode** -Separated Distcp, Logalyzer and Archiver into a tools jar. +Changed the default port for "hdfs:" URIs to be 8020, so that one may simply use URIs of the form "hdfs://example.com/dir/file". --- -* [HADOOP-3486](https://issues.apache.org/jira/browse/HADOOP-3486) | *Major* | **Change default for initial block report to 0 sec and document it in hadoop-defaults.xml** +* [HADOOP-2019](https://issues.apache.org/jira/browse/HADOOP-2019) | *Major* | **DistributedFileCache should support .tgz files in addition to jars and zip files** -Changed the default value of dfs.blockreport.initialDelay to be 0 seconds. +Added support for .tar, .tgz and .tar.gz files in DistributedCache. File sizes are limited to 2GB. --- -* [HADOOP-3483](https://issues.apache.org/jira/browse/HADOOP-3483) | *Major* | **[HOD] Improvements with cluster directory handling** +* [HADOOP-3058](https://issues.apache.org/jira/browse/HADOOP-3058) | *Minor* | **Hadoop DFS to report more replication metrics** -Modified HOD to create a cluster directory if one does not exist and to auto-deallocate a cluster while reallocating it, if it is already dead. +Added FSNamesystem status metrics. --- -* [HADOOP-3464](https://issues.apache.org/jira/browse/HADOOP-3464) | *Major* | **[HOD] HOD can improve error messages by reporting failures on compute nodes back to hod client** +* [HADOOP-3226](https://issues.apache.org/jira/browse/HADOOP-3226) | *Major* | **Run combiner when merging spills from map output** -Implemented a mechanism to transfer HOD errors that occur on compute nodes to the submit node running the HOD client, so users have good feedback on why an allocation failed. +Changed policy for running combiner. The combiner may be run multiple times as the map's output is sorted and merged. Additionally, it may be run on the reduce side as data is merged. The old semantics are available in Hadoop 0.18 if the user calls: +job.setCombineOnlyOnce(true); --- -* [HADOOP-3460](https://issues.apache.org/jira/browse/HADOOP-3460) | *Minor* | **SequenceFileAsBinaryOutputFormat** +* [HADOOP-3299](https://issues.apache.org/jira/browse/HADOOP-3299) | *Major* | **org.apache.hadoop.mapred.join.CompositeInputFormat does not initialize TextInput format files with the configuration resulting in an NullPointerException** -Created SequenceFileAsBinaryOutputFormat to write raw bytes as keys and values to a SequenceFile. +Changed the TextInputFormat and KeyValueTextInput classes to initialize the compressionCodecs member variable before dereferencing it. --- -* [HADOOP-3459](https://issues.apache.org/jira/browse/HADOOP-3459) | *Major* | **Change dfs -ls listing to closely match format on Linux** +* [HADOOP-3329](https://issues.apache.org/jira/browse/HADOOP-3329) | *Major* | **DatanodeDescriptor objects stored in FSImage may be out dated.** -Changed the output of the "fs -ls" command to more closely match familiar Linux format. Applications that parse the command output should be reviewed. +Changed format of file system image to not store locations of last block. --- -* [HADOOP-3452](https://issues.apache.org/jira/browse/HADOOP-3452) | *Minor* | **fsck exit code would be better if non-zero when FS corrupt** +* [HADOOP-2065](https://issues.apache.org/jira/browse/HADOOP-2065) | *Major* | **Replication policy for corrupted block** -Changed exit status of fsck to report whether the files system is healthy or corrupt. +Added "corrupt" flag to LocatedBlock to indicate that all replicas of the block thought to be corrupt. --- -* [HADOOP-3429](https://issues.apache.org/jira/browse/HADOOP-3429) | *Major* | **Increase the buffersize for the streaming parent java process's streams** +* [HADOOP-1702](https://issues.apache.org/jira/browse/HADOOP-1702) | *Major* | **Reduce buffer copies when data is written to DFS** -Increased the size of the buffer used in the communication between the Java task and the Streaming process to 128KB. +Reduced buffer copies as data is written to HDFS. The order of sending data bytes and control information has changed, but this will not be observed by client applications. --- -* [HADOOP-3427](https://issues.apache.org/jira/browse/HADOOP-3427) | *Major* | **In ReduceTask::fetchOutputs, wait for result can be improved slightly** +* [HADOOP-2656](https://issues.apache.org/jira/browse/HADOOP-2656) | *Major* | **Support for upgrading existing cluster to facilitate appends to HDFS files** -Changed shuffle scheduler policy to wait for notifications from shuffle threads before scheduling more. +Associated a generation stamp with each block. On data nodes, the generation stamp is stored as part of the file name of the block's meta-data file. --- -* [HADOOP-3417](https://issues.apache.org/jira/browse/HADOOP-3417) | *Major* | **JobClient should not have a static configuration for cli parsing** +* [HADOOP-3355](https://issues.apache.org/jira/browse/HADOOP-3355) | *Major* | **Configuration should accept decimal and hexadecimal values** -Removed the public class org.apache.hadoop.mapred.JobShell. -Command line options -libjars, -files and -archives are moved to GenericCommands. Thus applications have to implement org.apache.hadoop.util.Tool to use the options. +Added support for hexadecimal values in Configuration --- -* [HADOOP-3405](https://issues.apache.org/jira/browse/HADOOP-3405) | *Major* | **Make mapred internal classes package-local** +* [HADOOP-3390](https://issues.apache.org/jira/browse/HADOOP-3390) | *Major* | **Remove deprecated ClientProtocol.abandonFileInProgress()** -Refactored previously public classes MapTaskStatus, ReduceTaskStatus, JobSubmissionProtocol, CompletedJobStatusStore to be package local. +Removed deprecated ClientProtocol.abandonFileInProgress(). --- -* [HADOOP-3390](https://issues.apache.org/jira/browse/HADOOP-3390) | *Major* | **Remove deprecated ClientProtocol.abandonFileInProgress()** +* [HADOOP-3221](https://issues.apache.org/jira/browse/HADOOP-3221) | *Major* | **Need a "LineBasedTextInputFormat"** -Removed deprecated ClientProtocol.abandonFileInProgress(). +Added org.apache.hadoop.mapred.lib.NLineInputFormat ,which splits N lines of input as one split. N can be specified by configuration property "mapred.line.input.format.linespermap", which defaults to 1. --- -* [HADOOP-3379](https://issues.apache.org/jira/browse/HADOOP-3379) | *Blocker* | **Document the "stream.non.zero.exit.status.is.failure" knob for streaming** +* [HADOOP-3013](https://issues.apache.org/jira/browse/HADOOP-3013) | *Major* | **fsck to show (checksum) corrupted files** -Set default value for configuration property "stream.non.zero.exit.status.is.failure" to be "true". +fsck reports corrupt blocks in the system. --- -* [HADOOP-3376](https://issues.apache.org/jira/browse/HADOOP-3376) | *Major* | **[HOD] HOD should have a way to detect and deal with clusters that violate/exceed resource manager limits** +* [HADOOP-3336](https://issues.apache.org/jira/browse/HADOOP-3336) | *Major* | **Direct a subset of namenode RPC events for audit logging** -Modified HOD client to look for specific messages related to resource limit overruns and take appropriate actions - such as either failing to allocate the cluster, or issuing a warning to the user. A tool is provided, specific to Maui and Torque, that will set these specific messages. +Added a log4j appender that emits events from FSNamesystem for audit logging --- -* [HADOOP-3366](https://issues.apache.org/jira/browse/HADOOP-3366) | *Major* | **Shuffle/Merge improvements** +* [HADOOP-3339](https://issues.apache.org/jira/browse/HADOOP-3339) | *Major* | **DFS Write pipeline does not detect defective datanode correctly if it times out.** -Improved shuffle so that all fetched map-outputs are kept in-memory before being merged by stalling the shuffle so that the in-memory merge executes and frees up memory for the shuffle. +Improved failure handling of last Data Node in write pipeline. --- -* [HADOOP-3355](https://issues.apache.org/jira/browse/HADOOP-3355) | *Major* | **Configuration should accept decimal and hexadecimal values** +* [HADOOP-2867](https://issues.apache.org/jira/browse/HADOOP-2867) | *Major* | **Add a task's cwd to it's LD\_LIBRARY\_PATH** -Added support for hexadecimal values in Configuration +Added task's cwd to its LD\_LIBRARY\_PATH. --- -* [HADOOP-3339](https://issues.apache.org/jira/browse/HADOOP-3339) | *Major* | **DFS Write pipeline does not detect defective datanode correctly if it times out.** +* [HADOOP-1915](https://issues.apache.org/jira/browse/HADOOP-1915) | *Minor* | **adding counters methods using String (as opposed to Enum)** -Improved failure handling of last Data Node in write pipeline. +Provided a new method to update counters. "incrCounter(String group, String counter, long amount)" --- -* [HADOOP-3336](https://issues.apache.org/jira/browse/HADOOP-3336) | *Major* | **Direct a subset of namenode RPC events for audit logging** +* [HADOOP-3405](https://issues.apache.org/jira/browse/HADOOP-3405) | *Major* | **Make mapred internal classes package-local** -Added a log4j appender that emits events from FSNamesystem for audit logging +Refactored previously public classes MapTaskStatus, ReduceTaskStatus, JobSubmissionProtocol, CompletedJobStatusStore to be package local. --- -* [HADOOP-3329](https://issues.apache.org/jira/browse/HADOOP-3329) | *Major* | **DatanodeDescriptor objects stored in FSImage may be out dated.** +* [HADOOP-3232](https://issues.apache.org/jira/browse/HADOOP-3232) | *Critical* | **Datanodes time out** -Changed format of file system image to not store locations of last block. +Changed 'du' command to run in a seperate thread so that it does not block user. --- -* [HADOOP-3326](https://issues.apache.org/jira/browse/HADOOP-3326) | *Major* | **ReduceTask should not sleep for 200 ms while waiting for merge to finish** +* [HADOOP-3035](https://issues.apache.org/jira/browse/HADOOP-3035) | *Major* | **Data nodes should inform the name-node about block crc errors.** -Changed fetchOutputs() so that LocalFSMerger and InMemFSMergeThread threads are spawned only once. The thread gets notified when something is ready for merge. The merge happens when thresholds are met. +Changed protocol for transferring blocks between data nodes to report corrupt blocks to data node for re-replication from a good replica. --- -* [HADOOP-3317](https://issues.apache.org/jira/browse/HADOOP-3317) | *Minor* | **add default port for hdfs namenode** +* [HADOOP-3265](https://issues.apache.org/jira/browse/HADOOP-3265) | *Major* | **Remove deprecated API getFileCacheHints** -Changed the default port for "hdfs:" URIs to be 8020, so that one may simply use URIs of the form "hdfs\://example.com/dir/file". +Removed deprecated API getFileCacheHints --- -* [HADOOP-3310](https://issues.apache.org/jira/browse/HADOOP-3310) | *Major* | **Lease recovery for append** +* [HADOOP-4](https://issues.apache.org/jira/browse/HADOOP-4) | *Major* | **tool to mount dfs on linux** -Implemented Lease Recovery to sync the last bock of a file. Added ClientDatanodeProtocol for client trigging block recovery. Changed DatanodeProtocol to support block synchronization. Changed InterDatanodeProtocol to support block update. +Introduced FUSE module for HDFS. Module allows mount of HDFS as a Unix filesystem, and optionally the export of that mount point to other machines. Writes are disabled. rmdir, mv, mkdir, rm are supported, but not cp, touch, and the like. Usage information is attached to the Jira record. --- -* [HADOOP-3307](https://issues.apache.org/jira/browse/HADOOP-3307) | *Major* | **Archives in Hadoop.** +* [HADOOP-3246](https://issues.apache.org/jira/browse/HADOOP-3246) | *Major* | **FTP client over HDFS** -Introduced archive feature to Hadoop. A Map/Reduce job can be run to create an archive with indexes. A FileSystem abstraction is provided over the archive. +Introduced an FTPFileSystem backed by Apache Commons FTPClient to directly store data into HDFS. --- -* [HADOOP-3299](https://issues.apache.org/jira/browse/HADOOP-3299) | *Major* | **org.apache.hadoop.mapred.join.CompositeInputFormat does not initialize TextInput format files with the configuration resulting in an NullPointerException** +* [HADOOP-3310](https://issues.apache.org/jira/browse/HADOOP-3310) | *Major* | **Lease recovery for append** -Changed the TextInputFormat and KeyValueTextInput classes to initialize the compressionCodecs member variable before dereferencing it. +Implemented Lease Recovery to sync the last bock of a file. Added ClientDatanodeProtocol for client trigging block recovery. Changed DatanodeProtocol to support block synchronization. Changed InterDatanodeProtocol to support block update. --- -* [HADOOP-3283](https://issues.apache.org/jira/browse/HADOOP-3283) | *Major* | **Need a mechanism for data nodes to update generation stamps.** +* [HADOOP-3376](https://issues.apache.org/jira/browse/HADOOP-3376) | *Major* | **[HOD] HOD should have a way to detect and deal with clusters that violate/exceed resource manager limits** -Added an IPC server in DataNode and a new IPC protocol InterDatanodeProtocol. Added conf properties dfs.datanode.ipc.address and dfs.datanode.handler.count with defaults "0.0.0.0:50020" and 3, respectively. -Changed the serialization in DatanodeRegistration and DatanodeInfo, and therefore, updated the versionID in ClientProtocol, DatanodeProtocol, NamenodeProtocol. +Modified HOD client to look for specific messages related to resource limit overruns and take appropriate actions - such as either failing to allocate the cluster, or issuing a warning to the user. A tool is provided, specific to Maui and Torque, that will set these specific messages. --- -* [HADOOP-3265](https://issues.apache.org/jira/browse/HADOOP-3265) | *Major* | **Remove deprecated API getFileCacheHints** +* [HADOOP-3429](https://issues.apache.org/jira/browse/HADOOP-3429) | *Major* | **Increase the buffersize for the streaming parent java process's streams** -Removed deprecated API getFileCacheHints +Increased the size of the buffer used in the communication between the Java task and the Streaming process to 128KB. --- -* [HADOOP-3246](https://issues.apache.org/jira/browse/HADOOP-3246) | *Major* | **FTP client over HDFS** +* [HADOOP-3417](https://issues.apache.org/jira/browse/HADOOP-3417) | *Major* | **JobClient should not have a static configuration for cli parsing** -Introduced an FTPFileSystem backed by Apache Commons FTPClient to directly store data into HDFS. +Removed the public class org.apache.hadoop.mapred.JobShell. +Command line options -libjars, -files and -archives are moved to GenericCommands. Thus applications have to implement org.apache.hadoop.util.Tool to use the options. --- -* [HADOOP-3232](https://issues.apache.org/jira/browse/HADOOP-3232) | *Critical* | **Datanodes time out** +* [HADOOP-2132](https://issues.apache.org/jira/browse/HADOOP-2132) | *Critical* | **Killing successfully completed jobs moves them to failed** -Changed 'du' command to run in a seperate thread so that it does not block user. +Change "job -kill" to only allow a job that is in the RUNNING or PREP state to be killed. --- -* [HADOOP-3230](https://issues.apache.org/jira/browse/HADOOP-3230) | *Major* | **Add command line access to named counters** +* [HADOOP-2909](https://issues.apache.org/jira/browse/HADOOP-2909) | *Major* | **Improve IPC idle connection management** -Added command line tool "job -counter \<job-id\> \<group-name\> \<counter-name\>" to access counters. +Removed property ipc.client.maxidletime from the default configuration. The allowed idle time is twice ipc.client.connection.maxidletime. --- -* [HADOOP-3226](https://issues.apache.org/jira/browse/HADOOP-3226) | *Major* | **Run combiner when merging spills from map output** +* [HADOOP-3486](https://issues.apache.org/jira/browse/HADOOP-3486) | *Major* | **Change default for initial block report to 0 sec and document it in hadoop-defaults.xml** -Changed policy for running combiner. The combiner may be run multiple times as the map's output is sorted and merged. Additionally, it may be run on the reduce side as data is merged. The old semantics are available in Hadoop 0.18 if the user calls: -job.setCombineOnlyOnce(true); +Changed the default value of dfs.blockreport.initialDelay to be 0 seconds. --- -* [HADOOP-3221](https://issues.apache.org/jira/browse/HADOOP-3221) | *Major* | **Need a "LineBasedTextInputFormat"** +* [HADOOP-3459](https://issues.apache.org/jira/browse/HADOOP-3459) | *Major* | **Change dfs -ls listing to closely match format on Linux** -Added org.apache.hadoop.mapred.lib.NLineInputFormat ,which splits N lines of input as one split. N can be specified by configuration property "mapred.line.input.format.linespermap", which defaults to 1. +Changed the output of the "fs -ls" command to more closely match familiar Linux format. Applications that parse the command output should be reviewed. --- -* [HADOOP-3193](https://issues.apache.org/jira/browse/HADOOP-3193) | *Minor* | **Discovery of corrupt block reported in name node log** +* [HADOOP-3177](https://issues.apache.org/jira/browse/HADOOP-3177) | *Major* | **Expose DFSOutputStream.fsync API though the FileSystem interface** -Added reporter to FSNamesystem stateChangeLog, and a new metric to track the number of corrupted replicas. +Added a new public interface Syncable which declares the sync() operation. FSDataOutputStream implements Syncable. If the wrappedStream in FSDataOutputStream is Syncalbe, calling FSDataOutputStream.sync() is equivalent to call wrappedStream.sync(). Otherwise, FSDataOutputStream.sync() is a no-op. Both DistributedFileSystem and LocalFileSystem support the sync() operation. --- -* [HADOOP-3187](https://issues.apache.org/jira/browse/HADOOP-3187) | *Major* | **Quotas for name space management** +* [HADOOP-3113](https://issues.apache.org/jira/browse/HADOOP-3113) | *Major* | **DFSOututStream.flush() should flush data to real block file on DataNode.** -Introduced directory quota as hard limits on the number of names in the tree rooted at that directory. An administrator may set quotas on individual directories explicitly. Newly created directories have no associated quota. File/directory creations fault if the quota would be exceeded. The attempt to set a quota faults if the directory would be in violation of the new quota. +Added sync() method to FSDataOutputStream to really, really persist data in HDFS. InterDatanodeProtocol to implement this feature. --- -* [HADOOP-3184](https://issues.apache.org/jira/browse/HADOOP-3184) | *Major* | **HOD gracefully exclude "bad" nodes during ring formation** +* [HADOOP-3464](https://issues.apache.org/jira/browse/HADOOP-3464) | *Major* | **[HOD] HOD can improve error messages by reporting failures on compute nodes back to hod client** -Modified HOD to handle master (NameNode or JobTracker) failures on bad nodes by trying to bring them up on another node in the ring. Introduced new property ringmaster.max-master-failures to specify the maximum number of times a master is allowed to fail. +Implemented a mechanism to transfer HOD errors that occur on compute nodes to the submit node running the HOD client, so users have good feedback on why an allocation failed. --- -* [HADOOP-3177](https://issues.apache.org/jira/browse/HADOOP-3177) | *Major* | **Expose DFSOutputStream.fsync API though the FileSystem interface** +* [HADOOP-3452](https://issues.apache.org/jira/browse/HADOOP-3452) | *Minor* | **fsck exit code would be better if non-zero when FS corrupt** -Added a new public interface Syncable which declares the sync() operation. FSDataOutputStream implements Syncable. If the wrappedStream in FSDataOutputStream is Syncalbe, calling FSDataOutputStream.sync() is equivalent to call wrappedStream.sync(). Otherwise, FSDataOutputStream.sync() is a no-op. Both DistributedFileSystem and LocalFileSystem support the sync() operation. +Changed exit status of fsck to report whether the files system is healthy or corrupt. --- -* [HADOOP-3164](https://issues.apache.org/jira/browse/HADOOP-3164) | *Major* | **Use FileChannel.transferTo() when data is read from DataNode.** +* [HADOOP-1328](https://issues.apache.org/jira/browse/HADOOP-1328) | *Major* | **Hadoop Streaming needs to provide a way for the stream plugin to update global counters** -Changed data node to use FileChannel.tranferTo() to transfer block data. +Introduced a way for a streaming process to update global counters and status using stderr stream to emit information. Use "reporter:counter:\<group\>,\<counter\>,\<amount\> " to update a counter. Use "reporter:status:\<message\>" to update status. --- -* [HADOOP-3135](https://issues.apache.org/jira/browse/HADOOP-3135) | *Critical* | **if the 'mapred.system.dir' in the client jobconf is different from the JobTracker's value job submission fails** +* [HADOOP-3095](https://issues.apache.org/jira/browse/HADOOP-3095) | *Major* | **Validating input paths and creating splits is slow on S3** -Changed job submission protocol to not allow submission if the client's value of mapred.system.dir does not match the job tracker's. Deprecated JobConf.getSystemDir(); use JobClient.getSystemDir(). +Added overloaded method getFileBlockLocations(FileStatus, long, long). This is an incompatible change for FileSystem implementations which override getFileBlockLocations(Path, long, long). They should have the signature of this method changed to getFileBlockLocations(FileStatus, long, long) to work correctly. --- -* [HADOOP-3113](https://issues.apache.org/jira/browse/HADOOP-3113) | *Major* | **DFSOututStream.flush() should flush data to real block file on DataNode.** +* [HADOOP-2095](https://issues.apache.org/jira/browse/HADOOP-2095) | *Major* | **Reducer failed due to Out ofMemory** -Added sync() method to FSDataOutputStream to really, really persist data in HDFS. InterDatanodeProtocol to implement this feature. +Reduced in-memory copies of keys and values as they flow through the Map-Reduce framework. Changed the storage of intermediate map outputs to use new IFile instead of SequenceFile for better compression. --- -* [HADOOP-3095](https://issues.apache.org/jira/browse/HADOOP-3095) | *Major* | **Validating input paths and creating splits is slow on S3** +* [HADOOP-2427](https://issues.apache.org/jira/browse/HADOOP-2427) | *Major* | **Cleanup of mapred.local.dir after maptask is complete** -Added overloaded method getFileBlockLocations(FileStatus, long, long). This is an incompatible change for FileSystem implementations which override getFileBlockLocations(Path, long, long). They should have the signature of this method changed to getFileBlockLocations(FileStatus, long, long) to work correctly. +The current working directory of a task, i.e. ${mapred.local.dir}/taskTracker/jobcache/\<jobid\>/\<task\_dir\>/work is cleanedup, as soon as the task is finished. --- -* [HADOOP-3061](https://issues.apache.org/jira/browse/HADOOP-3061) | *Major* | **Writable for single byte and double** +* [HADOOP-3187](https://issues.apache.org/jira/browse/HADOOP-3187) | *Major* | **Quotas for name space management** -Introduced ByteWritable and DoubleWritable (implementing WritableComparable) implementations for Byte and Double. +Introduced directory quota as hard limits on the number of names in the tree rooted at that directory. An administrator may set quotas on individual directories explicitly. Newly created directories have no associated quota. File/directory creations fault if the quota would be exceeded. The attempt to set a quota faults if the directory would be in violation of the new quota. --- -* [HADOOP-3058](https://issues.apache.org/jira/browse/HADOOP-3058) | *Minor* | **Hadoop DFS to report more replication metrics** +* [HADOOP-3307](https://issues.apache.org/jira/browse/HADOOP-3307) | *Major* | **Archives in Hadoop.** -Added FSNamesystem status metrics. +Introduced archive feature to Hadoop. A Map/Reduce job can be run to create an archive with indexes. A FileSystem abstraction is provided over the archive. --- -* [HADOOP-3035](https://issues.apache.org/jira/browse/HADOOP-3035) | *Major* | **Data nodes should inform the name-node about block crc errors.** +* [HADOOP-3326](https://issues.apache.org/jira/browse/HADOOP-3326) | *Major* | **ReduceTask should not sleep for 200 ms while waiting for merge to finish** -Changed protocol for transferring blocks between data nodes to report corrupt blocks to data node for re-replication from a good replica. +Changed fetchOutputs() so that LocalFSMerger and InMemFSMergeThread threads are spawned only once. The thread gets notified when something is ready for merge. The merge happens when thresholds are met. --- -* [HADOOP-3013](https://issues.apache.org/jira/browse/HADOOP-3013) | *Major* | **fsck to show (checksum) corrupted files** +* [HADOOP-3483](https://issues.apache.org/jira/browse/HADOOP-3483) | *Major* | **[HOD] Improvements with cluster directory handling** -fsck reports corrupt blocks in the system. +Modified HOD to create a cluster directory if one does not exist and to auto-deallocate a cluster while reallocating it, if it is already dead. --- -* [HADOOP-2909](https://issues.apache.org/jira/browse/HADOOP-2909) | *Major* | **Improve IPC idle connection management** +* [HADOOP-236](https://issues.apache.org/jira/browse/HADOOP-236) | *Major* | **job tracker should refuse connection from a task tracker with a different version number** -Removed property ipc.client.maxidletime from the default configuration. The allowed idle time is twice ipc.client.connection.maxidletime. +Changed connection protocol job tracker and task tracker so that task tracker will not connect to a job tracker with a different build version. --- -* [HADOOP-2867](https://issues.apache.org/jira/browse/HADOOP-2867) | *Major* | **Add a task's cwd to it's LD\_LIBRARY\_PATH** +* [HADOOP-3427](https://issues.apache.org/jira/browse/HADOOP-3427) | *Major* | **In ReduceTask::fetchOutputs, wait for result can be improved slightly** -Added task's cwd to its LD\_LIBRARY\_PATH. +Changed shuffle scheduler policy to wait for notifications from shuffle threads before scheduling more. --- -* [HADOOP-2865](https://issues.apache.org/jira/browse/HADOOP-2865) | *Major* | **FsShell.ls() should print file attributes first then the path name.** +* [HADOOP-3184](https://issues.apache.org/jira/browse/HADOOP-3184) | *Major* | **HOD gracefully exclude "bad" nodes during ring formation** -Changed the output of the "fs -ls" command to more closely match familiar Linux format. Additional changes were made by HADOOP-3459. Applications that parse the command output should be reviewed. +Modified HOD to handle master (NameNode or JobTracker) failures on bad nodes by trying to bring them up on another node in the ring. Introduced new property ringmaster.max-master-failures to specify the maximum number of times a master is allowed to fail. --- -* [HADOOP-2797](https://issues.apache.org/jira/browse/HADOOP-2797) | *Critical* | **Withdraw CRC upgrade from HDFS** +* [HADOOP-3135](https://issues.apache.org/jira/browse/HADOOP-3135) | *Critical* | **if the 'mapred.system.dir' in the client jobconf is different from the JobTracker's value job submission fails** -Withdrew the upgrade-to-CRC facility. HDFS will no longer support upgrades from versions without CRCs for block data. Users upgrading from version 0.13 or earlier must first upgrade to an intermediate (0.14, 0.15, 0.16, 0.17) version before doing upgrade to version 0.18 or later. +Changed job submission protocol to not allow submission if the client's value of mapred.system.dir does not match the job tracker's. Deprecated JobConf.getSystemDir(); use JobClient.getSystemDir(). --- -* [HADOOP-2703](https://issues.apache.org/jira/browse/HADOOP-2703) | *Minor* | **New files under lease (before close) still shows up as MISSING files/blocks in fsck** +* [HADOOP-3460](https://issues.apache.org/jira/browse/HADOOP-3460) | *Minor* | **SequenceFileAsBinaryOutputFormat** -Changed fsck to ignore files opened for writing. Introduced new option "-openforwrite" to explicitly show open files. +Created SequenceFileAsBinaryOutputFormat to write raw bytes as keys and values to a SequenceFile. --- -* [HADOOP-2656](https://issues.apache.org/jira/browse/HADOOP-2656) | *Major* | **Support for upgrading existing cluster to facilitate appends to HDFS files** +* [HADOOP-3230](https://issues.apache.org/jira/browse/HADOOP-3230) | *Major* | **Add command line access to named counters** -Associated a generation stamp with each block. On data nodes, the generation stamp is stored as part of the file name of the block's meta-data file. +Added command line tool "job -counter \<job-id\> \<group-name\> \<counter-name\>" to access counters. --- -* [HADOOP-2585](https://issues.apache.org/jira/browse/HADOOP-2585) | *Major* | **Automatic namespace recovery from the secondary image.** +* [HADOOP-930](https://issues.apache.org/jira/browse/HADOOP-930) | *Major* | **Add support for reading regular (non-block-based) files from S3 in S3FileSystem** -Improved management of replicas of the name space image. If all replicas on the Name Node are lost, the latest check point can be loaded from the secondary Name Node. Use parameter "-importCheckpoint" and specify the location with "fs.checkpoint.dir." The directory structure on the secondary Name Node has changed to match the primary Name Node. +Added support for reading and writing native S3 files. Native S3 files are referenced using s3n URIs. See http://wiki.apache.org/hadoop/AmazonS3 for more details. --- -* [HADOOP-2427](https://issues.apache.org/jira/browse/HADOOP-2427) | *Major* | **Cleanup of mapred.local.dir after maptask is complete** +* [HADOOP-3193](https://issues.apache.org/jira/browse/HADOOP-3193) | *Minor* | **Discovery of corrupt block reported in name node log** -The current working directory of a task, i.e. ${mapred.local.dir}/taskTracker/jobcache/\<jobid\>/\<task\_dir\>/work is cleanedup, as soon as the task is finished. +Added reporter to FSNamesystem stateChangeLog, and a new metric to track the number of corrupted replicas. --- -* [HADOOP-2188](https://issues.apache.org/jira/browse/HADOOP-2188) | *Major* | **RPC should send a ping rather than use client timeouts** +* [HADOOP-3366](https://issues.apache.org/jira/browse/HADOOP-3366) | *Major* | **Shuffle/Merge improvements** -Replaced timeouts with pings to check that client connection is alive. Removed the property ipc.client.timeout from the default Hadoop configuration. Removed the metric RpcOpsDiscardedOPsNum. +Improved shuffle so that all fetched map-outputs are kept in-memory before being merged by stalling the shuffle so that the in-memory merge executes and frees up memory for the shuffle. --- -* [HADOOP-2181](https://issues.apache.org/jira/browse/HADOOP-2181) | *Minor* | **Input Split details for maps should be logged** +* [HADOOP-3512](https://issues.apache.org/jira/browse/HADOOP-3512) | *Major* | **Split map/reduce tools into separate jars** -Added logging for input splits in job tracker log and job history log. Added web UI for viewing input splits in the job UI and history UI. +Separated Distcp, Logalyzer and Archiver into a tools jar. --- -* [HADOOP-2132](https://issues.apache.org/jira/browse/HADOOP-2132) | *Critical* | **Killing successfully completed jobs moves them to failed** +* [HADOOP-3379](https://issues.apache.org/jira/browse/HADOOP-3379) | *Blocker* | **Document the "stream.non.zero.exit.status.is.failure" knob for streaming** -Change "job -kill" to only allow a job that is in the RUNNING or PREP state to be killed. +Set default value for configuration property "stream.non.zero.exit.status.is.failure" to be "true". --- -* [HADOOP-2095](https://issues.apache.org/jira/browse/HADOOP-2095) | *Major* | **Reducer failed due to Out ofMemory** +* [HADOOP-3564](https://issues.apache.org/jira/browse/HADOOP-3564) | *Blocker* | **Sometime after successful hod allocation datanode fails to come up with java.net.BindException for dfs.datanode.ipc.address** -Reduced in-memory copies of keys and values as they flow through the Map-Reduce framework. Changed the storage of intermediate map outputs to use new IFile instead of SequenceFile for better compression. +Modifed HOD to generate the dfs.datanode.ipc.address parameter in the hadoop-site.xml of datanodes that it launches. --- -* [HADOOP-2065](https://issues.apache.org/jira/browse/HADOOP-2065) | *Major* | **Replication policy for corrupted block** +* [HADOOP-3569](https://issues.apache.org/jira/browse/HADOOP-3569) | *Minor* | **KFS input stream read() returns 4 bytes instead of 1** -Added "corrupt" flag to LocatedBlock to indicate that all replicas of the block thought to be corrupt. +Fixed KFS to have read() read and return 1 byte instead of 4. --- -* [HADOOP-2019](https://issues.apache.org/jira/browse/HADOOP-2019) | *Major* | **DistributedFileCache should support .tgz files in addition to jars and zip files** +* [HADOOP-3598](https://issues.apache.org/jira/browse/HADOOP-3598) | *Blocker* | **Map-Reduce framework needlessly creates temporary \_${taskid} directories for Maps** -Added support for .tar, .tgz and .tar.gz files in DistributedCache. File sizes are limited to 2GB. +Changed Map-Reduce framework to no longer create temporary task output directories for staging outputs if staging outputs isn't necessary. ${mapred.out.dir}/\_temporary/\_${taskid} --- -* [HADOOP-1915](https://issues.apache.org/jira/browse/HADOOP-1915) | *Minor* | **adding counters methods using String (as opposed to Enum)** +* [HADOOP-3610](https://issues.apache.org/jira/browse/HADOOP-3610) | *Blocker* | **[HOD] HOD does not automatically create a cluster directory for the script option** -Provided a new method to update counters. "incrCounter(String group, String counter, long amount)" +Modified HOD to automatically create a cluster directory if the one specified with the script command does not exist. --- -* [HADOOP-1702](https://issues.apache.org/jira/browse/HADOOP-1702) | *Major* | **Reduce buffer copies when data is written to DFS** +* [HADOOP-3665](https://issues.apache.org/jira/browse/HADOOP-3665) | *Minor* | **WritableComparator newKey() fails for NullWritable** -Reduced buffer copies as data is written to HDFS. The order of sending data bytes and control information has changed, but this will not be observed by client applications. +**WARNING: No release note provided for this change.** --- -* [HADOOP-1328](https://issues.apache.org/jira/browse/HADOOP-1328) | *Major* | **Hadoop Streaming needs to provide a way for the stream plugin to update global counters** +* [HADOOP-3683](https://issues.apache.org/jira/browse/HADOOP-3683) | *Major* | **Hadoop dfs metric FilesListed shows number of files listed instead of operations** -Introduced a way for a streaming process to update global counters and status using stderr stream to emit information. Use "reporter:counter:\<group\>,\<counter\>,\<amount\> " to update a counter. Use "reporter:status:\<message\>" to update status. +Change FileListed to getNumGetListingOps and add CreateFileOps, DeleteFileOps and AddBlockOps metrics. --- -* [HADOOP-930](https://issues.apache.org/jira/browse/HADOOP-930) | *Major* | **Add support for reading regular (non-block-based) files from S3 in S3FileSystem** +* [HADOOP-3703](https://issues.apache.org/jira/browse/HADOOP-3703) | *Blocker* | **[HOD] logcondense needs to use the new pattern of output in hadoop dfs -lsr** -Added support for reading and writing native S3 files. Native S3 files are referenced using s3n URIs. See http://wiki.apache.org/hadoop/AmazonS3 for more details. +Modified logcondense.py to use the new format of hadoop dfs -lsr output. This version of logcondense would not work with previous versions of Hadoop and hence is incompatible. --- -* [HADOOP-544](https://issues.apache.org/jira/browse/HADOOP-544) | *Major* | **Replace the job, tip and task ids with objects.** +* [HADOOP-3677](https://issues.apache.org/jira/browse/HADOOP-3677) | *Blocker* | **Problems with generation stamp upgrade** -Introduced new classes JobID, TaskID and TaskAttemptID, which should be used instead of their string counterparts. Deprecated functions in JobClient, TaskReport, RunningJob, jobcontrol.Job and TaskCompletionEvent that use string arguments. Applications can use xxxID.toString() and xxxID.forName() methods to convert/restore objects to/from strings. +Simplify generation stamp upgrade by making is a local upgrade on datandodes. Deleted distributed upgrade. --- -* [HADOOP-236](https://issues.apache.org/jira/browse/HADOOP-236) | *Major* | **job tracker should refuse connection from a task tracker with a different version number** +* [HADOOP-3808](https://issues.apache.org/jira/browse/HADOOP-3808) | *Blocker* | **[HOD] Include job tracker RPC in notes attribute after job submission** -Changed connection protocol job tracker and task tracker so that task tracker will not connect to a job tracker with a different build version. +Modified HOD to include the RPC port of the JobTracker in the 'notes' attribute of the resource manager. The RPC port is included as the string 'Mapred RPC Port:\<port number\>'. Tools that depend on the value of the notes attribute must change to parse this new value. --- -* [HADOOP-4](https://issues.apache.org/jira/browse/HADOOP-4) | *Major* | **tool to mount dfs on linux** +* [HADOOP-3837](https://issues.apache.org/jira/browse/HADOOP-3837) | *Major* | **hadop streaming does not use progress reporting to detect hung tasks** -Introduced FUSE module for HDFS. Module allows mount of HDFS as a Unix filesystem, and optionally the export of that mount point to other machines. Writes are disabled. rmdir, mv, mkdir, rm are supported, but not cp, touch, and the like. Usage information is attached to the Jira record. +Changed streaming tasks to adhere to task timeout value specified in the job configuration.
http://git-wip-us.apache.org/repos/asf/hadoop/blob/19041008/hadoop-common-project/hadoop-common/src/site/markdown/release/0.18.1/CHANGES.0.18.1.md ---------------------------------------------------------------------- diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/release/0.18.1/CHANGES.0.18.1.md b/hadoop-common-project/hadoop-common/src/site/markdown/release/0.18.1/CHANGES.0.18.1.md index dec3ae1..87880c1 100644 --- a/hadoop-common-project/hadoop-common/src/site/markdown/release/0.18.1/CHANGES.0.18.1.md +++ b/hadoop-common-project/hadoop-common/src/site/markdown/release/0.18.1/CHANGES.0.18.1.md @@ -20,61 +20,27 @@ ## Release 0.18.1 - 2008-09-17 -### INCOMPATIBLE CHANGES: - -| JIRA | Summary | Priority | Component | Reporter | Contributor | -|:---- |:---- | :--- |:---- |:---- |:---- | - - -### IMPORTANT ISSUES: - -| JIRA | Summary | Priority | Component | Reporter | Contributor | -|:---- |:---- | :--- |:---- |:---- |:---- | - - -### NEW FEATURES: - -| JIRA | Summary | Priority | Component | Reporter | Contributor | -|:---- |:---- | :--- |:---- |:---- |:---- | ### IMPROVEMENTS: | JIRA | Summary | Priority | Component | Reporter | Contributor | |:---- |:---- | :--- |:---- |:---- |:---- | -| [HADOOP-4145](https://issues.apache.org/jira/browse/HADOOP-4145) | [HOD] Support an accounting plugin script for HOD | Blocker | contrib/hod | Hemanth Yamijala | Hemanth Yamijala | | [HADOOP-3934](https://issues.apache.org/jira/browse/HADOOP-3934) | Update log4j from 1.2.13 to 1.2.15 | Blocker | . | Don Kim | Owen O'Malley | +| [HADOOP-4145](https://issues.apache.org/jira/browse/HADOOP-4145) | [HOD] Support an accounting plugin script for HOD | Blocker | contrib/hod | Hemanth Yamijala | Hemanth Yamijala | ### BUG FIXES: | JIRA | Summary | Priority | Component | Reporter | Contributor | |:---- |:---- | :--- |:---- |:---- |:---- | -| [HADOOP-4161](https://issues.apache.org/jira/browse/HADOOP-4161) | [HOD] Uncaught exceptions can potentially hang hod-client. | Major | contrib/hod | Vinod Kumar Vavilapalli | Vinod Kumar Vavilapalli | -| [HADOOP-4060](https://issues.apache.org/jira/browse/HADOOP-4060) | [HOD] Make HOD to roll log files on the client | Blocker | contrib/hod | Hemanth Yamijala | Vinod Kumar Vavilapalli | -| [HADOOP-4046](https://issues.apache.org/jira/browse/HADOOP-4046) | WritableComparator's constructor should be protected instead of private | Major | io | Owen O'Malley | Chris Douglas | -| [HADOOP-4040](https://issues.apache.org/jira/browse/HADOOP-4040) | Remove the hardcoded ipc.client.connection.maxidletime setting from the TaskTracker.Child.main() | Blocker | . | Devaraj Das | Devaraj Das | -| [HADOOP-4037](https://issues.apache.org/jira/browse/HADOOP-4037) | HadoopQA contrib -1 comments due to inconsistency in eclipse plugin declared jars | Blocker | . | Nigel Daley | Nigel Daley | -| [HADOOP-3995](https://issues.apache.org/jira/browse/HADOOP-3995) | renameTo(src, dst) does not restore src name in case of quota failure. | Blocker | . | Raghu Angadi | Raghu Angadi | | [HADOOP-3940](https://issues.apache.org/jira/browse/HADOOP-3940) | Reduce often attempts in memory merge with no work | Minor | . | Chris Douglas | Chris Douglas | | [HADOOP-3821](https://issues.apache.org/jira/browse/HADOOP-3821) | SequenceFile's Reader.decompressorPool or Writer.decompressorPool gets into an inconsistent state when calling close() more than once | Major | io | Peter Voss | Arun C Murthy | - - -### TESTS: - -| JIRA | Summary | Priority | Component | Reporter | Contributor | -|:---- |:---- | :--- |:---- |:---- |:---- | - - -### SUB-TASKS: - -| JIRA | Summary | Priority | Component | Reporter | Contributor | -|:---- |:---- | :--- |:---- |:---- |:---- | - - -### OTHER: - -| JIRA | Summary | Priority | Component | Reporter | Contributor | -|:---- |:---- | :--- |:---- |:---- |:---- | +| [HADOOP-3995](https://issues.apache.org/jira/browse/HADOOP-3995) | renameTo(src, dst) does not restore src name in case of quota failure. | Blocker | . | Raghu Angadi | Raghu Angadi | +| [HADOOP-4037](https://issues.apache.org/jira/browse/HADOOP-4037) | HadoopQA contrib -1 comments due to inconsistency in eclipse plugin declared jars | Blocker | . | Nigel Daley | Nigel Daley | +| [HADOOP-4040](https://issues.apache.org/jira/browse/HADOOP-4040) | Remove the hardcoded ipc.client.connection.maxidletime setting from the TaskTracker.Child.main() | Blocker | . | Devaraj Das | Devaraj Das | +| [HADOOP-4046](https://issues.apache.org/jira/browse/HADOOP-4046) | WritableComparator's constructor should be protected instead of private | Major | io | Owen O'Malley | Chris Douglas | +| [HADOOP-4060](https://issues.apache.org/jira/browse/HADOOP-4060) | [HOD] Make HOD to roll log files on the client | Blocker | contrib/hod | Hemanth Yamijala | Vinod Kumar Vavilapalli | +| [HADOOP-4161](https://issues.apache.org/jira/browse/HADOOP-4161) | [HOD] Uncaught exceptions can potentially hang hod-client. | Major | contrib/hod | Vinod Kumar Vavilapalli | Vinod Kumar Vavilapalli | http://git-wip-us.apache.org/repos/asf/hadoop/blob/19041008/hadoop-common-project/hadoop-common/src/site/markdown/release/0.18.1/RELEASENOTES.0.18.1.md ---------------------------------------------------------------------- diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/release/0.18.1/RELEASENOTES.0.18.1.md b/hadoop-common-project/hadoop-common/src/site/markdown/release/0.18.1/RELEASENOTES.0.18.1.md index e9d1c06..4fa1510 100644 --- a/hadoop-common-project/hadoop-common/src/site/markdown/release/0.18.1/RELEASENOTES.0.18.1.md +++ b/hadoop-common-project/hadoop-common/src/site/markdown/release/0.18.1/RELEASENOTES.0.18.1.md @@ -23,16 +23,16 @@ These release notes cover new developer and user-facing incompatibilities, impor --- -* [HADOOP-4060](https://issues.apache.org/jira/browse/HADOOP-4060) | *Blocker* | **[HOD] Make HOD to roll log files on the client** +* [HADOOP-3995](https://issues.apache.org/jira/browse/HADOOP-3995) | *Blocker* | **renameTo(src, dst) does not restore src name in case of quota failure.** -HOD client was modified to roll over client logs being written to the cluster directory. A new configuration parameter, hod.log-rollover-count, was introduced to specify how many rollover log files to retain. +In case of quota failure on HDFS, rename does not restore source filename. --- -* [HADOOP-3995](https://issues.apache.org/jira/browse/HADOOP-3995) | *Blocker* | **renameTo(src, dst) does not restore src name in case of quota failure.** +* [HADOOP-4060](https://issues.apache.org/jira/browse/HADOOP-4060) | *Blocker* | **[HOD] Make HOD to roll log files on the client** -In case of quota failure on HDFS, rename does not restore source filename. +HOD client was modified to roll over client logs being written to the cluster directory. A new configuration parameter, hod.log-rollover-count, was introduced to specify how many rollover log files to retain. http://git-wip-us.apache.org/repos/asf/hadoop/blob/19041008/hadoop-common-project/hadoop-common/src/site/markdown/release/0.18.2/CHANGES.0.18.2.md ---------------------------------------------------------------------- diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/release/0.18.2/CHANGES.0.18.2.md b/hadoop-common-project/hadoop-common/src/site/markdown/release/0.18.2/CHANGES.0.18.2.md index d7e6ac7..4460629 100644 --- a/hadoop-common-project/hadoop-common/src/site/markdown/release/0.18.2/CHANGES.0.18.2.md +++ b/hadoop-common-project/hadoop-common/src/site/markdown/release/0.18.2/CHANGES.0.18.2.md @@ -27,18 +27,6 @@ | [HADOOP-4228](https://issues.apache.org/jira/browse/HADOOP-4228) | dfs datanode metrics, bytes\_read, bytes\_written overflows due to incorrect type used. | Blocker | metrics | Eric Yang | Eric Yang | -### IMPORTANT ISSUES: - -| JIRA | Summary | Priority | Component | Reporter | Contributor | -|:---- |:---- | :--- |:---- |:---- |:---- | - - -### NEW FEATURES: - -| JIRA | Summary | Priority | Component | Reporter | Contributor | -|:---- |:---- | :--- |:---- |:---- |:---- | - - ### IMPROVEMENTS: | JIRA | Summary | Priority | Component | Reporter | Contributor | @@ -50,27 +38,27 @@ | JIRA | Summary | Priority | Component | Reporter | Contributor | |:---- |:---- | :--- |:---- |:---- |:---- | -| [HADOOP-4533](https://issues.apache.org/jira/browse/HADOOP-4533) | HDFS client of hadoop 0.18.1 and HDFS server 0.18.2 (0.18 branch) not compatible | Blocker | . | Runping Qi | Hairong Kuang | -| [HADOOP-4526](https://issues.apache.org/jira/browse/HADOOP-4526) | fsck failing with NullPointerException (return value 0) | Major | . | Koji Noguchi | Hairong Kuang | -| [HADOOP-4517](https://issues.apache.org/jira/browse/HADOOP-4517) | unstable dfs when running jobs on 0.18.1 | Blocker | . | Christian Kunz | Tsz Wo Nicholas Sze | -| [HADOOP-4483](https://issues.apache.org/jira/browse/HADOOP-4483) | getBlockArray in DatanodeDescriptor does not honor passed in maxblocks value | Critical | . | Ahad Rana | Ahad Rana | -| [HADOOP-4469](https://issues.apache.org/jira/browse/HADOOP-4469) | ant jar file not being included in tar distribution | Blocker | build | Nigel Daley | Nigel Daley | -| [HADOOP-4467](https://issues.apache.org/jira/browse/HADOOP-4467) | SerializationFactory should use current context ClassLoader | Blocker | . | Chris K Wensel | Chris K Wensel | -| [HADOOP-4407](https://issues.apache.org/jira/browse/HADOOP-4407) | HADOOP-4395 should use a Java 1.5 API for 0.18 | Blocker | . | Tsz Wo Nicholas Sze | Tsz Wo Nicholas Sze | -| [HADOOP-4399](https://issues.apache.org/jira/browse/HADOOP-4399) | fuse-dfs per FD context is not thread safe and can cause segfaults and corruptions | Blocker | . | Pete Wyckoff | Pete Wyckoff | -| [HADOOP-4395](https://issues.apache.org/jira/browse/HADOOP-4395) | Reloading FSImage and FSEditLog may erase user and group information | Blocker | . | Tsz Wo Nicholas Sze | Tsz Wo Nicholas Sze | -| [HADOOP-4369](https://issues.apache.org/jira/browse/HADOOP-4369) | Metric Averages are not averages | Blocker | metrics | Brian Bockelman | Brian Bockelman | -| [HADOOP-4351](https://issues.apache.org/jira/browse/HADOOP-4351) | ArrayIndexOutOfBoundsException during fsck | Blocker | . | Brian Bockelman | Hairong Kuang | -| [HADOOP-4340](https://issues.apache.org/jira/browse/HADOOP-4340) | "hadoop jar" always returns exit code 0 (success) to the shell when jar throws a fatal exception | Major | . | David Litster | Arun C Murthy | -| [HADOOP-4326](https://issues.apache.org/jira/browse/HADOOP-4326) | ChecksumFileSystem does not override all create(...) methods | Blocker | fs | Tsz Wo Nicholas Sze | Tsz Wo Nicholas Sze | -| [HADOOP-4314](https://issues.apache.org/jira/browse/HADOOP-4314) | TestReplication fails quite often | Blocker | test | Raghu Angadi | Raghu Angadi | -| [HADOOP-4292](https://issues.apache.org/jira/browse/HADOOP-4292) | append() does not work for LocalFileSystem | Blocker | fs | Raghu Angadi | Hairong Kuang | | [HADOOP-4277](https://issues.apache.org/jira/browse/HADOOP-4277) | Checksum verification is disabled for LocalFS | Blocker | . | Raghu Angadi | Raghu Angadi | | [HADOOP-4271](https://issues.apache.org/jira/browse/HADOOP-4271) | Bug in FSInputChecker makes it possible to read from an invalid buffer | Blocker | fs | Ning Li | Ning Li | -| [HADOOP-3914](https://issues.apache.org/jira/browse/HADOOP-3914) | checksumOk implementation in DFSClient can break applications | Blocker | . | Christian Kunz | Christian Kunz | -| [HADOOP-3786](https://issues.apache.org/jira/browse/HADOOP-3786) | Changes in HOD documentation | Blocker | documentation | Suman Sehgal | Vinod Kumar Vavilapalli | | [HADOOP-3614](https://issues.apache.org/jira/browse/HADOOP-3614) | TestLeaseRecovery fails when run with assertions enabled. | Blocker | . | Konstantin Shvachko | Tsz Wo Nicholas Sze | +| [HADOOP-4314](https://issues.apache.org/jira/browse/HADOOP-4314) | TestReplication fails quite often | Blocker | test | Raghu Angadi | Raghu Angadi | +| [HADOOP-4326](https://issues.apache.org/jira/browse/HADOOP-4326) | ChecksumFileSystem does not override all create(...) methods | Blocker | fs | Tsz Wo Nicholas Sze | Tsz Wo Nicholas Sze | +| [HADOOP-4395](https://issues.apache.org/jira/browse/HADOOP-4395) | Reloading FSImage and FSEditLog may erase user and group information | Blocker | . | Tsz Wo Nicholas Sze | Tsz Wo Nicholas Sze | +| [HADOOP-4351](https://issues.apache.org/jira/browse/HADOOP-4351) | ArrayIndexOutOfBoundsException during fsck | Blocker | . | Brian Bockelman | Hairong Kuang | +| [HADOOP-4407](https://issues.apache.org/jira/browse/HADOOP-4407) | HADOOP-4395 should use a Java 1.5 API for 0.18 | Blocker | . | Tsz Wo Nicholas Sze | Tsz Wo Nicholas Sze | +| [HADOOP-4292](https://issues.apache.org/jira/browse/HADOOP-4292) | append() does not work for LocalFileSystem | Blocker | fs | Raghu Angadi | Hairong Kuang | | [HADOOP-3217](https://issues.apache.org/jira/browse/HADOOP-3217) | [HOD] Be less agressive when querying job status from resource manager. | Blocker | contrib/hod | Hemanth Yamijala | Hemanth Yamijala | +| [HADOOP-3786](https://issues.apache.org/jira/browse/HADOOP-3786) | Changes in HOD documentation | Blocker | documentation | Suman Sehgal | Vinod Kumar Vavilapalli | +| [HADOOP-4399](https://issues.apache.org/jira/browse/HADOOP-4399) | fuse-dfs per FD context is not thread safe and can cause segfaults and corruptions | Blocker | . | Pete Wyckoff | Pete Wyckoff | +| [HADOOP-4369](https://issues.apache.org/jira/browse/HADOOP-4369) | Metric Averages are not averages | Blocker | metrics | Brian Bockelman | Brian Bockelman | +| [HADOOP-4469](https://issues.apache.org/jira/browse/HADOOP-4469) | ant jar file not being included in tar distribution | Blocker | build | Nigel Daley | Nigel Daley | +| [HADOOP-3914](https://issues.apache.org/jira/browse/HADOOP-3914) | checksumOk implementation in DFSClient can break applications | Blocker | . | Christian Kunz | Christian Kunz | +| [HADOOP-4467](https://issues.apache.org/jira/browse/HADOOP-4467) | SerializationFactory should use current context ClassLoader | Blocker | . | Chris K Wensel | Chris K Wensel | +| [HADOOP-4517](https://issues.apache.org/jira/browse/HADOOP-4517) | unstable dfs when running jobs on 0.18.1 | Blocker | . | Christian Kunz | Tsz Wo Nicholas Sze | +| [HADOOP-4526](https://issues.apache.org/jira/browse/HADOOP-4526) | fsck failing with NullPointerException (return value 0) | Major | . | Koji Noguchi | Hairong Kuang | +| [HADOOP-4533](https://issues.apache.org/jira/browse/HADOOP-4533) | HDFS client of hadoop 0.18.1 and HDFS server 0.18.2 (0.18 branch) not compatible | Blocker | . | Runping Qi | Hairong Kuang | +| [HADOOP-4483](https://issues.apache.org/jira/browse/HADOOP-4483) | getBlockArray in DatanodeDescriptor does not honor passed in maxblocks value | Critical | . | Ahad Rana | Ahad Rana | +| [HADOOP-4340](https://issues.apache.org/jira/browse/HADOOP-4340) | "hadoop jar" always returns exit code 0 (success) to the shell when jar throws a fatal exception | Major | . | David Litster | Arun C Murthy | ### TESTS: @@ -80,15 +68,3 @@ | [HADOOP-4403](https://issues.apache.org/jira/browse/HADOOP-4403) | TestLeaseRecovery.testBlockSynchronization failed on trunk | Blocker | test | Hemanth Yamijala | Tsz Wo Nicholas Sze | -### SUB-TASKS: - -| JIRA | Summary | Priority | Component | Reporter | Contributor | -|:---- |:---- | :--- |:---- |:---- |:---- | - - -### OTHER: - -| JIRA | Summary | Priority | Component | Reporter | Contributor | -|:---- |:---- | :--- |:---- |:---- |:---- | - - http://git-wip-us.apache.org/repos/asf/hadoop/blob/19041008/hadoop-common-project/hadoop-common/src/site/markdown/release/0.18.2/RELEASENOTES.0.18.2.md ---------------------------------------------------------------------- diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/release/0.18.2/RELEASENOTES.0.18.2.md b/hadoop-common-project/hadoop-common/src/site/markdown/release/0.18.2/RELEASENOTES.0.18.2.md index e808cd3..e37a3d7 100644 --- a/hadoop-common-project/hadoop-common/src/site/markdown/release/0.18.2/RELEASENOTES.0.18.2.md +++ b/hadoop-common-project/hadoop-common/src/site/markdown/release/0.18.2/RELEASENOTES.0.18.2.md @@ -23,37 +23,37 @@ These release notes cover new developer and user-facing incompatibilities, impor --- -* [HADOOP-4467](https://issues.apache.org/jira/browse/HADOOP-4467) | *Blocker* | **SerializationFactory should use current context ClassLoader** +* [HADOOP-4277](https://issues.apache.org/jira/browse/HADOOP-4277) | *Blocker* | **Checksum verification is disabled for LocalFS** -SerializationFactory now uses the current context ClassLoader allowing for user supplied Serialization instances. +Checksum verification was mistakenly disabled for LocalFileSystem. --- -* [HADOOP-4314](https://issues.apache.org/jira/browse/HADOOP-4314) | *Blocker* | **TestReplication fails quite often** +* [HADOOP-4271](https://issues.apache.org/jira/browse/HADOOP-4271) | *Blocker* | **Bug in FSInputChecker makes it possible to read from an invalid buffer** -Simulated Datanodes should not include blocks that are still being written in their block report. +Checksum input stream can sometimes return invalid data to the user. --- -* [HADOOP-4277](https://issues.apache.org/jira/browse/HADOOP-4277) | *Blocker* | **Checksum verification is disabled for LocalFS** +* [HADOOP-4314](https://issues.apache.org/jira/browse/HADOOP-4314) | *Blocker* | **TestReplication fails quite often** -Checksum verification was mistakenly disabled for LocalFileSystem. +Simulated Datanodes should not include blocks that are still being written in their block report. --- -* [HADOOP-4271](https://issues.apache.org/jira/browse/HADOOP-4271) | *Blocker* | **Bug in FSInputChecker makes it possible to read from an invalid buffer** +* [HADOOP-4228](https://issues.apache.org/jira/browse/HADOOP-4228) | *Blocker* | **dfs datanode metrics, bytes\_read, bytes\_written overflows due to incorrect type used.** -Checksum input stream can sometimes return invalid data to the user. +Changed bytes\_read, bytes\_written to type long to prevent metrics overflow. --- -* [HADOOP-4228](https://issues.apache.org/jira/browse/HADOOP-4228) | *Blocker* | **dfs datanode metrics, bytes\_read, bytes\_written overflows due to incorrect type used.** +* [HADOOP-4467](https://issues.apache.org/jira/browse/HADOOP-4467) | *Blocker* | **SerializationFactory should use current context ClassLoader** -Changed bytes\_read, bytes\_written to type long to prevent metrics overflow. +SerializationFactory now uses the current context ClassLoader allowing for user supplied Serialization instances. http://git-wip-us.apache.org/repos/asf/hadoop/blob/19041008/hadoop-common-project/hadoop-common/src/site/markdown/release/0.18.3/CHANGES.0.18.3.md ---------------------------------------------------------------------- diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/release/0.18.3/CHANGES.0.18.3.md b/hadoop-common-project/hadoop-common/src/site/markdown/release/0.18.3/CHANGES.0.18.3.md index 7c33c5e..62249ef 100644 --- a/hadoop-common-project/hadoop-common/src/site/markdown/release/0.18.3/CHANGES.0.18.3.md +++ b/hadoop-common-project/hadoop-common/src/site/markdown/release/0.18.3/CHANGES.0.18.3.md @@ -24,21 +24,9 @@ | JIRA | Summary | Priority | Component | Reporter | Contributor | |:---- |:---- | :--- |:---- |:---- |:---- | -| [HADOOP-4997](https://issues.apache.org/jira/browse/HADOOP-4997) | workaround for tmp file handling on DataNodes in 0.18 (HADOOP-4663) | Blocker | . | Raghu Angadi | Raghu Angadi | -| [HADOOP-4659](https://issues.apache.org/jira/browse/HADOOP-4659) | Root cause of connection failure is being lost to code that uses it for delaying startup | Blocker | ipc | Steve Loughran | Steve Loughran | | [HADOOP-4061](https://issues.apache.org/jira/browse/HADOOP-4061) | Large number of decommission freezes the Namenode | Major | . | Koji Noguchi | Tsz Wo Nicholas Sze | - - -### IMPORTANT ISSUES: - -| JIRA | Summary | Priority | Component | Reporter | Contributor | -|:---- |:---- | :--- |:---- |:---- |:---- | - - -### NEW FEATURES: - -| JIRA | Summary | Priority | Component | Reporter | Contributor | -|:---- |:---- | :--- |:---- |:---- |:---- | +| [HADOOP-4659](https://issues.apache.org/jira/browse/HADOOP-4659) | Root cause of connection failure is being lost to code that uses it for delaying startup | Blocker | ipc | Steve Loughran | Steve Loughran | +| [HADOOP-4997](https://issues.apache.org/jira/browse/HADOOP-4997) | workaround for tmp file handling on DataNodes in 0.18 (HADOOP-4663) | Blocker | . | Raghu Angadi | Raghu Angadi | ### IMPROVEMENTS: @@ -52,64 +40,58 @@ | JIRA | Summary | Priority | Component | Reporter | Contributor | |:---- |:---- | :--- |:---- |:---- |:---- | -| [HADOOP-5077](https://issues.apache.org/jira/browse/HADOOP-5077) | JavaDoc errors in 0.18.3 | Blocker | util | Raghu Angadi | Raghu Angadi | -| [HADOOP-4983](https://issues.apache.org/jira/browse/HADOOP-4983) | Job counters sometimes go down as tasks run without task failures | Critical | . | Owen O'Malley | Amareshwari Sriramadasu | -| [HADOOP-4971](https://issues.apache.org/jira/browse/HADOOP-4971) | Block report times from datanodes could converge to same time. | Blocker | . | Raghu Angadi | Raghu Angadi | -| [HADOOP-4961](https://issues.apache.org/jira/browse/HADOOP-4961) | ConcurrentModificationException in lease recovery of empty files. | Blocker | . | Konstantin Shvachko | Konstantin Shvachko | -| [HADOOP-4951](https://issues.apache.org/jira/browse/HADOOP-4951) | Lease monitor does not own the LeaseManager lock in changing leases. | Blocker | . | Tsz Wo Nicholas Sze | Tsz Wo Nicholas Sze | -| [HADOOP-4935](https://issues.apache.org/jira/browse/HADOOP-4935) | Manual leaving of safe mode may lead to data lost | Major | . | Hairong Kuang | Konstantin Shvachko | -| [HADOOP-4924](https://issues.apache.org/jira/browse/HADOOP-4924) | Race condition in re-init of TaskTracker | Blocker | . | Devaraj Das | Devaraj Das | -| [HADOOP-4919](https://issues.apache.org/jira/browse/HADOOP-4919) | [HOD] Provide execute access to JT history directory path for group | Major | contrib/hod | Hemanth Yamijala | Peeyush Bishnoi | -| [HADOOP-4910](https://issues.apache.org/jira/browse/HADOOP-4910) | NameNode should exclude corrupt replicas when choosing excessive replicas to delete | Blocker | . | Hairong Kuang | Hairong Kuang | -| [HADOOP-4904](https://issues.apache.org/jira/browse/HADOOP-4904) | Deadlock while leaving safe mode. | Blocker | . | Konstantin Shvachko | Konstantin Shvachko | -| [HADOOP-4840](https://issues.apache.org/jira/browse/HADOOP-4840) | TestNodeCount sometimes fails with NullPointerException | Major | . | Hairong Kuang | Hairong Kuang | -| [HADOOP-4822](https://issues.apache.org/jira/browse/HADOOP-4822) | 0.18 cannot be compiled in Java 5. | Blocker | util | Tsz Wo Nicholas Sze | | -| [HADOOP-4810](https://issues.apache.org/jira/browse/HADOOP-4810) | Data lost at cluster startup time | Blocker | . | Hairong Kuang | Hairong Kuang | -| [HADOOP-4806](https://issues.apache.org/jira/browse/HADOOP-4806) | HDFS rename does not work correctly if src contains Java regular expression special characters | Blocker | . | Tsz Wo Nicholas Sze | Tsz Wo Nicholas Sze | -| [HADOOP-4797](https://issues.apache.org/jira/browse/HADOOP-4797) | RPC Server can leave a lot of direct buffers | Blocker | ipc | Raghu Angadi | Raghu Angadi | -| [HADOOP-4795](https://issues.apache.org/jira/browse/HADOOP-4795) | Lease monitor may get into an infinite loop | Blocker | . | Tsz Wo Nicholas Sze | Tsz Wo Nicholas Sze | -| [HADOOP-4778](https://issues.apache.org/jira/browse/HADOOP-4778) | Check for zero size block meta file when updating a block. | Blocker | . | Tsz Wo Nicholas Sze | Tsz Wo Nicholas Sze | -| [HADOOP-4746](https://issues.apache.org/jira/browse/HADOOP-4746) | Job output directory should be normalized | Blocker | . | Hairong Kuang | Hairong Kuang | -| [HADOOP-4742](https://issues.apache.org/jira/browse/HADOOP-4742) | Mistake delete replica in hadoop 0.18.1 | Blocker | . | Wang Xu | Wang Xu | -| [HADOOP-4734](https://issues.apache.org/jira/browse/HADOOP-4734) | Some lease recovery codes in 0.19 or trunk should also be committed in 0.18. | Blocker | . | Tsz Wo Nicholas Sze | Tsz Wo Nicholas Sze | -| [HADOOP-4726](https://issues.apache.org/jira/browse/HADOOP-4726) | documentation typos: "the the" | Minor | documentation | Tsz Wo Nicholas Sze | Edward J. Yoon | -| [HADOOP-4717](https://issues.apache.org/jira/browse/HADOOP-4717) | Removal of default port# in NameNode.getUri() cause a map/reduce job failed to prompt temporay output | Blocker | . | Hairong Kuang | Hairong Kuang | -| [HADOOP-4714](https://issues.apache.org/jira/browse/HADOOP-4714) | map tasks timing out during merge phase | Major | . | Christian Kunz | Jothi Padmanabhan | -| [HADOOP-4713](https://issues.apache.org/jira/browse/HADOOP-4713) | librecordio does not scale to large records | Blocker | record | Christian Kunz | Christian Kunz | +| [HADOOP-4257](https://issues.apache.org/jira/browse/HADOOP-4257) | TestLeaseRecovery2.testBlockSynchronization failing. | Blocker | test | Vinod Kumar Vavilapalli | Tsz Wo Nicholas Sze | +| [HADOOP-4499](https://issues.apache.org/jira/browse/HADOOP-4499) | DFSClient should invoke checksumOk only once. | Blocker | . | Raghu Angadi | Raghu Angadi | +| [HADOOP-4597](https://issues.apache.org/jira/browse/HADOOP-4597) | Under-replicated blocks are not calculated if the name-node is forced out of safe-mode. | Blocker | . | Konstantin Shvachko | Konstantin Shvachko | +| [HADOOP-4610](https://issues.apache.org/jira/browse/HADOOP-4610) | Always calculate mis-replicated blocks when safe-mode is turned off. | Major | . | Konstantin Shvachko | Konstantin Shvachko | +| [HADOOP-3121](https://issues.apache.org/jira/browse/HADOOP-3121) | dfs -lsr fail with "Could not get listing " | Minor | fs | Koji Noguchi | Tsz Wo Nicholas Sze | +| [HADOOP-3883](https://issues.apache.org/jira/browse/HADOOP-3883) | TestFileCreation fails once in a while | Blocker | test | Lohit Vijayarenu | Tsz Wo Nicholas Sze | +| [HADOOP-4556](https://issues.apache.org/jira/browse/HADOOP-4556) | Block went missing | Major | . | Robert Chansler | Hairong Kuang | +| [HADOOP-4643](https://issues.apache.org/jira/browse/HADOOP-4643) | NameNode should exclude excessive replicas when counting live replicas for a block | Major | . | Hairong Kuang | Hairong Kuang | | [HADOOP-4703](https://issues.apache.org/jira/browse/HADOOP-4703) | DataNode.createInterDataNodeProtocolProxy should not wait for proxy forever while recovering lease | Major | . | Hairong Kuang | Tsz Wo Nicholas Sze | -| [HADOOP-4702](https://issues.apache.org/jira/browse/HADOOP-4702) | Failed block replication leaves an incomplete block in receiver's tmp data directory | Blocker | . | Hairong Kuang | Hairong Kuang | -| [HADOOP-4679](https://issues.apache.org/jira/browse/HADOOP-4679) | Datanode prints tons of log messages: Waiting for threadgroup to exit, active theads is XX | Major | . | Hairong Kuang | Hairong Kuang | -| [HADOOP-4654](https://issues.apache.org/jira/browse/HADOOP-4654) | remove temporary output directory of failed tasks | Major | . | Christian Kunz | Amareshwari Sriramadasu | | [HADOOP-4647](https://issues.apache.org/jira/browse/HADOOP-4647) | NamenodeFsck creates a new DFSClient but never closes it | Major | . | Tsz Wo Nicholas Sze | Tsz Wo Nicholas Sze | -| [HADOOP-4643](https://issues.apache.org/jira/browse/HADOOP-4643) | NameNode should exclude excessive replicas when counting live replicas for a block | Major | . | Hairong Kuang | Hairong Kuang | -| [HADOOP-4635](https://issues.apache.org/jira/browse/HADOOP-4635) | Memory leak ? | Blocker | . | Marc-Olivier Fleury | Pete Wyckoff | -| [HADOOP-4620](https://issues.apache.org/jira/browse/HADOOP-4620) | Streaming mapper never completes if the mapper does not write to stdout | Major | . | Runping Qi | Ravi Gummadi | | [HADOOP-4616](https://issues.apache.org/jira/browse/HADOOP-4616) | assertion makes fuse-dfs exit when reading incomplete data | Blocker | . | Marc-Olivier Fleury | Pete Wyckoff | | [HADOOP-4614](https://issues.apache.org/jira/browse/HADOOP-4614) | "Too many open files" error while processing a large gzip file | Blocker | . | Abdul Qadeer | Yuri Pradkin | -| [HADOOP-4613](https://issues.apache.org/jira/browse/HADOOP-4613) | browseBlock.jsp does not generate "genstamp" property. | Major | . | Konstantin Shvachko | Johan Oskarsson | -| [HADOOP-4610](https://issues.apache.org/jira/browse/HADOOP-4610) | Always calculate mis-replicated blocks when safe-mode is turned off. | Major | . | Konstantin Shvachko | Konstantin Shvachko | -| [HADOOP-4597](https://issues.apache.org/jira/browse/HADOOP-4597) | Under-replicated blocks are not calculated if the name-node is forced out of safe-mode. | Blocker | . | Konstantin Shvachko | Konstantin Shvachko | -| [HADOOP-4556](https://issues.apache.org/jira/browse/HADOOP-4556) | Block went missing | Major | . | Robert Chansler | Hairong Kuang | | [HADOOP-4542](https://issues.apache.org/jira/browse/HADOOP-4542) | Fault in TestDistributedUpgrade | Minor | test | Robert Chansler | Raghu Angadi | -| [HADOOP-4499](https://issues.apache.org/jira/browse/HADOOP-4499) | DFSClient should invoke checksumOk only once. | Blocker | . | Raghu Angadi | Raghu Angadi | -| [HADOOP-4257](https://issues.apache.org/jira/browse/HADOOP-4257) | TestLeaseRecovery2.testBlockSynchronization failing. | Blocker | test | Vinod Kumar Vavilapalli | Tsz Wo Nicholas Sze | -| [HADOOP-3883](https://issues.apache.org/jira/browse/HADOOP-3883) | TestFileCreation fails once in a while | Blocker | test | Lohit Vijayarenu | Tsz Wo Nicholas Sze | -| [HADOOP-3121](https://issues.apache.org/jira/browse/HADOOP-3121) | dfs -lsr fail with "Could not get listing " | Minor | fs | Koji Noguchi | Tsz Wo Nicholas Sze | +| [HADOOP-4713](https://issues.apache.org/jira/browse/HADOOP-4713) | librecordio does not scale to large records | Blocker | record | Christian Kunz | Christian Kunz | +| [HADOOP-4635](https://issues.apache.org/jira/browse/HADOOP-4635) | Memory leak ? | Blocker | . | Marc-Olivier Fleury | Pete Wyckoff | +| [HADOOP-4714](https://issues.apache.org/jira/browse/HADOOP-4714) | map tasks timing out during merge phase | Major | . | Christian Kunz | Jothi Padmanabhan | +| [HADOOP-4726](https://issues.apache.org/jira/browse/HADOOP-4726) | documentation typos: "the the" | Minor | documentation | Tsz Wo Nicholas Sze | Edward J. Yoon | +| [HADOOP-4679](https://issues.apache.org/jira/browse/HADOOP-4679) | Datanode prints tons of log messages: Waiting for threadgroup to exit, active theads is XX | Major | . | Hairong Kuang | Hairong Kuang | +| [HADOOP-4734](https://issues.apache.org/jira/browse/HADOOP-4734) | Some lease recovery codes in 0.19 or trunk should also be committed in 0.18. | Blocker | . | Tsz Wo Nicholas Sze | Tsz Wo Nicholas Sze | +| [HADOOP-4746](https://issues.apache.org/jira/browse/HADOOP-4746) | Job output directory should be normalized | Blocker | . | Hairong Kuang | Hairong Kuang | +| [HADOOP-4654](https://issues.apache.org/jira/browse/HADOOP-4654) | remove temporary output directory of failed tasks | Major | . | Christian Kunz | Amareshwari Sriramadasu | +| [HADOOP-4717](https://issues.apache.org/jira/browse/HADOOP-4717) | Removal of default port# in NameNode.getUri() cause a map/reduce job failed to prompt temporay output | Blocker | . | Hairong Kuang | Hairong Kuang | +| [HADOOP-4778](https://issues.apache.org/jira/browse/HADOOP-4778) | Check for zero size block meta file when updating a block. | Blocker | . | Tsz Wo Nicholas Sze | Tsz Wo Nicholas Sze | +| [HADOOP-4742](https://issues.apache.org/jira/browse/HADOOP-4742) | Mistake delete replica in hadoop 0.18.1 | Blocker | . | Wang Xu | Wang Xu | +| [HADOOP-4702](https://issues.apache.org/jira/browse/HADOOP-4702) | Failed block replication leaves an incomplete block in receiver's tmp data directory | Blocker | . | Hairong Kuang | Hairong Kuang | +| [HADOOP-4613](https://issues.apache.org/jira/browse/HADOOP-4613) | browseBlock.jsp does not generate "genstamp" property. | Major | . | Konstantin Shvachko | Johan Oskarsson | +| [HADOOP-4806](https://issues.apache.org/jira/browse/HADOOP-4806) | HDFS rename does not work correctly if src contains Java regular expression special characters | Blocker | . | Tsz Wo Nicholas Sze | Tsz Wo Nicholas Sze | +| [HADOOP-4795](https://issues.apache.org/jira/browse/HADOOP-4795) | Lease monitor may get into an infinite loop | Blocker | . | Tsz Wo Nicholas Sze | Tsz Wo Nicholas Sze | +| [HADOOP-4822](https://issues.apache.org/jira/browse/HADOOP-4822) | 0.18 cannot be compiled in Java 5. | Blocker | util | Tsz Wo Nicholas Sze | | +| [HADOOP-4620](https://issues.apache.org/jira/browse/HADOOP-4620) | Streaming mapper never completes if the mapper does not write to stdout | Major | . | Runping Qi | Ravi Gummadi | +| [HADOOP-4810](https://issues.apache.org/jira/browse/HADOOP-4810) | Data lost at cluster startup time | Blocker | . | Hairong Kuang | Hairong Kuang | +| [HADOOP-4797](https://issues.apache.org/jira/browse/HADOOP-4797) | RPC Server can leave a lot of direct buffers | Blocker | ipc | Raghu Angadi | Raghu Angadi | +| [HADOOP-4840](https://issues.apache.org/jira/browse/HADOOP-4840) | TestNodeCount sometimes fails with NullPointerException | Major | . | Hairong Kuang | Hairong Kuang | +| [HADOOP-4904](https://issues.apache.org/jira/browse/HADOOP-4904) | Deadlock while leaving safe mode. | Blocker | . | Konstantin Shvachko | Konstantin Shvachko | +| [HADOOP-4919](https://issues.apache.org/jira/browse/HADOOP-4919) | [HOD] Provide execute access to JT history directory path for group | Major | contrib/hod | Hemanth Yamijala | Peeyush Bishnoi | | [HADOOP-1980](https://issues.apache.org/jira/browse/HADOOP-1980) | 'dfsadmin -safemode enter' should prevent the namenode from leaving safemode automatically after startup | Minor | . | Koji Noguchi | Konstantin Shvachko | - - -### TESTS: - -| JIRA | Summary | Priority | Component | Reporter | Contributor | -|:---- |:---- | :--- |:---- |:---- |:---- | +| [HADOOP-4924](https://issues.apache.org/jira/browse/HADOOP-4924) | Race condition in re-init of TaskTracker | Blocker | . | Devaraj Das | Devaraj Das | +| [HADOOP-4935](https://issues.apache.org/jira/browse/HADOOP-4935) | Manual leaving of safe mode may lead to data lost | Major | . | Hairong Kuang | Konstantin Shvachko | +| [HADOOP-4951](https://issues.apache.org/jira/browse/HADOOP-4951) | Lease monitor does not own the LeaseManager lock in changing leases. | Blocker | . | Tsz Wo Nicholas Sze | Tsz Wo Nicholas Sze | +| [HADOOP-4961](https://issues.apache.org/jira/browse/HADOOP-4961) | ConcurrentModificationException in lease recovery of empty files. | Blocker | . | Konstantin Shvachko | Konstantin Shvachko | +| [HADOOP-4971](https://issues.apache.org/jira/browse/HADOOP-4971) | Block report times from datanodes could converge to same time. | Blocker | . | Raghu Angadi | Raghu Angadi | +| [HADOOP-4910](https://issues.apache.org/jira/browse/HADOOP-4910) | NameNode should exclude corrupt replicas when choosing excessive replicas to delete | Blocker | . | Hairong Kuang | Hairong Kuang | +| [HADOOP-5077](https://issues.apache.org/jira/browse/HADOOP-5077) | JavaDoc errors in 0.18.3 | Blocker | util | Raghu Angadi | Raghu Angadi | +| [HADOOP-4983](https://issues.apache.org/jira/browse/HADOOP-4983) | Job counters sometimes go down as tasks run without task failures | Critical | . | Owen O'Malley | Amareshwari Sriramadasu | ### SUB-TASKS: | JIRA | Summary | Priority | Component | Reporter | Contributor | |:---- |:---- | :--- |:---- |:---- |:---- | -| [HADOOP-4824](https://issues.apache.org/jira/browse/HADOOP-4824) | Should not use File.setWritable(..) in 0.18 | Major | . | Tsz Wo Nicholas Sze | Hairong Kuang | | [HADOOP-4823](https://issues.apache.org/jira/browse/HADOOP-4823) | Should not use java.util.NavigableMap in 0.18 | Major | util | Tsz Wo Nicholas Sze | Tsz Wo Nicholas Sze | +| [HADOOP-4824](https://issues.apache.org/jira/browse/HADOOP-4824) | Should not use File.setWritable(..) in 0.18 | Major | . | Tsz Wo Nicholas Sze | Hairong Kuang | ### OTHER: http://git-wip-us.apache.org/repos/asf/hadoop/blob/19041008/hadoop-common-project/hadoop-common/src/site/markdown/release/0.18.3/RELEASENOTES.0.18.3.md ---------------------------------------------------------------------- diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/release/0.18.3/RELEASENOTES.0.18.3.md b/hadoop-common-project/hadoop-common/src/site/markdown/release/0.18.3/RELEASENOTES.0.18.3.md index 4dd9c06..c371afa 100644 --- a/hadoop-common-project/hadoop-common/src/site/markdown/release/0.18.3/RELEASENOTES.0.18.3.md +++ b/hadoop-common-project/hadoop-common/src/site/markdown/release/0.18.3/RELEASENOTES.0.18.3.md @@ -23,82 +23,82 @@ These release notes cover new developer and user-facing incompatibilities, impor --- -* [HADOOP-5077](https://issues.apache.org/jira/browse/HADOOP-5077) | *Blocker* | **JavaDoc errors in 0.18.3** +* [HADOOP-4150](https://issues.apache.org/jira/browse/HADOOP-4150) | *Blocker* | **Include librecordio as part of the release** -Fix couple of JavaDoc warnings. +Included librecordio in release for use by xerces-c (ant -Dlibrecordio=true -Dxercescroot=\<path to the xerces-c root\>) --- -* [HADOOP-4997](https://issues.apache.org/jira/browse/HADOOP-4997) | *Blocker* | **workaround for tmp file handling on DataNodes in 0.18 (HADOOP-4663)** +* [HADOOP-4061](https://issues.apache.org/jira/browse/HADOOP-4061) | *Major* | **Large number of decommission freezes the Namenode** -Revert tmp files handling on DataNodes back to 0.17. sync() introduced in 0.18 has less gaurantees. +Added a new conf property dfs.namenode.decommission.nodes.per.interval so that NameNode checks decommission status of x nodes for every y seconds, where x is the value of dfs.namenode.decommission.nodes.per.interval and y is the value of dfs.namenode.decommission.interval. --- -* [HADOOP-4971](https://issues.apache.org/jira/browse/HADOOP-4971) | *Blocker* | **Block report times from datanodes could converge to same time.** +* [HADOOP-4659](https://issues.apache.org/jira/browse/HADOOP-4659) | *Blocker* | **Root cause of connection failure is being lost to code that uses it for delaying startup** -A long (unexpected) delay at datanodes could make subsequent block reports from many datanode at the same time. +**WARNING: No release note provided for this change.** --- -* [HADOOP-4797](https://issues.apache.org/jira/browse/HADOOP-4797) | *Blocker* | **RPC Server can leave a lot of direct buffers** +* [HADOOP-4542](https://issues.apache.org/jira/browse/HADOOP-4542) | *Minor* | **Fault in TestDistributedUpgrade** -Improve how RPC server reads and writes large buffers. Avoids soft-leak of direct buffers and excess copies in NIO layer. +TestDistributedUpgrade used succeed for wrong reasons. --- -* [HADOOP-4679](https://issues.apache.org/jira/browse/HADOOP-4679) | *Major* | **Datanode prints tons of log messages: Waiting for threadgroup to exit, active theads is XX** +* [HADOOP-4635](https://issues.apache.org/jira/browse/HADOOP-4635) | *Blocker* | **Memory leak ?** -1. Only datanode's offerService thread shutdown the datanode to avoid deadlock; -2. Datanode checks disk in case of failure on creating a block file. +fix memory leak of user/group information in fuse-dfs --- -* [HADOOP-4659](https://issues.apache.org/jira/browse/HADOOP-4659) | *Blocker* | **Root cause of connection failure is being lost to code that uses it for delaying startup** +* [HADOOP-4679](https://issues.apache.org/jira/browse/HADOOP-4679) | *Major* | **Datanode prints tons of log messages: Waiting for threadgroup to exit, active theads is XX** -**WARNING: No release note provided for this incompatible change.** +1. Only datanode's offerService thread shutdown the datanode to avoid deadlock; +2. Datanode checks disk in case of failure on creating a block file. --- -* [HADOOP-4635](https://issues.apache.org/jira/browse/HADOOP-4635) | *Blocker* | **Memory leak ?** +* [HADOOP-4620](https://issues.apache.org/jira/browse/HADOOP-4620) | *Major* | **Streaming mapper never completes if the mapper does not write to stdout** -fix memory leak of user/group information in fuse-dfs +This patch HADOOP-4620.patch +(1) solves the hanging problem on map side with empty input and nonempty output â this map task generates output properly to intermediate files similar to other map tasks. +(2) solves the problem of hanging reducer with empty input to reduce task and nonempty output â this reduce task doesn't generate output if input to reduce task is empty. --- -* [HADOOP-4620](https://issues.apache.org/jira/browse/HADOOP-4620) | *Major* | **Streaming mapper never completes if the mapper does not write to stdout** +* [HADOOP-4797](https://issues.apache.org/jira/browse/HADOOP-4797) | *Blocker* | **RPC Server can leave a lot of direct buffers** -This patch HADOOP-4620.patch -(1) solves the hanging problem on map side with empty input and nonempty output â this map task generates output properly to intermediate files similar to other map tasks. -(2) solves the problem of hanging reducer with empty input to reduce task and nonempty output â this reduce task doesn't generate output if input to reduce task is empty. +Improve how RPC server reads and writes large buffers. Avoids soft-leak of direct buffers and excess copies in NIO layer. --- -* [HADOOP-4542](https://issues.apache.org/jira/browse/HADOOP-4542) | *Minor* | **Fault in TestDistributedUpgrade** +* [HADOOP-4971](https://issues.apache.org/jira/browse/HADOOP-4971) | *Blocker* | **Block report times from datanodes could converge to same time.** -TestDistributedUpgrade used succeed for wrong reasons. +A long (unexpected) delay at datanodes could make subsequent block reports from many datanode at the same time. --- -* [HADOOP-4150](https://issues.apache.org/jira/browse/HADOOP-4150) | *Blocker* | **Include librecordio as part of the release** +* [HADOOP-4997](https://issues.apache.org/jira/browse/HADOOP-4997) | *Blocker* | **workaround for tmp file handling on DataNodes in 0.18 (HADOOP-4663)** -Included librecordio in release for use by xerces-c (ant -Dlibrecordio=true -Dxercescroot=\<path to the xerces-c root\>) +Revert tmp files handling on DataNodes back to 0.17. sync() introduced in 0.18 has less gaurantees. --- -* [HADOOP-4061](https://issues.apache.org/jira/browse/HADOOP-4061) | *Major* | **Large number of decommission freezes the Namenode** +* [HADOOP-5077](https://issues.apache.org/jira/browse/HADOOP-5077) | *Blocker* | **JavaDoc errors in 0.18.3** -Added a new conf property dfs.namenode.decommission.nodes.per.interval so that NameNode checks decommission status of x nodes for every y seconds, where x is the value of dfs.namenode.decommission.nodes.per.interval and y is the value of dfs.namenode.decommission.interval. +Fix couple of JavaDoc warnings. --------------------------------------------------------------------- To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
