Repository: hadoop Updated Branches: refs/heads/branch-2.7 ee830b0d2 -> c05ba079e
Updating all CHANGES.txt files to move entires from future releases into 2.6.1 section given the large number of backports to 2.6.1. (cherry picked from commit fbbb7ff1ed111884d0375c177239073153139848) (cherry picked from commit cbf90f93f9ab3463307292ddfe6fe98ab8b37842) Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/c05ba079 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/c05ba079 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/c05ba079 Branch: refs/heads/branch-2.7 Commit: c05ba079e39509ff61478d6a5a70cc5c49c94707 Parents: ee830b0 Author: Vinod Kumar Vavilapalli (I am also known as @tshooter.) <vino...@apache.org> Authored: Thu Sep 10 13:45:57 2015 -0700 Committer: Vinod Kumar Vavilapalli <vino...@apache.org> Committed: Thu Sep 10 16:10:56 2015 -0700 ---------------------------------------------------------------------- hadoop-common-project/hadoop-common/CHANGES.txt | 112 +++--- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt | 263 +++++++------ hadoop-mapreduce-project/CHANGES.txt | 56 +-- hadoop-yarn-project/CHANGES.txt | 386 +++++++++---------- 4 files changed, 407 insertions(+), 410 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/hadoop/blob/c05ba079/hadoop-common-project/hadoop-common/CHANGES.txt ---------------------------------------------------------------------- diff --git a/hadoop-common-project/hadoop-common/CHANGES.txt b/hadoop-common-project/hadoop-common/CHANGES.txt index 49da893..c5d3d2b 100644 --- a/hadoop-common-project/hadoop-common/CHANGES.txt +++ b/hadoop-common-project/hadoop-common/CHANGES.txt @@ -10,9 +10,6 @@ Release 2.7.2 - UNRELEASED HADOOP-12232. Upgrade Tomcat dependency to 6.0.44. (cnauroth) - HADOOP-11491. HarFs incorrectly declared as requiring an authority. - (Brahma Reddy Battula via gera) - OPTIMIZATIONS BUG FIXES @@ -29,9 +26,6 @@ Release 2.7.2 - UNRELEASED HADOOP-12304. Applications using FileContext fail with the default file system configured to be wasb/s3/etc. (cnauroth) - HADOOP-11932. MetricsSinkAdapter may hang when being stopped. - (Brahma Reddy Battula via jianhe) - HADOOP-12061. Incorrect command in single cluster setup document. (Kengo Seki via aajisaka) @@ -47,9 +41,6 @@ Release 2.7.2 - UNRELEASED HADOOP-12213. Interrupted exception can occur when Client#stop is called. (Kuhu Shukla via ozawa) - HADOOP-8151. Error handling in snappy decompressor throws invalid - exceptions. (Matt Foley via harsh) - Release 2.7.1 - 2015-07-06 INCOMPATIBLE CHANGES @@ -63,9 +54,6 @@ Release 2.7.1 - 2015-07-06 OPTIMIZATIONS - HADOOP-11812. Implement listLocatedStatus for ViewFileSystem to speed up - split calculation (gera) - BUG FIXES HADOOP-11868. Invalid user logins trigger large backtraces in server log @@ -74,12 +62,6 @@ Release 2.7.1 - 2015-07-06 HADOOP-11872. "hadoop dfs" command prints message about using "yarn jar" on Windows(branch-2 only) (Varun Vasudev via cnauroth) - HADOOP-11730. Regression: s3n read failure recovery broken. - (Takenori Sato via stevel) - - HADOOP-11802. DomainSocketWatcher thread terminates sometimes after there - is an I/O error during requestShortCircuitShm (cmccabe) - HADOOP-11891. OsSecureRandom should lazily fill its reservoir (asuresh) HADOOP-9658. SnappyCodec#checkNativeCodeLoaded may unexpectedly fail when @@ -94,9 +76,6 @@ Release 2.7.1 - 2015-07-06 HADOOP-11973. Ensure ZkDelegationTokenSecretManager namespace znodes get created with ACLs. (Gregory Chanan via asuresh) - HADOOP-11934. Use of JavaKeyStoreProvider in LdapGroupsMapping causes - infinite loop. (Larry McCay via cnauroth) - HADOOP-12058. Fix dead links to DistCp and Hadoop Archives pages. (Kazuho Fujii via aajisaka) @@ -352,26 +331,17 @@ Release 2.7.0 - 2015-04-20 HADOOP-11801. Update BUILDING.txt for Ubuntu. (Gabor Liptak via Arpit Agarwal) - HADOOP-12280. Skip unit tests based on maven profile rather than - NativeCodeLoader.isNativeCodeLoaded (Masatake Iwasaki via Colin P. McCabe) - OPTIMIZATIONS HADOOP-11323. WritableComparator#compare keeps reference to byte array. (Wilfred Spiegelenburg via wang) - HADOOP-11238. Update the NameNode's Group Cache in the background when - possible (Chris Li via Colin P. McCabe) - HADOOP-10809. hadoop-azure: page blob support. (Dexter Bradshaw, Mostafa Elhemali, Eric Hanson, and Mike Liddell via cnauroth) HADOOP-11188. hadoop-azure: automatically expand page blobs when they become full. (Eric Hanson via cnauroth) - HADOOP-11506. Configuration variable expansion regex expensive for long - values. (Gera Shegalov via gera) - HADOOP-11231. Remove dead code in ServletUtil. (Li Lu via wheat9) HADOOP-11620. Add support for load balancing across a group of KMS for HA. @@ -486,9 +456,6 @@ Release 2.7.0 - 2015-04-20 HADOOP-11355. When accessing data in HDFS and the key has been deleted, a Null Pointer Exception is shown. (Arun Suresh via wang) - HADOOP-11343. Overflow is not properly handled in caclulating final iv for - AES CTR. (Jerry Chen via wang) - HADOOP-11354. ThrottledInputStream doesn't perform effective throttling. (Ted Yu via jing9) @@ -504,9 +471,6 @@ Release 2.7.0 - 2015-04-20 HADOOP-11369. Fix new findbugs warnings in hadoop-mapreduce-client, non-core directories. (Li Lu via wheat9) - HADOOP-11368. Fix SSLFactory truststore reloader thread leak in - KMSClientProvider. (Arun Suresh via wang) - HADOOP-11372. Fix new findbugs warnings in mapreduce-examples. (Li Lu via wheat9) @@ -604,9 +568,6 @@ Release 2.7.0 - 2015-04-20 HADOOP-11462. TestSocketIOWithTimeout needs change for PowerPC platform. (Ayappan via cnauroth) - HADOOP-11350. The size of header buffer of HttpServer is too small when - HTTPS is enabled. (Benoy Antony via wheat9) - HADOOP-10542 Potential null pointer dereference in Jets3tFileSystemStore retrieveBlock(). (Ted Yu via stevel) @@ -631,9 +592,6 @@ Release 2.7.0 - 2015-04-20 HADOOP-11507 Hadoop RPC Authentication problem with different user locale. (Talat UYARER via stevel) - HADOOP-11482. Use correct UGI when KMSClientProvider is called by a proxy - user. Contributed by Arun Suresh. - HADOOP-11499. Check of executorThreadsStarted in ValueQueue#submitRefillTask() evades lock acquisition (Ted Yu via jlowe) @@ -705,9 +663,6 @@ Release 2.7.0 - 2015-04-20 HADOOP-11570. S3AInputStream.close() downloads the remaining bytes of the object from S3. (Dan Hecht via stevel). - HADOOP-11295. RPC Server Reader thread can't shutdown if RPCCallQueue is - full. (Ming Ma via kihwal) - HADOOP-11599. Client#getTimeout should use IPC_CLIENT_PING_DEFAULT when IPC_CLIENT_PING_KEY is not configured. (zhihai xu via ozawa) @@ -720,9 +675,6 @@ Release 2.7.0 - 2015-04-20 HADOOP-9087. Queue size metric for metric sinks isn't actually maintained (Akira AJISAKA via jlowe) - HADOOP-11604. Prevent ConcurrentModificationException while closing domain - sockets during shutdown of DomainSocketWatcher thread. (cnauroth) - HADOOP-11612. Workaround for Curator's ChildReaper requiring Guava 15+. (rkanter) @@ -759,9 +711,6 @@ Release 2.7.0 - 2015-04-20 HADOOP-11666. Revert the format change of du output introduced by HADOOP-6857. (Byron Wong via aajisaka) - HADOOP-11674. oneByteBuf in CryptoInputStream and CryptoOutputStream - should be non static. (Sean Busbey via yliu) - HADOOP-11670. Regression: s3a auth setup broken. (Adam Budde via stevel) HADOOP-11602. Fix toUpperCase/toLowerCase to use Locale.ENGLISH. (ozawa) @@ -775,9 +724,6 @@ Release 2.7.0 - 2015-04-20 HADOOP-11693. Azure Storage FileSystem rename operations are throttled too aggressively to complete HBase WAL archiving. (Duo Xu via cnauroth) - HADOOP-11710. Make CryptoOutputStream behave like DFSOutputStream wrt - synchronization. (Sean Busbey via yliu) - HADOOP-11558. Fix dead links to doc of hadoop-tools. (Jean-Pierre Matsumoto via ozawa) @@ -853,7 +799,7 @@ Release 2.6.2 - UNRELEASED BUG FIXES -Release 2.6.1 - UNRELEASED +Release 2.6.1 - 2015-09-09 INCOMPATIBLE CHANGES @@ -864,11 +810,20 @@ Release 2.6.1 - UNRELEASED HADOOP-7139. Allow appending to existing SequenceFiles (kanaka kumar avvaru via vinayakumarb) + HADOOP-12280. Skip unit tests based on maven profile rather than + NativeCodeLoader.isNativeCodeLoaded (Masatake Iwasaki via Colin P. McCabe) + OPTIMIZATIONS + HADOOP-11238. Update the NameNode's Group Cache in the background when + possible (Chris Li via Colin P. McCabe) + + HADOOP-11506. Configuration variable expansion regex expensive for long + values. (Gera Shegalov via gera) + BUG FIXES - HADOOP-11466. FastByteComparisons: do not use UNSAFE_COMPARER on the SPARC + HADOOP-11466: FastByteComparisons: do not use UNSAFE_COMPARER on the SPARC architecture because it is slower there (Suman Somasundar via Colin P. McCabe) @@ -877,6 +832,51 @@ Release 2.6.1 - UNRELEASED HADOOP-11333. Fix deadlock in DomainSocketWatcher when the notification pipe is full (zhaoyunjiong via cmccabe) + HADOOP-11343. Overflow is not properly handled in caclulating final iv for + AES CTR. (Jerry Chen via wang) + + HADOOP-11368. Fix SSLFactory truststore reloader thread leak in + KMSClientProvider. (Arun Suresh via wang) + + HADOOP-11350. The size of header buffer of HttpServer is too small when + HTTPS is enabled. (Benoy Antony via wheat9) + + HADOOP-11482. Use correct UGI when KMSClientProvider is called by a proxy + user. Contributed by Arun Suresh. + + HADOOP-11295. RPC Server Reader thread can't shutdown if RPCCallQueue is + full. (Ming Ma via kihwal) + + HADOOP-11604. Prevent ConcurrentModificationException while closing domain + sockets during shutdown of DomainSocketWatcher thread. (cnauroth) + + HADOOP-11674. oneByteBuf in CryptoInputStream and CryptoOutputStream + should be non static. (Sean Busbey via yliu) + + HADOOP-11710. Make CryptoOutputStream behave like DFSOutputStream wrt + synchronization. (Sean Busbey via yliu) + + HADOOP-11812. Implement listLocatedStatus for ViewFileSystem to speed up + split calculation (gera) + + HADOOP-11730. Regression: s3n read failure recovery broken. + (Takenori Sato via stevel) + + HADOOP-11802: DomainSocketWatcher thread terminates sometimes after there + is an I/O error during requestShortCircuitShm (cmccabe) + + HADOOP-11491. HarFs incorrectly declared as requiring an authority. + (Brahma Reddy Battula via gera) + + HADOOP-8151. Error handling in snappy decompressor throws invalid + exceptions. (Matt Foley via harsh) + + HADOOP-11932. MetricsSinkAdapter may hang when being stopped. + (Brahma Reddy Battula via jianhe) + + HADOOP-11934. Use of JavaKeyStoreProvider in LdapGroupsMapping causes + infinite loop. (Larry McCay via cnauroth) + Release 2.6.0 - 2014-11-18 INCOMPATIBLE CHANGES http://git-wip-us.apache.org/repos/asf/hadoop/blob/c05ba079/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt ---------------------------------------------------------------------- diff --git a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt index e703aa9..4b7a030 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt +++ b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt @@ -16,9 +16,6 @@ Release 2.7.2 - UNRELEASED HDFS-7314. When the DFSClient lease cannot be renewed, abort open-for-write files rather than the entire DFSClient. (mingma) - HDFS-8846. Add a unit test for INotify functionality across a layout - version upgrade (Zhe Zhang via Colin P. McCabe) - OPTIMIZATIONS HDFS-8722. Optimize datanode writes for small writes and flushes (kihwal) @@ -36,9 +33,6 @@ Release 2.7.2 - UNRELEASED HDFS-8867. Enable optimized block reports. (Daryn Sharp via jing9) - HDFS-8863. The remaining space check in BlockPlacementPolicyDefault is - flawed. (Kihwal Lee via yliu) - HDFS-8891. HDFS concat should keep srcs order. (Yong Zhang via jing9) HDFS-8879. Quota by storage type usage incorrectly initialized upon namenode @@ -47,21 +41,9 @@ Release 2.7.2 - UNRELEASED HDFS-8995. Flaw in registration bookeeping can make DN die on reconnect. (Kihwal Lee via yliu) - HDFS-8384. Allow NN to startup if there are files having a lease but are not - under construction. (jing9) - HDFS-9033. dfsadmin -metasave prints "NaN" for cache used%. (Brahma Reddy Battula via aajisaka) - HDFS-8219. setStoragePolicy with folder behavior is different after cluster restart. - (surendra singh lilhore via Xiaoyu Yao) - - HDFS-8431. hdfs crypto class not found in Windows. - (Anu Engineer via cnauroth) - - HDFS-7609. Avoid retry cache collision when Standby NameNode loading edits. - (Ming Ma via jing9) - Release 2.7.1 - 2015-07-06 INCOMPATIBLE CHANGES @@ -92,22 +74,14 @@ Release 2.7.1 - 2015-07-06 HDFS-7546. Document, and set an accepting default for dfs.namenode.kerberos.principal.pattern (Harsh J via aw) - HDFS-8046. Allow better control of getContentSummary (kihwal) - OPTIMIZATIONS HDFS-7164. Feature documentation for HDFS-6581. (Arpit Agarwal) OPTIMIZATIONS - HDFS-8480. Fix performance and timeout issues in HDFS-7929 by using - hard-links to preserve old edit logs, instead of copying them. (Zhe Zhang - via Colin P. McCabe) BUG FIXES - HDFS-8127. NameNode Failover during HA upgrade can cause DataNode to - finalize upgrade. (jing9) - HDFS-8151. Always use snapshot path as source when invalid snapshot names are used for diff based distcp. (jing9) @@ -129,9 +103,6 @@ Release 2.7.1 - 2015-07-06 HDFS-8147. StorageGroup in Dispatcher should override equals nad hashCode. (surendra singh lilhore via szetszwo) - HDFS-8070. Pre-HDFS-7915 DFSClient cannot use short circuit on - post-HDFS-7915 DataNode (cmccabe) - HDFS-8273. FSNamesystem#Delete() should not call logSync() when holding the lock. (wheat9) @@ -144,17 +115,8 @@ Release 2.7.1 - 2015-07-06 HDFS-8305: HDFS INotify: the destination field of RenameOp should always end with the file name (cmccabe) - HDFS-7980. Incremental BlockReport will dramatically slow down namenode - startup. (Walter Su via szetszwo) - HDFS-8226. Non-HA rollback compatibility broken (J.Andreina via vinayakumarb) - HDFS-7894. Rolling upgrade readiness is not updated in jmx until query - command is issued. (Brahma Reddy Battula via kihwal) - - HDFS-8254. Standby namenode doesn't process DELETED_BLOCK if the add block - request is in edit log. (Rushabh S Shah via kihwal) - HDFS-7916. 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop (Rushabh S Shah via kihwal) @@ -163,9 +125,6 @@ Release 2.7.1 - 2015-07-06 HDFS-8405. Fix a typo in NamenodeFsck. (Takanobu Asanuma via szetszwo) - HDFS-8404. Pending block replication can get stuck using older genstamp - (Nathan Roberts via kihwal) - HDFS-8451. DFSClient probe for encryption testing interprets empty URI property for "enabled". (Steve Loughran via xyao) @@ -220,9 +179,6 @@ Release 2.7.0 - 2015-04-20 NEW FEATURES - HDFS-7278. Add a command that allows sysadmins to manually trigger full - block reports from a DN (cmccabe) - HDFS-6663. Admin command to track file and locations from block id. (Chen He via kihwal) @@ -316,9 +272,6 @@ Release 2.7.0 - 2015-04-20 HDFS-3342. SocketTimeoutException in BlockSender.sendChunks could have a better error message. (Yongjun Zhang via wang) - HDFS-7035. Make adding a new data directory to the DataNode an atomic - operation and improve error handling (Lei Xu via Colin P. McCabe) - HDFS-6917. Add an hdfs debug command to validate blocks, call recoverlease, etc. (cmccabe) @@ -404,9 +357,6 @@ Release 2.7.0 - 2015-04-20 HDFS-7462. Consolidate implementation of mkdirs() into a single class. (wheat9) - HDFS-7446. HDFS inotify should have the ability to determine what txid it - has read up to (cmccabe) - HDFS-6735. A minor optimization to avoid pread() be blocked by read() inside the same DFSInputStream (Lars Hofhansl via stack) @@ -449,9 +399,6 @@ Release 2.7.0 - 2015-04-20 HDFS-7528. Consolidate symlink-related implementation into a single class. (wheat9) - HDFS-7531. Improve the concurrent access on FsVolumeList (Lei Xu via Colin - P. McCabe) - HDFS-7373. Clean up temporary files after fsimage transfer failures. (kihwal) @@ -469,8 +416,6 @@ Release 2.7.0 - 2015-04-20 HADOOP-11470. Remove some uses of obsolete guava APIs from the hadoop codebase. (Sangjin Lee via Colin P. McCabe) - HDFS-7182. JMX metrics aren't accessible when NN is busy. (Ming Ma via jing9) - HDFS-7323. Move the get/setStoragePolicy commands out from dfsadmin. (jing9 via yliu) @@ -670,9 +615,6 @@ Release 2.7.0 - 2015-04-20 HDFS-7410. Support CreateFlags with append() to support hsync() for appending streams (Vinayakumar B via Colin P. McCabe) - HDFS-7742. Favoring decommissioning node for replication can cause a block - to stay underreplicated for long periods (Nathan Roberts via kihwal) - HDFS-7811. Avoid recursive call getStoragePolicyID in INodeFile#computeQuotaUsage. (Xiaoyu Yao and jing9) @@ -817,9 +759,6 @@ Release 2.7.0 - 2015-04-20 HDFS-7431. log message for InvalidMagicNumberException may be incorrect. (Yi Liu via cnauroth) - HDFS-7552. Change FsVolumeList toString() to fix - TestDataNodeVolumeFailureToleration (Liang Xie via Colin P. McCabe) - HDFS-7557. Fix spacing for a few keys in DFSConfigKeys.java (Colin P.McCabe) @@ -846,21 +785,9 @@ Release 2.7.0 - 2015-04-20 HDFS-7589. Break the dependency between libnative_mini_dfs and libhdfs. (Zhanwei Wang via cnauroth) - HDFS-7579. Improve log reporting during block report rpc failure. - (Charles Lamb via cnauroth) - - HDFS-7596. NameNode should prune dead storages from storageMap. - (Arpit Agarwal via cnauroth) - - HDFS-7533. Datanode sometimes does not shutdown on receiving upgrade - shutdown command (Eric Payne via kihwal) - HDFS-5445. PacketReceiver populates the packetLen field in PacketHeader incorrectly (Jonathan Mace via Colin P. McCabe) - HDFS-7470. SecondaryNameNode need twice memory when calling - reloadFromImageFile. (zhaoyunjiong via cnauroth) - HDFS-7585. Get TestEnhancedByteBufferAccess working on CPU architectures with page sizes other than 4096 (Sam Liu via Colin P. McCabe) @@ -878,15 +805,9 @@ Release 2.7.0 - 2015-04-20 HDFS-7496. Fix FsVolume removal race conditions on the DataNode by reference-counting the volume instances (lei via cmccabe) - HDFS-7610. Fix removal of dynamically added DN volumes (Lei (Eddy) Xu via - Colin P. McCabe) - HDFS-7548. Corrupt block reporting delayed until datablock scanner thread detects it (Rushabh Shah via kihwal) - HDFS-7575. Upgrade should generate a unique storage ID for each - volume. (Arpit Agarwal) - HDFS-3519. Checkpoint upload may interfere with a concurrent saveNamespace. (Ming Ma via cnauroth) @@ -927,9 +848,6 @@ Release 2.7.0 - 2015-04-20 HDFS-6651. Deletion failure can leak inodes permanently. (Jing Zhao via wheat9) - HDFS-7707. Edit log corruption due to delayed block removal again. - (Yongjun Zhang via kihwal) - HDFS-7734. Class cast exception in NameNode#main. (yliu via wang) HDFS-7719. BlockPoolSliceStorage#removeVolumes fails to remove some @@ -956,9 +874,6 @@ Release 2.7.0 - 2015-04-20 HDFS-7718. Store KeyProvider in ClientContext to avoid leaking key provider threads when using FileContext (Arun Suresh via Colin P. McCabe) - HDFS-7714. Simultaneous restart of HA NameNodes and DataNode can cause - DataNode to register successfully with only one NameNode.(vinayakumarb) - HDFS-7769. TestHDFSCLI should not create files in hdfs project root dir. (szetszwo) @@ -995,15 +910,9 @@ Release 2.7.0 - 2015-04-20 HDFS-6662. WebHDFS cannot open a file if its path contains "%". (Gerson Carlos via wheat9) - HDFS-7788. Post-2.6 namenode may not start up with an image containing - inodes created with an old release. (Rushabh Shah via kihwal) - HDFS-7814. Fix usage string of storageType parameter for "dfsadmin -setSpaceQuota/clrSpaceQuota". (Xiaoyu Yao via cnauroth) - HDFS-7009. Active NN and standby NN have different live nodes. - (Ming Ma via cnauroth) - HDFS-7807. libhdfs htable.c: fix htable resizing, add unit test (cmccabe) HDFS-7805. NameNode recovery prompt should be printed on console (Surendra @@ -1015,9 +924,6 @@ Release 2.7.0 - 2015-04-20 HDFS-7831. Fix the starting index and end condition of the loop in FileDiffList.findEarlierSnapshotBlocks(). (Konstantin Shvachko via jing9) - HDFS-7763. fix zkfc hung issue due to not catching exception in a corner - case. (Liang Xie via wang) - HDFS-7843. A truncated file is corrupted after rollback from a rolling upgrade. (szetszwo) @@ -1030,9 +936,6 @@ Release 2.7.0 - 2015-04-20 HDFS-7785. Improve diagnostics information for HttpPutFailedException. (Chengbing Liu via wheat9) - HDFS-7871. NameNodeEditLogRoller can keep printing "Swallowing exception" - message. (jing9) - HDFS-7757. Misleading error messages in FSImage.java. (Brahma Reddy Battula via Arpit Agarwal) @@ -1050,15 +953,9 @@ Release 2.7.0 - 2015-04-20 HDFS-7434. DatanodeID hashCode should not be mutable. (daryn via kihwal) - HDFS-7885. Datanode should not trust the generation stamp provided by - client. (Tsz Wo Nicholas Sze via jing9) - HDFS-7818. OffsetParam should return the default value instead of throwing NPE when the value is unspecified. (Eric Payne via wheat9) - HDFS-7830. DataNode does not release the volume lock when adding a volume - fails. (Lei Xu via Colin P. Mccabe) - HDFS-7880. Remove the tests for legacy Web UI in branch-2. (Brahma Reddy Battula via aajisaka) @@ -1073,9 +970,6 @@ Release 2.7.0 - 2015-04-20 HDFS-7886. Fix TestFileTruncate falures. (Plamen Jeliazkov and shv) - HDFS-7915. The DataNode can sometimes allocate a ShortCircuitShm slot and - fail to tell the DFSClient about it because of a network error (cmccabe) - HDFS-7946. TestDataNodeVolumeFailureReporting NPE on Windows. (Xiaoyu Yao via Arpit Agarwal) @@ -1107,18 +1001,10 @@ Release 2.7.0 - 2015-04-20 HDFS-7943. Append cannot handle the last block with length greater than the preferred block size. (jing9) - HDFS-7587. Edit log corruption can happen if append fails with a quota - violation. (jing9) - - HDFS-7929. inotify unable fetch pre-upgrade edit log segments once upgrade - starts (Zhe Zhang via Colin P. McCabe) - HDFS-7816. Unable to open webhdfs paths with "+". (wheat9 via kihwal) HDFS-7932. Speed up the shutdown of datanode during rolling upgrade.(kihwal) - HDFS-7930. commitBlockSynchronization() does not remove locations. (yliu) - HDFS-7957. Truncate should verify quota before making changes. (jing9) HDFS-6841. Use Time.monotonicNow() wherever applicable instead of Time.now() @@ -1126,16 +1012,6 @@ Release 2.7.0 - 2015-04-20 HDFS-7942. NFS: support regexp grouping in nfs.exports.allowed.hosts (brandonli) - HDFS-7881. TestHftpFileSystem#testSeek fails in branch-2. - (Brahma Reddy Battula via aajisaka) - - HDFS-7884. Fix NullPointerException in BlockSender when the generation stamp - provided by the client is larger than the one stored in the datanode. - (Brahma Reddy Battula via szetszwo) - - HDFS-7960. The full block report should prune zombie storages even if - they're not empty. (cmccabe and Eddy Xu via wang) - HDFS-7961. Trigger full block report after hot swapping disk. (Eddy Xu via wang) HDFS-7977. NFS couldn't take percentile intervals (brandonli) @@ -1164,15 +1040,9 @@ Release 2.7.0 - 2015-04-20 HDFS-8051. FsVolumeList#addVolume should release volume reference if not put it into BlockScanner. (Lei (Eddy) Xu via Colin P. McCabe) - HDFS-7999. FsDatasetImpl#createTemporary sometimes holds the FSDatasetImpl - lock for a very long time (sinago via cmccabe) - HDFS-8038. PBImageDelimitedTextWriter#getEntry output HDFS path in platform-specific format. (Xiaoyu Yao via cnauroth) - HDFS-8072. Reserved RBW space is not released if client terminates while - writing block. (Arpit Agarwal) - HDFS-8025. Addendum fix for HDFS-3087 Decomissioning on NN restart can complete without blocks being replicated. (Ming Ma via wang) @@ -1214,16 +1084,42 @@ Release 2.6.2 - UNRELEASED BUG FIXES -Release 2.6.1 - UNRELEASED +Release 2.6.1 - 2015-09-09 INCOMPATIBLE CHANGES NEW FEATURES + HDFS-7278. Add a command that allows sysadmins to manually trigger full + block reports from a DN (cmccabe) + IMPROVEMENTS + HDFS-7035. Make adding a new data directory to the DataNode an atomic + operation and improve error handling (Lei Xu via Colin P. McCabe) + + HDFS-7531. Improve the concurrent access on FsVolumeList (Lei Xu via Colin + P. McCabe) + + HDFS-7579. Improve log reporting during block report rpc failure. + (Charles Lamb via cnauroth) + + HDFS-7182. JMX metrics aren't accessible when NN is busy. (Ming Ma via jing9) + + HDFS-7596. NameNode should prune dead storages from storageMap. + (Arpit Agarwal via cnauroth) + + HDFS-8046. Allow better control of getContentSummary (kihwal) + + HDFS-8384. Allow NN to startup if there are files having a lease but are not + under construction. (jing9) + OPTIMIZATIONS + HDFS-8480. Fix performance and timeout issues in HDFS-7929 by using + hard-links to preserve old edit logs, instead of copying them. (Zhe Zhang + via Colin P. McCabe) + BUG FIXES HDFS-7425. NameNode block deletion logging uses incorrect appender. @@ -1247,8 +1143,7 @@ Release 2.6.1 - UNRELEASED HDFS-7733. NFS: readdir/readdirplus return null directory attribute on failure. (Arpit Agarwal) - HDFS-8486. DN startup may cause severe data loss (Daryn Sharp via Colin P. - McCabe) + HDFS-8486. DN startup may cause severe data loss. (daryn via cmccabe) HDFS-7213. processIncrementalBlockReport performance degradation. (Eric Payne via kihwal) @@ -1262,9 +1157,111 @@ Release 2.6.1 - UNRELEASED HDFS-7225. Remove stale block invalidation work when DN re-registers with different UUID. (Zhe Zhang and Andrew Wang) + HDFS-7533. Datanode sometimes does not shutdown on receiving upgrade + shutdown command (Eric Payne via kihwal) + + HDFS-7575. Upgrade should generate a unique storage ID for each + volume. (Arpit Agarwal) + + HDFS-7707. Edit log corruption due to delayed block removal again. + (Yongjun Zhang via kihwal) + + HDFS-7714. Simultaneous restart of HA NameNodes and DataNode can cause + DataNode to register successfully with only one NameNode.(vinayakumarb) + + HDFS-7788. Post-2.6 namenode may not start up with an image containing + inodes created with an old release. (Rushabh Shah via kihwal) + + HDFS-7009. Active NN and standby NN have different live nodes. + (Ming Ma via cnauroth) + + HDFS-7763. fix zkfc hung issue due to not catching exception in a corner + case. (Liang Xie via wang) + + HDFS-7871. NameNodeEditLogRoller can keep printing "Swallowing exception" + message. (jing9) + + HDFS-7885. Datanode should not trust the generation stamp provided by + client. (Tsz Wo Nicholas Sze via jing9) + + HDFS-7610. Fix removal of dynamically added DN volumes (Lei (Eddy) Xu via + Colin P. McCabe) + + HDFS-7830. DataNode does not release the volume lock when adding a volume + fails. (Lei Xu via Colin P. Mccabe) + + HDFS-7587. Edit log corruption can happen if append fails with a quota + violation. (jing9) + + HDFS-7929. inotify unable fetch pre-upgrade edit log segments once upgrade + starts (Zhe Zhang via Colin P. McCabe) + + HDFS-7930. commitBlockSynchronization() does not remove locations. (yliu) + + HDFS-7884. Fix NullPointerException in BlockSender when the generation stamp + provided by the client is larger than the one stored in the datanode. + (Brahma Reddy Battula via szetszwo) + + HDFS-7960. The full block report should prune zombie storages even if + they're not empty. (cmccabe and Eddy Xu via wang) + + HDFS-7742. Favoring decommissioning node for replication can cause a block + to stay underreplicated for long periods (Nathan Roberts via kihwal) + + HDFS-7999. FsDatasetImpl#createTemporary sometimes holds the FSDatasetImpl + lock for a very long time (sinago via cmccabe) + + HDFS-8072. Reserved RBW space is not released if client terminates while + writing block. (Arpit Agarwal) + + HDFS-8127. NameNode Failover during HA upgrade can cause DataNode to + finalize upgrade. (jing9) + + HDFS-7915. The DataNode can sometimes allocate a ShortCircuitShm slot and + fail to tell the DFSClient about it because of a network error (cmccabe) + + HDFS-8070. Pre-HDFS-7915 DFSClient cannot use short circuit on + post-HDFS-7915 DataNode (cmccabe) + + HDFS-8219. setStoragePolicy with folder behavior is different after cluster restart. + (surendra singh lilhore via Xiaoyu Yao) + + HDFS-7894. Rolling upgrade readiness is not updated in jmx until query + command is issued. (Brahma Reddy Battula via kihwal) + + HDFS-8254. Standby namenode doesn't process DELETED_BLOCK if the add block + request is in edit log. (Rushabh S Shah via kihwal) + + HDFS-8404. Pending block replication can get stuck using older genstamp + (Nathan Roberts via kihwal) + + HDFS-8431. hdfs crypto class not found in Windows. + (Anu Engineer via cnauroth) + + HDFS-7609. Avoid retry cache collision when Standby NameNode loading edits. + (Ming Ma via jing9) + HDFS-8270. create() always retried with hardcoded timeout when file already exists with open lease (J.Andreina via vinayakumarb) + HDFS-7980. Incremental BlockReport will dramatically slow down namenode + startup. (Walter Su via szetszwo) + + HDFS-7446. HDFS inotify should have the ability to determine what txid it + has read up to (cmccabe) + + HDFS-8846. Add a unit test for INotify functionality across a layout + version upgrade (Zhe Zhang via Colin P. McCabe) + + HDFS-7470. SecondaryNameNode need twice memory when calling + reloadFromImageFile. (zhaoyunjiong via cnauroth) + + HDFS-8863. The remaining space check in BlockPlacementPolicyDefault is + flawed. (Kihwal Lee via yliu) + + HDFS-7552. Change FsVolumeList toString() to fix + TestDataNodeVolumeFailureToleration (Liang Xie via Colin P. McCabe) + Release 2.6.0 - 2014-11-18 INCOMPATIBLE CHANGES http://git-wip-us.apache.org/repos/asf/hadoop/blob/c05ba079/hadoop-mapreduce-project/CHANGES.txt ---------------------------------------------------------------------- diff --git a/hadoop-mapreduce-project/CHANGES.txt b/hadoop-mapreduce-project/CHANGES.txt index 63b6129..8af159b 100644 --- a/hadoop-mapreduce-project/CHANGES.txt +++ b/hadoop-mapreduce-project/CHANGES.txt @@ -27,9 +27,6 @@ Release 2.7.2 - UNRELEASED MAPREDUCE-6474. ShuffleHandler can possibly exhaust nodemanager file descriptors (Kuhu Shukla via jlowe) - MAPREDUCE-5649. Reduce cannot use more than 2G memory for the final merge - (Gera Shegalov via jlowe) - Release 2.7.1 - 2015-07-06 INCOMPATIBLE CHANGES @@ -42,14 +39,6 @@ Release 2.7.1 - 2015-07-06 BUG FIXES - MAPREDUCE-6300. Task list sort by task id broken. (Siqi Li via aajisaka) - - MAPREDUCE-6238. MR2 can't run local jobs with -libjars command options - which is a regression from MR1 (zxu via rkanter) - - MAPREDUCE-6324. Fixed MapReduce uber jobs to not fail the udpate of AM-RM - tokens when they roll-over. (Jason Lowe via vinodkv) - MAPREDUCE-6252. JobHistoryServer should not fail when encountering a missing directory. (Craig Welch via devaraj) @@ -66,10 +55,6 @@ Release 2.7.1 - 2015-07-06 that they don't fail on history-server backed by DFSes with not so strong guarantees. (Craig Welch via vinodkv) - MAPREDUCE-6361. NPE issue in shuffle caused by concurrent issue between - copySucceeded() in one thread and copyFailed() in another thread on the - same host. (Junping Du via ozawa) - MAPREDUCE-6387. Serialize the recently added Task#encryptedSpillKey field at the end. (Arun Suresh via kasha) @@ -151,9 +136,6 @@ Release 2.7.0 - 2015-04-20 MAPREDUCE-6248. Exposed the internal MapReduce job's information as a public API in DistCp. (Jing Zhao via vinodkv) - MAPREDUCE-6267. Refactor JobSubmitter#copyAndConfigureFiles into it's own - class. (Chris Trezzo via kasha) - MAPREDUCE-6263. Configurable timeout between YARNRunner terminate the application and forcefully kill. (Eric Payne via junping_du) @@ -203,9 +185,6 @@ Release 2.7.0 - 2015-04-20 MAPREDUCE-4879. TeraOutputFormat may overwrite an existing output directory. (gera) - MAPREDUCE-6166. Reducers do not validate checksum of map outputs when - fetching directly to disk. (Eric Payne via gera) - MAPREDUCE-6045. need close the DataInputStream after open it in TestMapReduce.java (zxu via rkanter) @@ -215,9 +194,6 @@ Release 2.7.0 - 2015-04-20 MAPREDUCE-3283. mapred classpath CLI does not display the complete classpath (Varun Saxena via cnauroth) - MAPREDUCE-6230. Fixed RMContainerAllocator to update the new AMRMToken - service name properly. (Jason Lowe via jianhe) - MAPREDUCE-6231. Grep example job is not working on a fully-distributed cluster. (aajisaka) @@ -265,9 +241,6 @@ Release 2.7.0 - 2015-04-20 MAPREDUCE-6285. ClientServiceDelegate should not retry upon AuthenticationException. (Jonathan Eagles via ozawa) - MAPREDUCE-6303. Read timeout when retrying a fetch error can be fatal - to a reducer. (Jason Lowe via junping_du) - Release 2.6.2 - UNRELEASED INCOMPATIBLE CHANGES @@ -284,7 +257,7 @@ Release 2.6.2 - UNRELEASED cache files so that child processes running hadoop scripts can access these files. (Junping Du via vinodkv) -Release 2.6.1 - UNRELEASED +Release 2.6.1 - 2015-09-09 INCOMPATIBLE CHANGES @@ -299,6 +272,33 @@ Release 2.6.1 - UNRELEASED MAPREDUCE-6237. Multiple mappers with DBInputFormat don't work because of reusing conections. (Kannan Rajah via ozawa) + MAPREDUCE-6166. Reducers do not validate checksum of map outputs when + fetching directly to disk. (Eric Payne via gera) + + MAPREDUCE-6230. Fixed RMContainerAllocator to update the new AMRMToken + service name properly. (Jason Lowe via jianhe) + + MAPREDUCE-6303. Read timeout when retrying a fetch error can be fatal + to a reducer. (Jason Lowe via junping_du) + + MAPREDUCE-6300. Task list sort by task id broken. (Siqi Li via aajisaka) + + MAPREDUCE-6267. Refactor JobSubmitter#copyAndConfigureFiles into it's own + class. (Chris Trezzo via kasha) + + MAPREDUCE-6238. MR2 can't run local jobs with -libjars command options + which is a regression from MR1 (zxu via rkanter) + + MAPREDUCE-6324. Fixed MapReduce uber jobs to not fail the udpate of AM-RM + tokens when they roll-over. (Jason Lowe via vinodkv) + + MAPREDUCE-5649. Reduce cannot use more than 2G memory for the final merge + (Gera Shegalov via jlowe) + + MAPREDUCE-6361. NPE issue in shuffle caused by concurrent issue between + copySucceeded() in one thread and copyFailed() in another thread on the + same host. (Junping Du via ozawa) + Release 2.6.0 - 2014-11-18 INCOMPATIBLE CHANGES http://git-wip-us.apache.org/repos/asf/hadoop/blob/c05ba079/hadoop-yarn-project/CHANGES.txt ---------------------------------------------------------------------- diff --git a/hadoop-yarn-project/CHANGES.txt b/hadoop-yarn-project/CHANGES.txt index 9bd5cb1..c10fac4 100644 --- a/hadoop-yarn-project/CHANGES.txt +++ b/hadoop-yarn-project/CHANGES.txt @@ -17,18 +17,12 @@ Release 2.7.2 - UNRELEASED YARN-3967. Fetch the application report from the AHS if the RM does not know about it. (Mit Desai via xgong) - YARN-3978. Configurably turn off the saving of container info in Generic AHS - (Eric Payne via jeagles) - YARN-4092. Fixed UI redirection to print useful messages when both RMs are in standby mode. (Xuan Gong via jianhe) YARN-2801. Add documentation for node labels feature. (Wangda Tan and Naganarasimha G R via ozawa) - YARN-3700. Made generic history service load a number of latest applications - according to the parameter or the configuration. (Xuan Gong via zjshen) - OPTIMIZATIONS BUG FIXES @@ -56,14 +50,6 @@ Release 2.7.2 - UNRELEASED YARN-3925. ContainerLogsUtils#getContainerLogFile fails to read container log files from full disks. (zhihai xu via jlowe) - YARN-3990. AsyncDispatcher may overloaded with RMAppNodeUpdateEvent when - Node is connected/disconnected (Bibin A Chundatt via jlowe) - - YARN-3999. RM hangs on draing events. (Jian He via xgong) - - YARN-4047. ClientRMService getApplications has high scheduler lock contention - (Jason Lowe via jianhe) - YARN-3857: Memory leak in ResourceManager with SIMPLE mode. (mujunchao via zxu) @@ -118,9 +104,6 @@ Release 2.7.1 - 2015-07-06 BUG FIXES - YARN-3487. CapacityScheduler scheduler lock obtained unnecessarily when - calling getQueue (Jason Lowe via wangda) - YARN-3462. Patches applied for YARN-2424 are inconsistent between trunk and branch-2. (Naganarasimha G R via harsh) @@ -147,9 +130,6 @@ Release 2.7.1 - 2015-07-06 YARN-3516. killing ContainerLocalizer action doesn't take effect when private localizer receives FETCH_FAILURE status.(zhihai xu via xgong) - YARN-3464. Race condition in LocalizerRunner kills localizer before - localizing all resources. (Zhihai Xu via kasha) - YARN-3485. FairScheduler headroom calculation doesn't consider maxResources for Fifo and FairShare policies. (kasha) @@ -174,9 +154,6 @@ Release 2.7.1 - 2015-07-06 YARN-3434. Interaction between reservations and userlimit can result in significant ULF violation. (Thomas Graves via wangda) - YARN-3493. RM fails to come up with error "Failed to load/recover state" - when mem settings are changed. (Jian He via wangda) - YARN-3626. On Windows localized resources are not moved to the front of the classpath when they should be. (Craig Welch via xgong) @@ -186,20 +163,11 @@ Release 2.7.1 - 2015-07-06 YARN-3537. NPE when NodeManager.serviceInit fails and stopRecoveryStore invoked (Brahma Reddy Battula via jlowe) - YARN-3641. NodeManager: stopRecoveryStore() shouldn't be skipped when - exceptions happen in stopping NM's sub-services. (Junping Du via jlowe) - - YARN-3526. ApplicationMaster tracking URL is incorrectly redirected - on a QJM cluster. (Weiwei Yang via xgong) - YARN-3601. Fix UT TestRMFailover.testRMWebAppRedirect. (Weiwei Yang via xgong) YARN-3677. Fix findbugs warnings in yarn-server-resourcemanager. (Vinod Kumar Vavilapalli via ozawa) - YARN-2918. RM should not fail on startup if queue's configured labels do - not exist in cluster-node-labels. (Wangda Tan via jianhe) - YARN-3681. yarn cmd says "could not find main class 'queue'" in windows. (Craig Welch and Varun Saxena via xgong) @@ -224,21 +192,9 @@ Release 2.7.1 - 2015-07-06 YARN-3686. CapacityScheduler should trim default_node_label_expression. (Sunil G via wangda) - YARN-2900. Application (Attempt and Container) Not Found in AHS results - in Internal Server Error (500). (Zhijie Shen and Mit Desai via xgong) - - YARN-3725. App submission via REST API is broken in secure mode due to - Timeline DT service address is empty. (Zhijie Shen via wangda) - YARN-3753. RM failed to come up with "java.io.IOException: Wait for ZKClient creation timed outâ. (Jian He via xgong) - YARN-3585. NodeManager cannot exit on SHUTDOWN event triggered and NM - recovery is enabled (Rohith Sharmaks via jlowe) - - YARN-3733. Fix DominantRC#compare() does not work as expected if - cluster resource is empty. (Rohith Sharmaks via wangda) - YARN-3764. CapacityScheduler should forbid moving LeafQueue from one parent to another. (Wangda Tan via jianhe) @@ -251,12 +207,6 @@ Release 2.7.1 - 2015-07-06 YARN-3809. Failed to launch new attempts because ApplicationMasterLauncher's threads all hang (Jun Gong via jlowe) - YARN-3832. Resource Localization fails on a cluster due to existing cache - directories (Brahma Reddy Battula via jlowe) - - YARN-3850. NM fails to read files from full disks which can lead to - container logs being lost and other issues (Varun Saxena via jlowe) - Release 2.7.0 - 2015-04-20 INCOMPATIBLE CHANGES @@ -306,9 +256,6 @@ Release 2.7.0 - 2015-04-20 YARN-2994. Document work-preserving RM restart. (Jian He via ozawa) - YARN-3249. Add a 'kill application' button to Resource Manager's Web UI. - (Ryu Kobayashi via ozawa) - YARN-2786. Created a yarn cluster CLI and seeded with one command for listing node-labels collection. (Wangda Tan via vinodkv) @@ -379,8 +326,6 @@ Release 2.7.0 - 2015-04-20 YARN-1156. Enhance NodeManager AllocatedGB and AvailableGB metrics for aggregation of decimal values. (Tsuyoshi OZAWA via junping_du) - YARN-2301. Improved yarn container command. (Naganarasimha G R via jianhe) - YARN-2056. Disable preemption at Queue level (Eric Payne via jlowe) YARN-2762. Fixed RMAdminCLI to trim and check node-label related arguments @@ -434,12 +379,6 @@ Release 2.7.0 - 2015-04-20 YARN-2800. Remove MemoryNodeLabelsStore and add a way to enable/disable node labels feature. (Wangda Tan via ozawa) - YARN-3024. LocalizerRunner should give DIE action when all resources are - localized. (Chengbing Liu via xgong) - - YARN-3092. Created a common ResourceUsage class to track labeled resource - usages in Capacity Scheduler. (Wangda Tan via jianhe) - YARN-3086. Make NodeManager memory configurable in MiniYARNCluster. (Robert Metzger via ozawa) @@ -465,9 +404,6 @@ Release 2.7.0 - 2015-04-20 YARN-3022. Expose Container resource information from NodeManager for monitoring (adhoot via ranter) - YARN-3098. Created common QueueCapacities class in Capacity Scheduler to - track capacities-by-labels of queues. (Wangda Tan via jianhe) - YARN-3075. NodeLabelsManager implementation to retrieve label to node mapping (Varun Saxena via wangda) @@ -508,9 +444,6 @@ Release 2.7.0 - 2015-04-20 YARN-2079. Recover NonAggregatingLogHandler state upon nodemanager restart. (Jason Lowe via junping_du) - YARN-3124. Fixed CS LeafQueue/ParentQueue to use QueueCapacities to track - capacities-by-label. (Wangda Tan via jianhe) - YARN-3158. Correct log messages in ResourceTrackerService. (Varun Saxena via xgong) @@ -535,8 +468,6 @@ Release 2.7.0 - 2015-04-20 YARN-2799. Cleanup TestLogAggregationService based on the change in YARN-90. (Zhihai Xu via junping_du) - YARN-3230. Clarify application states on the web UI. (Jian He via wangda) - YARN-3237. AppLogAggregatorImpl fails to log error cause. (Rushabh S Shah via xgong) @@ -573,9 +504,6 @@ Release 2.7.0 - 2015-04-20 YARN-3122. Metrics for container's actual CPU usage. (Anubhav Dhoot via kasha) - YARN-1809. Synchronize RM and TimeLineServer Web-UIs. (Zhijie Shen and - Xuan Gong via jianhe) - YARN-2190. Added CPU and memory limit options to the default container executor for Windows containers. (Chuan Liu via jianhe) @@ -614,56 +542,26 @@ Release 2.7.0 - 2015-04-20 YARN-2713. "RM Home" link in NM should point to one of the RMs in an HA setup. (kasha) - YARN-2766. Made ApplicationHistoryManager return a sorted list of apps, - attempts and containers. (Robert Kanter via zjshen) - - YARN-2856. Fixed RMAppImpl to handle ATTEMPT_KILLED event at ACCEPTED state - on app recovery. (Rohith Sharmaks via jianhe) - YARN-2857. ConcurrentModificationException in ContainerLogAppender (Mohammad Kamrul Islam via jlowe) - YARN-2816. NM fail to start with NPE during container recovery (Zhihai Xu - via jlowe) - YARN-2432. RMStateStore should process the pending events before close. (Varun Saxena via jianhe) YARN-1703. Fixed ResourceManager web-proxy to close connections correctly. (Rohith Sharma via vinodkv) - YARN-2414. RM web UI: app page will crash if app is failed before any - attempt has been created (Wangda Tan via jlowe) - YARN-2870. Updated the command to run the timeline server in the document. (Masatake Iwasaki via zjshen) YARN-2878. Fix DockerContainerExecutor.apt.vm formatting. (Abin Shahab via jianhe) - YARN-2865. Fixed RM to always create a new RMContext when transtions from - StandBy to Active. (Rohith Sharmaks via jianhe) - YARN-2315. FairScheduler: Set current capacity in addition to capacity. (Zhihai Xu via kasha) - YARN-1984. LeveldbTimelineStore does not handle db exceptions properly - (Varun Saxena via jlowe) - YARN-2697. Remove useless RMAuthenticationHandler. (Haosong Huang via zjshen) - YARN-2906. CapacitySchedulerPage shows HTML tags for a queue's Active Users. - (Jason Lowe via jianhe) - - YARN-2905. AggregatedLogsBlock page can infinitely loop if the aggregated - log file is corrupted (Varun Saxena via jlowe) - - YARN-2894. Fixed a bug regarding application view acl when RM fails over. - (Rohith Sharmaks via jianhe) - - YARN-2874. Dead lock in "DelegationTokenRenewer" which blocks RM to execute - any further apps. (Naganarasimha G R via kasha) - YARN-2461. Fix PROCFS_USE_SMAPS_BASED_RSS_ENABLED property in YarnConfiguration. (rchiang via rkanter) @@ -676,18 +574,12 @@ Release 2.7.0 - 2015-04-20 YARN-2931. PublicLocalizer may fail until directory is initialized by LocalizeRunner. (Anubhav Dhoot via kasha) - YARN-2910. FSLeafQueue can throw ConcurrentModificationException. - (Wilfred Spiegelenburg via kasha) - YARN-2930. Fixed TestRMRestart#testRMRestartRecoveringNodeLabelManager intermittent failure. (Wangda Tan via jianhe) YARN-2924. Fixed RMAdminCLI to not convert node labels to lower case. (Wangda Tan via jianhe) - YARN-2917. Fixed potential deadlock when system.exit is called in AsyncDispatcher - (Rohith Sharmaks via jianhe) - YARN-2243. Order of arguments for Preconditions.checkNotNull() is wrong in SchedulerApplicationAttempt ctor. (devaraj) @@ -705,15 +597,9 @@ Release 2.7.0 - 2015-04-20 YARN-2944. InMemorySCMStore can not be instantiated with ReflectionUtils#newInstance. (Chris Trezzo via kasha) - YARN-2964. RM prematurely cancels tokens for jobs that submit jobs (oozie) - (Jian He via jlowe) - YARN-2675. containersKilled metrics is not updated when the container is killed during localization. (Zhihai Xu via kasha) - YARN-2952. Fixed incorrect version check in StateStore. (Rohith Sharmaks - via jianhe) - YARN-2975. FSLeafQueue app lists are accessed without required locks. (kasha) YARN-2977. Fixed intermittent TestNMClient failure. @@ -721,12 +607,6 @@ Release 2.7.0 - 2015-04-20 YARN-2939. Fix new findbugs warnings in hadoop-yarn-common. (Li Lu via junping_du) - YARN-2920. Changed CapacityScheduler to kill containers on nodes where - node labels are changed. (Wangda Tan via jianhe) - - YARN-2340. Fixed NPE when queue is stopped during RM restart. - (Rohith Sharmaks via jianhe) - YARN-2940. Fix new findbugs warnings in rest of the hadoop-yarn components. (Li Lu via junping_du) @@ -738,9 +618,6 @@ Release 2.7.0 - 2015-04-20 YARN-2988. Graph#save() may leak file descriptors. (Ted Yu via ozawa) - YARN-2992. ZKRMStateStore crashes due to session expiry. (Karthik Kambatla - via jianhe) - YARN-2938. Fixed new findbugs warnings in hadoop-yarn-resourcemanager and hadoop-yarn-applicationhistoryservice. (Varun Saxena via zjshen) @@ -750,15 +627,9 @@ Release 2.7.0 - 2015-04-20 YARN-2991. Fixed DrainDispatcher to reuse the draining code path in AsyncDispatcher. (Rohith Sharmaks via zjshen) - YARN-2922. ConcurrentModificationException in CapacityScheduler's LeafQueue. - (Rohith Sharmaks via ozawa) - YARN-2958. Made RMStateStore not update the last sequence number when updating the delegation token. (Varun Saxena via zjshen) - YARN-2978. Fixed potential NPE while getting queue info. (Varun Saxena via - jianhe) - YARN-2230. Fixed few configs description in yarn-default.xml. (Vijay Bhat via jianhe) @@ -768,18 +639,12 @@ Release 2.7.0 - 2015-04-20 YARN-2936. Changed YARNDelegationTokenIdentifier to set proto fields on getProto method. (Varun Saxena via jianhe) - YARN-2997. Fixed NodeStatusUpdater to not send alreay-sent completed - container statuses on heartbeat. (Chengbing Liu via jianhe) - YARN-3014. Replaces labels on a host should update all NM's labels on that host. (Wangda Tan via jianhe) YARN-3027. Scheduler should use totalAvailable resource from node instead of availableResource for maxAllocation. (adhoot via rkanter) - YARN-2637. Fixed max-am-resource-percent calculation in CapacityScheduler - when activating applications. (Craig Welch via jianhe) - YARN-2861. Fixed Timeline DT secret manager to not reuse RM's configs. (Zhijie Shen via jianhe) @@ -810,21 +675,12 @@ Release 2.7.0 - 2015-04-20 YARN-3088. LinuxContainerExecutor.deleteAsUser can throw NPE if native executor returns an error (Eric Payne via jlowe) - YARN-3011. Possible IllegalArgumentException in ResourceLocalizationService - might lead NM to crash. (Varun Saxena via jianhe) - - YARN-3103. AMRMClientImpl does not update AMRM token properly. (Jason Lowe - via jianhe) - YARN-3079. Scheduler should also update maximumAllocation when updateNodeResource. (Zhihai Xu via wangda) YARN-3029. FSDownload.unpack() uses local locale for FS case conversion, may not work everywhere. (Varun Saxena via ozawa) - YARN-3099. Capacity Scheduler LeafQueue/ParentQueue should use ResourceUsage - to track used-resources-by-label.(Wangda Tan via jianhe) - YARN-3077. Fixed RM to create zk root path recursively. (Chun Chen via jianhe) YARN-3113. Release audit warning for Sorting icons.psd. (stevel via kihwal) @@ -850,18 +706,12 @@ Release 2.7.0 - 2015-04-20 YARN-1537. Fix race condition in TestLocalResourcesTrackerImpl.testLocalResourceCache. (xgong via acmurthy) - YARN-2694. Ensure only single node label specified in ResourceRequest. - (Wangda Tan via jianhe) - YARN-3089. LinuxContainerExecutor does not handle file arguments to deleteAsUser (Eric Payne via jlowe) YARN-3143. RM Apps REST API can return NPE or entries missing id and other fields (jlowe) - YARN-3094. Reset timer for liveness monitors after RM recovery. (Jun Gong - via jianhe) - YARN-2971. RM uses conf instead of token service address to renew timeline delegation tokens (jeagles) @@ -871,10 +721,6 @@ Release 2.7.0 - 2015-04-20 YARN-2809. Implement workaround for linux kernel panic when removing cgroup (Nathan Roberts via jlowe) - YARN-2246. Made the proxy tracking URL always be - http(s)://proxy addr:port/proxy/<appId> to avoid duplicate sections. (Devaraj - K via zjshen) - YARN-3160. Fix non-atomic operation on nodeUpdateQueue in RMNodeImpl. (Chengbing Liu via junping_du) @@ -905,9 +751,6 @@ Release 2.7.0 - 2015-04-20 YARN-2749. Fix some testcases from TestLogAggregationService fails in trunk. (Xuan Gong via junping_du) - YARN-3207. Secondary filter matches entites which do not have the key being - filtered for. (Zhijie Shen via xgong) - YARN-3132. RMNodeLabelsManager should remove node from node-to-label mapping when node becomes deactivated. (Wangda Tan via jianhe) @@ -920,15 +763,9 @@ Release 2.7.0 - 2015-04-20 YARN-3194. RM should handle NMContainerStatuses sent by NM while registering if NM is Reconnected node (Rohith via jlowe) - YARN-3238. Connection timeouts to nodemanagers are retried at - multiple levels (Jason Lowe via xgong) - YARN-3247. TestQueueMappings should use CapacityScheduler explicitly. (Zhihai Xu via ozawa) - YARN-3239. WebAppProxy does not support a final tracking url which has - query fragments and params (Jian He via jlowe) - YARN-3256. TestClientToAMTokens#testClientTokenRace is not running against all Schedulers even when using ParameterizedSchedulerTestBase. (Anubhav Dhoot via devaraj) @@ -939,27 +776,12 @@ Release 2.7.0 - 2015-04-20 YARN-3265. Fixed a deadlock in CapacityScheduler by always passing a queue's available resource-limit from the parent queue. (Wangda Tan via vinodkv) - YARN-3222. Fixed RMNode to send scheduler events in sequential order when a - node reconnects. (Rohith Sharma K S via jianhe) - YARN-3131. YarnClientImpl should check FAILED and KILLED state in submitApplication (Chang Li via jlowe) - YARN-3231. FairScheduler: Changing queueMaxRunningApps interferes with pending - jobs. (Siqi Li via kasha) - - YARN-3242. Asynchrony in ZK-close can lead to ZKRMStateStore watcher receiving - events for old client. (Zhihai Xu via kasha) - - YARN-3227. Timeline renew delegation token fails when RM user's TGT is expired - (Zhijie Shen via xgong) - YARN-3275. CapacityScheduler: Preemption happening on non-preemptable queues (Eric Payne via jlowe) - YARN-3287. Made TimelineClient put methods do as the correct login context. - (Daryn Sharp and Jonathan Eagles via zjshen) - YARN-3300. Outstanding_resource_requests table should not be shown in AHS. (Xuan Gong via jianhe) @@ -974,9 +796,6 @@ Release 2.7.0 - 2015-04-20 YARN-3154. Added additional APIs in LogAggregationContext to avoid aggregating running logs of application when rolling is enabled. (Xuan Gong via vinodkv) - YARN-3267. Timelineserver applies the ACL rules after applying the limit on - the number of records (Chang Li via jeagles) - YARN-3171. Sort by Application id, AppAttempt and ContainerID doesn't work in ATS / RM web ui. (Naganarasimha G R via xgong) @@ -989,18 +808,12 @@ Release 2.7.0 - 2015-04-20 YARN-3379. Fixed missing data in localityTable and ResourceRequests table in RM WebUI. (Xuan Gong via jianhe) - YARN-3369. Missing NullPointer check in AppSchedulingInfo causes RM to die. - (Brahma Reddy Battula via wangda) - YARN-3384. TestLogAggregationService.verifyContainerLogs fails after YARN-2777. (Naganarasimha G R via ozawa) YARN-3336. FileSystem memory leak in DelegationTokenRenewer. (Zhihai Xu via cnauroth) - YARN-3393. Getting application(s) goes wrong when app finishes before - starting the attempt. (Zhijie Shen via xgong) - YARN-2213. Change proxy-user cookie log in AmIpFilter to DEBUG. (Varun Saxena via xgong) @@ -1011,9 +824,6 @@ Release 2.7.0 - 2015-04-20 YARN-3430. Made headroom data available on app attempt page of RM WebUI. (Xuan Gong via zjshen) - YARN-3055. Fixed ResourceManager's DelegationTokenRenewer to not stop token - renewal of applications part of a bigger workflow. (Daryn Sharp via vinodkv) - YARN-3466. Fix RM nodes web page to sort by node HTTP-address, #containers and node-label column (Jason Lowe via wangda) @@ -1029,20 +839,210 @@ Release 2.6.2 - UNRELEASED BUG FIXES -Release 2.6.1 - UNRELEASED +Release 2.6.1 - 2015-09-09 INCOMPATIBLE CHANGES NEW FEATURES + YARN-3249. Add a 'kill application' button to Resource Manager's Web UI. + (Ryu Kobayashi via ozawa) + IMPROVEMENTS + YARN-3230. Clarify application states on the web UI. (Jian He via wangda) + + YARN-1809. Synchronize RM and TimeLineServer Web-UIs. (Zhijie Shen and + Xuan Gong via jianhe) + + YARN-3092. Created a common ResourceUsage class to track labeled resource + usages in Capacity Scheduler. (Wangda Tan via jianhe) + + YARN-3098. Created common QueueCapacities class in Capacity Scheduler to + track capacities-by-labels of queues. (Wangda Tan via jianhe) + + YARN-2301. Improved yarn container command. (Naganarasimha G R via jianhe) + + YARN-3978. Configurably turn off the saving of container info in Generic AHS + (Eric Payne via jeagles) + OPTIMIZATIONS BUG FIXES - YARN-2890. MiniYarnCluster should turn on timeline service if - configured to do so. (Mit Desai via hitesh) + YARN-2856. Fixed RMAppImpl to handle ATTEMPT_KILLED event at ACCEPTED state + on app recovery. (Rohith Sharmaks via jianhe) + + YARN-2816. NM fail to start with NPE during container recovery (Zhihai Xu + via jlowe) + + YARN-2414. RM web UI: app page will crash if app is failed before any + attempt has been created (Wangda Tan via jlowe) + + YARN-2865. Fixed RM to always create a new RMContext when transtions from + StandBy to Active. (Rohith Sharmaks via jianhe) + + YARN-2906. CapacitySchedulerPage shows HTML tags for a queue's Active Users. + (Jason Lowe via jianhe) + + YARN-2905. AggregatedLogsBlock page can infinitely loop if the aggregated + log file is corrupted (Varun Saxena via jlowe) + + YARN-2890. MiniYARNCluster should start the timeline server based on the + configuration. (Mit Desai via zjshen) + + YARN-2894. Fixed a bug regarding application view acl when RM fails over. + (Rohith Sharmaks via jianhe) + + YARN-2874. Dead lock in "DelegationTokenRenewer" which blocks RM to execute + any further apps. (Naganarasimha G R via kasha) + + YARN-2910. FSLeafQueue can throw ConcurrentModificationException. + (Wilfred Spiegelenburg via kasha) + + YARN-2917. Fixed potential deadlock when system.exit is called in AsyncDispatcher + (Rohith Sharmaks via jianhe) + + YARN-2964. RM prematurely cancels tokens for jobs that submit jobs (oozie) + (Jian He via jlowe) + + YARN-1984. LeveldbTimelineStore does not handle db exceptions properly + (Varun Saxena via jlowe) + + YARN-2952. Fixed incorrect version check in StateStore. (Rohith Sharmaks + via jianhe) + + YARN-2340. Fixed NPE when queue is stopped during RM restart. + (Rohith Sharmaks via jianhe) + + YARN-2992. ZKRMStateStore crashes due to session expiry. (Karthik Kambatla + via jianhe) + + YARN-2922. ConcurrentModificationException in CapacityScheduler's LeafQueue. + (Rohith Sharmaks via ozawa) + + YARN-2997. Fixed NodeStatusUpdater to not send alreay-sent completed + container statuses on heartbeat. (Chengbing Liu via jianhe) + + YARN-3011. Possible IllegalArgumentException in ResourceLocalizationService + might lead NM to crash. (Varun Saxena via jianhe) + + YARN-3103. AMRMClientImpl does not update AMRM token properly. (Jason Lowe + via jianhe) + + YARN-3094. Reset timer for liveness monitors after RM recovery. (Jun Gong + via jianhe) + + YARN-2246. Made the proxy tracking URL always be + http(s)://proxy addr:port/proxy/<appId> to avoid duplicate sections. (Devaraj + K via zjshen) + + YARN-3207. Secondary filter matches entites which do not have the key being + filtered for. (Zhijie Shen via xgong) + + YARN-3238. Connection timeouts to nodemanagers are retried at + multiple levels (Jason Lowe via xgong) + + YARN-3239. WebAppProxy does not support a final tracking url which has + query fragments and params (Jian He via jlowe) + + YARN-3222. Fixed RMNode to send scheduler events in sequential order when a + node reconnects. (Rohith Sharma K S via jianhe) + + YARN-3231. FairScheduler: Changing queueMaxRunningApps interferes with pending + jobs. (Siqi Li via kasha) + + YARN-3242. Asynchrony in ZK-close can lead to ZKRMStateStore watcher receiving + events for old client. (Zhihai Xu via kasha) + + YARN-3227. Timeline renew delegation token fails when RM user's TGT is expired + (Zhijie Shen via xgong) + + YARN-3287. Made TimelineClient put methods do as the correct login context. + (Daryn Sharp and Jonathan Eagles via zjshen) + + YARN-3267. Timelineserver applies the ACL rules after applying the limit on + the number of records (Chang Li via jeagles) + + YARN-3369. Missing NullPointer check in AppSchedulingInfo causes RM to die. + (Brahma Reddy Battula via wangda) + + YARN-3393. Getting application(s) goes wrong when app finishes before + starting the attempt. (Zhijie Shen via xgong) + + YARN-3055. Fixed ResourceManager's DelegationTokenRenewer to not stop token + renewal of applications part of a bigger workflow. (Daryn Sharp via vinodkv) + + YARN-3493. RM fails to come up with error "Failed to load/recover state" + when mem settings are changed. (Jian He via wangda) + + YARN-3487. CapacityScheduler scheduler lock obtained unnecessarily when + calling getQueue (Jason Lowe via wangda) + + YARN-3024. LocalizerRunner should give DIE action when all resources are + localized. (Chengbing Liu via xgong) + + YARN-3464. Race condition in LocalizerRunner kills localizer before + localizing all resources. (Zhihai Xu via kasha) + + YARN-3641. NodeManager: stopRecoveryStore() shouldn't be skipped when + exceptions happen in stopping NM's sub-services. (Junping Du via jlowe) + + YARN-3526. ApplicationMaster tracking URL is incorrectly redirected + on a QJM cluster. (Weiwei Yang via xgong) + + YARN-2766. Made ApplicationHistoryManager return a sorted list of apps, + attempts and containers. (Robert Kanter via zjshen) + + YARN-3700. Made generic history service load a number of latest applications + according to the parameter or the configuration. (Xuan Gong via zjshen) + + YARN-2900. Application (Attempt and Container) Not Found in AHS results + in InternalServer Error (500). (Zhijie Shen and Mit Desai via xgong) + + YARN-3725. App submission via REST API is broken in secure mode due to + Timeline DT service address is empty. (Zhijie Shen via wangda) + + YARN-3585. NodeManager cannot exit on SHUTDOWN event triggered and NM + recovery is enabled (Rohith Sharmaks via jlowe) + + YARN-3832. Resource Localization fails on a cluster due to existing cache + directories (Brahma Reddy Battula via jlowe) + + YARN-3850. NM fails to read files from full disks which can lead to + container logs being lost and other issues (Varun Saxena via jlowe) + + YARN-3990. AsyncDispatcher may overloaded with RMAppNodeUpdateEvent when + Node is connected/disconnected (Bibin A Chundatt via jlowe) + + YARN-2637. Fixed max-am-resource-percent calculation in CapacityScheduler + when activating applications. (Craig Welch via jianhe) + + YARN-3733. Fix DominantRC#compare() does not work as expected if + cluster resource is empty. (Rohith Sharmaks via wangda) + + YARN-2920. Changed CapacityScheduler to kill containers on nodes where + node labels are changed. (Wangda Tan via jianhe) + + YARN-2978. Fixed potential NPE while getting queue info. (Varun Saxena via + jianhe) + + YARN-3099. Capacity Scheduler LeafQueue/ParentQueue should use ResourceUsage + to track used-resources-by-label.(Wangda Tan via jianhe) + + YARN-2694. Ensure only single node label specified in ResourceRequest. + (Wangda Tan via jianhe) + + YARN-3124. Fixed CS LeafQueue/ParentQueue to use QueueCapacities to track + capacities-by-label. (Wangda Tan via jianhe) + + YARN-2918. RM should not fail on startup if queue's configured labels do + not exist in cluster-node-labels. (Wangda Tan via jianhe) + + YARN-3999. RM hangs on draing events. (Jian He via xgong) + + YARN-4047. ClientRMService getApplications has high scheduler lock contention. + (Jason Lowe via jianhe) Release 2.6.0 - 2014-11-18