Repository: hadoop Updated Branches: refs/heads/branch-2 5decd8f9f -> cbf90f93f
Updating all CHANGES.txt files to move entires from future releases into 2.6.1 section given the large number of backports to 2.6.1. (cherry picked from commit fbbb7ff1ed111884d0375c177239073153139848) Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/cbf90f93 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/cbf90f93 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/cbf90f93 Branch: refs/heads/branch-2 Commit: cbf90f93f9ab3463307292ddfe6fe98ab8b37842 Parents: 5decd8f Author: Vinod Kumar Vavilapalli (I am also known as @tshooter.) <vino...@apache.org> Authored: Thu Sep 10 13:45:57 2015 -0700 Committer: Vinod Kumar Vavilapalli (I am also known as @tshooter.) <vino...@apache.org> Committed: Thu Sep 10 14:01:15 2015 -0700 ---------------------------------------------------------------------- hadoop-common-project/hadoop-common/CHANGES.txt | 112 +++--- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt | 263 +++++++------ hadoop-mapreduce-project/CHANGES.txt | 56 +-- hadoop-yarn-project/CHANGES.txt | 390 +++++++++---------- 4 files changed, 408 insertions(+), 413 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/hadoop/blob/cbf90f93/hadoop-common-project/hadoop-common/CHANGES.txt ---------------------------------------------------------------------- diff --git a/hadoop-common-project/hadoop-common/CHANGES.txt b/hadoop-common-project/hadoop-common/CHANGES.txt index 88b86a7..c900083 100644 --- a/hadoop-common-project/hadoop-common/CHANGES.txt +++ b/hadoop-common-project/hadoop-common/CHANGES.txt @@ -243,9 +243,6 @@ Release 2.8.0 - UNRELEASED command-line arguments passed by the user (Masatake Iwasaki via Colin P. McCabe) - HADOOP-12280. Skip unit tests based on maven profile rather than - NativeCodeLoader.isNativeCodeLoaded (Masatake Iwasaki via Colin P. McCabe) - HADOOP-12318. Expose underlying LDAP exceptions in SaslPlainServer. (Mike Yoder via atm) @@ -266,9 +263,6 @@ Release 2.8.0 - UNRELEASED HADOOP-11785. Reduce the number of listStatus operation in distcp buildListing (Zoran Dimitrijevic via Colin P. McCabe) - HADOOP-11812. Implement listLocatedStatus for ViewFileSystem to speed up - split calculation (gera) - HADOOP-11970. Replace uses of ThreadLocal<Random> with JDK7 ThreadLocalRandom. (Sean Busbey via Colin P. McCabe) @@ -304,9 +298,6 @@ Release 2.8.0 - UNRELEASED HADOOP-12352. Delay in checkpointing Trash can leave trash for 2 intervals before deleting (Casey Brotherton via harsh) - HADOOP-11802: DomainSocketWatcher thread terminates sometimes after there - is an I/O error during requestShortCircuitShm (cmccabe) - HADOOP-11568. Description on usage of classpath in hadoop command is incomplete. ( Archana T via vinayakumarb ) @@ -358,9 +349,6 @@ Release 2.8.0 - UNRELEASED HADOOP-11866. increase readability and reliability of checkstyle, shellcheck, and whitespace reports (aw) - HADOOP-11491. HarFs incorrectly declared as requiring an authority. - (Brahma Reddy Battula via gera) - HADOOP-11889. Make checkstyle runnable from root project (Gera Shegalov via jeagles) @@ -632,9 +620,6 @@ Release 2.7.2 - UNRELEASED HADOOP-12304. Applications using FileContext fail with the default file system configured to be wasb/s3/etc. (cnauroth) - HADOOP-11932. MetricsSinkAdapter may hang when being stopped. - (Brahma Reddy Battula via jianhe) - HADOOP-12061. Incorrect command in single cluster setup document. (Kengo Seki via aajisaka) @@ -668,12 +653,6 @@ Release 2.7.1 - 2015-07-06 HADOOP-11872. "hadoop dfs" command prints message about using "yarn jar" on Windows(branch-2 only) (Varun Vasudev via cnauroth) - HADOOP-11730. Regression: s3n read failure recovery broken. - (Takenori Sato via stevel) - - HADOOP-11802. DomainSocketWatcher thread terminates sometimes after there - is an I/O error during requestShortCircuitShm (cmccabe) - HADOOP-11891. OsSecureRandom should lazily fill its reservoir (asuresh) HADOOP-9658. SnappyCodec#checkNativeCodeLoaded may unexpectedly fail when @@ -688,9 +667,6 @@ Release 2.7.1 - 2015-07-06 HADOOP-11973. Ensure ZkDelegationTokenSecretManager namespace znodes get created with ACLs. (Gregory Chanan via asuresh) - HADOOP-11934. Use of JavaKeyStoreProvider in LdapGroupsMapping causes - infinite loop. (Larry McCay via cnauroth) - HADOOP-12058. Fix dead links to DistCp and Hadoop Archives pages. (Kazuho Fujii via aajisaka) @@ -957,18 +933,12 @@ Release 2.7.0 - 2015-04-20 HADOOP-11323. WritableComparator#compare keeps reference to byte array. (Wilfred Spiegelenburg via wang) - HADOOP-11238. Update the NameNode's Group Cache in the background when - possible (Chris Li via Colin P. McCabe) - HADOOP-10809. hadoop-azure: page blob support. (Dexter Bradshaw, Mostafa Elhemali, Eric Hanson, and Mike Liddell via cnauroth) HADOOP-11188. hadoop-azure: automatically expand page blobs when they become full. (Eric Hanson via cnauroth) - HADOOP-11506. Configuration variable expansion regex expensive for long - values. (Gera Shegalov via gera) - HADOOP-11231. Remove dead code in ServletUtil. (Li Lu via wheat9) HADOOP-11620. Add support for load balancing across a group of KMS for HA. @@ -1083,9 +1053,6 @@ Release 2.7.0 - 2015-04-20 HADOOP-11355. When accessing data in HDFS and the key has been deleted, a Null Pointer Exception is shown. (Arun Suresh via wang) - HADOOP-11343. Overflow is not properly handled in caclulating final iv for - AES CTR. (Jerry Chen via wang) - HADOOP-11354. ThrottledInputStream doesn't perform effective throttling. (Ted Yu via jing9) @@ -1101,9 +1068,6 @@ Release 2.7.0 - 2015-04-20 HADOOP-11369. Fix new findbugs warnings in hadoop-mapreduce-client, non-core directories. (Li Lu via wheat9) - HADOOP-11368. Fix SSLFactory truststore reloader thread leak in - KMSClientProvider. (Arun Suresh via wang) - HADOOP-11372. Fix new findbugs warnings in mapreduce-examples. (Li Lu via wheat9) @@ -1201,9 +1165,6 @@ Release 2.7.0 - 2015-04-20 HADOOP-11462. TestSocketIOWithTimeout needs change for PowerPC platform. (Ayappan via cnauroth) - HADOOP-11350. The size of header buffer of HttpServer is too small when - HTTPS is enabled. (Benoy Antony via wheat9) - HADOOP-10542 Potential null pointer dereference in Jets3tFileSystemStore retrieveBlock(). (Ted Yu via stevel) @@ -1228,9 +1189,6 @@ Release 2.7.0 - 2015-04-20 HADOOP-11507 Hadoop RPC Authentication problem with different user locale. (Talat UYARER via stevel) - HADOOP-11482. Use correct UGI when KMSClientProvider is called by a proxy - user. Contributed by Arun Suresh. - HADOOP-11499. Check of executorThreadsStarted in ValueQueue#submitRefillTask() evades lock acquisition (Ted Yu via jlowe) @@ -1302,9 +1260,6 @@ Release 2.7.0 - 2015-04-20 HADOOP-11570. S3AInputStream.close() downloads the remaining bytes of the object from S3. (Dan Hecht via stevel). - HADOOP-11295. RPC Server Reader thread can't shutdown if RPCCallQueue is - full. (Ming Ma via kihwal) - HADOOP-11599. Client#getTimeout should use IPC_CLIENT_PING_DEFAULT when IPC_CLIENT_PING_KEY is not configured. (zhihai xu via ozawa) @@ -1317,9 +1272,6 @@ Release 2.7.0 - 2015-04-20 HADOOP-9087. Queue size metric for metric sinks isn't actually maintained (Akira AJISAKA via jlowe) - HADOOP-11604. Prevent ConcurrentModificationException while closing domain - sockets during shutdown of DomainSocketWatcher thread. (cnauroth) - HADOOP-11612. Workaround for Curator's ChildReaper requiring Guava 15+. (rkanter) @@ -1356,9 +1308,6 @@ Release 2.7.0 - 2015-04-20 HADOOP-11666. Revert the format change of du output introduced by HADOOP-6857. (Byron Wong via aajisaka) - HADOOP-11674. oneByteBuf in CryptoInputStream and CryptoOutputStream - should be non static. (Sean Busbey via yliu) - HADOOP-11670. Regression: s3a auth setup broken. (Adam Budde via stevel) HADOOP-11602. Fix toUpperCase/toLowerCase to use Locale.ENGLISH. (ozawa) @@ -1372,9 +1321,6 @@ Release 2.7.0 - 2015-04-20 HADOOP-11693. Azure Storage FileSystem rename operations are throttled too aggressively to complete HBase WAL archiving. (Duo Xu via cnauroth) - HADOOP-11710. Make CryptoOutputStream behave like DFSOutputStream wrt - synchronization. (Sean Busbey via yliu) - HADOOP-11558. Fix dead links to doc of hadoop-tools. (Jean-Pierre Matsumoto via ozawa) @@ -1450,7 +1396,7 @@ Release 2.6.2 - UNRELEASED BUG FIXES -Release 2.6.1 - UNRELEASED +Release 2.6.1 - 2015-09-09 INCOMPATIBLE CHANGES @@ -1461,11 +1407,20 @@ Release 2.6.1 - UNRELEASED HADOOP-7139. Allow appending to existing SequenceFiles (kanaka kumar avvaru via vinayakumarb) + HADOOP-12280. Skip unit tests based on maven profile rather than + NativeCodeLoader.isNativeCodeLoaded (Masatake Iwasaki via Colin P. McCabe) + OPTIMIZATIONS + HADOOP-11238. Update the NameNode's Group Cache in the background when + possible (Chris Li via Colin P. McCabe) + + HADOOP-11506. Configuration variable expansion regex expensive for long + values. (Gera Shegalov via gera) + BUG FIXES - HADOOP-11466. FastByteComparisons: do not use UNSAFE_COMPARER on the SPARC + HADOOP-11466: FastByteComparisons: do not use UNSAFE_COMPARER on the SPARC architecture because it is slower there (Suman Somasundar via Colin P. McCabe) @@ -1474,6 +1429,51 @@ Release 2.6.1 - UNRELEASED HADOOP-11333. Fix deadlock in DomainSocketWatcher when the notification pipe is full (zhaoyunjiong via cmccabe) + HADOOP-11343. Overflow is not properly handled in caclulating final iv for + AES CTR. (Jerry Chen via wang) + + HADOOP-11368. Fix SSLFactory truststore reloader thread leak in + KMSClientProvider. (Arun Suresh via wang) + + HADOOP-11350. The size of header buffer of HttpServer is too small when + HTTPS is enabled. (Benoy Antony via wheat9) + + HADOOP-11482. Use correct UGI when KMSClientProvider is called by a proxy + user. Contributed by Arun Suresh. + + HADOOP-11295. RPC Server Reader thread can't shutdown if RPCCallQueue is + full. (Ming Ma via kihwal) + + HADOOP-11604. Prevent ConcurrentModificationException while closing domain + sockets during shutdown of DomainSocketWatcher thread. (cnauroth) + + HADOOP-11674. oneByteBuf in CryptoInputStream and CryptoOutputStream + should be non static. (Sean Busbey via yliu) + + HADOOP-11710. Make CryptoOutputStream behave like DFSOutputStream wrt + synchronization. (Sean Busbey via yliu) + + HADOOP-11812. Implement listLocatedStatus for ViewFileSystem to speed up + split calculation (gera) + + HADOOP-11730. Regression: s3n read failure recovery broken. + (Takenori Sato via stevel) + + HADOOP-11802: DomainSocketWatcher thread terminates sometimes after there + is an I/O error during requestShortCircuitShm (cmccabe) + + HADOOP-11491. HarFs incorrectly declared as requiring an authority. + (Brahma Reddy Battula via gera) + + HADOOP-8151. Error handling in snappy decompressor throws invalid + exceptions. (Matt Foley via harsh) + + HADOOP-11932. MetricsSinkAdapter may hang when being stopped. + (Brahma Reddy Battula via jianhe) + + HADOOP-11934. Use of JavaKeyStoreProvider in LdapGroupsMapping causes + infinite loop. (Larry McCay via cnauroth) + Release 2.6.0 - 2014-11-18 INCOMPATIBLE CHANGES http://git-wip-us.apache.org/repos/asf/hadoop/blob/cbf90f93/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt ---------------------------------------------------------------------- diff --git a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt index 44ec19c..961a3eb 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt +++ b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt @@ -93,8 +93,6 @@ Release 2.8.0 - UNRELEASED HDFS-8085. Move CorruptFileBlockIterator to a new hdfs.client.impl package. (szetszwo) - HDFS-8046. Allow better control of getContentSummary (kihwal) - HDFS-8076. Code cleanup for DFSInputStream: use offset instead of LocatedBlock when possible. (Zhe Zhang via wang) @@ -495,9 +493,6 @@ Release 2.8.0 - UNRELEASED HDFS-8900. Compact XAttrs to optimize memory footprint. (yliu) - HDFS-8846. Add a unit test for INotify functionality across a layout - version upgrade (Zhe Zhang via Colin P. McCabe) - HDFS-8951. Move the shortcircuit package to hdfs-client. (Mingliang Liu via wheat9) @@ -546,9 +541,6 @@ Release 2.8.0 - UNRELEASED HDFS-9012. Move o.a.h.hdfs.protocol.datatransfer.PipelineAck class to hadoop-hdfs-client module. (Mingliang Liu via wheat9) - HDFS-8384. Allow NN to startup if there are files having a lease but are not - under construction. (jing9) - HDFS-8984. Move replication queues related methods in FSNamesystem to BlockManager. (wheat9) @@ -696,9 +688,6 @@ Release 2.8.0 - UNRELEASED HDFS-8290. WebHDFS calls before namesystem initialization can cause NullPointerException. (cnauroth) - HDFS-8219. setStoragePolicy with folder behavior is different after cluster restart. - (surendra singh lilhore via Xiaoyu Yao) - HDFS-8310. Fix TestCLI.testAll "help: help for find" on Windows. (Kiran Kumar M R via Xiaoyu Yao) @@ -805,18 +794,12 @@ Release 2.8.0 - UNRELEASED HDFS-8268. Port conflict log for data node server is not sufficient (Mohammad Shahid Khan via vinayakumarb) - HDFS-8431. hdfs crypto class not found in Windows. - (Anu Engineer via cnauroth) - HDFS-8407. hdfsListDirectory must set errno to 0 on success (Masatake Iwasaki via Colin P. McCabe) HDFS-7401. Add block info to DFSInputStream' WARN message when it adds node to deadNodes (Arshad Mohammad via vinayakumarb) - HDFS-7609. Avoid retry cache collision when Standby NameNode loading edits. - (Ming Ma via jing9) - HDFS-8490. Typo in trace enabled log in ExceptionHandler of WebHDFS. (Archana T via ozawa) @@ -1018,9 +1001,6 @@ Release 2.7.2 - UNRELEASED HDFS-8867. Enable optimized block reports. (Daryn Sharp via jing9) - HDFS-8863. The remaining space check in BlockPlacementPolicyDefault is - flawed. (Kihwal Lee via yliu) - HDFS-8891. HDFS concat should keep srcs order. (Yong Zhang via jing9) HDFS-8995. Flaw in registration bookeeping can make DN die on reconnect. @@ -1065,15 +1045,9 @@ Release 2.7.1 - 2015-07-06 (Surendra Singh Lilhore via szetszwo) OPTIMIZATIONS - HDFS-8480. Fix performance and timeout issues in HDFS-7929 by using - hard-links to preserve old edit logs, instead of copying them. (Zhe Zhang - via Colin P. McCabe) BUG FIXES - HDFS-8127. NameNode Failover during HA upgrade can cause DataNode to - finalize upgrade. (jing9) - HDFS-8151. Always use snapshot path as source when invalid snapshot names are used for diff based distcp. (jing9) @@ -1095,9 +1069,6 @@ Release 2.7.1 - 2015-07-06 HDFS-8147. StorageGroup in Dispatcher should override equals nad hashCode. (surendra singh lilhore via szetszwo) - HDFS-8070. Pre-HDFS-7915 DFSClient cannot use short circuit on - post-HDFS-7915 DataNode (cmccabe) - HDFS-8273. FSNamesystem#Delete() should not call logSync() when holding the lock. (wheat9) @@ -1110,17 +1081,8 @@ Release 2.7.1 - 2015-07-06 HDFS-8305: HDFS INotify: the destination field of RenameOp should always end with the file name (cmccabe) - HDFS-7980. Incremental BlockReport will dramatically slow down namenode - startup. (Walter Su via szetszwo) - HDFS-8226. Non-HA rollback compatibility broken (J.Andreina via vinayakumarb) - HDFS-7894. Rolling upgrade readiness is not updated in jmx until query - command is issued. (Brahma Reddy Battula via kihwal) - - HDFS-8254. Standby namenode doesn't process DELETED_BLOCK if the add block - request is in edit log. (Rushabh S Shah via kihwal) - HDFS-7916. 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop (Rushabh S Shah via kihwal) @@ -1129,9 +1091,6 @@ Release 2.7.1 - 2015-07-06 HDFS-8405. Fix a typo in NamenodeFsck. (Takanobu Asanuma via szetszwo) - HDFS-8404. Pending block replication can get stuck using older genstamp - (Nathan Roberts via kihwal) - HDFS-8451. DFSClient probe for encryption testing interprets empty URI property for "enabled". (Steve Loughran via xyao) @@ -1183,9 +1142,6 @@ Release 2.7.0 - 2015-04-20 NEW FEATURES - HDFS-7278. Add a command that allows sysadmins to manually trigger full - block reports from a DN (cmccabe) - HDFS-6663. Admin command to track file and locations from block id. (Chen He via kihwal) @@ -1279,9 +1235,6 @@ Release 2.7.0 - 2015-04-20 HDFS-3342. SocketTimeoutException in BlockSender.sendChunks could have a better error message. (Yongjun Zhang via wang) - HDFS-7035. Make adding a new data directory to the DataNode an atomic - operation and improve error handling (Lei Xu via Colin P. McCabe) - HDFS-6917. Add an hdfs debug command to validate blocks, call recoverlease, etc. (cmccabe) @@ -1367,9 +1320,6 @@ Release 2.7.0 - 2015-04-20 HDFS-7462. Consolidate implementation of mkdirs() into a single class. (wheat9) - HDFS-7446. HDFS inotify should have the ability to determine what txid it - has read up to (cmccabe) - HDFS-6735. A minor optimization to avoid pread() be blocked by read() inside the same DFSInputStream (Lars Hofhansl via stack) @@ -1412,9 +1362,6 @@ Release 2.7.0 - 2015-04-20 HDFS-7528. Consolidate symlink-related implementation into a single class. (wheat9) - HDFS-7531. Improve the concurrent access on FsVolumeList (Lei Xu via Colin - P. McCabe) - HDFS-7373. Clean up temporary files after fsimage transfer failures. (kihwal) @@ -1432,8 +1379,6 @@ Release 2.7.0 - 2015-04-20 HADOOP-11470. Remove some uses of obsolete guava APIs from the hadoop codebase. (Sangjin Lee via Colin P. McCabe) - HDFS-7182. JMX metrics aren't accessible when NN is busy. (Ming Ma via jing9) - HDFS-7323. Move the get/setStoragePolicy commands out from dfsadmin. (jing9 via yliu) @@ -1633,9 +1578,6 @@ Release 2.7.0 - 2015-04-20 HDFS-7410. Support CreateFlags with append() to support hsync() for appending streams (Vinayakumar B via Colin P. McCabe) - HDFS-7742. Favoring decommissioning node for replication can cause a block - to stay underreplicated for long periods (Nathan Roberts via kihwal) - HDFS-8008. Support client-side back off when the datanodes are congested. (wheat9) @@ -1786,9 +1728,6 @@ Release 2.7.0 - 2015-04-20 HDFS-7431. log message for InvalidMagicNumberException may be incorrect. (Yi Liu via cnauroth) - HDFS-7552. Change FsVolumeList toString() to fix - TestDataNodeVolumeFailureToleration (Liang Xie via Colin P. McCabe) - HDFS-7557. Fix spacing for a few keys in DFSConfigKeys.java (Colin P.McCabe) @@ -1815,21 +1754,9 @@ Release 2.7.0 - 2015-04-20 HDFS-7589. Break the dependency between libnative_mini_dfs and libhdfs. (Zhanwei Wang via cnauroth) - HDFS-7579. Improve log reporting during block report rpc failure. - (Charles Lamb via cnauroth) - - HDFS-7596. NameNode should prune dead storages from storageMap. - (Arpit Agarwal via cnauroth) - - HDFS-7533. Datanode sometimes does not shutdown on receiving upgrade - shutdown command (Eric Payne via kihwal) - HDFS-5445. PacketReceiver populates the packetLen field in PacketHeader incorrectly (Jonathan Mace via Colin P. McCabe) - HDFS-7470. SecondaryNameNode need twice memory when calling - reloadFromImageFile. (zhaoyunjiong via cnauroth) - HDFS-7585. Get TestEnhancedByteBufferAccess working on CPU architectures with page sizes other than 4096 (Sam Liu via Colin P. McCabe) @@ -1847,15 +1774,9 @@ Release 2.7.0 - 2015-04-20 HDFS-7496. Fix FsVolume removal race conditions on the DataNode by reference-counting the volume instances (lei via cmccabe) - HDFS-7610. Fix removal of dynamically added DN volumes (Lei (Eddy) Xu via - Colin P. McCabe) - HDFS-7548. Corrupt block reporting delayed until datablock scanner thread detects it (Rushabh Shah via kihwal) - HDFS-7575. Upgrade should generate a unique storage ID for each - volume. (Arpit Agarwal) - HDFS-3519. Checkpoint upload may interfere with a concurrent saveNamespace. (Ming Ma via cnauroth) @@ -1896,9 +1817,6 @@ Release 2.7.0 - 2015-04-20 HDFS-6651. Deletion failure can leak inodes permanently. (Jing Zhao via wheat9) - HDFS-7707. Edit log corruption due to delayed block removal again. - (Yongjun Zhang via kihwal) - HDFS-7734. Class cast exception in NameNode#main. (yliu via wang) HDFS-7719. BlockPoolSliceStorage#removeVolumes fails to remove some @@ -1925,9 +1843,6 @@ Release 2.7.0 - 2015-04-20 HDFS-7718. Store KeyProvider in ClientContext to avoid leaking key provider threads when using FileContext (Arun Suresh via Colin P. McCabe) - HDFS-7714. Simultaneous restart of HA NameNodes and DataNode can cause - DataNode to register successfully with only one NameNode.(vinayakumarb) - HDFS-7769. TestHDFSCLI should not create files in hdfs project root dir. (szetszwo) @@ -1964,15 +1879,9 @@ Release 2.7.0 - 2015-04-20 HDFS-6662. WebHDFS cannot open a file if its path contains "%". (Gerson Carlos via wheat9) - HDFS-7788. Post-2.6 namenode may not start up with an image containing - inodes created with an old release. (Rushabh Shah via kihwal) - HDFS-7814. Fix usage string of storageType parameter for "dfsadmin -setSpaceQuota/clrSpaceQuota". (Xiaoyu Yao via cnauroth) - HDFS-7009. Active NN and standby NN have different live nodes. - (Ming Ma via cnauroth) - HDFS-7807. libhdfs htable.c: fix htable resizing, add unit test (cmccabe) HDFS-7805. NameNode recovery prompt should be printed on console (Surendra @@ -1984,9 +1893,6 @@ Release 2.7.0 - 2015-04-20 HDFS-7831. Fix the starting index and end condition of the loop in FileDiffList.findEarlierSnapshotBlocks(). (Konstantin Shvachko via jing9) - HDFS-7763. fix zkfc hung issue due to not catching exception in a corner - case. (Liang Xie via wang) - HDFS-7843. A truncated file is corrupted after rollback from a rolling upgrade. (szetszwo) @@ -1999,9 +1905,6 @@ Release 2.7.0 - 2015-04-20 HDFS-7785. Improve diagnostics information for HttpPutFailedException. (Chengbing Liu via wheat9) - HDFS-7871. NameNodeEditLogRoller can keep printing "Swallowing exception" - message. (jing9) - HDFS-7757. Misleading error messages in FSImage.java. (Brahma Reddy Battula via Arpit Agarwal) @@ -2019,15 +1922,9 @@ Release 2.7.0 - 2015-04-20 HDFS-7434. DatanodeID hashCode should not be mutable. (daryn via kihwal) - HDFS-7885. Datanode should not trust the generation stamp provided by - client. (Tsz Wo Nicholas Sze via jing9) - HDFS-7818. OffsetParam should return the default value instead of throwing NPE when the value is unspecified. (Eric Payne via wheat9) - HDFS-7830. DataNode does not release the volume lock when adding a volume - fails. (Lei Xu via Colin P. Mccabe) - HDFS-7880. Remove the tests for legacy Web UI in branch-2. (Brahma Reddy Battula via aajisaka) @@ -2040,9 +1937,6 @@ Release 2.7.0 - 2015-04-20 HDFS-7903. Cannot recover block after truncate and delete snapshot. (Plamen Jeliazkov via shv) - HDFS-7915. The DataNode can sometimes allocate a ShortCircuitShm slot and - fail to tell the DFSClient about it because of a network error (cmccabe) - HDFS-7886. Fix TestFileTruncate falures. (Plamen Jeliazkov and shv) HDFS-7946. TestDataNodeVolumeFailureReporting NPE on Windows. (Xiaoyu Yao @@ -2076,18 +1970,10 @@ Release 2.7.0 - 2015-04-20 HDFS-7943. Append cannot handle the last block with length greater than the preferred block size. (jing9) - HDFS-7587. Edit log corruption can happen if append fails with a quota - violation. (jing9) - - HDFS-7929. inotify unable fetch pre-upgrade edit log segments once upgrade - starts (Zhe Zhang via Colin P. McCabe) - HDFS-7816. Unable to open webhdfs paths with "+". (wheat9 via kihwal) HDFS-7932. Speed up the shutdown of datanode during rolling upgrade.(kihwal) - HDFS-7930. commitBlockSynchronization() does not remove locations. (yliu) - HDFS-7957. Truncate should verify quota before making changes. (jing9) HDFS-6841. Use Time.monotonicNow() wherever applicable instead of Time.now() @@ -2095,16 +1981,6 @@ Release 2.7.0 - 2015-04-20 HDFS-7942. NFS: support regexp grouping in nfs.exports.allowed.hosts (brandonli) - HDFS-7881. TestHftpFileSystem#testSeek fails in branch-2. - (Brahma Reddy Battula via aajisaka) - - HDFS-7884. Fix NullPointerException in BlockSender when the generation stamp - provided by the client is larger than the one stored in the datanode. - (Brahma Reddy Battula via szetszwo) - - HDFS-7960. The full block report should prune zombie storages even if - they're not empty. (cmccabe and Eddy Xu via wang) - HDFS-7961. Trigger full block report after hot swapping disk. (Eddy Xu via wang) HDFS-7977. NFS couldn't take percentile intervals (brandonli) @@ -2137,15 +2013,9 @@ Release 2.7.0 - 2015-04-20 HDFS-8051. FsVolumeList#addVolume should release volume reference if not put it into BlockScanner. (Lei (Eddy) Xu via Colin P. McCabe) - HDFS-7999. FsDatasetImpl#createTemporary sometimes holds the FSDatasetImpl - lock for a very long time (sinago via cmccabe) - HDFS-8038. PBImageDelimitedTextWriter#getEntry output HDFS path in platform-specific format. (Xiaoyu Yao via cnauroth) - HDFS-8072. Reserved RBW space is not released if client terminates while - writing block. (Arpit Agarwal) - HDFS-8025. Addendum fix for HDFS-3087 Decomissioning on NN restart can complete without blocks being replicated. (Ming Ma via wang) @@ -2190,16 +2060,42 @@ Release 2.6.2 - UNRELEASED BUG FIXES -Release 2.6.1 - UNRELEASED +Release 2.6.1 - 2015-09-09 INCOMPATIBLE CHANGES NEW FEATURES + HDFS-7278. Add a command that allows sysadmins to manually trigger full + block reports from a DN (cmccabe) + IMPROVEMENTS + HDFS-7035. Make adding a new data directory to the DataNode an atomic + operation and improve error handling (Lei Xu via Colin P. McCabe) + + HDFS-7531. Improve the concurrent access on FsVolumeList (Lei Xu via Colin + P. McCabe) + + HDFS-7579. Improve log reporting during block report rpc failure. + (Charles Lamb via cnauroth) + + HDFS-7182. JMX metrics aren't accessible when NN is busy. (Ming Ma via jing9) + + HDFS-7596. NameNode should prune dead storages from storageMap. + (Arpit Agarwal via cnauroth) + + HDFS-8046. Allow better control of getContentSummary (kihwal) + + HDFS-8384. Allow NN to startup if there are files having a lease but are not + under construction. (jing9) + OPTIMIZATIONS + HDFS-8480. Fix performance and timeout issues in HDFS-7929 by using + hard-links to preserve old edit logs, instead of copying them. (Zhe Zhang + via Colin P. McCabe) + BUG FIXES HDFS-7425. NameNode block deletion logging uses incorrect appender. @@ -2223,8 +2119,7 @@ Release 2.6.1 - UNRELEASED HDFS-7733. NFS: readdir/readdirplus return null directory attribute on failure. (Arpit Agarwal) - HDFS-8486. DN startup may cause severe data loss (Daryn Sharp via Colin P. - McCabe) + HDFS-8486. DN startup may cause severe data loss. (daryn via cmccabe) HDFS-7213. processIncrementalBlockReport performance degradation. (Eric Payne via kihwal) @@ -2238,9 +2133,111 @@ Release 2.6.1 - UNRELEASED HDFS-7225. Remove stale block invalidation work when DN re-registers with different UUID. (Zhe Zhang and Andrew Wang) + HDFS-7533. Datanode sometimes does not shutdown on receiving upgrade + shutdown command (Eric Payne via kihwal) + + HDFS-7575. Upgrade should generate a unique storage ID for each + volume. (Arpit Agarwal) + + HDFS-7707. Edit log corruption due to delayed block removal again. + (Yongjun Zhang via kihwal) + + HDFS-7714. Simultaneous restart of HA NameNodes and DataNode can cause + DataNode to register successfully with only one NameNode.(vinayakumarb) + + HDFS-7788. Post-2.6 namenode may not start up with an image containing + inodes created with an old release. (Rushabh Shah via kihwal) + + HDFS-7009. Active NN and standby NN have different live nodes. + (Ming Ma via cnauroth) + + HDFS-7763. fix zkfc hung issue due to not catching exception in a corner + case. (Liang Xie via wang) + + HDFS-7871. NameNodeEditLogRoller can keep printing "Swallowing exception" + message. (jing9) + + HDFS-7885. Datanode should not trust the generation stamp provided by + client. (Tsz Wo Nicholas Sze via jing9) + + HDFS-7610. Fix removal of dynamically added DN volumes (Lei (Eddy) Xu via + Colin P. McCabe) + + HDFS-7830. DataNode does not release the volume lock when adding a volume + fails. (Lei Xu via Colin P. Mccabe) + + HDFS-7587. Edit log corruption can happen if append fails with a quota + violation. (jing9) + + HDFS-7929. inotify unable fetch pre-upgrade edit log segments once upgrade + starts (Zhe Zhang via Colin P. McCabe) + + HDFS-7930. commitBlockSynchronization() does not remove locations. (yliu) + + HDFS-7884. Fix NullPointerException in BlockSender when the generation stamp + provided by the client is larger than the one stored in the datanode. + (Brahma Reddy Battula via szetszwo) + + HDFS-7960. The full block report should prune zombie storages even if + they're not empty. (cmccabe and Eddy Xu via wang) + + HDFS-7742. Favoring decommissioning node for replication can cause a block + to stay underreplicated for long periods (Nathan Roberts via kihwal) + + HDFS-7999. FsDatasetImpl#createTemporary sometimes holds the FSDatasetImpl + lock for a very long time (sinago via cmccabe) + + HDFS-8072. Reserved RBW space is not released if client terminates while + writing block. (Arpit Agarwal) + + HDFS-8127. NameNode Failover during HA upgrade can cause DataNode to + finalize upgrade. (jing9) + + HDFS-7915. The DataNode can sometimes allocate a ShortCircuitShm slot and + fail to tell the DFSClient about it because of a network error (cmccabe) + + HDFS-8070. Pre-HDFS-7915 DFSClient cannot use short circuit on + post-HDFS-7915 DataNode (cmccabe) + + HDFS-8219. setStoragePolicy with folder behavior is different after cluster restart. + (surendra singh lilhore via Xiaoyu Yao) + + HDFS-7894. Rolling upgrade readiness is not updated in jmx until query + command is issued. (Brahma Reddy Battula via kihwal) + + HDFS-8254. Standby namenode doesn't process DELETED_BLOCK if the add block + request is in edit log. (Rushabh S Shah via kihwal) + + HDFS-8404. Pending block replication can get stuck using older genstamp + (Nathan Roberts via kihwal) + + HDFS-8431. hdfs crypto class not found in Windows. + (Anu Engineer via cnauroth) + + HDFS-7609. Avoid retry cache collision when Standby NameNode loading edits. + (Ming Ma via jing9) + HDFS-8270. create() always retried with hardcoded timeout when file already exists with open lease (J.Andreina via vinayakumarb) + HDFS-7980. Incremental BlockReport will dramatically slow down namenode + startup. (Walter Su via szetszwo) + + HDFS-7446. HDFS inotify should have the ability to determine what txid it + has read up to (cmccabe) + + HDFS-8846. Add a unit test for INotify functionality across a layout + version upgrade (Zhe Zhang via Colin P. McCabe) + + HDFS-7470. SecondaryNameNode need twice memory when calling + reloadFromImageFile. (zhaoyunjiong via cnauroth) + + HDFS-8863. The remaining space check in BlockPlacementPolicyDefault is + flawed. (Kihwal Lee via yliu) + + HDFS-7552. Change FsVolumeList toString() to fix + TestDataNodeVolumeFailureToleration (Liang Xie via Colin P. McCabe) + Release 2.6.0 - 2014-11-18 INCOMPATIBLE CHANGES http://git-wip-us.apache.org/repos/asf/hadoop/blob/cbf90f93/hadoop-mapreduce-project/CHANGES.txt ---------------------------------------------------------------------- diff --git a/hadoop-mapreduce-project/CHANGES.txt b/hadoop-mapreduce-project/CHANGES.txt index ad4c6c5..e772078 100644 --- a/hadoop-mapreduce-project/CHANGES.txt +++ b/hadoop-mapreduce-project/CHANGES.txt @@ -176,9 +176,6 @@ Release 2.8.0 - UNRELEASED MAPREDUCE-6349. Fix typo in property org.apache.hadoop.mapreduce. lib.chain.Chain.REDUCER_INPUT_VALUE_CLASS. (Ray Chiang via ozawa) - MAPREDUCE-5649. Reduce cannot use more than 2G memory for the final merge - (Gera Shegalov via jlowe) - MAPREDUCE-6165. [JDK8] TestCombineFileInputFormat failed on JDK8. (Akira AJISAKA via ozawa) @@ -336,14 +333,6 @@ Release 2.7.1 - 2015-07-06 BUG FIXES - MAPREDUCE-6300. Task list sort by task id broken. (Siqi Li via aajisaka) - - MAPREDUCE-6238. MR2 can't run local jobs with -libjars command options - which is a regression from MR1 (zxu via rkanter) - - MAPREDUCE-6324. Fixed MapReduce uber jobs to not fail the udpate of AM-RM - tokens when they roll-over. (Jason Lowe via vinodkv) - MAPREDUCE-6252. JobHistoryServer should not fail when encountering a missing directory. (Craig Welch via devaraj) @@ -360,10 +349,6 @@ Release 2.7.1 - 2015-07-06 that they don't fail on history-server backed by DFSes with not so strong guarantees. (Craig Welch via vinodkv) - MAPREDUCE-6361. NPE issue in shuffle caused by concurrent issue between - copySucceeded() in one thread and copyFailed() in another thread on the - same host. (Junping Du via ozawa) - MAPREDUCE-6387. Serialize the recently added Task#encryptedSpillKey field at the end. (Arun Suresh via kasha) @@ -445,9 +430,6 @@ Release 2.7.0 - 2015-04-20 MAPREDUCE-6248. Exposed the internal MapReduce job's information as a public API in DistCp. (Jing Zhao via vinodkv) - MAPREDUCE-6267. Refactor JobSubmitter#copyAndConfigureFiles into it's own - class. (Chris Trezzo via kasha) - MAPREDUCE-6263. Configurable timeout between YARNRunner terminate the application and forcefully kill. (Eric Payne via junping_du) @@ -497,9 +479,6 @@ Release 2.7.0 - 2015-04-20 MAPREDUCE-4879. TeraOutputFormat may overwrite an existing output directory. (gera) - MAPREDUCE-6166. Reducers do not validate checksum of map outputs when - fetching directly to disk. (Eric Payne via gera) - MAPREDUCE-6045. need close the DataInputStream after open it in TestMapReduce.java (zxu via rkanter) @@ -509,9 +488,6 @@ Release 2.7.0 - 2015-04-20 MAPREDUCE-3283. mapred classpath CLI does not display the complete classpath (Varun Saxena via cnauroth) - MAPREDUCE-6230. Fixed RMContainerAllocator to update the new AMRMToken - service name properly. (Jason Lowe via jianhe) - MAPREDUCE-6231. Grep example job is not working on a fully-distributed cluster. (aajisaka) @@ -559,9 +535,6 @@ Release 2.7.0 - 2015-04-20 MAPREDUCE-6285. ClientServiceDelegate should not retry upon AuthenticationException. (Jonathan Eagles via ozawa) - MAPREDUCE-6303. Read timeout when retrying a fetch error can be fatal - to a reducer. (Jason Lowe via junping_du) - Release 2.6.2 - UNRELEASED INCOMPATIBLE CHANGES @@ -578,7 +551,7 @@ Release 2.6.2 - UNRELEASED cache files so that child processes running hadoop scripts can access these files. (Junping Du via vinodkv) -Release 2.6.1 - UNRELEASED +Release 2.6.1 - 2015-09-09 INCOMPATIBLE CHANGES @@ -593,6 +566,33 @@ Release 2.6.1 - UNRELEASED MAPREDUCE-6237. Multiple mappers with DBInputFormat don't work because of reusing conections. (Kannan Rajah via ozawa) + MAPREDUCE-6166. Reducers do not validate checksum of map outputs when + fetching directly to disk. (Eric Payne via gera) + + MAPREDUCE-6230. Fixed RMContainerAllocator to update the new AMRMToken + service name properly. (Jason Lowe via jianhe) + + MAPREDUCE-6303. Read timeout when retrying a fetch error can be fatal + to a reducer. (Jason Lowe via junping_du) + + MAPREDUCE-6300. Task list sort by task id broken. (Siqi Li via aajisaka) + + MAPREDUCE-6267. Refactor JobSubmitter#copyAndConfigureFiles into it's own + class. (Chris Trezzo via kasha) + + MAPREDUCE-6238. MR2 can't run local jobs with -libjars command options + which is a regression from MR1 (zxu via rkanter) + + MAPREDUCE-6324. Fixed MapReduce uber jobs to not fail the udpate of AM-RM + tokens when they roll-over. (Jason Lowe via vinodkv) + + MAPREDUCE-5649. Reduce cannot use more than 2G memory for the final merge + (Gera Shegalov via jlowe) + + MAPREDUCE-6361. NPE issue in shuffle caused by concurrent issue between + copySucceeded() in one thread and copyFailed() in another thread on the + same host. (Junping Du via ozawa) + Release 2.6.0 - 2014-11-18 INCOMPATIBLE CHANGES http://git-wip-us.apache.org/repos/asf/hadoop/blob/cbf90f93/hadoop-yarn-project/CHANGES.txt ---------------------------------------------------------------------- diff --git a/hadoop-yarn-project/CHANGES.txt b/hadoop-yarn-project/CHANGES.txt index 62cda3b..9975b7b 100644 --- a/hadoop-yarn-project/CHANGES.txt +++ b/hadoop-yarn-project/CHANGES.txt @@ -286,9 +286,6 @@ Release 2.8.0 - UNRELEASED YARN-3581. Deprecate -directlyAccessNodeLabelStore in RMAdminCLI. (Naganarasimha G R via wangda) - YARN-3700. Made generic history service load a number of latest applications - according to the parameter or the configuration. (Xuan Gong via zjshen) - YARN-3722. Merge multiple TestWebAppUtils into o.a.h.yarn.webapp.util.TestWebAppUtils. (Masatake Iwasaki via devaraj) @@ -446,9 +443,6 @@ Release 2.8.0 - UNRELEASED YARN-3459. Fix failiure of TestLog4jWarningErrorMetricsAppender. (Varun Vasudev via wangda) - YARN-2890. MiniYarnCluster should turn on timeline service if - configured to do so. (Mit Desai via hitesh) - YARN-3266. RMContext#inactiveNodes should have NodeId as map key. (Chengbing Liu via jianhe) @@ -745,9 +739,6 @@ Release 2.8.0 - UNRELEASED YARN-4005. Completed container whose app is finished is possibly not removed from NMStateStore. (Jun Gong via jianhe) - YARN-4047. ClientRMService getApplications has high scheduler lock contention. - (Jason Lowe via jianhe) - YARN-3987. Send AM container completed msg to NM once AM finishes. (sandflee via jianhe) @@ -792,9 +783,6 @@ Release 2.7.2 - UNRELEASED YARN-3967. Fetch the application report from the AHS if the RM does not know about it. (Mit Desai via xgong) - YARN-3978. Configurably turn off the saving of container info in Generic AHS - (Eric Payne via jeagles) - YARN-4092. Fixed UI redirection to print useful messages when both RMs are in standby mode. (Xuan Gong via jianhe) @@ -828,11 +816,6 @@ Release 2.7.2 - UNRELEASED YARN-3925. ContainerLogsUtils#getContainerLogFile fails to read container log files from full disks. (zhihai xu via jlowe) - YARN-3990. AsyncDispatcher may overloaded with RMAppNodeUpdateEvent when - Node is connected/disconnected (Bibin A Chundatt via jlowe) - - YARN-3999. RM hangs on draing events. (Jian He via xgong) - YARN-3857: Memory leak in ResourceManager with SIMPLE mode. (mujunchao via zxu) @@ -881,9 +864,6 @@ Release 2.7.1 - 2015-07-06 BUG FIXES - YARN-3487. CapacityScheduler scheduler lock obtained unnecessarily when - calling getQueue (Jason Lowe via wangda) - YARN-3462. Patches applied for YARN-2424 are inconsistent between trunk and branch-2. (Naganarasimha G R via harsh) @@ -910,9 +890,6 @@ Release 2.7.1 - 2015-07-06 YARN-3516. killing ContainerLocalizer action doesn't take effect when private localizer receives FETCH_FAILURE status.(zhihai xu via xgong) - YARN-3464. Race condition in LocalizerRunner kills localizer before - localizing all resources. (Zhihai Xu via kasha) - YARN-3485. FairScheduler headroom calculation doesn't consider maxResources for Fifo and FairShare policies. (kasha) @@ -937,10 +914,6 @@ Release 2.7.1 - 2015-07-06 YARN-3434. Interaction between reservations and userlimit can result in significant ULF violation (tgraves) - YARN-3493. RM fails to come up with error "Failed to load/recover state" - when mem settings are changed. (Jian He via wangda) - -<<<<<<< HEAD YARN-3626. On Windows localized resources are not moved to the front of the classpath when they should be. (Craig Welch via xgong) @@ -950,20 +923,11 @@ Release 2.7.1 - 2015-07-06 YARN-3537. NPE when NodeManager.serviceInit fails and stopRecoveryStore invoked (Brahma Reddy Battula via jlowe) - YARN-3641. NodeManager: stopRecoveryStore() shouldn't be skipped when - exceptions happen in stopping NM's sub-services. (Junping Du via jlowe) - - YARN-3526. ApplicationMaster tracking URL is incorrectly redirected - on a QJM cluster. (Weiwei Yang via xgong) - YARN-3601. Fix UT TestRMFailover.testRMWebAppRedirect. (Weiwei Yang via xgong) YARN-3677. Fix findbugs warnings in yarn-server-resourcemanager. (Vinod Kumar Vavilapalli via ozawa) - YARN-2918. RM should not fail on startup if queue's configured labels do - not exist in cluster-node-labels. (Wangda Tan via jianhe) - YARN-3681. yarn cmd says "could not find main class 'queue'" in windows. (Craig Welch and Varun Saxena via xgong) @@ -988,18 +952,6 @@ Release 2.7.1 - 2015-07-06 YARN-3686. CapacityScheduler should trim default_node_label_expression. (Sunil G via wangda) - YARN-2900. Application (Attempt and Container) Not Found in AHS results - in Internal Server Error (500). (Zhijie Shen and Mit Desai via xgong) - - YARN-3725. App submission via REST API is broken in secure mode due to - Timeline DT service address is empty. (Zhijie Shen via wangda) - - YARN-3585. NodeManager cannot exit on SHUTDOWN event triggered and NM - recovery is enabled (Rohith Sharmaks via jlowe) - - YARN-3733. Fix DominantRC#compare() does not work as expected if - cluster resource is empty. (Rohith Sharmaks via wangda) - YARN-3764. CapacityScheduler should forbid moving LeafQueue from one parent to another. (Wangda Tan via jianhe) @@ -1012,13 +964,6 @@ Release 2.7.1 - 2015-07-06 YARN-3809. Failed to launch new attempts because ApplicationMasterLauncher's threads all hang (Jun Gong via jlowe) - YARN-3832. Resource Localization fails on a cluster due to existing cache - directories (Brahma Reddy Battula via jlowe) - - YARN-3850. NM fails to read files from full disks which can lead to - container logs being lost and other issues (Varun Saxena via jlowe) - - Release 2.7.0 - 2015-04-20 INCOMPATIBLE CHANGES @@ -1068,9 +1013,6 @@ Release 2.7.0 - 2015-04-20 YARN-2994. Document work-preserving RM restart. (Jian He via ozawa) - YARN-3249. Add a 'kill application' button to Resource Manager's Web UI. - (Ryu Kobayashi via ozawa) - YARN-2786. Created a yarn cluster CLI and seeded with one command for listing node-labels collection. (Wangda Tan via vinodkv) @@ -1141,8 +1083,6 @@ Release 2.7.0 - 2015-04-20 YARN-1156. Enhance NodeManager AllocatedGB and AvailableGB metrics for aggregation of decimal values. (Tsuyoshi OZAWA via junping_du) - YARN-2301. Improved yarn container command. (Naganarasimha G R via jianhe) - YARN-2056. Disable preemption at Queue level (Eric Payne via jlowe) YARN-2762. Fixed RMAdminCLI to trim and check node-label related arguments @@ -1196,12 +1136,6 @@ Release 2.7.0 - 2015-04-20 YARN-2800. Remove MemoryNodeLabelsStore and add a way to enable/disable node labels feature. (Wangda Tan via ozawa) - YARN-3024. LocalizerRunner should give DIE action when all resources are - localized. (Chengbing Liu via xgong) - - YARN-3092. Created a common ResourceUsage class to track labeled resource - usages in Capacity Scheduler. (Wangda Tan via jianhe) - YARN-3086. Make NodeManager memory configurable in MiniYARNCluster. (Robert Metzger via ozawa) @@ -1227,9 +1161,6 @@ Release 2.7.0 - 2015-04-20 YARN-3022. Expose Container resource information from NodeManager for monitoring (adhoot via ranter) - YARN-3098. Created common QueueCapacities class in Capacity Scheduler to - track capacities-by-labels of queues. (Wangda Tan via jianhe) - YARN-3075. NodeLabelsManager implementation to retrieve label to node mapping (Varun Saxena via wangda) @@ -1270,9 +1201,6 @@ Release 2.7.0 - 2015-04-20 YARN-2079. Recover NonAggregatingLogHandler state upon nodemanager restart. (Jason Lowe via junping_du) - YARN-3124. Fixed CS LeafQueue/ParentQueue to use QueueCapacities to track - capacities-by-label. (Wangda Tan via jianhe) - YARN-3158. Correct log messages in ResourceTrackerService. (Varun Saxena via xgong) @@ -1297,8 +1225,6 @@ Release 2.7.0 - 2015-04-20 YARN-2799. Cleanup TestLogAggregationService based on the change in YARN-90. (Zhihai Xu via junping_du) - YARN-3230. Clarify application states on the web UI. (Jian He via wangda) - YARN-3237. AppLogAggregatorImpl fails to log error cause. (Rushabh S Shah via xgong) @@ -1335,9 +1261,6 @@ Release 2.7.0 - 2015-04-20 YARN-3122. Metrics for container's actual CPU usage. (Anubhav Dhoot via kasha) - YARN-1809. Synchronize RM and TimeLineServer Web-UIs. (Zhijie Shen and - Xuan Gong via jianhe) - YARN-2190. Added CPU and memory limit options to the default container executor for Windows containers. (Chuan Liu via jianhe) @@ -1372,56 +1295,26 @@ Release 2.7.0 - 2015-04-20 YARN-2713. "RM Home" link in NM should point to one of the RMs in an HA setup. (kasha) - YARN-2766. Made ApplicationHistoryManager return a sorted list of apps, - attempts and containers. (Robert Kanter via zjshen) - - YARN-2856. Fixed RMAppImpl to handle ATTEMPT_KILLED event at ACCEPTED state - on app recovery. (Rohith Sharmaks via jianhe) - YARN-2857. ConcurrentModificationException in ContainerLogAppender (Mohammad Kamrul Islam via jlowe) - YARN-2816. NM fail to start with NPE during container recovery (Zhihai Xu - via jlowe) - YARN-2432. RMStateStore should process the pending events before close. (Varun Saxena via jianhe) YARN-1703. Fixed ResourceManager web-proxy to close connections correctly. (Rohith Sharma via vinodkv) - YARN-2414. RM web UI: app page will crash if app is failed before any - attempt has been created (Wangda Tan via jlowe) - YARN-2870. Updated the command to run the timeline server in the document. (Masatake Iwasaki via zjshen) YARN-2878. Fix DockerContainerExecutor.apt.vm formatting. (Abin Shahab via jianhe) - YARN-2865. Fixed RM to always create a new RMContext when transtions from - StandBy to Active. (Rohith Sharmaks via jianhe) - YARN-2315. FairScheduler: Set current capacity in addition to capacity. (Zhihai Xu via kasha) - YARN-1984. LeveldbTimelineStore does not handle db exceptions properly - (Varun Saxena via jlowe) - YARN-2697. Remove useless RMAuthenticationHandler. (Haosong Huang via zjshen) - YARN-2906. CapacitySchedulerPage shows HTML tags for a queue's Active Users. - (Jason Lowe via jianhe) - - YARN-2905. AggregatedLogsBlock page can infinitely loop if the aggregated - log file is corrupted (Varun Saxena via jlowe) - - YARN-2894. Fixed a bug regarding application view acl when RM fails over. - (Rohith Sharmaks via jianhe) - - YARN-2874. Dead lock in "DelegationTokenRenewer" which blocks RM to execute - any further apps. (Naganarasimha G R via kasha) - YARN-2461. Fix PROCFS_USE_SMAPS_BASED_RSS_ENABLED property in YarnConfiguration. (rchiang via rkanter) @@ -1434,18 +1327,12 @@ Release 2.7.0 - 2015-04-20 YARN-2931. PublicLocalizer may fail until directory is initialized by LocalizeRunner. (Anubhav Dhoot via kasha) - YARN-2910. FSLeafQueue can throw ConcurrentModificationException. - (Wilfred Spiegelenburg via kasha) - YARN-2930. Fixed TestRMRestart#testRMRestartRecoveringNodeLabelManager intermittent failure. (Wangda Tan via jianhe) YARN-2924. Fixed RMAdminCLI to not convert node labels to lower case. (Wangda Tan via jianhe) - YARN-2917. Fixed potential deadlock when system.exit is called in AsyncDispatcher - (Rohith Sharmaks via jianhe) - YARN-2243. Order of arguments for Preconditions.checkNotNull() is wrong in SchedulerApplicationAttempt ctor. (devaraj) @@ -1463,15 +1350,9 @@ Release 2.7.0 - 2015-04-20 YARN-2944. InMemorySCMStore can not be instantiated with ReflectionUtils#newInstance. (Chris Trezzo via kasha) - YARN-2964. RM prematurely cancels tokens for jobs that submit jobs (oozie) - (Jian He via jlowe) - YARN-2675. containersKilled metrics is not updated when the container is killed during localization. (Zhihai Xu via kasha) - YARN-2952. Fixed incorrect version check in StateStore. (Rohith Sharmaks - via jianhe) - YARN-2975. FSLeafQueue app lists are accessed without required locks. (kasha) YARN-2977. Fixed intermittent TestNMClient failure. @@ -1479,12 +1360,6 @@ Release 2.7.0 - 2015-04-20 YARN-2939. Fix new findbugs warnings in hadoop-yarn-common. (Li Lu via junping_du) - YARN-2920. Changed CapacityScheduler to kill containers on nodes where - node labels are changed. (Wangda Tan via jianhe) - - YARN-2340. Fixed NPE when queue is stopped during RM restart. - (Rohith Sharmaks via jianhe) - YARN-2940. Fix new findbugs warnings in rest of the hadoop-yarn components. (Li Lu via junping_du) @@ -1496,9 +1371,6 @@ Release 2.7.0 - 2015-04-20 YARN-2988. Graph#save() may leak file descriptors. (Ted Yu via ozawa) - YARN-2992. ZKRMStateStore crashes due to session expiry. (Karthik Kambatla - via jianhe) - YARN-2938. Fixed new findbugs warnings in hadoop-yarn-resourcemanager and hadoop-yarn-applicationhistoryservice. (Varun Saxena via zjshen) @@ -1508,15 +1380,9 @@ Release 2.7.0 - 2015-04-20 YARN-2991. Fixed DrainDispatcher to reuse the draining code path in AsyncDispatcher. (Rohith Sharmaks via zjshen) - YARN-2922. ConcurrentModificationException in CapacityScheduler's LeafQueue. - (Rohith Sharmaks via ozawa) - YARN-2958. Made RMStateStore not update the last sequence number when updating the delegation token. (Varun Saxena via zjshen) - YARN-2978. Fixed potential NPE while getting queue info. (Varun Saxena via - jianhe) - YARN-2230. Fixed few configs description in yarn-default.xml. (Vijay Bhat via jianhe) @@ -1526,18 +1392,12 @@ Release 2.7.0 - 2015-04-20 YARN-2936. Changed YARNDelegationTokenIdentifier to set proto fields on getProto method. (Varun Saxena via jianhe) - YARN-2997. Fixed NodeStatusUpdater to not send alreay-sent completed - container statuses on heartbeat. (Chengbing Liu via jianhe) - YARN-3014. Replaces labels on a host should update all NM's labels on that host. (Wangda Tan via jianhe) YARN-3027. Scheduler should use totalAvailable resource from node instead of availableResource for maxAllocation. (adhoot via rkanter) - YARN-2637. Fixed max-am-resource-percent calculation in CapacityScheduler - when activating applications. (Craig Welch via jianhe) - YARN-2861. Fixed Timeline DT secret manager to not reuse RM's configs. (Zhijie Shen via jianhe) @@ -1568,21 +1428,12 @@ Release 2.7.0 - 2015-04-20 YARN-3088. LinuxContainerExecutor.deleteAsUser can throw NPE if native executor returns an error (Eric Payne via jlowe) - YARN-3011. Possible IllegalArgumentException in ResourceLocalizationService - might lead NM to crash. (Varun Saxena via jianhe) - - YARN-3103. AMRMClientImpl does not update AMRM token properly. (Jason Lowe - via jianhe) - YARN-3079. Scheduler should also update maximumAllocation when updateNodeResource. (Zhihai Xu via wangda) YARN-3029. FSDownload.unpack() uses local locale for FS case conversion, may not work everywhere. (Varun Saxena via ozawa) - YARN-3099. Capacity Scheduler LeafQueue/ParentQueue should use ResourceUsage - to track used-resources-by-label.(Wangda Tan via jianhe) - YARN-3077. Fixed RM to create zk root path recursively. (Chun Chen via jianhe) YARN-3113. Release audit warning for Sorting icons.psd. (stevel via kihwal) @@ -1608,18 +1459,12 @@ Release 2.7.0 - 2015-04-20 YARN-1537. Fix race condition in TestLocalResourcesTrackerImpl.testLocalResourceCache. (xgong via acmurthy) - YARN-2694. Ensure only single node label specified in ResourceRequest. - (Wangda Tan via jianhe) - YARN-3089. LinuxContainerExecutor does not handle file arguments to deleteAsUser (Eric Payne via jlowe) YARN-3143. RM Apps REST API can return NPE or entries missing id and other fields (jlowe) - YARN-3094. Reset timer for liveness monitors after RM recovery. (Jun Gong - via jianhe) - YARN-2971. RM uses conf instead of token service address to renew timeline delegation tokens (jeagles) @@ -1629,10 +1474,6 @@ Release 2.7.0 - 2015-04-20 YARN-2809. Implement workaround for linux kernel panic when removing cgroup (Nathan Roberts via jlowe) - YARN-2246. Made the proxy tracking URL always be - http(s)://proxy addr:port/proxy/<appId> to avoid duplicate sections. (Devaraj - K via zjshen) - YARN-3160. Fix non-atomic operation on nodeUpdateQueue in RMNodeImpl. (Chengbing Liu via junping_du) @@ -1663,9 +1504,6 @@ Release 2.7.0 - 2015-04-20 YARN-2749. Fix some testcases from TestLogAggregationService fails in trunk. (Xuan Gong via junping_du) - YARN-3207. Secondary filter matches entites which do not have the key being - filtered for. (Zhijie Shen via xgong) - YARN-3132. RMNodeLabelsManager should remove node from node-to-label mapping when node becomes deactivated. (Wangda Tan via jianhe) @@ -1678,15 +1516,9 @@ Release 2.7.0 - 2015-04-20 YARN-3194. RM should handle NMContainerStatuses sent by NM while registering if NM is Reconnected node (Rohith via jlowe) - YARN-3238. Connection timeouts to nodemanagers are retried at - multiple levels (Jason Lowe via xgong) - YARN-3247. TestQueueMappings should use CapacityScheduler explicitly. (Zhihai Xu via ozawa) - YARN-3239. WebAppProxy does not support a final tracking url which has - query fragments and params (Jian He via jlowe) - YARN-3256. TestClientToAMTokens#testClientTokenRace is not running against all Schedulers even when using ParameterizedSchedulerTestBase. (Anubhav Dhoot via devaraj) @@ -1697,27 +1529,12 @@ Release 2.7.0 - 2015-04-20 YARN-3265. Fixed a deadlock in CapacityScheduler by always passing a queue's available resource-limit from the parent queue. (Wangda Tan via vinodkv) - YARN-3222. Fixed RMNode to send scheduler events in sequential order when a - node reconnects. (Rohith Sharma K S via jianhe) - YARN-3131. YarnClientImpl should check FAILED and KILLED state in submitApplication (Chang Li via jlowe) - YARN-3231. FairScheduler: Changing queueMaxRunningApps interferes with pending - jobs. (Siqi Li via kasha) - - YARN-3242. Asynchrony in ZK-close can lead to ZKRMStateStore watcher receiving - events for old client. (Zhihai Xu via kasha) - - YARN-3227. Timeline renew delegation token fails when RM user's TGT is expired - (Zhijie Shen via xgong) - YARN-3275. CapacityScheduler: Preemption happening on non-preemptable queues (Eric Payne via jlowe) - YARN-3287. Made TimelineClient put methods do as the correct login context. - (Daryn Sharp and Jonathan Eagles via zjshen) - YARN-3300. Outstanding_resource_requests table should not be shown in AHS. (Xuan Gong via jianhe) @@ -1732,9 +1549,6 @@ Release 2.7.0 - 2015-04-20 YARN-3154. Added additional APIs in LogAggregationContext to avoid aggregating running logs of application when rolling is enabled. (Xuan Gong via vinodkv) - YARN-3267. Timelineserver applies the ACL rules after applying the limit on - the number of records (Chang Li via jeagles) - YARN-3171. Sort by Application id, AppAttempt and ContainerID doesn't work in ATS / RM web ui. (Naganarasimha G R via xgong) @@ -1747,18 +1561,12 @@ Release 2.7.0 - 2015-04-20 YARN-3379. Fixed missing data in localityTable and ResourceRequests table in RM WebUI. (Xuan Gong via jianhe) - YARN-3369. Missing NullPointer check in AppSchedulingInfo causes RM to die. - (Brahma Reddy Battula via wangda) - YARN-3384. TestLogAggregationService.verifyContainerLogs fails after YARN-2777. (Naganarasimha G R via ozawa) YARN-3336. FileSystem memory leak in DelegationTokenRenewer. (Zhihai Xu via cnauroth) - YARN-3393. Getting application(s) goes wrong when app finishes before - starting the attempt. (Zhijie Shen via xgong) - YARN-2213. Change proxy-user cookie log in AmIpFilter to DEBUG. (Varun Saxena via xgong) @@ -1772,9 +1580,6 @@ Release 2.7.0 - 2015-04-20 YARN-3466. Fix RM nodes web page to sort by node HTTP-address, #containers and node-label column (Jason Lowe via wangda) - YARN-3055. Fixed ResourceManager's DelegationTokenRenewer to not stop token - renewal of applications part of a bigger workflow. (Daryn Sharp via vinodkv) - Release 2.6.2 - UNRELEASED INCOMPATIBLE CHANGES @@ -1787,18 +1592,211 @@ Release 2.6.2 - UNRELEASED BUG FIXES -Release 2.6.1 - UNRELEASED +Release 2.6.1 - 2015-09-09 INCOMPATIBLE CHANGES NEW FEATURES + YARN-3249. Add a 'kill application' button to Resource Manager's Web UI. + (Ryu Kobayashi via ozawa) + IMPROVEMENTS + YARN-3230. Clarify application states on the web UI. (Jian He via wangda) + + YARN-1809. Synchronize RM and TimeLineServer Web-UIs. (Zhijie Shen and + Xuan Gong via jianhe) + + YARN-3092. Created a common ResourceUsage class to track labeled resource + usages in Capacity Scheduler. (Wangda Tan via jianhe) + + YARN-3098. Created common QueueCapacities class in Capacity Scheduler to + track capacities-by-labels of queues. (Wangda Tan via jianhe) + + YARN-2301. Improved yarn container command. (Naganarasimha G R via jianhe) + + YARN-3978. Configurably turn off the saving of container info in Generic AHS + (Eric Payne via jeagles) + OPTIMIZATIONS BUG FIXES + YARN-2856. Fixed RMAppImpl to handle ATTEMPT_KILLED event at ACCEPTED state + on app recovery. (Rohith Sharmaks via jianhe) + + YARN-2816. NM fail to start with NPE during container recovery (Zhihai Xu + via jlowe) + + YARN-2414. RM web UI: app page will crash if app is failed before any + attempt has been created (Wangda Tan via jlowe) + + YARN-2865. Fixed RM to always create a new RMContext when transtions from + StandBy to Active. (Rohith Sharmaks via jianhe) + + YARN-2906. CapacitySchedulerPage shows HTML tags for a queue's Active Users. + (Jason Lowe via jianhe) + + YARN-2905. AggregatedLogsBlock page can infinitely loop if the aggregated + log file is corrupted (Varun Saxena via jlowe) + + YARN-2890. MiniYARNCluster should start the timeline server based on the + configuration. (Mit Desai via zjshen) + + YARN-2894. Fixed a bug regarding application view acl when RM fails over. + (Rohith Sharmaks via jianhe) + + YARN-2874. Dead lock in "DelegationTokenRenewer" which blocks RM to execute + any further apps. (Naganarasimha G R via kasha) + + YARN-2910. FSLeafQueue can throw ConcurrentModificationException. + (Wilfred Spiegelenburg via kasha) + + YARN-2917. Fixed potential deadlock when system.exit is called in AsyncDispatcher + (Rohith Sharmaks via jianhe) + + YARN-2964. RM prematurely cancels tokens for jobs that submit jobs (oozie) + (Jian He via jlowe) + + YARN-1984. LeveldbTimelineStore does not handle db exceptions properly + (Varun Saxena via jlowe) + + YARN-2952. Fixed incorrect version check in StateStore. (Rohith Sharmaks + via jianhe) + + YARN-2340. Fixed NPE when queue is stopped during RM restart. + (Rohith Sharmaks via jianhe) + + YARN-2992. ZKRMStateStore crashes due to session expiry. (Karthik Kambatla + via jianhe) + + YARN-2922. ConcurrentModificationException in CapacityScheduler's LeafQueue. + (Rohith Sharmaks via ozawa) + + YARN-2997. Fixed NodeStatusUpdater to not send alreay-sent completed + container statuses on heartbeat. (Chengbing Liu via jianhe) + + YARN-3011. Possible IllegalArgumentException in ResourceLocalizationService + might lead NM to crash. (Varun Saxena via jianhe) + + YARN-3103. AMRMClientImpl does not update AMRM token properly. (Jason Lowe + via jianhe) + + YARN-3094. Reset timer for liveness monitors after RM recovery. (Jun Gong + via jianhe) + + YARN-2246. Made the proxy tracking URL always be + http(s)://proxy addr:port/proxy/<appId> to avoid duplicate sections. (Devaraj + K via zjshen) + + YARN-3207. Secondary filter matches entites which do not have the key being + filtered for. (Zhijie Shen via xgong) + + YARN-3238. Connection timeouts to nodemanagers are retried at + multiple levels (Jason Lowe via xgong) + + YARN-3239. WebAppProxy does not support a final tracking url which has + query fragments and params (Jian He via jlowe) + + YARN-3222. Fixed RMNode to send scheduler events in sequential order when a + node reconnects. (Rohith Sharma K S via jianhe) + + YARN-3231. FairScheduler: Changing queueMaxRunningApps interferes with pending + jobs. (Siqi Li via kasha) + + YARN-3242. Asynchrony in ZK-close can lead to ZKRMStateStore watcher receiving + events for old client. (Zhihai Xu via kasha) + + YARN-3227. Timeline renew delegation token fails when RM user's TGT is expired + (Zhijie Shen via xgong) + + YARN-3287. Made TimelineClient put methods do as the correct login context. + (Daryn Sharp and Jonathan Eagles via zjshen) + + YARN-3267. Timelineserver applies the ACL rules after applying the limit on + the number of records (Chang Li via jeagles) + + YARN-3369. Missing NullPointer check in AppSchedulingInfo causes RM to die. + (Brahma Reddy Battula via wangda) + + YARN-3393. Getting application(s) goes wrong when app finishes before + starting the attempt. (Zhijie Shen via xgong) + + YARN-3055. Fixed ResourceManager's DelegationTokenRenewer to not stop token + renewal of applications part of a bigger workflow. (Daryn Sharp via vinodkv) + + YARN-3493. RM fails to come up with error "Failed to load/recover state" + when mem settings are changed. (Jian He via wangda) + + YARN-3487. CapacityScheduler scheduler lock obtained unnecessarily when + calling getQueue (Jason Lowe via wangda) + + YARN-3024. LocalizerRunner should give DIE action when all resources are + localized. (Chengbing Liu via xgong) + + YARN-3464. Race condition in LocalizerRunner kills localizer before + localizing all resources. (Zhihai Xu via kasha) + + YARN-3641. NodeManager: stopRecoveryStore() shouldn't be skipped when + exceptions happen in stopping NM's sub-services. (Junping Du via jlowe) + + YARN-3526. ApplicationMaster tracking URL is incorrectly redirected + on a QJM cluster. (Weiwei Yang via xgong) + + YARN-2766. Made ApplicationHistoryManager return a sorted list of apps, + attempts and containers. (Robert Kanter via zjshen) + + YARN-3700. Made generic history service load a number of latest applications + according to the parameter or the configuration. (Xuan Gong via zjshen) + + YARN-2900. Application (Attempt and Container) Not Found in AHS results + in InternalServer Error (500). (Zhijie Shen and Mit Desai via xgong) + + YARN-3725. App submission via REST API is broken in secure mode due to + Timeline DT service address is empty. (Zhijie Shen via wangda) + + YARN-3585. NodeManager cannot exit on SHUTDOWN event triggered and NM + recovery is enabled (Rohith Sharmaks via jlowe) + + YARN-3832. Resource Localization fails on a cluster due to existing cache + directories (Brahma Reddy Battula via jlowe) + + YARN-3850. NM fails to read files from full disks which can lead to + container logs being lost and other issues (Varun Saxena via jlowe) + + YARN-3990. AsyncDispatcher may overloaded with RMAppNodeUpdateEvent when + Node is connected/disconnected (Bibin A Chundatt via jlowe) + + YARN-2637. Fixed max-am-resource-percent calculation in CapacityScheduler + when activating applications. (Craig Welch via jianhe) + + YARN-3733. Fix DominantRC#compare() does not work as expected if + cluster resource is empty. (Rohith Sharmaks via wangda) + + YARN-2920. Changed CapacityScheduler to kill containers on nodes where + node labels are changed. (Wangda Tan via jianhe) + + YARN-2978. Fixed potential NPE while getting queue info. (Varun Saxena via + jianhe) + + YARN-3099. Capacity Scheduler LeafQueue/ParentQueue should use ResourceUsage + to track used-resources-by-label.(Wangda Tan via jianhe) + + YARN-2694. Ensure only single node label specified in ResourceRequest. + (Wangda Tan via jianhe) + + YARN-3124. Fixed CS LeafQueue/ParentQueue to use QueueCapacities to track + capacities-by-label. (Wangda Tan via jianhe) + + YARN-2918. RM should not fail on startup if queue's configured labels do + not exist in cluster-node-labels. (Wangda Tan via jianhe) + + YARN-3999. RM hangs on draing events. (Jian He via xgong) + + YARN-4047. ClientRMService getApplications has high scheduler lock contention. + (Jason Lowe via jianhe) + Release 2.6.0 - 2014-11-18 INCOMPATIBLE CHANGES
