http://git-wip-us.apache.org/repos/asf/hadoop/blob/c6e37f67/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt ---------------------------------------------------------------------- diff --git a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt deleted file mode 100644 index bc232c8..0000000 --- a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt +++ /dev/null @@ -1,13728 +0,0 @@ - Hadoop HDFS Change Log - -Release 2.8.0 - UNRELEASED - - INCOMPATIBLE CHANGES - - HDFS-9047. Retire libwebhdfs. (wheat9) - - NEW FEATURES - - HDFS-7891. A block placement policy with best rack failure tolerance. - (Walter Su via szetszwo) - - HDFS-8131. Implement a space balanced block placement policy (Liu Shaohui - via kihwal) - - HDFS-8155. Support OAuth2 in WebHDFS. (jghoman) - - HDFS-9184. Logging HDFS operation's caller context into audit logs. - (Mingliang Liu via jitendra) - - HDFS-9244. Support nested encryption zones. (zhz) - - HDFS-9711. Integrate CSRF prevention filter in WebHDFS. (cnauroth) - - IMPROVEMENTS - - HDFS-9653. Added blocks pending deletion report to dfsadmin. - (Weiwei Yang via eyang) - - HDFS-9257. improve error message for "Absolute path required" in INode.java - to contain the rejected path (Marcell Szabo via harsh) - - HDFS-2390. dfsadmin -setBalancerBandwidth does not validate -ve value - (Gautam Gopalakrishnan via harsh) - - HDFS-8821. Explain message "Operation category X is not supported - in state standby" (Gautam Gopalakrishnan via harsh) - - HDFS-3918. EditLogTailer shouldn't log WARN when other node - is in standby mode (todd via harsh) - - HDFS-4396. Add START_MSG/SHUTDOWN_MSG for ZKFC - (Liang Xie via harsh) - - HDFS-7875. Improve log message when wrong value configured for - dfs.datanode.failed.volumes.tolerated. - (nijel via harsh) - - HDFS-2360. Ugly stacktrace when quota exceeds. (harsh) - - HDFS-7835. make initial sleeptime in locateFollowingBlock configurable for - DFSClient. (Zhihai Xu via Yongjun Zhang) - - HDFS-7829. Code clean up for LocatedBlock. (Takanobu Asanuma via jing9) - - HDFS-7854. Separate class DataStreamer out of DFSOutputStream. (Li Bo via - jing9) - - HDFS-7713. Implement mkdirs in the HDFS Web UI. (Ravi Prakash via wheat9) - - HDFS-7928. Scanning blocks from disk during rolling upgrade startup takes - a lot of time if disks are busy (Rushabh S Shah via kihwal) - - HDFS-7990. IBR delete ack should not be delayed. (daryn via kihwal) - - HDFS-8004. Use KeyProviderCryptoExtension#warmUpEncryptedKeys when creating - an encryption zone. (awang via asuresh) - - HDFS-6263. Remove DRFA.MaxBackupIndex config from log4j.properties. - (Abhiraj Butala via aajisaka) - - HDFS-6408. Remove redundant definitions in log4j.properties. - (Abhiraj Butala via aajisaka) - - HDFS-7890. Improve information on Top users for metrics in - RollingWindowsManager and lower log level (J.Andreina via vinayakumarb) - - HDFS-7645. Rolling upgrade is restoring blocks from trash multiple times. - (Vinayakumar B and Keisuke Ogiwara via Arpit Agarwal) - - HDFS-7944. Minor cleanup of BlockPoolManager#getAllNamenodeThreads. - (Arpit Agarwal) - - HDFS-7671. hdfs user guide should point to the common rack awareness doc. - (Kai Sasaki via aajisaka) - - HDFS-8009. Signal congestion on the DataNode. (wheat9) - - HDFS-7978. Add LOG.isDebugEnabled() guard for some LOG.debug(..). - (Walter Su via wang) - - HDFS-7888. Change DFSOutputStream and DataStreamer for convenience of - subclassing. (Li Bo via szetszwo) - - HDFS-8049. Add @InterfaceAudience.Private annotation to hdfs client - implementation. (Takuya Fukudome via szetszwo) - - HDFS-8079. Move dfs.client.retry.* confs from DFSConfigKeys to - HdfsClientConfigKeys.Retry. (szetszwo) - - HDFS-8073. Split BlockPlacementPolicyDefault.chooseTarget(..) so it - can be easily overrided. (Walter Su via vinayakumarb) - - HDFS-8080. Separate JSON related routines used by WebHdfsFileSystem to a - package local class. (wheat9) - - HDFS-8085. Move CorruptFileBlockIterator to a new hdfs.client.impl package. - (szetszwo) - - HDFS-8076. Code cleanup for DFSInputStream: use offset instead of - LocatedBlock when possible. (Zhe Zhang via wang) - - HDFS-8089. Move o.a.h.hdfs.web.resources.* to the client jars. (wheat9) - - HDFS-7979. Initialize block report IDs with a random number. (wang) - - HDFS-8101. DFSClient use of non-constant DFSConfigKeys pulls in WebHDFS - classes at runtime. (Sean Busbey via atm) - - HDFS-8102. Separate webhdfs retry configuration keys from DFSConfigKeys. - (wheat9) - - HDFS-8100. Refactor DFSClient.Conf to a standalone class and separates - short-circuit related conf to ShortCircuitConf. (szetszwo) - - HDFS-8103. Move BlockTokenSecretManager.AccessMode into - BlockTokenIdentifier. (wheat9) - - HDFS-8084. Move dfs.client.failover.* confs from DFSConfigKeys to - HdfsClientConfigKeys.Failover and fix typos in the dfs.http.client.* - configuration keys. (szetszwo) - - HDFS-7933. fsck should also report decommissioning replicas. - (Xiaoyu Yao via cnauroth) - - HDFS-8083. Move dfs.client.write.* conf from DFSConfigKeys to - HdfsClientConfigKeys.Write. (szetszwo) - - HDFS-8117. More accurate verification in SimulatedFSDataset: replace - DEFAULT_DATABYTE with patterned data. (Zhe Zhang via wang) - - HDFS-8144. Split TestLazyPersistFiles into multiple tests. (Arpit Agarwal) - - HDFS-8082. Move dfs.client.read.*, dfs.client.short.circuit.*, - dfs.client.mmap.* and dfs.client.hedged.read.* conf from DFSConfigKeys - to HdfsClientConfigKeys. (szetszwo) - - HDFS-8165. Move GRANDFATHER_GENERATION_STAMP and GRANDFATER_INODE_ID to - hdfs-client. (wheat9) - - HDFS-8152. Refactoring of lazy persist storage cases. (Arpit Agarwal) - - HDFS-8169. Move LocatedBlocks and related classes to hdfs-client. (wheat9) - - HDFS-8133. Improve readability of deleted block check (Daryn Sharp via - Colin P. McCabe) - - HDFS-8185. Separate client related routines in HAUtil into a new class. - (wheat9) - - HDFS-8218. Move classes that used by ClientProtocol into hdfs-client. - (wheat9) - - HDFS-4448. Allow HA NN to start in secure mode with wildcard address - configured (atm via asuresh) - - HDFS-8215. Refactor NamenodeFsck#check method. (Takanobu Asanuma - via szetszwo) - - HDFS-8052. Move WebHdfsFileSystem into hadoop-hdfs-client. (wheat9) - - HDFS-8176. Record from/to snapshots in audit log for snapshot diff report. - (J. Andreina via jing9) - - HDFS-8280. Code Cleanup in DFSInputStream. (Jing Zhao via wheat9) - - HDFS-8283. DataStreamer cleanup and some minor improvement. (szetszwo via - jing9) - - HDFS-5574. Remove buffer copy in BlockReader.skip. - (Binglin Chang via aajisaka) - - HDFS-8200. Refactor FSDirStatAndListingOp. (wheat9) - - HDFS-8292. Move conditional in fmt_time from dfs-dust.js to status.html. - (Charles Lamb via wang) - - HDFS-8086. Move LeaseRenewer to the hdfs.client.impl package. (Takanobu - Asanuma via szetszwo) - - HDFS-8249. Separate HdfsConstants into the client and the server side - class. (wheat9) - - HDFS-7397. Add more detail to the documentation for the conf key - "dfs.client.read.shortcircuit.streams.cache.size" (Brahma Reddy Battula via - Colin P. McCabe) - - HDFS-8237. Move all protocol classes used by ClientProtocol to hdfs-client. - (wheat9) - - HDFS-7758. Retire FsDatasetSpi#getVolumes() and use - FsDatasetSpi#getVolumeRefs() instead (Lei (Eddy) Xu via Colin P. McCabe) - - HDFS-8314. Move HdfsServerConstants#IO_FILE_BUFFER_SIZE and - SMALL_BUFFER_SIZE to the users. (Li Lu via wheat9) - - HDFS-7847. Modify NNThroughputBenchmark to be able to operate on a remote - NameNode (Charles Lamb via Colin P. McCabe) - - HDFS-8207. Improper log message when blockreport interval compared with - initial delay. (Brahma Reddy Battula and Ashish Singhi via ozawa) - - HDFS-7559. Create unit test to automatically compare HDFS related classes - and hdfs-default.xml. (Ray Chiang via asuresh) - - HDFS-5640. Add snapshot methods to FileContext. (Rakesh R via cnauroth) - - HDFS-8284. Update documentation about how to use HTrace with HDFS (Masatake - Iwasaki via Colin P. McCabe) - - HDFS-8113. Add check for null BlockCollection pointers in - BlockInfoContiguous structures (Chengbing Liu via Colin P. McCabe) - - HDFS-6757. Simplify lease manager with INodeID. (wheat9) - - HDFS-8327. Simplify quota calculations for snapshots and truncate. (wheat9) - - HDFS-8357. Consolidate parameters of INode.CleanSubtree() into a parameter - objects. (Li Lu via wheat9) - - HDFS-8255. Rename getBlockReplication to getPreferredBlockReplication. - (Contributed by Zhe Zhang) - - HDFS-6184. Capture NN's thread dump when it fails over. - (Ming Ma via aajisaka) - - HDFS-8350. Remove old webhdfs.xml and other outdated documentation stuff. - (Brahma Reddy Battula via aajisaka) - - HDFS-6888. Allow selectively audit logging ops (Chen He via vinayakumarb) - - HDFS-8397. Refactor the error handling code in DataStreamer. - (Tsz Wo Nicholas Sze via jing9) - - HDFS-8394. Move getAdditionalBlock() and related functionalities into a - separate class. (wheat9) - - HDFS-8157. Writes to RAM DISK reserve locked memory for block files. - (Arpit Agarwal) - - HDFS-8345. Storage policy APIs must be exposed via the FileSystem - interface. (Arpit Agarwal) - - HDFS-4185. Add a metric for number of active leases (Rakesh R via raviprak) - - HDFS-4383. Document the lease limits. (Arshad Mohammad via aajisaka) - - HDFS-8421. Move startFile() and related functions into FSDirWriteFileOp. - (wheat9) - - HDFS-8377. Support HTTP/2 in datanode. (Duo Zhang via wheat9) - - HDFS-8482. Rename BlockInfoContiguous to BlockInfo. (Zhe Zhang via wang) - - HDFS-8443. Document dfs.namenode.service.handler.count in hdfs-site.xml. - (J.Andreina via aajisaka) - - HDFS-8489. Subclass BlockInfo to represent contiguous blocks. - (Zhe Zhang via jing9) - - HDFS-8386. Improve synchronization of 'streamer' reference in - DFSOutputStream. (Rakesh R via wang) - - HDFS-8513. Rename BlockPlacementPolicyRackFaultTolarent to - BlockPlacementPolicyRackFaultTolerant. (wang) - - HDFS-8532. Make the visibility of DFSOutputStream#streamer member variable - to private. (Rakesh R via wang) - - HDFS-8535. Clarify that dfs usage in dfsadmin -report output includes all - block replicas. (Eddy Xu via wang) - - HDFS-8432. Introduce a minimum compatible layout version to allow downgrade - in more rolling upgrade use cases. (cnauroth) - - HDFS-8116. Cleanup uncessary if LOG.isDebugEnabled() from - RollingWindowManager. (Brahma Reddy Battula via xyao) - - HDFS-8553. Document hdfs class path options. - (Brahma Reddy Battula via cnauroth) - - HDFS-8552. Fix hdfs CLI usage message for namenode and zkfc. - (Brahma Reddy Battula via xyao) - - HDFS-8568. TestClusterId#testFormatWithEmptyClusterIdOption is failing. - (Rakesh R. via xyao) - - HDFS-8549. Abort the balancer if an upgrade is in progress. (wang) - - HDFS-8573. Move creation of restartMeta file logic from BlockReceiver to - ReplicaInPipeline. (Eddy Xu via wang) - - HDFS-7923. The DataNodes should rate-limit their full block reports by - asking the NN on heartbeat messages (cmccabe) - - HDFS-8540. Mover should exit with NO_MOVE_BLOCK if no block can be moved. - (surendra singh lilhore via szetszwo) - - HDFS-8606. Cleanup DFSOutputStream by removing unwanted changes - from HDFS-8386. (Rakesh R via szetszwo) - - HDFS-8238. Move ClientProtocol to the hdfs-client. - (Takanobu Asanuma via wheat9) - - HDFS-8446. Separate safemode related operations in GetBlockLocations(). - (wheat9) - - HDFS-8589. Fix unused imports in BPServiceActor and BlockReportLeaseManager - (cmccabe) - - HDFS-6249. Output AclEntry in PBImageXmlWriter. - (surendra singh lilhore via aajisaka) - - HDFS-8605. Merge Refactor of DFSOutputStream from HDFS-7285 branch. - (vinayakumarb via wang) - - HDFS-8582. Support getting a list of reconfigurable config properties and - do not generate spurious reconfig warnings (Lei (Eddy) Xu via Colin P. - McCabe) - - HDFS-8192. Eviction should key off used locked memory instead of - ram disk free space. (Arpit Agarwal) - - HDFS-9608. Merge HDFS-7912 to trunk and branch-2 (track BlockInfo instead - of Block in UnderReplicatedBlocks and PendingReplicationBlocks). - (Zhe Zhang via wang) - - HDFS-6564. Use slf4j instead of common-logging in hdfs-client. - (Rakesh R via wheat9) - - HDFS-8639. Add Option for NameNode HTTP port in MiniDFSClusterManager. - (Kai Sasaki via jing9) - - HDFS-8645. Resolve inconsistent code in TestReplicationPolicy between - trunk and branch-2. (Zhe Zhang via wang) - - HDFS-8462. Implement GETXATTRS and LISTXATTRS operations for WebImageViewer. - (Jagadesh Kiran N via aajisaka) - - HDFS-8640. Make reserved RBW space visible through JMX. (kanaka kumar - avvaru via Arpit Agarwal) - - HDFS-8665. Fix replication check in DFSTestUtils#waitForReplication. (wang) - - HDFS-8546. Use try with resources in DataStorage and Storage. (wang) - - HDFS-8651. Make hadoop-hdfs-project Native code -Wall-clean (Alan Burlison - via Colin P. McCabe) - - HDFS-8653. Code cleanup for DatanodeManager, DatanodeDescriptor and - DatanodeStorageInfo. (Zhe Zhang via wang) - - HDFS-8493. Consolidate truncate() related implementation in a single class. - (Rakesh R via wheat9) - - HDFS-8635. Migrate HDFS native build to new CMake framework (Alan Burlison - via Colin P. McCabe) - - HDFS-8666. Speedup the TestMover tests. (Walter Su via jing9) - - HDFS-8703. Merge refactor of DFSInputStream from ErasureCoding branch - (vinayakumarb) - - HDFS-8709. Clarify automatic sync in FSEditLog#logEdit. (wang) - - HDFS-8711. setSpaceQuota command should print the available storage type - when input storage type is wrong. (Brahma Reddy Battula via xyao) - - HDFS-8620. Clean up the checkstyle warinings about ClientProtocol. - (Takanobu Asanuma via wheat9) - - HDFS-8712. Remove 'public' and 'abstracta modifiers in FsVolumeSpi and - FsDatasetSpi (Lei (Eddy) Xu via vinayakumarb) - - HDFS-8726. Move protobuf files that define the client-sever protocols to - hdfs-client. (wheat9) - - HDFS-8751. Remove setBlocks API from INodeFile and misc code cleanup. (Zhe - Zhang via jing9) - - HDFS-8541. Mover should exit with NO_MOVE_PROGRESS if there is no move - progress. (Surendra Singh Lilhore via szetszwo) - - HDFS-8742. Inotify: Support event for OP_TRUNCATE. - (Surendra Singh Lilhore via aajisaka) - - HDFS-8794. Improve CorruptReplicasMap#corruptReplicasMap. (yliu) - - HDFS-7483. Display information per tier on the Namenode UI. - (Benoy Antony and wheat9 via wheat9) - - HDFS-8721. Add a metric for number of encryption zones. - (Rakesh R via cnauroth) - - HDFS-8495. Consolidate append() related implementation into a single class. - (Rakesh R via wheat9) - - HDFS-8795. Improve InvalidateBlocks#node2blocks. (yliu) - - HDFS-8797. WebHdfsFileSystem creates too many connections for pread. (jing9) - - HDFS-8730. Clean up the import statements in ClientProtocol. - (Takanobu Asanuma via wheat9) - - HDFS-8735. Inotify: All events classes should implement toString() API. - (Surendra Singh Lilhore via aajisaka) - - HDFS-7858. Improve HA Namenode Failover detection on the client. (asuresh) - - HDFS-8180. AbstractFileSystem Implementation for WebHdfs. (snayak via jghoman) - - HDFS-8811. Move BlockStoragePolicy name's constants from - HdfsServerConstants.java to HdfsConstants.java (vinayakumarb) - - HDFS-8822. Add SSD storagepolicy tests in TestBlockStoragePolicy# - testDefaultPolicies (vinayakumarb) - - HDFS-8816. Improve visualization for the Datanode tab in the NN UI. (wheat9) - - HDFS-7192. DN should ignore lazyPersist hint if the writer is - not local. (Arpit Agarwal) - - HDFS-6860. BlockStateChange logs are too noisy. (Chang Li and xyao via xyao) - - HDFS-8815. DFS getStoragePolicy implementation using single RPC call - (Surendra Singh Lilhore via vinayakumarb) - - HDFS-8856. Make LeaseManager#countPath O(1). (Arpit Agarwal) - - HDFS-8772. Fix TestStandbyIsHot#testDatanodeRestarts which occasionally fails. - (Walter Su via wang). - - HDFS-8818. Changes the global moveExecutor to per datanode executors and - changes MAX_SIZE_TO_MOVE to be configurable. (szetszwo) - - HDFS-8805. Archival Storage: getStoragePolicy should not need superuser privilege. - (Brahma Reddy Battula via jing9) - - HDFS-8887. Expose storage type and storage ID in BlockLocation. (wang) - - HDFS-8622. Implement GETCONTENTSUMMARY operation for WebImageViewer. - (Jagadesh Kiran N via aajisaka) - - HDFS-7649. Multihoming docs should emphasize using hostnames in - configurations. (Brahma Reddy Battula via Arpit Agarwal) - - HDFS-8824. Do not use small blocks for balancing the cluster. (szetszwo) - - HDFS-8883. NameNode Metrics : Add FSNameSystem lock Queue Length. - (Anu Engineer via xyao) - - HDFS-8713. Convert DatanodeDescriptor to use SLF4J logging. (wang) - - HDFS-6407. Add sorting and pagination in the datanode tab of the NN Web UI. - (wheat9) - - HDFS-8801. Convert BlockInfoUnderConstruction as a feature. - (Jing Zhao via wheat9) - - HDFS-8880. NameNode metrics logging. (Arpit Agarwal) - - HDFS-8278. When computing max-size-to-move in Balancer, count only the - storage with remaining >= default block size. (szetszwo) - - HDFS-8435. Support CreateFlag in WebHDFS. (Jakob Homan via cdouglas) - - HDFS-8826. In Balancer, add an option to specify the source node list - so that balancer only selects blocks to move from those nodes. (szetszwo) - - HDFS-8911. NameNode Metric : Add Editlog counters as a JMX metric. - (Anu Engineer via Arpit Agarwal) - - HDFS-8803. Move DfsClientConf to hdfs-client. (Mingliang Liu via wheat9) - - HDFS-8917. Cleanup BlockInfoUnderConstruction from comments and tests. - (Zhe Zhang via jing9) - - HDFS-8884. Fail-fast check in BlockPlacementPolicyDefault#chooseTarget. - (yliu) - - HDFS-8828. Utilize Snapshot diff report to build diff copy list in distcp. - (Yufei Gu via Yongjun Zhang) - - HDFS-8823. Move replication factor into individual blocks. (wheat9) - - HDFS-8934. Move ShortCircuitShm to hdfs-client. (Mingliang Liu via wheat9) - - HDFS-8928. Improvements for BlockUnderConstructionFeature: - ReplicaUnderConstruction as a separate class and replicas as an array. - (jing9) - - HDFS-8900. Compact XAttrs to optimize memory footprint. (yliu) - - HDFS-8951. Move the shortcircuit package to hdfs-client. - (Mingliang Liu via wheat9) - - HDFS-8896. DataNode object isn't GCed when shutdown, because it has GC - root in ShutdownHookManager. (Walter Su via jing9) - - HDFS-8248. Store INodeId instead of the INodeFile object in - BlockInfoContiguous. (wheat9) - - HDFS-8962. Clean up checkstyle warnings in o.a.h.hdfs.DfsClientConf. - (Mingliang Liu via wheat9) - - HDFS-8865. Improve quota initialization performance. (kihwal) - - HDFS-8938. Extract BlockToMarkCorrupt and ReplicationWork as standalone - classes from BlockManager. (Mingliang Liu via wheat9) - - HDFS-8925. Move BlockReaderLocal to hdfs-client. - (Mingliang Liu via wheat9) - - HDFS-8983. NameNode support for protected directories. (Arpit Agarwal) - - HDFS-8980. Remove unnecessary block replacement in INodeFile. (jing9) - - HDFS-8990. Move RemoteBlockReader to hdfs-client module. - (Mingliang via wheat9) - - HDFS-8946. Improve choosing datanode storage for block placement. (yliu) - - HDFS-8965. Harden edit log reading code against out of memory errors (cmccabe) - - HDFS-2070. Add more unit tests for FsShell getmerge (Daniel Templeton via - Colin P. McCabe) - - HDFS-328. Improve fs -setrep error message for invalid replication factors. - (Daniel Templeton via wang) - - HDFS-8890. Allow admin to specify which blockpools the balancer should run - on. (Chris Trezzo via mingma) - - HDFS-9002. Move o.a.h.hdfs.net/*Peer classes to hdfs-client. - (Mingliang Liu via wheat9) - - HDFS-9021. Use a yellow elephant rather than a blue one in diagram. (wang) - - HDFS-9012. Move o.a.h.hdfs.protocol.datatransfer.PipelineAck class to - hadoop-hdfs-client module. (Mingliang Liu via wheat9) - - HDFS-8984. Move replication queues related methods in FSNamesystem to - BlockManager. (wheat9) - - HDFS-9019. Adding informative message to sticky bit permission denied - exception. (xyao) - - HDFS-8860. Remove unused Replica copyOnWrite code (Lei (Eddy) Xu via Colin P. McCabe) - - HDFS-8716. Introduce a new config specifically for safe mode block count - (Chang Li via kihwal) - - HDFS-7116. Add a command to get the balancer bandwidth - (Rakesh R via vinayakumarb) - - HDFS-8974. Convert docs in xdoc format to markdown. - (Masatake Iwasaki via aajisaka) - - HDFS-6763. Initialize file system-wide quota once on transitioning to active - (kihwal) - - HDFS-9027. Refactor o.a.h.hdfs.DataStreamer#isLazyPersist() method. - (Mingliang Liu via Arpit Agarwal) - - HDFS-8996. Consolidate validateLog and scanLog in FJM#EditLogFile (Zhe - Zhang via Colin P. McCabe) - - HDFS-9010. Replace NameNode.DEFAULT_PORT with HdfsClientConfigKeys. - DFS_NAMENODE_RPC_PORT_DEFAULT config key. (Mingliang Liu via wheat9) - - HDFS-9065. Include commas on # of files, blocks, total filesystem objects - in NN Web UI. (Daniel Templeton via wheat9) - - HDFS-9008. Balancer#Parameters class could use a builder pattern. - (Chris Trezzo via mingma) - - HDFS-8953. DataNode Metrics logging (Kanaka Kumar Avvaru via vinayakumarb) - - HDFS-9082. Change the log level in WebHdfsFileSystem.initialize() from INFO - to DEBUG. (Santhosh Nayak via cnauroth) - - HDFS-7986. Allow files / directories to be deleted from the NameNode UI. - (Ravi Prakash via wheat9) - - HDFS-7995. Implement chmod in the HDFS Web UI. - (Ravi Prakash and Haohui Mai via wheat9) - - HDFS-9022. Move NameNode.getAddress() and NameNode.getUri() to - hadoop-hdfs-client. (Mingliang Liu via wheat9) - - HDFS-5802. NameNode does not check for inode type before traversing down a - path. (Xiao Chen via Yongjun Zhang) - - HDFS-9101. Remove deprecated NameNode.getUri() static helper method. - (Mingliang Liu via wheat9) - - HDFS-9111. Move hdfs-client protobuf convert methods from PBHelper to - PBHelperClient. (Mingliang Liu via wheat9) - - HADOOP-12428. Fix inconsistency between log-level guards and statements. - (Jagadesh Kiran N and Jackie Chang via ozawa) - - HDFS-9039. Separate client and server side methods of o.a.h.hdfs. - NameNodeProxies. (Mingliang Liu via wheat9) - - HDFS-8733. Keep server related definition in hdfs.proto on server side. - (Mingliang Liu via wheat9) - - HDFS-9130. Use GenericTestUtils#setLogLevel to the logging level. - (Mingliang Liu via wheat9) - - HDFS-9131 Move config keys used by hdfs-client to HdfsClientConfigKeys. - (Mingliang Liu via wheat9) - - HDFS-7529. Consolidate encryption zone related implementation into a single - class. (Rakesh R via wheat9) - - HDFS-9134. Move LEASE_{SOFTLIMIT,HARDLIMIT}_PERIOD constants from - HdfsServerConstants to HdfsConstants. (Mingliang Liu via wheat9) - - HDFS-5795. RemoteBlockReader2#checkSuccess() shoud print error status. - (Xiao Chen via Yongjun Zhang) - - HDFS-9112. Improve error message for Haadmin when multiple name service IDs - are configured. (Anu Engineer via jing9) - - HDFS-9132. Pass genstamp to ReplicaAccessorBuilder. (Colin Patrick McCabe via - Lei (Eddy) Xu) - - HDFS-9133. ExternalBlockReader and ReplicaAccessor need to return -1 on read - when at EOF. (Colin Patrick McCabe via Lei (Eddy) Xu) - - HDFS-8873. Allow the directoryScanner to be rate-limited (Daniel Templeton - via Colin P. McCabe) - - HDFS-8053. Move DFSIn/OutputStream and related classes to - hadoop-hdfs-client. (Mingliang Liu via wheat9) - - HDFS-9087. Add some jitter to DataNode.checkDiskErrorThread (Elliott Clark - via Colin P. McCabe) - - HDFS-8740. Move DistributedFileSystem to hadoop-hdfs-client. (Mingliang Liu - via wheat9) - - HDFS-9080. Update htrace version to 4.0.1 (cmccabe) - - HDFS-9148. Incorrect assert message in TestWriteToReplica#testWriteToTemporary - (Tony Wu via lei) - - HDFS-8859. Improve DataNode ReplicaMap memory footprint to save about 45%. - (yliu) - - HDFS-9165. Move entries in META-INF/services/o.a.h.fs.FileSystem to - hdfs-client. (Mingliang Liu via wheat9) - - HDFS-9166. Move hftp / hsftp filesystem to hfds-client. - (Mingliang Liu via wheat9) - - HDFS-8696. Make the lower and higher watermark in the DN Netty server - configurable. (Xiaobing Zhou via wheat9) - - HDFS-8971. Remove guards when calling LOG.debug() and LOG.trace() in client - package. (Mingliang Liu via wheat9) - - HDFS-9175. Change scope of 'AccessTokenProvider.getAccessToken()' and - 'CredentialBasedAccessTokenProvider.getCredential()' abstract methods to - public (Santhosh Nayak via cnauroth) - - HDFS-9015. Refactor TestReplicationPolicy to test different block placement - policies. (Ming Ma via lei) - - HDFS-8979. Clean up checkstyle warnings in hadoop-hdfs-client module. - (Mingliang Liu via wheat9) - - HDFS-9155. OEV should treat .XML files as XML even when the file name - extension is uppercase (nijel via cmccabe) - - HDFS-9170. Move libhdfs / fuse-dfs / libwebhdfs to hdfs-client. (wheat9) - - HDFS-8164. cTime is 0 in VERSION file for newly formatted NameNode. - (Xiao Chen via Yongjun Zhang) - - HDFS-9181. Better handling of exceptions thrown during upgrade shutdown. - (Wei-Chiu Chuang via Yongjun Zhang) - - HDFS-9110. Use Files.walkFileTree in NNUpgradeUtil#doPreUpgrade for - better efficiency. (Charlie Helin via wang) - - HDFS-8988. Use LightWeightHashSet instead of LightWeightLinkedSet in - BlockManager#excessReplicateMap. (yliu) - - HDFS-9006. Provide BlockPlacementPolicy that supports upgrade domain. - (Ming Ma via lei) - - HDFS-1172. Blocks in newly completed files are considered under-replicated - too quickly. (Masatake Iwasaki via jing9) - - HDFS-9238. Update TestFileCreation.testLeaseExpireHardLimit() to avoid using - DataNodeTestUtils.getFile(). (Tony Wu via lei) - - HDFS-9223. Code cleanup for DatanodeDescriptor and HeartbeatManager. (jing9) - - HDFS-9188. Make block corruption related tests FsDataset-agnostic. (lei) - - HDFS-9205. Do not schedule corrupt blocks for replication. (szetszwo) - - HDFS-9250. Add Precondition check to LocatedBlock#addCachedLoc. - (Xiao Chen via wang) - - HDFS-9251. Refactor TestWriteToReplica and TestFsDatasetImpl to avoid - explicitly creating Files in the tests code. (lei) - - HDFS-8647. Abstract BlockManager's rack policy into BlockPlacementPolicy. - (Brahma Reddy Battula via mingma) - - HDFS-9225. Fix intermittent test failure of - TestBlockManager.testBlocksAreNotUnderreplicatedInSingleRack. - (Masatake Iwasaki via wang) - - HDFS-7087. Ability to list /.reserved. (Xiao Chen via wang) - - HDFS-9264. Minor cleanup of operations on FsVolumeList#volumes. - (Walter Su via lei) - - HDFS-8808. dfs.image.transfer.bandwidthPerSec should not apply to - -bootstrapStandby (zhz) - - HDFS-4015. Safemode should count and report orphaned blocks. - (Anu Engineer via Arpit Agarwal) - - HDFS-7284. Add more debug info to - BlockInfoUnderConstruction#setGenerationStampAndVerifyReplicas. - (Wei-Chiu Chuang via Yongjun Zhang) - - HDFS-9284. fsck command should not print exception trace when file not - found. (Jagadesh Kiran N via wang) - - HDFS-9291. Fix TestInterDatanodeProtocol to be FsDataset-agnostic. (lei) - - HDFS-8945. Update the description about replica placement in HDFS - Architecture documentation. (Masatake Iwasaki via wang) - - HDFS-9292. Make TestFileConcorruption independent to underlying FsDataset - Implementation. (lei) - - HDFS-9259. Make SO_SNDBUF size configurable at DFSClient side for hdfs - write scenario. (Mingliang Liu via mingma) - - HDFS-9299. Give ReplicationMonitor a readable thread name (Staffan Friberg - via Colin P. McCabe) - - HDFS-9307. fuseConnect should be private to fuse_connect.c (Mingliang Liu - via Colin P. McCabe) - - HDFS-9311. Support optional offload of NameNode HA service health checks to - a separate RPC server. (cnauroth) - - HDFS-9255. Consolidate block recovery related implementation into a single - class. (Walter Su via zhz) - - HDFS-9295. Add a thorough test of the full KMS code path. - (Daniel Templeton via zhz) - - HDFS-8545. Refactor FS#getUsed() to use ContentSummary and add an API to fetch - the total file length from a specific path (J.Andreina via vinayakumarb) - - HDFS-9229. Expose size of NameNode directory as a metric. - (Surendra Singh Lilhore via zhz) - - HDFS-9339. Extend full test of KMS ACLs. (Daniel Templeton via zhz) - - HDFS-9007. Fix HDFS Balancer to honor upgrade domain policy. (Ming Ma via lei) - - HDFS-9331. Modify TestNameNodeMXBean#testNameNodeMXBeanInfo() to account for - filesystem entirely allocated for DFS use. (Tony Wu via lei) - - HDFS-9363. Add fetchReplica() to FsDatasetTestUtils to return FsDataset-agnostic - replica. (Tony Wu via lei) - - HDFS-9377. Fix findbugs warnings in FSDirSnapshotOp. - (Mingliang Liu via Yongjun Zhang) - - HDFS-9236. Missing sanity check for block size during block recovery. - (Tony Wu via Yongjun Zhang) - - HDFS-9379. Make NNThroughputBenchmark$BlockReportStats support more than 10 - datanodes. (Mingliang Liu via Arpit Agarwal) - - HDFS-9398. Make ByteArraryManager log message in one-line format. - (Mingliang Liu via szetszwo) - - HDFS-2261. AOP unit tests are not getting compiled or run. (wheat9) - - HDFS-9369. Use ctest to run tests for hadoop-hdfs-native-client. (wheat9) - - HDFS-9252. Change TestFileTruncate to use FsDatasetTestUtils to get block - file size and genstamp. (Lei (Eddy) Xu via cmccabe) - - HDFS-8056. Decommissioned dead nodes should continue to be counted as dead - after NN restart. (mingma) - - HDFS-9439. Include status of closeAck into exception message in DataNode#run. - (Xiao Chen via Yongjun Zhang) - - HDFS-9402. Switch DataNode.LOG to use slf4j. (Walter Su via wheat9) - - HDFS-9153. Pretty-format the output for DFSIO. (Kai Zheng via wheat9) - - HDFS-7988. Replace usage of ExactSizeInputStream with LimitInputStream. - (Walter Su via wheat9) - - HDFS-9314. Improve BlockPlacementPolicyDefault's picking of excess - replicas. (Xiao Chen via mingma) - - HDFS-8807. dfs.datanode.data.dir does not handle spaces between - storageType and URI correctly. (Anu Engineer via szetszwo) - - HDFS-9438. TestPipelinesFailover assumes Linux ifconfig. - (John Zhuge via Yongjun Zhang) - - HDFS-8512. WebHDFS : GETFILESTATUS should return LocatedBlock with storage - type info. (xyao) - - HDFS-9485. Make BlockManager#removeFromExcessReplicateMap accept BlockInfo - instead of Block. (Mingliang Liu via jing9) - - HDFS-9269. Update the documentation and wrapper for fuse-dfs. - (Wei-Chiu Chuang via zhz) - - HDFS-9474. TestPipelinesFailover should not fail when printing debug - message. (John Zhuge via Yongjun Zhang) - - HDFS-9214. Support reconfiguring dfs.datanode.balance.max.concurrent.moves - without DN restart. (Xiaobing Zhou via Arpit Agarwal) - - HDFS-9527. The return type of FSNamesystem.getBlockCollection should be - changed to INodeFile. (szetszwo) - - HDFS-9472. concat() API does not give proper exception messages on ./reserved - relative path (Rakesh R via umamahesh) - - HDFS-9528. Cleanup namenode audit/log/exception messages. (szetszwo via umamahesh) - - HDFS-9532. Detailed exception info is lost in reportTo methods of - ErrorReportAction and ReportBadBlockAction. (Yongjun Zhang) - - HDFS-9535. Newly completed blocks in IBR should not be considered - under-replicated too quickly. (Mingliang Liu via jing9) - - HDFS-9519. Some coding improvement in SecondaryNameNode#main. - (Xiao Chen via Yongjun Zhang) - - HDFS-9514. TestDistributedFileSystem.testDFSClientPeerWriteTimeout failing; - exception being swallowed. (Wei-Chiu Chuang via Yongjun Zhang) - - HDFS-8894. Set SO_KEEPALIVE on DN server sockets. - (Kanaka Kumar Avvaru via wang) - - HDFS-9557. Reduce object allocation in PB conversion - (Daryn Sharp via cnauroth) - - HDFS-9552. Document types of permission checks performed for HDFS - operations. (cnauroth) - - HDFS-9490. MiniDFSCluster should change block generation stamp via - FsDatasetTestUtils. (Tony Wu via lei) - - HDFS-8831. Trash Support for deletion in HDFS encryption zone. (xyao) - - HDFS-7779. Support changing ownership, group and replication in HDFS Web - UI. (Ravi Prakash via wheat9) - - HDFS-9630. DistCp minor refactoring and clean up. (Kai Zheng via zhz) - - HDFS-9415. Document dfs.cluster.administrators and - dfs.permissions.superusergroup. (Xiaobing Zhou via Arpit Agarwal) - - HDFS-9655. NN should start JVM pause monitor before loading fsimage. - (John Zhuge via Lei (Eddy) Xu) - - HDFS-8898. Create API and command-line argument to get quota and quota - usage without detailed content summary. (Ming Ma via kihwal) - - HDFS-9706. Log more details in debug logs in BlockReceiver's constructor. - (Xiao Chen via Yongjun Zhang) - - HDFS-9638. Improve DistCp Help and documentation. - (Wei-Chiu Chuang via Yongjun Zhang) - - HDFS-9721. Allow Delimited PB OIV tool to run upon fsimage that contains - INodeReference. (Xiao Chen via lei) - - HDFS-9503. Use generic option -fs for NNThroughputBenchmark instead of - -namenode. (Mingliang Liu via shv) - - HDFS-9777. Fix typos in DFSAdmin command line and documentation. - (Wei-Chiu Chuang via umamahesh) - - HDFS-9700. DFSClient and DFSOutputStream should set TCP_NODELAY on sockets - for DataTransferProtocol (Gary Helmling via iwasakims) - - HDFS-9644. Update encryption documentation to reflect nested EZs. (zhz) - - HDFS-9797. Log Standby exceptions thrown by RequestHedgingProxyProvider - at DEBUG Level (Inigo Goiri via asuresh) - - HDFS-9773. Remove dead code related to SimulatedFSDataset in tests. - (Brahma Reddy Battula via aajisaka) - - HDFS-8923. Add -source flag to balancer usage message. (Chris Trezzo - via szetszwo) - - HDFS-9764. DistCp doesn't print value for several arguments including - -numListstatusThreads. (Wei-Chiu Chuang via Yongjun Zhang) - - HDFS-9084. Pagination, sorting and filtering of files/directories in the - HDFS Web UI. (Ravi Prakash via ozawa) - - HDFS-9844. Correct path creation in getTrashRoot to handle root dir. (zhz) - - HDFS-9854. Log cipher suite negotiation more verbosely - (Wei-Chiu Chuang via cnauroth) - - HDFS-9843. Document distcp options required for copying between encrypted - locations. (Xiaoyu Yao via cnauroth) - - HDFS-9831. Document webhdfs retry configuration keys introduced by - HDFS-5219/HDFS-5122. (Xiaobing Zhou via xyao) - - HDFS-9887. WebHdfs socket timeouts should be configurable. - (Austin Donnelly and Chris Douglas via xyao) - - OPTIMIZATIONS - - HDFS-8026. Trace FSOutputSummer#writeChecksumChunks rather than - DFSOutputStream#writeChunk (cmccabe) - - HDFS-7433. Optimize performance of DatanodeManager's node map. - (daryn via kihwal) - - HDFS-8792. BlockManager#postponedMisreplicatedBlocks should use a - LightWeightHashSet to save memory (Yi Liu via Colin P. McCabe) - - HDFS-8845. DiskChecker should not traverse the entire tree (Chang Li via - Colin P. McCabe) - - HDFS-8862. BlockManager#excessReplicateMap should use a HashMap. (yliu) - - HDFS-8929. Add a metric to expose the timestamp of the last journal - (surendra singh lilhore via vinayakumarb) - - HDFS-8829. Make SO_RCVBUF and SO_SNDBUF size configurable for - DataTransferProtocol sockets and allow configuring auto-tuning (He Tianyi - via Colin P. McCabe) - - HDFS-9139. Enable parallel JUnit tests for HDFS Pre-commit - (Chris Nauroth and Vinayakumar B via vinayakumarb) - - HDFS-9145. Tracking methods that hold FSNamesytemLock for too long. - (Mingliang Liu via wheat9) - - HDFS-9167. Update pom.xml in other modules to depend on hdfs-client instead - of hdfs. (Mingliang Liu via wheat9) - - HDFS-9253. Refactor tests of libhdfs into a directory. (wheat9) - - HDFS-9280. Document NFS gateway export point parameter. (Xiao Chen via zhz) - - HDFS-9297. Update TestBlockMissingException to use corruptBlockOnDataNodesByDeletingBlockFile(). - (Tony Wu via lei) - - HDFS-9168. Move client side unit test to hadoop-hdfs-client. (wheat9) - - HDFS-9312. Fix TestReplication to be FsDataset-agnostic. (lei) - - HDFS-9308. Add truncateMeta() and deleteMeta() to MiniDFSCluster. (Tony Wu via lei) - - HDFS-9372. Remove dead code in DataStorage.recoverTransitionRead. - (Duo Zhang via wheat9) - - HDFS-9282. Make data directory count and storage raw capacity related tests - FsDataset-agnostic. (Tony Wu via lei) - - HDFS-9318. considerLoad factor can be improved. (Kuhu Shukla via kihwal) - - HDFS-8335. FSNamesystem should construct FSPermissionChecker only if - permission is enabled. (Gabor Liptak via wheat9) - - HDFS-9436. Make NNThroughputBenchmark$BlockReportStats run with 10 - datanodes by default. (Mingliang Liu via shv) - - HDFS-8999. Allow a file to be closed with COMMITTED but not yet COMPLETE - blocks. (szetszwo) - - HDFS-9715. Check storage ID uniqueness on datanode startup - (Lei (Eddy) Xu via vinayakumarb) - - HDFS-9726. Refactor IBR code to a new class. (szetszwo) - - HDFS-9686. Remove useless boxing/unboxing code. - (Kousuke Saruta via aajisaka) - - HDFS-9768. Reuse ObjectMapper instance in HDFS to improve the performance. - (Lin Yiqun via aajisaka) - - HDFS-9425. Expose number of blocks per volume as a metric - (Brahma Reddy Battula via vinayakumarb) - - HDFS-9710. DN can be configured to send block receipt IBRs in batches. - (szetszwo) - - BUG FIXES - - HDFS-8091: ACLStatus and XAttributes should be presented to - INodeAttributesProvider before returning to client (asuresh) - - HDFS-7501. TransactionsSinceLastCheckpoint can be negative on SBNs. - (Gautam Gopalakrishnan via harsh) - - HDFS-5356. MiniDFSCluster should close all open FileSystems when shutdown() - (Rakesh R via vinayakumarb) - - HDFS-7867. Update action param from "start" to "prepare" in rolling upgrade - javadoc (J.Andreina via vinayakumarb) - - HDFS-3325. When configuring "dfs.namenode.safemode.threshold-pct" to a value - greater or equal to 1 there is mismatch in the UI report - (J.Andreina via vinayakumarb) - - HDFS-8002. Website refers to /trash directory. (Brahma Reddy Battula via - aajisaka) - - HDFS-7261. storageMap is accessed without synchronization in - DatanodeDescriptor#updateHeartbeatState() (Brahma Reddy Battula via Colin - P. McCabe) - - HDFS-7997. The first non-existing xattr should also throw IOException. - (zhouyingchao via yliu) - - HDFS-7922. ShortCircuitCache#close is not releasing - ScheduledThreadPoolExecutors (Rakesh R via Colin P. McCabe) - - HDFS-5215. dfs.datanode.du.reserved is not considered while computing - available space ( Brahma Reddy Battula via Yongjun Zhang) - - HDFS-7725. Incorrect "nodes in service" metrics caused all writes to fail. - (Ming Ma via wang) - - HDFS-8096. DatanodeMetrics#blocksReplicated will get incremented early and - even for failed transfers (vinayakumarb) - - HDFS-7939. Two fsimage_rollback_* files are created which are not deleted - after rollback. (J.Andreina via vinayakumarb) - - HDFS-8111. NPE thrown when invalid FSImage filename given for - 'hdfs oiv_legacy' cmd ( surendra singh lilhore via vinayakumarb ) - - HDFS-7701. Support reporting per storage type quota and usage - with hadoop/hdfs shell. (Peter Shi via Arpit Agarwal) - - HDFS-6666. Abort NameNode and DataNode startup if security is enabled but - block access token is not enabled. (Vijay Bhat via cnauroth) - - HDFS-8055. NullPointerException when topology script is missing. - (Anu Engineer via cnauroth) - - HDFS-8142. DistributedFileSystem encryption zone commands should resolve - relative paths. (Rakesh R via wang) - - HDFS-7863. Missing description of some methods and parameters in javadoc of - FSDirDeleteOp. (Brahma Reddy Battula via ozawa) - - HDFS-8043. NPE in MiniDFSCluster teardown. (Brahma Reddy Battula via ozawa) - - HDFS-8173. NPE thrown at DataNode shutdown when HTTP server was not able to - create (surendra singh lilhore via vinayakumarb) - - HDFS-7993. Provide each Replica details in fsck (J.Andreina via vinayakumarb) - - HDFS-8217. During block recovery for truncate Log new Block Id in case of - copy-on-truncate is true. (vinayakumarb) - - HDFS-8231. StackTrace displayed at client while QuotaByStorageType exceeds - (J.Andreina and Xiaoyu Yao via vinayakumarb) - - HDFS-8191. Fix byte to integer casting in SimulatedFSDataset#simulatedByte. - (Zhe Zhang via wang) - - HDFS-8211. DataNode UUID is always null in the JMX counter. (Anu Engineer - via Arpit Agarwal) - - HDFS-8247. TestDiskspaceQuotaUpdate#testAppendOverTypeQuota is failing. - (Xiaoyu Yao via cnauroth) - - HDFS-8206. Fix the typos in hadoop-hdfs-httpfs. (Brahma Reddy Battula via xyao) - - HDFS-8205. CommandFormat#parse() should not parse option as - value of option. (Peter Shi and Xiaoyu Yao via Arpit Agarwal) - - HDFS-8232. Missing datanode counters when using Metrics2 sink interface. - (Anu Engineer via cnauroth) - - HDFS-8214. Secondary NN Web UI shows wrong date for Last Checkpoint. (clamb via wang) - - HDFS-8300. Fix unit test failures and findbugs warning caused by HDFS-8283. - (jing9) - - HDFS-8276. LazyPersistFileScrubber should be disabled if scrubber interval - configured zero. (Surendra Singh Lilhore via Arpit Agarwal) - - HDFS-8229. LAZY_PERSIST file gets deleted after NameNode restart. - (Surendra Singh Lilhore via Arpit Agarwal) - - HDFS-8309. Skip unit test using DataNodeTestUtils#injectDataDirFailure() on Windows. - (xyao) - - HDFS-8290. WebHDFS calls before namesystem initialization can cause - NullPointerException. (cnauroth) - - HDFS-8310. Fix TestCLI.testAll "help: help for find" on Windows. - (Kiran Kumar M R via Xiaoyu Yao) - - HDFS-2484. checkLease should throw FileNotFoundException when file does - not exist. (Rakesh R via shv) - - HDFS-7833. DataNode reconfiguration does not recalculate valid volumes - required, based on configured failed volumes tolerated. - (Lei (Eddy) Xu via cnauroth) - - HDFS-8325. Misspelling of threshold in log4j.properties for tests. - (Brahma Reddy Battula via aajisaka) - - HDFS-8321. CacheDirectives and CachePool operations should throw - RetriableException in safemode. (wheat9) - - HDFS-8037. CheckAccess in WebHDFS silently accepts malformed FsActions - parameters. (wheat9) - - HDFS-8257. Namenode rollingUpgrade option is incorrect in document - (J.Andreina via vinayakumarb) - - HDFS-8067. haadmin prints out stale help messages (Ajith S via vinayakumarb) - - HDFS-8174. Update replication count to live rep count in fsck report. (J.Andreina) - - HDFS-6291. FSImage may be left unclosed in BootstrapStandby#doRun() - (Sanghyun Yun via vinayakumarb) - - HDFS-7998. HDFS Federation : Command mentioned to add a NN to existing - federated cluster is wrong (Ajith S via vinayakumarb) - - HDFS-8222. Remove usage of "dfsadmin -upgradeProgress" from document which - is no longer supported. (J.Andreina via aajisaka) - - HDFS-8108. Fsck should provide the info on mandatory option to be used along with "-blocks , - -locations and -racks" (J.Andreina via umamahesh) - - HDFS-8187. Remove usage of "-setStoragePolicy" and "-getStoragePolicy" using - dfsadmin cmd (as it is not been supported) (J.Andreina via vinayakumarb) - - HDFS-8175. Provide information on snapshotDiff for supporting the comparison - between snapshot and current status (J.Andreina via vinayakumarb) - - HDFS-8209. Support different number of datanode directories in MiniDFSCluster. - (surendra singh lilhore via vinayakumarb) - - HDFS-6576. Datanode log is generating at root directory in security mode - (surendra singh lilhore via vinayakumarb) - - HDFS-3384. DataStreamer thread should be closed immediatly when failed to - setup a PipelineForAppendOrRecovery (Uma Maheswara Rao G via vinayakumarb) - - HDFS-6285. tidy an error log inside BlockReceiver. (Liang Xie via umamahesh) - - HDFS-8346. libwebhdfs build fails during link due to unresolved external - symbols. (Chris Nauroth via wheat9) - - HDFS-8274. NFS configuration nfs.dump.dir not working. (Ajith S via - Arpit Agarwal) - - HDFS-8340. Fix NFS documentation of nfs.wtmax. (Ajith S via Arpit Agarwal) - - HDFS-8311. DataStreamer.transfer() should timeout the socket InputStream. - (Esteban Gutierrez via Yongjun Zhang) - - HDFS-8326. Documentation about when checkpoints are run is out of date. - (Misty Stanley-Jones via xyao) - - HDFS-8097. TestFileTruncate is failing intermittently. (Rakesh R via - Arpit Agarwal) - - HDFS-8351. Remove namenode -finalize option from document. (aajisaka) - - HDFS-8362. Java Compilation Error in TestHdfsConfigFields.java - (Arshad Mohammad via vinayakumarb) - - HDFS-8358. TestTraceAdmin fails (Masatake Iwasaki via kihwal) - - HDFS-8380. Always call addStoredBlock on blocks which have been shifted - from one storage to another (cmccabe) - - HDFS-8243. Files written by TestHostsFiles and TestNameNodeMXBean are - causing Release Audit Warnings. (Ruth Wisniewski via Arpit Agarwal) - - HDFS-7728. Avoid updating quota usage while loading edits. - (Jing Zhao via wheat9) - - HDFS-8150. Make getFileChecksum fail for blocks under construction - (J.Andreina via vinayakumarb) - - HDFS-8371. Fix test failure in TestHdfsConfigFields for spanreceiver - properties. (Ray Chiang via aajisaka) - - HDFS-8403. Eliminate retries in TestFileCreation - #testOverwriteOpenForWrite. (Arpit Agarwal via wheat9) - - HDFS-6348. SecondaryNameNode not terminating properly on runtime exceptions - (Rakesh R via vinayakumarb) - - HDFS-8454. Remove unnecessary throttling in TestDatanodeDeath. - (Arpit Agarwal) - - HDFS-8268. Port conflict log for data node server is not sufficient - (Mohammad Shahid Khan via vinayakumarb) - - HDFS-8407. hdfsListDirectory must set errno to 0 on success (Masatake - Iwasaki via Colin P. McCabe) - - HDFS-7401. Add block info to DFSInputStream' WARN message when it adds - node to deadNodes (Arshad Mohammad via vinayakumarb) - - HDFS-8490. Typo in trace enabled log in ExceptionHandler of WebHDFS. - (Archana T via ozawa) - - HDFS-8256. "-storagepolicies , -blockId ,-replicaDetails " options are missed - out in usage and from documentation (J.Andreina via vinayakumarb) - - HDFS-8470. fsimage loading progress should update inode, delegation token and - cache pool count. (surendra singh lilhore via vinayakumarb) - - HDFS-3716. Purger should remove stale fsimage ckpt files - (J.Andreina via vinayakumarb) - - HDFS-8463. Calling DFSInputStream.seekToNewSource just after stream creation - causes NullPointerException (Masatake Iwasaki via kihwal) - - HDFS-8539. Hdfs doesnt have class 'debug' in windows. - (Anu Engineer via cnauroth) - - HDFS-8554. TestDatanodeLayoutUpgrade fails on Windows. (cnauroth) - - HDFS-8593. Calculation of effective layout version mishandles comparison to - current layout version in storage. (cnauroth) - - HDFS-8607. TestFileCorruption doesn't work as expected. (Walter Su via - Arpit Agarwal) - - HDFS-8592. SafeModeException never get unwrapped. (wheat9) - - HDFS-8548. Minicluster throws NPE on shutdown. - (surendra singh lilhore via xyao) - - HDFS-8551. Fix hdfs datanode CLI usage message. - (Brahma Reddy Battula via xyao) - - HDFS-8337. Accessing httpfs via webhdfs doesn't work from a jar with - kerberos. (Yongjun Zhang) - - HDFS-4366. Block Replication Policy Implementation May Skip Higher-Priority - Blocks for Lower-Priority Blocks (Derek Dagit via kihwal) - - HDFS-8542. WebHDFS getHomeDirectory behavior does not match specification. - (Kanaka Kumar Avvaru via jghoman) - - HDFS-8546. Prune cached replicas from DatanodeDescriptor state on replica - invalidation. (wang) - - HDFS-8586. Dead Datanode is allocated for write when client is from deadnode - (Brahma Reddy Battula via vinayakumarb) - - HDFS-8628. Update missing command option for fetchdt - (J.Andreina via vinayakumarb) - - HDFS-8687. Remove the duplicate usage message from Dfsck.java. (Brahma - Reddy Battula via Arpit Agarwal) - - HDFS-8579. Update HDFS usage with missing options - (J.Andreina via vinayakumarb) - - HDFS-8706. Fix typo in datanode startup options in HDFSCommands.html. - (Brahma Reddy Battula via Arpit Agarwal) - - HDFS-8577. Avoid retrying to recover lease on a file which does not exist - (J.Andreina via vinayakumarb) - - HDFS-8686. WebHdfsFileSystem#getXAttr(Path p, final String name) doesn't - work if namespace is in capitals (kanaka kumar avvaru via vinayakumarb) - - HDFS-8642. Make TestFileTruncate more reliable. (Rakesh R via - Arpit Agarwal) - - HDFS-8729. Fix TestFileTruncate#testTruncateWithDataNodesRestartImmediately - which occasionally failed. (Walter Su via jing9) - - HDFS-8749. Fix findbugs warnings in BlockManager.java. - (Brahma Reddy Battula via aajisaka) - - HDFS-2956. calling fetchdt without a --renewer argument throws NPE - (vinayakumarb) - - HDFS-7608. hdfs dfsclient newConnectedPeer has no write timeout (Xiaoyu Yao - via Colin P. McCabe) - - HDFS-8778. TestBlockReportRateLimiting#testLeaseExpiration can deadlock. - (Arpit Agarwal) - - HDFS-7582. Enforce maximum number of ACL entries separately per access - and default. (vinayakumarb) - - HDFS-8773. Few FSNamesystem metrics are not documented in the Metrics page. - (Rakesh R via cnauroth) - - HDFS-8810. Correct assertions in TestDFSInotifyEventInputStream class. - (Surendra Singh Lilhore via aajisaka) - - HDFS-8670. Better to exclude decommissioned nodes for namenode NodeUsage JMX - (J.Andreina via vinayakumarb) - - HDFS-8847. change TestHDFSContractAppend to not override - testRenameFileBeingAppended method. (Zhihai Xu) - - HDFS-8844. TestHDFSCLI does not cleanup the test directory (Masatake - Iwasaki via Colin P. McCabe) - - HDFS-8866. Typo in docs: Rumtime -> Runtime. (Gabor Liptak via jghoman) - - HDFS-8879. Quota by storage type usage incorrectly initialized upon namenode - restart. (xyao) - - HDFS-8565. Typo in dfshealth.html - Decomissioning. (nijel via xyao) - - HDFS-8908. TestAppendSnapshotTruncate may fail with IOException: Failed to - replace a bad datanode. (Tsz Wo Nicholas Sze via yliu) - - HDFS-8922. Link the native_mini_dfs test library with libdl, since IBM Java - requires it (Ayappan via Colin P. McCabe) - - HDFS-8809. HDFS fsck reports under construction blocks as "CORRUPT". (jing9) - - HDFS-8942. Update hyperlink to rack awareness page in HDFS Architecture - documentation. (Masatake Iwasaki via aajisaka) - - HDFS-8930. Block report lease may leak if the 2nd full block report comes - when NN is still in safemode (Colin P. McCabe via Jing Zhao) - - HDFS-8948. Use GenericTestUtils to set log levels in TestPread and - TestReplaceDatanodeOnFailure. (Mingliang Liu via wheat9) - - HDFS-8932. NPE thrown in NameNode when try to get TotalSyncCount metric - before editLogStream initialization. (Surendra Singh Lilhore via xyao) - - HDFS-8682. Should not remove decommissioned node,while calculating the - number of live/dead decommissioned node. (J. Andreina via vinayakumarb) - - HDFS-8961. Investigate lock issue in o.a.h.hdfs.shortcircuit. - DfsClientShmManager.EndpointShmManager. (Mingliang Liu via wheat9) - - HDFS-8969. Clean up findbugs warnings for HDFS-8823 and HDFS-8932. - (Anu Engineer via wheat9) - - HDFS-8963. Fix incorrect sign extension of xattr length in HDFS-8900. - (Colin Patrick McCabe via yliu) - - HDFS-8950. NameNode refresh doesn't remove DataNodes that are no longer in - the allowed list (Daniel Templeton) - - HDFS-8388. Time and Date format need to be in sync in NameNode UI page. - (Surendra Singh Lilhore via aajisaka) - - HDFS-9003. ForkJoin thread pool leaks. (Kihwal Lee via jing9) - - HDFS-8885. ByteRangeInputStream used in webhdfs does not override - available(). (Shradha Revankar via aajisaka) - - HDFS-9009. Send metrics logs to NullAppender by default. (Arpit Agarwal) - - HDFS-8964. When validating the edit log, do not read at or beyond the file - offset that is being written (Zhe Zhang via Colin P. McCabe) - - HDFS-8939. Test(S)WebHdfsFileContextMainOperations failing on branch-2. - (Chris Nauroth via jghoman) - - HDFS-8581. ContentSummary on / skips further counts on yielding lock - (J.Andreina via vinayakumarb) - - HDFS-9036. In BlockPlacementPolicyWithNodeGroup#chooseLocalStorage , random - node is selected eventhough fallbackToLocalRack is true. - (J.Andreina via vinayakumarb) - - HDFS-9041. Move entries in META-INF/services/o.a.h.fs.FileSystem to - hdfs-client. (Mingliang Liu via wheat9) - - HDFS-9069. TestNameNodeMetricsLogger failing -port in use. - (stevel) - - HDFS-9067. o.a.h.hdfs.server.datanode.fsdataset.impl.TestLazyWriter - is failing in trunk (Surendra Singh Lilhore via vinayakumarb) - - HDFS-9072. Fix random failures in TestJMXGet. - (J.Andreina via stevel) - - HDFS-9073. Fix failures in TestLazyPersistLockedMemory - testReleaseOnEviction(). (J.Andreina via stevel) - - HDFS-9063. Correctly handle snapshot path for getContentSummary. (jing9) - - HDFS-8780. Fetching live/dead datanode list with arg true for remove- - DecommissionNode,returns list with decom node. (J.Andreina via vinayakumarb) - - HDFS-9013. Deprecate NameNodeMXBean#getNNStarted in branch2 and remove from - trunk (Surendra Singh Lilhore via vinayakumarb) - - HDFS-9128. TestWebHdfsFileContextMainOperations and - TestSWebHdfsFileContextMainOperations fail due to invalid HDFS path on - Windows. (Chris Nauroth via wheat9) - - HDFS-9076. Log full path instead of inodeId in DFSClient - #closeAllFilesBeingWritten() (Surendra Singh Lilhore via vinayakumarb) - - HDFS-9123. Copying from the root to a subdirectory should be forbidden. - (Wei-Chiu Chuang via Yongjun Zhang) - - HDFS-9107. Prevent NN's unrecoverable death spiral after full GC (Daryn - Sharp via Colin P. McCabe) - - HDFS-9147. Fix the setting of visibleLength in ExternalBlockReader. (Colin - P. McCabe via Lei (Eddy) Xu) - - HDFS-9092. Nfs silently drops overlapping write requests and causes data - copying to fail. (Yongjun Zhang) - - HDFS-9141. Thread leak in Datanode#refreshVolumes. (Uma Maheswara Rao G - via yliu) - - HDFS-9174. Fix findbugs warnings in FSOutputSummer.tracer and - DirectoryScanner$ReportCompiler.currentThread. (Yi Liu via wheat9) - - HDFS-9001. DFSUtil.getNsServiceRpcUris() can return too many entries in a - non-HA, non-federated cluster. (Daniel Templeton via atm) - - HDFS-9100. HDFS Balancer does not respect dfs.client.use.datanode.hostname. - (Casey Brotherton via Yongjun Zhang) - - HDFS-9191. Typo in Hdfs.java. NoSuchElementException is misspelled. - (Catherine Palmer via jghoman) - - HDFS-9193. Fix incorrect references the usages of the DN in dfshealth.js. - (Chang Li via wheat9) - - HADOOP-11098. [JDK8] Max Non Heap Memory default changed between JDK7 - and 8 (ozawa). - - HDFS-9151. Mover should print the exit status/reason on console like - balancer tool. (Surendra singh lilhore via vinayakumarb) - - HDFS-9154. [OEV-Doc] : Document does not mention about "-f" and "-r" options - (nijel via vinayakumarb) - - HDFS-7899. Improve EOF error message (Jagadesh Kiran N via vinayakumarb) - - HDFS-9196. Fix TestWebHdfsContentLength. (Masatake Iwasaki via jing9) - - HDFS-9159. [OIV] : return value of the command is not correct if invalid - value specified in "-p (processor)" option (nijel via vinayakumarb) - - HDFS-9176. Fix TestDirectoryScanner#testThrottling often fails. - (Daniel Templeton via lei) - - HDFS-9211. Fix incorrect version in hadoop-hdfs-native-client/pom.xml - from HDFS-9170 branch-2 backport. (Eric Payne via wang) - - HDFS-9137. DeadLock between DataNode#refreshVolumes and - BPOfferService#registrationSucceeded. (Uma Maheswara Rao G via yliu) - - HDFS-9142. Separating Configuration object for namenode(s) in - MiniDFSCluster. (Siqi Li via mingma) - - HDFS-8941. DistributedFileSystem listCorruptFileBlocks API should - resolve relative path. (Rakesh R via wang) - - HDFS-9215. Suppress the RAT warnings in hdfs-native-client module. (wheat9) - - HDFS-9222. Add hadoop-hdfs-client as a dependency of - hadoop-hdfs-native-client. (Mingliang Liu via wheat9) - - HDFS-9224. TestFileTruncate fails intermittently with BindException - (Brahma Reddy Battula via vinayakumarb) - - HDFS-9160. [OIV-Doc] : Missing details of 'delimited' for processor options - (nijel via vinayakumarb) - - HDFS-9235. hdfs-native-client build getting errors when built with cmake - 2.6. (Eric Payne via wheat9) - - HDFS-8779. WebUI fails to display block IDs that are larger than 2^53 - 1. - (wheat9) - - HDFS-9187. Fix null pointer error in Globber when FS was not constructed - via FileSystem#createFileSystem (cmccabe) - - HDFS-9157. [OEV and OIV] : Unnecessary parsing for mandatory arguements if - '-h' option is specified as the only option (nijel via vinayakumarb) - - HDFS-9237. NPE at TestDataNodeVolumeFailureToleration#tearDown. - (Brahma Reddy Battula via ozawa) - - HDFS-9208. Disabling atime may fail clients like distCp. (Kihwal Lee via - yliu) - - HDFS-9270. TestShortCircuitLocalRead should not leave socket after unit - test (Masatake Iwasaki via Colin P. McCabe) - - HDFS-3059. ssl-server.xml causes NullPointer. (Xiao Chen via wang) - - HDFS-9274. Default value of dfs.datanode.directoryscan.throttle.limit.ms.per.sec - should be consistent. (Yi Liu via zhz) - - HDFS-9286. HttpFs does not parse ACL syntax correctly for operation - REMOVEACLENTRIES. (Wei-Chiu Chuang via cnauroth) - - HDFS-9301. HDFS clients can't construct HdfsConfiguration instances. - (Mingliang Liu via wheat9) - - HDFS-9304. Add HdfsClientConfigKeys class to TestHdfsConfigFields - #configurationClasses. (Mingliang Liu via wheat9) - - HDFS-9268. fuse_dfs chown crashes when uid is passed as -1 (cmccabe) - - HDFS-9231. fsck doesn't list correct file path when Bad Replicas/Blocks - are in a snapshot. (Xiao Chen via Yongjun Zhang) - - HDFS-9302. WebHDFS throws NullPointerException if newLength is not - provided. (Jagadesh Kiran N via yliu) - - HDFS-9297. Decomissioned capacity should not be considered for - configured/used capacity (Contributed by Kuhu Shukla) - - HDFS-9044. Give Priority to FavouredNodes , before selecting - nodes from FavouredNode's Node Group (J.Andreina via vinayakumarb) - - HDFS-9332. Fix Precondition failures from NameNodeEditLogRoller while - saving namespace. (wang) - - HDFS-9343. Empty caller context considered invalid. (Mingliang Liu via - Arpit Agarwal) - - HDFS-9329. TestBootstrapStandby#testRateThrottling is flaky because fsimage - size is smaller than IO buffer size. (zhz) - - HDFS-9313. Possible NullPointerException in BlockManager if no excess - replica can be chosen. (mingma) - - HDFS-9354. Fix TestBalancer#testBalancerWithZeroThreadsForMove on Windows. - (Xiaoyu Yao via cnauroth) - - HDFS-9362. TestAuditLogger#testAuditLoggerWithCallContext assumes Unix line - endings, fails on Windows. (cnauroth) - - HDFS-9351. checkNNStartup() need to be called when fsck calls - FSNamesystem.getSnapshottableDirs(). (Xiao Chen via Yongjun Zhang) - - HDFS-9357. NN UI renders icons of decommissioned DN incorrectly. - (Surendra Singh Lilhore via wheat9) - - HDFS-9360. Storage type usage isn't updated properly after file deletion. - (Ming Ma via xyao) - - HDFS-9378. hadoop-hdfs-client tests do not write logs. (cnauroth) - - HDFS-9384. TestWebHdfsContentLength intermittently hangs and fails due to - TCP conversation mismatch between client and server. (cnauroth) - - HDFS-9394. branch-2 hadoop-hdfs-client fails during FileSystem ServiceLoader - initialization, because HftpFileSystem is missing. - (Mingliang Liu via cnauroth) - - HDFS-9249. NPE is thrown if an IOException is thrown in NameNode constructor. - (Wei-Chiu Chuang via Yongjun Zhang) - - HDFS-9401. Fix findbugs warnings in BlockRecoveryWorker. - (Brahma Reddy Battula via waltersu4549) - - HDFS-9364. Unnecessary DNS resolution attempts when creating NameNodeProxies. - (Xiao Chen via zhz) - - HDFS-9234. WebHdfs: getContentSummary() should give quota for storage types. - (Surendra Singh Lilhore via xyao) - - HDFS-9245. Fix findbugs warnings in hdfs-nfs/WriteCtx. - (Mingliang Liu via xyao) - - HDFS-9396. Total files and directories on jmx and web UI on standby is - uninitialized. (kihwal) - - HDFS-9410. Some tests should always reset sysout and syserr. - (Xiao Chen via waltersu4549) - - HDFS-9413. getContentSummary() on standby should throw StandbyException. - (Brahma Reddy Battula via mingma) - - HDFS-9387. Fix namenodeUri parameter parsing in NNThroughputBenchmark. - (Mingliang Liu via xyao) - - HDFS-9421. NNThroughputBenchmark replication test NPE with -namenode option. - (Mingliang Liu via xyao) - - HDFS-9358. TestNodeCount#testNodeCount timed out. - (Masatake Iwasaki via waltersu4549) - - HDFS-9397. Fix typo for readChecksum() LOG.warn in BlockSender.java. - (Enrique Flores via Arpit Agarwal) - - HDFS-9400. TestRollingUpgradeRollback fails on branch-2. - (Brahma Reddy Battula via cnauroth) - - HDFS-9443. Disabling HDFS client socket cache causes logging message - printed to console for CLI commands. (Chris Nauroth via wheat9) - - HDFS-6885. Fix wrong use of BytesWritable in FSEditLogOp#RenameOp. - (Yi Liu via wheat9) - - HDFS-7897. Shutdown metrics when stopping JournalNode. - (zhouyingchao via wheat9) - - HDFS-9356. Decommissioning node does not have Last Contact value in the UI. - (Surendra Singh Lilhore via wheat9) - - HDFS-9024. Deprecate the TotalFiles metric. (Akira Ajisaka via wheat9) - - HDFS-9428. Fix intermittent failure of - TestDNFencing.testQueueingWithAppend. (Masatake Iwasaki via waltersu4549) - - HDFS-9435. TestBlockRecovery#testRBWReplicas is failing intermittently. - (Rakesh R via waltersu4549) - - HDFS-9433. DFS getEZForPath API on a non-existent file should throw - FileNotFoundException (Rakesh R via umamahesh) - - HDFS-6101. TestReplaceDatanodeOnFailure fails occasionally. - (Wei-Chiu Chuang via cnauroth) - - HDFS-8855. Webhdfs client leaks active NameNode connections. - (Xiaobing Zhou via xyao) - - HDFS-8335. FSNamesystem should construct FSPermissionChecker only if - permission is enabled. (Gabor Liptak via wheat9) - - HDFS-6694. TestPipelinesFailover.testPipelineRecoveryStress tests fail - intermittently with various symptoms - debugging patch. (Yongjun Zhang via - Arpit Agarwal) - - HDFS-9459. hadoop-hdfs-native-client fails test build on Windows after - transition to ctest. (Chris Nauroth via wheat9) - - HDFS-9407. TestFileTruncate should not use fixed NN port. - (Brahma Reddy Battula via shv) - - HDFS-9467. Fix data race accessing writeLockHeldTimeStamp in FSNamesystem. - (Mingliang Liu via jing9) - - HDFS-9336. deleteSnapshot throws NPE when snapshotname is null. - (Brahma Reddy Battula via aajisaka) - - HDFS-6533. TestBPOfferService#testBasicFunctionalitytest fails - intermittently. (Wei-Chiu Chuang via Arpit Agarwal) - - HDFS-9429. Tests in TestDFSAdminWithHA intermittently fail with - EOFException (Xiao Chen via Colin P. McCabe) - - HDFS-9198. Coalesce IBR processing in the NN. (Daryn Sharp via umamahesh) - - HDFS-9565. TestDistributedFileSystem.testLocatedFileStatusStorageIdsTypes - is flaky due to race condition. (Wei-Chiu Chuang via Arpit Agarwal) - - HDFS-9570. Minor typos, grammar, and case sensitivity cleanup in - HdfsPermissionsGuide.md's (Travis Campbell via aw) - - HDFS-9515. NPE when MiniDFSCluster#shutdown is invoked on uninitialized - reference. (Wei-Chiu Chuang via Arpit Agarwal) - - HDFS-9572. Prevent DataNode log spam if a client connects on the data - transfer port but sends no data. (cnauroth) - - HDFS-9571. Fix ASF Licence warnings in Jenkins reports - (Brahma Reddy Battula via cnauroth) - - HDFS-9393. After choosing favored nodes, choosing nodes for remaining - replicas should go through BlockPlacementPolicy - (J.Andreina via vinayakumarb) - - HDFS-9347. Invariant assumption in TestQuorumJournalManager.shutdown() - is wrong. (Wei-Chiu Chuang via zhz) - - HDFS-9484. NNThroughputBenchmark$BlockReportStats should not send empty - block reports. (Mingliang Liu via shv) - - HDFS-9589. Block files which have been hardlinked should be duplicated - before the DataNode appends to the them (cmccabe) - - HDFS-9458. TestBackupNode always binds to port 50070, which can cause bind - failures. (Xiao Chen via cnauroth) - - HDFS-7553. fix the TestDFSUpgradeWithHA due to BindException. - (Xiao Chen via cnauroth) - - HDFS-9605. Add links to failed volumes to explorer.html in HDFS Web UI. - (Archana T via wheat9) - - HDFS-9619. SimulatedFSDataset sometimes can not find blockpool for the - correct namenode (Wei-Chiu Chuang via vinayakumarb) - - HDFS-9626. TestBlockReplacement#testBlockReplacement fails occasionally. - (Xiao Chen via zhz) - - HDFS-9639. Inconsistent Logging in BootstrapStandby. (Xiaobing Zhou via - Arpit Agarwal) - - HDFS-9584. NPE in distcp when ssl configuration file does not exist in - class path. (Surendra Singh Lilhore via Xiaoyu Yao) - - HDFS-9493. Test o.a.h.hdfs.server.namenode.TestMetaSave fails in trunk. - (Tony Wu via lei) - - HDFS-9612. DistCp worker threads are not terminated after jobs are done. - (Wei-Chiu Chuang via Yongjun Zhang) - - HDFS-9623. Update example configuration of block state change log in - log4j.properties. (Masatake Iwasaki via aajisaka) - - HDFS-6054. MiniQJMHACluster should not use static port to avoid binding - failure in unit test. (Yongjun Zhang) - - HDFS-9682. Fix a typo "aplication" in HttpFS document. - (Weiwei Yang via aajisaka) - - HDFS-9566. Remove expensive 'BlocksMap#getStorages(Block b, final - DatanodeStorage.State state)' method (Daryn Sharp via vinayakumarb) - - HDFS-9708. FSNamesystem.initAuditLoggers() doesn't trim classnames - (Mingliang Liu via stevel) - - HDFS-9210. Fix some misuse of %n in VolumeScanner#printStats. - (Xiaoyu Yao) - - HDFS-9701. DN may deadlock when hot-swapping under load. (Xiao Chen via lei) - - HDFS-9718. HAUtil#getConfForOtherNodes should unset independent generic keys - before initialize (DENG FEI via vinayakumarb) - - HDFS-9739. DatanodeStorage.isValidStorageId() is broken - (Mingliang Liu via vinayakumarb) - - HDFS-9748. Avoid duplication in pendingReplications when - addExpectedReplicasToPending is called twice. (Walter Su via jing9) - - HDFS-9601. NNThroughputBenchmark.BlockReportStats should handle - NotReplicatedYetException on adding block (iwasakims) - - HDFS-9761. Rebalancer sleeps too long between iterations - (Mingliang Liu via cnauroth) - - HDFS-9713. DataXceiver#copyBlock should return if block is pinned. - (umamahesh) - - HDFS-9760. WebHDFS AuthFilter cannot be configured with custom AltKerberos - auth handler (Ryan Sasson via aw) - - HDFS-9779 . TestReplicationPolicyWithNodeGroup NODE variable picks wrong rack value - (Kuhu Shukla via umamahesh) - - HDFS-9788. Incompatible tag renumbering in HeartbeatResponseProto. (wang) - - HDFS-9790. HDFS Balancer should exit with a proper message if upgrade is - not finalized. (Xiaobing Zhou via Arpit Agarwal) - - HDFS-9801. ReconfigurableBase should update the cached configuration. - (Arpit Agarwal) - - HDFS-9456. BlockPlacementPolicyWithNodeGroup should override - verifyBlockPlacement(). (Xiaobing Zhou via junping_du) - - HDFS-9815. Move o.a.h.fs.Hdfs to hadoop-hdfs-client. - (Vinayakumar B via wheat9) - - HDFS-9799. Reimplement getCurrentTrashDir to remove incompatibility. (zhz) - - HDFS-6832. Fix the usage of 'hdfs namenode' command. - (Manjunath Ballur via aajisaka) - - HDFS-9839. Reduce verbosity of processReport logging. (Arpit Agarwal) - - HDFS-7452. skip StandbyException log for getCorruptFiles() - (Bramha Reddy Battula via vinayakumarb) - - HDFS-9842. dfs.datanode.balance.bandwidthPerSec should accept friendly - size units. (Lin Yiqun via Arpit Agarwal) - - HDFS-9549. TestCacheDirectives#testExceedsCapacity is flaky (Xiao Chen via - cmccabe) - - HDFS-9855. Modify TestAuditLoggerWithCommands to workaround the absence - of HDFS-8332 (Kuhu Shukla via kihwal) - - HDFS-9870. Remove unused imports from DFSUtil. - (Brahma Reddy Battula via cnauroth) - - HDFS-9881. DistributedFileSystem#getTrashRoot returns incorrect path for - encryption zones. (wang) - - HDFS-9886. Configuration properties for hedged read is broken. - (Akira AJISAKA via zhz) - -Release 2.7.3 - UNRELEASED - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - HDFS-7163. WebHdfsFileSystem should retry reads according to the configured - retry policy. (Eric Payne via kihwal) - - HDFS-9574. Reduce client failures during datanode restart (kihwal) - - HDFS-9569. Log the name of the fsimage being loaded for better - supportability. (Yongjun Zhang) - - HDFS-9634. webhdfs client side exceptions don't provide enough details - (Eric Payne via kihwal) - - HDFS-9654. Code refactoring for HDFS-8578. (szetszwo) - - HDFS-9669. TcpPeerServer should respect ipc.server.listen.queue.size - (Elliot Clark via cmccabe) - - HDFS-4946. Allow preferLocalNode in BlockPlacementPolicyDefault to be - configurable (James Kinley and Nathan Roberts via kihwal) - - HDFS-8578. On upgrade, Datanode should process all storage/data dirs in - parallel. (vinayakumarb and szetszwo via szetszwo) - - HDFS-9395. Make HDFS audit logging consistant (Kuhu Shukla via kihwal) - - HDFS-8791. block ID-based DN storage layout can be very slow for datanode - on ext4 (Chris Trezzo via kihwal) - - HDFS-9048. DistCp documentation is out-of-dated - (Daisuke Kobayashi via iwasakims) - - OPTIMIZATIONS - - BUG FIXES - - HDFS-9289. Make DataStreamer#block thread safe and verify genStamp in - commitBlock. (Chang Li via zhz) - - HDFS-4937. ReplicationMonitor can infinite-loop in - BlockPlacementPolicyDefault#chooseRandom(). (kihwal) - - HDFS-9383. TestByteArrayManager#testByteArrayManager fails. - (szetszwo via kihwal) - - HDFS-8785. TestDistributedFileSystem is failing in trunk. (Xiaoyu Yao) - - HDFS-9516. Truncate file fails with data dirs on multiple disks. - (Plamen Jeliazkov via shv) - - HDFS-9533. seen_txid in the shared edits directory is modified during - bootstrapping (kihwal) - - HDFS-9505. HDFS Architecture documentation needs to be refreshed. - (Masatake Iwasaki via aajisaka) - - HDFS-8914. Document HA support in the HDFS HdfsDesign.md. - (Lars Francke via wheat9) - - HDFS-9648. TestStartup.testImageChecksum is broken by HDFS-9569's message - change. (Wei-Chiu Chuang via Yongjun Zhang) - - HDFS-9661. Deadlock in DN.FsDatasetImpl between moveBlockAcrossStorage and - createRbw (ade via vinayakumarb) - - HDFS-9625. set replication for empty file failed when set storage policy - (DENG FEI via vinayakumarb) - - HDFS-9672. o.a.h.hdfs.TestLeaseRecovery2 fails intermittently (Mingliang Liu - via jitendra) - - HDFS-9690. ClientProtocol.addBlock is not idempotent after HDFS-8071. - (szetszwo) - - HDFS-9406. FSImage may get corrupted after deleting snapshot. - (Contributed by Jing Zhao, Stanislav Antic, Vinayakumar B, Yongjun Zhang) - - HDFS-9740. Use a reasonable limit in DFSTestUtil.waitForMetric() - (Chang Li via vinayakumarb) - - HDFS-9730. Storage ID update does not happen when there is a layout change - (Tsz Wo Nicholas Sze via kihwal) - - HDFS-9724. Degraded performance in WebHDFS listing as it does not reuse - ObjectMapper. (Akira AJISAKA via wheat9) - - HDFS-9752. Permanent write failures may happen to slow writers during - datanode rolling upgrades (Walter Su via kihwal) - - HDFS-9784. Example usage is not correct in Transparent Encryption document. - (Takashi Ohnishi via aajisaka) - - HDFS-9765. TestBlockScanner#testVolumeIteratorWithCaching fails - intermittently. (aajisaka) - - HDFS-9864. Correct reference for RENEWDELEGATIONTOKEN and - CANCELDELEGATIONTOKEN in webhdfs doc. (Brahma Reddy Battula via aajisaka) - - HDFS-9880. TestDatanodeRegistration fails occasionally (kihwal) - - HDFS-9766. TestDataNodeMetrics#testDataNodeTimeSpend fails intermittently. - (Xiao Chen via aajisaka) - - HDFS-9851. NameNode throws NPE when setPermission is called on a path that - does not exist. (Brahma Reddy Battula via aajisaka) - -Release 2.7.2 - 2016-01-25 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - HDFS-8659. Block scanner INFO message is spamming logs. (Yongjun Zhang) - - HDFS-8099. Change "DFSInputStream has been closed already" message to - debug log level (Charles Lamb via Colin P. McCabe) - - HDFS-9221. HdfsServerConstants#ReplicaState#getState should avoid calling - values() since it creates a temporary array. (Staffan Friberg via yliu) - - OPTIMIZATIONS - - HDFS-8722. Optimize datanode writes for small writes and flushes (kihwal) - - BUG FIXES - - HDFS-6945. BlockManager should remove a block from excessReplicateMap and - decrement ExcessBlocks metric when the block is removed. (aajisaka) - - HDFS-8806. Inconsistent metrics: number of missing blocks with replication - factor 1 not properly cleared. (Zhe Zhang via aajisaka) - - HDFS-8852. HDFS architecture documentation of version 2.x is outdated - about append write support. (Ajith S via aajisaka) - - HDFS-8867. Enable optimized block reports. (Daryn Sharp via jing9) - - HDFS-8891. HDFS concat should keep srcs order. (Yong Zhang via jing9) - - HDFS-8995. Flaw in registration bookeeping can make DN die on reconnect. - (Kihwal Lee via yliu) - - HDFS-9033. dfsadmin -metasave prints "NaN" for cache used%. - (Brahma Reddy Battula via aajisaka) - - HDFS-9042. Update document for the Storage policy name - (J.Andreina via vinayakumarb) - - HDFS-9043. Doc updation for commands in HDFS Federation - (J.Andreina via vinayakumab) - - HDFS-9106. Transfer failure during pipeline recovery causes permanent - write failures (kihwal) - - HDFS-8850. VolumeScanner thread exits with exception if there is no block - pool to be scanned but there are suspicious blocks. (Colin Patrick McCabe - via yliu) - - HDFS-9178. Slow datanode I/O can cause a wrong node to be marked bad - (kihwal) - - HDFS-8676. Delayed rolling upgrade finalization can cause heartbeat - expiration. (Walter Su via kihwal) - - HDFS-9220. Reading small file (< 512 bytes) that is open for append fails - due to incorrect checksum (Jing Zhao via kihwal) - - HDFS-9290. DFSClient#callAppend() is not backward compatible for slightly - older NameNodes. (Tony Wu via kihwal) - - HDFS-9305. Delayed heartbeat processing causes storm of subsequent - heartbeats. (Arpit Agarwal) - - HDFS-9317. Document fsck -blockId and -storagepolicy options in branch-2.7. - (aajisaka) - - HDFS-6481. DatanodeManager#getDatanodeStorageInfos() should check the - length of storageIDs. (szetszwo via Arpit Agarwal) - - HDFS-9426. Rollingupgrade finalization is not backward compatible - (Kihwal Lee via vinayakumarb) - - HDFS-9445. Datanode may deadlock while handling a bad volume. - (Wlater Su via Kihwal) - - HDFS-8326. Documentation about when checkpoints are run is out of date. - (Misty Stanley-Jones via xyao) - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - HDFS-8081. Split getAdditionalBlock() into two methods. (shv) - - HDFS-7931. DistributedFileSystem should not look for keyProvider in - cache if Encryption is disabled (asuresh) - - HDFS-8204. Mover/Balancer should not schedule two replicas to the same - datanode. (Walter Su via szetszwo) - - HDFS-7770. Need document for storage type label of data node storage - locations under dfs.data.dir. (Xiaoyu Yao via aajisaka) - - HDFS-8213. DFSClient should use hdfs.client.htrace HTrace configuration - prefix rather than hadoop.htrace (cmccabe) - - HDFS-8521. Add VisibleForTesting annotation to - BlockPoolSlice#selectReplicaToDelete. (cmccabe) - - HDFS-8361. Choose SSD over DISK in block placement. (szetszwo) - - HDFS-7546. Document, and set an accepting default for - dfs.namenode.kerberos.principal.pattern (Harsh J via aw) - - HDFS-7164. Feature documentation for HDFS-6581. (Arpit Agarwal) - - HDFS-8143. Mover should exit after some retry when failed to move blocks. - (Surendra Singh Lilhore via szetszwo) - - OPTIMIZATIONS - - BUG FIXES - - HDFS-8151. Always use snapshot path as source when invalid snapshot names - are used for diff based distcp. (jing9) - - HDFS-7934. Update RollingUpgrade rollback documentation: should use - bootstrapstandby for standby NN. (J. Andreina via jing9) - - HDFS-8149. The footer of the Web UI "Hadoop, 2014" is old. - (Brahma Reddy Battula via aajisaka) - - HDFS-8153. Error Message points to wrong parent directory in case of - path component name length error (Anu Engineer via jitendra) - - HDFS-8179. DFSClient#getServerDefaults returns null within 1 - hour of system start. (Xiaoyu Yao via Arpit Agarwal) - - HDFS-8163. Using monotonicNow for block report scheduling causes - test failures on recently restarted systems. (Arpit Agarwal) - - HDFS-8147. StorageGroup in Dispatcher should override equals nad hashCode. - (surendra singh lilhore via szetszwo) - - HDFS-8273. FSNamesystem#Delete() should not call logSync() when holding the - lock. (wheat9) - - HDFS-8269. getBlockLocations() does not resolve the .reserved path and - generates incorrect edit logs when updating the atime. (wheat9) - - HDFS-8091: ACLStatus and XAttributes should be presented to - INodeAttributesProvider before returning to client (asuresh) - - HDFS-8305: HDFS INotify: the destination field of RenameOp should always - end with the file name (cmccabe) - - HDFS-8226. Non-HA rollback compatibility broken (J.Andreina via vinayakumarb) - - HDFS-7916. 'reportBadBlocks' from datanodes to standby Node BPServiceActor - goes for infinite loop (Rushabh S Shah via kihwal) - - HDFS-6300. Prevent multiple balancers from running simultaneously - (Rakesh R via vinayakumarb) - - HDFS-8405. Fix a typo in NamenodeFsck. (Takanobu Asanuma via szetszwo) - - HDFS-8451. DFSClient probe for encryption testing interprets empty URI - property for "enabled". (Steve Loughran via xyao) - - HDFS-8523. Remove usage information on unsupported operation - "fsck -showprogress" from branch-2 (J.Andreina via vinayakumarb) - - HDFS-8522. Change heavily recorded NN logs from INFO to DEBUG level. (xyao) - - HDFS-8566. HDFS documentation about debug commands wrongly identifies them - as "hdfs dfs" commands (Surendra Singh Lilhore via Colin P. McCabe) - - HDFS-8583. Document that NFS gateway does not work with rpcbind - on SLES 11. (Arpit Agarwal) - - HDFS-8572. DN always uses HTTP/localhost@REALM principals in SPNEGO. - (wheat9) - - HDFS-8596. TestDistributedFileSystem et al tests are broken in branch-2 - due to incorrect setting of "datanode" attribute. (Yongjun Zhang) - - HDFS-8595. TestCommitBlockSynchronization fails in branch-2.7. (Patch - applies to all branches). (Arpit Agarwal) - - HDFS-8576. Lease recovery should return true if the lease can be released - and the file can be closed. (J.Andreina via szetszwo) - - HDFS-8544. Incorrect port specified in HFTP Guide document in branch-2. - (Brahma Reddy Battula via Yongjun Zhang) - - HDFS-8597. Fix TestFSImage#testZeroBlockSize on Windows. (Xiaoyu Yao) - - HDFS-4660. Block corruption can happen during pipeline recovery (kihwal) - - HDFS-8633. Fix setting of dfs.datanode.readahead.bytes in hdfs-default.xml - to match DFSConfigKeys. (Ray Chiang via Yongjun Zhang) - - HDFS-8626. Reserved RBW space is not released if creation of RBW File - fails. (kanaka kumar avvaru via Arpit Agarwal) - - HDFS08656. Preserve compatibility of ClientProtocol#rollingUpgrade after - finalization. (wang) - - HDFS-8681. BlockScanner is incorrectly disabled by default. - (Arpit Agarwal) - -Release 2.7.0 - 2015-04-20 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - HDFS-6663. Admin command to track file and locations from block id. - (Chen He via kihwal) - - HDFS-6982. nntop: topÂ-like tool for name node users. - (Maysam Yabandeh via wang) - - HDFS-7424. Add web UI for NFS gateway (brandonli) - - HDFS-7449. Add metrics to NFS gateway (brandonli) - - HDFS-3107. Introduce truncate. (Plamen Jeliazkov via shv) - - HDFS-7056. Snapshot support for truncate. (Plamen Jeliazkov and shv) - - HDFS-6673. Add delimited format support to PB OIV tool. (Eddy Xu via wang) - - HDFS-7655. Expose truncate API for Web HDFS. (yliu) - - HDFS-6133. Add a feature for replica pinning so that a pinned replica - will not be moved by Balancer/Mover. (zhaoyunjiong via szetszwo) - - HDFS-3689. Add support for variable length block. Contributed by Jing Zhao. - - HDFS-7584. Enable Quota Support for Storage Types (See breakdown of - tasks below) - - HDFS-7656. Expose truncate API for HDFS httpfs. (yliu) - - HDFS-6488. Support HDFS superuser in NFS gateway. (brandonli) - - HDFS-7838. Expose truncate API for libhdfs. (yliu) - - HDFS-6826. Plugin interface to enable delegation of HDFS authorization - assertions. (Arun Suresh via jitendra) - - IMPROVEMENTS - - HDFS-7752. Improve description for - "dfs.namenode.num.extra.edits.retained" - and "dfs.namenode.num.checkpoints.retained" properties on - hdfs-default.xml (Wellington Chevreuil via harsh) - - HDFS-7055. Add tracing to DFSInputStream (cmccabe) - - HDFS-7186. Document the "hadoop trace" command. (Masatake Iwasaki via Colin - P. McCabe) - - HDFS-7202. Should be able to omit package name of SpanReceiver on "hadoop - trace -add" (iwasakims via cmccabe) - - HDFS-7026. Introduce a string constant for "Failed to obtain user group - info...". (Yongjun Zhang via atm) - - HDFS-7209. Populate EDEK cache when creating encryption zone. (Yi Liu via wang) - - HDFS-6252. Phase out the old web UI in HDFS. (wheat9) - - HDFS-7266. HDFS Peercache enabled check should not lock on object (awang - via cmccabe) - - HDFS-7254. Add documentation for hot swaping DataNode drives (Lei Xu via - Colin P. McCabe) - - HDFS-6877. Avoid calling checkDisk when an HDFS volume is removed during a - write. (Lei Xu via Colin P. McCabe) - - HDFS-2486. Remove unnecessary priority level checks in - UnderReplicatedBlocks. (Uma Maheswara Rao G via szetszwo) - - HDFS-6824. Additional user documentation for HDFS encryption. (wang) - - HDFS-7165. Separate block metrics for files with replication count 1. - (Zhe Zhang via wang) - - HDFS-7222. Expose DataNode network errors as a metric. (Charles Lamb via wang) - - HDFS-7257. Add the time of last HA state transition to NN's /jmx page. - (Charles Lamb via wheat9) - - HDFS-7223. Tracing span description of IPC client is too long (iwasakims - via cmccabe) - - HDFS-7283. Bump DataNode OOM log from WARN to ERROR. - (Stephen Chu via wheat9) - - HDFS-5928. Show namespace and namenode ID on NN dfshealth page. - (Siqi Li via wheat9) - - HDFS-7280. Use netty 4 in WebImageViewer. (wheat9) - - HDFS-3342. SocketTimeoutException in BlockSender.sendChunks could - have a better error message. (Yongjun Zhang via wang) - - HDFS-6917. Add an hdfs debug command to validate blocks, call recoverlease, - etc. (cmccabe) - - HDFS-7356. Use DirectoryListing.hasMore() directly in nfs. (Li Lu via jing9) - - HDFS-7357. FSNamesystem.checkFileProgress should log file path. - (Tsz Wo Nicholas Sze via wheat9) - - HDFS-7335. Redundant checkOperation() in FSN.analyzeFileState(). - (Milan Desai via shv) - - HDFS-7333. Improve logging in Storage.tryLock(). (shv) - - HDFS-7361. TestCheckpoint fails after change of log message related to - locking violation. (shv) - - HDFS-7329. Improve logging when MiniDFSCluster fails to start. - (Byron Wong via shv) - - HDFS-7336. Unused member DFSInputStream.buffersize. (Milan Desai via shv) - - HDFS-7365. Remove hdfs.server.blockmanagement.MutableBlockCollection. - (Li Lu via wheat9) - - HDFS-7381. Decouple the management of block id and gen stamps from - FSNamesystem. (wheat9) - - HDFS-7375. Move FSClusterStats to o.a.h.h.hdfs.server.blockmanagement. - (wheat9) - - HDFS-7386. Replace check "port number < 1024" with shared isPrivilegedPort - method. (Yongjun Zhang via cnauroth) - - HDFS-7394. Log at INFO level, not WARN level, when InvalidToken is seen in - ShortCircuitCache (Keith Pak via Colin P. McCabe) - - HDFS-7279. Use netty to implement DatanodeWebHdfsMethods. (wheat9) - - HDFS-7404. Remove o.a.h.hdfs.server.datanode.web.resources. - (Li Lu via wheat9) - - HDFS-7398. Reset cached thread-local FSEditLogOp's on every - FSEditLog#logEdit. (Gera Shegalov via cnauroth) - - HDFS-7409. Allow dead nodes to finish decommissioning if all files are - fully replicated. (wang) - - HDFS-7413. Some unit tests should use NameNodeProtocols instead of - FSNameSystem. (wheat9) - - HDFS-7415. Move FSNameSystem.resolvePath() to FSDirectory. (wheat9) - - HDFS-7420. Delegate permission checks to FSDirectory. (wheat9) - - HDFS-7331. Add Datanode network counts to datanode jmx page. (Charles Lamb - via atm) - - HDFS-7412. Move RetryCache to NameNodeRpcServer. (wheat9) - - HDFS-7419. Improve error messages for DataNode hot swap drive feature (Lei - Xu via Colin P. Mccabe) - - HDFS-7436. Consolidate implementation of concat(). (wheat9) - - HDFS-7440. Consolidate snapshot related operations in a single class. - (wheat9) - - HDFS-6803 Document DFSClient#DFSInputStream expectations reading and preading - in concurrent context. (stack via stevel) - - HDFS-7310. Mover can give first priority to local DN if it has target storage type - available in local DN. (Vinayakumar B via umamahesh) - - HDFS-7210. Avoid two separate RPC's namenode.append() and namenode.getFileInfo() - for an append call from DFSClient. (Vinayakumar B via umamahesh) - - HDFS-745
<TRUNCATED>
