[jira] [Created] (HDFS-9518) LeaseRenewer - Improve Renew Method
BELUGA BEHR created HDFS-9518: - Summary: LeaseRenewer - Improve Renew Method Key: HDFS-9518 URL: https://issues.apache.org/jira/browse/HDFS-9518 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs-client Affects Versions: 2.7.1 Reporter: BELUGA BEHR Priority: Minor Replaced the current implementation of org.apache.hadoop.hdfs.client.impl.LeaseRenewer#renew() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-9551) Random VolumeChoosingPolicy
BELUGA BEHR created HDFS-9551: - Summary: Random VolumeChoosingPolicy Key: HDFS-9551 URL: https://issues.apache.org/jira/browse/HDFS-9551 Project: Hadoop HDFS Issue Type: Improvement Components: datanode Affects Versions: 2.7.1 Reporter: BELUGA BEHR Priority: Minor Please find attached a new implementation of VolumeChoosingPolicy. This implementation chooses volumes at random to place blocks. It is thread-safe and un-synchronized so there is less thread contention. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-9560) Fair AvailableSpaceVolumeChoosingPolicy
BELUGA BEHR created HDFS-9560: - Summary: Fair AvailableSpaceVolumeChoosingPolicy Key: HDFS-9560 URL: https://issues.apache.org/jira/browse/HDFS-9560 Project: Hadoop HDFS Issue Type: Improvement Reporter: BELUGA BEHR Priority: Minor I took a look at AvailableSpaceVolumeChoosingPolicy. It seems a bit overkill and includes some configuration items that seem a bit arbitrary with no real clear guidance on how to effectively use them: _dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction_ _dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold_ I have created an alternative implementation that does not require any external configuration, is thread-safe, and requires no synchronization. "Weighted Randomized Ordering" http://stackoverflow.com/questions/23971365/weighted-randomized-ordering Conceptually, a dart-board is constructed of several wedges, each wedge represents a disk volume. The more available space that a volume has relative to the other volumes, the larger its wedge. Then, a dart is thrown at the board and whichever wedge(volume) the dart lands on, that wedge is assigned the incoming block. Over time, the wedges balance and all have an equal chance of being "hit." -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-9593) BlockTokenIdentifier Code Review
BELUGA BEHR created HDFS-9593: - Summary: BlockTokenIdentifier Code Review Key: HDFS-9593 URL: https://issues.apache.org/jira/browse/HDFS-9593 Project: Hadoop HDFS Issue Type: Improvement Reporter: BELUGA BEHR Priority: Trivial Please find attached a code review of BlockTokenIdentifier. The only real change is using an explicit StringBuilder instead of an implicit one so that the internal buffer may be set more appropriately. The relatively small default buffer of 16 bytes would need to be expanded several times to fit the incoming string. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-9596) Remove "Shuffle" Method From DFSUtil
BELUGA BEHR created HDFS-9596: - Summary: Remove "Shuffle" Method From DFSUtil Key: HDFS-9596 URL: https://issues.apache.org/jira/browse/HDFS-9596 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 2.7.1 Reporter: BELUGA BEHR Priority: Trivial Attachments: Shuffle.HDFS-9596.patch DFSUtil contains a "shuffle" routine that shuffles arrays. With a little wrapping, Java Collections class can shuffle. This code is superfluous and undocumented. The method returns an array, though it does not specify if it returns the same array or a new, copy, array of shuffled items. It is only used in one place, so it is better to simply remove and substitute this implementation with the java Collections implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-11931) FileSystem Simplify / Optimize listStatus Method
BELUGA BEHR created HDFS-11931: -- Summary: FileSystem Simplify / Optimize listStatus Method Key: HDFS-11931 URL: https://issues.apache.org/jira/browse/HDFS-11931 Project: Hadoop HDFS Issue Type: Improvement Components: fs Affects Versions: 3.0.0-alpha3, 2.7.3 Reporter: BELUGA BEHR Priority: Minor {code:title=org.apache.hadoop.fs.FileSystem.listStatus(ArrayList, Path, PathFilter)} /* * Filter files/directories in the given path using the user-supplied path * filter. Results are added to the given array results. */ private void listStatus(ArrayList results, Path f, PathFilter filter) throws FileNotFoundException, IOException { FileStatus listing[] = listStatus(f); if (listing == null) { throw new IOException("Error accessing " + f); } for (int i = 0; i < listing.length; i++) { if (filter.accept(listing[i].getPath())) { results.add(listing[i]); } } } {code} {code:title=org.apache.hadoop.fs.FileSystem.listStatus(Path, PathFilter)} public FileStatus[] listStatus(Path f, PathFilter filter) throws FileNotFoundException, IOException { ArrayList results = new ArrayList(); listStatus(results, f, filter); return results.toArray(new FileStatus[results.size()]); } {code} We can be smarter about this: # Use enhanced for-loops # Optimize for the case where there are zero files in a directory, save on object instantiation # More encapsulated design -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13947) Review of DirectoryScanner Class
BELUGA BEHR created HDFS-13947: -- Summary: Review of DirectoryScanner Class Key: HDFS-13947 URL: https://issues.apache.org/jira/browse/HDFS-13947 Project: Hadoop HDFS Issue Type: Improvement Components: datanode Affects Versions: 3.2.0 Reporter: BELUGA BEHR Review of Directory Scanner. Replaced a lot of code with Guava MultiMap. Some general house cleaning and improved logging. For performance, using {{ArrayList}} instead of {{LinkedList}} where possible, especially since these lists can be quite large a LinkedList will consume a lot of memory and be slow to sort/iterate over. https://stackoverflow.com/questions/322715/when-to-use-linkedlist-over-arraylist-in-java -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13958) Miscellaneous Improvements for FsVolumeSpi
BELUGA BEHR created HDFS-13958: -- Summary: Miscellaneous Improvements for FsVolumeSpi Key: HDFS-13958 URL: https://issues.apache.org/jira/browse/HDFS-13958 Project: Hadoop HDFS Issue Type: Improvement Components: datanode Affects Versions: 3.3.0 Reporter: BELUGA BEHR The work in [HDFS-13947] allowed for using {{ArrayList}} instead of {{LinkedList}} when scanning DataNode local directories, however the {{FsVolumeSpi}} implementations were still using (and forcing) {{LinkedList}}. I propose changing the {{FsVolumeSpi}} signatures to allow for {{Collection}} instead of {{LinkedList}}. Since I'm looking at the code, I made some small improvements and check-style fixes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13967) HDFS Router Quota Class Review
BELUGA BEHR created HDFS-13967: -- Summary: HDFS Router Quota Class Review Key: HDFS-13967 URL: https://issues.apache.org/jira/browse/HDFS-13967 Project: Hadoop HDFS Issue Type: Improvement Components: federation, hdfs Affects Versions: 3.2.0 Reporter: BELUGA BEHR Attachments: HDFS-13967.1.patch # Use Guava library # Prefer ArrayList over LinkedList # Simplify code logic -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13968) BlockReceiver Array-Based Queue
BELUGA BEHR created HDFS-13968: -- Summary: BlockReceiver Array-Based Queue Key: HDFS-13968 URL: https://issues.apache.org/jira/browse/HDFS-13968 Project: Hadoop HDFS Issue Type: Improvement Components: datanode Affects Versions: 3.2.0 Environment: # Prefer Array implementation of Queue over LinkedList # Check-style clean up http://brianandstuff.com/2016/12/12/java-arraydeque-vs-linkedlist/ Reporter: BELUGA BEHR Attachments: HDFS-13968.1.patch -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13969) Server.java Prefer Array List
BELUGA BEHR created HDFS-13969: -- Summary: Server.java Prefer Array List Key: HDFS-13969 URL: https://issues.apache.org/jira/browse/HDFS-13969 Project: Hadoop HDFS Issue Type: Improvement Components: ipc Affects Versions: 3.2.0 Reporter: BELUGA BEHR Attachments: HDFS-13969.1.patch * Prefer ArrayDeque over LinkedList (faster, less memory overhead) * Address this code: {code} // // Remove calls that have been pending in the responseQueue // for a long time. // private void doPurge(RpcCall call, long now) { LinkedList responseQueue = call.connection.responseQueue; synchronized (responseQueue) { Iterator iter = responseQueue.listIterator(0); while (iter.hasNext()) { call = iter.next(); if (now > call.timestamp + PURGE_INTERVAL) { closeConnection(call.connection); break; } } } } {code} It says "Remove calls" (plural) but only one call will be removed because of the 'break' statement. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13970) CacheManager Directives Map
BELUGA BEHR created HDFS-13970: -- Summary: CacheManager Directives Map Key: HDFS-13970 URL: https://issues.apache.org/jira/browse/HDFS-13970 Project: Hadoop HDFS Issue Type: Improvement Components: caching, hdfs Affects Versions: 3.2.0 Reporter: BELUGA BEHR Attachments: HDFS-13970.1.patch # Use Guava Multimap to simplify code ## Currently, code uses a mix of LinkedList and ArrayList - just pick one ## Currently, {{directivesByPath}} structure is sorted but never used in a sorted way, it only performs remove and add operations, no iteration - use a {{Set}} instead of a {{List}} # The {{CacheDirective}} class needs a better hashcode implementation since it will be used in a Set. Do not instantiate a {{HashBuilder}} object every time {{hashcode}} is called. Ouch. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13971) Review of NetgroupCache
BELUGA BEHR created HDFS-13971: -- Summary: Review of NetgroupCache Key: HDFS-13971 URL: https://issues.apache.org/jira/browse/HDFS-13971 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs, security Affects Versions: 3.2.0 Reporter: BELUGA BEHR * Simplify code and performance by using Guava Multimap -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13978) Review of DiskBalancer
BELUGA BEHR created HDFS-13978: -- Summary: Review of DiskBalancer Key: HDFS-13978 URL: https://issues.apache.org/jira/browse/HDFS-13978 Project: Hadoop HDFS Issue Type: Improvement Components: datanode, hdfs Affects Versions: 3.2.0 Environment: * Use ArrayList instead of LinkedList. Especially because this code here uses the {{List#get()}} method, which is very slow for the LinkedList since it has to traverse every node to get to the offset: O(N^2) {code} ExtendedBlock getNextBlock(List poolIters, DiskBalancerWorkItem item) { Preconditions.checkNotNull(poolIters); int currentCount = 0; ExtendedBlock block = null; while (block == null && currentCount < poolIters.size()) { currentCount++; int index = poolIndex++ % poolIters.size(); FsVolumeSpi.BlockIterator currentPoolIter = poolIters.get(index); block = getBlockToCopy(currentPoolIter, item); } {code} * Do not "log and throw" errors. This is an anti-pattern and should be avoided. Log or throw, but don't do both. Removed some logging * Improved other logging statements * Improved the {{hashcode}} method of one of the inner classes. It was instantiating a List and performing iteration for every call. Replace with Eclipse-generated hashcode method. * Fixed compiler warnings for deprecated code or code that was not parameterized * Fix check style issue Reporter: BELUGA BEHR Attachments: HDFS-13978.1.patch -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13979) Review StateStoreFileSystemImpl Class
BELUGA BEHR created HDFS-13979: -- Summary: Review StateStoreFileSystemImpl Class Key: HDFS-13979 URL: https://issues.apache.org/jira/browse/HDFS-13979 Project: Hadoop HDFS Issue Type: Improvement Components: federation Affects Versions: 3.2.0 Reporter: BELUGA BEHR * Replace {{LinkedList}} with pre-sized {{ArrayList}} * Added exception information to WARN/ERROR logging -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13980) Review of DBNameNodeConnector Class
BELUGA BEHR created HDFS-13980: -- Summary: Review of DBNameNodeConnector Class Key: HDFS-13980 URL: https://issues.apache.org/jira/browse/HDFS-13980 Project: Hadoop HDFS Issue Type: Improvement Components: diskbalancer Affects Versions: 3.2.0 Environment: * Removed anti-pattern "log and throw". Log or throw, don't do both * Use pre-sized {{ArrayList}} instead of {{LinkedList}} * Fixed some formatting issues * Re-ordered imports into correct order Reporter: BELUGA BEHR Attachments: HDFS-13980.1.patch -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13981) Review of AvailableSpaceResolver.java
BELUGA BEHR created HDFS-13981: -- Summary: Review of AvailableSpaceResolver.java Key: HDFS-13981 URL: https://issues.apache.org/jira/browse/HDFS-13981 Project: Hadoop HDFS Issue Type: Improvement Components: federation Affects Versions: 3.2.0 Reporter: BELUGA BEHR Attachments: HDFS-13981.1.patch * No behavior changes, just optimizing and paring down the code -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13987) Review of RandomResolver Class
BELUGA BEHR created HDFS-13987: -- Summary: Review of RandomResolver Class Key: HDFS-13987 URL: https://issues.apache.org/jira/browse/HDFS-13987 Project: Hadoop HDFS Issue Type: Improvement Components: federation Affects Versions: 3.2.0 Reporter: BELUGA BEHR Attachments: HDFS-13987.1.patch * Use {{ThreadLocalRandom}} * Do not create a list, and copy, to pick a random index * An early-return in the method means that the 'ERROR' logging is skipped in some cases -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13988) Allow for Creating Temporary Files
BELUGA BEHR created HDFS-13988: -- Summary: Allow for Creating Temporary Files Key: HDFS-13988 URL: https://issues.apache.org/jira/browse/HDFS-13988 Project: Hadoop HDFS Issue Type: New Feature Components: hdfs, hdfs-client Affects Versions: 3.2.0 Reporter: BELUGA BEHR Allow clients to create temporary files in HDFS. {code:java|title=FileSystem.java} public FSDataOutputStreamBuilder createFile(Path path); public FSDataOutputStreamBuilder appendFile(Path path); // New feature public FSDataOutputStreamBuilder createTempFile(Path path, TimeUnit unit, long ttl); {code} This would register the file with the NameNode to have a TTL. The NameNode would be responsible for deleting the files after some custom TTL. This would be helpful in the case that a MapReduce (or Spark) job fails during execution and does not properly clean up after itself. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-13988) Allow for Creating Temporary Files
[ https://issues.apache.org/jira/browse/HDFS-13988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR resolved HDFS-13988. Resolution: Duplicate > Allow for Creating Temporary Files > -- > > Key: HDFS-13988 > URL: https://issues.apache.org/jira/browse/HDFS-13988 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs, hdfs-client >Affects Versions: 3.2.0 >Reporter: BELUGA BEHR >Priority: Major > > Allow clients to create temporary files in HDFS. > {code:java|title=FileSystem.java} > public FSDataOutputStreamBuilder createFile(Path path); > public FSDataOutputStreamBuilder appendFile(Path path); > // New feature > public FSDataOutputStreamBuilder createTempFile(Path path, TimeUnit unit, > long ttl); > {code} > This would register the file with the NameNode to have a TTL. The NameNode > would be responsible for deleting the files after some custom TTL. This > would be helpful in the case that a MapReduce (or Spark) job fails during > execution and does not properly clean up after itself. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13990) Synchronization Issue With HashResolver.java
BELUGA BEHR created HDFS-13990: -- Summary: Synchronization Issue With HashResolver.java Key: HDFS-13990 URL: https://issues.apache.org/jira/browse/HDFS-13990 Project: Hadoop HDFS Issue Type: Bug Components: federation Affects Versions: 3.2.0 Reporter: BELUGA BEHR Attachments: HDFS-13990.1.patch {code:java|title=HashResolver.java} private ConsistentHashRing getHashResolver(final Set namespaces) { int hash = namespaces.hashCode(); ConsistentHashRing resolver = this.hashResolverMap.get(hash); if (resolver == null) { resolver = new ConsistentHashRing(namespaces); this.hashResolverMap.put(hash, resolver); } return resolver; } {code} The {{hashResolverMap}} is a {{ConcurrentHashMap}} so presumably there is concern here for concurrency issues. However, there is no synchronization around this method, so two threads could call {{get(hash)}} both see a 'null' value and then both add two entries into the {{Map}}. Add synchronization here. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13991) Review of DiskBalancerCluster.java
BELUGA BEHR created HDFS-13991: -- Summary: Review of DiskBalancerCluster.java Key: HDFS-13991 URL: https://issues.apache.org/jira/browse/HDFS-13991 Project: Hadoop HDFS Issue Type: Improvement Components: diskbalancer Affects Versions: 3.2.0 Reporter: BELUGA BEHR Attachments: HDFS-13991.1.patch * Use {{ArrayList}} instead of {{LinkedList}} * Simplify the code * Use {{File}} object's file system agnostic features * Re-order import statements -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13994) DataNode BlockSender waitForMinLength
BELUGA BEHR created HDFS-13994: -- Summary: DataNode BlockSender waitForMinLength Key: HDFS-13994 URL: https://issues.apache.org/jira/browse/HDFS-13994 Project: Hadoop HDFS Issue Type: Improvement Components: datanode Affects Versions: 3.2.0 Reporter: BELUGA BEHR Attachments: HDFS-13994.1.patch {code:java|title=BlockSender.java} private static void waitForMinLength(ReplicaInPipeline rbw, long len) throws IOException { // Wait for 3 seconds for rbw replica to reach the minimum length for (int i = 0; i < 30 && rbw.getBytesOnDisk() < len; i++) { try { Thread.sleep(100); } catch (InterruptedException ie) { throw new IOException(ie); } } long bytesOnDisk = rbw.getBytesOnDisk(); if (bytesOnDisk < len) { throw new IOException( String.format("Need %d bytes, but only %d bytes available", len, bytesOnDisk)); } } {code} It is not very efficient to poll for status in this way. Instead, use {{notifyAll}} within the {{ReplicaInPipeline}} to notify the caller when the replica has reached a certain size. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14102) verifyBlockPlacement
BELUGA BEHR created HDFS-14102: -- Summary: verifyBlockPlacement Key: HDFS-14102 URL: https://issues.apache.org/jira/browse/HDFS-14102 Project: Hadoop HDFS Issue Type: Improvement Reporter: BELUGA BEHR {code:java|title=BlockPlacementPolicyDefault.java} // 1. Check that all locations are different. // 2. Count locations on different racks. Set racks = new TreeSet<>(); for (DatanodeInfo dn : locs) racks.add(dn.getNetworkLocation()); ... racks.size(){code} Here, the code is counting the number of distinct Network Locations. However, it is using a TreeSet which has overhead to maintain item order and uses a linked structure internally. This overhead is unneeded since all that is required here is a count. {quote}A NavigableSet implementation based on a TreeMap. The elements are ordered using their natural ordering, or by a Comparator provided at set creation time, depending on which constructor is used. This implementation provides guaranteed log(n) time cost for the basic operations (add, remove and contains). [https://docs.oracle.com/javase/7/docs/api/java/util/TreeSet.html] {quote} Use Java streams for readability and because it uses a {{HashSet}} under the covers to perform the distinct action. {{HashSet}} uses an array internally and has constant time performance for the {{add}} method. [https://github.com/apache/hadoop/blob/27978bcb66a9130cbf26d37ec454c0b7fcdc2530/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java#L1042] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14103) Review Logging of BlockPlacementPolicyDefault
BELUGA BEHR created HDFS-14103: -- Summary: Review Logging of BlockPlacementPolicyDefault Key: HDFS-14103 URL: https://issues.apache.org/jira/browse/HDFS-14103 Project: Hadoop HDFS Issue Type: Improvement Reporter: BELUGA BEHR Attachments: HDFS-14103.1.patch Review use of SLF4J in {{BlockPlacementPolicyDefault.java}} Other minor logging improvements. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14104) Review getImageTxIdToRetain
BELUGA BEHR created HDFS-14104: -- Summary: Review getImageTxIdToRetain Key: HDFS-14104 URL: https://issues.apache.org/jira/browse/HDFS-14104 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Affects Versions: 3.0.0 Reporter: BELUGA BEHR {code:java|title=NNStorageRetentionManager.java} private long getImageTxIdToRetain(FSImageTransactionalStorageInspector inspector) { List images = inspector.getFoundImages(); TreeSet imageTxIds = Sets.newTreeSet(); for (FSImageFile image : images) { imageTxIds.add(image.getCheckpointTxId()); } List imageTxIdsList = Lists.newArrayList(imageTxIds); if (imageTxIdsList.isEmpty()) { return 0; } Collections.reverse(imageTxIdsList); int toRetain = Math.min(numCheckpointsToRetain, imageTxIdsList.size()); long minTxId = imageTxIdsList.get(toRetain - 1); LOG.info("Going to retain " + toRetain + " images with txid >= " + minTxId); return minTxId; } {code} # Fix check style issues # Use SLF4J paramaterized logging # A lot of work gets done before checking if the list actually contains any entries and returning a 0. That should be the first thing that happens # Instead of building up the {{TreeSet}} in its natural order, then reversing the collection, simply use a reverse natural ordering to begin with and save a step. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14105) NamenodeFsck HashSet
BELUGA BEHR created HDFS-14105: -- Summary: NamenodeFsck HashSet Key: HDFS-14105 URL: https://issues.apache.org/jira/browse/HDFS-14105 Project: Hadoop HDFS Issue Type: Improvement Reporter: BELUGA BEHR Attachments: HDFS-14105.1.patch Use {{HashSet}} instead of {{TreeSet}}. {{TreeSet}} has the overhead of keeping elements in order even though ordering is not taken into consideration in this class. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14106) Improve NamenodeFsck copyBlock
BELUGA BEHR created HDFS-14106: -- Summary: Improve NamenodeFsck copyBlock Key: HDFS-14106 URL: https://issues.apache.org/jira/browse/HDFS-14106 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Affects Versions: 3.2.0 Environment: # Code is performing copy with a 1K buffer. 8K is the standard these ways # Improve code design; do not catch one's own exception, do not log and throw (only do one or the other, never both) # Refactor to make a new method for copy Reporter: BELUGA BEHR Attachments: HDFS-14106.1.patch -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14107) FileContext Delete on Exit Improvements
BELUGA BEHR created HDFS-14107: -- Summary: FileContext Delete on Exit Improvements Key: HDFS-14107 URL: https://issues.apache.org/jira/browse/HDFS-14107 Project: Hadoop HDFS Issue Type: Improvement Components: fs Affects Versions: 3.2.0 Reporter: BELUGA BEHR {code:java|FileContext.java} synchronized (DELETE_ON_EXIT) { Set>> set = DELETE_ON_EXIT.entrySet(); for (Entry> entry : set) { FileContext fc = entry.getKey(); Set paths = entry.getValue(); for (Path path : paths) { try { fc.delete(path, true); } catch (IOException e) { LOG.warn("Ignoring failure to deleteOnExit for path " + path); } } } DELETE_ON_EXIT.clear(); {code} # Include the {{IOException}} in the logging so that admins can know why the file was not deleted # Do not bother clearing out the data structure. This code is only called if the JVM is going down. Better to spend the time allowing another shutdown hook to run than to spend time cleaning this thing up. # Use Guava {{MultiMap}} for readability # Paths are currently stored in a {{TreeSet}}. This set implementation orders the files by names. It does not seem worth much to order the files. Use a faster {{HashSet}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14108) BlockManager Data Structures
BELUGA BEHR created HDFS-14108: -- Summary: BlockManager Data Structures Key: HDFS-14108 URL: https://issues.apache.org/jira/browse/HDFS-14108 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs Affects Versions: 3.2.0 Reporter: BELUGA BEHR Assignee: BELUGA BEHR Attachments: HDFS-14108.1.patch # Prefer {{ArrayList}} to {{LinkedList}} when simply adding/iterating # Prefer {{HashSet}} to {{TreeSet}} when no ordering is required # Other performance improvements # Check style fixes https://stackoverflow.com/questions/322715/when-to-use-linkedlist-over-arraylist-in-java {code:java} final Set excludedNodes = new HashSet<>(); for(BlockReconstructionWork rw : reconWork){ // Do no bother wasting time clearing out the collection, let GC do that work later excludedNodes.clear(); // use {{addAll}} here for (DatanodeDescriptor dn : rw.getContainingNodes()) { excludedNodes.add(dn); } {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14119) GreedyPlanner Parameter Logging
BELUGA BEHR created HDFS-14119: -- Summary: GreedyPlanner Parameter Logging Key: HDFS-14119 URL: https://issues.apache.org/jira/browse/HDFS-14119 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs Affects Versions: 3.3.0 Reporter: BELUGA BEHR Attachments: HDFS-14119.1.patch 1. Do not use {{String.format()}} in conjunction with SLF4J. Superfluous. {code:java} String message = String .format("Compute Plan for Node : %s:%d took %d ms ", node.getDataNodeName(), node.getDataNodePort(), endTime - startTime); LOG.info(message);{code} 2. Do not call an explicit toString() on an object with SLF4J parameter. Otherwise, the string will be created and then thrown away if the logger is not set to debug level. Just pass the object itself and the framework will call {{toString}} if needed. {code} LOG.debug("Step : {} ", nextStep.toString()); {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14177) BlockChecksumHelper Readability
BELUGA BEHR created HDFS-14177: -- Summary: BlockChecksumHelper Readability Key: HDFS-14177 URL: https://issues.apache.org/jira/browse/HDFS-14177 Project: Hadoop HDFS Issue Type: Improvement Components: datanode Affects Versions: 3.2.0 Reporter: BELUGA BEHR * Remove magic number * Replace routing code with libraries to increase readability -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14256) Review Logging of NameNode Class
BELUGA BEHR created HDFS-14256: -- Summary: Review Logging of NameNode Class Key: HDFS-14256 URL: https://issues.apache.org/jira/browse/HDFS-14256 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Affects Versions: 3.2.0 Reporter: BELUGA BEHR -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14258) Introduce Java Concurrent Package To DataXceiverServer Class
BELUGA BEHR created HDFS-14258: -- Summary: Introduce Java Concurrent Package To DataXceiverServer Class Key: HDFS-14258 URL: https://issues.apache.org/jira/browse/HDFS-14258 Project: Hadoop HDFS Issue Type: Improvement Components: datanode Affects Versions: 3.2.0 Reporter: BELUGA BEHR * Use Java concurrent package to replace current facilities in {{DataXceiverServer}}. * A little bit of extra clean up -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14260) Synchronized Method in BlockReceiver Can Be Replace with Atomic Value
BELUGA BEHR created HDFS-14260: -- Summary: Synchronized Method in BlockReceiver Can Be Replace with Atomic Value Key: HDFS-14260 URL: https://issues.apache.org/jira/browse/HDFS-14260 Project: Hadoop HDFS Issue Type: Improvement Components: datanode Affects Versions: 3.2.0 Reporter: BELUGA BEHR -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14287) DataXceiverServer May Double-Close
BELUGA BEHR created HDFS-14287: -- Summary: DataXceiverServer May Double-Close Key: HDFS-14287 URL: https://issues.apache.org/jira/browse/HDFS-14287 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 3.2.0 Reporter: BELUGA BEHR Attachments: HDFS-14287.1.patch When a call to [HDFS-14258], when a user calls the {{kill()}} method, it closes the {{PeerServer}}. This causes the thread reading from the {{PeerServer}} to interrupt and exit. On the way out, this thread also closes the {{PeerServer}} for good measure. I'm not sure what the affects of closing the {{PeerServer}} are twice, but it should be avoided. To be clear, this is not a regressions caused by [HDFS-14258], if anything, [HDFS-14258] makes it easier to fix. :) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14292) Introduce Java ExecutorService to DataNode
BELUGA BEHR created HDFS-14292: -- Summary: Introduce Java ExecutorService to DataNode Key: HDFS-14292 URL: https://issues.apache.org/jira/browse/HDFS-14292 Project: Hadoop HDFS Issue Type: Improvement Components: datanode Affects Versions: 3.2.0 Reporter: BELUGA BEHR I wanted to investigate {{dfs.datanode.max.transfer.threads}} from {{hdfs-site.xml}}. It is described as "Specifies the maximum number of threads to use for transferring data in and out of the DN." The default value is 4096. I found it interesting because 4096 threads sounds like a lot to me. I'm not sure how a system with 8-16 cores would react to this large a thread count. Intuitively, I would say that the overhead of context switching would be immense. During mt investigation, I discovered the [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216] setup in the {{DataXceiverServer}} class: # A peer connects to a DataNode # A new thread is spun up to service this connection # The thread runs to completion # The tread dies It would perhaps be better if we used a thread pool to better manage the lifecycle of the service threads and to allow the DataNode to re-use existing threads, saving on the need to create and spin-up threads on demand. In this JIRA, I have added a couple of things: # Added a thread pool that will always maintain a single thread running, always awaiting a new connection should one arrive. On-demand, it will create up to {{dfs.datanode.max.transfer.threads}}. A thread that has completed its prior duties will stay idle for up to 30 seconds, it will be retired if no new work has arrived. # Added new methods to the {{Peer}} Interface to allow for better logging and less code within each Thread ({{DataXceiver}}). # Updated the Thread code ({{DataXceiver}}) regarding its interactions with {{blockReceiver}} instance variable -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14293) Increase Default Size of dfs.stream-buffer-size
BELUGA BEHR created HDFS-14293: -- Summary: Increase Default Size of dfs.stream-buffer-size Key: HDFS-14293 URL: https://issues.apache.org/jira/browse/HDFS-14293 Project: Hadoop HDFS Issue Type: Improvement Components: datanode Affects Versions: 3.2.0 Reporter: BELUGA BEHR For many years (7+) now, the JDK has been using a default buffer size of [8192 bytes|https://github.com/openjdk-mirror/jdk7u-jdk/blob/master/src/share/classes/java/io/BufferedInputStream.java#L53]. Hadoop still defaults to a size half of that. The default is {{dfs.stream-buffer-size}} is 4096 bytes. https://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml Please increase default size to 8192. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14294) Deprecate DFSUtilClient#getSmallBufferSize
BELUGA BEHR created HDFS-14294: -- Summary: Deprecate DFSUtilClient#getSmallBufferSize Key: HDFS-14294 URL: https://issues.apache.org/jira/browse/HDFS-14294 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs Affects Versions: 3.2.0 Reporter: BELUGA BEHR {code:java|title=DFSUtilClient.java} public static int getIoFileBufferSize(Configuration conf) { return conf.getInt( CommonConfigurationKeysPublic.IO_FILE_BUFFER_SIZE_KEY, CommonConfigurationKeysPublic.IO_FILE_BUFFER_SIZE_DEFAULT); } public static int getSmallBufferSize(Configuration conf) { return Math.min(getIoFileBufferSize(conf) / 2, 512); } {code} This concept of a "small buffer size" seems a bit overkill. First of all, it's not documented that such a thing exists and that by adjusting {{dfs.stream-buffer-size}} an administrator is also scaling these other buffer sizes. Seconds, I think any "small" buffer size should just use the default JDK buffer sizes. Anything that benefits from being larger than the default JDK size should be the controlled by {{IO_FILE_BUFFER_SIZE_KEY}} / {{dfs.stream-buffer-size}} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14295) Add Threadpool for DataTransfers
BELUGA BEHR created HDFS-14295: -- Summary: Add Threadpool for DataTransfers Key: HDFS-14295 URL: https://issues.apache.org/jira/browse/HDFS-14295 Project: Hadoop HDFS Issue Type: Improvement Components: datanode Affects Versions: 3.2.0 Reporter: BELUGA BEHR When a DataNode data transfers a block, is spins up a new thread for each transfer. [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339] and [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022]. Instead, add the threads to a {{CachedThreadPool}} so that when their threads complete the transfer, they can be re-used for another transfer. This should save resources spent on creating and spinning up transfer threads. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14296) Prefer ArrayList over LinkedList in VolumeScanner
BELUGA BEHR created HDFS-14296: -- Summary: Prefer ArrayList over LinkedList in VolumeScanner Key: HDFS-14296 URL: https://issues.apache.org/jira/browse/HDFS-14296 Project: Hadoop HDFS Issue Type: Improvement Components: datanode Affects Versions: 3.2.0 Reporter: BELUGA BEHR Assignee: BELUGA BEHR Attachments: HDFS-14296.1.patch {quote}The {{size}}, {{isEmpty}}, {{get}}, {{set}}, {{iterator}}, and {{listIterator}} operations run in constant time. - ArrayList {quote} However, for a {{LinkedList}}, the entire list must be traversed to get to the desired index. Like [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/VolumeScanner.java#L384] Most of the time, the List is being iterated, which is quicker over a primitive array than walking a LinkedList. There is one place where an item is removed, potentially from the middle of the list [here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/VolumeScanner.java#L736] but the speed of removing from the middle of the list isn't bad; it's a system native array shift, and it only happens on the off chance that a block pool is removed from the DataNode. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13155) Not Checking Return Value for NULL
BELUGA BEHR created HDFS-13155: -- Summary: Not Checking Return Value for NULL Key: HDFS-13155 URL: https://issues.apache.org/jira/browse/HDFS-13155 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Affects Versions: 3.0.0 Reporter: BELUGA BEHR {code:title=BlockPlacementPolicyDefault.java} protected Node chooseTargetInOrder(int numOfReplicas, Node writer, final Set excludedNodes, final long blocksize, final int maxNodesPerRack, final List results, final boolean avoidStaleNodes, final boolean newBlock, EnumMap storageTypes) throws NotEnoughReplicasException { final int numOfResults = results.size(); if (numOfResults == 0) { writer = chooseLocalStorage(writer, excludedNodes, blocksize, maxNodesPerRack, results, avoidStaleNodes, storageTypes, true) .getDatanodeDescriptor(); if (--numOfReplicas == 0) { return writer; } } ... {code} The method {{chooseLocalStorage}} can return a _null_ value but it's not being checked here and the method {{getDatanodeDescriptor()}} is immediately being called on the result. Please check for a _null_ value first. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13156) HDFS Block Placement Policy - Client Local Rack
BELUGA BEHR created HDFS-13156: -- Summary: HDFS Block Placement Policy - Client Local Rack Key: HDFS-13156 URL: https://issues.apache.org/jira/browse/HDFS-13156 Project: Hadoop HDFS Issue Type: Improvement Components: documentation Affects Versions: 2.9.0 Reporter: BELUGA BEHR {quote}For the common case, when the replication factor is three, HDFS’s placement policy is to put one replica on the local machine if the writer is on a datanode, otherwise on a random datanode, another replica on a node in a different (remote) rack, and the last on a different node in the same remote rack. {quote} [https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html#Replica_Placement:_The_First_Baby_Steps] Having just looked over the Default Block Placement code, the way I understand this, is that, there are three basic scenarios: # HDFS client is running on a datanode inside the cluster # HDFS client is running on a node outside the cluster # HDFS client is running on a non-datanode inside the cluster The documentation is ambiguous concerning the third scenario. Please correct me if I'm wrong, but the way I understand the code, if there is an HDFS client inside the cluster, but it is not on a datanode, the first block will be placed on a datanode within the set of datanodes available on the local rack and not simply on any _random datanode_ from the set of all datanodes in the cluster. That is to say, if one rack has an HDFS Sink Flume Agent on a dedicated node, I should expect that every first block will be written to a _random datanode_ on the same rack as the HDFS Flume agent, assuming the network topology script is written to include this Flume node. If that is correct, can the documentation be updated to include this third common scenario? -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13157) Do Not Remove Blocks Sequentially During Dec omission
BELUGA BEHR created HDFS-13157: -- Summary: Do Not Remove Blocks Sequentially During Dec omission Key: HDFS-13157 URL: https://issues.apache.org/jira/browse/HDFS-13157 Project: Hadoop HDFS Issue Type: Improvement Components: datanode, namenode Affects Versions: 3.0.0 Reporter: BELUGA BEHR >From what I understand of [DataNode >decommissioning|https://github.com/apache/hadoop/blob/42a1c98597e6dba2e371510a6b2b6b1fb94e4090/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeAdminManager.java] > it appears that all the blocks are scheduled for removal _in order._. I'm >not 100% sure what the ordering is exactly, but I think it's loops through >each data volume and schedules each block to be replicated elsewhere. The net >affect is that during a decommission, all of the DataNode transfer threads >slam on a single volume until it is cleaned out. At which point, they all >slam on the next volume, etc. Please randomize the block list so that there is a more even distribution across all volumes when decommissioning a node. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13158) Fix Spelling MIstakes - DECOMISSIONED
BELUGA BEHR created HDFS-13158: -- Summary: Fix Spelling MIstakes - DECOMISSIONED Key: HDFS-13158 URL: https://issues.apache.org/jira/browse/HDFS-13158 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 3.0.0 Reporter: BELUGA BEHR https://github.com/apache/hadoop/search?l=Java&q=DECOMISSIONED There are references in the code to _DECOMISSIONED_ but the correct spelling is _DECOMMISSIONED_. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13167) DatanodeAdminManager Improvements
BELUGA BEHR created HDFS-13167: -- Summary: DatanodeAdminManager Improvements Key: HDFS-13167 URL: https://issues.apache.org/jira/browse/HDFS-13167 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs Affects Versions: 3.0.0 Reporter: BELUGA BEHR # Use Collection type Set instead of List for tracking nodes # Fix logging statements that are erroneously appending variables instead of using parameters # Miscellaneous small improvements As an example, the {{node}} variable is being appended to the string instead of being passed as an argument to the {{trace}} method for variable substitution. {code} LOG.trace("stopDecommission: Node {} in {}, nothing to do." + node, node.getAdminState()); {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13168) XmlImageVisitor - Prefer Array over LinkedList
BELUGA BEHR created HDFS-13168: -- Summary: XmlImageVisitor - Prefer Array over LinkedList Key: HDFS-13168 URL: https://issues.apache.org/jira/browse/HDFS-13168 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs Affects Versions: 3.0.0 Environment: {{ArrayDeque}} {quote}This class is likely to be faster than Stack when used as a stack, and faster than LinkedList when used as a queue.{quote} .. not to mention less memory fragmentation (single backing array v.s. many ArrayList nodes). https://docs.oracle.com/javase/8/docs/api/java/util/ArrayDeque.html Reporter: BELUGA BEHR Attachments: HDFS-13168.1.patch -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13193) Various Improvements for BlockTokenSecretManager.java
BELUGA BEHR created HDFS-13193: -- Summary: Various Improvements for BlockTokenSecretManager.java Key: HDFS-13193 URL: https://issues.apache.org/jira/browse/HDFS-13193 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs Affects Versions: 3.0.0 Reporter: BELUGA BEHR Attachments: HDFS-13193.1.patch Various improvements for class {{org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.java}} # Remove superfluous {{toString}} calls # Fix some checkstyle warnings # Re-implement method with O(N^2) with HashMultiSet - also improves readability # Increase code re-use with Apache Commons Library -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13428) Remove LinkedList From StateStoreFileImpl.java
BELUGA BEHR created HDFS-13428: -- Summary: Remove LinkedList From StateStoreFileImpl.java Key: HDFS-13428 URL: https://issues.apache.org/jira/browse/HDFS-13428 Project: Hadoop HDFS Issue Type: Improvement Components: federation Affects Versions: 3.0.1 Environment: Replace {{LinkedList}} with {{ArrayList}} implementation in the StateStoreFileImpl class. This is especially advantageous because we can pre-allocate the internal array before a copy occurs. {{ArrayList}} is faster for iterations and requires less memory than {{LinkedList}}. {code:java} protected List getChildren(String path) { List ret = new LinkedList<>(); File dir = new File(path); File[] files = dir.listFiles(); if (files != null) { for (File file : files) { String filename = file.getName(); ret.add(filename); } } return ret; }{code} Reporter: BELUGA BEHR Attachments: HDFS-13428.1.patch -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13447) Fix Typos - Node Not Chosen
BELUGA BEHR created HDFS-13447: -- Summary: Fix Typos - Node Not Chosen Key: HDFS-13447 URL: https://issues.apache.org/jira/browse/HDFS-13447 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Affects Versions: 3.0.1, 2.2.0 Reporter: BELUGA BEHR Attachments: HDFS-13447.1.patch Fix typo and improve: {code:java} private enum NodeNotChosenReason { NOT_IN_SERVICE("the node isn't in service"), NODE_STALE("the node is stale"), NODE_TOO_BUSY("the node is too busy"), TOO_MANY_NODES_ON_RACK("the rack has too many chosen nodes"), NOT_ENOUGH_STORAGE_SPACE("no enough storage space to place the block");{code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13448) HDFS Block Placement - Ignore Locality
BELUGA BEHR created HDFS-13448: -- Summary: HDFS Block Placement - Ignore Locality Key: HDFS-13448 URL: https://issues.apache.org/jira/browse/HDFS-13448 Project: Hadoop HDFS Issue Type: New Feature Components: block placement, hdfs-client Affects Versions: 3.0.1, 2.9.0 Reporter: BELUGA BEHR According to the HDFS Block Place Rules: {quote} /** * The replica placement strategy is that if the writer is on a datanode, * the 1st replica is placed on the local machine, * otherwise a random datanode. The 2nd replica is placed on a datanode * that is on a different rack. The 3rd replica is placed on a datanode * which is on a different node of the rack as the second replica. */ {quote} However, there is a hint for the hdfs-client that allows the block placement request to not put a block replica on the local datanode _where 'local' means the same host as the client is being run on._ {quote} /** * Advise that a block replica NOT be written to the local DataNode where * 'local' means the same host as the client is being run on. * * @see CreateFlag#NO_LOCAL_WRITE */ {quote} I propose that we add a new flag that allows the hdfs-client to request that the first block replica be placed on a random DataNode in the cluster. The subsequent block replicas should follow the normal block placement rules. The issue is that when the {{NO_LOCAL_WRITE}} is enabled, the first block replica is not placed on the local node, but it is still placed on the local rack. Where this comes into play is where you have, for example, a flume agent that is loading data into HDFS. If the Flume agent is running on a DataNode, then by default, the DataNode local to the Flume agent will always get the first block replica and this leads to un-even block placements, with the local node always filling up faster than any other node in the cluster. Modifying this example, if the DataNode is removed from the host where the Flume agent is running, or this {{NO_LOCAL_WRITE}} is enabled by Flume, then the default block placement policy will still prefer the local rack. This remedies the situation only so far as now the first block replica will always be distributed to a DataNode on the local rack. This new flag would allow a single Flume agent to distribute the blocks randomly, evenly, over the entire cluster instead of hot-spotting the local node or the local rack. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13684) Remove Use of Deprecated Whitebox Test Class
BELUGA BEHR created HDFS-13684: -- Summary: Remove Use of Deprecated Whitebox Test Class Key: HDFS-13684 URL: https://issues.apache.org/jira/browse/HDFS-13684 Project: Hadoop HDFS Issue Type: Improvement Components: test Affects Versions: 3.1.0, 2.10.0 Reporter: BELUGA BEHR Unit tests within the Hadoop suite are using the now deprecated class {{org.apache.hadoop.test.Whitebox}}. bq. This class was ported from org.mockito.internal.util.reflection.Whitebox since the class was removed in Mockito 2.1. Using this class is a bad practice. Consider refactoring instead of using this. As stated, refactor the existing tests to remove calls to this class... then remove this class from the project. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13685) Review and Refactor TestDFSOutputStream.java
BELUGA BEHR created HDFS-13685: -- Summary: Review and Refactor TestDFSOutputStream.java Key: HDFS-13685 URL: https://issues.apache.org/jira/browse/HDFS-13685 Project: Hadoop HDFS Issue Type: Sub-task Components: hdfs-client Affects Versions: 3.1.0, 2.10.0 Reporter: BELUGA BEHR Remove use of deprecated class {{org.apache.hadoop.test.Whitebox}} Refactor the necessary code to make mocking easier -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13812) Update Docs on Caching
BELUGA BEHR created HDFS-13812: -- Summary: Update Docs on Caching Key: HDFS-13812 URL: https://issues.apache.org/jira/browse/HDFS-13812 Project: Hadoop HDFS Issue Type: Improvement Components: documentation Affects Versions: 2.9.1 Reporter: BELUGA BEHR {quote} dfs.namenode.path.based.cache.refresh.interval.ms The NameNode will use this as the amount of milliseconds between subsequent path cache rescans. This calculates the blocks to cache and each DataNode containing a replica of the block that should cache it. By default, this parameter is set to 30, which is five minutes. {quote} [https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/CentralizedCacheManagement.html] However, this default value was change in [HDFS-6106] to 30 seconds. Please update docs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13820) Disable CacheReplicationMonitor If No Cached Paths Exist
BELUGA BEHR created HDFS-13820: -- Summary: Disable CacheReplicationMonitor If No Cached Paths Exist Key: HDFS-13820 URL: https://issues.apache.org/jira/browse/HDFS-13820 Project: Hadoop HDFS Issue Type: Improvement Components: caching Affects Versions: 2.10.0, 3.2.0 Reporter: BELUGA BEHR Stating with [HDFS-6106] the loop for checking caching is set to be every 30 seconds. Please implement a way to disable the {{CacheReplicationMonitor}} class if there are no paths specified. Adding the first cached path to the NameNode should kick off the {{CacheReplicationMonitor}} and when the last one is deleted, the {{CacheReplicationMonitor}} should be disabled again. Alternatively, provide a configuration flag to turn this feature off altogether. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-9457) Datanode Logging Improvement
BELUGA BEHR created HDFS-9457: - Summary: Datanode Logging Improvement Key: HDFS-9457 URL: https://issues.apache.org/jira/browse/HDFS-9457 Project: Hadoop HDFS Issue Type: Improvement Components: datanode Affects Versions: 2.7.0, 3.0.0 Reporter: BELUGA BEHR Priority: Trivial Please accept my patch for some minor clean-up of logging. Patch is against 3.0.0 but applies to previous versions as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-9492) RoundRobinVolumeChoosingPolicy
BELUGA BEHR created HDFS-9492: - Summary: RoundRobinVolumeChoosingPolicy Key: HDFS-9492 URL: https://issues.apache.org/jira/browse/HDFS-9492 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Affects Versions: 2.7.1 Reporter: BELUGA BEHR Priority: Trivial This is some general clean-up for: RoundRobinVolumeChoosingPolicy I have also updated and expanded the unit tests a bit. There is one error message being generated that I changed. I felt the previous Exception message was not that helpful and therefore it was possible to trim it down. If the exception message must be enhanced, the entire list of "volumes" should be included. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-9507) LeaseRenewer Logging Under-Reporting
BELUGA BEHR created HDFS-9507: - Summary: LeaseRenewer Logging Under-Reporting Key: HDFS-9507 URL: https://issues.apache.org/jira/browse/HDFS-9507 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Affects Versions: 2.7.1 Reporter: BELUGA BEHR Priority: Minor Why is it that in LeaseRenewer#run() failures to renew a lease on a file are reported with "warn" level logging, but in LeaseRenewer#renew() it is reported with a "debug" level warn? In LeaseRenewer#renew(), if the method renewLease() returns 'false' then the problem is silently discarded (continue, no Exception is thrown) and the next client in the list tries to renew. {code:title=LeaseRenewer.java|borderStyle=solid} private void run(final int id) throws InterruptedException { ... try { renew(); lastRenewed = Time.monotonicNow(); } catch (SocketTimeoutException ie) { LOG.warn("Failed to renew lease for " + clientsString() + " for " + (elapsed/1000) + " seconds. Aborting ...", ie); synchronized (this) { while (!dfsclients.isEmpty()) { DFSClient dfsClient = dfsclients.get(0); dfsClient.closeAllFilesBeingWritten(true); closeClient(dfsClient); } //Expire the current LeaseRenewer thread. emptyTime = 0; } break; } catch (IOException ie) { LOG.warn("Failed to renew lease for " + clientsString() + " for " + (elapsed/1000) + " seconds. Will retry shortly ...", ie); } } ... } private void renew() throws IOException { { ... if (!c.renewLease()) { LOG.debug("Did not renew lease for client {}", c); continue; } ... } {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)