[jira] [Created] (HDFS-9518) LeaseRenewer - Improve Renew Method
BELUGA BEHR created HDFS-9518: - Summary: LeaseRenewer - Improve Renew Method Key: HDFS-9518 URL: https://issues.apache.org/jira/browse/HDFS-9518 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs-client Affects Versions: 2.7.1 Reporter: BELUGA BEHR Priority: Minor Replaced the current implementation of org.apache.hadoop.hdfs.client.impl.LeaseRenewer#renew() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-9551) Random VolumeChoosingPolicy
BELUGA BEHR created HDFS-9551: - Summary: Random VolumeChoosingPolicy Key: HDFS-9551 URL: https://issues.apache.org/jira/browse/HDFS-9551 Project: Hadoop HDFS Issue Type: Improvement Components: datanode Affects Versions: 2.7.1 Reporter: BELUGA BEHR Priority: Minor Please find attached a new implementation of VolumeChoosingPolicy. This implementation chooses volumes at random to place blocks. It is thread-safe and un-synchronized so there is less thread contention. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-9560) Fair AvailableSpaceVolumeChoosingPolicy
BELUGA BEHR created HDFS-9560: - Summary: Fair AvailableSpaceVolumeChoosingPolicy Key: HDFS-9560 URL: https://issues.apache.org/jira/browse/HDFS-9560 Project: Hadoop HDFS Issue Type: Improvement Reporter: BELUGA BEHR Priority: Minor I took a look at AvailableSpaceVolumeChoosingPolicy. It seems a bit overkill and includes some configuration items that seem a bit arbitrary with no real clear guidance on how to effectively use them: _dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction_ _dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold_ I have created an alternative implementation that does not require any external configuration, is thread-safe, and requires no synchronization. "Weighted Randomized Ordering" http://stackoverflow.com/questions/23971365/weighted-randomized-ordering Conceptually, a dart-board is constructed of several wedges, each wedge represents a disk volume. The more available space that a volume has relative to the other volumes, the larger its wedge. Then, a dart is thrown at the board and whichever wedge(volume) the dart lands on, that wedge is assigned the incoming block. Over time, the wedges balance and all have an equal chance of being "hit." -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-9593) BlockTokenIdentifier Code Review
BELUGA BEHR created HDFS-9593: - Summary: BlockTokenIdentifier Code Review Key: HDFS-9593 URL: https://issues.apache.org/jira/browse/HDFS-9593 Project: Hadoop HDFS Issue Type: Improvement Reporter: BELUGA BEHR Priority: Trivial Please find attached a code review of BlockTokenIdentifier. The only real change is using an explicit StringBuilder instead of an implicit one so that the internal buffer may be set more appropriately. The relatively small default buffer of 16 bytes would need to be expanded several times to fit the incoming string. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-9596) Remove "Shuffle" Method From DFSUtil
BELUGA BEHR created HDFS-9596: - Summary: Remove "Shuffle" Method From DFSUtil Key: HDFS-9596 URL: https://issues.apache.org/jira/browse/HDFS-9596 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 2.7.1 Reporter: BELUGA BEHR Priority: Trivial Attachments: Shuffle.HDFS-9596.patch DFSUtil contains a "shuffle" routine that shuffles arrays. With a little wrapping, Java Collections class can shuffle. This code is superfluous and undocumented. The method returns an array, though it does not specify if it returns the same array or a new, copy, array of shuffled items. It is only used in one place, so it is better to simply remove and substitute this implementation with the java Collections implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-11931) FileSystem Simplify / Optimize listStatus Method
BELUGA BEHR created HDFS-11931:
--
Summary: FileSystem Simplify / Optimize listStatus Method
Key: HDFS-11931
URL: https://issues.apache.org/jira/browse/HDFS-11931
Project: Hadoop HDFS
Issue Type: Improvement
Components: fs
Affects Versions: 3.0.0-alpha3, 2.7.3
Reporter: BELUGA BEHR
Priority: Minor
{code:title=org.apache.hadoop.fs.FileSystem.listStatus(ArrayList,
Path, PathFilter)}
/*
* Filter files/directories in the given path using the user-supplied path
* filter. Results are added to the given array results.
*/
private void listStatus(ArrayList results, Path f,
PathFilter filter) throws FileNotFoundException, IOException {
FileStatus listing[] = listStatus(f);
if (listing == null) {
throw new IOException("Error accessing " + f);
}
for (int i = 0; i < listing.length; i++) {
if (filter.accept(listing[i].getPath())) {
results.add(listing[i]);
}
}
}
{code}
{code:title=org.apache.hadoop.fs.FileSystem.listStatus(Path, PathFilter)}
public FileStatus[] listStatus(Path f, PathFilter filter)
throws FileNotFoundException, IOException {
ArrayList results = new ArrayList();
listStatus(results, f, filter);
return results.toArray(new FileStatus[results.size()]);
}
{code}
We can be smarter about this:
# Use enhanced for-loops
# Optimize for the case where there are zero files in a directory, save on
object instantiation
# More encapsulated design
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13947) Review of DirectoryScanner Class
BELUGA BEHR created HDFS-13947:
--
Summary: Review of DirectoryScanner Class
Key: HDFS-13947
URL: https://issues.apache.org/jira/browse/HDFS-13947
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 3.2.0
Reporter: BELUGA BEHR
Review of Directory Scanner. Replaced a lot of code with Guava MultiMap.
Some general house cleaning and improved logging. For performance, using
{{ArrayList}} instead of {{LinkedList}} where possible, especially since these
lists can be quite large a LinkedList will consume a lot of memory and be slow
to sort/iterate over.
https://stackoverflow.com/questions/322715/when-to-use-linkedlist-over-arraylist-in-java
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13958) Miscellaneous Improvements for FsVolumeSpi
BELUGA BEHR created HDFS-13958:
--
Summary: Miscellaneous Improvements for FsVolumeSpi
Key: HDFS-13958
URL: https://issues.apache.org/jira/browse/HDFS-13958
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 3.3.0
Reporter: BELUGA BEHR
The work in [HDFS-13947] allowed for using {{ArrayList}} instead of
{{LinkedList}} when scanning DataNode local directories, however the
{{FsVolumeSpi}} implementations were still using (and forcing) {{LinkedList}}.
I propose changing the {{FsVolumeSpi}} signatures to allow for {{Collection}}
instead of {{LinkedList}}. Since I'm looking at the code, I made some small
improvements and check-style fixes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13967) HDFS Router Quota Class Review
BELUGA BEHR created HDFS-13967: -- Summary: HDFS Router Quota Class Review Key: HDFS-13967 URL: https://issues.apache.org/jira/browse/HDFS-13967 Project: Hadoop HDFS Issue Type: Improvement Components: federation, hdfs Affects Versions: 3.2.0 Reporter: BELUGA BEHR Attachments: HDFS-13967.1.patch # Use Guava library # Prefer ArrayList over LinkedList # Simplify code logic -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13968) BlockReceiver Array-Based Queue
BELUGA BEHR created HDFS-13968: -- Summary: BlockReceiver Array-Based Queue Key: HDFS-13968 URL: https://issues.apache.org/jira/browse/HDFS-13968 Project: Hadoop HDFS Issue Type: Improvement Components: datanode Affects Versions: 3.2.0 Environment: # Prefer Array implementation of Queue over LinkedList # Check-style clean up http://brianandstuff.com/2016/12/12/java-arraydeque-vs-linkedlist/ Reporter: BELUGA BEHR Attachments: HDFS-13968.1.patch -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13969) Server.java Prefer Array List
BELUGA BEHR created HDFS-13969:
--
Summary: Server.java Prefer Array List
Key: HDFS-13969
URL: https://issues.apache.org/jira/browse/HDFS-13969
Project: Hadoop HDFS
Issue Type: Improvement
Components: ipc
Affects Versions: 3.2.0
Reporter: BELUGA BEHR
Attachments: HDFS-13969.1.patch
* Prefer ArrayDeque over LinkedList (faster, less memory overhead)
* Address this code:
{code}
//
// Remove calls that have been pending in the responseQueue
// for a long time.
//
private void doPurge(RpcCall call, long now) {
LinkedList responseQueue = call.connection.responseQueue;
synchronized (responseQueue) {
Iterator iter = responseQueue.listIterator(0);
while (iter.hasNext()) {
call = iter.next();
if (now > call.timestamp + PURGE_INTERVAL) {
closeConnection(call.connection);
break;
}
}
}
}
{code}
It says "Remove calls" (plural) but only one call will be removed because of
the 'break' statement.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13970) CacheManager Directives Map
BELUGA BEHR created HDFS-13970:
--
Summary: CacheManager Directives Map
Key: HDFS-13970
URL: https://issues.apache.org/jira/browse/HDFS-13970
Project: Hadoop HDFS
Issue Type: Improvement
Components: caching, hdfs
Affects Versions: 3.2.0
Reporter: BELUGA BEHR
Attachments: HDFS-13970.1.patch
# Use Guava Multimap to simplify code
## Currently, code uses a mix of LinkedList and ArrayList - just pick one
## Currently, {{directivesByPath}} structure is sorted but never used in a
sorted way, it only performs remove and add operations, no iteration - use a
{{Set}} instead of a {{List}}
# The {{CacheDirective}} class needs a better hashcode implementation since it
will be used in a Set. Do not instantiate a {{HashBuilder}} object every time
{{hashcode}} is called. Ouch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13971) Review of NetgroupCache
BELUGA BEHR created HDFS-13971: -- Summary: Review of NetgroupCache Key: HDFS-13971 URL: https://issues.apache.org/jira/browse/HDFS-13971 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs, security Affects Versions: 3.2.0 Reporter: BELUGA BEHR * Simplify code and performance by using Guava Multimap -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13978) Review of DiskBalancer
BELUGA BEHR created HDFS-13978:
--
Summary: Review of DiskBalancer
Key: HDFS-13978
URL: https://issues.apache.org/jira/browse/HDFS-13978
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode, hdfs
Affects Versions: 3.2.0
Environment: * Use ArrayList instead of LinkedList. Especially because
this code here uses the {{List#get()}} method, which is very slow for the
LinkedList since it has to traverse every node to get to the offset: O(N^2)
{code}
ExtendedBlock getNextBlock(List poolIters,
DiskBalancerWorkItem item) {
Preconditions.checkNotNull(poolIters);
int currentCount = 0;
ExtendedBlock block = null;
while (block == null && currentCount < poolIters.size()) {
currentCount++;
int index = poolIndex++ % poolIters.size();
FsVolumeSpi.BlockIterator currentPoolIter = poolIters.get(index);
block = getBlockToCopy(currentPoolIter, item);
}
{code}
* Do not "log and throw" errors. This is an anti-pattern and should be
avoided. Log or throw, but don't do both. Removed some logging
* Improved other logging statements
* Improved the {{hashcode}} method of one of the inner classes. It was
instantiating a List and performing iteration for every call. Replace with
Eclipse-generated hashcode method.
* Fixed compiler warnings for deprecated code or code that was not parameterized
* Fix check style issue
Reporter: BELUGA BEHR
Attachments: HDFS-13978.1.patch
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13979) Review StateStoreFileSystemImpl Class
BELUGA BEHR created HDFS-13979:
--
Summary: Review StateStoreFileSystemImpl Class
Key: HDFS-13979
URL: https://issues.apache.org/jira/browse/HDFS-13979
Project: Hadoop HDFS
Issue Type: Improvement
Components: federation
Affects Versions: 3.2.0
Reporter: BELUGA BEHR
* Replace {{LinkedList}} with pre-sized {{ArrayList}}
* Added exception information to WARN/ERROR logging
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13980) Review of DBNameNodeConnector Class
BELUGA BEHR created HDFS-13980:
--
Summary: Review of DBNameNodeConnector Class
Key: HDFS-13980
URL: https://issues.apache.org/jira/browse/HDFS-13980
Project: Hadoop HDFS
Issue Type: Improvement
Components: diskbalancer
Affects Versions: 3.2.0
Environment: * Removed anti-pattern "log and throw". Log or throw,
don't do both
* Use pre-sized {{ArrayList}} instead of {{LinkedList}}
* Fixed some formatting issues
* Re-ordered imports into correct order
Reporter: BELUGA BEHR
Attachments: HDFS-13980.1.patch
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13981) Review of AvailableSpaceResolver.java
BELUGA BEHR created HDFS-13981: -- Summary: Review of AvailableSpaceResolver.java Key: HDFS-13981 URL: https://issues.apache.org/jira/browse/HDFS-13981 Project: Hadoop HDFS Issue Type: Improvement Components: federation Affects Versions: 3.2.0 Reporter: BELUGA BEHR Attachments: HDFS-13981.1.patch * No behavior changes, just optimizing and paring down the code -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13987) Review of RandomResolver Class
BELUGA BEHR created HDFS-13987:
--
Summary: Review of RandomResolver Class
Key: HDFS-13987
URL: https://issues.apache.org/jira/browse/HDFS-13987
Project: Hadoop HDFS
Issue Type: Improvement
Components: federation
Affects Versions: 3.2.0
Reporter: BELUGA BEHR
Attachments: HDFS-13987.1.patch
* Use {{ThreadLocalRandom}}
* Do not create a list, and copy, to pick a random index
* An early-return in the method means that the 'ERROR' logging is skipped in
some cases
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13988) Allow for Creating Temporary Files
BELUGA BEHR created HDFS-13988:
--
Summary: Allow for Creating Temporary Files
Key: HDFS-13988
URL: https://issues.apache.org/jira/browse/HDFS-13988
Project: Hadoop HDFS
Issue Type: New Feature
Components: hdfs, hdfs-client
Affects Versions: 3.2.0
Reporter: BELUGA BEHR
Allow clients to create temporary files in HDFS.
{code:java|title=FileSystem.java}
public FSDataOutputStreamBuilder createFile(Path path);
public FSDataOutputStreamBuilder appendFile(Path path);
// New feature
public FSDataOutputStreamBuilder createTempFile(Path path, TimeUnit unit, long
ttl);
{code}
This would register the file with the NameNode to have a TTL. The NameNode
would be responsible for deleting the files after some custom TTL. This would
be helpful in the case that a MapReduce (or Spark) job fails during execution
and does not properly clean up after itself.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-13988) Allow for Creating Temporary Files
[
https://issues.apache.org/jira/browse/HDFS-13988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR resolved HDFS-13988.
Resolution: Duplicate
> Allow for Creating Temporary Files
> --
>
> Key: HDFS-13988
> URL: https://issues.apache.org/jira/browse/HDFS-13988
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: hdfs, hdfs-client
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Priority: Major
>
> Allow clients to create temporary files in HDFS.
> {code:java|title=FileSystem.java}
> public FSDataOutputStreamBuilder createFile(Path path);
> public FSDataOutputStreamBuilder appendFile(Path path);
> // New feature
> public FSDataOutputStreamBuilder createTempFile(Path path, TimeUnit unit,
> long ttl);
> {code}
> This would register the file with the NameNode to have a TTL. The NameNode
> would be responsible for deleting the files after some custom TTL. This
> would be helpful in the case that a MapReduce (or Spark) job fails during
> execution and does not properly clean up after itself.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13990) Synchronization Issue With HashResolver.java
BELUGA BEHR created HDFS-13990:
--
Summary: Synchronization Issue With HashResolver.java
Key: HDFS-13990
URL: https://issues.apache.org/jira/browse/HDFS-13990
Project: Hadoop HDFS
Issue Type: Bug
Components: federation
Affects Versions: 3.2.0
Reporter: BELUGA BEHR
Attachments: HDFS-13990.1.patch
{code:java|title=HashResolver.java}
private ConsistentHashRing getHashResolver(final Set namespaces) {
int hash = namespaces.hashCode();
ConsistentHashRing resolver = this.hashResolverMap.get(hash);
if (resolver == null) {
resolver = new ConsistentHashRing(namespaces);
this.hashResolverMap.put(hash, resolver);
}
return resolver;
}
{code}
The {{hashResolverMap}} is a {{ConcurrentHashMap}} so presumably there is
concern here for concurrency issues. However, there is no synchronization
around this method, so two threads could call {{get(hash)}} both see a 'null'
value and then both add two entries into the {{Map}}. Add synchronization here.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13991) Review of DiskBalancerCluster.java
BELUGA BEHR created HDFS-13991:
--
Summary: Review of DiskBalancerCluster.java
Key: HDFS-13991
URL: https://issues.apache.org/jira/browse/HDFS-13991
Project: Hadoop HDFS
Issue Type: Improvement
Components: diskbalancer
Affects Versions: 3.2.0
Reporter: BELUGA BEHR
Attachments: HDFS-13991.1.patch
* Use {{ArrayList}} instead of {{LinkedList}}
* Simplify the code
* Use {{File}} object's file system agnostic features
* Re-order import statements
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13994) DataNode BlockSender waitForMinLength
BELUGA BEHR created HDFS-13994:
--
Summary: DataNode BlockSender waitForMinLength
Key: HDFS-13994
URL: https://issues.apache.org/jira/browse/HDFS-13994
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 3.2.0
Reporter: BELUGA BEHR
Attachments: HDFS-13994.1.patch
{code:java|title=BlockSender.java}
private static void waitForMinLength(ReplicaInPipeline rbw, long len)
throws IOException {
// Wait for 3 seconds for rbw replica to reach the minimum length
for (int i = 0; i < 30 && rbw.getBytesOnDisk() < len; i++) {
try {
Thread.sleep(100);
} catch (InterruptedException ie) {
throw new IOException(ie);
}
}
long bytesOnDisk = rbw.getBytesOnDisk();
if (bytesOnDisk < len) {
throw new IOException(
String.format("Need %d bytes, but only %d bytes available", len,
bytesOnDisk));
}
}
{code}
It is not very efficient to poll for status in this way. Instead, use
{{notifyAll}} within the {{ReplicaInPipeline}} to notify the caller when the
replica has reached a certain size.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14102) verifyBlockPlacement
BELUGA BEHR created HDFS-14102:
--
Summary: verifyBlockPlacement
Key: HDFS-14102
URL: https://issues.apache.org/jira/browse/HDFS-14102
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: BELUGA BEHR
{code:java|title=BlockPlacementPolicyDefault.java}
// 1. Check that all locations are different.
// 2. Count locations on different racks.
Set racks = new TreeSet<>();
for (DatanodeInfo dn : locs)
racks.add(dn.getNetworkLocation());
...
racks.size(){code}
Here, the code is counting the number of distinct Network Locations. However,
it is using a TreeSet which has overhead to maintain item order and uses a
linked structure internally. This overhead is unneeded since all that is
required here is a count.
{quote}A NavigableSet implementation based on a TreeMap. The elements are
ordered using their natural ordering, or by a Comparator provided at set
creation time, depending on which constructor is used.
This implementation provides guaranteed log(n) time cost for the basic
operations (add, remove and contains).
[https://docs.oracle.com/javase/7/docs/api/java/util/TreeSet.html]
{quote}
Use Java streams for readability and because it uses a {{HashSet}} under the
covers to perform the distinct action. {{HashSet}} uses an array internally and
has constant time performance for the {{add}} method.
[https://github.com/apache/hadoop/blob/27978bcb66a9130cbf26d37ec454c0b7fcdc2530/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java#L1042]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14103) Review Logging of BlockPlacementPolicyDefault
BELUGA BEHR created HDFS-14103:
--
Summary: Review Logging of BlockPlacementPolicyDefault
Key: HDFS-14103
URL: https://issues.apache.org/jira/browse/HDFS-14103
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: BELUGA BEHR
Attachments: HDFS-14103.1.patch
Review use of SLF4J in {{BlockPlacementPolicyDefault.java}}
Other minor logging improvements.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14104) Review getImageTxIdToRetain
BELUGA BEHR created HDFS-14104:
--
Summary: Review getImageTxIdToRetain
Key: HDFS-14104
URL: https://issues.apache.org/jira/browse/HDFS-14104
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Affects Versions: 3.0.0
Reporter: BELUGA BEHR
{code:java|title=NNStorageRetentionManager.java}
private long getImageTxIdToRetain(FSImageTransactionalStorageInspector
inspector) {
List images = inspector.getFoundImages();
TreeSet imageTxIds = Sets.newTreeSet();
for (FSImageFile image : images) {
imageTxIds.add(image.getCheckpointTxId());
}
List imageTxIdsList = Lists.newArrayList(imageTxIds);
if (imageTxIdsList.isEmpty()) {
return 0;
}
Collections.reverse(imageTxIdsList);
int toRetain = Math.min(numCheckpointsToRetain, imageTxIdsList.size());
long minTxId = imageTxIdsList.get(toRetain - 1);
LOG.info("Going to retain " + toRetain + " images with txid >= " +
minTxId);
return minTxId;
}
{code}
# Fix check style issues
# Use SLF4J paramaterized logging
# A lot of work gets done before checking if the list actually contains any
entries and returning a 0. That should be the first thing that happens
# Instead of building up the {{TreeSet}} in its natural order, then reversing
the collection, simply use a reverse natural ordering to begin with and save a
step.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14105) NamenodeFsck HashSet
BELUGA BEHR created HDFS-14105:
--
Summary: NamenodeFsck HashSet
Key: HDFS-14105
URL: https://issues.apache.org/jira/browse/HDFS-14105
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: BELUGA BEHR
Attachments: HDFS-14105.1.patch
Use {{HashSet}} instead of {{TreeSet}}. {{TreeSet}} has the overhead of
keeping elements in order even though ordering is not taken into consideration
in this class.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14106) Improve NamenodeFsck copyBlock
BELUGA BEHR created HDFS-14106: -- Summary: Improve NamenodeFsck copyBlock Key: HDFS-14106 URL: https://issues.apache.org/jira/browse/HDFS-14106 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Affects Versions: 3.2.0 Environment: # Code is performing copy with a 1K buffer. 8K is the standard these ways # Improve code design; do not catch one's own exception, do not log and throw (only do one or the other, never both) # Refactor to make a new method for copy Reporter: BELUGA BEHR Attachments: HDFS-14106.1.patch -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14107) FileContext Delete on Exit Improvements
BELUGA BEHR created HDFS-14107:
--
Summary: FileContext Delete on Exit Improvements
Key: HDFS-14107
URL: https://issues.apache.org/jira/browse/HDFS-14107
Project: Hadoop HDFS
Issue Type: Improvement
Components: fs
Affects Versions: 3.2.0
Reporter: BELUGA BEHR
{code:java|FileContext.java}
synchronized (DELETE_ON_EXIT) {
Set>> set = DELETE_ON_EXIT.entrySet();
for (Entry> entry : set) {
FileContext fc = entry.getKey();
Set paths = entry.getValue();
for (Path path : paths) {
try {
fc.delete(path, true);
} catch (IOException e) {
LOG.warn("Ignoring failure to deleteOnExit for path " + path);
}
}
}
DELETE_ON_EXIT.clear();
{code}
# Include the {{IOException}} in the logging so that admins can know why the
file was not deleted
# Do not bother clearing out the data structure. This code is only called if
the JVM is going down. Better to spend the time allowing another shutdown hook
to run than to spend time cleaning this thing up.
# Use Guava {{MultiMap}} for readability
# Paths are currently stored in a {{TreeSet}}. This set implementation orders
the files by names. It does not seem worth much to order the files. Use a
faster {{HashSet}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14108) BlockManager Data Structures
BELUGA BEHR created HDFS-14108:
--
Summary: BlockManager Data Structures
Key: HDFS-14108
URL: https://issues.apache.org/jira/browse/HDFS-14108
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs
Affects Versions: 3.2.0
Reporter: BELUGA BEHR
Assignee: BELUGA BEHR
Attachments: HDFS-14108.1.patch
# Prefer {{ArrayList}} to {{LinkedList}} when simply adding/iterating
# Prefer {{HashSet}} to {{TreeSet}} when no ordering is required
# Other performance improvements
# Check style fixes
https://stackoverflow.com/questions/322715/when-to-use-linkedlist-over-arraylist-in-java
{code:java}
final Set excludedNodes = new HashSet<>();
for(BlockReconstructionWork rw : reconWork){
// Do no bother wasting time clearing out the collection, let GC do that
work later
excludedNodes.clear();
// use {{addAll}} here
for (DatanodeDescriptor dn : rw.getContainingNodes()) {
excludedNodes.add(dn);
}
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14119) GreedyPlanner Parameter Logging
BELUGA BEHR created HDFS-14119:
--
Summary: GreedyPlanner Parameter Logging
Key: HDFS-14119
URL: https://issues.apache.org/jira/browse/HDFS-14119
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs
Affects Versions: 3.3.0
Reporter: BELUGA BEHR
Attachments: HDFS-14119.1.patch
1. Do not use {{String.format()}} in conjunction with SLF4J. Superfluous.
{code:java}
String message = String
.format("Compute Plan for Node : %s:%d took %d ms ",
node.getDataNodeName(), node.getDataNodePort(),
endTime - startTime);
LOG.info(message);{code}
2. Do not call an explicit toString() on an object with SLF4J parameter.
Otherwise, the string will be created and then thrown away if the logger is not
set to debug level. Just pass the object itself and the framework will call
{{toString}} if needed.
{code}
LOG.debug("Step : {} ", nextStep.toString());
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14177) BlockChecksumHelper Readability
BELUGA BEHR created HDFS-14177: -- Summary: BlockChecksumHelper Readability Key: HDFS-14177 URL: https://issues.apache.org/jira/browse/HDFS-14177 Project: Hadoop HDFS Issue Type: Improvement Components: datanode Affects Versions: 3.2.0 Reporter: BELUGA BEHR * Remove magic number * Replace routing code with libraries to increase readability -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14256) Review Logging of NameNode Class
BELUGA BEHR created HDFS-14256: -- Summary: Review Logging of NameNode Class Key: HDFS-14256 URL: https://issues.apache.org/jira/browse/HDFS-14256 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Affects Versions: 3.2.0 Reporter: BELUGA BEHR -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14258) Introduce Java Concurrent Package To DataXceiverServer Class
BELUGA BEHR created HDFS-14258:
--
Summary: Introduce Java Concurrent Package To DataXceiverServer
Class
Key: HDFS-14258
URL: https://issues.apache.org/jira/browse/HDFS-14258
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 3.2.0
Reporter: BELUGA BEHR
* Use Java concurrent package to replace current facilities in
{{DataXceiverServer}}.
* A little bit of extra clean up
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14260) Synchronized Method in BlockReceiver Can Be Replace with Atomic Value
BELUGA BEHR created HDFS-14260: -- Summary: Synchronized Method in BlockReceiver Can Be Replace with Atomic Value Key: HDFS-14260 URL: https://issues.apache.org/jira/browse/HDFS-14260 Project: Hadoop HDFS Issue Type: Improvement Components: datanode Affects Versions: 3.2.0 Reporter: BELUGA BEHR -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14287) DataXceiverServer May Double-Close
BELUGA BEHR created HDFS-14287:
--
Summary: DataXceiverServer May Double-Close
Key: HDFS-14287
URL: https://issues.apache.org/jira/browse/HDFS-14287
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 3.2.0
Reporter: BELUGA BEHR
Attachments: HDFS-14287.1.patch
When a call to [HDFS-14258], when a user calls the {{kill()}} method, it closes
the {{PeerServer}}. This causes the thread reading from the {{PeerServer}} to
interrupt and exit. On the way out, this thread also closes the {{PeerServer}}
for good measure. I'm not sure what the affects of closing the {{PeerServer}}
are twice, but it should be avoided.
To be clear, this is not a regressions caused by [HDFS-14258], if anything,
[HDFS-14258] makes it easier to fix. :)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14292) Introduce Java ExecutorService to DataNode
BELUGA BEHR created HDFS-14292:
--
Summary: Introduce Java ExecutorService to DataNode
Key: HDFS-14292
URL: https://issues.apache.org/jira/browse/HDFS-14292
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 3.2.0
Reporter: BELUGA BEHR
I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
{{hdfs-site.xml}}. It is described as "Specifies the maximum number of threads
to use for transferring data in and out of the DN." The default value is
4096. I found it interesting because 4096 threads sounds like a lot to me.
I'm not sure how a system with 8-16 cores would react to this large a thread
count. Intuitively, I would say that the overhead of context switching would
be immense.
During mt investigation, I discovered the
[following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
setup in the {{DataXceiverServer}} class:
# A peer connects to a DataNode
# A new thread is spun up to service this connection
# The thread runs to completion
# The tread dies
It would perhaps be better if we used a thread pool to better manage the
lifecycle of the service threads and to allow the DataNode to re-use existing
threads, saving on the need to create and spin-up threads on demand.
In this JIRA, I have added a couple of things:
# Added a thread pool that will always maintain a single thread running, always
awaiting a new connection should one arrive. On-demand, it will create up to
{{dfs.datanode.max.transfer.threads}}. A thread that has completed its prior
duties will stay idle for up to 30 seconds, it will be retired if no new work
has arrived.
# Added new methods to the {{Peer}} Interface to allow for better logging and
less code within each Thread ({{DataXceiver}}).
# Updated the Thread code ({{DataXceiver}}) regarding its interactions with
{{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14293) Increase Default Size of dfs.stream-buffer-size
BELUGA BEHR created HDFS-14293:
--
Summary: Increase Default Size of dfs.stream-buffer-size
Key: HDFS-14293
URL: https://issues.apache.org/jira/browse/HDFS-14293
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 3.2.0
Reporter: BELUGA BEHR
For many years (7+) now, the JDK has been using a default buffer size of [8192
bytes|https://github.com/openjdk-mirror/jdk7u-jdk/blob/master/src/share/classes/java/io/BufferedInputStream.java#L53].
Hadoop still defaults to a size half of that. The default is
{{dfs.stream-buffer-size}} is 4096 bytes.
https://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml
Please increase default size to 8192.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14294) Deprecate DFSUtilClient#getSmallBufferSize
BELUGA BEHR created HDFS-14294:
--
Summary: Deprecate DFSUtilClient#getSmallBufferSize
Key: HDFS-14294
URL: https://issues.apache.org/jira/browse/HDFS-14294
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs
Affects Versions: 3.2.0
Reporter: BELUGA BEHR
{code:java|title=DFSUtilClient.java}
public static int getIoFileBufferSize(Configuration conf) {
return conf.getInt(
CommonConfigurationKeysPublic.IO_FILE_BUFFER_SIZE_KEY,
CommonConfigurationKeysPublic.IO_FILE_BUFFER_SIZE_DEFAULT);
}
public static int getSmallBufferSize(Configuration conf) {
return Math.min(getIoFileBufferSize(conf) / 2, 512);
}
{code}
This concept of a "small buffer size" seems a bit overkill. First of all, it's
not documented that such a thing exists and that by adjusting
{{dfs.stream-buffer-size}} an administrator is also scaling these other buffer
sizes. Seconds, I think any "small" buffer size should just use the default JDK
buffer sizes. Anything that benefits from being larger than the default JDK
size should be the controlled by {{IO_FILE_BUFFER_SIZE_KEY}} /
{{dfs.stream-buffer-size}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14295) Add Threadpool for DataTransfers
BELUGA BEHR created HDFS-14295:
--
Summary: Add Threadpool for DataTransfers
Key: HDFS-14295
URL: https://issues.apache.org/jira/browse/HDFS-14295
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 3.2.0
Reporter: BELUGA BEHR
When a DataNode data transfers a block, is spins up a new thread for each
transfer.
[Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
and
[Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
Instead, add the threads to a {{CachedThreadPool}} so that when their
threads complete the transfer, they can be re-used for another transfer. This
should save resources spent on creating and spinning up transfer threads.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14296) Prefer ArrayList over LinkedList in VolumeScanner
BELUGA BEHR created HDFS-14296:
--
Summary: Prefer ArrayList over LinkedList in VolumeScanner
Key: HDFS-14296
URL: https://issues.apache.org/jira/browse/HDFS-14296
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 3.2.0
Reporter: BELUGA BEHR
Assignee: BELUGA BEHR
Attachments: HDFS-14296.1.patch
{quote}The {{size}}, {{isEmpty}}, {{get}}, {{set}}, {{iterator}}, and
{{listIterator}} operations run in constant time. - ArrayList
{quote}
However, for a {{LinkedList}}, the entire list must be traversed to get to the
desired index.
Like
[Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/VolumeScanner.java#L384]
Most of the time, the List is being iterated, which is quicker over a primitive
array than walking a LinkedList.
There is one place where an item is removed, potentially from the middle of the
list
[here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/VolumeScanner.java#L736]
but the speed of removing from the middle of the list isn't bad; it's a system
native array shift, and it only happens on the off chance that a block pool is
removed from the DataNode.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13155) Not Checking Return Value for NULL
BELUGA BEHR created HDFS-13155:
--
Summary: Not Checking Return Value for NULL
Key: HDFS-13155
URL: https://issues.apache.org/jira/browse/HDFS-13155
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Affects Versions: 3.0.0
Reporter: BELUGA BEHR
{code:title=BlockPlacementPolicyDefault.java}
protected Node chooseTargetInOrder(int numOfReplicas,
Node writer,
final Set excludedNodes,
final long blocksize,
final int maxNodesPerRack,
final List results,
final boolean avoidStaleNodes,
final boolean newBlock,
EnumMap storageTypes)
throws NotEnoughReplicasException {
final int numOfResults = results.size();
if (numOfResults == 0) {
writer = chooseLocalStorage(writer, excludedNodes, blocksize,
maxNodesPerRack, results, avoidStaleNodes, storageTypes, true)
.getDatanodeDescriptor();
if (--numOfReplicas == 0) {
return writer;
}
}
...
{code}
The method {{chooseLocalStorage}} can return a _null_ value but it's not being
checked here and the method {{getDatanodeDescriptor()}} is immediately being
called on the result. Please check for a _null_ value first.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13156) HDFS Block Placement Policy - Client Local Rack
BELUGA BEHR created HDFS-13156:
--
Summary: HDFS Block Placement Policy - Client Local Rack
Key: HDFS-13156
URL: https://issues.apache.org/jira/browse/HDFS-13156
Project: Hadoop HDFS
Issue Type: Improvement
Components: documentation
Affects Versions: 2.9.0
Reporter: BELUGA BEHR
{quote}For the common case, when the replication factor is three, HDFS’s
placement policy is to put one replica on the local machine if the writer is on
a datanode, otherwise on a random datanode, another replica on a node in a
different (remote) rack, and the last on a different node in the same remote
rack.
{quote}
[https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html#Replica_Placement:_The_First_Baby_Steps]
Having just looked over the Default Block Placement code, the way I understand
this, is that, there are three basic scenarios:
# HDFS client is running on a datanode inside the cluster
# HDFS client is running on a node outside the cluster
# HDFS client is running on a non-datanode inside the cluster
The documentation is ambiguous concerning the third scenario. Please correct me
if I'm wrong, but the way I understand the code, if there is an HDFS client
inside the cluster, but it is not on a datanode, the first block will be placed
on a datanode within the set of datanodes available on the local rack and not
simply on any _random datanode_ from the set of all datanodes in the cluster.
That is to say, if one rack has an HDFS Sink Flume Agent on a dedicated node, I
should expect that every first block will be written to a _random datanode_ on
the same rack as the HDFS Flume agent, assuming the network topology script is
written to include this Flume node.
If that is correct, can the documentation be updated to include this third
common scenario?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13157) Do Not Remove Blocks Sequentially During Dec omission
BELUGA BEHR created HDFS-13157: -- Summary: Do Not Remove Blocks Sequentially During Dec omission Key: HDFS-13157 URL: https://issues.apache.org/jira/browse/HDFS-13157 Project: Hadoop HDFS Issue Type: Improvement Components: datanode, namenode Affects Versions: 3.0.0 Reporter: BELUGA BEHR >From what I understand of [DataNode >decommissioning|https://github.com/apache/hadoop/blob/42a1c98597e6dba2e371510a6b2b6b1fb94e4090/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeAdminManager.java] > it appears that all the blocks are scheduled for removal _in order._. I'm >not 100% sure what the ordering is exactly, but I think it's loops through >each data volume and schedules each block to be replicated elsewhere. The net >affect is that during a decommission, all of the DataNode transfer threads >slam on a single volume until it is cleaned out. At which point, they all >slam on the next volume, etc. Please randomize the block list so that there is a more even distribution across all volumes when decommissioning a node. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13158) Fix Spelling MIstakes - DECOMISSIONED
BELUGA BEHR created HDFS-13158: -- Summary: Fix Spelling MIstakes - DECOMISSIONED Key: HDFS-13158 URL: https://issues.apache.org/jira/browse/HDFS-13158 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 3.0.0 Reporter: BELUGA BEHR https://github.com/apache/hadoop/search?l=Java&q=DECOMISSIONED There are references in the code to _DECOMISSIONED_ but the correct spelling is _DECOMMISSIONED_. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13167) DatanodeAdminManager Improvements
BELUGA BEHR created HDFS-13167:
--
Summary: DatanodeAdminManager Improvements
Key: HDFS-13167
URL: https://issues.apache.org/jira/browse/HDFS-13167
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs
Affects Versions: 3.0.0
Reporter: BELUGA BEHR
# Use Collection type Set instead of List for tracking nodes
# Fix logging statements that are erroneously appending variables instead of
using parameters
# Miscellaneous small improvements
As an example, the {{node}} variable is being appended to the string instead of
being passed as an argument to the {{trace}} method for variable substitution.
{code}
LOG.trace("stopDecommission: Node {} in {}, nothing to do." +
node, node.getAdminState());
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13168) XmlImageVisitor - Prefer Array over LinkedList
BELUGA BEHR created HDFS-13168:
--
Summary: XmlImageVisitor - Prefer Array over LinkedList
Key: HDFS-13168
URL: https://issues.apache.org/jira/browse/HDFS-13168
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs
Affects Versions: 3.0.0
Environment: {{ArrayDeque}}
{quote}This class is likely to be faster than Stack when used as a stack, and
faster than LinkedList when used as a queue.{quote}
.. not to mention less memory fragmentation (single backing array v.s. many
ArrayList nodes).
https://docs.oracle.com/javase/8/docs/api/java/util/ArrayDeque.html
Reporter: BELUGA BEHR
Attachments: HDFS-13168.1.patch
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13193) Various Improvements for BlockTokenSecretManager.java
BELUGA BEHR created HDFS-13193:
--
Summary: Various Improvements for BlockTokenSecretManager.java
Key: HDFS-13193
URL: https://issues.apache.org/jira/browse/HDFS-13193
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs
Affects Versions: 3.0.0
Reporter: BELUGA BEHR
Attachments: HDFS-13193.1.patch
Various improvements for class
{{org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.java}}
# Remove superfluous {{toString}} calls
# Fix some checkstyle warnings
# Re-implement method with O(N^2) with HashMultiSet - also improves readability
# Increase code re-use with Apache Commons Library
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13428) Remove LinkedList From StateStoreFileImpl.java
BELUGA BEHR created HDFS-13428:
--
Summary: Remove LinkedList From StateStoreFileImpl.java
Key: HDFS-13428
URL: https://issues.apache.org/jira/browse/HDFS-13428
Project: Hadoop HDFS
Issue Type: Improvement
Components: federation
Affects Versions: 3.0.1
Environment: Replace {{LinkedList}} with {{ArrayList}} implementation
in the StateStoreFileImpl class. This is especially advantageous because we
can pre-allocate the internal array before a copy occurs. {{ArrayList}} is
faster for iterations and requires less memory than {{LinkedList}}.
{code:java}
protected List getChildren(String path) {
List ret = new LinkedList<>();
File dir = new File(path);
File[] files = dir.listFiles();
if (files != null) {
for (File file : files) {
String filename = file.getName();
ret.add(filename);
}
}
return ret;
}{code}
Reporter: BELUGA BEHR
Attachments: HDFS-13428.1.patch
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13447) Fix Typos - Node Not Chosen
BELUGA BEHR created HDFS-13447:
--
Summary: Fix Typos - Node Not Chosen
Key: HDFS-13447
URL: https://issues.apache.org/jira/browse/HDFS-13447
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Affects Versions: 3.0.1, 2.2.0
Reporter: BELUGA BEHR
Attachments: HDFS-13447.1.patch
Fix typo and improve:
{code:java}
private enum NodeNotChosenReason {
NOT_IN_SERVICE("the node isn't in service"),
NODE_STALE("the node is stale"),
NODE_TOO_BUSY("the node is too busy"),
TOO_MANY_NODES_ON_RACK("the rack has too many chosen nodes"),
NOT_ENOUGH_STORAGE_SPACE("no enough storage space to place the block");{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13448) HDFS Block Placement - Ignore Locality
BELUGA BEHR created HDFS-13448:
--
Summary: HDFS Block Placement - Ignore Locality
Key: HDFS-13448
URL: https://issues.apache.org/jira/browse/HDFS-13448
Project: Hadoop HDFS
Issue Type: New Feature
Components: block placement, hdfs-client
Affects Versions: 3.0.1, 2.9.0
Reporter: BELUGA BEHR
According to the HDFS Block Place Rules:
{quote}
/**
* The replica placement strategy is that if the writer is on a datanode,
* the 1st replica is placed on the local machine,
* otherwise a random datanode. The 2nd replica is placed on a datanode
* that is on a different rack. The 3rd replica is placed on a datanode
* which is on a different node of the rack as the second replica.
*/
{quote}
However, there is a hint for the hdfs-client that allows the block placement
request to not put a block replica on the local datanode _where 'local' means
the same host as the client is being run on._
{quote}
/**
* Advise that a block replica NOT be written to the local DataNode where
* 'local' means the same host as the client is being run on.
*
* @see CreateFlag#NO_LOCAL_WRITE
*/
{quote}
I propose that we add a new flag that allows the hdfs-client to request that
the first block replica be placed on a random DataNode in the cluster. The
subsequent block replicas should follow the normal block placement rules.
The issue is that when the {{NO_LOCAL_WRITE}} is enabled, the first block
replica is not placed on the local node, but it is still placed on the local
rack. Where this comes into play is where you have, for example, a flume agent
that is loading data into HDFS.
If the Flume agent is running on a DataNode, then by default, the DataNode
local to the Flume agent will always get the first block replica and this leads
to un-even block placements, with the local node always filling up faster than
any other node in the cluster.
Modifying this example, if the DataNode is removed from the host where the
Flume agent is running, or this {{NO_LOCAL_WRITE}} is enabled by Flume, then
the default block placement policy will still prefer the local rack. This
remedies the situation only so far as now the first block replica will always
be distributed to a DataNode on the local rack.
This new flag would allow a single Flume agent to distribute the blocks
randomly, evenly, over the entire cluster instead of hot-spotting the local
node or the local rack.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13684) Remove Use of Deprecated Whitebox Test Class
BELUGA BEHR created HDFS-13684:
--
Summary: Remove Use of Deprecated Whitebox Test Class
Key: HDFS-13684
URL: https://issues.apache.org/jira/browse/HDFS-13684
Project: Hadoop HDFS
Issue Type: Improvement
Components: test
Affects Versions: 3.1.0, 2.10.0
Reporter: BELUGA BEHR
Unit tests within the Hadoop suite are using the now deprecated class
{{org.apache.hadoop.test.Whitebox}}.
bq. This class was ported from org.mockito.internal.util.reflection.Whitebox
since the class was removed in Mockito 2.1. Using this class is a bad practice.
Consider refactoring instead of using this.
As stated, refactor the existing tests to remove calls to this class... then
remove this class from the project.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13685) Review and Refactor TestDFSOutputStream.java
BELUGA BEHR created HDFS-13685:
--
Summary: Review and Refactor TestDFSOutputStream.java
Key: HDFS-13685
URL: https://issues.apache.org/jira/browse/HDFS-13685
Project: Hadoop HDFS
Issue Type: Sub-task
Components: hdfs-client
Affects Versions: 3.1.0, 2.10.0
Reporter: BELUGA BEHR
Remove use of deprecated class {{org.apache.hadoop.test.Whitebox}}
Refactor the necessary code to make mocking easier
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13812) Update Docs on Caching
BELUGA BEHR created HDFS-13812:
--
Summary: Update Docs on Caching
Key: HDFS-13812
URL: https://issues.apache.org/jira/browse/HDFS-13812
Project: Hadoop HDFS
Issue Type: Improvement
Components: documentation
Affects Versions: 2.9.1
Reporter: BELUGA BEHR
{quote}
dfs.namenode.path.based.cache.refresh.interval.ms
The NameNode will use this as the amount of milliseconds between subsequent
path cache rescans. This calculates the blocks to cache and each DataNode
containing a replica of the block that should cache it.
By default, this parameter is set to 30, which is five minutes.
{quote}
[https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/CentralizedCacheManagement.html]
However, this default value was change in [HDFS-6106] to 30 seconds. Please
update docs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13820) Disable CacheReplicationMonitor If No Cached Paths Exist
BELUGA BEHR created HDFS-13820:
--
Summary: Disable CacheReplicationMonitor If No Cached Paths Exist
Key: HDFS-13820
URL: https://issues.apache.org/jira/browse/HDFS-13820
Project: Hadoop HDFS
Issue Type: Improvement
Components: caching
Affects Versions: 2.10.0, 3.2.0
Reporter: BELUGA BEHR
Stating with [HDFS-6106] the loop for checking caching is set to be every 30
seconds.
Please implement a way to disable the {{CacheReplicationMonitor}} class if
there are no paths specified. Adding the first cached path to the NameNode
should kick off the {{CacheReplicationMonitor}} and when the last one is
deleted, the {{CacheReplicationMonitor}} should be disabled again.
Alternatively, provide a configuration flag to turn this feature off altogether.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-9457) Datanode Logging Improvement
BELUGA BEHR created HDFS-9457: - Summary: Datanode Logging Improvement Key: HDFS-9457 URL: https://issues.apache.org/jira/browse/HDFS-9457 Project: Hadoop HDFS Issue Type: Improvement Components: datanode Affects Versions: 2.7.0, 3.0.0 Reporter: BELUGA BEHR Priority: Trivial Please accept my patch for some minor clean-up of logging. Patch is against 3.0.0 but applies to previous versions as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-9492) RoundRobinVolumeChoosingPolicy
BELUGA BEHR created HDFS-9492: - Summary: RoundRobinVolumeChoosingPolicy Key: HDFS-9492 URL: https://issues.apache.org/jira/browse/HDFS-9492 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Affects Versions: 2.7.1 Reporter: BELUGA BEHR Priority: Trivial This is some general clean-up for: RoundRobinVolumeChoosingPolicy I have also updated and expanded the unit tests a bit. There is one error message being generated that I changed. I felt the previous Exception message was not that helpful and therefore it was possible to trim it down. If the exception message must be enhanced, the entire list of "volumes" should be included. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-9507) LeaseRenewer Logging Under-Reporting
BELUGA BEHR created HDFS-9507:
-
Summary: LeaseRenewer Logging Under-Reporting
Key: HDFS-9507
URL: https://issues.apache.org/jira/browse/HDFS-9507
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Affects Versions: 2.7.1
Reporter: BELUGA BEHR
Priority: Minor
Why is it that in LeaseRenewer#run() failures to renew a lease on a file are
reported with "warn" level logging, but in LeaseRenewer#renew() it is reported
with a "debug" level warn?
In LeaseRenewer#renew(), if the method renewLease() returns 'false' then the
problem is silently discarded (continue, no Exception is thrown) and the next
client in the list tries to renew.
{code:title=LeaseRenewer.java|borderStyle=solid}
private void run(final int id) throws InterruptedException {
...
try {
renew();
lastRenewed = Time.monotonicNow();
} catch (SocketTimeoutException ie) {
LOG.warn("Failed to renew lease for " + clientsString() + " for "
+ (elapsed/1000) + " seconds. Aborting ...", ie);
synchronized (this) {
while (!dfsclients.isEmpty()) {
DFSClient dfsClient = dfsclients.get(0);
dfsClient.closeAllFilesBeingWritten(true);
closeClient(dfsClient);
}
//Expire the current LeaseRenewer thread.
emptyTime = 0;
}
break;
} catch (IOException ie) {
LOG.warn("Failed to renew lease for " + clientsString() + " for "
+ (elapsed/1000) + " seconds. Will retry shortly ...", ie);
}
}
...
}
private void renew() throws IOException {
{
...
if (!c.renewLease()) {
LOG.debug("Did not renew lease for client {}", c);
continue;
}
...
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
