[jira] [Created] (HDFS-11039) Expose more configuration properties to hdfs-default.xml
Yi Liu created HDFS-11039: - Summary: Expose more configuration properties to hdfs-default.xml Key: HDFS-11039 URL: https://issues.apache.org/jira/browse/HDFS-11039 Project: Hadoop HDFS Issue Type: Improvement Components: documentation, newbie Reporter: Yi Liu Assignee: Jennica Pounds Priority: Minor There are some configuration properties for hdfs, but have not been exposed in hdfs-default.xml. It's convenient for Hadoop user/admin if we add them in the hdfs-default.xml. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-11020) Enable HDFS transparent encryption doc
Yi Liu created HDFS-11020: - Summary: Enable HDFS transparent encryption doc Key: HDFS-11020 URL: https://issues.apache.org/jira/browse/HDFS-11020 Project: Hadoop HDFS Issue Type: Task Components: documentation, encryption, fs Reporter: Yi Liu Assignee: Yi Liu Priority: Minor We need correct version of Openssl which supports hardware acceleration of AES CTR. Let's add more doc about how to configure the correct Openssl. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Reopened] (HDFS-4937) ReplicationMonitor can infinite-loop in BlockPlacementPolicyDefault#chooseRandom()
[
https://issues.apache.org/jira/browse/HDFS-4937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liu reopened HDFS-4937:
--
> ReplicationMonitor can infinite-loop in
> BlockPlacementPolicyDefault#chooseRandom()
> --
>
> Key: HDFS-4937
> URL: https://issues.apache.org/jira/browse/HDFS-4937
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.0.4-alpha, 0.23.8
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
> Labels: BB2015-05-TBR
> Fix For: 3.0.0, 2.7.2
>
> Attachments: HDFS-4937.patch, HDFS-4937.v1.patch, HDFS-4937.v2.patch
>
>
> When a large number of nodes are removed by refreshing node lists, the
> network topology is updated. If the refresh happens at the right moment, the
> replication monitor thread may stuck in the while loop of {{chooseRandom()}}.
> This is because the cached cluster size is used in the terminal condition
> check of the loop. This usually happens when a block with a high replication
> factor is being processed. Since replicas/rack is also calculated beforehand,
> no node choice may satisfy the goodness criteria if refreshing removed racks.
> All nodes will end up in the excluded list, but the size will still be less
> than the cached cluster size, so it will loop infinitely. This was observed
> in a production environment.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Reopened] (HDFS-9293) FSEditLog's 'OpInstanceCache' instance of threadLocal cache exists dirty 'rpcId',which may cause standby NN too busy to communicate
[ https://issues.apache.org/jira/browse/HDFS-9293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Liu reopened HDFS-9293: -- > FSEditLog's 'OpInstanceCache' instance of threadLocal cache exists dirty > 'rpcId',which may cause standby NN too busy to communicate > -- > > Key: HDFS-9293 > URL: https://issues.apache.org/jira/browse/HDFS-9293 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.2.0, 2.7.1 >Reporter: 邓飞 >Assignee: 邓飞 > Fix For: 2.7.1 > > > In our cluster (hadoop 2.2.0-HA,700+ DN),we found standby NN tail editlog > slowly,and hold the fsnamesystem writelock during the work and the DN's > heartbeart/blockreport IPC request blocked.Lead to Active NN remove stale DN > which can't send heartbeat because blocking at process Standby NN Regiest > common(FIXED at 2.7.1). > Below is the standby NN stack: > "Edit log tailer" prio=10 tid=0x7f28fcf35800 nid=0x1a7d runnable > [0x7f0dd1d76000] >java.lang.Thread.State: RUNNABLE > at java.util.PriorityQueue.remove(PriorityQueue.java:360) > at > org.apache.hadoop.util.LightWeightCache.put(LightWeightCache.java:217) > at org.apache.hadoop.ipc.RetryCache.addCacheEntry(RetryCache.java:270) > - locked <0x7f12817714b8> (a org.apache.hadoop.ipc.RetryCache) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.addCacheEntry(FSNamesystem.java:724) > at > org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:406) > at > org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:199) > at > org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:112) > at > org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:733) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.doTailEdits(EditLogTailer.java:227) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:321) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:279) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:296) > at > org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:456) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:292) > > When apply editLogOp,if the IPC retryCache is found,need to remove the > previous from priorityQueue(O(N)), The updateblock is don't need record > rpcId on editlog except 'client request updatePipeline',but we found many > 'UpdateBlocksOp' has repeat ipcId. > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HDFS-9293) FSEditLog's 'OpInstanceCache' instance of threadLocal cache exists dirty 'rpcId',which may cause standby NN too busy to communicate
[ https://issues.apache.org/jira/browse/HDFS-9293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Liu resolved HDFS-9293. -- Resolution: Duplicate > FSEditLog's 'OpInstanceCache' instance of threadLocal cache exists dirty > 'rpcId',which may cause standby NN too busy to communicate > -- > > Key: HDFS-9293 > URL: https://issues.apache.org/jira/browse/HDFS-9293 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.2.0, 2.7.1 >Reporter: 邓飞 >Assignee: 邓飞 > Fix For: 2.7.1 > > > In our cluster (hadoop 2.2.0-HA,700+ DN),we found standby NN tail editlog > slowly,and hold the fsnamesystem writelock during the work and the DN's > heartbeart/blockreport IPC request blocked.Lead to Active NN remove stale DN > which can't send heartbeat because blocking at process Standby NN Regiest > common(FIXED at 2.7.1). > Below is the standby NN stack: > "Edit log tailer" prio=10 tid=0x7f28fcf35800 nid=0x1a7d runnable > [0x7f0dd1d76000] >java.lang.Thread.State: RUNNABLE > at java.util.PriorityQueue.remove(PriorityQueue.java:360) > at > org.apache.hadoop.util.LightWeightCache.put(LightWeightCache.java:217) > at org.apache.hadoop.ipc.RetryCache.addCacheEntry(RetryCache.java:270) > - locked <0x7f12817714b8> (a org.apache.hadoop.ipc.RetryCache) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.addCacheEntry(FSNamesystem.java:724) > at > org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:406) > at > org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:199) > at > org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:112) > at > org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:733) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.doTailEdits(EditLogTailer.java:227) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:321) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:279) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:296) > at > org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:456) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:292) > > When apply editLogOp,if the IPC retryCache is found,need to remove the > previous from priorityQueue(O(N)), The updateblock is don't need record > rpcId on editlog except 'client request updatePipeline',but we found many > 'UpdateBlocksOp' has repeat ipcId. > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-9274) Default value of dfs.datanode.directoryscan.throttle.limit.ms.per.sec should be consistent
Yi Liu created HDFS-9274:
Summary: Default value of
dfs.datanode.directoryscan.throttle.limit.ms.per.sec should be consistent
Key: HDFS-9274
URL: https://issues.apache.org/jira/browse/HDFS-9274
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Reporter: Yi Liu
Assignee: Yi Liu
Priority: Trivial
Always see following error log while running:
{noformat}
ERROR datanode.DirectoryScanner (DirectoryScanner.java:(430)) -
dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1
ms/sec. Assuming default value of 1000
{noformat}
{code}
dfs.datanode.directoryscan.throttle.limit.ms.per.sec
0
...
{code}
The default value should be 1000 and consistent with
DFS_DATANODE_DIRECTORYSCAN_THROTTLE_LIMIT_MS_PER_SEC_DEFAULT
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HDFS-8398) Erasure Coding: Correctly calculate last striped block length in DFSStripedInputStream if it's under construction.
[ https://issues.apache.org/jira/browse/HDFS-8398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Liu resolved HDFS-8398. -- Resolution: Duplicate > Erasure Coding: Correctly calculate last striped block length in > DFSStripedInputStream if it's under construction. > -- > > Key: HDFS-8398 > URL: https://issues.apache.org/jira/browse/HDFS-8398 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Yi Liu >Assignee: Yi Liu > > Currently in DFSStripedInputStream, for continuous block, if it's under > construction, we need to read the block replica length from one of datanode > and use it as last block length. > For striped block, we need to read the length of all internal data blocks of > the striped group, then add them correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HDFS-8059) Erasure coding: revisit how to store EC schema and cellSize in NameNode
[
https://issues.apache.org/jira/browse/HDFS-8059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liu resolved HDFS-8059.
--
Resolution: Duplicate
> Erasure coding: revisit how to store EC schema and cellSize in NameNode
> ---
>
> Key: HDFS-8059
> URL: https://issues.apache.org/jira/browse/HDFS-8059
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-7285
>Reporter: Yi Liu
>Assignee: Yi Liu
> Attachments: HDFS-8059.001.patch
>
>
> Move {{dataBlockNum}} and {{parityBlockNum}} from BlockInfoStriped to
> INodeFile, and store them in {{FileWithStripedBlocksFeature}}.
> Ideally these two nums are the same for all striped blocks in a file, and
> store them in BlockInfoStriped will waste NN memory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HDFS-8087) Erasure Coding: Add more EC zone management APIs (get/list EC zone(s))
[ https://issues.apache.org/jira/browse/HDFS-8087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Liu resolved HDFS-8087. -- Resolution: Invalid Now, getErasureCodingPolicies and getErasureCodingPolicy are implemented, so close it as Invalid. > Erasure Coding: Add more EC zone management APIs (get/list EC zone(s)) > -- > > Key: HDFS-8087 > URL: https://issues.apache.org/jira/browse/HDFS-8087 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Yi Liu >Assignee: Yi Liu > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HDFS-8376) Erasure Coding: Update last cellsize calculation according to whether the erasure codec has chunk boundary
[
https://issues.apache.org/jira/browse/HDFS-8376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liu resolved HDFS-8376.
--
Resolution: Invalid
> Erasure Coding: Update last cellsize calculation according to whether the
> erasure codec has chunk boundary
> --
>
> Key: HDFS-8376
> URL: https://issues.apache.org/jira/browse/HDFS-8376
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Yi Liu
>Assignee: Yi Liu
>
> Current calculation for last cell size is as following. For parity cell, the
> last cell size is the same as the first data cell. But some erasure codec
> has chunk boundary, then the last cellsize for parity block is the codec
> chunk size.
> {code}
> private static int lastCellSize(int size, int cellSize, int numDataBlocks,
> int i) {
> if (i < numDataBlocks) {
> // parity block size (i.e. i >= numDataBlocks) is the same as
> // the first data block size (i.e. i = 0).
> size -= i*cellSize;
> if (size < 0) {
> size = 0;
> }
> }
> return size > cellSize? cellSize: size;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HDFS-9183) Cleanup the findbugs and other issues after HDFS EC merged to trunk.
[ https://issues.apache.org/jira/browse/HDFS-9183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Liu resolved HDFS-9183. -- Resolution: Duplicate Assignee: (was: Yi Liu) > Cleanup the findbugs and other issues after HDFS EC merged to trunk. > > > Key: HDFS-9183 > URL: https://issues.apache.org/jira/browse/HDFS-9183 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Yi Liu >Priority: Critical > > https://builds.apache.org/job/PreCommit-HDFS-Build/12754/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs-client.html > https://builds.apache.org/job/PreCommit-HDFS-Build/12754/artifact/patchprocess/patchReleaseAuditProblems.txt -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-9183) Cleanup the findbugs and other issues after HDFS EC merged to trunk.
Yi Liu created HDFS-9183: Summary: Cleanup the findbugs and other issues after HDFS EC merged to trunk. Key: HDFS-9183 URL: https://issues.apache.org/jira/browse/HDFS-9183 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 3.0.0 Reporter: Yi Liu Assignee: Yi Liu Priority: Critical https://builds.apache.org/job/PreCommit-HDFS-Build/12754/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs-client.html https://builds.apache.org/job/PreCommit-HDFS-Build/12754/artifact/patchprocess/patchReleaseAuditProblems.txt -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-9182) Cleanup the findbugs and other issues after HDFS EC merged to trunk.
Yi Liu created HDFS-9182: Summary: Cleanup the findbugs and other issues after HDFS EC merged to trunk. Key: HDFS-9182 URL: https://issues.apache.org/jira/browse/HDFS-9182 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 3.0.0 Reporter: Yi Liu Assignee: Yi Liu Priority: Critical https://builds.apache.org/job/PreCommit-HDFS-Build/12754/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs-client.html https://builds.apache.org/job/PreCommit-HDFS-Build/12754/artifact/patchprocess/patchReleaseAuditProblems.txt -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-9174) Fix the latest findbugs of FSOutputSummer.tracer and DirectoryScanner$ReportCompiler.currentThread
Yi Liu created HDFS-9174: Summary: Fix the latest findbugs of FSOutputSummer.tracer and DirectoryScanner$ReportCompiler.currentThread Key: HDFS-9174 URL: https://issues.apache.org/jira/browse/HDFS-9174 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.8.0 Reporter: Yi Liu Assignee: Yi Liu Priority: Minor https://builds.apache.org/job/PreCommit-HDFS-Build/12739/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html https://builds.apache.org/job/PreCommit-HDFS-Build/12739/artifact/patchprocess/trunkFindbugsWarningshadoop-common.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-9176) TestDirectoryScanner#testThrottling often fails.
Yi Liu created HDFS-9176: Summary: TestDirectoryScanner#testThrottling often fails. Key: HDFS-9176 URL: https://issues.apache.org/jira/browse/HDFS-9176 Project: Hadoop HDFS Issue Type: Bug Components: test Affects Versions: 2.8.0 Reporter: Yi Liu Priority: Minor https://builds.apache.org/job/PreCommit-HDFS-Build/12736/testReport/ https://builds.apache.org/job/PreCommit-HADOOP-Build/7732/testReport/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HDFS-8740) Move DistributedFileSystem to hadoop-hdfs-client
[ https://issues.apache.org/jira/browse/HDFS-8740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Liu reopened HDFS-8740: -- Yes, we should work on this. This JIRA should be the last step, it should be done after moving all client side related things, such as DFSClient. > Move DistributedFileSystem to hadoop-hdfs-client > > > Key: HDFS-8740 > URL: https://issues.apache.org/jira/browse/HDFS-8740 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: build >Reporter: Yi Liu >Assignee: Mingliang Liu > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-9053) Support large directories efficiently using B-Tree
Yi Liu created HDFS-9053: Summary: Support large directories efficiently using B-Tree Key: HDFS-9053 URL: https://issues.apache.org/jira/browse/HDFS-9053 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Reporter: Yi Liu Assignee: Yi Liu This is a long standing issue, we were trying to improve this in the past. Currently we use an ArrayList for the children under a directory, and the children are ordered in the list, for insert/delete/search, the time complexity is O(log n), but insertion/deleting causes re-allocations and copies of big arrays, so the operations are costly. For example, if the children grow to 1M size, the ArrayList will resize to > 1M capacity, so need > 1M * 4bytes = 4M continuous heap memory, it easily causes full GC in HDFS cluster where namenode heap memory is already highly used. I recap the 3 main issues: # Insertion/deletion operations in large directories are expensive because re-allocations and copies of big arrays. # Dynamically allocate several MB continuous heap memory which will be long-lived can easily cause full GC problem. # Even most children are removed later, but the directory INode still occupies same size heap memory, since the ArrayList will never shrink. This JIRA is similar to HDFS-7174 created by [~kihwal], but use B-Tree to solve the problem suggested by [~shv]. So the target of this JIRA is to implement a low memory footprint B-Tree and use it to replace ArrayList. If the elements size is not large (less than the maximum degree of B-Tree node), the B-Tree only has one root node which contains an array for the elements. And if the size grows large enough, it will split automatically, and if elements are removed, then B-Tree nodes can merge automatically (see more: https://en.wikipedia.org/wiki/B-tree). It will solve the above 3 issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-9035) Remove the max size conf of xattr
Yi Liu created HDFS-9035: Summary: Remove the max size conf of xattr Key: HDFS-9035 URL: https://issues.apache.org/jira/browse/HDFS-9035 Project: Hadoop HDFS Issue Type: Improvement Reporter: Yi Liu Assignee: Yi Liu Priority: Minor As discussed in HDFS-8900, now we have the max size hard-limit, and we can remove the max size config. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-9032) Small improvement for DatanodeManager#sortLocatedBlocks
Yi Liu created HDFS-9032: Summary: Small improvement for DatanodeManager#sortLocatedBlocks Key: HDFS-9032 URL: https://issues.apache.org/jira/browse/HDFS-9032 Project: Hadoop HDFS Issue Type: Improvement Reporter: Yi Liu Assignee: Yi Liu Priority: Minor This is a minor improvement: In sortLocatedBlocks, (in most cases) if locations of located block don't contain decommissioned/stale datanode, no need to call Arrays.sort. Also we can make the comparator as the class variable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-8988) Use LightWeightHashSet instead of LightWeightLinkedSet in BlockManager#excessReplicateMap
Yi Liu created HDFS-8988:
Summary: Use LightWeightHashSet instead of LightWeightLinkedSet in
BlockManager#excessReplicateMap
Key: HDFS-8988
URL: https://issues.apache.org/jira/browse/HDFS-8988
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Yi Liu
Assignee: Yi Liu
Priority: Minor
{code}
public final MapString, LightWeightLinkedSetBlock excessReplicateMap = new
HashMap();
{code}
{{LightWeightLinkedSet}} extends {{LightWeightHashSet}} and keeps elements in
order, but it requires more memory for each entry (2 references, totally 8
bytes).
We don't need to keep excess replicated blocks in order here, so should use
{{LightWeightHashSet}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8946) Improve choosing datanode storage for block placement
Yi Liu created HDFS-8946:
Summary: Improve choosing datanode storage for block placement
Key: HDFS-8946
URL: https://issues.apache.org/jira/browse/HDFS-8946
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Reporter: Yi Liu
Assignee: Yi Liu
This JIRA is to:
*1.* Improve chooseing datanode storage for block placement:
In {{BlockPlacementPolicyDefault}} ({{chooseLocalStorage}}, {{chooseRandom}}),
we have following logic to choose datanode storage to place block.
For given storage type, we iterate storages of the datanode. But for datanode,
it only cares about the storage type. In the loop, we check according to
Storage type and return the first storage if the storages of the type on the
datanode fit in requirement. So we can remove the iteration of storages, and
just need to do once to find a good storage of given type, it's efficient if
the storages of the type on the datanode don't fit in requirement since we
don't need to loop all storages and do the same check.
Besides, no need to shuffle the storages, since we only need to check according
to the storage type on the datanode once.
This also improves the logic and make it more clear.
{code}
if (excludedNodes.add(localMachine) // was not in the excluded list
isGoodDatanode(localDatanode, maxNodesPerRack, false,
results, avoidStaleNodes)) {
for (IteratorMap.EntryStorageType, Integer iter = storageTypes
.entrySet().iterator(); iter.hasNext(); ) {
Map.EntryStorageType, Integer entry = iter.next();
for (DatanodeStorageInfo localStorage : DFSUtil.shuffle(
localDatanode.getStorageInfos())) {
StorageType type = entry.getKey();
if (addIfIsGoodTarget(localStorage, excludedNodes, blocksize,
results, type) = 0) {
int num = entry.getValue();
...
{code}
(current logic above)
*2.* Improve the logic and remove some duplicated code
for example, In {{chooseLocalStorage}}, {{chooseRandom}}, we add the node to
excludeNodes before the {{for}}, and we do it again if we find it's a good
target. {{numOfAvailableNodes -= newExcludedNodes}} is duplicated too.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HDFS-8912) Implement ShrinkableHashMap extends java HashMap and use properly
[
https://issues.apache.org/jira/browse/HDFS-8912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liu resolved HDFS-8912.
--
Resolution: Invalid
I forgot one thing: many variables/methods of java {{HashMap}} we need to touch
in {{ShrinkableHashMap}} are {{package-private}}, so we can't simply extend
{{HashMap}}. Implementing a new {{ShrinkableHashMap}} is a bit heavy, close it
as Invalid.
Implement ShrinkableHashMap extends java HashMap and use properly
-
Key: HDFS-8912
URL: https://issues.apache.org/jira/browse/HDFS-8912
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Yi Liu
Assignee: Yi Liu
Currently {{LightWeightHashSet}} and {{LightWeightLinkedSet}} are used in
hdfs, there are two advantages compared to java HashSet: one is the entry
requires fewer memory, another is it's shrinkable. In real cluster, hdfs is
a long running service, and {{set}} may become large at some time and may
become small after that, so shrinking the {{set}} when size hits the shrink
threshold is necessary, it can improve the NN memory.
Same situation for {{map}}, some HashMap used in BlockManager (e.g., the
hashmap in CorruptReplicasMap), it's better to be shrinkable.
I think it's worth to implement ShrinkableHashMap extends the java HashMap,
for quick glance, seems few code is needed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8912) Implement ShrinkableHashMap extends java HashMap and use properly
Yi Liu created HDFS-8912:
Summary: Implement ShrinkableHashMap extends java HashMap and use
properly
Key: HDFS-8912
URL: https://issues.apache.org/jira/browse/HDFS-8912
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Yi Liu
Assignee: Yi Liu
Currently {{LightWeightHashSet}} and {{LightWeightLinkedSet}} are used in hdfs,
there are two advantages compared to java HashSet: one is the entry requires
fewer memory, another is it's shrinkable. In real cluster, hdfs is a long
running service, and {{set}} may become very large at some time and may become
small after that, so shrinking the {{set}} when size hits the shrink threshold
is necessary, it can improve the NN memory.
Same situation for {{map}}, some HashMap used in BlockManager (e.g., the
hashmap in CorruptReplicasMap), it's better to be shrinkable.
I think it's worth to implement ShrinkableHashMap extends the java HashMap,
for quick glance, seems few code is needed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8900) Improve XAttr memory footprint.
Yi Liu created HDFS-8900:
Summary: Improve XAttr memory footprint.
Key: HDFS-8900
URL: https://issues.apache.org/jira/browse/HDFS-8900
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Reporter: Yi Liu
Assignee: Yi Liu
{code}
private final ImmutableListXAttr xAttrs;
{code}
Currently we use above in XAttrFeature, it's not efficient from memory point of
view, since {{ImmutableList}} and {{XAttr}} have object memory overhead, and
each object has memory alignment.
We can use a {{byte[]}} in XAttrFeature and do some compact in {{XAttr}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8884) Improve BlockPlacementPolicyDefault#chooseTarget to check candidate datanode first
Yi Liu created HDFS-8884:
Summary: Improve BlockPlacementPolicyDefault#chooseTarget to check
candidate datanode first
Key: HDFS-8884
URL: https://issues.apache.org/jira/browse/HDFS-8884
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Yi Liu
Assignee: Yi Liu
In current BlockPlacementPolicyDefault, when choosing datanode storage to place
block, we have following logic:
{code}
final DatanodeStorageInfo[] storages = DFSUtil.shuffle(
chosenNode.getStorageInfos());
int i = 0;
boolean search = true;
for (IteratorMap.EntryStorageType, Integer iter = storageTypes
.entrySet().iterator(); search iter.hasNext(); ) {
Map.EntryStorageType, Integer entry = iter.next();
for (i = 0; i storages.length; i++) {
StorageType type = entry.getKey();
final int newExcludedNodes = addIfIsGoodTarget(storages[i],
{code}
We will iterate (actually two {{for}}) all storages of the candidate datanode
even the datanode itself is not good (e.g. decommissioned, stale, too busy..),
since currently we do all the check in {{addIfIsGoodTarget}}.
We can fail-fast: check the datanode related conditions first, if the datanode
is not good, then no need to shuffle and iterate the storages. Then it's more
efficient.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8859) Improve DataNode (ReplicaMap) memory footprint to save about 45%
Yi Liu created HDFS-8859:
Summary: Improve DataNode (ReplicaMap) memory footprint to save
about 45%
Key: HDFS-8859
URL: https://issues.apache.org/jira/browse/HDFS-8859
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Reporter: Yi Liu
Assignee: Yi Liu
By using following approach we can save about *45%* memory footprint for each
block replica in DataNode memory (This JIRA only talks about *ReplicaMap* in
DataNode), the details are:
In ReplicaMap,
{code}
private final MapString, MapLong, ReplicaInfo map =
new HashMapString, MapLong, ReplicaInfo();
{code}
Currently we use a HashMap {{MapLong, ReplicaInfo}} to store the replicas in
memory. The key is block id of the block replica which is already included in
{{ReplicaInfo}}, so this memory can be saved. Also HashMap Entry has a object
overhead. We can implement a lightweight Set which is similar to
{{LightWeightGSet}}, but not a fixed size ({{LightWeightGSet}} uses fix size
for the entries array, usually it's a big value, an example is {{BlocksMap}},
this can avoid full gc since no need to resize), also we should be able to get
Element through key.
Following is comparison of memory footprint If we implement a lightweight set
as described:
We can save:
{noformat}
SIZE (bytes) ITEM
20The Key: Long (12 bytes object overhead + 8 bytes
long)
12HashMap Entry object overhead
4 reference to the key in Entry
4 reference to the value in Entry
4 hash in Entry
{noformat}
Total: -44 bytes
We need to add:
{noformat}
SIZE (bytes) ITEM
4 a reference to next element in ReplicaInfo
{noformat}
Total: +4 bytes
So totally we can save 40bytes for each block replica
And currently one finalized replica needs around 46 bytes (notice: we ignore
memory alignment here).
We can save 1 - (4 + 46) / (44 + 46) = *45%* memory for each block replica in
DataNode.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8862) Improve BlockManager#excessReplicateMap
Yi Liu created HDFS-8862:
Summary: Improve BlockManager#excessReplicateMap
Key: HDFS-8862
URL: https://issues.apache.org/jira/browse/HDFS-8862
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Reporter: Yi Liu
Assignee: Yi Liu
Per [~cmccabe]'s comments in HDFS-8792, this JIRA is to discuss improving
{{BlockManager#excessReplicateMap}}.
That's right HashMap don't ever shrink when elements are removed, but TreeMap
entry needs to store more (memory) references (left, right, parent) than
HashMap entry (only one reference next), even when there is element removing
and cause some entry empty, the empty HashMap entry is just a {{null}}
reference (4 bytes), so they are close at this point. On the other hand, the
key of {{excessReplicateMap}} is datanode uuid, so the entries number is almost
fixed, so HashMap memory is good than TreeMap memory in this case. I think
the most important is the search/insert/remove performance, HashMap is
absolutely better than TreeMap. Because we don't need to sort, we should use
HashMap instead of TreeMap
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8795) Improve InvalidateBlocks
Yi Liu created HDFS-8795:
Summary: Improve InvalidateBlocks
Key: HDFS-8795
URL: https://issues.apache.org/jira/browse/HDFS-8795
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Yi Liu
Assignee: Yi Liu
Currently we use {{TreeMap}} for {{node2blocks}}, actually there are only two
place may need sorted: {{dump}}, {{getDatanodes}}. But {{getDatanodes}} is
called by {{computeInvalidateWork}}, and we do a shuffle there, so the sort is
unnecssary. For {{dump}}, certainly we can modify few log.
So we can use {{HashMap}}.
From memory and performance view, {{HashMap}} is better than {{TreeMap}}, a
simliar optimization HDFS-7433.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8794) Improve CorruptReplicasMap
Yi Liu created HDFS-8794:
Summary: Improve CorruptReplicasMap
Key: HDFS-8794
URL: https://issues.apache.org/jira/browse/HDFS-8794
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Yi Liu
Assignee: Yi Liu
Currently we use {{TreeMap}} for {{corruptReplicasMap}}, actually the only need
sorted place is {{getCorruptReplicaBlockIds}} which is used by test.
So we can use {{HashMap}}.
From memory and performance view, {{HashMap}} is better than {{TreeMap}}, a
simliar optimization HDFS-7433. Of course we need to make few change to
{{getCorruptReplicaBlockIds}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8792) Use LightWeightLinkedSet for BlockManager#postponedMisreplicatedBlocks
Yi Liu created HDFS-8792:
Summary: Use LightWeightLinkedSet for
BlockManager#postponedMisreplicatedBlocks
Key: HDFS-8792
URL: https://issues.apache.org/jira/browse/HDFS-8792
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Yi Liu
Assignee: Yi Liu
{{LightWeightLinkedSet}} requires fewer memory than java hashset.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8793) Improve BlockManager memory and performance
Yi Liu created HDFS-8793:
Summary: Improve BlockManager memory and performance
Key: HDFS-8793
URL: https://issues.apache.org/jira/browse/HDFS-8793
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Yi Liu
Assignee: Yi Liu
Umbrella JIRA to improve {{BlockManager}} memory and performance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8733) Keep server related definition in hdfs.proto on server side
Yi Liu created HDFS-8733: Summary: Keep server related definition in hdfs.proto on server side Key: HDFS-8733 URL: https://issues.apache.org/jira/browse/HDFS-8733 Project: Hadoop HDFS Issue Type: Sub-task Components: build Reporter: Yi Liu -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HDFS-8740) Move DistributedFileSystem to hadoop-hdfs-client
[ https://issues.apache.org/jira/browse/HDFS-8740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Liu resolved HDFS-8740. -- Resolution: Invalid Assignee: (was: Yi Liu) Move DistributedFileSystem to hadoop-hdfs-client Key: HDFS-8740 URL: https://issues.apache.org/jira/browse/HDFS-8740 Project: Hadoop HDFS Issue Type: Sub-task Components: build Reporter: Yi Liu -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-8740) Move DistributedFileSystem to hadoop-hdfs-client
Yi Liu created HDFS-8740: Summary: Move DistributedFileSystem to hadoop-hdfs-client Key: HDFS-8740 URL: https://issues.apache.org/jira/browse/HDFS-8740 Project: Hadoop HDFS Issue Type: Sub-task Components: build Reporter: Yi Liu Assignee: Yi Liu -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-8739) Move DFSClient to hadoop-hdfs-client
Yi Liu created HDFS-8739: Summary: Move DFSClient to hadoop-hdfs-client Key: HDFS-8739 URL: https://issues.apache.org/jira/browse/HDFS-8739 Project: Hadoop HDFS Issue Type: Sub-task Components: build Reporter: Yi Liu Assignee: Yi Liu -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HDFS-8684) Erasure Coding: fix some block number calculation for striped block
[
https://issues.apache.org/jira/browse/HDFS-8684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liu resolved HDFS-8684.
--
Resolution: Fixed
Hadoop Flags: Reviewed
Thanks Vinay and Walter for review, committed to the branch.
Erasure Coding: fix some block number calculation for striped block
---
Key: HDFS-8684
URL: https://issues.apache.org/jira/browse/HDFS-8684
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Yi Liu
Assignee: Yi Liu
Attachments: HDFS-8684-HDFS-7285.001.patch,
HDFS-8684-HDFS-7285.002.patch
in INodeFile#computeFileSize, the file size calucation for underconstruction
striped block is incorrect.
in BlockManager#chooseExcessReplicasStriped, the {{if (nonExcess.size() =
groupSize) {}} is incorrect.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8684) Erasure Coding: fix some block number calculation for striped block
Yi Liu created HDFS-8684: Summary: Erasure Coding: fix some block number calculation for striped block Key: HDFS-8684 URL: https://issues.apache.org/jira/browse/HDFS-8684 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Yi Liu Assignee: Yi Liu in INodeFile#computeFileSize, the file size calucation for underconstruction block is incorrect. in BlockManager#chooseExcessReplicasStriped, the groupSize calculation is incorrect. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-8667) Erasure Coding: refine striped block recovery logic and reuse in client and Datanode
Yi Liu created HDFS-8667: Summary: Erasure Coding: refine striped block recovery logic and reuse in client and Datanode Key: HDFS-8667 URL: https://issues.apache.org/jira/browse/HDFS-8667 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Yi Liu Assignee: Yi Liu Currently, there is some common logic of striped block recovery in client and datanode. This JIRA is to refine and reuse the common part. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-8668) Erasure Coding: revisit buffer used for encoding and decoding.
Yi Liu created HDFS-8668: Summary: Erasure Coding: revisit buffer used for encoding and decoding. Key: HDFS-8668 URL: https://issues.apache.org/jira/browse/HDFS-8668 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Yi Liu Assignee: Yi Liu For encoding and decoding buffers, currently some places use java heap ByteBuffer, some use direct byteBUffer, and some use java byte array. If the coder implementation is native, we should use direct ByteBuffer. This jira is to revisit all encoding/decoding buffers and improve them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HDFS-8559) Erasure Coding: fix non-protobuf fsimage for striped blocks
[
https://issues.apache.org/jira/browse/HDFS-8559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liu resolved HDFS-8559.
--
Resolution: Fixed
Hadoop Flags: Reviewed
Committed to branch. Thanks Jing for the contribution!
Erasure Coding: fix non-protobuf fsimage for striped blocks
---
Key: HDFS-8559
URL: https://issues.apache.org/jira/browse/HDFS-8559
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Jing Zhao
Assignee: Jing Zhao
Priority: Minor
Attachments: HDFS-8559.000.patch
For a legacy fsimage we currently always record its layoutversion as -51 so
that we can make sure it cannot be processed by a protobuf-based image
parser. Thus the following code in the parser always returns false and the
parse will fail.
{code}
NameNodeLayoutVersion.supports(
NameNodeLayoutVersion.Feature.ERASURE_CODING, imgVersion)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8587) Erasure Coding: fix the copy constructor of BlockInfoStriped and BlockInfoContiguous
Yi Liu created HDFS-8587:
Summary: Erasure Coding: fix the copy constructor of
BlockInfoStriped and BlockInfoContiguous
Key: HDFS-8587
URL: https://issues.apache.org/jira/browse/HDFS-8587
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Yi Liu
Assignee: Yi Liu
{code}
BlockInfoStriped(BlockInfoStriped b) {
this(b, b.getSchema());
this.setBlockCollection(b.getBlockCollection());
}
{code}
{code}
protected BlockInfoContiguous(BlockInfoContiguous from) {
this(from, from.getBlockCollection().getPreferredBlockReplication());
this.triplets = new Object[from.triplets.length];
this.setBlockCollection(from.getBlockCollection());
}
{code}
We should define a copy constructor in the {{BlockInfo}}, and call it from
these two {{subclass}}.I also notice a NullPointerException test failure of
{{TestBlockInfo.testCopyConstructor}} in latest branch which is related to this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8585) Remove dataBlockNum and parityBlockNum from StripedBlockProto
Yi Liu created HDFS-8585:
Summary: Remove dataBlockNum and parityBlockNum from
StripedBlockProto
Key: HDFS-8585
URL: https://issues.apache.org/jira/browse/HDFS-8585
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Yi Liu
Assignee: Yi Liu
Since in HDFS-8427, we remove {{dataBlockNum}} and {{parityBlockNum}}, and get
it from {{ECSchema}}, so we also need it from {{StripedBlockProto}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8460) Erasure Coding: stateful result doesn't match data occasionally
Yi Liu created HDFS-8460:
Summary: Erasure Coding: stateful result doesn't match data
occasionally
Key: HDFS-8460
URL: https://issues.apache.org/jira/browse/HDFS-8460
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Yi Liu
I found this issue in TestDFSStripedInputStream, {{testStatefulRead}} failed
occasionally.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8428) Erasure Coding: Fix the NullPointerException when deleting file
Yi Liu created HDFS-8428:
Summary: Erasure Coding: Fix the NullPointerException when
deleting file
Key: HDFS-8428
URL: https://issues.apache.org/jira/browse/HDFS-8428
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Yi Liu
Assignee: Yi Liu
In HDFS, when removing some file, NN will also remove all its blocks from
{{BlocksMap}}, and send {{DNA_INVALIDATE}} (invalidate blocks) commands to
datanodes. After datanodes successfully delete the block replicas, will report
{{DELETED_BLOCK}} to NameNode.
snippet code logic in {{BlockManager#processIncrementalBlockReport}} is as
following
{code}
case DELETED_BLOCK:
removeStoredBlock(storageInfo, getStoredBlock(rdbi.getBlock()), node);
...
{code}
{code}
private void removeStoredBlock(DatanodeStorageInfo storageInfo, Block block,
DatanodeDescriptor node) {
if (shouldPostponeBlocksFromFuture
namesystem.isGenStampInFuture(block)) {
queueReportedBlock(storageInfo, block, null,
QUEUE_REASON_FUTURE_GENSTAMP);
return;
}
removeStoredBlock(getStoredBlock(block), node);
}
{code}
In EC branch, we add {{getStoredBlock}}. There is {{NullPointerException}} when
handling {{DELETED_BLOCK}} of incrementalBlockReport from DataNode after delete
a file, since the block is already removed, we need to check.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8418) Fix the isNeededReplication calculation for Striped block in NN
Yi Liu created HDFS-8418:
Summary: Fix the isNeededReplication calculation for Striped block
in NN
Key: HDFS-8418
URL: https://issues.apache.org/jira/browse/HDFS-8418
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Yi Liu
Assignee: Yi Liu
Priority: Critical
Currently when calculating {{isNeededReplication}} for striped block, we use
BlockCollection#getPreferredBlockReplication to get expected replica number for
striped block. See an example:
{code}
public void checkReplication(BlockCollection bc) {
final short expected = bc.getPreferredBlockReplication();
for (BlockInfo block : bc.getBlocks()) {
final NumberReplicas n = countNodes(block);
if (isNeededReplication(block, expected, n.liveReplicas())) {
neededReplications.add(block, n.liveReplicas(),
n.decommissionedAndDecommissioning(), expected);
} else if (n.liveReplicas() expected) {
processOverReplicatedBlock(block, expected, null, null);
}
}
}
{code}
But actually it's not correct, for example, if the length of striped file is
less than a cell, then the expected replica of the block should be {{1 +
parityBlkNum}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8398) Erasure Coding: Correctly calculate last striped block length if it's under construction.
Yi Liu created HDFS-8398: Summary: Erasure Coding: Correctly calculate last striped block length if it's under construction. Key: HDFS-8398 URL: https://issues.apache.org/jira/browse/HDFS-8398 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Yi Liu Assignee: Yi Liu Currently, for continuous block, if it's under construction, we need to read the block replica length from one of datanode and use it as last block length. For striped block, we need to read the length of all internal data blocks of the striped group, then add them correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-8376) Erasure Coding: Update last cellsize calculation according to whether the erasure codec is liner
Yi Liu created HDFS-8376:
Summary: Erasure Coding: Update last cellsize calculation
according to whether the erasure codec is liner
Key: HDFS-8376
URL: https://issues.apache.org/jira/browse/HDFS-8376
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Yi Liu
Assignee: Yi Liu
Current calculation for last cell size is as following. For parity cell, the
last cell size is the same as the first data cell. But as discussed in
HDFS-8347 and on-line meeting, some erasure codec is not liner, then the last
cellsize for parity block is the codec chunk size.
{code}
private static int lastCellSize(int size, int cellSize, int numDataBlocks,
int i) {
if (i numDataBlocks) {
// parity block size (i.e. i = numDataBlocks) is the same as
// the first data block size (i.e. i = 0).
size -= i*cellSize;
if (size 0) {
size = 0;
}
}
return size cellSize? cellSize: size;
}
{code}
This JIRA also removes some duplicate code
{{StripedBlockUtil#constructStripedBlock}} and
{{StripedBlockUtil#getStripedBlockLength}} are duplicated with some other
methods, we can clean up.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HDFS-8363) Erasure Coding: DFSStripedInputStream#seekToNewSource
[
https://issues.apache.org/jira/browse/HDFS-8363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liu resolved HDFS-8363.
--
Resolution: Fixed
Hadoop Flags: Reviewed
Thanks Uma for review, commit to branch.
Erasure Coding: DFSStripedInputStream#seekToNewSource
-
Key: HDFS-8363
URL: https://issues.apache.org/jira/browse/HDFS-8363
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Yi Liu
Assignee: Yi Liu
Attachments: HDFS-8363.001.patch
Ideally, for striped block, an internal block is only on one datanode, so
{{DFSStripedInputStream#seekToNewSource}} should return *false*, we should
overwrite {{DFSInputStream#seekToNewSource}} for striped file.
This patch also remove the hardcode for {{groupSize}}
{code}
private final short groupSize = HdfsConstants.NUM_DATA_BLOCKS;
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8363) Erasure Coding: DFSStripedInputStream#seekToNewSource
Yi Liu created HDFS-8363:
Summary: Erasure Coding: DFSStripedInputStream#seekToNewSource
Key: HDFS-8363
URL: https://issues.apache.org/jira/browse/HDFS-8363
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Yi Liu
Assignee: Yi Liu
Ideally, for striped block, an internal block is only on one datanode, so
{{DFSStripedInputStream#seekToNewSource}} should return *false*, we should
overwrite {{DFSInputStream#seekToNewSource}} for striped file.
This patch also remove the hardcode for {{groupSize}}
{code}
private final short groupSize = HdfsConstants.NUM_DATA_BLOCKS;
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HDFS-8329) Erasure coding: Rename Striped block recovery to reconstruction to eliminate confusion.
[ https://issues.apache.org/jira/browse/HDFS-8329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Liu resolved HDFS-8329. -- Resolution: Duplicate Erasure coding: Rename Striped block recovery to reconstruction to eliminate confusion. --- Key: HDFS-8329 URL: https://issues.apache.org/jira/browse/HDFS-8329 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Yi Liu Assignee: Yi Liu Both in NN and DN, we use striped block recovery and sometime use reconstruction. The striped block recovery make people confused with block recovery, we should use striped block reconstruction to eliminate confusion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-8328) Follow-on to update decode for DataNode striped blocks reconstruction
Yi Liu created HDFS-8328: Summary: Follow-on to update decode for DataNode striped blocks reconstruction Key: HDFS-8328 URL: https://issues.apache.org/jira/browse/HDFS-8328 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Yi Liu Assignee: Yi Liu Current the decode for DataNode striped blocks reconstruction is a workaround, we need to update it after the decode fix in HADOOP-11847. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-8329) Erasure coding: Rename Striped block recovery to reconstruction to eliminate confusion.
Yi Liu created HDFS-8329: Summary: Erasure coding: Rename Striped block recovery to reconstruction to eliminate confusion. Key: HDFS-8329 URL: https://issues.apache.org/jira/browse/HDFS-8329 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Yi Liu Both in NN and DN, we use striped block recovery and sometime use reconstruction. The striped block recovery make people confused with block recovery, we should unify them to use striped block reconstruction. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-8313) Erasure Coding: DFSStripedOutputStream#close throws NullPointerException exception in some cases
Yi Liu created HDFS-8313:
Summary: Erasure Coding: DFSStripedOutputStream#close throws
NullPointerException exception in some cases
Key: HDFS-8313
URL: https://issues.apache.org/jira/browse/HDFS-8313
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Yi Liu
Assignee: Yi Liu
{code}
java.io.IOException: java.lang.NullPointerException
at
org.apache.hadoop.hdfs.DataStreamer$LastException.check(DataStreamer.java:193)
at
org.apache.hadoop.hdfs.DFSStripedOutputStream.closeImpl(DFSStripedOutputStream.java:422)
{code}
DFSStripedOutputStream#close throws NullPointerException exception in some
cases
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HDFS-8275) Erasure Coding: Implement batched listing of enrasure coding zones
[
https://issues.apache.org/jira/browse/HDFS-8275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liu resolved HDFS-8275.
--
Resolution: Duplicate
Hi Rakesh, this JIRA is duplicated with HDFS-8087.
Erasure Coding: Implement batched listing of enrasure coding zones
--
Key: HDFS-8275
URL: https://issues.apache.org/jira/browse/HDFS-8275
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Rakesh R
Assignee: Rakesh R
The idea of this jira is to provide batch API in {{DistributedFileSystem}} to
list the {{ECZoneInfo}}.
API signature:-
{code}
/**
* List all ErasureCoding zones. Incrementally fetches results from the
server.
*/
public RemoteIteratorECZoneInfo listErasureCodingZones() throws
IOException {
return dfs.listErasureCodingZones();
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8203) Erasure Coding: Seek and other Ops in DFSStripedInputStream.
Yi Liu created HDFS-8203:
Summary: Erasure Coding: Seek and other Ops in
DFSStripedInputStream.
Key: HDFS-8203
URL: https://issues.apache.org/jira/browse/HDFS-8203
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Yi Liu
Assignee: Yi Liu
In HDFS-7782 and HDFS-8033, we handle pread and stateful read for
{{DFSStripedInputStream}}, we also need handle other operations, such as
{{seek}}, zerocopy read ...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HDFS-8058) Erasure coding: use BlockInfo[] for both striped and contiguous blocks in INodeFile
[
https://issues.apache.org/jira/browse/HDFS-8058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liu resolved HDFS-8058.
--
Resolution: Invalid
Erasure coding: use BlockInfo[] for both striped and contiguous blocks in
INodeFile
---
Key: HDFS-8058
URL: https://issues.apache.org/jira/browse/HDFS-8058
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7285
Reporter: Yi Liu
Assignee: Yi Liu
Attachments: HDFS-8058.001.patch, HDFS-8058.002.patch
This JIRA is to use {{BlockInfo[] blocks}} for both striped and contiguous
blocks in INodeFile.
Currently {{FileWithStripedBlocksFeature}} keeps separate list for striped
blocks, and the methods there duplicate with those in INodeFile, and current
code need to judge {{isStriped}} then do different things. Also if file is
striped, the {{blocks}} in INodeFile occupy a reference memory space.
These are not necessary, and we can use the same {{blocks}} to make code more
clear.
I keep {{FileWithStripedBlocksFeature}} as empty for follow use: I will file
a new JIRA to move {{dataBlockNum}} and {{parityBlockNum}} from
*BlockInfoStriped* to INodeFile, since ideally they are the same for all
striped blocks in a file, and store them in block will waste NN memory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8087) Erasure Coding: Add more EC zone management APIs (get/list EC zone(s))
Yi Liu created HDFS-8087: Summary: Erasure Coding: Add more EC zone management APIs (get/list EC zone(s)) Key: HDFS-8087 URL: https://issues.apache.org/jira/browse/HDFS-8087 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Yi Liu Assignee: Yi Liu -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HDFS-8066) Erasure coding: Decommission handle for EC blocks.
[ https://issues.apache.org/jira/browse/HDFS-8066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Liu resolved HDFS-8066. -- Resolution: Implemented Take a look at the latest branch, this has been implemented in branch, so close it as implemented. Erasure coding: Decommission handle for EC blocks. -- Key: HDFS-8066 URL: https://issues.apache.org/jira/browse/HDFS-8066 Project: Hadoop HDFS Issue Type: Sub-task Affects Versions: HDFS-7285 Reporter: Yi Liu Assignee: Yi Liu This JIRA is to handle decommission for EC blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-8065) Support truncate at striped group boundary.
Yi Liu created HDFS-8065: Summary: Support truncate at striped group boundary. Key: HDFS-8065 URL: https://issues.apache.org/jira/browse/HDFS-8065 Project: Hadoop HDFS Issue Type: Sub-task Affects Versions: HDFS-7285 Reporter: Yi Liu Assignee: Yi Liu We can support truncate at striped group boundary firstly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-8066) Erasure coding: Decommission handle for EC blocks.
Yi Liu created HDFS-8066: Summary: Erasure coding: Decommission handle for EC blocks. Key: HDFS-8066 URL: https://issues.apache.org/jira/browse/HDFS-8066 Project: Hadoop HDFS Issue Type: Sub-task Affects Versions: HDFS-7285 Reporter: Yi Liu Assignee: Yi Liu This JIRA is to handle decommission for EC blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-8064) Erasure coding: DataNode support for block recovery of striped block groups
Yi Liu created HDFS-8064: Summary: Erasure coding: DataNode support for block recovery of striped block groups Key: HDFS-8064 URL: https://issues.apache.org/jira/browse/HDFS-8064 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode Affects Versions: HDFS-7285 Reporter: Yi Liu Assignee: Yi Liu This JIRA is for block recovery of striped block groups on DataNodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-8059) Move dataBlockNum and parityBlockNum from BlockInfoStriped to INodeFile
Yi Liu created HDFS-8059:
Summary: Move dataBlockNum and parityBlockNum from
BlockInfoStriped to INodeFile
Key: HDFS-8059
URL: https://issues.apache.org/jira/browse/HDFS-8059
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7285
Reporter: Yi Liu
Assignee: Yi Liu
Move {{dataBlockNum}} and {{parityBlockNum}} from BlockInfoStriped to
INodeFile, and store them in {{FileWithStripedBlocksFeature}}.
Ideally these two nums are the same for all striped blocks in a file, and store
them in BlockInfoStriped will waste NN memory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8058) Erasure coding: use BlockInfo[] for both striped and contiguous blocks in INodeFile
Yi Liu created HDFS-8058:
Summary: Erasure coding: use BlockInfo[] for both striped and
contiguous blocks in INodeFile
Key: HDFS-8058
URL: https://issues.apache.org/jira/browse/HDFS-8058
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7285
Reporter: Yi Liu
Assignee: Yi Liu
This JIRA is to use {{BlockInfo[] blocks}} for both striped and contiguous
blocks in INodeFile.
Currently {{FileWithStripedBlocksFeature}} keeps separate list for striped
blocks, and the methods there duplicate with those in INodeFile, and current
code need to judge {{isStriped}} then do different things. Also if file is
striped, the {{blocks}} in INodeFile occupy a reference memory space.
These are not necessary, and we can use the same {{blocks}} to make code more
clear.
I keep {{FileWithStripedBlocksFeature}} as empty for follow use: I will file a
new JIRA to move {{dataBlockNum}} and {{parityBlockNum}} to INodeFile, since
ideally they are the same for all striped blocks in a file, and store them in
block will waste NN memory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-7962) Remove duplicated log in BlockManager
Yi Liu created HDFS-7962:
Summary: Remove duplicated log in BlockManager
Key: HDFS-7962
URL: https://issues.apache.org/jira/browse/HDFS-7962
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Yi Liu
Assignee: Yi Liu
Priority: Minor
There are few duplicated log in {{BlockManager}}.
Also do few refinement of log.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HDFS-7911) Buffer Overflow when running HBase on HDFS Encryption Zone
[
https://issues.apache.org/jira/browse/HDFS-7911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liu resolved HDFS-7911.
--
Resolution: Duplicate
Buffer Overflow when running HBase on HDFS Encryption Zone
--
Key: HDFS-7911
URL: https://issues.apache.org/jira/browse/HDFS-7911
Project: Hadoop HDFS
Issue Type: Bug
Components: encryption
Affects Versions: 2.6.0
Reporter: Xiaoyu Yao
Assignee: Yi Liu
Priority: Blocker
Create an HDFS EZ for HBase under /apps/hbase with some basic testing passed,
including creating tables, listing, adding a few rows, scanning them, etc.
However, when doing bulk load 100's k rows. After 10 minutes or so, we get
the following error on the Region Server that owns the table.
{code}
2015-03-02 10:25:47,784 FATAL [regionserver60020-WAL.AsyncSyncer0]
wal.FSHLog: Error while AsyncSyncer sync, request close of hlog
java.io.IOException: java.nio.BufferOverflowException
at
org.apache.hadoop.crypto.JceAesCtrCryptoCodec$JceAesCtrCipher.process(JceAesCtrCryptoCodec.java:156)
at
org.apache.hadoop.crypto.JceAesCtrCryptoCodec$JceAesCtrCipher.encrypt(JceAesCtrCryptoCodec.java:127)
at
org.apache.hadoop.crypto.CryptoOutputStream.encrypt(CryptoOutputStream.java:162)
at
org.apache.hadoop.crypto.CryptoOutputStream.flush(CryptoOutputStream.java:232)
at
org.apache.hadoop.crypto.CryptoOutputStream.hflush(CryptoOutputStream.java:267)
at
org.apache.hadoop.crypto.CryptoOutputStream.sync(CryptoOutputStream.java:262)
at org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:123)
at
org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.sync(ProtobufLogWriter.java:165)
at
org.apache.hadoop.hbase.regionserver.wal.FSHLog$AsyncSyncer.run(FSHLog.java:1241)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.nio.BufferOverflowException
at java.nio.DirectByteBuffer.put(DirectByteBuffer.java:357)
at javax.crypto.CipherSpi.bufferCrypt(CipherSpi.java:823)
at javax.crypto.CipherSpi.engineUpdate(CipherSpi.java:546)
at javax.crypto.Cipher.update(Cipher.java:1760)
at
org.apache.hadoop.crypto.JceAesCtrCryptoCodec$JceAesCtrCipher.process(JceAesCtrCryptoCodec.java:145)
... 9 more
{code}
It looks like the HBase WAL (Write Ahead Log) use case is broken on the
CryptoOutputStream(). The use case has one flusher thread that keeps calling
the hflush() on WAL file while other roller threads are trying to write
concurrently to that same file handle.
As the class comments mentioned. *CryptoOutputStream encrypts data. It is
not thread-safe.* I check the code and it seems the buffer overflow is
related to the race between the CryptoOutputStream#write() and
CryptoOutputStream#flush() as both can call CryptoOutputStream#encrypt(). The
inBuffer/outBuffer of the CryptoOutputStream is not thread safe. They can be
changed during encrypt for flush() when write() is coming from other threads.
I have validated this with multi-threaded unit tests that mimic the HBase WAL
use case. For file not under encryption zone (*DFSOutputStream*),
multi-threaded flusher/writer works fine. For file under encryption zone
(*CryptoOutputStream*), multi-threaded flusher/writer randomly fails with
Buffer Overflow/Underflow.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-7902) Expose truncate API for libwebhdfs
Yi Liu created HDFS-7902: Summary: Expose truncate API for libwebhdfs Key: HDFS-7902 URL: https://issues.apache.org/jira/browse/HDFS-7902 Project: Hadoop HDFS Issue Type: Improvement Components: native, webhdfs Affects Versions: 2.7.0 Reporter: Yi Liu Assignee: Yi Liu As Colin suggested in HDFS-7838, we will add truncate support for libwebhdfs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7886) TestFileTruncate#testTruncateWithDataNodesRestart runs timeout sometimes
Yi Liu created HDFS-7886: Summary: TestFileTruncate#testTruncateWithDataNodesRestart runs timeout sometimes Key: HDFS-7886 URL: https://issues.apache.org/jira/browse/HDFS-7886 Project: Hadoop HDFS Issue Type: Bug Components: test Affects Versions: 2.7.0 Reporter: Yi Liu Priority: Minor https://builds.apache.org/job/PreCommit-HDFS-Build/9730//testReport/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7761) cleanup unnecssary code logic in LocatedBlock
Yi Liu created HDFS-7761:
Summary: cleanup unnecssary code logic in LocatedBlock
Key: HDFS-7761
URL: https://issues.apache.org/jira/browse/HDFS-7761
Project: Hadoop HDFS
Issue Type: Improvement
Affects Versions: 2.7.0
Reporter: Yi Liu
Assignee: Yi Liu
Priority: Minor
The usage of following two variables is unnecessary. We can remove them to make
code a bit brief.
{quote}
private final boolean hasStorageIDs;
private final boolean hasStorageTypes;
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-7760) Further update user doc for truncate
Yi Liu created HDFS-7760: Summary: Further update user doc for truncate Key: HDFS-7760 URL: https://issues.apache.org/jira/browse/HDFS-7760 Project: Hadoop HDFS Issue Type: Sub-task Components: documentation Affects Versions: 2.7.0 Reporter: Yi Liu Assignee: Yi Liu Priority: Minor The JIRA is to further update user doc for truncate, for example, WebHDFS and so on. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7741) Remove unnecessary synchronized in FSDataInputStream and HdfsDataInputStream
Yi Liu created HDFS-7741:
Summary: Remove unnecessary synchronized in FSDataInputStream and
HdfsDataInputStream
Key: HDFS-7741
URL: https://issues.apache.org/jira/browse/HDFS-7741
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Yi Liu
Assignee: Yi Liu
Priority: Minor
The {{synchronized}} for {{HdfsDataInputStream#getAllBlocks}} and
{{FSDataInputStream#seek}} are unnecessary.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-7677) DistributedFileSystem#truncate should resolve symlinks
Yi Liu created HDFS-7677:
Summary: DistributedFileSystem#truncate should resolve symlinks
Key: HDFS-7677
URL: https://issues.apache.org/jira/browse/HDFS-7677
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Yi Liu
Assignee: Yi Liu
We should resolve the symlinks in DistributedFileSystem#truncate as we do for
{{create}}, {{open}}, {{append}} and so on, I don't see any reason not support
it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-7656) Expose truncate API for HDFS httpfs
Yi Liu created HDFS-7656: Summary: Expose truncate API for HDFS httpfs Key: HDFS-7656 URL: https://issues.apache.org/jira/browse/HDFS-7656 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Yi Liu Assignee: Yi Liu This JIRA is to expose truncate API for Web HDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7655) Expose truncate API for Web HDFS
Yi Liu created HDFS-7655: Summary: Expose truncate API for Web HDFS Key: HDFS-7655 URL: https://issues.apache.org/jira/browse/HDFS-7655 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Yi Liu Assignee: Yi Liu This JIRA is to expose truncate API for Web HDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7641) Update archival storage user doc for list/set/get block storage policies
Yi Liu created HDFS-7641: Summary: Update archival storage user doc for list/set/get block storage policies Key: HDFS-7641 URL: https://issues.apache.org/jira/browse/HDFS-7641 Project: Hadoop HDFS Issue Type: Bug Components: documentation Affects Versions: 2.7.0 Reporter: Yi Liu Assignee: Yi Liu Priority: Minor After HDFS-7323, the list/set/get block storage policies commands are different, we should update the corresponding user doc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7637) Fix the check condition for reserved path
Yi Liu created HDFS-7637:
Summary: Fix the check condition for reserved path
Key: HDFS-7637
URL: https://issues.apache.org/jira/browse/HDFS-7637
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Yi Liu
Assignee: Yi Liu
Priority: Minor
Currently the {{.reserved}} patch check function is:
{code}
public static boolean isReservedName(String src) {
return src.startsWith(DOT_RESERVED_PATH_PREFIX);
}
{code}
And {{DOT_RESERVED_PATH_PREFIX}} is {{/.reserved}}, it should be
{{/.reserved/}}, other some directory may prefix with _/.reserved_.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-7634) Lazy persist (memory) file should not support truncate currently
Yi Liu created HDFS-7634:
Summary: Lazy persist (memory) file should not support truncate
currently
Key: HDFS-7634
URL: https://issues.apache.org/jira/browse/HDFS-7634
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Yi Liu
Assignee: Yi Liu
Similar with {{append}}, lazy persist (memory) file should not support truncate
currently. Quote the reason from HDFS-6581 design doc:
{quote}
Appends to files created with the LAZY_PERSISTflag will not be allowed in the
initial implementation to avoid the complexity of keeping inmemory and ondisk
replicas in sync on a given DataNode.
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-7623) Add htrace configuration properties to core-default.xml and update user doc about how to enable htrace
Yi Liu created HDFS-7623: Summary: Add htrace configuration properties to core-default.xml and update user doc about how to enable htrace Key: HDFS-7623 URL: https://issues.apache.org/jira/browse/HDFS-7623 Project: Hadoop HDFS Issue Type: Improvement Reporter: Yi Liu Assignee: Yi Liu This JIRA does following things: *1.* Add htrace configuration properties to core-default.xml *2.* update user doc about how to enable htrace. *3.* Few fix in user doc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HDFS-7631) Deep learn about hadoop
[ https://issues.apache.org/jira/browse/HDFS-7631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Liu resolved HDFS-7631. -- Resolution: Won't Fix Instead of create a JIRA, you can subscribe the user list and ask this question. Guide: http://hadoop.apache.org/mailing_lists.html Deep learn about hadoop --- Key: HDFS-7631 URL: https://issues.apache.org/jira/browse/HDFS-7631 Project: Hadoop HDFS Issue Type: Wish Reporter: frank Priority: Trivial I want to learn more about hadoop code. If there are any books that can help me. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7600) Refine hdfs admin classes to reuse common code
Yi Liu created HDFS-7600:
Summary: Refine hdfs admin classes to reuse common code
Key: HDFS-7600
URL: https://issues.apache.org/jira/browse/HDFS-7600
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Yi Liu
Assignee: Jing Zhao
As the review comment in HDFS-7323.
In {{StoragePolicyAdmin}} and other class under
{{org.apache.hadoop.hdfs.tools}}, such as {{CacheAdmin}}, {{CryptoAdmin}}.
There are too many common methods ({{getDFS}}, {{prettifyException}},
{{getOptionDescriptionListing}} ...) and also inner class ({{HelpCommand}}),
they are the same, it would be great if we can refine and shift them into a
common place.
This makes the code clean and can be re-used if we add new hdfs admin class in
future.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-7431) log message for InvalidMagicNumberException may be incorrect
Yi Liu created HDFS-7431:
Summary: log message for InvalidMagicNumberException may be
incorrect
Key: HDFS-7431
URL: https://issues.apache.org/jira/browse/HDFS-7431
Project: Hadoop HDFS
Issue Type: Bug
Components: security
Reporter: Yi Liu
Assignee: Yi Liu
For security mode, HDFS now supports that Datanodes don't require root or jsvc
if {{dfs.data.transfer.protection}} is configured.
Log message for {{InvalidMagicNumberException}}, we miss one case:
when the datanodes run on unprivileged port and
{{dfs.data.transfer.protection}} is configured to {{authentication}} but
{{dfs.encrypt.data.transfer}} is not configured. SASL handshake is required.
But a low version dfs client is used, then {{InvalidMagicNumberException}} is
thrown with log:
{quote}
Failed to read expected encryption handshake from client at Perhaps the
client is running an older version of Hadoop which does not support encryption
{quote}
Recently I run HDFS built on trunk and security is enabled, but the client is
2.5.1 version. Then I got the above log message, but actually I have not
configured encryption.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-7271) Find a way to make encryption zone deletion work with HDFS trash.
Yi Liu created HDFS-7271:
Summary: Find a way to make encryption zone deletion work with
HDFS trash.
Key: HDFS-7271
URL: https://issues.apache.org/jira/browse/HDFS-7271
Project: Hadoop HDFS
Issue Type: Bug
Components: encryption
Affects Versions: 2.6.0
Reporter: Yi Liu
Assignee: Yi Liu
Currently when HDFS trash is enabled, deletion of encryption zone will have
issue:
{quote}
rmr: Failed to move to trash: ... can't be moved from an encryption zone.
{quote}
A simple way is to add ignore trash flag for fs rm operation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HDFS-7271) Find a way to make encryption zone deletion work with HDFS trash.
[
https://issues.apache.org/jira/browse/HDFS-7271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liu resolved HDFS-7271.
--
Resolution: Invalid
The -skipTrash already exists for rm op, so resolve it as invalid.
Find a way to make encryption zone deletion work with HDFS trash.
-
Key: HDFS-7271
URL: https://issues.apache.org/jira/browse/HDFS-7271
Project: Hadoop HDFS
Issue Type: Bug
Components: encryption
Affects Versions: 2.6.0
Reporter: Yi Liu
Assignee: Yi Liu
Currently when HDFS trash is enabled, deletion of encryption zone will have
issue:
{quote}
rmr: Failed to move to trash: ... can't be moved from an encryption zone.
{quote}
A simple way is to add ignore trash flag for fs rm operation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HDFS-7256) Encryption Key created in Java Key Store after Namenode start unavailable for EZ Creation
[ https://issues.apache.org/jira/browse/HDFS-7256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Liu resolved HDFS-7256. -- Resolution: Not a Problem I mark it as Not a Problem, please feel free to reopen it if you have different opinions. Encryption Key created in Java Key Store after Namenode start unavailable for EZ Creation -- Key: HDFS-7256 URL: https://issues.apache.org/jira/browse/HDFS-7256 Project: Hadoop HDFS Issue Type: Bug Components: encryption, security Affects Versions: 2.6.0 Reporter: Xiaoyu Yao Hit an error on RemoteException: Key ezkey1 doesn't exist. when creating EZ with a Key created after NN starts. Briefly check the code and found that the KeyProivder is loaded by FSN only at the NN start. My work around is to restart the NN which triggers the reload of Key Provider. Is this expected? Repro Steps: Create a new Key after NN and KMS starts hadoop/bin/hadoop key create ezkey1 -size 256 -provider jceks://file/home/hadoop/kms.keystore List Keys hadoop@SaturnVm:~/deploy$ hadoop/bin/hadoop key list -provider jceks://file/home/hadoop/kms.keystore -metadata Listing keys for KeyProvider: jceks://file/home/hadoop/kms.keystore ezkey1 : cipher: AES/CTR/NoPadding, length: 256, description: null, created: Thu Oct 16 18:51:30 EDT 2014, version: 1, attributes: null key2 : cipher: AES/CTR/NoPadding, length: 128, description: null, created: Tue Oct 14 19:44:09 EDT 2014, version: 1, attributes: null key1 : cipher: AES/CTR/NoPadding, length: 128, description: null, created: Tue Oct 14 17:52:36 EDT 2014, version: 1, attributes: null Create Encryption Zone hadoop/bin/hdfs dfs -mkdir /Ez1 hadoop@SaturnVm:~/deploy$ hadoop/bin/hdfs crypto -createZone -keyName ezkey1 -path /Ez1 RemoteException: Key ezkey1 doesn't exist. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HDFS-7253) getBlockLocationsUpdateTimes missing handle exception may cause fsLock dead lock
[ https://issues.apache.org/jira/browse/HDFS-7253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Liu resolved HDFS-7253. -- Resolution: Invalid Thanks Carrey for reporting the issue, mark it as invalid since it's already fixed in HDFS-7045. getBlockLocationsUpdateTimes missing handle exception may cause fsLock dead lock Key: HDFS-7253 URL: https://issues.apache.org/jira/browse/HDFS-7253 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.2.0 Reporter: Carrey Zhan Attachments: Tester.java, Trigger.tgz, nn1013.jstack One day my active namenode hanged and I dumped the program stacks by jstack.In the stacks file, I saw most threads were waiting FSNamesystem.fsLock, both readLock and writeLock were unacquirable, but no thread was holding writeLock. I tried to access the web interface of this namenode but was blocked. and I tried to failover the active node to another namenode manually (zkfs did not discover this node was hanging) but it was also failed. So I killed this namenode trying to recover the production environment, then the failover was triggered, standby nn transited to active, and then, the new active namenode hanged. My following steps are useless and can be ignored. At last, I thought it was caused by an incorrect lock handling in FSNamesystem.getBlockLocationsUpdateTimes, which I will describe in the first comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7242) Code improvement for FSN#checkUnreadableBySuperuser
Yi Liu created HDFS-7242:
Summary: Code improvement for FSN#checkUnreadableBySuperuser
Key: HDFS-7242
URL: https://issues.apache.org/jira/browse/HDFS-7242
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Affects Versions: 2.6.0
Reporter: Yi Liu
Assignee: Yi Liu
Priority: Minor
_checkUnreadableBySuperuser_ is to check whether user can access specific path.
The code logic is not efficient. It does iteration check for all user, actually
we just need to check _super user_ and can save few cpu cycle.
{code}
private void checkUnreadableBySuperuser(FSPermissionChecker pc,
INode inode, int snapshotId)
throws IOException {
for (XAttr xattr : dir.getXAttrs(inode, snapshotId)) {
if (XAttrHelper.getPrefixName(xattr).
equals(SECURITY_XATTR_UNREADABLE_BY_SUPERUSER)) {
if (pc.isSuperUser()) {
throw new AccessControlException(Access is denied for +
pc.getUser() + since the superuser is not allowed to +
perform this operation.);
}
}
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-7243) HDFS concat operation should not be allowed in Encryption Zone
Yi Liu created HDFS-7243: Summary: HDFS concat operation should not be allowed in Encryption Zone Key: HDFS-7243 URL: https://issues.apache.org/jira/browse/HDFS-7243 Project: Hadoop HDFS Issue Type: Bug Components: encryption, namenode Affects Versions: 2.6.0 Reporter: Yi Liu Assignee: Yi Liu For HDFS encryption at rest, files in an encryption zone are using different data encryption keys, so concat should be disallowed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7252) small refine for use of isInAnEZ in FSNamesystem
Yi Liu created HDFS-7252:
Summary: small refine for use of isInAnEZ in FSNamesystem
Key: HDFS-7252
URL: https://issues.apache.org/jira/browse/HDFS-7252
Project: Hadoop HDFS
Issue Type: Improvement
Affects Versions: 2.6.0
Reporter: Yi Liu
Assignee: Yi Liu
Priority: Trivial
In {{FSN#startFileInt}}, _EncryptionZoneManager#getEncryptionZoneForPath_ is
invoked 3 times (_dir.isInAnEZ(iip)_, _dir.getEZForPath(iip)_,
_dir.getKeyName(iip)_) in following code, actually we just need one.
{code}
if (dir.isInAnEZ(iip)) {
EncryptionZone zone = dir.getEZForPath(iip);
protocolVersion = chooseProtocolVersion(zone, supportedVersions);
suite = zone.getSuite();
ezKeyName = dir.getKeyName(iip);
Preconditions.checkNotNull(protocolVersion);
Preconditions.checkNotNull(suite);
Preconditions.checkArgument(!suite.equals(CipherSuite.UNKNOWN),
Chose an UNKNOWN CipherSuite!);
Preconditions.checkNotNull(ezKeyName);
}
{code}
Also there are 2 times in following code, but just need one
{code}
if (dir.isInAnEZ(iip)) {
// The path is now within an EZ, but we're missing encryption parameters
if (suite == null || edek == null) {
throw new RetryStartFileException();
}
// Path is within an EZ and we have provided encryption parameters.
// Make sure that the generated EDEK matches the settings of the EZ.
String ezKeyName = dir.getKeyName(iip);
if (!ezKeyName.equals(edek.getEncryptionKeyName())) {
throw new RetryStartFileException();
}
feInfo = new FileEncryptionInfo(suite, version,
edek.getEncryptedKeyVersion().getMaterial(),
edek.getEncryptedKeyIv(),
ezKeyName, edek.getEncryptionKeyVersionName());
Preconditions.checkNotNull(feInfo);
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-7209) Improve fs-encryption performance: fill the key queue when creating encryption zone
Yi Liu created HDFS-7209: Summary: Improve fs-encryption performance: fill the key queue when creating encryption zone Key: HDFS-7209 URL: https://issues.apache.org/jira/browse/HDFS-7209 Project: Hadoop HDFS Issue Type: Improvement Components: encryption, performance Affects Versions: 2.6.0 Reporter: Yi Liu Assignee: Yi Liu Currently when creating file in an encryption zone for the first time, key provider will get bunch of keys from KMS and fill in the queue. It will take some time. We can initialize the key queue when creating the encryption zone by admin. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7205) Delegation token for KMS should only be got once if it already exists
Yi Liu created HDFS-7205:
Summary: Delegation token for KMS should only be got once if it
already exists
Key: HDFS-7205
URL: https://issues.apache.org/jira/browse/HDFS-7205
Project: Hadoop HDFS
Issue Type: Bug
Components: encryption
Affects Versions: 2.6.0
Reporter: Yi Liu
Assignee: Yi Liu
When submit MapReduce job in security mode, we need to collect delegation
tokens (i.e. delegation token for NameNode, KMS).
{{addDelegationTokens}} may be invoked several times, currently dt for NN is
got only once if exists. But dt for KMS is got every time, we should fix this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-7206) Fix warning of token.Token: Cannot find class for token kind kms-dt for KMS when running jobs on Encryption zones
Yi Liu created HDFS-7206: Summary: Fix warning of token.Token: Cannot find class for token kind kms-dt for KMS when running jobs on Encryption zones Key: HDFS-7206 URL: https://issues.apache.org/jira/browse/HDFS-7206 Project: Hadoop HDFS Issue Type: Bug Components: encryption Affects Versions: 2.6.0 Reporter: Yi Liu Assignee: Yi Liu This issue is produced when running MapReduce job and encryption zones are configured. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7195) Update user doc of secure mode about Datanodes don't require root or jsvc
Yi Liu created HDFS-7195:
Summary: Update user doc of secure mode about Datanodes don't
require root or jsvc
Key: HDFS-7195
URL: https://issues.apache.org/jira/browse/HDFS-7195
Project: Hadoop HDFS
Issue Type: Task
Components: security
Reporter: Yi Liu
Assignee: Yi Liu
HDFS-2856 adds support that Datanodes don't require root or jsvc. If
{{dfs.data.transfer.protection}} is configured and {{dfs.http.policy}} is
_HTTPS_ONLY_, then secure dataNode doesn't need to use privileged port.
This has not been updated in the latest user doc of secure mode. This JIRA is
to fix that.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-7193) value of dfs.webhdfs.enabled in user doc is incorrect.
Yi Liu created HDFS-7193:
Summary: value of dfs.webhdfs.enabled in user doc is incorrect.
Key: HDFS-7193
URL: https://issues.apache.org/jira/browse/HDFS-7193
Project: Hadoop HDFS
Issue Type: Bug
Components: documentation, webhdfs
Reporter: Yi Liu
Assignee: Yi Liu
Priority: Trivial
The default value for {{dfs.webhdfs.enabled}} should be {{true}}, not
_http/_HOST@REALM.TLD_.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-7045) Fix deadlock of open file (in some cases)
Yi Liu created HDFS-7045:
Summary: Fix deadlock of open file (in some cases)
Key: HDFS-7045
URL: https://issues.apache.org/jira/browse/HDFS-7045
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Reporter: Yi Liu
Assignee: Yi Liu
Priority: Critical
There will be deadlock for current logic as following ({{resolvePath}} could
throw exception, then deadlock):
In {{FSNamesystem#getBlockLocationsUpdateTimes}}:
{code}
...
if (isReadOp) { // first attempt is with readlock
checkOperation(OperationCategory.READ);
readLock();
} else { // second attempt is with write lock
checkOperation(OperationCategory.WRITE);
writeLock(); // writelock is needed to set accesstime
}
src = resolvePath(src, pathComponents);
try {
...
} finally {
if (isReadOp) {
readUnlock();
} else {
writeUnlock();
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-6885) Fix wrong use of BytesWritable in FSEditLogOp#RenameOp
Yi Liu created HDFS-6885:
Summary: Fix wrong use of BytesWritable in FSEditLogOp#RenameOp
Key: HDFS-6885
URL: https://issues.apache.org/jira/browse/HDFS-6885
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Reporter: Yi Liu
Assignee: Yi Liu
Priority: Minor
After readField using BytesWritable, the data length should be
{{writable.getLength()}}, instead of {{writable.getBytes().length}} which is
the buffer length.
This will cause returned {{Rename[]}} is longer than expected and may include
some incorrect values (They are Rename#NONE, and have not caused problem but
code is incorrect).
{code}
BytesWritable writable = new BytesWritable();
writable.readFields(in);
byte[] bytes = writable.getBytes();
Rename[] options = new Rename[bytes.length];
for (int i = 0; i bytes.length; i++) {
options[i] = Rename.valueOf(bytes[i]);
}
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HDFS-6886) Use single editlog record for creating file + overwrite.
Yi Liu created HDFS-6886: Summary: Use single editlog record for creating file + overwrite. Key: HDFS-6886 URL: https://issues.apache.org/jira/browse/HDFS-6886 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Reporter: Yi Liu Assignee: Yi Liu As discussed in HDFS-6871, as [~jingzhao] and [~cmccabe]'s suggestion, we could do further improvement to use one editlog record for creating file + overwrite in this JIRA. We could record the overwrite flag in editlog for creating file. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6901) Remove unnecessary CryptoCodec and KeyProvider.Options definition in FSNamesystem
Yi Liu created HDFS-6901: Summary: Remove unnecessary CryptoCodec and KeyProvider.Options definition in FSNamesystem Key: HDFS-6901 URL: https://issues.apache.org/jira/browse/HDFS-6901 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Reporter: Yi Liu Assignee: Yi Liu Priority: Minor CryptoCodec and KeyProvider.Options are not necessary in FSN. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6870) Blocks and INodes could leak for Rename with overwrite flag
Yi Liu created HDFS-6870:
Summary: Blocks and INodes could leak for Rename with overwrite
flag
Key: HDFS-6870
URL: https://issues.apache.org/jira/browse/HDFS-6870
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Yi Liu
Assignee: Yi Liu
Following code in FSDirectory#unprotectedRenameTo doesn't collect blocks and
INodes for non snapshot path.
{code}
if (removedDst != null) {
undoRemoveDst = false;
if (removedNum 0) {
BlocksMapUpdateInfo collectedBlocks = new BlocksMapUpdateInfo();
ListINode removedINodes = new ChunkedArrayListINode();
filesDeleted = removedDst.cleanSubtree(Snapshot.CURRENT_STATE_ID,
dstIIP.getLatestSnapshotId(), collectedBlocks, removedINodes,
true).get(Quota.NAMESPACE);
getFSNamesystem().removePathAndBlocks(src, collectedBlocks,
removedINodes, false);
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HDFS-6871) Improve NameNode performance when creating file
Yi Liu created HDFS-6871:
Summary: Improve NameNode performance when creating file
Key: HDFS-6871
URL: https://issues.apache.org/jira/browse/HDFS-6871
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode, performance
Reporter: Yi Liu
Assignee: Yi Liu
Priority: Critical
Fix For: 2.6.0
Creating file with overwrite flag will cause NN fall into flush edit logs and
block other requests if the file exists.
When we create a file with overwrite flag (default is true) in HDFS, NN will
remove original file if it exists. In FSNamesystem#startFileInternal, NN
already holds the write lock, it calls {{deleteInt}} if the file exists, there
is logSync in {{deleteInt}}. So in this case, logSync is under write lock, it
will heavily affect the NN performance.
We should ignore the force logSync in {{deleteInt}} in this case.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Reopened] (HDFS-6783) Fix HDFS CacheReplicationMonitor rescan logic
[
https://issues.apache.org/jira/browse/HDFS-6783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liu reopened HDFS-6783:
--
Fix HDFS CacheReplicationMonitor rescan logic
-
Key: HDFS-6783
URL: https://issues.apache.org/jira/browse/HDFS-6783
Project: Hadoop HDFS
Issue Type: Bug
Components: caching
Affects Versions: 3.0.0
Reporter: Yi Liu
Assignee: Yi Liu
Fix For: 3.0.0, 2.6.0
Attachments: HDFS-6783.001.patch, HDFS-6783.002.patch,
HDFS-6783.003.patch, HDFS-6783.004.patch, HDFS-6783.005.patch,
HDFS-6783.006.patch
In monitor thread, needsRescan is set to false before real scan starts, so
for {{waitForRescanIfNeeded}} will return for the first condition:
{code}
if (!needsRescan) {
return;
}
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HDFS-6846) NetworkTopology#sortByDistance should give nodes higher priority, which cache the block.
Yi Liu created HDFS-6846: Summary: NetworkTopology#sortByDistance should give nodes higher priority, which cache the block. Key: HDFS-6846 URL: https://issues.apache.org/jira/browse/HDFS-6846 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Affects Versions: 2.6.0 Reporter: Yi Liu Assignee: Yi Liu Currently there are 3 weights: * local * same rack * off rack But if some nodes cache the block, then it's faster if client read block from these nodes. So we should have some more weights as following: * local * cached same rack * same rack * cached off rack * off rack -- This message was sent by Atlassian JIRA (v6.2#6252)
