[jira] [Commented] (HDFS-7438) Consolidate implementation of rename()
[
https://issues.apache.org/jira/browse/HDFS-7438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14225897#comment-14225897
]
Hadoop QA commented on HDFS-7438:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12683763/HDFS-7438.001.patch
against trunk revision 4a31611.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestDFSPermission
org.apache.hadoop.security.TestPermissionSymlinks
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
org.apache.hadoop.hdfs.server.namenode.snapshot.TestDisallowModifyROSnapshot
org.apache.hadoop.fs.permission.TestStickyBit
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8843//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8843//console
This message is automatically generated.
Consolidate implementation of rename()
--
Key: HDFS-7438
URL: https://issues.apache.org/jira/browse/HDFS-7438
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Haohui Mai
Attachments: HDFS-7438.000.patch, HDFS-7438.001.patch
The implementation of {{rename()}} resides in both {{FSNameSystem}} and
{{FSDirectory}}. This jira proposes to consolidate them in a single class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-7447) Number of maximum Acl entries on a File/Folder should be made user configurable than hardcoding .
J.Andreina created HDFS-7447:
Summary: Number of maximum Acl entries on a File/Folder should be
made user configurable than hardcoding .
Key: HDFS-7447
URL: https://issues.apache.org/jira/browse/HDFS-7447
Project: Hadoop HDFS
Issue Type: Improvement
Components: security
Reporter: J.Andreina
By default on creating a folder1 will have 6 acl entries . On top of that
assigning acl on a folder1 exceeds 32 , then unable to assign acls for a
group/user to folder1.
{noformat}
2014-11-20 18:55:06,553 ERROR [qtp1279235236-17 - /rolexml/role/modrole] Error
occured while setting permissions for Resource:[ hdfs://hacluster/folder1 ] and
Error message is : Invalid ACL: ACL has 33 entries, which exceeds maximum of 32.
at
org.apache.hadoop.hdfs.server.namenode.AclTransformation.buildAndValidateAcl(AclTransformation.java:274)
at
org.apache.hadoop.hdfs.server.namenode.AclTransformation.mergeAclEntries(AclTransformation.java:181)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.unprotectedModifyAclEntries(FSDirectory.java:2771)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.modifyAclEntries(FSDirectory.java:2757)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.modifyAclEntries(FSNamesystem.java:7734)
{noformat}
Here value 32 is hardcoded , which can be made user configurable.
{noformat}
private static List buildAndValidateAcl(ArrayList aclBuilder)
throws AclException
{
if(aclBuilder.size() 32)
throw new AclException((new StringBuilder()).append(Invalid ACL:
ACL has ).append(aclBuilder.size()).append( entries, which exceeds maximum of
).append(32).append(.).toString());
:
:
}
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6133) Make Balancer support exclude specified path
[
https://issues.apache.org/jira/browse/HDFS-6133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14225982#comment-14225982
]
Hadoop QA commented on HDFS-6133:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12683784/HDFS-6133-5.patch
against trunk revision 4a31611.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8845//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8845//console
This message is automatically generated.
Make Balancer support exclude specified path
Key: HDFS-6133
URL: https://issues.apache.org/jira/browse/HDFS-6133
Project: Hadoop HDFS
Issue Type: Improvement
Components: balancer mover, namenode
Reporter: zhaoyunjiong
Assignee: zhaoyunjiong
Attachments: HDFS-6133-1.patch, HDFS-6133-2.patch, HDFS-6133-3.patch,
HDFS-6133-4.patch, HDFS-6133-5.patch, HDFS-6133.patch
Currently, run Balancer will destroying Regionserver's data locality.
If getBlocks could exclude blocks belongs to files which have specific path
prefix, like /hbase, then we can run Balancer without destroying
Regionserver's data locality.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7310) Mover can give first priority to local DN if it has target storage type available in local DN
[
https://issues.apache.org/jira/browse/HDFS-7310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14225981#comment-14225981
]
Hadoop QA commented on HDFS-7310:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12683779/HDFS-7310-004.patch
against trunk revision 4a31611.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8844//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8844//console
This message is automatically generated.
Mover can give first priority to local DN if it has target storage type
available in local DN
-
Key: HDFS-7310
URL: https://issues.apache.org/jira/browse/HDFS-7310
Project: Hadoop HDFS
Issue Type: Improvement
Components: balancer mover
Affects Versions: 3.0.0
Reporter: Uma Maheswara Rao G
Assignee: Vinayakumar B
Attachments: HDFS-7310-001.patch, HDFS-7310-002.patch,
HDFS-7310-003.patch, HDFS-7310-004.patch
Currently Mover logic may move blocks to any DN which had target storage
type. But if the src DN has target storage type then mover can give highest
priority to local DN. If local DN does not contains target storage type, then
it can assign to any DN as the current logic does.
This is a thought, have not go through the code fully yet.
Thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-6803) Documenting DFSClient#DFSInputStream expectations reading and preading in concurrent context
[ https://issues.apache.org/jira/browse/HDFS-6803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HDFS-6803: - Assignee: stack Documenting DFSClient#DFSInputStream expectations reading and preading in concurrent context Key: HDFS-6803 URL: https://issues.apache.org/jira/browse/HDFS-6803 Project: Hadoop HDFS Issue Type: Sub-task Components: hdfs-client Affects Versions: 2.4.1 Reporter: stack Assignee: stack Attachments: 9117.md.txt, DocumentingDFSClientDFSInputStream (1).pdf, DocumentingDFSClientDFSInputStream.v2.pdf, HDFS-6803v2.txt, HDFS-6803v3.txt, fsdatainputstream.md.v3.html Reviews of the patch posted the parent task suggest that we be more explicit about how DFSIS is expected to behave when being read by contending threads. It is also suggested that presumptions made internally be made explicit documenting expectations. Before we put up a patch we've made a document of assertions we'd like to make into tenets of DFSInputSteam. If agreement, we'll attach to this issue a patch that weaves the assumptions into DFSIS as javadoc and class comments. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HDFS-6803) Documenting DFSClient#DFSInputStream expectations reading and preading in concurrent context
[ https://issues.apache.org/jira/browse/HDFS-6803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HDFS-6803. -- Resolution: Fixed Fix Version/s: 2.7.0 committed patch -thanks! Documenting DFSClient#DFSInputStream expectations reading and preading in concurrent context Key: HDFS-6803 URL: https://issues.apache.org/jira/browse/HDFS-6803 Project: Hadoop HDFS Issue Type: Sub-task Components: hdfs-client Affects Versions: 2.4.1 Reporter: stack Assignee: stack Fix For: 2.7.0 Attachments: 9117.md.txt, DocumentingDFSClientDFSInputStream (1).pdf, DocumentingDFSClientDFSInputStream.v2.pdf, HDFS-6803v2.txt, HDFS-6803v3.txt, fsdatainputstream.md.v3.html Reviews of the patch posted the parent task suggest that we be more explicit about how DFSIS is expected to behave when being read by contending threads. It is also suggested that presumptions made internally be made explicit documenting expectations. Before we put up a patch we've made a document of assertions we'd like to make into tenets of DFSInputSteam. If agreement, we'll attach to this issue a patch that weaves the assumptions into DFSIS as javadoc and class comments. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7440) Consolidate snapshot related operations in a single class
[
https://issues.apache.org/jira/browse/HDFS-7440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226069#comment-14226069
]
Hudson commented on HDFS-7440:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #17 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/17/])
HDFS-7440. Consolidate snapshot related operations in a single class.
Contributed by Haohui Mai. (wheat9: rev
4a3161182905afaf450a60d02528161ed1f97471)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirSnapshotOp.java
Consolidate snapshot related operations in a single class
-
Key: HDFS-7440
URL: https://issues.apache.org/jira/browse/HDFS-7440
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Haohui Mai
Fix For: 2.7.0
Attachments: HDFS-7440.000.patch, HDFS-7440.001.patch,
HDFS-7440.002.patch
Currently the snapshot-related code scatters across both {{FSNameSystem}} and
{{FSDirectory}}. This jira proposes to consolidate the logic in a single
class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7097) Allow block reports to be processed during checkpointing on standby name node
[
https://issues.apache.org/jira/browse/HDFS-7097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226067#comment-14226067
]
Hudson commented on HDFS-7097:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #17 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/17/])
HDFS-7097. Allow block reports to be processed during checkpointing on standby
name node. (kihwal via wang) (wang: rev
f43a20c529ac3f104add95b222de6580757b3763)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestStandbyCheckpoints.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/StandbyCheckpointer.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
Allow block reports to be processed during checkpointing on standby name node
-
Key: HDFS-7097
URL: https://issues.apache.org/jira/browse/HDFS-7097
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Kihwal Lee
Assignee: Kihwal Lee
Priority: Critical
Attachments: HDFS-7097.patch, HDFS-7097.patch, HDFS-7097.patch,
HDFS-7097.patch, HDFS-7097.ultimate.trunk.patch
On a reasonably busy HDFS cluster, there are stream of creates, causing data
nodes to generate incremental block reports. When a standby name node is
checkpointing, RPC handler threads trying to process a full or incremental
block report is blocked on the name system's {{fsLock}}, because the
checkpointer acquires the read lock on it. This can create a serious problem
if the size of name space is big and checkpointing takes a long time.
All available RPC handlers can be tied up very quickly. If you have 100
handlers, it only takes 34 file creates. If a separate service RPC port is
not used, HA transition will have to wait in the call queue for minutes. Even
if a separate service RPC port is configured, hearbeats from datanodes will
be blocked. A standby NN with a big name space can lose all data nodes after
checkpointing. The rpc calls will also be retransmitted by data nodes many
times, filling up the call queue and potentially causing listen queue
overflow.
Since block reports are not modifying any state that is being saved to
fsimage, I propose letting them through during checkpointing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7435) PB encoding of block reports is very inefficient
[
https://issues.apache.org/jira/browse/HDFS-7435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226076#comment-14226076
]
Yi Liu commented on HDFS-7435:
--
I open HADOOP-11339 to reuse buffer for Hadoop RPC as stated in my above
comment.
PB encoding of block reports is very inefficient
Key: HDFS-7435
URL: https://issues.apache.org/jira/browse/HDFS-7435
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode, namenode
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Priority: Critical
Attachments: HDFS-7435.000.patch, HDFS-7435.patch
Block reports are encoded as a PB repeating long. Repeating fields use an
{{ArrayList}} with default capacity of 10. A block report containing tens or
hundreds of thousand of longs (3 for each replica) is extremely expensive
since the {{ArrayList}} must realloc many times. Also, decoding repeating
fields will box the primitive longs which must then be unboxed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6803) Documenting DFSClient#DFSInputStream expectations reading and preading in concurrent context
[ https://issues.apache.org/jira/browse/HDFS-6803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226079#comment-14226079 ] Hudson commented on HDFS-6803: -- FAILURE: Integrated in Hadoop-trunk-Commit #6609 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6609/]) HDFS-6803 Document DFSClient#DFSInputStream expectations reading and preading in concurrent context. (stack via stevel) (stevel: rev aa7dac335960950d2254a5a78bd1f0786a290538) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-common-project/hadoop-common/src/site/markdown/filesystem/fsdatainputstream.md Documenting DFSClient#DFSInputStream expectations reading and preading in concurrent context Key: HDFS-6803 URL: https://issues.apache.org/jira/browse/HDFS-6803 Project: Hadoop HDFS Issue Type: Sub-task Components: hdfs-client Affects Versions: 2.4.1 Reporter: stack Assignee: stack Fix For: 2.7.0 Attachments: 9117.md.txt, DocumentingDFSClientDFSInputStream (1).pdf, DocumentingDFSClientDFSInputStream.v2.pdf, HDFS-6803v2.txt, HDFS-6803v3.txt, fsdatainputstream.md.v3.html Reviews of the patch posted the parent task suggest that we be more explicit about how DFSIS is expected to behave when being read by contending threads. It is also suggested that presumptions made internally be made explicit documenting expectations. Before we put up a patch we've made a document of assertions we'd like to make into tenets of DFSInputSteam. If agreement, we'll attach to this issue a patch that weaves the assumptions into DFSIS as javadoc and class comments. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7097) Allow block reports to be processed during checkpointing on standby name node
[
https://issues.apache.org/jira/browse/HDFS-7097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226087#comment-14226087
]
Hudson commented on HDFS-7097:
--
SUCCESS: Integrated in Hadoop-Yarn-trunk #755 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/755/])
HDFS-7097. Allow block reports to be processed during checkpointing on standby
name node. (kihwal via wang) (wang: rev
f43a20c529ac3f104add95b222de6580757b3763)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/StandbyCheckpointer.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestStandbyCheckpoints.java
Allow block reports to be processed during checkpointing on standby name node
-
Key: HDFS-7097
URL: https://issues.apache.org/jira/browse/HDFS-7097
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Kihwal Lee
Assignee: Kihwal Lee
Priority: Critical
Attachments: HDFS-7097.patch, HDFS-7097.patch, HDFS-7097.patch,
HDFS-7097.patch, HDFS-7097.ultimate.trunk.patch
On a reasonably busy HDFS cluster, there are stream of creates, causing data
nodes to generate incremental block reports. When a standby name node is
checkpointing, RPC handler threads trying to process a full or incremental
block report is blocked on the name system's {{fsLock}}, because the
checkpointer acquires the read lock on it. This can create a serious problem
if the size of name space is big and checkpointing takes a long time.
All available RPC handlers can be tied up very quickly. If you have 100
handlers, it only takes 34 file creates. If a separate service RPC port is
not used, HA transition will have to wait in the call queue for minutes. Even
if a separate service RPC port is configured, hearbeats from datanodes will
be blocked. A standby NN with a big name space can lose all data nodes after
checkpointing. The rpc calls will also be retransmitted by data nodes many
times, filling up the call queue and potentially causing listen queue
overflow.
Since block reports are not modifying any state that is being saved to
fsimage, I propose letting them through during checkpointing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7440) Consolidate snapshot related operations in a single class
[
https://issues.apache.org/jira/browse/HDFS-7440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226089#comment-14226089
]
Hudson commented on HDFS-7440:
--
SUCCESS: Integrated in Hadoop-Yarn-trunk #755 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/755/])
HDFS-7440. Consolidate snapshot related operations in a single class.
Contributed by Haohui Mai. (wheat9: rev
4a3161182905afaf450a60d02528161ed1f97471)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirSnapshotOp.java
Consolidate snapshot related operations in a single class
-
Key: HDFS-7440
URL: https://issues.apache.org/jira/browse/HDFS-7440
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Haohui Mai
Fix For: 2.7.0
Attachments: HDFS-7440.000.patch, HDFS-7440.001.patch,
HDFS-7440.002.patch
Currently the snapshot-related code scatters across both {{FSNameSystem}} and
{{FSDirectory}}. This jira proposes to consolidate the logic in a single
class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7414) Namenode got shutdown and can't recover where edit update might be missed
[
https://issues.apache.org/jira/browse/HDFS-7414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226141#comment-14226141
]
Brahma Reddy Battula commented on HDFS-7414:
Seen following error from namenode log when it's initially came..
{noformat}
2014-11-20 05:01:18,430 | ERROR | main | Encountered exception on operation
CloseOp [length=0, inodeId=0,
path=/outDir2/_temporary/1/_temporary/attempt_1416390004064_0002_m_25_1/part-m-00025,
replication=2, mtime=1416409309023, atime=1416409290816, blockSize=67108864,
blocks=[blk_1073766144_25321, blk_1073766154_25331, blk_1073766160_25337],
permissions=mapred:supergroup:rw-r--r--, aclEntries=null, clientName=,
clientMachine=, opCode=OP_CLOSE, txid=162982] |
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:232
{noformat}
and let me describe full scenario..
Deleting DIr (which is having 100 files) and running mapreduce job..
I think, while gettingLastInodeInpath it may get null, since I am deleting the
dir where it edits will not synced..
{code}
final INodesInPath iip = fsDir.getLastINodeInPath(path);
final INodeFile file = INodeFile.valueOf(iip.getINode(0), path);
{code}
Anyone have Anythoughts on this..?
Namenode got shutdown and can't recover where edit update might be missed
-
Key: HDFS-7414
URL: https://issues.apache.org/jira/browse/HDFS-7414
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.4.1, 2.5.1
Reporter: Brahma Reddy Battula
Assignee: Brahma Reddy Battula
Priority: Blocker
Scenario:
Was running mapreduce job.
CPU usage crossed 190% for Datanode and machine became slow..
and seen the following exception ..
*Did not get the exact root cause, but as cpu usage more edit log updation
might be missed...Need dig to more...anyone have any thoughts.*
{noformat}
2014-11-20 05:01:18,430 | ERROR | main | Encountered exception on operation
CloseOp [length=0, inodeId=0,
path=/outDir2/_temporary/1/_temporary/attempt_1416390004064_0002_m_25_1/part-m-00025,
replication=2, mtime=1416409309023, atime=1416409290816, blockSize=67108864,
blocks=[blk_1073766144_25321, blk_1073766154_25331, blk_1073766160_25337],
permissions=mapred:supergroup:rw-r--r--, aclEntries=null, clientName=,
clientMachine=, opCode=OP_CLOSE, txid=162982] |
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:232)
java.io.FileNotFoundException: File does not exist:
/outDir2/_temporary/1/_temporary/attempt_1416390004064_0002_m_25_1/part-m-00025
at
org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:65)
at
org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:55)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:409)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:224)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:133)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:805)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:665)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:272)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:893)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:640)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:519)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:575)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:741)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:724)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1387)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1459)
2014-11-20 05:01:18,654 | WARN | main | Encountered exception loading
fsimage |
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:642)
java.io.FileNotFoundException: File does not exist:
/outDir2/_temporary/1/_temporary/attempt_1416390004064_0002_m_25_1/part-m-00025
at
org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:65)
at
org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:55)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:409)
at
[jira] [Commented] (HDFS-7097) Allow block reports to be processed during checkpointing on standby name node
[
https://issues.apache.org/jira/browse/HDFS-7097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226238#comment-14226238
]
Hudson commented on HDFS-7097:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #1945 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1945/])
HDFS-7097. Allow block reports to be processed during checkpointing on standby
name node. (kihwal via wang) (wang: rev
f43a20c529ac3f104add95b222de6580757b3763)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/StandbyCheckpointer.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestStandbyCheckpoints.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
Allow block reports to be processed during checkpointing on standby name node
-
Key: HDFS-7097
URL: https://issues.apache.org/jira/browse/HDFS-7097
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Kihwal Lee
Assignee: Kihwal Lee
Priority: Critical
Attachments: HDFS-7097.patch, HDFS-7097.patch, HDFS-7097.patch,
HDFS-7097.patch, HDFS-7097.ultimate.trunk.patch
On a reasonably busy HDFS cluster, there are stream of creates, causing data
nodes to generate incremental block reports. When a standby name node is
checkpointing, RPC handler threads trying to process a full or incremental
block report is blocked on the name system's {{fsLock}}, because the
checkpointer acquires the read lock on it. This can create a serious problem
if the size of name space is big and checkpointing takes a long time.
All available RPC handlers can be tied up very quickly. If you have 100
handlers, it only takes 34 file creates. If a separate service RPC port is
not used, HA transition will have to wait in the call queue for minutes. Even
if a separate service RPC port is configured, hearbeats from datanodes will
be blocked. A standby NN with a big name space can lose all data nodes after
checkpointing. The rpc calls will also be retransmitted by data nodes many
times, filling up the call queue and potentially causing listen queue
overflow.
Since block reports are not modifying any state that is being saved to
fsimage, I propose letting them through during checkpointing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7097) Allow block reports to be processed during checkpointing on standby name node
[
https://issues.apache.org/jira/browse/HDFS-7097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226248#comment-14226248
]
Hudson commented on HDFS-7097:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #17 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/17/])
HDFS-7097. Allow block reports to be processed during checkpointing on standby
name node. (kihwal via wang) (wang: rev
f43a20c529ac3f104add95b222de6580757b3763)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestStandbyCheckpoints.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/StandbyCheckpointer.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
Allow block reports to be processed during checkpointing on standby name node
-
Key: HDFS-7097
URL: https://issues.apache.org/jira/browse/HDFS-7097
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Kihwal Lee
Assignee: Kihwal Lee
Priority: Critical
Attachments: HDFS-7097.patch, HDFS-7097.patch, HDFS-7097.patch,
HDFS-7097.patch, HDFS-7097.ultimate.trunk.patch
On a reasonably busy HDFS cluster, there are stream of creates, causing data
nodes to generate incremental block reports. When a standby name node is
checkpointing, RPC handler threads trying to process a full or incremental
block report is blocked on the name system's {{fsLock}}, because the
checkpointer acquires the read lock on it. This can create a serious problem
if the size of name space is big and checkpointing takes a long time.
All available RPC handlers can be tied up very quickly. If you have 100
handlers, it only takes 34 file creates. If a separate service RPC port is
not used, HA transition will have to wait in the call queue for minutes. Even
if a separate service RPC port is configured, hearbeats from datanodes will
be blocked. A standby NN with a big name space can lose all data nodes after
checkpointing. The rpc calls will also be retransmitted by data nodes many
times, filling up the call queue and potentially causing listen queue
overflow.
Since block reports are not modifying any state that is being saved to
fsimage, I propose letting them through during checkpointing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7440) Consolidate snapshot related operations in a single class
[
https://issues.apache.org/jira/browse/HDFS-7440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226240#comment-14226240
]
Hudson commented on HDFS-7440:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #1945 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1945/])
HDFS-7440. Consolidate snapshot related operations in a single class.
Contributed by Haohui Mai. (wheat9: rev
4a3161182905afaf450a60d02528161ed1f97471)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirSnapshotOp.java
Consolidate snapshot related operations in a single class
-
Key: HDFS-7440
URL: https://issues.apache.org/jira/browse/HDFS-7440
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Haohui Mai
Fix For: 2.7.0
Attachments: HDFS-7440.000.patch, HDFS-7440.001.patch,
HDFS-7440.002.patch
Currently the snapshot-related code scatters across both {{FSNameSystem}} and
{{FSDirectory}}. This jira proposes to consolidate the logic in a single
class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7440) Consolidate snapshot related operations in a single class
[
https://issues.apache.org/jira/browse/HDFS-7440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226250#comment-14226250
]
Hudson commented on HDFS-7440:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #17 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/17/])
HDFS-7440. Consolidate snapshot related operations in a single class.
Contributed by Haohui Mai. (wheat9: rev
4a3161182905afaf450a60d02528161ed1f97471)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirSnapshotOp.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
Consolidate snapshot related operations in a single class
-
Key: HDFS-7440
URL: https://issues.apache.org/jira/browse/HDFS-7440
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Haohui Mai
Fix For: 2.7.0
Attachments: HDFS-7440.000.patch, HDFS-7440.001.patch,
HDFS-7440.002.patch
Currently the snapshot-related code scatters across both {{FSNameSystem}} and
{{FSDirectory}}. This jira proposes to consolidate the logic in a single
class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6407) new namenode UI, lost ability to sort columns in datanode tab
[ https://issues.apache.org/jira/browse/HDFS-6407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226255#comment-14226255 ] Benoy Antony commented on HDFS-6407: Why not use the plugin ? Its a widely used plugin and used in other Apache projects like Storm. Based on history, it's quite mature. It has a small size of 30 Kb. It seemed to function very well during my testing. Instead of re-inventing the wheel, why not use a standard solution ? Enabling sorting on a specific table with the plugin is quite straight forward as I explained in the comment above. It is desirable to sort any table. Currently , I added sorting on data nodes, snapshots and filesystem browsing. But if a table doesn't need sorting for some reason, it can be easily removed. BTW, pagination was never supported. It is normally required because client doesn't have all the data which is not the case here. new namenode UI, lost ability to sort columns in datanode tab - Key: HDFS-6407 URL: https://issues.apache.org/jira/browse/HDFS-6407 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.4.0 Reporter: Nathan Roberts Assignee: Benoy Antony Priority: Minor Attachments: HDFS-6407.patch, browse_directory.png, datanodes.png, snapshots.png old ui supported clicking on column header to sort on that column. The new ui seems to have dropped this very useful feature. There are a few tables in the Namenode UI to display datanodes information, directory listings and snapshots. When there are many items in the tables, it is useful to have ability to sort on the different columns. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7435) PB encoding of block reports is very inefficient
[
https://issues.apache.org/jira/browse/HDFS-7435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226276#comment-14226276
]
Daryn Sharp commented on HDFS-7435:
---
Upon quick scan, it appears the the chunk size is encoded in the buffer? If
yes, I think each side should be able chose an appropriate chunk size - as an
implementation detail - which is not leaked into the PB encoding. If we can
agree on that point, can we move the segmentation to a separate jira?
PB encoding of block reports is very inefficient
Key: HDFS-7435
URL: https://issues.apache.org/jira/browse/HDFS-7435
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode, namenode
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Priority: Critical
Attachments: HDFS-7435.000.patch, HDFS-7435.patch
Block reports are encoded as a PB repeating long. Repeating fields use an
{{ArrayList}} with default capacity of 10. A block report containing tens or
hundreds of thousand of longs (3 for each replica) is extremely expensive
since the {{ArrayList}} must realloc many times. Also, decoding repeating
fields will box the primitive longs which must then be unboxed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6803) Documenting DFSClient#DFSInputStream expectations reading and preading in concurrent context
[ https://issues.apache.org/jira/browse/HDFS-6803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226294#comment-14226294 ] Hudson commented on HDFS-6803: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1969 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1969/]) HDFS-6803 Document DFSClient#DFSInputStream expectations reading and preading in concurrent context. (stack via stevel) (stevel: rev aa7dac335960950d2254a5a78bd1f0786a290538) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-common-project/hadoop-common/src/site/markdown/filesystem/fsdatainputstream.md Documenting DFSClient#DFSInputStream expectations reading and preading in concurrent context Key: HDFS-6803 URL: https://issues.apache.org/jira/browse/HDFS-6803 Project: Hadoop HDFS Issue Type: Sub-task Components: hdfs-client Affects Versions: 2.4.1 Reporter: stack Assignee: stack Fix For: 2.7.0 Attachments: 9117.md.txt, DocumentingDFSClientDFSInputStream (1).pdf, DocumentingDFSClientDFSInputStream.v2.pdf, HDFS-6803v2.txt, HDFS-6803v3.txt, fsdatainputstream.md.v3.html Reviews of the patch posted the parent task suggest that we be more explicit about how DFSIS is expected to behave when being read by contending threads. It is also suggested that presumptions made internally be made explicit documenting expectations. Before we put up a patch we've made a document of assertions we'd like to make into tenets of DFSInputSteam. If agreement, we'll attach to this issue a patch that weaves the assumptions into DFSIS as javadoc and class comments. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7097) Allow block reports to be processed during checkpointing on standby name node
[
https://issues.apache.org/jira/browse/HDFS-7097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226302#comment-14226302
]
Hudson commented on HDFS-7097:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #1969 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1969/])
HDFS-7097. Allow block reports to be processed during checkpointing on standby
name node. (kihwal via wang) (wang: rev
f43a20c529ac3f104add95b222de6580757b3763)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/StandbyCheckpointer.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestStandbyCheckpoints.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java
Allow block reports to be processed during checkpointing on standby name node
-
Key: HDFS-7097
URL: https://issues.apache.org/jira/browse/HDFS-7097
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Kihwal Lee
Assignee: Kihwal Lee
Priority: Critical
Attachments: HDFS-7097.patch, HDFS-7097.patch, HDFS-7097.patch,
HDFS-7097.patch, HDFS-7097.ultimate.trunk.patch
On a reasonably busy HDFS cluster, there are stream of creates, causing data
nodes to generate incremental block reports. When a standby name node is
checkpointing, RPC handler threads trying to process a full or incremental
block report is blocked on the name system's {{fsLock}}, because the
checkpointer acquires the read lock on it. This can create a serious problem
if the size of name space is big and checkpointing takes a long time.
All available RPC handlers can be tied up very quickly. If you have 100
handlers, it only takes 34 file creates. If a separate service RPC port is
not used, HA transition will have to wait in the call queue for minutes. Even
if a separate service RPC port is configured, hearbeats from datanodes will
be blocked. A standby NN with a big name space can lose all data nodes after
checkpointing. The rpc calls will also be retransmitted by data nodes many
times, filling up the call queue and potentially causing listen queue
overflow.
Since block reports are not modifying any state that is being saved to
fsimage, I propose letting them through during checkpointing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7440) Consolidate snapshot related operations in a single class
[
https://issues.apache.org/jira/browse/HDFS-7440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226304#comment-14226304
]
Hudson commented on HDFS-7440:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #1969 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1969/])
HDFS-7440. Consolidate snapshot related operations in a single class.
Contributed by Haohui Mai. (wheat9: rev
4a3161182905afaf450a60d02528161ed1f97471)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirSnapshotOp.java
Consolidate snapshot related operations in a single class
-
Key: HDFS-7440
URL: https://issues.apache.org/jira/browse/HDFS-7440
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Haohui Mai
Fix For: 2.7.0
Attachments: HDFS-7440.000.patch, HDFS-7440.001.patch,
HDFS-7440.002.patch
Currently the snapshot-related code scatters across both {{FSNameSystem}} and
{{FSDirectory}}. This jira proposes to consolidate the logic in a single
class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6803) Documenting DFSClient#DFSInputStream expectations reading and preading in concurrent context
[ https://issues.apache.org/jira/browse/HDFS-6803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226311#comment-14226311 ] Hudson commented on HDFS-6803: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #17 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/17/]) HDFS-6803 Document DFSClient#DFSInputStream expectations reading and preading in concurrent context. (stack via stevel) (stevel: rev aa7dac335960950d2254a5a78bd1f0786a290538) * hadoop-common-project/hadoop-common/src/site/markdown/filesystem/fsdatainputstream.md * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Documenting DFSClient#DFSInputStream expectations reading and preading in concurrent context Key: HDFS-6803 URL: https://issues.apache.org/jira/browse/HDFS-6803 Project: Hadoop HDFS Issue Type: Sub-task Components: hdfs-client Affects Versions: 2.4.1 Reporter: stack Assignee: stack Fix For: 2.7.0 Attachments: 9117.md.txt, DocumentingDFSClientDFSInputStream (1).pdf, DocumentingDFSClientDFSInputStream.v2.pdf, HDFS-6803v2.txt, HDFS-6803v3.txt, fsdatainputstream.md.v3.html Reviews of the patch posted the parent task suggest that we be more explicit about how DFSIS is expected to behave when being read by contending threads. It is also suggested that presumptions made internally be made explicit documenting expectations. Before we put up a patch we've made a document of assertions we'd like to make into tenets of DFSInputSteam. If agreement, we'll attach to this issue a patch that weaves the assumptions into DFSIS as javadoc and class comments. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7097) Allow block reports to be processed during checkpointing on standby name node
[
https://issues.apache.org/jira/browse/HDFS-7097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226319#comment-14226319
]
Hudson commented on HDFS-7097:
--
SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #17 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/17/])
HDFS-7097. Allow block reports to be processed during checkpointing on standby
name node. (kihwal via wang) (wang: rev
f43a20c529ac3f104add95b222de6580757b3763)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/StandbyCheckpointer.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestStandbyCheckpoints.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java
Allow block reports to be processed during checkpointing on standby name node
-
Key: HDFS-7097
URL: https://issues.apache.org/jira/browse/HDFS-7097
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Kihwal Lee
Assignee: Kihwal Lee
Priority: Critical
Attachments: HDFS-7097.patch, HDFS-7097.patch, HDFS-7097.patch,
HDFS-7097.patch, HDFS-7097.ultimate.trunk.patch
On a reasonably busy HDFS cluster, there are stream of creates, causing data
nodes to generate incremental block reports. When a standby name node is
checkpointing, RPC handler threads trying to process a full or incremental
block report is blocked on the name system's {{fsLock}}, because the
checkpointer acquires the read lock on it. This can create a serious problem
if the size of name space is big and checkpointing takes a long time.
All available RPC handlers can be tied up very quickly. If you have 100
handlers, it only takes 34 file creates. If a separate service RPC port is
not used, HA transition will have to wait in the call queue for minutes. Even
if a separate service RPC port is configured, hearbeats from datanodes will
be blocked. A standby NN with a big name space can lose all data nodes after
checkpointing. The rpc calls will also be retransmitted by data nodes many
times, filling up the call queue and potentially causing listen queue
overflow.
Since block reports are not modifying any state that is being saved to
fsimage, I propose letting them through during checkpointing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7440) Consolidate snapshot related operations in a single class
[
https://issues.apache.org/jira/browse/HDFS-7440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226321#comment-14226321
]
Hudson commented on HDFS-7440:
--
SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #17 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/17/])
HDFS-7440. Consolidate snapshot related operations in a single class.
Contributed by Haohui Mai. (wheat9: rev
4a3161182905afaf450a60d02528161ed1f97471)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirSnapshotOp.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
Consolidate snapshot related operations in a single class
-
Key: HDFS-7440
URL: https://issues.apache.org/jira/browse/HDFS-7440
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Haohui Mai
Fix For: 2.7.0
Attachments: HDFS-7440.000.patch, HDFS-7440.001.patch,
HDFS-7440.002.patch
Currently the snapshot-related code scatters across both {{FSNameSystem}} and
{{FSDirectory}}. This jira proposes to consolidate the logic in a single
class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-7448) TestBookKeeperHACheckpoints fails in trunk build
Ted Yu created HDFS-7448:
Summary: TestBookKeeperHACheckpoints fails in trunk build
Key: HDFS-7448
URL: https://issues.apache.org/jira/browse/HDFS-7448
Project: Hadoop HDFS
Issue Type: Test
Reporter: Ted Yu
Priority: Minor
The test failed against both java 7 and java 8.
From https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/17/console :
{code}
testStandbyExceptionThrownDuringCheckpoint(org.apache.hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints)
Time elapsed: 6.822 sec ERROR!
org.apache.hadoop.ipc.RemoteException: File /testFile could only be replicated
to 0 nodes instead of minReplication (=1). There are 0 datanode(s) running and
no node(s) are excluded in this operation.
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1558)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3024)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:699)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:482)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:637)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:966)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2125)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2121)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1683)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2119)
at org.apache.hadoop.ipc.Client.call(Client.java:1468)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
at com.sun.proxy.$Proxy20.addBlock(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:399)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:101)
at com.sun.proxy.$Proxy21.addBlock(Unknown Source)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1544)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1361)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:600)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6803) Documenting DFSClient#DFSInputStream expectations reading and preading in concurrent context
[ https://issues.apache.org/jira/browse/HDFS-6803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226343#comment-14226343 ] stack commented on HDFS-6803: - [~stev...@iseran.com] Where do I send the $1.00? Documenting DFSClient#DFSInputStream expectations reading and preading in concurrent context Key: HDFS-6803 URL: https://issues.apache.org/jira/browse/HDFS-6803 Project: Hadoop HDFS Issue Type: Sub-task Components: hdfs-client Affects Versions: 2.4.1 Reporter: stack Assignee: stack Fix For: 2.7.0 Attachments: 9117.md.txt, DocumentingDFSClientDFSInputStream (1).pdf, DocumentingDFSClientDFSInputStream.v2.pdf, HDFS-6803v2.txt, HDFS-6803v3.txt, fsdatainputstream.md.v3.html Reviews of the patch posted the parent task suggest that we be more explicit about how DFSIS is expected to behave when being read by contending threads. It is also suggested that presumptions made internally be made explicit documenting expectations. Before we put up a patch we've made a document of assertions we'd like to make into tenets of DFSInputSteam. If agreement, we'll attach to this issue a patch that weaves the assumptions into DFSIS as javadoc and class comments. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6803) Documenting DFSClient#DFSInputStream expectations reading and preading in concurrent context
[ https://issues.apache.org/jira/browse/HDFS-6803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226405#comment-14226405 ] Steve Loughran commented on HDFS-6803: -- hang on to it until I need you to vote up something of mine Documenting DFSClient#DFSInputStream expectations reading and preading in concurrent context Key: HDFS-6803 URL: https://issues.apache.org/jira/browse/HDFS-6803 Project: Hadoop HDFS Issue Type: Sub-task Components: hdfs-client Affects Versions: 2.4.1 Reporter: stack Assignee: stack Fix For: 2.7.0 Attachments: 9117.md.txt, DocumentingDFSClientDFSInputStream (1).pdf, DocumentingDFSClientDFSInputStream.v2.pdf, HDFS-6803v2.txt, HDFS-6803v3.txt, fsdatainputstream.md.v3.html Reviews of the patch posted the parent task suggest that we be more explicit about how DFSIS is expected to behave when being read by contending threads. It is also suggested that presumptions made internally be made explicit documenting expectations. Before we put up a patch we've made a document of assertions we'd like to make into tenets of DFSInputSteam. If agreement, we'll attach to this issue a patch that weaves the assumptions into DFSIS as javadoc and class comments. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HDFS-7448) TestBookKeeperHACheckpoints fails in trunk build
[
https://issues.apache.org/jira/browse/HDFS-7448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA reassigned HDFS-7448:
---
Assignee: Akira AJISAKA
TestBookKeeperHACheckpoints fails in trunk build
Key: HDFS-7448
URL: https://issues.apache.org/jira/browse/HDFS-7448
Project: Hadoop HDFS
Issue Type: Test
Reporter: Ted Yu
Assignee: Akira AJISAKA
Priority: Minor
The test failed against both java 7 and java 8.
From https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/17/console :
{code}
testStandbyExceptionThrownDuringCheckpoint(org.apache.hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints)
Time elapsed: 6.822 sec ERROR!
org.apache.hadoop.ipc.RemoteException: File /testFile could only be
replicated to 0 nodes instead of minReplication (=1). There are 0
datanode(s) running and no node(s) are excluded in this operation.
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1558)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3024)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:699)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:482)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:637)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:966)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2125)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2121)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1683)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2119)
at org.apache.hadoop.ipc.Client.call(Client.java:1468)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
at com.sun.proxy.$Proxy20.addBlock(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:399)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:101)
at com.sun.proxy.$Proxy21.addBlock(Unknown Source)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1544)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1361)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:600)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7097) Allow block reports to be processed during checkpointing on standby name node
[
https://issues.apache.org/jira/browse/HDFS-7097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226450#comment-14226450
]
Akira AJISAKA commented on HDFS-7097:
-
The patch was committed to branch-2 and trunk. Updating the status.
Allow block reports to be processed during checkpointing on standby name node
-
Key: HDFS-7097
URL: https://issues.apache.org/jira/browse/HDFS-7097
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Kihwal Lee
Assignee: Kihwal Lee
Priority: Critical
Fix For: 2.7.0
Attachments: HDFS-7097.patch, HDFS-7097.patch, HDFS-7097.patch,
HDFS-7097.patch, HDFS-7097.ultimate.trunk.patch
On a reasonably busy HDFS cluster, there are stream of creates, causing data
nodes to generate incremental block reports. When a standby name node is
checkpointing, RPC handler threads trying to process a full or incremental
block report is blocked on the name system's {{fsLock}}, because the
checkpointer acquires the read lock on it. This can create a serious problem
if the size of name space is big and checkpointing takes a long time.
All available RPC handlers can be tied up very quickly. If you have 100
handlers, it only takes 34 file creates. If a separate service RPC port is
not used, HA transition will have to wait in the call queue for minutes. Even
if a separate service RPC port is configured, hearbeats from datanodes will
be blocked. A standby NN with a big name space can lose all data nodes after
checkpointing. The rpc calls will also be retransmitted by data nodes many
times, filling up the call queue and potentially causing listen queue
overflow.
Since block reports are not modifying any state that is being saved to
fsimage, I propose letting them through during checkpointing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7097) Allow block reports to be processed during checkpointing on standby name node
[
https://issues.apache.org/jira/browse/HDFS-7097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HDFS-7097:
Resolution: Fixed
Fix Version/s: 2.7.0
Status: Resolved (was: Patch Available)
Allow block reports to be processed during checkpointing on standby name node
-
Key: HDFS-7097
URL: https://issues.apache.org/jira/browse/HDFS-7097
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Kihwal Lee
Assignee: Kihwal Lee
Priority: Critical
Fix For: 2.7.0
Attachments: HDFS-7097.patch, HDFS-7097.patch, HDFS-7097.patch,
HDFS-7097.patch, HDFS-7097.ultimate.trunk.patch
On a reasonably busy HDFS cluster, there are stream of creates, causing data
nodes to generate incremental block reports. When a standby name node is
checkpointing, RPC handler threads trying to process a full or incremental
block report is blocked on the name system's {{fsLock}}, because the
checkpointer acquires the read lock on it. This can create a serious problem
if the size of name space is big and checkpointing takes a long time.
All available RPC handlers can be tied up very quickly. If you have 100
handlers, it only takes 34 file creates. If a separate service RPC port is
not used, HA transition will have to wait in the call queue for minutes. Even
if a separate service RPC port is configured, hearbeats from datanodes will
be blocked. A standby NN with a big name space can lose all data nodes after
checkpointing. The rpc calls will also be retransmitted by data nodes many
times, filling up the call queue and potentially causing listen queue
overflow.
Since block reports are not modifying any state that is being saved to
fsimage, I propose letting them through during checkpointing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7448) TestBookKeeperHACheckpoints fails in trunk build
[
https://issues.apache.org/jira/browse/HDFS-7448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HDFS-7448:
Attachment: HDFS-7448-001.patch
Attaching a simple patch to change the number of DNs.
TestBookKeeperHACheckpoints fails in trunk build
Key: HDFS-7448
URL: https://issues.apache.org/jira/browse/HDFS-7448
Project: Hadoop HDFS
Issue Type: Test
Reporter: Ted Yu
Assignee: Akira AJISAKA
Priority: Minor
Attachments: HDFS-7448-001.patch
The test failed against both java 7 and java 8.
From https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/17/console :
{code}
testStandbyExceptionThrownDuringCheckpoint(org.apache.hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints)
Time elapsed: 6.822 sec ERROR!
org.apache.hadoop.ipc.RemoteException: File /testFile could only be
replicated to 0 nodes instead of minReplication (=1). There are 0
datanode(s) running and no node(s) are excluded in this operation.
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1558)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3024)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:699)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:482)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:637)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:966)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2125)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2121)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1683)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2119)
at org.apache.hadoop.ipc.Client.call(Client.java:1468)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
at com.sun.proxy.$Proxy20.addBlock(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:399)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:101)
at com.sun.proxy.$Proxy21.addBlock(Unknown Source)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1544)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1361)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:600)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7448) TestBookKeeperHACheckpoints fails in trunk build
[
https://issues.apache.org/jira/browse/HDFS-7448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HDFS-7448:
Target Version/s: 2.7.0
Status: Patch Available (was: Open)
TestBookKeeperHACheckpoints fails in trunk build
Key: HDFS-7448
URL: https://issues.apache.org/jira/browse/HDFS-7448
Project: Hadoop HDFS
Issue Type: Test
Reporter: Ted Yu
Assignee: Akira AJISAKA
Priority: Minor
Attachments: HDFS-7448-001.patch
The test failed against both java 7 and java 8.
From https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/17/console :
{code}
testStandbyExceptionThrownDuringCheckpoint(org.apache.hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints)
Time elapsed: 6.822 sec ERROR!
org.apache.hadoop.ipc.RemoteException: File /testFile could only be
replicated to 0 nodes instead of minReplication (=1). There are 0
datanode(s) running and no node(s) are excluded in this operation.
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1558)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3024)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:699)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:482)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:637)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:966)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2125)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2121)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1683)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2119)
at org.apache.hadoop.ipc.Client.call(Client.java:1468)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
at com.sun.proxy.$Proxy20.addBlock(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:399)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:101)
at com.sun.proxy.$Proxy21.addBlock(Unknown Source)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1544)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1361)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:600)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7448) TestBookKeeperHACheckpoints fails in trunk build
[
https://issues.apache.org/jira/browse/HDFS-7448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226460#comment-14226460
]
Akira AJISAKA commented on HDFS-7448:
-
Thanks for the report, [~tedyu].
TestBookKeeperHACheckpoints fails in trunk build
Key: HDFS-7448
URL: https://issues.apache.org/jira/browse/HDFS-7448
Project: Hadoop HDFS
Issue Type: Test
Reporter: Ted Yu
Assignee: Akira AJISAKA
Priority: Minor
Attachments: HDFS-7448-001.patch
The test failed against both java 7 and java 8.
From https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/17/console :
{code}
testStandbyExceptionThrownDuringCheckpoint(org.apache.hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints)
Time elapsed: 6.822 sec ERROR!
org.apache.hadoop.ipc.RemoteException: File /testFile could only be
replicated to 0 nodes instead of minReplication (=1). There are 0
datanode(s) running and no node(s) are excluded in this operation.
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1558)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3024)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:699)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:482)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:637)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:966)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2125)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2121)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1683)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2119)
at org.apache.hadoop.ipc.Client.call(Client.java:1468)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
at com.sun.proxy.$Proxy20.addBlock(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:399)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:101)
at com.sun.proxy.$Proxy21.addBlock(Unknown Source)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1544)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1361)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:600)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7448) TestBookKeeperHACheckpoints fails in trunk build
[
https://issues.apache.org/jira/browse/HDFS-7448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226524#comment-14226524
]
Hadoop QA commented on HDFS-7448:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12683870/HDFS-7448-001.patch
against trunk revision aa7dac3.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8846//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8846//console
This message is automatically generated.
TestBookKeeperHACheckpoints fails in trunk build
Key: HDFS-7448
URL: https://issues.apache.org/jira/browse/HDFS-7448
Project: Hadoop HDFS
Issue Type: Test
Reporter: Ted Yu
Assignee: Akira AJISAKA
Priority: Minor
Attachments: HDFS-7448-001.patch
The test failed against both java 7 and java 8.
From https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/17/console :
{code}
testStandbyExceptionThrownDuringCheckpoint(org.apache.hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints)
Time elapsed: 6.822 sec ERROR!
org.apache.hadoop.ipc.RemoteException: File /testFile could only be
replicated to 0 nodes instead of minReplication (=1). There are 0
datanode(s) running and no node(s) are excluded in this operation.
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1558)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3024)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:699)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:482)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:637)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:966)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2125)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2121)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1683)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2119)
at org.apache.hadoop.ipc.Client.call(Client.java:1468)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
at com.sun.proxy.$Proxy20.addBlock(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:399)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:101)
at com.sun.proxy.$Proxy21.addBlock(Unknown Source)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1544)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1361)
at
[jira] [Commented] (HDFS-7310) Mover can give first priority to local DN if it has target storage type available in local DN
[ https://issues.apache.org/jira/browse/HDFS-7310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226529#comment-14226529 ] Uma Maheswara Rao G commented on HDFS-7310: --- Thanks a lot Vinay for the update. Latest patch looks good to me. +1 Will commit the patch shortly. Mover can give first priority to local DN if it has target storage type available in local DN - Key: HDFS-7310 URL: https://issues.apache.org/jira/browse/HDFS-7310 Project: Hadoop HDFS Issue Type: Improvement Components: balancer mover Affects Versions: 3.0.0 Reporter: Uma Maheswara Rao G Assignee: Vinayakumar B Attachments: HDFS-7310-001.patch, HDFS-7310-002.patch, HDFS-7310-003.patch, HDFS-7310-004.patch Currently Mover logic may move blocks to any DN which had target storage type. But if the src DN has target storage type then mover can give highest priority to local DN. If local DN does not contains target storage type, then it can assign to any DN as the current logic does. This is a thought, have not go through the code fully yet. Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7431) log message for InvalidMagicNumberException may be incorrect
[
https://issues.apache.org/jira/browse/HDFS-7431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226538#comment-14226538
]
Chris Nauroth commented on HDFS-7431:
-
Hello, Yi. Thank you for investigating this and posting a patch. I have a
possible idea for distinguishing the 2 cases. We throw
{{InvalidMagicNumberException}} from
{{SaslDataTransferServer#doSaslHandshake}}. Within this method, we have the
information we need to distinguish between the 2 cases:
* {{if (dnConf.getEncryptDataTransfer())}}, then it's the encrypted case.
* {{if (dnConf.getSaslPropsResolver() != null)}}, then it's the data transfer
protection case.
After checking that, we could throw exceptions with different messages
depending on the case. This could either be done with 2 distinct subclasses of
{{InvalidMagicNumberException}} or adding some kind of type tag as a member.
For the text of the messages, I suggest:
{code}
LOG.info(Failed to read expected encryption handshake from client +
at + peer.getRemoteAddressString() + . Perhaps the client +
is running an older version of Hadoop which does not support +
encryption);
{code}
{code}
LOG.info(Failed to read expected SASL data transfer protection
handshake from client +
at + peer.getRemoteAddressString() + . Perhaps the client +
is running an older version of Hadoop which does not support +
encryption);
{code}
What are your thoughts on this?
log message for InvalidMagicNumberException may be incorrect
Key: HDFS-7431
URL: https://issues.apache.org/jira/browse/HDFS-7431
Project: Hadoop HDFS
Issue Type: Bug
Components: security
Reporter: Yi Liu
Assignee: Yi Liu
Priority: Minor
Attachments: HDFS-7431.001.patch
For security mode, HDFS now supports that Datanodes don't require root or
jsvc if {{dfs.data.transfer.protection}} is configured.
Log message for {{InvalidMagicNumberException}}, we miss one case:
when the datanodes run on unprivileged port and
{{dfs.data.transfer.protection}} is configured to {{authentication}} but
{{dfs.encrypt.data.transfer}} is not configured. SASL handshake is required
and a low version dfs client is used, then {{InvalidMagicNumberException}} is
thrown and we write log:
{quote}
Failed to read expected encryption handshake from client at Perhaps the
client is running an older version of Hadoop which does not support encryption
{quote}
Recently I run HDFS built on trunk and security is enabled, but the client is
2.5.1 version. Then I got the above log message, but actually I have not
configured encryption.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7310) Mover can give first priority to local DN if it has target storage type available in local DN
[ https://issues.apache.org/jira/browse/HDFS-7310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226556#comment-14226556 ] Hudson commented on HDFS-7310: -- FAILURE: Integrated in Hadoop-trunk-Commit #6610 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6610/]) HDFS-7310. Mover can give first priority to local DN if it has target storage type available in local DN. (Vinayakumar B via umamahesh) (umamahesh: rev 058af60c56207907f2bedf76df4284e86d923e0c) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockReplacement.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeDescriptor.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfo.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/RamDiskAsyncLazyPersistService.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java Mover can give first priority to local DN if it has target storage type available in local DN - Key: HDFS-7310 URL: https://issues.apache.org/jira/browse/HDFS-7310 Project: Hadoop HDFS Issue Type: Improvement Components: balancer mover Affects Versions: 3.0.0 Reporter: Uma Maheswara Rao G Assignee: Vinayakumar B Attachments: HDFS-7310-001.patch, HDFS-7310-002.patch, HDFS-7310-003.patch, HDFS-7310-004.patch Currently Mover logic may move blocks to any DN which had target storage type. But if the src DN has target storage type then mover can give highest priority to local DN. If local DN does not contains target storage type, then it can assign to any DN as the current logic does. This is a thought, have not go through the code fully yet. Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7310) Mover can give first priority to local DN if it has target storage type available in local DN
[ https://issues.apache.org/jira/browse/HDFS-7310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-7310: -- Resolution: Fixed Fix Version/s: 2.7.0 3.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I have just committed this to trunk and branch-2 Mover can give first priority to local DN if it has target storage type available in local DN - Key: HDFS-7310 URL: https://issues.apache.org/jira/browse/HDFS-7310 Project: Hadoop HDFS Issue Type: Improvement Components: balancer mover Affects Versions: 3.0.0 Reporter: Uma Maheswara Rao G Assignee: Vinayakumar B Fix For: 3.0.0, 2.7.0 Attachments: HDFS-7310-001.patch, HDFS-7310-002.patch, HDFS-7310-003.patch, HDFS-7310-004.patch Currently Mover logic may move blocks to any DN which had target storage type. But if the src DN has target storage type then mover can give highest priority to local DN. If local DN does not contains target storage type, then it can assign to any DN as the current logic does. This is a thought, have not go through the code fully yet. Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7435) PB encoding of block reports is very inefficient
[
https://issues.apache.org/jira/browse/HDFS-7435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226567#comment-14226567
]
Jing Zhao commented on HDFS-7435:
-
In the current demo patch the max chunk size is not encoded in the buffer. It
is currently simply determined by the DataNode based on configuration. The
segmentation is done by the DataNode and NameNode does not need to know the max
chunk size. For simplicity each chunk still follows the same format with the
original long[] in BlockListAsLongs (i.e., it still encodes the number of
finalized blocks and number of uc replicas in the first two elements).
I guess to let DataNode be the only side doing segmentation can avoid NameNode
still allocating a big contiguous array before chunking. Thus I have to change
the {{optional bytes blocksBuffer}} into {{repeated bytes blocksBuffers}}.
Maybe we can use {{repeated bytes blocksBuffers}} here but assume the number of
buffer is always 1, then move the real segmentation change into a separate jira?
PB encoding of block reports is very inefficient
Key: HDFS-7435
URL: https://issues.apache.org/jira/browse/HDFS-7435
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode, namenode
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Priority: Critical
Attachments: HDFS-7435.000.patch, HDFS-7435.patch
Block reports are encoded as a PB repeating long. Repeating fields use an
{{ArrayList}} with default capacity of 10. A block report containing tens or
hundreds of thousand of longs (3 for each replica) is extremely expensive
since the {{ArrayList}} must realloc many times. Also, decoding repeating
fields will box the primitive longs which must then be unboxed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7444) convertToBlockUnderConstruction should preserve BlockCollection
[
https://issues.apache.org/jira/browse/HDFS-7444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226584#comment-14226584
]
Jing Zhao commented on HDFS-7444:
-
+1. Thanks for the fix, [~wheat9]!
convertToBlockUnderConstruction should preserve BlockCollection
---
Key: HDFS-7444
URL: https://issues.apache.org/jira/browse/HDFS-7444
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Haohui Mai
Assignee: Haohui Mai
Attachments: HDFS-7444.000.patch, HDFS-7444.001.patch
{{BlockInfo#convertToBlockUnderConstruction}} converts a {{BlockInfo}} object
to a {{BlockInfoUnderConstruction}} object. The callee instead of the caller
should preserve the {{BlockCollection}} field.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7438) Consolidate implementation of rename()
[
https://issues.apache.org/jira/browse/HDFS-7438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-7438:
-
Attachment: HDFS-7438.002.patch
Consolidate implementation of rename()
--
Key: HDFS-7438
URL: https://issues.apache.org/jira/browse/HDFS-7438
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Haohui Mai
Attachments: HDFS-7438.000.patch, HDFS-7438.001.patch,
HDFS-7438.002.patch
The implementation of {{rename()}} resides in both {{FSNameSystem}} and
{{FSDirectory}}. This jira proposes to consolidate them in a single class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7437) Storing block ids instead of BlockInfo object in INodeFile
[
https://issues.apache.org/jira/browse/HDFS-7437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-7437:
-
Attachment: HDFS-7437.001.patch
Storing block ids instead of BlockInfo object in INodeFile
--
Key: HDFS-7437
URL: https://issues.apache.org/jira/browse/HDFS-7437
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Haohui Mai
Assignee: Haohui Mai
Attachments: HDFS-7437.000.patch, HDFS-7437.001.patch
Currently {{INodeFile}} stores the lists of blocks as references of
{{BlockInfo}} instead of the block ids. This creates implicit dependency
between the namespace and the block manager.
The dependency blocks several recent efforts, such as separating the block
manager out as a standalone service, moving block information off heap, and
optimizing the memory usage of block manager.
This jira proposes to decouple the dependency by storing block ids instead of
object reference in {{INodeFile}} objects.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7444) convertToBlockUnderConstruction should preserve BlockCollection
[
https://issues.apache.org/jira/browse/HDFS-7444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226640#comment-14226640
]
Hudson commented on HDFS-7444:
--
FAILURE: Integrated in Hadoop-trunk-Commit #6611 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/6611/])
HDFS-7444. convertToBlockUnderConstruction should preserve BlockCollection.
Contributed by Haohui Mai. (wheat9: rev
f5b163117986886eaba8a0cc255ec741dd14c4c6)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java
convertToBlockUnderConstruction should preserve BlockCollection
---
Key: HDFS-7444
URL: https://issues.apache.org/jira/browse/HDFS-7444
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Haohui Mai
Assignee: Haohui Mai
Attachments: HDFS-7444.000.patch, HDFS-7444.001.patch
{{BlockInfo#convertToBlockUnderConstruction}} converts a {{BlockInfo}} object
to a {{BlockInfoUnderConstruction}} object. The callee instead of the caller
should preserve the {{BlockCollection}} field.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7437) Storing block ids instead of BlockInfo object in INodeFile
[
https://issues.apache.org/jira/browse/HDFS-7437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226641#comment-14226641
]
Hadoop QA commented on HDFS-7437:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12683891/HDFS-7437.001.patch
against trunk revision f5b1631.
{color:red}-1 patch{color}. Trunk compilation may be broken.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8849//console
This message is automatically generated.
Storing block ids instead of BlockInfo object in INodeFile
--
Key: HDFS-7437
URL: https://issues.apache.org/jira/browse/HDFS-7437
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Haohui Mai
Assignee: Haohui Mai
Attachments: HDFS-7437.000.patch, HDFS-7437.001.patch
Currently {{INodeFile}} stores the lists of blocks as references of
{{BlockInfo}} instead of the block ids. This creates implicit dependency
between the namespace and the block manager.
The dependency blocks several recent efforts, such as separating the block
manager out as a standalone service, moving block information off heap, and
optimizing the memory usage of block manager.
This jira proposes to decouple the dependency by storing block ids instead of
object reference in {{INodeFile}} objects.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7437) Storing block ids instead of BlockInfo object in INodeFile
[
https://issues.apache.org/jira/browse/HDFS-7437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-7437:
-
Resolution: Fixed
Fix Version/s: 2.7.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I've committed the patch to trunk and branch-2. Thanks [~jingzhao] for the
reviews.
Storing block ids instead of BlockInfo object in INodeFile
--
Key: HDFS-7437
URL: https://issues.apache.org/jira/browse/HDFS-7437
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Haohui Mai
Assignee: Haohui Mai
Fix For: 2.7.0
Attachments: HDFS-7437.000.patch, HDFS-7437.001.patch
Currently {{INodeFile}} stores the lists of blocks as references of
{{BlockInfo}} instead of the block ids. This creates implicit dependency
between the namespace and the block manager.
The dependency blocks several recent efforts, such as separating the block
manager out as a standalone service, moving block information off heap, and
optimizing the memory usage of block manager.
This jira proposes to decouple the dependency by storing block ids instead of
object reference in {{INodeFile}} objects.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Reopened] (HDFS-7437) Storing block ids instead of BlockInfo object in INodeFile
[
https://issues.apache.org/jira/browse/HDFS-7437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai reopened HDFS-7437:
--
Reopen the issue, resolved the wrong jira.
Storing block ids instead of BlockInfo object in INodeFile
--
Key: HDFS-7437
URL: https://issues.apache.org/jira/browse/HDFS-7437
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Haohui Mai
Assignee: Haohui Mai
Fix For: 2.7.0
Attachments: HDFS-7437.000.patch, HDFS-7437.001.patch
Currently {{INodeFile}} stores the lists of blocks as references of
{{BlockInfo}} instead of the block ids. This creates implicit dependency
between the namespace and the block manager.
The dependency blocks several recent efforts, such as separating the block
manager out as a standalone service, moving block information off heap, and
optimizing the memory usage of block manager.
This jira proposes to decouple the dependency by storing block ids instead of
object reference in {{INodeFile}} objects.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Issue Comment Deleted] (HDFS-7437) Storing block ids instead of BlockInfo object in INodeFile
[
https://issues.apache.org/jira/browse/HDFS-7437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-7437:
-
Comment: was deleted
(was: I've committed the patch to trunk and branch-2. Thanks [~jingzhao] for
the reviews.)
Storing block ids instead of BlockInfo object in INodeFile
--
Key: HDFS-7437
URL: https://issues.apache.org/jira/browse/HDFS-7437
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Haohui Mai
Assignee: Haohui Mai
Fix For: 2.7.0
Attachments: HDFS-7437.000.patch, HDFS-7437.001.patch
Currently {{INodeFile}} stores the lists of blocks as references of
{{BlockInfo}} instead of the block ids. This creates implicit dependency
between the namespace and the block manager.
The dependency blocks several recent efforts, such as separating the block
manager out as a standalone service, moving block information off heap, and
optimizing the memory usage of block manager.
This jira proposes to decouple the dependency by storing block ids instead of
object reference in {{INodeFile}} objects.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7444) convertToBlockUnderConstruction should preserve BlockCollection
[
https://issues.apache.org/jira/browse/HDFS-7444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-7444:
-
Resolution: Fixed
Fix Version/s: 2.7.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I've committed the patch to trunk and branch-2. Thanks [~jingzhao] for the
reviews
convertToBlockUnderConstruction should preserve BlockCollection
---
Key: HDFS-7444
URL: https://issues.apache.org/jira/browse/HDFS-7444
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Haohui Mai
Assignee: Haohui Mai
Fix For: 2.7.0
Attachments: HDFS-7444.000.patch, HDFS-7444.001.patch
{{BlockInfo#convertToBlockUnderConstruction}} converts a {{BlockInfo}} object
to a {{BlockInfoUnderConstruction}} object. The callee instead of the caller
should preserve the {{BlockCollection}} field.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7437) Storing block ids instead of BlockInfo object in INodeFile
[
https://issues.apache.org/jira/browse/HDFS-7437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-7437:
-
Fix Version/s: (was: 2.7.0)
Hadoop Flags: (was: Reviewed)
Status: Patch Available (was: Reopened)
Storing block ids instead of BlockInfo object in INodeFile
--
Key: HDFS-7437
URL: https://issues.apache.org/jira/browse/HDFS-7437
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Haohui Mai
Assignee: Haohui Mai
Attachments: HDFS-7437.000.patch, HDFS-7437.001.patch
Currently {{INodeFile}} stores the lists of blocks as references of
{{BlockInfo}} instead of the block ids. This creates implicit dependency
between the namespace and the block manager.
The dependency blocks several recent efforts, such as separating the block
manager out as a standalone service, moving block information off heap, and
optimizing the memory usage of block manager.
This jira proposes to decouple the dependency by storing block ids instead of
object reference in {{INodeFile}} objects.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7424) Add web UI for NFS gateway
[ https://issues.apache.org/jira/browse/HDFS-7424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-7424: - Issue Type: New Feature (was: Improvement) Add web UI for NFS gateway -- Key: HDFS-7424 URL: https://issues.apache.org/jira/browse/HDFS-7424 Project: Hadoop HDFS Issue Type: New Feature Components: nfs Affects Versions: 2.2.0 Reporter: Brandon Li Assignee: Brandon Li Attachments: HDFS-7424.001.patch, HDFS-7424.002.patch This JIRA is to track the effort to add web UI for NFS gateway to show some metrics and configuration related information. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7449) Add metrics to NFS gateway
Brandon Li created HDFS-7449: Summary: Add metrics to NFS gateway Key: HDFS-7449 URL: https://issues.apache.org/jira/browse/HDFS-7449 Project: Hadoop HDFS Issue Type: New Feature Components: nfs Affects Versions: 2.7.0 Reporter: Brandon Li Assignee: Brandon Li -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7444) convertToBlockUnderConstruction should preserve BlockCollection
[
https://issues.apache.org/jira/browse/HDFS-7444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226657#comment-14226657
]
Hudson commented on HDFS-7444:
--
SUCCESS: Integrated in Hadoop-trunk-Commit #6612 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/6612/])
HDFS-7444. Addendum patch to resolve conflicts between HDFS-7444 and HDFS-7310.
(wheat9: rev 978736d486a8775bc21d074010f58c28ae0fda41)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
convertToBlockUnderConstruction should preserve BlockCollection
---
Key: HDFS-7444
URL: https://issues.apache.org/jira/browse/HDFS-7444
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Haohui Mai
Assignee: Haohui Mai
Fix For: 2.7.0
Attachments: HDFS-7444.000.patch, HDFS-7444.001.patch
{{BlockInfo#convertToBlockUnderConstruction}} converts a {{BlockInfo}} object
to a {{BlockInfoUnderConstruction}} object. The callee instead of the caller
should preserve the {{BlockCollection}} field.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7310) Mover can give first priority to local DN if it has target storage type available in local DN
[ https://issues.apache.org/jira/browse/HDFS-7310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226658#comment-14226658 ] Hudson commented on HDFS-7310: -- SUCCESS: Integrated in Hadoop-trunk-Commit #6612 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6612/]) HDFS-7444. Addendum patch to resolve conflicts between HDFS-7444 and HDFS-7310. (wheat9: rev 978736d486a8775bc21d074010f58c28ae0fda41) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java Mover can give first priority to local DN if it has target storage type available in local DN - Key: HDFS-7310 URL: https://issues.apache.org/jira/browse/HDFS-7310 Project: Hadoop HDFS Issue Type: Improvement Components: balancer mover Affects Versions: 3.0.0 Reporter: Uma Maheswara Rao G Assignee: Vinayakumar B Fix For: 3.0.0, 2.7.0 Attachments: HDFS-7310-001.patch, HDFS-7310-002.patch, HDFS-7310-003.patch, HDFS-7310-004.patch Currently Mover logic may move blocks to any DN which had target storage type. But if the src DN has target storage type then mover can give highest priority to local DN. If local DN does not contains target storage type, then it can assign to any DN as the current logic does. This is a thought, have not go through the code fully yet. Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7310) Mover can give first priority to local DN if it has target storage type available in local DN
[ https://issues.apache.org/jira/browse/HDFS-7310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226665#comment-14226665 ] Jing Zhao commented on HDFS-7310: - Oops sorry I missed this jira... Thanks Uma for the review and Vinay for the work! Mover can give first priority to local DN if it has target storage type available in local DN - Key: HDFS-7310 URL: https://issues.apache.org/jira/browse/HDFS-7310 Project: Hadoop HDFS Issue Type: Improvement Components: balancer mover Affects Versions: 3.0.0 Reporter: Uma Maheswara Rao G Assignee: Vinayakumar B Fix For: 3.0.0, 2.7.0 Attachments: HDFS-7310-001.patch, HDFS-7310-002.patch, HDFS-7310-003.patch, HDFS-7310-004.patch Currently Mover logic may move blocks to any DN which had target storage type. But if the src DN has target storage type then mover can give highest priority to local DN. If local DN does not contains target storage type, then it can assign to any DN as the current logic does. This is a thought, have not go through the code fully yet. Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7023) use libexpat instead of libxml2 for libhdfs3
[
https://issues.apache.org/jira/browse/HDFS-7023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226705#comment-14226705
]
Colin Patrick McCabe commented on HDFS-7023:
bq. Seems that you forget to modify CMake file to add libexpat.
Good catch, I will take a look at that.
bq. I wonder, are there any interests on creating a configuration that does not
depend on XML parser at all? What we can do is to create a Options class which
captures all configuration parameters directly. The XML parser can set the
configuration parameters accordingly.
Hmm. Maybe I'm misunderstanding your comment, but I think this is already
implemented. The {{Config}} class doesn't depend on the XML parser, in my
patch... you can create a {{Config}} programmatically and set the values to
whatever you want. {{XmlConfigParser}} is just a helper class that just fills
in a {{Config}} class, by calling the normal setter methods. If we want to add
another helper class that fills it in by a different method later, we can.
use libexpat instead of libxml2 for libhdfs3
Key: HDFS-7023
URL: https://issues.apache.org/jira/browse/HDFS-7023
Project: Hadoop HDFS
Issue Type: Sub-task
Components: hdfs-client
Reporter: Zhanwei Wang
Assignee: Colin Patrick McCabe
Attachments: HDFS-7023.001.pnative.patch
As commented in HDFS-6994, libxml2 may has some thread safe issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7023) use libexpat instead of libxml2 for libhdfs3
[
https://issues.apache.org/jira/browse/HDFS-7023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HDFS-7023:
---
Attachment: HDFS-7023-pnative.002.patch
* Add {{FIND_PACKAGE}} for {{libexpat}} to {{CMakeLists.txt}}
use libexpat instead of libxml2 for libhdfs3
Key: HDFS-7023
URL: https://issues.apache.org/jira/browse/HDFS-7023
Project: Hadoop HDFS
Issue Type: Sub-task
Components: hdfs-client
Reporter: Zhanwei Wang
Assignee: Colin Patrick McCabe
Attachments: HDFS-7023-pnative.002.patch, HDFS-7023.001.pnative.patch
As commented in HDFS-6994, libxml2 may has some thread safe issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7446) HDFS inotify should have the ability to determine what txid it has read up to
[
https://issues.apache.org/jira/browse/HDFS-7446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226755#comment-14226755
]
Colin Patrick McCabe commented on HDFS-7446:
bq. It feels like we have a mismatch between the underlying data and our
objects. The need for the VHS-rewind in getTxidBatchSize is one example, what
we really want there is an iterator of EditEvents, with one EditEvents per txid
(name is just a suggestion).
Basically, the code in {{getTxidBatchSize}} is converting between what we get
back from the NameNode RPC (a bunch of Events, not grouped by txid) to what we
want to return (a batch of events that all have the same txid).
I guess we could do the conversion earlier, in
{{PBHelper#convert(GetEditsFromTxidResponseProto resp)}}. This would mean
pushing the batch concept into {{EventsList}}. I'm not sure there's much
benefit in this, though. {{EventsList}} is used on the {{NameNode}} as well,
and the NN doesn't care about grouping Events of the same txid into batches.
But we'd have to implement all that logic anyway if we went down this route.
(The NN has its own concept of batching... how many Events it sends back in a
single RPC... and of course it will not split a txid across RPCs. But that's
the only batching we get out of it.)
I admit that rewinding the iterator feels awkward. I was looking for a utility
function in {{Iterators}} or something that would help with this, but I didn't
find anything. Java doesn't have the concept of duplicating iterators that C++
does, so you can't just make a copy of the iterator and then push only that
copy forward. I wanted to avoid allocating an array until I knew what size I
wanted. I suppose I could create a {{LinkedList}} rather than an array, and
avoid the rewinding that way. That's not very efficient, though. One slightly
awkward function feels like an acceptable tradeoff for memory efficiency.
bq. The txid could also be moved into EditEvents which would also save some
bytes.
Most batches are small-- I think the median size will be a batch of size 1,
with maybe some outliers at 2, 3, or 4. I don't think there's a lot of memory
savings to be had by moving the txid into {{EditEvents}}.
A bigger issue is that we need (well maybe not need, but the code will be
really, really awkward if we don't) to add txid to the protobuf {{EventProto}}
structure in {{inotify.proto}}. This is because there can be gaps in the
txids returned by {{EventsListProto}}. Currently, all {{EventsListProto}} has
is firstTxid, lastTxid, and a list of Events. But you don't know what the txid
of any of those Events actually is... if firstTxid is 100 and lastTxid is 200,
and you have 50 events, where are they in that sequence? Who knows. And if
you add this to the PB, it's extremely awkward not to add it to the Java
{{Event}} class as well.
HDFS inotify should have the ability to determine what txid it has read up to
-
Key: HDFS-7446
URL: https://issues.apache.org/jira/browse/HDFS-7446
Project: Hadoop HDFS
Issue Type: Improvement
Components: dfsclient
Affects Versions: 2.6.0
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Attachments: HDFS-7446.001.patch
HDFS inotify should have the ability to determine what txid it has read up
to. This will allow users who want to avoid missing any events to record
this txid and use it to resume reading events at the spot they left off.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7342) Lease Recovery doesn't happen some times
[
https://issues.apache.org/jira/browse/HDFS-7342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226765#comment-14226765
]
Yongjun Zhang commented on HDFS-7342:
-
Hi [~vinayrpet],
Thanks for further explanation. I studied your latest two comments, and I agree
with your analysis: the case that penultimate block being in COMMITTEED sate
with minimal replication met may not exist. In other words, when penultimate
block is in COMMITTED state, the minimal replication is not met.
Going back to the source that triggerd this discussion, it's the scenario#1
described in:
https://issues.apache.org/jira/browse/HDFS-4882?focusedCommentId=14213992page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14213992
Here is what I think now:
scenario#1 exists for last block but not penultimate block
scenarion#2 exists for both penultimate and last block
My suggested solution for this jia is to share the same code that handles the
case when last block is COMMITTEED, with some modifications:
* For scenarion#1 (last block), my suggested solution is to forceComplete the
block before calling {{finalizeINodeFileUnderConstruction}}:
{code}
if (lastBlockState == BlockUCState.COMMITTED) {
getBlockManager().forceCompleteBlock(pendingFile,
(BlockInfoUnderConstruction) lastBlock);
}
finalizeINodeFileUnderConstruction(src, pendingFile,
iip.getLatestSnapshotId());
{code}
thus avoiding the exception thrown from {{finalizeINodeFileUnderConstruction}}
caused by
{code}
Preconditions.checkState(blocks[i].isComplete(), Failed to finalize
+ %s %s since blocks[%s] is non-complete, where blocks=%s.,
getClass().getSimpleName(), this, i, Arrays.asList(blocks));
{code}
* For scenarion#2, the same code can be shared to handle the case that when
penultimate block is COMMITTED with minimal replication NOT met. the lease will
be recovered, even if it's this kind of state.
On top of this suggested solution, we can add what Kihwal suggested.
What do you think?
Thanks.
Lease Recovery doesn't happen some times
Key: HDFS-7342
URL: https://issues.apache.org/jira/browse/HDFS-7342
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.0.0-alpha
Reporter: Ravi Prakash
Assignee: Ravi Prakash
Attachments: HDFS-7342.1.patch, HDFS-7342.2.patch, HDFS-7342.3.patch
In some cases, LeaseManager tries to recover a lease, but is not able to.
HDFS-4882 describes a possibility of that. We should fix this
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7384) 'getfacl' command and 'getAclStatus' output should be in sync
[
https://issues.apache.org/jira/browse/HDFS-7384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226786#comment-14226786
]
Chris Nauroth commented on HDFS-7384:
-
Hi [~vinayrpet]. Thank you for the follow-up. For this version of the patch,
I have one compatibility concern and a few minor nitpicks.
# {{AclStatus#getEffectivePermission}}: I think there is a compatibility
problem in this method. Let's assume this patch goes into 2.7.0, and then we
run a 2.7.0 client connected to a 2.6.0 NameNode. The old NameNode will not
populate the new permissions field in the outbound {{AclStatus}}. The 2.7.0
client would go into the null check path and not apply any mask, resulting in
{{hdfs dfs -getfacl}} reporting incorrect effective permissions. For
compatibility, I think the shell will need a way to detect that the NameNode
didn't populate permissions, and fall back to the current logic of using
permissions from {{FileStatus}}.
# {{AclStatus#getPermission}}: I suggest adding JavaDocs.
# {{AclStatus#Builder#setPermission}}: I suggest removing the word default
here, just to prevent any confusion that this is somehow related to default
ACLs. Same thing for the private {{AclStatus}} constructor.
# Let's update the documentation in WebHDFS.apt.vm to show the new fields in
the GETACLSTATUS example JSON response.
'getfacl' command and 'getAclStatus' output should be in sync
-
Key: HDFS-7384
URL: https://issues.apache.org/jira/browse/HDFS-7384
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Vinayakumar B
Assignee: Vinayakumar B
Attachments: HDFS-7384-001.patch, HDFS-7384-002.patch,
HDFS-7384-003.patch, HDFS-7384-004.patch, HDFS-7384-005.patch
*getfacl* command will print all the entries including basic and extended
entries, mask entries and effective permissions.
But, *getAclStatus* FileSystem API will return only extended ACL entries set
by the user. But this will not include the mask entry as well as effective
permissions.
To benefit the client using API, better to include 'mask' entry and effective
permissions in the return list of entries.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6735) A minor optimization to avoid pread() be blocked by read() inside the same DFSInputStream
[
https://issues.apache.org/jira/browse/HDFS-6735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226791#comment-14226791
]
Colin Patrick McCabe commented on HDFS-6735:
* {{readWithStrategy}} and {{blockSeekTo}} should be marked {{synchronized}}.
Yes, they are called from a {{synchronized}} function, but let's make it clear.
It's kind of confusing to see us fooling around with {{pos}} and other stuff
without seeing a {{synchronized}} on the function.
* We should document in a comment that we cannot try to take the DFSInputStream
lock when holding the infoLock. We need to be careful to avoid deadlock, and
maintaining this lock ordering is the easiest way.
* I noticed that in {{blockSeekTo}}, we are holding the {{infoLock}} when
calling {{BlockReaderFactory#build}}. It would be nice to avoid this. That
function does a lot of stuff... if we're creating a {{RemoteBlockReader2}}, it
potentially blocks while a TCP connection to the DataNode is opened. It seems
like all you need the {{infoLock}} for here is to get the {{cachingStrategy}}
and determine if {{shortCircuitForbidden}}, and you could pull this out into a
synchronized block prior to the {{Builder#build}} call, similar to how
{{actualGetFromOneDataNode}} does it.
Incidentally, the findbugs warning is probably because findbugs doesn't realize
that {{CachingStrategy}} is an immutable class, and so it's safe to access it
without locking. (The only thing you need locking for is actually reading the
current reference to the object, not for accessing the object itself.)
+1 once those are addressed
A minor optimization to avoid pread() be blocked by read() inside the same
DFSInputStream
-
Key: HDFS-6735
URL: https://issues.apache.org/jira/browse/HDFS-6735
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs-client
Affects Versions: 3.0.0
Reporter: Liang Xie
Assignee: Liang Xie
Attachments: HDFS-6735-v2.txt, HDFS-6735-v3.txt, HDFS-6735-v4.txt,
HDFS-6735-v5.txt, HDFS-6735-v6.txt, HDFS-6735.txt
In current DFSInputStream impl, there're a couple of coarser-grained locks in
read/pread path, and it has became a HBase read latency pain point so far. In
HDFS-6698, i made a minor patch against the first encourtered lock, around
getFileLength, in deed, after reading code and testing, it shows still other
locks we could improve.
In this jira, i'll make a patch against other locks, and a simple test case
to show the issue and the improved result.
This is important for HBase application, since in current HFile read path, we
issue all read()/pread() requests in the same DFSInputStream for one HFile.
(Multi streams solution is another story i had a plan to do, but probably
will take more time than i expected)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7435) PB encoding of block reports is very inefficient
[
https://issues.apache.org/jira/browse/HDFS-7435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-7435:
Attachment: HDFS-7435.001.patch
Fix a bug in the demo patch.
PB encoding of block reports is very inefficient
Key: HDFS-7435
URL: https://issues.apache.org/jira/browse/HDFS-7435
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode, namenode
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Priority: Critical
Attachments: HDFS-7435.000.patch, HDFS-7435.001.patch, HDFS-7435.patch
Block reports are encoded as a PB repeating long. Repeating fields use an
{{ArrayList}} with default capacity of 10. A block report containing tens or
hundreds of thousand of longs (3 for each replica) is extremely expensive
since the {{ArrayList}} must realloc many times. Also, decoding repeating
fields will box the primitive longs which must then be unboxed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7210) Avoid two separate RPC's namenode.append() and namenode.getFileInfo() for an append call from DFSClient
[
https://issues.apache.org/jira/browse/HDFS-7210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226841#comment-14226841
]
Hadoop QA commented on HDFS-7210:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12683498/HDFS-7210-005.patch
against trunk revision 058af60.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8847//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8847//console
This message is automatically generated.
Avoid two separate RPC's namenode.append() and namenode.getFileInfo() for an
append call from DFSClient
---
Key: HDFS-7210
URL: https://issues.apache.org/jira/browse/HDFS-7210
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs-client, namenode
Reporter: Vinayakumar B
Assignee: Vinayakumar B
Attachments: HDFS-7210-001.patch, HDFS-7210-002.patch,
HDFS-7210-003.patch, HDFS-7210-004.patch, HDFS-7210-005.patch
Currently DFSClient does 2 RPCs to namenode for an append operation.
{{append()}} for re-opening the file and getting the last block,
{{getFileInfo()}} Another on to get HdfsFileState
If we can combine result of these 2 calls and make one RPC, then it can
reduce load on NameNode.
For the backward compatibility we need to keep existing {{append()}} call as
is
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-7450) Consolidate GetFileInfo, GetListings and GetContentSummary into a single class
Haohui Mai created HDFS-7450:
Summary: Consolidate GetFileInfo, GetListings and
GetContentSummary into a single class
Key: HDFS-7450
URL: https://issues.apache.org/jira/browse/HDFS-7450
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Haohui Mai
This jira proposes to consolidate the implementation of {{GetFileInfo}},
{{GetListings}} and {{GetContentSummary}} into a single class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7450) Consolidate GetFileInfo, GetListings and GetContentSummary into a single class
[
https://issues.apache.org/jira/browse/HDFS-7450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-7450:
-
Status: Patch Available (was: Open)
Consolidate GetFileInfo, GetListings and GetContentSummary into a single class
--
Key: HDFS-7450
URL: https://issues.apache.org/jira/browse/HDFS-7450
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Haohui Mai
Attachments: HDFS-7450.000.patch
This jira proposes to consolidate the implementation of {{GetFileInfo}},
{{GetListings}} and {{GetContentSummary}} into a single class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7450) Consolidate GetFileInfo, GetListings and GetContentSummary into a single class
[
https://issues.apache.org/jira/browse/HDFS-7450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-7450:
-
Attachment: HDFS-7450.000.patch
Consolidate GetFileInfo, GetListings and GetContentSummary into a single class
--
Key: HDFS-7450
URL: https://issues.apache.org/jira/browse/HDFS-7450
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Haohui Mai
Attachments: HDFS-7450.000.patch
This jira proposes to consolidate the implementation of {{GetFileInfo}},
{{GetListings}} and {{GetContentSummary}} into a single class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7438) Consolidate implementation of rename()
[
https://issues.apache.org/jira/browse/HDFS-7438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226854#comment-14226854
]
Hadoop QA commented on HDFS-7438:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12683889/HDFS-7438.002.patch
against trunk revision 058af60.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8848//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/8848//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8848//console
This message is automatically generated.
Consolidate implementation of rename()
--
Key: HDFS-7438
URL: https://issues.apache.org/jira/browse/HDFS-7438
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Haohui Mai
Attachments: HDFS-7438.000.patch, HDFS-7438.001.patch,
HDFS-7438.002.patch
The implementation of {{rename()}} resides in both {{FSNameSystem}} and
{{FSDirectory}}. This jira proposes to consolidate them in a single class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7437) Storing block ids instead of BlockInfo object in INodeFile
[
https://issues.apache.org/jira/browse/HDFS-7437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226864#comment-14226864
]
Hadoop QA commented on HDFS-7437:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12683891/HDFS-7437.001.patch
against trunk revision 978736d.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8850//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8850//console
This message is automatically generated.
Storing block ids instead of BlockInfo object in INodeFile
--
Key: HDFS-7437
URL: https://issues.apache.org/jira/browse/HDFS-7437
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Haohui Mai
Assignee: Haohui Mai
Attachments: HDFS-7437.000.patch, HDFS-7437.001.patch
Currently {{INodeFile}} stores the lists of blocks as references of
{{BlockInfo}} instead of the block ids. This creates implicit dependency
between the namespace and the block manager.
The dependency blocks several recent efforts, such as separating the block
manager out as a standalone service, moving block information off heap, and
optimizing the memory usage of block manager.
This jira proposes to decouple the dependency by storing block ids instead of
object reference in {{INodeFile}} objects.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7450) Consolidate GetFileInfo, GetListings and GetContentSummary into a single class
[
https://issues.apache.org/jira/browse/HDFS-7450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-7450:
-
Attachment: HDFS-7450.001.patch
Consolidate GetFileInfo, GetListings and GetContentSummary into a single class
--
Key: HDFS-7450
URL: https://issues.apache.org/jira/browse/HDFS-7450
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Haohui Mai
Attachments: HDFS-7450.000.patch, HDFS-7450.001.patch
This jira proposes to consolidate the implementation of {{GetFileInfo}},
{{GetListings}} and {{GetContentSummary}} into a single class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7433) DatanodeMap lookups DatanodeID hashCodes are inefficient
[
https://issues.apache.org/jira/browse/HDFS-7433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226882#comment-14226882
]
Colin Patrick McCabe commented on HDFS-7433:
Can we separate out the {{DatanodeId#hashCode}} caching part of this into
another change? Although it's not a lot of lines of code, it feels kind of
tricky. And as I mentioned, it doesn't really relate to this, since we're
keying on Java's good old immutable String class here, not {{DatanodeId}}.
Otherwise, I'm +1 on the TreeMap - HashMap part.
DatanodeMap lookups DatanodeID hashCodes are inefficient
--
Key: HDFS-7433
URL: https://issues.apache.org/jira/browse/HDFS-7433
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Priority: Critical
Attachments: HDFS-7433.patch
The datanode map is currently a {{TreeMap}}. For many thousands of
datanodes, tree lookups are ~10X more expensive than a {{HashMap}}.
Insertions and removals are up to 100X more expensive.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6735) A minor optimization to avoid pread() be blocked by read() inside the same DFSInputStream
[ https://issues.apache.org/jira/browse/HDFS-6735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226896#comment-14226896 ] stack commented on HDFS-6735: - Interesting CachingStrategy can be changed on a DFSIS post-construction. Could avoid infolock on cachingstrategy if pre-made the readahead and dropbehinds but that's probably OTT. Nice doc'ing of locking strategy around data members. If we are doing a openInfo call, we can't service a filelength; I suppose thats how it should be; if we are updating our block info, file length could change and if updating block info, somethings up w/ the block set we currently have... At least the lock has climbed down from a lock on 'this'. Good. Patch LGTM (I like the Colin feedback above). Numbers still pretty good [~larsh]? A minor optimization to avoid pread() be blocked by read() inside the same DFSInputStream - Key: HDFS-6735 URL: https://issues.apache.org/jira/browse/HDFS-6735 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs-client Affects Versions: 3.0.0 Reporter: Liang Xie Assignee: Liang Xie Attachments: HDFS-6735-v2.txt, HDFS-6735-v3.txt, HDFS-6735-v4.txt, HDFS-6735-v5.txt, HDFS-6735-v6.txt, HDFS-6735.txt In current DFSInputStream impl, there're a couple of coarser-grained locks in read/pread path, and it has became a HBase read latency pain point so far. In HDFS-6698, i made a minor patch against the first encourtered lock, around getFileLength, in deed, after reading code and testing, it shows still other locks we could improve. In this jira, i'll make a patch against other locks, and a simple test case to show the issue and the improved result. This is important for HBase application, since in current HFile read path, we issue all read()/pread() requests in the same DFSInputStream for one HFile. (Multi streams solution is another story i had a plan to do, but probably will take more time than i expected) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7450) Consolidate GetFileInfo, GetListings and GetContentSummary into a single class
[
https://issues.apache.org/jira/browse/HDFS-7450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226901#comment-14226901
]
Hadoop QA commented on HDFS-7450:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12683928/HDFS-7450.001.patch
against trunk revision c1f2bb2.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8853//console
This message is automatically generated.
Consolidate GetFileInfo, GetListings and GetContentSummary into a single class
--
Key: HDFS-7450
URL: https://issues.apache.org/jira/browse/HDFS-7450
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Haohui Mai
Attachments: HDFS-7450.000.patch, HDFS-7450.001.patch
This jira proposes to consolidate the implementation of {{GetFileInfo}},
{{GetListings}} and {{GetContentSummary}} into a single class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7424) Add web UI for NFS gateway
[ https://issues.apache.org/jira/browse/HDFS-7424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226908#comment-14226908 ] Brandon Li commented on HDFS-7424: -- Uploaded a patch to address Haohui's comments. Add web UI for NFS gateway -- Key: HDFS-7424 URL: https://issues.apache.org/jira/browse/HDFS-7424 Project: Hadoop HDFS Issue Type: New Feature Components: nfs Affects Versions: 2.2.0 Reporter: Brandon Li Assignee: Brandon Li Attachments: HDFS-7424.001.patch, HDFS-7424.002.patch, HDFS-7424.003.patch This JIRA is to track the effort to add web UI for NFS gateway to show some metrics and configuration related information. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7424) Add web UI for NFS gateway
[ https://issues.apache.org/jira/browse/HDFS-7424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-7424: - Attachment: HDFS-7424.003.patch Add web UI for NFS gateway -- Key: HDFS-7424 URL: https://issues.apache.org/jira/browse/HDFS-7424 Project: Hadoop HDFS Issue Type: New Feature Components: nfs Affects Versions: 2.2.0 Reporter: Brandon Li Assignee: Brandon Li Attachments: HDFS-7424.001.patch, HDFS-7424.002.patch, HDFS-7424.003.patch This JIRA is to track the effort to add web UI for NFS gateway to show some metrics and configuration related information. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7424) Add web UI for NFS gateway
[
https://issues.apache.org/jira/browse/HDFS-7424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226912#comment-14226912
]
Haohui Mai commented on HDFS-7424:
--
{code}
+@InterfaceAudience.Private
{code}
This line can be removed since the class is a package-local class now. +1 after
addressing it.
Add web UI for NFS gateway
--
Key: HDFS-7424
URL: https://issues.apache.org/jira/browse/HDFS-7424
Project: Hadoop HDFS
Issue Type: New Feature
Components: nfs
Affects Versions: 2.2.0
Reporter: Brandon Li
Assignee: Brandon Li
Attachments: HDFS-7424.001.patch, HDFS-7424.002.patch,
HDFS-7424.003.patch
This JIRA is to track the effort to add web UI for NFS gateway to show some
metrics and configuration related information.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7438) Consolidate implementation of rename()
[
https://issues.apache.org/jira/browse/HDFS-7438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-7438:
-
Attachment: HDFS-7438.003.patch
Consolidate implementation of rename()
--
Key: HDFS-7438
URL: https://issues.apache.org/jira/browse/HDFS-7438
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Haohui Mai
Attachments: HDFS-7438.000.patch, HDFS-7438.001.patch,
HDFS-7438.002.patch, HDFS-7438.003.patch
The implementation of {{rename()}} resides in both {{FSNameSystem}} and
{{FSDirectory}}. This jira proposes to consolidate them in a single class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7435) PB encoding of block reports is very inefficient
[
https://issues.apache.org/jira/browse/HDFS-7435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226975#comment-14226975
]
Hadoop QA commented on HDFS-7435:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12683918/HDFS-7435.001.patch
against trunk revision 8f1454c.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 10 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestEncryptionZonesWithKMS
org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
The following test timeouts occurred in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestDFSClientRetries
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8851//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8851//console
This message is automatically generated.
PB encoding of block reports is very inefficient
Key: HDFS-7435
URL: https://issues.apache.org/jira/browse/HDFS-7435
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode, namenode
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Priority: Critical
Attachments: HDFS-7435.000.patch, HDFS-7435.001.patch, HDFS-7435.patch
Block reports are encoded as a PB repeating long. Repeating fields use an
{{ArrayList}} with default capacity of 10. A block report containing tens or
hundreds of thousand of longs (3 for each replica) is extremely expensive
since the {{ArrayList}} must realloc many times. Also, decoding repeating
fields will box the primitive longs which must then be unboxed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7450) Consolidate GetFileInfo, GetListings and GetContentSummary into a single class
[
https://issues.apache.org/jira/browse/HDFS-7450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-7450:
-
Attachment: HDFS-7450.002.patch
Consolidate GetFileInfo, GetListings and GetContentSummary into a single class
--
Key: HDFS-7450
URL: https://issues.apache.org/jira/browse/HDFS-7450
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Haohui Mai
Attachments: HDFS-7450.000.patch, HDFS-7450.001.patch,
HDFS-7450.002.patch
This jira proposes to consolidate the implementation of {{GetFileInfo}},
{{GetListings}} and {{GetContentSummary}} into a single class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7450) Consolidate GetFileInfo, GetListings and GetContentSummary into a single class
[
https://issues.apache.org/jira/browse/HDFS-7450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-7450:
-
Attachment: HDFS-7450.003.patch
Consolidate GetFileInfo, GetListings and GetContentSummary into a single class
--
Key: HDFS-7450
URL: https://issues.apache.org/jira/browse/HDFS-7450
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Haohui Mai
Attachments: HDFS-7450.000.patch, HDFS-7450.001.patch,
HDFS-7450.002.patch, HDFS-7450.003.patch
This jira proposes to consolidate the implementation of {{GetFileInfo}},
{{GetListings}} and {{GetContentSummary}} into a single class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6735) A minor optimization to avoid pread() be blocked by read() inside the same DFSInputStream
[ https://issues.apache.org/jira/browse/HDFS-6735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14227031#comment-14227031 ] Lars Hofhansl commented on HDFS-6735: - Per my comment above my preference would still be to just make the cachingStrategy reference volatile in DFSInputStream. It is immutable and hence the volatile reference would make access safe in all cases without any locking - the same is true for fileEncryptionInfo, btw (immutable already, just needs a volatile reference, no locking needed at all). I'll make a new patch. A minor optimization to avoid pread() be blocked by read() inside the same DFSInputStream - Key: HDFS-6735 URL: https://issues.apache.org/jira/browse/HDFS-6735 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs-client Affects Versions: 3.0.0 Reporter: Liang Xie Assignee: Liang Xie Attachments: HDFS-6735-v2.txt, HDFS-6735-v3.txt, HDFS-6735-v4.txt, HDFS-6735-v5.txt, HDFS-6735-v6.txt, HDFS-6735.txt In current DFSInputStream impl, there're a couple of coarser-grained locks in read/pread path, and it has became a HBase read latency pain point so far. In HDFS-6698, i made a minor patch against the first encourtered lock, around getFileLength, in deed, after reading code and testing, it shows still other locks we could improve. In this jira, i'll make a patch against other locks, and a simple test case to show the issue and the improved result. This is important for HBase application, since in current HFile read path, we issue all read()/pread() requests in the same DFSInputStream for one HFile. (Multi streams solution is another story i had a plan to do, but probably will take more time than i expected) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7424) Add web UI for NFS gateway
[ https://issues.apache.org/jira/browse/HDFS-7424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-7424: - Attachment: HDFS-7424.004.patch Add web UI for NFS gateway -- Key: HDFS-7424 URL: https://issues.apache.org/jira/browse/HDFS-7424 Project: Hadoop HDFS Issue Type: New Feature Components: nfs Affects Versions: 2.2.0 Reporter: Brandon Li Assignee: Brandon Li Attachments: HDFS-7424.001.patch, HDFS-7424.002.patch, HDFS-7424.003.patch, HDFS-7424.004.patch This JIRA is to track the effort to add web UI for NFS gateway to show some metrics and configuration related information. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-6735) A minor optimization to avoid pread() be blocked by read() inside the same DFSInputStream
[ https://issues.apache.org/jira/browse/HDFS-6735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HDFS-6735: Attachment: HDFS-6735-v6.txt Updated patch. The findbugs tweak is still necessary. Locking was correct before, findbugs does not seem to realize that all references to cachingStrategy is always guarded by the infoLock. I'll run a 2.4.1 version of this patch against HBase again. A minor optimization to avoid pread() be blocked by read() inside the same DFSInputStream - Key: HDFS-6735 URL: https://issues.apache.org/jira/browse/HDFS-6735 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs-client Affects Versions: 3.0.0 Reporter: Liang Xie Assignee: Liang Xie Attachments: HDFS-6735-v2.txt, HDFS-6735-v3.txt, HDFS-6735-v4.txt, HDFS-6735-v5.txt, HDFS-6735-v6.txt, HDFS-6735-v6.txt, HDFS-6735.txt In current DFSInputStream impl, there're a couple of coarser-grained locks in read/pread path, and it has became a HBase read latency pain point so far. In HDFS-6698, i made a minor patch against the first encourtered lock, around getFileLength, in deed, after reading code and testing, it shows still other locks we could improve. In this jira, i'll make a patch against other locks, and a simple test case to show the issue and the improved result. This is important for HBase application, since in current HFile read path, we issue all read()/pread() requests in the same DFSInputStream for one HFile. (Multi streams solution is another story i had a plan to do, but probably will take more time than i expected) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7424) Add web UI for NFS gateway
[
https://issues.apache.org/jira/browse/HDFS-7424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14227145#comment-14227145
]
Hadoop QA commented on HDFS-7424:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12683937/HDFS-7424.003.patch
against trunk revision c1f2bb2.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The following test timeouts occurred in
hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-nfs:
org.apache.hadoop.hdfs.server.namenode.TestBackupNode
org.apache.hadoop.hdfs.nfs.nfs3.TestNfs3HttpServer
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8854//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8854//console
This message is automatically generated.
Add web UI for NFS gateway
--
Key: HDFS-7424
URL: https://issues.apache.org/jira/browse/HDFS-7424
Project: Hadoop HDFS
Issue Type: New Feature
Components: nfs
Affects Versions: 2.2.0
Reporter: Brandon Li
Assignee: Brandon Li
Attachments: HDFS-7424.001.patch, HDFS-7424.002.patch,
HDFS-7424.003.patch, HDFS-7424.004.patch
This JIRA is to track the effort to add web UI for NFS gateway to show some
metrics and configuration related information.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6735) A minor optimization to avoid pread() be blocked by read() inside the same DFSInputStream
[ https://issues.apache.org/jira/browse/HDFS-6735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14227148#comment-14227148 ] Lars Hofhansl commented on HDFS-6735: - Tested -v6 with HBase. Still good from the DFSInputStream angle. I do see now that much more time is spent in ShortCircuitCache.fetchOrCreate and unref. (rechecked that is true to -v3 as well). It's still better, but the can is kicked down the road a bit. A minor optimization to avoid pread() be blocked by read() inside the same DFSInputStream - Key: HDFS-6735 URL: https://issues.apache.org/jira/browse/HDFS-6735 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs-client Affects Versions: 3.0.0 Reporter: Liang Xie Assignee: Liang Xie Attachments: HDFS-6735-v2.txt, HDFS-6735-v3.txt, HDFS-6735-v4.txt, HDFS-6735-v5.txt, HDFS-6735-v6.txt, HDFS-6735-v6.txt, HDFS-6735.txt In current DFSInputStream impl, there're a couple of coarser-grained locks in read/pread path, and it has became a HBase read latency pain point so far. In HDFS-6698, i made a minor patch against the first encourtered lock, around getFileLength, in deed, after reading code and testing, it shows still other locks we could improve. In this jira, i'll make a patch against other locks, and a simple test case to show the issue and the improved result. This is important for HBase application, since in current HFile read path, we issue all read()/pread() requests in the same DFSInputStream for one HFile. (Multi streams solution is another story i had a plan to do, but probably will take more time than i expected) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6735) A minor optimization to avoid pread() be blocked by read() inside the same DFSInputStream
[ https://issues.apache.org/jira/browse/HDFS-6735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14227155#comment-14227155 ] Lars Hofhansl commented on HDFS-6735: - So to be specific the improvement I see above is still there. Just that the next thing to tackle is the ShortCircuitCache. A minor optimization to avoid pread() be blocked by read() inside the same DFSInputStream - Key: HDFS-6735 URL: https://issues.apache.org/jira/browse/HDFS-6735 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs-client Affects Versions: 3.0.0 Reporter: Liang Xie Assignee: Liang Xie Attachments: HDFS-6735-v2.txt, HDFS-6735-v3.txt, HDFS-6735-v4.txt, HDFS-6735-v5.txt, HDFS-6735-v6.txt, HDFS-6735-v6.txt, HDFS-6735.txt In current DFSInputStream impl, there're a couple of coarser-grained locks in read/pread path, and it has became a HBase read latency pain point so far. In HDFS-6698, i made a minor patch against the first encourtered lock, around getFileLength, in deed, after reading code and testing, it shows still other locks we could improve. In this jira, i'll make a patch against other locks, and a simple test case to show the issue and the improved result. This is important for HBase application, since in current HFile read path, we issue all read()/pread() requests in the same DFSInputStream for one HFile. (Multi streams solution is another story i had a plan to do, but probably will take more time than i expected) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HDFS-6735) A minor optimization to avoid pread() be blocked by read() inside the same DFSInputStream
[ https://issues.apache.org/jira/browse/HDFS-6735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl reassigned HDFS-6735: --- Assignee: Lars Hofhansl (was: Liang Xie) A minor optimization to avoid pread() be blocked by read() inside the same DFSInputStream - Key: HDFS-6735 URL: https://issues.apache.org/jira/browse/HDFS-6735 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs-client Affects Versions: 3.0.0 Reporter: Liang Xie Assignee: Lars Hofhansl Attachments: HDFS-6735-v2.txt, HDFS-6735-v3.txt, HDFS-6735-v4.txt, HDFS-6735-v5.txt, HDFS-6735-v6.txt, HDFS-6735-v6.txt, HDFS-6735.txt In current DFSInputStream impl, there're a couple of coarser-grained locks in read/pread path, and it has became a HBase read latency pain point so far. In HDFS-6698, i made a minor patch against the first encourtered lock, around getFileLength, in deed, after reading code and testing, it shows still other locks we could improve. In this jira, i'll make a patch against other locks, and a simple test case to show the issue and the improved result. This is important for HBase application, since in current HFile read path, we issue all read()/pread() requests in the same DFSInputStream for one HFile. (Multi streams solution is another story i had a plan to do, but probably will take more time than i expected) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7438) Consolidate implementation of rename()
[
https://issues.apache.org/jira/browse/HDFS-7438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14227160#comment-14227160
]
Hadoop QA commented on HDFS-7438:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12683945/HDFS-7438.003.patch
against trunk revision c1f2bb2.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8856//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8856//console
This message is automatically generated.
Consolidate implementation of rename()
--
Key: HDFS-7438
URL: https://issues.apache.org/jira/browse/HDFS-7438
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Haohui Mai
Attachments: HDFS-7438.000.patch, HDFS-7438.001.patch,
HDFS-7438.002.patch, HDFS-7438.003.patch
The implementation of {{rename()}} resides in both {{FSNameSystem}} and
{{FSDirectory}}. This jira proposes to consolidate them in a single class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7450) Consolidate GetFileInfo, GetListings and GetContentSummary into a single class
[
https://issues.apache.org/jira/browse/HDFS-7450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14227173#comment-14227173
]
Hadoop QA commented on HDFS-7450:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12683948/HDFS-7450.003.patch
against trunk revision c1f2bb2.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8858//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8858//console
This message is automatically generated.
Consolidate GetFileInfo, GetListings and GetContentSummary into a single class
--
Key: HDFS-7450
URL: https://issues.apache.org/jira/browse/HDFS-7450
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Haohui Mai
Attachments: HDFS-7450.000.patch, HDFS-7450.001.patch,
HDFS-7450.002.patch, HDFS-7450.003.patch
This jira proposes to consolidate the implementation of {{GetFileInfo}},
{{GetListings}} and {{GetContentSummary}} into a single class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6735) A minor optimization to avoid pread() be blocked by read() inside the same DFSInputStream
[
https://issues.apache.org/jira/browse/HDFS-6735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14227174#comment-14227174
]
Hadoop QA commented on HDFS-6735:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12683956/HDFS-6735-v6.txt
against trunk revision c1f2bb2.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The test build failed in
hadoop-hdfs-project/hadoop-hdfs
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8860//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/8860//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8860//console
This message is automatically generated.
A minor optimization to avoid pread() be blocked by read() inside the same
DFSInputStream
-
Key: HDFS-6735
URL: https://issues.apache.org/jira/browse/HDFS-6735
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs-client
Affects Versions: 3.0.0
Reporter: Liang Xie
Assignee: Lars Hofhansl
Attachments: HDFS-6735-v2.txt, HDFS-6735-v3.txt, HDFS-6735-v4.txt,
HDFS-6735-v5.txt, HDFS-6735-v6.txt, HDFS-6735-v6.txt, HDFS-6735.txt
In current DFSInputStream impl, there're a couple of coarser-grained locks in
read/pread path, and it has became a HBase read latency pain point so far. In
HDFS-6698, i made a minor patch against the first encourtered lock, around
getFileLength, in deed, after reading code and testing, it shows still other
locks we could improve.
In this jira, i'll make a patch against other locks, and a simple test case
to show the issue and the improved result.
This is important for HBase application, since in current HFile read path, we
issue all read()/pread() requests in the same DFSInputStream for one HFile.
(Multi streams solution is another story i had a plan to do, but probably
will take more time than i expected)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7424) Add web UI for NFS gateway
[
https://issues.apache.org/jira/browse/HDFS-7424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14227223#comment-14227223
]
Hadoop QA commented on HDFS-7424:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12683955/HDFS-7424.004.patch
against trunk revision c1f2bb2.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The following test timeouts occurred in
hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-nfs:
org.apache.hadoop.hdfs.nfs.nfs3.TestNfs3HttpServer
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8859//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8859//console
This message is automatically generated.
Add web UI for NFS gateway
--
Key: HDFS-7424
URL: https://issues.apache.org/jira/browse/HDFS-7424
Project: Hadoop HDFS
Issue Type: New Feature
Components: nfs
Affects Versions: 2.2.0
Reporter: Brandon Li
Assignee: Brandon Li
Attachments: HDFS-7424.001.patch, HDFS-7424.002.patch,
HDFS-7424.003.patch, HDFS-7424.004.patch
This JIRA is to track the effort to add web UI for NFS gateway to show some
metrics and configuration related information.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HDFS-5098) Enhance FileSystem.Statistics to have locality information
[ https://issues.apache.org/jira/browse/HDFS-5098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha resolved HDFS-5098. -- Resolution: Duplicate Enhance FileSystem.Statistics to have locality information -- Key: HDFS-5098 URL: https://issues.apache.org/jira/browse/HDFS-5098 Project: Hadoop HDFS Issue Type: Improvement Reporter: Bikas Saha Assignee: Suresh Srinivas Fix For: 2.6.0 Currently in MR/Tez we dont have a good and accurate means to detect how much the the IO was actually done locally. Getting this information from the source of truth would be much better. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
