[jira] [Commented] (HDFS-4954) compile failure in branch-2: getFlushedOffset should catch or rethrow IOException
[ https://issues.apache.org/jira/browse/HDFS-4954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13698688#comment-13698688 ] Suresh Srinivas commented on HDFS-4954: --- This is not a problem in 2.1.0-beta? > compile failure in branch-2: getFlushedOffset should catch or rethrow > IOException > - > > Key: HDFS-4954 > URL: https://issues.apache.org/jira/browse/HDFS-4954 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.3.0 >Reporter: Brandon Li >Assignee: Brandon Li > > This is caused when merging HDFS-4762 from trunk to branch-2. > Unlike that in trunk, FSDataOutputStream.getPos() throws IOException, so > getFlushedOffset should catch or rethrow the IOException. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4942) Add retry cache support in Namenode
[ https://issues.apache.org/jira/browse/HDFS-4942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13698679#comment-13698679 ] Suresh Srinivas commented on HDFS-4942: --- UUID remains the same for only retried/retransmitted requests done automatically by the RPC layer and not for subsequent attempt a user/application makes. > Add retry cache support in Namenode > --- > > Key: HDFS-4942 > URL: https://issues.apache.org/jira/browse/HDFS-4942 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ha, namenode >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Attachments: HDFSRetryCache.pdf > > > In current HA mechanism with FailoverProxyProvider and non HA setups with > RetryProxy retry a request from the RPC layer. If the retried request has > already been processed at the namenode, the subsequent attempts fail for > non-idempotent operations such as create, append, delete, rename etc. This > will cause application failures during HA failover, network issues etc. > This jira proposes adding retry cache at the namenode to handle these > failures. More details in the comments. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4954) compile failure in branch-2: getFlushedOffset should catch or rethrow IOException
Brandon Li created HDFS-4954: Summary: compile failure in branch-2: getFlushedOffset should catch or rethrow IOException Key: HDFS-4954 URL: https://issues.apache.org/jira/browse/HDFS-4954 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.3.0 Reporter: Brandon Li Assignee: Brandon Li This is caused when merging HDFS-4762 from trunk to branch-2. Unlike that in trunk, FSDataOutputStream.getPos() throws IOException, so getFlushedOffset should catch or rethrow the IOException. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4942) Add retry cache support in Namenode
[ https://issues.apache.org/jira/browse/HDFS-4942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13698674#comment-13698674 ] Bikas Saha commented on HDFS-4942: -- Some clarifications would be good 1) User runs create command using CLI. Internally, the command is tried many times over RPC. Will these retries have the same UUID and will result in response coming from retry cache? I think yes. 2) Client in 1) above tries many times and fails. User retries the CLI command again. This will generate a new set of retries. Will these have the same UUID as the retries in 1) above. Will they get response from retry cache? > Add retry cache support in Namenode > --- > > Key: HDFS-4942 > URL: https://issues.apache.org/jira/browse/HDFS-4942 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ha, namenode >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Attachments: HDFSRetryCache.pdf > > > In current HA mechanism with FailoverProxyProvider and non HA setups with > RetryProxy retry a request from the RPC layer. If the retried request has > already been processed at the namenode, the subsequent attempts fail for > non-idempotent operations such as create, append, delete, rename etc. This > will cause application failures during HA failover, network issues etc. > This jira proposes adding retry cache at the namenode to handle these > failures. More details in the comments. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4913) Deleting file through fuse-dfs when using trash fails requiring root permissions
[
https://issues.apache.org/jira/browse/HDFS-4913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13698577#comment-13698577
]
Hadoop QA commented on HDFS-4913:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12590559/HDFS-4913.002.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4592//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4592//console
This message is automatically generated.
> Deleting file through fuse-dfs when using trash fails requiring root
> permissions
>
>
> Key: HDFS-4913
> URL: https://issues.apache.org/jira/browse/HDFS-4913
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: fuse-dfs
>Affects Versions: 2.0.3-alpha
>Reporter: Stephen Chu
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-4913.002.patch
>
>
> As _root_, I mounted HDFS with fuse-dfs using the -ousetrash option.
> As _testuser_, I cd into the mount and touch a test file at
> _/user/testuser/testFile1_. As the same user, I try to rm the file and run
> into an error:
> {code}
> [testuser@hdfs-vanilla-1 ~]$ cd /hdfs_mnt/user/testuser
> [testuser@hdfs-vanilla-1 testuser]$ touch testFile1
> [testuser@hdfs-vanilla-1 testuser]$ rm testFile1
> rm: cannot remove `testFile1': Unknown error 255
> {code}
> I check the fuse-dfs debug output, and it shows that we attempt to mkdir
> /user/root/.Trash, which testuser doesn't have permissions to.
> Ideally, we'd be able to remove testFile1 and have testFile1 be put into
> /user/testuser/.Trash instead of /user/root/.Trash.
> Error in debug:
> {code}
> unlink /user/testuser/testFile1
> hdfsCreateDirectory(/user/root/.Trash/Current/user/testuser):
> FileSystem#mkdirs error:
> org.apache.hadoop.security.AccessControlException: Permission denied:
> user=testuser, access=WRITE, inode="/user/root":root:supergroup:drwxr-xr-x
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:224)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:204)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:149)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:4716)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:4698)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:4672)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:3035)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:2999)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:2980)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:648)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTransla
[jira] [Commented] (HDFS-4750) Support NFSv3 interface to HDFS
[ https://issues.apache.org/jira/browse/HDFS-4750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13698561#comment-13698561 ] Brandon Li commented on HDFS-4750: -- I've merge the HADOOP-9009,HADOOP-9515,HDFS-4762,HDFS-4948 into branch-2 and branch-2.1 > Support NFSv3 interface to HDFS > --- > > Key: HDFS-4750 > URL: https://issues.apache.org/jira/browse/HDFS-4750 > Project: Hadoop HDFS > Issue Type: New Feature > Components: nfs >Affects Versions: 3.0.0 >Reporter: Brandon Li >Assignee: Brandon Li > Attachments: HADOOP-NFS-Proposal.pdf, HDFS-4750.patch, nfs-trunk.patch > > > Access HDFS is usually done through HDFS Client or webHDFS. Lack of seamless > integration with client’s file system makes it difficult for users and > impossible for some applications to access HDFS. NFS interface support is one > way for HDFS to have such easy integration. > This JIRA is to track the NFS protocol support for accessing HDFS. With HDFS > client, webHDFS and the NFS interface, HDFS will be easier to access and be > able support more applications and use cases. > We will upload the design document and the initial implementation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4762) Provide HDFS based NFSv3 and Mountd implementation
[ https://issues.apache.org/jira/browse/HDFS-4762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-4762: - Fix Version/s: 2.3.0 2.1.0-beta > Provide HDFS based NFSv3 and Mountd implementation > -- > > Key: HDFS-4762 > URL: https://issues.apache.org/jira/browse/HDFS-4762 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Affects Versions: 3.0.0 >Reporter: Brandon Li >Assignee: Brandon Li > Fix For: 3.0.0, 2.1.0-beta, 2.3.0 > > Attachments: HDFS-4762.patch, HDFS-4762.patch.2, HDFS-4762.patch.3, > HDFS-4762.patch.3, HDFS-4762.patch.4, HDFS-4762.patch.5, HDFS-4762.patch.6, > HDFS-4762.patch.7 > > > This is to track the implementation of NFSv3 to HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4948) mvn site for hadoop-hdfs-nfs fails
[
https://issues.apache.org/jira/browse/HDFS-4948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brandon Li updated HDFS-4948:
-
Fix Version/s: 2.3.0
2.1.0-beta
> mvn site for hadoop-hdfs-nfs fails
> --
>
> Key: HDFS-4948
> URL: https://issues.apache.org/jira/browse/HDFS-4948
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Robert Joseph Evans
>Assignee: Brandon Li
> Fix For: 3.0.0, 2.1.0-beta, 2.3.0
>
> Attachments: HDFS-4948.patch
>
>
> Running mvn site on trunk results in the following error.
> {noformat}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-antrun-plugin:1.6:run (default) on project
> hadoop-hdfs-nfs: An Ant BuildException has occured: Warning: Could not find
> file
> /home/evans/src/hadoop-git/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/resources/hdfs-nfs-default.xml
> to copy. -> [Help 1]
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4465) Optimize datanode ReplicasMap and ReplicaInfo
[
https://issues.apache.org/jira/browse/HDFS-4465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13698543#comment-13698543
]
Hadoop QA commented on HDFS-4465:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12590557/HDFS-4465.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4591//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4591//console
This message is automatically generated.
> Optimize datanode ReplicasMap and ReplicaInfo
> -
>
> Key: HDFS-4465
> URL: https://issues.apache.org/jira/browse/HDFS-4465
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 2.0.5-alpha
>Reporter: Suresh Srinivas
>Assignee: Aaron T. Myers
> Attachments: dn-memory-improvements.patch, HDFS-4465.patch,
> HDFS-4465.patch
>
>
> In Hadoop a lot of optimization has been done in namenode data structures to
> be memory efficient. Similar optimizations are necessary for Datanode
> process. With the growth in storage per datanode and number of blocks hosted
> on datanode, this jira intends to optimize long lived ReplicasMap and
> ReplicaInfo objects.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4860) Add additional attributes to JMX beans
[ https://issues.apache.org/jira/browse/HDFS-4860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13698537#comment-13698537 ] Konstantin Boudnik commented on HDFS-4860: -- I like the improvement. And the patch looks good for me +1 > Add additional attributes to JMX beans > -- > > Key: HDFS-4860 > URL: https://issues.apache.org/jira/browse/HDFS-4860 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 0.20.204.1, 3.0.0, 2.1.0-beta, 2.0.4-alpha >Reporter: Trevor Lorimer >Assignee: Trevor Lorimer > Attachments: HDFS-4860-3.diff > > > Currently the JMX bean returns much of the data contained on the HDFS Health > webpage (dfsHealth.html). However there are several other attributes that are > required to be added. > I intend to add the following items to the appropriate bean in parenthesis : > Started time (NameNodeInfo), > Compiled info (NameNodeInfo), > Jvm MaxHeap, MaxNonHeap (JvmMetrics) > Node Usage stats (i.e. Min, Median, Max, stdev) (NameNodeInfo), > Count of decommissioned Live and Dead nodes (FSNamesystemState), > Journal Status (NodeNameInfo) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4860) Add additional attributes to JMX beans
[ https://issues.apache.org/jira/browse/HDFS-4860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated HDFS-4860: - Assignee: Trevor Lorimer > Add additional attributes to JMX beans > -- > > Key: HDFS-4860 > URL: https://issues.apache.org/jira/browse/HDFS-4860 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 0.20.204.1, 3.0.0, 2.1.0-beta, 2.0.4-alpha >Reporter: Trevor Lorimer >Assignee: Trevor Lorimer > Attachments: HDFS-4860-3.diff > > > Currently the JMX bean returns much of the data contained on the HDFS Health > webpage (dfsHealth.html). However there are several other attributes that are > required to be added. > I intend to add the following items to the appropriate bean in parenthesis : > Started time (NameNodeInfo), > Compiled info (NameNodeInfo), > Jvm MaxHeap, MaxNonHeap (JvmMetrics) > Node Usage stats (i.e. Min, Median, Max, stdev) (NameNodeInfo), > Count of decommissioned Live and Dead nodes (FSNamesystemState), > Journal Status (NodeNameInfo) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4913) Deleting file through fuse-dfs when using trash fails requiring root permissions
[
https://issues.apache.org/jira/browse/HDFS-4913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HDFS-4913:
---
Attachment: HDFS-4913.002.patch
The issue here, I believe, is that fuse_dfs is always putting the file into
/user/root/.Trash/Current, whereas it should be putting it into
{{/user/${USERNAME}/.Trash/Current}}.
We can get username from the fuse_context. Although it comes as a UID, we can
map it back to a string. This patch does that.
> Deleting file through fuse-dfs when using trash fails requiring root
> permissions
>
>
> Key: HDFS-4913
> URL: https://issues.apache.org/jira/browse/HDFS-4913
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: fuse-dfs
>Affects Versions: 2.0.3-alpha
>Reporter: Stephen Chu
> Attachments: HDFS-4913.002.patch
>
>
> As _root_, I mounted HDFS with fuse-dfs using the -ousetrash option.
> As _testuser_, I cd into the mount and touch a test file at
> _/user/testuser/testFile1_. As the same user, I try to rm the file and run
> into an error:
> {code}
> [testuser@hdfs-vanilla-1 ~]$ cd /hdfs_mnt/user/testuser
> [testuser@hdfs-vanilla-1 testuser]$ touch testFile1
> [testuser@hdfs-vanilla-1 testuser]$ rm testFile1
> rm: cannot remove `testFile1': Unknown error 255
> {code}
> I check the fuse-dfs debug output, and it shows that we attempt to mkdir
> /user/root/.Trash, which testuser doesn't have permissions to.
> Ideally, we'd be able to remove testFile1 and have testFile1 be put into
> /user/testuser/.Trash instead of /user/root/.Trash.
> Error in debug:
> {code}
> unlink /user/testuser/testFile1
> hdfsCreateDirectory(/user/root/.Trash/Current/user/testuser):
> FileSystem#mkdirs error:
> org.apache.hadoop.security.AccessControlException: Permission denied:
> user=testuser, access=WRITE, inode="/user/root":root:supergroup:drwxr-xr-x
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:224)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:204)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:149)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:4716)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:4698)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:4672)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:3035)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:2999)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:2980)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:648)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:419)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:44970)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
> at
> org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1002)
> at
> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1701)
> at
> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1697)
> at
> java.security.AccessController.doPrivileged(Native Method)
> at
> javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
> at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1695)
>
[jira] [Updated] (HDFS-4913) Deleting file through fuse-dfs when using trash fails requiring root permissions
[
https://issues.apache.org/jira/browse/HDFS-4913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HDFS-4913:
---
Assignee: Colin Patrick McCabe
Status: Patch Available (was: Open)
> Deleting file through fuse-dfs when using trash fails requiring root
> permissions
>
>
> Key: HDFS-4913
> URL: https://issues.apache.org/jira/browse/HDFS-4913
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: fuse-dfs
>Affects Versions: 2.0.3-alpha
>Reporter: Stephen Chu
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-4913.002.patch
>
>
> As _root_, I mounted HDFS with fuse-dfs using the -ousetrash option.
> As _testuser_, I cd into the mount and touch a test file at
> _/user/testuser/testFile1_. As the same user, I try to rm the file and run
> into an error:
> {code}
> [testuser@hdfs-vanilla-1 ~]$ cd /hdfs_mnt/user/testuser
> [testuser@hdfs-vanilla-1 testuser]$ touch testFile1
> [testuser@hdfs-vanilla-1 testuser]$ rm testFile1
> rm: cannot remove `testFile1': Unknown error 255
> {code}
> I check the fuse-dfs debug output, and it shows that we attempt to mkdir
> /user/root/.Trash, which testuser doesn't have permissions to.
> Ideally, we'd be able to remove testFile1 and have testFile1 be put into
> /user/testuser/.Trash instead of /user/root/.Trash.
> Error in debug:
> {code}
> unlink /user/testuser/testFile1
> hdfsCreateDirectory(/user/root/.Trash/Current/user/testuser):
> FileSystem#mkdirs error:
> org.apache.hadoop.security.AccessControlException: Permission denied:
> user=testuser, access=WRITE, inode="/user/root":root:supergroup:drwxr-xr-x
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:224)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:204)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:149)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:4716)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:4698)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:4672)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:3035)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:2999)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:2980)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:648)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:419)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:44970)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
> at
> org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1002)
> at
> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1701)
> at
> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1697)
> at
> java.security.AccessController.doPrivileged(Native Method)
> at
> javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
> at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1695)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructor
[jira] [Updated] (HDFS-4465) Optimize datanode ReplicasMap and ReplicaInfo
[ https://issues.apache.org/jira/browse/HDFS-4465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-4465: - Attachment: HDFS-4465.patch Here's an updated patch which should address all of your feedback, Suresh. # Good thinking. I did some back of the envelope math which suggested that even 1% was probably higher than necessary for a typical DN. Switched this to 0.5%. # Per previous discussion, left it extending Block and added a comment. # Good thinking. Moved the parsing code to a separate static function and added a test for it. # In my testing with a DN with ~1MM blocks, this patch makes each replica go from using ~635 bytes per replica to ~250 bytes per replica, so about a 2.5x improvement. Note that to address the findbugs warning I had to add an exception to the findbugs exclude file, since in this patch I am very deliberately using the String(String) constructor so as to trim the underlying char[] array. > Optimize datanode ReplicasMap and ReplicaInfo > - > > Key: HDFS-4465 > URL: https://issues.apache.org/jira/browse/HDFS-4465 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 2.0.5-alpha >Reporter: Suresh Srinivas >Assignee: Aaron T. Myers > Attachments: dn-memory-improvements.patch, HDFS-4465.patch, > HDFS-4465.patch > > > In Hadoop a lot of optimization has been done in namenode data structures to > be memory efficient. Similar optimizations are necessary for Datanode > process. With the growth in storage per datanode and number of blocks hosted > on datanode, this jira intends to optimize long lived ReplicasMap and > ReplicaInfo objects. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4504) DFSOutputStream#close doesn't always release resources (such as leases)
[
https://issues.apache.org/jira/browse/HDFS-4504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13698487#comment-13698487
]
Colin Patrick McCabe commented on HDFS-4504:
That's an interesting idea-- calling recoverLease from the client itself. It
might have the advantage of requiring less new code, compared to adding a new
flag to {{complete()}}.
> DFSOutputStream#close doesn't always release resources (such as leases)
> ---
>
> Key: HDFS-4504
> URL: https://issues.apache.org/jira/browse/HDFS-4504
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-4504.001.patch, HDFS-4504.002.patch
>
>
> {{DFSOutputStream#close}} can throw an {{IOException}} in some cases. One
> example is if there is a pipeline error and then pipeline recovery fails.
> Unfortunately, in this case, some of the resources used by the
> {{DFSOutputStream}} are leaked. One particularly important resource is file
> leases.
> So it's possible for a long-lived HDFS client, such as Flume, to write many
> blocks to a file, but then fail to close it. Unfortunately, the
> {{LeaseRenewerThread}} inside the client will continue to renew the lease for
> the "undead" file. Future attempts to close the file will just rethrow the
> previous exception, and no progress can be made by the client.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-427) fuse-dfs should support symlinks
[
https://issues.apache.org/jira/browse/HDFS-427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13698482#comment-13698482
]
Colin Patrick McCabe commented on HDFS-427:
---
note: this also includes implementing the symlink resolving logic in
{{Trash#moveToAppropriateTrash}}
> fuse-dfs should support symlinks
>
>
> Key: HDFS-427
> URL: https://issues.apache.org/jira/browse/HDFS-427
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: fuse-dfs
>Reporter: Pete Wyckoff
>
> implement dfs_symlink(from, to) and dfs_readlink(path,buf,size)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4953) enable HDFS local reads via mmap
Colin Patrick McCabe created HDFS-4953: -- Summary: enable HDFS local reads via mmap Key: HDFS-4953 URL: https://issues.apache.org/jira/browse/HDFS-4953 Project: Hadoop HDFS Issue Type: New Feature Affects Versions: 2.2.0 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Currently, the short-circuit local read pathway allows HDFS clients to access files directly without going through the DataNode. However, all of these reads involve a copy at the operating system level, since they rely on the read() / pread() / etc family of kernel interfaces. We would like to enable HDFS to read local files via mmap. This would enable truly zero-copy reads. In the initial implementation, zero-copy reads will only be performed when checksums were disabled. Later, we can use the DataNode's cache awareness to only perform zero-copy reads when we know that checksum has already been verified. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HDFS-4942) Add retry cache support in Namenode
[ https://issues.apache.org/jira/browse/HDFS-4942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13698425#comment-13698425 ] Suresh Srinivas edited comment on HDFS-4942 at 7/3/13 12:21 AM: I created HADOOP-9688 to add unique request ID to RPC requests. I also have posted an early patch. was (Author: sureshms): I create HADOOP-9688 to add unique request ID to RPC requests. I also have posted an early patch. > Add retry cache support in Namenode > --- > > Key: HDFS-4942 > URL: https://issues.apache.org/jira/browse/HDFS-4942 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ha, namenode >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Attachments: HDFSRetryCache.pdf > > > In current HA mechanism with FailoverProxyProvider and non HA setups with > RetryProxy retry a request from the RPC layer. If the retried request has > already been processed at the namenode, the subsequent attempts fail for > non-idempotent operations such as create, append, delete, rename etc. This > will cause application failures during HA failover, network issues etc. > This jira proposes adding retry cache at the namenode to handle these > failures. More details in the comments. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4942) Add retry cache support in Namenode
[ https://issues.apache.org/jira/browse/HDFS-4942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13698425#comment-13698425 ] Suresh Srinivas commented on HDFS-4942: --- I create HADOOP-9688 to add unique request ID to RPC requests. I also have posted an early patch. > Add retry cache support in Namenode > --- > > Key: HDFS-4942 > URL: https://issues.apache.org/jira/browse/HDFS-4942 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ha, namenode >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Attachments: HDFSRetryCache.pdf > > > In current HA mechanism with FailoverProxyProvider and non HA setups with > RetryProxy retry a request from the RPC layer. If the retried request has > already been processed at the namenode, the subsequent attempts fail for > non-idempotent operations such as create, append, delete, rename etc. This > will cause application failures during HA failover, network issues etc. > This jira proposes adding retry cache at the namenode to handle these > failures. More details in the comments. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4942) Add retry cache support in Namenode
[
https://issues.apache.org/jira/browse/HDFS-4942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13698423#comment-13698423
]
Chris Nauroth commented on HDFS-4942:
-
The proposal looks good, and I'll be interested to see the analysis of the
individual RPC calls. Reminder on something that came up in offline
conversation: it appears that we can change
{{ClientProtocol#getDataEncryptionKey}} to annotate it as Idempotent. It
doesn't appear to mutate state. If a retry causes creation of multiple keys,
that shouldn't be a problem.
{quote}
Given that we plan on adding a unique identifier to every RPC request, should
we get this change done before 2.1.0-beta rc2 is built? This way 2.1.0-beta
clients can utilize retry cache as well.
{quote}
+1 for this idea. Adding the UUID now would be a low-risk change.
> Add retry cache support in Namenode
> ---
>
> Key: HDFS-4942
> URL: https://issues.apache.org/jira/browse/HDFS-4942
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: ha, namenode
>Reporter: Suresh Srinivas
>Assignee: Suresh Srinivas
> Attachments: HDFSRetryCache.pdf
>
>
> In current HA mechanism with FailoverProxyProvider and non HA setups with
> RetryProxy retry a request from the RPC layer. If the retried request has
> already been processed at the namenode, the subsequent attempts fail for
> non-idempotent operations such as create, append, delete, rename etc. This
> will cause application failures during HA failover, network issues etc.
> This jira proposes adding retry cache at the namenode to handle these
> failures. More details in the comments.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4948) mvn site for hadoop-hdfs-nfs fails
[
https://issues.apache.org/jira/browse/HDFS-4948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13698416#comment-13698416
]
Hudson commented on HDFS-4948:
--
Integrated in Hadoop-trunk-Commit #4036 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/4036/])
HDFS-4948. mvn site for hadoop-hdfs-nfs fails. Contributed by Brandon Li
(Revision 1499152)
Result = SUCCESS
brandonli :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1499152
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/pom.xml
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> mvn site for hadoop-hdfs-nfs fails
> --
>
> Key: HDFS-4948
> URL: https://issues.apache.org/jira/browse/HDFS-4948
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Robert Joseph Evans
>Assignee: Brandon Li
> Fix For: 3.0.0
>
> Attachments: HDFS-4948.patch
>
>
> Running mvn site on trunk results in the following error.
> {noformat}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-antrun-plugin:1.6:run (default) on project
> hadoop-hdfs-nfs: An Ant BuildException has occured: Warning: Could not find
> file
> /home/evans/src/hadoop-git/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/resources/hdfs-nfs-default.xml
> to copy. -> [Help 1]
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4763) Add script changes/utility for starting NFS gateway
[ https://issues.apache.org/jira/browse/HDFS-4763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-4763: - Component/s: nfs > Add script changes/utility for starting NFS gateway > --- > > Key: HDFS-4763 > URL: https://issues.apache.org/jira/browse/HDFS-4763 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Reporter: Brandon Li >Assignee: Brandon Li > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4750) Support NFSv3 interface to HDFS
[ https://issues.apache.org/jira/browse/HDFS-4750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-4750: - Component/s: nfs > Support NFSv3 interface to HDFS > --- > > Key: HDFS-4750 > URL: https://issues.apache.org/jira/browse/HDFS-4750 > Project: Hadoop HDFS > Issue Type: New Feature > Components: nfs >Affects Versions: 3.0.0 >Reporter: Brandon Li >Assignee: Brandon Li > Attachments: HADOOP-NFS-Proposal.pdf, HDFS-4750.patch, nfs-trunk.patch > > > Access HDFS is usually done through HDFS Client or webHDFS. Lack of seamless > integration with client’s file system makes it difficult for users and > impossible for some applications to access HDFS. NFS interface support is one > way for HDFS to have such easy integration. > This JIRA is to track the NFS protocol support for accessing HDFS. With HDFS > client, webHDFS and the NFS interface, HDFS will be easier to access and be > able support more applications and use cases. > We will upload the design document and the initial implementation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4947) Add NFS server export table to control export by hostname or IP range
[ https://issues.apache.org/jira/browse/HDFS-4947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-4947: - Component/s: nfs > Add NFS server export table to control export by hostname or IP range > - > > Key: HDFS-4947 > URL: https://issues.apache.org/jira/browse/HDFS-4947 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Affects Versions: 3.0.0 >Reporter: Brandon Li >Assignee: Jing Zhao > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4762) Provide HDFS based NFSv3 and Mountd implementation
[ https://issues.apache.org/jira/browse/HDFS-4762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13698409#comment-13698409 ] Brandon Li commented on HDFS-4762: -- It should be fixed now. > Provide HDFS based NFSv3 and Mountd implementation > -- > > Key: HDFS-4762 > URL: https://issues.apache.org/jira/browse/HDFS-4762 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Affects Versions: 3.0.0 >Reporter: Brandon Li >Assignee: Brandon Li > Fix For: 3.0.0 > > Attachments: HDFS-4762.patch, HDFS-4762.patch.2, HDFS-4762.patch.3, > HDFS-4762.patch.3, HDFS-4762.patch.4, HDFS-4762.patch.5, HDFS-4762.patch.6, > HDFS-4762.patch.7 > > > This is to track the implementation of NFSv3 to HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4948) mvn site for hadoop-hdfs-nfs fails
[
https://issues.apache.org/jira/browse/HDFS-4948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brandon Li updated HDFS-4948:
-
Fix Version/s: 3.0.0
> mvn site for hadoop-hdfs-nfs fails
> --
>
> Key: HDFS-4948
> URL: https://issues.apache.org/jira/browse/HDFS-4948
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Robert Joseph Evans
>Assignee: Brandon Li
> Fix For: 3.0.0
>
> Attachments: HDFS-4948.patch
>
>
> Running mvn site on trunk results in the following error.
> {noformat}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-antrun-plugin:1.6:run (default) on project
> hadoop-hdfs-nfs: An Ant BuildException has occured: Warning: Could not find
> file
> /home/evans/src/hadoop-git/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/resources/hdfs-nfs-default.xml
> to copy. -> [Help 1]
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4948) mvn site for hadoop-hdfs-nfs fails
[
https://issues.apache.org/jira/browse/HDFS-4948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brandon Li updated HDFS-4948:
-
Resolution: Fixed
Status: Resolved (was: Patch Available)
> mvn site for hadoop-hdfs-nfs fails
> --
>
> Key: HDFS-4948
> URL: https://issues.apache.org/jira/browse/HDFS-4948
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Robert Joseph Evans
>Assignee: Brandon Li
> Attachments: HDFS-4948.patch
>
>
> Running mvn site on trunk results in the following error.
> {noformat}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-antrun-plugin:1.6:run (default) on project
> hadoop-hdfs-nfs: An Ant BuildException has occured: Warning: Could not find
> file
> /home/evans/src/hadoop-git/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/resources/hdfs-nfs-default.xml
> to copy. -> [Help 1]
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4948) mvn site for hadoop-hdfs-nfs fails
[
https://issues.apache.org/jira/browse/HDFS-4948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13698407#comment-13698407
]
Brandon Li commented on HDFS-4948:
--
I've committed to trunk. Thanks!
> mvn site for hadoop-hdfs-nfs fails
> --
>
> Key: HDFS-4948
> URL: https://issues.apache.org/jira/browse/HDFS-4948
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Robert Joseph Evans
>Assignee: Brandon Li
> Attachments: HDFS-4948.patch
>
>
> Running mvn site on trunk results in the following error.
> {noformat}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-antrun-plugin:1.6:run (default) on project
> hadoop-hdfs-nfs: An Ant BuildException has occured: Warning: Could not find
> file
> /home/evans/src/hadoop-git/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/resources/hdfs-nfs-default.xml
> to copy. -> [Help 1]
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4952) dfs -ls hftp:// fails on secure hadoop2 cluster
yeshavora created HDFS-4952: --- Summary: dfs -ls hftp:// fails on secure hadoop2 cluster Key: HDFS-4952 URL: https://issues.apache.org/jira/browse/HDFS-4952 Project: Hadoop HDFS Issue Type: Bug Reporter: yeshavora Running :hadoop dfs -ls hftp://:/A WARN fs.FileSystem: Couldn't connect to http://:50470, assuming security is disabled ls: Security enabled but user not authenticated by filter -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4951) FsShell commands using secure httpfs throw exceptions due to missing TokenRenewer
Robert Kanter created HDFS-4951:
---
Summary: FsShell commands using secure httpfs throw exceptions due
to missing TokenRenewer
Key: HDFS-4951
URL: https://issues.apache.org/jira/browse/HDFS-4951
Project: Hadoop HDFS
Issue Type: Bug
Components: security
Affects Versions: 3.0.0
Reporter: Robert Kanter
Assignee: Robert Kanter
It looks like there isn't a {{TokenRenewer}} for HttpFS delegation tokens
({{HTTPFS_DELEGATION_TOKENS}} tokens, so when it goes to cancel the token, it
throws an exception:
{noformat}
$ hadoop fs -ls webhdfs://host:14000
// File listing omitted
13/06/21 13:09:04 WARN token.Token: No TokenRenewer defined for token kind
HTTPFS_DELEGATION_TOKEN
13/06/21 13:09:04 WARN util.ShutdownHookManager: ShutdownHook 'ClientFinalizer'
failed, java.lang.UnsupportedOperationException: Token cancel is not supported
for HTTPFS_DELEGATION_TOKEN tokens
java.lang.UnsupportedOperationException: Token cancel is not supported for
HTTPFS_DELEGATION_TOKEN tokens
at
org.apache.hadoop.security.token.Token$TrivialRenewer.cancel(Token.java:417)
at org.apache.hadoop.security.token.Token.cancel(Token.java:382)
at
org.apache.hadoop.fs.DelegationTokenRenewer$RenewAction.cancel(DelegationTokenRenewer.java:146)
at
org.apache.hadoop.fs.DelegationTokenRenewer$RenewAction.access$200(DelegationTokenRenewer.java:58)
at
org.apache.hadoop.fs.DelegationTokenRenewer.removeRenewAction(DelegationTokenRenewer.java:233)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.close(WebHdfsFileSystem.java:790)
at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2398)
at
org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:2414)
at
org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
{noformat}
WebHDFS doesn't have this problem because it has a {{TokenRenewer}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4951) FsShell commands using secure httpfs throw exceptions due to missing TokenRenewer
[
https://issues.apache.org/jira/browse/HDFS-4951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated HDFS-4951:
Description:
It looks like there isn't a {{TokenRenewer}} for HttpFS delegation tokens
({{HTTPFS_DELEGATION_TOKENS}} tokens, so when it goes to cancel the token, it
throws an exception:
{noformat}
$ hadoop fs -ls webhdfs://host:14000
// File listing omitted
13/06/21 13:09:04 WARN token.Token: No TokenRenewer defined for token kind
HTTPFS_DELEGATION_TOKEN
13/06/21 13:09:04 WARN util.ShutdownHookManager: ShutdownHook 'ClientFinalizer'
failed, java.lang.UnsupportedOperationException: Token cancel is not supported
for HTTPFS_DELEGATION_TOKEN tokens
java.lang.UnsupportedOperationException: Token cancel is not supported for
HTTPFS_DELEGATION_TOKEN tokens
at
org.apache.hadoop.security.token.Token$TrivialRenewer.cancel(Token.java:417)
at org.apache.hadoop.security.token.Token.cancel(Token.java:382)
at
org.apache.hadoop.fs.DelegationTokenRenewer$RenewAction.cancel(DelegationTokenRenewer.java:146)
at
org.apache.hadoop.fs.DelegationTokenRenewer$RenewAction.access$200(DelegationTokenRenewer.java:58)
at
org.apache.hadoop.fs.DelegationTokenRenewer.removeRenewAction(DelegationTokenRenewer.java:233)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.close(WebHdfsFileSystem.java:790)
at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2398)
at
org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:2414)
at
org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
{noformat}
WebHDFS doesn't have this problem because it has a {{TokenRenewer}} for its
delegation tokens ({{"WEBHDFS delegation"}} tokens).
was:
It looks like there isn't a {{TokenRenewer}} for HttpFS delegation tokens
({{HTTPFS_DELEGATION_TOKENS}} tokens, so when it goes to cancel the token, it
throws an exception:
{noformat}
$ hadoop fs -ls webhdfs://host:14000
// File listing omitted
13/06/21 13:09:04 WARN token.Token: No TokenRenewer defined for token kind
HTTPFS_DELEGATION_TOKEN
13/06/21 13:09:04 WARN util.ShutdownHookManager: ShutdownHook 'ClientFinalizer'
failed, java.lang.UnsupportedOperationException: Token cancel is not supported
for HTTPFS_DELEGATION_TOKEN tokens
java.lang.UnsupportedOperationException: Token cancel is not supported for
HTTPFS_DELEGATION_TOKEN tokens
at
org.apache.hadoop.security.token.Token$TrivialRenewer.cancel(Token.java:417)
at org.apache.hadoop.security.token.Token.cancel(Token.java:382)
at
org.apache.hadoop.fs.DelegationTokenRenewer$RenewAction.cancel(DelegationTokenRenewer.java:146)
at
org.apache.hadoop.fs.DelegationTokenRenewer$RenewAction.access$200(DelegationTokenRenewer.java:58)
at
org.apache.hadoop.fs.DelegationTokenRenewer.removeRenewAction(DelegationTokenRenewer.java:233)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.close(WebHdfsFileSystem.java:790)
at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2398)
at
org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:2414)
at
org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
{noformat}
WebHDFS doesn't have this problem because it has a {{TokenRenewer}}.
> FsShell commands using secure httpfs throw exceptions due to missing
> TokenRenewer
> -
>
> Key: HDFS-4951
> URL: https://issues.apache.org/jira/browse/HDFS-4951
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
>Affects Versions: 3.0.0
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>
> It looks like there isn't a {{TokenRenewer}} for HttpFS delegation tokens
> ({{HTTPFS_DELEGATION_TOKENS}} tokens, so when it goes to cancel the token, it
> throws an exception:
> {noformat}
> $ hadoop fs -ls webhdfs://host:14000
> // File listing omitted
> 13/06/21 13:09:04 WARN token.Token: No TokenRenewer defined for token kind
> HTTPFS_DELEGATION_TOKEN
> 13/06/21 13:09:04 WARN util.ShutdownHookManager: ShutdownHook
> 'ClientFinalizer' failed, java.lang.UnsupportedOperationException: Token
> cancel is not supported for HTTPFS_DELEGATION_TOKEN tokens
> java.lang.UnsupportedOperationException: Token cancel is not supported for
> HTTPFS_DELEGATION_TOKEN tokens
> at
> org.apache.hadoop.security.token.Token$TrivialRenewer.cancel(Token.java:417)
> at org.apache.hadoop.security.token.Token.cancel(Token.java:382)
> at
> org.apache.hadoop.fs.DelegationTokenRenewer$RenewAction.cancel(DelegationTokenRenewer.java:146)
> at
> org.apache.hadoop.fs.DelegationTokenRenewer$RenewAction.access$200(DelegationTokenRenewer.java:58)
> at
> org.a
[jira] [Commented] (HDFS-4948) mvn site for hadoop-hdfs-nfs fails
[
https://issues.apache.org/jira/browse/HDFS-4948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13698367#comment-13698367
]
Suresh Srinivas commented on HDFS-4948:
---
+1 for the change.
> mvn site for hadoop-hdfs-nfs fails
> --
>
> Key: HDFS-4948
> URL: https://issues.apache.org/jira/browse/HDFS-4948
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Robert Joseph Evans
>Assignee: Brandon Li
> Attachments: HDFS-4948.patch
>
>
> Running mvn site on trunk results in the following error.
> {noformat}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-antrun-plugin:1.6:run (default) on project
> hadoop-hdfs-nfs: An Ant BuildException has occured: Warning: Could not find
> file
> /home/evans/src/hadoop-git/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/resources/hdfs-nfs-default.xml
> to copy. -> [Help 1]
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4942) Add retry cache support in Namenode
[ https://issues.apache.org/jira/browse/HDFS-4942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-4942: -- Attachment: HDFSRetryCache.pdf Here is the early version of the design document. Still pending is the analysis of all the Namenode RPC requests. I will post that soon. Given that we plan on adding a unique identifier to every RPC request, should we get this change done before 2.1.0-beta rc2 is built? This way 2.1.0-beta clients can utilize retry cache as well. > Add retry cache support in Namenode > --- > > Key: HDFS-4942 > URL: https://issues.apache.org/jira/browse/HDFS-4942 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ha, namenode >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Attachments: HDFSRetryCache.pdf > > > In current HA mechanism with FailoverProxyProvider and non HA setups with > RetryProxy retry a request from the RPC layer. If the retried request has > already been processed at the namenode, the subsequent attempts fail for > non-idempotent operations such as create, append, delete, rename etc. This > will cause application failures during HA failover, network issues etc. > This jira proposes adding retry cache at the namenode to handle these > failures. More details in the comments. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4750) Support NFSv3 interface to HDFS
[ https://issues.apache.org/jira/browse/HDFS-4750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13698301#comment-13698301 ] Suresh Srinivas commented on HDFS-4750: --- [~brandonli] If this work is completed, given all the jiras going in, can you please merge this to branch 2.1? > Support NFSv3 interface to HDFS > --- > > Key: HDFS-4750 > URL: https://issues.apache.org/jira/browse/HDFS-4750 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 3.0.0 >Reporter: Brandon Li >Assignee: Brandon Li > Attachments: HADOOP-NFS-Proposal.pdf, HDFS-4750.patch, nfs-trunk.patch > > > Access HDFS is usually done through HDFS Client or webHDFS. Lack of seamless > integration with client’s file system makes it difficult for users and > impossible for some applications to access HDFS. NFS interface support is one > way for HDFS to have such easy integration. > This JIRA is to track the NFS protocol support for accessing HDFS. With HDFS > client, webHDFS and the NFS interface, HDFS will be easier to access and be > able support more applications and use cases. > We will upload the design document and the initial implementation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4948) mvn site for hadoop-hdfs-nfs fails
[
https://issues.apache.org/jira/browse/HDFS-4948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13698297#comment-13698297
]
Brandon Li commented on HDFS-4948:
--
Unit test is not needed since it's not a code change.
> mvn site for hadoop-hdfs-nfs fails
> --
>
> Key: HDFS-4948
> URL: https://issues.apache.org/jira/browse/HDFS-4948
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Robert Joseph Evans
>Assignee: Brandon Li
> Attachments: HDFS-4948.patch
>
>
> Running mvn site on trunk results in the following error.
> {noformat}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-antrun-plugin:1.6:run (default) on project
> hadoop-hdfs-nfs: An Ant BuildException has occured: Warning: Could not find
> file
> /home/evans/src/hadoop-git/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/resources/hdfs-nfs-default.xml
> to copy. -> [Help 1]
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4948) mvn site for hadoop-hdfs-nfs fails
[
https://issues.apache.org/jira/browse/HDFS-4948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13698288#comment-13698288
]
Hadoop QA commented on HDFS-4948:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12590502/HDFS-4948.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs-nfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4590//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4590//console
This message is automatically generated.
> mvn site for hadoop-hdfs-nfs fails
> --
>
> Key: HDFS-4948
> URL: https://issues.apache.org/jira/browse/HDFS-4948
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Robert Joseph Evans
>Assignee: Brandon Li
> Attachments: HDFS-4948.patch
>
>
> Running mvn site on trunk results in the following error.
> {noformat}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-antrun-plugin:1.6:run (default) on project
> hadoop-hdfs-nfs: An Ant BuildException has occured: Warning: Could not find
> file
> /home/evans/src/hadoop-git/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/resources/hdfs-nfs-default.xml
> to copy. -> [Help 1]
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4945) A Distributed and Cooperative NameNode Cluster for a Highly-Available HDFS
[ https://issues.apache.org/jira/browse/HDFS-4945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13698281#comment-13698281 ] Konstantin Shvachko commented on HDFS-4945: --- Yonghwan, the ideas sound interesting. But it looks to me like a new file system rather than a new feature of HDFS. Do you plan to replace HDFS or evolve it? I've been working on [Giraffa|http://code.google.com/a/apache-extras.org/p/giraffa/source/browse/?name=trunk] project. Is it similar to your ideas? You were saying "we" on several occasions. Who do you mean? > A Distributed and Cooperative NameNode Cluster for a Highly-Available HDFS > -- > > Key: HDFS-4945 > URL: https://issues.apache.org/jira/browse/HDFS-4945 > Project: Hadoop HDFS > Issue Type: New Feature > Components: auto-failover >Affects Versions: HA branch (HDFS-1623) >Reporter: Yonghwan Kim > Labels: documentation > > See the following comment for detailed description. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4948) mvn site for hadoop-hdfs-nfs fails
[
https://issues.apache.org/jira/browse/HDFS-4948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brandon Li updated HDFS-4948:
-
Status: Patch Available (was: Open)
> mvn site for hadoop-hdfs-nfs fails
> --
>
> Key: HDFS-4948
> URL: https://issues.apache.org/jira/browse/HDFS-4948
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Robert Joseph Evans
>Assignee: Brandon Li
> Attachments: HDFS-4948.patch
>
>
> Running mvn site on trunk results in the following error.
> {noformat}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-antrun-plugin:1.6:run (default) on project
> hadoop-hdfs-nfs: An Ant BuildException has occured: Warning: Could not find
> file
> /home/evans/src/hadoop-git/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/resources/hdfs-nfs-default.xml
> to copy. -> [Help 1]
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4950) newly created files in fuse_dfs appear to be length 0 for a while
[
https://issues.apache.org/jira/browse/HDFS-4950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HDFS-4950:
---
Attachment: 2013-07-02.incorrect-attr-trace.txt
2013-07-02.correct-noattr-trace.txt
here are two traces that demonstrate the problem. The issue seems to be that
after the fuse RELEASE operation, we never call GETATTR again when attribute
caching is enabled.
One easy fix for this would be to disable attribute caching entirely. This
would certainly fix the bug, but it might result in lower performance. As you
can see from the "correct noattr" trace, many more GETATTR operations are done
in this, all of which will hit the NameNode.
Can we live with the fuse attribute cache? This raises the question of how
we're "supposed" to invalidate the fuse_dfs attribute cache. I wasn't able to
find any documentation about this. I can see that fuse is checking the
attributes of the root directory after the release.
{code}
unique: 18, success, outsize: 16
unique: 19, opcode: RELEASE (18), nodeid: 2, insize: 64, pid: 0
release[140595351837776] flags: 0x8001
unique: 19, success, outsize: 16
unique: 20, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 6597
getattr /
unique: 20, success, outsize: 120
{code}
Is it possible that FUSE expects something to change there if a new file has
been added?
> newly created files in fuse_dfs appear to be length 0 for a while
> -
>
> Key: HDFS-4950
> URL: https://issues.apache.org/jira/browse/HDFS-4950
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: fuse-dfs
>Affects Versions: 2.2.0
>Reporter: Colin Patrick McCabe
> Attachments: 2013-07-02.correct-noattr-trace.txt,
> 2013-07-02.incorrect-attr-trace.txt
>
>
> For some reason, newly created files in fuse_dfs appear to be length 0 for a
> while.
> {code}
> cmccabe@keter:~> echo hi > hi
> cmccabe@keter:~> mv hi /mnt/tmp/hi
> cmccabe@keter:~> ls -l /mnt/tmp
> total 0
> -rw-r--r-- 1 cmccabe users 0 Jul 2 13:24 hi
> cmccabe@keter:~> cat /mnt/tmp/hi
> cmccabe@keter:~> cat /mnt/tmp/hi
> hi
> {code}
> Disabling FUSE attribute caching fixes this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4950) newly created files in fuse_dfs appear to be length 0 for a while due to attribute caching
[
https://issues.apache.org/jira/browse/HDFS-4950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HDFS-4950:
---
Summary: newly created files in fuse_dfs appear to be length 0 for a while
due to attribute caching (was: newly created files in fuse_dfs appear to be
length 0 for a while)
> newly created files in fuse_dfs appear to be length 0 for a while due to
> attribute caching
> --
>
> Key: HDFS-4950
> URL: https://issues.apache.org/jira/browse/HDFS-4950
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: fuse-dfs
>Affects Versions: 2.2.0
>Reporter: Colin Patrick McCabe
> Attachments: 2013-07-02.correct-noattr-trace.txt,
> 2013-07-02.incorrect-attr-trace.txt
>
>
> For some reason, newly created files in fuse_dfs appear to be length 0 for a
> while.
> {code}
> cmccabe@keter:~> echo hi > hi
> cmccabe@keter:~> mv hi /mnt/tmp/hi
> cmccabe@keter:~> ls -l /mnt/tmp
> total 0
> -rw-r--r-- 1 cmccabe users 0 Jul 2 13:24 hi
> cmccabe@keter:~> cat /mnt/tmp/hi
> cmccabe@keter:~> cat /mnt/tmp/hi
> hi
> {code}
> Disabling FUSE attribute caching fixes this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-4949) Centralized cache management in HDFS
[ https://issues.apache.org/jira/browse/HDFS-4949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins reassigned HDFS-4949: - Assignee: Andrew Wang > Centralized cache management in HDFS > > > Key: HDFS-4949 > URL: https://issues.apache.org/jira/browse/HDFS-4949 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: 3.0.0, 2.2.0 >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: caching-design-doc-2013-07-02.pdf > > > HDFS currently has no support for managing or exposing in-memory caches at > datanodes. This makes it harder for higher level application frameworks like > Hive, Pig, and Impala to effectively use cluster memory, because they cannot > explicitly cache important datasets or place their tasks for memory locality. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4950) newly created files in fuse_dfs appear to be length 0 for a while
[
https://issues.apache.org/jira/browse/HDFS-4950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HDFS-4950:
---
Component/s: fuse-dfs
Target Version/s: 2.2.0
Affects Version/s: 2.2.0
> newly created files in fuse_dfs appear to be length 0 for a while
> -
>
> Key: HDFS-4950
> URL: https://issues.apache.org/jira/browse/HDFS-4950
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: fuse-dfs
>Affects Versions: 2.2.0
>Reporter: Colin Patrick McCabe
>
> For some reason, newly created files in fuse_dfs appear to be length 0 for a
> while.
> {code}
> cmccabe@keter:~> echo hi > hi
> cmccabe@keter:~> mv hi /mnt/tmp/hi
> cmccabe@keter:~> ls -l /mnt/tmp
> total 0
> -rw-r--r-- 1 cmccabe users 0 Jul 2 13:24 hi
> cmccabe@keter:~> cat /mnt/tmp/hi
> cmccabe@keter:~> cat /mnt/tmp/hi
> hi
> {code}
> Disabling FUSE attribute caching fixes this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4950) newly created files in fuse_dfs appear to be length 0 for a while
Colin Patrick McCabe created HDFS-4950:
--
Summary: newly created files in fuse_dfs appear to be length 0 for
a while
Key: HDFS-4950
URL: https://issues.apache.org/jira/browse/HDFS-4950
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Colin Patrick McCabe
For some reason, newly created files in fuse_dfs appear to be length 0 for a
while.
{code}
cmccabe@keter:~> echo hi > hi
cmccabe@keter:~> mv hi /mnt/tmp/hi
cmccabe@keter:~> ls -l /mnt/tmp
total 0
-rw-r--r-- 1 cmccabe users 0 Jul 2 13:24 hi
cmccabe@keter:~> cat /mnt/tmp/hi
cmccabe@keter:~> cat /mnt/tmp/hi
hi
{code}
Disabling FUSE attribute caching fixes this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-3499) Make NetworkTopology support user specified topology class
[ https://issues.apache.org/jira/browse/HDFS-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du resolved HDFS-3499. -- Resolution: Fixed Fix Version/s: 1.2.0 2.1.0-beta HADOOP-8469 already address most of it, so resolve it here. > Make NetworkTopology support user specified topology class > -- > > Key: HDFS-3499 > URL: https://issues.apache.org/jira/browse/HDFS-3499 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Reporter: Junping Du >Assignee: Junping Du > Fix For: 2.1.0-beta, 1.2.0 > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4949) Centralized cache management in HDFS
[ https://issues.apache.org/jira/browse/HDFS-4949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-4949: -- Attachment: caching-design-doc-2013-07-02.pdf Here's a design doc that we've been working on internally. It proposes adding off-heap caches to each datanode using mmap and mlock, managed centrally by the NameNode. Any feedback welcomed. I'm hoping we can have a fruitful design discussion on this JIRA, then perhaps get a branch and start development. > Centralized cache management in HDFS > > > Key: HDFS-4949 > URL: https://issues.apache.org/jira/browse/HDFS-4949 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: 3.0.0, 2.2.0 >Reporter: Andrew Wang > Attachments: caching-design-doc-2013-07-02.pdf > > > HDFS currently has no support for managing or exposing in-memory caches at > datanodes. This makes it harder for higher level application frameworks like > Hive, Pig, and Impala to effectively use cluster memory, because they cannot > explicitly cache important datasets or place their tasks for memory locality. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4949) Centralized cache management in HDFS
Andrew Wang created HDFS-4949: - Summary: Centralized cache management in HDFS Key: HDFS-4949 URL: https://issues.apache.org/jira/browse/HDFS-4949 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, namenode Affects Versions: 3.0.0, 2.2.0 Reporter: Andrew Wang HDFS currently has no support for managing or exposing in-memory caches at datanodes. This makes it harder for higher level application frameworks like Hive, Pig, and Impala to effectively use cluster memory, because they cannot explicitly cache important datasets or place their tasks for memory locality. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4948) mvn site for hadoop-hdfs-nfs fails
[
https://issues.apache.org/jira/browse/HDFS-4948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brandon Li updated HDFS-4948:
-
Attachment: HDFS-4948.patch
Upload the patch which cleans the hadoop-hdfs-nfs/pom.xml file.
> mvn site for hadoop-hdfs-nfs fails
> --
>
> Key: HDFS-4948
> URL: https://issues.apache.org/jira/browse/HDFS-4948
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Robert Joseph Evans
>Assignee: Brandon Li
> Attachments: HDFS-4948.patch
>
>
> Running mvn site on trunk results in the following error.
> {noformat}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-antrun-plugin:1.6:run (default) on project
> hadoop-hdfs-nfs: An Ant BuildException has occured: Warning: Could not find
> file
> /home/evans/src/hadoop-git/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/resources/hdfs-nfs-default.xml
> to copy. -> [Help 1]
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4948) mvn site for hadoop-hdfs-nfs fails
[
https://issues.apache.org/jira/browse/HDFS-4948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brandon Li updated HDFS-4948:
-
Assignee: Brandon Li
> mvn site for hadoop-hdfs-nfs fails
> --
>
> Key: HDFS-4948
> URL: https://issues.apache.org/jira/browse/HDFS-4948
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Robert Joseph Evans
>Assignee: Brandon Li
>
> Running mvn site on trunk results in the following error.
> {noformat}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-antrun-plugin:1.6:run (default) on project
> hadoop-hdfs-nfs: An Ant BuildException has occured: Warning: Could not find
> file
> /home/evans/src/hadoop-git/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/resources/hdfs-nfs-default.xml
> to copy. -> [Help 1]
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4948) mvn site for hadoop-hdfs-nfs fails
Robert Joseph Evans created HDFS-4948:
-
Summary: mvn site for hadoop-hdfs-nfs fails
Key: HDFS-4948
URL: https://issues.apache.org/jira/browse/HDFS-4948
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Robert Joseph Evans
Running mvn site on trunk results in the following error.
{noformat}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-antrun-plugin:1.6:run (default) on project
hadoop-hdfs-nfs: An Ant BuildException has occured: Warning: Could not find
file
/home/evans/src/hadoop-git/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/resources/hdfs-nfs-default.xml
to copy. -> [Help 1]
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4860) Add additional attributes to JMX beans
[
https://issues.apache.org/jira/browse/HDFS-4860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13698104#comment-13698104
]
Hadoop QA commented on HDFS-4860:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12590460/HDFS-4860-3.diff
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4589//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4589//console
This message is automatically generated.
> Add additional attributes to JMX beans
> --
>
> Key: HDFS-4860
> URL: https://issues.apache.org/jira/browse/HDFS-4860
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 0.20.204.1, 3.0.0, 2.1.0-beta, 2.0.4-alpha
>Reporter: Trevor Lorimer
> Attachments: HDFS-4860-3.diff
>
>
> Currently the JMX bean returns much of the data contained on the HDFS Health
> webpage (dfsHealth.html). However there are several other attributes that are
> required to be added.
> I intend to add the following items to the appropriate bean in parenthesis :
> Started time (NameNodeInfo),
> Compiled info (NameNodeInfo),
> Jvm MaxHeap, MaxNonHeap (JvmMetrics)
> Node Usage stats (i.e. Min, Median, Max, stdev) (NameNodeInfo),
> Count of decommissioned Live and Dead nodes (FSNamesystemState),
> Journal Status (NodeNameInfo)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4851) Deadlock in pipeline recovery
[
https://issues.apache.org/jira/browse/HDFS-4851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13698073#comment-13698073
]
Hadoop QA commented on HDFS-4851:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12589794/hdfs-4851-1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4588//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4588//console
This message is automatically generated.
> Deadlock in pipeline recovery
> -
>
> Key: HDFS-4851
> URL: https://issues.apache.org/jira/browse/HDFS-4851
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 3.0.0, 2.0.4-alpha
>Reporter: Andrew Wang
>Assignee: Andrew Wang
> Attachments: hdfs-4851-1.patch
>
>
> Here's a deadlock scenario that cropped up during pipeline recovery, debugged
> through jstacks. Todd tipped me off to this one.
> # Pipeline fails, client initiates recovery. We have the old leftover
> DataXceiver, and a new one doing recovery.
> # New DataXceiver does {{recoverRbw}}, grabbing the {{FsDatasetImpl}} lock
> # Old DataXceiver is in {{BlockReceiver#computePartialChunkCrc}}, calls
> {{FsDatasetImpl#getTmpInputStreams}} and blocks on the {{FsDatasetImpl}} lock.
> # New DataXceiver {{ReplicaInPipeline#stopWriter}}, interrupting the old
> DataXceiver and then joining on it.
> # Boom, deadlock. New DX holds the {{FsDatasetImpl}} lock and is joining on
> the old DX, which is in turn waiting on the {{FsDatasetImpl}} lock.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4762) Provide HDFS based NFSv3 and Mountd implementation
[ https://issues.apache.org/jira/browse/HDFS-4762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13698048#comment-13698048 ] Brandon Li commented on HDFS-4762: -- My bad. I missed some files. Let me fix it. > Provide HDFS based NFSv3 and Mountd implementation > -- > > Key: HDFS-4762 > URL: https://issues.apache.org/jira/browse/HDFS-4762 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Affects Versions: 3.0.0 >Reporter: Brandon Li >Assignee: Brandon Li > Fix For: 3.0.0 > > Attachments: HDFS-4762.patch, HDFS-4762.patch.2, HDFS-4762.patch.3, > HDFS-4762.patch.3, HDFS-4762.patch.4, HDFS-4762.patch.5, HDFS-4762.patch.6, > HDFS-4762.patch.7 > > > This is to track the implementation of NFSv3 to HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4762) Provide HDFS based NFSv3 and Mountd implementation
[
https://issues.apache.org/jira/browse/HDFS-4762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13698039#comment-13698039
]
Kihwal Lee commented on HDFS-4762:
--
The trunk build is failing for me. Am I doing something wrong?
{noformat}
[ERROR] The project org.apache.hadoop:hadoop-hdfs-nfs:3.0.0-SNAPSHOT
(/home/kihwal/devel/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/pom.xml) has 1
error
[ERROR] 'dependencies.dependency.version' for
org.apache.hadoop:hadoop-nfs:jar is missing. @ line 44, column 17
{noformat}
> Provide HDFS based NFSv3 and Mountd implementation
> --
>
> Key: HDFS-4762
> URL: https://issues.apache.org/jira/browse/HDFS-4762
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: nfs
>Affects Versions: 3.0.0
>Reporter: Brandon Li
>Assignee: Brandon Li
> Fix For: 3.0.0
>
> Attachments: HDFS-4762.patch, HDFS-4762.patch.2, HDFS-4762.patch.3,
> HDFS-4762.patch.3, HDFS-4762.patch.4, HDFS-4762.patch.5, HDFS-4762.patch.6,
> HDFS-4762.patch.7
>
>
> This is to track the implementation of NFSv3 to HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4860) Add additional attributes to JMX beans
[ https://issues.apache.org/jira/browse/HDFS-4860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13698036#comment-13698036 ] Trevor Lorimer commented on HDFS-4860: -- Thanks for the notes Todd, in the previous version I was splitting the string for display reasons. For this latest update I am just sending the data as it is presented from JournalManager, the data can be reformatted as required by the consumer. > Add additional attributes to JMX beans > -- > > Key: HDFS-4860 > URL: https://issues.apache.org/jira/browse/HDFS-4860 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 0.20.204.1, 3.0.0, 2.1.0-beta, 2.0.4-alpha >Reporter: Trevor Lorimer > Attachments: HDFS-4860-3.diff > > > Currently the JMX bean returns much of the data contained on the HDFS Health > webpage (dfsHealth.html). However there are several other attributes that are > required to be added. > I intend to add the following items to the appropriate bean in parenthesis : > Started time (NameNodeInfo), > Compiled info (NameNodeInfo), > Jvm MaxHeap, MaxNonHeap (JvmMetrics) > Node Usage stats (i.e. Min, Median, Max, stdev) (NameNodeInfo), > Count of decommissioned Live and Dead nodes (FSNamesystemState), > Journal Status (NodeNameInfo) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4762) Provide HDFS based NFSv3 and Mountd implementation
[ https://issues.apache.org/jira/browse/HDFS-4762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13698025#comment-13698025 ] Hudson commented on HDFS-4762: -- Integrated in Hadoop-trunk-Commit #4030 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4030/]) HDFS-4762 Provide HDFS based NFSv3 and Mountd implementation. Contributed by Brandon Li (Revision 1499029) Result = FAILURE brandonli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1499029 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/README.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/dev-support * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/dev-support/findbugsExcludeFile.xml * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/pom.xml * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/mount * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/mount/Mountd.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/mount/RpcProgramMountd.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3 * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/AsyncDataService.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/DFSClientCache.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/LruCache.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/Nfs3.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/Nfs3Utils.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/OffsetRange.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/OpenFileCtx.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/RpcProgramNfs3.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/WriteCtx.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/WriteManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/test * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/test/java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/test/java/org * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/test/java/org/apache * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/test/java/org/apache/hadoop * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/test/java/org/apache/hadoop/hdfs * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/test/java/org/apache/hadoop/hdfs/nfs * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/test/java/org/apache/hadoop/hdfs/nfs/TestMountd.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/test/java/org/apache/hadoop/hdfs/nfs/TestOutOfOrderWrite.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/test/java/org/apache/hadoop/hdfs/nfs/TestPortmapRegister.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/test/java/org/apache/hadoop/hdfs/nfs/TestUdpServer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/test/java/org/apache/hadoop/hdfs/nfs/nfs3 * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/test/java/org/apache/hadoop/hdfs/nfs/nfs3/TestDFSClientCache.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/test/java/org/apache/hadoop/hdfs/nfs/nfs3/TestOffsetRange.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/test/java/org/apache/hadoop/hdfs/nfs/nfs3/TestRpcProgramNfs3.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/pom.xml > Provide
[jira] [Created] (HDFS-4947) Add NFS server export table to control export by hostname or IP range
Brandon Li created HDFS-4947: Summary: Add NFS server export table to control export by hostname or IP range Key: HDFS-4947 URL: https://issues.apache.org/jira/browse/HDFS-4947 Project: Hadoop HDFS Issue Type: Sub-task Affects Versions: 3.0.0 Reporter: Brandon Li Assignee: Jing Zhao -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4762) Provide HDFS based NFSv3 and Mountd implementation
[ https://issues.apache.org/jira/browse/HDFS-4762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13698021#comment-13698021 ] Brandon Li commented on HDFS-4762: -- Thank you, Nicholas. I've committed the patch to trunk. > Provide HDFS based NFSv3 and Mountd implementation > -- > > Key: HDFS-4762 > URL: https://issues.apache.org/jira/browse/HDFS-4762 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Affects Versions: 3.0.0 >Reporter: Brandon Li >Assignee: Brandon Li > Attachments: HDFS-4762.patch, HDFS-4762.patch.2, HDFS-4762.patch.3, > HDFS-4762.patch.3, HDFS-4762.patch.4, HDFS-4762.patch.5, HDFS-4762.patch.6, > HDFS-4762.patch.7 > > > This is to track the implementation of NFSv3 to HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4762) Provide HDFS based NFSv3 and Mountd implementation
[ https://issues.apache.org/jira/browse/HDFS-4762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-4762: - Fix Version/s: 3.0.0 > Provide HDFS based NFSv3 and Mountd implementation > -- > > Key: HDFS-4762 > URL: https://issues.apache.org/jira/browse/HDFS-4762 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Affects Versions: 3.0.0 >Reporter: Brandon Li >Assignee: Brandon Li > Fix For: 3.0.0 > > Attachments: HDFS-4762.patch, HDFS-4762.patch.2, HDFS-4762.patch.3, > HDFS-4762.patch.3, HDFS-4762.patch.4, HDFS-4762.patch.5, HDFS-4762.patch.6, > HDFS-4762.patch.7 > > > This is to track the implementation of NFSv3 to HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4762) Provide HDFS based NFSv3 and Mountd implementation
[ https://issues.apache.org/jira/browse/HDFS-4762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-4762: - Resolution: Fixed Target Version/s: 3.0.0 Status: Resolved (was: Patch Available) > Provide HDFS based NFSv3 and Mountd implementation > -- > > Key: HDFS-4762 > URL: https://issues.apache.org/jira/browse/HDFS-4762 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Affects Versions: 3.0.0 >Reporter: Brandon Li >Assignee: Brandon Li > Attachments: HDFS-4762.patch, HDFS-4762.patch.2, HDFS-4762.patch.3, > HDFS-4762.patch.3, HDFS-4762.patch.4, HDFS-4762.patch.5, HDFS-4762.patch.6, > HDFS-4762.patch.7 > > > This is to track the implementation of NFSv3 to HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-4841) FsShell commands using secure webhfds fail ClientFinalizer shutdown hook
[
https://issues.apache.org/jira/browse/HDFS-4841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter reassigned HDFS-4841:
---
Assignee: Robert Kanter
> FsShell commands using secure webhfds fail ClientFinalizer shutdown hook
>
>
> Key: HDFS-4841
> URL: https://issues.apache.org/jira/browse/HDFS-4841
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security, webhdfs
>Affects Versions: 3.0.0
>Reporter: Stephen Chu
>Assignee: Robert Kanter
> Attachments: core-site.xml,
> hadoop-root-namenode-hdfs-upgrade-pseudo.ent.cloudera.com.out, hdfs-site.xml,
> jsvc.out
>
>
> Hadoop version:
> {code}
> bash-4.1$ $HADOOP_HOME/bin/hadoop version
> Hadoop 3.0.0-SNAPSHOT
> Subversion git://github.com/apache/hadoop-common.git -r
> d5373b9c550a355d4e91330ba7cc8f4c7c3aac51
> Compiled by root on 2013-05-22T08:06Z
> From source with checksum 8c4cc9b1e8d6e8361431e00f64483f
> This command was run using
> /var/lib/hadoop-hdfs/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/hadoop-common-3.0.0-SNAPSHOT.jar
> {code}
> I'm seeing a problem when issuing FsShell commands using the webhdfs:// URI
> when security is enabled. The command completes but leaves a warning that
> ShutdownHook 'ClientFinalizer' failed.
> {code}
> bash-4.1$ hadoop-3.0.0-SNAPSHOT/bin/hadoop fs -ls
> webhdfs://hdfs-upgrade-pseudo.ent.cloudera.com:50070/
> 2013-05-22 09:46:55,710 INFO [main] util.Shell
> (Shell.java:isSetsidSupported(311)) - setsid exited with exit code 0
> Found 3 items
> drwxr-xr-x - hbase supergroup 0 2013-05-22 09:46
> webhdfs://hdfs-upgrade-pseudo.ent.cloudera.com:50070/hbase
> drwxr-xr-x - hdfs supergroup 0 2013-05-22 09:46
> webhdfs://hdfs-upgrade-pseudo.ent.cloudera.com:50070/tmp
> drwxr-xr-x - hdfs supergroup 0 2013-05-22 09:46
> webhdfs://hdfs-upgrade-pseudo.ent.cloudera.com:50070/user
> 2013-05-22 09:46:58,660 WARN [Thread-3] util.ShutdownHookManager
> (ShutdownHookManager.java:run(56)) - ShutdownHook 'ClientFinalizer' failed,
> java.lang.IllegalStateException: Shutdown in progress, cannot add a
> shutdownHook
> java.lang.IllegalStateException: Shutdown in progress, cannot add a
> shutdownHook
> at
> org.apache.hadoop.util.ShutdownHookManager.addShutdownHook(ShutdownHookManager.java:152)
> at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2400)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2372)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:352)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$DtRenewer.getWebHdfs(WebHdfsFileSystem.java:1001)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$DtRenewer.cancel(WebHdfsFileSystem.java:1013)
> at org.apache.hadoop.security.token.Token.cancel(Token.java:382)
> at
> org.apache.hadoop.fs.DelegationTokenRenewer$RenewAction.cancel(DelegationTokenRenewer.java:152)
> at
> org.apache.hadoop.fs.DelegationTokenRenewer$RenewAction.access$200(DelegationTokenRenewer.java:58)
> at
> org.apache.hadoop.fs.DelegationTokenRenewer.removeRenewAction(DelegationTokenRenewer.java:241)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.close(WebHdfsFileSystem.java:822)
> at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2446)
> at
> org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:2463)
> at
> org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
> {code}
> I've checked that FsShell + hdfs:// commands and WebHDFS operations through
> curl work successfully:
> {code}
> bash-4.1$ hadoop-3.0.0-SNAPSHOT/bin/hadoop fs -ls /
> 2013-05-22 09:46:43,663 INFO [main] util.Shell
> (Shell.java:isSetsidSupported(311)) - setsid exited with exit code 0
> Found 3 items
> drwxr-xr-x - hbase supergroup 0 2013-05-22 09:46 /hbase
> drwxr-xr-x - hdfs supergroup 0 2013-05-22 09:46 /tmp
> drwxr-xr-x - hdfs supergroup 0 2013-05-22 09:46 /user
> bash-4.1$ curl -i --negotiate -u :
> "http://hdfs-upgrade-pseudo.ent.cloudera.com:50070/webhdfs/v1/?op=GETHOMEDIRECTORY";
> HTTP/1.1 401
> Cache-Control: must-revalidate,no-cache,no-store
> Date: Wed, 22 May 2013 16:47:14 GMT
> Pragma: no-cache
> Date: Wed, 22 May 2013 16:47:14 GMT
> Pragma: no-cache
> Content-Type: text/html; charset=iso-8859-1
> WWW-Authenticate: Negotiate
> Set-Cookie: hadoop.auth=;Path=/;Expires=Thu, 01-Jan-1970 00:00:00 GMT
> Content-Length: 1358
> Server: Jetty(6.1.26)
> HTTP/1.1 200 OK
> Cache-Control: no-cache
> Expires: Thu, 01-Jan-1970 00:00:00 GMT
> Date: Wed, 22 May 2013 16:47:14 GMT
> Pragma: no-cache
> Date: Wed, 22 May 2013 16:47:14 GMT
> Pragma: no-cache
> Content-Type: application/json
> Set-Cookie:
> hadoop.auth="u=hd
[jira] [Updated] (HDFS-4860) Add additional attributes to JMX beans
[ https://issues.apache.org/jira/browse/HDFS-4860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Trevor Lorimer updated HDFS-4860: - Status: Patch Available (was: Open) > Add additional attributes to JMX beans > -- > > Key: HDFS-4860 > URL: https://issues.apache.org/jira/browse/HDFS-4860 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.0.4-alpha, 0.20.204.1, 3.0.0, 2.1.0-beta >Reporter: Trevor Lorimer > Attachments: HDFS-4860-3.diff > > > Currently the JMX bean returns much of the data contained on the HDFS Health > webpage (dfsHealth.html). However there are several other attributes that are > required to be added. > I intend to add the following items to the appropriate bean in parenthesis : > Started time (NameNodeInfo), > Compiled info (NameNodeInfo), > Jvm MaxHeap, MaxNonHeap (JvmMetrics) > Node Usage stats (i.e. Min, Median, Max, stdev) (NameNodeInfo), > Count of decommissioned Live and Dead nodes (FSNamesystemState), > Journal Status (NodeNameInfo) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4860) Add additional attributes to JMX beans
[ https://issues.apache.org/jira/browse/HDFS-4860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Trevor Lorimer updated HDFS-4860: - Attachment: HDFS-4860-3.diff Uploaded new patch HDFS-4860-3.diff > Add additional attributes to JMX beans > -- > > Key: HDFS-4860 > URL: https://issues.apache.org/jira/browse/HDFS-4860 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 0.20.204.1, 3.0.0, 2.1.0-beta, 2.0.4-alpha >Reporter: Trevor Lorimer > Attachments: HDFS-4860-3.diff > > > Currently the JMX bean returns much of the data contained on the HDFS Health > webpage (dfsHealth.html). However there are several other attributes that are > required to be added. > I intend to add the following items to the appropriate bean in parenthesis : > Started time (NameNodeInfo), > Compiled info (NameNodeInfo), > Jvm MaxHeap, MaxNonHeap (JvmMetrics) > Node Usage stats (i.e. Min, Median, Max, stdev) (NameNodeInfo), > Count of decommissioned Live and Dead nodes (FSNamesystemState), > Journal Status (NodeNameInfo) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4860) Add additional attributes to JMX beans
[ https://issues.apache.org/jira/browse/HDFS-4860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Trevor Lorimer updated HDFS-4860: - Status: Open (was: Patch Available) > Add additional attributes to JMX beans > -- > > Key: HDFS-4860 > URL: https://issues.apache.org/jira/browse/HDFS-4860 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.0.4-alpha, 0.20.204.1, 3.0.0, 2.1.0-beta >Reporter: Trevor Lorimer > Attachments: HDFS-4860-3.diff > > > Currently the JMX bean returns much of the data contained on the HDFS Health > webpage (dfsHealth.html). However there are several other attributes that are > required to be added. > I intend to add the following items to the appropriate bean in parenthesis : > Started time (NameNodeInfo), > Compiled info (NameNodeInfo), > Jvm MaxHeap, MaxNonHeap (JvmMetrics) > Node Usage stats (i.e. Min, Median, Max, stdev) (NameNodeInfo), > Count of decommissioned Live and Dead nodes (FSNamesystemState), > Journal Status (NodeNameInfo) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4860) Add additional attributes to JMX beans
[ https://issues.apache.org/jira/browse/HDFS-4860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Trevor Lorimer updated HDFS-4860: - Attachment: (was: HDFS-4860.diff) > Add additional attributes to JMX beans > -- > > Key: HDFS-4860 > URL: https://issues.apache.org/jira/browse/HDFS-4860 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 0.20.204.1, 3.0.0, 2.1.0-beta, 2.0.4-alpha >Reporter: Trevor Lorimer > > Currently the JMX bean returns much of the data contained on the HDFS Health > webpage (dfsHealth.html). However there are several other attributes that are > required to be added. > I intend to add the following items to the appropriate bean in parenthesis : > Started time (NameNodeInfo), > Compiled info (NameNodeInfo), > Jvm MaxHeap, MaxNonHeap (JvmMetrics) > Node Usage stats (i.e. Min, Median, Max, stdev) (NameNodeInfo), > Count of decommissioned Live and Dead nodes (FSNamesystemState), > Journal Status (NodeNameInfo) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4860) Add additional attributes to JMX beans
[
https://issues.apache.org/jira/browse/HDFS-4860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13697921#comment-13697921
]
Hadoop QA commented on HDFS-4860:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12590431/HDFS-4860.diff
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4586//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4586//console
This message is automatically generated.
> Add additional attributes to JMX beans
> --
>
> Key: HDFS-4860
> URL: https://issues.apache.org/jira/browse/HDFS-4860
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 0.20.204.1, 3.0.0, 2.1.0-beta, 2.0.4-alpha
>Reporter: Trevor Lorimer
> Attachments: HDFS-4860.diff
>
>
> Currently the JMX bean returns much of the data contained on the HDFS Health
> webpage (dfsHealth.html). However there are several other attributes that are
> required to be added.
> I intend to add the following items to the appropriate bean in parenthesis :
> Started time (NameNodeInfo),
> Compiled info (NameNodeInfo),
> Jvm MaxHeap, MaxNonHeap (JvmMetrics)
> Node Usage stats (i.e. Min, Median, Max, stdev) (NameNodeInfo),
> Count of decommissioned Live and Dead nodes (FSNamesystemState),
> Journal Status (NodeNameInfo)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4943) WebHdfsFileSystem does not work when original file path has encoded chars
[ https://issues.apache.org/jira/browse/HDFS-4943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13697859#comment-13697859 ] Hudson commented on HDFS-4943: -- Integrated in Hadoop-trunk-Commit #4029 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4029/]) HDFS-4943. WebHdfsFileSystem does not work when original file path has encoded chars. Contributed by Jerry He (Revision 1498962) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1498962 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsUrl.java > WebHdfsFileSystem does not work when original file path has encoded chars > -- > > Key: HDFS-4943 > URL: https://issues.apache.org/jira/browse/HDFS-4943 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 1.2.0, 1.1.2, 2.0.4-alpha >Reporter: Jerry He >Assignee: Jerry He >Priority: Minor > Fix For: 2.1.0-beta > > Attachments: HDFS-4943-trunk.patch, HDFS-4943-trunk-v2.patch > > > In HBase, the WAL (hlog) file name on hdfs is URL encoded. For example, > hdtest010%2C60020%2C1371000602151.1371058984668 > When we use webhdfs client to access the hlog file via httpfs, it does not > work in this case. > $ hadoop fs -ls hdfs:///user/biadmin/hbase_hlogs > > Found 1 items > -rw-r--r-- 3 biadmin supergroup 15049470 2013-06-12 10:45 > /user/biadmin/hbase_hlogs/hdtest010%2C60020%2C1371000602151.1371058984668 > $ hadoop fs -ls > hdfs:///user/biadmin/hbase_hlogs/hdtest010%2C60020%2C1371000602151.1371058984668 > Found 1 items > -rw-r--r-- 3 biadmin supergroup 15049470 2013-06-12 10:45 > /user/biadmin/hbase_hlogs/hdtest010%2C60020%2C1371000602151.1371058984668 > $ hadoop fs -ls webhdfs://hdtest010:14000/user/biadmin/hbase_hlogs > Found 1 items > -rw-r--r-- 3 biadmin supergroup 15049470 2013-06-12 10:45 > /user/biadmin/hbase_hlogs/hdtest010%2C60020%2C1371000602151.1371058984668 > $ > $ hadoop fs -ls > webhdfs://hdtest010:14000/user/biadmin/hbase_hlogs/hdtest010%2C60020%2C1371000602151.1371058984668 > 13/06/27 18:36:08 DEBUG web.WebHdfsFileSystem: Original exception is > org.apache.hadoop.ipc.RemoteException:java.io.FileNotFoundException:File does > not exist: > /user/biadmin/hbase_hlogs/hdtest010,60020,1371000602151.1371058984668 > at > org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:114) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:299) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$500(WebHdfsFileSystem.java:104) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem$Runner.getResponse(WebHdfsFileSystem.java:641) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem$Runner.run(WebHdfsFileSystem.java:538) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem.run(WebHdfsFileSystem.java:468) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus(WebHdfsFileSystem.java:662) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileStatus(WebHdfsFileSystem.java:673) > at org.apache.hadoop.fs.FileSystem.getFileStatus(FileSystem.java:1365) > at > org.apache.hadoop.fs.FileSystem.globStatusInternal(FileSystem.java:1048) > at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:987) > at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:965) > at org.apache.hadoop.fs.FsShell.ls(FsShell.java:573) > at org.apache.hadoop.fs.FsShell.doall(FsShell.java:1571) > at org.apache.hadoop.fs.FsShell.run(FsShell.java:1789) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) > at org.apache.hadoop.fs.FsShell.main(FsShell.java:1895) > ls: Cannot access > webhdfs://hdtest010:14000/user/biadmin/hbase_hlogs/hdtest010%2C60020%2C1371000602151.1371058984668: > No such file or directory. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4943) WebHdfsFileSystem does not work when original file path has encoded chars
[ https://issues.apache.org/jira/browse/HDFS-4943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-4943: - Resolution: Fixed Status: Resolved (was: Patch Available) > WebHdfsFileSystem does not work when original file path has encoded chars > -- > > Key: HDFS-4943 > URL: https://issues.apache.org/jira/browse/HDFS-4943 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 1.2.0, 1.1.2, 2.0.4-alpha >Reporter: Jerry He >Assignee: Jerry He >Priority: Minor > Fix For: 2.1.0-beta > > Attachments: HDFS-4943-trunk.patch, HDFS-4943-trunk-v2.patch > > > In HBase, the WAL (hlog) file name on hdfs is URL encoded. For example, > hdtest010%2C60020%2C1371000602151.1371058984668 > When we use webhdfs client to access the hlog file via httpfs, it does not > work in this case. > $ hadoop fs -ls hdfs:///user/biadmin/hbase_hlogs > > Found 1 items > -rw-r--r-- 3 biadmin supergroup 15049470 2013-06-12 10:45 > /user/biadmin/hbase_hlogs/hdtest010%2C60020%2C1371000602151.1371058984668 > $ hadoop fs -ls > hdfs:///user/biadmin/hbase_hlogs/hdtest010%2C60020%2C1371000602151.1371058984668 > Found 1 items > -rw-r--r-- 3 biadmin supergroup 15049470 2013-06-12 10:45 > /user/biadmin/hbase_hlogs/hdtest010%2C60020%2C1371000602151.1371058984668 > $ hadoop fs -ls webhdfs://hdtest010:14000/user/biadmin/hbase_hlogs > Found 1 items > -rw-r--r-- 3 biadmin supergroup 15049470 2013-06-12 10:45 > /user/biadmin/hbase_hlogs/hdtest010%2C60020%2C1371000602151.1371058984668 > $ > $ hadoop fs -ls > webhdfs://hdtest010:14000/user/biadmin/hbase_hlogs/hdtest010%2C60020%2C1371000602151.1371058984668 > 13/06/27 18:36:08 DEBUG web.WebHdfsFileSystem: Original exception is > org.apache.hadoop.ipc.RemoteException:java.io.FileNotFoundException:File does > not exist: > /user/biadmin/hbase_hlogs/hdtest010,60020,1371000602151.1371058984668 > at > org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:114) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:299) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$500(WebHdfsFileSystem.java:104) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem$Runner.getResponse(WebHdfsFileSystem.java:641) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem$Runner.run(WebHdfsFileSystem.java:538) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem.run(WebHdfsFileSystem.java:468) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus(WebHdfsFileSystem.java:662) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileStatus(WebHdfsFileSystem.java:673) > at org.apache.hadoop.fs.FileSystem.getFileStatus(FileSystem.java:1365) > at > org.apache.hadoop.fs.FileSystem.globStatusInternal(FileSystem.java:1048) > at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:987) > at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:965) > at org.apache.hadoop.fs.FsShell.ls(FsShell.java:573) > at org.apache.hadoop.fs.FsShell.doall(FsShell.java:1571) > at org.apache.hadoop.fs.FsShell.run(FsShell.java:1789) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) > at org.apache.hadoop.fs.FsShell.main(FsShell.java:1895) > ls: Cannot access > webhdfs://hdtest010:14000/user/biadmin/hbase_hlogs/hdtest010%2C60020%2C1371000602151.1371058984668: > No such file or directory. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4943) WebHdfsFileSystem does not work when original file path has encoded chars
[ https://issues.apache.org/jira/browse/HDFS-4943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-4943: - Fix Version/s: 2.1.0-beta I have committed this. Thanks, Jerry! > WebHdfsFileSystem does not work when original file path has encoded chars > -- > > Key: HDFS-4943 > URL: https://issues.apache.org/jira/browse/HDFS-4943 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 1.2.0, 1.1.2, 2.0.4-alpha >Reporter: Jerry He >Assignee: Jerry He >Priority: Minor > Fix For: 2.1.0-beta > > Attachments: HDFS-4943-trunk.patch, HDFS-4943-trunk-v2.patch > > > In HBase, the WAL (hlog) file name on hdfs is URL encoded. For example, > hdtest010%2C60020%2C1371000602151.1371058984668 > When we use webhdfs client to access the hlog file via httpfs, it does not > work in this case. > $ hadoop fs -ls hdfs:///user/biadmin/hbase_hlogs > > Found 1 items > -rw-r--r-- 3 biadmin supergroup 15049470 2013-06-12 10:45 > /user/biadmin/hbase_hlogs/hdtest010%2C60020%2C1371000602151.1371058984668 > $ hadoop fs -ls > hdfs:///user/biadmin/hbase_hlogs/hdtest010%2C60020%2C1371000602151.1371058984668 > Found 1 items > -rw-r--r-- 3 biadmin supergroup 15049470 2013-06-12 10:45 > /user/biadmin/hbase_hlogs/hdtest010%2C60020%2C1371000602151.1371058984668 > $ hadoop fs -ls webhdfs://hdtest010:14000/user/biadmin/hbase_hlogs > Found 1 items > -rw-r--r-- 3 biadmin supergroup 15049470 2013-06-12 10:45 > /user/biadmin/hbase_hlogs/hdtest010%2C60020%2C1371000602151.1371058984668 > $ > $ hadoop fs -ls > webhdfs://hdtest010:14000/user/biadmin/hbase_hlogs/hdtest010%2C60020%2C1371000602151.1371058984668 > 13/06/27 18:36:08 DEBUG web.WebHdfsFileSystem: Original exception is > org.apache.hadoop.ipc.RemoteException:java.io.FileNotFoundException:File does > not exist: > /user/biadmin/hbase_hlogs/hdtest010,60020,1371000602151.1371058984668 > at > org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:114) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:299) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$500(WebHdfsFileSystem.java:104) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem$Runner.getResponse(WebHdfsFileSystem.java:641) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem$Runner.run(WebHdfsFileSystem.java:538) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem.run(WebHdfsFileSystem.java:468) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus(WebHdfsFileSystem.java:662) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileStatus(WebHdfsFileSystem.java:673) > at org.apache.hadoop.fs.FileSystem.getFileStatus(FileSystem.java:1365) > at > org.apache.hadoop.fs.FileSystem.globStatusInternal(FileSystem.java:1048) > at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:987) > at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:965) > at org.apache.hadoop.fs.FsShell.ls(FsShell.java:573) > at org.apache.hadoop.fs.FsShell.doall(FsShell.java:1571) > at org.apache.hadoop.fs.FsShell.run(FsShell.java:1789) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) > at org.apache.hadoop.fs.FsShell.main(FsShell.java:1895) > ls: Cannot access > webhdfs://hdtest010:14000/user/biadmin/hbase_hlogs/hdtest010%2C60020%2C1371000602151.1371058984668: > No such file or directory. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4943) WebHdfsFileSystem does not work when original file path has encoded chars
[ https://issues.apache.org/jira/browse/HDFS-4943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-4943: - Fix Version/s: (was: 2.1.0-beta) Assignee: Jerry He Hadoop Flags: Reviewed +1 patch looks good. > WebHdfsFileSystem does not work when original file path has encoded chars > -- > > Key: HDFS-4943 > URL: https://issues.apache.org/jira/browse/HDFS-4943 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 1.2.0, 1.1.2, 2.0.4-alpha >Reporter: Jerry He >Assignee: Jerry He >Priority: Minor > Attachments: HDFS-4943-trunk.patch, HDFS-4943-trunk-v2.patch > > > In HBase, the WAL (hlog) file name on hdfs is URL encoded. For example, > hdtest010%2C60020%2C1371000602151.1371058984668 > When we use webhdfs client to access the hlog file via httpfs, it does not > work in this case. > $ hadoop fs -ls hdfs:///user/biadmin/hbase_hlogs > > Found 1 items > -rw-r--r-- 3 biadmin supergroup 15049470 2013-06-12 10:45 > /user/biadmin/hbase_hlogs/hdtest010%2C60020%2C1371000602151.1371058984668 > $ hadoop fs -ls > hdfs:///user/biadmin/hbase_hlogs/hdtest010%2C60020%2C1371000602151.1371058984668 > Found 1 items > -rw-r--r-- 3 biadmin supergroup 15049470 2013-06-12 10:45 > /user/biadmin/hbase_hlogs/hdtest010%2C60020%2C1371000602151.1371058984668 > $ hadoop fs -ls webhdfs://hdtest010:14000/user/biadmin/hbase_hlogs > Found 1 items > -rw-r--r-- 3 biadmin supergroup 15049470 2013-06-12 10:45 > /user/biadmin/hbase_hlogs/hdtest010%2C60020%2C1371000602151.1371058984668 > $ > $ hadoop fs -ls > webhdfs://hdtest010:14000/user/biadmin/hbase_hlogs/hdtest010%2C60020%2C1371000602151.1371058984668 > 13/06/27 18:36:08 DEBUG web.WebHdfsFileSystem: Original exception is > org.apache.hadoop.ipc.RemoteException:java.io.FileNotFoundException:File does > not exist: > /user/biadmin/hbase_hlogs/hdtest010,60020,1371000602151.1371058984668 > at > org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:114) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:299) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$500(WebHdfsFileSystem.java:104) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem$Runner.getResponse(WebHdfsFileSystem.java:641) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem$Runner.run(WebHdfsFileSystem.java:538) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem.run(WebHdfsFileSystem.java:468) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus(WebHdfsFileSystem.java:662) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileStatus(WebHdfsFileSystem.java:673) > at org.apache.hadoop.fs.FileSystem.getFileStatus(FileSystem.java:1365) > at > org.apache.hadoop.fs.FileSystem.globStatusInternal(FileSystem.java:1048) > at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:987) > at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:965) > at org.apache.hadoop.fs.FsShell.ls(FsShell.java:573) > at org.apache.hadoop.fs.FsShell.doall(FsShell.java:1571) > at org.apache.hadoop.fs.FsShell.run(FsShell.java:1789) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) > at org.apache.hadoop.fs.FsShell.main(FsShell.java:1895) > ls: Cannot access > webhdfs://hdtest010:14000/user/biadmin/hbase_hlogs/hdtest010%2C60020%2C1371000602151.1371058984668: > No such file or directory. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4762) Provide HDFS based NFSv3 and Mountd implementation
[ https://issues.apache.org/jira/browse/HDFS-4762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-4762: - Component/s: nfs Hadoop Flags: Reviewed +1 the latest patch looks good. > Provide HDFS based NFSv3 and Mountd implementation > -- > > Key: HDFS-4762 > URL: https://issues.apache.org/jira/browse/HDFS-4762 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Affects Versions: 3.0.0 >Reporter: Brandon Li >Assignee: Brandon Li > Attachments: HDFS-4762.patch, HDFS-4762.patch.2, HDFS-4762.patch.3, > HDFS-4762.patch.3, HDFS-4762.patch.4, HDFS-4762.patch.5, HDFS-4762.patch.6, > HDFS-4762.patch.7 > > > This is to track the implementation of NFSv3 to HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4762) Provide HDFS based NFSv3 and Mountd implementation
[
https://issues.apache.org/jira/browse/HDFS-4762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13697828#comment-13697828
]
Hadoop QA commented on HDFS-4762:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12590437/HDFS-4762.patch.7
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 7 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-nfs hadoop-hdfs-project/hadoop-hdfs-nfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4587//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4587//console
This message is automatically generated.
> Provide HDFS based NFSv3 and Mountd implementation
> --
>
> Key: HDFS-4762
> URL: https://issues.apache.org/jira/browse/HDFS-4762
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Brandon Li
>Assignee: Brandon Li
> Attachments: HDFS-4762.patch, HDFS-4762.patch.2, HDFS-4762.patch.3,
> HDFS-4762.patch.3, HDFS-4762.patch.4, HDFS-4762.patch.5, HDFS-4762.patch.6,
> HDFS-4762.patch.7
>
>
> This is to track the implementation of NFSv3 to HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4762) Provide HDFS based NFSv3 and Mountd implementation
[
https://issues.apache.org/jira/browse/HDFS-4762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brandon Li updated HDFS-4762:
-
Attachment: HDFS-4762.patch.7
{quote}The new patch still has "if (dumpFile.createNewFile())".{quote}
Sorry. Fixed in the new patch. Thanks!
> Provide HDFS based NFSv3 and Mountd implementation
> --
>
> Key: HDFS-4762
> URL: https://issues.apache.org/jira/browse/HDFS-4762
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Brandon Li
>Assignee: Brandon Li
> Attachments: HDFS-4762.patch, HDFS-4762.patch.2, HDFS-4762.patch.3,
> HDFS-4762.patch.3, HDFS-4762.patch.4, HDFS-4762.patch.5, HDFS-4762.patch.6,
> HDFS-4762.patch.7
>
>
> This is to track the implementation of NFSv3 to HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4762) Provide HDFS based NFSv3 and Mountd implementation
[ https://issues.apache.org/jira/browse/HDFS-4762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13697797#comment-13697797 ] Tsz Wo (Nicholas), SZE commented on HDFS-4762: -- > I removed "if (!dumpFile.createNewFile())" ... The new patch still has "if (dumpFile.createNewFile())". +1 on the patch once the if-statement is removed. > Provide HDFS based NFSv3 and Mountd implementation > -- > > Key: HDFS-4762 > URL: https://issues.apache.org/jira/browse/HDFS-4762 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Brandon Li >Assignee: Brandon Li > Attachments: HDFS-4762.patch, HDFS-4762.patch.2, HDFS-4762.patch.3, > HDFS-4762.patch.3, HDFS-4762.patch.4, HDFS-4762.patch.5, HDFS-4762.patch.6 > > > This is to track the implementation of NFSv3 to HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4860) Add additional attributes to JMX beans
[ https://issues.apache.org/jira/browse/HDFS-4860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Trevor Lorimer updated HDFS-4860: - Status: Open (was: Patch Available) > Add additional attributes to JMX beans > -- > > Key: HDFS-4860 > URL: https://issues.apache.org/jira/browse/HDFS-4860 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.0.4-alpha, 0.20.204.1, 3.0.0, 2.1.0-beta >Reporter: Trevor Lorimer > Attachments: HDFS-4860.diff > > > Currently the JMX bean returns much of the data contained on the HDFS Health > webpage (dfsHealth.html). However there are several other attributes that are > required to be added. > I intend to add the following items to the appropriate bean in parenthesis : > Started time (NameNodeInfo), > Compiled info (NameNodeInfo), > Jvm MaxHeap, MaxNonHeap (JvmMetrics) > Node Usage stats (i.e. Min, Median, Max, stdev) (NameNodeInfo), > Count of decommissioned Live and Dead nodes (FSNamesystemState), > Journal Status (NodeNameInfo) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4860) Add additional attributes to JMX beans
[ https://issues.apache.org/jira/browse/HDFS-4860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Trevor Lorimer updated HDFS-4860: - Status: Patch Available (was: Open) > Add additional attributes to JMX beans > -- > > Key: HDFS-4860 > URL: https://issues.apache.org/jira/browse/HDFS-4860 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.0.4-alpha, 0.20.204.1, 3.0.0, 2.1.0-beta >Reporter: Trevor Lorimer > Attachments: HDFS-4860.diff > > > Currently the JMX bean returns much of the data contained on the HDFS Health > webpage (dfsHealth.html). However there are several other attributes that are > required to be added. > I intend to add the following items to the appropriate bean in parenthesis : > Started time (NameNodeInfo), > Compiled info (NameNodeInfo), > Jvm MaxHeap, MaxNonHeap (JvmMetrics) > Node Usage stats (i.e. Min, Median, Max, stdev) (NameNodeInfo), > Count of decommissioned Live and Dead nodes (FSNamesystemState), > Journal Status (NodeNameInfo) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4860) Add additional attributes to JMX beans
[ https://issues.apache.org/jira/browse/HDFS-4860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Trevor Lorimer updated HDFS-4860: - Attachment: (was: 0002-HDFS-4860.patch) > Add additional attributes to JMX beans > -- > > Key: HDFS-4860 > URL: https://issues.apache.org/jira/browse/HDFS-4860 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 0.20.204.1, 3.0.0, 2.1.0-beta, 2.0.4-alpha >Reporter: Trevor Lorimer > Attachments: HDFS-4860.diff > > > Currently the JMX bean returns much of the data contained on the HDFS Health > webpage (dfsHealth.html). However there are several other attributes that are > required to be added. > I intend to add the following items to the appropriate bean in parenthesis : > Started time (NameNodeInfo), > Compiled info (NameNodeInfo), > Jvm MaxHeap, MaxNonHeap (JvmMetrics) > Node Usage stats (i.e. Min, Median, Max, stdev) (NameNodeInfo), > Count of decommissioned Live and Dead nodes (FSNamesystemState), > Journal Status (NodeNameInfo) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4860) Add additional attributes to JMX beans
[ https://issues.apache.org/jira/browse/HDFS-4860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Trevor Lorimer updated HDFS-4860: - Attachment: HDFS-4860.diff > Add additional attributes to JMX beans > -- > > Key: HDFS-4860 > URL: https://issues.apache.org/jira/browse/HDFS-4860 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 0.20.204.1, 3.0.0, 2.1.0-beta, 2.0.4-alpha >Reporter: Trevor Lorimer > Attachments: HDFS-4860.diff > > > Currently the JMX bean returns much of the data contained on the HDFS Health > webpage (dfsHealth.html). However there are several other attributes that are > required to be added. > I intend to add the following items to the appropriate bean in parenthesis : > Started time (NameNodeInfo), > Compiled info (NameNodeInfo), > Jvm MaxHeap, MaxNonHeap (JvmMetrics) > Node Usage stats (i.e. Min, Median, Max, stdev) (NameNodeInfo), > Count of decommissioned Live and Dead nodes (FSNamesystemState), > Journal Status (NodeNameInfo) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4887) TestNNThroughputBenchmark exits abruptly
[ https://issues.apache.org/jira/browse/HDFS-4887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13697792#comment-13697792 ] Kihwal Lee commented on HDFS-4887: -- I first tried using ExitUtil, but this test case and potentially others want to stop ReplicationMonitor thread and the rest of the system to be up. If NN exits, BlockManager is torn down and any subsequent checks by tests will race. > TestNNThroughputBenchmark exits abruptly > > > Key: HDFS-4887 > URL: https://issues.apache.org/jira/browse/HDFS-4887 > Project: Hadoop HDFS > Issue Type: Bug > Components: benchmarks, test >Affects Versions: 3.0.0, 2.1.0-beta >Reporter: Kihwal Lee >Assignee: Kihwal Lee > Attachments: HDFS-4887.patch, HDFS-4887.patch > > > After HDFS-4840, TestNNThroughputBenchmark exits in the middle. This is > because ReplicationMonitor is being stopped while NN is still running. > This is only valid during testing. In normal cases, ReplicationMonitor thread > runs all the time once started. In standby or safemode, it just skips > calculating DN work. I think NNThroughputBenchmark needs to use ExitUtil to > prevent termination, rather than modifying ReplicationMonitor. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4887) TestNNThroughputBenchmark exits abruptly
[
https://issues.apache.org/jira/browse/HDFS-4887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13697789#comment-13697789
]
Daryn Sharp commented on HDFS-4887:
---
There are methods called {{disableSystem(Exit|Halt)}} and
{{(get|reset)First(Exit|Halt)Exception}} which appear to be designed for
testing purposes. Could these be used in lieu of another testing specific
change?
> TestNNThroughputBenchmark exits abruptly
>
>
> Key: HDFS-4887
> URL: https://issues.apache.org/jira/browse/HDFS-4887
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: benchmarks, test
>Affects Versions: 3.0.0, 2.1.0-beta
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
> Attachments: HDFS-4887.patch, HDFS-4887.patch
>
>
> After HDFS-4840, TestNNThroughputBenchmark exits in the middle. This is
> because ReplicationMonitor is being stopped while NN is still running.
> This is only valid during testing. In normal cases, ReplicationMonitor thread
> runs all the time once started. In standby or safemode, it just skips
> calculating DN work. I think NNThroughputBenchmark needs to use ExitUtil to
> prevent termination, rather than modifying ReplicationMonitor.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4882) Namenode LeaseManager checkLeases() runs into infinite loop
[
https://issues.apache.org/jira/browse/HDFS-4882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13697787#comment-13697787
]
Zesheng Wu commented on HDFS-4882:
--
{quote}
File will not be finalized until it reached at least min replication blocks.
But block replication can be started once it is committed to DN.(here each DN
will finalize its block and report NN). On reaching min replication NN will
complete that block. If all Blocks in NN are in complete state then file can be
closed normally.
{quote}
Yes, I think you are right. But the block replication is not committed to DN,
so is not started.
> Namenode LeaseManager checkLeases() runs into infinite loop
> ---
>
> Key: HDFS-4882
> URL: https://issues.apache.org/jira/browse/HDFS-4882
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client, namenode
>Affects Versions: 2.0.0-alpha
>Reporter: Zesheng Wu
> Attachments: 4882.1.patch, 4882.patch, 4882.patch
>
>
> Scenario:
> 1. cluster with 4 DNs
> 2. the size of the file to be written is a little more than one block
> 3. write the first block to 3 DNs, DN1->DN2->DN3
> 4. all the data packets of first block is successfully acked and the client
> sets the pipeline stage to PIPELINE_CLOSE, but the last packet isn't sent out
> 5. DN2 and DN3 are down
> 6. client recovers the pipeline, but no new DN is added to the pipeline
> because of the current pipeline stage is PIPELINE_CLOSE
> 7. client continuously writes the last block, and try to close the file after
> written all the data
> 8. NN finds that the penultimate block doesn't has enough replica(our
> dfs.namenode.replication.min=2), and the client's close runs into indefinite
> loop(HDFS-2936), and at the same time, NN makes the last block's state to
> COMPLETE
> 9. shutdown the client
> 10. the file's lease exceeds hard limit
> 11. LeaseManager realizes that and begin to do lease recovery by call
> fsnamesystem.internalReleaseLease()
> 12. but the last block's state is COMPLETE, and this triggers lease manager's
> infinite loop and prints massive logs like this:
> {noformat}
> 2013-06-05,17:42:25,695 INFO
> org.apache.hadoop.hdfs.server.namenode.LeaseManager: Lease [Lease. Holder:
> DFSClient_NONMAPREDUCE_-1252656407_1, pendingcreates: 1] has expired hard
> limit
> 2013-06-05,17:42:25,695 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Recovering lease=[Lease.
> Holder: DFSClient_NONMAPREDUCE_-1252656407_1, pendingcreates: 1], src=
> /user/h_wuzesheng/test.dat
> 2013-06-05,17:42:25,695 WARN org.apache.hadoop.hdfs.StateChange: DIR*
> NameSystem.internalReleaseLease: File = /user/h_wuzesheng/test.dat, block
> blk_-7028017402720175688_1202597,
> lastBLockState=COMPLETE
> 2013-06-05,17:42:25,695 INFO
> org.apache.hadoop.hdfs.server.namenode.LeaseManager: Started block recovery
> for file /user/h_wuzesheng/test.dat lease [Lease. Holder: DFSClient_NONM
> APREDUCE_-1252656407_1, pendingcreates: 1]
> {noformat}
> (the 3rd line log is a debug log added by us)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4882) Namenode LeaseManager checkLeases() runs into infinite loop
[
https://issues.apache.org/jira/browse/HDFS-4882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13697784#comment-13697784
]
Zesheng Wu commented on HDFS-4882:
--
1. {quote}What you mean by 'the client triggers the data replication'?{quote}
I mean let the client goto the following replica transfer process:
{code}
//transfer replica
final DatanodeInfo src = d == 0? nodes[1]: nodes[d - 1];
final DatanodeInfo[] targets = {nodes[d]};
transfer(src, targets, lb.getBlockToken());
{code}
> Namenode LeaseManager checkLeases() runs into infinite loop
> ---
>
> Key: HDFS-4882
> URL: https://issues.apache.org/jira/browse/HDFS-4882
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client, namenode
>Affects Versions: 2.0.0-alpha
>Reporter: Zesheng Wu
> Attachments: 4882.1.patch, 4882.patch, 4882.patch
>
>
> Scenario:
> 1. cluster with 4 DNs
> 2. the size of the file to be written is a little more than one block
> 3. write the first block to 3 DNs, DN1->DN2->DN3
> 4. all the data packets of first block is successfully acked and the client
> sets the pipeline stage to PIPELINE_CLOSE, but the last packet isn't sent out
> 5. DN2 and DN3 are down
> 6. client recovers the pipeline, but no new DN is added to the pipeline
> because of the current pipeline stage is PIPELINE_CLOSE
> 7. client continuously writes the last block, and try to close the file after
> written all the data
> 8. NN finds that the penultimate block doesn't has enough replica(our
> dfs.namenode.replication.min=2), and the client's close runs into indefinite
> loop(HDFS-2936), and at the same time, NN makes the last block's state to
> COMPLETE
> 9. shutdown the client
> 10. the file's lease exceeds hard limit
> 11. LeaseManager realizes that and begin to do lease recovery by call
> fsnamesystem.internalReleaseLease()
> 12. but the last block's state is COMPLETE, and this triggers lease manager's
> infinite loop and prints massive logs like this:
> {noformat}
> 2013-06-05,17:42:25,695 INFO
> org.apache.hadoop.hdfs.server.namenode.LeaseManager: Lease [Lease. Holder:
> DFSClient_NONMAPREDUCE_-1252656407_1, pendingcreates: 1] has expired hard
> limit
> 2013-06-05,17:42:25,695 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Recovering lease=[Lease.
> Holder: DFSClient_NONMAPREDUCE_-1252656407_1, pendingcreates: 1], src=
> /user/h_wuzesheng/test.dat
> 2013-06-05,17:42:25,695 WARN org.apache.hadoop.hdfs.StateChange: DIR*
> NameSystem.internalReleaseLease: File = /user/h_wuzesheng/test.dat, block
> blk_-7028017402720175688_1202597,
> lastBLockState=COMPLETE
> 2013-06-05,17:42:25,695 INFO
> org.apache.hadoop.hdfs.server.namenode.LeaseManager: Started block recovery
> for file /user/h_wuzesheng/test.dat lease [Lease. Holder: DFSClient_NONM
> APREDUCE_-1252656407_1, pendingcreates: 1]
> {noformat}
> (the 3rd line log is a debug log added by us)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4888) Refactor and fix FSNamesystem.getTurnOffTip to sanity
[ https://issues.apache.org/jira/browse/HDFS-4888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13697775#comment-13697775 ] Hudson commented on HDFS-4888: -- Integrated in Hadoop-Mapreduce-trunk #1475 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1475/]) HDFS-4888. Refactor and fix FSNamesystem.getTurnOffTip. Contributed by Ravi Prakash. (Revision 1498665) Result = FAILURE kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1498665 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSafeMode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHASafeMode.java > Refactor and fix FSNamesystem.getTurnOffTip to sanity > - > > Key: HDFS-4888 > URL: https://issues.apache.org/jira/browse/HDFS-4888 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0, 2.0.4-alpha, 0.23.9 >Reporter: Ravi Prakash >Assignee: Ravi Prakash > Fix For: 3.0.0, 2.1.0-beta > > Attachments: HDFS-4888.patch, HDFS-4888.patch, HDFS-4888.patch > > > e.g. When resources are low, the command to leave safe mode is not printed. > This method is unnecessarily complex -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4888) Refactor and fix FSNamesystem.getTurnOffTip to sanity
[ https://issues.apache.org/jira/browse/HDFS-4888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13697760#comment-13697760 ] Hudson commented on HDFS-4888: -- Integrated in Hadoop-Hdfs-trunk #1448 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1448/]) HDFS-4888. Refactor and fix FSNamesystem.getTurnOffTip. Contributed by Ravi Prakash. (Revision 1498665) Result = FAILURE kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1498665 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSafeMode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHASafeMode.java > Refactor and fix FSNamesystem.getTurnOffTip to sanity > - > > Key: HDFS-4888 > URL: https://issues.apache.org/jira/browse/HDFS-4888 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0, 2.0.4-alpha, 0.23.9 >Reporter: Ravi Prakash >Assignee: Ravi Prakash > Fix For: 3.0.0, 2.1.0-beta > > Attachments: HDFS-4888.patch, HDFS-4888.patch, HDFS-4888.patch > > > e.g. When resources are low, the command to leave safe mode is not printed. > This method is unnecessarily complex -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4882) Namenode LeaseManager checkLeases() runs into infinite loop
[
https://issues.apache.org/jira/browse/HDFS-4882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13697743#comment-13697743
]
Uma Maheswara Rao G commented on HDFS-4882:
---
{quote}
When there's DN in the pipeline down and the pipeline stage is PIPELINE_CLOSE,
the client triggers the data replication, do not wait the NN to do this(NN
needs the file be finalized to do the replication, but finalized need all the
blocks have at least dfs.namenode.replication.min(=2) replicas, these two
conditions are contradicting).
{quote}
What you mean by 'the client triggers the data replication'?
File will not be finalized until it reached at least min replication blocks.
But block replication can be started once it is committed to DN.(here each DN
will finalize its block and report NN). On reaching min replication NN will
complete that block. If all Blocks in NN are in complete state then file can be
closed normally.
> Namenode LeaseManager checkLeases() runs into infinite loop
> ---
>
> Key: HDFS-4882
> URL: https://issues.apache.org/jira/browse/HDFS-4882
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client, namenode
>Affects Versions: 2.0.0-alpha
>Reporter: Zesheng Wu
> Attachments: 4882.1.patch, 4882.patch, 4882.patch
>
>
> Scenario:
> 1. cluster with 4 DNs
> 2. the size of the file to be written is a little more than one block
> 3. write the first block to 3 DNs, DN1->DN2->DN3
> 4. all the data packets of first block is successfully acked and the client
> sets the pipeline stage to PIPELINE_CLOSE, but the last packet isn't sent out
> 5. DN2 and DN3 are down
> 6. client recovers the pipeline, but no new DN is added to the pipeline
> because of the current pipeline stage is PIPELINE_CLOSE
> 7. client continuously writes the last block, and try to close the file after
> written all the data
> 8. NN finds that the penultimate block doesn't has enough replica(our
> dfs.namenode.replication.min=2), and the client's close runs into indefinite
> loop(HDFS-2936), and at the same time, NN makes the last block's state to
> COMPLETE
> 9. shutdown the client
> 10. the file's lease exceeds hard limit
> 11. LeaseManager realizes that and begin to do lease recovery by call
> fsnamesystem.internalReleaseLease()
> 12. but the last block's state is COMPLETE, and this triggers lease manager's
> infinite loop and prints massive logs like this:
> {noformat}
> 2013-06-05,17:42:25,695 INFO
> org.apache.hadoop.hdfs.server.namenode.LeaseManager: Lease [Lease. Holder:
> DFSClient_NONMAPREDUCE_-1252656407_1, pendingcreates: 1] has expired hard
> limit
> 2013-06-05,17:42:25,695 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Recovering lease=[Lease.
> Holder: DFSClient_NONMAPREDUCE_-1252656407_1, pendingcreates: 1], src=
> /user/h_wuzesheng/test.dat
> 2013-06-05,17:42:25,695 WARN org.apache.hadoop.hdfs.StateChange: DIR*
> NameSystem.internalReleaseLease: File = /user/h_wuzesheng/test.dat, block
> blk_-7028017402720175688_1202597,
> lastBLockState=COMPLETE
> 2013-06-05,17:42:25,695 INFO
> org.apache.hadoop.hdfs.server.namenode.LeaseManager: Started block recovery
> for file /user/h_wuzesheng/test.dat lease [Lease. Holder: DFSClient_NONM
> APREDUCE_-1252656407_1, pendingcreates: 1]
> {noformat}
> (the 3rd line log is a debug log added by us)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1539) prevent data loss when a cluster suffers a power loss
[ https://issues.apache.org/jira/browse/HDFS-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13697735#comment-13697735 ] Dave Latham commented on HDFS-1539: --- Does anyone have any performance numbers for enabling this? Or, does anyone just have some experience running this on significant workloads in production? (Especially HBase?) > prevent data loss when a cluster suffers a power loss > - > > Key: HDFS-1539 > URL: https://issues.apache.org/jira/browse/HDFS-1539 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, hdfs-client, namenode >Reporter: dhruba borthakur >Assignee: dhruba borthakur > Fix For: 0.23.0, 1.1.1 > > Attachments: syncOnClose1.txt, syncOnClose2_b-1.txt, syncOnClose2.txt > > > we have seen an instance where a external outage caused many datanodes to > reboot at around the same time. This resulted in many corrupted blocks. > These were recently written blocks; the current implementation of HDFS > Datanodes do not sync the data of a block file when the block is closed. > 1. Have a cluster-wide config setting that causes the datanode to sync a > block file when a block is finalized. > 2. Introduce a new parameter to the FileSystem.create() to trigger the new > behaviour, i.e. cause the datanode to sync a block-file when it is finalized. > 3. Implement the FSDataOutputStream.hsync() to cause all data written to the > specified file to be written to stable storage. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4837) Allow DFSAdmin to run when HDFS is not the default file system
[ https://issues.apache.org/jira/browse/HDFS-4837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13697715#comment-13697715 ] Uma Maheswara Rao G commented on HDFS-4837: --- Whether this is working with HA namenodes? if nameServiceAddress available, directly it will create connection right? Otherwise anyway this looks to be same as FileSystem.getDefaultUri(conf) and create address. I think having default behaviour only more correct I feel, As your installation default FS says something else. So, if you want to pass uri explicitely also, you can pass your specific configurations via command line. This is service address used by DN, backup node...etc. Could you please have a test with HA namenodes and confirm befor providing revised patch? ALso take care of calling getDFS in setBalancerBandwidth instead of getFS. Regards, Uma > Allow DFSAdmin to run when HDFS is not the default file system > -- > > Key: HDFS-4837 > URL: https://issues.apache.org/jira/browse/HDFS-4837 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Mostafa Elhemali >Assignee: Mostafa Elhemali > Attachments: HDFS-4837.patch > > > When Hadoop is running a different default file system than HDFS, but still > have HDFS namenode running, we are unable to run dfsadmin commands. > I suggest that DFSAdmin use the same mechanism as NameNode does today to get > its address: look at dfs.namenode.rpc-address, and if not set fallback on > getting it from the default file system. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4888) Refactor and fix FSNamesystem.getTurnOffTip to sanity
[ https://issues.apache.org/jira/browse/HDFS-4888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13697683#comment-13697683 ] Hudson commented on HDFS-4888: -- Integrated in Hadoop-Yarn-trunk #258 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/258/]) HDFS-4888. Refactor and fix FSNamesystem.getTurnOffTip. Contributed by Ravi Prakash. (Revision 1498665) Result = FAILURE kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1498665 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSafeMode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHASafeMode.java > Refactor and fix FSNamesystem.getTurnOffTip to sanity > - > > Key: HDFS-4888 > URL: https://issues.apache.org/jira/browse/HDFS-4888 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0, 2.0.4-alpha, 0.23.9 >Reporter: Ravi Prakash >Assignee: Ravi Prakash > Fix For: 3.0.0, 2.1.0-beta > > Attachments: HDFS-4888.patch, HDFS-4888.patch, HDFS-4888.patch > > > e.g. When resources are low, the command to leave safe mode is not printed. > This method is unnecessarily complex -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4504) DFSOutputStream#close doesn't always release resources (such as leases)
[
https://issues.apache.org/jira/browse/HDFS-4504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13697624#comment-13697624
]
Uma Maheswara Rao G commented on HDFS-4504:
---
Thanks Colin, for working on this issue.
Just to summarize:
Per my understanding here there are 2 issues, 1.leaving stale refernces when
there is failures in close call. 2. For long lived client, if cmpleFile fails,
no one will recover is as client will renewLease
for #1, fix would little straight forward.
for #2, Kihwal brought some cases above.
{quote}
•Extend complete() by adding an optional boolean arg, "force". Things will stay
compatible. If a new client is talking to an old NN, the file may not get
completed right away, but this is no worse than current behavior. The client
(lease renewer) can keep trying periodically. Probably less often than the
lease renewal. We may only allow this when lastBlock is present, since the
acked block length will reduce the risk of truncating valid data.
{quote}
Since the current close call already closes streamer, where we maintain this
last block? you mean we will introduce another structure for it and check
periodially in renewer/anyother thread?
(or) How about checking the filesBeingWritten file state. If the
FileBeingWritten state is closed from Clinet perspective but
completFile/flushbuffer failed. So, we will not remove that references staright
away from DFsClient. In this case, Renewer will check such files(closed) and
check real file status from NN. If the file closed from NN(isFileClosed added
in trunk I guess) , then remove from the fileBeingWritten list directly.
Otherwise make a call ourselves recoverLease (as we know no one is going to do
recover for such files).
> DFSOutputStream#close doesn't always release resources (such as leases)
> ---
>
> Key: HDFS-4504
> URL: https://issues.apache.org/jira/browse/HDFS-4504
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-4504.001.patch, HDFS-4504.002.patch
>
>
> {{DFSOutputStream#close}} can throw an {{IOException}} in some cases. One
> example is if there is a pipeline error and then pipeline recovery fails.
> Unfortunately, in this case, some of the resources used by the
> {{DFSOutputStream}} are leaked. One particularly important resource is file
> leases.
> So it's possible for a long-lived HDFS client, such as Flume, to write many
> blocks to a file, but then fail to close it. Unfortunately, the
> {{LeaseRenewerThread}} inside the client will continue to renew the lease for
> the "undead" file. Future attempts to close the file will just rethrow the
> previous exception, and no progress can be made by the client.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4372) Track NameNode startup progress
[
https://issues.apache.org/jira/browse/HDFS-4372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13697571#comment-13697571
]
Hadoop QA commented on HDFS-4372:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12590366/HDFS-4372.4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4584//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4584//console
This message is automatically generated.
> Track NameNode startup progress
> ---
>
> Key: HDFS-4372
> URL: https://issues.apache.org/jira/browse/HDFS-4372
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: 3.0.0
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HDFS-4372.1.patch, HDFS-4372.2.patch, HDFS-4372.3.patch,
> HDFS-4372.4.patch
>
>
> Track detailed progress information about the steps of NameNode startup to
> enable display to users.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
